Every number, with where it came from.
A series is a plain CSV in a folder of its topic and publisher. Each row says what was measured, when, by which source, and which program read it. A keeper adds the new values each night and logs what it did.
Last night's keeper, as it logged itself.
The statistics keeper finished at 02:44 UTC on dell-gb10-1, after 4 minutes 11 seconds. It read 213 series from their publishers: 11 new values in one series, 210 unchanged, and 2 BLS series that answered HTTP 503. The failures are in the log, not hidden.
The log is one JSON line; it is set out here a field a row.
02:44:30.421 job statistics command: node scripts/seed/run.mjs statistics dev --apply 02:44:30.421 counts series=213 added=11 unchanged=210 failed=2 appends=1 bytes=11193 02:44:30.421 error bls:JTS000000000000000JOL: https://api.bls.gov/publicAPI/v2/timeseries/data/JTS000000000000000JOL answered HTTP 503 02:44:30.421 error bls:JTS000000000000000QUR: https://api.bls.gov/publicAPI/v2/timeseries/data/JTS000000000000000QUR answered HTTP 503 02:44:30.421 event exit=0 ms=251203 host=Dell-gb10-1 reader=program:scripts/seed/data-keepers.mjs
Counted in the Drive
2,763 series files, 35.5 GB.
The statistics on dev, by topic. Most came in through the world-model import (BEA, the census, V-Dem, USDA, USGS); the rest the keepers read from their publishers. The daily stock bars are private by their terms and counted only. The series the keepers follow are declared in 65 files of CommandAGI's Drive: 22 from FRED, the rest from the ECB, BLS, the Bank of England and others.
- CSV files on dev
- 2,763
- of rows, each with its source and reader
- 35.5 GB
- declaration files of followed series, in production
- 65
- series the keeper read last night
- 213
cloud://home/user-commandagi/statistics/ (dev); series/ (production)Output case-studies/statistics/stats.jsonTool scripts/sites/case-studies.mjs statisticsRan on This computer (dell-gb10-1), reading the dev and production DrivesWhen Ask for a series by its id.
An agent asks for fred:CPIAUCSL or wdi:NY.GDP.MKTP.CD; the series is read on first demand and kept from then on.