CommandAGI · commandagi.com/for/world-model-import

CommandAGI
ワークベンチを開く
メニュー
ワークベンチを開くログイン

The world-model import

A world model, into plain files.

Each dataset's published stage becomes a few files of each kind, and every row names the stage and the raw input it came from. The import is staged, resumable and paced, and it logs every step.

Open the workbenchSee the datasets

Map, upload, index

Three steps, each one logged.

A dataset is read once from the world-model's drive and mapped to the files it becomes. Each file goes to the Drive whole, with its hash, so a file sent twice is the same file. Then its index by subject is built, so a question about one id reads only that id's rows.

How the world-model import worksEach dataset's published stage on the world-model drive, read only, is mapped into staged files by kind (entities to records/, assertions to relations/, observations to statistics/ and events to events/), each row with its evidence. Each staged file is uploaded whole with its hash, public or private by the dataset's rights, and the ledger records it. Once every table of a dataset is stored, its index by subject is written to indexes/world-model/. Every step is a line of runs.jsonl.Theworld-modeldrive142 datasets · readonlyMaprecords/ · relations/· statistics/ ·events/Uploadone PUT a file, withits hash; public orprivateIndexindexes/world-mo-del/<dataset>/runs.jsonlone line a stepHow the world-model import worksEach dataset's published stage on the world-model drive, read only, is mapped into staged files by kind (entities to records/, assertions to relations/, observations to statistics/ and events to events/), each row with its evidence. Each staged file is uploaded whole with its hash, public or private by the dataset's rights, and the ledger records it. Once every table of a dataset is stored, its index by subject is written to indexes/world-model/. Every step is a line of runs.jsonl.The world-model drive142 datasets · read onlyMaprecords/ · relations/ · statistics/ · events/Uploadone PUT a file, with its hash; public or privateIndexindexes/world-model/<dataset>/runs.jsonlone line a step

The import's own log.

The first lines of today's run, and its latest uploads. A file sent again with the same hash is same, not sent: the second fred_cpi line is a run that found nothing to do.

~/.local/share/commandagi/world-model-import/runs.jsonl (dell-gb10-1)real lines · 2026-11-04 · versions cut to 12 characters
10:07:08.816 map     fred_cpi@34fe03f554da  entity=4 assertion=3 observation=14681 files=6
10:07:11.442 map     congress_people@79e40ffafc41  entity=59146 assertion=255312 files=19
10:07:44.996 upload  dev fred_cpi public  sent=6 same=0 bytes=7681714
10:07:58.202 upload  dev fred_cpi public  sent=0 same=6 bytes=0
12:20:15.692 upload  dev airport_nodes private  sent=22 bytes=329711964
14:43:29.095 upload  dev alpaca_daily_bars private  sent=750 bytes=22442239573
14:52:12.356 upload  dev bea_input_output public  sent=75 bytes=1071284549

What dev holds so far

62.3 million rows, 21 datasets.

The import's ledger at 17:19 UTC, the newest line of each file. The import was still running: bea_national_regional was going up as this was counted. Private datasets (by their terms: daily stock bars, census microdata, airports, conflict events, listings) are stored private and counted only.

rows in 3,238 files, 39.3 GB
62.3 M
of 142 datasets stored on dev
21
indexed by subject: 13 public, 5 private
18
steps logged between 10:07 and 14:52 UTC: 2 maps, 22 uploads, 18 indexes
42
Rows stored on dev by kind
数値
Rows stored on dev by kind
KindRows
observation58305547
assertion2706834
event884496
entity406357
ProvenanceFile ~/.local/share/commandagi/world-model-import/ledger-dev.jsonl, index-dev.jsonl (dell-gb10-1)Output case-studies/world-model-import/stats.jsonTool scripts/statistics/world-model.mjs, counted by scripts/sites/case-studies.mjs world-model-importRan on dell-gb10-1, the computer that runs the importWhen

Ask the world model about one id.

The graph reads a subject's rows through the index, from every dataset that names it.

See the graph explorerSee the datasets