flybrain/tools/README.md

FlyWire artifact builder

build_flywire.py regenerates everything in data/fafb-v783 from the official FlyWire Codex v783 CSV exports. The committed artifacts are the output of one default run, so the build is the provenance record for the data: nothing in data/ is hand-edited.

Provenance

• Codex portal: https://codex.flywire.ai/
• Export bucket: https://storage.googleapis.com/flywire-data/codex/data/fafb/783/
• License: CC BY-NC 4.0 (see data/fafb-v783/ATTRIBUTION.md for the license, citations and the list of modifications).

Five exports are read. fetch_sources() downloads each one into .tools/flywire-v783/ (gitignored, several GB unpacked) and aborts on a checksum mismatch:

source file	sha256
classification.csv.gz	e946b552f4056dfc977707be0674609832c3f64332a22d69dc0d9615e7aae663
connections.csv.gz	d49dd692e59e153aa3c83f5257bfc0eff51247b86d7bb183386c6d1622c70fc9
consolidated_cell_types.csv.gz	8aba246d71dc40361677493629972ce3883048c3d02010adc42bda22962a1a2d
coordinates.csv.gz	14337121f451f98c2576cee72c24409ada5aaf7948b7c7ca8de9040296840e05
column_assignment.csv.gz	bdf4ce7f62cc63493d53eefad3816ff2dfd08b190e97b35a492e0e453df2f0f6

Neuron indices are the positions of root_ids sorted ascending across classification.csv.gz; every array, role list and edge in the dataset is expressed in that index space. Roles are anatomical labels from the Codex annotations, not inferred task functions; the command_<k> buckets and the transmitter sign table are project modeling choices.

Rerunning

The box enforces uv, so invoke Python through it:

uv run python3 tools/build_flywire.py

Options:

• --output DIR — write artifacts somewhere other than data/fafb-v783 (useful for diffing a rebuild against the committed copies before overwriting them).
• --command-buckets N — split the descending population into N round-robin command_<i % N> roles. Default 8, which is what the committed meta.json contains. Changing it changes meta.json only; the connectivity arrays are unaffected. Readout presets expect command_0..command_7, so a non-default value needs a matching decoder config.

A default rerun is reproducible: gzip streams are written with mtime=0 and an empty filename field, JSON is emitted with separators=(",", ":"), meta.roles is sorted by role name and circuit-roles.json keeps its fixed sensory, motor, descending, kenyon, mbon order, with the twenty-two macro_<type> populations appended after them in the order docs/design/macros.md section 11 lists the types. The run prints the edge count, the L1 input count and the circuit-role sizes.

• --macro-roles — rewrite only the macro_<type> roles of an existing circuit-roles.json in --output, with no downloads. The macro populations are a pure function of the mbon and motor lists already in that file (macro_roles()), so this path and a full run agree by construction; services/flysim/crates/flybrain-core/tests/macro_roles.rs checks the committed artifact against the same rule. It is how the committed copy gained the roles without re-deriving 2.7 M edges, and the diff it produced was additive to the byte: the file's first 150,440 bytes are unchanged and the twenty-two roles are appended before the closing braces.

The macro populations name no new neuron, edge or weight -- they are a relabelling of neurons the dataset already carries -- and they are deliberately outside the dataset fingerprint, so a checkpoint written before they existed still loads and flysim --print-compatibility is unchanged. Both loaders put them in meta.roles like any other role, and both fingerprint functions hash the metadata with the macro_* roles removed, which is the one place that exclusion lives.

Verifying a rebuild

meta.json carries the byte length, compressed length and sha256 of every .binz it describes, so a rebuild is self-checking. To confirm the committed tree instead:

cd data/fafb-v783 && sha256sum -c ../../tools/artifact-checksums.txt

Expected digests of the committed artifacts:

artifact	sha256
indptr.binz	d6d42d174e02ca33351a76596fcf69ff1202705cfe2d6bcd227f6c7905c4a8dd
targets.binz	2af5c1eaa58b8cf0bfa95eaed0b7473c7eb8f6a65693822021ec6b76879f4cc2
weights.binz	5babc87800879821d1399de4fc423c3ff29e490b485ad0f194dfb85216900847
positions.binz	3c45c0abc0523344b4359f2c03d00408758850803ea8d0e23aa59513f3e02879
classes.binz	806dcf394ef7dc341d19636703813d9581bf60eae9006c5d1d94425aebdb724b
viewer-edges.binz	1e78af6ee31866440a18f2124bdc2f5cbe9cbe6e7f8e2e352c6467426740439b
visual-indices.binz	35e689285e7c2edcd7646bddf891cea2d9163a378641f962d87c0016879c8531
visual-hemisphere.binz	261e49d942c6e6d2cadd0b03e06fe7a1cec801bf30b37e6deacf15ebfb117dea
visual-xy.binz	1212951beeb2496a37c83541e8b425b9f90a57774c6d43c62de9d23df43e8ea9
meta.json	d2c2cddfea686cf7691fe30cf2df8f95bc0fa9eef3b4f2ede62c759b28ea88f5
circuit-roles.json	0994d1346df1a06c7c3f268f5ab179fb2646e21bf5672afb6e695eb04ceeb8a9

A default build yields 139,255 neurons, 2,700,513 edges and 1,572 L1 retina columns.

The library's own fingerprintDataset() hashes the loaded arrays rather than the compressed files, so packages/brain/tests/dataset.test.ts is the end-to-end check that the artifacts in data/ still decode to the connectome the model was tuned against.

History

build_flywire.py and a separate build_circuit_roles.py were merged into this single script during extraction from the fly-plays-pokemon prototype. The circuit roles are now accumulated in the same pass over the sorted root_id list that builds meta.json, which is what made the two scripts agree on neuron indices in the first place.