flybrain/docs/overview.md

Overview

@flybrain/brain simulates a fruit-fly brain and turns its activity into discrete output channels. It holds four things: a connectome dataset format, a leaky integrate-and-fire (LIF) kernel with reward-modulated plasticity, a population-rate readout, and viewer geometry. It holds no game, no emulator, no reward rules and no server. Those live in the application that embeds it.

The library was extracted from the fly-plays-pokemon prototype so that more than one game demo can share one core. The default configuration reproduces that prototype's kernel bit for bit, and oracle tests hold it there (see verification).

Pipeline

image -> retina drive -> LIF network -> population rates -> decoder -> output channels

Stage	Module	Input	Output
retina drive	model/retina.ts	RGBA frame of any size	one drive value per retina column
LIF network	model/lif.ts	column drive, 1-ms ticks	spikes, per-role rate estimates
population rates	model/lif.ts	spike counts per role	role -> rate record, spikes/s
decoder	readout/decoder.ts	rates plus a calibration baseline	active channel names
output channels	caller, or readout/presets/	channel names	device input (for example a joypad mask)

Two side paths feed the network and never touch the readout:

reward (scalar, per frame) -> plasticity.reinforce() -> gains on selected KC->MBON edges
stimulation (pulse) -> network.stimulate() -> extra drive to the reward_pam role

reinforce() supplies the synthetic learning signal. stimulate() excites an anatomical population and has no causal link to the gain update (fly-plays-pokemon/docs/rewards-learning.md, "Plasticity").

Layer map

• dataset (src/dataset/): the artifact format, validation and fingerprinting. format.ts defines BrainDataset; load-node.ts and load-browser.ts are platform loaders that produce identical arrays and identical fingerprints. See dataset format.
• model (src/model/): lif.ts (the 1-ms kernel), plasticity.ts (reward-modulated STDP), retina.ts (image to column drive), rng.ts (deterministic xorshift), version.ts (version string hashing). See model and plasticity.
• readout (src/readout/): decoder.ts (PopulationDecoder) plus device presets under presets/. See readout.
• agent (src/agent/): NeuralAgent composes a network and a decoder into one per-frame call: a 2,500-ms warm-up with plasticity disabled and zero visual drive, one calibrate() against the resting rates, then per-frame stepping with remainder accumulation of 1000/59.7275 ms so a frame advances the brain by 16 or 17 integer 1-ms ticks. It also owns resetTransients() (the hook a host calls after rolling the game back), exportState() and importState() with rollback on a rejected checkpoint, and compatibility(). envelope.ts is the checksummed binary container (encodeEnvelope/decodeEnvelope) and the chunk helpers that split an AgentState into named typed-array chunks. See integration.
• view (src/view/): ConnectomeView, reachable only through the @flybrain/brain/view subpath because it needs a DOM and WebGL. It renders the fixed orthographic activity map from positions.binz, classes.binz, viewer-edges.binz and circuit-roles.json; classification and colours are options. layout.ts holds the pure helpers (normalizePositions, classifyByRoles).

Design principles

Anatomy-constrained, and roles are labels. Connectivity comes from the FlyWire FAFB Codex v783 export. Role lists (kenyon, mbon, descending, reward_pam, ...) are anatomical annotations from that export, not functions inferred from behaviour. The command_0..7 buckets are a round-robin partition of the descending population and the transmitter sign table is a modeling choice; both are called out as such in tools/README.md.

Synthetic learning signal. The modulator is m = tanh(R) for a caller-supplied scalar R. It is not a fitted model of dopamine release and the plastic sites are anatomical, not fitted compartments (model/plasticity.ts, class doc comment).

Fixed readout. The decoder calibrates once and then applies fixed ratios, timings and thresholds. No button-level gain adapts. Learning lives in the network (readout/decoder.ts, module doc comment).

Determinism and bit-exact checkpoints. Noise is one xorshift32 stream; membrane state is Float32; spike and eligibility timestamps are Float64 milliseconds. A checkpoint plus the dataset reproduces a run exactly. exportState() and importState() validate every field before writing anything, and a checkpoint is only accepted against a matching kernelVersion(), dataset fingerprint and plasticityVersion().

Default config pinned to the prototype. DEFAULT_LIF_CONFIG and DEFAULT_PLASTICITY_CONFIG carry the original constants, and the default version strings lif-1ms-f64-v2 and fly-kc-mbon-rstdp-v2 are kept verbatim so existing checkpoints stay loadable. Verbatim copies of the prototype's modules live in packages/brain/tests/legacy/ and are used as oracles.