alex/flybrain
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs(design): session-framework specs (bus-v1, ipc-v1, step-v1, workers-v1, publishing-v1, state-media-v1) and the MaleCNS modular-session plans
Some checks failed
ci / node 22 (test + typecheck) (push) Has been cancelled
Details
ci / rust stable (cargo test --workspace --release) (push) Has been cancelled
Details
ci / infra/tests/lint.sh (push) Has been cancelled
Details
ci / playwright apps/stage (allowed to fail) (push) Has been cancelled
Details

Browse source
This commit is contained in:
acamilo 2026-09-22 11:01:55 +00:00
parent 96677119ad
commit 05e93a95ef
11 changed files with 4026 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

268
docs/design/malecns-modular-implementation.md Normal file
View file

@ -0,0 +1,268 @@
# Implementation backlog: MaleCNS and modular sessions
Status: **planned, not started**. Written 2026-09-18. Companion to the
[design and code analysis](malecns-modular-sessions.md), based on `f7bc13a`.
This is the execution order for that proposal. It does not authorize deployment or a live
stream. Reconcile the baseline with merged macro/shop/recovery work before implementation.
Existing feed/control contracts and TypeScript oracle rules remain binding.
Concrete second-game audit: [Melee framework and emulator plan](melee-framework-audit.md).
Its MELEE-01/02 spikes specialize EMULATOR-01 below and can proceed alongside framework
extraction; they do not depend on importing MaleCNS first.
The [session framework contract set](session-framework/README.md) now specifies the new
multi-process architecture. Its [implementation guide](session-framework/implementation.md)
breaks RUNTIME/WIRE/STATE work into concrete schema, transport, coordinator and worker slices.
**Communications decision:** build [Flybus](session-framework/bus-v1.md), one Rust RPC/pub-sub
router with immutable external artifacts, delivery guards and last-owner GC. It serves workers,
application supervision, presentation and storage. No second direct-RPC system or NATS broker.
Application-specific orchestration and frontend are developed together; native frames leave
the environment, while resizing/compositing/streaming belongs to presentation.
## 1. Delivery strategy
Deliver working vertical slices; keep the existing FAFB/Game Boy composition usable throughout.
1. Establish a behavior baseline and explicit dataset/profile identities.
Independently build the generic Flybus example; it needs neither dataset nor emulator.
2. In parallel workstreams, characterize MaleCNS and extract the single-agent session runtime.
3. Demonstrate two isolated brains driving one ROM-free shared environment.
4. Expose that session through versioned feed/control contracts and a multi-agent broadcast.
5. Integrate a specifically chosen alternative emulator after its capability spike passes.
6. Finish physical package reorganization once the second consumer proves the boundaries.
**First milestone:** a reproducible headless MaleCNS run and a reusable single-agent session.
**Second milestone:** two flies in a synthetic arena with coherent resume and a local broadcast.
**Third milestone:** two flies in the chosen fighting game, with documented task scaffolding.
No elapsed-time estimate is committed before the import and emulator spikes establish their
unknowns. Split an item further when its contract and implementation cannot be reviewed together.
## 2. Work queue
Every item starts pending. Branch names are suggested implementation branches, not branches
already created. Builders use separate worktrees; the coordinator reviews contracts and results.
### FOUNDATION-01 — Pin existing behavior
- **Branch:** `test/session-baseline`
- **Depends on:** reconciliation with current main and related open work.
- Record effective legacy configuration, fingerprint, version strings and frame ordering.
- Add a ROM-free transition harness around the service's frame orchestration; capture brain
ticks, decoded actions, reward application and recovery effects with explicit clocks.
- Distinguish exact numerical replay from intentional macro/hold/transient reset on restore.
- **Done:** existing goldens pass; a trace fixture detects reordered vision/reward application;
legacy feed/API fixtures and compatibility identity are unchanged.
### FOUNDATION-02 — Specify bundle and behavior identities
- **Branch:** `feat/brain-profile-contract`
- **Depends on:** FOUNDATION-01.
- Specify dataset manifests, original-ID mapping, anatomical roles, sensory bindings,
readout bindings and composite behavior identity in a focused contract.
- Add profile resolution/validation around the existing core in TypeScript and Rust.
- Keep schema-1 fingerprinting and the legacy macro-role exception behind the legacy path.
- Add strict graph validation for new bundles, shared invalid fixtures and profile mismatch tests.
- **Done:** empty required populations, malformed CSR and incorrect profile restores fail;
legacy FAFB artifacts and default numerical version strings remain unchanged.
### DATA-01 — Acquire and normalize MaleCNS
- **Branch:** `feat/malecns-import`
- **Depends on:** FOUNDATION-02.
- Build a source-specific importer from official v1.0 tables with checksummed source locks.
- Reconcile the Codex versus neuPrint inventories, or explicitly select and document one.
- Preserve raw contact counts, transmitter evidence, original IDs and missing-data indicators.
- Emit deterministic graph bundles, weight≥1/weight≥5 comparison variants, and an exclusion/
clipping/coverage report. Decide schema-1 feasibility from measured weight ranges.
- Add attribution and license records with actual artifacts; keep raw downloads out of git.
- **Done:** repeated builds match; endpoint/role/index invariants pass; both loaders agree;
no service startup or ordinary unit test needs a download.
### DATA-02 — Characterize MaleCNS in the current kernel
- **Branch:** `feat/malecns-baseline-profile`
- **Depends on:** DATA-01.
- Audit L1 geometry, hemisphere handling, KC/MBON/PAM mappings and brain-versus-VNC motor roles.
- Define a versioned fixed readout; keep task action partitions out of anatomical truth.
- Run learning-off first, then learning-on, using fixed sensory traces and multiple seeds.
- Generate TS reference goldens and compare Rust exactly; measure activity, saturation,
initialization, memory and per-phase latency for both graph thresholds.
- **Done:** publish a reproducible characterization report and profile choice. Stop task
integration if required mappings are missing or dynamics are unusable; any recalibration
becomes a named profile rather than an edit to the legacy model.
### RUNTIME-01 — Separate environment execution from task interpretation
- **Branch:** `refactor/environment-task-boundary`
- **Depends on:** FOUNDATION-02.
- Specify controller ports, digital/analog controls, rational cadence, media descriptors,
observation ownership and backend capabilities.
- Wrap binjgb as the first environment; retain Game Boy FFI/cache/state behavior.
- Keep Pokémon memory inspection, objective routing, macros and reward rules in its task.
- The executor receives coherent current game state, progress/objective view and clock on
every step. This richer context is not implicitly passed to the neural sensory encoder.
- Preserve existing imports through a facade; avoid simultaneous directory moves.
- **Done:** existing single-agent action/reward traces match and a fake environment can be
driven through the same boundary without importing binjgb or task-specific addresses.
### RUNTIME-02 — Extract the single-agent session
- **Branch:** `refactor/session-runtime`
- **Depends on:** RUNTIME-01 and BUS-01..03 from the contract implementation guide.
- Move deterministic agent/environment/task orchestration out of `Sim` into a library.
- Keep HTTP, WebSocket serialization, wall-clock publication and process supervision in flysim.
- Route internal worker calls and observations through Flybus; keep public compatibility
adapters at the application edge. Domain caches own artifact handles for replay.
- Give session clock, action executor, task ledger and recovery state explicit owners.
- Wrap the existing composition with legacy ordering, checkpoint and reset semantics.
- **Done:** headless bus client runs a session without Twitch/browser; the legacy composition
passes its traces and restore tests; a slow snapshot consumer cannot stall simulation.
### RUNTIME-03 — Add synchronized multi-agent sessions
- **Branch:** `feat/multi-agent-arena`
- **Depends on:** RUNTIME-02.
- Implement a ROM-free two-player arena and per-port controller ownership.
- Evaluate both brains against one observation boundary; apply one complete action batch;
advance the world once. Start sequentially, then verify parallel execution equivalence.
- Isolate RNG, stimulation, decoder holds, gains, traces and rewards per agent; share only
immutable topology. Enforce a total worker budget and single-dispatcher pool ownership.
- Define participant failure, lateness and episode reset policies.
- **Done:** no cross-agent state leakage; swapping evaluation order leaves results unchanged;
one failed participant cannot accidentally advance a half-controlled match.
### STATE-01 — Capture and resume whole sessions
- **Branch:** `feat/session-checkpoints`
- **Depends on:** RUNTIME-03; specify the state contract during RUNTIME-02.
- Define the new envelope/manifest and preserve the `FLYSIM01` reader.
- Capture all agents, environment, task/executor/admission state and clock remainders at one
boundary. Bound off-thread write jobs and retain atomic manifest commit semantics.
- Validate all components before installing any restored state; define external-backend staging.
- **Done:** uninterrupted and resumed synthetic matches agree; corrupting any participant
refuses the generation without partial restore; crash-injection fallback tests pass.
### WIRE-01 — Introduce session feed/control v2
- **Branch:** `feat/session-protocol-v2`
- **Depends on:** FOUNDATION-02, RUNTIME-02; use RUNTIME-03 fixtures for integration.
- Write binding contracts before consumer implementation: descriptors, scoped agents/events,
media IDs/timestamps, task progress, targeted stimulation and retry/idempotency behavior.
- Implement Rust/TS codecs, schemas and a fake server; preserve the legacy v1 surface.
- Specify descriptor reconnect behavior, asset/index identity, bounded message sizes and audio gaps.
- **Done:** cross-language fixtures pass for unequal neuron counts and shared/private views;
duplicate attachment kinds no longer collide; ambiguous targets and incompatible schemas fail.
### PRESENTATION-01 — Compose multi-agent stage and bridge
- **Branch:** `feat/multi-agent-broadcast`
- **Depends on:** WIRE-01, RUNTIME-03.
- Replace stage store/scaler singletons with session/agent instances and one paint scheduler.
- Combine framework descriptors/measurements with application-owned state/cues over the bus.
Rendering may retain an artifact after message drop; last-use release returns delivery credit.
- Resolve geometry from hashed descriptors, preserve the Game Boy presentation, and add a
shared-match layout with explicit audio ownership.
- Route bridge commands/redemptions to persistent session/agent identities; test lost responses,
retries and restart without applying an interaction twice or to a different agent.
- **Done:** local synthetic match broadcast works; two-agent PNGs receive operator review;
browser/fixture/legibility checks pass; bridge remains template-only and quiet-mode capable.
### DATA-03 — Run MaleCNS through the complete application
- **Branch:** `feat/malecns-session`
- **Depends on:** DATA-02 and descriptor-aware assets from WIRE-01/PRESENTATION-01.
- Expose explicit profile selection and create a fresh MaleCNS state namespace.
- Verify task/controller bindings, stimulation capability and displayed anatomy identity.
- A narrow single-agent descriptor extension may ship earlier only with matching v1 contract
and consumer updates; do not publish MaleCNS spikes as implicit FAFB indices.
- **Done:** local one-hour soak and restore drill pass; paired learning-off/on observations
are recorded without claiming improved play; FAFB remains available unchanged.
### EMULATOR-01 — Establish the alternative backend's capabilities
- **Branch:** `spike/fighting-game-backend`
- **Depends on:** RUNTIME-01; can proceed alongside later runtime work.
- Choose the exact game/version and emulator; Melee/Dolphin is a candidate, not a commitment.
- Prove pause/step, simultaneous ports, analog input, frame/audio capture, state inspection,
save/restore, process lifecycle and achievable cadence with synthetic controller traces.
- Prefer a bus-connected helper if embedding would leak emulator internals into the session
library. Its native emulator protocol is an implementation detail, not a second framework API.
- **Done:** capability report includes pinned backend/content identity and reproducible results.
If bounded stepping or coherent restore fails, stop and revise the backend/requirements
before writing neural game logic. No game content enters repository fixtures.
### EMULATOR-02 — Build the two-fly fighting-game slice
- **Branch:** `feat/two-fly-fighting-game`
- **Depends on:** EMULATOR-01, STATE-01, PRESENTATION-01.
- Implement fixed controller mapping, match/round interpretation, positive attributed rewards,
observation policy and episode recovery. Display selected actions and actual controls.
- Validate one fly, two flies, round transitions, backend failure and resume in that order.
- Run side swaps and repeated seeds; compare learning-off and simple control baselines before
interpreting win rates. Separate show settings from controlled evaluation settings.
- **Done:** repeated local matches sustain declared cadence; restoration/failure policies work;
scaffold, interventions and limits are documented; reviewed match presentation is legible.
### PACKAGE-01 — Finalize reusable packages and release compositions
- **Branch:** `refactor/reusable-package-layout`
- **Depends on:** a useful second backend plus PRESENTATION-01.
- Extract proven crate/package boundaries from the design's module table; preserve facades.
- Move the Rust workspace only in a mechanical follow-up if it makes library consumption clearer.
- Update CI/build/vendor/golden paths, dataset/view asset packaging and compatibility preflight.
- Add minimal external-style Rust/TS consumers and a synthetic example composition.
- Reconcile current docs, stale template explanations and licensing/asset attribution.
- **Done:** both legacy and new compositions package successfully; incompatible state is
rejected before release selection; libraries run without importing broadcast services.
## 3. Dependency map and first execution batch
```text
FOUNDATION-01 → FOUNDATION-02 ┬→ DATA-01 → DATA-02 ───────────────→ DATA-03
└→ RUNTIME-01 → RUNTIME-02 → RUNTIME-03 → STATE-01
│ └→ WIRE-01 ───────┐
└→ EMULATOR-01 PRESENTATION-01
│
STATE-01 + EMULATOR-01 + PRESENTATION-01 → EMULATOR-02
second backend + presentation → PACKAGE-01
```
The item dependency lists are authoritative; the diagram is a reading aid. The detailed
contract guide adds BUS-01 (RPC), BUS-02 (pub/sub) and BUS-03 (artifacts/GC) before the
distributed RUNTIME-02/03 slices. These can proceed independently of MaleCNS import.
Start the preservation track at FOUNDATION-01; the generic bus track can start with the
contract guide's small RPC/pub-sub/artifact example. Then review FOUNDATION-02's contract
before assigning DATA-01 and RUNTIME-01 to independent worktrees.
Contract/schema authorship is serialized to avoid conflicting definitions. Deployment host
work remains serialized under the repository's claim protocol.
## 4. Definition of done for every implementation branch
- Scope and intentional behavior changes are stated; compatibility impact is explicit.
- Meaningful boundary tests cover the changed behavior; the TS oracle is not adjusted to
accommodate Rust output. Existing committed real-data goldens stay mandatory.
- `npm test`, `npm run typecheck`, `cargo test --workspace` (Rust workspace) and
`infra/tests/lint.sh` pass before merge. Visual changes also pass applicable Playwright
checks and PNG review. Optional full-MaleCNS/ROM runs record skips honestly.
- Performance-sensitive changes report representative activity, agent count, thread budget,
memory and tail latency. New experiments state what is modeled versus handwritten.
- Review the complete diff, merge with `--no-ff` when authorized, and update this queue with
commit, evidence and unresolved follow-ups. Rollback includes compatible state, not just code.
## 5. Decisions needed before the relevant work starts
| Decision | Deadline | Default recommendation |
| --- | --- | --- |
| MaleCNS inventory/filter policy | DATA-01 completion | Official versioned source; retain both threshold variants until measured |
| MaleCNS sensory/readout profile | DATA-02 | Audited L1 mapping with existing numerical model first |
| Exact fighting game and backend | EMULATOR-01 | Evaluate one concrete title/backend rather than supporting a console family at once |
| Number of flies and target resource budget | RUNTIME-03 performance gate | Two first; characterize four before promising it |
| Learning retention and sugar in matches | EMULATOR-02 task contract | Retention explicit; stimulation disabled in controlled comparisons |
| Package publication versus monorepo reuse | PACKAGE-01 | Monorepo libraries/examples first; public package publishing later |
There is no need to resolve these now to plan another feature. This backlog is ready for
resumption at FOUNDATION-01 and the independent BUS-01..03 track.

918
docs/design/malecns-modular-sessions.md Normal file
View file

@ -0,0 +1,918 @@
# MaleCNS and reusable streamed simulation sessions
Status: **proposal, not an implemented contract**. Written 2026-09-18 against `f7bc13a`
on `main`. This document covers two related projects: adding MaleCNS v1.0 as another
connectome, and extracting reusable modules for other emulators, embodied environments,
and multiple flies. No dataset, neural semantics, deployed configuration, or wire contract
is changed by this document.
The binding [feed](../feed-protocol.md) and [control](../control-api.md) contracts take
precedence. The TypeScript brain remains the oracle. Existing default versions
`lif-1ms-f64-v2` and `fly-kc-mbon-rstdp-v2` remain pinned.
Reading map: sections 2–3 contain the code audit and MaleCNS analysis; sections 4–6
define the proposed module/session boundaries; sections 7–8 give the extraction order,
implementation workstreams and acceptance gates; sections 9–10 record open questions and
sources.
Execution queue: [implementation backlog](malecns-modular-implementation.md), with branch-sized
deliverables, dependencies and completion criteria. Start at FOUNDATION-01 when work resumes.
Concrete follow-up: [Melee emulator and multi-fly framework audit](melee-framework-audit.md),
including source-checked Dolphin/libmelee integration options and full-stack performance gates.
Implementation contracts: [session framework](session-framework/README.md), defining private
Flybus RPC/pub-sub, lockstep phases, worker methods, artifact ownership, recovery and publication.
The [bus specification](session-framework/bus-v1.md) is the selected communications design:
one small Rust router, external immutable artifacts and delivery-scoped GC. Application
orchestration/presentation are developed together; tournaments are examples, not framework types.
## 1. Recommendation
1. **Add MaleCNS as a dataset/profile combination, not a replacement neural model.**
First run it through the existing LIF semantics with explicit, independently versioned
sensory, population, and readout mappings. Study different neuron dynamics separately.
2. **Make a session the unit of simulation ownership.** A session has one environment and
one or more independently stateful agents bound to its control ports. A shared match
advances once after all players have chosen actions from the same observation boundary.
3. **Extract along ownership and timing boundaries.** Separate anatomy, neural dynamics,
sensor encoding, action decoding, environment execution, task semantics, persistence,
observation transport, presentation, and audience interaction. Preserve current behavior
through a legacy composition while extracting these modules.
4. **Prove the design on a ROM-free two-player arena before a larger emulator.** Then build
a frame-stepped emulator integration. For “flies play Smash,” the first candidate should
be a specifically chosen title/backend, such as Melee with a pinned Dolphin integration;
“Smash” alone is not an emulator requirement.
5. **Retain a monorepo and one Rust workspace initially.** Reusable libraries do not require
a network of microservices, dynamic native plugins, or publishing unstable packages.
The two tracks can progress independently after the identity/profile boundary is established.
MaleCNS does not require multiplayer; multiplayer does not require MaleCNS. The first useful
deliverables are a reproducible MaleCNS characterization run and a behavior-preserving
single-agent session API, not a wholesale rewrite.
## 2. What the code actually does today
Paths below are relative to the repository root. Rust paths beginning `core/`, `gb/`, or
`sim/` in this document abbreviate `services/flysim/crates/flybrain-core/`,
`services/flysim/crates/flybrain-gb/`, and `services/flysim/crates/flysim/` respectively.
These aliases refer to **current** paths, not proposed directories.
| Boundary | Evidence inspected | Consequence |
| --- | --- | --- |
| Reference brain | `packages/brain/src/agent/agent.ts`; `core/src/agent.rs` | `NeuralAgent` composes network and decoder; image size and milliseconds/frame are configurable, but defaults are Game Boy-specific. It is already more reusable than the service. |
| Anatomy | `packages/brain/src/dataset/format.ts`; `core/src/dataset.rs` | CSR graph dimensions are dynamic. Weights are signed `i16`; roles and a two-dimensional visual-column table are part of the model input. Validation currently checks array lengths, not every graph invariant. |
| Dataset construction | `tools/build_flywire.py` | Five checksum-pinned Codex exports; stable indices from sorted root IDs; directed-pair aggregation; transmitter signs; clipping to ±32767; FAFB-specific role and L1-column extraction. Game macro populations are also generated here. |
| Neural dynamics | `core/src/lif.rs`; `packages/brain/src/model/lif.ts` | One-ms ticks, configurable gain/noise, role stimulation, image drive, and up to 64 tracked rate roles. It is not a general multimodal sensory API. |
| Learning | `core/src/plasticity.rs` | Strongest positive pre-role→post-role edges, default KC→MBON budget 16,384; gains/eligibility are per-agent. A caller supplies the reward scalar; PAM spikes do not generate it. |
| Parallelism | `core/src/lif.rs` (`SweepPlan`); `core/src/pool.rs` | Persistent deterministic within-brain pool. `broadcast` has one shared job slot and assumes one dispatcher at a time; cloning a plan is not permission to dispatch it concurrently. |
| Game interface | `gb/src/adapter.rs` | `GameAdapter` mixes reward detection, progress, decoder preset, recovery, ROM checks, and Pokémon-like tile/exit/objective queries. `MemoryReader::read8(u16)` is specifically a Game Boy-shaped interface. |
| Emulator | `gb/src/emulator.rs` | Concrete binjgb wrapper, 160×144 RGBA, eight-bit pad, one frame step, audio conversion, native save-state format. Explicit handles and `Send` allow multiple instances; `Sync` is deliberately absent. |
| Session ownership | `sim/src/simloop.rs` (`Sim`, `step_frame`) | One agent, emulator, adapter, ratchet, button mask, framebuffer, audio queue, sugar state, chat ring, and set of clocks. Application orchestration and game behavior share a large struct. |
| Actions | `sim/src/macros.rs`; `gb/src/macros.rs`; `gb/src/pokemon_red/macros/` | Neural channels select available macros, but the game-specific executor owns button sequences. The title screen uses raw input; scene handling, routing and targets are engineered behavior. |
| Recovery | `gb/src/recovery.rs`; `gb/src/ratchet.rs` | Game-only rewind retains brain clock, membrane, RNG, and gains; clears holds/eligibility and refreshes vision. A scalar progress ladder chooses a best save. This is not a general multiplayer reset policy. |
| Persistence | `sim/src/store.rs`; `gb/src/compatibility.rs` | Durable atomic envelope and manifest commit are reusable. Payload is one agent plus one emulator and ratchet. Compatibility names binjgb and `pokered`; native state identity includes size and target. |
| Public observation | `sim/src/snapshot.rs`; `packages/feed/src/{types,codec}.ts` | One flat brain/game snapshot; one attachment per kind; Game Boy buttons, fixed frame dimensions, Pokémon-shaped reward counters and a closed game-mode set. |
| Stage | `apps/stage/src/{App.tsx,feed/store.ts,feed/decode.ts}` | Good hot/paint/cold clock split, but mutable stores/scalers are singletons and the decoder checks 160×144 frames. Dataset URL is fixed to FAFB. |
| Game presentation | `apps/stage/src/games/` | A useful registry already exists, but config relabels v1 counters rather than declaring independent task schemas. |
| Twitch bridge | `services/bridge/src/{index,sim,commands,redemptions,templates}.ts` | Transport/client abstraction, test fakes, templates, rate limits, and redemption persistence are valuable. There is one sim URL and no agent target identity. |
| Packaging | `apps/stage/vite.config.ts`; `infra/build/package-release.sh`; `infra/05-deploy.sh` | Stage build copies FAFB artifacts; packaging defaults to FAFB; deploy preflights one compatibility string. Runtime/data/frontend are still one release composition. |
### 2.1 Behaviors to preserve before extracting
`Sim::step_frame` advances the brain from the previous visual input, decodes, applies
buttons, steps the emulator, sets the new visual input, samples rewards, stimulates per
reward event, reinforces their sum, updates macro availability, and observes the ratchet.
Control commands are drained before the frame step. This ordering is part of behavior.
Do not replace this with `NeuralAgent::tick` merely because it looks like a convenient
wrapper: the service currently orchestrates substeps to sample reward from the frame just
produced. Moving reward or visual drive across that boundary changes trajectories.
Other invariants:
- Deterministic arithmetic and per-target propagation order, including across thread counts.
- Browser/network clients never stall the sim; snapshots are latest-value/drop-oldest.
- Checkpoint capture is coherent; encoding and storage happen off the sim thread.
- Failed restore does not silently reset a run. Existing legacy restore policies remain exact.
- No public control endpoint for button presses or game-memory writes.
- Sugar and learning reward are distinct mechanisms. Chat text never becomes neural input.
- The current positive-only reward doctrine remains the default for all shipped tasks.
### 2.2 Existing compatibility gaps to handle deliberately
The dataset fingerprint hashes metadata (including anatomical circuit roles) and six arrays.
`macro_*` roles are merged **after** hashing to preserve old checkpoints. Both language
implementations document that changing those populations could restore rates onto different
neurons without invalidating the checkpoint. The kernel parameter version also omits role
names; the service compatibility string does not fully identify the decoder/action mapping.
Keep these historical behaviors in the legacy reader. For new profiles, add an explicit
behavior identity covering population bindings, sensor encoding, decoder configuration,
action executor, and reward catalog. Do not repair the old hash by changing it in place.
Open macro/shop and recovery branches existed when this proposal was written. Before
implementation, rebase the inventory against their merged state and rerun characterization;
this proposal neither incorporates nor supersedes their unmerged changes.
## 3. MaleCNS: anatomy, connectivity, and model are different things
### 3.1 Dataset comparison and evidence limits
| Property | FAFB v783 used here | MaleCNS v1.0 |
| --- | --- | --- |
| Specimen | Adult female | Adult male, independently imaged/reconstructed |
| Territory | Brain including optic lobes | Central brain, optic lobes, ventral nerve cord (VNC), intact neck connective |
| Local artifact | 139,255 neurons; 2,700,513 directed-pair edges | None imported in this repository |
| Available inventory counts | Codex lists 139,255 neurons | Codex lists 166,700; the Minecraft project's neuPrint `:Neuron` export reports 176,422. Selection rules must be reconciled before fixing a local count. |
| Input labels | Codex `root_id`, classification, consolidated types, column assignment | neuPrint/flat export `bodyId`/body IDs, class hierarchy, transmitter properties, sides, neuropils, cross-dataset type annotations |
| Added anatomical opportunity | Brain sensory→descending circuits | Brain↔VNC circuits, local motor circuitry, ascending feedback, additional sensory and motor populations |
| License evidence | Repository attribution: CC BY-NC 4.0 | Official MaleCNS download site: CC-BY; verify and retain the exact release license text when importing |
MaleCNS is not FlyWire with extra neurons appended. IDs and dense array indices do not
correspond. Homologous cell types and registered anatomical spaces enable comparisons, not
automatic one-to-one neuron matching, state transfer, or graph concatenation. Male-specific
and sexually dimorphic circuits make a universal matching assumption especially misleading.
The official MaleCNS site describes a finished, proofread and annotated CNS reconstruction.
This does not mean every synapse, cell type, or sensory column is equally certain. The
Minecraft derivative reports asymmetric visual-column coverage and missing soma positions;
our import must quantify coverage from its own pinned source. A connectome also does not
supply all synaptic physiology, electrical coupling, neuromodulation, body dynamics, or
behavioral competence.
### 3.2 What “connections” means
Separate these quantities in metadata, reports, and on-screen claims:
1. Source neuron/segment inventory and the chosen included-neuron inventory.
2. Individual synaptic contacts/partner pairs (and separately pre-sites and post-sites).
3. Directed neuron-pair edges after aggregation.
4. Retained edges and retained synaptic weight after confidence/weight filtering.
5. Effective signed weights after the simulation's transmitter and clipping policy.
The FAFB builder aggregates export rows by `(pre, post)` across rows, drops endpoints not
in its classification inventory, assigns a sign, and clips the summed magnitude. It adds
no explicit five-synapse threshold of its own. The source exports may already be filtered;
the builder cannot recover contacts absent upstream. Codex headline connection counts are
not necessarily the local aggregated graph's edge count.
The Minecraft project's provenance reports ~25.9 million MaleCNS neuron-pair edges at
weight ≥1 and a bundled derivative of 6,287,749 edges at weight ≥5, representing
90,296,905 of 125,024,863 neuron-to-neuron synapses. It removes 40 autapses. Those are
**that project's reported query/build results**, not counts independently reproduced here.
Its five-contact cutoff retains roughly 24% of edges but 72% of synaptic weight. It is a
performance/modeling choice, not the definition of a complete CNS.
The official bulk weight table includes **segments**, not only curated neurons. Loading
all rows as if they were all validated neurons would be a different experiment. Likewise,
neuPrint ROI-level adjacency rows must not be summed together with their already-aggregated
totals. Specify one authoritative edge representation and count every exclusion.
### 3.3 Acquisition and reproducible construction
Prefer official versioned bulk tables for the repeatable build, with a neuPrint query tool
for inspection and cross-checking. The official download page lists:
- `body-annotations-male-cns-v1.0-minconf-0.5.feather` — curated annotations.
- `body-neurotransmitters-male-cns-v1.0.feather` — neuron-level transmitter information.
- `body-stats-male-cns-v1.0-minconf-0.5.feather` — segment statistics; large and broader than
the curated neuron list.
- `connectome-weights-male-cns-v1.0-minconf-0.5.feather` — full segment connection graph,
approximately 1.1 GB as listed by upstream.
The much larger synaptic-point and partner tables are unnecessary for an initial point-neuron
simulation. Download them only for a question requiring synapse-level geometry. Neither
research downloads nor anatomy conversion belong in service startup or ordinary unit tests.
Proposed build stages:
```text
release manifest + checksummed source cache
→ source-specific parser
→ normalized neuron/edge tables + exclusion report
→ selected anatomical graph
→ model-specific signed-weight transform
→ runtime CSR bundle + profile bindings + separate viewer bundle
```
Each stage records source release, database revision if queried, query/filter definitions,
source byte hashes, tool revision, and output hashes. Fetch time is provenance, not a random
input to the semantic graph hash. Sort body IDs numerically; encode original IDs as strings
in JSON so this common format also preserves FAFB IDs beyond JavaScript's safe integer range.
Preserve original annotations and cross-dataset type aliases separately from normalized roles.
The first import report must reconcile the Codex/neuPrint count difference, or explicitly
choose and document one inventory without claiming equivalence. It must also report missing
IDs, unannotated neurons, empty required populations, missing geometry, unknown sides,
transmitter confidence/fallback counts, duplicate edges, autapses, clipped weights, and
retained contacts by region and threshold.
Use deterministic serialization and gzip headers, following our existing reproducible
builder rather than copying the Minecraft artifact's timestamp-dependent container format.
Keep the source cache outside tracked artifacts; pin published runtime bundles by digest.
When adding actual data, update `NOTICE`, `LICENSES.md` and bundle-local attribution/license
files in the same change. Keep FAFB-derived assets under their existing terms; a separately
licensed MaleCNS bundle does not relicense mixed fixtures, old goldens or viewer assets.
### 3.4 Graph and sign policy
Keep raw positive contact counts and transmitter evidence in the normalized data. Sign is a
model transform, not a measured property that should overwrite the source evidence.
The legacy policy assigns GABA/GLUT negative and other/unknown transmitters positive;
conflicting per-edge transmitter rows become `MIXED`, then positive. Do not silently extend
that policy to histaminergic photoreceptor input. A MaleCNS policy must explicitly define
histamine, monoamines, mixed/unknown labels, confidence fallbacks, and whether transmitter
is chosen per neuron or per connection. None implies receptor-specific physiology.
Recommended first profiles:
- **`malecns-v1-lif-baseline`**: existing kernel, explicitly versioned sign policy and L1
input mapping, fixed readout, plasticity initially disabled for characterization.
- **`malecns-v1-lif-learning`**: same anatomy/input/readout plus audited KC→MBON selection
and the existing reward rule. Learning is an experimental condition, not an assumed gain.
- Later **sensorimotor research profiles**: photoreceptors, mechanosensation, VNC outputs,
and possibly another neural model, each with its own identity and validation.
Build weight≥1 and weight≥5 variants as **different graph identities** for the benchmark.
Do not select a production cutoff until activity, retained connectivity, memory and speed
have been measured. Preserve autapses by default in the new canonical graph; if an experiment
removes them, record the policy. The claim that point-neuron models have no use for autapses
is not a reason to discard observed connectivity silently.
Schema-1 `i16` weights may be sufficient, but measure overflow rather than assume it. For
the first existing-kernel comparison, emit a schema-1-compatible runtime view only if its
quantization/clipping is explicitly reported. If wider weights are needed, implement an
additive format/loader and matching oracle path; do not reinterpret old `weights.binz`.
Strengthen validation before constructing a network: CSR starts at zero, is monotone, ends
at edge count; targets/roles/visual indices are in range; required populations are present;
geometry is finite where marked valid; array lengths and index widths are representable;
declared hashes and artifact sizes match. Test the same invalid fixtures in both languages.
### 3.5 Population and sensory mapping
Existing role predicates cannot be copied unchanged. The current builder recognizes Codex
names such as `Kenyon_Cell`, `brain_motor_neuron`, `DAN` and `PAM*`; MaleCNS uses another
annotation vocabulary. Maintain a reviewed mapping table with source predicates, resulting
counts, hemisphere policy, aliases, and citations. Resolve each profile's required roles
at startup; an absent role must be an unsupported capability, not a silent empty population.
In particular:
- **Brain motor ≠ all motor.** Current macro pools combine 96 MBONs and 110 brain motor
neurons. Adding hundreds of VNC motor neurons under `motor` would silently change that
behavior. Use qualified roles such as `brain.motor`, `vnc.motor`, `brain.descending`,
`mb.kenyon`, `mb.output`, and `mb.pam` in new profiles, with legacy aliases only as needed.
- **Action groups are not anatomical facts.** `command_*` buckets use index modulo eight;
`macro_*` groups use a round-robin MBON/brain-motor pool. Move their construction into a
versioned task readout profile, outside the anatomy builder. Do not describe those groups
as natural “attack,” “jump,” or game-objective circuits.
- **Rate budgets are finite.** Both kernels currently cap tracked populations at 64. A
full CNS has many more interesting populations. Select a bounded control/telemetry set
initially; arbitrary bulk population analysis belongs in offline tooling. A larger mask
is a measured, oracle-tested change, not an unbounded string map in the tick loop.
- **L1 first, photoreceptors later.** Reuse the existing luminance projection only after
auditing MaleCNS L1 hex coordinates, both sides, and a declared hex→2D transform. Soma
coordinates are not visual-field coordinates. Missing columns remain explicitly missing;
do not synthesize them from array order or infer them from another specimen's neuron IDs.
- **Input profile and display view differ.** A game image may be resized/cropped for the
agent while the full frame is shown to viewers. Record crop, orientation, color transform,
and sampling geometry. Neither player may accidentally receive another player's private
view or adapter-only task observations.
The existing `set_visual_frame` and `stimulate` API cannot represent a general collection of
odor/touch/proprioceptive inputs. A later sensory-drive interface needs explicit units,
target populations, additive/overriding rules, tick ordering, and deterministic noise streams.
It must be specified in TypeScript before a matching Rust implementation. Keep the old
image/stimulation path available byte-for-byte through its compatibility facade.
### 3.6 What MaleCNS lets us investigate
| Experiment | New capability | What still needs engineering/measurement |
| --- | --- | --- |
| Same game, another connectome | Compare datasets under a matched task interface | Cell-type mapping, gain/activity calibration, readout comparability, multiple seeds |
| Brain↔VNC control | Read actual descending, ascending and motor populations | Body/control mapping; gamepad commands are not muscles |
| Embodied fly arena | World smell/taste/touch/vision mapped into annotated sensory populations | Sensor transduction, proprioception and body dynamics; identify every reflex shortcut |
| Mixed-dataset two-fly match | FAFB and MaleCNS agents share one environment | Balanced observations, controller mapping, compute budgets, intervention rules |
| Circuit perturbation | Compare full graph with VNC feedback or defined pathways ablated | Separate graph identities; activity and behavioral controls; no biological claims from gameplay alone |
The Minecraft project is a useful engineering comparison, not our validation oracle. It uses
a Shiu-style current-based LIF model with synaptic dynamics/delay, reports gain calibration,
and explicitly supplies odor-approach reflexes and higher-level looming drive where its
simulated pathways do not work. Its source graph, numerical model and embodiment differ
from ours simultaneously. Do not attribute its behavior solely to MaleCNS.
### 3.7 Identity, restore, and performance
Name the components independently:
```text
anatomyId = source release + included inventory + graph/filter digest
modelId = numerical semantics + effective numeric configuration
sensorId = input encoding + anatomical binding digest
readoutId = population partition + decoder + action mapping digest
learningId = rule + selected-edge topology + reward-catalog identity
viewerId = positions/geometry + index mapping digest (presentation only)
```
The composite behavioral identity covers all behavior-affecting components; original source
IDs and display geometry cannot replace it. Same neuron count does not establish compatibility.
No FAFB neural checkpoint is restored into MaleCNS. A deliberately fresh MaleCNS brain may
start from a compatible game-only save, with a new run identity, clean calibration and reward
baselining; that is a new experiment, not continuation of the old fly. Gains are not mapped
between specimens by cell-type name.
For rough capacity planning, the current CSR is `4(N+1) + 6E` bytes, excluding roles,
geometry, derived propagation structures and mutable state. At 176,422 neurons that is
about 38.4 MB for 6.29 M edges, or 156.1 MB for 25.9 M edges (decimal MB). This is roughly
2.3× or 9.6× our edge count, not a prediction of the same slowdown. Spike activity, fan-out,
plasticity selection, memory bandwidth and sharding determine runtime cost. Checkpoint
copies and renderer assets also need separate memory budgets.
`LifNetwork` accepts shared `Arc<BrainDataset>` already. Reuse immutable anatomy across
same-profile agents; keep membrane, refractory state, RNG, rates, decoder holds, stimulation,
eligibility and learned gains private. Audit constructor-derived caches before moving them
into a shared topology object. Never share mutable gains merely because graphs match.
Measure headless one-, two-, and four-agent runs with plasticity on/off, fixed inputs and
representative activity. Record resident/peak memory, initialization, state capture cost,
per-phase p50/p95/p99, spike distribution, and real-time factor. Existing CUDA code is an
optional backend requiring its own new-dataset equivalence/capacity gate, not assumed capacity.
## 4. Reusable architecture
### 4.1 Define the nouns first
- **Dataset bundle:** immutable anatomical graph and source annotations.
- **Brain profile:** dataset plus numerical model, sensory/readout bindings and learning rule.
- **Agent:** one independently stateful brain, encoder, decoder and action executor.
- **Environment:** the world being advanced: one emulator instance, a linked-emulator group,
or an embodied simulator. Owns controller ports, world state, media, and native clock.
- **Task:** interpretation of environment state: rewards, progress, episode endings, allowed
macro actions and recovery policy. Pokémon is a task, not an environment API.
- **Session:** one environment plus agents, port assignments, scheduler, task state and clocks.
- **Application:** composes sessions/components and their presentation; owns supervision,
persistent identities/history, run/intervention rules and application-specific schemas.
- **Bus:** generic RPC/pub-sub routing and artifact ownership, with no game/simulation semantics.
- **Broadcast:** presentation of one or several sessions, plus chat and audience interactions.
An agent ID is not a Twitch username, controller port, array position or dataset ID. Session,
episode, agent, port, view, and event identities must be explicit and stable across restore.
### 4.2 Dependency direction
```text
source importers → dataset bundles
↓
neural core (TS oracle / Rust runtime)
↓
agent composition: sensors + readout + executor
↓
environment backend + task plugin → session runtime → observations/checkpoints
↑ ↓
command admission protocol adapters
↑ ↓
Twitch bridge stage / recorder
```
The neural core knows no emulator, task, network socket, chat, or UI. The environment knows
no neural populations or Twitch. The task may inspect backend-specific state through a
typed inspector, but neither inspection nor public presentation gives clients a memory-write
or controller-write API. Only the session commits agent-produced controls.
### 4.3 Proposed modules and staged layout
These are target responsibilities, not instructions to create every package immediately.
Start as modules; extract crates/packages once a second consumer demonstrates the boundary.
Keep the Rust workspace under `services/flysim` during semantic extraction so paths and
behavior do not change together. A later mechanical move can place reusable crates at the
root, updating CI/build/golden paths in one dedicated change.
| Module / eventual location | Owns | Extraction source |
| --- | --- | --- |
| `packages/brain` | Reference numerical behavior and legacy public facade | Existing package; keep imports compatible |
| `crates/flybrain-core` | Rust numerical kernel, plasticity, generic population decoder | Existing `core/`; leave compatibility re-exports for presets |
| `crates/fly-dataset` | Manifest validation, artifact loading, source-ID/index mapping | `core/src/dataset.rs`; retain legacy fingerprint implementation |
| `crates/flybus` | One Rust RPC/pub-sub client/router, immutable artifact store, delivery guards and GC | New generic library; embedded router or small executable, no separate worker transport |
| `tools/datasets/{fafb,malecns}` | Source-specific conversion to common bundles | Existing Python builder plus new importer; existing CLI wrapper remains |
| `crates/fly-session` | Agent ownership, clock coordination, action commit, event/reward routing | Orchestration extracted from `sim/src/simloop.rs` |
| `crates/fly-environment` | Backend capabilities, ports, observations, media, save-state interfaces | New small contract proven with binjgb and synthetic arena |
| `crates/fly-env-gb` | binjgb FFI, memory inspector and native save-state identity | `gb/src/{emulator,ffi}.rs`, build glue and vendor boundary |
| `crates/fly-task-pokemon`, `fly-task-platformer` | Audited reward rules, semantic state, macros, progress/recovery | `gb/src/{pokemon_red,platformer}/`; do not generalize tile routing into the core |
| `crates/fly-checkpoint` | Atomic storage and session envelope; legacy payload adapter | `sim/src/store.rs` plus core envelope helpers |
| `crates/fly-protocol` / `packages/feed` | Versioned wire schemas/codecs, legacy adapters, synthetic fixtures | `sim/src/snapshot.rs`, existing feed package; canonical schema with cross-language tests |
| `services/flysim` | Composition/config, HTTP/WS, process lifecycle, metrics | Thin host over reusable session library |
| `packages/stage-runtime` | Feed ingestion, per-session stores, paint loop, audio, fixture clock | Extract from `apps/stage/src/{feed,paint,audio,motion}` after multi-view prototype |
| `apps/stage` + presentation plugins | Layout, branding, task panels, audience-facing explanations | Existing page with legacy layout preserved |
| `services/bridge` + audience client module | Twitch transport/auth/redemptions; session-targeted interaction client | Existing bridge; extract provider-independent logic only when reused |
| `infra/` | Release composition, process supervision, capture, recordings | Existing tooling parameterized by session/broadcast manifest |
Use static Rust composition or a small closed registry initially, with trait boundaries at
backend/task seams. Do not require stable native dynamic-plugin ABI. An out-of-process
emulator helper implements the backend through Flybus RPC, using the same bus as application
events and publication. Native emulator protocols stay inside its adapter. This is not a
new public action API or a reason to maintain a second framework transport.
Keep backend-specific memory access private to its task implementation instead of widening
`read8(u16)` into a supposedly universal game-state abstraction.
### 4.4 Environment and agent contracts
Illustrative interfaces; concrete types must be written with tests during contract work:
```rust
trait Environment {
fn descriptor(&self) -> &EnvironmentDescriptor;
fn observe(&mut self) -> Result<WorldObservation>;
fn advance(&mut self, actions: &ActionBatch) -> Result<WorldStep>;
fn capture(&mut self) -> Result<EnvironmentCheckpoint>;
fn restore(&mut self, state: &EnvironmentCheckpoint) -> Result<()>;
}
// One decision boundary, one action per configured controller port.
struct ActionBatch {
session_tick: u64,
ports: Vec<PortAction>,
}
```
`descriptor` declares rational step duration, controller schemas, views, audio streams,
save/restore availability, task inspection capabilities and determinism level. Capture and
restore return an explicit unsupported error when unavailable; configuration validates that
the chosen recovery policy can work. `WorldObservation` is a frame-boundary snapshot or
immutable handle, not an object allowing agents to advance the backend.
Control schemas support digital buttons and bounded analog axes/triggers, with neutral
values, axis ranges, dead zones, and mutually exclusive directions where appropriate.
Preserve the Game Boy mask as one concrete codec. Analog controls need a fixed, versioned
decoder mapping; an 800-ms direction hold is not a sensible default for every fighting game.
Separate three observation surfaces:
1. **Agent sensory view:** pixels or declared synthetic senses the profile may consume.
2. **Task inspector:** audited state for reward/macro/episode logic. Access is part of the
disclosed scaffold, not implicitly available to the neural encoder.
3. **Broadcast view:** media and summaries for viewers, potentially richer than either player's
allowed sensory input.
Readout produces semantic channel activations or continuous signals. A task-local action
executor translates these into port actions, optionally running a selected macro. It is
explicitly resettable/checkpointable and reports selected action versus actual controller
output. Pokémon pathfinding, dialog logic, and objective catalogs stay in its task plugin.
Task outputs become scoped `RewardEvent`, `Progress`, `EpisodeEvent` and `ActionAvailability`.
Progress is a tagged value (`ladder`, `score`, `match`, `exploration`, or task extension), not
always a scalar rank. Rewards carry recipient agent/team, rule ID, event ID and observation
tick. A task cannot mutate neural state directly; the session routes accepted rewards once.
An agent-facing API similarly separates `advance_brain(interval)`, `decide(observation,
availability)`, `encode_next(sensory_view)`, and `apply_outcome(rewards, stimulation)`.
The session owns their order. Agents cannot call `Environment::advance`, select another
port, or inspect another agent's mutable state. Start with a concrete LIF agent composition;
introduce a controller trait when synthetic controllers or a second neural model require it.
Test controllers implement the same decision surface but are identified as non-neural agents
in descriptors and experiment records.
### 4.5 Example composition
Illustrative configuration, not syntax supported by today's `flysim.toml`:
```toml
[session]
id = "arena-demo"
environment = "synthetic-arena-v1"
task = "two-player-rounds-v1"
scheduler = "lockstep-v1"
master_seed = 1234
recovery = "round-reset-keep-gains-v1"
[[agents]]
id = "fly-a"
port = "player-1"
profile = "fafb-arena-baseline-v1"
sensory_view = "shared-camera"
[[agents]]
id = "fly-b"
port = "player-2"
profile = "malecns-arena-baseline-v1"
sensory_view = "shared-camera"
[broadcast]
layout = "shared-match-two-agents"
audio = "world"
audience_stimulation = false
```
Resolve profile IDs through a local, digest-pinned registry. Both agents may instead select
the same profile and share immutable topology while retaining independent state. The host
validates unique agent IDs and exclusive port ownership, view accessibility, profile/controller
compatibility, recovery capability and resource budget before starting. An independent-games
broadcast composes two such sessions; it does not misrepresent them as ports in one world.
## 5. Multiple flies: concurrency is not multiplayer
### 5.1 Three supported arrangements
| Arrangement | Ownership / synchronization | First use |
| --- | --- | --- |
| Independent flies in independent games | One session/process each; optional broadcast composition | Parallel streams and experiments; existing deployment pattern generalizes easily |
| Several flies in one game | One environment, several ports, one session barrier | Local fighting/multiplayer games |
| Linked emulator instances | One composite environment owns all instances and link state | A later link-cable experiment; requires cycle-accurate link support, not two independent frame loops |
For a shared arena there must not be one `Sim` loop per player, each calling `run_frame`.
That advances the world multiple times and gives an ordering advantage to one agent.
### 5.2 Shared-world step semantics
For decision boundary `t`, freeze observation `O[t]`, then:
1. Admit queued audience/operator commands against stable session/agent identities; log their
effective tick. Chat remains presentation state.
2. Each agent advances its brain for the same environment interval, using its previously
encoded sensory input. Its clock remainder and RNG are private.
3. Each agent decodes and advances its action executor using the same `O[t]` task boundary.
4. Barrier: collect all port actions, validate ownership/ranges, and commit one complete batch.
5. Advance the environment **once** to obtain `O[t+1]` and timestamped media.
6. Encode the next sensory inputs, evaluate task events from the completed transition, apply
explicitly routed stimulation and rewards, and compute next action availability.
7. Apply any whole-session episode/recovery transition; capture coherent state and publish.
Preserve the detailed legacy ordering inside the legacy single-agent composition. New
profiles identify their scheduling semantics explicitly rather than silently adopting a
different reward phase. Use rational environment time and integer substep accumulation for
new sessions; keep the legacy floating remainder arithmetic for old trajectories. The neural
clock may lead environment time by warm-up; persist that offset instead of pretending all
clocks start at zero. Rendering, physics and decision cadence may differ, but the backend
must define their relationship.
Start with sequential agent evaluation for reproducibility. Then compare parallel agent
evaluation against the same action trace. Cap total worker budget: `agents × brain_threads`
can otherwise oversubscribe the machine. Use private pools for concurrent agents or serialize
dispatch into a pool; the current `WorkerPool` must not be concurrently reused through a
cloned `SweepPlan`. Sharing immutable graph buffers is independent of scheduling workers.
If one agent is late, the default is to slow the **whole session** and report lag. A crashed
agent pauses/fails the match rather than silently becoming a neutral or scripted opponent.
A realtime external world that cannot pause requires a separate declared deadline/hold-last
policy, dropped-action telemetry and a different determinism claim. Do not hide that policy
inside the environment adapter or use wall-clock completion order as an action tie-breaker.
### 5.3 Match state and learning
Every agent has its own seed, calibration, gain vector, eligibility, reward totals and
stimulation cooldown. Derive seeds deterministically from a stored master seed and stable
agent ID; do not use thread scheduling or the default identical seed for every fly.
For an initial fighting-game task:
- Both agents receive the same shared camera unless the game has genuine private views.
- Controller-port swaps and seed repeats are part of evaluation; wins alone are confounded
by character, spawn, arena, action interface and side advantage.
- Award positive, explicitly attributed events such as a scored hit/round win; define
damage/self-damage/team attribution and duplicate detection before turning learning on.
A loss need not produce a negative reward; changing reward doctrine is a separate decision.
- Episode transitions may retain learned gains while clearing transient traces, or create
fresh brains for controlled trials. Record which policy was selected.
- Do not select a “best checkpoint” separately for each player in a shared world. There is
one world state. Tournament scores and historical results should not rewind with a match.
- Disable sugar for balanced evaluation. If enabled for a show, target a named agent under
a documented rule and log the intervention; do not call that an uncontrolled fair benchmark.
### 5.4 Checkpoint and recovery semantics
Distinguish three operations:
| Operation | Restored/reset state |
| --- | --- |
| Crash resume | Coherent environment, every agent, scheduler remainders, task ledgers, pending actions/commands and executor state at one boundary |
| Task recovery | Policy-defined world rewind/reset and per-agent transient clearing; continued gains/brain clocks only when declared, as in the legacy ratchet |
| New episode | Task initial world state, explicit retained/fresh agent policy, new episode identity; session event sequence remains monotonic |
Use a new session envelope version with a manifest mapping stable agent IDs to chunks and
listing graph/model/profile/backend/content/task/state-format identities. The current
envelope restricts chunk names to letters; do not simply append `agent/1/membrane` to it.
Specify a new container format or an explicit manifest-to-valid-chunk-name indirection.
Validate every participant into staged state before mutating any live participant. A failed
environment import must not leave half the brains restored. For an external emulator,
restore a replacement stopped process when transactional in-place validation is impossible.
Capture at the action barrier with no backend step in flight. Reuse atomic payload write,
manifest commit, hot/durable tiers and off-thread serialization; bound outstanding snapshot
jobs so repeated copies cannot exhaust memory under slow storage.
Exact replay requires action-executor and admission state, not just the neural envelope.
Legacy macros intentionally discard transient execution on restart; preserve that behavior
for v1 and label it as legacy continuation semantics, not exact session replay. New sessions
persist all behavior-affecting state or explicitly restart an episode under a documented rule.
Keep `FLYSIM01` readable through a legacy adapter. Never silently rewrite a checkpoint on
read. Conversion is an explicit offline operation writing a new directory and run record.
Environment identity includes title/content digest, backend build/configuration, relevant
platform/native state format, and patch/symbol provenance; state size alone is not sufficient.
## 6. Feed, stage and audience modules
### 6.1 Feed v2 is required
V1 is not a generic multi-agent format. Its attachment map forbids duplicate kinds, so two
`spikes` arrays cannot coexist; its button mask and 160×144 image are Game Boy-specific.
Platformer counters are currently folded onto names such as `pokedex` and `wildwin`.
Extending those conventions to fighting games would preserve syntax while losing meaning.
Keep v1 stable for the legacy session. Introduce a negotiated v2 or a separate `/v2/feed`
endpoint, with a descriptor delivered before dependent snapshots and available on reconnect.
This is the application's browser/presentation gateway over Flybus, not another internal
bus. Native participants use the same RPC/pub-sub protocol for all framework communication.
The contract change must update Rust, TypeScript, schema, fake service, fixture player,
stage and bridge together. Proposed shape:
```text
SessionDescriptor
sessionId, protocol, descriptorRevision, environment/task identities, clock definition
agents[{agentId, profileId, datasetId, neuronCount, roles, controlPort}]
ports[{portId, controllerSchema}]
views[{viewId, dimensions, pixelFormat, sensory/broadcast use}]
audioStreams[{streamId, sampleRate, channels}]
assets[{id, contentHash, datasetIndexHash, localUrl, license/credit}]
SessionSnapshot
descriptorRevision, seq, sessionTick, episodeId, environmentTime, wallTime, status
agents[{agentId, brainTime, rates, learning, selectedAction, actualControls, stimulation}]
progress: tagged task payload; events: scoped and sequenced
attachments[{id, kind, ownerId, byteLength, format, mediaTimestamp}]
```
Use bounded, schema-validated tagged payloads/namespaced task extensions, not arbitrary
unlimited JSON or executable server-supplied UI. Dataset identity and neuron index mapping
must accompany spike geometry: matching bitset length alone cannot establish alignment.
Include descriptor revision in every snapshot and reject stale/mismatched buffers. Large
monotonic IDs/times use a specified safe-integer bound or decimal strings across languages.
Publish one shared camera/audio stream once, not once per agent. Independent sessions may
have independent streams. Timestamp audio/video to the session clock; define discontinuities
on reset, reconnect and lag. Latest-value video/telemetry may drop, while audio needs a
bounded timestamped buffer and explicit gap handling. Durable event IDs permit recovering
missed events; a drop-oldest snapshot feed is not an exactly-once event log.
At 640×480 RGBA, native frame production is 36.864 MB/s at 30 Hz or 73.728 MB/s at 60 Hz.
Publish one immutable artifact and pass owned references through Flybus to all consumers.
Bytes stay outside router messages; producer/read/copy costs still need measurement. A
renderer retains its handle past message drop; cached RPC results and latest retention also
own data until release. Last-owner GC replaces coordinator-managed slots/reader acknowledgments.
Resizing, overlays, codecs and streaming are presentation-layer choices, not bus requirements.
### 6.2 Stage composition
Extract `createSessionStore()` rather than adding `agent2` fields to the singleton `hot`.
Own rate scalers, button afterglow, ticker state, fixture clock and audio queues per session/
agent. Keep one page paint scheduler; register surfaces against explicit view/agent IDs.
Geometry is loaded by descriptor/hash through an asset manifest, replacing the hardcoded
FAFB route in both `App.tsx` and `vite.config.ts`.
Retain the existing Game Boy layout as a presentation plugin. Add composition primitives for
a shared match view with two agent summaries, or independent session tiles with one focused
audio source. A generic fallback shows status, media, controls and task labels without
inventing Pokémon counters. Unknown optional extensions can be omitted; unknown required
capabilities or mismatched descriptors must be visible rather than silently showing Pokémon.
Combine common framework measurements with application-owned state/events. Describe values
by owner, type, units, range and timestamp; distinguish zero, unknown, unsupported and stale.
Game-specific progress/collections remain validated schema extensions. The application defines
its supervisory/story behavior alongside its UI, not inside a mandatory generic Director service.
Keep UI runtime dependencies out of the numerical package. The current `@flybrain/brain`
view exports and optional Three.js peer can remain compatibility re-exports when viewer
geometry/helpers move to a dedicated view module. Controller labels belong in controller
schemas, not a UI import of the neural package's Game Boy preset.
Review actual layout proposals as PNGs under `apps/stage/mockups/`, with existing legibility,
phone-scale and browser gates. This document proposes data/layout boundaries, not screen
copy or a replacement for visual sign-off.
### 6.3 Audience interaction
Keep Twitch authentication/EventSub and template-only replies in the application bridge. Extract a
session-targeted interaction client with explicit `sessionId`, optional `agentId`, interaction
kind and idempotency key. A multi-agent request with no target is rejected unless a fixed,
declared target policy exists; presentation focus must never choose the recipient.
Service admission owns per-session, per-agent and global limits. A profile advertises its
supported stimulation capability; an agent without PAM support returns unsupported rather
than pretending to accept “sugar.” `!stuck` becomes task-aware (ladder time versus round
time), while chat remains broadcast-scoped and independent of neural state. Future boons
use task/backend-declared capabilities with explicit target, timing and outcome; gifts do
not automatically become earned learning rewards. These need separate capability/admission
contracts before enabling them, carried over the same bus rather than an arbitrary write API.
For redemptions, persist the resolved target and request identity before retrying. Distinguish
an RPC timeout (HTTP on legacy v1) from a definite refusal: it may occur after the sim accepted the
effect. V2 needs a durable or explicitly recoverable deduplication/status contract so bridge
restart cannot apply the same pulse twice or retarget a redemption to a new match. Do not
promise exactly-once behavior from the bridge intent log alone. Existing v1 remains as-is.
### 6.4 Deployment and observability
A deployable composition selects runtime binary, backend/task, agent profiles, dataset
bundles, view assets, session state namespace and broadcast layout. Generate service config
from that manifest plus the operator's external environment. Keep tokens and network-specific
values outside this repository. Package viewer artifacts separately from the full simulation
graph so the browser does not need every edge.
Preserve existing process isolation: sim, bridge, browser, capture, local relay and Twitch
push can restart independently. One coordinator/session with separate worker processes is
the default composition; a shared match remains one logical failure/recovery group across
those workers. A worker restart cannot silently rejoin. Router failure also invalidates its
ephemeral handles/routes. Multi-session placement and presentation composition are application
deployment concerns; router scope sets an explicit shared-failure boundary.
Metrics distinguish session lag, environment step cost, per-agent step cost, barrier wait,
snapshot drops, audio discontinuities, checkpoint queue age and resource budget. Bound agent
labels to configured IDs; never label metrics by viewer name or arbitrary event text.
Health must distinguish paused, slow, disconnected backend and dead agent. A generic
watchdog cannot treat “no new Pokémon tiles” as a stall detector for all tasks.
Deployment preflight checks every referenced profile/bundle and complete checkpoint identity
before selecting a release. Switching the binary back does not convert newer state: retain
the previous release's state namespace for rollback. Twitch publishing still requires the
operator's explicit approval for that run; implementation benchmarks use local sinks.
## 7. Cleanup strategy: extract, then reorganize
Prioritize coupling that prevents a second application, rather than renaming everything.
1. **Characterize behavior and identify state owners.** Capture deterministic synthetic
observation→action→reward traces and legacy restore outcomes before moving code.
2. **Remove task construction from anatomy.** Introduce separately hashed readout bindings;
leave the current artifact and macro-role merge path frozen for legacy compatibility.
3. **Extract orchestration from transport.** A session step returns observations/events and
capture requests; it does not serialize HTTP/feed headers. `flysim` owns listener setup,
wall-clock publication and systemd integration.
4. **Split emulator from task.** Move binjgb behind an environment implementation, preserving
FFI/cache behavior. Move map/exit/objective queries into Pokémon task interfaces rather
than forcing every future game to implement them.
5. **Split task recovery from storage.** The ratchet decides a task transition; storage
commits an opaque coherent capture. Matches use round resets, not milestone archives.
6. **Version observation/control at the boundary.** Internal typed session snapshots become
v1 or v2 through adapters; core modules do not depend on wire enums.
7. **Make stage state instantiable and assets descriptor-driven.** Preserve hot/cold cadence
and fixture determinism while removing singletons and hardcoded dataset selection.
8. **Only then move directories/extract packages.** Keep re-exports/CLI wrappers during the
move, fix build/CI/vendor paths, and compile tiny consumers proving Rust/TS libraries can
be used without starting Twitch, a browser, or an emulator.
9. **Reconcile documentation.** Separate current contracts/reference from dated deployment
history. Audit contradictory “raw buttons only,” weighted-macro, throughput, token/setup,
and training-improvement claims against code. Update templates and scientific limitations
with actual profile capabilities, not a new generic claim of biological fidelity.
Do not introduce a generic reward engine, universal memory address model, central plugin
marketplace, or per-neuron network transport. Those abstractions have no demonstrated second
consumer and would obscure the useful, small seams already present.
## 8. Implementation plan and acceptance gates
Each row is a reviewable change or small workstream, not one large feature branch. Follow
the repository's branch/worktree build-and-review workflow. Contract changes precede their
consumers. No estimate here assumes an emulator backend or biological mapping already works.
| Phase | Deliverable and principal files | Dependencies | Acceptance / stop condition |
| --- | --- | --- | --- |
| P0 — baseline | Trace fixtures and boundary tests around `Sim::step_frame`, restore, dataset loader, stage singleton behavior; current-state documentation inventory | None; reconcile open branches first | Current TS/Rust goldens and legacy API/feed fixtures pinned; behavior ledger distinguishes intentional transient reset from exact replay |
| P1 — identities | Dataset/profile manifest, role resolver, behavior hash, synthetic fixtures; new strict validators in both languages | P0 | Missing/ambiguous roles fail; wrong profile refuses restore; FAFB legacy fingerprints/version strings unchanged |
| B1 — Flybus | Small Rust router/client with RPC, pub/sub, owned artifacts and GC; see BUS-01..03 in the contract implementation guide | Can start alongside P0/P1; no dataset/emulator dependency | Same semantics in-memory/Unix sockets; cache/retention holds safe; 480p three-reader test; no raw binary in message envelopes |
| M1 — MaleCNS import | `tools/datasets/malecns`, source lock, inventory/exclusion report, schema-compatible baseline bundle, attribution | P1 | Counts reconcile to selected inventory; reproducible output hashes; graph invariants and both loaders agree; no runtime downloads |
| M2 — headless characterization | Profile-specific L1/sensor binding, role/readout audit, learning-off/on benches and new goldens | M1 | Stable/finite activity measured over repeated seeds; no silent empty populations; exact TS/Rust state agreement; unsupported inputs remain marked unsupported |
| M3 — task integration | Explicit MaleCNS profile selected by service config and matching stage assets, fresh state namespace | M2 and minimal descriptor/asset support from P5 | Fresh brain on audited game state; sugar capability verified; stage index/geometry identity correct; one-hour local soak plus restore drill; no automatic promotion over FAFB |
| P2 — environment/task split | Environment contract, binjgb wrapper, task-specific inspector; facade for existing `flybrain-gb` imports | P1 | Legacy action/reward traces identical; ROM-free tests pass; optional ROM-backed sample confirms stepping/audio/state behavior |
| P3 — session library | Single-agent session owns clocks, agent/task/executor state; service composes bus-connected workers | P2, B1 | Same legacy order and compatibility; renderer/bridge restart leaves run intact; fake backend runs without binjgb/ROM/Twitch |
| P4 — multi-agent + persistence | Two-agent synthetic arena, action barrier, isolated state, session envelope/recovery and failure behavior | P3 | One world step per batch; no port/order advantage; resume/parallel-vs-sequential equivalence; failure cannot partly commit a match |
| P5 — feed/control v2 | Descriptor, scoped snapshots/events/media, targeted stimulation and idempotency, TS/Rust schema fixtures/fake server | P1, P3; validate with P4 fixture | V1 still works for legacy; multi-agent attachments cannot collide; unknown target/profile rejected; retry/reconnect tests pass |
| P6 — modular stage/bridge | Instantiable stores, dataset assets, shared/independent views, target-aware commands/redemptions | P4–P5 | PNG review for two-agent layout; browser legibility/fixture tests; correct audio ownership and no agent cross-talk |
| E1 — new emulator spike | Select title/backend; bus-connected helper or native wrapper; record native frame, ports, inspection and restore capabilities | P2, can run beside P4–P6; framework integration uses B1 | Reliable bounded step + simultaneous controls, pinned content/backend identity, reproducible state round trip; stop before task implementation if unavailable |
| E2 — fighting-game vertical slice | Two flies, chosen game task, analog/digital readout, episode logic, attributed rewards, match view | E1, P4–P6 | Repeated local matches and side swaps; measured compute headroom; documented scaffold and interventions; win-rate claims require controls |
| P7 — packaging/reorg | Optional root Rust workspace move, library consumers, profile-based release/asset manifests, updated infra and docs | Useful second backend + P6 | Four merge suites and affected browser gates pass; old composition deploys locally; preflight rejects incompatible multi-agent state |
M3 may use a small v1-compatible **additive descriptor extension**, if contracts and both
consumers are updated and the existing frame semantics stay unchanged. It must not publish
MaleCNS spikes under implicit FAFB geometry. Full multi-agent publishing still requires v2.
### 8.1 A concrete first alternative-emulator spike
Before committing to Melee/Dolphin or an N64 backend, establish:
- An exact title/version and backend revision, available to the operator externally.
- A supported pause/advance boundary with all configured controller ports applied together.
- Whether rendering is required for stepping and whether frame capture is synchronous.
- Sample rate/channel metadata, audio latency and timestamps.
- Analog sticks/triggers and button semantics; neutral state on disconnect.
- Save-state completeness, version/platform constraints and reproducibility after restore.
- Supported task-state inspection (match/round/port state) without guessing memory offsets.
- Headless/runtime packaging, process lifecycle, resource cost and failure behavior.
A library used for competitive tooling may expose controller and game-state APIs without
supporting arbitrary frame stepping or faithful visual capture. Verify capabilities instead
of assuming its name solves integration. Prefer a pinned private backend process if native
embedding would force emulator internals into our Rust runtime. Desktop keyboard automation
is unsuitable for simultaneous deterministic multi-port input.
Start with synthetic constant/alternating controller traces and a test opponent before neural
control. Such traces are backend tests, not public “fly playing” footage. Then validate one
agent, two agents, episode boundaries, crashes and resume in that order. No copyrighted game
content is added to source control or test fixtures.
### 8.2 Validation matrix
**Numerical/format:** existing `core/tests/golden_{toy,real,agent,restore,platformer,versions}.rs`
remain gates. Add pinned MaleCNS subgraph fixtures and a full-artifact optional golden run,
with generated TS goldens and exact Rust comparisons. Include malformed CSR, missing roles,
wide IDs, hemisphere/coordinate errors, overflowing weights, and graph/profile mismatch.
The subgraph tests prove arithmetic/loader agreement, not full-CNS dynamics.
**Scheduler:** synthetic backend asserts one advance per complete batch; swap agent evaluation
order, vary worker count, and inject a late/failing participant. Check equal observation
boundaries, deterministic seeds, no cross-agent gains/holds/stimulation, correct tick remainder,
and no reward twice at an episode boundary. Mixed profiles are allowed only if their clocks
and capabilities satisfy the same session contract.
**Persistence:** kill/fault injection around capture/write/manifest commit; corrupt one agent
chunk, backend state or profile hash; verify all-or-none restore and fallback reporting.
Compare uninterrupted and resumed new-session action/state traces. For legacy runs compare
against the documented transient-reset behavior instead of demanding a newly invented one.
**Protocol/UI/bridge:** cross-language v1/v2 fixtures; two agents with different neuron counts;
shared and private views; out-of-order descriptor/media, missing optional attachments, stale
snapshots and reconnect; replay seeking; duplicate redemption, lost HTTP response and bridge
restart; explicitly unsupported stimulation. Visual changes require PNG review and browser
tests, not prose approval of a hypothetical layout.
**Performance/science:** benchmark full graph and thresholded graph with learning disabled
and enabled, fixed sensory traces, multiple seeds and side swaps. For game-performance
claims compare against random/readout baselines and learning-off, reporting scaffold,
recovery, intervention and episode policies. Measure sustained real-time factor and tail
latency under two/four flies plus actual browser/capture load; do not extrapolate a single
kernel throughput figure to an entire show. Initial target is ≥1.0× sustained at the declared
agent count with p99 step time within its cadence budget and no growing queues; select a
resource/headroom margin from the measured backend before release.
Before each merge run the repository-required `npm test`, `npm run typecheck`,
`cargo test --workspace` from the Rust workspace, and `infra/tests/lint.sh`. Run affected
Playwright/PNG gates for stage changes. The existing committed FAFB real-data goldens remain
mandatory. New full-MaleCNS integration jobs and ROM-backed tests are explicit optional jobs
with recorded skips, never a hidden network/ROM dependency of normal CI.
## 9. Decisions and open questions
**Recommended decisions now:** preserve FAFB as the baseline; use official MaleCNS provenance;
freeze legacy arithmetic/identities; version profile behavior independently; make environment
and agent separate objects; use one session barrier for a shared match; retain static plugins
and process/session isolation; introduce v2 instead of stretching Game Boy fields indefinitely.
**Questions answered by spikes rather than assumptions:**
1. Which MaleCNS neuron inventory and confidence/threshold policy will be the published bundle?
Can we explain the differing inventories and quantify left/right sensory coverage?
2. Do existing LIF parameters give useful, stable activity on MaleCNS? If not, which explicit
profile calibration is justified, and does a different neural model warrant separate work?
3. Are verified L1 mappings adequate, or is the intended project really an embodied sensory
simulation requiring new encoders and VNC feedback?
4. Which Smash title/backend can satisfy deterministic stepping, simultaneous ports, media
capture and restoration at acceptable cost?
5. How many simultaneous brains fit the actual budget, with which mix of within-brain versus
between-brain workers? Is a compressed media path required?
6. Which match reset/learning/intervention policy defines the show, and which defines a
controlled comparison? They should be separate run configurations.
Success is not just “another connectome loads” or “a second pad moves.” It is a new session
assembled from modules whose anatomy, numerical model, controller, task, recovery and
presentation assumptions are explicit, testable and reusable without changing the old fly.
## 10. Sources and scope of the analysis
Repository evidence is enumerated in section 2 and tied to the baseline commit above. Existing
reference documents: [dataset format](../dataset-format.md), [model](../model.md),
[plasticity](../plasticity.md), [readout](../readout.md), [limitations](../limitations.md),
[macros](macros.md), [architecture tour](../architecture-tour.md), and
[contribution/compatibility rules](../../CONTRIBUTING.md). Historical status notes are not
evidence that an unmeasured experiment succeeded.
External sources inspected 2026-09-18:
- [Official MaleCNS overview](https://www.janelia.org/project-team/flyem/male-cns-connectome):
anatomical coverage, collaboration, release dates and licensing statement.
- [Official MaleCNS downloads](https://male-cns.janelia.org/download): versioned bulk tables,
confidence cutoffs, segment versus neuron distinction, coordinate units and API guidance.
- [FlyWire overview](https://flywire.ai/): FAFB reconstruction provenance and brain coverage.
- [Codex dataset listing](https://codex.flywire.ai/): portal inventory counts, which are not
assumed to be identical to neuPrint query inventories or local runtime graphs.
- [Minecraft fly README](https://github.com/blendi-remade/fly-brain-minecraft/blob/main/README.md)
and [provenance](https://github.com/blendi-remade/fly-brain-minecraft/blob/main/PROVENANCE.md):
a separately authored MaleCNS derivative, thresholding and mapping decisions, and disclosed
sensory/motor limitations. These moving links are comparison material, not a locked data
dependency; M1 must acquire its own official source lock.
This analysis reads the current implementation and upstream documentation. It does not
download/build the full MaleCNS dataset, independently validate the Minecraft benchmarks,
run a new emulator, or establish a performance/behavioral improvement. Those are explicit
deliverables with gates above.

766
docs/design/melee-framework-audit.md Normal file
View file

@ -0,0 +1,766 @@
# Melee: multi-fly runtime, emulator and broadcast audit
Status: **research and proposed implementation plan**. Written 2026-09-18 against local
`f7bc13a`. Extends the [modular-session design](malecns-modular-sessions.md) and
[implementation backlog](malecns-modular-implementation.md) with a concrete second game.
Existing [feed](../feed-protocol.md) and [control](../control-api.md) contracts still win.
No emulator, game image, deployment host or live broadcast was run for this audit.
Follow-on [session framework contracts](session-framework/README.md) define the concrete
multi-process synchronization and worker interfaces this backend will implement.
The subsequent [Flybus decision](session-framework/bus-v1.md) selects one small Rust router
for RPC and pub/sub, with immutable artifacts outside messages and delivery-scoped GC.
The application owns supervisory/presentation behavior; the bus owns no game or stream logic.
## 1. Executive decision
**Use Dolphin, initially a pinned mainline-based Slippi Dolphin build, as a separate backend
process. Use the maintained libmelee fork to accelerate the integration spike. Keep the
brains and session coordinator in Rust.** Prove its synchronization, rendered sensory input,
and recovery behavior before selecting a production build. Keep stock Dolphin plus a narrow
backend hook as the fallback if the Slippi path cannot satisfy those requirements cleanly.
Do not port Melee to native code, embed Dolphin into the neural crate, run one emulator per
fighter, or assume “libmelee has `step()`” supplies the complete environment contract.
Recommended first target:
- Melee US 1.02, local two-player versus, one emulator and two independently stateful flies.
- Existing FAFB brains first. MaleCNS is an independent axis of experimentation, not a
prerequisite for solving the emulator and multiplayer problems.
- Fixed declared characters, stage, stock/time rules and input profiles; no netplay rollback.
- Pixel-based sensory input with an explicit resolution/aspect transform; state inspection
is for task measurement, match lifecycle and display, not a hidden fighting policy.
- Fixed GameCube controller readout with bounded analog values and short frame-based holds.
- Native-resolution rendering initially, local broadcast at 30 fps first; simulation/input
continue at the backend's approximately 60-Hz cadence. Promote to a 60-fps show only after
the full media path and encoder are verified.
- Two-fly synthetic arena remains the framework test case before game integration.
The important scaling change is **one environment with many agents**, not simply a larger
ROM. Disc size is mostly a loading/storage concern. Runtime cost comes from PowerPC emulation,
graphics/audio, multiple neural simulations, synchronization and media copies.
### Confidence labels used below
- **Observed in source:** checked implementation or explicit upstream documentation.
- **Recommended:** proposed architecture/configuration, not implemented here.
- **Must measure:** cannot be established by reading code, including throughput, latency,
correct input-to-frame association and reliable recovery on the target platform.
## 2. What the Melee decompilation gives us
The supplied [doldecomp/melee](https://github.com/doldecomp/melee) repository is a matching
decompilation of **US 1.02**. Its README is `.github/README.md`, not the repository-root
`README.md`. The inspected revision is recorded in section 13.
The README and `config/GALE01/config.yml` identify the matching `main.dol` SHA-1 as
`08e0bf20134dfcb260699671004527b2d6bb1a45`. That identifies the executable, **not the entire
disc image**. Our run manifest must separately identify externally supplied game content,
effective executable, modifications, emulator build and task interpretation.
The decomp builds a GameCube executable, not a supported desktop port or emulator replacement.
The `dolphin` code within that source tree refers to Nintendo's SDK, not the Dolphin emulator
project. Rebuilding/relocating a DOL for instrumentation changes address identity; never use
stock symbol addresses on a shifted build.
### 2.1 Useful inspection map
| Upstream source | What was observed | How it helps our task |
| --- | --- | --- |
| `config/GALE01/{config.yml,symbols.txt}`; `docs/symbols.md` | Matching binary identity and named symbols with addresses, sections and attributes | Reproducible symbol/inspection manifest analogous to the Pokémon symbol generator |
| `src/melee/pl/player.h` and `player.c` | `StaticPlayer`, getters for stocks/damage/controller index, KO-by-player counters and self-destructs | Distinguish controller port, player slot and match attribution instead of assuming they are identical |
| `src/melee/ft/types.h` | `Fighter`, player/controller identifiers, buffered sticks/triggers/buttons, pressed/released edges, damage state, source-player field and move-instance information | Audit controls and potential reward attribution; source fields are hypotheses to validate against live transitions |
| `src/melee/gm/types.h` | Match frame/timer fields, `MatchEnd`, winner arrays and exit/results structures | Episode boundaries, timeout/results interpretation, one-time terminal rewards |
| `src/melee/gm/gmvsmelee.h` | Character/stage select, versus entry/exit, sudden-death and results transitions | An explicit lifecycle model instead of treating every screen as a playable frame |
| `src/melee/{cm,gr,mp,it}/` | Camera, stages, map/collision and item subsystems identified by upstream module structure | Follow-up inspection points for view geometry, hazards and projectiles; not all audited in this pass |
Examples of concrete distinctions:
- `StaticPlayer` has a controller index, a player ID and up to two sub-fighter entities.
Ice Climbers and transformations defeat “one visible fighter object = one agent.”
- The input struct tracks three-entry analog/button histories plus pressed/released buttons
and threshold timers. A constant button hold and a sequence of taps are different actions.
- Damage includes an annotated source-player number, but it must be checked for projectiles,
stale ownership, self-damage and indirect KOs before it becomes a reward source.
- The match structures include winner counts/arrays. A terminal result is not safely inferred
from whichever player's stock decrement happened to be sampled first.
### 2.2 How to use it without making the framework Melee-specific
Create a **task-local** inspection catalog: field name, binary/decomp revision, symbol or
pointer traversal, type/endianness, valid scenes, tested transitions and unsupported cases.
Prefer Slippi telemetry for fields it supplies reliably; use decomp-grounded inspection for
missing fields only after verification. Do not expose all emulator memory to every agent.
If memory inspection is needed, decode big-endian integer/float fields and emulated 32-bit
pointers explicitly. Never cast emulated bytes to a native Rust/C struct whose layout,
pointer width or bitfield ordering is different. Sample a consistent backend boundary rather
than reading a moving process asynchronously. Accessors named in the decomp explain semantics;
they are not functions our host process can call in place of an adapter.
Derive small constant/schema outputs and synthetic fixtures where appropriate. Do not vendor
the entire decomp, game executable, assets or save states merely to read stocks and damage.
The first working backend does not require rebuilding Melee. Custom hooks or patches are
separately identified scaffold and enter the run's behavior/content identity.
## 3. Emulator options and recommendation
All candidates below emulate GameCube software. The differentiator is the host integration,
not whether Melee can theoretically boot.
| Candidate | Evidence / useful capability | Gap or cost | Decision |
| --- | --- | --- | --- |
| **Mainline-based Slippi Dolphin + maintained libmelee** | Structured game/port events, controller pipes, explicit blocking-input support; Linux rendered path available in the ecosystem | Pixel/audio access and coherent external save-state control still need integration; pin emulator, Gecko codes and parser together | **First spike and preferred initial integration** |
| **Stock Dolphin + narrow host hook** | Source has `Core::DoFrameStep`, CPU-thread coordination and `State::{SaveToBuffer,LoadFromBuffer}` | These are internal APIs, not a stable remote environment SDK; own a small patch and state parser/telemetry bridge | Fallback or eventual generic Dolphin backend if its maintenance cost is justified |
| **Felk Python-scripting Dolphin branch** | Inspected stubs expose controller/memory/save-state scripting and rendered-frame events | Historical branch; `await frameadvance()` is documented as waiting for a rendered-frame event, not proof of a paused one-step transaction | Research reference or temporary probe, not default production dependency |
| **Custom EXI/fast-forward Slippi-Ishiiruka** | Maintained libmelee README describes accelerated ML mode and EXI inputs | That documented fast path disables rendering; inspected libmelee rejects non-Null graphics for the EXI_AI build | Useful for explicitly state-driven offline research, not the pixel-fed live baseline |
| **Libretro Dolphin core** | Potential common frontend ABI | No core-specific synchronization/state/render benchmark was performed; another integration layer does not remove task semantics | Defer rather than introduce an unverified second dependency stack |
| **Native game port built from the decomp** | Source enables modding/research | Matching DOL compilation is not native execution; graphics, OS/SDK, timing and assets remain substantial work | Outside this project's first Melee phase |
Use the maintained **`vladfi1/libmelee`**, not a floating install selected by an old tutorial.
`altf4/libmelee` says it is archived and points there. The maintained fork says it became
the PyPI `melee` source starting at 0.45.0; pin the actual chosen package/source revision and
its dependencies instead of assuming an unversioned `pip install melee` reproduces a run.
The maintained README describes raw-state compatibility, but the inspected `Console.step()`
still invokes `__fixframeindexing` and `__fixiasa`. Therefore, confirm field semantics from
the installed source and observation fixtures rather than trusting README wording. Our
adapter identifies parser/normalization revision as part of task identity.
### 3.1 What the blocking path actually does
**Observed in source:**
1. `libmelee.Console` defaults `blocking_input=False`. Setting it true writes
`Slippi/BlockingPipes` for its mainline backend.
2. `Console.step()` flushes its registered controllers and then dispatches game/menu events
until a frame boundary. It is not a method returning RGBA pixels or arbitrary game state.
3. Slippi's `EXI_DeviceSlippi.cpp` sets `g_need_input_for_frame` on game setup, menu frames
and frame bookends.
4. `Pipes.cpp::UpdateInput` checks the blocking setting and flag, waiting for commands through
`FLUSH`. Its Linux wait path uses `select`; the inspected Windows wait helper is not implemented.
5. `ControllerInterface::UpdateInput` updates devices and only then clears the flag. The
`FLUSH` handler explicitly avoids clearing it before the other devices have been read.
That is strong evidence for trying a Linux multi-port blocking backend. It is **not yet a
measurement** that action batch `t` affects exactly our desired frame `t+1`, that menu/game
boundaries behave identically, or that rendering/audio are coherent with the telemetry.
Keep at most **one batch outstanding**. The pipe implementation consumes buffered commands,
and a backlog of frame batches must not collapse into a latest-state input. Create an isolated
Dolphin user directory containing only the intended pipe devices: unused/abandoned devices
can participate in updates and leave blocking input waiting on a controller nobody drives.
For two flies, stage both full controller states before calling the single owner's
`Console.step()`. Do not give each agent a `Console` loop. Verify which input sample corresponds
to returned pre/post-frame telemetry with distinguishable action pulses and deliberate delays
on each port. “Both bots sent commands” is weaker than “both commands landed on one frame.”
### 3.2 What libmelee does not establish for us
- No framebuffer-returning or whole-session save/load interface was found in the inspected
`Console` API. `DumpConfig` configures media dumping; a dump is not automatically a bounded,
timestamped sensory-frame transport.
- GameCube pad values are stateful. Unchanged buttons remain held; the backend must emit an
explicit complete state or compute trustworthy deltas, including release/neutral values.
- The library applies analog normalization (`fix_analog_stick`, `fix_analog_trigger`). Define
our canonical ranges and apply conversion exactly once; test round trips at neutral,
extremes, diagonals, dead zones and trigger-click thresholds.
- Rollback skipping and internal controller flushes are present. Use local offline matches
first; do not confuse filtered rollback frames with advancing brains through speculative time.
- Initial game events can flush neutral input internally. Record startup as lifecycle scaffold;
do not attribute that to a neural decision.
- Spectator transport keepalive, pipe blocking and process liveness are separate. A healthy
connection does not prove the match or renderer is advancing.
## 4. Audit of our current system: keep, extract, replace
Rust path aliases: `core/`, `gb/`, `sim/` mean the respective crates under
`services/flysim/crates/` named `flybrain-core`, `flybrain-gb`, and `flysim`.
| Finding | Current evidence | Required change for Melee/multiple flies | Priority |
| --- | --- | --- | --- |
| Single world and brain bundled together | `sim/src/simloop.rs::Sim` owns one `NeuralAgent`, concrete `Emulator`, adapter and ratchet | Session owns one environment plus agent collection and explicit port map | Blocking |
| Direct Game Boy calls in frame loop | `step_frame`, `to_button_mask`, `set_buttons(u8)`, `run_frame`, fixed framebuffer copy | Backend interface with full action batch, rational cadence, observations and media capabilities | Blocking |
| Task trait carries Pokémon concepts | `gb/src/adapter.rs` includes `read8(u16)`, map/exit/objective hooks | Keep inspector and macros task-local; use generic reward/episode/progress outputs | Blocking |
| Timing defaults embed Game Boy | `core/src/agent.rs`, `sim/src/config.rs` | Session clock derived from backend, per-agent neural remainders; preserve old arithmetic in legacy facade | Blocking |
| Decoder tuned to walking through Pokémon maps | `packages/brain/src/readout/presets/gameboy.ts`: 800-ms directions, 85-ms A/B pulses with 480-ms cooldown | New fixed GameCube mapping; frame-scale controls, analog sticks/triggers, concurrent movement/action | Blocking |
| Neural code is already independently useful | `core/src/lif.rs`, `agent.rs`, `plasticity.rs`; TS counterparts | Reuse exact kernel and private per-agent state; do not replace neural semantics to integrate a game | Keep |
| Graph can be shared, pool cannot be concurrently dispatched | `Arc<BrainDataset>`, `SweepPlan`, `core/src/pool.rs` shared job slot | Immutable topology shared; distinct mutable state and controlled total scheduling budget | Blocking |
| CUDA exists but not an automatic service optimization | `core/src/lif/cuda.rs`; no `enable_cuda` call found in `sim/src/simloop.rs` | Explicit backend selection, equivalence/restore tests, profile first; don't promise GPU brains from graphics availability | Measured option |
| Snapshot header is a single Pokémon-shaped view | `sim/src/snapshot.rs`, `packages/feed/src/{types,codec}.ts` | Session descriptors, multiple agents/ports, task-specific progress, named attachments/media | Blocking for proper broadcast |
| Stage assumes Game Boy geometry and one fly | `apps/stage/src/{App.tsx,feed/store.ts,feed/decode.ts,lib/geometry.ts}` | Per-session/agent store instances, dynamic view aspect/dimensions, multi-agent match layout | Blocking for proper broadcast |
| Browser owns game audio | `apps/stage/src/audio/engine.ts`, 48-kHz feed, Pulse capture | Explicit audio producer and media-clock policy; do not play native Dolphin audio and forwarded PCM twice | Blocking |
| Capture already offers NVENC | `infra/bin/flycast-launch` | Reuse encode/relay/recording; measure new compositor/readback cost and revise 60-fps settings | Keep with changes |
| Encoder hardcodes H.264 level 4.1 | Both encoder functions in `flycast-launch` | 1080p60 needs a suitable level (normally 4.2 or automatic selection); changing only `FLY_FPS` is insufficient | Required for 1080p60 |
| Existing checkpoint payload is single-agent/binary-specific | `sim/src/store.rs`, `gb/src/compatibility.rs` | Coherent all-agent/world checkpoint plus backend/parser/patch/controller identity | Blocking for exact resume |
| Checkpoint writer queue is unbounded | `Sim::start_writer` uses `std::sync::mpsc::channel` | Bound/coalesce background work; larger emulator captures and several brain copies must not grow an unlimited queue | High |
| Existing “saved” event precedes durable commit | `checkpoint_with_reply` emits after enqueue; writer updates durable metrics on success | Distinguish capture/enqueue/commit/failure events; public status must not report a queued Melee save as durable | High |
| Recovery assumes a best progress ladder | `gb/src/{ratchet,recovery}.rs` | Match/episode reset policy; never rewind one player's world independently | Blocking |
| Health is mostly loop heartbeat | `sim/src/simloop.rs::Shared`, `sim/src/lib.rs` | Distinguish waiting at input barrier, intentional pause, backend timeout and deadlock; keep host supervision responsive | High |
| Deployment resource partitions reflect the old stack | `infra/units/flysim.service`, deploy cpuset construction | Budget Dolphin CPU/GPU plus N brains and media; measure and set new memory/process limits | Required before release |
| Bridge targets one sim | `services/bridge/src/{sim,commands,redemptions}.ts` | Explicit stable agent targeting and intervention policy; no viewer control-port endpoint | Before interactive show |
Do not interpret dated CPU/GPU measurements in the repository as current free capacity.
The records identify bandwidth contention and graphics-sharing constraints, but this audit
does not inspect the host or establish that it can run two brains plus Dolphin in real time.
## 5. A framework architecture that survives a third game
### 5.1 Four runtime components, not one enormous adapter
```text
Application / supervisory logic Presentation / recording
\ /
Flybus RPC + pub/sub
/ | \
Session coordinator Agent workers Backend helper
clock / barrier brain / encoder Python + libmelee initially
task + executors fixed readout owns Dolphin process/user dir
episode policy native pipes, parser, media/state hooks
\ | /
owned artifact references
|
immutable local store
Presentation owns stage/compositor → capture → local relay → optional Twitch push.
```
The helper is an **internal backend implementation**, not an audience-accessible controller
service. The session remains the sole authority assigning actions to ports. Python never
simulates the neurons or selects actions. Keep it if measurements say its overhead is small;
replace its internals with Rust only when an actual bottleneck or maintenance
requirement justifies it.
Its Flybus binding uses the same protocol as every component. Dolphin-specific input pipes
stay behind the environment adapter, not a second framework communications system.
An external emulator is a normal `Environment` implementation, not a special `if melee`
branch sprinkled throughout the session. For Game Boy, the same interface has an in-process
implementation. For an embodied world, it may be a native physics engine. Consumers do not
need to know which one owns the world.
### 5.2 Framework contracts to extract
| Contract | Owns | Must not know |
| --- | --- | --- |
| `Brain` / numerical core | Tick semantics, state, spikes, rates, learning updates | Game, process, controller labels or viewer |
| `SensorEncoder` | Declared observation→neural drive transform | Reward inspector state not declared as input |
| `Readout` | Fixed rate/signal→control-channel mapping | Opponent strategy, game addresses or pathfinding |
| `ActionExecutor` | Selected action + coherent game state + task progress + clock → controller state | Authority to invent a winning action when brain is silent |
| `Environment` | Native clock, port schema, action commit, observations/media, snapshot capabilities | Neural roles, Twitch or task reward weights |
| `Task` | Typed state interpretation, rewards, progress and episode outcomes | Direct neural mutation or direct controller writes |
| `EpisodePolicy` | Start/end/reset/recovery semantics | Hidden per-player rewind in a shared world |
| `Session` | Barrier, identity, agent isolation, routing, state capture and supervision | Melee memory offsets or Pokémon map IDs |
| `Flybus` | RPC/pub-sub routing, bounded deliveries, artifact ownership and GC | Game timing, neural roles, presentation/stream semantics |
| `Presentation` | Descriptor-driven layout, media/audio and task panels | Emulator stepping or access to controller pipes |
Use modules first, then crates/packages as second consumers appear. The existing monorepo
and Rust workspace can stay in place during extraction. Keep static compiled registries
initially; a stable dynamic-plugin ABI and public package registry are not prerequisites.
**Framework acceptance test:** adding a synthetic third environment/task requires a backend
implementation, a task/profile and a composition manifest, not edits to the session loop,
neural core, protocol enums or generic stage store. A game-specific presentation plugin is
allowed. Wire extensions must be namespaced/schema-validated, not hardcoded into every panel.
### 5.3 Backend RPCs over the common bus
Use Flybus for these conceptual capabilities; do not implement another socket/message
envelope. Exact method names/bodies are in the session-framework contracts. Pause/Resume
are session lifecycle intents; an environment adapter holds its boundary between Advances.
```text
Hello → backend build/content/patch identity, capabilities, cadence, ports, views
Initialize(runConfig) → Ready(epoch, observationBoundary)
Advance(epoch, expectedBoundary, completePortBatch)
→ StepResult(epoch, newBoundary, appliedBatchId, observation, mediaRefs)
Pause / Resume → explicit acknowledgment
Capture(epoch, boundary) → capture token + state digest + snapshot bytes/reference
Restore(captureToken) → new epoch + restored observation + success/failure
Shutdown → acknowledgment or bounded forced process termination
```
Only advertise `Capture/Restore` if implemented and tested. Otherwise expose an explicit
`restart_episode` capability and visible aborted-match policy; never claim exact resume.
Port batches use canonical buttons plus sticks in [-1,1] and triggers in [0,1], converted
once by the backend. Descriptor/schema versions pin conversions and active ports.
Commands and pub/sub envelopes contain metadata and ArtifactRefs; native media/checkpoint
bytes live in Flybus's managed immutable store. Include producing epoch/frame/sample identity
and format in domain descriptors. DeliveryGuard keeps bytes alive beyond message drop if an
encoder/renderer retains a handle. Cached RPC replies own handles for retry; last-owner GC
reclaims data. Start file-backed; pooled shared-memory reuse is an optional later optimization.
Router restart invalidates routes/handles and requires coherent session recovery.
One request in flight, explicit timeouts, bounded queues. After an uncertain `Advance`
response, do **not** resend blindly: the world may already have advanced. Resolve batch ID/
boundary through an idempotent reply cache or fail/recover the session. A pipe transport with
no application acknowledgment needs validation against returned input telemetry, not a claim
of atomicity it cannot prove.
Separate lifecycle I/O from the blocked advance operation so diagnostics/shutdown stay alive.
Do not issue a save operation scheduled on Dolphin's CPU thread while that same thread is
waiting forever for pipe input. Capture/pause needs a backend-owned quiescent point where
both the simulation and pending input consumption have known state.
## 6. Multiple flies and the timing contract
### 6.1 One match, one world clock
Two flies in Melee normally means two controller ports in **one Dolphin instance**. Four
flies means four ports, not four copies of Melee joined through netplay. Several independent
matches are separate sessions/process trees, orchestrated by application code developed with
its presentation. A tournament director is one example, not a mandatory framework service.
For each committed boundary:
1. Freeze each agent's allowed observation from the same world state.
2. Advance every brain by the environment's elapsed emulated time using its private remainder.
3. Decode independent actions; step any declared executors; assemble a complete port batch.
4. Commit the batch once and let the backend advance to the next acknowledged boundary.
5. Associate rendered sensory frames and telemetry with their actual producing boundary.
6. Route task events/rewards exactly once; encode the next input and publish a snapshot.
Warm-up settles/calibrates brains with learning off while the environment is held at its
initial boundary. The next observation must not be from a game that ran freely through the
warm-up. Fixed offsets between brain and environment clocks are recorded.
Use the selected backend's rational emulated cadence, not `GAMEBOY_MS_PER_FRAME` or an
unexamined exact 60. An approximately 60-Hz budget is about 16.7 ms, but logical game frame,
video interrupt, input poll, rendered presentation and Slippi frame bookend are distinct
events until the spike establishes their mapping. Session ticks are monotonic even when
Melee's signed frame number resets, starts before zero, or changes across menu scenes.
### 6.2 Render latency and agent fairness
Dual-threaded graphics can present frame `n` after telemetry for `n` is available. Label the
actual frame; do not attach “latest screenshot” to current state and assume equivalence.
Start with one declared fixed observation latency shared by all agents, and measure it.
If a pipeline deliberately adds one frame of latency, record that in the sensor profile.
The full-resolution spectator view may be delayed separately, provided overlays use the
matching presentation timestamps rather than future task data.
Also test a potential pipeline deadlock: the helper waits for a rendered image while Dolphin
is waiting for the next controller flush needed to reach that presentation event. Fix the
backend's rendezvous or select a declared previous-frame sensory latency; do not unblock it
with an undisclosed neutral gameplay input. Game-state bookends alone do not prove the GPU
has completed a matching frame.
Do not reduce brain integration from 1,000 to 500 ticks per emulated second to meet wall-clock
deadlines. That changes the model. A declared action-repeat interval can reduce decisions,
but normally still requires all neural ticks and correctly accumulated intermediate task
events; it does not halve the principal neural cost. Rendering every second game frame is
also a sensor change if the brain otherwise sees each frame, not merely a broadcast setting.
On slow compute, the default is to slow the entire local match and report real-time factor.
Do not let one fly continue while the other misses turns. On a dead participant/backend,
pause or abort the match visibly; neutral fallback play is not silently substituted.
### 6.3 No rollback netplay in the first release
Slippi supports online play, but we do not need it to connect two local flies. Online rollback
would require rewinding **all** neural states, RNG, decoder/executor state, reward ledgers
and admission decisions at the same speculative boundary as the game, then replaying inputs.
Filtering repeated frames in libmelee is not that system. Keep offline local matches and
assert monotonic committed observations per epoch; classify unexpected rollback as an error
or explicit recovery transition rather than double-rewarding it.
## 7. Controller, sensory and learning design
### 7.1 A GameCube controller is not an eight-bit pad
Support independent main and C sticks, analog shoulders, digital trigger clicks, face
buttons, start and D-pad. Movement and attack can overlap. Canonical neutral/release state
must be complete, so a missing command cannot accidentally leave attack or shield held.
At about 60 Hz, the current 800-ms direction hold is roughly **48 game frames** and the
85-ms pulse roughly five. Reusing these values would dominate the fly's behavior regardless
of the dataset. Create a fixed Melee readout with explicit decisions in integer game frames,
bounded analog mappings, dead zones, tie handling and pulse/hold policies. Start with a small
declared set of stick magnitudes and directions if that makes validation easier; continuously
valued mappings can follow as a separate profile.
Audit tap-jump, directional aerials/smash attacks, jump release, shields, simultaneous axes,
and conflicting inputs. Don't add state-conditioned auto-aim, automatic edge recovery or
combo execution under the label “controller mapping.” If later desired, publish those as
separate macro/assistance profiles with their own identity and comparison baseline.
Start/system controls are lifecycle-sensitive. During an active match the profile may omit
pause entirely; initial match setup and between-match reset are disclosed episode scaffolding.
This is not permission for an API to press buttons. Specify whether setup uses an audited
initial state, internal deterministic menu setup, or a reset hook, and mark those frames as
non-neural setup with learning disabled. That expands the legacy “all presses” phrasing and
requires a deliberate task-policy/documentation decision before shipping it.
### 7.2 What the fly sees
For the initial pixel profile, both flies receive the same shared game camera with fixed
crop/aspect treatment, independent of spectator overlays. The legacy 160×144 input should
not stretch a 4:3 scene silently. Compare an aspect-preserving downsample/letterbox transform
with a separately versioned input-size profile; changing kernel retina dimensions currently
affects the numeric configuration identity.
Current L1 projection samples luminance at 1,572 columns. Higher broadcast resolution does
not produce more sensory neurons, color recognition, motion estimation or knowledge of which
fighter the fly controls. Test whether each selected character remains visible across zoom
and stage movement; record the sparse sensory representation rather than assuming a human-
readable video is an adequate neural input.
Three distinct modes must not be conflated:
| Mode | Neural observation | Rendering implications |
| --- | --- | --- |
| Pixel baseline | Actual game image through fixed encoder | Needs real rendered frames even if no desktop GUI is shown |
| Structured-state experiment | Explicitly encoded positions, velocities, stocks, etc. | Can potentially use Null/fast-forward, but it is a new privileged-input model |
| Spectator-only rendering | Whatever the profile specifies; video for audience | May be independently compressed/delayed, never silently substituted for sensory input |
“Headless” can mean no GUI while still rendering; “Null graphics” generally means no useful
pixel observation. The documented EXI fast-forward speed path cannot be advertised as the
performance of our pixel-fed broadcast.
### 7.3 Reward and outcome attribution
Implement a `MeleeTask` with typed per-player observations, match state, a ledger and positive
reward events. Its schema belongs to the task, not a generic `GameMode` enum. Start small:
- Terminal match outcome, once, based on validated results/termination reason.
- Opponent damage and credited KOs only after verified ownership information is available.
- No reward for mere button activation, for losing a stock, or for scripted setup.
Do not reward A for every increase in B's percent: self-damage, stage effects, reflected
projectiles, teams and another sub-fighter can invalidate that inference. The decomp's source-
player and KO tables guide inspection, but no runtime correctness is claimed until tested.
Unknown attribution produces a logged observation without a guessed reward. Keep fractional
damage until the rule deliberately quantizes; HUD damage and fighter damage may differ.
Deduplication keys include epoch/match, producing frame and event identity. Handle multihits,
trades, simultaneous KOs, respawn percent reset, timeout, sudden death and disconnection as
separate cases. If source telemetry cannot distinguish a required case, narrow the first
ruleset or add a specific audited observation hook.
Learning remains private per fly and synthetic reward modulation remains distinct from PAM
stimulation. Disable learning during kernel/controller/backend characterization; later compare
learning-on with learning-off, repeat seeds and swap sides/characters. Retaining gains between
rounds is a run policy. Competitive success is not guaranteed by increased model complexity.
## 8. Performance plan: measure the actual critical path
### 8.1 Budget equation
For a lockstep pixel-fed match, approximate the critical wall-time interval as:
```text
T_step = T_brains + T_readout/task + T_bus_RPC
+ T_emulation_to_observation + T_required_render_readback + T_boundary_overhead
T_brains ≈ sum(T_agent_i) [sequential evaluation]
T_brains ≳ max(T_agent_i) + barrier cost [parallel with sufficient independent resources]
```
The parallel estimate is a lower bound, not a promise: shared caches, memory bandwidth,
GPU contention and scheduling can make every brain slower. Media publication/encoding and
storage should be off the critical path, but their resource use and state capture still
affect it. Do not obtain a “60 fps” claim solely from Dolphin's display counter while the
brains advance fewer milliseconds or repeat stale observations.
Proposed capacity gate: warm full-stack **unthrottled** throughput at least 1.2× the selected
game cadence for two flies, then a paced one-hour soak with no growing queues/lag and a
24-hour local endurance run before release. In the paced run, distinguish intentional wait
from compute time; report p50/p95/p99/max compute interval and deadline misses. The 1.2×
margin is a proposed engineering target, not a measured capability of current hardware.
### 8.2 CPU/GPU strategy
1. **Keep the first two brains on CPU.** Establish Dolphin JIT/render/media cost separately.
The repository's current service is CPU-composed even though a CUDA kernel exists.
2. **Allocate a total physical-core budget.** Compare sequential agents with modest within-
brain pools against agents running concurrently on disjoint core groups. Include Dolphin's
CPU/JIT thread, graphics worker, helper, browser, encoder and storage in the budget. Do not
launch four copies of the current per-brain pool size by default.
3. **Use native-resolution hardware rendering first.** Compare OpenGL/Vulkan on the chosen
build/platform; no blanket claim that one is faster. Measure render correctness and readback.
JIT is the performance baseline; an interpreter is a diagnostic baseline, not the live plan.
4. **Benchmark shader compilation and caches.** Report cold and warm starts separately. Choose
supported shader modes from measurements; a cache that hides startup hitches is not a
guarantee that a new stage/character will not compile something mid-match.
5. **Use NVENC where available, with a measured fallback policy.** Its encode engine does not
remove GPU rendering, memory allocation, color conversion or framebuffer-readback cost.
Automatic fallback to x264 can consume the cores the brains/emulator need; expose the
resulting degradation and test whether the declared session can still meet cadence.
6. **Only then test CUDA brains.** The current backend retains RNG/plasticity observation/rate
work on the host, uploads state inputs, and by default synchronizes membrane/refractory
state back each batch. It also allocates device graph/state per backend instance. Measure
two/four agents alongside Dolphin, browser graphics and NVENC; zero-copy shared graphs and
a GPU-wide scheduler are possible later work, not present features.
For CUDA, preserve bit-exactness, gain-update ordering and checkpoint synchronization.
Batching across a future game action boundary is not valid just because it improves kernel
throughput. Keep the TypeScript oracle and existing version strings intact.
The existing VirtualGL/Xvfb result demonstrates one Chromium rendering path, not that
Dolphin Vulkan/OpenGL works or is performant in the same container. Test the complete selected
graphics path. Reusing GPU passthrough requires no assumption of exclusive VRAM availability.
Resource availability must be measured in an approved, serialized deployment-host session.
### 8.3 Media bandwidth and copies
Uncompressed RGBA estimates, before copies/framing:
| Image/cadence | Bytes per second |
| --- | ---: |
| Existing 160×144 at 30 fps | 2.76 MB/s |
| 640×480 at 30 fps | 36.86 MB/s |
| 640×480 at 60 fps | 73.73 MB/s |
| 1920×1080 at 60 fps | 497.66 MB/s |
640×480 is a planning example, not an asserted fixed Dolphin framebuffer size. The backend
advertises actual dimensions/format/aspect. One shared camera is delivered once for the
match; two flies can sample one immutable image without duplicating its transport. If their
sensor transforms differ, encode separately against the same source frame.
Start with native frames as immutable artifacts: 480p is not an automatic reason for a codec
or second transport. Flybus carries handles only; multiple readers map the same object. Measure
renderer readback, optional seal copy and consumer reads separately from router overhead.
Sensor downsampling remains a declared profile operation (optionally optimized near capture
after measurement). Viewer scaling/composition/encoding belongs to presentation. Last-use
handles, not receipt acknowledgments, govern GC; required sensory input cannot be coalesced.
### 8.4 Benchmark ladder and decision records
| Run | Configuration | Question / recorded output |
| --- | --- | --- |
| B0 | Synthetic two-port backend plus common Flybus, no neurons | Routed RPC latency, pub/sub fairness, step barrier, timeout and artifact-GC behavior |
| B1 | Dolphin with fixed input traces, rendering/audio on, no brains | Cold/warm emulator cost, step timing, port alignment, render-to-state latency |
| B2 | Same run + helper/media extraction | Incremental parsing, copying, downsampling and audio cost |
| B3 | One FAFB brain | End-to-end reference and per-phase costs |
| B4 | Two FAFB brains, sequential vs parallel schedules | CPU/cache/bandwidth limits, balanced observation and input timing |
| B5 | B4 + actual stage, capture, relay, recording, checkpoints | Full critical path, A/V drift, encoder fallback and queue growth |
| B6 | Four brains and four active ports | Capacity characterization only until this independently passes the same gates |
| B7 | Matched MaleCNS and optional CUDA variants | Dataset and backend effects, measured independently before combined variants |
Record backend/content/patch/profile digests, physical-core allocation, exact graphics settings,
sensor/broadcast resolutions, all clock rates, resident/peak memory, VRAM, thread usage,
real-time factor, latency distributions, audio under/overruns and dropped frames by purpose.
Store operator-specific machine details externally and publish only the portable methodology
and non-identifying results. No measurements were performed by this document-writing task.
## 9. Broadcast architecture and audio ownership
### 9.1 Two viable routes
Both routes are **application/presentation implementations**, not additional framework buses.
The environment emits native frame/audio artifacts through Flybus in either case.
**Route A — stage receives game media.** A presentation gateway subscribes to native artifacts,
delivers them to the browser, and the stage composites game/overlays for capture. Start with
native uncompressed artifacts internally. Browser-edge delivery can remain raw or use a codec
after measurement; avoid encode→decode→encode unless its tradeoff is justified. The gateway
holds artifacts through conversion/use and then releases them; browser backpressure cannot
pin required sensory state without a bound. Overlay timestamps track displayed video.
**Route B — compositor combines native game output and stage overlay.** Dolphin supplies its
rendered output to a compositor; the browser supplies a separate overlay surface. This can
avoid moving full-resolution game pixels through JavaScript, but requires an explicit shared
clock and a new capture composition. The fly's sensory image still needs a frame-identified
path from the backend. Capturing a desktop window on a wall clock is insufficient to establish
which image a brain used at a given game boundary.
**Recommendation:** prototype Route A for the two-player local slice; benchmark Route B in
the media spike before committing to the long-run high-resolution pipeline. Preserve media
as a capability behind the environment interface so the choice does not change brain/task code.
The browser-facing v2 contract can reference media streams; it does not dictate the internal
bus, native sensor format or artifact-store implementation.
### 9.2 Audio and clock policy
Today the page plays binjgb PCM and stream SFX into the Pulse sink. With Dolphin, select one
of these explicitly:
- Dolphin PCM is captured/forwarded and played by the page, with native device output muted.
- Dolphin renders audio to the capture sink directly, and the page contributes only SFX.
Do not run both. Declare sample format/rate, resampling location, timestamps, buffering and
discontinuity handling. Pause/reset/restore must flush or relabel buffered old-episode audio.
If wall time falls behind, measure pitch/time-stretch behavior; do not let “async resample”
hide minutes of simulation lag. Game timestamps, not arbitrary browser receipt time, define
the intended A/V relationship.
Keep 30-fps broadcast and approximately 60-Hz gameplay as independent settings. For 1080p60,
the existing H.264 level 4.1 is too low for the normal macroblock-rate requirement; use a
compatible level such as 4.2 or encoder-selected level and validate the actual stream. Also
measure capture/compositor cadence, bitrate quality, encoder lookahead/latency, local recording
and audio synchronization. `FLY_FPS=60` alone is not a completed performance upgrade.
### 9.3 Presentation changes
The generic stage needs descriptor-driven game aspect ratio, two/four agent cards, per-port
button/stick indicators, per-agent learning/sugar state, shared match stocks/percent/results,
and scoped events. Neither “badges” nor “highest ladder rung” describes a match.
Separate task data schema from layout. Keep one compositor clock and one selected world audio
stream; keep neural maps and rate scalers private per agent/dataset identity. Defer expensive
four-avatar/whole-connectome rendering until measured. All actual layout decisions require
PNG mockups and existing legibility/browser checks. This text does not approve a screen.
## 10. Persistence and unattended operation
### 10.1 Savestates are a capability, not a libmelee assumption
Dolphin source provides buffer/file state operations, but `State.h` documents that operations
called off its CPU thread may be scheduled rather than executed immediately. An external
“save requested” is therefore not proof of a consistent capture at our agent boundary.
Slippi's internal rollback save-state commands likewise do not constitute an audited public
multi-agent checkpoint API.
The selected backend must supply acknowledgment of the frozen boundary, resulting state
digest and completion. Save every brain, RNG, rate/calibration state, learning state, sensor/
executor state, pending action identity, task ledger and environment together. Clock and
media epochs change on restore; discard pre-restore spectator/parser buffers.
Re-create or explicitly reinitialize libmelee's parser caches and controller history after
restore. An emulator savestate does not include an external helper's `_frame`, previous
game state, normalization state or queued pipe data. Test how game-start metadata is supplied
when loading into mid-match; some telemetry protocols may need reseeding or restarting.
When exact mid-match capture is unavailable, an initial prototype may visibly abort and
restart a match while retaining a declared brain checkpoint. Mark `resume=episode-restart`
in the descriptor. That is a deliberate narrower capability, not equivalent to crash resume.
It is not ready for a release that promises uninterrupted exact match continuation.
### 10.2 Storage and health
Bound pending captures and coalesce replaceable hot checkpoints. A durable request either
completes with a commit acknowledgment or fails explicitly; never drop it while reporting
success. Match-end result records are append-only and independent of world rewind.
Keep backend/helper/brain health separate from “game frame did not advance.” An intentional
pause or waiting barrier is not a crash; a dead helper must not keep the session green by
merely refreshing an HTTP heartbeat. Export last completed boundary, in-flight request age,
barrier participant status and renderer progress. A hard stall has a timeout and explicit
match abort/recovery path, not repeated blind restarts of unrelated services.
Manage Dolphin under the session's lifecycle or an explicitly coordinated systemd unit. If
Dolphin restarts, the session cannot keep sending frame `t+1` to a fresh match. Allocate unique
user directories, pipe names, telemetry ports and state namespaces per independent session.
Use deterministic configuration provisioning and checksums instead of reusing a developer's
desktop Dolphin settings or permitting auto-updates.
The existing `flysim.service` memory ceiling and CPU partition were sized for a different
process graph. Set new cgroup/resource limits from measured high-water marks; account for
backend process, multiple brain copies and checkpoint transients. Release preflight checks
the complete backend/game/patch/parser/profile identity. Rollback retains compatible state
as well as the old executable.
## 11. Staged implementation and go/no-go gates
This specializes the existing backlog rather than replacing its foundation/session work.
Melee-specific spikes can start before the full framework reorganization is finished.
| Item | Work and dependency | Evidence required before the next step |
| --- | --- | --- |
| **MELEE-01: backend selection spike** | Specializes EMULATOR-01. Pin mainline Slippi + maintained libmelee, content and Gecko codes; use isolated user config and two synthetic controllers | Boot/render/audio; block one then both ports; one batch/frame mapping; menu→match→results lifecycle; cleanup/restart. Choose this build or stock Dolphin + narrow hook based on results |
| **MELEE-02: capture/restore spike** | Alongside MELEE-01; prove sensory-frame identity, media export and save/load acknowledgment independently | Fixed pixel↔telemetry latency, bounded media storage, correct input after restore, parser/cache recovery. Explicit decision: exact resume or prototype-only episode restart |
| **MELEE-03: task observation audit** | Pin decomp; build field catalog, typed parser/inspector, lifecycle and synthetic event fixtures | Verified port/player/sub-fighter mapping; stocks/results; no guessed rewards; content/patch mismatches visibly disable unsupported semantic interpretation |
| **FRAMEWORK-01: generic backend + session** | Existing FOUNDATION-01/02, BUS-01..03 and RUNTIME-01/02; all workers/application events use Flybus | Same legacy Game Boy traces; synthetic environment uses identical session API; artifact-backed requests/results and no Melee logic in router/core |
| **FRAMEWORK-02: multi-agent and state** | Existing RUNTIME-03/STATE-01; integrate complete action batches, worker budget and chosen backend recovery capability | No cross-agent state leakage; one world step; changed evaluation order invariant; failed restore cannot partly install a match |
| **MELEE-04: fixed readout and sensory profile** | MELEE-01/02 + framework boundary; TS specification then Rust implementation for any new decoder semantics | Neutral/release, analog conversion, tap/hold/direction combinations, aspect-preserved neural input and recorded latency; no hidden combo/aim policy |
| **MELEE-05: first two-fly match** | MELEE-03/04 + FRAMEWORK-02; learning off, then audited positive rewards | Recorded action/observation timelines; paired side/seed trials; match terminal deduplication and visible reset/failure semantics |
| **MEDIA-01: full local show** | Existing WIRE-01/PRESENTATION-01; compare Route A/B, define audio owner and 30/60-fps profiles | PNG review, fake multi-agent fixtures, measured copies/latency/A/V drift; sustained media pipeline under checkpoint and shader-load events |
| **PERF-01: two-fly capacity gate** | Benchmark ladder B0–B5; optimize measured limiting phase | ≥1.2× warm unthrottled capacity target, one-hour paced soak and 24-hour local endurance; no queue/lag growth; documented CPU/GPU/memory envelope |
| **MELEE-06: expand carefully** | Passing two-fly slice | Four ports/teams, broader characters/stages, MaleCNS and CUDA are separate experiments, each with new tests and its own capacity result |
| **FRAMEWORK-03: finish packaging** | Existing PACKAGE-01 after useful second backend | Example third backend can be added without core/session/schema edits; isolated compositions package and preflight correctly |
**Stop conditions:** no dependable step/input barrier; no identifiable pixel source for a
pixel-input claim; inability to attribute rewards under the claimed ruleset; unsupported
restore marketed as exact resume; or sustained capacity below the declared cadence.
Respond by changing the explicit supported scope, backend or resources—not by quietly skipping
neural ticks, adding a gameplay bot, hiding game stalls or reporting guessed measurements.
### 11.1 Suggested first experiment script
The first implementation should be a local measurement harness, not the final stream:
1. Launch one pinned backend with known local content and two configured bot pads.
2. Enter a fixed local match through the declared setup procedure; record episode boundary.
3. Send distinct short left/right and A/jump pulse patterns on each port, including neutral
frames; log intended batches and observed raw/processed controller values.
4. Delay one port by a controlled wall-clock interval and verify no game boundary commits
until the complete batch is available. Repeat with port order reversed and four ports.
5. Capture a sequence of images/telemetry with frame identities; measure their association.
6. Save/restore at a known barrier if supported, replay the same actions, and compare task/
input traces; verify helper state and buffered media are reset coherently.
7. Kill the helper/backend separately and verify bounded failure without accidental continued
play or permanent hangs. Test paused-state health independently.
8. Measure compute with no brains, one brain, two brains, then the full broadcast stack.
Synthetic controller traces are test machinery, not footage presented as neural play. Keep
game content and environment-specific records outside source control; store portable metrics,
synthetic schemas and independently authored tests in the repository.
### 11.2 Test matrix that catches Melee-specific failures
- **Input:** two/four ports, inactive slots, delayed/missing flush, stale buffered commands,
full release, analog endpoints/deadzones, short taps, pressed versus held edges.
- **Identity:** controller↔player mapping, swapped ports, sub-fighters, transformations,
character/stage changes, wrong game revision, changed patch/parser normalization.
- **Events:** multi-hit, trade, self-damage, projectile ownership, stock reset, simultaneous
KO, timeout, sudden death, results re-entry, disconnect, restart after accepted reward.
- **Clocks/media:** game-frame reset, renderer lag, stale artifact/store identity, last-use GC, dropped
spectator frame versus required sensory frame, paused audio, mismatched overlay timestamps.
- **Recovery:** all-agent atomic validation, one corrupt state chunk, backend import failure,
helper parser not reinitialized, asynchronous save completion, hot-store coalescing and
durable-write failure. Test new exact resume separately from legacy transient-reset behavior.
- **Performance:** cold/warm shaders, high-activity matches, checkpoint capture bursts, CPU
encoder fallback, browser reconnect, two/four neural agents and measured GPU contention.
All implementation merges retain repository-required TS tests/typecheck, Rust workspace
tests and infra lint; UI changes add Playwright and PNG review. Game-backed jobs are explicit
operator-provided tests. Normal CI uses synthetic observations/backends and existing goldens.
## 12. Decisions to carry into implementation
| Question | Recommended answer now | Still requires evidence/choice |
| --- | --- | --- |
| Which emulator? | Dolphin, first trying mainline-based Slippi + maintained libmelee | Exact build selected by synchronized-input/media/state spikes |
| Use the decomp to run the game natively? | No; use it to audit task/state/controller semantics | Custom instrumentation only for specifically missing observations |
| One emulator per fly? | No for one match; one per independent session | Four-port capability must be tested, not inferred from two ports |
| Which brain? | Two existing FAFB agents for integration baseline | MaleCNS comparison after mappings/dynamics pass their independent gates |
| CPU or GPU brain? | CPU baseline, share immutable graph | CUDA versus CPU benchmark under Dolphin + capture, not in isolation |
| Inputs to the brain? | Pixels with explicit fixed transform | Structured state is a distinct optional research profile |
| Start with macros? | Fixed controller mapping, no hidden aim/combo policy | Any later assist profile is separately disclosed and evaluated |
| How fast? | Backend-native gameplay/input cadence, 30-fps initial show | Full-stack two-agent capacity; optional 60-fps broadcast and four flies |
| How to resume? | Whole-session coherent state where supported | Episode-restart prototype if exact state interface is not yet available |
| How generic? | Concrete environment/task/agent/session contracts and composition examples | Extract public packages only after second/third consumers prove the seam |
The first operator choices needed are the initial characters/stage/ruleset, desired show
cadence, and whether a visibly restarted match is acceptable during the prototype. They do
not block the synthetic framework work or source-level backend spike design.
## 13. Sources and audit scope
Local code evidence appears in section 4. Additional local files inspected include
`core/src/lif/cuda.rs`, `sim/src/pacing.rs`, `sim/src/simloop.rs::start_writer`,
`packages/brain/src/readout/presets/{gameboy,platformer}.ts`,
`apps/stage/src/audio/engine.ts`, `infra/bin/flycast-launch`,
`infra/units/flysim.service`, and the profiling/VirtualGL methods under `infra/docs/`.
External source snapshots inspected on 2026-09-18 (pin actual dependencies again at spike start):
| Repository/ref | Observed revision | Files used |
| --- | --- | --- |
| [doldecomp/melee](https://github.com/doldecomp/melee/tree/b9ec8a2eb48520753b2f8159ccc94d033fbf60ea) `master` | `b9ec8a2eb48520753b2f8159ccc94d033fbf60ea` | `.github/README.md`, `docs/symbols.md`, config, player/fighter/match headers and player implementation |
| [vladfi1/libmelee](https://github.com/vladfi1/libmelee/tree/bce21f09984b286e6d36bfd2939e4cd4691f94c2) `master` | `bce21f09984b286e6d36bfd2939e4cd4691f94c2` | README, `melee/console.py`, `melee/controller.py`, license metadata |
| [project-slippi/dolphin](https://github.com/project-slippi/dolphin/tree/41a7a3a110ed52999486ae1901c8fbb9a63d4f13) `slippi` | `41a7a3a110ed52999486ae1901c8fbb9a63d4f13` | Pipe backend, controller update loop, Slippi EXI events, `Core/State.h` |
| [dolphin-emu/dolphin](https://github.com/dolphin-emu/dolphin/tree/ee018d00e60b9eb727489908a8daec5c537f44a8) `master` | `ee018d00e60b9eb727489908a8daec5c537f44a8` | `Source/Core/Core/Core.h`, state/core module inventory |
| [Felk/dolphin](https://github.com/Felk/dolphin/tree/46b7eacd5c810c2d21ec5fe51ea1a9c61a7ceb3d) historical `scripting` branch | `46b7eacd5c810c2d21ec5fe51ea1a9c61a7ceb3d` | Scripting README, `python-stubs/dolphin/{event,savestate}.pyi` |
| [altf4/libmelee](https://github.com/altf4/libmelee/tree/1da979657122facd0750ea99cf6858255e198326) `main` | `1da979657122facd0750ea99cf6858255e198326` | Archive notice directing users to maintained fork |
Dolphin files inspected carry GPL-2.0-or-later headers; libmelee repository metadata reports
LGPL-3.0. Pin and retain actual dependency licenses/notices when packaging. A separate process
is an architectural boundary, not an assertion that distribution obligations disappear.
Source inspection supports the integration hypotheses and concrete constraints above. It
does not establish Dolphin throughput on the deployment hardware, verify any game-memory
field live, demonstrate a new neural behavior, or prove exact multi-port/frame/save semantics.
Those are the measured deliverables of MELEE-01/02 and the performance ladder.

122
docs/design/session-framework/README.md Normal file
View file

@ -0,0 +1,122 @@
# Application and session framework: architecture and contracts
Status: **implementation specification, draft 2**, 2026-09-18. This is a guide for future
agents; none of the new runtime is implemented yet. Baseline code is `83090a9` on
`docs/improvement-suggestions`. MUST/SHOULD requirements apply to the proposed new path, not
retroactively to existing [public feed](../../feed-protocol.md),
[control API](../../control-api.md), or legacy numerical/checkpoint behavior.
## Decisions from the architecture discussion
1. **Applications orchestrate components and develop their presentation alongside them.**
“Director” is application code, not a mandatory framework service. A tournament is an
example application, not the system's organizing data model.
2. **One lightweight Rust bus supports RPC and pub/sub everywhere internally.** Flybus replaces
separate direct worker transports and an application broker. No NATS dependency.
3. **Messages stay small; large artifacts live in managed storage.** Delivery guards and
explicit cache/retention owners keep data alive until its last actual use, then GC reclaims it.
4. **The router moves messages and tracks generic ownership.** It never schedules game frames,
understands macro actions, composites video or operates a stream.
5. **Sessions synchronize worlds; agent workers compute in parallel.** One logical clock is
not one execution thread. The coordinator alone commits complete world-control batches.
6. **Game-aware executors receive current game state and task progress.** Rich inspection data
does not become undeclared neural input.
7. **Native observations are framework outputs.** Resizing, overlays, browser delivery, audio
mixing, encoding, narration and streaming belong to the application/presentation layer.
## Read in this order
1. [Flybus v1](bus-v1.md) — authoritative wire/routing/RPC/pub-sub/artifact lifecycle contract.
2. [Session RPCs](ipc-v1.md) — domain payloads, worker capability negotiation and safe retries.
3. [Step protocol](step-v1.md) — session state machine, ordering and clocks.
4. [Worker/task interfaces](workers-v1.md) — exact method bodies and game-aware executor boundary.
5. [Session media/state](state-media-v1.md) — observation timing and coherent recovery.
6. [Application/presentation boundary](publishing-v1.md) — snapshots, flexible data and effects.
7. [Implementation guide](implementation.md) — sequenced build tasks and acceptance tests.
For context: [modular-session analysis](../malecns-modular-sessions.md) and
[Melee audit](../melee-framework-audit.md). Each contract owns its named subject; step ordering
wins over an informal diagram, and Flybus owns transport/resource rules. Resolve contradictions
before implementation. The [existing HTML report](report.html) is an overview, not a contract.
## 1. Composition and processes
```text
Application / supervisor Application presentation
run policies, identity, history UI, media composition, audience, stream
\ /
Flybus: RPC + pub/sub
/ | \
Session Agent workers Environment worker
coordinator brain + encoder emulator/world + native observations
task/executors + fixed readout
|
artifact store (same bus API)
```
This diagram is connectivity, not execution order. The step contract defines causal order.
One router can host several independent sessions and application consumers; a deployment may
choose one router per application for fault isolation. A router crash affects all its clients,
so choose that boundary deliberately. In-process mode still exercises routing and ownership.
Defaults: coordinator per session, worker per fly, environment worker per world. Task and
per-agent executors begin as coordinator-local libraries. An environment helper may own a
separate emulator child process and adapt its native protocol. There is no second framework
socket/lease API between those logical components.
Multiple players in one game share one environment/barrier. Independent games use independent
sessions. Linked emulators require a composite backend with link-appropriate timing.
## 2. Ownership and authority
| Owner | State and responsibility |
| --- | --- |
| Application | Composition, persistent personas/brain lineage, lifecycle policies, supported interventions, application schema/history |
| Session | Clock/epoch, port assignments, admission, task ledger, executor state, barriers and coherent recovery |
| Agent | Private membrane, RNG, rates, learning, sensory encoding, decoder and tick remainder |
| Environment | World, actual controller application, backend parser, native media and state capabilities |
| Flybus | Opaque endpoint/topic routing, delivery/call correlation, bounded queues, artifact-owner graph and GC |
| Storage client | Durable event/checkpoint writes and replay APIs; owns artifact handles during writes |
| Presentation | Application UI, display focus, clocks/buffers, media processing, audio/stream output and narrative cues |
Immutable graph data can be shared; neural mutable state cannot. Do not concurrently dispatch
the existing single-job WorkerPool through cloned handles from different brains.
Applications use declared session capabilities, not arbitrary emulator writes. Bus registration
and method privileges preserve a single controller authority for a session. Browser/audience
clients do not gain controller access by knowing a service name. Existing public rules remain.
## 3. Domain independence
The kernel knows no task or bus. The environment knows no neuron populations. A task interprets
game state and requests outcomes/recovery; its executor translates a selected decision using
read-only current game/progress context. The coordinator orders and applies these results.
The bus handles no such semantics. Presentation combines framework observations with an
application-owned schema and can change without changing simulation behavior.
Persistent AssetRefs identify installed release content. Transient Flybus ArtifactRefs identify
live bytes with ownership. Domain epoch/step identity and bus route/store incarnation are
different: the first protects simulation order, the second protects delivery/resource validity.
Never substitute game-frame number, persona identity or array position for either.
## 4. Initial scope and compatibility
First build a generic bus example (RPC + pub/sub + artifact retained beyond message lifetime),
then a synthetic two-agent session using it. One local machine, Unix sockets/in-memory parity,
immutable file-backed artifacts, fixed cadence, 1-ms LIF, direct control and exact-checkpoint
synthetic backend are sufficient. Dolphin, MaleCNS and richer effects are later integrations.
Deferred: cross-machine artifact access, durable broker queues, wildcard/queue-group routing,
dynamic native plugins, hot-join, speculative netplay rollback and pooled GPU buffers.
Keep legacy-gameboy-v1 distinct from lockstep-v1. Preserve TypeScript as oracle, existing default
versions, historical arithmetic/fingerprints and FLYSIM01 reader. New identities include sensor,
readout/executor/task/scheduler semantics. New public feed v2 is an application/presentation
gateway contract built on the same internal bus; it does not replace the bus or expose it raw.
## 5. Reuse criterion
Adding a third environment/application requires a backend, task/profile, composition and
application presentation. It must not require game-specific edits to the coordinator, router,
artifact manager, kernel, generic stores or transport. Schematized task extensions are valid;
an unchecked data blob or universal tournament schema is not a substitute for interfaces.

486
docs/design/session-framework/bus-v1.md Normal file
View file

@ -0,0 +1,486 @@
# Flybus v1: a small Rust RPC and pub/sub bus
Status: **draft 1**, 2026-09-18. Selected architecture for the new framework, not implemented
runtime code. This document supersedes the earlier proposal for direct worker sockets plus
a separate application bus and coordinator-owned buffer release protocol.
**One library, one router, one wire protocol. Small messages carry metadata and artifact
handles; large immutable bytes live in a managed local store. Ownership follows deliveries
and explicit retention.** The router moves messages and tracks generic resource ownership.
Applications own orchestration, presentation, effects and streaming. Sessions own simulation
ordering. The bus knows nothing about brains, game frames, matches or Twitch.
## 1. Scope and implementation shape
Start with a Rust/Tokio library, an embeddable router and an optional small router executable.
The same client API supports an in-memory test transport and Unix-domain stream sockets.
All participants use router semantics even when colocated; there is no separate fast-path
RPC protocol to maintain. Python helpers may implement a thin binding to the same schema;
browser integration belongs to the application's presentation gateway.
```text
Application / supervisor ─┐
Session coordinator ──────┤ managed artifact store
Agent workers ────────────┼── Flybus router ── immutable bytes
Environment backend ──────┤ handles in messages
Presentation / recording ─┤
Audience adapter ─────────┘
```
V1 supports one trusted local deployment with a shared local store. It does not require NATS,
a service mesh, durable broker queues, cross-machine artifact transfer, load-balanced stateful
workers, global transactions, dynamic plugins or a video codec. A private backend adapter may
still speak an emulator's native pipes/protocol internally; that is not a second framework bus.
Suggested crate: `flybus`, with `wire`, `router`, `client`, `artifact`, and `transport` modules.
Split crates only when useful. Keep session/game types out of this library. Use Tokio, serde
and a checked framing layer; persistent histories/checkpoints stay in application/session storage.
## 2. Client API
Illustrative Rust surface (not yet implemented):
```rust
let bus = Client::connect(config).await?;
let service = bus.register("agent.fly-a", service_config).await?;
let reply = bus.call(target, "Agent.Prepare", payload, attachments, budget).await?;
let subscription = bus.subscribe("session.demo.snapshots", subscription_config).await?;
bus.publish("session.demo.snapshots", payload, attachments).await?;
let mut writer = bus.artifacts().allocate(size, content_type).await?;
writer.write_all(&pixels)?;
let frame = writer.seal().await?; // consumes writer; immutable Artifact handle
bus.publish("world.demo.frame", metadata, [("frame", frame.clone())]).await?;
let message = subscription.next().await?;
let image = message.artifact("frame")?;
drop(message); // image still owns the delivery guard
render(image).await?; // last handle drop releases ownership
```
RPC, pub/sub and artifacts all use this client/connection. The artifact store is a data
structure/storage backend of the bus, not another messaging service. Bulk data does not
pass through the router's socket payloads or require a separate application data-transfer API.
`Artifact` is a read-only, cloneable handle. `ArtifactWriter` is unique and not cloneable;
sealing consumes its writable lifetime. Mapped slices cannot outlive their handle. Rust RAII
automates releases; other language bindings provide equivalent explicit close/context-manager
behavior. Garbage collection means reclaiming an unowned artifact, not inspecting game state.
## 3. Addressing and identities
| Type | Meaning |
| --- | --- |
| `routerId` | Fresh router/store incarnation; changes after restart |
| `clientId`, `clientIncarnation` | Configured participant identity and SDK client lifetime; reconnect creates a new incarnation |
| `connectionId` | Fresh connection; v1 does not resume its queues or delivery owners |
| `service`, `serviceIncarnation` | Named endpoint and opaque router-issued registration identity |
| `callId` | Unique RPC correlation ID for this client incarnation; not a domain operation ID |
| `topic`, `topicIncarnation`, `topicSequence` | Exact topic, declaration lifetime and acceptance order within it |
| `deliveryId` | One recipient's message delivery and artifact ownership root |
| `artifactId`, `generation` | Immutable byte object identity within a store incarnation |
| `ownerId` | A connection-owned delivery or explicit artifact hold |
Identifiers are bounded ASCII strings; scalar Id and U64 encodings match the common types
in [session RPC contracts](ipc-v1.md). Service/topic names use 1..192 characters from
`[a-z0-9._-]`, with no empty dot-separated segment. Exact names only in v1; wildcard routing
and queue groups are deferred. The router treats names as opaque addresses.
One live registration owns a service name. Duplicate registration fails; there is no implicit
round-robin balancing or replacement of a stateful agent. Registration returns the incarnation.
Callers pin it after discovery. If it changes, calls fail with `TARGET_CHANGED` rather than
silently reaching another brain/environment. `Worker.Hello` remains a domain capability RPC,
distinct from transport connection negotiation.
Service/topic access is configured per participant by the launcher/application. Presentation
subscribes to observations; it does not receive authority to invoke environment Advance.
Audience effects go through application/session admission. Naming a target is not authority
to control it. Public feed/control interfaces remain separate compatibility boundaries.
## 4. Wire envelope and framing
One connection handles both directions and every operation:
```text
u32 little-endian JSON byte length | UTF-8 JSON object
```
```ts
interface BusEnvelope {
protocol: "flybus"; major: 1; minor: 0;
id: Id; replyTo: Id | null;
kind: "command" | "reply" | "delivery" | "notice";
op: string;
body: object;
attachments: Attachment[];
}
interface ArtifactRef {
storeId: Id; artifactId: Id; generation: U64;
byteLength: U64; contentType: string;
digest: Digest | null;
}
interface Attachment {
name: Id; ref: ArtifactRef; ownerId: Id;
}
```
Maximum total JSON envelope is **65,536 bytes**. Up to 32 attachments, with unique names;
contentType is a nonempty ASCII string <=127 bytes. No pixel/base64/checkpoint bytes in JSON.
Artifact sizes are independent of envelope size. Domain schemas referencing artifacts MUST
enumerate every referenced artifact in attachments; generated client bindings enforce this.
The router validates attachment declarations/ownership, not the contents of domain payloads.
Commands have unique monotonically issued `id` serials per connection (canonical `msg-<U64>`).
Replies correlate with `replyTo`; routing notices/deliveries have router-generated IDs.
The router supplies authenticated sender/target metadata on deliveries; senders cannot forge
it by putting another participant's name in body. Domain `requestId` and callId remain distinct.
Reject duplicate JSON keys, invalid UTF-8, NaN/Infinity, unknown envelope fields, zero/oversize
frames and invalid ranges. Read length before allocating. Handle partial reads/writes and
serialize one writer per connection. There are no ancillary-FD tricks in the first file-backed
implementation. A future memory backend must retain the same client/ownership API.
### Connection negotiation
First command `bus.hello` has body `{clientId, clientIncarnation, supportedMajors}` and no
attachments. Its reply reports `{routerId, connectionId, selectedMajor, selectedMinor,
contractDigest, limits}`. The launcher provides expected client/registration privileges and
local endpoint/store configuration. Refuse incompatible majors or identity mismatch before
registration. Changes to these draft schemas change contractDigest; incompatible released
schemas require a major version bump.
The connection reader must dispatch incoming replies and requests without blocking on user
handlers. Blocking neural computation runs on dedicated workers; artifact I/O/hashing runs
outside the router's routing critical section. No mutable routing-state lock across slow I/O.
Example RPC command (the counter service is a generic test, not a built-in router feature):
```json
{
"protocol": "flybus", "major": 1, "minor": 0,
"id": "msg-7", "replyTo": null, "kind": "command", "op": "rpc.call",
"body": {
"callId": "call-2", "target": "example.counter", "expectedIncarnation": "service-a",
"method": "Counter.Increment", "payload": { "amount": 1 }
},
"attachments": []
}
```
An image-bearing publish has the same envelope shape. Its payload contains dimensions/time
and an application-defined attachment binding; its attachment entry contains ArtifactRef and
ownerId. byteLength may be `"1228800"`, while the entire message remains a small JSON object.
## 5. Operation registry
All operations use the envelope above. Replies are `{ok:true, value:object}` or
`{ok:false, error:{code, message, dispatch}}`, where dispatch is `not-dispatched`, `dispatched`
or `unknown`. Message length includes this wrapper. Transport error != domain error.
| Command | Body / reply value | Semantics |
| --- | --- | --- |
| `service.register` | `{name, maxQueued, maxInFlight}` → `{serviceIncarnation}` | Exclusive endpoint, bounded capacities |
| `service.unregister` | `{name, serviceIncarnation}` → `{removed}` | Stop new calls; queued calls fail; dispatched results follow §6 |
| `rpc.call` | `{callId, target, expectedIncarnation:null|Id, method, payload}` → `{accepted, serviceIncarnation}` | Admission acknowledgment only, eventual rpc.result follows |
| `rpc.reply` | `{callId, requestDeliveryId, outcome}` → `{routed}` | Reply only by the registered recipient; outcome is domain success/error |
| `rpc.cancel` | `{callId}` → `{state}` | Best effort; only canceled-before-dispatch establishes no invocation |
| `topic.declare` | `{name, retained:"none"|"latest"}` → `{declared, topicIncarnation}` | Compatible redeclaration allowed; conflicting settings fail |
| `topic.clear` | `{name}` → `{cleared}` | Release retained topic root; does not invalidate deliveries |
| `topic.delete` | `{name}` → `{deleted}` | Only when no subscribers; releases retained state |
| `subscribe` | `{topic, mode:"latest"|"bounded", maxQueued, maxInFlight, replayLatest}` → `{subscriptionId, topicIncarnation}` | Exact topic; no durable history |
| `unsubscribe` | `{subscriptionId}` → `{removed}` | Discard queued messages; already delivered handles remain valid |
| `publish` | `{topic, payload}` → `{topicSequence, subscribers, replaced}` | Atomic publication admission/fan-out, not consumption |
| `delivery.consumed` | `{deliveryIds:Id[]}` → `{released}` | Idempotent; processing and all local artifact uses have finished |
| `artifact.allocate` | `{byteLength, contentType}` → `{artifactId, generation, ownerId, writeLocation}` | Reserve quota and private writable staging storage |
| `artifact.seal` | `{artifactId, generation, ownerId, digest:null|Digest}` → `{ref, ownerId}` | Finish immutable publication; writeLocation no longer valid |
| `artifact.open` | `{ref, ownerId}` → `{readLocation}` | Resolve an owned sealed object for read-only mapping, never return its bytes |
| `artifact.retain` | `{ref, ownerId}` → `{ownerId}` | Create independent explicit hold while source ownership still exists |
| `artifact.release` | `{ownerIds:Id[]}` → `{released}` | Drop explicit holds/staging writers; not another client's owners |
Names/arrays/ranges follow §3/4/9. Release batches contain 1..64 IDs. No attachments are
allowed on management commands except rpc.call/rpc.reply/publish. `outcome` is a bounded
domain object; an incoming RPC result has its own attachments and delivery ownership.
`accepted`, `routed`, `removed`, `declared`, `cleared`, `deleted`, `replayLatest` are booleans;
`subscribers`, `replaced`, `released` are U64 counts. Released counts count newly released
roots, so an idempotent repeat may report zero. Queue/credit requests are integers 1..65535
and cannot exceed configured limits. Method strings are 1..128 printable ASCII characters.
call target is a service name; expectedIncarnation is its registration ID, not worker process ID.
Location grants are `{storeId, relativePath}` resolved by the client beneath the configured
local store root; absolute paths, parent traversal and symlink escapes are rejected. They
are SDK-private and do not appear in the application's ArtifactRef. Runtime paths are not
committed into application schemas or source configuration.
Deliveries:
- `rpc.request`: `{deliveryId, callId, caller, target, serviceIncarnation, method, payload}`.
- `rpc.result`: `{deliveryId, callId, responder, serviceIncarnation, outcome}`.
- `topic.message`: `{deliveryId, subscriptionId, topic, topicIncarnation, topicSequence, replaced, payload}`.
`caller`/`responder` include clientId and clientIncarnation. Delivery attachment ownerIds are
replaced by the recipient's deliveryId; source owner tokens are never delegated verbatim.
`topicSequence` and counters are U64 strings. The router assigns these IDs; the SDK exposes
typed payloads plus Artifact handles. Required bounded notices are route removal, subscription
closure and call failure. If even notice capacity is exhausted, close the connection instead
of silently losing control-plane correctness; disconnect is itself a typed client failure.
## 6. RPC behavior
callId uses `call-<U64>` with increasing serials per connected client. Keep an admission
watermark plus active-call entries; reused/retired call IDs are rejected, never executed again.
Reconnecting creates a new clientIncarnation/connection rather than reviving its old calls.
An RPC targets one registered service, not a broadcast subject. Preserve first-dispatch FIFO
per caller/service; responses may complete out of order and correlate by callId. Service
dispatchers can answer status concurrently with a long mutation, subject to their domain
state rules. The router does not implement frame barriers or numerical ordering.
Admission validates route, pinned incarnation, size, quotas and every source artifact owner.
It establishes request-delivery roots atomically before accepting. Rejection establishes no
delivery and drops any provisional roots. An accepted call is not proof that its handler ran.
Before any request bytes can reach the target, mark it dispatched; subsequent transport loss
is conservatively an unknown execution outcome.
The service publishes a reply with its own owned artifact handles. The router establishes
caller-result ownership before accepting that reply. It keeps only bounded call correlation
metadata until result delivery is consumed or the caller detaches/disconnects. It is not an
indefinite RPC result cache. A second rpc.reply for the same call is rejected, not routed twice.
Responding does not release the request's delivery guard; the handler does so when finished.
**No automatic retry or failover.** A deadline belongs to the calling client. On timeout the
client may send rpc.cancel, but cancellation after dispatch cannot undo work. Domain retries
use a fresh bus callId containing the **same domain requestId/body**, pinned to the same
service incarnation. Endpoint-level deduplication supplies safe replay; the router does not
infer it from method names. Never route a retry automatically to a restarted worker.
Queued cancellation releases its queued artifact roots and returns `cancelled-before-dispatch`.
After dispatch return `execution-unknown`; keep the recipient's delivery alive until consumed
or disconnected. A later reply to a detached call returns `routed:false`, with no caller-result
roots. The service still owns any retained result; domain state may have changed.
If a terminal result is already admitted, cancellation reports `completed`; the client drains
and consumes any result it no longer exposes to its caller. A retired/unknown correlation
reports `call-gone`. These four strings are the complete rpc.cancel state enum. None authorizes
re-execution, and cancelling a future does not abandon incoming delivery ownership.
To replay an artifact-bearing response safely, endpoint code caches **Artifact handles plus
payload**, not bare references. That cache owns explicit holds or delivery guards until domain
acknowledgment/eviction. Re-delivery gets new delivery IDs pointing to the same immutable bytes.
Once the domain cache expires it returns RESULT_EXPIRED; it cannot regenerate the operation
merely because the transport correlation entry was removed.
## 7. Pub/sub semantics
Topic declaration does not teach the router what a topic means. An application can publish
session observations, presentation cues, dataset jobs or unrelated typed events through the
same API. There are no hardcoded frame/brain topics inside the router.
- `latest`: one queued value per subscription, replacing only an **undelivered** value.
Replacing it releases that queue entry's artifact roots. Already delivered/in-use messages
are never reclaimed early. maxQueued is exactly 1 in this mode.
- `bounded`: FIFO queue, no coalescing or silent loss. When required queue/owner capacity
is unavailable, reject the publication with BACKPRESSURE before admitting any deliveries.
- maxInFlight credits are returned only by delivery.consumed, not socket write completion.
A latest subscriber with all credits in use still has one replaceable queued value.
Take an atomic subscriber/retention snapshot at admission. Validate and reserve all required
queue entries and artifact-owner budgets before accepting. For a bounded subscriber overflow,
reject the **whole publish**; no partial fan-out or retained-latest update. On acceptance,
assign one topicSequence and create roots for every delivery and optional retained value.
Different topics have no total ordering. Multiple publishers on one topic follow router
acceptance order, which is not automatically a deterministic application event order.
Publication reply counts accepted subscriptions/replaced queue entries, not consumers that
processed data. `replaced` on a delivery reports how many undelivered messages were coalesced
since that subscription's preceding delivery. Sequence gaps can also arise from joining late;
they are not evidence of a simulation step being skipped.
Optional `retained:latest` holds one last message and its artifacts independent of subscribers.
New subscriptions with replayLatest enqueue it before subsequent accepted publications;
bounded mode preserves that order, while latest mode may coalesce it before delivery under
the ordinary latest rule. Replay uses the original topicSequence, a fresh deliveryId and
explicit roots. Without retention,
zero-subscriber publication retains no artifact ownership after admission. Clearing a topic
releases only its retained root, not active consumers. Topic count and retained bytes are capped.
There is no durable replay, automatic redelivery, or exactly-once processing claim in v1.
Deleting/redeclaring a topic creates a fresh topicIncarnation; a reset sequence cannot be
mistaken for continuation of the deleted topic. Old subscription deliveries retain their
original incarnation and ownership until consumed.
If an application requires history, its recorder persists events and exposes a normal RPC
for recovery/query. Bus admission, message consumption and durable storage acknowledgment
are three different events. The bus must not conflate them.
## 8. Artifact lifecycle and garbage collection
### 8.1 Immutable object lifecycle
```text
ALLOCATED / WRITING → SEALED → referenced by owners → last owner drops → COLLECTED
└─ writer abandoned/disconnected ──────────────────────→ COLLECTED
```
`ArtifactRef` is an identity, not an address, filename or authority to read. Opening it requires
a current ownership root belonging to that connection. storeId is the router/store incarnation;
old handles fail after restart. V1 does not reuse an artifact ID/inode; generation is 1 and
remains in the contract for future pool implementations. Content hashes are optional for live
frames and mandatory for checkpoint/durable-content handoff, as specified by domain contracts.
The first storage backend uses runtime-configured local files, optionally on tmpfs. Producer
writes staging storage outside the message stream. Seal closes writable mappings/handles in
the SDK, checks length and any requested digest, then finishes an immutable store-owned
object before acknowledging. A correctness-first implementation may copy into a fresh sealed
inode; account for both allocations during sealing. No per-frame fsync for transient media.
Consumers resolve a store-issued readLocation through artifact.open and map/read it read-only.
Locations are private grants; they are not placed in application bodies or public browser feeds.
All filesystem access stays behind the client Artifact API. Do not inline binary data or create
a second bulk-transfer server just because it is stored outside the socket.
### 8.2 What owns an artifact?
The store tracks an ownership graph, not only a naive refcount incremented by every packet:
| Root | Lifetime |
| --- | --- |
| Producer explicit hold / active writer | Until last local handle releases, seal transfers its unique writer, or connection is lost |
| Accepted queued delivery | Until replaced/cancelled or transferred into recipient delivery ownership |
| In-flight delivery | Until message processing AND all extracted artifact uses finish |
| Retained latest topic | Until replaced, cleared, deleted, or router stops |
| Explicit retained hold | Until release; used for caches, rendering, checkpoint writes and forwarding |
Admission creates destination roots before the sender may relinquish source roots. A forward
or reply uses a live handle/owner; the client holds it until admission succeeds or definitively
fails. A timeout must not drop a source guard while an unsent operation might still be admitted.
The client cancels/discards the unsent frame or retains the guard until the transport outcome
is known; connection teardown ends that ambiguity for the old connection.
Every envelope must list its complete artifact set. Duplicate references in one delivery are
counted once. A retained topic and several consumers can reference the same physical bytes.
The router performs metadata updates only; it does not copy image bytes for fan-out.
### 8.3 Consumed means no remaining use
The incoming message owns a shared **DeliveryGuard**. Extracting an Artifact clones the guard;
dropping the message alone does not consume the delivery while a renderer/encoder still uses
its artifact. Local handle clones do not each require a bus round trip. Dropping the last
guard queues delivery.consumed through a bounded control lane.
V1 deliberately owns at delivery granularity: keeping one artifact from a message may keep
the other attachments alive too. For independent long-lived retention, artifact.retain creates
a specific explicit hold before the original guard is dropped. Domain RPC caches must use
that hold when they outlive message processing/credits. An acknowledgment of domain success
does not implicitly drop either the incoming guard or cached outgoing holds.
If an allocation/seal grant arrives after its caller abandoned the future, the SDK reactor
still processes and releases that grant. It must not leak an owner the application never saw.
An in-progress seal/copy has a bounded internal I/O hold; on producer disconnect it either
finishes cleanup or aborts safely, never publishes an ownerless object into a new connection.
Async task cancellation/drop of a response future is not necessarily consumption: the client
must own queued results until surfaced, explicitly discarded, or disconnected. Receivers must
await completion of asynchronous CPU/GPU use before releasing its guard. A pointer extracted
from a mapping cannot outlive its Artifact; FFI wrappers must enforce this lifetime explicitly.
Release commands are batched, idempotent and scoped to the connection that owns the IDs.
Delivery/hold IDs use monotonic per-connection serials. Keep issued watermarks plus active
owner maps; releasing an already retired ID is a no-op, a never-issued/future ID is an error.
This avoids a tombstone per frame forever. Control-lane exhaustion closes the connection
instead of silently losing releases and leaking an unbounded ownership graph.
### 8.4 Crash, disconnect and safe physical reclamation
On disconnect, unregister services/subscriptions; cancel queued deliveries and release that
connection's active writers/explicit/delivery roots. Retained topic roots remain router-owned.
Late replies and releases cannot attach to a new connection or service incarnation.
GC removes the registry entry and unlinks/closes the sealed object after its final root is
gone. Existing immutable file mappings may remain valid until the OS closes the last mapping;
do not overwrite their inode or reuse their bytes. Logical reclamation is not proof that a
disconnected process released its physical pages. The supervisor handles stuck processes;
resource measurements include OS mappings and client memory, not just registry totals.
No TTL may reclaim a live owned artifact. Limits may disconnect a consumer, triggering the
explicit cleanup above, but cannot overwrite memory under a renderer. Future pooled shared
memory must prove equivalent lifetime/generation safety before replacing immutable files.
Router restart creates a new routerId/storeId, loses routes/queues/retention and invalidates
all old handles. Live sessions fail their current epoch and use coherent recovery. Persistent
artifacts come from the application's durable store and are re-imported as new bus objects.
Orphan files from a stopped router are cleaned without treating them as durable checkpoints.
## 9. Bounds, scheduling and failure reporting
Configure limits explicitly; these defaults are a prototype starting point, not capacity data:
| Resource | Default |
| --- | ---: |
| Connected clients / services / topics | 64 / 256 / 512 |
| Subscriptions per client / total | 128 / 1024 |
| Control envelope | 64 KiB |
| Active calls per client | 64 |
| Service queued / in-flight calls | 16 / 16; worker dispatcher further limits mutations |
| Latest subscription queued / in-flight deliveries | 1 / 2 |
| Bounded subscription queued / in-flight deliveries | 64 / 16 |
| Active owners per client | 256 |
| Total artifact storage / per object | 512 MiB / 128 MiB |
| Per-client ordinary queued envelope bytes | 1 MiB |
| Reserved management/reply lane | 128 frames and 1 MiB per client |
Reserve an owner allowance for lifecycle/results separately from ordinary telemetry; memory
quotas account for staging/seal copies, queued deliveries and caches. Ownership metadata is
bounded even if many roots share one artifact. Disk-full, allocation failure or hash mismatch
returns a typed artifact error and cleans provisional storage/roots.
The router fairly services clients. Replies, release, cancellation and route-health control
cannot be starved by telemetry. Preserve FIFO for calls to a target despite lane scheduling;
classification is an explicit generic envelope operation/policy, not a topic-name heuristic.
No indefinite wait inside the router on subscriber readiness or artifact I/O. Admission is
bounded; rejected callers choose their own retry/fail/pause policy.
Transport errors include `INVALID_ENVELOPE`, `VERSION_MISMATCH`, `NOT_AUTHORIZED`,
`NO_SERVICE`, `TARGET_CHANGED`, `BACKPRESSURE`, `CALL_GONE`, `ARTIFACT_UNSEALED`,
`ARTIFACT_GONE`, `OWNER_INVALID`, `QUOTA_EXCEEDED`, `STORE_FAILURE`, `ROUTER_LOST`.
Before admission use dispatch:not-dispatched. Once dispatch might have occurred, report
unknown/dispatched conservatively; a caller-side timeout must not imply no mutation.
Bounded event subscriptions can reject publication; latest spectator subscriptions cannot
hold a required session transaction indefinitely. Per-client credit/owner budgets and
supervision enforce that distinction. Sustained pinned-artifact quota exhaustion is surfaced
as resource pressure, not solved by freeing live data. Session/app policies choose whether to
disconnect an observer, pause, or fail; the router does not know which outcome is appropriate.
## 10. Native-frame bandwidth check
At 640×480 RGBA, a frame is 1,228,800 bytes. At 60 fps, production is **73.728 MB/s**
(decimal); at 30 fps, 36.864 MB/s. These are planning dimensions; a backend advertises its
actual output. Two agent workers plus one presentation consumer can read the same immutable
frame object. Bus messages contain only references; there is no 3× byte fan-out through the
router. Readers still incur memory traffic/page faults, and renderer readback/seal copying
remain real costs. This is not a claim of zero-copy GPU capture or measured host performance.
The environment emits native game frames/audio. The application/presentation layer owns
resizing, overlays, compositing, browser delivery, codec choice and stream output. Do not put
1080p rendering, Twitch publishing or game-specific sampling logic in Flybus. A presentation
pipeline may itself exchange large artifacts through this same bus if useful.
## 11. Acceptance tests and implementation sequence
1. **Wire/router:** schema/framing, Hello, exclusive routes, pinned incarnations, request/reply,
disconnect and bounds. In-memory transport must pass the same tests as Unix sockets.
2. **Pub/sub:** exact topics, FIFO/bounded rejection, latest coalescing, retained replay/clear,
atomic fan-out and fair control/reply delivery under a saturated subscriber.
3. **Artifacts:** allocate/seal/read; publication before seal fails; fan-out owns one physical
object; last consumer releases; retaining an extracted frame after message drop works.
4. **Faults:** sender drops after admission; consumer dies mid-read; reply is lost; queued frame
is replaced; subscription closes with in-use deliveries; router restarts; old release arrives.
No double-free, use-after-reuse, unbounded tombstones or hidden operation replay.
5. **RPC cache:** endpoint retains an artifact-bearing result, original caller consumes it,
and a domain retry still returns valid bytes. Eviction drops the last cache hold correctly.
6. **Integration:** two parallel fake agents, complete-batch environment RPC, committed snapshot
publication and a deliberately slow presentation consumer over the same router.
7. **Performance:** 640×480×60 artifact production with three consumers, one delayed; measure
p50/p95/p99 RPC latency, router CPU, copy/readback cost separately, RSS, store live/peak bytes,
outstanding roots, collection lag and queue lengths. Compare one/two/four agent schedules.
The first executable example should show a counter RPC, a pub/sub observer, and a frame
artifact held past message consumption in one small Rust program. No game or browser required.
Distributed simulation ordering remains the [session contract's](step-v1.md) responsibility.

270
docs/design/session-framework/implementation.md Normal file
View file

@ -0,0 +1,270 @@
# Future-agent implementation guide
Status: **draft 2**, paired with the [contract index](README.md). This is the execution guide
for the new process architecture; it refines the broader
[MaleCNS/modular backlog](../malecns-modular-implementation.md), not a request to implement
every future feature in one branch.
## 1. Start here
Before coding:
1. Read repository instructions, the architecture tour, public feed/control contracts, and all
documents in this directory. Reconcile any newer main/feature-branch changes with this baseline.
2. Record what the next slice will change, its compatibility surface and expected tests.
3. Use a dedicated branch/worktree. Follow the repository's coordinator/build/review roles.
4. Preserve the TypeScript oracle, default numerical version strings and legacy deployment.
5. Resolve contract contradictions before implementing; do not fill gaps with an implicit
asynchronous best-effort policy or a public controller API.
The immediate target is **one small Rust Flybus example with RPC, pub/sub and a frame artifact
held beyond the message object's lifetime**. Then build the synthetic two-agent session over
that same router using in-memory and Unix-socket transports. No separate worker transport,
NATS service, coordinator-owned lease manager or raw-frame socket channel is to be implemented.
## 2. Proposed implementation map
Begin under the existing Rust workspace; extract physical package locations separately.
The following are proposed names, not files that exist today:
```text
services/flysim/crates/
fly-session-types/ scopes, contracts, schema validation, canonical digests
flybus/ generic router/client/wire/artifact store; embedded or standalone
fly-session-rpc/ domain schemas/deduplication over flybus; NOT another transport
fly-session/ phase machine, coordinator, admission, task/executor traits
fly-session-worker/ dispatch shell, status/shutdown, agent/environment adapters
fly-session-store/ participant captures, manifest commit, group recovery
services/flysim/crates/flysim/
legacy/ compatibility composition (extract without changing behavior)
composition/ new config/registries and worker launching
packages/feed/ existing public v1, later public v2 schemas/fixtures
packages/brain/ reference model/readout and new contract-relevant golden generators
```
Do not create empty crates to satisfy this tree. In the first slice, types/transport/coordinator
may be modules in one small crate; split when dependencies and consumers justify it. Keep
Melee parser/codecs and Game Boy FFI out of the session-types crate. Worker executables can
be subcommands of one binary initially; process boundaries do not require separate repos.
## 3. Ordered slices
### CONTRACT-01 — Executable schemas and trace format
**Inputs:** [bus spec](bus-v1.md), session RPC, step, worker, state/media and publishing documents.
**Implement:** Flybus wire schema separately from session domain schemas; common scalar types,
closed enums, method payload validation and schemas;
canonical digests; fixture loaders in Rust/TypeScript. Specify seed derivation and exact
checkpoint envelope bytes before their respective real-agent/store slices. Create the trace
format from step-v1 section 8, distinguishing behavior fields from operational IDs/time.
**Acceptance:**
- JSON round trips across both languages; U64/float boundaries reject correctly.
- Duplicate keys, invalid UTF-8, envelopes over 64 KiB and unknown required fields fail.
- Distinguish bus callId, domain requestId, artifact identity and delivery/hold owner tokens.
- Fixtures include rational zero/reduced form, overflow, duplicate ports and analog limits.
- Contract digest is generated from a documented canonical schema set, not source formatting.
**Stop:** do not wire a real worker until payload ambiguities and retry identity rules agree.
### BUS-01 — Router and RPC, in-memory and Unix socket parity
**Depends on:** CONTRACT-01.
**Implement:** one flybus crate with bounded framing, bus.hello, exclusive service registration,
incarnation-pinned RPC/reply/cancel, typed route/admission errors and independent read/write
dispatch. Begin with artifact-free calls; do not claim full bus conformance until BUS-03.
**Acceptance:**
- Partial frames/writes, disconnect after request, lost result and retransmission fixtures.
- No automatic retry/failover; timeout/cancel-after-dispatch reports uncertain execution.
- Service incarnation replacement is visible. Request/reply correlation survives out-of-order
replies; status RPC can respond while another handler is delayed. Saturation is bounded.
- Both transports produce equivalent behavior traces for the same scenario.
### BUS-02 — Pub/sub, retention and backpressure
**Depends on:** BUS-01.
**Implement:** exact topics, subscribe/unsubscribe, bounded FIFO and latest policies, optional
retained latest/clear, per-recipient delivery IDs/consumption credits and fair control lanes.
**Acceptance:** overflow rejects a bounded publication before partial fan-out; latest replaces
only queued messages; delivery consumption returns credits; unsubscribe preserves already-
delivered ownership; retained replay is ordered; stalled observers cannot starve RPC replies.
Durable event history is a storage client, not a second broker built into Flybus.
### BUS-03 — Artifact-backed messages and automatic lifetimes
**Depends on:** BUS-02.
**Implement:** immutable file-backed store, allocate/seal/open, bus attachment validation,
producer/queue/delivery/retention owners, RAII DeliveryGuard and explicit cache holds. Root
creation/admission is atomic. Start with ordinary local files/tmpfs; no pooled slot reuse yet.
**Acceptance:** last owner collects; extracted handle survives message drop; forward-before-
release is safe; lost replies/cache replay remain valid; disconnect releases logical ownership
without mutating still-mapped bytes; retained latest and queue replacement release correct roots.
Measure 640×480 RGBA×60 with three readers: one stored image, no raw pixels in router messages,
bounded CPU/RSS/owners/queues. Record reader/copy costs rather than claiming zero-copy capture.
### SESSION-01 — Synthetic sequential transaction
**Depends on:** BUS-03.
**Implement:** small fake agent workers, one counter/arena environment, identity executors
and a deterministic task. Follow Prepare→Advance→Evaluate→Commit exactly. Use explicit seeds
and rational clock accumulation; the fake model must expose a mutation counter for tests.
Implement domain request deduplication/result caches over bus calls; retaining result artifacts
is an endpoint responsibility. Domain Acknowledge differs from bus delivery.consumed.
**Acceptance:**
- One world advance per complete batch; every agent Prepared before Advance.
- Task evaluates once; every agent commits before next Prepare or committed publication.
- A synthetic 60-Hz/1-ms profile produces 16,17,17 ticks and remainder zero after three steps.
- Pause mid-step completes the step and pauses at its committed boundary.
- Bootstrap/warm-up cannot advance the environment or produce gameplay rewards.
- Domain retries use a new callId with the original requestId; they never repeat ticks/reward.
### SESSION-02 — Parallel processes and fault behavior
**Depends on:** SESSION-01.
**Implement:** one agent process per fly and one environment process under the coordinator;
compare with in-process and dedicated-thread variants. Enforce total thread budgets and
configured agent/port identities. Failure stops the epoch rather than neutralizing a player.
**Acceptance:** sequential, reversed order and parallel completion produce equivalent traces;
delayed one-agent result holds the world; worker/helper death has a bounded diagnosed outcome;
an uncertain Advance never creates a second batch; partial Commit never permits next-step play.
### MEDIA-01 — Native observation schemas and presentation handoff
**Depends on:** SESSION-02.
**Implement:** view/sample descriptors and producing-step validation on top of bus ArtifactRef,
not a second buffer system. Environment outputs native media; sensor transforms remain agent
profiles, while presentation owns viewer resizing/composition/audio/streaming.
**Acceptance:** bad strides/lengths/producer times fail; shared image reaches both agents through
owned attachments; spectators use latest subscriptions and cannot corrupt sensory state;
delayed rendering retains its handle. Distinguish AssetRef from transient ArtifactRef.
### AGENT-01 — Existing neural core worker
**Depends on:** SESSION-02, MEDIA-01 and the profile/identity foundation in the broader backlog.
**Implement:** adapter over existing LIF, plasticity, retina and fixed readout primitives;
reference-first composition/goldens; independently seeded agent state and shared immutable data.
Avoid using the old whole-frame `tick` wrapper if it changes the specified phase ordering.
**Acceptance:** per-agent state agrees with the reference across Prepare/Commit, stimulation,
zero/nonzero reward, warm-up and pauses. Two agents cannot share gains/RNG/holds; swapping
dispatch order and varying worker count preserves results. Keep 64-role limits explicit.
### ENV-01 — Game Boy compatibility environment
**Depends on:** AGENT-01 and environment/task extraction in the broader backlog.
**Implement:** binjgb environment, task-local memory inspector and identity/existing action
adapter. Keep `legacy-gameboy-v1` separately routed with exact old ordering/hash semantics.
**Acceptance:** legacy fixtures/goldens/restore outcomes unchanged; the new composition has
its own identity and public adapter selection. No console-specific state enters generic
session types. ROM-backed checks are optional explicit jobs, not required downloads.
### STATE-01 — Coherent all-participant checkpoint/recovery
**Depends on:** SESSION-02, MEDIA-01; validate with fake agents first, then AGENT-01/ENV-01.
**Implement:** exact new envelope schema, compatibility manifest, Capture/StageRestore/
ActivateRestore, bounded writer, durable commit acknowledgments and fresh-epoch fencing.
Keep old `FLYSIM01` reader separate. Payloads and coordinator state must refer to one boundary.
**Acceptance:** uninterrupted versus resumed synthetic/real-agent traces match after accounting
for new epoch metadata; corrupt any participant and installation fails as a group; lost save
reply doesn't advance durable metadata; failure during activation cannot resume half a world.
Checkpoint queue stress remains bounded; verify old media/parser data cannot cross recovery.
### PUBLISH-01 — Committed snapshots and observer isolation
**Depends on:** SESSION-02, MEDIA-01; public v2 contract work is a separate prerequisite to
publishing a supported multi-agent browser feed.
**Implement:** internal [publication boundary](publishing-v1.md) over the SAME bus, latest-value
observations, application-owned state/cues, bounded events, descriptor repair/query and a fake
multi-agent consumer. Presentation gateway owns browser delivery; no generic show/tournament
service or codec is added to the router. Then implement
the approved public v2 wire schemas/fixtures and stage adapters together.
**Acceptance:** browser disconnect/backpressure never advances/stalls the world; future agent
state is not mixed with old media; descriptor/index mismatch is visible; committed actions
are labeled as the transition that just ended. PNG/browser review gates apply to actual UI.
### DOLPHIN-01 — Substitute the backend, not the coordinator
**Depends on:** measured MELEE-01/02 spikes from the [Melee audit](../melee-framework-audit.md),
SESSION-02, MEDIA-01 and declared recovery support.
**Implement:** bus-connected helper over pinned Dolphin/libmelee or the chosen narrow hook,
complete port batch adapter, frame-identified sensory output and task-local Melee inspection.
Keep actual rendering/input/save semantics behind the same Environment API.
**Acceptance:** all generic backend conformance tests plus delayed-port/flush ordering,
game-frame reset, parser recovery, pixel latency and neutral/release tests. If exact snapshot
is unsupported, advertise episode-restart and use only the matching prototype policy.
## 4. Failure injection checklist
Tests must deliberately inject these cases; success-path demos are insufficient:
| Injection | Required invariant |
| --- | --- |
| Duplicate Prepare after lost reply | No extra ticks, RNG draws, stimulation or decode |
| Same batch with altered controls | Conflict, never a second world mutation |
| Lost Advance result after world step | Resolve same operation or fail epoch |
| One Commit fails after another succeeds | No next world step; coherent recovery only |
| Old worker replies after restore | Stale epoch/incarnation rejected |
| Message drops while extracted image is rendering | DeliveryGuard keeps bytes alive through last use |
| First agent releases a shared image early | Artifact remains until every other owner finishes |
| Cached RPC artifact is consumed by its first caller | Domain cache still owns it for retry |
| Latest queued frame is replaced | Only that queue root drops; in-use images remain valid |
| Router restarts during a world advance | Old handles/routes invalid; epoch fails and restores coherently |
| Viewer holds output indefinitely | Only spectator data is dropped/disconnected |
| Backend waits for input while capture is requested | No deadlock; capture only at valid quiescent boundary |
| Capture writer stalls | Finite queue/memory, honest durable status |
| StageRestore validates three participants, fourth fails | Nothing is resumed |
| ActivateRestore fails halfway | Group remains fenced, no new gameplay |
| Old epoch audio arrives after reset | Discontinuity handling; no stale playback as current |
## 5. Verification and handoff
Before each implementation merge, run the repository's required `npm test`,
`npm run typecheck`, `cargo test --workspace` from the Rust workspace, and
`infra/tests/lint.sh`. Run affected browser/PNG gates for presentation changes. Keep the
existing committed FAFB real-data goldens mandatory; larger new datasets and game-backed
jobs report explicit optional skips.
Performance checks report total physical-core allocation, one/two/four agents, within-agent
worker count, router/RPC latency, artifact production/read/copy time, critical-path percentiles,
memory peaks, owner/GC statistics and
bounded queue behavior. No host capacity claim follows from a local synthetic timing test.
Deployment-host work is separately authorized/claimed/serialized under repository rules.
Every completed slice leaves:
1. The implemented contract/schema revision and compatibility decisions.
2. A minimal runnable synthetic example and exact test commands/results.
3. Behavior traces demonstrating its acceptance criteria.
4. Known unsupported capabilities and remaining measured questions.
5. Updated planning status, with no claims that a stub provides real emulator semantics.
Do not start by moving every directory, adding a service mesh, or rewriting the model. The
first useful deliverable is the small artifact-backed bus example, followed by the synthetic
distributed step transaction using it. No one-off communication stack per component.

207
docs/design/session-framework/ipc-v1.md Normal file
View file

@ -0,0 +1,207 @@
# Session RPC contracts over Flybus
Status: **draft 2**, 2026-09-18. The filename is retained for existing links. This document
now defines **domain contracts carried by [Flybus v1](bus-v1.md)**. It no longer defines a
separate socket protocol, direct worker connections, or coordinator-owned buffer service.
The [architecture index](README.md) states scope and precedence. Public feed/control v1 stay
unchanged; this is the new internal session path.
## 1. One transport, domain-specific meaning
Every session/worker RPC is a Flybus call to a named, incarnation-pinned service. Pub/sub,
application supervision and artifact bookkeeping use the same bus. The router moves messages;
the receiver validates its method payload and the [session step machine](step-v1.md).
The bus owns framing, connection identity, route registration, bounded delivery and artifact
ownership. This document owns Scope, model-related scalar types, worker capability negotiation,
domain operation deduplication and errors. Domain request identity is independent of the bus
callId: a safe retry has a new transport callId but the original domain requestId/body.
## 2. Common domain types
```ts
type Id = string; // ^[a-z0-9][a-z0-9._-]{0,63}$
type U64 = string; // "0" or [1-9][0-9]*; <= 18446744073709551615
type Digest = string; // 64 lowercase hexadecimal digits (SHA-256)
interface Scope { sessionId: Id; epoch: Id; step: U64 }
interface RationalNs { numerator: U64; denominator: U64 }
interface SchemaRef { id: Id; version: number; digest: Digest }
interface TypedValue { schema: SchemaRef; value: object }
interface SessionRpcRequest { requestId: Id; scope: Scope | null; params: object }
```
All fields are required unless marked `?`. Schema version is integer 1..65535. Fractions
are reduced, denominators positive, durations positive; zero is encoded 0/1. Arithmetic is
checked. JSON numbers representing rates/rewards/controls are finite. Counters/clocks use
decimal strings. Task/profile schemas bound collections and numeric ranges before mutation.
First session composition limit: 4 agents, 4 ports and 64 rate roles per agent; these are
session/model limits, not limits on the number of application personas or generic bus clients.
Each TypedValue has a canonical JSON size limit of **32 KiB**, while the complete envelope
must still fit Flybus's 64-KiB maximum. Large typed state goes in a listed Artifact attachment
under an explicit schema, not an oversized inline object. Changing the old draft's 1-MiB
worker envelope to Flybus must not silently truncate a payload.
## 3. Request/reply mapping
Illustrative client call:
```text
bus.call(
target = {service: "agent.fly-a", expectedIncarnation: pinnedRegistration},
method = "Agent.Prepare",
payload = {requestId: "req-41", scope: {sessionId, epoch, step: "41"}, params},
attachments = ownedArtifactHandles
)
```
Flybus's eventual rpc.result `outcome` is one of:
```ts
interface SessionRpcSuccess {
type: "result"; requestId: Id; workerId: Id; incarnationId: Id;
scope: Scope | null; result: object;
}
interface SessionRpcFailure {
type: "error"; requestId: Id; workerId: Id; incarnationId: Id;
scope: Scope | null;
error: { code: ErrorCode; message: string; mutation: "none" | "applied" | "unknown" };
}
```
Replies echo the original scope. The receiver identity and bus service incarnation must
match the negotiated worker. Bus route/admission failure is not a SessionRpcFailure produced
by the handler. A bus admission acknowledgment is not an Agent.Prepare/Environment.Advance
completion. Only a matching terminal domain reply resolves a simulation phase.
ArtifactRefs inside request/result payloads must be declared in bus attachments and backed by
live owned handles. Domain canonical-body digests include the references but exclude changing
bus callIds, deliveryIds and owner tokens. A cached result owns Artifact handles independently
of the first delivery; it is not a JSON object holding unowned pointers.
## 4. Worker negotiation and status
After bus connection/registration, call Worker.Hello (`scope:null`):
```ts
interface HelloParams {
sessionId: Id; expectedWorkerId: Id;
role: "agent" | "environment" | "coordinator";
supportedMajors: number[];
}
interface HelloResult {
selectedMajor: 1; selectedMinor: 0;
workerId: Id; incarnationId: Id; role: "agent" | "environment" | "coordinator";
buildDigest: Digest; contractDigest: Digest;
capabilities: Id[];
limits: { maxAgents: number; maxPorts: number };
}
```
The bus supplies caller identity; do not accept a forged caller in params. Bind a worker's
session authority to the expected coordinator identity/incarnation during negotiation and
initialization. Wrong worker/role, no common major or missing required capability refuses
the composition. Required capabilities are agent-step-v1 and world-step-v1 for their roles;
checkpoint-v1 and pixel-observation-v1 are conditional. Artifact transport capability is
negotiated once by Flybus, not as another worker memory API.
Worker.Status has params `{}` and the caller's last known scope (null before initialization):
```ts
interface StatusResult {
state: "uninitialized" | "ready" | "preparing" | "prepared" | "advancing"
| "committing" | "capturing" | "staged-restore" | "restoring"
| "failed" | "stopping";
currentScope: Scope | null;
activeRequestId: Id | null; lastCompletedRequestId: Id | null;
lastBatchId: Id | null; progressCounter: U64;
}
```
ProgressCounter advances on computational/phase progress, not on status queries. Dispatch
Status through the same service without waiting for a long numerical operation. One mutation
executes at a time; at most one may be pending, and normal stepping pipelines neither. The
router's larger RPC capacity is not permission to overlap worker mutations. Never hold a
simulation lock while waiting for network I/O, artifact resolution or release bookkeeping.
## 5. Domain idempotency and retention
`requestId` is `req-` plus a canonical U64 serial, increasing for newly issued operations
per caller/worker pair. Retries reuse it unchanged even though bus callId changes. Flybus
preserves first-dispatch order per caller/service; worker handlers maintain that request
admission order while allowing read-only status alongside compute.
The operation key for step mutations is `(sessionId, epoch, step, method, workerId)`.
There is at most one Prepare, Commit or Advance for that key.
- Same key/request/body returns its cached reply, with fresh bus delivery ownership over
retained artifacts. It never repeats ticks, stimulation, controller execution or reward.
- Changed ID/body for an existing key is CONFLICT. Canonical comparison uses RFC 8785 over
method, scope and validated params. A rejected duplicate does not undo the earlier result.
- Check retained request identity before phase checks or artifact dereferencing. A duplicate
may arrive after the original input delivery was consumed; it needs only the cached result.
- Keep current/immediately previous step result records. Eviction never enables reexecution:
highest-issued request serial and step watermarks reject expired retries/old steps.
- Original expired serial → RESULT_EXPIRED; fresh serial naming an old step → STALE_STEP.
An exact duplicate arriving while execution is active receives the terminal domain error
IN_PROGRESS for that **bus call**. The original bus call still completes normally. Retry or
query Status later; no second mutation is started. This avoids multiple terminal replies to
one bus call and does not mistake IN_PROGRESS for the original operation's failure.
Lifecycle/capture replies are retained until Worker.Acknowledge:
`params:{requestIds:Id[]}` (1..16), result `{acknowledged:Id[]}`. It drops domain cache handles,
not another consumer's bus delivery. Already released/unknown IDs are ignored. Serial
watermarks reject reuse after acknowledgment without an unbounded tombstone list.
Bound unacknowledged lifecycle replies at 16, then BUSY before application. Status and
Acknowledge use a cache of their last 16 replies; current/previous step records have their
separate finite retention. Caches containing big artifacts consume bus owner/byte budgets;
configure capture limits consistently. Never evict a promised replay artifact but keep a
successful pointer-only reply. An intentionally expired result returns RESULT_EXPIRED.
## 6. Timeout and failure handling
Timeouts are measured on the caller's monotonic clock. Prototype defaults: probe after two
seconds without reply, fail after ten seconds without progress; long boot/capture have separate
budgets. These are failure-detection values, not a gameplay latency goal.
After an uncertain call:
1. Stop further world-step dispatch.
2. If the same bus/service/worker incarnation still exists, query Status or issue a fresh
bus call with the original domain requestId/body and retained input attachments.
3. Resolve only a matching terminal result. Never repeat an Advance with a new domain ID.
4. If routes/ownership were lost, incarnation changed or retained result expired, fail the
epoch and restore/reset the group.
Endpoint crash or bus restart is not covered by in-memory deduplication. Router/store restart
invalidates all transient artifacts and routes. Worker disconnect also invalidates its bus
owners and registration; v1 does not silently reattach that worker to an active epoch.
Recover coherently even if an OS process survived with some numerical state in memory.
## 7. Domain errors
| Code | Meaning |
| --- | --- |
| INVALID_ARGUMENT | Invalid schema/range, before mutation |
| UNSUPPORTED | Missing method/capability |
| IDENTITY_MISMATCH | Wrong session/profile/port/build/asset identity |
| STALE_EPOCH / STALE_STEP / FUTURE_STEP | Timeline/order mismatch |
| INVALID_PHASE | Wrong worker phase |
| CONFLICT | Existing logical operation with changed ID/body |
| IN_PROGRESS | Original operation still executing; duplicate bus call did not start work |
| BUSY | Domain capacity unavailable before admission |
| BUFFER_INVALID | Missing/unowned/mismatched artifact or invalid media shape |
| RESULT_EXPIRED | Safe replay is no longer available; never recompute to replace it |
| INCOMPATIBLE_STATE | Restore validation failed before activation |
| BACKEND_FAILURE / INTERNAL | Runtime fault, with explicit mutation certainty |
Messages are <=512 code points and exclude raw game memory/credentials. Errors after partial
mutation use unknown unless completion is established. No error authorizes skipping a fly,
pressing fallback controls, or continuing a partially committed match.
Worker.Shutdown, params `{reason:Id}`, returns `{stopping:true}` if responsive and terminates
the worker after replying. It does not imply saved state. Only configured supervisors may
invoke it; workers have no authority to shut down the coordinator. Shutdown/release notifications
travel on the same bus; there is no reverse lease socket or Buffer.Release/Reclaim RPC.

184
docs/design/session-framework/publishing-v1.md Normal file
View file

@ -0,0 +1,184 @@
# Application, presentation and audience contracts
Status: **draft 2**, 2026-09-18. All internal communication uses [Flybus](bus-v1.md). This
document defines ownership and logical data contracts, not the public feed v2 byte format.
Existing public v1 contracts remain unchanged for the legacy application.
## 1. Applications orchestrate; presentation owns the show
Fly Plays Pokémon is an application assembled from simulation and presentation components.
Its supervisory code and interface share an application-owned, versioned state/event schema.
A Melee competition or ecosystem can choose another schema. Director, tournament, bracket,
cast of 32 personas and story segments are examples, not mandatory framework services/types.
The framework supplies sessions, agents, backend/task interfaces, capability descriptions,
native observations and generic UI/client primitives. The application chooses lifecycle,
profiles, game-aware macros/recovery, identities/history, interventions and narrative behavior.
Frame-by-frame scheduling remains inside the session; the application need not RPC each tick.
Presentation owns selection/layout, resizing, overlays, compositing, audio mixing, encoding,
browser-facing delivery, recording choices and stream output. Native 480p game data is a
perfectly valid framework output. Neither the router nor generic session assumes Twitch,
1920×1080, particular colors, specific React components or an automatic tournament dashboard.
## 2. One bus, complementary data sources
```text
Session ── committed observations/events + artifact handles ──┐
├─ Flybus ── presentation application
Application ── own state/events/presentation cues ────────────┘
Presentation gateway ── application-defined browser delivery ── frontend
```
Example addresses (chosen by composition, not recognized by router code):
| Address | Pattern / purpose |
| --- | --- |
| `session.demo` | RPC: domain lifecycle/status/capabilities; exact methods require session API schemas |
| `session.demo.descriptor` | Pub/sub, retained latest: framework descriptor |
| `session.demo.snapshots` | Pub/sub, latest: committed simulation values and native media refs |
| `session.demo.events` | Pub/sub, bounded: scoped domain events; not a durable log |
| `app.pokemon.state` | Pub/sub, retained latest: application-specific state |
| `app.pokemon.cues` | Pub/sub: application narrative/presentation events under declared delivery policy |
| `app.pokemon` | RPC: application queries/admission, e.g. restore UI state or request a supported effect |
Descriptor revisions and scope link observations to schemas. Cross-topic ordering is not
guaranteed; a subscriber receiving an unknown descriptor revision must fetch it through the
application/session query contract or buffer a bounded number of snapshots, not infer shape.
Latest retained descriptors accelerate startup; RPC querying remains the repair path.
## 3. Common simulation descriptors and committed values
Types use [session RPC](ipc-v1.md), [workers](workers-v1.md), [state/media](state-media-v1.md):
```ts
interface SessionDescriptor {
sessionId: Id; revision: U64; compositionDigest: Digest;
schedulerId: "lockstep-v1";
environment: EnvironmentDescriptor; taskSchema: SchemaRef;
agents: {
agentId: Id; portId: Id; profileDigest: Digest;
datasetDigest: Digest; indexDigest: Digest; neuronCount: U64;
rateRoles: Id[]; supportedStimuli: Id[];
}[];
assets: AssetRef[];
}
interface CommittedSnapshot {
descriptorRevision: U64; publisherIncarnation: Id;
scope: Scope; episodeId: Id; sequence: U64; worldTime: RationalNs;
agents: {
agentId: Id; telemetry: AgentTelemetry;
selectedDecision: TypedValue | null;
appliedControls: PortControl | null;
}[];
progress: TypedValue;
media: { views: ViewRef[]; audio: AudioRef[] };
eventIds: Id[];
}
```
Publish only after all agent commits establish Ready(k). Decisions/controls describe the
transition ending at that boundary, null at initial boundary 0. Health updates are separate
and never claim an uncommitted future boundary. Every transient media reference is a declared
bus attachment held through publication admission. Ordinary snapshot publication is latest/
bounded and never waits for a spectator to consume it.
Publication sequence is monotonic within publisherIncarnation. Epoch determines simulation
timeline; router topicSequence only determines bus acceptance order. Never equate these.
Geometry/spike mapping requires indexDigest, not merely the same number of neurons. Persistent
AssetRefs survive release packaging; ephemeral ArtifactRefs never become permanent asset URLs.
## 4. Flexible data, authored UI
Common UI primitives should understand media, agents, controllers, typed measurements,
progressions, collections and events. Descriptors change infrequently; values change frequently.
Application-specific structures remain namespaced, schema-validated extensions, such as
`pokemon.progress.v1` or `melee.match.v1`. They are not mandatory fields on every snapshot.
Proposed measurement vocabulary to formalize with public v2 schemas:
```text
Definition: id, owner, label, kind, unit, optional range, schema revision
Sample: id, producing scope/time, validity, value
Validity: measured | unknown | unsupported | stale
```
Kinds include number/counter, gauge, duration, state, progression and collection with typed
item schemas. A measured zero is distinct from unknown. Stale values retain original timestamps.
Units/ranges are metadata, not pixel sizes. Unknown optional extensions may be omitted or shown
generically; unsupported required schemas are visible errors. No remote executable UI payloads.
Application state carries whatever the experience needs: progress history, featured fly,
competition records, season state or sponsor effects. It is developed with its presentation,
not forced into a framework-wide “show state”/tournament schema. An application can reuse
generic components and add its own panels without changing worker/transport contracts.
## 5. Artifact consumption and browser boundary
The native presentation client is a regular bus subscriber. Its renderer may hold an extracted
Artifact after dropping the message; the SDK delays consumption until actual use finishes.
Latest coalescing only drops queued values. A stalled consumer is constrained by finite credits,
owners and store budgets; it cannot make the router overwrite an in-use image.
The browser does not receive private owner tokens or local storage paths. A presentation
gateway resolves/copies/encodes artifacts into its chosen browser transport and then drops
its bus handles. That is an application-edge adapter, not a second framework communications
stack. Compositor/encoder/recorder processes inside the application can exchange their own
artifacts through the same bus when useful.
Dense spike publication is optional and identifies agent, index digest and covered ticks.
The runtime need not publish every neuron every millisecond. Required sensory data and
optional spectator data have distinct budgets; UI focus never changes an agent's input,
controller assignment or an already resolved stimulation/effect target.
## 6. Events, persistence and recovery visibility
Events identify session/epoch, source boundary, episode, optional agent, kind and typed payload.
Task events are emitted after committed transitions; capture/admission events describe their
actual phase. Bus publish acceptance and delivery consumption are not durable acknowledgments.
When durability is required, call a configured event-store client/service (over the same bus)
and await its append/commit acknowledgment under the session's configured policy. Pub/sub
remains useful for live observers; reconnecting clients query durable history through ordinary
RPCs. The initial conformance policy pauses at the next safe boundary if durable event
admission/commit fails, retaining only a bounded pending batch. No hidden durable broker queue.
After rollback publish old/new epochs, checkpoint identity and abandoned step ranges.
Application history can mark outcomes aborted/superseded; it does not erase records merely
because emulator time moved backward. Media reports the corresponding discontinuity.
## 7. Supervisory and audience effects
Application supervision uses bus RPCs for configured lifecycle/intervention capabilities
and pub/sub for application state/cues. The Twitch adapter can be a constrained bus client
of the application's admission service; viewers/browser clients never obtain worker control.
Legacy HTTP bridge behavior remains until deliberately migrated.
Effects are declared by the task/backend/profile: valid targets, parameter schema, timing,
duration/stacking, implementation capability and outcome events. Examples include neural
stimulation or future game items/modifiers, where verified implementations exist. Generic
game boons and a public v2 admission schema are follow-on work; initially new-session audience
effects remain disabled. No arbitrary controller/game-memory write endpoint is introduced.
```text
requested → rejected
→ accepted(target, epoch, earliestStep) → scheduled → applied → expired
└─ failed / cancelled
applied → rolled-back
```
The application persists interaction identity/target and defines retry, redemption, refund
and recovery policy. A gift is an intervention, not automatically an earned neural reward.
Presentation cues may be immediate; simulation effects apply at declared boundaries. The
supervisor does not bypass the complete-batch step barrier. Chat text remains presentation
data; template-only replies/quiet mode and existing no-public-button rules continue.
## 8. Presentation acceptance criteria
- Per-agent/session stores and rate/afterglow state, not one mutable global fly.
- Framework plus application schema streams, with descriptor repair/reconnect behavior.
- Native view dimensions and aspect; application-controlled output resolution and composition.
- Explicit audio ownership, timestamped overlays and bounded queues/discontinuities.
- Last-use artifact release, cached/replay-safe references and slow-observer isolation.
- Game-specific labels/views without Pokémon fields in generic runtime/router schemas.
- Actual UI changes reviewed as PNGs with browser/legibility gates. This contract is not screen approval.

212
docs/design/session-framework/state-media-v1.md Normal file
View file

@ -0,0 +1,212 @@
# Session artifacts, native media and recovery
Status: **draft 2**, 2026-09-18. [Flybus](bus-v1.md) owns generic artifact storage, delivery
ownership, retention and garbage collection. This document specifies the **domain meaning**
of those artifacts: native observations, clock association and coherent session checkpoints.
Read [session RPC](ipc-v1.md), [step ordering](step-v1.md) and [worker interfaces](workers-v1.md).
## 1. Use the bus ArtifactRef
Large payload fields use `ArtifactRef` from bus-v1, and every referenced artifact is listed
in the surrounding bus attachments. There is no separate BinaryRef, buffer-region registry,
coordinator lease endpoint, or Buffer.Release/Reclaim protocol. Profile/dataset release assets
use `AssetRef` (a persistent content identity); transient bus ArtifactRefs are not those assets.
The environment publishes one immutable native image. The coordinator may forward the same
owned handle to multiple agent Commit calls and publish it for presentation. Flybus creates
destination ownership before releasing the source. It does not send another copy of the
pixel bytes per recipient through its sockets.
An agent consumes/encodes pixels during Initialize/Commit and drops its handle when no longer
used. A renderer may keep its extracted handle after dropping the message; the DeliveryGuard
keeps the artifact alive until rendering has finished. A domain cached RPC result keeps its
own handles so replay remains valid after the original recipient consumes its delivery.
Content digests are optional on transient live frames, mandatory on checkpoint payloads and
persistent asset import. Ownership/index/byte-shape validation is always required. A digest
does not replace epoch or observation-time identity.
## 2. Native observation types
```ts
interface ViewDescriptor {
viewId: Id;
width: number; height: number;
format: "rgba8"; rowStride: number;
pixelAspect: { numerator: number; denominator: number };
observationDelaySteps: number;
}
interface ViewRef { viewId: Id; producedStep: U64; pixels: ArtifactRef }
interface AudioDescriptor {
streamId: Id; sampleRate: number; channels: number;
format: "f32le-interleaved";
}
interface AudioRef {
streamId: Id; firstSample: U64; sampleFrames: number;
samples: ArtifactRef; discontinuity: boolean;
}
```
Epoch is inherited from the domain observation; the bus treats it as opaque payload. View
dimensions are integers 1..4096, rowStride exactly 4×width, no padded rows in v1. Pixel aspect
numerator/denominator are positive integers <=65535; observationDelaySteps is integer 0..8.
Pixels are top-left RGBA8 and artifact length equals rowStride×height. Other formats require
a media-schema change, not special-case code inside the router.
Required sensory views have producedStep equal to
`max(0, observation.boundary - observationDelaySteps)`. Bootstrap may repeat O[0] until the
declared pipeline delay fills. Beyond that, missing/extra-delay sensory input is a step
failure, not an arbitrary latest frame. Observer publication may omit/coalesce frames while
preserving each artifact's actual producing boundary.
Audio sampleRate is integer 8000..192000, channels 1..8, sampleFrames 0..192000 per chunk.
Samples are finite f32; artifact length is sampleFrames×channels×4. firstSample identifies
the sample position relative to the episode's configured audio origin, with intended PTS
firstSample/sampleRate. Crash restore preserves sample position under a new epoch; first
chunk marks discontinuity. Within an epoch, chunks cannot overlap or go backwards.
The environment provides **native game output**. Sensor transformations belong to the agent
profile. Resizing for viewers, overlays, composition, audio mixing/resampling, encoding,
browser delivery and streaming belong to the application/presentation layer. No bus or
generic session configuration assumes a 1080p show or Twitch output.
640×480 RGBA at 60 fps produces 73.728 MB/s of raw image data. Artifact fan-out references
one stored object; reads and any staging/seal copy still consume memory bandwidth. This is
reasonable to measure before introducing codecs or pooled GPU buffers. Native dimensions
come from the backend, not a hardcoded GameCube or broadcast resolution.
## 3. Domain retention and backpressure
Use the same bus call/publish API for observations and artifacts. Bus ownership tracks bytes;
the session decides which observations are required and when they have been used.
| Use | Rule |
| --- | --- |
| Required agent input | Retain through encoding/Commit; no coalescing or overwrite |
| Step-result replay | Retain in the endpoint's current/previous-step cache until domain eviction |
| Spectator snapshot | Latest subscription, finite in-flight credits; release after actual use |
| Long rendering/storage job | Explicit artifact hold with a finite byte/count budget |
| Hot checkpoint | Coalesce only queued replaceable captures, releasing their holds |
| Durable checkpoint | Acknowledge after durable commit; reject/defer before capture when saturated |
Initial session defaults: two outstanding coherent captures and at most the bus-configured
latest/in-flight frame credits per observer. Presentation audio can target 250 ms and cap at
one second, but that is a presentation policy, not a bus or brain-clock requirement.
Budget cached step observations, active agent deliveries, retained latest and spectator holds
together. The producer dropping its handle does not free cached/queued/in-use objects.
A slow spectator exhausts its own credits; new latest messages replace its queued value.
If it violates configured resource policy, disconnect/restart that observer instead of freeing
live data or silently skipping simulation input. Global store exhaustion is an explicit fault
or pause condition; the router cannot guess that a particular live object is disposable.
No coordinator tracks per-reader socket acknowledgments or calls a producer's reclaim method.
The SDK and bus perform that bookkeeping. File-backed immutable mappings are safe after
unlink; physical pages disappear when all OS mappings close. Pooled reuse is deferred until
it provides equivalent safety. Bare handles in history are not persistent saved bytes.
## 4. Checkpoint identity and content
Exact-checkpoint sessions capture at Ready(k) or Paused(k), after all Agent.Commit replies,
with no Advance in flight. Block next Prepare until all participants supply immutable captures.
The manifest records:
- Envelope version, checkpoint ID and source session/epoch/step/episode/world time.
- Coordinator scheduler/configuration identity and exact port-to-agent map.
- Backend/content/patch/controller/parser/state-format compatibility.
- Per-agent profile/dataset/model identities, seed, tick count/remainder and payload digests.
- Task ledger, prior world inspection, per-agent executor state, next sensory/decision state
or reproducible reconstruction inputs, admission state and event watermarks.
- Payload names, lengths and hashes, including external-helper state required for exact resume.
Use a new envelope version; specify exact byte layout before production files. The historical
letter-only chunk-name constraint is not silently widened, and FLYSIM01 remains separately
readable. Persist payload **bytes and durable content identity**, not transient bus storeId,
artifact IDs, ownership tokens, mappings or pointers.
A checkpoint writer owns the bus Artifact handles until bytes are committed or the job fails.
It then drops them; durable files are outside Flybus's ephemeral GC. On restore, the durable
store imports fresh immutable bus artifacts. sourceScope is provenance, while the new handles
belong to the current router/store. Broker retention is never a substitute for a checkpoint.
## 5. State RPCs over the bus
Common to workers advertising checkpoint-v1:
```ts
interface CaptureParams { checkpointId: Id }
interface CaptureResult {
checkpointId: Id; boundary: U64;
compatibilityDigest: Digest;
payload: ArtifactRef; // listed attachment; digest required
}
interface StageRestoreParams {
checkpointId: Id; sourceScope: Scope;
compatibilityDigest: Digest;
payload: ArtifactRef; // newly imported owned attachment
}
interface StageRestoreResult { checkpointId: Id; restoreToken: Id }
interface ActivateRestoreParams { restoreToken: Id }
interface ActivateRestoreResult {
committedStep: U64; checkpointId: Id;
observation: WorldObservation | null; // environment required, agent null
}
```
State.Capture uses the committed scope. It completes after immutable capture exists, not when
a backend save was requested. The cached reply retains its artifact until Worker.Acknowledge.
The coordinator/writer obtains its own live ownership before acknowledging that cache.
State.StageRestore uses a proposed **new epoch** at source boundary k and is allowed only
on an uninitialized replacement or a quiescent worker. Launcher configuration supplies the
expected profile/backend identities; no implicit warm-up/reinitialization changes the saved
brain. It validates into replacement state, without exposing mutations to the live session.
After every participant and coordinator state validates, State.ActivateRestore installs
each staged token under that new scope without advancing a tick. Tokens are bound to scope/
payload/checkpoint and can activate only once; duplicate domain requests replay the cached
reply, while a fresh request trying to reuse an activated token is a conflict.
The environment returns the coherent restored observation with fresh artifact references and
restored time. It cannot advance gameplay to manufacture it. Capture/reconstruction therefore
covers render/inspection state and any pending sensor pipeline. Agent state agrees with it;
do not replay reward or recalibrate merely to fill missing cached data.
If emulator validation requires mutation, stage a stopped replacement emulator. If that cannot
provide externally atomic resume, advertise episode-restart, not exact-checkpoint. After all
activation acknowledgments, install the coordinator's staged task/executor/admission state
and establish Paused(new epoch,k). Failure during activation never permits half a group to run.
## 6. Durable commit, router failure and recovery
Write payload/envelope temporary generation, fsync, rename, fsync directory, then atomically
write/fsync/rename the store manifest and fsync its directory. **Manifest commit is the durable
commit point.** Unreferenced temporary generations are not automatic restore candidates.
Publish distinct captured/queued/committed/failed/superseded events over the same bus. Only
durable completion produces a saved acknowledgment/high-water mark. Failed writes release
owned ephemeral captures according to retry policy, without reporting false durability.
After participant/coordinator/router failure:
1. Stop steps, abandon the epoch and fence old participants/routes.
2. Connect to a live router and select a complete compatible durable checkpoint.
3. Import its payloads as new artifacts; stage/activate every participant and coordinator.
4. Verify identity/boundary, flush old media/parser queues and publish recovery/discontinuity.
5. Establish Paused(k), then resume only after the group invariant holds.
A router restart loses ephemeral topics, queues, roots and correlations. Continuing with
old handles is invalid even if some mapped bytes survived. Reconnect is not transparent
mid-step recovery. The durable log records old/new epochs and abandoned step ranges; rollback
can lose post-checkpoint work. Durable input replay requires a separate application/session
journal policy, not an exactly-once claim about Flybus.
## 7. Episode reset
Reset differs from crash restore. The application selects a policy; the coordinator records
the old episode's result/abort and creates a new epoch/episode at step zero. World initial
state and retained/fresh brain components are explicit. Gain retention, eligibility/hold
clearing, calibration and first sensory input are part of the policy, tested independently.
Legacy Pokémon ratchet behavior remains in the legacy composition. Shared competitive worlds
never restore one player's environment independently of the other players.

226
docs/design/session-framework/step-v1.md Normal file
View file

@ -0,0 +1,226 @@
# Lockstep step protocol v1
Status: **draft 2**. This is the authoritative new-session ordering contract. All method
arrows below are RPCs through the same [Flybus router](bus-v1.md); the router itself never
implements the barrier. Read [architecture](README.md) and [session RPC](ipc-v1.md) first. Method payloads are in
[worker interfaces](workers-v1.md).
## 1. Committed boundary
At `Ready(epoch, k)`:
- Environment state is at boundary `k`, with no outstanding action batch.
- Every agent has consumed the outcome of transition `k-1 → k`, including its new encoded
sensory input and rewards, and is ready to compute the decision for transition `k → k+1`.
- Task ledger, action-executor state, event identity and agent tick remainders agree with `k`.
- The current sensory observation may have a declared fixed render delay; its producing
boundary is explicit. “Ready” does not imply latest wall-clock screenshot.
- No normal step operation from an older boundary may mutate the session.
Only a committed boundary is eligible for a coherent checkpoint or normal pause. Boot/reset
establishes the same invariant with no preceding reward. The public snapshot represents this
boundary, not an in-progress combination of some new agent states and an old world.
## 2. State machine
```text
Starting → Ready(k) → Preparing(k) → Applying(k) → Observing(k+1)
↑ │
└──────────── Ready(k+1) ← Committing(k) ┘
Ready(k) → Paused(k) → Ready(k)
Ready(k) / Paused(k) → Capturing(k) → same boundary
any unresolved partial failure → Failed → Restoring(new epoch) → Paused(k)
terminal episode → Paused(k) → Resetting(new epoch) → Ready(0)
```
`Committing(k)` refers to completing transition `k → k+1`. Requests throughout that
transition carry `scope.step=k`; result fields identify `nextStep=k+1` where applicable.
Do not send Agent.Commit with step `k+1` merely because the observation is newer.
## 3. Transaction sequence
### Phase A: prepare all agents concurrently
At Ready(k), freeze the task's per-agent decision contexts and the coordinator's admitted
pre-step stimulation list. Inputs accepted after this cut wait for the next boundary.
Send `Agent.Prepare(scope=k)` to every active agent. Each worker:
1. Verifies its committed boundary/profile/context and applies admitted pre-step stimulation
in deterministic command sequence order. Chat text is never included.
2. Advances the numerical model for the environment interval, using the input encoded at
the preceding Commit (or initialization).
3. Reads rates and performs the fixed readout with the declared decision context.
4. Stores and returns `PreparedDecision`; it then enters Prepared(k) and waits for Commit.
This operation **mutates** the brain, RNG, clock and decoder. “Prepare” does not mean a
database transaction that can be rolled back cheaply. If another agent fails, do not ask a
prepared agent to prepare again or advance to the next step. Resolve/recover the whole session.
All agents see the same environment interval and the same world boundary, with only their
permitted view/context differences. Their completion order never affects port/action order.
### Phase B: build and apply one complete batch
After every PreparedDecision arrives:
1. Validate agent IDs, intent schemas and profile identities.
2. Run each task-local action executor once, in sorted agent-ID order, against coherent current
game state, task progress/objectives and clock from this boundary.
Direct-control profiles use an identity executor. Macro profiles are explicit extensions.
3. Assemble all configured port controls in descriptor port order; reject duplicates/missing
ports. Uncontrolled ports are configured neutral before the epoch, not supplied ad hoc.
4. Send exactly one `Environment.Advance(scope=k, batchId, controls)`.
The environment applies all controls at its agreed boundary, advances exactly one interval,
and returns StepResult for `k+1`. It MUST NOT advance another interval while waiting for
the next request. Transport/control scaffolding may have a measured fixed latency; it must
be declared in its descriptor and conformance tests.
### Phase C: observe and evaluate the task
The coordinator receives the environment result and verifies batch identity, boundary,
cadence, inspection schema and required sensory views. A missing spectator frame is tolerable;
a missing required sensory input is not silently replaced.
Call the task's `evaluate_transition` once with old/new inspection observations and applied
controls. It returns scoped rewards/stimulation, next decision contexts, progress/events and
an optional episode request. Commit its ledger update in memory and retain the result for
this transition. No task output directly writes controllers or neural state.
### Phase D: commit all agent outcomes concurrently
Send `Agent.Commit(scope=k)` with that agent's next sensory observation and routed outcomes.
Each agent, in this order:
1. Encodes/installs the next sensory input for the following Prepare.
2. Applies task-derived stimulation in returned event order.
3. Sums that agent's reward values in returned event order and calls reinforcement once at
its current brain time when learning is enabled. A zero sum still follows the profile's
specified legacy-equivalent reinforce behavior; do not optimize it away without evidence.
4. Retains the next decision context/digest and acknowledges committed boundary `k+1`.
There are no additional neural ticks in Commit. Sugar accepted for a future Prepare is not
silently merged with task reward modulation. The default synthetic learning mechanism remains
separate from the neural stimulation path.
Once **all** commits succeed, the coordinator advances its committed boundary to `k+1`,
finalizes the observation snapshot and scoped events, drops no-longer-needed artifact handles and
allows the next Prepare. If one Commit fails after others succeeded, the epoch is failed;
there is no partial-match continuation.
These phases establish logical coordination, not a distributed durable two-phase commit.
Crash recovery returns to the last complete checkpoint, not necessarily the last displayed step.
## 4. Sequence example
```text
Coordinator Agent A Agent B Environment
| Prepare(k) ------>| | |
| Prepare(k) ----------------------->| |
|<-- Prepared(A) ---| | |
|<-- Prepared(B) --------------------| |
| [executor + complete port batch] |
| Advance(k, batch-x) --------------------------------->|
|<-------------------- StepResult(k+1, batch-x) ----------|
| [task evaluation; route each reward once] |
| Commit(k, O[k+1], R_A) ->| | |
| Commit(k, O[k+1], R_B) ------------>| |
|<-- Committed(k+1) -----| | |
|<-- Committed(k+1) -----------------| |
| [Ready(k+1); publish; next boundary] |
```
The shared camera is one immutable artifact forwarded through bus-owned deliveries; both
workers can encode it without two renders or routing two full images through sockets.
Domain RPC replay caches retain handles, so consumption by one client cannot invalidate a
promised replay. Publication uses bus pub/sub and never waits for spectator consumption.
The coordinator retains each domain request's input handles until its terminal outcome is
resolved, beyond the shorter bus-admission lifetime, so safe domain retries still have valid
attachments. If ownership is lost, fail/recover instead of sending bare expired references.
The coordinator cannot publish a committed state as soon as the faster agent answers.
## 5. Time and pacing
The environment descriptor supplies a fixed reduced `stepDuration: RationalNs`. Each agent
profile supplies `tickDuration: RationalNs`. The existing LIF adapter uses exactly one ms.
For each Prepare:
```text
accumulator += environment step duration
ticks = floor(accumulator / model tick duration)
accumulator -= ticks * model tick duration
```
Use checked rational/integer arithmetic; remainder is always >=0 and < one model tick.
Do not accumulate rounded microseconds or nanoseconds for a fractional frame period.
Persist remainder, executed tick count and warm-up offset. Language implementations must
agree on remainder fixtures. Conversion to the legacy model's f64 millisecond clock must
preserve its representable integral ticks; refuse a run exceeding the supported exact range.
Example: a synthetic 60-Hz environment with a 1-ms model tick produces 16,17,17 ticks
over three steps, totaling 50. A real backend's measured/declared emulated cadence may
differ; never substitute this example's duration for Game Boy or Dolphin clocks.
Wall time is only for pacing, health and presentation. The coordinator schedules absolute
deadlines after committed boundaries; when behind, it omits sleep and reports lag. It does
not skip world steps, drop neural ticks, or let one agent advance more slowly than another.
Only one pacing authority is active. Backend throttling and coordinator pacing must be
configured/tested so they do not unintentionally double-throttle the session.
First-version profiles have a fixed cadence within an epoch. Supporting variable-duration
world advances requires a new capability and tests before enabling it.
## 6. Initialization, pause and episodes
Initialize the environment first while stopped, obtaining observation O[0]. Bootstrap the
task and initial decision contexts, then initialize agent workers with their permitted inputs.
Agent warm-up has learning disabled; calibration occurs on settled rates; no warm-up actions
advance the environment. All required acknowledgments establish Ready(0).
A normal pause request arriving mid-step means “finish this transition, then pause.” It does
not truncate neural computation or capture half an action batch. If completing the transition
is impossible, use failure/recovery, not an apparently successful Pause acknowledgment.
Paused workers retain state and answer Status; world controls do not advance the world.
Task terminal events are evaluated and their final rewards committed once. Before another
gameplay transition, enter Paused and apply the declared episode policy. Reset uses a new
epoch/episode and step 0. A profile may retain learned gains/brain state, but must identify
exactly what is retained, cleared, warmed or recalibrated. No worker independently resets.
Changing port assignment, agent membership, model/profile, cadence or task schema requires
a new composition/epoch. Hot-join and hot-swap during an active match are not v1 capabilities.
## 7. Failure rules
| Failure point | Required response |
| --- | --- |
| Before any Prepare admitted | Reject request/config or remain Ready; nothing advanced |
| Some agents Prepared | Stop dispatch; resolve matching requests or fail epoch/group restore |
| Advance acknowledgment lost | Query/retransmit same request to same incarnation; never new batch |
| World advanced, sensory data unavailable | Fail transition; do not reward/continue using guessed input |
| Task interpretation fails | Fail epoch; prepared brains/world already changed |
| Some Commit replies missing | Resolve exact requests; no next world step until all committed |
| Worker incarnation changes | All live participants belong to an invalid epoch; restore/reset together |
| Publisher/browser disconnected | Simulation continues; bound/drop spectator work |
| Durable storage fails | Report actual failure; apply configured pause/continue-with-stale-checkpoint policy |
Retries return original results; they never recompute a decision with updated rates or new
world data. A coordinator restart has no authority to assume any remote participant's phase;
recover from a coherent checkpoint into a new epoch or start an explicitly new episode.
## 8. Required trace assertions
The synthetic integration test must record, for every transition:
- Scope, Prepare request IDs, agent/profile IDs, tick counts/remainders and decision digests.
- Complete batch ID/control digest and acknowledged world boundary.
- Observation producing boundaries and task event/outcome IDs in order.
- All Commit acknowledgments and published committed boundary.
Evaluate agents sequentially, concurrently, and in reversed dispatch/completion order. All
committed state/action/reward results must match, excluding wall time, request IDs and other
explicitly operational metadata. Delayed/lost/duplicate messages must not add a neural tick,
world step, reward update or controller flush corresponding to another logical step.

367
docs/design/session-framework/workers-v1.md Normal file
View file

@ -0,0 +1,367 @@
# Worker and task interfaces v1
Status: **draft 2**. Uses [Flybus](bus-v1.md) for every call/publication and the domain
types/outcomes in [session RPC](ipc-v1.md), ordering in
[step protocol](step-v1.md), and binary/state types in [state and media](state-media-v1.md).
All method results below are the `result` object inside the domain result carried by a bus
rpc.result. Large inputs/outputs use owned bus attachments, never another worker data channel.
## Method registry
| Method | Caller → receiver | State/owner |
| --- | --- | --- |
| `Worker.Hello` | Authorized caller → named worker service | Domain identity/capabilities after bus negotiation |
| `Worker.Status` | Authorized caller → named worker service | Read-only; responsive during compute |
| `Worker.Acknowledge` | Coordinator → worker | Bounded lifecycle-result retention |
| `Worker.Shutdown` | Coordinator → worker | Terminal lifecycle request |
| `Agent.Initialize` | Coordinator → agent | Uninitialized → Ready(0) |
| `Agent.Prepare` | Coordinator → agent | Ready(k) → Prepared(k) |
| `Agent.Commit` | Coordinator → agent | Prepared(k) → Ready(k+1) |
| `Environment.Initialize` | Coordinator → environment | Uninitialized → boundary 0 |
| `Environment.Advance` | Coordinator → environment | Boundary k → boundary k+1 |
| `State.Capture` | Coordinator → agent/environment | Immutable snapshot of committed boundary |
| `State.StageRestore` | Coordinator → agent/environment | Validate replacement state under new epoch |
| `State.ActivateRestore` | Coordinator → agent/environment | Install staged state; remain quiescent |
State methods have payloads in [state and media](state-media-v1.md). Artifact lifetime and
message consumption are bus operations managed by the SDK, not Worker/Coordinator methods.
Worker.Acknowledge releases a domain result cache, distinct from consuming a bus delivery.
Task/executor interfaces below are local library methods, not extra communication protocols.
## 1. Shared data model
```ts
interface AssetRef {
id: Id; digest: Digest; byteLength: U64; format: Id;
}
interface SensoryInput {
boundary: U64; // environment boundary being observed
views: ViewRef[]; // only the views this agent is allowed to consume
structured: TypedValue | null;
}
interface Stimulus {
id: Id; kindId: Id; durationMs: number;
}
interface Reward {
eventId: Id; ruleId: Id; value: number;
}
interface AgentTelemetry {
brainTicks: U64;
populationRateHz: number;
rates: { roleId: Id; hz: number }[];
learning: { enabled: boolean; updates: U64; changed: U64; signal: number };
}
```
`AssetRef` names persistent content in a preprovisioned local registry; it is not an arbitrary path or
URL for a worker to fetch. A profile artifact contains all effective numerical, sensory,
readout, learning and schema identities. Dataset artifacts must already be installed and
verified. No implicit runtime network download or latest-version selection is allowed.
Transient `ArtifactRef` is instead defined by Flybus and resolves only through owned handles.
Views and structured input are separate capabilities. A pixel-only profile rejects non-null
structured input. Max views per sensory input: 8; any individual TypedValue is at most 32 KiB
of canonical JSON, and the complete bus envelope must fit 64 KiB. Larger typed state uses
an explicit artifact-backed schema. Frame bytes never go into JSON.
`Stimulus.kindId` resolves through a profile-declared capability to an anatomical binding
and fixed drive, including supported duration bounds. A caller cannot specify arbitrary
neuron indices or change drive values. `durationMs` must be finite and >0. Arrays of stimuli
or rewards are bounded to 64 per operation and retain their supplied order.
Rates must be finite/nonnegative, unique by role ID and in profile-defined order, at most
64 entries. Learning values must be finite; integer counters fit U64. Reward values are
finite; shipped positive-only task profiles reject negatives. Empty rewards do not imply a
different numerical rule. `id`/`eventId` is unique within its outcome or command namespace;
the coordinator assigns stable IDs before sending a mutating request.
## 2. Agent methods
### Agent.Initialize
Allowed only on an uninitialized agent with negotiated `agent-step-v1`. Scope is the new
epoch at step `0`; restore uses the state interface instead of Initialize.
```ts
interface AgentInitializeParams {
agentId: Id;
profile: AssetRef;
seed: number; // signed 32-bit integer, matching current RNG input
initialInput: SensoryInput;
initialDecisionContext: TypedValue;
workerThreads: number; // integer >=1; within launcher allocation
}
interface AgentInitializeResult {
agentId: Id; profileDigest: Digest; tickDuration: RationalNs;
warmupTicks: U64; committedStep: U64; // committedStep == "0"
decisionContextDigest: Digest;
telemetry: AgentTelemetry;
}
```
The profile fixes warm-up/calibration behavior and supported schema versions. Validate inputs
and required roles before model construction. Install the initial sensory input, warm the
brain with learning disabled, calibrate the fixed readout and establish Ready(0). Do not
generate gameplay rewards or controls that advance the world during warm-up.
Seed is persisted as run configuration and state; the coordinator derives independent seeds
from its recorded master seed and stable agent IDs under a versioned derivation algorithm.
That algorithm is part of composition identity and MUST be specified/tested before the real
agent slice; hand-selected explicit seeds are supported for the first synthetic composition.
Identical explicit seeds are allowed only when the experiment intentionally declares them.
The profile artifact digest identifies the profile definition; the capture compatibility
digest additionally covers the resolved seed, numerical model version and effective instance
configuration. Never assume identical profile digests make differently initialized state
interchangeable without matching that instance configuration.
### Agent.Prepare
Allowed from Ready(k), or as an exact domain retry under the session RPC deduplication rules.
```ts
interface PrepareParams {
agentId: Id; profileDigest: Digest;
interval: RationalNs;
decisionContextDigest: Digest;
preStepStimulations: Stimulus[];
}
interface PreparedDecision {
agentId: Id;
ticksAdvanced: U64; brainTicks: U64; remainder: RationalNs;
decision: TypedValue;
}
```
Verify the cached context digest from Initialize/last Commit, expected agent/profile and
interval. Apply pre-step stimuli, advance ticks and decode as specified by the step protocol.
The returned decision schema is the profile's registered intent schema. It may describe a
controller state or selected semantic action, but cannot assign a port or include hidden
task-inspection fields. Context may mask declared available actions; it cannot change the
readout weights, invent a default winner or inject arbitrary neural observations.
Successful response leaves the worker at Prepared(k). A duplicate returns the same decision,
ticks and remainder. It MUST NOT resample randomness, repeat stimulation or calibrate again.
### Agent.Commit
Allowed only at Prepared(k), matching the current transition and exact Prepare request.
```ts
interface CommitParams {
agentId: Id;
preparedRequestId: Id;
nextInput: SensoryInput; // boundary == k+1
nextDecisionContext: TypedValue;
rewards: Reward[];
taskStimulations: Stimulus[];
}
interface AgentCommitResult {
agentId: Id; committedStep: U64; // k+1
decisionContextDigest: Digest;
telemetry: AgentTelemetry;
}
```
Validate the complete request and required owned artifacts before applying it. Follow exact input→
stimulation→reinforcement ordering in the step protocol. A missing required view is an error,
not zero input. An outcome requesting unsupported learning/stimulation is an error, not a
silent no-op. Disabled learning is a declared profile/run state, not “unsupported.”
No tick is executed in Commit. Cache its response before accepting the following Prepare.
The next context is retained for that Prepare; its canonical digest is returned and checked.
After encoding/copying and all asynchronous use finish, the worker drops its input Artifact
handles. The SDK consumes the delivery when the last associated guard disappears. A stored
input pointer must retain its handle. Cached replies retain their own artifact ownership.
### Agent interface implementation boundary
An agent worker bundles numerical model, sensor encoder and readout. It need not copy the
legacy `NeuralAgent::tick` call order: the existing service already orchestrates substeps.
Use a small adapter over the reference primitives and preserve the new specified ordering.
New numerical semantics require reference-first implementation and new model identities;
this protocol is not permission to change the pinned default kernel.
## 3. Environment methods
### Controller and descriptor types
```ts
interface ControllerSchema {
schema: SchemaRef;
buttons: Id[]; // <=32, unique, fixed order
axes: { id: Id; range: "bipolar" | "unit"; neutral: number }[]; // <=16
}
interface PortControl {
portId: Id;
buttons: { id: Id; down: boolean }[];
axes: { id: Id; value: number }[];
}
interface EnvironmentDescriptor {
backendDigest: Digest; contentDigest: Digest; configurationDigest: Digest;
stepDuration: RationalNs;
ports: { portId: Id; controls: ControllerSchema }[];
inspectionSchema: SchemaRef;
views: ViewDescriptor[];
audio: AudioDescriptor[];
recovery: "exact-checkpoint" | "episode-restart";
determinism: "fixed-build" | "unverified";
}
```
Every active port control must include every declared button and axis in descriptor order.
All IDs must match exactly; no duplicates, extra controls or omissions. Bipolar axes are
finite [-1,1]; unit axes [0,1]. Neutral lies in range. Do not silently clamp an out-of-range
caller value. Hardware-specific quantization/dead zones are backend configuration, applied
exactly once and tested against observed controls.
`fixed-build` asserts tested repeatability under the pinned configuration, not universal
bit-exact behavior across CPU architectures, GPU drivers or emulator versions. Those limits
must be in the backend implementation guide and run manifest. `unverified` cannot satisfy
an exact-replay production composition without an explicit scope change.
### Environment.Initialize
```ts
interface EnvironmentInitializeParams {
backendConfig: AssetRef;
taskConfig: AssetRef;
episodeId: Id;
portBindings: { portId: Id; agentId: Id }[];
}
interface EnvironmentInitializeResult {
descriptor: EnvironmentDescriptor;
observation: WorldObservation; // boundary 0, worldTime zero
}
interface WorldObservation {
boundary: U64;
worldTime: RationalNs; // logical time since episode start; preserved on crash restore
engineFrame: string | null; // backend-defined signed counter; <=64 characters
sensoryViews: ViewRef[];
inspection: TypedValue;
broadcastViews: ViewRef[];
audio: AudioRef[];
}
```
Scope is new epoch step 0. Backend/task config artifacts identify exact game content,
patches, initial-state/setup policy, graphics/timing/parser and controller conversion.
No unresolved “latest” settings. Environment setup may be application-specific, but it is
declared lifecycle scaffold, not actions attributed to a fly. The environment is stopped
when it returns O[0] and cannot free-run during brain initialization.
The environment only needs backend-relevant portions of task setup, not reward rules or
neural policies. `taskConfig` resolves a declared setup configuration; the complete task
implementation and ledger stay in the coordinator.
### Environment.Advance
```ts
interface AdvanceParams {
batchId: Id;
controls: PortControl[];
}
interface StepResult {
batchId: Id;
appliedFromStep: U64; nextStep: U64;
appliedControlsDigest: Digest;
observation: WorldObservation;
}
```
Validate scope k, complete controls and identities before releasing a backend input barrier.
Batch IDs are unique within an epoch; reusing one for a different request/step is a conflict.
Apply the complete batch to its interval and advance one framework step. Record batch ID,
result and next boundary before acknowledging. StepResult's control digest is over validated
canonical requested controls; backend quantization does not silently change that definition.
Observed game-pad values, if provided, are separately schema-labeled inspection data.
The environment returns exactly boundary k+1, worldTime advanced by its stepDuration, and
the required sensory views or a typed failure. An adapter with measured input latency must
describe it in its versioned backend config and prove its frame mapping; it cannot claim an
unmeasured same-frame response. An emulator may execute internal cycles/polls, but cannot
hide multiple framework steps behind one result.
### Environment pause behavior
At a committed world boundary the backend already awaits the next Advance; normal session
pause does not need a separate per-frame RPC. When an emulator requires an explicit hardware/
CPU pause to hold that invariant, the adapter owns it and must prove it. Status/Shutdown
remain responsive. A render/audio worker may drain already-produced data while stopped,
but no new gameplay state may advance.
## 4. Coordinator-local task and executor interfaces
These are library interfaces in v1, not additional bus services. Equivalent typed interfaces
may be implemented in Rust; names below specify semantics rather than compilable code.
```text
Task.bootstrap(initialInspection, bindings)
→ perAgentDecisionContexts, progress, initialEvents
Task.evaluate_transition(scope, oldInspection, newInspection, appliedControls)
→ perAgentOutcomes, perAgentNextDecisionContexts, progress, events, episodeRequest
ActionExecutor.apply(scope, agentDecision, currentGameState, progressView, clock)
→ ControllerIntent, executionEvents
Task.capture / validate_restore / install_restore
ActionExecutor.capture / validate_restore / install_restore
```
Task owns a checkpointable ledger. The coordinator calls each transition evaluation exactly
once after its acknowledged world step and retains the output until all agent commits finish.
If task mutation is followed by failure, restore the group; never reevaluate against a later
observation. Task-produced outcomes are keyed by configured agent ID; unknown/missing agents
are errors. Every agent receives explicit outcome arrays, including empty ones.
`ControllerIntent` contains buttons/axes conforming to the assigned port's controller schema,
but not a port assignment. The coordinator supplies the port. Per-agent executor state is
private; a running macro may emit controls according to its declared policy, but only after
neural selection. The first implementation supports the stateless identity executor only.
The executor's currentGameState is a coherent read-only inspector view at this boundary;
progressView supplies task history/objectives. It updates its selected action every step
(movement, path replanning, interaction, completion), not merely replaying a blind button
sequence. These game-aware inputs stay in the task/executor layer. For an external backend,
they arrive as typed observation/artifact data over the same bus, not per-byte remote reads.
Task contexts sent to the neural readout are typed, bounded, versioned and allowlisted by its profile.
They may express boot state or available actions. They are distinct from neural sensory
input and broadcast telemetry. A profile using game-state features as neural input must
explicitly declare structured sensing; the task cannot smuggle it into an opaque context.
Task events carry `{id, kindId, sourceStep, agentId:null|Id, payload:TypedValue}`. Event order
is task-defined and deterministic; `sourceStep` is the newly reached boundary k+1 for a
transition event (0 for bootstrap events). Event IDs are derived deterministically from
epoch, source step, task/rule and event ordinal, encoded as an Id. Rewards and stimulation
referencing these events must
have a configured recipient. Broadcast text is generated by task/presentation templates,
not arbitrary raw inspector memory or incoming chat.
`episodeRequest` is either null or `{kind:"terminal", reason:Id, outcome:TypedValue}`.
It requests a coordinator-owned policy transition after final reward commit; it cannot reset
the environment directly. Generic progress is a TypedValue, not mandatory Pokémon ladder data.
## 5. Admission and audience boundary
The first synthetic implementation has no audience input. Later integration maps permitted
public requests into a coordinator admission record containing stable interaction ID, agent
ID, accepted epoch/earliest step and profile-supported stimulus. Only the coordinator can
place that stimulus in Prepare. Viewer names/chat never enter the neural worker contract.
Rate limits and target resolution occur before the step's command cut. A later-selected
UI focus or new match cannot retarget an already accepted interaction. Bus/domain RPC deduplication
does not itself define payment/redemption semantics across epoch recovery; a future public
v2 contract must specify accepted/applied/rolled-back/aborted states and reconciliation before
paid interactions are enabled. Do not inherit a claim of durable exactly-once stimulation
from these in-memory worker request caches.
## 6. Health, shutdown and extensions
All workers implement the common Hello/Status/Shutdown/Acknowledge methods. Capture/restore
methods are in the state contract and mandatory only when exact-checkpoint capability is
advertised. Unsupported methods return `UNSUPPORTED`, mutation none.
New task-specific fields belong in registered TypedValue schemas. New worker capabilities,
variable-duration stepping, subscriptions or additional sensor modalities require a contract
change and shared fixtures. An unconstrained plugin dictionary is not a substitute for that.