diff --git a/docs/design/malecns-modular-implementation.md b/docs/design/malecns-modular-implementation.md new file mode 100644 index 0000000..7b66b88 --- /dev/null +++ b/docs/design/malecns-modular-implementation.md @@ -0,0 +1,268 @@ +# Implementation backlog: MaleCNS and modular sessions + +Status: **planned, not started**. Written 2026-09-18. Companion to the +[design and code analysis](malecns-modular-sessions.md), based on `f7bc13a`. + +This is the execution order for that proposal. It does not authorize deployment or a live +stream. Reconcile the baseline with merged macro/shop/recovery work before implementation. +Existing feed/control contracts and TypeScript oracle rules remain binding. + +Concrete second-game audit: [Melee framework and emulator plan](melee-framework-audit.md). +Its MELEE-01/02 spikes specialize EMULATOR-01 below and can proceed alongside framework +extraction; they do not depend on importing MaleCNS first. + +The [session framework contract set](session-framework/README.md) now specifies the new +multi-process architecture. Its [implementation guide](session-framework/implementation.md) +breaks RUNTIME/WIRE/STATE work into concrete schema, transport, coordinator and worker slices. + +**Communications decision:** build [Flybus](session-framework/bus-v1.md), one Rust RPC/pub-sub +router with immutable external artifacts, delivery guards and last-owner GC. It serves workers, +application supervision, presentation and storage. No second direct-RPC system or NATS broker. +Application-specific orchestration and frontend are developed together; native frames leave +the environment, while resizing/compositing/streaming belongs to presentation. + +## 1. Delivery strategy + +Deliver working vertical slices; keep the existing FAFB/Game Boy composition usable throughout. + +1. Establish a behavior baseline and explicit dataset/profile identities. + Independently build the generic Flybus example; it needs neither dataset nor emulator. +2. In parallel workstreams, characterize MaleCNS and extract the single-agent session runtime. +3. Demonstrate two isolated brains driving one ROM-free shared environment. +4. Expose that session through versioned feed/control contracts and a multi-agent broadcast. +5. Integrate a specifically chosen alternative emulator after its capability spike passes. +6. Finish physical package reorganization once the second consumer proves the boundaries. + +**First milestone:** a reproducible headless MaleCNS run and a reusable single-agent session. +**Second milestone:** two flies in a synthetic arena with coherent resume and a local broadcast. +**Third milestone:** two flies in the chosen fighting game, with documented task scaffolding. + +No elapsed-time estimate is committed before the import and emulator spikes establish their +unknowns. Split an item further when its contract and implementation cannot be reviewed together. + +## 2. Work queue + +Every item starts pending. Branch names are suggested implementation branches, not branches +already created. Builders use separate worktrees; the coordinator reviews contracts and results. + +### FOUNDATION-01 — Pin existing behavior + +- **Branch:** `test/session-baseline` +- **Depends on:** reconciliation with current main and related open work. +- Record effective legacy configuration, fingerprint, version strings and frame ordering. +- Add a ROM-free transition harness around the service's frame orchestration; capture brain + ticks, decoded actions, reward application and recovery effects with explicit clocks. +- Distinguish exact numerical replay from intentional macro/hold/transient reset on restore. +- **Done:** existing goldens pass; a trace fixture detects reordered vision/reward application; + legacy feed/API fixtures and compatibility identity are unchanged. + +### FOUNDATION-02 — Specify bundle and behavior identities + +- **Branch:** `feat/brain-profile-contract` +- **Depends on:** FOUNDATION-01. +- Specify dataset manifests, original-ID mapping, anatomical roles, sensory bindings, + readout bindings and composite behavior identity in a focused contract. +- Add profile resolution/validation around the existing core in TypeScript and Rust. +- Keep schema-1 fingerprinting and the legacy macro-role exception behind the legacy path. +- Add strict graph validation for new bundles, shared invalid fixtures and profile mismatch tests. +- **Done:** empty required populations, malformed CSR and incorrect profile restores fail; + legacy FAFB artifacts and default numerical version strings remain unchanged. + +### DATA-01 — Acquire and normalize MaleCNS + +- **Branch:** `feat/malecns-import` +- **Depends on:** FOUNDATION-02. +- Build a source-specific importer from official v1.0 tables with checksummed source locks. +- Reconcile the Codex versus neuPrint inventories, or explicitly select and document one. +- Preserve raw contact counts, transmitter evidence, original IDs and missing-data indicators. +- Emit deterministic graph bundles, weight≥1/weight≥5 comparison variants, and an exclusion/ + clipping/coverage report. Decide schema-1 feasibility from measured weight ranges. +- Add attribution and license records with actual artifacts; keep raw downloads out of git. +- **Done:** repeated builds match; endpoint/role/index invariants pass; both loaders agree; + no service startup or ordinary unit test needs a download. + +### DATA-02 — Characterize MaleCNS in the current kernel + +- **Branch:** `feat/malecns-baseline-profile` +- **Depends on:** DATA-01. +- Audit L1 geometry, hemisphere handling, KC/MBON/PAM mappings and brain-versus-VNC motor roles. +- Define a versioned fixed readout; keep task action partitions out of anatomical truth. +- Run learning-off first, then learning-on, using fixed sensory traces and multiple seeds. +- Generate TS reference goldens and compare Rust exactly; measure activity, saturation, + initialization, memory and per-phase latency for both graph thresholds. +- **Done:** publish a reproducible characterization report and profile choice. Stop task + integration if required mappings are missing or dynamics are unusable; any recalibration + becomes a named profile rather than an edit to the legacy model. + +### RUNTIME-01 — Separate environment execution from task interpretation + +- **Branch:** `refactor/environment-task-boundary` +- **Depends on:** FOUNDATION-02. +- Specify controller ports, digital/analog controls, rational cadence, media descriptors, + observation ownership and backend capabilities. +- Wrap binjgb as the first environment; retain Game Boy FFI/cache/state behavior. +- Keep Pokémon memory inspection, objective routing, macros and reward rules in its task. +- The executor receives coherent current game state, progress/objective view and clock on + every step. This richer context is not implicitly passed to the neural sensory encoder. +- Preserve existing imports through a facade; avoid simultaneous directory moves. +- **Done:** existing single-agent action/reward traces match and a fake environment can be + driven through the same boundary without importing binjgb or task-specific addresses. + +### RUNTIME-02 — Extract the single-agent session + +- **Branch:** `refactor/session-runtime` +- **Depends on:** RUNTIME-01 and BUS-01..03 from the contract implementation guide. +- Move deterministic agent/environment/task orchestration out of `Sim` into a library. +- Keep HTTP, WebSocket serialization, wall-clock publication and process supervision in flysim. +- Route internal worker calls and observations through Flybus; keep public compatibility + adapters at the application edge. Domain caches own artifact handles for replay. +- Give session clock, action executor, task ledger and recovery state explicit owners. +- Wrap the existing composition with legacy ordering, checkpoint and reset semantics. +- **Done:** headless bus client runs a session without Twitch/browser; the legacy composition + passes its traces and restore tests; a slow snapshot consumer cannot stall simulation. + +### RUNTIME-03 — Add synchronized multi-agent sessions + +- **Branch:** `feat/multi-agent-arena` +- **Depends on:** RUNTIME-02. +- Implement a ROM-free two-player arena and per-port controller ownership. +- Evaluate both brains against one observation boundary; apply one complete action batch; + advance the world once. Start sequentially, then verify parallel execution equivalence. +- Isolate RNG, stimulation, decoder holds, gains, traces and rewards per agent; share only + immutable topology. Enforce a total worker budget and single-dispatcher pool ownership. +- Define participant failure, lateness and episode reset policies. +- **Done:** no cross-agent state leakage; swapping evaluation order leaves results unchanged; + one failed participant cannot accidentally advance a half-controlled match. + +### STATE-01 — Capture and resume whole sessions + +- **Branch:** `feat/session-checkpoints` +- **Depends on:** RUNTIME-03; specify the state contract during RUNTIME-02. +- Define the new envelope/manifest and preserve the `FLYSIM01` reader. +- Capture all agents, environment, task/executor/admission state and clock remainders at one + boundary. Bound off-thread write jobs and retain atomic manifest commit semantics. +- Validate all components before installing any restored state; define external-backend staging. +- **Done:** uninterrupted and resumed synthetic matches agree; corrupting any participant + refuses the generation without partial restore; crash-injection fallback tests pass. + +### WIRE-01 — Introduce session feed/control v2 + +- **Branch:** `feat/session-protocol-v2` +- **Depends on:** FOUNDATION-02, RUNTIME-02; use RUNTIME-03 fixtures for integration. +- Write binding contracts before consumer implementation: descriptors, scoped agents/events, + media IDs/timestamps, task progress, targeted stimulation and retry/idempotency behavior. +- Implement Rust/TS codecs, schemas and a fake server; preserve the legacy v1 surface. +- Specify descriptor reconnect behavior, asset/index identity, bounded message sizes and audio gaps. +- **Done:** cross-language fixtures pass for unequal neuron counts and shared/private views; + duplicate attachment kinds no longer collide; ambiguous targets and incompatible schemas fail. + +### PRESENTATION-01 — Compose multi-agent stage and bridge + +- **Branch:** `feat/multi-agent-broadcast` +- **Depends on:** WIRE-01, RUNTIME-03. +- Replace stage store/scaler singletons with session/agent instances and one paint scheduler. +- Combine framework descriptors/measurements with application-owned state/cues over the bus. + Rendering may retain an artifact after message drop; last-use release returns delivery credit. +- Resolve geometry from hashed descriptors, preserve the Game Boy presentation, and add a + shared-match layout with explicit audio ownership. +- Route bridge commands/redemptions to persistent session/agent identities; test lost responses, + retries and restart without applying an interaction twice or to a different agent. +- **Done:** local synthetic match broadcast works; two-agent PNGs receive operator review; + browser/fixture/legibility checks pass; bridge remains template-only and quiet-mode capable. + +### DATA-03 — Run MaleCNS through the complete application + +- **Branch:** `feat/malecns-session` +- **Depends on:** DATA-02 and descriptor-aware assets from WIRE-01/PRESENTATION-01. +- Expose explicit profile selection and create a fresh MaleCNS state namespace. +- Verify task/controller bindings, stimulation capability and displayed anatomy identity. +- A narrow single-agent descriptor extension may ship earlier only with matching v1 contract + and consumer updates; do not publish MaleCNS spikes as implicit FAFB indices. +- **Done:** local one-hour soak and restore drill pass; paired learning-off/on observations + are recorded without claiming improved play; FAFB remains available unchanged. + +### EMULATOR-01 — Establish the alternative backend's capabilities + +- **Branch:** `spike/fighting-game-backend` +- **Depends on:** RUNTIME-01; can proceed alongside later runtime work. +- Choose the exact game/version and emulator; Melee/Dolphin is a candidate, not a commitment. +- Prove pause/step, simultaneous ports, analog input, frame/audio capture, state inspection, + save/restore, process lifecycle and achievable cadence with synthetic controller traces. +- Prefer a bus-connected helper if embedding would leak emulator internals into the session + library. Its native emulator protocol is an implementation detail, not a second framework API. +- **Done:** capability report includes pinned backend/content identity and reproducible results. + If bounded stepping or coherent restore fails, stop and revise the backend/requirements + before writing neural game logic. No game content enters repository fixtures. + +### EMULATOR-02 — Build the two-fly fighting-game slice + +- **Branch:** `feat/two-fly-fighting-game` +- **Depends on:** EMULATOR-01, STATE-01, PRESENTATION-01. +- Implement fixed controller mapping, match/round interpretation, positive attributed rewards, + observation policy and episode recovery. Display selected actions and actual controls. +- Validate one fly, two flies, round transitions, backend failure and resume in that order. +- Run side swaps and repeated seeds; compare learning-off and simple control baselines before + interpreting win rates. Separate show settings from controlled evaluation settings. +- **Done:** repeated local matches sustain declared cadence; restoration/failure policies work; + scaffold, interventions and limits are documented; reviewed match presentation is legible. + +### PACKAGE-01 — Finalize reusable packages and release compositions + +- **Branch:** `refactor/reusable-package-layout` +- **Depends on:** a useful second backend plus PRESENTATION-01. +- Extract proven crate/package boundaries from the design's module table; preserve facades. +- Move the Rust workspace only in a mechanical follow-up if it makes library consumption clearer. +- Update CI/build/vendor/golden paths, dataset/view asset packaging and compatibility preflight. +- Add minimal external-style Rust/TS consumers and a synthetic example composition. +- Reconcile current docs, stale template explanations and licensing/asset attribution. +- **Done:** both legacy and new compositions package successfully; incompatible state is + rejected before release selection; libraries run without importing broadcast services. + +## 3. Dependency map and first execution batch + +```text +FOUNDATION-01 → FOUNDATION-02 ┬→ DATA-01 → DATA-02 ───────────────→ DATA-03 + └→ RUNTIME-01 → RUNTIME-02 → RUNTIME-03 → STATE-01 + │ └→ WIRE-01 ───────┐ + └→ EMULATOR-01 PRESENTATION-01 + │ + STATE-01 + EMULATOR-01 + PRESENTATION-01 → EMULATOR-02 + second backend + presentation → PACKAGE-01 +``` + +The item dependency lists are authoritative; the diagram is a reading aid. The detailed +contract guide adds BUS-01 (RPC), BUS-02 (pub/sub) and BUS-03 (artifacts/GC) before the +distributed RUNTIME-02/03 slices. These can proceed independently of MaleCNS import. + +Start the preservation track at FOUNDATION-01; the generic bus track can start with the +contract guide's small RPC/pub-sub/artifact example. Then review FOUNDATION-02's contract +before assigning DATA-01 and RUNTIME-01 to independent worktrees. +Contract/schema authorship is serialized to avoid conflicting definitions. Deployment host +work remains serialized under the repository's claim protocol. + +## 4. Definition of done for every implementation branch + +- Scope and intentional behavior changes are stated; compatibility impact is explicit. +- Meaningful boundary tests cover the changed behavior; the TS oracle is not adjusted to + accommodate Rust output. Existing committed real-data goldens stay mandatory. +- `npm test`, `npm run typecheck`, `cargo test --workspace` (Rust workspace) and + `infra/tests/lint.sh` pass before merge. Visual changes also pass applicable Playwright + checks and PNG review. Optional full-MaleCNS/ROM runs record skips honestly. +- Performance-sensitive changes report representative activity, agent count, thread budget, + memory and tail latency. New experiments state what is modeled versus handwritten. +- Review the complete diff, merge with `--no-ff` when authorized, and update this queue with + commit, evidence and unresolved follow-ups. Rollback includes compatible state, not just code. + +## 5. Decisions needed before the relevant work starts + +| Decision | Deadline | Default recommendation | +| --- | --- | --- | +| MaleCNS inventory/filter policy | DATA-01 completion | Official versioned source; retain both threshold variants until measured | +| MaleCNS sensory/readout profile | DATA-02 | Audited L1 mapping with existing numerical model first | +| Exact fighting game and backend | EMULATOR-01 | Evaluate one concrete title/backend rather than supporting a console family at once | +| Number of flies and target resource budget | RUNTIME-03 performance gate | Two first; characterize four before promising it | +| Learning retention and sugar in matches | EMULATOR-02 task contract | Retention explicit; stimulation disabled in controlled comparisons | +| Package publication versus monorepo reuse | PACKAGE-01 | Monorepo libraries/examples first; public package publishing later | + +There is no need to resolve these now to plan another feature. This backlog is ready for +resumption at FOUNDATION-01 and the independent BUS-01..03 track. diff --git a/docs/design/malecns-modular-sessions.md b/docs/design/malecns-modular-sessions.md new file mode 100644 index 0000000..9be1214 --- /dev/null +++ b/docs/design/malecns-modular-sessions.md @@ -0,0 +1,918 @@ +# MaleCNS and reusable streamed simulation sessions + +Status: **proposal, not an implemented contract**. Written 2026-09-18 against `f7bc13a` +on `main`. This document covers two related projects: adding MaleCNS v1.0 as another +connectome, and extracting reusable modules for other emulators, embodied environments, +and multiple flies. No dataset, neural semantics, deployed configuration, or wire contract +is changed by this document. + +The binding [feed](../feed-protocol.md) and [control](../control-api.md) contracts take +precedence. The TypeScript brain remains the oracle. Existing default versions +`lif-1ms-f64-v2` and `fly-kc-mbon-rstdp-v2` remain pinned. + +Reading map: sections 2–3 contain the code audit and MaleCNS analysis; sections 4–6 +define the proposed module/session boundaries; sections 7–8 give the extraction order, +implementation workstreams and acceptance gates; sections 9–10 record open questions and +sources. + +Execution queue: [implementation backlog](malecns-modular-implementation.md), with branch-sized +deliverables, dependencies and completion criteria. Start at FOUNDATION-01 when work resumes. + +Concrete follow-up: [Melee emulator and multi-fly framework audit](melee-framework-audit.md), +including source-checked Dolphin/libmelee integration options and full-stack performance gates. + +Implementation contracts: [session framework](session-framework/README.md), defining private +Flybus RPC/pub-sub, lockstep phases, worker methods, artifact ownership, recovery and publication. +The [bus specification](session-framework/bus-v1.md) is the selected communications design: +one small Rust router, external immutable artifacts and delivery-scoped GC. Application +orchestration/presentation are developed together; tournaments are examples, not framework types. + +## 1. Recommendation + +1. **Add MaleCNS as a dataset/profile combination, not a replacement neural model.** + First run it through the existing LIF semantics with explicit, independently versioned + sensory, population, and readout mappings. Study different neuron dynamics separately. +2. **Make a session the unit of simulation ownership.** A session has one environment and + one or more independently stateful agents bound to its control ports. A shared match + advances once after all players have chosen actions from the same observation boundary. +3. **Extract along ownership and timing boundaries.** Separate anatomy, neural dynamics, + sensor encoding, action decoding, environment execution, task semantics, persistence, + observation transport, presentation, and audience interaction. Preserve current behavior + through a legacy composition while extracting these modules. +4. **Prove the design on a ROM-free two-player arena before a larger emulator.** Then build + a frame-stepped emulator integration. For “flies play Smash,” the first candidate should + be a specifically chosen title/backend, such as Melee with a pinned Dolphin integration; + “Smash” alone is not an emulator requirement. +5. **Retain a monorepo and one Rust workspace initially.** Reusable libraries do not require + a network of microservices, dynamic native plugins, or publishing unstable packages. + +The two tracks can progress independently after the identity/profile boundary is established. +MaleCNS does not require multiplayer; multiplayer does not require MaleCNS. The first useful +deliverables are a reproducible MaleCNS characterization run and a behavior-preserving +single-agent session API, not a wholesale rewrite. + +## 2. What the code actually does today + +Paths below are relative to the repository root. Rust paths beginning `core/`, `gb/`, or +`sim/` in this document abbreviate `services/flysim/crates/flybrain-core/`, +`services/flysim/crates/flybrain-gb/`, and `services/flysim/crates/flysim/` respectively. +These aliases refer to **current** paths, not proposed directories. + +| Boundary | Evidence inspected | Consequence | +| --- | --- | --- | +| Reference brain | `packages/brain/src/agent/agent.ts`; `core/src/agent.rs` | `NeuralAgent` composes network and decoder; image size and milliseconds/frame are configurable, but defaults are Game Boy-specific. It is already more reusable than the service. | +| Anatomy | `packages/brain/src/dataset/format.ts`; `core/src/dataset.rs` | CSR graph dimensions are dynamic. Weights are signed `i16`; roles and a two-dimensional visual-column table are part of the model input. Validation currently checks array lengths, not every graph invariant. | +| Dataset construction | `tools/build_flywire.py` | Five checksum-pinned Codex exports; stable indices from sorted root IDs; directed-pair aggregation; transmitter signs; clipping to ±32767; FAFB-specific role and L1-column extraction. Game macro populations are also generated here. | +| Neural dynamics | `core/src/lif.rs`; `packages/brain/src/model/lif.ts` | One-ms ticks, configurable gain/noise, role stimulation, image drive, and up to 64 tracked rate roles. It is not a general multimodal sensory API. | +| Learning | `core/src/plasticity.rs` | Strongest positive pre-role→post-role edges, default KC→MBON budget 16,384; gains/eligibility are per-agent. A caller supplies the reward scalar; PAM spikes do not generate it. | +| Parallelism | `core/src/lif.rs` (`SweepPlan`); `core/src/pool.rs` | Persistent deterministic within-brain pool. `broadcast` has one shared job slot and assumes one dispatcher at a time; cloning a plan is not permission to dispatch it concurrently. | +| Game interface | `gb/src/adapter.rs` | `GameAdapter` mixes reward detection, progress, decoder preset, recovery, ROM checks, and Pokémon-like tile/exit/objective queries. `MemoryReader::read8(u16)` is specifically a Game Boy-shaped interface. | +| Emulator | `gb/src/emulator.rs` | Concrete binjgb wrapper, 160×144 RGBA, eight-bit pad, one frame step, audio conversion, native save-state format. Explicit handles and `Send` allow multiple instances; `Sync` is deliberately absent. | +| Session ownership | `sim/src/simloop.rs` (`Sim`, `step_frame`) | One agent, emulator, adapter, ratchet, button mask, framebuffer, audio queue, sugar state, chat ring, and set of clocks. Application orchestration and game behavior share a large struct. | +| Actions | `sim/src/macros.rs`; `gb/src/macros.rs`; `gb/src/pokemon_red/macros/` | Neural channels select available macros, but the game-specific executor owns button sequences. The title screen uses raw input; scene handling, routing and targets are engineered behavior. | +| Recovery | `gb/src/recovery.rs`; `gb/src/ratchet.rs` | Game-only rewind retains brain clock, membrane, RNG, and gains; clears holds/eligibility and refreshes vision. A scalar progress ladder chooses a best save. This is not a general multiplayer reset policy. | +| Persistence | `sim/src/store.rs`; `gb/src/compatibility.rs` | Durable atomic envelope and manifest commit are reusable. Payload is one agent plus one emulator and ratchet. Compatibility names binjgb and `pokered`; native state identity includes size and target. | +| Public observation | `sim/src/snapshot.rs`; `packages/feed/src/{types,codec}.ts` | One flat brain/game snapshot; one attachment per kind; Game Boy buttons, fixed frame dimensions, Pokémon-shaped reward counters and a closed game-mode set. | +| Stage | `apps/stage/src/{App.tsx,feed/store.ts,feed/decode.ts}` | Good hot/paint/cold clock split, but mutable stores/scalers are singletons and the decoder checks 160×144 frames. Dataset URL is fixed to FAFB. | +| Game presentation | `apps/stage/src/games/` | A useful registry already exists, but config relabels v1 counters rather than declaring independent task schemas. | +| Twitch bridge | `services/bridge/src/{index,sim,commands,redemptions,templates}.ts` | Transport/client abstraction, test fakes, templates, rate limits, and redemption persistence are valuable. There is one sim URL and no agent target identity. | +| Packaging | `apps/stage/vite.config.ts`; `infra/build/package-release.sh`; `infra/05-deploy.sh` | Stage build copies FAFB artifacts; packaging defaults to FAFB; deploy preflights one compatibility string. Runtime/data/frontend are still one release composition. | + +### 2.1 Behaviors to preserve before extracting + +`Sim::step_frame` advances the brain from the previous visual input, decodes, applies +buttons, steps the emulator, sets the new visual input, samples rewards, stimulates per +reward event, reinforces their sum, updates macro availability, and observes the ratchet. +Control commands are drained before the frame step. This ordering is part of behavior. + +Do not replace this with `NeuralAgent::tick` merely because it looks like a convenient +wrapper: the service currently orchestrates substeps to sample reward from the frame just +produced. Moving reward or visual drive across that boundary changes trajectories. + +Other invariants: + +- Deterministic arithmetic and per-target propagation order, including across thread counts. +- Browser/network clients never stall the sim; snapshots are latest-value/drop-oldest. +- Checkpoint capture is coherent; encoding and storage happen off the sim thread. +- Failed restore does not silently reset a run. Existing legacy restore policies remain exact. +- No public control endpoint for button presses or game-memory writes. +- Sugar and learning reward are distinct mechanisms. Chat text never becomes neural input. +- The current positive-only reward doctrine remains the default for all shipped tasks. + +### 2.2 Existing compatibility gaps to handle deliberately + +The dataset fingerprint hashes metadata (including anatomical circuit roles) and six arrays. +`macro_*` roles are merged **after** hashing to preserve old checkpoints. Both language +implementations document that changing those populations could restore rates onto different +neurons without invalidating the checkpoint. The kernel parameter version also omits role +names; the service compatibility string does not fully identify the decoder/action mapping. + +Keep these historical behaviors in the legacy reader. For new profiles, add an explicit +behavior identity covering population bindings, sensor encoding, decoder configuration, +action executor, and reward catalog. Do not repair the old hash by changing it in place. + +Open macro/shop and recovery branches existed when this proposal was written. Before +implementation, rebase the inventory against their merged state and rerun characterization; +this proposal neither incorporates nor supersedes their unmerged changes. + +## 3. MaleCNS: anatomy, connectivity, and model are different things + +### 3.1 Dataset comparison and evidence limits + +| Property | FAFB v783 used here | MaleCNS v1.0 | +| --- | --- | --- | +| Specimen | Adult female | Adult male, independently imaged/reconstructed | +| Territory | Brain including optic lobes | Central brain, optic lobes, ventral nerve cord (VNC), intact neck connective | +| Local artifact | 139,255 neurons; 2,700,513 directed-pair edges | None imported in this repository | +| Available inventory counts | Codex lists 139,255 neurons | Codex lists 166,700; the Minecraft project's neuPrint `:Neuron` export reports 176,422. Selection rules must be reconciled before fixing a local count. | +| Input labels | Codex `root_id`, classification, consolidated types, column assignment | neuPrint/flat export `bodyId`/body IDs, class hierarchy, transmitter properties, sides, neuropils, cross-dataset type annotations | +| Added anatomical opportunity | Brain sensory→descending circuits | Brain↔VNC circuits, local motor circuitry, ascending feedback, additional sensory and motor populations | +| License evidence | Repository attribution: CC BY-NC 4.0 | Official MaleCNS download site: CC-BY; verify and retain the exact release license text when importing | + +MaleCNS is not FlyWire with extra neurons appended. IDs and dense array indices do not +correspond. Homologous cell types and registered anatomical spaces enable comparisons, not +automatic one-to-one neuron matching, state transfer, or graph concatenation. Male-specific +and sexually dimorphic circuits make a universal matching assumption especially misleading. + +The official MaleCNS site describes a finished, proofread and annotated CNS reconstruction. +This does not mean every synapse, cell type, or sensory column is equally certain. The +Minecraft derivative reports asymmetric visual-column coverage and missing soma positions; +our import must quantify coverage from its own pinned source. A connectome also does not +supply all synaptic physiology, electrical coupling, neuromodulation, body dynamics, or +behavioral competence. + +### 3.2 What “connections” means + +Separate these quantities in metadata, reports, and on-screen claims: + +1. Source neuron/segment inventory and the chosen included-neuron inventory. +2. Individual synaptic contacts/partner pairs (and separately pre-sites and post-sites). +3. Directed neuron-pair edges after aggregation. +4. Retained edges and retained synaptic weight after confidence/weight filtering. +5. Effective signed weights after the simulation's transmitter and clipping policy. + +The FAFB builder aggregates export rows by `(pre, post)` across rows, drops endpoints not +in its classification inventory, assigns a sign, and clips the summed magnitude. It adds +no explicit five-synapse threshold of its own. The source exports may already be filtered; +the builder cannot recover contacts absent upstream. Codex headline connection counts are +not necessarily the local aggregated graph's edge count. + +The Minecraft project's provenance reports ~25.9 million MaleCNS neuron-pair edges at +weight ≥1 and a bundled derivative of 6,287,749 edges at weight ≥5, representing +90,296,905 of 125,024,863 neuron-to-neuron synapses. It removes 40 autapses. Those are +**that project's reported query/build results**, not counts independently reproduced here. +Its five-contact cutoff retains roughly 24% of edges but 72% of synaptic weight. It is a +performance/modeling choice, not the definition of a complete CNS. + +The official bulk weight table includes **segments**, not only curated neurons. Loading +all rows as if they were all validated neurons would be a different experiment. Likewise, +neuPrint ROI-level adjacency rows must not be summed together with their already-aggregated +totals. Specify one authoritative edge representation and count every exclusion. + +### 3.3 Acquisition and reproducible construction + +Prefer official versioned bulk tables for the repeatable build, with a neuPrint query tool +for inspection and cross-checking. The official download page lists: + +- `body-annotations-male-cns-v1.0-minconf-0.5.feather` — curated annotations. +- `body-neurotransmitters-male-cns-v1.0.feather` — neuron-level transmitter information. +- `body-stats-male-cns-v1.0-minconf-0.5.feather` — segment statistics; large and broader than + the curated neuron list. +- `connectome-weights-male-cns-v1.0-minconf-0.5.feather` — full segment connection graph, + approximately 1.1 GB as listed by upstream. + +The much larger synaptic-point and partner tables are unnecessary for an initial point-neuron +simulation. Download them only for a question requiring synapse-level geometry. Neither +research downloads nor anatomy conversion belong in service startup or ordinary unit tests. + +Proposed build stages: + +```text +release manifest + checksummed source cache + → source-specific parser + → normalized neuron/edge tables + exclusion report + → selected anatomical graph + → model-specific signed-weight transform + → runtime CSR bundle + profile bindings + separate viewer bundle +``` + +Each stage records source release, database revision if queried, query/filter definitions, +source byte hashes, tool revision, and output hashes. Fetch time is provenance, not a random +input to the semantic graph hash. Sort body IDs numerically; encode original IDs as strings +in JSON so this common format also preserves FAFB IDs beyond JavaScript's safe integer range. +Preserve original annotations and cross-dataset type aliases separately from normalized roles. + +The first import report must reconcile the Codex/neuPrint count difference, or explicitly +choose and document one inventory without claiming equivalence. It must also report missing +IDs, unannotated neurons, empty required populations, missing geometry, unknown sides, +transmitter confidence/fallback counts, duplicate edges, autapses, clipped weights, and +retained contacts by region and threshold. + +Use deterministic serialization and gzip headers, following our existing reproducible +builder rather than copying the Minecraft artifact's timestamp-dependent container format. +Keep the source cache outside tracked artifacts; pin published runtime bundles by digest. + +When adding actual data, update `NOTICE`, `LICENSES.md` and bundle-local attribution/license +files in the same change. Keep FAFB-derived assets under their existing terms; a separately +licensed MaleCNS bundle does not relicense mixed fixtures, old goldens or viewer assets. + +### 3.4 Graph and sign policy + +Keep raw positive contact counts and transmitter evidence in the normalized data. Sign is a +model transform, not a measured property that should overwrite the source evidence. + +The legacy policy assigns GABA/GLUT negative and other/unknown transmitters positive; +conflicting per-edge transmitter rows become `MIXED`, then positive. Do not silently extend +that policy to histaminergic photoreceptor input. A MaleCNS policy must explicitly define +histamine, monoamines, mixed/unknown labels, confidence fallbacks, and whether transmitter +is chosen per neuron or per connection. None implies receptor-specific physiology. + +Recommended first profiles: + +- **`malecns-v1-lif-baseline`**: existing kernel, explicitly versioned sign policy and L1 + input mapping, fixed readout, plasticity initially disabled for characterization. +- **`malecns-v1-lif-learning`**: same anatomy/input/readout plus audited KC→MBON selection + and the existing reward rule. Learning is an experimental condition, not an assumed gain. +- Later **sensorimotor research profiles**: photoreceptors, mechanosensation, VNC outputs, + and possibly another neural model, each with its own identity and validation. + +Build weight≥1 and weight≥5 variants as **different graph identities** for the benchmark. +Do not select a production cutoff until activity, retained connectivity, memory and speed +have been measured. Preserve autapses by default in the new canonical graph; if an experiment +removes them, record the policy. The claim that point-neuron models have no use for autapses +is not a reason to discard observed connectivity silently. + +Schema-1 `i16` weights may be sufficient, but measure overflow rather than assume it. For +the first existing-kernel comparison, emit a schema-1-compatible runtime view only if its +quantization/clipping is explicitly reported. If wider weights are needed, implement an +additive format/loader and matching oracle path; do not reinterpret old `weights.binz`. + +Strengthen validation before constructing a network: CSR starts at zero, is monotone, ends +at edge count; targets/roles/visual indices are in range; required populations are present; +geometry is finite where marked valid; array lengths and index widths are representable; +declared hashes and artifact sizes match. Test the same invalid fixtures in both languages. + +### 3.5 Population and sensory mapping + +Existing role predicates cannot be copied unchanged. The current builder recognizes Codex +names such as `Kenyon_Cell`, `brain_motor_neuron`, `DAN` and `PAM*`; MaleCNS uses another +annotation vocabulary. Maintain a reviewed mapping table with source predicates, resulting +counts, hemisphere policy, aliases, and citations. Resolve each profile's required roles +at startup; an absent role must be an unsupported capability, not a silent empty population. + +In particular: + +- **Brain motor ≠ all motor.** Current macro pools combine 96 MBONs and 110 brain motor + neurons. Adding hundreds of VNC motor neurons under `motor` would silently change that + behavior. Use qualified roles such as `brain.motor`, `vnc.motor`, `brain.descending`, + `mb.kenyon`, `mb.output`, and `mb.pam` in new profiles, with legacy aliases only as needed. +- **Action groups are not anatomical facts.** `command_*` buckets use index modulo eight; + `macro_*` groups use a round-robin MBON/brain-motor pool. Move their construction into a + versioned task readout profile, outside the anatomy builder. Do not describe those groups + as natural “attack,” “jump,” or game-objective circuits. +- **Rate budgets are finite.** Both kernels currently cap tracked populations at 64. A + full CNS has many more interesting populations. Select a bounded control/telemetry set + initially; arbitrary bulk population analysis belongs in offline tooling. A larger mask + is a measured, oracle-tested change, not an unbounded string map in the tick loop. +- **L1 first, photoreceptors later.** Reuse the existing luminance projection only after + auditing MaleCNS L1 hex coordinates, both sides, and a declared hex→2D transform. Soma + coordinates are not visual-field coordinates. Missing columns remain explicitly missing; + do not synthesize them from array order or infer them from another specimen's neuron IDs. +- **Input profile and display view differ.** A game image may be resized/cropped for the + agent while the full frame is shown to viewers. Record crop, orientation, color transform, + and sampling geometry. Neither player may accidentally receive another player's private + view or adapter-only task observations. + +The existing `set_visual_frame` and `stimulate` API cannot represent a general collection of +odor/touch/proprioceptive inputs. A later sensory-drive interface needs explicit units, +target populations, additive/overriding rules, tick ordering, and deterministic noise streams. +It must be specified in TypeScript before a matching Rust implementation. Keep the old +image/stimulation path available byte-for-byte through its compatibility facade. + +### 3.6 What MaleCNS lets us investigate + +| Experiment | New capability | What still needs engineering/measurement | +| --- | --- | --- | +| Same game, another connectome | Compare datasets under a matched task interface | Cell-type mapping, gain/activity calibration, readout comparability, multiple seeds | +| Brain↔VNC control | Read actual descending, ascending and motor populations | Body/control mapping; gamepad commands are not muscles | +| Embodied fly arena | World smell/taste/touch/vision mapped into annotated sensory populations | Sensor transduction, proprioception and body dynamics; identify every reflex shortcut | +| Mixed-dataset two-fly match | FAFB and MaleCNS agents share one environment | Balanced observations, controller mapping, compute budgets, intervention rules | +| Circuit perturbation | Compare full graph with VNC feedback or defined pathways ablated | Separate graph identities; activity and behavioral controls; no biological claims from gameplay alone | + +The Minecraft project is a useful engineering comparison, not our validation oracle. It uses +a Shiu-style current-based LIF model with synaptic dynamics/delay, reports gain calibration, +and explicitly supplies odor-approach reflexes and higher-level looming drive where its +simulated pathways do not work. Its source graph, numerical model and embodiment differ +from ours simultaneously. Do not attribute its behavior solely to MaleCNS. + +### 3.7 Identity, restore, and performance + +Name the components independently: + +```text +anatomyId = source release + included inventory + graph/filter digest +modelId = numerical semantics + effective numeric configuration +sensorId = input encoding + anatomical binding digest +readoutId = population partition + decoder + action mapping digest +learningId = rule + selected-edge topology + reward-catalog identity +viewerId = positions/geometry + index mapping digest (presentation only) +``` + +The composite behavioral identity covers all behavior-affecting components; original source +IDs and display geometry cannot replace it. Same neuron count does not establish compatibility. +No FAFB neural checkpoint is restored into MaleCNS. A deliberately fresh MaleCNS brain may +start from a compatible game-only save, with a new run identity, clean calibration and reward +baselining; that is a new experiment, not continuation of the old fly. Gains are not mapped +between specimens by cell-type name. + +For rough capacity planning, the current CSR is `4(N+1) + 6E` bytes, excluding roles, +geometry, derived propagation structures and mutable state. At 176,422 neurons that is +about 38.4 MB for 6.29 M edges, or 156.1 MB for 25.9 M edges (decimal MB). This is roughly +2.3× or 9.6× our edge count, not a prediction of the same slowdown. Spike activity, fan-out, +plasticity selection, memory bandwidth and sharding determine runtime cost. Checkpoint +copies and renderer assets also need separate memory budgets. + +`LifNetwork` accepts shared `Arc` already. Reuse immutable anatomy across +same-profile agents; keep membrane, refractory state, RNG, rates, decoder holds, stimulation, +eligibility and learned gains private. Audit constructor-derived caches before moving them +into a shared topology object. Never share mutable gains merely because graphs match. + +Measure headless one-, two-, and four-agent runs with plasticity on/off, fixed inputs and +representative activity. Record resident/peak memory, initialization, state capture cost, +per-phase p50/p95/p99, spike distribution, and real-time factor. Existing CUDA code is an +optional backend requiring its own new-dataset equivalence/capacity gate, not assumed capacity. + +## 4. Reusable architecture + +### 4.1 Define the nouns first + +- **Dataset bundle:** immutable anatomical graph and source annotations. +- **Brain profile:** dataset plus numerical model, sensory/readout bindings and learning rule. +- **Agent:** one independently stateful brain, encoder, decoder and action executor. +- **Environment:** the world being advanced: one emulator instance, a linked-emulator group, + or an embodied simulator. Owns controller ports, world state, media, and native clock. +- **Task:** interpretation of environment state: rewards, progress, episode endings, allowed + macro actions and recovery policy. Pokémon is a task, not an environment API. +- **Session:** one environment plus agents, port assignments, scheduler, task state and clocks. +- **Application:** composes sessions/components and their presentation; owns supervision, + persistent identities/history, run/intervention rules and application-specific schemas. +- **Bus:** generic RPC/pub-sub routing and artifact ownership, with no game/simulation semantics. +- **Broadcast:** presentation of one or several sessions, plus chat and audience interactions. + +An agent ID is not a Twitch username, controller port, array position or dataset ID. Session, +episode, agent, port, view, and event identities must be explicit and stable across restore. + +### 4.2 Dependency direction + +```text +source importers → dataset bundles + ↓ + neural core (TS oracle / Rust runtime) + ↓ + agent composition: sensors + readout + executor + ↓ +environment backend + task plugin → session runtime → observations/checkpoints + ↑ ↓ + command admission protocol adapters + ↑ ↓ + Twitch bridge stage / recorder +``` + +The neural core knows no emulator, task, network socket, chat, or UI. The environment knows +no neural populations or Twitch. The task may inspect backend-specific state through a +typed inspector, but neither inspection nor public presentation gives clients a memory-write +or controller-write API. Only the session commits agent-produced controls. + +### 4.3 Proposed modules and staged layout + +These are target responsibilities, not instructions to create every package immediately. +Start as modules; extract crates/packages once a second consumer demonstrates the boundary. +Keep the Rust workspace under `services/flysim` during semantic extraction so paths and +behavior do not change together. A later mechanical move can place reusable crates at the +root, updating CI/build/golden paths in one dedicated change. + +| Module / eventual location | Owns | Extraction source | +| --- | --- | --- | +| `packages/brain` | Reference numerical behavior and legacy public facade | Existing package; keep imports compatible | +| `crates/flybrain-core` | Rust numerical kernel, plasticity, generic population decoder | Existing `core/`; leave compatibility re-exports for presets | +| `crates/fly-dataset` | Manifest validation, artifact loading, source-ID/index mapping | `core/src/dataset.rs`; retain legacy fingerprint implementation | +| `crates/flybus` | One Rust RPC/pub-sub client/router, immutable artifact store, delivery guards and GC | New generic library; embedded router or small executable, no separate worker transport | +| `tools/datasets/{fafb,malecns}` | Source-specific conversion to common bundles | Existing Python builder plus new importer; existing CLI wrapper remains | +| `crates/fly-session` | Agent ownership, clock coordination, action commit, event/reward routing | Orchestration extracted from `sim/src/simloop.rs` | +| `crates/fly-environment` | Backend capabilities, ports, observations, media, save-state interfaces | New small contract proven with binjgb and synthetic arena | +| `crates/fly-env-gb` | binjgb FFI, memory inspector and native save-state identity | `gb/src/{emulator,ffi}.rs`, build glue and vendor boundary | +| `crates/fly-task-pokemon`, `fly-task-platformer` | Audited reward rules, semantic state, macros, progress/recovery | `gb/src/{pokemon_red,platformer}/`; do not generalize tile routing into the core | +| `crates/fly-checkpoint` | Atomic storage and session envelope; legacy payload adapter | `sim/src/store.rs` plus core envelope helpers | +| `crates/fly-protocol` / `packages/feed` | Versioned wire schemas/codecs, legacy adapters, synthetic fixtures | `sim/src/snapshot.rs`, existing feed package; canonical schema with cross-language tests | +| `services/flysim` | Composition/config, HTTP/WS, process lifecycle, metrics | Thin host over reusable session library | +| `packages/stage-runtime` | Feed ingestion, per-session stores, paint loop, audio, fixture clock | Extract from `apps/stage/src/{feed,paint,audio,motion}` after multi-view prototype | +| `apps/stage` + presentation plugins | Layout, branding, task panels, audience-facing explanations | Existing page with legacy layout preserved | +| `services/bridge` + audience client module | Twitch transport/auth/redemptions; session-targeted interaction client | Existing bridge; extract provider-independent logic only when reused | +| `infra/` | Release composition, process supervision, capture, recordings | Existing tooling parameterized by session/broadcast manifest | + +Use static Rust composition or a small closed registry initially, with trait boundaries at +backend/task seams. Do not require stable native dynamic-plugin ABI. An out-of-process +emulator helper implements the backend through Flybus RPC, using the same bus as application +events and publication. Native emulator protocols stay inside its adapter. This is not a +new public action API or a reason to maintain a second framework transport. +Keep backend-specific memory access private to its task implementation instead of widening +`read8(u16)` into a supposedly universal game-state abstraction. + +### 4.4 Environment and agent contracts + +Illustrative interfaces; concrete types must be written with tests during contract work: + +```rust +trait Environment { + fn descriptor(&self) -> &EnvironmentDescriptor; + fn observe(&mut self) -> Result; + fn advance(&mut self, actions: &ActionBatch) -> Result; + fn capture(&mut self) -> Result; + fn restore(&mut self, state: &EnvironmentCheckpoint) -> Result<()>; +} + +// One decision boundary, one action per configured controller port. +struct ActionBatch { + session_tick: u64, + ports: Vec, +} +``` + +`descriptor` declares rational step duration, controller schemas, views, audio streams, +save/restore availability, task inspection capabilities and determinism level. Capture and +restore return an explicit unsupported error when unavailable; configuration validates that +the chosen recovery policy can work. `WorldObservation` is a frame-boundary snapshot or +immutable handle, not an object allowing agents to advance the backend. + +Control schemas support digital buttons and bounded analog axes/triggers, with neutral +values, axis ranges, dead zones, and mutually exclusive directions where appropriate. +Preserve the Game Boy mask as one concrete codec. Analog controls need a fixed, versioned +decoder mapping; an 800-ms direction hold is not a sensible default for every fighting game. + +Separate three observation surfaces: + +1. **Agent sensory view:** pixels or declared synthetic senses the profile may consume. +2. **Task inspector:** audited state for reward/macro/episode logic. Access is part of the + disclosed scaffold, not implicitly available to the neural encoder. +3. **Broadcast view:** media and summaries for viewers, potentially richer than either player's + allowed sensory input. + +Readout produces semantic channel activations or continuous signals. A task-local action +executor translates these into port actions, optionally running a selected macro. It is +explicitly resettable/checkpointable and reports selected action versus actual controller +output. Pokémon pathfinding, dialog logic, and objective catalogs stay in its task plugin. + +Task outputs become scoped `RewardEvent`, `Progress`, `EpisodeEvent` and `ActionAvailability`. +Progress is a tagged value (`ladder`, `score`, `match`, `exploration`, or task extension), not +always a scalar rank. Rewards carry recipient agent/team, rule ID, event ID and observation +tick. A task cannot mutate neural state directly; the session routes accepted rewards once. + +An agent-facing API similarly separates `advance_brain(interval)`, `decide(observation, +availability)`, `encode_next(sensory_view)`, and `apply_outcome(rewards, stimulation)`. +The session owns their order. Agents cannot call `Environment::advance`, select another +port, or inspect another agent's mutable state. Start with a concrete LIF agent composition; +introduce a controller trait when synthetic controllers or a second neural model require it. +Test controllers implement the same decision surface but are identified as non-neural agents +in descriptors and experiment records. + +### 4.5 Example composition + +Illustrative configuration, not syntax supported by today's `flysim.toml`: + +```toml +[session] +id = "arena-demo" +environment = "synthetic-arena-v1" +task = "two-player-rounds-v1" +scheduler = "lockstep-v1" +master_seed = 1234 +recovery = "round-reset-keep-gains-v1" + +[[agents]] +id = "fly-a" +port = "player-1" +profile = "fafb-arena-baseline-v1" +sensory_view = "shared-camera" + +[[agents]] +id = "fly-b" +port = "player-2" +profile = "malecns-arena-baseline-v1" +sensory_view = "shared-camera" + +[broadcast] +layout = "shared-match-two-agents" +audio = "world" +audience_stimulation = false +``` + +Resolve profile IDs through a local, digest-pinned registry. Both agents may instead select +the same profile and share immutable topology while retaining independent state. The host +validates unique agent IDs and exclusive port ownership, view accessibility, profile/controller +compatibility, recovery capability and resource budget before starting. An independent-games +broadcast composes two such sessions; it does not misrepresent them as ports in one world. + +## 5. Multiple flies: concurrency is not multiplayer + +### 5.1 Three supported arrangements + +| Arrangement | Ownership / synchronization | First use | +| --- | --- | --- | +| Independent flies in independent games | One session/process each; optional broadcast composition | Parallel streams and experiments; existing deployment pattern generalizes easily | +| Several flies in one game | One environment, several ports, one session barrier | Local fighting/multiplayer games | +| Linked emulator instances | One composite environment owns all instances and link state | A later link-cable experiment; requires cycle-accurate link support, not two independent frame loops | + +For a shared arena there must not be one `Sim` loop per player, each calling `run_frame`. +That advances the world multiple times and gives an ordering advantage to one agent. + +### 5.2 Shared-world step semantics + +For decision boundary `t`, freeze observation `O[t]`, then: + +1. Admit queued audience/operator commands against stable session/agent identities; log their + effective tick. Chat remains presentation state. +2. Each agent advances its brain for the same environment interval, using its previously + encoded sensory input. Its clock remainder and RNG are private. +3. Each agent decodes and advances its action executor using the same `O[t]` task boundary. +4. Barrier: collect all port actions, validate ownership/ranges, and commit one complete batch. +5. Advance the environment **once** to obtain `O[t+1]` and timestamped media. +6. Encode the next sensory inputs, evaluate task events from the completed transition, apply + explicitly routed stimulation and rewards, and compute next action availability. +7. Apply any whole-session episode/recovery transition; capture coherent state and publish. + +Preserve the detailed legacy ordering inside the legacy single-agent composition. New +profiles identify their scheduling semantics explicitly rather than silently adopting a +different reward phase. Use rational environment time and integer substep accumulation for +new sessions; keep the legacy floating remainder arithmetic for old trajectories. The neural +clock may lead environment time by warm-up; persist that offset instead of pretending all +clocks start at zero. Rendering, physics and decision cadence may differ, but the backend +must define their relationship. + +Start with sequential agent evaluation for reproducibility. Then compare parallel agent +evaluation against the same action trace. Cap total worker budget: `agents × brain_threads` +can otherwise oversubscribe the machine. Use private pools for concurrent agents or serialize +dispatch into a pool; the current `WorkerPool` must not be concurrently reused through a +cloned `SweepPlan`. Sharing immutable graph buffers is independent of scheduling workers. + +If one agent is late, the default is to slow the **whole session** and report lag. A crashed +agent pauses/fails the match rather than silently becoming a neutral or scripted opponent. +A realtime external world that cannot pause requires a separate declared deadline/hold-last +policy, dropped-action telemetry and a different determinism claim. Do not hide that policy +inside the environment adapter or use wall-clock completion order as an action tie-breaker. + +### 5.3 Match state and learning + +Every agent has its own seed, calibration, gain vector, eligibility, reward totals and +stimulation cooldown. Derive seeds deterministically from a stored master seed and stable +agent ID; do not use thread scheduling or the default identical seed for every fly. + +For an initial fighting-game task: + +- Both agents receive the same shared camera unless the game has genuine private views. +- Controller-port swaps and seed repeats are part of evaluation; wins alone are confounded + by character, spawn, arena, action interface and side advantage. +- Award positive, explicitly attributed events such as a scored hit/round win; define + damage/self-damage/team attribution and duplicate detection before turning learning on. + A loss need not produce a negative reward; changing reward doctrine is a separate decision. +- Episode transitions may retain learned gains while clearing transient traces, or create + fresh brains for controlled trials. Record which policy was selected. +- Do not select a “best checkpoint” separately for each player in a shared world. There is + one world state. Tournament scores and historical results should not rewind with a match. +- Disable sugar for balanced evaluation. If enabled for a show, target a named agent under + a documented rule and log the intervention; do not call that an uncontrolled fair benchmark. + +### 5.4 Checkpoint and recovery semantics + +Distinguish three operations: + +| Operation | Restored/reset state | +| --- | --- | +| Crash resume | Coherent environment, every agent, scheduler remainders, task ledgers, pending actions/commands and executor state at one boundary | +| Task recovery | Policy-defined world rewind/reset and per-agent transient clearing; continued gains/brain clocks only when declared, as in the legacy ratchet | +| New episode | Task initial world state, explicit retained/fresh agent policy, new episode identity; session event sequence remains monotonic | + +Use a new session envelope version with a manifest mapping stable agent IDs to chunks and +listing graph/model/profile/backend/content/task/state-format identities. The current +envelope restricts chunk names to letters; do not simply append `agent/1/membrane` to it. +Specify a new container format or an explicit manifest-to-valid-chunk-name indirection. + +Validate every participant into staged state before mutating any live participant. A failed +environment import must not leave half the brains restored. For an external emulator, +restore a replacement stopped process when transactional in-place validation is impossible. +Capture at the action barrier with no backend step in flight. Reuse atomic payload write, +manifest commit, hot/durable tiers and off-thread serialization; bound outstanding snapshot +jobs so repeated copies cannot exhaust memory under slow storage. + +Exact replay requires action-executor and admission state, not just the neural envelope. +Legacy macros intentionally discard transient execution on restart; preserve that behavior +for v1 and label it as legacy continuation semantics, not exact session replay. New sessions +persist all behavior-affecting state or explicitly restart an episode under a documented rule. + +Keep `FLYSIM01` readable through a legacy adapter. Never silently rewrite a checkpoint on +read. Conversion is an explicit offline operation writing a new directory and run record. +Environment identity includes title/content digest, backend build/configuration, relevant +platform/native state format, and patch/symbol provenance; state size alone is not sufficient. + +## 6. Feed, stage and audience modules + +### 6.1 Feed v2 is required + +V1 is not a generic multi-agent format. Its attachment map forbids duplicate kinds, so two +`spikes` arrays cannot coexist; its button mask and 160×144 image are Game Boy-specific. +Platformer counters are currently folded onto names such as `pokedex` and `wildwin`. +Extending those conventions to fighting games would preserve syntax while losing meaning. + +Keep v1 stable for the legacy session. Introduce a negotiated v2 or a separate `/v2/feed` +endpoint, with a descriptor delivered before dependent snapshots and available on reconnect. +This is the application's browser/presentation gateway over Flybus, not another internal +bus. Native participants use the same RPC/pub-sub protocol for all framework communication. +The contract change must update Rust, TypeScript, schema, fake service, fixture player, +stage and bridge together. Proposed shape: + +```text +SessionDescriptor + sessionId, protocol, descriptorRevision, environment/task identities, clock definition + agents[{agentId, profileId, datasetId, neuronCount, roles, controlPort}] + ports[{portId, controllerSchema}] + views[{viewId, dimensions, pixelFormat, sensory/broadcast use}] + audioStreams[{streamId, sampleRate, channels}] + assets[{id, contentHash, datasetIndexHash, localUrl, license/credit}] + +SessionSnapshot + descriptorRevision, seq, sessionTick, episodeId, environmentTime, wallTime, status + agents[{agentId, brainTime, rates, learning, selectedAction, actualControls, stimulation}] + progress: tagged task payload; events: scoped and sequenced + attachments[{id, kind, ownerId, byteLength, format, mediaTimestamp}] +``` + +Use bounded, schema-validated tagged payloads/namespaced task extensions, not arbitrary +unlimited JSON or executable server-supplied UI. Dataset identity and neuron index mapping +must accompany spike geometry: matching bitset length alone cannot establish alignment. +Include descriptor revision in every snapshot and reject stale/mismatched buffers. Large +monotonic IDs/times use a specified safe-integer bound or decimal strings across languages. + +Publish one shared camera/audio stream once, not once per agent. Independent sessions may +have independent streams. Timestamp audio/video to the session clock; define discontinuities +on reset, reconnect and lag. Latest-value video/telemetry may drop, while audio needs a +bounded timestamped buffer and explicit gap handling. Durable event IDs permit recovering +missed events; a drop-oldest snapshot feed is not an exactly-once event log. + +At 640×480 RGBA, native frame production is 36.864 MB/s at 30 Hz or 73.728 MB/s at 60 Hz. +Publish one immutable artifact and pass owned references through Flybus to all consumers. +Bytes stay outside router messages; producer/read/copy costs still need measurement. A +renderer retains its handle past message drop; cached RPC results and latest retention also +own data until release. Last-owner GC replaces coordinator-managed slots/reader acknowledgments. +Resizing, overlays, codecs and streaming are presentation-layer choices, not bus requirements. + +### 6.2 Stage composition + +Extract `createSessionStore()` rather than adding `agent2` fields to the singleton `hot`. +Own rate scalers, button afterglow, ticker state, fixture clock and audio queues per session/ +agent. Keep one page paint scheduler; register surfaces against explicit view/agent IDs. +Geometry is loaded by descriptor/hash through an asset manifest, replacing the hardcoded +FAFB route in both `App.tsx` and `vite.config.ts`. + +Retain the existing Game Boy layout as a presentation plugin. Add composition primitives for +a shared match view with two agent summaries, or independent session tiles with one focused +audio source. A generic fallback shows status, media, controls and task labels without +inventing Pokémon counters. Unknown optional extensions can be omitted; unknown required +capabilities or mismatched descriptors must be visible rather than silently showing Pokémon. + +Combine common framework measurements with application-owned state/events. Describe values +by owner, type, units, range and timestamp; distinguish zero, unknown, unsupported and stale. +Game-specific progress/collections remain validated schema extensions. The application defines +its supervisory/story behavior alongside its UI, not inside a mandatory generic Director service. + +Keep UI runtime dependencies out of the numerical package. The current `@flybrain/brain` +view exports and optional Three.js peer can remain compatibility re-exports when viewer +geometry/helpers move to a dedicated view module. Controller labels belong in controller +schemas, not a UI import of the neural package's Game Boy preset. + +Review actual layout proposals as PNGs under `apps/stage/mockups/`, with existing legibility, +phone-scale and browser gates. This document proposes data/layout boundaries, not screen +copy or a replacement for visual sign-off. + +### 6.3 Audience interaction + +Keep Twitch authentication/EventSub and template-only replies in the application bridge. Extract a +session-targeted interaction client with explicit `sessionId`, optional `agentId`, interaction +kind and idempotency key. A multi-agent request with no target is rejected unless a fixed, +declared target policy exists; presentation focus must never choose the recipient. + +Service admission owns per-session, per-agent and global limits. A profile advertises its +supported stimulation capability; an agent without PAM support returns unsupported rather +than pretending to accept “sugar.” `!stuck` becomes task-aware (ladder time versus round +time), while chat remains broadcast-scoped and independent of neural state. Future boons +use task/backend-declared capabilities with explicit target, timing and outcome; gifts do +not automatically become earned learning rewards. These need separate capability/admission +contracts before enabling them, carried over the same bus rather than an arbitrary write API. + +For redemptions, persist the resolved target and request identity before retrying. Distinguish +an RPC timeout (HTTP on legacy v1) from a definite refusal: it may occur after the sim accepted the +effect. V2 needs a durable or explicitly recoverable deduplication/status contract so bridge +restart cannot apply the same pulse twice or retarget a redemption to a new match. Do not +promise exactly-once behavior from the bridge intent log alone. Existing v1 remains as-is. + +### 6.4 Deployment and observability + +A deployable composition selects runtime binary, backend/task, agent profiles, dataset +bundles, view assets, session state namespace and broadcast layout. Generate service config +from that manifest plus the operator's external environment. Keep tokens and network-specific +values outside this repository. Package viewer artifacts separately from the full simulation +graph so the browser does not need every edge. + +Preserve existing process isolation: sim, bridge, browser, capture, local relay and Twitch +push can restart independently. One coordinator/session with separate worker processes is +the default composition; a shared match remains one logical failure/recovery group across +those workers. A worker restart cannot silently rejoin. Router failure also invalidates its +ephemeral handles/routes. Multi-session placement and presentation composition are application +deployment concerns; router scope sets an explicit shared-failure boundary. + +Metrics distinguish session lag, environment step cost, per-agent step cost, barrier wait, +snapshot drops, audio discontinuities, checkpoint queue age and resource budget. Bound agent +labels to configured IDs; never label metrics by viewer name or arbitrary event text. +Health must distinguish paused, slow, disconnected backend and dead agent. A generic +watchdog cannot treat “no new Pokémon tiles” as a stall detector for all tasks. + +Deployment preflight checks every referenced profile/bundle and complete checkpoint identity +before selecting a release. Switching the binary back does not convert newer state: retain +the previous release's state namespace for rollback. Twitch publishing still requires the +operator's explicit approval for that run; implementation benchmarks use local sinks. + +## 7. Cleanup strategy: extract, then reorganize + +Prioritize coupling that prevents a second application, rather than renaming everything. + +1. **Characterize behavior and identify state owners.** Capture deterministic synthetic + observation→action→reward traces and legacy restore outcomes before moving code. +2. **Remove task construction from anatomy.** Introduce separately hashed readout bindings; + leave the current artifact and macro-role merge path frozen for legacy compatibility. +3. **Extract orchestration from transport.** A session step returns observations/events and + capture requests; it does not serialize HTTP/feed headers. `flysim` owns listener setup, + wall-clock publication and systemd integration. +4. **Split emulator from task.** Move binjgb behind an environment implementation, preserving + FFI/cache behavior. Move map/exit/objective queries into Pokémon task interfaces rather + than forcing every future game to implement them. +5. **Split task recovery from storage.** The ratchet decides a task transition; storage + commits an opaque coherent capture. Matches use round resets, not milestone archives. +6. **Version observation/control at the boundary.** Internal typed session snapshots become + v1 or v2 through adapters; core modules do not depend on wire enums. +7. **Make stage state instantiable and assets descriptor-driven.** Preserve hot/cold cadence + and fixture determinism while removing singletons and hardcoded dataset selection. +8. **Only then move directories/extract packages.** Keep re-exports/CLI wrappers during the + move, fix build/CI/vendor paths, and compile tiny consumers proving Rust/TS libraries can + be used without starting Twitch, a browser, or an emulator. +9. **Reconcile documentation.** Separate current contracts/reference from dated deployment + history. Audit contradictory “raw buttons only,” weighted-macro, throughput, token/setup, + and training-improvement claims against code. Update templates and scientific limitations + with actual profile capabilities, not a new generic claim of biological fidelity. + +Do not introduce a generic reward engine, universal memory address model, central plugin +marketplace, or per-neuron network transport. Those abstractions have no demonstrated second +consumer and would obscure the useful, small seams already present. + +## 8. Implementation plan and acceptance gates + +Each row is a reviewable change or small workstream, not one large feature branch. Follow +the repository's branch/worktree build-and-review workflow. Contract changes precede their +consumers. No estimate here assumes an emulator backend or biological mapping already works. + +| Phase | Deliverable and principal files | Dependencies | Acceptance / stop condition | +| --- | --- | --- | --- | +| P0 — baseline | Trace fixtures and boundary tests around `Sim::step_frame`, restore, dataset loader, stage singleton behavior; current-state documentation inventory | None; reconcile open branches first | Current TS/Rust goldens and legacy API/feed fixtures pinned; behavior ledger distinguishes intentional transient reset from exact replay | +| P1 — identities | Dataset/profile manifest, role resolver, behavior hash, synthetic fixtures; new strict validators in both languages | P0 | Missing/ambiguous roles fail; wrong profile refuses restore; FAFB legacy fingerprints/version strings unchanged | +| B1 — Flybus | Small Rust router/client with RPC, pub/sub, owned artifacts and GC; see BUS-01..03 in the contract implementation guide | Can start alongside P0/P1; no dataset/emulator dependency | Same semantics in-memory/Unix sockets; cache/retention holds safe; 480p three-reader test; no raw binary in message envelopes | +| M1 — MaleCNS import | `tools/datasets/malecns`, source lock, inventory/exclusion report, schema-compatible baseline bundle, attribution | P1 | Counts reconcile to selected inventory; reproducible output hashes; graph invariants and both loaders agree; no runtime downloads | +| M2 — headless characterization | Profile-specific L1/sensor binding, role/readout audit, learning-off/on benches and new goldens | M1 | Stable/finite activity measured over repeated seeds; no silent empty populations; exact TS/Rust state agreement; unsupported inputs remain marked unsupported | +| M3 — task integration | Explicit MaleCNS profile selected by service config and matching stage assets, fresh state namespace | M2 and minimal descriptor/asset support from P5 | Fresh brain on audited game state; sugar capability verified; stage index/geometry identity correct; one-hour local soak plus restore drill; no automatic promotion over FAFB | +| P2 — environment/task split | Environment contract, binjgb wrapper, task-specific inspector; facade for existing `flybrain-gb` imports | P1 | Legacy action/reward traces identical; ROM-free tests pass; optional ROM-backed sample confirms stepping/audio/state behavior | +| P3 — session library | Single-agent session owns clocks, agent/task/executor state; service composes bus-connected workers | P2, B1 | Same legacy order and compatibility; renderer/bridge restart leaves run intact; fake backend runs without binjgb/ROM/Twitch | +| P4 — multi-agent + persistence | Two-agent synthetic arena, action barrier, isolated state, session envelope/recovery and failure behavior | P3 | One world step per batch; no port/order advantage; resume/parallel-vs-sequential equivalence; failure cannot partly commit a match | +| P5 — feed/control v2 | Descriptor, scoped snapshots/events/media, targeted stimulation and idempotency, TS/Rust schema fixtures/fake server | P1, P3; validate with P4 fixture | V1 still works for legacy; multi-agent attachments cannot collide; unknown target/profile rejected; retry/reconnect tests pass | +| P6 — modular stage/bridge | Instantiable stores, dataset assets, shared/independent views, target-aware commands/redemptions | P4–P5 | PNG review for two-agent layout; browser legibility/fixture tests; correct audio ownership and no agent cross-talk | +| E1 — new emulator spike | Select title/backend; bus-connected helper or native wrapper; record native frame, ports, inspection and restore capabilities | P2, can run beside P4–P6; framework integration uses B1 | Reliable bounded step + simultaneous controls, pinned content/backend identity, reproducible state round trip; stop before task implementation if unavailable | +| E2 — fighting-game vertical slice | Two flies, chosen game task, analog/digital readout, episode logic, attributed rewards, match view | E1, P4–P6 | Repeated local matches and side swaps; measured compute headroom; documented scaffold and interventions; win-rate claims require controls | +| P7 — packaging/reorg | Optional root Rust workspace move, library consumers, profile-based release/asset manifests, updated infra and docs | Useful second backend + P6 | Four merge suites and affected browser gates pass; old composition deploys locally; preflight rejects incompatible multi-agent state | + +M3 may use a small v1-compatible **additive descriptor extension**, if contracts and both +consumers are updated and the existing frame semantics stay unchanged. It must not publish +MaleCNS spikes under implicit FAFB geometry. Full multi-agent publishing still requires v2. + +### 8.1 A concrete first alternative-emulator spike + +Before committing to Melee/Dolphin or an N64 backend, establish: + +- An exact title/version and backend revision, available to the operator externally. +- A supported pause/advance boundary with all configured controller ports applied together. +- Whether rendering is required for stepping and whether frame capture is synchronous. +- Sample rate/channel metadata, audio latency and timestamps. +- Analog sticks/triggers and button semantics; neutral state on disconnect. +- Save-state completeness, version/platform constraints and reproducibility after restore. +- Supported task-state inspection (match/round/port state) without guessing memory offsets. +- Headless/runtime packaging, process lifecycle, resource cost and failure behavior. + +A library used for competitive tooling may expose controller and game-state APIs without +supporting arbitrary frame stepping or faithful visual capture. Verify capabilities instead +of assuming its name solves integration. Prefer a pinned private backend process if native +embedding would force emulator internals into our Rust runtime. Desktop keyboard automation +is unsuitable for simultaneous deterministic multi-port input. + +Start with synthetic constant/alternating controller traces and a test opponent before neural +control. Such traces are backend tests, not public “fly playing” footage. Then validate one +agent, two agents, episode boundaries, crashes and resume in that order. No copyrighted game +content is added to source control or test fixtures. + +### 8.2 Validation matrix + +**Numerical/format:** existing `core/tests/golden_{toy,real,agent,restore,platformer,versions}.rs` +remain gates. Add pinned MaleCNS subgraph fixtures and a full-artifact optional golden run, +with generated TS goldens and exact Rust comparisons. Include malformed CSR, missing roles, +wide IDs, hemisphere/coordinate errors, overflowing weights, and graph/profile mismatch. +The subgraph tests prove arithmetic/loader agreement, not full-CNS dynamics. + +**Scheduler:** synthetic backend asserts one advance per complete batch; swap agent evaluation +order, vary worker count, and inject a late/failing participant. Check equal observation +boundaries, deterministic seeds, no cross-agent gains/holds/stimulation, correct tick remainder, +and no reward twice at an episode boundary. Mixed profiles are allowed only if their clocks +and capabilities satisfy the same session contract. + +**Persistence:** kill/fault injection around capture/write/manifest commit; corrupt one agent +chunk, backend state or profile hash; verify all-or-none restore and fallback reporting. +Compare uninterrupted and resumed new-session action/state traces. For legacy runs compare +against the documented transient-reset behavior instead of demanding a newly invented one. + +**Protocol/UI/bridge:** cross-language v1/v2 fixtures; two agents with different neuron counts; +shared and private views; out-of-order descriptor/media, missing optional attachments, stale +snapshots and reconnect; replay seeking; duplicate redemption, lost HTTP response and bridge +restart; explicitly unsupported stimulation. Visual changes require PNG review and browser +tests, not prose approval of a hypothetical layout. + +**Performance/science:** benchmark full graph and thresholded graph with learning disabled +and enabled, fixed sensory traces, multiple seeds and side swaps. For game-performance +claims compare against random/readout baselines and learning-off, reporting scaffold, +recovery, intervention and episode policies. Measure sustained real-time factor and tail +latency under two/four flies plus actual browser/capture load; do not extrapolate a single +kernel throughput figure to an entire show. Initial target is ≥1.0× sustained at the declared +agent count with p99 step time within its cadence budget and no growing queues; select a +resource/headroom margin from the measured backend before release. + +Before each merge run the repository-required `npm test`, `npm run typecheck`, +`cargo test --workspace` from the Rust workspace, and `infra/tests/lint.sh`. Run affected +Playwright/PNG gates for stage changes. The existing committed FAFB real-data goldens remain +mandatory. New full-MaleCNS integration jobs and ROM-backed tests are explicit optional jobs +with recorded skips, never a hidden network/ROM dependency of normal CI. + +## 9. Decisions and open questions + +**Recommended decisions now:** preserve FAFB as the baseline; use official MaleCNS provenance; +freeze legacy arithmetic/identities; version profile behavior independently; make environment +and agent separate objects; use one session barrier for a shared match; retain static plugins +and process/session isolation; introduce v2 instead of stretching Game Boy fields indefinitely. + +**Questions answered by spikes rather than assumptions:** + +1. Which MaleCNS neuron inventory and confidence/threshold policy will be the published bundle? + Can we explain the differing inventories and quantify left/right sensory coverage? +2. Do existing LIF parameters give useful, stable activity on MaleCNS? If not, which explicit + profile calibration is justified, and does a different neural model warrant separate work? +3. Are verified L1 mappings adequate, or is the intended project really an embodied sensory + simulation requiring new encoders and VNC feedback? +4. Which Smash title/backend can satisfy deterministic stepping, simultaneous ports, media + capture and restoration at acceptable cost? +5. How many simultaneous brains fit the actual budget, with which mix of within-brain versus + between-brain workers? Is a compressed media path required? +6. Which match reset/learning/intervention policy defines the show, and which defines a + controlled comparison? They should be separate run configurations. + +Success is not just “another connectome loads” or “a second pad moves.” It is a new session +assembled from modules whose anatomy, numerical model, controller, task, recovery and +presentation assumptions are explicit, testable and reusable without changing the old fly. + +## 10. Sources and scope of the analysis + +Repository evidence is enumerated in section 2 and tied to the baseline commit above. Existing +reference documents: [dataset format](../dataset-format.md), [model](../model.md), +[plasticity](../plasticity.md), [readout](../readout.md), [limitations](../limitations.md), +[macros](macros.md), [architecture tour](../architecture-tour.md), and +[contribution/compatibility rules](../../CONTRIBUTING.md). Historical status notes are not +evidence that an unmeasured experiment succeeded. + +External sources inspected 2026-09-18: + +- [Official MaleCNS overview](https://www.janelia.org/project-team/flyem/male-cns-connectome): + anatomical coverage, collaboration, release dates and licensing statement. +- [Official MaleCNS downloads](https://male-cns.janelia.org/download): versioned bulk tables, + confidence cutoffs, segment versus neuron distinction, coordinate units and API guidance. +- [FlyWire overview](https://flywire.ai/): FAFB reconstruction provenance and brain coverage. +- [Codex dataset listing](https://codex.flywire.ai/): portal inventory counts, which are not + assumed to be identical to neuPrint query inventories or local runtime graphs. +- [Minecraft fly README](https://github.com/blendi-remade/fly-brain-minecraft/blob/main/README.md) + and [provenance](https://github.com/blendi-remade/fly-brain-minecraft/blob/main/PROVENANCE.md): + a separately authored MaleCNS derivative, thresholding and mapping decisions, and disclosed + sensory/motor limitations. These moving links are comparison material, not a locked data + dependency; M1 must acquire its own official source lock. + +This analysis reads the current implementation and upstream documentation. It does not +download/build the full MaleCNS dataset, independently validate the Minecraft benchmarks, +run a new emulator, or establish a performance/behavioral improvement. Those are explicit +deliverables with gates above. diff --git a/docs/design/melee-framework-audit.md b/docs/design/melee-framework-audit.md new file mode 100644 index 0000000..db57f24 --- /dev/null +++ b/docs/design/melee-framework-audit.md @@ -0,0 +1,766 @@ +# Melee: multi-fly runtime, emulator and broadcast audit + +Status: **research and proposed implementation plan**. Written 2026-09-18 against local +`f7bc13a`. Extends the [modular-session design](malecns-modular-sessions.md) and +[implementation backlog](malecns-modular-implementation.md) with a concrete second game. +Existing [feed](../feed-protocol.md) and [control](../control-api.md) contracts still win. +No emulator, game image, deployment host or live broadcast was run for this audit. + +Follow-on [session framework contracts](session-framework/README.md) define the concrete +multi-process synchronization and worker interfaces this backend will implement. +The subsequent [Flybus decision](session-framework/bus-v1.md) selects one small Rust router +for RPC and pub/sub, with immutable artifacts outside messages and delivery-scoped GC. +The application owns supervisory/presentation behavior; the bus owns no game or stream logic. + +## 1. Executive decision + +**Use Dolphin, initially a pinned mainline-based Slippi Dolphin build, as a separate backend +process. Use the maintained libmelee fork to accelerate the integration spike. Keep the +brains and session coordinator in Rust.** Prove its synchronization, rendered sensory input, +and recovery behavior before selecting a production build. Keep stock Dolphin plus a narrow +backend hook as the fallback if the Slippi path cannot satisfy those requirements cleanly. + +Do not port Melee to native code, embed Dolphin into the neural crate, run one emulator per +fighter, or assume “libmelee has `step()`” supplies the complete environment contract. + +Recommended first target: + +- Melee US 1.02, local two-player versus, one emulator and two independently stateful flies. +- Existing FAFB brains first. MaleCNS is an independent axis of experimentation, not a + prerequisite for solving the emulator and multiplayer problems. +- Fixed declared characters, stage, stock/time rules and input profiles; no netplay rollback. +- Pixel-based sensory input with an explicit resolution/aspect transform; state inspection + is for task measurement, match lifecycle and display, not a hidden fighting policy. +- Fixed GameCube controller readout with bounded analog values and short frame-based holds. +- Native-resolution rendering initially, local broadcast at 30 fps first; simulation/input + continue at the backend's approximately 60-Hz cadence. Promote to a 60-fps show only after + the full media path and encoder are verified. +- Two-fly synthetic arena remains the framework test case before game integration. + +The important scaling change is **one environment with many agents**, not simply a larger +ROM. Disc size is mostly a loading/storage concern. Runtime cost comes from PowerPC emulation, +graphics/audio, multiple neural simulations, synchronization and media copies. + +### Confidence labels used below + +- **Observed in source:** checked implementation or explicit upstream documentation. +- **Recommended:** proposed architecture/configuration, not implemented here. +- **Must measure:** cannot be established by reading code, including throughput, latency, + correct input-to-frame association and reliable recovery on the target platform. + +## 2. What the Melee decompilation gives us + +The supplied [doldecomp/melee](https://github.com/doldecomp/melee) repository is a matching +decompilation of **US 1.02**. Its README is `.github/README.md`, not the repository-root +`README.md`. The inspected revision is recorded in section 13. + +The README and `config/GALE01/config.yml` identify the matching `main.dol` SHA-1 as +`08e0bf20134dfcb260699671004527b2d6bb1a45`. That identifies the executable, **not the entire +disc image**. Our run manifest must separately identify externally supplied game content, +effective executable, modifications, emulator build and task interpretation. + +The decomp builds a GameCube executable, not a supported desktop port or emulator replacement. +The `dolphin` code within that source tree refers to Nintendo's SDK, not the Dolphin emulator +project. Rebuilding/relocating a DOL for instrumentation changes address identity; never use +stock symbol addresses on a shifted build. + +### 2.1 Useful inspection map + +| Upstream source | What was observed | How it helps our task | +| --- | --- | --- | +| `config/GALE01/{config.yml,symbols.txt}`; `docs/symbols.md` | Matching binary identity and named symbols with addresses, sections and attributes | Reproducible symbol/inspection manifest analogous to the Pokémon symbol generator | +| `src/melee/pl/player.h` and `player.c` | `StaticPlayer`, getters for stocks/damage/controller index, KO-by-player counters and self-destructs | Distinguish controller port, player slot and match attribution instead of assuming they are identical | +| `src/melee/ft/types.h` | `Fighter`, player/controller identifiers, buffered sticks/triggers/buttons, pressed/released edges, damage state, source-player field and move-instance information | Audit controls and potential reward attribution; source fields are hypotheses to validate against live transitions | +| `src/melee/gm/types.h` | Match frame/timer fields, `MatchEnd`, winner arrays and exit/results structures | Episode boundaries, timeout/results interpretation, one-time terminal rewards | +| `src/melee/gm/gmvsmelee.h` | Character/stage select, versus entry/exit, sudden-death and results transitions | An explicit lifecycle model instead of treating every screen as a playable frame | +| `src/melee/{cm,gr,mp,it}/` | Camera, stages, map/collision and item subsystems identified by upstream module structure | Follow-up inspection points for view geometry, hazards and projectiles; not all audited in this pass | + +Examples of concrete distinctions: + +- `StaticPlayer` has a controller index, a player ID and up to two sub-fighter entities. + Ice Climbers and transformations defeat “one visible fighter object = one agent.” +- The input struct tracks three-entry analog/button histories plus pressed/released buttons + and threshold timers. A constant button hold and a sequence of taps are different actions. +- Damage includes an annotated source-player number, but it must be checked for projectiles, + stale ownership, self-damage and indirect KOs before it becomes a reward source. +- The match structures include winner counts/arrays. A terminal result is not safely inferred + from whichever player's stock decrement happened to be sampled first. + +### 2.2 How to use it without making the framework Melee-specific + +Create a **task-local** inspection catalog: field name, binary/decomp revision, symbol or +pointer traversal, type/endianness, valid scenes, tested transitions and unsupported cases. +Prefer Slippi telemetry for fields it supplies reliably; use decomp-grounded inspection for +missing fields only after verification. Do not expose all emulator memory to every agent. + +If memory inspection is needed, decode big-endian integer/float fields and emulated 32-bit +pointers explicitly. Never cast emulated bytes to a native Rust/C struct whose layout, +pointer width or bitfield ordering is different. Sample a consistent backend boundary rather +than reading a moving process asynchronously. Accessors named in the decomp explain semantics; +they are not functions our host process can call in place of an adapter. + +Derive small constant/schema outputs and synthetic fixtures where appropriate. Do not vendor +the entire decomp, game executable, assets or save states merely to read stocks and damage. +The first working backend does not require rebuilding Melee. Custom hooks or patches are +separately identified scaffold and enter the run's behavior/content identity. + +## 3. Emulator options and recommendation + +All candidates below emulate GameCube software. The differentiator is the host integration, +not whether Melee can theoretically boot. + +| Candidate | Evidence / useful capability | Gap or cost | Decision | +| --- | --- | --- | --- | +| **Mainline-based Slippi Dolphin + maintained libmelee** | Structured game/port events, controller pipes, explicit blocking-input support; Linux rendered path available in the ecosystem | Pixel/audio access and coherent external save-state control still need integration; pin emulator, Gecko codes and parser together | **First spike and preferred initial integration** | +| **Stock Dolphin + narrow host hook** | Source has `Core::DoFrameStep`, CPU-thread coordination and `State::{SaveToBuffer,LoadFromBuffer}` | These are internal APIs, not a stable remote environment SDK; own a small patch and state parser/telemetry bridge | Fallback or eventual generic Dolphin backend if its maintenance cost is justified | +| **Felk Python-scripting Dolphin branch** | Inspected stubs expose controller/memory/save-state scripting and rendered-frame events | Historical branch; `await frameadvance()` is documented as waiting for a rendered-frame event, not proof of a paused one-step transaction | Research reference or temporary probe, not default production dependency | +| **Custom EXI/fast-forward Slippi-Ishiiruka** | Maintained libmelee README describes accelerated ML mode and EXI inputs | That documented fast path disables rendering; inspected libmelee rejects non-Null graphics for the EXI_AI build | Useful for explicitly state-driven offline research, not the pixel-fed live baseline | +| **Libretro Dolphin core** | Potential common frontend ABI | No core-specific synchronization/state/render benchmark was performed; another integration layer does not remove task semantics | Defer rather than introduce an unverified second dependency stack | +| **Native game port built from the decomp** | Source enables modding/research | Matching DOL compilation is not native execution; graphics, OS/SDK, timing and assets remain substantial work | Outside this project's first Melee phase | + +Use the maintained **`vladfi1/libmelee`**, not a floating install selected by an old tutorial. +`altf4/libmelee` says it is archived and points there. The maintained fork says it became +the PyPI `melee` source starting at 0.45.0; pin the actual chosen package/source revision and +its dependencies instead of assuming an unversioned `pip install melee` reproduces a run. + +The maintained README describes raw-state compatibility, but the inspected `Console.step()` +still invokes `__fixframeindexing` and `__fixiasa`. Therefore, confirm field semantics from +the installed source and observation fixtures rather than trusting README wording. Our +adapter identifies parser/normalization revision as part of task identity. + +### 3.1 What the blocking path actually does + +**Observed in source:** + +1. `libmelee.Console` defaults `blocking_input=False`. Setting it true writes + `Slippi/BlockingPipes` for its mainline backend. +2. `Console.step()` flushes its registered controllers and then dispatches game/menu events + until a frame boundary. It is not a method returning RGBA pixels or arbitrary game state. +3. Slippi's `EXI_DeviceSlippi.cpp` sets `g_need_input_for_frame` on game setup, menu frames + and frame bookends. +4. `Pipes.cpp::UpdateInput` checks the blocking setting and flag, waiting for commands through + `FLUSH`. Its Linux wait path uses `select`; the inspected Windows wait helper is not implemented. +5. `ControllerInterface::UpdateInput` updates devices and only then clears the flag. The + `FLUSH` handler explicitly avoids clearing it before the other devices have been read. + +That is strong evidence for trying a Linux multi-port blocking backend. It is **not yet a +measurement** that action batch `t` affects exactly our desired frame `t+1`, that menu/game +boundaries behave identically, or that rendering/audio are coherent with the telemetry. + +Keep at most **one batch outstanding**. The pipe implementation consumes buffered commands, +and a backlog of frame batches must not collapse into a latest-state input. Create an isolated +Dolphin user directory containing only the intended pipe devices: unused/abandoned devices +can participate in updates and leave blocking input waiting on a controller nobody drives. + +For two flies, stage both full controller states before calling the single owner's +`Console.step()`. Do not give each agent a `Console` loop. Verify which input sample corresponds +to returned pre/post-frame telemetry with distinguishable action pulses and deliberate delays +on each port. “Both bots sent commands” is weaker than “both commands landed on one frame.” + +### 3.2 What libmelee does not establish for us + +- No framebuffer-returning or whole-session save/load interface was found in the inspected + `Console` API. `DumpConfig` configures media dumping; a dump is not automatically a bounded, + timestamped sensory-frame transport. +- GameCube pad values are stateful. Unchanged buttons remain held; the backend must emit an + explicit complete state or compute trustworthy deltas, including release/neutral values. +- The library applies analog normalization (`fix_analog_stick`, `fix_analog_trigger`). Define + our canonical ranges and apply conversion exactly once; test round trips at neutral, + extremes, diagonals, dead zones and trigger-click thresholds. +- Rollback skipping and internal controller flushes are present. Use local offline matches + first; do not confuse filtered rollback frames with advancing brains through speculative time. +- Initial game events can flush neutral input internally. Record startup as lifecycle scaffold; + do not attribute that to a neural decision. +- Spectator transport keepalive, pipe blocking and process liveness are separate. A healthy + connection does not prove the match or renderer is advancing. + +## 4. Audit of our current system: keep, extract, replace + +Rust path aliases: `core/`, `gb/`, `sim/` mean the respective crates under +`services/flysim/crates/` named `flybrain-core`, `flybrain-gb`, and `flysim`. + +| Finding | Current evidence | Required change for Melee/multiple flies | Priority | +| --- | --- | --- | --- | +| Single world and brain bundled together | `sim/src/simloop.rs::Sim` owns one `NeuralAgent`, concrete `Emulator`, adapter and ratchet | Session owns one environment plus agent collection and explicit port map | Blocking | +| Direct Game Boy calls in frame loop | `step_frame`, `to_button_mask`, `set_buttons(u8)`, `run_frame`, fixed framebuffer copy | Backend interface with full action batch, rational cadence, observations and media capabilities | Blocking | +| Task trait carries Pokémon concepts | `gb/src/adapter.rs` includes `read8(u16)`, map/exit/objective hooks | Keep inspector and macros task-local; use generic reward/episode/progress outputs | Blocking | +| Timing defaults embed Game Boy | `core/src/agent.rs`, `sim/src/config.rs` | Session clock derived from backend, per-agent neural remainders; preserve old arithmetic in legacy facade | Blocking | +| Decoder tuned to walking through Pokémon maps | `packages/brain/src/readout/presets/gameboy.ts`: 800-ms directions, 85-ms A/B pulses with 480-ms cooldown | New fixed GameCube mapping; frame-scale controls, analog sticks/triggers, concurrent movement/action | Blocking | +| Neural code is already independently useful | `core/src/lif.rs`, `agent.rs`, `plasticity.rs`; TS counterparts | Reuse exact kernel and private per-agent state; do not replace neural semantics to integrate a game | Keep | +| Graph can be shared, pool cannot be concurrently dispatched | `Arc`, `SweepPlan`, `core/src/pool.rs` shared job slot | Immutable topology shared; distinct mutable state and controlled total scheduling budget | Blocking | +| CUDA exists but not an automatic service optimization | `core/src/lif/cuda.rs`; no `enable_cuda` call found in `sim/src/simloop.rs` | Explicit backend selection, equivalence/restore tests, profile first; don't promise GPU brains from graphics availability | Measured option | +| Snapshot header is a single Pokémon-shaped view | `sim/src/snapshot.rs`, `packages/feed/src/{types,codec}.ts` | Session descriptors, multiple agents/ports, task-specific progress, named attachments/media | Blocking for proper broadcast | +| Stage assumes Game Boy geometry and one fly | `apps/stage/src/{App.tsx,feed/store.ts,feed/decode.ts,lib/geometry.ts}` | Per-session/agent store instances, dynamic view aspect/dimensions, multi-agent match layout | Blocking for proper broadcast | +| Browser owns game audio | `apps/stage/src/audio/engine.ts`, 48-kHz feed, Pulse capture | Explicit audio producer and media-clock policy; do not play native Dolphin audio and forwarded PCM twice | Blocking | +| Capture already offers NVENC | `infra/bin/flycast-launch` | Reuse encode/relay/recording; measure new compositor/readback cost and revise 60-fps settings | Keep with changes | +| Encoder hardcodes H.264 level 4.1 | Both encoder functions in `flycast-launch` | 1080p60 needs a suitable level (normally 4.2 or automatic selection); changing only `FLY_FPS` is insufficient | Required for 1080p60 | +| Existing checkpoint payload is single-agent/binary-specific | `sim/src/store.rs`, `gb/src/compatibility.rs` | Coherent all-agent/world checkpoint plus backend/parser/patch/controller identity | Blocking for exact resume | +| Checkpoint writer queue is unbounded | `Sim::start_writer` uses `std::sync::mpsc::channel` | Bound/coalesce background work; larger emulator captures and several brain copies must not grow an unlimited queue | High | +| Existing “saved” event precedes durable commit | `checkpoint_with_reply` emits after enqueue; writer updates durable metrics on success | Distinguish capture/enqueue/commit/failure events; public status must not report a queued Melee save as durable | High | +| Recovery assumes a best progress ladder | `gb/src/{ratchet,recovery}.rs` | Match/episode reset policy; never rewind one player's world independently | Blocking | +| Health is mostly loop heartbeat | `sim/src/simloop.rs::Shared`, `sim/src/lib.rs` | Distinguish waiting at input barrier, intentional pause, backend timeout and deadlock; keep host supervision responsive | High | +| Deployment resource partitions reflect the old stack | `infra/units/flysim.service`, deploy cpuset construction | Budget Dolphin CPU/GPU plus N brains and media; measure and set new memory/process limits | Required before release | +| Bridge targets one sim | `services/bridge/src/{sim,commands,redemptions}.ts` | Explicit stable agent targeting and intervention policy; no viewer control-port endpoint | Before interactive show | + +Do not interpret dated CPU/GPU measurements in the repository as current free capacity. +The records identify bandwidth contention and graphics-sharing constraints, but this audit +does not inspect the host or establish that it can run two brains plus Dolphin in real time. + +## 5. A framework architecture that survives a third game + +### 5.1 Four runtime components, not one enormous adapter + +```text +Application / supervisory logic Presentation / recording + \ / + Flybus RPC + pub/sub + / | \ + Session coordinator Agent workers Backend helper + clock / barrier brain / encoder Python + libmelee initially + task + executors fixed readout owns Dolphin process/user dir + episode policy native pipes, parser, media/state hooks + \ | / + owned artifact references + | + immutable local store + +Presentation owns stage/compositor → capture → local relay → optional Twitch push. +``` + +The helper is an **internal backend implementation**, not an audience-accessible controller +service. The session remains the sole authority assigning actions to ports. Python never +simulates the neurons or selects actions. Keep it if measurements say its overhead is small; +replace its internals with Rust only when an actual bottleneck or maintenance +requirement justifies it. +Its Flybus binding uses the same protocol as every component. Dolphin-specific input pipes +stay behind the environment adapter, not a second framework communications system. + +An external emulator is a normal `Environment` implementation, not a special `if melee` +branch sprinkled throughout the session. For Game Boy, the same interface has an in-process +implementation. For an embodied world, it may be a native physics engine. Consumers do not +need to know which one owns the world. + +### 5.2 Framework contracts to extract + +| Contract | Owns | Must not know | +| --- | --- | --- | +| `Brain` / numerical core | Tick semantics, state, spikes, rates, learning updates | Game, process, controller labels or viewer | +| `SensorEncoder` | Declared observation→neural drive transform | Reward inspector state not declared as input | +| `Readout` | Fixed rate/signal→control-channel mapping | Opponent strategy, game addresses or pathfinding | +| `ActionExecutor` | Selected action + coherent game state + task progress + clock → controller state | Authority to invent a winning action when brain is silent | +| `Environment` | Native clock, port schema, action commit, observations/media, snapshot capabilities | Neural roles, Twitch or task reward weights | +| `Task` | Typed state interpretation, rewards, progress and episode outcomes | Direct neural mutation or direct controller writes | +| `EpisodePolicy` | Start/end/reset/recovery semantics | Hidden per-player rewind in a shared world | +| `Session` | Barrier, identity, agent isolation, routing, state capture and supervision | Melee memory offsets or Pokémon map IDs | +| `Flybus` | RPC/pub-sub routing, bounded deliveries, artifact ownership and GC | Game timing, neural roles, presentation/stream semantics | +| `Presentation` | Descriptor-driven layout, media/audio and task panels | Emulator stepping or access to controller pipes | + +Use modules first, then crates/packages as second consumers appear. The existing monorepo +and Rust workspace can stay in place during extraction. Keep static compiled registries +initially; a stable dynamic-plugin ABI and public package registry are not prerequisites. + +**Framework acceptance test:** adding a synthetic third environment/task requires a backend +implementation, a task/profile and a composition manifest, not edits to the session loop, +neural core, protocol enums or generic stage store. A game-specific presentation plugin is +allowed. Wire extensions must be namespaced/schema-validated, not hardcoded into every panel. + +### 5.3 Backend RPCs over the common bus + +Use Flybus for these conceptual capabilities; do not implement another socket/message +envelope. Exact method names/bodies are in the session-framework contracts. Pause/Resume +are session lifecycle intents; an environment adapter holds its boundary between Advances. + +```text +Hello → backend build/content/patch identity, capabilities, cadence, ports, views +Initialize(runConfig) → Ready(epoch, observationBoundary) +Advance(epoch, expectedBoundary, completePortBatch) + → StepResult(epoch, newBoundary, appliedBatchId, observation, mediaRefs) +Pause / Resume → explicit acknowledgment +Capture(epoch, boundary) → capture token + state digest + snapshot bytes/reference +Restore(captureToken) → new epoch + restored observation + success/failure +Shutdown → acknowledgment or bounded forced process termination +``` + +Only advertise `Capture/Restore` if implemented and tested. Otherwise expose an explicit +`restart_episode` capability and visible aborted-match policy; never claim exact resume. +Port batches use canonical buttons plus sticks in [-1,1] and triggers in [0,1], converted +once by the backend. Descriptor/schema versions pin conversions and active ports. + +Commands and pub/sub envelopes contain metadata and ArtifactRefs; native media/checkpoint +bytes live in Flybus's managed immutable store. Include producing epoch/frame/sample identity +and format in domain descriptors. DeliveryGuard keeps bytes alive beyond message drop if an +encoder/renderer retains a handle. Cached RPC replies own handles for retry; last-owner GC +reclaims data. Start file-backed; pooled shared-memory reuse is an optional later optimization. +Router restart invalidates routes/handles and requires coherent session recovery. + +One request in flight, explicit timeouts, bounded queues. After an uncertain `Advance` +response, do **not** resend blindly: the world may already have advanced. Resolve batch ID/ +boundary through an idempotent reply cache or fail/recover the session. A pipe transport with +no application acknowledgment needs validation against returned input telemetry, not a claim +of atomicity it cannot prove. + +Separate lifecycle I/O from the blocked advance operation so diagnostics/shutdown stay alive. +Do not issue a save operation scheduled on Dolphin's CPU thread while that same thread is +waiting forever for pipe input. Capture/pause needs a backend-owned quiescent point where +both the simulation and pending input consumption have known state. + +## 6. Multiple flies and the timing contract + +### 6.1 One match, one world clock + +Two flies in Melee normally means two controller ports in **one Dolphin instance**. Four +flies means four ports, not four copies of Melee joined through netplay. Several independent +matches are separate sessions/process trees, orchestrated by application code developed with +its presentation. A tournament director is one example, not a mandatory framework service. + +For each committed boundary: + +1. Freeze each agent's allowed observation from the same world state. +2. Advance every brain by the environment's elapsed emulated time using its private remainder. +3. Decode independent actions; step any declared executors; assemble a complete port batch. +4. Commit the batch once and let the backend advance to the next acknowledged boundary. +5. Associate rendered sensory frames and telemetry with their actual producing boundary. +6. Route task events/rewards exactly once; encode the next input and publish a snapshot. + +Warm-up settles/calibrates brains with learning off while the environment is held at its +initial boundary. The next observation must not be from a game that ran freely through the +warm-up. Fixed offsets between brain and environment clocks are recorded. + +Use the selected backend's rational emulated cadence, not `GAMEBOY_MS_PER_FRAME` or an +unexamined exact 60. An approximately 60-Hz budget is about 16.7 ms, but logical game frame, +video interrupt, input poll, rendered presentation and Slippi frame bookend are distinct +events until the spike establishes their mapping. Session ticks are monotonic even when +Melee's signed frame number resets, starts before zero, or changes across menu scenes. + +### 6.2 Render latency and agent fairness + +Dual-threaded graphics can present frame `n` after telemetry for `n` is available. Label the +actual frame; do not attach “latest screenshot” to current state and assume equivalence. +Start with one declared fixed observation latency shared by all agents, and measure it. +If a pipeline deliberately adds one frame of latency, record that in the sensor profile. +The full-resolution spectator view may be delayed separately, provided overlays use the +matching presentation timestamps rather than future task data. + +Also test a potential pipeline deadlock: the helper waits for a rendered image while Dolphin +is waiting for the next controller flush needed to reach that presentation event. Fix the +backend's rendezvous or select a declared previous-frame sensory latency; do not unblock it +with an undisclosed neutral gameplay input. Game-state bookends alone do not prove the GPU +has completed a matching frame. + +Do not reduce brain integration from 1,000 to 500 ticks per emulated second to meet wall-clock +deadlines. That changes the model. A declared action-repeat interval can reduce decisions, +but normally still requires all neural ticks and correctly accumulated intermediate task +events; it does not halve the principal neural cost. Rendering every second game frame is +also a sensor change if the brain otherwise sees each frame, not merely a broadcast setting. + +On slow compute, the default is to slow the entire local match and report real-time factor. +Do not let one fly continue while the other misses turns. On a dead participant/backend, +pause or abort the match visibly; neutral fallback play is not silently substituted. + +### 6.3 No rollback netplay in the first release + +Slippi supports online play, but we do not need it to connect two local flies. Online rollback +would require rewinding **all** neural states, RNG, decoder/executor state, reward ledgers +and admission decisions at the same speculative boundary as the game, then replaying inputs. +Filtering repeated frames in libmelee is not that system. Keep offline local matches and +assert monotonic committed observations per epoch; classify unexpected rollback as an error +or explicit recovery transition rather than double-rewarding it. + +## 7. Controller, sensory and learning design + +### 7.1 A GameCube controller is not an eight-bit pad + +Support independent main and C sticks, analog shoulders, digital trigger clicks, face +buttons, start and D-pad. Movement and attack can overlap. Canonical neutral/release state +must be complete, so a missing command cannot accidentally leave attack or shield held. + +At about 60 Hz, the current 800-ms direction hold is roughly **48 game frames** and the +85-ms pulse roughly five. Reusing these values would dominate the fly's behavior regardless +of the dataset. Create a fixed Melee readout with explicit decisions in integer game frames, +bounded analog mappings, dead zones, tie handling and pulse/hold policies. Start with a small +declared set of stick magnitudes and directions if that makes validation easier; continuously +valued mappings can follow as a separate profile. + +Audit tap-jump, directional aerials/smash attacks, jump release, shields, simultaneous axes, +and conflicting inputs. Don't add state-conditioned auto-aim, automatic edge recovery or +combo execution under the label “controller mapping.” If later desired, publish those as +separate macro/assistance profiles with their own identity and comparison baseline. + +Start/system controls are lifecycle-sensitive. During an active match the profile may omit +pause entirely; initial match setup and between-match reset are disclosed episode scaffolding. +This is not permission for an API to press buttons. Specify whether setup uses an audited +initial state, internal deterministic menu setup, or a reset hook, and mark those frames as +non-neural setup with learning disabled. That expands the legacy “all presses” phrasing and +requires a deliberate task-policy/documentation decision before shipping it. + +### 7.2 What the fly sees + +For the initial pixel profile, both flies receive the same shared game camera with fixed +crop/aspect treatment, independent of spectator overlays. The legacy 160×144 input should +not stretch a 4:3 scene silently. Compare an aspect-preserving downsample/letterbox transform +with a separately versioned input-size profile; changing kernel retina dimensions currently +affects the numeric configuration identity. + +Current L1 projection samples luminance at 1,572 columns. Higher broadcast resolution does +not produce more sensory neurons, color recognition, motion estimation or knowledge of which +fighter the fly controls. Test whether each selected character remains visible across zoom +and stage movement; record the sparse sensory representation rather than assuming a human- +readable video is an adequate neural input. + +Three distinct modes must not be conflated: + +| Mode | Neural observation | Rendering implications | +| --- | --- | --- | +| Pixel baseline | Actual game image through fixed encoder | Needs real rendered frames even if no desktop GUI is shown | +| Structured-state experiment | Explicitly encoded positions, velocities, stocks, etc. | Can potentially use Null/fast-forward, but it is a new privileged-input model | +| Spectator-only rendering | Whatever the profile specifies; video for audience | May be independently compressed/delayed, never silently substituted for sensory input | + +“Headless” can mean no GUI while still rendering; “Null graphics” generally means no useful +pixel observation. The documented EXI fast-forward speed path cannot be advertised as the +performance of our pixel-fed broadcast. + +### 7.3 Reward and outcome attribution + +Implement a `MeleeTask` with typed per-player observations, match state, a ledger and positive +reward events. Its schema belongs to the task, not a generic `GameMode` enum. Start small: + +- Terminal match outcome, once, based on validated results/termination reason. +- Opponent damage and credited KOs only after verified ownership information is available. +- No reward for mere button activation, for losing a stock, or for scripted setup. + +Do not reward A for every increase in B's percent: self-damage, stage effects, reflected +projectiles, teams and another sub-fighter can invalidate that inference. The decomp's source- +player and KO tables guide inspection, but no runtime correctness is claimed until tested. +Unknown attribution produces a logged observation without a guessed reward. Keep fractional +damage until the rule deliberately quantizes; HUD damage and fighter damage may differ. + +Deduplication keys include epoch/match, producing frame and event identity. Handle multihits, +trades, simultaneous KOs, respawn percent reset, timeout, sudden death and disconnection as +separate cases. If source telemetry cannot distinguish a required case, narrow the first +ruleset or add a specific audited observation hook. + +Learning remains private per fly and synthetic reward modulation remains distinct from PAM +stimulation. Disable learning during kernel/controller/backend characterization; later compare +learning-on with learning-off, repeat seeds and swap sides/characters. Retaining gains between +rounds is a run policy. Competitive success is not guaranteed by increased model complexity. + +## 8. Performance plan: measure the actual critical path + +### 8.1 Budget equation + +For a lockstep pixel-fed match, approximate the critical wall-time interval as: + +```text +T_step = T_brains + T_readout/task + T_bus_RPC + + T_emulation_to_observation + T_required_render_readback + T_boundary_overhead + +T_brains ≈ sum(T_agent_i) [sequential evaluation] +T_brains ≳ max(T_agent_i) + barrier cost [parallel with sufficient independent resources] +``` + +The parallel estimate is a lower bound, not a promise: shared caches, memory bandwidth, +GPU contention and scheduling can make every brain slower. Media publication/encoding and +storage should be off the critical path, but their resource use and state capture still +affect it. Do not obtain a “60 fps” claim solely from Dolphin's display counter while the +brains advance fewer milliseconds or repeat stale observations. + +Proposed capacity gate: warm full-stack **unthrottled** throughput at least 1.2× the selected +game cadence for two flies, then a paced one-hour soak with no growing queues/lag and a +24-hour local endurance run before release. In the paced run, distinguish intentional wait +from compute time; report p50/p95/p99/max compute interval and deadline misses. The 1.2× +margin is a proposed engineering target, not a measured capability of current hardware. + +### 8.2 CPU/GPU strategy + +1. **Keep the first two brains on CPU.** Establish Dolphin JIT/render/media cost separately. + The repository's current service is CPU-composed even though a CUDA kernel exists. +2. **Allocate a total physical-core budget.** Compare sequential agents with modest within- + brain pools against agents running concurrently on disjoint core groups. Include Dolphin's + CPU/JIT thread, graphics worker, helper, browser, encoder and storage in the budget. Do not + launch four copies of the current per-brain pool size by default. +3. **Use native-resolution hardware rendering first.** Compare OpenGL/Vulkan on the chosen + build/platform; no blanket claim that one is faster. Measure render correctness and readback. + JIT is the performance baseline; an interpreter is a diagnostic baseline, not the live plan. +4. **Benchmark shader compilation and caches.** Report cold and warm starts separately. Choose + supported shader modes from measurements; a cache that hides startup hitches is not a + guarantee that a new stage/character will not compile something mid-match. +5. **Use NVENC where available, with a measured fallback policy.** Its encode engine does not + remove GPU rendering, memory allocation, color conversion or framebuffer-readback cost. + Automatic fallback to x264 can consume the cores the brains/emulator need; expose the + resulting degradation and test whether the declared session can still meet cadence. +6. **Only then test CUDA brains.** The current backend retains RNG/plasticity observation/rate + work on the host, uploads state inputs, and by default synchronizes membrane/refractory + state back each batch. It also allocates device graph/state per backend instance. Measure + two/four agents alongside Dolphin, browser graphics and NVENC; zero-copy shared graphs and + a GPU-wide scheduler are possible later work, not present features. + +For CUDA, preserve bit-exactness, gain-update ordering and checkpoint synchronization. +Batching across a future game action boundary is not valid just because it improves kernel +throughput. Keep the TypeScript oracle and existing version strings intact. + +The existing VirtualGL/Xvfb result demonstrates one Chromium rendering path, not that +Dolphin Vulkan/OpenGL works or is performant in the same container. Test the complete selected +graphics path. Reusing GPU passthrough requires no assumption of exclusive VRAM availability. +Resource availability must be measured in an approved, serialized deployment-host session. + +### 8.3 Media bandwidth and copies + +Uncompressed RGBA estimates, before copies/framing: + +| Image/cadence | Bytes per second | +| --- | ---: | +| Existing 160×144 at 30 fps | 2.76 MB/s | +| 640×480 at 30 fps | 36.86 MB/s | +| 640×480 at 60 fps | 73.73 MB/s | +| 1920×1080 at 60 fps | 497.66 MB/s | + +640×480 is a planning example, not an asserted fixed Dolphin framebuffer size. The backend +advertises actual dimensions/format/aspect. One shared camera is delivered once for the +match; two flies can sample one immutable image without duplicating its transport. If their +sensor transforms differ, encode separately against the same source frame. + +Start with native frames as immutable artifacts: 480p is not an automatic reason for a codec +or second transport. Flybus carries handles only; multiple readers map the same object. Measure +renderer readback, optional seal copy and consumer reads separately from router overhead. +Sensor downsampling remains a declared profile operation (optionally optimized near capture +after measurement). Viewer scaling/composition/encoding belongs to presentation. Last-use +handles, not receipt acknowledgments, govern GC; required sensory input cannot be coalesced. + +### 8.4 Benchmark ladder and decision records + +| Run | Configuration | Question / recorded output | +| --- | --- | --- | +| B0 | Synthetic two-port backend plus common Flybus, no neurons | Routed RPC latency, pub/sub fairness, step barrier, timeout and artifact-GC behavior | +| B1 | Dolphin with fixed input traces, rendering/audio on, no brains | Cold/warm emulator cost, step timing, port alignment, render-to-state latency | +| B2 | Same run + helper/media extraction | Incremental parsing, copying, downsampling and audio cost | +| B3 | One FAFB brain | End-to-end reference and per-phase costs | +| B4 | Two FAFB brains, sequential vs parallel schedules | CPU/cache/bandwidth limits, balanced observation and input timing | +| B5 | B4 + actual stage, capture, relay, recording, checkpoints | Full critical path, A/V drift, encoder fallback and queue growth | +| B6 | Four brains and four active ports | Capacity characterization only until this independently passes the same gates | +| B7 | Matched MaleCNS and optional CUDA variants | Dataset and backend effects, measured independently before combined variants | + +Record backend/content/patch/profile digests, physical-core allocation, exact graphics settings, +sensor/broadcast resolutions, all clock rates, resident/peak memory, VRAM, thread usage, +real-time factor, latency distributions, audio under/overruns and dropped frames by purpose. +Store operator-specific machine details externally and publish only the portable methodology +and non-identifying results. No measurements were performed by this document-writing task. + +## 9. Broadcast architecture and audio ownership + +### 9.1 Two viable routes + +Both routes are **application/presentation implementations**, not additional framework buses. +The environment emits native frame/audio artifacts through Flybus in either case. + +**Route A — stage receives game media.** A presentation gateway subscribes to native artifacts, +delivers them to the browser, and the stage composites game/overlays for capture. Start with +native uncompressed artifacts internally. Browser-edge delivery can remain raw or use a codec +after measurement; avoid encode→decode→encode unless its tradeoff is justified. The gateway +holds artifacts through conversion/use and then releases them; browser backpressure cannot +pin required sensory state without a bound. Overlay timestamps track displayed video. + +**Route B — compositor combines native game output and stage overlay.** Dolphin supplies its +rendered output to a compositor; the browser supplies a separate overlay surface. This can +avoid moving full-resolution game pixels through JavaScript, but requires an explicit shared +clock and a new capture composition. The fly's sensory image still needs a frame-identified +path from the backend. Capturing a desktop window on a wall clock is insufficient to establish +which image a brain used at a given game boundary. + +**Recommendation:** prototype Route A for the two-player local slice; benchmark Route B in +the media spike before committing to the long-run high-resolution pipeline. Preserve media +as a capability behind the environment interface so the choice does not change brain/task code. +The browser-facing v2 contract can reference media streams; it does not dictate the internal +bus, native sensor format or artifact-store implementation. + +### 9.2 Audio and clock policy + +Today the page plays binjgb PCM and stream SFX into the Pulse sink. With Dolphin, select one +of these explicitly: + +- Dolphin PCM is captured/forwarded and played by the page, with native device output muted. +- Dolphin renders audio to the capture sink directly, and the page contributes only SFX. + +Do not run both. Declare sample format/rate, resampling location, timestamps, buffering and +discontinuity handling. Pause/reset/restore must flush or relabel buffered old-episode audio. +If wall time falls behind, measure pitch/time-stretch behavior; do not let “async resample” +hide minutes of simulation lag. Game timestamps, not arbitrary browser receipt time, define +the intended A/V relationship. + +Keep 30-fps broadcast and approximately 60-Hz gameplay as independent settings. For 1080p60, +the existing H.264 level 4.1 is too low for the normal macroblock-rate requirement; use a +compatible level such as 4.2 or encoder-selected level and validate the actual stream. Also +measure capture/compositor cadence, bitrate quality, encoder lookahead/latency, local recording +and audio synchronization. `FLY_FPS=60` alone is not a completed performance upgrade. + +### 9.3 Presentation changes + +The generic stage needs descriptor-driven game aspect ratio, two/four agent cards, per-port +button/stick indicators, per-agent learning/sugar state, shared match stocks/percent/results, +and scoped events. Neither “badges” nor “highest ladder rung” describes a match. + +Separate task data schema from layout. Keep one compositor clock and one selected world audio +stream; keep neural maps and rate scalers private per agent/dataset identity. Defer expensive +four-avatar/whole-connectome rendering until measured. All actual layout decisions require +PNG mockups and existing legibility/browser checks. This text does not approve a screen. + +## 10. Persistence and unattended operation + +### 10.1 Savestates are a capability, not a libmelee assumption + +Dolphin source provides buffer/file state operations, but `State.h` documents that operations +called off its CPU thread may be scheduled rather than executed immediately. An external +“save requested” is therefore not proof of a consistent capture at our agent boundary. +Slippi's internal rollback save-state commands likewise do not constitute an audited public +multi-agent checkpoint API. + +The selected backend must supply acknowledgment of the frozen boundary, resulting state +digest and completion. Save every brain, RNG, rate/calibration state, learning state, sensor/ +executor state, pending action identity, task ledger and environment together. Clock and +media epochs change on restore; discard pre-restore spectator/parser buffers. + +Re-create or explicitly reinitialize libmelee's parser caches and controller history after +restore. An emulator savestate does not include an external helper's `_frame`, previous +game state, normalization state or queued pipe data. Test how game-start metadata is supplied +when loading into mid-match; some telemetry protocols may need reseeding or restarting. + +When exact mid-match capture is unavailable, an initial prototype may visibly abort and +restart a match while retaining a declared brain checkpoint. Mark `resume=episode-restart` +in the descriptor. That is a deliberate narrower capability, not equivalent to crash resume. +It is not ready for a release that promises uninterrupted exact match continuation. + +### 10.2 Storage and health + +Bound pending captures and coalesce replaceable hot checkpoints. A durable request either +completes with a commit acknowledgment or fails explicitly; never drop it while reporting +success. Match-end result records are append-only and independent of world rewind. + +Keep backend/helper/brain health separate from “game frame did not advance.” An intentional +pause or waiting barrier is not a crash; a dead helper must not keep the session green by +merely refreshing an HTTP heartbeat. Export last completed boundary, in-flight request age, +barrier participant status and renderer progress. A hard stall has a timeout and explicit +match abort/recovery path, not repeated blind restarts of unrelated services. + +Manage Dolphin under the session's lifecycle or an explicitly coordinated systemd unit. If +Dolphin restarts, the session cannot keep sending frame `t+1` to a fresh match. Allocate unique +user directories, pipe names, telemetry ports and state namespaces per independent session. +Use deterministic configuration provisioning and checksums instead of reusing a developer's +desktop Dolphin settings or permitting auto-updates. + +The existing `flysim.service` memory ceiling and CPU partition were sized for a different +process graph. Set new cgroup/resource limits from measured high-water marks; account for +backend process, multiple brain copies and checkpoint transients. Release preflight checks +the complete backend/game/patch/parser/profile identity. Rollback retains compatible state +as well as the old executable. + +## 11. Staged implementation and go/no-go gates + +This specializes the existing backlog rather than replacing its foundation/session work. +Melee-specific spikes can start before the full framework reorganization is finished. + +| Item | Work and dependency | Evidence required before the next step | +| --- | --- | --- | +| **MELEE-01: backend selection spike** | Specializes EMULATOR-01. Pin mainline Slippi + maintained libmelee, content and Gecko codes; use isolated user config and two synthetic controllers | Boot/render/audio; block one then both ports; one batch/frame mapping; menu→match→results lifecycle; cleanup/restart. Choose this build or stock Dolphin + narrow hook based on results | +| **MELEE-02: capture/restore spike** | Alongside MELEE-01; prove sensory-frame identity, media export and save/load acknowledgment independently | Fixed pixel↔telemetry latency, bounded media storage, correct input after restore, parser/cache recovery. Explicit decision: exact resume or prototype-only episode restart | +| **MELEE-03: task observation audit** | Pin decomp; build field catalog, typed parser/inspector, lifecycle and synthetic event fixtures | Verified port/player/sub-fighter mapping; stocks/results; no guessed rewards; content/patch mismatches visibly disable unsupported semantic interpretation | +| **FRAMEWORK-01: generic backend + session** | Existing FOUNDATION-01/02, BUS-01..03 and RUNTIME-01/02; all workers/application events use Flybus | Same legacy Game Boy traces; synthetic environment uses identical session API; artifact-backed requests/results and no Melee logic in router/core | +| **FRAMEWORK-02: multi-agent and state** | Existing RUNTIME-03/STATE-01; integrate complete action batches, worker budget and chosen backend recovery capability | No cross-agent state leakage; one world step; changed evaluation order invariant; failed restore cannot partly install a match | +| **MELEE-04: fixed readout and sensory profile** | MELEE-01/02 + framework boundary; TS specification then Rust implementation for any new decoder semantics | Neutral/release, analog conversion, tap/hold/direction combinations, aspect-preserved neural input and recorded latency; no hidden combo/aim policy | +| **MELEE-05: first two-fly match** | MELEE-03/04 + FRAMEWORK-02; learning off, then audited positive rewards | Recorded action/observation timelines; paired side/seed trials; match terminal deduplication and visible reset/failure semantics | +| **MEDIA-01: full local show** | Existing WIRE-01/PRESENTATION-01; compare Route A/B, define audio owner and 30/60-fps profiles | PNG review, fake multi-agent fixtures, measured copies/latency/A/V drift; sustained media pipeline under checkpoint and shader-load events | +| **PERF-01: two-fly capacity gate** | Benchmark ladder B0–B5; optimize measured limiting phase | ≥1.2× warm unthrottled capacity target, one-hour paced soak and 24-hour local endurance; no queue/lag growth; documented CPU/GPU/memory envelope | +| **MELEE-06: expand carefully** | Passing two-fly slice | Four ports/teams, broader characters/stages, MaleCNS and CUDA are separate experiments, each with new tests and its own capacity result | +| **FRAMEWORK-03: finish packaging** | Existing PACKAGE-01 after useful second backend | Example third backend can be added without core/session/schema edits; isolated compositions package and preflight correctly | + +**Stop conditions:** no dependable step/input barrier; no identifiable pixel source for a +pixel-input claim; inability to attribute rewards under the claimed ruleset; unsupported +restore marketed as exact resume; or sustained capacity below the declared cadence. +Respond by changing the explicit supported scope, backend or resources—not by quietly skipping +neural ticks, adding a gameplay bot, hiding game stalls or reporting guessed measurements. + +### 11.1 Suggested first experiment script + +The first implementation should be a local measurement harness, not the final stream: + +1. Launch one pinned backend with known local content and two configured bot pads. +2. Enter a fixed local match through the declared setup procedure; record episode boundary. +3. Send distinct short left/right and A/jump pulse patterns on each port, including neutral + frames; log intended batches and observed raw/processed controller values. +4. Delay one port by a controlled wall-clock interval and verify no game boundary commits + until the complete batch is available. Repeat with port order reversed and four ports. +5. Capture a sequence of images/telemetry with frame identities; measure their association. +6. Save/restore at a known barrier if supported, replay the same actions, and compare task/ + input traces; verify helper state and buffered media are reset coherently. +7. Kill the helper/backend separately and verify bounded failure without accidental continued + play or permanent hangs. Test paused-state health independently. +8. Measure compute with no brains, one brain, two brains, then the full broadcast stack. + +Synthetic controller traces are test machinery, not footage presented as neural play. Keep +game content and environment-specific records outside source control; store portable metrics, +synthetic schemas and independently authored tests in the repository. + +### 11.2 Test matrix that catches Melee-specific failures + +- **Input:** two/four ports, inactive slots, delayed/missing flush, stale buffered commands, + full release, analog endpoints/deadzones, short taps, pressed versus held edges. +- **Identity:** controller↔player mapping, swapped ports, sub-fighters, transformations, + character/stage changes, wrong game revision, changed patch/parser normalization. +- **Events:** multi-hit, trade, self-damage, projectile ownership, stock reset, simultaneous + KO, timeout, sudden death, results re-entry, disconnect, restart after accepted reward. +- **Clocks/media:** game-frame reset, renderer lag, stale artifact/store identity, last-use GC, dropped + spectator frame versus required sensory frame, paused audio, mismatched overlay timestamps. +- **Recovery:** all-agent atomic validation, one corrupt state chunk, backend import failure, + helper parser not reinitialized, asynchronous save completion, hot-store coalescing and + durable-write failure. Test new exact resume separately from legacy transient-reset behavior. +- **Performance:** cold/warm shaders, high-activity matches, checkpoint capture bursts, CPU + encoder fallback, browser reconnect, two/four neural agents and measured GPU contention. + +All implementation merges retain repository-required TS tests/typecheck, Rust workspace +tests and infra lint; UI changes add Playwright and PNG review. Game-backed jobs are explicit +operator-provided tests. Normal CI uses synthetic observations/backends and existing goldens. + +## 12. Decisions to carry into implementation + +| Question | Recommended answer now | Still requires evidence/choice | +| --- | --- | --- | +| Which emulator? | Dolphin, first trying mainline-based Slippi + maintained libmelee | Exact build selected by synchronized-input/media/state spikes | +| Use the decomp to run the game natively? | No; use it to audit task/state/controller semantics | Custom instrumentation only for specifically missing observations | +| One emulator per fly? | No for one match; one per independent session | Four-port capability must be tested, not inferred from two ports | +| Which brain? | Two existing FAFB agents for integration baseline | MaleCNS comparison after mappings/dynamics pass their independent gates | +| CPU or GPU brain? | CPU baseline, share immutable graph | CUDA versus CPU benchmark under Dolphin + capture, not in isolation | +| Inputs to the brain? | Pixels with explicit fixed transform | Structured state is a distinct optional research profile | +| Start with macros? | Fixed controller mapping, no hidden aim/combo policy | Any later assist profile is separately disclosed and evaluated | +| How fast? | Backend-native gameplay/input cadence, 30-fps initial show | Full-stack two-agent capacity; optional 60-fps broadcast and four flies | +| How to resume? | Whole-session coherent state where supported | Episode-restart prototype if exact state interface is not yet available | +| How generic? | Concrete environment/task/agent/session contracts and composition examples | Extract public packages only after second/third consumers prove the seam | + +The first operator choices needed are the initial characters/stage/ruleset, desired show +cadence, and whether a visibly restarted match is acceptable during the prototype. They do +not block the synthetic framework work or source-level backend spike design. + +## 13. Sources and audit scope + +Local code evidence appears in section 4. Additional local files inspected include +`core/src/lif/cuda.rs`, `sim/src/pacing.rs`, `sim/src/simloop.rs::start_writer`, +`packages/brain/src/readout/presets/{gameboy,platformer}.ts`, +`apps/stage/src/audio/engine.ts`, `infra/bin/flycast-launch`, +`infra/units/flysim.service`, and the profiling/VirtualGL methods under `infra/docs/`. + +External source snapshots inspected on 2026-09-18 (pin actual dependencies again at spike start): + +| Repository/ref | Observed revision | Files used | +| --- | --- | --- | +| [doldecomp/melee](https://github.com/doldecomp/melee/tree/b9ec8a2eb48520753b2f8159ccc94d033fbf60ea) `master` | `b9ec8a2eb48520753b2f8159ccc94d033fbf60ea` | `.github/README.md`, `docs/symbols.md`, config, player/fighter/match headers and player implementation | +| [vladfi1/libmelee](https://github.com/vladfi1/libmelee/tree/bce21f09984b286e6d36bfd2939e4cd4691f94c2) `master` | `bce21f09984b286e6d36bfd2939e4cd4691f94c2` | README, `melee/console.py`, `melee/controller.py`, license metadata | +| [project-slippi/dolphin](https://github.com/project-slippi/dolphin/tree/41a7a3a110ed52999486ae1901c8fbb9a63d4f13) `slippi` | `41a7a3a110ed52999486ae1901c8fbb9a63d4f13` | Pipe backend, controller update loop, Slippi EXI events, `Core/State.h` | +| [dolphin-emu/dolphin](https://github.com/dolphin-emu/dolphin/tree/ee018d00e60b9eb727489908a8daec5c537f44a8) `master` | `ee018d00e60b9eb727489908a8daec5c537f44a8` | `Source/Core/Core/Core.h`, state/core module inventory | +| [Felk/dolphin](https://github.com/Felk/dolphin/tree/46b7eacd5c810c2d21ec5fe51ea1a9c61a7ceb3d) historical `scripting` branch | `46b7eacd5c810c2d21ec5fe51ea1a9c61a7ceb3d` | Scripting README, `python-stubs/dolphin/{event,savestate}.pyi` | +| [altf4/libmelee](https://github.com/altf4/libmelee/tree/1da979657122facd0750ea99cf6858255e198326) `main` | `1da979657122facd0750ea99cf6858255e198326` | Archive notice directing users to maintained fork | + +Dolphin files inspected carry GPL-2.0-or-later headers; libmelee repository metadata reports +LGPL-3.0. Pin and retain actual dependency licenses/notices when packaging. A separate process +is an architectural boundary, not an assertion that distribution obligations disappear. + +Source inspection supports the integration hypotheses and concrete constraints above. It +does not establish Dolphin throughput on the deployment hardware, verify any game-memory +field live, demonstrate a new neural behavior, or prove exact multi-port/frame/save semantics. +Those are the measured deliverables of MELEE-01/02 and the performance ladder. diff --git a/docs/design/session-framework/README.md b/docs/design/session-framework/README.md new file mode 100644 index 0000000..ed13a4a --- /dev/null +++ b/docs/design/session-framework/README.md @@ -0,0 +1,122 @@ +# Application and session framework: architecture and contracts + +Status: **implementation specification, draft 2**, 2026-09-18. This is a guide for future +agents; none of the new runtime is implemented yet. Baseline code is `83090a9` on +`docs/improvement-suggestions`. MUST/SHOULD requirements apply to the proposed new path, not +retroactively to existing [public feed](../../feed-protocol.md), +[control API](../../control-api.md), or legacy numerical/checkpoint behavior. + +## Decisions from the architecture discussion + +1. **Applications orchestrate components and develop their presentation alongside them.** + “Director” is application code, not a mandatory framework service. A tournament is an + example application, not the system's organizing data model. +2. **One lightweight Rust bus supports RPC and pub/sub everywhere internally.** Flybus replaces + separate direct worker transports and an application broker. No NATS dependency. +3. **Messages stay small; large artifacts live in managed storage.** Delivery guards and + explicit cache/retention owners keep data alive until its last actual use, then GC reclaims it. +4. **The router moves messages and tracks generic ownership.** It never schedules game frames, + understands macro actions, composites video or operates a stream. +5. **Sessions synchronize worlds; agent workers compute in parallel.** One logical clock is + not one execution thread. The coordinator alone commits complete world-control batches. +6. **Game-aware executors receive current game state and task progress.** Rich inspection data + does not become undeclared neural input. +7. **Native observations are framework outputs.** Resizing, overlays, browser delivery, audio + mixing, encoding, narration and streaming belong to the application/presentation layer. + +## Read in this order + +1. [Flybus v1](bus-v1.md) — authoritative wire/routing/RPC/pub-sub/artifact lifecycle contract. +2. [Session RPCs](ipc-v1.md) — domain payloads, worker capability negotiation and safe retries. +3. [Step protocol](step-v1.md) — session state machine, ordering and clocks. +4. [Worker/task interfaces](workers-v1.md) — exact method bodies and game-aware executor boundary. +5. [Session media/state](state-media-v1.md) — observation timing and coherent recovery. +6. [Application/presentation boundary](publishing-v1.md) — snapshots, flexible data and effects. +7. [Implementation guide](implementation.md) — sequenced build tasks and acceptance tests. + +For context: [modular-session analysis](../malecns-modular-sessions.md) and +[Melee audit](../melee-framework-audit.md). Each contract owns its named subject; step ordering +wins over an informal diagram, and Flybus owns transport/resource rules. Resolve contradictions +before implementation. The [existing HTML report](report.html) is an overview, not a contract. + +## 1. Composition and processes + +```text +Application / supervisor Application presentation + run policies, identity, history UI, media composition, audience, stream + \ / + Flybus: RPC + pub/sub + / | \ + Session Agent workers Environment worker + coordinator brain + encoder emulator/world + native observations + task/executors + fixed readout + | + artifact store (same bus API) +``` + +This diagram is connectivity, not execution order. The step contract defines causal order. +One router can host several independent sessions and application consumers; a deployment may +choose one router per application for fault isolation. A router crash affects all its clients, +so choose that boundary deliberately. In-process mode still exercises routing and ownership. + +Defaults: coordinator per session, worker per fly, environment worker per world. Task and +per-agent executors begin as coordinator-local libraries. An environment helper may own a +separate emulator child process and adapt its native protocol. There is no second framework +socket/lease API between those logical components. + +Multiple players in one game share one environment/barrier. Independent games use independent +sessions. Linked emulators require a composite backend with link-appropriate timing. + +## 2. Ownership and authority + +| Owner | State and responsibility | +| --- | --- | +| Application | Composition, persistent personas/brain lineage, lifecycle policies, supported interventions, application schema/history | +| Session | Clock/epoch, port assignments, admission, task ledger, executor state, barriers and coherent recovery | +| Agent | Private membrane, RNG, rates, learning, sensory encoding, decoder and tick remainder | +| Environment | World, actual controller application, backend parser, native media and state capabilities | +| Flybus | Opaque endpoint/topic routing, delivery/call correlation, bounded queues, artifact-owner graph and GC | +| Storage client | Durable event/checkpoint writes and replay APIs; owns artifact handles during writes | +| Presentation | Application UI, display focus, clocks/buffers, media processing, audio/stream output and narrative cues | + +Immutable graph data can be shared; neural mutable state cannot. Do not concurrently dispatch +the existing single-job WorkerPool through cloned handles from different brains. + +Applications use declared session capabilities, not arbitrary emulator writes. Bus registration +and method privileges preserve a single controller authority for a session. Browser/audience +clients do not gain controller access by knowing a service name. Existing public rules remain. + +## 3. Domain independence + +The kernel knows no task or bus. The environment knows no neuron populations. A task interprets +game state and requests outcomes/recovery; its executor translates a selected decision using +read-only current game/progress context. The coordinator orders and applies these results. +The bus handles no such semantics. Presentation combines framework observations with an +application-owned schema and can change without changing simulation behavior. + +Persistent AssetRefs identify installed release content. Transient Flybus ArtifactRefs identify +live bytes with ownership. Domain epoch/step identity and bus route/store incarnation are +different: the first protects simulation order, the second protects delivery/resource validity. +Never substitute game-frame number, persona identity or array position for either. + +## 4. Initial scope and compatibility + +First build a generic bus example (RPC + pub/sub + artifact retained beyond message lifetime), +then a synthetic two-agent session using it. One local machine, Unix sockets/in-memory parity, +immutable file-backed artifacts, fixed cadence, 1-ms LIF, direct control and exact-checkpoint +synthetic backend are sufficient. Dolphin, MaleCNS and richer effects are later integrations. + +Deferred: cross-machine artifact access, durable broker queues, wildcard/queue-group routing, +dynamic native plugins, hot-join, speculative netplay rollback and pooled GPU buffers. + +Keep legacy-gameboy-v1 distinct from lockstep-v1. Preserve TypeScript as oracle, existing default +versions, historical arithmetic/fingerprints and FLYSIM01 reader. New identities include sensor, +readout/executor/task/scheduler semantics. New public feed v2 is an application/presentation +gateway contract built on the same internal bus; it does not replace the bus or expose it raw. + +## 5. Reuse criterion + +Adding a third environment/application requires a backend, task/profile, composition and +application presentation. It must not require game-specific edits to the coordinator, router, +artifact manager, kernel, generic stores or transport. Schematized task extensions are valid; +an unchecked data blob or universal tournament schema is not a substitute for interfaces. diff --git a/docs/design/session-framework/bus-v1.md b/docs/design/session-framework/bus-v1.md new file mode 100644 index 0000000..6839783 --- /dev/null +++ b/docs/design/session-framework/bus-v1.md @@ -0,0 +1,486 @@ +# Flybus v1: a small Rust RPC and pub/sub bus + +Status: **draft 1**, 2026-09-18. Selected architecture for the new framework, not implemented +runtime code. This document supersedes the earlier proposal for direct worker sockets plus +a separate application bus and coordinator-owned buffer release protocol. + +**One library, one router, one wire protocol. Small messages carry metadata and artifact +handles; large immutable bytes live in a managed local store. Ownership follows deliveries +and explicit retention.** The router moves messages and tracks generic resource ownership. +Applications own orchestration, presentation, effects and streaming. Sessions own simulation +ordering. The bus knows nothing about brains, game frames, matches or Twitch. + +## 1. Scope and implementation shape + +Start with a Rust/Tokio library, an embeddable router and an optional small router executable. +The same client API supports an in-memory test transport and Unix-domain stream sockets. +All participants use router semantics even when colocated; there is no separate fast-path +RPC protocol to maintain. Python helpers may implement a thin binding to the same schema; +browser integration belongs to the application's presentation gateway. + +```text +Application / supervisor ─┐ +Session coordinator ──────┤ managed artifact store +Agent workers ────────────┼── Flybus router ── immutable bytes +Environment backend ──────┤ handles in messages +Presentation / recording ─┤ +Audience adapter ─────────┘ +``` + +V1 supports one trusted local deployment with a shared local store. It does not require NATS, +a service mesh, durable broker queues, cross-machine artifact transfer, load-balanced stateful +workers, global transactions, dynamic plugins or a video codec. A private backend adapter may +still speak an emulator's native pipes/protocol internally; that is not a second framework bus. + +Suggested crate: `flybus`, with `wire`, `router`, `client`, `artifact`, and `transport` modules. +Split crates only when useful. Keep session/game types out of this library. Use Tokio, serde +and a checked framing layer; persistent histories/checkpoints stay in application/session storage. + +## 2. Client API + +Illustrative Rust surface (not yet implemented): + +```rust +let bus = Client::connect(config).await?; +let service = bus.register("agent.fly-a", service_config).await?; +let reply = bus.call(target, "Agent.Prepare", payload, attachments, budget).await?; +let subscription = bus.subscribe("session.demo.snapshots", subscription_config).await?; +bus.publish("session.demo.snapshots", payload, attachments).await?; + +let mut writer = bus.artifacts().allocate(size, content_type).await?; +writer.write_all(&pixels)?; +let frame = writer.seal().await?; // consumes writer; immutable Artifact handle +bus.publish("world.demo.frame", metadata, [("frame", frame.clone())]).await?; + +let message = subscription.next().await?; +let image = message.artifact("frame")?; +drop(message); // image still owns the delivery guard +render(image).await?; // last handle drop releases ownership +``` + +RPC, pub/sub and artifacts all use this client/connection. The artifact store is a data +structure/storage backend of the bus, not another messaging service. Bulk data does not +pass through the router's socket payloads or require a separate application data-transfer API. + +`Artifact` is a read-only, cloneable handle. `ArtifactWriter` is unique and not cloneable; +sealing consumes its writable lifetime. Mapped slices cannot outlive their handle. Rust RAII +automates releases; other language bindings provide equivalent explicit close/context-manager +behavior. Garbage collection means reclaiming an unowned artifact, not inspecting game state. + +## 3. Addressing and identities + +| Type | Meaning | +| --- | --- | +| `routerId` | Fresh router/store incarnation; changes after restart | +| `clientId`, `clientIncarnation` | Configured participant identity and SDK client lifetime; reconnect creates a new incarnation | +| `connectionId` | Fresh connection; v1 does not resume its queues or delivery owners | +| `service`, `serviceIncarnation` | Named endpoint and opaque router-issued registration identity | +| `callId` | Unique RPC correlation ID for this client incarnation; not a domain operation ID | +| `topic`, `topicIncarnation`, `topicSequence` | Exact topic, declaration lifetime and acceptance order within it | +| `deliveryId` | One recipient's message delivery and artifact ownership root | +| `artifactId`, `generation` | Immutable byte object identity within a store incarnation | +| `ownerId` | A connection-owned delivery or explicit artifact hold | + +Identifiers are bounded ASCII strings; scalar Id and U64 encodings match the common types +in [session RPC contracts](ipc-v1.md). Service/topic names use 1..192 characters from +`[a-z0-9._-]`, with no empty dot-separated segment. Exact names only in v1; wildcard routing +and queue groups are deferred. The router treats names as opaque addresses. + +One live registration owns a service name. Duplicate registration fails; there is no implicit +round-robin balancing or replacement of a stateful agent. Registration returns the incarnation. +Callers pin it after discovery. If it changes, calls fail with `TARGET_CHANGED` rather than +silently reaching another brain/environment. `Worker.Hello` remains a domain capability RPC, +distinct from transport connection negotiation. + +Service/topic access is configured per participant by the launcher/application. Presentation +subscribes to observations; it does not receive authority to invoke environment Advance. +Audience effects go through application/session admission. Naming a target is not authority +to control it. Public feed/control interfaces remain separate compatibility boundaries. + +## 4. Wire envelope and framing + +One connection handles both directions and every operation: + +```text +u32 little-endian JSON byte length | UTF-8 JSON object +``` + +```ts +interface BusEnvelope { + protocol: "flybus"; major: 1; minor: 0; + id: Id; replyTo: Id | null; + kind: "command" | "reply" | "delivery" | "notice"; + op: string; + body: object; + attachments: Attachment[]; +} +interface ArtifactRef { + storeId: Id; artifactId: Id; generation: U64; + byteLength: U64; contentType: string; + digest: Digest | null; +} +interface Attachment { + name: Id; ref: ArtifactRef; ownerId: Id; +} +``` + +Maximum total JSON envelope is **65,536 bytes**. Up to 32 attachments, with unique names; +contentType is a nonempty ASCII string <=127 bytes. No pixel/base64/checkpoint bytes in JSON. +Artifact sizes are independent of envelope size. Domain schemas referencing artifacts MUST +enumerate every referenced artifact in attachments; generated client bindings enforce this. +The router validates attachment declarations/ownership, not the contents of domain payloads. + +Commands have unique monotonically issued `id` serials per connection (canonical `msg-`). +Replies correlate with `replyTo`; routing notices/deliveries have router-generated IDs. +The router supplies authenticated sender/target metadata on deliveries; senders cannot forge +it by putting another participant's name in body. Domain `requestId` and callId remain distinct. + +Reject duplicate JSON keys, invalid UTF-8, NaN/Infinity, unknown envelope fields, zero/oversize +frames and invalid ranges. Read length before allocating. Handle partial reads/writes and +serialize one writer per connection. There are no ancillary-FD tricks in the first file-backed +implementation. A future memory backend must retain the same client/ownership API. + +### Connection negotiation + +First command `bus.hello` has body `{clientId, clientIncarnation, supportedMajors}` and no +attachments. Its reply reports `{routerId, connectionId, selectedMajor, selectedMinor, +contractDigest, limits}`. The launcher provides expected client/registration privileges and +local endpoint/store configuration. Refuse incompatible majors or identity mismatch before +registration. Changes to these draft schemas change contractDigest; incompatible released +schemas require a major version bump. + +The connection reader must dispatch incoming replies and requests without blocking on user +handlers. Blocking neural computation runs on dedicated workers; artifact I/O/hashing runs +outside the router's routing critical section. No mutable routing-state lock across slow I/O. + +Example RPC command (the counter service is a generic test, not a built-in router feature): + +```json +{ + "protocol": "flybus", "major": 1, "minor": 0, + "id": "msg-7", "replyTo": null, "kind": "command", "op": "rpc.call", + "body": { + "callId": "call-2", "target": "example.counter", "expectedIncarnation": "service-a", + "method": "Counter.Increment", "payload": { "amount": 1 } + }, + "attachments": [] +} +``` + +An image-bearing publish has the same envelope shape. Its payload contains dimensions/time +and an application-defined attachment binding; its attachment entry contains ArtifactRef and +ownerId. byteLength may be `"1228800"`, while the entire message remains a small JSON object. + +## 5. Operation registry + +All operations use the envelope above. Replies are `{ok:true, value:object}` or +`{ok:false, error:{code, message, dispatch}}`, where dispatch is `not-dispatched`, `dispatched` +or `unknown`. Message length includes this wrapper. Transport error != domain error. + +| Command | Body / reply value | Semantics | +| --- | --- | --- | +| `service.register` | `{name, maxQueued, maxInFlight}` → `{serviceIncarnation}` | Exclusive endpoint, bounded capacities | +| `service.unregister` | `{name, serviceIncarnation}` → `{removed}` | Stop new calls; queued calls fail; dispatched results follow §6 | +| `rpc.call` | `{callId, target, expectedIncarnation:null|Id, method, payload}` → `{accepted, serviceIncarnation}` | Admission acknowledgment only, eventual rpc.result follows | +| `rpc.reply` | `{callId, requestDeliveryId, outcome}` → `{routed}` | Reply only by the registered recipient; outcome is domain success/error | +| `rpc.cancel` | `{callId}` → `{state}` | Best effort; only canceled-before-dispatch establishes no invocation | +| `topic.declare` | `{name, retained:"none"|"latest"}` → `{declared, topicIncarnation}` | Compatible redeclaration allowed; conflicting settings fail | +| `topic.clear` | `{name}` → `{cleared}` | Release retained topic root; does not invalidate deliveries | +| `topic.delete` | `{name}` → `{deleted}` | Only when no subscribers; releases retained state | +| `subscribe` | `{topic, mode:"latest"|"bounded", maxQueued, maxInFlight, replayLatest}` → `{subscriptionId, topicIncarnation}` | Exact topic; no durable history | +| `unsubscribe` | `{subscriptionId}` → `{removed}` | Discard queued messages; already delivered handles remain valid | +| `publish` | `{topic, payload}` → `{topicSequence, subscribers, replaced}` | Atomic publication admission/fan-out, not consumption | +| `delivery.consumed` | `{deliveryIds:Id[]}` → `{released}` | Idempotent; processing and all local artifact uses have finished | +| `artifact.allocate` | `{byteLength, contentType}` → `{artifactId, generation, ownerId, writeLocation}` | Reserve quota and private writable staging storage | +| `artifact.seal` | `{artifactId, generation, ownerId, digest:null|Digest}` → `{ref, ownerId}` | Finish immutable publication; writeLocation no longer valid | +| `artifact.open` | `{ref, ownerId}` → `{readLocation}` | Resolve an owned sealed object for read-only mapping, never return its bytes | +| `artifact.retain` | `{ref, ownerId}` → `{ownerId}` | Create independent explicit hold while source ownership still exists | +| `artifact.release` | `{ownerIds:Id[]}` → `{released}` | Drop explicit holds/staging writers; not another client's owners | + +Names/arrays/ranges follow §3/4/9. Release batches contain 1..64 IDs. No attachments are +allowed on management commands except rpc.call/rpc.reply/publish. `outcome` is a bounded +domain object; an incoming RPC result has its own attachments and delivery ownership. + +`accepted`, `routed`, `removed`, `declared`, `cleared`, `deleted`, `replayLatest` are booleans; +`subscribers`, `replaced`, `released` are U64 counts. Released counts count newly released +roots, so an idempotent repeat may report zero. Queue/credit requests are integers 1..65535 +and cannot exceed configured limits. Method strings are 1..128 printable ASCII characters. +call target is a service name; expectedIncarnation is its registration ID, not worker process ID. +Location grants are `{storeId, relativePath}` resolved by the client beneath the configured +local store root; absolute paths, parent traversal and symlink escapes are rejected. They +are SDK-private and do not appear in the application's ArtifactRef. Runtime paths are not +committed into application schemas or source configuration. + +Deliveries: + +- `rpc.request`: `{deliveryId, callId, caller, target, serviceIncarnation, method, payload}`. +- `rpc.result`: `{deliveryId, callId, responder, serviceIncarnation, outcome}`. +- `topic.message`: `{deliveryId, subscriptionId, topic, topicIncarnation, topicSequence, replaced, payload}`. + +`caller`/`responder` include clientId and clientIncarnation. Delivery attachment ownerIds are +replaced by the recipient's deliveryId; source owner tokens are never delegated verbatim. +`topicSequence` and counters are U64 strings. The router assigns these IDs; the SDK exposes +typed payloads plus Artifact handles. Required bounded notices are route removal, subscription +closure and call failure. If even notice capacity is exhausted, close the connection instead +of silently losing control-plane correctness; disconnect is itself a typed client failure. + +## 6. RPC behavior + +callId uses `call-` with increasing serials per connected client. Keep an admission +watermark plus active-call entries; reused/retired call IDs are rejected, never executed again. +Reconnecting creates a new clientIncarnation/connection rather than reviving its old calls. +An RPC targets one registered service, not a broadcast subject. Preserve first-dispatch FIFO +per caller/service; responses may complete out of order and correlate by callId. Service +dispatchers can answer status concurrently with a long mutation, subject to their domain +state rules. The router does not implement frame barriers or numerical ordering. + +Admission validates route, pinned incarnation, size, quotas and every source artifact owner. +It establishes request-delivery roots atomically before accepting. Rejection establishes no +delivery and drops any provisional roots. An accepted call is not proof that its handler ran. +Before any request bytes can reach the target, mark it dispatched; subsequent transport loss +is conservatively an unknown execution outcome. + +The service publishes a reply with its own owned artifact handles. The router establishes +caller-result ownership before accepting that reply. It keeps only bounded call correlation +metadata until result delivery is consumed or the caller detaches/disconnects. It is not an +indefinite RPC result cache. A second rpc.reply for the same call is rejected, not routed twice. +Responding does not release the request's delivery guard; the handler does so when finished. + +**No automatic retry or failover.** A deadline belongs to the calling client. On timeout the +client may send rpc.cancel, but cancellation after dispatch cannot undo work. Domain retries +use a fresh bus callId containing the **same domain requestId/body**, pinned to the same +service incarnation. Endpoint-level deduplication supplies safe replay; the router does not +infer it from method names. Never route a retry automatically to a restarted worker. + +Queued cancellation releases its queued artifact roots and returns `cancelled-before-dispatch`. +After dispatch return `execution-unknown`; keep the recipient's delivery alive until consumed +or disconnected. A later reply to a detached call returns `routed:false`, with no caller-result +roots. The service still owns any retained result; domain state may have changed. +If a terminal result is already admitted, cancellation reports `completed`; the client drains +and consumes any result it no longer exposes to its caller. A retired/unknown correlation +reports `call-gone`. These four strings are the complete rpc.cancel state enum. None authorizes +re-execution, and cancelling a future does not abandon incoming delivery ownership. + +To replay an artifact-bearing response safely, endpoint code caches **Artifact handles plus +payload**, not bare references. That cache owns explicit holds or delivery guards until domain +acknowledgment/eviction. Re-delivery gets new delivery IDs pointing to the same immutable bytes. +Once the domain cache expires it returns RESULT_EXPIRED; it cannot regenerate the operation +merely because the transport correlation entry was removed. + +## 7. Pub/sub semantics + +Topic declaration does not teach the router what a topic means. An application can publish +session observations, presentation cues, dataset jobs or unrelated typed events through the +same API. There are no hardcoded frame/brain topics inside the router. + +- `latest`: one queued value per subscription, replacing only an **undelivered** value. + Replacing it releases that queue entry's artifact roots. Already delivered/in-use messages + are never reclaimed early. maxQueued is exactly 1 in this mode. +- `bounded`: FIFO queue, no coalescing or silent loss. When required queue/owner capacity + is unavailable, reject the publication with BACKPRESSURE before admitting any deliveries. +- maxInFlight credits are returned only by delivery.consumed, not socket write completion. + A latest subscriber with all credits in use still has one replaceable queued value. + +Take an atomic subscriber/retention snapshot at admission. Validate and reserve all required +queue entries and artifact-owner budgets before accepting. For a bounded subscriber overflow, +reject the **whole publish**; no partial fan-out or retained-latest update. On acceptance, +assign one topicSequence and create roots for every delivery and optional retained value. +Different topics have no total ordering. Multiple publishers on one topic follow router +acceptance order, which is not automatically a deterministic application event order. + +Publication reply counts accepted subscriptions/replaced queue entries, not consumers that +processed data. `replaced` on a delivery reports how many undelivered messages were coalesced +since that subscription's preceding delivery. Sequence gaps can also arise from joining late; +they are not evidence of a simulation step being skipped. + +Optional `retained:latest` holds one last message and its artifacts independent of subscribers. +New subscriptions with replayLatest enqueue it before subsequent accepted publications; +bounded mode preserves that order, while latest mode may coalesce it before delivery under +the ordinary latest rule. Replay uses the original topicSequence, a fresh deliveryId and +explicit roots. Without retention, +zero-subscriber publication retains no artifact ownership after admission. Clearing a topic +releases only its retained root, not active consumers. Topic count and retained bytes are capped. + +There is no durable replay, automatic redelivery, or exactly-once processing claim in v1. +Deleting/redeclaring a topic creates a fresh topicIncarnation; a reset sequence cannot be +mistaken for continuation of the deleted topic. Old subscription deliveries retain their +original incarnation and ownership until consumed. +If an application requires history, its recorder persists events and exposes a normal RPC +for recovery/query. Bus admission, message consumption and durable storage acknowledgment +are three different events. The bus must not conflate them. + +## 8. Artifact lifecycle and garbage collection + +### 8.1 Immutable object lifecycle + +```text +ALLOCATED / WRITING → SEALED → referenced by owners → last owner drops → COLLECTED + └─ writer abandoned/disconnected ──────────────────────→ COLLECTED +``` + +`ArtifactRef` is an identity, not an address, filename or authority to read. Opening it requires +a current ownership root belonging to that connection. storeId is the router/store incarnation; +old handles fail after restart. V1 does not reuse an artifact ID/inode; generation is 1 and +remains in the contract for future pool implementations. Content hashes are optional for live +frames and mandatory for checkpoint/durable-content handoff, as specified by domain contracts. + +The first storage backend uses runtime-configured local files, optionally on tmpfs. Producer +writes staging storage outside the message stream. Seal closes writable mappings/handles in +the SDK, checks length and any requested digest, then finishes an immutable store-owned +object before acknowledging. A correctness-first implementation may copy into a fresh sealed +inode; account for both allocations during sealing. No per-frame fsync for transient media. + +Consumers resolve a store-issued readLocation through artifact.open and map/read it read-only. +Locations are private grants; they are not placed in application bodies or public browser feeds. +All filesystem access stays behind the client Artifact API. Do not inline binary data or create +a second bulk-transfer server just because it is stored outside the socket. + +### 8.2 What owns an artifact? + +The store tracks an ownership graph, not only a naive refcount incremented by every packet: + +| Root | Lifetime | +| --- | --- | +| Producer explicit hold / active writer | Until last local handle releases, seal transfers its unique writer, or connection is lost | +| Accepted queued delivery | Until replaced/cancelled or transferred into recipient delivery ownership | +| In-flight delivery | Until message processing AND all extracted artifact uses finish | +| Retained latest topic | Until replaced, cleared, deleted, or router stops | +| Explicit retained hold | Until release; used for caches, rendering, checkpoint writes and forwarding | + +Admission creates destination roots before the sender may relinquish source roots. A forward +or reply uses a live handle/owner; the client holds it until admission succeeds or definitively +fails. A timeout must not drop a source guard while an unsent operation might still be admitted. +The client cancels/discards the unsent frame or retains the guard until the transport outcome +is known; connection teardown ends that ambiguity for the old connection. + +Every envelope must list its complete artifact set. Duplicate references in one delivery are +counted once. A retained topic and several consumers can reference the same physical bytes. +The router performs metadata updates only; it does not copy image bytes for fan-out. + +### 8.3 Consumed means no remaining use + +The incoming message owns a shared **DeliveryGuard**. Extracting an Artifact clones the guard; +dropping the message alone does not consume the delivery while a renderer/encoder still uses +its artifact. Local handle clones do not each require a bus round trip. Dropping the last +guard queues delivery.consumed through a bounded control lane. + +V1 deliberately owns at delivery granularity: keeping one artifact from a message may keep +the other attachments alive too. For independent long-lived retention, artifact.retain creates +a specific explicit hold before the original guard is dropped. Domain RPC caches must use +that hold when they outlive message processing/credits. An acknowledgment of domain success +does not implicitly drop either the incoming guard or cached outgoing holds. + +If an allocation/seal grant arrives after its caller abandoned the future, the SDK reactor +still processes and releases that grant. It must not leak an owner the application never saw. +An in-progress seal/copy has a bounded internal I/O hold; on producer disconnect it either +finishes cleanup or aborts safely, never publishes an ownerless object into a new connection. + +Async task cancellation/drop of a response future is not necessarily consumption: the client +must own queued results until surfaced, explicitly discarded, or disconnected. Receivers must +await completion of asynchronous CPU/GPU use before releasing its guard. A pointer extracted +from a mapping cannot outlive its Artifact; FFI wrappers must enforce this lifetime explicitly. + +Release commands are batched, idempotent and scoped to the connection that owns the IDs. +Delivery/hold IDs use monotonic per-connection serials. Keep issued watermarks plus active +owner maps; releasing an already retired ID is a no-op, a never-issued/future ID is an error. +This avoids a tombstone per frame forever. Control-lane exhaustion closes the connection +instead of silently losing releases and leaking an unbounded ownership graph. + +### 8.4 Crash, disconnect and safe physical reclamation + +On disconnect, unregister services/subscriptions; cancel queued deliveries and release that +connection's active writers/explicit/delivery roots. Retained topic roots remain router-owned. +Late replies and releases cannot attach to a new connection or service incarnation. + +GC removes the registry entry and unlinks/closes the sealed object after its final root is +gone. Existing immutable file mappings may remain valid until the OS closes the last mapping; +do not overwrite their inode or reuse their bytes. Logical reclamation is not proof that a +disconnected process released its physical pages. The supervisor handles stuck processes; +resource measurements include OS mappings and client memory, not just registry totals. + +No TTL may reclaim a live owned artifact. Limits may disconnect a consumer, triggering the +explicit cleanup above, but cannot overwrite memory under a renderer. Future pooled shared +memory must prove equivalent lifetime/generation safety before replacing immutable files. + +Router restart creates a new routerId/storeId, loses routes/queues/retention and invalidates +all old handles. Live sessions fail their current epoch and use coherent recovery. Persistent +artifacts come from the application's durable store and are re-imported as new bus objects. +Orphan files from a stopped router are cleaned without treating them as durable checkpoints. + +## 9. Bounds, scheduling and failure reporting + +Configure limits explicitly; these defaults are a prototype starting point, not capacity data: + +| Resource | Default | +| --- | ---: | +| Connected clients / services / topics | 64 / 256 / 512 | +| Subscriptions per client / total | 128 / 1024 | +| Control envelope | 64 KiB | +| Active calls per client | 64 | +| Service queued / in-flight calls | 16 / 16; worker dispatcher further limits mutations | +| Latest subscription queued / in-flight deliveries | 1 / 2 | +| Bounded subscription queued / in-flight deliveries | 64 / 16 | +| Active owners per client | 256 | +| Total artifact storage / per object | 512 MiB / 128 MiB | +| Per-client ordinary queued envelope bytes | 1 MiB | +| Reserved management/reply lane | 128 frames and 1 MiB per client | + +Reserve an owner allowance for lifecycle/results separately from ordinary telemetry; memory +quotas account for staging/seal copies, queued deliveries and caches. Ownership metadata is +bounded even if many roots share one artifact. Disk-full, allocation failure or hash mismatch +returns a typed artifact error and cleans provisional storage/roots. + +The router fairly services clients. Replies, release, cancellation and route-health control +cannot be starved by telemetry. Preserve FIFO for calls to a target despite lane scheduling; +classification is an explicit generic envelope operation/policy, not a topic-name heuristic. +No indefinite wait inside the router on subscriber readiness or artifact I/O. Admission is +bounded; rejected callers choose their own retry/fail/pause policy. + +Transport errors include `INVALID_ENVELOPE`, `VERSION_MISMATCH`, `NOT_AUTHORIZED`, +`NO_SERVICE`, `TARGET_CHANGED`, `BACKPRESSURE`, `CALL_GONE`, `ARTIFACT_UNSEALED`, +`ARTIFACT_GONE`, `OWNER_INVALID`, `QUOTA_EXCEEDED`, `STORE_FAILURE`, `ROUTER_LOST`. +Before admission use dispatch:not-dispatched. Once dispatch might have occurred, report +unknown/dispatched conservatively; a caller-side timeout must not imply no mutation. + +Bounded event subscriptions can reject publication; latest spectator subscriptions cannot +hold a required session transaction indefinitely. Per-client credit/owner budgets and +supervision enforce that distinction. Sustained pinned-artifact quota exhaustion is surfaced +as resource pressure, not solved by freeing live data. Session/app policies choose whether to +disconnect an observer, pause, or fail; the router does not know which outcome is appropriate. + +## 10. Native-frame bandwidth check + +At 640×480 RGBA, a frame is 1,228,800 bytes. At 60 fps, production is **73.728 MB/s** +(decimal); at 30 fps, 36.864 MB/s. These are planning dimensions; a backend advertises its +actual output. Two agent workers plus one presentation consumer can read the same immutable +frame object. Bus messages contain only references; there is no 3× byte fan-out through the +router. Readers still incur memory traffic/page faults, and renderer readback/seal copying +remain real costs. This is not a claim of zero-copy GPU capture or measured host performance. + +The environment emits native game frames/audio. The application/presentation layer owns +resizing, overlays, compositing, browser delivery, codec choice and stream output. Do not put +1080p rendering, Twitch publishing or game-specific sampling logic in Flybus. A presentation +pipeline may itself exchange large artifacts through this same bus if useful. + +## 11. Acceptance tests and implementation sequence + +1. **Wire/router:** schema/framing, Hello, exclusive routes, pinned incarnations, request/reply, + disconnect and bounds. In-memory transport must pass the same tests as Unix sockets. +2. **Pub/sub:** exact topics, FIFO/bounded rejection, latest coalescing, retained replay/clear, + atomic fan-out and fair control/reply delivery under a saturated subscriber. +3. **Artifacts:** allocate/seal/read; publication before seal fails; fan-out owns one physical + object; last consumer releases; retaining an extracted frame after message drop works. +4. **Faults:** sender drops after admission; consumer dies mid-read; reply is lost; queued frame + is replaced; subscription closes with in-use deliveries; router restarts; old release arrives. + No double-free, use-after-reuse, unbounded tombstones or hidden operation replay. +5. **RPC cache:** endpoint retains an artifact-bearing result, original caller consumes it, + and a domain retry still returns valid bytes. Eviction drops the last cache hold correctly. +6. **Integration:** two parallel fake agents, complete-batch environment RPC, committed snapshot + publication and a deliberately slow presentation consumer over the same router. +7. **Performance:** 640×480×60 artifact production with three consumers, one delayed; measure + p50/p95/p99 RPC latency, router CPU, copy/readback cost separately, RSS, store live/peak bytes, + outstanding roots, collection lag and queue lengths. Compare one/two/four agent schedules. + +The first executable example should show a counter RPC, a pub/sub observer, and a frame +artifact held past message consumption in one small Rust program. No game or browser required. +Distributed simulation ordering remains the [session contract's](step-v1.md) responsibility. diff --git a/docs/design/session-framework/implementation.md b/docs/design/session-framework/implementation.md new file mode 100644 index 0000000..124d7d1 --- /dev/null +++ b/docs/design/session-framework/implementation.md @@ -0,0 +1,270 @@ +# Future-agent implementation guide + +Status: **draft 2**, paired with the [contract index](README.md). This is the execution guide +for the new process architecture; it refines the broader +[MaleCNS/modular backlog](../malecns-modular-implementation.md), not a request to implement +every future feature in one branch. + +## 1. Start here + +Before coding: + +1. Read repository instructions, the architecture tour, public feed/control contracts, and all + documents in this directory. Reconcile any newer main/feature-branch changes with this baseline. +2. Record what the next slice will change, its compatibility surface and expected tests. +3. Use a dedicated branch/worktree. Follow the repository's coordinator/build/review roles. +4. Preserve the TypeScript oracle, default numerical version strings and legacy deployment. +5. Resolve contract contradictions before implementing; do not fill gaps with an implicit + asynchronous best-effort policy or a public controller API. + +The immediate target is **one small Rust Flybus example with RPC, pub/sub and a frame artifact +held beyond the message object's lifetime**. Then build the synthetic two-agent session over +that same router using in-memory and Unix-socket transports. No separate worker transport, +NATS service, coordinator-owned lease manager or raw-frame socket channel is to be implemented. + +## 2. Proposed implementation map + +Begin under the existing Rust workspace; extract physical package locations separately. +The following are proposed names, not files that exist today: + +```text +services/flysim/crates/ + fly-session-types/ scopes, contracts, schema validation, canonical digests + flybus/ generic router/client/wire/artifact store; embedded or standalone + fly-session-rpc/ domain schemas/deduplication over flybus; NOT another transport + fly-session/ phase machine, coordinator, admission, task/executor traits + fly-session-worker/ dispatch shell, status/shutdown, agent/environment adapters + fly-session-store/ participant captures, manifest commit, group recovery + +services/flysim/crates/flysim/ + legacy/ compatibility composition (extract without changing behavior) + composition/ new config/registries and worker launching + +packages/feed/ existing public v1, later public v2 schemas/fixtures +packages/brain/ reference model/readout and new contract-relevant golden generators +``` + +Do not create empty crates to satisfy this tree. In the first slice, types/transport/coordinator +may be modules in one small crate; split when dependencies and consumers justify it. Keep +Melee parser/codecs and Game Boy FFI out of the session-types crate. Worker executables can +be subcommands of one binary initially; process boundaries do not require separate repos. + +## 3. Ordered slices + +### CONTRACT-01 — Executable schemas and trace format + +**Inputs:** [bus spec](bus-v1.md), session RPC, step, worker, state/media and publishing documents. + +**Implement:** Flybus wire schema separately from session domain schemas; common scalar types, +closed enums, method payload validation and schemas; +canonical digests; fixture loaders in Rust/TypeScript. Specify seed derivation and exact +checkpoint envelope bytes before their respective real-agent/store slices. Create the trace +format from step-v1 section 8, distinguishing behavior fields from operational IDs/time. + +**Acceptance:** + +- JSON round trips across both languages; U64/float boundaries reject correctly. +- Duplicate keys, invalid UTF-8, envelopes over 64 KiB and unknown required fields fail. +- Distinguish bus callId, domain requestId, artifact identity and delivery/hold owner tokens. +- Fixtures include rational zero/reduced form, overflow, duplicate ports and analog limits. +- Contract digest is generated from a documented canonical schema set, not source formatting. + +**Stop:** do not wire a real worker until payload ambiguities and retry identity rules agree. + +### BUS-01 — Router and RPC, in-memory and Unix socket parity + +**Depends on:** CONTRACT-01. + +**Implement:** one flybus crate with bounded framing, bus.hello, exclusive service registration, +incarnation-pinned RPC/reply/cancel, typed route/admission errors and independent read/write +dispatch. Begin with artifact-free calls; do not claim full bus conformance until BUS-03. + +**Acceptance:** + +- Partial frames/writes, disconnect after request, lost result and retransmission fixtures. +- No automatic retry/failover; timeout/cancel-after-dispatch reports uncertain execution. +- Service incarnation replacement is visible. Request/reply correlation survives out-of-order + replies; status RPC can respond while another handler is delayed. Saturation is bounded. +- Both transports produce equivalent behavior traces for the same scenario. + +### BUS-02 — Pub/sub, retention and backpressure + +**Depends on:** BUS-01. + +**Implement:** exact topics, subscribe/unsubscribe, bounded FIFO and latest policies, optional +retained latest/clear, per-recipient delivery IDs/consumption credits and fair control lanes. + +**Acceptance:** overflow rejects a bounded publication before partial fan-out; latest replaces +only queued messages; delivery consumption returns credits; unsubscribe preserves already- +delivered ownership; retained replay is ordered; stalled observers cannot starve RPC replies. +Durable event history is a storage client, not a second broker built into Flybus. + +### BUS-03 — Artifact-backed messages and automatic lifetimes + +**Depends on:** BUS-02. + +**Implement:** immutable file-backed store, allocate/seal/open, bus attachment validation, +producer/queue/delivery/retention owners, RAII DeliveryGuard and explicit cache holds. Root +creation/admission is atomic. Start with ordinary local files/tmpfs; no pooled slot reuse yet. + +**Acceptance:** last owner collects; extracted handle survives message drop; forward-before- +release is safe; lost replies/cache replay remain valid; disconnect releases logical ownership +without mutating still-mapped bytes; retained latest and queue replacement release correct roots. +Measure 640×480 RGBA×60 with three readers: one stored image, no raw pixels in router messages, +bounded CPU/RSS/owners/queues. Record reader/copy costs rather than claiming zero-copy capture. + +### SESSION-01 — Synthetic sequential transaction + +**Depends on:** BUS-03. + +**Implement:** small fake agent workers, one counter/arena environment, identity executors +and a deterministic task. Follow Prepare→Advance→Evaluate→Commit exactly. Use explicit seeds +and rational clock accumulation; the fake model must expose a mutation counter for tests. +Implement domain request deduplication/result caches over bus calls; retaining result artifacts +is an endpoint responsibility. Domain Acknowledge differs from bus delivery.consumed. + +**Acceptance:** + +- One world advance per complete batch; every agent Prepared before Advance. +- Task evaluates once; every agent commits before next Prepare or committed publication. +- A synthetic 60-Hz/1-ms profile produces 16,17,17 ticks and remainder zero after three steps. +- Pause mid-step completes the step and pauses at its committed boundary. +- Bootstrap/warm-up cannot advance the environment or produce gameplay rewards. +- Domain retries use a new callId with the original requestId; they never repeat ticks/reward. + +### SESSION-02 — Parallel processes and fault behavior + +**Depends on:** SESSION-01. + +**Implement:** one agent process per fly and one environment process under the coordinator; +compare with in-process and dedicated-thread variants. Enforce total thread budgets and +configured agent/port identities. Failure stops the epoch rather than neutralizing a player. + +**Acceptance:** sequential, reversed order and parallel completion produce equivalent traces; +delayed one-agent result holds the world; worker/helper death has a bounded diagnosed outcome; +an uncertain Advance never creates a second batch; partial Commit never permits next-step play. + +### MEDIA-01 — Native observation schemas and presentation handoff + +**Depends on:** SESSION-02. + +**Implement:** view/sample descriptors and producing-step validation on top of bus ArtifactRef, +not a second buffer system. Environment outputs native media; sensor transforms remain agent +profiles, while presentation owns viewer resizing/composition/audio/streaming. + +**Acceptance:** bad strides/lengths/producer times fail; shared image reaches both agents through +owned attachments; spectators use latest subscriptions and cannot corrupt sensory state; +delayed rendering retains its handle. Distinguish AssetRef from transient ArtifactRef. + +### AGENT-01 — Existing neural core worker + +**Depends on:** SESSION-02, MEDIA-01 and the profile/identity foundation in the broader backlog. + +**Implement:** adapter over existing LIF, plasticity, retina and fixed readout primitives; +reference-first composition/goldens; independently seeded agent state and shared immutable data. +Avoid using the old whole-frame `tick` wrapper if it changes the specified phase ordering. + +**Acceptance:** per-agent state agrees with the reference across Prepare/Commit, stimulation, +zero/nonzero reward, warm-up and pauses. Two agents cannot share gains/RNG/holds; swapping +dispatch order and varying worker count preserves results. Keep 64-role limits explicit. + +### ENV-01 — Game Boy compatibility environment + +**Depends on:** AGENT-01 and environment/task extraction in the broader backlog. + +**Implement:** binjgb environment, task-local memory inspector and identity/existing action +adapter. Keep `legacy-gameboy-v1` separately routed with exact old ordering/hash semantics. + +**Acceptance:** legacy fixtures/goldens/restore outcomes unchanged; the new composition has +its own identity and public adapter selection. No console-specific state enters generic +session types. ROM-backed checks are optional explicit jobs, not required downloads. + +### STATE-01 — Coherent all-participant checkpoint/recovery + +**Depends on:** SESSION-02, MEDIA-01; validate with fake agents first, then AGENT-01/ENV-01. + +**Implement:** exact new envelope schema, compatibility manifest, Capture/StageRestore/ +ActivateRestore, bounded writer, durable commit acknowledgments and fresh-epoch fencing. +Keep old `FLYSIM01` reader separate. Payloads and coordinator state must refer to one boundary. + +**Acceptance:** uninterrupted versus resumed synthetic/real-agent traces match after accounting +for new epoch metadata; corrupt any participant and installation fails as a group; lost save +reply doesn't advance durable metadata; failure during activation cannot resume half a world. +Checkpoint queue stress remains bounded; verify old media/parser data cannot cross recovery. + +### PUBLISH-01 — Committed snapshots and observer isolation + +**Depends on:** SESSION-02, MEDIA-01; public v2 contract work is a separate prerequisite to +publishing a supported multi-agent browser feed. + +**Implement:** internal [publication boundary](publishing-v1.md) over the SAME bus, latest-value +observations, application-owned state/cues, bounded events, descriptor repair/query and a fake +multi-agent consumer. Presentation gateway owns browser delivery; no generic show/tournament +service or codec is added to the router. Then implement +the approved public v2 wire schemas/fixtures and stage adapters together. + +**Acceptance:** browser disconnect/backpressure never advances/stalls the world; future agent +state is not mixed with old media; descriptor/index mismatch is visible; committed actions +are labeled as the transition that just ended. PNG/browser review gates apply to actual UI. + +### DOLPHIN-01 — Substitute the backend, not the coordinator + +**Depends on:** measured MELEE-01/02 spikes from the [Melee audit](../melee-framework-audit.md), +SESSION-02, MEDIA-01 and declared recovery support. + +**Implement:** bus-connected helper over pinned Dolphin/libmelee or the chosen narrow hook, +complete port batch adapter, frame-identified sensory output and task-local Melee inspection. +Keep actual rendering/input/save semantics behind the same Environment API. + +**Acceptance:** all generic backend conformance tests plus delayed-port/flush ordering, +game-frame reset, parser recovery, pixel latency and neutral/release tests. If exact snapshot +is unsupported, advertise episode-restart and use only the matching prototype policy. + +## 4. Failure injection checklist + +Tests must deliberately inject these cases; success-path demos are insufficient: + +| Injection | Required invariant | +| --- | --- | +| Duplicate Prepare after lost reply | No extra ticks, RNG draws, stimulation or decode | +| Same batch with altered controls | Conflict, never a second world mutation | +| Lost Advance result after world step | Resolve same operation or fail epoch | +| One Commit fails after another succeeds | No next world step; coherent recovery only | +| Old worker replies after restore | Stale epoch/incarnation rejected | +| Message drops while extracted image is rendering | DeliveryGuard keeps bytes alive through last use | +| First agent releases a shared image early | Artifact remains until every other owner finishes | +| Cached RPC artifact is consumed by its first caller | Domain cache still owns it for retry | +| Latest queued frame is replaced | Only that queue root drops; in-use images remain valid | +| Router restarts during a world advance | Old handles/routes invalid; epoch fails and restores coherently | +| Viewer holds output indefinitely | Only spectator data is dropped/disconnected | +| Backend waits for input while capture is requested | No deadlock; capture only at valid quiescent boundary | +| Capture writer stalls | Finite queue/memory, honest durable status | +| StageRestore validates three participants, fourth fails | Nothing is resumed | +| ActivateRestore fails halfway | Group remains fenced, no new gameplay | +| Old epoch audio arrives after reset | Discontinuity handling; no stale playback as current | + +## 5. Verification and handoff + +Before each implementation merge, run the repository's required `npm test`, +`npm run typecheck`, `cargo test --workspace` from the Rust workspace, and +`infra/tests/lint.sh`. Run affected browser/PNG gates for presentation changes. Keep the +existing committed FAFB real-data goldens mandatory; larger new datasets and game-backed +jobs report explicit optional skips. + +Performance checks report total physical-core allocation, one/two/four agents, within-agent +worker count, router/RPC latency, artifact production/read/copy time, critical-path percentiles, +memory peaks, owner/GC statistics and +bounded queue behavior. No host capacity claim follows from a local synthetic timing test. +Deployment-host work is separately authorized/claimed/serialized under repository rules. + +Every completed slice leaves: + +1. The implemented contract/schema revision and compatibility decisions. +2. A minimal runnable synthetic example and exact test commands/results. +3. Behavior traces demonstrating its acceptance criteria. +4. Known unsupported capabilities and remaining measured questions. +5. Updated planning status, with no claims that a stub provides real emulator semantics. + +Do not start by moving every directory, adding a service mesh, or rewriting the model. The +first useful deliverable is the small artifact-backed bus example, followed by the synthetic +distributed step transaction using it. No one-off communication stack per component. diff --git a/docs/design/session-framework/ipc-v1.md b/docs/design/session-framework/ipc-v1.md new file mode 100644 index 0000000..98778f9 --- /dev/null +++ b/docs/design/session-framework/ipc-v1.md @@ -0,0 +1,207 @@ +# Session RPC contracts over Flybus + +Status: **draft 2**, 2026-09-18. The filename is retained for existing links. This document +now defines **domain contracts carried by [Flybus v1](bus-v1.md)**. It no longer defines a +separate socket protocol, direct worker connections, or coordinator-owned buffer service. +The [architecture index](README.md) states scope and precedence. Public feed/control v1 stay +unchanged; this is the new internal session path. + +## 1. One transport, domain-specific meaning + +Every session/worker RPC is a Flybus call to a named, incarnation-pinned service. Pub/sub, +application supervision and artifact bookkeeping use the same bus. The router moves messages; +the receiver validates its method payload and the [session step machine](step-v1.md). + +The bus owns framing, connection identity, route registration, bounded delivery and artifact +ownership. This document owns Scope, model-related scalar types, worker capability negotiation, +domain operation deduplication and errors. Domain request identity is independent of the bus +callId: a safe retry has a new transport callId but the original domain requestId/body. + +## 2. Common domain types + +```ts +type Id = string; // ^[a-z0-9][a-z0-9._-]{0,63}$ +type U64 = string; // "0" or [1-9][0-9]*; <= 18446744073709551615 +type Digest = string; // 64 lowercase hexadecimal digits (SHA-256) +interface Scope { sessionId: Id; epoch: Id; step: U64 } +interface RationalNs { numerator: U64; denominator: U64 } +interface SchemaRef { id: Id; version: number; digest: Digest } +interface TypedValue { schema: SchemaRef; value: object } +interface SessionRpcRequest { requestId: Id; scope: Scope | null; params: object } +``` + +All fields are required unless marked `?`. Schema version is integer 1..65535. Fractions +are reduced, denominators positive, durations positive; zero is encoded 0/1. Arithmetic is +checked. JSON numbers representing rates/rewards/controls are finite. Counters/clocks use +decimal strings. Task/profile schemas bound collections and numeric ranges before mutation. +First session composition limit: 4 agents, 4 ports and 64 rate roles per agent; these are +session/model limits, not limits on the number of application personas or generic bus clients. + +Each TypedValue has a canonical JSON size limit of **32 KiB**, while the complete envelope +must still fit Flybus's 64-KiB maximum. Large typed state goes in a listed Artifact attachment +under an explicit schema, not an oversized inline object. Changing the old draft's 1-MiB +worker envelope to Flybus must not silently truncate a payload. + +## 3. Request/reply mapping + +Illustrative client call: + +```text +bus.call( + target = {service: "agent.fly-a", expectedIncarnation: pinnedRegistration}, + method = "Agent.Prepare", + payload = {requestId: "req-41", scope: {sessionId, epoch, step: "41"}, params}, + attachments = ownedArtifactHandles +) +``` + +Flybus's eventual rpc.result `outcome` is one of: + +```ts +interface SessionRpcSuccess { + type: "result"; requestId: Id; workerId: Id; incarnationId: Id; + scope: Scope | null; result: object; +} +interface SessionRpcFailure { + type: "error"; requestId: Id; workerId: Id; incarnationId: Id; + scope: Scope | null; + error: { code: ErrorCode; message: string; mutation: "none" | "applied" | "unknown" }; +} +``` + +Replies echo the original scope. The receiver identity and bus service incarnation must +match the negotiated worker. Bus route/admission failure is not a SessionRpcFailure produced +by the handler. A bus admission acknowledgment is not an Agent.Prepare/Environment.Advance +completion. Only a matching terminal domain reply resolves a simulation phase. + +ArtifactRefs inside request/result payloads must be declared in bus attachments and backed by +live owned handles. Domain canonical-body digests include the references but exclude changing +bus callIds, deliveryIds and owner tokens. A cached result owns Artifact handles independently +of the first delivery; it is not a JSON object holding unowned pointers. + +## 4. Worker negotiation and status + +After bus connection/registration, call Worker.Hello (`scope:null`): + +```ts +interface HelloParams { + sessionId: Id; expectedWorkerId: Id; + role: "agent" | "environment" | "coordinator"; + supportedMajors: number[]; +} +interface HelloResult { + selectedMajor: 1; selectedMinor: 0; + workerId: Id; incarnationId: Id; role: "agent" | "environment" | "coordinator"; + buildDigest: Digest; contractDigest: Digest; + capabilities: Id[]; + limits: { maxAgents: number; maxPorts: number }; +} +``` + +The bus supplies caller identity; do not accept a forged caller in params. Bind a worker's +session authority to the expected coordinator identity/incarnation during negotiation and +initialization. Wrong worker/role, no common major or missing required capability refuses +the composition. Required capabilities are agent-step-v1 and world-step-v1 for their roles; +checkpoint-v1 and pixel-observation-v1 are conditional. Artifact transport capability is +negotiated once by Flybus, not as another worker memory API. + +Worker.Status has params `{}` and the caller's last known scope (null before initialization): + +```ts +interface StatusResult { + state: "uninitialized" | "ready" | "preparing" | "prepared" | "advancing" + | "committing" | "capturing" | "staged-restore" | "restoring" + | "failed" | "stopping"; + currentScope: Scope | null; + activeRequestId: Id | null; lastCompletedRequestId: Id | null; + lastBatchId: Id | null; progressCounter: U64; +} +``` + +ProgressCounter advances on computational/phase progress, not on status queries. Dispatch +Status through the same service without waiting for a long numerical operation. One mutation +executes at a time; at most one may be pending, and normal stepping pipelines neither. The +router's larger RPC capacity is not permission to overlap worker mutations. Never hold a +simulation lock while waiting for network I/O, artifact resolution or release bookkeeping. + +## 5. Domain idempotency and retention + +`requestId` is `req-` plus a canonical U64 serial, increasing for newly issued operations +per caller/worker pair. Retries reuse it unchanged even though bus callId changes. Flybus +preserves first-dispatch order per caller/service; worker handlers maintain that request +admission order while allowing read-only status alongside compute. + +The operation key for step mutations is `(sessionId, epoch, step, method, workerId)`. +There is at most one Prepare, Commit or Advance for that key. + +- Same key/request/body returns its cached reply, with fresh bus delivery ownership over + retained artifacts. It never repeats ticks, stimulation, controller execution or reward. +- Changed ID/body for an existing key is CONFLICT. Canonical comparison uses RFC 8785 over + method, scope and validated params. A rejected duplicate does not undo the earlier result. +- Check retained request identity before phase checks or artifact dereferencing. A duplicate + may arrive after the original input delivery was consumed; it needs only the cached result. +- Keep current/immediately previous step result records. Eviction never enables reexecution: + highest-issued request serial and step watermarks reject expired retries/old steps. +- Original expired serial → RESULT_EXPIRED; fresh serial naming an old step → STALE_STEP. + +An exact duplicate arriving while execution is active receives the terminal domain error +IN_PROGRESS for that **bus call**. The original bus call still completes normally. Retry or +query Status later; no second mutation is started. This avoids multiple terminal replies to +one bus call and does not mistake IN_PROGRESS for the original operation's failure. + +Lifecycle/capture replies are retained until Worker.Acknowledge: +`params:{requestIds:Id[]}` (1..16), result `{acknowledged:Id[]}`. It drops domain cache handles, +not another consumer's bus delivery. Already released/unknown IDs are ignored. Serial +watermarks reject reuse after acknowledgment without an unbounded tombstone list. + +Bound unacknowledged lifecycle replies at 16, then BUSY before application. Status and +Acknowledge use a cache of their last 16 replies; current/previous step records have their +separate finite retention. Caches containing big artifacts consume bus owner/byte budgets; +configure capture limits consistently. Never evict a promised replay artifact but keep a +successful pointer-only reply. An intentionally expired result returns RESULT_EXPIRED. + +## 6. Timeout and failure handling + +Timeouts are measured on the caller's monotonic clock. Prototype defaults: probe after two +seconds without reply, fail after ten seconds without progress; long boot/capture have separate +budgets. These are failure-detection values, not a gameplay latency goal. + +After an uncertain call: + +1. Stop further world-step dispatch. +2. If the same bus/service/worker incarnation still exists, query Status or issue a fresh + bus call with the original domain requestId/body and retained input attachments. +3. Resolve only a matching terminal result. Never repeat an Advance with a new domain ID. +4. If routes/ownership were lost, incarnation changed or retained result expired, fail the + epoch and restore/reset the group. + +Endpoint crash or bus restart is not covered by in-memory deduplication. Router/store restart +invalidates all transient artifacts and routes. Worker disconnect also invalidates its bus +owners and registration; v1 does not silently reattach that worker to an active epoch. +Recover coherently even if an OS process survived with some numerical state in memory. + +## 7. Domain errors + +| Code | Meaning | +| --- | --- | +| INVALID_ARGUMENT | Invalid schema/range, before mutation | +| UNSUPPORTED | Missing method/capability | +| IDENTITY_MISMATCH | Wrong session/profile/port/build/asset identity | +| STALE_EPOCH / STALE_STEP / FUTURE_STEP | Timeline/order mismatch | +| INVALID_PHASE | Wrong worker phase | +| CONFLICT | Existing logical operation with changed ID/body | +| IN_PROGRESS | Original operation still executing; duplicate bus call did not start work | +| BUSY | Domain capacity unavailable before admission | +| BUFFER_INVALID | Missing/unowned/mismatched artifact or invalid media shape | +| RESULT_EXPIRED | Safe replay is no longer available; never recompute to replace it | +| INCOMPATIBLE_STATE | Restore validation failed before activation | +| BACKEND_FAILURE / INTERNAL | Runtime fault, with explicit mutation certainty | + +Messages are <=512 code points and exclude raw game memory/credentials. Errors after partial +mutation use unknown unless completion is established. No error authorizes skipping a fly, +pressing fallback controls, or continuing a partially committed match. + +Worker.Shutdown, params `{reason:Id}`, returns `{stopping:true}` if responsive and terminates +the worker after replying. It does not imply saved state. Only configured supervisors may +invoke it; workers have no authority to shut down the coordinator. Shutdown/release notifications +travel on the same bus; there is no reverse lease socket or Buffer.Release/Reclaim RPC. diff --git a/docs/design/session-framework/publishing-v1.md b/docs/design/session-framework/publishing-v1.md new file mode 100644 index 0000000..01cd30b --- /dev/null +++ b/docs/design/session-framework/publishing-v1.md @@ -0,0 +1,184 @@ +# Application, presentation and audience contracts + +Status: **draft 2**, 2026-09-18. All internal communication uses [Flybus](bus-v1.md). This +document defines ownership and logical data contracts, not the public feed v2 byte format. +Existing public v1 contracts remain unchanged for the legacy application. + +## 1. Applications orchestrate; presentation owns the show + +Fly Plays Pokémon is an application assembled from simulation and presentation components. +Its supervisory code and interface share an application-owned, versioned state/event schema. +A Melee competition or ecosystem can choose another schema. Director, tournament, bracket, +cast of 32 personas and story segments are examples, not mandatory framework services/types. + +The framework supplies sessions, agents, backend/task interfaces, capability descriptions, +native observations and generic UI/client primitives. The application chooses lifecycle, +profiles, game-aware macros/recovery, identities/history, interventions and narrative behavior. +Frame-by-frame scheduling remains inside the session; the application need not RPC each tick. + +Presentation owns selection/layout, resizing, overlays, compositing, audio mixing, encoding, +browser-facing delivery, recording choices and stream output. Native 480p game data is a +perfectly valid framework output. Neither the router nor generic session assumes Twitch, +1920×1080, particular colors, specific React components or an automatic tournament dashboard. + +## 2. One bus, complementary data sources + +```text +Session ── committed observations/events + artifact handles ──┐ + ├─ Flybus ── presentation application +Application ── own state/events/presentation cues ────────────┘ +Presentation gateway ── application-defined browser delivery ── frontend +``` + +Example addresses (chosen by composition, not recognized by router code): + +| Address | Pattern / purpose | +| --- | --- | +| `session.demo` | RPC: domain lifecycle/status/capabilities; exact methods require session API schemas | +| `session.demo.descriptor` | Pub/sub, retained latest: framework descriptor | +| `session.demo.snapshots` | Pub/sub, latest: committed simulation values and native media refs | +| `session.demo.events` | Pub/sub, bounded: scoped domain events; not a durable log | +| `app.pokemon.state` | Pub/sub, retained latest: application-specific state | +| `app.pokemon.cues` | Pub/sub: application narrative/presentation events under declared delivery policy | +| `app.pokemon` | RPC: application queries/admission, e.g. restore UI state or request a supported effect | + +Descriptor revisions and scope link observations to schemas. Cross-topic ordering is not +guaranteed; a subscriber receiving an unknown descriptor revision must fetch it through the +application/session query contract or buffer a bounded number of snapshots, not infer shape. +Latest retained descriptors accelerate startup; RPC querying remains the repair path. + +## 3. Common simulation descriptors and committed values + +Types use [session RPC](ipc-v1.md), [workers](workers-v1.md), [state/media](state-media-v1.md): + +```ts +interface SessionDescriptor { + sessionId: Id; revision: U64; compositionDigest: Digest; + schedulerId: "lockstep-v1"; + environment: EnvironmentDescriptor; taskSchema: SchemaRef; + agents: { + agentId: Id; portId: Id; profileDigest: Digest; + datasetDigest: Digest; indexDigest: Digest; neuronCount: U64; + rateRoles: Id[]; supportedStimuli: Id[]; + }[]; + assets: AssetRef[]; +} +interface CommittedSnapshot { + descriptorRevision: U64; publisherIncarnation: Id; + scope: Scope; episodeId: Id; sequence: U64; worldTime: RationalNs; + agents: { + agentId: Id; telemetry: AgentTelemetry; + selectedDecision: TypedValue | null; + appliedControls: PortControl | null; + }[]; + progress: TypedValue; + media: { views: ViewRef[]; audio: AudioRef[] }; + eventIds: Id[]; +} +``` + +Publish only after all agent commits establish Ready(k). Decisions/controls describe the +transition ending at that boundary, null at initial boundary 0. Health updates are separate +and never claim an uncommitted future boundary. Every transient media reference is a declared +bus attachment held through publication admission. Ordinary snapshot publication is latest/ +bounded and never waits for a spectator to consume it. + +Publication sequence is monotonic within publisherIncarnation. Epoch determines simulation +timeline; router topicSequence only determines bus acceptance order. Never equate these. +Geometry/spike mapping requires indexDigest, not merely the same number of neurons. Persistent +AssetRefs survive release packaging; ephemeral ArtifactRefs never become permanent asset URLs. + +## 4. Flexible data, authored UI + +Common UI primitives should understand media, agents, controllers, typed measurements, +progressions, collections and events. Descriptors change infrequently; values change frequently. +Application-specific structures remain namespaced, schema-validated extensions, such as +`pokemon.progress.v1` or `melee.match.v1`. They are not mandatory fields on every snapshot. + +Proposed measurement vocabulary to formalize with public v2 schemas: + +```text +Definition: id, owner, label, kind, unit, optional range, schema revision +Sample: id, producing scope/time, validity, value +Validity: measured | unknown | unsupported | stale +``` + +Kinds include number/counter, gauge, duration, state, progression and collection with typed +item schemas. A measured zero is distinct from unknown. Stale values retain original timestamps. +Units/ranges are metadata, not pixel sizes. Unknown optional extensions may be omitted or shown +generically; unsupported required schemas are visible errors. No remote executable UI payloads. + +Application state carries whatever the experience needs: progress history, featured fly, +competition records, season state or sponsor effects. It is developed with its presentation, +not forced into a framework-wide “show state”/tournament schema. An application can reuse +generic components and add its own panels without changing worker/transport contracts. + +## 5. Artifact consumption and browser boundary + +The native presentation client is a regular bus subscriber. Its renderer may hold an extracted +Artifact after dropping the message; the SDK delays consumption until actual use finishes. +Latest coalescing only drops queued values. A stalled consumer is constrained by finite credits, +owners and store budgets; it cannot make the router overwrite an in-use image. + +The browser does not receive private owner tokens or local storage paths. A presentation +gateway resolves/copies/encodes artifacts into its chosen browser transport and then drops +its bus handles. That is an application-edge adapter, not a second framework communications +stack. Compositor/encoder/recorder processes inside the application can exchange their own +artifacts through the same bus when useful. + +Dense spike publication is optional and identifies agent, index digest and covered ticks. +The runtime need not publish every neuron every millisecond. Required sensory data and +optional spectator data have distinct budgets; UI focus never changes an agent's input, +controller assignment or an already resolved stimulation/effect target. + +## 6. Events, persistence and recovery visibility + +Events identify session/epoch, source boundary, episode, optional agent, kind and typed payload. +Task events are emitted after committed transitions; capture/admission events describe their +actual phase. Bus publish acceptance and delivery consumption are not durable acknowledgments. + +When durability is required, call a configured event-store client/service (over the same bus) +and await its append/commit acknowledgment under the session's configured policy. Pub/sub +remains useful for live observers; reconnecting clients query durable history through ordinary +RPCs. The initial conformance policy pauses at the next safe boundary if durable event +admission/commit fails, retaining only a bounded pending batch. No hidden durable broker queue. + +After rollback publish old/new epochs, checkpoint identity and abandoned step ranges. +Application history can mark outcomes aborted/superseded; it does not erase records merely +because emulator time moved backward. Media reports the corresponding discontinuity. + +## 7. Supervisory and audience effects + +Application supervision uses bus RPCs for configured lifecycle/intervention capabilities +and pub/sub for application state/cues. The Twitch adapter can be a constrained bus client +of the application's admission service; viewers/browser clients never obtain worker control. +Legacy HTTP bridge behavior remains until deliberately migrated. + +Effects are declared by the task/backend/profile: valid targets, parameter schema, timing, +duration/stacking, implementation capability and outcome events. Examples include neural +stimulation or future game items/modifiers, where verified implementations exist. Generic +game boons and a public v2 admission schema are follow-on work; initially new-session audience +effects remain disabled. No arbitrary controller/game-memory write endpoint is introduced. + +```text +requested → rejected + → accepted(target, epoch, earliestStep) → scheduled → applied → expired + └─ failed / cancelled + applied → rolled-back +``` + +The application persists interaction identity/target and defines retry, redemption, refund +and recovery policy. A gift is an intervention, not automatically an earned neural reward. +Presentation cues may be immediate; simulation effects apply at declared boundaries. The +supervisor does not bypass the complete-batch step barrier. Chat text remains presentation +data; template-only replies/quiet mode and existing no-public-button rules continue. + +## 8. Presentation acceptance criteria + +- Per-agent/session stores and rate/afterglow state, not one mutable global fly. +- Framework plus application schema streams, with descriptor repair/reconnect behavior. +- Native view dimensions and aspect; application-controlled output resolution and composition. +- Explicit audio ownership, timestamped overlays and bounded queues/discontinuities. +- Last-use artifact release, cached/replay-safe references and slow-observer isolation. +- Game-specific labels/views without Pokémon fields in generic runtime/router schemas. +- Actual UI changes reviewed as PNGs with browser/legibility gates. This contract is not screen approval. diff --git a/docs/design/session-framework/state-media-v1.md b/docs/design/session-framework/state-media-v1.md new file mode 100644 index 0000000..a8c7a86 --- /dev/null +++ b/docs/design/session-framework/state-media-v1.md @@ -0,0 +1,212 @@ +# Session artifacts, native media and recovery + +Status: **draft 2**, 2026-09-18. [Flybus](bus-v1.md) owns generic artifact storage, delivery +ownership, retention and garbage collection. This document specifies the **domain meaning** +of those artifacts: native observations, clock association and coherent session checkpoints. +Read [session RPC](ipc-v1.md), [step ordering](step-v1.md) and [worker interfaces](workers-v1.md). + +## 1. Use the bus ArtifactRef + +Large payload fields use `ArtifactRef` from bus-v1, and every referenced artifact is listed +in the surrounding bus attachments. There is no separate BinaryRef, buffer-region registry, +coordinator lease endpoint, or Buffer.Release/Reclaim protocol. Profile/dataset release assets +use `AssetRef` (a persistent content identity); transient bus ArtifactRefs are not those assets. + +The environment publishes one immutable native image. The coordinator may forward the same +owned handle to multiple agent Commit calls and publish it for presentation. Flybus creates +destination ownership before releasing the source. It does not send another copy of the +pixel bytes per recipient through its sockets. + +An agent consumes/encodes pixels during Initialize/Commit and drops its handle when no longer +used. A renderer may keep its extracted handle after dropping the message; the DeliveryGuard +keeps the artifact alive until rendering has finished. A domain cached RPC result keeps its +own handles so replay remains valid after the original recipient consumes its delivery. + +Content digests are optional on transient live frames, mandatory on checkpoint payloads and +persistent asset import. Ownership/index/byte-shape validation is always required. A digest +does not replace epoch or observation-time identity. + +## 2. Native observation types + +```ts +interface ViewDescriptor { + viewId: Id; + width: number; height: number; + format: "rgba8"; rowStride: number; + pixelAspect: { numerator: number; denominator: number }; + observationDelaySteps: number; +} +interface ViewRef { viewId: Id; producedStep: U64; pixels: ArtifactRef } +interface AudioDescriptor { + streamId: Id; sampleRate: number; channels: number; + format: "f32le-interleaved"; +} +interface AudioRef { + streamId: Id; firstSample: U64; sampleFrames: number; + samples: ArtifactRef; discontinuity: boolean; +} +``` + +Epoch is inherited from the domain observation; the bus treats it as opaque payload. View +dimensions are integers 1..4096, rowStride exactly 4×width, no padded rows in v1. Pixel aspect +numerator/denominator are positive integers <=65535; observationDelaySteps is integer 0..8. +Pixels are top-left RGBA8 and artifact length equals rowStride×height. Other formats require +a media-schema change, not special-case code inside the router. + +Required sensory views have producedStep equal to +`max(0, observation.boundary - observationDelaySteps)`. Bootstrap may repeat O[0] until the +declared pipeline delay fills. Beyond that, missing/extra-delay sensory input is a step +failure, not an arbitrary latest frame. Observer publication may omit/coalesce frames while +preserving each artifact's actual producing boundary. + +Audio sampleRate is integer 8000..192000, channels 1..8, sampleFrames 0..192000 per chunk. +Samples are finite f32; artifact length is sampleFrames×channels×4. firstSample identifies +the sample position relative to the episode's configured audio origin, with intended PTS +firstSample/sampleRate. Crash restore preserves sample position under a new epoch; first +chunk marks discontinuity. Within an epoch, chunks cannot overlap or go backwards. + +The environment provides **native game output**. Sensor transformations belong to the agent +profile. Resizing for viewers, overlays, composition, audio mixing/resampling, encoding, +browser delivery and streaming belong to the application/presentation layer. No bus or +generic session configuration assumes a 1080p show or Twitch output. + +640×480 RGBA at 60 fps produces 73.728 MB/s of raw image data. Artifact fan-out references +one stored object; reads and any staging/seal copy still consume memory bandwidth. This is +reasonable to measure before introducing codecs or pooled GPU buffers. Native dimensions +come from the backend, not a hardcoded GameCube or broadcast resolution. + +## 3. Domain retention and backpressure + +Use the same bus call/publish API for observations and artifacts. Bus ownership tracks bytes; +the session decides which observations are required and when they have been used. + +| Use | Rule | +| --- | --- | +| Required agent input | Retain through encoding/Commit; no coalescing or overwrite | +| Step-result replay | Retain in the endpoint's current/previous-step cache until domain eviction | +| Spectator snapshot | Latest subscription, finite in-flight credits; release after actual use | +| Long rendering/storage job | Explicit artifact hold with a finite byte/count budget | +| Hot checkpoint | Coalesce only queued replaceable captures, releasing their holds | +| Durable checkpoint | Acknowledge after durable commit; reject/defer before capture when saturated | + +Initial session defaults: two outstanding coherent captures and at most the bus-configured +latest/in-flight frame credits per observer. Presentation audio can target 250 ms and cap at +one second, but that is a presentation policy, not a bus or brain-clock requirement. + +Budget cached step observations, active agent deliveries, retained latest and spectator holds +together. The producer dropping its handle does not free cached/queued/in-use objects. +A slow spectator exhausts its own credits; new latest messages replace its queued value. +If it violates configured resource policy, disconnect/restart that observer instead of freeing +live data or silently skipping simulation input. Global store exhaustion is an explicit fault +or pause condition; the router cannot guess that a particular live object is disposable. + +No coordinator tracks per-reader socket acknowledgments or calls a producer's reclaim method. +The SDK and bus perform that bookkeeping. File-backed immutable mappings are safe after +unlink; physical pages disappear when all OS mappings close. Pooled reuse is deferred until +it provides equivalent safety. Bare handles in history are not persistent saved bytes. + +## 4. Checkpoint identity and content + +Exact-checkpoint sessions capture at Ready(k) or Paused(k), after all Agent.Commit replies, +with no Advance in flight. Block next Prepare until all participants supply immutable captures. + +The manifest records: + +- Envelope version, checkpoint ID and source session/epoch/step/episode/world time. +- Coordinator scheduler/configuration identity and exact port-to-agent map. +- Backend/content/patch/controller/parser/state-format compatibility. +- Per-agent profile/dataset/model identities, seed, tick count/remainder and payload digests. +- Task ledger, prior world inspection, per-agent executor state, next sensory/decision state + or reproducible reconstruction inputs, admission state and event watermarks. +- Payload names, lengths and hashes, including external-helper state required for exact resume. + +Use a new envelope version; specify exact byte layout before production files. The historical +letter-only chunk-name constraint is not silently widened, and FLYSIM01 remains separately +readable. Persist payload **bytes and durable content identity**, not transient bus storeId, +artifact IDs, ownership tokens, mappings or pointers. + +A checkpoint writer owns the bus Artifact handles until bytes are committed or the job fails. +It then drops them; durable files are outside Flybus's ephemeral GC. On restore, the durable +store imports fresh immutable bus artifacts. sourceScope is provenance, while the new handles +belong to the current router/store. Broker retention is never a substitute for a checkpoint. + +## 5. State RPCs over the bus + +Common to workers advertising checkpoint-v1: + +```ts +interface CaptureParams { checkpointId: Id } +interface CaptureResult { + checkpointId: Id; boundary: U64; + compatibilityDigest: Digest; + payload: ArtifactRef; // listed attachment; digest required +} +interface StageRestoreParams { + checkpointId: Id; sourceScope: Scope; + compatibilityDigest: Digest; + payload: ArtifactRef; // newly imported owned attachment +} +interface StageRestoreResult { checkpointId: Id; restoreToken: Id } +interface ActivateRestoreParams { restoreToken: Id } +interface ActivateRestoreResult { + committedStep: U64; checkpointId: Id; + observation: WorldObservation | null; // environment required, agent null +} +``` + +State.Capture uses the committed scope. It completes after immutable capture exists, not when +a backend save was requested. The cached reply retains its artifact until Worker.Acknowledge. +The coordinator/writer obtains its own live ownership before acknowledging that cache. + +State.StageRestore uses a proposed **new epoch** at source boundary k and is allowed only +on an uninitialized replacement or a quiescent worker. Launcher configuration supplies the +expected profile/backend identities; no implicit warm-up/reinitialization changes the saved +brain. It validates into replacement state, without exposing mutations to the live session. + +After every participant and coordinator state validates, State.ActivateRestore installs +each staged token under that new scope without advancing a tick. Tokens are bound to scope/ +payload/checkpoint and can activate only once; duplicate domain requests replay the cached +reply, while a fresh request trying to reuse an activated token is a conflict. + +The environment returns the coherent restored observation with fresh artifact references and +restored time. It cannot advance gameplay to manufacture it. Capture/reconstruction therefore +covers render/inspection state and any pending sensor pipeline. Agent state agrees with it; +do not replay reward or recalibrate merely to fill missing cached data. + +If emulator validation requires mutation, stage a stopped replacement emulator. If that cannot +provide externally atomic resume, advertise episode-restart, not exact-checkpoint. After all +activation acknowledgments, install the coordinator's staged task/executor/admission state +and establish Paused(new epoch,k). Failure during activation never permits half a group to run. + +## 6. Durable commit, router failure and recovery + +Write payload/envelope temporary generation, fsync, rename, fsync directory, then atomically +write/fsync/rename the store manifest and fsync its directory. **Manifest commit is the durable +commit point.** Unreferenced temporary generations are not automatic restore candidates. + +Publish distinct captured/queued/committed/failed/superseded events over the same bus. Only +durable completion produces a saved acknowledgment/high-water mark. Failed writes release +owned ephemeral captures according to retry policy, without reporting false durability. + +After participant/coordinator/router failure: + +1. Stop steps, abandon the epoch and fence old participants/routes. +2. Connect to a live router and select a complete compatible durable checkpoint. +3. Import its payloads as new artifacts; stage/activate every participant and coordinator. +4. Verify identity/boundary, flush old media/parser queues and publish recovery/discontinuity. +5. Establish Paused(k), then resume only after the group invariant holds. + +A router restart loses ephemeral topics, queues, roots and correlations. Continuing with +old handles is invalid even if some mapped bytes survived. Reconnect is not transparent +mid-step recovery. The durable log records old/new epochs and abandoned step ranges; rollback +can lose post-checkpoint work. Durable input replay requires a separate application/session +journal policy, not an exactly-once claim about Flybus. + +## 7. Episode reset + +Reset differs from crash restore. The application selects a policy; the coordinator records +the old episode's result/abort and creates a new epoch/episode at step zero. World initial +state and retained/fresh brain components are explicit. Gain retention, eligibility/hold +clearing, calibration and first sensory input are part of the policy, tested independently. +Legacy Pokémon ratchet behavior remains in the legacy composition. Shared competitive worlds +never restore one player's environment independently of the other players. diff --git a/docs/design/session-framework/step-v1.md b/docs/design/session-framework/step-v1.md new file mode 100644 index 0000000..3dea51f --- /dev/null +++ b/docs/design/session-framework/step-v1.md @@ -0,0 +1,226 @@ +# Lockstep step protocol v1 + +Status: **draft 2**. This is the authoritative new-session ordering contract. All method +arrows below are RPCs through the same [Flybus router](bus-v1.md); the router itself never +implements the barrier. Read [architecture](README.md) and [session RPC](ipc-v1.md) first. Method payloads are in +[worker interfaces](workers-v1.md). + +## 1. Committed boundary + +At `Ready(epoch, k)`: + +- Environment state is at boundary `k`, with no outstanding action batch. +- Every agent has consumed the outcome of transition `k-1 → k`, including its new encoded + sensory input and rewards, and is ready to compute the decision for transition `k → k+1`. +- Task ledger, action-executor state, event identity and agent tick remainders agree with `k`. +- The current sensory observation may have a declared fixed render delay; its producing + boundary is explicit. “Ready” does not imply latest wall-clock screenshot. +- No normal step operation from an older boundary may mutate the session. + +Only a committed boundary is eligible for a coherent checkpoint or normal pause. Boot/reset +establishes the same invariant with no preceding reward. The public snapshot represents this +boundary, not an in-progress combination of some new agent states and an old world. + +## 2. State machine + +```text +Starting → Ready(k) → Preparing(k) → Applying(k) → Observing(k+1) + ↑ │ + └──────────── Ready(k+1) ← Committing(k) ┘ + +Ready(k) → Paused(k) → Ready(k) +Ready(k) / Paused(k) → Capturing(k) → same boundary +any unresolved partial failure → Failed → Restoring(new epoch) → Paused(k) +terminal episode → Paused(k) → Resetting(new epoch) → Ready(0) +``` + +`Committing(k)` refers to completing transition `k → k+1`. Requests throughout that +transition carry `scope.step=k`; result fields identify `nextStep=k+1` where applicable. +Do not send Agent.Commit with step `k+1` merely because the observation is newer. + +## 3. Transaction sequence + +### Phase A: prepare all agents concurrently + +At Ready(k), freeze the task's per-agent decision contexts and the coordinator's admitted +pre-step stimulation list. Inputs accepted after this cut wait for the next boundary. + +Send `Agent.Prepare(scope=k)` to every active agent. Each worker: + +1. Verifies its committed boundary/profile/context and applies admitted pre-step stimulation + in deterministic command sequence order. Chat text is never included. +2. Advances the numerical model for the environment interval, using the input encoded at + the preceding Commit (or initialization). +3. Reads rates and performs the fixed readout with the declared decision context. +4. Stores and returns `PreparedDecision`; it then enters Prepared(k) and waits for Commit. + +This operation **mutates** the brain, RNG, clock and decoder. “Prepare” does not mean a +database transaction that can be rolled back cheaply. If another agent fails, do not ask a +prepared agent to prepare again or advance to the next step. Resolve/recover the whole session. + +All agents see the same environment interval and the same world boundary, with only their +permitted view/context differences. Their completion order never affects port/action order. + +### Phase B: build and apply one complete batch + +After every PreparedDecision arrives: + +1. Validate agent IDs, intent schemas and profile identities. +2. Run each task-local action executor once, in sorted agent-ID order, against coherent current + game state, task progress/objectives and clock from this boundary. + Direct-control profiles use an identity executor. Macro profiles are explicit extensions. +3. Assemble all configured port controls in descriptor port order; reject duplicates/missing + ports. Uncontrolled ports are configured neutral before the epoch, not supplied ad hoc. +4. Send exactly one `Environment.Advance(scope=k, batchId, controls)`. + +The environment applies all controls at its agreed boundary, advances exactly one interval, +and returns StepResult for `k+1`. It MUST NOT advance another interval while waiting for +the next request. Transport/control scaffolding may have a measured fixed latency; it must +be declared in its descriptor and conformance tests. + +### Phase C: observe and evaluate the task + +The coordinator receives the environment result and verifies batch identity, boundary, +cadence, inspection schema and required sensory views. A missing spectator frame is tolerable; +a missing required sensory input is not silently replaced. + +Call the task's `evaluate_transition` once with old/new inspection observations and applied +controls. It returns scoped rewards/stimulation, next decision contexts, progress/events and +an optional episode request. Commit its ledger update in memory and retain the result for +this transition. No task output directly writes controllers or neural state. + +### Phase D: commit all agent outcomes concurrently + +Send `Agent.Commit(scope=k)` with that agent's next sensory observation and routed outcomes. +Each agent, in this order: + +1. Encodes/installs the next sensory input for the following Prepare. +2. Applies task-derived stimulation in returned event order. +3. Sums that agent's reward values in returned event order and calls reinforcement once at + its current brain time when learning is enabled. A zero sum still follows the profile's + specified legacy-equivalent reinforce behavior; do not optimize it away without evidence. +4. Retains the next decision context/digest and acknowledges committed boundary `k+1`. + +There are no additional neural ticks in Commit. Sugar accepted for a future Prepare is not +silently merged with task reward modulation. The default synthetic learning mechanism remains +separate from the neural stimulation path. + +Once **all** commits succeed, the coordinator advances its committed boundary to `k+1`, +finalizes the observation snapshot and scoped events, drops no-longer-needed artifact handles and +allows the next Prepare. If one Commit fails after others succeeded, the epoch is failed; +there is no partial-match continuation. + +These phases establish logical coordination, not a distributed durable two-phase commit. +Crash recovery returns to the last complete checkpoint, not necessarily the last displayed step. + +## 4. Sequence example + +```text +Coordinator Agent A Agent B Environment + | Prepare(k) ------>| | | + | Prepare(k) ----------------------->| | + |<-- Prepared(A) ---| | | + |<-- Prepared(B) --------------------| | + | [executor + complete port batch] | + | Advance(k, batch-x) --------------------------------->| + |<-------------------- StepResult(k+1, batch-x) ----------| + | [task evaluation; route each reward once] | + | Commit(k, O[k+1], R_A) ->| | | + | Commit(k, O[k+1], R_B) ------------>| | + |<-- Committed(k+1) -----| | | + |<-- Committed(k+1) -----------------| | + | [Ready(k+1); publish; next boundary] | +``` + +The shared camera is one immutable artifact forwarded through bus-owned deliveries; both +workers can encode it without two renders or routing two full images through sockets. +Domain RPC replay caches retain handles, so consumption by one client cannot invalidate a +promised replay. Publication uses bus pub/sub and never waits for spectator consumption. +The coordinator retains each domain request's input handles until its terminal outcome is +resolved, beyond the shorter bus-admission lifetime, so safe domain retries still have valid +attachments. If ownership is lost, fail/recover instead of sending bare expired references. +The coordinator cannot publish a committed state as soon as the faster agent answers. + +## 5. Time and pacing + +The environment descriptor supplies a fixed reduced `stepDuration: RationalNs`. Each agent +profile supplies `tickDuration: RationalNs`. The existing LIF adapter uses exactly one ms. + +For each Prepare: + +```text +accumulator += environment step duration +ticks = floor(accumulator / model tick duration) +accumulator -= ticks * model tick duration +``` + +Use checked rational/integer arithmetic; remainder is always >=0 and < one model tick. +Do not accumulate rounded microseconds or nanoseconds for a fractional frame period. +Persist remainder, executed tick count and warm-up offset. Language implementations must +agree on remainder fixtures. Conversion to the legacy model's f64 millisecond clock must +preserve its representable integral ticks; refuse a run exceeding the supported exact range. + +Example: a synthetic 60-Hz environment with a 1-ms model tick produces 16,17,17 ticks +over three steps, totaling 50. A real backend's measured/declared emulated cadence may +differ; never substitute this example's duration for Game Boy or Dolphin clocks. + +Wall time is only for pacing, health and presentation. The coordinator schedules absolute +deadlines after committed boundaries; when behind, it omits sleep and reports lag. It does +not skip world steps, drop neural ticks, or let one agent advance more slowly than another. +Only one pacing authority is active. Backend throttling and coordinator pacing must be +configured/tested so they do not unintentionally double-throttle the session. + +First-version profiles have a fixed cadence within an epoch. Supporting variable-duration +world advances requires a new capability and tests before enabling it. + +## 6. Initialization, pause and episodes + +Initialize the environment first while stopped, obtaining observation O[0]. Bootstrap the +task and initial decision contexts, then initialize agent workers with their permitted inputs. +Agent warm-up has learning disabled; calibration occurs on settled rates; no warm-up actions +advance the environment. All required acknowledgments establish Ready(0). + +A normal pause request arriving mid-step means “finish this transition, then pause.” It does +not truncate neural computation or capture half an action batch. If completing the transition +is impossible, use failure/recovery, not an apparently successful Pause acknowledgment. +Paused workers retain state and answer Status; world controls do not advance the world. + +Task terminal events are evaluated and their final rewards committed once. Before another +gameplay transition, enter Paused and apply the declared episode policy. Reset uses a new +epoch/episode and step 0. A profile may retain learned gains/brain state, but must identify +exactly what is retained, cleared, warmed or recalibrated. No worker independently resets. + +Changing port assignment, agent membership, model/profile, cadence or task schema requires +a new composition/epoch. Hot-join and hot-swap during an active match are not v1 capabilities. + +## 7. Failure rules + +| Failure point | Required response | +| --- | --- | +| Before any Prepare admitted | Reject request/config or remain Ready; nothing advanced | +| Some agents Prepared | Stop dispatch; resolve matching requests or fail epoch/group restore | +| Advance acknowledgment lost | Query/retransmit same request to same incarnation; never new batch | +| World advanced, sensory data unavailable | Fail transition; do not reward/continue using guessed input | +| Task interpretation fails | Fail epoch; prepared brains/world already changed | +| Some Commit replies missing | Resolve exact requests; no next world step until all committed | +| Worker incarnation changes | All live participants belong to an invalid epoch; restore/reset together | +| Publisher/browser disconnected | Simulation continues; bound/drop spectator work | +| Durable storage fails | Report actual failure; apply configured pause/continue-with-stale-checkpoint policy | + +Retries return original results; they never recompute a decision with updated rates or new +world data. A coordinator restart has no authority to assume any remote participant's phase; +recover from a coherent checkpoint into a new epoch or start an explicitly new episode. + +## 8. Required trace assertions + +The synthetic integration test must record, for every transition: + +- Scope, Prepare request IDs, agent/profile IDs, tick counts/remainders and decision digests. +- Complete batch ID/control digest and acknowledged world boundary. +- Observation producing boundaries and task event/outcome IDs in order. +- All Commit acknowledgments and published committed boundary. + +Evaluate agents sequentially, concurrently, and in reversed dispatch/completion order. All +committed state/action/reward results must match, excluding wall time, request IDs and other +explicitly operational metadata. Delayed/lost/duplicate messages must not add a neural tick, +world step, reward update or controller flush corresponding to another logical step. diff --git a/docs/design/session-framework/workers-v1.md b/docs/design/session-framework/workers-v1.md new file mode 100644 index 0000000..45b45eb --- /dev/null +++ b/docs/design/session-framework/workers-v1.md @@ -0,0 +1,367 @@ +# Worker and task interfaces v1 + +Status: **draft 2**. Uses [Flybus](bus-v1.md) for every call/publication and the domain +types/outcomes in [session RPC](ipc-v1.md), ordering in +[step protocol](step-v1.md), and binary/state types in [state and media](state-media-v1.md). +All method results below are the `result` object inside the domain result carried by a bus +rpc.result. Large inputs/outputs use owned bus attachments, never another worker data channel. + +## Method registry + +| Method | Caller → receiver | State/owner | +| --- | --- | --- | +| `Worker.Hello` | Authorized caller → named worker service | Domain identity/capabilities after bus negotiation | +| `Worker.Status` | Authorized caller → named worker service | Read-only; responsive during compute | +| `Worker.Acknowledge` | Coordinator → worker | Bounded lifecycle-result retention | +| `Worker.Shutdown` | Coordinator → worker | Terminal lifecycle request | +| `Agent.Initialize` | Coordinator → agent | Uninitialized → Ready(0) | +| `Agent.Prepare` | Coordinator → agent | Ready(k) → Prepared(k) | +| `Agent.Commit` | Coordinator → agent | Prepared(k) → Ready(k+1) | +| `Environment.Initialize` | Coordinator → environment | Uninitialized → boundary 0 | +| `Environment.Advance` | Coordinator → environment | Boundary k → boundary k+1 | +| `State.Capture` | Coordinator → agent/environment | Immutable snapshot of committed boundary | +| `State.StageRestore` | Coordinator → agent/environment | Validate replacement state under new epoch | +| `State.ActivateRestore` | Coordinator → agent/environment | Install staged state; remain quiescent | + +State methods have payloads in [state and media](state-media-v1.md). Artifact lifetime and +message consumption are bus operations managed by the SDK, not Worker/Coordinator methods. +Worker.Acknowledge releases a domain result cache, distinct from consuming a bus delivery. +Task/executor interfaces below are local library methods, not extra communication protocols. + +## 1. Shared data model + +```ts +interface AssetRef { + id: Id; digest: Digest; byteLength: U64; format: Id; +} +interface SensoryInput { + boundary: U64; // environment boundary being observed + views: ViewRef[]; // only the views this agent is allowed to consume + structured: TypedValue | null; +} +interface Stimulus { + id: Id; kindId: Id; durationMs: number; +} +interface Reward { + eventId: Id; ruleId: Id; value: number; +} +interface AgentTelemetry { + brainTicks: U64; + populationRateHz: number; + rates: { roleId: Id; hz: number }[]; + learning: { enabled: boolean; updates: U64; changed: U64; signal: number }; +} +``` + +`AssetRef` names persistent content in a preprovisioned local registry; it is not an arbitrary path or +URL for a worker to fetch. A profile artifact contains all effective numerical, sensory, +readout, learning and schema identities. Dataset artifacts must already be installed and +verified. No implicit runtime network download or latest-version selection is allowed. + +Transient `ArtifactRef` is instead defined by Flybus and resolves only through owned handles. +Views and structured input are separate capabilities. A pixel-only profile rejects non-null +structured input. Max views per sensory input: 8; any individual TypedValue is at most 32 KiB +of canonical JSON, and the complete bus envelope must fit 64 KiB. Larger typed state uses +an explicit artifact-backed schema. Frame bytes never go into JSON. + +`Stimulus.kindId` resolves through a profile-declared capability to an anatomical binding +and fixed drive, including supported duration bounds. A caller cannot specify arbitrary +neuron indices or change drive values. `durationMs` must be finite and >0. Arrays of stimuli +or rewards are bounded to 64 per operation and retain their supplied order. + +Rates must be finite/nonnegative, unique by role ID and in profile-defined order, at most +64 entries. Learning values must be finite; integer counters fit U64. Reward values are +finite; shipped positive-only task profiles reject negatives. Empty rewards do not imply a +different numerical rule. `id`/`eventId` is unique within its outcome or command namespace; +the coordinator assigns stable IDs before sending a mutating request. + +## 2. Agent methods + +### Agent.Initialize + +Allowed only on an uninitialized agent with negotiated `agent-step-v1`. Scope is the new +epoch at step `0`; restore uses the state interface instead of Initialize. + +```ts +interface AgentInitializeParams { + agentId: Id; + profile: AssetRef; + seed: number; // signed 32-bit integer, matching current RNG input + initialInput: SensoryInput; + initialDecisionContext: TypedValue; + workerThreads: number; // integer >=1; within launcher allocation +} +interface AgentInitializeResult { + agentId: Id; profileDigest: Digest; tickDuration: RationalNs; + warmupTicks: U64; committedStep: U64; // committedStep == "0" + decisionContextDigest: Digest; + telemetry: AgentTelemetry; +} +``` + +The profile fixes warm-up/calibration behavior and supported schema versions. Validate inputs +and required roles before model construction. Install the initial sensory input, warm the +brain with learning disabled, calibrate the fixed readout and establish Ready(0). Do not +generate gameplay rewards or controls that advance the world during warm-up. + +Seed is persisted as run configuration and state; the coordinator derives independent seeds +from its recorded master seed and stable agent IDs under a versioned derivation algorithm. +That algorithm is part of composition identity and MUST be specified/tested before the real +agent slice; hand-selected explicit seeds are supported for the first synthetic composition. +Identical explicit seeds are allowed only when the experiment intentionally declares them. +The profile artifact digest identifies the profile definition; the capture compatibility +digest additionally covers the resolved seed, numerical model version and effective instance +configuration. Never assume identical profile digests make differently initialized state +interchangeable without matching that instance configuration. + +### Agent.Prepare + +Allowed from Ready(k), or as an exact domain retry under the session RPC deduplication rules. + +```ts +interface PrepareParams { + agentId: Id; profileDigest: Digest; + interval: RationalNs; + decisionContextDigest: Digest; + preStepStimulations: Stimulus[]; +} +interface PreparedDecision { + agentId: Id; + ticksAdvanced: U64; brainTicks: U64; remainder: RationalNs; + decision: TypedValue; +} +``` + +Verify the cached context digest from Initialize/last Commit, expected agent/profile and +interval. Apply pre-step stimuli, advance ticks and decode as specified by the step protocol. +The returned decision schema is the profile's registered intent schema. It may describe a +controller state or selected semantic action, but cannot assign a port or include hidden +task-inspection fields. Context may mask declared available actions; it cannot change the +readout weights, invent a default winner or inject arbitrary neural observations. + +Successful response leaves the worker at Prepared(k). A duplicate returns the same decision, +ticks and remainder. It MUST NOT resample randomness, repeat stimulation or calibrate again. + +### Agent.Commit + +Allowed only at Prepared(k), matching the current transition and exact Prepare request. + +```ts +interface CommitParams { + agentId: Id; + preparedRequestId: Id; + nextInput: SensoryInput; // boundary == k+1 + nextDecisionContext: TypedValue; + rewards: Reward[]; + taskStimulations: Stimulus[]; +} +interface AgentCommitResult { + agentId: Id; committedStep: U64; // k+1 + decisionContextDigest: Digest; + telemetry: AgentTelemetry; +} +``` + +Validate the complete request and required owned artifacts before applying it. Follow exact input→ +stimulation→reinforcement ordering in the step protocol. A missing required view is an error, +not zero input. An outcome requesting unsupported learning/stimulation is an error, not a +silent no-op. Disabled learning is a declared profile/run state, not “unsupported.” + +No tick is executed in Commit. Cache its response before accepting the following Prepare. +The next context is retained for that Prepare; its canonical digest is returned and checked. +After encoding/copying and all asynchronous use finish, the worker drops its input Artifact +handles. The SDK consumes the delivery when the last associated guard disappears. A stored +input pointer must retain its handle. Cached replies retain their own artifact ownership. + +### Agent interface implementation boundary + +An agent worker bundles numerical model, sensor encoder and readout. It need not copy the +legacy `NeuralAgent::tick` call order: the existing service already orchestrates substeps. +Use a small adapter over the reference primitives and preserve the new specified ordering. +New numerical semantics require reference-first implementation and new model identities; +this protocol is not permission to change the pinned default kernel. + +## 3. Environment methods + +### Controller and descriptor types + +```ts +interface ControllerSchema { + schema: SchemaRef; + buttons: Id[]; // <=32, unique, fixed order + axes: { id: Id; range: "bipolar" | "unit"; neutral: number }[]; // <=16 +} +interface PortControl { + portId: Id; + buttons: { id: Id; down: boolean }[]; + axes: { id: Id; value: number }[]; +} +interface EnvironmentDescriptor { + backendDigest: Digest; contentDigest: Digest; configurationDigest: Digest; + stepDuration: RationalNs; + ports: { portId: Id; controls: ControllerSchema }[]; + inspectionSchema: SchemaRef; + views: ViewDescriptor[]; + audio: AudioDescriptor[]; + recovery: "exact-checkpoint" | "episode-restart"; + determinism: "fixed-build" | "unverified"; +} +``` + +Every active port control must include every declared button and axis in descriptor order. +All IDs must match exactly; no duplicates, extra controls or omissions. Bipolar axes are +finite [-1,1]; unit axes [0,1]. Neutral lies in range. Do not silently clamp an out-of-range +caller value. Hardware-specific quantization/dead zones are backend configuration, applied +exactly once and tested against observed controls. + +`fixed-build` asserts tested repeatability under the pinned configuration, not universal +bit-exact behavior across CPU architectures, GPU drivers or emulator versions. Those limits +must be in the backend implementation guide and run manifest. `unverified` cannot satisfy +an exact-replay production composition without an explicit scope change. + +### Environment.Initialize + +```ts +interface EnvironmentInitializeParams { + backendConfig: AssetRef; + taskConfig: AssetRef; + episodeId: Id; + portBindings: { portId: Id; agentId: Id }[]; +} +interface EnvironmentInitializeResult { + descriptor: EnvironmentDescriptor; + observation: WorldObservation; // boundary 0, worldTime zero +} +interface WorldObservation { + boundary: U64; + worldTime: RationalNs; // logical time since episode start; preserved on crash restore + engineFrame: string | null; // backend-defined signed counter; <=64 characters + sensoryViews: ViewRef[]; + inspection: TypedValue; + broadcastViews: ViewRef[]; + audio: AudioRef[]; +} +``` + +Scope is new epoch step 0. Backend/task config artifacts identify exact game content, +patches, initial-state/setup policy, graphics/timing/parser and controller conversion. +No unresolved “latest” settings. Environment setup may be application-specific, but it is +declared lifecycle scaffold, not actions attributed to a fly. The environment is stopped +when it returns O[0] and cannot free-run during brain initialization. + +The environment only needs backend-relevant portions of task setup, not reward rules or +neural policies. `taskConfig` resolves a declared setup configuration; the complete task +implementation and ledger stay in the coordinator. + +### Environment.Advance + +```ts +interface AdvanceParams { + batchId: Id; + controls: PortControl[]; +} +interface StepResult { + batchId: Id; + appliedFromStep: U64; nextStep: U64; + appliedControlsDigest: Digest; + observation: WorldObservation; +} +``` + +Validate scope k, complete controls and identities before releasing a backend input barrier. +Batch IDs are unique within an epoch; reusing one for a different request/step is a conflict. +Apply the complete batch to its interval and advance one framework step. Record batch ID, +result and next boundary before acknowledging. StepResult's control digest is over validated +canonical requested controls; backend quantization does not silently change that definition. +Observed game-pad values, if provided, are separately schema-labeled inspection data. + +The environment returns exactly boundary k+1, worldTime advanced by its stepDuration, and +the required sensory views or a typed failure. An adapter with measured input latency must +describe it in its versioned backend config and prove its frame mapping; it cannot claim an +unmeasured same-frame response. An emulator may execute internal cycles/polls, but cannot +hide multiple framework steps behind one result. + +### Environment pause behavior + +At a committed world boundary the backend already awaits the next Advance; normal session +pause does not need a separate per-frame RPC. When an emulator requires an explicit hardware/ +CPU pause to hold that invariant, the adapter owns it and must prove it. Status/Shutdown +remain responsive. A render/audio worker may drain already-produced data while stopped, +but no new gameplay state may advance. + +## 4. Coordinator-local task and executor interfaces + +These are library interfaces in v1, not additional bus services. Equivalent typed interfaces +may be implemented in Rust; names below specify semantics rather than compilable code. + +```text +Task.bootstrap(initialInspection, bindings) + → perAgentDecisionContexts, progress, initialEvents + +Task.evaluate_transition(scope, oldInspection, newInspection, appliedControls) + → perAgentOutcomes, perAgentNextDecisionContexts, progress, events, episodeRequest + +ActionExecutor.apply(scope, agentDecision, currentGameState, progressView, clock) + → ControllerIntent, executionEvents + +Task.capture / validate_restore / install_restore +ActionExecutor.capture / validate_restore / install_restore +``` + +Task owns a checkpointable ledger. The coordinator calls each transition evaluation exactly +once after its acknowledged world step and retains the output until all agent commits finish. +If task mutation is followed by failure, restore the group; never reevaluate against a later +observation. Task-produced outcomes are keyed by configured agent ID; unknown/missing agents +are errors. Every agent receives explicit outcome arrays, including empty ones. + +`ControllerIntent` contains buttons/axes conforming to the assigned port's controller schema, +but not a port assignment. The coordinator supplies the port. Per-agent executor state is +private; a running macro may emit controls according to its declared policy, but only after +neural selection. The first implementation supports the stateless identity executor only. + +The executor's currentGameState is a coherent read-only inspector view at this boundary; +progressView supplies task history/objectives. It updates its selected action every step +(movement, path replanning, interaction, completion), not merely replaying a blind button +sequence. These game-aware inputs stay in the task/executor layer. For an external backend, +they arrive as typed observation/artifact data over the same bus, not per-byte remote reads. + +Task contexts sent to the neural readout are typed, bounded, versioned and allowlisted by its profile. +They may express boot state or available actions. They are distinct from neural sensory +input and broadcast telemetry. A profile using game-state features as neural input must +explicitly declare structured sensing; the task cannot smuggle it into an opaque context. + +Task events carry `{id, kindId, sourceStep, agentId:null|Id, payload:TypedValue}`. Event order +is task-defined and deterministic; `sourceStep` is the newly reached boundary k+1 for a +transition event (0 for bootstrap events). Event IDs are derived deterministically from +epoch, source step, task/rule and event ordinal, encoded as an Id. Rewards and stimulation +referencing these events must +have a configured recipient. Broadcast text is generated by task/presentation templates, +not arbitrary raw inspector memory or incoming chat. + +`episodeRequest` is either null or `{kind:"terminal", reason:Id, outcome:TypedValue}`. +It requests a coordinator-owned policy transition after final reward commit; it cannot reset +the environment directly. Generic progress is a TypedValue, not mandatory Pokémon ladder data. + +## 5. Admission and audience boundary + +The first synthetic implementation has no audience input. Later integration maps permitted +public requests into a coordinator admission record containing stable interaction ID, agent +ID, accepted epoch/earliest step and profile-supported stimulus. Only the coordinator can +place that stimulus in Prepare. Viewer names/chat never enter the neural worker contract. + +Rate limits and target resolution occur before the step's command cut. A later-selected +UI focus or new match cannot retarget an already accepted interaction. Bus/domain RPC deduplication +does not itself define payment/redemption semantics across epoch recovery; a future public +v2 contract must specify accepted/applied/rolled-back/aborted states and reconciliation before +paid interactions are enabled. Do not inherit a claim of durable exactly-once stimulation +from these in-memory worker request caches. + +## 6. Health, shutdown and extensions + +All workers implement the common Hello/Status/Shutdown/Acknowledge methods. Capture/restore +methods are in the state contract and mandatory only when exact-checkpoint capability is +advertised. Unsupported methods return `UNSUPPORTED`, mutation none. + +New task-specific fields belong in registered TypedValue schemas. New worker capabilities, +variable-duration stepping, subscriptions or additional sensor modalities require a contract +change and shared fixtures. An unconstrained plugin dictionary is not a substitute for that.