278 lines
16 KiB
Markdown
278 lines
16 KiB
Markdown
# Feed protocol v1 (flysim to flystage)
|
|
|
|
The simulation service publishes snapshots over a WebSocket on `ws://127.0.0.1:7400/feed`. The
|
|
page is a display: it never sends anything but a one-time `hello`. Every consumer (stage, bridge,
|
|
tests) uses the same message shape, defined in TypeScript in `packages/feed` and in Rust with
|
|
serde in `services/flysim/crates/flysim`. A JSON schema test in each side pins the shape.
|
|
|
|
## Framing
|
|
|
|
Each snapshot is ONE binary WebSocket message:
|
|
|
|
```
|
|
u32 LE headerLength | header JSON (UTF-8) | attachments...
|
|
```
|
|
|
|
Attachments follow the header in the order listed by `header.attachments`, each as
|
|
`u32 LE byteLength | bytes`, so a consumer can skip attachments it does not want. The header
|
|
is small (under 4 KB); attachments carry the bulk.
|
|
|
|
Cadence: 30 snapshots per second wall-clock while running, 2 per second while paused or
|
|
booting (header only). The service drops snapshots rather than queueing when the socket is slow.
|
|
|
|
## Header
|
|
|
|
```ts
|
|
interface FeedHeader {
|
|
protocol: 1;
|
|
seq: number; // monotonically increasing
|
|
wallMs: number; // Date.now() at publish
|
|
status: 'booting' | 'running' | 'paused' | 'recovering' | 'error';
|
|
realtimeFactor: number; // simulated ms per wall ms over the last second
|
|
uptimeSeconds: number; // since service start
|
|
runSeconds: number; // total simulated seconds across restores (from checkpoint)
|
|
brainMs: number; // simulation clock
|
|
frame: number; // emulator frame counter
|
|
buttons: number; // bitmask, GAMEBOY_BUTTON_BITS order (up,down,left,right,a,b,start,select)
|
|
rates: Record<string, number>; // Hz per tracked role: command_0..7, steer_left, steer_right,
|
|
// forward, backward, proboscis, reward_pam, and in macros
|
|
// mode the bound macro types' macro_* roles
|
|
populationRate: number; // Hz
|
|
spikeCount: number; // set bits in the spikes attachment (0 when omitted)
|
|
learning: { enabled: boolean; updates: number; changed: number; synapses: number; signal: number };
|
|
game: {
|
|
mode: 'BOOT' | 'OVERWORLD' | 'BATTLE' | 'TRANSITION' | 'DEMO' | 'SAFARI' | 'UNKNOWN';
|
|
// closed set; adapters fold onto it (see below)
|
|
semanticRewards: boolean; // false when the ROM hash is not the audited one
|
|
map: number | null; // current map id when known
|
|
badges: number; // 0..8
|
|
uniqueLocations: number; // exploration coverage
|
|
rewardTotal: number;
|
|
rewardCounts: Record<RewardKind, number>;
|
|
// The macro buttons (docs/design/macros.md section 12). Added 2026-09-16, additive,
|
|
// optional: a producer that predates them omits all five and a consumer falls back to
|
|
// raw-mode rendering.
|
|
scene?: Scene; // closed set; the scene whose macro buttons are bound
|
|
macroMode?: 'raw' | 'macros';
|
|
palette?: PaletteSlot[]; // the scene's bound macro buttons, ascending by slot,
|
|
// one per macro type at most; [] in raw mode
|
|
macro?: RunningMacro | null; // the macro that owns the buttons
|
|
macroOutcome?: MacroOutcome | null; // the one that finished most recently
|
|
padEmptyMs?: number; // BRAIN ms the pad has had nothing on it in a playable
|
|
// scene; 0 otherwise. Report only.
|
|
};
|
|
milestone: {
|
|
rank: number; // position on the ratchet ladder, 0..total-1
|
|
label: string; // human label, e.g. "Left the bedroom"
|
|
next: string; // label of the rung the fly is going for (see below)
|
|
sinceSeconds: number; // simulated seconds at current rank (the stuck-o-meter)
|
|
attempts: number; // game rollbacks since reaching this rank
|
|
total?: number; // rungs the game's ladder has (>= 1); omitted by older producers
|
|
};
|
|
sugar: {
|
|
active: boolean; // a stimulation pulse is currently applied
|
|
remainingMs: number;
|
|
cooldownMs: number; // until the next viewer sugar is accepted
|
|
lastBy: string | null; // display name of the last redeemer
|
|
todayCount: number;
|
|
};
|
|
events: FeedEvent[]; // events since the previous snapshot (usually empty)
|
|
chat?: ChatLine[]; // last N (<= 12, `[chat] ring`) lines accepted by the control API,
|
|
// oldest first. ABSENT ENTIRELY (not an empty array) while
|
|
// `[chat] enabled = false`, so a page can tell "chat is off" from
|
|
// "nobody has said anything yet". Chat never reaches the simulation.
|
|
attachments: AttachmentKind[];
|
|
}
|
|
|
|
type RewardKind = 'story' | 'explore' | 'area' | 'pokedex' | 'trainer' | 'wildwin' | 'badge';
|
|
|
|
type Scene =
|
|
| 'title' | 'overworld' | 'dialog' | 'menu'
|
|
| 'battle' | 'battle-switch' | 'shop' | 'pc' | 'unknown';
|
|
|
|
interface PaletteSlot {
|
|
slot: number; // the macro TYPE's own index, 0..30; unbound types are
|
|
// omitted, and a type's index never changes
|
|
name: string; // the macro's name, at most 14 characters
|
|
gloss: string; // two or three words of what it does here ("nearest door")
|
|
channel: string; // the type's short channel tag, e.g. "MB·GO", "MB·ATK"
|
|
}
|
|
|
|
interface RunningMacro {
|
|
slot: number;
|
|
name: string;
|
|
sinceMs: number; // BRAIN milliseconds since it started (a duration, not a clock)
|
|
}
|
|
|
|
interface MacroOutcome {
|
|
slot: number;
|
|
name: string;
|
|
outcome: 'done' | 'blocked' | 'timeout' | 'refused';
|
|
atMs: number; // the BRAIN clock at which it finished (a timestamp)
|
|
}
|
|
|
|
interface FeedEvent {
|
|
id: number; // monotonically increasing across the run
|
|
wallMs: number;
|
|
brainMs: number;
|
|
kind: 'reward' | 'sugar' | 'milestone' | 'recovery' | 'checkpoint' | 'viewer' | 'system'
|
|
| 'macro';
|
|
label: string; // short human text, template-generated, never raw chat
|
|
value?: number; // reward value, milestone rank, or macro slot
|
|
rewardKind?: RewardKind;
|
|
by?: string; // viewer display name for sugar/viewer events
|
|
}
|
|
|
|
interface ChatLine {
|
|
id: number; // the event id of the accepted line
|
|
wallMs: number;
|
|
by: string; // validated display name (packages/feed names.ts rules)
|
|
text: string; // sanitized: <= 200 code points, letters/digits/space and a fixed
|
|
// punctuation allowlist only — no control, zero-width or bidi
|
|
// characters, no combining marks, no emoji, no URLs, deny-list
|
|
// filtered; the service refuses anything else whole, never trimmed.
|
|
// The rules live in packages/feed/src/chat.ts and, identically, in
|
|
// services/flysim/crates/flysim/src/chat.rs, pinned to each other
|
|
// by packages/feed/tests/fixtures/chat-cases.json
|
|
bot?: boolean; // true for the bridge's own template replies
|
|
}
|
|
|
|
type AttachmentKind = 'frame' | 'audio' | 'spikes';
|
|
```
|
|
|
|
`game.mode`, `game.scene` and `RewardKind` are closed sets, because every consumer switches
|
|
exhaustively on them.
|
|
An adapter with other states or other reward kinds folds onto these names rather than extending
|
|
them; the per-game config in `apps/stage/src/games/` supplies the words on screen.
|
|
|
|
| Adapter mode | `game.mode` | Why |
|
|
| --- | --- | --- |
|
|
| Pokémon `BOOT` / `OVERWORLD` / `BATTLE` / `TRANSITION` / `DEMO / SAFARI` | as named | — |
|
|
| Platformer `IN LEVEL <world>-<stage>` | `OVERWORLD` | Both mean "controllable in the world". The level itself is `game.map`, so nothing is lost. |
|
|
| Platformer `GAME OVER` | `TRANSITION` | The run is over and the fly is not controllable. The recovery event says what happened. |
|
|
| Platformer `DEMO` (attract demo) | `DEMO` | — |
|
|
| `UNSUPPORTED ROM · SEMANTIC REWARDS OFF` | `UNKNOWN` | `game.semanticRewards` already carries it. |
|
|
|
|
The platformer's nine reward kinds share the seven published counters: `band` -> `explore`,
|
|
`coin` -> `wildwin`, `score` -> `area`, `powerup` -> `pokedex`, `life` -> `trainer`,
|
|
`level` -> `story`, `world` -> `badge`. `started` and `clear` map to nothing: each pays once in a
|
|
lifetime, so a counter for them is noise, and their events carry their own labels.
|
|
|
|
## The macro buttons (2026-09-16)
|
|
|
|
`docs/design/macros.md` section 12 is the design; these are the fields, and where the two differ
|
|
this file wins. Additive: `protocol` stays 1, the five fields are optional in
|
|
`packages/feed/src/schema.json`, and a consumer that ignores them renders exactly what it rendered
|
|
before. The Rust producer always sends all five, in both modes.
|
|
|
|
- **`game.scene`** is the scene whose macro buttons are bound, from the closed set above. It is
|
|
`unknown` in raw mode — nothing is on the pad but the eight buttons, so no scene is claimed —
|
|
and `title` through the intro, where nothing is bound by design and the readout's boot variant
|
|
applies.
|
|
- **`game.macroMode`** is `raw` or `macros`, straight from `flysim.toml`'s `[macros] mode`. A
|
|
consumer that does not know a mode name renders the raw layout rather than publishing a word it
|
|
cannot explain.
|
|
- **`game.palette`** carries the scene's **bound** macro buttons, ascending by slot, at most one
|
|
per macro type. `slot` is the macro *type's own index* since `docs/design/macros.md` section 14
|
|
(2026-09-17) — not a Game Boy button, not a rank, and no longer a position among the bound ones,
|
|
which is what let a cell move between two frames as a precondition came and went. A type that
|
|
this scene does not bind is omitted rather than sent as a null, and the page draws its cell dim.
|
|
Always `[]` in raw mode.
|
|
- **`game.macro`** is the macro that owns the buttons, or `null`. `sinceMs` is a *duration* in
|
|
brain milliseconds, so a page can show how long it has been running without doing arithmetic
|
|
against `brainMs`.
|
|
- **`game.macroOutcome`** is the macro that finished most recently and is **kept until the next
|
|
one starts**, so the page can light a cell's result for a beat rather than racing a single
|
|
frame. `atMs` is a brain-clock *timestamp*, so it can be compared with `brainMs`.
|
|
|
|
- **`game.padEmptyMs`** is how long the pad has had nothing on it **in a playable scene**, in
|
|
brain milliseconds, and 0 otherwise — in raw mode, on the title screen, when something is bound,
|
|
and while a macro is running, because a running macro owns the pad. It is **report only**: no
|
|
producer acts on it and no consumer should, and nothing in the loop reads it back. It exists
|
|
because an empty pad is the doctrine working — nothing presses for the fly, so a scene with no
|
|
button waits — and on screen that is indistinguishable from a hang
|
|
(`docs/design/macros.md` section 13.1). `fly-watchdog` exports it as `fly_pad_empty_seconds`.
|
|
|
|
`game.macro` and `game.macroOutcome` carry that same slot, so a display lights the cell that ran
|
|
without matching on the name.
|
|
|
|
### One channel per macro type (2026-09-16)
|
|
|
|
`docs/design/macros.md` sections 11 and 12: a macro type is a button pressed by its own neuron
|
|
population, and the decoder's second exclusive group picks between the types the scene has bound
|
|
(`docs/readout.md`, "Macro group"). Two consequences here.
|
|
|
|
- **`game.palette[i].channel`** is that type's short tag, which the page draws as the cell's glyph
|
|
— `MB·GO`, `MB·ATK`. The rate role behind it is `macro_` plus the macro's name lowercased
|
|
with spaces as underscores (`GO ITEM` -> `macro_go_item`), which is how a consumer finds the
|
|
type's Hz in `rates`. It is also what orders the cells on screen: the cells are drawn in the
|
|
contract's fixed type order, which since section 14 *is* `slot`. Every entry a dealt mode sends
|
|
carries it; it is absent only from a producer that predates this paragraph.
|
|
- **`prior` is removed.** There is no rank, no ladder and no blend left to publish: whichever
|
|
bound channel wins the decision starts, and `game.macro` says which did. A producer still
|
|
sending `prior` is stale, and a consumer ignores the field.
|
|
|
|
`rates` carries the bound types' `macro_*` roles alongside `command_0..7`, which is what the
|
|
SENSES tab's MACROS row draws. Those populations are shared out over the mushroom body output
|
|
neurons and the brain motor neurons, so a rate there is the mushroom body's, not a ninth button.
|
|
|
|
One `macro` event per start and per finish, so the ticker and `events.jsonl` both carry them.
|
|
Labels are exactly `<NAME> start` on a start and `<NAME> done|blocked|timeout|refused` on a
|
|
finish; `value` is the slot. An unbound type produces no event of any kind: it never entered the
|
|
decision, so there is nothing to report.
|
|
|
|
### Why `macroMode` and not `mode`
|
|
|
|
`docs/design/macros.md` asks for `game.mode: "raw" | "macros"`. That name has been taken since
|
|
protocol v1 by the adapter's own mode string (`BOOT`, `OVERWORLD`, …), which is a
|
|
closed set every consumer switches exhaustively on, and repointing it would be a breaking change
|
|
to a published field for a feature that is explicitly additive. So the mode is published as
|
|
`game.macroMode`, and the design doc's field list is wrong on that one name. `macroMode` is the
|
|
name both sides are built against.
|
|
|
|
## `milestone.next` is the rung the fly is going for (2026-09-17)
|
|
|
|
`next` was the label of `rank + 1`. It is the label of the adapter's own **next rung** now —
|
|
`GameAdapter::next_rung`, the lowest rung the run has *not* earned — falling back to `rank + 1` for
|
|
an adapter that does not answer. The type is unchanged (a string, never empty: the current label is
|
|
still the fallback at the top of the ladder), so no client changes and no version bump.
|
|
|
|
Why: the rank is the *maximum* over satisfied rungs, and a ladder's rungs are not a chain. The live
|
|
save of 2026-09-17 had Pokémon Red's rungs 6 and 7 — Oak's parcel delivered, and the Pokédex —
|
|
unearned while rung 8 (Viridian City, a map) was stood on, so the rank read 8 and the header said
|
|
"→ VIRIDIAN FOREST" while `GO OBJECTIVE` was walking two maps south to deliver a parcel. The screen
|
|
was naming a rung nothing was working toward. `rank`, `label`, `sinceSeconds`, `attempts` and
|
|
`total` are untouched, and so is the ratchet: the rank is still the maximum, because that is what
|
|
"best reached" means.
|
|
|
|
## Attachments
|
|
|
|
- `frame`: 160 x 144 RGBA, 92,160 bytes, the emulator framebuffer after the last frame.
|
|
- `audio`: interleaved stereo `f32` PCM at 48,000 Hz (Web Audio's native rate on Linux, so the
|
|
page never resamples), all samples produced since the previous snapshot (about 1,600 frames =
|
|
12,800 bytes at 30 Hz). binjgb emits unsigned 8-bit stereo; the service converts. The page keeps
|
|
a ring buffer with varispeed drift correction.
|
|
- `spikes`: a bitset of `ceil(neurons / 8)` bytes (17,407 bytes for 139,255 neurons), bit `i`
|
|
(byte `i >> 3`, mask `1 << (i & 7)`) set when neuron `i` spiked at least once since the previous
|
|
snapshot. Fixed size, order-free, and exactly what the page's density accumulator consumes.
|
|
`header.spikeCount` carries the number of set bits.
|
|
|
|
Bandwidth at 30 Hz: about 3.7 MB/s on localhost (frame 92 KB, spikes 17 KB, audio 13 KB).
|
|
|
|
## Client hello
|
|
|
|
On connect the client sends one JSON text message:
|
|
`{"protocol":1,"client":"stage"|"bridge"|"test","wants":["frame","audio","spikes"]}`.
|
|
The service honours `wants` per connection. The bridge asks for no attachments.
|
|
|
|
## Compatibility
|
|
|
|
`protocol` bumps on any breaking change. Adding optional header fields is not breaking.
|
|
|
|
## Audio source note (2026-09-15)
|
|
|
|
binjgb's mixer emits unsigned 8-bit samples that are unipolar: silence is 0, not 128 (measured
|
|
range 0..44 over a Pokémon Red boot). The service converts with `v / 255` to `[0, 1]` and then
|
|
applies a DC-blocking one-pole high-pass (`y[n] = x[n] - x[n-1] + 0.995 * y[n-1]`) per channel
|
|
before publishing, so the `audio` attachment is ordinary bipolar f32 PCM centred on zero. The page
|
|
does no further offset correction.
|