From 3c614c87f433f5b97a216d6192564c3ca6a6aa9f Mon Sep 17 00:00:00 2001 From: acamilo Date: Wed, 23 Sep 2026 08:36:00 +0000 Subject: [PATCH] docs: flybus.md amendments for the feed on the bus, its sizing and its store lifecycle The two pending decisions, taken for the feed with EDGE-01 and dated: sizing from the measured 122,367-byte snapshot (the frame is 92,160 bytes, not the 1.2 MB the list assumed) with the worst case of seven stuck latest subscribers at about 3 MB inside a 32 MiB tmpfs store; and the lifecycle, flysim owning the router under /run/fly/bus and starting first, the edge After= and Requires= it and reconnecting by itself across a crash. Plus the design as built, the tour's pointer, the example config and the flybus README's no-longer-true line. --- docs/architecture-tour.md | 7 +- docs/design/flybus.md | 104 +++++++++++++++++++++--- services/flysim/crates/flybus/README.md | 5 +- services/flysim/flysim.toml.example | 7 ++ 4 files changed, 109 insertions(+), 14 deletions(-) diff --git a/docs/architecture-tour.md b/docs/architecture-tour.md index 9afac6f..9f0f01b 100644 --- a/docs/architecture-tour.md +++ b/docs/architecture-tour.md @@ -131,6 +131,9 @@ publish. Pacing uses absolute deadlines at 1.0x by default; it never skips frame snapshot at 30 Hz: a JSON header (status, rates, learning stats, game mode, milestone rank and total, sugar state, events, chat ring) followed by attachments: RGBA frame, f32 stereo 48 kHz audio (binjgb's unipolar u8 converted and DC-blocked), and a 17,407-byte spike bitset. + flysim serves it itself by default; with `FLY_FEED_VIA=bus` it publishes each snapshot on an + embedded flybus router and the `fly-edge` process serves the same bytes + (`docs/design/flybus.md`, "Feed over the bus"). - Control API (`docs/control-api.md`): loopback HTTP :7401. `POST /stimulate` (sugar: a timed PAM pulse, rate-limited server side), `POST /reward` (present, disabled by config), `POST /chat` (sanitized, deny-listed, ring of 12), `/status`, `/checkpoint`, `/pause`, `/resume`, @@ -159,8 +162,8 @@ sequenceDiagram S->>S: every 5 s hot copy, every 300 s durable checkpoint ``` -Where: `services/flysim/crates/flysim/src/{main,config,simloop,pacing,snapshot,feed,api,chat,store,eventlog,metrics}.rs`, -`docs/design/flysim.md`. +Where: `services/flysim/crates/flysim/src/{main,config,simloop,pacing,snapshot,feed,feedbus,api,chat,store,eventlog,metrics}.rs`, +`services/flysim/crates/fly-edge`, `docs/design/flysim.md`. ## 4. Stage page diff --git a/docs/design/flybus.md b/docs/design/flybus.md index 1dcff3b..369b8f0 100644 --- a/docs/design/flybus.md +++ b/docs/design/flybus.md @@ -1,6 +1,7 @@ # flybus: the communications bus -Status: **crate landed, nothing wired onto it**. Written 2026-09-22. Index only; the +Status: **crate landed; the feed rides it behind `FLY_FEED_VIA=bus`, off by default**. +Written 2026-09-22, amended 2026-09-23 (EDGE-01, below). Index only; the authority for the API and the wire format is the crate's own [README](../../services/flysim/crates/flybus/README.md), and the audit of the crate against the draft is the [conformance report](session-framework/bus-conformance.md). @@ -37,8 +38,8 @@ does not change any published contract by existing. ## Crate layout -`services/flysim/crates/flybus`, a workspace member of the flysim workspace; no other crate -depends on it yet. +`services/flysim/crates/flybus`, a workspace member of the flysim workspace. `flysim` depends +on it for the feed publisher (`src/feedbus.rs`) and `fly-edge` for the subscriber. | Module | Contents | | --- | --- | @@ -59,16 +60,99 @@ allocate/seal/read with quotas and router restarts, plus the conformance suites ## Wiring still pending -- **flysim publisher.** Router startup inside the sim service, a store root under its - runtime directory, and snapshot publication as artifact plus header envelope. +- ~~**flysim publisher.**~~ Done 2026-09-23 behind `FLY_FEED_VIA=bus`: see "Feed over the bus". - **flysim control services.** The control endpoints as RPC services with grants, so the "no button endpoint" structural guarantee is expressed as a grant table. - **Stage and bridge clients.** Both are TypeScript/Node; the crate is Rust only, so either a binding or a thin translating edge process is required before they leave the WebSocket - and HTTP surfaces. -- **Sizing.** `max_store_bytes`, `max_retained_bytes` and `max_latest_in_flight` need values - chosen for 1.2 MB frames at 30 to 60 Hz with a slow consumer, not the defaults. -- **Lifecycle.** Orphaned store directories are cleaned only when a new router starts on the - same root, so service restart order and the store root's location need a decision. + and HTTP surfaces. The operator chose the edge process (port decisions, 2026-09-23); for + the feed it exists (`fly-edge`), and they keep the WebSocket contract unchanged. The + control API (:7401) is the next slice and stays in flysim until then. +- ~~**Sizing.**~~ Decided 2026-09-23: amendment "Feed sizing" below. +- ~~**Lifecycle.**~~ Decided 2026-09-23: amendment "Feed store lifecycle" below. - **Migration order.** The feed is the cheaper first move; control should follow only once the bus carries the feed in production for a full session. + +## Feed over the bus (2026-09-23, EDGE-01) + +`feed.via` (`FLY_FEED_VIA`) picks who serves `ws://127.0.0.1:7400/feed`. `direct` is the +default and is the behaviour that predates the bus. With `bus`: + +```text + sim thread --watch--> publisher task --flybus (in memory)--> Router + (unchanged) (flysim-bus runtime) | /edge.sock + v (bound to "fly-edge") + fly-edge: Subscription -> watch -> flysim::feed :7400 +``` + +- flysim does not bind `feed.bind`. It starts a `Router` on a runtime of its own (two + threads, `flysim-bus`), store root `/store`, closed policy: `flysim` may declare + and publish `fly.feed.snapshots`, `fly-edge` may only subscribe to it, and the Unix socket + `/edge.sock` is launcher-bound to `fly-edge`. +- The topic is `retained: latest`. Each publication is one snapshot: the attachments the + header lists as sealed artifacts named `frame` (`image/x-rgba`), `audio` + (`audio/x-f32le`), `spikes` (`application/x-spike-bitset`), and the header as the payload + `{"header": {...}}`. A header over 48 KiB of JSON goes as a `header` artifact instead, so + the 65,536-byte envelope limit can never make a snapshot unpublishable. +- The sim thread is untouched. The publisher reads the same `watch` slot the direct server + reads, so a slow bus skips snapshots the way a slow WebSocket client does, and nothing on + the bus can hold the loop's publish. `fly_bus_published_total` and + `fly_bus_publish_failures_total` count it. +- `fly-edge` subscribes `latest`, one in flight, with replay, rebuilds each `Snapshot` with + `feedbus::receive` and serves it with flysim's own `feed::router`. `hello`, `wants`, + drop-oldest, the idle header and the framing are therefore the same code, and the bytes are + the same bytes: `crates/fly-edge/tests/parity.rs` replays the committed stage fixtures + through both paths at once and requires byte-equal messages per client flavour. +- `fly_frames_sent_total` and `fly_feed_clients` move to the edge with the clients; it exports + them under the same names on `FLY_EDGE_METRICS_ADDR` (`127.0.0.1:9102` in + `infra/units/flyedge.service`), and watchdog check 2 follows `FLY_FEED_VIA` in `fly.env` to + them. flysim's own copies read 0 in bus mode; `/status` is otherwise unchanged. +- Nothing about the fly changes: the readout, the reward catalog, the adapter version and the + compatibility string are byte-identical in both modes (`--print-compatibility`). + +### Amendment 2026-09-23: feed sizing + +Measured on the live fly (release build, the real cartridge): a running snapshot is a +**92,160-byte** frame (160x144 RGBA; not the 640x480 "1.2 MB" the pending list assumed), a +**17,407-byte** spike bitset (139,255 neurons), about **12,800 bytes** of audio at realtime +(1,600 stereo f32 frames per 30 Hz snapshot at 48 kHz) and a 2 to 3 KB header: **122,367 +bytes** of artifacts, about 3.7 MB/s at 30 Hz. `flysim::feedbus::limits()`: + +| Limit | Value | Why | +| --- | --- | --- | +| `max_clients` | 8 | flysim's publisher, the edge, and room for a recorder or a probe | +| `max_latest_in_flight` | 2 | the default; the edge asks for 1 | +| `max_artifact_bytes` | 4 MiB | ten seconds of audio that piled up behind a late publish | +| `max_store_bytes` | 32 MiB | tmpfs, so RAM; ten times the worst case below | +| `max_retained_bytes` | 8 MiB | one retained snapshot, plus a large header artifact | +| `max_owners_per_client` / reserved | 64 / 8 | three artifacts per delivery, a few deliveries | +| others | small counts | one topic, no services | + +A `latest` subscriber that never consumes pins at most its queued slot plus its in-flight +credits (3 snapshots); the topic pins one retained value; the publisher holds one snapshot of +staging plus the sealed copy while sealing. Seven stuck subscribers are therefore 24 snapshots, +about 3 MB, and publication never waits on any of them (a latest subscriber is never a +reason to refuse a publication, bus-v1 section 9). Measured in +`crates/fly-edge/tests/stall.rs`: a subscriber that hoards every delivery holds the store at +4 snapshots (489,468 bytes) while 179 of 179 snapshots are published, with pacer lag 0. + +### Amendment 2026-09-23: feed store lifecycle + +- **Location.** `feed.bus_dir` (`FLY_BUS_DIR`), `/run/fly/bus` on the containers: tmpfs, + 0700, owned by `fly`, created by tmpfiles and again by flysim. The store root is + `/store`, the socket `/edge.sock`. A reboot empties it. +- **Owner.** The router lives in flysim; its lifetime is flysim's. flysim removes a stale + socket file at start, and `Router::new` removes any store directory whose `flock` is free, + i.e. one a crashed flysim left behind. A clean stop removes its own directory. The edge owns + nothing on disk. +- **Order.** flysim first, the edge after it: `flyedge.service` is `After=` and + `Requires=flysim.service`, so an explicit stop or restart of flysim (the unstick rule's + restart included) takes the edge with it. A crash-restart of flysim needs nothing: the edge + sees the connection close, drops every WebSocket client, unbinds :7400 and reconnects every + 500 ms, binding :7400 again only when the first snapshot of the new router arrives. To the + stage that is exactly a flysim restart in direct mode: refused, then back. +- **Default.** `flyedge.service` is in no target and `07-enable.sh` does not enable it; + `05-deploy.sh` writes `FLY_FEED_VIA=direct` unless the env file says otherwise. The switch + and the way back are in the unit's header. +- **Migration order** is unchanged: the feed first; control only after the bus has carried + the feed in production for a full session. diff --git a/services/flysim/crates/flybus/README.md b/services/flysim/crates/flybus/README.md index 125391a..003fbe4 100644 --- a/services/flysim/crates/flybus/README.md +++ b/services/flysim/crates/flybus/README.md @@ -14,8 +14,9 @@ Where this crate narrows or extends the draft, the difference is listed under sections 2 to 11 is audited against this code, with the test that proves it, in `docs/design/session-framework/bus-conformance.md`. -Nothing in the crate is specific to a game, a brain or a stream. It is a workspace member and -no other crate depends on it yet. +Nothing in the crate is specific to a game, a brain or a stream. It is a workspace member; +`flysim` embeds a router for the feed (`FLY_FEED_VIA=bus`, `flysim::feedbus`) and `fly-edge` +subscribes to it (`docs/design/flybus.md`, "Feed over the bus"). ## Layout diff --git a/services/flysim/flysim.toml.example b/services/flysim/flysim.toml.example index ed3ca17..eb98f50 100644 --- a/services/flysim/flysim.toml.example +++ b/services/flysim/flysim.toml.example @@ -62,6 +62,13 @@ bind = "127.0.0.1:7400" # Audio attachment rate. 48 kHz is Web Audio's native rate on Linux, so the page never resamples. # env: FLYSIM_FEED_AUDIO_HZ audio_hz = 48000 +# Who serves `bind`: "direct" (flysim, the default) or "bus" (flysim publishes on an embedded +# flybus router and the `fly-edge` process serves the same bytes; docs/design/flybus.md). +# env: FLY_FEED_VIA, FLYSIM_FEED_VIA +via = "direct" +# The bus router's socket and artifact store in "bus" mode. tmpfs. +# env: FLY_BUS_DIR, FLYSIM_FEED_BUS_DIR +bus_dir = "/run/fly/bus" [control] # http://127.0.0.1:7401 — docs/control-api.md. Loopback only; there is no auth because nothing