diff --git a/docs/design/flysim.md b/docs/design/flysim.md index abdddab..5e7f0b5 100644 --- a/docs/design/flysim.md +++ b/docs/design/flysim.md @@ -355,6 +355,50 @@ on-screen ticker cannot disagree with what the sim did. against the prototype's WASM size and diffs a known save. If they match, prototype checkpoints import and the segment records the shared tag; if not, milestone saves must be re-earned and that is a stated M3 finding. +- **Restoring across an adapter version** (2026-09-22). The compatibility string is compared + whole, so bumping the reward adapter refuses every checkpoint the previous one wrote -- which is + the right default and was, until now, the only behaviour. It is the wrong default for a change + that only *adds* a rule: `pokered-unique8-v6` adds the catch reward and one counter, + `catchCounts`, and means the same thing as `v5` for every other field, so a `v5` run is + resumable and throwing it away would be a choice nobody made deliberately. + + So there is one narrow, opt-in migration, `flybrain_gb::compatibility::decide`, and it requires + **all three** of: + + 1. the two compatibility strings differ in the adapter segment (segment 1) and **nowhere else**. + A dataset, kernel, plasticity, emulator-revision, symbol-provenance or state-format + difference is still a refusal: none of those has a migration, and a fly restored across one + is a different fly; + 2. the running adapter's `migrates_from()` lists the checkpoint's adapter, so the code that will + read that state says out loud that it can. Pokémon Red's list is `["pokered-unique8-v5"]` and + nothing else -- `v4` is excluded because its ledger holds no `boundary:` keys and resuming it + would pay a second time for every exit already found, and `v3` because its stored rank is a + rung on a different ladder; + 3. the deploy names the same adapter id in **`FLY_ACCEPT_ADAPTERS`** (comma- or + space-separated). Unset or empty migrates nothing, which is what every deploy before this one + did. + + Condition 2 without 3 would make the migration silent; condition 3 without 2 would let an + operator wave through a pair nobody wrote a migration for. `infra/05-deploy.sh`'s compatibility + gate applies the same rule before it flips the `current` symlink, and writes the variable into + `/etc/fly/fly.env` so flysim applies it at restore -- the two must agree, or a deploy would pass + a gate that flysim then fails, which is the black stream the gate exists to prevent. The + migration itself is `PokemonRedReward::import_state` doing what it already did: `catchCounts` is + absent from a `v5` state and restores empty, which is the truth about a run that was never paid + for a catch. `STATE_VERSION` does not move, because the schema did not. + +- **Restarting a run from an earlier rung** (2026-09-22). `FLY_RESET_STATE=1` throws the run away; + `infra/bin/fly-reset-to-milestone ` keeps it and rewinds it. It archives both stores to a + dated directory, rewrites `milestone-.checkpoint` with the ratchet's `attempts` and + `recoveries` at zero (so the restarted run does not begin with its recovery budget already + spent), installs it as the newest generation of the hot and durable stores, removes the + milestone archives above N, and clears the event log -- whose id sequence the restored + checkpoint's `lastEventId` rewinds. `best` is not touched: the archive's own `best` is the rung + it was taken at, and the rank the stream shows is recomputed by the adapter from the restored + game state. The implementation is `flysim::reset` (`flysim --reset-to-milestone N`) rather than + the shell script, because two of those steps are inside the envelope. The sequence around it is + in `infra/docs/runbook.md`. + - **A running macro is not checkpointed** (2026-09-16, `docs/design/macros.md`). Palette mode's state — the scene, the palette, the running macro, its plan and its frame count — is transient, like the readout's blocked-direction cooldown and for the same reason: a restore that resumed a diff --git a/docs/design/macros-wram.md b/docs/design/macros-wram.md index 3b55030..1798322 100644 --- a/docs/design/macros-wram.md +++ b/docs/design/macros-wram.md @@ -118,6 +118,20 @@ Addresses are at the pinned commit. "Verified" is one of: | which slot is out | `wPlayerMonNumber` | `$cc2f` | 0-based party slot | ROM, trace | | the enemy | `wEnemyMonSpecies`, `wEnemyMonHP`, `wEnemyMonLevel`, `wEnemyMonMaxHP` | `$cfe5`, `$cfe6`, `$cff3`, `$cff4` | HP big-endian. Not written on the frame a battle starts — the reward adapter's own comment says the same — so the enemy is `None` for the first few hundred frames of a battle. | ROM (the rival's Squirtle, level 5, 20/20, and `None` on the first frame), trace | | how many moves | `wNumMovesMinusOne` | `$cd6c` | the move count minus one, valid in a battle | trace | +| **a ball kept this one** | `wCapturedMonSpecies` | `$d11c` | **new 2026-09-22** (the catch reward, `docs/rewards-learning.md`). `ram/wram.asm`'s own comment is "0 if no mon was captured". `ItemUseBall` zeroes it before every throw (`.canUseBall`) and writes `wEnemyMonSpecies` into it only on the branch that keeps the Pokémon; `UseBagItem`'s `.returnAfterCapturingMon` zeroes it again and sets `wBattleResult` to 2 on the way out of the battle. It is therefore non-zero for the hundreds of frames the catch's text and Pokédex screen take, and zero everywhere else. The value is the **internal** species index, like `wEnemyMonSpecies` and unlike `wPokedexOwned`'s bit index. Address resolved by `services/flysim/tools/resolve_wram.py`, bracketed by `wFontLoaded` and `wForcePlayerToChooseMon`. | survey (`tests/rom_catch.rs`: a real wild battle from a rung-9 checkpoint, balls thrown by the `THROW BALL` macro, the byte read out of the running game), trace (`pokemon_red/tests.rs`) | + +`wBattleResult` (`$cf0b`) is the second half of that row and is worth its own sentence: it is 0 +for a win, 1 for a loss, and 2 on exactly two paths in the whole game -- `.returnAfterCapturingMon` +and a *link* battle whose opponent ran (`engine/battle/core.asm`), which this cartridge never has. +So "the captured-species byte was non-zero during the battle **and** the result is 2" is a catch +and nothing else. `InitBattleVariables`, `ResetStatusAndHalveMoneyOnBlackout` and +`HandleFlyWarpOrDungeonWarp` all clear it, so a stale 2 cannot survive into the next battle. + +Not used for the catch, and why: `wPartyCount` (`$d163`) rises on a catch **only** when the party +has room -- a full party sends the Pokémon to `wBoxCount` instead -- and it also rises for a gift, +a trade and a Pokémon withdrawn from the PC. Reading a catch off it would need a second rule to +tell those apart. The cartridge's own flag needs none, which is why the row above is the one the +adapter reads. ### Battle menu and cursor, own turn against forced switch diff --git a/docs/design/session-framework/bus-conformance.md b/docs/design/session-framework/bus-conformance.md index 00e72b3..83e2114 100644 --- a/docs/design/session-framework/bus-conformance.md +++ b/docs/design/session-framework/bus-conformance.md @@ -153,15 +153,15 @@ and once over a Unix socket, so `tests/rpc.rs::request_reply_roundtrip` means | `latest`: one queued value, replacing only an undelivered one; replacement releases that entry's roots; delivered or in-use messages are never reclaimed early; maxQueued is exactly 1 | conforms | `router/state.rs::{op_publish (latest branch), op_subscribe}` | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both), `tests/conformance_artifacts.rs::latest_mode_holds_at_most_two_roots_delivered_plus_queued` (both) | | `bounded`: FIFO, no coalescing or silent loss; when capacity is unavailable, reject with `BACKPRESSURE` before admitting any delivery | conforms | `router/state.rs::op_publish` (pre-checks every subscriber) | `tests/pubsub.rs::bounded_fifo_and_atomic_backpressure` (both), `tests/conformance_routing.rs::bounded_overflow_rolls_back_all_artifact_roots` (both) | | maxInFlight credits return only on `delivery.consumed`, not on socket write completion | conforms | `router/state.rs::release_owner` (credit returned when the owner is released) | `tests/pubsub.rs::credits_return_only_on_consume` (both), `tests/conformance_routing.rs::bounded_credit_waits_for_every_extracted_artifact` (both) | -| A latest subscriber with all credits in use still has one replaceable queued value | conforms | `router/state.rs::{Sub::queue, dispatch_topic}` (queue and credits are separate) | `tests/bus_acceptance.rs::both_transports_produce_equivalent_behaviour_traces` (events 21 and 24: `publish replaced=1`, then `latest seq=3 replaced=1`) | +| A latest subscriber with all credits in use still has one replaceable queued value | conforms | `router/state.rs::{Sub::queue, dispatch_topic}` (queue and credits are separate) | `tests/bus_acceptance.rs::both_transports_produce_equivalent_behaviour_traces` (events 21 and 24: `publish replaced=1`, then `latest seq=3 replaced=1`), `tests/integration.rs::session_over_one_router` (both: a renderer held for the whole run keeps one delivery in flight and one replaceable value, and receives snapshots 1 and 20 of 20) | | Atomic subscriber/retention snapshot at admission; validate and reserve every queue entry and owner budget before accepting | conforms: one mutex, validate-then-mutate | `router/state.rs::op_publish` | `tests/artifacts.rs::failed_admission_is_atomic` (both) | | A bounded overflow rejects the whole publish: no partial fan-out, no retained-latest update | conforms | `router/state.rs::op_publish` | `tests/conformance_routing.rs::bounded_overflow_rolls_back_all_artifact_roots` (both) | | On acceptance, one `topicSequence` and roots for every delivery and the optional retained value | conforms; a refused publication spends no sequence number | `router/state.rs::op_publish` (`t.sequence += 1` after the checks) | `tests/pubsub.rs::bounded_fifo_and_atomic_backpressure` (both) | | Different topics have no total ordering; multiple publishers follow router acceptance order | conforms: per-topic sequence only | `router/state.rs::Topic::sequence` | `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) | | The publication reply counts accepted subscriptions and replaced queue entries, not consumers that processed data | conforms | `router/state.rs::op_publish` reply | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both) | -| `replaced` on a delivery reports how many undelivered messages were coalesced since that subscription's preceding delivery | conforms | `router/state.rs::{Sub::replaced, dispatch_topic}` (taken at dispatch) | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both) | +| `replaced` on a delivery reports how many undelivered messages were coalesced since that subscription's preceding delivery | conforms | `router/state.rs::{Sub::replaced, dispatch_topic}` (taken at dispatch) | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both), `tests/integration.rs::session_over_one_router` (both: the 18 replacements the publisher was told about at admission are the same 18 the renderer is told about on delivery, and are exactly the snapshots it did not receive) | | Optional `retained:latest` holds one last message and its artifacts independent of subscribers | conforms | `router/state.rs::op_publish` (retain branch) | `tests/conformance_artifacts.rs::retained_topic_value_holds_a_root_independent_of_subscribers` (both) | -| `replayLatest` enqueues the retained value before subsequent accepted publications; bounded preserves the order, latest may coalesce it | conforms | `router/state.rs::op_subscribe` (replay is enqueued under the subscribe lock) | `tests/conformance_routing.rs::latest_replay_is_ordered_ahead_of_a_racing_publish` (both), `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) | +| `replayLatest` enqueues the retained value before subsequent accepted publications; bounded preserves the order, latest may coalesce it | conforms | `router/state.rs::op_subscribe` (replay is enqueued under the subscribe lock) | `tests/conformance_routing.rs::latest_replay_is_ordered_ahead_of_a_racing_publish` (both: one racing publication to a bounded and a latest subscription at once; bounded must deliver replay then publication, and the latest branch is chosen by that publication's own `replaced` count, never by which side of the race the dispatcher won), `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) | | Replay uses the original topicSequence, a fresh deliveryId and explicit roots | conforms | `router/state.rs::op_subscribe` (`add_roots`, the same `Arc`) | `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) | | Without retention, a zero-subscriber publication retains no ownership after admission | conforms | `router/state.rs::op_publish` | `tests/pubsub.rs::zero_subscriber_publish_retains_nothing` (both) | | Clearing a topic releases only its retained root, not active consumers | conforms | `router/state.rs::op_clear` | `tests/conformance_routing.rs::cleared_topic_gives_no_replay_until_a_fresh_publish` (both) | @@ -260,7 +260,7 @@ and once over a Unix socket, so `tests/rpc.rs::request_reply_roundtrip` means | The thirteen transport error codes exist with those names | conforms | `error.rs::ErrorCode` | `tests/wire.rs::body_errors_keep_the_connection` (both) | | Three more codes: `CONFLICT`, `NO_TOPIC`, `ARTIFACT_MISMATCH` | deviates-allowed: "Transport errors **include** ..." is not an exhaustive list, and each names a refusal the draft requires but leaves unnamed. Recorded as an amendment in bus-v1 section 12 | `error.rs::ErrorCode` | `tests/pubsub.rs::subscription_and_topic_validation` (both), `tests/artifacts.rs::seal_checks_length_and_digest` (both) | | Before admission report `not-dispatched`; once dispatch might have occurred report `dispatched` or `unknown` conservatively | conforms | `error.rs::BusError::new` (not-dispatched by default), `router/state.rs` dispatched notices, `client/reactor.rs::fail_all` (unknown) | `tests/rpc.rs::{cancellation_states, service_disconnect_fails_calls}` (both), `tests/sol_review_races.rs::writer_failure_terminates_reader_and_pending_work` | -| Bounded subscriptions can reject a publication; latest spectators cannot hold a session transaction indefinitely | conforms: a latest subscriber never causes `BACKPRESSURE` | `router/state.rs::op_publish` (the latest branch skips every capacity check) | `tests/bus_acceptance.rs::a_latest_subscriber_never_refuses_a_publication` (both: 100 publications of 60 KB into one unconsumed slot, six times the bounded pool, none refused, 98 coalesced), with `tests/pubsub.rs::{bounded_fifo_and_atomic_backpressure, saturated_subscriber_does_not_block_control}` (both) for the bounded half | +| Bounded subscriptions can reject a publication; latest spectators cannot hold a session transaction indefinitely | conforms: a latest subscriber never causes `BACKPRESSURE` | `router/state.rs::op_publish` (the latest branch skips every capacity check) | `tests/bus_acceptance.rs::a_latest_subscriber_never_refuses_a_publication` (both: 100 publications of 60 KB into one unconsumed slot, six times the bounded pool, none refused, 98 coalesced), with `tests/pubsub.rs::{bounded_fifo_and_atomic_backpressure, saturated_subscriber_does_not_block_control}` (both) for the bounded half, and `tests/integration.rs::session_over_one_router` (both: 20 snapshot publications accepted by both subscriptions while the presentation consumer reads nothing) | | Sustained pinned-artifact quota exhaustion is surfaced as pressure, not solved by freeing live data | conforms: `QUOTA_EXCEEDED`, never eviction | `router/state.rs::{op_allocate, op_seal}` | `tests/artifacts.rs::quotas_are_enforced` (both) | | Session and application policies choose disconnect, pause or fail; the router does not know which | conforms by absence | `router/state.rs` | `tests/pubsub.rs::saturated_subscriber_does_not_block_control` (both) | @@ -284,7 +284,7 @@ and once over a Unix socket, so `tests/rpc.rs::request_reply_roundtrip` means | 3. Artifacts: allocate/seal/read; publication before seal fails; fan-out owns one object; the last consumer releases; a retained extracted frame survives a message drop | conforms | `tests/artifacts.rs` (28), `tests/conformance_artifacts.rs` (36) | | 4. Faults: sender drops after admission, consumer dies mid-read, reply lost, queued frame replaced, subscription closes with in-use deliveries, router restarts, old release arrives; no double-free, use-after-reuse, unbounded tombstones or hidden replay | conforms | `tests/conformance_routing.rs::{caller_disconnect_detaches_dispatched_call_but_service_keeps_serving, subscriber_disconnect_releases_queued_and_delivered_artifacts_but_not_retention}`, `tests/artifacts.rs::{release_ids_are_watermarked, router_restart_invalidates_old_handles}`, `tests/bus_acceptance.rs::{a_lost_result_leaks_no_roots_and_the_endpoint_cache_still_replays, disconnect_releases_logical_ownership_without_mutating_open_bytes}`, `tests/sol_rereview_regressions.rs` (11) | | 5. RPC cache: an endpoint retains an artifact-bearing result, the original caller consumes it, a domain retry still returns valid bytes, eviction drops the last hold | conforms | `tests/rpc.rs::endpoint_cache_replays_artifact_results` (both), `tests/bus_acceptance.rs::a_retransmission_repeats_the_domain_request_under_a_fresh_call_id` (both) | -| 6. Integration: two parallel fake agents, a complete-batch environment RPC, committed snapshot publication and a deliberately slow presentation consumer on one router | conforms | `tests/integration.rs::session_over_one_router` (both) | +| 6. Integration: two parallel fake agents, a complete-batch environment RPC, committed snapshot publication and a deliberately slow presentation consumer on one router | conforms; the consumer is slow by construction, held until the publisher's completion is observed, so its coalescing is forced rather than raced for | `tests/integration.rs::session_over_one_router` (both) | | 7. Performance: 640x480x60 with three consumers, one delayed; p50/p95/p99 RPC latency, router CPU, copy and readback cost separately, RSS, store live and peak bytes, outstanding roots, collection lag, queue lengths, for one, two and four agents | conforms | `tests/perf.rs::frames_at_60hz_with_three_consumers` (`--ignored`); numbers below | | The first executable example: a counter RPC, a pub/sub observer and a frame artifact held past message consumption, in one small Rust program, no game or browser | conforms | `examples/demo.rs` (`cargo run -p flybus --example demo`), asserted by `tests/example_demo.rs::the_example_shows_a_counter_rpc_an_observer_and_a_held_frame` | @@ -438,18 +438,30 @@ poll-until-the-router-settles the rest of the file already uses for router-side no sleep, no timing constant, and the assertion now has the precondition its contract sentence names. **360 runs after the fix, 0 failures** (240 debug, 120 release). -Two other intermittent failures were seen in the same sweep and are **not** fixed here, since -they belong to the bus slice rather than to this one: +Two other intermittent failures were seen in the same sweep. They belonged to the bus slice +rather than to this one and were fixed there, in the same way and for the same reason: - `tests/example_demo.rs::the_example_shows_a_counter_rpc_an_observer_and_a_held_frame`, - 2 failures in 40 standalone runs plus 1 in 12 full-suite runs. It prints + 2 failures in 40 standalone runs plus 1 in 12 full-suite runs. It printed "while the frame is held: 1 artifact(s), 2 root(s)" instead of 1 root: the producer's hold release is queued on the control lane and had not been applied when the example read the - counts. The same shape of gap, in the guide deliverable's printed output. -- `tests/integration.rs::unix_socket::session_over_one_router`, 1 failure in 12 full-suite - runs and 0 in 40 standalone runs, at the assertion that the deliberately slow consumer - skipped snapshots. Under load it kept up, so the assertion is a timing claim about the - machine. + counts. `examples/demo.rs` now waits for that release before reading the counts, the same + bounded poll it already used for collection eight lines below, so the line the guide quotes + is an observation rather than a race. The printed output is unchanged. +- `tests/integration.rs::session_over_one_router`, 1 failure in 12 full-suite runs and 0 in 40 + standalone runs, at the assertion that the deliberately slow consumer skipped snapshots. + Under load it kept up, so the assertion was a timing claim about the machine: section 7 + permits a latest subscriber to miss values, it does not oblige it to. The renderer is now + held until the publisher's twentieth receipt has returned -- the publisher's completion + observed, not timed -- so the coalescing is forced by construction, and the test asserts the + guarantees that do hold: each delivery carries the frame of the snapshot it announces, + deliveries arrive in publication order, the last value received is the latest published, the + renderer receives snapshots 1 and 20 of 20, both subscriptions accept all twenty publications + while the spectator reads nothing, and the eighteen replacements reported to the publisher at + admission are the same eighteen reported to the renderer on delivery and are exactly the + snapshots it did not receive. 16 failures in 40 runs beside four busy loops before, 0 in 40 + after; the whole crate went from 10 failed runs in 20 to 0, and the workspace suite from 2 + in 5 to 0. ## Contradictions diff --git a/docs/design/session-framework/checkpoint-envelope-v1.md b/docs/design/session-framework/checkpoint-envelope-v1.md index c83bb70..d58ed98 100644 --- a/docs/design/session-framework/checkpoint-envelope-v1.md +++ b/docs/design/session-framework/checkpoint-envelope-v1.md @@ -105,6 +105,31 @@ names; a manifest missing any of them is not a complete checkpoint. | `helperState` | External-helper state required for exact resume, as payload names | | `payloads` | `[{name, byteLength, digest}]`, mirroring the payload table | +**Amendment, 2026-09-22 (STATE-01).** The table above names a holder for every payload except +the environment's own, although section 6's fixture has one (`world`) and a group install has +to map it by name like any other participant's. The manifest therefore also records: + +| Field | Contents | +| --- | --- | +| `environment` | `{workerId, payload}`: which worker the world belonged to and the payload name holding its state | + +The reference implementations' required-field set was also missing `helperState`, which this +section has listed from the start. Both are now in `REQUIRED_MANIFEST_FIELDS` in Rust and in +TypeScript, and the fixture was regenerated by the existing example. The schema set is +untouched, so `contractDigest` is unchanged. + +`coordinator.eventWatermarks` is `{lastSourceStep, issued}`. The fixture illustrated +`{lastEventId, lastOrdinal}`, and it is the illustration that changed: an event id is derived +from the epoch, so a watermark spelled as one cannot be compared across the restore that +gives the session a new epoch, while a source step and an issued count can. + +**A required-manifest-field change is compatibility-relevant and `contractDigest` does not +cover it.** The digest is taken over the schema set, and this manifest is not in it, so +`envelopeVersion` is the only thing that can carry such a change. It stays `1` here only +because no production `FLYSESS1` file exists yet: once one does, adding or removing a required +manifest field **must** bump `envelopeVersion`, because a reader of the older version would +otherwise accept a file it cannot completely read, or refuse one it could. + `payloads` is redundant with the table on purpose: the table is what a reader needs to map bytes, and the manifest is what a store lists, compares and reports without opening the payload area. A reader checks that the two agree. diff --git a/docs/design/session-framework/state-media-v1.md b/docs/design/session-framework/state-media-v1.md index 2669786..0061761 100644 --- a/docs/design/session-framework/state-media-v1.md +++ b/docs/design/session-framework/state-media-v1.md @@ -190,6 +190,28 @@ restored time. It cannot advance gameplay to manufacture it. Capture/reconstruct covers render/inspection state and any pending sensor pipeline. Agent state agrees with it; do not replay reward or recalibrate merely to fill missing cached data. +**Amendment, 2026-09-22 (STATE-01).** Three readings of this section, made explicit because +they are now enforced: + +- `compatibilityDigest` on `CaptureResult` and `StageRestoreParams` is the **participant's** + capture compatibility digest of [worker interfaces](workers-v1.md) section 2 -- profile, + resolved seed, numerical model version and effective instance configuration for an agent; + backend, content, patch, controller and parser identity for an environment. It is not the + manifest's `compatibility` block of section 4, which is the composition's and which the + coordinator compares before anything is asked to stage. Both exist because they answer + different questions, and a restore that passed the second could still be handing an agent + another agent's brain. +- The observation `ActivateRestore` returns ran no transition, so it carries **no audio + chunk**, and one in it is refused. Section 2's chunk is the audio of an interval and this + observation covers none; MEDIA-01 implemented that rule as "boundary 0 carries no chunk", + which is true of the only such observation that slice could produce and false of this one. + The rule is about provenance, not about the boundary number. +- A participant that staged into a group install the coordinator then abandoned must be + **replaced** before another restore, exactly as one that activated must. It is holding a + validated replacement state that nothing installed, and [session RPC](ipc-v1.md) section 6 + already refuses to silently reattach such a participant to an active epoch. Without this the + group's second attempt meets its own leftovers and calls them a conflict. + If emulator validation requires mutation, stage a stopped replacement emulator. If that cannot provide externally atomic resume, advertise episode-restart, not exact-checkpoint. After all activation acknowledgments, install the coordinator's staged task/executor/admission state diff --git a/docs/rewards-learning.md b/docs/rewards-learning.md index e3130ee..1975629 100644 --- a/docs/rewards-learning.md +++ b/docs/rewards-learning.md @@ -1,6 +1,6 @@ # Rewards and learning -The live reward catalog of the Pokémon Red adapter, `pokered-unique8-v5`. The code of record is +The live reward catalog of the Pokémon Red adapter, `pokered-unique8-v6`. The code of record is `services/flysim/crates/flybrain-gb/src/pokemon_red/` (`catalog.rs` holds the values, `mod.rs` the gates and the rules); this page says what each rule pays for and why it is allowed to. The prototype's own `docs/rewards-learning.md` in `fly-plays-pokemon` is where the first seven rules @@ -24,6 +24,7 @@ change what the fly can do. | `battle` | `wildwin` | +0.1, +0.05, +0.0333 | 100 ms | At most three observed wild KOs per `(map, species, level)` | | `badge` | `badge` | +3 | 400 ms | Each newly set badge bit | | `boundary` | `explore` | +0.05, +0.10 | 100 ms | First tile adjacent to one of the map's exits, and the exit tile itself; once per `(map, exit)` for the lifetime of the ledger | +| `catch` | `wildwin` | +0.30, +0.10 | 150 ms | A wild Pokémon kept by a ball: +0.30 for a species this run had never owned, +0.10 for a repeat; at most three payouts per species for the lifetime of the ledger | Every value is positive: there are no loss or blackout penalties, and `catalog::rule("blackout")` is `None` by test. The values in one frame sum into `R`, and the network reinforces once with @@ -33,6 +34,54 @@ The feed-kind column is `RewardKind::from_adapter` in `services/flysim/crates/fl `docs/feed-protocol.md` publishes seven counters, and an adapter kind that has no counter of its own shares the nearest one. It still reaches the page as an event with its own label. +Two consequences of that sharing are worth stating rather than discovering. `catch` publishes on +`wildwin` because a catch is a wild battle the fly won by keeping the Pokémon, and *not* on +`pokedex` because the `species` rule already pays for the Pokédex bit the same catch sets -- +counting it twice would be the dishonest option. And the stage's ticker copy is keyed on the feed +kind, not on the catalog kind (`apps/stage/src/games/pokemon-red.ts`), so the row for a catch +currently reads "wild win". The event's own label, `CAUGHT #`, is what reaches the event +log, `/status` and the checkpoint. Changing the ticker copy means opening the feed's closed kind +set, which this rule deliberately did not do. + +## Catch rewards + +The operator's decision of 2026-09-22: the fly is paid for *keeping* a wild Pokémon, not only for +knocking one out. The rule is one kind with two payouts, the way `boundary` is. + +**How a catch is read.** From `wCapturedMonSpecies` (`$d11c`), whose comment in `ram/wram.asm` at +the pinned commit is "0 if no mon was captured". `ItemUseBall` zeroes it before every throw +(`.canUseBall`) and writes `wEnemyMonSpecies` into it only on the branch that keeps the Pokémon; +`UseBagItem`'s `.returnAfterCapturingMon` zeroes it again and sets `wBattleResult` to 2 on the way +out of the battle. `wBattleResult` is 2 on exactly two paths in the whole game -- that one, and a +link battle whose opponent ran -- so requiring both the species and the result means a byte read +out of a half-initialised battle cannot pay. The adapter records the species during the battle and +pays on the way out, where the wild-KO payout already lives. + +Not from `wPartyCount`. A catch with a full party raises `wBoxCount` instead, and `wPartyCount` +also rises for a gift, a trade and a Pokémon taken out of the PC, so it would need a second rule +to mean anything. The cartridge's own flag needs none. + +**What counts as a new species.** The `species` payout inside the same battle. Nothing but a catch +can set a `wPokedexOwned` bit during a wild battle, so a `species` payout between the battle +starting and the ball keeping the Pokémon *is* that Pokémon being new to the run. It is read this +way rather than off `wCapturedMonSpecies` because that byte is the cartridge's **internal** species +index while the owned bitset is by **Pokédex number**, and nothing in WRAM converts between the two +(`docs/design/macros-wram.md` section 2, "species numbering"). A battle restored from a checkpoint +written before this rule existed carries no "species payouts when it started", which reads as +"cannot tell" and pays the repeat amount: the conservative half, and at most 0.20 once. + +**The budget.** Three payouts per species for the lifetime of the ledger, the same cap and the +same reason as the wild-KO rule's three: a species the fly can find over and over is a farm, and +three is enough for the behaviour to be learned. A rollback blocks every species already paid, +exactly as it blocks every wild-KO key already paid, so the same catch cannot be replayed for +reward. A Safari Zone or old-man battle pays nothing, because the whole sample is dropped a step +earlier with a visible mode; a trainer battle pays nothing, because balls cannot be thrown in one. + +**The scale.** 0.30 on its own is below a new Pokédex entry (0.50), below a story flag (1.0) and +well below a badge (3.0). A catch of a new species pays 0.80 across two kinds, which sits between +a story flag and a badge -- deliberately, because it is the one event that is both a discovery and +a thing the fly had to do on purpose. + ## Gates Semantic rewards are enabled for exactly one cartridge, the SHA-256 in `SUPPORTED_ROM`. Any other @@ -128,7 +177,19 @@ body picks the macro; the descending neurons press the buttons.** ## Honesty -The catalog now includes exits. That is worth saying plainly on the honesty panel, because paying +The catalog now includes catches. The honesty panel's copy is not data-driven from the catalog -- +`apps/stage/src/lib/schedule.ts`'s rotating card is four written lines and lists no kinds -- so +there was nothing to regenerate and the copy is unchanged. The sentences below are where the +argument lives. + +Paying for a catch does not move the fly: the ball is thrown by a macro the mushroom body chose +among the ones the battle scene put on the pad, and the payout is read out of WRAM after the +frame. What it does do is make one of the palette's existing macros worth choosing, which is the +same kind of pressure every other rule applies. The cap is what keeps it from becoming a farm: a +run that finds one patch of grass and throws balls at the same species all night earns 0.50 from +it and then nothing. + +The catalog also includes exits. That is worth saying plainly on the honesty panel, because paying for a door is closer to telling the fly where to go than paying for a badge is: - **still no button path.** Nothing in the adapter chooses or biases a button. The reward is read diff --git a/docs/stream-mvp-plan.md b/docs/stream-mvp-plan.md index 29404f0..6588bfc 100644 --- a/docs/stream-mvp-plan.md +++ b/docs/stream-mvp-plan.md @@ -736,3 +736,11 @@ rewritten separately. (73/73) because 82% of the fixed run is battle time; Fable shipped it on the same judgement as v0.4.6 and started row 50 (MOVE n blocked on an unresponsive move list). The on-screen chat ring now survives a sim restart (sidecar in the hot dir, never in the checkpoint). +- 2026-09-22 (v0.5.0, the operator's decision): the fly is paid for keeping a wild Pokémon. New + catalog kind `catch` (0.30 for a species this run never caught, 0.10 for a repeat, three payouts + per species), read from the captured-species byte and the battle result together; the existing + species rule still pays on top. Adapter `pokered-unique8-v6`; the compatibility string differs + in the adapter segment only, and a deploy with `FLY_ACCEPT_ADAPTERS=pokered-unique8-v5` migrates + a v5 checkpoint instead of refusing it. `fly-reset-to-milestone ` restarts the run from a + ladder rung (archives both stores first). The live run restarts from rung 7 with this release, so + the ladder is climbed again with the catch reward and the row-54 walks in place. diff --git a/infra/05-deploy.sh b/infra/05-deploy.sh old mode 100755 new mode 100644 index 3b50552..705fb13 --- a/infra/05-deploy.sh +++ b/infra/05-deploy.sh @@ -186,6 +186,36 @@ else log "05-deploy: CPUSET unset — heavy in-container steps run unpinned (no partition configured)" fi +# Whether the only difference between two compatibility strings is the adapter +# segment, and FLY_ACCEPT_ADAPTERS names the adapter the live checkpoints carry. +# +# The bash half of flybrain_gb::compatibility::decide, which is what flysim +# itself applies at restore. Both have to agree: a gate that let a deploy +# through and a flysim that then refused every checkpoint would be the black +# stream this whole section exists to prevent. The string is +# {kernel}/{adapter}/{fingerprint}/{plasticity}/binjgb:{rev}/pokered:{commit}/statefmt:{id}, +# so the adapter is segment 1 and nothing else may move. +adapter_migration_accepted() { + local live="$1" new="$2" accepted="$3" + local -a live_parts new_parts + IFS='/' read -r -a live_parts <<< "$live" + IFS='/' read -r -a new_parts <<< "$new" + [ "${#live_parts[@]}" -eq "${#new_parts[@]}" ] || return 1 + local i differing=0 index=-1 + for ((i = 0; i < ${#live_parts[@]}; i++)); do + if [ "${live_parts[$i]}" != "${new_parts[$i]}" ]; then + differing=$((differing + 1)) + index=$i + fi + done + [ "$differing" -eq 1 ] && [ "$index" -eq 1 ] || return 1 + local entry + for entry in ${accepted//,/ }; do + [ "$entry" = "${live_parts[1]}" ] && return 0 + done + return 1 +} + # cpu_pin CMD [ARGS...] — run CMD inside the container on the page cpus. # Falls through to a plain ct_exec when no partition is configured, so this is # a no-op on an unpartitioned container rather than a new failure mode (a @@ -262,6 +292,16 @@ if [ -n "$RELEASE_TARBALL" ]; then # checkpoints and the on-screen chat ring's sidecar — so the new build warms # up fresh. Everything learned so far is thrown away, which is why it is not # the default. + # + # FLY_ACCEPT_ADAPTERS is the *other* override, and the opposite one: it keeps + # the run. It names adapter version strings whose checkpoints the new build + # may migrate — e.g. FLY_ACCEPT_ADAPTERS=pokered-unique8-v5 for the deploy + # that adds the catch reward. It only applies when the adapter segment is the + # ONLY difference between the two strings and the new build's adapter says it + # can read that one; a dataset, kernel, emulator or state-format change is + # still a refusal, because none of those has a migration. The same variable is + # written into /etc/fly/fly.env below, so flysim applies the same rule at + # restore that this gate applied at deploy. # ----------------------------------------------------------------------- state_dir="${FLY_STATE_DIR:-/srv/fly/state}" hot_dir="${FLY_STATE_HOT_DIR:-/run/fly/state}" @@ -287,6 +327,11 @@ if [ -n "$RELEASE_TARBALL" ]; then log "05-deploy: no decodable checkpoint in ${state_dir} — nothing to compare, continuing" elif [ "$new_compat" = "$live_compat" ]; then log "05-deploy: checkpoint compatibility matches the live state, the new build will restore it" + elif [ -n "${FLY_ACCEPT_ADAPTERS:-}" ] \ + && adapter_migration_accepted "$live_compat" "$new_compat" "$FLY_ACCEPT_ADAPTERS"; then + log "05-deploy: FLY_ACCEPT_ADAPTERS=${FLY_ACCEPT_ADAPTERS} — the adapter version is the only difference, and it is named; the run is KEPT and migrated" + log "05-deploy: live: $live_compat" + log "05-deploy: new: $new_compat" elif [ "${FLY_RESET_STATE:-0}" = 1 ]; then archive="${state_dir}.$(date -u +%Y%m%d%H%M%S)" log "05-deploy: FLY_RESET_STATE=1 — compatibility CHANGED, archiving the durable state to ${archive} and clearing the hot ring" @@ -300,8 +345,12 @@ if [ -n "$RELEASE_TARBALL" ]; then die "05-deploy: REFUSING to deploy release ${version}: its checkpoint compatibility string does not match the live state in ${state_dir}, so flysim would refuse every checkpoint there and then refuse to start at all — a black stream. live state: ${live_compat} new build: ${new_compat} -The difference is usually an adapter/ladder or dataset version bump. Two ways forward: +The difference is usually an adapter/ladder or dataset version bump. Three ways forward: * deploy a build whose string matches (check out the commit the running release was built from), or + * if the ADAPTER VERSION is the only segment that differs and the new build documents a + migration from the old one, re-run with FLY_ACCEPT_ADAPTERS set to the adapter id in the live + string (e.g. FLY_ACCEPT_ADAPTERS=pokered-unique8-v5). The run is kept; flysim applies the same + rule at restore. See docs/design/flysim.md, \"Restoring across an adapter version\", or * accept losing everything the brain has learned and re-run with FLY_RESET_STATE=1, which archives ${state_dir}'s checkpoints to ${state_dir}. (kept, not deleted) and clears ${hot_dir} so the new build warms up fresh. @@ -457,6 +506,16 @@ trap 'rm -f "$tmp_fly_env" "$tmp_flypush_env"' EXIT if [[ -n "${FLY_MACRO_BLOCKED_MINUTES:-}" ]]; then echo "FLY_MACRO_BLOCKED_MINUTES=${FLY_MACRO_BLOCKED_MINUTES}" fi + # Adapter versions whose checkpoints this build may migrate + # (flybrain_gb::compatibility, docs/design/flysim.md "Restoring across an + # adapter version"). Only written when it is set, because the safe state is + # absent: an empty or missing variable migrates nothing, which is what every + # deploy before 2026-09-22 did. It stays in fly.env for as long as the + # operator leaves it on the deploy command line, so removing the opt-in is + # one deploy without it. + if [[ -n "${FLY_ACCEPT_ADAPTERS:-}" ]]; then + echo "FLY_ACCEPT_ADAPTERS=${FLY_ACCEPT_ADAPTERS}" + fi # flybridge (services/bridge/src/config.ts). Nothing wrote these before, so # flybridge.service had no EnvironmentFile= at all and the service refused to # start with "CHANNEL is required / BOT_USER is required / GAME_TITLE is @@ -607,7 +666,7 @@ fi # --------------------------------------------------------------------------- log "05-deploy: converging bin/ helpers to /opt/fly/bin" ct_exec "$CTID" -- mkdir -p /opt/fly/bin -for name in fly-watchdog fly-recap fly-retention flypush flystage-launch flycast-launch wait-for-x wait-for-stage wait-for-health; do +for name in fly-watchdog fly-recap fly-retention fly-reset-to-milestone flypush flystage-launch flycast-launch wait-for-x wait-for-stage wait-for-health; do converge_file "$CTID" "$INFRA_DIR/bin/$name" "/opt/fly/bin/$name" 0755 root:root >/dev/null done diff --git a/infra/bin/fly-reset-to-milestone b/infra/bin/fly-reset-to-milestone new file mode 100755 index 0000000..732c343 --- /dev/null +++ b/infra/bin/fly-reset-to-milestone @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# infra/bin/fly-reset-to-milestone — restart the run from an earlier ladder rung, +# instead of from scratch. +# +# The operator's decision of 2026-09-22: "restart the live run from an early +# checkpoint instead of from scratch". 05-deploy's FLY_RESET_STATE=1 cannot do +# that — it archives the durable state and the next start warms up a fresh fly, +# losing everything the brain has learned. This promotes one milestone archive +# (milestone-.checkpoint, written at the first commit at a new best rank and +# never rotated away) to being what both stores restore. +# +# Usage: fly-reset-to-milestone +# Run INSIDE the container, as root, with flysim STOPPED. It refuses +# otherwise, and it refuses a rung this run never reached. +# +# The whole sequence — stop, reset, deploy with the adapter opt-in, start, +# verify the rank — is in infra/docs/runbook.md, "Restart the run from a rung". +# Nothing here is destructive on its own: every file in both stores is copied to +# a dated directory next to the durable one before anything is rewritten. +set -euo pipefail + +: "${FLY_STATE_DIR:=/srv/fly/state}" +: "${FLY_STATE_HOT_DIR:=/run/fly/state}" +: "${FLY_RELEASE_DIR:=/opt/fly/current}" +: "${FLY_SERVICE:=flysim.service}" +: "${FLY_USER:=fly}" + +FLYSIM="${FLY_BIN:-${FLY_RELEASE_DIR}/flysim}" + +log() { echo "fly-reset-to-milestone: $*" >&2; } +die() { log "$*"; exit 1; } + +RANK="${1:-}" +if [ "$#" -ne 1 ] || ! [[ "$RANK" =~ ^[0-9]+$ ]]; then + die "usage: fly-reset-to-milestone (e.g. fly-reset-to-milestone 9)" +fi + +# --- refusals ---------------------------------------------------------------- +# A running flysim owns both stores: it commits a hot checkpoint every few +# seconds and a durable one every few minutes, so a reset underneath it would be +# overwritten within the minute and the tool would have lied. +if command -v systemctl >/dev/null 2>&1 && systemctl is-active --quiet "$FLY_SERVICE"; then + die "$FLY_SERVICE is running. Stop it first: systemctl stop $FLY_SERVICE" +fi +[ -x "$FLYSIM" ] || die "no flysim binary at $FLYSIM (set FLY_BIN to point at one)" + +milestone="${FLY_STATE_DIR}/milestone-${RANK}.checkpoint" +# The binary refuses this too, and refuses before it copies anything; checking +# here as well is what makes the message name the rungs that do exist. +if [ ! -f "$milestone" ]; then + log "no milestone archive for rung ${RANK}: $milestone does not exist." + log "rungs this run reached:" + ls -1 "${FLY_STATE_DIR}"/milestone-*.checkpoint 2>/dev/null \ + | sed 's|.*/milestone-||; s|\.checkpoint$||' | sort -n | tr '\n' ' ' >&2 || true + echo >&2 + exit 1 +fi + +# --- the reset --------------------------------------------------------------- +log "resetting to rung ${RANK} (durable ${FLY_STATE_DIR}, hot ${FLY_STATE_HOT_DIR})" +FLY_STATE="$FLY_STATE_DIR" FLY_STATE_HOT="$FLY_STATE_HOT_DIR" \ + "$FLYSIM" --reset-to-milestone "$RANK" + +# flysim runs unprivileged; this tool runs as root, so everything it wrote and +# everything it archived has to go back to the service account. +if command -v chown >/dev/null 2>&1 && id "$FLY_USER" >/dev/null 2>&1; then + chown -R "${FLY_USER}:${FLY_USER}" "$FLY_STATE_DIR" "$FLY_STATE_HOT_DIR" 2>/dev/null || true + for dir in "${FLY_STATE_DIR}".reset-*; do + [ -d "$dir" ] && chown -R "${FLY_USER}:${FLY_USER}" "$dir" + done +fi + +log "done. Next, per infra/docs/runbook.md:" +log " 1. deploy the build whose adapter wrote that checkpoint, or deploy the new" +log " one with FLY_ACCEPT_ADAPTERS set to the checkpoint's adapter id" +log " 2. systemctl start $FLY_SERVICE" +log " 3. curl -s localhost:7401/status | grep -o '\"rank\":[0-9]*'" diff --git a/infra/docs/runbook.md b/infra/docs/runbook.md index a07a413..97e4efa 100644 --- a/infra/docs/runbook.md +++ b/infra/docs/runbook.md @@ -231,6 +231,59 @@ auto-reset (`docs/design/flysim.md` section 8: "no automatic fresh start, ever") is deliberate — a silent reset would be indistinguishable from real progress on stream. A deliberate reset means moving `/srv/fly/state` aside by hand. +## Restart the run from a rung + +When the run has to go back to an earlier milestone rather than start over — the operator's +decision of 2026-09-22 was "restart the live run from an early checkpoint instead of from +scratch". `FLY_RESET_STATE=1` is the wrong tool: it archives the durable state and the next start +warms up a fresh fly, losing everything the brain has learned. + +`infra/bin/fly-reset-to-milestone ` promotes `milestone-.checkpoint` to being what both +stores restore, with the ratchet's attempts and recoveries back at zero. It copies every file in +both stores to `/srv/fly/state.reset-` first, so it is reversible by hand. It refuses while +flysim is running, and refuses a rung this run never reached. + +The whole sequence, in order. Claim the container in the host's agent claim log first, like any +other work on it. + +``` +CTID= +N=9 # the rung to restart from + +# 1. what rungs exist at all +pct exec $CTID -- ls -1 /srv/fly/state/milestone-*.checkpoint + +# 2. stop flysim (it owns both stores; a reset underneath it is overwritten within the minute) +pct exec $CTID -- systemctl stop flysim.service + +# 3. the reset. Prints what it did, one line per step. +pct exec $CTID -- /opt/fly/bin/fly-reset-to-milestone $N + +# 4. deploy. Two cases: +# (a) the running release already wrote that checkpoint -> nothing to deploy, skip to 5. +# (b) the new build bumps the ADAPTER VERSION and nothing else -> name the checkpoint's +# adapter so the gate and flysim both migrate instead of refusing: +FLY_ACCEPT_ADAPTERS=pokered-unique8-v5 infra/05-deploy.sh +# The gate logs "the adapter version is the only difference, and it is named; the run is KEPT +# and migrated", and writes FLY_ACCEPT_ADAPTERS into /etc/fly/fly.env so flysim applies the +# same rule at restore. Anything else about the string differing is still a refusal. + +# 5. start +pct exec $CTID -- systemctl start flysim.service + +# 6. verify: the rank is the rung, and the restore came from the generation the tool wrote +pct exec $CTID -- curl -s http://127.0.0.1:7401/status | jq '.milestone.rank, .game.badges, .checkpoint' +pct exec $CTID -- journalctl -u flysim -n 40 --no-pager | grep -E 'restored|migration|compatibility' +``` + +Step 6 is the one that must be read rather than assumed. The rank is recomputed by the adapter +from the restored game state, not taken from the ratchet, so a rank that is *not* N means the +milestone archive was taken somewhere other than where its name says — stop and look before +starting a stream on it. + +To undo: stop flysim, move the contents of `/srv/fly/state.reset-/durable` back into +`/srv/fly/state`, delete the generation the tool wrote, and start again. + ## Restore from the backup host ``` diff --git a/infra/env/example.env b/infra/env/example.env index ddab36f..fc19970 100644 --- a/infra/env/example.env +++ b/infra/env/example.env @@ -317,6 +317,20 @@ FLY_MACRO_MODE=raw # target once more. Unset means the default, 10. # FLY_MACRO_BLOCKED_MINUTES=10 +# --- restoring across an adapter version ------------------------------------ +# Adapter version strings whose checkpoints this build may migrate, comma- or +# space-separated (docs/design/flysim.md, "Restoring across an adapter +# version"). Unset -- the default, and what every deploy before 2026-09-22 did +# -- migrates nothing: a build whose compatibility string differs from the live +# state's is refused by 05-deploy's gate and by flysim at restore. +# +# It applies only when the ADAPTER segment is the only difference between the +# two strings AND the new build's adapter declares a migration from that one. A +# dataset, kernel, plasticity, emulator or state-format difference is still a +# refusal. Set it for the one deploy that needs it and leave it out afterwards; +# 05-deploy writes it into /etc/fly/fly.env only while it is set. +# FLY_ACCEPT_ADAPTERS=pokered-unique8-v5 + # --- push mode -------------------------------------------------------------- # local: flypush.service stays disabled, everything else identical to prod. # twitch: flypush.service is enabled by 07-enable.sh. diff --git a/packages/session-types/src/checkpoint.ts b/packages/session-types/src/checkpoint.ts index 63aac19..2df48f7 100644 --- a/packages/session-types/src/checkpoint.ts +++ b/packages/session-types/src/checkpoint.ts @@ -223,7 +223,12 @@ export function decode(input: Uint8Array): Envelope { }; } -/** The manifest fields state-media-v1 section 4 requires. */ +/** + * The manifest fields state-media-v1 section 4 requires. + * + * `helperState` and `environment` join the list under the 2026-09-22 amendment to + * checkpoint-envelope-v1 section 3. + */ export const REQUIRED_MANIFEST_FIELDS = [ 'envelopeVersion', 'checkpointId', @@ -236,6 +241,8 @@ export const REQUIRED_MANIFEST_FIELDS = [ 'compatibility', 'agents', 'coordinator', + 'environment', + 'helperState', 'payloads', ] as const; diff --git a/services/flysim/crates/fly-session-types/examples/update_fixtures.rs b/services/flysim/crates/fly-session-types/examples/update_fixtures.rs index 238287e..bad3693 100644 --- a/services/flysim/crates/fly-session-types/examples/update_fixtures.rs +++ b/services/flysim/crates/fly-session-types/examples/update_fixtures.rs @@ -197,8 +197,9 @@ fn checkpoint_envelope() -> String { "priorInspection": "prior-inspection", "executorState": [{"agentId": "fly-a", "payload": "executor-fly-a"}], "admissionState": null, - "eventWatermarks": {"lastEventId": "evt-1", "lastOrdinal": "7"}, + "eventWatermarks": {"lastSourceStep": "42", "issued": "7"}, }, + "environment": {"workerId": "arena", "payload": "world"}, "helperState": [], "payloads": payload_table(), }); diff --git a/services/flysim/crates/fly-session-types/fixtures/checkpoint-envelope.json b/services/flysim/crates/fly-session-types/fixtures/checkpoint-envelope.json index 6636bae..b726585 100644 --- a/services/flysim/crates/fly-session-types/fixtures/checkpoint-envelope.json +++ b/services/flysim/crates/fly-session-types/fixtures/checkpoint-envelope.json @@ -59,10 +59,14 @@ ], "admissionState": null, "eventWatermarks": { - "lastEventId": "evt-1", - "lastOrdinal": "7" + "lastSourceStep": "42", + "issued": "7" } }, + "environment": { + "workerId": "arena", + "payload": "world" + }, "helperState": [], "payloads": [ { @@ -115,49 +119,49 @@ } ], "envelope": { - "base64": "RkxZU0VTUzEBAAAAIAAAAAAIAAAFAAAAIAgAAAAAAAB7ImFnZW50cyI6W3siYWdlbnRJZCI6ImZseS1hIiwiYnJhaW5UaWNrcyI6IjI1MzQiLCJkYXRhc2V0RGlnZXN0IjoiNmMwYWYxZjA3ODRlZjYzYTM5M2VlNzdkNjE0ZTgyNDZjNjI1MDUxMzYwZjNmMWE0ODgzODM3NGM1ZDM1NWI1MiIsIm1vZGVsVmVyc2lvbiI6ImxpZi0xbXMtZjY0LXYyIiwicGF5bG9hZCI6ImFnZW50LWZseS1hIiwicGxhc3RpY2l0eVZlcnNpb24iOiJmbHkta2MtbWJvbi1yc3RkcC12MiIsInByb2ZpbGVEaWdlc3QiOiIxOTAwZWFiNmMwMjg0ODNkNzEyNjU5OWVlNmY1MGRlMGQyNzkwN2I1YzY1ZmE5MDUyNDU4MGI0YjBmOTg1MmIwIiwicmVtYWluZGVyIjp7ImRlbm9taW5hdG9yIjoiMyIsIm51bWVyYXRvciI6IjEwMDAwMDAifSwic2VlZCI6LTE4NDk0NjA2M31dLCJjaGVja3BvaW50SWQiOiJja3B0LTEiLCJjb21wYXRpYmlsaXR5Ijp7ImJhY2tlbmREaWdlc3QiOiIxMGUwOGE0MTllODUwZWJhMWViYmExOGZkZDI4ZWI3ZWMxYjdlOGJhYTliY2MzYjk3M2UyYjg4OTFlYzcyNmJlIiwiY29udGVudERpZ2VzdCI6ImVkNzAwMmI0MzllOWFjODQ1ZjIyMzU3ZDgyMmJhYzE0NDQ3MzBmYmRiNjAxNmQzZWM5NDMyMjk3YjllYzlmNzMiLCJjb250cm9sbGVyRGlnZXN0IjoiYzE0NzIxMzViMTRjNzdjOGJlZjk4ZTczZjcwMjA4MzI1ZmEwZGNmMWU2YmQ2NjhhZTliMzFhOWNlYTI5NWZlNyIsInBhcnNlckRpZ2VzdCI6ImIxN2Q0NTEyMTE1MDkyOGYyMTQ2YWY0OWUxOTVlZmYxZWVmNWQ2NzMyNWJlMjczYTczM2ZiNzRhY2FkYWEzNDIiLCJwYXRjaERpZ2VzdCI6ImE0ODk1ZWI0NGFmYzMzNmZlY2JiYTZlNTIwY2Q2N2UxNzhkYWNlMDI3NjY1NWQxMDJmY2VmZmE4ZTVmNzA1NzAiLCJzdGF0ZUZvcm1hdElkIjoiZmx5c2Vzcy0xIn0sImNvbXBvc2l0aW9uRGlnZXN0IjoiNzMwZDcyNWM4YTU5ZDNhNzMwM2RlZjJiZWQwNDFhNTc3ZWRiNDI1NWFhYmQ0ODg5Y2UxMjkxODMxMWQ5NTJmMCIsImNvb3JkaW5hdG9yIjp7ImFkbWlzc2lvblN0YXRlIjpudWxsLCJldmVudFdhdGVybWFya3MiOnsibGFzdEV2ZW50SWQiOiJldnQtMSIsImxhc3RPcmRpbmFsIjoiNyJ9LCJleGVjdXRvclN0YXRlIjpbeyJhZ2VudElkIjoiZmx5LWEiLCJwYXlsb2FkIjoiZXhlY3V0b3ItZmx5LWEifV0sInByaW9ySW5zcGVjdGlvbiI6InByaW9yLWluc3BlY3Rpb24iLCJ0YXNrTGVkZ2VyIjoidGFzay1sZWRnZXIifSwiZW52ZWxvcGVWZXJzaW9uIjoxLCJlcGlzb2RlSWQiOiJlcGlzb2RlLTEiLCJoZWxwZXJTdGF0ZSI6W10sInBheWxvYWRzIjpbeyJieXRlTGVuZ3RoIjoiMTciLCJkaWdlc3QiOiIxMzIxZGZmYjBjZGM2ZjkwOTJjYmY3ZmEyYTVmYzY4YmJlZDEyYzk5M2Q1YWQzOTgyNjQwMTI4MTBjZTliZjkzIiwibmFtZSI6ImFnZW50LWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTQiLCJkaWdlc3QiOiIzYWVlNjBkZjdlMjllZmViYTdmNWY5OWZjNTg2NzY0N2IzNmFlYmZmMWQ1ZDNjODM4ZGJmZjMyMzEyMmU2NDYyIiwibmFtZSI6ImV4ZWN1dG9yLWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTEiLCJkaWdlc3QiOiI0MGIwMGVkMmJiYmE5MDFkNjgyMDVmZjcxYjA0YTQ0YjllZTUzYzUxY2IzMTA5YWEyY2VhYTQ0ZjFjNDU3MjdlIiwibmFtZSI6InRhc2stbGVkZ2VyIn0seyJieXRlTGVuZ3RoIjoiMTAiLCJkaWdlc3QiOiIyYzEzYjdiNGQ5YTk5MTY4MDFhYjkxOTFjMzE0ZjMxYjA0NWU5YjljNWI2NjlhNmMwNDc0ZjAyMTdlZjc1YmY1IiwibmFtZSI6InByaW9yLWluc3BlY3Rpb24ifSx7ImJ5dGVMZW5ndGgiOiI2NCIsImRpZ2VzdCI6ImY1YTVmZDQyZDE2YTIwMzAyNzk4ZWY2ZWQzMDk5NzliNDMwMDNkMjMyMGQ5ZjBlOGVhOTgzMWE5Mjc1OWZiNGIiLCJuYW1lIjoid29ybGQifV0sInBvcnRNYXAiOlt7ImFnZW50SWQiOiJmbHktYSIsInBvcnRJZCI6InBvcnQtMSJ9XSwic2NoZWR1bGVySWQiOiJsb2Nrc3RlcC12MSIsInNvdXJjZVNjb3BlIjp7ImVwb2NoIjoiZXBvY2gtMSIsInNlc3Npb25JZCI6ImRlbW8iLCJzdGVwIjoiNDIifSwid29ybGRUaW1lIjp7ImRlbm9taW5hdG9yIjoiMSIsIm51bWVyYXRvciI6IjcwMDAwMDAwMCJ9fWFnZW50LWZseS1hAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABQCgAAAAAAABEAAAAAAAAAEyHf+wzcb5CSy/f6Kl/Gi77RLJk9WtOYJkASgQzpv5NleGVjdXRvci1mbHktYQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaAoAAAAAAAAOAAAAAAAAADruYN9+Ke/rp/X5n8WGdkezauv/HV08g42/8yMSLmRidGFzay1sZWRnZXIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHgKAAAAAAAACwAAAAAAAABAsA7Su7qQHWggX/cbBKRLnuU8UcsxCaos6qRPHEVyfnByaW9yLWluc3BlY3Rpb24AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACICgAAAAAAAAoAAAAAAAAALBO3tNmpkWgBq5GRwxTzGwRem5xbZppsBHTwIX73W/V3b3JsZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAmAoAAAAAAABAAAAAAAAAAPWl/ULRaiAwJ5jvbtMJl5tDAD0jINnw6OqYMaknWftLYWdlbnQgc3RhdGUgYnl0ZXMAAAAAAAAAZXhlY3V0b3Igc3RhdGUAAHsicmFuayI6MTB9AAAAAAB7Im1hcCI6NDB9AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgLAAAAAAAAq++fEx+FvDZho/eB4imbENN4HZrGNC2OCAsI7/gp9r5GTFlTRVNTRg==", - "byteLength": 2824, + "base64": "RkxZU0VTUzEBAAAAIAAAADAIAAAFAAAAUAgAAAAAAAB7ImFnZW50cyI6W3siYWdlbnRJZCI6ImZseS1hIiwiYnJhaW5UaWNrcyI6IjI1MzQiLCJkYXRhc2V0RGlnZXN0IjoiNmMwYWYxZjA3ODRlZjYzYTM5M2VlNzdkNjE0ZTgyNDZjNjI1MDUxMzYwZjNmMWE0ODgzODM3NGM1ZDM1NWI1MiIsIm1vZGVsVmVyc2lvbiI6ImxpZi0xbXMtZjY0LXYyIiwicGF5bG9hZCI6ImFnZW50LWZseS1hIiwicGxhc3RpY2l0eVZlcnNpb24iOiJmbHkta2MtbWJvbi1yc3RkcC12MiIsInByb2ZpbGVEaWdlc3QiOiIxOTAwZWFiNmMwMjg0ODNkNzEyNjU5OWVlNmY1MGRlMGQyNzkwN2I1YzY1ZmE5MDUyNDU4MGI0YjBmOTg1MmIwIiwicmVtYWluZGVyIjp7ImRlbm9taW5hdG9yIjoiMyIsIm51bWVyYXRvciI6IjEwMDAwMDAifSwic2VlZCI6LTE4NDk0NjA2M31dLCJjaGVja3BvaW50SWQiOiJja3B0LTEiLCJjb21wYXRpYmlsaXR5Ijp7ImJhY2tlbmREaWdlc3QiOiIxMGUwOGE0MTllODUwZWJhMWViYmExOGZkZDI4ZWI3ZWMxYjdlOGJhYTliY2MzYjk3M2UyYjg4OTFlYzcyNmJlIiwiY29udGVudERpZ2VzdCI6ImVkNzAwMmI0MzllOWFjODQ1ZjIyMzU3ZDgyMmJhYzE0NDQ3MzBmYmRiNjAxNmQzZWM5NDMyMjk3YjllYzlmNzMiLCJjb250cm9sbGVyRGlnZXN0IjoiYzE0NzIxMzViMTRjNzdjOGJlZjk4ZTczZjcwMjA4MzI1ZmEwZGNmMWU2YmQ2NjhhZTliMzFhOWNlYTI5NWZlNyIsInBhcnNlckRpZ2VzdCI6ImIxN2Q0NTEyMTE1MDkyOGYyMTQ2YWY0OWUxOTVlZmYxZWVmNWQ2NzMyNWJlMjczYTczM2ZiNzRhY2FkYWEzNDIiLCJwYXRjaERpZ2VzdCI6ImE0ODk1ZWI0NGFmYzMzNmZlY2JiYTZlNTIwY2Q2N2UxNzhkYWNlMDI3NjY1NWQxMDJmY2VmZmE4ZTVmNzA1NzAiLCJzdGF0ZUZvcm1hdElkIjoiZmx5c2Vzcy0xIn0sImNvbXBvc2l0aW9uRGlnZXN0IjoiNzMwZDcyNWM4YTU5ZDNhNzMwM2RlZjJiZWQwNDFhNTc3ZWRiNDI1NWFhYmQ0ODg5Y2UxMjkxODMxMWQ5NTJmMCIsImNvb3JkaW5hdG9yIjp7ImFkbWlzc2lvblN0YXRlIjpudWxsLCJldmVudFdhdGVybWFya3MiOnsiaXNzdWVkIjoiNyIsImxhc3RTb3VyY2VTdGVwIjoiNDIifSwiZXhlY3V0b3JTdGF0ZSI6W3siYWdlbnRJZCI6ImZseS1hIiwicGF5bG9hZCI6ImV4ZWN1dG9yLWZseS1hIn1dLCJwcmlvckluc3BlY3Rpb24iOiJwcmlvci1pbnNwZWN0aW9uIiwidGFza0xlZGdlciI6InRhc2stbGVkZ2VyIn0sImVudmVsb3BlVmVyc2lvbiI6MSwiZW52aXJvbm1lbnQiOnsicGF5bG9hZCI6IndvcmxkIiwid29ya2VySWQiOiJhcmVuYSJ9LCJlcGlzb2RlSWQiOiJlcGlzb2RlLTEiLCJoZWxwZXJTdGF0ZSI6W10sInBheWxvYWRzIjpbeyJieXRlTGVuZ3RoIjoiMTciLCJkaWdlc3QiOiIxMzIxZGZmYjBjZGM2ZjkwOTJjYmY3ZmEyYTVmYzY4YmJlZDEyYzk5M2Q1YWQzOTgyNjQwMTI4MTBjZTliZjkzIiwibmFtZSI6ImFnZW50LWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTQiLCJkaWdlc3QiOiIzYWVlNjBkZjdlMjllZmViYTdmNWY5OWZjNTg2NzY0N2IzNmFlYmZmMWQ1ZDNjODM4ZGJmZjMyMzEyMmU2NDYyIiwibmFtZSI6ImV4ZWN1dG9yLWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTEiLCJkaWdlc3QiOiI0MGIwMGVkMmJiYmE5MDFkNjgyMDVmZjcxYjA0YTQ0YjllZTUzYzUxY2IzMTA5YWEyY2VhYTQ0ZjFjNDU3MjdlIiwibmFtZSI6InRhc2stbGVkZ2VyIn0seyJieXRlTGVuZ3RoIjoiMTAiLCJkaWdlc3QiOiIyYzEzYjdiNGQ5YTk5MTY4MDFhYjkxOTFjMzE0ZjMxYjA0NWU5YjljNWI2NjlhNmMwNDc0ZjAyMTdlZjc1YmY1IiwibmFtZSI6InByaW9yLWluc3BlY3Rpb24ifSx7ImJ5dGVMZW5ndGgiOiI2NCIsImRpZ2VzdCI6ImY1YTVmZDQyZDE2YTIwMzAyNzk4ZWY2ZWQzMDk5NzliNDMwMDNkMjMyMGQ5ZjBlOGVhOTgzMWE5Mjc1OWZiNGIiLCJuYW1lIjoid29ybGQifV0sInBvcnRNYXAiOlt7ImFnZW50SWQiOiJmbHktYSIsInBvcnRJZCI6InBvcnQtMSJ9XSwic2NoZWR1bGVySWQiOiJsb2Nrc3RlcC12MSIsInNvdXJjZVNjb3BlIjp7ImVwb2NoIjoiZXBvY2gtMSIsInNlc3Npb25JZCI6ImRlbW8iLCJzdGVwIjoiNDIifSwid29ybGRUaW1lIjp7ImRlbm9taW5hdG9yIjoiMSIsIm51bWVyYXRvciI6IjcwMDAwMDAwMCJ9fWFnZW50LWZseS1hAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACACgAAAAAAABEAAAAAAAAAEyHf+wzcb5CSy/f6Kl/Gi77RLJk9WtOYJkASgQzpv5NleGVjdXRvci1mbHktYQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAmAoAAAAAAAAOAAAAAAAAADruYN9+Ke/rp/X5n8WGdkezauv/HV08g42/8yMSLmRidGFzay1sZWRnZXIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKgKAAAAAAAACwAAAAAAAABAsA7Su7qQHWggX/cbBKRLnuU8UcsxCaos6qRPHEVyfnByaW9yLWluc3BlY3Rpb24AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC4CgAAAAAAAAoAAAAAAAAALBO3tNmpkWgBq5GRwxTzGwRem5xbZppsBHTwIX73W/V3b3JsZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAyAoAAAAAAABAAAAAAAAAAPWl/ULRaiAwJ5jvbtMJl5tDAD0jINnw6OqYMaknWftLYWdlbnQgc3RhdGUgYnl0ZXMAAAAAAAAAZXhlY3V0b3Igc3RhdGUAAHsicmFuayI6MTB9AAAAAAB7Im1hcCI6NDB9AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADgLAAAAAAAAX4L9WkdX0MViD3h5YJgf7VQgwocWU4XJoKX6MUwl+8hGTFlTRVNTRg==", + "byteLength": 2872, "layout": { "headerBytes": 32, "manifestOffset": "32", - "manifestBytes": 2048, - "tableOffset": "2080", + "manifestBytes": 2096, + "tableOffset": "2128", "tableEntryBytes": 112, "entries": [ { "name": "agent-fly-a", - "offset": "2640", + "offset": "2688", "byteLength": "17", "digest": "1321dffb0cdc6f9092cbf7fa2a5fc68bbed12c993d5ad398264012810ce9bf93" }, { "name": "executor-fly-a", - "offset": "2664", + "offset": "2712", "byteLength": "14", "digest": "3aee60df7e29efeba7f5f99fc5867647b36aebff1d5d3c838dbff323122e6462" }, { "name": "task-ledger", - "offset": "2680", + "offset": "2728", "byteLength": "11", "digest": "40b00ed2bbba901d68205ff71b04a44b9ee53c51cb3109aa2ceaa44f1c45727e" }, { "name": "prior-inspection", - "offset": "2696", + "offset": "2744", "byteLength": "10", "digest": "2c13b7b4d9a9916801ab9191c314f31b045e9b9c5b669a6c0474f0217ef75bf5" }, { "name": "world", - "offset": "2712", + "offset": "2760", "byteLength": "64", "digest": "f5a5fd42d16a20302798ef6ed309979b43003d2320d9f0e8ea9831a92759fb4b" } ], - "footerOffset": "2776", + "footerOffset": "2824", "footerBytes": 48, - "totalBytes": "2824" + "totalBytes": "2872" } }, "corruption": [ @@ -178,17 +182,17 @@ }, { "name": "a flipped payload byte", - "offset": 2640, + "offset": 2688, "reason": "every payload carries its own digest" }, { "name": "a flipped footer digest byte", - "offset": 2784, + "offset": 2832, "reason": "the footer digest must match the contents" }, { "name": "a flipped footer magic byte", - "offset": 2816, + "offset": 2864, "reason": "a truncated file cannot look complete" } ] diff --git a/services/flysim/crates/fly-session-types/src/checkpoint.rs b/services/flysim/crates/fly-session-types/src/checkpoint.rs index 341f5c5..985fcb1 100644 --- a/services/flysim/crates/fly-session-types/src/checkpoint.rs +++ b/services/flysim/crates/fly-session-types/src/checkpoint.rs @@ -296,6 +296,11 @@ pub fn decode(bytes: &[u8]) -> Result { /// The manifest fields state-media-v1 section 4 requires, checked as a set: a manifest that /// omits one of them is not a complete checkpoint. +/// +/// `helperState` and `environment` join the list under the 2026-09-22 amendment to +/// checkpoint-envelope-v1 section 3: the first has been in that section's table from the +/// start and was missing here, and the second is the holder of the world's own payload, which +/// the table named for every other participant and not for the environment. pub const REQUIRED_MANIFEST_FIELDS: &[&str] = &[ "envelopeVersion", "checkpointId", @@ -308,6 +313,8 @@ pub const REQUIRED_MANIFEST_FIELDS: &[&str] = &[ "compatibility", "agents", "coordinator", + "environment", + "helperState", "payloads", ]; diff --git a/services/flysim/crates/fly-session/README.md b/services/flysim/crates/fly-session/README.md index 1d3b0ae..d01f1b0 100644 --- a/services/flysim/crates/fly-session/README.md +++ b/services/flysim/crates/fly-session/README.md @@ -44,6 +44,7 @@ Ready(k) ─ Prepare all agents concurrently ─────────── | `metrics` | Latency percentiles and the machine's core and memory counters | | `measure` | The execution-mode comparison of the guide's section 5 | | `cli` | The binary's subcommands: `agent`, `environment`, `measure` | +| `state` | The durable checkpoint store over `FLYSESS1`: compatibility, generations, the bounded writer | | `harness` | The runnable composition: router, the flies, one arena, one coordinator | ## Execution modes and the launcher @@ -178,6 +179,44 @@ harness.shutdown().await; event ids derived from epoch, source step, rule and ordinal. - **Executors.** The stateless identity executor only, as v1 specifies. +## Checkpoints and recovery + +The durable store is `state`, over the `FLYSESS1` layout the contract crate owns. + +- **One boundary, every participant.** `Coordinator::capture` runs at `Ready(k)` or + `Paused(k)` only. It takes its queue slot *before* the first `State.Capture`, so a saturated + writer refuses the capture rather than queueing it without bound, and the refusal is a + `BUSY` a stepping session survives rather than an epoch failure. +- **Capture and durability are two events.** `State.Capture` completes when an immutable + capture exists; `Coordinator::await_durable` completes when the store manifest rename has + happened, which is the durable commit point. Only the second moves the durable mark, and the + three ways it can end without one are told apart: `Failed` (the write stopped), + `ReplyLost` (the write finished and the acknowledgment did not arrive) and + `DeadlineExpired` (the caller's own budget ran out while the save was still going). + `Coordinator::resolve_durable` then asks the store about the *same* checkpoint instead of + saving again. +- **The writer is bounded twice**, and the two bounds refuse at different moments. The + outstanding-capture bound is taken before a capture is requested; the byte budget cannot be, + because a capture's size is not known until it exists, so it refuses at submit and releases + the payloads with the refusal. The writer owns its payload handles until the bytes are + committed or the job fails. A queued *replaceable* capture is superseded by a later one, + releasing its holds; a durable one never is. +- **The install is a group.** A restore selects a complete compatible generation, imports its + payloads as fresh artifacts, stages every participant, validates the coordinator's own + ledgers, and only then activates. A failure anywhere leaves the fence closed, and every + participant that got as far as staging is recorded as one that must be replaced before + another restore is attempted. +- **The fence lifts once.** `Failed -> Restoring(k) -> Paused(k)`, at the end of a complete + install and nowhere else. A fenced session takes no step, publishes nothing, captures + nothing and holds no artifact handle. +- **Nothing old crosses.** The fence drops every media handle; the restore imports fresh + artifacts; the environment re-renders its pending sensor pipeline from recorded + reconstruction inputs; and the new epoch's first audio chunk resumes the preserved sample + position and marks the discontinuity. +- **Epoch metadata in a trace.** `scope.epoch`, the batch id and every task event id are + derived from the epoch, so a resumed run's behaviour is compared through + `EpochRebase`, which rewrites exactly those and fails on anything it does not recognise. + ## Before you add a check to a reply Ask which kind of reply it is. Is the far side **reporting what it did**, in which case a @@ -212,9 +251,9 @@ pointing the other way -- they are what make a partial commit and an incomplete - **Fake workers.** There is no neural model and no emulator. What is modelled exactly is the ordering, the identity rules and the retry rules, not any numerical behaviour. -- **No state methods.** `State.Capture`, `State.StageRestore` and `State.ActivateRestore` are - STATE-01. The phase machine has their edges (`Capturing`, `Restoring`) and the workers do not - advertise them as implemented methods. +- **One environment, one task.** A checkpoint records the composition it was taken from, and a + restore refuses one taken under another backend, content, patch, controller or parser + identity. It does not migrate between compositions, and it does not try. - **No audience input.** The admitted pre-step stimulation list exists and is always empty. - **Pacing is coarse.** The pacing deadline rounds one step to whole nanoseconds for sleeping only; simulation time stays rational and that rounding never re-enters the accumulator. @@ -275,6 +314,9 @@ The three integration suites do not all run over both transports, and cannot: - `tests/processes.rs` runs over the Unix socket only, in all three execution modes. A participant in a process of its own has no in-memory transport to reach the router by, so the mode is the axis that suite varies and the transport is fixed. +- `tests/media.rs` and `tests/state.rs` run over both transports *and* in all three execution + modes: each acceptance body is written once and registered twice, by `both_transports!` in + the in-process composition and by `all_modes!` over the socket. - `tests/session.rs`: one world advance per complete batch; every agent Prepared before the advance; one task evaluation per transition; every agent committed before the next Prepare or @@ -291,6 +333,14 @@ The three integration suites do not all run over both transports, and cannot: allocation -- plus the sequential/reversed/parallel trace comparison across all three modes and the two process-mode section 4 rows: a router restart during a world advance, and an old worker's reply after a restart. +- `tests/state.rs`: the STATE-01 acceptance bullets -- an uninterrupted run and a resumed run + committing the same behaviour once the epoch metadata is rebased, a corrupt payload failing + the install as a group for every participant and for the coordinator's own ledger, a lost + save reply and an uncommitted store manifest both leaving the durable mark where it was, a + refused activation resuming no part of the world, the capture queue staying bounded under a + stalled writer, and old media and another parser's state failing to cross a recovery -- + plus the once-only restore token, the superseded replaceable capture, and the fence that + lifts only through a complete restore. - `tests/failures.rs`: a duplicate Prepare after a lost reply; a duplicate Commit; the same batch with altered controls; a lost Advance result; a cached artifact consumed by its first caller; one Commit failing after another succeeded; a replaced registration; a reply from diff --git a/services/flysim/crates/fly-session/src/agent.rs b/services/flysim/crates/fly-session/src/agent.rs index a52822f..22b5ac5 100644 --- a/services/flysim/crates/fly-session/src/agent.rs +++ b/services/flysim/crates/fly-session/src/agent.rs @@ -7,7 +7,7 @@ //! stimulation, then reinforces once, and executes no tick at all. Every mutating step bumps //! one counter, which is how a test proves a duplicate request changed nothing. -use std::collections::BTreeMap; +use std::collections::{BTreeMap, BTreeSet}; use serde_json::Value; @@ -193,6 +193,12 @@ pub struct AgentFaults { pub prepare_delay_ms: u64, /// Hold `Agent.Commit` open for this long. pub commit_delay_ms: u64, + /// Refuse `State.StageRestore`, so a group install meets one participant that will not + /// validate while the others already have. + pub fail_stage_restore: bool, + /// Refuse `State.ActivateRestore` after this worker has already staged, so a group meets + /// a failure halfway through activation. + pub fail_activate_restore: bool, } /// One fake agent worker's configuration. @@ -234,6 +240,10 @@ pub struct FakeAgentWorker { context: Option, context_digest: Option, prepared: Option<(DomainRequestId, PreparedDecision)>, + /// A validated replacement state that the live session cannot see yet. + staged: Option, + /// Restore tokens this worker has activated. A token activates once. + activated: BTreeSet, } impl FakeAgentWorker { @@ -248,10 +258,17 @@ impl FakeAgentWorker { context: None, context_digest: None, prepared: None, + staged: None, + activated: BTreeSet::new(), config, } } + /// True while a validated replacement state is staged and not yet activated. + pub fn has_staged_restore(&self) -> bool { + self.staged.is_some() + } + pub fn status(&self) -> StatusCell { self.status.clone() } @@ -657,7 +674,11 @@ impl WorkerEndpoint for FakeAgentWorker { } fn capabilities(&self) -> Vec { - vec![id("agent-step-v1"), id("pixel-observation-v1")] + vec![ + id("agent-step-v1"), + id("pixel-observation-v1"), + id(crate::state::CHECKPOINT_CAPABILITY), + ] } fn status_cell(&self) -> StatusCell { @@ -669,7 +690,14 @@ impl WorkerEndpoint for FakeAgentWorker { } fn methods(&self) -> Vec<&'static str> { - vec!["Agent.Initialize", "Agent.Prepare", "Agent.Commit"] + vec![ + "Agent.Initialize", + "Agent.Prepare", + "Agent.Commit", + "State.Capture", + "State.StageRestore", + "State.ActivateRestore", + ] } fn handle<'a>(&'a mut self, ctx: HandlerCtx<'a>) -> BoxFuture<'a, DomainResult> { @@ -678,6 +706,9 @@ impl WorkerEndpoint for FakeAgentWorker { "Agent.Initialize" => self.initialize(&ctx).await, "Agent.Prepare" => self.prepare(&ctx).await, "Agent.Commit" => self.commit(&ctx).await, + "State.Capture" => self.state_capture(&ctx).await, + "State.StageRestore" => self.state_stage_restore(&ctx).await, + "State.ActivateRestore" => self.state_activate_restore(&ctx).await, other => Err(DomainError::before( ErrorCode::Unsupported, format!("{other} is not an agent method"), @@ -690,7 +721,12 @@ impl WorkerEndpoint for FakeAgentWorker { /// The retention class table an agent endpoint follows, for a caller that wants it. pub fn agent_op_class(method: &str) -> Option { match method { - "Agent.Initialize" => Some(OpClass::Lifecycle), + // `ipc-v1` section 5: lifecycle *and capture* replies are retained until + // `Worker.Acknowledge`, which is also what lets a duplicate restore request replay + // its cached reply rather than staging or activating twice. + "Agent.Initialize" | "State.Capture" | "State.StageRestore" | "State.ActivateRestore" => { + Some(OpClass::Lifecycle) + } "Agent.Prepare" | "Agent.Commit" => Some(OpClass::StepMutation), _ => None, } @@ -712,3 +748,532 @@ pub fn synthetic_profile(agent_id: &Id, tick_duration: &RationalNs, warmup_ticks /// The per-agent contexts a bootstrap produced, keyed by agent id. pub type Contexts = BTreeMap; + +// ------------------------------------------------------------------------------------------- +// STATE-01: capture and restore + +/// The numerical model version this worker implements. It is part of a capture's +/// compatibility identity: the same profile and seed under another model is not the same +/// state (`workers-v1` section 2). +pub const MODEL_VERSION: &str = "fake-lcg-v1"; + +/// The plasticity rule version, for the same reason. +pub const PLASTICITY_VERSION: &str = "fake-reinforce-v1"; + +/// The version this payload layout is written and read under. +pub const AGENT_PAYLOAD_VERSION: u64 = 1; + +/// The dataset identity a synthetic agent resolves. +/// +/// There is no connectome dataset behind this worker, and a checkpoint says so with a stable +/// identity rather than omitting the field: "no dataset" has to be distinguishable from "the +/// dataset was not recorded". +pub fn dataset_digest() -> Digest { + digest_of_bytes(b"fly-session/no-dataset-v1") +} + +/// The capture compatibility digest of one agent (`workers-v1` section 2). +/// +/// The profile digest identifies the profile definition; this additionally covers the +/// resolved seed, the numerical model version and the plasticity rule, because two agents +/// with the same profile digest and different seeds hold state that is not interchangeable. +/// Every field it covers is one the checkpoint manifest already records in that agent's row, +/// so a restore derives the expected digest from the manifest rather than from the payload it +/// is about to validate. +pub fn agent_compatibility_digest( + agent_id: &Id, + profile_digest: &Digest, + dataset_digest: &Digest, + model_version: &str, + plasticity_version: &str, + seed: i32, +) -> Digest { + let value = serde_json::json!({ + "agentId": agent_id.as_str(), + "profileDigest": profile_digest.as_str(), + "datasetDigest": dataset_digest.as_str(), + "modelVersion": model_version, + "plasticityVersion": plasticity_version, + "seed": seed, + }); + digest_of(&value).expect("an agent compatibility block canonicalizes") +} + +impl FakeModel { + /// Every field of the model, so a resumed agent is this agent and not a fresh one. + fn capture(&self) -> Value { + serde_json::json!({ + "seed": self.seed, + "state": self.state.to_string(), + "mutations": self.mutations.to_string(), + "ticks": self.ticks.to_string(), + "stimulations": self.stimulations.to_string(), + "reinforcements": self.reinforcements.to_string(), + "learningEnabled": self.learning_enabled, + "learningUpdates": self.learning_updates.to_string(), + "learningChanged": self.learning_changed.to_string(), + "lastSignal": self.last_signal, + "inputValue": self.input_value.to_string(), + "inputInstalls": self.input_installs.to_string(), + }) + } + + fn restored(value: &Value) -> DomainResult { + let number = |key: &str| -> DomainResult { + value + .get(key) + .and_then(Value::as_str) + .ok_or_else(|| incompatible(format!("the agent payload has no {key}")))? + .parse::() + .map_err(|_| incompatible(format!("the agent payload's {key} is not a U64"))) + }; + let seed = value + .get("seed") + .and_then(Value::as_i64) + .and_then(|v| i32::try_from(v).ok()) + .ok_or_else(|| incompatible("the agent payload has no seed"))?; + let input_value = value + .get("inputValue") + .and_then(Value::as_str) + .ok_or_else(|| incompatible("the agent payload has no inputValue"))? + .parse::() + .map_err(|_| incompatible("the agent payload's inputValue is not an integer"))?; + let last_signal = value + .get("lastSignal") + .and_then(Value::as_f64) + .filter(|v| v.is_finite()) + .ok_or_else(|| incompatible("the agent payload's lastSignal is not finite"))?; + let learning_enabled = value + .get("learningEnabled") + .and_then(Value::as_bool) + .ok_or_else(|| incompatible("the agent payload has no learningEnabled"))?; + Ok(FakeModel { + seed, + state: number("state")?, + mutations: number("mutations")?, + ticks: number("ticks")?, + stimulations: number("stimulations")?, + reinforcements: number("reinforcements")?, + learning_enabled, + learning_updates: number("learningUpdates")?, + learning_changed: number("learningChanged")?, + last_signal, + input_value, + input_installs: number("inputInstalls")?, + }) + } +} + +fn incompatible(message: impl std::fmt::Display) -> DomainError { + DomainError::before(ErrorCode::IncompatibleState, message) +} + +/// One staged restore, held outside the live agent until it is activated. +struct StagedAgent { + token: Id, + checkpoint_id: Id, + scope: Scope, + model: FakeModel, + accumulator: TickAccumulator, + context: TypedValue, + profile: AssetRef, + committed_step: u64, +} + +impl FakeAgentWorker { + /// This worker's own compatibility identity, from its configuration and a resolved seed. + fn compatibility_digest(&self, profile: &AssetRef, seed: i32) -> Digest { + agent_compatibility_digest( + &self.config.agent_id, + &profile.digest, + &dataset_digest(), + MODEL_VERSION, + PLASTICITY_VERSION, + seed, + ) + } + + /// `State.Capture`: an immutable snapshot of this agent at its committed boundary. + /// + /// It is allowed at `Ready(k)` only. A Prepared agent holds half a transition, and there + /// is no coherent boundary to file that under. + async fn state_capture(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult { + let scope = ctx.scope()?.clone(); + self.check_epoch(&scope)?; + let AgentPhase::Ready(k) = self.phase.clone() else { + return Err(DomainError::before( + ErrorCode::InvalidPhase, + format!( + "State.Capture needs a quiescent Ready(k); this worker is {:?}", + self.phase + ), + )); + }; + if scope.step != k { + return Err(DomainError::before( + if scope.step < k { ErrorCode::StaleStep } else { ErrorCode::FutureStep }, + "State.Capture names a boundary this worker is not at", + )); + } + let params: CaptureParams = ctx.params()?; + let profile = self.profile.clone().expect("initialized"); + let context = self.context.clone().expect("initialized"); + let accumulator = self.accumulator.as_ref().expect("initialized"); + let previous = self.status.state(); + self.status.set_state(WorkerState::Capturing); + let payload = serde_json::json!({ + "payloadVersion": AGENT_PAYLOAD_VERSION, + "kind": "agent", + "agentId": self.config.agent_id.as_str(), + "checkpointId": params.checkpoint_id.as_str(), + "sourceScope": scope.to_json(), + "committedStep": k.to_string(), + "profile": profile.to_json(), + "modelVersion": MODEL_VERSION, + "plasticityVersion": PLASTICITY_VERSION, + "datasetDigest": dataset_digest().as_str(), + "model": self.model.capture(), + "accumulator": { + "tickDuration": accumulator.tick_duration().to_json(), + "remainder": accumulator.remainder().to_json(), + "executedTicks": accumulator.executed_ticks().to_string(), + "warmupOffset": accumulator.warmup_offset().to_string(), + }, + "context": context.to_json(), + }); + let bytes = canonicalize(&payload) + .map_err(|e| DomainError::invalid(format!("State.Capture: {}", e.0)))? + .into_bytes(); + let digest = digest_of_bytes(&bytes); + let artifact = crate::state::seal_payload(ctx.client, &bytes, &digest).await?; + // Capture is a read of the model, not a mutation of it: nothing above changed a + // counter, and the worker goes back to the boundary it was already at. + self.status.set_state(previous); + let result = CaptureResult { + checkpoint_id: params.checkpoint_id, + boundary: k, + compatibility_digest: self.compatibility_digest(&profile, self.model.seed()), + payload: artifact.reference().clone(), + }; + Ok(HandlerReply::with_artifacts( + object(result.to_json()), + vec![(crate::state::PAYLOAD_ATTACHMENT.to_owned(), artifact)], + )) + } + + /// `State.StageRestore`: validate a replacement state into a staging slot. + /// + /// Nothing the live session can see changes here, and the worker keeps whatever state it + /// had. It is allowed on an uninitialized replacement or a quiescent worker only; a + /// failed one is neither, which is why a group that failed is replaced rather than + /// reused. + async fn state_stage_restore(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult { + let scope = ctx.scope()?.clone(); + if scope.session_id != self.config.session_id { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "this worker belongs to another session", + )); + } + match &self.phase { + AgentPhase::Uninitialized | AgentPhase::Ready(_) => {} + other => { + return Err(DomainError::before( + ErrorCode::InvalidPhase, + format!( + "State.StageRestore needs an uninitialized replacement or a quiescent \ +worker; this worker is {other:?}" + ), + )); + } + } + if let Some(epoch) = &self.epoch + && *epoch == scope.epoch + { + return Err(DomainError::before( + ErrorCode::StaleEpoch, + "State.StageRestore proposes the epoch this worker is already running", + )); + } + let params: StageRestoreParams = ctx.params()?; + if params.source_scope.step != scope.step { + return Err(DomainError::invalid( + "State.StageRestore's scope step must be the source boundary", + )); + } + let artifact = ctx.artifact(crate::state::PAYLOAD_ATTACHMENT)?; + if artifact.reference() != ¶ms.payload { + return Err(DomainError::before( + ErrorCode::BufferInvalid, + "the staged payload attachment is not the artifact the request names", + )); + } + let bytes = artifact.read_all().await.map_err(|e| { + DomainError::before( + ErrorCode::BufferInvalid, + format!("the staged payload could not be read: {}", e.message), + ) + })?; + let declared = params + .payload + .digest + .clone() + .ok_or_else(|| incompatible("a checkpoint payload must carry a content digest"))?; + let actual = digest_of_bytes(&bytes); + if actual != declared || bytes.len() as u64 != params.payload.byte_length { + return Err(incompatible( + "the staged payload is not the content the request declares", + )); + } + let value: Value = serde_json::from_slice(&bytes) + .map_err(|e| DomainError::invalid(format!("the staged payload is not JSON: {e}")))?; + let text = |key: &str| -> DomainResult { + value + .get(key) + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| incompatible(format!("the agent payload has no {key}"))) + }; + if value.get("payloadVersion").and_then(Value::as_u64) != Some(AGENT_PAYLOAD_VERSION) { + return Err(incompatible("the agent payload is another payload version")); + } + if text("kind")? != "agent" { + return Err(incompatible("this payload is not an agent's state")); + } + if text("agentId")? != self.config.agent_id { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "the staged payload belongs to another agent", + )); + } + if text("checkpointId")? != params.checkpoint_id { + return Err(incompatible("the staged payload belongs to another checkpoint")); + } + if text("modelVersion")? != MODEL_VERSION || text("plasticityVersion")? != PLASTICITY_VERSION + { + return Err(incompatible( + "the staged payload was captured under another numerical model", + )); + } + let source_scope = Scope::from_json( + value + .get("sourceScope") + .ok_or_else(|| incompatible("the agent payload has no sourceScope"))?, + ) + .map_err(|e| incompatible(format!("the agent payload's sourceScope: {}", e.0)))?; + if source_scope != params.source_scope { + return Err(incompatible( + "the staged payload was captured at another source scope", + )); + } + let committed_step: u64 = text("committedStep")? + .parse() + .map_err(|_| incompatible("the agent payload's committedStep is not a U64"))?; + if committed_step != params.source_scope.step { + return Err(incompatible( + "the staged payload's committed step is not the source boundary", + )); + } + let profile = AssetRef::from_json( + value + .get("profile") + .ok_or_else(|| incompatible("the agent payload has no profile"))?, + ) + .map_err(|e| incompatible(format!("the agent payload's profile: {}", e.0)))?; + let model = FakeModel::restored( + value + .get("model") + .ok_or_else(|| incompatible("the agent payload has no model"))?, + )?; + // The compatibility digest is recomputed from this worker's own configuration and the + // identity the payload declares. A capture of the same profile under another seed, or + // of another agent's brain, fails here and never reaches activation. + let computed = self.compatibility_digest(&profile, model.seed()); + if computed != params.compatibility_digest { + return Err(incompatible(format!( + "the staged state's compatibility {computed} is not the {} the restore \ +requires", + params.compatibility_digest + ))); + } + let accumulator_value = value + .get("accumulator") + .ok_or_else(|| incompatible("the agent payload has no accumulator"))?; + let rational = |key: &str| -> DomainResult { + RationalNs::from_json( + accumulator_value + .get(key) + .ok_or_else(|| incompatible(format!("the accumulator has no {key}")))?, + ) + .map_err(|e| incompatible(format!("the accumulator's {key}: {}", e.0))) + }; + let counter = |key: &str| -> DomainResult { + accumulator_value + .get(key) + .and_then(Value::as_str) + .ok_or_else(|| incompatible(format!("the accumulator has no {key}")))? + .parse::() + .map_err(|_| incompatible(format!("the accumulator's {key} is not a U64"))) + }; + let tick_duration = rational("tickDuration")?; + if tick_duration != self.config.tick_duration { + return Err(incompatible( + "the staged state was captured at another model tick duration", + )); + } + let accumulator = TickAccumulator::restored( + tick_duration, + rational("remainder")?, + counter("executedTicks")?, + counter("warmupOffset")?, + ) + .map_err(incompatible)?; + let context = TypedValue::from_json( + value + .get("context") + .ok_or_else(|| incompatible("the agent payload has no context"))?, + ) + .map_err(|e| incompatible(format!("the agent payload's context: {}", e.0)))?; + FakeAgentWorker::available_actions(&context)?; + + if self.config.faults.fail_stage_restore { + // The row where a group validates three participants and the fourth does not. + // Nothing is staged here and nothing is staged anywhere else either: the + // coordinator abandons the whole install. + return Err(incompatible( + "injected staging refusal: this participant's replacement state does not \ +validate", + )); + } + // One staged restore at a time. A second proposal replaces nothing silently. + if let Some(staged) = &self.staged { + return Err(DomainError::before( + ErrorCode::Conflict, + format!( + "this worker already holds the staged restore {} for checkpoint {}", + staged.token, staged.checkpoint_id + ), + )); + } + let token = restore_token(¶ms.checkpoint_id, &scope, &actual, &self.config.incarnation_id); + if self.activated.contains(&token) { + return Err(DomainError::before( + ErrorCode::Conflict, + "this exact restore was already activated on this worker", + )); + } + self.staged = Some(StagedAgent { + token: token.clone(), + checkpoint_id: params.checkpoint_id.clone(), + scope: scope.clone(), + model, + accumulator, + context, + profile, + committed_step, + }); + self.status.set_state(WorkerState::StagedRestore); + let result = StageRestoreResult { + checkpoint_id: params.checkpoint_id, + restore_token: token, + }; + Ok(HandlerReply::from(&result)) + } + + /// `State.ActivateRestore`: install the staged state under its new scope, without a tick. + /// + /// The token activates once. A duplicate domain request replays the cached reply through + /// the shell's result cache; a fresh request naming an already activated token is a + /// conflict, which is what stops a second group from being resumed from the same bytes. + async fn state_activate_restore( + &mut self, + ctx: &HandlerCtx<'_>, + ) -> DomainResult { + let params: ActivateRestoreParams = ctx.params()?; + if self.activated.contains(¶ms.restore_token) { + return Err(DomainError::before( + ErrorCode::Conflict, + "this restore token has already been activated", + )); + } + let Some(staged) = self.staged.take() else { + return Err(DomainError::before( + ErrorCode::InvalidPhase, + "this worker holds no staged restore", + )); + }; + if staged.token != params.restore_token { + // Put it back: naming another token is not a reason to discard this one. + let token = staged.token.clone(); + self.staged = Some(staged); + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + format!("this worker's staged restore is {token}, not {}", params.restore_token), + )); + } + if self.config.faults.fail_activate_restore { + let token = staged.token.clone(); + self.staged = Some(staged); + self.status.set_state(WorkerState::Failed); + return Err(DomainError::new( + ErrorCode::BackendFailure, + format!("injected activation failure; {token} stays staged and unresumed"), + MutationCertainty::None, + )); + } + self.status.set_state(WorkerState::Restoring); + let StagedAgent { + token, + checkpoint_id, + scope, + model, + accumulator, + context, + profile, + committed_step, + } = staged; + self.model = model; + self.accumulator = Some(accumulator); + self.context_digest = Some(context.digest()); + self.context = Some(context); + self.profile = Some(profile); + self.epoch = Some(scope.epoch.clone()); + self.prepared = None; + self.phase = AgentPhase::Ready(committed_step); + self.activated.insert(token); + self.status.set_state(WorkerState::Ready); + self.status.set_scope(Some(scope_at( + &scope.session_id, + &scope.epoch, + committed_step, + ))); + self.status.advance_to(self.model.mutations()); + let result = ActivateRestoreResult { + committed_step, + checkpoint_id, + // An agent returns a null observation; the environment returns the world's. + observation: None, + }; + result + .validate_for_role(Role::Agent) + .map_err(|e| DomainError::invalid(e.0))?; + Ok(HandlerReply::from(&result)) + } +} + +/// A restore token bound to the checkpoint, the proposed scope, the payload bytes and the +/// worker incarnation staging them. +/// +/// `state-media-v1` section 5 binds a token to scope, payload and checkpoint. Binding it to +/// the incarnation as well is what keeps a token minted by a worker that has since been +/// replaced from activating anything on its replacement. +pub fn restore_token(checkpoint_id: &Id, scope: &Scope, payload_digest: &Digest, incarnation: &Id) -> Id { + let digest = digest_of_bytes( + format!( + "fly-session/restore-token-v1\n{checkpoint_id}\n{}\n{}\n{}\n{payload_digest}\n{incarnation}\n", + scope.session_id, scope.epoch, scope.step + ) + .as_bytes(), + ); + parse_id(&format!("rt-{}", &digest[..32])).expect("a hex suffix is an Id") +} diff --git a/services/flysim/crates/fly-session/src/cli.rs b/services/flysim/crates/fly-session/src/cli.rs index 1f7bc68..bbd6bbf 100644 --- a/services/flysim/crates/fly-session/src/cli.rs +++ b/services/flysim/crates/fly-session/src/cli.rs @@ -46,8 +46,10 @@ Worker options (agent and environment): agent: --agent ID --port ID --tick-numerator N --tick-denominator N --warmup-ticks N [--prepare-delay-ms N] [--commit-delay-ms N] [--fail-commit-at-step N] + [--fail-stage-restore 0|1] [--fail-activate-restore 0|1] environment: --worker ID --ports p1,p2 --step-numerator N --step-denominator N [--advance-delay-ms N] [--omit-view-at-boundary N] + [--fail-stage-restore 0|1] [--fail-activate-restore 0|1] Measure options: --steps N transitions per run (default 200) @@ -145,6 +147,17 @@ impl Options { } } + /// A flag whose value is `0` or `1`. Anything else is an error naming it, so a + /// mistyped injection is a failed launch rather than a fault that never fires. + fn flag(&self, name: &str) -> Result { + match self.0.get(name) { + None => Ok(false), + Some(value) if value == "0" => Ok(false), + Some(value) if value == "1" => Ok(true), + Some(value) => Err(format!("--{name}: {value:?} is not 0 or 1")), + } + } + fn opt_u64(&self, name: &str) -> Result, String> { match self.0.get(name) { None => Ok(None), @@ -197,6 +210,8 @@ fn serve(role: &str, options: &Options) -> Result<(), String> { fail_commit_at_step: options.opt_u64(flags::FAIL_COMMIT_AT_STEP)?, prepare_delay_ms: options.u64(flags::PREPARE_DELAY_MS, 0)?, commit_delay_ms: options.u64(flags::COMMIT_DELAY_MS, 0)?, + fail_stage_restore: options.flag(flags::FAIL_STAGE_RESTORE)?, + fail_activate_restore: options.flag(flags::FAIL_ACTIVATE_RESTORE)?, }, client_id: client_id.clone(), service: service.clone(), @@ -219,6 +234,8 @@ fn serve(role: &str, options: &Options) -> Result<(), String> { omit_audio_at_boundary: options.opt_u64(flags::OMIT_AUDIO_AT_BOUNDARY)?, overlapping_audio_at_boundary: options .opt_u64(flags::OVERLAPPING_AUDIO_AT_BOUNDARY)?, + fail_stage_restore: options.flag(flags::FAIL_STAGE_RESTORE)?, + fail_activate_restore: options.flag(flags::FAIL_ACTIVATE_RESTORE)?, }, client_id: client_id.clone(), service: service.clone(), diff --git a/services/flysim/crates/fly-session/src/clock.rs b/services/flysim/crates/fly-session/src/clock.rs index 080c12a..aa50ff2 100644 --- a/services/flysim/crates/fly-session/src/clock.rs +++ b/services/flysim/crates/fly-session/src/clock.rs @@ -29,6 +29,30 @@ impl TickAccumulator { }) } + /// The exact accumulator a capture recorded. + /// + /// The remainder is restored, never rounded or reset: a resumed agent that started its + /// first interval from zero would drift away from the run it is supposed to continue. + pub fn restored( + tick_duration: RationalNs, + remainder: RationalNs, + executed_ticks: u64, + warmup_offset: u64, + ) -> Result { + let mut accumulator = TickAccumulator::new(tick_duration)?; + remainder.validate().map_err(|e| e.0)?; + if remainder >= tick_duration { + return Err("a captured remainder is not below one model tick".to_owned()); + } + if warmup_offset > executed_ticks { + return Err("a captured warm-up offset exceeds the executed tick count".to_owned()); + } + accumulator.remainder = remainder; + accumulator.executed_ticks = executed_ticks; + accumulator.warmup_offset = warmup_offset; + Ok(accumulator) + } + pub fn tick_duration(&self) -> RationalNs { self.tick_duration } diff --git a/services/flysim/crates/fly-session/src/coordinator.rs b/services/flysim/crates/fly-session/src/coordinator.rs index f9d74c5..78f7858 100644 --- a/services/flysim/crates/fly-session/src/coordinator.rs +++ b/services/flysim/crates/fly-session/src/coordinator.rs @@ -141,6 +141,16 @@ pub struct Deadlines { pub resolve_attempts: u32, /// A separate, larger budget for `Worker.Hello` and the `Initialize` methods. pub boot: Duration, + /// A separate budget for the `State.*` methods, which `ipc-v1` section 6 gives one: + /// a capture serializes a participant and a restore validates and installs one, and + /// neither is a step whose latency the probe was chosen for. + pub capture: Duration, + /// How long a caller waits for a *durable* acknowledgment. + /// + /// It is not [`Deadlines::capture`]: that one bounds a call to a participant, and this + /// one bounds two `fsync`s, a queue the caller shares with other captures and a disk. + /// Reusing the call budget here would make a slow disk look like an unresponsive worker. + pub durable: Duration, } /// How long the resolution waits between attempts. @@ -167,6 +177,8 @@ impl Default for Deadlines { // Over sixteen seconds of pauses: the budget above is what terminates. resolve_attempts: 8192, boot: Duration::from_secs(30), + capture: Duration::from_secs(30), + durable: Duration::from_secs(60), } } } @@ -177,6 +189,8 @@ pub struct Topics { pub descriptor: String, pub snapshots: String, pub events: String, + /// Where the distinct captured/queued/committed/failed/superseded checkpoint events go. + pub checkpoints: String, } impl Topics { @@ -185,6 +199,7 @@ impl Topics { descriptor: format!("session.{session_id}.descriptor"), snapshots: format!("session.{session_id}.snapshots"), events: format!("session.{session_id}.events"), + checkpoints: format!("session.{session_id}.checkpoints"), } } } @@ -202,6 +217,11 @@ pub struct AgentSlot { pub tick_duration: RationalNs, pub warmup_ticks: u64, pub committed_step: u64, + /// The tick count and remainder this agent last reported, which are what the checkpoint + /// manifest records for it. They are metadata about the payload, never a substitute for + /// it: the agent's own capture is the state that is restored. + pub brain_ticks: u64, + pub remainder: RationalNs, context: TypedValue, context_digest: Digest, prepared: Option, @@ -226,6 +246,8 @@ impl AgentSlot { tick_duration: RationalNs::ZERO, warmup_ticks: 0, committed_step: 0, + brain_ticks: 0, + remainder: RationalNs::ZERO, context: TypedValue::new(crate::task::context_schema(), Value::Object(Map::new())) .expect("an empty context object is a valid typed value"), context_digest: digest_of_bytes(b""), @@ -302,6 +324,12 @@ pub struct Coordinator { /// Set when the epoch failed: every old handle, route and reply is invalid from here on /// and only a coherent restore may lift it. fenced: bool, + /// The durable store's bounded writer, when this composition has one. + writer: Option, + /// The last checkpoint whose saved acknowledgment arrived, and its boundary. + durable: Option<(Id, u64)>, + /// Participants that installed a restore a group install then abandoned. + tainted: BTreeSet, started: std::time::Instant, last_advance_request: Option, last_commit_requests: Vec, @@ -364,6 +392,9 @@ impl Coordinator { metrics: Metrics::default(), blame: None, fenced: false, + writer: None, + durable: None, + tainted: BTreeSet::new(), started: std::time::Instant::now(), last_advance_request: None, last_commit_requests: Vec::new(), @@ -492,8 +523,16 @@ impl Coordinator { let (from, to) = self.phases.fail(); self.trace.phase(from, to); self.fenced = true; + // Whatever replies this session still owed an acknowledgment for belong to + // participants of an invalid epoch. Carrying them across a recovery would send + // `Worker.Acknowledge` to a registration that is gone. + self.lifecycle_acks.clear(); + // Every handle this session held on the old epoch's media goes with the fence: a + // recovery imports fresh artifacts and never expects one of these back. self.views.clear(); self.pending_views.clear(); + self.audio.clear(); + self.pending_audio.clear(); match &participant { Some(who) => self.audit.push(format!("fail:{detail}:{who}")), None => self.audit.push(format!("fail:{detail}")), @@ -634,6 +673,10 @@ impl Coordinator { (self.topics.descriptor.clone(), flybus::Retained::Latest), (self.topics.snapshots.clone(), flybus::Retained::Latest), (self.topics.events.clone(), flybus::Retained::None), + // Checkpoint events are a stream of distinct facts, not a latest value: a + // "committed" that replaced a "queued" would erase the distinction the durable + // commit rules are built on. + (self.topics.checkpoints.clone(), flybus::Retained::None), ] { self.bus.declare_topic(&name, retained).await.map_err(|e| { let error = DomainError::new( @@ -719,6 +762,15 @@ impl Coordinator { if let Err(e) = media::check_required_views(&result.descriptor, &result.observation) { return Err(self.fail_now(e, "observation-0")); } + // O[0] ran no transition either, so it carries no chunk, and that is checked rather + // than assumed from its boundary number. + if let Err(e) = media::check_required_audio( + &result.descriptor, + &result.observation, + media::ObservationOrigin::Installed, + ) { + return Err(self.fail_now(e, "observation-0")); + } self.descriptor = Some(result.descriptor); self.observation = Some(result.observation); self.lifecycle_acks.push((worker, reply.request_id.clone())); @@ -818,6 +870,10 @@ impl Coordinator { self.agents[index].tick_duration = result.tick_duration; self.agents[index].warmup_ticks = result.warmup_ticks; self.agents[index].committed_step = 0; + // At boundary 0 the agent has executed exactly its warm-up, with no interval + // consumed, so the remainder is zero. + self.agents[index].brain_ticks = result.telemetry.brain_ticks; + self.agents[index].remainder = RationalNs::ZERO; self.lifecycle_acks.push((slot_worker, reply.request_id.clone())); let agent_id = self.agents[index].agent_id.clone(); self.audit.push(format!("agent.initialize:{agent_id}")); @@ -1034,6 +1090,8 @@ impl Coordinator { self.blame(Some(worker.worker_id.clone())); let deadline = if method.ends_with("Initialize") || method == "Worker.Hello" { self.deadlines.boot + } else if method.starts_with("State.") { + self.deadlines.capture } else { self.deadlines.probe }; @@ -1716,6 +1774,12 @@ impl Coordinator { method, )); } + if let Some(slot) = self.agents.iter_mut().find(|slot| slot.agent_id == agent_id) { + // What the agent will be at once this transition commits: Prepare is the only + // phase that advances the accumulator. + slot.brain_ticks = decision.brain_ticks; + slot.remainder = decision.remainder; + } self.audit.push(format!("prepared:{agent_id}@{k}")); self.stats.prepares += 1; self.blame(None); @@ -2134,7 +2198,11 @@ impl Coordinator { } // Audio has no sensory role here, but its chunks still cannot overlap or go backwards // inside an epoch, and a stale one must not reach presentation as current. - if let Err(e) = media::check_required_audio(descriptor, &result.observation) { + if let Err(e) = media::check_required_audio( + descriptor, + &result.observation, + media::ObservationOrigin::Transition, + ) { return Err(self.fail_now(e, "step-result")); } if let Err(e) = self.timelines.accept(descriptor, &result.observation) { @@ -2629,3 +2697,1432 @@ impl Coordinator { Ok(()) } } + +// ------------------------------------------------------------------------------------------- +// STATE-01: coherent all-participant capture and recovery + +/// A capture that exists and has been queued, whose durable outcome has not arrived yet. +/// +/// `State.Capture` completes when an immutable capture exists, not when a backend save was +/// requested, and only durable completion produces a saved acknowledgment. Those are two +/// events, so they are two calls. +#[derive(Debug)] +pub struct CaptureTicket { + pub checkpoint_id: Id, + pub boundary: u64, + receiver: tokio::sync::oneshot::Receiver, +} + +/// What one completed group restore installed. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct RestoreReport { + pub checkpoint_id: Id, + pub boundary: u64, + pub epoch: Id, + /// Every participant that staged, in the order they were asked. + pub staged: Vec, + /// Every participant that activated, in the order they were asked. + pub activated: Vec, + /// The once-only token each participant staged under, so a caller can prove a token + /// activates once rather than being told so. + pub tokens: Vec<(Id, Id)>, + /// The artifacts the payloads were imported as. None of them existed before this restore. + pub imported: Vec, +} + +/// The payload name the coordinator's own session record travels under. +impl Coordinator { + /// Attaches the durable store's bounded writer. Without one this session captures nothing + /// and says so, rather than pretending to. + pub fn attach_store(&mut self, writer: crate::state::CheckpointWriter) { + self.writer = Some(writer); + } + + pub fn writer(&self) -> Option<&crate::state::CheckpointWriter> { + self.writer.as_ref() + } + + /// Stops the checkpoint writer and waits for its task. + pub async fn shutdown_store(&mut self) { + if let Some(writer) = self.writer.take() { + writer.shutdown().await; + } + } + + /// The last checkpoint whose *saved acknowledgment* this coordinator received, and its + /// boundary. It moves on durable completion and on nothing else. + pub fn durable(&self) -> Option<(Id, u64)> { + self.durable.clone() + } + + /// Participants that installed a restore in a group install that then failed. + /// + /// They hold state no group ever resumed. A further restore is refused until each one has + /// been replaced, which is how "a failure during activation cannot resume half a world" + /// survives the next attempt as well as this one. + pub fn tainted(&self) -> Vec { + self.tainted.iter().cloned().collect() + } + + /// The live composition's compatibility identities. + pub fn compatibility(&self) -> Outcome { + match &self.descriptor { + Some(descriptor) => Ok(crate::state::Compatibility::of(descriptor)), + None => Err(SessionFailure { + error: DomainError::before( + ErrorCode::InvalidPhase, + "the session has no environment descriptor to take compatibility from", + ), + phase: self.phases.phase().label(), + detail: "compatibility".to_owned(), + participant: None, + }), + } + } + + fn agent_compatibility(&self, slot: &AgentSlot) -> Digest { + crate::agent::agent_compatibility_digest( + &slot.agent_id, + &slot.profile.digest, + &crate::agent::dataset_digest(), + crate::agent::MODEL_VERSION, + crate::agent::PLASTICITY_VERSION, + slot.seed, + ) + } + + /// The coordinator's own session record: what it must hold again to resume this boundary. + /// + /// `state-media-v1` section 4 puts the next decision state, the admission state and the + /// event watermarks in the coordinator's own payloads. This is that payload: the + /// composition has no audience input, so the admission record says so explicitly rather + /// than being absent. + fn coordinator_record(&self, boundary: u64) -> Value { + let (last_source_step, issued) = self.task.event_watermarks(); + json!({ + "payloadVersion": 1, + "kind": "coordinator", + "committedStep": boundary.to_string(), + "admission": { + "audienceInput": "none-configured", + "admitted": Value::Array(Vec::new()), + }, + "eventWatermarks": { + "lastSourceStep": last_source_step.to_string(), + "issued": issued.to_string(), + }, + "audioPositions": Value::Object( + self.timelines + .positions() + .into_iter() + .map(|(stream, sample)| (stream, Value::String(sample.to_string()))) + .collect(), + ), + "agents": Value::Array( + self.agents + .iter() + .map(|slot| json!({ + "agentId": slot.agent_id.as_str(), + "portId": slot.port_id.as_str(), + "profile": slot.profile.to_json(), + "seed": slot.seed, + "workerThreads": slot.worker_threads.to_string(), + "committedStep": slot.committed_step.to_string(), + "brainTicks": slot.brain_ticks.to_string(), + "remainder": slot.remainder.to_json(), + "context": slot.context.to_json(), + })) + .collect(), + ), + }) + } + + /// Seals one coordinator-owned payload as an immutable artifact. + async fn seal_own(&mut self, name: &str, value: &Value) -> Outcome { + let bytes = match canonicalize(value) { + Ok(text) => text.into_bytes(), + Err(e) => { + return Err(self.fail_now( + DomainError::invalid(format!("checkpoint payload {name}: {}", e.0)), + "capture", + )); + } + }; + let digest = digest_of_bytes(&bytes); + let artifact = match crate::state::seal_payload(&self.bus, &bytes, &digest).await { + Ok(artifact) => artifact, + Err(e) => return Err(self.fail_now(e, "capture")), + }; + Ok(crate::state::CapturedPayload { + name: name.to_owned(), + byte_length: bytes.len() as u64, + digest, + artifact, + }) + } + + /// Takes one coherent all-participant capture at the committed boundary and queues it. + /// + /// The queue slot is taken *first*: a saturated writer refuses before a single + /// `State.Capture` is sent, which is the only way "reject or defer before capture" can be + /// true rather than aspirational. + pub async fn capture(&mut self, checkpoint_id: &Id, replaceable: bool) -> Outcome { + if self.fenced { + return Err(SessionFailure { + error: DomainError::before( + ErrorCode::InvalidPhase, + "the epoch is fenced; a fenced session captures nothing", + ), + phase: self.phases.phase().label(), + detail: "fenced".to_owned(), + participant: None, + }); + } + // A capture that arrives while a transition is in flight is a race, not a bug in the + // transaction: the supervisor asking for one does not know where the session is. It + // is refused by name and the epoch is untouched, unlike a phase edge the machine + // itself takes. + let Some(boundary) = self.phases.phase().committed_boundary() else { + self.audit.push(format!("capture-refused:{checkpoint_id}")); + return Err(SessionFailure { + error: DomainError::before( + ErrorCode::InvalidPhase, + "a coherent checkpoint is taken at a committed boundary only", + ), + phase: self.phases.phase().label(), + detail: "capture".to_owned(), + participant: None, + }); + }; + let origin = self.phases.phase(); + if self.writer.is_none() { + return Err(SessionFailure { + error: DomainError::before( + ErrorCode::Unsupported, + "this session has no checkpoint store attached", + ), + phase: self.phases.phase().label(), + detail: "capture".to_owned(), + participant: None, + }); + } + // Before anything is captured. + let reservation = match self.writer.as_ref().expect("checked").reserve() { + Ok(reservation) => reservation, + Err(e) => { + // A refused capture is not a session failure: the boundary stands, the + // session keeps stepping and the caller is told the queue is full. + self.audit.push(format!("capture-refused:{checkpoint_id}")); + return Err(SessionFailure { + error: e, + phase: self.phases.phase().label(), + detail: "capture".to_owned(), + participant: None, + }); + } + }; + self.transition(Phase::Capturing(boundary))?; + let result = self + .capture_group(checkpoint_id, boundary, replaceable, reservation) + .await; + match result { + Ok(ticket) => { + self.transition(origin)?; + self.audit.push(format!("captured:{checkpoint_id}@{boundary}")); + Ok(ticket) + } + Err(e) => Err(e), + } + } + + async fn capture_group( + &mut self, + checkpoint_id: &Id, + boundary: u64, + replaceable: bool, + reservation: crate::state::Reservation, + ) -> Outcome { + let scope = self.scope(boundary); + let compatibility = self.compatibility()?; + let params = object(CaptureParams { checkpoint_id: checkpoint_id.clone() }.to_json()); + let want = vec![crate::state::PAYLOAD_ATTACHMENT.to_owned()]; + let mut payloads = Vec::new(); + let mut acknowledge: Vec<(WorkerRef, DomainRequestId)> = Vec::new(); + + // The world first, then the agents in sorted order: one boundary, every participant. + let environment = self.environment.clone(); + let reply = self + .call( + &environment, + "State.Capture", + Some(scope.clone()), + params.clone(), + &[], + &want, + ) + .await?; + let world: CaptureResult = reply.parse().map_err(|e| self.fail_now(e, "capture"))?; + let world_payload = self.accept_capture( + &reply, + &world, + checkpoint_id, + boundary, + &compatibility.digest(), + crate::state::WORLD_PAYLOAD, + )?; + payloads.push(world_payload); + acknowledge.push((environment, reply.request_id.clone())); + + let mut agent_rows = Vec::new(); + for index in 0..self.agents.len() { + let slot_worker = self.agents[index].worker.clone(); + let agent_id = self.agents[index].agent_id.clone(); + let expected = self.agent_compatibility(&self.agents[index]); + let reply = self + .call( + &slot_worker, + "State.Capture", + Some(scope.clone()), + params.clone(), + &[], + &want, + ) + .await?; + let captured: CaptureResult = reply.parse().map_err(|e| self.fail_now(e, "capture"))?; + let name = crate::state::agent_payload(&agent_id); + let payload = + self.accept_capture(&reply, &captured, checkpoint_id, boundary, &expected, &name)?; + payloads.push(payload); + acknowledge.push((slot_worker, reply.request_id.clone())); + let slot = &self.agents[index]; + agent_rows.push(crate::state::AgentEntry { + agent_id: agent_id.clone(), + profile_digest: slot.profile.digest.clone(), + dataset_digest: crate::agent::dataset_digest(), + model_version: crate::agent::MODEL_VERSION.to_owned(), + plasticity_version: crate::agent::PLASTICITY_VERSION.to_owned(), + seed: slot.seed, + brain_ticks: slot.brain_ticks, + remainder: slot.remainder, + payload: name, + }); + } + + // The coordinator's own ledgers, sealed the same way so the writer treats every + // payload alike. + let ledger = self.task.capture().map_err(|e| self.fail_now(e, "capture"))?; + payloads.push( + self.seal_own(crate::state::TASK_LEDGER_PAYLOAD, &ledger.to_json()) + .await?, + ); + let inspection = self + .observation + .as_ref() + .map(|observation| observation.inspection.to_json()) + .ok_or_else(|| { + DomainError::before(ErrorCode::InvalidPhase, "the session never bootstrapped") + }); + let inspection = match inspection { + Ok(value) => value, + Err(e) => return Err(self.fail_now(e, "capture")), + }; + payloads.push( + self.seal_own(crate::state::PRIOR_INSPECTION_PAYLOAD, &inspection) + .await?, + ); + let mut executor_rows = Vec::new(); + for agent_id in self.agent_ids() { + let state = { + let executor = self.executors.get(&agent_id).ok_or_else(|| { + DomainError::before( + ErrorCode::IdentityMismatch, + format!("{agent_id} has no configured action executor"), + ) + }); + match executor { + Ok(executor) => executor.capture().map_err(|e| (e, agent_id.clone())), + Err(e) => Err((e, agent_id.clone())), + } + }; + let state = match state { + Ok(state) => state, + Err((e, who)) => { + self.blame(Some(who)); + return Err(self.fail_now(e, "capture")); + } + }; + let name = crate::state::executor_payload(&agent_id); + payloads.push(self.seal_own(&name, &state.to_json()).await?); + executor_rows.push((agent_id, name)); + } + let record = self.coordinator_record(boundary); + payloads.push( + self.seal_own(crate::state::ADMISSION_PAYLOAD, &record) + .await?, + ); + + let (last_source_step, issued) = self.task.event_watermarks(); + let world_time = match self.observation.as_ref().map(|o| o.world_time) { + Some(world_time) => Ok(world_time), + None => Err(self.fail_now( + DomainError::before( + ErrorCode::InvalidPhase, + "the session has no observation, so it is at no world time to record", + ), + "capture", + )), + }; + let manifest = crate::state::CheckpointManifest { + checkpoint_id: checkpoint_id.clone(), + source_scope: scope.clone(), + episode_id: self.episode_id.clone(), + world_time: world_time?, + scheduler_id: "lockstep-v1".to_owned(), + composition_digest: self.composition_digest(), + port_map: self + .agents + .iter() + .map(|slot| (slot.port_id.clone(), slot.agent_id.clone())) + .collect(), + compatibility: compatibility.clone(), + agents: agent_rows, + coordinator: crate::state::CoordinatorEntry { + task_ledger: crate::state::TASK_LEDGER_PAYLOAD.to_owned(), + prior_inspection: crate::state::PRIOR_INSPECTION_PAYLOAD.to_owned(), + executor_state: executor_rows, + admission_state: crate::state::ADMISSION_PAYLOAD.to_owned(), + event_watermarks: crate::state::EventWatermarks { + last_source_step, + issued, + }, + }, + environment: crate::state::EnvironmentEntry { + worker_id: self.environment.worker_id.clone(), + payload: crate::state::WORLD_PAYLOAD.to_owned(), + }, + // No external helper takes part in this composition, and the manifest says so + // rather than leaving the field out. + helper_state: Vec::new(), + payloads: payloads + .iter() + .map(|p| (p.name.clone(), p.byte_length, p.digest.clone())) + .collect(), + }; + + let submission = crate::state::CaptureSubmission { + checkpoint_id: checkpoint_id.clone(), + boundary, + session_id: self.session_id.clone(), + epoch: self.epoch.clone(), + episode_id: self.episode_id.clone(), + compatibility_digest: compatibility.digest(), + manifest: manifest.to_json(), + payloads, + replaceable, + }; + // The writer takes its own ownership of every payload here. Only then are the + // workers' cached capture replies released. + let receiver = { + let writer = self.writer.as_ref().expect("checked"); + match writer.submit(reservation, submission).await { + Ok(receiver) => receiver, + Err(e) => return Err(self.fail_now(e, "capture")), + } + }; + self.publish_checkpoint_event("captured", checkpoint_id, boundary, None).await?; + for (worker, request_id) in acknowledge { + self.lifecycle_acks.push((worker, request_id)); + } + self.acknowledge_lifecycle().await?; + Ok(CaptureTicket { + checkpoint_id: checkpoint_id.clone(), + boundary, + receiver, + }) + } + + /// Checks one participant's capture and turns it into a payload the writer can own. + fn accept_capture( + &mut self, + reply: &DomainReply, + result: &CaptureResult, + checkpoint_id: &Id, + boundary: u64, + expected_compatibility: &Digest, + name: &str, + ) -> Outcome { + if result.checkpoint_id != *checkpoint_id { + return Err(self.fail_now( + DomainError::before( + ErrorCode::IdentityMismatch, + "a capture names another checkpoint", + ), + "capture", + )); + } + if result.boundary != boundary { + return Err(self.fail_now( + DomainError::before( + ErrorCode::IdentityMismatch, + "a capture names another boundary; one checkpoint is one boundary", + ), + "capture", + )); + } + if result.compatibility_digest != *expected_compatibility { + return Err(self.fail_now( + DomainError::before( + ErrorCode::IncompatibleState, + "a capture reports a compatibility identity the composition does not hold", + ), + "capture", + )); + } + let digest = match &result.payload.digest { + Some(digest) => digest.clone(), + None => { + return Err(self.fail_now( + DomainError::before( + ErrorCode::BufferInvalid, + "a checkpoint payload must carry a content digest", + ), + "capture", + )); + } + }; + let artifact = match reply.artifacts.get(crate::state::PAYLOAD_ATTACHMENT) { + Some(artifact) if artifact.reference() == &result.payload => artifact.clone(), + _ => { + return Err(self.fail_now( + DomainError::before( + ErrorCode::BufferInvalid, + "a capture arrived without a live owned handle on its payload", + ), + "capture", + )); + } + }; + Ok(crate::state::CapturedPayload { + name: name.to_owned(), + byte_length: result.payload.byte_length, + digest, + artifact, + }) + } + + /// Waits for one capture's durable outcome and moves the durable mark only on a commit. + /// + /// A lost reply is an outcome here, not a hang and not a save: the high-water mark stays + /// where it was until [`Coordinator::resolve_durable`] asks the store about the same + /// operation. + pub async fn await_durable(&mut self, ticket: CaptureTicket) -> Outcome { + let CaptureTicket { checkpoint_id, boundary, receiver } = ticket; + let budget = self.deadlines.durable; + let outcome = + crate::state::CheckpointWriter::wait(receiver, &checkpoint_id, budget).await; + if outcome.is_durable() { + self.durable = Some((checkpoint_id.clone(), boundary)); + self.audit.push(format!("durable:{checkpoint_id}@{boundary}")); + } else { + // Named rather than lumped together: a failed write, a superseded capture, a lost + // reply and an expired caller budget are four different things to have to explain. + self.audit + .push(format!("not-durable:{}:{checkpoint_id}@{boundary}", outcome.event())); + } + Ok(outcome) + } + + /// Resolves a save whose reply was lost, by asking the store's durable metadata about the + /// *same* checkpoint. It never saves again. + /// + /// `Some(boundary)` means the store manifest lists that generation, which is the durable + /// commit point; `None` means it does not, and an unreferenced generation file stays + /// unreferenced. + pub async fn resolve_durable(&mut self, checkpoint_id: &Id) -> Outcome> { + let Some(writer) = self.writer.as_ref() else { + return Err(self.fail_now( + DomainError::before( + ErrorCode::Unsupported, + "this session has no checkpoint store attached", + ), + "resolve-durable", + )); + }; + let wanted = checkpoint_id.clone(); + let found = writer + .with_store(move |store| store.lookup(&wanted).map(|g| g.boundary)) + .await; + match found { + Some(boundary) => { + self.durable = Some((checkpoint_id.clone(), boundary)); + self.audit.push(format!("durable:{checkpoint_id}@{boundary}")); + Ok(Some(boundary)) + } + None => { + self.audit.push(format!("not-durable:{checkpoint_id}")); + Ok(None) + } + } + } + + /// Captures and waits for the durable outcome, which is what an ordinary caller wants. + pub async fn checkpoint(&mut self, checkpoint_id: &Id) -> Outcome { + let ticket = self.capture(checkpoint_id, false).await?; + self.await_durable(ticket).await + } + + async fn publish_checkpoint_event( + &mut self, + event: &str, + checkpoint_id: &Id, + boundary: u64, + detail: Option<&str>, + ) -> Outcome<()> { + let payload = json!({ + "event": event, + "sessionId": self.session_id.as_str(), + "epoch": self.epoch.as_str(), + "checkpointId": checkpoint_id.as_str(), + "boundary": boundary.to_string(), + "detail": detail.map_or(Value::Null, |d| Value::String(d.to_owned())), + }); + let topic = self.topics.checkpoints.clone(); + self.publish(&topic, object(payload), Vec::new()).await + } + + // --------------------------------------------------------------------------------------- + // Recovery + + /// Points the coordinator at a replacement participant while the epoch is fenced. + /// + /// A replacement is the only way a fenced participant comes back: `step-v1` section 7's + /// incarnation row says every live participant of a failed epoch belongs to an invalid + /// one, so the reference is exchanged deliberately here and never repaired in place. + pub fn replace_participant(&mut self, worker_id: &Id, worker: WorkerRef) -> Outcome<()> { + if !self.fenced { + return Err(self.fail_now( + DomainError::before( + ErrorCode::InvalidPhase, + "participants are replaced while the epoch is fenced, not during play", + ), + "replace", + )); + } + if worker.worker_id != *worker_id { + return Err(self.fail_now( + DomainError::before( + ErrorCode::IdentityMismatch, + "the replacement reference names another worker", + ), + "replace", + )); + } + if self.environment.worker_id == *worker_id { + self.environment = worker; + self.tainted.remove(worker_id); + self.audit.push(format!("replaced:{worker_id}")); + return Ok(()); + } + match self.agents.iter_mut().find(|slot| slot.agent_id == *worker_id) { + Some(slot) => { + slot.worker = worker; + self.tainted.remove(worker_id); + self.audit.push(format!("replaced:{worker_id}")); + Ok(()) + } + None => Err(self.fail_now( + DomainError::before( + ErrorCode::IdentityMismatch, + format!("{worker_id} is not a participant of this composition"), + ), + "replace", + )), + } + } + + /// Installs a coherent checkpoint into a fresh epoch and lifts the fence. + /// + /// The whole of `state-media-v1` section 6 in order: select a complete compatible durable + /// checkpoint, import its payloads as *new* artifacts, stage every participant, activate + /// every participant, install the coordinator's own staged state, verify identity and + /// boundary, flush the old media and parser state, and establish `Paused(k)`. A failure + /// at any point leaves the fence exactly where it was. + pub async fn restore( + &mut self, + checkpoint_id: Option<&Id>, + new_epoch: &Id, + ) -> Outcome { + if self.phases.phase() != Phase::Failed { + return Err(self.fail_now( + DomainError::before( + ErrorCode::InvalidPhase, + "a coherent restore starts from a failed epoch", + ), + "restore", + )); + } + if *new_epoch == self.epoch { + return Err(self.fail_now( + DomainError::before( + ErrorCode::StaleEpoch, + "a restore installs a fresh epoch, never the one that failed", + ), + "restore", + )); + } + if !self.tainted.is_empty() { + let who: Vec<&str> = self.tainted.iter().map(String::as_str).collect(); + return Err(self.fail_now( + DomainError::before( + ErrorCode::InvalidPhase, + format!( + "{} installed a restore that no group resumed and must be replaced first", + who.join(", ") + ), + ), + "restore", + )); + } + let Some(writer) = self.writer.as_ref() else { + return Err(self.fail_now( + DomainError::before( + ErrorCode::Unsupported, + "this session has no checkpoint store attached", + ), + "restore", + )); + }; + let wanted = checkpoint_id.cloned(); + let read = writer + .with_store(move |store| { + let record = store.select(wanted.as_ref())?; + let envelope = store.read(&record)?; + Ok::<_, DomainError>((record, envelope)) + }) + .await; + let (record, envelope) = match read { + Ok(pair) => pair, + Err(e) => return Err(self.fail_now(e, "restore")), + }; + let manifest = match crate::state::CheckpointManifest::from_json(&envelope.manifest) { + Ok(manifest) => manifest, + Err(e) => { + return Err(self.fail_now( + DomainError::before(ErrorCode::IncompatibleState, e), + "restore", + )); + } + }; + if let Err(e) = self.check_restore_identity(&manifest, &record) { + return Err(self.fail_now(e, "restore")); + } + let boundary = manifest.source_scope.step; + self.transition(Phase::Restoring(boundary))?; + let outcome = self.restore_group(&envelope, &manifest, new_epoch, boundary).await; + match outcome { + Ok(report) => Ok(report), + Err(e) => Err(e), + } + } + + /// Marks every participant of an abandoned group install as one that must be replaced. + /// + /// A participant that staged holds a replacement state nothing installed; one that + /// activated holds installed state no group resumed. Neither is a participant this + /// session may reuse, and `ipc-v1` section 6's last paragraph is explicit that v1 does + /// not silently reattach one. + fn taint_group(&mut self, staged: &[(Id, WorkerRef, Id)]) { + for (who, _, _) in staged { + self.tainted.insert(who.clone()); + } + } + + /// Every identity a restore checks before a single participant is asked to stage. + fn check_restore_identity( + &self, + manifest: &crate::state::CheckpointManifest, + record: &crate::state::GenerationRecord, + ) -> DomainResult<()> { + if manifest.source_scope.session_id != self.session_id { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "the checkpoint belongs to another session", + )); + } + if manifest.episode_id != self.episode_id { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "the checkpoint belongs to another episode", + )); + } + if manifest.scheduler_id != "lockstep-v1" { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + format!( + "the checkpoint was scheduled by {}, not lockstep-v1", + manifest.scheduler_id + ), + )); + } + if record.compatibility_digest != manifest.compatibility.digest() { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the store manifest and the envelope disagree about compatibility", + )); + } + let live = match &self.descriptor { + Some(descriptor) => crate::state::Compatibility::of(descriptor), + None => { + return Err(DomainError::before( + ErrorCode::InvalidPhase, + "the session has no environment descriptor to compare compatibility with", + )); + } + }; + // Names the identity that differs -- backend, content, patch, controller, parser or + // state format -- rather than one opaque digest mismatch. + manifest + .compatibility + .compare(&live) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e))?; + let live_ports: Vec<(Id, Id)> = self + .agents + .iter() + .map(|slot| (slot.port_id.clone(), slot.agent_id.clone())) + .collect(); + if manifest.port_map != live_ports { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "the checkpoint's port map is not this composition's", + )); + } + if manifest.environment.worker_id != self.environment.worker_id { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "the checkpoint's world is not this composition's environment", + )); + } + let recorded: Vec = manifest.agents.iter().map(|a| a.agent_id.clone()).collect(); + if recorded != self.agent_ids() { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "the checkpoint's agents are not this composition's", + )); + } + for (row, slot) in manifest.agents.iter().zip(&self.agents) { + if row.profile_digest != slot.profile.digest || row.seed != slot.seed { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + format!( + "{}'s checkpoint was taken under another profile or seed", + row.agent_id + ), + )); + } + } + Ok(()) + } + + #[allow(clippy::too_many_lines)] + async fn restore_group( + &mut self, + envelope: &fly_session_types::checkpoint::Envelope, + manifest: &crate::state::CheckpointManifest, + new_epoch: &Id, + boundary: u64, + ) -> Outcome { + let scope = scope_at(&self.session_id, new_epoch, boundary); + // Step 3: import the payloads as *new* artifacts. Nothing the fence dropped is asked + // to come back, and a router that restarted has none of the old roots anyway. + let mut imported: BTreeMap = BTreeMap::new(); + for (name, bytes) in &envelope.payloads { + let digest = digest_of_bytes(bytes); + let artifact = match crate::state::seal_payload(&self.bus, bytes, &digest).await { + Ok(artifact) => artifact, + Err(e) => return Err(self.fail_now(e, "restore")), + }; + imported.insert(name.clone(), (artifact, digest, bytes.len() as u64)); + } + let imported_ids: Vec = imported + .values() + .map(|(artifact, _, _)| artifact.reference().artifact_id.clone()) + .collect(); + + // Step 3, continued: stage every participant. A refusal anywhere leaves nothing + // staged that will ever be activated, because the whole install is abandoned. + let mut staged: Vec<(Id, WorkerRef, Id)> = Vec::new(); + let mut order: Vec<(Id, WorkerRef, String, Digest)> = Vec::new(); + order.push(( + self.environment.worker_id.clone(), + self.environment.clone(), + manifest.environment.payload.clone(), + manifest.compatibility.digest(), + )); + for index in 0..self.agents.len() { + let slot = &self.agents[index]; + let row = manifest + .agents + .iter() + .find(|row| row.agent_id == slot.agent_id) + .expect("the agent set was checked"); + order.push(( + slot.agent_id.clone(), + slot.worker.clone(), + row.payload.clone(), + crate::agent::agent_compatibility_digest( + &row.agent_id, + &row.profile_digest, + &row.dataset_digest, + &row.model_version, + &row.plasticity_version, + row.seed, + ), + )); + } + for (who, worker, payload_name, compatibility_digest) in &order { + let Some((artifact, _, _)) = imported.get(payload_name) else { + return Err(self.fail_now( + DomainError::before( + ErrorCode::IncompatibleState, + format!("the checkpoint has no payload {payload_name} for {who}"), + ), + "stage-restore", + )); + }; + let params = StageRestoreParams { + checkpoint_id: manifest.checkpoint_id.clone(), + source_scope: manifest.source_scope.clone(), + compatibility_digest: compatibility_digest.clone(), + payload: artifact.reference().clone(), + }; + let attachments = [(crate::state::PAYLOAD_ATTACHMENT, artifact)]; + let reply = match self + .call( + worker, + "State.StageRestore", + Some(scope.clone()), + object(params.to_json()), + &attachments, + &[], + ) + .await + { + Ok(reply) => reply, + Err(failure) => { + // Whoever already staged is holding a replacement state this group will + // never install. Nothing is resumed, and none of them is reused. + for (done, _, _) in &staged { + self.tainted.insert(done.clone()); + } + return Err(failure); + } + }; + let result: StageRestoreResult = match reply.parse() { + Ok(result) => result, + Err(e) => { + for (done, _, _) in &staged { + self.tainted.insert(done.clone()); + } + return Err(self.fail_now(e, "stage-restore")); + } + }; + if result.checkpoint_id != manifest.checkpoint_id { + for (done, _, _) in &staged { + self.tainted.insert(done.clone()); + } + return Err(self.fail_now( + DomainError::before( + ErrorCode::IdentityMismatch, + "a staged restore names another checkpoint", + ), + "stage-restore", + )); + } + self.lifecycle_acks.push((worker.clone(), reply.request_id.clone())); + self.audit.push(format!("staged:{who}")); + staged.push((who.clone(), worker.clone(), result.restore_token)); + } + + // `state-media-v1` section 5: activation happens "after every participant **and + // coordinator** state validates". The coordinator's own staged ledgers are checked + // here, before a single token is activated, so a checkpoint whose task ledger or + // executor state is unreadable installs nothing anywhere. + let payloads: BTreeMap> = envelope + .payloads + .iter() + .map(|(name, bytes)| (name.clone(), bytes.clone())) + .collect(); + let descriptor = match &self.descriptor { + Some(descriptor) => descriptor.clone(), + None => { + self.taint_group(&staged); + return Err(self.fail_now( + DomainError::before( + ErrorCode::InvalidPhase, + "the session has no environment descriptor to restore against", + ), + "restore", + )); + } + }; + if let Err(e) = Coordinator::validate_coordinator_state( + manifest, + &payloads, + &descriptor, + self.agents.len(), + &*self.task, + &self.executors, + ) { + self.taint_group(&staged); + return Err(self.fail_now(e, "restore")); + } + + // Step 3, last part: activate. Anything that activates before a failure holds state + // no group resumed, so it is recorded as tainted and must be replaced. + let mut activated: Vec = Vec::new(); + let mut restored_observation: Option<(WorldObservation, BTreeMap)> = + None; + for (who, worker, token) in &staged { + let params = ActivateRestoreParams { restore_token: token.clone() }; + let want = if *who == self.environment.worker_id { + self.media_names.clone() + } else { + Vec::new() + }; + let reply = match self + .call( + worker, + "State.ActivateRestore", + Some(scope.clone()), + object(params.to_json()), + &[], + &want, + ) + .await + { + Ok(reply) => reply, + Err(failure) => { + self.taint_group(&staged); + return Err(failure); + } + }; + let result: ActivateRestoreResult = match reply.parse() { + Ok(result) => result, + Err(e) => { + self.taint_group(&staged); + return Err(self.fail_now(e, "activate-restore")); + } + }; + let role = if *who == self.environment.worker_id { + Role::Environment + } else { + Role::Agent + }; + if let Err(e) = result.validate_for_role(role) { + self.taint_group(&staged); + return Err(self.fail_now(DomainError::invalid(e.0), "activate-restore")); + } + if result.committed_step != boundary || result.checkpoint_id != manifest.checkpoint_id { + self.taint_group(&staged); + return Err(self.fail_now( + DomainError::before( + ErrorCode::IdentityMismatch, + "an activation names another boundary or checkpoint", + ), + "activate-restore", + )); + } + if let Some(observation) = result.observation { + let (views, audio) = media::split_attachments(reply.artifacts); + if !audio.is_empty() { + self.taint_group(&staged); + return Err(self.fail_now( + DomainError::new( + ErrorCode::BufferInvalid, + "a restored observation carried an audio chunk; no interval was played", + MutationCertainty::Unknown, + ), + "activate-restore", + )); + } + restored_observation = Some((observation, views)); + } + self.lifecycle_acks.push((worker.clone(), reply.request_id.clone())); + self.audit.push(format!("activated:{who}")); + activated.push(who.clone()); + } + + // Step 4: verify identity and boundary, and flush the old media and parser state. + let Some((observation, views)) = restored_observation else { + self.taint_group(&staged); + return Err(self.fail_now( + DomainError::before( + ErrorCode::IncompatibleState, + "no participant returned the restored world observation", + ), + "activate-restore", + )); + }; + let install = self.install_restored( + manifest, + new_epoch, + boundary, + &descriptor, + observation, + views, + &payloads, + ); + if let Err(e) = install { + self.taint_group(&staged); + return Err(self.fail_now(e, "restore")); + } + + // Step 5: Paused(k), and only now is the fence lifted. + self.epoch = new_epoch.clone(); + self.transition(Phase::Paused(boundary))?; + self.fenced = false; + self.acknowledge_lifecycle().await?; + self.publish_checkpoint_event("restored", &manifest.checkpoint_id, boundary, None) + .await?; + self.publish_descriptor().await?; + self.publish_snapshot(boundary, &BTreeMap::new(), &[], &[]).await?; + self.audit.push(format!("restored:{}@{boundary}", manifest.checkpoint_id)); + Ok(RestoreReport { + checkpoint_id: manifest.checkpoint_id.clone(), + boundary, + epoch: new_epoch.clone(), + staged: staged.iter().map(|(who, _, _)| who.clone()).collect(), + tokens: staged + .iter() + .map(|(who, _, token)| (who.clone(), token.clone())) + .collect(), + activated, + imported: imported_ids, + }) + } + + /// Reads one of the checkpoint's payloads as JSON, or says which one is unreadable. + fn read_payload(payloads: &BTreeMap>, name: &str) -> DomainResult { + let bytes = payloads.get(name).ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + format!("the checkpoint has no payload {name}"), + ) + })?; + serde_json::from_slice(bytes).map_err(|e| { + DomainError::before( + ErrorCode::IncompatibleState, + format!("payload {name} is not JSON: {e}"), + ) + }) + } + + /// Validates every coordinator-owned payload, changing nothing. + /// + /// This runs after the group has staged and before anything activates, which is the order + /// `state-media-v1` section 5 sets. It is a separate pass from the install below on + /// purpose: a group that cannot be resumed coherently must not have resumed some of it. + fn validate_coordinator_state( + manifest: &crate::state::CheckpointManifest, + payloads: &BTreeMap>, + descriptor: &EnvironmentDescriptor, + agents: usize, + task: &dyn crate::task::Task, + executors: &BTreeMap>, + ) -> DomainResult<()> { + let ledger = TypedValue::from_json(&Coordinator::read_payload( + payloads, + &manifest.coordinator.task_ledger, + )?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + task.validate_restore(&ledger)?; + for (agent_id, name) in &manifest.coordinator.executor_state { + let state = TypedValue::from_json(&Coordinator::read_payload(payloads, name)?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + let executor = executors.get(agent_id).ok_or_else(|| { + DomainError::before( + ErrorCode::IdentityMismatch, + format!("{agent_id} has no configured action executor"), + ) + })?; + executor.validate_restore(&state)?; + } + let inspection = TypedValue::from_json(&Coordinator::read_payload( + payloads, + &manifest.coordinator.prior_inspection, + )?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + if inspection.schema != descriptor.inspection_schema { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the recorded prior inspection is not the declared inspection schema", + )); + } + let record = + Coordinator::read_payload(payloads, &manifest.coordinator.admission_state)?; + let recorded = record + .get("agents") + .and_then(Value::as_array) + .ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the coordinator record has no agent list", + ) + })?; + if recorded.len() != agents { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the coordinator record names another number of agents", + )); + } + if record.get("audioPositions").and_then(Value::as_object).is_none() { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the coordinator record has no audio positions", + )); + } + Ok(()) + } + + /// Installs the coordinator's own staged state, after every participant activated. + #[allow(clippy::too_many_arguments)] + fn install_restored( + &mut self, + manifest: &crate::state::CheckpointManifest, + new_epoch: &Id, + boundary: u64, + descriptor: &EnvironmentDescriptor, + observation: WorldObservation, + views: BTreeMap, + payloads: &BTreeMap>, + ) -> DomainResult<()> { + if observation.boundary != boundary { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "the restored observation is not at the restored boundary", + )); + } + if observation.world_time != manifest.world_time { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the restored observation's world time is not the checkpoint's", + )); + } + observation + .validate_against(descriptor) + .map_err(|e| DomainError::new(ErrorCode::BufferInvalid, e, MutationCertainty::Unknown))?; + media::check_required_views(descriptor, &observation)?; + // An installed observation ran no transition, so it carries no chunk. Old epoch audio + // arriving as current is exactly what this refuses. + media::check_required_audio(descriptor, &observation, media::ObservationOrigin::Installed)?; + for view in &observation.sensory_views { + let name = media::view_attachment(&view.view_id); + match views.get(&name) { + Some(artifact) if artifact.reference() == &view.pixels => {} + _ => { + return Err(DomainError::new( + ErrorCode::BufferInvalid, + format!( + "the restored view {} arrived without a live owned handle", + view.view_id + ), + MutationCertainty::Unknown, + )); + } + } + } + + let read = |name: &str| Coordinator::read_payload(payloads, name); + + let ledger = TypedValue::from_json(&read(&manifest.coordinator.task_ledger)?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + self.task.validate_restore(&ledger)?; + for (agent_id, name) in &manifest.coordinator.executor_state { + let state = TypedValue::from_json(&read(name)?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + let executor = self.executors.get(agent_id).ok_or_else(|| { + DomainError::before( + ErrorCode::IdentityMismatch, + format!("{agent_id} has no configured action executor"), + ) + })?; + executor.validate_restore(&state)?; + } + let record = read(&manifest.coordinator.admission_state)?; + let inspection = TypedValue::from_json(&read(&manifest.coordinator.prior_inspection)?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + if inspection.schema != descriptor.inspection_schema { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the recorded prior inspection is not the declared inspection schema", + )); + } + let agents = record + .get("agents") + .and_then(Value::as_array) + .ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the coordinator record has no agent list", + ) + })? + .clone(); + if agents.len() != self.agents.len() { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the coordinator record names another number of agents", + )); + } + let positions = record + .get("audioPositions") + .and_then(Value::as_object) + .ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the coordinator record has no audio positions", + ) + })?; + let mut audio_positions = BTreeMap::new(); + for (stream, value) in positions { + let sample: u64 = value + .as_str() + .ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "a recorded audio position is not a canonical U64", + ) + })? + .parse() + .map_err(|_| { + DomainError::before( + ErrorCode::IncompatibleState, + "a recorded audio position is not a canonical U64", + ) + })?; + audio_positions.insert(stream.clone(), sample); + } + // Everything above validated. From here the coordinator installs, in one pass. + self.task.install_restore(new_epoch, &ledger)?; + for (agent_id, name) in &manifest.coordinator.executor_state { + let state = TypedValue::from_json(&read(name)?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + let executor = self + .executors + .get_mut(agent_id) + .expect("checked immediately above"); + executor.install_restore(&state)?; + } + for value in &agents { + let agent_id = value.get("agentId").and_then(Value::as_str).ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "a coordinator agent record has no agentId", + ) + })?; + let slot = self + .agents + .iter_mut() + .find(|slot| slot.agent_id == agent_id) + .ok_or_else(|| { + DomainError::before( + ErrorCode::IdentityMismatch, + format!("the coordinator record names {agent_id}, which is not configured"), + ) + })?; + let context = TypedValue::from_json(value.get("context").ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "a coordinator agent record has no decision context", + ) + })?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + let committed: u64 = value + .get("committedStep") + .and_then(Value::as_str) + .ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "a coordinator agent record has no committedStep", + ) + })? + .parse() + .map_err(|_| { + DomainError::before( + ErrorCode::IncompatibleState, + "a recorded committed step is not a canonical U64", + ) + })?; + if committed != boundary { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "a coordinator agent record is at another boundary", + )); + } + let brain_ticks: u64 = value + .get("brainTicks") + .and_then(Value::as_str) + .ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "a coordinator agent record has no brainTicks", + ) + })? + .parse() + .map_err(|_| { + DomainError::before( + ErrorCode::IncompatibleState, + "a recorded tick count is not a canonical U64", + ) + })?; + let remainder = RationalNs::from_json(value.get("remainder").ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "a coordinator agent record has no remainder", + ) + })?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + slot.context_digest = context.digest(); + slot.context = context; + slot.committed_step = committed; + slot.brain_ticks = brain_ticks; + slot.remainder = remainder; + slot.prepared = None; + slot.prepare_request = None; + } + // Old media and old parser state are replaced, never carried: the previous epoch's + // handles were dropped by the fence and the timelines start again at their preserved + // sample positions with a discontinuity. + self.views = views; + self.pending_views.clear(); + self.audio.clear(); + self.pending_audio.clear(); + self.timelines = AudioTimelines::restored(descriptor, &audio_positions)?; + self.observation = Some(observation); + self.episode = None; + self.last_advance_request = None; + self.last_commit_requests.clear(); + self.pacing = Some(Pacing::new(descriptor.step_duration)); + Ok(()) + } + + /// The epoch-derived identities of this session's behaviour trace, mapped onto `to_epoch`. + /// + /// `step-v1` section 8 compares behaviour across runs. A resumed run runs in a new epoch, + /// and scope, batch identity and every task event identity are derived from it, so a + /// comparison either accounts for that or compares nothing. This is what "accounting for + /// new epoch metadata" is: an explicit, total rewrite of the epoch-derived fields, which + /// fails rather than passing anything through it does not recognise. + pub fn rebase(&self, to_epoch: &Id) -> Outcome { + let events = self.task.rebase_ids(to_epoch).map_err(|e| SessionFailure { + error: e, + phase: self.phases.phase().label(), + detail: "rebase".to_owned(), + participant: None, + })?; + Ok(EpochRebase { + from: self.epoch.clone(), + to: to_epoch.clone(), + events, + }) + } +} diff --git a/services/flysim/crates/fly-session/src/environment.rs b/services/flysim/crates/fly-session/src/environment.rs index 8a2947b..9c1bd85 100644 --- a/services/flysim/crates/fly-session/src/environment.rs +++ b/services/flysim/crates/fly-session/src/environment.rs @@ -13,6 +13,8 @@ use std::collections::BTreeSet; +use serde_json::Value; + use crate::media::{self, AudioSource, RenderCounter, ViewPipeline}; use crate::task::{controller_schema_ref, inspection, inspection_schema}; // `crate::types` is this crate's facade over the shared `fly-session-types` crate; the @@ -48,6 +50,12 @@ pub struct EnvironmentFaults { pub omit_audio_at_boundary: Option, /// Emit an audio chunk that starts before the previous chunk ended. pub overlapping_audio_at_boundary: Option, + /// Refuse `State.StageRestore`, so a group install meets a participant that will not + /// validate. + pub fail_stage_restore: bool, + /// Refuse `State.ActivateRestore` after staging, so a group meets a failure halfway + /// through activation. + pub fail_activate_restore: bool, } #[derive(Clone, Debug)] @@ -86,6 +94,10 @@ pub struct CounterEnvironment { audio: Option, /// The frame served at the previous boundary, kept only so a fault can serve it again. previous_view: Option<(ViewRef, flybus::Artifact)>, + /// A validated replacement world the live session cannot see yet. + staged: Option, + /// Restore tokens this world has activated. A token activates once. + activated: BTreeSet, } impl CounterEnvironment { @@ -104,10 +116,17 @@ impl CounterEnvironment { pipeline: None, audio: None, previous_view: None, + staged: None, + activated: BTreeSet::new(), config, } } + /// True while a validated replacement world is staged and not yet activated. + pub fn has_staged_restore(&self) -> bool { + self.staged.is_some() + } + pub fn status(&self) -> StatusCell { self.status.clone() } @@ -477,7 +496,7 @@ impl WorkerEndpoint for CounterEnvironment { vec![ id("world-step-v1"), id("pixel-observation-v1"), - id("checkpoint-v1"), + id(crate::state::CHECKPOINT_CAPABILITY), ] } @@ -490,7 +509,13 @@ impl WorkerEndpoint for CounterEnvironment { } fn methods(&self) -> Vec<&'static str> { - vec!["Environment.Initialize", "Environment.Advance"] + vec![ + "Environment.Initialize", + "Environment.Advance", + "State.Capture", + "State.StageRestore", + "State.ActivateRestore", + ] } fn handle<'a>(&'a mut self, ctx: HandlerCtx<'a>) -> BoxFuture<'a, DomainResult> { @@ -498,6 +523,9 @@ impl WorkerEndpoint for CounterEnvironment { match ctx.method { "Environment.Initialize" => self.initialize(&ctx).await, "Environment.Advance" => self.advance(&ctx).await, + "State.Capture" => self.state_capture(&ctx).await, + "State.StageRestore" => self.state_stage_restore(&ctx).await, + "State.ActivateRestore" => self.state_activate_restore(&ctx).await, other => Err(DomainError::before( ErrorCode::Unsupported, format!("{other} is not an environment method"), @@ -516,3 +544,488 @@ pub fn synthetic_asset(asset_id: &str, body: &str) -> AssetRef { format: id("fly-config-v1"), } } + +// ------------------------------------------------------------------------------------------- +// STATE-01: capture and restore + +/// The version this payload layout is written and read under. +pub const WORLD_PAYLOAD_VERSION: u64 = 1; + +fn incompatible(message: impl std::fmt::Display) -> DomainError { + DomainError::before(ErrorCode::IncompatibleState, message) +} + +/// One staged restore, held outside the live world until it is activated. +struct StagedWorld { + token: Id, + checkpoint_id: Id, + scope: Scope, + episode_id: Id, + descriptor: EnvironmentDescriptor, + boundary: u64, + counter: i64, + world_time: RationalNs, + advances: u64, + frames: Vec<(u64, i64)>, + audio_next_sample: u64, + audio_phase: u64, + audio_accumulator: u128, + audio_denominator: u128, +} + +impl CounterEnvironment { + /// `State.Capture`: the world at its committed boundary, including its pending sensor + /// pipeline. + /// + /// The pipeline is recorded as reconstruction inputs -- the producing boundary and the + /// world counter of every retained frame -- and never as an artifact identity: a + /// transient artifact belongs to the router that is running now, and a checkpoint outlives + /// it. + async fn state_capture(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult { + let scope = ctx.scope()?.clone(); + let Some(descriptor) = self.descriptor.clone() else { + return Err(DomainError::before( + ErrorCode::InvalidPhase, + "this environment is uninitialized", + )); + }; + if scope.session_id != self.config.session_id { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "this environment belongs to another session", + )); + } + match &self.epoch { + Some(epoch) if *epoch == scope.epoch => {} + _ => { + return Err(DomainError::before( + ErrorCode::StaleEpoch, + "State.Capture names an epoch this environment has left", + )); + } + } + if scope.step != self.boundary { + return Err(DomainError::before( + if scope.step < self.boundary { + ErrorCode::StaleStep + } else { + ErrorCode::FutureStep + }, + "State.Capture must name the boundary the world is at", + )); + } + let params: CaptureParams = ctx.params()?; + let pipeline = self + .pipeline + .as_ref() + .ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no view pipeline"))?; + let audio = self + .audio + .as_ref() + .ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no audio source"))?; + let (accumulator, denominator) = audio.accumulator(); + let previous = self.status.state(); + self.status.set_state(WorkerState::Capturing); + let payload = serde_json::json!({ + "payloadVersion": WORLD_PAYLOAD_VERSION, + "kind": "world", + "workerId": self.config.worker_id.as_str(), + "checkpointId": params.checkpoint_id.as_str(), + "sourceScope": scope.to_json(), + "episodeId": self.episode_id.clone().expect("initialized").as_str(), + "committedStep": self.boundary.to_string(), + "counter": self.counter.to_string(), + "worldTime": self.world_time.to_json(), + "advances": self.advances.to_string(), + "descriptor": descriptor.to_json(), + "pipeline": { + // The declared delay's whole queue, oldest first. + "frames": pipeline + .retained() + .into_iter() + .map(|(boundary, counter)| serde_json::json!({ + "boundary": boundary.to_string(), + "counter": counter.to_string(), + })) + .collect::>(), + }, + "audio": { + "nextSample": audio.next_sample().to_string(), + "phase": audio.phase().to_string(), + "accumulator": accumulator.to_string(), + "denominator": denominator.to_string(), + "chunks": audio.chunks().to_string(), + }, + }); + let bytes = canonicalize(&payload) + .map_err(|e| DomainError::invalid(format!("State.Capture: {}", e.0)))? + .into_bytes(); + let digest = digest_of_bytes(&bytes); + let artifact = crate::state::seal_payload(ctx.client, &bytes, &digest).await?; + // A capture reads the world; it does not advance it. + self.status.set_state(previous); + let result = CaptureResult { + checkpoint_id: params.checkpoint_id, + boundary: self.boundary, + compatibility_digest: crate::state::Compatibility::of(&descriptor).digest(), + payload: artifact.reference().clone(), + }; + Ok(HandlerReply::with_artifacts( + object(result.to_json()), + vec![(crate::state::PAYLOAD_ATTACHMENT.to_owned(), artifact)], + )) + } + + /// `State.StageRestore`: validate a replacement world into a staging slot. + async fn state_stage_restore(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult { + let scope = ctx.scope()?.clone(); + if scope.session_id != self.config.session_id { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "this environment belongs to another session", + )); + } + if let Some(epoch) = &self.epoch + && *epoch == scope.epoch + { + return Err(DomainError::before( + ErrorCode::StaleEpoch, + "State.StageRestore proposes the epoch this environment is already running", + )); + } + if self.descriptor.is_some() { + // A world that is already running a boundary is not a quiescent replacement: the + // group replaces it rather than restoring over a live one. + return Err(DomainError::before( + ErrorCode::InvalidPhase, + "State.StageRestore needs an uninitialized replacement environment", + )); + } + let params: StageRestoreParams = ctx.params()?; + if params.source_scope.step != scope.step { + return Err(DomainError::invalid( + "State.StageRestore's scope step must be the source boundary", + )); + } + let artifact = ctx.artifact(crate::state::PAYLOAD_ATTACHMENT)?; + if artifact.reference() != ¶ms.payload { + return Err(DomainError::before( + ErrorCode::BufferInvalid, + "the staged payload attachment is not the artifact the request names", + )); + } + let bytes = artifact.read_all().await.map_err(|e| { + DomainError::before( + ErrorCode::BufferInvalid, + format!("the staged payload could not be read: {}", e.message), + ) + })?; + let declared = params + .payload + .digest + .clone() + .ok_or_else(|| incompatible("a checkpoint payload must carry a content digest"))?; + let actual = digest_of_bytes(&bytes); + if actual != declared || bytes.len() as u64 != params.payload.byte_length { + return Err(incompatible( + "the staged payload is not the content the request declares", + )); + } + let value: Value = serde_json::from_slice(&bytes) + .map_err(|e| DomainError::invalid(format!("the staged payload is not JSON: {e}")))?; + let text = |key: &str| -> DomainResult { + value + .get(key) + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| incompatible(format!("the world payload has no {key}"))) + }; + let number = |key: &str| -> DomainResult { + text(key)? + .parse::() + .map_err(|_| incompatible(format!("the world payload's {key} is not a U64"))) + }; + if value.get("payloadVersion").and_then(Value::as_u64) != Some(WORLD_PAYLOAD_VERSION) { + return Err(incompatible("the world payload is another payload version")); + } + if text("kind")? != "world" { + return Err(incompatible("this payload is not a world's state")); + } + if text("workerId")? != self.config.worker_id { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "the staged payload belongs to another world", + )); + } + if text("checkpointId")? != params.checkpoint_id { + return Err(incompatible("the staged payload belongs to another checkpoint")); + } + let source_scope = Scope::from_json( + value + .get("sourceScope") + .ok_or_else(|| incompatible("the world payload has no sourceScope"))?, + ) + .map_err(|e| incompatible(format!("the world payload's sourceScope: {}", e.0)))?; + if source_scope != params.source_scope { + return Err(incompatible( + "the staged payload was captured at another source scope", + )); + } + let committed_step = number("committedStep")?; + if committed_step != params.source_scope.step { + return Err(incompatible( + "the staged payload's committed step is not the source boundary", + )); + } + let descriptor = EnvironmentDescriptor::from_json( + value + .get("descriptor") + .ok_or_else(|| incompatible("the world payload has no descriptor"))?, + ) + .map_err(|e| incompatible(format!("the world payload's descriptor: {}", e.0)))?; + // The replacement builds the descriptor it would advertise and compares. A world + // started with other ports, another cadence or another declared render delay is a + // different backend, not this one resumed. + let live = self.build_descriptor()?; + if descriptor != live { + return Err(incompatible( + "the staged world was captured under another environment descriptor", + )); + } + let expected = crate::state::Compatibility::of(&descriptor).digest(); + if expected != params.compatibility_digest { + return Err(incompatible(format!( + "the staged world's compatibility {expected} is not the {} the restore requires", + params.compatibility_digest + ))); + } + let counter: i64 = text("counter")? + .parse() + .map_err(|_| incompatible("the world payload's counter is not an integer"))?; + let world_time = RationalNs::from_json( + value + .get("worldTime") + .ok_or_else(|| incompatible("the world payload has no worldTime"))?, + ) + .map_err(|e| incompatible(format!("the world payload's worldTime: {}", e.0)))?; + let pipeline_value = value + .get("pipeline") + .and_then(|p| p.get("frames")) + .and_then(Value::as_array) + .ok_or_else(|| incompatible("the world payload has no pipeline frames"))?; + let mut frames = Vec::with_capacity(pipeline_value.len()); + for frame in pipeline_value { + let boundary = frame + .get("boundary") + .and_then(Value::as_str) + .ok_or_else(|| incompatible("a captured frame has no boundary"))? + .parse::() + .map_err(|_| incompatible("a captured frame's boundary is not a U64"))?; + let frame_counter = frame + .get("counter") + .and_then(Value::as_str) + .ok_or_else(|| incompatible("a captured frame has no counter"))? + .parse::() + .map_err(|_| incompatible("a captured frame's counter is not an integer"))?; + frames.push((boundary, frame_counter)); + } + match frames.last() { + Some((boundary, _)) if *boundary == committed_step => {} + _ => { + return Err(incompatible( + "the captured pipeline does not end at the committed boundary", + )); + } + } + let audio_value = value + .get("audio") + .ok_or_else(|| incompatible("the world payload has no audio state"))?; + let audio_number = |key: &str| -> DomainResult { + audio_value + .get(key) + .and_then(Value::as_str) + .ok_or_else(|| incompatible(format!("the captured audio state has no {key}")))? + .parse::() + .map_err(|_| incompatible(format!("the captured audio {key} is not a number"))) + }; + let audio_next_sample = u64::try_from(audio_number("nextSample")?) + .map_err(|_| incompatible("the captured audio position is outside U64"))?; + let audio_phase = u64::try_from(audio_number("phase")?) + .map_err(|_| incompatible("the captured audio phase is outside U64"))?; + + if self.config.faults.fail_stage_restore { + return Err(incompatible( + "injected staging refusal: this participant's replacement state does not \ +validate", + )); + } + if let Some(staged) = &self.staged { + return Err(DomainError::before( + ErrorCode::Conflict, + format!( + "this environment already holds the staged restore {} for checkpoint {}", + staged.token, staged.checkpoint_id + ), + )); + } + let token = crate::agent::restore_token( + ¶ms.checkpoint_id, + &scope, + &actual, + &self.config.incarnation_id, + ); + if self.activated.contains(&token) { + return Err(DomainError::before( + ErrorCode::Conflict, + "this exact restore was already activated on this environment", + )); + } + self.staged = Some(StagedWorld { + token: token.clone(), + checkpoint_id: params.checkpoint_id.clone(), + scope, + episode_id: parse_id(&text("episodeId")?) + .map_err(|e| incompatible(format!("the world payload's episodeId {e}")))?, + descriptor, + boundary: committed_step, + counter, + world_time, + advances: number("advances")?, + frames, + audio_next_sample, + audio_phase, + audio_accumulator: audio_number("accumulator")?, + audio_denominator: audio_number("denominator")?, + }); + self.status.set_state(WorkerState::StagedRestore); + let result = StageRestoreResult { + checkpoint_id: params.checkpoint_id, + restore_token: token, + }; + Ok(HandlerReply::from(&result)) + } + + /// `State.ActivateRestore`: install the staged world and return its coherent observation. + /// + /// Nothing advances. The pipeline's frames are rendered again into fresh artifacts of the + /// current store, which is what "the durable store imports fresh immutable bus artifacts" + /// means on the producing side, and the observation carries no audio chunk because no + /// interval was played. + async fn state_activate_restore( + &mut self, + ctx: &HandlerCtx<'_>, + ) -> DomainResult { + let params: ActivateRestoreParams = ctx.params()?; + if self.activated.contains(¶ms.restore_token) { + return Err(DomainError::before( + ErrorCode::Conflict, + "this restore token has already been activated", + )); + } + let Some(staged) = self.staged.take() else { + return Err(DomainError::before( + ErrorCode::InvalidPhase, + "this environment holds no staged restore", + )); + }; + if staged.token != params.restore_token { + let token = staged.token.clone(); + self.staged = Some(staged); + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + format!( + "this environment's staged restore is {token}, not {}", + params.restore_token + ), + )); + } + if self.config.faults.fail_activate_restore { + let token = staged.token.clone(); + self.staged = Some(staged); + self.status.set_state(WorkerState::Failed); + return Err(DomainError::new( + ErrorCode::BackendFailure, + format!("injected activation failure; {token} stays staged and unresumed"), + MutationCertainty::None, + )); + } + self.status.set_state(WorkerState::Restoring); + let mut pipeline = ViewPipeline::new( + CounterEnvironment::view_descriptor(self.config.observation_delay_steps), + self.config.renders.clone(), + ); + pipeline.restore(ctx.client, &staged.frames).await?; + let audio = AudioSource::restored_from( + CounterEnvironment::audio_descriptor(), + staged.audio_next_sample, + staged.audio_phase, + staged.audio_accumulator, + staged.audio_denominator, + )?; + self.epoch = Some(staged.scope.epoch.clone()); + self.episode_id = Some(staged.episode_id.clone()); + self.descriptor = Some(staged.descriptor.clone()); + self.boundary = staged.boundary; + self.counter = staged.counter; + self.world_time = staged.world_time; + self.advances = staged.advances; + // Batch ids are unique within an epoch, and this is a new one. Keeping the old set + // would refuse nothing extra: a request under the old epoch is already refused by its + // scope. + self.batches.clear(); + self.pipeline = Some(pipeline); + self.audio = Some(audio); + self.previous_view = None; + self.activated.insert(staged.token); + self.status.set_state(WorkerState::Ready); + self.status.set_scope(Some(scope_at( + &staged.scope.session_id, + &staged.scope.epoch, + staged.boundary, + ))); + + let (observation, attachments) = self.restored_observation()?; + let result = ActivateRestoreResult { + committed_step: staged.boundary, + checkpoint_id: staged.checkpoint_id, + observation: Some(observation), + }; + result + .validate_for_role(Role::Environment) + .map_err(|e| DomainError::invalid(e.0))?; + let mut reply = HandlerReply::from(&result); + reply.artifacts = attachments; + Ok(reply) + } + + /// The observation the restored world is already at: no render, no advance, no audio. + fn restored_observation( + &mut self, + ) -> DomainResult<(WorldObservation, Vec<(String, flybus::Artifact)>)> { + let boundary = self.boundary; + let counter = self.counter; + let pipeline = self + .pipeline + .as_ref() + .ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no view pipeline"))?; + let (view, artifact) = pipeline.at(boundary).ok_or_else(|| { + incompatible("the restored pipeline holds no frame for the restored boundary") + })?; + self.previous_view = Some((view.clone(), artifact.clone())); + let observation = WorldObservation { + boundary, + world_time: self.world_time, + engine_frame: Some(boundary.to_string()), + sensory_views: vec![view.clone()], + inspection: inspection(counter, boundary), + broadcast_views: vec![view.clone()], + // No interval was played, so there is no chunk. A chunk here would be an old + // epoch's audio offered as current. + audio: Vec::new(), + }; + Ok(( + observation, + vec![(media::view_attachment(&view.view_id), artifact)], + )) + } +} diff --git a/services/flysim/crates/fly-session/src/harness.rs b/services/flysim/crates/fly-session/src/harness.rs index c537a9b..53159d1 100644 --- a/services/flysim/crates/fly-session/src/harness.rs +++ b/services/flysim/crates/fly-session/src/harness.rs @@ -26,6 +26,7 @@ use crate::media::{RenderCounter, SensorLog}; use crate::launcher::{ AgentLaunch, EnvironmentLaunch, Launcher, ReapOutcome, SUPERVISOR_CLIENT, ThreadBudget, }; +use crate::state::{CheckpointStore, CheckpointWriter, StoreConfig, StoreFaults, WriterConfig, WriterFaults}; use crate::task::{ActionExecutor, CounterTask, IdentityExecutor, Terminal}; // `crate::types` is this crate's facade over the shared `fly-session-types` crate; the // glob keeps the contract's own names in sight instead of restating them. @@ -85,6 +86,14 @@ pub struct HarnessConfig { /// The threads reserved for the coordinator, its router and its store. pub coordinator_threads: usize, pub environment_threads: usize, + /// How many committed generations the durable checkpoint store keeps. + pub store: StoreConfig, + /// The durable write faults this composition injects. + pub store_faults: StoreFaults, + /// The checkpoint queue's bounds. + pub writer: WriterConfig, + /// The writer faults this composition injects. + pub writer_faults: WriterFaults, } impl Default for HarnessConfig { @@ -107,6 +116,10 @@ impl Default for HarnessConfig { thread_budget: None, coordinator_threads: 1, environment_threads: 1, + store: StoreConfig::default(), + store_faults: StoreFaults::default(), + writer: WriterConfig::default(), + writer_faults: WriterFaults::default(), } } } @@ -134,6 +147,14 @@ const ENV_SERVICE: &str = "env.arena"; const ENV_CLIENT: &str = "environment"; const ENV_WORKER: &str = "arena"; const COORDINATOR_CLIENT: &str = "coordinator"; +/// The checkpoint writer's own bus identity. It publishes checkpoint events and nothing else. +const WRITER_CLIENT: &str = "checkpoint-writer"; + +/// How many times one participant may be replaced in a composition. +/// +/// Each replacement connects under its own client id, so a restart is visibly a new +/// participant rather than a silent reattachment, and the policy has to name them all. +const MAX_GENERATIONS: u32 = 8; fn agent_service(agent_id: &Id) -> String { format!("agent.{agent_id}") @@ -172,6 +193,10 @@ pub struct SessionHarness { /// The supervisor. It owns every participant's lifetime and thread allocation. pub launcher: Launcher, observers: Mutex>, + /// Which generation of each participant is running: 1 is the one the composition started. + generations: BTreeMap, + /// Where the durable checkpoint store lives, for a test that reads the files themselves. + checkpoint_root: std::path::PathBuf, } impl SessionHarness { @@ -204,12 +229,16 @@ impl SessionHarness { g.call = vec![Pattern::prefix("agent."), Pattern::prefix("env.")]; }), ) + // The writer publishes the checkpoint events and never calls a participant. + .client(WRITER_CLIENT, grants(|g| g.publish = vec![Pattern::prefix("session.")])) .client(ENV_CLIENT, grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)])) - .client( - &format!("{ENV_CLIENT}-r2"), - grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]), - ) .client("observer", grants(|g| g.subscribe = vec![Pattern::prefix("session.")])); + for generation in 2..=MAX_GENERATIONS { + policy = policy.client( + &format!("{ENV_CLIENT}-r{generation}"), + grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]), + ); + } for spec in &config.agents { let service = agent_service(&spec.agent_id); policy = policy.client( @@ -218,10 +247,12 @@ impl SessionHarness { ); // A replacement worker connects under its own client id, so a restart is visibly a // new participant rather than a silent reattachment to the active epoch. - policy = policy.client( - &format!("{}-r2", agent_client(&spec.agent_id)), - grants(|g| g.register = vec![Pattern::exact(&service)]), - ); + for generation in 2..=MAX_GENERATIONS { + policy = policy.client( + &format!("{}-r{generation}", agent_client(&spec.agent_id)), + grants(|g| g.register = vec![Pattern::exact(&service)]), + ); + } } let mut router_config = RouterConfig::new(&store_root); router_config.policy = policy; @@ -299,6 +330,21 @@ impl SessionHarness { } let coordinator_client = launcher.connect(COORDINATOR_CLIENT).await?; + // The durable store lives beside the router's artifact store and never inside it: a + // committed generation is outside the bus's ephemeral collection. + let checkpoint_root = root.join("checkpoints"); + let mut store = CheckpointStore::open(&checkpoint_root, config.store).map_err(refusal)?; + *store.faults_mut() = config.store_faults.clone(); + let writer_client = launcher.connect(WRITER_CLIENT).await?; + let writer = CheckpointWriter::start( + store, + config.writer, + config.writer_faults.clone(), + Some(( + writer_client, + format!("session.{}.checkpoints", config.session_id), + )), + ); let executors: BTreeMap> = config .agents .iter() @@ -316,6 +362,8 @@ impl SessionHarness { Box::new(CounterTask::new(&config.epoch, config.terminal)), executors, ); + let mut coordinator = coordinator; + coordinator.attach_store(writer); Ok(SessionHarness { coordinator, @@ -326,9 +374,16 @@ impl SessionHarness { sensors, launcher, observers: Mutex::new(Vec::new()), + generations: BTreeMap::new(), + checkpoint_root, }) } + /// Where the durable checkpoint store's generations and store manifest live. + pub fn checkpoint_root(&self) -> &std::path::Path { + &self.checkpoint_root + } + pub fn router(&self) -> &Router { self.launcher.router() } @@ -373,10 +428,11 @@ impl SessionHarness { .find(|spec| spec.agent_id == *agent_id) .expect("a configured agent") .clone(); + let generation = self.next_generation(agent_id)?; self.launcher.kill(agent_id).await; let tick_duration = millis(self.config.tick_ms).expect("a positive tick"); - let incarnation_id = - parse_id(&format!("{agent_id}-inc-2")).expect("an agent id plus a suffix is an Id"); + let incarnation_id = parse_id(&format!("{agent_id}-inc-{generation}")) + .expect("an agent id plus a suffix is an Id"); self.launcher .launch_agent(AgentLaunch { session_id: self.config.session_id.clone(), @@ -390,7 +446,7 @@ impl SessionHarness { // predecessor wrote, so a restore's sensory input is visible beside it. sensors: self.sensors.get(agent_id).cloned().unwrap_or_default(), faults: spec.faults.clone(), - client_id: format!("{}-r2", agent_client(agent_id)), + client_id: format!("{}-r{generation}", agent_client(agent_id)), service: agent_service(agent_id), }) .await @@ -403,6 +459,102 @@ impl SessionHarness { }) } + /// Replaces the environment with a fresh, uninitialized incarnation, as a restore needs. + pub async fn restart_environment(&mut self) -> Result { + let worker_id = id(ENV_WORKER); + let generation = self.next_generation(&worker_id)?; + self.launcher.kill(&worker_id).await; + let step_duration = hz(self.config.step_hz).expect("a positive cadence"); + let incarnation_id = parse_id(&format!("arena-inc-{generation}")) + .expect("a worker id plus a suffix is an Id"); + self.launcher + .launch_environment(EnvironmentLaunch { + session_id: self.config.session_id.clone(), + worker_id: worker_id.clone(), + incarnation_id: incarnation_id.clone(), + step_duration, + ports: self.config.agents.iter().map(|a| a.port_id.clone()).collect(), + worker_threads: self.config.environment_threads, + observation_delay_steps: self.config.observation_delay_steps, + renders: self.renders.clone(), + faults: self.config.environment_faults.clone(), + client_id: format!("{ENV_CLIENT}-r{generation}"), + service: ENV_SERVICE.to_owned(), + }) + .await + .map_err(refusal)?; + let worker = self.launcher.worker(&worker_id).expect("just launched"); + Ok(Restarted { + service: worker.identity.service.clone(), + service_incarnation: worker.service_incarnation.clone(), + incarnation_id, + }) + } + + fn next_generation(&mut self, worker_id: &Id) -> Result { + let slot = self.generations.entry(worker_id.clone()).or_insert(1); + if *slot >= MAX_GENERATIONS { + return Err(flybus::BusError::new( + flybus::ErrorCode::QuotaExceeded, + format!( + "{worker_id} has used all {MAX_GENERATIONS} configured client identities; a composition declares how many replacements it allows" + ), + )); + } + *slot += 1; + Ok(*slot) + } + + /// Replaces every participant and points the fenced coordinator at the replacements. + /// + /// This is what a recovery does before it restores: the old participants belong to an + /// invalid epoch, and the references the coordinator pinned are exchanged deliberately. + pub async fn replace_all_participants(&mut self) -> Result<(), flybus::BusError> { + let environment = self.environment_id(); + self.restart_environment().await?; + let worker = self + .launcher + .worker(&environment) + .expect("just launched") + .worker_ref(); + self.coordinator + .replace_participant(&environment, worker) + .map_err(|e| refusal(e.error))?; + for agent_id in self.config.agents.iter().map(|a| a.agent_id.clone()).collect::>() { + self.restart_agent(&agent_id).await?; + let worker = self + .launcher + .worker(&agent_id) + .expect("just launched") + .worker_ref(); + self.coordinator + .replace_participant(&agent_id, worker) + .map_err(|e| refusal(e.error))?; + } + Ok(()) + } + + /// Changes one agent's injected faults, so the replacement the next restart launches is + /// a participant without them. + /// + /// A fault is launch configuration, so clearing one is a relaunch and not a live change: + /// the worker running now keeps whatever it was started with. + pub fn set_agent_faults(&mut self, agent_id: &Id, faults: AgentFaults) { + if let Some(spec) = self + .config + .agents + .iter_mut() + .find(|spec| spec.agent_id == *agent_id) + { + spec.faults = faults; + } + } + + /// Changes the environment's injected faults, with the same relaunch rule. + pub fn set_environment_faults(&mut self, faults: EnvironmentFaults) { + self.config.environment_faults = faults; + } + /// Ends one participant without asking it, as a crash would. pub async fn kill(&mut self, worker_id: &Id) -> ReapOutcome { self.launcher.kill(worker_id).await @@ -466,7 +618,10 @@ impl SessionHarness { /// Reaps every participant and closes the router. pub async fn shutdown(self) { - let SessionHarness { coordinator, mut launcher, observers, .. } = self; + let SessionHarness { mut coordinator, mut launcher, observers, .. } = self; + // The writer task owns artifact handles and a blocking store. Leaving it running + // would leave both behind. + coordinator.shutdown_store().await; drop(coordinator); launcher.reap_all(&id("shutdown")).await; for observer in observers.into_inner().expect("not poisoned") { diff --git a/services/flysim/crates/fly-session/src/launcher.rs b/services/flysim/crates/fly-session/src/launcher.rs index c80551e..43d6705 100644 --- a/services/flysim/crates/fly-session/src/launcher.rs +++ b/services/flysim/crates/fly-session/src/launcher.rs @@ -1231,6 +1231,8 @@ pub(crate) mod flags { pub const PREPARE_DELAY_MS: &str = "prepare-delay-ms"; pub const COMMIT_DELAY_MS: &str = "commit-delay-ms"; pub const FAIL_COMMIT_AT_STEP: &str = "fail-commit-at-step"; + pub const FAIL_STAGE_RESTORE: &str = "fail-stage-restore"; + pub const FAIL_ACTIVATE_RESTORE: &str = "fail-activate-restore"; pub const WORKER: &str = "worker"; pub const PORTS: &str = "ports"; @@ -1271,6 +1273,8 @@ pub(crate) mod flags { PREPARE_DELAY_MS, COMMIT_DELAY_MS, FAIL_COMMIT_AT_STEP, + FAIL_STAGE_RESTORE, + FAIL_ACTIVATE_RESTORE, ]; /// What only the environment is given, media options included. pub const ENVIRONMENT_ONLY: &[&str] = &[ @@ -1285,6 +1289,8 @@ pub(crate) mod flags { TRUNCATED_VIEW_AT_BOUNDARY, OMIT_AUDIO_AT_BOUNDARY, OVERLAPPING_AUDIO_AT_BOUNDARY, + FAIL_STAGE_RESTORE, + FAIL_ACTIVATE_RESTORE, ]; /// What a measurement run or one of its row children is given. pub const MEASURE: &[&str] = &[MODE, AGENTS, STEPS, WARMUP_STEPS, WORKER_THREADS, MODES]; @@ -1327,6 +1333,11 @@ impl Started { arg(flags::WARMUP_TICKS, spec.warmup_ticks), arg(flags::PREPARE_DELAY_MS, spec.faults.prepare_delay_ms), arg(flags::COMMIT_DELAY_MS, spec.faults.commit_delay_ms), + arg(flags::FAIL_STAGE_RESTORE, u64::from(spec.faults.fail_stage_restore)), + arg( + flags::FAIL_ACTIVATE_RESTORE, + u64::from(spec.faults.fail_activate_restore), + ), ]; if let Some(step) = spec.faults.fail_commit_at_step { args.push(arg(flags::FAIL_COMMIT_AT_STEP, step)); @@ -1345,6 +1356,11 @@ impl Started { // The media options a world in another process needs to be exactly this // world. Its render counter and its agents' sensor logs stay there. arg(flags::OBSERVATION_DELAY_STEPS, spec.observation_delay_steps), + arg(flags::FAIL_STAGE_RESTORE, u64::from(spec.faults.fail_stage_restore)), + arg( + flags::FAIL_ACTIVATE_RESTORE, + u64::from(spec.faults.fail_activate_restore), + ), ]; for (flag, boundary) in [ (flags::OMIT_VIEW_AT_BOUNDARY, spec.faults.omit_view_at_boundary), @@ -1458,6 +1474,8 @@ mod flag_tests { truncated_view_at_boundary: Some(3), omit_audio_at_boundary: Some(4), overlapping_audio_at_boundary: Some(5), + fail_stage_restore: true, + fail_activate_restore: true, } } @@ -1491,6 +1509,8 @@ mod flag_tests { fail_commit_at_step: Some(2), prepare_delay_ms: 1, commit_delay_ms: 2, + fail_stage_restore: true, + fail_activate_restore: true, }, client_id: "worker-fly-a".to_owned(), service: "agent.fly-a".to_owned(), @@ -1535,6 +1555,8 @@ mod flag_tests { flags::TRUNCATED_VIEW_AT_BOUNDARY, flags::OMIT_AUDIO_AT_BOUNDARY, flags::OVERLAPPING_AUDIO_AT_BOUNDARY, + flags::FAIL_STAGE_RESTORE, + flags::FAIL_ACTIVATE_RESTORE, ] { assert!(written.contains(&format!("--{flag}")), "--{flag} is not written"); } diff --git a/services/flysim/crates/fly-session/src/lib.rs b/services/flysim/crates/fly-session/src/lib.rs index 15722b5..e6d66c0 100644 --- a/services/flysim/crates/fly-session/src/lib.rs +++ b/services/flysim/crates/fly-session/src/lib.rs @@ -34,6 +34,7 @@ pub mod media; pub mod metrics; pub mod phase; pub mod rpc; +pub mod state; pub mod task; pub mod worker; diff --git a/services/flysim/crates/fly-session/src/media.rs b/services/flysim/crates/fly-session/src/media.rs index 870ead9..1d6c138 100644 --- a/services/flysim/crates/fly-session/src/media.rs +++ b/services/flysim/crates/fly-session/src/media.rs @@ -133,7 +133,14 @@ pub fn arena_frame(descriptor: &ViewDescriptor, counter: i64, boundary: u64) -> /// cannot be served an arbitrary stale image. pub struct ViewPipeline { descriptor: ViewDescriptor, - frames: VecDeque<(u64, flybus::Artifact)>, + /// Each retained frame: its producing boundary, the world counter it was rendered from + /// and the owned handle on its immutable bytes. + /// + /// The counter is kept because it is the whole of the reconstruction input: a checkpoint + /// records `(boundary, counter)` per retained frame and a restore re-renders them into + /// fresh artifacts of the current store, rather than persisting a transient artifact + /// identity that cannot survive a router restart. + frames: VecDeque<(u64, i64, flybus::Artifact)>, renders: RenderCounter, } @@ -160,7 +167,7 @@ impl ViewPipeline { let bytes = arena_frame(&self.descriptor, counter, boundary); let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?; self.renders.bump(); - self.frames.push_back((boundary, artifact)); + self.frames.push_back((boundary, counter, artifact)); // Keep exactly the frames a declared delay can still require. while self.frames.len() > self.descriptor.observation_delay_steps as usize + 1 { self.frames.pop_front(); @@ -168,6 +175,50 @@ impl ViewPipeline { Ok(()) } + /// The reconstruction inputs of every retained frame, oldest first. + /// + /// This is what a checkpoint records for the pending sensor pipeline: the producing + /// boundary and the world counter, never an artifact identity. + pub fn retained(&self) -> Vec<(u64, i64)> { + self.frames + .iter() + .map(|(boundary, counter, _)| (*boundary, *counter)) + .collect() + } + + /// Rebuilds the pipeline from recorded reconstruction inputs, into fresh artifacts. + /// + /// Every frame is rendered again in the current store, so nothing a fence dropped is + /// expected to come back and no old artifact identity crosses the recovery. + pub async fn restore( + &mut self, + client: &flybus::Client, + frames: &[(u64, i64)], + ) -> DomainResult<()> { + if frames.len() > self.descriptor.observation_delay_steps as usize + 1 { + return Err(media_error(format!( + "a captured pipeline of {} frames does not fit a declared delay of {}", + frames.len(), + self.descriptor.observation_delay_steps + ))); + } + for window in frames.windows(2) { + if window[1].0 != window[0].0 + 1 { + return Err(media_error( + "a captured pipeline's producing boundaries are not consecutive", + )); + } + } + self.frames.clear(); + for (boundary, counter) in frames { + let bytes = arena_frame(&self.descriptor, *counter, *boundary); + let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?; + self.renders.bump(); + self.frames.push_back((*boundary, *counter, artifact)); + } + Ok(()) + } + /// Seals a frame of the wrong length, which is what a broken backend produces. The /// reference it returns describes the artifact honestly, so the shape check is the thing /// under test rather than a lie in the payload. @@ -181,7 +232,7 @@ impl ViewPipeline { bytes.truncate(bytes.len() - self.descriptor.row_stride as usize); let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?; self.renders.bump(); - self.frames.push_back((boundary, artifact)); + self.frames.push_back((boundary, counter, artifact)); while self.frames.len() > self.descriptor.observation_delay_steps as usize + 2 { self.frames.pop_front(); } @@ -198,8 +249,8 @@ impl ViewPipeline { pub fn frame_produced_at(&self, produced: u64) -> Option<(ViewRef, flybus::Artifact)> { self.frames .iter() - .find(|(step, _)| *step == produced) - .map(|(step, artifact)| { + .find(|(step, _, _)| *step == produced) + .map(|(step, _, artifact)| { ( ViewRef { view_id: self.descriptor.view_id.clone(), @@ -257,6 +308,52 @@ impl AudioSource { source } + /// The exact state a capture recorded: sample position, waveform phase and the + /// unconsumed fraction of a frame. + /// + /// Restoring the position alone would restart the waveform and round the remainder away, + /// which is a resample the restore rules refuse. The first chunk of the new epoch marks + /// the discontinuity the recovery established. + pub fn restored_from( + descriptor: AudioDescriptor, + next_sample: u64, + phase: u64, + accumulator: u128, + denominator: u128, + ) -> DomainResult { + if denominator == 0 { + return Err(DomainError::invalid( + "audio: a captured accumulator denominator of zero", + )); + } + if accumulator >= denominator { + return Err(DomainError::invalid( + "audio: a captured accumulator is not below one whole frame", + )); + } + if phase >= descriptor.sample_rate { + return Err(DomainError::invalid( + "audio: a captured phase is not below the sample rate", + )); + } + let mut source = AudioSource::new(descriptor, next_sample); + source.discontinuous = true; + source.phase = phase; + source.accumulator = accumulator; + source.denominator = denominator; + Ok(source) + } + + /// The waveform phase, for a capture. + pub fn phase(&self) -> u64 { + self.phase + } + + /// The unconsumed fraction of a frame and the denominator it is over, for a capture. + pub fn accumulator(&self) -> (u128, u128) { + (self.accumulator, self.denominator) + } + pub fn descriptor(&self) -> &AudioDescriptor { &self.descriptor } @@ -439,18 +536,47 @@ pub fn check_required_views( Ok(()) } -/// Every declared audio stream produces exactly one chunk per transition. +/// Where an observation came from. +/// +/// `state-media-v1` section 2 makes a chunk the audio of an *interval*, so whether an +/// observation must carry one is a question about its provenance and not about its boundary +/// number. MEDIA-01 wrote the rule as "boundary 0 carries no chunk", which is true of the one +/// observation that slice could produce without a transition and false of the other one: +/// `State.ActivateRestore` installs a coherent observation at boundary `k` without advancing +/// gameplay, and it covers no interval either. Naming the provenance is the fix; exempting +/// the restored observation from the validator instead would have left "must a chunk exist" +/// unanswered exactly where a stale chunk would do the most damage. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum ObservationOrigin { + /// The observation a completed transition produced. Its interval has audio. + Transition, + /// An observation established at a boundary without running a transition: + /// `Environment.Initialize`'s `O[0]` and `State.ActivateRestore`'s restored observation. + /// It covers no interval, so it carries no chunk and one in it is refused. + Installed, +} + +/// Every declared audio stream produces exactly one chunk per transition, and none at all in +/// an observation that is not one. /// /// The contract states the shape and the ordering of chunks, not whether one has to exist, so /// this is MEDIA-01's choice and it is deliberate: a session that tolerates a silently missing /// chunk cannot tell "this world produced no audio for this interval" from "the chunk was -/// lost", and the second is the case the retention rules care about. Boundary 0 has no -/// preceding interval and so carries no chunk. +/// lost", and the second is the case the retention rules care about. The mirror of that, which +/// STATE-01 needs, is that an installed observation carrying a chunk is a stale chunk being +/// offered as current, and is refused for the same reason. pub fn check_required_audio( descriptor: &EnvironmentDescriptor, observation: &WorldObservation, + origin: ObservationOrigin, ) -> DomainResult<()> { - if observation.boundary == 0 { + if origin == ObservationOrigin::Installed { + if let Some(chunk) = observation.audio.first() { + return Err(media_error(format!( + "audio stream {} produced a chunk for an observation that ran no transition", + chunk.stream_id + ))); + } return Ok(()); } for stream in &descriptor.audio { diff --git a/services/flysim/crates/fly-session/src/state.rs b/services/flysim/crates/fly-session/src/state.rs new file mode 100644 index 0000000..aa42d3c --- /dev/null +++ b/services/flysim/crates/fly-session/src/state.rs @@ -0,0 +1,1558 @@ +//! STATE-01: the coherent all-participant checkpoint store over the `FLYSESS1` envelope. +//! +//! [`fly_session_types::checkpoint`] owns the byte layout. This module owns everything +//! `checkpoint-envelope-v1` section 7 defers to this slice: generations, rotation, the store +//! manifest and its durable commit point, the bounded capture queue, the compatibility +//! comparison and the group fence. +//! +//! ```text +//! State.Capture ──> every participant, at one committed boundary +//! ──> one envelope: manifest + one payload per participant and per +//! coordinator-owned ledger +//! writer ──> temp, fsync, rename, fsync dir, then the store manifest the same way +//! ^^^^ the store manifest rename is the durable commit point +//! State.StageRestore ──> validated into replacement state, once-only token +//! State.ActivateRestore ──> installed under the new epoch, without a tick +//! ``` +//! +//! Nothing here is best-effort. A saturated queue is a named `BUSY` refusal taken *before* a +//! capture is requested; a lost save reply is an explicit outcome that leaves durable +//! metadata where it was; an incompatible checkpoint names the field that differs; and an +//! unreferenced generation is never a restore candidate. +//! +//! `FLYSIM01` (`crates/flybrain-core/src/envelope.rs`, read by the legacy flysim store) is a +//! different format with a different magic and a different reader, and nothing here touches +//! it. + +use std::collections::VecDeque; +use std::io::Write as _; +use std::path::{Path, PathBuf}; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; + +use serde_json::{Map, Value, json}; + +use fly_session_types::checkpoint::{self, Envelope}; + +// `crate::types` is this crate's facade over the shared `fly-session-types` crate; the glob +// keeps the contract's own names in sight instead of restating them. +use crate::types::*; + +/// The state-format identity this slice writes and reads. A checkpoint recorded under any +/// other one is refused by name rather than attempted. +pub const STATE_FORMAT_ID: &str = "flysess-1"; + +/// The worker capability `state-media-v1` section 5 makes the State methods conditional on. +pub const CHECKPOINT_CAPABILITY: &str = "checkpoint-v1"; + +/// The store manifest's own version. It is not the envelope version. +pub const STORE_MANIFEST_VERSION: u32 = 1; + +/// The file the store manifest is committed to. Its rename is the durable commit point. +pub const STORE_MANIFEST_FILE: &str = "manifest.json"; + +/// The content type a checkpoint payload travels under as a bus artifact. +pub const PAYLOAD_CONTENT_TYPE: &str = "application/x-fly-checkpoint-payload"; + +/// The attachment name a checkpoint payload travels under, in both directions. +pub const PAYLOAD_ATTACHMENT: &str = "payload"; + +/// Seals one checkpoint payload as an immutable artifact against its content digest. +/// +/// `state-media-v1` section 1 makes a digest mandatory on checkpoint payloads, so this seals +/// with one rather than computing it afterwards: a store that wrote the wrong bytes finds out +/// here and not at the next restore. +pub async fn seal_payload( + client: &flybus::Client, + bytes: &[u8], + digest: &Digest, +) -> DomainResult { + let mut writer = client + .artifacts() + .allocate(bytes.len() as u64, PAYLOAD_CONTENT_TYPE) + .await + .map_err(|e| store_error(format!("payload allocate: {}", e.message)))?; + writer + .write_all(bytes) + .map_err(|e| store_error(format!("payload write: {e}")))?; + let artifact = writer + .seal_with_digest(Some(digest.clone())) + .await + .map_err(|e| store_error(format!("payload seal: {}", e.message)))?; + match &artifact.reference().digest { + Some(sealed) if sealed == digest => Ok(artifact), + _ => Err(store_error("a sealed checkpoint payload has no matching content digest")), + } +} + +fn store_error(what: impl std::fmt::Display) -> DomainError { + DomainError::new(ErrorCode::BackendFailure, what, MutationCertainty::Unknown) +} + +fn incompatible(what: impl std::fmt::Display) -> DomainError { + DomainError::before(ErrorCode::IncompatibleState, what) +} + +// ---------------------------------------------------------------------------------------------- +// Payload names + +/// The payload name one agent's captured state is filed under. +pub fn agent_payload(agent_id: &str) -> String { + format!("agent-{agent_id}") +} + +/// The payload name one agent's action-executor state is filed under. +pub fn executor_payload(agent_id: &str) -> String { + format!("executor-{agent_id}") +} + +/// The environment's payload name. +pub const WORLD_PAYLOAD: &str = "world"; +/// The task ledger's payload name. +pub const TASK_LEDGER_PAYLOAD: &str = "task-ledger"; +/// The prior world inspection's payload name. +pub const PRIOR_INSPECTION_PAYLOAD: &str = "prior-inspection"; +/// The coordinator's admission state and event watermarks. +pub const ADMISSION_PAYLOAD: &str = "coordinator-admission"; + +// ---------------------------------------------------------------------------------------------- +// Compatibility + +/// The compatibility identities `state-media-v1` section 4 requires a checkpoint to record. +/// +/// Each one is a separate field on purpose: a restore that fails says which identity differs, +/// instead of reporting one opaque digest mismatch. The synthetic composition maps them onto +/// the identities it actually has: +/// +/// | Field | Where it comes from | +/// | --- | --- | +/// | `backend_digest` | the environment descriptor's backend identity | +/// | `content_digest` | the environment descriptor's content identity | +/// | `patch_digest` | the environment descriptor's resolved configuration identity | +/// | `controller_digest` | every declared port's controller schema, in descriptor order | +/// | `parser_digest` | the inspection schema and the task schema that reads it | +/// | `state_format_id` | [`STATE_FORMAT_ID`] | +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Compatibility { + pub backend_digest: Digest, + pub content_digest: Digest, + pub patch_digest: Digest, + pub controller_digest: Digest, + pub parser_digest: Digest, + pub state_format_id: Id, +} + +impl Compatibility { + /// The compatibility of one world, from the descriptor it advertises. + /// + /// Every identity comes from the descriptor, which is what lets the environment compute + /// exactly this digest for its own capture while the coordinator computes it for the + /// composition. The task's own schema is deliberately not in here: the captured task + /// ledger carries its schema and refuses another one, so folding it in would put one + /// identity in two places. + pub fn of(descriptor: &EnvironmentDescriptor) -> Compatibility { + let controllers = Value::Array( + descriptor + .ports + .iter() + .map(|port| { + json!({ + "portId": port.port_id.as_str(), + "controls": port.controls.to_json(), + }) + }) + .collect(), + ); + let parser = json!({ + "inspectionSchema": descriptor.inspection_schema.to_json(), + }); + Compatibility { + backend_digest: descriptor.backend_digest.clone(), + content_digest: descriptor.content_digest.clone(), + patch_digest: descriptor.configuration_digest.clone(), + controller_digest: digest_of(&controllers) + .expect("a validated controller schema canonicalizes"), + parser_digest: digest_of(&parser).expect("a validated schema reference canonicalizes"), + state_format_id: id(STATE_FORMAT_ID), + } + } + + pub fn to_json(&self) -> Value { + json!({ + "backendDigest": self.backend_digest.as_str(), + "contentDigest": self.content_digest.as_str(), + "patchDigest": self.patch_digest.as_str(), + "controllerDigest": self.controller_digest.as_str(), + "parserDigest": self.parser_digest.as_str(), + "stateFormatId": self.state_format_id.as_str(), + }) + } + + pub fn from_json(value: &Value) -> Result { + let digest = |key: &str| -> Result { + let text = value + .get(key) + .and_then(Value::as_str) + .ok_or_else(|| format!("compatibility: {key} is missing or not a string"))?; + if !is_digest(text) { + return Err(format!("compatibility: {key} is not a digest")); + } + Ok(text.to_owned()) + }; + let state_format_id = value + .get("stateFormatId") + .and_then(Value::as_str) + .ok_or_else(|| "compatibility: stateFormatId is missing".to_owned())?; + Ok(Compatibility { + backend_digest: digest("backendDigest")?, + content_digest: digest("contentDigest")?, + patch_digest: digest("patchDigest")?, + controller_digest: digest("controllerDigest")?, + parser_digest: digest("parserDigest")?, + state_format_id: parse_id(state_format_id) + .map_err(|e| format!("compatibility: stateFormatId {e}"))?, + }) + } + + /// The single digest a participant echoes in its capture and its stage request. + pub fn digest(&self) -> Digest { + digest_of(&self.to_json()).expect("a compatibility block canonicalizes") + } + + /// Names the first identity that differs. There is no tolerance and no "close enough". + pub fn compare(&self, live: &Compatibility) -> Result<(), String> { + for (field, recorded, current) in [ + ("stateFormatId", &self.state_format_id, &live.state_format_id), + ("backend", &self.backend_digest, &live.backend_digest), + ("content", &self.content_digest, &live.content_digest), + ("patch", &self.patch_digest, &live.patch_digest), + ("controller", &self.controller_digest, &live.controller_digest), + ("parser", &self.parser_digest, &live.parser_digest), + ] { + if recorded != current { + return Err(format!( + "the checkpoint's {field} identity {recorded} is not this composition's {current}" + )); + } + } + Ok(()) + } +} + +// ---------------------------------------------------------------------------------------------- +// The store manifest + +/// One committed generation, as the store manifest lists it. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct GenerationRecord { + pub checkpoint_id: Id, + /// The generation's file name inside the store directory. + pub file: String, + pub session_id: Id, + pub epoch: Id, + pub episode_id: Id, + pub boundary: u64, + pub compatibility_digest: Digest, + /// The SHA-256 of the whole envelope file, so a store can compare without opening it. + pub envelope_digest: Digest, + pub byte_length: u64, +} + +impl GenerationRecord { + fn to_json(&self) -> Value { + json!({ + "checkpointId": self.checkpoint_id.as_str(), + "file": self.file.as_str(), + "sessionId": self.session_id.as_str(), + "epoch": self.epoch.as_str(), + "episodeId": self.episode_id.as_str(), + "boundary": self.boundary.to_string(), + "compatibilityDigest": self.compatibility_digest.as_str(), + "envelopeDigest": self.envelope_digest.as_str(), + "byteLength": self.byte_length.to_string(), + }) + } + + fn from_json(value: &Value) -> Result { + let text = |key: &str| -> Result { + value + .get(key) + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| format!("store manifest: a generation has no {key}")) + }; + let number = |key: &str| -> Result { + text(key)? + .parse::() + .map_err(|_| format!("store manifest: {key} is not a canonical U64")) + }; + Ok(GenerationRecord { + checkpoint_id: parse_id(&text("checkpointId")?)?, + file: text("file")?, + session_id: parse_id(&text("sessionId")?)?, + epoch: parse_id(&text("epoch")?)?, + episode_id: parse_id(&text("episodeId")?)?, + boundary: number("boundary")?, + compatibility_digest: text("compatibilityDigest")?, + envelope_digest: text("envelopeDigest")?, + byte_length: number("byteLength")?, + }) + } +} + +/// The store's durable metadata: which generations exist and how far durability has reached. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct StoreManifest { + pub generations: Vec, + /// The newest committed checkpoint, which is the high-water mark. `None` until the first + /// durable commit; that is "nothing has been committed", not a default. + pub high_water: Option, +} + +impl StoreManifest { + fn to_json(&self) -> Value { + json!({ + "storeManifestVersion": STORE_MANIFEST_VERSION, + "stateFormatId": STATE_FORMAT_ID, + "highWater": self.high_water.as_ref().map_or(Value::Null, |h| h.as_str().into()), + "generations": Value::Array(self.generations.iter().map(GenerationRecord::to_json).collect()), + }) + } + + fn from_json(value: &Value) -> Result { + let version = value + .get("storeManifestVersion") + .and_then(Value::as_u64) + .ok_or_else(|| "store manifest: no storeManifestVersion".to_owned())?; + if version != u64::from(STORE_MANIFEST_VERSION) { + return Err(format!("store manifest: unsupported version {version}")); + } + match value.get("stateFormatId").and_then(Value::as_str) { + Some(STATE_FORMAT_ID) => {} + Some(other) => { + return Err(format!("store manifest: state format {other} is not {STATE_FORMAT_ID}")); + } + None => return Err("store manifest: no stateFormatId".to_owned()), + } + let generations = value + .get("generations") + .and_then(Value::as_array) + .ok_or_else(|| "store manifest: generations must be an array".to_owned())? + .iter() + .map(GenerationRecord::from_json) + .collect::, _>>()?; + let high_water = match value.get("highWater") { + Some(Value::Null) | None => None, + Some(Value::String(s)) => Some(parse_id(s)?), + Some(_) => return Err("store manifest: highWater is neither null nor an Id".to_owned()), + }; + if let Some(mark) = &high_water + && !generations.iter().any(|g| g.checkpoint_id == *mark) + { + return Err("store manifest: the high-water mark names no listed generation".to_owned()); + } + Ok(StoreManifest { generations, high_water }) + } +} + +// ---------------------------------------------------------------------------------------------- +// The store + +/// How many committed generations the store keeps. +#[derive(Clone, Copy, Debug)] +pub struct StoreConfig { + /// Generations retained after a commit. The oldest are dropped, and only once the + /// manifest that no longer references them is itself committed. + pub keep_generations: usize, +} + +impl Default for StoreConfig { + fn default() -> StoreConfig { + StoreConfig { keep_generations: 3 } + } +} + +/// Deliberate durable-write faults, for the failure rows this slice has to demonstrate. +#[derive(Clone, Debug, Default)] +pub struct StoreFaults { + /// Stop after the generation file has been renamed and before the store manifest is + /// committed. The generation is then an unreferenced file, which is never a candidate. + pub stop_before_manifest_commit: bool, +} + +/// The durable checkpoint store: generation files, one store manifest and the commit order of +/// `checkpoint-envelope-v1` section 5. +pub struct CheckpointStore { + root: PathBuf, + config: StoreConfig, + manifest: StoreManifest, + faults: StoreFaults, + commits: u64, + /// Generations the committed manifest no longer references and whose files could not be + /// removed. + /// + /// Rotation happens after the durable commit point, so a file that will not unlink is a + /// leaked file and never a lost checkpoint. It is recorded rather than swallowed, because + /// a store that keeps failing to rotate is filling a disk quietly. + unrotated: Vec, +} + +impl CheckpointStore { + /// Opens or creates a store at `root`, reading whatever it already committed. + /// + /// A directory with no manifest is an empty store: nothing has been committed there yet. + /// A manifest that cannot be read is a failure, not an empty store. + pub fn open(root: impl Into, config: StoreConfig) -> DomainResult { + let root: PathBuf = root.into(); + std::fs::create_dir_all(&root) + .map_err(|e| store_error(format!("checkpoint store {}: {e}", root.display())))?; + let path = root.join(STORE_MANIFEST_FILE); + let manifest = match std::fs::read(&path) { + Ok(bytes) => { + let value: Value = serde_json::from_slice(&bytes) + .map_err(|e| store_error(format!("store manifest: {e}")))?; + StoreManifest::from_json(&value).map_err(store_error)? + } + Err(e) if e.kind() == std::io::ErrorKind::NotFound => StoreManifest::default(), + Err(e) => return Err(store_error(format!("store manifest: {e}"))), + }; + Ok(CheckpointStore { + root, + config, + manifest, + faults: StoreFaults::default(), + commits: 0, + unrotated: Vec::new(), + }) + } + + pub fn root(&self) -> &Path { + &self.root + } + + /// Re-reads the store manifest from disk. + /// + /// The store manifest is the durable metadata, and this session is not necessarily the + /// only thing that has ever written it: a previous run, a repair or an operator may have + /// committed or removed a generation. Reading it again is how a store finds that out, + /// rather than trusting a copy it happens to be holding. + pub fn reload(&mut self) -> DomainResult<()> { + let reopened = CheckpointStore::open(self.root.clone(), self.config)?; + self.manifest = reopened.manifest; + Ok(()) + } + + pub fn manifest(&self) -> &StoreManifest { + &self.manifest + } + + pub fn faults_mut(&mut self) -> &mut StoreFaults { + &mut self.faults + } + + /// How many durable commits this store has completed. + pub fn commits(&self) -> u64 { + self.commits + } + + /// Generations the committed manifest dropped whose files are still on disk. + pub fn unrotated(&self) -> &[String] { + &self.unrotated + } + + /// The committed generation with this checkpoint id, if the manifest lists it. + /// + /// This is the query a coordinator uses to resolve a save whose reply it never saw: it + /// asks the durable metadata about the *same* operation rather than saving again. + pub fn lookup(&self, checkpoint_id: &Id) -> Option<&GenerationRecord> { + self.manifest + .generations + .iter() + .find(|g| g.checkpoint_id == *checkpoint_id) + } + + /// The newest committed generation, or an explicit refusal when nothing is committed. + pub fn high_water(&self) -> Option<&GenerationRecord> { + let mark = self.manifest.high_water.as_ref()?; + self.lookup(mark) + } + + /// Selects a restore candidate: the named generation, or the high-water one. + pub fn select(&self, checkpoint_id: Option<&Id>) -> DomainResult { + match checkpoint_id { + Some(wanted) => self.lookup(wanted).cloned().ok_or_else(|| { + incompatible(format!( + "the store has no committed generation {wanted}; an unreferenced \ +temporary is never a restore candidate" + )) + }), + None => self.high_water().cloned().ok_or_else(|| { + incompatible("the store has committed no checkpoint to restore from") + }), + } + } + + /// Reads one committed generation back and validates the whole envelope. + pub fn read(&self, record: &GenerationRecord) -> DomainResult { + let path = self.root.join(&record.file); + let bytes = std::fs::read(&path) + .map_err(|e| store_error(format!("generation {}: {e}", record.file)))?; + if bytes.len() as u64 != record.byte_length { + return Err(incompatible(format!( + "generation {} is {} bytes; the store manifest records {}", + record.file, + bytes.len(), + record.byte_length + ))); + } + if digest_of_bytes(&bytes) != record.envelope_digest { + return Err(incompatible(format!( + "generation {} does not match the digest the store manifest records", + record.file + ))); + } + let envelope = checkpoint::decode(&bytes) + .map_err(|e| incompatible(format!("generation {}: {}", record.file, e.0)))?; + checkpoint::validate_manifest(&envelope) + .map_err(|e| incompatible(format!("generation {}: {}", record.file, e.0)))?; + Ok(envelope) + } + + /// The durable commit sequence of `checkpoint-envelope-v1` section 5, in that order. + /// + /// Blocking by construction: it fsyncs. The writer runs it off the session's runtime. + fn commit(&mut self, record: GenerationRecord, bytes: &[u8]) -> DomainResult<()> { + let temporary = self.root.join(format!("tmp-{}.flysess", record.checkpoint_id)); + let final_path = self.root.join(&record.file); + // 1. write the envelope to a temporary generation file, 2. fsync it + { + let mut file = std::fs::File::create(&temporary) + .map_err(|e| store_error(format!("generation temporary: {e}")))?; + file.write_all(bytes) + .map_err(|e| store_error(format!("generation temporary: {e}")))?; + file.sync_all() + .map_err(|e| store_error(format!("generation fsync: {e}")))?; + } + // 3. rename it to its final generation name, 4. fsync the store directory + std::fs::rename(&temporary, &final_path) + .map_err(|e| store_error(format!("generation rename: {e}")))?; + sync_dir(&self.root)?; + if self.faults.stop_before_manifest_commit { + // The generation file exists and nothing references it. It is not a restore + // candidate and the high-water mark has not moved. + return Err(store_error( + "injected failure after the generation was renamed and before the store \ +manifest was committed", + )); + } + // 5. write the store manifest to its own temporary, fsync, rename, fsync the directory. + let mut next = self.manifest.clone(); + next.generations.retain(|g| g.checkpoint_id != record.checkpoint_id); + next.generations.push(record.clone()); + next.high_water = Some(record.checkpoint_id.clone()); + let dropped = if next.generations.len() > self.config.keep_generations { + let excess = next.generations.len() - self.config.keep_generations; + next.generations.drain(..excess).collect::>() + } else { + Vec::new() + }; + let text = canonicalize(&next.to_json()) + .map_err(|e| store_error(format!("store manifest: {}", e.0)))?; + let manifest_temporary = self.root.join("tmp-manifest.json"); + { + let mut file = std::fs::File::create(&manifest_temporary) + .map_err(|e| store_error(format!("store manifest temporary: {e}")))?; + file.write_all(text.as_bytes()) + .map_err(|e| store_error(format!("store manifest temporary: {e}")))?; + file.sync_all() + .map_err(|e| store_error(format!("store manifest fsync: {e}")))?; + } + std::fs::rename(&manifest_temporary, self.root.join(STORE_MANIFEST_FILE)) + .map_err(|e| store_error(format!("store manifest rename: {e}")))?; + sync_dir(&self.root)?; + // Past the durable commit point. Rotation removes only files the committed manifest + // no longer references. + self.manifest = next; + self.commits += 1; + for old in dropped { + if std::fs::remove_file(self.root.join(&old.file)).is_err() { + self.unrotated.push(old.file); + } + } + Ok(()) + } +} + +fn sync_dir(path: &Path) -> DomainResult<()> { + let dir = std::fs::File::open(path) + .map_err(|e| store_error(format!("store directory {}: {e}", path.display())))?; + dir.sync_all() + .map_err(|e| store_error(format!("store directory fsync: {e}"))) +} + +// ---------------------------------------------------------------------------------------------- +// The checkpoint manifest this slice writes and reads + +/// One agent's row in a checkpoint manifest. +#[derive(Clone, Debug, PartialEq)] +pub struct AgentEntry { + pub agent_id: Id, + pub profile_digest: Digest, + pub dataset_digest: Digest, + pub model_version: String, + pub plasticity_version: String, + pub seed: i32, + pub brain_ticks: u64, + pub remainder: RationalNs, + pub payload: String, +} + +impl AgentEntry { + fn to_json(&self) -> Value { + json!({ + "agentId": self.agent_id.as_str(), + "profileDigest": self.profile_digest.as_str(), + "datasetDigest": self.dataset_digest.as_str(), + "modelVersion": self.model_version.as_str(), + "plasticityVersion": self.plasticity_version.as_str(), + "seed": self.seed, + "brainTicks": self.brain_ticks.to_string(), + "remainder": self.remainder.to_json(), + "payload": self.payload.as_str(), + }) + } + + fn from_json(value: &Value) -> Result { + let text = |key: &str| -> Result { + value + .get(key) + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| format!("checkpoint manifest: an agent row has no {key}")) + }; + let seed = value + .get("seed") + .and_then(Value::as_i64) + .ok_or_else(|| "checkpoint manifest: an agent row has no seed".to_owned())?; + let seed = i32::try_from(seed) + .map_err(|_| "checkpoint manifest: a seed is outside i32".to_owned())?; + let remainder = RationalNs::from_json( + value + .get("remainder") + .ok_or_else(|| "checkpoint manifest: an agent row has no remainder".to_owned())?, + ) + .map_err(|e| format!("checkpoint manifest: remainder: {}", e.0))?; + Ok(AgentEntry { + agent_id: parse_id(&text("agentId")?)?, + profile_digest: text("profileDigest")?, + dataset_digest: text("datasetDigest")?, + model_version: text("modelVersion")?, + plasticity_version: text("plasticityVersion")?, + seed, + brain_ticks: text("brainTicks")? + .parse() + .map_err(|_| "checkpoint manifest: brainTicks is not a canonical U64".to_owned())?, + remainder, + payload: text("payload")?, + }) + } +} + +/// The coordinator-owned state a checkpoint records, each as a payload name. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CoordinatorEntry { + pub task_ledger: String, + pub prior_inspection: String, + pub executor_state: Vec<(Id, String)>, + pub admission_state: String, + pub event_watermarks: EventWatermarks, +} + +/// The event identity a resumed epoch continues from. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct EventWatermarks { + /// The highest source step any recorded event belongs to. + pub last_source_step: u64, + /// How many events the ledger has issued. + pub issued: u64, +} + +impl EventWatermarks { + fn to_json(&self) -> Value { + json!({ + "lastSourceStep": self.last_source_step.to_string(), + "issued": self.issued.to_string(), + }) + } + + fn from_json(value: &Value) -> Result { + let number = |key: &str| -> Result { + value + .get(key) + .and_then(Value::as_str) + .ok_or_else(|| format!("checkpoint manifest: eventWatermarks has no {key}"))? + .parse() + .map_err(|_| format!("checkpoint manifest: {key} is not a canonical U64")) + }; + Ok(EventWatermarks { + last_source_step: number("lastSourceStep")?, + issued: number("issued")?, + }) + } +} + +impl CoordinatorEntry { + fn to_json(&self) -> Value { + json!({ + "taskLedger": self.task_ledger.as_str(), + "priorInspection": self.prior_inspection.as_str(), + "executorState": Value::Array( + self.executor_state + .iter() + .map(|(agent_id, payload)| json!({ + "agentId": agent_id.as_str(), + "payload": payload.as_str(), + })) + .collect(), + ), + "admissionState": self.admission_state.as_str(), + "eventWatermarks": self.event_watermarks.to_json(), + }) + } + + fn from_json(value: &Value) -> Result { + let text = |key: &str| -> Result { + value + .get(key) + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| format!("checkpoint manifest: coordinator has no {key}")) + }; + let executors = value + .get("executorState") + .and_then(Value::as_array) + .ok_or_else(|| "checkpoint manifest: executorState must be an array".to_owned())?; + let mut executor_state = Vec::with_capacity(executors.len()); + for entry in executors { + let agent_id = entry + .get("agentId") + .and_then(Value::as_str) + .ok_or_else(|| "checkpoint manifest: an executor row has no agentId".to_owned())?; + let payload = entry + .get("payload") + .and_then(Value::as_str) + .ok_or_else(|| "checkpoint manifest: an executor row has no payload".to_owned())?; + executor_state.push((parse_id(agent_id)?, payload.to_owned())); + } + let watermarks = value + .get("eventWatermarks") + .ok_or_else(|| "checkpoint manifest: coordinator has no eventWatermarks".to_owned())?; + Ok(CoordinatorEntry { + task_ledger: text("taskLedger")?, + prior_inspection: text("priorInspection")?, + executor_state, + admission_state: text("admissionState")?, + event_watermarks: EventWatermarks::from_json(watermarks)?, + }) + } +} + +/// The environment's row: which worker the world belonged to and which payload holds it. +/// +/// `checkpoint-envelope-v1` section 3 named a holder for every payload except the world's; +/// the 2026-09-22 amendment to that section adds this one. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct EnvironmentEntry { + pub worker_id: Id, + pub payload: String, +} + +impl EnvironmentEntry { + fn to_json(&self) -> Value { + json!({ + "workerId": self.worker_id.as_str(), + "payload": self.payload.as_str(), + }) + } + + fn from_json(value: &Value) -> Result { + let text = |key: &str| -> Result { + value + .get(key) + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| format!("checkpoint manifest: environment has no {key}")) + }; + Ok(EnvironmentEntry { + worker_id: parse_id(&text("workerId")?)?, + payload: text("payload")?, + }) + } +} + +/// A complete checkpoint manifest, in the field names `checkpoint-envelope-v1` section 3 sets. +#[derive(Clone, Debug, PartialEq)] +pub struct CheckpointManifest { + pub checkpoint_id: Id, + pub source_scope: Scope, + pub episode_id: Id, + pub world_time: RationalNs, + pub scheduler_id: String, + pub composition_digest: Digest, + pub port_map: Vec<(Id, Id)>, + pub compatibility: Compatibility, + pub agents: Vec, + pub coordinator: CoordinatorEntry, + pub environment: EnvironmentEntry, + /// External-helper state required for exact resume, as payload names. The synthetic + /// composition has no external helper, so it records an empty list rather than omitting + /// the field: "no helper" is a statement, not a missing one. + pub helper_state: Vec, + pub payloads: Vec<(String, u64, Digest)>, +} + +impl CheckpointManifest { + pub fn to_json(&self) -> Value { + json!({ + "envelopeVersion": checkpoint::VERSION, + "checkpointId": self.checkpoint_id.as_str(), + "sourceScope": self.source_scope.to_json(), + "episodeId": self.episode_id.as_str(), + "worldTime": self.world_time.to_json(), + "schedulerId": self.scheduler_id.as_str(), + "compositionDigest": self.composition_digest.as_str(), + "portMap": Value::Array( + self.port_map + .iter() + .map(|(port_id, agent_id)| json!({ + "portId": port_id.as_str(), + "agentId": agent_id.as_str(), + })) + .collect(), + ), + "compatibility": self.compatibility.to_json(), + "agents": Value::Array(self.agents.iter().map(AgentEntry::to_json).collect()), + "coordinator": self.coordinator.to_json(), + "environment": self.environment.to_json(), + "helperState": Value::Array( + self.helper_state.iter().map(|n| Value::String(n.clone())).collect(), + ), + "payloads": Value::Array( + self.payloads + .iter() + .map(|(name, length, digest)| json!({ + "name": name.as_str(), + "byteLength": length.to_string(), + "digest": digest.as_str(), + })) + .collect(), + ), + }) + } + + /// Reads one back, refusing an incomplete manifest rather than filling anything in. + pub fn from_json(value: &Value) -> Result { + for field in checkpoint::REQUIRED_MANIFEST_FIELDS { + if value.get(*field).is_none() { + return Err(format!("checkpoint manifest: missing {field:?}")); + } + } + let text = |key: &str| -> Result { + value + .get(key) + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| format!("checkpoint manifest: {key} is missing or not a string")) + }; + let source_scope = Scope::from_json(&value["sourceScope"]) + .map_err(|e| format!("checkpoint manifest: sourceScope: {}", e.0))?; + let world_time = RationalNs::from_json(&value["worldTime"]) + .map_err(|e| format!("checkpoint manifest: worldTime: {}", e.0))?; + let mut port_map = Vec::new(); + for entry in value["portMap"] + .as_array() + .ok_or_else(|| "checkpoint manifest: portMap must be an array".to_owned())? + { + let port_id = entry + .get("portId") + .and_then(Value::as_str) + .ok_or_else(|| "checkpoint manifest: a port map row has no portId".to_owned())?; + let agent_id = entry + .get("agentId") + .and_then(Value::as_str) + .ok_or_else(|| "checkpoint manifest: a port map row has no agentId".to_owned())?; + port_map.push((parse_id(port_id)?, parse_id(agent_id)?)); + } + let agents = value["agents"] + .as_array() + .ok_or_else(|| "checkpoint manifest: agents must be an array".to_owned())? + .iter() + .map(AgentEntry::from_json) + .collect::, _>>()?; + let mut helper_state = Vec::new(); + for entry in value["helperState"] + .as_array() + .ok_or_else(|| "checkpoint manifest: helperState must be an array".to_owned())? + { + helper_state.push( + entry + .as_str() + .ok_or_else(|| "checkpoint manifest: a helper state entry is not a payload name".to_owned())? + .to_owned(), + ); + } + let mut payloads = Vec::new(); + for entry in value["payloads"] + .as_array() + .ok_or_else(|| "checkpoint manifest: payloads must be an array".to_owned())? + { + let name = entry + .get("name") + .and_then(Value::as_str) + .ok_or_else(|| "checkpoint manifest: a payload row has no name".to_owned())?; + let length: u64 = entry + .get("byteLength") + .and_then(Value::as_str) + .ok_or_else(|| "checkpoint manifest: a payload row has no byteLength".to_owned())? + .parse() + .map_err(|_| "checkpoint manifest: byteLength is not a canonical U64".to_owned())?; + let digest = entry + .get("digest") + .and_then(Value::as_str) + .ok_or_else(|| "checkpoint manifest: a payload row has no digest".to_owned())?; + payloads.push((name.to_owned(), length, digest.to_owned())); + } + Ok(CheckpointManifest { + checkpoint_id: parse_id(&text("checkpointId")?)?, + source_scope, + episode_id: parse_id(&text("episodeId")?)?, + world_time, + scheduler_id: text("schedulerId")?, + composition_digest: text("compositionDigest")?, + port_map, + compatibility: Compatibility::from_json(&value["compatibility"])?, + agents, + coordinator: CoordinatorEntry::from_json(&value["coordinator"])?, + environment: EnvironmentEntry::from_json( + value + .get("environment") + .ok_or_else(|| "checkpoint manifest: missing \"environment\"".to_owned())?, + )?, + helper_state, + payloads, + }) + } + + /// The payload name this manifest files one participant's state under. + pub fn payload_of(&self, worker_id: &Id) -> Option<&str> { + if self.environment.worker_id == *worker_id { + return Some(self.environment.payload.as_str()); + } + self.agents + .iter() + .find(|a| a.agent_id == *worker_id) + .map(|a| a.payload.as_str()) + } +} + +// ---------------------------------------------------------------------------------------------- +// The bounded writer + +/// One participant's captured payload, with the owned handle the writer keeps until the bytes +/// are committed or the job fails. +pub struct CapturedPayload { + pub name: String, + pub artifact: flybus::Artifact, + pub byte_length: u64, + pub digest: Digest, +} + +/// What a coordinator hands the writer once every participant has captured. +pub struct CaptureSubmission { + pub checkpoint_id: Id, + pub boundary: u64, + pub session_id: Id, + pub epoch: Id, + pub episode_id: Id, + pub compatibility_digest: Digest, + pub manifest: Value, + pub payloads: Vec, + /// A hot checkpoint may replace a queued hot checkpoint, releasing its holds. A durable + /// one never is: the retention table coalesces only queued replaceable captures. + pub replaceable: bool, +} + +impl CaptureSubmission { + fn byte_length(&self) -> u64 { + self.payloads.iter().map(|p| p.byte_length).sum() + } +} + +/// How one durable save ended. Every variant is a statement; none of them is a default. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum SaveOutcome { + /// Past the store manifest rename. This is the only variant that is a saved + /// acknowledgment and the only one that moves a high-water mark. + Committed { checkpoint_id: Id, boundary: u64, file: String }, + /// The write failed and its owned captures were released under the retry policy. It + /// never reports false durability. + Failed { checkpoint_id: Id, reason: String }, + /// A later replaceable capture took this one's place in the queue before it was written. + Superseded { checkpoint_id: Id, by: Id }, + /// The writer finished this job and its reply channel was gone before the outcome could + /// be delivered. The write is over and its result is unknown from here. + ReplyLost { checkpoint_id: Id }, + /// The caller's own budget ran out while the job was still queued or being written. The + /// save is not over: it may commit after this is reported. + /// + /// It is a different fact from [`SaveOutcome::ReplyLost`] and is named separately because + /// diagnosing one as the other is exactly the implicit best-effort reading these + /// contracts refuse. Both leave durable metadata where it was, and for both the caller + /// resolves the *same* operation against the store manifest instead of saving again -- + /// but only one of them is a save that has already stopped. + DeadlineExpired { checkpoint_id: Id }, +} + +impl SaveOutcome { + pub fn checkpoint_id(&self) -> &Id { + match self { + SaveOutcome::Committed { checkpoint_id, .. } + | SaveOutcome::Failed { checkpoint_id, .. } + | SaveOutcome::Superseded { checkpoint_id, .. } + | SaveOutcome::ReplyLost { checkpoint_id } + | SaveOutcome::DeadlineExpired { checkpoint_id } => checkpoint_id, + } + } + + /// The event name this outcome publishes under. + /// + /// Only the three the writer itself produces are ever published; the two caller-side + /// outcomes are what a caller saw, not what the store did, and the store does not announce + /// them on its own topic. + pub fn event(&self) -> &'static str { + match self { + SaveOutcome::Committed { .. } => "committed", + SaveOutcome::Failed { .. } => "failed", + SaveOutcome::Superseded { .. } => "superseded", + SaveOutcome::ReplyLost { .. } | SaveOutcome::DeadlineExpired { .. } => "failed", + } + } + + /// True only past the durable commit point. + pub fn is_durable(&self) -> bool { + matches!(self, SaveOutcome::Committed { .. }) + } +} + +/// What the writer itself can produce for one job. +/// +/// The two caller-side outcomes -- a lost reply and an expired caller deadline -- are not in +/// here, because the writer cannot observe either. Keeping them out is what stops the writer's +/// own bookkeeping from carrying arms that can never run. +#[derive(Clone, Debug, PartialEq, Eq)] +enum WriteOutcome { + Committed { boundary: u64, file: String }, + Failed { reason: String }, +} + +impl WriteOutcome { + fn into_save(self, checkpoint_id: &Id) -> SaveOutcome { + match self { + WriteOutcome::Committed { boundary, file } => SaveOutcome::Committed { + checkpoint_id: checkpoint_id.clone(), + boundary, + file, + }, + WriteOutcome::Failed { reason } => SaveOutcome::Failed { + checkpoint_id: checkpoint_id.clone(), + reason, + }, + } + } +} + +/// What a failed write does with the ephemeral captures it owns. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum RetryPolicy { + /// Release the owned captures and report the failure. The default: a capture is cheap to + /// take again at the next committed boundary, and holding one is not free. + ReleaseAndReport, + /// Try the durable sequence again, up to `attempts` times in total, keeping the owned + /// captures until they are committed or the attempts are spent. + RetryThenRelease { attempts: u32 }, +} + +/// The writer's bounds. Both are finite, and they refuse at different moments. +/// +/// [`WriterConfig::queue_capacity`] is the one a capture is refused *before* it is requested: +/// [`CheckpointWriter::reserve`] takes its slot first, which is what the durable row of +/// `state-media-v1` section 3 means by rejecting before capture. The byte budget cannot work +/// that way, because how many bytes a capture is worth is not known until the participants +/// have produced it; it is checked at [`CheckpointWriter::submit`], so an oversized capture is +/// refused after it exists and before it is queued, and its payloads are released with the +/// refusal. Both are named `BUSY` refusals and neither fails the epoch. +#[derive(Clone, Copy, Debug)] +pub struct WriterConfig { + /// Outstanding coherent captures. `state-media-v1` section 3's initial session default + /// is two. Refused before a capture is requested. + pub queue_capacity: usize, + /// The total payload bytes the queue may hold. Refused at submit, once the size is known. + pub max_queued_bytes: u64, + pub retry: RetryPolicy, +} + +impl Default for WriterConfig { + fn default() -> WriterConfig { + WriterConfig { + queue_capacity: 2, + max_queued_bytes: 64 * 1024 * 1024, + retry: RetryPolicy::ReleaseAndReport, + } + } +} + +/// Deliberate writer faults, for the rows this slice has to demonstrate. +#[derive(Clone, Default)] +pub struct WriterFaults { + /// Hold every job until the gate is opened, so a test can fill the queue on purpose. + pub gate: Option>, + /// Drop this job's reply channel after the durable sequence ran, which is a lost save + /// reply. + pub drop_reply_for: Option, +} + +impl std::fmt::Debug for WriterFaults { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("WriterFaults") + .field("gate", &self.gate.is_some()) + .field("drop_reply_for", &self.drop_reply_for) + .finish() + } +} + +/// The writer's own counters, so "bounded" is something a test reads rather than believes. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct WriterStats { + pub queued: u64, + pub committed: u64, + pub failed: u64, + pub superseded: u64, + /// Submissions refused because the queue or its byte budget was full. + pub rejected: u64, + /// Checkpoint events the bus would not take. The durable outcome the caller receives is + /// the authority; a publication that failed is counted here rather than disappearing. + pub events_dropped: u64, + /// The deepest the queue ever got. + pub peak_queue: usize, + /// The most payload bytes the queue ever held. + pub peak_bytes: u64, +} + +struct Job { + submission: CaptureSubmission, + reply: tokio::sync::oneshot::Sender, + permit: tokio::sync::OwnedSemaphorePermit, +} + +struct WriterShared { + config: WriterConfig, + faults: WriterFaults, + permits: Arc, + queue: std::sync::Mutex>, + queued_bytes: AtomicU64, + stats: std::sync::Mutex, + wake: tokio::sync::Notify, + stop: AtomicBool, + store: Arc>, + events: Option<(flybus::Client, String)>, +} + +/// A queue slot, taken before a capture is requested so a saturated writer refuses early. +/// +/// `state-media-v1` section 3's durable row says to reject or defer *before capture* when +/// saturated. Holding the slot from before `State.Capture` until the outcome is delivered is +/// what makes that true rather than aspirational. +pub struct Reservation { + permit: tokio::sync::OwnedSemaphorePermit, +} + +/// The bounded checkpoint writer. +pub struct CheckpointWriter { + shared: Arc, + task: tokio::task::JoinHandle<()>, +} + +impl CheckpointWriter { + /// Starts the writer over `store`. `events` is the bus client and topic the distinct + /// captured/queued/committed/failed/superseded publications go to. + pub fn start( + store: CheckpointStore, + config: WriterConfig, + faults: WriterFaults, + events: Option<(flybus::Client, String)>, + ) -> CheckpointWriter { + let shared = Arc::new(WriterShared { + config, + faults, + permits: Arc::new(tokio::sync::Semaphore::new(config.queue_capacity)), + queue: std::sync::Mutex::new(VecDeque::new()), + queued_bytes: AtomicU64::new(0), + stats: std::sync::Mutex::new(WriterStats::default()), + wake: tokio::sync::Notify::new(), + stop: AtomicBool::new(false), + store: Arc::new(std::sync::Mutex::new(store)), + events, + }); + let task = tokio::spawn(run_writer(shared.clone())); + CheckpointWriter { shared, task } + } + + pub fn stats(&self) -> WriterStats { + *self.shared.stats.lock().expect("the writer stats are never poisoned") + } + + pub fn config(&self) -> WriterConfig { + self.shared.config + } + + /// How many captures are outstanding: queued or being written. + pub fn outstanding(&self) -> usize { + self.shared.config.queue_capacity - self.shared.permits.available_permits() + } + + /// Takes a queue slot, or refuses by name. Nothing is captured without one. + pub fn reserve(&self) -> DomainResult { + match self.shared.permits.clone().try_acquire_owned() { + Ok(permit) => Ok(Reservation { permit }), + Err(_) => { + self.shared + .stats + .lock() + .expect("the writer stats are never poisoned") + .rejected += 1; + Err(DomainError::before( + ErrorCode::Busy, + format!( + "the checkpoint queue already holds its {} outstanding captures; a \ +capture is refused before it is requested rather than queued without bound", + self.shared.config.queue_capacity + ), + )) + } + } + } + + /// Hands one complete capture to the writer and returns the channel its outcome arrives + /// on. The reservation becomes the job's slot. + pub async fn submit( + &self, + reservation: Reservation, + submission: CaptureSubmission, + ) -> DomainResult> { + let bytes = submission.byte_length(); + let budget = self.shared.config.max_queued_bytes; + let (reply, receiver) = tokio::sync::oneshot::channel(); + let checkpoint_id = submission.checkpoint_id.clone(); + let queued_event = submission.as_event("queued"); + // Reading the byte total, deciding on it and changing it are one critical section. + // Only one caller submits today, so a split could not be observed -- but a bound that + // is only correct while nobody else is submitting is not a bound. + let superseded = { + let mut queue = self.shared.queue.lock().expect("the writer queue is never poisoned"); + let replaced_index = if submission.replaceable { + queue.iter().position(|job| job.submission.replaceable) + } else { + None + }; + // A capture that will take a queued replaceable one's place frees its bytes, so + // the budget is decided against what the queue will hold and not what it holds. + let freed = replaced_index.map_or(0, |index| queue[index].submission.byte_length()); + let held = self.shared.queued_bytes.load(Ordering::SeqCst); + let after = held.saturating_sub(freed) + bytes; + if after > budget { + self.shared + .stats + .lock() + .expect("the writer stats are never poisoned") + .rejected += 1; + return Err(DomainError::before( + ErrorCode::Busy, + format!( + "the checkpoint queue would hold {after} of {budget} bytes; the byte \ +budget is finite and refuses before it is exceeded" + ), + )); + } + let replaced = replaced_index.map(|index| queue.remove(index).expect("just found")); + if let Some(old) = &replaced { + self.shared + .queued_bytes + .fetch_sub(old.submission.byte_length(), Ordering::SeqCst); + } + queue.push_back(Job { submission, reply, permit: reservation.permit }); + self.shared.queued_bytes.fetch_add(bytes, Ordering::SeqCst); + let mut stats = self.shared.stats.lock().expect("the writer stats are never poisoned"); + stats.queued += 1; + stats.peak_queue = stats.peak_queue.max(queue.len()); + // Sampled after the superseded job's bytes are gone, so the peak is a total the + // queue really held. + stats.peak_bytes = stats + .peak_bytes + .max(self.shared.queued_bytes.load(Ordering::SeqCst)); + replaced + }; + if let Some(old) = superseded { + self.shared + .stats + .lock() + .expect("the writer stats are never poisoned") + .superseded += 1; + let outcome = SaveOutcome::Superseded { + checkpoint_id: old.submission.checkpoint_id.clone(), + by: checkpoint_id.clone(), + }; + publish_event( + &self.shared.events, + &self.shared.stats, + &outcome_event(&old.submission, &outcome), + ) + .await; + // A caller that dropped its ticket is not waiting for this; the store's own + // metadata is the durable record either way. + let _ = old.reply.send(outcome); + // Dropping the job releases its owned captures and its queue slot. + drop(old.submission); + drop(old.permit); + } + publish_event(&self.shared.events, &self.shared.stats, &object(queued_event)).await; + self.shared.wake.notify_one(); + Ok(receiver) + } + + /// Waits for one save's outcome, within the caller's own budget. + /// + /// The two ways this ends without an outcome are different facts and are reported as + /// themselves: the channel closing means the writer finished and the reply did not reach + /// here, and the budget running out means the save is still going. Neither is a hang and + /// neither is a save. + pub async fn wait( + receiver: tokio::sync::oneshot::Receiver, + checkpoint_id: &Id, + budget: std::time::Duration, + ) -> SaveOutcome { + match tokio::time::timeout(budget, receiver).await { + Ok(Ok(outcome)) => outcome, + Ok(Err(_)) => SaveOutcome::ReplyLost { checkpoint_id: checkpoint_id.clone() }, + Err(_) => SaveOutcome::DeadlineExpired { checkpoint_id: checkpoint_id.clone() }, + } + } + + /// Runs `f` against the store, which is how a caller resolves a save whose reply it lost. + /// + /// The store's own work is blocking -- it fsyncs -- so it is reached on a blocking thread + /// rather than from the session's runtime. + pub async fn with_store(&self, f: F) -> T + where + F: FnOnce(&mut CheckpointStore) -> T + Send + 'static, + T: Send + 'static, + { + let store = self.shared.store.clone(); + tokio::task::spawn_blocking(move || { + let mut store = store.lock().expect("the checkpoint store is never poisoned"); + f(&mut store) + }) + .await + .expect("the checkpoint store task is never cancelled") + } + + /// Stops the writer and waits for its task. A leftover writer task would hold artifact + /// handles the session has finished with. + pub async fn shutdown(self) { + self.shared.stop.store(true, Ordering::SeqCst); + self.shared.wake.notify_one(); + let _ = self.task.await; + let mut queue = self.shared.queue.lock().expect("the writer queue is never poisoned"); + queue.clear(); + } +} + +impl CaptureSubmission { + fn as_event(&self, event: &'static str) -> Value { + json!({ + "event": event, + "checkpointId": self.checkpoint_id.as_str(), + "sessionId": self.session_id.as_str(), + "epoch": self.epoch.as_str(), + "episodeId": self.episode_id.as_str(), + "boundary": self.boundary.to_string(), + "byteLength": self.byte_length().to_string(), + }) + } +} + +fn outcome_event(submission: &CaptureSubmission, outcome: &SaveOutcome) -> Map { + let mut payload = object(submission.as_event(outcome.event())); + match outcome { + SaveOutcome::Committed { file, .. } => { + payload.insert("generation".into(), file.as_str().into()); + payload.insert("durable".into(), true.into()); + } + SaveOutcome::Failed { reason, .. } => { + payload.insert("reason".into(), reason.as_str().into()); + payload.insert("durable".into(), false.into()); + } + SaveOutcome::Superseded { by, .. } => { + payload.insert("supersededBy".into(), by.as_str().into()); + payload.insert("durable".into(), false.into()); + } + SaveOutcome::ReplyLost { .. } => { + payload.insert("reason".into(), "the save reply was lost".into()); + payload.insert("durable".into(), false.into()); + } + SaveOutcome::DeadlineExpired { .. } => { + payload.insert("reason".into(), "the caller's durable budget expired".into()); + payload.insert("durable".into(), false.into()); + } + } + payload +} + +async fn publish_event( + events: &Option<(flybus::Client, String)>, + stats: &std::sync::Mutex, + payload: &Map, +) { + let Some((client, topic)) = events else { return }; + // A checkpoint event is telemetry about the store, not the durable record. A publication + // that cannot be delivered never changes what was committed, so the outcome the caller + // receives stays the authority -- and the drop is counted rather than retried, because a + // retry here would be exactly the implicit best-effort policy these contracts refuse. + if client.publish(topic, payload.clone(), &[]).await.is_err() { + stats + .lock() + .expect("the writer stats are never poisoned") + .events_dropped += 1; + } +} + +async fn run_writer(shared: Arc) { + loop { + let job = { + let mut queue = shared.queue.lock().expect("the writer queue is never poisoned"); + queue.pop_front() + }; + let Some(job) = job else { + if shared.stop.load(Ordering::SeqCst) { + return; + } + shared.wake.notified().await; + continue; + }; + if let Some(gate) = &shared.faults.gate { + // A deliberately stalled writer. The queue in front of it stays bounded, which is + // the point of the stall. + if let Ok(permit) = gate.acquire().await { + permit.forget(); + } + } + let Job { submission, reply, permit } = job; + shared + .queued_bytes + .fetch_sub(submission.byte_length(), Ordering::SeqCst); + let written = write_one(&shared, &submission).await; + { + let mut stats = shared.stats.lock().expect("the writer stats are never poisoned"); + match &written { + WriteOutcome::Committed { .. } => stats.committed += 1, + WriteOutcome::Failed { .. } => stats.failed += 1, + } + } + let outcome = written.into_save(&submission.checkpoint_id); + publish_event(&shared.events, &shared.stats, &outcome_event(&submission, &outcome)).await; + let lost = shared.faults.drop_reply_for.as_ref() == Some(&submission.checkpoint_id); + // The writer owned these handles until the bytes were committed or the job failed. + // The job is over, so they and its queue slot go before the outcome is delivered: + // a caller that reads the queue depth the moment its outcome arrives must not see a + // slot this job has finished with. + drop(submission); + drop(permit); + if lost { + // The bytes are committed and the acknowledgment is lost. Durable metadata is in + // the store manifest, which is exactly where the caller must look. + drop(reply); + } else { + // A caller that dropped its ticket is not waiting for this. + let _ = reply.send(outcome); + } + } +} + +async fn write_one(shared: &Arc, submission: &CaptureSubmission) -> WriteOutcome { + let mut payloads = Vec::with_capacity(submission.payloads.len()); + for payload in &submission.payloads { + let bytes = match payload.artifact.read_all().await { + Ok(bytes) => bytes, + Err(e) => { + return WriteOutcome::Failed { + reason: format!("payload {}: {}", payload.name, e.message), + }; + } + }; + if bytes.len() as u64 != payload.byte_length || digest_of_bytes(&bytes) != payload.digest { + return WriteOutcome::Failed { + reason: format!( + "payload {} is not the content its capture declared", + payload.name + ), + }; + } + payloads.push((payload.name.clone(), bytes)); + } + let bytes = match checkpoint::encode(&submission.manifest, &payloads) { + Ok(bytes) => bytes, + Err(e) => { + return WriteOutcome::Failed { + reason: format!("envelope: {}", e.0), + }; + } + }; + let record = GenerationRecord { + checkpoint_id: submission.checkpoint_id.clone(), + file: format!("{}.flysess", submission.checkpoint_id), + session_id: submission.session_id.clone(), + epoch: submission.epoch.clone(), + episode_id: submission.episode_id.clone(), + boundary: submission.boundary, + compatibility_digest: submission.compatibility_digest.clone(), + envelope_digest: digest_of_bytes(&bytes), + byte_length: bytes.len() as u64, + }; + let attempts = match shared.config.retry { + RetryPolicy::ReleaseAndReport => 1, + RetryPolicy::RetryThenRelease { attempts } => attempts.max(1), + }; + let mut last = String::new(); + for _ in 0..attempts { + let record = record.clone(); + let bytes = bytes.clone(); + let store = shared.store.clone(); + // The durable sequence fsyncs twice. It runs on a blocking thread so the session's + // runtime is never the thing waiting on a disk. + let result = tokio::task::spawn_blocking(move || { + let mut store = store.lock().expect("the checkpoint store is never poisoned"); + store.commit(record, &bytes) + }) + .await; + match result { + Ok(Ok(())) => { + return WriteOutcome::Committed { + boundary: submission.boundary, + file: format!("{}.flysess", submission.checkpoint_id), + }; + } + Ok(Err(e)) => last = e.message, + Err(e) => last = format!("the checkpoint writer stopped: {e}"), + } + } + WriteOutcome::Failed { reason: last } +} diff --git a/services/flysim/crates/fly-session/src/task.rs b/services/flysim/crates/fly-session/src/task.rs index b48e909..2eebbef 100644 --- a/services/flysim/crates/fly-session/src/task.rs +++ b/services/flysim/crates/fly-session/src/task.rs @@ -39,6 +39,16 @@ pub fn episode_schema() -> SchemaRef { synthetic_schema("arena.episode.v1", 1) } +/// The schema of a captured task ledger. +pub fn ledger_schema() -> SchemaRef { + synthetic_schema("arena.ledger.v1", 1) +} + +/// The schema of a captured action-executor state. +pub fn executor_schema() -> SchemaRef { + synthetic_schema("arena.executor.v1", 1) +} + pub fn controller_schema_ref() -> SchemaRef { synthetic_schema("arena.controller.v1", 1) } @@ -86,6 +96,29 @@ pub trait Task: Send { /// How many times `evaluate_transition` has run. A transition must evaluate once. fn evaluations(&self) -> u64; + + /// The checkpointable ledger at a committed boundary (`workers-v1` section 4). + fn capture(&self) -> DomainResult; + + /// Validates a captured ledger without installing it, so a group install can fail before + /// anything is changed. + fn validate_restore(&self, state: &TypedValue) -> DomainResult<()>; + + /// Installs a validated ledger under `epoch`. Event identity is derived from the epoch, + /// so the new one is part of the install rather than something the ledger keeps from the + /// epoch it was captured in. + fn install_restore(&mut self, epoch: &Id, state: &TypedValue) -> DomainResult<()>; + + /// Every event identity this ledger has issued, mapped onto the identity it would have + /// under `to_epoch`. + /// + /// `workers-v1` section 4 derives an event id from the epoch, so a trace recorded in one + /// epoch cannot be compared with a trace recorded in another until these are rebased. + /// The ledger owns the derivation, so it is the only thing that can do it. + fn rebase_ids(&self, to_epoch: &Id) -> DomainResult>; + + /// How far event identity has reached: the highest source step and the number issued. + fn event_watermarks(&self) -> (u64, u64); } /// Translates one selected decision into a controller intent, with no port assignment. @@ -98,6 +131,15 @@ pub trait ActionExecutor: Send { progress: &TypedValue, clock: &RationalNs, ) -> DomainResult<(ControllerIntent, Vec)>; + + /// Per-executor state at a committed boundary (`workers-v1` section 4). + fn capture(&self) -> DomainResult; + + /// Validates a captured executor state without installing it. + fn validate_restore(&self, state: &TypedValue) -> DomainResult<()>; + + /// Installs a validated executor state. + fn install_restore(&mut self, state: &TypedValue) -> DomainResult<()>; } /// The only executor v1 supports: it passes a direct-control decision through unchanged. @@ -123,6 +165,37 @@ impl ActionExecutor for IdentityExecutor { .map_err(|e| DomainError::invalid(format!("decision: {e}")))?; Ok((intent, Vec::new())) } + + /// The identity executor is stateless, and says so rather than capturing nothing. + /// + /// An empty object would be indistinguishable from a stateful executor whose capture went + /// missing, so the capture names the executor it came from and a restore refuses any + /// other one. + fn capture(&self) -> DomainResult { + TypedValue::new(executor_schema(), json!({"executor": "identity-v1"})) + .map_err(|e| DomainError::invalid(e.0)) + } + + fn validate_restore(&self, state: &TypedValue) -> DomainResult<()> { + if state.schema != executor_schema() { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the captured executor state does not carry the executor schema", + )); + } + match state.value.get("executor").and_then(Value::as_str) { + Some("identity-v1") => Ok(()), + other => Err(DomainError::before( + ErrorCode::IncompatibleState, + format!("the captured executor is {other:?}, not the identity executor"), + )), + } + } + + fn install_restore(&mut self, state: &TypedValue) -> DomainResult<()> { + // Stateless: validation is the whole of the install, and it is not skipped. + self.validate_restore(state) + } } /// When the counter task asks for a terminal episode transition. @@ -146,6 +219,10 @@ pub struct CounterTask { evaluations: u64, total_reward: f64, counter: i64, + /// The highest source step any issued event belongs to, and how many were issued. These + /// are the event watermarks a checkpoint records and a resumed epoch continues from. + last_source_step: u64, + issued_events: u64, terminal: Terminal, } @@ -159,6 +236,8 @@ impl CounterTask { evaluations: 0, total_reward: 0.0, counter: 0, + last_source_step: 0, + issued_events: 0, terminal, } } @@ -239,6 +318,7 @@ impl Task for CounterTask { payload: TypedValue::new(event_schema(), json!({"counter": self.counter})) .expect("a synthetic typed value fits the contract"), }]; + self.issued_events += events.len() as u64; Ok(Bootstrap { contexts, progress: self.progress_value(), events }) } @@ -314,6 +394,8 @@ impl Task for CounterTask { )); } + self.last_source_step = self.last_source_step.max(source_step); + self.issued_events += events.len() as u64; let next_contexts = self .agents .iter() @@ -346,6 +428,173 @@ impl Task for CounterTask { fn evaluations(&self) -> u64 { self.evaluations } + + fn capture(&self) -> DomainResult { + TypedValue::new( + ledger_schema(), + json!({ + "epoch": self.epoch.as_str(), + "agents": self.agents.iter().map(String::as_str).collect::>(), + "bindings": self + .bindings + .iter() + .map(|b| json!({"portId": b.port_id.as_str(), "agentId": b.agent_id.as_str()})) + .collect::>(), + "transitions": self.transitions, + "evaluations": self.evaluations, + "totalReward": self.total_reward, + "counter": self.counter, + "lastSourceStep": self.last_source_step, + "issuedEvents": self.issued_events, + }), + ) + .map_err(|e| DomainError::invalid(e.0)) + } + + fn validate_restore(&self, state: &TypedValue) -> DomainResult<()> { + if state.schema != ledger_schema() { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the captured ledger does not carry this task's schema", + )); + } + for field in [ + "epoch", + "agents", + "bindings", + "transitions", + "evaluations", + "totalReward", + "counter", + "lastSourceStep", + "issuedEvents", + ] { + if state.value.get(field).is_none() { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + format!("the captured ledger has no {field}"), + )); + } + } + let bindings = state + .value + .get("bindings") + .and_then(Value::as_array) + .ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the captured ledger's bindings are not a list", + ) + })?; + if bindings.len() != self.bindings.len() && !self.bindings.is_empty() { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the captured ledger binds another number of ports", + )); + } + Ok(()) + } + + fn install_restore(&mut self, epoch: &Id, state: &TypedValue) -> DomainResult<()> { + self.validate_restore(state)?; + let number = |key: &str| -> DomainResult { + state.value.get(key).and_then(Value::as_u64).ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + format!("the captured ledger's {key} is not a whole number"), + ) + }) + }; + let mut agents = Vec::new(); + for value in state.value["agents"].as_array().expect("validated") { + let agent = value.as_str().ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the captured ledger names an agent that is not a string", + ) + })?; + agents.push(parse_id(agent).map_err(|e| { + DomainError::before(ErrorCode::IncompatibleState, format!("ledger: {e}")) + })?); + } + let mut bindings = Vec::new(); + for value in state.value["bindings"].as_array().expect("validated") { + let port_id = value.get("portId").and_then(Value::as_str).ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the captured ledger has a binding with no portId", + ) + })?; + let agent_id = value.get("agentId").and_then(Value::as_str).ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the captured ledger has a binding with no agentId", + ) + })?; + bindings.push(PortBinding { + port_id: parse_id(port_id).map_err(|e| { + DomainError::before(ErrorCode::IncompatibleState, format!("ledger: {e}")) + })?, + agent_id: parse_id(agent_id).map_err(|e| { + DomainError::before(ErrorCode::IncompatibleState, format!("ledger: {e}")) + })?, + }); + } + let counter = state.value.get("counter").and_then(Value::as_i64).ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the captured ledger's counter is not an integer", + ) + })?; + let total_reward = state + .value + .get("totalReward") + .and_then(Value::as_f64) + .filter(|v| v.is_finite()) + .ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the captured ledger's totalReward is not a finite number", + ) + })?; + // The epoch is the caller's, not the capture's: event identity belongs to the epoch + // the ledger is being installed into. + self.epoch = epoch.clone(); + self.agents = agents; + self.bindings = bindings; + self.transitions = number("transitions")?; + self.evaluations = number("evaluations")?; + self.total_reward = total_reward; + self.counter = counter; + self.last_source_step = number("lastSourceStep")?; + self.issued_events = number("issuedEvents")?; + Ok(()) + } + + fn rebase_ids(&self, to_epoch: &Id) -> DomainResult> { + let mut out = BTreeMap::new(); + out.insert( + event_id(&self.epoch, 0, "bootstrap", 0), + event_id(to_epoch, 0, "bootstrap", 0), + ); + // The counter task issues exactly one `counter-delta` event per bound port per + // evaluated transition, in descriptor port order, so every identity it has ever + // issued is re-derivable from its ledger without keeping a list of them. + let ports = self.bindings.len() as u32; + for source_step in 1..=self.last_source_step { + for ordinal in 0..ports { + out.insert( + event_id(&self.epoch, source_step, "counter-delta", ordinal), + event_id(to_epoch, source_step, "counter-delta", ordinal), + ); + } + } + Ok(out) + } + + fn event_watermarks(&self) -> (u64, u64) { + (self.last_source_step, self.issued_events) + } } /// The inspection value the counter environment publishes. diff --git a/services/flysim/crates/fly-session/src/types.rs b/services/flysim/crates/fly-session/src/types.rs index c52c276..07041c2 100644 --- a/services/flysim/crates/fly-session/src/types.rs +++ b/services/flysim/crates/fly-session/src/types.rs @@ -8,13 +8,18 @@ //! `Id` and `Digest` are type aliases, because the shared crate carries both as validated //! `String`s from `flybus::wire` rather than forking the encodings into newtypes. +use std::collections::BTreeMap; + use serde_json::{Map, Value}; pub use fly_session_types::ArtifactRef; pub use fly_session_types::canonical::{ self, OperationKey, body_digest, canonicalize, digest_of, sha256_hex, }; -pub use fly_session_types::media::{AudioDescriptor, AudioRef, ViewDescriptor, ViewRef}; +pub use fly_session_types::media::{ + ActivateRestoreParams, ActivateRestoreResult, AudioDescriptor, AudioRef, CaptureParams, + CaptureResult, StageRestoreParams, StageRestoreResult, ViewDescriptor, ViewRef, +}; pub use fly_session_types::rpc::{ ErrorCode, MutationCertainty, SessionRpcFailure, SessionRpcOutcome, SessionRpcRequest, SessionRpcSuccess, @@ -214,6 +219,57 @@ pub fn outcome_identity( } } +/// The epoch-derived identities of a behaviour trace, rewritten onto one reference epoch. +/// +/// `step-v1` section 8 compares committed behaviour across runs, excluding wall time, request +/// ids "and other explicitly operational metadata". A resumed run's epoch is neither: it is +/// behaviour metadata, and `scope.epoch`, the batch id and every task event id are derived +/// from it. Comparing the two runs therefore means rewriting exactly those three things and +/// nothing else, which is what this does -- and it **fails** on anything it does not +/// recognise instead of passing it through, so a field that silently stopped being rebased +/// would fail the comparison rather than weaken it. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct EpochRebase { + pub from: Id, + pub to: Id, + /// Every event identity the task issued under `from`, and the identity it has under `to`. + pub events: BTreeMap, +} + +impl EpochRebase { + /// Rewrites one behaviour record. An identity this rebase does not know is an error. + pub fn apply(&self, behaviour: &TraceBehaviour) -> Result { + if behaviour.scope.epoch != self.from { + return Err(format!( + "this behaviour was recorded in epoch {}, not {}", + behaviour.scope.epoch, self.from + )); + } + let mut out = behaviour.clone(); + out.scope = Scope::new(&behaviour.scope.session_id, &self.to, behaviour.scope.step) + .map_err(|e| e.0)?; + let prefix = format!("batch-{}-", self.from); + let suffix = behaviour + .batch_id + .strip_prefix(&prefix) + .ok_or_else(|| format!("the batch id {} is not derived from {}", behaviour.batch_id, self.from))?; + out.batch_id = parse_id(&format!("batch-{}-{suffix}", self.to))?; + let map = |ids: &[Id]| -> Result, String> { + ids.iter() + .map(|id| { + self.events + .get(id) + .cloned() + .ok_or_else(|| format!("no rebased identity for the event {id}")) + }) + .collect() + }; + out.outcome_ids = map(&behaviour.outcome_ids)?; + out.event_ids = map(&behaviour.event_ids)?; + Ok(out) + } +} + /// One session phase transition, recorded whether or not it ends a step. #[derive(Clone, Debug, PartialEq, Eq)] pub struct PhaseTransition { @@ -253,6 +309,28 @@ impl TraceLog { .collect() } + /// Every transition's behaviour, rebased onto one epoch and canonicalized. + /// + /// This is the comparison a resumed run is held to: the same strings as + /// [`TraceLog::behavior`], with the epoch metadata accounted for and nothing else changed. + /// A resumed run's log holds transitions from two epochs -- the ones before the checkpoint + /// and the ones after the restore -- so a transition already recorded in `rebase.to` is + /// kept as it stands and one recorded in `rebase.from` is rewritten. A transition in a + /// third epoch is an error; there is no pass-through case. + pub fn behavior_rebased(&self, rebase: &EpochRebase) -> Result, String> { + self.transitions + .iter() + .map(|t| { + let behaviour = if t.behaviour.scope.epoch == rebase.to { + t.behaviour.clone() + } else { + rebase.apply(&t.behaviour)? + }; + canonicalize(&behaviour.to_json()).map_err(|e| e.0) + }) + .collect() + } + /// The phase path, as `from -> to` strings. pub fn phase_path(&self) -> Vec { self.phases.iter().map(|p| format!("{} -> {}", p.from, p.to)).collect() diff --git a/services/flysim/crates/fly-session/src/worker.rs b/services/flysim/crates/fly-session/src/worker.rs index 327d66b..d717878 100644 --- a/services/flysim/crates/fly-session/src/worker.rs +++ b/services/flysim/crates/fly-session/src/worker.rs @@ -597,7 +597,17 @@ async fn execute( fn classify_default(method: &str) -> Option { match method { "Agent.Prepare" | "Agent.Commit" | "Environment.Advance" => Some(OpClass::StepMutation), - "Agent.Initialize" | "Environment.Initialize" => Some(OpClass::Lifecycle), + // `ipc-v1` section 5 retains lifecycle *and capture* replies until + // `Worker.Acknowledge`. The restore methods join them: their replies carry a + // once-only token and, for an environment, the restored observation's artifact, and a + // duplicate domain request must replay that reply rather than stage or activate a + // second time. They are not step mutations -- they carry no committed step of their + // own and are not keyed by one. + "Agent.Initialize" + | "Environment.Initialize" + | "State.Capture" + | "State.StageRestore" + | "State.ActivateRestore" => Some(OpClass::Lifecycle), "Worker.Hello" | "Worker.Status" | "Worker.Acknowledge" | "Worker.Shutdown" => { Some(OpClass::ReadOnly) } diff --git a/services/flysim/crates/fly-session/tests/processes.rs b/services/flysim/crates/fly-session/tests/processes.rs index e8e19f5..980ef3d 100644 --- a/services/flysim/crates/fly-session/tests/processes.rs +++ b/services/flysim/crates/fly-session/tests/processes.rs @@ -173,6 +173,8 @@ async fn a_slow_participant_is_resolved_rather_than_failed(mode: ExecutionMode) resolve: Duration::from_secs(15), resolve_attempts: u32::MAX, boot: Duration::from_secs(30), + capture: Duration::from_secs(30), + durable: Duration::from_secs(60), }; let reports = within("run", f.harness.coordinator.run(2)) .await @@ -262,6 +264,8 @@ async fn a_resolution_says_which_of_its_two_bounds_ended_it(mode: ExecutionMode) resolve: Duration::from_millis(200), resolve_attempts: u32::MAX, boot: Duration::from_secs(30), + capture: Duration::from_secs(30), + durable: Duration::from_secs(60), }; let failure = within("step", f.harness.coordinator.step()) .await @@ -288,6 +292,8 @@ async fn a_resolution_says_which_of_its_two_bounds_ended_it(mode: ExecutionMode) resolve: Duration::from_secs(3_600), resolve_attempts: 3, boot: Duration::from_secs(30), + capture: Duration::from_secs(30), + durable: Duration::from_secs(60), }; let failure = within("step", f.harness.coordinator.step()) .await diff --git a/services/flysim/crates/fly-session/tests/state.rs b/services/flysim/crates/fly-session/tests/state.rs new file mode 100644 index 0000000..b287c04 --- /dev/null +++ b/services/flysim/crates/fly-session/tests/state.rs @@ -0,0 +1,1002 @@ +//! STATE-01 acceptance: one coherent all-participant checkpoint, and the recovery that +//! installs it into a fresh epoch. +//! +//! Every test here is one of the slice's acceptance bullets or one of the failure-injection +//! rows the implementation guide's section 4 assigns to it. Each of them runs over both +//! transports and in all three execution modes: the recovery path crosses a process boundary +//! in exactly the places the media path does, so a row that holds in one mode has to hold in +//! all of them. +//! +//! The byte layout itself is proved against the contract crate in +//! `fly-session-types/tests/checkpoint_envelope.rs`; these prove the session's use of it. + +mod common; + +use std::collections::BTreeSet; +use std::path::Path; +use std::sync::Arc; +use std::time::Duration; + +use serde_json::{Value, json}; + +use common::{Fixture, count, fixture, fly_a, fly_b, within}; +use fly_session::agent::AgentFaults; +use fly_session::harness::{ExecutionMode, HarnessConfig, Via}; +use fly_session::phase::Phase; +use fly_session::state::{SaveOutcome, WriterFaults}; +use fly_session::types::*; +use fly_session_types::checkpoint; + +both_transports!( + an_uninterrupted_run_and_a_resumed_run_produce_matching_traces, + corrupting_any_single_participants_payload_fails_the_install_as_a_group, + a_lost_save_reply_does_not_advance_durable_metadata, + a_failure_during_activation_cannot_resume_half_a_world, + the_checkpoint_queue_under_stress_stays_bounded, + old_media_cannot_cross_a_recovery, + a_checkpoint_taken_by_another_parser_is_refused_by_name, + a_group_where_one_participant_will_not_stage_resumes_nothing, + a_restore_token_activates_only_once, + the_fence_lifts_only_through_a_coherent_restore, + a_queued_replaceable_capture_is_superseded_rather_than_duplicated, + a_capture_is_refused_anywhere_but_a_committed_boundary, +); + +all_modes!( + matching_traces_in_every_mode, + a_group_install_fails_as_a_group_in_every_mode, + a_lost_save_reply_holds_durable_metadata_in_every_mode, + half_an_activation_resumes_nothing_in_every_mode, + the_queue_stays_bounded_in_every_mode, + old_media_cannot_cross_in_every_mode, + another_parser_is_refused_in_every_mode, + a_refused_stage_resumes_nothing_in_every_mode, + a_token_activates_once_in_every_mode, + the_fence_lifts_only_by_restore_in_every_mode, +); + +const BEFORE: u64 = 2; +const AFTER: u64 = 2; + +fn ckpt(n: u32) -> Id { + id(&format!("ckpt-{n}")) +} + +/// A fixture in one transport and one execution mode. +/// +/// `Via` is the composition's; a dedicated thread or a separate process reaches the router +/// over a socket whatever it says, which the launcher decides and this does not second-guess. +async fn fx(via: Via, mode: ExecutionMode, config: HarnessConfig) -> Fixture { + fixture(via, HarnessConfig { mode, ..config }).await +} + +/// Runs to a committed boundary and takes one durable checkpoint there. +async fn run_and_checkpoint(f: &mut Fixture, steps: u64, checkpoint_id: &Id) { + within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap(); + within("run", f.harness.coordinator.run(steps)).await.unwrap(); + let outcome = within("checkpoint", f.harness.coordinator.checkpoint(checkpoint_id)) + .await + .unwrap(); + match outcome { + SaveOutcome::Committed { boundary, .. } => assert_eq!(boundary, steps), + other => panic!("the checkpoint was not committed: {other:?}"), + } + assert_eq!( + f.harness.coordinator.durable(), + Some((checkpoint_id.clone(), steps)), + "a committed save is the only thing that moves the durable mark" + ); +} + +/// Fails the epoch the way a participant death does, and checks the fence closed. +async fn fail_the_epoch(f: &mut Fixture) { + f.harness.kill(&fly_a()).await; + let failure = within("step", f.harness.coordinator.step()) + .await + .expect_err("a dead participant fails the epoch"); + assert!( + failure.participant.is_some(), + "a diagnosed failure names its participant: {failure}" + ); + assert_eq!(f.harness.coordinator.phase(), Phase::Failed); + assert!(f.harness.coordinator.is_fenced()); + assert_eq!( + f.harness.coordinator.live_view_handles(), + 0, + "the fence drops every handle the old epoch held" + ); +} + +/// The committed generation's bytes, as they are on disk. +fn generation_bytes(root: &Path, checkpoint_id: &Id) -> Vec { + std::fs::read(root.join(format!("{checkpoint_id}.flysess"))).expect("a committed generation") +} + +/// Writes a generation file and makes the store manifest describe it, as a repair tool or a +/// previous run would have left it. +fn write_generation(root: &Path, checkpoint_id: &Id, bytes: &[u8]) { + std::fs::write(root.join(format!("{checkpoint_id}.flysess")), bytes).expect("writable"); + let path = root.join("manifest.json"); + let mut manifest: Value = + serde_json::from_slice(&std::fs::read(&path).expect("a store manifest")).expect("json"); + let generations = manifest["generations"].as_array_mut().expect("an array"); + for generation in generations.iter_mut() { + if generation["checkpointId"] == json!(checkpoint_id.as_str()) { + generation["byteLength"] = json!(bytes.len().to_string()); + generation["envelopeDigest"] = json!(digest_of_bytes(bytes)); + let envelope = checkpoint::decode(bytes).expect("a well formed envelope"); + let compatibility = fly_session::state::Compatibility::from_json( + &envelope.manifest["compatibility"], + ) + .expect("a compatibility block"); + generation["compatibilityDigest"] = json!(compatibility.digest()); + } + } + std::fs::write(&path, serde_json::to_vec(&manifest).expect("json")).expect("writable"); +} + +/// Re-encodes one committed generation after `edit` has changed its manifest or its payloads. +fn rewrite_generation( + root: &Path, + checkpoint_id: &Id, + edit: impl FnOnce(&mut Value, &mut Vec<(String, Vec)>), +) { + let bytes = generation_bytes(root, checkpoint_id); + let envelope = checkpoint::decode(&bytes).expect("a committed generation decodes"); + let mut manifest = envelope.manifest.clone(); + let mut payloads = envelope.payloads.clone(); + edit(&mut manifest, &mut payloads); + // The manifest's payload table mirrors the envelope's, so it is rebuilt from the bytes + // that are actually being written rather than left to disagree. + manifest["payloads"] = Value::Array( + payloads + .iter() + .map(|(name, bytes)| { + json!({ + "name": name, + "byteLength": bytes.len().to_string(), + "digest": digest_of_bytes(bytes), + }) + }) + .collect(), + ); + let rewritten = checkpoint::encode(&manifest, &payloads).expect("a valid envelope"); + write_generation(root, checkpoint_id, &rewritten); +} + +/// Makes the store read its durable metadata again, after a test has edited it. +async fn reload_store(f: &mut Fixture) { + f.harness + .coordinator + .writer() + .expect("a store is attached") + .with_store(|store| store.reload()) + .await + .expect("the store manifest is still readable"); +} + +/// The names of every payload the checkpoint holds, participants first. +fn payload_names(root: &Path, checkpoint_id: &Id) -> Vec { + let bytes = generation_bytes(root, checkpoint_id); + checkpoint::decode(&bytes) + .expect("decodes") + .payloads + .into_iter() + .map(|(name, _)| name) + .collect() +} + +/// Every artifact identity this session's committed boundary is holding. +fn live_artifact_ids(f: &Fixture) -> BTreeSet { + f.harness + .coordinator + .media_handles() + .into_iter() + .map(|(_, reference)| reference.artifact_id) + .collect() +} + +/// Asserts that no participant is running the proposed epoch: nothing was installed. +async fn nothing_is_installed(f: &mut Fixture, epoch: &Id) { + let ids: Vec = std::iter::once(f.harness.environment_id()) + .chain(f.harness.config.agents.iter().map(|a| a.agent_id.clone())) + .collect(); + for worker_id in ids { + let (_, launcher) = f.harness.parts(); + let Ok(status) = launcher.health_check(&worker_id).await else { + // A participant that is gone is certainly not running the new epoch. + continue; + }; + if let Some(scope) = status.current_scope { + assert_ne!( + scope.epoch, *epoch, + "{worker_id} is running the epoch the abandoned install proposed" + ); + } + } + assert_eq!(f.harness.coordinator.phase(), Phase::Failed); + assert!(f.harness.coordinator.is_fenced()); + let refused = within("step", f.harness.coordinator.step()) + .await + .expect_err("a fenced session takes no step"); + assert_eq!(refused.error.code, ErrorCode::InvalidPhase); +} + +// =============================================================================================== +// Acceptance: an uninterrupted run and a resumed run produce matching traces + +async fn an_uninterrupted_run_and_a_resumed_run_produce_matching_traces(via: Via) { + matching_traces(via, ExecutionMode::InProcess).await; +} + +async fn matching_traces_in_every_mode(mode: ExecutionMode) { + matching_traces(Via::Unix, mode).await; +} + +/// The reference run and the resumed run commit the same behaviour, once the epoch metadata +/// the restore necessarily changed is accounted for. +/// +/// `step-v1` section 8's split is the whole of the comparison: the behaviour is what must +/// match, the request ids and wall time are excluded because they are operational, and the +/// epoch is neither -- it is behaviour metadata, so it is rewritten explicitly and everything +/// else is compared byte for byte. +async fn matching_traces(via: Via, mode: ExecutionMode) { + let mut reference = fx(via, mode, HarnessConfig::default()).await; + within("bootstrap", reference.harness.coordinator.bootstrap()).await.unwrap(); + within("run", reference.harness.coordinator.run(BEFORE + AFTER)).await.unwrap(); + let expected = reference.harness.coordinator.trace.behavior(); + assert_eq!(expected.len() as u64, BEFORE + AFTER); + reference.shutdown().await; + + let mut f = fx(via, mode, HarnessConfig::default()).await; + let checkpoint_id = ckpt(1); + run_and_checkpoint(&mut f, BEFORE, &checkpoint_id).await; + fail_the_epoch(&mut f).await; + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let report = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e2")), + ) + .await + .unwrap(); + assert_eq!(report.boundary, BEFORE); + assert_eq!(report.epoch, id("e2")); + assert_eq!(report.activated.len(), report.staged.len()); + assert!(!f.harness.coordinator.is_fenced(), "a coherent restore lifts the fence"); + assert_eq!(f.harness.coordinator.phase(), Phase::Paused(BEFORE)); + + f.harness.coordinator.resume().unwrap(); + within("resume", f.harness.coordinator.run(AFTER)).await.unwrap(); + assert_eq!(f.harness.coordinator.phase(), Phase::Ready(BEFORE + AFTER)); + + // Without accounting for the epoch the two traces disagree, which is what makes the + // rebase a statement rather than a formality. + let raw = f.harness.coordinator.trace.behavior(); + assert_eq!(raw.len() as u64, BEFORE + AFTER); + assert_ne!(raw, expected, "the resumed run runs in a different epoch"); + + let rebase = f.harness.coordinator.rebase(&id("e1")).unwrap(); + let resumed = f.harness.coordinator.trace.behavior_rebased(&rebase).unwrap(); + assert_eq!( + resumed, expected, + "a resumed run commits the behaviour the uninterrupted run committed" + ); + f.shutdown().await; +} + +// =============================================================================================== +// Acceptance: corrupt any participant and installation fails as a group + +async fn corrupting_any_single_participants_payload_fails_the_install_as_a_group(via: Via) { + group_install_is_all_or_nothing(via, ExecutionMode::InProcess).await; +} + +async fn a_group_install_fails_as_a_group_in_every_mode(mode: ExecutionMode) { + group_install_is_all_or_nothing(Via::Unix, mode).await; +} + +/// One corrupted payload -- any participant's, and the coordinator's own ledger too -- fails +/// the whole install, and the group afterwards is exactly as it was. +async fn group_install_is_all_or_nothing(via: Via, mode: ExecutionMode) { + let mut f = fx(via, mode, HarnessConfig::default()).await; + let checkpoint_id = ckpt(1); + run_and_checkpoint(&mut f, BEFORE, &checkpoint_id).await; + let root = f.harness.checkpoint_root().to_path_buf(); + let good = generation_bytes(&root, &checkpoint_id); + let names = payload_names(&root, &checkpoint_id); + // Every participant's payload, plus one of the coordinator's own ledgers. + let mut corrupt: Vec = names + .iter() + .filter(|name| name.starts_with("agent-") || *name == "world") + .cloned() + .collect(); + corrupt.push("task-ledger".to_owned()); + assert!(corrupt.len() >= 3, "the composition has several payloads: {names:?}"); + fail_the_epoch(&mut f).await; + + let mut epoch = 1u32; + for target in &corrupt { + epoch += 1; + let proposed = id(&format!("e{epoch}")); + // Each round starts from the intact bytes, so exactly one payload is corrupt. + write_generation(&root, &checkpoint_id, &good); + // A well formed envelope whose digests all agree, so what fails is the participant + // reading its own bytes and not the envelope reader in front of it. + let name = target.clone(); + rewrite_generation(&root, &checkpoint_id, |_manifest, payloads| { + for (payload, bytes) in payloads.iter_mut() { + if *payload == name { + let last = bytes.len() - 1; + bytes[last] ^= 0xff; + } + } + }); + reload_store(&mut f).await; + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let failure = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &proposed), + ) + .await + .expect_err(&format!("a corrupt {target} must fail the install")); + assert!( + matches!( + failure.error.code, + ErrorCode::IncompatibleState | ErrorCode::InvalidArgument + ), + "a corrupt {target} is an explicit refusal, not {failure}" + ); + nothing_is_installed(&mut f, &proposed).await; + let tainted = f.harness.coordinator.tainted(); + if target == "world" { + assert!( + tainted.is_empty(), + "the first participant asked refused, so nothing staged: {tainted:?}" + ); + } else { + assert!( + !tainted.is_empty(), + "a participant that staged into an abandoned install must be replaced" + ); + } + } + + // The same group, the same store, the intact bytes: the failures above installed nothing + // that stops this from working. + epoch += 1; + write_generation(&root, &checkpoint_id, &good); + reload_store(&mut f).await; + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let report = within( + "restore", + f.harness + .coordinator + .restore(Some(&checkpoint_id), &id(&format!("e{epoch}"))), + ) + .await + .unwrap(); + assert_eq!(report.boundary, BEFORE); + f.harness.coordinator.resume().unwrap(); + within("resume", f.harness.coordinator.run(1)).await.unwrap(); + f.shutdown().await; +} + +// =============================================================================================== +// Acceptance: a lost save reply does not advance durable metadata + +async fn a_lost_save_reply_does_not_advance_durable_metadata(via: Via) { + a_lost_save_reply(via, ExecutionMode::InProcess).await; +} + +async fn a_lost_save_reply_holds_durable_metadata_in_every_mode(mode: ExecutionMode) { + a_lost_save_reply(Via::Unix, mode).await; +} + +/// Three ways a save can end without a saved acknowledgment. None moves the mark, and each +/// is reported as itself rather than as the others. +async fn a_lost_save_reply(via: Via, mode: ExecutionMode) { + let lost = ckpt(1); + let config = HarnessConfig { + writer_faults: WriterFaults { + drop_reply_for: Some(lost.clone()), + ..WriterFaults::default() + }, + ..HarnessConfig::default() + }; + let mut f = fx(via, mode, config).await; + within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap(); + within("run", f.harness.coordinator.run(1)).await.unwrap(); + + let ticket = within("capture", f.harness.coordinator.capture(&lost, false)) + .await + .unwrap(); + let outcome = within("durable", f.harness.coordinator.await_durable(ticket)) + .await + .unwrap(); + assert_eq!(outcome, SaveOutcome::ReplyLost { checkpoint_id: lost.clone() }); + assert_eq!( + f.harness.coordinator.durable(), + None, + "a lost save reply never moves the durable mark" + ); + assert_eq!(count(&f.harness.coordinator.audit, &format!("durable:{lost}@1")), 0); + + // The resolution asks the durable metadata about the *same* operation. It never saves + // again, and here the bytes did reach the store manifest. + let resolved = within("resolve", f.harness.coordinator.resolve_durable(&lost)) + .await + .unwrap(); + assert_eq!(resolved, Some(1)); + assert_eq!(f.harness.coordinator.durable(), Some((lost.clone(), 1))); + let commits = f + .harness + .coordinator + .writer() + .expect("a store") + .with_store(|store| store.commits()) + .await; + assert_eq!(commits, 1, "the resolution queried the store; it did not save again"); + f.shutdown().await; + + // The other half: the generation file is renamed and the store manifest is never + // committed. That generation is unreferenced, so it is not a candidate and the mark is + // still where it was. + let orphan = ckpt(2); + let mut config = HarnessConfig::default(); + config.store_faults.stop_before_manifest_commit = true; + let mut g = fx(via, mode, config).await; + within("bootstrap", g.harness.coordinator.bootstrap()).await.unwrap(); + within("run", g.harness.coordinator.run(1)).await.unwrap(); + let outcome = within("checkpoint", g.harness.coordinator.checkpoint(&orphan)) + .await + .unwrap(); + match outcome { + SaveOutcome::Failed { checkpoint_id, .. } => assert_eq!(checkpoint_id, orphan), + other => panic!("an uncommitted manifest is not a saved checkpoint: {other:?}"), + } + assert_eq!(g.harness.coordinator.durable(), None); + let root = g.harness.checkpoint_root().to_path_buf(); + assert!( + root.join(format!("{orphan}.flysess")).is_file(), + "the generation file was written and renamed" + ); + let resolved = within("resolve", g.harness.coordinator.resolve_durable(&orphan)) + .await + .unwrap(); + assert_eq!(resolved, None, "an unreferenced generation is never a restore candidate"); + assert_eq!(g.harness.coordinator.durable(), None); + g.shutdown().await; + + // The third: the caller's own budget runs out while the save is still going. That is a + // different fact from a lost reply -- this save has not stopped -- and it is reported as + // itself, because diagnosing one as the other is the implicit reading these rules refuse. + let slow = ckpt(3); + let gate = Arc::new(tokio::sync::Semaphore::new(0)); + let mut config = HarnessConfig::default(); + config.writer_faults.gate = Some(gate.clone()); + let mut h = fx(via, mode, config).await; + within("bootstrap", h.harness.coordinator.bootstrap()).await.unwrap(); + within("run", h.harness.coordinator.run(1)).await.unwrap(); + // The durable budget is the caller's own and is not the call budget: this shortens the + // wait for an acknowledgment without shortening a single call to a participant. + h.harness.coordinator.deadlines.durable = Duration::from_millis(100); + let ticket = within("capture", h.harness.coordinator.capture(&slow, false)) + .await + .unwrap(); + let outcome = within("durable", h.harness.coordinator.await_durable(ticket)) + .await + .unwrap(); + assert_eq!( + outcome, + SaveOutcome::DeadlineExpired { checkpoint_id: slow.clone() }, + "an expired caller budget is not a lost reply" + ); + assert_ne!(outcome, SaveOutcome::ReplyLost { checkpoint_id: slow.clone() }); + assert_eq!(h.harness.coordinator.durable(), None); + assert_eq!( + count( + &h.harness.coordinator.audit, + &format!("not-durable:failed:{slow}@1") + ), + 1, + "the session records that this capture is not durable: {:?}", + h.harness.coordinator.audit + ); + + // It really was still going: once the writer is let past its gate the same operation + // commits, and resolving it is what moves the mark. + gate.add_permits(16); + let deadline = std::time::Instant::now() + Duration::from_secs(10); + let resolved = loop { + let found = within("resolve", h.harness.coordinator.resolve_durable(&slow)) + .await + .unwrap(); + if found.is_some() || std::time::Instant::now() >= deadline { + break found; + } + tokio::time::sleep(Duration::from_millis(10)).await; + }; + assert_eq!(resolved, Some(1), "the save the caller stopped waiting for still committed"); + assert_eq!(h.harness.coordinator.durable(), Some((slow, 1))); + h.shutdown().await; +} + +// =============================================================================================== +// Acceptance: a failure during activation cannot resume half a world + +async fn a_failure_during_activation_cannot_resume_half_a_world(via: Via) { + half_an_activation(via, ExecutionMode::InProcess).await; +} + +async fn half_an_activation_resumes_nothing_in_every_mode(mode: ExecutionMode) { + half_an_activation(Via::Unix, mode).await; +} + +/// The world and the first agent activate; the second refuses. Nothing plays, the fence +/// stays closed, and the group cannot be resumed until every participant is replaced. +async fn half_an_activation(via: Via, mode: ExecutionMode) { + let mut f = fx(via, mode, HarnessConfig::default()).await; + let checkpoint_id = ckpt(1); + run_and_checkpoint(&mut f, BEFORE, &checkpoint_id).await; + fail_the_epoch(&mut f).await; + + // The replacement for fly-b refuses to activate after it has staged. + f.harness.set_agent_faults( + &fly_b(), + AgentFaults { fail_activate_restore: true, ..AgentFaults::default() }, + ); + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let failure = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e2")), + ) + .await + .expect_err("a refused activation fails the install"); + assert_eq!(failure.participant.as_deref(), Some(fly_b().as_str())); + assert_eq!(f.harness.coordinator.phase(), Phase::Failed); + assert!(f.harness.coordinator.is_fenced(), "the group stays fenced"); + let refused = within("step", f.harness.coordinator.step()) + .await + .expect_err("half a world never runs"); + assert_eq!(refused.error.code, ErrorCode::InvalidPhase); + + // The participants that got as far as installing hold state no group resumed. Another + // restore over them is refused by name rather than attempted. + let tainted = f.harness.coordinator.tainted(); + assert!(tainted.contains(&f.harness.environment_id()), "{tainted:?}"); + assert!(tainted.contains(&fly_a()), "{tainted:?}"); + let refused = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e3")), + ) + .await + .expect_err("the half-installed group is not restored over"); + assert_eq!(refused.error.code, ErrorCode::InvalidPhase); + assert!(refused.error.message.contains(&fly_a()), "{refused}"); + + // A fresh group, without the injected refusal, resumes the same boundary. + f.harness.set_agent_faults(&fly_b(), AgentFaults::default()); + within("replace", f.harness.replace_all_participants()).await.unwrap(); + assert!(f.harness.coordinator.tainted().is_empty()); + let report = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e4")), + ) + .await + .unwrap(); + assert_eq!(report.boundary, BEFORE); + assert!(!f.harness.coordinator.is_fenced()); + f.harness.coordinator.resume().unwrap(); + within("resume", f.harness.coordinator.run(1)).await.unwrap(); + f.shutdown().await; +} + +// =============================================================================================== +// Acceptance: the checkpoint queue under stress stays bounded + +async fn the_checkpoint_queue_under_stress_stays_bounded(via: Via) { + queue_stays_bounded(via, ExecutionMode::InProcess).await; +} + +async fn the_queue_stays_bounded_in_every_mode(mode: ExecutionMode) { + queue_stays_bounded(Via::Unix, mode).await; +} + +/// With the writer stalled, the queue fills to its configured bound and the next capture is +/// refused *before* a single participant is asked for one. +async fn queue_stays_bounded(via: Via, mode: ExecutionMode) { + let gate = Arc::new(tokio::sync::Semaphore::new(0)); + let mut config = HarnessConfig::default(); + config.writer_faults.gate = Some(gate.clone()); + config.writer.queue_capacity = 2; + let mut f = fx(via, mode, config).await; + within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap(); + within("run", f.harness.coordinator.run(1)).await.unwrap(); + + let first = within("capture", f.harness.coordinator.capture(&ckpt(1), false)) + .await + .unwrap(); + let second = within("capture", f.harness.coordinator.capture(&ckpt(2), false)) + .await + .unwrap(); + assert_eq!(f.harness.coordinator.writer().expect("a store").outstanding(), 2); + + let environment = f.harness.environment_id(); + let before = within("status", f.harness.progress_of(&environment)).await.unwrap(); + let refused = within("capture", f.harness.coordinator.capture(&ckpt(3), false)) + .await + .expect_err("a full queue refuses"); + assert_eq!(refused.error.code, ErrorCode::Busy); + assert_eq!(refused.error.mutation, MutationCertainty::None); + let after = within("status", f.harness.progress_of(&environment)).await.unwrap(); + assert_eq!( + before, after, + "a refused capture asks no participant for one: rejection happens before capture" + ); + assert_eq!(count(&f.harness.coordinator.audit, "capture-refused:ckpt-3"), 1); + + // A refused capture is not a failed session: the world keeps stepping. + assert!(!f.harness.coordinator.is_fenced()); + within("run", f.harness.coordinator.run(1)).await.unwrap(); + assert_eq!(f.harness.coordinator.phase(), Phase::Ready(2)); + + let stats = f.harness.coordinator.writer().expect("a store").stats(); + assert_eq!(stats.rejected, 1); + assert!(stats.peak_queue <= 2, "the queue never exceeded its bound: {stats:?}"); + assert!(stats.peak_bytes <= 64 * 1024 * 1024); + + gate.add_permits(16); + for (ticket, expected) in [(first, ckpt(1)), (second, ckpt(2))] { + let outcome = within("durable", f.harness.coordinator.await_durable(ticket)) + .await + .unwrap(); + match outcome { + SaveOutcome::Committed { checkpoint_id, .. } => assert_eq!(checkpoint_id, expected), + other => panic!("an unstalled writer commits: {other:?}"), + } + } + let stats = f.harness.coordinator.writer().expect("a store").stats(); + assert_eq!(stats.committed, 2); + assert_eq!(f.harness.coordinator.writer().expect("a store").outstanding(), 0); + f.shutdown().await; +} + +/// A queued replaceable capture is replaced by the next one, releasing its holds, rather than +/// both being written. +async fn a_queued_replaceable_capture_is_superseded_rather_than_duplicated(via: Via) { + let gate = Arc::new(tokio::sync::Semaphore::new(0)); + let mut config = HarnessConfig::default(); + config.writer_faults.gate = Some(gate.clone()); + config.writer.queue_capacity = 3; + let mut f = fx(via, ExecutionMode::InProcess, config).await; + within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap(); + within("run", f.harness.coordinator.run(1)).await.unwrap(); + + // The first job is taken by the writer and stalls at the gate; the next two are queued. + let durable = within("capture", f.harness.coordinator.capture(&ckpt(1), false)) + .await + .unwrap(); + let hot = within("capture", f.harness.coordinator.capture(&ckpt(2), true)) + .await + .unwrap(); + let newer = within("capture", f.harness.coordinator.capture(&ckpt(3), true)) + .await + .unwrap(); + + let superseded = within("durable", f.harness.coordinator.await_durable(hot)) + .await + .unwrap(); + assert_eq!( + superseded, + SaveOutcome::Superseded { checkpoint_id: ckpt(2), by: ckpt(3) }, + "only a queued replaceable capture is coalesced" + ); + gate.add_permits(16); + for ticket in [durable, newer] { + let outcome = within("durable", f.harness.coordinator.await_durable(ticket)) + .await + .unwrap(); + assert!(matches!(outcome, SaveOutcome::Committed { .. }), "{outcome:?}"); + } + let stats = f.harness.coordinator.writer().expect("a store").stats(); + assert_eq!(stats.superseded, 1); + assert_eq!(stats.committed, 2, "the superseded capture was never written"); + f.shutdown().await; +} + +// =============================================================================================== +// Acceptance: old media and old parser data cannot cross a recovery + +async fn old_media_cannot_cross_a_recovery(via: Via) { + old_media_cannot_cross(via, ExecutionMode::InProcess).await; +} + +async fn old_media_cannot_cross_in_every_mode(mode: ExecutionMode) { + old_media_cannot_cross(Via::Unix, mode).await; +} + +/// Nothing the fence dropped comes back: the payloads are re-imported as fresh artifacts, the +/// restored view is a new object, and the new epoch's first audio chunk resumes the preserved +/// sample position and marks the discontinuity. +async fn old_media_cannot_cross(via: Via, mode: ExecutionMode) { + let mut f = fx(via, mode, HarnessConfig::default()).await; + let checkpoint_id = ckpt(1); + run_and_checkpoint(&mut f, BEFORE, &checkpoint_id).await; + let old_ids = live_artifact_ids(&f); + assert!(!old_ids.is_empty(), "the committed boundary holds media handles"); + let positions = f.harness.coordinator.audio_positions(); + assert!(positions.values().any(|sample| *sample > 0), "audio has played"); + + fail_the_epoch(&mut f).await; + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let report = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e2")), + ) + .await + .unwrap(); + for imported in &report.imported { + assert!( + !old_ids.contains(imported), + "a checkpoint payload was imported as an artifact the old epoch already had" + ); + } + let new_ids = live_artifact_ids(&f); + assert!(!new_ids.is_empty()); + assert!( + new_ids.is_disjoint(&old_ids), + "the restored boundary's media are fresh artifacts, not the old epoch's" + ); + assert_eq!( + f.harness.coordinator.audio_positions(), + positions, + "crash restore preserves the sample position" + ); + + f.harness.coordinator.resume().unwrap(); + within("resume", f.harness.coordinator.run(1)).await.unwrap(); + let chunk = f + .harness + .coordinator + .observation() + .expect("a restored world") + .audio + .first() + .cloned() + .expect("one chunk per transition"); + assert!( + chunk.discontinuity, + "the first chunk of a fresh epoch marks the discontinuity the recovery established" + ); + assert_eq!( + chunk.first_sample, + positions["arena"], + "and it starts where the checkpointed stream stopped" + ); + f.shutdown().await; +} + +async fn a_checkpoint_taken_by_another_parser_is_refused_by_name(via: Via) { + another_parser_is_refused(via, ExecutionMode::InProcess).await; +} + +async fn another_parser_is_refused_in_every_mode(mode: ExecutionMode) { + another_parser_is_refused(Via::Unix, mode).await; +} + +/// A checkpoint whose recorded parser identity is not this composition's is refused before a +/// single participant is asked to stage, and the refusal names the identity that differs. +async fn another_parser_is_refused(via: Via, mode: ExecutionMode) { + let mut f = fx(via, mode, HarnessConfig::default()).await; + let checkpoint_id = ckpt(1); + run_and_checkpoint(&mut f, 1, &checkpoint_id).await; + let root = f.harness.checkpoint_root().to_path_buf(); + fail_the_epoch(&mut f).await; + rewrite_generation(&root, &checkpoint_id, |manifest, _payloads| { + manifest["compatibility"]["parserDigest"] = + json!(digest_of_bytes(b"some other inspection schema")); + }); + reload_store(&mut f).await; + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let failure = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e2")), + ) + .await + .expect_err("another parser's state is not this composition's"); + assert_eq!(failure.error.code, ErrorCode::IncompatibleState); + assert!( + failure.error.message.contains("parser"), + "the refusal names the identity that differs: {failure}" + ); + assert!( + f.harness.coordinator.tainted().is_empty(), + "nothing was asked to stage, so nothing has to be replaced" + ); + nothing_is_installed(&mut f, &id("e2")).await; + f.shutdown().await; +} + +// =============================================================================================== +// Failure-injection rows + +async fn a_group_where_one_participant_will_not_stage_resumes_nothing(via: Via) { + a_refused_stage(via, ExecutionMode::InProcess).await; +} + +async fn a_refused_stage_resumes_nothing_in_every_mode(mode: ExecutionMode) { + a_refused_stage(Via::Unix, mode).await; +} + +/// The checklist row: the install validates the participants ahead of it and the last one +/// fails. Nothing is resumed. +async fn a_refused_stage(via: Via, mode: ExecutionMode) { + let mut f = fx(via, mode, HarnessConfig::default()).await; + let checkpoint_id = ckpt(1); + run_and_checkpoint(&mut f, 1, &checkpoint_id).await; + fail_the_epoch(&mut f).await; + f.harness.set_agent_faults( + &fly_b(), + AgentFaults { fail_stage_restore: true, ..AgentFaults::default() }, + ); + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let failure = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e2")), + ) + .await + .expect_err("a participant that will not validate stops the install"); + assert_eq!(failure.error.code, ErrorCode::IncompatibleState); + assert_eq!(failure.participant.as_deref(), Some(fly_b().as_str())); + assert_eq!( + count(&f.harness.coordinator.audit, &format!("activated:{}", fly_a())), + 0, + "no participant activates when one of them will not stage" + ); + nothing_is_installed(&mut f, &id("e2")).await; + f.shutdown().await; +} + +async fn a_restore_token_activates_only_once(via: Via) { + a_token_activates_once(via, ExecutionMode::InProcess).await; +} + +async fn a_token_activates_once_in_every_mode(mode: ExecutionMode) { + a_token_activates_once(Via::Unix, mode).await; +} + +/// A token is bound to its checkpoint, scope and payload and activates once. A fresh request +/// naming it again is a conflict, and the old epoch's scope is stale on the new participants. +async fn a_token_activates_once(via: Via, mode: ExecutionMode) { + let mut f = fx(via, mode, HarnessConfig::default()).await; + let checkpoint_id = ckpt(1); + run_and_checkpoint(&mut f, 1, &checkpoint_id).await; + fail_the_epoch(&mut f).await; + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let report = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e2")), + ) + .await + .unwrap(); + let (who, token) = report.tokens.first().cloned().expect("a staged token"); + let worker = if who == f.harness.environment_id() { + f.harness.coordinator.environment_ref().clone() + } else { + f.harness.coordinator.agent_ref(&who).expect("a participant").clone() + }; + let scope = scope_at("demo", "e2", 1); + let refused = within( + "activate", + f.harness.coordinator.probe_raw( + &worker, + "State.ActivateRestore", + Some(scope), + json!({"restoreToken": token.as_str()}), + ), + ) + .await + .expect_err("a token activates once"); + assert_eq!(refused.code, ErrorCode::Conflict); + + // And the epoch the restore left behind is stale on every replacement. + let stale = within( + "prepare", + f.harness.coordinator.probe_raw( + &worker, + if who == f.harness.environment_id() { "Environment.Advance" } else { "Agent.Prepare" }, + Some(scope_at("demo", "e1", 1)), + json!({}), + ), + ) + .await + .expect_err("the old epoch is gone"); + assert!( + matches!(stale.code, ErrorCode::StaleEpoch | ErrorCode::InvalidArgument), + "an old-epoch request is refused: {stale}" + ); + f.shutdown().await; +} + +async fn the_fence_lifts_only_through_a_coherent_restore(via: Via) { + the_fence_lifts_only_by_restore(via, ExecutionMode::InProcess).await; +} + +async fn the_fence_lifts_only_by_restore_in_every_mode(mode: ExecutionMode) { + the_fence_lifts_only_by_restore(Via::Unix, mode).await; +} + +/// Nothing but a coherent restore moves a fenced session, and the restore lands on +/// `Paused(k)` rather than straight back into play. +async fn the_fence_lifts_only_by_restore(via: Via, mode: ExecutionMode) { + let mut f = fx(via, mode, HarnessConfig::default()).await; + let checkpoint_id = ckpt(1); + run_and_checkpoint(&mut f, 1, &checkpoint_id).await; + fail_the_epoch(&mut f).await; + + for refused in [ + within("step", f.harness.coordinator.step()).await.err(), + within("bootstrap", f.harness.coordinator.bootstrap()).await.err(), + within("capture", f.harness.coordinator.capture(&ckpt(9), false)) + .await + .err(), + ] { + let refused = refused.expect("a fenced session refuses"); + assert_eq!(refused.error.code, ErrorCode::InvalidPhase); + } + assert!(f.harness.coordinator.resume().is_err(), "a fenced session does not resume"); + assert!(f.harness.coordinator.is_fenced()); + + // A restore into the epoch that failed is refused: recovery establishes a fresh one. + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let same = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e1")), + ) + .await + .expect_err("a restore installs a fresh epoch"); + assert_eq!(same.error.code, ErrorCode::StaleEpoch); + + let report = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e2")), + ) + .await + .unwrap(); + assert_eq!(report.boundary, 1); + assert!(!f.harness.coordinator.is_fenced()); + assert_eq!(f.harness.coordinator.phase(), Phase::Paused(1)); + f.harness.coordinator.resume().unwrap(); + within("resume", f.harness.coordinator.run(1)).await.unwrap(); + assert_eq!(f.harness.coordinator.phase(), Phase::Ready(2)); + f.shutdown().await; +} + +/// A capture belongs to a committed boundary, and the phase machine is what says so. +async fn a_capture_is_refused_anywhere_but_a_committed_boundary(via: Via) { + let mut f = fx(via, ExecutionMode::InProcess, HarnessConfig::default()).await; + // Before bootstrap the session is Starting, which is not a boundary at all. + let refused = within("capture", f.harness.coordinator.capture(&ckpt(1), false)) + .await + .expect_err("Starting is not a committed boundary"); + assert_eq!(refused.error.code, ErrorCode::InvalidPhase); + f.shutdown().await; + + // From a pause, which is a committed boundary, it works, and it returns there. + let mut f = fx(via, ExecutionMode::InProcess, HarnessConfig::default()).await; + within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap(); + within("run", f.harness.coordinator.run(1)).await.unwrap(); + f.harness.coordinator.request_pause(); + within("run", f.harness.coordinator.run(1)).await.unwrap(); + assert_eq!(f.harness.coordinator.phase(), Phase::Paused(2)); + let outcome = within("checkpoint", f.harness.coordinator.checkpoint(&ckpt(1))) + .await + .unwrap(); + assert!(matches!(outcome, SaveOutcome::Committed { boundary: 2, .. }), "{outcome:?}"); + assert_eq!( + f.harness.coordinator.phase(), + Phase::Paused(2), + "a capture returns to the boundary it came from" + ); + f.shutdown().await; +} diff --git a/services/flysim/crates/flybrain-gb/src/adapter.rs b/services/flysim/crates/flybrain-gb/src/adapter.rs index 15d8456..9e24815 100644 --- a/services/flysim/crates/flybrain-gb/src/adapter.rs +++ b/services/flysim/crates/flybrain-gb/src/adapter.rs @@ -56,7 +56,7 @@ impl MemoryReader for &mut dyn MemoryReader { /// One reward payout in one frame. /// /// `kind` is an adapter-owned interned name (Pokémon: `milestone`, -/// `exploration`, `map`, `species`, `trainer`, `battle`, `badge`, `boundary`); it is the +/// `exploration`, `map`, `species`, `trainer`, `battle`, `badge`, `boundary`, `catch`); it is the /// key the statistics counters and the on-screen ticker group by. Field names /// serialize exactly as the prototype's `RewardEvent` did, so a checkpoint /// written by either implementation reads in the other. @@ -241,9 +241,21 @@ impl std::error::Error for AdapterError {} /// A game, as the sim loop sees it. pub trait GameAdapter: Send { /// Adapter version string, pinned into the checkpoint compatibility string. - /// Pokémon: `pokered-unique8-v5`. + /// Pokémon: `pokered-unique8-v6`. fn id(&self) -> &'static str; + /// Earlier [`GameAdapter::id`]s whose checkpoints this build can read, by a migration + /// this adapter has written down and tested. + /// + /// The default is empty: an adapter migrates from nothing unless it says otherwise, which + /// is the behaviour every adapter had before this existed. It is only half of the gate -- + /// [`crate::compatibility::decide`] also requires the operator to have named the same id in + /// `FLY_ACCEPT_ADAPTERS` for that deploy -- so listing an id here never migrates a live run + /// on its own. + fn migrates_from(&self) -> &'static [&'static str] { + &[] + } + /// Whether semantic rewards are enabled for this cartridge. An adapter that /// says no must still sample without paying anything, so the stream keeps /// running with a visible "rewards off" mode. @@ -470,6 +482,10 @@ mod tests { let platformer = adapter_for_with_rom_pin("platformer", Some(&"a".repeat(64))).unwrap(); assert_ne!(pokemon.id(), platformer.id()); + assert!( + !platformer.migrates_from().contains(&pokemon.id()), + "a migration never crosses games" + ); assert_ne!(pokemon.symbol_provenance(), platformer.symbol_provenance()); // And two ROM revisions of the same game cannot either. let other = adapter_for_with_rom_pin("platformer", Some(&"b".repeat(64))).unwrap(); diff --git a/services/flysim/crates/flybrain-gb/src/compatibility.rs b/services/flysim/crates/flybrain-gb/src/compatibility.rs index 7e183c0..b3fc959 100644 --- a/services/flysim/crates/flybrain-gb/src/compatibility.rs +++ b/services/flysim/crates/flybrain-gb/src/compatibility.rs @@ -30,7 +30,7 @@ pub const PROTOTYPE_PLASTICITY_VERSION: &str = "fly-kc-mbon-rstdp-v2"; pub struct Compatibility<'a> { /// `kernelVersion(config)` from the neural library. pub neural_kernel_version: &'a str, - /// The adapter's version string, e.g. `pokered-unique8-v5`. + /// The adapter's version string, e.g. `pokered-unique8-v6`. pub adapter: &'a str, /// The dataset's seven SHA-256 digests joined with `:`. pub dataset_fingerprint: &'a str, @@ -67,6 +67,92 @@ impl Compatibility<'_> { } } +/// Position of the adapter's version string in [`Compatibility::string`]. +/// +/// `{kernel}/{adapter}/{fingerprint}/{plasticity}/binjgb:{rev}/pokered:{commit}/statefmt:{id}`, +/// so the adapter is segment one. Nothing else in the string may move for a migration to be +/// considered: a different kernel, dataset, plasticity, emulator revision, symbol provenance or +/// state format is a different *fly*, not a different reward rule. +const ADAPTER_SEGMENT: usize = 1; + +/// The environment variable that opts a deploy into the adapter migration. +/// +/// Read by flysim at restore and by `infra/05-deploy.sh`'s compatibility gate. Comma- or +/// whitespace-separated adapter ids, e.g. `FLY_ACCEPT_ADAPTERS=pokered-unique8-v5`. +pub const ACCEPT_ADAPTERS_ENV: &str = "FLY_ACCEPT_ADAPTERS"; + +/// What a build may do with a checkpoint whose compatibility string is not its own. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum RestoreDecision { + /// Byte-identical. Restore it, as every build always has. + Exact, + /// Every segment but the adapter's is identical, this build's adapter says it can migrate + /// from that one, and the operator named it in [`ACCEPT_ADAPTERS_ENV`]. Restore it. + MigrateAdapter { from: String }, + /// Refuse, and say which of the three conditions failed. + Refuse(&'static str), +} + +/// Decide whether `checkpoint`'s compatibility string may be restored under `current`. +/// +/// Three conditions, all required, in the order they are cheapest to explain: +/// +/// 1. the two strings differ in the adapter segment and **nowhere else**; +/// 2. `migrates_from` -- the running adapter's own list -- contains the checkpoint's adapter, so +/// the code that will read that state says out loud that it can; +/// 3. `accepted` -- [`ACCEPT_ADAPTERS_ENV`] as the operator set it for this deploy -- contains it +/// too, so no build ever migrates a run by itself. +/// +/// Condition 2 without condition 3 would make the migration silent; condition 3 without condition +/// 2 would let an operator wave through a pair nobody wrote a migration for. Neither alone is +/// enough, which is why both are here. +pub fn decide( + checkpoint: &str, + current: &str, + migrates_from: &[&str], + accepted: &[String], +) -> RestoreDecision { + if checkpoint == current { + return RestoreDecision::Exact; + } + let old: Vec<&str> = checkpoint.split('/').collect(); + let new: Vec<&str> = current.split('/').collect(); + if old.len() != new.len() { + return RestoreDecision::Refuse("the two compatibility strings do not have the same shape"); + } + let differing: Vec = (0..old.len()).filter(|&index| old[index] != new[index]).collect(); + if differing != [ADAPTER_SEGMENT] { + return RestoreDecision::Refuse( + "more than the adapter version differs; nothing but a reward-rule change can migrate", + ); + } + let from = old[ADAPTER_SEGMENT]; + if !migrates_from.contains(&from) { + return RestoreDecision::Refuse("this build's adapter has no migration from that adapter"); + } + if !accepted.iter().any(|name| name == from) { + return RestoreDecision::Refuse( + "the checkpoint's adapter is not in FLY_ACCEPT_ADAPTERS, so the migration was not \ + asked for", + ); + } + RestoreDecision::MigrateAdapter { from: from.to_string() } +} + +/// Parse [`ACCEPT_ADAPTERS_ENV`]: comma- or whitespace-separated, empty entries dropped. +/// +/// An unset variable and an empty one are the same thing -- no migration -- so that clearing the +/// opt-in is one edit rather than two. +pub fn accepted_adapters(value: Option<&str>) -> Vec { + value + .unwrap_or_default() + .split([',', ' ', '\t', '\n']) + .map(str::trim) + .filter(|entry| !entry.is_empty()) + .map(str::to_string) + .collect() +} + /// `-`: the two things that decide whether a /// binjgb save state written elsewhere can be memcpy'd back in here. pub fn state_format_id() -> String { @@ -94,7 +180,7 @@ mod tests { assert_eq!( fixture().prototype_string(), concat!( - "lif-1ms-f64-v2/pokered-unique8-v5/aa:bb:cc:dd:ee:ff:00/", + "lif-1ms-f64-v2/pokered-unique8-v6/aa:bb:cc:dd:ee:ff:00/", "fly-kc-mbon-rstdp-v2/", "binjgb:c60e138da5a795ebb55e56b11b7e90024e41112c/", "pokered:0cd19d3b877b7dc66d12c7050bed9a7f38154d4b", @@ -102,6 +188,73 @@ mod tests { ); } + fn with_adapter(adapter: &'static str) -> String { + Compatibility { adapter, ..fixture() }.string() + } + + #[test] + fn an_identical_string_restores_without_any_opt_in() { + let current = with_adapter("pokered-unique8-v6"); + assert_eq!(decide(¤t, ¤t, &[], &[]), RestoreDecision::Exact); + } + + #[test] + fn a_v5_checkpoint_restores_under_v6_only_with_the_opt_in() { + let old = with_adapter("pokered-unique8-v5"); + let new = with_adapter("pokered-unique8-v6"); + let migrates = ["pokered-unique8-v5"]; + + assert!(matches!(decide(&old, &new, &migrates, &[]), RestoreDecision::Refuse(_))); + assert_eq!( + decide(&old, &new, &migrates, &accepted_adapters(Some("pokered-unique8-v5"))), + RestoreDecision::MigrateAdapter { from: "pokered-unique8-v5".to_string() } + ); + // And only for a pair the running adapter says it can migrate. + assert!(matches!( + decide(&old, &new, &[], &accepted_adapters(Some("pokered-unique8-v5"))), + RestoreDecision::Refuse(_) + )); + } + + #[test] + fn nothing_but_the_adapter_segment_may_move() { + let migrates = ["pokered-unique8-v5"]; + let accepted = accepted_adapters(Some("pokered-unique8-v5")); + let new = with_adapter("pokered-unique8-v6"); + + // A different dataset, with the same adapter bump, is not a migration. + let other_dataset = Compatibility { + adapter: "pokered-unique8-v5", + dataset_fingerprint: "00:11:22:33:44:55:66", + ..fixture() + } + .string(); + assert!(matches!( + decide(&other_dataset, &new, &migrates, &accepted), + RestoreDecision::Refuse(_) + )); + + // Neither is a different kernel, and neither is a string of another shape. + let other_kernel = + Compatibility { adapter: "pokered-unique8-v5", neural_kernel_version: "lif-1ms-f64-v3", ..fixture() } + .string(); + assert!(matches!( + decide(&other_kernel, &new, &migrates, &accepted), + RestoreDecision::Refuse(_) + )); + assert!(matches!(decide("a/b", &new, &migrates, &accepted), RestoreDecision::Refuse(_))); + } + + #[test] + fn the_opt_in_list_is_separated_by_commas_or_spaces() { + assert!(accepted_adapters(None).is_empty()); + assert!(accepted_adapters(Some(" ")).is_empty()); + assert_eq!( + accepted_adapters(Some("pokered-unique8-v5, pokered-unique8-v4")), + vec!["pokered-unique8-v5".to_string(), "pokered-unique8-v4".to_string()] + ); + } + #[test] fn the_state_format_segment_is_appended_not_interleaved() { let full = fixture().string(); diff --git a/services/flysim/crates/flybrain-gb/src/pokemon_red/catalog.rs b/services/flysim/crates/flybrain-gb/src/pokemon_red/catalog.rs index d4016d3..4e7a0e6 100644 --- a/services/flysim/crates/flybrain-gb/src/pokemon_red/catalog.rs +++ b/services/flysim/crates/flybrain-gb/src/pokemon_red/catalog.rs @@ -1,4 +1,4 @@ -//! The `pokered-unique8-v5` reward catalog. +//! The `pokered-unique8-v6` reward catalog. //! //! A direct port of the prototype's `src/reward/catalog.ts`, including the //! declaration order, which is the order `counts` and `last` serialize in. @@ -20,8 +20,18 @@ pub mod kind { pub const BATTLE: &str = "battle"; pub const BADGE: &str = "badge"; pub const BOUNDARY: &str = "boundary"; + pub const CATCH: &str = "catch"; } +/// What a `catch` of a species this run has already caught pays. +/// +/// Not a multiple of the rule's catalog value, because no binary float scales 0.30 into +/// exactly 0.10: `0.3 * (1.0 / 3.0)` is `0.09999999999999999`, and that number would reach +/// the ticker, the checkpoint and `docs/rewards-learning.md`'s table as itself. `boundary`'s +/// two payouts are 0.05 and 0.10, which a scale of two does express exactly, so that rule +/// still goes through the scaling path. +pub const CATCH_REPEAT_VALUE: f64 = 0.10; + #[derive(Debug, Clone, Copy)] pub struct RewardRule { pub kind: &'static str, @@ -34,7 +44,7 @@ pub struct RewardRule { pub stimulation_ms: u32, } -pub const REWARDS: [RewardRule; 8] = [ +pub const REWARDS: [RewardRule; 9] = [ RewardRule { kind: kind::MILESTONE, label: "Story", @@ -99,6 +109,22 @@ pub const REWARDS: [RewardRule; 8] = [ value: 0.05, stimulation_ms: 100, }, + // The operator's decision of 2026-09-22: the fly is paid for *keeping* a wild Pokémon, not + // only for knocking one out. Appended rather than slotted next to `species` for the same + // reason `boundary` was appended -- the declaration order is the key order `counts` + // serializes in, and every checkpoint already written carries the first eight in this order. + // + // One rule, two payouts, like `boundary`: this value is what a species this run has never + // caught pays, and [`CATCH_REPEAT_VALUE`] is what a repeat pays. The existing `species` + // rule is untouched and still pays 0.50 the first time a species is owned by any means, so + // a first catch of a new species pays 0.50 + 0.30 across two kinds. + RewardRule { + kind: kind::CATCH, + label: "Catch", + trigger: "Wild Pokémon caught; 0.10 for a species already caught; max 3 per species", + value: 0.30, + stimulation_ms: 150, + }, ]; /// Position of `kind` in [`REWARDS`], or `None` for an unknown kind. This is @@ -199,10 +225,34 @@ mod tests { // separate kinds. assert_eq!(rule(kind::BOUNDARY).unwrap().value, 0.05); assert_eq!(rule(kind::BOUNDARY).unwrap().stimulation_ms, 100); + // Nor the prototype's: the operator's catch rule, `pokered-unique8-v6`. + assert_eq!(rule(kind::CATCH).unwrap().value, 0.30); + assert_eq!(CATCH_REPEAT_VALUE, 0.10); + assert_eq!(rule(kind::CATCH).unwrap().stimulation_ms, 150); assert!(rule("blackout").is_none(), "the catalog has no penalties"); assert!(REWARDS.iter().all(|rule| rule.value > 0.0)); } + #[test] + fn the_catch_rule_is_last_so_the_older_key_order_does_not_move() { + let order: Vec<&str> = REWARDS.iter().map(|rule| rule.kind).collect(); + assert_eq!( + order, + vec![ + kind::MILESTONE, + kind::EXPLORATION, + kind::MAP, + kind::SPECIES, + kind::TRAINER, + kind::BATTLE, + kind::BADGE, + kind::BOUNDARY, + kind::CATCH, + ] + ); + assert_eq!(index(kind::CATCH), Some(REWARDS.len() - 1)); + } + #[test] fn empty_counts_lists_every_kind_at_zero() { let counts = Counts::default(); diff --git a/services/flysim/crates/flybrain-gb/src/pokemon_red/mod.rs b/services/flysim/crates/flybrain-gb/src/pokemon_red/mod.rs index 469d708..7d4058f 100644 --- a/services/flysim/crates/flybrain-gb/src/pokemon_red/mod.rs +++ b/services/flysim/crates/flybrain-gb/src/pokemon_red/mod.rs @@ -1,11 +1,12 @@ -//! The Pokémon Red reward adapter, `pokered-unique8-v5`. +//! The Pokémon Red reward adapter, `pokered-unique8-v6`. //! //! A port of the prototype's `src/reward/pokemon-red.ts`. The gates and budgets //! are unchanged; `docs/rewards-learning.md` holds the live rule table and the //! source evidence behind each gate. v4 replaced the 16-rung boot-to-badges //! ladder with the 38 rungs of `docs/design/ladder.md`; v5 adds one reward rule, //! `boundary` (`docs/design/room-escape.md` section 2), which pays the first step -//! next to and the first step onto each of a map's exits. +//! next to and the first step onto each of a map's exits; v6 adds `catch`, the +//! operator's decision of 2026-09-22, which pays for keeping a wild Pokémon. pub mod catalog; #[cfg(test)] @@ -32,13 +33,33 @@ use symbols::ram; /// Adapter version, pinned into the checkpoint compatibility string. /// -/// `v5` is the `boundary` rule. Bumping it is what rejects every checkpoint written -/// by `v4`: the string is compared whole before a restore is attempted, so a ledger -/// that has never recorded a single `boundary:` key can never be resumed as though -/// its exits were already collected. (`v4` was the 38-rung ladder, and rejected -/// `v3` for the same reason: a stored rank that meant "4 badges" on the old ladder -/// could not be read as a rung on the new one.) Pre-launch, so no run is lost. -pub const REWARD_ADAPTER: &str = "pokered-unique8-v5"; +/// `v6` is the `catch` rule. Bumping it is what makes a `v5` checkpoint a decision +/// rather than an accident: the compatibility string is compared whole before a +/// restore is attempted, so a `v5` run is refused by default and resumed only when +/// the operator names it in `FLY_ACCEPT_ADAPTERS` +/// ([`crate::compatibility::RestoreDecision`], `docs/design/flysim.md`). That +/// migration is safe in one direction only, and only for this pair: `v5`'s ledger is +/// a `v6` ledger with the catch counter absent, and an absent counter reads as zero. +/// +/// (`v5` was the `boundary` rule, and rejected `v4` because a ledger that had never +/// recorded a `boundary:` key could not be resumed as though its exits were already +/// collected. `v4` was the 38-rung ladder, and rejected `v3` because a stored rank +/// that meant "4 badges" on the old ladder is not a rung on the new one. Neither of +/// those is a migration: this one is, because nothing a `v5` ledger holds means +/// something different under `v6`.) +pub const REWARD_ADAPTER: &str = "pokered-unique8-v6"; + +/// Adapter ids whose checkpoints `v6` can read. +/// +/// Exactly one, and it is one because the `catch` rule adds a counter and changes nothing else: +/// a `v5` ledger restores as a `v6` ledger with `catchCounts` empty, and every other byte of the +/// state means what it meant. `v4` is not here -- its `seen` ledger holds no `boundary:` keys, so +/// resuming it would pay a second time for every exit the run had already found -- and neither is +/// `v3`, whose stored rank is a rung on a different ladder. +/// +/// Listing an id here is necessary but not sufficient: `FLY_ACCEPT_ADAPTERS` must name it too +/// (`crate::compatibility::decide`, `docs/design/flysim.md`). +pub const MIGRATES_FROM: &[&str] = &["pokered-unique8-v5"]; /// The only cartridge semantic rewards are enabled for. Even the canonical /// pret build stays disabled until reviewed; see `docs/rewards-learning.md`. @@ -60,8 +81,22 @@ pub const SUPPORTED_ROM: &str = /// [`REWARD_ADAPTER`] is the gate that refuses such a checkpoint anyway, and it is /// the right gate, because the objection to loading one is about semantics rather /// than shape. +/// +/// *Not* bumped for the `catch` rule either, and this time the answer matters, +/// because `v5` checkpoints are meant to be restorable under `v6`. The rule adds one +/// counter, `catchCounts`, and nothing else: every other field keeps its name, its +/// shape and its meaning, and a state written without the counter restores with it +/// empty, which is the truth about a run that was never paid for a catch. That is the +/// whole of the documented `v5` -> `v6` migration; see +/// [`crate::compatibility::RestoreDecision`]. pub const STATE_VERSION: u64 = 4; +/// Catch payouts one species may earn in the lifetime of a run's ledger. +/// +/// The same cap and the same reason as the wild-KO rule's three: a species the fly can +/// find over and over is a farm, and three is enough for the behaviour to be learned. +const MAX_CATCH_PAYOUTS: u64 = 3; + const BADGE_NAMES: [&str; 8] = [ "BOULDER", "CASCADE", "THUNDER", "RAINBOW", "SOUL", "MARSH", "VOLCANO", "EARTH", ]; @@ -325,6 +360,26 @@ struct Battle { wild: bool, saw_living: bool, ko: bool, + /// Lifetime `species` payouts when this battle started. + /// + /// The "never owned this run" test for the catch rule, and an exact one: the only + /// thing that can set a `wPokedexOwned` bit during a wild battle is the catch + /// itself, so a `species` payout between the battle starting and the ball keeping + /// the Pokémon *is* that Pokémon being new. It is read this way rather than from + /// `wCapturedMonSpecies` directly because that byte is the cartridge's **internal** + /// species index and the owned bitset is by **Pokédex number**; the two numberings + /// differ and nothing in WRAM converts between them + /// (`docs/design/macros-wram.md` section 2, "species numbering"). + /// + /// `None` for a battle restored from a checkpoint written before this existed, + /// which reads as "cannot tell" and pays the repeat amount rather than guessing + /// generously. + species_at_start: Option, + /// The internal species index `wCapturedMonSpecies` named, once a ball has kept one. + captured: Option, + /// Whether that catch was a species this run had never owned, decided on the frame + /// the capture was observed. + captured_new: bool, } /// Immutable per-sample byte cache. Each address requested during one sample @@ -370,6 +425,10 @@ pub struct PokemonRedReward { tiles: OrderedSet, tile_counts: BTreeMap, wild_wins: BTreeMap, + /// Catch payouts per species, by the cartridge's internal species index as a decimal + /// string. The one field `v6` adds to the checkpoint; absent in a `v5` state, which + /// reads as every species at zero. + catch_counts: BTreeMap, replay_blocked: OrderedSet, counts: Counts, total: f64, @@ -420,6 +479,7 @@ impl PokemonRedReward { tiles: OrderedSet::new(), tile_counts: BTreeMap::new(), wild_wins: BTreeMap::new(), + catch_counts: BTreeMap::new(), replay_blocked: OrderedSet::new(), counts: Counts::default(), total: 0.0, @@ -567,8 +627,9 @@ impl PokemonRedReward { } /// Forget observations a rollback invalidates. Lifetime novelty survives, - /// and every wild-KO key paid so far is blocked from paying again, because - /// after a rollback the same battle could otherwise be replayed for reward. + /// and every wild-KO key and every caught species paid so far is blocked from + /// paying again, because after a rollback the same battle -- or the same catch -- + /// could otherwise be replayed for reward. pub fn clear_transient(&mut self) { self.location.clear(); self.stable = 0; @@ -578,6 +639,10 @@ impl PokemonRedReward { for key in keys { self.replay_blocked.insert(&key); } + let caught: Vec = self.catch_counts.keys().cloned().collect(); + for species in caught { + self.replay_blocked.insert(&format!("catch:{species}")); + } } /// Sample WRAM after one completed frame and return this frame's payouts. @@ -690,14 +755,29 @@ impl PokemonRedReward { if in_battle == 1 || in_battle == 2 || in_battle == 255 { self.mode = "BATTLE".to_string(); self.stable = 0; + let species_paid = self.counts.get(kind::SPECIES); if self.battle.is_none() && in_battle != 255 { self.battle = Some(Battle { key: battle_key(memory, map), wild: in_battle == 1, saw_living: false, ko: false, + species_at_start: Some(species_paid), + captured: None, + captured_new: false, }); } + // The cartridge's own answer to "was one caught": `ram/wram.asm`'s comment on + // this byte is "0 if no mon was captured". `ItemUseBall` zeroes it before every + // throw and writes `wEnemyMonSpecies` into it only on the branch that keeps the + // Pokémon, and `UseBagItem`'s `.returnAfterCapturingMon` zeroes it again on the + // way out of the battle -- so it is non-zero for the hundreds of frames the + // catch's own text and Pokédex screen take, and zero everywhere else. + // + // Read rather than derived from `wPartyCount`, because a catch with a full party + // raises `wBoxCount` instead, and because `wPartyCount` also rises for a gift, a + // trade and a revive out of the PC. + let captured = memory.read8(ram::wCapturedMonSpecies); if let Some(battle) = &mut self.battle { let hp = word(memory, ram::wEnemyMonHP); let max = word(memory, ram::wEnemyMonMaxHP); @@ -710,6 +790,11 @@ impl PokemonRedReward { if battle.saw_living && hp == 0 { battle.ko = true; } + if battle.wild && captured != 0 && battle.captured.is_none() { + battle.captured = Some(captured); + battle.captured_new = + battle.species_at_start.is_some_and(|before| species_paid > before); + } } } else if in_battle == 0 { self.mode = "OVERWORLD".to_string(); @@ -728,6 +813,41 @@ impl PokemonRedReward { } self.wild_wins.insert(battle.key.clone(), (count + 1).min(3)); } + // The catch rule (`docs/rewards-learning.md`, the operator 2026-09-22). + // + // Paid on the way out of the battle rather than on the capture frame, so that + // it lands in the same place the wild-KO payout does and cannot fire twice for + // one battle. `wBattleResult` is 2 on exactly two paths in the game: + // `UseBagItem`'s `.returnAfterCapturingMon`, which is this one, and a link + // battle whose opponent ran (`engine/battle/core.asm`), which this cartridge + // never has. Requiring it as well as the captured species means a byte read + // out of a half-initialised battle cannot pay. + if let Some(species) = battle.captured + && battle.wild + && result == 2 + { + let key = species.to_string(); + let paid = self.catch_counts.get(&key).copied().unwrap_or(0); + if paid < MAX_CATCH_PAYOUTS + && !self.replay_blocked.contains(&format!("catch:{key}")) + { + let value = if battle.captured_new { + catalog::rule(kind::CATCH) + .expect("the catch rule is in the catalog") + .value + } else { + catalog::CATCH_REPEAT_VALUE + }; + self.emit_amount( + &mut emitted, + kind::CATCH, + format!("CAUGHT #{species}"), + value, + brain_ms, + ); + } + self.catch_counts.insert(key, (paid + 1).min(MAX_CATCH_PAYOUTS)); + } } let location = format!("{map}:{x}:{y}"); self.stable = if self.location == location { self.stable + 1 } else { 1 }; @@ -825,13 +945,33 @@ impl PokemonRedReward { label: String, scale: f64, brain_ms: f64, + ) { + let rule = catalog::rule(kind).expect("emit is only called with catalog kinds"); + self.emit_amount(emitted, kind, label, rule.value * scale, brain_ms); + } + + /// [`PokemonRedReward::emit`] with the payout stated outright instead of as a multiple + /// of the catalog value. + /// + /// One rule needs it. `catch` pays 0.30 for a species this run has not caught and 0.10 + /// for one it has, and no binary float scales the first into exactly the second: + /// `0.3 * (1.0 / 3.0)` is `0.09999999999999999`, and that is the number that would reach + /// the ticker and the checkpoint. `boundary`'s pair, 0.05 and 0.10, *is* an exact scale + /// of two, so that rule still goes through [`PokemonRedReward::emit`]. + fn emit_amount( + &mut self, + emitted: &mut Vec, + kind: &'static str, + label: String, + value: f64, + brain_ms: f64, ) { let rule = catalog::rule(kind).expect("emit is only called with catalog kinds"); let event = RewardEvent { kind, label, brain_ms, - value: rule.value * scale, + value, stimulation_ms: rule.stimulation_ms, }; emitted.push(event.clone()); @@ -991,6 +1131,10 @@ impl PokemonRedReward { "tiles": self.tiles.as_slice(), "tileCounts": self.tile_counts, "wildWins": self.wild_wins, + // The one field v6 adds. A v5 state does not carry it and restores with it + // empty, which is the documented v5 -> v6 migration and the truth about a run + // that was never paid for a catch. + "catchCounts": self.catch_counts, "replayBlocked": self.replay_blocked.as_slice(), "counts": self.counts, "total": self.total, @@ -1011,6 +1155,9 @@ impl PokemonRedReward { "wild": battle.wild, "sawLiving": battle.saw_living, "ko": battle.ko, + "speciesAtStart": battle.species_at_start, + "captured": battle.captured, + "capturedNew": battle.captured_new, })), "mode": self.mode, }) @@ -1056,6 +1203,13 @@ impl PokemonRedReward { let counts_raw = counted_record(input.get("counts")).ok_or(BAD_CHECKPOINT)?; let tile_counts_raw = counted_record(input.get("tileCounts")).ok_or(BAD_CHECKPOINT)?; let wild_wins_raw = counted_record(input.get("wildWins")).ok_or(BAD_CHECKPOINT)?; + // Absent in every v5 state, and that absence is the migration: no species has been + // paid for a catch, because the rule did not exist. Present but malformed is still + // an error, the same as every other counter here. + let catch_counts = match input.get("catchCounts") { + None | Some(Value::Null) => BTreeMap::new(), + Some(value) => counted_record(Some(value)).ok_or(BAD_CHECKPOINT)?, + }; let recent = recent_raw .iter() @@ -1078,6 +1232,19 @@ impl PokemonRedReward { wild: value.get("wild").and_then(Value::as_bool).ok_or(BAD_HISTORY)?, saw_living: value.get("sawLiving").and_then(Value::as_bool).ok_or(BAD_HISTORY)?, ko: value.get("ko").and_then(Value::as_bool).ok_or(BAD_HISTORY)?, + // All three are v6's, and all three are optional for the same reason + // `catchCounts` is. A v5 battle carries no `speciesAtStart`, which reads as + // "cannot tell whether the caught species was new" and pays the repeat + // amount: the conservative half of the rule, and at most 0.20 once. + species_at_start: value.get("speciesAtStart").and_then(Value::as_u64), + captured: value + .get("captured") + .and_then(Value::as_u64) + .and_then(|species| u8::try_from(species).ok()), + captured_new: value + .get("capturedNew") + .and_then(Value::as_bool) + .unwrap_or(false), }), }; let replay_blocked = match input.get("replayBlocked") { @@ -1103,6 +1270,7 @@ impl PokemonRedReward { } self.tile_counts = tile_counts; self.wild_wins = wild_wins_raw; + self.catch_counts = catch_counts; self.replay_blocked = replay_blocked.iter().map(String::as_str).collect(); self.counts = Counts::default(); for (key, count) in &counts_raw { @@ -1174,6 +1342,10 @@ impl GameAdapter for PokemonRedReward { REWARD_ADAPTER } + fn migrates_from(&self) -> &'static [&'static str] { + MIGRATES_FROM + } + fn rom_allowed(&self, sha256: &str) -> bool { sha256 == SUPPORTED_ROM } diff --git a/services/flysim/crates/flybrain-gb/src/pokemon_red/symbols.rs b/services/flysim/crates/flybrain-gb/src/pokemon_red/symbols.rs index 30119fa..014b4ad 100644 --- a/services/flysim/crates/flybrain-gb/src/pokemon_red/symbols.rs +++ b/services/flysim/crates/flybrain-gb/src/pokemon_red/symbols.rs @@ -47,6 +47,7 @@ pub mod ram { pub const wBattleType: u16 = 0xd05a; // 53338 pub const wTrainerNo: u16 = 0xd05d; // 53341 pub const wPartyMenuTypeOrMessageID: u16 = 0xd07d; // 53373 + pub const wCapturedMonSpecies: u16 = 0xd11c; // 53532 pub const wForcePlayerToChooseMon: u16 = 0xd11f; // 53535 pub const wTextBoxID: u16 = 0xd125; // 53541 pub const wPartyCount: u16 = 0xd163; // 53603 diff --git a/services/flysim/crates/flybrain-gb/src/pokemon_red/tests.rs b/services/flysim/crates/flybrain-gb/src/pokemon_red/tests.rs index e0bede2..87559c1 100644 --- a/services/flysim/crates/flybrain-gb/src/pokemon_red/tests.rs +++ b/services/flysim/crates/flybrain-gb/src/pokemon_red/tests.rs @@ -83,6 +83,34 @@ impl Fixture { self.visit(x, 0) } + /// One wild battle that ends in a ball keeping the Pokémon, byte for byte as the + /// cartridge writes it at the pinned commit. + /// + /// `InitBattleVariables` clears `wBattleResult`; `ItemUseBall`'s capture branch sets the + /// Pokédex bit (for a species the player did not already own) and writes + /// `wEnemyMonSpecies` into `wCapturedMonSpecies`; `UseBagItem`'s + /// `.returnAfterCapturingMon` then zeroes that byte, sets `wBattleResult` to 2 and leaves + /// the battle. `dex` is the Pokédex *number* minus one, i.e. the bit index, and `None` is a + /// species this run already owns. + fn catch(&mut self, species: u8, dex: Option) -> Vec { + self.memory.set(ram::wBattleResult, 0); + self.memory.set(ram::wIsInBattle, 1); + self.memory.set(ram::wEnemyMonSpecies, species); + self.memory.set(ram::wEnemyMonHP + 1, 10); + self.memory.set(ram::wEnemyMonMaxHP + 1, 10); + let mut events = self.sample(); + if let Some(index) = dex { + self.memory.or(ram::wPokedexOwned + (index >> 3), 1 << (index & 7)); + } + self.memory.set(ram::wCapturedMonSpecies, species); + events.extend(self.sample()); + self.memory.set(ram::wCapturedMonSpecies, 0); + self.memory.set(ram::wBattleResult, 2); + self.memory.set(ram::wIsInBattle, 0); + events.extend(self.sample()); + events + } + /// Write a warp table: `wNumberOfWarps` plus one four-byte `Y, X, warp id, map id` entry per /// `(x, y)`, the layout `ram/wram.asm` documents at the pinned commit. fn warps(&mut self, warps: &[(u8, u8)]) { @@ -320,6 +348,136 @@ fn a_wild_run_capture_or_single_faint_never_pays_a_ko_while_a_verified_ko_does() assert_eq!(f.reward.statistics().counts[kind::BATTLE], 1); } +#[test] +fn a_catch_pays_the_new_species_amount_once_the_repeat_amount_after_and_stops_at_three() { + let mut f = Fixture::new(); + f.sample(); + + // A species this run has never owned: the cartridge sets the Pokédex bit on the way + // through, so the existing `species` rule pays 0.50 and the new rule pays 0.30. + let first = f.catch(0xb0, Some(3)); + assert_eq!(kinds(&first), ["species", "catch"]); + assert_eq!(labels(&first), ["OWNED #4", "CAUGHT #176"]); + assert!((first[0].value - 0.5).abs() < 1e-12, "the species rule is untouched"); + assert!((first[1].value - 0.30).abs() < 1e-12); + + // The same species again: a repeat, twice, and then the cap. + for _ in 0..2 { + let again = f.catch(0xb0, None); + assert_eq!(kinds(&again), ["catch"]); + assert_eq!(again[0].value, 0.10, "the repeat amount is exactly 0.10, not 0.3/3"); + } + assert!(f.catch(0xb0, None).is_empty(), "three payouts per species is the cap"); + assert_eq!(f.reward.statistics().counts[kind::CATCH], 3); + + // Another species starts its own count, and its own 0.30. + let other = f.catch(0x99, Some(0)); + assert_eq!(kinds(&other), ["species", "catch"]); + assert!((other[1].value - 0.30).abs() < 1e-12); +} + +#[test] +fn a_catch_of_a_species_this_run_already_owns_pays_the_repeat_amount() { + let mut f = Fixture::new(); + f.sample(); + // Owned before the battle -- a gift, a trade, an evolution -- so no Pokédex bit is set + // during it and the catch is not a new species. + f.memory.or(ram::wPokedexOwned, 1); + assert_eq!(kinds(&f.sample()), ["species"]); + + let events = f.catch(0x99, None); + assert_eq!(kinds(&events), ["catch"]); + assert_eq!(events[0].value, 0.10); +} + +#[test] +fn nothing_but_a_wild_catch_pays_the_catch_rule() { + // A trainer battle: balls cannot be thrown, and `wIsInBattle` is 2. + let mut f = Fixture::new(); + f.sample(); + f.memory.set(ram::wIsInBattle, 2); + f.memory.set(ram::wEnemyMonHP + 1, 10); + f.memory.set(ram::wEnemyMonMaxHP + 1, 10); + f.sample(); + f.memory.set(ram::wCapturedMonSpecies, 0xb0); + f.sample(); + f.memory.set(ram::wCapturedMonSpecies, 0); + f.memory.set(ram::wBattleResult, 2); + f.memory.set(ram::wIsInBattle, 0); + assert!(f.sample().is_empty(), "a trainer battle never pays the catch rule"); + + // The Safari Zone and the old man's tutorial are excluded a step earlier: the whole + // sample is dropped with a visible mode, so no battle is ever opened. + for battle_type in [1u8, 2] { + let mut f = Fixture::new(); + f.sample(); + f.memory.set(ram::wBattleType, battle_type); + f.memory.set(ram::wIsInBattle, 1); + assert!(f.sample().is_empty()); + f.memory.set(ram::wCapturedMonSpecies, 0xb0); + assert!(f.sample().is_empty()); + f.memory.set(ram::wCapturedMonSpecies, 0); + f.memory.set(ram::wBattleResult, 2); + f.memory.set(ram::wIsInBattle, 0); + f.memory.set(ram::wBattleType, 0); + assert!(f.sample().is_empty()); + assert_eq!(f.reward.statistics().counts[kind::CATCH], 0); + } + + // A ball that missed: `wCapturedMonSpecies` never leaves zero and the battle ends as a + // run or a loss. + let mut f = Fixture::new(); + f.sample(); + f.memory.set(ram::wIsInBattle, 1); + f.memory.set(ram::wEnemyMonHP + 1, 10); + f.memory.set(ram::wEnemyMonMaxHP + 1, 10); + f.sample(); + f.memory.set(ram::wIsInBattle, 0); + assert!(f.sample().is_empty()); +} + +#[test] +fn a_rollback_cannot_replay_a_catch() { + let mut f = Fixture::new(); + f.sample(); + assert_eq!(kinds(&f.catch(0xb0, Some(3))), ["species", "catch"]); + + f.reward.clear_transient(); + let state = f.reward.export_state(); + f.reward.import_state(&state).unwrap(); + assert!(f.catch(0xb0, None).is_empty(), "an already-paid species cannot pay after rollback"); + assert_eq!(f.reward.statistics().counts[kind::CATCH], 1); +} + +#[test] +fn a_v5_state_restores_under_v6_with_the_catch_counter_at_zero() { + let mut f = Fixture::new(); + f.sample(); + f.catch(0xb0, Some(3)); + let v6 = f.reward.export_state(); + assert_eq!(v6["catchCounts"], json!({ "176": 1 })); + + // The v5 shape is this one without the counter the rule added: same `version`, same field + // names, same meanings. That is the whole of the documented migration. + let mut v5 = v6.clone(); + v5.as_object_mut().unwrap().remove("catchCounts"); + assert_eq!(v5["version"], json!(STATE_VERSION), "v5 and v6 states share a schema version"); + + let mut restored = PokemonRedReward::new(); + restored.import_state(&v5).unwrap(); + let mut expected = v6.clone(); + expected["catchCounts"] = json!({}); + assert_eq!(restored.export_state(), expected, "the counter starts at 0, nothing else moves"); + + // A genuine v5 `counts` object carries eight kinds and no `catch`, which reads as zero. + let mut older = v5.clone(); + older["counts"].as_object_mut().unwrap().remove("catch"); + let mut restored = PokemonRedReward::new(); + restored.import_state(&older).unwrap(); + assert_eq!(restored.statistics().counts[kind::CATCH], 0); + assert_eq!(restored.statistics().counts[kind::SPECIES], 1); +} + #[test] fn a_repeated_wild_ko_decays_then_stops() { let mut f = Fixture::new(); @@ -378,6 +536,7 @@ fn malformed_checkpoint_fields_are_named_in_the_error() { ("counts", json!([])), ("tileCounts", json!("not a record")), ("wildWins", json!(3)), + ("catchCounts", json!(3)), ] { let mut broken = good.clone(); broken[field] = wrong; @@ -706,7 +865,8 @@ fn the_recent_ticker_keeps_the_newest_eight_events_newest_first() { #[test] fn the_adapter_reports_its_identity_and_pinned_rom() { let reward = PokemonRedReward::new(); - assert_eq!(reward.id(), "pokered-unique8-v5"); + assert_eq!(reward.id(), "pokered-unique8-v6"); + assert_eq!(reward.migrates_from(), ["pokered-unique8-v5"]); assert!(reward.rom_allowed(SUPPORTED_ROM)); assert!(!reward.rom_allowed( "5ca7ba01642a3b27b0cc0b5349b52792795b62d3ed977e98a09390659af96b7b" @@ -716,6 +876,9 @@ fn the_adapter_reports_its_identity_and_pinned_rom() { assert_eq!(symbols::ram::wNumberOfWarps, 0xd3ae); assert_eq!(symbols::ram::wWarpEntries, 0xd3af); assert_eq!(symbols::ram::wCurMapConnections, 0xd370); + // Resolved from ram/wram.asm by services/flysim/tools/resolve_wram.py, bracketed by + // wFontLoaded and wForcePlayerToChooseMon; never written out by hand. + assert_eq!(symbols::ram::wCapturedMonSpecies, 0xd11c); assert_eq!(symbols::MILESTONES.len(), 17); } diff --git a/services/flysim/crates/flybus/examples/demo.rs b/services/flysim/crates/flybus/examples/demo.rs index 12830de..73f3fb5 100644 --- a/services/flysim/crates/flybus/examples/demo.rs +++ b/services/flysim/crates/flybus/examples/demo.rs @@ -99,8 +99,17 @@ pub async fn run() -> Result, Box> { "published sequence {} to {} subscriber(s)", receipt.topic_sequence, receipt.subscribers )); - // The producer lets go of its own hold; the delivery keeps the bytes alive. + // The producer lets go of its own hold; the delivery keeps the bytes alive. The release + // travels the control lane like any other operation, so the count below waits for it + // instead of reading a number that may still include it. drop(frame); + let released = Instant::now() + Duration::from_secs(10); + while router.stats().artifact_roots > 1 { + if Instant::now() > released { + return Err("the producer's own hold was never released".into()); + } + tokio::time::sleep(Duration::from_millis(1)).await; + } let message = frames.next().await.ok_or("the subscription closed")?; let image = message.artifact("frame")?; diff --git a/services/flysim/crates/flybus/tests/conformance_artifacts.rs b/services/flysim/crates/flybus/tests/conformance_artifacts.rs index cf8a8d2..2eb874b 100644 --- a/services/flysim/crates/flybus/tests/conformance_artifacts.rs +++ b/services/flysim/crates/flybus/tests/conformance_artifacts.rs @@ -179,7 +179,9 @@ async fn collection_waits_for_every_retained_owner(via: Via) { s.owners == 0 && s.sealed_artifacts == 0 && s.store_bytes == 0 }) .await; - assert_eq!(e.files("sealed"), 0); + // The unlink follows the registry update, outside the router lock: wait for the file to + // go rather than assume the two happen together. + e.settle_files("sealed", 0).await; } // ------------------------------------------------------------------------------------------- @@ -521,7 +523,7 @@ async fn disconnect_abandons_an_unsealed_writer(via: Via) { s.artifacts == 0 && s.store_bytes == 0 && s.owners == 0 }) .await; - assert_eq!(e.files("staging"), 0); + e.settle_files("staging", 0).await; } /// An abrupt disconnect must release an explicit hold too, when it was the object's only root. @@ -538,7 +540,7 @@ async fn disconnect_releases_an_explicit_hold(via: Via) { s.sealed_artifacts == 0 && s.owners == 0 && s.store_bytes == 0 }) .await; - assert_eq!(e.files("sealed"), 0); + e.settle_files("sealed", 0).await; } /// A vanished subscriber must give up both a delivery it already holds and one still queued diff --git a/services/flysim/crates/flybus/tests/conformance_routing.rs b/services/flysim/crates/flybus/tests/conformance_routing.rs index 3c0ef30..d8aaa7f 100644 --- a/services/flysim/crates/flybus/tests/conformance_routing.rs +++ b/services/flysim/crates/flybus/tests/conformance_routing.rs @@ -420,14 +420,20 @@ async fn bounded_overflow_rolls_back_all_artifact_roots(via: Via) { } /// bus-v1 section 7: "New subscriptions with replayLatest enqueue it before subsequent accepted -/// publications." A fresh `latest` subscription's replay claims its first in-flight credit -/// immediately (there is nothing else competing for it yet), so a publish accepted right after -/// subscribing must still be observed strictly after the replay, never ahead of or merged with -/// it: each keeps its own delivery. +/// publications ... bounded mode preserves that order, while latest mode may coalesce it before +/// delivery under the ordinary latest rule." The replay is enqueued under the subscribe lock, so +/// a publication admitted after `subscribe` returned is always behind it; what the two modes do +/// with that order is what differs, and one racing publication is put to both at once. +/// +/// The bounded subscription must deliver both values, replay first. The latest subscription +/// either does the same or replaces the still-queued replay, and the router says which in the +/// racing publication's own `replaced` count rather than the test guessing from how fast the +/// dispatcher ran: what it may never do is reorder the two or lose the newer value. async fn latest_replay_is_ordered_ahead_of_a_racing_publish(via: Via) { let e = env(via).await; let admin = e.client("admin").await; let reader = e.client("reader").await; + let viewer = e.client("viewer").await; admin .declare_topic("t.replay-race", Retained::Latest) .await @@ -436,26 +442,66 @@ async fn latest_replay_is_ordered_ahead_of_a_racing_publish(via: Via) { .publish("t.replay-race", obj(json!({"v": "old"})), &[]) .await .unwrap(); - let mut sub = reader + let mut fifo = reader + .subscribe( + "t.replay-race", + SubscriptionConfig::bounded().in_flight(1).replay(true), + ) + .await + .unwrap(); + let mut coalescing = viewer .subscribe( "t.replay-race", SubscriptionConfig::latest().in_flight(1).replay(true), ) .await .unwrap(); - admin + let racing = admin .publish("t.replay-race", obj(json!({"v": "new"})), &[]) .await .unwrap(); - let first = within("the replay arrives first", sub.next()) + assert_eq!( + racing.subscribers, 2, + "one publication, admitted behind both replays" + ); + + let first = within("the replay arrives first", fifo.next()) .await .unwrap(); assert_eq!(first.payload()["v"], "old"); drop(first); // the sole in-flight credit must return before the queued second value moves - let second = within("the racing publish follows, not coalesced away", sub.next()) + let second = within("the racing publish follows, not coalesced away", fifo.next()) .await .unwrap(); - assert_eq!(second.payload()["v"], "new"); + assert_eq!( + (second.payload()["v"].as_str(), second.replaced()), + (Some("new"), 0), + "bounded preserves the order and coalesces nothing" + ); + + let m = within("the latest subscription's first delivery", coalescing.next()) + .await + .unwrap(); + if racing.replaced == 1 { + assert_eq!( + (m.payload()["v"].as_str(), m.replaced()), + (Some("new"), 1), + "a replay still queued is replaced by the newer value, and the delivery says so" + ); + drop(m); + quiet("nothing behind a coalesced replay", coalescing.next()).await; + } else { + assert_eq!( + (racing.replaced, m.payload()["v"].as_str(), m.replaced()), + (0, Some("old"), 0), + "a replay already in flight keeps its own delivery" + ); + drop(m); + let after = within("the racing publish follows it", coalescing.next()) + .await + .unwrap(); + assert_eq!(after.payload()["v"], "new"); + } } /// bus-v1 section 7: clearing releases only the retained root; a later `replayLatest` diff --git a/services/flysim/crates/flybus/tests/integration.rs b/services/flysim/crates/flybus/tests/integration.rs index c4c95ba..b658f63 100644 --- a/services/flysim/crates/flybus/tests/integration.rs +++ b/services/flysim/crates/flybus/tests/integration.rs @@ -1,6 +1,11 @@ //! bus-v1 section 11 item 6: two parallel fake agents, complete-batch environment RPC, //! committed snapshot publication and a deliberately slow presentation consumer, all over one //! router. Generic services only; nothing here knows what a brain or a game is. +//! +//! The presentation consumer is held until the publisher's own completion is observed, so the +//! latest subscription has to coalesce instead of happening to: section 7 lets a latest +//! subscriber miss values, it does not oblige it to, and a test that demands a miss it cannot +//! force is asserting how fast the machine is. mod common; @@ -148,20 +153,31 @@ async fn session_over_one_router(via: Via) { ) .await .unwrap(); + // A renderer that does not read a single snapshot until the publisher has finished every + // one of them. The hold ends on the publisher's observed completion, never on a timer. + let (release, held) = tokio::sync::oneshot::channel::<()>(); let presenting = tokio::spawn(async move { + held.await.unwrap(); let mut seen = Vec::new(); + let mut coalesced = 0; while let Some(m) = slow.next().await { + let (sequence, replaced) = (m.topic_sequence(), m.replaced()); let frame = m.artifact("frame").unwrap(); drop(m); tokio::time::sleep(Duration::from_millis(25)).await; // a slow renderer let bytes = frame.read_all().await.unwrap(); let step = bytes[0] as u64; + assert_eq!( + sequence, step, + "a delivery carries the frame of the snapshot it announces" + ); + coalesced += replaced; seen.push((step, frame.reference().artifact_id.clone())); if step == STEPS { break; } } - seen + (seen, coalesced) }); let recorder = e.client("recorder").await; let mut all = recorder @@ -179,6 +195,7 @@ async fn session_over_one_router(via: Via) { seq }); + let mut replaced_at_admission = 0; for step in 1..=STEPS { let advanced = coordinator .call_and_wait( @@ -226,8 +243,16 @@ async fn session_over_one_router(via: Via) { ) .await .unwrap(); - assert_eq!(receipt.topic_sequence, step); + assert_eq!( + (receipt.topic_sequence, receipt.subscribers), + (step, 2), + "both subscriptions accept every publication: the stalled latest spectator neither \ + refuses one nor drops out of the fan-out" + ); + replaced_at_admission += receipt.replaced; } + // The publisher is finished, observably, so the renderer may start. + release.send(()).unwrap(); let recorded = within("recorder", recording).await.unwrap(); assert_eq!( @@ -235,17 +260,30 @@ async fn session_over_one_router(via: Via) { (1..=STEPS).map(|s| (s, s)).collect::>(), "the bounded recorder misses nothing" ); - let presented = within("presenter", presenting).await.unwrap(); - assert_eq!( - presented.last().unwrap().0, - STEPS, - "the slow consumer ends on the latest snapshot" - ); + let (presented, coalesced) = within("presenter", presenting).await.unwrap(); + let steps: Vec = presented.iter().map(|(s, _)| *s).collect(); assert!( - presented.len() < STEPS as usize, - "the slow consumer skipped snapshots: {presented:?}" + steps.windows(2).all(|w| w[0] < w[1]), + "what a latest subscription does deliver arrives in publication order: {presented:?}" + ); + assert_eq!( + steps.last().copied(), + Some(STEPS), + "the slow consumer ends on the latest snapshot: {presented:?}" + ); + assert_eq!( + steps, + vec![1, STEPS], + "held for the whole run, the subscription keeps the one delivery already in flight and \ + one replaceable queued value, so the renderer sees the first snapshot and the last, \ + and the eighteen between them were coalesced: {presented:?}" + ); + assert_eq!( + (coalesced, replaced_at_admission), + (STEPS - 2, STEPS - 2), + "every snapshot the renderer missed is counted as a replacement, to the publisher at \ + admission and to the renderer on its next delivery: none is lost silently" ); - assert!(presented.windows(2).all(|w| w[0].0 < w[1].0)); for t in agents { t.abort(); diff --git a/services/flysim/crates/flybus/tests/sol_review_races.rs b/services/flysim/crates/flybus/tests/sol_review_races.rs index 9d26b1b..793df64 100644 --- a/services/flysim/crates/flybus/tests/sol_review_races.rs +++ b/services/flysim/crates/flybus/tests/sol_review_races.rs @@ -266,13 +266,14 @@ async fn pending_connections_are_bounded_and_hello_expires() { let router = Router::new(config).unwrap(); let pending = router.connect_in_memory_as("first"); - tokio::time::timeout(Duration::from_millis(20), async { + // Registration happens on the router's own task. How long that takes is this box's + // business; that it happens is the router's. + within("the pending connection occupies the only slot", async { while router.stats().connections != 1 { - tokio::task::yield_now().await; + tokio::time::sleep(Duration::from_millis(1)).await; } }) - .await - .unwrap(); + .await; assert_eq!(router.stats().connections, 1); let refused = Client::connect( router.connect_in_memory_as("second"), @@ -280,7 +281,14 @@ async fn pending_connections_are_bounded_and_hello_expires() { ) .await; assert_eq!(refused.unwrap_err().code, ErrorCode::RouterLost); - tokio::time::sleep(Duration::from_millis(80)).await; + // The 40 ms hello timeout expires on the router's clock: wait for the expiry to be + // observed rather than sleep past it and read the count once. + within("the pending Hello expires", async { + while router.stats().connections != 0 { + tokio::time::sleep(Duration::from_millis(1)).await; + } + }) + .await; assert_eq!(router.stats().connections, 0, "pending Hello timed out"); drop(pending); diff --git a/services/flysim/crates/flysim/src/lib.rs b/services/flysim/crates/flysim/src/lib.rs index 65a1b70..be481b0 100644 --- a/services/flysim/crates/flysim/src/lib.rs +++ b/services/flysim/crates/flysim/src/lib.rs @@ -30,6 +30,7 @@ pub mod metrics; pub mod pacing; pub mod profile; pub mod ratelimit; +pub mod reset; pub mod sdnotify; pub mod simloop; pub mod snapshot; diff --git a/services/flysim/crates/flysim/src/main.rs b/services/flysim/crates/flysim/src/main.rs index 5d2aca3..b0c4a89 100644 --- a/services/flysim/crates/flysim/src/main.rs +++ b/services/flysim/crates/flysim/src/main.rs @@ -42,6 +42,15 @@ struct Args { /// the "nothing to compare" case for a fresh container. #[arg(long, value_name = "DIR")] print_state_compatibility: Option, + + /// Restart the run from the milestone archive for this ladder rung, and exit. + /// + /// Run with flysim stopped: it rewrites both checkpoint stores. + /// `infra/bin/fly-reset-to-milestone` is the operator-facing wrapper and the sequence + /// around it is in `infra/docs/runbook.md`. The current state is copied to a dated + /// directory first, so this is reversible by hand. + #[arg(long, value_name = "RANK")] + reset_to_milestone: Option, } fn main() -> Result<()> { @@ -66,6 +75,17 @@ fn main() -> Result<()> { return Ok(()); } + if let Some(rank) = args.reset_to_milestone { + let stamp = flysim::reset::utc_stamp(flysim::eventlog::now_wall_ms()); + let durable = config.paths.save_dir.clone(); + let hot = config.paths.hot_dir.clone(); + let archive = flysim::reset::default_archive_dir(&durable, &stamp); + for line in flysim::reset::reset_to_milestone(&durable, &hot, rank, &archive)? { + println!("{line}"); + } + return Ok(()); + } + flysim::run(config) } diff --git a/services/flysim/crates/flysim/src/reset.rs b/services/flysim/crates/flysim/src/reset.rs new file mode 100644 index 0000000..acf575e --- /dev/null +++ b/services/flysim/crates/flysim/src/reset.rs @@ -0,0 +1,450 @@ +//! Restarting a run from an earlier rung, on disk, with flysim stopped. +//! +//! The operator's decision of 2026-09-22 was "restart the live run from an early checkpoint +//! instead of from scratch". `FLY_RESET_STATE=1` cannot do that: it archives everything and the +//! next start warms up a fresh fly. What this does instead is promote one milestone archive -- +//! `milestone-.checkpoint`, which `store::Store::commit` writes at the first commit at a new +//! best rank and which no rotation ever unlinks -- to being the only thing either store will +//! restore. +//! +//! `infra/bin/fly-reset-to-milestone` is the operator-facing wrapper; it refuses to run while +//! flysim is up, calls `flysim --reset-to-milestone N`, and fixes ownership afterwards. The work +//! is here rather than in that script because two of the steps are inside the `FLYSIM01` +//! envelope: the ratchet's attempts and recoveries counters live in the checkpoint's manifest, +//! and a shell script has no business rewriting one. +//! +//! What it does, in order, and nothing else: +//! +//! 1. **archives** every file in the durable and hot stores into a dated directory, by copying, +//! so a step that fails later has destroyed nothing; +//! 2. **rewrites** the rung's archive with the ratchet's `attempts` and `recoveries` at zero, so +//! the recovery budget is not already spent when the restarted run begins. `best` is left +//! alone: the archive's own `best` is the rung it was taken at, which is exactly what the +//! restarted run is at, and the rank the stream shows is recomputed by the adapter from the +//! restored game state anyway; +//! 3. **installs** it as the newest generation in both stores, so the restore order +//! (`store::restore_order`: hot latest, hot previous, durable latest, ...) reaches it first; +//! 4. **clears** the milestone archives above N -- rungs the run had reached and is now below -- +//! and the generation files of the run being abandoned; +//! 5. **clears the session ledgers**: the event log `events.jsonl` and its rotations. The +//! checkpoint carries `lastEventId`, so restoring an old checkpoint over a newer log would +//! re-issue ids the log already holds. The macro layer's own session ledgers (blocked, +//! talked, reached, pushed-back) are memory-only by contract +//! (`docs/design/macros.md` section 12.1: "a restored run offers every target once more"), +//! so stopping flysim is what resets those and this has nothing to do. + +use std::collections::BTreeMap; +use std::path::{Path, PathBuf}; + +use anyhow::{Context, Result, bail}; + +use crate::store::{self, Store, StoreManifest}; + +/// Generations kept by the stores this tool writes. Only used for the rotation bound, which +/// this tool does not trigger; the durable store's own value. +const KEEP_GENERATIONS: usize = 8; + +/// A generation number no commit ever allocates, so `Store::candidates` drops the +/// generation-file half of an archive entry and offers only `milestone-.checkpoint`. +/// +/// `Sim::boot` allocates `highest_generation() + 1`, which is 1 or more, so 0 names no file. +/// The milestone archives below the rung being restored are kept exactly this way: still on +/// disk, still restorable as a deeper fallback, and with no generation file pretending to be +/// their contents. +const NO_GENERATION: u64 = 0; + +/// What the reset did, one line per step, for the operator's terminal and the run record. +pub type Report = Vec; + +/// Promote `rank`'s milestone archive to be the restore source of both stores. +/// +/// `archive` is the dated directory the current state is copied into; it must not exist. +/// Refuses if the milestone archive is missing, which is the "that rung was never reached" +/// case and the one mistake worth refusing rather than guessing at. +pub fn reset_to_milestone( + durable_dir: &Path, + hot_dir: &Path, + rank: u32, + archive: &Path, +) -> Result { + let durable = Store::new(durable_dir, KEEP_GENERATIONS); + let hot = Store::new(hot_dir, KEEP_GENERATIONS); + let source = durable.archive_path(rank); + if !source.is_file() { + bail!( + "no milestone archive for rung {rank}: {} does not exist. `ls {}` shows the rungs \ + this run actually reached.", + source.display(), + durable_dir.display() + ); + } + if archive.exists() { + bail!("the archive directory {} already exists", archive.display()); + } + + let mut report: Report = Vec::new(); + + // 1. Copy everything aside first. + let copied_durable = copy_tree(durable_dir, &archive.join("durable"))?; + let copied_hot = copy_tree(hot_dir, &archive.join("hot"))?; + report.push(format!( + "archived {copied_durable} durable and {copied_hot} hot files to {}", + archive.display() + )); + + // 2. Zero the two recovery counters inside the envelope. + let mut checkpoint = store::load(&source) + .with_context(|| format!("decoding {}", source.display()))?; + let spent = (checkpoint.runtime.ratchet.attempts, checkpoint.runtime.ratchet.recoveries); + checkpoint.runtime.ratchet.attempts = 0; + checkpoint.runtime.ratchet.recoveries = 0; + + let generation = durable.highest_generation().max(hot.highest_generation()) + 1; + checkpoint.runtime.generation = generation; + let bytes = store::encode(&checkpoint.agent, &checkpoint.runtime)?; + report.push(format!( + "rung {rank} (best {}, ladder rank recomputed from the game state): ratchet attempts \ + {} -> 0, recoveries {} -> 0", + checkpoint.runtime.ratchet.best, spent.0, spent.1 + )); + + // 3/4. Clear both stores, keeping the milestone archives at or below this rung, and write + // the promoted state as the newest generation of each. + let kept = clear_store(durable_dir, Some(rank))?; + clear_store(hot_dir, None)?; + report.push(format!( + "cleared the hot store and every milestone archive above rung {rank}; kept {} at or \ + below it: {kept:?}", + kept.len() + )); + + // The hot store lives on a tmpfs that a stopped container may not have mounted yet, so it + // is created rather than assumed; the durable one already exists or the milestone archive + // above could not have been read. + durable.create()?; + hot.create()?; + store::write_atomic(&durable.generation_path(generation), &bytes)?; + store::write_atomic(&durable.archive_path(rank), &bytes)?; + store::write_atomic(&hot.generation_path(generation), &bytes)?; + + let mut archives: BTreeMap = + kept.iter().map(|rung| (*rung, NO_GENERATION)).collect(); + archives.insert(rank, generation); + let durable_manifest = StoreManifest { + generation, + latest: Some(generation), + previous: None, + archives, + }; + write_manifest(&durable, &durable_manifest)?; + write_manifest( + &hot, + &StoreManifest { + generation, + latest: Some(generation), + previous: None, + archives: BTreeMap::new(), + }, + )?; + report.push(format!( + "generation {generation} is now hot latest and durable latest in {} and {}", + hot_dir.display(), + durable_dir.display() + )); + + // 5. The session ledgers. + let logs = remove_matching(durable_dir, |name| { + name == "events.jsonl" || (name.starts_with("events-") && name.ends_with(".jsonl")) + })?; + report.push(format!( + "reset the session ledgers: {logs} event-log files removed (the macro layer's are \ + memory-only and are reset by stopping flysim)" + )); + + Ok(report) +} + +fn write_manifest(store: &Store, manifest: &StoreManifest) -> Result<()> { + store.create()?; + store::write_atomic(&store.manifest_path(), &serde_json::to_vec_pretty(manifest)?) +} + +/// Copy every regular file of `from` into `to`, creating `to`. A missing source is zero files, +/// not an error: the hot store lives on a tmpfs that a stopped container does not have. +fn copy_tree(from: &Path, to: &Path) -> Result { + if !from.is_dir() { + return Ok(0); + } + std::fs::create_dir_all(to) + .with_context(|| format!("creating {}", to.display()))?; + let mut copied = 0; + for entry in std::fs::read_dir(from)?.flatten() { + if !entry.file_type().is_ok_and(|kind| kind.is_file()) { + continue; + } + std::fs::copy(entry.path(), to.join(entry.file_name())) + .with_context(|| format!("copying {}", entry.path().display()))?; + copied += 1; + } + Ok(copied) +} + +/// Remove every checkpoint, tmp file and manifest from `dir`, keeping `milestone-.checkpoint` +/// for `r <= keep_up_to`. Returns the rungs kept, ascending. +fn clear_store(dir: &Path, keep_up_to: Option) -> Result> { + let mut kept = Vec::new(); + if !dir.is_dir() { + return Ok(kept); + } + for entry in std::fs::read_dir(dir)?.flatten() { + let name = entry.file_name().to_string_lossy().to_string(); + let milestone = name + .strip_prefix("milestone-") + .and_then(|rest| rest.strip_suffix(".checkpoint")) + .and_then(|rung| rung.parse::().ok()); + let remove = match milestone { + // The promoted rung's own archive is rewritten straight after this, so it is + // removed here like the rest and reinstated with the counters cleared. + Some(rung) => match keep_up_to { + Some(limit) if rung < limit => { + kept.push(rung); + false + } + _ => true, + }, + None => { + name == "manifest.json" + || name.ends_with(".checkpoint") + || name.ends_with(".checkpoint.tmp") + } + }; + if remove { + std::fs::remove_file(entry.path()) + .with_context(|| format!("removing {}", entry.path().display()))?; + } + } + kept.sort_unstable(); + Ok(kept) +} + +fn remove_matching(dir: &Path, wanted: impl Fn(&str) -> bool) -> Result { + if !dir.is_dir() { + return Ok(0); + } + let mut removed = 0; + for entry in std::fs::read_dir(dir)?.flatten() { + if wanted(&entry.file_name().to_string_lossy()) { + std::fs::remove_file(entry.path())?; + removed += 1; + } + } + Ok(removed) +} + +/// `YYYYMMDDTHHMMSSZ` in UTC, for the dated archive directory's name. +/// +/// Built on the event log's own calendar conversion, which is the service's only one; the time of +/// day is arithmetic on the same millisecond count. +pub fn utc_stamp(wall_ms: u64) -> String { + let second_of_day = (wall_ms % 86_400_000) / 1_000; + format!( + "{}T{:02}{:02}{:02}Z", + crate::eventlog::utc_day(wall_ms), + second_of_day / 3_600, + (second_of_day / 60) % 60, + second_of_day % 60, + ) +} + +/// The default dated archive directory: a sibling of the durable store, which is its own +/// mountpoint and so cannot be renamed -- the same shape `infra/05-deploy.sh` uses for +/// `FLY_RESET_STATE=1`. +pub fn default_archive_dir(durable_dir: &Path, stamp: &str) -> PathBuf { + let mut name = durable_dir.as_os_str().to_os_string(); + name.push(format!(".reset-{stamp}")); + PathBuf::from(name) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A real `FLYSIM01` envelope, small but structurally complete, so these tests decode and + /// re-encode what the running service writes rather than a stand-in. + fn envelope(generation: u64, ratchet: flybrain_gb::RatchetState) -> Vec { + use flybrain_core::decoder::DecoderState; + use flybrain_core::lif::LifState; + use flybrain_core::ordered::NumberMap; + use flybrain_core::plasticity::PlasticityState; + + let agent = flybrain_core::agent::AgentState { + version: 1, + remainder: 0.75, + warmed_up: true, + network: LifState { + membrane: vec![0.5, -0.25], + refractory: vec![0, 3], + last_spike_ms: vec![-1_000_000.0, 12.0], + visual_drive: vec![0.1], + rng: -12_345, + reward_remaining: 40.0, + ms: 9_000.0, + population_rate: 1.5, + rates: NumberMap::from_pairs([("forward", 2.0)]), + plasticity: PlasticityState { + version: "fly-kc-mbon-rstdp-v2".to_string(), + topology: 42, + enabled: true, + updates: 3.0, + signal: 0.25, + gains: vec![1.0, 0.9], + traces: vec![0.0, 0.1], + touched: vec![0.0, 8_000.0], + }, + }, + decoder: DecoderState { + version: 4, + calibrated: true, + baseline: NumberMap::from_pairs([("forward", 1.0)]), + held_until: NumberMap::new(), + next_allowed: NumberMap::new(), + next_decision: 100.0, + current: None, + fatigue: NumberMap::new(), + macro_next_decision: 0.0, + macro_current: None, + macro_fatigue: NumberMap::new(), + }, + }; + let runtime = store::RuntimeState { + generation, + wall_ms: 1_700_000_000_000, + rom_sha256: "ab".repeat(32), + emulator_frame: 12_345, + compatibility: "kernel/pokered-unique8-v6/fingerprint".to_string(), + speed: 1.0, + buttons: 0, + rank_since_ms: 4_242.0, + last_event_id: 77, + reward: serde_json::json!({ "version": 4, "total": 1.25 }), + ratchet, + emulator: vec![7; 64], + framebuffer: vec![9; 32], + ratchet_game: vec![1, 2, 3], + ratchet_frame: vec![4, 5, 6], + }; + store::encode(&agent, &runtime).unwrap() + } + + /// A store dir holding a milestone archive for each rung in `rungs`, their generations, a + /// manifest and an event log, all written through the store's own commit path. + fn state_dir(root: &Path, rungs: &[u32], ratchet: flybrain_gb::RatchetState) -> Store { + let store = Store::new(root, KEEP_GENERATIONS); + store.create().unwrap(); + for (index, rung) in rungs.iter().enumerate() { + let generation = index as u64 + 1; + store.commit(generation, &envelope(generation, ratchet), Some(*rung)).unwrap(); + } + std::fs::write(root.join("events.jsonl"), b"{}\n").unwrap(); + std::fs::write(root.join("events-20260921.jsonl"), b"{}\n").unwrap(); + store + } + + #[test] + fn a_missing_rung_is_refused_and_nothing_is_touched() { + let tmp = tempfile::tempdir().unwrap(); + let durable = tmp.path().join("state"); + let hot = tmp.path().join("hot"); + state_dir(&durable, &[3, 5], flybrain_gb::RatchetState::default()); + let before = std::fs::read_dir(&durable).unwrap().flatten().count(); + + let error = reset_to_milestone(&durable, &hot, 9, &tmp.path().join("archive")) + .unwrap_err() + .to_string(); + assert!(error.contains("no milestone archive for rung 9"), "{error}"); + assert_eq!(std::fs::read_dir(&durable).unwrap().flatten().count(), before); + assert!(!tmp.path().join("archive").exists(), "nothing was archived"); + } + + #[test] + fn the_rung_becomes_both_stores_latest_with_the_recovery_budget_back() { + let tmp = tempfile::tempdir().unwrap(); + let durable = tmp.path().join("state"); + let hot = tmp.path().join("hot"); + let spent = flybrain_gb::RatchetState { + best: 5, + attempts: 3, + recoveries: 11, + ..flybrain_gb::RatchetState::default() + }; + state_dir(&durable, &[3, 5, 9, 11], spent); + state_dir(&hot, &[11], spent); + let archive = tmp.path().join("archive-20260922"); + + let report = reset_to_milestone(&durable, &hot, 5, &archive).unwrap(); + assert!(report.iter().any(|line| line.contains("attempts 3 -> 0")), "{report:?}"); + + // Everything that was there is in the archive. + assert!(archive.join("durable/milestone-11.checkpoint").is_file()); + assert!(archive.join("durable/events.jsonl").is_file()); + assert!(archive.join("hot/manifest.json").is_file()); + + // The rungs above 5 are gone; the ones below it stay as deeper fallbacks. + assert!(!durable.join("milestone-9.checkpoint").exists()); + assert!(!durable.join("milestone-11.checkpoint").exists()); + assert!(durable.join("milestone-3.checkpoint").is_file()); + assert!(durable.join("milestone-5.checkpoint").is_file()); + assert!(!durable.join("events.jsonl").exists()); + assert!(!durable.join("events-20260921.jsonl").exists()); + assert!(!hot.join("milestone-11.checkpoint").exists()); + + // Both stores restore the rung, and the counters are back. + for store in [Store::new(&hot, KEEP_GENERATIONS), Store::new(&durable, KEEP_GENERATIONS)] { + let candidates = store.candidates("x"); + let first = store::load(&candidates[0].path).unwrap(); + assert_eq!(first.runtime.ratchet.best, 5); + assert_eq!((first.runtime.ratchet.attempts, first.runtime.ratchet.recoveries), (0, 0)); + } + // ... including through the promoted milestone archive itself. + let archived = store::load(&durable.join("milestone-5.checkpoint")).unwrap(); + assert_eq!((archived.runtime.ratchet.attempts, archived.runtime.ratchet.recoveries), (0, 0)); + + // The rungs below it are offered, and only as their own archive files. + let manifest = Store::new(&durable, KEEP_GENERATIONS).manifest().unwrap(); + assert_eq!(manifest.archives.get(&3), Some(&NO_GENERATION)); + assert_eq!(manifest.latest, manifest.archives.get(&5).copied()); + assert!( + Store::new(&durable, KEEP_GENERATIONS) + .candidates("durable") + .iter() + .any(|candidate| candidate.path.ends_with("milestone-3.checkpoint")) + ); + } + + #[test] + fn a_second_reset_refuses_to_write_over_an_existing_archive() { + let tmp = tempfile::tempdir().unwrap(); + let durable = tmp.path().join("state"); + let hot = tmp.path().join("hot"); + state_dir(&durable, &[4], flybrain_gb::RatchetState::default()); + let archive = tmp.path().join("archive"); + reset_to_milestone(&durable, &hot, 4, &archive).unwrap(); + let error = reset_to_milestone(&durable, &hot, 4, &archive).unwrap_err().to_string(); + assert!(error.contains("already exists"), "{error}"); + } + + #[test] + fn the_stamp_is_the_event_logs_calendar_plus_a_time_of_day() { + assert_eq!(utc_stamp(0), "19700101T000000Z"); + // 2026-09-22T16:15:00Z + assert_eq!(utc_stamp(1_790_093_700_000), "20260922T161500Z"); + } + + #[test] + fn the_default_archive_is_a_dated_sibling_of_the_store() { + assert_eq!( + default_archive_dir(Path::new("/srv/fly/state"), "20260922T161500Z"), + PathBuf::from("/srv/fly/state.reset-20260922T161500Z") + ); + } +} diff --git a/services/flysim/crates/flysim/src/simloop.rs b/services/flysim/crates/flysim/src/simloop.rs index 2ab4bc5..f5ecd55 100644 --- a/services/flysim/crates/flysim/src/simloop.rs +++ b/services/flysim/crates/flysim/src/simloop.rs @@ -718,12 +718,36 @@ impl Sim { if runtime.rom_sha256 != self.rom_sha256 { bail!("checkpoint is for another cartridge ({})", runtime.rom_sha256); } - if runtime.compatibility != self.compatibility { - bail!( - "compatibility mismatch\n checkpoint: {}\n this build: {}", + // Byte-identical, or the one documented migration the operator asked for. + // + // `FLY_ACCEPT_ADAPTERS` is read here rather than carried in `Config` because it is a + // property of a *deploy*, not of a run: `infra/05-deploy.sh` writes it into + // `/etc/fly/fly.env` only for the deploy that needs it, and an operator who wants the + // migration off again deletes one line. An empty or unset variable is no migration at + // all, which is what every deploy before this one did. + let accepted = flybrain_gb::compatibility::accepted_adapters( + std::env::var(flybrain_gb::compatibility::ACCEPT_ADAPTERS_ENV).ok().as_deref(), + ); + match flybrain_gb::compatibility::decide( + &runtime.compatibility, + &self.compatibility, + self.adapter.migrates_from(), + &accepted, + ) { + flybrain_gb::compatibility::RestoreDecision::Exact => {} + flybrain_gb::compatibility::RestoreDecision::MigrateAdapter { from } => { + tracing::warn!( + from = %from, + to = %self.adapter.id(), + "restoring a checkpoint from an earlier adapter, by the migration \ + FLY_ACCEPT_ADAPTERS opted this deploy into" + ); + } + flybrain_gb::compatibility::RestoreDecision::Refuse(reason) => bail!( + "compatibility mismatch: {reason}\n checkpoint: {}\n this build: {}", runtime.compatibility, self.compatibility - ); + ), } if runtime.framebuffer.len() != FRAMEBUFFER_LEN { bail!("checkpoint framebuffer is {} bytes", runtime.framebuffer.len()); diff --git a/services/flysim/crates/flysim/src/snapshot.rs b/services/flysim/crates/flysim/src/snapshot.rs index f1c450d..dbcced1 100644 --- a/services/flysim/crates/flysim/src/snapshot.rs +++ b/services/flysim/crates/flysim/src/snapshot.rs @@ -241,8 +241,8 @@ pub struct FeedMacroOutcome { } /// Reward categories the feed reports counts for. The adapter's own interned kinds -/// (`milestone`, `exploration`, `map`, `species`, `trainer`, `battle`, `badge`, `boundary`) map -/// onto these. +/// (`milestone`, `exploration`, `map`, `species`, `trainer`, `battle`, `badge`, `boundary`, +/// `catch`) map onto these. #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] #[serde(rename_all = "lowercase")] pub enum RewardKind { @@ -282,6 +282,15 @@ impl RewardKind { // protocol is concerned: finding a door is finding somewhere new, and the design asks // for no new feed kind. "boundary" => Self::Explore, + // `catch` is a wild battle the fly won by keeping the Pokémon, so it publishes on + // the same counter a wild KO does. The feed's kinds are a closed set + // (`docs/feed-protocol.md`) and this rule asked for no new one. + // + // Deliberately *not* `pokedex`: on a catch of a species this run has never owned, + // the cartridge sets the Pokédex bit and the adapter's existing `species` rule pays + // for it on the same frame, so the `pokedex` counter already moves. Mapping `catch` + // there as well would count one event twice. + "catch" => Self::Wildwin, // The platformer. "band" => Self::Explore, "coin" => Self::Wildwin, @@ -739,6 +748,7 @@ mod tests { } assert_eq!(RewardKind::from_adapter("nonsense"), None); assert_eq!(RewardKind::from_adapter("boundary"), Some(RewardKind::Explore)); + assert_eq!(RewardKind::from_adapter("catch"), Some(RewardKind::Wildwin)); } #[test] diff --git a/services/flysim/crates/flysim/tests/compat_migration.rs b/services/flysim/crates/flysim/tests/compat_migration.rs new file mode 100644 index 0000000..ce47b05 --- /dev/null +++ b/services/flysim/crates/flysim/tests/compat_migration.rs @@ -0,0 +1,224 @@ +//! A `v5` checkpoint restored under `v6`: accepted with the opt-in, refused without it. +//! +//! The unit tests in `flybrain-gb` cover the decision function and the adapter's own state +//! migration separately. This is the two of them against one artefact: a real `FLYSIM01` +//! envelope carrying a `pokered-unique8-v5` compatibility string and a `v5` reward ledger — +//! written, encoded, decoded, and then put through exactly what `Sim::try_restore` puts a +//! candidate through. +//! +//! No ROM and no dataset, deliberately. Building a `Sim` would need both, and neither is part of +//! the question: what decides a restore is the compatibility string and `import_state`. + +use flybrain_gb::GameAdapter; +use flybrain_gb::compatibility::{RestoreDecision, accepted_adapters, decide}; +use flybrain_gb::pokemon_red::PokemonRedReward; +use flysim::store::{self, RuntimeState}; + +/// The live string's shape, with the adapter left open. The dataset fingerprint is shortened — +/// nothing here parses it, and a seven-digest one would be 455 characters of noise. +fn compatibility(adapter: &str) -> String { + format!( + "lif-1ms-f64-v2/{adapter}/aabbccddeeff00/fly-kc-mbon-rstdp-v2/\ + binjgb:c60e138da5a795ebb55e56b11b7e90024e41112c/\ + pokered:0cd19d3b877b7dc66d12c7050bed9a7f38154d4b/statefmt:199616-x86_64-unknown-linux-gnu" + ) +} + +/// A `v5` reward ledger: `STATE_VERSION` 4, every field `v5` wrote, and **no** `catchCounts`. +/// +/// Written out by hand rather than exported from an adapter, because an exported one would be a +/// `v6` state with the counter deleted — this is the shape the release box's checkpoints really +/// carry, field for field. +fn v5_reward() -> serde_json::Value { + serde_json::json!({ + "version": 4, + "seen": ["adventure", "map:0", "early:outside", "dex:3", "boundary:0:edge:n:near"], + "tiles": ["0:5:6", "0:5:7"], + "tileCounts": { "0": 2 }, + "wildWins": { "0:112:4": 2 }, + "replayBlocked": [], + "counts": { + "milestone": 2, "exploration": 0, "map": 1, "species": 1, + "trainer": 0, "battle": 2, "badge": 0, "boundary": 1 + }, + "total": 2.05, + "recent": [{ "kind": "species", "label": "OWNED #4", "brainMs": 1234.5, "value": 0.5 }], + "last": { "species": { "kind": "species", "label": "OWNED #4", "brainMs": 1234.5, "value": 0.5 } }, + "initialized": true, + "sawBoot": true, + "location": "0:5:7", + "stable": 9, + "progress": 3, + "badges": 0, + "battle": null, + "mode": "OVERWORLD" + }) +} + +fn v5_checkpoint() -> Vec { + use flybrain_core::decoder::DecoderState; + use flybrain_core::lif::LifState; + use flybrain_core::ordered::NumberMap; + use flybrain_core::plasticity::PlasticityState; + + let agent = flybrain_core::agent::AgentState { + version: 1, + remainder: 0.25, + warmed_up: true, + network: LifState { + membrane: vec![0.1, -0.2], + refractory: vec![0, 1], + last_spike_ms: vec![-1_000_000.0, 5.0], + visual_drive: vec![0.3], + rng: 42, + reward_remaining: 0.0, + ms: 1_234.5, + population_rate: 1.0, + rates: NumberMap::from_pairs([("forward", 1.0)]), + plasticity: PlasticityState { + version: "fly-kc-mbon-rstdp-v2".to_string(), + topology: 7, + enabled: true, + updates: 1.0, + signal: 0.0, + gains: vec![1.0], + traces: vec![0.0], + touched: vec![0.0], + }, + }, + decoder: DecoderState { + version: 4, + calibrated: true, + baseline: NumberMap::from_pairs([("forward", 1.0)]), + held_until: NumberMap::new(), + next_allowed: NumberMap::new(), + next_decision: 0.0, + current: None, + fatigue: NumberMap::new(), + macro_next_decision: 0.0, + macro_current: None, + macro_fatigue: NumberMap::new(), + }, + }; + let runtime = RuntimeState { + generation: 41, + wall_ms: 1_790_000_000_000, + rom_sha256: flybrain_gb::pokemon_red::SUPPORTED_ROM.to_string(), + emulator_frame: 1_000_000, + compatibility: compatibility("pokered-unique8-v5"), + speed: 1.0, + buttons: 0, + rank_since_ms: 1_000.0, + last_event_id: 4_242, + reward: v5_reward(), + ratchet: flybrain_gb::RatchetState { best: 3, attempts: 1, recoveries: 4, ..Default::default() }, + emulator: vec![3; 64], + framebuffer: vec![0; 32], + ratchet_game: vec![1], + ratchet_frame: vec![2], + }; + store::encode(&agent, &runtime).expect("the fixture encodes") +} + +#[test] +fn a_v5_checkpoint_is_refused_under_v6_without_the_opt_in() { + let checkpoint = store::decode(&v5_checkpoint()).expect("the fixture decodes"); + let adapter = PokemonRedReward::new(); + let current = compatibility(adapter.id()); + assert_ne!(checkpoint.runtime.compatibility, current, "v6 is not v5"); + + for opt_in in [None, Some(""), Some("pokered-unique8-v4"), Some("some-other-adapter")] { + assert!( + matches!( + decide( + &checkpoint.runtime.compatibility, + ¤t, + adapter.migrates_from(), + &accepted_adapters(opt_in), + ), + RestoreDecision::Refuse(_) + ), + "FLY_ACCEPT_ADAPTERS={opt_in:?} must not migrate anything" + ); + } +} + +#[test] +fn a_v5_checkpoint_restores_under_v6_with_the_opt_in_and_the_counter_starts_at_zero() { + let checkpoint = store::decode(&v5_checkpoint()).expect("the fixture decodes"); + let mut adapter = PokemonRedReward::new(); + let current = compatibility(adapter.id()); + + assert_eq!( + decide( + &checkpoint.runtime.compatibility, + ¤t, + adapter.migrates_from(), + &accepted_adapters(Some("pokered-unique8-v5")), + ), + RestoreDecision::MigrateAdapter { from: "pokered-unique8-v5".to_string() } + ); + + // The migration itself: `import_state`, exactly as `Sim::try_restore` calls it. + adapter.import_state(&checkpoint.runtime.reward).expect("a v5 ledger is a valid v6 ledger"); + + let after = adapter.export_state(); + assert_eq!(after["catchCounts"], serde_json::json!({}), "the new counter starts at 0"); + assert_eq!(after["counts"]["catch"], serde_json::json!(0)); + + // And nothing else moved: every field the v5 state carried round-trips to the same value, + // and the only key v6 adds is the counter. + // + // `counts` is the one field that is not byte-identical, and it is not a change of meaning: + // it serializes every kind in the catalog, so a v6 state lists `catch` where a v5 state had + // nothing to list. Every kind the v5 state did carry keeps its number. + let before = v5_reward(); + for (key, value) in before.as_object().unwrap() { + if key == "counts" { + for (kind, count) in value.as_object().unwrap() { + assert_eq!(&after["counts"][kind], count, "counts.{kind}"); + } + let added: Vec<&String> = after["counts"] + .as_object() + .unwrap() + .keys() + .filter(|kind| !value.as_object().unwrap().contains_key(*kind)) + .collect(); + assert_eq!(added, vec!["catch"], "v6 counts one more kind and no others"); + continue; + } + assert_eq!(&after[key], value, "{key} must survive the migration byte for byte"); + } + let added: Vec<&String> = after + .as_object() + .unwrap() + .keys() + .filter(|key| !before.as_object().unwrap().contains_key(*key)) + .collect(); + assert_eq!(added, vec!["catchCounts"], "v6 adds one field and no others"); + + // The rest of what a restore reads is untouched by the migration. + assert_eq!(adapter.progress().rank, 3); + assert_eq!(checkpoint.runtime.last_event_id, 4_242); + assert_eq!(checkpoint.runtime.ratchet.best, 3); +} + +#[test] +fn nothing_but_the_adapter_segment_may_differ_for_the_migration_to_apply() { + let adapter = PokemonRedReward::new(); + let accepted = accepted_adapters(Some("pokered-unique8-v5")); + let current = compatibility(adapter.id()); + + // A v5 string whose state format also moved: a different build, not a rule change. + let other_abi = compatibility("pokered-unique8-v5").replace("199616", "199617"); + assert!(matches!( + decide(&other_abi, ¤t, adapter.migrates_from(), &accepted), + RestoreDecision::Refuse(_) + )); + + // And an identical string needs no opt-in at all. + assert_eq!( + decide(¤t, ¤t, adapter.migrates_from(), &[]), + RestoreDecision::Exact + ); +} diff --git a/services/flysim/crates/flysim/tests/rom_catch.rs b/services/flysim/crates/flysim/tests/rom_catch.rs new file mode 100644 index 0000000..014048b --- /dev/null +++ b/services/flysim/crates/flysim/tests/rom_catch.rs @@ -0,0 +1,257 @@ +//! The catch reward against the real cartridge. +//! +//! Gated on `FLY_ROM` *and* on a checkpoint, the way every ROM test in this workspace is, and +//! skips cleanly without either — the cartridge never enters this repository and a checkpoint is +//! not a fixture, it is the state the release box was really in: +//! +//! ```sh +//! FLY_ROM="$HOME/roms/pokemon-red.gb" \ +//! FLY_CATCH_CHECKPOINT=.local/checkpoints/ \ +//! cargo test --release -p flysim --test rom_catch -- --nocapture +//! ``` +//! +//! ## What only the cartridge can answer +//! +//! The synthetic trace in `pokemon_red/tests.rs` writes `wCapturedMonSpecies`, `wBattleResult` +//! and the Pokédex bit itself, from the disassembly. It cannot say that those are the bytes +//! *this* cartridge writes when a ball keeps a Pokémon, in that order, on frames an adapter +//! sampling once a frame actually sees. That is this test, and it is the "survey" half of +//! `docs/design/macros-wram.md`'s evidence for the row: a real battle, real button presses, and +//! the byte read out of the running game rather than written into a fake one. +//! +//! ## How the catch is produced +//! +//! No steering and no scripted button sequence: the shipping macro palette, the shipping macro +//! layer and the shipping decoder, with a stub readout that leans on one macro population at a +//! time — the same driver `tests/rom_macros_mode.rs` uses and for the same reason. The one thing +//! this harness does that the rotation does not is lean on `THROW BALL`'s channel while a wild +//! battle is up, because the question here is what the adapter reads from a catch, not whether a +//! game-blind readout finds its way to one. +//! +//! The checkpoint must hold at least one ball in the bag. The macro palette can buy one +//! (`BUY BALL`, `MB·PBALL`, inside a mart), but that is a walk across a city and back and it is a +//! different test's question; this one says out loud that it skipped. + +use flybrain_core::decoder::PopulationDecoder; +use flybrain_core::decoder::gameboy::gameboy_decoder_config_with_macros; +use flybrain_core::ordered::NumberMap; +use flybrain_gb::adapter::RewardEvent; +use flybrain_gb::pokemon_red::state; +use flybrain_gb::pokemon_red::symbols::ram; +use flybrain_gb::pokemon_red::{PokemonRedReward, catalog}; +use flybrain_gb::{ + AdapterLedger, DEFAULT_AUDIO_FRAMES, DEFAULT_AUDIO_FREQUENCY, Emulator, GameAdapter, +}; +use flysim::config::Config; +use flysim::macros::{MacroLayer, macro_layer}; +use flysim::snapshot::MacroMode; + +const MS_PER_FRAME: f64 = 1000.0 / 59.7275; +const SEED: u32 = 20_260_922; +/// The hot population's rate against every other one's, which is also the stub's calibration +/// rate — so a channel that is not the hot one scores exactly 1.0. +const HOT: f64 = 16.0; +const REST: f64 = 10.0; +/// `THROW BALL`'s channel (`pokemon_red::macros::palette`). +const BALL: &str = "MB·BALL"; +/// Frames the stub leans on one channel before the rotation moves on, the shape of the real +/// group's hysteresis-then-fatigue rotation. +const BURST_FRAMES: u32 = 24; + +fn rates(hot: Option<&str>) -> NumberMap { + let mut rates = NumberMap::new(); + for channel in flybrain_gb::macro_channels("pokemon-red") { + rates.set(channel, REST); + } + for bucket in 0..8 { + rates.set(&format!("command_{bucket}"), REST); + } + if let Some(channel) = hot { + rates.set(channel, HOT); + } + rates +} + +fn rom() -> Option> { + let path = std::env::var_os("FLY_ROM")?; + match std::fs::read(&path) { + Ok(bytes) => Some(bytes), + Err(error) => panic!("FLY_ROM is set to {path:?} but could not be read: {error}"), + } +} + +fn checkpoint() -> Option { + let path = std::env::var_os("FLY_CATCH_CHECKPOINT")?; + Some( + flysim::store::load(std::path::Path::new(&path)) + .expect("the checkpoint should be a FLYSIM01 envelope"), + ) +} + +struct Run { + gb: Emulator, + adapter: PokemonRedReward, + layer: MacroLayer, + decoder: PopulationDecoder, + channels: Vec<&'static str>, + ms: f64, + frame: u32, + /// Every payout the adapter has made since the run started. + payouts: Vec, +} + +impl Run { + fn resume(rom: &[u8], checkpoint: &flysim::store::Checkpoint) -> Self { + let mut gb = Emulator::new(rom, DEFAULT_AUDIO_FREQUENCY, DEFAULT_AUDIO_FRAMES) + .expect("binjgb should accept the cartridge"); + let mut adapter = PokemonRedReward::new(); + gb.import_state(&checkpoint.runtime.emulator).expect("the checkpoint's emulator state"); + // A checkpoint written by an earlier adapter rebaselines rather than failing, which is + // exactly the `v5` -> `v6` case this rule ships with. + adapter.import_state(&checkpoint.runtime.reward).expect("the checkpoint's reward ledger"); + let channels = flybrain_gb::macro_channels("pokemon-red"); + let preset = gameboy_decoder_config_with_macros(&channels); + let hold_ms = preset.macros.as_ref().expect("the preset has a macro group").hold_ms; + let mut decoder = PopulationDecoder::new(preset).expect("the preset is well formed"); + decoder.calibrate(&rates(None)); + let mut config = Config::default(); + config.loop_.game = "pokemon-red".to_string(); + config.macros.mode = MacroMode::Macros; + config.validate().expect("pokemon-red has a palette in macros mode"); + let mut layer = macro_layer(&config, hold_ms, SEED).expect("a layer in macros mode"); + let _ = layer.observe(&mut gb, &AdapterLedger(&adapter), 0.0); + Self { + gb, + adapter, + layer, + decoder, + channels, + ms: 0.0, + frame: 0, + payouts: Vec::new(), + } + } + + fn byte(&mut self, address: u16) -> u8 { + self.gb.read_wram(address) + } + + fn in_wild_battle(&mut self) -> bool { + self.byte(ram::wIsInBattle) == 1 + } + + /// Balls in the bag, of any kind (`constants/item_constants.asm`: MASTER_BALL 1, + /// ULTRA_BALL 2, GREAT_BALL 3, POKE_BALL 4 — the same four `THROW BALL` looks for). + fn balls(&mut self) -> usize { + state::bag(&mut self.gb) + .iter() + .filter(|item| (0x01..=0x04).contains(&item.id) && item.count > 0) + .map(|item| usize::from(item.count)) + .sum() + } + + fn step(&mut self) { + // Lean on `THROW BALL` while a wild battle is up; otherwise rotate, which is what gets + // the fly into the grass in the first place. + let hot = if self.in_wild_battle() { + Some(BALL) + } else { + let slot = (self.frame / BURST_FRAMES) as usize % self.channels.len(); + Some(self.channels[slot]) + }; + let bound = self.layer.bound_channels(); + let active = self.decoder.decode_bound(&rates(hot), self.ms, false, None, Some(&bound)); + let mask = { + let ledger = AdapterLedger(&self.adapter); + self.layer.decide(&active, 0, self.ms, &mut self.gb, &ledger).mask + }; + self.gb.set_buttons(mask as u8); + self.gb.run_frame().expect("a frame should complete"); + self.ms += MS_PER_FRAME; + self.frame += 1; + let ms = self.ms; + self.payouts.extend(self.adapter.sample(&mut self.gb, ms)); + let ledger = AdapterLedger(&self.adapter); + let _ = self.layer.observe(&mut self.gb, &ledger, ms); + } + + fn catches(&self) -> Vec<&RewardEvent> { + self.payouts.iter().filter(|event| event.kind == catalog::kind::CATCH).collect() + } +} + +#[test] +fn a_catch_on_the_cartridge_pays_the_catch_rule_once_with_the_species_in_its_label() { + let Some(rom) = rom() else { + eprintln!("skipped: FLY_ROM is not set"); + return; + }; + let Some(checkpoint) = checkpoint() else { + eprintln!("skipped: no FLY_CATCH_CHECKPOINT"); + return; + }; + let mut run = Run::resume(&rom, &checkpoint); + let balls = run.balls(); + if balls == 0 { + eprintln!( + "skipped: the checkpoint's bag holds no ball (map {:#04x}). `BUY BALL` can buy one \ + inside a mart; point FLY_CATCH_CHECKPOINT at a state that already has one.", + run.adapter.map_id().unwrap_or(u32::MAX) + ); + return; + } + eprintln!("bag holds {balls} balls; map {:#04x}", run.adapter.map_id().unwrap_or(u32::MAX)); + + // Twenty brain minutes is generous for a forest checkpoint: the live run threw 28 balls in + // its first Viridian Forest session (`pokemon_red::macros::palette`). + let budget = 20 * 60 * 60; + let mut battles = 0u32; + let mut was_in_battle = false; + for _ in 0..budget { + run.step(); + let now = run.in_wild_battle(); + if now && !was_in_battle { + battles += 1; + } + was_in_battle = now; + if !run.catches().is_empty() { + break; + } + } + + let catches = run.catches(); + assert!( + !catches.is_empty(), + "no catch in {:.1} brain minutes: {battles} wild battles, {} balls left, map {:#04x}, \ + macros {:?}", + run.ms / 60_000.0, + run.balls(), + run.adapter.map_id().unwrap_or(u32::MAX), + run.layer.counts() + ); + + let caught = catches[0]; + eprintln!( + "caught after {:.1} brain minutes and {battles} wild battles: {} for {}", + run.ms / 60_000.0, + caught.label, + caught.value + ); + assert!(caught.label.starts_with("CAUGHT #"), "{}", caught.label); + assert!( + (caught.value - 0.30).abs() < 1e-12 || caught.value == catalog::CATCH_REPEAT_VALUE, + "a catch pays one of the rule's two amounts, not {}", + caught.value + ); + // The cartridge's own flag is clear again by the time the payout lands, which is what makes + // the payout a battle-exit event rather than a per-frame one. + assert_eq!(run.byte(ram::wCapturedMonSpecies), 0); + assert_eq!( + run.adapter.progress().counts[catalog::kind::CATCH], + 1, + "one battle, one payout" + ); + // And a species the run is paid for catching is a species the Pokédex knows: the same event + // sets the bit the `species` rule reads, whether or not it was new to this run. + assert!(run.balls() < balls, "a ball was spent"); +} diff --git a/services/flysim/tools/gen_symbols.py b/services/flysim/tools/gen_symbols.py index 71ce56c..b499e43 100644 --- a/services/flysim/tools/gen_symbols.py +++ b/services/flysim/tools/gen_symbols.py @@ -200,6 +200,15 @@ EXTRA_RAM = ( 'wCurMapTileset', 'wTilesetBank', 'wTilesetBlocksPtr', + # The catch reward (`docs/rewards-learning.md`, `docs/design/macros-wram.md` section 2). + # ram/wram.asm's own comment is "0 if no mon was captured": ItemUseBall zeroes it before + # every throw and writes wEnemyMonSpecies into it only on the branch that keeps the + # Pokemon, and UseBagItem zeroes it again on the way out of the battle. It is the + # cartridge's own answer to "was this one caught", and the only signal that needs no + # second rule to tell a catch apart from a gift, a trade or an evolution. + # services/flysim/tools/resolve_wram.py is the second reading of it, from ram/wram.asm at + # this commit, bracketed by wFontLoaded and wForcePlayerToChooseMon. + 'wCapturedMonSpecies', ) diff --git a/services/flysim/tools/resolve_wram.py b/services/flysim/tools/resolve_wram.py index 4e67b06..c7973cb 100644 --- a/services/flysim/tools/resolve_wram.py +++ b/services/flysim/tools/resolve_wram.py @@ -49,6 +49,33 @@ WANTED = { # not bank 0, so this is the read the memory seam grew a bank for. 'wTilesetBank': 'the ROM bank the blockset lives in', 'wTilesetBlocksPtr': 'blocks to tiles, 16 bytes per block', + # The catch reward (`docs/rewards-learning.md`, `docs/design/macros-wram.md` + # section 10). ram/wram.asm's own comment is "0 if no mon was captured": + # ItemUseBall zeroes it before every throw and writes wEnemyMonSpecies into it + # only on the branch that keeps the caught Pokemon, and UseBagItem zeroes it + # again on the way out of the battle. It is the cartridge's own answer to "was + # this one caught", and the only signal that needs no second rule to tell a + # catch apart from a gift, a trade or an evolution. + 'wCapturedMonSpecies': 'the species a ball just caught, 0 for none', +} + + +#: Constants the decomp defines through its `const` enumeration rather than with a +#: plain `EQU`, so `constants()` cannot evaluate their expressions. They matter here +#: because `NUM_TMS + NUM_HMS` is the size of `wMonHLearnset`, and that one +#: declaration is what kills the cursor on its way through the battle engine's +#: scratch bytes -- the region `wCapturedMonSpecies` lives in. +#: +#: Each is *counted* from the decomp rather than written out by hand, which is the +#: same rule the rest of this tool follows. `DEF NUM_HMS EQU const_value - HM01` is +#: by construction the number of `add_hm` definitions after `HM01`, and +#: `item_constants.asm`'s own `ASSERT NUM_TMS == const_value - TM01` ties `NUM_TMS` +#: to the number of `add_tm` definitions -- so `NUM_TMS` is counted *and* compared +#: against the literal the same file declares, and a decomp that moved one without +#: the other stops the run instead of producing an address. +COUNTED = { + 'NUM_HMS': ('constants/item_constants.asm', r'^\s*add_hm\s+\w+'), + 'NUM_TMS': ('constants/item_constants.asm', r'^\s*add_tm\s+\w+'), } @@ -68,6 +95,10 @@ def constants(root: Path) -> dict[str, int]: BLOCK_WIDTH`). A name whose expression never becomes evaluable is simply left out, which kills the cursor at any declaration that uses it. """ + counted = { + name: len(re.findall(pattern, (root / path).read_text(), re.M)) + for name, (path, pattern) in COUNTED.items() + } pending: dict[str, str] = {} sources = sorted((root / 'constants').glob('*.asm')) + sorted( (root / 'constants').glob('*.inc') @@ -77,7 +108,7 @@ def constants(root: Path) -> dict[str, int]: r'^\s*(?:DEF|def)\s+(\w+)\s+(?:EQU|equ)\s+([^;\n]+)', path.read_text(), re.M ): pending.setdefault(name, value.strip()) - out: dict[str, int] = {} + out: dict[str, int] = dict(counted) while pending: progressed = False for name in list(pending): @@ -89,6 +120,12 @@ def constants(root: Path) -> dict[str, int]: progressed = True if not progressed: break + for name, value in counted.items(): + if out.get(name, value) != value: + raise SystemExit( + f'{name}: the decomp declares {out[name]} and defines {value} of them' + ) + out[name] = value return out @@ -367,11 +404,17 @@ def main() -> None: raise SystemExit('the walk disagrees with symbols.rs; nothing emitted') print(f'{checked} of {len(table)} pinned addresses re-derived from wram.asm, no disagreement') - missing = [name for name in WANTED if name not in resolved] + # A name this tool has already emitted is pinned, so the walk meets it as an + # anchor rather than resolving it: it was re-derived all the same, and the + # comparison above is what says so. + missing = [name for name in WANTED if name not in resolved and name not in table] if missing: raise SystemExit(f'unanchored, so not resolved: {", ".join(missing)}') for name in WANTED: - print(f'{name} = ${resolved[name]:04x} ({WANTED[name]})') + if name in resolved: + print(f'{name} = ${resolved[name]:04x} ({WANTED[name]})') + else: + print(f'{name} = ${table[name]:04x} (already pinned; {WANTED[name]})') if not args.emit: return