Merge main at 5512900: STATE-01, and the flybus coalescing fix

The flybus session_over_one_router failure my workspace runs were
counting is already fixed on main: the coalescing branch forces the
coalescing deterministically instead of asserting that a slow consumer
must skip. Merging before the runs so they measure a tree that exists.

One conflict, in the crate README, and it was two sections both newly
added at the same anchor rather than two versions of one thing. Both are
kept: STATE-01's checkpoint and recovery section, then the guidance on
which replies permit a subset, which stays immediately above the
contract-narrowing section where someone adding a check will meet it.

coordinator.rs and the process tests auto-merged. Checked rather than
assumed: the acknowledge equality check is still gone, acknowledge_replies
and last_resolution_attempts are present, STATE-01's capture, durable
and rebase surfaces are present, and both test changes survived.
This commit is contained in:
acamilo 2026-09-22 19:49:14 +00:00
commit e3132362cf
51 changed files with 8102 additions and 125 deletions

View file

@ -355,6 +355,50 @@ on-screen ticker cannot disagree with what the sim did.
against the prototype's WASM size and diffs a known save. If they match, prototype checkpoints against the prototype's WASM size and diffs a known save. If they match, prototype checkpoints
import and the segment records the shared tag; if not, milestone saves must be re-earned and that import and the segment records the shared tag; if not, milestone saves must be re-earned and that
is a stated M3 finding. is a stated M3 finding.
- **Restoring across an adapter version** (2026-09-22). The compatibility string is compared
whole, so bumping the reward adapter refuses every checkpoint the previous one wrote -- which is
the right default and was, until now, the only behaviour. It is the wrong default for a change
that only *adds* a rule: `pokered-unique8-v6` adds the catch reward and one counter,
`catchCounts`, and means the same thing as `v5` for every other field, so a `v5` run is
resumable and throwing it away would be a choice nobody made deliberately.
So there is one narrow, opt-in migration, `flybrain_gb::compatibility::decide`, and it requires
**all three** of:
1. the two compatibility strings differ in the adapter segment (segment 1) and **nowhere else**.
A dataset, kernel, plasticity, emulator-revision, symbol-provenance or state-format
difference is still a refusal: none of those has a migration, and a fly restored across one
is a different fly;
2. the running adapter's `migrates_from()` lists the checkpoint's adapter, so the code that will
read that state says out loud that it can. Pokémon Red's list is `["pokered-unique8-v5"]` and
nothing else -- `v4` is excluded because its ledger holds no `boundary:` keys and resuming it
would pay a second time for every exit already found, and `v3` because its stored rank is a
rung on a different ladder;
3. the deploy names the same adapter id in **`FLY_ACCEPT_ADAPTERS`** (comma- or
space-separated). Unset or empty migrates nothing, which is what every deploy before this one
did.
Condition 2 without 3 would make the migration silent; condition 3 without 2 would let an
operator wave through a pair nobody wrote a migration for. `infra/05-deploy.sh`'s compatibility
gate applies the same rule before it flips the `current` symlink, and writes the variable into
`/etc/fly/fly.env` so flysim applies it at restore -- the two must agree, or a deploy would pass
a gate that flysim then fails, which is the black stream the gate exists to prevent. The
migration itself is `PokemonRedReward::import_state` doing what it already did: `catchCounts` is
absent from a `v5` state and restores empty, which is the truth about a run that was never paid
for a catch. `STATE_VERSION` does not move, because the schema did not.
- **Restarting a run from an earlier rung** (2026-09-22). `FLY_RESET_STATE=1` throws the run away;
`infra/bin/fly-reset-to-milestone <N>` keeps it and rewinds it. It archives both stores to a
dated directory, rewrites `milestone-<N>.checkpoint` with the ratchet's `attempts` and
`recoveries` at zero (so the restarted run does not begin with its recovery budget already
spent), installs it as the newest generation of the hot and durable stores, removes the
milestone archives above N, and clears the event log -- whose id sequence the restored
checkpoint's `lastEventId` rewinds. `best` is not touched: the archive's own `best` is the rung
it was taken at, and the rank the stream shows is recomputed by the adapter from the restored
game state. The implementation is `flysim::reset` (`flysim --reset-to-milestone N`) rather than
the shell script, because two of those steps are inside the envelope. The sequence around it is
in `infra/docs/runbook.md`.
- **A running macro is not checkpointed** (2026-09-16, `docs/design/macros.md`). Palette mode's - **A running macro is not checkpointed** (2026-09-16, `docs/design/macros.md`). Palette mode's
state — the scene, the palette, the running macro, its plan and its frame count — is transient, state — the scene, the palette, the running macro, its plan and its frame count — is transient,
like the readout's blocked-direction cooldown and for the same reason: a restore that resumed a like the readout's blocked-direction cooldown and for the same reason: a restore that resumed a

View file

@ -118,6 +118,20 @@ Addresses are at the pinned commit. "Verified" is one of:
| which slot is out | `wPlayerMonNumber` | `$cc2f` | 0-based party slot | ROM, trace | | which slot is out | `wPlayerMonNumber` | `$cc2f` | 0-based party slot | ROM, trace |
| the enemy | `wEnemyMonSpecies`, `wEnemyMonHP`, `wEnemyMonLevel`, `wEnemyMonMaxHP` | `$cfe5`, `$cfe6`, `$cff3`, `$cff4` | HP big-endian. Not written on the frame a battle starts — the reward adapter's own comment says the same — so the enemy is `None` for the first few hundred frames of a battle. | ROM (the rival's Squirtle, level 5, 20/20, and `None` on the first frame), trace | | the enemy | `wEnemyMonSpecies`, `wEnemyMonHP`, `wEnemyMonLevel`, `wEnemyMonMaxHP` | `$cfe5`, `$cfe6`, `$cff3`, `$cff4` | HP big-endian. Not written on the frame a battle starts — the reward adapter's own comment says the same — so the enemy is `None` for the first few hundred frames of a battle. | ROM (the rival's Squirtle, level 5, 20/20, and `None` on the first frame), trace |
| how many moves | `wNumMovesMinusOne` | `$cd6c` | the move count minus one, valid in a battle | trace | | how many moves | `wNumMovesMinusOne` | `$cd6c` | the move count minus one, valid in a battle | trace |
| **a ball kept this one** | `wCapturedMonSpecies` | `$d11c` | **new 2026-09-22** (the catch reward, `docs/rewards-learning.md`). `ram/wram.asm`'s own comment is "0 if no mon was captured". `ItemUseBall` zeroes it before every throw (`.canUseBall`) and writes `wEnemyMonSpecies` into it only on the branch that keeps the Pokémon; `UseBagItem`'s `.returnAfterCapturingMon` zeroes it again and sets `wBattleResult` to 2 on the way out of the battle. It is therefore non-zero for the hundreds of frames the catch's text and Pokédex screen take, and zero everywhere else. The value is the **internal** species index, like `wEnemyMonSpecies` and unlike `wPokedexOwned`'s bit index. Address resolved by `services/flysim/tools/resolve_wram.py`, bracketed by `wFontLoaded` and `wForcePlayerToChooseMon`. | survey (`tests/rom_catch.rs`: a real wild battle from a rung-9 checkpoint, balls thrown by the `THROW BALL` macro, the byte read out of the running game), trace (`pokemon_red/tests.rs`) |
`wBattleResult` (`$cf0b`) is the second half of that row and is worth its own sentence: it is 0
for a win, 1 for a loss, and 2 on exactly two paths in the whole game -- `.returnAfterCapturingMon`
and a *link* battle whose opponent ran (`engine/battle/core.asm`), which this cartridge never has.
So "the captured-species byte was non-zero during the battle **and** the result is 2" is a catch
and nothing else. `InitBattleVariables`, `ResetStatusAndHalveMoneyOnBlackout` and
`HandleFlyWarpOrDungeonWarp` all clear it, so a stale 2 cannot survive into the next battle.
Not used for the catch, and why: `wPartyCount` (`$d163`) rises on a catch **only** when the party
has room -- a full party sends the Pokémon to `wBoxCount` instead -- and it also rises for a gift,
a trade and a Pokémon withdrawn from the PC. Reading a catch off it would need a second rule to
tell those apart. The cartridge's own flag needs none, which is why the row above is the one the
adapter reads.
### Battle menu and cursor, own turn against forced switch ### Battle menu and cursor, own turn against forced switch

View file

@ -153,15 +153,15 @@ and once over a Unix socket, so `tests/rpc.rs::request_reply_roundtrip` means
| `latest`: one queued value, replacing only an undelivered one; replacement releases that entry's roots; delivered or in-use messages are never reclaimed early; maxQueued is exactly 1 | conforms | `router/state.rs::{op_publish (latest branch), op_subscribe}` | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both), `tests/conformance_artifacts.rs::latest_mode_holds_at_most_two_roots_delivered_plus_queued` (both) | | `latest`: one queued value, replacing only an undelivered one; replacement releases that entry's roots; delivered or in-use messages are never reclaimed early; maxQueued is exactly 1 | conforms | `router/state.rs::{op_publish (latest branch), op_subscribe}` | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both), `tests/conformance_artifacts.rs::latest_mode_holds_at_most_two_roots_delivered_plus_queued` (both) |
| `bounded`: FIFO, no coalescing or silent loss; when capacity is unavailable, reject with `BACKPRESSURE` before admitting any delivery | conforms | `router/state.rs::op_publish` (pre-checks every subscriber) | `tests/pubsub.rs::bounded_fifo_and_atomic_backpressure` (both), `tests/conformance_routing.rs::bounded_overflow_rolls_back_all_artifact_roots` (both) | | `bounded`: FIFO, no coalescing or silent loss; when capacity is unavailable, reject with `BACKPRESSURE` before admitting any delivery | conforms | `router/state.rs::op_publish` (pre-checks every subscriber) | `tests/pubsub.rs::bounded_fifo_and_atomic_backpressure` (both), `tests/conformance_routing.rs::bounded_overflow_rolls_back_all_artifact_roots` (both) |
| maxInFlight credits return only on `delivery.consumed`, not on socket write completion | conforms | `router/state.rs::release_owner` (credit returned when the owner is released) | `tests/pubsub.rs::credits_return_only_on_consume` (both), `tests/conformance_routing.rs::bounded_credit_waits_for_every_extracted_artifact` (both) | | maxInFlight credits return only on `delivery.consumed`, not on socket write completion | conforms | `router/state.rs::release_owner` (credit returned when the owner is released) | `tests/pubsub.rs::credits_return_only_on_consume` (both), `tests/conformance_routing.rs::bounded_credit_waits_for_every_extracted_artifact` (both) |
| A latest subscriber with all credits in use still has one replaceable queued value | conforms | `router/state.rs::{Sub::queue, dispatch_topic}` (queue and credits are separate) | `tests/bus_acceptance.rs::both_transports_produce_equivalent_behaviour_traces` (events 21 and 24: `publish replaced=1`, then `latest seq=3 replaced=1`) | | A latest subscriber with all credits in use still has one replaceable queued value | conforms | `router/state.rs::{Sub::queue, dispatch_topic}` (queue and credits are separate) | `tests/bus_acceptance.rs::both_transports_produce_equivalent_behaviour_traces` (events 21 and 24: `publish replaced=1`, then `latest seq=3 replaced=1`), `tests/integration.rs::session_over_one_router` (both: a renderer held for the whole run keeps one delivery in flight and one replaceable value, and receives snapshots 1 and 20 of 20) |
| Atomic subscriber/retention snapshot at admission; validate and reserve every queue entry and owner budget before accepting | conforms: one mutex, validate-then-mutate | `router/state.rs::op_publish` | `tests/artifacts.rs::failed_admission_is_atomic` (both) | | Atomic subscriber/retention snapshot at admission; validate and reserve every queue entry and owner budget before accepting | conforms: one mutex, validate-then-mutate | `router/state.rs::op_publish` | `tests/artifacts.rs::failed_admission_is_atomic` (both) |
| A bounded overflow rejects the whole publish: no partial fan-out, no retained-latest update | conforms | `router/state.rs::op_publish` | `tests/conformance_routing.rs::bounded_overflow_rolls_back_all_artifact_roots` (both) | | A bounded overflow rejects the whole publish: no partial fan-out, no retained-latest update | conforms | `router/state.rs::op_publish` | `tests/conformance_routing.rs::bounded_overflow_rolls_back_all_artifact_roots` (both) |
| On acceptance, one `topicSequence` and roots for every delivery and the optional retained value | conforms; a refused publication spends no sequence number | `router/state.rs::op_publish` (`t.sequence += 1` after the checks) | `tests/pubsub.rs::bounded_fifo_and_atomic_backpressure` (both) | | On acceptance, one `topicSequence` and roots for every delivery and the optional retained value | conforms; a refused publication spends no sequence number | `router/state.rs::op_publish` (`t.sequence += 1` after the checks) | `tests/pubsub.rs::bounded_fifo_and_atomic_backpressure` (both) |
| Different topics have no total ordering; multiple publishers follow router acceptance order | conforms: per-topic sequence only | `router/state.rs::Topic::sequence` | `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) | | Different topics have no total ordering; multiple publishers follow router acceptance order | conforms: per-topic sequence only | `router/state.rs::Topic::sequence` | `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) |
| The publication reply counts accepted subscriptions and replaced queue entries, not consumers that processed data | conforms | `router/state.rs::op_publish` reply | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both) | | The publication reply counts accepted subscriptions and replaced queue entries, not consumers that processed data | conforms | `router/state.rs::op_publish` reply | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both) |
| `replaced` on a delivery reports how many undelivered messages were coalesced since that subscription's preceding delivery | conforms | `router/state.rs::{Sub::replaced, dispatch_topic}` (taken at dispatch) | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both) | | `replaced` on a delivery reports how many undelivered messages were coalesced since that subscription's preceding delivery | conforms | `router/state.rs::{Sub::replaced, dispatch_topic}` (taken at dispatch) | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both), `tests/integration.rs::session_over_one_router` (both: the 18 replacements the publisher was told about at admission are the same 18 the renderer is told about on delivery, and are exactly the snapshots it did not receive) |
| Optional `retained:latest` holds one last message and its artifacts independent of subscribers | conforms | `router/state.rs::op_publish` (retain branch) | `tests/conformance_artifacts.rs::retained_topic_value_holds_a_root_independent_of_subscribers` (both) | | Optional `retained:latest` holds one last message and its artifacts independent of subscribers | conforms | `router/state.rs::op_publish` (retain branch) | `tests/conformance_artifacts.rs::retained_topic_value_holds_a_root_independent_of_subscribers` (both) |
| `replayLatest` enqueues the retained value before subsequent accepted publications; bounded preserves the order, latest may coalesce it | conforms | `router/state.rs::op_subscribe` (replay is enqueued under the subscribe lock) | `tests/conformance_routing.rs::latest_replay_is_ordered_ahead_of_a_racing_publish` (both), `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) | | `replayLatest` enqueues the retained value before subsequent accepted publications; bounded preserves the order, latest may coalesce it | conforms | `router/state.rs::op_subscribe` (replay is enqueued under the subscribe lock) | `tests/conformance_routing.rs::latest_replay_is_ordered_ahead_of_a_racing_publish` (both: one racing publication to a bounded and a latest subscription at once; bounded must deliver replay then publication, and the latest branch is chosen by that publication's own `replaced` count, never by which side of the race the dispatcher won), `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) |
| Replay uses the original topicSequence, a fresh deliveryId and explicit roots | conforms | `router/state.rs::op_subscribe` (`add_roots`, the same `Arc<TopicMsg>`) | `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) | | Replay uses the original topicSequence, a fresh deliveryId and explicit roots | conforms | `router/state.rs::op_subscribe` (`add_roots`, the same `Arc<TopicMsg>`) | `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) |
| Without retention, a zero-subscriber publication retains no ownership after admission | conforms | `router/state.rs::op_publish` | `tests/pubsub.rs::zero_subscriber_publish_retains_nothing` (both) | | Without retention, a zero-subscriber publication retains no ownership after admission | conforms | `router/state.rs::op_publish` | `tests/pubsub.rs::zero_subscriber_publish_retains_nothing` (both) |
| Clearing a topic releases only its retained root, not active consumers | conforms | `router/state.rs::op_clear` | `tests/conformance_routing.rs::cleared_topic_gives_no_replay_until_a_fresh_publish` (both) | | Clearing a topic releases only its retained root, not active consumers | conforms | `router/state.rs::op_clear` | `tests/conformance_routing.rs::cleared_topic_gives_no_replay_until_a_fresh_publish` (both) |
@ -260,7 +260,7 @@ and once over a Unix socket, so `tests/rpc.rs::request_reply_roundtrip` means
| The thirteen transport error codes exist with those names | conforms | `error.rs::ErrorCode` | `tests/wire.rs::body_errors_keep_the_connection` (both) | | The thirteen transport error codes exist with those names | conforms | `error.rs::ErrorCode` | `tests/wire.rs::body_errors_keep_the_connection` (both) |
| Three more codes: `CONFLICT`, `NO_TOPIC`, `ARTIFACT_MISMATCH` | deviates-allowed: "Transport errors **include** ..." is not an exhaustive list, and each names a refusal the draft requires but leaves unnamed. Recorded as an amendment in bus-v1 section 12 | `error.rs::ErrorCode` | `tests/pubsub.rs::subscription_and_topic_validation` (both), `tests/artifacts.rs::seal_checks_length_and_digest` (both) | | Three more codes: `CONFLICT`, `NO_TOPIC`, `ARTIFACT_MISMATCH` | deviates-allowed: "Transport errors **include** ..." is not an exhaustive list, and each names a refusal the draft requires but leaves unnamed. Recorded as an amendment in bus-v1 section 12 | `error.rs::ErrorCode` | `tests/pubsub.rs::subscription_and_topic_validation` (both), `tests/artifacts.rs::seal_checks_length_and_digest` (both) |
| Before admission report `not-dispatched`; once dispatch might have occurred report `dispatched` or `unknown` conservatively | conforms | `error.rs::BusError::new` (not-dispatched by default), `router/state.rs` dispatched notices, `client/reactor.rs::fail_all` (unknown) | `tests/rpc.rs::{cancellation_states, service_disconnect_fails_calls}` (both), `tests/sol_review_races.rs::writer_failure_terminates_reader_and_pending_work` | | Before admission report `not-dispatched`; once dispatch might have occurred report `dispatched` or `unknown` conservatively | conforms | `error.rs::BusError::new` (not-dispatched by default), `router/state.rs` dispatched notices, `client/reactor.rs::fail_all` (unknown) | `tests/rpc.rs::{cancellation_states, service_disconnect_fails_calls}` (both), `tests/sol_review_races.rs::writer_failure_terminates_reader_and_pending_work` |
| Bounded subscriptions can reject a publication; latest spectators cannot hold a session transaction indefinitely | conforms: a latest subscriber never causes `BACKPRESSURE` | `router/state.rs::op_publish` (the latest branch skips every capacity check) | `tests/bus_acceptance.rs::a_latest_subscriber_never_refuses_a_publication` (both: 100 publications of 60 KB into one unconsumed slot, six times the bounded pool, none refused, 98 coalesced), with `tests/pubsub.rs::{bounded_fifo_and_atomic_backpressure, saturated_subscriber_does_not_block_control}` (both) for the bounded half | | Bounded subscriptions can reject a publication; latest spectators cannot hold a session transaction indefinitely | conforms: a latest subscriber never causes `BACKPRESSURE` | `router/state.rs::op_publish` (the latest branch skips every capacity check) | `tests/bus_acceptance.rs::a_latest_subscriber_never_refuses_a_publication` (both: 100 publications of 60 KB into one unconsumed slot, six times the bounded pool, none refused, 98 coalesced), with `tests/pubsub.rs::{bounded_fifo_and_atomic_backpressure, saturated_subscriber_does_not_block_control}` (both) for the bounded half, and `tests/integration.rs::session_over_one_router` (both: 20 snapshot publications accepted by both subscriptions while the presentation consumer reads nothing) |
| Sustained pinned-artifact quota exhaustion is surfaced as pressure, not solved by freeing live data | conforms: `QUOTA_EXCEEDED`, never eviction | `router/state.rs::{op_allocate, op_seal}` | `tests/artifacts.rs::quotas_are_enforced` (both) | | Sustained pinned-artifact quota exhaustion is surfaced as pressure, not solved by freeing live data | conforms: `QUOTA_EXCEEDED`, never eviction | `router/state.rs::{op_allocate, op_seal}` | `tests/artifacts.rs::quotas_are_enforced` (both) |
| Session and application policies choose disconnect, pause or fail; the router does not know which | conforms by absence | `router/state.rs` | `tests/pubsub.rs::saturated_subscriber_does_not_block_control` (both) | | Session and application policies choose disconnect, pause or fail; the router does not know which | conforms by absence | `router/state.rs` | `tests/pubsub.rs::saturated_subscriber_does_not_block_control` (both) |
@ -284,7 +284,7 @@ and once over a Unix socket, so `tests/rpc.rs::request_reply_roundtrip` means
| 3. Artifacts: allocate/seal/read; publication before seal fails; fan-out owns one object; the last consumer releases; a retained extracted frame survives a message drop | conforms | `tests/artifacts.rs` (28), `tests/conformance_artifacts.rs` (36) | | 3. Artifacts: allocate/seal/read; publication before seal fails; fan-out owns one object; the last consumer releases; a retained extracted frame survives a message drop | conforms | `tests/artifacts.rs` (28), `tests/conformance_artifacts.rs` (36) |
| 4. Faults: sender drops after admission, consumer dies mid-read, reply lost, queued frame replaced, subscription closes with in-use deliveries, router restarts, old release arrives; no double-free, use-after-reuse, unbounded tombstones or hidden replay | conforms | `tests/conformance_routing.rs::{caller_disconnect_detaches_dispatched_call_but_service_keeps_serving, subscriber_disconnect_releases_queued_and_delivered_artifacts_but_not_retention}`, `tests/artifacts.rs::{release_ids_are_watermarked, router_restart_invalidates_old_handles}`, `tests/bus_acceptance.rs::{a_lost_result_leaks_no_roots_and_the_endpoint_cache_still_replays, disconnect_releases_logical_ownership_without_mutating_open_bytes}`, `tests/sol_rereview_regressions.rs` (11) | | 4. Faults: sender drops after admission, consumer dies mid-read, reply lost, queued frame replaced, subscription closes with in-use deliveries, router restarts, old release arrives; no double-free, use-after-reuse, unbounded tombstones or hidden replay | conforms | `tests/conformance_routing.rs::{caller_disconnect_detaches_dispatched_call_but_service_keeps_serving, subscriber_disconnect_releases_queued_and_delivered_artifacts_but_not_retention}`, `tests/artifacts.rs::{release_ids_are_watermarked, router_restart_invalidates_old_handles}`, `tests/bus_acceptance.rs::{a_lost_result_leaks_no_roots_and_the_endpoint_cache_still_replays, disconnect_releases_logical_ownership_without_mutating_open_bytes}`, `tests/sol_rereview_regressions.rs` (11) |
| 5. RPC cache: an endpoint retains an artifact-bearing result, the original caller consumes it, a domain retry still returns valid bytes, eviction drops the last hold | conforms | `tests/rpc.rs::endpoint_cache_replays_artifact_results` (both), `tests/bus_acceptance.rs::a_retransmission_repeats_the_domain_request_under_a_fresh_call_id` (both) | | 5. RPC cache: an endpoint retains an artifact-bearing result, the original caller consumes it, a domain retry still returns valid bytes, eviction drops the last hold | conforms | `tests/rpc.rs::endpoint_cache_replays_artifact_results` (both), `tests/bus_acceptance.rs::a_retransmission_repeats_the_domain_request_under_a_fresh_call_id` (both) |
| 6. Integration: two parallel fake agents, a complete-batch environment RPC, committed snapshot publication and a deliberately slow presentation consumer on one router | conforms | `tests/integration.rs::session_over_one_router` (both) | | 6. Integration: two parallel fake agents, a complete-batch environment RPC, committed snapshot publication and a deliberately slow presentation consumer on one router | conforms; the consumer is slow by construction, held until the publisher's completion is observed, so its coalescing is forced rather than raced for | `tests/integration.rs::session_over_one_router` (both) |
| 7. Performance: 640x480x60 with three consumers, one delayed; p50/p95/p99 RPC latency, router CPU, copy and readback cost separately, RSS, store live and peak bytes, outstanding roots, collection lag, queue lengths, for one, two and four agents | conforms | `tests/perf.rs::frames_at_60hz_with_three_consumers` (`--ignored`); numbers below | | 7. Performance: 640x480x60 with three consumers, one delayed; p50/p95/p99 RPC latency, router CPU, copy and readback cost separately, RSS, store live and peak bytes, outstanding roots, collection lag, queue lengths, for one, two and four agents | conforms | `tests/perf.rs::frames_at_60hz_with_three_consumers` (`--ignored`); numbers below |
| The first executable example: a counter RPC, a pub/sub observer and a frame artifact held past message consumption, in one small Rust program, no game or browser | conforms | `examples/demo.rs` (`cargo run -p flybus --example demo`), asserted by `tests/example_demo.rs::the_example_shows_a_counter_rpc_an_observer_and_a_held_frame` | | The first executable example: a counter RPC, a pub/sub observer and a frame artifact held past message consumption, in one small Rust program, no game or browser | conforms | `examples/demo.rs` (`cargo run -p flybus --example demo`), asserted by `tests/example_demo.rs::the_example_shows_a_counter_rpc_an_observer_and_a_held_frame` |
@ -438,18 +438,30 @@ poll-until-the-router-settles the rest of the file already uses for router-side
no sleep, no timing constant, and the assertion now has the precondition its contract sentence no sleep, no timing constant, and the assertion now has the precondition its contract sentence
names. **360 runs after the fix, 0 failures** (240 debug, 120 release). names. **360 runs after the fix, 0 failures** (240 debug, 120 release).
Two other intermittent failures were seen in the same sweep and are **not** fixed here, since Two other intermittent failures were seen in the same sweep. They belonged to the bus slice
they belong to the bus slice rather than to this one: rather than to this one and were fixed there, in the same way and for the same reason:
- `tests/example_demo.rs::the_example_shows_a_counter_rpc_an_observer_and_a_held_frame`, - `tests/example_demo.rs::the_example_shows_a_counter_rpc_an_observer_and_a_held_frame`,
2 failures in 40 standalone runs plus 1 in 12 full-suite runs. It prints 2 failures in 40 standalone runs plus 1 in 12 full-suite runs. It printed
"while the frame is held: 1 artifact(s), 2 root(s)" instead of 1 root: the producer's hold "while the frame is held: 1 artifact(s), 2 root(s)" instead of 1 root: the producer's hold
release is queued on the control lane and had not been applied when the example read the release is queued on the control lane and had not been applied when the example read the
counts. The same shape of gap, in the guide deliverable's printed output. counts. `examples/demo.rs` now waits for that release before reading the counts, the same
- `tests/integration.rs::unix_socket::session_over_one_router`, 1 failure in 12 full-suite bounded poll it already used for collection eight lines below, so the line the guide quotes
runs and 0 in 40 standalone runs, at the assertion that the deliberately slow consumer is an observation rather than a race. The printed output is unchanged.
skipped snapshots. Under load it kept up, so the assertion is a timing claim about the - `tests/integration.rs::session_over_one_router`, 1 failure in 12 full-suite runs and 0 in 40
machine. standalone runs, at the assertion that the deliberately slow consumer skipped snapshots.
Under load it kept up, so the assertion was a timing claim about the machine: section 7
permits a latest subscriber to miss values, it does not oblige it to. The renderer is now
held until the publisher's twentieth receipt has returned -- the publisher's completion
observed, not timed -- so the coalescing is forced by construction, and the test asserts the
guarantees that do hold: each delivery carries the frame of the snapshot it announces,
deliveries arrive in publication order, the last value received is the latest published, the
renderer receives snapshots 1 and 20 of 20, both subscriptions accept all twenty publications
while the spectator reads nothing, and the eighteen replacements reported to the publisher at
admission are the same eighteen reported to the renderer on delivery and are exactly the
snapshots it did not receive. 16 failures in 40 runs beside four busy loops before, 0 in 40
after; the whole crate went from 10 failed runs in 20 to 0, and the workspace suite from 2
in 5 to 0.
## Contradictions ## Contradictions

View file

@ -105,6 +105,31 @@ names; a manifest missing any of them is not a complete checkpoint.
| `helperState` | External-helper state required for exact resume, as payload names | | `helperState` | External-helper state required for exact resume, as payload names |
| `payloads` | `[{name, byteLength, digest}]`, mirroring the payload table | | `payloads` | `[{name, byteLength, digest}]`, mirroring the payload table |
**Amendment, 2026-09-22 (STATE-01).** The table above names a holder for every payload except
the environment's own, although section 6's fixture has one (`world`) and a group install has
to map it by name like any other participant's. The manifest therefore also records:
| Field | Contents |
| --- | --- |
| `environment` | `{workerId, payload}`: which worker the world belonged to and the payload name holding its state |
The reference implementations' required-field set was also missing `helperState`, which this
section has listed from the start. Both are now in `REQUIRED_MANIFEST_FIELDS` in Rust and in
TypeScript, and the fixture was regenerated by the existing example. The schema set is
untouched, so `contractDigest` is unchanged.
`coordinator.eventWatermarks` is `{lastSourceStep, issued}`. The fixture illustrated
`{lastEventId, lastOrdinal}`, and it is the illustration that changed: an event id is derived
from the epoch, so a watermark spelled as one cannot be compared across the restore that
gives the session a new epoch, while a source step and an issued count can.
**A required-manifest-field change is compatibility-relevant and `contractDigest` does not
cover it.** The digest is taken over the schema set, and this manifest is not in it, so
`envelopeVersion` is the only thing that can carry such a change. It stays `1` here only
because no production `FLYSESS1` file exists yet: once one does, adding or removing a required
manifest field **must** bump `envelopeVersion`, because a reader of the older version would
otherwise accept a file it cannot completely read, or refuse one it could.
`payloads` is redundant with the table on purpose: the table is what a reader needs to map `payloads` is redundant with the table on purpose: the table is what a reader needs to map
bytes, and the manifest is what a store lists, compares and reports without opening the bytes, and the manifest is what a store lists, compares and reports without opening the
payload area. A reader checks that the two agree. payload area. A reader checks that the two agree.

View file

@ -190,6 +190,28 @@ restored time. It cannot advance gameplay to manufacture it. Capture/reconstruct
covers render/inspection state and any pending sensor pipeline. Agent state agrees with it; covers render/inspection state and any pending sensor pipeline. Agent state agrees with it;
do not replay reward or recalibrate merely to fill missing cached data. do not replay reward or recalibrate merely to fill missing cached data.
**Amendment, 2026-09-22 (STATE-01).** Three readings of this section, made explicit because
they are now enforced:
- `compatibilityDigest` on `CaptureResult` and `StageRestoreParams` is the **participant's**
capture compatibility digest of [worker interfaces](workers-v1.md) section 2 -- profile,
resolved seed, numerical model version and effective instance configuration for an agent;
backend, content, patch, controller and parser identity for an environment. It is not the
manifest's `compatibility` block of section 4, which is the composition's and which the
coordinator compares before anything is asked to stage. Both exist because they answer
different questions, and a restore that passed the second could still be handing an agent
another agent's brain.
- The observation `ActivateRestore` returns ran no transition, so it carries **no audio
chunk**, and one in it is refused. Section 2's chunk is the audio of an interval and this
observation covers none; MEDIA-01 implemented that rule as "boundary 0 carries no chunk",
which is true of the only such observation that slice could produce and false of this one.
The rule is about provenance, not about the boundary number.
- A participant that staged into a group install the coordinator then abandoned must be
**replaced** before another restore, exactly as one that activated must. It is holding a
validated replacement state that nothing installed, and [session RPC](ipc-v1.md) section 6
already refuses to silently reattach such a participant to an active epoch. Without this the
group's second attempt meets its own leftovers and calls them a conflict.
If emulator validation requires mutation, stage a stopped replacement emulator. If that cannot If emulator validation requires mutation, stage a stopped replacement emulator. If that cannot
provide externally atomic resume, advertise episode-restart, not exact-checkpoint. After all provide externally atomic resume, advertise episode-restart, not exact-checkpoint. After all
activation acknowledgments, install the coordinator's staged task/executor/admission state activation acknowledgments, install the coordinator's staged task/executor/admission state

View file

@ -1,6 +1,6 @@
# Rewards and learning # Rewards and learning
The live reward catalog of the Pokémon Red adapter, `pokered-unique8-v5`. The code of record is The live reward catalog of the Pokémon Red adapter, `pokered-unique8-v6`. The code of record is
`services/flysim/crates/flybrain-gb/src/pokemon_red/` (`catalog.rs` holds the values, `mod.rs` the `services/flysim/crates/flybrain-gb/src/pokemon_red/` (`catalog.rs` holds the values, `mod.rs` the
gates and the rules); this page says what each rule pays for and why it is allowed to. The gates and the rules); this page says what each rule pays for and why it is allowed to. The
prototype's own `docs/rewards-learning.md` in `fly-plays-pokemon` is where the first seven rules prototype's own `docs/rewards-learning.md` in `fly-plays-pokemon` is where the first seven rules
@ -24,6 +24,7 @@ change what the fly can do.
| `battle` | `wildwin` | +0.1, +0.05, +0.0333 | 100 ms | At most three observed wild KOs per `(map, species, level)` | | `battle` | `wildwin` | +0.1, +0.05, +0.0333 | 100 ms | At most three observed wild KOs per `(map, species, level)` |
| `badge` | `badge` | +3 | 400 ms | Each newly set badge bit | | `badge` | `badge` | +3 | 400 ms | Each newly set badge bit |
| `boundary` | `explore` | +0.05, +0.10 | 100 ms | First tile adjacent to one of the map's exits, and the exit tile itself; once per `(map, exit)` for the lifetime of the ledger | | `boundary` | `explore` | +0.05, +0.10 | 100 ms | First tile adjacent to one of the map's exits, and the exit tile itself; once per `(map, exit)` for the lifetime of the ledger |
| `catch` | `wildwin` | +0.30, +0.10 | 150 ms | A wild Pokémon kept by a ball: +0.30 for a species this run had never owned, +0.10 for a repeat; at most three payouts per species for the lifetime of the ledger |
Every value is positive: there are no loss or blackout penalties, and `catalog::rule("blackout")` Every value is positive: there are no loss or blackout penalties, and `catalog::rule("blackout")`
is `None` by test. The values in one frame sum into `R`, and the network reinforces once with is `None` by test. The values in one frame sum into `R`, and the network reinforces once with
@ -33,6 +34,54 @@ The feed-kind column is `RewardKind::from_adapter` in `services/flysim/crates/fl
`docs/feed-protocol.md` publishes seven counters, and an adapter kind that has no counter of its `docs/feed-protocol.md` publishes seven counters, and an adapter kind that has no counter of its
own shares the nearest one. It still reaches the page as an event with its own label. own shares the nearest one. It still reaches the page as an event with its own label.
Two consequences of that sharing are worth stating rather than discovering. `catch` publishes on
`wildwin` because a catch is a wild battle the fly won by keeping the Pokémon, and *not* on
`pokedex` because the `species` rule already pays for the Pokédex bit the same catch sets --
counting it twice would be the dishonest option. And the stage's ticker copy is keyed on the feed
kind, not on the catalog kind (`apps/stage/src/games/pokemon-red.ts`), so the row for a catch
currently reads "wild win". The event's own label, `CAUGHT #<species>`, is what reaches the event
log, `/status` and the checkpoint. Changing the ticker copy means opening the feed's closed kind
set, which this rule deliberately did not do.
## Catch rewards
The operator's decision of 2026-09-22: the fly is paid for *keeping* a wild Pokémon, not only for
knocking one out. The rule is one kind with two payouts, the way `boundary` is.
**How a catch is read.** From `wCapturedMonSpecies` (`$d11c`), whose comment in `ram/wram.asm` at
the pinned commit is "0 if no mon was captured". `ItemUseBall` zeroes it before every throw
(`.canUseBall`) and writes `wEnemyMonSpecies` into it only on the branch that keeps the Pokémon;
`UseBagItem`'s `.returnAfterCapturingMon` zeroes it again and sets `wBattleResult` to 2 on the way
out of the battle. `wBattleResult` is 2 on exactly two paths in the whole game -- that one, and a
link battle whose opponent ran -- so requiring both the species and the result means a byte read
out of a half-initialised battle cannot pay. The adapter records the species during the battle and
pays on the way out, where the wild-KO payout already lives.
Not from `wPartyCount`. A catch with a full party raises `wBoxCount` instead, and `wPartyCount`
also rises for a gift, a trade and a Pokémon taken out of the PC, so it would need a second rule
to mean anything. The cartridge's own flag needs none.
**What counts as a new species.** The `species` payout inside the same battle. Nothing but a catch
can set a `wPokedexOwned` bit during a wild battle, so a `species` payout between the battle
starting and the ball keeping the Pokémon *is* that Pokémon being new to the run. It is read this
way rather than off `wCapturedMonSpecies` because that byte is the cartridge's **internal** species
index while the owned bitset is by **Pokédex number**, and nothing in WRAM converts between the two
(`docs/design/macros-wram.md` section 2, "species numbering"). A battle restored from a checkpoint
written before this rule existed carries no "species payouts when it started", which reads as
"cannot tell" and pays the repeat amount: the conservative half, and at most 0.20 once.
**The budget.** Three payouts per species for the lifetime of the ledger, the same cap and the
same reason as the wild-KO rule's three: a species the fly can find over and over is a farm, and
three is enough for the behaviour to be learned. A rollback blocks every species already paid,
exactly as it blocks every wild-KO key already paid, so the same catch cannot be replayed for
reward. A Safari Zone or old-man battle pays nothing, because the whole sample is dropped a step
earlier with a visible mode; a trainer battle pays nothing, because balls cannot be thrown in one.
**The scale.** 0.30 on its own is below a new Pokédex entry (0.50), below a story flag (1.0) and
well below a badge (3.0). A catch of a new species pays 0.80 across two kinds, which sits between
a story flag and a badge -- deliberately, because it is the one event that is both a discovery and
a thing the fly had to do on purpose.
## Gates ## Gates
Semantic rewards are enabled for exactly one cartridge, the SHA-256 in `SUPPORTED_ROM`. Any other Semantic rewards are enabled for exactly one cartridge, the SHA-256 in `SUPPORTED_ROM`. Any other
@ -128,7 +177,19 @@ body picks the macro; the descending neurons press the buttons.**
## Honesty ## Honesty
The catalog now includes exits. That is worth saying plainly on the honesty panel, because paying The catalog now includes catches. The honesty panel's copy is not data-driven from the catalog --
`apps/stage/src/lib/schedule.ts`'s rotating card is four written lines and lists no kinds -- so
there was nothing to regenerate and the copy is unchanged. The sentences below are where the
argument lives.
Paying for a catch does not move the fly: the ball is thrown by a macro the mushroom body chose
among the ones the battle scene put on the pad, and the payout is read out of WRAM after the
frame. What it does do is make one of the palette's existing macros worth choosing, which is the
same kind of pressure every other rule applies. The cap is what keeps it from becoming a farm: a
run that finds one patch of grass and throws balls at the same species all night earns 0.50 from
it and then nothing.
The catalog also includes exits. That is worth saying plainly on the honesty panel, because paying
for a door is closer to telling the fly where to go than paying for a badge is: for a door is closer to telling the fly where to go than paying for a badge is:
- **still no button path.** Nothing in the adapter chooses or biases a button. The reward is read - **still no button path.** Nothing in the adapter chooses or biases a button. The reward is read

View file

@ -736,3 +736,11 @@ rewritten separately.
(73/73) because 82% of the fixed run is battle time; Fable shipped it on the same judgement as (73/73) because 82% of the fixed run is battle time; Fable shipped it on the same judgement as
v0.4.6 and started row 50 (MOVE n blocked on an unresponsive move list). The on-screen chat ring v0.4.6 and started row 50 (MOVE n blocked on an unresponsive move list). The on-screen chat ring
now survives a sim restart (sidecar in the hot dir, never in the checkpoint). now survives a sim restart (sidecar in the hot dir, never in the checkpoint).
- 2026-09-22 (v0.5.0, the operator's decision): the fly is paid for keeping a wild Pokémon. New
catalog kind `catch` (0.30 for a species this run never caught, 0.10 for a repeat, three payouts
per species), read from the captured-species byte and the battle result together; the existing
species rule still pays on top. Adapter `pokered-unique8-v6`; the compatibility string differs
in the adapter segment only, and a deploy with `FLY_ACCEPT_ADAPTERS=pokered-unique8-v5` migrates
a v5 checkpoint instead of refusing it. `fly-reset-to-milestone <N>` restarts the run from a
ladder rung (archives both stores first). The live run restarts from rung 7 with this release, so
the ladder is climbed again with the catch reward and the row-54 walks in place.

63
infra/05-deploy.sh Executable file → Normal file
View file

@ -186,6 +186,36 @@ else
log "05-deploy: CPUSET unset — heavy in-container steps run unpinned (no partition configured)" log "05-deploy: CPUSET unset — heavy in-container steps run unpinned (no partition configured)"
fi fi
# Whether the only difference between two compatibility strings is the adapter
# segment, and FLY_ACCEPT_ADAPTERS names the adapter the live checkpoints carry.
#
# The bash half of flybrain_gb::compatibility::decide, which is what flysim
# itself applies at restore. Both have to agree: a gate that let a deploy
# through and a flysim that then refused every checkpoint would be the black
# stream this whole section exists to prevent. The string is
# {kernel}/{adapter}/{fingerprint}/{plasticity}/binjgb:{rev}/pokered:{commit}/statefmt:{id},
# so the adapter is segment 1 and nothing else may move.
adapter_migration_accepted() {
local live="$1" new="$2" accepted="$3"
local -a live_parts new_parts
IFS='/' read -r -a live_parts <<< "$live"
IFS='/' read -r -a new_parts <<< "$new"
[ "${#live_parts[@]}" -eq "${#new_parts[@]}" ] || return 1
local i differing=0 index=-1
for ((i = 0; i < ${#live_parts[@]}; i++)); do
if [ "${live_parts[$i]}" != "${new_parts[$i]}" ]; then
differing=$((differing + 1))
index=$i
fi
done
[ "$differing" -eq 1 ] && [ "$index" -eq 1 ] || return 1
local entry
for entry in ${accepted//,/ }; do
[ "$entry" = "${live_parts[1]}" ] && return 0
done
return 1
}
# cpu_pin CMD [ARGS...] — run CMD inside the container on the page cpus. # cpu_pin CMD [ARGS...] — run CMD inside the container on the page cpus.
# Falls through to a plain ct_exec when no partition is configured, so this is # Falls through to a plain ct_exec when no partition is configured, so this is
# a no-op on an unpartitioned container rather than a new failure mode (a # a no-op on an unpartitioned container rather than a new failure mode (a
@ -262,6 +292,16 @@ if [ -n "$RELEASE_TARBALL" ]; then
# checkpoints and the on-screen chat ring's sidecar — so the new build warms # checkpoints and the on-screen chat ring's sidecar — so the new build warms
# up fresh. Everything learned so far is thrown away, which is why it is not # up fresh. Everything learned so far is thrown away, which is why it is not
# the default. # the default.
#
# FLY_ACCEPT_ADAPTERS is the *other* override, and the opposite one: it keeps
# the run. It names adapter version strings whose checkpoints the new build
# may migrate — e.g. FLY_ACCEPT_ADAPTERS=pokered-unique8-v5 for the deploy
# that adds the catch reward. It only applies when the adapter segment is the
# ONLY difference between the two strings and the new build's adapter says it
# can read that one; a dataset, kernel, emulator or state-format change is
# still a refusal, because none of those has a migration. The same variable is
# written into /etc/fly/fly.env below, so flysim applies the same rule at
# restore that this gate applied at deploy.
# ----------------------------------------------------------------------- # -----------------------------------------------------------------------
state_dir="${FLY_STATE_DIR:-/srv/fly/state}" state_dir="${FLY_STATE_DIR:-/srv/fly/state}"
hot_dir="${FLY_STATE_HOT_DIR:-/run/fly/state}" hot_dir="${FLY_STATE_HOT_DIR:-/run/fly/state}"
@ -287,6 +327,11 @@ if [ -n "$RELEASE_TARBALL" ]; then
log "05-deploy: no decodable checkpoint in ${state_dir} — nothing to compare, continuing" log "05-deploy: no decodable checkpoint in ${state_dir} — nothing to compare, continuing"
elif [ "$new_compat" = "$live_compat" ]; then elif [ "$new_compat" = "$live_compat" ]; then
log "05-deploy: checkpoint compatibility matches the live state, the new build will restore it" log "05-deploy: checkpoint compatibility matches the live state, the new build will restore it"
elif [ -n "${FLY_ACCEPT_ADAPTERS:-}" ] \
&& adapter_migration_accepted "$live_compat" "$new_compat" "$FLY_ACCEPT_ADAPTERS"; then
log "05-deploy: FLY_ACCEPT_ADAPTERS=${FLY_ACCEPT_ADAPTERS} — the adapter version is the only difference, and it is named; the run is KEPT and migrated"
log "05-deploy: live: $live_compat"
log "05-deploy: new: $new_compat"
elif [ "${FLY_RESET_STATE:-0}" = 1 ]; then elif [ "${FLY_RESET_STATE:-0}" = 1 ]; then
archive="${state_dir}.$(date -u +%Y%m%d%H%M%S)" archive="${state_dir}.$(date -u +%Y%m%d%H%M%S)"
log "05-deploy: FLY_RESET_STATE=1 — compatibility CHANGED, archiving the durable state to ${archive} and clearing the hot ring" log "05-deploy: FLY_RESET_STATE=1 — compatibility CHANGED, archiving the durable state to ${archive} and clearing the hot ring"
@ -300,8 +345,12 @@ if [ -n "$RELEASE_TARBALL" ]; then
die "05-deploy: REFUSING to deploy release ${version}: its checkpoint compatibility string does not match the live state in ${state_dir}, so flysim would refuse every checkpoint there and then refuse to start at all — a black stream. die "05-deploy: REFUSING to deploy release ${version}: its checkpoint compatibility string does not match the live state in ${state_dir}, so flysim would refuse every checkpoint there and then refuse to start at all — a black stream.
live state: ${live_compat} live state: ${live_compat}
new build: ${new_compat} new build: ${new_compat}
The difference is usually an adapter/ladder or dataset version bump. Two ways forward: The difference is usually an adapter/ladder or dataset version bump. Three ways forward:
* deploy a build whose string matches (check out the commit the running release was built from), or * deploy a build whose string matches (check out the commit the running release was built from), or
* if the ADAPTER VERSION is the only segment that differs and the new build documents a
migration from the old one, re-run with FLY_ACCEPT_ADAPTERS set to the adapter id in the live
string (e.g. FLY_ACCEPT_ADAPTERS=pokered-unique8-v5). The run is kept; flysim applies the same
rule at restore. See docs/design/flysim.md, \"Restoring across an adapter version\", or
* accept losing everything the brain has learned and re-run with FLY_RESET_STATE=1, which * accept losing everything the brain has learned and re-run with FLY_RESET_STATE=1, which
archives ${state_dir}'s checkpoints to ${state_dir}.<timestamp> (kept, not deleted) and archives ${state_dir}'s checkpoints to ${state_dir}.<timestamp> (kept, not deleted) and
clears ${hot_dir} so the new build warms up fresh. clears ${hot_dir} so the new build warms up fresh.
@ -457,6 +506,16 @@ trap 'rm -f "$tmp_fly_env" "$tmp_flypush_env"' EXIT
if [[ -n "${FLY_MACRO_BLOCKED_MINUTES:-}" ]]; then if [[ -n "${FLY_MACRO_BLOCKED_MINUTES:-}" ]]; then
echo "FLY_MACRO_BLOCKED_MINUTES=${FLY_MACRO_BLOCKED_MINUTES}" echo "FLY_MACRO_BLOCKED_MINUTES=${FLY_MACRO_BLOCKED_MINUTES}"
fi fi
# Adapter versions whose checkpoints this build may migrate
# (flybrain_gb::compatibility, docs/design/flysim.md "Restoring across an
# adapter version"). Only written when it is set, because the safe state is
# absent: an empty or missing variable migrates nothing, which is what every
# deploy before 2026-09-22 did. It stays in fly.env for as long as the
# operator leaves it on the deploy command line, so removing the opt-in is
# one deploy without it.
if [[ -n "${FLY_ACCEPT_ADAPTERS:-}" ]]; then
echo "FLY_ACCEPT_ADAPTERS=${FLY_ACCEPT_ADAPTERS}"
fi
# flybridge (services/bridge/src/config.ts). Nothing wrote these before, so # flybridge (services/bridge/src/config.ts). Nothing wrote these before, so
# flybridge.service had no EnvironmentFile= at all and the service refused to # flybridge.service had no EnvironmentFile= at all and the service refused to
# start with "CHANNEL is required / BOT_USER is required / GAME_TITLE is # start with "CHANNEL is required / BOT_USER is required / GAME_TITLE is
@ -607,7 +666,7 @@ fi
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
log "05-deploy: converging bin/ helpers to /opt/fly/bin" log "05-deploy: converging bin/ helpers to /opt/fly/bin"
ct_exec "$CTID" -- mkdir -p /opt/fly/bin ct_exec "$CTID" -- mkdir -p /opt/fly/bin
for name in fly-watchdog fly-recap fly-retention flypush flystage-launch flycast-launch wait-for-x wait-for-stage wait-for-health; do for name in fly-watchdog fly-recap fly-retention fly-reset-to-milestone flypush flystage-launch flycast-launch wait-for-x wait-for-stage wait-for-health; do
converge_file "$CTID" "$INFRA_DIR/bin/$name" "/opt/fly/bin/$name" 0755 root:root >/dev/null converge_file "$CTID" "$INFRA_DIR/bin/$name" "/opt/fly/bin/$name" 0755 root:root >/dev/null
done done

View file

@ -0,0 +1,77 @@
#!/usr/bin/env bash
# infra/bin/fly-reset-to-milestone — restart the run from an earlier ladder rung,
# instead of from scratch.
#
# The operator's decision of 2026-09-22: "restart the live run from an early
# checkpoint instead of from scratch". 05-deploy's FLY_RESET_STATE=1 cannot do
# that — it archives the durable state and the next start warms up a fresh fly,
# losing everything the brain has learned. This promotes one milestone archive
# (milestone-<N>.checkpoint, written at the first commit at a new best rank and
# never rotated away) to being what both stores restore.
#
# Usage: fly-reset-to-milestone <N>
# Run INSIDE the container, as root, with flysim STOPPED. It refuses
# otherwise, and it refuses a rung this run never reached.
#
# The whole sequence — stop, reset, deploy with the adapter opt-in, start,
# verify the rank — is in infra/docs/runbook.md, "Restart the run from a rung".
# Nothing here is destructive on its own: every file in both stores is copied to
# a dated directory next to the durable one before anything is rewritten.
set -euo pipefail
: "${FLY_STATE_DIR:=/srv/fly/state}"
: "${FLY_STATE_HOT_DIR:=/run/fly/state}"
: "${FLY_RELEASE_DIR:=/opt/fly/current}"
: "${FLY_SERVICE:=flysim.service}"
: "${FLY_USER:=fly}"
FLYSIM="${FLY_BIN:-${FLY_RELEASE_DIR}/flysim}"
log() { echo "fly-reset-to-milestone: $*" >&2; }
die() { log "$*"; exit 1; }
RANK="${1:-}"
if [ "$#" -ne 1 ] || ! [[ "$RANK" =~ ^[0-9]+$ ]]; then
die "usage: fly-reset-to-milestone <rung> (e.g. fly-reset-to-milestone 9)"
fi
# --- refusals ----------------------------------------------------------------
# A running flysim owns both stores: it commits a hot checkpoint every few
# seconds and a durable one every few minutes, so a reset underneath it would be
# overwritten within the minute and the tool would have lied.
if command -v systemctl >/dev/null 2>&1 && systemctl is-active --quiet "$FLY_SERVICE"; then
die "$FLY_SERVICE is running. Stop it first: systemctl stop $FLY_SERVICE"
fi
[ -x "$FLYSIM" ] || die "no flysim binary at $FLYSIM (set FLY_BIN to point at one)"
milestone="${FLY_STATE_DIR}/milestone-${RANK}.checkpoint"
# The binary refuses this too, and refuses before it copies anything; checking
# here as well is what makes the message name the rungs that do exist.
if [ ! -f "$milestone" ]; then
log "no milestone archive for rung ${RANK}: $milestone does not exist."
log "rungs this run reached:"
ls -1 "${FLY_STATE_DIR}"/milestone-*.checkpoint 2>/dev/null \
| sed 's|.*/milestone-||; s|\.checkpoint$||' | sort -n | tr '\n' ' ' >&2 || true
echo >&2
exit 1
fi
# --- the reset ---------------------------------------------------------------
log "resetting to rung ${RANK} (durable ${FLY_STATE_DIR}, hot ${FLY_STATE_HOT_DIR})"
FLY_STATE="$FLY_STATE_DIR" FLY_STATE_HOT="$FLY_STATE_HOT_DIR" \
"$FLYSIM" --reset-to-milestone "$RANK"
# flysim runs unprivileged; this tool runs as root, so everything it wrote and
# everything it archived has to go back to the service account.
if command -v chown >/dev/null 2>&1 && id "$FLY_USER" >/dev/null 2>&1; then
chown -R "${FLY_USER}:${FLY_USER}" "$FLY_STATE_DIR" "$FLY_STATE_HOT_DIR" 2>/dev/null || true
for dir in "${FLY_STATE_DIR}".reset-*; do
[ -d "$dir" ] && chown -R "${FLY_USER}:${FLY_USER}" "$dir"
done
fi
log "done. Next, per infra/docs/runbook.md:"
log " 1. deploy the build whose adapter wrote that checkpoint, or deploy the new"
log " one with FLY_ACCEPT_ADAPTERS set to the checkpoint's adapter id"
log " 2. systemctl start $FLY_SERVICE"
log " 3. curl -s localhost:7401/status | grep -o '\"rank\":[0-9]*'"

View file

@ -231,6 +231,59 @@ auto-reset (`docs/design/flysim.md` section 8: "no automatic fresh start, ever")
is deliberate — a silent reset would be indistinguishable from real progress on stream. is deliberate — a silent reset would be indistinguishable from real progress on stream.
A deliberate reset means moving `/srv/fly/state` aside by hand. A deliberate reset means moving `/srv/fly/state` aside by hand.
## Restart the run from a rung
When the run has to go back to an earlier milestone rather than start over — the operator's
decision of 2026-09-22 was "restart the live run from an early checkpoint instead of from
scratch". `FLY_RESET_STATE=1` is the wrong tool: it archives the durable state and the next start
warms up a fresh fly, losing everything the brain has learned.
`infra/bin/fly-reset-to-milestone <N>` promotes `milestone-<N>.checkpoint` to being what both
stores restore, with the ratchet's attempts and recoveries back at zero. It copies every file in
both stores to `/srv/fly/state.reset-<UTC>` first, so it is reversible by hand. It refuses while
flysim is running, and refuses a rung this run never reached.
The whole sequence, in order. Claim the container in the host's agent claim log first, like any
other work on it.
```
CTID=<release-ctid>
N=9 # the rung to restart from
# 1. what rungs exist at all
pct exec $CTID -- ls -1 /srv/fly/state/milestone-*.checkpoint
# 2. stop flysim (it owns both stores; a reset underneath it is overwritten within the minute)
pct exec $CTID -- systemctl stop flysim.service
# 3. the reset. Prints what it did, one line per step.
pct exec $CTID -- /opt/fly/bin/fly-reset-to-milestone $N
# 4. deploy. Two cases:
# (a) the running release already wrote that checkpoint -> nothing to deploy, skip to 5.
# (b) the new build bumps the ADAPTER VERSION and nothing else -> name the checkpoint's
# adapter so the gate and flysim both migrate instead of refusing:
FLY_ACCEPT_ADAPTERS=pokered-unique8-v5 infra/05-deploy.sh <release-env> <release-tarball>
# The gate logs "the adapter version is the only difference, and it is named; the run is KEPT
# and migrated", and writes FLY_ACCEPT_ADAPTERS into /etc/fly/fly.env so flysim applies the
# same rule at restore. Anything else about the string differing is still a refusal.
# 5. start
pct exec $CTID -- systemctl start flysim.service
# 6. verify: the rank is the rung, and the restore came from the generation the tool wrote
pct exec $CTID -- curl -s http://127.0.0.1:7401/status | jq '.milestone.rank, .game.badges, .checkpoint'
pct exec $CTID -- journalctl -u flysim -n 40 --no-pager | grep -E 'restored|migration|compatibility'
```
Step 6 is the one that must be read rather than assumed. The rank is recomputed by the adapter
from the restored game state, not taken from the ratchet, so a rank that is *not* N means the
milestone archive was taken somewhere other than where its name says — stop and look before
starting a stream on it.
To undo: stop flysim, move the contents of `/srv/fly/state.reset-<UTC>/durable` back into
`/srv/fly/state`, delete the generation the tool wrote, and start again.
## Restore from the backup host ## Restore from the backup host
``` ```

14
infra/env/example.env vendored
View file

@ -317,6 +317,20 @@ FLY_MACRO_MODE=raw
# target once more. Unset means the default, 10. # target once more. Unset means the default, 10.
# FLY_MACRO_BLOCKED_MINUTES=10 # FLY_MACRO_BLOCKED_MINUTES=10
# --- restoring across an adapter version ------------------------------------
# Adapter version strings whose checkpoints this build may migrate, comma- or
# space-separated (docs/design/flysim.md, "Restoring across an adapter
# version"). Unset -- the default, and what every deploy before 2026-09-22 did
# -- migrates nothing: a build whose compatibility string differs from the live
# state's is refused by 05-deploy's gate and by flysim at restore.
#
# It applies only when the ADAPTER segment is the only difference between the
# two strings AND the new build's adapter declares a migration from that one. A
# dataset, kernel, plasticity, emulator or state-format difference is still a
# refusal. Set it for the one deploy that needs it and leave it out afterwards;
# 05-deploy writes it into /etc/fly/fly.env only while it is set.
# FLY_ACCEPT_ADAPTERS=pokered-unique8-v5
# --- push mode -------------------------------------------------------------- # --- push mode --------------------------------------------------------------
# local: flypush.service stays disabled, everything else identical to prod. # local: flypush.service stays disabled, everything else identical to prod.
# twitch: flypush.service is enabled by 07-enable.sh. # twitch: flypush.service is enabled by 07-enable.sh.

View file

@ -223,7 +223,12 @@ export function decode(input: Uint8Array): Envelope {
}; };
} }
/** The manifest fields state-media-v1 section 4 requires. */ /**
* The manifest fields state-media-v1 section 4 requires.
*
* `helperState` and `environment` join the list under the 2026-09-22 amendment to
* checkpoint-envelope-v1 section 3.
*/
export const REQUIRED_MANIFEST_FIELDS = [ export const REQUIRED_MANIFEST_FIELDS = [
'envelopeVersion', 'envelopeVersion',
'checkpointId', 'checkpointId',
@ -236,6 +241,8 @@ export const REQUIRED_MANIFEST_FIELDS = [
'compatibility', 'compatibility',
'agents', 'agents',
'coordinator', 'coordinator',
'environment',
'helperState',
'payloads', 'payloads',
] as const; ] as const;

View file

@ -197,8 +197,9 @@ fn checkpoint_envelope() -> String {
"priorInspection": "prior-inspection", "priorInspection": "prior-inspection",
"executorState": [{"agentId": "fly-a", "payload": "executor-fly-a"}], "executorState": [{"agentId": "fly-a", "payload": "executor-fly-a"}],
"admissionState": null, "admissionState": null,
"eventWatermarks": {"lastEventId": "evt-1", "lastOrdinal": "7"}, "eventWatermarks": {"lastSourceStep": "42", "issued": "7"},
}, },
"environment": {"workerId": "arena", "payload": "world"},
"helperState": [], "helperState": [],
"payloads": payload_table(), "payloads": payload_table(),
}); });

View file

@ -59,10 +59,14 @@
], ],
"admissionState": null, "admissionState": null,
"eventWatermarks": { "eventWatermarks": {
"lastEventId": "evt-1", "lastSourceStep": "42",
"lastOrdinal": "7" "issued": "7"
} }
}, },
"environment": {
"workerId": "arena",
"payload": "world"
},
"helperState": [], "helperState": [],
"payloads": [ "payloads": [
{ {
@ -115,49 +119,49 @@
} }
], ],
"envelope": { "envelope": {
"base64": "RkxZU0VTUzEBAAAAIAAAAAAIAAAFAAAAIAgAAAAAAAB7ImFnZW50cyI6W3siYWdlbnRJZCI6ImZseS1hIiwiYnJhaW5UaWNrcyI6IjI1MzQiLCJkYXRhc2V0RGlnZXN0IjoiNmMwYWYxZjA3ODRlZjYzYTM5M2VlNzdkNjE0ZTgyNDZjNjI1MDUxMzYwZjNmMWE0ODgzODM3NGM1ZDM1NWI1MiIsIm1vZGVsVmVyc2lvbiI6ImxpZi0xbXMtZjY0LXYyIiwicGF5bG9hZCI6ImFnZW50LWZseS1hIiwicGxhc3RpY2l0eVZlcnNpb24iOiJmbHkta2MtbWJvbi1yc3RkcC12MiIsInByb2ZpbGVEaWdlc3QiOiIxOTAwZWFiNmMwMjg0ODNkNzEyNjU5OWVlNmY1MGRlMGQyNzkwN2I1YzY1ZmE5MDUyNDU4MGI0YjBmOTg1MmIwIiwicmVtYWluZGVyIjp7ImRlbm9taW5hdG9yIjoiMyIsIm51bWVyYXRvciI6IjEwMDAwMDAifSwic2VlZCI6LTE4NDk0NjA2M31dLCJjaGVja3BvaW50SWQiOiJja3B0LTEiLCJjb21wYXRpYmlsaXR5Ijp7ImJhY2tlbmREaWdlc3QiOiIxMGUwOGE0MTllODUwZWJhMWViYmExOGZkZDI4ZWI3ZWMxYjdlOGJhYTliY2MzYjk3M2UyYjg4OTFlYzcyNmJlIiwiY29udGVudERpZ2VzdCI6ImVkNzAwMmI0MzllOWFjODQ1ZjIyMzU3ZDgyMmJhYzE0NDQ3MzBmYmRiNjAxNmQzZWM5NDMyMjk3YjllYzlmNzMiLCJjb250cm9sbGVyRGlnZXN0IjoiYzE0NzIxMzViMTRjNzdjOGJlZjk4ZTczZjcwMjA4MzI1ZmEwZGNmMWU2YmQ2NjhhZTliMzFhOWNlYTI5NWZlNyIsInBhcnNlckRpZ2VzdCI6ImIxN2Q0NTEyMTE1MDkyOGYyMTQ2YWY0OWUxOTVlZmYxZWVmNWQ2NzMyNWJlMjczYTczM2ZiNzRhY2FkYWEzNDIiLCJwYXRjaERpZ2VzdCI6ImE0ODk1ZWI0NGFmYzMzNmZlY2JiYTZlNTIwY2Q2N2UxNzhkYWNlMDI3NjY1NWQxMDJmY2VmZmE4ZTVmNzA1NzAiLCJzdGF0ZUZvcm1hdElkIjoiZmx5c2Vzcy0xIn0sImNvbXBvc2l0aW9uRGlnZXN0IjoiNzMwZDcyNWM4YTU5ZDNhNzMwM2RlZjJiZWQwNDFhNTc3ZWRiNDI1NWFhYmQ0ODg5Y2UxMjkxODMxMWQ5NTJmMCIsImNvb3JkaW5hdG9yIjp7ImFkbWlzc2lvblN0YXRlIjpudWxsLCJldmVudFdhdGVybWFya3MiOnsibGFzdEV2ZW50SWQiOiJldnQtMSIsImxhc3RPcmRpbmFsIjoiNyJ9LCJleGVjdXRvclN0YXRlIjpbeyJhZ2VudElkIjoiZmx5LWEiLCJwYXlsb2FkIjoiZXhlY3V0b3ItZmx5LWEifV0sInByaW9ySW5zcGVjdGlvbiI6InByaW9yLWluc3BlY3Rpb24iLCJ0YXNrTGVkZ2VyIjoidGFzay1sZWRnZXIifSwiZW52ZWxvcGVWZXJzaW9uIjoxLCJlcGlzb2RlSWQiOiJlcGlzb2RlLTEiLCJoZWxwZXJTdGF0ZSI6W10sInBheWxvYWRzIjpbeyJieXRlTGVuZ3RoIjoiMTciLCJkaWdlc3QiOiIxMzIxZGZmYjBjZGM2ZjkwOTJjYmY3ZmEyYTVmYzY4YmJlZDEyYzk5M2Q1YWQzOTgyNjQwMTI4MTBjZTliZjkzIiwibmFtZSI6ImFnZW50LWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTQiLCJkaWdlc3QiOiIzYWVlNjBkZjdlMjllZmViYTdmNWY5OWZjNTg2NzY0N2IzNmFlYmZmMWQ1ZDNjODM4ZGJmZjMyMzEyMmU2NDYyIiwibmFtZSI6ImV4ZWN1dG9yLWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTEiLCJkaWdlc3QiOiI0MGIwMGVkMmJiYmE5MDFkNjgyMDVmZjcxYjA0YTQ0YjllZTUzYzUxY2IzMTA5YWEyY2VhYTQ0ZjFjNDU3MjdlIiwibmFtZSI6InRhc2stbGVkZ2VyIn0seyJieXRlTGVuZ3RoIjoiMTAiLCJkaWdlc3QiOiIyYzEzYjdiNGQ5YTk5MTY4MDFhYjkxOTFjMzE0ZjMxYjA0NWU5YjljNWI2NjlhNmMwNDc0ZjAyMTdlZjc1YmY1IiwibmFtZSI6InByaW9yLWluc3BlY3Rpb24ifSx7ImJ5dGVMZW5ndGgiOiI2NCIsImRpZ2VzdCI6ImY1YTVmZDQyZDE2YTIwMzAyNzk4ZWY2ZWQzMDk5NzliNDMwMDNkMjMyMGQ5ZjBlOGVhOTgzMWE5Mjc1OWZiNGIiLCJuYW1lIjoid29ybGQifV0sInBvcnRNYXAiOlt7ImFnZW50SWQiOiJmbHktYSIsInBvcnRJZCI6InBvcnQtMSJ9XSwic2NoZWR1bGVySWQiOiJsb2Nrc3RlcC12MSIsInNvdXJjZVNjb3BlIjp7ImVwb2NoIjoiZXBvY2gtMSIsInNlc3Npb25JZCI6ImRlbW8iLCJzdGVwIjoiNDIifSwid29ybGRUaW1lIjp7ImRlbm9taW5hdG9yIjoiMSIsIm51bWVyYXRvciI6IjcwMDAwMDAwMCJ9fWFnZW50LWZseS1hAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABQCgAAAAAAABEAAAAAAAAAEyHf+wzcb5CSy/f6Kl/Gi77RLJk9WtOYJkASgQzpv5NleGVjdXRvci1mbHktYQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaAoAAAAAAAAOAAAAAAAAADruYN9+Ke/rp/X5n8WGdkezauv/HV08g42/8yMSLmRidGFzay1sZWRnZXIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHgKAAAAAAAACwAAAAAAAABAsA7Su7qQHWggX/cbBKRLnuU8UcsxCaos6qRPHEVyfnByaW9yLWluc3BlY3Rpb24AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACICgAAAAAAAAoAAAAAAAAALBO3tNmpkWgBq5GRwxTzGwRem5xbZppsBHTwIX73W/V3b3JsZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAmAoAAAAAAABAAAAAAAAAAPWl/ULRaiAwJ5jvbtMJl5tDAD0jINnw6OqYMaknWftLYWdlbnQgc3RhdGUgYnl0ZXMAAAAAAAAAZXhlY3V0b3Igc3RhdGUAAHsicmFuayI6MTB9AAAAAAB7Im1hcCI6NDB9AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgLAAAAAAAAq++fEx+FvDZho/eB4imbENN4HZrGNC2OCAsI7/gp9r5GTFlTRVNTRg==", "base64": "RkxZU0VTUzEBAAAAIAAAADAIAAAFAAAAUAgAAAAAAAB7ImFnZW50cyI6W3siYWdlbnRJZCI6ImZseS1hIiwiYnJhaW5UaWNrcyI6IjI1MzQiLCJkYXRhc2V0RGlnZXN0IjoiNmMwYWYxZjA3ODRlZjYzYTM5M2VlNzdkNjE0ZTgyNDZjNjI1MDUxMzYwZjNmMWE0ODgzODM3NGM1ZDM1NWI1MiIsIm1vZGVsVmVyc2lvbiI6ImxpZi0xbXMtZjY0LXYyIiwicGF5bG9hZCI6ImFnZW50LWZseS1hIiwicGxhc3RpY2l0eVZlcnNpb24iOiJmbHkta2MtbWJvbi1yc3RkcC12MiIsInByb2ZpbGVEaWdlc3QiOiIxOTAwZWFiNmMwMjg0ODNkNzEyNjU5OWVlNmY1MGRlMGQyNzkwN2I1YzY1ZmE5MDUyNDU4MGI0YjBmOTg1MmIwIiwicmVtYWluZGVyIjp7ImRlbm9taW5hdG9yIjoiMyIsIm51bWVyYXRvciI6IjEwMDAwMDAifSwic2VlZCI6LTE4NDk0NjA2M31dLCJjaGVja3BvaW50SWQiOiJja3B0LTEiLCJjb21wYXRpYmlsaXR5Ijp7ImJhY2tlbmREaWdlc3QiOiIxMGUwOGE0MTllODUwZWJhMWViYmExOGZkZDI4ZWI3ZWMxYjdlOGJhYTliY2MzYjk3M2UyYjg4OTFlYzcyNmJlIiwiY29udGVudERpZ2VzdCI6ImVkNzAwMmI0MzllOWFjODQ1ZjIyMzU3ZDgyMmJhYzE0NDQ3MzBmYmRiNjAxNmQzZWM5NDMyMjk3YjllYzlmNzMiLCJjb250cm9sbGVyRGlnZXN0IjoiYzE0NzIxMzViMTRjNzdjOGJlZjk4ZTczZjcwMjA4MzI1ZmEwZGNmMWU2YmQ2NjhhZTliMzFhOWNlYTI5NWZlNyIsInBhcnNlckRpZ2VzdCI6ImIxN2Q0NTEyMTE1MDkyOGYyMTQ2YWY0OWUxOTVlZmYxZWVmNWQ2NzMyNWJlMjczYTczM2ZiNzRhY2FkYWEzNDIiLCJwYXRjaERpZ2VzdCI6ImE0ODk1ZWI0NGFmYzMzNmZlY2JiYTZlNTIwY2Q2N2UxNzhkYWNlMDI3NjY1NWQxMDJmY2VmZmE4ZTVmNzA1NzAiLCJzdGF0ZUZvcm1hdElkIjoiZmx5c2Vzcy0xIn0sImNvbXBvc2l0aW9uRGlnZXN0IjoiNzMwZDcyNWM4YTU5ZDNhNzMwM2RlZjJiZWQwNDFhNTc3ZWRiNDI1NWFhYmQ0ODg5Y2UxMjkxODMxMWQ5NTJmMCIsImNvb3JkaW5hdG9yIjp7ImFkbWlzc2lvblN0YXRlIjpudWxsLCJldmVudFdhdGVybWFya3MiOnsiaXNzdWVkIjoiNyIsImxhc3RTb3VyY2VTdGVwIjoiNDIifSwiZXhlY3V0b3JTdGF0ZSI6W3siYWdlbnRJZCI6ImZseS1hIiwicGF5bG9hZCI6ImV4ZWN1dG9yLWZseS1hIn1dLCJwcmlvckluc3BlY3Rpb24iOiJwcmlvci1pbnNwZWN0aW9uIiwidGFza0xlZGdlciI6InRhc2stbGVkZ2VyIn0sImVudmVsb3BlVmVyc2lvbiI6MSwiZW52aXJvbm1lbnQiOnsicGF5bG9hZCI6IndvcmxkIiwid29ya2VySWQiOiJhcmVuYSJ9LCJlcGlzb2RlSWQiOiJlcGlzb2RlLTEiLCJoZWxwZXJTdGF0ZSI6W10sInBheWxvYWRzIjpbeyJieXRlTGVuZ3RoIjoiMTciLCJkaWdlc3QiOiIxMzIxZGZmYjBjZGM2ZjkwOTJjYmY3ZmEyYTVmYzY4YmJlZDEyYzk5M2Q1YWQzOTgyNjQwMTI4MTBjZTliZjkzIiwibmFtZSI6ImFnZW50LWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTQiLCJkaWdlc3QiOiIzYWVlNjBkZjdlMjllZmViYTdmNWY5OWZjNTg2NzY0N2IzNmFlYmZmMWQ1ZDNjODM4ZGJmZjMyMzEyMmU2NDYyIiwibmFtZSI6ImV4ZWN1dG9yLWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTEiLCJkaWdlc3QiOiI0MGIwMGVkMmJiYmE5MDFkNjgyMDVmZjcxYjA0YTQ0YjllZTUzYzUxY2IzMTA5YWEyY2VhYTQ0ZjFjNDU3MjdlIiwibmFtZSI6InRhc2stbGVkZ2VyIn0seyJieXRlTGVuZ3RoIjoiMTAiLCJkaWdlc3QiOiIyYzEzYjdiNGQ5YTk5MTY4MDFhYjkxOTFjMzE0ZjMxYjA0NWU5YjljNWI2NjlhNmMwNDc0ZjAyMTdlZjc1YmY1IiwibmFtZSI6InByaW9yLWluc3BlY3Rpb24ifSx7ImJ5dGVMZW5ndGgiOiI2NCIsImRpZ2VzdCI6ImY1YTVmZDQyZDE2YTIwMzAyNzk4ZWY2ZWQzMDk5NzliNDMwMDNkMjMyMGQ5ZjBlOGVhOTgzMWE5Mjc1OWZiNGIiLCJuYW1lIjoid29ybGQifV0sInBvcnRNYXAiOlt7ImFnZW50SWQiOiJmbHktYSIsInBvcnRJZCI6InBvcnQtMSJ9XSwic2NoZWR1bGVySWQiOiJsb2Nrc3RlcC12MSIsInNvdXJjZVNjb3BlIjp7ImVwb2NoIjoiZXBvY2gtMSIsInNlc3Npb25JZCI6ImRlbW8iLCJzdGVwIjoiNDIifSwid29ybGRUaW1lIjp7ImRlbm9taW5hdG9yIjoiMSIsIm51bWVyYXRvciI6IjcwMDAwMDAwMCJ9fWFnZW50LWZseS1hAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACACgAAAAAAABEAAAAAAAAAEyHf+wzcb5CSy/f6Kl/Gi77RLJk9WtOYJkASgQzpv5NleGVjdXRvci1mbHktYQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAmAoAAAAAAAAOAAAAAAAAADruYN9+Ke/rp/X5n8WGdkezauv/HV08g42/8yMSLmRidGFzay1sZWRnZXIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKgKAAAAAAAACwAAAAAAAABAsA7Su7qQHWggX/cbBKRLnuU8UcsxCaos6qRPHEVyfnByaW9yLWluc3BlY3Rpb24AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC4CgAAAAAAAAoAAAAAAAAALBO3tNmpkWgBq5GRwxTzGwRem5xbZppsBHTwIX73W/V3b3JsZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAyAoAAAAAAABAAAAAAAAAAPWl/ULRaiAwJ5jvbtMJl5tDAD0jINnw6OqYMaknWftLYWdlbnQgc3RhdGUgYnl0ZXMAAAAAAAAAZXhlY3V0b3Igc3RhdGUAAHsicmFuayI6MTB9AAAAAAB7Im1hcCI6NDB9AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADgLAAAAAAAAX4L9WkdX0MViD3h5YJgf7VQgwocWU4XJoKX6MUwl+8hGTFlTRVNTRg==",
"byteLength": 2824, "byteLength": 2872,
"layout": { "layout": {
"headerBytes": 32, "headerBytes": 32,
"manifestOffset": "32", "manifestOffset": "32",
"manifestBytes": 2048, "manifestBytes": 2096,
"tableOffset": "2080", "tableOffset": "2128",
"tableEntryBytes": 112, "tableEntryBytes": 112,
"entries": [ "entries": [
{ {
"name": "agent-fly-a", "name": "agent-fly-a",
"offset": "2640", "offset": "2688",
"byteLength": "17", "byteLength": "17",
"digest": "1321dffb0cdc6f9092cbf7fa2a5fc68bbed12c993d5ad398264012810ce9bf93" "digest": "1321dffb0cdc6f9092cbf7fa2a5fc68bbed12c993d5ad398264012810ce9bf93"
}, },
{ {
"name": "executor-fly-a", "name": "executor-fly-a",
"offset": "2664", "offset": "2712",
"byteLength": "14", "byteLength": "14",
"digest": "3aee60df7e29efeba7f5f99fc5867647b36aebff1d5d3c838dbff323122e6462" "digest": "3aee60df7e29efeba7f5f99fc5867647b36aebff1d5d3c838dbff323122e6462"
}, },
{ {
"name": "task-ledger", "name": "task-ledger",
"offset": "2680", "offset": "2728",
"byteLength": "11", "byteLength": "11",
"digest": "40b00ed2bbba901d68205ff71b04a44b9ee53c51cb3109aa2ceaa44f1c45727e" "digest": "40b00ed2bbba901d68205ff71b04a44b9ee53c51cb3109aa2ceaa44f1c45727e"
}, },
{ {
"name": "prior-inspection", "name": "prior-inspection",
"offset": "2696", "offset": "2744",
"byteLength": "10", "byteLength": "10",
"digest": "2c13b7b4d9a9916801ab9191c314f31b045e9b9c5b669a6c0474f0217ef75bf5" "digest": "2c13b7b4d9a9916801ab9191c314f31b045e9b9c5b669a6c0474f0217ef75bf5"
}, },
{ {
"name": "world", "name": "world",
"offset": "2712", "offset": "2760",
"byteLength": "64", "byteLength": "64",
"digest": "f5a5fd42d16a20302798ef6ed309979b43003d2320d9f0e8ea9831a92759fb4b" "digest": "f5a5fd42d16a20302798ef6ed309979b43003d2320d9f0e8ea9831a92759fb4b"
} }
], ],
"footerOffset": "2776", "footerOffset": "2824",
"footerBytes": 48, "footerBytes": 48,
"totalBytes": "2824" "totalBytes": "2872"
} }
}, },
"corruption": [ "corruption": [
@ -178,17 +182,17 @@
}, },
{ {
"name": "a flipped payload byte", "name": "a flipped payload byte",
"offset": 2640, "offset": 2688,
"reason": "every payload carries its own digest" "reason": "every payload carries its own digest"
}, },
{ {
"name": "a flipped footer digest byte", "name": "a flipped footer digest byte",
"offset": 2784, "offset": 2832,
"reason": "the footer digest must match the contents" "reason": "the footer digest must match the contents"
}, },
{ {
"name": "a flipped footer magic byte", "name": "a flipped footer magic byte",
"offset": 2816, "offset": 2864,
"reason": "a truncated file cannot look complete" "reason": "a truncated file cannot look complete"
} }
] ]

View file

@ -296,6 +296,11 @@ pub fn decode(bytes: &[u8]) -> Result<Envelope> {
/// The manifest fields state-media-v1 section 4 requires, checked as a set: a manifest that /// The manifest fields state-media-v1 section 4 requires, checked as a set: a manifest that
/// omits one of them is not a complete checkpoint. /// omits one of them is not a complete checkpoint.
///
/// `helperState` and `environment` join the list under the 2026-09-22 amendment to
/// checkpoint-envelope-v1 section 3: the first has been in that section's table from the
/// start and was missing here, and the second is the holder of the world's own payload, which
/// the table named for every other participant and not for the environment.
pub const REQUIRED_MANIFEST_FIELDS: &[&str] = &[ pub const REQUIRED_MANIFEST_FIELDS: &[&str] = &[
"envelopeVersion", "envelopeVersion",
"checkpointId", "checkpointId",
@ -308,6 +313,8 @@ pub const REQUIRED_MANIFEST_FIELDS: &[&str] = &[
"compatibility", "compatibility",
"agents", "agents",
"coordinator", "coordinator",
"environment",
"helperState",
"payloads", "payloads",
]; ];

View file

@ -44,6 +44,7 @@ Ready(k) ─ Prepare all agents concurrently ───────────
| `metrics` | Latency percentiles and the machine's core and memory counters | | `metrics` | Latency percentiles and the machine's core and memory counters |
| `measure` | The execution-mode comparison of the guide's section 5 | | `measure` | The execution-mode comparison of the guide's section 5 |
| `cli` | The binary's subcommands: `agent`, `environment`, `measure` | | `cli` | The binary's subcommands: `agent`, `environment`, `measure` |
| `state` | The durable checkpoint store over `FLYSESS1`: compatibility, generations, the bounded writer |
| `harness` | The runnable composition: router, the flies, one arena, one coordinator | | `harness` | The runnable composition: router, the flies, one arena, one coordinator |
## Execution modes and the launcher ## Execution modes and the launcher
@ -178,6 +179,44 @@ harness.shutdown().await;
event ids derived from epoch, source step, rule and ordinal. event ids derived from epoch, source step, rule and ordinal.
- **Executors.** The stateless identity executor only, as v1 specifies. - **Executors.** The stateless identity executor only, as v1 specifies.
## Checkpoints and recovery
The durable store is `state`, over the `FLYSESS1` layout the contract crate owns.
- **One boundary, every participant.** `Coordinator::capture` runs at `Ready(k)` or
`Paused(k)` only. It takes its queue slot *before* the first `State.Capture`, so a saturated
writer refuses the capture rather than queueing it without bound, and the refusal is a
`BUSY` a stepping session survives rather than an epoch failure.
- **Capture and durability are two events.** `State.Capture` completes when an immutable
capture exists; `Coordinator::await_durable` completes when the store manifest rename has
happened, which is the durable commit point. Only the second moves the durable mark, and the
three ways it can end without one are told apart: `Failed` (the write stopped),
`ReplyLost` (the write finished and the acknowledgment did not arrive) and
`DeadlineExpired` (the caller's own budget ran out while the save was still going).
`Coordinator::resolve_durable` then asks the store about the *same* checkpoint instead of
saving again.
- **The writer is bounded twice**, and the two bounds refuse at different moments. The
outstanding-capture bound is taken before a capture is requested; the byte budget cannot be,
because a capture's size is not known until it exists, so it refuses at submit and releases
the payloads with the refusal. The writer owns its payload handles until the bytes are
committed or the job fails. A queued *replaceable* capture is superseded by a later one,
releasing its holds; a durable one never is.
- **The install is a group.** A restore selects a complete compatible generation, imports its
payloads as fresh artifacts, stages every participant, validates the coordinator's own
ledgers, and only then activates. A failure anywhere leaves the fence closed, and every
participant that got as far as staging is recorded as one that must be replaced before
another restore is attempted.
- **The fence lifts once.** `Failed -> Restoring(k) -> Paused(k)`, at the end of a complete
install and nowhere else. A fenced session takes no step, publishes nothing, captures
nothing and holds no artifact handle.
- **Nothing old crosses.** The fence drops every media handle; the restore imports fresh
artifacts; the environment re-renders its pending sensor pipeline from recorded
reconstruction inputs; and the new epoch's first audio chunk resumes the preserved sample
position and marks the discontinuity.
- **Epoch metadata in a trace.** `scope.epoch`, the batch id and every task event id are
derived from the epoch, so a resumed run's behaviour is compared through
`EpochRebase`, which rewrites exactly those and fails on anything it does not recognise.
## Before you add a check to a reply ## Before you add a check to a reply
Ask which kind of reply it is. Is the far side **reporting what it did**, in which case a Ask which kind of reply it is. Is the far side **reporting what it did**, in which case a
@ -212,9 +251,9 @@ pointing the other way -- they are what make a partial commit and an incomplete
- **Fake workers.** There is no neural model and no emulator. What is modelled exactly is the - **Fake workers.** There is no neural model and no emulator. What is modelled exactly is the
ordering, the identity rules and the retry rules, not any numerical behaviour. ordering, the identity rules and the retry rules, not any numerical behaviour.
- **No state methods.** `State.Capture`, `State.StageRestore` and `State.ActivateRestore` are - **One environment, one task.** A checkpoint records the composition it was taken from, and a
STATE-01. The phase machine has their edges (`Capturing`, `Restoring`) and the workers do not restore refuses one taken under another backend, content, patch, controller or parser
advertise them as implemented methods. identity. It does not migrate between compositions, and it does not try.
- **No audience input.** The admitted pre-step stimulation list exists and is always empty. - **No audience input.** The admitted pre-step stimulation list exists and is always empty.
- **Pacing is coarse.** The pacing deadline rounds one step to whole nanoseconds for sleeping - **Pacing is coarse.** The pacing deadline rounds one step to whole nanoseconds for sleeping
only; simulation time stays rational and that rounding never re-enters the accumulator. only; simulation time stays rational and that rounding never re-enters the accumulator.
@ -275,6 +314,9 @@ The three integration suites do not all run over both transports, and cannot:
- `tests/processes.rs` runs over the Unix socket only, in all three execution modes. A - `tests/processes.rs` runs over the Unix socket only, in all three execution modes. A
participant in a process of its own has no in-memory transport to reach the router by, so participant in a process of its own has no in-memory transport to reach the router by, so
the mode is the axis that suite varies and the transport is fixed. the mode is the axis that suite varies and the transport is fixed.
- `tests/media.rs` and `tests/state.rs` run over both transports *and* in all three execution
modes: each acceptance body is written once and registered twice, by `both_transports!` in
the in-process composition and by `all_modes!` over the socket.
- `tests/session.rs`: one world advance per complete batch; every agent Prepared before the - `tests/session.rs`: one world advance per complete batch; every agent Prepared before the
advance; one task evaluation per transition; every agent committed before the next Prepare or advance; one task evaluation per transition; every agent committed before the next Prepare or
@ -291,6 +333,14 @@ The three integration suites do not all run over both transports, and cannot:
allocation -- plus the sequential/reversed/parallel trace comparison across all three modes allocation -- plus the sequential/reversed/parallel trace comparison across all three modes
and the two process-mode section 4 rows: a router restart during a world advance, and an old and the two process-mode section 4 rows: a router restart during a world advance, and an old
worker's reply after a restart. worker's reply after a restart.
- `tests/state.rs`: the STATE-01 acceptance bullets -- an uninterrupted run and a resumed run
committing the same behaviour once the epoch metadata is rebased, a corrupt payload failing
the install as a group for every participant and for the coordinator's own ledger, a lost
save reply and an uncommitted store manifest both leaving the durable mark where it was, a
refused activation resuming no part of the world, the capture queue staying bounded under a
stalled writer, and old media and another parser's state failing to cross a recovery --
plus the once-only restore token, the superseded replaceable capture, and the fence that
lifts only through a complete restore.
- `tests/failures.rs`: a duplicate Prepare after a lost reply; a duplicate Commit; the same - `tests/failures.rs`: a duplicate Prepare after a lost reply; a duplicate Commit; the same
batch with altered controls; a lost Advance result; a cached artifact consumed by its first batch with altered controls; a lost Advance result; a cached artifact consumed by its first
caller; one Commit failing after another succeeded; a replaced registration; a reply from caller; one Commit failing after another succeeded; a replaced registration; a reply from

View file

@ -7,7 +7,7 @@
//! stimulation, then reinforces once, and executes no tick at all. Every mutating step bumps //! stimulation, then reinforces once, and executes no tick at all. Every mutating step bumps
//! one counter, which is how a test proves a duplicate request changed nothing. //! one counter, which is how a test proves a duplicate request changed nothing.
use std::collections::BTreeMap; use std::collections::{BTreeMap, BTreeSet};
use serde_json::Value; use serde_json::Value;
@ -193,6 +193,12 @@ pub struct AgentFaults {
pub prepare_delay_ms: u64, pub prepare_delay_ms: u64,
/// Hold `Agent.Commit` open for this long. /// Hold `Agent.Commit` open for this long.
pub commit_delay_ms: u64, pub commit_delay_ms: u64,
/// Refuse `State.StageRestore`, so a group install meets one participant that will not
/// validate while the others already have.
pub fail_stage_restore: bool,
/// Refuse `State.ActivateRestore` after this worker has already staged, so a group meets
/// a failure halfway through activation.
pub fail_activate_restore: bool,
} }
/// One fake agent worker's configuration. /// One fake agent worker's configuration.
@ -234,6 +240,10 @@ pub struct FakeAgentWorker {
context: Option<TypedValue>, context: Option<TypedValue>,
context_digest: Option<Digest>, context_digest: Option<Digest>,
prepared: Option<(DomainRequestId, PreparedDecision)>, prepared: Option<(DomainRequestId, PreparedDecision)>,
/// A validated replacement state that the live session cannot see yet.
staged: Option<StagedAgent>,
/// Restore tokens this worker has activated. A token activates once.
activated: BTreeSet<Id>,
} }
impl FakeAgentWorker { impl FakeAgentWorker {
@ -248,10 +258,17 @@ impl FakeAgentWorker {
context: None, context: None,
context_digest: None, context_digest: None,
prepared: None, prepared: None,
staged: None,
activated: BTreeSet::new(),
config, config,
} }
} }
/// True while a validated replacement state is staged and not yet activated.
pub fn has_staged_restore(&self) -> bool {
self.staged.is_some()
}
pub fn status(&self) -> StatusCell { pub fn status(&self) -> StatusCell {
self.status.clone() self.status.clone()
} }
@ -657,7 +674,11 @@ impl WorkerEndpoint for FakeAgentWorker {
} }
fn capabilities(&self) -> Vec<Id> { fn capabilities(&self) -> Vec<Id> {
vec![id("agent-step-v1"), id("pixel-observation-v1")] vec![
id("agent-step-v1"),
id("pixel-observation-v1"),
id(crate::state::CHECKPOINT_CAPABILITY),
]
} }
fn status_cell(&self) -> StatusCell { fn status_cell(&self) -> StatusCell {
@ -669,7 +690,14 @@ impl WorkerEndpoint for FakeAgentWorker {
} }
fn methods(&self) -> Vec<&'static str> { fn methods(&self) -> Vec<&'static str> {
vec!["Agent.Initialize", "Agent.Prepare", "Agent.Commit"] vec![
"Agent.Initialize",
"Agent.Prepare",
"Agent.Commit",
"State.Capture",
"State.StageRestore",
"State.ActivateRestore",
]
} }
fn handle<'a>(&'a mut self, ctx: HandlerCtx<'a>) -> BoxFuture<'a, DomainResult<HandlerReply>> { fn handle<'a>(&'a mut self, ctx: HandlerCtx<'a>) -> BoxFuture<'a, DomainResult<HandlerReply>> {
@ -678,6 +706,9 @@ impl WorkerEndpoint for FakeAgentWorker {
"Agent.Initialize" => self.initialize(&ctx).await, "Agent.Initialize" => self.initialize(&ctx).await,
"Agent.Prepare" => self.prepare(&ctx).await, "Agent.Prepare" => self.prepare(&ctx).await,
"Agent.Commit" => self.commit(&ctx).await, "Agent.Commit" => self.commit(&ctx).await,
"State.Capture" => self.state_capture(&ctx).await,
"State.StageRestore" => self.state_stage_restore(&ctx).await,
"State.ActivateRestore" => self.state_activate_restore(&ctx).await,
other => Err(DomainError::before( other => Err(DomainError::before(
ErrorCode::Unsupported, ErrorCode::Unsupported,
format!("{other} is not an agent method"), format!("{other} is not an agent method"),
@ -690,7 +721,12 @@ impl WorkerEndpoint for FakeAgentWorker {
/// The retention class table an agent endpoint follows, for a caller that wants it. /// The retention class table an agent endpoint follows, for a caller that wants it.
pub fn agent_op_class(method: &str) -> Option<OpClass> { pub fn agent_op_class(method: &str) -> Option<OpClass> {
match method { match method {
"Agent.Initialize" => Some(OpClass::Lifecycle), // `ipc-v1` section 5: lifecycle *and capture* replies are retained until
// `Worker.Acknowledge`, which is also what lets a duplicate restore request replay
// its cached reply rather than staging or activating twice.
"Agent.Initialize" | "State.Capture" | "State.StageRestore" | "State.ActivateRestore" => {
Some(OpClass::Lifecycle)
}
"Agent.Prepare" | "Agent.Commit" => Some(OpClass::StepMutation), "Agent.Prepare" | "Agent.Commit" => Some(OpClass::StepMutation),
_ => None, _ => None,
} }
@ -712,3 +748,532 @@ pub fn synthetic_profile(agent_id: &Id, tick_duration: &RationalNs, warmup_ticks
/// The per-agent contexts a bootstrap produced, keyed by agent id. /// The per-agent contexts a bootstrap produced, keyed by agent id.
pub type Contexts = BTreeMap<Id, TypedValue>; pub type Contexts = BTreeMap<Id, TypedValue>;
// -------------------------------------------------------------------------------------------
// STATE-01: capture and restore
/// The numerical model version this worker implements. It is part of a capture's
/// compatibility identity: the same profile and seed under another model is not the same
/// state (`workers-v1` section 2).
pub const MODEL_VERSION: &str = "fake-lcg-v1";
/// The plasticity rule version, for the same reason.
pub const PLASTICITY_VERSION: &str = "fake-reinforce-v1";
/// The version this payload layout is written and read under.
pub const AGENT_PAYLOAD_VERSION: u64 = 1;
/// The dataset identity a synthetic agent resolves.
///
/// There is no connectome dataset behind this worker, and a checkpoint says so with a stable
/// identity rather than omitting the field: "no dataset" has to be distinguishable from "the
/// dataset was not recorded".
pub fn dataset_digest() -> Digest {
digest_of_bytes(b"fly-session/no-dataset-v1")
}
/// The capture compatibility digest of one agent (`workers-v1` section 2).
///
/// The profile digest identifies the profile definition; this additionally covers the
/// resolved seed, the numerical model version and the plasticity rule, because two agents
/// with the same profile digest and different seeds hold state that is not interchangeable.
/// Every field it covers is one the checkpoint manifest already records in that agent's row,
/// so a restore derives the expected digest from the manifest rather than from the payload it
/// is about to validate.
pub fn agent_compatibility_digest(
agent_id: &Id,
profile_digest: &Digest,
dataset_digest: &Digest,
model_version: &str,
plasticity_version: &str,
seed: i32,
) -> Digest {
let value = serde_json::json!({
"agentId": agent_id.as_str(),
"profileDigest": profile_digest.as_str(),
"datasetDigest": dataset_digest.as_str(),
"modelVersion": model_version,
"plasticityVersion": plasticity_version,
"seed": seed,
});
digest_of(&value).expect("an agent compatibility block canonicalizes")
}
impl FakeModel {
/// Every field of the model, so a resumed agent is this agent and not a fresh one.
fn capture(&self) -> Value {
serde_json::json!({
"seed": self.seed,
"state": self.state.to_string(),
"mutations": self.mutations.to_string(),
"ticks": self.ticks.to_string(),
"stimulations": self.stimulations.to_string(),
"reinforcements": self.reinforcements.to_string(),
"learningEnabled": self.learning_enabled,
"learningUpdates": self.learning_updates.to_string(),
"learningChanged": self.learning_changed.to_string(),
"lastSignal": self.last_signal,
"inputValue": self.input_value.to_string(),
"inputInstalls": self.input_installs.to_string(),
})
}
fn restored(value: &Value) -> DomainResult<FakeModel> {
let number = |key: &str| -> DomainResult<u64> {
value
.get(key)
.and_then(Value::as_str)
.ok_or_else(|| incompatible(format!("the agent payload has no {key}")))?
.parse::<u64>()
.map_err(|_| incompatible(format!("the agent payload's {key} is not a U64")))
};
let seed = value
.get("seed")
.and_then(Value::as_i64)
.and_then(|v| i32::try_from(v).ok())
.ok_or_else(|| incompatible("the agent payload has no seed"))?;
let input_value = value
.get("inputValue")
.and_then(Value::as_str)
.ok_or_else(|| incompatible("the agent payload has no inputValue"))?
.parse::<i64>()
.map_err(|_| incompatible("the agent payload's inputValue is not an integer"))?;
let last_signal = value
.get("lastSignal")
.and_then(Value::as_f64)
.filter(|v| v.is_finite())
.ok_or_else(|| incompatible("the agent payload's lastSignal is not finite"))?;
let learning_enabled = value
.get("learningEnabled")
.and_then(Value::as_bool)
.ok_or_else(|| incompatible("the agent payload has no learningEnabled"))?;
Ok(FakeModel {
seed,
state: number("state")?,
mutations: number("mutations")?,
ticks: number("ticks")?,
stimulations: number("stimulations")?,
reinforcements: number("reinforcements")?,
learning_enabled,
learning_updates: number("learningUpdates")?,
learning_changed: number("learningChanged")?,
last_signal,
input_value,
input_installs: number("inputInstalls")?,
})
}
}
fn incompatible(message: impl std::fmt::Display) -> DomainError {
DomainError::before(ErrorCode::IncompatibleState, message)
}
/// One staged restore, held outside the live agent until it is activated.
struct StagedAgent {
token: Id,
checkpoint_id: Id,
scope: Scope,
model: FakeModel,
accumulator: TickAccumulator,
context: TypedValue,
profile: AssetRef,
committed_step: u64,
}
impl FakeAgentWorker {
/// This worker's own compatibility identity, from its configuration and a resolved seed.
fn compatibility_digest(&self, profile: &AssetRef, seed: i32) -> Digest {
agent_compatibility_digest(
&self.config.agent_id,
&profile.digest,
&dataset_digest(),
MODEL_VERSION,
PLASTICITY_VERSION,
seed,
)
}
/// `State.Capture`: an immutable snapshot of this agent at its committed boundary.
///
/// It is allowed at `Ready(k)` only. A Prepared agent holds half a transition, and there
/// is no coherent boundary to file that under.
async fn state_capture(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult<HandlerReply> {
let scope = ctx.scope()?.clone();
self.check_epoch(&scope)?;
let AgentPhase::Ready(k) = self.phase.clone() else {
return Err(DomainError::before(
ErrorCode::InvalidPhase,
format!(
"State.Capture needs a quiescent Ready(k); this worker is {:?}",
self.phase
),
));
};
if scope.step != k {
return Err(DomainError::before(
if scope.step < k { ErrorCode::StaleStep } else { ErrorCode::FutureStep },
"State.Capture names a boundary this worker is not at",
));
}
let params: CaptureParams = ctx.params()?;
let profile = self.profile.clone().expect("initialized");
let context = self.context.clone().expect("initialized");
let accumulator = self.accumulator.as_ref().expect("initialized");
let previous = self.status.state();
self.status.set_state(WorkerState::Capturing);
let payload = serde_json::json!({
"payloadVersion": AGENT_PAYLOAD_VERSION,
"kind": "agent",
"agentId": self.config.agent_id.as_str(),
"checkpointId": params.checkpoint_id.as_str(),
"sourceScope": scope.to_json(),
"committedStep": k.to_string(),
"profile": profile.to_json(),
"modelVersion": MODEL_VERSION,
"plasticityVersion": PLASTICITY_VERSION,
"datasetDigest": dataset_digest().as_str(),
"model": self.model.capture(),
"accumulator": {
"tickDuration": accumulator.tick_duration().to_json(),
"remainder": accumulator.remainder().to_json(),
"executedTicks": accumulator.executed_ticks().to_string(),
"warmupOffset": accumulator.warmup_offset().to_string(),
},
"context": context.to_json(),
});
let bytes = canonicalize(&payload)
.map_err(|e| DomainError::invalid(format!("State.Capture: {}", e.0)))?
.into_bytes();
let digest = digest_of_bytes(&bytes);
let artifact = crate::state::seal_payload(ctx.client, &bytes, &digest).await?;
// Capture is a read of the model, not a mutation of it: nothing above changed a
// counter, and the worker goes back to the boundary it was already at.
self.status.set_state(previous);
let result = CaptureResult {
checkpoint_id: params.checkpoint_id,
boundary: k,
compatibility_digest: self.compatibility_digest(&profile, self.model.seed()),
payload: artifact.reference().clone(),
};
Ok(HandlerReply::with_artifacts(
object(result.to_json()),
vec![(crate::state::PAYLOAD_ATTACHMENT.to_owned(), artifact)],
))
}
/// `State.StageRestore`: validate a replacement state into a staging slot.
///
/// Nothing the live session can see changes here, and the worker keeps whatever state it
/// had. It is allowed on an uninitialized replacement or a quiescent worker only; a
/// failed one is neither, which is why a group that failed is replaced rather than
/// reused.
async fn state_stage_restore(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult<HandlerReply> {
let scope = ctx.scope()?.clone();
if scope.session_id != self.config.session_id {
return Err(DomainError::before(
ErrorCode::IdentityMismatch,
"this worker belongs to another session",
));
}
match &self.phase {
AgentPhase::Uninitialized | AgentPhase::Ready(_) => {}
other => {
return Err(DomainError::before(
ErrorCode::InvalidPhase,
format!(
"State.StageRestore needs an uninitialized replacement or a quiescent \
worker; this worker is {other:?}"
),
));
}
}
if let Some(epoch) = &self.epoch
&& *epoch == scope.epoch
{
return Err(DomainError::before(
ErrorCode::StaleEpoch,
"State.StageRestore proposes the epoch this worker is already running",
));
}
let params: StageRestoreParams = ctx.params()?;
if params.source_scope.step != scope.step {
return Err(DomainError::invalid(
"State.StageRestore's scope step must be the source boundary",
));
}
let artifact = ctx.artifact(crate::state::PAYLOAD_ATTACHMENT)?;
if artifact.reference() != &params.payload {
return Err(DomainError::before(
ErrorCode::BufferInvalid,
"the staged payload attachment is not the artifact the request names",
));
}
let bytes = artifact.read_all().await.map_err(|e| {
DomainError::before(
ErrorCode::BufferInvalid,
format!("the staged payload could not be read: {}", e.message),
)
})?;
let declared = params
.payload
.digest
.clone()
.ok_or_else(|| incompatible("a checkpoint payload must carry a content digest"))?;
let actual = digest_of_bytes(&bytes);
if actual != declared || bytes.len() as u64 != params.payload.byte_length {
return Err(incompatible(
"the staged payload is not the content the request declares",
));
}
let value: Value = serde_json::from_slice(&bytes)
.map_err(|e| DomainError::invalid(format!("the staged payload is not JSON: {e}")))?;
let text = |key: &str| -> DomainResult<String> {
value
.get(key)
.and_then(Value::as_str)
.map(str::to_owned)
.ok_or_else(|| incompatible(format!("the agent payload has no {key}")))
};
if value.get("payloadVersion").and_then(Value::as_u64) != Some(AGENT_PAYLOAD_VERSION) {
return Err(incompatible("the agent payload is another payload version"));
}
if text("kind")? != "agent" {
return Err(incompatible("this payload is not an agent's state"));
}
if text("agentId")? != self.config.agent_id {
return Err(DomainError::before(
ErrorCode::IdentityMismatch,
"the staged payload belongs to another agent",
));
}
if text("checkpointId")? != params.checkpoint_id {
return Err(incompatible("the staged payload belongs to another checkpoint"));
}
if text("modelVersion")? != MODEL_VERSION || text("plasticityVersion")? != PLASTICITY_VERSION
{
return Err(incompatible(
"the staged payload was captured under another numerical model",
));
}
let source_scope = Scope::from_json(
value
.get("sourceScope")
.ok_or_else(|| incompatible("the agent payload has no sourceScope"))?,
)
.map_err(|e| incompatible(format!("the agent payload's sourceScope: {}", e.0)))?;
if source_scope != params.source_scope {
return Err(incompatible(
"the staged payload was captured at another source scope",
));
}
let committed_step: u64 = text("committedStep")?
.parse()
.map_err(|_| incompatible("the agent payload's committedStep is not a U64"))?;
if committed_step != params.source_scope.step {
return Err(incompatible(
"the staged payload's committed step is not the source boundary",
));
}
let profile = AssetRef::from_json(
value
.get("profile")
.ok_or_else(|| incompatible("the agent payload has no profile"))?,
)
.map_err(|e| incompatible(format!("the agent payload's profile: {}", e.0)))?;
let model = FakeModel::restored(
value
.get("model")
.ok_or_else(|| incompatible("the agent payload has no model"))?,
)?;
// The compatibility digest is recomputed from this worker's own configuration and the
// identity the payload declares. A capture of the same profile under another seed, or
// of another agent's brain, fails here and never reaches activation.
let computed = self.compatibility_digest(&profile, model.seed());
if computed != params.compatibility_digest {
return Err(incompatible(format!(
"the staged state's compatibility {computed} is not the {} the restore \
requires",
params.compatibility_digest
)));
}
let accumulator_value = value
.get("accumulator")
.ok_or_else(|| incompatible("the agent payload has no accumulator"))?;
let rational = |key: &str| -> DomainResult<RationalNs> {
RationalNs::from_json(
accumulator_value
.get(key)
.ok_or_else(|| incompatible(format!("the accumulator has no {key}")))?,
)
.map_err(|e| incompatible(format!("the accumulator's {key}: {}", e.0)))
};
let counter = |key: &str| -> DomainResult<u64> {
accumulator_value
.get(key)
.and_then(Value::as_str)
.ok_or_else(|| incompatible(format!("the accumulator has no {key}")))?
.parse::<u64>()
.map_err(|_| incompatible(format!("the accumulator's {key} is not a U64")))
};
let tick_duration = rational("tickDuration")?;
if tick_duration != self.config.tick_duration {
return Err(incompatible(
"the staged state was captured at another model tick duration",
));
}
let accumulator = TickAccumulator::restored(
tick_duration,
rational("remainder")?,
counter("executedTicks")?,
counter("warmupOffset")?,
)
.map_err(incompatible)?;
let context = TypedValue::from_json(
value
.get("context")
.ok_or_else(|| incompatible("the agent payload has no context"))?,
)
.map_err(|e| incompatible(format!("the agent payload's context: {}", e.0)))?;
FakeAgentWorker::available_actions(&context)?;
if self.config.faults.fail_stage_restore {
// The row where a group validates three participants and the fourth does not.
// Nothing is staged here and nothing is staged anywhere else either: the
// coordinator abandons the whole install.
return Err(incompatible(
"injected staging refusal: this participant's replacement state does not \
validate",
));
}
// One staged restore at a time. A second proposal replaces nothing silently.
if let Some(staged) = &self.staged {
return Err(DomainError::before(
ErrorCode::Conflict,
format!(
"this worker already holds the staged restore {} for checkpoint {}",
staged.token, staged.checkpoint_id
),
));
}
let token = restore_token(&params.checkpoint_id, &scope, &actual, &self.config.incarnation_id);
if self.activated.contains(&token) {
return Err(DomainError::before(
ErrorCode::Conflict,
"this exact restore was already activated on this worker",
));
}
self.staged = Some(StagedAgent {
token: token.clone(),
checkpoint_id: params.checkpoint_id.clone(),
scope: scope.clone(),
model,
accumulator,
context,
profile,
committed_step,
});
self.status.set_state(WorkerState::StagedRestore);
let result = StageRestoreResult {
checkpoint_id: params.checkpoint_id,
restore_token: token,
};
Ok(HandlerReply::from(&result))
}
/// `State.ActivateRestore`: install the staged state under its new scope, without a tick.
///
/// The token activates once. A duplicate domain request replays the cached reply through
/// the shell's result cache; a fresh request naming an already activated token is a
/// conflict, which is what stops a second group from being resumed from the same bytes.
async fn state_activate_restore(
&mut self,
ctx: &HandlerCtx<'_>,
) -> DomainResult<HandlerReply> {
let params: ActivateRestoreParams = ctx.params()?;
if self.activated.contains(&params.restore_token) {
return Err(DomainError::before(
ErrorCode::Conflict,
"this restore token has already been activated",
));
}
let Some(staged) = self.staged.take() else {
return Err(DomainError::before(
ErrorCode::InvalidPhase,
"this worker holds no staged restore",
));
};
if staged.token != params.restore_token {
// Put it back: naming another token is not a reason to discard this one.
let token = staged.token.clone();
self.staged = Some(staged);
return Err(DomainError::before(
ErrorCode::IdentityMismatch,
format!("this worker's staged restore is {token}, not {}", params.restore_token),
));
}
if self.config.faults.fail_activate_restore {
let token = staged.token.clone();
self.staged = Some(staged);
self.status.set_state(WorkerState::Failed);
return Err(DomainError::new(
ErrorCode::BackendFailure,
format!("injected activation failure; {token} stays staged and unresumed"),
MutationCertainty::None,
));
}
self.status.set_state(WorkerState::Restoring);
let StagedAgent {
token,
checkpoint_id,
scope,
model,
accumulator,
context,
profile,
committed_step,
} = staged;
self.model = model;
self.accumulator = Some(accumulator);
self.context_digest = Some(context.digest());
self.context = Some(context);
self.profile = Some(profile);
self.epoch = Some(scope.epoch.clone());
self.prepared = None;
self.phase = AgentPhase::Ready(committed_step);
self.activated.insert(token);
self.status.set_state(WorkerState::Ready);
self.status.set_scope(Some(scope_at(
&scope.session_id,
&scope.epoch,
committed_step,
)));
self.status.advance_to(self.model.mutations());
let result = ActivateRestoreResult {
committed_step,
checkpoint_id,
// An agent returns a null observation; the environment returns the world's.
observation: None,
};
result
.validate_for_role(Role::Agent)
.map_err(|e| DomainError::invalid(e.0))?;
Ok(HandlerReply::from(&result))
}
}
/// A restore token bound to the checkpoint, the proposed scope, the payload bytes and the
/// worker incarnation staging them.
///
/// `state-media-v1` section 5 binds a token to scope, payload and checkpoint. Binding it to
/// the incarnation as well is what keeps a token minted by a worker that has since been
/// replaced from activating anything on its replacement.
pub fn restore_token(checkpoint_id: &Id, scope: &Scope, payload_digest: &Digest, incarnation: &Id) -> Id {
let digest = digest_of_bytes(
format!(
"fly-session/restore-token-v1\n{checkpoint_id}\n{}\n{}\n{}\n{payload_digest}\n{incarnation}\n",
scope.session_id, scope.epoch, scope.step
)
.as_bytes(),
);
parse_id(&format!("rt-{}", &digest[..32])).expect("a hex suffix is an Id")
}

View file

@ -46,8 +46,10 @@ Worker options (agent and environment):
agent: --agent ID --port ID --tick-numerator N --tick-denominator N agent: --agent ID --port ID --tick-numerator N --tick-denominator N
--warmup-ticks N [--prepare-delay-ms N] [--commit-delay-ms N] --warmup-ticks N [--prepare-delay-ms N] [--commit-delay-ms N]
[--fail-commit-at-step N] [--fail-commit-at-step N]
[--fail-stage-restore 0|1] [--fail-activate-restore 0|1]
environment: --worker ID --ports p1,p2 --step-numerator N --step-denominator N environment: --worker ID --ports p1,p2 --step-numerator N --step-denominator N
[--advance-delay-ms N] [--omit-view-at-boundary N] [--advance-delay-ms N] [--omit-view-at-boundary N]
[--fail-stage-restore 0|1] [--fail-activate-restore 0|1]
Measure options: Measure options:
--steps N transitions per run (default 200) --steps N transitions per run (default 200)
@ -145,6 +147,17 @@ impl Options {
} }
} }
/// A flag whose value is `0` or `1`. Anything else is an error naming it, so a
/// mistyped injection is a failed launch rather than a fault that never fires.
fn flag(&self, name: &str) -> Result<bool, String> {
match self.0.get(name) {
None => Ok(false),
Some(value) if value == "0" => Ok(false),
Some(value) if value == "1" => Ok(true),
Some(value) => Err(format!("--{name}: {value:?} is not 0 or 1")),
}
}
fn opt_u64(&self, name: &str) -> Result<Option<u64>, String> { fn opt_u64(&self, name: &str) -> Result<Option<u64>, String> {
match self.0.get(name) { match self.0.get(name) {
None => Ok(None), None => Ok(None),
@ -197,6 +210,8 @@ fn serve(role: &str, options: &Options) -> Result<(), String> {
fail_commit_at_step: options.opt_u64(flags::FAIL_COMMIT_AT_STEP)?, fail_commit_at_step: options.opt_u64(flags::FAIL_COMMIT_AT_STEP)?,
prepare_delay_ms: options.u64(flags::PREPARE_DELAY_MS, 0)?, prepare_delay_ms: options.u64(flags::PREPARE_DELAY_MS, 0)?,
commit_delay_ms: options.u64(flags::COMMIT_DELAY_MS, 0)?, commit_delay_ms: options.u64(flags::COMMIT_DELAY_MS, 0)?,
fail_stage_restore: options.flag(flags::FAIL_STAGE_RESTORE)?,
fail_activate_restore: options.flag(flags::FAIL_ACTIVATE_RESTORE)?,
}, },
client_id: client_id.clone(), client_id: client_id.clone(),
service: service.clone(), service: service.clone(),
@ -219,6 +234,8 @@ fn serve(role: &str, options: &Options) -> Result<(), String> {
omit_audio_at_boundary: options.opt_u64(flags::OMIT_AUDIO_AT_BOUNDARY)?, omit_audio_at_boundary: options.opt_u64(flags::OMIT_AUDIO_AT_BOUNDARY)?,
overlapping_audio_at_boundary: options overlapping_audio_at_boundary: options
.opt_u64(flags::OVERLAPPING_AUDIO_AT_BOUNDARY)?, .opt_u64(flags::OVERLAPPING_AUDIO_AT_BOUNDARY)?,
fail_stage_restore: options.flag(flags::FAIL_STAGE_RESTORE)?,
fail_activate_restore: options.flag(flags::FAIL_ACTIVATE_RESTORE)?,
}, },
client_id: client_id.clone(), client_id: client_id.clone(),
service: service.clone(), service: service.clone(),

View file

@ -29,6 +29,30 @@ impl TickAccumulator {
}) })
} }
/// The exact accumulator a capture recorded.
///
/// The remainder is restored, never rounded or reset: a resumed agent that started its
/// first interval from zero would drift away from the run it is supposed to continue.
pub fn restored(
tick_duration: RationalNs,
remainder: RationalNs,
executed_ticks: u64,
warmup_offset: u64,
) -> Result<TickAccumulator, String> {
let mut accumulator = TickAccumulator::new(tick_duration)?;
remainder.validate().map_err(|e| e.0)?;
if remainder >= tick_duration {
return Err("a captured remainder is not below one model tick".to_owned());
}
if warmup_offset > executed_ticks {
return Err("a captured warm-up offset exceeds the executed tick count".to_owned());
}
accumulator.remainder = remainder;
accumulator.executed_ticks = executed_ticks;
accumulator.warmup_offset = warmup_offset;
Ok(accumulator)
}
pub fn tick_duration(&self) -> RationalNs { pub fn tick_duration(&self) -> RationalNs {
self.tick_duration self.tick_duration
} }

File diff suppressed because it is too large Load diff

View file

@ -13,6 +13,8 @@
use std::collections::BTreeSet; use std::collections::BTreeSet;
use serde_json::Value;
use crate::media::{self, AudioSource, RenderCounter, ViewPipeline}; use crate::media::{self, AudioSource, RenderCounter, ViewPipeline};
use crate::task::{controller_schema_ref, inspection, inspection_schema}; use crate::task::{controller_schema_ref, inspection, inspection_schema};
// `crate::types` is this crate's facade over the shared `fly-session-types` crate; the // `crate::types` is this crate's facade over the shared `fly-session-types` crate; the
@ -48,6 +50,12 @@ pub struct EnvironmentFaults {
pub omit_audio_at_boundary: Option<u64>, pub omit_audio_at_boundary: Option<u64>,
/// Emit an audio chunk that starts before the previous chunk ended. /// Emit an audio chunk that starts before the previous chunk ended.
pub overlapping_audio_at_boundary: Option<u64>, pub overlapping_audio_at_boundary: Option<u64>,
/// Refuse `State.StageRestore`, so a group install meets a participant that will not
/// validate.
pub fail_stage_restore: bool,
/// Refuse `State.ActivateRestore` after staging, so a group meets a failure halfway
/// through activation.
pub fail_activate_restore: bool,
} }
#[derive(Clone, Debug)] #[derive(Clone, Debug)]
@ -86,6 +94,10 @@ pub struct CounterEnvironment {
audio: Option<AudioSource>, audio: Option<AudioSource>,
/// The frame served at the previous boundary, kept only so a fault can serve it again. /// The frame served at the previous boundary, kept only so a fault can serve it again.
previous_view: Option<(ViewRef, flybus::Artifact)>, previous_view: Option<(ViewRef, flybus::Artifact)>,
/// A validated replacement world the live session cannot see yet.
staged: Option<StagedWorld>,
/// Restore tokens this world has activated. A token activates once.
activated: BTreeSet<Id>,
} }
impl CounterEnvironment { impl CounterEnvironment {
@ -104,10 +116,17 @@ impl CounterEnvironment {
pipeline: None, pipeline: None,
audio: None, audio: None,
previous_view: None, previous_view: None,
staged: None,
activated: BTreeSet::new(),
config, config,
} }
} }
/// True while a validated replacement world is staged and not yet activated.
pub fn has_staged_restore(&self) -> bool {
self.staged.is_some()
}
pub fn status(&self) -> StatusCell { pub fn status(&self) -> StatusCell {
self.status.clone() self.status.clone()
} }
@ -477,7 +496,7 @@ impl WorkerEndpoint for CounterEnvironment {
vec![ vec![
id("world-step-v1"), id("world-step-v1"),
id("pixel-observation-v1"), id("pixel-observation-v1"),
id("checkpoint-v1"), id(crate::state::CHECKPOINT_CAPABILITY),
] ]
} }
@ -490,7 +509,13 @@ impl WorkerEndpoint for CounterEnvironment {
} }
fn methods(&self) -> Vec<&'static str> { fn methods(&self) -> Vec<&'static str> {
vec!["Environment.Initialize", "Environment.Advance"] vec![
"Environment.Initialize",
"Environment.Advance",
"State.Capture",
"State.StageRestore",
"State.ActivateRestore",
]
} }
fn handle<'a>(&'a mut self, ctx: HandlerCtx<'a>) -> BoxFuture<'a, DomainResult<HandlerReply>> { fn handle<'a>(&'a mut self, ctx: HandlerCtx<'a>) -> BoxFuture<'a, DomainResult<HandlerReply>> {
@ -498,6 +523,9 @@ impl WorkerEndpoint for CounterEnvironment {
match ctx.method { match ctx.method {
"Environment.Initialize" => self.initialize(&ctx).await, "Environment.Initialize" => self.initialize(&ctx).await,
"Environment.Advance" => self.advance(&ctx).await, "Environment.Advance" => self.advance(&ctx).await,
"State.Capture" => self.state_capture(&ctx).await,
"State.StageRestore" => self.state_stage_restore(&ctx).await,
"State.ActivateRestore" => self.state_activate_restore(&ctx).await,
other => Err(DomainError::before( other => Err(DomainError::before(
ErrorCode::Unsupported, ErrorCode::Unsupported,
format!("{other} is not an environment method"), format!("{other} is not an environment method"),
@ -516,3 +544,488 @@ pub fn synthetic_asset(asset_id: &str, body: &str) -> AssetRef {
format: id("fly-config-v1"), format: id("fly-config-v1"),
} }
} }
// -------------------------------------------------------------------------------------------
// STATE-01: capture and restore
/// The version this payload layout is written and read under.
pub const WORLD_PAYLOAD_VERSION: u64 = 1;
fn incompatible(message: impl std::fmt::Display) -> DomainError {
DomainError::before(ErrorCode::IncompatibleState, message)
}
/// One staged restore, held outside the live world until it is activated.
struct StagedWorld {
token: Id,
checkpoint_id: Id,
scope: Scope,
episode_id: Id,
descriptor: EnvironmentDescriptor,
boundary: u64,
counter: i64,
world_time: RationalNs,
advances: u64,
frames: Vec<(u64, i64)>,
audio_next_sample: u64,
audio_phase: u64,
audio_accumulator: u128,
audio_denominator: u128,
}
impl CounterEnvironment {
/// `State.Capture`: the world at its committed boundary, including its pending sensor
/// pipeline.
///
/// The pipeline is recorded as reconstruction inputs -- the producing boundary and the
/// world counter of every retained frame -- and never as an artifact identity: a
/// transient artifact belongs to the router that is running now, and a checkpoint outlives
/// it.
async fn state_capture(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult<HandlerReply> {
let scope = ctx.scope()?.clone();
let Some(descriptor) = self.descriptor.clone() else {
return Err(DomainError::before(
ErrorCode::InvalidPhase,
"this environment is uninitialized",
));
};
if scope.session_id != self.config.session_id {
return Err(DomainError::before(
ErrorCode::IdentityMismatch,
"this environment belongs to another session",
));
}
match &self.epoch {
Some(epoch) if *epoch == scope.epoch => {}
_ => {
return Err(DomainError::before(
ErrorCode::StaleEpoch,
"State.Capture names an epoch this environment has left",
));
}
}
if scope.step != self.boundary {
return Err(DomainError::before(
if scope.step < self.boundary {
ErrorCode::StaleStep
} else {
ErrorCode::FutureStep
},
"State.Capture must name the boundary the world is at",
));
}
let params: CaptureParams = ctx.params()?;
let pipeline = self
.pipeline
.as_ref()
.ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no view pipeline"))?;
let audio = self
.audio
.as_ref()
.ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no audio source"))?;
let (accumulator, denominator) = audio.accumulator();
let previous = self.status.state();
self.status.set_state(WorkerState::Capturing);
let payload = serde_json::json!({
"payloadVersion": WORLD_PAYLOAD_VERSION,
"kind": "world",
"workerId": self.config.worker_id.as_str(),
"checkpointId": params.checkpoint_id.as_str(),
"sourceScope": scope.to_json(),
"episodeId": self.episode_id.clone().expect("initialized").as_str(),
"committedStep": self.boundary.to_string(),
"counter": self.counter.to_string(),
"worldTime": self.world_time.to_json(),
"advances": self.advances.to_string(),
"descriptor": descriptor.to_json(),
"pipeline": {
// The declared delay's whole queue, oldest first.
"frames": pipeline
.retained()
.into_iter()
.map(|(boundary, counter)| serde_json::json!({
"boundary": boundary.to_string(),
"counter": counter.to_string(),
}))
.collect::<Vec<_>>(),
},
"audio": {
"nextSample": audio.next_sample().to_string(),
"phase": audio.phase().to_string(),
"accumulator": accumulator.to_string(),
"denominator": denominator.to_string(),
"chunks": audio.chunks().to_string(),
},
});
let bytes = canonicalize(&payload)
.map_err(|e| DomainError::invalid(format!("State.Capture: {}", e.0)))?
.into_bytes();
let digest = digest_of_bytes(&bytes);
let artifact = crate::state::seal_payload(ctx.client, &bytes, &digest).await?;
// A capture reads the world; it does not advance it.
self.status.set_state(previous);
let result = CaptureResult {
checkpoint_id: params.checkpoint_id,
boundary: self.boundary,
compatibility_digest: crate::state::Compatibility::of(&descriptor).digest(),
payload: artifact.reference().clone(),
};
Ok(HandlerReply::with_artifacts(
object(result.to_json()),
vec![(crate::state::PAYLOAD_ATTACHMENT.to_owned(), artifact)],
))
}
/// `State.StageRestore`: validate a replacement world into a staging slot.
async fn state_stage_restore(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult<HandlerReply> {
let scope = ctx.scope()?.clone();
if scope.session_id != self.config.session_id {
return Err(DomainError::before(
ErrorCode::IdentityMismatch,
"this environment belongs to another session",
));
}
if let Some(epoch) = &self.epoch
&& *epoch == scope.epoch
{
return Err(DomainError::before(
ErrorCode::StaleEpoch,
"State.StageRestore proposes the epoch this environment is already running",
));
}
if self.descriptor.is_some() {
// A world that is already running a boundary is not a quiescent replacement: the
// group replaces it rather than restoring over a live one.
return Err(DomainError::before(
ErrorCode::InvalidPhase,
"State.StageRestore needs an uninitialized replacement environment",
));
}
let params: StageRestoreParams = ctx.params()?;
if params.source_scope.step != scope.step {
return Err(DomainError::invalid(
"State.StageRestore's scope step must be the source boundary",
));
}
let artifact = ctx.artifact(crate::state::PAYLOAD_ATTACHMENT)?;
if artifact.reference() != &params.payload {
return Err(DomainError::before(
ErrorCode::BufferInvalid,
"the staged payload attachment is not the artifact the request names",
));
}
let bytes = artifact.read_all().await.map_err(|e| {
DomainError::before(
ErrorCode::BufferInvalid,
format!("the staged payload could not be read: {}", e.message),
)
})?;
let declared = params
.payload
.digest
.clone()
.ok_or_else(|| incompatible("a checkpoint payload must carry a content digest"))?;
let actual = digest_of_bytes(&bytes);
if actual != declared || bytes.len() as u64 != params.payload.byte_length {
return Err(incompatible(
"the staged payload is not the content the request declares",
));
}
let value: Value = serde_json::from_slice(&bytes)
.map_err(|e| DomainError::invalid(format!("the staged payload is not JSON: {e}")))?;
let text = |key: &str| -> DomainResult<String> {
value
.get(key)
.and_then(Value::as_str)
.map(str::to_owned)
.ok_or_else(|| incompatible(format!("the world payload has no {key}")))
};
let number = |key: &str| -> DomainResult<u64> {
text(key)?
.parse::<u64>()
.map_err(|_| incompatible(format!("the world payload's {key} is not a U64")))
};
if value.get("payloadVersion").and_then(Value::as_u64) != Some(WORLD_PAYLOAD_VERSION) {
return Err(incompatible("the world payload is another payload version"));
}
if text("kind")? != "world" {
return Err(incompatible("this payload is not a world's state"));
}
if text("workerId")? != self.config.worker_id {
return Err(DomainError::before(
ErrorCode::IdentityMismatch,
"the staged payload belongs to another world",
));
}
if text("checkpointId")? != params.checkpoint_id {
return Err(incompatible("the staged payload belongs to another checkpoint"));
}
let source_scope = Scope::from_json(
value
.get("sourceScope")
.ok_or_else(|| incompatible("the world payload has no sourceScope"))?,
)
.map_err(|e| incompatible(format!("the world payload's sourceScope: {}", e.0)))?;
if source_scope != params.source_scope {
return Err(incompatible(
"the staged payload was captured at another source scope",
));
}
let committed_step = number("committedStep")?;
if committed_step != params.source_scope.step {
return Err(incompatible(
"the staged payload's committed step is not the source boundary",
));
}
let descriptor = EnvironmentDescriptor::from_json(
value
.get("descriptor")
.ok_or_else(|| incompatible("the world payload has no descriptor"))?,
)
.map_err(|e| incompatible(format!("the world payload's descriptor: {}", e.0)))?;
// The replacement builds the descriptor it would advertise and compares. A world
// started with other ports, another cadence or another declared render delay is a
// different backend, not this one resumed.
let live = self.build_descriptor()?;
if descriptor != live {
return Err(incompatible(
"the staged world was captured under another environment descriptor",
));
}
let expected = crate::state::Compatibility::of(&descriptor).digest();
if expected != params.compatibility_digest {
return Err(incompatible(format!(
"the staged world's compatibility {expected} is not the {} the restore requires",
params.compatibility_digest
)));
}
let counter: i64 = text("counter")?
.parse()
.map_err(|_| incompatible("the world payload's counter is not an integer"))?;
let world_time = RationalNs::from_json(
value
.get("worldTime")
.ok_or_else(|| incompatible("the world payload has no worldTime"))?,
)
.map_err(|e| incompatible(format!("the world payload's worldTime: {}", e.0)))?;
let pipeline_value = value
.get("pipeline")
.and_then(|p| p.get("frames"))
.and_then(Value::as_array)
.ok_or_else(|| incompatible("the world payload has no pipeline frames"))?;
let mut frames = Vec::with_capacity(pipeline_value.len());
for frame in pipeline_value {
let boundary = frame
.get("boundary")
.and_then(Value::as_str)
.ok_or_else(|| incompatible("a captured frame has no boundary"))?
.parse::<u64>()
.map_err(|_| incompatible("a captured frame's boundary is not a U64"))?;
let frame_counter = frame
.get("counter")
.and_then(Value::as_str)
.ok_or_else(|| incompatible("a captured frame has no counter"))?
.parse::<i64>()
.map_err(|_| incompatible("a captured frame's counter is not an integer"))?;
frames.push((boundary, frame_counter));
}
match frames.last() {
Some((boundary, _)) if *boundary == committed_step => {}
_ => {
return Err(incompatible(
"the captured pipeline does not end at the committed boundary",
));
}
}
let audio_value = value
.get("audio")
.ok_or_else(|| incompatible("the world payload has no audio state"))?;
let audio_number = |key: &str| -> DomainResult<u128> {
audio_value
.get(key)
.and_then(Value::as_str)
.ok_or_else(|| incompatible(format!("the captured audio state has no {key}")))?
.parse::<u128>()
.map_err(|_| incompatible(format!("the captured audio {key} is not a number")))
};
let audio_next_sample = u64::try_from(audio_number("nextSample")?)
.map_err(|_| incompatible("the captured audio position is outside U64"))?;
let audio_phase = u64::try_from(audio_number("phase")?)
.map_err(|_| incompatible("the captured audio phase is outside U64"))?;
if self.config.faults.fail_stage_restore {
return Err(incompatible(
"injected staging refusal: this participant's replacement state does not \
validate",
));
}
if let Some(staged) = &self.staged {
return Err(DomainError::before(
ErrorCode::Conflict,
format!(
"this environment already holds the staged restore {} for checkpoint {}",
staged.token, staged.checkpoint_id
),
));
}
let token = crate::agent::restore_token(
&params.checkpoint_id,
&scope,
&actual,
&self.config.incarnation_id,
);
if self.activated.contains(&token) {
return Err(DomainError::before(
ErrorCode::Conflict,
"this exact restore was already activated on this environment",
));
}
self.staged = Some(StagedWorld {
token: token.clone(),
checkpoint_id: params.checkpoint_id.clone(),
scope,
episode_id: parse_id(&text("episodeId")?)
.map_err(|e| incompatible(format!("the world payload's episodeId {e}")))?,
descriptor,
boundary: committed_step,
counter,
world_time,
advances: number("advances")?,
frames,
audio_next_sample,
audio_phase,
audio_accumulator: audio_number("accumulator")?,
audio_denominator: audio_number("denominator")?,
});
self.status.set_state(WorkerState::StagedRestore);
let result = StageRestoreResult {
checkpoint_id: params.checkpoint_id,
restore_token: token,
};
Ok(HandlerReply::from(&result))
}
/// `State.ActivateRestore`: install the staged world and return its coherent observation.
///
/// Nothing advances. The pipeline's frames are rendered again into fresh artifacts of the
/// current store, which is what "the durable store imports fresh immutable bus artifacts"
/// means on the producing side, and the observation carries no audio chunk because no
/// interval was played.
async fn state_activate_restore(
&mut self,
ctx: &HandlerCtx<'_>,
) -> DomainResult<HandlerReply> {
let params: ActivateRestoreParams = ctx.params()?;
if self.activated.contains(&params.restore_token) {
return Err(DomainError::before(
ErrorCode::Conflict,
"this restore token has already been activated",
));
}
let Some(staged) = self.staged.take() else {
return Err(DomainError::before(
ErrorCode::InvalidPhase,
"this environment holds no staged restore",
));
};
if staged.token != params.restore_token {
let token = staged.token.clone();
self.staged = Some(staged);
return Err(DomainError::before(
ErrorCode::IdentityMismatch,
format!(
"this environment's staged restore is {token}, not {}",
params.restore_token
),
));
}
if self.config.faults.fail_activate_restore {
let token = staged.token.clone();
self.staged = Some(staged);
self.status.set_state(WorkerState::Failed);
return Err(DomainError::new(
ErrorCode::BackendFailure,
format!("injected activation failure; {token} stays staged and unresumed"),
MutationCertainty::None,
));
}
self.status.set_state(WorkerState::Restoring);
let mut pipeline = ViewPipeline::new(
CounterEnvironment::view_descriptor(self.config.observation_delay_steps),
self.config.renders.clone(),
);
pipeline.restore(ctx.client, &staged.frames).await?;
let audio = AudioSource::restored_from(
CounterEnvironment::audio_descriptor(),
staged.audio_next_sample,
staged.audio_phase,
staged.audio_accumulator,
staged.audio_denominator,
)?;
self.epoch = Some(staged.scope.epoch.clone());
self.episode_id = Some(staged.episode_id.clone());
self.descriptor = Some(staged.descriptor.clone());
self.boundary = staged.boundary;
self.counter = staged.counter;
self.world_time = staged.world_time;
self.advances = staged.advances;
// Batch ids are unique within an epoch, and this is a new one. Keeping the old set
// would refuse nothing extra: a request under the old epoch is already refused by its
// scope.
self.batches.clear();
self.pipeline = Some(pipeline);
self.audio = Some(audio);
self.previous_view = None;
self.activated.insert(staged.token);
self.status.set_state(WorkerState::Ready);
self.status.set_scope(Some(scope_at(
&staged.scope.session_id,
&staged.scope.epoch,
staged.boundary,
)));
let (observation, attachments) = self.restored_observation()?;
let result = ActivateRestoreResult {
committed_step: staged.boundary,
checkpoint_id: staged.checkpoint_id,
observation: Some(observation),
};
result
.validate_for_role(Role::Environment)
.map_err(|e| DomainError::invalid(e.0))?;
let mut reply = HandlerReply::from(&result);
reply.artifacts = attachments;
Ok(reply)
}
/// The observation the restored world is already at: no render, no advance, no audio.
fn restored_observation(
&mut self,
) -> DomainResult<(WorldObservation, Vec<(String, flybus::Artifact)>)> {
let boundary = self.boundary;
let counter = self.counter;
let pipeline = self
.pipeline
.as_ref()
.ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no view pipeline"))?;
let (view, artifact) = pipeline.at(boundary).ok_or_else(|| {
incompatible("the restored pipeline holds no frame for the restored boundary")
})?;
self.previous_view = Some((view.clone(), artifact.clone()));
let observation = WorldObservation {
boundary,
world_time: self.world_time,
engine_frame: Some(boundary.to_string()),
sensory_views: vec![view.clone()],
inspection: inspection(counter, boundary),
broadcast_views: vec![view.clone()],
// No interval was played, so there is no chunk. A chunk here would be an old
// epoch's audio offered as current.
audio: Vec::new(),
};
Ok((
observation,
vec![(media::view_attachment(&view.view_id), artifact)],
))
}
}

View file

@ -26,6 +26,7 @@ use crate::media::{RenderCounter, SensorLog};
use crate::launcher::{ use crate::launcher::{
AgentLaunch, EnvironmentLaunch, Launcher, ReapOutcome, SUPERVISOR_CLIENT, ThreadBudget, AgentLaunch, EnvironmentLaunch, Launcher, ReapOutcome, SUPERVISOR_CLIENT, ThreadBudget,
}; };
use crate::state::{CheckpointStore, CheckpointWriter, StoreConfig, StoreFaults, WriterConfig, WriterFaults};
use crate::task::{ActionExecutor, CounterTask, IdentityExecutor, Terminal}; use crate::task::{ActionExecutor, CounterTask, IdentityExecutor, Terminal};
// `crate::types` is this crate's facade over the shared `fly-session-types` crate; the // `crate::types` is this crate's facade over the shared `fly-session-types` crate; the
// glob keeps the contract's own names in sight instead of restating them. // glob keeps the contract's own names in sight instead of restating them.
@ -85,6 +86,14 @@ pub struct HarnessConfig {
/// The threads reserved for the coordinator, its router and its store. /// The threads reserved for the coordinator, its router and its store.
pub coordinator_threads: usize, pub coordinator_threads: usize,
pub environment_threads: usize, pub environment_threads: usize,
/// How many committed generations the durable checkpoint store keeps.
pub store: StoreConfig,
/// The durable write faults this composition injects.
pub store_faults: StoreFaults,
/// The checkpoint queue's bounds.
pub writer: WriterConfig,
/// The writer faults this composition injects.
pub writer_faults: WriterFaults,
} }
impl Default for HarnessConfig { impl Default for HarnessConfig {
@ -107,6 +116,10 @@ impl Default for HarnessConfig {
thread_budget: None, thread_budget: None,
coordinator_threads: 1, coordinator_threads: 1,
environment_threads: 1, environment_threads: 1,
store: StoreConfig::default(),
store_faults: StoreFaults::default(),
writer: WriterConfig::default(),
writer_faults: WriterFaults::default(),
} }
} }
} }
@ -134,6 +147,14 @@ const ENV_SERVICE: &str = "env.arena";
const ENV_CLIENT: &str = "environment"; const ENV_CLIENT: &str = "environment";
const ENV_WORKER: &str = "arena"; const ENV_WORKER: &str = "arena";
const COORDINATOR_CLIENT: &str = "coordinator"; const COORDINATOR_CLIENT: &str = "coordinator";
/// The checkpoint writer's own bus identity. It publishes checkpoint events and nothing else.
const WRITER_CLIENT: &str = "checkpoint-writer";
/// How many times one participant may be replaced in a composition.
///
/// Each replacement connects under its own client id, so a restart is visibly a new
/// participant rather than a silent reattachment, and the policy has to name them all.
const MAX_GENERATIONS: u32 = 8;
fn agent_service(agent_id: &Id) -> String { fn agent_service(agent_id: &Id) -> String {
format!("agent.{agent_id}") format!("agent.{agent_id}")
@ -172,6 +193,10 @@ pub struct SessionHarness {
/// The supervisor. It owns every participant's lifetime and thread allocation. /// The supervisor. It owns every participant's lifetime and thread allocation.
pub launcher: Launcher, pub launcher: Launcher,
observers: Mutex<Vec<Client>>, observers: Mutex<Vec<Client>>,
/// Which generation of each participant is running: 1 is the one the composition started.
generations: BTreeMap<Id, u32>,
/// Where the durable checkpoint store lives, for a test that reads the files themselves.
checkpoint_root: std::path::PathBuf,
} }
impl SessionHarness { impl SessionHarness {
@ -204,12 +229,16 @@ impl SessionHarness {
g.call = vec![Pattern::prefix("agent."), Pattern::prefix("env.")]; g.call = vec![Pattern::prefix("agent."), Pattern::prefix("env.")];
}), }),
) )
// The writer publishes the checkpoint events and never calls a participant.
.client(WRITER_CLIENT, grants(|g| g.publish = vec![Pattern::prefix("session.")]))
.client(ENV_CLIENT, grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)])) .client(ENV_CLIENT, grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]))
.client(
&format!("{ENV_CLIENT}-r2"),
grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]),
)
.client("observer", grants(|g| g.subscribe = vec![Pattern::prefix("session.")])); .client("observer", grants(|g| g.subscribe = vec![Pattern::prefix("session.")]));
for generation in 2..=MAX_GENERATIONS {
policy = policy.client(
&format!("{ENV_CLIENT}-r{generation}"),
grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]),
);
}
for spec in &config.agents { for spec in &config.agents {
let service = agent_service(&spec.agent_id); let service = agent_service(&spec.agent_id);
policy = policy.client( policy = policy.client(
@ -218,10 +247,12 @@ impl SessionHarness {
); );
// A replacement worker connects under its own client id, so a restart is visibly a // A replacement worker connects under its own client id, so a restart is visibly a
// new participant rather than a silent reattachment to the active epoch. // new participant rather than a silent reattachment to the active epoch.
policy = policy.client( for generation in 2..=MAX_GENERATIONS {
&format!("{}-r2", agent_client(&spec.agent_id)), policy = policy.client(
grants(|g| g.register = vec![Pattern::exact(&service)]), &format!("{}-r{generation}", agent_client(&spec.agent_id)),
); grants(|g| g.register = vec![Pattern::exact(&service)]),
);
}
} }
let mut router_config = RouterConfig::new(&store_root); let mut router_config = RouterConfig::new(&store_root);
router_config.policy = policy; router_config.policy = policy;
@ -299,6 +330,21 @@ impl SessionHarness {
} }
let coordinator_client = launcher.connect(COORDINATOR_CLIENT).await?; let coordinator_client = launcher.connect(COORDINATOR_CLIENT).await?;
// The durable store lives beside the router's artifact store and never inside it: a
// committed generation is outside the bus's ephemeral collection.
let checkpoint_root = root.join("checkpoints");
let mut store = CheckpointStore::open(&checkpoint_root, config.store).map_err(refusal)?;
*store.faults_mut() = config.store_faults.clone();
let writer_client = launcher.connect(WRITER_CLIENT).await?;
let writer = CheckpointWriter::start(
store,
config.writer,
config.writer_faults.clone(),
Some((
writer_client,
format!("session.{}.checkpoints", config.session_id),
)),
);
let executors: BTreeMap<Id, Box<dyn ActionExecutor>> = config let executors: BTreeMap<Id, Box<dyn ActionExecutor>> = config
.agents .agents
.iter() .iter()
@ -316,6 +362,8 @@ impl SessionHarness {
Box::new(CounterTask::new(&config.epoch, config.terminal)), Box::new(CounterTask::new(&config.epoch, config.terminal)),
executors, executors,
); );
let mut coordinator = coordinator;
coordinator.attach_store(writer);
Ok(SessionHarness { Ok(SessionHarness {
coordinator, coordinator,
@ -326,9 +374,16 @@ impl SessionHarness {
sensors, sensors,
launcher, launcher,
observers: Mutex::new(Vec::new()), observers: Mutex::new(Vec::new()),
generations: BTreeMap::new(),
checkpoint_root,
}) })
} }
/// Where the durable checkpoint store's generations and store manifest live.
pub fn checkpoint_root(&self) -> &std::path::Path {
&self.checkpoint_root
}
pub fn router(&self) -> &Router { pub fn router(&self) -> &Router {
self.launcher.router() self.launcher.router()
} }
@ -373,10 +428,11 @@ impl SessionHarness {
.find(|spec| spec.agent_id == *agent_id) .find(|spec| spec.agent_id == *agent_id)
.expect("a configured agent") .expect("a configured agent")
.clone(); .clone();
let generation = self.next_generation(agent_id)?;
self.launcher.kill(agent_id).await; self.launcher.kill(agent_id).await;
let tick_duration = millis(self.config.tick_ms).expect("a positive tick"); let tick_duration = millis(self.config.tick_ms).expect("a positive tick");
let incarnation_id = let incarnation_id = parse_id(&format!("{agent_id}-inc-{generation}"))
parse_id(&format!("{agent_id}-inc-2")).expect("an agent id plus a suffix is an Id"); .expect("an agent id plus a suffix is an Id");
self.launcher self.launcher
.launch_agent(AgentLaunch { .launch_agent(AgentLaunch {
session_id: self.config.session_id.clone(), session_id: self.config.session_id.clone(),
@ -390,7 +446,7 @@ impl SessionHarness {
// predecessor wrote, so a restore's sensory input is visible beside it. // predecessor wrote, so a restore's sensory input is visible beside it.
sensors: self.sensors.get(agent_id).cloned().unwrap_or_default(), sensors: self.sensors.get(agent_id).cloned().unwrap_or_default(),
faults: spec.faults.clone(), faults: spec.faults.clone(),
client_id: format!("{}-r2", agent_client(agent_id)), client_id: format!("{}-r{generation}", agent_client(agent_id)),
service: agent_service(agent_id), service: agent_service(agent_id),
}) })
.await .await
@ -403,6 +459,102 @@ impl SessionHarness {
}) })
} }
/// Replaces the environment with a fresh, uninitialized incarnation, as a restore needs.
pub async fn restart_environment(&mut self) -> Result<Restarted, flybus::BusError> {
let worker_id = id(ENV_WORKER);
let generation = self.next_generation(&worker_id)?;
self.launcher.kill(&worker_id).await;
let step_duration = hz(self.config.step_hz).expect("a positive cadence");
let incarnation_id = parse_id(&format!("arena-inc-{generation}"))
.expect("a worker id plus a suffix is an Id");
self.launcher
.launch_environment(EnvironmentLaunch {
session_id: self.config.session_id.clone(),
worker_id: worker_id.clone(),
incarnation_id: incarnation_id.clone(),
step_duration,
ports: self.config.agents.iter().map(|a| a.port_id.clone()).collect(),
worker_threads: self.config.environment_threads,
observation_delay_steps: self.config.observation_delay_steps,
renders: self.renders.clone(),
faults: self.config.environment_faults.clone(),
client_id: format!("{ENV_CLIENT}-r{generation}"),
service: ENV_SERVICE.to_owned(),
})
.await
.map_err(refusal)?;
let worker = self.launcher.worker(&worker_id).expect("just launched");
Ok(Restarted {
service: worker.identity.service.clone(),
service_incarnation: worker.service_incarnation.clone(),
incarnation_id,
})
}
fn next_generation(&mut self, worker_id: &Id) -> Result<u32, flybus::BusError> {
let slot = self.generations.entry(worker_id.clone()).or_insert(1);
if *slot >= MAX_GENERATIONS {
return Err(flybus::BusError::new(
flybus::ErrorCode::QuotaExceeded,
format!(
"{worker_id} has used all {MAX_GENERATIONS} configured client identities; a composition declares how many replacements it allows"
),
));
}
*slot += 1;
Ok(*slot)
}
/// Replaces every participant and points the fenced coordinator at the replacements.
///
/// This is what a recovery does before it restores: the old participants belong to an
/// invalid epoch, and the references the coordinator pinned are exchanged deliberately.
pub async fn replace_all_participants(&mut self) -> Result<(), flybus::BusError> {
let environment = self.environment_id();
self.restart_environment().await?;
let worker = self
.launcher
.worker(&environment)
.expect("just launched")
.worker_ref();
self.coordinator
.replace_participant(&environment, worker)
.map_err(|e| refusal(e.error))?;
for agent_id in self.config.agents.iter().map(|a| a.agent_id.clone()).collect::<Vec<_>>() {
self.restart_agent(&agent_id).await?;
let worker = self
.launcher
.worker(&agent_id)
.expect("just launched")
.worker_ref();
self.coordinator
.replace_participant(&agent_id, worker)
.map_err(|e| refusal(e.error))?;
}
Ok(())
}
/// Changes one agent's injected faults, so the replacement the next restart launches is
/// a participant without them.
///
/// A fault is launch configuration, so clearing one is a relaunch and not a live change:
/// the worker running now keeps whatever it was started with.
pub fn set_agent_faults(&mut self, agent_id: &Id, faults: AgentFaults) {
if let Some(spec) = self
.config
.agents
.iter_mut()
.find(|spec| spec.agent_id == *agent_id)
{
spec.faults = faults;
}
}
/// Changes the environment's injected faults, with the same relaunch rule.
pub fn set_environment_faults(&mut self, faults: EnvironmentFaults) {
self.config.environment_faults = faults;
}
/// Ends one participant without asking it, as a crash would. /// Ends one participant without asking it, as a crash would.
pub async fn kill(&mut self, worker_id: &Id) -> ReapOutcome { pub async fn kill(&mut self, worker_id: &Id) -> ReapOutcome {
self.launcher.kill(worker_id).await self.launcher.kill(worker_id).await
@ -466,7 +618,10 @@ impl SessionHarness {
/// Reaps every participant and closes the router. /// Reaps every participant and closes the router.
pub async fn shutdown(self) { pub async fn shutdown(self) {
let SessionHarness { coordinator, mut launcher, observers, .. } = self; let SessionHarness { mut coordinator, mut launcher, observers, .. } = self;
// The writer task owns artifact handles and a blocking store. Leaving it running
// would leave both behind.
coordinator.shutdown_store().await;
drop(coordinator); drop(coordinator);
launcher.reap_all(&id("shutdown")).await; launcher.reap_all(&id("shutdown")).await;
for observer in observers.into_inner().expect("not poisoned") { for observer in observers.into_inner().expect("not poisoned") {

View file

@ -1231,6 +1231,8 @@ pub(crate) mod flags {
pub const PREPARE_DELAY_MS: &str = "prepare-delay-ms"; pub const PREPARE_DELAY_MS: &str = "prepare-delay-ms";
pub const COMMIT_DELAY_MS: &str = "commit-delay-ms"; pub const COMMIT_DELAY_MS: &str = "commit-delay-ms";
pub const FAIL_COMMIT_AT_STEP: &str = "fail-commit-at-step"; pub const FAIL_COMMIT_AT_STEP: &str = "fail-commit-at-step";
pub const FAIL_STAGE_RESTORE: &str = "fail-stage-restore";
pub const FAIL_ACTIVATE_RESTORE: &str = "fail-activate-restore";
pub const WORKER: &str = "worker"; pub const WORKER: &str = "worker";
pub const PORTS: &str = "ports"; pub const PORTS: &str = "ports";
@ -1271,6 +1273,8 @@ pub(crate) mod flags {
PREPARE_DELAY_MS, PREPARE_DELAY_MS,
COMMIT_DELAY_MS, COMMIT_DELAY_MS,
FAIL_COMMIT_AT_STEP, FAIL_COMMIT_AT_STEP,
FAIL_STAGE_RESTORE,
FAIL_ACTIVATE_RESTORE,
]; ];
/// What only the environment is given, media options included. /// What only the environment is given, media options included.
pub const ENVIRONMENT_ONLY: &[&str] = &[ pub const ENVIRONMENT_ONLY: &[&str] = &[
@ -1285,6 +1289,8 @@ pub(crate) mod flags {
TRUNCATED_VIEW_AT_BOUNDARY, TRUNCATED_VIEW_AT_BOUNDARY,
OMIT_AUDIO_AT_BOUNDARY, OMIT_AUDIO_AT_BOUNDARY,
OVERLAPPING_AUDIO_AT_BOUNDARY, OVERLAPPING_AUDIO_AT_BOUNDARY,
FAIL_STAGE_RESTORE,
FAIL_ACTIVATE_RESTORE,
]; ];
/// What a measurement run or one of its row children is given. /// What a measurement run or one of its row children is given.
pub const MEASURE: &[&str] = &[MODE, AGENTS, STEPS, WARMUP_STEPS, WORKER_THREADS, MODES]; pub const MEASURE: &[&str] = &[MODE, AGENTS, STEPS, WARMUP_STEPS, WORKER_THREADS, MODES];
@ -1327,6 +1333,11 @@ impl Started {
arg(flags::WARMUP_TICKS, spec.warmup_ticks), arg(flags::WARMUP_TICKS, spec.warmup_ticks),
arg(flags::PREPARE_DELAY_MS, spec.faults.prepare_delay_ms), arg(flags::PREPARE_DELAY_MS, spec.faults.prepare_delay_ms),
arg(flags::COMMIT_DELAY_MS, spec.faults.commit_delay_ms), arg(flags::COMMIT_DELAY_MS, spec.faults.commit_delay_ms),
arg(flags::FAIL_STAGE_RESTORE, u64::from(spec.faults.fail_stage_restore)),
arg(
flags::FAIL_ACTIVATE_RESTORE,
u64::from(spec.faults.fail_activate_restore),
),
]; ];
if let Some(step) = spec.faults.fail_commit_at_step { if let Some(step) = spec.faults.fail_commit_at_step {
args.push(arg(flags::FAIL_COMMIT_AT_STEP, step)); args.push(arg(flags::FAIL_COMMIT_AT_STEP, step));
@ -1345,6 +1356,11 @@ impl Started {
// The media options a world in another process needs to be exactly this // The media options a world in another process needs to be exactly this
// world. Its render counter and its agents' sensor logs stay there. // world. Its render counter and its agents' sensor logs stay there.
arg(flags::OBSERVATION_DELAY_STEPS, spec.observation_delay_steps), arg(flags::OBSERVATION_DELAY_STEPS, spec.observation_delay_steps),
arg(flags::FAIL_STAGE_RESTORE, u64::from(spec.faults.fail_stage_restore)),
arg(
flags::FAIL_ACTIVATE_RESTORE,
u64::from(spec.faults.fail_activate_restore),
),
]; ];
for (flag, boundary) in [ for (flag, boundary) in [
(flags::OMIT_VIEW_AT_BOUNDARY, spec.faults.omit_view_at_boundary), (flags::OMIT_VIEW_AT_BOUNDARY, spec.faults.omit_view_at_boundary),
@ -1458,6 +1474,8 @@ mod flag_tests {
truncated_view_at_boundary: Some(3), truncated_view_at_boundary: Some(3),
omit_audio_at_boundary: Some(4), omit_audio_at_boundary: Some(4),
overlapping_audio_at_boundary: Some(5), overlapping_audio_at_boundary: Some(5),
fail_stage_restore: true,
fail_activate_restore: true,
} }
} }
@ -1491,6 +1509,8 @@ mod flag_tests {
fail_commit_at_step: Some(2), fail_commit_at_step: Some(2),
prepare_delay_ms: 1, prepare_delay_ms: 1,
commit_delay_ms: 2, commit_delay_ms: 2,
fail_stage_restore: true,
fail_activate_restore: true,
}, },
client_id: "worker-fly-a".to_owned(), client_id: "worker-fly-a".to_owned(),
service: "agent.fly-a".to_owned(), service: "agent.fly-a".to_owned(),
@ -1535,6 +1555,8 @@ mod flag_tests {
flags::TRUNCATED_VIEW_AT_BOUNDARY, flags::TRUNCATED_VIEW_AT_BOUNDARY,
flags::OMIT_AUDIO_AT_BOUNDARY, flags::OMIT_AUDIO_AT_BOUNDARY,
flags::OVERLAPPING_AUDIO_AT_BOUNDARY, flags::OVERLAPPING_AUDIO_AT_BOUNDARY,
flags::FAIL_STAGE_RESTORE,
flags::FAIL_ACTIVATE_RESTORE,
] { ] {
assert!(written.contains(&format!("--{flag}")), "--{flag} is not written"); assert!(written.contains(&format!("--{flag}")), "--{flag} is not written");
} }

View file

@ -34,6 +34,7 @@ pub mod media;
pub mod metrics; pub mod metrics;
pub mod phase; pub mod phase;
pub mod rpc; pub mod rpc;
pub mod state;
pub mod task; pub mod task;
pub mod worker; pub mod worker;

View file

@ -133,7 +133,14 @@ pub fn arena_frame(descriptor: &ViewDescriptor, counter: i64, boundary: u64) ->
/// cannot be served an arbitrary stale image. /// cannot be served an arbitrary stale image.
pub struct ViewPipeline { pub struct ViewPipeline {
descriptor: ViewDescriptor, descriptor: ViewDescriptor,
frames: VecDeque<(u64, flybus::Artifact)>, /// Each retained frame: its producing boundary, the world counter it was rendered from
/// and the owned handle on its immutable bytes.
///
/// The counter is kept because it is the whole of the reconstruction input: a checkpoint
/// records `(boundary, counter)` per retained frame and a restore re-renders them into
/// fresh artifacts of the current store, rather than persisting a transient artifact
/// identity that cannot survive a router restart.
frames: VecDeque<(u64, i64, flybus::Artifact)>,
renders: RenderCounter, renders: RenderCounter,
} }
@ -160,7 +167,7 @@ impl ViewPipeline {
let bytes = arena_frame(&self.descriptor, counter, boundary); let bytes = arena_frame(&self.descriptor, counter, boundary);
let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?; let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?;
self.renders.bump(); self.renders.bump();
self.frames.push_back((boundary, artifact)); self.frames.push_back((boundary, counter, artifact));
// Keep exactly the frames a declared delay can still require. // Keep exactly the frames a declared delay can still require.
while self.frames.len() > self.descriptor.observation_delay_steps as usize + 1 { while self.frames.len() > self.descriptor.observation_delay_steps as usize + 1 {
self.frames.pop_front(); self.frames.pop_front();
@ -168,6 +175,50 @@ impl ViewPipeline {
Ok(()) Ok(())
} }
/// The reconstruction inputs of every retained frame, oldest first.
///
/// This is what a checkpoint records for the pending sensor pipeline: the producing
/// boundary and the world counter, never an artifact identity.
pub fn retained(&self) -> Vec<(u64, i64)> {
self.frames
.iter()
.map(|(boundary, counter, _)| (*boundary, *counter))
.collect()
}
/// Rebuilds the pipeline from recorded reconstruction inputs, into fresh artifacts.
///
/// Every frame is rendered again in the current store, so nothing a fence dropped is
/// expected to come back and no old artifact identity crosses the recovery.
pub async fn restore(
&mut self,
client: &flybus::Client,
frames: &[(u64, i64)],
) -> DomainResult<()> {
if frames.len() > self.descriptor.observation_delay_steps as usize + 1 {
return Err(media_error(format!(
"a captured pipeline of {} frames does not fit a declared delay of {}",
frames.len(),
self.descriptor.observation_delay_steps
)));
}
for window in frames.windows(2) {
if window[1].0 != window[0].0 + 1 {
return Err(media_error(
"a captured pipeline's producing boundaries are not consecutive",
));
}
}
self.frames.clear();
for (boundary, counter) in frames {
let bytes = arena_frame(&self.descriptor, *counter, *boundary);
let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?;
self.renders.bump();
self.frames.push_back((*boundary, *counter, artifact));
}
Ok(())
}
/// Seals a frame of the wrong length, which is what a broken backend produces. The /// Seals a frame of the wrong length, which is what a broken backend produces. The
/// reference it returns describes the artifact honestly, so the shape check is the thing /// reference it returns describes the artifact honestly, so the shape check is the thing
/// under test rather than a lie in the payload. /// under test rather than a lie in the payload.
@ -181,7 +232,7 @@ impl ViewPipeline {
bytes.truncate(bytes.len() - self.descriptor.row_stride as usize); bytes.truncate(bytes.len() - self.descriptor.row_stride as usize);
let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?; let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?;
self.renders.bump(); self.renders.bump();
self.frames.push_back((boundary, artifact)); self.frames.push_back((boundary, counter, artifact));
while self.frames.len() > self.descriptor.observation_delay_steps as usize + 2 { while self.frames.len() > self.descriptor.observation_delay_steps as usize + 2 {
self.frames.pop_front(); self.frames.pop_front();
} }
@ -198,8 +249,8 @@ impl ViewPipeline {
pub fn frame_produced_at(&self, produced: u64) -> Option<(ViewRef, flybus::Artifact)> { pub fn frame_produced_at(&self, produced: u64) -> Option<(ViewRef, flybus::Artifact)> {
self.frames self.frames
.iter() .iter()
.find(|(step, _)| *step == produced) .find(|(step, _, _)| *step == produced)
.map(|(step, artifact)| { .map(|(step, _, artifact)| {
( (
ViewRef { ViewRef {
view_id: self.descriptor.view_id.clone(), view_id: self.descriptor.view_id.clone(),
@ -257,6 +308,52 @@ impl AudioSource {
source source
} }
/// The exact state a capture recorded: sample position, waveform phase and the
/// unconsumed fraction of a frame.
///
/// Restoring the position alone would restart the waveform and round the remainder away,
/// which is a resample the restore rules refuse. The first chunk of the new epoch marks
/// the discontinuity the recovery established.
pub fn restored_from(
descriptor: AudioDescriptor,
next_sample: u64,
phase: u64,
accumulator: u128,
denominator: u128,
) -> DomainResult<AudioSource> {
if denominator == 0 {
return Err(DomainError::invalid(
"audio: a captured accumulator denominator of zero",
));
}
if accumulator >= denominator {
return Err(DomainError::invalid(
"audio: a captured accumulator is not below one whole frame",
));
}
if phase >= descriptor.sample_rate {
return Err(DomainError::invalid(
"audio: a captured phase is not below the sample rate",
));
}
let mut source = AudioSource::new(descriptor, next_sample);
source.discontinuous = true;
source.phase = phase;
source.accumulator = accumulator;
source.denominator = denominator;
Ok(source)
}
/// The waveform phase, for a capture.
pub fn phase(&self) -> u64 {
self.phase
}
/// The unconsumed fraction of a frame and the denominator it is over, for a capture.
pub fn accumulator(&self) -> (u128, u128) {
(self.accumulator, self.denominator)
}
pub fn descriptor(&self) -> &AudioDescriptor { pub fn descriptor(&self) -> &AudioDescriptor {
&self.descriptor &self.descriptor
} }
@ -439,18 +536,47 @@ pub fn check_required_views(
Ok(()) Ok(())
} }
/// Every declared audio stream produces exactly one chunk per transition. /// Where an observation came from.
///
/// `state-media-v1` section 2 makes a chunk the audio of an *interval*, so whether an
/// observation must carry one is a question about its provenance and not about its boundary
/// number. MEDIA-01 wrote the rule as "boundary 0 carries no chunk", which is true of the one
/// observation that slice could produce without a transition and false of the other one:
/// `State.ActivateRestore` installs a coherent observation at boundary `k` without advancing
/// gameplay, and it covers no interval either. Naming the provenance is the fix; exempting
/// the restored observation from the validator instead would have left "must a chunk exist"
/// unanswered exactly where a stale chunk would do the most damage.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ObservationOrigin {
/// The observation a completed transition produced. Its interval has audio.
Transition,
/// An observation established at a boundary without running a transition:
/// `Environment.Initialize`'s `O[0]` and `State.ActivateRestore`'s restored observation.
/// It covers no interval, so it carries no chunk and one in it is refused.
Installed,
}
/// Every declared audio stream produces exactly one chunk per transition, and none at all in
/// an observation that is not one.
/// ///
/// The contract states the shape and the ordering of chunks, not whether one has to exist, so /// The contract states the shape and the ordering of chunks, not whether one has to exist, so
/// this is MEDIA-01's choice and it is deliberate: a session that tolerates a silently missing /// this is MEDIA-01's choice and it is deliberate: a session that tolerates a silently missing
/// chunk cannot tell "this world produced no audio for this interval" from "the chunk was /// chunk cannot tell "this world produced no audio for this interval" from "the chunk was
/// lost", and the second is the case the retention rules care about. Boundary 0 has no /// lost", and the second is the case the retention rules care about. The mirror of that, which
/// preceding interval and so carries no chunk. /// STATE-01 needs, is that an installed observation carrying a chunk is a stale chunk being
/// offered as current, and is refused for the same reason.
pub fn check_required_audio( pub fn check_required_audio(
descriptor: &EnvironmentDescriptor, descriptor: &EnvironmentDescriptor,
observation: &WorldObservation, observation: &WorldObservation,
origin: ObservationOrigin,
) -> DomainResult<()> { ) -> DomainResult<()> {
if observation.boundary == 0 { if origin == ObservationOrigin::Installed {
if let Some(chunk) = observation.audio.first() {
return Err(media_error(format!(
"audio stream {} produced a chunk for an observation that ran no transition",
chunk.stream_id
)));
}
return Ok(()); return Ok(());
} }
for stream in &descriptor.audio { for stream in &descriptor.audio {

File diff suppressed because it is too large Load diff

View file

@ -39,6 +39,16 @@ pub fn episode_schema() -> SchemaRef {
synthetic_schema("arena.episode.v1", 1) synthetic_schema("arena.episode.v1", 1)
} }
/// The schema of a captured task ledger.
pub fn ledger_schema() -> SchemaRef {
synthetic_schema("arena.ledger.v1", 1)
}
/// The schema of a captured action-executor state.
pub fn executor_schema() -> SchemaRef {
synthetic_schema("arena.executor.v1", 1)
}
pub fn controller_schema_ref() -> SchemaRef { pub fn controller_schema_ref() -> SchemaRef {
synthetic_schema("arena.controller.v1", 1) synthetic_schema("arena.controller.v1", 1)
} }
@ -86,6 +96,29 @@ pub trait Task: Send {
/// How many times `evaluate_transition` has run. A transition must evaluate once. /// How many times `evaluate_transition` has run. A transition must evaluate once.
fn evaluations(&self) -> u64; fn evaluations(&self) -> u64;
/// The checkpointable ledger at a committed boundary (`workers-v1` section 4).
fn capture(&self) -> DomainResult<TypedValue>;
/// Validates a captured ledger without installing it, so a group install can fail before
/// anything is changed.
fn validate_restore(&self, state: &TypedValue) -> DomainResult<()>;
/// Installs a validated ledger under `epoch`. Event identity is derived from the epoch,
/// so the new one is part of the install rather than something the ledger keeps from the
/// epoch it was captured in.
fn install_restore(&mut self, epoch: &Id, state: &TypedValue) -> DomainResult<()>;
/// Every event identity this ledger has issued, mapped onto the identity it would have
/// under `to_epoch`.
///
/// `workers-v1` section 4 derives an event id from the epoch, so a trace recorded in one
/// epoch cannot be compared with a trace recorded in another until these are rebased.
/// The ledger owns the derivation, so it is the only thing that can do it.
fn rebase_ids(&self, to_epoch: &Id) -> DomainResult<BTreeMap<Id, Id>>;
/// How far event identity has reached: the highest source step and the number issued.
fn event_watermarks(&self) -> (u64, u64);
} }
/// Translates one selected decision into a controller intent, with no port assignment. /// Translates one selected decision into a controller intent, with no port assignment.
@ -98,6 +131,15 @@ pub trait ActionExecutor: Send {
progress: &TypedValue, progress: &TypedValue,
clock: &RationalNs, clock: &RationalNs,
) -> DomainResult<(ControllerIntent, Vec<TaskEvent>)>; ) -> DomainResult<(ControllerIntent, Vec<TaskEvent>)>;
/// Per-executor state at a committed boundary (`workers-v1` section 4).
fn capture(&self) -> DomainResult<TypedValue>;
/// Validates a captured executor state without installing it.
fn validate_restore(&self, state: &TypedValue) -> DomainResult<()>;
/// Installs a validated executor state.
fn install_restore(&mut self, state: &TypedValue) -> DomainResult<()>;
} }
/// The only executor v1 supports: it passes a direct-control decision through unchanged. /// The only executor v1 supports: it passes a direct-control decision through unchanged.
@ -123,6 +165,37 @@ impl ActionExecutor for IdentityExecutor {
.map_err(|e| DomainError::invalid(format!("decision: {e}")))?; .map_err(|e| DomainError::invalid(format!("decision: {e}")))?;
Ok((intent, Vec::new())) Ok((intent, Vec::new()))
} }
/// The identity executor is stateless, and says so rather than capturing nothing.
///
/// An empty object would be indistinguishable from a stateful executor whose capture went
/// missing, so the capture names the executor it came from and a restore refuses any
/// other one.
fn capture(&self) -> DomainResult<TypedValue> {
TypedValue::new(executor_schema(), json!({"executor": "identity-v1"}))
.map_err(|e| DomainError::invalid(e.0))
}
fn validate_restore(&self, state: &TypedValue) -> DomainResult<()> {
if state.schema != executor_schema() {
return Err(DomainError::before(
ErrorCode::IncompatibleState,
"the captured executor state does not carry the executor schema",
));
}
match state.value.get("executor").and_then(Value::as_str) {
Some("identity-v1") => Ok(()),
other => Err(DomainError::before(
ErrorCode::IncompatibleState,
format!("the captured executor is {other:?}, not the identity executor"),
)),
}
}
fn install_restore(&mut self, state: &TypedValue) -> DomainResult<()> {
// Stateless: validation is the whole of the install, and it is not skipped.
self.validate_restore(state)
}
} }
/// When the counter task asks for a terminal episode transition. /// When the counter task asks for a terminal episode transition.
@ -146,6 +219,10 @@ pub struct CounterTask {
evaluations: u64, evaluations: u64,
total_reward: f64, total_reward: f64,
counter: i64, counter: i64,
/// The highest source step any issued event belongs to, and how many were issued. These
/// are the event watermarks a checkpoint records and a resumed epoch continues from.
last_source_step: u64,
issued_events: u64,
terminal: Terminal, terminal: Terminal,
} }
@ -159,6 +236,8 @@ impl CounterTask {
evaluations: 0, evaluations: 0,
total_reward: 0.0, total_reward: 0.0,
counter: 0, counter: 0,
last_source_step: 0,
issued_events: 0,
terminal, terminal,
} }
} }
@ -239,6 +318,7 @@ impl Task for CounterTask {
payload: TypedValue::new(event_schema(), json!({"counter": self.counter})) payload: TypedValue::new(event_schema(), json!({"counter": self.counter}))
.expect("a synthetic typed value fits the contract"), .expect("a synthetic typed value fits the contract"),
}]; }];
self.issued_events += events.len() as u64;
Ok(Bootstrap { contexts, progress: self.progress_value(), events }) Ok(Bootstrap { contexts, progress: self.progress_value(), events })
} }
@ -314,6 +394,8 @@ impl Task for CounterTask {
)); ));
} }
self.last_source_step = self.last_source_step.max(source_step);
self.issued_events += events.len() as u64;
let next_contexts = self let next_contexts = self
.agents .agents
.iter() .iter()
@ -346,6 +428,173 @@ impl Task for CounterTask {
fn evaluations(&self) -> u64 { fn evaluations(&self) -> u64 {
self.evaluations self.evaluations
} }
fn capture(&self) -> DomainResult<TypedValue> {
TypedValue::new(
ledger_schema(),
json!({
"epoch": self.epoch.as_str(),
"agents": self.agents.iter().map(String::as_str).collect::<Vec<_>>(),
"bindings": self
.bindings
.iter()
.map(|b| json!({"portId": b.port_id.as_str(), "agentId": b.agent_id.as_str()}))
.collect::<Vec<_>>(),
"transitions": self.transitions,
"evaluations": self.evaluations,
"totalReward": self.total_reward,
"counter": self.counter,
"lastSourceStep": self.last_source_step,
"issuedEvents": self.issued_events,
}),
)
.map_err(|e| DomainError::invalid(e.0))
}
fn validate_restore(&self, state: &TypedValue) -> DomainResult<()> {
if state.schema != ledger_schema() {
return Err(DomainError::before(
ErrorCode::IncompatibleState,
"the captured ledger does not carry this task's schema",
));
}
for field in [
"epoch",
"agents",
"bindings",
"transitions",
"evaluations",
"totalReward",
"counter",
"lastSourceStep",
"issuedEvents",
] {
if state.value.get(field).is_none() {
return Err(DomainError::before(
ErrorCode::IncompatibleState,
format!("the captured ledger has no {field}"),
));
}
}
let bindings = state
.value
.get("bindings")
.and_then(Value::as_array)
.ok_or_else(|| {
DomainError::before(
ErrorCode::IncompatibleState,
"the captured ledger's bindings are not a list",
)
})?;
if bindings.len() != self.bindings.len() && !self.bindings.is_empty() {
return Err(DomainError::before(
ErrorCode::IncompatibleState,
"the captured ledger binds another number of ports",
));
}
Ok(())
}
fn install_restore(&mut self, epoch: &Id, state: &TypedValue) -> DomainResult<()> {
self.validate_restore(state)?;
let number = |key: &str| -> DomainResult<u64> {
state.value.get(key).and_then(Value::as_u64).ok_or_else(|| {
DomainError::before(
ErrorCode::IncompatibleState,
format!("the captured ledger's {key} is not a whole number"),
)
})
};
let mut agents = Vec::new();
for value in state.value["agents"].as_array().expect("validated") {
let agent = value.as_str().ok_or_else(|| {
DomainError::before(
ErrorCode::IncompatibleState,
"the captured ledger names an agent that is not a string",
)
})?;
agents.push(parse_id(agent).map_err(|e| {
DomainError::before(ErrorCode::IncompatibleState, format!("ledger: {e}"))
})?);
}
let mut bindings = Vec::new();
for value in state.value["bindings"].as_array().expect("validated") {
let port_id = value.get("portId").and_then(Value::as_str).ok_or_else(|| {
DomainError::before(
ErrorCode::IncompatibleState,
"the captured ledger has a binding with no portId",
)
})?;
let agent_id = value.get("agentId").and_then(Value::as_str).ok_or_else(|| {
DomainError::before(
ErrorCode::IncompatibleState,
"the captured ledger has a binding with no agentId",
)
})?;
bindings.push(PortBinding {
port_id: parse_id(port_id).map_err(|e| {
DomainError::before(ErrorCode::IncompatibleState, format!("ledger: {e}"))
})?,
agent_id: parse_id(agent_id).map_err(|e| {
DomainError::before(ErrorCode::IncompatibleState, format!("ledger: {e}"))
})?,
});
}
let counter = state.value.get("counter").and_then(Value::as_i64).ok_or_else(|| {
DomainError::before(
ErrorCode::IncompatibleState,
"the captured ledger's counter is not an integer",
)
})?;
let total_reward = state
.value
.get("totalReward")
.and_then(Value::as_f64)
.filter(|v| v.is_finite())
.ok_or_else(|| {
DomainError::before(
ErrorCode::IncompatibleState,
"the captured ledger's totalReward is not a finite number",
)
})?;
// The epoch is the caller's, not the capture's: event identity belongs to the epoch
// the ledger is being installed into.
self.epoch = epoch.clone();
self.agents = agents;
self.bindings = bindings;
self.transitions = number("transitions")?;
self.evaluations = number("evaluations")?;
self.total_reward = total_reward;
self.counter = counter;
self.last_source_step = number("lastSourceStep")?;
self.issued_events = number("issuedEvents")?;
Ok(())
}
fn rebase_ids(&self, to_epoch: &Id) -> DomainResult<BTreeMap<Id, Id>> {
let mut out = BTreeMap::new();
out.insert(
event_id(&self.epoch, 0, "bootstrap", 0),
event_id(to_epoch, 0, "bootstrap", 0),
);
// The counter task issues exactly one `counter-delta` event per bound port per
// evaluated transition, in descriptor port order, so every identity it has ever
// issued is re-derivable from its ledger without keeping a list of them.
let ports = self.bindings.len() as u32;
for source_step in 1..=self.last_source_step {
for ordinal in 0..ports {
out.insert(
event_id(&self.epoch, source_step, "counter-delta", ordinal),
event_id(to_epoch, source_step, "counter-delta", ordinal),
);
}
}
Ok(out)
}
fn event_watermarks(&self) -> (u64, u64) {
(self.last_source_step, self.issued_events)
}
} }
/// The inspection value the counter environment publishes. /// The inspection value the counter environment publishes.

View file

@ -8,13 +8,18 @@
//! `Id` and `Digest` are type aliases, because the shared crate carries both as validated //! `Id` and `Digest` are type aliases, because the shared crate carries both as validated
//! `String`s from `flybus::wire` rather than forking the encodings into newtypes. //! `String`s from `flybus::wire` rather than forking the encodings into newtypes.
use std::collections::BTreeMap;
use serde_json::{Map, Value}; use serde_json::{Map, Value};
pub use fly_session_types::ArtifactRef; pub use fly_session_types::ArtifactRef;
pub use fly_session_types::canonical::{ pub use fly_session_types::canonical::{
self, OperationKey, body_digest, canonicalize, digest_of, sha256_hex, self, OperationKey, body_digest, canonicalize, digest_of, sha256_hex,
}; };
pub use fly_session_types::media::{AudioDescriptor, AudioRef, ViewDescriptor, ViewRef}; pub use fly_session_types::media::{
ActivateRestoreParams, ActivateRestoreResult, AudioDescriptor, AudioRef, CaptureParams,
CaptureResult, StageRestoreParams, StageRestoreResult, ViewDescriptor, ViewRef,
};
pub use fly_session_types::rpc::{ pub use fly_session_types::rpc::{
ErrorCode, MutationCertainty, SessionRpcFailure, SessionRpcOutcome, SessionRpcRequest, ErrorCode, MutationCertainty, SessionRpcFailure, SessionRpcOutcome, SessionRpcRequest,
SessionRpcSuccess, SessionRpcSuccess,
@ -214,6 +219,57 @@ pub fn outcome_identity(
} }
} }
/// The epoch-derived identities of a behaviour trace, rewritten onto one reference epoch.
///
/// `step-v1` section 8 compares committed behaviour across runs, excluding wall time, request
/// ids "and other explicitly operational metadata". A resumed run's epoch is neither: it is
/// behaviour metadata, and `scope.epoch`, the batch id and every task event id are derived
/// from it. Comparing the two runs therefore means rewriting exactly those three things and
/// nothing else, which is what this does -- and it **fails** on anything it does not
/// recognise instead of passing it through, so a field that silently stopped being rebased
/// would fail the comparison rather than weaken it.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct EpochRebase {
pub from: Id,
pub to: Id,
/// Every event identity the task issued under `from`, and the identity it has under `to`.
pub events: BTreeMap<Id, Id>,
}
impl EpochRebase {
/// Rewrites one behaviour record. An identity this rebase does not know is an error.
pub fn apply(&self, behaviour: &TraceBehaviour) -> Result<TraceBehaviour, String> {
if behaviour.scope.epoch != self.from {
return Err(format!(
"this behaviour was recorded in epoch {}, not {}",
behaviour.scope.epoch, self.from
));
}
let mut out = behaviour.clone();
out.scope = Scope::new(&behaviour.scope.session_id, &self.to, behaviour.scope.step)
.map_err(|e| e.0)?;
let prefix = format!("batch-{}-", self.from);
let suffix = behaviour
.batch_id
.strip_prefix(&prefix)
.ok_or_else(|| format!("the batch id {} is not derived from {}", behaviour.batch_id, self.from))?;
out.batch_id = parse_id(&format!("batch-{}-{suffix}", self.to))?;
let map = |ids: &[Id]| -> Result<Vec<Id>, String> {
ids.iter()
.map(|id| {
self.events
.get(id)
.cloned()
.ok_or_else(|| format!("no rebased identity for the event {id}"))
})
.collect()
};
out.outcome_ids = map(&behaviour.outcome_ids)?;
out.event_ids = map(&behaviour.event_ids)?;
Ok(out)
}
}
/// One session phase transition, recorded whether or not it ends a step. /// One session phase transition, recorded whether or not it ends a step.
#[derive(Clone, Debug, PartialEq, Eq)] #[derive(Clone, Debug, PartialEq, Eq)]
pub struct PhaseTransition { pub struct PhaseTransition {
@ -253,6 +309,28 @@ impl TraceLog {
.collect() .collect()
} }
/// Every transition's behaviour, rebased onto one epoch and canonicalized.
///
/// This is the comparison a resumed run is held to: the same strings as
/// [`TraceLog::behavior`], with the epoch metadata accounted for and nothing else changed.
/// A resumed run's log holds transitions from two epochs -- the ones before the checkpoint
/// and the ones after the restore -- so a transition already recorded in `rebase.to` is
/// kept as it stands and one recorded in `rebase.from` is rewritten. A transition in a
/// third epoch is an error; there is no pass-through case.
pub fn behavior_rebased(&self, rebase: &EpochRebase) -> Result<Vec<String>, String> {
self.transitions
.iter()
.map(|t| {
let behaviour = if t.behaviour.scope.epoch == rebase.to {
t.behaviour.clone()
} else {
rebase.apply(&t.behaviour)?
};
canonicalize(&behaviour.to_json()).map_err(|e| e.0)
})
.collect()
}
/// The phase path, as `from -> to` strings. /// The phase path, as `from -> to` strings.
pub fn phase_path(&self) -> Vec<String> { pub fn phase_path(&self) -> Vec<String> {
self.phases.iter().map(|p| format!("{} -> {}", p.from, p.to)).collect() self.phases.iter().map(|p| format!("{} -> {}", p.from, p.to)).collect()

View file

@ -597,7 +597,17 @@ async fn execute<E: WorkerEndpoint>(
fn classify_default(method: &str) -> Option<OpClass> { fn classify_default(method: &str) -> Option<OpClass> {
match method { match method {
"Agent.Prepare" | "Agent.Commit" | "Environment.Advance" => Some(OpClass::StepMutation), "Agent.Prepare" | "Agent.Commit" | "Environment.Advance" => Some(OpClass::StepMutation),
"Agent.Initialize" | "Environment.Initialize" => Some(OpClass::Lifecycle), // `ipc-v1` section 5 retains lifecycle *and capture* replies until
// `Worker.Acknowledge`. The restore methods join them: their replies carry a
// once-only token and, for an environment, the restored observation's artifact, and a
// duplicate domain request must replay that reply rather than stage or activate a
// second time. They are not step mutations -- they carry no committed step of their
// own and are not keyed by one.
"Agent.Initialize"
| "Environment.Initialize"
| "State.Capture"
| "State.StageRestore"
| "State.ActivateRestore" => Some(OpClass::Lifecycle),
"Worker.Hello" | "Worker.Status" | "Worker.Acknowledge" | "Worker.Shutdown" => { "Worker.Hello" | "Worker.Status" | "Worker.Acknowledge" | "Worker.Shutdown" => {
Some(OpClass::ReadOnly) Some(OpClass::ReadOnly)
} }

View file

@ -173,6 +173,8 @@ async fn a_slow_participant_is_resolved_rather_than_failed(mode: ExecutionMode)
resolve: Duration::from_secs(15), resolve: Duration::from_secs(15),
resolve_attempts: u32::MAX, resolve_attempts: u32::MAX,
boot: Duration::from_secs(30), boot: Duration::from_secs(30),
capture: Duration::from_secs(30),
durable: Duration::from_secs(60),
}; };
let reports = within("run", f.harness.coordinator.run(2)) let reports = within("run", f.harness.coordinator.run(2))
.await .await
@ -262,6 +264,8 @@ async fn a_resolution_says_which_of_its_two_bounds_ended_it(mode: ExecutionMode)
resolve: Duration::from_millis(200), resolve: Duration::from_millis(200),
resolve_attempts: u32::MAX, resolve_attempts: u32::MAX,
boot: Duration::from_secs(30), boot: Duration::from_secs(30),
capture: Duration::from_secs(30),
durable: Duration::from_secs(60),
}; };
let failure = within("step", f.harness.coordinator.step()) let failure = within("step", f.harness.coordinator.step())
.await .await
@ -288,6 +292,8 @@ async fn a_resolution_says_which_of_its_two_bounds_ended_it(mode: ExecutionMode)
resolve: Duration::from_secs(3_600), resolve: Duration::from_secs(3_600),
resolve_attempts: 3, resolve_attempts: 3,
boot: Duration::from_secs(30), boot: Duration::from_secs(30),
capture: Duration::from_secs(30),
durable: Duration::from_secs(60),
}; };
let failure = within("step", f.harness.coordinator.step()) let failure = within("step", f.harness.coordinator.step())
.await .await

File diff suppressed because it is too large Load diff

View file

@ -56,7 +56,7 @@ impl MemoryReader for &mut dyn MemoryReader {
/// One reward payout in one frame. /// One reward payout in one frame.
/// ///
/// `kind` is an adapter-owned interned name (Pokémon: `milestone`, /// `kind` is an adapter-owned interned name (Pokémon: `milestone`,
/// `exploration`, `map`, `species`, `trainer`, `battle`, `badge`, `boundary`); it is the /// `exploration`, `map`, `species`, `trainer`, `battle`, `badge`, `boundary`, `catch`); it is the
/// key the statistics counters and the on-screen ticker group by. Field names /// key the statistics counters and the on-screen ticker group by. Field names
/// serialize exactly as the prototype's `RewardEvent` did, so a checkpoint /// serialize exactly as the prototype's `RewardEvent` did, so a checkpoint
/// written by either implementation reads in the other. /// written by either implementation reads in the other.
@ -241,9 +241,21 @@ impl std::error::Error for AdapterError {}
/// A game, as the sim loop sees it. /// A game, as the sim loop sees it.
pub trait GameAdapter: Send { pub trait GameAdapter: Send {
/// Adapter version string, pinned into the checkpoint compatibility string. /// Adapter version string, pinned into the checkpoint compatibility string.
/// Pokémon: `pokered-unique8-v5`. /// Pokémon: `pokered-unique8-v6`.
fn id(&self) -> &'static str; fn id(&self) -> &'static str;
/// Earlier [`GameAdapter::id`]s whose checkpoints this build can read, by a migration
/// this adapter has written down and tested.
///
/// The default is empty: an adapter migrates from nothing unless it says otherwise, which
/// is the behaviour every adapter had before this existed. It is only half of the gate --
/// [`crate::compatibility::decide`] also requires the operator to have named the same id in
/// `FLY_ACCEPT_ADAPTERS` for that deploy -- so listing an id here never migrates a live run
/// on its own.
fn migrates_from(&self) -> &'static [&'static str] {
&[]
}
/// Whether semantic rewards are enabled for this cartridge. An adapter that /// Whether semantic rewards are enabled for this cartridge. An adapter that
/// says no must still sample without paying anything, so the stream keeps /// says no must still sample without paying anything, so the stream keeps
/// running with a visible "rewards off" mode. /// running with a visible "rewards off" mode.
@ -470,6 +482,10 @@ mod tests {
let platformer = let platformer =
adapter_for_with_rom_pin("platformer", Some(&"a".repeat(64))).unwrap(); adapter_for_with_rom_pin("platformer", Some(&"a".repeat(64))).unwrap();
assert_ne!(pokemon.id(), platformer.id()); assert_ne!(pokemon.id(), platformer.id());
assert!(
!platformer.migrates_from().contains(&pokemon.id()),
"a migration never crosses games"
);
assert_ne!(pokemon.symbol_provenance(), platformer.symbol_provenance()); assert_ne!(pokemon.symbol_provenance(), platformer.symbol_provenance());
// And two ROM revisions of the same game cannot either. // And two ROM revisions of the same game cannot either.
let other = adapter_for_with_rom_pin("platformer", Some(&"b".repeat(64))).unwrap(); let other = adapter_for_with_rom_pin("platformer", Some(&"b".repeat(64))).unwrap();

View file

@ -30,7 +30,7 @@ pub const PROTOTYPE_PLASTICITY_VERSION: &str = "fly-kc-mbon-rstdp-v2";
pub struct Compatibility<'a> { pub struct Compatibility<'a> {
/// `kernelVersion(config)` from the neural library. /// `kernelVersion(config)` from the neural library.
pub neural_kernel_version: &'a str, pub neural_kernel_version: &'a str,
/// The adapter's version string, e.g. `pokered-unique8-v5`. /// The adapter's version string, e.g. `pokered-unique8-v6`.
pub adapter: &'a str, pub adapter: &'a str,
/// The dataset's seven SHA-256 digests joined with `:`. /// The dataset's seven SHA-256 digests joined with `:`.
pub dataset_fingerprint: &'a str, pub dataset_fingerprint: &'a str,
@ -67,6 +67,92 @@ impl Compatibility<'_> {
} }
} }
/// Position of the adapter's version string in [`Compatibility::string`].
///
/// `{kernel}/{adapter}/{fingerprint}/{plasticity}/binjgb:{rev}/pokered:{commit}/statefmt:{id}`,
/// so the adapter is segment one. Nothing else in the string may move for a migration to be
/// considered: a different kernel, dataset, plasticity, emulator revision, symbol provenance or
/// state format is a different *fly*, not a different reward rule.
const ADAPTER_SEGMENT: usize = 1;
/// The environment variable that opts a deploy into the adapter migration.
///
/// Read by flysim at restore and by `infra/05-deploy.sh`'s compatibility gate. Comma- or
/// whitespace-separated adapter ids, e.g. `FLY_ACCEPT_ADAPTERS=pokered-unique8-v5`.
pub const ACCEPT_ADAPTERS_ENV: &str = "FLY_ACCEPT_ADAPTERS";
/// What a build may do with a checkpoint whose compatibility string is not its own.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum RestoreDecision {
/// Byte-identical. Restore it, as every build always has.
Exact,
/// Every segment but the adapter's is identical, this build's adapter says it can migrate
/// from that one, and the operator named it in [`ACCEPT_ADAPTERS_ENV`]. Restore it.
MigrateAdapter { from: String },
/// Refuse, and say which of the three conditions failed.
Refuse(&'static str),
}
/// Decide whether `checkpoint`'s compatibility string may be restored under `current`.
///
/// Three conditions, all required, in the order they are cheapest to explain:
///
/// 1. the two strings differ in the adapter segment and **nowhere else**;
/// 2. `migrates_from` -- the running adapter's own list -- contains the checkpoint's adapter, so
/// the code that will read that state says out loud that it can;
/// 3. `accepted` -- [`ACCEPT_ADAPTERS_ENV`] as the operator set it for this deploy -- contains it
/// too, so no build ever migrates a run by itself.
///
/// Condition 2 without condition 3 would make the migration silent; condition 3 without condition
/// 2 would let an operator wave through a pair nobody wrote a migration for. Neither alone is
/// enough, which is why both are here.
pub fn decide(
checkpoint: &str,
current: &str,
migrates_from: &[&str],
accepted: &[String],
) -> RestoreDecision {
if checkpoint == current {
return RestoreDecision::Exact;
}
let old: Vec<&str> = checkpoint.split('/').collect();
let new: Vec<&str> = current.split('/').collect();
if old.len() != new.len() {
return RestoreDecision::Refuse("the two compatibility strings do not have the same shape");
}
let differing: Vec<usize> = (0..old.len()).filter(|&index| old[index] != new[index]).collect();
if differing != [ADAPTER_SEGMENT] {
return RestoreDecision::Refuse(
"more than the adapter version differs; nothing but a reward-rule change can migrate",
);
}
let from = old[ADAPTER_SEGMENT];
if !migrates_from.contains(&from) {
return RestoreDecision::Refuse("this build's adapter has no migration from that adapter");
}
if !accepted.iter().any(|name| name == from) {
return RestoreDecision::Refuse(
"the checkpoint's adapter is not in FLY_ACCEPT_ADAPTERS, so the migration was not \
asked for",
);
}
RestoreDecision::MigrateAdapter { from: from.to_string() }
}
/// Parse [`ACCEPT_ADAPTERS_ENV`]: comma- or whitespace-separated, empty entries dropped.
///
/// An unset variable and an empty one are the same thing -- no migration -- so that clearing the
/// opt-in is one edit rather than two.
pub fn accepted_adapters(value: Option<&str>) -> Vec<String> {
value
.unwrap_or_default()
.split([',', ' ', '\t', '\n'])
.map(str::trim)
.filter(|entry| !entry.is_empty())
.map(str::to_string)
.collect()
}
/// `<state size>-<target triple>`: the two things that decide whether a /// `<state size>-<target triple>`: the two things that decide whether a
/// binjgb save state written elsewhere can be memcpy'd back in here. /// binjgb save state written elsewhere can be memcpy'd back in here.
pub fn state_format_id() -> String { pub fn state_format_id() -> String {
@ -94,7 +180,7 @@ mod tests {
assert_eq!( assert_eq!(
fixture().prototype_string(), fixture().prototype_string(),
concat!( concat!(
"lif-1ms-f64-v2/pokered-unique8-v5/aa:bb:cc:dd:ee:ff:00/", "lif-1ms-f64-v2/pokered-unique8-v6/aa:bb:cc:dd:ee:ff:00/",
"fly-kc-mbon-rstdp-v2/", "fly-kc-mbon-rstdp-v2/",
"binjgb:c60e138da5a795ebb55e56b11b7e90024e41112c/", "binjgb:c60e138da5a795ebb55e56b11b7e90024e41112c/",
"pokered:0cd19d3b877b7dc66d12c7050bed9a7f38154d4b", "pokered:0cd19d3b877b7dc66d12c7050bed9a7f38154d4b",
@ -102,6 +188,73 @@ mod tests {
); );
} }
fn with_adapter(adapter: &'static str) -> String {
Compatibility { adapter, ..fixture() }.string()
}
#[test]
fn an_identical_string_restores_without_any_opt_in() {
let current = with_adapter("pokered-unique8-v6");
assert_eq!(decide(&current, &current, &[], &[]), RestoreDecision::Exact);
}
#[test]
fn a_v5_checkpoint_restores_under_v6_only_with_the_opt_in() {
let old = with_adapter("pokered-unique8-v5");
let new = with_adapter("pokered-unique8-v6");
let migrates = ["pokered-unique8-v5"];
assert!(matches!(decide(&old, &new, &migrates, &[]), RestoreDecision::Refuse(_)));
assert_eq!(
decide(&old, &new, &migrates, &accepted_adapters(Some("pokered-unique8-v5"))),
RestoreDecision::MigrateAdapter { from: "pokered-unique8-v5".to_string() }
);
// And only for a pair the running adapter says it can migrate.
assert!(matches!(
decide(&old, &new, &[], &accepted_adapters(Some("pokered-unique8-v5"))),
RestoreDecision::Refuse(_)
));
}
#[test]
fn nothing_but_the_adapter_segment_may_move() {
let migrates = ["pokered-unique8-v5"];
let accepted = accepted_adapters(Some("pokered-unique8-v5"));
let new = with_adapter("pokered-unique8-v6");
// A different dataset, with the same adapter bump, is not a migration.
let other_dataset = Compatibility {
adapter: "pokered-unique8-v5",
dataset_fingerprint: "00:11:22:33:44:55:66",
..fixture()
}
.string();
assert!(matches!(
decide(&other_dataset, &new, &migrates, &accepted),
RestoreDecision::Refuse(_)
));
// Neither is a different kernel, and neither is a string of another shape.
let other_kernel =
Compatibility { adapter: "pokered-unique8-v5", neural_kernel_version: "lif-1ms-f64-v3", ..fixture() }
.string();
assert!(matches!(
decide(&other_kernel, &new, &migrates, &accepted),
RestoreDecision::Refuse(_)
));
assert!(matches!(decide("a/b", &new, &migrates, &accepted), RestoreDecision::Refuse(_)));
}
#[test]
fn the_opt_in_list_is_separated_by_commas_or_spaces() {
assert!(accepted_adapters(None).is_empty());
assert!(accepted_adapters(Some(" ")).is_empty());
assert_eq!(
accepted_adapters(Some("pokered-unique8-v5, pokered-unique8-v4")),
vec!["pokered-unique8-v5".to_string(), "pokered-unique8-v4".to_string()]
);
}
#[test] #[test]
fn the_state_format_segment_is_appended_not_interleaved() { fn the_state_format_segment_is_appended_not_interleaved() {
let full = fixture().string(); let full = fixture().string();

View file

@ -1,4 +1,4 @@
//! The `pokered-unique8-v5` reward catalog. //! The `pokered-unique8-v6` reward catalog.
//! //!
//! A direct port of the prototype's `src/reward/catalog.ts`, including the //! A direct port of the prototype's `src/reward/catalog.ts`, including the
//! declaration order, which is the order `counts` and `last` serialize in. //! declaration order, which is the order `counts` and `last` serialize in.
@ -20,8 +20,18 @@ pub mod kind {
pub const BATTLE: &str = "battle"; pub const BATTLE: &str = "battle";
pub const BADGE: &str = "badge"; pub const BADGE: &str = "badge";
pub const BOUNDARY: &str = "boundary"; pub const BOUNDARY: &str = "boundary";
pub const CATCH: &str = "catch";
} }
/// What a `catch` of a species this run has already caught pays.
///
/// Not a multiple of the rule's catalog value, because no binary float scales 0.30 into
/// exactly 0.10: `0.3 * (1.0 / 3.0)` is `0.09999999999999999`, and that number would reach
/// the ticker, the checkpoint and `docs/rewards-learning.md`'s table as itself. `boundary`'s
/// two payouts are 0.05 and 0.10, which a scale of two does express exactly, so that rule
/// still goes through the scaling path.
pub const CATCH_REPEAT_VALUE: f64 = 0.10;
#[derive(Debug, Clone, Copy)] #[derive(Debug, Clone, Copy)]
pub struct RewardRule { pub struct RewardRule {
pub kind: &'static str, pub kind: &'static str,
@ -34,7 +44,7 @@ pub struct RewardRule {
pub stimulation_ms: u32, pub stimulation_ms: u32,
} }
pub const REWARDS: [RewardRule; 8] = [ pub const REWARDS: [RewardRule; 9] = [
RewardRule { RewardRule {
kind: kind::MILESTONE, kind: kind::MILESTONE,
label: "Story", label: "Story",
@ -99,6 +109,22 @@ pub const REWARDS: [RewardRule; 8] = [
value: 0.05, value: 0.05,
stimulation_ms: 100, stimulation_ms: 100,
}, },
// The operator's decision of 2026-09-22: the fly is paid for *keeping* a wild Pokémon, not
// only for knocking one out. Appended rather than slotted next to `species` for the same
// reason `boundary` was appended -- the declaration order is the key order `counts`
// serializes in, and every checkpoint already written carries the first eight in this order.
//
// One rule, two payouts, like `boundary`: this value is what a species this run has never
// caught pays, and [`CATCH_REPEAT_VALUE`] is what a repeat pays. The existing `species`
// rule is untouched and still pays 0.50 the first time a species is owned by any means, so
// a first catch of a new species pays 0.50 + 0.30 across two kinds.
RewardRule {
kind: kind::CATCH,
label: "Catch",
trigger: "Wild Pokémon caught; 0.10 for a species already caught; max 3 per species",
value: 0.30,
stimulation_ms: 150,
},
]; ];
/// Position of `kind` in [`REWARDS`], or `None` for an unknown kind. This is /// Position of `kind` in [`REWARDS`], or `None` for an unknown kind. This is
@ -199,10 +225,34 @@ mod tests {
// separate kinds. // separate kinds.
assert_eq!(rule(kind::BOUNDARY).unwrap().value, 0.05); assert_eq!(rule(kind::BOUNDARY).unwrap().value, 0.05);
assert_eq!(rule(kind::BOUNDARY).unwrap().stimulation_ms, 100); assert_eq!(rule(kind::BOUNDARY).unwrap().stimulation_ms, 100);
// Nor the prototype's: the operator's catch rule, `pokered-unique8-v6`.
assert_eq!(rule(kind::CATCH).unwrap().value, 0.30);
assert_eq!(CATCH_REPEAT_VALUE, 0.10);
assert_eq!(rule(kind::CATCH).unwrap().stimulation_ms, 150);
assert!(rule("blackout").is_none(), "the catalog has no penalties"); assert!(rule("blackout").is_none(), "the catalog has no penalties");
assert!(REWARDS.iter().all(|rule| rule.value > 0.0)); assert!(REWARDS.iter().all(|rule| rule.value > 0.0));
} }
#[test]
fn the_catch_rule_is_last_so_the_older_key_order_does_not_move() {
let order: Vec<&str> = REWARDS.iter().map(|rule| rule.kind).collect();
assert_eq!(
order,
vec![
kind::MILESTONE,
kind::EXPLORATION,
kind::MAP,
kind::SPECIES,
kind::TRAINER,
kind::BATTLE,
kind::BADGE,
kind::BOUNDARY,
kind::CATCH,
]
);
assert_eq!(index(kind::CATCH), Some(REWARDS.len() - 1));
}
#[test] #[test]
fn empty_counts_lists_every_kind_at_zero() { fn empty_counts_lists_every_kind_at_zero() {
let counts = Counts::default(); let counts = Counts::default();

View file

@ -1,11 +1,12 @@
//! The Pokémon Red reward adapter, `pokered-unique8-v5`. //! The Pokémon Red reward adapter, `pokered-unique8-v6`.
//! //!
//! A port of the prototype's `src/reward/pokemon-red.ts`. The gates and budgets //! A port of the prototype's `src/reward/pokemon-red.ts`. The gates and budgets
//! are unchanged; `docs/rewards-learning.md` holds the live rule table and the //! are unchanged; `docs/rewards-learning.md` holds the live rule table and the
//! source evidence behind each gate. v4 replaced the 16-rung boot-to-badges //! source evidence behind each gate. v4 replaced the 16-rung boot-to-badges
//! ladder with the 38 rungs of `docs/design/ladder.md`; v5 adds one reward rule, //! ladder with the 38 rungs of `docs/design/ladder.md`; v5 adds one reward rule,
//! `boundary` (`docs/design/room-escape.md` section 2), which pays the first step //! `boundary` (`docs/design/room-escape.md` section 2), which pays the first step
//! next to and the first step onto each of a map's exits. //! next to and the first step onto each of a map's exits; v6 adds `catch`, the
//! operator's decision of 2026-09-22, which pays for keeping a wild Pokémon.
pub mod catalog; pub mod catalog;
#[cfg(test)] #[cfg(test)]
@ -32,13 +33,33 @@ use symbols::ram;
/// Adapter version, pinned into the checkpoint compatibility string. /// Adapter version, pinned into the checkpoint compatibility string.
/// ///
/// `v5` is the `boundary` rule. Bumping it is what rejects every checkpoint written /// `v6` is the `catch` rule. Bumping it is what makes a `v5` checkpoint a decision
/// by `v4`: the string is compared whole before a restore is attempted, so a ledger /// rather than an accident: the compatibility string is compared whole before a
/// that has never recorded a single `boundary:` key can never be resumed as though /// restore is attempted, so a `v5` run is refused by default and resumed only when
/// its exits were already collected. (`v4` was the 38-rung ladder, and rejected /// the operator names it in `FLY_ACCEPT_ADAPTERS`
/// `v3` for the same reason: a stored rank that meant "4 badges" on the old ladder /// ([`crate::compatibility::RestoreDecision`], `docs/design/flysim.md`). That
/// could not be read as a rung on the new one.) Pre-launch, so no run is lost. /// migration is safe in one direction only, and only for this pair: `v5`'s ledger is
pub const REWARD_ADAPTER: &str = "pokered-unique8-v5"; /// a `v6` ledger with the catch counter absent, and an absent counter reads as zero.
///
/// (`v5` was the `boundary` rule, and rejected `v4` because a ledger that had never
/// recorded a `boundary:` key could not be resumed as though its exits were already
/// collected. `v4` was the 38-rung ladder, and rejected `v3` because a stored rank
/// that meant "4 badges" on the old ladder is not a rung on the new one. Neither of
/// those is a migration: this one is, because nothing a `v5` ledger holds means
/// something different under `v6`.)
pub const REWARD_ADAPTER: &str = "pokered-unique8-v6";
/// Adapter ids whose checkpoints `v6` can read.
///
/// Exactly one, and it is one because the `catch` rule adds a counter and changes nothing else:
/// a `v5` ledger restores as a `v6` ledger with `catchCounts` empty, and every other byte of the
/// state means what it meant. `v4` is not here -- its `seen` ledger holds no `boundary:` keys, so
/// resuming it would pay a second time for every exit the run had already found -- and neither is
/// `v3`, whose stored rank is a rung on a different ladder.
///
/// Listing an id here is necessary but not sufficient: `FLY_ACCEPT_ADAPTERS` must name it too
/// (`crate::compatibility::decide`, `docs/design/flysim.md`).
pub const MIGRATES_FROM: &[&str] = &["pokered-unique8-v5"];
/// The only cartridge semantic rewards are enabled for. Even the canonical /// The only cartridge semantic rewards are enabled for. Even the canonical
/// pret build stays disabled until reviewed; see `docs/rewards-learning.md`. /// pret build stays disabled until reviewed; see `docs/rewards-learning.md`.
@ -60,8 +81,22 @@ pub const SUPPORTED_ROM: &str =
/// [`REWARD_ADAPTER`] is the gate that refuses such a checkpoint anyway, and it is /// [`REWARD_ADAPTER`] is the gate that refuses such a checkpoint anyway, and it is
/// the right gate, because the objection to loading one is about semantics rather /// the right gate, because the objection to loading one is about semantics rather
/// than shape. /// than shape.
///
/// *Not* bumped for the `catch` rule either, and this time the answer matters,
/// because `v5` checkpoints are meant to be restorable under `v6`. The rule adds one
/// counter, `catchCounts`, and nothing else: every other field keeps its name, its
/// shape and its meaning, and a state written without the counter restores with it
/// empty, which is the truth about a run that was never paid for a catch. That is the
/// whole of the documented `v5` -> `v6` migration; see
/// [`crate::compatibility::RestoreDecision`].
pub const STATE_VERSION: u64 = 4; pub const STATE_VERSION: u64 = 4;
/// Catch payouts one species may earn in the lifetime of a run's ledger.
///
/// The same cap and the same reason as the wild-KO rule's three: a species the fly can
/// find over and over is a farm, and three is enough for the behaviour to be learned.
const MAX_CATCH_PAYOUTS: u64 = 3;
const BADGE_NAMES: [&str; 8] = [ const BADGE_NAMES: [&str; 8] = [
"BOULDER", "CASCADE", "THUNDER", "RAINBOW", "SOUL", "MARSH", "VOLCANO", "EARTH", "BOULDER", "CASCADE", "THUNDER", "RAINBOW", "SOUL", "MARSH", "VOLCANO", "EARTH",
]; ];
@ -325,6 +360,26 @@ struct Battle {
wild: bool, wild: bool,
saw_living: bool, saw_living: bool,
ko: bool, ko: bool,
/// Lifetime `species` payouts when this battle started.
///
/// The "never owned this run" test for the catch rule, and an exact one: the only
/// thing that can set a `wPokedexOwned` bit during a wild battle is the catch
/// itself, so a `species` payout between the battle starting and the ball keeping
/// the Pokémon *is* that Pokémon being new. It is read this way rather than from
/// `wCapturedMonSpecies` directly because that byte is the cartridge's **internal**
/// species index and the owned bitset is by **Pokédex number**; the two numberings
/// differ and nothing in WRAM converts between them
/// (`docs/design/macros-wram.md` section 2, "species numbering").
///
/// `None` for a battle restored from a checkpoint written before this existed,
/// which reads as "cannot tell" and pays the repeat amount rather than guessing
/// generously.
species_at_start: Option<u64>,
/// The internal species index `wCapturedMonSpecies` named, once a ball has kept one.
captured: Option<u8>,
/// Whether that catch was a species this run had never owned, decided on the frame
/// the capture was observed.
captured_new: bool,
} }
/// Immutable per-sample byte cache. Each address requested during one sample /// Immutable per-sample byte cache. Each address requested during one sample
@ -370,6 +425,10 @@ pub struct PokemonRedReward {
tiles: OrderedSet, tiles: OrderedSet,
tile_counts: BTreeMap<u8, u64>, tile_counts: BTreeMap<u8, u64>,
wild_wins: BTreeMap<String, u64>, wild_wins: BTreeMap<String, u64>,
/// Catch payouts per species, by the cartridge's internal species index as a decimal
/// string. The one field `v6` adds to the checkpoint; absent in a `v5` state, which
/// reads as every species at zero.
catch_counts: BTreeMap<String, u64>,
replay_blocked: OrderedSet, replay_blocked: OrderedSet,
counts: Counts, counts: Counts,
total: f64, total: f64,
@ -420,6 +479,7 @@ impl PokemonRedReward {
tiles: OrderedSet::new(), tiles: OrderedSet::new(),
tile_counts: BTreeMap::new(), tile_counts: BTreeMap::new(),
wild_wins: BTreeMap::new(), wild_wins: BTreeMap::new(),
catch_counts: BTreeMap::new(),
replay_blocked: OrderedSet::new(), replay_blocked: OrderedSet::new(),
counts: Counts::default(), counts: Counts::default(),
total: 0.0, total: 0.0,
@ -567,8 +627,9 @@ impl PokemonRedReward {
} }
/// Forget observations a rollback invalidates. Lifetime novelty survives, /// Forget observations a rollback invalidates. Lifetime novelty survives,
/// and every wild-KO key paid so far is blocked from paying again, because /// and every wild-KO key and every caught species paid so far is blocked from
/// after a rollback the same battle could otherwise be replayed for reward. /// paying again, because after a rollback the same battle -- or the same catch --
/// could otherwise be replayed for reward.
pub fn clear_transient(&mut self) { pub fn clear_transient(&mut self) {
self.location.clear(); self.location.clear();
self.stable = 0; self.stable = 0;
@ -578,6 +639,10 @@ impl PokemonRedReward {
for key in keys { for key in keys {
self.replay_blocked.insert(&key); self.replay_blocked.insert(&key);
} }
let caught: Vec<String> = self.catch_counts.keys().cloned().collect();
for species in caught {
self.replay_blocked.insert(&format!("catch:{species}"));
}
} }
/// Sample WRAM after one completed frame and return this frame's payouts. /// Sample WRAM after one completed frame and return this frame's payouts.
@ -690,14 +755,29 @@ impl PokemonRedReward {
if in_battle == 1 || in_battle == 2 || in_battle == 255 { if in_battle == 1 || in_battle == 2 || in_battle == 255 {
self.mode = "BATTLE".to_string(); self.mode = "BATTLE".to_string();
self.stable = 0; self.stable = 0;
let species_paid = self.counts.get(kind::SPECIES);
if self.battle.is_none() && in_battle != 255 { if self.battle.is_none() && in_battle != 255 {
self.battle = Some(Battle { self.battle = Some(Battle {
key: battle_key(memory, map), key: battle_key(memory, map),
wild: in_battle == 1, wild: in_battle == 1,
saw_living: false, saw_living: false,
ko: false, ko: false,
species_at_start: Some(species_paid),
captured: None,
captured_new: false,
}); });
} }
// The cartridge's own answer to "was one caught": `ram/wram.asm`'s comment on
// this byte is "0 if no mon was captured". `ItemUseBall` zeroes it before every
// throw and writes `wEnemyMonSpecies` into it only on the branch that keeps the
// Pokémon, and `UseBagItem`'s `.returnAfterCapturingMon` zeroes it again on the
// way out of the battle -- so it is non-zero for the hundreds of frames the
// catch's own text and Pokédex screen take, and zero everywhere else.
//
// Read rather than derived from `wPartyCount`, because a catch with a full party
// raises `wBoxCount` instead, and because `wPartyCount` also rises for a gift, a
// trade and a revive out of the PC.
let captured = memory.read8(ram::wCapturedMonSpecies);
if let Some(battle) = &mut self.battle { if let Some(battle) = &mut self.battle {
let hp = word(memory, ram::wEnemyMonHP); let hp = word(memory, ram::wEnemyMonHP);
let max = word(memory, ram::wEnemyMonMaxHP); let max = word(memory, ram::wEnemyMonMaxHP);
@ -710,6 +790,11 @@ impl PokemonRedReward {
if battle.saw_living && hp == 0 { if battle.saw_living && hp == 0 {
battle.ko = true; battle.ko = true;
} }
if battle.wild && captured != 0 && battle.captured.is_none() {
battle.captured = Some(captured);
battle.captured_new =
battle.species_at_start.is_some_and(|before| species_paid > before);
}
} }
} else if in_battle == 0 { } else if in_battle == 0 {
self.mode = "OVERWORLD".to_string(); self.mode = "OVERWORLD".to_string();
@ -728,6 +813,41 @@ impl PokemonRedReward {
} }
self.wild_wins.insert(battle.key.clone(), (count + 1).min(3)); self.wild_wins.insert(battle.key.clone(), (count + 1).min(3));
} }
// The catch rule (`docs/rewards-learning.md`, the operator 2026-09-22).
//
// Paid on the way out of the battle rather than on the capture frame, so that
// it lands in the same place the wild-KO payout does and cannot fire twice for
// one battle. `wBattleResult` is 2 on exactly two paths in the game:
// `UseBagItem`'s `.returnAfterCapturingMon`, which is this one, and a link
// battle whose opponent ran (`engine/battle/core.asm`), which this cartridge
// never has. Requiring it as well as the captured species means a byte read
// out of a half-initialised battle cannot pay.
if let Some(species) = battle.captured
&& battle.wild
&& result == 2
{
let key = species.to_string();
let paid = self.catch_counts.get(&key).copied().unwrap_or(0);
if paid < MAX_CATCH_PAYOUTS
&& !self.replay_blocked.contains(&format!("catch:{key}"))
{
let value = if battle.captured_new {
catalog::rule(kind::CATCH)
.expect("the catch rule is in the catalog")
.value
} else {
catalog::CATCH_REPEAT_VALUE
};
self.emit_amount(
&mut emitted,
kind::CATCH,
format!("CAUGHT #{species}"),
value,
brain_ms,
);
}
self.catch_counts.insert(key, (paid + 1).min(MAX_CATCH_PAYOUTS));
}
} }
let location = format!("{map}:{x}:{y}"); let location = format!("{map}:{x}:{y}");
self.stable = if self.location == location { self.stable + 1 } else { 1 }; self.stable = if self.location == location { self.stable + 1 } else { 1 };
@ -825,13 +945,33 @@ impl PokemonRedReward {
label: String, label: String,
scale: f64, scale: f64,
brain_ms: f64, brain_ms: f64,
) {
let rule = catalog::rule(kind).expect("emit is only called with catalog kinds");
self.emit_amount(emitted, kind, label, rule.value * scale, brain_ms);
}
/// [`PokemonRedReward::emit`] with the payout stated outright instead of as a multiple
/// of the catalog value.
///
/// One rule needs it. `catch` pays 0.30 for a species this run has not caught and 0.10
/// for one it has, and no binary float scales the first into exactly the second:
/// `0.3 * (1.0 / 3.0)` is `0.09999999999999999`, and that is the number that would reach
/// the ticker and the checkpoint. `boundary`'s pair, 0.05 and 0.10, *is* an exact scale
/// of two, so that rule still goes through [`PokemonRedReward::emit`].
fn emit_amount(
&mut self,
emitted: &mut Vec<RewardEvent>,
kind: &'static str,
label: String,
value: f64,
brain_ms: f64,
) { ) {
let rule = catalog::rule(kind).expect("emit is only called with catalog kinds"); let rule = catalog::rule(kind).expect("emit is only called with catalog kinds");
let event = RewardEvent { let event = RewardEvent {
kind, kind,
label, label,
brain_ms, brain_ms,
value: rule.value * scale, value,
stimulation_ms: rule.stimulation_ms, stimulation_ms: rule.stimulation_ms,
}; };
emitted.push(event.clone()); emitted.push(event.clone());
@ -991,6 +1131,10 @@ impl PokemonRedReward {
"tiles": self.tiles.as_slice(), "tiles": self.tiles.as_slice(),
"tileCounts": self.tile_counts, "tileCounts": self.tile_counts,
"wildWins": self.wild_wins, "wildWins": self.wild_wins,
// The one field v6 adds. A v5 state does not carry it and restores with it
// empty, which is the documented v5 -> v6 migration and the truth about a run
// that was never paid for a catch.
"catchCounts": self.catch_counts,
"replayBlocked": self.replay_blocked.as_slice(), "replayBlocked": self.replay_blocked.as_slice(),
"counts": self.counts, "counts": self.counts,
"total": self.total, "total": self.total,
@ -1011,6 +1155,9 @@ impl PokemonRedReward {
"wild": battle.wild, "wild": battle.wild,
"sawLiving": battle.saw_living, "sawLiving": battle.saw_living,
"ko": battle.ko, "ko": battle.ko,
"speciesAtStart": battle.species_at_start,
"captured": battle.captured,
"capturedNew": battle.captured_new,
})), })),
"mode": self.mode, "mode": self.mode,
}) })
@ -1056,6 +1203,13 @@ impl PokemonRedReward {
let counts_raw = counted_record(input.get("counts")).ok_or(BAD_CHECKPOINT)?; let counts_raw = counted_record(input.get("counts")).ok_or(BAD_CHECKPOINT)?;
let tile_counts_raw = counted_record(input.get("tileCounts")).ok_or(BAD_CHECKPOINT)?; let tile_counts_raw = counted_record(input.get("tileCounts")).ok_or(BAD_CHECKPOINT)?;
let wild_wins_raw = counted_record(input.get("wildWins")).ok_or(BAD_CHECKPOINT)?; let wild_wins_raw = counted_record(input.get("wildWins")).ok_or(BAD_CHECKPOINT)?;
// Absent in every v5 state, and that absence is the migration: no species has been
// paid for a catch, because the rule did not exist. Present but malformed is still
// an error, the same as every other counter here.
let catch_counts = match input.get("catchCounts") {
None | Some(Value::Null) => BTreeMap::new(),
Some(value) => counted_record(Some(value)).ok_or(BAD_CHECKPOINT)?,
};
let recent = recent_raw let recent = recent_raw
.iter() .iter()
@ -1078,6 +1232,19 @@ impl PokemonRedReward {
wild: value.get("wild").and_then(Value::as_bool).ok_or(BAD_HISTORY)?, wild: value.get("wild").and_then(Value::as_bool).ok_or(BAD_HISTORY)?,
saw_living: value.get("sawLiving").and_then(Value::as_bool).ok_or(BAD_HISTORY)?, saw_living: value.get("sawLiving").and_then(Value::as_bool).ok_or(BAD_HISTORY)?,
ko: value.get("ko").and_then(Value::as_bool).ok_or(BAD_HISTORY)?, ko: value.get("ko").and_then(Value::as_bool).ok_or(BAD_HISTORY)?,
// All three are v6's, and all three are optional for the same reason
// `catchCounts` is. A v5 battle carries no `speciesAtStart`, which reads as
// "cannot tell whether the caught species was new" and pays the repeat
// amount: the conservative half of the rule, and at most 0.20 once.
species_at_start: value.get("speciesAtStart").and_then(Value::as_u64),
captured: value
.get("captured")
.and_then(Value::as_u64)
.and_then(|species| u8::try_from(species).ok()),
captured_new: value
.get("capturedNew")
.and_then(Value::as_bool)
.unwrap_or(false),
}), }),
}; };
let replay_blocked = match input.get("replayBlocked") { let replay_blocked = match input.get("replayBlocked") {
@ -1103,6 +1270,7 @@ impl PokemonRedReward {
} }
self.tile_counts = tile_counts; self.tile_counts = tile_counts;
self.wild_wins = wild_wins_raw; self.wild_wins = wild_wins_raw;
self.catch_counts = catch_counts;
self.replay_blocked = replay_blocked.iter().map(String::as_str).collect(); self.replay_blocked = replay_blocked.iter().map(String::as_str).collect();
self.counts = Counts::default(); self.counts = Counts::default();
for (key, count) in &counts_raw { for (key, count) in &counts_raw {
@ -1174,6 +1342,10 @@ impl GameAdapter for PokemonRedReward {
REWARD_ADAPTER REWARD_ADAPTER
} }
fn migrates_from(&self) -> &'static [&'static str] {
MIGRATES_FROM
}
fn rom_allowed(&self, sha256: &str) -> bool { fn rom_allowed(&self, sha256: &str) -> bool {
sha256 == SUPPORTED_ROM sha256 == SUPPORTED_ROM
} }

View file

@ -47,6 +47,7 @@ pub mod ram {
pub const wBattleType: u16 = 0xd05a; // 53338 pub const wBattleType: u16 = 0xd05a; // 53338
pub const wTrainerNo: u16 = 0xd05d; // 53341 pub const wTrainerNo: u16 = 0xd05d; // 53341
pub const wPartyMenuTypeOrMessageID: u16 = 0xd07d; // 53373 pub const wPartyMenuTypeOrMessageID: u16 = 0xd07d; // 53373
pub const wCapturedMonSpecies: u16 = 0xd11c; // 53532
pub const wForcePlayerToChooseMon: u16 = 0xd11f; // 53535 pub const wForcePlayerToChooseMon: u16 = 0xd11f; // 53535
pub const wTextBoxID: u16 = 0xd125; // 53541 pub const wTextBoxID: u16 = 0xd125; // 53541
pub const wPartyCount: u16 = 0xd163; // 53603 pub const wPartyCount: u16 = 0xd163; // 53603

View file

@ -83,6 +83,34 @@ impl Fixture {
self.visit(x, 0) self.visit(x, 0)
} }
/// One wild battle that ends in a ball keeping the Pokémon, byte for byte as the
/// cartridge writes it at the pinned commit.
///
/// `InitBattleVariables` clears `wBattleResult`; `ItemUseBall`'s capture branch sets the
/// Pokédex bit (for a species the player did not already own) and writes
/// `wEnemyMonSpecies` into `wCapturedMonSpecies`; `UseBagItem`'s
/// `.returnAfterCapturingMon` then zeroes that byte, sets `wBattleResult` to 2 and leaves
/// the battle. `dex` is the Pokédex *number* minus one, i.e. the bit index, and `None` is a
/// species this run already owns.
fn catch(&mut self, species: u8, dex: Option<u16>) -> Vec<RewardEvent> {
self.memory.set(ram::wBattleResult, 0);
self.memory.set(ram::wIsInBattle, 1);
self.memory.set(ram::wEnemyMonSpecies, species);
self.memory.set(ram::wEnemyMonHP + 1, 10);
self.memory.set(ram::wEnemyMonMaxHP + 1, 10);
let mut events = self.sample();
if let Some(index) = dex {
self.memory.or(ram::wPokedexOwned + (index >> 3), 1 << (index & 7));
}
self.memory.set(ram::wCapturedMonSpecies, species);
events.extend(self.sample());
self.memory.set(ram::wCapturedMonSpecies, 0);
self.memory.set(ram::wBattleResult, 2);
self.memory.set(ram::wIsInBattle, 0);
events.extend(self.sample());
events
}
/// Write a warp table: `wNumberOfWarps` plus one four-byte `Y, X, warp id, map id` entry per /// Write a warp table: `wNumberOfWarps` plus one four-byte `Y, X, warp id, map id` entry per
/// `(x, y)`, the layout `ram/wram.asm` documents at the pinned commit. /// `(x, y)`, the layout `ram/wram.asm` documents at the pinned commit.
fn warps(&mut self, warps: &[(u8, u8)]) { fn warps(&mut self, warps: &[(u8, u8)]) {
@ -320,6 +348,136 @@ fn a_wild_run_capture_or_single_faint_never_pays_a_ko_while_a_verified_ko_does()
assert_eq!(f.reward.statistics().counts[kind::BATTLE], 1); assert_eq!(f.reward.statistics().counts[kind::BATTLE], 1);
} }
#[test]
fn a_catch_pays_the_new_species_amount_once_the_repeat_amount_after_and_stops_at_three() {
let mut f = Fixture::new();
f.sample();
// A species this run has never owned: the cartridge sets the Pokédex bit on the way
// through, so the existing `species` rule pays 0.50 and the new rule pays 0.30.
let first = f.catch(0xb0, Some(3));
assert_eq!(kinds(&first), ["species", "catch"]);
assert_eq!(labels(&first), ["OWNED #4", "CAUGHT #176"]);
assert!((first[0].value - 0.5).abs() < 1e-12, "the species rule is untouched");
assert!((first[1].value - 0.30).abs() < 1e-12);
// The same species again: a repeat, twice, and then the cap.
for _ in 0..2 {
let again = f.catch(0xb0, None);
assert_eq!(kinds(&again), ["catch"]);
assert_eq!(again[0].value, 0.10, "the repeat amount is exactly 0.10, not 0.3/3");
}
assert!(f.catch(0xb0, None).is_empty(), "three payouts per species is the cap");
assert_eq!(f.reward.statistics().counts[kind::CATCH], 3);
// Another species starts its own count, and its own 0.30.
let other = f.catch(0x99, Some(0));
assert_eq!(kinds(&other), ["species", "catch"]);
assert!((other[1].value - 0.30).abs() < 1e-12);
}
#[test]
fn a_catch_of_a_species_this_run_already_owns_pays_the_repeat_amount() {
let mut f = Fixture::new();
f.sample();
// Owned before the battle -- a gift, a trade, an evolution -- so no Pokédex bit is set
// during it and the catch is not a new species.
f.memory.or(ram::wPokedexOwned, 1);
assert_eq!(kinds(&f.sample()), ["species"]);
let events = f.catch(0x99, None);
assert_eq!(kinds(&events), ["catch"]);
assert_eq!(events[0].value, 0.10);
}
#[test]
fn nothing_but_a_wild_catch_pays_the_catch_rule() {
// A trainer battle: balls cannot be thrown, and `wIsInBattle` is 2.
let mut f = Fixture::new();
f.sample();
f.memory.set(ram::wIsInBattle, 2);
f.memory.set(ram::wEnemyMonHP + 1, 10);
f.memory.set(ram::wEnemyMonMaxHP + 1, 10);
f.sample();
f.memory.set(ram::wCapturedMonSpecies, 0xb0);
f.sample();
f.memory.set(ram::wCapturedMonSpecies, 0);
f.memory.set(ram::wBattleResult, 2);
f.memory.set(ram::wIsInBattle, 0);
assert!(f.sample().is_empty(), "a trainer battle never pays the catch rule");
// The Safari Zone and the old man's tutorial are excluded a step earlier: the whole
// sample is dropped with a visible mode, so no battle is ever opened.
for battle_type in [1u8, 2] {
let mut f = Fixture::new();
f.sample();
f.memory.set(ram::wBattleType, battle_type);
f.memory.set(ram::wIsInBattle, 1);
assert!(f.sample().is_empty());
f.memory.set(ram::wCapturedMonSpecies, 0xb0);
assert!(f.sample().is_empty());
f.memory.set(ram::wCapturedMonSpecies, 0);
f.memory.set(ram::wBattleResult, 2);
f.memory.set(ram::wIsInBattle, 0);
f.memory.set(ram::wBattleType, 0);
assert!(f.sample().is_empty());
assert_eq!(f.reward.statistics().counts[kind::CATCH], 0);
}
// A ball that missed: `wCapturedMonSpecies` never leaves zero and the battle ends as a
// run or a loss.
let mut f = Fixture::new();
f.sample();
f.memory.set(ram::wIsInBattle, 1);
f.memory.set(ram::wEnemyMonHP + 1, 10);
f.memory.set(ram::wEnemyMonMaxHP + 1, 10);
f.sample();
f.memory.set(ram::wIsInBattle, 0);
assert!(f.sample().is_empty());
}
#[test]
fn a_rollback_cannot_replay_a_catch() {
let mut f = Fixture::new();
f.sample();
assert_eq!(kinds(&f.catch(0xb0, Some(3))), ["species", "catch"]);
f.reward.clear_transient();
let state = f.reward.export_state();
f.reward.import_state(&state).unwrap();
assert!(f.catch(0xb0, None).is_empty(), "an already-paid species cannot pay after rollback");
assert_eq!(f.reward.statistics().counts[kind::CATCH], 1);
}
#[test]
fn a_v5_state_restores_under_v6_with_the_catch_counter_at_zero() {
let mut f = Fixture::new();
f.sample();
f.catch(0xb0, Some(3));
let v6 = f.reward.export_state();
assert_eq!(v6["catchCounts"], json!({ "176": 1 }));
// The v5 shape is this one without the counter the rule added: same `version`, same field
// names, same meanings. That is the whole of the documented migration.
let mut v5 = v6.clone();
v5.as_object_mut().unwrap().remove("catchCounts");
assert_eq!(v5["version"], json!(STATE_VERSION), "v5 and v6 states share a schema version");
let mut restored = PokemonRedReward::new();
restored.import_state(&v5).unwrap();
let mut expected = v6.clone();
expected["catchCounts"] = json!({});
assert_eq!(restored.export_state(), expected, "the counter starts at 0, nothing else moves");
// A genuine v5 `counts` object carries eight kinds and no `catch`, which reads as zero.
let mut older = v5.clone();
older["counts"].as_object_mut().unwrap().remove("catch");
let mut restored = PokemonRedReward::new();
restored.import_state(&older).unwrap();
assert_eq!(restored.statistics().counts[kind::CATCH], 0);
assert_eq!(restored.statistics().counts[kind::SPECIES], 1);
}
#[test] #[test]
fn a_repeated_wild_ko_decays_then_stops() { fn a_repeated_wild_ko_decays_then_stops() {
let mut f = Fixture::new(); let mut f = Fixture::new();
@ -378,6 +536,7 @@ fn malformed_checkpoint_fields_are_named_in_the_error() {
("counts", json!([])), ("counts", json!([])),
("tileCounts", json!("not a record")), ("tileCounts", json!("not a record")),
("wildWins", json!(3)), ("wildWins", json!(3)),
("catchCounts", json!(3)),
] { ] {
let mut broken = good.clone(); let mut broken = good.clone();
broken[field] = wrong; broken[field] = wrong;
@ -706,7 +865,8 @@ fn the_recent_ticker_keeps_the_newest_eight_events_newest_first() {
#[test] #[test]
fn the_adapter_reports_its_identity_and_pinned_rom() { fn the_adapter_reports_its_identity_and_pinned_rom() {
let reward = PokemonRedReward::new(); let reward = PokemonRedReward::new();
assert_eq!(reward.id(), "pokered-unique8-v5"); assert_eq!(reward.id(), "pokered-unique8-v6");
assert_eq!(reward.migrates_from(), ["pokered-unique8-v5"]);
assert!(reward.rom_allowed(SUPPORTED_ROM)); assert!(reward.rom_allowed(SUPPORTED_ROM));
assert!(!reward.rom_allowed( assert!(!reward.rom_allowed(
"5ca7ba01642a3b27b0cc0b5349b52792795b62d3ed977e98a09390659af96b7b" "5ca7ba01642a3b27b0cc0b5349b52792795b62d3ed977e98a09390659af96b7b"
@ -716,6 +876,9 @@ fn the_adapter_reports_its_identity_and_pinned_rom() {
assert_eq!(symbols::ram::wNumberOfWarps, 0xd3ae); assert_eq!(symbols::ram::wNumberOfWarps, 0xd3ae);
assert_eq!(symbols::ram::wWarpEntries, 0xd3af); assert_eq!(symbols::ram::wWarpEntries, 0xd3af);
assert_eq!(symbols::ram::wCurMapConnections, 0xd370); assert_eq!(symbols::ram::wCurMapConnections, 0xd370);
// Resolved from ram/wram.asm by services/flysim/tools/resolve_wram.py, bracketed by
// wFontLoaded and wForcePlayerToChooseMon; never written out by hand.
assert_eq!(symbols::ram::wCapturedMonSpecies, 0xd11c);
assert_eq!(symbols::MILESTONES.len(), 17); assert_eq!(symbols::MILESTONES.len(), 17);
} }

View file

@ -99,8 +99,17 @@ pub async fn run() -> Result<Vec<String>, Box<dyn std::error::Error>> {
"published sequence {} to {} subscriber(s)", "published sequence {} to {} subscriber(s)",
receipt.topic_sequence, receipt.subscribers receipt.topic_sequence, receipt.subscribers
)); ));
// The producer lets go of its own hold; the delivery keeps the bytes alive. // The producer lets go of its own hold; the delivery keeps the bytes alive. The release
// travels the control lane like any other operation, so the count below waits for it
// instead of reading a number that may still include it.
drop(frame); drop(frame);
let released = Instant::now() + Duration::from_secs(10);
while router.stats().artifact_roots > 1 {
if Instant::now() > released {
return Err("the producer's own hold was never released".into());
}
tokio::time::sleep(Duration::from_millis(1)).await;
}
let message = frames.next().await.ok_or("the subscription closed")?; let message = frames.next().await.ok_or("the subscription closed")?;
let image = message.artifact("frame")?; let image = message.artifact("frame")?;

View file

@ -179,7 +179,9 @@ async fn collection_waits_for_every_retained_owner(via: Via) {
s.owners == 0 && s.sealed_artifacts == 0 && s.store_bytes == 0 s.owners == 0 && s.sealed_artifacts == 0 && s.store_bytes == 0
}) })
.await; .await;
assert_eq!(e.files("sealed"), 0); // The unlink follows the registry update, outside the router lock: wait for the file to
// go rather than assume the two happen together.
e.settle_files("sealed", 0).await;
} }
// ------------------------------------------------------------------------------------------- // -------------------------------------------------------------------------------------------
@ -521,7 +523,7 @@ async fn disconnect_abandons_an_unsealed_writer(via: Via) {
s.artifacts == 0 && s.store_bytes == 0 && s.owners == 0 s.artifacts == 0 && s.store_bytes == 0 && s.owners == 0
}) })
.await; .await;
assert_eq!(e.files("staging"), 0); e.settle_files("staging", 0).await;
} }
/// An abrupt disconnect must release an explicit hold too, when it was the object's only root. /// An abrupt disconnect must release an explicit hold too, when it was the object's only root.
@ -538,7 +540,7 @@ async fn disconnect_releases_an_explicit_hold(via: Via) {
s.sealed_artifacts == 0 && s.owners == 0 && s.store_bytes == 0 s.sealed_artifacts == 0 && s.owners == 0 && s.store_bytes == 0
}) })
.await; .await;
assert_eq!(e.files("sealed"), 0); e.settle_files("sealed", 0).await;
} }
/// A vanished subscriber must give up both a delivery it already holds and one still queued /// A vanished subscriber must give up both a delivery it already holds and one still queued

View file

@ -420,14 +420,20 @@ async fn bounded_overflow_rolls_back_all_artifact_roots(via: Via) {
} }
/// bus-v1 section 7: "New subscriptions with replayLatest enqueue it before subsequent accepted /// bus-v1 section 7: "New subscriptions with replayLatest enqueue it before subsequent accepted
/// publications." A fresh `latest` subscription's replay claims its first in-flight credit /// publications ... bounded mode preserves that order, while latest mode may coalesce it before
/// immediately (there is nothing else competing for it yet), so a publish accepted right after /// delivery under the ordinary latest rule." The replay is enqueued under the subscribe lock, so
/// subscribing must still be observed strictly after the replay, never ahead of or merged with /// a publication admitted after `subscribe` returned is always behind it; what the two modes do
/// it: each keeps its own delivery. /// with that order is what differs, and one racing publication is put to both at once.
///
/// The bounded subscription must deliver both values, replay first. The latest subscription
/// either does the same or replaces the still-queued replay, and the router says which in the
/// racing publication's own `replaced` count rather than the test guessing from how fast the
/// dispatcher ran: what it may never do is reorder the two or lose the newer value.
async fn latest_replay_is_ordered_ahead_of_a_racing_publish(via: Via) { async fn latest_replay_is_ordered_ahead_of_a_racing_publish(via: Via) {
let e = env(via).await; let e = env(via).await;
let admin = e.client("admin").await; let admin = e.client("admin").await;
let reader = e.client("reader").await; let reader = e.client("reader").await;
let viewer = e.client("viewer").await;
admin admin
.declare_topic("t.replay-race", Retained::Latest) .declare_topic("t.replay-race", Retained::Latest)
.await .await
@ -436,26 +442,66 @@ async fn latest_replay_is_ordered_ahead_of_a_racing_publish(via: Via) {
.publish("t.replay-race", obj(json!({"v": "old"})), &[]) .publish("t.replay-race", obj(json!({"v": "old"})), &[])
.await .await
.unwrap(); .unwrap();
let mut sub = reader let mut fifo = reader
.subscribe(
"t.replay-race",
SubscriptionConfig::bounded().in_flight(1).replay(true),
)
.await
.unwrap();
let mut coalescing = viewer
.subscribe( .subscribe(
"t.replay-race", "t.replay-race",
SubscriptionConfig::latest().in_flight(1).replay(true), SubscriptionConfig::latest().in_flight(1).replay(true),
) )
.await .await
.unwrap(); .unwrap();
admin let racing = admin
.publish("t.replay-race", obj(json!({"v": "new"})), &[]) .publish("t.replay-race", obj(json!({"v": "new"})), &[])
.await .await
.unwrap(); .unwrap();
let first = within("the replay arrives first", sub.next()) assert_eq!(
racing.subscribers, 2,
"one publication, admitted behind both replays"
);
let first = within("the replay arrives first", fifo.next())
.await .await
.unwrap(); .unwrap();
assert_eq!(first.payload()["v"], "old"); assert_eq!(first.payload()["v"], "old");
drop(first); // the sole in-flight credit must return before the queued second value moves drop(first); // the sole in-flight credit must return before the queued second value moves
let second = within("the racing publish follows, not coalesced away", sub.next()) let second = within("the racing publish follows, not coalesced away", fifo.next())
.await .await
.unwrap(); .unwrap();
assert_eq!(second.payload()["v"], "new"); assert_eq!(
(second.payload()["v"].as_str(), second.replaced()),
(Some("new"), 0),
"bounded preserves the order and coalesces nothing"
);
let m = within("the latest subscription's first delivery", coalescing.next())
.await
.unwrap();
if racing.replaced == 1 {
assert_eq!(
(m.payload()["v"].as_str(), m.replaced()),
(Some("new"), 1),
"a replay still queued is replaced by the newer value, and the delivery says so"
);
drop(m);
quiet("nothing behind a coalesced replay", coalescing.next()).await;
} else {
assert_eq!(
(racing.replaced, m.payload()["v"].as_str(), m.replaced()),
(0, Some("old"), 0),
"a replay already in flight keeps its own delivery"
);
drop(m);
let after = within("the racing publish follows it", coalescing.next())
.await
.unwrap();
assert_eq!(after.payload()["v"], "new");
}
} }
/// bus-v1 section 7: clearing releases only the retained root; a later `replayLatest` /// bus-v1 section 7: clearing releases only the retained root; a later `replayLatest`

View file

@ -1,6 +1,11 @@
//! bus-v1 section 11 item 6: two parallel fake agents, complete-batch environment RPC, //! bus-v1 section 11 item 6: two parallel fake agents, complete-batch environment RPC,
//! committed snapshot publication and a deliberately slow presentation consumer, all over one //! committed snapshot publication and a deliberately slow presentation consumer, all over one
//! router. Generic services only; nothing here knows what a brain or a game is. //! router. Generic services only; nothing here knows what a brain or a game is.
//!
//! The presentation consumer is held until the publisher's own completion is observed, so the
//! latest subscription has to coalesce instead of happening to: section 7 lets a latest
//! subscriber miss values, it does not oblige it to, and a test that demands a miss it cannot
//! force is asserting how fast the machine is.
mod common; mod common;
@ -148,20 +153,31 @@ async fn session_over_one_router(via: Via) {
) )
.await .await
.unwrap(); .unwrap();
// A renderer that does not read a single snapshot until the publisher has finished every
// one of them. The hold ends on the publisher's observed completion, never on a timer.
let (release, held) = tokio::sync::oneshot::channel::<()>();
let presenting = tokio::spawn(async move { let presenting = tokio::spawn(async move {
held.await.unwrap();
let mut seen = Vec::new(); let mut seen = Vec::new();
let mut coalesced = 0;
while let Some(m) = slow.next().await { while let Some(m) = slow.next().await {
let (sequence, replaced) = (m.topic_sequence(), m.replaced());
let frame = m.artifact("frame").unwrap(); let frame = m.artifact("frame").unwrap();
drop(m); drop(m);
tokio::time::sleep(Duration::from_millis(25)).await; // a slow renderer tokio::time::sleep(Duration::from_millis(25)).await; // a slow renderer
let bytes = frame.read_all().await.unwrap(); let bytes = frame.read_all().await.unwrap();
let step = bytes[0] as u64; let step = bytes[0] as u64;
assert_eq!(
sequence, step,
"a delivery carries the frame of the snapshot it announces"
);
coalesced += replaced;
seen.push((step, frame.reference().artifact_id.clone())); seen.push((step, frame.reference().artifact_id.clone()));
if step == STEPS { if step == STEPS {
break; break;
} }
} }
seen (seen, coalesced)
}); });
let recorder = e.client("recorder").await; let recorder = e.client("recorder").await;
let mut all = recorder let mut all = recorder
@ -179,6 +195,7 @@ async fn session_over_one_router(via: Via) {
seq seq
}); });
let mut replaced_at_admission = 0;
for step in 1..=STEPS { for step in 1..=STEPS {
let advanced = coordinator let advanced = coordinator
.call_and_wait( .call_and_wait(
@ -226,8 +243,16 @@ async fn session_over_one_router(via: Via) {
) )
.await .await
.unwrap(); .unwrap();
assert_eq!(receipt.topic_sequence, step); assert_eq!(
(receipt.topic_sequence, receipt.subscribers),
(step, 2),
"both subscriptions accept every publication: the stalled latest spectator neither \
refuses one nor drops out of the fan-out"
);
replaced_at_admission += receipt.replaced;
} }
// The publisher is finished, observably, so the renderer may start.
release.send(()).unwrap();
let recorded = within("recorder", recording).await.unwrap(); let recorded = within("recorder", recording).await.unwrap();
assert_eq!( assert_eq!(
@ -235,17 +260,30 @@ async fn session_over_one_router(via: Via) {
(1..=STEPS).map(|s| (s, s)).collect::<Vec<_>>(), (1..=STEPS).map(|s| (s, s)).collect::<Vec<_>>(),
"the bounded recorder misses nothing" "the bounded recorder misses nothing"
); );
let presented = within("presenter", presenting).await.unwrap(); let (presented, coalesced) = within("presenter", presenting).await.unwrap();
assert_eq!( let steps: Vec<u64> = presented.iter().map(|(s, _)| *s).collect();
presented.last().unwrap().0,
STEPS,
"the slow consumer ends on the latest snapshot"
);
assert!( assert!(
presented.len() < STEPS as usize, steps.windows(2).all(|w| w[0] < w[1]),
"the slow consumer skipped snapshots: {presented:?}" "what a latest subscription does deliver arrives in publication order: {presented:?}"
);
assert_eq!(
steps.last().copied(),
Some(STEPS),
"the slow consumer ends on the latest snapshot: {presented:?}"
);
assert_eq!(
steps,
vec![1, STEPS],
"held for the whole run, the subscription keeps the one delivery already in flight and \
one replaceable queued value, so the renderer sees the first snapshot and the last, \
and the eighteen between them were coalesced: {presented:?}"
);
assert_eq!(
(coalesced, replaced_at_admission),
(STEPS - 2, STEPS - 2),
"every snapshot the renderer missed is counted as a replacement, to the publisher at \
admission and to the renderer on its next delivery: none is lost silently"
); );
assert!(presented.windows(2).all(|w| w[0].0 < w[1].0));
for t in agents { for t in agents {
t.abort(); t.abort();

View file

@ -266,13 +266,14 @@ async fn pending_connections_are_bounded_and_hello_expires() {
let router = Router::new(config).unwrap(); let router = Router::new(config).unwrap();
let pending = router.connect_in_memory_as("first"); let pending = router.connect_in_memory_as("first");
tokio::time::timeout(Duration::from_millis(20), async { // Registration happens on the router's own task. How long that takes is this box's
// business; that it happens is the router's.
within("the pending connection occupies the only slot", async {
while router.stats().connections != 1 { while router.stats().connections != 1 {
tokio::task::yield_now().await; tokio::time::sleep(Duration::from_millis(1)).await;
} }
}) })
.await .await;
.unwrap();
assert_eq!(router.stats().connections, 1); assert_eq!(router.stats().connections, 1);
let refused = Client::connect( let refused = Client::connect(
router.connect_in_memory_as("second"), router.connect_in_memory_as("second"),
@ -280,7 +281,14 @@ async fn pending_connections_are_bounded_and_hello_expires() {
) )
.await; .await;
assert_eq!(refused.unwrap_err().code, ErrorCode::RouterLost); assert_eq!(refused.unwrap_err().code, ErrorCode::RouterLost);
tokio::time::sleep(Duration::from_millis(80)).await; // The 40 ms hello timeout expires on the router's clock: wait for the expiry to be
// observed rather than sleep past it and read the count once.
within("the pending Hello expires", async {
while router.stats().connections != 0 {
tokio::time::sleep(Duration::from_millis(1)).await;
}
})
.await;
assert_eq!(router.stats().connections, 0, "pending Hello timed out"); assert_eq!(router.stats().connections, 0, "pending Hello timed out");
drop(pending); drop(pending);

View file

@ -30,6 +30,7 @@ pub mod metrics;
pub mod pacing; pub mod pacing;
pub mod profile; pub mod profile;
pub mod ratelimit; pub mod ratelimit;
pub mod reset;
pub mod sdnotify; pub mod sdnotify;
pub mod simloop; pub mod simloop;
pub mod snapshot; pub mod snapshot;

View file

@ -42,6 +42,15 @@ struct Args {
/// the "nothing to compare" case for a fresh container. /// the "nothing to compare" case for a fresh container.
#[arg(long, value_name = "DIR")] #[arg(long, value_name = "DIR")]
print_state_compatibility: Option<std::path::PathBuf>, print_state_compatibility: Option<std::path::PathBuf>,
/// Restart the run from the milestone archive for this ladder rung, and exit.
///
/// Run with flysim stopped: it rewrites both checkpoint stores.
/// `infra/bin/fly-reset-to-milestone` is the operator-facing wrapper and the sequence
/// around it is in `infra/docs/runbook.md`. The current state is copied to a dated
/// directory first, so this is reversible by hand.
#[arg(long, value_name = "RANK")]
reset_to_milestone: Option<u32>,
} }
fn main() -> Result<()> { fn main() -> Result<()> {
@ -66,6 +75,17 @@ fn main() -> Result<()> {
return Ok(()); return Ok(());
} }
if let Some(rank) = args.reset_to_milestone {
let stamp = flysim::reset::utc_stamp(flysim::eventlog::now_wall_ms());
let durable = config.paths.save_dir.clone();
let hot = config.paths.hot_dir.clone();
let archive = flysim::reset::default_archive_dir(&durable, &stamp);
for line in flysim::reset::reset_to_milestone(&durable, &hot, rank, &archive)? {
println!("{line}");
}
return Ok(());
}
flysim::run(config) flysim::run(config)
} }

View file

@ -0,0 +1,450 @@
//! Restarting a run from an earlier rung, on disk, with flysim stopped.
//!
//! The operator's decision of 2026-09-22 was "restart the live run from an early checkpoint
//! instead of from scratch". `FLY_RESET_STATE=1` cannot do that: it archives everything and the
//! next start warms up a fresh fly. What this does instead is promote one milestone archive --
//! `milestone-<N>.checkpoint`, which `store::Store::commit` writes at the first commit at a new
//! best rank and which no rotation ever unlinks -- to being the only thing either store will
//! restore.
//!
//! `infra/bin/fly-reset-to-milestone` is the operator-facing wrapper; it refuses to run while
//! flysim is up, calls `flysim --reset-to-milestone N`, and fixes ownership afterwards. The work
//! is here rather than in that script because two of the steps are inside the `FLYSIM01`
//! envelope: the ratchet's attempts and recoveries counters live in the checkpoint's manifest,
//! and a shell script has no business rewriting one.
//!
//! What it does, in order, and nothing else:
//!
//! 1. **archives** every file in the durable and hot stores into a dated directory, by copying,
//! so a step that fails later has destroyed nothing;
//! 2. **rewrites** the rung's archive with the ratchet's `attempts` and `recoveries` at zero, so
//! the recovery budget is not already spent when the restarted run begins. `best` is left
//! alone: the archive's own `best` is the rung it was taken at, which is exactly what the
//! restarted run is at, and the rank the stream shows is recomputed by the adapter from the
//! restored game state anyway;
//! 3. **installs** it as the newest generation in both stores, so the restore order
//! (`store::restore_order`: hot latest, hot previous, durable latest, ...) reaches it first;
//! 4. **clears** the milestone archives above N -- rungs the run had reached and is now below --
//! and the generation files of the run being abandoned;
//! 5. **clears the session ledgers**: the event log `events.jsonl` and its rotations. The
//! checkpoint carries `lastEventId`, so restoring an old checkpoint over a newer log would
//! re-issue ids the log already holds. The macro layer's own session ledgers (blocked,
//! talked, reached, pushed-back) are memory-only by contract
//! (`docs/design/macros.md` section 12.1: "a restored run offers every target once more"),
//! so stopping flysim is what resets those and this has nothing to do.
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use anyhow::{Context, Result, bail};
use crate::store::{self, Store, StoreManifest};
/// Generations kept by the stores this tool writes. Only used for the rotation bound, which
/// this tool does not trigger; the durable store's own value.
const KEEP_GENERATIONS: usize = 8;
/// A generation number no commit ever allocates, so `Store::candidates` drops the
/// generation-file half of an archive entry and offers only `milestone-<rank>.checkpoint`.
///
/// `Sim::boot` allocates `highest_generation() + 1`, which is 1 or more, so 0 names no file.
/// The milestone archives below the rung being restored are kept exactly this way: still on
/// disk, still restorable as a deeper fallback, and with no generation file pretending to be
/// their contents.
const NO_GENERATION: u64 = 0;
/// What the reset did, one line per step, for the operator's terminal and the run record.
pub type Report = Vec<String>;
/// Promote `rank`'s milestone archive to be the restore source of both stores.
///
/// `archive` is the dated directory the current state is copied into; it must not exist.
/// Refuses if the milestone archive is missing, which is the "that rung was never reached"
/// case and the one mistake worth refusing rather than guessing at.
pub fn reset_to_milestone(
durable_dir: &Path,
hot_dir: &Path,
rank: u32,
archive: &Path,
) -> Result<Report> {
let durable = Store::new(durable_dir, KEEP_GENERATIONS);
let hot = Store::new(hot_dir, KEEP_GENERATIONS);
let source = durable.archive_path(rank);
if !source.is_file() {
bail!(
"no milestone archive for rung {rank}: {} does not exist. `ls {}` shows the rungs \
this run actually reached.",
source.display(),
durable_dir.display()
);
}
if archive.exists() {
bail!("the archive directory {} already exists", archive.display());
}
let mut report: Report = Vec::new();
// 1. Copy everything aside first.
let copied_durable = copy_tree(durable_dir, &archive.join("durable"))?;
let copied_hot = copy_tree(hot_dir, &archive.join("hot"))?;
report.push(format!(
"archived {copied_durable} durable and {copied_hot} hot files to {}",
archive.display()
));
// 2. Zero the two recovery counters inside the envelope.
let mut checkpoint = store::load(&source)
.with_context(|| format!("decoding {}", source.display()))?;
let spent = (checkpoint.runtime.ratchet.attempts, checkpoint.runtime.ratchet.recoveries);
checkpoint.runtime.ratchet.attempts = 0;
checkpoint.runtime.ratchet.recoveries = 0;
let generation = durable.highest_generation().max(hot.highest_generation()) + 1;
checkpoint.runtime.generation = generation;
let bytes = store::encode(&checkpoint.agent, &checkpoint.runtime)?;
report.push(format!(
"rung {rank} (best {}, ladder rank recomputed from the game state): ratchet attempts \
{} -> 0, recoveries {} -> 0",
checkpoint.runtime.ratchet.best, spent.0, spent.1
));
// 3/4. Clear both stores, keeping the milestone archives at or below this rung, and write
// the promoted state as the newest generation of each.
let kept = clear_store(durable_dir, Some(rank))?;
clear_store(hot_dir, None)?;
report.push(format!(
"cleared the hot store and every milestone archive above rung {rank}; kept {} at or \
below it: {kept:?}",
kept.len()
));
// The hot store lives on a tmpfs that a stopped container may not have mounted yet, so it
// is created rather than assumed; the durable one already exists or the milestone archive
// above could not have been read.
durable.create()?;
hot.create()?;
store::write_atomic(&durable.generation_path(generation), &bytes)?;
store::write_atomic(&durable.archive_path(rank), &bytes)?;
store::write_atomic(&hot.generation_path(generation), &bytes)?;
let mut archives: BTreeMap<u32, u64> =
kept.iter().map(|rung| (*rung, NO_GENERATION)).collect();
archives.insert(rank, generation);
let durable_manifest = StoreManifest {
generation,
latest: Some(generation),
previous: None,
archives,
};
write_manifest(&durable, &durable_manifest)?;
write_manifest(
&hot,
&StoreManifest {
generation,
latest: Some(generation),
previous: None,
archives: BTreeMap::new(),
},
)?;
report.push(format!(
"generation {generation} is now hot latest and durable latest in {} and {}",
hot_dir.display(),
durable_dir.display()
));
// 5. The session ledgers.
let logs = remove_matching(durable_dir, |name| {
name == "events.jsonl" || (name.starts_with("events-") && name.ends_with(".jsonl"))
})?;
report.push(format!(
"reset the session ledgers: {logs} event-log files removed (the macro layer's are \
memory-only and are reset by stopping flysim)"
));
Ok(report)
}
fn write_manifest(store: &Store, manifest: &StoreManifest) -> Result<()> {
store.create()?;
store::write_atomic(&store.manifest_path(), &serde_json::to_vec_pretty(manifest)?)
}
/// Copy every regular file of `from` into `to`, creating `to`. A missing source is zero files,
/// not an error: the hot store lives on a tmpfs that a stopped container does not have.
fn copy_tree(from: &Path, to: &Path) -> Result<usize> {
if !from.is_dir() {
return Ok(0);
}
std::fs::create_dir_all(to)
.with_context(|| format!("creating {}", to.display()))?;
let mut copied = 0;
for entry in std::fs::read_dir(from)?.flatten() {
if !entry.file_type().is_ok_and(|kind| kind.is_file()) {
continue;
}
std::fs::copy(entry.path(), to.join(entry.file_name()))
.with_context(|| format!("copying {}", entry.path().display()))?;
copied += 1;
}
Ok(copied)
}
/// Remove every checkpoint, tmp file and manifest from `dir`, keeping `milestone-<r>.checkpoint`
/// for `r <= keep_up_to`. Returns the rungs kept, ascending.
fn clear_store(dir: &Path, keep_up_to: Option<u32>) -> Result<Vec<u32>> {
let mut kept = Vec::new();
if !dir.is_dir() {
return Ok(kept);
}
for entry in std::fs::read_dir(dir)?.flatten() {
let name = entry.file_name().to_string_lossy().to_string();
let milestone = name
.strip_prefix("milestone-")
.and_then(|rest| rest.strip_suffix(".checkpoint"))
.and_then(|rung| rung.parse::<u32>().ok());
let remove = match milestone {
// The promoted rung's own archive is rewritten straight after this, so it is
// removed here like the rest and reinstated with the counters cleared.
Some(rung) => match keep_up_to {
Some(limit) if rung < limit => {
kept.push(rung);
false
}
_ => true,
},
None => {
name == "manifest.json"
|| name.ends_with(".checkpoint")
|| name.ends_with(".checkpoint.tmp")
}
};
if remove {
std::fs::remove_file(entry.path())
.with_context(|| format!("removing {}", entry.path().display()))?;
}
}
kept.sort_unstable();
Ok(kept)
}
fn remove_matching(dir: &Path, wanted: impl Fn(&str) -> bool) -> Result<usize> {
if !dir.is_dir() {
return Ok(0);
}
let mut removed = 0;
for entry in std::fs::read_dir(dir)?.flatten() {
if wanted(&entry.file_name().to_string_lossy()) {
std::fs::remove_file(entry.path())?;
removed += 1;
}
}
Ok(removed)
}
/// `YYYYMMDDTHHMMSSZ` in UTC, for the dated archive directory's name.
///
/// Built on the event log's own calendar conversion, which is the service's only one; the time of
/// day is arithmetic on the same millisecond count.
pub fn utc_stamp(wall_ms: u64) -> String {
let second_of_day = (wall_ms % 86_400_000) / 1_000;
format!(
"{}T{:02}{:02}{:02}Z",
crate::eventlog::utc_day(wall_ms),
second_of_day / 3_600,
(second_of_day / 60) % 60,
second_of_day % 60,
)
}
/// The default dated archive directory: a sibling of the durable store, which is its own
/// mountpoint and so cannot be renamed -- the same shape `infra/05-deploy.sh` uses for
/// `FLY_RESET_STATE=1`.
pub fn default_archive_dir(durable_dir: &Path, stamp: &str) -> PathBuf {
let mut name = durable_dir.as_os_str().to_os_string();
name.push(format!(".reset-{stamp}"));
PathBuf::from(name)
}
#[cfg(test)]
mod tests {
use super::*;
/// A real `FLYSIM01` envelope, small but structurally complete, so these tests decode and
/// re-encode what the running service writes rather than a stand-in.
fn envelope(generation: u64, ratchet: flybrain_gb::RatchetState) -> Vec<u8> {
use flybrain_core::decoder::DecoderState;
use flybrain_core::lif::LifState;
use flybrain_core::ordered::NumberMap;
use flybrain_core::plasticity::PlasticityState;
let agent = flybrain_core::agent::AgentState {
version: 1,
remainder: 0.75,
warmed_up: true,
network: LifState {
membrane: vec![0.5, -0.25],
refractory: vec![0, 3],
last_spike_ms: vec![-1_000_000.0, 12.0],
visual_drive: vec![0.1],
rng: -12_345,
reward_remaining: 40.0,
ms: 9_000.0,
population_rate: 1.5,
rates: NumberMap::from_pairs([("forward", 2.0)]),
plasticity: PlasticityState {
version: "fly-kc-mbon-rstdp-v2".to_string(),
topology: 42,
enabled: true,
updates: 3.0,
signal: 0.25,
gains: vec![1.0, 0.9],
traces: vec![0.0, 0.1],
touched: vec![0.0, 8_000.0],
},
},
decoder: DecoderState {
version: 4,
calibrated: true,
baseline: NumberMap::from_pairs([("forward", 1.0)]),
held_until: NumberMap::new(),
next_allowed: NumberMap::new(),
next_decision: 100.0,
current: None,
fatigue: NumberMap::new(),
macro_next_decision: 0.0,
macro_current: None,
macro_fatigue: NumberMap::new(),
},
};
let runtime = store::RuntimeState {
generation,
wall_ms: 1_700_000_000_000,
rom_sha256: "ab".repeat(32),
emulator_frame: 12_345,
compatibility: "kernel/pokered-unique8-v6/fingerprint".to_string(),
speed: 1.0,
buttons: 0,
rank_since_ms: 4_242.0,
last_event_id: 77,
reward: serde_json::json!({ "version": 4, "total": 1.25 }),
ratchet,
emulator: vec![7; 64],
framebuffer: vec![9; 32],
ratchet_game: vec![1, 2, 3],
ratchet_frame: vec![4, 5, 6],
};
store::encode(&agent, &runtime).unwrap()
}
/// A store dir holding a milestone archive for each rung in `rungs`, their generations, a
/// manifest and an event log, all written through the store's own commit path.
fn state_dir(root: &Path, rungs: &[u32], ratchet: flybrain_gb::RatchetState) -> Store {
let store = Store::new(root, KEEP_GENERATIONS);
store.create().unwrap();
for (index, rung) in rungs.iter().enumerate() {
let generation = index as u64 + 1;
store.commit(generation, &envelope(generation, ratchet), Some(*rung)).unwrap();
}
std::fs::write(root.join("events.jsonl"), b"{}\n").unwrap();
std::fs::write(root.join("events-20260921.jsonl"), b"{}\n").unwrap();
store
}
#[test]
fn a_missing_rung_is_refused_and_nothing_is_touched() {
let tmp = tempfile::tempdir().unwrap();
let durable = tmp.path().join("state");
let hot = tmp.path().join("hot");
state_dir(&durable, &[3, 5], flybrain_gb::RatchetState::default());
let before = std::fs::read_dir(&durable).unwrap().flatten().count();
let error = reset_to_milestone(&durable, &hot, 9, &tmp.path().join("archive"))
.unwrap_err()
.to_string();
assert!(error.contains("no milestone archive for rung 9"), "{error}");
assert_eq!(std::fs::read_dir(&durable).unwrap().flatten().count(), before);
assert!(!tmp.path().join("archive").exists(), "nothing was archived");
}
#[test]
fn the_rung_becomes_both_stores_latest_with_the_recovery_budget_back() {
let tmp = tempfile::tempdir().unwrap();
let durable = tmp.path().join("state");
let hot = tmp.path().join("hot");
let spent = flybrain_gb::RatchetState {
best: 5,
attempts: 3,
recoveries: 11,
..flybrain_gb::RatchetState::default()
};
state_dir(&durable, &[3, 5, 9, 11], spent);
state_dir(&hot, &[11], spent);
let archive = tmp.path().join("archive-20260922");
let report = reset_to_milestone(&durable, &hot, 5, &archive).unwrap();
assert!(report.iter().any(|line| line.contains("attempts 3 -> 0")), "{report:?}");
// Everything that was there is in the archive.
assert!(archive.join("durable/milestone-11.checkpoint").is_file());
assert!(archive.join("durable/events.jsonl").is_file());
assert!(archive.join("hot/manifest.json").is_file());
// The rungs above 5 are gone; the ones below it stay as deeper fallbacks.
assert!(!durable.join("milestone-9.checkpoint").exists());
assert!(!durable.join("milestone-11.checkpoint").exists());
assert!(durable.join("milestone-3.checkpoint").is_file());
assert!(durable.join("milestone-5.checkpoint").is_file());
assert!(!durable.join("events.jsonl").exists());
assert!(!durable.join("events-20260921.jsonl").exists());
assert!(!hot.join("milestone-11.checkpoint").exists());
// Both stores restore the rung, and the counters are back.
for store in [Store::new(&hot, KEEP_GENERATIONS), Store::new(&durable, KEEP_GENERATIONS)] {
let candidates = store.candidates("x");
let first = store::load(&candidates[0].path).unwrap();
assert_eq!(first.runtime.ratchet.best, 5);
assert_eq!((first.runtime.ratchet.attempts, first.runtime.ratchet.recoveries), (0, 0));
}
// ... including through the promoted milestone archive itself.
let archived = store::load(&durable.join("milestone-5.checkpoint")).unwrap();
assert_eq!((archived.runtime.ratchet.attempts, archived.runtime.ratchet.recoveries), (0, 0));
// The rungs below it are offered, and only as their own archive files.
let manifest = Store::new(&durable, KEEP_GENERATIONS).manifest().unwrap();
assert_eq!(manifest.archives.get(&3), Some(&NO_GENERATION));
assert_eq!(manifest.latest, manifest.archives.get(&5).copied());
assert!(
Store::new(&durable, KEEP_GENERATIONS)
.candidates("durable")
.iter()
.any(|candidate| candidate.path.ends_with("milestone-3.checkpoint"))
);
}
#[test]
fn a_second_reset_refuses_to_write_over_an_existing_archive() {
let tmp = tempfile::tempdir().unwrap();
let durable = tmp.path().join("state");
let hot = tmp.path().join("hot");
state_dir(&durable, &[4], flybrain_gb::RatchetState::default());
let archive = tmp.path().join("archive");
reset_to_milestone(&durable, &hot, 4, &archive).unwrap();
let error = reset_to_milestone(&durable, &hot, 4, &archive).unwrap_err().to_string();
assert!(error.contains("already exists"), "{error}");
}
#[test]
fn the_stamp_is_the_event_logs_calendar_plus_a_time_of_day() {
assert_eq!(utc_stamp(0), "19700101T000000Z");
// 2026-09-22T16:15:00Z
assert_eq!(utc_stamp(1_790_093_700_000), "20260922T161500Z");
}
#[test]
fn the_default_archive_is_a_dated_sibling_of_the_store() {
assert_eq!(
default_archive_dir(Path::new("/srv/fly/state"), "20260922T161500Z"),
PathBuf::from("/srv/fly/state.reset-20260922T161500Z")
);
}
}

View file

@ -718,12 +718,36 @@ impl Sim {
if runtime.rom_sha256 != self.rom_sha256 { if runtime.rom_sha256 != self.rom_sha256 {
bail!("checkpoint is for another cartridge ({})", runtime.rom_sha256); bail!("checkpoint is for another cartridge ({})", runtime.rom_sha256);
} }
if runtime.compatibility != self.compatibility { // Byte-identical, or the one documented migration the operator asked for.
bail!( //
"compatibility mismatch\n checkpoint: {}\n this build: {}", // `FLY_ACCEPT_ADAPTERS` is read here rather than carried in `Config` because it is a
// property of a *deploy*, not of a run: `infra/05-deploy.sh` writes it into
// `/etc/fly/fly.env` only for the deploy that needs it, and an operator who wants the
// migration off again deletes one line. An empty or unset variable is no migration at
// all, which is what every deploy before this one did.
let accepted = flybrain_gb::compatibility::accepted_adapters(
std::env::var(flybrain_gb::compatibility::ACCEPT_ADAPTERS_ENV).ok().as_deref(),
);
match flybrain_gb::compatibility::decide(
&runtime.compatibility,
&self.compatibility,
self.adapter.migrates_from(),
&accepted,
) {
flybrain_gb::compatibility::RestoreDecision::Exact => {}
flybrain_gb::compatibility::RestoreDecision::MigrateAdapter { from } => {
tracing::warn!(
from = %from,
to = %self.adapter.id(),
"restoring a checkpoint from an earlier adapter, by the migration \
FLY_ACCEPT_ADAPTERS opted this deploy into"
);
}
flybrain_gb::compatibility::RestoreDecision::Refuse(reason) => bail!(
"compatibility mismatch: {reason}\n checkpoint: {}\n this build: {}",
runtime.compatibility, runtime.compatibility,
self.compatibility self.compatibility
); ),
} }
if runtime.framebuffer.len() != FRAMEBUFFER_LEN { if runtime.framebuffer.len() != FRAMEBUFFER_LEN {
bail!("checkpoint framebuffer is {} bytes", runtime.framebuffer.len()); bail!("checkpoint framebuffer is {} bytes", runtime.framebuffer.len());

View file

@ -241,8 +241,8 @@ pub struct FeedMacroOutcome {
} }
/// Reward categories the feed reports counts for. The adapter's own interned kinds /// Reward categories the feed reports counts for. The adapter's own interned kinds
/// (`milestone`, `exploration`, `map`, `species`, `trainer`, `battle`, `badge`, `boundary`) map /// (`milestone`, `exploration`, `map`, `species`, `trainer`, `battle`, `badge`, `boundary`,
/// onto these. /// `catch`) map onto these.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")] #[serde(rename_all = "lowercase")]
pub enum RewardKind { pub enum RewardKind {
@ -282,6 +282,15 @@ impl RewardKind {
// protocol is concerned: finding a door is finding somewhere new, and the design asks // protocol is concerned: finding a door is finding somewhere new, and the design asks
// for no new feed kind. // for no new feed kind.
"boundary" => Self::Explore, "boundary" => Self::Explore,
// `catch` is a wild battle the fly won by keeping the Pokémon, so it publishes on
// the same counter a wild KO does. The feed's kinds are a closed set
// (`docs/feed-protocol.md`) and this rule asked for no new one.
//
// Deliberately *not* `pokedex`: on a catch of a species this run has never owned,
// the cartridge sets the Pokédex bit and the adapter's existing `species` rule pays
// for it on the same frame, so the `pokedex` counter already moves. Mapping `catch`
// there as well would count one event twice.
"catch" => Self::Wildwin,
// The platformer. // The platformer.
"band" => Self::Explore, "band" => Self::Explore,
"coin" => Self::Wildwin, "coin" => Self::Wildwin,
@ -739,6 +748,7 @@ mod tests {
} }
assert_eq!(RewardKind::from_adapter("nonsense"), None); assert_eq!(RewardKind::from_adapter("nonsense"), None);
assert_eq!(RewardKind::from_adapter("boundary"), Some(RewardKind::Explore)); assert_eq!(RewardKind::from_adapter("boundary"), Some(RewardKind::Explore));
assert_eq!(RewardKind::from_adapter("catch"), Some(RewardKind::Wildwin));
} }
#[test] #[test]

View file

@ -0,0 +1,224 @@
//! A `v5` checkpoint restored under `v6`: accepted with the opt-in, refused without it.
//!
//! The unit tests in `flybrain-gb` cover the decision function and the adapter's own state
//! migration separately. This is the two of them against one artefact: a real `FLYSIM01`
//! envelope carrying a `pokered-unique8-v5` compatibility string and a `v5` reward ledger —
//! written, encoded, decoded, and then put through exactly what `Sim::try_restore` puts a
//! candidate through.
//!
//! No ROM and no dataset, deliberately. Building a `Sim` would need both, and neither is part of
//! the question: what decides a restore is the compatibility string and `import_state`.
use flybrain_gb::GameAdapter;
use flybrain_gb::compatibility::{RestoreDecision, accepted_adapters, decide};
use flybrain_gb::pokemon_red::PokemonRedReward;
use flysim::store::{self, RuntimeState};
/// The live string's shape, with the adapter left open. The dataset fingerprint is shortened —
/// nothing here parses it, and a seven-digest one would be 455 characters of noise.
fn compatibility(adapter: &str) -> String {
format!(
"lif-1ms-f64-v2/{adapter}/aabbccddeeff00/fly-kc-mbon-rstdp-v2/\
binjgb:c60e138da5a795ebb55e56b11b7e90024e41112c/\
pokered:0cd19d3b877b7dc66d12c7050bed9a7f38154d4b/statefmt:199616-x86_64-unknown-linux-gnu"
)
}
/// A `v5` reward ledger: `STATE_VERSION` 4, every field `v5` wrote, and **no** `catchCounts`.
///
/// Written out by hand rather than exported from an adapter, because an exported one would be a
/// `v6` state with the counter deleted — this is the shape the release box's checkpoints really
/// carry, field for field.
fn v5_reward() -> serde_json::Value {
serde_json::json!({
"version": 4,
"seen": ["adventure", "map:0", "early:outside", "dex:3", "boundary:0:edge:n:near"],
"tiles": ["0:5:6", "0:5:7"],
"tileCounts": { "0": 2 },
"wildWins": { "0:112:4": 2 },
"replayBlocked": [],
"counts": {
"milestone": 2, "exploration": 0, "map": 1, "species": 1,
"trainer": 0, "battle": 2, "badge": 0, "boundary": 1
},
"total": 2.05,
"recent": [{ "kind": "species", "label": "OWNED #4", "brainMs": 1234.5, "value": 0.5 }],
"last": { "species": { "kind": "species", "label": "OWNED #4", "brainMs": 1234.5, "value": 0.5 } },
"initialized": true,
"sawBoot": true,
"location": "0:5:7",
"stable": 9,
"progress": 3,
"badges": 0,
"battle": null,
"mode": "OVERWORLD"
})
}
fn v5_checkpoint() -> Vec<u8> {
use flybrain_core::decoder::DecoderState;
use flybrain_core::lif::LifState;
use flybrain_core::ordered::NumberMap;
use flybrain_core::plasticity::PlasticityState;
let agent = flybrain_core::agent::AgentState {
version: 1,
remainder: 0.25,
warmed_up: true,
network: LifState {
membrane: vec![0.1, -0.2],
refractory: vec![0, 1],
last_spike_ms: vec![-1_000_000.0, 5.0],
visual_drive: vec![0.3],
rng: 42,
reward_remaining: 0.0,
ms: 1_234.5,
population_rate: 1.0,
rates: NumberMap::from_pairs([("forward", 1.0)]),
plasticity: PlasticityState {
version: "fly-kc-mbon-rstdp-v2".to_string(),
topology: 7,
enabled: true,
updates: 1.0,
signal: 0.0,
gains: vec![1.0],
traces: vec![0.0],
touched: vec![0.0],
},
},
decoder: DecoderState {
version: 4,
calibrated: true,
baseline: NumberMap::from_pairs([("forward", 1.0)]),
held_until: NumberMap::new(),
next_allowed: NumberMap::new(),
next_decision: 0.0,
current: None,
fatigue: NumberMap::new(),
macro_next_decision: 0.0,
macro_current: None,
macro_fatigue: NumberMap::new(),
},
};
let runtime = RuntimeState {
generation: 41,
wall_ms: 1_790_000_000_000,
rom_sha256: flybrain_gb::pokemon_red::SUPPORTED_ROM.to_string(),
emulator_frame: 1_000_000,
compatibility: compatibility("pokered-unique8-v5"),
speed: 1.0,
buttons: 0,
rank_since_ms: 1_000.0,
last_event_id: 4_242,
reward: v5_reward(),
ratchet: flybrain_gb::RatchetState { best: 3, attempts: 1, recoveries: 4, ..Default::default() },
emulator: vec![3; 64],
framebuffer: vec![0; 32],
ratchet_game: vec![1],
ratchet_frame: vec![2],
};
store::encode(&agent, &runtime).expect("the fixture encodes")
}
#[test]
fn a_v5_checkpoint_is_refused_under_v6_without_the_opt_in() {
let checkpoint = store::decode(&v5_checkpoint()).expect("the fixture decodes");
let adapter = PokemonRedReward::new();
let current = compatibility(adapter.id());
assert_ne!(checkpoint.runtime.compatibility, current, "v6 is not v5");
for opt_in in [None, Some(""), Some("pokered-unique8-v4"), Some("some-other-adapter")] {
assert!(
matches!(
decide(
&checkpoint.runtime.compatibility,
&current,
adapter.migrates_from(),
&accepted_adapters(opt_in),
),
RestoreDecision::Refuse(_)
),
"FLY_ACCEPT_ADAPTERS={opt_in:?} must not migrate anything"
);
}
}
#[test]
fn a_v5_checkpoint_restores_under_v6_with_the_opt_in_and_the_counter_starts_at_zero() {
let checkpoint = store::decode(&v5_checkpoint()).expect("the fixture decodes");
let mut adapter = PokemonRedReward::new();
let current = compatibility(adapter.id());
assert_eq!(
decide(
&checkpoint.runtime.compatibility,
&current,
adapter.migrates_from(),
&accepted_adapters(Some("pokered-unique8-v5")),
),
RestoreDecision::MigrateAdapter { from: "pokered-unique8-v5".to_string() }
);
// The migration itself: `import_state`, exactly as `Sim::try_restore` calls it.
adapter.import_state(&checkpoint.runtime.reward).expect("a v5 ledger is a valid v6 ledger");
let after = adapter.export_state();
assert_eq!(after["catchCounts"], serde_json::json!({}), "the new counter starts at 0");
assert_eq!(after["counts"]["catch"], serde_json::json!(0));
// And nothing else moved: every field the v5 state carried round-trips to the same value,
// and the only key v6 adds is the counter.
//
// `counts` is the one field that is not byte-identical, and it is not a change of meaning:
// it serializes every kind in the catalog, so a v6 state lists `catch` where a v5 state had
// nothing to list. Every kind the v5 state did carry keeps its number.
let before = v5_reward();
for (key, value) in before.as_object().unwrap() {
if key == "counts" {
for (kind, count) in value.as_object().unwrap() {
assert_eq!(&after["counts"][kind], count, "counts.{kind}");
}
let added: Vec<&String> = after["counts"]
.as_object()
.unwrap()
.keys()
.filter(|kind| !value.as_object().unwrap().contains_key(*kind))
.collect();
assert_eq!(added, vec!["catch"], "v6 counts one more kind and no others");
continue;
}
assert_eq!(&after[key], value, "{key} must survive the migration byte for byte");
}
let added: Vec<&String> = after
.as_object()
.unwrap()
.keys()
.filter(|key| !before.as_object().unwrap().contains_key(*key))
.collect();
assert_eq!(added, vec!["catchCounts"], "v6 adds one field and no others");
// The rest of what a restore reads is untouched by the migration.
assert_eq!(adapter.progress().rank, 3);
assert_eq!(checkpoint.runtime.last_event_id, 4_242);
assert_eq!(checkpoint.runtime.ratchet.best, 3);
}
#[test]
fn nothing_but_the_adapter_segment_may_differ_for_the_migration_to_apply() {
let adapter = PokemonRedReward::new();
let accepted = accepted_adapters(Some("pokered-unique8-v5"));
let current = compatibility(adapter.id());
// A v5 string whose state format also moved: a different build, not a rule change.
let other_abi = compatibility("pokered-unique8-v5").replace("199616", "199617");
assert!(matches!(
decide(&other_abi, &current, adapter.migrates_from(), &accepted),
RestoreDecision::Refuse(_)
));
// And an identical string needs no opt-in at all.
assert_eq!(
decide(&current, &current, adapter.migrates_from(), &[]),
RestoreDecision::Exact
);
}

View file

@ -0,0 +1,257 @@
//! The catch reward against the real cartridge.
//!
//! Gated on `FLY_ROM` *and* on a checkpoint, the way every ROM test in this workspace is, and
//! skips cleanly without either — the cartridge never enters this repository and a checkpoint is
//! not a fixture, it is the state the release box was really in:
//!
//! ```sh
//! FLY_ROM="$HOME/roms/pokemon-red.gb" \
//! FLY_CATCH_CHECKPOINT=.local/checkpoints/<a rung-9 forest checkpoint> \
//! cargo test --release -p flysim --test rom_catch -- --nocapture
//! ```
//!
//! ## What only the cartridge can answer
//!
//! The synthetic trace in `pokemon_red/tests.rs` writes `wCapturedMonSpecies`, `wBattleResult`
//! and the Pokédex bit itself, from the disassembly. It cannot say that those are the bytes
//! *this* cartridge writes when a ball keeps a Pokémon, in that order, on frames an adapter
//! sampling once a frame actually sees. That is this test, and it is the "survey" half of
//! `docs/design/macros-wram.md`'s evidence for the row: a real battle, real button presses, and
//! the byte read out of the running game rather than written into a fake one.
//!
//! ## How the catch is produced
//!
//! No steering and no scripted button sequence: the shipping macro palette, the shipping macro
//! layer and the shipping decoder, with a stub readout that leans on one macro population at a
//! time — the same driver `tests/rom_macros_mode.rs` uses and for the same reason. The one thing
//! this harness does that the rotation does not is lean on `THROW BALL`'s channel while a wild
//! battle is up, because the question here is what the adapter reads from a catch, not whether a
//! game-blind readout finds its way to one.
//!
//! The checkpoint must hold at least one ball in the bag. The macro palette can buy one
//! (`BUY BALL`, `MB·PBALL`, inside a mart), but that is a walk across a city and back and it is a
//! different test's question; this one says out loud that it skipped.
use flybrain_core::decoder::PopulationDecoder;
use flybrain_core::decoder::gameboy::gameboy_decoder_config_with_macros;
use flybrain_core::ordered::NumberMap;
use flybrain_gb::adapter::RewardEvent;
use flybrain_gb::pokemon_red::state;
use flybrain_gb::pokemon_red::symbols::ram;
use flybrain_gb::pokemon_red::{PokemonRedReward, catalog};
use flybrain_gb::{
AdapterLedger, DEFAULT_AUDIO_FRAMES, DEFAULT_AUDIO_FREQUENCY, Emulator, GameAdapter,
};
use flysim::config::Config;
use flysim::macros::{MacroLayer, macro_layer};
use flysim::snapshot::MacroMode;
const MS_PER_FRAME: f64 = 1000.0 / 59.7275;
const SEED: u32 = 20_260_922;
/// The hot population's rate against every other one's, which is also the stub's calibration
/// rate — so a channel that is not the hot one scores exactly 1.0.
const HOT: f64 = 16.0;
const REST: f64 = 10.0;
/// `THROW BALL`'s channel (`pokemon_red::macros::palette`).
const BALL: &str = "MB·BALL";
/// Frames the stub leans on one channel before the rotation moves on, the shape of the real
/// group's hysteresis-then-fatigue rotation.
const BURST_FRAMES: u32 = 24;
fn rates(hot: Option<&str>) -> NumberMap {
let mut rates = NumberMap::new();
for channel in flybrain_gb::macro_channels("pokemon-red") {
rates.set(channel, REST);
}
for bucket in 0..8 {
rates.set(&format!("command_{bucket}"), REST);
}
if let Some(channel) = hot {
rates.set(channel, HOT);
}
rates
}
fn rom() -> Option<Vec<u8>> {
let path = std::env::var_os("FLY_ROM")?;
match std::fs::read(&path) {
Ok(bytes) => Some(bytes),
Err(error) => panic!("FLY_ROM is set to {path:?} but could not be read: {error}"),
}
}
fn checkpoint() -> Option<flysim::store::Checkpoint> {
let path = std::env::var_os("FLY_CATCH_CHECKPOINT")?;
Some(
flysim::store::load(std::path::Path::new(&path))
.expect("the checkpoint should be a FLYSIM01 envelope"),
)
}
struct Run {
gb: Emulator,
adapter: PokemonRedReward,
layer: MacroLayer,
decoder: PopulationDecoder,
channels: Vec<&'static str>,
ms: f64,
frame: u32,
/// Every payout the adapter has made since the run started.
payouts: Vec<RewardEvent>,
}
impl Run {
fn resume(rom: &[u8], checkpoint: &flysim::store::Checkpoint) -> Self {
let mut gb = Emulator::new(rom, DEFAULT_AUDIO_FREQUENCY, DEFAULT_AUDIO_FRAMES)
.expect("binjgb should accept the cartridge");
let mut adapter = PokemonRedReward::new();
gb.import_state(&checkpoint.runtime.emulator).expect("the checkpoint's emulator state");
// A checkpoint written by an earlier adapter rebaselines rather than failing, which is
// exactly the `v5` -> `v6` case this rule ships with.
adapter.import_state(&checkpoint.runtime.reward).expect("the checkpoint's reward ledger");
let channels = flybrain_gb::macro_channels("pokemon-red");
let preset = gameboy_decoder_config_with_macros(&channels);
let hold_ms = preset.macros.as_ref().expect("the preset has a macro group").hold_ms;
let mut decoder = PopulationDecoder::new(preset).expect("the preset is well formed");
decoder.calibrate(&rates(None));
let mut config = Config::default();
config.loop_.game = "pokemon-red".to_string();
config.macros.mode = MacroMode::Macros;
config.validate().expect("pokemon-red has a palette in macros mode");
let mut layer = macro_layer(&config, hold_ms, SEED).expect("a layer in macros mode");
let _ = layer.observe(&mut gb, &AdapterLedger(&adapter), 0.0);
Self {
gb,
adapter,
layer,
decoder,
channels,
ms: 0.0,
frame: 0,
payouts: Vec::new(),
}
}
fn byte(&mut self, address: u16) -> u8 {
self.gb.read_wram(address)
}
fn in_wild_battle(&mut self) -> bool {
self.byte(ram::wIsInBattle) == 1
}
/// Balls in the bag, of any kind (`constants/item_constants.asm`: MASTER_BALL 1,
/// ULTRA_BALL 2, GREAT_BALL 3, POKE_BALL 4 — the same four `THROW BALL` looks for).
fn balls(&mut self) -> usize {
state::bag(&mut self.gb)
.iter()
.filter(|item| (0x01..=0x04).contains(&item.id) && item.count > 0)
.map(|item| usize::from(item.count))
.sum()
}
fn step(&mut self) {
// Lean on `THROW BALL` while a wild battle is up; otherwise rotate, which is what gets
// the fly into the grass in the first place.
let hot = if self.in_wild_battle() {
Some(BALL)
} else {
let slot = (self.frame / BURST_FRAMES) as usize % self.channels.len();
Some(self.channels[slot])
};
let bound = self.layer.bound_channels();
let active = self.decoder.decode_bound(&rates(hot), self.ms, false, None, Some(&bound));
let mask = {
let ledger = AdapterLedger(&self.adapter);
self.layer.decide(&active, 0, self.ms, &mut self.gb, &ledger).mask
};
self.gb.set_buttons(mask as u8);
self.gb.run_frame().expect("a frame should complete");
self.ms += MS_PER_FRAME;
self.frame += 1;
let ms = self.ms;
self.payouts.extend(self.adapter.sample(&mut self.gb, ms));
let ledger = AdapterLedger(&self.adapter);
let _ = self.layer.observe(&mut self.gb, &ledger, ms);
}
fn catches(&self) -> Vec<&RewardEvent> {
self.payouts.iter().filter(|event| event.kind == catalog::kind::CATCH).collect()
}
}
#[test]
fn a_catch_on_the_cartridge_pays_the_catch_rule_once_with_the_species_in_its_label() {
let Some(rom) = rom() else {
eprintln!("skipped: FLY_ROM is not set");
return;
};
let Some(checkpoint) = checkpoint() else {
eprintln!("skipped: no FLY_CATCH_CHECKPOINT");
return;
};
let mut run = Run::resume(&rom, &checkpoint);
let balls = run.balls();
if balls == 0 {
eprintln!(
"skipped: the checkpoint's bag holds no ball (map {:#04x}). `BUY BALL` can buy one \
inside a mart; point FLY_CATCH_CHECKPOINT at a state that already has one.",
run.adapter.map_id().unwrap_or(u32::MAX)
);
return;
}
eprintln!("bag holds {balls} balls; map {:#04x}", run.adapter.map_id().unwrap_or(u32::MAX));
// Twenty brain minutes is generous for a forest checkpoint: the live run threw 28 balls in
// its first Viridian Forest session (`pokemon_red::macros::palette`).
let budget = 20 * 60 * 60;
let mut battles = 0u32;
let mut was_in_battle = false;
for _ in 0..budget {
run.step();
let now = run.in_wild_battle();
if now && !was_in_battle {
battles += 1;
}
was_in_battle = now;
if !run.catches().is_empty() {
break;
}
}
let catches = run.catches();
assert!(
!catches.is_empty(),
"no catch in {:.1} brain minutes: {battles} wild battles, {} balls left, map {:#04x}, \
macros {:?}",
run.ms / 60_000.0,
run.balls(),
run.adapter.map_id().unwrap_or(u32::MAX),
run.layer.counts()
);
let caught = catches[0];
eprintln!(
"caught after {:.1} brain minutes and {battles} wild battles: {} for {}",
run.ms / 60_000.0,
caught.label,
caught.value
);
assert!(caught.label.starts_with("CAUGHT #"), "{}", caught.label);
assert!(
(caught.value - 0.30).abs() < 1e-12 || caught.value == catalog::CATCH_REPEAT_VALUE,
"a catch pays one of the rule's two amounts, not {}",
caught.value
);
// The cartridge's own flag is clear again by the time the payout lands, which is what makes
// the payout a battle-exit event rather than a per-frame one.
assert_eq!(run.byte(ram::wCapturedMonSpecies), 0);
assert_eq!(
run.adapter.progress().counts[catalog::kind::CATCH],
1,
"one battle, one payout"
);
// And a species the run is paid for catching is a species the Pokédex knows: the same event
// sets the bit the `species` rule reads, whether or not it was new to this run.
assert!(run.balls() < balls, "a ball was spent");
}

View file

@ -200,6 +200,15 @@ EXTRA_RAM = (
'wCurMapTileset', 'wCurMapTileset',
'wTilesetBank', 'wTilesetBank',
'wTilesetBlocksPtr', 'wTilesetBlocksPtr',
# The catch reward (`docs/rewards-learning.md`, `docs/design/macros-wram.md` section 2).
# ram/wram.asm's own comment is "0 if no mon was captured": ItemUseBall zeroes it before
# every throw and writes wEnemyMonSpecies into it only on the branch that keeps the
# Pokemon, and UseBagItem zeroes it again on the way out of the battle. It is the
# cartridge's own answer to "was this one caught", and the only signal that needs no
# second rule to tell a catch apart from a gift, a trade or an evolution.
# services/flysim/tools/resolve_wram.py is the second reading of it, from ram/wram.asm at
# this commit, bracketed by wFontLoaded and wForcePlayerToChooseMon.
'wCapturedMonSpecies',
) )

View file

@ -49,6 +49,33 @@ WANTED = {
# not bank 0, so this is the read the memory seam grew a bank for. # not bank 0, so this is the read the memory seam grew a bank for.
'wTilesetBank': 'the ROM bank the blockset lives in', 'wTilesetBank': 'the ROM bank the blockset lives in',
'wTilesetBlocksPtr': 'blocks to tiles, 16 bytes per block', 'wTilesetBlocksPtr': 'blocks to tiles, 16 bytes per block',
# The catch reward (`docs/rewards-learning.md`, `docs/design/macros-wram.md`
# section 10). ram/wram.asm's own comment is "0 if no mon was captured":
# ItemUseBall zeroes it before every throw and writes wEnemyMonSpecies into it
# only on the branch that keeps the caught Pokemon, and UseBagItem zeroes it
# again on the way out of the battle. It is the cartridge's own answer to "was
# this one caught", and the only signal that needs no second rule to tell a
# catch apart from a gift, a trade or an evolution.
'wCapturedMonSpecies': 'the species a ball just caught, 0 for none',
}
#: Constants the decomp defines through its `const` enumeration rather than with a
#: plain `EQU`, so `constants()` cannot evaluate their expressions. They matter here
#: because `NUM_TMS + NUM_HMS` is the size of `wMonHLearnset`, and that one
#: declaration is what kills the cursor on its way through the battle engine's
#: scratch bytes -- the region `wCapturedMonSpecies` lives in.
#:
#: Each is *counted* from the decomp rather than written out by hand, which is the
#: same rule the rest of this tool follows. `DEF NUM_HMS EQU const_value - HM01` is
#: by construction the number of `add_hm` definitions after `HM01`, and
#: `item_constants.asm`'s own `ASSERT NUM_TMS == const_value - TM01` ties `NUM_TMS`
#: to the number of `add_tm` definitions -- so `NUM_TMS` is counted *and* compared
#: against the literal the same file declares, and a decomp that moved one without
#: the other stops the run instead of producing an address.
COUNTED = {
'NUM_HMS': ('constants/item_constants.asm', r'^\s*add_hm\s+\w+'),
'NUM_TMS': ('constants/item_constants.asm', r'^\s*add_tm\s+\w+'),
} }
@ -68,6 +95,10 @@ def constants(root: Path) -> dict[str, int]:
BLOCK_WIDTH`). A name whose expression never becomes evaluable is simply left BLOCK_WIDTH`). A name whose expression never becomes evaluable is simply left
out, which kills the cursor at any declaration that uses it. out, which kills the cursor at any declaration that uses it.
""" """
counted = {
name: len(re.findall(pattern, (root / path).read_text(), re.M))
for name, (path, pattern) in COUNTED.items()
}
pending: dict[str, str] = {} pending: dict[str, str] = {}
sources = sorted((root / 'constants').glob('*.asm')) + sorted( sources = sorted((root / 'constants').glob('*.asm')) + sorted(
(root / 'constants').glob('*.inc') (root / 'constants').glob('*.inc')
@ -77,7 +108,7 @@ def constants(root: Path) -> dict[str, int]:
r'^\s*(?:DEF|def)\s+(\w+)\s+(?:EQU|equ)\s+([^;\n]+)', path.read_text(), re.M r'^\s*(?:DEF|def)\s+(\w+)\s+(?:EQU|equ)\s+([^;\n]+)', path.read_text(), re.M
): ):
pending.setdefault(name, value.strip()) pending.setdefault(name, value.strip())
out: dict[str, int] = {} out: dict[str, int] = dict(counted)
while pending: while pending:
progressed = False progressed = False
for name in list(pending): for name in list(pending):
@ -89,6 +120,12 @@ def constants(root: Path) -> dict[str, int]:
progressed = True progressed = True
if not progressed: if not progressed:
break break
for name, value in counted.items():
if out.get(name, value) != value:
raise SystemExit(
f'{name}: the decomp declares {out[name]} and defines {value} of them'
)
out[name] = value
return out return out
@ -367,11 +404,17 @@ def main() -> None:
raise SystemExit('the walk disagrees with symbols.rs; nothing emitted') raise SystemExit('the walk disagrees with symbols.rs; nothing emitted')
print(f'{checked} of {len(table)} pinned addresses re-derived from wram.asm, no disagreement') print(f'{checked} of {len(table)} pinned addresses re-derived from wram.asm, no disagreement')
missing = [name for name in WANTED if name not in resolved] # A name this tool has already emitted is pinned, so the walk meets it as an
# anchor rather than resolving it: it was re-derived all the same, and the
# comparison above is what says so.
missing = [name for name in WANTED if name not in resolved and name not in table]
if missing: if missing:
raise SystemExit(f'unanchored, so not resolved: {", ".join(missing)}') raise SystemExit(f'unanchored, so not resolved: {", ".join(missing)}')
for name in WANTED: for name in WANTED:
print(f'{name} = ${resolved[name]:04x} ({WANTED[name]})') if name in resolved:
print(f'{name} = ${resolved[name]:04x} ({WANTED[name]})')
else:
print(f'{name} = ${table[name]:04x} (already pinned; {WANTED[name]})')
if not args.emit: if not args.emit:
return return