From f2dc5d434a2fa8811cef8e407ab511f11f331b3b Mon Sep 17 00:00:00 2001 From: dev Date: Tue, 22 Sep 2026 17:05:25 +0000 Subject: [PATCH 1/8] flybus: assert the latest-mode guarantee, not the machine's timing session_over_one_router asserted that a latest subscriber must drop snapshots. bus-v1 section 7 says a latest subscription replaces an undelivered value; it lets a consumer miss values, it does not oblige it to. Under contention the publisher was slow enough that the renderer kept up, saw all twenty snapshots and the test failed on conforming behaviour: 16 of 40 runs beside four busy loops, and 9 of 20 whole-crate runs. The renderer is now held until the publisher's completion is observed rather than until a timer expires, so the coalescing is forced instead of raced for: the subscription keeps the one delivery in flight and one replaceable queued value, and the renderer receives snapshots 1 and 20 of 20. The assertions are the guarantees that hold -- what arrives is in publication order, the last value is the latest published, the stalled spectator never refuses a publication or drops out of the fan-out, and every snapshot the renderer missed is counted as a replacement to the publisher at admission and to the renderer on delivery, so nothing is lost silently. Three more tests in the crate asserted the same kind of race: - latest_replay_is_ordered_ahead_of_a_racing_publish demanded the non-coalesced outcome of a race section 7 allows either way ("bounded mode preserves that order, while latest mode may coalesce it"). It now puts one racing publication to a bounded and a latest subscription at once: bounded must deliver the replay and then the publication, and the latest branch is chosen by that publication's own replaced count. - collection_waits_for_every_retained_owner, and the two disconnect cleanup tests beside it, read the store directory for the unlink that follows the registry update outside the router lock. They use settle_files, like every other unlink check in the suite. - The demo example printed a root count taken before the producer's own release had reached the router, so the line the guide quotes was a race. It waits for the release, the same way it already waits for collection; the printed output is unchanged. Nothing under flybus/src is touched: no routing defect was found. The conformance rows for the credit/queue split, the replaced count, the replay ordering and the latest spectator that cannot refuse a publication now cite what each rewritten test actually varies. --- .../session-framework/bus-conformance.md | 10 +-- .../flysim/crates/flybus/examples/demo.rs | 11 +++- .../flybus/tests/conformance_artifacts.rs | 8 ++- .../flybus/tests/conformance_routing.rs | 64 ++++++++++++++++--- .../flysim/crates/flybus/tests/integration.rs | 61 ++++++++++++++---- 5 files changed, 125 insertions(+), 29 deletions(-) diff --git a/docs/design/session-framework/bus-conformance.md b/docs/design/session-framework/bus-conformance.md index cd90df6..a987554 100644 --- a/docs/design/session-framework/bus-conformance.md +++ b/docs/design/session-framework/bus-conformance.md @@ -153,15 +153,15 @@ and once over a Unix socket, so `tests/rpc.rs::request_reply_roundtrip` means | `latest`: one queued value, replacing only an undelivered one; replacement releases that entry's roots; delivered or in-use messages are never reclaimed early; maxQueued is exactly 1 | conforms | `router/state.rs::{op_publish (latest branch), op_subscribe}` | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both), `tests/conformance_artifacts.rs::latest_mode_holds_at_most_two_roots_delivered_plus_queued` (both) | | `bounded`: FIFO, no coalescing or silent loss; when capacity is unavailable, reject with `BACKPRESSURE` before admitting any delivery | conforms | `router/state.rs::op_publish` (pre-checks every subscriber) | `tests/pubsub.rs::bounded_fifo_and_atomic_backpressure` (both), `tests/conformance_routing.rs::bounded_overflow_rolls_back_all_artifact_roots` (both) | | maxInFlight credits return only on `delivery.consumed`, not on socket write completion | conforms | `router/state.rs::release_owner` (credit returned when the owner is released) | `tests/pubsub.rs::credits_return_only_on_consume` (both), `tests/conformance_routing.rs::bounded_credit_waits_for_every_extracted_artifact` (both) | -| A latest subscriber with all credits in use still has one replaceable queued value | conforms | `router/state.rs::{Sub::queue, dispatch_topic}` (queue and credits are separate) | `tests/bus_acceptance.rs::both_transports_produce_equivalent_behaviour_traces` (events 21 and 24: `publish replaced=1`, then `latest seq=3 replaced=1`) | +| A latest subscriber with all credits in use still has one replaceable queued value | conforms | `router/state.rs::{Sub::queue, dispatch_topic}` (queue and credits are separate) | `tests/bus_acceptance.rs::both_transports_produce_equivalent_behaviour_traces` (events 21 and 24: `publish replaced=1`, then `latest seq=3 replaced=1`), `tests/integration.rs::session_over_one_router` (both: a renderer held for the whole run keeps one delivery in flight and one replaceable value, and receives snapshots 1 and 20 of 20) | | Atomic subscriber/retention snapshot at admission; validate and reserve every queue entry and owner budget before accepting | conforms: one mutex, validate-then-mutate | `router/state.rs::op_publish` | `tests/artifacts.rs::failed_admission_is_atomic` (both) | | A bounded overflow rejects the whole publish: no partial fan-out, no retained-latest update | conforms | `router/state.rs::op_publish` | `tests/conformance_routing.rs::bounded_overflow_rolls_back_all_artifact_roots` (both) | | On acceptance, one `topicSequence` and roots for every delivery and the optional retained value | conforms; a refused publication spends no sequence number | `router/state.rs::op_publish` (`t.sequence += 1` after the checks) | `tests/pubsub.rs::bounded_fifo_and_atomic_backpressure` (both) | | Different topics have no total ordering; multiple publishers follow router acceptance order | conforms: per-topic sequence only | `router/state.rs::Topic::sequence` | `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) | | The publication reply counts accepted subscriptions and replaced queue entries, not consumers that processed data | conforms | `router/state.rs::op_publish` reply | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both) | -| `replaced` on a delivery reports how many undelivered messages were coalesced since that subscription's preceding delivery | conforms | `router/state.rs::{Sub::replaced, dispatch_topic}` (taken at dispatch) | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both) | +| `replaced` on a delivery reports how many undelivered messages were coalesced since that subscription's preceding delivery | conforms | `router/state.rs::{Sub::replaced, dispatch_topic}` (taken at dispatch) | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both), `tests/integration.rs::session_over_one_router` (both: the 18 replacements the publisher was told about at admission are the same 18 the renderer is told about on delivery, and are exactly the snapshots it did not receive) | | Optional `retained:latest` holds one last message and its artifacts independent of subscribers | conforms | `router/state.rs::op_publish` (retain branch) | `tests/conformance_artifacts.rs::retained_topic_value_holds_a_root_independent_of_subscribers` (both) | -| `replayLatest` enqueues the retained value before subsequent accepted publications; bounded preserves the order, latest may coalesce it | conforms | `router/state.rs::op_subscribe` (replay is enqueued under the subscribe lock) | `tests/conformance_routing.rs::latest_replay_is_ordered_ahead_of_a_racing_publish` (both), `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) | +| `replayLatest` enqueues the retained value before subsequent accepted publications; bounded preserves the order, latest may coalesce it | conforms | `router/state.rs::op_subscribe` (replay is enqueued under the subscribe lock) | `tests/conformance_routing.rs::latest_replay_is_ordered_ahead_of_a_racing_publish` (both: one racing publication to a bounded and a latest subscription at once; bounded must deliver replay then publication, and the latest branch is chosen by that publication's own `replaced` count, never by which side of the race the dispatcher won), `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) | | Replay uses the original topicSequence, a fresh deliveryId and explicit roots | conforms | `router/state.rs::op_subscribe` (`add_roots`, the same `Arc`) | `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) | | Without retention, a zero-subscriber publication retains no ownership after admission | conforms | `router/state.rs::op_publish` | `tests/pubsub.rs::zero_subscriber_publish_retains_nothing` (both) | | Clearing a topic releases only its retained root, not active consumers | conforms | `router/state.rs::op_clear` | `tests/conformance_routing.rs::cleared_topic_gives_no_replay_until_a_fresh_publish` (both) | @@ -260,7 +260,7 @@ and once over a Unix socket, so `tests/rpc.rs::request_reply_roundtrip` means | The thirteen transport error codes exist with those names | conforms | `error.rs::ErrorCode` | `tests/wire.rs::body_errors_keep_the_connection` (both) | | Three more codes: `CONFLICT`, `NO_TOPIC`, `ARTIFACT_MISMATCH` | deviates-allowed: "Transport errors **include** ..." is not an exhaustive list, and each names a refusal the draft requires but leaves unnamed. Recorded as an amendment in bus-v1 section 12 | `error.rs::ErrorCode` | `tests/pubsub.rs::subscription_and_topic_validation` (both), `tests/artifacts.rs::seal_checks_length_and_digest` (both) | | Before admission report `not-dispatched`; once dispatch might have occurred report `dispatched` or `unknown` conservatively | conforms | `error.rs::BusError::new` (not-dispatched by default), `router/state.rs` dispatched notices, `client/reactor.rs::fail_all` (unknown) | `tests/rpc.rs::{cancellation_states, service_disconnect_fails_calls}` (both), `tests/sol_review_races.rs::writer_failure_terminates_reader_and_pending_work` | -| Bounded subscriptions can reject a publication; latest spectators cannot hold a session transaction indefinitely | conforms: a latest subscriber never causes `BACKPRESSURE` | `router/state.rs::op_publish` (the latest branch skips every capacity check) | `tests/bus_acceptance.rs::a_latest_subscriber_never_refuses_a_publication` (both: 100 publications of 60 KB into one unconsumed slot, six times the bounded pool, none refused, 98 coalesced), with `tests/pubsub.rs::{bounded_fifo_and_atomic_backpressure, saturated_subscriber_does_not_block_control}` (both) for the bounded half | +| Bounded subscriptions can reject a publication; latest spectators cannot hold a session transaction indefinitely | conforms: a latest subscriber never causes `BACKPRESSURE` | `router/state.rs::op_publish` (the latest branch skips every capacity check) | `tests/bus_acceptance.rs::a_latest_subscriber_never_refuses_a_publication` (both: 100 publications of 60 KB into one unconsumed slot, six times the bounded pool, none refused, 98 coalesced), with `tests/pubsub.rs::{bounded_fifo_and_atomic_backpressure, saturated_subscriber_does_not_block_control}` (both) for the bounded half, and `tests/integration.rs::session_over_one_router` (both: 20 snapshot publications accepted by both subscriptions while the presentation consumer reads nothing) | | Sustained pinned-artifact quota exhaustion is surfaced as pressure, not solved by freeing live data | conforms: `QUOTA_EXCEEDED`, never eviction | `router/state.rs::{op_allocate, op_seal}` | `tests/artifacts.rs::quotas_are_enforced` (both) | | Session and application policies choose disconnect, pause or fail; the router does not know which | conforms by absence | `router/state.rs` | `tests/pubsub.rs::saturated_subscriber_does_not_block_control` (both) | @@ -284,7 +284,7 @@ and once over a Unix socket, so `tests/rpc.rs::request_reply_roundtrip` means | 3. Artifacts: allocate/seal/read; publication before seal fails; fan-out owns one object; the last consumer releases; a retained extracted frame survives a message drop | conforms | `tests/artifacts.rs` (28), `tests/conformance_artifacts.rs` (36) | | 4. Faults: sender drops after admission, consumer dies mid-read, reply lost, queued frame replaced, subscription closes with in-use deliveries, router restarts, old release arrives; no double-free, use-after-reuse, unbounded tombstones or hidden replay | conforms | `tests/conformance_routing.rs::{caller_disconnect_detaches_dispatched_call_but_service_keeps_serving, subscriber_disconnect_releases_queued_and_delivered_artifacts_but_not_retention}`, `tests/artifacts.rs::{release_ids_are_watermarked, router_restart_invalidates_old_handles}`, `tests/bus_acceptance.rs::{a_lost_result_leaks_no_roots_and_the_endpoint_cache_still_replays, disconnect_releases_logical_ownership_without_mutating_open_bytes}`, `tests/sol_rereview_regressions.rs` (11) | | 5. RPC cache: an endpoint retains an artifact-bearing result, the original caller consumes it, a domain retry still returns valid bytes, eviction drops the last hold | conforms | `tests/rpc.rs::endpoint_cache_replays_artifact_results` (both), `tests/bus_acceptance.rs::a_retransmission_repeats_the_domain_request_under_a_fresh_call_id` (both) | -| 6. Integration: two parallel fake agents, a complete-batch environment RPC, committed snapshot publication and a deliberately slow presentation consumer on one router | conforms | `tests/integration.rs::session_over_one_router` (both) | +| 6. Integration: two parallel fake agents, a complete-batch environment RPC, committed snapshot publication and a deliberately slow presentation consumer on one router | conforms; the consumer is slow by construction, held until the publisher's completion is observed, so its coalescing is forced rather than raced for | `tests/integration.rs::session_over_one_router` (both) | | 7. Performance: 640x480x60 with three consumers, one delayed; p50/p95/p99 RPC latency, router CPU, copy and readback cost separately, RSS, store live and peak bytes, outstanding roots, collection lag, queue lengths, for one, two and four agents | conforms | `tests/perf.rs::frames_at_60hz_with_three_consumers` (`--ignored`); numbers below | | The first executable example: a counter RPC, a pub/sub observer and a frame artifact held past message consumption, in one small Rust program, no game or browser | conforms | `examples/demo.rs` (`cargo run -p flybus --example demo`), asserted by `tests/example_demo.rs::the_example_shows_a_counter_rpc_an_observer_and_a_held_frame` | diff --git a/services/flysim/crates/flybus/examples/demo.rs b/services/flysim/crates/flybus/examples/demo.rs index 12830de..73f3fb5 100644 --- a/services/flysim/crates/flybus/examples/demo.rs +++ b/services/flysim/crates/flybus/examples/demo.rs @@ -99,8 +99,17 @@ pub async fn run() -> Result, Box> { "published sequence {} to {} subscriber(s)", receipt.topic_sequence, receipt.subscribers )); - // The producer lets go of its own hold; the delivery keeps the bytes alive. + // The producer lets go of its own hold; the delivery keeps the bytes alive. The release + // travels the control lane like any other operation, so the count below waits for it + // instead of reading a number that may still include it. drop(frame); + let released = Instant::now() + Duration::from_secs(10); + while router.stats().artifact_roots > 1 { + if Instant::now() > released { + return Err("the producer's own hold was never released".into()); + } + tokio::time::sleep(Duration::from_millis(1)).await; + } let message = frames.next().await.ok_or("the subscription closed")?; let image = message.artifact("frame")?; diff --git a/services/flysim/crates/flybus/tests/conformance_artifacts.rs b/services/flysim/crates/flybus/tests/conformance_artifacts.rs index cf8a8d2..2eb874b 100644 --- a/services/flysim/crates/flybus/tests/conformance_artifacts.rs +++ b/services/flysim/crates/flybus/tests/conformance_artifacts.rs @@ -179,7 +179,9 @@ async fn collection_waits_for_every_retained_owner(via: Via) { s.owners == 0 && s.sealed_artifacts == 0 && s.store_bytes == 0 }) .await; - assert_eq!(e.files("sealed"), 0); + // The unlink follows the registry update, outside the router lock: wait for the file to + // go rather than assume the two happen together. + e.settle_files("sealed", 0).await; } // ------------------------------------------------------------------------------------------- @@ -521,7 +523,7 @@ async fn disconnect_abandons_an_unsealed_writer(via: Via) { s.artifacts == 0 && s.store_bytes == 0 && s.owners == 0 }) .await; - assert_eq!(e.files("staging"), 0); + e.settle_files("staging", 0).await; } /// An abrupt disconnect must release an explicit hold too, when it was the object's only root. @@ -538,7 +540,7 @@ async fn disconnect_releases_an_explicit_hold(via: Via) { s.sealed_artifacts == 0 && s.owners == 0 && s.store_bytes == 0 }) .await; - assert_eq!(e.files("sealed"), 0); + e.settle_files("sealed", 0).await; } /// A vanished subscriber must give up both a delivery it already holds and one still queued diff --git a/services/flysim/crates/flybus/tests/conformance_routing.rs b/services/flysim/crates/flybus/tests/conformance_routing.rs index 3c0ef30..d8aaa7f 100644 --- a/services/flysim/crates/flybus/tests/conformance_routing.rs +++ b/services/flysim/crates/flybus/tests/conformance_routing.rs @@ -420,14 +420,20 @@ async fn bounded_overflow_rolls_back_all_artifact_roots(via: Via) { } /// bus-v1 section 7: "New subscriptions with replayLatest enqueue it before subsequent accepted -/// publications." A fresh `latest` subscription's replay claims its first in-flight credit -/// immediately (there is nothing else competing for it yet), so a publish accepted right after -/// subscribing must still be observed strictly after the replay, never ahead of or merged with -/// it: each keeps its own delivery. +/// publications ... bounded mode preserves that order, while latest mode may coalesce it before +/// delivery under the ordinary latest rule." The replay is enqueued under the subscribe lock, so +/// a publication admitted after `subscribe` returned is always behind it; what the two modes do +/// with that order is what differs, and one racing publication is put to both at once. +/// +/// The bounded subscription must deliver both values, replay first. The latest subscription +/// either does the same or replaces the still-queued replay, and the router says which in the +/// racing publication's own `replaced` count rather than the test guessing from how fast the +/// dispatcher ran: what it may never do is reorder the two or lose the newer value. async fn latest_replay_is_ordered_ahead_of_a_racing_publish(via: Via) { let e = env(via).await; let admin = e.client("admin").await; let reader = e.client("reader").await; + let viewer = e.client("viewer").await; admin .declare_topic("t.replay-race", Retained::Latest) .await @@ -436,26 +442,66 @@ async fn latest_replay_is_ordered_ahead_of_a_racing_publish(via: Via) { .publish("t.replay-race", obj(json!({"v": "old"})), &[]) .await .unwrap(); - let mut sub = reader + let mut fifo = reader + .subscribe( + "t.replay-race", + SubscriptionConfig::bounded().in_flight(1).replay(true), + ) + .await + .unwrap(); + let mut coalescing = viewer .subscribe( "t.replay-race", SubscriptionConfig::latest().in_flight(1).replay(true), ) .await .unwrap(); - admin + let racing = admin .publish("t.replay-race", obj(json!({"v": "new"})), &[]) .await .unwrap(); - let first = within("the replay arrives first", sub.next()) + assert_eq!( + racing.subscribers, 2, + "one publication, admitted behind both replays" + ); + + let first = within("the replay arrives first", fifo.next()) .await .unwrap(); assert_eq!(first.payload()["v"], "old"); drop(first); // the sole in-flight credit must return before the queued second value moves - let second = within("the racing publish follows, not coalesced away", sub.next()) + let second = within("the racing publish follows, not coalesced away", fifo.next()) .await .unwrap(); - assert_eq!(second.payload()["v"], "new"); + assert_eq!( + (second.payload()["v"].as_str(), second.replaced()), + (Some("new"), 0), + "bounded preserves the order and coalesces nothing" + ); + + let m = within("the latest subscription's first delivery", coalescing.next()) + .await + .unwrap(); + if racing.replaced == 1 { + assert_eq!( + (m.payload()["v"].as_str(), m.replaced()), + (Some("new"), 1), + "a replay still queued is replaced by the newer value, and the delivery says so" + ); + drop(m); + quiet("nothing behind a coalesced replay", coalescing.next()).await; + } else { + assert_eq!( + (racing.replaced, m.payload()["v"].as_str(), m.replaced()), + (0, Some("old"), 0), + "a replay already in flight keeps its own delivery" + ); + drop(m); + let after = within("the racing publish follows it", coalescing.next()) + .await + .unwrap(); + assert_eq!(after.payload()["v"], "new"); + } } /// bus-v1 section 7: clearing releases only the retained root; a later `replayLatest` diff --git a/services/flysim/crates/flybus/tests/integration.rs b/services/flysim/crates/flybus/tests/integration.rs index c4c95ba..f18136b 100644 --- a/services/flysim/crates/flybus/tests/integration.rs +++ b/services/flysim/crates/flybus/tests/integration.rs @@ -1,6 +1,11 @@ //! bus-v1 section 11 item 6: two parallel fake agents, complete-batch environment RPC, //! committed snapshot publication and a deliberately slow presentation consumer, all over one //! router. Generic services only; nothing here knows what a brain or a game is. +//! +//! The presentation consumer is held until the publisher's own completion is observed, so the +//! latest subscription has to coalesce instead of happening to: section 7 lets a latest +//! subscriber miss values, it does not oblige it to, and a test that demands a miss it cannot +//! force is asserting how fast the machine is. mod common; @@ -148,20 +153,32 @@ async fn session_over_one_router(via: Via) { ) .await .unwrap(); + // A renderer that does not read a single snapshot until the publisher has finished every + // one of them. The hold ends on the publisher's observed completion, never on a timer. + let (release, held) = tokio::sync::oneshot::channel::<()>(); let presenting = tokio::spawn(async move { + held.await.unwrap(); let mut seen = Vec::new(); + let mut coalesced = 0; while let Some(m) = slow.next().await { + let (sequence, replaced) = (m.topic_sequence(), m.replaced()); let frame = m.artifact("frame").unwrap(); drop(m); tokio::time::sleep(Duration::from_millis(25)).await; // a slow renderer let bytes = frame.read_all().await.unwrap(); let step = bytes[0] as u64; + assert_eq!( + (step, sequence), + (step, step), + "a delivery carries the frame of the snapshot it announces" + ); + coalesced += replaced; seen.push((step, frame.reference().artifact_id.clone())); if step == STEPS { break; } } - seen + (seen, coalesced) }); let recorder = e.client("recorder").await; let mut all = recorder @@ -179,6 +196,7 @@ async fn session_over_one_router(via: Via) { seq }); + let mut replaced_at_admission = 0; for step in 1..=STEPS { let advanced = coordinator .call_and_wait( @@ -226,8 +244,16 @@ async fn session_over_one_router(via: Via) { ) .await .unwrap(); - assert_eq!(receipt.topic_sequence, step); + assert_eq!( + (receipt.topic_sequence, receipt.subscribers), + (step, 2), + "both subscriptions accept every publication: the stalled latest spectator neither \ + refuses one nor drops out of the fan-out" + ); + replaced_at_admission += receipt.replaced; } + // The publisher is finished, observably, so the renderer may start. + release.send(()).unwrap(); let recorded = within("recorder", recording).await.unwrap(); assert_eq!( @@ -235,17 +261,30 @@ async fn session_over_one_router(via: Via) { (1..=STEPS).map(|s| (s, s)).collect::>(), "the bounded recorder misses nothing" ); - let presented = within("presenter", presenting).await.unwrap(); - assert_eq!( - presented.last().unwrap().0, - STEPS, - "the slow consumer ends on the latest snapshot" - ); + let (presented, coalesced) = within("presenter", presenting).await.unwrap(); + let steps: Vec = presented.iter().map(|(s, _)| *s).collect(); assert!( - presented.len() < STEPS as usize, - "the slow consumer skipped snapshots: {presented:?}" + steps.windows(2).all(|w| w[0] < w[1]), + "what a latest subscription does deliver arrives in publication order: {presented:?}" + ); + assert_eq!( + steps.last().copied(), + Some(STEPS), + "the slow consumer ends on the latest snapshot: {presented:?}" + ); + assert_eq!( + steps, + vec![1, STEPS], + "held for the whole run, the subscription keeps the one delivery already in flight and \ + one replaceable queued value, so the renderer sees the first snapshot and the last, \ + and the eighteen between them were coalesced: {presented:?}" + ); + assert_eq!( + (coalesced, replaced_at_admission), + (STEPS - 2, STEPS - 2), + "every snapshot the renderer missed is counted as a replacement, to the publisher at \ + admission and to the renderer on its next delivery: none is lost silently" ); - assert!(presented.windows(2).all(|w| w[0].0 < w[1].0)); for t in agents { t.abort(); From 9bed780cb96c462a81de0a647247fac260889f04 Mon Sep 17 00:00:00 2001 From: dev Date: Tue, 22 Sep 2026 17:12:46 +0000 Subject: [PATCH 2/8] flybus: wait for the pending-connection events instead of timing them pending_connections_are_bounded_and_hello_expires gave the router 20 ms to register a connection and then slept 80 ms past a 40 ms hello timeout before reading the count once. Both are claims about how fast this box is, not about the router: the first failed once in 20 whole-crate runs beside four busy loops. Both now wait for the event they are about, under the suite's ordinary ten-second bound, so the test still fails if a pending connection never registers or a pending Hello never expires. --- .../crates/flybus/tests/sol_review_races.rs | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-) diff --git a/services/flysim/crates/flybus/tests/sol_review_races.rs b/services/flysim/crates/flybus/tests/sol_review_races.rs index 9d26b1b..793df64 100644 --- a/services/flysim/crates/flybus/tests/sol_review_races.rs +++ b/services/flysim/crates/flybus/tests/sol_review_races.rs @@ -266,13 +266,14 @@ async fn pending_connections_are_bounded_and_hello_expires() { let router = Router::new(config).unwrap(); let pending = router.connect_in_memory_as("first"); - tokio::time::timeout(Duration::from_millis(20), async { + // Registration happens on the router's own task. How long that takes is this box's + // business; that it happens is the router's. + within("the pending connection occupies the only slot", async { while router.stats().connections != 1 { - tokio::task::yield_now().await; + tokio::time::sleep(Duration::from_millis(1)).await; } }) - .await - .unwrap(); + .await; assert_eq!(router.stats().connections, 1); let refused = Client::connect( router.connect_in_memory_as("second"), @@ -280,7 +281,14 @@ async fn pending_connections_are_bounded_and_hello_expires() { ) .await; assert_eq!(refused.unwrap_err().code, ErrorCode::RouterLost); - tokio::time::sleep(Duration::from_millis(80)).await; + // The 40 ms hello timeout expires on the router's clock: wait for the expiry to be + // observed rather than sleep past it and read the count once. + within("the pending Hello expires", async { + while router.stats().connections != 0 { + tokio::time::sleep(Duration::from_millis(1)).await; + } + }) + .await; assert_eq!(router.stats().connections, 0, "pending Hello timed out"); drop(pending); From 23a4d7379b245883fc5db9b697f05584e571dd0a Mon Sep 17 00:00:00 2001 From: alex Date: Tue, 22 Sep 2026 17:30:00 +0000 Subject: [PATCH 3/8] rewards: a catch reward, adapter v6 with a v5 migration, and a rung restart The operator's decision of 2026-09-22: pay the fly for keeping a wild Pokemon, bump the adapter properly, and restart the live run from an early checkpoint rather than from scratch. The rule. `catch` is the catalog's ninth kind, appended so the key order `counts` serializes in does not move. 0.30 for a species this run had never owned, 0.10 for a repeat, three payouts per species for the lifetime of the ledger; the `species` rule is untouched, so a first catch of a new species pays 0.80 across two kinds. The catch is read from `wCapturedMonSpecies` ($d11c), whose comment in ram/wram.asm is "0 if no mon was captured": ItemUseBall zeroes it before every throw and writes wEnemyMonSpecies into it only on the branch that keeps the Pokemon, and UseBagItem's `.returnAfterCapturingMon` zeroes it again and sets wBattleResult to 2 -- a value written on exactly two paths in the game, that one and a link battle whose opponent ran. Both are required, so a byte read out of a half-initialised battle cannot pay. Not wPartyCount: a catch with a full party raises wBoxCount instead, and wPartyCount also rises for a gift, a trade and a PC withdrawal. "Never owned this run" is the `species` payout inside the same battle, because nothing else can set a Pokedex bit during one. It is not read off the captured species byte: that is the cartridge's internal index while the owned bitset is by Pokedex number, and nothing in WRAM converts between them. The address was resolved by tools/resolve_wram.py, not written by hand. The tool needed NUM_TMS and NUM_HMS, which the decomp defines through its `const` enumeration, so it now counts them from the file's own add_tm/add_hm definitions and cross-checks NUM_TMS against the literal the same file declares. The feed's kinds are closed, so `catch` publishes on `wildwin` and nothing in packages/feed or apps/stage changed. Deliberately not `pokedex`: the `species` rule already pays for the bit the same catch sets. The stage's ticker copy is keyed on the feed kind, so a catch row reads "wild win" -- stated in docs/rewards-learning.md rather than left to be discovered. v5 -> v6. STATE_VERSION stays 4: the rule adds one counter, `catchCounts`, and changes nothing else, so a v5 state restores with it empty. That migration is opt-in and needs all three of: the adapter segment being the only difference between the two compatibility strings, the running adapter listing the checkpoint's adapter in `migrates_from()`, and the deploy naming it in FLY_ACCEPT_ADAPTERS. flysim applies the rule at restore and 05-deploy's gate applies the same rule before it flips the symlink, writing the variable into fly.env so the two cannot disagree. The restart. infra/bin/fly-reset-to-milestone archives both stores to a dated directory, rewrites milestone-.checkpoint with the ratchet's attempts and recoveries at zero, installs it as the newest generation of both stores, clears the milestone archives above N and the event log, and prints what it did. It refuses while flysim is running and refuses a rung the run never reached. The envelope work is in flysim::reset (`flysim --reset-to-milestone N`); the shell script is the operator's wrapper. Tests: catalog values and order; a synthetic WRAM trace of a catch (new, repeat, cap, already-owned species, trainer/Safari/old-man/missed-ball negatives, rollback replay); a v5 state restoring with the counter at zero; a v5 checkpoint fixture accepted with the opt-in and refused without it; the reset tool against copies of a state dir in temp directories; and a ROM-gated catch from a rung-9 forest checkpoint, driven by the shipping THROW BALL macro. The compatibility string differs from main's in exactly one segment, checked by splitting both on `/`: pokered-unique8-v5 -> pokered-unique8-v6. --- docs/design/flysim.md | 44 ++ docs/design/macros-wram.md | 14 + docs/rewards-learning.md | 65 ++- infra/05-deploy.sh | 63 ++- infra/bin/fly-reset-to-milestone | 77 +++ infra/docs/runbook.md | 53 +++ infra/env/example.env | 14 + .../flysim/crates/flybrain-gb/src/adapter.rs | 20 +- .../crates/flybrain-gb/src/compatibility.rs | 157 +++++- .../flybrain-gb/src/pokemon_red/catalog.rs | 54 ++- .../crates/flybrain-gb/src/pokemon_red/mod.rs | 196 +++++++- .../flybrain-gb/src/pokemon_red/symbols.rs | 1 + .../flybrain-gb/src/pokemon_red/tests.rs | 165 ++++++- services/flysim/crates/flysim/src/lib.rs | 1 + services/flysim/crates/flysim/src/main.rs | 20 + services/flysim/crates/flysim/src/reset.rs | 450 ++++++++++++++++++ services/flysim/crates/flysim/src/simloop.rs | 32 +- services/flysim/crates/flysim/src/snapshot.rs | 14 +- .../crates/flysim/tests/compat_migration.rs | 224 +++++++++ .../flysim/crates/flysim/tests/rom_catch.rs | 257 ++++++++++ services/flysim/tools/gen_symbols.py | 9 + services/flysim/tools/resolve_wram.py | 49 +- 22 files changed, 1947 insertions(+), 32 deletions(-) create mode 100755 infra/bin/fly-reset-to-milestone create mode 100644 services/flysim/crates/flysim/src/reset.rs create mode 100644 services/flysim/crates/flysim/tests/compat_migration.rs create mode 100644 services/flysim/crates/flysim/tests/rom_catch.rs diff --git a/docs/design/flysim.md b/docs/design/flysim.md index abdddab..5e7f0b5 100644 --- a/docs/design/flysim.md +++ b/docs/design/flysim.md @@ -355,6 +355,50 @@ on-screen ticker cannot disagree with what the sim did. against the prototype's WASM size and diffs a known save. If they match, prototype checkpoints import and the segment records the shared tag; if not, milestone saves must be re-earned and that is a stated M3 finding. +- **Restoring across an adapter version** (2026-09-22). The compatibility string is compared + whole, so bumping the reward adapter refuses every checkpoint the previous one wrote -- which is + the right default and was, until now, the only behaviour. It is the wrong default for a change + that only *adds* a rule: `pokered-unique8-v6` adds the catch reward and one counter, + `catchCounts`, and means the same thing as `v5` for every other field, so a `v5` run is + resumable and throwing it away would be a choice nobody made deliberately. + + So there is one narrow, opt-in migration, `flybrain_gb::compatibility::decide`, and it requires + **all three** of: + + 1. the two compatibility strings differ in the adapter segment (segment 1) and **nowhere else**. + A dataset, kernel, plasticity, emulator-revision, symbol-provenance or state-format + difference is still a refusal: none of those has a migration, and a fly restored across one + is a different fly; + 2. the running adapter's `migrates_from()` lists the checkpoint's adapter, so the code that will + read that state says out loud that it can. Pokémon Red's list is `["pokered-unique8-v5"]` and + nothing else -- `v4` is excluded because its ledger holds no `boundary:` keys and resuming it + would pay a second time for every exit already found, and `v3` because its stored rank is a + rung on a different ladder; + 3. the deploy names the same adapter id in **`FLY_ACCEPT_ADAPTERS`** (comma- or + space-separated). Unset or empty migrates nothing, which is what every deploy before this one + did. + + Condition 2 without 3 would make the migration silent; condition 3 without 2 would let an + operator wave through a pair nobody wrote a migration for. `infra/05-deploy.sh`'s compatibility + gate applies the same rule before it flips the `current` symlink, and writes the variable into + `/etc/fly/fly.env` so flysim applies it at restore -- the two must agree, or a deploy would pass + a gate that flysim then fails, which is the black stream the gate exists to prevent. The + migration itself is `PokemonRedReward::import_state` doing what it already did: `catchCounts` is + absent from a `v5` state and restores empty, which is the truth about a run that was never paid + for a catch. `STATE_VERSION` does not move, because the schema did not. + +- **Restarting a run from an earlier rung** (2026-09-22). `FLY_RESET_STATE=1` throws the run away; + `infra/bin/fly-reset-to-milestone ` keeps it and rewinds it. It archives both stores to a + dated directory, rewrites `milestone-.checkpoint` with the ratchet's `attempts` and + `recoveries` at zero (so the restarted run does not begin with its recovery budget already + spent), installs it as the newest generation of the hot and durable stores, removes the + milestone archives above N, and clears the event log -- whose id sequence the restored + checkpoint's `lastEventId` rewinds. `best` is not touched: the archive's own `best` is the rung + it was taken at, and the rank the stream shows is recomputed by the adapter from the restored + game state. The implementation is `flysim::reset` (`flysim --reset-to-milestone N`) rather than + the shell script, because two of those steps are inside the envelope. The sequence around it is + in `infra/docs/runbook.md`. + - **A running macro is not checkpointed** (2026-09-16, `docs/design/macros.md`). Palette mode's state — the scene, the palette, the running macro, its plan and its frame count — is transient, like the readout's blocked-direction cooldown and for the same reason: a restore that resumed a diff --git a/docs/design/macros-wram.md b/docs/design/macros-wram.md index 0c721a3..a3253de 100644 --- a/docs/design/macros-wram.md +++ b/docs/design/macros-wram.md @@ -118,6 +118,20 @@ Addresses are at the pinned commit. "Verified" is one of: | which slot is out | `wPlayerMonNumber` | `$cc2f` | 0-based party slot | ROM, trace | | the enemy | `wEnemyMonSpecies`, `wEnemyMonHP`, `wEnemyMonLevel`, `wEnemyMonMaxHP` | `$cfe5`, `$cfe6`, `$cff3`, `$cff4` | HP big-endian. Not written on the frame a battle starts — the reward adapter's own comment says the same — so the enemy is `None` for the first few hundred frames of a battle. | ROM (the rival's Squirtle, level 5, 20/20, and `None` on the first frame), trace | | how many moves | `wNumMovesMinusOne` | `$cd6c` | the move count minus one, valid in a battle | trace | +| **a ball kept this one** | `wCapturedMonSpecies` | `$d11c` | **new 2026-09-22** (the catch reward, `docs/rewards-learning.md`). `ram/wram.asm`'s own comment is "0 if no mon was captured". `ItemUseBall` zeroes it before every throw (`.canUseBall`) and writes `wEnemyMonSpecies` into it only on the branch that keeps the Pokémon; `UseBagItem`'s `.returnAfterCapturingMon` zeroes it again and sets `wBattleResult` to 2 on the way out of the battle. It is therefore non-zero for the hundreds of frames the catch's text and Pokédex screen take, and zero everywhere else. The value is the **internal** species index, like `wEnemyMonSpecies` and unlike `wPokedexOwned`'s bit index. Address resolved by `services/flysim/tools/resolve_wram.py`, bracketed by `wFontLoaded` and `wForcePlayerToChooseMon`. | survey (`tests/rom_catch.rs`: a real wild battle from a rung-9 checkpoint, balls thrown by the `THROW BALL` macro, the byte read out of the running game), trace (`pokemon_red/tests.rs`) | + +`wBattleResult` (`$cf0b`) is the second half of that row and is worth its own sentence: it is 0 +for a win, 1 for a loss, and 2 on exactly two paths in the whole game -- `.returnAfterCapturingMon` +and a *link* battle whose opponent ran (`engine/battle/core.asm`), which this cartridge never has. +So "the captured-species byte was non-zero during the battle **and** the result is 2" is a catch +and nothing else. `InitBattleVariables`, `ResetStatusAndHalveMoneyOnBlackout` and +`HandleFlyWarpOrDungeonWarp` all clear it, so a stale 2 cannot survive into the next battle. + +Not used for the catch, and why: `wPartyCount` (`$d163`) rises on a catch **only** when the party +has room -- a full party sends the Pokémon to `wBoxCount` instead -- and it also rises for a gift, +a trade and a Pokémon withdrawn from the PC. Reading a catch off it would need a second rule to +tell those apart. The cartridge's own flag needs none, which is why the row above is the one the +adapter reads. ### Battle menu and cursor, own turn against forced switch diff --git a/docs/rewards-learning.md b/docs/rewards-learning.md index e3130ee..1975629 100644 --- a/docs/rewards-learning.md +++ b/docs/rewards-learning.md @@ -1,6 +1,6 @@ # Rewards and learning -The live reward catalog of the Pokémon Red adapter, `pokered-unique8-v5`. The code of record is +The live reward catalog of the Pokémon Red adapter, `pokered-unique8-v6`. The code of record is `services/flysim/crates/flybrain-gb/src/pokemon_red/` (`catalog.rs` holds the values, `mod.rs` the gates and the rules); this page says what each rule pays for and why it is allowed to. The prototype's own `docs/rewards-learning.md` in `fly-plays-pokemon` is where the first seven rules @@ -24,6 +24,7 @@ change what the fly can do. | `battle` | `wildwin` | +0.1, +0.05, +0.0333 | 100 ms | At most three observed wild KOs per `(map, species, level)` | | `badge` | `badge` | +3 | 400 ms | Each newly set badge bit | | `boundary` | `explore` | +0.05, +0.10 | 100 ms | First tile adjacent to one of the map's exits, and the exit tile itself; once per `(map, exit)` for the lifetime of the ledger | +| `catch` | `wildwin` | +0.30, +0.10 | 150 ms | A wild Pokémon kept by a ball: +0.30 for a species this run had never owned, +0.10 for a repeat; at most three payouts per species for the lifetime of the ledger | Every value is positive: there are no loss or blackout penalties, and `catalog::rule("blackout")` is `None` by test. The values in one frame sum into `R`, and the network reinforces once with @@ -33,6 +34,54 @@ The feed-kind column is `RewardKind::from_adapter` in `services/flysim/crates/fl `docs/feed-protocol.md` publishes seven counters, and an adapter kind that has no counter of its own shares the nearest one. It still reaches the page as an event with its own label. +Two consequences of that sharing are worth stating rather than discovering. `catch` publishes on +`wildwin` because a catch is a wild battle the fly won by keeping the Pokémon, and *not* on +`pokedex` because the `species` rule already pays for the Pokédex bit the same catch sets -- +counting it twice would be the dishonest option. And the stage's ticker copy is keyed on the feed +kind, not on the catalog kind (`apps/stage/src/games/pokemon-red.ts`), so the row for a catch +currently reads "wild win". The event's own label, `CAUGHT #`, is what reaches the event +log, `/status` and the checkpoint. Changing the ticker copy means opening the feed's closed kind +set, which this rule deliberately did not do. + +## Catch rewards + +The operator's decision of 2026-09-22: the fly is paid for *keeping* a wild Pokémon, not only for +knocking one out. The rule is one kind with two payouts, the way `boundary` is. + +**How a catch is read.** From `wCapturedMonSpecies` (`$d11c`), whose comment in `ram/wram.asm` at +the pinned commit is "0 if no mon was captured". `ItemUseBall` zeroes it before every throw +(`.canUseBall`) and writes `wEnemyMonSpecies` into it only on the branch that keeps the Pokémon; +`UseBagItem`'s `.returnAfterCapturingMon` zeroes it again and sets `wBattleResult` to 2 on the way +out of the battle. `wBattleResult` is 2 on exactly two paths in the whole game -- that one, and a +link battle whose opponent ran -- so requiring both the species and the result means a byte read +out of a half-initialised battle cannot pay. The adapter records the species during the battle and +pays on the way out, where the wild-KO payout already lives. + +Not from `wPartyCount`. A catch with a full party raises `wBoxCount` instead, and `wPartyCount` +also rises for a gift, a trade and a Pokémon taken out of the PC, so it would need a second rule +to mean anything. The cartridge's own flag needs none. + +**What counts as a new species.** The `species` payout inside the same battle. Nothing but a catch +can set a `wPokedexOwned` bit during a wild battle, so a `species` payout between the battle +starting and the ball keeping the Pokémon *is* that Pokémon being new to the run. It is read this +way rather than off `wCapturedMonSpecies` because that byte is the cartridge's **internal** species +index while the owned bitset is by **Pokédex number**, and nothing in WRAM converts between the two +(`docs/design/macros-wram.md` section 2, "species numbering"). A battle restored from a checkpoint +written before this rule existed carries no "species payouts when it started", which reads as +"cannot tell" and pays the repeat amount: the conservative half, and at most 0.20 once. + +**The budget.** Three payouts per species for the lifetime of the ledger, the same cap and the +same reason as the wild-KO rule's three: a species the fly can find over and over is a farm, and +three is enough for the behaviour to be learned. A rollback blocks every species already paid, +exactly as it blocks every wild-KO key already paid, so the same catch cannot be replayed for +reward. A Safari Zone or old-man battle pays nothing, because the whole sample is dropped a step +earlier with a visible mode; a trainer battle pays nothing, because balls cannot be thrown in one. + +**The scale.** 0.30 on its own is below a new Pokédex entry (0.50), below a story flag (1.0) and +well below a badge (3.0). A catch of a new species pays 0.80 across two kinds, which sits between +a story flag and a badge -- deliberately, because it is the one event that is both a discovery and +a thing the fly had to do on purpose. + ## Gates Semantic rewards are enabled for exactly one cartridge, the SHA-256 in `SUPPORTED_ROM`. Any other @@ -128,7 +177,19 @@ body picks the macro; the descending neurons press the buttons.** ## Honesty -The catalog now includes exits. That is worth saying plainly on the honesty panel, because paying +The catalog now includes catches. The honesty panel's copy is not data-driven from the catalog -- +`apps/stage/src/lib/schedule.ts`'s rotating card is four written lines and lists no kinds -- so +there was nothing to regenerate and the copy is unchanged. The sentences below are where the +argument lives. + +Paying for a catch does not move the fly: the ball is thrown by a macro the mushroom body chose +among the ones the battle scene put on the pad, and the payout is read out of WRAM after the +frame. What it does do is make one of the palette's existing macros worth choosing, which is the +same kind of pressure every other rule applies. The cap is what keeps it from becoming a farm: a +run that finds one patch of grass and throws balls at the same species all night earns 0.50 from +it and then nothing. + +The catalog also includes exits. That is worth saying plainly on the honesty panel, because paying for a door is closer to telling the fly where to go than paying for a badge is: - **still no button path.** Nothing in the adapter chooses or biases a button. The reward is read diff --git a/infra/05-deploy.sh b/infra/05-deploy.sh index 892bf3e..ccc58c6 100755 --- a/infra/05-deploy.sh +++ b/infra/05-deploy.sh @@ -186,6 +186,36 @@ else log "05-deploy: CPUSET unset — heavy in-container steps run unpinned (no partition configured)" fi +# Whether the only difference between two compatibility strings is the adapter +# segment, and FLY_ACCEPT_ADAPTERS names the adapter the live checkpoints carry. +# +# The bash half of flybrain_gb::compatibility::decide, which is what flysim +# itself applies at restore. Both have to agree: a gate that let a deploy +# through and a flysim that then refused every checkpoint would be the black +# stream this whole section exists to prevent. The string is +# {kernel}/{adapter}/{fingerprint}/{plasticity}/binjgb:{rev}/pokered:{commit}/statefmt:{id}, +# so the adapter is segment 1 and nothing else may move. +adapter_migration_accepted() { + local live="$1" new="$2" accepted="$3" + local -a live_parts new_parts + IFS='/' read -r -a live_parts <<< "$live" + IFS='/' read -r -a new_parts <<< "$new" + [ "${#live_parts[@]}" -eq "${#new_parts[@]}" ] || return 1 + local i differing=0 index=-1 + for ((i = 0; i < ${#live_parts[@]}; i++)); do + if [ "${live_parts[$i]}" != "${new_parts[$i]}" ]; then + differing=$((differing + 1)) + index=$i + fi + done + [ "$differing" -eq 1 ] && [ "$index" -eq 1 ] || return 1 + local entry + for entry in ${accepted//,/ }; do + [ "$entry" = "${live_parts[1]}" ] && return 0 + done + return 1 +} + # cpu_pin CMD [ARGS...] — run CMD inside the container on the page cpus. # Falls through to a plain ct_exec when no partition is configured, so this is # a no-op on an unpartitioned container rather than a new failure mode (a @@ -261,6 +291,16 @@ if [ -n "$RELEASE_TARBALL" ]; then # checkpoints (kept, never deleted) and clears the tmpfs hot ring, so the # new build warms up fresh. Everything learned so far is thrown away, which # is why it is not the default. + # + # FLY_ACCEPT_ADAPTERS is the *other* override, and the opposite one: it keeps + # the run. It names adapter version strings whose checkpoints the new build + # may migrate — e.g. FLY_ACCEPT_ADAPTERS=pokered-unique8-v5 for the deploy + # that adds the catch reward. It only applies when the adapter segment is the + # ONLY difference between the two strings and the new build's adapter says it + # can read that one; a dataset, kernel, emulator or state-format change is + # still a refusal, because none of those has a migration. The same variable is + # written into /etc/fly/fly.env below, so flysim applies the same rule at + # restore that this gate applied at deploy. # ----------------------------------------------------------------------- state_dir="${FLY_STATE_DIR:-/srv/fly/state}" hot_dir="${FLY_STATE_HOT_DIR:-/run/fly/state}" @@ -286,6 +326,11 @@ if [ -n "$RELEASE_TARBALL" ]; then log "05-deploy: no decodable checkpoint in ${state_dir} — nothing to compare, continuing" elif [ "$new_compat" = "$live_compat" ]; then log "05-deploy: checkpoint compatibility matches the live state, the new build will restore it" + elif [ -n "${FLY_ACCEPT_ADAPTERS:-}" ] \ + && adapter_migration_accepted "$live_compat" "$new_compat" "$FLY_ACCEPT_ADAPTERS"; then + log "05-deploy: FLY_ACCEPT_ADAPTERS=${FLY_ACCEPT_ADAPTERS} — the adapter version is the only difference, and it is named; the run is KEPT and migrated" + log "05-deploy: live: $live_compat" + log "05-deploy: new: $new_compat" elif [ "${FLY_RESET_STATE:-0}" = 1 ]; then archive="${state_dir}.$(date -u +%Y%m%d%H%M%S)" log "05-deploy: FLY_RESET_STATE=1 — compatibility CHANGED, archiving the durable state to ${archive} and clearing the hot ring" @@ -299,8 +344,12 @@ if [ -n "$RELEASE_TARBALL" ]; then die "05-deploy: REFUSING to deploy release ${version}: its checkpoint compatibility string does not match the live state in ${state_dir}, so flysim would refuse every checkpoint there and then refuse to start at all — a black stream. live state: ${live_compat} new build: ${new_compat} -The difference is usually an adapter/ladder or dataset version bump. Two ways forward: +The difference is usually an adapter/ladder or dataset version bump. Three ways forward: * deploy a build whose string matches (check out the commit the running release was built from), or + * if the ADAPTER VERSION is the only segment that differs and the new build documents a + migration from the old one, re-run with FLY_ACCEPT_ADAPTERS set to the adapter id in the live + string (e.g. FLY_ACCEPT_ADAPTERS=pokered-unique8-v5). The run is kept; flysim applies the same + rule at restore. See docs/design/flysim.md, \"Restoring across an adapter version\", or * accept losing everything the brain has learned and re-run with FLY_RESET_STATE=1, which archives ${state_dir}'s checkpoints to ${state_dir}. (kept, not deleted) and clears ${hot_dir} so the new build warms up fresh. @@ -456,6 +505,16 @@ trap 'rm -f "$tmp_fly_env" "$tmp_flypush_env"' EXIT if [[ -n "${FLY_MACRO_BLOCKED_MINUTES:-}" ]]; then echo "FLY_MACRO_BLOCKED_MINUTES=${FLY_MACRO_BLOCKED_MINUTES}" fi + # Adapter versions whose checkpoints this build may migrate + # (flybrain_gb::compatibility, docs/design/flysim.md "Restoring across an + # adapter version"). Only written when it is set, because the safe state is + # absent: an empty or missing variable migrates nothing, which is what every + # deploy before 2026-09-22 did. It stays in fly.env for as long as the + # operator leaves it on the deploy command line, so removing the opt-in is + # one deploy without it. + if [[ -n "${FLY_ACCEPT_ADAPTERS:-}" ]]; then + echo "FLY_ACCEPT_ADAPTERS=${FLY_ACCEPT_ADAPTERS}" + fi # flybridge (services/bridge/src/config.ts). Nothing wrote these before, so # flybridge.service had no EnvironmentFile= at all and the service refused to # start with "CHANNEL is required / BOT_USER is required / GAME_TITLE is @@ -606,7 +665,7 @@ fi # --------------------------------------------------------------------------- log "05-deploy: converging bin/ helpers to /opt/fly/bin" ct_exec "$CTID" -- mkdir -p /opt/fly/bin -for name in fly-watchdog fly-recap fly-retention flypush flystage-launch flycast-launch wait-for-x wait-for-stage wait-for-health; do +for name in fly-watchdog fly-recap fly-retention fly-reset-to-milestone flypush flystage-launch flycast-launch wait-for-x wait-for-stage wait-for-health; do converge_file "$CTID" "$INFRA_DIR/bin/$name" "/opt/fly/bin/$name" 0755 root:root >/dev/null done diff --git a/infra/bin/fly-reset-to-milestone b/infra/bin/fly-reset-to-milestone new file mode 100755 index 0000000..732c343 --- /dev/null +++ b/infra/bin/fly-reset-to-milestone @@ -0,0 +1,77 @@ +#!/usr/bin/env bash +# infra/bin/fly-reset-to-milestone — restart the run from an earlier ladder rung, +# instead of from scratch. +# +# The operator's decision of 2026-09-22: "restart the live run from an early +# checkpoint instead of from scratch". 05-deploy's FLY_RESET_STATE=1 cannot do +# that — it archives the durable state and the next start warms up a fresh fly, +# losing everything the brain has learned. This promotes one milestone archive +# (milestone-.checkpoint, written at the first commit at a new best rank and +# never rotated away) to being what both stores restore. +# +# Usage: fly-reset-to-milestone +# Run INSIDE the container, as root, with flysim STOPPED. It refuses +# otherwise, and it refuses a rung this run never reached. +# +# The whole sequence — stop, reset, deploy with the adapter opt-in, start, +# verify the rank — is in infra/docs/runbook.md, "Restart the run from a rung". +# Nothing here is destructive on its own: every file in both stores is copied to +# a dated directory next to the durable one before anything is rewritten. +set -euo pipefail + +: "${FLY_STATE_DIR:=/srv/fly/state}" +: "${FLY_STATE_HOT_DIR:=/run/fly/state}" +: "${FLY_RELEASE_DIR:=/opt/fly/current}" +: "${FLY_SERVICE:=flysim.service}" +: "${FLY_USER:=fly}" + +FLYSIM="${FLY_BIN:-${FLY_RELEASE_DIR}/flysim}" + +log() { echo "fly-reset-to-milestone: $*" >&2; } +die() { log "$*"; exit 1; } + +RANK="${1:-}" +if [ "$#" -ne 1 ] || ! [[ "$RANK" =~ ^[0-9]+$ ]]; then + die "usage: fly-reset-to-milestone (e.g. fly-reset-to-milestone 9)" +fi + +# --- refusals ---------------------------------------------------------------- +# A running flysim owns both stores: it commits a hot checkpoint every few +# seconds and a durable one every few minutes, so a reset underneath it would be +# overwritten within the minute and the tool would have lied. +if command -v systemctl >/dev/null 2>&1 && systemctl is-active --quiet "$FLY_SERVICE"; then + die "$FLY_SERVICE is running. Stop it first: systemctl stop $FLY_SERVICE" +fi +[ -x "$FLYSIM" ] || die "no flysim binary at $FLYSIM (set FLY_BIN to point at one)" + +milestone="${FLY_STATE_DIR}/milestone-${RANK}.checkpoint" +# The binary refuses this too, and refuses before it copies anything; checking +# here as well is what makes the message name the rungs that do exist. +if [ ! -f "$milestone" ]; then + log "no milestone archive for rung ${RANK}: $milestone does not exist." + log "rungs this run reached:" + ls -1 "${FLY_STATE_DIR}"/milestone-*.checkpoint 2>/dev/null \ + | sed 's|.*/milestone-||; s|\.checkpoint$||' | sort -n | tr '\n' ' ' >&2 || true + echo >&2 + exit 1 +fi + +# --- the reset --------------------------------------------------------------- +log "resetting to rung ${RANK} (durable ${FLY_STATE_DIR}, hot ${FLY_STATE_HOT_DIR})" +FLY_STATE="$FLY_STATE_DIR" FLY_STATE_HOT="$FLY_STATE_HOT_DIR" \ + "$FLYSIM" --reset-to-milestone "$RANK" + +# flysim runs unprivileged; this tool runs as root, so everything it wrote and +# everything it archived has to go back to the service account. +if command -v chown >/dev/null 2>&1 && id "$FLY_USER" >/dev/null 2>&1; then + chown -R "${FLY_USER}:${FLY_USER}" "$FLY_STATE_DIR" "$FLY_STATE_HOT_DIR" 2>/dev/null || true + for dir in "${FLY_STATE_DIR}".reset-*; do + [ -d "$dir" ] && chown -R "${FLY_USER}:${FLY_USER}" "$dir" + done +fi + +log "done. Next, per infra/docs/runbook.md:" +log " 1. deploy the build whose adapter wrote that checkpoint, or deploy the new" +log " one with FLY_ACCEPT_ADAPTERS set to the checkpoint's adapter id" +log " 2. systemctl start $FLY_SERVICE" +log " 3. curl -s localhost:7401/status | grep -o '\"rank\":[0-9]*'" diff --git a/infra/docs/runbook.md b/infra/docs/runbook.md index a07a413..97e4efa 100644 --- a/infra/docs/runbook.md +++ b/infra/docs/runbook.md @@ -231,6 +231,59 @@ auto-reset (`docs/design/flysim.md` section 8: "no automatic fresh start, ever") is deliberate — a silent reset would be indistinguishable from real progress on stream. A deliberate reset means moving `/srv/fly/state` aside by hand. +## Restart the run from a rung + +When the run has to go back to an earlier milestone rather than start over — the operator's +decision of 2026-09-22 was "restart the live run from an early checkpoint instead of from +scratch". `FLY_RESET_STATE=1` is the wrong tool: it archives the durable state and the next start +warms up a fresh fly, losing everything the brain has learned. + +`infra/bin/fly-reset-to-milestone ` promotes `milestone-.checkpoint` to being what both +stores restore, with the ratchet's attempts and recoveries back at zero. It copies every file in +both stores to `/srv/fly/state.reset-` first, so it is reversible by hand. It refuses while +flysim is running, and refuses a rung this run never reached. + +The whole sequence, in order. Claim the container in the host's agent claim log first, like any +other work on it. + +``` +CTID= +N=9 # the rung to restart from + +# 1. what rungs exist at all +pct exec $CTID -- ls -1 /srv/fly/state/milestone-*.checkpoint + +# 2. stop flysim (it owns both stores; a reset underneath it is overwritten within the minute) +pct exec $CTID -- systemctl stop flysim.service + +# 3. the reset. Prints what it did, one line per step. +pct exec $CTID -- /opt/fly/bin/fly-reset-to-milestone $N + +# 4. deploy. Two cases: +# (a) the running release already wrote that checkpoint -> nothing to deploy, skip to 5. +# (b) the new build bumps the ADAPTER VERSION and nothing else -> name the checkpoint's +# adapter so the gate and flysim both migrate instead of refusing: +FLY_ACCEPT_ADAPTERS=pokered-unique8-v5 infra/05-deploy.sh +# The gate logs "the adapter version is the only difference, and it is named; the run is KEPT +# and migrated", and writes FLY_ACCEPT_ADAPTERS into /etc/fly/fly.env so flysim applies the +# same rule at restore. Anything else about the string differing is still a refusal. + +# 5. start +pct exec $CTID -- systemctl start flysim.service + +# 6. verify: the rank is the rung, and the restore came from the generation the tool wrote +pct exec $CTID -- curl -s http://127.0.0.1:7401/status | jq '.milestone.rank, .game.badges, .checkpoint' +pct exec $CTID -- journalctl -u flysim -n 40 --no-pager | grep -E 'restored|migration|compatibility' +``` + +Step 6 is the one that must be read rather than assumed. The rank is recomputed by the adapter +from the restored game state, not taken from the ratchet, so a rank that is *not* N means the +milestone archive was taken somewhere other than where its name says — stop and look before +starting a stream on it. + +To undo: stop flysim, move the contents of `/srv/fly/state.reset-/durable` back into +`/srv/fly/state`, delete the generation the tool wrote, and start again. + ## Restore from the backup host ``` diff --git a/infra/env/example.env b/infra/env/example.env index ddab36f..fc19970 100644 --- a/infra/env/example.env +++ b/infra/env/example.env @@ -317,6 +317,20 @@ FLY_MACRO_MODE=raw # target once more. Unset means the default, 10. # FLY_MACRO_BLOCKED_MINUTES=10 +# --- restoring across an adapter version ------------------------------------ +# Adapter version strings whose checkpoints this build may migrate, comma- or +# space-separated (docs/design/flysim.md, "Restoring across an adapter +# version"). Unset -- the default, and what every deploy before 2026-09-22 did +# -- migrates nothing: a build whose compatibility string differs from the live +# state's is refused by 05-deploy's gate and by flysim at restore. +# +# It applies only when the ADAPTER segment is the only difference between the +# two strings AND the new build's adapter declares a migration from that one. A +# dataset, kernel, plasticity, emulator or state-format difference is still a +# refusal. Set it for the one deploy that needs it and leave it out afterwards; +# 05-deploy writes it into /etc/fly/fly.env only while it is set. +# FLY_ACCEPT_ADAPTERS=pokered-unique8-v5 + # --- push mode -------------------------------------------------------------- # local: flypush.service stays disabled, everything else identical to prod. # twitch: flypush.service is enabled by 07-enable.sh. diff --git a/services/flysim/crates/flybrain-gb/src/adapter.rs b/services/flysim/crates/flybrain-gb/src/adapter.rs index 15d8456..9e24815 100644 --- a/services/flysim/crates/flybrain-gb/src/adapter.rs +++ b/services/flysim/crates/flybrain-gb/src/adapter.rs @@ -56,7 +56,7 @@ impl MemoryReader for &mut dyn MemoryReader { /// One reward payout in one frame. /// /// `kind` is an adapter-owned interned name (Pokémon: `milestone`, -/// `exploration`, `map`, `species`, `trainer`, `battle`, `badge`, `boundary`); it is the +/// `exploration`, `map`, `species`, `trainer`, `battle`, `badge`, `boundary`, `catch`); it is the /// key the statistics counters and the on-screen ticker group by. Field names /// serialize exactly as the prototype's `RewardEvent` did, so a checkpoint /// written by either implementation reads in the other. @@ -241,9 +241,21 @@ impl std::error::Error for AdapterError {} /// A game, as the sim loop sees it. pub trait GameAdapter: Send { /// Adapter version string, pinned into the checkpoint compatibility string. - /// Pokémon: `pokered-unique8-v5`. + /// Pokémon: `pokered-unique8-v6`. fn id(&self) -> &'static str; + /// Earlier [`GameAdapter::id`]s whose checkpoints this build can read, by a migration + /// this adapter has written down and tested. + /// + /// The default is empty: an adapter migrates from nothing unless it says otherwise, which + /// is the behaviour every adapter had before this existed. It is only half of the gate -- + /// [`crate::compatibility::decide`] also requires the operator to have named the same id in + /// `FLY_ACCEPT_ADAPTERS` for that deploy -- so listing an id here never migrates a live run + /// on its own. + fn migrates_from(&self) -> &'static [&'static str] { + &[] + } + /// Whether semantic rewards are enabled for this cartridge. An adapter that /// says no must still sample without paying anything, so the stream keeps /// running with a visible "rewards off" mode. @@ -470,6 +482,10 @@ mod tests { let platformer = adapter_for_with_rom_pin("platformer", Some(&"a".repeat(64))).unwrap(); assert_ne!(pokemon.id(), platformer.id()); + assert!( + !platformer.migrates_from().contains(&pokemon.id()), + "a migration never crosses games" + ); assert_ne!(pokemon.symbol_provenance(), platformer.symbol_provenance()); // And two ROM revisions of the same game cannot either. let other = adapter_for_with_rom_pin("platformer", Some(&"b".repeat(64))).unwrap(); diff --git a/services/flysim/crates/flybrain-gb/src/compatibility.rs b/services/flysim/crates/flybrain-gb/src/compatibility.rs index 7e183c0..b3fc959 100644 --- a/services/flysim/crates/flybrain-gb/src/compatibility.rs +++ b/services/flysim/crates/flybrain-gb/src/compatibility.rs @@ -30,7 +30,7 @@ pub const PROTOTYPE_PLASTICITY_VERSION: &str = "fly-kc-mbon-rstdp-v2"; pub struct Compatibility<'a> { /// `kernelVersion(config)` from the neural library. pub neural_kernel_version: &'a str, - /// The adapter's version string, e.g. `pokered-unique8-v5`. + /// The adapter's version string, e.g. `pokered-unique8-v6`. pub adapter: &'a str, /// The dataset's seven SHA-256 digests joined with `:`. pub dataset_fingerprint: &'a str, @@ -67,6 +67,92 @@ impl Compatibility<'_> { } } +/// Position of the adapter's version string in [`Compatibility::string`]. +/// +/// `{kernel}/{adapter}/{fingerprint}/{plasticity}/binjgb:{rev}/pokered:{commit}/statefmt:{id}`, +/// so the adapter is segment one. Nothing else in the string may move for a migration to be +/// considered: a different kernel, dataset, plasticity, emulator revision, symbol provenance or +/// state format is a different *fly*, not a different reward rule. +const ADAPTER_SEGMENT: usize = 1; + +/// The environment variable that opts a deploy into the adapter migration. +/// +/// Read by flysim at restore and by `infra/05-deploy.sh`'s compatibility gate. Comma- or +/// whitespace-separated adapter ids, e.g. `FLY_ACCEPT_ADAPTERS=pokered-unique8-v5`. +pub const ACCEPT_ADAPTERS_ENV: &str = "FLY_ACCEPT_ADAPTERS"; + +/// What a build may do with a checkpoint whose compatibility string is not its own. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum RestoreDecision { + /// Byte-identical. Restore it, as every build always has. + Exact, + /// Every segment but the adapter's is identical, this build's adapter says it can migrate + /// from that one, and the operator named it in [`ACCEPT_ADAPTERS_ENV`]. Restore it. + MigrateAdapter { from: String }, + /// Refuse, and say which of the three conditions failed. + Refuse(&'static str), +} + +/// Decide whether `checkpoint`'s compatibility string may be restored under `current`. +/// +/// Three conditions, all required, in the order they are cheapest to explain: +/// +/// 1. the two strings differ in the adapter segment and **nowhere else**; +/// 2. `migrates_from` -- the running adapter's own list -- contains the checkpoint's adapter, so +/// the code that will read that state says out loud that it can; +/// 3. `accepted` -- [`ACCEPT_ADAPTERS_ENV`] as the operator set it for this deploy -- contains it +/// too, so no build ever migrates a run by itself. +/// +/// Condition 2 without condition 3 would make the migration silent; condition 3 without condition +/// 2 would let an operator wave through a pair nobody wrote a migration for. Neither alone is +/// enough, which is why both are here. +pub fn decide( + checkpoint: &str, + current: &str, + migrates_from: &[&str], + accepted: &[String], +) -> RestoreDecision { + if checkpoint == current { + return RestoreDecision::Exact; + } + let old: Vec<&str> = checkpoint.split('/').collect(); + let new: Vec<&str> = current.split('/').collect(); + if old.len() != new.len() { + return RestoreDecision::Refuse("the two compatibility strings do not have the same shape"); + } + let differing: Vec = (0..old.len()).filter(|&index| old[index] != new[index]).collect(); + if differing != [ADAPTER_SEGMENT] { + return RestoreDecision::Refuse( + "more than the adapter version differs; nothing but a reward-rule change can migrate", + ); + } + let from = old[ADAPTER_SEGMENT]; + if !migrates_from.contains(&from) { + return RestoreDecision::Refuse("this build's adapter has no migration from that adapter"); + } + if !accepted.iter().any(|name| name == from) { + return RestoreDecision::Refuse( + "the checkpoint's adapter is not in FLY_ACCEPT_ADAPTERS, so the migration was not \ + asked for", + ); + } + RestoreDecision::MigrateAdapter { from: from.to_string() } +} + +/// Parse [`ACCEPT_ADAPTERS_ENV`]: comma- or whitespace-separated, empty entries dropped. +/// +/// An unset variable and an empty one are the same thing -- no migration -- so that clearing the +/// opt-in is one edit rather than two. +pub fn accepted_adapters(value: Option<&str>) -> Vec { + value + .unwrap_or_default() + .split([',', ' ', '\t', '\n']) + .map(str::trim) + .filter(|entry| !entry.is_empty()) + .map(str::to_string) + .collect() +} + /// `-`: the two things that decide whether a /// binjgb save state written elsewhere can be memcpy'd back in here. pub fn state_format_id() -> String { @@ -94,7 +180,7 @@ mod tests { assert_eq!( fixture().prototype_string(), concat!( - "lif-1ms-f64-v2/pokered-unique8-v5/aa:bb:cc:dd:ee:ff:00/", + "lif-1ms-f64-v2/pokered-unique8-v6/aa:bb:cc:dd:ee:ff:00/", "fly-kc-mbon-rstdp-v2/", "binjgb:c60e138da5a795ebb55e56b11b7e90024e41112c/", "pokered:0cd19d3b877b7dc66d12c7050bed9a7f38154d4b", @@ -102,6 +188,73 @@ mod tests { ); } + fn with_adapter(adapter: &'static str) -> String { + Compatibility { adapter, ..fixture() }.string() + } + + #[test] + fn an_identical_string_restores_without_any_opt_in() { + let current = with_adapter("pokered-unique8-v6"); + assert_eq!(decide(¤t, ¤t, &[], &[]), RestoreDecision::Exact); + } + + #[test] + fn a_v5_checkpoint_restores_under_v6_only_with_the_opt_in() { + let old = with_adapter("pokered-unique8-v5"); + let new = with_adapter("pokered-unique8-v6"); + let migrates = ["pokered-unique8-v5"]; + + assert!(matches!(decide(&old, &new, &migrates, &[]), RestoreDecision::Refuse(_))); + assert_eq!( + decide(&old, &new, &migrates, &accepted_adapters(Some("pokered-unique8-v5"))), + RestoreDecision::MigrateAdapter { from: "pokered-unique8-v5".to_string() } + ); + // And only for a pair the running adapter says it can migrate. + assert!(matches!( + decide(&old, &new, &[], &accepted_adapters(Some("pokered-unique8-v5"))), + RestoreDecision::Refuse(_) + )); + } + + #[test] + fn nothing_but_the_adapter_segment_may_move() { + let migrates = ["pokered-unique8-v5"]; + let accepted = accepted_adapters(Some("pokered-unique8-v5")); + let new = with_adapter("pokered-unique8-v6"); + + // A different dataset, with the same adapter bump, is not a migration. + let other_dataset = Compatibility { + adapter: "pokered-unique8-v5", + dataset_fingerprint: "00:11:22:33:44:55:66", + ..fixture() + } + .string(); + assert!(matches!( + decide(&other_dataset, &new, &migrates, &accepted), + RestoreDecision::Refuse(_) + )); + + // Neither is a different kernel, and neither is a string of another shape. + let other_kernel = + Compatibility { adapter: "pokered-unique8-v5", neural_kernel_version: "lif-1ms-f64-v3", ..fixture() } + .string(); + assert!(matches!( + decide(&other_kernel, &new, &migrates, &accepted), + RestoreDecision::Refuse(_) + )); + assert!(matches!(decide("a/b", &new, &migrates, &accepted), RestoreDecision::Refuse(_))); + } + + #[test] + fn the_opt_in_list_is_separated_by_commas_or_spaces() { + assert!(accepted_adapters(None).is_empty()); + assert!(accepted_adapters(Some(" ")).is_empty()); + assert_eq!( + accepted_adapters(Some("pokered-unique8-v5, pokered-unique8-v4")), + vec!["pokered-unique8-v5".to_string(), "pokered-unique8-v4".to_string()] + ); + } + #[test] fn the_state_format_segment_is_appended_not_interleaved() { let full = fixture().string(); diff --git a/services/flysim/crates/flybrain-gb/src/pokemon_red/catalog.rs b/services/flysim/crates/flybrain-gb/src/pokemon_red/catalog.rs index d4016d3..4e7a0e6 100644 --- a/services/flysim/crates/flybrain-gb/src/pokemon_red/catalog.rs +++ b/services/flysim/crates/flybrain-gb/src/pokemon_red/catalog.rs @@ -1,4 +1,4 @@ -//! The `pokered-unique8-v5` reward catalog. +//! The `pokered-unique8-v6` reward catalog. //! //! A direct port of the prototype's `src/reward/catalog.ts`, including the //! declaration order, which is the order `counts` and `last` serialize in. @@ -20,8 +20,18 @@ pub mod kind { pub const BATTLE: &str = "battle"; pub const BADGE: &str = "badge"; pub const BOUNDARY: &str = "boundary"; + pub const CATCH: &str = "catch"; } +/// What a `catch` of a species this run has already caught pays. +/// +/// Not a multiple of the rule's catalog value, because no binary float scales 0.30 into +/// exactly 0.10: `0.3 * (1.0 / 3.0)` is `0.09999999999999999`, and that number would reach +/// the ticker, the checkpoint and `docs/rewards-learning.md`'s table as itself. `boundary`'s +/// two payouts are 0.05 and 0.10, which a scale of two does express exactly, so that rule +/// still goes through the scaling path. +pub const CATCH_REPEAT_VALUE: f64 = 0.10; + #[derive(Debug, Clone, Copy)] pub struct RewardRule { pub kind: &'static str, @@ -34,7 +44,7 @@ pub struct RewardRule { pub stimulation_ms: u32, } -pub const REWARDS: [RewardRule; 8] = [ +pub const REWARDS: [RewardRule; 9] = [ RewardRule { kind: kind::MILESTONE, label: "Story", @@ -99,6 +109,22 @@ pub const REWARDS: [RewardRule; 8] = [ value: 0.05, stimulation_ms: 100, }, + // The operator's decision of 2026-09-22: the fly is paid for *keeping* a wild Pokémon, not + // only for knocking one out. Appended rather than slotted next to `species` for the same + // reason `boundary` was appended -- the declaration order is the key order `counts` + // serializes in, and every checkpoint already written carries the first eight in this order. + // + // One rule, two payouts, like `boundary`: this value is what a species this run has never + // caught pays, and [`CATCH_REPEAT_VALUE`] is what a repeat pays. The existing `species` + // rule is untouched and still pays 0.50 the first time a species is owned by any means, so + // a first catch of a new species pays 0.50 + 0.30 across two kinds. + RewardRule { + kind: kind::CATCH, + label: "Catch", + trigger: "Wild Pokémon caught; 0.10 for a species already caught; max 3 per species", + value: 0.30, + stimulation_ms: 150, + }, ]; /// Position of `kind` in [`REWARDS`], or `None` for an unknown kind. This is @@ -199,10 +225,34 @@ mod tests { // separate kinds. assert_eq!(rule(kind::BOUNDARY).unwrap().value, 0.05); assert_eq!(rule(kind::BOUNDARY).unwrap().stimulation_ms, 100); + // Nor the prototype's: the operator's catch rule, `pokered-unique8-v6`. + assert_eq!(rule(kind::CATCH).unwrap().value, 0.30); + assert_eq!(CATCH_REPEAT_VALUE, 0.10); + assert_eq!(rule(kind::CATCH).unwrap().stimulation_ms, 150); assert!(rule("blackout").is_none(), "the catalog has no penalties"); assert!(REWARDS.iter().all(|rule| rule.value > 0.0)); } + #[test] + fn the_catch_rule_is_last_so_the_older_key_order_does_not_move() { + let order: Vec<&str> = REWARDS.iter().map(|rule| rule.kind).collect(); + assert_eq!( + order, + vec![ + kind::MILESTONE, + kind::EXPLORATION, + kind::MAP, + kind::SPECIES, + kind::TRAINER, + kind::BATTLE, + kind::BADGE, + kind::BOUNDARY, + kind::CATCH, + ] + ); + assert_eq!(index(kind::CATCH), Some(REWARDS.len() - 1)); + } + #[test] fn empty_counts_lists_every_kind_at_zero() { let counts = Counts::default(); diff --git a/services/flysim/crates/flybrain-gb/src/pokemon_red/mod.rs b/services/flysim/crates/flybrain-gb/src/pokemon_red/mod.rs index 469d708..7d4058f 100644 --- a/services/flysim/crates/flybrain-gb/src/pokemon_red/mod.rs +++ b/services/flysim/crates/flybrain-gb/src/pokemon_red/mod.rs @@ -1,11 +1,12 @@ -//! The Pokémon Red reward adapter, `pokered-unique8-v5`. +//! The Pokémon Red reward adapter, `pokered-unique8-v6`. //! //! A port of the prototype's `src/reward/pokemon-red.ts`. The gates and budgets //! are unchanged; `docs/rewards-learning.md` holds the live rule table and the //! source evidence behind each gate. v4 replaced the 16-rung boot-to-badges //! ladder with the 38 rungs of `docs/design/ladder.md`; v5 adds one reward rule, //! `boundary` (`docs/design/room-escape.md` section 2), which pays the first step -//! next to and the first step onto each of a map's exits. +//! next to and the first step onto each of a map's exits; v6 adds `catch`, the +//! operator's decision of 2026-09-22, which pays for keeping a wild Pokémon. pub mod catalog; #[cfg(test)] @@ -32,13 +33,33 @@ use symbols::ram; /// Adapter version, pinned into the checkpoint compatibility string. /// -/// `v5` is the `boundary` rule. Bumping it is what rejects every checkpoint written -/// by `v4`: the string is compared whole before a restore is attempted, so a ledger -/// that has never recorded a single `boundary:` key can never be resumed as though -/// its exits were already collected. (`v4` was the 38-rung ladder, and rejected -/// `v3` for the same reason: a stored rank that meant "4 badges" on the old ladder -/// could not be read as a rung on the new one.) Pre-launch, so no run is lost. -pub const REWARD_ADAPTER: &str = "pokered-unique8-v5"; +/// `v6` is the `catch` rule. Bumping it is what makes a `v5` checkpoint a decision +/// rather than an accident: the compatibility string is compared whole before a +/// restore is attempted, so a `v5` run is refused by default and resumed only when +/// the operator names it in `FLY_ACCEPT_ADAPTERS` +/// ([`crate::compatibility::RestoreDecision`], `docs/design/flysim.md`). That +/// migration is safe in one direction only, and only for this pair: `v5`'s ledger is +/// a `v6` ledger with the catch counter absent, and an absent counter reads as zero. +/// +/// (`v5` was the `boundary` rule, and rejected `v4` because a ledger that had never +/// recorded a `boundary:` key could not be resumed as though its exits were already +/// collected. `v4` was the 38-rung ladder, and rejected `v3` because a stored rank +/// that meant "4 badges" on the old ladder is not a rung on the new one. Neither of +/// those is a migration: this one is, because nothing a `v5` ledger holds means +/// something different under `v6`.) +pub const REWARD_ADAPTER: &str = "pokered-unique8-v6"; + +/// Adapter ids whose checkpoints `v6` can read. +/// +/// Exactly one, and it is one because the `catch` rule adds a counter and changes nothing else: +/// a `v5` ledger restores as a `v6` ledger with `catchCounts` empty, and every other byte of the +/// state means what it meant. `v4` is not here -- its `seen` ledger holds no `boundary:` keys, so +/// resuming it would pay a second time for every exit the run had already found -- and neither is +/// `v3`, whose stored rank is a rung on a different ladder. +/// +/// Listing an id here is necessary but not sufficient: `FLY_ACCEPT_ADAPTERS` must name it too +/// (`crate::compatibility::decide`, `docs/design/flysim.md`). +pub const MIGRATES_FROM: &[&str] = &["pokered-unique8-v5"]; /// The only cartridge semantic rewards are enabled for. Even the canonical /// pret build stays disabled until reviewed; see `docs/rewards-learning.md`. @@ -60,8 +81,22 @@ pub const SUPPORTED_ROM: &str = /// [`REWARD_ADAPTER`] is the gate that refuses such a checkpoint anyway, and it is /// the right gate, because the objection to loading one is about semantics rather /// than shape. +/// +/// *Not* bumped for the `catch` rule either, and this time the answer matters, +/// because `v5` checkpoints are meant to be restorable under `v6`. The rule adds one +/// counter, `catchCounts`, and nothing else: every other field keeps its name, its +/// shape and its meaning, and a state written without the counter restores with it +/// empty, which is the truth about a run that was never paid for a catch. That is the +/// whole of the documented `v5` -> `v6` migration; see +/// [`crate::compatibility::RestoreDecision`]. pub const STATE_VERSION: u64 = 4; +/// Catch payouts one species may earn in the lifetime of a run's ledger. +/// +/// The same cap and the same reason as the wild-KO rule's three: a species the fly can +/// find over and over is a farm, and three is enough for the behaviour to be learned. +const MAX_CATCH_PAYOUTS: u64 = 3; + const BADGE_NAMES: [&str; 8] = [ "BOULDER", "CASCADE", "THUNDER", "RAINBOW", "SOUL", "MARSH", "VOLCANO", "EARTH", ]; @@ -325,6 +360,26 @@ struct Battle { wild: bool, saw_living: bool, ko: bool, + /// Lifetime `species` payouts when this battle started. + /// + /// The "never owned this run" test for the catch rule, and an exact one: the only + /// thing that can set a `wPokedexOwned` bit during a wild battle is the catch + /// itself, so a `species` payout between the battle starting and the ball keeping + /// the Pokémon *is* that Pokémon being new. It is read this way rather than from + /// `wCapturedMonSpecies` directly because that byte is the cartridge's **internal** + /// species index and the owned bitset is by **Pokédex number**; the two numberings + /// differ and nothing in WRAM converts between them + /// (`docs/design/macros-wram.md` section 2, "species numbering"). + /// + /// `None` for a battle restored from a checkpoint written before this existed, + /// which reads as "cannot tell" and pays the repeat amount rather than guessing + /// generously. + species_at_start: Option, + /// The internal species index `wCapturedMonSpecies` named, once a ball has kept one. + captured: Option, + /// Whether that catch was a species this run had never owned, decided on the frame + /// the capture was observed. + captured_new: bool, } /// Immutable per-sample byte cache. Each address requested during one sample @@ -370,6 +425,10 @@ pub struct PokemonRedReward { tiles: OrderedSet, tile_counts: BTreeMap, wild_wins: BTreeMap, + /// Catch payouts per species, by the cartridge's internal species index as a decimal + /// string. The one field `v6` adds to the checkpoint; absent in a `v5` state, which + /// reads as every species at zero. + catch_counts: BTreeMap, replay_blocked: OrderedSet, counts: Counts, total: f64, @@ -420,6 +479,7 @@ impl PokemonRedReward { tiles: OrderedSet::new(), tile_counts: BTreeMap::new(), wild_wins: BTreeMap::new(), + catch_counts: BTreeMap::new(), replay_blocked: OrderedSet::new(), counts: Counts::default(), total: 0.0, @@ -567,8 +627,9 @@ impl PokemonRedReward { } /// Forget observations a rollback invalidates. Lifetime novelty survives, - /// and every wild-KO key paid so far is blocked from paying again, because - /// after a rollback the same battle could otherwise be replayed for reward. + /// and every wild-KO key and every caught species paid so far is blocked from + /// paying again, because after a rollback the same battle -- or the same catch -- + /// could otherwise be replayed for reward. pub fn clear_transient(&mut self) { self.location.clear(); self.stable = 0; @@ -578,6 +639,10 @@ impl PokemonRedReward { for key in keys { self.replay_blocked.insert(&key); } + let caught: Vec = self.catch_counts.keys().cloned().collect(); + for species in caught { + self.replay_blocked.insert(&format!("catch:{species}")); + } } /// Sample WRAM after one completed frame and return this frame's payouts. @@ -690,14 +755,29 @@ impl PokemonRedReward { if in_battle == 1 || in_battle == 2 || in_battle == 255 { self.mode = "BATTLE".to_string(); self.stable = 0; + let species_paid = self.counts.get(kind::SPECIES); if self.battle.is_none() && in_battle != 255 { self.battle = Some(Battle { key: battle_key(memory, map), wild: in_battle == 1, saw_living: false, ko: false, + species_at_start: Some(species_paid), + captured: None, + captured_new: false, }); } + // The cartridge's own answer to "was one caught": `ram/wram.asm`'s comment on + // this byte is "0 if no mon was captured". `ItemUseBall` zeroes it before every + // throw and writes `wEnemyMonSpecies` into it only on the branch that keeps the + // Pokémon, and `UseBagItem`'s `.returnAfterCapturingMon` zeroes it again on the + // way out of the battle -- so it is non-zero for the hundreds of frames the + // catch's own text and Pokédex screen take, and zero everywhere else. + // + // Read rather than derived from `wPartyCount`, because a catch with a full party + // raises `wBoxCount` instead, and because `wPartyCount` also rises for a gift, a + // trade and a revive out of the PC. + let captured = memory.read8(ram::wCapturedMonSpecies); if let Some(battle) = &mut self.battle { let hp = word(memory, ram::wEnemyMonHP); let max = word(memory, ram::wEnemyMonMaxHP); @@ -710,6 +790,11 @@ impl PokemonRedReward { if battle.saw_living && hp == 0 { battle.ko = true; } + if battle.wild && captured != 0 && battle.captured.is_none() { + battle.captured = Some(captured); + battle.captured_new = + battle.species_at_start.is_some_and(|before| species_paid > before); + } } } else if in_battle == 0 { self.mode = "OVERWORLD".to_string(); @@ -728,6 +813,41 @@ impl PokemonRedReward { } self.wild_wins.insert(battle.key.clone(), (count + 1).min(3)); } + // The catch rule (`docs/rewards-learning.md`, the operator 2026-09-22). + // + // Paid on the way out of the battle rather than on the capture frame, so that + // it lands in the same place the wild-KO payout does and cannot fire twice for + // one battle. `wBattleResult` is 2 on exactly two paths in the game: + // `UseBagItem`'s `.returnAfterCapturingMon`, which is this one, and a link + // battle whose opponent ran (`engine/battle/core.asm`), which this cartridge + // never has. Requiring it as well as the captured species means a byte read + // out of a half-initialised battle cannot pay. + if let Some(species) = battle.captured + && battle.wild + && result == 2 + { + let key = species.to_string(); + let paid = self.catch_counts.get(&key).copied().unwrap_or(0); + if paid < MAX_CATCH_PAYOUTS + && !self.replay_blocked.contains(&format!("catch:{key}")) + { + let value = if battle.captured_new { + catalog::rule(kind::CATCH) + .expect("the catch rule is in the catalog") + .value + } else { + catalog::CATCH_REPEAT_VALUE + }; + self.emit_amount( + &mut emitted, + kind::CATCH, + format!("CAUGHT #{species}"), + value, + brain_ms, + ); + } + self.catch_counts.insert(key, (paid + 1).min(MAX_CATCH_PAYOUTS)); + } } let location = format!("{map}:{x}:{y}"); self.stable = if self.location == location { self.stable + 1 } else { 1 }; @@ -825,13 +945,33 @@ impl PokemonRedReward { label: String, scale: f64, brain_ms: f64, + ) { + let rule = catalog::rule(kind).expect("emit is only called with catalog kinds"); + self.emit_amount(emitted, kind, label, rule.value * scale, brain_ms); + } + + /// [`PokemonRedReward::emit`] with the payout stated outright instead of as a multiple + /// of the catalog value. + /// + /// One rule needs it. `catch` pays 0.30 for a species this run has not caught and 0.10 + /// for one it has, and no binary float scales the first into exactly the second: + /// `0.3 * (1.0 / 3.0)` is `0.09999999999999999`, and that is the number that would reach + /// the ticker and the checkpoint. `boundary`'s pair, 0.05 and 0.10, *is* an exact scale + /// of two, so that rule still goes through [`PokemonRedReward::emit`]. + fn emit_amount( + &mut self, + emitted: &mut Vec, + kind: &'static str, + label: String, + value: f64, + brain_ms: f64, ) { let rule = catalog::rule(kind).expect("emit is only called with catalog kinds"); let event = RewardEvent { kind, label, brain_ms, - value: rule.value * scale, + value, stimulation_ms: rule.stimulation_ms, }; emitted.push(event.clone()); @@ -991,6 +1131,10 @@ impl PokemonRedReward { "tiles": self.tiles.as_slice(), "tileCounts": self.tile_counts, "wildWins": self.wild_wins, + // The one field v6 adds. A v5 state does not carry it and restores with it + // empty, which is the documented v5 -> v6 migration and the truth about a run + // that was never paid for a catch. + "catchCounts": self.catch_counts, "replayBlocked": self.replay_blocked.as_slice(), "counts": self.counts, "total": self.total, @@ -1011,6 +1155,9 @@ impl PokemonRedReward { "wild": battle.wild, "sawLiving": battle.saw_living, "ko": battle.ko, + "speciesAtStart": battle.species_at_start, + "captured": battle.captured, + "capturedNew": battle.captured_new, })), "mode": self.mode, }) @@ -1056,6 +1203,13 @@ impl PokemonRedReward { let counts_raw = counted_record(input.get("counts")).ok_or(BAD_CHECKPOINT)?; let tile_counts_raw = counted_record(input.get("tileCounts")).ok_or(BAD_CHECKPOINT)?; let wild_wins_raw = counted_record(input.get("wildWins")).ok_or(BAD_CHECKPOINT)?; + // Absent in every v5 state, and that absence is the migration: no species has been + // paid for a catch, because the rule did not exist. Present but malformed is still + // an error, the same as every other counter here. + let catch_counts = match input.get("catchCounts") { + None | Some(Value::Null) => BTreeMap::new(), + Some(value) => counted_record(Some(value)).ok_or(BAD_CHECKPOINT)?, + }; let recent = recent_raw .iter() @@ -1078,6 +1232,19 @@ impl PokemonRedReward { wild: value.get("wild").and_then(Value::as_bool).ok_or(BAD_HISTORY)?, saw_living: value.get("sawLiving").and_then(Value::as_bool).ok_or(BAD_HISTORY)?, ko: value.get("ko").and_then(Value::as_bool).ok_or(BAD_HISTORY)?, + // All three are v6's, and all three are optional for the same reason + // `catchCounts` is. A v5 battle carries no `speciesAtStart`, which reads as + // "cannot tell whether the caught species was new" and pays the repeat + // amount: the conservative half of the rule, and at most 0.20 once. + species_at_start: value.get("speciesAtStart").and_then(Value::as_u64), + captured: value + .get("captured") + .and_then(Value::as_u64) + .and_then(|species| u8::try_from(species).ok()), + captured_new: value + .get("capturedNew") + .and_then(Value::as_bool) + .unwrap_or(false), }), }; let replay_blocked = match input.get("replayBlocked") { @@ -1103,6 +1270,7 @@ impl PokemonRedReward { } self.tile_counts = tile_counts; self.wild_wins = wild_wins_raw; + self.catch_counts = catch_counts; self.replay_blocked = replay_blocked.iter().map(String::as_str).collect(); self.counts = Counts::default(); for (key, count) in &counts_raw { @@ -1174,6 +1342,10 @@ impl GameAdapter for PokemonRedReward { REWARD_ADAPTER } + fn migrates_from(&self) -> &'static [&'static str] { + MIGRATES_FROM + } + fn rom_allowed(&self, sha256: &str) -> bool { sha256 == SUPPORTED_ROM } diff --git a/services/flysim/crates/flybrain-gb/src/pokemon_red/symbols.rs b/services/flysim/crates/flybrain-gb/src/pokemon_red/symbols.rs index 30119fa..014b4ad 100644 --- a/services/flysim/crates/flybrain-gb/src/pokemon_red/symbols.rs +++ b/services/flysim/crates/flybrain-gb/src/pokemon_red/symbols.rs @@ -47,6 +47,7 @@ pub mod ram { pub const wBattleType: u16 = 0xd05a; // 53338 pub const wTrainerNo: u16 = 0xd05d; // 53341 pub const wPartyMenuTypeOrMessageID: u16 = 0xd07d; // 53373 + pub const wCapturedMonSpecies: u16 = 0xd11c; // 53532 pub const wForcePlayerToChooseMon: u16 = 0xd11f; // 53535 pub const wTextBoxID: u16 = 0xd125; // 53541 pub const wPartyCount: u16 = 0xd163; // 53603 diff --git a/services/flysim/crates/flybrain-gb/src/pokemon_red/tests.rs b/services/flysim/crates/flybrain-gb/src/pokemon_red/tests.rs index e0bede2..87559c1 100644 --- a/services/flysim/crates/flybrain-gb/src/pokemon_red/tests.rs +++ b/services/flysim/crates/flybrain-gb/src/pokemon_red/tests.rs @@ -83,6 +83,34 @@ impl Fixture { self.visit(x, 0) } + /// One wild battle that ends in a ball keeping the Pokémon, byte for byte as the + /// cartridge writes it at the pinned commit. + /// + /// `InitBattleVariables` clears `wBattleResult`; `ItemUseBall`'s capture branch sets the + /// Pokédex bit (for a species the player did not already own) and writes + /// `wEnemyMonSpecies` into `wCapturedMonSpecies`; `UseBagItem`'s + /// `.returnAfterCapturingMon` then zeroes that byte, sets `wBattleResult` to 2 and leaves + /// the battle. `dex` is the Pokédex *number* minus one, i.e. the bit index, and `None` is a + /// species this run already owns. + fn catch(&mut self, species: u8, dex: Option) -> Vec { + self.memory.set(ram::wBattleResult, 0); + self.memory.set(ram::wIsInBattle, 1); + self.memory.set(ram::wEnemyMonSpecies, species); + self.memory.set(ram::wEnemyMonHP + 1, 10); + self.memory.set(ram::wEnemyMonMaxHP + 1, 10); + let mut events = self.sample(); + if let Some(index) = dex { + self.memory.or(ram::wPokedexOwned + (index >> 3), 1 << (index & 7)); + } + self.memory.set(ram::wCapturedMonSpecies, species); + events.extend(self.sample()); + self.memory.set(ram::wCapturedMonSpecies, 0); + self.memory.set(ram::wBattleResult, 2); + self.memory.set(ram::wIsInBattle, 0); + events.extend(self.sample()); + events + } + /// Write a warp table: `wNumberOfWarps` plus one four-byte `Y, X, warp id, map id` entry per /// `(x, y)`, the layout `ram/wram.asm` documents at the pinned commit. fn warps(&mut self, warps: &[(u8, u8)]) { @@ -320,6 +348,136 @@ fn a_wild_run_capture_or_single_faint_never_pays_a_ko_while_a_verified_ko_does() assert_eq!(f.reward.statistics().counts[kind::BATTLE], 1); } +#[test] +fn a_catch_pays_the_new_species_amount_once_the_repeat_amount_after_and_stops_at_three() { + let mut f = Fixture::new(); + f.sample(); + + // A species this run has never owned: the cartridge sets the Pokédex bit on the way + // through, so the existing `species` rule pays 0.50 and the new rule pays 0.30. + let first = f.catch(0xb0, Some(3)); + assert_eq!(kinds(&first), ["species", "catch"]); + assert_eq!(labels(&first), ["OWNED #4", "CAUGHT #176"]); + assert!((first[0].value - 0.5).abs() < 1e-12, "the species rule is untouched"); + assert!((first[1].value - 0.30).abs() < 1e-12); + + // The same species again: a repeat, twice, and then the cap. + for _ in 0..2 { + let again = f.catch(0xb0, None); + assert_eq!(kinds(&again), ["catch"]); + assert_eq!(again[0].value, 0.10, "the repeat amount is exactly 0.10, not 0.3/3"); + } + assert!(f.catch(0xb0, None).is_empty(), "three payouts per species is the cap"); + assert_eq!(f.reward.statistics().counts[kind::CATCH], 3); + + // Another species starts its own count, and its own 0.30. + let other = f.catch(0x99, Some(0)); + assert_eq!(kinds(&other), ["species", "catch"]); + assert!((other[1].value - 0.30).abs() < 1e-12); +} + +#[test] +fn a_catch_of_a_species_this_run_already_owns_pays_the_repeat_amount() { + let mut f = Fixture::new(); + f.sample(); + // Owned before the battle -- a gift, a trade, an evolution -- so no Pokédex bit is set + // during it and the catch is not a new species. + f.memory.or(ram::wPokedexOwned, 1); + assert_eq!(kinds(&f.sample()), ["species"]); + + let events = f.catch(0x99, None); + assert_eq!(kinds(&events), ["catch"]); + assert_eq!(events[0].value, 0.10); +} + +#[test] +fn nothing_but_a_wild_catch_pays_the_catch_rule() { + // A trainer battle: balls cannot be thrown, and `wIsInBattle` is 2. + let mut f = Fixture::new(); + f.sample(); + f.memory.set(ram::wIsInBattle, 2); + f.memory.set(ram::wEnemyMonHP + 1, 10); + f.memory.set(ram::wEnemyMonMaxHP + 1, 10); + f.sample(); + f.memory.set(ram::wCapturedMonSpecies, 0xb0); + f.sample(); + f.memory.set(ram::wCapturedMonSpecies, 0); + f.memory.set(ram::wBattleResult, 2); + f.memory.set(ram::wIsInBattle, 0); + assert!(f.sample().is_empty(), "a trainer battle never pays the catch rule"); + + // The Safari Zone and the old man's tutorial are excluded a step earlier: the whole + // sample is dropped with a visible mode, so no battle is ever opened. + for battle_type in [1u8, 2] { + let mut f = Fixture::new(); + f.sample(); + f.memory.set(ram::wBattleType, battle_type); + f.memory.set(ram::wIsInBattle, 1); + assert!(f.sample().is_empty()); + f.memory.set(ram::wCapturedMonSpecies, 0xb0); + assert!(f.sample().is_empty()); + f.memory.set(ram::wCapturedMonSpecies, 0); + f.memory.set(ram::wBattleResult, 2); + f.memory.set(ram::wIsInBattle, 0); + f.memory.set(ram::wBattleType, 0); + assert!(f.sample().is_empty()); + assert_eq!(f.reward.statistics().counts[kind::CATCH], 0); + } + + // A ball that missed: `wCapturedMonSpecies` never leaves zero and the battle ends as a + // run or a loss. + let mut f = Fixture::new(); + f.sample(); + f.memory.set(ram::wIsInBattle, 1); + f.memory.set(ram::wEnemyMonHP + 1, 10); + f.memory.set(ram::wEnemyMonMaxHP + 1, 10); + f.sample(); + f.memory.set(ram::wIsInBattle, 0); + assert!(f.sample().is_empty()); +} + +#[test] +fn a_rollback_cannot_replay_a_catch() { + let mut f = Fixture::new(); + f.sample(); + assert_eq!(kinds(&f.catch(0xb0, Some(3))), ["species", "catch"]); + + f.reward.clear_transient(); + let state = f.reward.export_state(); + f.reward.import_state(&state).unwrap(); + assert!(f.catch(0xb0, None).is_empty(), "an already-paid species cannot pay after rollback"); + assert_eq!(f.reward.statistics().counts[kind::CATCH], 1); +} + +#[test] +fn a_v5_state_restores_under_v6_with_the_catch_counter_at_zero() { + let mut f = Fixture::new(); + f.sample(); + f.catch(0xb0, Some(3)); + let v6 = f.reward.export_state(); + assert_eq!(v6["catchCounts"], json!({ "176": 1 })); + + // The v5 shape is this one without the counter the rule added: same `version`, same field + // names, same meanings. That is the whole of the documented migration. + let mut v5 = v6.clone(); + v5.as_object_mut().unwrap().remove("catchCounts"); + assert_eq!(v5["version"], json!(STATE_VERSION), "v5 and v6 states share a schema version"); + + let mut restored = PokemonRedReward::new(); + restored.import_state(&v5).unwrap(); + let mut expected = v6.clone(); + expected["catchCounts"] = json!({}); + assert_eq!(restored.export_state(), expected, "the counter starts at 0, nothing else moves"); + + // A genuine v5 `counts` object carries eight kinds and no `catch`, which reads as zero. + let mut older = v5.clone(); + older["counts"].as_object_mut().unwrap().remove("catch"); + let mut restored = PokemonRedReward::new(); + restored.import_state(&older).unwrap(); + assert_eq!(restored.statistics().counts[kind::CATCH], 0); + assert_eq!(restored.statistics().counts[kind::SPECIES], 1); +} + #[test] fn a_repeated_wild_ko_decays_then_stops() { let mut f = Fixture::new(); @@ -378,6 +536,7 @@ fn malformed_checkpoint_fields_are_named_in_the_error() { ("counts", json!([])), ("tileCounts", json!("not a record")), ("wildWins", json!(3)), + ("catchCounts", json!(3)), ] { let mut broken = good.clone(); broken[field] = wrong; @@ -706,7 +865,8 @@ fn the_recent_ticker_keeps_the_newest_eight_events_newest_first() { #[test] fn the_adapter_reports_its_identity_and_pinned_rom() { let reward = PokemonRedReward::new(); - assert_eq!(reward.id(), "pokered-unique8-v5"); + assert_eq!(reward.id(), "pokered-unique8-v6"); + assert_eq!(reward.migrates_from(), ["pokered-unique8-v5"]); assert!(reward.rom_allowed(SUPPORTED_ROM)); assert!(!reward.rom_allowed( "5ca7ba01642a3b27b0cc0b5349b52792795b62d3ed977e98a09390659af96b7b" @@ -716,6 +876,9 @@ fn the_adapter_reports_its_identity_and_pinned_rom() { assert_eq!(symbols::ram::wNumberOfWarps, 0xd3ae); assert_eq!(symbols::ram::wWarpEntries, 0xd3af); assert_eq!(symbols::ram::wCurMapConnections, 0xd370); + // Resolved from ram/wram.asm by services/flysim/tools/resolve_wram.py, bracketed by + // wFontLoaded and wForcePlayerToChooseMon; never written out by hand. + assert_eq!(symbols::ram::wCapturedMonSpecies, 0xd11c); assert_eq!(symbols::MILESTONES.len(), 17); } diff --git a/services/flysim/crates/flysim/src/lib.rs b/services/flysim/crates/flysim/src/lib.rs index 65a1b70..be481b0 100644 --- a/services/flysim/crates/flysim/src/lib.rs +++ b/services/flysim/crates/flysim/src/lib.rs @@ -30,6 +30,7 @@ pub mod metrics; pub mod pacing; pub mod profile; pub mod ratelimit; +pub mod reset; pub mod sdnotify; pub mod simloop; pub mod snapshot; diff --git a/services/flysim/crates/flysim/src/main.rs b/services/flysim/crates/flysim/src/main.rs index 5d2aca3..b0c4a89 100644 --- a/services/flysim/crates/flysim/src/main.rs +++ b/services/flysim/crates/flysim/src/main.rs @@ -42,6 +42,15 @@ struct Args { /// the "nothing to compare" case for a fresh container. #[arg(long, value_name = "DIR")] print_state_compatibility: Option, + + /// Restart the run from the milestone archive for this ladder rung, and exit. + /// + /// Run with flysim stopped: it rewrites both checkpoint stores. + /// `infra/bin/fly-reset-to-milestone` is the operator-facing wrapper and the sequence + /// around it is in `infra/docs/runbook.md`. The current state is copied to a dated + /// directory first, so this is reversible by hand. + #[arg(long, value_name = "RANK")] + reset_to_milestone: Option, } fn main() -> Result<()> { @@ -66,6 +75,17 @@ fn main() -> Result<()> { return Ok(()); } + if let Some(rank) = args.reset_to_milestone { + let stamp = flysim::reset::utc_stamp(flysim::eventlog::now_wall_ms()); + let durable = config.paths.save_dir.clone(); + let hot = config.paths.hot_dir.clone(); + let archive = flysim::reset::default_archive_dir(&durable, &stamp); + for line in flysim::reset::reset_to_milestone(&durable, &hot, rank, &archive)? { + println!("{line}"); + } + return Ok(()); + } + flysim::run(config) } diff --git a/services/flysim/crates/flysim/src/reset.rs b/services/flysim/crates/flysim/src/reset.rs new file mode 100644 index 0000000..acf575e --- /dev/null +++ b/services/flysim/crates/flysim/src/reset.rs @@ -0,0 +1,450 @@ +//! Restarting a run from an earlier rung, on disk, with flysim stopped. +//! +//! The operator's decision of 2026-09-22 was "restart the live run from an early checkpoint +//! instead of from scratch". `FLY_RESET_STATE=1` cannot do that: it archives everything and the +//! next start warms up a fresh fly. What this does instead is promote one milestone archive -- +//! `milestone-.checkpoint`, which `store::Store::commit` writes at the first commit at a new +//! best rank and which no rotation ever unlinks -- to being the only thing either store will +//! restore. +//! +//! `infra/bin/fly-reset-to-milestone` is the operator-facing wrapper; it refuses to run while +//! flysim is up, calls `flysim --reset-to-milestone N`, and fixes ownership afterwards. The work +//! is here rather than in that script because two of the steps are inside the `FLYSIM01` +//! envelope: the ratchet's attempts and recoveries counters live in the checkpoint's manifest, +//! and a shell script has no business rewriting one. +//! +//! What it does, in order, and nothing else: +//! +//! 1. **archives** every file in the durable and hot stores into a dated directory, by copying, +//! so a step that fails later has destroyed nothing; +//! 2. **rewrites** the rung's archive with the ratchet's `attempts` and `recoveries` at zero, so +//! the recovery budget is not already spent when the restarted run begins. `best` is left +//! alone: the archive's own `best` is the rung it was taken at, which is exactly what the +//! restarted run is at, and the rank the stream shows is recomputed by the adapter from the +//! restored game state anyway; +//! 3. **installs** it as the newest generation in both stores, so the restore order +//! (`store::restore_order`: hot latest, hot previous, durable latest, ...) reaches it first; +//! 4. **clears** the milestone archives above N -- rungs the run had reached and is now below -- +//! and the generation files of the run being abandoned; +//! 5. **clears the session ledgers**: the event log `events.jsonl` and its rotations. The +//! checkpoint carries `lastEventId`, so restoring an old checkpoint over a newer log would +//! re-issue ids the log already holds. The macro layer's own session ledgers (blocked, +//! talked, reached, pushed-back) are memory-only by contract +//! (`docs/design/macros.md` section 12.1: "a restored run offers every target once more"), +//! so stopping flysim is what resets those and this has nothing to do. + +use std::collections::BTreeMap; +use std::path::{Path, PathBuf}; + +use anyhow::{Context, Result, bail}; + +use crate::store::{self, Store, StoreManifest}; + +/// Generations kept by the stores this tool writes. Only used for the rotation bound, which +/// this tool does not trigger; the durable store's own value. +const KEEP_GENERATIONS: usize = 8; + +/// A generation number no commit ever allocates, so `Store::candidates` drops the +/// generation-file half of an archive entry and offers only `milestone-.checkpoint`. +/// +/// `Sim::boot` allocates `highest_generation() + 1`, which is 1 or more, so 0 names no file. +/// The milestone archives below the rung being restored are kept exactly this way: still on +/// disk, still restorable as a deeper fallback, and with no generation file pretending to be +/// their contents. +const NO_GENERATION: u64 = 0; + +/// What the reset did, one line per step, for the operator's terminal and the run record. +pub type Report = Vec; + +/// Promote `rank`'s milestone archive to be the restore source of both stores. +/// +/// `archive` is the dated directory the current state is copied into; it must not exist. +/// Refuses if the milestone archive is missing, which is the "that rung was never reached" +/// case and the one mistake worth refusing rather than guessing at. +pub fn reset_to_milestone( + durable_dir: &Path, + hot_dir: &Path, + rank: u32, + archive: &Path, +) -> Result { + let durable = Store::new(durable_dir, KEEP_GENERATIONS); + let hot = Store::new(hot_dir, KEEP_GENERATIONS); + let source = durable.archive_path(rank); + if !source.is_file() { + bail!( + "no milestone archive for rung {rank}: {} does not exist. `ls {}` shows the rungs \ + this run actually reached.", + source.display(), + durable_dir.display() + ); + } + if archive.exists() { + bail!("the archive directory {} already exists", archive.display()); + } + + let mut report: Report = Vec::new(); + + // 1. Copy everything aside first. + let copied_durable = copy_tree(durable_dir, &archive.join("durable"))?; + let copied_hot = copy_tree(hot_dir, &archive.join("hot"))?; + report.push(format!( + "archived {copied_durable} durable and {copied_hot} hot files to {}", + archive.display() + )); + + // 2. Zero the two recovery counters inside the envelope. + let mut checkpoint = store::load(&source) + .with_context(|| format!("decoding {}", source.display()))?; + let spent = (checkpoint.runtime.ratchet.attempts, checkpoint.runtime.ratchet.recoveries); + checkpoint.runtime.ratchet.attempts = 0; + checkpoint.runtime.ratchet.recoveries = 0; + + let generation = durable.highest_generation().max(hot.highest_generation()) + 1; + checkpoint.runtime.generation = generation; + let bytes = store::encode(&checkpoint.agent, &checkpoint.runtime)?; + report.push(format!( + "rung {rank} (best {}, ladder rank recomputed from the game state): ratchet attempts \ + {} -> 0, recoveries {} -> 0", + checkpoint.runtime.ratchet.best, spent.0, spent.1 + )); + + // 3/4. Clear both stores, keeping the milestone archives at or below this rung, and write + // the promoted state as the newest generation of each. + let kept = clear_store(durable_dir, Some(rank))?; + clear_store(hot_dir, None)?; + report.push(format!( + "cleared the hot store and every milestone archive above rung {rank}; kept {} at or \ + below it: {kept:?}", + kept.len() + )); + + // The hot store lives on a tmpfs that a stopped container may not have mounted yet, so it + // is created rather than assumed; the durable one already exists or the milestone archive + // above could not have been read. + durable.create()?; + hot.create()?; + store::write_atomic(&durable.generation_path(generation), &bytes)?; + store::write_atomic(&durable.archive_path(rank), &bytes)?; + store::write_atomic(&hot.generation_path(generation), &bytes)?; + + let mut archives: BTreeMap = + kept.iter().map(|rung| (*rung, NO_GENERATION)).collect(); + archives.insert(rank, generation); + let durable_manifest = StoreManifest { + generation, + latest: Some(generation), + previous: None, + archives, + }; + write_manifest(&durable, &durable_manifest)?; + write_manifest( + &hot, + &StoreManifest { + generation, + latest: Some(generation), + previous: None, + archives: BTreeMap::new(), + }, + )?; + report.push(format!( + "generation {generation} is now hot latest and durable latest in {} and {}", + hot_dir.display(), + durable_dir.display() + )); + + // 5. The session ledgers. + let logs = remove_matching(durable_dir, |name| { + name == "events.jsonl" || (name.starts_with("events-") && name.ends_with(".jsonl")) + })?; + report.push(format!( + "reset the session ledgers: {logs} event-log files removed (the macro layer's are \ + memory-only and are reset by stopping flysim)" + )); + + Ok(report) +} + +fn write_manifest(store: &Store, manifest: &StoreManifest) -> Result<()> { + store.create()?; + store::write_atomic(&store.manifest_path(), &serde_json::to_vec_pretty(manifest)?) +} + +/// Copy every regular file of `from` into `to`, creating `to`. A missing source is zero files, +/// not an error: the hot store lives on a tmpfs that a stopped container does not have. +fn copy_tree(from: &Path, to: &Path) -> Result { + if !from.is_dir() { + return Ok(0); + } + std::fs::create_dir_all(to) + .with_context(|| format!("creating {}", to.display()))?; + let mut copied = 0; + for entry in std::fs::read_dir(from)?.flatten() { + if !entry.file_type().is_ok_and(|kind| kind.is_file()) { + continue; + } + std::fs::copy(entry.path(), to.join(entry.file_name())) + .with_context(|| format!("copying {}", entry.path().display()))?; + copied += 1; + } + Ok(copied) +} + +/// Remove every checkpoint, tmp file and manifest from `dir`, keeping `milestone-.checkpoint` +/// for `r <= keep_up_to`. Returns the rungs kept, ascending. +fn clear_store(dir: &Path, keep_up_to: Option) -> Result> { + let mut kept = Vec::new(); + if !dir.is_dir() { + return Ok(kept); + } + for entry in std::fs::read_dir(dir)?.flatten() { + let name = entry.file_name().to_string_lossy().to_string(); + let milestone = name + .strip_prefix("milestone-") + .and_then(|rest| rest.strip_suffix(".checkpoint")) + .and_then(|rung| rung.parse::().ok()); + let remove = match milestone { + // The promoted rung's own archive is rewritten straight after this, so it is + // removed here like the rest and reinstated with the counters cleared. + Some(rung) => match keep_up_to { + Some(limit) if rung < limit => { + kept.push(rung); + false + } + _ => true, + }, + None => { + name == "manifest.json" + || name.ends_with(".checkpoint") + || name.ends_with(".checkpoint.tmp") + } + }; + if remove { + std::fs::remove_file(entry.path()) + .with_context(|| format!("removing {}", entry.path().display()))?; + } + } + kept.sort_unstable(); + Ok(kept) +} + +fn remove_matching(dir: &Path, wanted: impl Fn(&str) -> bool) -> Result { + if !dir.is_dir() { + return Ok(0); + } + let mut removed = 0; + for entry in std::fs::read_dir(dir)?.flatten() { + if wanted(&entry.file_name().to_string_lossy()) { + std::fs::remove_file(entry.path())?; + removed += 1; + } + } + Ok(removed) +} + +/// `YYYYMMDDTHHMMSSZ` in UTC, for the dated archive directory's name. +/// +/// Built on the event log's own calendar conversion, which is the service's only one; the time of +/// day is arithmetic on the same millisecond count. +pub fn utc_stamp(wall_ms: u64) -> String { + let second_of_day = (wall_ms % 86_400_000) / 1_000; + format!( + "{}T{:02}{:02}{:02}Z", + crate::eventlog::utc_day(wall_ms), + second_of_day / 3_600, + (second_of_day / 60) % 60, + second_of_day % 60, + ) +} + +/// The default dated archive directory: a sibling of the durable store, which is its own +/// mountpoint and so cannot be renamed -- the same shape `infra/05-deploy.sh` uses for +/// `FLY_RESET_STATE=1`. +pub fn default_archive_dir(durable_dir: &Path, stamp: &str) -> PathBuf { + let mut name = durable_dir.as_os_str().to_os_string(); + name.push(format!(".reset-{stamp}")); + PathBuf::from(name) +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A real `FLYSIM01` envelope, small but structurally complete, so these tests decode and + /// re-encode what the running service writes rather than a stand-in. + fn envelope(generation: u64, ratchet: flybrain_gb::RatchetState) -> Vec { + use flybrain_core::decoder::DecoderState; + use flybrain_core::lif::LifState; + use flybrain_core::ordered::NumberMap; + use flybrain_core::plasticity::PlasticityState; + + let agent = flybrain_core::agent::AgentState { + version: 1, + remainder: 0.75, + warmed_up: true, + network: LifState { + membrane: vec![0.5, -0.25], + refractory: vec![0, 3], + last_spike_ms: vec![-1_000_000.0, 12.0], + visual_drive: vec![0.1], + rng: -12_345, + reward_remaining: 40.0, + ms: 9_000.0, + population_rate: 1.5, + rates: NumberMap::from_pairs([("forward", 2.0)]), + plasticity: PlasticityState { + version: "fly-kc-mbon-rstdp-v2".to_string(), + topology: 42, + enabled: true, + updates: 3.0, + signal: 0.25, + gains: vec![1.0, 0.9], + traces: vec![0.0, 0.1], + touched: vec![0.0, 8_000.0], + }, + }, + decoder: DecoderState { + version: 4, + calibrated: true, + baseline: NumberMap::from_pairs([("forward", 1.0)]), + held_until: NumberMap::new(), + next_allowed: NumberMap::new(), + next_decision: 100.0, + current: None, + fatigue: NumberMap::new(), + macro_next_decision: 0.0, + macro_current: None, + macro_fatigue: NumberMap::new(), + }, + }; + let runtime = store::RuntimeState { + generation, + wall_ms: 1_700_000_000_000, + rom_sha256: "ab".repeat(32), + emulator_frame: 12_345, + compatibility: "kernel/pokered-unique8-v6/fingerprint".to_string(), + speed: 1.0, + buttons: 0, + rank_since_ms: 4_242.0, + last_event_id: 77, + reward: serde_json::json!({ "version": 4, "total": 1.25 }), + ratchet, + emulator: vec![7; 64], + framebuffer: vec![9; 32], + ratchet_game: vec![1, 2, 3], + ratchet_frame: vec![4, 5, 6], + }; + store::encode(&agent, &runtime).unwrap() + } + + /// A store dir holding a milestone archive for each rung in `rungs`, their generations, a + /// manifest and an event log, all written through the store's own commit path. + fn state_dir(root: &Path, rungs: &[u32], ratchet: flybrain_gb::RatchetState) -> Store { + let store = Store::new(root, KEEP_GENERATIONS); + store.create().unwrap(); + for (index, rung) in rungs.iter().enumerate() { + let generation = index as u64 + 1; + store.commit(generation, &envelope(generation, ratchet), Some(*rung)).unwrap(); + } + std::fs::write(root.join("events.jsonl"), b"{}\n").unwrap(); + std::fs::write(root.join("events-20260921.jsonl"), b"{}\n").unwrap(); + store + } + + #[test] + fn a_missing_rung_is_refused_and_nothing_is_touched() { + let tmp = tempfile::tempdir().unwrap(); + let durable = tmp.path().join("state"); + let hot = tmp.path().join("hot"); + state_dir(&durable, &[3, 5], flybrain_gb::RatchetState::default()); + let before = std::fs::read_dir(&durable).unwrap().flatten().count(); + + let error = reset_to_milestone(&durable, &hot, 9, &tmp.path().join("archive")) + .unwrap_err() + .to_string(); + assert!(error.contains("no milestone archive for rung 9"), "{error}"); + assert_eq!(std::fs::read_dir(&durable).unwrap().flatten().count(), before); + assert!(!tmp.path().join("archive").exists(), "nothing was archived"); + } + + #[test] + fn the_rung_becomes_both_stores_latest_with_the_recovery_budget_back() { + let tmp = tempfile::tempdir().unwrap(); + let durable = tmp.path().join("state"); + let hot = tmp.path().join("hot"); + let spent = flybrain_gb::RatchetState { + best: 5, + attempts: 3, + recoveries: 11, + ..flybrain_gb::RatchetState::default() + }; + state_dir(&durable, &[3, 5, 9, 11], spent); + state_dir(&hot, &[11], spent); + let archive = tmp.path().join("archive-20260922"); + + let report = reset_to_milestone(&durable, &hot, 5, &archive).unwrap(); + assert!(report.iter().any(|line| line.contains("attempts 3 -> 0")), "{report:?}"); + + // Everything that was there is in the archive. + assert!(archive.join("durable/milestone-11.checkpoint").is_file()); + assert!(archive.join("durable/events.jsonl").is_file()); + assert!(archive.join("hot/manifest.json").is_file()); + + // The rungs above 5 are gone; the ones below it stay as deeper fallbacks. + assert!(!durable.join("milestone-9.checkpoint").exists()); + assert!(!durable.join("milestone-11.checkpoint").exists()); + assert!(durable.join("milestone-3.checkpoint").is_file()); + assert!(durable.join("milestone-5.checkpoint").is_file()); + assert!(!durable.join("events.jsonl").exists()); + assert!(!durable.join("events-20260921.jsonl").exists()); + assert!(!hot.join("milestone-11.checkpoint").exists()); + + // Both stores restore the rung, and the counters are back. + for store in [Store::new(&hot, KEEP_GENERATIONS), Store::new(&durable, KEEP_GENERATIONS)] { + let candidates = store.candidates("x"); + let first = store::load(&candidates[0].path).unwrap(); + assert_eq!(first.runtime.ratchet.best, 5); + assert_eq!((first.runtime.ratchet.attempts, first.runtime.ratchet.recoveries), (0, 0)); + } + // ... including through the promoted milestone archive itself. + let archived = store::load(&durable.join("milestone-5.checkpoint")).unwrap(); + assert_eq!((archived.runtime.ratchet.attempts, archived.runtime.ratchet.recoveries), (0, 0)); + + // The rungs below it are offered, and only as their own archive files. + let manifest = Store::new(&durable, KEEP_GENERATIONS).manifest().unwrap(); + assert_eq!(manifest.archives.get(&3), Some(&NO_GENERATION)); + assert_eq!(manifest.latest, manifest.archives.get(&5).copied()); + assert!( + Store::new(&durable, KEEP_GENERATIONS) + .candidates("durable") + .iter() + .any(|candidate| candidate.path.ends_with("milestone-3.checkpoint")) + ); + } + + #[test] + fn a_second_reset_refuses_to_write_over_an_existing_archive() { + let tmp = tempfile::tempdir().unwrap(); + let durable = tmp.path().join("state"); + let hot = tmp.path().join("hot"); + state_dir(&durable, &[4], flybrain_gb::RatchetState::default()); + let archive = tmp.path().join("archive"); + reset_to_milestone(&durable, &hot, 4, &archive).unwrap(); + let error = reset_to_milestone(&durable, &hot, 4, &archive).unwrap_err().to_string(); + assert!(error.contains("already exists"), "{error}"); + } + + #[test] + fn the_stamp_is_the_event_logs_calendar_plus_a_time_of_day() { + assert_eq!(utc_stamp(0), "19700101T000000Z"); + // 2026-09-22T16:15:00Z + assert_eq!(utc_stamp(1_790_093_700_000), "20260922T161500Z"); + } + + #[test] + fn the_default_archive_is_a_dated_sibling_of_the_store() { + assert_eq!( + default_archive_dir(Path::new("/srv/fly/state"), "20260922T161500Z"), + PathBuf::from("/srv/fly/state.reset-20260922T161500Z") + ); + } +} diff --git a/services/flysim/crates/flysim/src/simloop.rs b/services/flysim/crates/flysim/src/simloop.rs index ecf3440..8677f45 100644 --- a/services/flysim/crates/flysim/src/simloop.rs +++ b/services/flysim/crates/flysim/src/simloop.rs @@ -708,12 +708,36 @@ impl Sim { if runtime.rom_sha256 != self.rom_sha256 { bail!("checkpoint is for another cartridge ({})", runtime.rom_sha256); } - if runtime.compatibility != self.compatibility { - bail!( - "compatibility mismatch\n checkpoint: {}\n this build: {}", + // Byte-identical, or the one documented migration the operator asked for. + // + // `FLY_ACCEPT_ADAPTERS` is read here rather than carried in `Config` because it is a + // property of a *deploy*, not of a run: `infra/05-deploy.sh` writes it into + // `/etc/fly/fly.env` only for the deploy that needs it, and an operator who wants the + // migration off again deletes one line. An empty or unset variable is no migration at + // all, which is what every deploy before this one did. + let accepted = flybrain_gb::compatibility::accepted_adapters( + std::env::var(flybrain_gb::compatibility::ACCEPT_ADAPTERS_ENV).ok().as_deref(), + ); + match flybrain_gb::compatibility::decide( + &runtime.compatibility, + &self.compatibility, + self.adapter.migrates_from(), + &accepted, + ) { + flybrain_gb::compatibility::RestoreDecision::Exact => {} + flybrain_gb::compatibility::RestoreDecision::MigrateAdapter { from } => { + tracing::warn!( + from = %from, + to = %self.adapter.id(), + "restoring a checkpoint from an earlier adapter, by the migration \ + FLY_ACCEPT_ADAPTERS opted this deploy into" + ); + } + flybrain_gb::compatibility::RestoreDecision::Refuse(reason) => bail!( + "compatibility mismatch: {reason}\n checkpoint: {}\n this build: {}", runtime.compatibility, self.compatibility - ); + ), } if runtime.framebuffer.len() != FRAMEBUFFER_LEN { bail!("checkpoint framebuffer is {} bytes", runtime.framebuffer.len()); diff --git a/services/flysim/crates/flysim/src/snapshot.rs b/services/flysim/crates/flysim/src/snapshot.rs index f1c450d..dbcced1 100644 --- a/services/flysim/crates/flysim/src/snapshot.rs +++ b/services/flysim/crates/flysim/src/snapshot.rs @@ -241,8 +241,8 @@ pub struct FeedMacroOutcome { } /// Reward categories the feed reports counts for. The adapter's own interned kinds -/// (`milestone`, `exploration`, `map`, `species`, `trainer`, `battle`, `badge`, `boundary`) map -/// onto these. +/// (`milestone`, `exploration`, `map`, `species`, `trainer`, `battle`, `badge`, `boundary`, +/// `catch`) map onto these. #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] #[serde(rename_all = "lowercase")] pub enum RewardKind { @@ -282,6 +282,15 @@ impl RewardKind { // protocol is concerned: finding a door is finding somewhere new, and the design asks // for no new feed kind. "boundary" => Self::Explore, + // `catch` is a wild battle the fly won by keeping the Pokémon, so it publishes on + // the same counter a wild KO does. The feed's kinds are a closed set + // (`docs/feed-protocol.md`) and this rule asked for no new one. + // + // Deliberately *not* `pokedex`: on a catch of a species this run has never owned, + // the cartridge sets the Pokédex bit and the adapter's existing `species` rule pays + // for it on the same frame, so the `pokedex` counter already moves. Mapping `catch` + // there as well would count one event twice. + "catch" => Self::Wildwin, // The platformer. "band" => Self::Explore, "coin" => Self::Wildwin, @@ -739,6 +748,7 @@ mod tests { } assert_eq!(RewardKind::from_adapter("nonsense"), None); assert_eq!(RewardKind::from_adapter("boundary"), Some(RewardKind::Explore)); + assert_eq!(RewardKind::from_adapter("catch"), Some(RewardKind::Wildwin)); } #[test] diff --git a/services/flysim/crates/flysim/tests/compat_migration.rs b/services/flysim/crates/flysim/tests/compat_migration.rs new file mode 100644 index 0000000..13950b7 --- /dev/null +++ b/services/flysim/crates/flysim/tests/compat_migration.rs @@ -0,0 +1,224 @@ +//! A `v5` checkpoint restored under `v6`: accepted with the opt-in, refused without it. +//! +//! The unit tests in `flybrain-gb` cover the decision function and the adapter's own state +//! migration separately. This is the two of them against one artefact: a real `FLYSIM01` +//! envelope carrying a `pokered-unique8-v5` compatibility string and a `v5` reward ledger — +//! written, encoded, decoded, and then put through exactly what `Sim::try_restore` puts a +//! candidate through. +//! +//! No ROM and no dataset, deliberately. Building a `Sim` would need both, and neither is part of +//! the question: what decides a restore is the compatibility string and `import_state`. + +use flybrain_gb::GameAdapter; +use flybrain_gb::compatibility::{RestoreDecision, accepted_adapters, decide}; +use flybrain_gb::pokemon_red::PokemonRedReward; +use flysim::store::{self, RuntimeState}; + +/// The live string's shape, with the adapter left open. The dataset fingerprint is shortened — +/// nothing here parses it, and a seven-digest one would be 455 characters of noise. +fn compatibility(adapter: &str) -> String { + format!( + "lif-1ms-f64-v2/{adapter}/aa:bb:cc:dd:ee:ff:00/fly-kc-mbon-rstdp-v2/\ + binjgb:c60e138da5a795ebb55e56b11b7e90024e41112c/\ + pokered:0cd19d3b877b7dc66d12c7050bed9a7f38154d4b/statefmt:199616-x86_64-unknown-linux-gnu" + ) +} + +/// A `v5` reward ledger: `STATE_VERSION` 4, every field `v5` wrote, and **no** `catchCounts`. +/// +/// Written out by hand rather than exported from an adapter, because an exported one would be a +/// `v6` state with the counter deleted — this is the shape the release box's checkpoints really +/// carry, field for field. +fn v5_reward() -> serde_json::Value { + serde_json::json!({ + "version": 4, + "seen": ["adventure", "map:0", "early:outside", "dex:3", "boundary:0:edge:n:near"], + "tiles": ["0:5:6", "0:5:7"], + "tileCounts": { "0": 2 }, + "wildWins": { "0:112:4": 2 }, + "replayBlocked": [], + "counts": { + "milestone": 2, "exploration": 0, "map": 1, "species": 1, + "trainer": 0, "battle": 2, "badge": 0, "boundary": 1 + }, + "total": 2.05, + "recent": [{ "kind": "species", "label": "OWNED #4", "brainMs": 1234.5, "value": 0.5 }], + "last": { "species": { "kind": "species", "label": "OWNED #4", "brainMs": 1234.5, "value": 0.5 } }, + "initialized": true, + "sawBoot": true, + "location": "0:5:7", + "stable": 9, + "progress": 3, + "badges": 0, + "battle": null, + "mode": "OVERWORLD" + }) +} + +fn v5_checkpoint() -> Vec { + use flybrain_core::decoder::DecoderState; + use flybrain_core::lif::LifState; + use flybrain_core::ordered::NumberMap; + use flybrain_core::plasticity::PlasticityState; + + let agent = flybrain_core::agent::AgentState { + version: 1, + remainder: 0.25, + warmed_up: true, + network: LifState { + membrane: vec![0.1, -0.2], + refractory: vec![0, 1], + last_spike_ms: vec![-1_000_000.0, 5.0], + visual_drive: vec![0.3], + rng: 42, + reward_remaining: 0.0, + ms: 1_234.5, + population_rate: 1.0, + rates: NumberMap::from_pairs([("forward", 1.0)]), + plasticity: PlasticityState { + version: "fly-kc-mbon-rstdp-v2".to_string(), + topology: 7, + enabled: true, + updates: 1.0, + signal: 0.0, + gains: vec![1.0], + traces: vec![0.0], + touched: vec![0.0], + }, + }, + decoder: DecoderState { + version: 4, + calibrated: true, + baseline: NumberMap::from_pairs([("forward", 1.0)]), + held_until: NumberMap::new(), + next_allowed: NumberMap::new(), + next_decision: 0.0, + current: None, + fatigue: NumberMap::new(), + macro_next_decision: 0.0, + macro_current: None, + macro_fatigue: NumberMap::new(), + }, + }; + let runtime = RuntimeState { + generation: 41, + wall_ms: 1_790_000_000_000, + rom_sha256: flybrain_gb::pokemon_red::SUPPORTED_ROM.to_string(), + emulator_frame: 1_000_000, + compatibility: compatibility("pokered-unique8-v5"), + speed: 1.0, + buttons: 0, + rank_since_ms: 1_000.0, + last_event_id: 4_242, + reward: v5_reward(), + ratchet: flybrain_gb::RatchetState { best: 3, attempts: 1, recoveries: 4, ..Default::default() }, + emulator: vec![3; 64], + framebuffer: vec![0; 32], + ratchet_game: vec![1], + ratchet_frame: vec![2], + }; + store::encode(&agent, &runtime).expect("the fixture encodes") +} + +#[test] +fn a_v5_checkpoint_is_refused_under_v6_without_the_opt_in() { + let checkpoint = store::decode(&v5_checkpoint()).expect("the fixture decodes"); + let adapter = PokemonRedReward::new(); + let current = compatibility(adapter.id()); + assert_ne!(checkpoint.runtime.compatibility, current, "v6 is not v5"); + + for opt_in in [None, Some(""), Some("pokered-unique8-v4"), Some("some-other-adapter")] { + assert!( + matches!( + decide( + &checkpoint.runtime.compatibility, + ¤t, + adapter.migrates_from(), + &accepted_adapters(opt_in), + ), + RestoreDecision::Refuse(_) + ), + "FLY_ACCEPT_ADAPTERS={opt_in:?} must not migrate anything" + ); + } +} + +#[test] +fn a_v5_checkpoint_restores_under_v6_with_the_opt_in_and_the_counter_starts_at_zero() { + let checkpoint = store::decode(&v5_checkpoint()).expect("the fixture decodes"); + let mut adapter = PokemonRedReward::new(); + let current = compatibility(adapter.id()); + + assert_eq!( + decide( + &checkpoint.runtime.compatibility, + ¤t, + adapter.migrates_from(), + &accepted_adapters(Some("pokered-unique8-v5")), + ), + RestoreDecision::MigrateAdapter { from: "pokered-unique8-v5".to_string() } + ); + + // The migration itself: `import_state`, exactly as `Sim::try_restore` calls it. + adapter.import_state(&checkpoint.runtime.reward).expect("a v5 ledger is a valid v6 ledger"); + + let after = adapter.export_state(); + assert_eq!(after["catchCounts"], serde_json::json!({}), "the new counter starts at 0"); + assert_eq!(after["counts"]["catch"], serde_json::json!(0)); + + // And nothing else moved: every field the v5 state carried round-trips to the same value, + // and the only key v6 adds is the counter. + // + // `counts` is the one field that is not byte-identical, and it is not a change of meaning: + // it serializes every kind in the catalog, so a v6 state lists `catch` where a v5 state had + // nothing to list. Every kind the v5 state did carry keeps its number. + let before = v5_reward(); + for (key, value) in before.as_object().unwrap() { + if key == "counts" { + for (kind, count) in value.as_object().unwrap() { + assert_eq!(&after["counts"][kind], count, "counts.{kind}"); + } + let added: Vec<&String> = after["counts"] + .as_object() + .unwrap() + .keys() + .filter(|kind| !value.as_object().unwrap().contains_key(*kind)) + .collect(); + assert_eq!(added, vec!["catch"], "v6 counts one more kind and no others"); + continue; + } + assert_eq!(&after[key], value, "{key} must survive the migration byte for byte"); + } + let added: Vec<&String> = after + .as_object() + .unwrap() + .keys() + .filter(|key| !before.as_object().unwrap().contains_key(*key)) + .collect(); + assert_eq!(added, vec!["catchCounts"], "v6 adds one field and no others"); + + // The rest of what a restore reads is untouched by the migration. + assert_eq!(adapter.progress().rank, 3); + assert_eq!(checkpoint.runtime.last_event_id, 4_242); + assert_eq!(checkpoint.runtime.ratchet.best, 3); +} + +#[test] +fn nothing_but_the_adapter_segment_may_differ_for_the_migration_to_apply() { + let adapter = PokemonRedReward::new(); + let accepted = accepted_adapters(Some("pokered-unique8-v5")); + let current = compatibility(adapter.id()); + + // A v5 string whose state format also moved: a different build, not a rule change. + let other_abi = compatibility("pokered-unique8-v5").replace("199616", "199617"); + assert!(matches!( + decide(&other_abi, ¤t, adapter.migrates_from(), &accepted), + RestoreDecision::Refuse(_) + )); + + // And an identical string needs no opt-in at all. + assert_eq!( + decide(¤t, ¤t, adapter.migrates_from(), &[]), + RestoreDecision::Exact + ); +} diff --git a/services/flysim/crates/flysim/tests/rom_catch.rs b/services/flysim/crates/flysim/tests/rom_catch.rs new file mode 100644 index 0000000..014048b --- /dev/null +++ b/services/flysim/crates/flysim/tests/rom_catch.rs @@ -0,0 +1,257 @@ +//! The catch reward against the real cartridge. +//! +//! Gated on `FLY_ROM` *and* on a checkpoint, the way every ROM test in this workspace is, and +//! skips cleanly without either — the cartridge never enters this repository and a checkpoint is +//! not a fixture, it is the state the release box was really in: +//! +//! ```sh +//! FLY_ROM="$HOME/roms/pokemon-red.gb" \ +//! FLY_CATCH_CHECKPOINT=.local/checkpoints/ \ +//! cargo test --release -p flysim --test rom_catch -- --nocapture +//! ``` +//! +//! ## What only the cartridge can answer +//! +//! The synthetic trace in `pokemon_red/tests.rs` writes `wCapturedMonSpecies`, `wBattleResult` +//! and the Pokédex bit itself, from the disassembly. It cannot say that those are the bytes +//! *this* cartridge writes when a ball keeps a Pokémon, in that order, on frames an adapter +//! sampling once a frame actually sees. That is this test, and it is the "survey" half of +//! `docs/design/macros-wram.md`'s evidence for the row: a real battle, real button presses, and +//! the byte read out of the running game rather than written into a fake one. +//! +//! ## How the catch is produced +//! +//! No steering and no scripted button sequence: the shipping macro palette, the shipping macro +//! layer and the shipping decoder, with a stub readout that leans on one macro population at a +//! time — the same driver `tests/rom_macros_mode.rs` uses and for the same reason. The one thing +//! this harness does that the rotation does not is lean on `THROW BALL`'s channel while a wild +//! battle is up, because the question here is what the adapter reads from a catch, not whether a +//! game-blind readout finds its way to one. +//! +//! The checkpoint must hold at least one ball in the bag. The macro palette can buy one +//! (`BUY BALL`, `MB·PBALL`, inside a mart), but that is a walk across a city and back and it is a +//! different test's question; this one says out loud that it skipped. + +use flybrain_core::decoder::PopulationDecoder; +use flybrain_core::decoder::gameboy::gameboy_decoder_config_with_macros; +use flybrain_core::ordered::NumberMap; +use flybrain_gb::adapter::RewardEvent; +use flybrain_gb::pokemon_red::state; +use flybrain_gb::pokemon_red::symbols::ram; +use flybrain_gb::pokemon_red::{PokemonRedReward, catalog}; +use flybrain_gb::{ + AdapterLedger, DEFAULT_AUDIO_FRAMES, DEFAULT_AUDIO_FREQUENCY, Emulator, GameAdapter, +}; +use flysim::config::Config; +use flysim::macros::{MacroLayer, macro_layer}; +use flysim::snapshot::MacroMode; + +const MS_PER_FRAME: f64 = 1000.0 / 59.7275; +const SEED: u32 = 20_260_922; +/// The hot population's rate against every other one's, which is also the stub's calibration +/// rate — so a channel that is not the hot one scores exactly 1.0. +const HOT: f64 = 16.0; +const REST: f64 = 10.0; +/// `THROW BALL`'s channel (`pokemon_red::macros::palette`). +const BALL: &str = "MB·BALL"; +/// Frames the stub leans on one channel before the rotation moves on, the shape of the real +/// group's hysteresis-then-fatigue rotation. +const BURST_FRAMES: u32 = 24; + +fn rates(hot: Option<&str>) -> NumberMap { + let mut rates = NumberMap::new(); + for channel in flybrain_gb::macro_channels("pokemon-red") { + rates.set(channel, REST); + } + for bucket in 0..8 { + rates.set(&format!("command_{bucket}"), REST); + } + if let Some(channel) = hot { + rates.set(channel, HOT); + } + rates +} + +fn rom() -> Option> { + let path = std::env::var_os("FLY_ROM")?; + match std::fs::read(&path) { + Ok(bytes) => Some(bytes), + Err(error) => panic!("FLY_ROM is set to {path:?} but could not be read: {error}"), + } +} + +fn checkpoint() -> Option { + let path = std::env::var_os("FLY_CATCH_CHECKPOINT")?; + Some( + flysim::store::load(std::path::Path::new(&path)) + .expect("the checkpoint should be a FLYSIM01 envelope"), + ) +} + +struct Run { + gb: Emulator, + adapter: PokemonRedReward, + layer: MacroLayer, + decoder: PopulationDecoder, + channels: Vec<&'static str>, + ms: f64, + frame: u32, + /// Every payout the adapter has made since the run started. + payouts: Vec, +} + +impl Run { + fn resume(rom: &[u8], checkpoint: &flysim::store::Checkpoint) -> Self { + let mut gb = Emulator::new(rom, DEFAULT_AUDIO_FREQUENCY, DEFAULT_AUDIO_FRAMES) + .expect("binjgb should accept the cartridge"); + let mut adapter = PokemonRedReward::new(); + gb.import_state(&checkpoint.runtime.emulator).expect("the checkpoint's emulator state"); + // A checkpoint written by an earlier adapter rebaselines rather than failing, which is + // exactly the `v5` -> `v6` case this rule ships with. + adapter.import_state(&checkpoint.runtime.reward).expect("the checkpoint's reward ledger"); + let channels = flybrain_gb::macro_channels("pokemon-red"); + let preset = gameboy_decoder_config_with_macros(&channels); + let hold_ms = preset.macros.as_ref().expect("the preset has a macro group").hold_ms; + let mut decoder = PopulationDecoder::new(preset).expect("the preset is well formed"); + decoder.calibrate(&rates(None)); + let mut config = Config::default(); + config.loop_.game = "pokemon-red".to_string(); + config.macros.mode = MacroMode::Macros; + config.validate().expect("pokemon-red has a palette in macros mode"); + let mut layer = macro_layer(&config, hold_ms, SEED).expect("a layer in macros mode"); + let _ = layer.observe(&mut gb, &AdapterLedger(&adapter), 0.0); + Self { + gb, + adapter, + layer, + decoder, + channels, + ms: 0.0, + frame: 0, + payouts: Vec::new(), + } + } + + fn byte(&mut self, address: u16) -> u8 { + self.gb.read_wram(address) + } + + fn in_wild_battle(&mut self) -> bool { + self.byte(ram::wIsInBattle) == 1 + } + + /// Balls in the bag, of any kind (`constants/item_constants.asm`: MASTER_BALL 1, + /// ULTRA_BALL 2, GREAT_BALL 3, POKE_BALL 4 — the same four `THROW BALL` looks for). + fn balls(&mut self) -> usize { + state::bag(&mut self.gb) + .iter() + .filter(|item| (0x01..=0x04).contains(&item.id) && item.count > 0) + .map(|item| usize::from(item.count)) + .sum() + } + + fn step(&mut self) { + // Lean on `THROW BALL` while a wild battle is up; otherwise rotate, which is what gets + // the fly into the grass in the first place. + let hot = if self.in_wild_battle() { + Some(BALL) + } else { + let slot = (self.frame / BURST_FRAMES) as usize % self.channels.len(); + Some(self.channels[slot]) + }; + let bound = self.layer.bound_channels(); + let active = self.decoder.decode_bound(&rates(hot), self.ms, false, None, Some(&bound)); + let mask = { + let ledger = AdapterLedger(&self.adapter); + self.layer.decide(&active, 0, self.ms, &mut self.gb, &ledger).mask + }; + self.gb.set_buttons(mask as u8); + self.gb.run_frame().expect("a frame should complete"); + self.ms += MS_PER_FRAME; + self.frame += 1; + let ms = self.ms; + self.payouts.extend(self.adapter.sample(&mut self.gb, ms)); + let ledger = AdapterLedger(&self.adapter); + let _ = self.layer.observe(&mut self.gb, &ledger, ms); + } + + fn catches(&self) -> Vec<&RewardEvent> { + self.payouts.iter().filter(|event| event.kind == catalog::kind::CATCH).collect() + } +} + +#[test] +fn a_catch_on_the_cartridge_pays_the_catch_rule_once_with_the_species_in_its_label() { + let Some(rom) = rom() else { + eprintln!("skipped: FLY_ROM is not set"); + return; + }; + let Some(checkpoint) = checkpoint() else { + eprintln!("skipped: no FLY_CATCH_CHECKPOINT"); + return; + }; + let mut run = Run::resume(&rom, &checkpoint); + let balls = run.balls(); + if balls == 0 { + eprintln!( + "skipped: the checkpoint's bag holds no ball (map {:#04x}). `BUY BALL` can buy one \ + inside a mart; point FLY_CATCH_CHECKPOINT at a state that already has one.", + run.adapter.map_id().unwrap_or(u32::MAX) + ); + return; + } + eprintln!("bag holds {balls} balls; map {:#04x}", run.adapter.map_id().unwrap_or(u32::MAX)); + + // Twenty brain minutes is generous for a forest checkpoint: the live run threw 28 balls in + // its first Viridian Forest session (`pokemon_red::macros::palette`). + let budget = 20 * 60 * 60; + let mut battles = 0u32; + let mut was_in_battle = false; + for _ in 0..budget { + run.step(); + let now = run.in_wild_battle(); + if now && !was_in_battle { + battles += 1; + } + was_in_battle = now; + if !run.catches().is_empty() { + break; + } + } + + let catches = run.catches(); + assert!( + !catches.is_empty(), + "no catch in {:.1} brain minutes: {battles} wild battles, {} balls left, map {:#04x}, \ + macros {:?}", + run.ms / 60_000.0, + run.balls(), + run.adapter.map_id().unwrap_or(u32::MAX), + run.layer.counts() + ); + + let caught = catches[0]; + eprintln!( + "caught after {:.1} brain minutes and {battles} wild battles: {} for {}", + run.ms / 60_000.0, + caught.label, + caught.value + ); + assert!(caught.label.starts_with("CAUGHT #"), "{}", caught.label); + assert!( + (caught.value - 0.30).abs() < 1e-12 || caught.value == catalog::CATCH_REPEAT_VALUE, + "a catch pays one of the rule's two amounts, not {}", + caught.value + ); + // The cartridge's own flag is clear again by the time the payout lands, which is what makes + // the payout a battle-exit event rather than a per-frame one. + assert_eq!(run.byte(ram::wCapturedMonSpecies), 0); + assert_eq!( + run.adapter.progress().counts[catalog::kind::CATCH], + 1, + "one battle, one payout" + ); + // And a species the run is paid for catching is a species the Pokédex knows: the same event + // sets the bit the `species` rule reads, whether or not it was new to this run. + assert!(run.balls() < balls, "a ball was spent"); +} diff --git a/services/flysim/tools/gen_symbols.py b/services/flysim/tools/gen_symbols.py index 71ce56c..b499e43 100644 --- a/services/flysim/tools/gen_symbols.py +++ b/services/flysim/tools/gen_symbols.py @@ -200,6 +200,15 @@ EXTRA_RAM = ( 'wCurMapTileset', 'wTilesetBank', 'wTilesetBlocksPtr', + # The catch reward (`docs/rewards-learning.md`, `docs/design/macros-wram.md` section 2). + # ram/wram.asm's own comment is "0 if no mon was captured": ItemUseBall zeroes it before + # every throw and writes wEnemyMonSpecies into it only on the branch that keeps the + # Pokemon, and UseBagItem zeroes it again on the way out of the battle. It is the + # cartridge's own answer to "was this one caught", and the only signal that needs no + # second rule to tell a catch apart from a gift, a trade or an evolution. + # services/flysim/tools/resolve_wram.py is the second reading of it, from ram/wram.asm at + # this commit, bracketed by wFontLoaded and wForcePlayerToChooseMon. + 'wCapturedMonSpecies', ) diff --git a/services/flysim/tools/resolve_wram.py b/services/flysim/tools/resolve_wram.py index 4e67b06..c7973cb 100644 --- a/services/flysim/tools/resolve_wram.py +++ b/services/flysim/tools/resolve_wram.py @@ -49,6 +49,33 @@ WANTED = { # not bank 0, so this is the read the memory seam grew a bank for. 'wTilesetBank': 'the ROM bank the blockset lives in', 'wTilesetBlocksPtr': 'blocks to tiles, 16 bytes per block', + # The catch reward (`docs/rewards-learning.md`, `docs/design/macros-wram.md` + # section 10). ram/wram.asm's own comment is "0 if no mon was captured": + # ItemUseBall zeroes it before every throw and writes wEnemyMonSpecies into it + # only on the branch that keeps the caught Pokemon, and UseBagItem zeroes it + # again on the way out of the battle. It is the cartridge's own answer to "was + # this one caught", and the only signal that needs no second rule to tell a + # catch apart from a gift, a trade or an evolution. + 'wCapturedMonSpecies': 'the species a ball just caught, 0 for none', +} + + +#: Constants the decomp defines through its `const` enumeration rather than with a +#: plain `EQU`, so `constants()` cannot evaluate their expressions. They matter here +#: because `NUM_TMS + NUM_HMS` is the size of `wMonHLearnset`, and that one +#: declaration is what kills the cursor on its way through the battle engine's +#: scratch bytes -- the region `wCapturedMonSpecies` lives in. +#: +#: Each is *counted* from the decomp rather than written out by hand, which is the +#: same rule the rest of this tool follows. `DEF NUM_HMS EQU const_value - HM01` is +#: by construction the number of `add_hm` definitions after `HM01`, and +#: `item_constants.asm`'s own `ASSERT NUM_TMS == const_value - TM01` ties `NUM_TMS` +#: to the number of `add_tm` definitions -- so `NUM_TMS` is counted *and* compared +#: against the literal the same file declares, and a decomp that moved one without +#: the other stops the run instead of producing an address. +COUNTED = { + 'NUM_HMS': ('constants/item_constants.asm', r'^\s*add_hm\s+\w+'), + 'NUM_TMS': ('constants/item_constants.asm', r'^\s*add_tm\s+\w+'), } @@ -68,6 +95,10 @@ def constants(root: Path) -> dict[str, int]: BLOCK_WIDTH`). A name whose expression never becomes evaluable is simply left out, which kills the cursor at any declaration that uses it. """ + counted = { + name: len(re.findall(pattern, (root / path).read_text(), re.M)) + for name, (path, pattern) in COUNTED.items() + } pending: dict[str, str] = {} sources = sorted((root / 'constants').glob('*.asm')) + sorted( (root / 'constants').glob('*.inc') @@ -77,7 +108,7 @@ def constants(root: Path) -> dict[str, int]: r'^\s*(?:DEF|def)\s+(\w+)\s+(?:EQU|equ)\s+([^;\n]+)', path.read_text(), re.M ): pending.setdefault(name, value.strip()) - out: dict[str, int] = {} + out: dict[str, int] = dict(counted) while pending: progressed = False for name in list(pending): @@ -89,6 +120,12 @@ def constants(root: Path) -> dict[str, int]: progressed = True if not progressed: break + for name, value in counted.items(): + if out.get(name, value) != value: + raise SystemExit( + f'{name}: the decomp declares {out[name]} and defines {value} of them' + ) + out[name] = value return out @@ -367,11 +404,17 @@ def main() -> None: raise SystemExit('the walk disagrees with symbols.rs; nothing emitted') print(f'{checked} of {len(table)} pinned addresses re-derived from wram.asm, no disagreement') - missing = [name for name in WANTED if name not in resolved] + # A name this tool has already emitted is pinned, so the walk meets it as an + # anchor rather than resolving it: it was re-derived all the same, and the + # comparison above is what says so. + missing = [name for name in WANTED if name not in resolved and name not in table] if missing: raise SystemExit(f'unanchored, so not resolved: {", ".join(missing)}') for name in WANTED: - print(f'{name} = ${resolved[name]:04x} ({WANTED[name]})') + if name in resolved: + print(f'{name} = ${resolved[name]:04x} ({WANTED[name]})') + else: + print(f'{name} = ${table[name]:04x} (already pinned; {WANTED[name]})') if not args.emit: return From 6655a1b1c6a939ed8a51513db074eb614d288915 Mon Sep 17 00:00:00 2001 From: dev Date: Tue, 22 Sep 2026 17:43:43 +0000 Subject: [PATCH 4/8] session: coherent all-participant checkpoint and recovery STATE-01 over the FLYSESS1 envelope CONTRACT-01 specified. fly-session gains a `state` module: the durable store with its generations, its rotation and the commit order of checkpoint-envelope-v1 section 5, where the store manifest rename is the durable commit point; a compatibility block whose comparison names the identity that differs rather than one opaque digest; and a bounded writer that owns its payload handles until the bytes are committed or the job fails. The writer's queue slot is taken before the first State.Capture, so a saturated writer refuses a capture rather than queueing it without bound, and the refusal is a BUSY the stepping session survives. Capture and durability are two events: a capture completes when an immutable capture exists, and only the store manifest rename moves the durable mark. A lost save reply is an outcome, and the resolution asks the store about the same checkpoint instead of saving again. Both worker roles implement State.Capture, State.StageRestore and State.ActivateRestore, with once-only restore tokens bound to checkpoint, scope, payload and incarnation. A restore selects a complete compatible generation, imports every payload as a fresh artifact, stages the group, validates the coordinator's own ledgers, and only then activates; a failure anywhere leaves the fence closed and records every participant that staged as one that must be replaced. The fence lifts at Failed -> Restoring(k) -> Paused(k) and nowhere else. The task and the action executor gain the capture/validate_restore/install_restore interfaces workers-v1 section 4 lists, and the ledger can re-derive the event identities it issued under another epoch, which is what lets a resumed run's behaviour trace be compared with an uninterrupted one. media: check_required_audio now takes the observation's provenance instead of exempting boundary 0. A chunk is the audio of an interval, and the observation ActivateRestore installs covers none. checkpoint-envelope-v1 section 3 gains a dated amendment adding `environment` to the manifest, the holder of the world's own payload, which the table named for every other participant; `helperState`, which that table already listed, joins the required-field set in Rust and TypeScript. The fixture was regenerated by the existing example; the schema set and contractDigest are unchanged. state-media-v1 section 5 gains a dated amendment for three readings this slice enforces: the State RPCs' compatibilityDigest is the participant's, not the manifest's composition-level block; a restored observation carries no audio chunk; and a participant that staged into an abandoned install must be replaced. --- .../checkpoint-envelope-v1.md | 13 + .../session-framework/state-media-v1.md | 22 + packages/session-types/src/checkpoint.ts | 9 +- .../examples/update_fixtures.rs | 1 + .../fixtures/checkpoint-envelope.json | 32 +- .../fly-session-types/src/checkpoint.rs | 7 + services/flysim/crates/fly-session/README.md | 50 +- .../flysim/crates/fly-session/src/agent.rs | 573 ++++++- services/flysim/crates/fly-session/src/cli.rs | 17 + .../flysim/crates/fly-session/src/clock.rs | 24 + .../crates/fly-session/src/coordinator.rs | 1489 +++++++++++++++- .../crates/fly-session/src/environment.rs | 517 +++++- .../flysim/crates/fly-session/src/harness.rs | 179 +- .../flysim/crates/fly-session/src/launcher.rs | 22 + services/flysim/crates/fly-session/src/lib.rs | 1 + .../flysim/crates/fly-session/src/media.rs | 144 +- .../flysim/crates/fly-session/src/state.rs | 1494 +++++++++++++++++ .../flysim/crates/fly-session/src/task.rs | 249 +++ .../flysim/crates/fly-session/src/types.rs | 80 +- .../flysim/crates/fly-session/src/worker.rs | 12 +- .../crates/fly-session/tests/processes.rs | 3 + .../flysim/crates/fly-session/tests/state.rs | 947 +++++++++++ 22 files changed, 5837 insertions(+), 48 deletions(-) create mode 100644 services/flysim/crates/fly-session/src/state.rs create mode 100644 services/flysim/crates/fly-session/tests/state.rs diff --git a/docs/design/session-framework/checkpoint-envelope-v1.md b/docs/design/session-framework/checkpoint-envelope-v1.md index c83bb70..7cb833f 100644 --- a/docs/design/session-framework/checkpoint-envelope-v1.md +++ b/docs/design/session-framework/checkpoint-envelope-v1.md @@ -105,6 +105,19 @@ names; a manifest missing any of them is not a complete checkpoint. | `helperState` | External-helper state required for exact resume, as payload names | | `payloads` | `[{name, byteLength, digest}]`, mirroring the payload table | +**Amendment, 2026-09-22 (STATE-01).** The table above names a holder for every payload except +the environment's own, although section 6's fixture has one (`world`) and a group install has +to map it by name like any other participant's. The manifest therefore also records: + +| Field | Contents | +| --- | --- | +| `environment` | `{workerId, payload}`: which worker the world belonged to and the payload name holding its state | + +The reference implementations' required-field set was also missing `helperState`, which this +section has listed from the start. Both are now in `REQUIRED_MANIFEST_FIELDS` in Rust and in +TypeScript, and the fixture was regenerated by the existing example. The schema set is +untouched, so `contractDigest` is unchanged. + `payloads` is redundant with the table on purpose: the table is what a reader needs to map bytes, and the manifest is what a store lists, compares and reports without opening the payload area. A reader checks that the two agree. diff --git a/docs/design/session-framework/state-media-v1.md b/docs/design/session-framework/state-media-v1.md index 2669786..0061761 100644 --- a/docs/design/session-framework/state-media-v1.md +++ b/docs/design/session-framework/state-media-v1.md @@ -190,6 +190,28 @@ restored time. It cannot advance gameplay to manufacture it. Capture/reconstruct covers render/inspection state and any pending sensor pipeline. Agent state agrees with it; do not replay reward or recalibrate merely to fill missing cached data. +**Amendment, 2026-09-22 (STATE-01).** Three readings of this section, made explicit because +they are now enforced: + +- `compatibilityDigest` on `CaptureResult` and `StageRestoreParams` is the **participant's** + capture compatibility digest of [worker interfaces](workers-v1.md) section 2 -- profile, + resolved seed, numerical model version and effective instance configuration for an agent; + backend, content, patch, controller and parser identity for an environment. It is not the + manifest's `compatibility` block of section 4, which is the composition's and which the + coordinator compares before anything is asked to stage. Both exist because they answer + different questions, and a restore that passed the second could still be handing an agent + another agent's brain. +- The observation `ActivateRestore` returns ran no transition, so it carries **no audio + chunk**, and one in it is refused. Section 2's chunk is the audio of an interval and this + observation covers none; MEDIA-01 implemented that rule as "boundary 0 carries no chunk", + which is true of the only such observation that slice could produce and false of this one. + The rule is about provenance, not about the boundary number. +- A participant that staged into a group install the coordinator then abandoned must be + **replaced** before another restore, exactly as one that activated must. It is holding a + validated replacement state that nothing installed, and [session RPC](ipc-v1.md) section 6 + already refuses to silently reattach such a participant to an active epoch. Without this the + group's second attempt meets its own leftovers and calls them a conflict. + If emulator validation requires mutation, stage a stopped replacement emulator. If that cannot provide externally atomic resume, advertise episode-restart, not exact-checkpoint. After all activation acknowledgments, install the coordinator's staged task/executor/admission state diff --git a/packages/session-types/src/checkpoint.ts b/packages/session-types/src/checkpoint.ts index 63aac19..2df48f7 100644 --- a/packages/session-types/src/checkpoint.ts +++ b/packages/session-types/src/checkpoint.ts @@ -223,7 +223,12 @@ export function decode(input: Uint8Array): Envelope { }; } -/** The manifest fields state-media-v1 section 4 requires. */ +/** + * The manifest fields state-media-v1 section 4 requires. + * + * `helperState` and `environment` join the list under the 2026-09-22 amendment to + * checkpoint-envelope-v1 section 3. + */ export const REQUIRED_MANIFEST_FIELDS = [ 'envelopeVersion', 'checkpointId', @@ -236,6 +241,8 @@ export const REQUIRED_MANIFEST_FIELDS = [ 'compatibility', 'agents', 'coordinator', + 'environment', + 'helperState', 'payloads', ] as const; diff --git a/services/flysim/crates/fly-session-types/examples/update_fixtures.rs b/services/flysim/crates/fly-session-types/examples/update_fixtures.rs index 238287e..d68b865 100644 --- a/services/flysim/crates/fly-session-types/examples/update_fixtures.rs +++ b/services/flysim/crates/fly-session-types/examples/update_fixtures.rs @@ -199,6 +199,7 @@ fn checkpoint_envelope() -> String { "admissionState": null, "eventWatermarks": {"lastEventId": "evt-1", "lastOrdinal": "7"}, }, + "environment": {"workerId": "arena", "payload": "world"}, "helperState": [], "payloads": payload_table(), }); diff --git a/services/flysim/crates/fly-session-types/fixtures/checkpoint-envelope.json b/services/flysim/crates/fly-session-types/fixtures/checkpoint-envelope.json index 6636bae..70453c3 100644 --- a/services/flysim/crates/fly-session-types/fixtures/checkpoint-envelope.json +++ b/services/flysim/crates/fly-session-types/fixtures/checkpoint-envelope.json @@ -63,6 +63,10 @@ "lastOrdinal": "7" } }, + "environment": { + "workerId": "arena", + "payload": "world" + }, "helperState": [], "payloads": [ { @@ -115,49 +119,49 @@ } ], "envelope": { - "base64": "RkxZU0VTUzEBAAAAIAAAAAAIAAAFAAAAIAgAAAAAAAB7ImFnZW50cyI6W3siYWdlbnRJZCI6ImZseS1hIiwiYnJhaW5UaWNrcyI6IjI1MzQiLCJkYXRhc2V0RGlnZXN0IjoiNmMwYWYxZjA3ODRlZjYzYTM5M2VlNzdkNjE0ZTgyNDZjNjI1MDUxMzYwZjNmMWE0ODgzODM3NGM1ZDM1NWI1MiIsIm1vZGVsVmVyc2lvbiI6ImxpZi0xbXMtZjY0LXYyIiwicGF5bG9hZCI6ImFnZW50LWZseS1hIiwicGxhc3RpY2l0eVZlcnNpb24iOiJmbHkta2MtbWJvbi1yc3RkcC12MiIsInByb2ZpbGVEaWdlc3QiOiIxOTAwZWFiNmMwMjg0ODNkNzEyNjU5OWVlNmY1MGRlMGQyNzkwN2I1YzY1ZmE5MDUyNDU4MGI0YjBmOTg1MmIwIiwicmVtYWluZGVyIjp7ImRlbm9taW5hdG9yIjoiMyIsIm51bWVyYXRvciI6IjEwMDAwMDAifSwic2VlZCI6LTE4NDk0NjA2M31dLCJjaGVja3BvaW50SWQiOiJja3B0LTEiLCJjb21wYXRpYmlsaXR5Ijp7ImJhY2tlbmREaWdlc3QiOiIxMGUwOGE0MTllODUwZWJhMWViYmExOGZkZDI4ZWI3ZWMxYjdlOGJhYTliY2MzYjk3M2UyYjg4OTFlYzcyNmJlIiwiY29udGVudERpZ2VzdCI6ImVkNzAwMmI0MzllOWFjODQ1ZjIyMzU3ZDgyMmJhYzE0NDQ3MzBmYmRiNjAxNmQzZWM5NDMyMjk3YjllYzlmNzMiLCJjb250cm9sbGVyRGlnZXN0IjoiYzE0NzIxMzViMTRjNzdjOGJlZjk4ZTczZjcwMjA4MzI1ZmEwZGNmMWU2YmQ2NjhhZTliMzFhOWNlYTI5NWZlNyIsInBhcnNlckRpZ2VzdCI6ImIxN2Q0NTEyMTE1MDkyOGYyMTQ2YWY0OWUxOTVlZmYxZWVmNWQ2NzMyNWJlMjczYTczM2ZiNzRhY2FkYWEzNDIiLCJwYXRjaERpZ2VzdCI6ImE0ODk1ZWI0NGFmYzMzNmZlY2JiYTZlNTIwY2Q2N2UxNzhkYWNlMDI3NjY1NWQxMDJmY2VmZmE4ZTVmNzA1NzAiLCJzdGF0ZUZvcm1hdElkIjoiZmx5c2Vzcy0xIn0sImNvbXBvc2l0aW9uRGlnZXN0IjoiNzMwZDcyNWM4YTU5ZDNhNzMwM2RlZjJiZWQwNDFhNTc3ZWRiNDI1NWFhYmQ0ODg5Y2UxMjkxODMxMWQ5NTJmMCIsImNvb3JkaW5hdG9yIjp7ImFkbWlzc2lvblN0YXRlIjpudWxsLCJldmVudFdhdGVybWFya3MiOnsibGFzdEV2ZW50SWQiOiJldnQtMSIsImxhc3RPcmRpbmFsIjoiNyJ9LCJleGVjdXRvclN0YXRlIjpbeyJhZ2VudElkIjoiZmx5LWEiLCJwYXlsb2FkIjoiZXhlY3V0b3ItZmx5LWEifV0sInByaW9ySW5zcGVjdGlvbiI6InByaW9yLWluc3BlY3Rpb24iLCJ0YXNrTGVkZ2VyIjoidGFzay1sZWRnZXIifSwiZW52ZWxvcGVWZXJzaW9uIjoxLCJlcGlzb2RlSWQiOiJlcGlzb2RlLTEiLCJoZWxwZXJTdGF0ZSI6W10sInBheWxvYWRzIjpbeyJieXRlTGVuZ3RoIjoiMTciLCJkaWdlc3QiOiIxMzIxZGZmYjBjZGM2ZjkwOTJjYmY3ZmEyYTVmYzY4YmJlZDEyYzk5M2Q1YWQzOTgyNjQwMTI4MTBjZTliZjkzIiwibmFtZSI6ImFnZW50LWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTQiLCJkaWdlc3QiOiIzYWVlNjBkZjdlMjllZmViYTdmNWY5OWZjNTg2NzY0N2IzNmFlYmZmMWQ1ZDNjODM4ZGJmZjMyMzEyMmU2NDYyIiwibmFtZSI6ImV4ZWN1dG9yLWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTEiLCJkaWdlc3QiOiI0MGIwMGVkMmJiYmE5MDFkNjgyMDVmZjcxYjA0YTQ0YjllZTUzYzUxY2IzMTA5YWEyY2VhYTQ0ZjFjNDU3MjdlIiwibmFtZSI6InRhc2stbGVkZ2VyIn0seyJieXRlTGVuZ3RoIjoiMTAiLCJkaWdlc3QiOiIyYzEzYjdiNGQ5YTk5MTY4MDFhYjkxOTFjMzE0ZjMxYjA0NWU5YjljNWI2NjlhNmMwNDc0ZjAyMTdlZjc1YmY1IiwibmFtZSI6InByaW9yLWluc3BlY3Rpb24ifSx7ImJ5dGVMZW5ndGgiOiI2NCIsImRpZ2VzdCI6ImY1YTVmZDQyZDE2YTIwMzAyNzk4ZWY2ZWQzMDk5NzliNDMwMDNkMjMyMGQ5ZjBlOGVhOTgzMWE5Mjc1OWZiNGIiLCJuYW1lIjoid29ybGQifV0sInBvcnRNYXAiOlt7ImFnZW50SWQiOiJmbHktYSIsInBvcnRJZCI6InBvcnQtMSJ9XSwic2NoZWR1bGVySWQiOiJsb2Nrc3RlcC12MSIsInNvdXJjZVNjb3BlIjp7ImVwb2NoIjoiZXBvY2gtMSIsInNlc3Npb25JZCI6ImRlbW8iLCJzdGVwIjoiNDIifSwid29ybGRUaW1lIjp7ImRlbm9taW5hdG9yIjoiMSIsIm51bWVyYXRvciI6IjcwMDAwMDAwMCJ9fWFnZW50LWZseS1hAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABQCgAAAAAAABEAAAAAAAAAEyHf+wzcb5CSy/f6Kl/Gi77RLJk9WtOYJkASgQzpv5NleGVjdXRvci1mbHktYQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaAoAAAAAAAAOAAAAAAAAADruYN9+Ke/rp/X5n8WGdkezauv/HV08g42/8yMSLmRidGFzay1sZWRnZXIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHgKAAAAAAAACwAAAAAAAABAsA7Su7qQHWggX/cbBKRLnuU8UcsxCaos6qRPHEVyfnByaW9yLWluc3BlY3Rpb24AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACICgAAAAAAAAoAAAAAAAAALBO3tNmpkWgBq5GRwxTzGwRem5xbZppsBHTwIX73W/V3b3JsZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAmAoAAAAAAABAAAAAAAAAAPWl/ULRaiAwJ5jvbtMJl5tDAD0jINnw6OqYMaknWftLYWdlbnQgc3RhdGUgYnl0ZXMAAAAAAAAAZXhlY3V0b3Igc3RhdGUAAHsicmFuayI6MTB9AAAAAAB7Im1hcCI6NDB9AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgLAAAAAAAAq++fEx+FvDZho/eB4imbENN4HZrGNC2OCAsI7/gp9r5GTFlTRVNTRg==", - "byteLength": 2824, + "base64": "RkxZU0VTUzEBAAAAIAAAADUIAAAFAAAAWAgAAAAAAAB7ImFnZW50cyI6W3siYWdlbnRJZCI6ImZseS1hIiwiYnJhaW5UaWNrcyI6IjI1MzQiLCJkYXRhc2V0RGlnZXN0IjoiNmMwYWYxZjA3ODRlZjYzYTM5M2VlNzdkNjE0ZTgyNDZjNjI1MDUxMzYwZjNmMWE0ODgzODM3NGM1ZDM1NWI1MiIsIm1vZGVsVmVyc2lvbiI6ImxpZi0xbXMtZjY0LXYyIiwicGF5bG9hZCI6ImFnZW50LWZseS1hIiwicGxhc3RpY2l0eVZlcnNpb24iOiJmbHkta2MtbWJvbi1yc3RkcC12MiIsInByb2ZpbGVEaWdlc3QiOiIxOTAwZWFiNmMwMjg0ODNkNzEyNjU5OWVlNmY1MGRlMGQyNzkwN2I1YzY1ZmE5MDUyNDU4MGI0YjBmOTg1MmIwIiwicmVtYWluZGVyIjp7ImRlbm9taW5hdG9yIjoiMyIsIm51bWVyYXRvciI6IjEwMDAwMDAifSwic2VlZCI6LTE4NDk0NjA2M31dLCJjaGVja3BvaW50SWQiOiJja3B0LTEiLCJjb21wYXRpYmlsaXR5Ijp7ImJhY2tlbmREaWdlc3QiOiIxMGUwOGE0MTllODUwZWJhMWViYmExOGZkZDI4ZWI3ZWMxYjdlOGJhYTliY2MzYjk3M2UyYjg4OTFlYzcyNmJlIiwiY29udGVudERpZ2VzdCI6ImVkNzAwMmI0MzllOWFjODQ1ZjIyMzU3ZDgyMmJhYzE0NDQ3MzBmYmRiNjAxNmQzZWM5NDMyMjk3YjllYzlmNzMiLCJjb250cm9sbGVyRGlnZXN0IjoiYzE0NzIxMzViMTRjNzdjOGJlZjk4ZTczZjcwMjA4MzI1ZmEwZGNmMWU2YmQ2NjhhZTliMzFhOWNlYTI5NWZlNyIsInBhcnNlckRpZ2VzdCI6ImIxN2Q0NTEyMTE1MDkyOGYyMTQ2YWY0OWUxOTVlZmYxZWVmNWQ2NzMyNWJlMjczYTczM2ZiNzRhY2FkYWEzNDIiLCJwYXRjaERpZ2VzdCI6ImE0ODk1ZWI0NGFmYzMzNmZlY2JiYTZlNTIwY2Q2N2UxNzhkYWNlMDI3NjY1NWQxMDJmY2VmZmE4ZTVmNzA1NzAiLCJzdGF0ZUZvcm1hdElkIjoiZmx5c2Vzcy0xIn0sImNvbXBvc2l0aW9uRGlnZXN0IjoiNzMwZDcyNWM4YTU5ZDNhNzMwM2RlZjJiZWQwNDFhNTc3ZWRiNDI1NWFhYmQ0ODg5Y2UxMjkxODMxMWQ5NTJmMCIsImNvb3JkaW5hdG9yIjp7ImFkbWlzc2lvblN0YXRlIjpudWxsLCJldmVudFdhdGVybWFya3MiOnsibGFzdEV2ZW50SWQiOiJldnQtMSIsImxhc3RPcmRpbmFsIjoiNyJ9LCJleGVjdXRvclN0YXRlIjpbeyJhZ2VudElkIjoiZmx5LWEiLCJwYXlsb2FkIjoiZXhlY3V0b3ItZmx5LWEifV0sInByaW9ySW5zcGVjdGlvbiI6InByaW9yLWluc3BlY3Rpb24iLCJ0YXNrTGVkZ2VyIjoidGFzay1sZWRnZXIifSwiZW52ZWxvcGVWZXJzaW9uIjoxLCJlbnZpcm9ubWVudCI6eyJwYXlsb2FkIjoid29ybGQiLCJ3b3JrZXJJZCI6ImFyZW5hIn0sImVwaXNvZGVJZCI6ImVwaXNvZGUtMSIsImhlbHBlclN0YXRlIjpbXSwicGF5bG9hZHMiOlt7ImJ5dGVMZW5ndGgiOiIxNyIsImRpZ2VzdCI6IjEzMjFkZmZiMGNkYzZmOTA5MmNiZjdmYTJhNWZjNjhiYmVkMTJjOTkzZDVhZDM5ODI2NDAxMjgxMGNlOWJmOTMiLCJuYW1lIjoiYWdlbnQtZmx5LWEifSx7ImJ5dGVMZW5ndGgiOiIxNCIsImRpZ2VzdCI6IjNhZWU2MGRmN2UyOWVmZWJhN2Y1Zjk5ZmM1ODY3NjQ3YjM2YWViZmYxZDVkM2M4MzhkYmZmMzIzMTIyZTY0NjIiLCJuYW1lIjoiZXhlY3V0b3ItZmx5LWEifSx7ImJ5dGVMZW5ndGgiOiIxMSIsImRpZ2VzdCI6IjQwYjAwZWQyYmJiYTkwMWQ2ODIwNWZmNzFiMDRhNDRiOWVlNTNjNTFjYjMxMDlhYTJjZWFhNDRmMWM0NTcyN2UiLCJuYW1lIjoidGFzay1sZWRnZXIifSx7ImJ5dGVMZW5ndGgiOiIxMCIsImRpZ2VzdCI6IjJjMTNiN2I0ZDlhOTkxNjgwMWFiOTE5MWMzMTRmMzFiMDQ1ZTliOWM1YjY2OWE2YzA0NzRmMDIxN2VmNzViZjUiLCJuYW1lIjoicHJpb3ItaW5zcGVjdGlvbiJ9LHsiYnl0ZUxlbmd0aCI6IjY0IiwiZGlnZXN0IjoiZjVhNWZkNDJkMTZhMjAzMDI3OThlZjZlZDMwOTk3OWI0MzAwM2QyMzIwZDlmMGU4ZWE5ODMxYTkyNzU5ZmI0YiIsIm5hbWUiOiJ3b3JsZCJ9XSwicG9ydE1hcCI6W3siYWdlbnRJZCI6ImZseS1hIiwicG9ydElkIjoicG9ydC0xIn1dLCJzY2hlZHVsZXJJZCI6ImxvY2tzdGVwLXYxIiwic291cmNlU2NvcGUiOnsiZXBvY2giOiJlcG9jaC0xIiwic2Vzc2lvbklkIjoiZGVtbyIsInN0ZXAiOiI0MiJ9LCJ3b3JsZFRpbWUiOnsiZGVub21pbmF0b3IiOiIxIiwibnVtZXJhdG9yIjoiNzAwMDAwMDAwIn19AAAAYWdlbnQtZmx5LWEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIgKAAAAAAAAEQAAAAAAAAATId/7DNxvkJLL9/oqX8aLvtEsmT1a05gmQBKBDOm/k2V4ZWN1dG9yLWZseS1hAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACgCgAAAAAAAA4AAAAAAAAAOu5g334p7+un9fmfxYZ2R7Nq6/8dXTyDjb/zIxIuZGJ0YXNrLWxlZGdlcgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAsAoAAAAAAAALAAAAAAAAAECwDtK7upAdaCBf9xsEpEue5TxRyzEJqizqpE8cRXJ+cHJpb3ItaW5zcGVjdGlvbgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAKAAAAAAAACgAAAAAAAAAsE7e02amRaAGrkZHDFPMbBF6bnFtmmmwEdPAhfvdb9XdvcmxkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADQCgAAAAAAAEAAAAAAAAAA9aX9QtFqIDAnmO9u0wmXm0MAPSMg2fDo6pgxqSdZ+0thZ2VudCBzdGF0ZSBieXRlcwAAAAAAAABleGVjdXRvciBzdGF0ZQAAeyJyYW5rIjoxMH0AAAAAAHsibWFwIjo0MH0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAsAAAAAAAAhBlh9AWLTtVKckmeIzNn4DO5Yhn2C1nUA1T1RfxMXlkZMWVNFU1NG", + "byteLength": 2880, "layout": { "headerBytes": 32, "manifestOffset": "32", - "manifestBytes": 2048, - "tableOffset": "2080", + "manifestBytes": 2101, + "tableOffset": "2136", "tableEntryBytes": 112, "entries": [ { "name": "agent-fly-a", - "offset": "2640", + "offset": "2696", "byteLength": "17", "digest": "1321dffb0cdc6f9092cbf7fa2a5fc68bbed12c993d5ad398264012810ce9bf93" }, { "name": "executor-fly-a", - "offset": "2664", + "offset": "2720", "byteLength": "14", "digest": "3aee60df7e29efeba7f5f99fc5867647b36aebff1d5d3c838dbff323122e6462" }, { "name": "task-ledger", - "offset": "2680", + "offset": "2736", "byteLength": "11", "digest": "40b00ed2bbba901d68205ff71b04a44b9ee53c51cb3109aa2ceaa44f1c45727e" }, { "name": "prior-inspection", - "offset": "2696", + "offset": "2752", "byteLength": "10", "digest": "2c13b7b4d9a9916801ab9191c314f31b045e9b9c5b669a6c0474f0217ef75bf5" }, { "name": "world", - "offset": "2712", + "offset": "2768", "byteLength": "64", "digest": "f5a5fd42d16a20302798ef6ed309979b43003d2320d9f0e8ea9831a92759fb4b" } ], - "footerOffset": "2776", + "footerOffset": "2832", "footerBytes": 48, - "totalBytes": "2824" + "totalBytes": "2880" } }, "corruption": [ @@ -178,17 +182,17 @@ }, { "name": "a flipped payload byte", - "offset": 2640, + "offset": 2696, "reason": "every payload carries its own digest" }, { "name": "a flipped footer digest byte", - "offset": 2784, + "offset": 2840, "reason": "the footer digest must match the contents" }, { "name": "a flipped footer magic byte", - "offset": 2816, + "offset": 2872, "reason": "a truncated file cannot look complete" } ] diff --git a/services/flysim/crates/fly-session-types/src/checkpoint.rs b/services/flysim/crates/fly-session-types/src/checkpoint.rs index 341f5c5..985fcb1 100644 --- a/services/flysim/crates/fly-session-types/src/checkpoint.rs +++ b/services/flysim/crates/fly-session-types/src/checkpoint.rs @@ -296,6 +296,11 @@ pub fn decode(bytes: &[u8]) -> Result { /// The manifest fields state-media-v1 section 4 requires, checked as a set: a manifest that /// omits one of them is not a complete checkpoint. +/// +/// `helperState` and `environment` join the list under the 2026-09-22 amendment to +/// checkpoint-envelope-v1 section 3: the first has been in that section's table from the +/// start and was missing here, and the second is the holder of the world's own payload, which +/// the table named for every other participant and not for the environment. pub const REQUIRED_MANIFEST_FIELDS: &[&str] = &[ "envelopeVersion", "checkpointId", @@ -308,6 +313,8 @@ pub const REQUIRED_MANIFEST_FIELDS: &[&str] = &[ "compatibility", "agents", "coordinator", + "environment", + "helperState", "payloads", ]; diff --git a/services/flysim/crates/fly-session/README.md b/services/flysim/crates/fly-session/README.md index 8e114e2..33f0594 100644 --- a/services/flysim/crates/fly-session/README.md +++ b/services/flysim/crates/fly-session/README.md @@ -44,6 +44,7 @@ Ready(k) ─ Prepare all agents concurrently ─────────── | `metrics` | Latency percentiles and the machine's core and memory counters | | `measure` | The execution-mode comparison of the guide's section 5 | | `cli` | The binary's subcommands: `agent`, `environment`, `measure` | +| `state` | The durable checkpoint store over `FLYSESS1`: compatibility, generations, the bounded writer | | `harness` | The runnable composition: router, the flies, one arena, one coordinator | ## Execution modes and the launcher @@ -178,6 +179,38 @@ harness.shutdown().await; event ids derived from epoch, source step, rule and ordinal. - **Executors.** The stateless identity executor only, as v1 specifies. +## Checkpoints and recovery + +The durable store is `state`, over the `FLYSESS1` layout the contract crate owns. + +- **One boundary, every participant.** `Coordinator::capture` runs at `Ready(k)` or + `Paused(k)` only. It takes its queue slot *before* the first `State.Capture`, so a saturated + writer refuses the capture rather than queueing it without bound, and the refusal is a + `BUSY` a stepping session survives rather than an epoch failure. +- **Capture and durability are two events.** `State.Capture` completes when an immutable + capture exists; `Coordinator::await_durable` completes when the store manifest rename has + happened, which is the durable commit point. Only the second moves the durable mark. A lost + save reply is `SaveOutcome::ReplyLost`, and `Coordinator::resolve_durable` then asks the + store about the *same* checkpoint instead of saving again. +- **The writer is bounded twice**, by outstanding captures and by queued bytes, and it owns + its payload handles until the bytes are committed or the job fails. A queued *replaceable* + capture is superseded by a later one, releasing its holds; a durable one never is. +- **The install is a group.** A restore selects a complete compatible generation, imports its + payloads as fresh artifacts, stages every participant, validates the coordinator's own + ledgers, and only then activates. A failure anywhere leaves the fence closed, and every + participant that got as far as staging is recorded as one that must be replaced before + another restore is attempted. +- **The fence lifts once.** `Failed -> Restoring(k) -> Paused(k)`, at the end of a complete + install and nowhere else. A fenced session takes no step, publishes nothing, captures + nothing and holds no artifact handle. +- **Nothing old crosses.** The fence drops every media handle; the restore imports fresh + artifacts; the environment re-renders its pending sensor pipeline from recorded + reconstruction inputs; and the new epoch's first audio chunk resumes the preserved sample + position and marks the discontinuity. +- **Epoch metadata in a trace.** `scope.epoch`, the batch id and every task event id are + derived from the epoch, so a resumed run's behaviour is compared through + `EpochRebase`, which rewrites exactly those and fails on anything it does not recognise. + ## Where this crate narrows or adds to the contract crate - **Required views.** `WorldObservation::validate_against` checks the views a result carries @@ -196,9 +229,9 @@ harness.shutdown().await; - **Fake workers.** There is no neural model and no emulator. What is modelled exactly is the ordering, the identity rules and the retry rules, not any numerical behaviour. -- **No state methods.** `State.Capture`, `State.StageRestore` and `State.ActivateRestore` are - STATE-01. The phase machine has their edges (`Capturing`, `Restoring`) and the workers do not - advertise them as implemented methods. +- **One environment, one task.** A checkpoint records the composition it was taken from, and a + restore refuses one taken under another backend, content, patch, controller or parser + identity. It does not migrate between compositions, and it does not try. - **No audience input.** The admitted pre-step stimulation list exists and is always empty. - **Pacing is coarse.** The pacing deadline rounds one step to whole nanoseconds for sleeping only; simulation time stays rational and that rounding never re-enters the accumulator. @@ -259,6 +292,9 @@ The three integration suites do not all run over both transports, and cannot: - `tests/processes.rs` runs over the Unix socket only, in all three execution modes. A participant in a process of its own has no in-memory transport to reach the router by, so the mode is the axis that suite varies and the transport is fixed. +- `tests/media.rs` and `tests/state.rs` run over both transports *and* in all three execution + modes: each acceptance body is written once and registered twice, by `both_transports!` in + the in-process composition and by `all_modes!` over the socket. - `tests/session.rs`: one world advance per complete batch; every agent Prepared before the advance; one task evaluation per transition; every agent committed before the next Prepare or @@ -275,6 +311,14 @@ The three integration suites do not all run over both transports, and cannot: allocation -- plus the sequential/reversed/parallel trace comparison across all three modes and the two process-mode section 4 rows: a router restart during a world advance, and an old worker's reply after a restart. +- `tests/state.rs`: the STATE-01 acceptance bullets -- an uninterrupted run and a resumed run + committing the same behaviour once the epoch metadata is rebased, a corrupt payload failing + the install as a group for every participant and for the coordinator's own ledger, a lost + save reply and an uncommitted store manifest both leaving the durable mark where it was, a + refused activation resuming no part of the world, the capture queue staying bounded under a + stalled writer, and old media and another parser's state failing to cross a recovery -- + plus the once-only restore token, the superseded replaceable capture, and the fence that + lifts only through a complete restore. - `tests/failures.rs`: a duplicate Prepare after a lost reply; a duplicate Commit; the same batch with altered controls; a lost Advance result; a cached artifact consumed by its first caller; one Commit failing after another succeeded; a replaced registration; a reply from diff --git a/services/flysim/crates/fly-session/src/agent.rs b/services/flysim/crates/fly-session/src/agent.rs index a52822f..22b5ac5 100644 --- a/services/flysim/crates/fly-session/src/agent.rs +++ b/services/flysim/crates/fly-session/src/agent.rs @@ -7,7 +7,7 @@ //! stimulation, then reinforces once, and executes no tick at all. Every mutating step bumps //! one counter, which is how a test proves a duplicate request changed nothing. -use std::collections::BTreeMap; +use std::collections::{BTreeMap, BTreeSet}; use serde_json::Value; @@ -193,6 +193,12 @@ pub struct AgentFaults { pub prepare_delay_ms: u64, /// Hold `Agent.Commit` open for this long. pub commit_delay_ms: u64, + /// Refuse `State.StageRestore`, so a group install meets one participant that will not + /// validate while the others already have. + pub fail_stage_restore: bool, + /// Refuse `State.ActivateRestore` after this worker has already staged, so a group meets + /// a failure halfway through activation. + pub fail_activate_restore: bool, } /// One fake agent worker's configuration. @@ -234,6 +240,10 @@ pub struct FakeAgentWorker { context: Option, context_digest: Option, prepared: Option<(DomainRequestId, PreparedDecision)>, + /// A validated replacement state that the live session cannot see yet. + staged: Option, + /// Restore tokens this worker has activated. A token activates once. + activated: BTreeSet, } impl FakeAgentWorker { @@ -248,10 +258,17 @@ impl FakeAgentWorker { context: None, context_digest: None, prepared: None, + staged: None, + activated: BTreeSet::new(), config, } } + /// True while a validated replacement state is staged and not yet activated. + pub fn has_staged_restore(&self) -> bool { + self.staged.is_some() + } + pub fn status(&self) -> StatusCell { self.status.clone() } @@ -657,7 +674,11 @@ impl WorkerEndpoint for FakeAgentWorker { } fn capabilities(&self) -> Vec { - vec![id("agent-step-v1"), id("pixel-observation-v1")] + vec![ + id("agent-step-v1"), + id("pixel-observation-v1"), + id(crate::state::CHECKPOINT_CAPABILITY), + ] } fn status_cell(&self) -> StatusCell { @@ -669,7 +690,14 @@ impl WorkerEndpoint for FakeAgentWorker { } fn methods(&self) -> Vec<&'static str> { - vec!["Agent.Initialize", "Agent.Prepare", "Agent.Commit"] + vec![ + "Agent.Initialize", + "Agent.Prepare", + "Agent.Commit", + "State.Capture", + "State.StageRestore", + "State.ActivateRestore", + ] } fn handle<'a>(&'a mut self, ctx: HandlerCtx<'a>) -> BoxFuture<'a, DomainResult> { @@ -678,6 +706,9 @@ impl WorkerEndpoint for FakeAgentWorker { "Agent.Initialize" => self.initialize(&ctx).await, "Agent.Prepare" => self.prepare(&ctx).await, "Agent.Commit" => self.commit(&ctx).await, + "State.Capture" => self.state_capture(&ctx).await, + "State.StageRestore" => self.state_stage_restore(&ctx).await, + "State.ActivateRestore" => self.state_activate_restore(&ctx).await, other => Err(DomainError::before( ErrorCode::Unsupported, format!("{other} is not an agent method"), @@ -690,7 +721,12 @@ impl WorkerEndpoint for FakeAgentWorker { /// The retention class table an agent endpoint follows, for a caller that wants it. pub fn agent_op_class(method: &str) -> Option { match method { - "Agent.Initialize" => Some(OpClass::Lifecycle), + // `ipc-v1` section 5: lifecycle *and capture* replies are retained until + // `Worker.Acknowledge`, which is also what lets a duplicate restore request replay + // its cached reply rather than staging or activating twice. + "Agent.Initialize" | "State.Capture" | "State.StageRestore" | "State.ActivateRestore" => { + Some(OpClass::Lifecycle) + } "Agent.Prepare" | "Agent.Commit" => Some(OpClass::StepMutation), _ => None, } @@ -712,3 +748,532 @@ pub fn synthetic_profile(agent_id: &Id, tick_duration: &RationalNs, warmup_ticks /// The per-agent contexts a bootstrap produced, keyed by agent id. pub type Contexts = BTreeMap; + +// ------------------------------------------------------------------------------------------- +// STATE-01: capture and restore + +/// The numerical model version this worker implements. It is part of a capture's +/// compatibility identity: the same profile and seed under another model is not the same +/// state (`workers-v1` section 2). +pub const MODEL_VERSION: &str = "fake-lcg-v1"; + +/// The plasticity rule version, for the same reason. +pub const PLASTICITY_VERSION: &str = "fake-reinforce-v1"; + +/// The version this payload layout is written and read under. +pub const AGENT_PAYLOAD_VERSION: u64 = 1; + +/// The dataset identity a synthetic agent resolves. +/// +/// There is no connectome dataset behind this worker, and a checkpoint says so with a stable +/// identity rather than omitting the field: "no dataset" has to be distinguishable from "the +/// dataset was not recorded". +pub fn dataset_digest() -> Digest { + digest_of_bytes(b"fly-session/no-dataset-v1") +} + +/// The capture compatibility digest of one agent (`workers-v1` section 2). +/// +/// The profile digest identifies the profile definition; this additionally covers the +/// resolved seed, the numerical model version and the plasticity rule, because two agents +/// with the same profile digest and different seeds hold state that is not interchangeable. +/// Every field it covers is one the checkpoint manifest already records in that agent's row, +/// so a restore derives the expected digest from the manifest rather than from the payload it +/// is about to validate. +pub fn agent_compatibility_digest( + agent_id: &Id, + profile_digest: &Digest, + dataset_digest: &Digest, + model_version: &str, + plasticity_version: &str, + seed: i32, +) -> Digest { + let value = serde_json::json!({ + "agentId": agent_id.as_str(), + "profileDigest": profile_digest.as_str(), + "datasetDigest": dataset_digest.as_str(), + "modelVersion": model_version, + "plasticityVersion": plasticity_version, + "seed": seed, + }); + digest_of(&value).expect("an agent compatibility block canonicalizes") +} + +impl FakeModel { + /// Every field of the model, so a resumed agent is this agent and not a fresh one. + fn capture(&self) -> Value { + serde_json::json!({ + "seed": self.seed, + "state": self.state.to_string(), + "mutations": self.mutations.to_string(), + "ticks": self.ticks.to_string(), + "stimulations": self.stimulations.to_string(), + "reinforcements": self.reinforcements.to_string(), + "learningEnabled": self.learning_enabled, + "learningUpdates": self.learning_updates.to_string(), + "learningChanged": self.learning_changed.to_string(), + "lastSignal": self.last_signal, + "inputValue": self.input_value.to_string(), + "inputInstalls": self.input_installs.to_string(), + }) + } + + fn restored(value: &Value) -> DomainResult { + let number = |key: &str| -> DomainResult { + value + .get(key) + .and_then(Value::as_str) + .ok_or_else(|| incompatible(format!("the agent payload has no {key}")))? + .parse::() + .map_err(|_| incompatible(format!("the agent payload's {key} is not a U64"))) + }; + let seed = value + .get("seed") + .and_then(Value::as_i64) + .and_then(|v| i32::try_from(v).ok()) + .ok_or_else(|| incompatible("the agent payload has no seed"))?; + let input_value = value + .get("inputValue") + .and_then(Value::as_str) + .ok_or_else(|| incompatible("the agent payload has no inputValue"))? + .parse::() + .map_err(|_| incompatible("the agent payload's inputValue is not an integer"))?; + let last_signal = value + .get("lastSignal") + .and_then(Value::as_f64) + .filter(|v| v.is_finite()) + .ok_or_else(|| incompatible("the agent payload's lastSignal is not finite"))?; + let learning_enabled = value + .get("learningEnabled") + .and_then(Value::as_bool) + .ok_or_else(|| incompatible("the agent payload has no learningEnabled"))?; + Ok(FakeModel { + seed, + state: number("state")?, + mutations: number("mutations")?, + ticks: number("ticks")?, + stimulations: number("stimulations")?, + reinforcements: number("reinforcements")?, + learning_enabled, + learning_updates: number("learningUpdates")?, + learning_changed: number("learningChanged")?, + last_signal, + input_value, + input_installs: number("inputInstalls")?, + }) + } +} + +fn incompatible(message: impl std::fmt::Display) -> DomainError { + DomainError::before(ErrorCode::IncompatibleState, message) +} + +/// One staged restore, held outside the live agent until it is activated. +struct StagedAgent { + token: Id, + checkpoint_id: Id, + scope: Scope, + model: FakeModel, + accumulator: TickAccumulator, + context: TypedValue, + profile: AssetRef, + committed_step: u64, +} + +impl FakeAgentWorker { + /// This worker's own compatibility identity, from its configuration and a resolved seed. + fn compatibility_digest(&self, profile: &AssetRef, seed: i32) -> Digest { + agent_compatibility_digest( + &self.config.agent_id, + &profile.digest, + &dataset_digest(), + MODEL_VERSION, + PLASTICITY_VERSION, + seed, + ) + } + + /// `State.Capture`: an immutable snapshot of this agent at its committed boundary. + /// + /// It is allowed at `Ready(k)` only. A Prepared agent holds half a transition, and there + /// is no coherent boundary to file that under. + async fn state_capture(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult { + let scope = ctx.scope()?.clone(); + self.check_epoch(&scope)?; + let AgentPhase::Ready(k) = self.phase.clone() else { + return Err(DomainError::before( + ErrorCode::InvalidPhase, + format!( + "State.Capture needs a quiescent Ready(k); this worker is {:?}", + self.phase + ), + )); + }; + if scope.step != k { + return Err(DomainError::before( + if scope.step < k { ErrorCode::StaleStep } else { ErrorCode::FutureStep }, + "State.Capture names a boundary this worker is not at", + )); + } + let params: CaptureParams = ctx.params()?; + let profile = self.profile.clone().expect("initialized"); + let context = self.context.clone().expect("initialized"); + let accumulator = self.accumulator.as_ref().expect("initialized"); + let previous = self.status.state(); + self.status.set_state(WorkerState::Capturing); + let payload = serde_json::json!({ + "payloadVersion": AGENT_PAYLOAD_VERSION, + "kind": "agent", + "agentId": self.config.agent_id.as_str(), + "checkpointId": params.checkpoint_id.as_str(), + "sourceScope": scope.to_json(), + "committedStep": k.to_string(), + "profile": profile.to_json(), + "modelVersion": MODEL_VERSION, + "plasticityVersion": PLASTICITY_VERSION, + "datasetDigest": dataset_digest().as_str(), + "model": self.model.capture(), + "accumulator": { + "tickDuration": accumulator.tick_duration().to_json(), + "remainder": accumulator.remainder().to_json(), + "executedTicks": accumulator.executed_ticks().to_string(), + "warmupOffset": accumulator.warmup_offset().to_string(), + }, + "context": context.to_json(), + }); + let bytes = canonicalize(&payload) + .map_err(|e| DomainError::invalid(format!("State.Capture: {}", e.0)))? + .into_bytes(); + let digest = digest_of_bytes(&bytes); + let artifact = crate::state::seal_payload(ctx.client, &bytes, &digest).await?; + // Capture is a read of the model, not a mutation of it: nothing above changed a + // counter, and the worker goes back to the boundary it was already at. + self.status.set_state(previous); + let result = CaptureResult { + checkpoint_id: params.checkpoint_id, + boundary: k, + compatibility_digest: self.compatibility_digest(&profile, self.model.seed()), + payload: artifact.reference().clone(), + }; + Ok(HandlerReply::with_artifacts( + object(result.to_json()), + vec![(crate::state::PAYLOAD_ATTACHMENT.to_owned(), artifact)], + )) + } + + /// `State.StageRestore`: validate a replacement state into a staging slot. + /// + /// Nothing the live session can see changes here, and the worker keeps whatever state it + /// had. It is allowed on an uninitialized replacement or a quiescent worker only; a + /// failed one is neither, which is why a group that failed is replaced rather than + /// reused. + async fn state_stage_restore(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult { + let scope = ctx.scope()?.clone(); + if scope.session_id != self.config.session_id { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "this worker belongs to another session", + )); + } + match &self.phase { + AgentPhase::Uninitialized | AgentPhase::Ready(_) => {} + other => { + return Err(DomainError::before( + ErrorCode::InvalidPhase, + format!( + "State.StageRestore needs an uninitialized replacement or a quiescent \ +worker; this worker is {other:?}" + ), + )); + } + } + if let Some(epoch) = &self.epoch + && *epoch == scope.epoch + { + return Err(DomainError::before( + ErrorCode::StaleEpoch, + "State.StageRestore proposes the epoch this worker is already running", + )); + } + let params: StageRestoreParams = ctx.params()?; + if params.source_scope.step != scope.step { + return Err(DomainError::invalid( + "State.StageRestore's scope step must be the source boundary", + )); + } + let artifact = ctx.artifact(crate::state::PAYLOAD_ATTACHMENT)?; + if artifact.reference() != ¶ms.payload { + return Err(DomainError::before( + ErrorCode::BufferInvalid, + "the staged payload attachment is not the artifact the request names", + )); + } + let bytes = artifact.read_all().await.map_err(|e| { + DomainError::before( + ErrorCode::BufferInvalid, + format!("the staged payload could not be read: {}", e.message), + ) + })?; + let declared = params + .payload + .digest + .clone() + .ok_or_else(|| incompatible("a checkpoint payload must carry a content digest"))?; + let actual = digest_of_bytes(&bytes); + if actual != declared || bytes.len() as u64 != params.payload.byte_length { + return Err(incompatible( + "the staged payload is not the content the request declares", + )); + } + let value: Value = serde_json::from_slice(&bytes) + .map_err(|e| DomainError::invalid(format!("the staged payload is not JSON: {e}")))?; + let text = |key: &str| -> DomainResult { + value + .get(key) + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| incompatible(format!("the agent payload has no {key}"))) + }; + if value.get("payloadVersion").and_then(Value::as_u64) != Some(AGENT_PAYLOAD_VERSION) { + return Err(incompatible("the agent payload is another payload version")); + } + if text("kind")? != "agent" { + return Err(incompatible("this payload is not an agent's state")); + } + if text("agentId")? != self.config.agent_id { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "the staged payload belongs to another agent", + )); + } + if text("checkpointId")? != params.checkpoint_id { + return Err(incompatible("the staged payload belongs to another checkpoint")); + } + if text("modelVersion")? != MODEL_VERSION || text("plasticityVersion")? != PLASTICITY_VERSION + { + return Err(incompatible( + "the staged payload was captured under another numerical model", + )); + } + let source_scope = Scope::from_json( + value + .get("sourceScope") + .ok_or_else(|| incompatible("the agent payload has no sourceScope"))?, + ) + .map_err(|e| incompatible(format!("the agent payload's sourceScope: {}", e.0)))?; + if source_scope != params.source_scope { + return Err(incompatible( + "the staged payload was captured at another source scope", + )); + } + let committed_step: u64 = text("committedStep")? + .parse() + .map_err(|_| incompatible("the agent payload's committedStep is not a U64"))?; + if committed_step != params.source_scope.step { + return Err(incompatible( + "the staged payload's committed step is not the source boundary", + )); + } + let profile = AssetRef::from_json( + value + .get("profile") + .ok_or_else(|| incompatible("the agent payload has no profile"))?, + ) + .map_err(|e| incompatible(format!("the agent payload's profile: {}", e.0)))?; + let model = FakeModel::restored( + value + .get("model") + .ok_or_else(|| incompatible("the agent payload has no model"))?, + )?; + // The compatibility digest is recomputed from this worker's own configuration and the + // identity the payload declares. A capture of the same profile under another seed, or + // of another agent's brain, fails here and never reaches activation. + let computed = self.compatibility_digest(&profile, model.seed()); + if computed != params.compatibility_digest { + return Err(incompatible(format!( + "the staged state's compatibility {computed} is not the {} the restore \ +requires", + params.compatibility_digest + ))); + } + let accumulator_value = value + .get("accumulator") + .ok_or_else(|| incompatible("the agent payload has no accumulator"))?; + let rational = |key: &str| -> DomainResult { + RationalNs::from_json( + accumulator_value + .get(key) + .ok_or_else(|| incompatible(format!("the accumulator has no {key}")))?, + ) + .map_err(|e| incompatible(format!("the accumulator's {key}: {}", e.0))) + }; + let counter = |key: &str| -> DomainResult { + accumulator_value + .get(key) + .and_then(Value::as_str) + .ok_or_else(|| incompatible(format!("the accumulator has no {key}")))? + .parse::() + .map_err(|_| incompatible(format!("the accumulator's {key} is not a U64"))) + }; + let tick_duration = rational("tickDuration")?; + if tick_duration != self.config.tick_duration { + return Err(incompatible( + "the staged state was captured at another model tick duration", + )); + } + let accumulator = TickAccumulator::restored( + tick_duration, + rational("remainder")?, + counter("executedTicks")?, + counter("warmupOffset")?, + ) + .map_err(incompatible)?; + let context = TypedValue::from_json( + value + .get("context") + .ok_or_else(|| incompatible("the agent payload has no context"))?, + ) + .map_err(|e| incompatible(format!("the agent payload's context: {}", e.0)))?; + FakeAgentWorker::available_actions(&context)?; + + if self.config.faults.fail_stage_restore { + // The row where a group validates three participants and the fourth does not. + // Nothing is staged here and nothing is staged anywhere else either: the + // coordinator abandons the whole install. + return Err(incompatible( + "injected staging refusal: this participant's replacement state does not \ +validate", + )); + } + // One staged restore at a time. A second proposal replaces nothing silently. + if let Some(staged) = &self.staged { + return Err(DomainError::before( + ErrorCode::Conflict, + format!( + "this worker already holds the staged restore {} for checkpoint {}", + staged.token, staged.checkpoint_id + ), + )); + } + let token = restore_token(¶ms.checkpoint_id, &scope, &actual, &self.config.incarnation_id); + if self.activated.contains(&token) { + return Err(DomainError::before( + ErrorCode::Conflict, + "this exact restore was already activated on this worker", + )); + } + self.staged = Some(StagedAgent { + token: token.clone(), + checkpoint_id: params.checkpoint_id.clone(), + scope: scope.clone(), + model, + accumulator, + context, + profile, + committed_step, + }); + self.status.set_state(WorkerState::StagedRestore); + let result = StageRestoreResult { + checkpoint_id: params.checkpoint_id, + restore_token: token, + }; + Ok(HandlerReply::from(&result)) + } + + /// `State.ActivateRestore`: install the staged state under its new scope, without a tick. + /// + /// The token activates once. A duplicate domain request replays the cached reply through + /// the shell's result cache; a fresh request naming an already activated token is a + /// conflict, which is what stops a second group from being resumed from the same bytes. + async fn state_activate_restore( + &mut self, + ctx: &HandlerCtx<'_>, + ) -> DomainResult { + let params: ActivateRestoreParams = ctx.params()?; + if self.activated.contains(¶ms.restore_token) { + return Err(DomainError::before( + ErrorCode::Conflict, + "this restore token has already been activated", + )); + } + let Some(staged) = self.staged.take() else { + return Err(DomainError::before( + ErrorCode::InvalidPhase, + "this worker holds no staged restore", + )); + }; + if staged.token != params.restore_token { + // Put it back: naming another token is not a reason to discard this one. + let token = staged.token.clone(); + self.staged = Some(staged); + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + format!("this worker's staged restore is {token}, not {}", params.restore_token), + )); + } + if self.config.faults.fail_activate_restore { + let token = staged.token.clone(); + self.staged = Some(staged); + self.status.set_state(WorkerState::Failed); + return Err(DomainError::new( + ErrorCode::BackendFailure, + format!("injected activation failure; {token} stays staged and unresumed"), + MutationCertainty::None, + )); + } + self.status.set_state(WorkerState::Restoring); + let StagedAgent { + token, + checkpoint_id, + scope, + model, + accumulator, + context, + profile, + committed_step, + } = staged; + self.model = model; + self.accumulator = Some(accumulator); + self.context_digest = Some(context.digest()); + self.context = Some(context); + self.profile = Some(profile); + self.epoch = Some(scope.epoch.clone()); + self.prepared = None; + self.phase = AgentPhase::Ready(committed_step); + self.activated.insert(token); + self.status.set_state(WorkerState::Ready); + self.status.set_scope(Some(scope_at( + &scope.session_id, + &scope.epoch, + committed_step, + ))); + self.status.advance_to(self.model.mutations()); + let result = ActivateRestoreResult { + committed_step, + checkpoint_id, + // An agent returns a null observation; the environment returns the world's. + observation: None, + }; + result + .validate_for_role(Role::Agent) + .map_err(|e| DomainError::invalid(e.0))?; + Ok(HandlerReply::from(&result)) + } +} + +/// A restore token bound to the checkpoint, the proposed scope, the payload bytes and the +/// worker incarnation staging them. +/// +/// `state-media-v1` section 5 binds a token to scope, payload and checkpoint. Binding it to +/// the incarnation as well is what keeps a token minted by a worker that has since been +/// replaced from activating anything on its replacement. +pub fn restore_token(checkpoint_id: &Id, scope: &Scope, payload_digest: &Digest, incarnation: &Id) -> Id { + let digest = digest_of_bytes( + format!( + "fly-session/restore-token-v1\n{checkpoint_id}\n{}\n{}\n{}\n{payload_digest}\n{incarnation}\n", + scope.session_id, scope.epoch, scope.step + ) + .as_bytes(), + ); + parse_id(&format!("rt-{}", &digest[..32])).expect("a hex suffix is an Id") +} diff --git a/services/flysim/crates/fly-session/src/cli.rs b/services/flysim/crates/fly-session/src/cli.rs index 1f7bc68..bbd6bbf 100644 --- a/services/flysim/crates/fly-session/src/cli.rs +++ b/services/flysim/crates/fly-session/src/cli.rs @@ -46,8 +46,10 @@ Worker options (agent and environment): agent: --agent ID --port ID --tick-numerator N --tick-denominator N --warmup-ticks N [--prepare-delay-ms N] [--commit-delay-ms N] [--fail-commit-at-step N] + [--fail-stage-restore 0|1] [--fail-activate-restore 0|1] environment: --worker ID --ports p1,p2 --step-numerator N --step-denominator N [--advance-delay-ms N] [--omit-view-at-boundary N] + [--fail-stage-restore 0|1] [--fail-activate-restore 0|1] Measure options: --steps N transitions per run (default 200) @@ -145,6 +147,17 @@ impl Options { } } + /// A flag whose value is `0` or `1`. Anything else is an error naming it, so a + /// mistyped injection is a failed launch rather than a fault that never fires. + fn flag(&self, name: &str) -> Result { + match self.0.get(name) { + None => Ok(false), + Some(value) if value == "0" => Ok(false), + Some(value) if value == "1" => Ok(true), + Some(value) => Err(format!("--{name}: {value:?} is not 0 or 1")), + } + } + fn opt_u64(&self, name: &str) -> Result, String> { match self.0.get(name) { None => Ok(None), @@ -197,6 +210,8 @@ fn serve(role: &str, options: &Options) -> Result<(), String> { fail_commit_at_step: options.opt_u64(flags::FAIL_COMMIT_AT_STEP)?, prepare_delay_ms: options.u64(flags::PREPARE_DELAY_MS, 0)?, commit_delay_ms: options.u64(flags::COMMIT_DELAY_MS, 0)?, + fail_stage_restore: options.flag(flags::FAIL_STAGE_RESTORE)?, + fail_activate_restore: options.flag(flags::FAIL_ACTIVATE_RESTORE)?, }, client_id: client_id.clone(), service: service.clone(), @@ -219,6 +234,8 @@ fn serve(role: &str, options: &Options) -> Result<(), String> { omit_audio_at_boundary: options.opt_u64(flags::OMIT_AUDIO_AT_BOUNDARY)?, overlapping_audio_at_boundary: options .opt_u64(flags::OVERLAPPING_AUDIO_AT_BOUNDARY)?, + fail_stage_restore: options.flag(flags::FAIL_STAGE_RESTORE)?, + fail_activate_restore: options.flag(flags::FAIL_ACTIVATE_RESTORE)?, }, client_id: client_id.clone(), service: service.clone(), diff --git a/services/flysim/crates/fly-session/src/clock.rs b/services/flysim/crates/fly-session/src/clock.rs index 080c12a..aa50ff2 100644 --- a/services/flysim/crates/fly-session/src/clock.rs +++ b/services/flysim/crates/fly-session/src/clock.rs @@ -29,6 +29,30 @@ impl TickAccumulator { }) } + /// The exact accumulator a capture recorded. + /// + /// The remainder is restored, never rounded or reset: a resumed agent that started its + /// first interval from zero would drift away from the run it is supposed to continue. + pub fn restored( + tick_duration: RationalNs, + remainder: RationalNs, + executed_ticks: u64, + warmup_offset: u64, + ) -> Result { + let mut accumulator = TickAccumulator::new(tick_duration)?; + remainder.validate().map_err(|e| e.0)?; + if remainder >= tick_duration { + return Err("a captured remainder is not below one model tick".to_owned()); + } + if warmup_offset > executed_ticks { + return Err("a captured warm-up offset exceeds the executed tick count".to_owned()); + } + accumulator.remainder = remainder; + accumulator.executed_ticks = executed_ticks; + accumulator.warmup_offset = warmup_offset; + Ok(accumulator) + } + pub fn tick_duration(&self) -> RationalNs { self.tick_duration } diff --git a/services/flysim/crates/fly-session/src/coordinator.rs b/services/flysim/crates/fly-session/src/coordinator.rs index e7e4bb7..bde1183 100644 --- a/services/flysim/crates/fly-session/src/coordinator.rs +++ b/services/flysim/crates/fly-session/src/coordinator.rs @@ -141,6 +141,10 @@ pub struct Deadlines { pub resolve_attempts: u32, /// A separate, larger budget for `Worker.Hello` and the `Initialize` methods. pub boot: Duration, + /// A separate budget for the `State.*` methods, which `ipc-v1` section 6 gives one: + /// a capture serializes a participant and a restore validates and installs one, and + /// neither is a step whose latency the probe was chosen for. + pub capture: Duration, } /// How long the resolution waits between attempts. @@ -167,6 +171,7 @@ impl Default for Deadlines { // Over sixteen seconds of pauses: the budget above is what terminates. resolve_attempts: 8192, boot: Duration::from_secs(30), + capture: Duration::from_secs(30), } } } @@ -177,6 +182,8 @@ pub struct Topics { pub descriptor: String, pub snapshots: String, pub events: String, + /// Where the distinct captured/queued/committed/failed/superseded checkpoint events go. + pub checkpoints: String, } impl Topics { @@ -185,6 +192,7 @@ impl Topics { descriptor: format!("session.{session_id}.descriptor"), snapshots: format!("session.{session_id}.snapshots"), events: format!("session.{session_id}.events"), + checkpoints: format!("session.{session_id}.checkpoints"), } } } @@ -202,6 +210,11 @@ pub struct AgentSlot { pub tick_duration: RationalNs, pub warmup_ticks: u64, pub committed_step: u64, + /// The tick count and remainder this agent last reported, which are what the checkpoint + /// manifest records for it. They are metadata about the payload, never a substitute for + /// it: the agent's own capture is the state that is restored. + pub brain_ticks: u64, + pub remainder: RationalNs, context: TypedValue, context_digest: Digest, prepared: Option, @@ -226,6 +239,8 @@ impl AgentSlot { tick_duration: RationalNs::ZERO, warmup_ticks: 0, committed_step: 0, + brain_ticks: 0, + remainder: RationalNs::ZERO, context: TypedValue::new(crate::task::context_schema(), Value::Object(Map::new())) .expect("an empty context object is a valid typed value"), context_digest: digest_of_bytes(b""), @@ -300,6 +315,12 @@ pub struct Coordinator { /// Set when the epoch failed: every old handle, route and reply is invalid from here on /// and only a coherent restore may lift it. fenced: bool, + /// The durable store's bounded writer, when this composition has one. + writer: Option, + /// The last checkpoint whose saved acknowledgment arrived, and its boundary. + durable: Option<(Id, u64)>, + /// Participants that installed a restore a group install then abandoned. + tainted: BTreeSet, started: std::time::Instant, last_advance_request: Option, last_commit_requests: Vec, @@ -361,6 +382,9 @@ impl Coordinator { metrics: Metrics::default(), blame: None, fenced: false, + writer: None, + durable: None, + tainted: BTreeSet::new(), started: std::time::Instant::now(), last_advance_request: None, last_commit_requests: Vec::new(), @@ -489,8 +513,16 @@ impl Coordinator { let (from, to) = self.phases.fail(); self.trace.phase(from, to); self.fenced = true; + // Whatever replies this session still owed an acknowledgment for belong to + // participants of an invalid epoch. Carrying them across a recovery would send + // `Worker.Acknowledge` to a registration that is gone. + self.lifecycle_acks.clear(); + // Every handle this session held on the old epoch's media goes with the fence: a + // recovery imports fresh artifacts and never expects one of these back. self.views.clear(); self.pending_views.clear(); + self.audio.clear(); + self.pending_audio.clear(); match &participant { Some(who) => self.audit.push(format!("fail:{detail}:{who}")), None => self.audit.push(format!("fail:{detail}")), @@ -631,6 +663,10 @@ impl Coordinator { (self.topics.descriptor.clone(), flybus::Retained::Latest), (self.topics.snapshots.clone(), flybus::Retained::Latest), (self.topics.events.clone(), flybus::Retained::None), + // Checkpoint events are a stream of distinct facts, not a latest value: a + // "committed" that replaced a "queued" would erase the distinction the durable + // commit rules are built on. + (self.topics.checkpoints.clone(), flybus::Retained::None), ] { self.bus.declare_topic(&name, retained).await.map_err(|e| { let error = DomainError::new( @@ -716,6 +752,15 @@ impl Coordinator { if let Err(e) = media::check_required_views(&result.descriptor, &result.observation) { return Err(self.fail_now(e, "observation-0")); } + // O[0] ran no transition either, so it carries no chunk, and that is checked rather + // than assumed from its boundary number. + if let Err(e) = media::check_required_audio( + &result.descriptor, + &result.observation, + media::ObservationOrigin::Installed, + ) { + return Err(self.fail_now(e, "observation-0")); + } self.descriptor = Some(result.descriptor); self.observation = Some(result.observation); self.lifecycle_acks.push((worker, reply.request_id.clone())); @@ -815,6 +860,10 @@ impl Coordinator { self.agents[index].tick_duration = result.tick_duration; self.agents[index].warmup_ticks = result.warmup_ticks; self.agents[index].committed_step = 0; + // At boundary 0 the agent has executed exactly its warm-up, with no interval + // consumed, so the remainder is zero. + self.agents[index].brain_ticks = result.telemetry.brain_ticks; + self.agents[index].remainder = RationalNs::ZERO; self.lifecycle_acks.push((slot_worker, reply.request_id.clone())); let agent_id = self.agents[index].agent_id.clone(); self.audit.push(format!("agent.initialize:{agent_id}")); @@ -1013,6 +1062,8 @@ impl Coordinator { self.blame(Some(worker.worker_id.clone())); let deadline = if method.ends_with("Initialize") || method == "Worker.Hello" { self.deadlines.boot + } else if method.starts_with("State.") { + self.deadlines.capture } else { self.deadlines.probe }; @@ -1692,6 +1743,12 @@ impl Coordinator { method, )); } + if let Some(slot) = self.agents.iter_mut().find(|slot| slot.agent_id == agent_id) { + // What the agent will be at once this transition commits: Prepare is the only + // phase that advances the accumulator. + slot.brain_ticks = decision.brain_ticks; + slot.remainder = decision.remainder; + } self.audit.push(format!("prepared:{agent_id}@{k}")); self.stats.prepares += 1; self.blame(None); @@ -2110,7 +2167,11 @@ impl Coordinator { } // Audio has no sensory role here, but its chunks still cannot overlap or go backwards // inside an epoch, and a stale one must not reach presentation as current. - if let Err(e) = media::check_required_audio(descriptor, &result.observation) { + if let Err(e) = media::check_required_audio( + descriptor, + &result.observation, + media::ObservationOrigin::Transition, + ) { return Err(self.fail_now(e, "step-result")); } if let Err(e) = self.timelines.accept(descriptor, &result.observation) { @@ -2605,3 +2666,1429 @@ impl Coordinator { Ok(()) } } + +// ------------------------------------------------------------------------------------------- +// STATE-01: coherent all-participant capture and recovery + +/// A capture that exists and has been queued, whose durable outcome has not arrived yet. +/// +/// `State.Capture` completes when an immutable capture exists, not when a backend save was +/// requested, and only durable completion produces a saved acknowledgment. Those are two +/// events, so they are two calls. +#[derive(Debug)] +pub struct CaptureTicket { + pub checkpoint_id: Id, + pub boundary: u64, + receiver: tokio::sync::oneshot::Receiver, +} + +/// What one completed group restore installed. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct RestoreReport { + pub checkpoint_id: Id, + pub boundary: u64, + pub epoch: Id, + /// Every participant that staged, in the order they were asked. + pub staged: Vec, + /// Every participant that activated, in the order they were asked. + pub activated: Vec, + /// The once-only token each participant staged under, so a caller can prove a token + /// activates once rather than being told so. + pub tokens: Vec<(Id, Id)>, + /// The artifacts the payloads were imported as. None of them existed before this restore. + pub imported: Vec, +} + +/// The payload name the coordinator's own session record travels under. +impl Coordinator { + /// Attaches the durable store's bounded writer. Without one this session captures nothing + /// and says so, rather than pretending to. + pub fn attach_store(&mut self, writer: crate::state::CheckpointWriter) { + self.writer = Some(writer); + } + + pub fn writer(&self) -> Option<&crate::state::CheckpointWriter> { + self.writer.as_ref() + } + + /// Stops the checkpoint writer and waits for its task. + pub async fn shutdown_store(&mut self) { + if let Some(writer) = self.writer.take() { + writer.shutdown().await; + } + } + + /// The last checkpoint whose *saved acknowledgment* this coordinator received, and its + /// boundary. It moves on durable completion and on nothing else. + pub fn durable(&self) -> Option<(Id, u64)> { + self.durable.clone() + } + + /// Participants that installed a restore in a group install that then failed. + /// + /// They hold state no group ever resumed. A further restore is refused until each one has + /// been replaced, which is how "a failure during activation cannot resume half a world" + /// survives the next attempt as well as this one. + pub fn tainted(&self) -> Vec { + self.tainted.iter().cloned().collect() + } + + /// The live composition's compatibility identities. + pub fn compatibility(&self) -> Outcome { + match &self.descriptor { + Some(descriptor) => Ok(crate::state::Compatibility::of(descriptor)), + None => Err(SessionFailure { + error: DomainError::before( + ErrorCode::InvalidPhase, + "the session has no environment descriptor to take compatibility from", + ), + phase: self.phases.phase().label(), + detail: "compatibility".to_owned(), + participant: None, + }), + } + } + + fn agent_compatibility(&self, slot: &AgentSlot) -> Digest { + crate::agent::agent_compatibility_digest( + &slot.agent_id, + &slot.profile.digest, + &crate::agent::dataset_digest(), + crate::agent::MODEL_VERSION, + crate::agent::PLASTICITY_VERSION, + slot.seed, + ) + } + + /// The coordinator's own session record: what it must hold again to resume this boundary. + /// + /// `state-media-v1` section 4 puts the next decision state, the admission state and the + /// event watermarks in the coordinator's own payloads. This is that payload: the + /// composition has no audience input, so the admission record says so explicitly rather + /// than being absent. + fn coordinator_record(&self, boundary: u64) -> Value { + let (last_source_step, issued) = self.task.event_watermarks(); + json!({ + "payloadVersion": 1, + "kind": "coordinator", + "committedStep": boundary.to_string(), + "admission": { + "audienceInput": "none-configured", + "admitted": Value::Array(Vec::new()), + }, + "eventWatermarks": { + "lastSourceStep": last_source_step.to_string(), + "issued": issued.to_string(), + }, + "audioPositions": Value::Object( + self.timelines + .positions() + .into_iter() + .map(|(stream, sample)| (stream, Value::String(sample.to_string()))) + .collect(), + ), + "agents": Value::Array( + self.agents + .iter() + .map(|slot| json!({ + "agentId": slot.agent_id.as_str(), + "portId": slot.port_id.as_str(), + "profile": slot.profile.to_json(), + "seed": slot.seed, + "workerThreads": slot.worker_threads.to_string(), + "committedStep": slot.committed_step.to_string(), + "brainTicks": slot.brain_ticks.to_string(), + "remainder": slot.remainder.to_json(), + "context": slot.context.to_json(), + })) + .collect(), + ), + }) + } + + /// Seals one coordinator-owned payload as an immutable artifact. + async fn seal_own(&mut self, name: &str, value: &Value) -> Outcome { + let bytes = match canonicalize(value) { + Ok(text) => text.into_bytes(), + Err(e) => { + return Err(self.fail_now( + DomainError::invalid(format!("checkpoint payload {name}: {}", e.0)), + "capture", + )); + } + }; + let digest = digest_of_bytes(&bytes); + let artifact = match crate::state::seal_payload(&self.bus, &bytes, &digest).await { + Ok(artifact) => artifact, + Err(e) => return Err(self.fail_now(e, "capture")), + }; + Ok(crate::state::CapturedPayload { + name: name.to_owned(), + byte_length: bytes.len() as u64, + digest, + artifact, + }) + } + + /// Takes one coherent all-participant capture at the committed boundary and queues it. + /// + /// The queue slot is taken *first*: a saturated writer refuses before a single + /// `State.Capture` is sent, which is the only way "reject or defer before capture" can be + /// true rather than aspirational. + pub async fn capture(&mut self, checkpoint_id: &Id, replaceable: bool) -> Outcome { + if self.fenced { + return Err(SessionFailure { + error: DomainError::before( + ErrorCode::InvalidPhase, + "the epoch is fenced; a fenced session captures nothing", + ), + phase: self.phases.phase().label(), + detail: "fenced".to_owned(), + participant: None, + }); + } + // A capture that arrives while a transition is in flight is a race, not a bug in the + // transaction: the supervisor asking for one does not know where the session is. It + // is refused by name and the epoch is untouched, unlike a phase edge the machine + // itself takes. + let Some(boundary) = self.phases.phase().committed_boundary() else { + self.audit.push(format!("capture-refused:{checkpoint_id}")); + return Err(SessionFailure { + error: DomainError::before( + ErrorCode::InvalidPhase, + "a coherent checkpoint is taken at a committed boundary only", + ), + phase: self.phases.phase().label(), + detail: "capture".to_owned(), + participant: None, + }); + }; + let origin = self.phases.phase(); + if self.writer.is_none() { + return Err(SessionFailure { + error: DomainError::before( + ErrorCode::Unsupported, + "this session has no checkpoint store attached", + ), + phase: self.phases.phase().label(), + detail: "capture".to_owned(), + participant: None, + }); + } + // Before anything is captured. + let reservation = match self.writer.as_ref().expect("checked").reserve() { + Ok(reservation) => reservation, + Err(e) => { + // A refused capture is not a session failure: the boundary stands, the + // session keeps stepping and the caller is told the queue is full. + self.audit.push(format!("capture-refused:{checkpoint_id}")); + return Err(SessionFailure { + error: e, + phase: self.phases.phase().label(), + detail: "capture".to_owned(), + participant: None, + }); + } + }; + self.transition(Phase::Capturing(boundary))?; + let result = self + .capture_group(checkpoint_id, boundary, replaceable, reservation) + .await; + match result { + Ok(ticket) => { + self.transition(origin)?; + self.audit.push(format!("captured:{checkpoint_id}@{boundary}")); + Ok(ticket) + } + Err(e) => Err(e), + } + } + + async fn capture_group( + &mut self, + checkpoint_id: &Id, + boundary: u64, + replaceable: bool, + reservation: crate::state::Reservation, + ) -> Outcome { + let scope = self.scope(boundary); + let compatibility = self.compatibility()?; + let params = object(CaptureParams { checkpoint_id: checkpoint_id.clone() }.to_json()); + let want = vec![crate::state::PAYLOAD_ATTACHMENT.to_owned()]; + let mut payloads = Vec::new(); + let mut acknowledge: Vec<(WorkerRef, DomainRequestId)> = Vec::new(); + + // The world first, then the agents in sorted order: one boundary, every participant. + let environment = self.environment.clone(); + let reply = self + .call( + &environment, + "State.Capture", + Some(scope.clone()), + params.clone(), + &[], + &want, + ) + .await?; + let world: CaptureResult = reply.parse().map_err(|e| self.fail_now(e, "capture"))?; + let world_payload = self.accept_capture( + &reply, + &world, + checkpoint_id, + boundary, + &compatibility.digest(), + crate::state::WORLD_PAYLOAD, + )?; + payloads.push(world_payload); + acknowledge.push((environment, reply.request_id.clone())); + + let mut agent_rows = Vec::new(); + for index in 0..self.agents.len() { + let slot_worker = self.agents[index].worker.clone(); + let agent_id = self.agents[index].agent_id.clone(); + let expected = self.agent_compatibility(&self.agents[index]); + let reply = self + .call( + &slot_worker, + "State.Capture", + Some(scope.clone()), + params.clone(), + &[], + &want, + ) + .await?; + let captured: CaptureResult = reply.parse().map_err(|e| self.fail_now(e, "capture"))?; + let name = crate::state::agent_payload(&agent_id); + let payload = + self.accept_capture(&reply, &captured, checkpoint_id, boundary, &expected, &name)?; + payloads.push(payload); + acknowledge.push((slot_worker, reply.request_id.clone())); + let slot = &self.agents[index]; + agent_rows.push(crate::state::AgentEntry { + agent_id: agent_id.clone(), + profile_digest: slot.profile.digest.clone(), + dataset_digest: crate::agent::dataset_digest(), + model_version: crate::agent::MODEL_VERSION.to_owned(), + plasticity_version: crate::agent::PLASTICITY_VERSION.to_owned(), + seed: slot.seed, + brain_ticks: slot.brain_ticks, + remainder: slot.remainder, + payload: name, + }); + } + + // The coordinator's own ledgers, sealed the same way so the writer treats every + // payload alike. + let ledger = self.task.capture().map_err(|e| self.fail_now(e, "capture"))?; + payloads.push( + self.seal_own(crate::state::TASK_LEDGER_PAYLOAD, &ledger.to_json()) + .await?, + ); + let inspection = self + .observation + .as_ref() + .map(|observation| observation.inspection.to_json()) + .ok_or_else(|| { + DomainError::before(ErrorCode::InvalidPhase, "the session never bootstrapped") + }); + let inspection = match inspection { + Ok(value) => value, + Err(e) => return Err(self.fail_now(e, "capture")), + }; + payloads.push( + self.seal_own(crate::state::PRIOR_INSPECTION_PAYLOAD, &inspection) + .await?, + ); + let mut executor_rows = Vec::new(); + for agent_id in self.agent_ids() { + let state = { + let executor = self.executors.get(&agent_id).ok_or_else(|| { + DomainError::before( + ErrorCode::IdentityMismatch, + format!("{agent_id} has no configured action executor"), + ) + }); + match executor { + Ok(executor) => executor.capture().map_err(|e| (e, agent_id.clone())), + Err(e) => Err((e, agent_id.clone())), + } + }; + let state = match state { + Ok(state) => state, + Err((e, who)) => { + self.blame(Some(who)); + return Err(self.fail_now(e, "capture")); + } + }; + let name = crate::state::executor_payload(&agent_id); + payloads.push(self.seal_own(&name, &state.to_json()).await?); + executor_rows.push((agent_id, name)); + } + let record = self.coordinator_record(boundary); + payloads.push( + self.seal_own(crate::state::ADMISSION_PAYLOAD, &record) + .await?, + ); + + let (last_source_step, issued) = self.task.event_watermarks(); + let world_time = match self.observation.as_ref().map(|o| o.world_time) { + Some(world_time) => Ok(world_time), + None => Err(self.fail_now( + DomainError::before( + ErrorCode::InvalidPhase, + "the session has no observation, so it is at no world time to record", + ), + "capture", + )), + }; + let manifest = crate::state::CheckpointManifest { + checkpoint_id: checkpoint_id.clone(), + source_scope: scope.clone(), + episode_id: self.episode_id.clone(), + world_time: world_time?, + scheduler_id: "lockstep-v1".to_owned(), + composition_digest: self.composition_digest(), + port_map: self + .agents + .iter() + .map(|slot| (slot.port_id.clone(), slot.agent_id.clone())) + .collect(), + compatibility: compatibility.clone(), + agents: agent_rows, + coordinator: crate::state::CoordinatorEntry { + task_ledger: crate::state::TASK_LEDGER_PAYLOAD.to_owned(), + prior_inspection: crate::state::PRIOR_INSPECTION_PAYLOAD.to_owned(), + executor_state: executor_rows, + admission_state: crate::state::ADMISSION_PAYLOAD.to_owned(), + event_watermarks: crate::state::EventWatermarks { + last_source_step, + issued, + }, + }, + environment: crate::state::EnvironmentEntry { + worker_id: self.environment.worker_id.clone(), + payload: crate::state::WORLD_PAYLOAD.to_owned(), + }, + // No external helper takes part in this composition, and the manifest says so + // rather than leaving the field out. + helper_state: Vec::new(), + payloads: payloads + .iter() + .map(|p| (p.name.clone(), p.byte_length, p.digest.clone())) + .collect(), + }; + + let submission = crate::state::CaptureSubmission { + checkpoint_id: checkpoint_id.clone(), + boundary, + session_id: self.session_id.clone(), + epoch: self.epoch.clone(), + episode_id: self.episode_id.clone(), + compatibility_digest: compatibility.digest(), + manifest: manifest.to_json(), + payloads, + replaceable, + }; + // The writer takes its own ownership of every payload here. Only then are the + // workers' cached capture replies released. + let receiver = { + let writer = self.writer.as_ref().expect("checked"); + match writer.submit(reservation, submission).await { + Ok(receiver) => receiver, + Err(e) => return Err(self.fail_now(e, "capture")), + } + }; + self.publish_checkpoint_event("captured", checkpoint_id, boundary, None).await?; + for (worker, request_id) in acknowledge { + self.lifecycle_acks.push((worker, request_id)); + } + self.acknowledge_lifecycle().await?; + Ok(CaptureTicket { + checkpoint_id: checkpoint_id.clone(), + boundary, + receiver, + }) + } + + /// Checks one participant's capture and turns it into a payload the writer can own. + fn accept_capture( + &mut self, + reply: &DomainReply, + result: &CaptureResult, + checkpoint_id: &Id, + boundary: u64, + expected_compatibility: &Digest, + name: &str, + ) -> Outcome { + if result.checkpoint_id != *checkpoint_id { + return Err(self.fail_now( + DomainError::before( + ErrorCode::IdentityMismatch, + "a capture names another checkpoint", + ), + "capture", + )); + } + if result.boundary != boundary { + return Err(self.fail_now( + DomainError::before( + ErrorCode::IdentityMismatch, + "a capture names another boundary; one checkpoint is one boundary", + ), + "capture", + )); + } + if result.compatibility_digest != *expected_compatibility { + return Err(self.fail_now( + DomainError::before( + ErrorCode::IncompatibleState, + "a capture reports a compatibility identity the composition does not hold", + ), + "capture", + )); + } + let digest = match &result.payload.digest { + Some(digest) => digest.clone(), + None => { + return Err(self.fail_now( + DomainError::before( + ErrorCode::BufferInvalid, + "a checkpoint payload must carry a content digest", + ), + "capture", + )); + } + }; + let artifact = match reply.artifacts.get(crate::state::PAYLOAD_ATTACHMENT) { + Some(artifact) if artifact.reference() == &result.payload => artifact.clone(), + _ => { + return Err(self.fail_now( + DomainError::before( + ErrorCode::BufferInvalid, + "a capture arrived without a live owned handle on its payload", + ), + "capture", + )); + } + }; + Ok(crate::state::CapturedPayload { + name: name.to_owned(), + byte_length: result.payload.byte_length, + digest, + artifact, + }) + } + + /// Waits for one capture's durable outcome and moves the durable mark only on a commit. + /// + /// A lost reply is an outcome here, not a hang and not a save: the high-water mark stays + /// where it was until [`Coordinator::resolve_durable`] asks the store about the same + /// operation. + pub async fn await_durable(&mut self, ticket: CaptureTicket) -> Outcome { + let CaptureTicket { checkpoint_id, boundary, receiver } = ticket; + let budget = self.deadlines.capture; + let outcome = + crate::state::CheckpointWriter::wait(receiver, &checkpoint_id, budget).await; + if let crate::state::SaveOutcome::Committed { .. } = &outcome { + self.durable = Some((checkpoint_id.clone(), boundary)); + self.audit.push(format!("durable:{checkpoint_id}@{boundary}")); + } else { + self.audit.push(format!("not-durable:{checkpoint_id}@{boundary}")); + } + Ok(outcome) + } + + /// Resolves a save whose reply was lost, by asking the store's durable metadata about the + /// *same* checkpoint. It never saves again. + /// + /// `Some(boundary)` means the store manifest lists that generation, which is the durable + /// commit point; `None` means it does not, and an unreferenced generation file stays + /// unreferenced. + pub async fn resolve_durable(&mut self, checkpoint_id: &Id) -> Outcome> { + let Some(writer) = self.writer.as_ref() else { + return Err(self.fail_now( + DomainError::before( + ErrorCode::Unsupported, + "this session has no checkpoint store attached", + ), + "resolve-durable", + )); + }; + let wanted = checkpoint_id.clone(); + let found = writer + .with_store(move |store| store.lookup(&wanted).map(|g| g.boundary)) + .await; + match found { + Some(boundary) => { + self.durable = Some((checkpoint_id.clone(), boundary)); + self.audit.push(format!("durable:{checkpoint_id}@{boundary}")); + Ok(Some(boundary)) + } + None => { + self.audit.push(format!("not-durable:{checkpoint_id}")); + Ok(None) + } + } + } + + /// Captures and waits for the durable outcome, which is what an ordinary caller wants. + pub async fn checkpoint(&mut self, checkpoint_id: &Id) -> Outcome { + let ticket = self.capture(checkpoint_id, false).await?; + self.await_durable(ticket).await + } + + async fn publish_checkpoint_event( + &mut self, + event: &str, + checkpoint_id: &Id, + boundary: u64, + detail: Option<&str>, + ) -> Outcome<()> { + let payload = json!({ + "event": event, + "sessionId": self.session_id.as_str(), + "epoch": self.epoch.as_str(), + "checkpointId": checkpoint_id.as_str(), + "boundary": boundary.to_string(), + "detail": detail.map_or(Value::Null, |d| Value::String(d.to_owned())), + }); + let topic = self.topics.checkpoints.clone(); + self.publish(&topic, object(payload), Vec::new()).await + } + + // --------------------------------------------------------------------------------------- + // Recovery + + /// Points the coordinator at a replacement participant while the epoch is fenced. + /// + /// A replacement is the only way a fenced participant comes back: `step-v1` section 7's + /// incarnation row says every live participant of a failed epoch belongs to an invalid + /// one, so the reference is exchanged deliberately here and never repaired in place. + pub fn replace_participant(&mut self, worker_id: &Id, worker: WorkerRef) -> Outcome<()> { + if !self.fenced { + return Err(self.fail_now( + DomainError::before( + ErrorCode::InvalidPhase, + "participants are replaced while the epoch is fenced, not during play", + ), + "replace", + )); + } + if worker.worker_id != *worker_id { + return Err(self.fail_now( + DomainError::before( + ErrorCode::IdentityMismatch, + "the replacement reference names another worker", + ), + "replace", + )); + } + if self.environment.worker_id == *worker_id { + self.environment = worker; + self.tainted.remove(worker_id); + self.audit.push(format!("replaced:{worker_id}")); + return Ok(()); + } + match self.agents.iter_mut().find(|slot| slot.agent_id == *worker_id) { + Some(slot) => { + slot.worker = worker; + self.tainted.remove(worker_id); + self.audit.push(format!("replaced:{worker_id}")); + Ok(()) + } + None => Err(self.fail_now( + DomainError::before( + ErrorCode::IdentityMismatch, + format!("{worker_id} is not a participant of this composition"), + ), + "replace", + )), + } + } + + /// Installs a coherent checkpoint into a fresh epoch and lifts the fence. + /// + /// The whole of `state-media-v1` section 6 in order: select a complete compatible durable + /// checkpoint, import its payloads as *new* artifacts, stage every participant, activate + /// every participant, install the coordinator's own staged state, verify identity and + /// boundary, flush the old media and parser state, and establish `Paused(k)`. A failure + /// at any point leaves the fence exactly where it was. + pub async fn restore( + &mut self, + checkpoint_id: Option<&Id>, + new_epoch: &Id, + ) -> Outcome { + if self.phases.phase() != Phase::Failed { + return Err(self.fail_now( + DomainError::before( + ErrorCode::InvalidPhase, + "a coherent restore starts from a failed epoch", + ), + "restore", + )); + } + if *new_epoch == self.epoch { + return Err(self.fail_now( + DomainError::before( + ErrorCode::StaleEpoch, + "a restore installs a fresh epoch, never the one that failed", + ), + "restore", + )); + } + if !self.tainted.is_empty() { + let who: Vec<&str> = self.tainted.iter().map(String::as_str).collect(); + return Err(self.fail_now( + DomainError::before( + ErrorCode::InvalidPhase, + format!( + "{} installed a restore that no group resumed and must be replaced first", + who.join(", ") + ), + ), + "restore", + )); + } + let Some(writer) = self.writer.as_ref() else { + return Err(self.fail_now( + DomainError::before( + ErrorCode::Unsupported, + "this session has no checkpoint store attached", + ), + "restore", + )); + }; + let wanted = checkpoint_id.cloned(); + let read = writer + .with_store(move |store| { + let record = store.select(wanted.as_ref())?; + let envelope = store.read(&record)?; + Ok::<_, DomainError>((record, envelope)) + }) + .await; + let (record, envelope) = match read { + Ok(pair) => pair, + Err(e) => return Err(self.fail_now(e, "restore")), + }; + let manifest = match crate::state::CheckpointManifest::from_json(&envelope.manifest) { + Ok(manifest) => manifest, + Err(e) => { + return Err(self.fail_now( + DomainError::before(ErrorCode::IncompatibleState, e), + "restore", + )); + } + }; + if let Err(e) = self.check_restore_identity(&manifest, &record) { + return Err(self.fail_now(e, "restore")); + } + let boundary = manifest.source_scope.step; + self.transition(Phase::Restoring(boundary))?; + let outcome = self.restore_group(&envelope, &manifest, new_epoch, boundary).await; + match outcome { + Ok(report) => Ok(report), + Err(e) => Err(e), + } + } + + /// Marks every participant of an abandoned group install as one that must be replaced. + /// + /// A participant that staged holds a replacement state nothing installed; one that + /// activated holds installed state no group resumed. Neither is a participant this + /// session may reuse, and `ipc-v1` section 6's last paragraph is explicit that v1 does + /// not silently reattach one. + fn taint_group(&mut self, staged: &[(Id, WorkerRef, Id)]) { + for (who, _, _) in staged { + self.tainted.insert(who.clone()); + } + } + + /// Every identity a restore checks before a single participant is asked to stage. + fn check_restore_identity( + &self, + manifest: &crate::state::CheckpointManifest, + record: &crate::state::GenerationRecord, + ) -> DomainResult<()> { + if manifest.source_scope.session_id != self.session_id { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "the checkpoint belongs to another session", + )); + } + if manifest.episode_id != self.episode_id { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "the checkpoint belongs to another episode", + )); + } + if manifest.scheduler_id != "lockstep-v1" { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + format!( + "the checkpoint was scheduled by {}, not lockstep-v1", + manifest.scheduler_id + ), + )); + } + if record.compatibility_digest != manifest.compatibility.digest() { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the store manifest and the envelope disagree about compatibility", + )); + } + let live = match &self.descriptor { + Some(descriptor) => crate::state::Compatibility::of(descriptor), + None => { + return Err(DomainError::before( + ErrorCode::InvalidPhase, + "the session has no environment descriptor to compare compatibility with", + )); + } + }; + // Names the identity that differs -- backend, content, patch, controller, parser or + // state format -- rather than one opaque digest mismatch. + manifest + .compatibility + .compare(&live) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e))?; + let live_ports: Vec<(Id, Id)> = self + .agents + .iter() + .map(|slot| (slot.port_id.clone(), slot.agent_id.clone())) + .collect(); + if manifest.port_map != live_ports { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "the checkpoint's port map is not this composition's", + )); + } + if manifest.environment.worker_id != self.environment.worker_id { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "the checkpoint's world is not this composition's environment", + )); + } + let recorded: Vec = manifest.agents.iter().map(|a| a.agent_id.clone()).collect(); + if recorded != self.agent_ids() { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "the checkpoint's agents are not this composition's", + )); + } + for (row, slot) in manifest.agents.iter().zip(&self.agents) { + if row.profile_digest != slot.profile.digest || row.seed != slot.seed { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + format!( + "{}'s checkpoint was taken under another profile or seed", + row.agent_id + ), + )); + } + } + Ok(()) + } + + #[allow(clippy::too_many_lines)] + async fn restore_group( + &mut self, + envelope: &fly_session_types::checkpoint::Envelope, + manifest: &crate::state::CheckpointManifest, + new_epoch: &Id, + boundary: u64, + ) -> Outcome { + let scope = scope_at(&self.session_id, new_epoch, boundary); + // Step 3: import the payloads as *new* artifacts. Nothing the fence dropped is asked + // to come back, and a router that restarted has none of the old roots anyway. + let mut imported: BTreeMap = BTreeMap::new(); + for (name, bytes) in &envelope.payloads { + let digest = digest_of_bytes(bytes); + let artifact = match crate::state::seal_payload(&self.bus, bytes, &digest).await { + Ok(artifact) => artifact, + Err(e) => return Err(self.fail_now(e, "restore")), + }; + imported.insert(name.clone(), (artifact, digest, bytes.len() as u64)); + } + let imported_ids: Vec = imported + .values() + .map(|(artifact, _, _)| artifact.reference().artifact_id.clone()) + .collect(); + + // Step 3, continued: stage every participant. A refusal anywhere leaves nothing + // staged that will ever be activated, because the whole install is abandoned. + let mut staged: Vec<(Id, WorkerRef, Id)> = Vec::new(); + let mut order: Vec<(Id, WorkerRef, String, Digest)> = Vec::new(); + order.push(( + self.environment.worker_id.clone(), + self.environment.clone(), + manifest.environment.payload.clone(), + manifest.compatibility.digest(), + )); + for index in 0..self.agents.len() { + let slot = &self.agents[index]; + let row = manifest + .agents + .iter() + .find(|row| row.agent_id == slot.agent_id) + .expect("the agent set was checked"); + order.push(( + slot.agent_id.clone(), + slot.worker.clone(), + row.payload.clone(), + crate::agent::agent_compatibility_digest( + &row.agent_id, + &row.profile_digest, + &row.dataset_digest, + &row.model_version, + &row.plasticity_version, + row.seed, + ), + )); + } + for (who, worker, payload_name, compatibility_digest) in &order { + let Some((artifact, _, _)) = imported.get(payload_name) else { + return Err(self.fail_now( + DomainError::before( + ErrorCode::IncompatibleState, + format!("the checkpoint has no payload {payload_name} for {who}"), + ), + "stage-restore", + )); + }; + let params = StageRestoreParams { + checkpoint_id: manifest.checkpoint_id.clone(), + source_scope: manifest.source_scope.clone(), + compatibility_digest: compatibility_digest.clone(), + payload: artifact.reference().clone(), + }; + let attachments = [(crate::state::PAYLOAD_ATTACHMENT, artifact)]; + let reply = match self + .call( + worker, + "State.StageRestore", + Some(scope.clone()), + object(params.to_json()), + &attachments, + &[], + ) + .await + { + Ok(reply) => reply, + Err(failure) => { + // Whoever already staged is holding a replacement state this group will + // never install. Nothing is resumed, and none of them is reused. + for (done, _, _) in &staged { + self.tainted.insert(done.clone()); + } + return Err(failure); + } + }; + let result: StageRestoreResult = match reply.parse() { + Ok(result) => result, + Err(e) => { + for (done, _, _) in &staged { + self.tainted.insert(done.clone()); + } + return Err(self.fail_now(e, "stage-restore")); + } + }; + if result.checkpoint_id != manifest.checkpoint_id { + for (done, _, _) in &staged { + self.tainted.insert(done.clone()); + } + return Err(self.fail_now( + DomainError::before( + ErrorCode::IdentityMismatch, + "a staged restore names another checkpoint", + ), + "stage-restore", + )); + } + self.lifecycle_acks.push((worker.clone(), reply.request_id.clone())); + self.audit.push(format!("staged:{who}")); + staged.push((who.clone(), worker.clone(), result.restore_token)); + } + + // `state-media-v1` section 5: activation happens "after every participant **and + // coordinator** state validates". The coordinator's own staged ledgers are checked + // here, before a single token is activated, so a checkpoint whose task ledger or + // executor state is unreadable installs nothing anywhere. + let payloads: BTreeMap> = envelope + .payloads + .iter() + .map(|(name, bytes)| (name.clone(), bytes.clone())) + .collect(); + let descriptor = match &self.descriptor { + Some(descriptor) => descriptor.clone(), + None => { + self.taint_group(&staged); + return Err(self.fail_now( + DomainError::before( + ErrorCode::InvalidPhase, + "the session has no environment descriptor to restore against", + ), + "restore", + )); + } + }; + if let Err(e) = Coordinator::validate_coordinator_state( + manifest, + &payloads, + &descriptor, + self.agents.len(), + &*self.task, + &self.executors, + ) { + self.taint_group(&staged); + return Err(self.fail_now(e, "restore")); + } + + // Step 3, last part: activate. Anything that activates before a failure holds state + // no group resumed, so it is recorded as tainted and must be replaced. + let mut activated: Vec = Vec::new(); + let mut restored_observation: Option<(WorldObservation, BTreeMap)> = + None; + for (who, worker, token) in &staged { + let params = ActivateRestoreParams { restore_token: token.clone() }; + let want = if *who == self.environment.worker_id { + self.media_names.clone() + } else { + Vec::new() + }; + let reply = match self + .call( + worker, + "State.ActivateRestore", + Some(scope.clone()), + object(params.to_json()), + &[], + &want, + ) + .await + { + Ok(reply) => reply, + Err(failure) => { + self.taint_group(&staged); + return Err(failure); + } + }; + let result: ActivateRestoreResult = match reply.parse() { + Ok(result) => result, + Err(e) => { + self.taint_group(&staged); + return Err(self.fail_now(e, "activate-restore")); + } + }; + let role = if *who == self.environment.worker_id { + Role::Environment + } else { + Role::Agent + }; + if let Err(e) = result.validate_for_role(role) { + self.taint_group(&staged); + return Err(self.fail_now(DomainError::invalid(e.0), "activate-restore")); + } + if result.committed_step != boundary || result.checkpoint_id != manifest.checkpoint_id { + self.taint_group(&staged); + return Err(self.fail_now( + DomainError::before( + ErrorCode::IdentityMismatch, + "an activation names another boundary or checkpoint", + ), + "activate-restore", + )); + } + if let Some(observation) = result.observation { + let (views, audio) = media::split_attachments(reply.artifacts); + if !audio.is_empty() { + self.taint_group(&staged); + return Err(self.fail_now( + DomainError::new( + ErrorCode::BufferInvalid, + "a restored observation carried an audio chunk; no interval was played", + MutationCertainty::Unknown, + ), + "activate-restore", + )); + } + restored_observation = Some((observation, views)); + } + self.lifecycle_acks.push((worker.clone(), reply.request_id.clone())); + self.audit.push(format!("activated:{who}")); + activated.push(who.clone()); + } + + // Step 4: verify identity and boundary, and flush the old media and parser state. + let Some((observation, views)) = restored_observation else { + self.taint_group(&staged); + return Err(self.fail_now( + DomainError::before( + ErrorCode::IncompatibleState, + "no participant returned the restored world observation", + ), + "activate-restore", + )); + }; + let install = self.install_restored( + manifest, + new_epoch, + boundary, + &descriptor, + observation, + views, + &payloads, + ); + if let Err(e) = install { + self.taint_group(&staged); + return Err(self.fail_now(e, "restore")); + } + + // Step 5: Paused(k), and only now is the fence lifted. + self.epoch = new_epoch.clone(); + self.transition(Phase::Paused(boundary))?; + self.fenced = false; + self.acknowledge_lifecycle().await?; + self.publish_checkpoint_event("restored", &manifest.checkpoint_id, boundary, None) + .await?; + self.publish_descriptor().await?; + self.publish_snapshot(boundary, &BTreeMap::new(), &[], &[]).await?; + self.audit.push(format!("restored:{}@{boundary}", manifest.checkpoint_id)); + Ok(RestoreReport { + checkpoint_id: manifest.checkpoint_id.clone(), + boundary, + epoch: new_epoch.clone(), + staged: staged.iter().map(|(who, _, _)| who.clone()).collect(), + tokens: staged + .iter() + .map(|(who, _, token)| (who.clone(), token.clone())) + .collect(), + activated, + imported: imported_ids, + }) + } + + /// Reads one of the checkpoint's payloads as JSON, or says which one is unreadable. + fn read_payload(payloads: &BTreeMap>, name: &str) -> DomainResult { + let bytes = payloads.get(name).ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + format!("the checkpoint has no payload {name}"), + ) + })?; + serde_json::from_slice(bytes).map_err(|e| { + DomainError::before( + ErrorCode::IncompatibleState, + format!("payload {name} is not JSON: {e}"), + ) + }) + } + + /// Validates every coordinator-owned payload, changing nothing. + /// + /// This runs after the group has staged and before anything activates, which is the order + /// `state-media-v1` section 5 sets. It is a separate pass from the install below on + /// purpose: a group that cannot be resumed coherently must not have resumed some of it. + fn validate_coordinator_state( + manifest: &crate::state::CheckpointManifest, + payloads: &BTreeMap>, + descriptor: &EnvironmentDescriptor, + agents: usize, + task: &dyn crate::task::Task, + executors: &BTreeMap>, + ) -> DomainResult<()> { + let ledger = TypedValue::from_json(&Coordinator::read_payload( + payloads, + &manifest.coordinator.task_ledger, + )?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + task.validate_restore(&ledger)?; + for (agent_id, name) in &manifest.coordinator.executor_state { + let state = TypedValue::from_json(&Coordinator::read_payload(payloads, name)?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + let executor = executors.get(agent_id).ok_or_else(|| { + DomainError::before( + ErrorCode::IdentityMismatch, + format!("{agent_id} has no configured action executor"), + ) + })?; + executor.validate_restore(&state)?; + } + let inspection = TypedValue::from_json(&Coordinator::read_payload( + payloads, + &manifest.coordinator.prior_inspection, + )?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + if inspection.schema != descriptor.inspection_schema { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the recorded prior inspection is not the declared inspection schema", + )); + } + let record = + Coordinator::read_payload(payloads, &manifest.coordinator.admission_state)?; + let recorded = record + .get("agents") + .and_then(Value::as_array) + .ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the coordinator record has no agent list", + ) + })?; + if recorded.len() != agents { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the coordinator record names another number of agents", + )); + } + if record.get("audioPositions").and_then(Value::as_object).is_none() { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the coordinator record has no audio positions", + )); + } + Ok(()) + } + + /// Installs the coordinator's own staged state, after every participant activated. + #[allow(clippy::too_many_arguments)] + fn install_restored( + &mut self, + manifest: &crate::state::CheckpointManifest, + new_epoch: &Id, + boundary: u64, + descriptor: &EnvironmentDescriptor, + observation: WorldObservation, + views: BTreeMap, + payloads: &BTreeMap>, + ) -> DomainResult<()> { + if observation.boundary != boundary { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "the restored observation is not at the restored boundary", + )); + } + if observation.world_time != manifest.world_time { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the restored observation's world time is not the checkpoint's", + )); + } + observation + .validate_against(descriptor) + .map_err(|e| DomainError::new(ErrorCode::BufferInvalid, e, MutationCertainty::Unknown))?; + media::check_required_views(descriptor, &observation)?; + // An installed observation ran no transition, so it carries no chunk. Old epoch audio + // arriving as current is exactly what this refuses. + media::check_required_audio(descriptor, &observation, media::ObservationOrigin::Installed)?; + for view in &observation.sensory_views { + let name = media::view_attachment(&view.view_id); + match views.get(&name) { + Some(artifact) if artifact.reference() == &view.pixels => {} + _ => { + return Err(DomainError::new( + ErrorCode::BufferInvalid, + format!( + "the restored view {} arrived without a live owned handle", + view.view_id + ), + MutationCertainty::Unknown, + )); + } + } + } + + let read = |name: &str| Coordinator::read_payload(payloads, name); + + let ledger = TypedValue::from_json(&read(&manifest.coordinator.task_ledger)?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + self.task.validate_restore(&ledger)?; + for (agent_id, name) in &manifest.coordinator.executor_state { + let state = TypedValue::from_json(&read(name)?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + let executor = self.executors.get(agent_id).ok_or_else(|| { + DomainError::before( + ErrorCode::IdentityMismatch, + format!("{agent_id} has no configured action executor"), + ) + })?; + executor.validate_restore(&state)?; + } + let record = read(&manifest.coordinator.admission_state)?; + let inspection = TypedValue::from_json(&read(&manifest.coordinator.prior_inspection)?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + if inspection.schema != descriptor.inspection_schema { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the recorded prior inspection is not the declared inspection schema", + )); + } + let agents = record + .get("agents") + .and_then(Value::as_array) + .ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the coordinator record has no agent list", + ) + })? + .clone(); + if agents.len() != self.agents.len() { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the coordinator record names another number of agents", + )); + } + let positions = record + .get("audioPositions") + .and_then(Value::as_object) + .ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the coordinator record has no audio positions", + ) + })?; + let mut audio_positions = BTreeMap::new(); + for (stream, value) in positions { + let sample: u64 = value + .as_str() + .ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "a recorded audio position is not a canonical U64", + ) + })? + .parse() + .map_err(|_| { + DomainError::before( + ErrorCode::IncompatibleState, + "a recorded audio position is not a canonical U64", + ) + })?; + audio_positions.insert(stream.clone(), sample); + } + // Everything above validated. From here the coordinator installs, in one pass. + self.task.install_restore(new_epoch, &ledger)?; + for (agent_id, name) in &manifest.coordinator.executor_state { + let state = TypedValue::from_json(&read(name)?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + let executor = self + .executors + .get_mut(agent_id) + .expect("checked immediately above"); + executor.install_restore(&state)?; + } + for value in &agents { + let agent_id = value.get("agentId").and_then(Value::as_str).ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "a coordinator agent record has no agentId", + ) + })?; + let slot = self + .agents + .iter_mut() + .find(|slot| slot.agent_id == agent_id) + .ok_or_else(|| { + DomainError::before( + ErrorCode::IdentityMismatch, + format!("the coordinator record names {agent_id}, which is not configured"), + ) + })?; + let context = TypedValue::from_json(value.get("context").ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "a coordinator agent record has no decision context", + ) + })?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + let committed: u64 = value + .get("committedStep") + .and_then(Value::as_str) + .ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "a coordinator agent record has no committedStep", + ) + })? + .parse() + .map_err(|_| { + DomainError::before( + ErrorCode::IncompatibleState, + "a recorded committed step is not a canonical U64", + ) + })?; + if committed != boundary { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "a coordinator agent record is at another boundary", + )); + } + let brain_ticks: u64 = value + .get("brainTicks") + .and_then(Value::as_str) + .ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "a coordinator agent record has no brainTicks", + ) + })? + .parse() + .map_err(|_| { + DomainError::before( + ErrorCode::IncompatibleState, + "a recorded tick count is not a canonical U64", + ) + })?; + let remainder = RationalNs::from_json(value.get("remainder").ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "a coordinator agent record has no remainder", + ) + })?) + .map_err(|e| DomainError::before(ErrorCode::IncompatibleState, e.0))?; + slot.context_digest = context.digest(); + slot.context = context; + slot.committed_step = committed; + slot.brain_ticks = brain_ticks; + slot.remainder = remainder; + slot.prepared = None; + slot.prepare_request = None; + } + // Old media and old parser state are replaced, never carried: the previous epoch's + // handles were dropped by the fence and the timelines start again at their preserved + // sample positions with a discontinuity. + self.views = views; + self.pending_views.clear(); + self.audio.clear(); + self.pending_audio.clear(); + self.timelines = AudioTimelines::restored(descriptor, &audio_positions)?; + self.observation = Some(observation); + self.episode = None; + self.last_advance_request = None; + self.last_commit_requests.clear(); + self.pacing = Some(Pacing::new(descriptor.step_duration)); + Ok(()) + } + + /// The epoch-derived identities of this session's behaviour trace, mapped onto `to_epoch`. + /// + /// `step-v1` section 8 compares behaviour across runs. A resumed run runs in a new epoch, + /// and scope, batch identity and every task event identity are derived from it, so a + /// comparison either accounts for that or compares nothing. This is what "accounting for + /// new epoch metadata" is: an explicit, total rewrite of the epoch-derived fields, which + /// fails rather than passing anything through it does not recognise. + pub fn rebase(&self, to_epoch: &Id) -> Outcome { + let events = self.task.rebase_ids(to_epoch).map_err(|e| SessionFailure { + error: e, + phase: self.phases.phase().label(), + detail: "rebase".to_owned(), + participant: None, + })?; + Ok(EpochRebase { + from: self.epoch.clone(), + to: to_epoch.clone(), + events, + }) + } +} diff --git a/services/flysim/crates/fly-session/src/environment.rs b/services/flysim/crates/fly-session/src/environment.rs index 8a2947b..9c1bd85 100644 --- a/services/flysim/crates/fly-session/src/environment.rs +++ b/services/flysim/crates/fly-session/src/environment.rs @@ -13,6 +13,8 @@ use std::collections::BTreeSet; +use serde_json::Value; + use crate::media::{self, AudioSource, RenderCounter, ViewPipeline}; use crate::task::{controller_schema_ref, inspection, inspection_schema}; // `crate::types` is this crate's facade over the shared `fly-session-types` crate; the @@ -48,6 +50,12 @@ pub struct EnvironmentFaults { pub omit_audio_at_boundary: Option, /// Emit an audio chunk that starts before the previous chunk ended. pub overlapping_audio_at_boundary: Option, + /// Refuse `State.StageRestore`, so a group install meets a participant that will not + /// validate. + pub fail_stage_restore: bool, + /// Refuse `State.ActivateRestore` after staging, so a group meets a failure halfway + /// through activation. + pub fail_activate_restore: bool, } #[derive(Clone, Debug)] @@ -86,6 +94,10 @@ pub struct CounterEnvironment { audio: Option, /// The frame served at the previous boundary, kept only so a fault can serve it again. previous_view: Option<(ViewRef, flybus::Artifact)>, + /// A validated replacement world the live session cannot see yet. + staged: Option, + /// Restore tokens this world has activated. A token activates once. + activated: BTreeSet, } impl CounterEnvironment { @@ -104,10 +116,17 @@ impl CounterEnvironment { pipeline: None, audio: None, previous_view: None, + staged: None, + activated: BTreeSet::new(), config, } } + /// True while a validated replacement world is staged and not yet activated. + pub fn has_staged_restore(&self) -> bool { + self.staged.is_some() + } + pub fn status(&self) -> StatusCell { self.status.clone() } @@ -477,7 +496,7 @@ impl WorkerEndpoint for CounterEnvironment { vec![ id("world-step-v1"), id("pixel-observation-v1"), - id("checkpoint-v1"), + id(crate::state::CHECKPOINT_CAPABILITY), ] } @@ -490,7 +509,13 @@ impl WorkerEndpoint for CounterEnvironment { } fn methods(&self) -> Vec<&'static str> { - vec!["Environment.Initialize", "Environment.Advance"] + vec![ + "Environment.Initialize", + "Environment.Advance", + "State.Capture", + "State.StageRestore", + "State.ActivateRestore", + ] } fn handle<'a>(&'a mut self, ctx: HandlerCtx<'a>) -> BoxFuture<'a, DomainResult> { @@ -498,6 +523,9 @@ impl WorkerEndpoint for CounterEnvironment { match ctx.method { "Environment.Initialize" => self.initialize(&ctx).await, "Environment.Advance" => self.advance(&ctx).await, + "State.Capture" => self.state_capture(&ctx).await, + "State.StageRestore" => self.state_stage_restore(&ctx).await, + "State.ActivateRestore" => self.state_activate_restore(&ctx).await, other => Err(DomainError::before( ErrorCode::Unsupported, format!("{other} is not an environment method"), @@ -516,3 +544,488 @@ pub fn synthetic_asset(asset_id: &str, body: &str) -> AssetRef { format: id("fly-config-v1"), } } + +// ------------------------------------------------------------------------------------------- +// STATE-01: capture and restore + +/// The version this payload layout is written and read under. +pub const WORLD_PAYLOAD_VERSION: u64 = 1; + +fn incompatible(message: impl std::fmt::Display) -> DomainError { + DomainError::before(ErrorCode::IncompatibleState, message) +} + +/// One staged restore, held outside the live world until it is activated. +struct StagedWorld { + token: Id, + checkpoint_id: Id, + scope: Scope, + episode_id: Id, + descriptor: EnvironmentDescriptor, + boundary: u64, + counter: i64, + world_time: RationalNs, + advances: u64, + frames: Vec<(u64, i64)>, + audio_next_sample: u64, + audio_phase: u64, + audio_accumulator: u128, + audio_denominator: u128, +} + +impl CounterEnvironment { + /// `State.Capture`: the world at its committed boundary, including its pending sensor + /// pipeline. + /// + /// The pipeline is recorded as reconstruction inputs -- the producing boundary and the + /// world counter of every retained frame -- and never as an artifact identity: a + /// transient artifact belongs to the router that is running now, and a checkpoint outlives + /// it. + async fn state_capture(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult { + let scope = ctx.scope()?.clone(); + let Some(descriptor) = self.descriptor.clone() else { + return Err(DomainError::before( + ErrorCode::InvalidPhase, + "this environment is uninitialized", + )); + }; + if scope.session_id != self.config.session_id { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "this environment belongs to another session", + )); + } + match &self.epoch { + Some(epoch) if *epoch == scope.epoch => {} + _ => { + return Err(DomainError::before( + ErrorCode::StaleEpoch, + "State.Capture names an epoch this environment has left", + )); + } + } + if scope.step != self.boundary { + return Err(DomainError::before( + if scope.step < self.boundary { + ErrorCode::StaleStep + } else { + ErrorCode::FutureStep + }, + "State.Capture must name the boundary the world is at", + )); + } + let params: CaptureParams = ctx.params()?; + let pipeline = self + .pipeline + .as_ref() + .ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no view pipeline"))?; + let audio = self + .audio + .as_ref() + .ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no audio source"))?; + let (accumulator, denominator) = audio.accumulator(); + let previous = self.status.state(); + self.status.set_state(WorkerState::Capturing); + let payload = serde_json::json!({ + "payloadVersion": WORLD_PAYLOAD_VERSION, + "kind": "world", + "workerId": self.config.worker_id.as_str(), + "checkpointId": params.checkpoint_id.as_str(), + "sourceScope": scope.to_json(), + "episodeId": self.episode_id.clone().expect("initialized").as_str(), + "committedStep": self.boundary.to_string(), + "counter": self.counter.to_string(), + "worldTime": self.world_time.to_json(), + "advances": self.advances.to_string(), + "descriptor": descriptor.to_json(), + "pipeline": { + // The declared delay's whole queue, oldest first. + "frames": pipeline + .retained() + .into_iter() + .map(|(boundary, counter)| serde_json::json!({ + "boundary": boundary.to_string(), + "counter": counter.to_string(), + })) + .collect::>(), + }, + "audio": { + "nextSample": audio.next_sample().to_string(), + "phase": audio.phase().to_string(), + "accumulator": accumulator.to_string(), + "denominator": denominator.to_string(), + "chunks": audio.chunks().to_string(), + }, + }); + let bytes = canonicalize(&payload) + .map_err(|e| DomainError::invalid(format!("State.Capture: {}", e.0)))? + .into_bytes(); + let digest = digest_of_bytes(&bytes); + let artifact = crate::state::seal_payload(ctx.client, &bytes, &digest).await?; + // A capture reads the world; it does not advance it. + self.status.set_state(previous); + let result = CaptureResult { + checkpoint_id: params.checkpoint_id, + boundary: self.boundary, + compatibility_digest: crate::state::Compatibility::of(&descriptor).digest(), + payload: artifact.reference().clone(), + }; + Ok(HandlerReply::with_artifacts( + object(result.to_json()), + vec![(crate::state::PAYLOAD_ATTACHMENT.to_owned(), artifact)], + )) + } + + /// `State.StageRestore`: validate a replacement world into a staging slot. + async fn state_stage_restore(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult { + let scope = ctx.scope()?.clone(); + if scope.session_id != self.config.session_id { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "this environment belongs to another session", + )); + } + if let Some(epoch) = &self.epoch + && *epoch == scope.epoch + { + return Err(DomainError::before( + ErrorCode::StaleEpoch, + "State.StageRestore proposes the epoch this environment is already running", + )); + } + if self.descriptor.is_some() { + // A world that is already running a boundary is not a quiescent replacement: the + // group replaces it rather than restoring over a live one. + return Err(DomainError::before( + ErrorCode::InvalidPhase, + "State.StageRestore needs an uninitialized replacement environment", + )); + } + let params: StageRestoreParams = ctx.params()?; + if params.source_scope.step != scope.step { + return Err(DomainError::invalid( + "State.StageRestore's scope step must be the source boundary", + )); + } + let artifact = ctx.artifact(crate::state::PAYLOAD_ATTACHMENT)?; + if artifact.reference() != ¶ms.payload { + return Err(DomainError::before( + ErrorCode::BufferInvalid, + "the staged payload attachment is not the artifact the request names", + )); + } + let bytes = artifact.read_all().await.map_err(|e| { + DomainError::before( + ErrorCode::BufferInvalid, + format!("the staged payload could not be read: {}", e.message), + ) + })?; + let declared = params + .payload + .digest + .clone() + .ok_or_else(|| incompatible("a checkpoint payload must carry a content digest"))?; + let actual = digest_of_bytes(&bytes); + if actual != declared || bytes.len() as u64 != params.payload.byte_length { + return Err(incompatible( + "the staged payload is not the content the request declares", + )); + } + let value: Value = serde_json::from_slice(&bytes) + .map_err(|e| DomainError::invalid(format!("the staged payload is not JSON: {e}")))?; + let text = |key: &str| -> DomainResult { + value + .get(key) + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| incompatible(format!("the world payload has no {key}"))) + }; + let number = |key: &str| -> DomainResult { + text(key)? + .parse::() + .map_err(|_| incompatible(format!("the world payload's {key} is not a U64"))) + }; + if value.get("payloadVersion").and_then(Value::as_u64) != Some(WORLD_PAYLOAD_VERSION) { + return Err(incompatible("the world payload is another payload version")); + } + if text("kind")? != "world" { + return Err(incompatible("this payload is not a world's state")); + } + if text("workerId")? != self.config.worker_id { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "the staged payload belongs to another world", + )); + } + if text("checkpointId")? != params.checkpoint_id { + return Err(incompatible("the staged payload belongs to another checkpoint")); + } + let source_scope = Scope::from_json( + value + .get("sourceScope") + .ok_or_else(|| incompatible("the world payload has no sourceScope"))?, + ) + .map_err(|e| incompatible(format!("the world payload's sourceScope: {}", e.0)))?; + if source_scope != params.source_scope { + return Err(incompatible( + "the staged payload was captured at another source scope", + )); + } + let committed_step = number("committedStep")?; + if committed_step != params.source_scope.step { + return Err(incompatible( + "the staged payload's committed step is not the source boundary", + )); + } + let descriptor = EnvironmentDescriptor::from_json( + value + .get("descriptor") + .ok_or_else(|| incompatible("the world payload has no descriptor"))?, + ) + .map_err(|e| incompatible(format!("the world payload's descriptor: {}", e.0)))?; + // The replacement builds the descriptor it would advertise and compares. A world + // started with other ports, another cadence or another declared render delay is a + // different backend, not this one resumed. + let live = self.build_descriptor()?; + if descriptor != live { + return Err(incompatible( + "the staged world was captured under another environment descriptor", + )); + } + let expected = crate::state::Compatibility::of(&descriptor).digest(); + if expected != params.compatibility_digest { + return Err(incompatible(format!( + "the staged world's compatibility {expected} is not the {} the restore requires", + params.compatibility_digest + ))); + } + let counter: i64 = text("counter")? + .parse() + .map_err(|_| incompatible("the world payload's counter is not an integer"))?; + let world_time = RationalNs::from_json( + value + .get("worldTime") + .ok_or_else(|| incompatible("the world payload has no worldTime"))?, + ) + .map_err(|e| incompatible(format!("the world payload's worldTime: {}", e.0)))?; + let pipeline_value = value + .get("pipeline") + .and_then(|p| p.get("frames")) + .and_then(Value::as_array) + .ok_or_else(|| incompatible("the world payload has no pipeline frames"))?; + let mut frames = Vec::with_capacity(pipeline_value.len()); + for frame in pipeline_value { + let boundary = frame + .get("boundary") + .and_then(Value::as_str) + .ok_or_else(|| incompatible("a captured frame has no boundary"))? + .parse::() + .map_err(|_| incompatible("a captured frame's boundary is not a U64"))?; + let frame_counter = frame + .get("counter") + .and_then(Value::as_str) + .ok_or_else(|| incompatible("a captured frame has no counter"))? + .parse::() + .map_err(|_| incompatible("a captured frame's counter is not an integer"))?; + frames.push((boundary, frame_counter)); + } + match frames.last() { + Some((boundary, _)) if *boundary == committed_step => {} + _ => { + return Err(incompatible( + "the captured pipeline does not end at the committed boundary", + )); + } + } + let audio_value = value + .get("audio") + .ok_or_else(|| incompatible("the world payload has no audio state"))?; + let audio_number = |key: &str| -> DomainResult { + audio_value + .get(key) + .and_then(Value::as_str) + .ok_or_else(|| incompatible(format!("the captured audio state has no {key}")))? + .parse::() + .map_err(|_| incompatible(format!("the captured audio {key} is not a number"))) + }; + let audio_next_sample = u64::try_from(audio_number("nextSample")?) + .map_err(|_| incompatible("the captured audio position is outside U64"))?; + let audio_phase = u64::try_from(audio_number("phase")?) + .map_err(|_| incompatible("the captured audio phase is outside U64"))?; + + if self.config.faults.fail_stage_restore { + return Err(incompatible( + "injected staging refusal: this participant's replacement state does not \ +validate", + )); + } + if let Some(staged) = &self.staged { + return Err(DomainError::before( + ErrorCode::Conflict, + format!( + "this environment already holds the staged restore {} for checkpoint {}", + staged.token, staged.checkpoint_id + ), + )); + } + let token = crate::agent::restore_token( + ¶ms.checkpoint_id, + &scope, + &actual, + &self.config.incarnation_id, + ); + if self.activated.contains(&token) { + return Err(DomainError::before( + ErrorCode::Conflict, + "this exact restore was already activated on this environment", + )); + } + self.staged = Some(StagedWorld { + token: token.clone(), + checkpoint_id: params.checkpoint_id.clone(), + scope, + episode_id: parse_id(&text("episodeId")?) + .map_err(|e| incompatible(format!("the world payload's episodeId {e}")))?, + descriptor, + boundary: committed_step, + counter, + world_time, + advances: number("advances")?, + frames, + audio_next_sample, + audio_phase, + audio_accumulator: audio_number("accumulator")?, + audio_denominator: audio_number("denominator")?, + }); + self.status.set_state(WorkerState::StagedRestore); + let result = StageRestoreResult { + checkpoint_id: params.checkpoint_id, + restore_token: token, + }; + Ok(HandlerReply::from(&result)) + } + + /// `State.ActivateRestore`: install the staged world and return its coherent observation. + /// + /// Nothing advances. The pipeline's frames are rendered again into fresh artifacts of the + /// current store, which is what "the durable store imports fresh immutable bus artifacts" + /// means on the producing side, and the observation carries no audio chunk because no + /// interval was played. + async fn state_activate_restore( + &mut self, + ctx: &HandlerCtx<'_>, + ) -> DomainResult { + let params: ActivateRestoreParams = ctx.params()?; + if self.activated.contains(¶ms.restore_token) { + return Err(DomainError::before( + ErrorCode::Conflict, + "this restore token has already been activated", + )); + } + let Some(staged) = self.staged.take() else { + return Err(DomainError::before( + ErrorCode::InvalidPhase, + "this environment holds no staged restore", + )); + }; + if staged.token != params.restore_token { + let token = staged.token.clone(); + self.staged = Some(staged); + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + format!( + "this environment's staged restore is {token}, not {}", + params.restore_token + ), + )); + } + if self.config.faults.fail_activate_restore { + let token = staged.token.clone(); + self.staged = Some(staged); + self.status.set_state(WorkerState::Failed); + return Err(DomainError::new( + ErrorCode::BackendFailure, + format!("injected activation failure; {token} stays staged and unresumed"), + MutationCertainty::None, + )); + } + self.status.set_state(WorkerState::Restoring); + let mut pipeline = ViewPipeline::new( + CounterEnvironment::view_descriptor(self.config.observation_delay_steps), + self.config.renders.clone(), + ); + pipeline.restore(ctx.client, &staged.frames).await?; + let audio = AudioSource::restored_from( + CounterEnvironment::audio_descriptor(), + staged.audio_next_sample, + staged.audio_phase, + staged.audio_accumulator, + staged.audio_denominator, + )?; + self.epoch = Some(staged.scope.epoch.clone()); + self.episode_id = Some(staged.episode_id.clone()); + self.descriptor = Some(staged.descriptor.clone()); + self.boundary = staged.boundary; + self.counter = staged.counter; + self.world_time = staged.world_time; + self.advances = staged.advances; + // Batch ids are unique within an epoch, and this is a new one. Keeping the old set + // would refuse nothing extra: a request under the old epoch is already refused by its + // scope. + self.batches.clear(); + self.pipeline = Some(pipeline); + self.audio = Some(audio); + self.previous_view = None; + self.activated.insert(staged.token); + self.status.set_state(WorkerState::Ready); + self.status.set_scope(Some(scope_at( + &staged.scope.session_id, + &staged.scope.epoch, + staged.boundary, + ))); + + let (observation, attachments) = self.restored_observation()?; + let result = ActivateRestoreResult { + committed_step: staged.boundary, + checkpoint_id: staged.checkpoint_id, + observation: Some(observation), + }; + result + .validate_for_role(Role::Environment) + .map_err(|e| DomainError::invalid(e.0))?; + let mut reply = HandlerReply::from(&result); + reply.artifacts = attachments; + Ok(reply) + } + + /// The observation the restored world is already at: no render, no advance, no audio. + fn restored_observation( + &mut self, + ) -> DomainResult<(WorldObservation, Vec<(String, flybus::Artifact)>)> { + let boundary = self.boundary; + let counter = self.counter; + let pipeline = self + .pipeline + .as_ref() + .ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no view pipeline"))?; + let (view, artifact) = pipeline.at(boundary).ok_or_else(|| { + incompatible("the restored pipeline holds no frame for the restored boundary") + })?; + self.previous_view = Some((view.clone(), artifact.clone())); + let observation = WorldObservation { + boundary, + world_time: self.world_time, + engine_frame: Some(boundary.to_string()), + sensory_views: vec![view.clone()], + inspection: inspection(counter, boundary), + broadcast_views: vec![view.clone()], + // No interval was played, so there is no chunk. A chunk here would be an old + // epoch's audio offered as current. + audio: Vec::new(), + }; + Ok(( + observation, + vec![(media::view_attachment(&view.view_id), artifact)], + )) + } +} diff --git a/services/flysim/crates/fly-session/src/harness.rs b/services/flysim/crates/fly-session/src/harness.rs index c537a9b..53159d1 100644 --- a/services/flysim/crates/fly-session/src/harness.rs +++ b/services/flysim/crates/fly-session/src/harness.rs @@ -26,6 +26,7 @@ use crate::media::{RenderCounter, SensorLog}; use crate::launcher::{ AgentLaunch, EnvironmentLaunch, Launcher, ReapOutcome, SUPERVISOR_CLIENT, ThreadBudget, }; +use crate::state::{CheckpointStore, CheckpointWriter, StoreConfig, StoreFaults, WriterConfig, WriterFaults}; use crate::task::{ActionExecutor, CounterTask, IdentityExecutor, Terminal}; // `crate::types` is this crate's facade over the shared `fly-session-types` crate; the // glob keeps the contract's own names in sight instead of restating them. @@ -85,6 +86,14 @@ pub struct HarnessConfig { /// The threads reserved for the coordinator, its router and its store. pub coordinator_threads: usize, pub environment_threads: usize, + /// How many committed generations the durable checkpoint store keeps. + pub store: StoreConfig, + /// The durable write faults this composition injects. + pub store_faults: StoreFaults, + /// The checkpoint queue's bounds. + pub writer: WriterConfig, + /// The writer faults this composition injects. + pub writer_faults: WriterFaults, } impl Default for HarnessConfig { @@ -107,6 +116,10 @@ impl Default for HarnessConfig { thread_budget: None, coordinator_threads: 1, environment_threads: 1, + store: StoreConfig::default(), + store_faults: StoreFaults::default(), + writer: WriterConfig::default(), + writer_faults: WriterFaults::default(), } } } @@ -134,6 +147,14 @@ const ENV_SERVICE: &str = "env.arena"; const ENV_CLIENT: &str = "environment"; const ENV_WORKER: &str = "arena"; const COORDINATOR_CLIENT: &str = "coordinator"; +/// The checkpoint writer's own bus identity. It publishes checkpoint events and nothing else. +const WRITER_CLIENT: &str = "checkpoint-writer"; + +/// How many times one participant may be replaced in a composition. +/// +/// Each replacement connects under its own client id, so a restart is visibly a new +/// participant rather than a silent reattachment, and the policy has to name them all. +const MAX_GENERATIONS: u32 = 8; fn agent_service(agent_id: &Id) -> String { format!("agent.{agent_id}") @@ -172,6 +193,10 @@ pub struct SessionHarness { /// The supervisor. It owns every participant's lifetime and thread allocation. pub launcher: Launcher, observers: Mutex>, + /// Which generation of each participant is running: 1 is the one the composition started. + generations: BTreeMap, + /// Where the durable checkpoint store lives, for a test that reads the files themselves. + checkpoint_root: std::path::PathBuf, } impl SessionHarness { @@ -204,12 +229,16 @@ impl SessionHarness { g.call = vec![Pattern::prefix("agent."), Pattern::prefix("env.")]; }), ) + // The writer publishes the checkpoint events and never calls a participant. + .client(WRITER_CLIENT, grants(|g| g.publish = vec![Pattern::prefix("session.")])) .client(ENV_CLIENT, grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)])) - .client( - &format!("{ENV_CLIENT}-r2"), - grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]), - ) .client("observer", grants(|g| g.subscribe = vec![Pattern::prefix("session.")])); + for generation in 2..=MAX_GENERATIONS { + policy = policy.client( + &format!("{ENV_CLIENT}-r{generation}"), + grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]), + ); + } for spec in &config.agents { let service = agent_service(&spec.agent_id); policy = policy.client( @@ -218,10 +247,12 @@ impl SessionHarness { ); // A replacement worker connects under its own client id, so a restart is visibly a // new participant rather than a silent reattachment to the active epoch. - policy = policy.client( - &format!("{}-r2", agent_client(&spec.agent_id)), - grants(|g| g.register = vec![Pattern::exact(&service)]), - ); + for generation in 2..=MAX_GENERATIONS { + policy = policy.client( + &format!("{}-r{generation}", agent_client(&spec.agent_id)), + grants(|g| g.register = vec![Pattern::exact(&service)]), + ); + } } let mut router_config = RouterConfig::new(&store_root); router_config.policy = policy; @@ -299,6 +330,21 @@ impl SessionHarness { } let coordinator_client = launcher.connect(COORDINATOR_CLIENT).await?; + // The durable store lives beside the router's artifact store and never inside it: a + // committed generation is outside the bus's ephemeral collection. + let checkpoint_root = root.join("checkpoints"); + let mut store = CheckpointStore::open(&checkpoint_root, config.store).map_err(refusal)?; + *store.faults_mut() = config.store_faults.clone(); + let writer_client = launcher.connect(WRITER_CLIENT).await?; + let writer = CheckpointWriter::start( + store, + config.writer, + config.writer_faults.clone(), + Some(( + writer_client, + format!("session.{}.checkpoints", config.session_id), + )), + ); let executors: BTreeMap> = config .agents .iter() @@ -316,6 +362,8 @@ impl SessionHarness { Box::new(CounterTask::new(&config.epoch, config.terminal)), executors, ); + let mut coordinator = coordinator; + coordinator.attach_store(writer); Ok(SessionHarness { coordinator, @@ -326,9 +374,16 @@ impl SessionHarness { sensors, launcher, observers: Mutex::new(Vec::new()), + generations: BTreeMap::new(), + checkpoint_root, }) } + /// Where the durable checkpoint store's generations and store manifest live. + pub fn checkpoint_root(&self) -> &std::path::Path { + &self.checkpoint_root + } + pub fn router(&self) -> &Router { self.launcher.router() } @@ -373,10 +428,11 @@ impl SessionHarness { .find(|spec| spec.agent_id == *agent_id) .expect("a configured agent") .clone(); + let generation = self.next_generation(agent_id)?; self.launcher.kill(agent_id).await; let tick_duration = millis(self.config.tick_ms).expect("a positive tick"); - let incarnation_id = - parse_id(&format!("{agent_id}-inc-2")).expect("an agent id plus a suffix is an Id"); + let incarnation_id = parse_id(&format!("{agent_id}-inc-{generation}")) + .expect("an agent id plus a suffix is an Id"); self.launcher .launch_agent(AgentLaunch { session_id: self.config.session_id.clone(), @@ -390,7 +446,7 @@ impl SessionHarness { // predecessor wrote, so a restore's sensory input is visible beside it. sensors: self.sensors.get(agent_id).cloned().unwrap_or_default(), faults: spec.faults.clone(), - client_id: format!("{}-r2", agent_client(agent_id)), + client_id: format!("{}-r{generation}", agent_client(agent_id)), service: agent_service(agent_id), }) .await @@ -403,6 +459,102 @@ impl SessionHarness { }) } + /// Replaces the environment with a fresh, uninitialized incarnation, as a restore needs. + pub async fn restart_environment(&mut self) -> Result { + let worker_id = id(ENV_WORKER); + let generation = self.next_generation(&worker_id)?; + self.launcher.kill(&worker_id).await; + let step_duration = hz(self.config.step_hz).expect("a positive cadence"); + let incarnation_id = parse_id(&format!("arena-inc-{generation}")) + .expect("a worker id plus a suffix is an Id"); + self.launcher + .launch_environment(EnvironmentLaunch { + session_id: self.config.session_id.clone(), + worker_id: worker_id.clone(), + incarnation_id: incarnation_id.clone(), + step_duration, + ports: self.config.agents.iter().map(|a| a.port_id.clone()).collect(), + worker_threads: self.config.environment_threads, + observation_delay_steps: self.config.observation_delay_steps, + renders: self.renders.clone(), + faults: self.config.environment_faults.clone(), + client_id: format!("{ENV_CLIENT}-r{generation}"), + service: ENV_SERVICE.to_owned(), + }) + .await + .map_err(refusal)?; + let worker = self.launcher.worker(&worker_id).expect("just launched"); + Ok(Restarted { + service: worker.identity.service.clone(), + service_incarnation: worker.service_incarnation.clone(), + incarnation_id, + }) + } + + fn next_generation(&mut self, worker_id: &Id) -> Result { + let slot = self.generations.entry(worker_id.clone()).or_insert(1); + if *slot >= MAX_GENERATIONS { + return Err(flybus::BusError::new( + flybus::ErrorCode::QuotaExceeded, + format!( + "{worker_id} has used all {MAX_GENERATIONS} configured client identities; a composition declares how many replacements it allows" + ), + )); + } + *slot += 1; + Ok(*slot) + } + + /// Replaces every participant and points the fenced coordinator at the replacements. + /// + /// This is what a recovery does before it restores: the old participants belong to an + /// invalid epoch, and the references the coordinator pinned are exchanged deliberately. + pub async fn replace_all_participants(&mut self) -> Result<(), flybus::BusError> { + let environment = self.environment_id(); + self.restart_environment().await?; + let worker = self + .launcher + .worker(&environment) + .expect("just launched") + .worker_ref(); + self.coordinator + .replace_participant(&environment, worker) + .map_err(|e| refusal(e.error))?; + for agent_id in self.config.agents.iter().map(|a| a.agent_id.clone()).collect::>() { + self.restart_agent(&agent_id).await?; + let worker = self + .launcher + .worker(&agent_id) + .expect("just launched") + .worker_ref(); + self.coordinator + .replace_participant(&agent_id, worker) + .map_err(|e| refusal(e.error))?; + } + Ok(()) + } + + /// Changes one agent's injected faults, so the replacement the next restart launches is + /// a participant without them. + /// + /// A fault is launch configuration, so clearing one is a relaunch and not a live change: + /// the worker running now keeps whatever it was started with. + pub fn set_agent_faults(&mut self, agent_id: &Id, faults: AgentFaults) { + if let Some(spec) = self + .config + .agents + .iter_mut() + .find(|spec| spec.agent_id == *agent_id) + { + spec.faults = faults; + } + } + + /// Changes the environment's injected faults, with the same relaunch rule. + pub fn set_environment_faults(&mut self, faults: EnvironmentFaults) { + self.config.environment_faults = faults; + } + /// Ends one participant without asking it, as a crash would. pub async fn kill(&mut self, worker_id: &Id) -> ReapOutcome { self.launcher.kill(worker_id).await @@ -466,7 +618,10 @@ impl SessionHarness { /// Reaps every participant and closes the router. pub async fn shutdown(self) { - let SessionHarness { coordinator, mut launcher, observers, .. } = self; + let SessionHarness { mut coordinator, mut launcher, observers, .. } = self; + // The writer task owns artifact handles and a blocking store. Leaving it running + // would leave both behind. + coordinator.shutdown_store().await; drop(coordinator); launcher.reap_all(&id("shutdown")).await; for observer in observers.into_inner().expect("not poisoned") { diff --git a/services/flysim/crates/fly-session/src/launcher.rs b/services/flysim/crates/fly-session/src/launcher.rs index c80551e..43d6705 100644 --- a/services/flysim/crates/fly-session/src/launcher.rs +++ b/services/flysim/crates/fly-session/src/launcher.rs @@ -1231,6 +1231,8 @@ pub(crate) mod flags { pub const PREPARE_DELAY_MS: &str = "prepare-delay-ms"; pub const COMMIT_DELAY_MS: &str = "commit-delay-ms"; pub const FAIL_COMMIT_AT_STEP: &str = "fail-commit-at-step"; + pub const FAIL_STAGE_RESTORE: &str = "fail-stage-restore"; + pub const FAIL_ACTIVATE_RESTORE: &str = "fail-activate-restore"; pub const WORKER: &str = "worker"; pub const PORTS: &str = "ports"; @@ -1271,6 +1273,8 @@ pub(crate) mod flags { PREPARE_DELAY_MS, COMMIT_DELAY_MS, FAIL_COMMIT_AT_STEP, + FAIL_STAGE_RESTORE, + FAIL_ACTIVATE_RESTORE, ]; /// What only the environment is given, media options included. pub const ENVIRONMENT_ONLY: &[&str] = &[ @@ -1285,6 +1289,8 @@ pub(crate) mod flags { TRUNCATED_VIEW_AT_BOUNDARY, OMIT_AUDIO_AT_BOUNDARY, OVERLAPPING_AUDIO_AT_BOUNDARY, + FAIL_STAGE_RESTORE, + FAIL_ACTIVATE_RESTORE, ]; /// What a measurement run or one of its row children is given. pub const MEASURE: &[&str] = &[MODE, AGENTS, STEPS, WARMUP_STEPS, WORKER_THREADS, MODES]; @@ -1327,6 +1333,11 @@ impl Started { arg(flags::WARMUP_TICKS, spec.warmup_ticks), arg(flags::PREPARE_DELAY_MS, spec.faults.prepare_delay_ms), arg(flags::COMMIT_DELAY_MS, spec.faults.commit_delay_ms), + arg(flags::FAIL_STAGE_RESTORE, u64::from(spec.faults.fail_stage_restore)), + arg( + flags::FAIL_ACTIVATE_RESTORE, + u64::from(spec.faults.fail_activate_restore), + ), ]; if let Some(step) = spec.faults.fail_commit_at_step { args.push(arg(flags::FAIL_COMMIT_AT_STEP, step)); @@ -1345,6 +1356,11 @@ impl Started { // The media options a world in another process needs to be exactly this // world. Its render counter and its agents' sensor logs stay there. arg(flags::OBSERVATION_DELAY_STEPS, spec.observation_delay_steps), + arg(flags::FAIL_STAGE_RESTORE, u64::from(spec.faults.fail_stage_restore)), + arg( + flags::FAIL_ACTIVATE_RESTORE, + u64::from(spec.faults.fail_activate_restore), + ), ]; for (flag, boundary) in [ (flags::OMIT_VIEW_AT_BOUNDARY, spec.faults.omit_view_at_boundary), @@ -1458,6 +1474,8 @@ mod flag_tests { truncated_view_at_boundary: Some(3), omit_audio_at_boundary: Some(4), overlapping_audio_at_boundary: Some(5), + fail_stage_restore: true, + fail_activate_restore: true, } } @@ -1491,6 +1509,8 @@ mod flag_tests { fail_commit_at_step: Some(2), prepare_delay_ms: 1, commit_delay_ms: 2, + fail_stage_restore: true, + fail_activate_restore: true, }, client_id: "worker-fly-a".to_owned(), service: "agent.fly-a".to_owned(), @@ -1535,6 +1555,8 @@ mod flag_tests { flags::TRUNCATED_VIEW_AT_BOUNDARY, flags::OMIT_AUDIO_AT_BOUNDARY, flags::OVERLAPPING_AUDIO_AT_BOUNDARY, + flags::FAIL_STAGE_RESTORE, + flags::FAIL_ACTIVATE_RESTORE, ] { assert!(written.contains(&format!("--{flag}")), "--{flag} is not written"); } diff --git a/services/flysim/crates/fly-session/src/lib.rs b/services/flysim/crates/fly-session/src/lib.rs index 15722b5..e6d66c0 100644 --- a/services/flysim/crates/fly-session/src/lib.rs +++ b/services/flysim/crates/fly-session/src/lib.rs @@ -34,6 +34,7 @@ pub mod media; pub mod metrics; pub mod phase; pub mod rpc; +pub mod state; pub mod task; pub mod worker; diff --git a/services/flysim/crates/fly-session/src/media.rs b/services/flysim/crates/fly-session/src/media.rs index 870ead9..1d6c138 100644 --- a/services/flysim/crates/fly-session/src/media.rs +++ b/services/flysim/crates/fly-session/src/media.rs @@ -133,7 +133,14 @@ pub fn arena_frame(descriptor: &ViewDescriptor, counter: i64, boundary: u64) -> /// cannot be served an arbitrary stale image. pub struct ViewPipeline { descriptor: ViewDescriptor, - frames: VecDeque<(u64, flybus::Artifact)>, + /// Each retained frame: its producing boundary, the world counter it was rendered from + /// and the owned handle on its immutable bytes. + /// + /// The counter is kept because it is the whole of the reconstruction input: a checkpoint + /// records `(boundary, counter)` per retained frame and a restore re-renders them into + /// fresh artifacts of the current store, rather than persisting a transient artifact + /// identity that cannot survive a router restart. + frames: VecDeque<(u64, i64, flybus::Artifact)>, renders: RenderCounter, } @@ -160,7 +167,7 @@ impl ViewPipeline { let bytes = arena_frame(&self.descriptor, counter, boundary); let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?; self.renders.bump(); - self.frames.push_back((boundary, artifact)); + self.frames.push_back((boundary, counter, artifact)); // Keep exactly the frames a declared delay can still require. while self.frames.len() > self.descriptor.observation_delay_steps as usize + 1 { self.frames.pop_front(); @@ -168,6 +175,50 @@ impl ViewPipeline { Ok(()) } + /// The reconstruction inputs of every retained frame, oldest first. + /// + /// This is what a checkpoint records for the pending sensor pipeline: the producing + /// boundary and the world counter, never an artifact identity. + pub fn retained(&self) -> Vec<(u64, i64)> { + self.frames + .iter() + .map(|(boundary, counter, _)| (*boundary, *counter)) + .collect() + } + + /// Rebuilds the pipeline from recorded reconstruction inputs, into fresh artifacts. + /// + /// Every frame is rendered again in the current store, so nothing a fence dropped is + /// expected to come back and no old artifact identity crosses the recovery. + pub async fn restore( + &mut self, + client: &flybus::Client, + frames: &[(u64, i64)], + ) -> DomainResult<()> { + if frames.len() > self.descriptor.observation_delay_steps as usize + 1 { + return Err(media_error(format!( + "a captured pipeline of {} frames does not fit a declared delay of {}", + frames.len(), + self.descriptor.observation_delay_steps + ))); + } + for window in frames.windows(2) { + if window[1].0 != window[0].0 + 1 { + return Err(media_error( + "a captured pipeline's producing boundaries are not consecutive", + )); + } + } + self.frames.clear(); + for (boundary, counter) in frames { + let bytes = arena_frame(&self.descriptor, *counter, *boundary); + let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?; + self.renders.bump(); + self.frames.push_back((*boundary, *counter, artifact)); + } + Ok(()) + } + /// Seals a frame of the wrong length, which is what a broken backend produces. The /// reference it returns describes the artifact honestly, so the shape check is the thing /// under test rather than a lie in the payload. @@ -181,7 +232,7 @@ impl ViewPipeline { bytes.truncate(bytes.len() - self.descriptor.row_stride as usize); let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?; self.renders.bump(); - self.frames.push_back((boundary, artifact)); + self.frames.push_back((boundary, counter, artifact)); while self.frames.len() > self.descriptor.observation_delay_steps as usize + 2 { self.frames.pop_front(); } @@ -198,8 +249,8 @@ impl ViewPipeline { pub fn frame_produced_at(&self, produced: u64) -> Option<(ViewRef, flybus::Artifact)> { self.frames .iter() - .find(|(step, _)| *step == produced) - .map(|(step, artifact)| { + .find(|(step, _, _)| *step == produced) + .map(|(step, _, artifact)| { ( ViewRef { view_id: self.descriptor.view_id.clone(), @@ -257,6 +308,52 @@ impl AudioSource { source } + /// The exact state a capture recorded: sample position, waveform phase and the + /// unconsumed fraction of a frame. + /// + /// Restoring the position alone would restart the waveform and round the remainder away, + /// which is a resample the restore rules refuse. The first chunk of the new epoch marks + /// the discontinuity the recovery established. + pub fn restored_from( + descriptor: AudioDescriptor, + next_sample: u64, + phase: u64, + accumulator: u128, + denominator: u128, + ) -> DomainResult { + if denominator == 0 { + return Err(DomainError::invalid( + "audio: a captured accumulator denominator of zero", + )); + } + if accumulator >= denominator { + return Err(DomainError::invalid( + "audio: a captured accumulator is not below one whole frame", + )); + } + if phase >= descriptor.sample_rate { + return Err(DomainError::invalid( + "audio: a captured phase is not below the sample rate", + )); + } + let mut source = AudioSource::new(descriptor, next_sample); + source.discontinuous = true; + source.phase = phase; + source.accumulator = accumulator; + source.denominator = denominator; + Ok(source) + } + + /// The waveform phase, for a capture. + pub fn phase(&self) -> u64 { + self.phase + } + + /// The unconsumed fraction of a frame and the denominator it is over, for a capture. + pub fn accumulator(&self) -> (u128, u128) { + (self.accumulator, self.denominator) + } + pub fn descriptor(&self) -> &AudioDescriptor { &self.descriptor } @@ -439,18 +536,47 @@ pub fn check_required_views( Ok(()) } -/// Every declared audio stream produces exactly one chunk per transition. +/// Where an observation came from. +/// +/// `state-media-v1` section 2 makes a chunk the audio of an *interval*, so whether an +/// observation must carry one is a question about its provenance and not about its boundary +/// number. MEDIA-01 wrote the rule as "boundary 0 carries no chunk", which is true of the one +/// observation that slice could produce without a transition and false of the other one: +/// `State.ActivateRestore` installs a coherent observation at boundary `k` without advancing +/// gameplay, and it covers no interval either. Naming the provenance is the fix; exempting +/// the restored observation from the validator instead would have left "must a chunk exist" +/// unanswered exactly where a stale chunk would do the most damage. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum ObservationOrigin { + /// The observation a completed transition produced. Its interval has audio. + Transition, + /// An observation established at a boundary without running a transition: + /// `Environment.Initialize`'s `O[0]` and `State.ActivateRestore`'s restored observation. + /// It covers no interval, so it carries no chunk and one in it is refused. + Installed, +} + +/// Every declared audio stream produces exactly one chunk per transition, and none at all in +/// an observation that is not one. /// /// The contract states the shape and the ordering of chunks, not whether one has to exist, so /// this is MEDIA-01's choice and it is deliberate: a session that tolerates a silently missing /// chunk cannot tell "this world produced no audio for this interval" from "the chunk was -/// lost", and the second is the case the retention rules care about. Boundary 0 has no -/// preceding interval and so carries no chunk. +/// lost", and the second is the case the retention rules care about. The mirror of that, which +/// STATE-01 needs, is that an installed observation carrying a chunk is a stale chunk being +/// offered as current, and is refused for the same reason. pub fn check_required_audio( descriptor: &EnvironmentDescriptor, observation: &WorldObservation, + origin: ObservationOrigin, ) -> DomainResult<()> { - if observation.boundary == 0 { + if origin == ObservationOrigin::Installed { + if let Some(chunk) = observation.audio.first() { + return Err(media_error(format!( + "audio stream {} produced a chunk for an observation that ran no transition", + chunk.stream_id + ))); + } return Ok(()); } for stream in &descriptor.audio { diff --git a/services/flysim/crates/fly-session/src/state.rs b/services/flysim/crates/fly-session/src/state.rs new file mode 100644 index 0000000..93ba77b --- /dev/null +++ b/services/flysim/crates/fly-session/src/state.rs @@ -0,0 +1,1494 @@ +//! STATE-01: the coherent all-participant checkpoint store over the `FLYSESS1` envelope. +//! +//! [`fly_session_types::checkpoint`] owns the byte layout. This module owns everything +//! `checkpoint-envelope-v1` section 7 defers to this slice: generations, rotation, the store +//! manifest and its durable commit point, the bounded capture queue, the compatibility +//! comparison and the group fence. +//! +//! ```text +//! State.Capture ──> every participant, at one committed boundary +//! ──> one envelope: manifest + one payload per participant and per +//! coordinator-owned ledger +//! writer ──> temp, fsync, rename, fsync dir, then the store manifest the same way +//! ^^^^ the store manifest rename is the durable commit point +//! State.StageRestore ──> validated into replacement state, once-only token +//! State.ActivateRestore ──> installed under the new epoch, without a tick +//! ``` +//! +//! Nothing here is best-effort. A saturated queue is a named `BUSY` refusal taken *before* a +//! capture is requested; a lost save reply is an explicit outcome that leaves durable +//! metadata where it was; an incompatible checkpoint names the field that differs; and an +//! unreferenced generation is never a restore candidate. +//! +//! `FLYSIM01` (`crates/flybrain-core/src/envelope.rs`, read by the legacy flysim store) is a +//! different format with a different magic and a different reader, and nothing here touches +//! it. + +use std::collections::VecDeque; +use std::io::Write as _; +use std::path::{Path, PathBuf}; +use std::sync::Arc; +use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; + +use serde_json::{Map, Value, json}; + +use fly_session_types::checkpoint::{self, Envelope}; + +// `crate::types` is this crate's facade over the shared `fly-session-types` crate; the glob +// keeps the contract's own names in sight instead of restating them. +use crate::types::*; + +/// The state-format identity this slice writes and reads. A checkpoint recorded under any +/// other one is refused by name rather than attempted. +pub const STATE_FORMAT_ID: &str = "flysess-1"; + +/// The worker capability `state-media-v1` section 5 makes the State methods conditional on. +pub const CHECKPOINT_CAPABILITY: &str = "checkpoint-v1"; + +/// The store manifest's own version. It is not the envelope version. +pub const STORE_MANIFEST_VERSION: u32 = 1; + +/// The file the store manifest is committed to. Its rename is the durable commit point. +pub const STORE_MANIFEST_FILE: &str = "manifest.json"; + +/// The content type a checkpoint payload travels under as a bus artifact. +pub const PAYLOAD_CONTENT_TYPE: &str = "application/x-fly-checkpoint-payload"; + +/// The attachment name a checkpoint payload travels under, in both directions. +pub const PAYLOAD_ATTACHMENT: &str = "payload"; + +/// Seals one checkpoint payload as an immutable artifact against its content digest. +/// +/// `state-media-v1` section 1 makes a digest mandatory on checkpoint payloads, so this seals +/// with one rather than computing it afterwards: a store that wrote the wrong bytes finds out +/// here and not at the next restore. +pub async fn seal_payload( + client: &flybus::Client, + bytes: &[u8], + digest: &Digest, +) -> DomainResult { + let mut writer = client + .artifacts() + .allocate(bytes.len() as u64, PAYLOAD_CONTENT_TYPE) + .await + .map_err(|e| store_error(format!("payload allocate: {}", e.message)))?; + writer + .write_all(bytes) + .map_err(|e| store_error(format!("payload write: {e}")))?; + let artifact = writer + .seal_with_digest(Some(digest.clone())) + .await + .map_err(|e| store_error(format!("payload seal: {}", e.message)))?; + match &artifact.reference().digest { + Some(sealed) if sealed == digest => Ok(artifact), + _ => Err(store_error("a sealed checkpoint payload has no matching content digest")), + } +} + +fn store_error(what: impl std::fmt::Display) -> DomainError { + DomainError::new(ErrorCode::BackendFailure, what, MutationCertainty::Unknown) +} + +fn incompatible(what: impl std::fmt::Display) -> DomainError { + DomainError::before(ErrorCode::IncompatibleState, what) +} + +// ---------------------------------------------------------------------------------------------- +// Payload names + +/// The payload name one agent's captured state is filed under. +pub fn agent_payload(agent_id: &str) -> String { + format!("agent-{agent_id}") +} + +/// The payload name one agent's action-executor state is filed under. +pub fn executor_payload(agent_id: &str) -> String { + format!("executor-{agent_id}") +} + +/// The environment's payload name. +pub const WORLD_PAYLOAD: &str = "world"; +/// The task ledger's payload name. +pub const TASK_LEDGER_PAYLOAD: &str = "task-ledger"; +/// The prior world inspection's payload name. +pub const PRIOR_INSPECTION_PAYLOAD: &str = "prior-inspection"; +/// The coordinator's admission state and event watermarks. +pub const ADMISSION_PAYLOAD: &str = "coordinator-admission"; + +// ---------------------------------------------------------------------------------------------- +// Compatibility + +/// The compatibility identities `state-media-v1` section 4 requires a checkpoint to record. +/// +/// Each one is a separate field on purpose: a restore that fails says which identity differs, +/// instead of reporting one opaque digest mismatch. The synthetic composition maps them onto +/// the identities it actually has: +/// +/// | Field | Where it comes from | +/// | --- | --- | +/// | `backend_digest` | the environment descriptor's backend identity | +/// | `content_digest` | the environment descriptor's content identity | +/// | `patch_digest` | the environment descriptor's resolved configuration identity | +/// | `controller_digest` | every declared port's controller schema, in descriptor order | +/// | `parser_digest` | the inspection schema and the task schema that reads it | +/// | `state_format_id` | [`STATE_FORMAT_ID`] | +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct Compatibility { + pub backend_digest: Digest, + pub content_digest: Digest, + pub patch_digest: Digest, + pub controller_digest: Digest, + pub parser_digest: Digest, + pub state_format_id: Id, +} + +impl Compatibility { + /// The compatibility of one world, from the descriptor it advertises. + /// + /// Every identity comes from the descriptor, which is what lets the environment compute + /// exactly this digest for its own capture while the coordinator computes it for the + /// composition. The task's own schema is deliberately not in here: the captured task + /// ledger carries its schema and refuses another one, so folding it in would put one + /// identity in two places. + pub fn of(descriptor: &EnvironmentDescriptor) -> Compatibility { + let controllers = Value::Array( + descriptor + .ports + .iter() + .map(|port| { + json!({ + "portId": port.port_id.as_str(), + "controls": port.controls.to_json(), + }) + }) + .collect(), + ); + let parser = json!({ + "inspectionSchema": descriptor.inspection_schema.to_json(), + }); + Compatibility { + backend_digest: descriptor.backend_digest.clone(), + content_digest: descriptor.content_digest.clone(), + patch_digest: descriptor.configuration_digest.clone(), + controller_digest: digest_of(&controllers) + .expect("a validated controller schema canonicalizes"), + parser_digest: digest_of(&parser).expect("a validated schema reference canonicalizes"), + state_format_id: id(STATE_FORMAT_ID), + } + } + + pub fn to_json(&self) -> Value { + json!({ + "backendDigest": self.backend_digest.as_str(), + "contentDigest": self.content_digest.as_str(), + "patchDigest": self.patch_digest.as_str(), + "controllerDigest": self.controller_digest.as_str(), + "parserDigest": self.parser_digest.as_str(), + "stateFormatId": self.state_format_id.as_str(), + }) + } + + pub fn from_json(value: &Value) -> Result { + let digest = |key: &str| -> Result { + let text = value + .get(key) + .and_then(Value::as_str) + .ok_or_else(|| format!("compatibility: {key} is missing or not a string"))?; + if !is_digest(text) { + return Err(format!("compatibility: {key} is not a digest")); + } + Ok(text.to_owned()) + }; + let state_format_id = value + .get("stateFormatId") + .and_then(Value::as_str) + .ok_or_else(|| "compatibility: stateFormatId is missing".to_owned())?; + Ok(Compatibility { + backend_digest: digest("backendDigest")?, + content_digest: digest("contentDigest")?, + patch_digest: digest("patchDigest")?, + controller_digest: digest("controllerDigest")?, + parser_digest: digest("parserDigest")?, + state_format_id: parse_id(state_format_id) + .map_err(|e| format!("compatibility: stateFormatId {e}"))?, + }) + } + + /// The single digest a participant echoes in its capture and its stage request. + pub fn digest(&self) -> Digest { + digest_of(&self.to_json()).expect("a compatibility block canonicalizes") + } + + /// Names the first identity that differs. There is no tolerance and no "close enough". + pub fn compare(&self, live: &Compatibility) -> Result<(), String> { + for (field, recorded, current) in [ + ("stateFormatId", &self.state_format_id, &live.state_format_id), + ("backend", &self.backend_digest, &live.backend_digest), + ("content", &self.content_digest, &live.content_digest), + ("patch", &self.patch_digest, &live.patch_digest), + ("controller", &self.controller_digest, &live.controller_digest), + ("parser", &self.parser_digest, &live.parser_digest), + ] { + if recorded != current { + return Err(format!( + "the checkpoint's {field} identity {recorded} is not this composition's {current}" + )); + } + } + Ok(()) + } +} + +// ---------------------------------------------------------------------------------------------- +// The store manifest + +/// One committed generation, as the store manifest lists it. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct GenerationRecord { + pub checkpoint_id: Id, + /// The generation's file name inside the store directory. + pub file: String, + pub session_id: Id, + pub epoch: Id, + pub episode_id: Id, + pub boundary: u64, + pub compatibility_digest: Digest, + /// The SHA-256 of the whole envelope file, so a store can compare without opening it. + pub envelope_digest: Digest, + pub byte_length: u64, +} + +impl GenerationRecord { + fn to_json(&self) -> Value { + json!({ + "checkpointId": self.checkpoint_id.as_str(), + "file": self.file.as_str(), + "sessionId": self.session_id.as_str(), + "epoch": self.epoch.as_str(), + "episodeId": self.episode_id.as_str(), + "boundary": self.boundary.to_string(), + "compatibilityDigest": self.compatibility_digest.as_str(), + "envelopeDigest": self.envelope_digest.as_str(), + "byteLength": self.byte_length.to_string(), + }) + } + + fn from_json(value: &Value) -> Result { + let text = |key: &str| -> Result { + value + .get(key) + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| format!("store manifest: a generation has no {key}")) + }; + let number = |key: &str| -> Result { + text(key)? + .parse::() + .map_err(|_| format!("store manifest: {key} is not a canonical U64")) + }; + Ok(GenerationRecord { + checkpoint_id: parse_id(&text("checkpointId")?)?, + file: text("file")?, + session_id: parse_id(&text("sessionId")?)?, + epoch: parse_id(&text("epoch")?)?, + episode_id: parse_id(&text("episodeId")?)?, + boundary: number("boundary")?, + compatibility_digest: text("compatibilityDigest")?, + envelope_digest: text("envelopeDigest")?, + byte_length: number("byteLength")?, + }) + } +} + +/// The store's durable metadata: which generations exist and how far durability has reached. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct StoreManifest { + pub generations: Vec, + /// The newest committed checkpoint, which is the high-water mark. `None` until the first + /// durable commit; that is "nothing has been committed", not a default. + pub high_water: Option, +} + +impl StoreManifest { + fn to_json(&self) -> Value { + json!({ + "storeManifestVersion": STORE_MANIFEST_VERSION, + "stateFormatId": STATE_FORMAT_ID, + "highWater": self.high_water.as_ref().map_or(Value::Null, |h| h.as_str().into()), + "generations": Value::Array(self.generations.iter().map(GenerationRecord::to_json).collect()), + }) + } + + fn from_json(value: &Value) -> Result { + let version = value + .get("storeManifestVersion") + .and_then(Value::as_u64) + .ok_or_else(|| "store manifest: no storeManifestVersion".to_owned())?; + if version != u64::from(STORE_MANIFEST_VERSION) { + return Err(format!("store manifest: unsupported version {version}")); + } + match value.get("stateFormatId").and_then(Value::as_str) { + Some(STATE_FORMAT_ID) => {} + Some(other) => { + return Err(format!("store manifest: state format {other} is not {STATE_FORMAT_ID}")); + } + None => return Err("store manifest: no stateFormatId".to_owned()), + } + let generations = value + .get("generations") + .and_then(Value::as_array) + .ok_or_else(|| "store manifest: generations must be an array".to_owned())? + .iter() + .map(GenerationRecord::from_json) + .collect::, _>>()?; + let high_water = match value.get("highWater") { + Some(Value::Null) | None => None, + Some(Value::String(s)) => Some(parse_id(s)?), + Some(_) => return Err("store manifest: highWater is neither null nor an Id".to_owned()), + }; + if let Some(mark) = &high_water + && !generations.iter().any(|g| g.checkpoint_id == *mark) + { + return Err("store manifest: the high-water mark names no listed generation".to_owned()); + } + Ok(StoreManifest { generations, high_water }) + } +} + +// ---------------------------------------------------------------------------------------------- +// The store + +/// How many committed generations the store keeps. +#[derive(Clone, Copy, Debug)] +pub struct StoreConfig { + /// Generations retained after a commit. The oldest are dropped, and only once the + /// manifest that no longer references them is itself committed. + pub keep_generations: usize, +} + +impl Default for StoreConfig { + fn default() -> StoreConfig { + StoreConfig { keep_generations: 3 } + } +} + +/// Deliberate durable-write faults, for the failure rows this slice has to demonstrate. +#[derive(Clone, Debug, Default)] +pub struct StoreFaults { + /// Stop after the generation file has been renamed and before the store manifest is + /// committed. The generation is then an unreferenced file, which is never a candidate. + pub stop_before_manifest_commit: bool, +} + +/// The durable checkpoint store: generation files, one store manifest and the commit order of +/// `checkpoint-envelope-v1` section 5. +pub struct CheckpointStore { + root: PathBuf, + config: StoreConfig, + manifest: StoreManifest, + faults: StoreFaults, + commits: u64, + /// Generations the committed manifest no longer references and whose files could not be + /// removed. + /// + /// Rotation happens after the durable commit point, so a file that will not unlink is a + /// leaked file and never a lost checkpoint. It is recorded rather than swallowed, because + /// a store that keeps failing to rotate is filling a disk quietly. + unrotated: Vec, +} + +impl CheckpointStore { + /// Opens or creates a store at `root`, reading whatever it already committed. + /// + /// A directory with no manifest is an empty store: nothing has been committed there yet. + /// A manifest that cannot be read is a failure, not an empty store. + pub fn open(root: impl Into, config: StoreConfig) -> DomainResult { + let root: PathBuf = root.into(); + std::fs::create_dir_all(&root) + .map_err(|e| store_error(format!("checkpoint store {}: {e}", root.display())))?; + let path = root.join(STORE_MANIFEST_FILE); + let manifest = match std::fs::read(&path) { + Ok(bytes) => { + let value: Value = serde_json::from_slice(&bytes) + .map_err(|e| store_error(format!("store manifest: {e}")))?; + StoreManifest::from_json(&value).map_err(store_error)? + } + Err(e) if e.kind() == std::io::ErrorKind::NotFound => StoreManifest::default(), + Err(e) => return Err(store_error(format!("store manifest: {e}"))), + }; + Ok(CheckpointStore { + root, + config, + manifest, + faults: StoreFaults::default(), + commits: 0, + unrotated: Vec::new(), + }) + } + + pub fn root(&self) -> &Path { + &self.root + } + + /// Re-reads the store manifest from disk. + /// + /// The store manifest is the durable metadata, and this session is not necessarily the + /// only thing that has ever written it: a previous run, a repair or an operator may have + /// committed or removed a generation. Reading it again is how a store finds that out, + /// rather than trusting a copy it happens to be holding. + pub fn reload(&mut self) -> DomainResult<()> { + let reopened = CheckpointStore::open(self.root.clone(), self.config)?; + self.manifest = reopened.manifest; + Ok(()) + } + + pub fn manifest(&self) -> &StoreManifest { + &self.manifest + } + + pub fn faults_mut(&mut self) -> &mut StoreFaults { + &mut self.faults + } + + /// How many durable commits this store has completed. + pub fn commits(&self) -> u64 { + self.commits + } + + /// Generations the committed manifest dropped whose files are still on disk. + pub fn unrotated(&self) -> &[String] { + &self.unrotated + } + + /// The committed generation with this checkpoint id, if the manifest lists it. + /// + /// This is the query a coordinator uses to resolve a save whose reply it never saw: it + /// asks the durable metadata about the *same* operation rather than saving again. + pub fn lookup(&self, checkpoint_id: &Id) -> Option<&GenerationRecord> { + self.manifest + .generations + .iter() + .find(|g| g.checkpoint_id == *checkpoint_id) + } + + /// The newest committed generation, or an explicit refusal when nothing is committed. + pub fn high_water(&self) -> Option<&GenerationRecord> { + let mark = self.manifest.high_water.as_ref()?; + self.lookup(mark) + } + + /// Selects a restore candidate: the named generation, or the high-water one. + pub fn select(&self, checkpoint_id: Option<&Id>) -> DomainResult { + match checkpoint_id { + Some(wanted) => self.lookup(wanted).cloned().ok_or_else(|| { + incompatible(format!( + "the store has no committed generation {wanted}; an unreferenced \ +temporary is never a restore candidate" + )) + }), + None => self.high_water().cloned().ok_or_else(|| { + incompatible("the store has committed no checkpoint to restore from") + }), + } + } + + /// Reads one committed generation back and validates the whole envelope. + pub fn read(&self, record: &GenerationRecord) -> DomainResult { + let path = self.root.join(&record.file); + let bytes = std::fs::read(&path) + .map_err(|e| store_error(format!("generation {}: {e}", record.file)))?; + if bytes.len() as u64 != record.byte_length { + return Err(incompatible(format!( + "generation {} is {} bytes; the store manifest records {}", + record.file, + bytes.len(), + record.byte_length + ))); + } + if digest_of_bytes(&bytes) != record.envelope_digest { + return Err(incompatible(format!( + "generation {} does not match the digest the store manifest records", + record.file + ))); + } + let envelope = checkpoint::decode(&bytes) + .map_err(|e| incompatible(format!("generation {}: {}", record.file, e.0)))?; + checkpoint::validate_manifest(&envelope) + .map_err(|e| incompatible(format!("generation {}: {}", record.file, e.0)))?; + Ok(envelope) + } + + /// The durable commit sequence of `checkpoint-envelope-v1` section 5, in that order. + /// + /// Blocking by construction: it fsyncs. The writer runs it off the session's runtime. + fn commit(&mut self, record: GenerationRecord, bytes: &[u8]) -> DomainResult<()> { + let temporary = self.root.join(format!("tmp-{}.flysess", record.checkpoint_id)); + let final_path = self.root.join(&record.file); + // 1. write the envelope to a temporary generation file, 2. fsync it + { + let mut file = std::fs::File::create(&temporary) + .map_err(|e| store_error(format!("generation temporary: {e}")))?; + file.write_all(bytes) + .map_err(|e| store_error(format!("generation temporary: {e}")))?; + file.sync_all() + .map_err(|e| store_error(format!("generation fsync: {e}")))?; + } + // 3. rename it to its final generation name, 4. fsync the store directory + std::fs::rename(&temporary, &final_path) + .map_err(|e| store_error(format!("generation rename: {e}")))?; + sync_dir(&self.root)?; + if self.faults.stop_before_manifest_commit { + // The generation file exists and nothing references it. It is not a restore + // candidate and the high-water mark has not moved. + return Err(store_error( + "injected failure after the generation was renamed and before the store \ +manifest was committed", + )); + } + // 5. write the store manifest to its own temporary, fsync, rename, fsync the directory. + let mut next = self.manifest.clone(); + next.generations.retain(|g| g.checkpoint_id != record.checkpoint_id); + next.generations.push(record.clone()); + next.high_water = Some(record.checkpoint_id.clone()); + let dropped = if next.generations.len() > self.config.keep_generations { + let excess = next.generations.len() - self.config.keep_generations; + next.generations.drain(..excess).collect::>() + } else { + Vec::new() + }; + let text = canonicalize(&next.to_json()) + .map_err(|e| store_error(format!("store manifest: {}", e.0)))?; + let manifest_temporary = self.root.join("tmp-manifest.json"); + { + let mut file = std::fs::File::create(&manifest_temporary) + .map_err(|e| store_error(format!("store manifest temporary: {e}")))?; + file.write_all(text.as_bytes()) + .map_err(|e| store_error(format!("store manifest temporary: {e}")))?; + file.sync_all() + .map_err(|e| store_error(format!("store manifest fsync: {e}")))?; + } + std::fs::rename(&manifest_temporary, self.root.join(STORE_MANIFEST_FILE)) + .map_err(|e| store_error(format!("store manifest rename: {e}")))?; + sync_dir(&self.root)?; + // Past the durable commit point. Rotation removes only files the committed manifest + // no longer references. + self.manifest = next; + self.commits += 1; + for old in dropped { + if std::fs::remove_file(self.root.join(&old.file)).is_err() { + self.unrotated.push(old.file); + } + } + Ok(()) + } +} + +fn sync_dir(path: &Path) -> DomainResult<()> { + let dir = std::fs::File::open(path) + .map_err(|e| store_error(format!("store directory {}: {e}", path.display())))?; + dir.sync_all() + .map_err(|e| store_error(format!("store directory fsync: {e}"))) +} + +// ---------------------------------------------------------------------------------------------- +// The checkpoint manifest this slice writes and reads + +/// One agent's row in a checkpoint manifest. +#[derive(Clone, Debug, PartialEq)] +pub struct AgentEntry { + pub agent_id: Id, + pub profile_digest: Digest, + pub dataset_digest: Digest, + pub model_version: String, + pub plasticity_version: String, + pub seed: i32, + pub brain_ticks: u64, + pub remainder: RationalNs, + pub payload: String, +} + +impl AgentEntry { + fn to_json(&self) -> Value { + json!({ + "agentId": self.agent_id.as_str(), + "profileDigest": self.profile_digest.as_str(), + "datasetDigest": self.dataset_digest.as_str(), + "modelVersion": self.model_version.as_str(), + "plasticityVersion": self.plasticity_version.as_str(), + "seed": self.seed, + "brainTicks": self.brain_ticks.to_string(), + "remainder": self.remainder.to_json(), + "payload": self.payload.as_str(), + }) + } + + fn from_json(value: &Value) -> Result { + let text = |key: &str| -> Result { + value + .get(key) + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| format!("checkpoint manifest: an agent row has no {key}")) + }; + let seed = value + .get("seed") + .and_then(Value::as_i64) + .ok_or_else(|| "checkpoint manifest: an agent row has no seed".to_owned())?; + let seed = i32::try_from(seed) + .map_err(|_| "checkpoint manifest: a seed is outside i32".to_owned())?; + let remainder = RationalNs::from_json( + value + .get("remainder") + .ok_or_else(|| "checkpoint manifest: an agent row has no remainder".to_owned())?, + ) + .map_err(|e| format!("checkpoint manifest: remainder: {}", e.0))?; + Ok(AgentEntry { + agent_id: parse_id(&text("agentId")?)?, + profile_digest: text("profileDigest")?, + dataset_digest: text("datasetDigest")?, + model_version: text("modelVersion")?, + plasticity_version: text("plasticityVersion")?, + seed, + brain_ticks: text("brainTicks")? + .parse() + .map_err(|_| "checkpoint manifest: brainTicks is not a canonical U64".to_owned())?, + remainder, + payload: text("payload")?, + }) + } +} + +/// The coordinator-owned state a checkpoint records, each as a payload name. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct CoordinatorEntry { + pub task_ledger: String, + pub prior_inspection: String, + pub executor_state: Vec<(Id, String)>, + pub admission_state: String, + pub event_watermarks: EventWatermarks, +} + +/// The event identity a resumed epoch continues from. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct EventWatermarks { + /// The highest source step any recorded event belongs to. + pub last_source_step: u64, + /// How many events the ledger has issued. + pub issued: u64, +} + +impl EventWatermarks { + fn to_json(&self) -> Value { + json!({ + "lastSourceStep": self.last_source_step.to_string(), + "issued": self.issued.to_string(), + }) + } + + fn from_json(value: &Value) -> Result { + let number = |key: &str| -> Result { + value + .get(key) + .and_then(Value::as_str) + .ok_or_else(|| format!("checkpoint manifest: eventWatermarks has no {key}"))? + .parse() + .map_err(|_| format!("checkpoint manifest: {key} is not a canonical U64")) + }; + Ok(EventWatermarks { + last_source_step: number("lastSourceStep")?, + issued: number("issued")?, + }) + } +} + +impl CoordinatorEntry { + fn to_json(&self) -> Value { + json!({ + "taskLedger": self.task_ledger.as_str(), + "priorInspection": self.prior_inspection.as_str(), + "executorState": Value::Array( + self.executor_state + .iter() + .map(|(agent_id, payload)| json!({ + "agentId": agent_id.as_str(), + "payload": payload.as_str(), + })) + .collect(), + ), + "admissionState": self.admission_state.as_str(), + "eventWatermarks": self.event_watermarks.to_json(), + }) + } + + fn from_json(value: &Value) -> Result { + let text = |key: &str| -> Result { + value + .get(key) + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| format!("checkpoint manifest: coordinator has no {key}")) + }; + let executors = value + .get("executorState") + .and_then(Value::as_array) + .ok_or_else(|| "checkpoint manifest: executorState must be an array".to_owned())?; + let mut executor_state = Vec::with_capacity(executors.len()); + for entry in executors { + let agent_id = entry + .get("agentId") + .and_then(Value::as_str) + .ok_or_else(|| "checkpoint manifest: an executor row has no agentId".to_owned())?; + let payload = entry + .get("payload") + .and_then(Value::as_str) + .ok_or_else(|| "checkpoint manifest: an executor row has no payload".to_owned())?; + executor_state.push((parse_id(agent_id)?, payload.to_owned())); + } + let watermarks = value + .get("eventWatermarks") + .ok_or_else(|| "checkpoint manifest: coordinator has no eventWatermarks".to_owned())?; + Ok(CoordinatorEntry { + task_ledger: text("taskLedger")?, + prior_inspection: text("priorInspection")?, + executor_state, + admission_state: text("admissionState")?, + event_watermarks: EventWatermarks::from_json(watermarks)?, + }) + } +} + +/// The environment's row: which worker the world belonged to and which payload holds it. +/// +/// `checkpoint-envelope-v1` section 3 named a holder for every payload except the world's; +/// the 2026-09-22 amendment to that section adds this one. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct EnvironmentEntry { + pub worker_id: Id, + pub payload: String, +} + +impl EnvironmentEntry { + fn to_json(&self) -> Value { + json!({ + "workerId": self.worker_id.as_str(), + "payload": self.payload.as_str(), + }) + } + + fn from_json(value: &Value) -> Result { + let text = |key: &str| -> Result { + value + .get(key) + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| format!("checkpoint manifest: environment has no {key}")) + }; + Ok(EnvironmentEntry { + worker_id: parse_id(&text("workerId")?)?, + payload: text("payload")?, + }) + } +} + +/// A complete checkpoint manifest, in the field names `checkpoint-envelope-v1` section 3 sets. +#[derive(Clone, Debug, PartialEq)] +pub struct CheckpointManifest { + pub checkpoint_id: Id, + pub source_scope: Scope, + pub episode_id: Id, + pub world_time: RationalNs, + pub scheduler_id: String, + pub composition_digest: Digest, + pub port_map: Vec<(Id, Id)>, + pub compatibility: Compatibility, + pub agents: Vec, + pub coordinator: CoordinatorEntry, + pub environment: EnvironmentEntry, + /// External-helper state required for exact resume, as payload names. The synthetic + /// composition has no external helper, so it records an empty list rather than omitting + /// the field: "no helper" is a statement, not a missing one. + pub helper_state: Vec, + pub payloads: Vec<(String, u64, Digest)>, +} + +impl CheckpointManifest { + pub fn to_json(&self) -> Value { + json!({ + "envelopeVersion": checkpoint::VERSION, + "checkpointId": self.checkpoint_id.as_str(), + "sourceScope": self.source_scope.to_json(), + "episodeId": self.episode_id.as_str(), + "worldTime": self.world_time.to_json(), + "schedulerId": self.scheduler_id.as_str(), + "compositionDigest": self.composition_digest.as_str(), + "portMap": Value::Array( + self.port_map + .iter() + .map(|(port_id, agent_id)| json!({ + "portId": port_id.as_str(), + "agentId": agent_id.as_str(), + })) + .collect(), + ), + "compatibility": self.compatibility.to_json(), + "agents": Value::Array(self.agents.iter().map(AgentEntry::to_json).collect()), + "coordinator": self.coordinator.to_json(), + "environment": self.environment.to_json(), + "helperState": Value::Array( + self.helper_state.iter().map(|n| Value::String(n.clone())).collect(), + ), + "payloads": Value::Array( + self.payloads + .iter() + .map(|(name, length, digest)| json!({ + "name": name.as_str(), + "byteLength": length.to_string(), + "digest": digest.as_str(), + })) + .collect(), + ), + }) + } + + /// Reads one back, refusing an incomplete manifest rather than filling anything in. + pub fn from_json(value: &Value) -> Result { + for field in checkpoint::REQUIRED_MANIFEST_FIELDS { + if value.get(*field).is_none() { + return Err(format!("checkpoint manifest: missing {field:?}")); + } + } + let text = |key: &str| -> Result { + value + .get(key) + .and_then(Value::as_str) + .map(str::to_owned) + .ok_or_else(|| format!("checkpoint manifest: {key} is missing or not a string")) + }; + let source_scope = Scope::from_json(&value["sourceScope"]) + .map_err(|e| format!("checkpoint manifest: sourceScope: {}", e.0))?; + let world_time = RationalNs::from_json(&value["worldTime"]) + .map_err(|e| format!("checkpoint manifest: worldTime: {}", e.0))?; + let mut port_map = Vec::new(); + for entry in value["portMap"] + .as_array() + .ok_or_else(|| "checkpoint manifest: portMap must be an array".to_owned())? + { + let port_id = entry + .get("portId") + .and_then(Value::as_str) + .ok_or_else(|| "checkpoint manifest: a port map row has no portId".to_owned())?; + let agent_id = entry + .get("agentId") + .and_then(Value::as_str) + .ok_or_else(|| "checkpoint manifest: a port map row has no agentId".to_owned())?; + port_map.push((parse_id(port_id)?, parse_id(agent_id)?)); + } + let agents = value["agents"] + .as_array() + .ok_or_else(|| "checkpoint manifest: agents must be an array".to_owned())? + .iter() + .map(AgentEntry::from_json) + .collect::, _>>()?; + let mut helper_state = Vec::new(); + for entry in value["helperState"] + .as_array() + .ok_or_else(|| "checkpoint manifest: helperState must be an array".to_owned())? + { + helper_state.push( + entry + .as_str() + .ok_or_else(|| "checkpoint manifest: a helper state entry is not a payload name".to_owned())? + .to_owned(), + ); + } + let mut payloads = Vec::new(); + for entry in value["payloads"] + .as_array() + .ok_or_else(|| "checkpoint manifest: payloads must be an array".to_owned())? + { + let name = entry + .get("name") + .and_then(Value::as_str) + .ok_or_else(|| "checkpoint manifest: a payload row has no name".to_owned())?; + let length: u64 = entry + .get("byteLength") + .and_then(Value::as_str) + .ok_or_else(|| "checkpoint manifest: a payload row has no byteLength".to_owned())? + .parse() + .map_err(|_| "checkpoint manifest: byteLength is not a canonical U64".to_owned())?; + let digest = entry + .get("digest") + .and_then(Value::as_str) + .ok_or_else(|| "checkpoint manifest: a payload row has no digest".to_owned())?; + payloads.push((name.to_owned(), length, digest.to_owned())); + } + Ok(CheckpointManifest { + checkpoint_id: parse_id(&text("checkpointId")?)?, + source_scope, + episode_id: parse_id(&text("episodeId")?)?, + world_time, + scheduler_id: text("schedulerId")?, + composition_digest: text("compositionDigest")?, + port_map, + compatibility: Compatibility::from_json(&value["compatibility"])?, + agents, + coordinator: CoordinatorEntry::from_json(&value["coordinator"])?, + environment: EnvironmentEntry::from_json( + value + .get("environment") + .ok_or_else(|| "checkpoint manifest: missing \"environment\"".to_owned())?, + )?, + helper_state, + payloads, + }) + } + + /// The payload name this manifest files one participant's state under. + pub fn payload_of(&self, worker_id: &Id) -> Option<&str> { + if self.environment.worker_id == *worker_id { + return Some(self.environment.payload.as_str()); + } + self.agents + .iter() + .find(|a| a.agent_id == *worker_id) + .map(|a| a.payload.as_str()) + } +} + +// ---------------------------------------------------------------------------------------------- +// The bounded writer + +/// One participant's captured payload, with the owned handle the writer keeps until the bytes +/// are committed or the job fails. +pub struct CapturedPayload { + pub name: String, + pub artifact: flybus::Artifact, + pub byte_length: u64, + pub digest: Digest, +} + +/// What a coordinator hands the writer once every participant has captured. +pub struct CaptureSubmission { + pub checkpoint_id: Id, + pub boundary: u64, + pub session_id: Id, + pub epoch: Id, + pub episode_id: Id, + pub compatibility_digest: Digest, + pub manifest: Value, + pub payloads: Vec, + /// A hot checkpoint may replace a queued hot checkpoint, releasing its holds. A durable + /// one never is: the retention table coalesces only queued replaceable captures. + pub replaceable: bool, +} + +impl CaptureSubmission { + fn byte_length(&self) -> u64 { + self.payloads.iter().map(|p| p.byte_length).sum() + } +} + +/// How one durable save ended. Every variant is a statement; none of them is a default. +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum SaveOutcome { + /// Past the store manifest rename. This is the only variant that is a saved + /// acknowledgment and the only one that moves a high-water mark. + Committed { checkpoint_id: Id, boundary: u64, file: String }, + /// The write failed and its owned captures were released under the retry policy. It + /// never reports false durability. + Failed { checkpoint_id: Id, reason: String }, + /// A later replaceable capture took this one's place in the queue before it was written. + Superseded { checkpoint_id: Id, by: Id }, + /// The writer's reply never arrived. The operation's outcome is unknown from here, so + /// durable metadata does not move; the caller resolves the *same* operation against the + /// store manifest instead of saving again. + ReplyLost { checkpoint_id: Id }, +} + +impl SaveOutcome { + pub fn checkpoint_id(&self) -> &Id { + match self { + SaveOutcome::Committed { checkpoint_id, .. } + | SaveOutcome::Failed { checkpoint_id, .. } + | SaveOutcome::Superseded { checkpoint_id, .. } + | SaveOutcome::ReplyLost { checkpoint_id } => checkpoint_id, + } + } + + /// The event name this outcome publishes under. + pub fn event(&self) -> &'static str { + match self { + SaveOutcome::Committed { .. } => "committed", + SaveOutcome::Failed { .. } => "failed", + SaveOutcome::Superseded { .. } => "superseded", + SaveOutcome::ReplyLost { .. } => "failed", + } + } +} + +/// What a failed write does with the ephemeral captures it owns. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum RetryPolicy { + /// Release the owned captures and report the failure. The default: a capture is cheap to + /// take again at the next committed boundary, and holding one is not free. + ReleaseAndReport, + /// Try the durable sequence again, up to `attempts` times in total, keeping the owned + /// captures until they are committed or the attempts are spent. + RetryThenRelease { attempts: u32 }, +} + +/// The writer's bounds. Both are finite and both refuse before a capture is requested. +#[derive(Clone, Copy, Debug)] +pub struct WriterConfig { + /// Outstanding coherent captures. `state-media-v1` section 3's initial session default + /// is two. + pub queue_capacity: usize, + /// The total payload bytes the queue may hold. + pub max_queued_bytes: u64, + pub retry: RetryPolicy, +} + +impl Default for WriterConfig { + fn default() -> WriterConfig { + WriterConfig { + queue_capacity: 2, + max_queued_bytes: 64 * 1024 * 1024, + retry: RetryPolicy::ReleaseAndReport, + } + } +} + +/// Deliberate writer faults, for the rows this slice has to demonstrate. +#[derive(Clone, Default)] +pub struct WriterFaults { + /// Hold every job until the gate is opened, so a test can fill the queue on purpose. + pub gate: Option>, + /// Drop this job's reply channel after the durable sequence ran, which is a lost save + /// reply. + pub drop_reply_for: Option, +} + +impl std::fmt::Debug for WriterFaults { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("WriterFaults") + .field("gate", &self.gate.is_some()) + .field("drop_reply_for", &self.drop_reply_for) + .finish() + } +} + +/// The writer's own counters, so "bounded" is something a test reads rather than believes. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct WriterStats { + pub queued: u64, + pub committed: u64, + pub failed: u64, + pub superseded: u64, + /// Submissions refused because the queue or its byte budget was full. + pub rejected: u64, + /// Checkpoint events the bus would not take. The durable outcome the caller receives is + /// the authority; a publication that failed is counted here rather than disappearing. + pub events_dropped: u64, + /// The deepest the queue ever got. + pub peak_queue: usize, + /// The most payload bytes the queue ever held. + pub peak_bytes: u64, +} + +struct Job { + submission: CaptureSubmission, + reply: tokio::sync::oneshot::Sender, + permit: tokio::sync::OwnedSemaphorePermit, +} + +struct WriterShared { + config: WriterConfig, + faults: WriterFaults, + permits: Arc, + queue: std::sync::Mutex>, + queued_bytes: AtomicU64, + stats: std::sync::Mutex, + wake: tokio::sync::Notify, + stop: AtomicBool, + store: Arc>, + events: Option<(flybus::Client, String)>, +} + +/// A queue slot, taken before a capture is requested so a saturated writer refuses early. +/// +/// `state-media-v1` section 3's durable row says to reject or defer *before capture* when +/// saturated. Holding the slot from before `State.Capture` until the outcome is delivered is +/// what makes that true rather than aspirational. +pub struct Reservation { + permit: tokio::sync::OwnedSemaphorePermit, +} + +/// The bounded checkpoint writer. +pub struct CheckpointWriter { + shared: Arc, + task: tokio::task::JoinHandle<()>, +} + +impl CheckpointWriter { + /// Starts the writer over `store`. `events` is the bus client and topic the distinct + /// captured/queued/committed/failed/superseded publications go to. + pub fn start( + store: CheckpointStore, + config: WriterConfig, + faults: WriterFaults, + events: Option<(flybus::Client, String)>, + ) -> CheckpointWriter { + let shared = Arc::new(WriterShared { + config, + faults, + permits: Arc::new(tokio::sync::Semaphore::new(config.queue_capacity)), + queue: std::sync::Mutex::new(VecDeque::new()), + queued_bytes: AtomicU64::new(0), + stats: std::sync::Mutex::new(WriterStats::default()), + wake: tokio::sync::Notify::new(), + stop: AtomicBool::new(false), + store: Arc::new(std::sync::Mutex::new(store)), + events, + }); + let task = tokio::spawn(run_writer(shared.clone())); + CheckpointWriter { shared, task } + } + + pub fn stats(&self) -> WriterStats { + *self.shared.stats.lock().expect("the writer stats are never poisoned") + } + + pub fn config(&self) -> WriterConfig { + self.shared.config + } + + /// How many captures are outstanding: queued or being written. + pub fn outstanding(&self) -> usize { + self.shared.config.queue_capacity - self.shared.permits.available_permits() + } + + /// Takes a queue slot, or refuses by name. Nothing is captured without one. + pub fn reserve(&self) -> DomainResult { + match self.shared.permits.clone().try_acquire_owned() { + Ok(permit) => Ok(Reservation { permit }), + Err(_) => { + self.shared + .stats + .lock() + .expect("the writer stats are never poisoned") + .rejected += 1; + Err(DomainError::before( + ErrorCode::Busy, + format!( + "the checkpoint queue already holds its {} outstanding captures; a \ +capture is refused before it is requested rather than queued without bound", + self.shared.config.queue_capacity + ), + )) + } + } + } + + /// Hands one complete capture to the writer and returns the channel its outcome arrives + /// on. The reservation becomes the job's slot. + pub async fn submit( + &self, + reservation: Reservation, + submission: CaptureSubmission, + ) -> DomainResult> { + let bytes = submission.byte_length(); + let budget = self.shared.config.max_queued_bytes; + let held = self.shared.queued_bytes.load(Ordering::SeqCst); + if held + bytes > budget { + self.shared + .stats + .lock() + .expect("the writer stats are never poisoned") + .rejected += 1; + return Err(DomainError::before( + ErrorCode::Busy, + format!( + "the checkpoint queue holds {held} of {budget} bytes and this capture adds \ +{bytes}; the byte budget is finite and refuses before it is exceeded" + ), + )); + } + let (reply, receiver) = tokio::sync::oneshot::channel(); + let checkpoint_id = submission.checkpoint_id.clone(); + let queued_event = submission.as_event("queued"); + let superseded = { + let mut queue = self.shared.queue.lock().expect("the writer queue is never poisoned"); + let replaced = if submission.replaceable { + queue + .iter() + .position(|job| job.submission.replaceable) + .map(|index| queue.remove(index).expect("just found")) + } else { + None + }; + queue.push_back(Job { submission, reply, permit: reservation.permit }); + self.shared.queued_bytes.fetch_add(bytes, Ordering::SeqCst); + let mut stats = self.shared.stats.lock().expect("the writer stats are never poisoned"); + stats.queued += 1; + stats.peak_queue = stats.peak_queue.max(queue.len()); + stats.peak_bytes = stats + .peak_bytes + .max(self.shared.queued_bytes.load(Ordering::SeqCst)); + replaced + }; + if let Some(old) = superseded { + self.shared + .queued_bytes + .fetch_sub(old.submission.byte_length(), Ordering::SeqCst); + self.shared + .stats + .lock() + .expect("the writer stats are never poisoned") + .superseded += 1; + let outcome = SaveOutcome::Superseded { + checkpoint_id: old.submission.checkpoint_id.clone(), + by: checkpoint_id.clone(), + }; + publish_event( + &self.shared.events, + &self.shared.stats, + &outcome_event(&old.submission, &outcome), + ) + .await; + // A caller that dropped its ticket is not waiting for this; the store's own + // metadata is the durable record either way. + let _ = old.reply.send(outcome); + // Dropping the job releases its owned captures and its queue slot. + drop(old.submission); + drop(old.permit); + } + publish_event(&self.shared.events, &self.shared.stats, &object(queued_event)).await; + self.shared.wake.notify_one(); + Ok(receiver) + } + + /// Waits for one save's outcome. A dropped reply channel is a lost save reply, which is + /// an outcome and not a hang. + pub async fn wait( + receiver: tokio::sync::oneshot::Receiver, + checkpoint_id: &Id, + budget: std::time::Duration, + ) -> SaveOutcome { + match tokio::time::timeout(budget, receiver).await { + Ok(Ok(outcome)) => outcome, + Ok(Err(_)) | Err(_) => SaveOutcome::ReplyLost { checkpoint_id: checkpoint_id.clone() }, + } + } + + /// Runs `f` against the store, which is how a caller resolves a save whose reply it lost. + /// + /// The store's own work is blocking -- it fsyncs -- so it is reached on a blocking thread + /// rather than from the session's runtime. + pub async fn with_store(&self, f: F) -> T + where + F: FnOnce(&mut CheckpointStore) -> T + Send + 'static, + T: Send + 'static, + { + let store = self.shared.store.clone(); + tokio::task::spawn_blocking(move || { + let mut store = store.lock().expect("the checkpoint store is never poisoned"); + f(&mut store) + }) + .await + .expect("the checkpoint store task is never cancelled") + } + + /// Stops the writer and waits for its task. A leftover writer task would hold artifact + /// handles the session has finished with. + pub async fn shutdown(self) { + self.shared.stop.store(true, Ordering::SeqCst); + self.shared.wake.notify_one(); + let _ = self.task.await; + let mut queue = self.shared.queue.lock().expect("the writer queue is never poisoned"); + queue.clear(); + } +} + +impl CaptureSubmission { + fn as_event(&self, event: &'static str) -> Value { + json!({ + "event": event, + "checkpointId": self.checkpoint_id.as_str(), + "sessionId": self.session_id.as_str(), + "epoch": self.epoch.as_str(), + "episodeId": self.episode_id.as_str(), + "boundary": self.boundary.to_string(), + "byteLength": self.byte_length().to_string(), + }) + } +} + +fn outcome_event(submission: &CaptureSubmission, outcome: &SaveOutcome) -> Map { + let mut payload = object(submission.as_event(outcome.event())); + match outcome { + SaveOutcome::Committed { file, .. } => { + payload.insert("generation".into(), file.as_str().into()); + payload.insert("durable".into(), true.into()); + } + SaveOutcome::Failed { reason, .. } => { + payload.insert("reason".into(), reason.as_str().into()); + payload.insert("durable".into(), false.into()); + } + SaveOutcome::Superseded { by, .. } => { + payload.insert("supersededBy".into(), by.as_str().into()); + payload.insert("durable".into(), false.into()); + } + SaveOutcome::ReplyLost { .. } => { + payload.insert("reason".into(), "the save reply was lost".into()); + payload.insert("durable".into(), false.into()); + } + } + payload +} + +async fn publish_event( + events: &Option<(flybus::Client, String)>, + stats: &std::sync::Mutex, + payload: &Map, +) { + let Some((client, topic)) = events else { return }; + // A checkpoint event is telemetry about the store, not the durable record. A publication + // that cannot be delivered never changes what was committed, so the outcome the caller + // receives stays the authority -- and the drop is counted rather than retried, because a + // retry here would be exactly the implicit best-effort policy these contracts refuse. + if client.publish(topic, payload.clone(), &[]).await.is_err() { + stats + .lock() + .expect("the writer stats are never poisoned") + .events_dropped += 1; + } +} + +async fn run_writer(shared: Arc) { + loop { + let job = { + let mut queue = shared.queue.lock().expect("the writer queue is never poisoned"); + queue.pop_front() + }; + let Some(job) = job else { + if shared.stop.load(Ordering::SeqCst) { + return; + } + shared.wake.notified().await; + continue; + }; + if let Some(gate) = &shared.faults.gate { + // A deliberately stalled writer. The queue in front of it stays bounded, which is + // the point of the stall. + if let Ok(permit) = gate.acquire().await { + permit.forget(); + } + } + let Job { submission, reply, permit } = job; + shared + .queued_bytes + .fetch_sub(submission.byte_length(), Ordering::SeqCst); + let outcome = write_one(&shared, &submission).await; + { + let mut stats = shared.stats.lock().expect("the writer stats are never poisoned"); + match &outcome { + SaveOutcome::Committed { .. } => stats.committed += 1, + SaveOutcome::Failed { .. } | SaveOutcome::ReplyLost { .. } => stats.failed += 1, + SaveOutcome::Superseded { .. } => stats.superseded += 1, + } + } + publish_event(&shared.events, &shared.stats, &outcome_event(&submission, &outcome)).await; + let lost = shared.faults.drop_reply_for.as_ref() == Some(&submission.checkpoint_id); + // The writer owned these handles until the bytes were committed or the job failed. + // The job is over, so they and its queue slot go before the outcome is delivered: + // a caller that reads the queue depth the moment its outcome arrives must not see a + // slot this job has finished with. + drop(submission); + drop(permit); + if lost { + // The bytes are committed and the acknowledgment is lost. Durable metadata is in + // the store manifest, which is exactly where the caller must look. + drop(reply); + } else { + // A caller that dropped its ticket is not waiting for this. + let _ = reply.send(outcome); + } + } +} + +async fn write_one(shared: &Arc, submission: &CaptureSubmission) -> SaveOutcome { + let mut payloads = Vec::with_capacity(submission.payloads.len()); + for payload in &submission.payloads { + let bytes = match payload.artifact.read_all().await { + Ok(bytes) => bytes, + Err(e) => { + return SaveOutcome::Failed { + checkpoint_id: submission.checkpoint_id.clone(), + reason: format!("payload {}: {}", payload.name, e.message), + }; + } + }; + if bytes.len() as u64 != payload.byte_length || digest_of_bytes(&bytes) != payload.digest { + return SaveOutcome::Failed { + checkpoint_id: submission.checkpoint_id.clone(), + reason: format!( + "payload {} is not the content its capture declared", + payload.name + ), + }; + } + payloads.push((payload.name.clone(), bytes)); + } + let bytes = match checkpoint::encode(&submission.manifest, &payloads) { + Ok(bytes) => bytes, + Err(e) => { + return SaveOutcome::Failed { + checkpoint_id: submission.checkpoint_id.clone(), + reason: format!("envelope: {}", e.0), + }; + } + }; + let record = GenerationRecord { + checkpoint_id: submission.checkpoint_id.clone(), + file: format!("{}.flysess", submission.checkpoint_id), + session_id: submission.session_id.clone(), + epoch: submission.epoch.clone(), + episode_id: submission.episode_id.clone(), + boundary: submission.boundary, + compatibility_digest: submission.compatibility_digest.clone(), + envelope_digest: digest_of_bytes(&bytes), + byte_length: bytes.len() as u64, + }; + let attempts = match shared.config.retry { + RetryPolicy::ReleaseAndReport => 1, + RetryPolicy::RetryThenRelease { attempts } => attempts.max(1), + }; + let mut last = String::new(); + for _ in 0..attempts { + let record = record.clone(); + let bytes = bytes.clone(); + let store = shared.store.clone(); + // The durable sequence fsyncs twice. It runs on a blocking thread so the session's + // runtime is never the thing waiting on a disk. + let result = tokio::task::spawn_blocking(move || { + let mut store = store.lock().expect("the checkpoint store is never poisoned"); + store.commit(record, &bytes) + }) + .await; + match result { + Ok(Ok(())) => { + return SaveOutcome::Committed { + checkpoint_id: submission.checkpoint_id.clone(), + boundary: submission.boundary, + file: format!("{}.flysess", submission.checkpoint_id), + }; + } + Ok(Err(e)) => last = e.message, + Err(e) => last = format!("the checkpoint writer stopped: {e}"), + } + } + SaveOutcome::Failed { + checkpoint_id: submission.checkpoint_id.clone(), + reason: last, + } +} diff --git a/services/flysim/crates/fly-session/src/task.rs b/services/flysim/crates/fly-session/src/task.rs index b48e909..2eebbef 100644 --- a/services/flysim/crates/fly-session/src/task.rs +++ b/services/flysim/crates/fly-session/src/task.rs @@ -39,6 +39,16 @@ pub fn episode_schema() -> SchemaRef { synthetic_schema("arena.episode.v1", 1) } +/// The schema of a captured task ledger. +pub fn ledger_schema() -> SchemaRef { + synthetic_schema("arena.ledger.v1", 1) +} + +/// The schema of a captured action-executor state. +pub fn executor_schema() -> SchemaRef { + synthetic_schema("arena.executor.v1", 1) +} + pub fn controller_schema_ref() -> SchemaRef { synthetic_schema("arena.controller.v1", 1) } @@ -86,6 +96,29 @@ pub trait Task: Send { /// How many times `evaluate_transition` has run. A transition must evaluate once. fn evaluations(&self) -> u64; + + /// The checkpointable ledger at a committed boundary (`workers-v1` section 4). + fn capture(&self) -> DomainResult; + + /// Validates a captured ledger without installing it, so a group install can fail before + /// anything is changed. + fn validate_restore(&self, state: &TypedValue) -> DomainResult<()>; + + /// Installs a validated ledger under `epoch`. Event identity is derived from the epoch, + /// so the new one is part of the install rather than something the ledger keeps from the + /// epoch it was captured in. + fn install_restore(&mut self, epoch: &Id, state: &TypedValue) -> DomainResult<()>; + + /// Every event identity this ledger has issued, mapped onto the identity it would have + /// under `to_epoch`. + /// + /// `workers-v1` section 4 derives an event id from the epoch, so a trace recorded in one + /// epoch cannot be compared with a trace recorded in another until these are rebased. + /// The ledger owns the derivation, so it is the only thing that can do it. + fn rebase_ids(&self, to_epoch: &Id) -> DomainResult>; + + /// How far event identity has reached: the highest source step and the number issued. + fn event_watermarks(&self) -> (u64, u64); } /// Translates one selected decision into a controller intent, with no port assignment. @@ -98,6 +131,15 @@ pub trait ActionExecutor: Send { progress: &TypedValue, clock: &RationalNs, ) -> DomainResult<(ControllerIntent, Vec)>; + + /// Per-executor state at a committed boundary (`workers-v1` section 4). + fn capture(&self) -> DomainResult; + + /// Validates a captured executor state without installing it. + fn validate_restore(&self, state: &TypedValue) -> DomainResult<()>; + + /// Installs a validated executor state. + fn install_restore(&mut self, state: &TypedValue) -> DomainResult<()>; } /// The only executor v1 supports: it passes a direct-control decision through unchanged. @@ -123,6 +165,37 @@ impl ActionExecutor for IdentityExecutor { .map_err(|e| DomainError::invalid(format!("decision: {e}")))?; Ok((intent, Vec::new())) } + + /// The identity executor is stateless, and says so rather than capturing nothing. + /// + /// An empty object would be indistinguishable from a stateful executor whose capture went + /// missing, so the capture names the executor it came from and a restore refuses any + /// other one. + fn capture(&self) -> DomainResult { + TypedValue::new(executor_schema(), json!({"executor": "identity-v1"})) + .map_err(|e| DomainError::invalid(e.0)) + } + + fn validate_restore(&self, state: &TypedValue) -> DomainResult<()> { + if state.schema != executor_schema() { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the captured executor state does not carry the executor schema", + )); + } + match state.value.get("executor").and_then(Value::as_str) { + Some("identity-v1") => Ok(()), + other => Err(DomainError::before( + ErrorCode::IncompatibleState, + format!("the captured executor is {other:?}, not the identity executor"), + )), + } + } + + fn install_restore(&mut self, state: &TypedValue) -> DomainResult<()> { + // Stateless: validation is the whole of the install, and it is not skipped. + self.validate_restore(state) + } } /// When the counter task asks for a terminal episode transition. @@ -146,6 +219,10 @@ pub struct CounterTask { evaluations: u64, total_reward: f64, counter: i64, + /// The highest source step any issued event belongs to, and how many were issued. These + /// are the event watermarks a checkpoint records and a resumed epoch continues from. + last_source_step: u64, + issued_events: u64, terminal: Terminal, } @@ -159,6 +236,8 @@ impl CounterTask { evaluations: 0, total_reward: 0.0, counter: 0, + last_source_step: 0, + issued_events: 0, terminal, } } @@ -239,6 +318,7 @@ impl Task for CounterTask { payload: TypedValue::new(event_schema(), json!({"counter": self.counter})) .expect("a synthetic typed value fits the contract"), }]; + self.issued_events += events.len() as u64; Ok(Bootstrap { contexts, progress: self.progress_value(), events }) } @@ -314,6 +394,8 @@ impl Task for CounterTask { )); } + self.last_source_step = self.last_source_step.max(source_step); + self.issued_events += events.len() as u64; let next_contexts = self .agents .iter() @@ -346,6 +428,173 @@ impl Task for CounterTask { fn evaluations(&self) -> u64 { self.evaluations } + + fn capture(&self) -> DomainResult { + TypedValue::new( + ledger_schema(), + json!({ + "epoch": self.epoch.as_str(), + "agents": self.agents.iter().map(String::as_str).collect::>(), + "bindings": self + .bindings + .iter() + .map(|b| json!({"portId": b.port_id.as_str(), "agentId": b.agent_id.as_str()})) + .collect::>(), + "transitions": self.transitions, + "evaluations": self.evaluations, + "totalReward": self.total_reward, + "counter": self.counter, + "lastSourceStep": self.last_source_step, + "issuedEvents": self.issued_events, + }), + ) + .map_err(|e| DomainError::invalid(e.0)) + } + + fn validate_restore(&self, state: &TypedValue) -> DomainResult<()> { + if state.schema != ledger_schema() { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the captured ledger does not carry this task's schema", + )); + } + for field in [ + "epoch", + "agents", + "bindings", + "transitions", + "evaluations", + "totalReward", + "counter", + "lastSourceStep", + "issuedEvents", + ] { + if state.value.get(field).is_none() { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + format!("the captured ledger has no {field}"), + )); + } + } + let bindings = state + .value + .get("bindings") + .and_then(Value::as_array) + .ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the captured ledger's bindings are not a list", + ) + })?; + if bindings.len() != self.bindings.len() && !self.bindings.is_empty() { + return Err(DomainError::before( + ErrorCode::IncompatibleState, + "the captured ledger binds another number of ports", + )); + } + Ok(()) + } + + fn install_restore(&mut self, epoch: &Id, state: &TypedValue) -> DomainResult<()> { + self.validate_restore(state)?; + let number = |key: &str| -> DomainResult { + state.value.get(key).and_then(Value::as_u64).ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + format!("the captured ledger's {key} is not a whole number"), + ) + }) + }; + let mut agents = Vec::new(); + for value in state.value["agents"].as_array().expect("validated") { + let agent = value.as_str().ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the captured ledger names an agent that is not a string", + ) + })?; + agents.push(parse_id(agent).map_err(|e| { + DomainError::before(ErrorCode::IncompatibleState, format!("ledger: {e}")) + })?); + } + let mut bindings = Vec::new(); + for value in state.value["bindings"].as_array().expect("validated") { + let port_id = value.get("portId").and_then(Value::as_str).ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the captured ledger has a binding with no portId", + ) + })?; + let agent_id = value.get("agentId").and_then(Value::as_str).ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the captured ledger has a binding with no agentId", + ) + })?; + bindings.push(PortBinding { + port_id: parse_id(port_id).map_err(|e| { + DomainError::before(ErrorCode::IncompatibleState, format!("ledger: {e}")) + })?, + agent_id: parse_id(agent_id).map_err(|e| { + DomainError::before(ErrorCode::IncompatibleState, format!("ledger: {e}")) + })?, + }); + } + let counter = state.value.get("counter").and_then(Value::as_i64).ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the captured ledger's counter is not an integer", + ) + })?; + let total_reward = state + .value + .get("totalReward") + .and_then(Value::as_f64) + .filter(|v| v.is_finite()) + .ok_or_else(|| { + DomainError::before( + ErrorCode::IncompatibleState, + "the captured ledger's totalReward is not a finite number", + ) + })?; + // The epoch is the caller's, not the capture's: event identity belongs to the epoch + // the ledger is being installed into. + self.epoch = epoch.clone(); + self.agents = agents; + self.bindings = bindings; + self.transitions = number("transitions")?; + self.evaluations = number("evaluations")?; + self.total_reward = total_reward; + self.counter = counter; + self.last_source_step = number("lastSourceStep")?; + self.issued_events = number("issuedEvents")?; + Ok(()) + } + + fn rebase_ids(&self, to_epoch: &Id) -> DomainResult> { + let mut out = BTreeMap::new(); + out.insert( + event_id(&self.epoch, 0, "bootstrap", 0), + event_id(to_epoch, 0, "bootstrap", 0), + ); + // The counter task issues exactly one `counter-delta` event per bound port per + // evaluated transition, in descriptor port order, so every identity it has ever + // issued is re-derivable from its ledger without keeping a list of them. + let ports = self.bindings.len() as u32; + for source_step in 1..=self.last_source_step { + for ordinal in 0..ports { + out.insert( + event_id(&self.epoch, source_step, "counter-delta", ordinal), + event_id(to_epoch, source_step, "counter-delta", ordinal), + ); + } + } + Ok(out) + } + + fn event_watermarks(&self) -> (u64, u64) { + (self.last_source_step, self.issued_events) + } } /// The inspection value the counter environment publishes. diff --git a/services/flysim/crates/fly-session/src/types.rs b/services/flysim/crates/fly-session/src/types.rs index c52c276..07041c2 100644 --- a/services/flysim/crates/fly-session/src/types.rs +++ b/services/flysim/crates/fly-session/src/types.rs @@ -8,13 +8,18 @@ //! `Id` and `Digest` are type aliases, because the shared crate carries both as validated //! `String`s from `flybus::wire` rather than forking the encodings into newtypes. +use std::collections::BTreeMap; + use serde_json::{Map, Value}; pub use fly_session_types::ArtifactRef; pub use fly_session_types::canonical::{ self, OperationKey, body_digest, canonicalize, digest_of, sha256_hex, }; -pub use fly_session_types::media::{AudioDescriptor, AudioRef, ViewDescriptor, ViewRef}; +pub use fly_session_types::media::{ + ActivateRestoreParams, ActivateRestoreResult, AudioDescriptor, AudioRef, CaptureParams, + CaptureResult, StageRestoreParams, StageRestoreResult, ViewDescriptor, ViewRef, +}; pub use fly_session_types::rpc::{ ErrorCode, MutationCertainty, SessionRpcFailure, SessionRpcOutcome, SessionRpcRequest, SessionRpcSuccess, @@ -214,6 +219,57 @@ pub fn outcome_identity( } } +/// The epoch-derived identities of a behaviour trace, rewritten onto one reference epoch. +/// +/// `step-v1` section 8 compares committed behaviour across runs, excluding wall time, request +/// ids "and other explicitly operational metadata". A resumed run's epoch is neither: it is +/// behaviour metadata, and `scope.epoch`, the batch id and every task event id are derived +/// from it. Comparing the two runs therefore means rewriting exactly those three things and +/// nothing else, which is what this does -- and it **fails** on anything it does not +/// recognise instead of passing it through, so a field that silently stopped being rebased +/// would fail the comparison rather than weaken it. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct EpochRebase { + pub from: Id, + pub to: Id, + /// Every event identity the task issued under `from`, and the identity it has under `to`. + pub events: BTreeMap, +} + +impl EpochRebase { + /// Rewrites one behaviour record. An identity this rebase does not know is an error. + pub fn apply(&self, behaviour: &TraceBehaviour) -> Result { + if behaviour.scope.epoch != self.from { + return Err(format!( + "this behaviour was recorded in epoch {}, not {}", + behaviour.scope.epoch, self.from + )); + } + let mut out = behaviour.clone(); + out.scope = Scope::new(&behaviour.scope.session_id, &self.to, behaviour.scope.step) + .map_err(|e| e.0)?; + let prefix = format!("batch-{}-", self.from); + let suffix = behaviour + .batch_id + .strip_prefix(&prefix) + .ok_or_else(|| format!("the batch id {} is not derived from {}", behaviour.batch_id, self.from))?; + out.batch_id = parse_id(&format!("batch-{}-{suffix}", self.to))?; + let map = |ids: &[Id]| -> Result, String> { + ids.iter() + .map(|id| { + self.events + .get(id) + .cloned() + .ok_or_else(|| format!("no rebased identity for the event {id}")) + }) + .collect() + }; + out.outcome_ids = map(&behaviour.outcome_ids)?; + out.event_ids = map(&behaviour.event_ids)?; + Ok(out) + } +} + /// One session phase transition, recorded whether or not it ends a step. #[derive(Clone, Debug, PartialEq, Eq)] pub struct PhaseTransition { @@ -253,6 +309,28 @@ impl TraceLog { .collect() } + /// Every transition's behaviour, rebased onto one epoch and canonicalized. + /// + /// This is the comparison a resumed run is held to: the same strings as + /// [`TraceLog::behavior`], with the epoch metadata accounted for and nothing else changed. + /// A resumed run's log holds transitions from two epochs -- the ones before the checkpoint + /// and the ones after the restore -- so a transition already recorded in `rebase.to` is + /// kept as it stands and one recorded in `rebase.from` is rewritten. A transition in a + /// third epoch is an error; there is no pass-through case. + pub fn behavior_rebased(&self, rebase: &EpochRebase) -> Result, String> { + self.transitions + .iter() + .map(|t| { + let behaviour = if t.behaviour.scope.epoch == rebase.to { + t.behaviour.clone() + } else { + rebase.apply(&t.behaviour)? + }; + canonicalize(&behaviour.to_json()).map_err(|e| e.0) + }) + .collect() + } + /// The phase path, as `from -> to` strings. pub fn phase_path(&self) -> Vec { self.phases.iter().map(|p| format!("{} -> {}", p.from, p.to)).collect() diff --git a/services/flysim/crates/fly-session/src/worker.rs b/services/flysim/crates/fly-session/src/worker.rs index 327d66b..d717878 100644 --- a/services/flysim/crates/fly-session/src/worker.rs +++ b/services/flysim/crates/fly-session/src/worker.rs @@ -597,7 +597,17 @@ async fn execute( fn classify_default(method: &str) -> Option { match method { "Agent.Prepare" | "Agent.Commit" | "Environment.Advance" => Some(OpClass::StepMutation), - "Agent.Initialize" | "Environment.Initialize" => Some(OpClass::Lifecycle), + // `ipc-v1` section 5 retains lifecycle *and capture* replies until + // `Worker.Acknowledge`. The restore methods join them: their replies carry a + // once-only token and, for an environment, the restored observation's artifact, and a + // duplicate domain request must replay that reply rather than stage or activate a + // second time. They are not step mutations -- they carry no committed step of their + // own and are not keyed by one. + "Agent.Initialize" + | "Environment.Initialize" + | "State.Capture" + | "State.StageRestore" + | "State.ActivateRestore" => Some(OpClass::Lifecycle), "Worker.Hello" | "Worker.Status" | "Worker.Acknowledge" | "Worker.Shutdown" => { Some(OpClass::ReadOnly) } diff --git a/services/flysim/crates/fly-session/tests/processes.rs b/services/flysim/crates/fly-session/tests/processes.rs index d2dcd53..17dd0e2 100644 --- a/services/flysim/crates/fly-session/tests/processes.rs +++ b/services/flysim/crates/fly-session/tests/processes.rs @@ -121,6 +121,7 @@ async fn a_slow_participant_is_resolved_rather_than_failed(mode: ExecutionMode) resolve: Duration::from_secs(20), resolve_attempts: 4096, boot: Duration::from_secs(30), + capture: Duration::from_secs(30), }; within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap(); let reports = within("run", f.harness.coordinator.run(2)) @@ -191,6 +192,7 @@ async fn a_resolution_says_which_of_its_two_bounds_ended_it(mode: ExecutionMode) resolve: Duration::from_millis(300), resolve_attempts: 8192, boot: Duration::from_secs(30), + capture: Duration::from_secs(30), }; within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap(); let started = Instant::now(); @@ -221,6 +223,7 @@ async fn a_resolution_says_which_of_its_two_bounds_ended_it(mode: ExecutionMode) resolve: Duration::from_secs(600), resolve_attempts: 3, boot: Duration::from_secs(30), + capture: Duration::from_secs(30), }; within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap(); let failure = within("step", f.harness.coordinator.step()) diff --git a/services/flysim/crates/fly-session/tests/state.rs b/services/flysim/crates/fly-session/tests/state.rs new file mode 100644 index 0000000..d7b3a0f --- /dev/null +++ b/services/flysim/crates/fly-session/tests/state.rs @@ -0,0 +1,947 @@ +//! STATE-01 acceptance: one coherent all-participant checkpoint, and the recovery that +//! installs it into a fresh epoch. +//! +//! Every test here is one of the slice's acceptance bullets or one of the failure-injection +//! rows the implementation guide's section 4 assigns to it. Each of them runs over both +//! transports and in all three execution modes: the recovery path crosses a process boundary +//! in exactly the places the media path does, so a row that holds in one mode has to hold in +//! all of them. +//! +//! The byte layout itself is proved against the contract crate in +//! `fly-session-types/tests/checkpoint_envelope.rs`; these prove the session's use of it. + +mod common; + +use std::collections::BTreeSet; +use std::path::Path; +use std::sync::Arc; + +use serde_json::{Value, json}; + +use common::{Fixture, count, fixture, fly_a, fly_b, within}; +use fly_session::agent::AgentFaults; +use fly_session::harness::{ExecutionMode, HarnessConfig, Via}; +use fly_session::phase::Phase; +use fly_session::state::{SaveOutcome, WriterFaults}; +use fly_session::types::*; +use fly_session_types::checkpoint; + +both_transports!( + an_uninterrupted_run_and_a_resumed_run_produce_matching_traces, + corrupting_any_single_participants_payload_fails_the_install_as_a_group, + a_lost_save_reply_does_not_advance_durable_metadata, + a_failure_during_activation_cannot_resume_half_a_world, + the_checkpoint_queue_under_stress_stays_bounded, + old_media_cannot_cross_a_recovery, + a_checkpoint_taken_by_another_parser_is_refused_by_name, + a_group_where_one_participant_will_not_stage_resumes_nothing, + a_restore_token_activates_only_once, + the_fence_lifts_only_through_a_coherent_restore, + a_queued_replaceable_capture_is_superseded_rather_than_duplicated, + a_capture_is_refused_anywhere_but_a_committed_boundary, +); + +all_modes!( + matching_traces_in_every_mode, + a_group_install_fails_as_a_group_in_every_mode, + a_lost_save_reply_holds_durable_metadata_in_every_mode, + half_an_activation_resumes_nothing_in_every_mode, + the_queue_stays_bounded_in_every_mode, + old_media_cannot_cross_in_every_mode, + another_parser_is_refused_in_every_mode, + a_refused_stage_resumes_nothing_in_every_mode, + a_token_activates_once_in_every_mode, + the_fence_lifts_only_by_restore_in_every_mode, +); + +const BEFORE: u64 = 2; +const AFTER: u64 = 2; + +fn ckpt(n: u32) -> Id { + id(&format!("ckpt-{n}")) +} + +/// A fixture in one transport and one execution mode. +/// +/// `Via` is the composition's; a dedicated thread or a separate process reaches the router +/// over a socket whatever it says, which the launcher decides and this does not second-guess. +async fn fx(via: Via, mode: ExecutionMode, config: HarnessConfig) -> Fixture { + fixture(via, HarnessConfig { mode, ..config }).await +} + +/// Runs to a committed boundary and takes one durable checkpoint there. +async fn run_and_checkpoint(f: &mut Fixture, steps: u64, checkpoint_id: &Id) { + within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap(); + within("run", f.harness.coordinator.run(steps)).await.unwrap(); + let outcome = within("checkpoint", f.harness.coordinator.checkpoint(checkpoint_id)) + .await + .unwrap(); + match outcome { + SaveOutcome::Committed { boundary, .. } => assert_eq!(boundary, steps), + other => panic!("the checkpoint was not committed: {other:?}"), + } + assert_eq!( + f.harness.coordinator.durable(), + Some((checkpoint_id.clone(), steps)), + "a committed save is the only thing that moves the durable mark" + ); +} + +/// Fails the epoch the way a participant death does, and checks the fence closed. +async fn fail_the_epoch(f: &mut Fixture) { + f.harness.kill(&fly_a()).await; + let failure = within("step", f.harness.coordinator.step()) + .await + .expect_err("a dead participant fails the epoch"); + assert!( + failure.participant.is_some(), + "a diagnosed failure names its participant: {failure}" + ); + assert_eq!(f.harness.coordinator.phase(), Phase::Failed); + assert!(f.harness.coordinator.is_fenced()); + assert_eq!( + f.harness.coordinator.live_view_handles(), + 0, + "the fence drops every handle the old epoch held" + ); +} + +/// The committed generation's bytes, as they are on disk. +fn generation_bytes(root: &Path, checkpoint_id: &Id) -> Vec { + std::fs::read(root.join(format!("{checkpoint_id}.flysess"))).expect("a committed generation") +} + +/// Writes a generation file and makes the store manifest describe it, as a repair tool or a +/// previous run would have left it. +fn write_generation(root: &Path, checkpoint_id: &Id, bytes: &[u8]) { + std::fs::write(root.join(format!("{checkpoint_id}.flysess")), bytes).expect("writable"); + let path = root.join("manifest.json"); + let mut manifest: Value = + serde_json::from_slice(&std::fs::read(&path).expect("a store manifest")).expect("json"); + let generations = manifest["generations"].as_array_mut().expect("an array"); + for generation in generations.iter_mut() { + if generation["checkpointId"] == json!(checkpoint_id.as_str()) { + generation["byteLength"] = json!(bytes.len().to_string()); + generation["envelopeDigest"] = json!(digest_of_bytes(bytes)); + let envelope = checkpoint::decode(bytes).expect("a well formed envelope"); + let compatibility = fly_session::state::Compatibility::from_json( + &envelope.manifest["compatibility"], + ) + .expect("a compatibility block"); + generation["compatibilityDigest"] = json!(compatibility.digest()); + } + } + std::fs::write(&path, serde_json::to_vec(&manifest).expect("json")).expect("writable"); +} + +/// Re-encodes one committed generation after `edit` has changed its manifest or its payloads. +fn rewrite_generation( + root: &Path, + checkpoint_id: &Id, + edit: impl FnOnce(&mut Value, &mut Vec<(String, Vec)>), +) { + let bytes = generation_bytes(root, checkpoint_id); + let envelope = checkpoint::decode(&bytes).expect("a committed generation decodes"); + let mut manifest = envelope.manifest.clone(); + let mut payloads = envelope.payloads.clone(); + edit(&mut manifest, &mut payloads); + // The manifest's payload table mirrors the envelope's, so it is rebuilt from the bytes + // that are actually being written rather than left to disagree. + manifest["payloads"] = Value::Array( + payloads + .iter() + .map(|(name, bytes)| { + json!({ + "name": name, + "byteLength": bytes.len().to_string(), + "digest": digest_of_bytes(bytes), + }) + }) + .collect(), + ); + let rewritten = checkpoint::encode(&manifest, &payloads).expect("a valid envelope"); + write_generation(root, checkpoint_id, &rewritten); +} + +/// Makes the store read its durable metadata again, after a test has edited it. +async fn reload_store(f: &mut Fixture) { + f.harness + .coordinator + .writer() + .expect("a store is attached") + .with_store(|store| store.reload()) + .await + .expect("the store manifest is still readable"); +} + +/// The names of every payload the checkpoint holds, participants first. +fn payload_names(root: &Path, checkpoint_id: &Id) -> Vec { + let bytes = generation_bytes(root, checkpoint_id); + checkpoint::decode(&bytes) + .expect("decodes") + .payloads + .into_iter() + .map(|(name, _)| name) + .collect() +} + +/// Every artifact identity this session's committed boundary is holding. +fn live_artifact_ids(f: &Fixture) -> BTreeSet { + f.harness + .coordinator + .media_handles() + .into_iter() + .map(|(_, reference)| reference.artifact_id) + .collect() +} + +/// Asserts that no participant is running the proposed epoch: nothing was installed. +async fn nothing_is_installed(f: &mut Fixture, epoch: &Id) { + let ids: Vec = std::iter::once(f.harness.environment_id()) + .chain(f.harness.config.agents.iter().map(|a| a.agent_id.clone())) + .collect(); + for worker_id in ids { + let (_, launcher) = f.harness.parts(); + let Ok(status) = launcher.health_check(&worker_id).await else { + // A participant that is gone is certainly not running the new epoch. + continue; + }; + if let Some(scope) = status.current_scope { + assert_ne!( + scope.epoch, *epoch, + "{worker_id} is running the epoch the abandoned install proposed" + ); + } + } + assert_eq!(f.harness.coordinator.phase(), Phase::Failed); + assert!(f.harness.coordinator.is_fenced()); + let refused = within("step", f.harness.coordinator.step()) + .await + .expect_err("a fenced session takes no step"); + assert_eq!(refused.error.code, ErrorCode::InvalidPhase); +} + +// =============================================================================================== +// Acceptance: an uninterrupted run and a resumed run produce matching traces + +async fn an_uninterrupted_run_and_a_resumed_run_produce_matching_traces(via: Via) { + matching_traces(via, ExecutionMode::InProcess).await; +} + +async fn matching_traces_in_every_mode(mode: ExecutionMode) { + matching_traces(Via::Unix, mode).await; +} + +/// The reference run and the resumed run commit the same behaviour, once the epoch metadata +/// the restore necessarily changed is accounted for. +/// +/// `step-v1` section 8's split is the whole of the comparison: the behaviour is what must +/// match, the request ids and wall time are excluded because they are operational, and the +/// epoch is neither -- it is behaviour metadata, so it is rewritten explicitly and everything +/// else is compared byte for byte. +async fn matching_traces(via: Via, mode: ExecutionMode) { + let mut reference = fx(via, mode, HarnessConfig::default()).await; + within("bootstrap", reference.harness.coordinator.bootstrap()).await.unwrap(); + within("run", reference.harness.coordinator.run(BEFORE + AFTER)).await.unwrap(); + let expected = reference.harness.coordinator.trace.behavior(); + assert_eq!(expected.len() as u64, BEFORE + AFTER); + reference.shutdown().await; + + let mut f = fx(via, mode, HarnessConfig::default()).await; + let checkpoint_id = ckpt(1); + run_and_checkpoint(&mut f, BEFORE, &checkpoint_id).await; + fail_the_epoch(&mut f).await; + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let report = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e2")), + ) + .await + .unwrap(); + assert_eq!(report.boundary, BEFORE); + assert_eq!(report.epoch, id("e2")); + assert_eq!(report.activated.len(), report.staged.len()); + assert!(!f.harness.coordinator.is_fenced(), "a coherent restore lifts the fence"); + assert_eq!(f.harness.coordinator.phase(), Phase::Paused(BEFORE)); + + f.harness.coordinator.resume().unwrap(); + within("resume", f.harness.coordinator.run(AFTER)).await.unwrap(); + assert_eq!(f.harness.coordinator.phase(), Phase::Ready(BEFORE + AFTER)); + + // Without accounting for the epoch the two traces disagree, which is what makes the + // rebase a statement rather than a formality. + let raw = f.harness.coordinator.trace.behavior(); + assert_eq!(raw.len() as u64, BEFORE + AFTER); + assert_ne!(raw, expected, "the resumed run runs in a different epoch"); + + let rebase = f.harness.coordinator.rebase(&id("e1")).unwrap(); + let resumed = f.harness.coordinator.trace.behavior_rebased(&rebase).unwrap(); + assert_eq!( + resumed, expected, + "a resumed run commits the behaviour the uninterrupted run committed" + ); + f.shutdown().await; +} + +// =============================================================================================== +// Acceptance: corrupt any participant and installation fails as a group + +async fn corrupting_any_single_participants_payload_fails_the_install_as_a_group(via: Via) { + group_install_is_all_or_nothing(via, ExecutionMode::InProcess).await; +} + +async fn a_group_install_fails_as_a_group_in_every_mode(mode: ExecutionMode) { + group_install_is_all_or_nothing(Via::Unix, mode).await; +} + +/// One corrupted payload -- any participant's, and the coordinator's own ledger too -- fails +/// the whole install, and the group afterwards is exactly as it was. +async fn group_install_is_all_or_nothing(via: Via, mode: ExecutionMode) { + let mut f = fx(via, mode, HarnessConfig::default()).await; + let checkpoint_id = ckpt(1); + run_and_checkpoint(&mut f, BEFORE, &checkpoint_id).await; + let root = f.harness.checkpoint_root().to_path_buf(); + let good = generation_bytes(&root, &checkpoint_id); + let names = payload_names(&root, &checkpoint_id); + // Every participant's payload, plus one of the coordinator's own ledgers. + let mut corrupt: Vec = names + .iter() + .filter(|name| name.starts_with("agent-") || *name == "world") + .cloned() + .collect(); + corrupt.push("task-ledger".to_owned()); + assert!(corrupt.len() >= 3, "the composition has several payloads: {names:?}"); + fail_the_epoch(&mut f).await; + + let mut epoch = 1u32; + for target in &corrupt { + epoch += 1; + let proposed = id(&format!("e{epoch}")); + // Each round starts from the intact bytes, so exactly one payload is corrupt. + write_generation(&root, &checkpoint_id, &good); + // A well formed envelope whose digests all agree, so what fails is the participant + // reading its own bytes and not the envelope reader in front of it. + let name = target.clone(); + rewrite_generation(&root, &checkpoint_id, |_manifest, payloads| { + for (payload, bytes) in payloads.iter_mut() { + if *payload == name { + let last = bytes.len() - 1; + bytes[last] ^= 0xff; + } + } + }); + reload_store(&mut f).await; + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let failure = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &proposed), + ) + .await + .expect_err(&format!("a corrupt {target} must fail the install")); + assert!( + matches!( + failure.error.code, + ErrorCode::IncompatibleState | ErrorCode::InvalidArgument + ), + "a corrupt {target} is an explicit refusal, not {failure}" + ); + nothing_is_installed(&mut f, &proposed).await; + let tainted = f.harness.coordinator.tainted(); + if target == "world" { + assert!( + tainted.is_empty(), + "the first participant asked refused, so nothing staged: {tainted:?}" + ); + } else { + assert!( + !tainted.is_empty(), + "a participant that staged into an abandoned install must be replaced" + ); + } + } + + // The same group, the same store, the intact bytes: the failures above installed nothing + // that stops this from working. + epoch += 1; + write_generation(&root, &checkpoint_id, &good); + reload_store(&mut f).await; + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let report = within( + "restore", + f.harness + .coordinator + .restore(Some(&checkpoint_id), &id(&format!("e{epoch}"))), + ) + .await + .unwrap(); + assert_eq!(report.boundary, BEFORE); + f.harness.coordinator.resume().unwrap(); + within("resume", f.harness.coordinator.run(1)).await.unwrap(); + f.shutdown().await; +} + +// =============================================================================================== +// Acceptance: a lost save reply does not advance durable metadata + +async fn a_lost_save_reply_does_not_advance_durable_metadata(via: Via) { + a_lost_save_reply(via, ExecutionMode::InProcess).await; +} + +async fn a_lost_save_reply_holds_durable_metadata_in_every_mode(mode: ExecutionMode) { + a_lost_save_reply(Via::Unix, mode).await; +} + +/// Two ways a save can end without a saved acknowledgment, and neither moves the mark. +async fn a_lost_save_reply(via: Via, mode: ExecutionMode) { + let lost = ckpt(1); + let config = HarnessConfig { + writer_faults: WriterFaults { + drop_reply_for: Some(lost.clone()), + ..WriterFaults::default() + }, + ..HarnessConfig::default() + }; + let mut f = fx(via, mode, config).await; + within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap(); + within("run", f.harness.coordinator.run(1)).await.unwrap(); + + let ticket = within("capture", f.harness.coordinator.capture(&lost, false)) + .await + .unwrap(); + let outcome = within("durable", f.harness.coordinator.await_durable(ticket)) + .await + .unwrap(); + assert_eq!(outcome, SaveOutcome::ReplyLost { checkpoint_id: lost.clone() }); + assert_eq!( + f.harness.coordinator.durable(), + None, + "a lost save reply never moves the durable mark" + ); + assert_eq!(count(&f.harness.coordinator.audit, &format!("durable:{lost}@1")), 0); + + // The resolution asks the durable metadata about the *same* operation. It never saves + // again, and here the bytes did reach the store manifest. + let resolved = within("resolve", f.harness.coordinator.resolve_durable(&lost)) + .await + .unwrap(); + assert_eq!(resolved, Some(1)); + assert_eq!(f.harness.coordinator.durable(), Some((lost.clone(), 1))); + let commits = f + .harness + .coordinator + .writer() + .expect("a store") + .with_store(|store| store.commits()) + .await; + assert_eq!(commits, 1, "the resolution queried the store; it did not save again"); + f.shutdown().await; + + // The other half: the generation file is renamed and the store manifest is never + // committed. That generation is unreferenced, so it is not a candidate and the mark is + // still where it was. + let orphan = ckpt(2); + let mut config = HarnessConfig::default(); + config.store_faults.stop_before_manifest_commit = true; + let mut g = fx(via, mode, config).await; + within("bootstrap", g.harness.coordinator.bootstrap()).await.unwrap(); + within("run", g.harness.coordinator.run(1)).await.unwrap(); + let outcome = within("checkpoint", g.harness.coordinator.checkpoint(&orphan)) + .await + .unwrap(); + match outcome { + SaveOutcome::Failed { checkpoint_id, .. } => assert_eq!(checkpoint_id, orphan), + other => panic!("an uncommitted manifest is not a saved checkpoint: {other:?}"), + } + assert_eq!(g.harness.coordinator.durable(), None); + let root = g.harness.checkpoint_root().to_path_buf(); + assert!( + root.join(format!("{orphan}.flysess")).is_file(), + "the generation file was written and renamed" + ); + let resolved = within("resolve", g.harness.coordinator.resolve_durable(&orphan)) + .await + .unwrap(); + assert_eq!(resolved, None, "an unreferenced generation is never a restore candidate"); + assert_eq!(g.harness.coordinator.durable(), None); + g.shutdown().await; +} + +// =============================================================================================== +// Acceptance: a failure during activation cannot resume half a world + +async fn a_failure_during_activation_cannot_resume_half_a_world(via: Via) { + half_an_activation(via, ExecutionMode::InProcess).await; +} + +async fn half_an_activation_resumes_nothing_in_every_mode(mode: ExecutionMode) { + half_an_activation(Via::Unix, mode).await; +} + +/// The world and the first agent activate; the second refuses. Nothing plays, the fence +/// stays closed, and the group cannot be resumed until every participant is replaced. +async fn half_an_activation(via: Via, mode: ExecutionMode) { + let mut f = fx(via, mode, HarnessConfig::default()).await; + let checkpoint_id = ckpt(1); + run_and_checkpoint(&mut f, BEFORE, &checkpoint_id).await; + fail_the_epoch(&mut f).await; + + // The replacement for fly-b refuses to activate after it has staged. + f.harness.set_agent_faults( + &fly_b(), + AgentFaults { fail_activate_restore: true, ..AgentFaults::default() }, + ); + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let failure = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e2")), + ) + .await + .expect_err("a refused activation fails the install"); + assert_eq!(failure.participant.as_deref(), Some(fly_b().as_str())); + assert_eq!(f.harness.coordinator.phase(), Phase::Failed); + assert!(f.harness.coordinator.is_fenced(), "the group stays fenced"); + let refused = within("step", f.harness.coordinator.step()) + .await + .expect_err("half a world never runs"); + assert_eq!(refused.error.code, ErrorCode::InvalidPhase); + + // The participants that got as far as installing hold state no group resumed. Another + // restore over them is refused by name rather than attempted. + let tainted = f.harness.coordinator.tainted(); + assert!(tainted.contains(&f.harness.environment_id()), "{tainted:?}"); + assert!(tainted.contains(&fly_a()), "{tainted:?}"); + let refused = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e3")), + ) + .await + .expect_err("the half-installed group is not restored over"); + assert_eq!(refused.error.code, ErrorCode::InvalidPhase); + assert!(refused.error.message.contains(&fly_a()), "{refused}"); + + // A fresh group, without the injected refusal, resumes the same boundary. + f.harness.set_agent_faults(&fly_b(), AgentFaults::default()); + within("replace", f.harness.replace_all_participants()).await.unwrap(); + assert!(f.harness.coordinator.tainted().is_empty()); + let report = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e4")), + ) + .await + .unwrap(); + assert_eq!(report.boundary, BEFORE); + assert!(!f.harness.coordinator.is_fenced()); + f.harness.coordinator.resume().unwrap(); + within("resume", f.harness.coordinator.run(1)).await.unwrap(); + f.shutdown().await; +} + +// =============================================================================================== +// Acceptance: the checkpoint queue under stress stays bounded + +async fn the_checkpoint_queue_under_stress_stays_bounded(via: Via) { + queue_stays_bounded(via, ExecutionMode::InProcess).await; +} + +async fn the_queue_stays_bounded_in_every_mode(mode: ExecutionMode) { + queue_stays_bounded(Via::Unix, mode).await; +} + +/// With the writer stalled, the queue fills to its configured bound and the next capture is +/// refused *before* a single participant is asked for one. +async fn queue_stays_bounded(via: Via, mode: ExecutionMode) { + let gate = Arc::new(tokio::sync::Semaphore::new(0)); + let mut config = HarnessConfig::default(); + config.writer_faults.gate = Some(gate.clone()); + config.writer.queue_capacity = 2; + let mut f = fx(via, mode, config).await; + within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap(); + within("run", f.harness.coordinator.run(1)).await.unwrap(); + + let first = within("capture", f.harness.coordinator.capture(&ckpt(1), false)) + .await + .unwrap(); + let second = within("capture", f.harness.coordinator.capture(&ckpt(2), false)) + .await + .unwrap(); + assert_eq!(f.harness.coordinator.writer().expect("a store").outstanding(), 2); + + let environment = f.harness.environment_id(); + let before = within("status", f.harness.progress_of(&environment)).await.unwrap(); + let refused = within("capture", f.harness.coordinator.capture(&ckpt(3), false)) + .await + .expect_err("a full queue refuses"); + assert_eq!(refused.error.code, ErrorCode::Busy); + assert_eq!(refused.error.mutation, MutationCertainty::None); + let after = within("status", f.harness.progress_of(&environment)).await.unwrap(); + assert_eq!( + before, after, + "a refused capture asks no participant for one: rejection happens before capture" + ); + assert_eq!(count(&f.harness.coordinator.audit, "capture-refused:ckpt-3"), 1); + + // A refused capture is not a failed session: the world keeps stepping. + assert!(!f.harness.coordinator.is_fenced()); + within("run", f.harness.coordinator.run(1)).await.unwrap(); + assert_eq!(f.harness.coordinator.phase(), Phase::Ready(2)); + + let stats = f.harness.coordinator.writer().expect("a store").stats(); + assert_eq!(stats.rejected, 1); + assert!(stats.peak_queue <= 2, "the queue never exceeded its bound: {stats:?}"); + assert!(stats.peak_bytes <= 64 * 1024 * 1024); + + gate.add_permits(16); + for (ticket, expected) in [(first, ckpt(1)), (second, ckpt(2))] { + let outcome = within("durable", f.harness.coordinator.await_durable(ticket)) + .await + .unwrap(); + match outcome { + SaveOutcome::Committed { checkpoint_id, .. } => assert_eq!(checkpoint_id, expected), + other => panic!("an unstalled writer commits: {other:?}"), + } + } + let stats = f.harness.coordinator.writer().expect("a store").stats(); + assert_eq!(stats.committed, 2); + assert_eq!(f.harness.coordinator.writer().expect("a store").outstanding(), 0); + f.shutdown().await; +} + +/// A queued replaceable capture is replaced by the next one, releasing its holds, rather than +/// both being written. +async fn a_queued_replaceable_capture_is_superseded_rather_than_duplicated(via: Via) { + let gate = Arc::new(tokio::sync::Semaphore::new(0)); + let mut config = HarnessConfig::default(); + config.writer_faults.gate = Some(gate.clone()); + config.writer.queue_capacity = 3; + let mut f = fx(via, ExecutionMode::InProcess, config).await; + within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap(); + within("run", f.harness.coordinator.run(1)).await.unwrap(); + + // The first job is taken by the writer and stalls at the gate; the next two are queued. + let durable = within("capture", f.harness.coordinator.capture(&ckpt(1), false)) + .await + .unwrap(); + let hot = within("capture", f.harness.coordinator.capture(&ckpt(2), true)) + .await + .unwrap(); + let newer = within("capture", f.harness.coordinator.capture(&ckpt(3), true)) + .await + .unwrap(); + + let superseded = within("durable", f.harness.coordinator.await_durable(hot)) + .await + .unwrap(); + assert_eq!( + superseded, + SaveOutcome::Superseded { checkpoint_id: ckpt(2), by: ckpt(3) }, + "only a queued replaceable capture is coalesced" + ); + gate.add_permits(16); + for ticket in [durable, newer] { + let outcome = within("durable", f.harness.coordinator.await_durable(ticket)) + .await + .unwrap(); + assert!(matches!(outcome, SaveOutcome::Committed { .. }), "{outcome:?}"); + } + let stats = f.harness.coordinator.writer().expect("a store").stats(); + assert_eq!(stats.superseded, 1); + assert_eq!(stats.committed, 2, "the superseded capture was never written"); + f.shutdown().await; +} + +// =============================================================================================== +// Acceptance: old media and old parser data cannot cross a recovery + +async fn old_media_cannot_cross_a_recovery(via: Via) { + old_media_cannot_cross(via, ExecutionMode::InProcess).await; +} + +async fn old_media_cannot_cross_in_every_mode(mode: ExecutionMode) { + old_media_cannot_cross(Via::Unix, mode).await; +} + +/// Nothing the fence dropped comes back: the payloads are re-imported as fresh artifacts, the +/// restored view is a new object, and the new epoch's first audio chunk resumes the preserved +/// sample position and marks the discontinuity. +async fn old_media_cannot_cross(via: Via, mode: ExecutionMode) { + let mut f = fx(via, mode, HarnessConfig::default()).await; + let checkpoint_id = ckpt(1); + run_and_checkpoint(&mut f, BEFORE, &checkpoint_id).await; + let old_ids = live_artifact_ids(&f); + assert!(!old_ids.is_empty(), "the committed boundary holds media handles"); + let positions = f.harness.coordinator.audio_positions(); + assert!(positions.values().any(|sample| *sample > 0), "audio has played"); + + fail_the_epoch(&mut f).await; + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let report = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e2")), + ) + .await + .unwrap(); + for imported in &report.imported { + assert!( + !old_ids.contains(imported), + "a checkpoint payload was imported as an artifact the old epoch already had" + ); + } + let new_ids = live_artifact_ids(&f); + assert!(!new_ids.is_empty()); + assert!( + new_ids.is_disjoint(&old_ids), + "the restored boundary's media are fresh artifacts, not the old epoch's" + ); + assert_eq!( + f.harness.coordinator.audio_positions(), + positions, + "crash restore preserves the sample position" + ); + + f.harness.coordinator.resume().unwrap(); + within("resume", f.harness.coordinator.run(1)).await.unwrap(); + let chunk = f + .harness + .coordinator + .observation() + .expect("a restored world") + .audio + .first() + .cloned() + .expect("one chunk per transition"); + assert!( + chunk.discontinuity, + "the first chunk of a fresh epoch marks the discontinuity the recovery established" + ); + assert_eq!( + chunk.first_sample, + positions["arena"], + "and it starts where the checkpointed stream stopped" + ); + f.shutdown().await; +} + +async fn a_checkpoint_taken_by_another_parser_is_refused_by_name(via: Via) { + another_parser_is_refused(via, ExecutionMode::InProcess).await; +} + +async fn another_parser_is_refused_in_every_mode(mode: ExecutionMode) { + another_parser_is_refused(Via::Unix, mode).await; +} + +/// A checkpoint whose recorded parser identity is not this composition's is refused before a +/// single participant is asked to stage, and the refusal names the identity that differs. +async fn another_parser_is_refused(via: Via, mode: ExecutionMode) { + let mut f = fx(via, mode, HarnessConfig::default()).await; + let checkpoint_id = ckpt(1); + run_and_checkpoint(&mut f, 1, &checkpoint_id).await; + let root = f.harness.checkpoint_root().to_path_buf(); + fail_the_epoch(&mut f).await; + rewrite_generation(&root, &checkpoint_id, |manifest, _payloads| { + manifest["compatibility"]["parserDigest"] = + json!(digest_of_bytes(b"some other inspection schema")); + }); + reload_store(&mut f).await; + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let failure = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e2")), + ) + .await + .expect_err("another parser's state is not this composition's"); + assert_eq!(failure.error.code, ErrorCode::IncompatibleState); + assert!( + failure.error.message.contains("parser"), + "the refusal names the identity that differs: {failure}" + ); + assert!( + f.harness.coordinator.tainted().is_empty(), + "nothing was asked to stage, so nothing has to be replaced" + ); + nothing_is_installed(&mut f, &id("e2")).await; + f.shutdown().await; +} + +// =============================================================================================== +// Failure-injection rows + +async fn a_group_where_one_participant_will_not_stage_resumes_nothing(via: Via) { + a_refused_stage(via, ExecutionMode::InProcess).await; +} + +async fn a_refused_stage_resumes_nothing_in_every_mode(mode: ExecutionMode) { + a_refused_stage(Via::Unix, mode).await; +} + +/// The checklist row: the install validates the participants ahead of it and the last one +/// fails. Nothing is resumed. +async fn a_refused_stage(via: Via, mode: ExecutionMode) { + let mut f = fx(via, mode, HarnessConfig::default()).await; + let checkpoint_id = ckpt(1); + run_and_checkpoint(&mut f, 1, &checkpoint_id).await; + fail_the_epoch(&mut f).await; + f.harness.set_agent_faults( + &fly_b(), + AgentFaults { fail_stage_restore: true, ..AgentFaults::default() }, + ); + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let failure = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e2")), + ) + .await + .expect_err("a participant that will not validate stops the install"); + assert_eq!(failure.error.code, ErrorCode::IncompatibleState); + assert_eq!(failure.participant.as_deref(), Some(fly_b().as_str())); + assert_eq!( + count(&f.harness.coordinator.audit, &format!("activated:{}", fly_a())), + 0, + "no participant activates when one of them will not stage" + ); + nothing_is_installed(&mut f, &id("e2")).await; + f.shutdown().await; +} + +async fn a_restore_token_activates_only_once(via: Via) { + a_token_activates_once(via, ExecutionMode::InProcess).await; +} + +async fn a_token_activates_once_in_every_mode(mode: ExecutionMode) { + a_token_activates_once(Via::Unix, mode).await; +} + +/// A token is bound to its checkpoint, scope and payload and activates once. A fresh request +/// naming it again is a conflict, and the old epoch's scope is stale on the new participants. +async fn a_token_activates_once(via: Via, mode: ExecutionMode) { + let mut f = fx(via, mode, HarnessConfig::default()).await; + let checkpoint_id = ckpt(1); + run_and_checkpoint(&mut f, 1, &checkpoint_id).await; + fail_the_epoch(&mut f).await; + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let report = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e2")), + ) + .await + .unwrap(); + let (who, token) = report.tokens.first().cloned().expect("a staged token"); + let worker = if who == f.harness.environment_id() { + f.harness.coordinator.environment_ref().clone() + } else { + f.harness.coordinator.agent_ref(&who).expect("a participant").clone() + }; + let scope = scope_at("demo", "e2", 1); + let refused = within( + "activate", + f.harness.coordinator.probe_raw( + &worker, + "State.ActivateRestore", + Some(scope), + json!({"restoreToken": token.as_str()}), + ), + ) + .await + .expect_err("a token activates once"); + assert_eq!(refused.code, ErrorCode::Conflict); + + // And the epoch the restore left behind is stale on every replacement. + let stale = within( + "prepare", + f.harness.coordinator.probe_raw( + &worker, + if who == f.harness.environment_id() { "Environment.Advance" } else { "Agent.Prepare" }, + Some(scope_at("demo", "e1", 1)), + json!({}), + ), + ) + .await + .expect_err("the old epoch is gone"); + assert!( + matches!(stale.code, ErrorCode::StaleEpoch | ErrorCode::InvalidArgument), + "an old-epoch request is refused: {stale}" + ); + f.shutdown().await; +} + +async fn the_fence_lifts_only_through_a_coherent_restore(via: Via) { + the_fence_lifts_only_by_restore(via, ExecutionMode::InProcess).await; +} + +async fn the_fence_lifts_only_by_restore_in_every_mode(mode: ExecutionMode) { + the_fence_lifts_only_by_restore(Via::Unix, mode).await; +} + +/// Nothing but a coherent restore moves a fenced session, and the restore lands on +/// `Paused(k)` rather than straight back into play. +async fn the_fence_lifts_only_by_restore(via: Via, mode: ExecutionMode) { + let mut f = fx(via, mode, HarnessConfig::default()).await; + let checkpoint_id = ckpt(1); + run_and_checkpoint(&mut f, 1, &checkpoint_id).await; + fail_the_epoch(&mut f).await; + + for refused in [ + within("step", f.harness.coordinator.step()).await.err(), + within("bootstrap", f.harness.coordinator.bootstrap()).await.err(), + within("capture", f.harness.coordinator.capture(&ckpt(9), false)) + .await + .err(), + ] { + let refused = refused.expect("a fenced session refuses"); + assert_eq!(refused.error.code, ErrorCode::InvalidPhase); + } + assert!(f.harness.coordinator.resume().is_err(), "a fenced session does not resume"); + assert!(f.harness.coordinator.is_fenced()); + + // A restore into the epoch that failed is refused: recovery establishes a fresh one. + within("replace", f.harness.replace_all_participants()).await.unwrap(); + let same = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e1")), + ) + .await + .expect_err("a restore installs a fresh epoch"); + assert_eq!(same.error.code, ErrorCode::StaleEpoch); + + let report = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint_id), &id("e2")), + ) + .await + .unwrap(); + assert_eq!(report.boundary, 1); + assert!(!f.harness.coordinator.is_fenced()); + assert_eq!(f.harness.coordinator.phase(), Phase::Paused(1)); + f.harness.coordinator.resume().unwrap(); + within("resume", f.harness.coordinator.run(1)).await.unwrap(); + assert_eq!(f.harness.coordinator.phase(), Phase::Ready(2)); + f.shutdown().await; +} + +/// A capture belongs to a committed boundary, and the phase machine is what says so. +async fn a_capture_is_refused_anywhere_but_a_committed_boundary(via: Via) { + let mut f = fx(via, ExecutionMode::InProcess, HarnessConfig::default()).await; + // Before bootstrap the session is Starting, which is not a boundary at all. + let refused = within("capture", f.harness.coordinator.capture(&ckpt(1), false)) + .await + .expect_err("Starting is not a committed boundary"); + assert_eq!(refused.error.code, ErrorCode::InvalidPhase); + f.shutdown().await; + + // From a pause, which is a committed boundary, it works, and it returns there. + let mut f = fx(via, ExecutionMode::InProcess, HarnessConfig::default()).await; + within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap(); + within("run", f.harness.coordinator.run(1)).await.unwrap(); + f.harness.coordinator.request_pause(); + within("run", f.harness.coordinator.run(1)).await.unwrap(); + assert_eq!(f.harness.coordinator.phase(), Phase::Paused(2)); + let outcome = within("checkpoint", f.harness.coordinator.checkpoint(&ckpt(1))) + .await + .unwrap(); + assert!(matches!(outcome, SaveOutcome::Committed { boundary: 2, .. }), "{outcome:?}"); + assert_eq!( + f.harness.coordinator.phase(), + Phase::Paused(2), + "a capture returns to the boundary it came from" + ); + f.shutdown().await; +} From 2cc8d0294d46f647777cb9bba33d138f653c6c23 Mon Sep 17 00:00:00 2001 From: dev Date: Tue, 22 Sep 2026 18:28:54 +0000 Subject: [PATCH 5/8] docs: the two sweep flakes are fixed, and one assertion that could not fail bus-conformance.md still listed the example_demo root count and session_over_one_router as not fixed here. Both are fixed on main now, so the rows say what each test does instead: the example waits for the producer's hold release before reading the counts, with its printed output unchanged, and the integration renderer is held until the publisher's twentieth receipt has returned, so its coalescing is forced and the assertions are order, freshness, acceptance under a stalled spectator and the replacement accounting, with the before and after counts. The delivery check in that test compared (step, sequence) against (step, step), whose first element can never fail. It compares sequence against step. --- .../session-framework/bus-conformance.md | 28 +++++++++++++------ .../flysim/crates/flybus/tests/integration.rs | 3 +- 2 files changed, 21 insertions(+), 10 deletions(-) diff --git a/docs/design/session-framework/bus-conformance.md b/docs/design/session-framework/bus-conformance.md index 5c52155..83e2114 100644 --- a/docs/design/session-framework/bus-conformance.md +++ b/docs/design/session-framework/bus-conformance.md @@ -438,18 +438,30 @@ poll-until-the-router-settles the rest of the file already uses for router-side no sleep, no timing constant, and the assertion now has the precondition its contract sentence names. **360 runs after the fix, 0 failures** (240 debug, 120 release). -Two other intermittent failures were seen in the same sweep and are **not** fixed here, since -they belong to the bus slice rather than to this one: +Two other intermittent failures were seen in the same sweep. They belonged to the bus slice +rather than to this one and were fixed there, in the same way and for the same reason: - `tests/example_demo.rs::the_example_shows_a_counter_rpc_an_observer_and_a_held_frame`, - 2 failures in 40 standalone runs plus 1 in 12 full-suite runs. It prints + 2 failures in 40 standalone runs plus 1 in 12 full-suite runs. It printed "while the frame is held: 1 artifact(s), 2 root(s)" instead of 1 root: the producer's hold release is queued on the control lane and had not been applied when the example read the - counts. The same shape of gap, in the guide deliverable's printed output. -- `tests/integration.rs::unix_socket::session_over_one_router`, 1 failure in 12 full-suite - runs and 0 in 40 standalone runs, at the assertion that the deliberately slow consumer - skipped snapshots. Under load it kept up, so the assertion is a timing claim about the - machine. + counts. `examples/demo.rs` now waits for that release before reading the counts, the same + bounded poll it already used for collection eight lines below, so the line the guide quotes + is an observation rather than a race. The printed output is unchanged. +- `tests/integration.rs::session_over_one_router`, 1 failure in 12 full-suite runs and 0 in 40 + standalone runs, at the assertion that the deliberately slow consumer skipped snapshots. + Under load it kept up, so the assertion was a timing claim about the machine: section 7 + permits a latest subscriber to miss values, it does not oblige it to. The renderer is now + held until the publisher's twentieth receipt has returned -- the publisher's completion + observed, not timed -- so the coalescing is forced by construction, and the test asserts the + guarantees that do hold: each delivery carries the frame of the snapshot it announces, + deliveries arrive in publication order, the last value received is the latest published, the + renderer receives snapshots 1 and 20 of 20, both subscriptions accept all twenty publications + while the spectator reads nothing, and the eighteen replacements reported to the publisher at + admission are the same eighteen reported to the renderer on delivery and are exactly the + snapshots it did not receive. 16 failures in 40 runs beside four busy loops before, 0 in 40 + after; the whole crate went from 10 failed runs in 20 to 0, and the workspace suite from 2 + in 5 to 0. ## Contradictions diff --git a/services/flysim/crates/flybus/tests/integration.rs b/services/flysim/crates/flybus/tests/integration.rs index f18136b..b658f63 100644 --- a/services/flysim/crates/flybus/tests/integration.rs +++ b/services/flysim/crates/flybus/tests/integration.rs @@ -168,8 +168,7 @@ async fn session_over_one_router(via: Via) { let bytes = frame.read_all().await.unwrap(); let step = bytes[0] as u64; assert_eq!( - (step, sequence), - (step, step), + sequence, step, "a delivery carries the frame of the snapshot it announces" ); coalesced += replaced; From 55221a7c77be0bae51f7fc21948ee391195d735f Mon Sep 17 00:00:00 2001 From: acamilo Date: Tue, 22 Sep 2026 18:41:17 +0000 Subject: [PATCH 6/8] docs: v0.5.0 status --- docs/stream-mvp-plan.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/stream-mvp-plan.md b/docs/stream-mvp-plan.md index 29404f0..6588bfc 100644 --- a/docs/stream-mvp-plan.md +++ b/docs/stream-mvp-plan.md @@ -736,3 +736,11 @@ rewritten separately. (73/73) because 82% of the fixed run is battle time; Fable shipped it on the same judgement as v0.4.6 and started row 50 (MOVE n blocked on an unresponsive move list). The on-screen chat ring now survives a sim restart (sidecar in the hot dir, never in the checkpoint). +- 2026-09-22 (v0.5.0, the operator's decision): the fly is paid for keeping a wild Pokémon. New + catalog kind `catch` (0.30 for a species this run never caught, 0.10 for a repeat, three payouts + per species), read from the captured-species byte and the battle result together; the existing + species rule still pays on top. Adapter `pokered-unique8-v6`; the compatibility string differs + in the adapter segment only, and a deploy with `FLY_ACCEPT_ADAPTERS=pokered-unique8-v5` migrates + a v5 checkpoint instead of refusing it. `fly-reset-to-milestone ` restarts the run from a + ladder rung (archives both stores first). The live run restarts from rung 7 with this release, so + the ladder is climbed again with the catch reward and the row-54 walks in place. From 45621903be44096d98293da8b2a66cc6508efae2 Mon Sep 17 00:00:00 2001 From: dev Date: Tue, 22 Sep 2026 18:45:51 +0000 Subject: [PATCH 7/8] session: name the two ways a durable wait ends without an acknowledgment Review follow-up on the checkpoint store. A dropped reply channel and an expired caller budget were both reported as ReplyLost. They are different facts -- the first means the write is over and its outcome did not reach here, the second means the save is still going -- so they are now separate outcomes, and the durable wait has its own budget rather than borrowing the one that bounds a call to a participant. Both still leave durable metadata where it was, and for both the resolution asks the store about the same checkpoint. The writer's two bounds refuse at different moments and the comment claimed otherwise: the outstanding-capture bound is taken before a capture is requested, and the byte budget cannot be, because a capture's size is not known until it exists. The byte check, the decision and the change to the byte total are now one critical section, the peak is sampled after a superseded job's bytes are gone, and the writer's own bookkeeping is over a type that holds only the outcomes a writer can produce. The manifest's coordinator.eventWatermarks is {lastSourceStep, issued}; the fixture illustrated {lastEventId, lastOrdinal}, and the illustration is what changed, because an event id is derived from the epoch and cannot be compared across the restore that gives the session a new one. checkpoint-envelope-v1 section 3 also now says, under the same dated amendment, that a required-manifest-field change must bump envelopeVersion once production files exist: contractDigest is taken over the schema set and does not cover this manifest, so the envelope version is the only thing that can carry such a change. --- .../checkpoint-envelope-v1.md | 12 ++ .../examples/update_fixtures.rs | 2 +- .../fixtures/checkpoint-envelope.json | 32 ++-- services/flysim/crates/fly-session/README.md | 18 +- .../crates/fly-session/src/coordinator.rs | 16 +- .../flysim/crates/fly-session/src/state.rs | 168 ++++++++++++------ .../crates/fly-session/tests/processes.rs | 3 + .../flysim/crates/fly-session/tests/state.rs | 57 +++++- 8 files changed, 229 insertions(+), 79 deletions(-) diff --git a/docs/design/session-framework/checkpoint-envelope-v1.md b/docs/design/session-framework/checkpoint-envelope-v1.md index 7cb833f..d58ed98 100644 --- a/docs/design/session-framework/checkpoint-envelope-v1.md +++ b/docs/design/session-framework/checkpoint-envelope-v1.md @@ -118,6 +118,18 @@ section has listed from the start. Both are now in `REQUIRED_MANIFEST_FIELDS` in TypeScript, and the fixture was regenerated by the existing example. The schema set is untouched, so `contractDigest` is unchanged. +`coordinator.eventWatermarks` is `{lastSourceStep, issued}`. The fixture illustrated +`{lastEventId, lastOrdinal}`, and it is the illustration that changed: an event id is derived +from the epoch, so a watermark spelled as one cannot be compared across the restore that +gives the session a new epoch, while a source step and an issued count can. + +**A required-manifest-field change is compatibility-relevant and `contractDigest` does not +cover it.** The digest is taken over the schema set, and this manifest is not in it, so +`envelopeVersion` is the only thing that can carry such a change. It stays `1` here only +because no production `FLYSESS1` file exists yet: once one does, adding or removing a required +manifest field **must** bump `envelopeVersion`, because a reader of the older version would +otherwise accept a file it cannot completely read, or refuse one it could. + `payloads` is redundant with the table on purpose: the table is what a reader needs to map bytes, and the manifest is what a store lists, compares and reports without opening the payload area. A reader checks that the two agree. diff --git a/services/flysim/crates/fly-session-types/examples/update_fixtures.rs b/services/flysim/crates/fly-session-types/examples/update_fixtures.rs index d68b865..bad3693 100644 --- a/services/flysim/crates/fly-session-types/examples/update_fixtures.rs +++ b/services/flysim/crates/fly-session-types/examples/update_fixtures.rs @@ -197,7 +197,7 @@ fn checkpoint_envelope() -> String { "priorInspection": "prior-inspection", "executorState": [{"agentId": "fly-a", "payload": "executor-fly-a"}], "admissionState": null, - "eventWatermarks": {"lastEventId": "evt-1", "lastOrdinal": "7"}, + "eventWatermarks": {"lastSourceStep": "42", "issued": "7"}, }, "environment": {"workerId": "arena", "payload": "world"}, "helperState": [], diff --git a/services/flysim/crates/fly-session-types/fixtures/checkpoint-envelope.json b/services/flysim/crates/fly-session-types/fixtures/checkpoint-envelope.json index 70453c3..b726585 100644 --- a/services/flysim/crates/fly-session-types/fixtures/checkpoint-envelope.json +++ b/services/flysim/crates/fly-session-types/fixtures/checkpoint-envelope.json @@ -59,8 +59,8 @@ ], "admissionState": null, "eventWatermarks": { - "lastEventId": "evt-1", - "lastOrdinal": "7" + "lastSourceStep": "42", + "issued": "7" } }, "environment": { @@ -119,49 +119,49 @@ } ], "envelope": { - "base64": "RkxZU0VTUzEBAAAAIAAAADUIAAAFAAAAWAgAAAAAAAB7ImFnZW50cyI6W3siYWdlbnRJZCI6ImZseS1hIiwiYnJhaW5UaWNrcyI6IjI1MzQiLCJkYXRhc2V0RGlnZXN0IjoiNmMwYWYxZjA3ODRlZjYzYTM5M2VlNzdkNjE0ZTgyNDZjNjI1MDUxMzYwZjNmMWE0ODgzODM3NGM1ZDM1NWI1MiIsIm1vZGVsVmVyc2lvbiI6ImxpZi0xbXMtZjY0LXYyIiwicGF5bG9hZCI6ImFnZW50LWZseS1hIiwicGxhc3RpY2l0eVZlcnNpb24iOiJmbHkta2MtbWJvbi1yc3RkcC12MiIsInByb2ZpbGVEaWdlc3QiOiIxOTAwZWFiNmMwMjg0ODNkNzEyNjU5OWVlNmY1MGRlMGQyNzkwN2I1YzY1ZmE5MDUyNDU4MGI0YjBmOTg1MmIwIiwicmVtYWluZGVyIjp7ImRlbm9taW5hdG9yIjoiMyIsIm51bWVyYXRvciI6IjEwMDAwMDAifSwic2VlZCI6LTE4NDk0NjA2M31dLCJjaGVja3BvaW50SWQiOiJja3B0LTEiLCJjb21wYXRpYmlsaXR5Ijp7ImJhY2tlbmREaWdlc3QiOiIxMGUwOGE0MTllODUwZWJhMWViYmExOGZkZDI4ZWI3ZWMxYjdlOGJhYTliY2MzYjk3M2UyYjg4OTFlYzcyNmJlIiwiY29udGVudERpZ2VzdCI6ImVkNzAwMmI0MzllOWFjODQ1ZjIyMzU3ZDgyMmJhYzE0NDQ3MzBmYmRiNjAxNmQzZWM5NDMyMjk3YjllYzlmNzMiLCJjb250cm9sbGVyRGlnZXN0IjoiYzE0NzIxMzViMTRjNzdjOGJlZjk4ZTczZjcwMjA4MzI1ZmEwZGNmMWU2YmQ2NjhhZTliMzFhOWNlYTI5NWZlNyIsInBhcnNlckRpZ2VzdCI6ImIxN2Q0NTEyMTE1MDkyOGYyMTQ2YWY0OWUxOTVlZmYxZWVmNWQ2NzMyNWJlMjczYTczM2ZiNzRhY2FkYWEzNDIiLCJwYXRjaERpZ2VzdCI6ImE0ODk1ZWI0NGFmYzMzNmZlY2JiYTZlNTIwY2Q2N2UxNzhkYWNlMDI3NjY1NWQxMDJmY2VmZmE4ZTVmNzA1NzAiLCJzdGF0ZUZvcm1hdElkIjoiZmx5c2Vzcy0xIn0sImNvbXBvc2l0aW9uRGlnZXN0IjoiNzMwZDcyNWM4YTU5ZDNhNzMwM2RlZjJiZWQwNDFhNTc3ZWRiNDI1NWFhYmQ0ODg5Y2UxMjkxODMxMWQ5NTJmMCIsImNvb3JkaW5hdG9yIjp7ImFkbWlzc2lvblN0YXRlIjpudWxsLCJldmVudFdhdGVybWFya3MiOnsibGFzdEV2ZW50SWQiOiJldnQtMSIsImxhc3RPcmRpbmFsIjoiNyJ9LCJleGVjdXRvclN0YXRlIjpbeyJhZ2VudElkIjoiZmx5LWEiLCJwYXlsb2FkIjoiZXhlY3V0b3ItZmx5LWEifV0sInByaW9ySW5zcGVjdGlvbiI6InByaW9yLWluc3BlY3Rpb24iLCJ0YXNrTGVkZ2VyIjoidGFzay1sZWRnZXIifSwiZW52ZWxvcGVWZXJzaW9uIjoxLCJlbnZpcm9ubWVudCI6eyJwYXlsb2FkIjoid29ybGQiLCJ3b3JrZXJJZCI6ImFyZW5hIn0sImVwaXNvZGVJZCI6ImVwaXNvZGUtMSIsImhlbHBlclN0YXRlIjpbXSwicGF5bG9hZHMiOlt7ImJ5dGVMZW5ndGgiOiIxNyIsImRpZ2VzdCI6IjEzMjFkZmZiMGNkYzZmOTA5MmNiZjdmYTJhNWZjNjhiYmVkMTJjOTkzZDVhZDM5ODI2NDAxMjgxMGNlOWJmOTMiLCJuYW1lIjoiYWdlbnQtZmx5LWEifSx7ImJ5dGVMZW5ndGgiOiIxNCIsImRpZ2VzdCI6IjNhZWU2MGRmN2UyOWVmZWJhN2Y1Zjk5ZmM1ODY3NjQ3YjM2YWViZmYxZDVkM2M4MzhkYmZmMzIzMTIyZTY0NjIiLCJuYW1lIjoiZXhlY3V0b3ItZmx5LWEifSx7ImJ5dGVMZW5ndGgiOiIxMSIsImRpZ2VzdCI6IjQwYjAwZWQyYmJiYTkwMWQ2ODIwNWZmNzFiMDRhNDRiOWVlNTNjNTFjYjMxMDlhYTJjZWFhNDRmMWM0NTcyN2UiLCJuYW1lIjoidGFzay1sZWRnZXIifSx7ImJ5dGVMZW5ndGgiOiIxMCIsImRpZ2VzdCI6IjJjMTNiN2I0ZDlhOTkxNjgwMWFiOTE5MWMzMTRmMzFiMDQ1ZTliOWM1YjY2OWE2YzA0NzRmMDIxN2VmNzViZjUiLCJuYW1lIjoicHJpb3ItaW5zcGVjdGlvbiJ9LHsiYnl0ZUxlbmd0aCI6IjY0IiwiZGlnZXN0IjoiZjVhNWZkNDJkMTZhMjAzMDI3OThlZjZlZDMwOTk3OWI0MzAwM2QyMzIwZDlmMGU4ZWE5ODMxYTkyNzU5ZmI0YiIsIm5hbWUiOiJ3b3JsZCJ9XSwicG9ydE1hcCI6W3siYWdlbnRJZCI6ImZseS1hIiwicG9ydElkIjoicG9ydC0xIn1dLCJzY2hlZHVsZXJJZCI6ImxvY2tzdGVwLXYxIiwic291cmNlU2NvcGUiOnsiZXBvY2giOiJlcG9jaC0xIiwic2Vzc2lvbklkIjoiZGVtbyIsInN0ZXAiOiI0MiJ9LCJ3b3JsZFRpbWUiOnsiZGVub21pbmF0b3IiOiIxIiwibnVtZXJhdG9yIjoiNzAwMDAwMDAwIn19AAAAYWdlbnQtZmx5LWEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAIgKAAAAAAAAEQAAAAAAAAATId/7DNxvkJLL9/oqX8aLvtEsmT1a05gmQBKBDOm/k2V4ZWN1dG9yLWZseS1hAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACgCgAAAAAAAA4AAAAAAAAAOu5g334p7+un9fmfxYZ2R7Nq6/8dXTyDjb/zIxIuZGJ0YXNrLWxlZGdlcgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAsAoAAAAAAAALAAAAAAAAAECwDtK7upAdaCBf9xsEpEue5TxRyzEJqizqpE8cRXJ+cHJpb3ItaW5zcGVjdGlvbgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMAKAAAAAAAACgAAAAAAAAAsE7e02amRaAGrkZHDFPMbBF6bnFtmmmwEdPAhfvdb9XdvcmxkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADQCgAAAAAAAEAAAAAAAAAA9aX9QtFqIDAnmO9u0wmXm0MAPSMg2fDo6pgxqSdZ+0thZ2VudCBzdGF0ZSBieXRlcwAAAAAAAABleGVjdXRvciBzdGF0ZQAAeyJyYW5rIjoxMH0AAAAAAHsibWFwIjo0MH0AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAsAAAAAAAAhBlh9AWLTtVKckmeIzNn4DO5Yhn2C1nUA1T1RfxMXlkZMWVNFU1NG", - "byteLength": 2880, + "base64": "RkxZU0VTUzEBAAAAIAAAADAIAAAFAAAAUAgAAAAAAAB7ImFnZW50cyI6W3siYWdlbnRJZCI6ImZseS1hIiwiYnJhaW5UaWNrcyI6IjI1MzQiLCJkYXRhc2V0RGlnZXN0IjoiNmMwYWYxZjA3ODRlZjYzYTM5M2VlNzdkNjE0ZTgyNDZjNjI1MDUxMzYwZjNmMWE0ODgzODM3NGM1ZDM1NWI1MiIsIm1vZGVsVmVyc2lvbiI6ImxpZi0xbXMtZjY0LXYyIiwicGF5bG9hZCI6ImFnZW50LWZseS1hIiwicGxhc3RpY2l0eVZlcnNpb24iOiJmbHkta2MtbWJvbi1yc3RkcC12MiIsInByb2ZpbGVEaWdlc3QiOiIxOTAwZWFiNmMwMjg0ODNkNzEyNjU5OWVlNmY1MGRlMGQyNzkwN2I1YzY1ZmE5MDUyNDU4MGI0YjBmOTg1MmIwIiwicmVtYWluZGVyIjp7ImRlbm9taW5hdG9yIjoiMyIsIm51bWVyYXRvciI6IjEwMDAwMDAifSwic2VlZCI6LTE4NDk0NjA2M31dLCJjaGVja3BvaW50SWQiOiJja3B0LTEiLCJjb21wYXRpYmlsaXR5Ijp7ImJhY2tlbmREaWdlc3QiOiIxMGUwOGE0MTllODUwZWJhMWViYmExOGZkZDI4ZWI3ZWMxYjdlOGJhYTliY2MzYjk3M2UyYjg4OTFlYzcyNmJlIiwiY29udGVudERpZ2VzdCI6ImVkNzAwMmI0MzllOWFjODQ1ZjIyMzU3ZDgyMmJhYzE0NDQ3MzBmYmRiNjAxNmQzZWM5NDMyMjk3YjllYzlmNzMiLCJjb250cm9sbGVyRGlnZXN0IjoiYzE0NzIxMzViMTRjNzdjOGJlZjk4ZTczZjcwMjA4MzI1ZmEwZGNmMWU2YmQ2NjhhZTliMzFhOWNlYTI5NWZlNyIsInBhcnNlckRpZ2VzdCI6ImIxN2Q0NTEyMTE1MDkyOGYyMTQ2YWY0OWUxOTVlZmYxZWVmNWQ2NzMyNWJlMjczYTczM2ZiNzRhY2FkYWEzNDIiLCJwYXRjaERpZ2VzdCI6ImE0ODk1ZWI0NGFmYzMzNmZlY2JiYTZlNTIwY2Q2N2UxNzhkYWNlMDI3NjY1NWQxMDJmY2VmZmE4ZTVmNzA1NzAiLCJzdGF0ZUZvcm1hdElkIjoiZmx5c2Vzcy0xIn0sImNvbXBvc2l0aW9uRGlnZXN0IjoiNzMwZDcyNWM4YTU5ZDNhNzMwM2RlZjJiZWQwNDFhNTc3ZWRiNDI1NWFhYmQ0ODg5Y2UxMjkxODMxMWQ5NTJmMCIsImNvb3JkaW5hdG9yIjp7ImFkbWlzc2lvblN0YXRlIjpudWxsLCJldmVudFdhdGVybWFya3MiOnsiaXNzdWVkIjoiNyIsImxhc3RTb3VyY2VTdGVwIjoiNDIifSwiZXhlY3V0b3JTdGF0ZSI6W3siYWdlbnRJZCI6ImZseS1hIiwicGF5bG9hZCI6ImV4ZWN1dG9yLWZseS1hIn1dLCJwcmlvckluc3BlY3Rpb24iOiJwcmlvci1pbnNwZWN0aW9uIiwidGFza0xlZGdlciI6InRhc2stbGVkZ2VyIn0sImVudmVsb3BlVmVyc2lvbiI6MSwiZW52aXJvbm1lbnQiOnsicGF5bG9hZCI6IndvcmxkIiwid29ya2VySWQiOiJhcmVuYSJ9LCJlcGlzb2RlSWQiOiJlcGlzb2RlLTEiLCJoZWxwZXJTdGF0ZSI6W10sInBheWxvYWRzIjpbeyJieXRlTGVuZ3RoIjoiMTciLCJkaWdlc3QiOiIxMzIxZGZmYjBjZGM2ZjkwOTJjYmY3ZmEyYTVmYzY4YmJlZDEyYzk5M2Q1YWQzOTgyNjQwMTI4MTBjZTliZjkzIiwibmFtZSI6ImFnZW50LWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTQiLCJkaWdlc3QiOiIzYWVlNjBkZjdlMjllZmViYTdmNWY5OWZjNTg2NzY0N2IzNmFlYmZmMWQ1ZDNjODM4ZGJmZjMyMzEyMmU2NDYyIiwibmFtZSI6ImV4ZWN1dG9yLWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTEiLCJkaWdlc3QiOiI0MGIwMGVkMmJiYmE5MDFkNjgyMDVmZjcxYjA0YTQ0YjllZTUzYzUxY2IzMTA5YWEyY2VhYTQ0ZjFjNDU3MjdlIiwibmFtZSI6InRhc2stbGVkZ2VyIn0seyJieXRlTGVuZ3RoIjoiMTAiLCJkaWdlc3QiOiIyYzEzYjdiNGQ5YTk5MTY4MDFhYjkxOTFjMzE0ZjMxYjA0NWU5YjljNWI2NjlhNmMwNDc0ZjAyMTdlZjc1YmY1IiwibmFtZSI6InByaW9yLWluc3BlY3Rpb24ifSx7ImJ5dGVMZW5ndGgiOiI2NCIsImRpZ2VzdCI6ImY1YTVmZDQyZDE2YTIwMzAyNzk4ZWY2ZWQzMDk5NzliNDMwMDNkMjMyMGQ5ZjBlOGVhOTgzMWE5Mjc1OWZiNGIiLCJuYW1lIjoid29ybGQifV0sInBvcnRNYXAiOlt7ImFnZW50SWQiOiJmbHktYSIsInBvcnRJZCI6InBvcnQtMSJ9XSwic2NoZWR1bGVySWQiOiJsb2Nrc3RlcC12MSIsInNvdXJjZVNjb3BlIjp7ImVwb2NoIjoiZXBvY2gtMSIsInNlc3Npb25JZCI6ImRlbW8iLCJzdGVwIjoiNDIifSwid29ybGRUaW1lIjp7ImRlbm9taW5hdG9yIjoiMSIsIm51bWVyYXRvciI6IjcwMDAwMDAwMCJ9fWFnZW50LWZseS1hAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACACgAAAAAAABEAAAAAAAAAEyHf+wzcb5CSy/f6Kl/Gi77RLJk9WtOYJkASgQzpv5NleGVjdXRvci1mbHktYQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAmAoAAAAAAAAOAAAAAAAAADruYN9+Ke/rp/X5n8WGdkezauv/HV08g42/8yMSLmRidGFzay1sZWRnZXIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKgKAAAAAAAACwAAAAAAAABAsA7Su7qQHWggX/cbBKRLnuU8UcsxCaos6qRPHEVyfnByaW9yLWluc3BlY3Rpb24AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC4CgAAAAAAAAoAAAAAAAAALBO3tNmpkWgBq5GRwxTzGwRem5xbZppsBHTwIX73W/V3b3JsZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAyAoAAAAAAABAAAAAAAAAAPWl/ULRaiAwJ5jvbtMJl5tDAD0jINnw6OqYMaknWftLYWdlbnQgc3RhdGUgYnl0ZXMAAAAAAAAAZXhlY3V0b3Igc3RhdGUAAHsicmFuayI6MTB9AAAAAAB7Im1hcCI6NDB9AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADgLAAAAAAAAX4L9WkdX0MViD3h5YJgf7VQgwocWU4XJoKX6MUwl+8hGTFlTRVNTRg==", + "byteLength": 2872, "layout": { "headerBytes": 32, "manifestOffset": "32", - "manifestBytes": 2101, - "tableOffset": "2136", + "manifestBytes": 2096, + "tableOffset": "2128", "tableEntryBytes": 112, "entries": [ { "name": "agent-fly-a", - "offset": "2696", + "offset": "2688", "byteLength": "17", "digest": "1321dffb0cdc6f9092cbf7fa2a5fc68bbed12c993d5ad398264012810ce9bf93" }, { "name": "executor-fly-a", - "offset": "2720", + "offset": "2712", "byteLength": "14", "digest": "3aee60df7e29efeba7f5f99fc5867647b36aebff1d5d3c838dbff323122e6462" }, { "name": "task-ledger", - "offset": "2736", + "offset": "2728", "byteLength": "11", "digest": "40b00ed2bbba901d68205ff71b04a44b9ee53c51cb3109aa2ceaa44f1c45727e" }, { "name": "prior-inspection", - "offset": "2752", + "offset": "2744", "byteLength": "10", "digest": "2c13b7b4d9a9916801ab9191c314f31b045e9b9c5b669a6c0474f0217ef75bf5" }, { "name": "world", - "offset": "2768", + "offset": "2760", "byteLength": "64", "digest": "f5a5fd42d16a20302798ef6ed309979b43003d2320d9f0e8ea9831a92759fb4b" } ], - "footerOffset": "2832", + "footerOffset": "2824", "footerBytes": 48, - "totalBytes": "2880" + "totalBytes": "2872" } }, "corruption": [ @@ -182,17 +182,17 @@ }, { "name": "a flipped payload byte", - "offset": 2696, + "offset": 2688, "reason": "every payload carries its own digest" }, { "name": "a flipped footer digest byte", - "offset": 2840, + "offset": 2832, "reason": "the footer digest must match the contents" }, { "name": "a flipped footer magic byte", - "offset": 2872, + "offset": 2864, "reason": "a truncated file cannot look complete" } ] diff --git a/services/flysim/crates/fly-session/README.md b/services/flysim/crates/fly-session/README.md index 33f0594..76b8b72 100644 --- a/services/flysim/crates/fly-session/README.md +++ b/services/flysim/crates/fly-session/README.md @@ -189,12 +189,18 @@ The durable store is `state`, over the `FLYSESS1` layout the contract crate owns `BUSY` a stepping session survives rather than an epoch failure. - **Capture and durability are two events.** `State.Capture` completes when an immutable capture exists; `Coordinator::await_durable` completes when the store manifest rename has - happened, which is the durable commit point. Only the second moves the durable mark. A lost - save reply is `SaveOutcome::ReplyLost`, and `Coordinator::resolve_durable` then asks the - store about the *same* checkpoint instead of saving again. -- **The writer is bounded twice**, by outstanding captures and by queued bytes, and it owns - its payload handles until the bytes are committed or the job fails. A queued *replaceable* - capture is superseded by a later one, releasing its holds; a durable one never is. + happened, which is the durable commit point. Only the second moves the durable mark, and the + three ways it can end without one are told apart: `Failed` (the write stopped), + `ReplyLost` (the write finished and the acknowledgment did not arrive) and + `DeadlineExpired` (the caller's own budget ran out while the save was still going). + `Coordinator::resolve_durable` then asks the store about the *same* checkpoint instead of + saving again. +- **The writer is bounded twice**, and the two bounds refuse at different moments. The + outstanding-capture bound is taken before a capture is requested; the byte budget cannot be, + because a capture's size is not known until it exists, so it refuses at submit and releases + the payloads with the refusal. The writer owns its payload handles until the bytes are + committed or the job fails. A queued *replaceable* capture is superseded by a later one, + releasing its holds; a durable one never is. - **The install is a group.** A restore selects a complete compatible generation, imports its payloads as fresh artifacts, stages every participant, validates the coordinator's own ledgers, and only then activates. A failure anywhere leaves the fence closed, and every diff --git a/services/flysim/crates/fly-session/src/coordinator.rs b/services/flysim/crates/fly-session/src/coordinator.rs index bde1183..f01d72b 100644 --- a/services/flysim/crates/fly-session/src/coordinator.rs +++ b/services/flysim/crates/fly-session/src/coordinator.rs @@ -145,6 +145,12 @@ pub struct Deadlines { /// a capture serializes a participant and a restore validates and installs one, and /// neither is a step whose latency the probe was chosen for. pub capture: Duration, + /// How long a caller waits for a *durable* acknowledgment. + /// + /// It is not [`Deadlines::capture`]: that one bounds a call to a participant, and this + /// one bounds two `fsync`s, a queue the caller shares with other captures and a disk. + /// Reusing the call budget here would make a slow disk look like an unresponsive worker. + pub durable: Duration, } /// How long the resolution waits between attempts. @@ -172,6 +178,7 @@ impl Default for Deadlines { resolve_attempts: 8192, boot: Duration::from_secs(30), capture: Duration::from_secs(30), + durable: Duration::from_secs(60), } } } @@ -3186,14 +3193,17 @@ impl Coordinator { /// operation. pub async fn await_durable(&mut self, ticket: CaptureTicket) -> Outcome { let CaptureTicket { checkpoint_id, boundary, receiver } = ticket; - let budget = self.deadlines.capture; + let budget = self.deadlines.durable; let outcome = crate::state::CheckpointWriter::wait(receiver, &checkpoint_id, budget).await; - if let crate::state::SaveOutcome::Committed { .. } = &outcome { + if outcome.is_durable() { self.durable = Some((checkpoint_id.clone(), boundary)); self.audit.push(format!("durable:{checkpoint_id}@{boundary}")); } else { - self.audit.push(format!("not-durable:{checkpoint_id}@{boundary}")); + // Named rather than lumped together: a failed write, a superseded capture, a lost + // reply and an expired caller budget are four different things to have to explain. + self.audit + .push(format!("not-durable:{}:{checkpoint_id}@{boundary}", outcome.event())); } Ok(outcome) } diff --git a/services/flysim/crates/fly-session/src/state.rs b/services/flysim/crates/fly-session/src/state.rs index 93ba77b..aa42d3c 100644 --- a/services/flysim/crates/fly-session/src/state.rs +++ b/services/flysim/crates/fly-session/src/state.rs @@ -999,10 +999,18 @@ pub enum SaveOutcome { Failed { checkpoint_id: Id, reason: String }, /// A later replaceable capture took this one's place in the queue before it was written. Superseded { checkpoint_id: Id, by: Id }, - /// The writer's reply never arrived. The operation's outcome is unknown from here, so - /// durable metadata does not move; the caller resolves the *same* operation against the - /// store manifest instead of saving again. + /// The writer finished this job and its reply channel was gone before the outcome could + /// be delivered. The write is over and its result is unknown from here. ReplyLost { checkpoint_id: Id }, + /// The caller's own budget ran out while the job was still queued or being written. The + /// save is not over: it may commit after this is reported. + /// + /// It is a different fact from [`SaveOutcome::ReplyLost`] and is named separately because + /// diagnosing one as the other is exactly the implicit best-effort reading these + /// contracts refuse. Both leave durable metadata where it was, and for both the caller + /// resolves the *same* operation against the store manifest instead of saving again -- + /// but only one of them is a save that has already stopped. + DeadlineExpired { checkpoint_id: Id }, } impl SaveOutcome { @@ -1011,17 +1019,54 @@ impl SaveOutcome { SaveOutcome::Committed { checkpoint_id, .. } | SaveOutcome::Failed { checkpoint_id, .. } | SaveOutcome::Superseded { checkpoint_id, .. } - | SaveOutcome::ReplyLost { checkpoint_id } => checkpoint_id, + | SaveOutcome::ReplyLost { checkpoint_id } + | SaveOutcome::DeadlineExpired { checkpoint_id } => checkpoint_id, } } /// The event name this outcome publishes under. + /// + /// Only the three the writer itself produces are ever published; the two caller-side + /// outcomes are what a caller saw, not what the store did, and the store does not announce + /// them on its own topic. pub fn event(&self) -> &'static str { match self { SaveOutcome::Committed { .. } => "committed", SaveOutcome::Failed { .. } => "failed", SaveOutcome::Superseded { .. } => "superseded", - SaveOutcome::ReplyLost { .. } => "failed", + SaveOutcome::ReplyLost { .. } | SaveOutcome::DeadlineExpired { .. } => "failed", + } + } + + /// True only past the durable commit point. + pub fn is_durable(&self) -> bool { + matches!(self, SaveOutcome::Committed { .. }) + } +} + +/// What the writer itself can produce for one job. +/// +/// The two caller-side outcomes -- a lost reply and an expired caller deadline -- are not in +/// here, because the writer cannot observe either. Keeping them out is what stops the writer's +/// own bookkeeping from carrying arms that can never run. +#[derive(Clone, Debug, PartialEq, Eq)] +enum WriteOutcome { + Committed { boundary: u64, file: String }, + Failed { reason: String }, +} + +impl WriteOutcome { + fn into_save(self, checkpoint_id: &Id) -> SaveOutcome { + match self { + WriteOutcome::Committed { boundary, file } => SaveOutcome::Committed { + checkpoint_id: checkpoint_id.clone(), + boundary, + file, + }, + WriteOutcome::Failed { reason } => SaveOutcome::Failed { + checkpoint_id: checkpoint_id.clone(), + reason, + }, } } } @@ -1037,13 +1082,21 @@ pub enum RetryPolicy { RetryThenRelease { attempts: u32 }, } -/// The writer's bounds. Both are finite and both refuse before a capture is requested. +/// The writer's bounds. Both are finite, and they refuse at different moments. +/// +/// [`WriterConfig::queue_capacity`] is the one a capture is refused *before* it is requested: +/// [`CheckpointWriter::reserve`] takes its slot first, which is what the durable row of +/// `state-media-v1` section 3 means by rejecting before capture. The byte budget cannot work +/// that way, because how many bytes a capture is worth is not known until the participants +/// have produced it; it is checked at [`CheckpointWriter::submit`], so an oversized capture is +/// refused after it exists and before it is queued, and its payloads are released with the +/// refusal. Both are named `BUSY` refusals and neither fails the epoch. #[derive(Clone, Copy, Debug)] pub struct WriterConfig { /// Outstanding coherent captures. `state-media-v1` section 3's initial session default - /// is two. + /// is two. Refused before a capture is requested. pub queue_capacity: usize, - /// The total payload bytes the queue may hold. + /// The total payload bytes the queue may hold. Refused at submit, once the size is known. pub max_queued_bytes: u64, pub retry: RetryPolicy, } @@ -1198,48 +1251,57 @@ capture is refused before it is requested rather than queued without bound", ) -> DomainResult> { let bytes = submission.byte_length(); let budget = self.shared.config.max_queued_bytes; - let held = self.shared.queued_bytes.load(Ordering::SeqCst); - if held + bytes > budget { - self.shared - .stats - .lock() - .expect("the writer stats are never poisoned") - .rejected += 1; - return Err(DomainError::before( - ErrorCode::Busy, - format!( - "the checkpoint queue holds {held} of {budget} bytes and this capture adds \ -{bytes}; the byte budget is finite and refuses before it is exceeded" - ), - )); - } let (reply, receiver) = tokio::sync::oneshot::channel(); let checkpoint_id = submission.checkpoint_id.clone(); let queued_event = submission.as_event("queued"); + // Reading the byte total, deciding on it and changing it are one critical section. + // Only one caller submits today, so a split could not be observed -- but a bound that + // is only correct while nobody else is submitting is not a bound. let superseded = { let mut queue = self.shared.queue.lock().expect("the writer queue is never poisoned"); - let replaced = if submission.replaceable { - queue - .iter() - .position(|job| job.submission.replaceable) - .map(|index| queue.remove(index).expect("just found")) + let replaced_index = if submission.replaceable { + queue.iter().position(|job| job.submission.replaceable) } else { None }; + // A capture that will take a queued replaceable one's place frees its bytes, so + // the budget is decided against what the queue will hold and not what it holds. + let freed = replaced_index.map_or(0, |index| queue[index].submission.byte_length()); + let held = self.shared.queued_bytes.load(Ordering::SeqCst); + let after = held.saturating_sub(freed) + bytes; + if after > budget { + self.shared + .stats + .lock() + .expect("the writer stats are never poisoned") + .rejected += 1; + return Err(DomainError::before( + ErrorCode::Busy, + format!( + "the checkpoint queue would hold {after} of {budget} bytes; the byte \ +budget is finite and refuses before it is exceeded" + ), + )); + } + let replaced = replaced_index.map(|index| queue.remove(index).expect("just found")); + if let Some(old) = &replaced { + self.shared + .queued_bytes + .fetch_sub(old.submission.byte_length(), Ordering::SeqCst); + } queue.push_back(Job { submission, reply, permit: reservation.permit }); self.shared.queued_bytes.fetch_add(bytes, Ordering::SeqCst); let mut stats = self.shared.stats.lock().expect("the writer stats are never poisoned"); stats.queued += 1; stats.peak_queue = stats.peak_queue.max(queue.len()); + // Sampled after the superseded job's bytes are gone, so the peak is a total the + // queue really held. stats.peak_bytes = stats .peak_bytes .max(self.shared.queued_bytes.load(Ordering::SeqCst)); replaced }; if let Some(old) = superseded { - self.shared - .queued_bytes - .fetch_sub(old.submission.byte_length(), Ordering::SeqCst); self.shared .stats .lock() @@ -1267,8 +1329,12 @@ capture is refused before it is requested rather than queued without bound", Ok(receiver) } - /// Waits for one save's outcome. A dropped reply channel is a lost save reply, which is - /// an outcome and not a hang. + /// Waits for one save's outcome, within the caller's own budget. + /// + /// The two ways this ends without an outcome are different facts and are reported as + /// themselves: the channel closing means the writer finished and the reply did not reach + /// here, and the budget running out means the save is still going. Neither is a hang and + /// neither is a save. pub async fn wait( receiver: tokio::sync::oneshot::Receiver, checkpoint_id: &Id, @@ -1276,7 +1342,8 @@ capture is refused before it is requested rather than queued without bound", ) -> SaveOutcome { match tokio::time::timeout(budget, receiver).await { Ok(Ok(outcome)) => outcome, - Ok(Err(_)) | Err(_) => SaveOutcome::ReplyLost { checkpoint_id: checkpoint_id.clone() }, + Ok(Err(_)) => SaveOutcome::ReplyLost { checkpoint_id: checkpoint_id.clone() }, + Err(_) => SaveOutcome::DeadlineExpired { checkpoint_id: checkpoint_id.clone() }, } } @@ -1342,6 +1409,10 @@ fn outcome_event(submission: &CaptureSubmission, outcome: &SaveOutcome) -> Map { + payload.insert("reason".into(), "the caller's durable budget expired".into()); + payload.insert("durable".into(), false.into()); + } } payload } @@ -1388,15 +1459,15 @@ async fn run_writer(shared: Arc) { shared .queued_bytes .fetch_sub(submission.byte_length(), Ordering::SeqCst); - let outcome = write_one(&shared, &submission).await; + let written = write_one(&shared, &submission).await; { let mut stats = shared.stats.lock().expect("the writer stats are never poisoned"); - match &outcome { - SaveOutcome::Committed { .. } => stats.committed += 1, - SaveOutcome::Failed { .. } | SaveOutcome::ReplyLost { .. } => stats.failed += 1, - SaveOutcome::Superseded { .. } => stats.superseded += 1, + match &written { + WriteOutcome::Committed { .. } => stats.committed += 1, + WriteOutcome::Failed { .. } => stats.failed += 1, } } + let outcome = written.into_save(&submission.checkpoint_id); publish_event(&shared.events, &shared.stats, &outcome_event(&submission, &outcome)).await; let lost = shared.faults.drop_reply_for.as_ref() == Some(&submission.checkpoint_id); // The writer owned these handles until the bytes were committed or the job failed. @@ -1416,21 +1487,19 @@ async fn run_writer(shared: Arc) { } } -async fn write_one(shared: &Arc, submission: &CaptureSubmission) -> SaveOutcome { +async fn write_one(shared: &Arc, submission: &CaptureSubmission) -> WriteOutcome { let mut payloads = Vec::with_capacity(submission.payloads.len()); for payload in &submission.payloads { let bytes = match payload.artifact.read_all().await { Ok(bytes) => bytes, Err(e) => { - return SaveOutcome::Failed { - checkpoint_id: submission.checkpoint_id.clone(), + return WriteOutcome::Failed { reason: format!("payload {}: {}", payload.name, e.message), }; } }; if bytes.len() as u64 != payload.byte_length || digest_of_bytes(&bytes) != payload.digest { - return SaveOutcome::Failed { - checkpoint_id: submission.checkpoint_id.clone(), + return WriteOutcome::Failed { reason: format!( "payload {} is not the content its capture declared", payload.name @@ -1442,8 +1511,7 @@ async fn write_one(shared: &Arc, submission: &CaptureSubmission) - let bytes = match checkpoint::encode(&submission.manifest, &payloads) { Ok(bytes) => bytes, Err(e) => { - return SaveOutcome::Failed { - checkpoint_id: submission.checkpoint_id.clone(), + return WriteOutcome::Failed { reason: format!("envelope: {}", e.0), }; } @@ -1477,8 +1545,7 @@ async fn write_one(shared: &Arc, submission: &CaptureSubmission) - .await; match result { Ok(Ok(())) => { - return SaveOutcome::Committed { - checkpoint_id: submission.checkpoint_id.clone(), + return WriteOutcome::Committed { boundary: submission.boundary, file: format!("{}.flysess", submission.checkpoint_id), }; @@ -1487,8 +1554,5 @@ async fn write_one(shared: &Arc, submission: &CaptureSubmission) - Err(e) => last = format!("the checkpoint writer stopped: {e}"), } } - SaveOutcome::Failed { - checkpoint_id: submission.checkpoint_id.clone(), - reason: last, - } + WriteOutcome::Failed { reason: last } } diff --git a/services/flysim/crates/fly-session/tests/processes.rs b/services/flysim/crates/fly-session/tests/processes.rs index 17dd0e2..0eaa267 100644 --- a/services/flysim/crates/fly-session/tests/processes.rs +++ b/services/flysim/crates/fly-session/tests/processes.rs @@ -122,6 +122,7 @@ async fn a_slow_participant_is_resolved_rather_than_failed(mode: ExecutionMode) resolve_attempts: 4096, boot: Duration::from_secs(30), capture: Duration::from_secs(30), + durable: Duration::from_secs(60), }; within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap(); let reports = within("run", f.harness.coordinator.run(2)) @@ -193,6 +194,7 @@ async fn a_resolution_says_which_of_its_two_bounds_ended_it(mode: ExecutionMode) resolve_attempts: 8192, boot: Duration::from_secs(30), capture: Duration::from_secs(30), + durable: Duration::from_secs(60), }; within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap(); let started = Instant::now(); @@ -224,6 +226,7 @@ async fn a_resolution_says_which_of_its_two_bounds_ended_it(mode: ExecutionMode) resolve_attempts: 3, boot: Duration::from_secs(30), capture: Duration::from_secs(30), + durable: Duration::from_secs(60), }; within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap(); let failure = within("step", f.harness.coordinator.step()) diff --git a/services/flysim/crates/fly-session/tests/state.rs b/services/flysim/crates/fly-session/tests/state.rs index d7b3a0f..b287c04 100644 --- a/services/flysim/crates/fly-session/tests/state.rs +++ b/services/flysim/crates/fly-session/tests/state.rs @@ -15,6 +15,7 @@ mod common; use std::collections::BTreeSet; use std::path::Path; use std::sync::Arc; +use std::time::Duration; use serde_json::{Value, json}; @@ -391,7 +392,8 @@ async fn a_lost_save_reply_holds_durable_metadata_in_every_mode(mode: ExecutionM a_lost_save_reply(Via::Unix, mode).await; } -/// Two ways a save can end without a saved acknowledgment, and neither moves the mark. +/// Three ways a save can end without a saved acknowledgment. None moves the mark, and each +/// is reported as itself rather than as the others. async fn a_lost_save_reply(via: Via, mode: ExecutionMode) { let lost = ckpt(1); let config = HarnessConfig { @@ -464,6 +466,59 @@ async fn a_lost_save_reply(via: Via, mode: ExecutionMode) { assert_eq!(resolved, None, "an unreferenced generation is never a restore candidate"); assert_eq!(g.harness.coordinator.durable(), None); g.shutdown().await; + + // The third: the caller's own budget runs out while the save is still going. That is a + // different fact from a lost reply -- this save has not stopped -- and it is reported as + // itself, because diagnosing one as the other is the implicit reading these rules refuse. + let slow = ckpt(3); + let gate = Arc::new(tokio::sync::Semaphore::new(0)); + let mut config = HarnessConfig::default(); + config.writer_faults.gate = Some(gate.clone()); + let mut h = fx(via, mode, config).await; + within("bootstrap", h.harness.coordinator.bootstrap()).await.unwrap(); + within("run", h.harness.coordinator.run(1)).await.unwrap(); + // The durable budget is the caller's own and is not the call budget: this shortens the + // wait for an acknowledgment without shortening a single call to a participant. + h.harness.coordinator.deadlines.durable = Duration::from_millis(100); + let ticket = within("capture", h.harness.coordinator.capture(&slow, false)) + .await + .unwrap(); + let outcome = within("durable", h.harness.coordinator.await_durable(ticket)) + .await + .unwrap(); + assert_eq!( + outcome, + SaveOutcome::DeadlineExpired { checkpoint_id: slow.clone() }, + "an expired caller budget is not a lost reply" + ); + assert_ne!(outcome, SaveOutcome::ReplyLost { checkpoint_id: slow.clone() }); + assert_eq!(h.harness.coordinator.durable(), None); + assert_eq!( + count( + &h.harness.coordinator.audit, + &format!("not-durable:failed:{slow}@1") + ), + 1, + "the session records that this capture is not durable: {:?}", + h.harness.coordinator.audit + ); + + // It really was still going: once the writer is let past its gate the same operation + // commits, and resolving it is what moves the mark. + gate.add_permits(16); + let deadline = std::time::Instant::now() + Duration::from_secs(10); + let resolved = loop { + let found = within("resolve", h.harness.coordinator.resolve_durable(&slow)) + .await + .unwrap(); + if found.is_some() || std::time::Instant::now() >= deadline { + break found; + } + tokio::time::sleep(Duration::from_millis(10)).await; + }; + assert_eq!(resolved, Some(1), "the save the caller stopped waiting for still committed"); + assert_eq!(h.harness.coordinator.durable(), Some((slow, 1))); + h.shutdown().await; } // =============================================================================================== From 55129002a4713785801c9e620b47c5c86feb500f Mon Sep 17 00:00:00 2001 From: acamilo Date: Tue, 22 Sep 2026 18:59:17 +0000 Subject: [PATCH 8/8] tests: the placeholder fingerprint in the migration test no longer reads as a MAC address --- services/flysim/crates/flysim/tests/compat_migration.rs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/services/flysim/crates/flysim/tests/compat_migration.rs b/services/flysim/crates/flysim/tests/compat_migration.rs index 13950b7..ce47b05 100644 --- a/services/flysim/crates/flysim/tests/compat_migration.rs +++ b/services/flysim/crates/flysim/tests/compat_migration.rs @@ -18,7 +18,7 @@ use flysim::store::{self, RuntimeState}; /// nothing here parses it, and a seven-digest one would be 455 characters of noise. fn compatibility(adapter: &str) -> String { format!( - "lif-1ms-f64-v2/{adapter}/aa:bb:cc:dd:ee:ff:00/fly-kc-mbon-rstdp-v2/\ + "lif-1ms-f64-v2/{adapter}/aabbccddeeff00/fly-kc-mbon-rstdp-v2/\ binjgb:c60e138da5a795ebb55e56b11b7e90024e41112c/\ pokered:0cd19d3b877b7dc66d12c7050bed9a7f38154d4b/statefmt:199616-x86_64-unknown-linux-gnu" )