Compare commits
9 commits
537cdd8ad9
...
55129002a4
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
55129002a4 | ||
|
|
083abe5f1f | ||
|
|
388e112ad5 | ||
|
|
45621903be | ||
|
|
55221a7c77 | ||
|
|
a8afd8f448 | ||
|
|
db30708c3f | ||
|
|
6655a1b1c6 | ||
|
|
23a4d7379b |
45 changed files with 7945 additions and 83 deletions
|
|
@ -355,6 +355,50 @@ on-screen ticker cannot disagree with what the sim did.
|
|||
against the prototype's WASM size and diffs a known save. If they match, prototype checkpoints
|
||||
import and the segment records the shared tag; if not, milestone saves must be re-earned and that
|
||||
is a stated M3 finding.
|
||||
- **Restoring across an adapter version** (2026-09-22). The compatibility string is compared
|
||||
whole, so bumping the reward adapter refuses every checkpoint the previous one wrote -- which is
|
||||
the right default and was, until now, the only behaviour. It is the wrong default for a change
|
||||
that only *adds* a rule: `pokered-unique8-v6` adds the catch reward and one counter,
|
||||
`catchCounts`, and means the same thing as `v5` for every other field, so a `v5` run is
|
||||
resumable and throwing it away would be a choice nobody made deliberately.
|
||||
|
||||
So there is one narrow, opt-in migration, `flybrain_gb::compatibility::decide`, and it requires
|
||||
**all three** of:
|
||||
|
||||
1. the two compatibility strings differ in the adapter segment (segment 1) and **nowhere else**.
|
||||
A dataset, kernel, plasticity, emulator-revision, symbol-provenance or state-format
|
||||
difference is still a refusal: none of those has a migration, and a fly restored across one
|
||||
is a different fly;
|
||||
2. the running adapter's `migrates_from()` lists the checkpoint's adapter, so the code that will
|
||||
read that state says out loud that it can. Pokémon Red's list is `["pokered-unique8-v5"]` and
|
||||
nothing else -- `v4` is excluded because its ledger holds no `boundary:` keys and resuming it
|
||||
would pay a second time for every exit already found, and `v3` because its stored rank is a
|
||||
rung on a different ladder;
|
||||
3. the deploy names the same adapter id in **`FLY_ACCEPT_ADAPTERS`** (comma- or
|
||||
space-separated). Unset or empty migrates nothing, which is what every deploy before this one
|
||||
did.
|
||||
|
||||
Condition 2 without 3 would make the migration silent; condition 3 without 2 would let an
|
||||
operator wave through a pair nobody wrote a migration for. `infra/05-deploy.sh`'s compatibility
|
||||
gate applies the same rule before it flips the `current` symlink, and writes the variable into
|
||||
`/etc/fly/fly.env` so flysim applies it at restore -- the two must agree, or a deploy would pass
|
||||
a gate that flysim then fails, which is the black stream the gate exists to prevent. The
|
||||
migration itself is `PokemonRedReward::import_state` doing what it already did: `catchCounts` is
|
||||
absent from a `v5` state and restores empty, which is the truth about a run that was never paid
|
||||
for a catch. `STATE_VERSION` does not move, because the schema did not.
|
||||
|
||||
- **Restarting a run from an earlier rung** (2026-09-22). `FLY_RESET_STATE=1` throws the run away;
|
||||
`infra/bin/fly-reset-to-milestone <N>` keeps it and rewinds it. It archives both stores to a
|
||||
dated directory, rewrites `milestone-<N>.checkpoint` with the ratchet's `attempts` and
|
||||
`recoveries` at zero (so the restarted run does not begin with its recovery budget already
|
||||
spent), installs it as the newest generation of the hot and durable stores, removes the
|
||||
milestone archives above N, and clears the event log -- whose id sequence the restored
|
||||
checkpoint's `lastEventId` rewinds. `best` is not touched: the archive's own `best` is the rung
|
||||
it was taken at, and the rank the stream shows is recomputed by the adapter from the restored
|
||||
game state. The implementation is `flysim::reset` (`flysim --reset-to-milestone N`) rather than
|
||||
the shell script, because two of those steps are inside the envelope. The sequence around it is
|
||||
in `infra/docs/runbook.md`.
|
||||
|
||||
- **A running macro is not checkpointed** (2026-09-16, `docs/design/macros.md`). Palette mode's
|
||||
state — the scene, the palette, the running macro, its plan and its frame count — is transient,
|
||||
like the readout's blocked-direction cooldown and for the same reason: a restore that resumed a
|
||||
|
|
|
|||
|
|
@ -118,6 +118,20 @@ Addresses are at the pinned commit. "Verified" is one of:
|
|||
| which slot is out | `wPlayerMonNumber` | `$cc2f` | 0-based party slot | ROM, trace |
|
||||
| the enemy | `wEnemyMonSpecies`, `wEnemyMonHP`, `wEnemyMonLevel`, `wEnemyMonMaxHP` | `$cfe5`, `$cfe6`, `$cff3`, `$cff4` | HP big-endian. Not written on the frame a battle starts — the reward adapter's own comment says the same — so the enemy is `None` for the first few hundred frames of a battle. | ROM (the rival's Squirtle, level 5, 20/20, and `None` on the first frame), trace |
|
||||
| how many moves | `wNumMovesMinusOne` | `$cd6c` | the move count minus one, valid in a battle | trace |
|
||||
| **a ball kept this one** | `wCapturedMonSpecies` | `$d11c` | **new 2026-09-22** (the catch reward, `docs/rewards-learning.md`). `ram/wram.asm`'s own comment is "0 if no mon was captured". `ItemUseBall` zeroes it before every throw (`.canUseBall`) and writes `wEnemyMonSpecies` into it only on the branch that keeps the Pokémon; `UseBagItem`'s `.returnAfterCapturingMon` zeroes it again and sets `wBattleResult` to 2 on the way out of the battle. It is therefore non-zero for the hundreds of frames the catch's text and Pokédex screen take, and zero everywhere else. The value is the **internal** species index, like `wEnemyMonSpecies` and unlike `wPokedexOwned`'s bit index. Address resolved by `services/flysim/tools/resolve_wram.py`, bracketed by `wFontLoaded` and `wForcePlayerToChooseMon`. | survey (`tests/rom_catch.rs`: a real wild battle from a rung-9 checkpoint, balls thrown by the `THROW BALL` macro, the byte read out of the running game), trace (`pokemon_red/tests.rs`) |
|
||||
|
||||
`wBattleResult` (`$cf0b`) is the second half of that row and is worth its own sentence: it is 0
|
||||
for a win, 1 for a loss, and 2 on exactly two paths in the whole game -- `.returnAfterCapturingMon`
|
||||
and a *link* battle whose opponent ran (`engine/battle/core.asm`), which this cartridge never has.
|
||||
So "the captured-species byte was non-zero during the battle **and** the result is 2" is a catch
|
||||
and nothing else. `InitBattleVariables`, `ResetStatusAndHalveMoneyOnBlackout` and
|
||||
`HandleFlyWarpOrDungeonWarp` all clear it, so a stale 2 cannot survive into the next battle.
|
||||
|
||||
Not used for the catch, and why: `wPartyCount` (`$d163`) rises on a catch **only** when the party
|
||||
has room -- a full party sends the Pokémon to `wBoxCount` instead -- and it also rises for a gift,
|
||||
a trade and a Pokémon withdrawn from the PC. Reading a catch off it would need a second rule to
|
||||
tell those apart. The cartridge's own flag needs none, which is why the row above is the one the
|
||||
adapter reads.
|
||||
|
||||
### Battle menu and cursor, own turn against forced switch
|
||||
|
||||
|
|
|
|||
|
|
@ -105,6 +105,31 @@ names; a manifest missing any of them is not a complete checkpoint.
|
|||
| `helperState` | External-helper state required for exact resume, as payload names |
|
||||
| `payloads` | `[{name, byteLength, digest}]`, mirroring the payload table |
|
||||
|
||||
**Amendment, 2026-09-22 (STATE-01).** The table above names a holder for every payload except
|
||||
the environment's own, although section 6's fixture has one (`world`) and a group install has
|
||||
to map it by name like any other participant's. The manifest therefore also records:
|
||||
|
||||
| Field | Contents |
|
||||
| --- | --- |
|
||||
| `environment` | `{workerId, payload}`: which worker the world belonged to and the payload name holding its state |
|
||||
|
||||
The reference implementations' required-field set was also missing `helperState`, which this
|
||||
section has listed from the start. Both are now in `REQUIRED_MANIFEST_FIELDS` in Rust and in
|
||||
TypeScript, and the fixture was regenerated by the existing example. The schema set is
|
||||
untouched, so `contractDigest` is unchanged.
|
||||
|
||||
`coordinator.eventWatermarks` is `{lastSourceStep, issued}`. The fixture illustrated
|
||||
`{lastEventId, lastOrdinal}`, and it is the illustration that changed: an event id is derived
|
||||
from the epoch, so a watermark spelled as one cannot be compared across the restore that
|
||||
gives the session a new epoch, while a source step and an issued count can.
|
||||
|
||||
**A required-manifest-field change is compatibility-relevant and `contractDigest` does not
|
||||
cover it.** The digest is taken over the schema set, and this manifest is not in it, so
|
||||
`envelopeVersion` is the only thing that can carry such a change. It stays `1` here only
|
||||
because no production `FLYSESS1` file exists yet: once one does, adding or removing a required
|
||||
manifest field **must** bump `envelopeVersion`, because a reader of the older version would
|
||||
otherwise accept a file it cannot completely read, or refuse one it could.
|
||||
|
||||
`payloads` is redundant with the table on purpose: the table is what a reader needs to map
|
||||
bytes, and the manifest is what a store lists, compares and reports without opening the
|
||||
payload area. A reader checks that the two agree.
|
||||
|
|
|
|||
|
|
@ -190,6 +190,28 @@ restored time. It cannot advance gameplay to manufacture it. Capture/reconstruct
|
|||
covers render/inspection state and any pending sensor pipeline. Agent state agrees with it;
|
||||
do not replay reward or recalibrate merely to fill missing cached data.
|
||||
|
||||
**Amendment, 2026-09-22 (STATE-01).** Three readings of this section, made explicit because
|
||||
they are now enforced:
|
||||
|
||||
- `compatibilityDigest` on `CaptureResult` and `StageRestoreParams` is the **participant's**
|
||||
capture compatibility digest of [worker interfaces](workers-v1.md) section 2 -- profile,
|
||||
resolved seed, numerical model version and effective instance configuration for an agent;
|
||||
backend, content, patch, controller and parser identity for an environment. It is not the
|
||||
manifest's `compatibility` block of section 4, which is the composition's and which the
|
||||
coordinator compares before anything is asked to stage. Both exist because they answer
|
||||
different questions, and a restore that passed the second could still be handing an agent
|
||||
another agent's brain.
|
||||
- The observation `ActivateRestore` returns ran no transition, so it carries **no audio
|
||||
chunk**, and one in it is refused. Section 2's chunk is the audio of an interval and this
|
||||
observation covers none; MEDIA-01 implemented that rule as "boundary 0 carries no chunk",
|
||||
which is true of the only such observation that slice could produce and false of this one.
|
||||
The rule is about provenance, not about the boundary number.
|
||||
- A participant that staged into a group install the coordinator then abandoned must be
|
||||
**replaced** before another restore, exactly as one that activated must. It is holding a
|
||||
validated replacement state that nothing installed, and [session RPC](ipc-v1.md) section 6
|
||||
already refuses to silently reattach such a participant to an active epoch. Without this the
|
||||
group's second attempt meets its own leftovers and calls them a conflict.
|
||||
|
||||
If emulator validation requires mutation, stage a stopped replacement emulator. If that cannot
|
||||
provide externally atomic resume, advertise episode-restart, not exact-checkpoint. After all
|
||||
activation acknowledgments, install the coordinator's staged task/executor/admission state
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
# Rewards and learning
|
||||
|
||||
The live reward catalog of the Pokémon Red adapter, `pokered-unique8-v5`. The code of record is
|
||||
The live reward catalog of the Pokémon Red adapter, `pokered-unique8-v6`. The code of record is
|
||||
`services/flysim/crates/flybrain-gb/src/pokemon_red/` (`catalog.rs` holds the values, `mod.rs` the
|
||||
gates and the rules); this page says what each rule pays for and why it is allowed to. The
|
||||
prototype's own `docs/rewards-learning.md` in `fly-plays-pokemon` is where the first seven rules
|
||||
|
|
@ -24,6 +24,7 @@ change what the fly can do.
|
|||
| `battle` | `wildwin` | +0.1, +0.05, +0.0333 | 100 ms | At most three observed wild KOs per `(map, species, level)` |
|
||||
| `badge` | `badge` | +3 | 400 ms | Each newly set badge bit |
|
||||
| `boundary` | `explore` | +0.05, +0.10 | 100 ms | First tile adjacent to one of the map's exits, and the exit tile itself; once per `(map, exit)` for the lifetime of the ledger |
|
||||
| `catch` | `wildwin` | +0.30, +0.10 | 150 ms | A wild Pokémon kept by a ball: +0.30 for a species this run had never owned, +0.10 for a repeat; at most three payouts per species for the lifetime of the ledger |
|
||||
|
||||
Every value is positive: there are no loss or blackout penalties, and `catalog::rule("blackout")`
|
||||
is `None` by test. The values in one frame sum into `R`, and the network reinforces once with
|
||||
|
|
@ -33,6 +34,54 @@ The feed-kind column is `RewardKind::from_adapter` in `services/flysim/crates/fl
|
|||
`docs/feed-protocol.md` publishes seven counters, and an adapter kind that has no counter of its
|
||||
own shares the nearest one. It still reaches the page as an event with its own label.
|
||||
|
||||
Two consequences of that sharing are worth stating rather than discovering. `catch` publishes on
|
||||
`wildwin` because a catch is a wild battle the fly won by keeping the Pokémon, and *not* on
|
||||
`pokedex` because the `species` rule already pays for the Pokédex bit the same catch sets --
|
||||
counting it twice would be the dishonest option. And the stage's ticker copy is keyed on the feed
|
||||
kind, not on the catalog kind (`apps/stage/src/games/pokemon-red.ts`), so the row for a catch
|
||||
currently reads "wild win". The event's own label, `CAUGHT #<species>`, is what reaches the event
|
||||
log, `/status` and the checkpoint. Changing the ticker copy means opening the feed's closed kind
|
||||
set, which this rule deliberately did not do.
|
||||
|
||||
## Catch rewards
|
||||
|
||||
The operator's decision of 2026-09-22: the fly is paid for *keeping* a wild Pokémon, not only for
|
||||
knocking one out. The rule is one kind with two payouts, the way `boundary` is.
|
||||
|
||||
**How a catch is read.** From `wCapturedMonSpecies` (`$d11c`), whose comment in `ram/wram.asm` at
|
||||
the pinned commit is "0 if no mon was captured". `ItemUseBall` zeroes it before every throw
|
||||
(`.canUseBall`) and writes `wEnemyMonSpecies` into it only on the branch that keeps the Pokémon;
|
||||
`UseBagItem`'s `.returnAfterCapturingMon` zeroes it again and sets `wBattleResult` to 2 on the way
|
||||
out of the battle. `wBattleResult` is 2 on exactly two paths in the whole game -- that one, and a
|
||||
link battle whose opponent ran -- so requiring both the species and the result means a byte read
|
||||
out of a half-initialised battle cannot pay. The adapter records the species during the battle and
|
||||
pays on the way out, where the wild-KO payout already lives.
|
||||
|
||||
Not from `wPartyCount`. A catch with a full party raises `wBoxCount` instead, and `wPartyCount`
|
||||
also rises for a gift, a trade and a Pokémon taken out of the PC, so it would need a second rule
|
||||
to mean anything. The cartridge's own flag needs none.
|
||||
|
||||
**What counts as a new species.** The `species` payout inside the same battle. Nothing but a catch
|
||||
can set a `wPokedexOwned` bit during a wild battle, so a `species` payout between the battle
|
||||
starting and the ball keeping the Pokémon *is* that Pokémon being new to the run. It is read this
|
||||
way rather than off `wCapturedMonSpecies` because that byte is the cartridge's **internal** species
|
||||
index while the owned bitset is by **Pokédex number**, and nothing in WRAM converts between the two
|
||||
(`docs/design/macros-wram.md` section 2, "species numbering"). A battle restored from a checkpoint
|
||||
written before this rule existed carries no "species payouts when it started", which reads as
|
||||
"cannot tell" and pays the repeat amount: the conservative half, and at most 0.20 once.
|
||||
|
||||
**The budget.** Three payouts per species for the lifetime of the ledger, the same cap and the
|
||||
same reason as the wild-KO rule's three: a species the fly can find over and over is a farm, and
|
||||
three is enough for the behaviour to be learned. A rollback blocks every species already paid,
|
||||
exactly as it blocks every wild-KO key already paid, so the same catch cannot be replayed for
|
||||
reward. A Safari Zone or old-man battle pays nothing, because the whole sample is dropped a step
|
||||
earlier with a visible mode; a trainer battle pays nothing, because balls cannot be thrown in one.
|
||||
|
||||
**The scale.** 0.30 on its own is below a new Pokédex entry (0.50), below a story flag (1.0) and
|
||||
well below a badge (3.0). A catch of a new species pays 0.80 across two kinds, which sits between
|
||||
a story flag and a badge -- deliberately, because it is the one event that is both a discovery and
|
||||
a thing the fly had to do on purpose.
|
||||
|
||||
## Gates
|
||||
|
||||
Semantic rewards are enabled for exactly one cartridge, the SHA-256 in `SUPPORTED_ROM`. Any other
|
||||
|
|
@ -128,7 +177,19 @@ body picks the macro; the descending neurons press the buttons.**
|
|||
|
||||
## Honesty
|
||||
|
||||
The catalog now includes exits. That is worth saying plainly on the honesty panel, because paying
|
||||
The catalog now includes catches. The honesty panel's copy is not data-driven from the catalog --
|
||||
`apps/stage/src/lib/schedule.ts`'s rotating card is four written lines and lists no kinds -- so
|
||||
there was nothing to regenerate and the copy is unchanged. The sentences below are where the
|
||||
argument lives.
|
||||
|
||||
Paying for a catch does not move the fly: the ball is thrown by a macro the mushroom body chose
|
||||
among the ones the battle scene put on the pad, and the payout is read out of WRAM after the
|
||||
frame. What it does do is make one of the palette's existing macros worth choosing, which is the
|
||||
same kind of pressure every other rule applies. The cap is what keeps it from becoming a farm: a
|
||||
run that finds one patch of grass and throws balls at the same species all night earns 0.50 from
|
||||
it and then nothing.
|
||||
|
||||
The catalog also includes exits. That is worth saying plainly on the honesty panel, because paying
|
||||
for a door is closer to telling the fly where to go than paying for a badge is:
|
||||
|
||||
- **still no button path.** Nothing in the adapter chooses or biases a button. The reward is read
|
||||
|
|
|
|||
|
|
@ -736,3 +736,11 @@ rewritten separately.
|
|||
(73/73) because 82% of the fixed run is battle time; Fable shipped it on the same judgement as
|
||||
v0.4.6 and started row 50 (MOVE n blocked on an unresponsive move list). The on-screen chat ring
|
||||
now survives a sim restart (sidecar in the hot dir, never in the checkpoint).
|
||||
- 2026-09-22 (v0.5.0, the operator's decision): the fly is paid for keeping a wild Pokémon. New
|
||||
catalog kind `catch` (0.30 for a species this run never caught, 0.10 for a repeat, three payouts
|
||||
per species), read from the captured-species byte and the battle result together; the existing
|
||||
species rule still pays on top. Adapter `pokered-unique8-v6`; the compatibility string differs
|
||||
in the adapter segment only, and a deploy with `FLY_ACCEPT_ADAPTERS=pokered-unique8-v5` migrates
|
||||
a v5 checkpoint instead of refusing it. `fly-reset-to-milestone <N>` restarts the run from a
|
||||
ladder rung (archives both stores first). The live run restarts from rung 7 with this release, so
|
||||
the ladder is climbed again with the catch reward and the row-54 walks in place.
|
||||
|
|
|
|||
63
infra/05-deploy.sh
Executable file → Normal file
63
infra/05-deploy.sh
Executable file → Normal file
|
|
@ -186,6 +186,36 @@ else
|
|||
log "05-deploy: CPUSET unset — heavy in-container steps run unpinned (no partition configured)"
|
||||
fi
|
||||
|
||||
# Whether the only difference between two compatibility strings is the adapter
|
||||
# segment, and FLY_ACCEPT_ADAPTERS names the adapter the live checkpoints carry.
|
||||
#
|
||||
# The bash half of flybrain_gb::compatibility::decide, which is what flysim
|
||||
# itself applies at restore. Both have to agree: a gate that let a deploy
|
||||
# through and a flysim that then refused every checkpoint would be the black
|
||||
# stream this whole section exists to prevent. The string is
|
||||
# {kernel}/{adapter}/{fingerprint}/{plasticity}/binjgb:{rev}/pokered:{commit}/statefmt:{id},
|
||||
# so the adapter is segment 1 and nothing else may move.
|
||||
adapter_migration_accepted() {
|
||||
local live="$1" new="$2" accepted="$3"
|
||||
local -a live_parts new_parts
|
||||
IFS='/' read -r -a live_parts <<< "$live"
|
||||
IFS='/' read -r -a new_parts <<< "$new"
|
||||
[ "${#live_parts[@]}" -eq "${#new_parts[@]}" ] || return 1
|
||||
local i differing=0 index=-1
|
||||
for ((i = 0; i < ${#live_parts[@]}; i++)); do
|
||||
if [ "${live_parts[$i]}" != "${new_parts[$i]}" ]; then
|
||||
differing=$((differing + 1))
|
||||
index=$i
|
||||
fi
|
||||
done
|
||||
[ "$differing" -eq 1 ] && [ "$index" -eq 1 ] || return 1
|
||||
local entry
|
||||
for entry in ${accepted//,/ }; do
|
||||
[ "$entry" = "${live_parts[1]}" ] && return 0
|
||||
done
|
||||
return 1
|
||||
}
|
||||
|
||||
# cpu_pin CMD [ARGS...] — run CMD inside the container on the page cpus.
|
||||
# Falls through to a plain ct_exec when no partition is configured, so this is
|
||||
# a no-op on an unpartitioned container rather than a new failure mode (a
|
||||
|
|
@ -262,6 +292,16 @@ if [ -n "$RELEASE_TARBALL" ]; then
|
|||
# checkpoints and the on-screen chat ring's sidecar — so the new build warms
|
||||
# up fresh. Everything learned so far is thrown away, which is why it is not
|
||||
# the default.
|
||||
#
|
||||
# FLY_ACCEPT_ADAPTERS is the *other* override, and the opposite one: it keeps
|
||||
# the run. It names adapter version strings whose checkpoints the new build
|
||||
# may migrate — e.g. FLY_ACCEPT_ADAPTERS=pokered-unique8-v5 for the deploy
|
||||
# that adds the catch reward. It only applies when the adapter segment is the
|
||||
# ONLY difference between the two strings and the new build's adapter says it
|
||||
# can read that one; a dataset, kernel, emulator or state-format change is
|
||||
# still a refusal, because none of those has a migration. The same variable is
|
||||
# written into /etc/fly/fly.env below, so flysim applies the same rule at
|
||||
# restore that this gate applied at deploy.
|
||||
# -----------------------------------------------------------------------
|
||||
state_dir="${FLY_STATE_DIR:-/srv/fly/state}"
|
||||
hot_dir="${FLY_STATE_HOT_DIR:-/run/fly/state}"
|
||||
|
|
@ -287,6 +327,11 @@ if [ -n "$RELEASE_TARBALL" ]; then
|
|||
log "05-deploy: no decodable checkpoint in ${state_dir} — nothing to compare, continuing"
|
||||
elif [ "$new_compat" = "$live_compat" ]; then
|
||||
log "05-deploy: checkpoint compatibility matches the live state, the new build will restore it"
|
||||
elif [ -n "${FLY_ACCEPT_ADAPTERS:-}" ] \
|
||||
&& adapter_migration_accepted "$live_compat" "$new_compat" "$FLY_ACCEPT_ADAPTERS"; then
|
||||
log "05-deploy: FLY_ACCEPT_ADAPTERS=${FLY_ACCEPT_ADAPTERS} — the adapter version is the only difference, and it is named; the run is KEPT and migrated"
|
||||
log "05-deploy: live: $live_compat"
|
||||
log "05-deploy: new: $new_compat"
|
||||
elif [ "${FLY_RESET_STATE:-0}" = 1 ]; then
|
||||
archive="${state_dir}.$(date -u +%Y%m%d%H%M%S)"
|
||||
log "05-deploy: FLY_RESET_STATE=1 — compatibility CHANGED, archiving the durable state to ${archive} and clearing the hot ring"
|
||||
|
|
@ -300,8 +345,12 @@ if [ -n "$RELEASE_TARBALL" ]; then
|
|||
die "05-deploy: REFUSING to deploy release ${version}: its checkpoint compatibility string does not match the live state in ${state_dir}, so flysim would refuse every checkpoint there and then refuse to start at all — a black stream.
|
||||
live state: ${live_compat}
|
||||
new build: ${new_compat}
|
||||
The difference is usually an adapter/ladder or dataset version bump. Two ways forward:
|
||||
The difference is usually an adapter/ladder or dataset version bump. Three ways forward:
|
||||
* deploy a build whose string matches (check out the commit the running release was built from), or
|
||||
* if the ADAPTER VERSION is the only segment that differs and the new build documents a
|
||||
migration from the old one, re-run with FLY_ACCEPT_ADAPTERS set to the adapter id in the live
|
||||
string (e.g. FLY_ACCEPT_ADAPTERS=pokered-unique8-v5). The run is kept; flysim applies the same
|
||||
rule at restore. See docs/design/flysim.md, \"Restoring across an adapter version\", or
|
||||
* accept losing everything the brain has learned and re-run with FLY_RESET_STATE=1, which
|
||||
archives ${state_dir}'s checkpoints to ${state_dir}.<timestamp> (kept, not deleted) and
|
||||
clears ${hot_dir} so the new build warms up fresh.
|
||||
|
|
@ -457,6 +506,16 @@ trap 'rm -f "$tmp_fly_env" "$tmp_flypush_env"' EXIT
|
|||
if [[ -n "${FLY_MACRO_BLOCKED_MINUTES:-}" ]]; then
|
||||
echo "FLY_MACRO_BLOCKED_MINUTES=${FLY_MACRO_BLOCKED_MINUTES}"
|
||||
fi
|
||||
# Adapter versions whose checkpoints this build may migrate
|
||||
# (flybrain_gb::compatibility, docs/design/flysim.md "Restoring across an
|
||||
# adapter version"). Only written when it is set, because the safe state is
|
||||
# absent: an empty or missing variable migrates nothing, which is what every
|
||||
# deploy before 2026-09-22 did. It stays in fly.env for as long as the
|
||||
# operator leaves it on the deploy command line, so removing the opt-in is
|
||||
# one deploy without it.
|
||||
if [[ -n "${FLY_ACCEPT_ADAPTERS:-}" ]]; then
|
||||
echo "FLY_ACCEPT_ADAPTERS=${FLY_ACCEPT_ADAPTERS}"
|
||||
fi
|
||||
# flybridge (services/bridge/src/config.ts). Nothing wrote these before, so
|
||||
# flybridge.service had no EnvironmentFile= at all and the service refused to
|
||||
# start with "CHANNEL is required / BOT_USER is required / GAME_TITLE is
|
||||
|
|
@ -607,7 +666,7 @@ fi
|
|||
# ---------------------------------------------------------------------------
|
||||
log "05-deploy: converging bin/ helpers to /opt/fly/bin"
|
||||
ct_exec "$CTID" -- mkdir -p /opt/fly/bin
|
||||
for name in fly-watchdog fly-recap fly-retention flypush flystage-launch flycast-launch wait-for-x wait-for-stage wait-for-health; do
|
||||
for name in fly-watchdog fly-recap fly-retention fly-reset-to-milestone flypush flystage-launch flycast-launch wait-for-x wait-for-stage wait-for-health; do
|
||||
converge_file "$CTID" "$INFRA_DIR/bin/$name" "/opt/fly/bin/$name" 0755 root:root >/dev/null
|
||||
done
|
||||
|
||||
|
|
|
|||
77
infra/bin/fly-reset-to-milestone
Executable file
77
infra/bin/fly-reset-to-milestone
Executable file
|
|
@ -0,0 +1,77 @@
|
|||
#!/usr/bin/env bash
|
||||
# infra/bin/fly-reset-to-milestone — restart the run from an earlier ladder rung,
|
||||
# instead of from scratch.
|
||||
#
|
||||
# The operator's decision of 2026-09-22: "restart the live run from an early
|
||||
# checkpoint instead of from scratch". 05-deploy's FLY_RESET_STATE=1 cannot do
|
||||
# that — it archives the durable state and the next start warms up a fresh fly,
|
||||
# losing everything the brain has learned. This promotes one milestone archive
|
||||
# (milestone-<N>.checkpoint, written at the first commit at a new best rank and
|
||||
# never rotated away) to being what both stores restore.
|
||||
#
|
||||
# Usage: fly-reset-to-milestone <N>
|
||||
# Run INSIDE the container, as root, with flysim STOPPED. It refuses
|
||||
# otherwise, and it refuses a rung this run never reached.
|
||||
#
|
||||
# The whole sequence — stop, reset, deploy with the adapter opt-in, start,
|
||||
# verify the rank — is in infra/docs/runbook.md, "Restart the run from a rung".
|
||||
# Nothing here is destructive on its own: every file in both stores is copied to
|
||||
# a dated directory next to the durable one before anything is rewritten.
|
||||
set -euo pipefail
|
||||
|
||||
: "${FLY_STATE_DIR:=/srv/fly/state}"
|
||||
: "${FLY_STATE_HOT_DIR:=/run/fly/state}"
|
||||
: "${FLY_RELEASE_DIR:=/opt/fly/current}"
|
||||
: "${FLY_SERVICE:=flysim.service}"
|
||||
: "${FLY_USER:=fly}"
|
||||
|
||||
FLYSIM="${FLY_BIN:-${FLY_RELEASE_DIR}/flysim}"
|
||||
|
||||
log() { echo "fly-reset-to-milestone: $*" >&2; }
|
||||
die() { log "$*"; exit 1; }
|
||||
|
||||
RANK="${1:-}"
|
||||
if [ "$#" -ne 1 ] || ! [[ "$RANK" =~ ^[0-9]+$ ]]; then
|
||||
die "usage: fly-reset-to-milestone <rung> (e.g. fly-reset-to-milestone 9)"
|
||||
fi
|
||||
|
||||
# --- refusals ----------------------------------------------------------------
|
||||
# A running flysim owns both stores: it commits a hot checkpoint every few
|
||||
# seconds and a durable one every few minutes, so a reset underneath it would be
|
||||
# overwritten within the minute and the tool would have lied.
|
||||
if command -v systemctl >/dev/null 2>&1 && systemctl is-active --quiet "$FLY_SERVICE"; then
|
||||
die "$FLY_SERVICE is running. Stop it first: systemctl stop $FLY_SERVICE"
|
||||
fi
|
||||
[ -x "$FLYSIM" ] || die "no flysim binary at $FLYSIM (set FLY_BIN to point at one)"
|
||||
|
||||
milestone="${FLY_STATE_DIR}/milestone-${RANK}.checkpoint"
|
||||
# The binary refuses this too, and refuses before it copies anything; checking
|
||||
# here as well is what makes the message name the rungs that do exist.
|
||||
if [ ! -f "$milestone" ]; then
|
||||
log "no milestone archive for rung ${RANK}: $milestone does not exist."
|
||||
log "rungs this run reached:"
|
||||
ls -1 "${FLY_STATE_DIR}"/milestone-*.checkpoint 2>/dev/null \
|
||||
| sed 's|.*/milestone-||; s|\.checkpoint$||' | sort -n | tr '\n' ' ' >&2 || true
|
||||
echo >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --- the reset ---------------------------------------------------------------
|
||||
log "resetting to rung ${RANK} (durable ${FLY_STATE_DIR}, hot ${FLY_STATE_HOT_DIR})"
|
||||
FLY_STATE="$FLY_STATE_DIR" FLY_STATE_HOT="$FLY_STATE_HOT_DIR" \
|
||||
"$FLYSIM" --reset-to-milestone "$RANK"
|
||||
|
||||
# flysim runs unprivileged; this tool runs as root, so everything it wrote and
|
||||
# everything it archived has to go back to the service account.
|
||||
if command -v chown >/dev/null 2>&1 && id "$FLY_USER" >/dev/null 2>&1; then
|
||||
chown -R "${FLY_USER}:${FLY_USER}" "$FLY_STATE_DIR" "$FLY_STATE_HOT_DIR" 2>/dev/null || true
|
||||
for dir in "${FLY_STATE_DIR}".reset-*; do
|
||||
[ -d "$dir" ] && chown -R "${FLY_USER}:${FLY_USER}" "$dir"
|
||||
done
|
||||
fi
|
||||
|
||||
log "done. Next, per infra/docs/runbook.md:"
|
||||
log " 1. deploy the build whose adapter wrote that checkpoint, or deploy the new"
|
||||
log " one with FLY_ACCEPT_ADAPTERS set to the checkpoint's adapter id"
|
||||
log " 2. systemctl start $FLY_SERVICE"
|
||||
log " 3. curl -s localhost:7401/status | grep -o '\"rank\":[0-9]*'"
|
||||
|
|
@ -231,6 +231,59 @@ auto-reset (`docs/design/flysim.md` section 8: "no automatic fresh start, ever")
|
|||
is deliberate — a silent reset would be indistinguishable from real progress on stream.
|
||||
A deliberate reset means moving `/srv/fly/state` aside by hand.
|
||||
|
||||
## Restart the run from a rung
|
||||
|
||||
When the run has to go back to an earlier milestone rather than start over — the operator's
|
||||
decision of 2026-09-22 was "restart the live run from an early checkpoint instead of from
|
||||
scratch". `FLY_RESET_STATE=1` is the wrong tool: it archives the durable state and the next start
|
||||
warms up a fresh fly, losing everything the brain has learned.
|
||||
|
||||
`infra/bin/fly-reset-to-milestone <N>` promotes `milestone-<N>.checkpoint` to being what both
|
||||
stores restore, with the ratchet's attempts and recoveries back at zero. It copies every file in
|
||||
both stores to `/srv/fly/state.reset-<UTC>` first, so it is reversible by hand. It refuses while
|
||||
flysim is running, and refuses a rung this run never reached.
|
||||
|
||||
The whole sequence, in order. Claim the container in the host's agent claim log first, like any
|
||||
other work on it.
|
||||
|
||||
```
|
||||
CTID=<release-ctid>
|
||||
N=9 # the rung to restart from
|
||||
|
||||
# 1. what rungs exist at all
|
||||
pct exec $CTID -- ls -1 /srv/fly/state/milestone-*.checkpoint
|
||||
|
||||
# 2. stop flysim (it owns both stores; a reset underneath it is overwritten within the minute)
|
||||
pct exec $CTID -- systemctl stop flysim.service
|
||||
|
||||
# 3. the reset. Prints what it did, one line per step.
|
||||
pct exec $CTID -- /opt/fly/bin/fly-reset-to-milestone $N
|
||||
|
||||
# 4. deploy. Two cases:
|
||||
# (a) the running release already wrote that checkpoint -> nothing to deploy, skip to 5.
|
||||
# (b) the new build bumps the ADAPTER VERSION and nothing else -> name the checkpoint's
|
||||
# adapter so the gate and flysim both migrate instead of refusing:
|
||||
FLY_ACCEPT_ADAPTERS=pokered-unique8-v5 infra/05-deploy.sh <release-env> <release-tarball>
|
||||
# The gate logs "the adapter version is the only difference, and it is named; the run is KEPT
|
||||
# and migrated", and writes FLY_ACCEPT_ADAPTERS into /etc/fly/fly.env so flysim applies the
|
||||
# same rule at restore. Anything else about the string differing is still a refusal.
|
||||
|
||||
# 5. start
|
||||
pct exec $CTID -- systemctl start flysim.service
|
||||
|
||||
# 6. verify: the rank is the rung, and the restore came from the generation the tool wrote
|
||||
pct exec $CTID -- curl -s http://127.0.0.1:7401/status | jq '.milestone.rank, .game.badges, .checkpoint'
|
||||
pct exec $CTID -- journalctl -u flysim -n 40 --no-pager | grep -E 'restored|migration|compatibility'
|
||||
```
|
||||
|
||||
Step 6 is the one that must be read rather than assumed. The rank is recomputed by the adapter
|
||||
from the restored game state, not taken from the ratchet, so a rank that is *not* N means the
|
||||
milestone archive was taken somewhere other than where its name says — stop and look before
|
||||
starting a stream on it.
|
||||
|
||||
To undo: stop flysim, move the contents of `/srv/fly/state.reset-<UTC>/durable` back into
|
||||
`/srv/fly/state`, delete the generation the tool wrote, and start again.
|
||||
|
||||
## Restore from the backup host
|
||||
|
||||
```
|
||||
|
|
|
|||
14
infra/env/example.env
vendored
14
infra/env/example.env
vendored
|
|
@ -317,6 +317,20 @@ FLY_MACRO_MODE=raw
|
|||
# target once more. Unset means the default, 10.
|
||||
# FLY_MACRO_BLOCKED_MINUTES=10
|
||||
|
||||
# --- restoring across an adapter version ------------------------------------
|
||||
# Adapter version strings whose checkpoints this build may migrate, comma- or
|
||||
# space-separated (docs/design/flysim.md, "Restoring across an adapter
|
||||
# version"). Unset -- the default, and what every deploy before 2026-09-22 did
|
||||
# -- migrates nothing: a build whose compatibility string differs from the live
|
||||
# state's is refused by 05-deploy's gate and by flysim at restore.
|
||||
#
|
||||
# It applies only when the ADAPTER segment is the only difference between the
|
||||
# two strings AND the new build's adapter declares a migration from that one. A
|
||||
# dataset, kernel, plasticity, emulator or state-format difference is still a
|
||||
# refusal. Set it for the one deploy that needs it and leave it out afterwards;
|
||||
# 05-deploy writes it into /etc/fly/fly.env only while it is set.
|
||||
# FLY_ACCEPT_ADAPTERS=pokered-unique8-v5
|
||||
|
||||
# --- push mode --------------------------------------------------------------
|
||||
# local: flypush.service stays disabled, everything else identical to prod.
|
||||
# twitch: flypush.service is enabled by 07-enable.sh.
|
||||
|
|
|
|||
|
|
@ -223,7 +223,12 @@ export function decode(input: Uint8Array): Envelope {
|
|||
};
|
||||
}
|
||||
|
||||
/** The manifest fields state-media-v1 section 4 requires. */
|
||||
/**
|
||||
* The manifest fields state-media-v1 section 4 requires.
|
||||
*
|
||||
* `helperState` and `environment` join the list under the 2026-09-22 amendment to
|
||||
* checkpoint-envelope-v1 section 3.
|
||||
*/
|
||||
export const REQUIRED_MANIFEST_FIELDS = [
|
||||
'envelopeVersion',
|
||||
'checkpointId',
|
||||
|
|
@ -236,6 +241,8 @@ export const REQUIRED_MANIFEST_FIELDS = [
|
|||
'compatibility',
|
||||
'agents',
|
||||
'coordinator',
|
||||
'environment',
|
||||
'helperState',
|
||||
'payloads',
|
||||
] as const;
|
||||
|
||||
|
|
|
|||
|
|
@ -197,8 +197,9 @@ fn checkpoint_envelope() -> String {
|
|||
"priorInspection": "prior-inspection",
|
||||
"executorState": [{"agentId": "fly-a", "payload": "executor-fly-a"}],
|
||||
"admissionState": null,
|
||||
"eventWatermarks": {"lastEventId": "evt-1", "lastOrdinal": "7"},
|
||||
"eventWatermarks": {"lastSourceStep": "42", "issued": "7"},
|
||||
},
|
||||
"environment": {"workerId": "arena", "payload": "world"},
|
||||
"helperState": [],
|
||||
"payloads": payload_table(),
|
||||
});
|
||||
|
|
|
|||
|
|
@ -59,10 +59,14 @@
|
|||
],
|
||||
"admissionState": null,
|
||||
"eventWatermarks": {
|
||||
"lastEventId": "evt-1",
|
||||
"lastOrdinal": "7"
|
||||
"lastSourceStep": "42",
|
||||
"issued": "7"
|
||||
}
|
||||
},
|
||||
"environment": {
|
||||
"workerId": "arena",
|
||||
"payload": "world"
|
||||
},
|
||||
"helperState": [],
|
||||
"payloads": [
|
||||
{
|
||||
|
|
@ -115,49 +119,49 @@
|
|||
}
|
||||
],
|
||||
"envelope": {
|
||||
"base64": "RkxZU0VTUzEBAAAAIAAAAAAIAAAFAAAAIAgAAAAAAAB7ImFnZW50cyI6W3siYWdlbnRJZCI6ImZseS1hIiwiYnJhaW5UaWNrcyI6IjI1MzQiLCJkYXRhc2V0RGlnZXN0IjoiNmMwYWYxZjA3ODRlZjYzYTM5M2VlNzdkNjE0ZTgyNDZjNjI1MDUxMzYwZjNmMWE0ODgzODM3NGM1ZDM1NWI1MiIsIm1vZGVsVmVyc2lvbiI6ImxpZi0xbXMtZjY0LXYyIiwicGF5bG9hZCI6ImFnZW50LWZseS1hIiwicGxhc3RpY2l0eVZlcnNpb24iOiJmbHkta2MtbWJvbi1yc3RkcC12MiIsInByb2ZpbGVEaWdlc3QiOiIxOTAwZWFiNmMwMjg0ODNkNzEyNjU5OWVlNmY1MGRlMGQyNzkwN2I1YzY1ZmE5MDUyNDU4MGI0YjBmOTg1MmIwIiwicmVtYWluZGVyIjp7ImRlbm9taW5hdG9yIjoiMyIsIm51bWVyYXRvciI6IjEwMDAwMDAifSwic2VlZCI6LTE4NDk0NjA2M31dLCJjaGVja3BvaW50SWQiOiJja3B0LTEiLCJjb21wYXRpYmlsaXR5Ijp7ImJhY2tlbmREaWdlc3QiOiIxMGUwOGE0MTllODUwZWJhMWViYmExOGZkZDI4ZWI3ZWMxYjdlOGJhYTliY2MzYjk3M2UyYjg4OTFlYzcyNmJlIiwiY29udGVudERpZ2VzdCI6ImVkNzAwMmI0MzllOWFjODQ1ZjIyMzU3ZDgyMmJhYzE0NDQ3MzBmYmRiNjAxNmQzZWM5NDMyMjk3YjllYzlmNzMiLCJjb250cm9sbGVyRGlnZXN0IjoiYzE0NzIxMzViMTRjNzdjOGJlZjk4ZTczZjcwMjA4MzI1ZmEwZGNmMWU2YmQ2NjhhZTliMzFhOWNlYTI5NWZlNyIsInBhcnNlckRpZ2VzdCI6ImIxN2Q0NTEyMTE1MDkyOGYyMTQ2YWY0OWUxOTVlZmYxZWVmNWQ2NzMyNWJlMjczYTczM2ZiNzRhY2FkYWEzNDIiLCJwYXRjaERpZ2VzdCI6ImE0ODk1ZWI0NGFmYzMzNmZlY2JiYTZlNTIwY2Q2N2UxNzhkYWNlMDI3NjY1NWQxMDJmY2VmZmE4ZTVmNzA1NzAiLCJzdGF0ZUZvcm1hdElkIjoiZmx5c2Vzcy0xIn0sImNvbXBvc2l0aW9uRGlnZXN0IjoiNzMwZDcyNWM4YTU5ZDNhNzMwM2RlZjJiZWQwNDFhNTc3ZWRiNDI1NWFhYmQ0ODg5Y2UxMjkxODMxMWQ5NTJmMCIsImNvb3JkaW5hdG9yIjp7ImFkbWlzc2lvblN0YXRlIjpudWxsLCJldmVudFdhdGVybWFya3MiOnsibGFzdEV2ZW50SWQiOiJldnQtMSIsImxhc3RPcmRpbmFsIjoiNyJ9LCJleGVjdXRvclN0YXRlIjpbeyJhZ2VudElkIjoiZmx5LWEiLCJwYXlsb2FkIjoiZXhlY3V0b3ItZmx5LWEifV0sInByaW9ySW5zcGVjdGlvbiI6InByaW9yLWluc3BlY3Rpb24iLCJ0YXNrTGVkZ2VyIjoidGFzay1sZWRnZXIifSwiZW52ZWxvcGVWZXJzaW9uIjoxLCJlcGlzb2RlSWQiOiJlcGlzb2RlLTEiLCJoZWxwZXJTdGF0ZSI6W10sInBheWxvYWRzIjpbeyJieXRlTGVuZ3RoIjoiMTciLCJkaWdlc3QiOiIxMzIxZGZmYjBjZGM2ZjkwOTJjYmY3ZmEyYTVmYzY4YmJlZDEyYzk5M2Q1YWQzOTgyNjQwMTI4MTBjZTliZjkzIiwibmFtZSI6ImFnZW50LWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTQiLCJkaWdlc3QiOiIzYWVlNjBkZjdlMjllZmViYTdmNWY5OWZjNTg2NzY0N2IzNmFlYmZmMWQ1ZDNjODM4ZGJmZjMyMzEyMmU2NDYyIiwibmFtZSI6ImV4ZWN1dG9yLWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTEiLCJkaWdlc3QiOiI0MGIwMGVkMmJiYmE5MDFkNjgyMDVmZjcxYjA0YTQ0YjllZTUzYzUxY2IzMTA5YWEyY2VhYTQ0ZjFjNDU3MjdlIiwibmFtZSI6InRhc2stbGVkZ2VyIn0seyJieXRlTGVuZ3RoIjoiMTAiLCJkaWdlc3QiOiIyYzEzYjdiNGQ5YTk5MTY4MDFhYjkxOTFjMzE0ZjMxYjA0NWU5YjljNWI2NjlhNmMwNDc0ZjAyMTdlZjc1YmY1IiwibmFtZSI6InByaW9yLWluc3BlY3Rpb24ifSx7ImJ5dGVMZW5ndGgiOiI2NCIsImRpZ2VzdCI6ImY1YTVmZDQyZDE2YTIwMzAyNzk4ZWY2ZWQzMDk5NzliNDMwMDNkMjMyMGQ5ZjBlOGVhOTgzMWE5Mjc1OWZiNGIiLCJuYW1lIjoid29ybGQifV0sInBvcnRNYXAiOlt7ImFnZW50SWQiOiJmbHktYSIsInBvcnRJZCI6InBvcnQtMSJ9XSwic2NoZWR1bGVySWQiOiJsb2Nrc3RlcC12MSIsInNvdXJjZVNjb3BlIjp7ImVwb2NoIjoiZXBvY2gtMSIsInNlc3Npb25JZCI6ImRlbW8iLCJzdGVwIjoiNDIifSwid29ybGRUaW1lIjp7ImRlbm9taW5hdG9yIjoiMSIsIm51bWVyYXRvciI6IjcwMDAwMDAwMCJ9fWFnZW50LWZseS1hAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABQCgAAAAAAABEAAAAAAAAAEyHf+wzcb5CSy/f6Kl/Gi77RLJk9WtOYJkASgQzpv5NleGVjdXRvci1mbHktYQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaAoAAAAAAAAOAAAAAAAAADruYN9+Ke/rp/X5n8WGdkezauv/HV08g42/8yMSLmRidGFzay1sZWRnZXIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHgKAAAAAAAACwAAAAAAAABAsA7Su7qQHWggX/cbBKRLnuU8UcsxCaos6qRPHEVyfnByaW9yLWluc3BlY3Rpb24AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACICgAAAAAAAAoAAAAAAAAALBO3tNmpkWgBq5GRwxTzGwRem5xbZppsBHTwIX73W/V3b3JsZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAmAoAAAAAAABAAAAAAAAAAPWl/ULRaiAwJ5jvbtMJl5tDAD0jINnw6OqYMaknWftLYWdlbnQgc3RhdGUgYnl0ZXMAAAAAAAAAZXhlY3V0b3Igc3RhdGUAAHsicmFuayI6MTB9AAAAAAB7Im1hcCI6NDB9AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgLAAAAAAAAq++fEx+FvDZho/eB4imbENN4HZrGNC2OCAsI7/gp9r5GTFlTRVNTRg==",
|
||||
"byteLength": 2824,
|
||||
"base64": "RkxZU0VTUzEBAAAAIAAAADAIAAAFAAAAUAgAAAAAAAB7ImFnZW50cyI6W3siYWdlbnRJZCI6ImZseS1hIiwiYnJhaW5UaWNrcyI6IjI1MzQiLCJkYXRhc2V0RGlnZXN0IjoiNmMwYWYxZjA3ODRlZjYzYTM5M2VlNzdkNjE0ZTgyNDZjNjI1MDUxMzYwZjNmMWE0ODgzODM3NGM1ZDM1NWI1MiIsIm1vZGVsVmVyc2lvbiI6ImxpZi0xbXMtZjY0LXYyIiwicGF5bG9hZCI6ImFnZW50LWZseS1hIiwicGxhc3RpY2l0eVZlcnNpb24iOiJmbHkta2MtbWJvbi1yc3RkcC12MiIsInByb2ZpbGVEaWdlc3QiOiIxOTAwZWFiNmMwMjg0ODNkNzEyNjU5OWVlNmY1MGRlMGQyNzkwN2I1YzY1ZmE5MDUyNDU4MGI0YjBmOTg1MmIwIiwicmVtYWluZGVyIjp7ImRlbm9taW5hdG9yIjoiMyIsIm51bWVyYXRvciI6IjEwMDAwMDAifSwic2VlZCI6LTE4NDk0NjA2M31dLCJjaGVja3BvaW50SWQiOiJja3B0LTEiLCJjb21wYXRpYmlsaXR5Ijp7ImJhY2tlbmREaWdlc3QiOiIxMGUwOGE0MTllODUwZWJhMWViYmExOGZkZDI4ZWI3ZWMxYjdlOGJhYTliY2MzYjk3M2UyYjg4OTFlYzcyNmJlIiwiY29udGVudERpZ2VzdCI6ImVkNzAwMmI0MzllOWFjODQ1ZjIyMzU3ZDgyMmJhYzE0NDQ3MzBmYmRiNjAxNmQzZWM5NDMyMjk3YjllYzlmNzMiLCJjb250cm9sbGVyRGlnZXN0IjoiYzE0NzIxMzViMTRjNzdjOGJlZjk4ZTczZjcwMjA4MzI1ZmEwZGNmMWU2YmQ2NjhhZTliMzFhOWNlYTI5NWZlNyIsInBhcnNlckRpZ2VzdCI6ImIxN2Q0NTEyMTE1MDkyOGYyMTQ2YWY0OWUxOTVlZmYxZWVmNWQ2NzMyNWJlMjczYTczM2ZiNzRhY2FkYWEzNDIiLCJwYXRjaERpZ2VzdCI6ImE0ODk1ZWI0NGFmYzMzNmZlY2JiYTZlNTIwY2Q2N2UxNzhkYWNlMDI3NjY1NWQxMDJmY2VmZmE4ZTVmNzA1NzAiLCJzdGF0ZUZvcm1hdElkIjoiZmx5c2Vzcy0xIn0sImNvbXBvc2l0aW9uRGlnZXN0IjoiNzMwZDcyNWM4YTU5ZDNhNzMwM2RlZjJiZWQwNDFhNTc3ZWRiNDI1NWFhYmQ0ODg5Y2UxMjkxODMxMWQ5NTJmMCIsImNvb3JkaW5hdG9yIjp7ImFkbWlzc2lvblN0YXRlIjpudWxsLCJldmVudFdhdGVybWFya3MiOnsiaXNzdWVkIjoiNyIsImxhc3RTb3VyY2VTdGVwIjoiNDIifSwiZXhlY3V0b3JTdGF0ZSI6W3siYWdlbnRJZCI6ImZseS1hIiwicGF5bG9hZCI6ImV4ZWN1dG9yLWZseS1hIn1dLCJwcmlvckluc3BlY3Rpb24iOiJwcmlvci1pbnNwZWN0aW9uIiwidGFza0xlZGdlciI6InRhc2stbGVkZ2VyIn0sImVudmVsb3BlVmVyc2lvbiI6MSwiZW52aXJvbm1lbnQiOnsicGF5bG9hZCI6IndvcmxkIiwid29ya2VySWQiOiJhcmVuYSJ9LCJlcGlzb2RlSWQiOiJlcGlzb2RlLTEiLCJoZWxwZXJTdGF0ZSI6W10sInBheWxvYWRzIjpbeyJieXRlTGVuZ3RoIjoiMTciLCJkaWdlc3QiOiIxMzIxZGZmYjBjZGM2ZjkwOTJjYmY3ZmEyYTVmYzY4YmJlZDEyYzk5M2Q1YWQzOTgyNjQwMTI4MTBjZTliZjkzIiwibmFtZSI6ImFnZW50LWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTQiLCJkaWdlc3QiOiIzYWVlNjBkZjdlMjllZmViYTdmNWY5OWZjNTg2NzY0N2IzNmFlYmZmMWQ1ZDNjODM4ZGJmZjMyMzEyMmU2NDYyIiwibmFtZSI6ImV4ZWN1dG9yLWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTEiLCJkaWdlc3QiOiI0MGIwMGVkMmJiYmE5MDFkNjgyMDVmZjcxYjA0YTQ0YjllZTUzYzUxY2IzMTA5YWEyY2VhYTQ0ZjFjNDU3MjdlIiwibmFtZSI6InRhc2stbGVkZ2VyIn0seyJieXRlTGVuZ3RoIjoiMTAiLCJkaWdlc3QiOiIyYzEzYjdiNGQ5YTk5MTY4MDFhYjkxOTFjMzE0ZjMxYjA0NWU5YjljNWI2NjlhNmMwNDc0ZjAyMTdlZjc1YmY1IiwibmFtZSI6InByaW9yLWluc3BlY3Rpb24ifSx7ImJ5dGVMZW5ndGgiOiI2NCIsImRpZ2VzdCI6ImY1YTVmZDQyZDE2YTIwMzAyNzk4ZWY2ZWQzMDk5NzliNDMwMDNkMjMyMGQ5ZjBlOGVhOTgzMWE5Mjc1OWZiNGIiLCJuYW1lIjoid29ybGQifV0sInBvcnRNYXAiOlt7ImFnZW50SWQiOiJmbHktYSIsInBvcnRJZCI6InBvcnQtMSJ9XSwic2NoZWR1bGVySWQiOiJsb2Nrc3RlcC12MSIsInNvdXJjZVNjb3BlIjp7ImVwb2NoIjoiZXBvY2gtMSIsInNlc3Npb25JZCI6ImRlbW8iLCJzdGVwIjoiNDIifSwid29ybGRUaW1lIjp7ImRlbm9taW5hdG9yIjoiMSIsIm51bWVyYXRvciI6IjcwMDAwMDAwMCJ9fWFnZW50LWZseS1hAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACACgAAAAAAABEAAAAAAAAAEyHf+wzcb5CSy/f6Kl/Gi77RLJk9WtOYJkASgQzpv5NleGVjdXRvci1mbHktYQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAmAoAAAAAAAAOAAAAAAAAADruYN9+Ke/rp/X5n8WGdkezauv/HV08g42/8yMSLmRidGFzay1sZWRnZXIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKgKAAAAAAAACwAAAAAAAABAsA7Su7qQHWggX/cbBKRLnuU8UcsxCaos6qRPHEVyfnByaW9yLWluc3BlY3Rpb24AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC4CgAAAAAAAAoAAAAAAAAALBO3tNmpkWgBq5GRwxTzGwRem5xbZppsBHTwIX73W/V3b3JsZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAyAoAAAAAAABAAAAAAAAAAPWl/ULRaiAwJ5jvbtMJl5tDAD0jINnw6OqYMaknWftLYWdlbnQgc3RhdGUgYnl0ZXMAAAAAAAAAZXhlY3V0b3Igc3RhdGUAAHsicmFuayI6MTB9AAAAAAB7Im1hcCI6NDB9AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADgLAAAAAAAAX4L9WkdX0MViD3h5YJgf7VQgwocWU4XJoKX6MUwl+8hGTFlTRVNTRg==",
|
||||
"byteLength": 2872,
|
||||
"layout": {
|
||||
"headerBytes": 32,
|
||||
"manifestOffset": "32",
|
||||
"manifestBytes": 2048,
|
||||
"tableOffset": "2080",
|
||||
"manifestBytes": 2096,
|
||||
"tableOffset": "2128",
|
||||
"tableEntryBytes": 112,
|
||||
"entries": [
|
||||
{
|
||||
"name": "agent-fly-a",
|
||||
"offset": "2640",
|
||||
"offset": "2688",
|
||||
"byteLength": "17",
|
||||
"digest": "1321dffb0cdc6f9092cbf7fa2a5fc68bbed12c993d5ad398264012810ce9bf93"
|
||||
},
|
||||
{
|
||||
"name": "executor-fly-a",
|
||||
"offset": "2664",
|
||||
"offset": "2712",
|
||||
"byteLength": "14",
|
||||
"digest": "3aee60df7e29efeba7f5f99fc5867647b36aebff1d5d3c838dbff323122e6462"
|
||||
},
|
||||
{
|
||||
"name": "task-ledger",
|
||||
"offset": "2680",
|
||||
"offset": "2728",
|
||||
"byteLength": "11",
|
||||
"digest": "40b00ed2bbba901d68205ff71b04a44b9ee53c51cb3109aa2ceaa44f1c45727e"
|
||||
},
|
||||
{
|
||||
"name": "prior-inspection",
|
||||
"offset": "2696",
|
||||
"offset": "2744",
|
||||
"byteLength": "10",
|
||||
"digest": "2c13b7b4d9a9916801ab9191c314f31b045e9b9c5b669a6c0474f0217ef75bf5"
|
||||
},
|
||||
{
|
||||
"name": "world",
|
||||
"offset": "2712",
|
||||
"offset": "2760",
|
||||
"byteLength": "64",
|
||||
"digest": "f5a5fd42d16a20302798ef6ed309979b43003d2320d9f0e8ea9831a92759fb4b"
|
||||
}
|
||||
],
|
||||
"footerOffset": "2776",
|
||||
"footerOffset": "2824",
|
||||
"footerBytes": 48,
|
||||
"totalBytes": "2824"
|
||||
"totalBytes": "2872"
|
||||
}
|
||||
},
|
||||
"corruption": [
|
||||
|
|
@ -178,17 +182,17 @@
|
|||
},
|
||||
{
|
||||
"name": "a flipped payload byte",
|
||||
"offset": 2640,
|
||||
"offset": 2688,
|
||||
"reason": "every payload carries its own digest"
|
||||
},
|
||||
{
|
||||
"name": "a flipped footer digest byte",
|
||||
"offset": 2784,
|
||||
"offset": 2832,
|
||||
"reason": "the footer digest must match the contents"
|
||||
},
|
||||
{
|
||||
"name": "a flipped footer magic byte",
|
||||
"offset": 2816,
|
||||
"offset": 2864,
|
||||
"reason": "a truncated file cannot look complete"
|
||||
}
|
||||
]
|
||||
|
|
|
|||
|
|
@ -296,6 +296,11 @@ pub fn decode(bytes: &[u8]) -> Result<Envelope> {
|
|||
|
||||
/// The manifest fields state-media-v1 section 4 requires, checked as a set: a manifest that
|
||||
/// omits one of them is not a complete checkpoint.
|
||||
///
|
||||
/// `helperState` and `environment` join the list under the 2026-09-22 amendment to
|
||||
/// checkpoint-envelope-v1 section 3: the first has been in that section's table from the
|
||||
/// start and was missing here, and the second is the holder of the world's own payload, which
|
||||
/// the table named for every other participant and not for the environment.
|
||||
pub const REQUIRED_MANIFEST_FIELDS: &[&str] = &[
|
||||
"envelopeVersion",
|
||||
"checkpointId",
|
||||
|
|
@ -308,6 +313,8 @@ pub const REQUIRED_MANIFEST_FIELDS: &[&str] = &[
|
|||
"compatibility",
|
||||
"agents",
|
||||
"coordinator",
|
||||
"environment",
|
||||
"helperState",
|
||||
"payloads",
|
||||
];
|
||||
|
||||
|
|
|
|||
|
|
@ -44,6 +44,7 @@ Ready(k) ─ Prepare all agents concurrently ───────────
|
|||
| `metrics` | Latency percentiles and the machine's core and memory counters |
|
||||
| `measure` | The execution-mode comparison of the guide's section 5 |
|
||||
| `cli` | The binary's subcommands: `agent`, `environment`, `measure` |
|
||||
| `state` | The durable checkpoint store over `FLYSESS1`: compatibility, generations, the bounded writer |
|
||||
| `harness` | The runnable composition: router, the flies, one arena, one coordinator |
|
||||
|
||||
## Execution modes and the launcher
|
||||
|
|
@ -178,6 +179,44 @@ harness.shutdown().await;
|
|||
event ids derived from epoch, source step, rule and ordinal.
|
||||
- **Executors.** The stateless identity executor only, as v1 specifies.
|
||||
|
||||
## Checkpoints and recovery
|
||||
|
||||
The durable store is `state`, over the `FLYSESS1` layout the contract crate owns.
|
||||
|
||||
- **One boundary, every participant.** `Coordinator::capture` runs at `Ready(k)` or
|
||||
`Paused(k)` only. It takes its queue slot *before* the first `State.Capture`, so a saturated
|
||||
writer refuses the capture rather than queueing it without bound, and the refusal is a
|
||||
`BUSY` a stepping session survives rather than an epoch failure.
|
||||
- **Capture and durability are two events.** `State.Capture` completes when an immutable
|
||||
capture exists; `Coordinator::await_durable` completes when the store manifest rename has
|
||||
happened, which is the durable commit point. Only the second moves the durable mark, and the
|
||||
three ways it can end without one are told apart: `Failed` (the write stopped),
|
||||
`ReplyLost` (the write finished and the acknowledgment did not arrive) and
|
||||
`DeadlineExpired` (the caller's own budget ran out while the save was still going).
|
||||
`Coordinator::resolve_durable` then asks the store about the *same* checkpoint instead of
|
||||
saving again.
|
||||
- **The writer is bounded twice**, and the two bounds refuse at different moments. The
|
||||
outstanding-capture bound is taken before a capture is requested; the byte budget cannot be,
|
||||
because a capture's size is not known until it exists, so it refuses at submit and releases
|
||||
the payloads with the refusal. The writer owns its payload handles until the bytes are
|
||||
committed or the job fails. A queued *replaceable* capture is superseded by a later one,
|
||||
releasing its holds; a durable one never is.
|
||||
- **The install is a group.** A restore selects a complete compatible generation, imports its
|
||||
payloads as fresh artifacts, stages every participant, validates the coordinator's own
|
||||
ledgers, and only then activates. A failure anywhere leaves the fence closed, and every
|
||||
participant that got as far as staging is recorded as one that must be replaced before
|
||||
another restore is attempted.
|
||||
- **The fence lifts once.** `Failed -> Restoring(k) -> Paused(k)`, at the end of a complete
|
||||
install and nowhere else. A fenced session takes no step, publishes nothing, captures
|
||||
nothing and holds no artifact handle.
|
||||
- **Nothing old crosses.** The fence drops every media handle; the restore imports fresh
|
||||
artifacts; the environment re-renders its pending sensor pipeline from recorded
|
||||
reconstruction inputs; and the new epoch's first audio chunk resumes the preserved sample
|
||||
position and marks the discontinuity.
|
||||
- **Epoch metadata in a trace.** `scope.epoch`, the batch id and every task event id are
|
||||
derived from the epoch, so a resumed run's behaviour is compared through
|
||||
`EpochRebase`, which rewrites exactly those and fails on anything it does not recognise.
|
||||
|
||||
## Where this crate narrows or adds to the contract crate
|
||||
|
||||
- **Required views.** `WorldObservation::validate_against` checks the views a result carries
|
||||
|
|
@ -196,9 +235,9 @@ harness.shutdown().await;
|
|||
|
||||
- **Fake workers.** There is no neural model and no emulator. What is modelled exactly is the
|
||||
ordering, the identity rules and the retry rules, not any numerical behaviour.
|
||||
- **No state methods.** `State.Capture`, `State.StageRestore` and `State.ActivateRestore` are
|
||||
STATE-01. The phase machine has their edges (`Capturing`, `Restoring`) and the workers do not
|
||||
advertise them as implemented methods.
|
||||
- **One environment, one task.** A checkpoint records the composition it was taken from, and a
|
||||
restore refuses one taken under another backend, content, patch, controller or parser
|
||||
identity. It does not migrate between compositions, and it does not try.
|
||||
- **No audience input.** The admitted pre-step stimulation list exists and is always empty.
|
||||
- **Pacing is coarse.** The pacing deadline rounds one step to whole nanoseconds for sleeping
|
||||
only; simulation time stays rational and that rounding never re-enters the accumulator.
|
||||
|
|
@ -259,6 +298,9 @@ The three integration suites do not all run over both transports, and cannot:
|
|||
- `tests/processes.rs` runs over the Unix socket only, in all three execution modes. A
|
||||
participant in a process of its own has no in-memory transport to reach the router by, so
|
||||
the mode is the axis that suite varies and the transport is fixed.
|
||||
- `tests/media.rs` and `tests/state.rs` run over both transports *and* in all three execution
|
||||
modes: each acceptance body is written once and registered twice, by `both_transports!` in
|
||||
the in-process composition and by `all_modes!` over the socket.
|
||||
|
||||
- `tests/session.rs`: one world advance per complete batch; every agent Prepared before the
|
||||
advance; one task evaluation per transition; every agent committed before the next Prepare or
|
||||
|
|
@ -275,6 +317,14 @@ The three integration suites do not all run over both transports, and cannot:
|
|||
allocation -- plus the sequential/reversed/parallel trace comparison across all three modes
|
||||
and the two process-mode section 4 rows: a router restart during a world advance, and an old
|
||||
worker's reply after a restart.
|
||||
- `tests/state.rs`: the STATE-01 acceptance bullets -- an uninterrupted run and a resumed run
|
||||
committing the same behaviour once the epoch metadata is rebased, a corrupt payload failing
|
||||
the install as a group for every participant and for the coordinator's own ledger, a lost
|
||||
save reply and an uncommitted store manifest both leaving the durable mark where it was, a
|
||||
refused activation resuming no part of the world, the capture queue staying bounded under a
|
||||
stalled writer, and old media and another parser's state failing to cross a recovery --
|
||||
plus the once-only restore token, the superseded replaceable capture, and the fence that
|
||||
lifts only through a complete restore.
|
||||
- `tests/failures.rs`: a duplicate Prepare after a lost reply; a duplicate Commit; the same
|
||||
batch with altered controls; a lost Advance result; a cached artifact consumed by its first
|
||||
caller; one Commit failing after another succeeded; a replaced registration; a reply from
|
||||
|
|
|
|||
|
|
@ -7,7 +7,7 @@
|
|||
//! stimulation, then reinforces once, and executes no tick at all. Every mutating step bumps
|
||||
//! one counter, which is how a test proves a duplicate request changed nothing.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::collections::{BTreeMap, BTreeSet};
|
||||
|
||||
use serde_json::Value;
|
||||
|
||||
|
|
@ -193,6 +193,12 @@ pub struct AgentFaults {
|
|||
pub prepare_delay_ms: u64,
|
||||
/// Hold `Agent.Commit` open for this long.
|
||||
pub commit_delay_ms: u64,
|
||||
/// Refuse `State.StageRestore`, so a group install meets one participant that will not
|
||||
/// validate while the others already have.
|
||||
pub fail_stage_restore: bool,
|
||||
/// Refuse `State.ActivateRestore` after this worker has already staged, so a group meets
|
||||
/// a failure halfway through activation.
|
||||
pub fail_activate_restore: bool,
|
||||
}
|
||||
|
||||
/// One fake agent worker's configuration.
|
||||
|
|
@ -234,6 +240,10 @@ pub struct FakeAgentWorker {
|
|||
context: Option<TypedValue>,
|
||||
context_digest: Option<Digest>,
|
||||
prepared: Option<(DomainRequestId, PreparedDecision)>,
|
||||
/// A validated replacement state that the live session cannot see yet.
|
||||
staged: Option<StagedAgent>,
|
||||
/// Restore tokens this worker has activated. A token activates once.
|
||||
activated: BTreeSet<Id>,
|
||||
}
|
||||
|
||||
impl FakeAgentWorker {
|
||||
|
|
@ -248,10 +258,17 @@ impl FakeAgentWorker {
|
|||
context: None,
|
||||
context_digest: None,
|
||||
prepared: None,
|
||||
staged: None,
|
||||
activated: BTreeSet::new(),
|
||||
config,
|
||||
}
|
||||
}
|
||||
|
||||
/// True while a validated replacement state is staged and not yet activated.
|
||||
pub fn has_staged_restore(&self) -> bool {
|
||||
self.staged.is_some()
|
||||
}
|
||||
|
||||
pub fn status(&self) -> StatusCell {
|
||||
self.status.clone()
|
||||
}
|
||||
|
|
@ -657,7 +674,11 @@ impl WorkerEndpoint for FakeAgentWorker {
|
|||
}
|
||||
|
||||
fn capabilities(&self) -> Vec<Id> {
|
||||
vec![id("agent-step-v1"), id("pixel-observation-v1")]
|
||||
vec![
|
||||
id("agent-step-v1"),
|
||||
id("pixel-observation-v1"),
|
||||
id(crate::state::CHECKPOINT_CAPABILITY),
|
||||
]
|
||||
}
|
||||
|
||||
fn status_cell(&self) -> StatusCell {
|
||||
|
|
@ -669,7 +690,14 @@ impl WorkerEndpoint for FakeAgentWorker {
|
|||
}
|
||||
|
||||
fn methods(&self) -> Vec<&'static str> {
|
||||
vec!["Agent.Initialize", "Agent.Prepare", "Agent.Commit"]
|
||||
vec![
|
||||
"Agent.Initialize",
|
||||
"Agent.Prepare",
|
||||
"Agent.Commit",
|
||||
"State.Capture",
|
||||
"State.StageRestore",
|
||||
"State.ActivateRestore",
|
||||
]
|
||||
}
|
||||
|
||||
fn handle<'a>(&'a mut self, ctx: HandlerCtx<'a>) -> BoxFuture<'a, DomainResult<HandlerReply>> {
|
||||
|
|
@ -678,6 +706,9 @@ impl WorkerEndpoint for FakeAgentWorker {
|
|||
"Agent.Initialize" => self.initialize(&ctx).await,
|
||||
"Agent.Prepare" => self.prepare(&ctx).await,
|
||||
"Agent.Commit" => self.commit(&ctx).await,
|
||||
"State.Capture" => self.state_capture(&ctx).await,
|
||||
"State.StageRestore" => self.state_stage_restore(&ctx).await,
|
||||
"State.ActivateRestore" => self.state_activate_restore(&ctx).await,
|
||||
other => Err(DomainError::before(
|
||||
ErrorCode::Unsupported,
|
||||
format!("{other} is not an agent method"),
|
||||
|
|
@ -690,7 +721,12 @@ impl WorkerEndpoint for FakeAgentWorker {
|
|||
/// The retention class table an agent endpoint follows, for a caller that wants it.
|
||||
pub fn agent_op_class(method: &str) -> Option<OpClass> {
|
||||
match method {
|
||||
"Agent.Initialize" => Some(OpClass::Lifecycle),
|
||||
// `ipc-v1` section 5: lifecycle *and capture* replies are retained until
|
||||
// `Worker.Acknowledge`, which is also what lets a duplicate restore request replay
|
||||
// its cached reply rather than staging or activating twice.
|
||||
"Agent.Initialize" | "State.Capture" | "State.StageRestore" | "State.ActivateRestore" => {
|
||||
Some(OpClass::Lifecycle)
|
||||
}
|
||||
"Agent.Prepare" | "Agent.Commit" => Some(OpClass::StepMutation),
|
||||
_ => None,
|
||||
}
|
||||
|
|
@ -712,3 +748,532 @@ pub fn synthetic_profile(agent_id: &Id, tick_duration: &RationalNs, warmup_ticks
|
|||
|
||||
/// The per-agent contexts a bootstrap produced, keyed by agent id.
|
||||
pub type Contexts = BTreeMap<Id, TypedValue>;
|
||||
|
||||
// -------------------------------------------------------------------------------------------
|
||||
// STATE-01: capture and restore
|
||||
|
||||
/// The numerical model version this worker implements. It is part of a capture's
|
||||
/// compatibility identity: the same profile and seed under another model is not the same
|
||||
/// state (`workers-v1` section 2).
|
||||
pub const MODEL_VERSION: &str = "fake-lcg-v1";
|
||||
|
||||
/// The plasticity rule version, for the same reason.
|
||||
pub const PLASTICITY_VERSION: &str = "fake-reinforce-v1";
|
||||
|
||||
/// The version this payload layout is written and read under.
|
||||
pub const AGENT_PAYLOAD_VERSION: u64 = 1;
|
||||
|
||||
/// The dataset identity a synthetic agent resolves.
|
||||
///
|
||||
/// There is no connectome dataset behind this worker, and a checkpoint says so with a stable
|
||||
/// identity rather than omitting the field: "no dataset" has to be distinguishable from "the
|
||||
/// dataset was not recorded".
|
||||
pub fn dataset_digest() -> Digest {
|
||||
digest_of_bytes(b"fly-session/no-dataset-v1")
|
||||
}
|
||||
|
||||
/// The capture compatibility digest of one agent (`workers-v1` section 2).
|
||||
///
|
||||
/// The profile digest identifies the profile definition; this additionally covers the
|
||||
/// resolved seed, the numerical model version and the plasticity rule, because two agents
|
||||
/// with the same profile digest and different seeds hold state that is not interchangeable.
|
||||
/// Every field it covers is one the checkpoint manifest already records in that agent's row,
|
||||
/// so a restore derives the expected digest from the manifest rather than from the payload it
|
||||
/// is about to validate.
|
||||
pub fn agent_compatibility_digest(
|
||||
agent_id: &Id,
|
||||
profile_digest: &Digest,
|
||||
dataset_digest: &Digest,
|
||||
model_version: &str,
|
||||
plasticity_version: &str,
|
||||
seed: i32,
|
||||
) -> Digest {
|
||||
let value = serde_json::json!({
|
||||
"agentId": agent_id.as_str(),
|
||||
"profileDigest": profile_digest.as_str(),
|
||||
"datasetDigest": dataset_digest.as_str(),
|
||||
"modelVersion": model_version,
|
||||
"plasticityVersion": plasticity_version,
|
||||
"seed": seed,
|
||||
});
|
||||
digest_of(&value).expect("an agent compatibility block canonicalizes")
|
||||
}
|
||||
|
||||
impl FakeModel {
|
||||
/// Every field of the model, so a resumed agent is this agent and not a fresh one.
|
||||
fn capture(&self) -> Value {
|
||||
serde_json::json!({
|
||||
"seed": self.seed,
|
||||
"state": self.state.to_string(),
|
||||
"mutations": self.mutations.to_string(),
|
||||
"ticks": self.ticks.to_string(),
|
||||
"stimulations": self.stimulations.to_string(),
|
||||
"reinforcements": self.reinforcements.to_string(),
|
||||
"learningEnabled": self.learning_enabled,
|
||||
"learningUpdates": self.learning_updates.to_string(),
|
||||
"learningChanged": self.learning_changed.to_string(),
|
||||
"lastSignal": self.last_signal,
|
||||
"inputValue": self.input_value.to_string(),
|
||||
"inputInstalls": self.input_installs.to_string(),
|
||||
})
|
||||
}
|
||||
|
||||
fn restored(value: &Value) -> DomainResult<FakeModel> {
|
||||
let number = |key: &str| -> DomainResult<u64> {
|
||||
value
|
||||
.get(key)
|
||||
.and_then(Value::as_str)
|
||||
.ok_or_else(|| incompatible(format!("the agent payload has no {key}")))?
|
||||
.parse::<u64>()
|
||||
.map_err(|_| incompatible(format!("the agent payload's {key} is not a U64")))
|
||||
};
|
||||
let seed = value
|
||||
.get("seed")
|
||||
.and_then(Value::as_i64)
|
||||
.and_then(|v| i32::try_from(v).ok())
|
||||
.ok_or_else(|| incompatible("the agent payload has no seed"))?;
|
||||
let input_value = value
|
||||
.get("inputValue")
|
||||
.and_then(Value::as_str)
|
||||
.ok_or_else(|| incompatible("the agent payload has no inputValue"))?
|
||||
.parse::<i64>()
|
||||
.map_err(|_| incompatible("the agent payload's inputValue is not an integer"))?;
|
||||
let last_signal = value
|
||||
.get("lastSignal")
|
||||
.and_then(Value::as_f64)
|
||||
.filter(|v| v.is_finite())
|
||||
.ok_or_else(|| incompatible("the agent payload's lastSignal is not finite"))?;
|
||||
let learning_enabled = value
|
||||
.get("learningEnabled")
|
||||
.and_then(Value::as_bool)
|
||||
.ok_or_else(|| incompatible("the agent payload has no learningEnabled"))?;
|
||||
Ok(FakeModel {
|
||||
seed,
|
||||
state: number("state")?,
|
||||
mutations: number("mutations")?,
|
||||
ticks: number("ticks")?,
|
||||
stimulations: number("stimulations")?,
|
||||
reinforcements: number("reinforcements")?,
|
||||
learning_enabled,
|
||||
learning_updates: number("learningUpdates")?,
|
||||
learning_changed: number("learningChanged")?,
|
||||
last_signal,
|
||||
input_value,
|
||||
input_installs: number("inputInstalls")?,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
fn incompatible(message: impl std::fmt::Display) -> DomainError {
|
||||
DomainError::before(ErrorCode::IncompatibleState, message)
|
||||
}
|
||||
|
||||
/// One staged restore, held outside the live agent until it is activated.
|
||||
struct StagedAgent {
|
||||
token: Id,
|
||||
checkpoint_id: Id,
|
||||
scope: Scope,
|
||||
model: FakeModel,
|
||||
accumulator: TickAccumulator,
|
||||
context: TypedValue,
|
||||
profile: AssetRef,
|
||||
committed_step: u64,
|
||||
}
|
||||
|
||||
impl FakeAgentWorker {
|
||||
/// This worker's own compatibility identity, from its configuration and a resolved seed.
|
||||
fn compatibility_digest(&self, profile: &AssetRef, seed: i32) -> Digest {
|
||||
agent_compatibility_digest(
|
||||
&self.config.agent_id,
|
||||
&profile.digest,
|
||||
&dataset_digest(),
|
||||
MODEL_VERSION,
|
||||
PLASTICITY_VERSION,
|
||||
seed,
|
||||
)
|
||||
}
|
||||
|
||||
/// `State.Capture`: an immutable snapshot of this agent at its committed boundary.
|
||||
///
|
||||
/// It is allowed at `Ready(k)` only. A Prepared agent holds half a transition, and there
|
||||
/// is no coherent boundary to file that under.
|
||||
async fn state_capture(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult<HandlerReply> {
|
||||
let scope = ctx.scope()?.clone();
|
||||
self.check_epoch(&scope)?;
|
||||
let AgentPhase::Ready(k) = self.phase.clone() else {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::InvalidPhase,
|
||||
format!(
|
||||
"State.Capture needs a quiescent Ready(k); this worker is {:?}",
|
||||
self.phase
|
||||
),
|
||||
));
|
||||
};
|
||||
if scope.step != k {
|
||||
return Err(DomainError::before(
|
||||
if scope.step < k { ErrorCode::StaleStep } else { ErrorCode::FutureStep },
|
||||
"State.Capture names a boundary this worker is not at",
|
||||
));
|
||||
}
|
||||
let params: CaptureParams = ctx.params()?;
|
||||
let profile = self.profile.clone().expect("initialized");
|
||||
let context = self.context.clone().expect("initialized");
|
||||
let accumulator = self.accumulator.as_ref().expect("initialized");
|
||||
let previous = self.status.state();
|
||||
self.status.set_state(WorkerState::Capturing);
|
||||
let payload = serde_json::json!({
|
||||
"payloadVersion": AGENT_PAYLOAD_VERSION,
|
||||
"kind": "agent",
|
||||
"agentId": self.config.agent_id.as_str(),
|
||||
"checkpointId": params.checkpoint_id.as_str(),
|
||||
"sourceScope": scope.to_json(),
|
||||
"committedStep": k.to_string(),
|
||||
"profile": profile.to_json(),
|
||||
"modelVersion": MODEL_VERSION,
|
||||
"plasticityVersion": PLASTICITY_VERSION,
|
||||
"datasetDigest": dataset_digest().as_str(),
|
||||
"model": self.model.capture(),
|
||||
"accumulator": {
|
||||
"tickDuration": accumulator.tick_duration().to_json(),
|
||||
"remainder": accumulator.remainder().to_json(),
|
||||
"executedTicks": accumulator.executed_ticks().to_string(),
|
||||
"warmupOffset": accumulator.warmup_offset().to_string(),
|
||||
},
|
||||
"context": context.to_json(),
|
||||
});
|
||||
let bytes = canonicalize(&payload)
|
||||
.map_err(|e| DomainError::invalid(format!("State.Capture: {}", e.0)))?
|
||||
.into_bytes();
|
||||
let digest = digest_of_bytes(&bytes);
|
||||
let artifact = crate::state::seal_payload(ctx.client, &bytes, &digest).await?;
|
||||
// Capture is a read of the model, not a mutation of it: nothing above changed a
|
||||
// counter, and the worker goes back to the boundary it was already at.
|
||||
self.status.set_state(previous);
|
||||
let result = CaptureResult {
|
||||
checkpoint_id: params.checkpoint_id,
|
||||
boundary: k,
|
||||
compatibility_digest: self.compatibility_digest(&profile, self.model.seed()),
|
||||
payload: artifact.reference().clone(),
|
||||
};
|
||||
Ok(HandlerReply::with_artifacts(
|
||||
object(result.to_json()),
|
||||
vec![(crate::state::PAYLOAD_ATTACHMENT.to_owned(), artifact)],
|
||||
))
|
||||
}
|
||||
|
||||
/// `State.StageRestore`: validate a replacement state into a staging slot.
|
||||
///
|
||||
/// Nothing the live session can see changes here, and the worker keeps whatever state it
|
||||
/// had. It is allowed on an uninitialized replacement or a quiescent worker only; a
|
||||
/// failed one is neither, which is why a group that failed is replaced rather than
|
||||
/// reused.
|
||||
async fn state_stage_restore(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult<HandlerReply> {
|
||||
let scope = ctx.scope()?.clone();
|
||||
if scope.session_id != self.config.session_id {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::IdentityMismatch,
|
||||
"this worker belongs to another session",
|
||||
));
|
||||
}
|
||||
match &self.phase {
|
||||
AgentPhase::Uninitialized | AgentPhase::Ready(_) => {}
|
||||
other => {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::InvalidPhase,
|
||||
format!(
|
||||
"State.StageRestore needs an uninitialized replacement or a quiescent \
|
||||
worker; this worker is {other:?}"
|
||||
),
|
||||
));
|
||||
}
|
||||
}
|
||||
if let Some(epoch) = &self.epoch
|
||||
&& *epoch == scope.epoch
|
||||
{
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::StaleEpoch,
|
||||
"State.StageRestore proposes the epoch this worker is already running",
|
||||
));
|
||||
}
|
||||
let params: StageRestoreParams = ctx.params()?;
|
||||
if params.source_scope.step != scope.step {
|
||||
return Err(DomainError::invalid(
|
||||
"State.StageRestore's scope step must be the source boundary",
|
||||
));
|
||||
}
|
||||
let artifact = ctx.artifact(crate::state::PAYLOAD_ATTACHMENT)?;
|
||||
if artifact.reference() != ¶ms.payload {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::BufferInvalid,
|
||||
"the staged payload attachment is not the artifact the request names",
|
||||
));
|
||||
}
|
||||
let bytes = artifact.read_all().await.map_err(|e| {
|
||||
DomainError::before(
|
||||
ErrorCode::BufferInvalid,
|
||||
format!("the staged payload could not be read: {}", e.message),
|
||||
)
|
||||
})?;
|
||||
let declared = params
|
||||
.payload
|
||||
.digest
|
||||
.clone()
|
||||
.ok_or_else(|| incompatible("a checkpoint payload must carry a content digest"))?;
|
||||
let actual = digest_of_bytes(&bytes);
|
||||
if actual != declared || bytes.len() as u64 != params.payload.byte_length {
|
||||
return Err(incompatible(
|
||||
"the staged payload is not the content the request declares",
|
||||
));
|
||||
}
|
||||
let value: Value = serde_json::from_slice(&bytes)
|
||||
.map_err(|e| DomainError::invalid(format!("the staged payload is not JSON: {e}")))?;
|
||||
let text = |key: &str| -> DomainResult<String> {
|
||||
value
|
||||
.get(key)
|
||||
.and_then(Value::as_str)
|
||||
.map(str::to_owned)
|
||||
.ok_or_else(|| incompatible(format!("the agent payload has no {key}")))
|
||||
};
|
||||
if value.get("payloadVersion").and_then(Value::as_u64) != Some(AGENT_PAYLOAD_VERSION) {
|
||||
return Err(incompatible("the agent payload is another payload version"));
|
||||
}
|
||||
if text("kind")? != "agent" {
|
||||
return Err(incompatible("this payload is not an agent's state"));
|
||||
}
|
||||
if text("agentId")? != self.config.agent_id {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::IdentityMismatch,
|
||||
"the staged payload belongs to another agent",
|
||||
));
|
||||
}
|
||||
if text("checkpointId")? != params.checkpoint_id {
|
||||
return Err(incompatible("the staged payload belongs to another checkpoint"));
|
||||
}
|
||||
if text("modelVersion")? != MODEL_VERSION || text("plasticityVersion")? != PLASTICITY_VERSION
|
||||
{
|
||||
return Err(incompatible(
|
||||
"the staged payload was captured under another numerical model",
|
||||
));
|
||||
}
|
||||
let source_scope = Scope::from_json(
|
||||
value
|
||||
.get("sourceScope")
|
||||
.ok_or_else(|| incompatible("the agent payload has no sourceScope"))?,
|
||||
)
|
||||
.map_err(|e| incompatible(format!("the agent payload's sourceScope: {}", e.0)))?;
|
||||
if source_scope != params.source_scope {
|
||||
return Err(incompatible(
|
||||
"the staged payload was captured at another source scope",
|
||||
));
|
||||
}
|
||||
let committed_step: u64 = text("committedStep")?
|
||||
.parse()
|
||||
.map_err(|_| incompatible("the agent payload's committedStep is not a U64"))?;
|
||||
if committed_step != params.source_scope.step {
|
||||
return Err(incompatible(
|
||||
"the staged payload's committed step is not the source boundary",
|
||||
));
|
||||
}
|
||||
let profile = AssetRef::from_json(
|
||||
value
|
||||
.get("profile")
|
||||
.ok_or_else(|| incompatible("the agent payload has no profile"))?,
|
||||
)
|
||||
.map_err(|e| incompatible(format!("the agent payload's profile: {}", e.0)))?;
|
||||
let model = FakeModel::restored(
|
||||
value
|
||||
.get("model")
|
||||
.ok_or_else(|| incompatible("the agent payload has no model"))?,
|
||||
)?;
|
||||
// The compatibility digest is recomputed from this worker's own configuration and the
|
||||
// identity the payload declares. A capture of the same profile under another seed, or
|
||||
// of another agent's brain, fails here and never reaches activation.
|
||||
let computed = self.compatibility_digest(&profile, model.seed());
|
||||
if computed != params.compatibility_digest {
|
||||
return Err(incompatible(format!(
|
||||
"the staged state's compatibility {computed} is not the {} the restore \
|
||||
requires",
|
||||
params.compatibility_digest
|
||||
)));
|
||||
}
|
||||
let accumulator_value = value
|
||||
.get("accumulator")
|
||||
.ok_or_else(|| incompatible("the agent payload has no accumulator"))?;
|
||||
let rational = |key: &str| -> DomainResult<RationalNs> {
|
||||
RationalNs::from_json(
|
||||
accumulator_value
|
||||
.get(key)
|
||||
.ok_or_else(|| incompatible(format!("the accumulator has no {key}")))?,
|
||||
)
|
||||
.map_err(|e| incompatible(format!("the accumulator's {key}: {}", e.0)))
|
||||
};
|
||||
let counter = |key: &str| -> DomainResult<u64> {
|
||||
accumulator_value
|
||||
.get(key)
|
||||
.and_then(Value::as_str)
|
||||
.ok_or_else(|| incompatible(format!("the accumulator has no {key}")))?
|
||||
.parse::<u64>()
|
||||
.map_err(|_| incompatible(format!("the accumulator's {key} is not a U64")))
|
||||
};
|
||||
let tick_duration = rational("tickDuration")?;
|
||||
if tick_duration != self.config.tick_duration {
|
||||
return Err(incompatible(
|
||||
"the staged state was captured at another model tick duration",
|
||||
));
|
||||
}
|
||||
let accumulator = TickAccumulator::restored(
|
||||
tick_duration,
|
||||
rational("remainder")?,
|
||||
counter("executedTicks")?,
|
||||
counter("warmupOffset")?,
|
||||
)
|
||||
.map_err(incompatible)?;
|
||||
let context = TypedValue::from_json(
|
||||
value
|
||||
.get("context")
|
||||
.ok_or_else(|| incompatible("the agent payload has no context"))?,
|
||||
)
|
||||
.map_err(|e| incompatible(format!("the agent payload's context: {}", e.0)))?;
|
||||
FakeAgentWorker::available_actions(&context)?;
|
||||
|
||||
if self.config.faults.fail_stage_restore {
|
||||
// The row where a group validates three participants and the fourth does not.
|
||||
// Nothing is staged here and nothing is staged anywhere else either: the
|
||||
// coordinator abandons the whole install.
|
||||
return Err(incompatible(
|
||||
"injected staging refusal: this participant's replacement state does not \
|
||||
validate",
|
||||
));
|
||||
}
|
||||
// One staged restore at a time. A second proposal replaces nothing silently.
|
||||
if let Some(staged) = &self.staged {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::Conflict,
|
||||
format!(
|
||||
"this worker already holds the staged restore {} for checkpoint {}",
|
||||
staged.token, staged.checkpoint_id
|
||||
),
|
||||
));
|
||||
}
|
||||
let token = restore_token(¶ms.checkpoint_id, &scope, &actual, &self.config.incarnation_id);
|
||||
if self.activated.contains(&token) {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::Conflict,
|
||||
"this exact restore was already activated on this worker",
|
||||
));
|
||||
}
|
||||
self.staged = Some(StagedAgent {
|
||||
token: token.clone(),
|
||||
checkpoint_id: params.checkpoint_id.clone(),
|
||||
scope: scope.clone(),
|
||||
model,
|
||||
accumulator,
|
||||
context,
|
||||
profile,
|
||||
committed_step,
|
||||
});
|
||||
self.status.set_state(WorkerState::StagedRestore);
|
||||
let result = StageRestoreResult {
|
||||
checkpoint_id: params.checkpoint_id,
|
||||
restore_token: token,
|
||||
};
|
||||
Ok(HandlerReply::from(&result))
|
||||
}
|
||||
|
||||
/// `State.ActivateRestore`: install the staged state under its new scope, without a tick.
|
||||
///
|
||||
/// The token activates once. A duplicate domain request replays the cached reply through
|
||||
/// the shell's result cache; a fresh request naming an already activated token is a
|
||||
/// conflict, which is what stops a second group from being resumed from the same bytes.
|
||||
async fn state_activate_restore(
|
||||
&mut self,
|
||||
ctx: &HandlerCtx<'_>,
|
||||
) -> DomainResult<HandlerReply> {
|
||||
let params: ActivateRestoreParams = ctx.params()?;
|
||||
if self.activated.contains(¶ms.restore_token) {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::Conflict,
|
||||
"this restore token has already been activated",
|
||||
));
|
||||
}
|
||||
let Some(staged) = self.staged.take() else {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::InvalidPhase,
|
||||
"this worker holds no staged restore",
|
||||
));
|
||||
};
|
||||
if staged.token != params.restore_token {
|
||||
// Put it back: naming another token is not a reason to discard this one.
|
||||
let token = staged.token.clone();
|
||||
self.staged = Some(staged);
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::IdentityMismatch,
|
||||
format!("this worker's staged restore is {token}, not {}", params.restore_token),
|
||||
));
|
||||
}
|
||||
if self.config.faults.fail_activate_restore {
|
||||
let token = staged.token.clone();
|
||||
self.staged = Some(staged);
|
||||
self.status.set_state(WorkerState::Failed);
|
||||
return Err(DomainError::new(
|
||||
ErrorCode::BackendFailure,
|
||||
format!("injected activation failure; {token} stays staged and unresumed"),
|
||||
MutationCertainty::None,
|
||||
));
|
||||
}
|
||||
self.status.set_state(WorkerState::Restoring);
|
||||
let StagedAgent {
|
||||
token,
|
||||
checkpoint_id,
|
||||
scope,
|
||||
model,
|
||||
accumulator,
|
||||
context,
|
||||
profile,
|
||||
committed_step,
|
||||
} = staged;
|
||||
self.model = model;
|
||||
self.accumulator = Some(accumulator);
|
||||
self.context_digest = Some(context.digest());
|
||||
self.context = Some(context);
|
||||
self.profile = Some(profile);
|
||||
self.epoch = Some(scope.epoch.clone());
|
||||
self.prepared = None;
|
||||
self.phase = AgentPhase::Ready(committed_step);
|
||||
self.activated.insert(token);
|
||||
self.status.set_state(WorkerState::Ready);
|
||||
self.status.set_scope(Some(scope_at(
|
||||
&scope.session_id,
|
||||
&scope.epoch,
|
||||
committed_step,
|
||||
)));
|
||||
self.status.advance_to(self.model.mutations());
|
||||
let result = ActivateRestoreResult {
|
||||
committed_step,
|
||||
checkpoint_id,
|
||||
// An agent returns a null observation; the environment returns the world's.
|
||||
observation: None,
|
||||
};
|
||||
result
|
||||
.validate_for_role(Role::Agent)
|
||||
.map_err(|e| DomainError::invalid(e.0))?;
|
||||
Ok(HandlerReply::from(&result))
|
||||
}
|
||||
}
|
||||
|
||||
/// A restore token bound to the checkpoint, the proposed scope, the payload bytes and the
|
||||
/// worker incarnation staging them.
|
||||
///
|
||||
/// `state-media-v1` section 5 binds a token to scope, payload and checkpoint. Binding it to
|
||||
/// the incarnation as well is what keeps a token minted by a worker that has since been
|
||||
/// replaced from activating anything on its replacement.
|
||||
pub fn restore_token(checkpoint_id: &Id, scope: &Scope, payload_digest: &Digest, incarnation: &Id) -> Id {
|
||||
let digest = digest_of_bytes(
|
||||
format!(
|
||||
"fly-session/restore-token-v1\n{checkpoint_id}\n{}\n{}\n{}\n{payload_digest}\n{incarnation}\n",
|
||||
scope.session_id, scope.epoch, scope.step
|
||||
)
|
||||
.as_bytes(),
|
||||
);
|
||||
parse_id(&format!("rt-{}", &digest[..32])).expect("a hex suffix is an Id")
|
||||
}
|
||||
|
|
|
|||
|
|
@ -46,8 +46,10 @@ Worker options (agent and environment):
|
|||
agent: --agent ID --port ID --tick-numerator N --tick-denominator N
|
||||
--warmup-ticks N [--prepare-delay-ms N] [--commit-delay-ms N]
|
||||
[--fail-commit-at-step N]
|
||||
[--fail-stage-restore 0|1] [--fail-activate-restore 0|1]
|
||||
environment: --worker ID --ports p1,p2 --step-numerator N --step-denominator N
|
||||
[--advance-delay-ms N] [--omit-view-at-boundary N]
|
||||
[--fail-stage-restore 0|1] [--fail-activate-restore 0|1]
|
||||
|
||||
Measure options:
|
||||
--steps N transitions per run (default 200)
|
||||
|
|
@ -145,6 +147,17 @@ impl Options {
|
|||
}
|
||||
}
|
||||
|
||||
/// A flag whose value is `0` or `1`. Anything else is an error naming it, so a
|
||||
/// mistyped injection is a failed launch rather than a fault that never fires.
|
||||
fn flag(&self, name: &str) -> Result<bool, String> {
|
||||
match self.0.get(name) {
|
||||
None => Ok(false),
|
||||
Some(value) if value == "0" => Ok(false),
|
||||
Some(value) if value == "1" => Ok(true),
|
||||
Some(value) => Err(format!("--{name}: {value:?} is not 0 or 1")),
|
||||
}
|
||||
}
|
||||
|
||||
fn opt_u64(&self, name: &str) -> Result<Option<u64>, String> {
|
||||
match self.0.get(name) {
|
||||
None => Ok(None),
|
||||
|
|
@ -197,6 +210,8 @@ fn serve(role: &str, options: &Options) -> Result<(), String> {
|
|||
fail_commit_at_step: options.opt_u64(flags::FAIL_COMMIT_AT_STEP)?,
|
||||
prepare_delay_ms: options.u64(flags::PREPARE_DELAY_MS, 0)?,
|
||||
commit_delay_ms: options.u64(flags::COMMIT_DELAY_MS, 0)?,
|
||||
fail_stage_restore: options.flag(flags::FAIL_STAGE_RESTORE)?,
|
||||
fail_activate_restore: options.flag(flags::FAIL_ACTIVATE_RESTORE)?,
|
||||
},
|
||||
client_id: client_id.clone(),
|
||||
service: service.clone(),
|
||||
|
|
@ -219,6 +234,8 @@ fn serve(role: &str, options: &Options) -> Result<(), String> {
|
|||
omit_audio_at_boundary: options.opt_u64(flags::OMIT_AUDIO_AT_BOUNDARY)?,
|
||||
overlapping_audio_at_boundary: options
|
||||
.opt_u64(flags::OVERLAPPING_AUDIO_AT_BOUNDARY)?,
|
||||
fail_stage_restore: options.flag(flags::FAIL_STAGE_RESTORE)?,
|
||||
fail_activate_restore: options.flag(flags::FAIL_ACTIVATE_RESTORE)?,
|
||||
},
|
||||
client_id: client_id.clone(),
|
||||
service: service.clone(),
|
||||
|
|
|
|||
|
|
@ -29,6 +29,30 @@ impl TickAccumulator {
|
|||
})
|
||||
}
|
||||
|
||||
/// The exact accumulator a capture recorded.
|
||||
///
|
||||
/// The remainder is restored, never rounded or reset: a resumed agent that started its
|
||||
/// first interval from zero would drift away from the run it is supposed to continue.
|
||||
pub fn restored(
|
||||
tick_duration: RationalNs,
|
||||
remainder: RationalNs,
|
||||
executed_ticks: u64,
|
||||
warmup_offset: u64,
|
||||
) -> Result<TickAccumulator, String> {
|
||||
let mut accumulator = TickAccumulator::new(tick_duration)?;
|
||||
remainder.validate().map_err(|e| e.0)?;
|
||||
if remainder >= tick_duration {
|
||||
return Err("a captured remainder is not below one model tick".to_owned());
|
||||
}
|
||||
if warmup_offset > executed_ticks {
|
||||
return Err("a captured warm-up offset exceeds the executed tick count".to_owned());
|
||||
}
|
||||
accumulator.remainder = remainder;
|
||||
accumulator.executed_ticks = executed_ticks;
|
||||
accumulator.warmup_offset = warmup_offset;
|
||||
Ok(accumulator)
|
||||
}
|
||||
|
||||
pub fn tick_duration(&self) -> RationalNs {
|
||||
self.tick_duration
|
||||
}
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -13,6 +13,8 @@
|
|||
|
||||
use std::collections::BTreeSet;
|
||||
|
||||
use serde_json::Value;
|
||||
|
||||
use crate::media::{self, AudioSource, RenderCounter, ViewPipeline};
|
||||
use crate::task::{controller_schema_ref, inspection, inspection_schema};
|
||||
// `crate::types` is this crate's facade over the shared `fly-session-types` crate; the
|
||||
|
|
@ -48,6 +50,12 @@ pub struct EnvironmentFaults {
|
|||
pub omit_audio_at_boundary: Option<u64>,
|
||||
/// Emit an audio chunk that starts before the previous chunk ended.
|
||||
pub overlapping_audio_at_boundary: Option<u64>,
|
||||
/// Refuse `State.StageRestore`, so a group install meets a participant that will not
|
||||
/// validate.
|
||||
pub fail_stage_restore: bool,
|
||||
/// Refuse `State.ActivateRestore` after staging, so a group meets a failure halfway
|
||||
/// through activation.
|
||||
pub fail_activate_restore: bool,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
|
|
@ -86,6 +94,10 @@ pub struct CounterEnvironment {
|
|||
audio: Option<AudioSource>,
|
||||
/// The frame served at the previous boundary, kept only so a fault can serve it again.
|
||||
previous_view: Option<(ViewRef, flybus::Artifact)>,
|
||||
/// A validated replacement world the live session cannot see yet.
|
||||
staged: Option<StagedWorld>,
|
||||
/// Restore tokens this world has activated. A token activates once.
|
||||
activated: BTreeSet<Id>,
|
||||
}
|
||||
|
||||
impl CounterEnvironment {
|
||||
|
|
@ -104,10 +116,17 @@ impl CounterEnvironment {
|
|||
pipeline: None,
|
||||
audio: None,
|
||||
previous_view: None,
|
||||
staged: None,
|
||||
activated: BTreeSet::new(),
|
||||
config,
|
||||
}
|
||||
}
|
||||
|
||||
/// True while a validated replacement world is staged and not yet activated.
|
||||
pub fn has_staged_restore(&self) -> bool {
|
||||
self.staged.is_some()
|
||||
}
|
||||
|
||||
pub fn status(&self) -> StatusCell {
|
||||
self.status.clone()
|
||||
}
|
||||
|
|
@ -477,7 +496,7 @@ impl WorkerEndpoint for CounterEnvironment {
|
|||
vec![
|
||||
id("world-step-v1"),
|
||||
id("pixel-observation-v1"),
|
||||
id("checkpoint-v1"),
|
||||
id(crate::state::CHECKPOINT_CAPABILITY),
|
||||
]
|
||||
}
|
||||
|
||||
|
|
@ -490,7 +509,13 @@ impl WorkerEndpoint for CounterEnvironment {
|
|||
}
|
||||
|
||||
fn methods(&self) -> Vec<&'static str> {
|
||||
vec!["Environment.Initialize", "Environment.Advance"]
|
||||
vec![
|
||||
"Environment.Initialize",
|
||||
"Environment.Advance",
|
||||
"State.Capture",
|
||||
"State.StageRestore",
|
||||
"State.ActivateRestore",
|
||||
]
|
||||
}
|
||||
|
||||
fn handle<'a>(&'a mut self, ctx: HandlerCtx<'a>) -> BoxFuture<'a, DomainResult<HandlerReply>> {
|
||||
|
|
@ -498,6 +523,9 @@ impl WorkerEndpoint for CounterEnvironment {
|
|||
match ctx.method {
|
||||
"Environment.Initialize" => self.initialize(&ctx).await,
|
||||
"Environment.Advance" => self.advance(&ctx).await,
|
||||
"State.Capture" => self.state_capture(&ctx).await,
|
||||
"State.StageRestore" => self.state_stage_restore(&ctx).await,
|
||||
"State.ActivateRestore" => self.state_activate_restore(&ctx).await,
|
||||
other => Err(DomainError::before(
|
||||
ErrorCode::Unsupported,
|
||||
format!("{other} is not an environment method"),
|
||||
|
|
@ -516,3 +544,488 @@ pub fn synthetic_asset(asset_id: &str, body: &str) -> AssetRef {
|
|||
format: id("fly-config-v1"),
|
||||
}
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------------------------
|
||||
// STATE-01: capture and restore
|
||||
|
||||
/// The version this payload layout is written and read under.
|
||||
pub const WORLD_PAYLOAD_VERSION: u64 = 1;
|
||||
|
||||
fn incompatible(message: impl std::fmt::Display) -> DomainError {
|
||||
DomainError::before(ErrorCode::IncompatibleState, message)
|
||||
}
|
||||
|
||||
/// One staged restore, held outside the live world until it is activated.
|
||||
struct StagedWorld {
|
||||
token: Id,
|
||||
checkpoint_id: Id,
|
||||
scope: Scope,
|
||||
episode_id: Id,
|
||||
descriptor: EnvironmentDescriptor,
|
||||
boundary: u64,
|
||||
counter: i64,
|
||||
world_time: RationalNs,
|
||||
advances: u64,
|
||||
frames: Vec<(u64, i64)>,
|
||||
audio_next_sample: u64,
|
||||
audio_phase: u64,
|
||||
audio_accumulator: u128,
|
||||
audio_denominator: u128,
|
||||
}
|
||||
|
||||
impl CounterEnvironment {
|
||||
/// `State.Capture`: the world at its committed boundary, including its pending sensor
|
||||
/// pipeline.
|
||||
///
|
||||
/// The pipeline is recorded as reconstruction inputs -- the producing boundary and the
|
||||
/// world counter of every retained frame -- and never as an artifact identity: a
|
||||
/// transient artifact belongs to the router that is running now, and a checkpoint outlives
|
||||
/// it.
|
||||
async fn state_capture(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult<HandlerReply> {
|
||||
let scope = ctx.scope()?.clone();
|
||||
let Some(descriptor) = self.descriptor.clone() else {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::InvalidPhase,
|
||||
"this environment is uninitialized",
|
||||
));
|
||||
};
|
||||
if scope.session_id != self.config.session_id {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::IdentityMismatch,
|
||||
"this environment belongs to another session",
|
||||
));
|
||||
}
|
||||
match &self.epoch {
|
||||
Some(epoch) if *epoch == scope.epoch => {}
|
||||
_ => {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::StaleEpoch,
|
||||
"State.Capture names an epoch this environment has left",
|
||||
));
|
||||
}
|
||||
}
|
||||
if scope.step != self.boundary {
|
||||
return Err(DomainError::before(
|
||||
if scope.step < self.boundary {
|
||||
ErrorCode::StaleStep
|
||||
} else {
|
||||
ErrorCode::FutureStep
|
||||
},
|
||||
"State.Capture must name the boundary the world is at",
|
||||
));
|
||||
}
|
||||
let params: CaptureParams = ctx.params()?;
|
||||
let pipeline = self
|
||||
.pipeline
|
||||
.as_ref()
|
||||
.ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no view pipeline"))?;
|
||||
let audio = self
|
||||
.audio
|
||||
.as_ref()
|
||||
.ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no audio source"))?;
|
||||
let (accumulator, denominator) = audio.accumulator();
|
||||
let previous = self.status.state();
|
||||
self.status.set_state(WorkerState::Capturing);
|
||||
let payload = serde_json::json!({
|
||||
"payloadVersion": WORLD_PAYLOAD_VERSION,
|
||||
"kind": "world",
|
||||
"workerId": self.config.worker_id.as_str(),
|
||||
"checkpointId": params.checkpoint_id.as_str(),
|
||||
"sourceScope": scope.to_json(),
|
||||
"episodeId": self.episode_id.clone().expect("initialized").as_str(),
|
||||
"committedStep": self.boundary.to_string(),
|
||||
"counter": self.counter.to_string(),
|
||||
"worldTime": self.world_time.to_json(),
|
||||
"advances": self.advances.to_string(),
|
||||
"descriptor": descriptor.to_json(),
|
||||
"pipeline": {
|
||||
// The declared delay's whole queue, oldest first.
|
||||
"frames": pipeline
|
||||
.retained()
|
||||
.into_iter()
|
||||
.map(|(boundary, counter)| serde_json::json!({
|
||||
"boundary": boundary.to_string(),
|
||||
"counter": counter.to_string(),
|
||||
}))
|
||||
.collect::<Vec<_>>(),
|
||||
},
|
||||
"audio": {
|
||||
"nextSample": audio.next_sample().to_string(),
|
||||
"phase": audio.phase().to_string(),
|
||||
"accumulator": accumulator.to_string(),
|
||||
"denominator": denominator.to_string(),
|
||||
"chunks": audio.chunks().to_string(),
|
||||
},
|
||||
});
|
||||
let bytes = canonicalize(&payload)
|
||||
.map_err(|e| DomainError::invalid(format!("State.Capture: {}", e.0)))?
|
||||
.into_bytes();
|
||||
let digest = digest_of_bytes(&bytes);
|
||||
let artifact = crate::state::seal_payload(ctx.client, &bytes, &digest).await?;
|
||||
// A capture reads the world; it does not advance it.
|
||||
self.status.set_state(previous);
|
||||
let result = CaptureResult {
|
||||
checkpoint_id: params.checkpoint_id,
|
||||
boundary: self.boundary,
|
||||
compatibility_digest: crate::state::Compatibility::of(&descriptor).digest(),
|
||||
payload: artifact.reference().clone(),
|
||||
};
|
||||
Ok(HandlerReply::with_artifacts(
|
||||
object(result.to_json()),
|
||||
vec![(crate::state::PAYLOAD_ATTACHMENT.to_owned(), artifact)],
|
||||
))
|
||||
}
|
||||
|
||||
/// `State.StageRestore`: validate a replacement world into a staging slot.
|
||||
async fn state_stage_restore(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult<HandlerReply> {
|
||||
let scope = ctx.scope()?.clone();
|
||||
if scope.session_id != self.config.session_id {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::IdentityMismatch,
|
||||
"this environment belongs to another session",
|
||||
));
|
||||
}
|
||||
if let Some(epoch) = &self.epoch
|
||||
&& *epoch == scope.epoch
|
||||
{
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::StaleEpoch,
|
||||
"State.StageRestore proposes the epoch this environment is already running",
|
||||
));
|
||||
}
|
||||
if self.descriptor.is_some() {
|
||||
// A world that is already running a boundary is not a quiescent replacement: the
|
||||
// group replaces it rather than restoring over a live one.
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::InvalidPhase,
|
||||
"State.StageRestore needs an uninitialized replacement environment",
|
||||
));
|
||||
}
|
||||
let params: StageRestoreParams = ctx.params()?;
|
||||
if params.source_scope.step != scope.step {
|
||||
return Err(DomainError::invalid(
|
||||
"State.StageRestore's scope step must be the source boundary",
|
||||
));
|
||||
}
|
||||
let artifact = ctx.artifact(crate::state::PAYLOAD_ATTACHMENT)?;
|
||||
if artifact.reference() != ¶ms.payload {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::BufferInvalid,
|
||||
"the staged payload attachment is not the artifact the request names",
|
||||
));
|
||||
}
|
||||
let bytes = artifact.read_all().await.map_err(|e| {
|
||||
DomainError::before(
|
||||
ErrorCode::BufferInvalid,
|
||||
format!("the staged payload could not be read: {}", e.message),
|
||||
)
|
||||
})?;
|
||||
let declared = params
|
||||
.payload
|
||||
.digest
|
||||
.clone()
|
||||
.ok_or_else(|| incompatible("a checkpoint payload must carry a content digest"))?;
|
||||
let actual = digest_of_bytes(&bytes);
|
||||
if actual != declared || bytes.len() as u64 != params.payload.byte_length {
|
||||
return Err(incompatible(
|
||||
"the staged payload is not the content the request declares",
|
||||
));
|
||||
}
|
||||
let value: Value = serde_json::from_slice(&bytes)
|
||||
.map_err(|e| DomainError::invalid(format!("the staged payload is not JSON: {e}")))?;
|
||||
let text = |key: &str| -> DomainResult<String> {
|
||||
value
|
||||
.get(key)
|
||||
.and_then(Value::as_str)
|
||||
.map(str::to_owned)
|
||||
.ok_or_else(|| incompatible(format!("the world payload has no {key}")))
|
||||
};
|
||||
let number = |key: &str| -> DomainResult<u64> {
|
||||
text(key)?
|
||||
.parse::<u64>()
|
||||
.map_err(|_| incompatible(format!("the world payload's {key} is not a U64")))
|
||||
};
|
||||
if value.get("payloadVersion").and_then(Value::as_u64) != Some(WORLD_PAYLOAD_VERSION) {
|
||||
return Err(incompatible("the world payload is another payload version"));
|
||||
}
|
||||
if text("kind")? != "world" {
|
||||
return Err(incompatible("this payload is not a world's state"));
|
||||
}
|
||||
if text("workerId")? != self.config.worker_id {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::IdentityMismatch,
|
||||
"the staged payload belongs to another world",
|
||||
));
|
||||
}
|
||||
if text("checkpointId")? != params.checkpoint_id {
|
||||
return Err(incompatible("the staged payload belongs to another checkpoint"));
|
||||
}
|
||||
let source_scope = Scope::from_json(
|
||||
value
|
||||
.get("sourceScope")
|
||||
.ok_or_else(|| incompatible("the world payload has no sourceScope"))?,
|
||||
)
|
||||
.map_err(|e| incompatible(format!("the world payload's sourceScope: {}", e.0)))?;
|
||||
if source_scope != params.source_scope {
|
||||
return Err(incompatible(
|
||||
"the staged payload was captured at another source scope",
|
||||
));
|
||||
}
|
||||
let committed_step = number("committedStep")?;
|
||||
if committed_step != params.source_scope.step {
|
||||
return Err(incompatible(
|
||||
"the staged payload's committed step is not the source boundary",
|
||||
));
|
||||
}
|
||||
let descriptor = EnvironmentDescriptor::from_json(
|
||||
value
|
||||
.get("descriptor")
|
||||
.ok_or_else(|| incompatible("the world payload has no descriptor"))?,
|
||||
)
|
||||
.map_err(|e| incompatible(format!("the world payload's descriptor: {}", e.0)))?;
|
||||
// The replacement builds the descriptor it would advertise and compares. A world
|
||||
// started with other ports, another cadence or another declared render delay is a
|
||||
// different backend, not this one resumed.
|
||||
let live = self.build_descriptor()?;
|
||||
if descriptor != live {
|
||||
return Err(incompatible(
|
||||
"the staged world was captured under another environment descriptor",
|
||||
));
|
||||
}
|
||||
let expected = crate::state::Compatibility::of(&descriptor).digest();
|
||||
if expected != params.compatibility_digest {
|
||||
return Err(incompatible(format!(
|
||||
"the staged world's compatibility {expected} is not the {} the restore requires",
|
||||
params.compatibility_digest
|
||||
)));
|
||||
}
|
||||
let counter: i64 = text("counter")?
|
||||
.parse()
|
||||
.map_err(|_| incompatible("the world payload's counter is not an integer"))?;
|
||||
let world_time = RationalNs::from_json(
|
||||
value
|
||||
.get("worldTime")
|
||||
.ok_or_else(|| incompatible("the world payload has no worldTime"))?,
|
||||
)
|
||||
.map_err(|e| incompatible(format!("the world payload's worldTime: {}", e.0)))?;
|
||||
let pipeline_value = value
|
||||
.get("pipeline")
|
||||
.and_then(|p| p.get("frames"))
|
||||
.and_then(Value::as_array)
|
||||
.ok_or_else(|| incompatible("the world payload has no pipeline frames"))?;
|
||||
let mut frames = Vec::with_capacity(pipeline_value.len());
|
||||
for frame in pipeline_value {
|
||||
let boundary = frame
|
||||
.get("boundary")
|
||||
.and_then(Value::as_str)
|
||||
.ok_or_else(|| incompatible("a captured frame has no boundary"))?
|
||||
.parse::<u64>()
|
||||
.map_err(|_| incompatible("a captured frame's boundary is not a U64"))?;
|
||||
let frame_counter = frame
|
||||
.get("counter")
|
||||
.and_then(Value::as_str)
|
||||
.ok_or_else(|| incompatible("a captured frame has no counter"))?
|
||||
.parse::<i64>()
|
||||
.map_err(|_| incompatible("a captured frame's counter is not an integer"))?;
|
||||
frames.push((boundary, frame_counter));
|
||||
}
|
||||
match frames.last() {
|
||||
Some((boundary, _)) if *boundary == committed_step => {}
|
||||
_ => {
|
||||
return Err(incompatible(
|
||||
"the captured pipeline does not end at the committed boundary",
|
||||
));
|
||||
}
|
||||
}
|
||||
let audio_value = value
|
||||
.get("audio")
|
||||
.ok_or_else(|| incompatible("the world payload has no audio state"))?;
|
||||
let audio_number = |key: &str| -> DomainResult<u128> {
|
||||
audio_value
|
||||
.get(key)
|
||||
.and_then(Value::as_str)
|
||||
.ok_or_else(|| incompatible(format!("the captured audio state has no {key}")))?
|
||||
.parse::<u128>()
|
||||
.map_err(|_| incompatible(format!("the captured audio {key} is not a number")))
|
||||
};
|
||||
let audio_next_sample = u64::try_from(audio_number("nextSample")?)
|
||||
.map_err(|_| incompatible("the captured audio position is outside U64"))?;
|
||||
let audio_phase = u64::try_from(audio_number("phase")?)
|
||||
.map_err(|_| incompatible("the captured audio phase is outside U64"))?;
|
||||
|
||||
if self.config.faults.fail_stage_restore {
|
||||
return Err(incompatible(
|
||||
"injected staging refusal: this participant's replacement state does not \
|
||||
validate",
|
||||
));
|
||||
}
|
||||
if let Some(staged) = &self.staged {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::Conflict,
|
||||
format!(
|
||||
"this environment already holds the staged restore {} for checkpoint {}",
|
||||
staged.token, staged.checkpoint_id
|
||||
),
|
||||
));
|
||||
}
|
||||
let token = crate::agent::restore_token(
|
||||
¶ms.checkpoint_id,
|
||||
&scope,
|
||||
&actual,
|
||||
&self.config.incarnation_id,
|
||||
);
|
||||
if self.activated.contains(&token) {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::Conflict,
|
||||
"this exact restore was already activated on this environment",
|
||||
));
|
||||
}
|
||||
self.staged = Some(StagedWorld {
|
||||
token: token.clone(),
|
||||
checkpoint_id: params.checkpoint_id.clone(),
|
||||
scope,
|
||||
episode_id: parse_id(&text("episodeId")?)
|
||||
.map_err(|e| incompatible(format!("the world payload's episodeId {e}")))?,
|
||||
descriptor,
|
||||
boundary: committed_step,
|
||||
counter,
|
||||
world_time,
|
||||
advances: number("advances")?,
|
||||
frames,
|
||||
audio_next_sample,
|
||||
audio_phase,
|
||||
audio_accumulator: audio_number("accumulator")?,
|
||||
audio_denominator: audio_number("denominator")?,
|
||||
});
|
||||
self.status.set_state(WorkerState::StagedRestore);
|
||||
let result = StageRestoreResult {
|
||||
checkpoint_id: params.checkpoint_id,
|
||||
restore_token: token,
|
||||
};
|
||||
Ok(HandlerReply::from(&result))
|
||||
}
|
||||
|
||||
/// `State.ActivateRestore`: install the staged world and return its coherent observation.
|
||||
///
|
||||
/// Nothing advances. The pipeline's frames are rendered again into fresh artifacts of the
|
||||
/// current store, which is what "the durable store imports fresh immutable bus artifacts"
|
||||
/// means on the producing side, and the observation carries no audio chunk because no
|
||||
/// interval was played.
|
||||
async fn state_activate_restore(
|
||||
&mut self,
|
||||
ctx: &HandlerCtx<'_>,
|
||||
) -> DomainResult<HandlerReply> {
|
||||
let params: ActivateRestoreParams = ctx.params()?;
|
||||
if self.activated.contains(¶ms.restore_token) {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::Conflict,
|
||||
"this restore token has already been activated",
|
||||
));
|
||||
}
|
||||
let Some(staged) = self.staged.take() else {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::InvalidPhase,
|
||||
"this environment holds no staged restore",
|
||||
));
|
||||
};
|
||||
if staged.token != params.restore_token {
|
||||
let token = staged.token.clone();
|
||||
self.staged = Some(staged);
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::IdentityMismatch,
|
||||
format!(
|
||||
"this environment's staged restore is {token}, not {}",
|
||||
params.restore_token
|
||||
),
|
||||
));
|
||||
}
|
||||
if self.config.faults.fail_activate_restore {
|
||||
let token = staged.token.clone();
|
||||
self.staged = Some(staged);
|
||||
self.status.set_state(WorkerState::Failed);
|
||||
return Err(DomainError::new(
|
||||
ErrorCode::BackendFailure,
|
||||
format!("injected activation failure; {token} stays staged and unresumed"),
|
||||
MutationCertainty::None,
|
||||
));
|
||||
}
|
||||
self.status.set_state(WorkerState::Restoring);
|
||||
let mut pipeline = ViewPipeline::new(
|
||||
CounterEnvironment::view_descriptor(self.config.observation_delay_steps),
|
||||
self.config.renders.clone(),
|
||||
);
|
||||
pipeline.restore(ctx.client, &staged.frames).await?;
|
||||
let audio = AudioSource::restored_from(
|
||||
CounterEnvironment::audio_descriptor(),
|
||||
staged.audio_next_sample,
|
||||
staged.audio_phase,
|
||||
staged.audio_accumulator,
|
||||
staged.audio_denominator,
|
||||
)?;
|
||||
self.epoch = Some(staged.scope.epoch.clone());
|
||||
self.episode_id = Some(staged.episode_id.clone());
|
||||
self.descriptor = Some(staged.descriptor.clone());
|
||||
self.boundary = staged.boundary;
|
||||
self.counter = staged.counter;
|
||||
self.world_time = staged.world_time;
|
||||
self.advances = staged.advances;
|
||||
// Batch ids are unique within an epoch, and this is a new one. Keeping the old set
|
||||
// would refuse nothing extra: a request under the old epoch is already refused by its
|
||||
// scope.
|
||||
self.batches.clear();
|
||||
self.pipeline = Some(pipeline);
|
||||
self.audio = Some(audio);
|
||||
self.previous_view = None;
|
||||
self.activated.insert(staged.token);
|
||||
self.status.set_state(WorkerState::Ready);
|
||||
self.status.set_scope(Some(scope_at(
|
||||
&staged.scope.session_id,
|
||||
&staged.scope.epoch,
|
||||
staged.boundary,
|
||||
)));
|
||||
|
||||
let (observation, attachments) = self.restored_observation()?;
|
||||
let result = ActivateRestoreResult {
|
||||
committed_step: staged.boundary,
|
||||
checkpoint_id: staged.checkpoint_id,
|
||||
observation: Some(observation),
|
||||
};
|
||||
result
|
||||
.validate_for_role(Role::Environment)
|
||||
.map_err(|e| DomainError::invalid(e.0))?;
|
||||
let mut reply = HandlerReply::from(&result);
|
||||
reply.artifacts = attachments;
|
||||
Ok(reply)
|
||||
}
|
||||
|
||||
/// The observation the restored world is already at: no render, no advance, no audio.
|
||||
fn restored_observation(
|
||||
&mut self,
|
||||
) -> DomainResult<(WorldObservation, Vec<(String, flybus::Artifact)>)> {
|
||||
let boundary = self.boundary;
|
||||
let counter = self.counter;
|
||||
let pipeline = self
|
||||
.pipeline
|
||||
.as_ref()
|
||||
.ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no view pipeline"))?;
|
||||
let (view, artifact) = pipeline.at(boundary).ok_or_else(|| {
|
||||
incompatible("the restored pipeline holds no frame for the restored boundary")
|
||||
})?;
|
||||
self.previous_view = Some((view.clone(), artifact.clone()));
|
||||
let observation = WorldObservation {
|
||||
boundary,
|
||||
world_time: self.world_time,
|
||||
engine_frame: Some(boundary.to_string()),
|
||||
sensory_views: vec![view.clone()],
|
||||
inspection: inspection(counter, boundary),
|
||||
broadcast_views: vec![view.clone()],
|
||||
// No interval was played, so there is no chunk. A chunk here would be an old
|
||||
// epoch's audio offered as current.
|
||||
audio: Vec::new(),
|
||||
};
|
||||
Ok((
|
||||
observation,
|
||||
vec![(media::view_attachment(&view.view_id), artifact)],
|
||||
))
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -26,6 +26,7 @@ use crate::media::{RenderCounter, SensorLog};
|
|||
use crate::launcher::{
|
||||
AgentLaunch, EnvironmentLaunch, Launcher, ReapOutcome, SUPERVISOR_CLIENT, ThreadBudget,
|
||||
};
|
||||
use crate::state::{CheckpointStore, CheckpointWriter, StoreConfig, StoreFaults, WriterConfig, WriterFaults};
|
||||
use crate::task::{ActionExecutor, CounterTask, IdentityExecutor, Terminal};
|
||||
// `crate::types` is this crate's facade over the shared `fly-session-types` crate; the
|
||||
// glob keeps the contract's own names in sight instead of restating them.
|
||||
|
|
@ -85,6 +86,14 @@ pub struct HarnessConfig {
|
|||
/// The threads reserved for the coordinator, its router and its store.
|
||||
pub coordinator_threads: usize,
|
||||
pub environment_threads: usize,
|
||||
/// How many committed generations the durable checkpoint store keeps.
|
||||
pub store: StoreConfig,
|
||||
/// The durable write faults this composition injects.
|
||||
pub store_faults: StoreFaults,
|
||||
/// The checkpoint queue's bounds.
|
||||
pub writer: WriterConfig,
|
||||
/// The writer faults this composition injects.
|
||||
pub writer_faults: WriterFaults,
|
||||
}
|
||||
|
||||
impl Default for HarnessConfig {
|
||||
|
|
@ -107,6 +116,10 @@ impl Default for HarnessConfig {
|
|||
thread_budget: None,
|
||||
coordinator_threads: 1,
|
||||
environment_threads: 1,
|
||||
store: StoreConfig::default(),
|
||||
store_faults: StoreFaults::default(),
|
||||
writer: WriterConfig::default(),
|
||||
writer_faults: WriterFaults::default(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -134,6 +147,14 @@ const ENV_SERVICE: &str = "env.arena";
|
|||
const ENV_CLIENT: &str = "environment";
|
||||
const ENV_WORKER: &str = "arena";
|
||||
const COORDINATOR_CLIENT: &str = "coordinator";
|
||||
/// The checkpoint writer's own bus identity. It publishes checkpoint events and nothing else.
|
||||
const WRITER_CLIENT: &str = "checkpoint-writer";
|
||||
|
||||
/// How many times one participant may be replaced in a composition.
|
||||
///
|
||||
/// Each replacement connects under its own client id, so a restart is visibly a new
|
||||
/// participant rather than a silent reattachment, and the policy has to name them all.
|
||||
const MAX_GENERATIONS: u32 = 8;
|
||||
|
||||
fn agent_service(agent_id: &Id) -> String {
|
||||
format!("agent.{agent_id}")
|
||||
|
|
@ -172,6 +193,10 @@ pub struct SessionHarness {
|
|||
/// The supervisor. It owns every participant's lifetime and thread allocation.
|
||||
pub launcher: Launcher,
|
||||
observers: Mutex<Vec<Client>>,
|
||||
/// Which generation of each participant is running: 1 is the one the composition started.
|
||||
generations: BTreeMap<Id, u32>,
|
||||
/// Where the durable checkpoint store lives, for a test that reads the files themselves.
|
||||
checkpoint_root: std::path::PathBuf,
|
||||
}
|
||||
|
||||
impl SessionHarness {
|
||||
|
|
@ -204,12 +229,16 @@ impl SessionHarness {
|
|||
g.call = vec![Pattern::prefix("agent."), Pattern::prefix("env.")];
|
||||
}),
|
||||
)
|
||||
// The writer publishes the checkpoint events and never calls a participant.
|
||||
.client(WRITER_CLIENT, grants(|g| g.publish = vec![Pattern::prefix("session.")]))
|
||||
.client(ENV_CLIENT, grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]))
|
||||
.client(
|
||||
&format!("{ENV_CLIENT}-r2"),
|
||||
grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]),
|
||||
)
|
||||
.client("observer", grants(|g| g.subscribe = vec![Pattern::prefix("session.")]));
|
||||
for generation in 2..=MAX_GENERATIONS {
|
||||
policy = policy.client(
|
||||
&format!("{ENV_CLIENT}-r{generation}"),
|
||||
grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]),
|
||||
);
|
||||
}
|
||||
for spec in &config.agents {
|
||||
let service = agent_service(&spec.agent_id);
|
||||
policy = policy.client(
|
||||
|
|
@ -218,10 +247,12 @@ impl SessionHarness {
|
|||
);
|
||||
// A replacement worker connects under its own client id, so a restart is visibly a
|
||||
// new participant rather than a silent reattachment to the active epoch.
|
||||
policy = policy.client(
|
||||
&format!("{}-r2", agent_client(&spec.agent_id)),
|
||||
grants(|g| g.register = vec![Pattern::exact(&service)]),
|
||||
);
|
||||
for generation in 2..=MAX_GENERATIONS {
|
||||
policy = policy.client(
|
||||
&format!("{}-r{generation}", agent_client(&spec.agent_id)),
|
||||
grants(|g| g.register = vec![Pattern::exact(&service)]),
|
||||
);
|
||||
}
|
||||
}
|
||||
let mut router_config = RouterConfig::new(&store_root);
|
||||
router_config.policy = policy;
|
||||
|
|
@ -299,6 +330,21 @@ impl SessionHarness {
|
|||
}
|
||||
|
||||
let coordinator_client = launcher.connect(COORDINATOR_CLIENT).await?;
|
||||
// The durable store lives beside the router's artifact store and never inside it: a
|
||||
// committed generation is outside the bus's ephemeral collection.
|
||||
let checkpoint_root = root.join("checkpoints");
|
||||
let mut store = CheckpointStore::open(&checkpoint_root, config.store).map_err(refusal)?;
|
||||
*store.faults_mut() = config.store_faults.clone();
|
||||
let writer_client = launcher.connect(WRITER_CLIENT).await?;
|
||||
let writer = CheckpointWriter::start(
|
||||
store,
|
||||
config.writer,
|
||||
config.writer_faults.clone(),
|
||||
Some((
|
||||
writer_client,
|
||||
format!("session.{}.checkpoints", config.session_id),
|
||||
)),
|
||||
);
|
||||
let executors: BTreeMap<Id, Box<dyn ActionExecutor>> = config
|
||||
.agents
|
||||
.iter()
|
||||
|
|
@ -316,6 +362,8 @@ impl SessionHarness {
|
|||
Box::new(CounterTask::new(&config.epoch, config.terminal)),
|
||||
executors,
|
||||
);
|
||||
let mut coordinator = coordinator;
|
||||
coordinator.attach_store(writer);
|
||||
|
||||
Ok(SessionHarness {
|
||||
coordinator,
|
||||
|
|
@ -326,9 +374,16 @@ impl SessionHarness {
|
|||
sensors,
|
||||
launcher,
|
||||
observers: Mutex::new(Vec::new()),
|
||||
generations: BTreeMap::new(),
|
||||
checkpoint_root,
|
||||
})
|
||||
}
|
||||
|
||||
/// Where the durable checkpoint store's generations and store manifest live.
|
||||
pub fn checkpoint_root(&self) -> &std::path::Path {
|
||||
&self.checkpoint_root
|
||||
}
|
||||
|
||||
pub fn router(&self) -> &Router {
|
||||
self.launcher.router()
|
||||
}
|
||||
|
|
@ -373,10 +428,11 @@ impl SessionHarness {
|
|||
.find(|spec| spec.agent_id == *agent_id)
|
||||
.expect("a configured agent")
|
||||
.clone();
|
||||
let generation = self.next_generation(agent_id)?;
|
||||
self.launcher.kill(agent_id).await;
|
||||
let tick_duration = millis(self.config.tick_ms).expect("a positive tick");
|
||||
let incarnation_id =
|
||||
parse_id(&format!("{agent_id}-inc-2")).expect("an agent id plus a suffix is an Id");
|
||||
let incarnation_id = parse_id(&format!("{agent_id}-inc-{generation}"))
|
||||
.expect("an agent id plus a suffix is an Id");
|
||||
self.launcher
|
||||
.launch_agent(AgentLaunch {
|
||||
session_id: self.config.session_id.clone(),
|
||||
|
|
@ -390,7 +446,7 @@ impl SessionHarness {
|
|||
// predecessor wrote, so a restore's sensory input is visible beside it.
|
||||
sensors: self.sensors.get(agent_id).cloned().unwrap_or_default(),
|
||||
faults: spec.faults.clone(),
|
||||
client_id: format!("{}-r2", agent_client(agent_id)),
|
||||
client_id: format!("{}-r{generation}", agent_client(agent_id)),
|
||||
service: agent_service(agent_id),
|
||||
})
|
||||
.await
|
||||
|
|
@ -403,6 +459,102 @@ impl SessionHarness {
|
|||
})
|
||||
}
|
||||
|
||||
/// Replaces the environment with a fresh, uninitialized incarnation, as a restore needs.
|
||||
pub async fn restart_environment(&mut self) -> Result<Restarted, flybus::BusError> {
|
||||
let worker_id = id(ENV_WORKER);
|
||||
let generation = self.next_generation(&worker_id)?;
|
||||
self.launcher.kill(&worker_id).await;
|
||||
let step_duration = hz(self.config.step_hz).expect("a positive cadence");
|
||||
let incarnation_id = parse_id(&format!("arena-inc-{generation}"))
|
||||
.expect("a worker id plus a suffix is an Id");
|
||||
self.launcher
|
||||
.launch_environment(EnvironmentLaunch {
|
||||
session_id: self.config.session_id.clone(),
|
||||
worker_id: worker_id.clone(),
|
||||
incarnation_id: incarnation_id.clone(),
|
||||
step_duration,
|
||||
ports: self.config.agents.iter().map(|a| a.port_id.clone()).collect(),
|
||||
worker_threads: self.config.environment_threads,
|
||||
observation_delay_steps: self.config.observation_delay_steps,
|
||||
renders: self.renders.clone(),
|
||||
faults: self.config.environment_faults.clone(),
|
||||
client_id: format!("{ENV_CLIENT}-r{generation}"),
|
||||
service: ENV_SERVICE.to_owned(),
|
||||
})
|
||||
.await
|
||||
.map_err(refusal)?;
|
||||
let worker = self.launcher.worker(&worker_id).expect("just launched");
|
||||
Ok(Restarted {
|
||||
service: worker.identity.service.clone(),
|
||||
service_incarnation: worker.service_incarnation.clone(),
|
||||
incarnation_id,
|
||||
})
|
||||
}
|
||||
|
||||
fn next_generation(&mut self, worker_id: &Id) -> Result<u32, flybus::BusError> {
|
||||
let slot = self.generations.entry(worker_id.clone()).or_insert(1);
|
||||
if *slot >= MAX_GENERATIONS {
|
||||
return Err(flybus::BusError::new(
|
||||
flybus::ErrorCode::QuotaExceeded,
|
||||
format!(
|
||||
"{worker_id} has used all {MAX_GENERATIONS} configured client identities; a composition declares how many replacements it allows"
|
||||
),
|
||||
));
|
||||
}
|
||||
*slot += 1;
|
||||
Ok(*slot)
|
||||
}
|
||||
|
||||
/// Replaces every participant and points the fenced coordinator at the replacements.
|
||||
///
|
||||
/// This is what a recovery does before it restores: the old participants belong to an
|
||||
/// invalid epoch, and the references the coordinator pinned are exchanged deliberately.
|
||||
pub async fn replace_all_participants(&mut self) -> Result<(), flybus::BusError> {
|
||||
let environment = self.environment_id();
|
||||
self.restart_environment().await?;
|
||||
let worker = self
|
||||
.launcher
|
||||
.worker(&environment)
|
||||
.expect("just launched")
|
||||
.worker_ref();
|
||||
self.coordinator
|
||||
.replace_participant(&environment, worker)
|
||||
.map_err(|e| refusal(e.error))?;
|
||||
for agent_id in self.config.agents.iter().map(|a| a.agent_id.clone()).collect::<Vec<_>>() {
|
||||
self.restart_agent(&agent_id).await?;
|
||||
let worker = self
|
||||
.launcher
|
||||
.worker(&agent_id)
|
||||
.expect("just launched")
|
||||
.worker_ref();
|
||||
self.coordinator
|
||||
.replace_participant(&agent_id, worker)
|
||||
.map_err(|e| refusal(e.error))?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Changes one agent's injected faults, so the replacement the next restart launches is
|
||||
/// a participant without them.
|
||||
///
|
||||
/// A fault is launch configuration, so clearing one is a relaunch and not a live change:
|
||||
/// the worker running now keeps whatever it was started with.
|
||||
pub fn set_agent_faults(&mut self, agent_id: &Id, faults: AgentFaults) {
|
||||
if let Some(spec) = self
|
||||
.config
|
||||
.agents
|
||||
.iter_mut()
|
||||
.find(|spec| spec.agent_id == *agent_id)
|
||||
{
|
||||
spec.faults = faults;
|
||||
}
|
||||
}
|
||||
|
||||
/// Changes the environment's injected faults, with the same relaunch rule.
|
||||
pub fn set_environment_faults(&mut self, faults: EnvironmentFaults) {
|
||||
self.config.environment_faults = faults;
|
||||
}
|
||||
|
||||
/// Ends one participant without asking it, as a crash would.
|
||||
pub async fn kill(&mut self, worker_id: &Id) -> ReapOutcome {
|
||||
self.launcher.kill(worker_id).await
|
||||
|
|
@ -466,7 +618,10 @@ impl SessionHarness {
|
|||
|
||||
/// Reaps every participant and closes the router.
|
||||
pub async fn shutdown(self) {
|
||||
let SessionHarness { coordinator, mut launcher, observers, .. } = self;
|
||||
let SessionHarness { mut coordinator, mut launcher, observers, .. } = self;
|
||||
// The writer task owns artifact handles and a blocking store. Leaving it running
|
||||
// would leave both behind.
|
||||
coordinator.shutdown_store().await;
|
||||
drop(coordinator);
|
||||
launcher.reap_all(&id("shutdown")).await;
|
||||
for observer in observers.into_inner().expect("not poisoned") {
|
||||
|
|
|
|||
|
|
@ -1231,6 +1231,8 @@ pub(crate) mod flags {
|
|||
pub const PREPARE_DELAY_MS: &str = "prepare-delay-ms";
|
||||
pub const COMMIT_DELAY_MS: &str = "commit-delay-ms";
|
||||
pub const FAIL_COMMIT_AT_STEP: &str = "fail-commit-at-step";
|
||||
pub const FAIL_STAGE_RESTORE: &str = "fail-stage-restore";
|
||||
pub const FAIL_ACTIVATE_RESTORE: &str = "fail-activate-restore";
|
||||
|
||||
pub const WORKER: &str = "worker";
|
||||
pub const PORTS: &str = "ports";
|
||||
|
|
@ -1271,6 +1273,8 @@ pub(crate) mod flags {
|
|||
PREPARE_DELAY_MS,
|
||||
COMMIT_DELAY_MS,
|
||||
FAIL_COMMIT_AT_STEP,
|
||||
FAIL_STAGE_RESTORE,
|
||||
FAIL_ACTIVATE_RESTORE,
|
||||
];
|
||||
/// What only the environment is given, media options included.
|
||||
pub const ENVIRONMENT_ONLY: &[&str] = &[
|
||||
|
|
@ -1285,6 +1289,8 @@ pub(crate) mod flags {
|
|||
TRUNCATED_VIEW_AT_BOUNDARY,
|
||||
OMIT_AUDIO_AT_BOUNDARY,
|
||||
OVERLAPPING_AUDIO_AT_BOUNDARY,
|
||||
FAIL_STAGE_RESTORE,
|
||||
FAIL_ACTIVATE_RESTORE,
|
||||
];
|
||||
/// What a measurement run or one of its row children is given.
|
||||
pub const MEASURE: &[&str] = &[MODE, AGENTS, STEPS, WARMUP_STEPS, WORKER_THREADS, MODES];
|
||||
|
|
@ -1327,6 +1333,11 @@ impl Started {
|
|||
arg(flags::WARMUP_TICKS, spec.warmup_ticks),
|
||||
arg(flags::PREPARE_DELAY_MS, spec.faults.prepare_delay_ms),
|
||||
arg(flags::COMMIT_DELAY_MS, spec.faults.commit_delay_ms),
|
||||
arg(flags::FAIL_STAGE_RESTORE, u64::from(spec.faults.fail_stage_restore)),
|
||||
arg(
|
||||
flags::FAIL_ACTIVATE_RESTORE,
|
||||
u64::from(spec.faults.fail_activate_restore),
|
||||
),
|
||||
];
|
||||
if let Some(step) = spec.faults.fail_commit_at_step {
|
||||
args.push(arg(flags::FAIL_COMMIT_AT_STEP, step));
|
||||
|
|
@ -1345,6 +1356,11 @@ impl Started {
|
|||
// The media options a world in another process needs to be exactly this
|
||||
// world. Its render counter and its agents' sensor logs stay there.
|
||||
arg(flags::OBSERVATION_DELAY_STEPS, spec.observation_delay_steps),
|
||||
arg(flags::FAIL_STAGE_RESTORE, u64::from(spec.faults.fail_stage_restore)),
|
||||
arg(
|
||||
flags::FAIL_ACTIVATE_RESTORE,
|
||||
u64::from(spec.faults.fail_activate_restore),
|
||||
),
|
||||
];
|
||||
for (flag, boundary) in [
|
||||
(flags::OMIT_VIEW_AT_BOUNDARY, spec.faults.omit_view_at_boundary),
|
||||
|
|
@ -1458,6 +1474,8 @@ mod flag_tests {
|
|||
truncated_view_at_boundary: Some(3),
|
||||
omit_audio_at_boundary: Some(4),
|
||||
overlapping_audio_at_boundary: Some(5),
|
||||
fail_stage_restore: true,
|
||||
fail_activate_restore: true,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1491,6 +1509,8 @@ mod flag_tests {
|
|||
fail_commit_at_step: Some(2),
|
||||
prepare_delay_ms: 1,
|
||||
commit_delay_ms: 2,
|
||||
fail_stage_restore: true,
|
||||
fail_activate_restore: true,
|
||||
},
|
||||
client_id: "worker-fly-a".to_owned(),
|
||||
service: "agent.fly-a".to_owned(),
|
||||
|
|
@ -1535,6 +1555,8 @@ mod flag_tests {
|
|||
flags::TRUNCATED_VIEW_AT_BOUNDARY,
|
||||
flags::OMIT_AUDIO_AT_BOUNDARY,
|
||||
flags::OVERLAPPING_AUDIO_AT_BOUNDARY,
|
||||
flags::FAIL_STAGE_RESTORE,
|
||||
flags::FAIL_ACTIVATE_RESTORE,
|
||||
] {
|
||||
assert!(written.contains(&format!("--{flag}")), "--{flag} is not written");
|
||||
}
|
||||
|
|
|
|||
|
|
@ -34,6 +34,7 @@ pub mod media;
|
|||
pub mod metrics;
|
||||
pub mod phase;
|
||||
pub mod rpc;
|
||||
pub mod state;
|
||||
pub mod task;
|
||||
pub mod worker;
|
||||
|
||||
|
|
|
|||
|
|
@ -133,7 +133,14 @@ pub fn arena_frame(descriptor: &ViewDescriptor, counter: i64, boundary: u64) ->
|
|||
/// cannot be served an arbitrary stale image.
|
||||
pub struct ViewPipeline {
|
||||
descriptor: ViewDescriptor,
|
||||
frames: VecDeque<(u64, flybus::Artifact)>,
|
||||
/// Each retained frame: its producing boundary, the world counter it was rendered from
|
||||
/// and the owned handle on its immutable bytes.
|
||||
///
|
||||
/// The counter is kept because it is the whole of the reconstruction input: a checkpoint
|
||||
/// records `(boundary, counter)` per retained frame and a restore re-renders them into
|
||||
/// fresh artifacts of the current store, rather than persisting a transient artifact
|
||||
/// identity that cannot survive a router restart.
|
||||
frames: VecDeque<(u64, i64, flybus::Artifact)>,
|
||||
renders: RenderCounter,
|
||||
}
|
||||
|
||||
|
|
@ -160,7 +167,7 @@ impl ViewPipeline {
|
|||
let bytes = arena_frame(&self.descriptor, counter, boundary);
|
||||
let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?;
|
||||
self.renders.bump();
|
||||
self.frames.push_back((boundary, artifact));
|
||||
self.frames.push_back((boundary, counter, artifact));
|
||||
// Keep exactly the frames a declared delay can still require.
|
||||
while self.frames.len() > self.descriptor.observation_delay_steps as usize + 1 {
|
||||
self.frames.pop_front();
|
||||
|
|
@ -168,6 +175,50 @@ impl ViewPipeline {
|
|||
Ok(())
|
||||
}
|
||||
|
||||
/// The reconstruction inputs of every retained frame, oldest first.
|
||||
///
|
||||
/// This is what a checkpoint records for the pending sensor pipeline: the producing
|
||||
/// boundary and the world counter, never an artifact identity.
|
||||
pub fn retained(&self) -> Vec<(u64, i64)> {
|
||||
self.frames
|
||||
.iter()
|
||||
.map(|(boundary, counter, _)| (*boundary, *counter))
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// Rebuilds the pipeline from recorded reconstruction inputs, into fresh artifacts.
|
||||
///
|
||||
/// Every frame is rendered again in the current store, so nothing a fence dropped is
|
||||
/// expected to come back and no old artifact identity crosses the recovery.
|
||||
pub async fn restore(
|
||||
&mut self,
|
||||
client: &flybus::Client,
|
||||
frames: &[(u64, i64)],
|
||||
) -> DomainResult<()> {
|
||||
if frames.len() > self.descriptor.observation_delay_steps as usize + 1 {
|
||||
return Err(media_error(format!(
|
||||
"a captured pipeline of {} frames does not fit a declared delay of {}",
|
||||
frames.len(),
|
||||
self.descriptor.observation_delay_steps
|
||||
)));
|
||||
}
|
||||
for window in frames.windows(2) {
|
||||
if window[1].0 != window[0].0 + 1 {
|
||||
return Err(media_error(
|
||||
"a captured pipeline's producing boundaries are not consecutive",
|
||||
));
|
||||
}
|
||||
}
|
||||
self.frames.clear();
|
||||
for (boundary, counter) in frames {
|
||||
let bytes = arena_frame(&self.descriptor, *counter, *boundary);
|
||||
let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?;
|
||||
self.renders.bump();
|
||||
self.frames.push_back((*boundary, *counter, artifact));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Seals a frame of the wrong length, which is what a broken backend produces. The
|
||||
/// reference it returns describes the artifact honestly, so the shape check is the thing
|
||||
/// under test rather than a lie in the payload.
|
||||
|
|
@ -181,7 +232,7 @@ impl ViewPipeline {
|
|||
bytes.truncate(bytes.len() - self.descriptor.row_stride as usize);
|
||||
let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?;
|
||||
self.renders.bump();
|
||||
self.frames.push_back((boundary, artifact));
|
||||
self.frames.push_back((boundary, counter, artifact));
|
||||
while self.frames.len() > self.descriptor.observation_delay_steps as usize + 2 {
|
||||
self.frames.pop_front();
|
||||
}
|
||||
|
|
@ -198,8 +249,8 @@ impl ViewPipeline {
|
|||
pub fn frame_produced_at(&self, produced: u64) -> Option<(ViewRef, flybus::Artifact)> {
|
||||
self.frames
|
||||
.iter()
|
||||
.find(|(step, _)| *step == produced)
|
||||
.map(|(step, artifact)| {
|
||||
.find(|(step, _, _)| *step == produced)
|
||||
.map(|(step, _, artifact)| {
|
||||
(
|
||||
ViewRef {
|
||||
view_id: self.descriptor.view_id.clone(),
|
||||
|
|
@ -257,6 +308,52 @@ impl AudioSource {
|
|||
source
|
||||
}
|
||||
|
||||
/// The exact state a capture recorded: sample position, waveform phase and the
|
||||
/// unconsumed fraction of a frame.
|
||||
///
|
||||
/// Restoring the position alone would restart the waveform and round the remainder away,
|
||||
/// which is a resample the restore rules refuse. The first chunk of the new epoch marks
|
||||
/// the discontinuity the recovery established.
|
||||
pub fn restored_from(
|
||||
descriptor: AudioDescriptor,
|
||||
next_sample: u64,
|
||||
phase: u64,
|
||||
accumulator: u128,
|
||||
denominator: u128,
|
||||
) -> DomainResult<AudioSource> {
|
||||
if denominator == 0 {
|
||||
return Err(DomainError::invalid(
|
||||
"audio: a captured accumulator denominator of zero",
|
||||
));
|
||||
}
|
||||
if accumulator >= denominator {
|
||||
return Err(DomainError::invalid(
|
||||
"audio: a captured accumulator is not below one whole frame",
|
||||
));
|
||||
}
|
||||
if phase >= descriptor.sample_rate {
|
||||
return Err(DomainError::invalid(
|
||||
"audio: a captured phase is not below the sample rate",
|
||||
));
|
||||
}
|
||||
let mut source = AudioSource::new(descriptor, next_sample);
|
||||
source.discontinuous = true;
|
||||
source.phase = phase;
|
||||
source.accumulator = accumulator;
|
||||
source.denominator = denominator;
|
||||
Ok(source)
|
||||
}
|
||||
|
||||
/// The waveform phase, for a capture.
|
||||
pub fn phase(&self) -> u64 {
|
||||
self.phase
|
||||
}
|
||||
|
||||
/// The unconsumed fraction of a frame and the denominator it is over, for a capture.
|
||||
pub fn accumulator(&self) -> (u128, u128) {
|
||||
(self.accumulator, self.denominator)
|
||||
}
|
||||
|
||||
pub fn descriptor(&self) -> &AudioDescriptor {
|
||||
&self.descriptor
|
||||
}
|
||||
|
|
@ -439,18 +536,47 @@ pub fn check_required_views(
|
|||
Ok(())
|
||||
}
|
||||
|
||||
/// Every declared audio stream produces exactly one chunk per transition.
|
||||
/// Where an observation came from.
|
||||
///
|
||||
/// `state-media-v1` section 2 makes a chunk the audio of an *interval*, so whether an
|
||||
/// observation must carry one is a question about its provenance and not about its boundary
|
||||
/// number. MEDIA-01 wrote the rule as "boundary 0 carries no chunk", which is true of the one
|
||||
/// observation that slice could produce without a transition and false of the other one:
|
||||
/// `State.ActivateRestore` installs a coherent observation at boundary `k` without advancing
|
||||
/// gameplay, and it covers no interval either. Naming the provenance is the fix; exempting
|
||||
/// the restored observation from the validator instead would have left "must a chunk exist"
|
||||
/// unanswered exactly where a stale chunk would do the most damage.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum ObservationOrigin {
|
||||
/// The observation a completed transition produced. Its interval has audio.
|
||||
Transition,
|
||||
/// An observation established at a boundary without running a transition:
|
||||
/// `Environment.Initialize`'s `O[0]` and `State.ActivateRestore`'s restored observation.
|
||||
/// It covers no interval, so it carries no chunk and one in it is refused.
|
||||
Installed,
|
||||
}
|
||||
|
||||
/// Every declared audio stream produces exactly one chunk per transition, and none at all in
|
||||
/// an observation that is not one.
|
||||
///
|
||||
/// The contract states the shape and the ordering of chunks, not whether one has to exist, so
|
||||
/// this is MEDIA-01's choice and it is deliberate: a session that tolerates a silently missing
|
||||
/// chunk cannot tell "this world produced no audio for this interval" from "the chunk was
|
||||
/// lost", and the second is the case the retention rules care about. Boundary 0 has no
|
||||
/// preceding interval and so carries no chunk.
|
||||
/// lost", and the second is the case the retention rules care about. The mirror of that, which
|
||||
/// STATE-01 needs, is that an installed observation carrying a chunk is a stale chunk being
|
||||
/// offered as current, and is refused for the same reason.
|
||||
pub fn check_required_audio(
|
||||
descriptor: &EnvironmentDescriptor,
|
||||
observation: &WorldObservation,
|
||||
origin: ObservationOrigin,
|
||||
) -> DomainResult<()> {
|
||||
if observation.boundary == 0 {
|
||||
if origin == ObservationOrigin::Installed {
|
||||
if let Some(chunk) = observation.audio.first() {
|
||||
return Err(media_error(format!(
|
||||
"audio stream {} produced a chunk for an observation that ran no transition",
|
||||
chunk.stream_id
|
||||
)));
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
for stream in &descriptor.audio {
|
||||
|
|
|
|||
1558
services/flysim/crates/fly-session/src/state.rs
Normal file
1558
services/flysim/crates/fly-session/src/state.rs
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -39,6 +39,16 @@ pub fn episode_schema() -> SchemaRef {
|
|||
synthetic_schema("arena.episode.v1", 1)
|
||||
}
|
||||
|
||||
/// The schema of a captured task ledger.
|
||||
pub fn ledger_schema() -> SchemaRef {
|
||||
synthetic_schema("arena.ledger.v1", 1)
|
||||
}
|
||||
|
||||
/// The schema of a captured action-executor state.
|
||||
pub fn executor_schema() -> SchemaRef {
|
||||
synthetic_schema("arena.executor.v1", 1)
|
||||
}
|
||||
|
||||
pub fn controller_schema_ref() -> SchemaRef {
|
||||
synthetic_schema("arena.controller.v1", 1)
|
||||
}
|
||||
|
|
@ -86,6 +96,29 @@ pub trait Task: Send {
|
|||
|
||||
/// How many times `evaluate_transition` has run. A transition must evaluate once.
|
||||
fn evaluations(&self) -> u64;
|
||||
|
||||
/// The checkpointable ledger at a committed boundary (`workers-v1` section 4).
|
||||
fn capture(&self) -> DomainResult<TypedValue>;
|
||||
|
||||
/// Validates a captured ledger without installing it, so a group install can fail before
|
||||
/// anything is changed.
|
||||
fn validate_restore(&self, state: &TypedValue) -> DomainResult<()>;
|
||||
|
||||
/// Installs a validated ledger under `epoch`. Event identity is derived from the epoch,
|
||||
/// so the new one is part of the install rather than something the ledger keeps from the
|
||||
/// epoch it was captured in.
|
||||
fn install_restore(&mut self, epoch: &Id, state: &TypedValue) -> DomainResult<()>;
|
||||
|
||||
/// Every event identity this ledger has issued, mapped onto the identity it would have
|
||||
/// under `to_epoch`.
|
||||
///
|
||||
/// `workers-v1` section 4 derives an event id from the epoch, so a trace recorded in one
|
||||
/// epoch cannot be compared with a trace recorded in another until these are rebased.
|
||||
/// The ledger owns the derivation, so it is the only thing that can do it.
|
||||
fn rebase_ids(&self, to_epoch: &Id) -> DomainResult<BTreeMap<Id, Id>>;
|
||||
|
||||
/// How far event identity has reached: the highest source step and the number issued.
|
||||
fn event_watermarks(&self) -> (u64, u64);
|
||||
}
|
||||
|
||||
/// Translates one selected decision into a controller intent, with no port assignment.
|
||||
|
|
@ -98,6 +131,15 @@ pub trait ActionExecutor: Send {
|
|||
progress: &TypedValue,
|
||||
clock: &RationalNs,
|
||||
) -> DomainResult<(ControllerIntent, Vec<TaskEvent>)>;
|
||||
|
||||
/// Per-executor state at a committed boundary (`workers-v1` section 4).
|
||||
fn capture(&self) -> DomainResult<TypedValue>;
|
||||
|
||||
/// Validates a captured executor state without installing it.
|
||||
fn validate_restore(&self, state: &TypedValue) -> DomainResult<()>;
|
||||
|
||||
/// Installs a validated executor state.
|
||||
fn install_restore(&mut self, state: &TypedValue) -> DomainResult<()>;
|
||||
}
|
||||
|
||||
/// The only executor v1 supports: it passes a direct-control decision through unchanged.
|
||||
|
|
@ -123,6 +165,37 @@ impl ActionExecutor for IdentityExecutor {
|
|||
.map_err(|e| DomainError::invalid(format!("decision: {e}")))?;
|
||||
Ok((intent, Vec::new()))
|
||||
}
|
||||
|
||||
/// The identity executor is stateless, and says so rather than capturing nothing.
|
||||
///
|
||||
/// An empty object would be indistinguishable from a stateful executor whose capture went
|
||||
/// missing, so the capture names the executor it came from and a restore refuses any
|
||||
/// other one.
|
||||
fn capture(&self) -> DomainResult<TypedValue> {
|
||||
TypedValue::new(executor_schema(), json!({"executor": "identity-v1"}))
|
||||
.map_err(|e| DomainError::invalid(e.0))
|
||||
}
|
||||
|
||||
fn validate_restore(&self, state: &TypedValue) -> DomainResult<()> {
|
||||
if state.schema != executor_schema() {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::IncompatibleState,
|
||||
"the captured executor state does not carry the executor schema",
|
||||
));
|
||||
}
|
||||
match state.value.get("executor").and_then(Value::as_str) {
|
||||
Some("identity-v1") => Ok(()),
|
||||
other => Err(DomainError::before(
|
||||
ErrorCode::IncompatibleState,
|
||||
format!("the captured executor is {other:?}, not the identity executor"),
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
fn install_restore(&mut self, state: &TypedValue) -> DomainResult<()> {
|
||||
// Stateless: validation is the whole of the install, and it is not skipped.
|
||||
self.validate_restore(state)
|
||||
}
|
||||
}
|
||||
|
||||
/// When the counter task asks for a terminal episode transition.
|
||||
|
|
@ -146,6 +219,10 @@ pub struct CounterTask {
|
|||
evaluations: u64,
|
||||
total_reward: f64,
|
||||
counter: i64,
|
||||
/// The highest source step any issued event belongs to, and how many were issued. These
|
||||
/// are the event watermarks a checkpoint records and a resumed epoch continues from.
|
||||
last_source_step: u64,
|
||||
issued_events: u64,
|
||||
terminal: Terminal,
|
||||
}
|
||||
|
||||
|
|
@ -159,6 +236,8 @@ impl CounterTask {
|
|||
evaluations: 0,
|
||||
total_reward: 0.0,
|
||||
counter: 0,
|
||||
last_source_step: 0,
|
||||
issued_events: 0,
|
||||
terminal,
|
||||
}
|
||||
}
|
||||
|
|
@ -239,6 +318,7 @@ impl Task for CounterTask {
|
|||
payload: TypedValue::new(event_schema(), json!({"counter": self.counter}))
|
||||
.expect("a synthetic typed value fits the contract"),
|
||||
}];
|
||||
self.issued_events += events.len() as u64;
|
||||
Ok(Bootstrap { contexts, progress: self.progress_value(), events })
|
||||
}
|
||||
|
||||
|
|
@ -314,6 +394,8 @@ impl Task for CounterTask {
|
|||
));
|
||||
}
|
||||
|
||||
self.last_source_step = self.last_source_step.max(source_step);
|
||||
self.issued_events += events.len() as u64;
|
||||
let next_contexts = self
|
||||
.agents
|
||||
.iter()
|
||||
|
|
@ -346,6 +428,173 @@ impl Task for CounterTask {
|
|||
fn evaluations(&self) -> u64 {
|
||||
self.evaluations
|
||||
}
|
||||
|
||||
fn capture(&self) -> DomainResult<TypedValue> {
|
||||
TypedValue::new(
|
||||
ledger_schema(),
|
||||
json!({
|
||||
"epoch": self.epoch.as_str(),
|
||||
"agents": self.agents.iter().map(String::as_str).collect::<Vec<_>>(),
|
||||
"bindings": self
|
||||
.bindings
|
||||
.iter()
|
||||
.map(|b| json!({"portId": b.port_id.as_str(), "agentId": b.agent_id.as_str()}))
|
||||
.collect::<Vec<_>>(),
|
||||
"transitions": self.transitions,
|
||||
"evaluations": self.evaluations,
|
||||
"totalReward": self.total_reward,
|
||||
"counter": self.counter,
|
||||
"lastSourceStep": self.last_source_step,
|
||||
"issuedEvents": self.issued_events,
|
||||
}),
|
||||
)
|
||||
.map_err(|e| DomainError::invalid(e.0))
|
||||
}
|
||||
|
||||
fn validate_restore(&self, state: &TypedValue) -> DomainResult<()> {
|
||||
if state.schema != ledger_schema() {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::IncompatibleState,
|
||||
"the captured ledger does not carry this task's schema",
|
||||
));
|
||||
}
|
||||
for field in [
|
||||
"epoch",
|
||||
"agents",
|
||||
"bindings",
|
||||
"transitions",
|
||||
"evaluations",
|
||||
"totalReward",
|
||||
"counter",
|
||||
"lastSourceStep",
|
||||
"issuedEvents",
|
||||
] {
|
||||
if state.value.get(field).is_none() {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::IncompatibleState,
|
||||
format!("the captured ledger has no {field}"),
|
||||
));
|
||||
}
|
||||
}
|
||||
let bindings = state
|
||||
.value
|
||||
.get("bindings")
|
||||
.and_then(Value::as_array)
|
||||
.ok_or_else(|| {
|
||||
DomainError::before(
|
||||
ErrorCode::IncompatibleState,
|
||||
"the captured ledger's bindings are not a list",
|
||||
)
|
||||
})?;
|
||||
if bindings.len() != self.bindings.len() && !self.bindings.is_empty() {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::IncompatibleState,
|
||||
"the captured ledger binds another number of ports",
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn install_restore(&mut self, epoch: &Id, state: &TypedValue) -> DomainResult<()> {
|
||||
self.validate_restore(state)?;
|
||||
let number = |key: &str| -> DomainResult<u64> {
|
||||
state.value.get(key).and_then(Value::as_u64).ok_or_else(|| {
|
||||
DomainError::before(
|
||||
ErrorCode::IncompatibleState,
|
||||
format!("the captured ledger's {key} is not a whole number"),
|
||||
)
|
||||
})
|
||||
};
|
||||
let mut agents = Vec::new();
|
||||
for value in state.value["agents"].as_array().expect("validated") {
|
||||
let agent = value.as_str().ok_or_else(|| {
|
||||
DomainError::before(
|
||||
ErrorCode::IncompatibleState,
|
||||
"the captured ledger names an agent that is not a string",
|
||||
)
|
||||
})?;
|
||||
agents.push(parse_id(agent).map_err(|e| {
|
||||
DomainError::before(ErrorCode::IncompatibleState, format!("ledger: {e}"))
|
||||
})?);
|
||||
}
|
||||
let mut bindings = Vec::new();
|
||||
for value in state.value["bindings"].as_array().expect("validated") {
|
||||
let port_id = value.get("portId").and_then(Value::as_str).ok_or_else(|| {
|
||||
DomainError::before(
|
||||
ErrorCode::IncompatibleState,
|
||||
"the captured ledger has a binding with no portId",
|
||||
)
|
||||
})?;
|
||||
let agent_id = value.get("agentId").and_then(Value::as_str).ok_or_else(|| {
|
||||
DomainError::before(
|
||||
ErrorCode::IncompatibleState,
|
||||
"the captured ledger has a binding with no agentId",
|
||||
)
|
||||
})?;
|
||||
bindings.push(PortBinding {
|
||||
port_id: parse_id(port_id).map_err(|e| {
|
||||
DomainError::before(ErrorCode::IncompatibleState, format!("ledger: {e}"))
|
||||
})?,
|
||||
agent_id: parse_id(agent_id).map_err(|e| {
|
||||
DomainError::before(ErrorCode::IncompatibleState, format!("ledger: {e}"))
|
||||
})?,
|
||||
});
|
||||
}
|
||||
let counter = state.value.get("counter").and_then(Value::as_i64).ok_or_else(|| {
|
||||
DomainError::before(
|
||||
ErrorCode::IncompatibleState,
|
||||
"the captured ledger's counter is not an integer",
|
||||
)
|
||||
})?;
|
||||
let total_reward = state
|
||||
.value
|
||||
.get("totalReward")
|
||||
.and_then(Value::as_f64)
|
||||
.filter(|v| v.is_finite())
|
||||
.ok_or_else(|| {
|
||||
DomainError::before(
|
||||
ErrorCode::IncompatibleState,
|
||||
"the captured ledger's totalReward is not a finite number",
|
||||
)
|
||||
})?;
|
||||
// The epoch is the caller's, not the capture's: event identity belongs to the epoch
|
||||
// the ledger is being installed into.
|
||||
self.epoch = epoch.clone();
|
||||
self.agents = agents;
|
||||
self.bindings = bindings;
|
||||
self.transitions = number("transitions")?;
|
||||
self.evaluations = number("evaluations")?;
|
||||
self.total_reward = total_reward;
|
||||
self.counter = counter;
|
||||
self.last_source_step = number("lastSourceStep")?;
|
||||
self.issued_events = number("issuedEvents")?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn rebase_ids(&self, to_epoch: &Id) -> DomainResult<BTreeMap<Id, Id>> {
|
||||
let mut out = BTreeMap::new();
|
||||
out.insert(
|
||||
event_id(&self.epoch, 0, "bootstrap", 0),
|
||||
event_id(to_epoch, 0, "bootstrap", 0),
|
||||
);
|
||||
// The counter task issues exactly one `counter-delta` event per bound port per
|
||||
// evaluated transition, in descriptor port order, so every identity it has ever
|
||||
// issued is re-derivable from its ledger without keeping a list of them.
|
||||
let ports = self.bindings.len() as u32;
|
||||
for source_step in 1..=self.last_source_step {
|
||||
for ordinal in 0..ports {
|
||||
out.insert(
|
||||
event_id(&self.epoch, source_step, "counter-delta", ordinal),
|
||||
event_id(to_epoch, source_step, "counter-delta", ordinal),
|
||||
);
|
||||
}
|
||||
}
|
||||
Ok(out)
|
||||
}
|
||||
|
||||
fn event_watermarks(&self) -> (u64, u64) {
|
||||
(self.last_source_step, self.issued_events)
|
||||
}
|
||||
}
|
||||
|
||||
/// The inspection value the counter environment publishes.
|
||||
|
|
|
|||
|
|
@ -8,13 +8,18 @@
|
|||
//! `Id` and `Digest` are type aliases, because the shared crate carries both as validated
|
||||
//! `String`s from `flybus::wire` rather than forking the encodings into newtypes.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
use serde_json::{Map, Value};
|
||||
|
||||
pub use fly_session_types::ArtifactRef;
|
||||
pub use fly_session_types::canonical::{
|
||||
self, OperationKey, body_digest, canonicalize, digest_of, sha256_hex,
|
||||
};
|
||||
pub use fly_session_types::media::{AudioDescriptor, AudioRef, ViewDescriptor, ViewRef};
|
||||
pub use fly_session_types::media::{
|
||||
ActivateRestoreParams, ActivateRestoreResult, AudioDescriptor, AudioRef, CaptureParams,
|
||||
CaptureResult, StageRestoreParams, StageRestoreResult, ViewDescriptor, ViewRef,
|
||||
};
|
||||
pub use fly_session_types::rpc::{
|
||||
ErrorCode, MutationCertainty, SessionRpcFailure, SessionRpcOutcome, SessionRpcRequest,
|
||||
SessionRpcSuccess,
|
||||
|
|
@ -214,6 +219,57 @@ pub fn outcome_identity(
|
|||
}
|
||||
}
|
||||
|
||||
/// The epoch-derived identities of a behaviour trace, rewritten onto one reference epoch.
|
||||
///
|
||||
/// `step-v1` section 8 compares committed behaviour across runs, excluding wall time, request
|
||||
/// ids "and other explicitly operational metadata". A resumed run's epoch is neither: it is
|
||||
/// behaviour metadata, and `scope.epoch`, the batch id and every task event id are derived
|
||||
/// from it. Comparing the two runs therefore means rewriting exactly those three things and
|
||||
/// nothing else, which is what this does -- and it **fails** on anything it does not
|
||||
/// recognise instead of passing it through, so a field that silently stopped being rebased
|
||||
/// would fail the comparison rather than weaken it.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct EpochRebase {
|
||||
pub from: Id,
|
||||
pub to: Id,
|
||||
/// Every event identity the task issued under `from`, and the identity it has under `to`.
|
||||
pub events: BTreeMap<Id, Id>,
|
||||
}
|
||||
|
||||
impl EpochRebase {
|
||||
/// Rewrites one behaviour record. An identity this rebase does not know is an error.
|
||||
pub fn apply(&self, behaviour: &TraceBehaviour) -> Result<TraceBehaviour, String> {
|
||||
if behaviour.scope.epoch != self.from {
|
||||
return Err(format!(
|
||||
"this behaviour was recorded in epoch {}, not {}",
|
||||
behaviour.scope.epoch, self.from
|
||||
));
|
||||
}
|
||||
let mut out = behaviour.clone();
|
||||
out.scope = Scope::new(&behaviour.scope.session_id, &self.to, behaviour.scope.step)
|
||||
.map_err(|e| e.0)?;
|
||||
let prefix = format!("batch-{}-", self.from);
|
||||
let suffix = behaviour
|
||||
.batch_id
|
||||
.strip_prefix(&prefix)
|
||||
.ok_or_else(|| format!("the batch id {} is not derived from {}", behaviour.batch_id, self.from))?;
|
||||
out.batch_id = parse_id(&format!("batch-{}-{suffix}", self.to))?;
|
||||
let map = |ids: &[Id]| -> Result<Vec<Id>, String> {
|
||||
ids.iter()
|
||||
.map(|id| {
|
||||
self.events
|
||||
.get(id)
|
||||
.cloned()
|
||||
.ok_or_else(|| format!("no rebased identity for the event {id}"))
|
||||
})
|
||||
.collect()
|
||||
};
|
||||
out.outcome_ids = map(&behaviour.outcome_ids)?;
|
||||
out.event_ids = map(&behaviour.event_ids)?;
|
||||
Ok(out)
|
||||
}
|
||||
}
|
||||
|
||||
/// One session phase transition, recorded whether or not it ends a step.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct PhaseTransition {
|
||||
|
|
@ -253,6 +309,28 @@ impl TraceLog {
|
|||
.collect()
|
||||
}
|
||||
|
||||
/// Every transition's behaviour, rebased onto one epoch and canonicalized.
|
||||
///
|
||||
/// This is the comparison a resumed run is held to: the same strings as
|
||||
/// [`TraceLog::behavior`], with the epoch metadata accounted for and nothing else changed.
|
||||
/// A resumed run's log holds transitions from two epochs -- the ones before the checkpoint
|
||||
/// and the ones after the restore -- so a transition already recorded in `rebase.to` is
|
||||
/// kept as it stands and one recorded in `rebase.from` is rewritten. A transition in a
|
||||
/// third epoch is an error; there is no pass-through case.
|
||||
pub fn behavior_rebased(&self, rebase: &EpochRebase) -> Result<Vec<String>, String> {
|
||||
self.transitions
|
||||
.iter()
|
||||
.map(|t| {
|
||||
let behaviour = if t.behaviour.scope.epoch == rebase.to {
|
||||
t.behaviour.clone()
|
||||
} else {
|
||||
rebase.apply(&t.behaviour)?
|
||||
};
|
||||
canonicalize(&behaviour.to_json()).map_err(|e| e.0)
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// The phase path, as `from -> to` strings.
|
||||
pub fn phase_path(&self) -> Vec<String> {
|
||||
self.phases.iter().map(|p| format!("{} -> {}", p.from, p.to)).collect()
|
||||
|
|
|
|||
|
|
@ -597,7 +597,17 @@ async fn execute<E: WorkerEndpoint>(
|
|||
fn classify_default(method: &str) -> Option<OpClass> {
|
||||
match method {
|
||||
"Agent.Prepare" | "Agent.Commit" | "Environment.Advance" => Some(OpClass::StepMutation),
|
||||
"Agent.Initialize" | "Environment.Initialize" => Some(OpClass::Lifecycle),
|
||||
// `ipc-v1` section 5 retains lifecycle *and capture* replies until
|
||||
// `Worker.Acknowledge`. The restore methods join them: their replies carry a
|
||||
// once-only token and, for an environment, the restored observation's artifact, and a
|
||||
// duplicate domain request must replay that reply rather than stage or activate a
|
||||
// second time. They are not step mutations -- they carry no committed step of their
|
||||
// own and are not keyed by one.
|
||||
"Agent.Initialize"
|
||||
| "Environment.Initialize"
|
||||
| "State.Capture"
|
||||
| "State.StageRestore"
|
||||
| "State.ActivateRestore" => Some(OpClass::Lifecycle),
|
||||
"Worker.Hello" | "Worker.Status" | "Worker.Acknowledge" | "Worker.Shutdown" => {
|
||||
Some(OpClass::ReadOnly)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -121,6 +121,8 @@ async fn a_slow_participant_is_resolved_rather_than_failed(mode: ExecutionMode)
|
|||
resolve: Duration::from_secs(20),
|
||||
resolve_attempts: 4096,
|
||||
boot: Duration::from_secs(30),
|
||||
capture: Duration::from_secs(30),
|
||||
durable: Duration::from_secs(60),
|
||||
};
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let reports = within("run", f.harness.coordinator.run(2))
|
||||
|
|
@ -191,6 +193,8 @@ async fn a_resolution_says_which_of_its_two_bounds_ended_it(mode: ExecutionMode)
|
|||
resolve: Duration::from_millis(300),
|
||||
resolve_attempts: 8192,
|
||||
boot: Duration::from_secs(30),
|
||||
capture: Duration::from_secs(30),
|
||||
durable: Duration::from_secs(60),
|
||||
};
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let started = Instant::now();
|
||||
|
|
@ -221,6 +225,8 @@ async fn a_resolution_says_which_of_its_two_bounds_ended_it(mode: ExecutionMode)
|
|||
resolve: Duration::from_secs(600),
|
||||
resolve_attempts: 3,
|
||||
boot: Duration::from_secs(30),
|
||||
capture: Duration::from_secs(30),
|
||||
durable: Duration::from_secs(60),
|
||||
};
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let failure = within("step", f.harness.coordinator.step())
|
||||
|
|
|
|||
1002
services/flysim/crates/fly-session/tests/state.rs
Normal file
1002
services/flysim/crates/fly-session/tests/state.rs
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -56,7 +56,7 @@ impl MemoryReader for &mut dyn MemoryReader {
|
|||
/// One reward payout in one frame.
|
||||
///
|
||||
/// `kind` is an adapter-owned interned name (Pokémon: `milestone`,
|
||||
/// `exploration`, `map`, `species`, `trainer`, `battle`, `badge`, `boundary`); it is the
|
||||
/// `exploration`, `map`, `species`, `trainer`, `battle`, `badge`, `boundary`, `catch`); it is the
|
||||
/// key the statistics counters and the on-screen ticker group by. Field names
|
||||
/// serialize exactly as the prototype's `RewardEvent` did, so a checkpoint
|
||||
/// written by either implementation reads in the other.
|
||||
|
|
@ -241,9 +241,21 @@ impl std::error::Error for AdapterError {}
|
|||
/// A game, as the sim loop sees it.
|
||||
pub trait GameAdapter: Send {
|
||||
/// Adapter version string, pinned into the checkpoint compatibility string.
|
||||
/// Pokémon: `pokered-unique8-v5`.
|
||||
/// Pokémon: `pokered-unique8-v6`.
|
||||
fn id(&self) -> &'static str;
|
||||
|
||||
/// Earlier [`GameAdapter::id`]s whose checkpoints this build can read, by a migration
|
||||
/// this adapter has written down and tested.
|
||||
///
|
||||
/// The default is empty: an adapter migrates from nothing unless it says otherwise, which
|
||||
/// is the behaviour every adapter had before this existed. It is only half of the gate --
|
||||
/// [`crate::compatibility::decide`] also requires the operator to have named the same id in
|
||||
/// `FLY_ACCEPT_ADAPTERS` for that deploy -- so listing an id here never migrates a live run
|
||||
/// on its own.
|
||||
fn migrates_from(&self) -> &'static [&'static str] {
|
||||
&[]
|
||||
}
|
||||
|
||||
/// Whether semantic rewards are enabled for this cartridge. An adapter that
|
||||
/// says no must still sample without paying anything, so the stream keeps
|
||||
/// running with a visible "rewards off" mode.
|
||||
|
|
@ -470,6 +482,10 @@ mod tests {
|
|||
let platformer =
|
||||
adapter_for_with_rom_pin("platformer", Some(&"a".repeat(64))).unwrap();
|
||||
assert_ne!(pokemon.id(), platformer.id());
|
||||
assert!(
|
||||
!platformer.migrates_from().contains(&pokemon.id()),
|
||||
"a migration never crosses games"
|
||||
);
|
||||
assert_ne!(pokemon.symbol_provenance(), platformer.symbol_provenance());
|
||||
// And two ROM revisions of the same game cannot either.
|
||||
let other = adapter_for_with_rom_pin("platformer", Some(&"b".repeat(64))).unwrap();
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ pub const PROTOTYPE_PLASTICITY_VERSION: &str = "fly-kc-mbon-rstdp-v2";
|
|||
pub struct Compatibility<'a> {
|
||||
/// `kernelVersion(config)` from the neural library.
|
||||
pub neural_kernel_version: &'a str,
|
||||
/// The adapter's version string, e.g. `pokered-unique8-v5`.
|
||||
/// The adapter's version string, e.g. `pokered-unique8-v6`.
|
||||
pub adapter: &'a str,
|
||||
/// The dataset's seven SHA-256 digests joined with `:`.
|
||||
pub dataset_fingerprint: &'a str,
|
||||
|
|
@ -67,6 +67,92 @@ impl Compatibility<'_> {
|
|||
}
|
||||
}
|
||||
|
||||
/// Position of the adapter's version string in [`Compatibility::string`].
|
||||
///
|
||||
/// `{kernel}/{adapter}/{fingerprint}/{plasticity}/binjgb:{rev}/pokered:{commit}/statefmt:{id}`,
|
||||
/// so the adapter is segment one. Nothing else in the string may move for a migration to be
|
||||
/// considered: a different kernel, dataset, plasticity, emulator revision, symbol provenance or
|
||||
/// state format is a different *fly*, not a different reward rule.
|
||||
const ADAPTER_SEGMENT: usize = 1;
|
||||
|
||||
/// The environment variable that opts a deploy into the adapter migration.
|
||||
///
|
||||
/// Read by flysim at restore and by `infra/05-deploy.sh`'s compatibility gate. Comma- or
|
||||
/// whitespace-separated adapter ids, e.g. `FLY_ACCEPT_ADAPTERS=pokered-unique8-v5`.
|
||||
pub const ACCEPT_ADAPTERS_ENV: &str = "FLY_ACCEPT_ADAPTERS";
|
||||
|
||||
/// What a build may do with a checkpoint whose compatibility string is not its own.
|
||||
#[derive(Debug, Clone, PartialEq, Eq)]
|
||||
pub enum RestoreDecision {
|
||||
/// Byte-identical. Restore it, as every build always has.
|
||||
Exact,
|
||||
/// Every segment but the adapter's is identical, this build's adapter says it can migrate
|
||||
/// from that one, and the operator named it in [`ACCEPT_ADAPTERS_ENV`]. Restore it.
|
||||
MigrateAdapter { from: String },
|
||||
/// Refuse, and say which of the three conditions failed.
|
||||
Refuse(&'static str),
|
||||
}
|
||||
|
||||
/// Decide whether `checkpoint`'s compatibility string may be restored under `current`.
|
||||
///
|
||||
/// Three conditions, all required, in the order they are cheapest to explain:
|
||||
///
|
||||
/// 1. the two strings differ in the adapter segment and **nowhere else**;
|
||||
/// 2. `migrates_from` -- the running adapter's own list -- contains the checkpoint's adapter, so
|
||||
/// the code that will read that state says out loud that it can;
|
||||
/// 3. `accepted` -- [`ACCEPT_ADAPTERS_ENV`] as the operator set it for this deploy -- contains it
|
||||
/// too, so no build ever migrates a run by itself.
|
||||
///
|
||||
/// Condition 2 without condition 3 would make the migration silent; condition 3 without condition
|
||||
/// 2 would let an operator wave through a pair nobody wrote a migration for. Neither alone is
|
||||
/// enough, which is why both are here.
|
||||
pub fn decide(
|
||||
checkpoint: &str,
|
||||
current: &str,
|
||||
migrates_from: &[&str],
|
||||
accepted: &[String],
|
||||
) -> RestoreDecision {
|
||||
if checkpoint == current {
|
||||
return RestoreDecision::Exact;
|
||||
}
|
||||
let old: Vec<&str> = checkpoint.split('/').collect();
|
||||
let new: Vec<&str> = current.split('/').collect();
|
||||
if old.len() != new.len() {
|
||||
return RestoreDecision::Refuse("the two compatibility strings do not have the same shape");
|
||||
}
|
||||
let differing: Vec<usize> = (0..old.len()).filter(|&index| old[index] != new[index]).collect();
|
||||
if differing != [ADAPTER_SEGMENT] {
|
||||
return RestoreDecision::Refuse(
|
||||
"more than the adapter version differs; nothing but a reward-rule change can migrate",
|
||||
);
|
||||
}
|
||||
let from = old[ADAPTER_SEGMENT];
|
||||
if !migrates_from.contains(&from) {
|
||||
return RestoreDecision::Refuse("this build's adapter has no migration from that adapter");
|
||||
}
|
||||
if !accepted.iter().any(|name| name == from) {
|
||||
return RestoreDecision::Refuse(
|
||||
"the checkpoint's adapter is not in FLY_ACCEPT_ADAPTERS, so the migration was not \
|
||||
asked for",
|
||||
);
|
||||
}
|
||||
RestoreDecision::MigrateAdapter { from: from.to_string() }
|
||||
}
|
||||
|
||||
/// Parse [`ACCEPT_ADAPTERS_ENV`]: comma- or whitespace-separated, empty entries dropped.
|
||||
///
|
||||
/// An unset variable and an empty one are the same thing -- no migration -- so that clearing the
|
||||
/// opt-in is one edit rather than two.
|
||||
pub fn accepted_adapters(value: Option<&str>) -> Vec<String> {
|
||||
value
|
||||
.unwrap_or_default()
|
||||
.split([',', ' ', '\t', '\n'])
|
||||
.map(str::trim)
|
||||
.filter(|entry| !entry.is_empty())
|
||||
.map(str::to_string)
|
||||
.collect()
|
||||
}
|
||||
|
||||
/// `<state size>-<target triple>`: the two things that decide whether a
|
||||
/// binjgb save state written elsewhere can be memcpy'd back in here.
|
||||
pub fn state_format_id() -> String {
|
||||
|
|
@ -94,7 +180,7 @@ mod tests {
|
|||
assert_eq!(
|
||||
fixture().prototype_string(),
|
||||
concat!(
|
||||
"lif-1ms-f64-v2/pokered-unique8-v5/aa:bb:cc:dd:ee:ff:00/",
|
||||
"lif-1ms-f64-v2/pokered-unique8-v6/aa:bb:cc:dd:ee:ff:00/",
|
||||
"fly-kc-mbon-rstdp-v2/",
|
||||
"binjgb:c60e138da5a795ebb55e56b11b7e90024e41112c/",
|
||||
"pokered:0cd19d3b877b7dc66d12c7050bed9a7f38154d4b",
|
||||
|
|
@ -102,6 +188,73 @@ mod tests {
|
|||
);
|
||||
}
|
||||
|
||||
fn with_adapter(adapter: &'static str) -> String {
|
||||
Compatibility { adapter, ..fixture() }.string()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn an_identical_string_restores_without_any_opt_in() {
|
||||
let current = with_adapter("pokered-unique8-v6");
|
||||
assert_eq!(decide(¤t, ¤t, &[], &[]), RestoreDecision::Exact);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_v5_checkpoint_restores_under_v6_only_with_the_opt_in() {
|
||||
let old = with_adapter("pokered-unique8-v5");
|
||||
let new = with_adapter("pokered-unique8-v6");
|
||||
let migrates = ["pokered-unique8-v5"];
|
||||
|
||||
assert!(matches!(decide(&old, &new, &migrates, &[]), RestoreDecision::Refuse(_)));
|
||||
assert_eq!(
|
||||
decide(&old, &new, &migrates, &accepted_adapters(Some("pokered-unique8-v5"))),
|
||||
RestoreDecision::MigrateAdapter { from: "pokered-unique8-v5".to_string() }
|
||||
);
|
||||
// And only for a pair the running adapter says it can migrate.
|
||||
assert!(matches!(
|
||||
decide(&old, &new, &[], &accepted_adapters(Some("pokered-unique8-v5"))),
|
||||
RestoreDecision::Refuse(_)
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn nothing_but_the_adapter_segment_may_move() {
|
||||
let migrates = ["pokered-unique8-v5"];
|
||||
let accepted = accepted_adapters(Some("pokered-unique8-v5"));
|
||||
let new = with_adapter("pokered-unique8-v6");
|
||||
|
||||
// A different dataset, with the same adapter bump, is not a migration.
|
||||
let other_dataset = Compatibility {
|
||||
adapter: "pokered-unique8-v5",
|
||||
dataset_fingerprint: "00:11:22:33:44:55:66",
|
||||
..fixture()
|
||||
}
|
||||
.string();
|
||||
assert!(matches!(
|
||||
decide(&other_dataset, &new, &migrates, &accepted),
|
||||
RestoreDecision::Refuse(_)
|
||||
));
|
||||
|
||||
// Neither is a different kernel, and neither is a string of another shape.
|
||||
let other_kernel =
|
||||
Compatibility { adapter: "pokered-unique8-v5", neural_kernel_version: "lif-1ms-f64-v3", ..fixture() }
|
||||
.string();
|
||||
assert!(matches!(
|
||||
decide(&other_kernel, &new, &migrates, &accepted),
|
||||
RestoreDecision::Refuse(_)
|
||||
));
|
||||
assert!(matches!(decide("a/b", &new, &migrates, &accepted), RestoreDecision::Refuse(_)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_opt_in_list_is_separated_by_commas_or_spaces() {
|
||||
assert!(accepted_adapters(None).is_empty());
|
||||
assert!(accepted_adapters(Some(" ")).is_empty());
|
||||
assert_eq!(
|
||||
accepted_adapters(Some("pokered-unique8-v5, pokered-unique8-v4")),
|
||||
vec!["pokered-unique8-v5".to_string(), "pokered-unique8-v4".to_string()]
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_state_format_segment_is_appended_not_interleaved() {
|
||||
let full = fixture().string();
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
//! The `pokered-unique8-v5` reward catalog.
|
||||
//! The `pokered-unique8-v6` reward catalog.
|
||||
//!
|
||||
//! A direct port of the prototype's `src/reward/catalog.ts`, including the
|
||||
//! declaration order, which is the order `counts` and `last` serialize in.
|
||||
|
|
@ -20,8 +20,18 @@ pub mod kind {
|
|||
pub const BATTLE: &str = "battle";
|
||||
pub const BADGE: &str = "badge";
|
||||
pub const BOUNDARY: &str = "boundary";
|
||||
pub const CATCH: &str = "catch";
|
||||
}
|
||||
|
||||
/// What a `catch` of a species this run has already caught pays.
|
||||
///
|
||||
/// Not a multiple of the rule's catalog value, because no binary float scales 0.30 into
|
||||
/// exactly 0.10: `0.3 * (1.0 / 3.0)` is `0.09999999999999999`, and that number would reach
|
||||
/// the ticker, the checkpoint and `docs/rewards-learning.md`'s table as itself. `boundary`'s
|
||||
/// two payouts are 0.05 and 0.10, which a scale of two does express exactly, so that rule
|
||||
/// still goes through the scaling path.
|
||||
pub const CATCH_REPEAT_VALUE: f64 = 0.10;
|
||||
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
pub struct RewardRule {
|
||||
pub kind: &'static str,
|
||||
|
|
@ -34,7 +44,7 @@ pub struct RewardRule {
|
|||
pub stimulation_ms: u32,
|
||||
}
|
||||
|
||||
pub const REWARDS: [RewardRule; 8] = [
|
||||
pub const REWARDS: [RewardRule; 9] = [
|
||||
RewardRule {
|
||||
kind: kind::MILESTONE,
|
||||
label: "Story",
|
||||
|
|
@ -99,6 +109,22 @@ pub const REWARDS: [RewardRule; 8] = [
|
|||
value: 0.05,
|
||||
stimulation_ms: 100,
|
||||
},
|
||||
// The operator's decision of 2026-09-22: the fly is paid for *keeping* a wild Pokémon, not
|
||||
// only for knocking one out. Appended rather than slotted next to `species` for the same
|
||||
// reason `boundary` was appended -- the declaration order is the key order `counts`
|
||||
// serializes in, and every checkpoint already written carries the first eight in this order.
|
||||
//
|
||||
// One rule, two payouts, like `boundary`: this value is what a species this run has never
|
||||
// caught pays, and [`CATCH_REPEAT_VALUE`] is what a repeat pays. The existing `species`
|
||||
// rule is untouched and still pays 0.50 the first time a species is owned by any means, so
|
||||
// a first catch of a new species pays 0.50 + 0.30 across two kinds.
|
||||
RewardRule {
|
||||
kind: kind::CATCH,
|
||||
label: "Catch",
|
||||
trigger: "Wild Pokémon caught; 0.10 for a species already caught; max 3 per species",
|
||||
value: 0.30,
|
||||
stimulation_ms: 150,
|
||||
},
|
||||
];
|
||||
|
||||
/// Position of `kind` in [`REWARDS`], or `None` for an unknown kind. This is
|
||||
|
|
@ -199,10 +225,34 @@ mod tests {
|
|||
// separate kinds.
|
||||
assert_eq!(rule(kind::BOUNDARY).unwrap().value, 0.05);
|
||||
assert_eq!(rule(kind::BOUNDARY).unwrap().stimulation_ms, 100);
|
||||
// Nor the prototype's: the operator's catch rule, `pokered-unique8-v6`.
|
||||
assert_eq!(rule(kind::CATCH).unwrap().value, 0.30);
|
||||
assert_eq!(CATCH_REPEAT_VALUE, 0.10);
|
||||
assert_eq!(rule(kind::CATCH).unwrap().stimulation_ms, 150);
|
||||
assert!(rule("blackout").is_none(), "the catalog has no penalties");
|
||||
assert!(REWARDS.iter().all(|rule| rule.value > 0.0));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_catch_rule_is_last_so_the_older_key_order_does_not_move() {
|
||||
let order: Vec<&str> = REWARDS.iter().map(|rule| rule.kind).collect();
|
||||
assert_eq!(
|
||||
order,
|
||||
vec![
|
||||
kind::MILESTONE,
|
||||
kind::EXPLORATION,
|
||||
kind::MAP,
|
||||
kind::SPECIES,
|
||||
kind::TRAINER,
|
||||
kind::BATTLE,
|
||||
kind::BADGE,
|
||||
kind::BOUNDARY,
|
||||
kind::CATCH,
|
||||
]
|
||||
);
|
||||
assert_eq!(index(kind::CATCH), Some(REWARDS.len() - 1));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_counts_lists_every_kind_at_zero() {
|
||||
let counts = Counts::default();
|
||||
|
|
|
|||
|
|
@ -1,11 +1,12 @@
|
|||
//! The Pokémon Red reward adapter, `pokered-unique8-v5`.
|
||||
//! The Pokémon Red reward adapter, `pokered-unique8-v6`.
|
||||
//!
|
||||
//! A port of the prototype's `src/reward/pokemon-red.ts`. The gates and budgets
|
||||
//! are unchanged; `docs/rewards-learning.md` holds the live rule table and the
|
||||
//! source evidence behind each gate. v4 replaced the 16-rung boot-to-badges
|
||||
//! ladder with the 38 rungs of `docs/design/ladder.md`; v5 adds one reward rule,
|
||||
//! `boundary` (`docs/design/room-escape.md` section 2), which pays the first step
|
||||
//! next to and the first step onto each of a map's exits.
|
||||
//! next to and the first step onto each of a map's exits; v6 adds `catch`, the
|
||||
//! operator's decision of 2026-09-22, which pays for keeping a wild Pokémon.
|
||||
|
||||
pub mod catalog;
|
||||
#[cfg(test)]
|
||||
|
|
@ -32,13 +33,33 @@ use symbols::ram;
|
|||
|
||||
/// Adapter version, pinned into the checkpoint compatibility string.
|
||||
///
|
||||
/// `v5` is the `boundary` rule. Bumping it is what rejects every checkpoint written
|
||||
/// by `v4`: the string is compared whole before a restore is attempted, so a ledger
|
||||
/// that has never recorded a single `boundary:` key can never be resumed as though
|
||||
/// its exits were already collected. (`v4` was the 38-rung ladder, and rejected
|
||||
/// `v3` for the same reason: a stored rank that meant "4 badges" on the old ladder
|
||||
/// could not be read as a rung on the new one.) Pre-launch, so no run is lost.
|
||||
pub const REWARD_ADAPTER: &str = "pokered-unique8-v5";
|
||||
/// `v6` is the `catch` rule. Bumping it is what makes a `v5` checkpoint a decision
|
||||
/// rather than an accident: the compatibility string is compared whole before a
|
||||
/// restore is attempted, so a `v5` run is refused by default and resumed only when
|
||||
/// the operator names it in `FLY_ACCEPT_ADAPTERS`
|
||||
/// ([`crate::compatibility::RestoreDecision`], `docs/design/flysim.md`). That
|
||||
/// migration is safe in one direction only, and only for this pair: `v5`'s ledger is
|
||||
/// a `v6` ledger with the catch counter absent, and an absent counter reads as zero.
|
||||
///
|
||||
/// (`v5` was the `boundary` rule, and rejected `v4` because a ledger that had never
|
||||
/// recorded a `boundary:` key could not be resumed as though its exits were already
|
||||
/// collected. `v4` was the 38-rung ladder, and rejected `v3` because a stored rank
|
||||
/// that meant "4 badges" on the old ladder is not a rung on the new one. Neither of
|
||||
/// those is a migration: this one is, because nothing a `v5` ledger holds means
|
||||
/// something different under `v6`.)
|
||||
pub const REWARD_ADAPTER: &str = "pokered-unique8-v6";
|
||||
|
||||
/// Adapter ids whose checkpoints `v6` can read.
|
||||
///
|
||||
/// Exactly one, and it is one because the `catch` rule adds a counter and changes nothing else:
|
||||
/// a `v5` ledger restores as a `v6` ledger with `catchCounts` empty, and every other byte of the
|
||||
/// state means what it meant. `v4` is not here -- its `seen` ledger holds no `boundary:` keys, so
|
||||
/// resuming it would pay a second time for every exit the run had already found -- and neither is
|
||||
/// `v3`, whose stored rank is a rung on a different ladder.
|
||||
///
|
||||
/// Listing an id here is necessary but not sufficient: `FLY_ACCEPT_ADAPTERS` must name it too
|
||||
/// (`crate::compatibility::decide`, `docs/design/flysim.md`).
|
||||
pub const MIGRATES_FROM: &[&str] = &["pokered-unique8-v5"];
|
||||
|
||||
/// The only cartridge semantic rewards are enabled for. Even the canonical
|
||||
/// pret build stays disabled until reviewed; see `docs/rewards-learning.md`.
|
||||
|
|
@ -60,8 +81,22 @@ pub const SUPPORTED_ROM: &str =
|
|||
/// [`REWARD_ADAPTER`] is the gate that refuses such a checkpoint anyway, and it is
|
||||
/// the right gate, because the objection to loading one is about semantics rather
|
||||
/// than shape.
|
||||
///
|
||||
/// *Not* bumped for the `catch` rule either, and this time the answer matters,
|
||||
/// because `v5` checkpoints are meant to be restorable under `v6`. The rule adds one
|
||||
/// counter, `catchCounts`, and nothing else: every other field keeps its name, its
|
||||
/// shape and its meaning, and a state written without the counter restores with it
|
||||
/// empty, which is the truth about a run that was never paid for a catch. That is the
|
||||
/// whole of the documented `v5` -> `v6` migration; see
|
||||
/// [`crate::compatibility::RestoreDecision`].
|
||||
pub const STATE_VERSION: u64 = 4;
|
||||
|
||||
/// Catch payouts one species may earn in the lifetime of a run's ledger.
|
||||
///
|
||||
/// The same cap and the same reason as the wild-KO rule's three: a species the fly can
|
||||
/// find over and over is a farm, and three is enough for the behaviour to be learned.
|
||||
const MAX_CATCH_PAYOUTS: u64 = 3;
|
||||
|
||||
const BADGE_NAMES: [&str; 8] = [
|
||||
"BOULDER", "CASCADE", "THUNDER", "RAINBOW", "SOUL", "MARSH", "VOLCANO", "EARTH",
|
||||
];
|
||||
|
|
@ -325,6 +360,26 @@ struct Battle {
|
|||
wild: bool,
|
||||
saw_living: bool,
|
||||
ko: bool,
|
||||
/// Lifetime `species` payouts when this battle started.
|
||||
///
|
||||
/// The "never owned this run" test for the catch rule, and an exact one: the only
|
||||
/// thing that can set a `wPokedexOwned` bit during a wild battle is the catch
|
||||
/// itself, so a `species` payout between the battle starting and the ball keeping
|
||||
/// the Pokémon *is* that Pokémon being new. It is read this way rather than from
|
||||
/// `wCapturedMonSpecies` directly because that byte is the cartridge's **internal**
|
||||
/// species index and the owned bitset is by **Pokédex number**; the two numberings
|
||||
/// differ and nothing in WRAM converts between them
|
||||
/// (`docs/design/macros-wram.md` section 2, "species numbering").
|
||||
///
|
||||
/// `None` for a battle restored from a checkpoint written before this existed,
|
||||
/// which reads as "cannot tell" and pays the repeat amount rather than guessing
|
||||
/// generously.
|
||||
species_at_start: Option<u64>,
|
||||
/// The internal species index `wCapturedMonSpecies` named, once a ball has kept one.
|
||||
captured: Option<u8>,
|
||||
/// Whether that catch was a species this run had never owned, decided on the frame
|
||||
/// the capture was observed.
|
||||
captured_new: bool,
|
||||
}
|
||||
|
||||
/// Immutable per-sample byte cache. Each address requested during one sample
|
||||
|
|
@ -370,6 +425,10 @@ pub struct PokemonRedReward {
|
|||
tiles: OrderedSet,
|
||||
tile_counts: BTreeMap<u8, u64>,
|
||||
wild_wins: BTreeMap<String, u64>,
|
||||
/// Catch payouts per species, by the cartridge's internal species index as a decimal
|
||||
/// string. The one field `v6` adds to the checkpoint; absent in a `v5` state, which
|
||||
/// reads as every species at zero.
|
||||
catch_counts: BTreeMap<String, u64>,
|
||||
replay_blocked: OrderedSet,
|
||||
counts: Counts,
|
||||
total: f64,
|
||||
|
|
@ -420,6 +479,7 @@ impl PokemonRedReward {
|
|||
tiles: OrderedSet::new(),
|
||||
tile_counts: BTreeMap::new(),
|
||||
wild_wins: BTreeMap::new(),
|
||||
catch_counts: BTreeMap::new(),
|
||||
replay_blocked: OrderedSet::new(),
|
||||
counts: Counts::default(),
|
||||
total: 0.0,
|
||||
|
|
@ -567,8 +627,9 @@ impl PokemonRedReward {
|
|||
}
|
||||
|
||||
/// Forget observations a rollback invalidates. Lifetime novelty survives,
|
||||
/// and every wild-KO key paid so far is blocked from paying again, because
|
||||
/// after a rollback the same battle could otherwise be replayed for reward.
|
||||
/// and every wild-KO key and every caught species paid so far is blocked from
|
||||
/// paying again, because after a rollback the same battle -- or the same catch --
|
||||
/// could otherwise be replayed for reward.
|
||||
pub fn clear_transient(&mut self) {
|
||||
self.location.clear();
|
||||
self.stable = 0;
|
||||
|
|
@ -578,6 +639,10 @@ impl PokemonRedReward {
|
|||
for key in keys {
|
||||
self.replay_blocked.insert(&key);
|
||||
}
|
||||
let caught: Vec<String> = self.catch_counts.keys().cloned().collect();
|
||||
for species in caught {
|
||||
self.replay_blocked.insert(&format!("catch:{species}"));
|
||||
}
|
||||
}
|
||||
|
||||
/// Sample WRAM after one completed frame and return this frame's payouts.
|
||||
|
|
@ -690,14 +755,29 @@ impl PokemonRedReward {
|
|||
if in_battle == 1 || in_battle == 2 || in_battle == 255 {
|
||||
self.mode = "BATTLE".to_string();
|
||||
self.stable = 0;
|
||||
let species_paid = self.counts.get(kind::SPECIES);
|
||||
if self.battle.is_none() && in_battle != 255 {
|
||||
self.battle = Some(Battle {
|
||||
key: battle_key(memory, map),
|
||||
wild: in_battle == 1,
|
||||
saw_living: false,
|
||||
ko: false,
|
||||
species_at_start: Some(species_paid),
|
||||
captured: None,
|
||||
captured_new: false,
|
||||
});
|
||||
}
|
||||
// The cartridge's own answer to "was one caught": `ram/wram.asm`'s comment on
|
||||
// this byte is "0 if no mon was captured". `ItemUseBall` zeroes it before every
|
||||
// throw and writes `wEnemyMonSpecies` into it only on the branch that keeps the
|
||||
// Pokémon, and `UseBagItem`'s `.returnAfterCapturingMon` zeroes it again on the
|
||||
// way out of the battle -- so it is non-zero for the hundreds of frames the
|
||||
// catch's own text and Pokédex screen take, and zero everywhere else.
|
||||
//
|
||||
// Read rather than derived from `wPartyCount`, because a catch with a full party
|
||||
// raises `wBoxCount` instead, and because `wPartyCount` also rises for a gift, a
|
||||
// trade and a revive out of the PC.
|
||||
let captured = memory.read8(ram::wCapturedMonSpecies);
|
||||
if let Some(battle) = &mut self.battle {
|
||||
let hp = word(memory, ram::wEnemyMonHP);
|
||||
let max = word(memory, ram::wEnemyMonMaxHP);
|
||||
|
|
@ -710,6 +790,11 @@ impl PokemonRedReward {
|
|||
if battle.saw_living && hp == 0 {
|
||||
battle.ko = true;
|
||||
}
|
||||
if battle.wild && captured != 0 && battle.captured.is_none() {
|
||||
battle.captured = Some(captured);
|
||||
battle.captured_new =
|
||||
battle.species_at_start.is_some_and(|before| species_paid > before);
|
||||
}
|
||||
}
|
||||
} else if in_battle == 0 {
|
||||
self.mode = "OVERWORLD".to_string();
|
||||
|
|
@ -728,6 +813,41 @@ impl PokemonRedReward {
|
|||
}
|
||||
self.wild_wins.insert(battle.key.clone(), (count + 1).min(3));
|
||||
}
|
||||
// The catch rule (`docs/rewards-learning.md`, the operator 2026-09-22).
|
||||
//
|
||||
// Paid on the way out of the battle rather than on the capture frame, so that
|
||||
// it lands in the same place the wild-KO payout does and cannot fire twice for
|
||||
// one battle. `wBattleResult` is 2 on exactly two paths in the game:
|
||||
// `UseBagItem`'s `.returnAfterCapturingMon`, which is this one, and a link
|
||||
// battle whose opponent ran (`engine/battle/core.asm`), which this cartridge
|
||||
// never has. Requiring it as well as the captured species means a byte read
|
||||
// out of a half-initialised battle cannot pay.
|
||||
if let Some(species) = battle.captured
|
||||
&& battle.wild
|
||||
&& result == 2
|
||||
{
|
||||
let key = species.to_string();
|
||||
let paid = self.catch_counts.get(&key).copied().unwrap_or(0);
|
||||
if paid < MAX_CATCH_PAYOUTS
|
||||
&& !self.replay_blocked.contains(&format!("catch:{key}"))
|
||||
{
|
||||
let value = if battle.captured_new {
|
||||
catalog::rule(kind::CATCH)
|
||||
.expect("the catch rule is in the catalog")
|
||||
.value
|
||||
} else {
|
||||
catalog::CATCH_REPEAT_VALUE
|
||||
};
|
||||
self.emit_amount(
|
||||
&mut emitted,
|
||||
kind::CATCH,
|
||||
format!("CAUGHT #{species}"),
|
||||
value,
|
||||
brain_ms,
|
||||
);
|
||||
}
|
||||
self.catch_counts.insert(key, (paid + 1).min(MAX_CATCH_PAYOUTS));
|
||||
}
|
||||
}
|
||||
let location = format!("{map}:{x}:{y}");
|
||||
self.stable = if self.location == location { self.stable + 1 } else { 1 };
|
||||
|
|
@ -825,13 +945,33 @@ impl PokemonRedReward {
|
|||
label: String,
|
||||
scale: f64,
|
||||
brain_ms: f64,
|
||||
) {
|
||||
let rule = catalog::rule(kind).expect("emit is only called with catalog kinds");
|
||||
self.emit_amount(emitted, kind, label, rule.value * scale, brain_ms);
|
||||
}
|
||||
|
||||
/// [`PokemonRedReward::emit`] with the payout stated outright instead of as a multiple
|
||||
/// of the catalog value.
|
||||
///
|
||||
/// One rule needs it. `catch` pays 0.30 for a species this run has not caught and 0.10
|
||||
/// for one it has, and no binary float scales the first into exactly the second:
|
||||
/// `0.3 * (1.0 / 3.0)` is `0.09999999999999999`, and that is the number that would reach
|
||||
/// the ticker and the checkpoint. `boundary`'s pair, 0.05 and 0.10, *is* an exact scale
|
||||
/// of two, so that rule still goes through [`PokemonRedReward::emit`].
|
||||
fn emit_amount(
|
||||
&mut self,
|
||||
emitted: &mut Vec<RewardEvent>,
|
||||
kind: &'static str,
|
||||
label: String,
|
||||
value: f64,
|
||||
brain_ms: f64,
|
||||
) {
|
||||
let rule = catalog::rule(kind).expect("emit is only called with catalog kinds");
|
||||
let event = RewardEvent {
|
||||
kind,
|
||||
label,
|
||||
brain_ms,
|
||||
value: rule.value * scale,
|
||||
value,
|
||||
stimulation_ms: rule.stimulation_ms,
|
||||
};
|
||||
emitted.push(event.clone());
|
||||
|
|
@ -991,6 +1131,10 @@ impl PokemonRedReward {
|
|||
"tiles": self.tiles.as_slice(),
|
||||
"tileCounts": self.tile_counts,
|
||||
"wildWins": self.wild_wins,
|
||||
// The one field v6 adds. A v5 state does not carry it and restores with it
|
||||
// empty, which is the documented v5 -> v6 migration and the truth about a run
|
||||
// that was never paid for a catch.
|
||||
"catchCounts": self.catch_counts,
|
||||
"replayBlocked": self.replay_blocked.as_slice(),
|
||||
"counts": self.counts,
|
||||
"total": self.total,
|
||||
|
|
@ -1011,6 +1155,9 @@ impl PokemonRedReward {
|
|||
"wild": battle.wild,
|
||||
"sawLiving": battle.saw_living,
|
||||
"ko": battle.ko,
|
||||
"speciesAtStart": battle.species_at_start,
|
||||
"captured": battle.captured,
|
||||
"capturedNew": battle.captured_new,
|
||||
})),
|
||||
"mode": self.mode,
|
||||
})
|
||||
|
|
@ -1056,6 +1203,13 @@ impl PokemonRedReward {
|
|||
let counts_raw = counted_record(input.get("counts")).ok_or(BAD_CHECKPOINT)?;
|
||||
let tile_counts_raw = counted_record(input.get("tileCounts")).ok_or(BAD_CHECKPOINT)?;
|
||||
let wild_wins_raw = counted_record(input.get("wildWins")).ok_or(BAD_CHECKPOINT)?;
|
||||
// Absent in every v5 state, and that absence is the migration: no species has been
|
||||
// paid for a catch, because the rule did not exist. Present but malformed is still
|
||||
// an error, the same as every other counter here.
|
||||
let catch_counts = match input.get("catchCounts") {
|
||||
None | Some(Value::Null) => BTreeMap::new(),
|
||||
Some(value) => counted_record(Some(value)).ok_or(BAD_CHECKPOINT)?,
|
||||
};
|
||||
|
||||
let recent = recent_raw
|
||||
.iter()
|
||||
|
|
@ -1078,6 +1232,19 @@ impl PokemonRedReward {
|
|||
wild: value.get("wild").and_then(Value::as_bool).ok_or(BAD_HISTORY)?,
|
||||
saw_living: value.get("sawLiving").and_then(Value::as_bool).ok_or(BAD_HISTORY)?,
|
||||
ko: value.get("ko").and_then(Value::as_bool).ok_or(BAD_HISTORY)?,
|
||||
// All three are v6's, and all three are optional for the same reason
|
||||
// `catchCounts` is. A v5 battle carries no `speciesAtStart`, which reads as
|
||||
// "cannot tell whether the caught species was new" and pays the repeat
|
||||
// amount: the conservative half of the rule, and at most 0.20 once.
|
||||
species_at_start: value.get("speciesAtStart").and_then(Value::as_u64),
|
||||
captured: value
|
||||
.get("captured")
|
||||
.and_then(Value::as_u64)
|
||||
.and_then(|species| u8::try_from(species).ok()),
|
||||
captured_new: value
|
||||
.get("capturedNew")
|
||||
.and_then(Value::as_bool)
|
||||
.unwrap_or(false),
|
||||
}),
|
||||
};
|
||||
let replay_blocked = match input.get("replayBlocked") {
|
||||
|
|
@ -1103,6 +1270,7 @@ impl PokemonRedReward {
|
|||
}
|
||||
self.tile_counts = tile_counts;
|
||||
self.wild_wins = wild_wins_raw;
|
||||
self.catch_counts = catch_counts;
|
||||
self.replay_blocked = replay_blocked.iter().map(String::as_str).collect();
|
||||
self.counts = Counts::default();
|
||||
for (key, count) in &counts_raw {
|
||||
|
|
@ -1174,6 +1342,10 @@ impl GameAdapter for PokemonRedReward {
|
|||
REWARD_ADAPTER
|
||||
}
|
||||
|
||||
fn migrates_from(&self) -> &'static [&'static str] {
|
||||
MIGRATES_FROM
|
||||
}
|
||||
|
||||
fn rom_allowed(&self, sha256: &str) -> bool {
|
||||
sha256 == SUPPORTED_ROM
|
||||
}
|
||||
|
|
|
|||
|
|
@ -47,6 +47,7 @@ pub mod ram {
|
|||
pub const wBattleType: u16 = 0xd05a; // 53338
|
||||
pub const wTrainerNo: u16 = 0xd05d; // 53341
|
||||
pub const wPartyMenuTypeOrMessageID: u16 = 0xd07d; // 53373
|
||||
pub const wCapturedMonSpecies: u16 = 0xd11c; // 53532
|
||||
pub const wForcePlayerToChooseMon: u16 = 0xd11f; // 53535
|
||||
pub const wTextBoxID: u16 = 0xd125; // 53541
|
||||
pub const wPartyCount: u16 = 0xd163; // 53603
|
||||
|
|
|
|||
|
|
@ -83,6 +83,34 @@ impl Fixture {
|
|||
self.visit(x, 0)
|
||||
}
|
||||
|
||||
/// One wild battle that ends in a ball keeping the Pokémon, byte for byte as the
|
||||
/// cartridge writes it at the pinned commit.
|
||||
///
|
||||
/// `InitBattleVariables` clears `wBattleResult`; `ItemUseBall`'s capture branch sets the
|
||||
/// Pokédex bit (for a species the player did not already own) and writes
|
||||
/// `wEnemyMonSpecies` into `wCapturedMonSpecies`; `UseBagItem`'s
|
||||
/// `.returnAfterCapturingMon` then zeroes that byte, sets `wBattleResult` to 2 and leaves
|
||||
/// the battle. `dex` is the Pokédex *number* minus one, i.e. the bit index, and `None` is a
|
||||
/// species this run already owns.
|
||||
fn catch(&mut self, species: u8, dex: Option<u16>) -> Vec<RewardEvent> {
|
||||
self.memory.set(ram::wBattleResult, 0);
|
||||
self.memory.set(ram::wIsInBattle, 1);
|
||||
self.memory.set(ram::wEnemyMonSpecies, species);
|
||||
self.memory.set(ram::wEnemyMonHP + 1, 10);
|
||||
self.memory.set(ram::wEnemyMonMaxHP + 1, 10);
|
||||
let mut events = self.sample();
|
||||
if let Some(index) = dex {
|
||||
self.memory.or(ram::wPokedexOwned + (index >> 3), 1 << (index & 7));
|
||||
}
|
||||
self.memory.set(ram::wCapturedMonSpecies, species);
|
||||
events.extend(self.sample());
|
||||
self.memory.set(ram::wCapturedMonSpecies, 0);
|
||||
self.memory.set(ram::wBattleResult, 2);
|
||||
self.memory.set(ram::wIsInBattle, 0);
|
||||
events.extend(self.sample());
|
||||
events
|
||||
}
|
||||
|
||||
/// Write a warp table: `wNumberOfWarps` plus one four-byte `Y, X, warp id, map id` entry per
|
||||
/// `(x, y)`, the layout `ram/wram.asm` documents at the pinned commit.
|
||||
fn warps(&mut self, warps: &[(u8, u8)]) {
|
||||
|
|
@ -320,6 +348,136 @@ fn a_wild_run_capture_or_single_faint_never_pays_a_ko_while_a_verified_ko_does()
|
|||
assert_eq!(f.reward.statistics().counts[kind::BATTLE], 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_catch_pays_the_new_species_amount_once_the_repeat_amount_after_and_stops_at_three() {
|
||||
let mut f = Fixture::new();
|
||||
f.sample();
|
||||
|
||||
// A species this run has never owned: the cartridge sets the Pokédex bit on the way
|
||||
// through, so the existing `species` rule pays 0.50 and the new rule pays 0.30.
|
||||
let first = f.catch(0xb0, Some(3));
|
||||
assert_eq!(kinds(&first), ["species", "catch"]);
|
||||
assert_eq!(labels(&first), ["OWNED #4", "CAUGHT #176"]);
|
||||
assert!((first[0].value - 0.5).abs() < 1e-12, "the species rule is untouched");
|
||||
assert!((first[1].value - 0.30).abs() < 1e-12);
|
||||
|
||||
// The same species again: a repeat, twice, and then the cap.
|
||||
for _ in 0..2 {
|
||||
let again = f.catch(0xb0, None);
|
||||
assert_eq!(kinds(&again), ["catch"]);
|
||||
assert_eq!(again[0].value, 0.10, "the repeat amount is exactly 0.10, not 0.3/3");
|
||||
}
|
||||
assert!(f.catch(0xb0, None).is_empty(), "three payouts per species is the cap");
|
||||
assert_eq!(f.reward.statistics().counts[kind::CATCH], 3);
|
||||
|
||||
// Another species starts its own count, and its own 0.30.
|
||||
let other = f.catch(0x99, Some(0));
|
||||
assert_eq!(kinds(&other), ["species", "catch"]);
|
||||
assert!((other[1].value - 0.30).abs() < 1e-12);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_catch_of_a_species_this_run_already_owns_pays_the_repeat_amount() {
|
||||
let mut f = Fixture::new();
|
||||
f.sample();
|
||||
// Owned before the battle -- a gift, a trade, an evolution -- so no Pokédex bit is set
|
||||
// during it and the catch is not a new species.
|
||||
f.memory.or(ram::wPokedexOwned, 1);
|
||||
assert_eq!(kinds(&f.sample()), ["species"]);
|
||||
|
||||
let events = f.catch(0x99, None);
|
||||
assert_eq!(kinds(&events), ["catch"]);
|
||||
assert_eq!(events[0].value, 0.10);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn nothing_but_a_wild_catch_pays_the_catch_rule() {
|
||||
// A trainer battle: balls cannot be thrown, and `wIsInBattle` is 2.
|
||||
let mut f = Fixture::new();
|
||||
f.sample();
|
||||
f.memory.set(ram::wIsInBattle, 2);
|
||||
f.memory.set(ram::wEnemyMonHP + 1, 10);
|
||||
f.memory.set(ram::wEnemyMonMaxHP + 1, 10);
|
||||
f.sample();
|
||||
f.memory.set(ram::wCapturedMonSpecies, 0xb0);
|
||||
f.sample();
|
||||
f.memory.set(ram::wCapturedMonSpecies, 0);
|
||||
f.memory.set(ram::wBattleResult, 2);
|
||||
f.memory.set(ram::wIsInBattle, 0);
|
||||
assert!(f.sample().is_empty(), "a trainer battle never pays the catch rule");
|
||||
|
||||
// The Safari Zone and the old man's tutorial are excluded a step earlier: the whole
|
||||
// sample is dropped with a visible mode, so no battle is ever opened.
|
||||
for battle_type in [1u8, 2] {
|
||||
let mut f = Fixture::new();
|
||||
f.sample();
|
||||
f.memory.set(ram::wBattleType, battle_type);
|
||||
f.memory.set(ram::wIsInBattle, 1);
|
||||
assert!(f.sample().is_empty());
|
||||
f.memory.set(ram::wCapturedMonSpecies, 0xb0);
|
||||
assert!(f.sample().is_empty());
|
||||
f.memory.set(ram::wCapturedMonSpecies, 0);
|
||||
f.memory.set(ram::wBattleResult, 2);
|
||||
f.memory.set(ram::wIsInBattle, 0);
|
||||
f.memory.set(ram::wBattleType, 0);
|
||||
assert!(f.sample().is_empty());
|
||||
assert_eq!(f.reward.statistics().counts[kind::CATCH], 0);
|
||||
}
|
||||
|
||||
// A ball that missed: `wCapturedMonSpecies` never leaves zero and the battle ends as a
|
||||
// run or a loss.
|
||||
let mut f = Fixture::new();
|
||||
f.sample();
|
||||
f.memory.set(ram::wIsInBattle, 1);
|
||||
f.memory.set(ram::wEnemyMonHP + 1, 10);
|
||||
f.memory.set(ram::wEnemyMonMaxHP + 1, 10);
|
||||
f.sample();
|
||||
f.memory.set(ram::wIsInBattle, 0);
|
||||
assert!(f.sample().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_rollback_cannot_replay_a_catch() {
|
||||
let mut f = Fixture::new();
|
||||
f.sample();
|
||||
assert_eq!(kinds(&f.catch(0xb0, Some(3))), ["species", "catch"]);
|
||||
|
||||
f.reward.clear_transient();
|
||||
let state = f.reward.export_state();
|
||||
f.reward.import_state(&state).unwrap();
|
||||
assert!(f.catch(0xb0, None).is_empty(), "an already-paid species cannot pay after rollback");
|
||||
assert_eq!(f.reward.statistics().counts[kind::CATCH], 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_v5_state_restores_under_v6_with_the_catch_counter_at_zero() {
|
||||
let mut f = Fixture::new();
|
||||
f.sample();
|
||||
f.catch(0xb0, Some(3));
|
||||
let v6 = f.reward.export_state();
|
||||
assert_eq!(v6["catchCounts"], json!({ "176": 1 }));
|
||||
|
||||
// The v5 shape is this one without the counter the rule added: same `version`, same field
|
||||
// names, same meanings. That is the whole of the documented migration.
|
||||
let mut v5 = v6.clone();
|
||||
v5.as_object_mut().unwrap().remove("catchCounts");
|
||||
assert_eq!(v5["version"], json!(STATE_VERSION), "v5 and v6 states share a schema version");
|
||||
|
||||
let mut restored = PokemonRedReward::new();
|
||||
restored.import_state(&v5).unwrap();
|
||||
let mut expected = v6.clone();
|
||||
expected["catchCounts"] = json!({});
|
||||
assert_eq!(restored.export_state(), expected, "the counter starts at 0, nothing else moves");
|
||||
|
||||
// A genuine v5 `counts` object carries eight kinds and no `catch`, which reads as zero.
|
||||
let mut older = v5.clone();
|
||||
older["counts"].as_object_mut().unwrap().remove("catch");
|
||||
let mut restored = PokemonRedReward::new();
|
||||
restored.import_state(&older).unwrap();
|
||||
assert_eq!(restored.statistics().counts[kind::CATCH], 0);
|
||||
assert_eq!(restored.statistics().counts[kind::SPECIES], 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_repeated_wild_ko_decays_then_stops() {
|
||||
let mut f = Fixture::new();
|
||||
|
|
@ -378,6 +536,7 @@ fn malformed_checkpoint_fields_are_named_in_the_error() {
|
|||
("counts", json!([])),
|
||||
("tileCounts", json!("not a record")),
|
||||
("wildWins", json!(3)),
|
||||
("catchCounts", json!(3)),
|
||||
] {
|
||||
let mut broken = good.clone();
|
||||
broken[field] = wrong;
|
||||
|
|
@ -706,7 +865,8 @@ fn the_recent_ticker_keeps_the_newest_eight_events_newest_first() {
|
|||
#[test]
|
||||
fn the_adapter_reports_its_identity_and_pinned_rom() {
|
||||
let reward = PokemonRedReward::new();
|
||||
assert_eq!(reward.id(), "pokered-unique8-v5");
|
||||
assert_eq!(reward.id(), "pokered-unique8-v6");
|
||||
assert_eq!(reward.migrates_from(), ["pokered-unique8-v5"]);
|
||||
assert!(reward.rom_allowed(SUPPORTED_ROM));
|
||||
assert!(!reward.rom_allowed(
|
||||
"5ca7ba01642a3b27b0cc0b5349b52792795b62d3ed977e98a09390659af96b7b"
|
||||
|
|
@ -716,6 +876,9 @@ fn the_adapter_reports_its_identity_and_pinned_rom() {
|
|||
assert_eq!(symbols::ram::wNumberOfWarps, 0xd3ae);
|
||||
assert_eq!(symbols::ram::wWarpEntries, 0xd3af);
|
||||
assert_eq!(symbols::ram::wCurMapConnections, 0xd370);
|
||||
// Resolved from ram/wram.asm by services/flysim/tools/resolve_wram.py, bracketed by
|
||||
// wFontLoaded and wForcePlayerToChooseMon; never written out by hand.
|
||||
assert_eq!(symbols::ram::wCapturedMonSpecies, 0xd11c);
|
||||
assert_eq!(symbols::MILESTONES.len(), 17);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -30,6 +30,7 @@ pub mod metrics;
|
|||
pub mod pacing;
|
||||
pub mod profile;
|
||||
pub mod ratelimit;
|
||||
pub mod reset;
|
||||
pub mod sdnotify;
|
||||
pub mod simloop;
|
||||
pub mod snapshot;
|
||||
|
|
|
|||
|
|
@ -42,6 +42,15 @@ struct Args {
|
|||
/// the "nothing to compare" case for a fresh container.
|
||||
#[arg(long, value_name = "DIR")]
|
||||
print_state_compatibility: Option<std::path::PathBuf>,
|
||||
|
||||
/// Restart the run from the milestone archive for this ladder rung, and exit.
|
||||
///
|
||||
/// Run with flysim stopped: it rewrites both checkpoint stores.
|
||||
/// `infra/bin/fly-reset-to-milestone` is the operator-facing wrapper and the sequence
|
||||
/// around it is in `infra/docs/runbook.md`. The current state is copied to a dated
|
||||
/// directory first, so this is reversible by hand.
|
||||
#[arg(long, value_name = "RANK")]
|
||||
reset_to_milestone: Option<u32>,
|
||||
}
|
||||
|
||||
fn main() -> Result<()> {
|
||||
|
|
@ -66,6 +75,17 @@ fn main() -> Result<()> {
|
|||
return Ok(());
|
||||
}
|
||||
|
||||
if let Some(rank) = args.reset_to_milestone {
|
||||
let stamp = flysim::reset::utc_stamp(flysim::eventlog::now_wall_ms());
|
||||
let durable = config.paths.save_dir.clone();
|
||||
let hot = config.paths.hot_dir.clone();
|
||||
let archive = flysim::reset::default_archive_dir(&durable, &stamp);
|
||||
for line in flysim::reset::reset_to_milestone(&durable, &hot, rank, &archive)? {
|
||||
println!("{line}");
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
flysim::run(config)
|
||||
}
|
||||
|
||||
|
|
|
|||
450
services/flysim/crates/flysim/src/reset.rs
Normal file
450
services/flysim/crates/flysim/src/reset.rs
Normal file
|
|
@ -0,0 +1,450 @@
|
|||
//! Restarting a run from an earlier rung, on disk, with flysim stopped.
|
||||
//!
|
||||
//! The operator's decision of 2026-09-22 was "restart the live run from an early checkpoint
|
||||
//! instead of from scratch". `FLY_RESET_STATE=1` cannot do that: it archives everything and the
|
||||
//! next start warms up a fresh fly. What this does instead is promote one milestone archive --
|
||||
//! `milestone-<N>.checkpoint`, which `store::Store::commit` writes at the first commit at a new
|
||||
//! best rank and which no rotation ever unlinks -- to being the only thing either store will
|
||||
//! restore.
|
||||
//!
|
||||
//! `infra/bin/fly-reset-to-milestone` is the operator-facing wrapper; it refuses to run while
|
||||
//! flysim is up, calls `flysim --reset-to-milestone N`, and fixes ownership afterwards. The work
|
||||
//! is here rather than in that script because two of the steps are inside the `FLYSIM01`
|
||||
//! envelope: the ratchet's attempts and recoveries counters live in the checkpoint's manifest,
|
||||
//! and a shell script has no business rewriting one.
|
||||
//!
|
||||
//! What it does, in order, and nothing else:
|
||||
//!
|
||||
//! 1. **archives** every file in the durable and hot stores into a dated directory, by copying,
|
||||
//! so a step that fails later has destroyed nothing;
|
||||
//! 2. **rewrites** the rung's archive with the ratchet's `attempts` and `recoveries` at zero, so
|
||||
//! the recovery budget is not already spent when the restarted run begins. `best` is left
|
||||
//! alone: the archive's own `best` is the rung it was taken at, which is exactly what the
|
||||
//! restarted run is at, and the rank the stream shows is recomputed by the adapter from the
|
||||
//! restored game state anyway;
|
||||
//! 3. **installs** it as the newest generation in both stores, so the restore order
|
||||
//! (`store::restore_order`: hot latest, hot previous, durable latest, ...) reaches it first;
|
||||
//! 4. **clears** the milestone archives above N -- rungs the run had reached and is now below --
|
||||
//! and the generation files of the run being abandoned;
|
||||
//! 5. **clears the session ledgers**: the event log `events.jsonl` and its rotations. The
|
||||
//! checkpoint carries `lastEventId`, so restoring an old checkpoint over a newer log would
|
||||
//! re-issue ids the log already holds. The macro layer's own session ledgers (blocked,
|
||||
//! talked, reached, pushed-back) are memory-only by contract
|
||||
//! (`docs/design/macros.md` section 12.1: "a restored run offers every target once more"),
|
||||
//! so stopping flysim is what resets those and this has nothing to do.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::path::{Path, PathBuf};
|
||||
|
||||
use anyhow::{Context, Result, bail};
|
||||
|
||||
use crate::store::{self, Store, StoreManifest};
|
||||
|
||||
/// Generations kept by the stores this tool writes. Only used for the rotation bound, which
|
||||
/// this tool does not trigger; the durable store's own value.
|
||||
const KEEP_GENERATIONS: usize = 8;
|
||||
|
||||
/// A generation number no commit ever allocates, so `Store::candidates` drops the
|
||||
/// generation-file half of an archive entry and offers only `milestone-<rank>.checkpoint`.
|
||||
///
|
||||
/// `Sim::boot` allocates `highest_generation() + 1`, which is 1 or more, so 0 names no file.
|
||||
/// The milestone archives below the rung being restored are kept exactly this way: still on
|
||||
/// disk, still restorable as a deeper fallback, and with no generation file pretending to be
|
||||
/// their contents.
|
||||
const NO_GENERATION: u64 = 0;
|
||||
|
||||
/// What the reset did, one line per step, for the operator's terminal and the run record.
|
||||
pub type Report = Vec<String>;
|
||||
|
||||
/// Promote `rank`'s milestone archive to be the restore source of both stores.
|
||||
///
|
||||
/// `archive` is the dated directory the current state is copied into; it must not exist.
|
||||
/// Refuses if the milestone archive is missing, which is the "that rung was never reached"
|
||||
/// case and the one mistake worth refusing rather than guessing at.
|
||||
pub fn reset_to_milestone(
|
||||
durable_dir: &Path,
|
||||
hot_dir: &Path,
|
||||
rank: u32,
|
||||
archive: &Path,
|
||||
) -> Result<Report> {
|
||||
let durable = Store::new(durable_dir, KEEP_GENERATIONS);
|
||||
let hot = Store::new(hot_dir, KEEP_GENERATIONS);
|
||||
let source = durable.archive_path(rank);
|
||||
if !source.is_file() {
|
||||
bail!(
|
||||
"no milestone archive for rung {rank}: {} does not exist. `ls {}` shows the rungs \
|
||||
this run actually reached.",
|
||||
source.display(),
|
||||
durable_dir.display()
|
||||
);
|
||||
}
|
||||
if archive.exists() {
|
||||
bail!("the archive directory {} already exists", archive.display());
|
||||
}
|
||||
|
||||
let mut report: Report = Vec::new();
|
||||
|
||||
// 1. Copy everything aside first.
|
||||
let copied_durable = copy_tree(durable_dir, &archive.join("durable"))?;
|
||||
let copied_hot = copy_tree(hot_dir, &archive.join("hot"))?;
|
||||
report.push(format!(
|
||||
"archived {copied_durable} durable and {copied_hot} hot files to {}",
|
||||
archive.display()
|
||||
));
|
||||
|
||||
// 2. Zero the two recovery counters inside the envelope.
|
||||
let mut checkpoint = store::load(&source)
|
||||
.with_context(|| format!("decoding {}", source.display()))?;
|
||||
let spent = (checkpoint.runtime.ratchet.attempts, checkpoint.runtime.ratchet.recoveries);
|
||||
checkpoint.runtime.ratchet.attempts = 0;
|
||||
checkpoint.runtime.ratchet.recoveries = 0;
|
||||
|
||||
let generation = durable.highest_generation().max(hot.highest_generation()) + 1;
|
||||
checkpoint.runtime.generation = generation;
|
||||
let bytes = store::encode(&checkpoint.agent, &checkpoint.runtime)?;
|
||||
report.push(format!(
|
||||
"rung {rank} (best {}, ladder rank recomputed from the game state): ratchet attempts \
|
||||
{} -> 0, recoveries {} -> 0",
|
||||
checkpoint.runtime.ratchet.best, spent.0, spent.1
|
||||
));
|
||||
|
||||
// 3/4. Clear both stores, keeping the milestone archives at or below this rung, and write
|
||||
// the promoted state as the newest generation of each.
|
||||
let kept = clear_store(durable_dir, Some(rank))?;
|
||||
clear_store(hot_dir, None)?;
|
||||
report.push(format!(
|
||||
"cleared the hot store and every milestone archive above rung {rank}; kept {} at or \
|
||||
below it: {kept:?}",
|
||||
kept.len()
|
||||
));
|
||||
|
||||
// The hot store lives on a tmpfs that a stopped container may not have mounted yet, so it
|
||||
// is created rather than assumed; the durable one already exists or the milestone archive
|
||||
// above could not have been read.
|
||||
durable.create()?;
|
||||
hot.create()?;
|
||||
store::write_atomic(&durable.generation_path(generation), &bytes)?;
|
||||
store::write_atomic(&durable.archive_path(rank), &bytes)?;
|
||||
store::write_atomic(&hot.generation_path(generation), &bytes)?;
|
||||
|
||||
let mut archives: BTreeMap<u32, u64> =
|
||||
kept.iter().map(|rung| (*rung, NO_GENERATION)).collect();
|
||||
archives.insert(rank, generation);
|
||||
let durable_manifest = StoreManifest {
|
||||
generation,
|
||||
latest: Some(generation),
|
||||
previous: None,
|
||||
archives,
|
||||
};
|
||||
write_manifest(&durable, &durable_manifest)?;
|
||||
write_manifest(
|
||||
&hot,
|
||||
&StoreManifest {
|
||||
generation,
|
||||
latest: Some(generation),
|
||||
previous: None,
|
||||
archives: BTreeMap::new(),
|
||||
},
|
||||
)?;
|
||||
report.push(format!(
|
||||
"generation {generation} is now hot latest and durable latest in {} and {}",
|
||||
hot_dir.display(),
|
||||
durable_dir.display()
|
||||
));
|
||||
|
||||
// 5. The session ledgers.
|
||||
let logs = remove_matching(durable_dir, |name| {
|
||||
name == "events.jsonl" || (name.starts_with("events-") && name.ends_with(".jsonl"))
|
||||
})?;
|
||||
report.push(format!(
|
||||
"reset the session ledgers: {logs} event-log files removed (the macro layer's are \
|
||||
memory-only and are reset by stopping flysim)"
|
||||
));
|
||||
|
||||
Ok(report)
|
||||
}
|
||||
|
||||
fn write_manifest(store: &Store, manifest: &StoreManifest) -> Result<()> {
|
||||
store.create()?;
|
||||
store::write_atomic(&store.manifest_path(), &serde_json::to_vec_pretty(manifest)?)
|
||||
}
|
||||
|
||||
/// Copy every regular file of `from` into `to`, creating `to`. A missing source is zero files,
|
||||
/// not an error: the hot store lives on a tmpfs that a stopped container does not have.
|
||||
fn copy_tree(from: &Path, to: &Path) -> Result<usize> {
|
||||
if !from.is_dir() {
|
||||
return Ok(0);
|
||||
}
|
||||
std::fs::create_dir_all(to)
|
||||
.with_context(|| format!("creating {}", to.display()))?;
|
||||
let mut copied = 0;
|
||||
for entry in std::fs::read_dir(from)?.flatten() {
|
||||
if !entry.file_type().is_ok_and(|kind| kind.is_file()) {
|
||||
continue;
|
||||
}
|
||||
std::fs::copy(entry.path(), to.join(entry.file_name()))
|
||||
.with_context(|| format!("copying {}", entry.path().display()))?;
|
||||
copied += 1;
|
||||
}
|
||||
Ok(copied)
|
||||
}
|
||||
|
||||
/// Remove every checkpoint, tmp file and manifest from `dir`, keeping `milestone-<r>.checkpoint`
|
||||
/// for `r <= keep_up_to`. Returns the rungs kept, ascending.
|
||||
fn clear_store(dir: &Path, keep_up_to: Option<u32>) -> Result<Vec<u32>> {
|
||||
let mut kept = Vec::new();
|
||||
if !dir.is_dir() {
|
||||
return Ok(kept);
|
||||
}
|
||||
for entry in std::fs::read_dir(dir)?.flatten() {
|
||||
let name = entry.file_name().to_string_lossy().to_string();
|
||||
let milestone = name
|
||||
.strip_prefix("milestone-")
|
||||
.and_then(|rest| rest.strip_suffix(".checkpoint"))
|
||||
.and_then(|rung| rung.parse::<u32>().ok());
|
||||
let remove = match milestone {
|
||||
// The promoted rung's own archive is rewritten straight after this, so it is
|
||||
// removed here like the rest and reinstated with the counters cleared.
|
||||
Some(rung) => match keep_up_to {
|
||||
Some(limit) if rung < limit => {
|
||||
kept.push(rung);
|
||||
false
|
||||
}
|
||||
_ => true,
|
||||
},
|
||||
None => {
|
||||
name == "manifest.json"
|
||||
|| name.ends_with(".checkpoint")
|
||||
|| name.ends_with(".checkpoint.tmp")
|
||||
}
|
||||
};
|
||||
if remove {
|
||||
std::fs::remove_file(entry.path())
|
||||
.with_context(|| format!("removing {}", entry.path().display()))?;
|
||||
}
|
||||
}
|
||||
kept.sort_unstable();
|
||||
Ok(kept)
|
||||
}
|
||||
|
||||
fn remove_matching(dir: &Path, wanted: impl Fn(&str) -> bool) -> Result<usize> {
|
||||
if !dir.is_dir() {
|
||||
return Ok(0);
|
||||
}
|
||||
let mut removed = 0;
|
||||
for entry in std::fs::read_dir(dir)?.flatten() {
|
||||
if wanted(&entry.file_name().to_string_lossy()) {
|
||||
std::fs::remove_file(entry.path())?;
|
||||
removed += 1;
|
||||
}
|
||||
}
|
||||
Ok(removed)
|
||||
}
|
||||
|
||||
/// `YYYYMMDDTHHMMSSZ` in UTC, for the dated archive directory's name.
|
||||
///
|
||||
/// Built on the event log's own calendar conversion, which is the service's only one; the time of
|
||||
/// day is arithmetic on the same millisecond count.
|
||||
pub fn utc_stamp(wall_ms: u64) -> String {
|
||||
let second_of_day = (wall_ms % 86_400_000) / 1_000;
|
||||
format!(
|
||||
"{}T{:02}{:02}{:02}Z",
|
||||
crate::eventlog::utc_day(wall_ms),
|
||||
second_of_day / 3_600,
|
||||
(second_of_day / 60) % 60,
|
||||
second_of_day % 60,
|
||||
)
|
||||
}
|
||||
|
||||
/// The default dated archive directory: a sibling of the durable store, which is its own
|
||||
/// mountpoint and so cannot be renamed -- the same shape `infra/05-deploy.sh` uses for
|
||||
/// `FLY_RESET_STATE=1`.
|
||||
pub fn default_archive_dir(durable_dir: &Path, stamp: &str) -> PathBuf {
|
||||
let mut name = durable_dir.as_os_str().to_os_string();
|
||||
name.push(format!(".reset-{stamp}"));
|
||||
PathBuf::from(name)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// A real `FLYSIM01` envelope, small but structurally complete, so these tests decode and
|
||||
/// re-encode what the running service writes rather than a stand-in.
|
||||
fn envelope(generation: u64, ratchet: flybrain_gb::RatchetState) -> Vec<u8> {
|
||||
use flybrain_core::decoder::DecoderState;
|
||||
use flybrain_core::lif::LifState;
|
||||
use flybrain_core::ordered::NumberMap;
|
||||
use flybrain_core::plasticity::PlasticityState;
|
||||
|
||||
let agent = flybrain_core::agent::AgentState {
|
||||
version: 1,
|
||||
remainder: 0.75,
|
||||
warmed_up: true,
|
||||
network: LifState {
|
||||
membrane: vec![0.5, -0.25],
|
||||
refractory: vec![0, 3],
|
||||
last_spike_ms: vec![-1_000_000.0, 12.0],
|
||||
visual_drive: vec![0.1],
|
||||
rng: -12_345,
|
||||
reward_remaining: 40.0,
|
||||
ms: 9_000.0,
|
||||
population_rate: 1.5,
|
||||
rates: NumberMap::from_pairs([("forward", 2.0)]),
|
||||
plasticity: PlasticityState {
|
||||
version: "fly-kc-mbon-rstdp-v2".to_string(),
|
||||
topology: 42,
|
||||
enabled: true,
|
||||
updates: 3.0,
|
||||
signal: 0.25,
|
||||
gains: vec![1.0, 0.9],
|
||||
traces: vec![0.0, 0.1],
|
||||
touched: vec![0.0, 8_000.0],
|
||||
},
|
||||
},
|
||||
decoder: DecoderState {
|
||||
version: 4,
|
||||
calibrated: true,
|
||||
baseline: NumberMap::from_pairs([("forward", 1.0)]),
|
||||
held_until: NumberMap::new(),
|
||||
next_allowed: NumberMap::new(),
|
||||
next_decision: 100.0,
|
||||
current: None,
|
||||
fatigue: NumberMap::new(),
|
||||
macro_next_decision: 0.0,
|
||||
macro_current: None,
|
||||
macro_fatigue: NumberMap::new(),
|
||||
},
|
||||
};
|
||||
let runtime = store::RuntimeState {
|
||||
generation,
|
||||
wall_ms: 1_700_000_000_000,
|
||||
rom_sha256: "ab".repeat(32),
|
||||
emulator_frame: 12_345,
|
||||
compatibility: "kernel/pokered-unique8-v6/fingerprint".to_string(),
|
||||
speed: 1.0,
|
||||
buttons: 0,
|
||||
rank_since_ms: 4_242.0,
|
||||
last_event_id: 77,
|
||||
reward: serde_json::json!({ "version": 4, "total": 1.25 }),
|
||||
ratchet,
|
||||
emulator: vec![7; 64],
|
||||
framebuffer: vec![9; 32],
|
||||
ratchet_game: vec![1, 2, 3],
|
||||
ratchet_frame: vec![4, 5, 6],
|
||||
};
|
||||
store::encode(&agent, &runtime).unwrap()
|
||||
}
|
||||
|
||||
/// A store dir holding a milestone archive for each rung in `rungs`, their generations, a
|
||||
/// manifest and an event log, all written through the store's own commit path.
|
||||
fn state_dir(root: &Path, rungs: &[u32], ratchet: flybrain_gb::RatchetState) -> Store {
|
||||
let store = Store::new(root, KEEP_GENERATIONS);
|
||||
store.create().unwrap();
|
||||
for (index, rung) in rungs.iter().enumerate() {
|
||||
let generation = index as u64 + 1;
|
||||
store.commit(generation, &envelope(generation, ratchet), Some(*rung)).unwrap();
|
||||
}
|
||||
std::fs::write(root.join("events.jsonl"), b"{}\n").unwrap();
|
||||
std::fs::write(root.join("events-20260921.jsonl"), b"{}\n").unwrap();
|
||||
store
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_missing_rung_is_refused_and_nothing_is_touched() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let durable = tmp.path().join("state");
|
||||
let hot = tmp.path().join("hot");
|
||||
state_dir(&durable, &[3, 5], flybrain_gb::RatchetState::default());
|
||||
let before = std::fs::read_dir(&durable).unwrap().flatten().count();
|
||||
|
||||
let error = reset_to_milestone(&durable, &hot, 9, &tmp.path().join("archive"))
|
||||
.unwrap_err()
|
||||
.to_string();
|
||||
assert!(error.contains("no milestone archive for rung 9"), "{error}");
|
||||
assert_eq!(std::fs::read_dir(&durable).unwrap().flatten().count(), before);
|
||||
assert!(!tmp.path().join("archive").exists(), "nothing was archived");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_rung_becomes_both_stores_latest_with_the_recovery_budget_back() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let durable = tmp.path().join("state");
|
||||
let hot = tmp.path().join("hot");
|
||||
let spent = flybrain_gb::RatchetState {
|
||||
best: 5,
|
||||
attempts: 3,
|
||||
recoveries: 11,
|
||||
..flybrain_gb::RatchetState::default()
|
||||
};
|
||||
state_dir(&durable, &[3, 5, 9, 11], spent);
|
||||
state_dir(&hot, &[11], spent);
|
||||
let archive = tmp.path().join("archive-20260922");
|
||||
|
||||
let report = reset_to_milestone(&durable, &hot, 5, &archive).unwrap();
|
||||
assert!(report.iter().any(|line| line.contains("attempts 3 -> 0")), "{report:?}");
|
||||
|
||||
// Everything that was there is in the archive.
|
||||
assert!(archive.join("durable/milestone-11.checkpoint").is_file());
|
||||
assert!(archive.join("durable/events.jsonl").is_file());
|
||||
assert!(archive.join("hot/manifest.json").is_file());
|
||||
|
||||
// The rungs above 5 are gone; the ones below it stay as deeper fallbacks.
|
||||
assert!(!durable.join("milestone-9.checkpoint").exists());
|
||||
assert!(!durable.join("milestone-11.checkpoint").exists());
|
||||
assert!(durable.join("milestone-3.checkpoint").is_file());
|
||||
assert!(durable.join("milestone-5.checkpoint").is_file());
|
||||
assert!(!durable.join("events.jsonl").exists());
|
||||
assert!(!durable.join("events-20260921.jsonl").exists());
|
||||
assert!(!hot.join("milestone-11.checkpoint").exists());
|
||||
|
||||
// Both stores restore the rung, and the counters are back.
|
||||
for store in [Store::new(&hot, KEEP_GENERATIONS), Store::new(&durable, KEEP_GENERATIONS)] {
|
||||
let candidates = store.candidates("x");
|
||||
let first = store::load(&candidates[0].path).unwrap();
|
||||
assert_eq!(first.runtime.ratchet.best, 5);
|
||||
assert_eq!((first.runtime.ratchet.attempts, first.runtime.ratchet.recoveries), (0, 0));
|
||||
}
|
||||
// ... including through the promoted milestone archive itself.
|
||||
let archived = store::load(&durable.join("milestone-5.checkpoint")).unwrap();
|
||||
assert_eq!((archived.runtime.ratchet.attempts, archived.runtime.ratchet.recoveries), (0, 0));
|
||||
|
||||
// The rungs below it are offered, and only as their own archive files.
|
||||
let manifest = Store::new(&durable, KEEP_GENERATIONS).manifest().unwrap();
|
||||
assert_eq!(manifest.archives.get(&3), Some(&NO_GENERATION));
|
||||
assert_eq!(manifest.latest, manifest.archives.get(&5).copied());
|
||||
assert!(
|
||||
Store::new(&durable, KEEP_GENERATIONS)
|
||||
.candidates("durable")
|
||||
.iter()
|
||||
.any(|candidate| candidate.path.ends_with("milestone-3.checkpoint"))
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_second_reset_refuses_to_write_over_an_existing_archive() {
|
||||
let tmp = tempfile::tempdir().unwrap();
|
||||
let durable = tmp.path().join("state");
|
||||
let hot = tmp.path().join("hot");
|
||||
state_dir(&durable, &[4], flybrain_gb::RatchetState::default());
|
||||
let archive = tmp.path().join("archive");
|
||||
reset_to_milestone(&durable, &hot, 4, &archive).unwrap();
|
||||
let error = reset_to_milestone(&durable, &hot, 4, &archive).unwrap_err().to_string();
|
||||
assert!(error.contains("already exists"), "{error}");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_stamp_is_the_event_logs_calendar_plus_a_time_of_day() {
|
||||
assert_eq!(utc_stamp(0), "19700101T000000Z");
|
||||
// 2026-09-22T16:15:00Z
|
||||
assert_eq!(utc_stamp(1_790_093_700_000), "20260922T161500Z");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_default_archive_is_a_dated_sibling_of_the_store() {
|
||||
assert_eq!(
|
||||
default_archive_dir(Path::new("/srv/fly/state"), "20260922T161500Z"),
|
||||
PathBuf::from("/srv/fly/state.reset-20260922T161500Z")
|
||||
);
|
||||
}
|
||||
}
|
||||
|
|
@ -718,12 +718,36 @@ impl Sim {
|
|||
if runtime.rom_sha256 != self.rom_sha256 {
|
||||
bail!("checkpoint is for another cartridge ({})", runtime.rom_sha256);
|
||||
}
|
||||
if runtime.compatibility != self.compatibility {
|
||||
bail!(
|
||||
"compatibility mismatch\n checkpoint: {}\n this build: {}",
|
||||
// Byte-identical, or the one documented migration the operator asked for.
|
||||
//
|
||||
// `FLY_ACCEPT_ADAPTERS` is read here rather than carried in `Config` because it is a
|
||||
// property of a *deploy*, not of a run: `infra/05-deploy.sh` writes it into
|
||||
// `/etc/fly/fly.env` only for the deploy that needs it, and an operator who wants the
|
||||
// migration off again deletes one line. An empty or unset variable is no migration at
|
||||
// all, which is what every deploy before this one did.
|
||||
let accepted = flybrain_gb::compatibility::accepted_adapters(
|
||||
std::env::var(flybrain_gb::compatibility::ACCEPT_ADAPTERS_ENV).ok().as_deref(),
|
||||
);
|
||||
match flybrain_gb::compatibility::decide(
|
||||
&runtime.compatibility,
|
||||
&self.compatibility,
|
||||
self.adapter.migrates_from(),
|
||||
&accepted,
|
||||
) {
|
||||
flybrain_gb::compatibility::RestoreDecision::Exact => {}
|
||||
flybrain_gb::compatibility::RestoreDecision::MigrateAdapter { from } => {
|
||||
tracing::warn!(
|
||||
from = %from,
|
||||
to = %self.adapter.id(),
|
||||
"restoring a checkpoint from an earlier adapter, by the migration \
|
||||
FLY_ACCEPT_ADAPTERS opted this deploy into"
|
||||
);
|
||||
}
|
||||
flybrain_gb::compatibility::RestoreDecision::Refuse(reason) => bail!(
|
||||
"compatibility mismatch: {reason}\n checkpoint: {}\n this build: {}",
|
||||
runtime.compatibility,
|
||||
self.compatibility
|
||||
);
|
||||
),
|
||||
}
|
||||
if runtime.framebuffer.len() != FRAMEBUFFER_LEN {
|
||||
bail!("checkpoint framebuffer is {} bytes", runtime.framebuffer.len());
|
||||
|
|
|
|||
|
|
@ -241,8 +241,8 @@ pub struct FeedMacroOutcome {
|
|||
}
|
||||
|
||||
/// Reward categories the feed reports counts for. The adapter's own interned kinds
|
||||
/// (`milestone`, `exploration`, `map`, `species`, `trainer`, `battle`, `badge`, `boundary`) map
|
||||
/// onto these.
|
||||
/// (`milestone`, `exploration`, `map`, `species`, `trainer`, `battle`, `badge`, `boundary`,
|
||||
/// `catch`) map onto these.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
pub enum RewardKind {
|
||||
|
|
@ -282,6 +282,15 @@ impl RewardKind {
|
|||
// protocol is concerned: finding a door is finding somewhere new, and the design asks
|
||||
// for no new feed kind.
|
||||
"boundary" => Self::Explore,
|
||||
// `catch` is a wild battle the fly won by keeping the Pokémon, so it publishes on
|
||||
// the same counter a wild KO does. The feed's kinds are a closed set
|
||||
// (`docs/feed-protocol.md`) and this rule asked for no new one.
|
||||
//
|
||||
// Deliberately *not* `pokedex`: on a catch of a species this run has never owned,
|
||||
// the cartridge sets the Pokédex bit and the adapter's existing `species` rule pays
|
||||
// for it on the same frame, so the `pokedex` counter already moves. Mapping `catch`
|
||||
// there as well would count one event twice.
|
||||
"catch" => Self::Wildwin,
|
||||
// The platformer.
|
||||
"band" => Self::Explore,
|
||||
"coin" => Self::Wildwin,
|
||||
|
|
@ -739,6 +748,7 @@ mod tests {
|
|||
}
|
||||
assert_eq!(RewardKind::from_adapter("nonsense"), None);
|
||||
assert_eq!(RewardKind::from_adapter("boundary"), Some(RewardKind::Explore));
|
||||
assert_eq!(RewardKind::from_adapter("catch"), Some(RewardKind::Wildwin));
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
|
|
|||
224
services/flysim/crates/flysim/tests/compat_migration.rs
Normal file
224
services/flysim/crates/flysim/tests/compat_migration.rs
Normal file
|
|
@ -0,0 +1,224 @@
|
|||
//! A `v5` checkpoint restored under `v6`: accepted with the opt-in, refused without it.
|
||||
//!
|
||||
//! The unit tests in `flybrain-gb` cover the decision function and the adapter's own state
|
||||
//! migration separately. This is the two of them against one artefact: a real `FLYSIM01`
|
||||
//! envelope carrying a `pokered-unique8-v5` compatibility string and a `v5` reward ledger —
|
||||
//! written, encoded, decoded, and then put through exactly what `Sim::try_restore` puts a
|
||||
//! candidate through.
|
||||
//!
|
||||
//! No ROM and no dataset, deliberately. Building a `Sim` would need both, and neither is part of
|
||||
//! the question: what decides a restore is the compatibility string and `import_state`.
|
||||
|
||||
use flybrain_gb::GameAdapter;
|
||||
use flybrain_gb::compatibility::{RestoreDecision, accepted_adapters, decide};
|
||||
use flybrain_gb::pokemon_red::PokemonRedReward;
|
||||
use flysim::store::{self, RuntimeState};
|
||||
|
||||
/// The live string's shape, with the adapter left open. The dataset fingerprint is shortened —
|
||||
/// nothing here parses it, and a seven-digest one would be 455 characters of noise.
|
||||
fn compatibility(adapter: &str) -> String {
|
||||
format!(
|
||||
"lif-1ms-f64-v2/{adapter}/aabbccddeeff00/fly-kc-mbon-rstdp-v2/\
|
||||
binjgb:c60e138da5a795ebb55e56b11b7e90024e41112c/\
|
||||
pokered:0cd19d3b877b7dc66d12c7050bed9a7f38154d4b/statefmt:199616-x86_64-unknown-linux-gnu"
|
||||
)
|
||||
}
|
||||
|
||||
/// A `v5` reward ledger: `STATE_VERSION` 4, every field `v5` wrote, and **no** `catchCounts`.
|
||||
///
|
||||
/// Written out by hand rather than exported from an adapter, because an exported one would be a
|
||||
/// `v6` state with the counter deleted — this is the shape the release box's checkpoints really
|
||||
/// carry, field for field.
|
||||
fn v5_reward() -> serde_json::Value {
|
||||
serde_json::json!({
|
||||
"version": 4,
|
||||
"seen": ["adventure", "map:0", "early:outside", "dex:3", "boundary:0:edge:n:near"],
|
||||
"tiles": ["0:5:6", "0:5:7"],
|
||||
"tileCounts": { "0": 2 },
|
||||
"wildWins": { "0:112:4": 2 },
|
||||
"replayBlocked": [],
|
||||
"counts": {
|
||||
"milestone": 2, "exploration": 0, "map": 1, "species": 1,
|
||||
"trainer": 0, "battle": 2, "badge": 0, "boundary": 1
|
||||
},
|
||||
"total": 2.05,
|
||||
"recent": [{ "kind": "species", "label": "OWNED #4", "brainMs": 1234.5, "value": 0.5 }],
|
||||
"last": { "species": { "kind": "species", "label": "OWNED #4", "brainMs": 1234.5, "value": 0.5 } },
|
||||
"initialized": true,
|
||||
"sawBoot": true,
|
||||
"location": "0:5:7",
|
||||
"stable": 9,
|
||||
"progress": 3,
|
||||
"badges": 0,
|
||||
"battle": null,
|
||||
"mode": "OVERWORLD"
|
||||
})
|
||||
}
|
||||
|
||||
fn v5_checkpoint() -> Vec<u8> {
|
||||
use flybrain_core::decoder::DecoderState;
|
||||
use flybrain_core::lif::LifState;
|
||||
use flybrain_core::ordered::NumberMap;
|
||||
use flybrain_core::plasticity::PlasticityState;
|
||||
|
||||
let agent = flybrain_core::agent::AgentState {
|
||||
version: 1,
|
||||
remainder: 0.25,
|
||||
warmed_up: true,
|
||||
network: LifState {
|
||||
membrane: vec![0.1, -0.2],
|
||||
refractory: vec![0, 1],
|
||||
last_spike_ms: vec![-1_000_000.0, 5.0],
|
||||
visual_drive: vec![0.3],
|
||||
rng: 42,
|
||||
reward_remaining: 0.0,
|
||||
ms: 1_234.5,
|
||||
population_rate: 1.0,
|
||||
rates: NumberMap::from_pairs([("forward", 1.0)]),
|
||||
plasticity: PlasticityState {
|
||||
version: "fly-kc-mbon-rstdp-v2".to_string(),
|
||||
topology: 7,
|
||||
enabled: true,
|
||||
updates: 1.0,
|
||||
signal: 0.0,
|
||||
gains: vec![1.0],
|
||||
traces: vec![0.0],
|
||||
touched: vec![0.0],
|
||||
},
|
||||
},
|
||||
decoder: DecoderState {
|
||||
version: 4,
|
||||
calibrated: true,
|
||||
baseline: NumberMap::from_pairs([("forward", 1.0)]),
|
||||
held_until: NumberMap::new(),
|
||||
next_allowed: NumberMap::new(),
|
||||
next_decision: 0.0,
|
||||
current: None,
|
||||
fatigue: NumberMap::new(),
|
||||
macro_next_decision: 0.0,
|
||||
macro_current: None,
|
||||
macro_fatigue: NumberMap::new(),
|
||||
},
|
||||
};
|
||||
let runtime = RuntimeState {
|
||||
generation: 41,
|
||||
wall_ms: 1_790_000_000_000,
|
||||
rom_sha256: flybrain_gb::pokemon_red::SUPPORTED_ROM.to_string(),
|
||||
emulator_frame: 1_000_000,
|
||||
compatibility: compatibility("pokered-unique8-v5"),
|
||||
speed: 1.0,
|
||||
buttons: 0,
|
||||
rank_since_ms: 1_000.0,
|
||||
last_event_id: 4_242,
|
||||
reward: v5_reward(),
|
||||
ratchet: flybrain_gb::RatchetState { best: 3, attempts: 1, recoveries: 4, ..Default::default() },
|
||||
emulator: vec![3; 64],
|
||||
framebuffer: vec![0; 32],
|
||||
ratchet_game: vec![1],
|
||||
ratchet_frame: vec![2],
|
||||
};
|
||||
store::encode(&agent, &runtime).expect("the fixture encodes")
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_v5_checkpoint_is_refused_under_v6_without_the_opt_in() {
|
||||
let checkpoint = store::decode(&v5_checkpoint()).expect("the fixture decodes");
|
||||
let adapter = PokemonRedReward::new();
|
||||
let current = compatibility(adapter.id());
|
||||
assert_ne!(checkpoint.runtime.compatibility, current, "v6 is not v5");
|
||||
|
||||
for opt_in in [None, Some(""), Some("pokered-unique8-v4"), Some("some-other-adapter")] {
|
||||
assert!(
|
||||
matches!(
|
||||
decide(
|
||||
&checkpoint.runtime.compatibility,
|
||||
¤t,
|
||||
adapter.migrates_from(),
|
||||
&accepted_adapters(opt_in),
|
||||
),
|
||||
RestoreDecision::Refuse(_)
|
||||
),
|
||||
"FLY_ACCEPT_ADAPTERS={opt_in:?} must not migrate anything"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_v5_checkpoint_restores_under_v6_with_the_opt_in_and_the_counter_starts_at_zero() {
|
||||
let checkpoint = store::decode(&v5_checkpoint()).expect("the fixture decodes");
|
||||
let mut adapter = PokemonRedReward::new();
|
||||
let current = compatibility(adapter.id());
|
||||
|
||||
assert_eq!(
|
||||
decide(
|
||||
&checkpoint.runtime.compatibility,
|
||||
¤t,
|
||||
adapter.migrates_from(),
|
||||
&accepted_adapters(Some("pokered-unique8-v5")),
|
||||
),
|
||||
RestoreDecision::MigrateAdapter { from: "pokered-unique8-v5".to_string() }
|
||||
);
|
||||
|
||||
// The migration itself: `import_state`, exactly as `Sim::try_restore` calls it.
|
||||
adapter.import_state(&checkpoint.runtime.reward).expect("a v5 ledger is a valid v6 ledger");
|
||||
|
||||
let after = adapter.export_state();
|
||||
assert_eq!(after["catchCounts"], serde_json::json!({}), "the new counter starts at 0");
|
||||
assert_eq!(after["counts"]["catch"], serde_json::json!(0));
|
||||
|
||||
// And nothing else moved: every field the v5 state carried round-trips to the same value,
|
||||
// and the only key v6 adds is the counter.
|
||||
//
|
||||
// `counts` is the one field that is not byte-identical, and it is not a change of meaning:
|
||||
// it serializes every kind in the catalog, so a v6 state lists `catch` where a v5 state had
|
||||
// nothing to list. Every kind the v5 state did carry keeps its number.
|
||||
let before = v5_reward();
|
||||
for (key, value) in before.as_object().unwrap() {
|
||||
if key == "counts" {
|
||||
for (kind, count) in value.as_object().unwrap() {
|
||||
assert_eq!(&after["counts"][kind], count, "counts.{kind}");
|
||||
}
|
||||
let added: Vec<&String> = after["counts"]
|
||||
.as_object()
|
||||
.unwrap()
|
||||
.keys()
|
||||
.filter(|kind| !value.as_object().unwrap().contains_key(*kind))
|
||||
.collect();
|
||||
assert_eq!(added, vec!["catch"], "v6 counts one more kind and no others");
|
||||
continue;
|
||||
}
|
||||
assert_eq!(&after[key], value, "{key} must survive the migration byte for byte");
|
||||
}
|
||||
let added: Vec<&String> = after
|
||||
.as_object()
|
||||
.unwrap()
|
||||
.keys()
|
||||
.filter(|key| !before.as_object().unwrap().contains_key(*key))
|
||||
.collect();
|
||||
assert_eq!(added, vec!["catchCounts"], "v6 adds one field and no others");
|
||||
|
||||
// The rest of what a restore reads is untouched by the migration.
|
||||
assert_eq!(adapter.progress().rank, 3);
|
||||
assert_eq!(checkpoint.runtime.last_event_id, 4_242);
|
||||
assert_eq!(checkpoint.runtime.ratchet.best, 3);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn nothing_but_the_adapter_segment_may_differ_for_the_migration_to_apply() {
|
||||
let adapter = PokemonRedReward::new();
|
||||
let accepted = accepted_adapters(Some("pokered-unique8-v5"));
|
||||
let current = compatibility(adapter.id());
|
||||
|
||||
// A v5 string whose state format also moved: a different build, not a rule change.
|
||||
let other_abi = compatibility("pokered-unique8-v5").replace("199616", "199617");
|
||||
assert!(matches!(
|
||||
decide(&other_abi, ¤t, adapter.migrates_from(), &accepted),
|
||||
RestoreDecision::Refuse(_)
|
||||
));
|
||||
|
||||
// And an identical string needs no opt-in at all.
|
||||
assert_eq!(
|
||||
decide(¤t, ¤t, adapter.migrates_from(), &[]),
|
||||
RestoreDecision::Exact
|
||||
);
|
||||
}
|
||||
257
services/flysim/crates/flysim/tests/rom_catch.rs
Normal file
257
services/flysim/crates/flysim/tests/rom_catch.rs
Normal file
|
|
@ -0,0 +1,257 @@
|
|||
//! The catch reward against the real cartridge.
|
||||
//!
|
||||
//! Gated on `FLY_ROM` *and* on a checkpoint, the way every ROM test in this workspace is, and
|
||||
//! skips cleanly without either — the cartridge never enters this repository and a checkpoint is
|
||||
//! not a fixture, it is the state the release box was really in:
|
||||
//!
|
||||
//! ```sh
|
||||
//! FLY_ROM="$HOME/roms/pokemon-red.gb" \
|
||||
//! FLY_CATCH_CHECKPOINT=.local/checkpoints/<a rung-9 forest checkpoint> \
|
||||
//! cargo test --release -p flysim --test rom_catch -- --nocapture
|
||||
//! ```
|
||||
//!
|
||||
//! ## What only the cartridge can answer
|
||||
//!
|
||||
//! The synthetic trace in `pokemon_red/tests.rs` writes `wCapturedMonSpecies`, `wBattleResult`
|
||||
//! and the Pokédex bit itself, from the disassembly. It cannot say that those are the bytes
|
||||
//! *this* cartridge writes when a ball keeps a Pokémon, in that order, on frames an adapter
|
||||
//! sampling once a frame actually sees. That is this test, and it is the "survey" half of
|
||||
//! `docs/design/macros-wram.md`'s evidence for the row: a real battle, real button presses, and
|
||||
//! the byte read out of the running game rather than written into a fake one.
|
||||
//!
|
||||
//! ## How the catch is produced
|
||||
//!
|
||||
//! No steering and no scripted button sequence: the shipping macro palette, the shipping macro
|
||||
//! layer and the shipping decoder, with a stub readout that leans on one macro population at a
|
||||
//! time — the same driver `tests/rom_macros_mode.rs` uses and for the same reason. The one thing
|
||||
//! this harness does that the rotation does not is lean on `THROW BALL`'s channel while a wild
|
||||
//! battle is up, because the question here is what the adapter reads from a catch, not whether a
|
||||
//! game-blind readout finds its way to one.
|
||||
//!
|
||||
//! The checkpoint must hold at least one ball in the bag. The macro palette can buy one
|
||||
//! (`BUY BALL`, `MB·PBALL`, inside a mart), but that is a walk across a city and back and it is a
|
||||
//! different test's question; this one says out loud that it skipped.
|
||||
|
||||
use flybrain_core::decoder::PopulationDecoder;
|
||||
use flybrain_core::decoder::gameboy::gameboy_decoder_config_with_macros;
|
||||
use flybrain_core::ordered::NumberMap;
|
||||
use flybrain_gb::adapter::RewardEvent;
|
||||
use flybrain_gb::pokemon_red::state;
|
||||
use flybrain_gb::pokemon_red::symbols::ram;
|
||||
use flybrain_gb::pokemon_red::{PokemonRedReward, catalog};
|
||||
use flybrain_gb::{
|
||||
AdapterLedger, DEFAULT_AUDIO_FRAMES, DEFAULT_AUDIO_FREQUENCY, Emulator, GameAdapter,
|
||||
};
|
||||
use flysim::config::Config;
|
||||
use flysim::macros::{MacroLayer, macro_layer};
|
||||
use flysim::snapshot::MacroMode;
|
||||
|
||||
const MS_PER_FRAME: f64 = 1000.0 / 59.7275;
|
||||
const SEED: u32 = 20_260_922;
|
||||
/// The hot population's rate against every other one's, which is also the stub's calibration
|
||||
/// rate — so a channel that is not the hot one scores exactly 1.0.
|
||||
const HOT: f64 = 16.0;
|
||||
const REST: f64 = 10.0;
|
||||
/// `THROW BALL`'s channel (`pokemon_red::macros::palette`).
|
||||
const BALL: &str = "MB·BALL";
|
||||
/// Frames the stub leans on one channel before the rotation moves on, the shape of the real
|
||||
/// group's hysteresis-then-fatigue rotation.
|
||||
const BURST_FRAMES: u32 = 24;
|
||||
|
||||
fn rates(hot: Option<&str>) -> NumberMap {
|
||||
let mut rates = NumberMap::new();
|
||||
for channel in flybrain_gb::macro_channels("pokemon-red") {
|
||||
rates.set(channel, REST);
|
||||
}
|
||||
for bucket in 0..8 {
|
||||
rates.set(&format!("command_{bucket}"), REST);
|
||||
}
|
||||
if let Some(channel) = hot {
|
||||
rates.set(channel, HOT);
|
||||
}
|
||||
rates
|
||||
}
|
||||
|
||||
fn rom() -> Option<Vec<u8>> {
|
||||
let path = std::env::var_os("FLY_ROM")?;
|
||||
match std::fs::read(&path) {
|
||||
Ok(bytes) => Some(bytes),
|
||||
Err(error) => panic!("FLY_ROM is set to {path:?} but could not be read: {error}"),
|
||||
}
|
||||
}
|
||||
|
||||
fn checkpoint() -> Option<flysim::store::Checkpoint> {
|
||||
let path = std::env::var_os("FLY_CATCH_CHECKPOINT")?;
|
||||
Some(
|
||||
flysim::store::load(std::path::Path::new(&path))
|
||||
.expect("the checkpoint should be a FLYSIM01 envelope"),
|
||||
)
|
||||
}
|
||||
|
||||
struct Run {
|
||||
gb: Emulator,
|
||||
adapter: PokemonRedReward,
|
||||
layer: MacroLayer,
|
||||
decoder: PopulationDecoder,
|
||||
channels: Vec<&'static str>,
|
||||
ms: f64,
|
||||
frame: u32,
|
||||
/// Every payout the adapter has made since the run started.
|
||||
payouts: Vec<RewardEvent>,
|
||||
}
|
||||
|
||||
impl Run {
|
||||
fn resume(rom: &[u8], checkpoint: &flysim::store::Checkpoint) -> Self {
|
||||
let mut gb = Emulator::new(rom, DEFAULT_AUDIO_FREQUENCY, DEFAULT_AUDIO_FRAMES)
|
||||
.expect("binjgb should accept the cartridge");
|
||||
let mut adapter = PokemonRedReward::new();
|
||||
gb.import_state(&checkpoint.runtime.emulator).expect("the checkpoint's emulator state");
|
||||
// A checkpoint written by an earlier adapter rebaselines rather than failing, which is
|
||||
// exactly the `v5` -> `v6` case this rule ships with.
|
||||
adapter.import_state(&checkpoint.runtime.reward).expect("the checkpoint's reward ledger");
|
||||
let channels = flybrain_gb::macro_channels("pokemon-red");
|
||||
let preset = gameboy_decoder_config_with_macros(&channels);
|
||||
let hold_ms = preset.macros.as_ref().expect("the preset has a macro group").hold_ms;
|
||||
let mut decoder = PopulationDecoder::new(preset).expect("the preset is well formed");
|
||||
decoder.calibrate(&rates(None));
|
||||
let mut config = Config::default();
|
||||
config.loop_.game = "pokemon-red".to_string();
|
||||
config.macros.mode = MacroMode::Macros;
|
||||
config.validate().expect("pokemon-red has a palette in macros mode");
|
||||
let mut layer = macro_layer(&config, hold_ms, SEED).expect("a layer in macros mode");
|
||||
let _ = layer.observe(&mut gb, &AdapterLedger(&adapter), 0.0);
|
||||
Self {
|
||||
gb,
|
||||
adapter,
|
||||
layer,
|
||||
decoder,
|
||||
channels,
|
||||
ms: 0.0,
|
||||
frame: 0,
|
||||
payouts: Vec::new(),
|
||||
}
|
||||
}
|
||||
|
||||
fn byte(&mut self, address: u16) -> u8 {
|
||||
self.gb.read_wram(address)
|
||||
}
|
||||
|
||||
fn in_wild_battle(&mut self) -> bool {
|
||||
self.byte(ram::wIsInBattle) == 1
|
||||
}
|
||||
|
||||
/// Balls in the bag, of any kind (`constants/item_constants.asm`: MASTER_BALL 1,
|
||||
/// ULTRA_BALL 2, GREAT_BALL 3, POKE_BALL 4 — the same four `THROW BALL` looks for).
|
||||
fn balls(&mut self) -> usize {
|
||||
state::bag(&mut self.gb)
|
||||
.iter()
|
||||
.filter(|item| (0x01..=0x04).contains(&item.id) && item.count > 0)
|
||||
.map(|item| usize::from(item.count))
|
||||
.sum()
|
||||
}
|
||||
|
||||
fn step(&mut self) {
|
||||
// Lean on `THROW BALL` while a wild battle is up; otherwise rotate, which is what gets
|
||||
// the fly into the grass in the first place.
|
||||
let hot = if self.in_wild_battle() {
|
||||
Some(BALL)
|
||||
} else {
|
||||
let slot = (self.frame / BURST_FRAMES) as usize % self.channels.len();
|
||||
Some(self.channels[slot])
|
||||
};
|
||||
let bound = self.layer.bound_channels();
|
||||
let active = self.decoder.decode_bound(&rates(hot), self.ms, false, None, Some(&bound));
|
||||
let mask = {
|
||||
let ledger = AdapterLedger(&self.adapter);
|
||||
self.layer.decide(&active, 0, self.ms, &mut self.gb, &ledger).mask
|
||||
};
|
||||
self.gb.set_buttons(mask as u8);
|
||||
self.gb.run_frame().expect("a frame should complete");
|
||||
self.ms += MS_PER_FRAME;
|
||||
self.frame += 1;
|
||||
let ms = self.ms;
|
||||
self.payouts.extend(self.adapter.sample(&mut self.gb, ms));
|
||||
let ledger = AdapterLedger(&self.adapter);
|
||||
let _ = self.layer.observe(&mut self.gb, &ledger, ms);
|
||||
}
|
||||
|
||||
fn catches(&self) -> Vec<&RewardEvent> {
|
||||
self.payouts.iter().filter(|event| event.kind == catalog::kind::CATCH).collect()
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_catch_on_the_cartridge_pays_the_catch_rule_once_with_the_species_in_its_label() {
|
||||
let Some(rom) = rom() else {
|
||||
eprintln!("skipped: FLY_ROM is not set");
|
||||
return;
|
||||
};
|
||||
let Some(checkpoint) = checkpoint() else {
|
||||
eprintln!("skipped: no FLY_CATCH_CHECKPOINT");
|
||||
return;
|
||||
};
|
||||
let mut run = Run::resume(&rom, &checkpoint);
|
||||
let balls = run.balls();
|
||||
if balls == 0 {
|
||||
eprintln!(
|
||||
"skipped: the checkpoint's bag holds no ball (map {:#04x}). `BUY BALL` can buy one \
|
||||
inside a mart; point FLY_CATCH_CHECKPOINT at a state that already has one.",
|
||||
run.adapter.map_id().unwrap_or(u32::MAX)
|
||||
);
|
||||
return;
|
||||
}
|
||||
eprintln!("bag holds {balls} balls; map {:#04x}", run.adapter.map_id().unwrap_or(u32::MAX));
|
||||
|
||||
// Twenty brain minutes is generous for a forest checkpoint: the live run threw 28 balls in
|
||||
// its first Viridian Forest session (`pokemon_red::macros::palette`).
|
||||
let budget = 20 * 60 * 60;
|
||||
let mut battles = 0u32;
|
||||
let mut was_in_battle = false;
|
||||
for _ in 0..budget {
|
||||
run.step();
|
||||
let now = run.in_wild_battle();
|
||||
if now && !was_in_battle {
|
||||
battles += 1;
|
||||
}
|
||||
was_in_battle = now;
|
||||
if !run.catches().is_empty() {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
let catches = run.catches();
|
||||
assert!(
|
||||
!catches.is_empty(),
|
||||
"no catch in {:.1} brain minutes: {battles} wild battles, {} balls left, map {:#04x}, \
|
||||
macros {:?}",
|
||||
run.ms / 60_000.0,
|
||||
run.balls(),
|
||||
run.adapter.map_id().unwrap_or(u32::MAX),
|
||||
run.layer.counts()
|
||||
);
|
||||
|
||||
let caught = catches[0];
|
||||
eprintln!(
|
||||
"caught after {:.1} brain minutes and {battles} wild battles: {} for {}",
|
||||
run.ms / 60_000.0,
|
||||
caught.label,
|
||||
caught.value
|
||||
);
|
||||
assert!(caught.label.starts_with("CAUGHT #"), "{}", caught.label);
|
||||
assert!(
|
||||
(caught.value - 0.30).abs() < 1e-12 || caught.value == catalog::CATCH_REPEAT_VALUE,
|
||||
"a catch pays one of the rule's two amounts, not {}",
|
||||
caught.value
|
||||
);
|
||||
// The cartridge's own flag is clear again by the time the payout lands, which is what makes
|
||||
// the payout a battle-exit event rather than a per-frame one.
|
||||
assert_eq!(run.byte(ram::wCapturedMonSpecies), 0);
|
||||
assert_eq!(
|
||||
run.adapter.progress().counts[catalog::kind::CATCH],
|
||||
1,
|
||||
"one battle, one payout"
|
||||
);
|
||||
// And a species the run is paid for catching is a species the Pokédex knows: the same event
|
||||
// sets the bit the `species` rule reads, whether or not it was new to this run.
|
||||
assert!(run.balls() < balls, "a ball was spent");
|
||||
}
|
||||
|
|
@ -200,6 +200,15 @@ EXTRA_RAM = (
|
|||
'wCurMapTileset',
|
||||
'wTilesetBank',
|
||||
'wTilesetBlocksPtr',
|
||||
# The catch reward (`docs/rewards-learning.md`, `docs/design/macros-wram.md` section 2).
|
||||
# ram/wram.asm's own comment is "0 if no mon was captured": ItemUseBall zeroes it before
|
||||
# every throw and writes wEnemyMonSpecies into it only on the branch that keeps the
|
||||
# Pokemon, and UseBagItem zeroes it again on the way out of the battle. It is the
|
||||
# cartridge's own answer to "was this one caught", and the only signal that needs no
|
||||
# second rule to tell a catch apart from a gift, a trade or an evolution.
|
||||
# services/flysim/tools/resolve_wram.py is the second reading of it, from ram/wram.asm at
|
||||
# this commit, bracketed by wFontLoaded and wForcePlayerToChooseMon.
|
||||
'wCapturedMonSpecies',
|
||||
)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -49,6 +49,33 @@ WANTED = {
|
|||
# not bank 0, so this is the read the memory seam grew a bank for.
|
||||
'wTilesetBank': 'the ROM bank the blockset lives in',
|
||||
'wTilesetBlocksPtr': 'blocks to tiles, 16 bytes per block',
|
||||
# The catch reward (`docs/rewards-learning.md`, `docs/design/macros-wram.md`
|
||||
# section 10). ram/wram.asm's own comment is "0 if no mon was captured":
|
||||
# ItemUseBall zeroes it before every throw and writes wEnemyMonSpecies into it
|
||||
# only on the branch that keeps the caught Pokemon, and UseBagItem zeroes it
|
||||
# again on the way out of the battle. It is the cartridge's own answer to "was
|
||||
# this one caught", and the only signal that needs no second rule to tell a
|
||||
# catch apart from a gift, a trade or an evolution.
|
||||
'wCapturedMonSpecies': 'the species a ball just caught, 0 for none',
|
||||
}
|
||||
|
||||
|
||||
#: Constants the decomp defines through its `const` enumeration rather than with a
|
||||
#: plain `EQU`, so `constants()` cannot evaluate their expressions. They matter here
|
||||
#: because `NUM_TMS + NUM_HMS` is the size of `wMonHLearnset`, and that one
|
||||
#: declaration is what kills the cursor on its way through the battle engine's
|
||||
#: scratch bytes -- the region `wCapturedMonSpecies` lives in.
|
||||
#:
|
||||
#: Each is *counted* from the decomp rather than written out by hand, which is the
|
||||
#: same rule the rest of this tool follows. `DEF NUM_HMS EQU const_value - HM01` is
|
||||
#: by construction the number of `add_hm` definitions after `HM01`, and
|
||||
#: `item_constants.asm`'s own `ASSERT NUM_TMS == const_value - TM01` ties `NUM_TMS`
|
||||
#: to the number of `add_tm` definitions -- so `NUM_TMS` is counted *and* compared
|
||||
#: against the literal the same file declares, and a decomp that moved one without
|
||||
#: the other stops the run instead of producing an address.
|
||||
COUNTED = {
|
||||
'NUM_HMS': ('constants/item_constants.asm', r'^\s*add_hm\s+\w+'),
|
||||
'NUM_TMS': ('constants/item_constants.asm', r'^\s*add_tm\s+\w+'),
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -68,6 +95,10 @@ def constants(root: Path) -> dict[str, int]:
|
|||
BLOCK_WIDTH`). A name whose expression never becomes evaluable is simply left
|
||||
out, which kills the cursor at any declaration that uses it.
|
||||
"""
|
||||
counted = {
|
||||
name: len(re.findall(pattern, (root / path).read_text(), re.M))
|
||||
for name, (path, pattern) in COUNTED.items()
|
||||
}
|
||||
pending: dict[str, str] = {}
|
||||
sources = sorted((root / 'constants').glob('*.asm')) + sorted(
|
||||
(root / 'constants').glob('*.inc')
|
||||
|
|
@ -77,7 +108,7 @@ def constants(root: Path) -> dict[str, int]:
|
|||
r'^\s*(?:DEF|def)\s+(\w+)\s+(?:EQU|equ)\s+([^;\n]+)', path.read_text(), re.M
|
||||
):
|
||||
pending.setdefault(name, value.strip())
|
||||
out: dict[str, int] = {}
|
||||
out: dict[str, int] = dict(counted)
|
||||
while pending:
|
||||
progressed = False
|
||||
for name in list(pending):
|
||||
|
|
@ -89,6 +120,12 @@ def constants(root: Path) -> dict[str, int]:
|
|||
progressed = True
|
||||
if not progressed:
|
||||
break
|
||||
for name, value in counted.items():
|
||||
if out.get(name, value) != value:
|
||||
raise SystemExit(
|
||||
f'{name}: the decomp declares {out[name]} and defines {value} of them'
|
||||
)
|
||||
out[name] = value
|
||||
return out
|
||||
|
||||
|
||||
|
|
@ -367,11 +404,17 @@ def main() -> None:
|
|||
raise SystemExit('the walk disagrees with symbols.rs; nothing emitted')
|
||||
print(f'{checked} of {len(table)} pinned addresses re-derived from wram.asm, no disagreement')
|
||||
|
||||
missing = [name for name in WANTED if name not in resolved]
|
||||
# A name this tool has already emitted is pinned, so the walk meets it as an
|
||||
# anchor rather than resolving it: it was re-derived all the same, and the
|
||||
# comparison above is what says so.
|
||||
missing = [name for name in WANTED if name not in resolved and name not in table]
|
||||
if missing:
|
||||
raise SystemExit(f'unanchored, so not resolved: {", ".join(missing)}')
|
||||
for name in WANTED:
|
||||
print(f'{name} = ${resolved[name]:04x} ({WANTED[name]})')
|
||||
if name in resolved:
|
||||
print(f'{name} = ${resolved[name]:04x} ({WANTED[name]})')
|
||||
else:
|
||||
print(f'{name} = ${table[name]:04x} (already pinned; {WANTED[name]})')
|
||||
|
||||
if not args.emit:
|
||||
return
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue