Merge main into feat/sf-publish-01
# Conflicts: # services/flysim/crates/fly-session/README.md # services/flysim/crates/fly-session/src/coordinator.rs # services/flysim/crates/fly-session/src/harness.rs # services/flysim/crates/fly-session/src/launcher.rs
This commit is contained in:
commit
e27306f171
23 changed files with 6017 additions and 57 deletions
|
|
@ -105,6 +105,31 @@ names; a manifest missing any of them is not a complete checkpoint.
|
||||||
| `helperState` | External-helper state required for exact resume, as payload names |
|
| `helperState` | External-helper state required for exact resume, as payload names |
|
||||||
| `payloads` | `[{name, byteLength, digest}]`, mirroring the payload table |
|
| `payloads` | `[{name, byteLength, digest}]`, mirroring the payload table |
|
||||||
|
|
||||||
|
**Amendment, 2026-09-22 (STATE-01).** The table above names a holder for every payload except
|
||||||
|
the environment's own, although section 6's fixture has one (`world`) and a group install has
|
||||||
|
to map it by name like any other participant's. The manifest therefore also records:
|
||||||
|
|
||||||
|
| Field | Contents |
|
||||||
|
| --- | --- |
|
||||||
|
| `environment` | `{workerId, payload}`: which worker the world belonged to and the payload name holding its state |
|
||||||
|
|
||||||
|
The reference implementations' required-field set was also missing `helperState`, which this
|
||||||
|
section has listed from the start. Both are now in `REQUIRED_MANIFEST_FIELDS` in Rust and in
|
||||||
|
TypeScript, and the fixture was regenerated by the existing example. The schema set is
|
||||||
|
untouched, so `contractDigest` is unchanged.
|
||||||
|
|
||||||
|
`coordinator.eventWatermarks` is `{lastSourceStep, issued}`. The fixture illustrated
|
||||||
|
`{lastEventId, lastOrdinal}`, and it is the illustration that changed: an event id is derived
|
||||||
|
from the epoch, so a watermark spelled as one cannot be compared across the restore that
|
||||||
|
gives the session a new epoch, while a source step and an issued count can.
|
||||||
|
|
||||||
|
**A required-manifest-field change is compatibility-relevant and `contractDigest` does not
|
||||||
|
cover it.** The digest is taken over the schema set, and this manifest is not in it, so
|
||||||
|
`envelopeVersion` is the only thing that can carry such a change. It stays `1` here only
|
||||||
|
because no production `FLYSESS1` file exists yet: once one does, adding or removing a required
|
||||||
|
manifest field **must** bump `envelopeVersion`, because a reader of the older version would
|
||||||
|
otherwise accept a file it cannot completely read, or refuse one it could.
|
||||||
|
|
||||||
`payloads` is redundant with the table on purpose: the table is what a reader needs to map
|
`payloads` is redundant with the table on purpose: the table is what a reader needs to map
|
||||||
bytes, and the manifest is what a store lists, compares and reports without opening the
|
bytes, and the manifest is what a store lists, compares and reports without opening the
|
||||||
payload area. A reader checks that the two agree.
|
payload area. A reader checks that the two agree.
|
||||||
|
|
|
||||||
|
|
@ -205,6 +205,28 @@ restored time. It cannot advance gameplay to manufacture it. Capture/reconstruct
|
||||||
covers render/inspection state and any pending sensor pipeline. Agent state agrees with it;
|
covers render/inspection state and any pending sensor pipeline. Agent state agrees with it;
|
||||||
do not replay reward or recalibrate merely to fill missing cached data.
|
do not replay reward or recalibrate merely to fill missing cached data.
|
||||||
|
|
||||||
|
**Amendment, 2026-09-22 (STATE-01).** Three readings of this section, made explicit because
|
||||||
|
they are now enforced:
|
||||||
|
|
||||||
|
- `compatibilityDigest` on `CaptureResult` and `StageRestoreParams` is the **participant's**
|
||||||
|
capture compatibility digest of [worker interfaces](workers-v1.md) section 2 -- profile,
|
||||||
|
resolved seed, numerical model version and effective instance configuration for an agent;
|
||||||
|
backend, content, patch, controller and parser identity for an environment. It is not the
|
||||||
|
manifest's `compatibility` block of section 4, which is the composition's and which the
|
||||||
|
coordinator compares before anything is asked to stage. Both exist because they answer
|
||||||
|
different questions, and a restore that passed the second could still be handing an agent
|
||||||
|
another agent's brain.
|
||||||
|
- The observation `ActivateRestore` returns ran no transition, so it carries **no audio
|
||||||
|
chunk**, and one in it is refused. Section 2's chunk is the audio of an interval and this
|
||||||
|
observation covers none; MEDIA-01 implemented that rule as "boundary 0 carries no chunk",
|
||||||
|
which is true of the only such observation that slice could produce and false of this one.
|
||||||
|
The rule is about provenance, not about the boundary number.
|
||||||
|
- A participant that staged into a group install the coordinator then abandoned must be
|
||||||
|
**replaced** before another restore, exactly as one that activated must. It is holding a
|
||||||
|
validated replacement state that nothing installed, and [session RPC](ipc-v1.md) section 6
|
||||||
|
already refuses to silently reattach such a participant to an active epoch. Without this the
|
||||||
|
group's second attempt meets its own leftovers and calls them a conflict.
|
||||||
|
|
||||||
If emulator validation requires mutation, stage a stopped replacement emulator. If that cannot
|
If emulator validation requires mutation, stage a stopped replacement emulator. If that cannot
|
||||||
provide externally atomic resume, advertise episode-restart, not exact-checkpoint. After all
|
provide externally atomic resume, advertise episode-restart, not exact-checkpoint. After all
|
||||||
activation acknowledgments, install the coordinator's staged task/executor/admission state
|
activation acknowledgments, install the coordinator's staged task/executor/admission state
|
||||||
|
|
|
||||||
|
|
@ -223,7 +223,12 @@ export function decode(input: Uint8Array): Envelope {
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
/** The manifest fields state-media-v1 section 4 requires. */
|
/**
|
||||||
|
* The manifest fields state-media-v1 section 4 requires.
|
||||||
|
*
|
||||||
|
* `helperState` and `environment` join the list under the 2026-09-22 amendment to
|
||||||
|
* checkpoint-envelope-v1 section 3.
|
||||||
|
*/
|
||||||
export const REQUIRED_MANIFEST_FIELDS = [
|
export const REQUIRED_MANIFEST_FIELDS = [
|
||||||
'envelopeVersion',
|
'envelopeVersion',
|
||||||
'checkpointId',
|
'checkpointId',
|
||||||
|
|
@ -236,6 +241,8 @@ export const REQUIRED_MANIFEST_FIELDS = [
|
||||||
'compatibility',
|
'compatibility',
|
||||||
'agents',
|
'agents',
|
||||||
'coordinator',
|
'coordinator',
|
||||||
|
'environment',
|
||||||
|
'helperState',
|
||||||
'payloads',
|
'payloads',
|
||||||
] as const;
|
] as const;
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -197,8 +197,9 @@ fn checkpoint_envelope() -> String {
|
||||||
"priorInspection": "prior-inspection",
|
"priorInspection": "prior-inspection",
|
||||||
"executorState": [{"agentId": "fly-a", "payload": "executor-fly-a"}],
|
"executorState": [{"agentId": "fly-a", "payload": "executor-fly-a"}],
|
||||||
"admissionState": null,
|
"admissionState": null,
|
||||||
"eventWatermarks": {"lastEventId": "evt-1", "lastOrdinal": "7"},
|
"eventWatermarks": {"lastSourceStep": "42", "issued": "7"},
|
||||||
},
|
},
|
||||||
|
"environment": {"workerId": "arena", "payload": "world"},
|
||||||
"helperState": [],
|
"helperState": [],
|
||||||
"payloads": payload_table(),
|
"payloads": payload_table(),
|
||||||
});
|
});
|
||||||
|
|
|
||||||
|
|
@ -59,10 +59,14 @@
|
||||||
],
|
],
|
||||||
"admissionState": null,
|
"admissionState": null,
|
||||||
"eventWatermarks": {
|
"eventWatermarks": {
|
||||||
"lastEventId": "evt-1",
|
"lastSourceStep": "42",
|
||||||
"lastOrdinal": "7"
|
"issued": "7"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"environment": {
|
||||||
|
"workerId": "arena",
|
||||||
|
"payload": "world"
|
||||||
|
},
|
||||||
"helperState": [],
|
"helperState": [],
|
||||||
"payloads": [
|
"payloads": [
|
||||||
{
|
{
|
||||||
|
|
@ -115,49 +119,49 @@
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"envelope": {
|
"envelope": {
|
||||||
"base64": "RkxZU0VTUzEBAAAAIAAAAAAIAAAFAAAAIAgAAAAAAAB7ImFnZW50cyI6W3siYWdlbnRJZCI6ImZseS1hIiwiYnJhaW5UaWNrcyI6IjI1MzQiLCJkYXRhc2V0RGlnZXN0IjoiNmMwYWYxZjA3ODRlZjYzYTM5M2VlNzdkNjE0ZTgyNDZjNjI1MDUxMzYwZjNmMWE0ODgzODM3NGM1ZDM1NWI1MiIsIm1vZGVsVmVyc2lvbiI6ImxpZi0xbXMtZjY0LXYyIiwicGF5bG9hZCI6ImFnZW50LWZseS1hIiwicGxhc3RpY2l0eVZlcnNpb24iOiJmbHkta2MtbWJvbi1yc3RkcC12MiIsInByb2ZpbGVEaWdlc3QiOiIxOTAwZWFiNmMwMjg0ODNkNzEyNjU5OWVlNmY1MGRlMGQyNzkwN2I1YzY1ZmE5MDUyNDU4MGI0YjBmOTg1MmIwIiwicmVtYWluZGVyIjp7ImRlbm9taW5hdG9yIjoiMyIsIm51bWVyYXRvciI6IjEwMDAwMDAifSwic2VlZCI6LTE4NDk0NjA2M31dLCJjaGVja3BvaW50SWQiOiJja3B0LTEiLCJjb21wYXRpYmlsaXR5Ijp7ImJhY2tlbmREaWdlc3QiOiIxMGUwOGE0MTllODUwZWJhMWViYmExOGZkZDI4ZWI3ZWMxYjdlOGJhYTliY2MzYjk3M2UyYjg4OTFlYzcyNmJlIiwiY29udGVudERpZ2VzdCI6ImVkNzAwMmI0MzllOWFjODQ1ZjIyMzU3ZDgyMmJhYzE0NDQ3MzBmYmRiNjAxNmQzZWM5NDMyMjk3YjllYzlmNzMiLCJjb250cm9sbGVyRGlnZXN0IjoiYzE0NzIxMzViMTRjNzdjOGJlZjk4ZTczZjcwMjA4MzI1ZmEwZGNmMWU2YmQ2NjhhZTliMzFhOWNlYTI5NWZlNyIsInBhcnNlckRpZ2VzdCI6ImIxN2Q0NTEyMTE1MDkyOGYyMTQ2YWY0OWUxOTVlZmYxZWVmNWQ2NzMyNWJlMjczYTczM2ZiNzRhY2FkYWEzNDIiLCJwYXRjaERpZ2VzdCI6ImE0ODk1ZWI0NGFmYzMzNmZlY2JiYTZlNTIwY2Q2N2UxNzhkYWNlMDI3NjY1NWQxMDJmY2VmZmE4ZTVmNzA1NzAiLCJzdGF0ZUZvcm1hdElkIjoiZmx5c2Vzcy0xIn0sImNvbXBvc2l0aW9uRGlnZXN0IjoiNzMwZDcyNWM4YTU5ZDNhNzMwM2RlZjJiZWQwNDFhNTc3ZWRiNDI1NWFhYmQ0ODg5Y2UxMjkxODMxMWQ5NTJmMCIsImNvb3JkaW5hdG9yIjp7ImFkbWlzc2lvblN0YXRlIjpudWxsLCJldmVudFdhdGVybWFya3MiOnsibGFzdEV2ZW50SWQiOiJldnQtMSIsImxhc3RPcmRpbmFsIjoiNyJ9LCJleGVjdXRvclN0YXRlIjpbeyJhZ2VudElkIjoiZmx5LWEiLCJwYXlsb2FkIjoiZXhlY3V0b3ItZmx5LWEifV0sInByaW9ySW5zcGVjdGlvbiI6InByaW9yLWluc3BlY3Rpb24iLCJ0YXNrTGVkZ2VyIjoidGFzay1sZWRnZXIifSwiZW52ZWxvcGVWZXJzaW9uIjoxLCJlcGlzb2RlSWQiOiJlcGlzb2RlLTEiLCJoZWxwZXJTdGF0ZSI6W10sInBheWxvYWRzIjpbeyJieXRlTGVuZ3RoIjoiMTciLCJkaWdlc3QiOiIxMzIxZGZmYjBjZGM2ZjkwOTJjYmY3ZmEyYTVmYzY4YmJlZDEyYzk5M2Q1YWQzOTgyNjQwMTI4MTBjZTliZjkzIiwibmFtZSI6ImFnZW50LWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTQiLCJkaWdlc3QiOiIzYWVlNjBkZjdlMjllZmViYTdmNWY5OWZjNTg2NzY0N2IzNmFlYmZmMWQ1ZDNjODM4ZGJmZjMyMzEyMmU2NDYyIiwibmFtZSI6ImV4ZWN1dG9yLWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTEiLCJkaWdlc3QiOiI0MGIwMGVkMmJiYmE5MDFkNjgyMDVmZjcxYjA0YTQ0YjllZTUzYzUxY2IzMTA5YWEyY2VhYTQ0ZjFjNDU3MjdlIiwibmFtZSI6InRhc2stbGVkZ2VyIn0seyJieXRlTGVuZ3RoIjoiMTAiLCJkaWdlc3QiOiIyYzEzYjdiNGQ5YTk5MTY4MDFhYjkxOTFjMzE0ZjMxYjA0NWU5YjljNWI2NjlhNmMwNDc0ZjAyMTdlZjc1YmY1IiwibmFtZSI6InByaW9yLWluc3BlY3Rpb24ifSx7ImJ5dGVMZW5ndGgiOiI2NCIsImRpZ2VzdCI6ImY1YTVmZDQyZDE2YTIwMzAyNzk4ZWY2ZWQzMDk5NzliNDMwMDNkMjMyMGQ5ZjBlOGVhOTgzMWE5Mjc1OWZiNGIiLCJuYW1lIjoid29ybGQifV0sInBvcnRNYXAiOlt7ImFnZW50SWQiOiJmbHktYSIsInBvcnRJZCI6InBvcnQtMSJ9XSwic2NoZWR1bGVySWQiOiJsb2Nrc3RlcC12MSIsInNvdXJjZVNjb3BlIjp7ImVwb2NoIjoiZXBvY2gtMSIsInNlc3Npb25JZCI6ImRlbW8iLCJzdGVwIjoiNDIifSwid29ybGRUaW1lIjp7ImRlbm9taW5hdG9yIjoiMSIsIm51bWVyYXRvciI6IjcwMDAwMDAwMCJ9fWFnZW50LWZseS1hAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABQCgAAAAAAABEAAAAAAAAAEyHf+wzcb5CSy/f6Kl/Gi77RLJk9WtOYJkASgQzpv5NleGVjdXRvci1mbHktYQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAaAoAAAAAAAAOAAAAAAAAADruYN9+Ke/rp/X5n8WGdkezauv/HV08g42/8yMSLmRidGFzay1sZWRnZXIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHgKAAAAAAAACwAAAAAAAABAsA7Su7qQHWggX/cbBKRLnuU8UcsxCaos6qRPHEVyfnByaW9yLWluc3BlY3Rpb24AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACICgAAAAAAAAoAAAAAAAAALBO3tNmpkWgBq5GRwxTzGwRem5xbZppsBHTwIX73W/V3b3JsZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAmAoAAAAAAABAAAAAAAAAAPWl/ULRaiAwJ5jvbtMJl5tDAD0jINnw6OqYMaknWftLYWdlbnQgc3RhdGUgYnl0ZXMAAAAAAAAAZXhlY3V0b3Igc3RhdGUAAHsicmFuayI6MTB9AAAAAAB7Im1hcCI6NDB9AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgLAAAAAAAAq++fEx+FvDZho/eB4imbENN4HZrGNC2OCAsI7/gp9r5GTFlTRVNTRg==",
|
"base64": "RkxZU0VTUzEBAAAAIAAAADAIAAAFAAAAUAgAAAAAAAB7ImFnZW50cyI6W3siYWdlbnRJZCI6ImZseS1hIiwiYnJhaW5UaWNrcyI6IjI1MzQiLCJkYXRhc2V0RGlnZXN0IjoiNmMwYWYxZjA3ODRlZjYzYTM5M2VlNzdkNjE0ZTgyNDZjNjI1MDUxMzYwZjNmMWE0ODgzODM3NGM1ZDM1NWI1MiIsIm1vZGVsVmVyc2lvbiI6ImxpZi0xbXMtZjY0LXYyIiwicGF5bG9hZCI6ImFnZW50LWZseS1hIiwicGxhc3RpY2l0eVZlcnNpb24iOiJmbHkta2MtbWJvbi1yc3RkcC12MiIsInByb2ZpbGVEaWdlc3QiOiIxOTAwZWFiNmMwMjg0ODNkNzEyNjU5OWVlNmY1MGRlMGQyNzkwN2I1YzY1ZmE5MDUyNDU4MGI0YjBmOTg1MmIwIiwicmVtYWluZGVyIjp7ImRlbm9taW5hdG9yIjoiMyIsIm51bWVyYXRvciI6IjEwMDAwMDAifSwic2VlZCI6LTE4NDk0NjA2M31dLCJjaGVja3BvaW50SWQiOiJja3B0LTEiLCJjb21wYXRpYmlsaXR5Ijp7ImJhY2tlbmREaWdlc3QiOiIxMGUwOGE0MTllODUwZWJhMWViYmExOGZkZDI4ZWI3ZWMxYjdlOGJhYTliY2MzYjk3M2UyYjg4OTFlYzcyNmJlIiwiY29udGVudERpZ2VzdCI6ImVkNzAwMmI0MzllOWFjODQ1ZjIyMzU3ZDgyMmJhYzE0NDQ3MzBmYmRiNjAxNmQzZWM5NDMyMjk3YjllYzlmNzMiLCJjb250cm9sbGVyRGlnZXN0IjoiYzE0NzIxMzViMTRjNzdjOGJlZjk4ZTczZjcwMjA4MzI1ZmEwZGNmMWU2YmQ2NjhhZTliMzFhOWNlYTI5NWZlNyIsInBhcnNlckRpZ2VzdCI6ImIxN2Q0NTEyMTE1MDkyOGYyMTQ2YWY0OWUxOTVlZmYxZWVmNWQ2NzMyNWJlMjczYTczM2ZiNzRhY2FkYWEzNDIiLCJwYXRjaERpZ2VzdCI6ImE0ODk1ZWI0NGFmYzMzNmZlY2JiYTZlNTIwY2Q2N2UxNzhkYWNlMDI3NjY1NWQxMDJmY2VmZmE4ZTVmNzA1NzAiLCJzdGF0ZUZvcm1hdElkIjoiZmx5c2Vzcy0xIn0sImNvbXBvc2l0aW9uRGlnZXN0IjoiNzMwZDcyNWM4YTU5ZDNhNzMwM2RlZjJiZWQwNDFhNTc3ZWRiNDI1NWFhYmQ0ODg5Y2UxMjkxODMxMWQ5NTJmMCIsImNvb3JkaW5hdG9yIjp7ImFkbWlzc2lvblN0YXRlIjpudWxsLCJldmVudFdhdGVybWFya3MiOnsiaXNzdWVkIjoiNyIsImxhc3RTb3VyY2VTdGVwIjoiNDIifSwiZXhlY3V0b3JTdGF0ZSI6W3siYWdlbnRJZCI6ImZseS1hIiwicGF5bG9hZCI6ImV4ZWN1dG9yLWZseS1hIn1dLCJwcmlvckluc3BlY3Rpb24iOiJwcmlvci1pbnNwZWN0aW9uIiwidGFza0xlZGdlciI6InRhc2stbGVkZ2VyIn0sImVudmVsb3BlVmVyc2lvbiI6MSwiZW52aXJvbm1lbnQiOnsicGF5bG9hZCI6IndvcmxkIiwid29ya2VySWQiOiJhcmVuYSJ9LCJlcGlzb2RlSWQiOiJlcGlzb2RlLTEiLCJoZWxwZXJTdGF0ZSI6W10sInBheWxvYWRzIjpbeyJieXRlTGVuZ3RoIjoiMTciLCJkaWdlc3QiOiIxMzIxZGZmYjBjZGM2ZjkwOTJjYmY3ZmEyYTVmYzY4YmJlZDEyYzk5M2Q1YWQzOTgyNjQwMTI4MTBjZTliZjkzIiwibmFtZSI6ImFnZW50LWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTQiLCJkaWdlc3QiOiIzYWVlNjBkZjdlMjllZmViYTdmNWY5OWZjNTg2NzY0N2IzNmFlYmZmMWQ1ZDNjODM4ZGJmZjMyMzEyMmU2NDYyIiwibmFtZSI6ImV4ZWN1dG9yLWZseS1hIn0seyJieXRlTGVuZ3RoIjoiMTEiLCJkaWdlc3QiOiI0MGIwMGVkMmJiYmE5MDFkNjgyMDVmZjcxYjA0YTQ0YjllZTUzYzUxY2IzMTA5YWEyY2VhYTQ0ZjFjNDU3MjdlIiwibmFtZSI6InRhc2stbGVkZ2VyIn0seyJieXRlTGVuZ3RoIjoiMTAiLCJkaWdlc3QiOiIyYzEzYjdiNGQ5YTk5MTY4MDFhYjkxOTFjMzE0ZjMxYjA0NWU5YjljNWI2NjlhNmMwNDc0ZjAyMTdlZjc1YmY1IiwibmFtZSI6InByaW9yLWluc3BlY3Rpb24ifSx7ImJ5dGVMZW5ndGgiOiI2NCIsImRpZ2VzdCI6ImY1YTVmZDQyZDE2YTIwMzAyNzk4ZWY2ZWQzMDk5NzliNDMwMDNkMjMyMGQ5ZjBlOGVhOTgzMWE5Mjc1OWZiNGIiLCJuYW1lIjoid29ybGQifV0sInBvcnRNYXAiOlt7ImFnZW50SWQiOiJmbHktYSIsInBvcnRJZCI6InBvcnQtMSJ9XSwic2NoZWR1bGVySWQiOiJsb2Nrc3RlcC12MSIsInNvdXJjZVNjb3BlIjp7ImVwb2NoIjoiZXBvY2gtMSIsInNlc3Npb25JZCI6ImRlbW8iLCJzdGVwIjoiNDIifSwid29ybGRUaW1lIjp7ImRlbm9taW5hdG9yIjoiMSIsIm51bWVyYXRvciI6IjcwMDAwMDAwMCJ9fWFnZW50LWZseS1hAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACACgAAAAAAABEAAAAAAAAAEyHf+wzcb5CSy/f6Kl/Gi77RLJk9WtOYJkASgQzpv5NleGVjdXRvci1mbHktYQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAmAoAAAAAAAAOAAAAAAAAADruYN9+Ke/rp/X5n8WGdkezauv/HV08g42/8yMSLmRidGFzay1sZWRnZXIAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAKgKAAAAAAAACwAAAAAAAABAsA7Su7qQHWggX/cbBKRLnuU8UcsxCaos6qRPHEVyfnByaW9yLWluc3BlY3Rpb24AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC4CgAAAAAAAAoAAAAAAAAALBO3tNmpkWgBq5GRwxTzGwRem5xbZppsBHTwIX73W/V3b3JsZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAyAoAAAAAAABAAAAAAAAAAPWl/ULRaiAwJ5jvbtMJl5tDAD0jINnw6OqYMaknWftLYWdlbnQgc3RhdGUgYnl0ZXMAAAAAAAAAZXhlY3V0b3Igc3RhdGUAAHsicmFuayI6MTB9AAAAAAB7Im1hcCI6NDB9AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADgLAAAAAAAAX4L9WkdX0MViD3h5YJgf7VQgwocWU4XJoKX6MUwl+8hGTFlTRVNTRg==",
|
||||||
"byteLength": 2824,
|
"byteLength": 2872,
|
||||||
"layout": {
|
"layout": {
|
||||||
"headerBytes": 32,
|
"headerBytes": 32,
|
||||||
"manifestOffset": "32",
|
"manifestOffset": "32",
|
||||||
"manifestBytes": 2048,
|
"manifestBytes": 2096,
|
||||||
"tableOffset": "2080",
|
"tableOffset": "2128",
|
||||||
"tableEntryBytes": 112,
|
"tableEntryBytes": 112,
|
||||||
"entries": [
|
"entries": [
|
||||||
{
|
{
|
||||||
"name": "agent-fly-a",
|
"name": "agent-fly-a",
|
||||||
"offset": "2640",
|
"offset": "2688",
|
||||||
"byteLength": "17",
|
"byteLength": "17",
|
||||||
"digest": "1321dffb0cdc6f9092cbf7fa2a5fc68bbed12c993d5ad398264012810ce9bf93"
|
"digest": "1321dffb0cdc6f9092cbf7fa2a5fc68bbed12c993d5ad398264012810ce9bf93"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"name": "executor-fly-a",
|
"name": "executor-fly-a",
|
||||||
"offset": "2664",
|
"offset": "2712",
|
||||||
"byteLength": "14",
|
"byteLength": "14",
|
||||||
"digest": "3aee60df7e29efeba7f5f99fc5867647b36aebff1d5d3c838dbff323122e6462"
|
"digest": "3aee60df7e29efeba7f5f99fc5867647b36aebff1d5d3c838dbff323122e6462"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"name": "task-ledger",
|
"name": "task-ledger",
|
||||||
"offset": "2680",
|
"offset": "2728",
|
||||||
"byteLength": "11",
|
"byteLength": "11",
|
||||||
"digest": "40b00ed2bbba901d68205ff71b04a44b9ee53c51cb3109aa2ceaa44f1c45727e"
|
"digest": "40b00ed2bbba901d68205ff71b04a44b9ee53c51cb3109aa2ceaa44f1c45727e"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"name": "prior-inspection",
|
"name": "prior-inspection",
|
||||||
"offset": "2696",
|
"offset": "2744",
|
||||||
"byteLength": "10",
|
"byteLength": "10",
|
||||||
"digest": "2c13b7b4d9a9916801ab9191c314f31b045e9b9c5b669a6c0474f0217ef75bf5"
|
"digest": "2c13b7b4d9a9916801ab9191c314f31b045e9b9c5b669a6c0474f0217ef75bf5"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"name": "world",
|
"name": "world",
|
||||||
"offset": "2712",
|
"offset": "2760",
|
||||||
"byteLength": "64",
|
"byteLength": "64",
|
||||||
"digest": "f5a5fd42d16a20302798ef6ed309979b43003d2320d9f0e8ea9831a92759fb4b"
|
"digest": "f5a5fd42d16a20302798ef6ed309979b43003d2320d9f0e8ea9831a92759fb4b"
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"footerOffset": "2776",
|
"footerOffset": "2824",
|
||||||
"footerBytes": 48,
|
"footerBytes": 48,
|
||||||
"totalBytes": "2824"
|
"totalBytes": "2872"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
"corruption": [
|
"corruption": [
|
||||||
|
|
@ -178,17 +182,17 @@
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"name": "a flipped payload byte",
|
"name": "a flipped payload byte",
|
||||||
"offset": 2640,
|
"offset": 2688,
|
||||||
"reason": "every payload carries its own digest"
|
"reason": "every payload carries its own digest"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"name": "a flipped footer digest byte",
|
"name": "a flipped footer digest byte",
|
||||||
"offset": 2784,
|
"offset": 2832,
|
||||||
"reason": "the footer digest must match the contents"
|
"reason": "the footer digest must match the contents"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"name": "a flipped footer magic byte",
|
"name": "a flipped footer magic byte",
|
||||||
"offset": 2816,
|
"offset": 2864,
|
||||||
"reason": "a truncated file cannot look complete"
|
"reason": "a truncated file cannot look complete"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
|
|
|
||||||
|
|
@ -296,6 +296,11 @@ pub fn decode(bytes: &[u8]) -> Result<Envelope> {
|
||||||
|
|
||||||
/// The manifest fields state-media-v1 section 4 requires, checked as a set: a manifest that
|
/// The manifest fields state-media-v1 section 4 requires, checked as a set: a manifest that
|
||||||
/// omits one of them is not a complete checkpoint.
|
/// omits one of them is not a complete checkpoint.
|
||||||
|
///
|
||||||
|
/// `helperState` and `environment` join the list under the 2026-09-22 amendment to
|
||||||
|
/// checkpoint-envelope-v1 section 3: the first has been in that section's table from the
|
||||||
|
/// start and was missing here, and the second is the holder of the world's own payload, which
|
||||||
|
/// the table named for every other participant and not for the environment.
|
||||||
pub const REQUIRED_MANIFEST_FIELDS: &[&str] = &[
|
pub const REQUIRED_MANIFEST_FIELDS: &[&str] = &[
|
||||||
"envelopeVersion",
|
"envelopeVersion",
|
||||||
"checkpointId",
|
"checkpointId",
|
||||||
|
|
@ -308,6 +313,8 @@ pub const REQUIRED_MANIFEST_FIELDS: &[&str] = &[
|
||||||
"compatibility",
|
"compatibility",
|
||||||
"agents",
|
"agents",
|
||||||
"coordinator",
|
"coordinator",
|
||||||
|
"environment",
|
||||||
|
"helperState",
|
||||||
"payloads",
|
"payloads",
|
||||||
];
|
];
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -45,6 +45,7 @@ Ready(k) ─ Prepare all agents concurrently ───────────
|
||||||
| `metrics` | Latency percentiles and the machine's core and memory counters |
|
| `metrics` | Latency percentiles and the machine's core and memory counters |
|
||||||
| `measure` | The execution-mode comparison of the guide's section 5 |
|
| `measure` | The execution-mode comparison of the guide's section 5 |
|
||||||
| `cli` | The binary's subcommands: `agent`, `environment`, `measure` |
|
| `cli` | The binary's subcommands: `agent`, `environment`, `measure` |
|
||||||
|
| `state` | The durable checkpoint store over `FLYSESS1`: compatibility, generations, the bounded writer |
|
||||||
| `harness` | The runnable composition: router, the flies, one arena, one coordinator |
|
| `harness` | The runnable composition: router, the flies, one arena, one coordinator |
|
||||||
|
|
||||||
## Execution modes and the launcher
|
## Execution modes and the launcher
|
||||||
|
|
@ -179,6 +180,44 @@ harness.shutdown().await;
|
||||||
event ids derived from epoch, source step, rule and ordinal.
|
event ids derived from epoch, source step, rule and ordinal.
|
||||||
- **Executors.** The stateless identity executor only, as v1 specifies.
|
- **Executors.** The stateless identity executor only, as v1 specifies.
|
||||||
|
|
||||||
|
## Checkpoints and recovery
|
||||||
|
|
||||||
|
The durable store is `state`, over the `FLYSESS1` layout the contract crate owns.
|
||||||
|
|
||||||
|
- **One boundary, every participant.** `Coordinator::capture` runs at `Ready(k)` or
|
||||||
|
`Paused(k)` only. It takes its queue slot *before* the first `State.Capture`, so a saturated
|
||||||
|
writer refuses the capture rather than queueing it without bound, and the refusal is a
|
||||||
|
`BUSY` a stepping session survives rather than an epoch failure.
|
||||||
|
- **Capture and durability are two events.** `State.Capture` completes when an immutable
|
||||||
|
capture exists; `Coordinator::await_durable` completes when the store manifest rename has
|
||||||
|
happened, which is the durable commit point. Only the second moves the durable mark, and the
|
||||||
|
three ways it can end without one are told apart: `Failed` (the write stopped),
|
||||||
|
`ReplyLost` (the write finished and the acknowledgment did not arrive) and
|
||||||
|
`DeadlineExpired` (the caller's own budget ran out while the save was still going).
|
||||||
|
`Coordinator::resolve_durable` then asks the store about the *same* checkpoint instead of
|
||||||
|
saving again.
|
||||||
|
- **The writer is bounded twice**, and the two bounds refuse at different moments. The
|
||||||
|
outstanding-capture bound is taken before a capture is requested; the byte budget cannot be,
|
||||||
|
because a capture's size is not known until it exists, so it refuses at submit and releases
|
||||||
|
the payloads with the refusal. The writer owns its payload handles until the bytes are
|
||||||
|
committed or the job fails. A queued *replaceable* capture is superseded by a later one,
|
||||||
|
releasing its holds; a durable one never is.
|
||||||
|
- **The install is a group.** A restore selects a complete compatible generation, imports its
|
||||||
|
payloads as fresh artifacts, stages every participant, validates the coordinator's own
|
||||||
|
ledgers, and only then activates. A failure anywhere leaves the fence closed, and every
|
||||||
|
participant that got as far as staging is recorded as one that must be replaced before
|
||||||
|
another restore is attempted.
|
||||||
|
- **The fence lifts once.** `Failed -> Restoring(k) -> Paused(k)`, at the end of a complete
|
||||||
|
install and nowhere else. A fenced session takes no step, publishes nothing, captures
|
||||||
|
nothing and holds no artifact handle.
|
||||||
|
- **Nothing old crosses.** The fence drops every media handle; the restore imports fresh
|
||||||
|
artifacts; the environment re-renders its pending sensor pipeline from recorded
|
||||||
|
reconstruction inputs; and the new epoch's first audio chunk resumes the preserved sample
|
||||||
|
position and marks the discontinuity.
|
||||||
|
- **Epoch metadata in a trace.** `scope.epoch`, the batch id and every task event id are
|
||||||
|
derived from the epoch, so a resumed run's behaviour is compared through
|
||||||
|
`EpochRebase`, which rewrites exactly those and fails on anything it does not recognise.
|
||||||
|
|
||||||
## Where this crate narrows or adds to the contract crate
|
## Where this crate narrows or adds to the contract crate
|
||||||
|
|
||||||
- **Required views.** `WorldObservation::validate_against` checks the views a result carries
|
- **Required views.** `WorldObservation::validate_against` checks the views a result carries
|
||||||
|
|
@ -220,9 +259,9 @@ them. `implementation.md` sequences those after this slice and together with eac
|
||||||
|
|
||||||
- **Fake workers.** There is no neural model and no emulator. What is modelled exactly is the
|
- **Fake workers.** There is no neural model and no emulator. What is modelled exactly is the
|
||||||
ordering, the identity rules and the retry rules, not any numerical behaviour.
|
ordering, the identity rules and the retry rules, not any numerical behaviour.
|
||||||
- **No state methods.** `State.Capture`, `State.StageRestore` and `State.ActivateRestore` are
|
- **One environment, one task.** A checkpoint records the composition it was taken from, and a
|
||||||
STATE-01. The phase machine has their edges (`Capturing`, `Restoring`) and the workers do not
|
restore refuses one taken under another backend, content, patch, controller or parser
|
||||||
advertise them as implemented methods.
|
identity. It does not migrate between compositions, and it does not try.
|
||||||
- **No audience input.** The admitted pre-step stimulation list exists and is always empty.
|
- **No audience input.** The admitted pre-step stimulation list exists and is always empty.
|
||||||
- **One descriptor revision.** A revision changes when the composition does, and the only
|
- **One descriptor revision.** A revision changes when the composition does, and the only
|
||||||
in-session path to that is a group restore into a fresh epoch, which is STATE-01's. The
|
in-session path to that is a group restore into a fresh epoch, which is STATE-01's. The
|
||||||
|
|
@ -291,11 +330,13 @@ The three integration suites do not all run over both transports, and cannot:
|
||||||
- `tests/processes.rs` runs over the Unix socket only, in all three execution modes. A
|
- `tests/processes.rs` runs over the Unix socket only, in all three execution modes. A
|
||||||
participant in a process of its own has no in-memory transport to reach the router by, so
|
participant in a process of its own has no in-memory transport to reach the router by, so
|
||||||
the mode is the axis that suite varies and the transport is fixed.
|
the mode is the axis that suite varies and the transport is fixed.
|
||||||
- `tests/media.rs` and `tests/publishing.rs` run over both transports, and each also generates
|
- `tests/media.rs`, `tests/state.rs` and `tests/publishing.rs` run over both transports *and*
|
||||||
a subset once per execution mode. The publication boundary lives in the coordinator, so
|
in the execution modes: each acceptance body is written once and registered twice, by
|
||||||
unlike the render counter and the sensor log it crosses no process boundary and stays fully
|
`both_transports!` in the in-process composition and by `all_modes!` over the socket.
|
||||||
observable in all three modes; `the_publication_boundary_holds_in_every_execution_mode`
|
`tests/publishing.rs` registers a subset that way rather than all of it, because the
|
||||||
asserts that rather than assuming it.
|
publication boundary lives in the coordinator: unlike the render counter and the sensor log
|
||||||
|
it crosses no process boundary and stays fully observable in all three modes, which
|
||||||
|
`the_publication_boundary_holds_in_every_execution_mode` asserts rather than assumes.
|
||||||
|
|
||||||
- `tests/session.rs`: one world advance per complete batch; every agent Prepared before the
|
- `tests/session.rs`: one world advance per complete batch; every agent Prepared before the
|
||||||
advance; one task evaluation per transition; every agent committed before the next Prepare or
|
advance; one task evaluation per transition; every agent committed before the next Prepare or
|
||||||
|
|
@ -320,6 +361,14 @@ The three integration suites do not all run over both transports, and cannot:
|
||||||
committed action being the transition that just ended, one snapshot carrying every agent,
|
committed action being the transition that just ended, one snapshot carrying every agent,
|
||||||
application-owned state and cues, a held event batch, and the read-only query service --
|
application-owned state and cues, a held event batch, and the read-only query service --
|
||||||
plus the first two generated once per execution mode by `all_modes!`.
|
plus the first two generated once per execution mode by `all_modes!`.
|
||||||
|
- `tests/state.rs`: the STATE-01 acceptance bullets -- an uninterrupted run and a resumed run
|
||||||
|
committing the same behaviour once the epoch metadata is rebased, a corrupt payload failing
|
||||||
|
the install as a group for every participant and for the coordinator's own ledger, a lost
|
||||||
|
save reply and an uncommitted store manifest both leaving the durable mark where it was, a
|
||||||
|
refused activation resuming no part of the world, the capture queue staying bounded under a
|
||||||
|
stalled writer, and old media and another parser's state failing to cross a recovery --
|
||||||
|
plus the once-only restore token, the superseded replaceable capture, and the fence that
|
||||||
|
lifts only through a complete restore.
|
||||||
- `tests/failures.rs`: a duplicate Prepare after a lost reply; a duplicate Commit; the same
|
- `tests/failures.rs`: a duplicate Prepare after a lost reply; a duplicate Commit; the same
|
||||||
batch with altered controls; a lost Advance result; a cached artifact consumed by its first
|
batch with altered controls; a lost Advance result; a cached artifact consumed by its first
|
||||||
caller; one Commit failing after another succeeded; a replaced registration; a reply from
|
caller; one Commit failing after another succeeded; a replaced registration; a reply from
|
||||||
|
|
|
||||||
|
|
@ -7,7 +7,7 @@
|
||||||
//! stimulation, then reinforces once, and executes no tick at all. Every mutating step bumps
|
//! stimulation, then reinforces once, and executes no tick at all. Every mutating step bumps
|
||||||
//! one counter, which is how a test proves a duplicate request changed nothing.
|
//! one counter, which is how a test proves a duplicate request changed nothing.
|
||||||
|
|
||||||
use std::collections::BTreeMap;
|
use std::collections::{BTreeMap, BTreeSet};
|
||||||
|
|
||||||
use serde_json::Value;
|
use serde_json::Value;
|
||||||
|
|
||||||
|
|
@ -193,6 +193,12 @@ pub struct AgentFaults {
|
||||||
pub prepare_delay_ms: u64,
|
pub prepare_delay_ms: u64,
|
||||||
/// Hold `Agent.Commit` open for this long.
|
/// Hold `Agent.Commit` open for this long.
|
||||||
pub commit_delay_ms: u64,
|
pub commit_delay_ms: u64,
|
||||||
|
/// Refuse `State.StageRestore`, so a group install meets one participant that will not
|
||||||
|
/// validate while the others already have.
|
||||||
|
pub fail_stage_restore: bool,
|
||||||
|
/// Refuse `State.ActivateRestore` after this worker has already staged, so a group meets
|
||||||
|
/// a failure halfway through activation.
|
||||||
|
pub fail_activate_restore: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// One fake agent worker's configuration.
|
/// One fake agent worker's configuration.
|
||||||
|
|
@ -238,6 +244,10 @@ pub struct FakeAgentWorker {
|
||||||
context: Option<TypedValue>,
|
context: Option<TypedValue>,
|
||||||
context_digest: Option<Digest>,
|
context_digest: Option<Digest>,
|
||||||
prepared: Option<(DomainRequestId, PreparedDecision)>,
|
prepared: Option<(DomainRequestId, PreparedDecision)>,
|
||||||
|
/// A validated replacement state that the live session cannot see yet.
|
||||||
|
staged: Option<StagedAgent>,
|
||||||
|
/// Restore tokens this worker has activated. A token activates once.
|
||||||
|
activated: BTreeSet<Id>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl FakeAgentWorker {
|
impl FakeAgentWorker {
|
||||||
|
|
@ -252,10 +262,17 @@ impl FakeAgentWorker {
|
||||||
context: None,
|
context: None,
|
||||||
context_digest: None,
|
context_digest: None,
|
||||||
prepared: None,
|
prepared: None,
|
||||||
|
staged: None,
|
||||||
|
activated: BTreeSet::new(),
|
||||||
config,
|
config,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// True while a validated replacement state is staged and not yet activated.
|
||||||
|
pub fn has_staged_restore(&self) -> bool {
|
||||||
|
self.staged.is_some()
|
||||||
|
}
|
||||||
|
|
||||||
pub fn status(&self) -> StatusCell {
|
pub fn status(&self) -> StatusCell {
|
||||||
self.status.clone()
|
self.status.clone()
|
||||||
}
|
}
|
||||||
|
|
@ -666,7 +683,11 @@ impl WorkerEndpoint for FakeAgentWorker {
|
||||||
}
|
}
|
||||||
|
|
||||||
fn capabilities(&self) -> Vec<Id> {
|
fn capabilities(&self) -> Vec<Id> {
|
||||||
vec![id("agent-step-v1"), id("pixel-observation-v1")]
|
vec![
|
||||||
|
id("agent-step-v1"),
|
||||||
|
id("pixel-observation-v1"),
|
||||||
|
id(crate::state::CHECKPOINT_CAPABILITY),
|
||||||
|
]
|
||||||
}
|
}
|
||||||
|
|
||||||
fn status_cell(&self) -> StatusCell {
|
fn status_cell(&self) -> StatusCell {
|
||||||
|
|
@ -678,7 +699,14 @@ impl WorkerEndpoint for FakeAgentWorker {
|
||||||
}
|
}
|
||||||
|
|
||||||
fn methods(&self) -> Vec<&'static str> {
|
fn methods(&self) -> Vec<&'static str> {
|
||||||
vec!["Agent.Initialize", "Agent.Prepare", "Agent.Commit"]
|
vec![
|
||||||
|
"Agent.Initialize",
|
||||||
|
"Agent.Prepare",
|
||||||
|
"Agent.Commit",
|
||||||
|
"State.Capture",
|
||||||
|
"State.StageRestore",
|
||||||
|
"State.ActivateRestore",
|
||||||
|
]
|
||||||
}
|
}
|
||||||
|
|
||||||
fn handle<'a>(&'a mut self, ctx: HandlerCtx<'a>) -> BoxFuture<'a, DomainResult<HandlerReply>> {
|
fn handle<'a>(&'a mut self, ctx: HandlerCtx<'a>) -> BoxFuture<'a, DomainResult<HandlerReply>> {
|
||||||
|
|
@ -687,6 +715,9 @@ impl WorkerEndpoint for FakeAgentWorker {
|
||||||
"Agent.Initialize" => self.initialize(&ctx).await,
|
"Agent.Initialize" => self.initialize(&ctx).await,
|
||||||
"Agent.Prepare" => self.prepare(&ctx).await,
|
"Agent.Prepare" => self.prepare(&ctx).await,
|
||||||
"Agent.Commit" => self.commit(&ctx).await,
|
"Agent.Commit" => self.commit(&ctx).await,
|
||||||
|
"State.Capture" => self.state_capture(&ctx).await,
|
||||||
|
"State.StageRestore" => self.state_stage_restore(&ctx).await,
|
||||||
|
"State.ActivateRestore" => self.state_activate_restore(&ctx).await,
|
||||||
other => Err(DomainError::before(
|
other => Err(DomainError::before(
|
||||||
ErrorCode::Unsupported,
|
ErrorCode::Unsupported,
|
||||||
format!("{other} is not an agent method"),
|
format!("{other} is not an agent method"),
|
||||||
|
|
@ -699,7 +730,12 @@ impl WorkerEndpoint for FakeAgentWorker {
|
||||||
/// The retention class table an agent endpoint follows, for a caller that wants it.
|
/// The retention class table an agent endpoint follows, for a caller that wants it.
|
||||||
pub fn agent_op_class(method: &str) -> Option<OpClass> {
|
pub fn agent_op_class(method: &str) -> Option<OpClass> {
|
||||||
match method {
|
match method {
|
||||||
"Agent.Initialize" => Some(OpClass::Lifecycle),
|
// `ipc-v1` section 5: lifecycle *and capture* replies are retained until
|
||||||
|
// `Worker.Acknowledge`, which is also what lets a duplicate restore request replay
|
||||||
|
// its cached reply rather than staging or activating twice.
|
||||||
|
"Agent.Initialize" | "State.Capture" | "State.StageRestore" | "State.ActivateRestore" => {
|
||||||
|
Some(OpClass::Lifecycle)
|
||||||
|
}
|
||||||
"Agent.Prepare" | "Agent.Commit" => Some(OpClass::StepMutation),
|
"Agent.Prepare" | "Agent.Commit" => Some(OpClass::StepMutation),
|
||||||
_ => None,
|
_ => None,
|
||||||
}
|
}
|
||||||
|
|
@ -769,3 +805,532 @@ pub fn synthetic_profile(agent_id: &Id, tick_duration: &RationalNs, warmup_ticks
|
||||||
|
|
||||||
/// The per-agent contexts a bootstrap produced, keyed by agent id.
|
/// The per-agent contexts a bootstrap produced, keyed by agent id.
|
||||||
pub type Contexts = BTreeMap<Id, TypedValue>;
|
pub type Contexts = BTreeMap<Id, TypedValue>;
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------------------------------
|
||||||
|
// STATE-01: capture and restore
|
||||||
|
|
||||||
|
/// The numerical model version this worker implements. It is part of a capture's
|
||||||
|
/// compatibility identity: the same profile and seed under another model is not the same
|
||||||
|
/// state (`workers-v1` section 2).
|
||||||
|
pub const MODEL_VERSION: &str = "fake-lcg-v1";
|
||||||
|
|
||||||
|
/// The plasticity rule version, for the same reason.
|
||||||
|
pub const PLASTICITY_VERSION: &str = "fake-reinforce-v1";
|
||||||
|
|
||||||
|
/// The version this payload layout is written and read under.
|
||||||
|
pub const AGENT_PAYLOAD_VERSION: u64 = 1;
|
||||||
|
|
||||||
|
/// The dataset identity a synthetic agent resolves.
|
||||||
|
///
|
||||||
|
/// There is no connectome dataset behind this worker, and a checkpoint says so with a stable
|
||||||
|
/// identity rather than omitting the field: "no dataset" has to be distinguishable from "the
|
||||||
|
/// dataset was not recorded".
|
||||||
|
pub fn dataset_digest() -> Digest {
|
||||||
|
digest_of_bytes(b"fly-session/no-dataset-v1")
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The capture compatibility digest of one agent (`workers-v1` section 2).
|
||||||
|
///
|
||||||
|
/// The profile digest identifies the profile definition; this additionally covers the
|
||||||
|
/// resolved seed, the numerical model version and the plasticity rule, because two agents
|
||||||
|
/// with the same profile digest and different seeds hold state that is not interchangeable.
|
||||||
|
/// Every field it covers is one the checkpoint manifest already records in that agent's row,
|
||||||
|
/// so a restore derives the expected digest from the manifest rather than from the payload it
|
||||||
|
/// is about to validate.
|
||||||
|
pub fn agent_compatibility_digest(
|
||||||
|
agent_id: &Id,
|
||||||
|
profile_digest: &Digest,
|
||||||
|
dataset_digest: &Digest,
|
||||||
|
model_version: &str,
|
||||||
|
plasticity_version: &str,
|
||||||
|
seed: i32,
|
||||||
|
) -> Digest {
|
||||||
|
let value = serde_json::json!({
|
||||||
|
"agentId": agent_id.as_str(),
|
||||||
|
"profileDigest": profile_digest.as_str(),
|
||||||
|
"datasetDigest": dataset_digest.as_str(),
|
||||||
|
"modelVersion": model_version,
|
||||||
|
"plasticityVersion": plasticity_version,
|
||||||
|
"seed": seed,
|
||||||
|
});
|
||||||
|
digest_of(&value).expect("an agent compatibility block canonicalizes")
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FakeModel {
|
||||||
|
/// Every field of the model, so a resumed agent is this agent and not a fresh one.
|
||||||
|
fn capture(&self) -> Value {
|
||||||
|
serde_json::json!({
|
||||||
|
"seed": self.seed,
|
||||||
|
"state": self.state.to_string(),
|
||||||
|
"mutations": self.mutations.to_string(),
|
||||||
|
"ticks": self.ticks.to_string(),
|
||||||
|
"stimulations": self.stimulations.to_string(),
|
||||||
|
"reinforcements": self.reinforcements.to_string(),
|
||||||
|
"learningEnabled": self.learning_enabled,
|
||||||
|
"learningUpdates": self.learning_updates.to_string(),
|
||||||
|
"learningChanged": self.learning_changed.to_string(),
|
||||||
|
"lastSignal": self.last_signal,
|
||||||
|
"inputValue": self.input_value.to_string(),
|
||||||
|
"inputInstalls": self.input_installs.to_string(),
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn restored(value: &Value) -> DomainResult<FakeModel> {
|
||||||
|
let number = |key: &str| -> DomainResult<u64> {
|
||||||
|
value
|
||||||
|
.get(key)
|
||||||
|
.and_then(Value::as_str)
|
||||||
|
.ok_or_else(|| incompatible(format!("the agent payload has no {key}")))?
|
||||||
|
.parse::<u64>()
|
||||||
|
.map_err(|_| incompatible(format!("the agent payload's {key} is not a U64")))
|
||||||
|
};
|
||||||
|
let seed = value
|
||||||
|
.get("seed")
|
||||||
|
.and_then(Value::as_i64)
|
||||||
|
.and_then(|v| i32::try_from(v).ok())
|
||||||
|
.ok_or_else(|| incompatible("the agent payload has no seed"))?;
|
||||||
|
let input_value = value
|
||||||
|
.get("inputValue")
|
||||||
|
.and_then(Value::as_str)
|
||||||
|
.ok_or_else(|| incompatible("the agent payload has no inputValue"))?
|
||||||
|
.parse::<i64>()
|
||||||
|
.map_err(|_| incompatible("the agent payload's inputValue is not an integer"))?;
|
||||||
|
let last_signal = value
|
||||||
|
.get("lastSignal")
|
||||||
|
.and_then(Value::as_f64)
|
||||||
|
.filter(|v| v.is_finite())
|
||||||
|
.ok_or_else(|| incompatible("the agent payload's lastSignal is not finite"))?;
|
||||||
|
let learning_enabled = value
|
||||||
|
.get("learningEnabled")
|
||||||
|
.and_then(Value::as_bool)
|
||||||
|
.ok_or_else(|| incompatible("the agent payload has no learningEnabled"))?;
|
||||||
|
Ok(FakeModel {
|
||||||
|
seed,
|
||||||
|
state: number("state")?,
|
||||||
|
mutations: number("mutations")?,
|
||||||
|
ticks: number("ticks")?,
|
||||||
|
stimulations: number("stimulations")?,
|
||||||
|
reinforcements: number("reinforcements")?,
|
||||||
|
learning_enabled,
|
||||||
|
learning_updates: number("learningUpdates")?,
|
||||||
|
learning_changed: number("learningChanged")?,
|
||||||
|
last_signal,
|
||||||
|
input_value,
|
||||||
|
input_installs: number("inputInstalls")?,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn incompatible(message: impl std::fmt::Display) -> DomainError {
|
||||||
|
DomainError::before(ErrorCode::IncompatibleState, message)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One staged restore, held outside the live agent until it is activated.
|
||||||
|
struct StagedAgent {
|
||||||
|
token: Id,
|
||||||
|
checkpoint_id: Id,
|
||||||
|
scope: Scope,
|
||||||
|
model: FakeModel,
|
||||||
|
accumulator: TickAccumulator,
|
||||||
|
context: TypedValue,
|
||||||
|
profile: AssetRef,
|
||||||
|
committed_step: u64,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl FakeAgentWorker {
|
||||||
|
/// This worker's own compatibility identity, from its configuration and a resolved seed.
|
||||||
|
fn compatibility_digest(&self, profile: &AssetRef, seed: i32) -> Digest {
|
||||||
|
agent_compatibility_digest(
|
||||||
|
&self.config.agent_id,
|
||||||
|
&profile.digest,
|
||||||
|
&dataset_digest(),
|
||||||
|
MODEL_VERSION,
|
||||||
|
PLASTICITY_VERSION,
|
||||||
|
seed,
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `State.Capture`: an immutable snapshot of this agent at its committed boundary.
|
||||||
|
///
|
||||||
|
/// It is allowed at `Ready(k)` only. A Prepared agent holds half a transition, and there
|
||||||
|
/// is no coherent boundary to file that under.
|
||||||
|
async fn state_capture(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult<HandlerReply> {
|
||||||
|
let scope = ctx.scope()?.clone();
|
||||||
|
self.check_epoch(&scope)?;
|
||||||
|
let AgentPhase::Ready(k) = self.phase.clone() else {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::InvalidPhase,
|
||||||
|
format!(
|
||||||
|
"State.Capture needs a quiescent Ready(k); this worker is {:?}",
|
||||||
|
self.phase
|
||||||
|
),
|
||||||
|
));
|
||||||
|
};
|
||||||
|
if scope.step != k {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
if scope.step < k { ErrorCode::StaleStep } else { ErrorCode::FutureStep },
|
||||||
|
"State.Capture names a boundary this worker is not at",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let params: CaptureParams = ctx.params()?;
|
||||||
|
let profile = self.profile.clone().expect("initialized");
|
||||||
|
let context = self.context.clone().expect("initialized");
|
||||||
|
let accumulator = self.accumulator.as_ref().expect("initialized");
|
||||||
|
let previous = self.status.state();
|
||||||
|
self.status.set_state(WorkerState::Capturing);
|
||||||
|
let payload = serde_json::json!({
|
||||||
|
"payloadVersion": AGENT_PAYLOAD_VERSION,
|
||||||
|
"kind": "agent",
|
||||||
|
"agentId": self.config.agent_id.as_str(),
|
||||||
|
"checkpointId": params.checkpoint_id.as_str(),
|
||||||
|
"sourceScope": scope.to_json(),
|
||||||
|
"committedStep": k.to_string(),
|
||||||
|
"profile": profile.to_json(),
|
||||||
|
"modelVersion": MODEL_VERSION,
|
||||||
|
"plasticityVersion": PLASTICITY_VERSION,
|
||||||
|
"datasetDigest": dataset_digest().as_str(),
|
||||||
|
"model": self.model.capture(),
|
||||||
|
"accumulator": {
|
||||||
|
"tickDuration": accumulator.tick_duration().to_json(),
|
||||||
|
"remainder": accumulator.remainder().to_json(),
|
||||||
|
"executedTicks": accumulator.executed_ticks().to_string(),
|
||||||
|
"warmupOffset": accumulator.warmup_offset().to_string(),
|
||||||
|
},
|
||||||
|
"context": context.to_json(),
|
||||||
|
});
|
||||||
|
let bytes = canonicalize(&payload)
|
||||||
|
.map_err(|e| DomainError::invalid(format!("State.Capture: {}", e.0)))?
|
||||||
|
.into_bytes();
|
||||||
|
let digest = digest_of_bytes(&bytes);
|
||||||
|
let artifact = crate::state::seal_payload(ctx.client, &bytes, &digest).await?;
|
||||||
|
// Capture is a read of the model, not a mutation of it: nothing above changed a
|
||||||
|
// counter, and the worker goes back to the boundary it was already at.
|
||||||
|
self.status.set_state(previous);
|
||||||
|
let result = CaptureResult {
|
||||||
|
checkpoint_id: params.checkpoint_id,
|
||||||
|
boundary: k,
|
||||||
|
compatibility_digest: self.compatibility_digest(&profile, self.model.seed()),
|
||||||
|
payload: artifact.reference().clone(),
|
||||||
|
};
|
||||||
|
Ok(HandlerReply::with_artifacts(
|
||||||
|
object(result.to_json()),
|
||||||
|
vec![(crate::state::PAYLOAD_ATTACHMENT.to_owned(), artifact)],
|
||||||
|
))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `State.StageRestore`: validate a replacement state into a staging slot.
|
||||||
|
///
|
||||||
|
/// Nothing the live session can see changes here, and the worker keeps whatever state it
|
||||||
|
/// had. It is allowed on an uninitialized replacement or a quiescent worker only; a
|
||||||
|
/// failed one is neither, which is why a group that failed is replaced rather than
|
||||||
|
/// reused.
|
||||||
|
async fn state_stage_restore(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult<HandlerReply> {
|
||||||
|
let scope = ctx.scope()?.clone();
|
||||||
|
if scope.session_id != self.config.session_id {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::IdentityMismatch,
|
||||||
|
"this worker belongs to another session",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
match &self.phase {
|
||||||
|
AgentPhase::Uninitialized | AgentPhase::Ready(_) => {}
|
||||||
|
other => {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::InvalidPhase,
|
||||||
|
format!(
|
||||||
|
"State.StageRestore needs an uninitialized replacement or a quiescent \
|
||||||
|
worker; this worker is {other:?}"
|
||||||
|
),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if let Some(epoch) = &self.epoch
|
||||||
|
&& *epoch == scope.epoch
|
||||||
|
{
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::StaleEpoch,
|
||||||
|
"State.StageRestore proposes the epoch this worker is already running",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let params: StageRestoreParams = ctx.params()?;
|
||||||
|
if params.source_scope.step != scope.step {
|
||||||
|
return Err(DomainError::invalid(
|
||||||
|
"State.StageRestore's scope step must be the source boundary",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let artifact = ctx.artifact(crate::state::PAYLOAD_ATTACHMENT)?;
|
||||||
|
if artifact.reference() != ¶ms.payload {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::BufferInvalid,
|
||||||
|
"the staged payload attachment is not the artifact the request names",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let bytes = artifact.read_all().await.map_err(|e| {
|
||||||
|
DomainError::before(
|
||||||
|
ErrorCode::BufferInvalid,
|
||||||
|
format!("the staged payload could not be read: {}", e.message),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
let declared = params
|
||||||
|
.payload
|
||||||
|
.digest
|
||||||
|
.clone()
|
||||||
|
.ok_or_else(|| incompatible("a checkpoint payload must carry a content digest"))?;
|
||||||
|
let actual = digest_of_bytes(&bytes);
|
||||||
|
if actual != declared || bytes.len() as u64 != params.payload.byte_length {
|
||||||
|
return Err(incompatible(
|
||||||
|
"the staged payload is not the content the request declares",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let value: Value = serde_json::from_slice(&bytes)
|
||||||
|
.map_err(|e| DomainError::invalid(format!("the staged payload is not JSON: {e}")))?;
|
||||||
|
let text = |key: &str| -> DomainResult<String> {
|
||||||
|
value
|
||||||
|
.get(key)
|
||||||
|
.and_then(Value::as_str)
|
||||||
|
.map(str::to_owned)
|
||||||
|
.ok_or_else(|| incompatible(format!("the agent payload has no {key}")))
|
||||||
|
};
|
||||||
|
if value.get("payloadVersion").and_then(Value::as_u64) != Some(AGENT_PAYLOAD_VERSION) {
|
||||||
|
return Err(incompatible("the agent payload is another payload version"));
|
||||||
|
}
|
||||||
|
if text("kind")? != "agent" {
|
||||||
|
return Err(incompatible("this payload is not an agent's state"));
|
||||||
|
}
|
||||||
|
if text("agentId")? != self.config.agent_id {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::IdentityMismatch,
|
||||||
|
"the staged payload belongs to another agent",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
if text("checkpointId")? != params.checkpoint_id {
|
||||||
|
return Err(incompatible("the staged payload belongs to another checkpoint"));
|
||||||
|
}
|
||||||
|
if text("modelVersion")? != MODEL_VERSION || text("plasticityVersion")? != PLASTICITY_VERSION
|
||||||
|
{
|
||||||
|
return Err(incompatible(
|
||||||
|
"the staged payload was captured under another numerical model",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let source_scope = Scope::from_json(
|
||||||
|
value
|
||||||
|
.get("sourceScope")
|
||||||
|
.ok_or_else(|| incompatible("the agent payload has no sourceScope"))?,
|
||||||
|
)
|
||||||
|
.map_err(|e| incompatible(format!("the agent payload's sourceScope: {}", e.0)))?;
|
||||||
|
if source_scope != params.source_scope {
|
||||||
|
return Err(incompatible(
|
||||||
|
"the staged payload was captured at another source scope",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let committed_step: u64 = text("committedStep")?
|
||||||
|
.parse()
|
||||||
|
.map_err(|_| incompatible("the agent payload's committedStep is not a U64"))?;
|
||||||
|
if committed_step != params.source_scope.step {
|
||||||
|
return Err(incompatible(
|
||||||
|
"the staged payload's committed step is not the source boundary",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let profile = AssetRef::from_json(
|
||||||
|
value
|
||||||
|
.get("profile")
|
||||||
|
.ok_or_else(|| incompatible("the agent payload has no profile"))?,
|
||||||
|
)
|
||||||
|
.map_err(|e| incompatible(format!("the agent payload's profile: {}", e.0)))?;
|
||||||
|
let model = FakeModel::restored(
|
||||||
|
value
|
||||||
|
.get("model")
|
||||||
|
.ok_or_else(|| incompatible("the agent payload has no model"))?,
|
||||||
|
)?;
|
||||||
|
// The compatibility digest is recomputed from this worker's own configuration and the
|
||||||
|
// identity the payload declares. A capture of the same profile under another seed, or
|
||||||
|
// of another agent's brain, fails here and never reaches activation.
|
||||||
|
let computed = self.compatibility_digest(&profile, model.seed());
|
||||||
|
if computed != params.compatibility_digest {
|
||||||
|
return Err(incompatible(format!(
|
||||||
|
"the staged state's compatibility {computed} is not the {} the restore \
|
||||||
|
requires",
|
||||||
|
params.compatibility_digest
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
let accumulator_value = value
|
||||||
|
.get("accumulator")
|
||||||
|
.ok_or_else(|| incompatible("the agent payload has no accumulator"))?;
|
||||||
|
let rational = |key: &str| -> DomainResult<RationalNs> {
|
||||||
|
RationalNs::from_json(
|
||||||
|
accumulator_value
|
||||||
|
.get(key)
|
||||||
|
.ok_or_else(|| incompatible(format!("the accumulator has no {key}")))?,
|
||||||
|
)
|
||||||
|
.map_err(|e| incompatible(format!("the accumulator's {key}: {}", e.0)))
|
||||||
|
};
|
||||||
|
let counter = |key: &str| -> DomainResult<u64> {
|
||||||
|
accumulator_value
|
||||||
|
.get(key)
|
||||||
|
.and_then(Value::as_str)
|
||||||
|
.ok_or_else(|| incompatible(format!("the accumulator has no {key}")))?
|
||||||
|
.parse::<u64>()
|
||||||
|
.map_err(|_| incompatible(format!("the accumulator's {key} is not a U64")))
|
||||||
|
};
|
||||||
|
let tick_duration = rational("tickDuration")?;
|
||||||
|
if tick_duration != self.config.tick_duration {
|
||||||
|
return Err(incompatible(
|
||||||
|
"the staged state was captured at another model tick duration",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let accumulator = TickAccumulator::restored(
|
||||||
|
tick_duration,
|
||||||
|
rational("remainder")?,
|
||||||
|
counter("executedTicks")?,
|
||||||
|
counter("warmupOffset")?,
|
||||||
|
)
|
||||||
|
.map_err(incompatible)?;
|
||||||
|
let context = TypedValue::from_json(
|
||||||
|
value
|
||||||
|
.get("context")
|
||||||
|
.ok_or_else(|| incompatible("the agent payload has no context"))?,
|
||||||
|
)
|
||||||
|
.map_err(|e| incompatible(format!("the agent payload's context: {}", e.0)))?;
|
||||||
|
FakeAgentWorker::available_actions(&context)?;
|
||||||
|
|
||||||
|
if self.config.faults.fail_stage_restore {
|
||||||
|
// The row where a group validates three participants and the fourth does not.
|
||||||
|
// Nothing is staged here and nothing is staged anywhere else either: the
|
||||||
|
// coordinator abandons the whole install.
|
||||||
|
return Err(incompatible(
|
||||||
|
"injected staging refusal: this participant's replacement state does not \
|
||||||
|
validate",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
// One staged restore at a time. A second proposal replaces nothing silently.
|
||||||
|
if let Some(staged) = &self.staged {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::Conflict,
|
||||||
|
format!(
|
||||||
|
"this worker already holds the staged restore {} for checkpoint {}",
|
||||||
|
staged.token, staged.checkpoint_id
|
||||||
|
),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let token = restore_token(¶ms.checkpoint_id, &scope, &actual, &self.config.incarnation_id);
|
||||||
|
if self.activated.contains(&token) {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::Conflict,
|
||||||
|
"this exact restore was already activated on this worker",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
self.staged = Some(StagedAgent {
|
||||||
|
token: token.clone(),
|
||||||
|
checkpoint_id: params.checkpoint_id.clone(),
|
||||||
|
scope: scope.clone(),
|
||||||
|
model,
|
||||||
|
accumulator,
|
||||||
|
context,
|
||||||
|
profile,
|
||||||
|
committed_step,
|
||||||
|
});
|
||||||
|
self.status.set_state(WorkerState::StagedRestore);
|
||||||
|
let result = StageRestoreResult {
|
||||||
|
checkpoint_id: params.checkpoint_id,
|
||||||
|
restore_token: token,
|
||||||
|
};
|
||||||
|
Ok(HandlerReply::from(&result))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `State.ActivateRestore`: install the staged state under its new scope, without a tick.
|
||||||
|
///
|
||||||
|
/// The token activates once. A duplicate domain request replays the cached reply through
|
||||||
|
/// the shell's result cache; a fresh request naming an already activated token is a
|
||||||
|
/// conflict, which is what stops a second group from being resumed from the same bytes.
|
||||||
|
async fn state_activate_restore(
|
||||||
|
&mut self,
|
||||||
|
ctx: &HandlerCtx<'_>,
|
||||||
|
) -> DomainResult<HandlerReply> {
|
||||||
|
let params: ActivateRestoreParams = ctx.params()?;
|
||||||
|
if self.activated.contains(¶ms.restore_token) {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::Conflict,
|
||||||
|
"this restore token has already been activated",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let Some(staged) = self.staged.take() else {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::InvalidPhase,
|
||||||
|
"this worker holds no staged restore",
|
||||||
|
));
|
||||||
|
};
|
||||||
|
if staged.token != params.restore_token {
|
||||||
|
// Put it back: naming another token is not a reason to discard this one.
|
||||||
|
let token = staged.token.clone();
|
||||||
|
self.staged = Some(staged);
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::IdentityMismatch,
|
||||||
|
format!("this worker's staged restore is {token}, not {}", params.restore_token),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
if self.config.faults.fail_activate_restore {
|
||||||
|
let token = staged.token.clone();
|
||||||
|
self.staged = Some(staged);
|
||||||
|
self.status.set_state(WorkerState::Failed);
|
||||||
|
return Err(DomainError::new(
|
||||||
|
ErrorCode::BackendFailure,
|
||||||
|
format!("injected activation failure; {token} stays staged and unresumed"),
|
||||||
|
MutationCertainty::None,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
self.status.set_state(WorkerState::Restoring);
|
||||||
|
let StagedAgent {
|
||||||
|
token,
|
||||||
|
checkpoint_id,
|
||||||
|
scope,
|
||||||
|
model,
|
||||||
|
accumulator,
|
||||||
|
context,
|
||||||
|
profile,
|
||||||
|
committed_step,
|
||||||
|
} = staged;
|
||||||
|
self.model = model;
|
||||||
|
self.accumulator = Some(accumulator);
|
||||||
|
self.context_digest = Some(context.digest());
|
||||||
|
self.context = Some(context);
|
||||||
|
self.profile = Some(profile);
|
||||||
|
self.epoch = Some(scope.epoch.clone());
|
||||||
|
self.prepared = None;
|
||||||
|
self.phase = AgentPhase::Ready(committed_step);
|
||||||
|
self.activated.insert(token);
|
||||||
|
self.status.set_state(WorkerState::Ready);
|
||||||
|
self.status.set_scope(Some(scope_at(
|
||||||
|
&scope.session_id,
|
||||||
|
&scope.epoch,
|
||||||
|
committed_step,
|
||||||
|
)));
|
||||||
|
self.status.advance_to(self.model.mutations());
|
||||||
|
let result = ActivateRestoreResult {
|
||||||
|
committed_step,
|
||||||
|
checkpoint_id,
|
||||||
|
// An agent returns a null observation; the environment returns the world's.
|
||||||
|
observation: None,
|
||||||
|
};
|
||||||
|
result
|
||||||
|
.validate_for_role(Role::Agent)
|
||||||
|
.map_err(|e| DomainError::invalid(e.0))?;
|
||||||
|
Ok(HandlerReply::from(&result))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A restore token bound to the checkpoint, the proposed scope, the payload bytes and the
|
||||||
|
/// worker incarnation staging them.
|
||||||
|
///
|
||||||
|
/// `state-media-v1` section 5 binds a token to scope, payload and checkpoint. Binding it to
|
||||||
|
/// the incarnation as well is what keeps a token minted by a worker that has since been
|
||||||
|
/// replaced from activating anything on its replacement.
|
||||||
|
pub fn restore_token(checkpoint_id: &Id, scope: &Scope, payload_digest: &Digest, incarnation: &Id) -> Id {
|
||||||
|
let digest = digest_of_bytes(
|
||||||
|
format!(
|
||||||
|
"fly-session/restore-token-v1\n{checkpoint_id}\n{}\n{}\n{}\n{payload_digest}\n{incarnation}\n",
|
||||||
|
scope.session_id, scope.epoch, scope.step
|
||||||
|
)
|
||||||
|
.as_bytes(),
|
||||||
|
);
|
||||||
|
parse_id(&format!("rt-{}", &digest[..32])).expect("a hex suffix is an Id")
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -46,8 +46,10 @@ Worker options (agent and environment):
|
||||||
agent: --agent ID --port ID --tick-numerator N --tick-denominator N
|
agent: --agent ID --port ID --tick-numerator N --tick-denominator N
|
||||||
--warmup-ticks N [--prepare-delay-ms N] [--commit-delay-ms N]
|
--warmup-ticks N [--prepare-delay-ms N] [--commit-delay-ms N]
|
||||||
[--fail-commit-at-step N]
|
[--fail-commit-at-step N]
|
||||||
|
[--fail-stage-restore 0|1] [--fail-activate-restore 0|1]
|
||||||
environment: --worker ID --ports p1,p2 --step-numerator N --step-denominator N
|
environment: --worker ID --ports p1,p2 --step-numerator N --step-denominator N
|
||||||
[--advance-delay-ms N] [--omit-view-at-boundary N]
|
[--advance-delay-ms N] [--omit-view-at-boundary N]
|
||||||
|
[--fail-stage-restore 0|1] [--fail-activate-restore 0|1]
|
||||||
|
|
||||||
Measure options:
|
Measure options:
|
||||||
--steps N transitions per run (default 200)
|
--steps N transitions per run (default 200)
|
||||||
|
|
@ -145,6 +147,17 @@ impl Options {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A flag whose value is `0` or `1`. Anything else is an error naming it, so a
|
||||||
|
/// mistyped injection is a failed launch rather than a fault that never fires.
|
||||||
|
fn flag(&self, name: &str) -> Result<bool, String> {
|
||||||
|
match self.0.get(name) {
|
||||||
|
None => Ok(false),
|
||||||
|
Some(value) if value == "0" => Ok(false),
|
||||||
|
Some(value) if value == "1" => Ok(true),
|
||||||
|
Some(value) => Err(format!("--{name}: {value:?} is not 0 or 1")),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
fn opt_u64(&self, name: &str) -> Result<Option<u64>, String> {
|
fn opt_u64(&self, name: &str) -> Result<Option<u64>, String> {
|
||||||
match self.0.get(name) {
|
match self.0.get(name) {
|
||||||
None => Ok(None),
|
None => Ok(None),
|
||||||
|
|
@ -198,6 +211,8 @@ fn serve(role: &str, options: &Options) -> Result<(), String> {
|
||||||
fail_commit_at_step: options.opt_u64(flags::FAIL_COMMIT_AT_STEP)?,
|
fail_commit_at_step: options.opt_u64(flags::FAIL_COMMIT_AT_STEP)?,
|
||||||
prepare_delay_ms: options.u64(flags::PREPARE_DELAY_MS, 0)?,
|
prepare_delay_ms: options.u64(flags::PREPARE_DELAY_MS, 0)?,
|
||||||
commit_delay_ms: options.u64(flags::COMMIT_DELAY_MS, 0)?,
|
commit_delay_ms: options.u64(flags::COMMIT_DELAY_MS, 0)?,
|
||||||
|
fail_stage_restore: options.flag(flags::FAIL_STAGE_RESTORE)?,
|
||||||
|
fail_activate_restore: options.flag(flags::FAIL_ACTIVATE_RESTORE)?,
|
||||||
},
|
},
|
||||||
client_id: client_id.clone(),
|
client_id: client_id.clone(),
|
||||||
service: service.clone(),
|
service: service.clone(),
|
||||||
|
|
@ -220,6 +235,8 @@ fn serve(role: &str, options: &Options) -> Result<(), String> {
|
||||||
omit_audio_at_boundary: options.opt_u64(flags::OMIT_AUDIO_AT_BOUNDARY)?,
|
omit_audio_at_boundary: options.opt_u64(flags::OMIT_AUDIO_AT_BOUNDARY)?,
|
||||||
overlapping_audio_at_boundary: options
|
overlapping_audio_at_boundary: options
|
||||||
.opt_u64(flags::OVERLAPPING_AUDIO_AT_BOUNDARY)?,
|
.opt_u64(flags::OVERLAPPING_AUDIO_AT_BOUNDARY)?,
|
||||||
|
fail_stage_restore: options.flag(flags::FAIL_STAGE_RESTORE)?,
|
||||||
|
fail_activate_restore: options.flag(flags::FAIL_ACTIVATE_RESTORE)?,
|
||||||
},
|
},
|
||||||
client_id: client_id.clone(),
|
client_id: client_id.clone(),
|
||||||
service: service.clone(),
|
service: service.clone(),
|
||||||
|
|
|
||||||
|
|
@ -29,6 +29,30 @@ impl TickAccumulator {
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The exact accumulator a capture recorded.
|
||||||
|
///
|
||||||
|
/// The remainder is restored, never rounded or reset: a resumed agent that started its
|
||||||
|
/// first interval from zero would drift away from the run it is supposed to continue.
|
||||||
|
pub fn restored(
|
||||||
|
tick_duration: RationalNs,
|
||||||
|
remainder: RationalNs,
|
||||||
|
executed_ticks: u64,
|
||||||
|
warmup_offset: u64,
|
||||||
|
) -> Result<TickAccumulator, String> {
|
||||||
|
let mut accumulator = TickAccumulator::new(tick_duration)?;
|
||||||
|
remainder.validate().map_err(|e| e.0)?;
|
||||||
|
if remainder >= tick_duration {
|
||||||
|
return Err("a captured remainder is not below one model tick".to_owned());
|
||||||
|
}
|
||||||
|
if warmup_offset > executed_ticks {
|
||||||
|
return Err("a captured warm-up offset exceeds the executed tick count".to_owned());
|
||||||
|
}
|
||||||
|
accumulator.remainder = remainder;
|
||||||
|
accumulator.executed_ticks = executed_ticks;
|
||||||
|
accumulator.warmup_offset = warmup_offset;
|
||||||
|
Ok(accumulator)
|
||||||
|
}
|
||||||
|
|
||||||
pub fn tick_duration(&self) -> RationalNs {
|
pub fn tick_duration(&self) -> RationalNs {
|
||||||
self.tick_duration
|
self.tick_duration
|
||||||
}
|
}
|
||||||
|
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -13,6 +13,8 @@
|
||||||
|
|
||||||
use std::collections::BTreeSet;
|
use std::collections::BTreeSet;
|
||||||
|
|
||||||
|
use serde_json::Value;
|
||||||
|
|
||||||
use crate::media::{self, AudioSource, RenderCounter, ViewPipeline};
|
use crate::media::{self, AudioSource, RenderCounter, ViewPipeline};
|
||||||
use crate::task::{controller_schema_ref, inspection, inspection_schema};
|
use crate::task::{controller_schema_ref, inspection, inspection_schema};
|
||||||
// `crate::types` is this crate's facade over the shared `fly-session-types` crate; the
|
// `crate::types` is this crate's facade over the shared `fly-session-types` crate; the
|
||||||
|
|
@ -48,6 +50,12 @@ pub struct EnvironmentFaults {
|
||||||
pub omit_audio_at_boundary: Option<u64>,
|
pub omit_audio_at_boundary: Option<u64>,
|
||||||
/// Emit an audio chunk that starts before the previous chunk ended.
|
/// Emit an audio chunk that starts before the previous chunk ended.
|
||||||
pub overlapping_audio_at_boundary: Option<u64>,
|
pub overlapping_audio_at_boundary: Option<u64>,
|
||||||
|
/// Refuse `State.StageRestore`, so a group install meets a participant that will not
|
||||||
|
/// validate.
|
||||||
|
pub fail_stage_restore: bool,
|
||||||
|
/// Refuse `State.ActivateRestore` after staging, so a group meets a failure halfway
|
||||||
|
/// through activation.
|
||||||
|
pub fail_activate_restore: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Clone, Debug)]
|
#[derive(Clone, Debug)]
|
||||||
|
|
@ -86,6 +94,10 @@ pub struct CounterEnvironment {
|
||||||
audio: Option<AudioSource>,
|
audio: Option<AudioSource>,
|
||||||
/// The frame served at the previous boundary, kept only so a fault can serve it again.
|
/// The frame served at the previous boundary, kept only so a fault can serve it again.
|
||||||
previous_view: Option<(ViewRef, flybus::Artifact)>,
|
previous_view: Option<(ViewRef, flybus::Artifact)>,
|
||||||
|
/// A validated replacement world the live session cannot see yet.
|
||||||
|
staged: Option<StagedWorld>,
|
||||||
|
/// Restore tokens this world has activated. A token activates once.
|
||||||
|
activated: BTreeSet<Id>,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl CounterEnvironment {
|
impl CounterEnvironment {
|
||||||
|
|
@ -104,10 +116,17 @@ impl CounterEnvironment {
|
||||||
pipeline: None,
|
pipeline: None,
|
||||||
audio: None,
|
audio: None,
|
||||||
previous_view: None,
|
previous_view: None,
|
||||||
|
staged: None,
|
||||||
|
activated: BTreeSet::new(),
|
||||||
config,
|
config,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// True while a validated replacement world is staged and not yet activated.
|
||||||
|
pub fn has_staged_restore(&self) -> bool {
|
||||||
|
self.staged.is_some()
|
||||||
|
}
|
||||||
|
|
||||||
pub fn status(&self) -> StatusCell {
|
pub fn status(&self) -> StatusCell {
|
||||||
self.status.clone()
|
self.status.clone()
|
||||||
}
|
}
|
||||||
|
|
@ -477,7 +496,7 @@ impl WorkerEndpoint for CounterEnvironment {
|
||||||
vec![
|
vec![
|
||||||
id("world-step-v1"),
|
id("world-step-v1"),
|
||||||
id("pixel-observation-v1"),
|
id("pixel-observation-v1"),
|
||||||
id("checkpoint-v1"),
|
id(crate::state::CHECKPOINT_CAPABILITY),
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -490,7 +509,13 @@ impl WorkerEndpoint for CounterEnvironment {
|
||||||
}
|
}
|
||||||
|
|
||||||
fn methods(&self) -> Vec<&'static str> {
|
fn methods(&self) -> Vec<&'static str> {
|
||||||
vec!["Environment.Initialize", "Environment.Advance"]
|
vec![
|
||||||
|
"Environment.Initialize",
|
||||||
|
"Environment.Advance",
|
||||||
|
"State.Capture",
|
||||||
|
"State.StageRestore",
|
||||||
|
"State.ActivateRestore",
|
||||||
|
]
|
||||||
}
|
}
|
||||||
|
|
||||||
fn handle<'a>(&'a mut self, ctx: HandlerCtx<'a>) -> BoxFuture<'a, DomainResult<HandlerReply>> {
|
fn handle<'a>(&'a mut self, ctx: HandlerCtx<'a>) -> BoxFuture<'a, DomainResult<HandlerReply>> {
|
||||||
|
|
@ -498,6 +523,9 @@ impl WorkerEndpoint for CounterEnvironment {
|
||||||
match ctx.method {
|
match ctx.method {
|
||||||
"Environment.Initialize" => self.initialize(&ctx).await,
|
"Environment.Initialize" => self.initialize(&ctx).await,
|
||||||
"Environment.Advance" => self.advance(&ctx).await,
|
"Environment.Advance" => self.advance(&ctx).await,
|
||||||
|
"State.Capture" => self.state_capture(&ctx).await,
|
||||||
|
"State.StageRestore" => self.state_stage_restore(&ctx).await,
|
||||||
|
"State.ActivateRestore" => self.state_activate_restore(&ctx).await,
|
||||||
other => Err(DomainError::before(
|
other => Err(DomainError::before(
|
||||||
ErrorCode::Unsupported,
|
ErrorCode::Unsupported,
|
||||||
format!("{other} is not an environment method"),
|
format!("{other} is not an environment method"),
|
||||||
|
|
@ -516,3 +544,488 @@ pub fn synthetic_asset(asset_id: &str, body: &str) -> AssetRef {
|
||||||
format: id("fly-config-v1"),
|
format: id("fly-config-v1"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// -------------------------------------------------------------------------------------------
|
||||||
|
// STATE-01: capture and restore
|
||||||
|
|
||||||
|
/// The version this payload layout is written and read under.
|
||||||
|
pub const WORLD_PAYLOAD_VERSION: u64 = 1;
|
||||||
|
|
||||||
|
fn incompatible(message: impl std::fmt::Display) -> DomainError {
|
||||||
|
DomainError::before(ErrorCode::IncompatibleState, message)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One staged restore, held outside the live world until it is activated.
|
||||||
|
struct StagedWorld {
|
||||||
|
token: Id,
|
||||||
|
checkpoint_id: Id,
|
||||||
|
scope: Scope,
|
||||||
|
episode_id: Id,
|
||||||
|
descriptor: EnvironmentDescriptor,
|
||||||
|
boundary: u64,
|
||||||
|
counter: i64,
|
||||||
|
world_time: RationalNs,
|
||||||
|
advances: u64,
|
||||||
|
frames: Vec<(u64, i64)>,
|
||||||
|
audio_next_sample: u64,
|
||||||
|
audio_phase: u64,
|
||||||
|
audio_accumulator: u128,
|
||||||
|
audio_denominator: u128,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl CounterEnvironment {
|
||||||
|
/// `State.Capture`: the world at its committed boundary, including its pending sensor
|
||||||
|
/// pipeline.
|
||||||
|
///
|
||||||
|
/// The pipeline is recorded as reconstruction inputs -- the producing boundary and the
|
||||||
|
/// world counter of every retained frame -- and never as an artifact identity: a
|
||||||
|
/// transient artifact belongs to the router that is running now, and a checkpoint outlives
|
||||||
|
/// it.
|
||||||
|
async fn state_capture(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult<HandlerReply> {
|
||||||
|
let scope = ctx.scope()?.clone();
|
||||||
|
let Some(descriptor) = self.descriptor.clone() else {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::InvalidPhase,
|
||||||
|
"this environment is uninitialized",
|
||||||
|
));
|
||||||
|
};
|
||||||
|
if scope.session_id != self.config.session_id {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::IdentityMismatch,
|
||||||
|
"this environment belongs to another session",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
match &self.epoch {
|
||||||
|
Some(epoch) if *epoch == scope.epoch => {}
|
||||||
|
_ => {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::StaleEpoch,
|
||||||
|
"State.Capture names an epoch this environment has left",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if scope.step != self.boundary {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
if scope.step < self.boundary {
|
||||||
|
ErrorCode::StaleStep
|
||||||
|
} else {
|
||||||
|
ErrorCode::FutureStep
|
||||||
|
},
|
||||||
|
"State.Capture must name the boundary the world is at",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let params: CaptureParams = ctx.params()?;
|
||||||
|
let pipeline = self
|
||||||
|
.pipeline
|
||||||
|
.as_ref()
|
||||||
|
.ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no view pipeline"))?;
|
||||||
|
let audio = self
|
||||||
|
.audio
|
||||||
|
.as_ref()
|
||||||
|
.ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no audio source"))?;
|
||||||
|
let (accumulator, denominator) = audio.accumulator();
|
||||||
|
let previous = self.status.state();
|
||||||
|
self.status.set_state(WorkerState::Capturing);
|
||||||
|
let payload = serde_json::json!({
|
||||||
|
"payloadVersion": WORLD_PAYLOAD_VERSION,
|
||||||
|
"kind": "world",
|
||||||
|
"workerId": self.config.worker_id.as_str(),
|
||||||
|
"checkpointId": params.checkpoint_id.as_str(),
|
||||||
|
"sourceScope": scope.to_json(),
|
||||||
|
"episodeId": self.episode_id.clone().expect("initialized").as_str(),
|
||||||
|
"committedStep": self.boundary.to_string(),
|
||||||
|
"counter": self.counter.to_string(),
|
||||||
|
"worldTime": self.world_time.to_json(),
|
||||||
|
"advances": self.advances.to_string(),
|
||||||
|
"descriptor": descriptor.to_json(),
|
||||||
|
"pipeline": {
|
||||||
|
// The declared delay's whole queue, oldest first.
|
||||||
|
"frames": pipeline
|
||||||
|
.retained()
|
||||||
|
.into_iter()
|
||||||
|
.map(|(boundary, counter)| serde_json::json!({
|
||||||
|
"boundary": boundary.to_string(),
|
||||||
|
"counter": counter.to_string(),
|
||||||
|
}))
|
||||||
|
.collect::<Vec<_>>(),
|
||||||
|
},
|
||||||
|
"audio": {
|
||||||
|
"nextSample": audio.next_sample().to_string(),
|
||||||
|
"phase": audio.phase().to_string(),
|
||||||
|
"accumulator": accumulator.to_string(),
|
||||||
|
"denominator": denominator.to_string(),
|
||||||
|
"chunks": audio.chunks().to_string(),
|
||||||
|
},
|
||||||
|
});
|
||||||
|
let bytes = canonicalize(&payload)
|
||||||
|
.map_err(|e| DomainError::invalid(format!("State.Capture: {}", e.0)))?
|
||||||
|
.into_bytes();
|
||||||
|
let digest = digest_of_bytes(&bytes);
|
||||||
|
let artifact = crate::state::seal_payload(ctx.client, &bytes, &digest).await?;
|
||||||
|
// A capture reads the world; it does not advance it.
|
||||||
|
self.status.set_state(previous);
|
||||||
|
let result = CaptureResult {
|
||||||
|
checkpoint_id: params.checkpoint_id,
|
||||||
|
boundary: self.boundary,
|
||||||
|
compatibility_digest: crate::state::Compatibility::of(&descriptor).digest(),
|
||||||
|
payload: artifact.reference().clone(),
|
||||||
|
};
|
||||||
|
Ok(HandlerReply::with_artifacts(
|
||||||
|
object(result.to_json()),
|
||||||
|
vec![(crate::state::PAYLOAD_ATTACHMENT.to_owned(), artifact)],
|
||||||
|
))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `State.StageRestore`: validate a replacement world into a staging slot.
|
||||||
|
async fn state_stage_restore(&mut self, ctx: &HandlerCtx<'_>) -> DomainResult<HandlerReply> {
|
||||||
|
let scope = ctx.scope()?.clone();
|
||||||
|
if scope.session_id != self.config.session_id {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::IdentityMismatch,
|
||||||
|
"this environment belongs to another session",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
if let Some(epoch) = &self.epoch
|
||||||
|
&& *epoch == scope.epoch
|
||||||
|
{
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::StaleEpoch,
|
||||||
|
"State.StageRestore proposes the epoch this environment is already running",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
if self.descriptor.is_some() {
|
||||||
|
// A world that is already running a boundary is not a quiescent replacement: the
|
||||||
|
// group replaces it rather than restoring over a live one.
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::InvalidPhase,
|
||||||
|
"State.StageRestore needs an uninitialized replacement environment",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let params: StageRestoreParams = ctx.params()?;
|
||||||
|
if params.source_scope.step != scope.step {
|
||||||
|
return Err(DomainError::invalid(
|
||||||
|
"State.StageRestore's scope step must be the source boundary",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let artifact = ctx.artifact(crate::state::PAYLOAD_ATTACHMENT)?;
|
||||||
|
if artifact.reference() != ¶ms.payload {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::BufferInvalid,
|
||||||
|
"the staged payload attachment is not the artifact the request names",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let bytes = artifact.read_all().await.map_err(|e| {
|
||||||
|
DomainError::before(
|
||||||
|
ErrorCode::BufferInvalid,
|
||||||
|
format!("the staged payload could not be read: {}", e.message),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
let declared = params
|
||||||
|
.payload
|
||||||
|
.digest
|
||||||
|
.clone()
|
||||||
|
.ok_or_else(|| incompatible("a checkpoint payload must carry a content digest"))?;
|
||||||
|
let actual = digest_of_bytes(&bytes);
|
||||||
|
if actual != declared || bytes.len() as u64 != params.payload.byte_length {
|
||||||
|
return Err(incompatible(
|
||||||
|
"the staged payload is not the content the request declares",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let value: Value = serde_json::from_slice(&bytes)
|
||||||
|
.map_err(|e| DomainError::invalid(format!("the staged payload is not JSON: {e}")))?;
|
||||||
|
let text = |key: &str| -> DomainResult<String> {
|
||||||
|
value
|
||||||
|
.get(key)
|
||||||
|
.and_then(Value::as_str)
|
||||||
|
.map(str::to_owned)
|
||||||
|
.ok_or_else(|| incompatible(format!("the world payload has no {key}")))
|
||||||
|
};
|
||||||
|
let number = |key: &str| -> DomainResult<u64> {
|
||||||
|
text(key)?
|
||||||
|
.parse::<u64>()
|
||||||
|
.map_err(|_| incompatible(format!("the world payload's {key} is not a U64")))
|
||||||
|
};
|
||||||
|
if value.get("payloadVersion").and_then(Value::as_u64) != Some(WORLD_PAYLOAD_VERSION) {
|
||||||
|
return Err(incompatible("the world payload is another payload version"));
|
||||||
|
}
|
||||||
|
if text("kind")? != "world" {
|
||||||
|
return Err(incompatible("this payload is not a world's state"));
|
||||||
|
}
|
||||||
|
if text("workerId")? != self.config.worker_id {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::IdentityMismatch,
|
||||||
|
"the staged payload belongs to another world",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
if text("checkpointId")? != params.checkpoint_id {
|
||||||
|
return Err(incompatible("the staged payload belongs to another checkpoint"));
|
||||||
|
}
|
||||||
|
let source_scope = Scope::from_json(
|
||||||
|
value
|
||||||
|
.get("sourceScope")
|
||||||
|
.ok_or_else(|| incompatible("the world payload has no sourceScope"))?,
|
||||||
|
)
|
||||||
|
.map_err(|e| incompatible(format!("the world payload's sourceScope: {}", e.0)))?;
|
||||||
|
if source_scope != params.source_scope {
|
||||||
|
return Err(incompatible(
|
||||||
|
"the staged payload was captured at another source scope",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let committed_step = number("committedStep")?;
|
||||||
|
if committed_step != params.source_scope.step {
|
||||||
|
return Err(incompatible(
|
||||||
|
"the staged payload's committed step is not the source boundary",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let descriptor = EnvironmentDescriptor::from_json(
|
||||||
|
value
|
||||||
|
.get("descriptor")
|
||||||
|
.ok_or_else(|| incompatible("the world payload has no descriptor"))?,
|
||||||
|
)
|
||||||
|
.map_err(|e| incompatible(format!("the world payload's descriptor: {}", e.0)))?;
|
||||||
|
// The replacement builds the descriptor it would advertise and compares. A world
|
||||||
|
// started with other ports, another cadence or another declared render delay is a
|
||||||
|
// different backend, not this one resumed.
|
||||||
|
let live = self.build_descriptor()?;
|
||||||
|
if descriptor != live {
|
||||||
|
return Err(incompatible(
|
||||||
|
"the staged world was captured under another environment descriptor",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let expected = crate::state::Compatibility::of(&descriptor).digest();
|
||||||
|
if expected != params.compatibility_digest {
|
||||||
|
return Err(incompatible(format!(
|
||||||
|
"the staged world's compatibility {expected} is not the {} the restore requires",
|
||||||
|
params.compatibility_digest
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
let counter: i64 = text("counter")?
|
||||||
|
.parse()
|
||||||
|
.map_err(|_| incompatible("the world payload's counter is not an integer"))?;
|
||||||
|
let world_time = RationalNs::from_json(
|
||||||
|
value
|
||||||
|
.get("worldTime")
|
||||||
|
.ok_or_else(|| incompatible("the world payload has no worldTime"))?,
|
||||||
|
)
|
||||||
|
.map_err(|e| incompatible(format!("the world payload's worldTime: {}", e.0)))?;
|
||||||
|
let pipeline_value = value
|
||||||
|
.get("pipeline")
|
||||||
|
.and_then(|p| p.get("frames"))
|
||||||
|
.and_then(Value::as_array)
|
||||||
|
.ok_or_else(|| incompatible("the world payload has no pipeline frames"))?;
|
||||||
|
let mut frames = Vec::with_capacity(pipeline_value.len());
|
||||||
|
for frame in pipeline_value {
|
||||||
|
let boundary = frame
|
||||||
|
.get("boundary")
|
||||||
|
.and_then(Value::as_str)
|
||||||
|
.ok_or_else(|| incompatible("a captured frame has no boundary"))?
|
||||||
|
.parse::<u64>()
|
||||||
|
.map_err(|_| incompatible("a captured frame's boundary is not a U64"))?;
|
||||||
|
let frame_counter = frame
|
||||||
|
.get("counter")
|
||||||
|
.and_then(Value::as_str)
|
||||||
|
.ok_or_else(|| incompatible("a captured frame has no counter"))?
|
||||||
|
.parse::<i64>()
|
||||||
|
.map_err(|_| incompatible("a captured frame's counter is not an integer"))?;
|
||||||
|
frames.push((boundary, frame_counter));
|
||||||
|
}
|
||||||
|
match frames.last() {
|
||||||
|
Some((boundary, _)) if *boundary == committed_step => {}
|
||||||
|
_ => {
|
||||||
|
return Err(incompatible(
|
||||||
|
"the captured pipeline does not end at the committed boundary",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let audio_value = value
|
||||||
|
.get("audio")
|
||||||
|
.ok_or_else(|| incompatible("the world payload has no audio state"))?;
|
||||||
|
let audio_number = |key: &str| -> DomainResult<u128> {
|
||||||
|
audio_value
|
||||||
|
.get(key)
|
||||||
|
.and_then(Value::as_str)
|
||||||
|
.ok_or_else(|| incompatible(format!("the captured audio state has no {key}")))?
|
||||||
|
.parse::<u128>()
|
||||||
|
.map_err(|_| incompatible(format!("the captured audio {key} is not a number")))
|
||||||
|
};
|
||||||
|
let audio_next_sample = u64::try_from(audio_number("nextSample")?)
|
||||||
|
.map_err(|_| incompatible("the captured audio position is outside U64"))?;
|
||||||
|
let audio_phase = u64::try_from(audio_number("phase")?)
|
||||||
|
.map_err(|_| incompatible("the captured audio phase is outside U64"))?;
|
||||||
|
|
||||||
|
if self.config.faults.fail_stage_restore {
|
||||||
|
return Err(incompatible(
|
||||||
|
"injected staging refusal: this participant's replacement state does not \
|
||||||
|
validate",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
if let Some(staged) = &self.staged {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::Conflict,
|
||||||
|
format!(
|
||||||
|
"this environment already holds the staged restore {} for checkpoint {}",
|
||||||
|
staged.token, staged.checkpoint_id
|
||||||
|
),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let token = crate::agent::restore_token(
|
||||||
|
¶ms.checkpoint_id,
|
||||||
|
&scope,
|
||||||
|
&actual,
|
||||||
|
&self.config.incarnation_id,
|
||||||
|
);
|
||||||
|
if self.activated.contains(&token) {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::Conflict,
|
||||||
|
"this exact restore was already activated on this environment",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
self.staged = Some(StagedWorld {
|
||||||
|
token: token.clone(),
|
||||||
|
checkpoint_id: params.checkpoint_id.clone(),
|
||||||
|
scope,
|
||||||
|
episode_id: parse_id(&text("episodeId")?)
|
||||||
|
.map_err(|e| incompatible(format!("the world payload's episodeId {e}")))?,
|
||||||
|
descriptor,
|
||||||
|
boundary: committed_step,
|
||||||
|
counter,
|
||||||
|
world_time,
|
||||||
|
advances: number("advances")?,
|
||||||
|
frames,
|
||||||
|
audio_next_sample,
|
||||||
|
audio_phase,
|
||||||
|
audio_accumulator: audio_number("accumulator")?,
|
||||||
|
audio_denominator: audio_number("denominator")?,
|
||||||
|
});
|
||||||
|
self.status.set_state(WorkerState::StagedRestore);
|
||||||
|
let result = StageRestoreResult {
|
||||||
|
checkpoint_id: params.checkpoint_id,
|
||||||
|
restore_token: token,
|
||||||
|
};
|
||||||
|
Ok(HandlerReply::from(&result))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `State.ActivateRestore`: install the staged world and return its coherent observation.
|
||||||
|
///
|
||||||
|
/// Nothing advances. The pipeline's frames are rendered again into fresh artifacts of the
|
||||||
|
/// current store, which is what "the durable store imports fresh immutable bus artifacts"
|
||||||
|
/// means on the producing side, and the observation carries no audio chunk because no
|
||||||
|
/// interval was played.
|
||||||
|
async fn state_activate_restore(
|
||||||
|
&mut self,
|
||||||
|
ctx: &HandlerCtx<'_>,
|
||||||
|
) -> DomainResult<HandlerReply> {
|
||||||
|
let params: ActivateRestoreParams = ctx.params()?;
|
||||||
|
if self.activated.contains(¶ms.restore_token) {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::Conflict,
|
||||||
|
"this restore token has already been activated",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let Some(staged) = self.staged.take() else {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::InvalidPhase,
|
||||||
|
"this environment holds no staged restore",
|
||||||
|
));
|
||||||
|
};
|
||||||
|
if staged.token != params.restore_token {
|
||||||
|
let token = staged.token.clone();
|
||||||
|
self.staged = Some(staged);
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::IdentityMismatch,
|
||||||
|
format!(
|
||||||
|
"this environment's staged restore is {token}, not {}",
|
||||||
|
params.restore_token
|
||||||
|
),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
if self.config.faults.fail_activate_restore {
|
||||||
|
let token = staged.token.clone();
|
||||||
|
self.staged = Some(staged);
|
||||||
|
self.status.set_state(WorkerState::Failed);
|
||||||
|
return Err(DomainError::new(
|
||||||
|
ErrorCode::BackendFailure,
|
||||||
|
format!("injected activation failure; {token} stays staged and unresumed"),
|
||||||
|
MutationCertainty::None,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
self.status.set_state(WorkerState::Restoring);
|
||||||
|
let mut pipeline = ViewPipeline::new(
|
||||||
|
CounterEnvironment::view_descriptor(self.config.observation_delay_steps),
|
||||||
|
self.config.renders.clone(),
|
||||||
|
);
|
||||||
|
pipeline.restore(ctx.client, &staged.frames).await?;
|
||||||
|
let audio = AudioSource::restored_from(
|
||||||
|
CounterEnvironment::audio_descriptor(),
|
||||||
|
staged.audio_next_sample,
|
||||||
|
staged.audio_phase,
|
||||||
|
staged.audio_accumulator,
|
||||||
|
staged.audio_denominator,
|
||||||
|
)?;
|
||||||
|
self.epoch = Some(staged.scope.epoch.clone());
|
||||||
|
self.episode_id = Some(staged.episode_id.clone());
|
||||||
|
self.descriptor = Some(staged.descriptor.clone());
|
||||||
|
self.boundary = staged.boundary;
|
||||||
|
self.counter = staged.counter;
|
||||||
|
self.world_time = staged.world_time;
|
||||||
|
self.advances = staged.advances;
|
||||||
|
// Batch ids are unique within an epoch, and this is a new one. Keeping the old set
|
||||||
|
// would refuse nothing extra: a request under the old epoch is already refused by its
|
||||||
|
// scope.
|
||||||
|
self.batches.clear();
|
||||||
|
self.pipeline = Some(pipeline);
|
||||||
|
self.audio = Some(audio);
|
||||||
|
self.previous_view = None;
|
||||||
|
self.activated.insert(staged.token);
|
||||||
|
self.status.set_state(WorkerState::Ready);
|
||||||
|
self.status.set_scope(Some(scope_at(
|
||||||
|
&staged.scope.session_id,
|
||||||
|
&staged.scope.epoch,
|
||||||
|
staged.boundary,
|
||||||
|
)));
|
||||||
|
|
||||||
|
let (observation, attachments) = self.restored_observation()?;
|
||||||
|
let result = ActivateRestoreResult {
|
||||||
|
committed_step: staged.boundary,
|
||||||
|
checkpoint_id: staged.checkpoint_id,
|
||||||
|
observation: Some(observation),
|
||||||
|
};
|
||||||
|
result
|
||||||
|
.validate_for_role(Role::Environment)
|
||||||
|
.map_err(|e| DomainError::invalid(e.0))?;
|
||||||
|
let mut reply = HandlerReply::from(&result);
|
||||||
|
reply.artifacts = attachments;
|
||||||
|
Ok(reply)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The observation the restored world is already at: no render, no advance, no audio.
|
||||||
|
fn restored_observation(
|
||||||
|
&mut self,
|
||||||
|
) -> DomainResult<(WorldObservation, Vec<(String, flybus::Artifact)>)> {
|
||||||
|
let boundary = self.boundary;
|
||||||
|
let counter = self.counter;
|
||||||
|
let pipeline = self
|
||||||
|
.pipeline
|
||||||
|
.as_ref()
|
||||||
|
.ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no view pipeline"))?;
|
||||||
|
let (view, artifact) = pipeline.at(boundary).ok_or_else(|| {
|
||||||
|
incompatible("the restored pipeline holds no frame for the restored boundary")
|
||||||
|
})?;
|
||||||
|
self.previous_view = Some((view.clone(), artifact.clone()));
|
||||||
|
let observation = WorldObservation {
|
||||||
|
boundary,
|
||||||
|
world_time: self.world_time,
|
||||||
|
engine_frame: Some(boundary.to_string()),
|
||||||
|
sensory_views: vec![view.clone()],
|
||||||
|
inspection: inspection(counter, boundary),
|
||||||
|
broadcast_views: vec![view.clone()],
|
||||||
|
// No interval was played, so there is no chunk. A chunk here would be an old
|
||||||
|
// epoch's audio offered as current.
|
||||||
|
audio: Vec::new(),
|
||||||
|
};
|
||||||
|
Ok((
|
||||||
|
observation,
|
||||||
|
vec![(media::view_attachment(&view.view_id), artifact)],
|
||||||
|
))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -26,6 +26,7 @@ use crate::media::{RenderCounter, SensorLog};
|
||||||
use crate::launcher::{
|
use crate::launcher::{
|
||||||
AgentLaunch, EnvironmentLaunch, Launcher, ReapOutcome, SUPERVISOR_CLIENT, ThreadBudget,
|
AgentLaunch, EnvironmentLaunch, Launcher, ReapOutcome, SUPERVISOR_CLIENT, ThreadBudget,
|
||||||
};
|
};
|
||||||
|
use crate::state::{CheckpointStore, CheckpointWriter, StoreConfig, StoreFaults, WriterConfig, WriterFaults};
|
||||||
use crate::task::{ActionExecutor, CounterTask, IdentityExecutor, Terminal};
|
use crate::task::{ActionExecutor, CounterTask, IdentityExecutor, Terminal};
|
||||||
// `crate::types` is this crate's facade over the shared `fly-session-types` crate; the
|
// `crate::types` is this crate's facade over the shared `fly-session-types` crate; the
|
||||||
// glob keeps the contract's own names in sight instead of restating them.
|
// glob keeps the contract's own names in sight instead of restating them.
|
||||||
|
|
@ -90,6 +91,14 @@ pub struct HarnessConfig {
|
||||||
/// The threads reserved for the coordinator, its router and its store.
|
/// The threads reserved for the coordinator, its router and its store.
|
||||||
pub coordinator_threads: usize,
|
pub coordinator_threads: usize,
|
||||||
pub environment_threads: usize,
|
pub environment_threads: usize,
|
||||||
|
/// How many committed generations the durable checkpoint store keeps.
|
||||||
|
pub store: StoreConfig,
|
||||||
|
/// The durable write faults this composition injects.
|
||||||
|
pub store_faults: StoreFaults,
|
||||||
|
/// The checkpoint queue's bounds.
|
||||||
|
pub writer: WriterConfig,
|
||||||
|
/// The writer faults this composition injects.
|
||||||
|
pub writer_faults: WriterFaults,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl Default for HarnessConfig {
|
impl Default for HarnessConfig {
|
||||||
|
|
@ -112,6 +121,10 @@ impl Default for HarnessConfig {
|
||||||
thread_budget: None,
|
thread_budget: None,
|
||||||
coordinator_threads: 1,
|
coordinator_threads: 1,
|
||||||
environment_threads: 1,
|
environment_threads: 1,
|
||||||
|
store: StoreConfig::default(),
|
||||||
|
store_faults: StoreFaults::default(),
|
||||||
|
writer: WriterConfig::default(),
|
||||||
|
writer_faults: WriterFaults::default(),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -139,6 +152,14 @@ const ENV_SERVICE: &str = "env.arena";
|
||||||
const ENV_CLIENT: &str = "environment";
|
const ENV_CLIENT: &str = "environment";
|
||||||
const ENV_WORKER: &str = "arena";
|
const ENV_WORKER: &str = "arena";
|
||||||
const COORDINATOR_CLIENT: &str = "coordinator";
|
const COORDINATOR_CLIENT: &str = "coordinator";
|
||||||
|
/// The checkpoint writer's own bus identity. It publishes checkpoint events and nothing else.
|
||||||
|
const WRITER_CLIENT: &str = "checkpoint-writer";
|
||||||
|
|
||||||
|
/// How many times one participant may be replaced in a composition.
|
||||||
|
///
|
||||||
|
/// Each replacement connects under its own client id, so a restart is visibly a new
|
||||||
|
/// participant rather than a silent reattachment, and the policy has to name them all.
|
||||||
|
const MAX_GENERATIONS: u32 = 8;
|
||||||
|
|
||||||
fn agent_service(agent_id: &Id) -> String {
|
fn agent_service(agent_id: &Id) -> String {
|
||||||
format!("agent.{agent_id}")
|
format!("agent.{agent_id}")
|
||||||
|
|
@ -187,6 +208,10 @@ pub struct SessionHarness {
|
||||||
observers: Mutex<Vec<Client>>,
|
observers: Mutex<Vec<Client>>,
|
||||||
/// Which configured observer identity the next consumer takes.
|
/// Which configured observer identity the next consumer takes.
|
||||||
next_observer: std::sync::atomic::AtomicUsize,
|
next_observer: std::sync::atomic::AtomicUsize,
|
||||||
|
/// Which generation of each participant is running: 1 is the one the composition started.
|
||||||
|
generations: BTreeMap<Id, u32>,
|
||||||
|
/// Where the durable checkpoint store lives, for a test that reads the files themselves.
|
||||||
|
checkpoint_root: std::path::PathBuf,
|
||||||
}
|
}
|
||||||
|
|
||||||
impl SessionHarness {
|
impl SessionHarness {
|
||||||
|
|
@ -223,11 +248,9 @@ impl SessionHarness {
|
||||||
g.call = vec![Pattern::prefix("agent."), Pattern::prefix("env.")];
|
g.call = vec![Pattern::prefix("agent."), Pattern::prefix("env.")];
|
||||||
}),
|
}),
|
||||||
)
|
)
|
||||||
|
// The writer publishes the checkpoint events and never calls a participant.
|
||||||
|
.client(WRITER_CLIENT, grants(|g| g.publish = vec![Pattern::prefix("session.")]))
|
||||||
.client(ENV_CLIENT, grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]))
|
.client(ENV_CLIENT, grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]))
|
||||||
.client(
|
|
||||||
&format!("{ENV_CLIENT}-r2"),
|
|
||||||
grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]),
|
|
||||||
)
|
|
||||||
// A presentation consumer subscribes and may call the repair service. It can
|
// A presentation consumer subscribes and may call the repair service. It can
|
||||||
// publish nothing, register nothing and reach no worker: "viewers/browser clients
|
// publish nothing, register nothing and reach no worker: "viewers/browser clients
|
||||||
// never obtain worker control" (publishing-v1 section 7). A bus client id is one
|
// never obtain worker control" (publishing-v1 section 7). A bus client id is one
|
||||||
|
|
@ -256,6 +279,14 @@ impl SessionHarness {
|
||||||
g.manage_topics = vec![Pattern::prefix("session.")];
|
g.manage_topics = vec![Pattern::prefix("session.")];
|
||||||
}),
|
}),
|
||||||
);
|
);
|
||||||
|
// A replacement environment connects under its own client id, one per generation;
|
||||||
|
// this subsumes the single `-r2` identity the publication slice had configured.
|
||||||
|
for generation in 2..=MAX_GENERATIONS {
|
||||||
|
policy = policy.client(
|
||||||
|
&format!("{ENV_CLIENT}-r{generation}"),
|
||||||
|
grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]),
|
||||||
|
);
|
||||||
|
}
|
||||||
for spec in &config.agents {
|
for spec in &config.agents {
|
||||||
let service = agent_service(&spec.agent_id);
|
let service = agent_service(&spec.agent_id);
|
||||||
policy = policy.client(
|
policy = policy.client(
|
||||||
|
|
@ -264,10 +295,12 @@ impl SessionHarness {
|
||||||
);
|
);
|
||||||
// A replacement worker connects under its own client id, so a restart is visibly a
|
// A replacement worker connects under its own client id, so a restart is visibly a
|
||||||
// new participant rather than a silent reattachment to the active epoch.
|
// new participant rather than a silent reattachment to the active epoch.
|
||||||
policy = policy.client(
|
for generation in 2..=MAX_GENERATIONS {
|
||||||
&format!("{}-r2", agent_client(&spec.agent_id)),
|
policy = policy.client(
|
||||||
grants(|g| g.register = vec![Pattern::exact(&service)]),
|
&format!("{}-r{generation}", agent_client(&spec.agent_id)),
|
||||||
);
|
grants(|g| g.register = vec![Pattern::exact(&service)]),
|
||||||
|
);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
let mut router_config = RouterConfig::new(&store_root);
|
let mut router_config = RouterConfig::new(&store_root);
|
||||||
router_config.policy = policy;
|
router_config.policy = policy;
|
||||||
|
|
@ -346,6 +379,21 @@ impl SessionHarness {
|
||||||
}
|
}
|
||||||
|
|
||||||
let coordinator_client = launcher.connect(COORDINATOR_CLIENT).await?;
|
let coordinator_client = launcher.connect(COORDINATOR_CLIENT).await?;
|
||||||
|
// The durable store lives beside the router's artifact store and never inside it: a
|
||||||
|
// committed generation is outside the bus's ephemeral collection.
|
||||||
|
let checkpoint_root = root.join("checkpoints");
|
||||||
|
let mut store = CheckpointStore::open(&checkpoint_root, config.store).map_err(refusal)?;
|
||||||
|
*store.faults_mut() = config.store_faults.clone();
|
||||||
|
let writer_client = launcher.connect(WRITER_CLIENT).await?;
|
||||||
|
let writer = CheckpointWriter::start(
|
||||||
|
store,
|
||||||
|
config.writer,
|
||||||
|
config.writer_faults.clone(),
|
||||||
|
Some((
|
||||||
|
writer_client,
|
||||||
|
format!("session.{}.checkpoints", config.session_id),
|
||||||
|
)),
|
||||||
|
);
|
||||||
let executors: BTreeMap<Id, Box<dyn ActionExecutor>> = config
|
let executors: BTreeMap<Id, Box<dyn ActionExecutor>> = config
|
||||||
.agents
|
.agents
|
||||||
.iter()
|
.iter()
|
||||||
|
|
@ -363,6 +411,8 @@ impl SessionHarness {
|
||||||
Box::new(CounterTask::new(&config.epoch, config.terminal)),
|
Box::new(CounterTask::new(&config.epoch, config.terminal)),
|
||||||
executors,
|
executors,
|
||||||
);
|
);
|
||||||
|
let mut coordinator = coordinator;
|
||||||
|
coordinator.attach_store(writer);
|
||||||
|
|
||||||
Ok(SessionHarness {
|
Ok(SessionHarness {
|
||||||
coordinator,
|
coordinator,
|
||||||
|
|
@ -374,9 +424,16 @@ impl SessionHarness {
|
||||||
launcher,
|
launcher,
|
||||||
observers: Mutex::new(Vec::new()),
|
observers: Mutex::new(Vec::new()),
|
||||||
next_observer: std::sync::atomic::AtomicUsize::new(0),
|
next_observer: std::sync::atomic::AtomicUsize::new(0),
|
||||||
|
generations: BTreeMap::new(),
|
||||||
|
checkpoint_root,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Where the durable checkpoint store's generations and store manifest live.
|
||||||
|
pub fn checkpoint_root(&self) -> &std::path::Path {
|
||||||
|
&self.checkpoint_root
|
||||||
|
}
|
||||||
|
|
||||||
pub fn router(&self) -> &Router {
|
pub fn router(&self) -> &Router {
|
||||||
self.launcher.router()
|
self.launcher.router()
|
||||||
}
|
}
|
||||||
|
|
@ -469,10 +526,11 @@ impl SessionHarness {
|
||||||
.find(|spec| spec.agent_id == *agent_id)
|
.find(|spec| spec.agent_id == *agent_id)
|
||||||
.expect("a configured agent")
|
.expect("a configured agent")
|
||||||
.clone();
|
.clone();
|
||||||
|
let generation = self.next_generation(agent_id)?;
|
||||||
self.launcher.kill(agent_id).await;
|
self.launcher.kill(agent_id).await;
|
||||||
let tick_duration = millis(self.config.tick_ms).expect("a positive tick");
|
let tick_duration = millis(self.config.tick_ms).expect("a positive tick");
|
||||||
let incarnation_id =
|
let incarnation_id = parse_id(&format!("{agent_id}-inc-{generation}"))
|
||||||
parse_id(&format!("{agent_id}-inc-2")).expect("an agent id plus a suffix is an Id");
|
.expect("an agent id plus a suffix is an Id");
|
||||||
self.launcher
|
self.launcher
|
||||||
.launch_agent(AgentLaunch {
|
.launch_agent(AgentLaunch {
|
||||||
session_id: self.config.session_id.clone(),
|
session_id: self.config.session_id.clone(),
|
||||||
|
|
@ -487,7 +545,7 @@ impl SessionHarness {
|
||||||
// predecessor wrote, so a restore's sensory input is visible beside it.
|
// predecessor wrote, so a restore's sensory input is visible beside it.
|
||||||
sensors: self.sensors.get(agent_id).cloned().unwrap_or_default(),
|
sensors: self.sensors.get(agent_id).cloned().unwrap_or_default(),
|
||||||
faults: spec.faults.clone(),
|
faults: spec.faults.clone(),
|
||||||
client_id: format!("{}-r2", agent_client(agent_id)),
|
client_id: format!("{}-r{generation}", agent_client(agent_id)),
|
||||||
service: agent_service(agent_id),
|
service: agent_service(agent_id),
|
||||||
})
|
})
|
||||||
.await
|
.await
|
||||||
|
|
@ -500,6 +558,102 @@ impl SessionHarness {
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Replaces the environment with a fresh, uninitialized incarnation, as a restore needs.
|
||||||
|
pub async fn restart_environment(&mut self) -> Result<Restarted, flybus::BusError> {
|
||||||
|
let worker_id = id(ENV_WORKER);
|
||||||
|
let generation = self.next_generation(&worker_id)?;
|
||||||
|
self.launcher.kill(&worker_id).await;
|
||||||
|
let step_duration = hz(self.config.step_hz).expect("a positive cadence");
|
||||||
|
let incarnation_id = parse_id(&format!("arena-inc-{generation}"))
|
||||||
|
.expect("a worker id plus a suffix is an Id");
|
||||||
|
self.launcher
|
||||||
|
.launch_environment(EnvironmentLaunch {
|
||||||
|
session_id: self.config.session_id.clone(),
|
||||||
|
worker_id: worker_id.clone(),
|
||||||
|
incarnation_id: incarnation_id.clone(),
|
||||||
|
step_duration,
|
||||||
|
ports: self.config.agents.iter().map(|a| a.port_id.clone()).collect(),
|
||||||
|
worker_threads: self.config.environment_threads,
|
||||||
|
observation_delay_steps: self.config.observation_delay_steps,
|
||||||
|
renders: self.renders.clone(),
|
||||||
|
faults: self.config.environment_faults.clone(),
|
||||||
|
client_id: format!("{ENV_CLIENT}-r{generation}"),
|
||||||
|
service: ENV_SERVICE.to_owned(),
|
||||||
|
})
|
||||||
|
.await
|
||||||
|
.map_err(refusal)?;
|
||||||
|
let worker = self.launcher.worker(&worker_id).expect("just launched");
|
||||||
|
Ok(Restarted {
|
||||||
|
service: worker.identity.service.clone(),
|
||||||
|
service_incarnation: worker.service_incarnation.clone(),
|
||||||
|
incarnation_id,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
fn next_generation(&mut self, worker_id: &Id) -> Result<u32, flybus::BusError> {
|
||||||
|
let slot = self.generations.entry(worker_id.clone()).or_insert(1);
|
||||||
|
if *slot >= MAX_GENERATIONS {
|
||||||
|
return Err(flybus::BusError::new(
|
||||||
|
flybus::ErrorCode::QuotaExceeded,
|
||||||
|
format!(
|
||||||
|
"{worker_id} has used all {MAX_GENERATIONS} configured client identities; a composition declares how many replacements it allows"
|
||||||
|
),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
*slot += 1;
|
||||||
|
Ok(*slot)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Replaces every participant and points the fenced coordinator at the replacements.
|
||||||
|
///
|
||||||
|
/// This is what a recovery does before it restores: the old participants belong to an
|
||||||
|
/// invalid epoch, and the references the coordinator pinned are exchanged deliberately.
|
||||||
|
pub async fn replace_all_participants(&mut self) -> Result<(), flybus::BusError> {
|
||||||
|
let environment = self.environment_id();
|
||||||
|
self.restart_environment().await?;
|
||||||
|
let worker = self
|
||||||
|
.launcher
|
||||||
|
.worker(&environment)
|
||||||
|
.expect("just launched")
|
||||||
|
.worker_ref();
|
||||||
|
self.coordinator
|
||||||
|
.replace_participant(&environment, worker)
|
||||||
|
.map_err(|e| refusal(e.error))?;
|
||||||
|
for agent_id in self.config.agents.iter().map(|a| a.agent_id.clone()).collect::<Vec<_>>() {
|
||||||
|
self.restart_agent(&agent_id).await?;
|
||||||
|
let worker = self
|
||||||
|
.launcher
|
||||||
|
.worker(&agent_id)
|
||||||
|
.expect("just launched")
|
||||||
|
.worker_ref();
|
||||||
|
self.coordinator
|
||||||
|
.replace_participant(&agent_id, worker)
|
||||||
|
.map_err(|e| refusal(e.error))?;
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Changes one agent's injected faults, so the replacement the next restart launches is
|
||||||
|
/// a participant without them.
|
||||||
|
///
|
||||||
|
/// A fault is launch configuration, so clearing one is a relaunch and not a live change:
|
||||||
|
/// the worker running now keeps whatever it was started with.
|
||||||
|
pub fn set_agent_faults(&mut self, agent_id: &Id, faults: AgentFaults) {
|
||||||
|
if let Some(spec) = self
|
||||||
|
.config
|
||||||
|
.agents
|
||||||
|
.iter_mut()
|
||||||
|
.find(|spec| spec.agent_id == *agent_id)
|
||||||
|
{
|
||||||
|
spec.faults = faults;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Changes the environment's injected faults, with the same relaunch rule.
|
||||||
|
pub fn set_environment_faults(&mut self, faults: EnvironmentFaults) {
|
||||||
|
self.config.environment_faults = faults;
|
||||||
|
}
|
||||||
|
|
||||||
/// Ends one participant without asking it, as a crash would.
|
/// Ends one participant without asking it, as a crash would.
|
||||||
pub async fn kill(&mut self, worker_id: &Id) -> ReapOutcome {
|
pub async fn kill(&mut self, worker_id: &Id) -> ReapOutcome {
|
||||||
self.launcher.kill(worker_id).await
|
self.launcher.kill(worker_id).await
|
||||||
|
|
@ -563,7 +717,10 @@ impl SessionHarness {
|
||||||
|
|
||||||
/// Reaps every participant and closes the router.
|
/// Reaps every participant and closes the router.
|
||||||
pub async fn shutdown(self) {
|
pub async fn shutdown(self) {
|
||||||
let SessionHarness { coordinator, mut launcher, observers, .. } = self;
|
let SessionHarness { mut coordinator, mut launcher, observers, .. } = self;
|
||||||
|
// The writer task owns artifact handles and a blocking store. Leaving it running
|
||||||
|
// would leave both behind.
|
||||||
|
coordinator.shutdown_store().await;
|
||||||
drop(coordinator);
|
drop(coordinator);
|
||||||
launcher.reap_all(&id("shutdown")).await;
|
launcher.reap_all(&id("shutdown")).await;
|
||||||
for observer in observers.into_inner().expect("not poisoned") {
|
for observer in observers.into_inner().expect("not poisoned") {
|
||||||
|
|
|
||||||
|
|
@ -1235,6 +1235,8 @@ pub(crate) mod flags {
|
||||||
pub const COMMIT_DELAY_MS: &str = "commit-delay-ms";
|
pub const COMMIT_DELAY_MS: &str = "commit-delay-ms";
|
||||||
pub const FAIL_COMMIT_AT_STEP: &str = "fail-commit-at-step";
|
pub const FAIL_COMMIT_AT_STEP: &str = "fail-commit-at-step";
|
||||||
pub const GRAPH_VARIANT: &str = "graph-variant";
|
pub const GRAPH_VARIANT: &str = "graph-variant";
|
||||||
|
pub const FAIL_STAGE_RESTORE: &str = "fail-stage-restore";
|
||||||
|
pub const FAIL_ACTIVATE_RESTORE: &str = "fail-activate-restore";
|
||||||
|
|
||||||
pub const WORKER: &str = "worker";
|
pub const WORKER: &str = "worker";
|
||||||
pub const PORTS: &str = "ports";
|
pub const PORTS: &str = "ports";
|
||||||
|
|
@ -1276,6 +1278,8 @@ pub(crate) mod flags {
|
||||||
COMMIT_DELAY_MS,
|
COMMIT_DELAY_MS,
|
||||||
FAIL_COMMIT_AT_STEP,
|
FAIL_COMMIT_AT_STEP,
|
||||||
GRAPH_VARIANT,
|
GRAPH_VARIANT,
|
||||||
|
FAIL_STAGE_RESTORE,
|
||||||
|
FAIL_ACTIVATE_RESTORE,
|
||||||
];
|
];
|
||||||
/// What only the environment is given, media options included.
|
/// What only the environment is given, media options included.
|
||||||
pub const ENVIRONMENT_ONLY: &[&str] = &[
|
pub const ENVIRONMENT_ONLY: &[&str] = &[
|
||||||
|
|
@ -1290,6 +1294,8 @@ pub(crate) mod flags {
|
||||||
TRUNCATED_VIEW_AT_BOUNDARY,
|
TRUNCATED_VIEW_AT_BOUNDARY,
|
||||||
OMIT_AUDIO_AT_BOUNDARY,
|
OMIT_AUDIO_AT_BOUNDARY,
|
||||||
OVERLAPPING_AUDIO_AT_BOUNDARY,
|
OVERLAPPING_AUDIO_AT_BOUNDARY,
|
||||||
|
FAIL_STAGE_RESTORE,
|
||||||
|
FAIL_ACTIVATE_RESTORE,
|
||||||
];
|
];
|
||||||
/// What a measurement run or one of its row children is given.
|
/// What a measurement run or one of its row children is given.
|
||||||
pub const MEASURE: &[&str] = &[MODE, AGENTS, STEPS, WARMUP_STEPS, WORKER_THREADS, MODES];
|
pub const MEASURE: &[&str] = &[MODE, AGENTS, STEPS, WARMUP_STEPS, WORKER_THREADS, MODES];
|
||||||
|
|
@ -1333,6 +1339,11 @@ impl Started {
|
||||||
arg(flags::PREPARE_DELAY_MS, spec.faults.prepare_delay_ms),
|
arg(flags::PREPARE_DELAY_MS, spec.faults.prepare_delay_ms),
|
||||||
arg(flags::COMMIT_DELAY_MS, spec.faults.commit_delay_ms),
|
arg(flags::COMMIT_DELAY_MS, spec.faults.commit_delay_ms),
|
||||||
arg(flags::GRAPH_VARIANT, spec.graph_variant),
|
arg(flags::GRAPH_VARIANT, spec.graph_variant),
|
||||||
|
arg(flags::FAIL_STAGE_RESTORE, u64::from(spec.faults.fail_stage_restore)),
|
||||||
|
arg(
|
||||||
|
flags::FAIL_ACTIVATE_RESTORE,
|
||||||
|
u64::from(spec.faults.fail_activate_restore),
|
||||||
|
),
|
||||||
];
|
];
|
||||||
if let Some(step) = spec.faults.fail_commit_at_step {
|
if let Some(step) = spec.faults.fail_commit_at_step {
|
||||||
args.push(arg(flags::FAIL_COMMIT_AT_STEP, step));
|
args.push(arg(flags::FAIL_COMMIT_AT_STEP, step));
|
||||||
|
|
@ -1351,6 +1362,11 @@ impl Started {
|
||||||
// The media options a world in another process needs to be exactly this
|
// The media options a world in another process needs to be exactly this
|
||||||
// world. Its render counter and its agents' sensor logs stay there.
|
// world. Its render counter and its agents' sensor logs stay there.
|
||||||
arg(flags::OBSERVATION_DELAY_STEPS, spec.observation_delay_steps),
|
arg(flags::OBSERVATION_DELAY_STEPS, spec.observation_delay_steps),
|
||||||
|
arg(flags::FAIL_STAGE_RESTORE, u64::from(spec.faults.fail_stage_restore)),
|
||||||
|
arg(
|
||||||
|
flags::FAIL_ACTIVATE_RESTORE,
|
||||||
|
u64::from(spec.faults.fail_activate_restore),
|
||||||
|
),
|
||||||
];
|
];
|
||||||
for (flag, boundary) in [
|
for (flag, boundary) in [
|
||||||
(flags::OMIT_VIEW_AT_BOUNDARY, spec.faults.omit_view_at_boundary),
|
(flags::OMIT_VIEW_AT_BOUNDARY, spec.faults.omit_view_at_boundary),
|
||||||
|
|
@ -1465,6 +1481,8 @@ mod flag_tests {
|
||||||
truncated_view_at_boundary: Some(3),
|
truncated_view_at_boundary: Some(3),
|
||||||
omit_audio_at_boundary: Some(4),
|
omit_audio_at_boundary: Some(4),
|
||||||
overlapping_audio_at_boundary: Some(5),
|
overlapping_audio_at_boundary: Some(5),
|
||||||
|
fail_stage_restore: true,
|
||||||
|
fail_activate_restore: true,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -1499,6 +1517,8 @@ mod flag_tests {
|
||||||
fail_commit_at_step: Some(2),
|
fail_commit_at_step: Some(2),
|
||||||
prepare_delay_ms: 1,
|
prepare_delay_ms: 1,
|
||||||
commit_delay_ms: 2,
|
commit_delay_ms: 2,
|
||||||
|
fail_stage_restore: true,
|
||||||
|
fail_activate_restore: true,
|
||||||
},
|
},
|
||||||
client_id: "worker-fly-a".to_owned(),
|
client_id: "worker-fly-a".to_owned(),
|
||||||
service: "agent.fly-a".to_owned(),
|
service: "agent.fly-a".to_owned(),
|
||||||
|
|
@ -1543,6 +1563,8 @@ mod flag_tests {
|
||||||
flags::TRUNCATED_VIEW_AT_BOUNDARY,
|
flags::TRUNCATED_VIEW_AT_BOUNDARY,
|
||||||
flags::OMIT_AUDIO_AT_BOUNDARY,
|
flags::OMIT_AUDIO_AT_BOUNDARY,
|
||||||
flags::OVERLAPPING_AUDIO_AT_BOUNDARY,
|
flags::OVERLAPPING_AUDIO_AT_BOUNDARY,
|
||||||
|
flags::FAIL_STAGE_RESTORE,
|
||||||
|
flags::FAIL_ACTIVATE_RESTORE,
|
||||||
] {
|
] {
|
||||||
assert!(written.contains(&format!("--{flag}")), "--{flag} is not written");
|
assert!(written.contains(&format!("--{flag}")), "--{flag} is not written");
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -35,6 +35,7 @@ pub mod metrics;
|
||||||
pub mod phase;
|
pub mod phase;
|
||||||
pub mod publish;
|
pub mod publish;
|
||||||
pub mod rpc;
|
pub mod rpc;
|
||||||
|
pub mod state;
|
||||||
pub mod task;
|
pub mod task;
|
||||||
pub mod worker;
|
pub mod worker;
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -133,7 +133,14 @@ pub fn arena_frame(descriptor: &ViewDescriptor, counter: i64, boundary: u64) ->
|
||||||
/// cannot be served an arbitrary stale image.
|
/// cannot be served an arbitrary stale image.
|
||||||
pub struct ViewPipeline {
|
pub struct ViewPipeline {
|
||||||
descriptor: ViewDescriptor,
|
descriptor: ViewDescriptor,
|
||||||
frames: VecDeque<(u64, flybus::Artifact)>,
|
/// Each retained frame: its producing boundary, the world counter it was rendered from
|
||||||
|
/// and the owned handle on its immutable bytes.
|
||||||
|
///
|
||||||
|
/// The counter is kept because it is the whole of the reconstruction input: a checkpoint
|
||||||
|
/// records `(boundary, counter)` per retained frame and a restore re-renders them into
|
||||||
|
/// fresh artifacts of the current store, rather than persisting a transient artifact
|
||||||
|
/// identity that cannot survive a router restart.
|
||||||
|
frames: VecDeque<(u64, i64, flybus::Artifact)>,
|
||||||
renders: RenderCounter,
|
renders: RenderCounter,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -160,7 +167,7 @@ impl ViewPipeline {
|
||||||
let bytes = arena_frame(&self.descriptor, counter, boundary);
|
let bytes = arena_frame(&self.descriptor, counter, boundary);
|
||||||
let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?;
|
let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?;
|
||||||
self.renders.bump();
|
self.renders.bump();
|
||||||
self.frames.push_back((boundary, artifact));
|
self.frames.push_back((boundary, counter, artifact));
|
||||||
// Keep exactly the frames a declared delay can still require.
|
// Keep exactly the frames a declared delay can still require.
|
||||||
while self.frames.len() > self.descriptor.observation_delay_steps as usize + 1 {
|
while self.frames.len() > self.descriptor.observation_delay_steps as usize + 1 {
|
||||||
self.frames.pop_front();
|
self.frames.pop_front();
|
||||||
|
|
@ -168,6 +175,50 @@ impl ViewPipeline {
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The reconstruction inputs of every retained frame, oldest first.
|
||||||
|
///
|
||||||
|
/// This is what a checkpoint records for the pending sensor pipeline: the producing
|
||||||
|
/// boundary and the world counter, never an artifact identity.
|
||||||
|
pub fn retained(&self) -> Vec<(u64, i64)> {
|
||||||
|
self.frames
|
||||||
|
.iter()
|
||||||
|
.map(|(boundary, counter, _)| (*boundary, *counter))
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Rebuilds the pipeline from recorded reconstruction inputs, into fresh artifacts.
|
||||||
|
///
|
||||||
|
/// Every frame is rendered again in the current store, so nothing a fence dropped is
|
||||||
|
/// expected to come back and no old artifact identity crosses the recovery.
|
||||||
|
pub async fn restore(
|
||||||
|
&mut self,
|
||||||
|
client: &flybus::Client,
|
||||||
|
frames: &[(u64, i64)],
|
||||||
|
) -> DomainResult<()> {
|
||||||
|
if frames.len() > self.descriptor.observation_delay_steps as usize + 1 {
|
||||||
|
return Err(media_error(format!(
|
||||||
|
"a captured pipeline of {} frames does not fit a declared delay of {}",
|
||||||
|
frames.len(),
|
||||||
|
self.descriptor.observation_delay_steps
|
||||||
|
)));
|
||||||
|
}
|
||||||
|
for window in frames.windows(2) {
|
||||||
|
if window[1].0 != window[0].0 + 1 {
|
||||||
|
return Err(media_error(
|
||||||
|
"a captured pipeline's producing boundaries are not consecutive",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
self.frames.clear();
|
||||||
|
for (boundary, counter) in frames {
|
||||||
|
let bytes = arena_frame(&self.descriptor, *counter, *boundary);
|
||||||
|
let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?;
|
||||||
|
self.renders.bump();
|
||||||
|
self.frames.push_back((*boundary, *counter, artifact));
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
/// Seals a frame of the wrong length, which is what a broken backend produces. The
|
/// Seals a frame of the wrong length, which is what a broken backend produces. The
|
||||||
/// reference it returns describes the artifact honestly, so the shape check is the thing
|
/// reference it returns describes the artifact honestly, so the shape check is the thing
|
||||||
/// under test rather than a lie in the payload.
|
/// under test rather than a lie in the payload.
|
||||||
|
|
@ -181,7 +232,7 @@ impl ViewPipeline {
|
||||||
bytes.truncate(bytes.len() - self.descriptor.row_stride as usize);
|
bytes.truncate(bytes.len() - self.descriptor.row_stride as usize);
|
||||||
let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?;
|
let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?;
|
||||||
self.renders.bump();
|
self.renders.bump();
|
||||||
self.frames.push_back((boundary, artifact));
|
self.frames.push_back((boundary, counter, artifact));
|
||||||
while self.frames.len() > self.descriptor.observation_delay_steps as usize + 2 {
|
while self.frames.len() > self.descriptor.observation_delay_steps as usize + 2 {
|
||||||
self.frames.pop_front();
|
self.frames.pop_front();
|
||||||
}
|
}
|
||||||
|
|
@ -198,8 +249,8 @@ impl ViewPipeline {
|
||||||
pub fn frame_produced_at(&self, produced: u64) -> Option<(ViewRef, flybus::Artifact)> {
|
pub fn frame_produced_at(&self, produced: u64) -> Option<(ViewRef, flybus::Artifact)> {
|
||||||
self.frames
|
self.frames
|
||||||
.iter()
|
.iter()
|
||||||
.find(|(step, _)| *step == produced)
|
.find(|(step, _, _)| *step == produced)
|
||||||
.map(|(step, artifact)| {
|
.map(|(step, _, artifact)| {
|
||||||
(
|
(
|
||||||
ViewRef {
|
ViewRef {
|
||||||
view_id: self.descriptor.view_id.clone(),
|
view_id: self.descriptor.view_id.clone(),
|
||||||
|
|
@ -257,6 +308,52 @@ impl AudioSource {
|
||||||
source
|
source
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The exact state a capture recorded: sample position, waveform phase and the
|
||||||
|
/// unconsumed fraction of a frame.
|
||||||
|
///
|
||||||
|
/// Restoring the position alone would restart the waveform and round the remainder away,
|
||||||
|
/// which is a resample the restore rules refuse. The first chunk of the new epoch marks
|
||||||
|
/// the discontinuity the recovery established.
|
||||||
|
pub fn restored_from(
|
||||||
|
descriptor: AudioDescriptor,
|
||||||
|
next_sample: u64,
|
||||||
|
phase: u64,
|
||||||
|
accumulator: u128,
|
||||||
|
denominator: u128,
|
||||||
|
) -> DomainResult<AudioSource> {
|
||||||
|
if denominator == 0 {
|
||||||
|
return Err(DomainError::invalid(
|
||||||
|
"audio: a captured accumulator denominator of zero",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
if accumulator >= denominator {
|
||||||
|
return Err(DomainError::invalid(
|
||||||
|
"audio: a captured accumulator is not below one whole frame",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
if phase >= descriptor.sample_rate {
|
||||||
|
return Err(DomainError::invalid(
|
||||||
|
"audio: a captured phase is not below the sample rate",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let mut source = AudioSource::new(descriptor, next_sample);
|
||||||
|
source.discontinuous = true;
|
||||||
|
source.phase = phase;
|
||||||
|
source.accumulator = accumulator;
|
||||||
|
source.denominator = denominator;
|
||||||
|
Ok(source)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The waveform phase, for a capture.
|
||||||
|
pub fn phase(&self) -> u64 {
|
||||||
|
self.phase
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The unconsumed fraction of a frame and the denominator it is over, for a capture.
|
||||||
|
pub fn accumulator(&self) -> (u128, u128) {
|
||||||
|
(self.accumulator, self.denominator)
|
||||||
|
}
|
||||||
|
|
||||||
pub fn descriptor(&self) -> &AudioDescriptor {
|
pub fn descriptor(&self) -> &AudioDescriptor {
|
||||||
&self.descriptor
|
&self.descriptor
|
||||||
}
|
}
|
||||||
|
|
@ -451,18 +548,47 @@ pub fn check_required_views(
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Every declared audio stream produces exactly one chunk per transition.
|
/// Where an observation came from.
|
||||||
|
///
|
||||||
|
/// `state-media-v1` section 2 makes a chunk the audio of an *interval*, so whether an
|
||||||
|
/// observation must carry one is a question about its provenance and not about its boundary
|
||||||
|
/// number. MEDIA-01 wrote the rule as "boundary 0 carries no chunk", which is true of the one
|
||||||
|
/// observation that slice could produce without a transition and false of the other one:
|
||||||
|
/// `State.ActivateRestore` installs a coherent observation at boundary `k` without advancing
|
||||||
|
/// gameplay, and it covers no interval either. Naming the provenance is the fix; exempting
|
||||||
|
/// the restored observation from the validator instead would have left "must a chunk exist"
|
||||||
|
/// unanswered exactly where a stale chunk would do the most damage.
|
||||||
|
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||||
|
pub enum ObservationOrigin {
|
||||||
|
/// The observation a completed transition produced. Its interval has audio.
|
||||||
|
Transition,
|
||||||
|
/// An observation established at a boundary without running a transition:
|
||||||
|
/// `Environment.Initialize`'s `O[0]` and `State.ActivateRestore`'s restored observation.
|
||||||
|
/// It covers no interval, so it carries no chunk and one in it is refused.
|
||||||
|
Installed,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Every declared audio stream produces exactly one chunk per transition, and none at all in
|
||||||
|
/// an observation that is not one.
|
||||||
///
|
///
|
||||||
/// The contract states the shape and the ordering of chunks, not whether one has to exist, so
|
/// The contract states the shape and the ordering of chunks, not whether one has to exist, so
|
||||||
/// this is MEDIA-01's choice and it is deliberate: a session that tolerates a silently missing
|
/// this is MEDIA-01's choice and it is deliberate: a session that tolerates a silently missing
|
||||||
/// chunk cannot tell "this world produced no audio for this interval" from "the chunk was
|
/// chunk cannot tell "this world produced no audio for this interval" from "the chunk was
|
||||||
/// lost", and the second is the case the retention rules care about. Boundary 0 has no
|
/// lost", and the second is the case the retention rules care about. The mirror of that, which
|
||||||
/// preceding interval and so carries no chunk.
|
/// STATE-01 needs, is that an installed observation carrying a chunk is a stale chunk being
|
||||||
|
/// offered as current, and is refused for the same reason.
|
||||||
pub fn check_required_audio(
|
pub fn check_required_audio(
|
||||||
descriptor: &EnvironmentDescriptor,
|
descriptor: &EnvironmentDescriptor,
|
||||||
observation: &WorldObservation,
|
observation: &WorldObservation,
|
||||||
|
origin: ObservationOrigin,
|
||||||
) -> DomainResult<()> {
|
) -> DomainResult<()> {
|
||||||
if observation.boundary == 0 {
|
if origin == ObservationOrigin::Installed {
|
||||||
|
if let Some(chunk) = observation.audio.first() {
|
||||||
|
return Err(media_error(format!(
|
||||||
|
"audio stream {} produced a chunk for an observation that ran no transition",
|
||||||
|
chunk.stream_id
|
||||||
|
)));
|
||||||
|
}
|
||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
for stream in &descriptor.audio {
|
for stream in &descriptor.audio {
|
||||||
|
|
|
||||||
|
|
@ -500,6 +500,10 @@ pub struct Publisher {
|
||||||
descriptor_topic: TopicPolicy,
|
descriptor_topic: TopicPolicy,
|
||||||
snapshot_topic: TopicPolicy,
|
snapshot_topic: TopicPolicy,
|
||||||
event_topic: TopicPolicy,
|
event_topic: TopicPolicy,
|
||||||
|
/// The STATE-01 checkpoint stream. A stream of distinct facts, so it is a bounded
|
||||||
|
/// delivery and never a latest value: a "committed" that replaced a "queued" would erase
|
||||||
|
/// the distinction the durable commit rules are built on.
|
||||||
|
checkpoint_topic: TopicPolicy,
|
||||||
outbox: EventOutbox,
|
outbox: EventOutbox,
|
||||||
state: SharedState,
|
state: SharedState,
|
||||||
ledger: Ledger,
|
ledger: Ledger,
|
||||||
|
|
@ -520,6 +524,7 @@ impl Publisher {
|
||||||
descriptor_topic: TopicPolicy::latest(&topics.descriptor),
|
descriptor_topic: TopicPolicy::latest(&topics.descriptor),
|
||||||
snapshot_topic: TopicPolicy::latest(&topics.snapshots),
|
snapshot_topic: TopicPolicy::latest(&topics.snapshots),
|
||||||
event_topic: TopicPolicy::bounded(&topics.events, EVENT_BATCH_DEPTH),
|
event_topic: TopicPolicy::bounded(&topics.events, EVENT_BATCH_DEPTH),
|
||||||
|
checkpoint_topic: TopicPolicy::bounded(&topics.checkpoints, EVENT_BATCH_DEPTH),
|
||||||
outbox: EventOutbox::new(EVENT_BATCH_DEPTH),
|
outbox: EventOutbox::new(EVENT_BATCH_DEPTH),
|
||||||
state: Arc::new(Mutex::new(PublishedState::default())),
|
state: Arc::new(Mutex::new(PublishedState::default())),
|
||||||
ledger: Ledger::default(),
|
ledger: Ledger::default(),
|
||||||
|
|
@ -533,6 +538,7 @@ impl Publisher {
|
||||||
self.descriptor_topic.clone(),
|
self.descriptor_topic.clone(),
|
||||||
self.snapshot_topic.clone(),
|
self.snapshot_topic.clone(),
|
||||||
self.event_topic.clone(),
|
self.event_topic.clone(),
|
||||||
|
self.checkpoint_topic.clone(),
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -675,6 +681,22 @@ impl Publisher {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
impl Publisher {
|
||||||
|
/// Publishes one checkpoint fact on the checkpoint stream.
|
||||||
|
///
|
||||||
|
/// It goes through the same named outcomes as everything else: a durable-commit fact that
|
||||||
|
/// an observer refuses is counted and does not fail the session, because the durable
|
||||||
|
/// acknowledgment is the store's, not the subscriber's -- "bus publish acceptance and
|
||||||
|
/// delivery consumption are not durable acknowledgments" (publishing-v1 section 6).
|
||||||
|
pub async fn publish_checkpoint(&mut self, payload: Map<String, Value>) -> PublicationOutcome {
|
||||||
|
let topic = self.checkpoint_topic.topic.clone();
|
||||||
|
let outcome =
|
||||||
|
PublicationOutcome::from_bus(&topic, self.bus.publish(&topic, payload, &[]).await);
|
||||||
|
self.ledger.record(&outcome);
|
||||||
|
outcome
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// ----------------------------------------------------------------------------------------------
|
// ----------------------------------------------------------------------------------------------
|
||||||
// The query service: the repair path
|
// The query service: the repair path
|
||||||
|
|
||||||
|
|
|
||||||
1558
services/flysim/crates/fly-session/src/state.rs
Normal file
1558
services/flysim/crates/fly-session/src/state.rs
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -39,6 +39,16 @@ pub fn episode_schema() -> SchemaRef {
|
||||||
synthetic_schema("arena.episode.v1", 1)
|
synthetic_schema("arena.episode.v1", 1)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The schema of a captured task ledger.
|
||||||
|
pub fn ledger_schema() -> SchemaRef {
|
||||||
|
synthetic_schema("arena.ledger.v1", 1)
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The schema of a captured action-executor state.
|
||||||
|
pub fn executor_schema() -> SchemaRef {
|
||||||
|
synthetic_schema("arena.executor.v1", 1)
|
||||||
|
}
|
||||||
|
|
||||||
pub fn controller_schema_ref() -> SchemaRef {
|
pub fn controller_schema_ref() -> SchemaRef {
|
||||||
synthetic_schema("arena.controller.v1", 1)
|
synthetic_schema("arena.controller.v1", 1)
|
||||||
}
|
}
|
||||||
|
|
@ -86,6 +96,29 @@ pub trait Task: Send {
|
||||||
|
|
||||||
/// How many times `evaluate_transition` has run. A transition must evaluate once.
|
/// How many times `evaluate_transition` has run. A transition must evaluate once.
|
||||||
fn evaluations(&self) -> u64;
|
fn evaluations(&self) -> u64;
|
||||||
|
|
||||||
|
/// The checkpointable ledger at a committed boundary (`workers-v1` section 4).
|
||||||
|
fn capture(&self) -> DomainResult<TypedValue>;
|
||||||
|
|
||||||
|
/// Validates a captured ledger without installing it, so a group install can fail before
|
||||||
|
/// anything is changed.
|
||||||
|
fn validate_restore(&self, state: &TypedValue) -> DomainResult<()>;
|
||||||
|
|
||||||
|
/// Installs a validated ledger under `epoch`. Event identity is derived from the epoch,
|
||||||
|
/// so the new one is part of the install rather than something the ledger keeps from the
|
||||||
|
/// epoch it was captured in.
|
||||||
|
fn install_restore(&mut self, epoch: &Id, state: &TypedValue) -> DomainResult<()>;
|
||||||
|
|
||||||
|
/// Every event identity this ledger has issued, mapped onto the identity it would have
|
||||||
|
/// under `to_epoch`.
|
||||||
|
///
|
||||||
|
/// `workers-v1` section 4 derives an event id from the epoch, so a trace recorded in one
|
||||||
|
/// epoch cannot be compared with a trace recorded in another until these are rebased.
|
||||||
|
/// The ledger owns the derivation, so it is the only thing that can do it.
|
||||||
|
fn rebase_ids(&self, to_epoch: &Id) -> DomainResult<BTreeMap<Id, Id>>;
|
||||||
|
|
||||||
|
/// How far event identity has reached: the highest source step and the number issued.
|
||||||
|
fn event_watermarks(&self) -> (u64, u64);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Translates one selected decision into a controller intent, with no port assignment.
|
/// Translates one selected decision into a controller intent, with no port assignment.
|
||||||
|
|
@ -98,6 +131,15 @@ pub trait ActionExecutor: Send {
|
||||||
progress: &TypedValue,
|
progress: &TypedValue,
|
||||||
clock: &RationalNs,
|
clock: &RationalNs,
|
||||||
) -> DomainResult<(ControllerIntent, Vec<TaskEvent>)>;
|
) -> DomainResult<(ControllerIntent, Vec<TaskEvent>)>;
|
||||||
|
|
||||||
|
/// Per-executor state at a committed boundary (`workers-v1` section 4).
|
||||||
|
fn capture(&self) -> DomainResult<TypedValue>;
|
||||||
|
|
||||||
|
/// Validates a captured executor state without installing it.
|
||||||
|
fn validate_restore(&self, state: &TypedValue) -> DomainResult<()>;
|
||||||
|
|
||||||
|
/// Installs a validated executor state.
|
||||||
|
fn install_restore(&mut self, state: &TypedValue) -> DomainResult<()>;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The only executor v1 supports: it passes a direct-control decision through unchanged.
|
/// The only executor v1 supports: it passes a direct-control decision through unchanged.
|
||||||
|
|
@ -123,6 +165,37 @@ impl ActionExecutor for IdentityExecutor {
|
||||||
.map_err(|e| DomainError::invalid(format!("decision: {e}")))?;
|
.map_err(|e| DomainError::invalid(format!("decision: {e}")))?;
|
||||||
Ok((intent, Vec::new()))
|
Ok((intent, Vec::new()))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The identity executor is stateless, and says so rather than capturing nothing.
|
||||||
|
///
|
||||||
|
/// An empty object would be indistinguishable from a stateful executor whose capture went
|
||||||
|
/// missing, so the capture names the executor it came from and a restore refuses any
|
||||||
|
/// other one.
|
||||||
|
fn capture(&self) -> DomainResult<TypedValue> {
|
||||||
|
TypedValue::new(executor_schema(), json!({"executor": "identity-v1"}))
|
||||||
|
.map_err(|e| DomainError::invalid(e.0))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn validate_restore(&self, state: &TypedValue) -> DomainResult<()> {
|
||||||
|
if state.schema != executor_schema() {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::IncompatibleState,
|
||||||
|
"the captured executor state does not carry the executor schema",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
match state.value.get("executor").and_then(Value::as_str) {
|
||||||
|
Some("identity-v1") => Ok(()),
|
||||||
|
other => Err(DomainError::before(
|
||||||
|
ErrorCode::IncompatibleState,
|
||||||
|
format!("the captured executor is {other:?}, not the identity executor"),
|
||||||
|
)),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn install_restore(&mut self, state: &TypedValue) -> DomainResult<()> {
|
||||||
|
// Stateless: validation is the whole of the install, and it is not skipped.
|
||||||
|
self.validate_restore(state)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// When the counter task asks for a terminal episode transition.
|
/// When the counter task asks for a terminal episode transition.
|
||||||
|
|
@ -146,6 +219,10 @@ pub struct CounterTask {
|
||||||
evaluations: u64,
|
evaluations: u64,
|
||||||
total_reward: f64,
|
total_reward: f64,
|
||||||
counter: i64,
|
counter: i64,
|
||||||
|
/// The highest source step any issued event belongs to, and how many were issued. These
|
||||||
|
/// are the event watermarks a checkpoint records and a resumed epoch continues from.
|
||||||
|
last_source_step: u64,
|
||||||
|
issued_events: u64,
|
||||||
terminal: Terminal,
|
terminal: Terminal,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -159,6 +236,8 @@ impl CounterTask {
|
||||||
evaluations: 0,
|
evaluations: 0,
|
||||||
total_reward: 0.0,
|
total_reward: 0.0,
|
||||||
counter: 0,
|
counter: 0,
|
||||||
|
last_source_step: 0,
|
||||||
|
issued_events: 0,
|
||||||
terminal,
|
terminal,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -239,6 +318,7 @@ impl Task for CounterTask {
|
||||||
payload: TypedValue::new(event_schema(), json!({"counter": self.counter}))
|
payload: TypedValue::new(event_schema(), json!({"counter": self.counter}))
|
||||||
.expect("a synthetic typed value fits the contract"),
|
.expect("a synthetic typed value fits the contract"),
|
||||||
}];
|
}];
|
||||||
|
self.issued_events += events.len() as u64;
|
||||||
Ok(Bootstrap { contexts, progress: self.progress_value(), events })
|
Ok(Bootstrap { contexts, progress: self.progress_value(), events })
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -314,6 +394,8 @@ impl Task for CounterTask {
|
||||||
));
|
));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
self.last_source_step = self.last_source_step.max(source_step);
|
||||||
|
self.issued_events += events.len() as u64;
|
||||||
let next_contexts = self
|
let next_contexts = self
|
||||||
.agents
|
.agents
|
||||||
.iter()
|
.iter()
|
||||||
|
|
@ -346,6 +428,173 @@ impl Task for CounterTask {
|
||||||
fn evaluations(&self) -> u64 {
|
fn evaluations(&self) -> u64 {
|
||||||
self.evaluations
|
self.evaluations
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn capture(&self) -> DomainResult<TypedValue> {
|
||||||
|
TypedValue::new(
|
||||||
|
ledger_schema(),
|
||||||
|
json!({
|
||||||
|
"epoch": self.epoch.as_str(),
|
||||||
|
"agents": self.agents.iter().map(String::as_str).collect::<Vec<_>>(),
|
||||||
|
"bindings": self
|
||||||
|
.bindings
|
||||||
|
.iter()
|
||||||
|
.map(|b| json!({"portId": b.port_id.as_str(), "agentId": b.agent_id.as_str()}))
|
||||||
|
.collect::<Vec<_>>(),
|
||||||
|
"transitions": self.transitions,
|
||||||
|
"evaluations": self.evaluations,
|
||||||
|
"totalReward": self.total_reward,
|
||||||
|
"counter": self.counter,
|
||||||
|
"lastSourceStep": self.last_source_step,
|
||||||
|
"issuedEvents": self.issued_events,
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
.map_err(|e| DomainError::invalid(e.0))
|
||||||
|
}
|
||||||
|
|
||||||
|
fn validate_restore(&self, state: &TypedValue) -> DomainResult<()> {
|
||||||
|
if state.schema != ledger_schema() {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::IncompatibleState,
|
||||||
|
"the captured ledger does not carry this task's schema",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
for field in [
|
||||||
|
"epoch",
|
||||||
|
"agents",
|
||||||
|
"bindings",
|
||||||
|
"transitions",
|
||||||
|
"evaluations",
|
||||||
|
"totalReward",
|
||||||
|
"counter",
|
||||||
|
"lastSourceStep",
|
||||||
|
"issuedEvents",
|
||||||
|
] {
|
||||||
|
if state.value.get(field).is_none() {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::IncompatibleState,
|
||||||
|
format!("the captured ledger has no {field}"),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
let bindings = state
|
||||||
|
.value
|
||||||
|
.get("bindings")
|
||||||
|
.and_then(Value::as_array)
|
||||||
|
.ok_or_else(|| {
|
||||||
|
DomainError::before(
|
||||||
|
ErrorCode::IncompatibleState,
|
||||||
|
"the captured ledger's bindings are not a list",
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
if bindings.len() != self.bindings.len() && !self.bindings.is_empty() {
|
||||||
|
return Err(DomainError::before(
|
||||||
|
ErrorCode::IncompatibleState,
|
||||||
|
"the captured ledger binds another number of ports",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn install_restore(&mut self, epoch: &Id, state: &TypedValue) -> DomainResult<()> {
|
||||||
|
self.validate_restore(state)?;
|
||||||
|
let number = |key: &str| -> DomainResult<u64> {
|
||||||
|
state.value.get(key).and_then(Value::as_u64).ok_or_else(|| {
|
||||||
|
DomainError::before(
|
||||||
|
ErrorCode::IncompatibleState,
|
||||||
|
format!("the captured ledger's {key} is not a whole number"),
|
||||||
|
)
|
||||||
|
})
|
||||||
|
};
|
||||||
|
let mut agents = Vec::new();
|
||||||
|
for value in state.value["agents"].as_array().expect("validated") {
|
||||||
|
let agent = value.as_str().ok_or_else(|| {
|
||||||
|
DomainError::before(
|
||||||
|
ErrorCode::IncompatibleState,
|
||||||
|
"the captured ledger names an agent that is not a string",
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
agents.push(parse_id(agent).map_err(|e| {
|
||||||
|
DomainError::before(ErrorCode::IncompatibleState, format!("ledger: {e}"))
|
||||||
|
})?);
|
||||||
|
}
|
||||||
|
let mut bindings = Vec::new();
|
||||||
|
for value in state.value["bindings"].as_array().expect("validated") {
|
||||||
|
let port_id = value.get("portId").and_then(Value::as_str).ok_or_else(|| {
|
||||||
|
DomainError::before(
|
||||||
|
ErrorCode::IncompatibleState,
|
||||||
|
"the captured ledger has a binding with no portId",
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
let agent_id = value.get("agentId").and_then(Value::as_str).ok_or_else(|| {
|
||||||
|
DomainError::before(
|
||||||
|
ErrorCode::IncompatibleState,
|
||||||
|
"the captured ledger has a binding with no agentId",
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
bindings.push(PortBinding {
|
||||||
|
port_id: parse_id(port_id).map_err(|e| {
|
||||||
|
DomainError::before(ErrorCode::IncompatibleState, format!("ledger: {e}"))
|
||||||
|
})?,
|
||||||
|
agent_id: parse_id(agent_id).map_err(|e| {
|
||||||
|
DomainError::before(ErrorCode::IncompatibleState, format!("ledger: {e}"))
|
||||||
|
})?,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
let counter = state.value.get("counter").and_then(Value::as_i64).ok_or_else(|| {
|
||||||
|
DomainError::before(
|
||||||
|
ErrorCode::IncompatibleState,
|
||||||
|
"the captured ledger's counter is not an integer",
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
let total_reward = state
|
||||||
|
.value
|
||||||
|
.get("totalReward")
|
||||||
|
.and_then(Value::as_f64)
|
||||||
|
.filter(|v| v.is_finite())
|
||||||
|
.ok_or_else(|| {
|
||||||
|
DomainError::before(
|
||||||
|
ErrorCode::IncompatibleState,
|
||||||
|
"the captured ledger's totalReward is not a finite number",
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
// The epoch is the caller's, not the capture's: event identity belongs to the epoch
|
||||||
|
// the ledger is being installed into.
|
||||||
|
self.epoch = epoch.clone();
|
||||||
|
self.agents = agents;
|
||||||
|
self.bindings = bindings;
|
||||||
|
self.transitions = number("transitions")?;
|
||||||
|
self.evaluations = number("evaluations")?;
|
||||||
|
self.total_reward = total_reward;
|
||||||
|
self.counter = counter;
|
||||||
|
self.last_source_step = number("lastSourceStep")?;
|
||||||
|
self.issued_events = number("issuedEvents")?;
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
fn rebase_ids(&self, to_epoch: &Id) -> DomainResult<BTreeMap<Id, Id>> {
|
||||||
|
let mut out = BTreeMap::new();
|
||||||
|
out.insert(
|
||||||
|
event_id(&self.epoch, 0, "bootstrap", 0),
|
||||||
|
event_id(to_epoch, 0, "bootstrap", 0),
|
||||||
|
);
|
||||||
|
// The counter task issues exactly one `counter-delta` event per bound port per
|
||||||
|
// evaluated transition, in descriptor port order, so every identity it has ever
|
||||||
|
// issued is re-derivable from its ledger without keeping a list of them.
|
||||||
|
let ports = self.bindings.len() as u32;
|
||||||
|
for source_step in 1..=self.last_source_step {
|
||||||
|
for ordinal in 0..ports {
|
||||||
|
out.insert(
|
||||||
|
event_id(&self.epoch, source_step, "counter-delta", ordinal),
|
||||||
|
event_id(to_epoch, source_step, "counter-delta", ordinal),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
Ok(out)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn event_watermarks(&self) -> (u64, u64) {
|
||||||
|
(self.last_source_step, self.issued_events)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The inspection value the counter environment publishes.
|
/// The inspection value the counter environment publishes.
|
||||||
|
|
|
||||||
|
|
@ -8,13 +8,18 @@
|
||||||
//! `Id` and `Digest` are type aliases, because the shared crate carries both as validated
|
//! `Id` and `Digest` are type aliases, because the shared crate carries both as validated
|
||||||
//! `String`s from `flybus::wire` rather than forking the encodings into newtypes.
|
//! `String`s from `flybus::wire` rather than forking the encodings into newtypes.
|
||||||
|
|
||||||
|
use std::collections::BTreeMap;
|
||||||
|
|
||||||
use serde_json::{Map, Value};
|
use serde_json::{Map, Value};
|
||||||
|
|
||||||
pub use fly_session_types::ArtifactRef;
|
pub use fly_session_types::ArtifactRef;
|
||||||
pub use fly_session_types::canonical::{
|
pub use fly_session_types::canonical::{
|
||||||
self, OperationKey, body_digest, canonicalize, digest_of, sha256_hex,
|
self, OperationKey, body_digest, canonicalize, digest_of, sha256_hex,
|
||||||
};
|
};
|
||||||
pub use fly_session_types::media::{AudioDescriptor, AudioRef, ViewDescriptor, ViewRef};
|
pub use fly_session_types::media::{
|
||||||
|
ActivateRestoreParams, ActivateRestoreResult, AudioDescriptor, AudioRef, CaptureParams,
|
||||||
|
CaptureResult, StageRestoreParams, StageRestoreResult, ViewDescriptor, ViewRef,
|
||||||
|
};
|
||||||
pub use fly_session_types::rpc::{
|
pub use fly_session_types::rpc::{
|
||||||
ErrorCode, MutationCertainty, SessionRpcFailure, SessionRpcOutcome, SessionRpcRequest,
|
ErrorCode, MutationCertainty, SessionRpcFailure, SessionRpcOutcome, SessionRpcRequest,
|
||||||
SessionRpcSuccess,
|
SessionRpcSuccess,
|
||||||
|
|
@ -217,6 +222,57 @@ pub fn outcome_identity(
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The epoch-derived identities of a behaviour trace, rewritten onto one reference epoch.
|
||||||
|
///
|
||||||
|
/// `step-v1` section 8 compares committed behaviour across runs, excluding wall time, request
|
||||||
|
/// ids "and other explicitly operational metadata". A resumed run's epoch is neither: it is
|
||||||
|
/// behaviour metadata, and `scope.epoch`, the batch id and every task event id are derived
|
||||||
|
/// from it. Comparing the two runs therefore means rewriting exactly those three things and
|
||||||
|
/// nothing else, which is what this does -- and it **fails** on anything it does not
|
||||||
|
/// recognise instead of passing it through, so a field that silently stopped being rebased
|
||||||
|
/// would fail the comparison rather than weaken it.
|
||||||
|
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||||
|
pub struct EpochRebase {
|
||||||
|
pub from: Id,
|
||||||
|
pub to: Id,
|
||||||
|
/// Every event identity the task issued under `from`, and the identity it has under `to`.
|
||||||
|
pub events: BTreeMap<Id, Id>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl EpochRebase {
|
||||||
|
/// Rewrites one behaviour record. An identity this rebase does not know is an error.
|
||||||
|
pub fn apply(&self, behaviour: &TraceBehaviour) -> Result<TraceBehaviour, String> {
|
||||||
|
if behaviour.scope.epoch != self.from {
|
||||||
|
return Err(format!(
|
||||||
|
"this behaviour was recorded in epoch {}, not {}",
|
||||||
|
behaviour.scope.epoch, self.from
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let mut out = behaviour.clone();
|
||||||
|
out.scope = Scope::new(&behaviour.scope.session_id, &self.to, behaviour.scope.step)
|
||||||
|
.map_err(|e| e.0)?;
|
||||||
|
let prefix = format!("batch-{}-", self.from);
|
||||||
|
let suffix = behaviour
|
||||||
|
.batch_id
|
||||||
|
.strip_prefix(&prefix)
|
||||||
|
.ok_or_else(|| format!("the batch id {} is not derived from {}", behaviour.batch_id, self.from))?;
|
||||||
|
out.batch_id = parse_id(&format!("batch-{}-{suffix}", self.to))?;
|
||||||
|
let map = |ids: &[Id]| -> Result<Vec<Id>, String> {
|
||||||
|
ids.iter()
|
||||||
|
.map(|id| {
|
||||||
|
self.events
|
||||||
|
.get(id)
|
||||||
|
.cloned()
|
||||||
|
.ok_or_else(|| format!("no rebased identity for the event {id}"))
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
};
|
||||||
|
out.outcome_ids = map(&behaviour.outcome_ids)?;
|
||||||
|
out.event_ids = map(&behaviour.event_ids)?;
|
||||||
|
Ok(out)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// One session phase transition, recorded whether or not it ends a step.
|
/// One session phase transition, recorded whether or not it ends a step.
|
||||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||||
pub struct PhaseTransition {
|
pub struct PhaseTransition {
|
||||||
|
|
@ -256,6 +312,28 @@ impl TraceLog {
|
||||||
.collect()
|
.collect()
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Every transition's behaviour, rebased onto one epoch and canonicalized.
|
||||||
|
///
|
||||||
|
/// This is the comparison a resumed run is held to: the same strings as
|
||||||
|
/// [`TraceLog::behavior`], with the epoch metadata accounted for and nothing else changed.
|
||||||
|
/// A resumed run's log holds transitions from two epochs -- the ones before the checkpoint
|
||||||
|
/// and the ones after the restore -- so a transition already recorded in `rebase.to` is
|
||||||
|
/// kept as it stands and one recorded in `rebase.from` is rewritten. A transition in a
|
||||||
|
/// third epoch is an error; there is no pass-through case.
|
||||||
|
pub fn behavior_rebased(&self, rebase: &EpochRebase) -> Result<Vec<String>, String> {
|
||||||
|
self.transitions
|
||||||
|
.iter()
|
||||||
|
.map(|t| {
|
||||||
|
let behaviour = if t.behaviour.scope.epoch == rebase.to {
|
||||||
|
t.behaviour.clone()
|
||||||
|
} else {
|
||||||
|
rebase.apply(&t.behaviour)?
|
||||||
|
};
|
||||||
|
canonicalize(&behaviour.to_json()).map_err(|e| e.0)
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
}
|
||||||
|
|
||||||
/// The phase path, as `from -> to` strings.
|
/// The phase path, as `from -> to` strings.
|
||||||
pub fn phase_path(&self) -> Vec<String> {
|
pub fn phase_path(&self) -> Vec<String> {
|
||||||
self.phases.iter().map(|p| format!("{} -> {}", p.from, p.to)).collect()
|
self.phases.iter().map(|p| format!("{} -> {}", p.from, p.to)).collect()
|
||||||
|
|
|
||||||
|
|
@ -597,7 +597,17 @@ async fn execute<E: WorkerEndpoint>(
|
||||||
fn classify_default(method: &str) -> Option<OpClass> {
|
fn classify_default(method: &str) -> Option<OpClass> {
|
||||||
match method {
|
match method {
|
||||||
"Agent.Prepare" | "Agent.Commit" | "Environment.Advance" => Some(OpClass::StepMutation),
|
"Agent.Prepare" | "Agent.Commit" | "Environment.Advance" => Some(OpClass::StepMutation),
|
||||||
"Agent.Initialize" | "Environment.Initialize" => Some(OpClass::Lifecycle),
|
// `ipc-v1` section 5 retains lifecycle *and capture* replies until
|
||||||
|
// `Worker.Acknowledge`. The restore methods join them: their replies carry a
|
||||||
|
// once-only token and, for an environment, the restored observation's artifact, and a
|
||||||
|
// duplicate domain request must replay that reply rather than stage or activate a
|
||||||
|
// second time. They are not step mutations -- they carry no committed step of their
|
||||||
|
// own and are not keyed by one.
|
||||||
|
"Agent.Initialize"
|
||||||
|
| "Environment.Initialize"
|
||||||
|
| "State.Capture"
|
||||||
|
| "State.StageRestore"
|
||||||
|
| "State.ActivateRestore" => Some(OpClass::Lifecycle),
|
||||||
"Worker.Hello" | "Worker.Status" | "Worker.Acknowledge" | "Worker.Shutdown" => {
|
"Worker.Hello" | "Worker.Status" | "Worker.Acknowledge" | "Worker.Shutdown" => {
|
||||||
Some(OpClass::ReadOnly)
|
Some(OpClass::ReadOnly)
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -121,6 +121,8 @@ async fn a_slow_participant_is_resolved_rather_than_failed(mode: ExecutionMode)
|
||||||
resolve: Duration::from_secs(20),
|
resolve: Duration::from_secs(20),
|
||||||
resolve_attempts: 4096,
|
resolve_attempts: 4096,
|
||||||
boot: Duration::from_secs(30),
|
boot: Duration::from_secs(30),
|
||||||
|
capture: Duration::from_secs(30),
|
||||||
|
durable: Duration::from_secs(60),
|
||||||
};
|
};
|
||||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||||
let reports = within("run", f.harness.coordinator.run(2))
|
let reports = within("run", f.harness.coordinator.run(2))
|
||||||
|
|
@ -191,6 +193,8 @@ async fn a_resolution_says_which_of_its_two_bounds_ended_it(mode: ExecutionMode)
|
||||||
resolve: Duration::from_millis(300),
|
resolve: Duration::from_millis(300),
|
||||||
resolve_attempts: 8192,
|
resolve_attempts: 8192,
|
||||||
boot: Duration::from_secs(30),
|
boot: Duration::from_secs(30),
|
||||||
|
capture: Duration::from_secs(30),
|
||||||
|
durable: Duration::from_secs(60),
|
||||||
};
|
};
|
||||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||||
let started = Instant::now();
|
let started = Instant::now();
|
||||||
|
|
@ -221,6 +225,8 @@ async fn a_resolution_says_which_of_its_two_bounds_ended_it(mode: ExecutionMode)
|
||||||
resolve: Duration::from_secs(600),
|
resolve: Duration::from_secs(600),
|
||||||
resolve_attempts: 3,
|
resolve_attempts: 3,
|
||||||
boot: Duration::from_secs(30),
|
boot: Duration::from_secs(30),
|
||||||
|
capture: Duration::from_secs(30),
|
||||||
|
durable: Duration::from_secs(60),
|
||||||
};
|
};
|
||||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||||
let failure = within("step", f.harness.coordinator.step())
|
let failure = within("step", f.harness.coordinator.step())
|
||||||
|
|
|
||||||
1002
services/flysim/crates/fly-session/tests/state.rs
Normal file
1002
services/flysim/crates/fly-session/tests/state.rs
Normal file
File diff suppressed because it is too large
Load diff
Loading…
Add table
Reference in a new issue