diff --git a/docs/design/session-framework/checkpoint-envelope-v1.md b/docs/design/session-framework/checkpoint-envelope-v1.md index d58ed98..6e5017a 100644 --- a/docs/design/session-framework/checkpoint-envelope-v1.md +++ b/docs/design/session-framework/checkpoint-envelope-v1.md @@ -100,11 +100,21 @@ names; a manifest missing any of them is not a complete checkpoint. | `compositionDigest` | Coordinator scheduler and configuration identity | | `portMap` | The exact port-to-agent map, `[{portId, agentId}]` | | `compatibility` | Backend, content, patch, controller, parser and state-format identities | -| `agents` | Per agent: profile, dataset and model identities, resolved seed, tick count, remainder and the payload name holding its state | +| `agents` | Per agent: profile, dataset, **index** and model identities, resolved seed, tick count, remainder and the payload name holding its state | | `coordinator` | Task ledger, prior world inspection, per-agent executor state, admission state and event watermarks, each as a payload name or an inline value | | `helperState` | External-helper state required for exact resume, as payload names | | `payloads` | `[{name, byteLength, digest}]`, mirroring the payload table | +**Amendment, 2026-09-22 (PUBLISH-01).** The `agents` row gains `indexDigest`, the index the +agent attested to at `Agent.Initialize`, and it joins that agent's compatibility identity. +Without it a replacement fly that built another graph -- the same dataset, the same neuron +count, another index -- passed the group check and was then published under its predecessor's +`indexDigest`, which is a graph identity crossing a recovery and exactly what section 5's rules +exist to prevent. It is recorded from the worker's attestation rather than recomputed from the +dataset, because the point is that the two can disagree. `envelopeVersion` stays `1`, which the +required-manifest-field rule below allows only while no production `FLYSESS1` file exists; once +one does, adding a required manifest field must bump it. + **Amendment, 2026-09-22 (STATE-01).** The table above names a holder for every payload except the environment's own, although section 6's fixture has one (`world`) and a group install has to map it by name like any other participant's. The manifest therefore also records: diff --git a/docs/design/session-framework/publishing-v1.md b/docs/design/session-framework/publishing-v1.md index 01cd30b..7258845 100644 --- a/docs/design/session-framework/publishing-v1.md +++ b/docs/design/session-framework/publishing-v1.md @@ -42,6 +42,20 @@ Example addresses (chosen by composition, not recognized by router code): | `app.pokemon.cues` | Pub/sub: application narrative/presentation events under declared delivery policy | | `app.pokemon` | RPC: application queries/admission, e.g. restore UI state or request a supported effect | +**Amendment, 2026-09-22 (PUBLISH-01).** The repair path above needs exact methods, and +"exact methods require session API schemas" left the row unbuildable. The session registers +one **read-only** service, `session..query`, with exactly two methods, both ordinary +[session RPCs](ipc-v1.md) answering from what the session already published: +`Session.GetDescriptor` takes an optional `{revision: U64}` and returns that `SessionDescriptor` +or, with no revision, the newest; `Session.GetSnapshot` takes no parameters and returns the +latest `CommittedSnapshot`. A revision the session never published is `IDENTITY_MISMATCH`, not +an empty answer. Nothing on this service mutates, selects a participant or reaches a worker, so +it is not the controller API section 7 rules out; adding a third method that did would be. +These two names are **internal and provisional**: they are what the internal boundary needs in +order to be buildable now, and the later public v2 step is free to rename them, supersede them +or expose a different repair surface entirely. Nothing about them is browser-facing, and the +public step does not inherit them by default merely because they landed first. + Descriptor revisions and scope link observations to schemas. Cross-topic ordering is not guaranteed; a subscriber receiving an unknown descriptor revision must fetch it through the application/session query contract or buffer a bounded number of snapshots, not infer shape. @@ -77,6 +91,16 @@ interface CommittedSnapshot { } ``` +**Amendment, 2026-09-22 (PUBLISH-01).** "Null at initial boundary 0" is the rule for a +boundary this epoch *produced*. A group restore ([state/media](state-media-v1.md) section 5) +re-establishes a committed boundary `k > 0` that this epoch did not run a transition into, and +the abandoned epoch's decisions are not this session's to republish under a new epoch. So the +rule is: `selectedDecision` and `appliedControls` are null at boundary 0 and at a boundary +*installed* by a restore, present otherwise, and always **together** and for **every agent or +none**. A snapshot where one fly carries an action and another does not would be two different +boundaries in one value, and is refused. Without this, the section 6 requirement to publish the +recovery could not be met at all: the restored boundary's snapshot would be unrepresentable. + Publish only after all agent commits establish Ready(k). Decisions/controls describe the transition ending at that boundary, null at initial boundary 0. Health updates are separate and never claim an uncommitted future boundary. Every transient media reference is a declared diff --git a/docs/design/session-framework/state-media-v1.md b/docs/design/session-framework/state-media-v1.md index 0061761..0285287 100644 --- a/docs/design/session-framework/state-media-v1.md +++ b/docs/design/session-framework/state-media-v1.md @@ -117,6 +117,21 @@ If it violates configured resource policy, disconnect/restart that observer inst live data or silently skipping simulation input. Global store exhaustion is an explicit fault or pause condition; the router cannot guess that a particular live object is disposable. +**Amendment, 2026-09-22 (PUBLISH-01).** "Disconnect/restart that observer" names an action +no participant can take under [Flybus v1](bus-v1.md). Section 5 there makes publish admission +all or nothing -- "for a bounded subscriber overflow, reject the **whole** publish; no partial +fan-out or retained-latest update" -- and the router exposes no per-subscriber eviction, so a +session meeting a full bounded queue cannot drop that one subscriber and deliver to the rest. +The realisable reading, which the session now implements, is three-part: observation topics +are published `latest`, and a latest subscriber can never refuse a publication (it loses its +own queued value and is told how many by `replaced`); a bounded subscriber's refusal, which +`bus-v1` section 6 explicitly permits, is a named and counted publication outcome that takes +no world step, stalls nothing and fences no epoch, and the exact value stays recoverable +through the [publishing-v1](publishing-v1.md) section 2 query path; and disconnecting the +offender is an operator action against the topic the ledger names, not something the session +performs. A per-subscriber drop would need a router operation Flybus v1 does not have, and +inventing one here would be a transport change written into the wrong document. + No coordinator tracks per-reader socket acknowledgments or calls a producer's reclaim method. The SDK and bus perform that bookkeeping. File-backed immutable mappings are safe after unlink; physical pages disappear when all OS mappings close. Pooled reuse is deferred until diff --git a/docs/design/session-framework/workers-v1.md b/docs/design/session-framework/workers-v1.md index 996d5cb..9692dc6 100644 --- a/docs/design/session-framework/workers-v1.md +++ b/docs/design/session-framework/workers-v1.md @@ -96,9 +96,29 @@ interface AgentInitializeResult { warmupTicks: U64; committedStep: U64; // committedStep == "0" decisionContextDigest: Digest; telemetry: AgentTelemetry; + graph: AgentGraph; +} +interface AgentGraph { + datasetDigest: Digest; indexDigest: Digest; neuronCount: U64; + rateRoles: Id[]; // <=64, unique; AgentTelemetry.rates is in this order + supportedStimuli: Id[]; // <=64, unique; an undeclared kind is UNSUPPORTED } ``` +**Amendment, 2026-09-22 (PUBLISH-01).** `AgentInitializeResult` gains `graph`, because +[publishing-v1](publishing-v1.md) section 3 requires `datasetDigest`, `indexDigest`, +`neuronCount`, `rateRoles` and `supportedStimuli` in every published `AgentDescriptor` and no +worker method carried any of them. Without this the only available source is the composition +that asked for the agent, so a descriptor could only ever agree with itself and the section 3 +rule that "geometry/spike mapping requires indexDigest, not merely the same number of neurons" +would have nothing to compare. Initialize is where the agent has just loaded its dataset and +built its index, so the attestation belongs there. `rateRoles` is the "profile-defined order" +section 1 already requires `AgentTelemetry.rates` to be in, and the result is refused when the +two disagree; `supportedStimuli` is the profile capability section 1 already requires a +stimulus kind to resolve through, and a kind outside it is refused with `UNSUPPORTED` before +the model is touched. It changes `contractDigest`, which [session RPC](ipc-v1.md) section 4 +already provides for. + **Amendment, 2026-09-22 (SESSION-02).** `HelloResult.limits` gains `workerThreads`, an integer >=1 reporting the allocation the launcher started that worker within, because "within launcher allocation" above had no wire-level proof: the launcher passes the number to diff --git a/packages/session-types/src/publishing.ts b/packages/session-types/src/publishing.ts index 855460a..104e777 100644 --- a/packages/session-types/src/publishing.ts +++ b/packages/session-types/src/publishing.ts @@ -16,6 +16,7 @@ import { type EnvironmentDescriptor, MAX_AGENTS, MAX_RATE_ROLES, + MAX_SUPPORTED_STIMULI, type PortControl, findPort, readAgentTelemetry, @@ -26,8 +27,9 @@ import { validateTelemetryRoles, } from './workers'; +export { MAX_SUPPORTED_STIMULI } from './workers'; + /** Not stated by a document; this crate's choices, published in the schema set. */ -export const MAX_SUPPORTED_STIMULI = 64; export const MAX_ASSETS = 64; export const MAX_SNAPSHOT_EVENTS = 64; @@ -166,16 +168,24 @@ export function readCommittedSnapshot(value: unknown): CommittedSnapshot { const atBoundaryZero = u64(scope.step) === 0n; for (const agent of agents) { // "Decisions/controls describe the transition ending at that boundary, null at initial - // boundary 0." (publishing-v1 section 3) + // boundary 0." (publishing-v1 section 3, and its 2026-09-22 amendment for a boundary that + // was installed rather than produced.) if (atBoundaryZero && (agent.selectedDecision !== null || agent.appliedControls !== null)) { fail('CommittedSnapshot: at boundary 0 selectedDecision and appliedControls are null'); } - if (!atBoundaryZero && (agent.selectedDecision === null || agent.appliedControls === null)) { + if ((agent.selectedDecision === null) !== (agent.appliedControls === null)) { fail( - 'CommittedSnapshot: past boundary 0 every agent has a decision and applied controls', + 'CommittedSnapshot: selectedDecision and appliedControls are null together or present together', ); } } + // A boundary is produced by a transition or installed by one, and the whole snapshot says + // which: every agent carries the transition that ended here, or none does. + if (agents.some((a) => (a.selectedDecision === null) !== (agents[0].selectedDecision === null))) { + fail( + 'CommittedSnapshot: either every agent carries the transition that ended here, or none does', + ); + } return { descriptorRevision, publisherIncarnation, diff --git a/packages/session-types/src/workers.ts b/packages/session-types/src/workers.ts index 6f93a9d..904c5f2 100644 --- a/packages/session-types/src/workers.ts +++ b/packages/session-types/src/workers.ts @@ -35,6 +35,8 @@ import { readSchemaRef, readTypedValue, readNullableTypedValue } from './common' export const MAX_AGENTS = 4; export const MAX_PORTS = 4; export const MAX_RATE_ROLES = 64; +/** Declared stimulus kinds per agent. Not a stated bound; recorded in the schema set. */ +export const MAX_SUPPORTED_STIMULI = 64; export const MAX_STIMULI = 64; export const MAX_REWARDS = 64; export const MAX_BUTTONS = 32; @@ -257,6 +259,14 @@ export interface AgentInitializeParams { workerThreads: number; } +export interface AgentGraph { + datasetDigest: Digest; + indexDigest: Digest; + neuronCount: U64; + rateRoles: Id[]; + supportedStimuli: Id[]; +} + export interface AgentInitializeResult { agentId: Id; profileDigest: Digest; @@ -265,6 +275,7 @@ export interface AgentInitializeResult { committedStep: U64; decisionContextDigest: Digest; telemetry: AgentTelemetry; + graph: AgentGraph; } export interface PrepareParams { @@ -313,6 +324,21 @@ export function readAgentInitializeParams(value: unknown): AgentInitializeParams return params; } +export function readAgentGraph(value: unknown): AgentGraph { + const reader = new Reader(value, 'AgentGraph'); + const graph: AgentGraph = { + datasetDigest: reader.digest('datasetDigest'), + indexDigest: reader.digest('indexDigest'), + neuronCount: reader.u64('neuronCount'), + rateRoles: reader.idList('rateRoles', 0, MAX_RATE_ROLES), + supportedStimuli: reader.idList('supportedStimuli', 0, MAX_SUPPORTED_STIMULI), + }; + reader.finish(); + requireUnique(graph.rateRoles, 'AgentGraph.rateRoles'); + requireUnique(graph.supportedStimuli, 'AgentGraph.supportedStimuli'); + return graph; +} + export function readAgentInitializeResult(value: unknown): AgentInitializeResult { const reader = new Reader(value, 'AgentInitializeResult'); const result: AgentInitializeResult = { @@ -323,12 +349,15 @@ export function readAgentInitializeResult(value: unknown): AgentInitializeResult committedStep: reader.u64('committedStep'), decisionContextDigest: reader.digest('decisionContextDigest'), telemetry: readAgentTelemetry(reader.value('telemetry')), + graph: readAgentGraph(reader.value('graph')), }; reader.finish(); requirePositiveRational(result.tickDuration, 'AgentInitializeResult.tickDuration'); if (u64(result.committedStep) !== 0n) { fail('AgentInitializeResult: committedStep must be "0"'); } + // The rates a worker reports and the role order it declares are one statement. + validateTelemetryRoles(result.telemetry, result.graph.rateRoles); return result; } diff --git a/services/flysim/crates/fly-session-types/fixtures/contract-digest.json b/services/flysim/crates/fly-session-types/fixtures/contract-digest.json index 8845d36..90e4419 100644 --- a/services/flysim/crates/fly-session-types/fixtures/contract-digest.json +++ b/services/flysim/crates/fly-session-types/fixtures/contract-digest.json @@ -1,9 +1,9 @@ { "description": "contractDigest is the SHA-256 of the canonical schema set in schema-set.json.", - "contractDigest": "d8f29a49b5df05ad8f75f7f5790a3f8cde9c5ad23a685137474c649c3c9da36d", + "contractDigest": "7f4b11d6737e5097c6889657479527174ddf496e50b60322b281e6c7490bcb4a", "schemaSetVersion": 1, - "schemaSetBytes": 26814, - "types": 53, + "schemaSetBytes": 27470, + "types": 54, "enums": 11, "limits": 26 } diff --git a/services/flysim/crates/fly-session-types/fixtures/invalid.json b/services/flysim/crates/fly-session-types/fixtures/invalid.json index cbce2b4..b43edec 100644 --- a/services/flysim/crates/fly-session-types/fixtures/invalid.json +++ b/services/flysim/crates/fly-session-types/fixtures/invalid.json @@ -2746,6 +2746,19 @@ "changed": "2", "signal": 0.5 } + }, + "graph": { + "datasetDigest": "6c0af1f0784ef63a393ee77d614e8246c625051360f3f1a48838374c5d355b52", + "indexDigest": "52a7a9b441f686ebea398e3d65dbaa87b4fde4f8b2231fff951700548f6a4e42", + "neuronCount": "139255", + "rateRoles": [ + "kenyon", + "mbon" + ], + "supportedStimuli": [ + "sugar", + "shock" + ] } }, "reason": "initialization establishes Ready(0)" @@ -2782,10 +2795,256 @@ "changed": "2", "signal": 0.5 } + }, + "graph": { + "datasetDigest": "6c0af1f0784ef63a393ee77d614e8246c625051360f3f1a48838374c5d355b52", + "indexDigest": "52a7a9b441f686ebea398e3d65dbaa87b4fde4f8b2231fff951700548f6a4e42", + "neuronCount": "139255", + "rateRoles": [ + "kenyon", + "mbon" + ], + "supportedStimuli": [ + "sugar", + "shock" + ] } }, "reason": "durations are positive" }, + { + "name": "agent initialize result without a graph identity", + "type": "AgentInitializeResult", + "value": { + "agentId": "fly-a", + "profileDigest": "1900eab6c028483d7126599ee6f50de0d27907b5c65fa90524580b4b0f9852b0", + "tickDuration": { + "numerator": "1000000", + "denominator": "1" + }, + "warmupTicks": "2500", + "committedStep": "0", + "decisionContextDigest": "ea7792a26f405e2ae9c6f49ca93bbe6076ceac0a1fc53d83426c7d7f2d9377e4", + "telemetry": { + "brainTicks": "2500", + "populationRateHz": 12.5, + "rates": [ + { + "roleId": "kenyon", + "hz": 3.25 + }, + { + "roleId": "mbon", + "hz": 0.0 + } + ], + "learning": { + "enabled": true, + "updates": "4", + "changed": "2", + "signal": 0.5 + } + } + }, + "reason": "publishing-v1 3 needs datasetDigest, indexDigest, neuronCount, rateRoles and supportedStimuli, and only the worker knows them" + }, + { + "name": "agent initialize result whose index digest is not a digest", + "type": "AgentInitializeResult", + "value": { + "agentId": "fly-a", + "profileDigest": "1900eab6c028483d7126599ee6f50de0d27907b5c65fa90524580b4b0f9852b0", + "tickDuration": { + "numerator": "1000000", + "denominator": "1" + }, + "warmupTicks": "2500", + "committedStep": "0", + "decisionContextDigest": "ea7792a26f405e2ae9c6f49ca93bbe6076ceac0a1fc53d83426c7d7f2d9377e4", + "telemetry": { + "brainTicks": "2500", + "populationRateHz": 12.5, + "rates": [ + { + "roleId": "kenyon", + "hz": 3.25 + }, + { + "roleId": "mbon", + "hz": 0.0 + } + ], + "learning": { + "enabled": true, + "updates": "4", + "changed": "2", + "signal": 0.5 + } + }, + "graph": { + "datasetDigest": "6c0af1f0784ef63a393ee77d614e8246c625051360f3f1a48838374c5d355b52", + "indexDigest": "not-a-digest", + "neuronCount": "139255", + "rateRoles": [ + "kenyon", + "mbon" + ], + "supportedStimuli": [ + "sugar", + "shock" + ] + } + }, + "reason": "digests are 64 lowercase hex digits" + }, + { + "name": "agent initialize result whose rates are not in the declared role order", + "type": "AgentInitializeResult", + "value": { + "agentId": "fly-a", + "profileDigest": "1900eab6c028483d7126599ee6f50de0d27907b5c65fa90524580b4b0f9852b0", + "tickDuration": { + "numerator": "1000000", + "denominator": "1" + }, + "warmupTicks": "2500", + "committedStep": "0", + "decisionContextDigest": "ea7792a26f405e2ae9c6f49ca93bbe6076ceac0a1fc53d83426c7d7f2d9377e4", + "telemetry": { + "brainTicks": "2500", + "populationRateHz": 12.5, + "rates": [ + { + "roleId": "kenyon", + "hz": 3.25 + }, + { + "roleId": "mbon", + "hz": 0.0 + } + ], + "learning": { + "enabled": true, + "updates": "4", + "changed": "2", + "signal": 0.5 + } + }, + "graph": { + "datasetDigest": "6c0af1f0784ef63a393ee77d614e8246c625051360f3f1a48838374c5d355b52", + "indexDigest": "52a7a9b441f686ebea398e3d65dbaa87b4fde4f8b2231fff951700548f6a4e42", + "neuronCount": "139255", + "rateRoles": [ + "mbon", + "kenyon" + ], + "supportedStimuli": [ + "sugar", + "shock" + ] + } + }, + "reason": "AgentTelemetry.rates is in graph.rateRoles order" + }, + { + "name": "agent initialize result declaring a role it reports no rate for", + "type": "AgentInitializeResult", + "value": { + "agentId": "fly-a", + "profileDigest": "1900eab6c028483d7126599ee6f50de0d27907b5c65fa90524580b4b0f9852b0", + "tickDuration": { + "numerator": "1000000", + "denominator": "1" + }, + "warmupTicks": "2500", + "committedStep": "0", + "decisionContextDigest": "ea7792a26f405e2ae9c6f49ca93bbe6076ceac0a1fc53d83426c7d7f2d9377e4", + "telemetry": { + "brainTicks": "2500", + "populationRateHz": 12.5, + "rates": [ + { + "roleId": "kenyon", + "hz": 3.25 + }, + { + "roleId": "mbon", + "hz": 0.0 + } + ], + "learning": { + "enabled": true, + "updates": "4", + "changed": "2", + "signal": 0.5 + } + }, + "graph": { + "datasetDigest": "6c0af1f0784ef63a393ee77d614e8246c625051360f3f1a48838374c5d355b52", + "indexDigest": "52a7a9b441f686ebea398e3d65dbaa87b4fde4f8b2231fff951700548f6a4e42", + "neuronCount": "139255", + "rateRoles": [ + "kenyon", + "mbon", + "pn" + ], + "supportedStimuli": [ + "sugar", + "shock" + ] + } + }, + "reason": "AgentTelemetry.rates is exactly graph.rateRoles" + }, + { + "name": "agent initialize result repeating a supported stimulus", + "type": "AgentInitializeResult", + "value": { + "agentId": "fly-a", + "profileDigest": "1900eab6c028483d7126599ee6f50de0d27907b5c65fa90524580b4b0f9852b0", + "tickDuration": { + "numerator": "1000000", + "denominator": "1" + }, + "warmupTicks": "2500", + "committedStep": "0", + "decisionContextDigest": "ea7792a26f405e2ae9c6f49ca93bbe6076ceac0a1fc53d83426c7d7f2d9377e4", + "telemetry": { + "brainTicks": "2500", + "populationRateHz": 12.5, + "rates": [ + { + "roleId": "kenyon", + "hz": 3.25 + }, + { + "roleId": "mbon", + "hz": 0.0 + } + ], + "learning": { + "enabled": true, + "updates": "4", + "changed": "2", + "signal": 0.5 + } + }, + "graph": { + "datasetDigest": "6c0af1f0784ef63a393ee77d614e8246c625051360f3f1a48838374c5d355b52", + "indexDigest": "52a7a9b441f686ebea398e3d65dbaa87b4fde4f8b2231fff951700548f6a4e42", + "neuronCount": "139255", + "rateRoles": [ + "kenyon", + "mbon" + ], + "supportedStimuli": [ + "sugar", + "sugar" + ] + } + }, + "reason": "supportedStimuli is unique" + }, { "name": "hello result for an agent without agent-step-v1", "type": "HelloResult", @@ -3909,7 +4168,179 @@ }, "eventIds": [] }, - "reason": "a committed transition has applied controls" + "reason": "selectedDecision and appliedControls are null together or present together" + }, + { + "name": "snapshot where one agent carries the transition and another does not", + "type": "CommittedSnapshot", + "value": { + "descriptorRevision": "7", + "publisherIncarnation": "pub-1", + "scope": { + "sessionId": "demo", + "epoch": "epoch-1", + "step": "42" + }, + "episodeId": "episode-1", + "sequence": "42", + "worldTime": { + "numerator": "700000000", + "denominator": "1" + }, + "agents": [ + { + "agentId": "fly-a", + "telemetry": { + "brainTicks": "2534", + "populationRateHz": 12.5, + "rates": [ + { + "roleId": "kenyon", + "hz": 3.25 + }, + { + "roleId": "mbon", + "hz": 0.0 + } + ], + "learning": { + "enabled": true, + "updates": "4", + "changed": "2", + "signal": 0.5 + } + }, + "selectedDecision": { + "schema": { + "id": "gameboy.intent.v1", + "version": 1, + "digest": "e70451b26fb5462ec63729d8355912dad01c3963a95bec0e6a8452375717ad9d" + }, + "value": { + "press": "a" + } + }, + "appliedControls": { + "portId": "port-1", + "buttons": [ + { + "id": "a", + "down": true + }, + { + "id": "b", + "down": false + }, + { + "id": "start", + "down": false + }, + { + "id": "select", + "down": false + }, + { + "id": "up", + "down": false + }, + { + "id": "down", + "down": false + }, + { + "id": "left", + "down": false + }, + { + "id": "right", + "down": false + } + ], + "axes": [ + { + "id": "stick-x", + "value": 0.0 + }, + { + "id": "trigger", + "value": 0.0 + } + ] + } + }, + { + "agentId": "fly-b", + "telemetry": { + "brainTicks": "2534", + "populationRateHz": 12.5, + "rates": [ + { + "roleId": "kenyon", + "hz": 3.25 + }, + { + "roleId": "mbon", + "hz": 0.0 + } + ], + "learning": { + "enabled": true, + "updates": "4", + "changed": "2", + "signal": 0.5 + } + }, + "selectedDecision": null, + "appliedControls": null + } + ], + "progress": { + "schema": { + "id": "pokemon.progress.v1", + "version": 1, + "digest": "80fa973157f334bb3208dfff6f7a2f298e79f73f07c72c33f5725975c6cff8e1" + }, + "value": { + "rank": 10 + } + }, + "media": { + "views": [ + { + "viewId": "screen", + "producedStep": "42", + "pixels": { + "storeId": "store-1", + "artifactId": "frame-1", + "generation": "1", + "byteLength": "92160", + "contentType": "image/x-rgba8", + "digest": null + } + } + ], + "audio": [ + { + "streamId": "mix", + "firstSample": "33600", + "sampleFrames": 800, + "samples": { + "storeId": "store-1", + "artifactId": "audio-1", + "generation": "1", + "byteLength": "6400", + "contentType": "audio/x-f32le", + "digest": null + }, + "discontinuity": false + } + ] + }, + "eventIds": [ + "evt-1" + ] + }, + "reason": "a boundary is produced or installed for the whole composition, never per agent" }, { "name": "trace whose commit acknowledgment is the old boundary", diff --git a/services/flysim/crates/fly-session-types/fixtures/schema-set.json b/services/flysim/crates/fly-session-types/fixtures/schema-set.json index e1a175f..7078001 100644 --- a/services/flysim/crates/fly-session-types/fixtures/schema-set.json +++ b/services/flysim/crates/fly-session-types/fixtures/schema-set.json @@ -1 +1 @@ -{"contract":"fly-session-types","enums":[{"members":["f32le-interleaved"],"name":"AudioFormat","source":"state-media-v1 2"},{"members":["bipolar","unit"],"name":"AxisRange","source":"workers-v1 3"},{"members":["fixed-build","unverified"],"name":"Determinism","source":"workers-v1 3"},{"members":["terminal"],"name":"EpisodeRequestKind","source":"workers-v1 4"},{"members":["INVALID_ARGUMENT","UNSUPPORTED","IDENTITY_MISMATCH","STALE_EPOCH","STALE_STEP","FUTURE_STEP","INVALID_PHASE","CONFLICT","IN_PROGRESS","BUSY","BUFFER_INVALID","RESULT_EXPIRED","INCOMPATIBLE_STATE","BACKEND_FAILURE","INTERNAL"],"name":"ErrorCode","source":"ipc-v1 7"},{"members":["none","applied","unknown"],"name":"MutationCertainty","source":"ipc-v1 3"},{"members":["exact-checkpoint","episode-restart"],"name":"Recovery","source":"workers-v1 3"},{"members":["agent","environment","coordinator"],"name":"Role","source":"ipc-v1 4"},{"members":["lockstep-v1"],"name":"SchedulerId","source":"publishing-v1 3"},{"members":["rgba8"],"name":"ViewFormat","source":"state-media-v1 2"},{"members":["uninitialized","ready","preparing","prepared","advancing","committing","capturing","staged-restore","restoring","failed","stopping"],"name":"WorkerState","source":"ipc-v1 4"}],"limits":[{"name":"maxAcknowledge","source":"ipc-v1 5","value":16},{"name":"maxAgents","source":"ipc-v1 2","value":4},{"name":"maxAssets","source":"crate","value":64},{"name":"maxAttachments","source":"bus-v1 4","value":32},{"name":"maxAudioStreams","source":"crate","value":8},{"name":"maxAxes","source":"workers-v1 3","value":16},{"name":"maxButtons","source":"workers-v1 3","value":32},{"name":"maxCapabilities","source":"crate","value":32},{"name":"maxEngineFrameLength","source":"workers-v1 3","value":64},{"name":"maxEnvelopeBytes","source":"bus-v1 4","value":65536},{"name":"maxMessageCodePoints","source":"ipc-v1 7","value":512},{"name":"maxObservationDelaySteps","source":"state-media-v1 2","value":8},{"name":"maxPixelAspectPart","source":"state-media-v1 2","value":65535},{"name":"maxPorts","source":"ipc-v1 2","value":4},{"name":"maxRateRoles","source":"ipc-v1 2","value":64},{"name":"maxRewardsPerOperation","source":"workers-v1 1","value":64},{"name":"maxSampleFrames","source":"state-media-v1 2","value":192000},{"name":"maxSchemaVersion","source":"ipc-v1 2","value":65535},{"name":"maxSnapshotEvents","source":"crate","value":64},{"name":"maxStimuliPerOperation","source":"workers-v1 1","value":64},{"name":"maxSupportedMajors","source":"crate","value":8},{"name":"maxSupportedStimuli","source":"crate","value":64},{"name":"maxTypedValueBytes","source":"ipc-v1 2","value":32768},{"name":"maxViewDimension","source":"state-media-v1 2","value":4096},{"name":"maxViews","source":"workers-v1 1","value":8},{"name":"maxWorkerThreads","source":"workers-v1 2","value":4096}],"scalars":{"ArtifactIdentity":"storeId, artifactId and generation of a bus ArtifactRef","BusCallId":"call-","Digest":"64 lowercase hexadecimal digits (SHA-256)","DomainRequestId":"req-","Id":"^[a-z0-9][a-z0-9._-]{0,63}$","OwnerToken":"dlv- or own-","U64":"\"0\" or [1-9][0-9]*, <= 18446744073709551615"},"types":[{"fields":[{"constraint":"1..=16, unique","kind":"array","name":"requestIds","required":true}],"name":"AcknowledgeParams","source":"ipc-v1 5"},{"fields":[{"constraint":"<= 16, unique, a subset of the request","kind":"array","name":"acknowledged","required":true}],"name":"AcknowledgeResult","source":"ipc-v1 5"},{"fields":[{"constraint":"","kind":"Id","name":"restoreToken","required":true}],"name":"ActivateRestoreParams","source":"state-media-v1 5"},{"fields":[{"constraint":"","kind":"U64","name":"committedStep","required":true},{"constraint":"","kind":"Id","name":"checkpointId","required":true},{"constraint":"required from an environment, null from an agent","kind":"WorldObservation|null","name":"observation","required":false}],"name":"ActivateRestoreResult","source":"state-media-v1 5"},{"fields":[{"constraint":"unique within an epoch","kind":"Id","name":"batchId","required":true},{"constraint":"one complete batch: every declared port once, descriptor order","kind":"array","name":"controls","required":true}],"name":"AdvanceParams","source":"workers-v1 3"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"k+1","kind":"U64","name":"committedStep","required":true},{"constraint":"","kind":"Digest","name":"decisionContextDigest","required":true},{"constraint":"","kind":"AgentTelemetry","name":"telemetry","required":true}],"name":"AgentCommitResult","source":"workers-v1 2"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"","kind":"Id","name":"portId","required":true},{"constraint":"","kind":"Digest","name":"profileDigest","required":true},{"constraint":"","kind":"Digest","name":"datasetDigest","required":true},{"constraint":"geometry mapping needs this, not neuronCount","kind":"Digest","name":"indexDigest","required":true},{"constraint":"","kind":"U64","name":"neuronCount","required":true},{"constraint":"<= 64, unique","kind":"array","name":"rateRoles","required":true},{"constraint":"unique","kind":"array","name":"supportedStimuli","required":true}],"name":"AgentDescriptor","source":"publishing-v1 3"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"","kind":"AssetRef","name":"profile","required":true},{"constraint":"signed 32-bit","kind":"int","name":"seed","required":true},{"constraint":"","kind":"SensoryInput","name":"initialInput","required":true},{"constraint":"","kind":"TypedValue","name":"initialDecisionContext","required":true},{"constraint":">= 1","kind":"int","name":"workerThreads","required":true}],"name":"AgentInitializeParams","source":"workers-v1 2"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"","kind":"Digest","name":"profileDigest","required":true},{"constraint":"positive","kind":"RationalNs","name":"tickDuration","required":true},{"constraint":"","kind":"U64","name":"warmupTicks","required":true},{"constraint":"\"0\"","kind":"U64","name":"committedStep","required":true},{"constraint":"","kind":"Digest","name":"decisionContextDigest","required":true},{"constraint":"","kind":"AgentTelemetry","name":"telemetry","required":true}],"name":"AgentInitializeResult","source":"workers-v1 2"},{"fields":[{"constraint":"","kind":"U64","name":"brainTicks","required":true},{"constraint":"finite and nonnegative","kind":"number","name":"populationRateHz","required":true},{"constraint":"<= 64, unique roleId, profile order, finite nonnegative hz","kind":"array<{roleId:Id,hz:number}>","name":"rates","required":true},{"constraint":"changed <= updates; signal finite","kind":"{enabled:bool,updates:U64,changed:U64,signal:number}","name":"learning","required":true}],"name":"AgentTelemetry","source":"workers-v1 1"},{"fields":[{"constraint":"","kind":"Id","name":"id","required":true},{"constraint":"","kind":"Digest","name":"digest","required":true},{"constraint":"positive","kind":"U64","name":"byteLength","required":true},{"constraint":"","kind":"Id","name":"format","required":true}],"name":"AssetRef","source":"workers-v1 1"},{"fields":[{"constraint":"","kind":"Id","name":"streamId","required":true},{"constraint":"8000..=192000","kind":"int","name":"sampleRate","required":true},{"constraint":"1..=8","kind":"int","name":"channels","required":true},{"constraint":"","kind":"AudioFormat","name":"format","required":true}],"name":"AudioDescriptor","source":"state-media-v1 2"},{"fields":[{"constraint":"","kind":"Id","name":"streamId","required":true},{"constraint":"no overlap or rewind within an epoch","kind":"U64","name":"firstSample","required":true},{"constraint":"0..=192000","kind":"int","name":"sampleFrames","required":true},{"constraint":"byteLength == sampleFrames x channels x 4, finite f32","kind":"ArtifactRef","name":"samples","required":true},{"constraint":"true on the first chunk after restore","kind":"bool","name":"discontinuity","required":true}],"name":"AudioRef","source":"state-media-v1 2"},{"fields":[{"constraint":"","kind":"Id","name":"checkpointId","required":true}],"name":"CaptureParams","source":"state-media-v1 5"},{"fields":[{"constraint":"","kind":"Id","name":"checkpointId","required":true},{"constraint":"the committed boundary","kind":"U64","name":"boundary","required":true},{"constraint":"","kind":"Digest","name":"compatibilityDigest","required":true},{"constraint":"listed attachment; digest required","kind":"ArtifactRef","name":"payload","required":true}],"name":"CaptureResult","source":"state-media-v1 5"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"","kind":"DomainRequestId","name":"preparedRequestId","required":true},{"constraint":"boundary == scope.step + 1","kind":"SensoryInput","name":"nextInput","required":true},{"constraint":"","kind":"TypedValue","name":"nextDecisionContext","required":true},{"constraint":"<= 64, unique eventId, order retained","kind":"array","name":"rewards","required":true},{"constraint":"<= 64, unique id, order retained","kind":"array","name":"taskStimulations","required":true}],"name":"CommitParams","source":"workers-v1 2"},{"fields":[{"constraint":"","kind":"U64","name":"descriptorRevision","required":true},{"constraint":"","kind":"Id","name":"publisherIncarnation","required":true},{"constraint":"the committed boundary","kind":"Scope","name":"scope","required":true},{"constraint":"","kind":"Id","name":"episodeId","required":true},{"constraint":"monotonic within publisherIncarnation","kind":"U64","name":"sequence","required":true},{"constraint":"","kind":"RationalNs","name":"worldTime","required":true},{"constraint":"1..=4, unique agentId, telemetry in profile role order","kind":"array","name":"agents","required":true},{"constraint":"","kind":"TypedValue","name":"progress","required":true},{"constraint":"declared attachments held through publication admission","kind":"{views:array,audio:array}","name":"media","required":true},{"constraint":"unique, task order","kind":"array","name":"eventIds","required":true}],"name":"CommittedSnapshot","source":"publishing-v1 3"},{"fields":[{"constraint":"","kind":"SchemaRef","name":"schema","required":true},{"constraint":"<= 32, unique, fixed order","kind":"array","name":"buttons","required":true},{"constraint":"<= 16, unique id, neutral inside its range","kind":"array<{id:Id,range:AxisRange,neutral:number}>","name":"axes","required":true}],"name":"ControllerSchema","source":"workers-v1 3"},{"fields":[{"constraint":"","kind":"Digest","name":"backendDigest","required":true},{"constraint":"","kind":"Digest","name":"contentDigest","required":true},{"constraint":"","kind":"Digest","name":"configurationDigest","required":true},{"constraint":"fixed, reduced, positive","kind":"RationalNs","name":"stepDuration","required":true},{"constraint":"1..=4, unique portId, fixed order","kind":"array<{portId:Id,controls:ControllerSchema}>","name":"ports","required":true},{"constraint":"","kind":"SchemaRef","name":"inspectionSchema","required":true},{"constraint":"<= 8, unique viewId","kind":"array","name":"views","required":true},{"constraint":"unique streamId","kind":"array","name":"audio","required":true},{"constraint":"","kind":"Recovery","name":"recovery","required":true},{"constraint":"","kind":"Determinism","name":"determinism","required":true}],"name":"EnvironmentDescriptor","source":"workers-v1 3"},{"fields":[{"constraint":"","kind":"AssetRef","name":"backendConfig","required":true},{"constraint":"","kind":"AssetRef","name":"taskConfig","required":true},{"constraint":"","kind":"Id","name":"episodeId","required":true},{"constraint":"1..=4, unique portId and unique agentId","kind":"array<{portId:Id,agentId:Id}>","name":"portBindings","required":true}],"name":"EnvironmentInitializeParams","source":"workers-v1 3"},{"fields":[{"constraint":"","kind":"EnvironmentDescriptor","name":"descriptor","required":true},{"constraint":"boundary 0 and worldTime 0/1","kind":"WorldObservation","name":"observation","required":true}],"name":"EnvironmentInitializeResult","source":"workers-v1 3"},{"fields":[{"constraint":"","kind":"EpisodeRequestKind","name":"kind","required":true},{"constraint":"","kind":"Id","name":"reason","required":true},{"constraint":"","kind":"TypedValue","name":"outcome","required":true}],"name":"EpisodeRequest","source":"workers-v1 4"},{"fields":[{"constraint":"","kind":"Id","name":"sessionId","required":true},{"constraint":"","kind":"Id","name":"expectedWorkerId","required":true},{"constraint":"","kind":"Role","name":"role","required":true},{"constraint":"nonempty, unique, 1..=65535","kind":"array","name":"supportedMajors","required":true}],"name":"HelloParams","source":"ipc-v1 4"},{"fields":[{"constraint":"1","kind":"const","name":"selectedMajor","required":true},{"constraint":"0","kind":"const","name":"selectedMinor","required":true},{"constraint":"","kind":"Id","name":"workerId","required":true},{"constraint":"","kind":"Id","name":"incarnationId","required":true},{"constraint":"","kind":"Role","name":"role","required":true},{"constraint":"","kind":"Digest","name":"buildDigest","required":true},{"constraint":"","kind":"Digest","name":"contractDigest","required":true},{"constraint":"unique; agent-step-v1 or world-step-v1 required for that role","kind":"array","name":"capabilities","required":true},{"constraint":"1..=4 agents, 1..=4 ports, and the launcher allocation this worker runs in","kind":"{maxAgents:int,maxPorts:int,workerThreads:int}","name":"limits","required":true}],"name":"HelloResult","source":"ipc-v1 4"},{"fields":[{"constraint":"","kind":"Id","name":"portId","required":true},{"constraint":"every declared button, descriptor order, no extras","kind":"array<{id:Id,down:bool}>","name":"buttons","required":true},{"constraint":"every declared axis in order; bipolar [-1,1], unit [0,1], refused not clamped","kind":"array<{id:Id,value:number}>","name":"axes","required":true}],"name":"PortControl","source":"workers-v1 3"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"","kind":"Digest","name":"profileDigest","required":true},{"constraint":"positive","kind":"RationalNs","name":"interval","required":true},{"constraint":"","kind":"Digest","name":"decisionContextDigest","required":true},{"constraint":"<= 64, unique id, supplied order retained","kind":"array","name":"preStepStimulations","required":true}],"name":"PrepareParams","source":"workers-v1 2"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"<= brainTicks","kind":"U64","name":"ticksAdvanced","required":true},{"constraint":"","kind":"U64","name":"brainTicks","required":true},{"constraint":">= 0 and < one model tick","kind":"RationalNs","name":"remainder","required":true},{"constraint":"the profile's registered intent schema","kind":"TypedValue","name":"decision","required":true}],"name":"PreparedDecision","source":"workers-v1 2"},{"fields":[{"constraint":"reduced against denominator","kind":"U64","name":"numerator","required":true},{"constraint":"positive; zero is encoded 0/1; arithmetic is checked","kind":"U64","name":"denominator","required":true}],"name":"RationalNs","source":"ipc-v1 2"},{"fields":[{"constraint":"unique within its outcome namespace","kind":"Id","name":"eventId","required":true},{"constraint":"","kind":"Id","name":"ruleId","required":true},{"constraint":"finite; positive-only profiles reject negatives","kind":"number","name":"value","required":true}],"name":"Reward","source":"workers-v1 1"},{"fields":[{"constraint":"","kind":"Id","name":"id","required":true},{"constraint":"1..=65535","kind":"int","name":"version","required":true},{"constraint":"64 lowercase hex digits","kind":"Digest","name":"digest","required":true}],"name":"SchemaRef","source":"ipc-v1 2"},{"fields":[{"constraint":"^[a-z0-9][a-z0-9._-]{0,63}$","kind":"Id","name":"sessionId","required":true},{"constraint":"^[a-z0-9][a-z0-9._-]{0,63}$","kind":"Id","name":"epoch","required":true},{"constraint":"decimal string, <= 18446744073709551615","kind":"U64","name":"step","required":true}],"name":"Scope","source":"ipc-v1 2"},{"fields":[{"constraint":"the observed environment boundary","kind":"U64","name":"boundary","required":true},{"constraint":"<= 8, unique viewId, producedStep <= boundary","kind":"array","name":"views","required":true},{"constraint":"a pixel-only profile rejects non-null","kind":"TypedValue|null","name":"structured","required":false}],"name":"SensoryInput","source":"workers-v1 1"},{"fields":[{"constraint":"","kind":"Id","name":"sessionId","required":true},{"constraint":"","kind":"U64","name":"revision","required":true},{"constraint":"","kind":"Digest","name":"compositionDigest","required":true},{"constraint":"","kind":"SchedulerId","name":"schedulerId","required":true},{"constraint":"","kind":"EnvironmentDescriptor","name":"environment","required":true},{"constraint":"","kind":"SchemaRef","name":"taskSchema","required":true},{"constraint":"1..=4, unique agentId and portId, each portId declared by the environment","kind":"array","name":"agents","required":true},{"constraint":"unique id","kind":"array","name":"assets","required":true}],"name":"SessionDescriptor","source":"publishing-v1 3"},{"fields":[{"constraint":"\"error\"","kind":"const","name":"type","required":true},{"constraint":"echoes the request","kind":"DomainRequestId","name":"requestId","required":true},{"constraint":"","kind":"Id","name":"workerId","required":true},{"constraint":"","kind":"Id","name":"incarnationId","required":true},{"constraint":"echoes the request scope","kind":"Scope|null","name":"scope","required":false},{"constraint":"","kind":"ErrorCode","name":"error.code","required":true},{"constraint":"<= 512 code points","kind":"string","name":"error.message","required":true},{"constraint":"none for every code raised before mutation","kind":"MutationCertainty","name":"error.mutation","required":true}],"name":"SessionRpcFailure","source":"ipc-v1 3"},{"fields":[{"constraint":"req-","kind":"DomainRequestId","name":"requestId","required":true},{"constraint":"null for lifecycle calls","kind":"Scope|null","name":"scope","required":false},{"constraint":"no bus callId/deliveryId/ownerId keys","kind":"object","name":"params","required":true}],"name":"SessionRpcRequest","source":"ipc-v1 2"},{"fields":[{"constraint":"\"result\"","kind":"const","name":"type","required":true},{"constraint":"echoes the request","kind":"DomainRequestId","name":"requestId","required":true},{"constraint":"","kind":"Id","name":"workerId","required":true},{"constraint":"","kind":"Id","name":"incarnationId","required":true},{"constraint":"echoes the request scope","kind":"Scope|null","name":"scope","required":false},{"constraint":"","kind":"object","name":"result","required":true}],"name":"SessionRpcSuccess","source":"ipc-v1 3"},{"fields":[{"constraint":"","kind":"Id","name":"reason","required":true}],"name":"ShutdownParams","source":"ipc-v1 7"},{"fields":[{"constraint":"true","kind":"const","name":"stopping","required":true}],"name":"ShutdownResult","source":"ipc-v1 7"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"","kind":"AgentTelemetry","name":"telemetry","required":true},{"constraint":"null exactly at boundary 0","kind":"TypedValue|null","name":"selectedDecision","required":false},{"constraint":"null exactly at boundary 0; the agent's assigned port","kind":"PortControl|null","name":"appliedControls","required":false}],"name":"SnapshotAgent","source":"publishing-v1 3"},{"fields":[{"constraint":"","kind":"Id","name":"checkpointId","required":true},{"constraint":"provenance, not the new handles","kind":"Scope","name":"sourceScope","required":true},{"constraint":"","kind":"Digest","name":"compatibilityDigest","required":true},{"constraint":"newly imported; digest required","kind":"ArtifactRef","name":"payload","required":true}],"name":"StageRestoreParams","source":"state-media-v1 5"},{"fields":[{"constraint":"","kind":"Id","name":"checkpointId","required":true},{"constraint":"activates once, bound to scope and payload","kind":"Id","name":"restoreToken","required":true}],"name":"StageRestoreResult","source":"state-media-v1 5"},{"fields":[{"constraint":"","kind":"WorkerState","name":"state","required":true},{"constraint":"null before initialization","kind":"Scope|null","name":"currentScope","required":false},{"constraint":"","kind":"DomainRequestId|null","name":"activeRequestId","required":false},{"constraint":"","kind":"DomainRequestId|null","name":"lastCompletedRequestId","required":false},{"constraint":"","kind":"Id|null","name":"lastBatchId","required":false},{"constraint":"advances on progress, not on status queries","kind":"U64","name":"progressCounter","required":true}],"name":"StatusResult","source":"ipc-v1 4"},{"fields":[{"constraint":"echoes the request","kind":"Id","name":"batchId","required":true},{"constraint":"k","kind":"U64","name":"appliedFromStep","required":true},{"constraint":"appliedFromStep + 1","kind":"U64","name":"nextStep","required":true},{"constraint":"over validated canonical requested controls","kind":"Digest","name":"appliedControlsDigest","required":true},{"constraint":"boundary == nextStep","kind":"WorldObservation","name":"observation","required":true}],"name":"StepResult","source":"workers-v1 3"},{"fields":[{"constraint":"unique within its command namespace","kind":"Id","name":"id","required":true},{"constraint":"resolved through a profile capability","kind":"Id","name":"kindId","required":true},{"constraint":"finite and > 0","kind":"number","name":"durationMs","required":true}],"name":"Stimulus","source":"workers-v1 1"},{"fields":[{"constraint":"derived from epoch, source step, rule and ordinal","kind":"Id","name":"id","required":true},{"constraint":"","kind":"Id","name":"kindId","required":true},{"constraint":"the newly reached boundary, 0 for bootstrap","kind":"U64","name":"sourceStep","required":true},{"constraint":"","kind":"Id|null","name":"agentId","required":false},{"constraint":"","kind":"TypedValue","name":"payload","required":true}],"name":"TaskEvent","source":"workers-v1 4"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"","kind":"Digest","name":"profileDigest","required":true},{"constraint":"","kind":"U64","name":"ticksAdvanced","required":true},{"constraint":"","kind":"U64","name":"brainTicks","required":true},{"constraint":"","kind":"RationalNs","name":"remainder","required":true},{"constraint":"","kind":"Digest","name":"decisionDigest","required":true},{"constraint":"the commit acknowledgment","kind":"U64","name":"committedStep","required":true}],"name":"TraceAgent","source":"step-v1 8"},{"fields":[{"constraint":"","kind":"Scope","name":"scope","required":true},{"constraint":"sorted by agentId, independent of dispatch order","kind":"array","name":"agents","required":true},{"constraint":"","kind":"Id","name":"batchId","required":true},{"constraint":"","kind":"Digest","name":"controlDigest","required":true},{"constraint":"the world boundary the environment acknowledged","kind":"U64","name":"acknowledgedBoundary","required":true},{"constraint":"sorted by viewId","kind":"array<{viewId:Id,producedStep:U64}>","name":"observationBoundaries","required":true},{"constraint":"task outcome ids in task order","kind":"array","name":"outcomeIds","required":true},{"constraint":"task event ids in task order","kind":"array","name":"eventIds","required":true},{"constraint":"","kind":"U64","name":"publishedBoundary","required":true}],"name":"TraceBehaviour","source":"step-v1 8"},{"fields":[{"constraint":"","kind":"U64","name":"wallTimeNs","required":true},{"constraint":"","kind":"array<{agentId:Id,requestId:DomainRequestId}>","name":"prepareRequestIds","required":true},{"constraint":"","kind":"DomainRequestId","name":"advanceRequestId","required":true},{"constraint":"","kind":"array<{agentId:Id,requestId:DomainRequestId}>","name":"commitRequestIds","required":true},{"constraint":"call-","kind":"array","name":"busCallIds","required":true},{"constraint":"dlv- or own-","kind":"array","name":"deliveryIds","required":true}],"name":"TraceOperational","source":"step-v1 8"},{"fields":[{"constraint":"compared between runs","kind":"TraceBehaviour","name":"behaviour","required":true},{"constraint":"recorded, never compared: wall time and transport identities","kind":"TraceOperational","name":"operational","required":true}],"name":"TransitionTrace","source":"step-v1 8"},{"fields":[{"constraint":"","kind":"SchemaRef","name":"schema","required":true},{"constraint":"canonical JSON of the whole TypedValue <= 32768 bytes","kind":"object","name":"value","required":true}],"name":"TypedValue","source":"ipc-v1 2"},{"fields":[{"constraint":"","kind":"Id","name":"viewId","required":true},{"constraint":"1..=4096","kind":"int","name":"width","required":true},{"constraint":"1..=4096","kind":"int","name":"height","required":true},{"constraint":"","kind":"ViewFormat","name":"format","required":true},{"constraint":"exactly 4 x width","kind":"int","name":"rowStride","required":true},{"constraint":"positive integers <= 65535","kind":"{numerator:int,denominator:int}","name":"pixelAspect","required":true},{"constraint":"0..=8","kind":"int","name":"observationDelaySteps","required":true}],"name":"ViewDescriptor","source":"state-media-v1 2"},{"fields":[{"constraint":"","kind":"Id","name":"viewId","required":true},{"constraint":"max(0, boundary - observationDelaySteps) for required sensory views","kind":"U64","name":"producedStep","required":true},{"constraint":"listed attachment; byteLength == rowStride x height","kind":"ArtifactRef","name":"pixels","required":true}],"name":"ViewRef","source":"state-media-v1 2"},{"fields":[{"constraint":"","kind":"U64","name":"boundary","required":true},{"constraint":"logical time since episode start","kind":"RationalNs","name":"worldTime","required":true},{"constraint":"<= 64 characters","kind":"string|null","name":"engineFrame","required":false},{"constraint":"<= 8, unique viewId","kind":"array","name":"sensoryViews","required":true},{"constraint":"the descriptor's inspectionSchema","kind":"TypedValue","name":"inspection","required":true},{"constraint":"<= 8, unique viewId","kind":"array","name":"broadcastViews","required":true},{"constraint":"unique streamId","kind":"array","name":"audio","required":true}],"name":"WorldObservation","source":"workers-v1 3"}],"version":1} +{"contract":"fly-session-types","enums":[{"members":["f32le-interleaved"],"name":"AudioFormat","source":"state-media-v1 2"},{"members":["bipolar","unit"],"name":"AxisRange","source":"workers-v1 3"},{"members":["fixed-build","unverified"],"name":"Determinism","source":"workers-v1 3"},{"members":["terminal"],"name":"EpisodeRequestKind","source":"workers-v1 4"},{"members":["INVALID_ARGUMENT","UNSUPPORTED","IDENTITY_MISMATCH","STALE_EPOCH","STALE_STEP","FUTURE_STEP","INVALID_PHASE","CONFLICT","IN_PROGRESS","BUSY","BUFFER_INVALID","RESULT_EXPIRED","INCOMPATIBLE_STATE","BACKEND_FAILURE","INTERNAL"],"name":"ErrorCode","source":"ipc-v1 7"},{"members":["none","applied","unknown"],"name":"MutationCertainty","source":"ipc-v1 3"},{"members":["exact-checkpoint","episode-restart"],"name":"Recovery","source":"workers-v1 3"},{"members":["agent","environment","coordinator"],"name":"Role","source":"ipc-v1 4"},{"members":["lockstep-v1"],"name":"SchedulerId","source":"publishing-v1 3"},{"members":["rgba8"],"name":"ViewFormat","source":"state-media-v1 2"},{"members":["uninitialized","ready","preparing","prepared","advancing","committing","capturing","staged-restore","restoring","failed","stopping"],"name":"WorkerState","source":"ipc-v1 4"}],"limits":[{"name":"maxAcknowledge","source":"ipc-v1 5","value":16},{"name":"maxAgents","source":"ipc-v1 2","value":4},{"name":"maxAssets","source":"crate","value":64},{"name":"maxAttachments","source":"bus-v1 4","value":32},{"name":"maxAudioStreams","source":"crate","value":8},{"name":"maxAxes","source":"workers-v1 3","value":16},{"name":"maxButtons","source":"workers-v1 3","value":32},{"name":"maxCapabilities","source":"crate","value":32},{"name":"maxEngineFrameLength","source":"workers-v1 3","value":64},{"name":"maxEnvelopeBytes","source":"bus-v1 4","value":65536},{"name":"maxMessageCodePoints","source":"ipc-v1 7","value":512},{"name":"maxObservationDelaySteps","source":"state-media-v1 2","value":8},{"name":"maxPixelAspectPart","source":"state-media-v1 2","value":65535},{"name":"maxPorts","source":"ipc-v1 2","value":4},{"name":"maxRateRoles","source":"ipc-v1 2","value":64},{"name":"maxRewardsPerOperation","source":"workers-v1 1","value":64},{"name":"maxSampleFrames","source":"state-media-v1 2","value":192000},{"name":"maxSchemaVersion","source":"ipc-v1 2","value":65535},{"name":"maxSnapshotEvents","source":"crate","value":64},{"name":"maxStimuliPerOperation","source":"workers-v1 1","value":64},{"name":"maxSupportedMajors","source":"crate","value":8},{"name":"maxSupportedStimuli","source":"crate","value":64},{"name":"maxTypedValueBytes","source":"ipc-v1 2","value":32768},{"name":"maxViewDimension","source":"state-media-v1 2","value":4096},{"name":"maxViews","source":"workers-v1 1","value":8},{"name":"maxWorkerThreads","source":"workers-v1 2","value":4096}],"scalars":{"ArtifactIdentity":"storeId, artifactId and generation of a bus ArtifactRef","BusCallId":"call-","Digest":"64 lowercase hexadecimal digits (SHA-256)","DomainRequestId":"req-","Id":"^[a-z0-9][a-z0-9._-]{0,63}$","OwnerToken":"dlv- or own-","U64":"\"0\" or [1-9][0-9]*, <= 18446744073709551615"},"types":[{"fields":[{"constraint":"1..=16, unique","kind":"array","name":"requestIds","required":true}],"name":"AcknowledgeParams","source":"ipc-v1 5"},{"fields":[{"constraint":"<= 16, unique, a subset of the request","kind":"array","name":"acknowledged","required":true}],"name":"AcknowledgeResult","source":"ipc-v1 5"},{"fields":[{"constraint":"","kind":"Id","name":"restoreToken","required":true}],"name":"ActivateRestoreParams","source":"state-media-v1 5"},{"fields":[{"constraint":"","kind":"U64","name":"committedStep","required":true},{"constraint":"","kind":"Id","name":"checkpointId","required":true},{"constraint":"required from an environment, null from an agent","kind":"WorldObservation|null","name":"observation","required":false}],"name":"ActivateRestoreResult","source":"state-media-v1 5"},{"fields":[{"constraint":"unique within an epoch","kind":"Id","name":"batchId","required":true},{"constraint":"one complete batch: every declared port once, descriptor order","kind":"array","name":"controls","required":true}],"name":"AdvanceParams","source":"workers-v1 3"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"k+1","kind":"U64","name":"committedStep","required":true},{"constraint":"","kind":"Digest","name":"decisionContextDigest","required":true},{"constraint":"","kind":"AgentTelemetry","name":"telemetry","required":true}],"name":"AgentCommitResult","source":"workers-v1 2"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"","kind":"Id","name":"portId","required":true},{"constraint":"","kind":"Digest","name":"profileDigest","required":true},{"constraint":"","kind":"Digest","name":"datasetDigest","required":true},{"constraint":"geometry mapping needs this, not neuronCount","kind":"Digest","name":"indexDigest","required":true},{"constraint":"","kind":"U64","name":"neuronCount","required":true},{"constraint":"<= 64, unique","kind":"array","name":"rateRoles","required":true},{"constraint":"unique","kind":"array","name":"supportedStimuli","required":true}],"name":"AgentDescriptor","source":"publishing-v1 3"},{"fields":[{"constraint":"","kind":"Digest","name":"datasetDigest","required":true},{"constraint":"geometry mapping needs this, not neuronCount","kind":"Digest","name":"indexDigest","required":true},{"constraint":"","kind":"U64","name":"neuronCount","required":true},{"constraint":"<= 64, unique","kind":"array","name":"rateRoles","required":true},{"constraint":"<= 64, unique","kind":"array","name":"supportedStimuli","required":true}],"name":"AgentGraph","source":"workers-v1 2"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"","kind":"AssetRef","name":"profile","required":true},{"constraint":"signed 32-bit","kind":"int","name":"seed","required":true},{"constraint":"","kind":"SensoryInput","name":"initialInput","required":true},{"constraint":"","kind":"TypedValue","name":"initialDecisionContext","required":true},{"constraint":">= 1","kind":"int","name":"workerThreads","required":true}],"name":"AgentInitializeParams","source":"workers-v1 2"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"","kind":"Digest","name":"profileDigest","required":true},{"constraint":"positive","kind":"RationalNs","name":"tickDuration","required":true},{"constraint":"","kind":"U64","name":"warmupTicks","required":true},{"constraint":"\"0\"","kind":"U64","name":"committedStep","required":true},{"constraint":"","kind":"Digest","name":"decisionContextDigest","required":true},{"constraint":"","kind":"AgentTelemetry","name":"telemetry","required":true},{"constraint":"rates are in graph.rateRoles order","kind":"AgentGraph","name":"graph","required":true}],"name":"AgentInitializeResult","source":"workers-v1 2"},{"fields":[{"constraint":"","kind":"U64","name":"brainTicks","required":true},{"constraint":"finite and nonnegative","kind":"number","name":"populationRateHz","required":true},{"constraint":"<= 64, unique roleId, profile order, finite nonnegative hz","kind":"array<{roleId:Id,hz:number}>","name":"rates","required":true},{"constraint":"changed <= updates; signal finite","kind":"{enabled:bool,updates:U64,changed:U64,signal:number}","name":"learning","required":true}],"name":"AgentTelemetry","source":"workers-v1 1"},{"fields":[{"constraint":"","kind":"Id","name":"id","required":true},{"constraint":"","kind":"Digest","name":"digest","required":true},{"constraint":"positive","kind":"U64","name":"byteLength","required":true},{"constraint":"","kind":"Id","name":"format","required":true}],"name":"AssetRef","source":"workers-v1 1"},{"fields":[{"constraint":"","kind":"Id","name":"streamId","required":true},{"constraint":"8000..=192000","kind":"int","name":"sampleRate","required":true},{"constraint":"1..=8","kind":"int","name":"channels","required":true},{"constraint":"","kind":"AudioFormat","name":"format","required":true}],"name":"AudioDescriptor","source":"state-media-v1 2"},{"fields":[{"constraint":"","kind":"Id","name":"streamId","required":true},{"constraint":"no overlap or rewind within an epoch","kind":"U64","name":"firstSample","required":true},{"constraint":"0..=192000","kind":"int","name":"sampleFrames","required":true},{"constraint":"byteLength == sampleFrames x channels x 4, finite f32","kind":"ArtifactRef","name":"samples","required":true},{"constraint":"true on the first chunk after restore","kind":"bool","name":"discontinuity","required":true}],"name":"AudioRef","source":"state-media-v1 2"},{"fields":[{"constraint":"","kind":"Id","name":"checkpointId","required":true}],"name":"CaptureParams","source":"state-media-v1 5"},{"fields":[{"constraint":"","kind":"Id","name":"checkpointId","required":true},{"constraint":"the committed boundary","kind":"U64","name":"boundary","required":true},{"constraint":"","kind":"Digest","name":"compatibilityDigest","required":true},{"constraint":"listed attachment; digest required","kind":"ArtifactRef","name":"payload","required":true}],"name":"CaptureResult","source":"state-media-v1 5"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"","kind":"DomainRequestId","name":"preparedRequestId","required":true},{"constraint":"boundary == scope.step + 1","kind":"SensoryInput","name":"nextInput","required":true},{"constraint":"","kind":"TypedValue","name":"nextDecisionContext","required":true},{"constraint":"<= 64, unique eventId, order retained","kind":"array","name":"rewards","required":true},{"constraint":"<= 64, unique id, order retained","kind":"array","name":"taskStimulations","required":true}],"name":"CommitParams","source":"workers-v1 2"},{"fields":[{"constraint":"","kind":"U64","name":"descriptorRevision","required":true},{"constraint":"","kind":"Id","name":"publisherIncarnation","required":true},{"constraint":"the committed boundary","kind":"Scope","name":"scope","required":true},{"constraint":"","kind":"Id","name":"episodeId","required":true},{"constraint":"monotonic within publisherIncarnation","kind":"U64","name":"sequence","required":true},{"constraint":"","kind":"RationalNs","name":"worldTime","required":true},{"constraint":"1..=4, unique agentId, telemetry in profile role order","kind":"array","name":"agents","required":true},{"constraint":"","kind":"TypedValue","name":"progress","required":true},{"constraint":"declared attachments held through publication admission","kind":"{views:array,audio:array}","name":"media","required":true},{"constraint":"unique, task order","kind":"array","name":"eventIds","required":true}],"name":"CommittedSnapshot","source":"publishing-v1 3"},{"fields":[{"constraint":"","kind":"SchemaRef","name":"schema","required":true},{"constraint":"<= 32, unique, fixed order","kind":"array","name":"buttons","required":true},{"constraint":"<= 16, unique id, neutral inside its range","kind":"array<{id:Id,range:AxisRange,neutral:number}>","name":"axes","required":true}],"name":"ControllerSchema","source":"workers-v1 3"},{"fields":[{"constraint":"","kind":"Digest","name":"backendDigest","required":true},{"constraint":"","kind":"Digest","name":"contentDigest","required":true},{"constraint":"","kind":"Digest","name":"configurationDigest","required":true},{"constraint":"fixed, reduced, positive","kind":"RationalNs","name":"stepDuration","required":true},{"constraint":"1..=4, unique portId, fixed order","kind":"array<{portId:Id,controls:ControllerSchema}>","name":"ports","required":true},{"constraint":"","kind":"SchemaRef","name":"inspectionSchema","required":true},{"constraint":"<= 8, unique viewId","kind":"array","name":"views","required":true},{"constraint":"unique streamId","kind":"array","name":"audio","required":true},{"constraint":"","kind":"Recovery","name":"recovery","required":true},{"constraint":"","kind":"Determinism","name":"determinism","required":true}],"name":"EnvironmentDescriptor","source":"workers-v1 3"},{"fields":[{"constraint":"","kind":"AssetRef","name":"backendConfig","required":true},{"constraint":"","kind":"AssetRef","name":"taskConfig","required":true},{"constraint":"","kind":"Id","name":"episodeId","required":true},{"constraint":"1..=4, unique portId and unique agentId","kind":"array<{portId:Id,agentId:Id}>","name":"portBindings","required":true}],"name":"EnvironmentInitializeParams","source":"workers-v1 3"},{"fields":[{"constraint":"","kind":"EnvironmentDescriptor","name":"descriptor","required":true},{"constraint":"boundary 0 and worldTime 0/1","kind":"WorldObservation","name":"observation","required":true}],"name":"EnvironmentInitializeResult","source":"workers-v1 3"},{"fields":[{"constraint":"","kind":"EpisodeRequestKind","name":"kind","required":true},{"constraint":"","kind":"Id","name":"reason","required":true},{"constraint":"","kind":"TypedValue","name":"outcome","required":true}],"name":"EpisodeRequest","source":"workers-v1 4"},{"fields":[{"constraint":"","kind":"Id","name":"sessionId","required":true},{"constraint":"","kind":"Id","name":"expectedWorkerId","required":true},{"constraint":"","kind":"Role","name":"role","required":true},{"constraint":"nonempty, unique, 1..=65535","kind":"array","name":"supportedMajors","required":true}],"name":"HelloParams","source":"ipc-v1 4"},{"fields":[{"constraint":"1","kind":"const","name":"selectedMajor","required":true},{"constraint":"0","kind":"const","name":"selectedMinor","required":true},{"constraint":"","kind":"Id","name":"workerId","required":true},{"constraint":"","kind":"Id","name":"incarnationId","required":true},{"constraint":"","kind":"Role","name":"role","required":true},{"constraint":"","kind":"Digest","name":"buildDigest","required":true},{"constraint":"","kind":"Digest","name":"contractDigest","required":true},{"constraint":"unique; agent-step-v1 or world-step-v1 required for that role","kind":"array","name":"capabilities","required":true},{"constraint":"1..=4 agents, 1..=4 ports, and the launcher allocation this worker runs in","kind":"{maxAgents:int,maxPorts:int,workerThreads:int}","name":"limits","required":true}],"name":"HelloResult","source":"ipc-v1 4"},{"fields":[{"constraint":"","kind":"Id","name":"portId","required":true},{"constraint":"every declared button, descriptor order, no extras","kind":"array<{id:Id,down:bool}>","name":"buttons","required":true},{"constraint":"every declared axis in order; bipolar [-1,1], unit [0,1], refused not clamped","kind":"array<{id:Id,value:number}>","name":"axes","required":true}],"name":"PortControl","source":"workers-v1 3"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"","kind":"Digest","name":"profileDigest","required":true},{"constraint":"positive","kind":"RationalNs","name":"interval","required":true},{"constraint":"","kind":"Digest","name":"decisionContextDigest","required":true},{"constraint":"<= 64, unique id, supplied order retained","kind":"array","name":"preStepStimulations","required":true}],"name":"PrepareParams","source":"workers-v1 2"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"<= brainTicks","kind":"U64","name":"ticksAdvanced","required":true},{"constraint":"","kind":"U64","name":"brainTicks","required":true},{"constraint":">= 0 and < one model tick","kind":"RationalNs","name":"remainder","required":true},{"constraint":"the profile's registered intent schema","kind":"TypedValue","name":"decision","required":true}],"name":"PreparedDecision","source":"workers-v1 2"},{"fields":[{"constraint":"reduced against denominator","kind":"U64","name":"numerator","required":true},{"constraint":"positive; zero is encoded 0/1; arithmetic is checked","kind":"U64","name":"denominator","required":true}],"name":"RationalNs","source":"ipc-v1 2"},{"fields":[{"constraint":"unique within its outcome namespace","kind":"Id","name":"eventId","required":true},{"constraint":"","kind":"Id","name":"ruleId","required":true},{"constraint":"finite; positive-only profiles reject negatives","kind":"number","name":"value","required":true}],"name":"Reward","source":"workers-v1 1"},{"fields":[{"constraint":"","kind":"Id","name":"id","required":true},{"constraint":"1..=65535","kind":"int","name":"version","required":true},{"constraint":"64 lowercase hex digits","kind":"Digest","name":"digest","required":true}],"name":"SchemaRef","source":"ipc-v1 2"},{"fields":[{"constraint":"^[a-z0-9][a-z0-9._-]{0,63}$","kind":"Id","name":"sessionId","required":true},{"constraint":"^[a-z0-9][a-z0-9._-]{0,63}$","kind":"Id","name":"epoch","required":true},{"constraint":"decimal string, <= 18446744073709551615","kind":"U64","name":"step","required":true}],"name":"Scope","source":"ipc-v1 2"},{"fields":[{"constraint":"the observed environment boundary","kind":"U64","name":"boundary","required":true},{"constraint":"<= 8, unique viewId, producedStep <= boundary","kind":"array","name":"views","required":true},{"constraint":"a pixel-only profile rejects non-null","kind":"TypedValue|null","name":"structured","required":false}],"name":"SensoryInput","source":"workers-v1 1"},{"fields":[{"constraint":"","kind":"Id","name":"sessionId","required":true},{"constraint":"","kind":"U64","name":"revision","required":true},{"constraint":"","kind":"Digest","name":"compositionDigest","required":true},{"constraint":"","kind":"SchedulerId","name":"schedulerId","required":true},{"constraint":"","kind":"EnvironmentDescriptor","name":"environment","required":true},{"constraint":"","kind":"SchemaRef","name":"taskSchema","required":true},{"constraint":"1..=4, unique agentId and portId, each portId declared by the environment","kind":"array","name":"agents","required":true},{"constraint":"unique id","kind":"array","name":"assets","required":true}],"name":"SessionDescriptor","source":"publishing-v1 3"},{"fields":[{"constraint":"\"error\"","kind":"const","name":"type","required":true},{"constraint":"echoes the request","kind":"DomainRequestId","name":"requestId","required":true},{"constraint":"","kind":"Id","name":"workerId","required":true},{"constraint":"","kind":"Id","name":"incarnationId","required":true},{"constraint":"echoes the request scope","kind":"Scope|null","name":"scope","required":false},{"constraint":"","kind":"ErrorCode","name":"error.code","required":true},{"constraint":"<= 512 code points","kind":"string","name":"error.message","required":true},{"constraint":"none for every code raised before mutation","kind":"MutationCertainty","name":"error.mutation","required":true}],"name":"SessionRpcFailure","source":"ipc-v1 3"},{"fields":[{"constraint":"req-","kind":"DomainRequestId","name":"requestId","required":true},{"constraint":"null for lifecycle calls","kind":"Scope|null","name":"scope","required":false},{"constraint":"no bus callId/deliveryId/ownerId keys","kind":"object","name":"params","required":true}],"name":"SessionRpcRequest","source":"ipc-v1 2"},{"fields":[{"constraint":"\"result\"","kind":"const","name":"type","required":true},{"constraint":"echoes the request","kind":"DomainRequestId","name":"requestId","required":true},{"constraint":"","kind":"Id","name":"workerId","required":true},{"constraint":"","kind":"Id","name":"incarnationId","required":true},{"constraint":"echoes the request scope","kind":"Scope|null","name":"scope","required":false},{"constraint":"","kind":"object","name":"result","required":true}],"name":"SessionRpcSuccess","source":"ipc-v1 3"},{"fields":[{"constraint":"","kind":"Id","name":"reason","required":true}],"name":"ShutdownParams","source":"ipc-v1 7"},{"fields":[{"constraint":"true","kind":"const","name":"stopping","required":true}],"name":"ShutdownResult","source":"ipc-v1 7"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"","kind":"AgentTelemetry","name":"telemetry","required":true},{"constraint":"null at boundary 0 and at an installed boundary; null or present for every agent together","kind":"TypedValue|null","name":"selectedDecision","required":false},{"constraint":"null with selectedDecision; the agent's assigned port","kind":"PortControl|null","name":"appliedControls","required":false}],"name":"SnapshotAgent","source":"publishing-v1 3"},{"fields":[{"constraint":"","kind":"Id","name":"checkpointId","required":true},{"constraint":"provenance, not the new handles","kind":"Scope","name":"sourceScope","required":true},{"constraint":"","kind":"Digest","name":"compatibilityDigest","required":true},{"constraint":"newly imported; digest required","kind":"ArtifactRef","name":"payload","required":true}],"name":"StageRestoreParams","source":"state-media-v1 5"},{"fields":[{"constraint":"","kind":"Id","name":"checkpointId","required":true},{"constraint":"activates once, bound to scope and payload","kind":"Id","name":"restoreToken","required":true}],"name":"StageRestoreResult","source":"state-media-v1 5"},{"fields":[{"constraint":"","kind":"WorkerState","name":"state","required":true},{"constraint":"null before initialization","kind":"Scope|null","name":"currentScope","required":false},{"constraint":"","kind":"DomainRequestId|null","name":"activeRequestId","required":false},{"constraint":"","kind":"DomainRequestId|null","name":"lastCompletedRequestId","required":false},{"constraint":"","kind":"Id|null","name":"lastBatchId","required":false},{"constraint":"advances on progress, not on status queries","kind":"U64","name":"progressCounter","required":true}],"name":"StatusResult","source":"ipc-v1 4"},{"fields":[{"constraint":"echoes the request","kind":"Id","name":"batchId","required":true},{"constraint":"k","kind":"U64","name":"appliedFromStep","required":true},{"constraint":"appliedFromStep + 1","kind":"U64","name":"nextStep","required":true},{"constraint":"over validated canonical requested controls","kind":"Digest","name":"appliedControlsDigest","required":true},{"constraint":"boundary == nextStep","kind":"WorldObservation","name":"observation","required":true}],"name":"StepResult","source":"workers-v1 3"},{"fields":[{"constraint":"unique within its command namespace","kind":"Id","name":"id","required":true},{"constraint":"resolved through a profile capability","kind":"Id","name":"kindId","required":true},{"constraint":"finite and > 0","kind":"number","name":"durationMs","required":true}],"name":"Stimulus","source":"workers-v1 1"},{"fields":[{"constraint":"derived from epoch, source step, rule and ordinal","kind":"Id","name":"id","required":true},{"constraint":"","kind":"Id","name":"kindId","required":true},{"constraint":"the newly reached boundary, 0 for bootstrap","kind":"U64","name":"sourceStep","required":true},{"constraint":"","kind":"Id|null","name":"agentId","required":false},{"constraint":"","kind":"TypedValue","name":"payload","required":true}],"name":"TaskEvent","source":"workers-v1 4"},{"fields":[{"constraint":"","kind":"Id","name":"agentId","required":true},{"constraint":"","kind":"Digest","name":"profileDigest","required":true},{"constraint":"","kind":"U64","name":"ticksAdvanced","required":true},{"constraint":"","kind":"U64","name":"brainTicks","required":true},{"constraint":"","kind":"RationalNs","name":"remainder","required":true},{"constraint":"","kind":"Digest","name":"decisionDigest","required":true},{"constraint":"the commit acknowledgment","kind":"U64","name":"committedStep","required":true}],"name":"TraceAgent","source":"step-v1 8"},{"fields":[{"constraint":"","kind":"Scope","name":"scope","required":true},{"constraint":"sorted by agentId, independent of dispatch order","kind":"array","name":"agents","required":true},{"constraint":"","kind":"Id","name":"batchId","required":true},{"constraint":"","kind":"Digest","name":"controlDigest","required":true},{"constraint":"the world boundary the environment acknowledged","kind":"U64","name":"acknowledgedBoundary","required":true},{"constraint":"sorted by viewId","kind":"array<{viewId:Id,producedStep:U64}>","name":"observationBoundaries","required":true},{"constraint":"task outcome ids in task order","kind":"array","name":"outcomeIds","required":true},{"constraint":"task event ids in task order","kind":"array","name":"eventIds","required":true},{"constraint":"","kind":"U64","name":"publishedBoundary","required":true}],"name":"TraceBehaviour","source":"step-v1 8"},{"fields":[{"constraint":"","kind":"U64","name":"wallTimeNs","required":true},{"constraint":"","kind":"array<{agentId:Id,requestId:DomainRequestId}>","name":"prepareRequestIds","required":true},{"constraint":"","kind":"DomainRequestId","name":"advanceRequestId","required":true},{"constraint":"","kind":"array<{agentId:Id,requestId:DomainRequestId}>","name":"commitRequestIds","required":true},{"constraint":"call-","kind":"array","name":"busCallIds","required":true},{"constraint":"dlv- or own-","kind":"array","name":"deliveryIds","required":true}],"name":"TraceOperational","source":"step-v1 8"},{"fields":[{"constraint":"compared between runs","kind":"TraceBehaviour","name":"behaviour","required":true},{"constraint":"recorded, never compared: wall time and transport identities","kind":"TraceOperational","name":"operational","required":true}],"name":"TransitionTrace","source":"step-v1 8"},{"fields":[{"constraint":"","kind":"SchemaRef","name":"schema","required":true},{"constraint":"canonical JSON of the whole TypedValue <= 32768 bytes","kind":"object","name":"value","required":true}],"name":"TypedValue","source":"ipc-v1 2"},{"fields":[{"constraint":"","kind":"Id","name":"viewId","required":true},{"constraint":"1..=4096","kind":"int","name":"width","required":true},{"constraint":"1..=4096","kind":"int","name":"height","required":true},{"constraint":"","kind":"ViewFormat","name":"format","required":true},{"constraint":"exactly 4 x width","kind":"int","name":"rowStride","required":true},{"constraint":"positive integers <= 65535","kind":"{numerator:int,denominator:int}","name":"pixelAspect","required":true},{"constraint":"0..=8","kind":"int","name":"observationDelaySteps","required":true}],"name":"ViewDescriptor","source":"state-media-v1 2"},{"fields":[{"constraint":"","kind":"Id","name":"viewId","required":true},{"constraint":"max(0, boundary - observationDelaySteps) for required sensory views","kind":"U64","name":"producedStep","required":true},{"constraint":"listed attachment; byteLength == rowStride x height","kind":"ArtifactRef","name":"pixels","required":true}],"name":"ViewRef","source":"state-media-v1 2"},{"fields":[{"constraint":"","kind":"U64","name":"boundary","required":true},{"constraint":"logical time since episode start","kind":"RationalNs","name":"worldTime","required":true},{"constraint":"<= 64 characters","kind":"string|null","name":"engineFrame","required":false},{"constraint":"<= 8, unique viewId","kind":"array","name":"sensoryViews","required":true},{"constraint":"the descriptor's inspectionSchema","kind":"TypedValue","name":"inspection","required":true},{"constraint":"<= 8, unique viewId","kind":"array","name":"broadcastViews","required":true},{"constraint":"unique streamId","kind":"array","name":"audio","required":true}],"name":"WorldObservation","source":"workers-v1 3"}],"version":1} diff --git a/services/flysim/crates/fly-session-types/fixtures/valid.json b/services/flysim/crates/fly-session-types/fixtures/valid.json index dea3a64..8f13023 100644 --- a/services/flysim/crates/fly-session-types/fixtures/valid.json +++ b/services/flysim/crates/fly-session-types/fixtures/valid.json @@ -470,11 +470,24 @@ "changed": "2", "signal": 0.5 } + }, + "graph": { + "datasetDigest": "6c0af1f0784ef63a393ee77d614e8246c625051360f3f1a48838374c5d355b52", + "indexDigest": "52a7a9b441f686ebea398e3d65dbaa87b4fde4f8b2231fff951700548f6a4e42", + "neuronCount": "139255", + "rateRoles": [ + "kenyon", + "mbon" + ], + "supportedStimuli": [ + "sugar", + "shock" + ] } }, "note": "", - "canonical": "{\"agentId\":\"fly-a\",\"committedStep\":\"0\",\"decisionContextDigest\":\"ea7792a26f405e2ae9c6f49ca93bbe6076ceac0a1fc53d83426c7d7f2d9377e4\",\"profileDigest\":\"1900eab6c028483d7126599ee6f50de0d27907b5c65fa90524580b4b0f9852b0\",\"telemetry\":{\"brainTicks\":\"2500\",\"learning\":{\"changed\":\"2\",\"enabled\":true,\"signal\":0.5,\"updates\":\"4\"},\"populationRateHz\":12.5,\"rates\":[{\"hz\":3.25,\"roleId\":\"kenyon\"},{\"hz\":0,\"roleId\":\"mbon\"}]},\"tickDuration\":{\"denominator\":\"1\",\"numerator\":\"1000000\"},\"warmupTicks\":\"2500\"}", - "digest": "a220160c75d758720fe43145089939201ee35c86bb100fae0c4efdd3c4eafaa6" + "canonical": "{\"agentId\":\"fly-a\",\"committedStep\":\"0\",\"decisionContextDigest\":\"ea7792a26f405e2ae9c6f49ca93bbe6076ceac0a1fc53d83426c7d7f2d9377e4\",\"graph\":{\"datasetDigest\":\"6c0af1f0784ef63a393ee77d614e8246c625051360f3f1a48838374c5d355b52\",\"indexDigest\":\"52a7a9b441f686ebea398e3d65dbaa87b4fde4f8b2231fff951700548f6a4e42\",\"neuronCount\":\"139255\",\"rateRoles\":[\"kenyon\",\"mbon\"],\"supportedStimuli\":[\"sugar\",\"shock\"]},\"profileDigest\":\"1900eab6c028483d7126599ee6f50de0d27907b5c65fa90524580b4b0f9852b0\",\"telemetry\":{\"brainTicks\":\"2500\",\"learning\":{\"changed\":\"2\",\"enabled\":true,\"signal\":0.5,\"updates\":\"4\"},\"populationRateHz\":12.5,\"rates\":[{\"hz\":3.25,\"roleId\":\"kenyon\"},{\"hz\":0,\"roleId\":\"mbon\"}]},\"tickDuration\":{\"denominator\":\"1\",\"numerator\":\"1000000\"},\"warmupTicks\":\"2500\"}", + "digest": "707803be4867dc2f0f3d4bccedfaa7b97b1e52b9af78d21ae6957caba1a7e3f8" }, { "name": "prepare params", @@ -2598,6 +2611,100 @@ "canonical": "{\"agents\":[{\"agentId\":\"fly-a\",\"appliedControls\":{\"axes\":[{\"id\":\"stick-x\",\"value\":0},{\"id\":\"trigger\",\"value\":0}],\"buttons\":[{\"down\":true,\"id\":\"a\"},{\"down\":false,\"id\":\"b\"},{\"down\":false,\"id\":\"start\"},{\"down\":false,\"id\":\"select\"},{\"down\":false,\"id\":\"up\"},{\"down\":false,\"id\":\"down\"},{\"down\":false,\"id\":\"left\"},{\"down\":false,\"id\":\"right\"}],\"portId\":\"port-1\"},\"selectedDecision\":{\"schema\":{\"digest\":\"e70451b26fb5462ec63729d8355912dad01c3963a95bec0e6a8452375717ad9d\",\"id\":\"gameboy.intent.v1\",\"version\":1},\"value\":{\"press\":\"a\"}},\"telemetry\":{\"brainTicks\":\"2534\",\"learning\":{\"changed\":\"2\",\"enabled\":true,\"signal\":0.5,\"updates\":\"4\"},\"populationRateHz\":12.5,\"rates\":[{\"hz\":3.25,\"roleId\":\"kenyon\"},{\"hz\":0,\"roleId\":\"mbon\"}]}}],\"descriptorRevision\":\"7\",\"episodeId\":\"episode-1\",\"eventIds\":[\"evt-1\"],\"media\":{\"audio\":[{\"discontinuity\":false,\"firstSample\":\"33600\",\"sampleFrames\":800,\"samples\":{\"artifactId\":\"audio-1\",\"byteLength\":\"6400\",\"contentType\":\"audio/x-f32le\",\"digest\":null,\"generation\":\"1\",\"storeId\":\"store-1\"},\"streamId\":\"mix\"}],\"views\":[{\"pixels\":{\"artifactId\":\"frame-1\",\"byteLength\":\"92160\",\"contentType\":\"image/x-rgba8\",\"digest\":null,\"generation\":\"1\",\"storeId\":\"store-1\"},\"producedStep\":\"42\",\"viewId\":\"screen\"}]},\"progress\":{\"schema\":{\"digest\":\"80fa973157f334bb3208dfff6f7a2f298e79f73f07c72c33f5725975c6cff8e1\",\"id\":\"pokemon.progress.v1\",\"version\":1},\"value\":{\"rank\":10}},\"publisherIncarnation\":\"pub-1\",\"scope\":{\"epoch\":\"epoch-1\",\"sessionId\":\"demo\",\"step\":\"42\"},\"sequence\":\"42\",\"worldTime\":{\"denominator\":\"1\",\"numerator\":\"700000000\"}}", "digest": "d5ed83ccb866318b3cf17b53c8a5b1dea2f404b10649f6b640bf5f2b88a31435" }, + { + "name": "committed snapshot at a boundary installed by a restore", + "type": "CommittedSnapshot", + "value": { + "descriptorRevision": "7", + "publisherIncarnation": "pub-1", + "scope": { + "sessionId": "demo", + "epoch": "epoch-1", + "step": "42" + }, + "episodeId": "episode-1", + "sequence": "42", + "worldTime": { + "numerator": "700000000", + "denominator": "1" + }, + "agents": [ + { + "agentId": "fly-a", + "telemetry": { + "brainTicks": "2534", + "populationRateHz": 12.5, + "rates": [ + { + "roleId": "kenyon", + "hz": 3.25 + }, + { + "roleId": "mbon", + "hz": 0.0 + } + ], + "learning": { + "enabled": true, + "updates": "4", + "changed": "2", + "signal": 0.5 + } + }, + "selectedDecision": null, + "appliedControls": null + } + ], + "progress": { + "schema": { + "id": "pokemon.progress.v1", + "version": 1, + "digest": "80fa973157f334bb3208dfff6f7a2f298e79f73f07c72c33f5725975c6cff8e1" + }, + "value": { + "rank": 10 + } + }, + "media": { + "views": [ + { + "viewId": "screen", + "producedStep": "42", + "pixels": { + "storeId": "store-1", + "artifactId": "frame-1", + "generation": "1", + "byteLength": "92160", + "contentType": "image/x-rgba8", + "digest": null + } + } + ], + "audio": [ + { + "streamId": "mix", + "firstSample": "33600", + "sampleFrames": 800, + "samples": { + "storeId": "store-1", + "artifactId": "audio-1", + "generation": "1", + "byteLength": "6400", + "contentType": "audio/x-f32le", + "digest": null + }, + "discontinuity": false + } + ] + }, + "eventIds": [ + "evt-1" + ] + }, + "note": "a restore re-establishes a committed boundary this epoch did not run a transition into", + "canonical": "{\"agents\":[{\"agentId\":\"fly-a\",\"appliedControls\":null,\"selectedDecision\":null,\"telemetry\":{\"brainTicks\":\"2534\",\"learning\":{\"changed\":\"2\",\"enabled\":true,\"signal\":0.5,\"updates\":\"4\"},\"populationRateHz\":12.5,\"rates\":[{\"hz\":3.25,\"roleId\":\"kenyon\"},{\"hz\":0,\"roleId\":\"mbon\"}]}}],\"descriptorRevision\":\"7\",\"episodeId\":\"episode-1\",\"eventIds\":[\"evt-1\"],\"media\":{\"audio\":[{\"discontinuity\":false,\"firstSample\":\"33600\",\"sampleFrames\":800,\"samples\":{\"artifactId\":\"audio-1\",\"byteLength\":\"6400\",\"contentType\":\"audio/x-f32le\",\"digest\":null,\"generation\":\"1\",\"storeId\":\"store-1\"},\"streamId\":\"mix\"}],\"views\":[{\"pixels\":{\"artifactId\":\"frame-1\",\"byteLength\":\"92160\",\"contentType\":\"image/x-rgba8\",\"digest\":null,\"generation\":\"1\",\"storeId\":\"store-1\"},\"producedStep\":\"42\",\"viewId\":\"screen\"}]},\"progress\":{\"schema\":{\"digest\":\"80fa973157f334bb3208dfff6f7a2f298e79f73f07c72c33f5725975c6cff8e1\",\"id\":\"pokemon.progress.v1\",\"version\":1},\"value\":{\"rank\":10}},\"publisherIncarnation\":\"pub-1\",\"scope\":{\"epoch\":\"epoch-1\",\"sessionId\":\"demo\",\"step\":\"42\"},\"sequence\":\"42\",\"worldTime\":{\"denominator\":\"1\",\"numerator\":\"700000000\"}}", + "digest": "79a0ef10ed65c0720c34c7d99de1fe87ec2f0c26028994562cc4655b0f6acd1f" + }, { "name": "transition trace", "type": "TransitionTrace", diff --git a/services/flysim/crates/fly-session-types/src/publishing.rs b/services/flysim/crates/fly-session-types/src/publishing.rs index b90d8e5..10ba25c 100644 --- a/services/flysim/crates/fly-session-types/src/publishing.rs +++ b/services/flysim/crates/fly-session-types/src/publishing.rs @@ -15,8 +15,9 @@ use crate::workers::{ AgentTelemetry, AssetRef, EnvironmentDescriptor, MAX_AGENTS, MAX_RATE_ROLES, PortControl, }; -/// Declared stimulus kinds per agent. Not a stated bound; recorded in the schema set. -pub const MAX_SUPPORTED_STIMULI: usize = 64; +/// Declared stimulus kinds per agent, re-exported from its defining module. +pub use crate::workers::MAX_SUPPORTED_STIMULI; + /// Installed assets in one descriptor. Not a stated bound; recorded in the schema set. pub const MAX_ASSETS: usize = 64; /// Scoped event ids in one snapshot. Not a stated bound; recorded in the schema set. @@ -413,7 +414,8 @@ impl DomainType for CommittedSnapshot { controls.validate()?; } // "Decisions/controls describe the transition ending at that boundary, null at - // initial boundary 0." (publishing-v1 section 3) + // initial boundary 0." (publishing-v1 section 3, and its 2026-09-22 amendment for + // a boundary that was installed rather than produced.) if self.scope.step == 0 && (agent.selected_decision.is_some() || agent.applied_controls.is_some()) { @@ -421,14 +423,24 @@ impl DomainType for CommittedSnapshot { "CommittedSnapshot: at boundary 0 selectedDecision and appliedControls are null", ); } - if self.scope.step > 0 - && (agent.selected_decision.is_none() || agent.applied_controls.is_none()) - { + if agent.selected_decision.is_some() != agent.applied_controls.is_some() { return err( - "CommittedSnapshot: past boundary 0 every agent has a decision and applied controls", + "CommittedSnapshot: selectedDecision and appliedControls are null together or present together", ); } } + // A boundary is produced by a transition or installed by one, and the whole snapshot + // says which: every agent carries the transition that ended here, or none does. A + // mixture would be one fly's action beside another fly's silence at the same boundary. + if self + .agents + .iter() + .any(|a| a.selected_decision.is_some() != self.agents[0].selected_decision.is_some()) + { + return err( + "CommittedSnapshot: either every agent carries the transition that ended here, or none does", + ); + } self.progress.validate()?; if self.views.len() > MAX_VIEWS { return err("CommittedSnapshot: at most 8 views"); diff --git a/services/flysim/crates/fly-session-types/src/schema.rs b/services/flysim/crates/fly-session-types/src/schema.rs index 6317620..57ed6a7 100644 --- a/services/flysim/crates/fly-session-types/src/schema.rs +++ b/services/flysim/crates/fly-session-types/src/schema.rs @@ -438,6 +438,22 @@ pub const SCHEMAS: &[TypeSchema] = &[ req("committedStep", "U64", "\"0\""), req("decisionContextDigest", "Digest", ""), req("telemetry", "AgentTelemetry", ""), + req("graph", "AgentGraph", "rates are in graph.rateRoles order"), + ], + }, + TypeSchema { + name: "AgentGraph", + source: "workers-v1 2", + fields: &[ + req("datasetDigest", "Digest", ""), + req( + "indexDigest", + "Digest", + "geometry mapping needs this, not neuronCount", + ), + req("neuronCount", "U64", ""), + req("rateRoles", "array", "<= 64, unique"), + req("supportedStimuli", "array", "<= 64, unique"), ], }, TypeSchema { @@ -924,12 +940,12 @@ pub const SCHEMAS: &[TypeSchema] = &[ opt( "selectedDecision", "TypedValue|null", - "null exactly at boundary 0", + "null at boundary 0 and at an installed boundary; null or present for every agent together", ), opt( "appliedControls", "PortControl|null", - "null exactly at boundary 0; the agent's assigned port", + "null with selectedDecision; the agent's assigned port", ), ], }, diff --git a/services/flysim/crates/fly-session-types/src/workers.rs b/services/flysim/crates/fly-session-types/src/workers.rs index 7ff4d07..9a785c9 100644 --- a/services/flysim/crates/fly-session-types/src/workers.rs +++ b/services/flysim/crates/fly-session-types/src/workers.rs @@ -19,6 +19,8 @@ pub const MAX_AGENTS: usize = 4; pub const MAX_PORTS: usize = 4; /// 64 rate roles per agent. pub const MAX_RATE_ROLES: usize = 64; +/// Declared stimulus kinds per agent. Not a stated bound; recorded in the schema set. +pub const MAX_SUPPORTED_STIMULI: usize = 64; /// Arrays of stimuli or rewards are bounded to 64 per operation (workers-v1 section 1). pub const MAX_STIMULI: usize = 64; /// Arrays of stimuli or rewards are bounded to 64 per operation. @@ -706,6 +708,92 @@ pub struct AgentInitializeResult { pub committed_step: u64, pub decision_context_digest: String, pub telemetry: AgentTelemetry, + /// The graph identity this agent actually loaded, which is what a descriptor publishes. + /// + /// `publishing-v1` section 3 requires `datasetDigest`, `indexDigest`, `neuronCount`, + /// `rateRoles` and `supportedStimuli` in every `AgentDescriptor`, and before the + /// 2026-09-22 amendment to `workers-v1` section 2 no worker method carried them: a + /// coordinator could only have restated its own configuration. The worker attests + /// instead, so a fly that built another index is a visible mismatch rather than a + /// descriptor that agrees with itself. + pub graph: AgentGraph, +} + +/// What one agent's loaded graph is, as the agent reports it. +/// +/// `neuronCount` does not identify a mapping: "geometry/spike mapping requires indexDigest, +/// not merely the same number of neurons" (publishing-v1 section 3), so both travel and a +/// consumer compares the digest. +#[derive(Clone, Debug, PartialEq)] +pub struct AgentGraph { + pub dataset_digest: String, + pub index_digest: String, + pub neuron_count: u64, + /// The profile-defined rate-role order. `AgentTelemetry.rates` is in exactly this order. + pub rate_roles: Vec, + pub supported_stimuli: Vec, +} + +impl AgentGraph { + pub fn from_json(value: &Value) -> Result { + let mut f = Fields::new(value, "AgentGraph")?; + let dataset_digest = f.string("datasetDigest")?.to_owned(); + let index_digest = f.string("indexDigest")?.to_owned(); + let neuron_count = f.u64_string("neuronCount")?; + let rate_roles = id_list(&mut f, "rateRoles", 0, MAX_RATE_ROLES)?; + let supported_stimuli = id_list(&mut f, "supportedStimuli", 0, MAX_SUPPORTED_STIMULI)?; + f.finish()?; + let g = AgentGraph { + dataset_digest, + index_digest, + neuron_count, + rate_roles, + supported_stimuli, + }; + g.validate()?; + Ok(g) + } + + pub fn to_json(&self) -> Value { + obj(vec![ + ("datasetDigest", self.dataset_digest.clone().into()), + ("indexDigest", self.index_digest.clone().into()), + ("neuronCount", u64_json(self.neuron_count)), + ( + "rateRoles", + Value::Array(self.rate_roles.iter().map(|r| r.clone().into()).collect()), + ), + ( + "supportedStimuli", + Value::Array( + self.supported_stimuli + .iter() + .map(|s| s.clone().into()) + .collect(), + ), + ), + ]) + } + + pub fn validate(&self) -> Result<()> { + if !is_digest(&self.dataset_digest) || !is_digest(&self.index_digest) { + return err("AgentGraph: datasetDigest and indexDigest must be 64 lowercase hex digits"); + } + if self.rate_roles.len() > MAX_RATE_ROLES { + return err("AgentGraph: at most 64 rate roles"); + } + if self.supported_stimuli.len() > MAX_SUPPORTED_STIMULI { + return err("AgentGraph: at most 64 supported stimuli"); + } + require_unique( + self.rate_roles.iter().map(String::as_str), + "AgentGraph.rateRoles", + )?; + require_unique( + self.supported_stimuli.iter().map(String::as_str), + "AgentGraph.supportedStimuli", + ) + } } impl DomainType for AgentInitializeResult { @@ -720,6 +808,7 @@ impl DomainType for AgentInitializeResult { let committed_step = f.u64_string("committedStep")?; let decision_context_digest = f.string("decisionContextDigest")?.to_owned(); let telemetry = AgentTelemetry::from_json(f.value("telemetry")?)?; + let graph = AgentGraph::from_json(f.value("graph")?)?; f.finish()?; let r = AgentInitializeResult { agent_id, @@ -729,6 +818,7 @@ impl DomainType for AgentInitializeResult { committed_step, decision_context_digest, telemetry, + graph, }; r.validate()?; Ok(r) @@ -746,6 +836,7 @@ impl DomainType for AgentInitializeResult { self.decision_context_digest.clone().into(), ), ("telemetry", self.telemetry.to_json()), + ("graph", self.graph.to_json()), ]) } @@ -762,7 +853,10 @@ impl DomainType for AgentInitializeResult { if self.committed_step != 0 { return err("AgentInitializeResult: committedStep must be \"0\""); } - self.telemetry.validate() + self.graph.validate()?; + // The rates a worker reports and the role order it declares are one statement, so a + // descriptor built from the second can never mislabel the first. + self.telemetry.validate_against_roles(&self.graph.rate_roles) } } diff --git a/services/flysim/crates/fly-session/README.md b/services/flysim/crates/fly-session/README.md index 76b8b72..9b8ee00 100644 --- a/services/flysim/crates/fly-session/README.md +++ b/services/flysim/crates/fly-session/README.md @@ -39,6 +39,7 @@ Ready(k) ─ Prepare all agents concurrently ─────────── | `environment` | The counter arena: one complete batch per advance, one native frame | | `task` | The task and executor traits, the deterministic counter task, the identity executor | | `rpc` | Domain calls: `req-` serials, incarnation pinning, the retry rule | +| `publish` | The publication boundary: declared delivery policies, named publication outcomes, the bounded event batch, the read-only repair service, an application channel and a fake multi-agent consumer | | `coordinator` | The transaction, the trace, the failure rules and the publication boundary | | `launcher` | The supervisor: thread budget, identities, start, health check, reap | | `metrics` | Latency percentiles and the machine's core and memory counters | @@ -231,6 +232,29 @@ The durable store is `state`, over the `FLYSESS1` layout the contract crate owns it takes -- the transition finishes, then the session pauses at the boundary it just committed -- is now written into the section 2 machine as a dated amendment. +## The publication boundary + +`publishing-v1` on the same bus, with nothing added to the router: + +| Address | Delivery | Contents | +| --- | --- | --- | +| `session..descriptor` | retained latest | `SessionDescriptor`, built from what each participant attested to | +| `session..snapshots` | retained latest | `CommittedSnapshot` plus the boundary's media handles | +| `session..events` | bounded, depth 64 | the transition's task events, with a `droppedBefore` count | +| `session..query` | RPC, read-only | `Session.GetDescriptor`, `Session.GetSnapshot` | +| `.state`, `.cues` | the application's own | whatever the experience needs, under the application's schema | + +Every publication returns a named outcome: `Accepted`, `RefusedByObserver` or `Faulted`. Only +`BACKPRESSURE` is an observer's refusal, and a refusal takes no world step, stalls nothing and +fences no epoch -- it is counted per topic in the ledger and the exact value stays readable +through the query service. Anything else is the session's own fault and fails the epoch. A +snapshot is checked before it is published and again when it is read: every frame comes from +the boundary its declared delay implies, every handle is the artifact its reference names, +audio never goes backwards, and the snapshot agrees with the descriptor revision it names. + +What is **not** here: the approved public v2 wire schemas and the stage adapters that speak +them. `implementation.md` sequences those after this slice and together with each other. + ## Limitations - **Fake workers.** There is no neural model and no emulator. What is modelled exactly is the @@ -239,6 +263,14 @@ The durable store is `state`, over the `FLYSESS1` layout the contract crate owns restore refuses one taken under another backend, content, patch, controller or parser identity. It does not migrate between compositions, and it does not try. - **No audience input.** The admitted pre-step stimulation list exists and is always empty. +- **One descriptor revision.** A revision changes when the composition does, and the only + in-session path to that is a group restore into a fresh epoch, which is STATE-01's. The + session publishes revision 1; the repair path, the revision cache and the index-change rule + are exercised against a second revision published by a `Publisher` of a second composition. +- **No per-subscriber eviction.** A bounded subscriber may refuse a publication, and Flybus v1 + has no operation to drop that one subscriber, so the refusal costs every subscriber that + boundary's delivery on a stream whose contract is "latest". See the 2026-09-22 amendment to + `state-media-v1` section 3. - **Pacing is coarse.** The pacing deadline rounds one step to whole nanoseconds for sleeping only; simulation time stays rational and that rounding never re-enters the accumulator. @@ -298,9 +330,13 @@ The three integration suites do not all run over both transports, and cannot: - `tests/processes.rs` runs over the Unix socket only, in all three execution modes. A participant in a process of its own has no in-memory transport to reach the router by, so the mode is the axis that suite varies and the transport is fixed. -- `tests/media.rs` and `tests/state.rs` run over both transports *and* in all three execution - modes: each acceptance body is written once and registered twice, by `both_transports!` in - the in-process composition and by `all_modes!` over the socket. +- `tests/media.rs`, `tests/state.rs` and `tests/publishing.rs` run over both transports *and* + in the execution modes: each acceptance body is written once and registered twice, by + `both_transports!` in the in-process composition and by `all_modes!` over the socket. + `tests/publishing.rs` registers a subset that way rather than all of it, because the + publication boundary lives in the coordinator: unlike the render counter and the sensor log + it crosses no process boundary and stays fully observable in all three modes, which + `the_publication_boundary_holds_in_every_execution_mode` asserts rather than assumes. - `tests/session.rs`: one world advance per complete batch; every agent Prepared before the advance; one task evaluation per transition; every agent committed before the next Prepare or @@ -317,6 +353,14 @@ The three integration suites do not all run over both transports, and cannot: allocation -- plus the sequential/reversed/parallel trace comparison across all three modes and the two process-mode section 4 rows: a router restart during a world advance, and an old worker's reply after a restart. +- `tests/publishing.rs`: the PUBLISH-01 acceptance bullets over both transports -- a consumer + that disconnects and one that stops consuming, a bounded observer's named refusal, every + boundary's media belonging to that boundary, a frame and a handle from another boundary + refused, an unheld revision repaired rather than inferred, a revision that was never + published, an index that moved under a mapped consumer, boundary 0's null decision, the + committed action being the transition that just ended, one snapshot carrying every agent, + application-owned state and cues, a held event batch, and the read-only query service -- + plus the first two generated once per execution mode by `all_modes!`. - `tests/state.rs`: the STATE-01 acceptance bullets -- an uninterrupted run and a resumed run committing the same behaviour once the epoch metadata is rebased, a corrupt payload failing the install as a group for every participant and for the coordinator's own ledger, a lost diff --git a/services/flysim/crates/fly-session/src/agent.rs b/services/flysim/crates/fly-session/src/agent.rs index 22b5ac5..ab2e1a1 100644 --- a/services/flysim/crates/fly-session/src/agent.rs +++ b/services/flysim/crates/fly-session/src/agent.rs @@ -212,6 +212,10 @@ pub struct AgentConfig { /// The thread allocation the launcher started this worker within. `workers-v1` requires /// `Agent.Initialize`'s `workerThreads` to lie inside it. pub worker_threads: usize, + /// Which graph this fly built. Two variants have the same `neuronCount` and different + /// `indexDigest`, which is the case `publishing-v1` section 3 says a consumer must not + /// mistake for the same mapping. + pub graph_variant: u64, /// Records every view this agent read, so a test can see which artifact reached it. /// /// It is this process's log: an agent with a process of its own writes to its own copy, @@ -456,6 +460,9 @@ impl FakeAgentWorker { committed_step: 0, decision_context_digest: self.context_digest.clone().expect("just set"), telemetry: self.model.telemetry(), + // The worker attests to the graph it loaded. A descriptor built from this can + // disagree with the composition; one built from the composition never could. + graph: synthetic_graph(&self.config.agent_id, self.config.graph_variant), }; Ok(HandlerReply::from(&result)) } @@ -507,6 +514,7 @@ impl FakeAgentWorker { } for stimulus in ¶ms.pre_step_stimulations { stimulus.validate().map_err(DomainError::invalid)?; + check_supported(stimulus)?; } let available = FakeAgentWorker::available_actions(self.context.as_ref().expect("initialized"))?; @@ -594,6 +602,7 @@ impl FakeAgentWorker { } for stimulus in ¶ms.task_stimulations { stimulus.validate().map_err(DomainError::invalid)?; + check_supported(stimulus)?; } params.next_decision_context.validate().map_err(DomainError::invalid)?; FakeAgentWorker::available_actions(¶ms.next_decision_context)?; @@ -733,13 +742,61 @@ pub fn agent_op_class(method: &str) -> Option { } /// A synthetic profile asset for one agent. The digest covers its effective identities. +/// Refuses a stimulus kind this profile does not resolve, before the model is touched. +/// +/// `supportedStimuli` in a published descriptor is exactly this list, so the declaration is +/// what the worker enforces rather than a label printed beside it. +fn check_supported(stimulus: &Stimulus) -> DomainResult<()> { + if SUPPORTED_STIMULI.contains(&stimulus.kind_id.as_str()) { + return Ok(()); + } + Err(DomainError::before( + ErrorCode::Unsupported, + format!( + "stimulus kind {} is not one this profile resolves", + stimulus.kind_id + ), + )) +} + +/// The rate roles this fake model reports, in the order it reports them. +pub const RATE_ROLES: [&str; 2] = ["kc", "mbon"]; + +/// The stimulus kinds this synthetic profile resolves. An undeclared kind is refused before +/// the model is touched, so `supportedStimuli` in a descriptor is what the worker enforces +/// rather than a label beside it. +pub const SUPPORTED_STIMULI: [&str; 1] = ["arena.milestone"]; + +/// This fly's graph identity. Every variant has the same neuron count and its own index, so +/// "the same number of neurons" can never be mistaken for the same mapping. +pub const NEURON_COUNT: u64 = 1024; + +pub fn synthetic_graph(agent_id: &Id, variant: u64) -> AgentGraph { + AgentGraph { + dataset_digest: digest_of_bytes( + format!("arena-dataset-v1\nvariant={variant}\n").as_bytes(), + ), + index_digest: digest_of_bytes( + format!( + "arena-index-v1\nagent={agent_id}\nvariant={variant}\nneurons={NEURON_COUNT}\n" + ) + .as_bytes(), + ), + neuron_count: NEURON_COUNT, + rate_roles: RATE_ROLES.iter().map(|r| id(r)).collect(), + supported_stimuli: SUPPORTED_STIMULI.iter().map(|s| id(s)).collect(), + } +} + pub fn synthetic_profile(agent_id: &Id, tick_duration: &RationalNs, warmup_ticks: u64) -> AssetRef { let text = format!( "arena-direct-v1\nagent={agent_id}\ntick={}/{}\nwarmup={warmup_ticks}\n", tick_duration.numerator, tick_duration.denominator ); AssetRef { - id: id("arena-direct-v1"), + // One installed asset per fly: a descriptor's `assets` are unique by id, and two + // profiles that differ in content are two assets, not one id with two digests. + id: parse_id(&format!("arena-direct-v1-{agent_id}")).expect("a prefix plus an agent id"), digest: digest_of_bytes(text.as_bytes()), byte_length: text.len() as u64, format: id("fly-profile-v1"), @@ -787,6 +844,7 @@ pub fn agent_compatibility_digest( model_version: &str, plasticity_version: &str, seed: i32, + index_digest: &Digest, ) -> Digest { let value = serde_json::json!({ "agentId": agent_id.as_str(), @@ -795,6 +853,11 @@ pub fn agent_compatibility_digest( "modelVersion": model_version, "plasticityVersion": plasticity_version, "seed": seed, + // The index the worker actually built, not a value recomputed from the dataset: the + // whole point is that the two can disagree. Without it a replacement fly that built + // another graph restores cleanly and is then published under its predecessor's + // `indexDigest`, which is the predecessor's graph identity crossing a recovery. + "indexDigest": index_digest.as_str(), }); digest_of(&value).expect("an agent compatibility block canonicalizes") } @@ -883,13 +946,15 @@ struct StagedAgent { impl FakeAgentWorker { /// This worker's own compatibility identity, from its configuration and a resolved seed. fn compatibility_digest(&self, profile: &AssetRef, seed: i32) -> Digest { + let graph = synthetic_graph(&self.config.agent_id, self.config.graph_variant); agent_compatibility_digest( &self.config.agent_id, &profile.digest, - &dataset_digest(), + &graph.dataset_digest, MODEL_VERSION, PLASTICITY_VERSION, seed, + &graph.index_digest, ) } @@ -1090,10 +1155,16 @@ worker; this worker is {other:?}" // of another agent's brain, fails here and never reaches activation. let computed = self.compatibility_digest(&profile, model.seed()); if computed != params.compatibility_digest { + let graph = synthetic_graph(&self.config.agent_id, self.config.graph_variant); return Err(incompatible(format!( - "the staged state's compatibility {computed} is not the {} the restore \ -requires", - params.compatibility_digest + "the staged state's compatibility {} is not the {computed} this worker is: \ +profile {}, dataset {}, index {}, model {MODEL_VERSION}, plasticity {PLASTICITY_VERSION}, \ +seed {}", + params.compatibility_digest, + profile.digest, + graph.dataset_digest, + graph.index_digest, + model.seed() ))); } let accumulator_value = value diff --git a/services/flysim/crates/fly-session/src/cli.rs b/services/flysim/crates/fly-session/src/cli.rs index bbd6bbf..603e375 100644 --- a/services/flysim/crates/fly-session/src/cli.rs +++ b/services/flysim/crates/fly-session/src/cli.rs @@ -204,6 +204,7 @@ fn serve(role: &str, options: &Options) -> Result<(), String> { tick_duration: options.rational(flags::TICK_NUMERATOR, flags::TICK_DENOMINATOR)?, warmup_ticks: options.u64(flags::WARMUP_TICKS, 0)?, worker_threads: threads, + graph_variant: options.u64(flags::GRAPH_VARIANT, 0)?, // This process's own log. The supervisor reads what crosses the bus, not this. sensors: crate::media::SensorLog::new(), faults: AgentFaults { diff --git a/services/flysim/crates/fly-session/src/coordinator.rs b/services/flysim/crates/fly-session/src/coordinator.rs index f01d72b..7d68aa1 100644 --- a/services/flysim/crates/fly-session/src/coordinator.rs +++ b/services/flysim/crates/fly-session/src/coordinator.rs @@ -16,6 +16,7 @@ use serde_json::{Map, Value, json}; use crate::clock::Pacing; use crate::media::{self, AudioTimelines}; +use crate::publish::PublicationOutcome; use crate::metrics::Metrics; use crate::phase::{Phase, PhaseMachine}; use crate::rpc::{self, DomainReply, Serials, WorkerRef}; @@ -54,6 +55,14 @@ pub struct Injections { pub altered_advance_controls: bool, /// Read and release the Advance result's frame, then replay the same operation. pub consume_advance_artifact_then_retry: bool, + /// Publish this boundary's snapshot naming the previous boundary's frame: new agent + /// state beside an older observation. + pub stale_published_view: bool, + /// Publish this boundary's snapshot with a handle that is not the artifact the snapshot + /// references: the same name, the same shape, another object. + pub substituted_published_handle: bool, + /// Ask an agent to apply a stimulus kind its published descriptor does not declare. + pub undeclared_stimulus: bool, } /// What an injection produced, for a test to assert on. @@ -183,6 +192,14 @@ impl Default for Deadlines { } } +/// The descriptor revision this slice publishes. +/// +/// A revision changes when the composition does -- a replaced fly with another index, a +/// different port assignment -- and the only in-session path to that is a group restore into +/// a fresh epoch, which is STATE-01's. So a session establishes revision 1 and the repair +/// path, not a revision counter with nothing to count. +pub const DESCRIPTOR_REVISION: u64 = 1; + /// The bus addresses this session publishes on. Chosen by the composition, not the router. #[derive(Clone, Debug)] pub struct Topics { @@ -217,6 +234,11 @@ pub struct AgentSlot { pub tick_duration: RationalNs, pub warmup_ticks: u64, pub committed_step: u64, + /// The graph this fly attested to at `Agent.Initialize`, which is what the published + /// descriptor says about it. `None` before initialization. + pub graph: Option, + /// The telemetry of the last committed boundary, which is what the snapshot publishes. + pub telemetry: Option, /// The tick count and remainder this agent last reported, which are what the checkpoint /// manifest records for it. They are metadata about the payload, never a substitute for /// it: the agent's own capture is the state that is restored. @@ -246,6 +268,8 @@ impl AgentSlot { tick_duration: RationalNs::ZERO, warmup_ticks: 0, committed_step: 0, + graph: None, + telemetry: None, brain_ticks: 0, remainder: RationalNs::ZERO, context: TypedValue::new(crate::task::context_schema(), Value::Object(Map::new())) @@ -294,6 +318,16 @@ pub struct Coordinator { media_names: Vec, serials: Serials, topics: Topics, + /// The publication boundary. Everything this session publishes goes through it, and + /// every outcome it returns is a named one. + publisher: crate::publish::Publisher, + /// The composition as published. Built from what the live participants attested to, + /// never restated from the configuration that asked for them. + session_descriptor: Option, + /// The revision the next descriptor publication carries. + descriptor_revision: u64, + /// The read-only repair service. Held so it stops with the session. + query: Option, pacing: Option, /// Set by whoever asks for a normal pause, possibly while a transition is in flight. pause: std::sync::Arc, @@ -331,6 +365,17 @@ pub struct Coordinator { started: std::time::Instant, last_advance_request: Option, last_commit_requests: Vec, + /// The previous committed boundary's broadcast references. Data only: no handle, no owner, + /// no retention, and nothing reads it but the publication fault injections. + previous_broadcast_views: Vec, +} + +/// The broadcast references of an observation, or none when there is no observation yet. +fn observation_views_of(observation: &Option) -> Vec { + observation + .as_ref() + .map(|o| o.broadcast_views.clone()) + .unwrap_or_default() } impl Coordinator { @@ -349,6 +394,7 @@ impl Coordinator { // Sorted agent-id order is the executor and control order, so it is fixed here once. agents.sort_by(|a, b| a.agent_id.cmp(&b.agent_id)); let topics = Topics::for_session(&session_id); + let publisher = crate::publish::Publisher::new(bus.clone(), &session_id, &epoch, &topics); Coordinator { bus, session_id, @@ -371,6 +417,10 @@ impl Coordinator { media::audio_attachment(crate::environment::AUDIO_STREAM_ID), ], serials: Serials::default(), + publisher, + session_descriptor: None, + descriptor_revision: DESCRIPTOR_REVISION, + query: None, topics, pacing: None, pause: std::sync::Arc::new(std::sync::atomic::AtomicBool::new(false)), @@ -395,6 +445,7 @@ impl Coordinator { started: std::time::Instant::now(), last_advance_request: None, last_commit_requests: Vec::new(), + previous_broadcast_views: Vec::new(), } } @@ -410,6 +461,37 @@ impl Coordinator { &self.topics } + /// The composition this session published, once it has. + pub fn session_descriptor(&self) -> Option<&SessionDescriptor> { + self.session_descriptor.as_ref() + } + + /// The revision the last published descriptor carried. + pub fn descriptor_revision(&self) -> u64 { + self.descriptor_revision + } + + /// The sequence the next published snapshot will carry. + pub fn published_sequence(&self) -> u64 { + self.publisher.sequence() + } + + /// What this session published and what became of it: accepted, refused by an observer, + /// or faulted, per topic. + pub fn ledger(&self) -> &crate::publish::Ledger { + self.publisher.ledger() + } + + /// The events the bounded batch is still holding because an observer refused them. + pub fn pending_events(&self) -> usize { + self.publisher.outbox().len() + } + + /// The read-only state the repair service answers from. + pub fn published_state(&self) -> crate::publish::SharedState { + self.publisher.state() + } + pub fn epoch(&self) -> &Id { &self.epoch } @@ -665,26 +747,14 @@ impl Coordinator { Ok(()) } + /// Declares every framework topic under the delivery policy the publisher holds. async fn declare_topics(&mut self) -> Outcome<()> { - for (name, retained) in [ - (self.topics.descriptor.clone(), flybus::Retained::Latest), - (self.topics.snapshots.clone(), flybus::Retained::Latest), - (self.topics.events.clone(), flybus::Retained::None), - // Checkpoint events are a stream of distinct facts, not a latest value: a - // "committed" that replaced a "queued" would erase the distinction the durable - // commit rules are built on. - (self.topics.checkpoints.clone(), flybus::Retained::None), - ] { - self.bus.declare_topic(&name, retained).await.map_err(|e| { - let error = DomainError::new( - ErrorCode::BackendFailure, - format!("declaring {name}: {}", e.message), - MutationCertainty::None, - ); - self.fail_now(error, "declare-topic") - })?; + // Every framework topic, including the checkpoint stream, is declared in one place + // under the delivery policy the publisher holds. + match self.publisher.declare().await { + Ok(()) => Ok(()), + Err(e) => Err(self.fail_now(e, "declare-topic")), } - Ok(()) } async fn initialize_environment(&mut self) -> Outcome<()> { @@ -864,6 +934,10 @@ impl Coordinator { .telemetry .validate() .map_err(|e| self.fail_now(DomainError::invalid(e), "agent-initialize"))?; + // What the fly says it built. The descriptor publishes this, so a composition that + // loaded another index is visible in the descriptor rather than only in a log line. + self.agents[index].graph = Some(result.graph.clone()); + self.agents[index].telemetry = Some(result.telemetry.clone()); self.agents[index].tick_duration = result.tick_duration; self.agents[index].warmup_ticks = result.warmup_ticks; self.agents[index].committed_step = 0; @@ -1587,8 +1661,30 @@ impl Coordinator { self.agents[index].context_digest = context.digest(); self.agents[index].context = context; self.agents[index].committed_step = k + 1; + // The telemetry of the transition that just ended, which is what this boundary's + // snapshot publishes. Without this the slot would keep whatever `Agent.Initialize` + // reported and every snapshot would label warm-up telemetry as boundary k. + let telemetry = commits + .iter() + .find(|(id, _)| *id == agent_id) + .map(|(_, result)| result.telemetry.clone()); + match telemetry { + Some(telemetry) => self.agents[index].telemetry = Some(telemetry), + None => { + return Err(self.fail_now( + DomainError::before( + ErrorCode::IdentityMismatch, + format!("agent {agent_id} committed without telemetry"), + ), + "commit", + )); + } + } } // The previous boundary's handles are no longer needed; the new ones take over. + // The references -- which are data, not ownership -- are kept for one boundary, so a + // publication fault injection can name an older frame without retaining it. + self.previous_broadcast_views = observation_views_of(&self.observation); self.views = new_views; self.audio = new_audio; self.pending_views.clear(); @@ -2250,13 +2346,23 @@ impl Coordinator { .expect("every agent prepared"); let outcome = outcomes.get(&agent_id).cloned().unwrap_or_default(); let next_context = next_contexts.get(&agent_id).cloned().expect("checked"); + let mut task_stimulations = outcome.stimulations.clone(); + if self.injections.undeclared_stimulus && self.injections.at_step == k { + // A kind outside the agent's published `supportedStimuli`. The declaration is + // only worth publishing if the worker enforces it. + task_stimulations.push(Stimulus { + id: parse_id(&format!("stim-undeclared-{k}")).expect("a serial makes an Id"), + kind_id: id("arena.undeclared"), + duration_ms: 1.0, + }); + } let params = CommitParams { agent_id: agent_id.clone(), prepared_request_id: prepared_request.clone(), next_input: self.sensory_input(observation, k + 1), next_decision_context: next_context, rewards: outcome.rewards.clone(), - task_stimulations: outcome.stimulations.clone(), + task_stimulations, }; let params = match params.to_json() { Value::Object(m) => m, @@ -2513,59 +2619,115 @@ impl Coordinator { // ----------------------------------------------------------------------------------- // Publication - async fn publish( - &mut self, - topic: &str, - payload: Map, - attachments: Vec<(String, flybus::Artifact)>, - ) -> Outcome<()> { - let refs: Vec<(&str, &flybus::Artifact)> = - attachments.iter().map(|(n, a)| (n.as_str(), a)).collect(); - match self.bus.publish(topic, payload, &refs).await { - Ok(_) => Ok(()), - Err(e) => { - // A disconnected or backpressured observer never stalls the world; only a - // real resource fault reaches here, and it fails the epoch honestly. - let error = DomainError::new( - ErrorCode::BackendFailure, - format!("publishing {topic}: {}", e.message), - MutationCertainty::None, - ); - Err(self.fail_now(error, "publish")) + /// Sends one publication and turns its outcome into the session's response to it. + /// + /// An observer's refusal is counted and the world carries on: "ordinary snapshot + /// publication is latest/bounded and never waits for a spectator to consume it" + /// (publishing-v1 section 3), and `bus-v1` section 6 allows a bounded subscriber to reject + /// a publication. A session resource fault is not an observer and fails the epoch. + fn settle(&mut self, outcome: PublicationOutcome, detail: &str) -> Outcome { + match outcome.fault() { + Some(error) => Err(self.fail_now(error, detail)), + None => { + if outcome.is_refused() { + self.audit.push(format!("refused:{}", outcome.topic())); + } + Ok(outcome) } } } + /// The composition as the live participants attested to it. + /// + /// Every agent row comes from that agent's own `Agent.Initialize` reply, so a descriptor + /// can disagree with the configuration that asked for the composition. One restated from + /// the configuration never could, and `publishing-v1` section 3 needs the disagreement to + /// be visible: "geometry/spike mapping requires indexDigest, not merely the same number + /// of neurons". + fn build_descriptor(&self, revision: u64) -> DomainResult { + let environment = self.descriptor.clone().ok_or_else(|| { + DomainError::before(ErrorCode::InvalidPhase, "no environment descriptor") + })?; + let mut agents = Vec::new(); + let mut assets = Vec::new(); + for slot in &self.agents { + let graph = slot.graph.clone().ok_or_else(|| { + DomainError::before( + ErrorCode::InvalidPhase, + format!("agent {} has not attested to a graph", slot.agent_id), + ) + })?; + agents.push(AgentDescriptor { + agent_id: slot.agent_id.clone(), + port_id: slot.port_id.clone(), + profile_digest: slot.profile.digest.clone(), + dataset_digest: graph.dataset_digest, + index_digest: graph.index_digest, + neuron_count: graph.neuron_count, + rate_roles: graph.rate_roles, + supported_stimuli: graph.supported_stimuli, + }); + assets.push(slot.profile.clone()); + } + let descriptor = SessionDescriptor { + session_id: self.session_id.clone(), + revision, + composition_digest: self.composition_digest(), + environment, + task_schema: self.task.schema(), + agents, + assets, + }; + descriptor.validate().map_err(DomainError::invalid)?; + Ok(descriptor) + } + + /// Publishes the composition and starts the read-only repair service beside it. + /// Publishes the composition, advancing the revision when the composition changed. + /// + /// A revision identifies a composition, so republishing an unchanged one keeps its number + /// and a changed one takes the next: a group restore establishes a fresh epoch, which is a + /// new `compositionDigest`, and a consumer that held the old revision has to be told rather + /// than handed the same number with different contents. The publisher refuses the second + /// case outright, so this is where the number moves. async fn publish_descriptor(&mut self) -> Outcome<()> { - let descriptor = self.descriptor.clone().expect("bootstrapped"); - let agents: Vec = self - .agents - .iter() - .map(|slot| { - json!({ - "agentId": slot.agent_id.as_str(), - "portId": slot.port_id.as_str(), - "profileDigest": slot.profile.digest.as_str(), - "tickDuration": slot.tick_duration.to_json(), - "warmupTicks": slot.warmup_ticks.to_string(), - }) - }) - .collect(); - let payload = json!({ - "sessionId": self.session_id.as_str(), - "revision": "1", - "compositionDigest": self.composition_digest().as_str(), - "schedulerId": "lockstep-v1", - "environment": descriptor.to_json(), - "taskSchema": self.task.schema().to_json(), - "agents": agents, - }); - let topic = self.topics.descriptor.clone(); - self.publish(&topic, match payload { - Value::Object(m) => m, - _ => Map::new(), - }, Vec::new()) - .await + let mut descriptor = match self.build_descriptor(self.descriptor_revision) { + Ok(descriptor) => descriptor, + Err(e) => return Err(self.fail_now(e, "descriptor")), + }; + if let Some(published) = &self.session_descriptor { + let mut same = descriptor.clone(); + same.revision = published.revision; + if same != *published { + self.descriptor_revision += 1; + descriptor.revision = self.descriptor_revision; + self.audit + .push(format!("descriptor-revision:{}", self.descriptor_revision)); + } + } + let outcome = match self.publisher.publish_descriptor(&descriptor).await { + Ok(outcome) => outcome, + Err(e) => return Err(self.fail_now(e, "descriptor")), + }; + self.settle(outcome, "descriptor")?; + self.session_descriptor = Some(descriptor); + if self.query.is_none() { + let state = self.publisher.state(); + let service = + crate::publish::QueryService::start(self.bus.clone(), &self.session_id, state) + .await + .map_err(|e| { + let error = DomainError::new( + ErrorCode::BackendFailure, + format!("registering the session query service: {}", e.message), + MutationCertainty::None, + ); + self.fail_now(error, "query-service") + })?; + self.query = Some(service); + } + self.audit.push("publish:descriptor".to_owned()); + Ok(()) } /// The composition identity: session, epoch, agents, ports and the contract revision. @@ -2585,22 +2747,15 @@ impl Coordinator { digest_of_bytes(text.as_bytes()) } + /// Offers this boundary's events to the bounded batch and publishes what it holds. async fn publish_events(&mut self, source_step: u64, events: &[TaskEvent]) -> Outcome<()> { - if events.is_empty() { - return Ok(()); + match self.publisher.publish_events(source_step, events).await { + None => Ok(()), + Some(outcome) => { + self.settle(outcome, "events")?; + Ok(()) + } } - let payload = json!({ - "sessionId": self.session_id.as_str(), - "epoch": self.epoch.as_str(), - "sourceStep": source_step.to_string(), - "events": Value::Array(events.iter().map(DomainType::to_json).collect()), - }); - let topic = self.topics.events.clone(); - self.publish(&topic, match payload { - Value::Object(m) => m, - _ => Map::new(), - }, Vec::new()) - .await } /// Publishes the committed boundary. Never an in-progress mix of new agent state and an @@ -2621,55 +2776,180 @@ impl Coordinator { "publish", )); } + let descriptor = match self.session_descriptor.clone() { + Some(descriptor) => descriptor, + None => { + return Err(self.fail_now( + DomainError::before(ErrorCode::InvalidPhase, "no descriptor was published"), + "publish", + )); + } + }; let observation = self.observation.clone().expect("bootstrapped"); - let agents: Vec = self - .agents - .iter() - .map(|slot| { - let control = controls - .iter() - .find(|c| c.port_id == slot.port_id) - .map(|c| c.to_json()); - json!({ - "agentId": slot.agent_id.as_str(), - "selectedDecision": decisions - .get(&slot.agent_id) - .map(|d| d.to_json()), - "appliedControls": control, - "committedStep": slot.committed_step.to_string(), - }) - }) - .collect(); - let payload = json!({ - "descriptorRevision": "1", - "publisherIncarnation": self.bus.info().connection_id.clone(), - "scope": self.scope(boundary).to_json(), - "episodeId": self.episode_id.as_str(), - "sequence": self.stats.publications.to_string(), - "worldTime": observation.world_time.to_json(), - "agents": agents, - "progress": self.task.progress().to_json(), - "media": json!({ - "views": Value::Array(observation.broadcast_views.iter().map(DomainType::to_json).collect()), - "audio": Value::Array(observation.audio.iter().map(DomainType::to_json).collect()), - }), - "eventIds": event_ids.iter().map(Id::as_str).collect::>(), - }); - // The same owned handles the agents were given, published once for presentation. - let mut attachments = self.view_attachments(); - attachments.extend(self.audio.iter().map(|(n, a)| (n.clone(), a.clone()))); - let topic = self.topics.snapshots.clone(); - self.publish( - &topic, - match payload { - Value::Object(m) => m, - _ => Map::new(), - }, - attachments, - ) - .await?; - self.stats.publications += 1; - self.audit.push(format!("publish:{boundary}")); + let mut agents = Vec::new(); + for slot in &self.agents { + if slot.committed_step != boundary { + // A snapshot names one boundary. An agent that is not at it would be future + // state beside this world, which is the thing this check exists to refuse. + return Err(self.fail_now( + DomainError::before( + ErrorCode::IdentityMismatch, + format!( + "agent {} is committed at {} and the snapshot is boundary {boundary}", + slot.agent_id, slot.committed_step + ), + ), + "publish", + )); + } + let telemetry = match slot.telemetry.clone() { + Some(telemetry) => telemetry, + None => { + return Err(self.fail_now( + DomainError::before( + ErrorCode::InvalidPhase, + format!("agent {} reported no telemetry", slot.agent_id), + ), + "publish", + )); + } + }; + agents.push(SnapshotAgent { + agent_id: slot.agent_id.clone(), + telemetry, + // "Decisions/controls describe the transition ending at that boundary, null at + // initial boundary 0." These are the decisions of the transition that ended + // here, never the ones prepared for the transition about to start. + selected_decision: decisions.get(&slot.agent_id).cloned(), + applied_controls: controls.iter().find(|c| c.port_id == slot.port_id).cloned(), + }); + } + let mut views = observation.broadcast_views.clone(); + if self.injections.stale_published_view && self.injections.at_step + 1 == boundary { + // The injection of "new agent state with old media": the agents are at this + // boundary and the frame is the previous one's. + let stale = self.previous_broadcast_views.clone(); + if stale.is_empty() { + return Err(self.fail_now( + DomainError::invalid("no previous boundary to take a stale view from"), + "publish", + )); + } + views = stale; + self.injection_log.push(InjectionOutcome { + what: "stale-published-view".to_owned(), + code: None, + identical: false, + }); + } + let snapshot = CommittedSnapshot { + descriptor_revision: descriptor.revision, + publisher_incarnation: self.publisher.incarnation(), + scope: self.scope(boundary), + episode_id: self.episode_id.clone(), + sequence: self.publisher.sequence(), + world_time: observation.world_time, + agents, + progress: self.task.progress(), + views, + audio: observation.audio.clone(), + event_ids: event_ids.to_vec(), + }; + // The same owned handles the agents were given, published once for presentation. A + // referenced frame with no handle, or a handle that is another boundary's object, is + // refused by `check_publication` before anything reaches a subscriber. + let mut attachments = Vec::new(); + for view in &snapshot.views { + let name = media::view_attachment(&view.view_id); + match self.views.get(&name) { + Some(artifact) => attachments.push((name, artifact.clone())), + None => { + return Err(self.fail_now( + DomainError::new( + ErrorCode::BufferInvalid, + format!("this boundary holds no handle for view {}", view.view_id), + MutationCertainty::None, + ), + "publish", + )); + } + } + } + for chunk in &snapshot.audio { + let name = media::audio_attachment(&chunk.stream_id); + match self.audio.get(&name) { + Some(artifact) => attachments.push((name, artifact.clone())), + None => { + return Err(self.fail_now( + DomainError::new( + ErrorCode::BufferInvalid, + format!( + "this boundary holds no handle for stream {}", + chunk.stream_id + ), + MutationCertainty::None, + ), + "publish", + )); + } + } + } + if self.injections.substituted_published_handle && self.injections.at_step + 1 == boundary { + // The same attachment name and the same bytes, a different object. Only the + // artifact identity sees it, which is why the check compares that and not names. + let (name, artifact) = match attachments.first() { + Some(first) => first.clone(), + None => { + return Err(self.fail_now( + DomainError::invalid("no attachment to substitute"), + "publish", + )); + } + }; + let bytes = match artifact.read_all().await { + Ok(bytes) => bytes, + Err(e) => { + return Err(self.fail_now( + DomainError::new( + ErrorCode::BufferInvalid, + e.message, + MutationCertainty::None, + ), + "publish", + )); + } + }; + let copy = match media::seal_copy( + &self.bus, + artifact.reference().content_type.clone(), + &bytes, + ) + .await + { + Ok(copy) => copy, + Err(e) => return Err(self.fail_now(e, "publish")), + }; + attachments[0] = (name, copy); + self.injection_log.push(InjectionOutcome { + what: "substituted-published-handle".to_owned(), + code: None, + identical: false, + }); + } + let positions = self.timelines.positions(); + let outcome = match self + .publisher + .publish_snapshot(&descriptor, &snapshot, &attachments, &positions) + .await + { + Ok(outcome) => outcome, + Err(e) => return Err(self.fail_now(e, "publish")), + }; + let outcome = self.settle(outcome, "publish")?; + if outcome.is_accepted() { + self.stats.publications += 1; + self.audit.push(format!("publish:{boundary}")); + } Ok(()) } } @@ -2756,15 +3036,33 @@ impl Coordinator { } } - fn agent_compatibility(&self, slot: &AgentSlot) -> Digest { - crate::agent::agent_compatibility_digest( + /// One agent's compatibility identity, from what that agent attested to at + /// `Agent.Initialize` rather than from anything this coordinator recomputed. + /// + /// The graph belongs here because `state-media-v1`'s recovery rules say old parser or + /// media state must not cross a recovery, and a graph identity is exactly that: without it + /// a replacement fly that built another index passes the group check and is then published + /// under its predecessor's `indexDigest`. An agent that has not attested yet has no + /// compatibility, which is a refusal rather than a guessed digest. + fn agent_compatibility(&self, slot: &AgentSlot) -> DomainResult { + let graph = slot.graph.as_ref().ok_or_else(|| { + DomainError::before( + ErrorCode::InvalidPhase, + format!( + "agent {} has not attested to a graph, so it has no compatibility identity", + slot.agent_id + ), + ) + })?; + Ok(crate::agent::agent_compatibility_digest( &slot.agent_id, &slot.profile.digest, - &crate::agent::dataset_digest(), + &graph.dataset_digest, crate::agent::MODEL_VERSION, crate::agent::PLASTICITY_VERSION, slot.seed, - ) + &graph.index_digest, + )) } /// The coordinator's own session record: what it must hold again to resume this boundary. @@ -2953,7 +3251,10 @@ impl Coordinator { for index in 0..self.agents.len() { let slot_worker = self.agents[index].worker.clone(); let agent_id = self.agents[index].agent_id.clone(); - let expected = self.agent_compatibility(&self.agents[index]); + let expected = match self.agent_compatibility(&self.agents[index]) { + Ok(expected) => expected, + Err(e) => return Err(self.fail_now(e, "capture")), + }; let reply = self .call( &slot_worker, @@ -2971,10 +3272,25 @@ impl Coordinator { payloads.push(payload); acknowledge.push((slot_worker, reply.request_id.clone())); let slot = &self.agents[index]; + // The graph identities come from what this agent attested to, not from a value + // the coordinator recomputed; that is the whole point of recording them. + let graph = match slot.graph.clone() { + Some(graph) => graph, + None => { + return Err(self.fail_now( + DomainError::before( + ErrorCode::InvalidPhase, + format!("agent {agent_id} has not attested to a graph"), + ), + "capture", + )); + } + }; agent_rows.push(crate::state::AgentEntry { agent_id: agent_id.clone(), profile_digest: slot.profile.digest.clone(), - dataset_digest: crate::agent::dataset_digest(), + dataset_digest: graph.dataset_digest, + index_digest: graph.index_digest, model_version: crate::agent::MODEL_VERSION.to_owned(), plasticity_version: crate::agent::PLASTICITY_VERSION.to_owned(), seed: slot.seed, @@ -3262,8 +3578,9 @@ impl Coordinator { "boundary": boundary.to_string(), "detail": detail.map_or(Value::Null, |d| Value::String(d.to_owned())), }); - let topic = self.topics.checkpoints.clone(); - self.publish(&topic, object(payload), Vec::new()).await + let outcome = self.publisher.publish_checkpoint(object(payload)).await; + self.settle(outcome, "checkpoint-event")?; + Ok(()) } // --------------------------------------------------------------------------------------- @@ -3552,6 +3869,7 @@ impl Coordinator { &row.model_version, &row.plasticity_version, row.seed, + &row.index_digest, ), )); } diff --git a/services/flysim/crates/fly-session/src/harness.rs b/services/flysim/crates/fly-session/src/harness.rs index 53159d1..f86a179 100644 --- a/services/flysim/crates/fly-session/src/harness.rs +++ b/services/flysim/crates/fly-session/src/harness.rs @@ -47,6 +47,10 @@ pub struct AgentSpec { /// The threads this agent asks the launcher for. `Agent.Initialize` carries exactly what /// the launcher allocated, which `workers-v1` requires it to lie within. pub worker_threads: usize, + /// Which graph this fly builds. A replacement worker started on another variant has the + /// same neuron count and another `indexDigest`, which is the composition change a + /// descriptor revision exists to make visible. + pub graph_variant: u64, } impl AgentSpec { @@ -58,6 +62,7 @@ impl AgentSpec { seed, faults: AgentFaults::default(), worker_threads: 1, + graph_variant: 0, } } } @@ -164,6 +169,14 @@ fn agent_client(agent_id: &Id) -> String { format!("worker-{agent_id}") } +/// What a presentation consumer may do: subscribe, and ask the read-only repair service. +fn consumer_grants() -> Grants { + grants(|g| { + g.subscribe = vec![Pattern::prefix("session."), Pattern::prefix("app.")]; + g.call = vec![Pattern::prefix("session.")]; + }) +} + fn grants(f: impl FnOnce(&mut Grants)) -> Grants { let mut g = Grants::default(); f(&mut g); @@ -193,6 +206,8 @@ pub struct SessionHarness { /// The supervisor. It owns every participant's lifetime and thread allocation. pub launcher: Launcher, observers: Mutex>, + /// Which configured observer identity the next consumer takes. + next_observer: std::sync::atomic::AtomicUsize, /// Which generation of each participant is running: 1 is the one the composition started. generations: BTreeMap, /// Where the durable checkpoint store lives, for a test that reads the files themselves. @@ -221,6 +236,10 @@ impl SessionHarness { g.call = vec![Pattern::prefix("agent."), Pattern::prefix("env.")]; g.publish = vec![Pattern::prefix("session.")]; g.manage_topics = vec![Pattern::prefix("session.")]; + // The read-only repair service of publishing-v1 section 2. It is the + // session's own address and answers two queries; naming it is not + // authority over anything, and no method on it mutates. + g.register = vec![Pattern::prefix("session.")]; }), ) .client( @@ -232,7 +251,36 @@ impl SessionHarness { // The writer publishes the checkpoint events and never calls a participant. .client(WRITER_CLIENT, grants(|g| g.publish = vec![Pattern::prefix("session.")])) .client(ENV_CLIENT, grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)])) - .client("observer", grants(|g| g.subscribe = vec![Pattern::prefix("session.")])); + // A presentation consumer subscribes and may call the repair service. It can + // publish nothing, register nothing and reach no worker: "viewers/browser clients + // never obtain worker control" (publishing-v1 section 7). A bus client id is one + // connection, so a composition with several consumers configures several of them; + // they are the same grants, because a second viewer is not a more privileged one. + .client("observer", consumer_grants()) + .client("observer-2", consumer_grants()) + .client("observer-3", consumer_grants()) + .client("observer-4", consumer_grants()) + // The application's own publisher. Its addresses are its own, and it has no + // reach into the session's. + .client( + "application", + grants(|g| { + g.publish = vec![Pattern::prefix("app.")]; + g.manage_topics = vec![Pattern::prefix("app.")]; + g.subscribe = vec![Pattern::prefix("session.")]; + }), + ) + // The publication boundary, when a composition places it on a client of its own + // rather than on the coordinator's. + .client( + "publisher", + grants(|g| { + g.publish = vec![Pattern::prefix("session.")]; + g.manage_topics = vec![Pattern::prefix("session.")]; + }), + ); + // A replacement environment connects under its own client id, one per generation; + // this subsumes the single `-r2` identity the publication slice had configured. for generation in 2..=MAX_GENERATIONS { policy = policy.client( &format!("{ENV_CLIENT}-r{generation}"), @@ -307,6 +355,7 @@ impl SessionHarness { tick_duration, warmup_ticks: config.warmup_ticks, worker_threads: spec.worker_threads, + graph_variant: spec.graph_variant, sensors: sensors[&spec.agent_id].clone(), faults: spec.faults.clone(), client_id: agent_client(&spec.agent_id), @@ -374,6 +423,7 @@ impl SessionHarness { sensors, launcher, observers: Mutex::new(Vec::new()), + next_observer: std::sync::atomic::AtomicUsize::new(0), generations: BTreeMap::new(), checkpoint_root, }) @@ -410,17 +460,65 @@ impl SessionHarness { } /// An extra subscriber, for a test that watches the published boundaries. + /// + /// Each call takes the next configured observer identity: one bus client id is one + /// connection, so two consumers are two configured participants and not one identity + /// used twice. pub async fn observer(&self) -> Result { - let client = self.launcher.connect("observer").await?; + let index = self + .next_observer + .fetch_add(1, std::sync::atomic::Ordering::SeqCst); + let id = match index { + 0 => "observer".to_owned(), + n => format!("observer-{}", n + 1), + }; + let client = self.launcher.connect(&id).await?; self.observers.lock().expect("not poisoned").push(client.clone()); Ok(client) } + /// A client for the application that owns its own state and cues. + pub async fn application(&self) -> Result { + let client = self.launcher.connect("application").await?; + self.observers.lock().expect("not poisoned").push(client.clone()); + Ok(client) + } + + /// A client for a publication boundary of its own. + pub async fn publisher(&self) -> Result { + let client = self.launcher.connect("publisher").await?; + self.observers.lock().expect("not poisoned").push(client.clone()); + Ok(client) + } + + /// A fake multi-agent presentation consumer attached to this session's topics. + pub async fn consumer(&self) -> Result { + let client = self.observer().await?; + crate::publish::PresentationConsumer::attach( + client, + &self.config.session_id, + self.coordinator.topics(), + ) + .await + } + /// Replaces one agent's worker with a fresh incarnation, as a restore would. /// /// The coordinator still pins the old registration, so its next call to that agent fails /// rather than silently reaching another brain. pub async fn restart_agent(&mut self, agent_id: &Id) -> Result { + self.restart_agent_on_graph(agent_id, None).await + } + + /// Replaces one agent's worker, optionally with a fly that built another graph. + /// + /// `Some(variant)` is the composition change a descriptor revision exists for: the same + /// neuron count, another `indexDigest`. + pub async fn restart_agent_on_graph( + &mut self, + agent_id: &Id, + graph_variant: Option, + ) -> Result { let spec = self .config .agents @@ -442,6 +540,7 @@ impl SessionHarness { tick_duration, warmup_ticks: self.config.warmup_ticks, worker_threads: spec.worker_threads, + graph_variant: graph_variant.unwrap_or(spec.graph_variant), // The same log: a replacement worker in this process keeps writing where its // predecessor wrote, so a restore's sensory input is visible beside it. sensors: self.sensors.get(agent_id).cloned().unwrap_or_default(), @@ -550,6 +649,22 @@ impl SessionHarness { } } + /// Changes which graph one agent builds, so the replacement the next restart launches is + /// a fly with the same neuron count and another index. + /// + /// The same relaunch rule as a fault: the worker running now keeps what it was started + /// with, and the change reaches the composition through the next replacement. + pub fn set_agent_graph(&mut self, agent_id: &Id, graph_variant: u64) { + if let Some(spec) = self + .config + .agents + .iter_mut() + .find(|spec| spec.agent_id == *agent_id) + { + spec.graph_variant = graph_variant; + } + } + /// Changes the environment's injected faults, with the same relaunch rule. pub fn set_environment_faults(&mut self, faults: EnvironmentFaults) { self.config.environment_faults = faults; diff --git a/services/flysim/crates/fly-session/src/launcher.rs b/services/flysim/crates/fly-session/src/launcher.rs index 43d6705..f778858 100644 --- a/services/flysim/crates/fly-session/src/launcher.rs +++ b/services/flysim/crates/fly-session/src/launcher.rs @@ -231,6 +231,9 @@ pub struct AgentLaunch { /// gets a fresh log in that process, which the supervisor cannot read. pub sensors: crate::media::SensorLog, pub faults: AgentFaults, + /// Which graph this fly builds. Crosses a process boundary as argv, like every other + /// thing a worker is started with. + pub graph_variant: u64, /// The configured client id. A replacement worker connects under its own. pub client_id: String, pub service: String, @@ -1231,6 +1234,7 @@ pub(crate) mod flags { pub const PREPARE_DELAY_MS: &str = "prepare-delay-ms"; pub const COMMIT_DELAY_MS: &str = "commit-delay-ms"; pub const FAIL_COMMIT_AT_STEP: &str = "fail-commit-at-step"; + pub const GRAPH_VARIANT: &str = "graph-variant"; pub const FAIL_STAGE_RESTORE: &str = "fail-stage-restore"; pub const FAIL_ACTIVATE_RESTORE: &str = "fail-activate-restore"; @@ -1273,6 +1277,7 @@ pub(crate) mod flags { PREPARE_DELAY_MS, COMMIT_DELAY_MS, FAIL_COMMIT_AT_STEP, + GRAPH_VARIANT, FAIL_STAGE_RESTORE, FAIL_ACTIVATE_RESTORE, ]; @@ -1333,6 +1338,7 @@ impl Started { arg(flags::WARMUP_TICKS, spec.warmup_ticks), arg(flags::PREPARE_DELAY_MS, spec.faults.prepare_delay_ms), arg(flags::COMMIT_DELAY_MS, spec.faults.commit_delay_ms), + arg(flags::GRAPH_VARIANT, spec.graph_variant), arg(flags::FAIL_STAGE_RESTORE, u64::from(spec.faults.fail_stage_restore)), arg( flags::FAIL_ACTIVATE_RESTORE, @@ -1393,6 +1399,7 @@ pub(crate) fn agent_config(spec: &AgentLaunch, worker_threads: usize) -> AgentCo tick_duration: spec.tick_duration, warmup_ticks: spec.warmup_ticks, worker_threads, + graph_variant: spec.graph_variant, sensors: spec.sensors.clone(), faults: spec.faults.clone(), } @@ -1504,6 +1511,7 @@ mod flag_tests { tick_duration: RationalNs::new(1, 1_000).expect("a tick"), warmup_ticks: 10, worker_threads: 1, + graph_variant: 3, sensors: crate::media::SensorLog::new(), faults: AgentFaults { fail_commit_at_step: Some(2), diff --git a/services/flysim/crates/fly-session/src/lib.rs b/services/flysim/crates/fly-session/src/lib.rs index e6d66c0..6849001 100644 --- a/services/flysim/crates/fly-session/src/lib.rs +++ b/services/flysim/crates/fly-session/src/lib.rs @@ -33,6 +33,7 @@ pub mod measure; pub mod media; pub mod metrics; pub mod phase; +pub mod publish; pub mod rpc; pub mod state; pub mod task; diff --git a/services/flysim/crates/fly-session/src/media.rs b/services/flysim/crates/fly-session/src/media.rs index 1d6c138..c73ed54 100644 --- a/services/flysim/crates/fly-session/src/media.rs +++ b/services/flysim/crates/fly-session/src/media.rs @@ -442,6 +442,18 @@ impl AudioSource { } } +/// Allocates, writes and seals one immutable artifact of an arbitrary content type. +/// +/// Used where a test needs a second object with the same bytes, so that "the handle is not +/// the artifact the payload names" can be produced without corrupting the bytes. +pub async fn seal_copy( + client: &flybus::Client, + content_type: String, + bytes: &[u8], +) -> DomainResult { + seal(client, &content_type, bytes).await +} + /// Allocates, writes and seals one immutable artifact. async fn seal( client: &flybus::Client, diff --git a/services/flysim/crates/fly-session/src/publish.rs b/services/flysim/crates/fly-session/src/publish.rs new file mode 100644 index 0000000..8390298 --- /dev/null +++ b/services/flysim/crates/fly-session/src/publish.rs @@ -0,0 +1,1487 @@ +//! The publication boundary of `publishing-v1`: what leaves the session, and what a refusal is. +//! +//! This is the PUBLISH-01 slice. It is an *internal* boundary on the *same* bus: there is no +//! second transport, no gateway process, no codec and no show or tournament service. What it +//! adds is the separation the contract asks for: +//! +//! ```text +//! session..descriptor retained latest framework: what this composition is +//! session..snapshots retained latest framework: the values of one committed boundary +//! session..events bounded framework: scoped domain events, not a log +//! .state retained latest application-owned, application-shaped +//! .cues bounded application-owned, under a declared policy +//! session..query RPC, read-only the repair path for a missed descriptor +//! ``` +//! +//! Three rules decide everything below. +//! +//! 1. **A publication outcome is named.** [`PublicationOutcome`] is `Accepted`, +//! `RefusedByObserver` or `Faulted`; nothing is dropped, retried or defaulted silently. +//! `bus-v1` section 6 says plainly that "bounded event subscriptions can reject publication; +//! latest spectator subscriptions cannot hold a required session transaction indefinitely", +//! so a refusal is a thing the contract expects and this module counts, not an error to +//! swallow. Only `BACKPRESSURE` is an observer's refusal. A store quota, a lost router or an +//! unreadable payload is the session's own fault and fails the epoch. +//! 2. **An observer never moves the world.** Publication happens after the committed boundary +//! is established. A refusal changes no phase, takes no step and releases no handle, and a +//! latest observation topic supersedes the refused value at the next boundary, so a slow or +//! bounded observer costs its own delivery and one boundary of a stream whose contract is +//! "latest" -- never a tick, never a stall. The repair path recovers the exact value. +//! 3. **Agent state and media are one statement.** [`check_publication`] refuses a snapshot +//! whose media does not belong to the boundary its agent state belongs to, before anything +//! is published, so "future agent state with old media" is a failure rather than a frame. +//! +//! What this module deliberately does **not** contain: the approved public v2 wire schemas and +//! the stage adapters that speak them. `implementation.md` sequences those after this slice and +//! together with each other, and inventing a public byte format here would be exactly the +//! unapproved contract that ordering exists to prevent. + +use std::collections::{BTreeMap, VecDeque}; +use std::sync::{Arc, Mutex}; + +use serde_json::{Map, Value, json}; + +// `crate::types` is this crate's facade over the shared `fly-session-types` crate; the glob +// keeps the contract's own names in sight instead of restating them. +use crate::types::*; + +/// How many events one bounded batch may hold before the oldest are explicitly dropped. +pub const EVENT_BATCH_DEPTH: usize = 64; + +/// The in-flight credits a framework consumer takes on an observation topic. +pub const SPECTATOR_CREDITS: u32 = 2; + +// ---------------------------------------------------------------------------------------------- +// Policy + +/// The delivery a topic is published under. Declared once, never inferred per message. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Delivery { + /// One replaceable value. A consumer that falls behind loses intermediate values and + /// nothing else; that loss is the contract of the topic, not an accident of load. + LatestValue, + /// A bounded batch. Nothing is coalesced: when the batch is full the oldest entries are + /// dropped by an explicit count that travels with the next accepted batch. + BoundedBatch { depth: usize }, +} + +impl Delivery { + /// Whether the router retains the last value for a late subscriber. + /// + /// A latest observation is retained, because a consumer that arrives mid-session must be + /// able to start; a bounded event stream is not, because it is "not a durable log" + /// (publishing-v1 section 2) and a retained tail would look like one. + pub fn retained(self) -> flybus::Retained { + match self { + Delivery::LatestValue => flybus::Retained::Latest, + Delivery::BoundedBatch { .. } => flybus::Retained::None, + } + } + + /// The subscription a consumer of this topic takes. + pub fn subscription(self) -> flybus::SubscriptionConfig { + match self { + Delivery::LatestValue => flybus::SubscriptionConfig::latest() + .in_flight(SPECTATOR_CREDITS) + .replay(true), + Delivery::BoundedBatch { depth } => { + flybus::SubscriptionConfig::bounded().queued(depth as u32) + } + } + } +} + +/// One published address and the delivery it was declared under. +#[derive(Clone, Debug)] +pub struct TopicPolicy { + pub topic: String, + pub delivery: Delivery, +} + +impl TopicPolicy { + pub fn latest(topic: impl Into) -> TopicPolicy { + TopicPolicy { + topic: topic.into(), + delivery: Delivery::LatestValue, + } + } + + pub fn bounded(topic: impl Into, depth: usize) -> TopicPolicy { + TopicPolicy { + topic: topic.into(), + delivery: Delivery::BoundedBatch { depth }, + } + } +} + +// ---------------------------------------------------------------------------------------------- +// Outcomes + +/// What became of one publication. Every path through [`Publisher`] returns one of these. +#[derive(Clone, Debug, PartialEq)] +pub enum PublicationOutcome { + /// The router admitted it. `replaced` counts the queued values it coalesced away, which + /// is the only loss a latest subscriber can suffer and is reported, not hidden. + Accepted { + topic: String, + topic_sequence: u64, + subscribers: u64, + replaced: u64, + }, + /// A subscriber refused it. `bus-v1` section 5 rejects the whole publish for a bounded + /// subscriber's full queue, so this is an observer's doing: the world is untouched, the + /// value is recoverable through the query service, and the offender is named. + RefusedByObserver { topic: String, detail: String }, + /// The session's own resource or identity fault. This one fails the epoch. + Faulted { topic: String, detail: String }, +} + +impl PublicationOutcome { + pub fn topic(&self) -> &str { + match self { + PublicationOutcome::Accepted { topic, .. } + | PublicationOutcome::RefusedByObserver { topic, .. } + | PublicationOutcome::Faulted { topic, .. } => topic, + } + } + + pub fn is_accepted(&self) -> bool { + matches!(self, PublicationOutcome::Accepted { .. }) + } + + pub fn is_refused(&self) -> bool { + matches!(self, PublicationOutcome::RefusedByObserver { .. }) + } + + /// The domain error a *fault* carries, or `None` for an accepted or refused publication. + /// + /// The certainty is `none`: publication happens after the boundary is committed and + /// mutates no participant, so a failed publish has changed nothing in the world. + pub fn fault(&self) -> Option { + match self { + PublicationOutcome::Faulted { topic, detail } => Some(DomainError::new( + ErrorCode::BackendFailure, + format!("publishing {topic}: {detail}"), + MutationCertainty::None, + )), + _ => None, + } + } + + fn from_bus(topic: &str, result: Result) -> Self { + match result { + Ok(receipt) => PublicationOutcome::Accepted { + topic: topic.to_owned(), + topic_sequence: receipt.topic_sequence, + subscribers: receipt.subscribers, + replaced: receipt.replaced, + }, + // The one code an observer can cause. Everything else is ours. + Err(e) if e.code == flybus::ErrorCode::Backpressure => { + PublicationOutcome::RefusedByObserver { + topic: topic.to_owned(), + detail: e.message, + } + } + Err(e) => PublicationOutcome::Faulted { + topic: topic.to_owned(), + detail: format!("{:?}: {}", e.code, e.message), + }, + } + } +} + +/// Per-topic publication counters, for assertions and for an operator. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct TopicCounters { + pub accepted: u64, + pub refused: u64, + pub faulted: u64, + /// Values a latest subscriber's queue coalesced away, as the router reported them. + pub replaced: u64, +} + +/// What this session published and what happened to it. +#[derive(Clone, Debug, Default)] +pub struct Ledger { + topics: BTreeMap, + /// Events dropped from the bounded batch because the batch was full, cumulative. + pub events_dropped: u64, + /// Batches held for a later boundary because an observer refused them. + pub events_held: u64, + last: Option, +} + +impl Ledger { + pub fn counters(&self, topic: &str) -> TopicCounters { + self.topics.get(topic).cloned().unwrap_or_default() + } + + pub fn last(&self) -> Option<&PublicationOutcome> { + self.last.as_ref() + } + + /// Every refusal this session has met, by topic. + pub fn refusals(&self) -> u64 { + self.topics.values().map(|c| c.refused).sum() + } + + fn record(&mut self, outcome: &PublicationOutcome) { + let entry = self.topics.entry(outcome.topic().to_owned()).or_default(); + match outcome { + PublicationOutcome::Accepted { replaced, .. } => { + entry.accepted += 1; + entry.replaced += replaced; + } + PublicationOutcome::RefusedByObserver { .. } => entry.refused += 1, + PublicationOutcome::Faulted { .. } => entry.faulted += 1, + } + self.last = Some(outcome.clone()); + } +} + +// ---------------------------------------------------------------------------------------------- +// Published state and the repair path + +/// Everything the query service can answer from: the descriptor revisions this session has +/// published, and the latest committed snapshot. +/// +/// "Latest retained descriptors accelerate startup; RPC querying remains the repair path" +/// (publishing-v1 section 2). A consumer that meets a revision it does not hold asks here +/// instead of guessing the shape of the values it is reading. +#[derive(Clone, Debug, Default)] +pub struct PublishedState { + descriptors: BTreeMap, + latest: Option, +} + +impl PublishedState { + pub fn descriptor(&self, revision: u64) -> Option<&SessionDescriptor> { + self.descriptors.get(&revision) + } + + pub fn newest_descriptor(&self) -> Option<&SessionDescriptor> { + self.descriptors.values().next_back() + } + + pub fn latest_snapshot(&self) -> Option<&CommittedSnapshot> { + self.latest.as_ref() + } + + pub fn revisions(&self) -> Vec { + self.descriptors.keys().copied().collect() + } +} + +/// The shared handle the publisher writes and the query service reads. +pub type SharedState = Arc>; + +fn lock(state: &SharedState) -> std::sync::MutexGuard<'_, PublishedState> { + state + .lock() + .expect("the published state is never held across a panic") +} + +// ---------------------------------------------------------------------------------------------- +// The bounded event batch + +/// The pending event batch of `publishing-v1` section 6: bounded, explicit about loss. +/// +/// Events are "scoped domain events; not a durable log" (section 2), and "bus publish +/// acceptance and delivery consumption are not durable acknowledgments" (section 6). So this +/// keeps a bounded batch and says exactly what it could not keep: `droppedBefore` travels with +/// the next accepted batch, so a subscriber reads a count rather than inferring a gap. What it +/// never does is grow without bound, retry forever or forget quietly. +#[derive(Clone, Debug, Default)] +pub struct EventOutbox { + depth: usize, + pending: VecDeque<(u64, TaskEvent)>, + dropped_since_accepted: u64, +} + +impl EventOutbox { + pub fn new(depth: usize) -> EventOutbox { + EventOutbox { + depth, + pending: VecDeque::new(), + dropped_since_accepted: 0, + } + } + + /// Adds this boundary's events, dropping the oldest if the batch is over its depth. + /// Returns how many were dropped by this call. + pub fn offer(&mut self, source_step: u64, events: &[TaskEvent]) -> u64 { + for event in events { + self.pending.push_back((source_step, event.clone())); + } + let mut dropped = 0; + while self.pending.len() > self.depth { + self.pending.pop_front(); + dropped += 1; + } + self.dropped_since_accepted += dropped; + dropped + } + + pub fn is_empty(&self) -> bool { + self.pending.is_empty() + } + + pub fn len(&self) -> usize { + self.pending.len() + } + + pub fn dropped_since_accepted(&self) -> u64 { + self.dropped_since_accepted + } + + fn payload(&self, session_id: &Id, epoch: &Id) -> Map { + let events: Vec = self + .pending + .iter() + .map(|(source_step, event)| { + let mut value = event.to_json(); + if let Value::Object(map) = &mut value { + map.insert("sourceStep".to_owned(), source_step.to_string().into()); + } + value + }) + .collect(); + object(json!({ + "sessionId": session_id.as_str(), + "epoch": epoch.as_str(), + "droppedBefore": self.dropped_since_accepted.to_string(), + "events": Value::Array(events), + })) + } + + fn accepted(&mut self) { + self.pending.clear(); + self.dropped_since_accepted = 0; + } +} + +// ---------------------------------------------------------------------------------------------- +// Coherence + +/// Refuses a snapshot whose media does not belong to the boundary its agent state belongs to. +/// +/// This is the "future agent state is never mixed with old media" rule, checked on the way out +/// and again on the way in. Four things must agree: +/// +/// 1. Each published view's `producedStep` is exactly what its declared delay implies for this +/// boundary. A frame from an older boundary is a refusal, not a substitution. +/// 2. Each published audio chunk's sample range ends where the stream's next chunk begins, so +/// a chunk from a previous transition cannot ride along under a new boundary. +/// 3. Every referenced view and chunk has its attachment, and no attachment is present that +/// the payload does not reference. An extra handle is an old boundary's frame. +/// 4. The snapshot agrees with the descriptor it names: revision, agent set, port assignment, +/// rate roles. +pub fn check_publication( + descriptor: &SessionDescriptor, + snapshot: &CommittedSnapshot, + attachments: &[(String, ArtifactRef)], + audio_next_sample: &BTreeMap, +) -> DomainResult<()> { + snapshot + .validate_against(descriptor) + .map_err(|e| coherence(format!("snapshot and descriptor disagree: {e}")))?; + check_views(descriptor, snapshot)?; + check_attachments(snapshot, attachments)?; + for chunk in &snapshot.audio { + let end = chunk + .first_sample + .checked_add(chunk.sample_frames) + .ok_or_else(|| { + coherence(format!( + "audio stream {} overflows its sample position", + chunk.stream_id + )) + })?; + match audio_next_sample.get(&chunk.stream_id) { + Some(next) if *next == end => {} + Some(next) => { + return Err(coherence(format!( + "audio stream {} covers samples {}..{end} and this boundary ends at {next}", + chunk.stream_id, chunk.first_sample + ))); + } + None => { + return Err(coherence(format!( + "audio stream {} has no accepted position at this boundary", + chunk.stream_id + ))); + } + } + } + Ok(()) +} + +/// Every published view comes from exactly the boundary its declared delay implies. +pub fn check_views( + descriptor: &SessionDescriptor, + snapshot: &CommittedSnapshot, +) -> DomainResult<()> { + let boundary = snapshot.scope.step; + for view in &snapshot.views { + let declared = descriptor + .environment + .views + .iter() + .find(|v| v.view_id == view.view_id) + .ok_or_else(|| coherence(format!("view {} is not declared", view.view_id)))?; + let want = declared.required_produced_step(boundary); + if view.produced_step != want { + return Err(coherence(format!( + "view {} at boundary {boundary} was produced at {}, and its declared delay of {} requires {want}", + view.view_id, view.produced_step, declared.observation_delay_steps + ))); + } + } + Ok(()) +} + +/// The handles and the references are the same media: no missing frame, no extra one, and +/// each handle is the artifact its reference names. +/// +/// The identity comparison is the half that matters: an attachment set that matches by *name* +/// while one handle is the previous boundary's object is exactly "old media under new agent +/// state", and only the `ArtifactRef` sees it. +pub fn check_attachments( + snapshot: &CommittedSnapshot, + attachments: &[(String, ArtifactRef)], +) -> DomainResult<()> { + let mut want: Vec<(String, &ArtifactRef)> = snapshot + .views + .iter() + .map(|v| (crate::media::view_attachment(&v.view_id), &v.pixels)) + .chain( + snapshot + .audio + .iter() + .map(|a| (crate::media::audio_attachment(&a.stream_id), &a.samples)), + ) + .collect(); + want.sort_by(|a, b| a.0.cmp(&b.0)); + let mut given: Vec<(String, &ArtifactRef)> = + attachments.iter().map(|(n, r)| (n.clone(), r)).collect(); + given.sort_by(|a, b| a.0.cmp(&b.0)); + if want.len() != given.len() || want.iter().zip(&given).any(|(w, g)| w.0 != g.0) { + let want: Vec<&String> = want.iter().map(|(n, _)| n).collect(); + let given: Vec<&String> = given.iter().map(|(n, _)| n).collect(); + return Err(coherence(format!( + "the published handles {given:?} are not the media the snapshot references {want:?}" + ))); + } + for ((name, reference), (_, handle)) in want.iter().zip(&given) { + if reference != handle { + return Err(coherence(format!( + "the handle published as {name} is artifact {} generation {}, and the snapshot references {} generation {}", + handle.artifact_id, handle.generation, reference.artifact_id, reference.generation + ))); + } + } + Ok(()) +} + +fn coherence(message: impl std::fmt::Display) -> DomainError { + // Nothing was published, so nothing downstream saw a mixed boundary. + DomainError::new(ErrorCode::BufferInvalid, message, MutationCertainty::None) +} + +// ---------------------------------------------------------------------------------------------- +// The publisher + +/// The session's publication path. One bus client, three framework topics, named outcomes. +pub struct Publisher { + bus: flybus::Client, + session_id: Id, + epoch: Id, + descriptor_topic: TopicPolicy, + snapshot_topic: TopicPolicy, + event_topic: TopicPolicy, + /// The STATE-01 checkpoint stream. A stream of distinct facts, so it is a bounded + /// delivery and never a latest value: a "committed" that replaced a "queued" would erase + /// the distinction the durable commit rules are built on. + checkpoint_topic: TopicPolicy, + outbox: EventOutbox, + state: SharedState, + ledger: Ledger, + sequence: u64, +} + +impl Publisher { + pub fn new( + bus: flybus::Client, + session_id: &Id, + epoch: &Id, + topics: &crate::coordinator::Topics, + ) -> Publisher { + Publisher { + bus, + session_id: session_id.clone(), + epoch: epoch.clone(), + descriptor_topic: TopicPolicy::latest(&topics.descriptor), + snapshot_topic: TopicPolicy::latest(&topics.snapshots), + event_topic: TopicPolicy::bounded(&topics.events, EVENT_BATCH_DEPTH), + checkpoint_topic: TopicPolicy::bounded(&topics.checkpoints, EVENT_BATCH_DEPTH), + outbox: EventOutbox::new(EVENT_BATCH_DEPTH), + state: Arc::new(Mutex::new(PublishedState::default())), + ledger: Ledger::default(), + sequence: 0, + } + } + + /// The declared policies, in publication order. + pub fn policies(&self) -> Vec { + vec![ + self.descriptor_topic.clone(), + self.snapshot_topic.clone(), + self.event_topic.clone(), + self.checkpoint_topic.clone(), + ] + } + + pub fn ledger(&self) -> &Ledger { + &self.ledger + } + + pub fn state(&self) -> SharedState { + Arc::clone(&self.state) + } + + pub fn outbox(&self) -> &EventOutbox { + &self.outbox + } + + /// The next publication sequence, which is monotonic within this publisher incarnation. + pub fn sequence(&self) -> u64 { + self.sequence + } + + pub fn incarnation(&self) -> String { + self.bus.info().connection_id.clone() + } + + /// Declares every framework topic under its policy. A topic already declared compatibly + /// is accepted; a conflicting declaration is a fault here, not a silent reuse. + pub async fn declare(&self) -> DomainResult<()> { + for policy in self.policies() { + self.bus + .declare_topic(&policy.topic, policy.delivery.retained()) + .await + .map_err(|e| { + DomainError::new( + ErrorCode::BackendFailure, + format!("declaring {}: {}", policy.topic, e.message), + MutationCertainty::None, + ) + })?; + } + Ok(()) + } + + /// Publishes a descriptor revision and records it as answerable by the query service. + /// + /// The revision is recorded even when the publication is refused: the repair path exists + /// exactly for the consumer that did not receive it. + pub async fn publish_descriptor( + &mut self, + descriptor: &SessionDescriptor, + ) -> DomainResult { + descriptor.validate().map_err(DomainError::invalid)?; + { + let mut state = lock(&self.state); + if let Some(previous) = state.descriptor(descriptor.revision) + && previous != descriptor + { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + format!( + "descriptor revision {} was already published with another composition", + descriptor.revision + ), + )); + } + state + .descriptors + .insert(descriptor.revision, descriptor.clone()); + } + let topic = self.descriptor_topic.topic.clone(); + let outcome = PublicationOutcome::from_bus( + &topic, + self.bus + .publish(&topic, object(descriptor.to_json()), &[]) + .await, + ); + self.ledger.record(&outcome); + Ok(outcome) + } + + /// Publishes one committed boundary with the media handles it references. + /// + /// The coherence check runs first, so a snapshot that mixes boundaries never reaches a + /// subscriber; the sequence advances only on a publication the router accepted, so + /// "monotonic within publisherIncarnation" counts published values and not attempts. + pub async fn publish_snapshot( + &mut self, + descriptor: &SessionDescriptor, + snapshot: &CommittedSnapshot, + attachments: &[(String, flybus::Artifact)], + audio_next_sample: &BTreeMap, + ) -> DomainResult { + let named: Vec<(String, ArtifactRef)> = attachments + .iter() + .map(|(name, artifact)| (name.clone(), artifact.reference().clone())) + .collect(); + check_publication(descriptor, snapshot, &named, audio_next_sample)?; + let refs: Vec<(&str, &flybus::Artifact)> = + attachments.iter().map(|(n, a)| (n.as_str(), a)).collect(); + let topic = self.snapshot_topic.topic.clone(); + let outcome = PublicationOutcome::from_bus( + &topic, + self.bus + .publish(&topic, object(snapshot.to_json()), &refs) + .await, + ); + // The value is recorded whether or not the router admitted it, exactly as a descriptor + // revision is: the repair path exists for the consumer that did not receive it, and the + // `state-media-v1` amendment promises the exact value stays recoverable through the + // query service. The sequence advances with the value rather than with the delivery, + // so two different snapshots can never share one sequence number. + self.sequence += 1; + lock(&self.state).latest = Some(snapshot.clone()); + self.ledger.record(&outcome); + Ok(outcome) + } + + /// Offers this boundary's events to the bounded batch and publishes what it holds. + /// + /// An empty batch publishes nothing and is not an outcome. A refused batch is held for the + /// next boundary and counted; what the depth pushed out is counted too and travels with + /// the next accepted batch as `droppedBefore`. + pub async fn publish_events( + &mut self, + source_step: u64, + events: &[TaskEvent], + ) -> Option { + let dropped = self.outbox.offer(source_step, events); + self.ledger.events_dropped += dropped; + if self.outbox.is_empty() { + return None; + } + let payload = self.outbox.payload(&self.session_id, &self.epoch); + let topic = self.event_topic.topic.clone(); + let outcome = + PublicationOutcome::from_bus(&topic, self.bus.publish(&topic, payload, &[]).await); + match &outcome { + PublicationOutcome::Accepted { .. } => self.outbox.accepted(), + PublicationOutcome::RefusedByObserver { .. } => self.ledger.events_held += 1, + PublicationOutcome::Faulted { .. } => {} + } + self.ledger.record(&outcome); + Some(outcome) + } +} + +impl Publisher { + /// Publishes one checkpoint fact on the checkpoint stream. + /// + /// It goes through the same named outcomes as everything else: a durable-commit fact that + /// an observer refuses is counted and does not fail the session, because the durable + /// acknowledgment is the store's, not the subscriber's -- "bus publish acceptance and + /// delivery consumption are not durable acknowledgments" (publishing-v1 section 6). + pub async fn publish_checkpoint(&mut self, payload: Map) -> PublicationOutcome { + let topic = self.checkpoint_topic.topic.clone(); + let outcome = + PublicationOutcome::from_bus(&topic, self.bus.publish(&topic, payload, &[]).await); + self.ledger.record(&outcome); + outcome + } +} + +// ---------------------------------------------------------------------------------------------- +// The query service: the repair path + +/// The read-only service name a session answers descriptor queries on. +pub fn query_service(session_id: &Id) -> String { + format!("session.{session_id}.query") +} + +/// `Session.GetDescriptor`: one published revision, or the newest. +pub const GET_DESCRIPTOR: &str = "Session.GetDescriptor"; +/// `Session.GetSnapshot`: the latest committed snapshot, without its media handles. +pub const GET_SNAPSHOT: &str = "Session.GetSnapshot"; + +/// A running query service. +/// +/// It is the repair path of `publishing-v1` section 2 and nothing else: two read methods over +/// state the publisher already published. It takes no parameters that select a participant, it +/// mutates nothing, and it is not a controller API -- there is no method here that could +/// advance, pause, stimulate, restore or reconfigure anything. +pub struct QueryService { + task: tokio::task::JoinHandle<()>, + /// Answers this service produced and could not deliver, because the caller was gone or + /// the router refused the reply. A read that nobody received is not a read that happened, + /// and this module drops nothing silently. + undeliverable: Arc, +} + +impl QueryService { + pub async fn start( + client: flybus::Client, + session_id: &Id, + state: SharedState, + ) -> Result { + let name = query_service(session_id); + let mut service = client + .register(&name, flybus::ServiceConfig::default()) + .await?; + let worker_id = session_id.clone(); + // The bus connection id names this publisher incarnation. It has to be an `Id` to + // travel in a reply, and a connection id that is not one is a refusal here rather + // than a fallback name that two incarnations could share. + let incarnation = parse_id(&format!("query-{}", client.info().connection_id)) + .map_err(|e| flybus::BusError::new(flybus::ErrorCode::InvalidEnvelope, e))?; + let undeliverable = Arc::new(std::sync::atomic::AtomicU64::new(0)); + let undelivered = Arc::clone(&undeliverable); + let task = tokio::spawn(async move { + while let Some(request) = service.next().await { + let method = request.method().to_owned(); + let responder = request.responder(); + let parsed = + SessionRpcRequest::from_json(&Value::Object(request.payload().clone())); + drop(request); + let outcome = match parsed { + Ok(parsed) => answer(&parsed, &method, &worker_id, &incarnation, &state), + Err(e) => failure_outcome( + &DomainRequestId::from_serial(0), + &worker_id, + &incarnation, + None, + DomainError::invalid(format!("{method}: {e}")), + ), + }; + if responder.reply(outcome.to_outcome(), &[]).await.is_err() { + undelivered.fetch_add(1, std::sync::atomic::Ordering::SeqCst); + } + } + }); + Ok(QueryService { task, undeliverable }) + } + + /// How many answers this service could not deliver. + pub fn undeliverable(&self) -> u64 { + self.undeliverable.load(std::sync::atomic::Ordering::SeqCst) + } + + /// Ends the service. Dropping one does the same thing. + pub fn stop(self) { + drop(self); + } +} + +impl Drop for QueryService { + /// A dropped session leaves no task reading a service it no longer answers for. + fn drop(&mut self) { + self.task.abort(); + } +} + +fn answer( + request: &SessionRpcRequest, + method: &str, + worker_id: &Id, + incarnation: &Id, + state: &SharedState, +) -> SessionRpcOutcome { + let result = match method { + GET_DESCRIPTOR => get_descriptor(request, state), + GET_SNAPSHOT => get_snapshot(request, state), + other => Err(DomainError::before( + ErrorCode::Unsupported, + format!("{other} is not a method of the session query service"), + )), + }; + match result { + Ok(result) => success_outcome( + &request.request_id, + worker_id, + incarnation, + request.scope.clone(), + result, + ), + Err(error) => failure_outcome( + &request.request_id, + worker_id, + incarnation, + request.scope.clone(), + error, + ), + } +} + +fn get_descriptor( + request: &SessionRpcRequest, + state: &SharedState, +) -> DomainResult> { + let wanted = match request.params.get("revision") { + None | Some(Value::Null) => None, + Some(Value::String(text)) => Some( + text.parse::() + .map_err(|_| DomainError::invalid("revision is a decimal U64 string"))?, + ), + Some(_) => return Err(DomainError::invalid("revision is a decimal U64 string")), + }; + let state = lock(state); + let descriptor = match wanted { + Some(revision) => state.descriptor(revision).ok_or_else(|| { + // A revision this session never published is an answer, not an empty result. + DomainError::before( + ErrorCode::IdentityMismatch, + format!( + "descriptor revision {revision} was never published; this session has {:?}", + state.revisions() + ), + ) + })?, + None => state.newest_descriptor().ok_or_else(|| { + DomainError::before(ErrorCode::InvalidPhase, "no descriptor has been published") + })?, + }; + Ok(object(descriptor.to_json())) +} + +fn get_snapshot( + request: &SessionRpcRequest, + state: &SharedState, +) -> DomainResult> { + if !request.params.as_object().is_some_and(Map::is_empty) { + return Err(DomainError::invalid( + "Session.GetSnapshot takes no parameters", + )); + } + let state = lock(state); + let snapshot = state.latest_snapshot().ok_or_else(|| { + DomainError::before(ErrorCode::InvalidPhase, "no snapshot has been published") + })?; + Ok(object(snapshot.to_json())) +} + +// ---------------------------------------------------------------------------------------------- +// Application-owned state and cues + +/// An application's own publication channel: its schema, its topics, its delivery policy. +/// +/// The framework supplies the channel and nothing about what travels on it. "Application state +/// carries whatever the experience needs ... It is developed with its presentation, not forced +/// into a framework-wide show state/tournament schema" (publishing-v1 section 4). So the topics +/// are named by the application, the values are `TypedValue`s under the application's own +/// namespaced schema, and this module never looks inside one. There is no director here, no +/// bracket, no cast and no game. +pub struct ApplicationChannel { + bus: flybus::Client, + state_topic: TopicPolicy, + cue_topic: TopicPolicy, + ledger: Ledger, + revision: u64, +} + +impl ApplicationChannel { + /// `prefix` is the application's own address root, for example `app.counter`. + pub fn new(bus: flybus::Client, prefix: &str, cue_depth: usize) -> ApplicationChannel { + ApplicationChannel { + bus, + state_topic: TopicPolicy::latest(format!("{prefix}.state")), + cue_topic: TopicPolicy::bounded(format!("{prefix}.cues"), cue_depth), + ledger: Ledger::default(), + revision: 0, + } + } + + pub fn state_topic(&self) -> &str { + &self.state_topic.topic + } + + pub fn cue_topic(&self) -> &str { + &self.cue_topic.topic + } + + pub fn ledger(&self) -> &Ledger { + &self.ledger + } + + pub async fn declare(&self) -> DomainResult<()> { + for policy in [&self.state_topic, &self.cue_topic] { + self.bus + .declare_topic(&policy.topic, policy.delivery.retained()) + .await + .map_err(|e| { + DomainError::new( + ErrorCode::BackendFailure, + format!("declaring {}: {}", policy.topic, e.message), + MutationCertainty::None, + ) + })?; + } + Ok(()) + } + + /// Publishes application state at a committed boundary it names. + /// + /// The boundary travels with it because a consumer combines this with the framework + /// snapshot, and cross-topic ordering is not guaranteed: "a subscriber receiving an unknown + /// descriptor revision must fetch it ... or buffer a bounded number of snapshots, not infer + /// shape" (publishing-v1 section 2). A named boundary is how the two are joined. + pub async fn publish_state(&mut self, boundary: u64, value: &TypedValue) -> PublicationOutcome { + self.revision += 1; + let payload = object(json!({ + "boundary": boundary.to_string(), + "revision": self.revision.to_string(), + "state": value.to_json(), + })); + let topic = self.state_topic.topic.clone(); + let outcome = + PublicationOutcome::from_bus(&topic, self.bus.publish(&topic, payload, &[]).await); + self.ledger.record(&outcome); + outcome + } + + /// Publishes a presentation cue. Cues are presentation data: "presentation cues may be + /// immediate; simulation effects apply at declared boundaries" (section 7). Nothing here + /// reaches a worker, a controller or the world. + pub async fn publish_cue( + &mut self, + boundary: u64, + kind: &Id, + value: &TypedValue, + ) -> PublicationOutcome { + let payload = object(json!({ + "boundary": boundary.to_string(), + "kind": kind.as_str(), + "cue": value.to_json(), + })); + let topic = self.cue_topic.topic.clone(); + let outcome = + PublicationOutcome::from_bus(&topic, self.bus.publish(&topic, payload, &[]).await); + self.ledger.record(&outcome); + outcome + } +} + +// ---------------------------------------------------------------------------------------------- +// The fake multi-agent consumer + +/// One agent's committed values as a consumer reads them. +/// +/// A presentation consumer is a *multi-agent* consumer: one snapshot carries the whole +/// composition, so there is no per-fly stream to join and no "current fly" to be stale. +#[derive(Clone, Debug, PartialEq)] +pub struct AgentView { + pub agent_id: Id, + /// The index the descriptor says this agent's rates and geometry belong to. A consumer + /// that maps anything spatial compares this, not `neuronCount`. + pub index_digest: Digest, + pub brain_ticks: u64, + pub rates: Vec<(Id, f64)>, + /// The decision of the transition that *ended* at this boundary, null only at boundary 0. + pub decision: Option, + pub controls: Option, +} + +/// One committed boundary as a consumer reads it, with the media handles it still owns. +pub struct SnapshotView { + pub boundary: u64, + pub descriptor_revision: u64, + pub sequence: u64, + /// How many undelivered snapshots the router coalesced away before this one. + pub replaced: u64, + pub agents: Vec, + pub views: Vec, + pub audio: Vec, + /// The extracted handles, held after the message is dropped. + pub artifacts: BTreeMap, +} + +impl SnapshotView { + pub fn agent(&self, agent_id: &Id) -> Option<&AgentView> { + self.agents.iter().find(|a| a.agent_id == *agent_id) + } +} + +/// One bounded event batch as a consumer reads it. +#[derive(Clone, Debug, PartialEq)] +pub struct EventBatchView { + pub epoch: Id, + /// How many events the publisher's bounded batch dropped before this one. A count, never + /// a gap the consumer has to infer. + pub dropped_before: u64, + pub event_ids: Vec, +} + +/// What one poll of the event stream produced. +#[derive(Clone, Debug, PartialEq)] +pub enum ConsumerEvents { + Batch(EventBatchView), + /// A batch this consumer could not read. Distinct from the end of the stream. + Unreadable { detail: String }, +} + +/// What one poll of a consumer produced. +#[derive(Clone, Debug, PartialEq)] +pub enum ConsumerOutcome { + /// A descriptor this consumer now holds, and the agents it describes. + Composition { revision: u64, agents: Vec }, + /// A snapshot whose descriptor revision this consumer holds and which agrees with it. + Read { boundary: u64, agents: Vec }, + /// A snapshot naming a descriptor revision this consumer has never seen. Nothing is + /// inferred from it: the consumer repairs through the query service and reads it again. + UnknownRevision { revision: u64 }, + /// The composition changed under an agent this consumer had already mapped: the same + /// neuron count, another index. Visible, named, and never silently remapped. + IndexChanged { + agent_id: Id, + from: Digest, + to: Digest, + }, + /// The snapshot does not agree with the descriptor it names, or its media does not belong + /// to its boundary. + Incoherent { detail: String }, +} + +/// A presentation-side consumer of one session, over the same bus. +/// +/// It is a regular bus subscriber (publishing-v1 section 5): latest subscriptions with finite +/// credits on the observation topics, a bounded subscription on events, and one read-only RPC +/// for repair. It is deliberately *not* a gateway: it resolves no artifact into a browser +/// transport, encodes nothing and holds no private owner token on anyone's behalf. +pub struct PresentationConsumer { + bus: flybus::Client, + query: String, + descriptors: flybus::Subscription, + snapshots: flybus::Subscription, + events: flybus::Subscription, + held: Vec, + cached: BTreeMap, + /// The index digest this consumer has mapped geometry against, per agent. + mapped: BTreeMap, + /// Where each audio stream's last accepted chunk ended. + audio_end: BTreeMap, + last: Option, + repairs: u64, + serial: u64, + coalesced: u64, +} + +impl PresentationConsumer { + pub async fn attach( + bus: flybus::Client, + session_id: &Id, + topics: &crate::coordinator::Topics, + ) -> Result { + let descriptors = bus + .subscribe(&topics.descriptor, Delivery::LatestValue.subscription()) + .await?; + let snapshots = bus + .subscribe(&topics.snapshots, Delivery::LatestValue.subscription()) + .await?; + let events = bus + .subscribe( + &topics.events, + Delivery::BoundedBatch { + depth: EVENT_BATCH_DEPTH, + } + .subscription(), + ) + .await?; + Ok(PresentationConsumer { + bus, + query: query_service(session_id), + descriptors, + snapshots, + events, + held: Vec::new(), + cached: BTreeMap::new(), + mapped: BTreeMap::new(), + audio_end: BTreeMap::new(), + last: None, + repairs: 0, + serial: 0, + coalesced: 0, + }) + } + + /// How many descriptor revisions this consumer holds. + pub fn revisions(&self) -> Vec { + self.cached.keys().copied().collect() + } + + pub fn descriptor(&self, revision: u64) -> Option<&SessionDescriptor> { + self.cached.get(&revision) + } + + /// How many times this consumer had to ask the query service for a descriptor. + pub fn repairs(&self) -> u64 { + self.repairs + } + + /// How many snapshots the router coalesced away in this consumer's own queue. + pub fn coalesced(&self) -> u64 { + self.coalesced + } + + pub fn last(&self) -> Option<&SnapshotView> { + self.last.as_ref() + } + + /// Takes the next descriptor from the retained-latest topic. + /// + /// A composition that moves an agent this consumer has already mapped geometry against is + /// reported here and *not* cached: remapping silently is the one thing a consumer holding + /// a spatial mapping must not do. (publishing-v1 section 3) + pub async fn take_descriptor(&mut self) -> Option { + let message = self.descriptors.next().await?; + let descriptor = + match SessionDescriptor::from_json(&Value::Object(message.payload().clone())) { + Ok(descriptor) => descriptor, + Err(e) => { + return Some(ConsumerOutcome::Incoherent { + detail: e.to_string(), + }); + } + }; + drop(message); + for agent in &descriptor.agents { + if let Some(mapped) = self.mapped.get(&agent.agent_id) + && *mapped != agent.index_digest + { + return Some(ConsumerOutcome::IndexChanged { + agent_id: agent.agent_id.clone(), + from: mapped.clone(), + to: agent.index_digest.clone(), + }); + } + } + let revision = descriptor.revision; + let agents = descriptor + .agents + .iter() + .map(|a| a.agent_id.clone()) + .collect(); + self.cached.insert(revision, descriptor); + Some(ConsumerOutcome::Composition { revision, agents }) + } + + /// Asks the query service for a revision this consumer does not hold. + /// + /// This is the repair path, and it is an ordinary RPC on the same bus. A revision the + /// session never published comes back as a named error rather than an empty answer. + pub async fn repair(&mut self, revision: u64) -> DomainResult { + self.serial += 1; + self.repairs += 1; + let request = SessionRpcRequest { + request_id: DomainRequestId::from_serial(self.serial), + scope: None, + params: json!({ "revision": revision.to_string() }), + }; + let mut pending = self + .bus + .call( + &self.query, + None, + GET_DESCRIPTOR, + object(request.to_json()), + &[], + ) + .await + .map_err(|e| { + DomainError::new( + ErrorCode::BackendFailure, + e.message, + MutationCertainty::None, + ) + })?; + let result = pending.result().await.map_err(|e| { + DomainError::new( + ErrorCode::BackendFailure, + e.message, + MutationCertainty::None, + ) + })?; + let outcome = SessionRpcOutcome::from_json(&Value::Object(result.outcome().clone())) + .map_err(DomainError::invalid)?; + drop(result); + let value = outcome_result(&outcome)?; + let descriptor = SessionDescriptor::from_json(value).map_err(DomainError::invalid)?; + if descriptor.revision != revision { + return Err(DomainError::before( + ErrorCode::IdentityMismatch, + "the query service answered with another revision", + )); + } + self.cached.insert(revision, descriptor); + Ok(revision) + } + + /// Takes the next committed snapshot and reads it against the descriptor it names. + pub async fn take_snapshot(&mut self) -> Option { + let message = self.snapshots.next().await?; + self.coalesced += message.replaced(); + Some(self.read(message)) + } + + /// Takes a snapshot without reading or releasing it, which is what a viewer that stopped + /// rendering does. Its credits run out; nothing else in the session notices. + pub async fn hold_snapshot(&mut self) -> bool { + match self.snapshots.next().await { + Some(message) => { + self.coalesced += message.replaced(); + self.held.push(message); + true + } + None => false, + } + } + + /// Holds a snapshot if one is queued right now, and answers `false` if none is. + /// + /// The bounded form: a test that means "take whatever is there" must not be able to + /// block on a stream that is deliberately not producing. + pub fn try_hold_snapshot(&mut self) -> bool { + match self.snapshots.try_next() { + Some(message) => { + self.coalesced += message.replaced(); + self.held.push(message); + true + } + None => false, + } + } + + /// Reads a snapshot if one is queued right now. + pub fn try_take_snapshot(&mut self) -> Option { + let message = self.snapshots.try_next()?; + self.coalesced += message.replaced(); + Some(self.read(message)) + } + + /// Releases everything this consumer was holding, returning its credits. + pub fn release(&mut self) { + self.held.clear(); + } + + pub fn held(&self) -> usize { + self.held.len() + } + + /// Takes the next bounded event batch. + /// + /// `None` is the end of the stream and nothing else. A batch this consumer cannot read + /// comes back as [`ConsumerEvents::Unreadable`], because "there are no more events" and + /// "that one made no sense" are different facts and one value cannot carry both: a + /// consumer that saw the second as the first would stop reading a live stream. + pub async fn take_events(&mut self) -> Option { + let message = self.events.next().await?; + let payload = message.payload().clone(); + drop(message); + let unreadable = |what: &str| { + Some(ConsumerEvents::Unreadable { + detail: format!("an event batch has no readable {what}"), + }) + }; + let Some(epoch) = payload.get("epoch").and_then(Value::as_str).map(id) else { + return unreadable("epoch"); + }; + let Some(dropped_before) = payload + .get("droppedBefore") + .and_then(Value::as_str) + .and_then(|t| t.parse::().ok()) + else { + return unreadable("droppedBefore"); + }; + let Some(events) = payload.get("events").and_then(Value::as_array) else { + return unreadable("events array"); + }; + let event_ids = events + .iter() + .filter_map(|e| e.get("id").and_then(Value::as_str).map(str::to_owned)) + .collect(); + Some(ConsumerEvents::Batch(EventBatchView { + epoch, + dropped_before, + event_ids, + })) + } + + /// Records that this consumer has mapped geometry against the agents of `revision`. + /// + /// A consumer that never calls this never claims a mapping, and a later index change is + /// simply a new descriptor. One that does claim it gets [`ConsumerOutcome::IndexChanged`] + /// when the composition moves under it. + pub fn map_geometry(&mut self, revision: u64) -> DomainResult<()> { + let descriptor = self.cached.get(&revision).ok_or_else(|| { + DomainError::before( + ErrorCode::IdentityMismatch, + format!("revision {revision} is not held by this consumer"), + ) + })?; + for agent in &descriptor.agents { + self.mapped + .insert(agent.agent_id.clone(), agent.index_digest.clone()); + } + Ok(()) + } + + pub fn mapped_index(&self, agent_id: &Id) -> Option<&Digest> { + self.mapped.get(agent_id) + } + + fn read(&mut self, message: flybus::Message) -> ConsumerOutcome { + let sequence = message.topic_sequence(); + let replaced = message.replaced(); + let snapshot = match CommittedSnapshot::from_json(&Value::Object(message.payload().clone())) + { + Ok(snapshot) => snapshot, + Err(e) => { + return ConsumerOutcome::Incoherent { + detail: e.to_string(), + }; + } + }; + let Some(descriptor) = self.cached.get(&snapshot.descriptor_revision).cloned() else { + // Not an error and not a guess: the consumer buffers nothing and infers nothing, + // it repairs. (publishing-v1 section 2) + return ConsumerOutcome::UnknownRevision { + revision: snapshot.descriptor_revision, + }; + }; + for agent in &descriptor.agents { + if let Some(mapped) = self.mapped.get(&agent.agent_id) + && *mapped != agent.index_digest + { + return ConsumerOutcome::IndexChanged { + agent_id: agent.agent_id.clone(), + from: mapped.clone(), + to: agent.index_digest.clone(), + }; + } + } + let names: Vec = message.attachment_names().map(str::to_owned).collect(); + let mut handles = Vec::new(); + for name in &names { + match message.artifact(name) { + Ok(artifact) => handles.push((name.clone(), artifact.reference().clone())), + Err(e) => { + return ConsumerOutcome::Incoherent { + detail: format!("attachment {name}: {}", e.message), + }; + } + } + } + if let Err(e) = check_received(&descriptor, &snapshot, &self.audio_end, &handles) { + return ConsumerOutcome::Incoherent { + detail: e.to_string(), + }; + } + // The renderer keeps its handles after the delivery is gone; the bytes stay alive + // because the extracted handle owns them. (publishing-v1 section 5) + let mut artifacts = BTreeMap::new(); + for name in &names { + match message.artifact(name) { + Ok(artifact) => { + artifacts.insert(name.clone(), artifact); + } + Err(e) => { + return ConsumerOutcome::Incoherent { + detail: format!("attachment {name}: {}", e.message), + }; + } + } + } + drop(message); + for chunk in &snapshot.audio { + self.audio_end.insert( + chunk.stream_id.clone(), + chunk.first_sample + chunk.sample_frames, + ); + } + let agents: Vec = snapshot + .agents + .iter() + .map(|agent| { + let declared = descriptor + .agents + .iter() + .find(|a| a.agent_id == agent.agent_id) + .expect("validate_against checked the agent set"); + AgentView { + agent_id: agent.agent_id.clone(), + index_digest: declared.index_digest.clone(), + brain_ticks: agent.telemetry.brain_ticks, + rates: agent + .telemetry + .rates + .iter() + .map(|r| (r.role_id.clone(), r.hz)) + .collect(), + decision: agent.selected_decision.clone(), + controls: agent.applied_controls.clone(), + } + }) + .collect(); + let outcome = ConsumerOutcome::Read { + boundary: snapshot.scope.step, + agents: agents.iter().map(|a| a.agent_id.clone()).collect(), + }; + self.last = Some(SnapshotView { + boundary: snapshot.scope.step, + descriptor_revision: snapshot.descriptor_revision, + sequence, + replaced, + agents, + views: snapshot.views.clone(), + audio: snapshot.audio.clone(), + artifacts, + }); + outcome + } +} + +/// The receiving half of the coherence rule. +/// +/// A consumer does not trust that the publisher checked: it checks the same statement from the +/// other side, against the audio position it last accepted rather than the one the publisher +/// holds. That is what catches a chunk from an earlier transition arriving under a later +/// boundary, which is the one shape of "old media" a publisher-side check cannot see. +pub fn check_received( + descriptor: &SessionDescriptor, + snapshot: &CommittedSnapshot, + audio_end: &BTreeMap, + attachments: &[(String, ArtifactRef)], +) -> DomainResult<()> { + snapshot + .validate_against(descriptor) + .map_err(|e| coherence(format!("snapshot and descriptor disagree: {e}")))?; + check_views(descriptor, snapshot)?; + check_attachments(snapshot, attachments)?; + for chunk in &snapshot.audio { + match audio_end.get(&chunk.stream_id) { + // A stream this consumer has not heard yet, or one that says it skipped: both are + // declared states, not assumptions. + None => {} + Some(_) if chunk.discontinuity => {} + // Forward is the latest subscription's own contract: a consumer that fell behind + // was told so by `replaced`, and the boundaries in between are values it chose + // not to receive. Backwards or overlapping is old media under a new boundary, + // which no delivery policy explains. + Some(end) if chunk.first_sample >= *end => {} + Some(end) => { + return Err(coherence(format!( + "audio stream {} starts at {} and the last chunk this consumer read ended at {end}", + chunk.stream_id, chunk.first_sample + ))); + } + } + } + Ok(()) +} diff --git a/services/flysim/crates/fly-session/src/state.rs b/services/flysim/crates/fly-session/src/state.rs index aa42d3c..220310f 100644 --- a/services/flysim/crates/fly-session/src/state.rs +++ b/services/flysim/crates/fly-session/src/state.rs @@ -599,6 +599,10 @@ pub struct AgentEntry { pub agent_id: Id, pub profile_digest: Digest, pub dataset_digest: Digest, + /// The index the agent attested to at `Agent.Initialize`. It is part of the agent's + /// compatibility identity, so a replacement that built another graph cannot install this + /// payload -- the graph identity does not cross the recovery. + pub index_digest: Digest, pub model_version: String, pub plasticity_version: String, pub seed: i32, @@ -613,6 +617,7 @@ impl AgentEntry { "agentId": self.agent_id.as_str(), "profileDigest": self.profile_digest.as_str(), "datasetDigest": self.dataset_digest.as_str(), + "indexDigest": self.index_digest.as_str(), "modelVersion": self.model_version.as_str(), "plasticityVersion": self.plasticity_version.as_str(), "seed": self.seed, @@ -646,6 +651,7 @@ impl AgentEntry { agent_id: parse_id(&text("agentId")?)?, profile_digest: text("profileDigest")?, dataset_digest: text("datasetDigest")?, + index_digest: text("indexDigest")?, model_version: text("modelVersion")?, plasticity_version: text("plasticityVersion")?, seed, diff --git a/services/flysim/crates/fly-session/src/types.rs b/services/flysim/crates/fly-session/src/types.rs index 07041c2..8648308 100644 --- a/services/flysim/crates/fly-session/src/types.rs +++ b/services/flysim/crates/fly-session/src/types.rs @@ -32,9 +32,12 @@ pub use fly_session_types::schema::contract_digest; pub use fly_session_types::trace::{ TraceAgent, TraceBehaviour, TraceObservation, TraceOperational, TraceRequest, TransitionTrace, }; +pub use fly_session_types::publishing::{ + AgentDescriptor, CommittedSnapshot, SessionDescriptor, SnapshotAgent, +}; pub use fly_session_types::workers::{ AcknowledgeParams, AcknowledgeResult, AdvanceParams, AgentCommitResult, AgentInitializeParams, - AgentInitializeResult, AgentTelemetry, AssetRef, AxisRange, AxisSchema, AxisValue, ButtonState, + AgentGraph, AgentInitializeResult, AgentTelemetry, AssetRef, AxisRange, AxisSchema, AxisValue, ButtonState, CommitParams, ControllerSchema, Determinism, EnvironmentDescriptor, EnvironmentInitializeParams, EnvironmentInitializeResult, EpisodeRequest, HelloParams, HelloResult, LearningTelemetry, MAX_ACKNOWLEDGE, MAX_AGENTS, MAX_PORTS, MAX_RATE_ROLES, diff --git a/services/flysim/crates/fly-session/tests/publishing.rs b/services/flysim/crates/fly-session/tests/publishing.rs new file mode 100644 index 0000000..b5e7b93 --- /dev/null +++ b/services/flysim/crates/fly-session/tests/publishing.rs @@ -0,0 +1,1384 @@ +//! PUBLISH-01 acceptance: committed snapshots, observer isolation and the repair path. +//! +//! Every test here is one of the slice's acceptance bullets, or one sentence of +//! `publishing-v1` the session now enforces. The boundary itself is `fly_session::publish`: +//! the same bus, named delivery policies, named outcomes, and a fake multi-agent consumer that +//! is an ordinary subscriber. No public v2 wire schema is exercised, because none is approved. + +mod common; + +use std::collections::BTreeMap; +use std::time::Duration; + +use common::{Fixture, default_fixture, fixture, fly_a, fly_b, mode_fixture, within}; +use fly_session::coordinator::{DESCRIPTOR_REVISION, Injections}; +use fly_session::harness::{AgentSpec, ExecutionMode, HarnessConfig, Via}; +use fly_session::publish::{ + ApplicationChannel, ConsumerEvents, ConsumerOutcome, Delivery, EVENT_BATCH_DEPTH, EventOutbox, + GET_SNAPSHOT, PresentationConsumer, PublicationOutcome, TopicPolicy, check_publication, + query_service, +}; +use fly_session::types::*; +use serde_json::json; + +both_transports!( + a_consumer_that_disconnects_neither_advances_nor_stalls_the_world, + a_consumer_that_stops_consuming_costs_only_its_own_credits, + a_bounded_observer_refusal_is_named_and_never_fails_the_epoch, + every_published_snapshot_carries_its_own_boundarys_media, + a_published_frame_from_another_boundary_is_refused, + a_published_handle_that_is_not_the_referenced_artifact_is_refused, + an_unheld_descriptor_revision_is_repaired_rather_than_inferred, + a_revision_that_was_never_published_is_a_named_answer, + a_changed_index_digest_is_named_rather_than_remapped, + boundary_zero_publishes_no_decision_and_no_controls, + a_committed_action_is_the_transition_that_just_ended, + one_snapshot_carries_every_agent_in_the_composition, + application_state_and_cues_are_the_applications_own, + a_refused_event_batch_is_held_and_counted_not_lost, + the_query_service_answers_reads_and_nothing_else, + a_refused_snapshot_is_still_what_the_repair_path_answers, + a_replacement_that_built_another_index_cannot_install_the_checkpoint, + a_malformed_event_batch_is_unreadable_and_not_the_end_of_the_stream, + the_published_descriptor_is_what_the_workers_attested_to, + a_stimulus_kind_the_descriptor_does_not_declare_is_refused, + a_restored_boundary_publishes_a_new_revision_and_no_transition, +); + +all_modes!( + the_publication_boundary_holds_in_every_execution_mode, + a_stalled_observer_never_moves_the_world_in_any_execution_mode, +); + +const STEPS: u64 = 4; + +/// Polls until `ok` holds, so a test never asserts something that has not happened yet. +async fn until(what: &str, mut ok: impl FnMut() -> bool) { + let deadline = std::time::Instant::now() + Duration::from_secs(10); + while !ok() { + assert!( + std::time::Instant::now() < deadline, + "{what}: never happened" + ); + tokio::time::sleep(Duration::from_millis(5)).await; + } +} + +/// A started session at Ready(0), with boundary 0 already published. +async fn started(via: Via) -> Fixture { + let mut f = default_fixture(via).await; + f.harness + .coordinator + .bootstrap() + .await + .expect("the session bootstraps"); + f +} + +async fn started_in_mode(mode: ExecutionMode) -> Fixture { + let mut f = mode_fixture(mode, HarnessConfig::default()).await; + f.harness + .coordinator + .bootstrap() + .await + .expect("the session bootstraps"); + f +} + +/// Reads snapshots until the consumer reports the boundary it is waiting for. +async fn read_until(consumer: &mut PresentationConsumer, boundary: u64) -> u64 { + loop { + match within("a snapshot", consumer.take_snapshot()).await { + Some(ConsumerOutcome::Read { boundary: at, .. }) if at >= boundary => return at, + Some(ConsumerOutcome::Read { .. }) => {} + other => panic!("expected a readable snapshot, got {other:?}"), + } + } +} + +// ------------------------------------------------------------------------------------------ +// "A browser consumer disconnecting, or applying backpressure, never advances the world and +// never stalls it." + +/// A consumer that goes away mid-run. The world takes exactly the steps it was asked for. +async fn a_consumer_that_disconnects_neither_advances_nor_stalls_the_world(via: Via) { + let mut f = started(via).await; + let mut consumer = f.harness.consumer().await.expect("a consumer attaches"); + assert!(matches!( + within("the descriptor", consumer.take_descriptor()).await, + Some(ConsumerOutcome::Composition { revision, .. }) if revision == DESCRIPTOR_REVISION + )); + f.harness.coordinator.run(1).await.expect("one transition"); + read_until(&mut consumer, 1).await; + + // The browser goes away. Nothing in the session is told, and nothing waits for it. + drop(consumer); + + let started_at = std::time::Instant::now(); + let before = f.harness.coordinator.stats(); + f.harness + .coordinator + .run(STEPS) + .await + .expect("the world carries on"); + let after = f.harness.coordinator.stats(); + assert_eq!( + after.advances - before.advances, + STEPS, + "a departed observer neither added a world step nor removed one" + ); + assert_eq!(after.publications - before.publications, STEPS); + assert!( + !f.harness.coordinator.is_fenced(), + "a departed observer never fences an epoch" + ); + assert!( + started_at.elapsed() < Duration::from_secs(10), + "a departed observer never stalls the world" + ); + assert_eq!(f.harness.coordinator.ledger().refusals(), 0); + f.shutdown().await; +} + +/// A viewer that holds its deliveries and stops rendering. Its own credits run out; the +/// world's boundaries do not. +async fn a_consumer_that_stops_consuming_costs_only_its_own_credits(via: Via) { + let mut f = started(via).await; + let mut slow = f + .harness + .consumer() + .await + .expect("a slow consumer attaches"); + let mut healthy = f + .harness + .consumer() + .await + .expect("a healthy consumer attaches"); + + // The slow one takes its deliveries and never releases them: a viewer holding output. + until("the slow viewer has a delivery", || { + slow.try_hold_snapshot() + }) + .await; + while slow.try_hold_snapshot() && slow.held() < 4 {} + let held = slow.held(); + assert!(held > 0); + + f.harness + .coordinator + .run(STEPS) + .await + .expect("the world carries on"); + assert_eq!(f.harness.coordinator.stats().advances, STEPS); + assert_eq!( + f.harness.coordinator.ledger().refusals(), + 0, + "a latest subscriber cannot refuse" + ); + assert_eq!( + slow.held(), + held, + "the slow viewer took nothing more once its credits were gone" + ); + + // The healthy one is unaffected and reads the newest boundary. + assert!(matches!( + within("the descriptor", healthy.take_descriptor()).await, + Some(ConsumerOutcome::Composition { .. }) + )); + let at = read_until(&mut healthy, STEPS).await; + assert_eq!(at, STEPS); + assert_eq!( + healthy.last().expect("a snapshot").boundary, + STEPS, + "the healthy consumer reads the newest boundary while the slow one holds" + ); + f.shutdown().await; +} + +/// A bounded subscriber whose queue is full refuses a publication -- `bus-v1` section 6 says +/// it may. The refusal is a named outcome on a named topic, the world takes every step it was +/// asked for, the epoch is not fenced, and when the offender goes away the next boundary +/// reaches the well-behaved consumer. +async fn a_bounded_observer_refusal_is_named_and_never_fails_the_epoch(via: Via) { + let mut f = started(via).await; + let healthy_first = f + .harness + .consumer() + .await + .expect("a healthy consumer attaches"); + let offender_client = f.harness.observer().await.expect("an observer client"); + let snapshots = f.harness.coordinator.topics().snapshots.clone(); + let mut offender = offender_client + .subscribe( + &snapshots, + flybus::SubscriptionConfig::bounded().queued(1).in_flight(1), + ) + .await + .expect("a bounded subscription"); + + let before = f.harness.coordinator.stats().advances; + f.harness + .coordinator + .run(STEPS) + .await + .expect("an observer never fails the epoch"); + assert_eq!( + f.harness.coordinator.stats().advances - before, + STEPS, + "a refused publication never costs the world a step" + ); + assert!(!f.harness.coordinator.is_fenced()); + let counters = f.harness.coordinator.ledger().counters(&snapshots); + assert!( + counters.refused > 0, + "the refusal is counted on its own topic: {counters:?}" + ); + assert!( + matches!( + f.harness.coordinator.ledger().last(), + Some(PublicationOutcome::RefusedByObserver { .. }) + ), + "the last outcome names the refusal rather than defaulting to success" + ); + assert_eq!( + counters.faulted, 0, + "an observer's refusal is never the session's fault" + ); + + // The offender leaves. The next boundary is published and the healthy consumer reads it. + drop(offender.try_next()); + drop(offender); + offender_client.close().await; + drop(healthy_first); + let mut healthy = f + .harness + .consumer() + .await + .expect("a healthy consumer attaches"); + f.harness + .coordinator + .run(2) + .await + .expect("the world carries on"); + assert!(matches!( + within("the descriptor", healthy.take_descriptor()).await, + Some(ConsumerOutcome::Composition { .. }) + )); + let at = read_until(&mut healthy, STEPS + 1).await; + assert!( + at > STEPS, + "a boundary published after the offender left reaches a consumer" + ); + f.shutdown().await; +} + +// ------------------------------------------------------------------------------------------ +// "Future agent state is never mixed with old media." + +/// Every snapshot's media belongs to the boundary its agent state belongs to: the frame is the +/// one this boundary's declared delay requires, the handle is the artifact the payload names, +/// the audio continues where the last chunk ended, and the agents are committed at it. +async fn every_published_snapshot_carries_its_own_boundarys_media(via: Via) { + let mut f = started(via).await; + let mut consumer = f.harness.consumer().await.expect("a consumer attaches"); + assert!(matches!( + within("the descriptor", consumer.take_descriptor()).await, + Some(ConsumerOutcome::Composition { .. }) + )); + let mut seen: Vec<(u64, String)> = Vec::new(); + let mut ticks: BTreeMap = BTreeMap::new(); + for step in 1..=STEPS { + f.harness.coordinator.run(1).await.expect("a transition"); + let at = read_until(&mut consumer, step).await; + let view = consumer.last().expect("a snapshot"); + assert_eq!(view.boundary, at); + for reference in &view.views { + // The counter arena declares no render delay, so the frame of boundary k is + // produced at k. The consumer checked this itself before it got here. + assert_eq!(reference.produced_step, at, "the frame is this boundary's"); + let handle = view + .artifacts + .get(&fly_session::media::view_attachment(&reference.view_id)) + .expect("the referenced frame has its handle"); + assert_eq!( + handle.reference(), + &reference.pixels, + "the handle is that artifact" + ); + seen.push((at, reference.pixels.artifact_id.clone())); + } + for agent in &view.agents { + let previous = ticks.insert(agent.agent_id.clone(), agent.brain_ticks); + match previous { + None => assert!(agent.brain_ticks > 0, "the agent has run by boundary {at}"), + // Telemetry is the state of the transition that ended here. Republishing the + // previous boundary's numbers -- or `Agent.Initialize`'s warm-up numbers -- + // would be old agent state labelled as this boundary, which is the same + // mislabelling as old media. + Some(before) => assert!( + agent.brain_ticks > before, + "the telemetry of {} advanced into boundary {at}: {before} -> {}", + agent.agent_id, + agent.brain_ticks + ), + } + } + } + let mut ids: Vec<&String> = seen.iter().map(|(_, id)| id).collect(); + ids.sort(); + ids.dedup(); + assert_eq!( + ids.len(), + seen.len(), + "no boundary republished another boundary's frame: {seen:?}" + ); + f.shutdown().await; +} + +/// The injection of new agent state with the previous boundary's frame. The publication is +/// refused before it reaches a subscriber, and the consumer never sees a mixed boundary. +async fn a_published_frame_from_another_boundary_is_refused(via: Via) { + let mut f = started(via).await; + let mut consumer = f.harness.consumer().await.expect("a consumer attaches"); + f.harness + .coordinator + .run(1) + .await + .expect("one clean transition"); + f.harness.coordinator.injections = Injections { + at_step: 1, + stale_published_view: true, + ..Injections::default() + }; + let failure = f + .harness + .coordinator + .run(1) + .await + .expect_err("a snapshot that mixes boundaries is refused"); + assert_eq!(failure.error.code, ErrorCode::BufferInvalid, "{failure:?}"); + assert!( + failure.error.message.contains("produced at"), + "the refusal names the boundary the frame came from: {}", + failure.error.message + ); + assert_eq!( + failure.error.mutation, + MutationCertainty::None, + "nothing was published, so nothing downstream saw it" + ); + + // Whatever the consumer read, none of it is the mixed boundary. + assert!(matches!( + within("the descriptor", consumer.take_descriptor()).await, + Some(ConsumerOutcome::Composition { .. }) + )); + until("a snapshot of the clean boundary", || { + matches!( + consumer.try_take_snapshot(), + Some(ConsumerOutcome::Read { boundary: 1, .. }) + ) + }) + .await; + assert_eq!(consumer.last().expect("a snapshot").boundary, 1); + while let Some(outcome) = consumer.try_take_snapshot() { + match outcome { + ConsumerOutcome::Read { boundary, .. } => { + assert!(boundary <= 1, "the mixed boundary was never published"); + } + other => panic!("the consumer saw {other:?}"), + } + } + f.shutdown().await; +} + +/// The same attachment names and the same bytes, another object. Only the artifact identity +/// sees it, and the publication is refused. +async fn a_published_handle_that_is_not_the_referenced_artifact_is_refused(via: Via) { + let mut f = started(via).await; + f.harness.coordinator.injections = Injections { + at_step: 0, + substituted_published_handle: true, + ..Injections::default() + }; + let failure = f + .harness + .coordinator + .run(1) + .await + .expect_err("a handle that is not the referenced artifact is refused"); + assert_eq!(failure.error.code, ErrorCode::BufferInvalid, "{failure:?}"); + assert!( + failure.error.message.contains("generation"), + "the refusal names the object it got: {}", + failure.error.message + ); + f.shutdown().await; +} + +// ------------------------------------------------------------------------------------------ +// "A descriptor or index mismatch is visible rather than silently tolerated." + +/// A consumer that meets a revision it does not hold repairs through the query service. It +/// infers nothing from the snapshot in the meantime. +async fn an_unheld_descriptor_revision_is_repaired_rather_than_inferred(via: Via) { + let mut f = started(via).await; + let mut consumer = f.harness.consumer().await.expect("a consumer attaches"); + f.harness.coordinator.run(1).await.expect("a transition"); + + // The snapshot arrives first: cross-topic ordering is not guaranteed, and this consumer + // has deliberately not read its descriptor topic. + let outcome = within("a snapshot", consumer.take_snapshot()).await; + assert_eq!( + outcome, + Some(ConsumerOutcome::UnknownRevision { + revision: DESCRIPTOR_REVISION + }), + "an unknown revision is reported, not guessed" + ); + assert!( + consumer.last().is_none(), + "nothing was read out of a snapshot it cannot shape" + ); + + let repaired = consumer + .repair(DESCRIPTOR_REVISION) + .await + .expect("the repair path answers"); + assert_eq!(repaired, DESCRIPTOR_REVISION); + assert_eq!(consumer.repairs(), 1); + f.harness.coordinator.run(1).await.expect("a transition"); + let at = read_until(&mut consumer, 1).await; + assert!(at >= 1, "with the descriptor in hand the same stream reads"); + f.shutdown().await; +} + +/// A revision this session never published is an answer, not an empty result. +async fn a_revision_that_was_never_published_is_a_named_answer(via: Via) { + let f = started(via).await; + let mut consumer = f.harness.consumer().await.expect("a consumer attaches"); + let error = consumer + .repair(99) + .await + .expect_err("revision 99 was never published"); + assert_eq!(error.code, ErrorCode::IdentityMismatch, "{error:?}"); + assert!(error.message.contains("99"), "{}", error.message); + assert_eq!( + error.mutation, + MutationCertainty::None, + "a read never mutates" + ); + f.shutdown().await; +} + +/// A graph identity does not cross a recovery. +/// +/// The index a fly attested to at `Agent.Initialize` is part of its compatibility identity, so +/// a replacement that built another graph -- the same dataset, the same neuron count, another +/// index -- cannot install a checkpoint taken under the first one, and the refusal names what +/// this worker is rather than only that two digests differ. The identical replacement still +/// installs, so the check refuses the case it is about and nothing else. +async fn a_replacement_that_built_another_index_cannot_install_the_checkpoint(via: Via) { + for (variant, refused) in [(0u64, false), (1, true)] { + let mut f = started(via).await; + let checkpoint = id("ck-graph"); + f.harness.coordinator.run(1).await.expect("one transition"); + within("checkpoint", f.harness.coordinator.checkpoint(&checkpoint)) + .await + .expect("a committed checkpoint"); + + f.harness.set_agent_graph(&fly_a(), variant); + f.harness.kill(&fly_a()).await; + f.harness + .coordinator + .step() + .await + .expect_err("a dead participant fails the epoch"); + within("replace", f.harness.replace_all_participants()) + .await + .expect("replacements"); + let outcome = within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint), &id("e2")), + ) + .await; + match (refused, outcome) { + (false, Ok(_)) => {} + (false, Err(e)) => panic!("the identical replacement must still install: {e}"), + (true, Ok(_)) => panic!("a fly that built another index installed the checkpoint"), + (true, Err(failure)) => { + assert_eq!(failure.error.code, ErrorCode::IncompatibleState, "{failure:?}"); + let built = fly_session::agent::synthetic_graph(&fly_a(), 1); + assert!( + failure.error.message.contains(&built.index_digest), + "the refusal names the index this worker built: {}", + failure.error.message + ); + assert!( + f.harness.coordinator.is_fenced(), + "and the group stays fenced" + ); + } + } + f.shutdown().await; + } +} + +/// The same neuron count, another index. A consumer that has mapped geometry is told, and the +/// new composition is not quietly cached over the one it mapped. +/// +/// The two descriptors here come from two real sessions, and deliberately not from a restore: +/// since the index is part of an agent's compatibility identity, a restore whose replacement +/// built another graph is now refused before it can install, which +/// `a_replacement_that_built_another_index_cannot_install_the_checkpoint` proves. A published +/// index therefore changes between compositions rather than across a recovery, and this is that +/// case. +async fn a_changed_index_digest_is_named_rather_than_remapped(via: Via) { + // Two real compositions that differ only in the graph one fly built. + let first = started(via).await; + let mut second = fixture( + via, + HarnessConfig { + agents: vec![ + AgentSpec { + graph_variant: 1, + ..AgentSpec::new("fly-a", "p1", 7) + }, + AgentSpec { + graph_variant: 0, + ..AgentSpec::new("fly-b", "p2", 11) + }, + ], + ..HarnessConfig::default() + }, + ) + .await; + second + .harness + .coordinator + .bootstrap() + .await + .expect("the second session bootstraps"); + let revised = { + let mut revised = second + .harness + .coordinator + .session_descriptor() + .expect("a published descriptor") + .clone(); + revised.revision = DESCRIPTOR_REVISION + 1; + revised + }; + let original = first + .harness + .coordinator + .session_descriptor() + .expect("a published descriptor") + .clone(); + let moved = original + .agents + .iter() + .find(|a| a.agent_id == fly_a()) + .expect("fly-a"); + let arrived = revised + .agents + .iter() + .find(|a| a.agent_id == fly_a()) + .expect("fly-a"); + assert_eq!( + moved.neuron_count, arrived.neuron_count, + "the same number of neurons" + ); + assert_ne!(moved.index_digest, arrived.index_digest, "and another index"); + + let mut consumer = first.harness.consumer().await.expect("a consumer attaches"); + let held = match within("the descriptor", consumer.take_descriptor()).await { + Some(ConsumerOutcome::Composition { revision, .. }) => revision, + other => panic!("expected a composition, got {other:?}"), + }; + consumer.map_geometry(held).expect("this consumer maps geometry"); + assert_eq!(consumer.mapped_index(&fly_a()), Some(&moved.index_digest)); + + // The composition changes under it. + let publisher_client = first + .harness + .publisher() + .await + .expect("a publishing client"); + let mut publisher = fly_session::publish::Publisher::new( + publisher_client, + &first.harness.config.session_id, + &first.harness.config.epoch, + first.harness.coordinator.topics(), + ); + publisher + .publish_descriptor(&revised) + .await + .expect("the revision publishes"); + let outcome = within("the revised descriptor", consumer.take_descriptor()).await; + assert_eq!( + outcome, + Some(ConsumerOutcome::IndexChanged { + agent_id: fly_a(), + from: moved.index_digest.clone(), + to: arrived.index_digest.clone(), + }), + "the change is named" + ); + assert_eq!( + consumer.mapped_index(&fly_a()), + Some(&moved.index_digest), + "and nothing was remapped behind the consumer's back" + ); + assert!( + consumer.descriptor(DESCRIPTOR_REVISION + 1).is_none(), + "the revision it refused is not in its cache either" + ); + second.shutdown().await; + first.shutdown().await; +} + +/// The descriptor says what the workers said, not what the composition asked for. +async fn the_published_descriptor_is_what_the_workers_attested_to(via: Via) { + let f = started(via).await; + let descriptor = f + .harness + .coordinator + .session_descriptor() + .expect("a descriptor") + .clone(); + assert_eq!(descriptor.revision, DESCRIPTOR_REVISION); + assert_eq!(descriptor.agents.len(), 2); + for agent in &descriptor.agents { + let expected = fly_session::agent::synthetic_graph(&agent.agent_id, 0); + assert_eq!( + agent.index_digest, expected.index_digest, + "the fly's own index" + ); + assert_eq!(agent.dataset_digest, expected.dataset_digest); + assert_eq!(agent.neuron_count, expected.neuron_count); + assert_eq!( + agent.rate_roles, expected.rate_roles, + "the profile's rate order" + ); + assert_eq!(agent.supported_stimuli, expected.supported_stimuli); + assert!( + descriptor.environment.port(&agent.port_id).is_some(), + "every agent's port is declared by the environment" + ); + } + let mut asset_ids: Vec<&Id> = descriptor.assets.iter().map(|a| &a.id).collect(); + asset_ids.sort(); + asset_ids.dedup(); + assert_eq!( + asset_ids.len(), + descriptor.assets.len(), + "assets are unique by id" + ); + f.shutdown().await; +} + +/// `supportedStimuli` is enforced, not advertised: a kind outside the list the descriptor +/// publishes is refused before the model is touched, so the declaration is worth reading. +async fn a_stimulus_kind_the_descriptor_does_not_declare_is_refused(via: Via) { + let mut f = started(via).await; + let declared = f + .harness + .coordinator + .session_descriptor() + .expect("a descriptor") + .agents + .iter() + .find(|a| a.agent_id == fly_a()) + .expect("fly-a") + .supported_stimuli + .clone(); + assert!(!declared.contains(&id("arena.undeclared")), "{declared:?}"); + + // A clean transition first, so the refusal is the injection and not the composition. + f.harness.coordinator.run(1).await.expect("one clean transition"); + let before = f.harness.coordinator.stats().advances; + f.harness.coordinator.injections = Injections { + at_step: 1, + undeclared_stimulus: true, + ..Injections::default() + }; + let failure = f + .harness + .coordinator + .run(1) + .await + .expect_err("an undeclared stimulus kind is refused"); + assert_eq!(failure.error.code, ErrorCode::Unsupported, "{failure:?}"); + assert_eq!( + failure.error.mutation, + MutationCertainty::None, + "refused before the model is touched" + ); + assert!( + failure.error.message.contains("arena.undeclared"), + "the refusal names the kind: {}", + failure.error.message + ); + assert!(failure.participant.is_some(), "and the participant it came from"); + assert_eq!( + f.harness.coordinator.stats().advances, + before, + "the refused transition never completed, so no boundary was added" + ); + assert!( + f.harness.coordinator.is_fenced(), + "a commit that refused after the world moved fences the epoch" + ); + f.shutdown().await; +} + +// ------------------------------------------------------------------------------------------ +// "A committed action is labelled as the transition that just ended, not the one about to +// start." + +async fn boundary_zero_publishes_no_decision_and_no_controls(via: Via) { + let f = started(via).await; + let state = f.harness.coordinator.published_state(); + let snapshot = { + let state = state.lock().expect("not poisoned"); + state + .latest_snapshot() + .expect("boundary 0 was published") + .clone() + }; + assert_eq!(snapshot.scope.step, 0); + for agent in &snapshot.agents { + assert!( + agent.selected_decision.is_none(), + "boundary 0 ended no transition" + ); + assert!(agent.applied_controls.is_none()); + } + f.shutdown().await; +} + +/// The decision published at boundary k+1 is the one the agent prepared for the transition +/// k -> k+1, never the one it is about to prepare for k+1 -> k+2. +async fn a_committed_action_is_the_transition_that_just_ended(via: Via) { + let mut f = started(via).await; + let mut consumer = f.harness.consumer().await.expect("a consumer attaches"); + assert!(matches!( + within("the descriptor", consumer.take_descriptor()).await, + Some(ConsumerOutcome::Composition { .. }) + )); + let mut published: Vec<(u64, Digest)> = Vec::new(); + for step in 1..=STEPS { + f.harness.coordinator.run(1).await.expect("a transition"); + read_until(&mut consumer, step).await; + let view = consumer.last().expect("a snapshot"); + let agent = view.agent(&fly_a()).expect("fly-a is in every snapshot"); + let decision = agent + .decision + .as_ref() + .expect("past boundary 0 there is a decision"); + published.push((view.boundary, typed_digest(decision))); + assert!( + agent.controls.is_some(), + "and the controls that were applied" + ); + } + // The trace records, per transition, the decision digest the coordinator actually sent to + // the world. The snapshot at boundary k+1 must carry the transition ending there. + let trace = &f.harness.coordinator.trace.transitions; + assert_eq!(trace.len() as u64, STEPS); + for (index, (boundary, digest)) in published.iter().enumerate() { + let transition = &trace[index]; + assert_eq!( + transition.behaviour.published_boundary, *boundary, + "the snapshot at {boundary} is the transition that ended there" + ); + let traced = transition + .behaviour + .agents + .iter() + .find(|a| a.agent_id == fly_a()) + .expect("fly-a in the trace"); + assert_eq!( + traced.decision_digest, *digest, + "the decision published at boundary {boundary} is the one that produced it" + ); + assert_eq!(traced.committed_step, *boundary); + } + // And it is not the next transition's, which the following trace row holds. + if STEPS > 1 { + let first = &published[0]; + let next = trace[1] + .behaviour + .agents + .iter() + .find(|a| a.agent_id == fly_a()) + .expect("fly-a in the trace"); + assert_ne!( + first.1, next.decision_digest, + "boundary 1 did not publish the decision of the transition about to start" + ); + } + f.shutdown().await; +} + +/// One snapshot is the whole composition: a presentation consumer is a multi-agent consumer +/// and has no per-fly stream to join. +async fn one_snapshot_carries_every_agent_in_the_composition(via: Via) { + let mut f = started(via).await; + let mut consumer = f.harness.consumer().await.expect("a consumer attaches"); + assert!(matches!( + within("the descriptor", consumer.take_descriptor()).await, + Some(ConsumerOutcome::Composition { .. }) + )); + f.harness.coordinator.run(1).await.expect("a transition"); + let outcome = match within("a snapshot", consumer.take_snapshot()).await { + Some(outcome) => outcome, + None => panic!("the snapshot stream ended"), + }; + let agents = match outcome { + ConsumerOutcome::Read { agents, .. } => agents, + other => panic!("expected a readable snapshot, got {other:?}"), + }; + assert_eq!(agents, vec![fly_a(), fly_b()], "every agent, in one value"); + let view = consumer.last().expect("a snapshot"); + for agent_id in [fly_a(), fly_b()] { + let agent = view.agent(&agent_id).expect("in the snapshot"); + assert_eq!( + agent.rates.len(), + 2, + "telemetry in the descriptor's rate-role order" + ); + assert_eq!(agent.rates[0].0, id("kc")); + assert_eq!(agent.rates[1].0, id("mbon")); + assert!(!agent.index_digest.is_empty()); + } + f.shutdown().await; +} + +/// A group restore re-establishes a committed boundary this epoch did not run a transition +/// into. Two things follow, and both are published rather than inferred: the composition is a +/// new one, because a fresh epoch is a new `compositionDigest`, so the descriptor takes the +/// next revision; and the restored boundary carries no decision and no controls for any agent, +/// because the abandoned epoch's actions are not this session's to republish. +async fn a_restored_boundary_publishes_a_new_revision_and_no_transition(via: Via) { + let mut f = started(via).await; + let checkpoint = id("ck-1"); + f.harness.coordinator.run(1).await.expect("one transition"); + let outcome = within("checkpoint", f.harness.coordinator.checkpoint(&checkpoint)) + .await + .expect("a committed checkpoint"); + assert!(matches!(outcome, fly_session::state::SaveOutcome::Committed { .. }), "{outcome:?}"); + let first = f.harness.coordinator.descriptor_revision(); + + // The snapshot of a boundary this epoch produced does carry the transition. + let produced = { + let state = f.harness.coordinator.published_state(); + let state = state.lock().expect("not poisoned"); + state.latest_snapshot().expect("boundary 1").clone() + }; + assert_eq!(produced.scope.step, 1); + assert!(produced.agents.iter().all(|a| a.selected_decision.is_some())); + + // Fail the epoch and restore into a fresh one. + f.harness.kill(&fly_a()).await; + f.harness.coordinator.step().await.expect_err("a dead participant fails the epoch"); + within("replace", f.harness.replace_all_participants()) + .await + .expect("replacements"); + within( + "restore", + f.harness.coordinator.restore(Some(&checkpoint), &id("e2")), + ) + .await + .expect("a coherent group restore"); + + assert_eq!( + f.harness.coordinator.descriptor_revision(), + first + 1, + "a fresh epoch is a new composition, so the revision advanced" + ); + let restored = { + let state = f.harness.coordinator.published_state(); + let state = state.lock().expect("not poisoned"); + state.latest_snapshot().expect("the restored boundary").clone() + }; + assert_eq!(restored.scope.step, 1, "the same committed boundary"); + assert_eq!(restored.descriptor_revision, first + 1); + assert!( + restored.agents.iter().all(|a| a.selected_decision.is_none() && a.applied_controls.is_none()), + "an installed boundary carries no transition for any agent" + ); + f.shutdown().await; +} + +// ------------------------------------------------------------------------------------------ +// Application-owned state and cues, bounded events, and the read-only repair service. + +/// The application publishes its own state and cues, on its own addresses, under its own +/// schema. Nothing about them is framework-shaped. +async fn application_state_and_cues_are_the_applications_own(via: Via) { + let mut f = started(via).await; + let client = f + .harness + .application() + .await + .expect("an application client"); + let mut channel = ApplicationChannel::new(client, "app.counter", 16); + channel + .declare() + .await + .expect("the application declares its own topics"); + assert_eq!(channel.state_topic(), "app.counter.state"); + assert_eq!(channel.cue_topic(), "app.counter.cues"); + + let watcher = f.harness.observer().await.expect("an observer client"); + let mut states = watcher + .subscribe(channel.state_topic(), Delivery::LatestValue.subscription()) + .await + .expect("a latest subscription"); + let mut cues = watcher + .subscribe( + channel.cue_topic(), + Delivery::BoundedBatch { depth: 16 }.subscription(), + ) + .await + .expect("a bounded subscription"); + + f.harness.coordinator.run(1).await.expect("a transition"); + // The application's schema: its own namespace, not a framework field. + let schema = SchemaRef { + id: id("counter.show.v1"), + version: 1, + digest: digest_of_bytes(b"counter.show.v1"), + }; + let state = typed(schema.clone(), json!({ "featured": "fly-a", "streak": 3 })) + .expect("an application value"); + let outcome = channel.publish_state(1, &state).await; + assert!(outcome.is_accepted(), "{outcome:?}"); + let cue = typed(schema, json!({ "line": "fly-a takes the lead" })).expect("a cue"); + assert!( + channel + .publish_cue(1, &id("counter.headline"), &cue) + .await + .is_accepted() + ); + + let message = within("the application state", states.next()) + .await + .expect("a state message"); + assert_eq!( + message.payload().get("boundary").and_then(|v| v.as_str()), + Some("1") + ); + let published = TypedValue::from_json(message.payload().get("state").expect("state")) + .expect("an application typed value"); + assert_eq!(published.schema.id, id("counter.show.v1")); + drop(message); + let message = within("the cue", cues.next()).await.expect("a cue message"); + assert_eq!( + message.payload().get("kind").and_then(|v| v.as_str()), + Some("counter.headline") + ); + drop(message); + + // Nothing the application published is on a session topic, and nothing on a session topic + // knows the application's schema. + let descriptor = f + .harness + .coordinator + .session_descriptor() + .expect("a descriptor"); + assert_ne!(descriptor.task_schema.id, id("counter.show.v1")); + f.shutdown().await; +} + +/// A refused event batch is held, counted and delivered later. Nothing is dropped quietly, +/// and what the bounded depth does push out travels as a count. +async fn a_refused_event_batch_is_held_and_counted_not_lost(via: Via) { + let mut f = started(via).await; + let events_topic = f.harness.coordinator.topics().events.clone(); + let offender_client = f.harness.observer().await.expect("an observer client"); + let mut offender = offender_client + .subscribe( + &events_topic, + flybus::SubscriptionConfig::bounded().queued(1).in_flight(1), + ) + .await + .expect("a bounded subscription"); + + f.harness + .coordinator + .run(STEPS) + .await + .expect("an event observer never fails the epoch"); + assert_eq!(f.harness.coordinator.stats().advances, STEPS); + let counters = f.harness.coordinator.ledger().counters(&events_topic); + assert!(counters.refused > 0, "the refusal is counted: {counters:?}"); + assert!(f.harness.coordinator.ledger().events_held > 0); + assert!( + f.harness.coordinator.pending_events() > 0, + "the refused events are held in a bounded batch, not dropped" + ); + + // The offender starts consuming; the held batch goes out at the next boundary. + drop(within("an event delivery", offender.next()).await); + drop(offender.try_next()); + let held = f.harness.coordinator.pending_events(); + f.harness.coordinator.run(1).await.expect("a transition"); + until("the held batch is published", || { + f.harness.coordinator.pending_events() < held + }) + .await; + offender_client.close().await; + f.shutdown().await; +} + +/// A publication an observer refused is exactly the value the repair path exists to hand back. +/// +/// The `state-media-v1` amendment promises the exact value stays recoverable through the query +/// path, so recording it cannot depend on whether the router admitted the delivery -- that is +/// the case the promise is about. +async fn a_refused_snapshot_is_still_what_the_repair_path_answers(via: Via) { + let mut f = started(via).await; + let snapshots = f.harness.coordinator.topics().snapshots.clone(); + let offender_client = f.harness.observer().await.expect("an observer client"); + let _offender = offender_client + .subscribe( + &snapshots, + flybus::SubscriptionConfig::bounded().queued(1).in_flight(1), + ) + .await + .expect("a bounded subscription"); + + f.harness.coordinator.run(STEPS).await.expect("the world carries on"); + let counters = f.harness.coordinator.ledger().counters(&snapshots); + assert!(counters.refused > 0, "a refusal is what this test is about: {counters:?}"); + + let latest = { + let state = f.harness.coordinator.published_state(); + let state = state.lock().expect("not poisoned"); + state.latest_snapshot().expect("a snapshot").clone() + }; + assert_eq!( + latest.scope.step, STEPS, + "the newest committed boundary is recoverable even though its delivery was refused" + ); + assert_eq!( + latest.sequence + 1, + f.harness.coordinator.published_sequence(), + "the sequence advanced with the value, not with the delivery" + ); + + // And the query service answers it over the bus, not just the state behind it. + let mut consumer = f.harness.consumer().await.expect("a consumer attaches"); + let answered = within("the repair path", consumer.repair(DESCRIPTOR_REVISION)) + .await + .expect("the repair path answers"); + assert_eq!(answered, DESCRIPTOR_REVISION); + offender_client.close().await; + f.shutdown().await; +} + +/// An event batch a consumer cannot read is reported as unreadable, and the stream goes on. +/// +/// The two facts differ: a consumer that read "that one made no sense" as "there are no more +/// events" would stop reading a live stream, which is the silent default this module exists to +/// refuse. The malformed batch is published by a second publisher on the session's own address, +/// because a well-behaved session cannot produce one. +async fn a_malformed_event_batch_is_unreadable_and_not_the_end_of_the_stream(via: Via) { + let mut f = started(via).await; + let mut consumer = f.harness.consumer().await.expect("a consumer attaches"); + let events_topic = f.harness.coordinator.topics().events.clone(); + let intruder = f.harness.publisher().await.expect("a publishing client"); + + // A batch with no droppedBefore: readable JSON, unreadable as a batch. + intruder + .publish( + &events_topic, + object(json!({ + "sessionId": f.harness.config.session_id.as_str(), + "epoch": f.harness.config.epoch.as_str(), + "events": [], + })), + &[], + ) + .await + .expect("the malformed batch publishes"); + + match within("the malformed batch", consumer.take_events()).await { + Some(ConsumerEvents::Unreadable { detail }) => { + assert!( + detail.contains("droppedBefore"), + "the report names what it could not read: {detail}" + ); + } + other => panic!("a malformed batch must not read as {other:?}"), + } + + // The stream did not end: the next real batch still arrives and reads. + f.harness.coordinator.run(1).await.expect("a transition"); + match within("the next batch", consumer.take_events()).await { + Some(ConsumerEvents::Batch(batch)) => { + assert_eq!(batch.epoch, f.harness.config.epoch); + assert!(!batch.event_ids.is_empty(), "the transition produced events"); + } + other => panic!("expected a readable batch after the malformed one, got {other:?}"), + } + intruder.close().await; + f.shutdown().await; +} + +/// Two reads and nothing else. There is no method on this service that could move anything. +async fn the_query_service_answers_reads_and_nothing_else(via: Via) { + let mut f = started(via).await; + f.harness.coordinator.run(1).await.expect("a transition"); + let client = f.harness.observer().await.expect("an observer client"); + let service = query_service(&f.harness.config.session_id); + + let request = SessionRpcRequest { + request_id: DomainRequestId::from_serial(1), + scope: None, + params: json!({}), + }; + let mut pending = client + .call(&service, None, GET_SNAPSHOT, object(request.to_json()), &[]) + .await + .expect("the service answers"); + let result = pending.result().await.expect("a reply"); + let outcome = + SessionRpcOutcome::from_json(&serde_json::Value::Object(result.outcome().clone())) + .expect("a domain outcome"); + drop(result); + let snapshot = CommittedSnapshot::from_json(outcome_result(&outcome).expect("a result")) + .expect("a committed snapshot"); + assert_eq!(snapshot.scope.step, 1); + assert_eq!( + snapshot.descriptor_revision, + f.harness.coordinator.descriptor_revision(), + "a read answers the composition the session is publishing under" + ); + assert!( + snapshot.agents.iter().all(|a| a.selected_decision.is_some()), + "and the values of the transition that ended at it" + ); + + // A method it does not implement is a named refusal, not a default. + let request = SessionRpcRequest { + request_id: DomainRequestId::from_serial(2), + scope: None, + params: json!({}), + }; + let mut pending = client + .call( + &service, + None, + "Session.Advance", + object(request.to_json()), + &[], + ) + .await + .expect("the service answers"); + let result = pending.result().await.expect("a reply"); + let outcome = + SessionRpcOutcome::from_json(&serde_json::Value::Object(result.outcome().clone())) + .expect("a domain outcome"); + drop(result); + let error = outcome_result(&outcome).expect_err("no such method"); + assert_eq!(error.code, ErrorCode::Unsupported, "{error:?}"); + assert_eq!( + f.harness.coordinator.stats().advances, + 1, + "and nothing advanced" + ); + client.close().await; + f.shutdown().await; +} + +// ------------------------------------------------------------------------------------------ +// Every execution mode + +/// The publication boundary is inside the coordinator, so it crosses no process boundary: it +/// must therefore behave identically in all three modes, and this asserts that rather than +/// assuming it. +async fn the_publication_boundary_holds_in_every_execution_mode(mode: ExecutionMode) { + let mut f = started_in_mode(mode).await; + let mut consumer = f.harness.consumer().await.expect("a consumer attaches"); + let revision = match within("the descriptor", consumer.take_descriptor()).await { + Some(ConsumerOutcome::Composition { revision, agents }) => { + assert_eq!(agents, vec![fly_a(), fly_b()]); + revision + } + other => panic!("expected a composition, got {other:?}"), + }; + assert_eq!(revision, DESCRIPTOR_REVISION); + let descriptor = consumer.descriptor(revision).expect("held").clone(); + for agent in &descriptor.agents { + // The graph identity is attested over the bus, so it crosses a process boundary like + // every other reply and is observable in every mode. + assert_eq!( + agent.index_digest, + fly_session::agent::synthetic_graph(&agent.agent_id, 0).index_digest + ); + } + f.harness + .coordinator + .run(STEPS) + .await + .expect("the world runs"); + let at = read_until(&mut consumer, STEPS).await; + assert_eq!(at, STEPS); + let view = consumer.last().expect("a snapshot"); + assert_eq!(view.agents.len(), 2); + assert!(view.agent(&fly_a()).expect("fly-a").decision.is_some()); + assert_eq!( + f.harness + .coordinator + .ledger() + .counters(&f.harness.coordinator.topics().snapshots) + .accepted, + STEPS + 1 + ); + f.shutdown().await; +} + +/// A stalled observer costs only itself, in every mode. +async fn a_stalled_observer_never_moves_the_world_in_any_execution_mode(mode: ExecutionMode) { + let mut f = started_in_mode(mode).await; + let mut slow = f + .harness + .consumer() + .await + .expect("a slow consumer attaches"); + until("the slow viewer has a delivery", || { + slow.try_hold_snapshot() + }) + .await; + while slow.try_hold_snapshot() && slow.held() < 4 {} + let before = f.harness.coordinator.stats(); + let started_at = std::time::Instant::now(); + f.harness + .coordinator + .run(STEPS) + .await + .expect("the world carries on"); + assert_eq!( + f.harness.coordinator.stats().advances - before.advances, + STEPS + ); + assert!(!f.harness.coordinator.is_fenced()); + assert_eq!(f.harness.coordinator.ledger().refusals(), 0); + assert!(started_at.elapsed() < Duration::from_secs(10)); + f.shutdown().await; +} + +// ------------------------------------------------------------------------------------------ +// The policy and the batch, without a session + +#[test] +fn a_topic_policy_declares_its_delivery_once() { + let latest = TopicPolicy::latest("session.demo.snapshots"); + assert_eq!(latest.delivery, Delivery::LatestValue); + assert_eq!(latest.delivery.retained(), flybus::Retained::Latest); + assert_eq!(latest.delivery.subscription().mode, flybus::Mode::Latest); + let bounded = TopicPolicy::bounded("session.demo.events", 8); + assert_eq!(bounded.delivery, Delivery::BoundedBatch { depth: 8 }); + // A bounded stream is never retained: it is "not a durable log", and a retained tail + // would be the beginning of one. + assert_eq!(bounded.delivery.retained(), flybus::Retained::None); + assert_eq!(bounded.delivery.subscription().max_queued, 8); +} + +#[test] +fn a_full_event_batch_drops_the_oldest_by_an_explicit_count() { + let mut outbox = EventOutbox::new(EVENT_BATCH_DEPTH); + let event = |n: u64| TaskEvent { + id: id(&format!("ev-{n}")), + kind_id: id("arena.counter-delta"), + source_step: 1, + agent_id: None, + payload: typed( + SchemaRef { + id: id("t.v1"), + version: 1, + digest: digest_of_bytes(b"t.v1"), + }, + json!({ "n": n }), + ) + .expect("a typed value"), + }; + for n in 0..EVENT_BATCH_DEPTH as u64 { + assert_eq!( + outbox.offer(1, &[event(n)]), + 0, + "nothing is dropped inside the depth" + ); + } + assert_eq!(outbox.len(), EVENT_BATCH_DEPTH); + assert_eq!( + outbox.offer(2, &[event(999)]), + 1, + "the oldest is dropped, and counted" + ); + assert_eq!(outbox.dropped_since_accepted(), 1); + assert_eq!( + outbox.len(), + EVENT_BATCH_DEPTH, + "and the batch stays bounded" + ); +} + +#[tokio::test] +async fn a_snapshot_that_disagrees_with_its_descriptor_is_refused() { + let f = started(Via::Memory).await; + let descriptor = f + .harness + .coordinator + .session_descriptor() + .expect("a descriptor") + .clone(); + let snapshot = { + let state = f.harness.coordinator.published_state(); + let state = state.lock().expect("not poisoned"); + state.latest_snapshot().expect("boundary 0").clone() + }; + let positions: BTreeMap = snapshot + .audio + .iter() + .map(|a| (a.stream_id.clone(), a.first_sample + a.sample_frames)) + .collect(); + let attachments: Vec<(String, ArtifactRef)> = snapshot + .views + .iter() + .map(|v| { + ( + fly_session::media::view_attachment(&v.view_id), + v.pixels.clone(), + ) + }) + .chain(snapshot.audio.iter().map(|a| { + ( + fly_session::media::audio_attachment(&a.stream_id), + a.samples.clone(), + ) + })) + .collect(); + + let mut wrong = snapshot.clone(); + wrong.descriptor_revision = DESCRIPTOR_REVISION + 1; + let error = check_publication(&descriptor, &wrong, &attachments, &positions) + .expect_err("another revision is a disagreement"); + assert_eq!(error.code, ErrorCode::BufferInvalid, "{error:?}"); + + let mut renamed = snapshot.clone(); + renamed.agents[0].agent_id = id("fly-z"); + let error = check_publication(&descriptor, &renamed, &attachments, &positions) + .expect_err("an agent the descriptor does not declare"); + assert!(error.message.contains("fly-z"), "{}", error.message); + + // And a snapshot the session actually published passes the same check. + check_publication(&descriptor, &snapshot, &attachments, &positions) + .expect("what the session published agrees with the descriptor it named"); + f.shutdown().await; +}