Compare commits

..

15 commits

Author SHA1 Message Date
acamilo
56db91cd9b docs: record the session framework wave 2
Some checks are pending
ci / node 22 (test + typecheck) (push) Waiting to run
ci / rust stable (cargo test --workspace --release) (push) Waiting to run
ci / infra/tests/lint.sh (push) Waiting to run
ci / playwright apps/stage (allowed to fail) (push) Waiting to run
2026-09-22 16:42:17 +00:00
acamilo
a301527b09 Merge feat/sf-media-01: native view and audio observations, the shared frame, and the spectator path 2026-09-22 16:30:04 +00:00
acamilo
66ec2e5c6b session: an unknown launch option is an error, not a no-op
The command line between a launcher and a participant in its own process
was coupled by string convention on both sides and checked by neither: the
parser accepted any --flag it did not know, so a renamed or dropped option
would have become a fault that never fires or a delay that is never
applied, with nothing failing.

Every flag now has one name, in launcher::flags, written by the launcher's
argv and read by the parser. Options::parse takes the sets its command
allows and refuses anything outside them by name, so a mismatch fails the
launch. measure and measure-row use the same constants for their own
flags.

Three tests hold the two sides together: every flag a launch writes is one
its command accepts, every media option is written for a separate process,
and an unknown or wrong-command flag is refused by name. The four media
faults now also run end to end in all three execution modes, each failing
its own transition, so the wiring across a process boundary is proved
rather than assumed.
2026-09-22 16:28:59 +00:00
acamilo
f456fe9522 Merge main: the per-fly processes, the launcher and the thread budgets
Keep-both everywhere the two slices met. lib.rs takes both module sets.
AgentConfig keeps worker_threads and the sensor log; EnvironmentConfig
keeps worker_threads, the render delay and the render counter.
coordinator.rs keeps the two-stage resolution and its blame() beside the
media split of the Advance reply's attachments, and its imports take both.
harness.rs is main's launcher-based file with this slice's media
instrumentation re-applied on top.

The media instrumentation is shared memory, so it now follows the
launcher's own rule for the progress counter: sensor_log and renders
return None for a participant with a process of its own rather than a
misleading zero. The launcher carries the sensor log and the render
counter to a participant in this process and the render delay and the four
media faults on the command line to one in another process, where the
child builds its own log and counter.

The media path itself is mode-agnostic and is now tested as such: one
image per boundary, forwarded to every agent and published once, asserted
over the bus in all three execution modes, with the shared-memory
assertions made only where those participants live.
2026-09-22 16:07:30 +00:00
acamilo
ba545d9225 Merge feat/sf-session-02: per-fly processes, a launcher with thread budgets, and the fault behaviour 2026-09-22 16:00:57 +00:00
acamilo
a3c1c125cd session: negative fixtures for the allocation, and honest resolution bounds
Two review notes.

The launcher allocation became a required field of HelloResult.limits
with no negative fixture behind it. Three rows now cover it: missing,
above maxWorkerThreads, and zero. Both readers reject all three, and no
derived fixture moved, because invalid.json is not one of them.

The resolution's two bounds disagreed. 512 attempts at a 2 ms pause
give up near 1.5 s, so the attempt count silently pre-empted the 8 s
budget the doc comment advertised. The budget is now the working limit
and says so: the guard is 8192 attempts, over sixteen seconds of pauses
against an eight-second budget, so at the default values the budget is
always what fires. Which one did is no longer arithmetic either --
ResolutionEnd records it, the failure message names the bound and its
size, and the code, the doc comment and the README all state the same
numbers. a_resolution_says_which_of_its_two_bounds_ended_it drives each
bound to the end in every execution mode.
2026-09-22 16:00:13 +00:00
acamilo
77c8ee4558 session: resolve uncertain calls, and measure each mode in its own process
Review fixes for SESSION-02.

An expired caller deadline was becoming a failed epoch without the
ipc-v1 section 6 resolution. That procedure existed and was correct and
had exactly one caller, a test injection, so the deadline this slice
introduced bypassed it and a merely slow participant lost its epoch.
Deadlines is now the two-stage shape section 6 describes -- a probe,
then a bounded resolve budget and attempt count -- call_owned returns a
typed CallOutcome so an expiry is distinguishable from a refusal, and
Prepare, Commit, Advance and the lifecycle calls all query the same
request id against the same incarnation before the epoch can fail. This
is also step-v1 section 7's Advance row, which was imperative about it.

The coordinator peak-RSS column was measuring the measuring process.
VmHWM never falls and every row shared one process, so the column was
cumulative and the mode ranking reversed when the rows were reordered.
Each row now runs in a measure-row child of its own. The corrected
numbers say the opposite of what the first report claimed: the
coordinator's own peak is roughly flat across the modes and lowest in
process mode, and the cost of the split is the children.

workers-v1 section 2 bounded Agent.Initialize's workerThreads by
"within launcher allocation" and named no wire for it. Dated amendment:
HelloResult.limits gains workerThreads, the worker reports what its
launcher gave it, and the launcher refuses one that disagrees. The
schema set, the shared fixtures and the TypeScript package move
together; contractDigest changes, which ipc-v1 section 4 provides for.

Also: the stale-epoch row now reaches the stale-epoch path against a
live agent process and asserts exact codes on both halves; the
router-restart row asserts the handle drop it claimed; frames are
counted from the behaviour trace instead of calculated; the README says
which suites run over which transports; bootstrap is fence-guarded; the
shutdown reason is an Id rather than a silent fallback; and
agent_mutations returns None rather than zero where the counter lives
in another process.
2026-09-22 15:41:42 +00:00
acamilo
a48e0ace4c docs: the fresh-epoch discontinuity reason rests on section 6 alone 2026-09-22 15:18:47 +00:00
acamilo
079842f818 media: no best-effort defaults in the audio path
Review round two.

A restored timeline refuses a declared stream with no recorded sample
position instead of resuming it at zero, which would have restarted the
episode's audio clock silently.

A declared stream that produces no chunk for a transition now fails the
step. The contract does not say a chunk must exist; this slice requires
one, because a silently missing chunk cannot be told from a lost one.
Boundary 0 is exempt: no interval precedes it.

The fresh-epoch discontinuity refusal is dropped rather than written into
the amendment. Only the restore direction is stated, and sections 6 and 7
both have a fresh timeline publishing a discontinuity after a recovery or
a reset, so refusing the flag at an origin contradicted them. The dated
amendment now says the requirement is one-directional.

A snapshot with an unreadable chunk, or one whose handle is not attached,
is now unreadable as a whole rather than quietly carrying less audio than
was published. ViewPipeline::is_bootstrap_repeat is removed.

New tests: a missing chunk fails the step, restored timelines need every
declared stream's position, and a 7 Hz world's seven chunks sum to exactly
one second of 48 kHz samples. The bus-conformance reconnect row also cites
the test that actually replaces an incarnation.
2026-09-22 15:15:06 +00:00
acamilo
903629db00 docs: what MEDIA-01 enforces in state-media-v1 section 2
Two readings of the section that are now checked rather than assumed: the
bootstrap window is exactly the boundaries where max(0, boundary - delay)
is zero and its repeated O[0] is the same artifact, and a chunk that
continues the previous one exactly cannot claim a discontinuity, which is
what keeps the restore rule checkable instead of advisory.
2026-09-22 14:54:48 +00:00
acamilo
438eaf3130 Merge main at cb9a88c: the Pewter macro work, which this slice does not touch 2026-09-22 14:33:30 +00:00
acamilo
7b7ebcdf28 session: parallel processes, a launcher and the fault behaviour
SESSION-02: one agent process per fly and one environment process under
the coordinator over the Unix-socket transport, compared against the
in-process composition and a dedicated-thread variant. The mode is the
only thing that changes; the composition, the coordinator, the workers
and the router are the same code in all three.

The launcher is the configured supervisor. It owns a total thread budget
with one allocation per participant, refused as BUSY before anything
starts when the total cannot cover it; the configured client, service,
worker and port identities, proved in Worker.Hello before the
coordinator pins a registration; Worker.Status health on the
supervisor's own monotonic clock at the ipc-v1 section 6 budgets; and
reaping, where Worker.Shutdown is the request and the operating system
is the guarantee.

The worker executable is a subcommand of this crate's one binary, which
is what implementation.md section 2 allows in place of a separate
worker crate.

The coordinator's fault behaviour: every failure names the participant
it is attributed to, every domain call has a caller-side deadline so a
dead participant is a diagnosed outcome rather than a hang, and failing
fences the epoch -- the boundary stops, the handles drop, and no further
transition or publication is allowed. Agent.Initialize now carries the
launcher's allocation, and an agent refuses one asking for more.

tests/processes.rs proves every acceptance bullet once per execution
mode, and the two section 4 rows SESSION-01 could not reach in one
process: a router restart during a world advance, and an old worker's
reply after a restart. measure compares the three modes at one, two and
four agents; its table is in the crate README, and it is not a capacity
claim.
2026-09-22 14:33:25 +00:00
acamilo
8d999b2881 Merge branch 'main' into feat/sf-media-01 2026-09-22 14:30:34 +00:00
acamilo
8afeaf6c92 bus: wait for the teardown the disconnect test asserts
caller_disconnect_cleanup_works_before_and_after_consumption failed 37 times
in 240 runs under load: the reply raced the router's own teardown. op_reply
reports routed:false only once disconnect has marked the call detached, and
the router runs that when its connection task reads EOF, while Client::close
waits for this client's reader only. The reply could therefore reach a
connection that was still closing and be routed. Nothing escaped -- teardown
releases those roots -- but the flag was read one step early.

The test now settles on the caller's connection being gone before asserting
the section 6 sentence, which is the poll the rest of the file already uses.
360 runs after the fix, 0 failures.

bus-conformance.md records the mechanism, the counts and the two other
intermittent failures seen in the same sweep, which are left to the bus slice.
2026-09-22 14:29:57 +00:00
acamilo
40c6a88949 media: native observation schemas and the presentation handoff
Puts MEDIA-01 on top of the bus ArtifactRef rather than beside it.

fly-session-types gains the two validators a descriptor and a reference
cannot carry on their own: AudioTimeline, which holds one stream chunk
sequence for one epoch (no overlap, no backwards, the first chunk after a
restore marks the discontinuity), and check_imported_asset, which checks
that a transient artifact carries an installed asset content without ever
converting one identity into the other.

fly-session gains a media module: a ViewPipeline that renders one native
frame per boundary and serves the frame the declared observationDelaySteps
requires (so bootstrap repeats O[0] exactly while the pipeline fills), an
AudioSource with an exact rational sample budget and a deterministic
integer-phase waveform, the Phase C acceptance checks, a latest-subscription
Spectator with finite credits, a SensorLog recording what each agent read,
and an AssetRegistry for installed persistent content.

The counter arena now emits a real 32x24 RGBA8 pattern per boundary and one
audio chunk per transition; the coordinator forwards the one owned view
handle to every agent Commit and publishes the same handle, with the audio
handles, for presentation.

36 new tests: 12 shape rules against the contract crate and 24 session-level
acceptance tests over both transports.
2026-09-22 14:28:58 +00:00
35 changed files with 7242 additions and 397 deletions

View file

@ -119,7 +119,7 @@ and once over a Unix socket, so `tests/rpc.rs::request_reply_roundtrip` means
| Requirement | Status | Code | Test |
| --- | --- | --- | --- |
| `call-<U64>` with increasing serials per connected client; reused or retired ids are rejected, never executed again | conforms: a syntactically valid id advances the watermark even when admission is refused | `router/state.rs::op_call` (`call_watermark`) | `tests/sol_review_races.rs::rejected_call_id_still_advances_monotonic_watermark`, `tests/rpc.rs::raw_call_ids_and_forged_replies` (both) |
| Reconnecting creates a new incarnation rather than reviving old calls | conforms | `router/state.rs::{hello, disconnect}` | `tests/sol_review_races.rs::caller_disconnect_cleanup_works_before_and_after_consumption` |
| Reconnecting creates a new incarnation rather than reviving old calls | conforms | `router/state.rs::{hello, disconnect}` | `tests/conformance_routing.rs::unpinned_call_after_incarnation_replacement_reaches_the_new_holder` (both), `tests/sol_review_races.rs::caller_disconnect_cleanup_works_before_and_after_consumption` for the old calls half (its synchronisation was fixed on 2026-09-22; see "A flaky test and what it was measuring") |
| An RPC targets one registered service, not a broadcast subject | conforms | `router/state.rs::op_call` | `tests/rpc.rs::request_reply_roundtrip` (both) |
| First-dispatch FIFO per caller and service; responses may complete out of order and correlate by callId | conforms | `router/state.rs::{Svc::queue, dispatch_rpc}` | `tests/rpc.rs::fifo_dispatch_and_out_of_order_completion` (both), `tests/conformance_routing.rs::out_of_order_replies_correlate_across_concurrent_callers` (both) |
| A service dispatcher can answer status concurrently with a long mutation | conforms | `router/state.rs::dispatch_rpc` (in-flight credits, not one-at-a-time) | `tests/bus_acceptance.rs::a_status_rpc_responds_while_another_handler_is_delayed` (both) |
@ -412,6 +412,45 @@ What the numbers do and do not say:
cores against 0.33 to 0.46), because the clients own the copies. A thread that exits between
two samples takes its CPU with it, so the router figure is a floor.
## A flaky test and what it was measuring, 2026-09-22 (MEDIA-01)
`tests/sol_review_races.rs::caller_disconnect_cleanup_works_before_and_after_consumption`
failed intermittently on `main` after the bus slice merged. Reproduced here at
**37 failures in 240 runs** (four parallel loops of 60, debug, on the loaded dev VM), always
on the same line and always the same way: `responder.reply(...)` returned `routed:true` where
the test asserted `false`.
The mechanism is a synchronisation gap in the test, not a routing defect.
`router/state.rs::op_reply` returns `routed:false` only when `call.detached` is set, and for a
disconnected caller that flag is set by `router/state.rs::disconnect`, which the router runs
when **its** connection task reads EOF. `Client::close` documents what it waits for — "flushes
queued releases, closes the connection and waits until the reader has stopped ... the router
releases what they owned" — which is the client side only. So after `close()` returns, the
router may not have torn the caller's connection down yet, and a reply that reaches it first is
routed to a connection that is already closing. Nothing escapes: `disconnect` then releases
that connection's roots along with the queued result, which is why the test's own later
`settle` calls always passed. Only the `routed` flag, read one step too early, was wrong.
The fix is in the test: it now waits for the teardown it is talking about
(`e.settle("caller-a disconnected", |s| s.connections == 1)`) before asserting the
reply-to-a-detached-call sentence of section 6. That is the same bounded
poll-until-the-router-settles the rest of the file already uses for router-side consequences;
no sleep, no timing constant, and the assertion now has the precondition its contract sentence
names. **360 runs after the fix, 0 failures** (240 debug, 120 release).
Two other intermittent failures were seen in the same sweep and are **not** fixed here, since
they belong to the bus slice rather than to this one:
- `tests/example_demo.rs::the_example_shows_a_counter_rpc_an_observer_and_a_held_frame`,
2 failures in 40 standalone runs plus 1 in 12 full-suite runs. It prints
"while the frame is held: 1 artifact(s), 2 root(s)" instead of 1 root: the producer's hold
release is queued on the control lane and had not been applied when the example read the
counts. The same shape of gap, in the guide deliverable's printed output.
- `tests/integration.rs::unix_socket::session_over_one_router`, 1 failure in 12 full-suite
runs and 0 in 40 standalone runs, at the assertion that the deliberately slow consumer
skipped snapshots. Under load it kept up, so the assertion is a timing claim about the
machine.
## Contradictions
Two, both inside bus-v1, both minor, neither resolved by changing code. Both were referred to

View file

@ -94,10 +94,14 @@ interface HelloResult {
workerId: Id; incarnationId: Id; role: "agent" | "environment" | "coordinator";
buildDigest: Digest; contractDigest: Digest;
capabilities: Id[];
limits: { maxAgents: number; maxPorts: number };
limits: { maxAgents: number; maxPorts: number; workerThreads: number };
}
```
`limits.workerThreads` is the thread allocation the worker's launcher started it within; it
is an integer >=1 and its rule belongs to [worker interfaces](workers-v1.md) section 2, whose
2026-09-22 amendment added it.
The bus supplies caller identity; do not accept a forged caller in params. Bind a worker's
session authority to the expected coordinator identity/incarnation during negotiation and
initialization. Wrong worker/role, no common major or missing required capability refuses

View file

@ -65,6 +65,23 @@ the sample position relative to the episode's configured audio origin, with inte
firstSample/sampleRate. Crash restore preserves sample position under a new epoch; first
chunk marks discontinuity. Within an epoch, chunks cannot overlap or go backwards.
**Amendment, 2026-09-22 (MEDIA-01).** Two readings of the paragraphs above, made explicit
because they are now enforced:
- The bootstrap window is exactly the boundaries where `max(0, boundary - observationDelaySteps)`
is zero, that is `boundary <= observationDelaySteps`. Inside it the repeated `O[0]` is the
**same artifact**, not a fresh render of the same scene; outside it the producing boundary
advances one per step, and a frame from any other boundary -- older or newer -- is a step
failure. A producer therefore keeps a queue of `observationDelaySteps + 1` frames and nothing
more, so there is no older frame available to substitute.
- Within an epoch, `discontinuity` marks a range the stream actually skipped. The first chunk
after a restore marks it, and a later chunk may mark it when it starts past where the previous
chunk ended; a chunk that continues the previous one exactly is continuous by construction and
its flag is refused. Without that reading the restore rule is advisory, because a stream could
set the flag on every chunk and satisfy it by accident. The requirement is one-directional: a
fresh epoch's first chunk **may** mark a discontinuity, because section 6's recovery
establishes a fresh timeline and publishes one.
The environment provides **native game output**. Sensor transformations belong to the agent
profile. Resizing for viewers, overlays, composition, audio mixing/resampling, encoding,
browser delivery and streaming belong to the application/presentation layer. No bus or

View file

@ -99,6 +99,16 @@ interface AgentInitializeResult {
}
```
**Amendment, 2026-09-22 (SESSION-02).** `HelloResult.limits` gains `workerThreads`, an
integer >=1 reporting the allocation the launcher started that worker within, because
"within launcher allocation" above had no wire-level proof: the launcher passes the number to
the worker out of band, and a coordinator that is not also its own launcher had no contract
path to it. Hello is where a worker already proves its identity and reports its limits, so the
allocation belongs there. A caller asking for more than the worker reports is refused with
`BUSY` before the model is constructed, which this section already required; the amendment
only makes the number visible to whoever must respect it. It changes `contractDigest`, which
[session RPC](ipc-v1.md) section 4 already provides for.
The profile fixes warm-up/calibration behavior and supported schema versions. Validate inputs
and required roles before model construction. Install the initial sensory input, warm the
brain with learning disabled, calibrate the fixed readout and establish Ready(0). Do not

View file

@ -684,3 +684,44 @@ then purge; then the stale-doc pass.
ground with a new trap (row 54: GO FRONTIER, GO HEAL, GO ROUTE cycling, GO HEAL x204 at net 0);
Fable shipped it anyway: the hunt criterion compares within ground both arms reach, and a trap on
newly opened ground is a new row, not a regression. Row 54 review started at once.
## 2026-09-22 - session framework wave 2
Per-fly processes and native observations landed on top of the wave-1 contract and
lockstep session.
- The session runs in three execution modes that share one coordinator, one worker and
one router: in process, one thread per participant, and one process per fly plus one
for the environment over Unix sockets. The worker is a subcommand of the existing
binary, not a new crate. A launcher owns the thread budget, proves each participant's
configured identity and allocation on the wire before the coordinator pins anything,
polls health on its own clock, and reaps its children.
- A caller deadline that expires no longer fails the epoch on its own. It runs the
contract's resolution procedure against the same request and the same incarnation, so
a merely slow participant completes its step, and the epoch fails only on a definite
refusal, a lost incarnation or an exhausted budget. Which of the two bounds ended a
resolution is recorded and named in the failure rather than inferred.
- Failures name the participant, and a failed session fences its epoch: the boundary
stops, handles are dropped, and no further transition or publication is possible.
Worker death, helper death, a router restart mid-advance and stale replies after a
restart all have bounded, diagnosed outcomes, each proved in every mode.
- The environment now emits native output: one shared RGBA frame per boundary reaching
both flies through owned attachments, and one audio chunk per transition on an exact
rational sample budget. Observation delay is a real queue, so nothing stale can be
substituted. Spectators watch a latest subscription with finite credits and cannot
perturb what the flies sense. Audio never enters sensory input.
- Every media rule is enforced rather than assumed: strides, dimensions, formats,
lengths, producing step, timeline continuity, and the distinction between a persistent
asset and a transient artifact. A missing or malformed frame or chunk fails its step.
- Three contract silences were closed by dated amendments rather than by convention: the
launcher's thread allocation is now on the wire in the worker's hello, the audio
discontinuity rule is one-directional, and a mid-step pause is defined.
Measured on the development box, not capacity claims: with two flies the critical path
per transition sat near 10 to 12 ms at the median across all three modes, so a process
boundary costs little at the median and shows in the tail. What the split costs is
memory, roughly 5.7 MiB per participant process, while the coordinator's own footprint
is flat and lowest once the workers leave it.
Two flaky bus tests predating this work assert timing rather than contract and are being
rewritten separately.

View file

@ -43,6 +43,8 @@ export const MAX_ACKNOWLEDGE = 16;
export const MAX_ENGINE_FRAME_LEN = 64;
/** Not stated by a document; this crate's choice, published in the schema set. */
export const MAX_CAPABILITIES = 32;
/** The largest `workerThreads` a launcher may allocate to one worker (workers-v1 2). */
export const MAX_WORKER_THREADS = 4096;
export const MAX_SUPPORTED_MAJORS = 8;
export const MAX_MESSAGE_CODE_POINTS = 512;
@ -305,7 +307,7 @@ export function readAgentInitializeParams(value: unknown): AgentInitializeParams
seed: reader.int('seed', -2_147_483_648, 2_147_483_647),
initialInput: readSensoryInput(reader.value('initialInput')),
initialDecisionContext: readTypedValue(reader.value('initialDecisionContext')),
workerThreads: reader.int('workerThreads', 1, 4_096),
workerThreads: reader.int('workerThreads', 1, MAX_WORKER_THREADS),
};
reader.finish();
return params;
@ -776,7 +778,12 @@ export interface HelloResult {
buildDigest: Digest;
contractDigest: Digest;
capabilities: Id[];
limits: { maxAgents: number; maxPorts: number };
/**
* `workerThreads` is the thread allocation this worker was launched within, added by the
* 2026-09-22 amendment to workers-v1 section 2: the bound "within launcher allocation" had
* no wire on which a caller could learn the allocation.
*/
limits: { maxAgents: number; maxPorts: number; workerThreads: number };
}
export interface StatusResult {
@ -860,6 +867,7 @@ export function readHelloResult(value: unknown): HelloResult {
const limits = {
maxAgents: limitsReader.int('maxAgents', 1, MAX_AGENTS),
maxPorts: limitsReader.int('maxPorts', 1, MAX_PORTS),
workerThreads: limitsReader.int('workerThreads', 1, MAX_WORKER_THREADS),
};
limitsReader.finish();
reader.finish();

View file

@ -1,9 +1,9 @@
{
"description": "contractDigest is the SHA-256 of the canonical schema set in schema-set.json.",
"contractDigest": "7932aef30c4d2d16e428081affc4e0ad187987f5b361138d553e54fd7f843b50",
"contractDigest": "d8f29a49b5df05ad8f75f7f5790a3f8cde9c5ad23a685137474c649c3c9da36d",
"schemaSetVersion": 1,
"schemaSetBytes": 26685,
"schemaSetBytes": 26814,
"types": 53,
"enums": 11,
"limits": 25
"limits": 26
}

View file

@ -2802,7 +2802,8 @@
],
"limits": {
"maxAgents": 4,
"maxPorts": 4
"maxPorts": 4,
"workerThreads": 1
}
},
"reason": "an agent must advertise agent-step-v1"
@ -2823,7 +2824,8 @@
],
"limits": {
"maxAgents": 5,
"maxPorts": 4
"maxPorts": 4,
"workerThreads": 1
}
},
"reason": "the first composition allows four agents"
@ -2844,11 +2846,77 @@
],
"limits": {
"maxAgents": 4,
"maxPorts": 4
"maxPorts": 4,
"workerThreads": 1
}
},
"reason": "v1 selects major 1"
},
{
"name": "hello result without its launcher allocation",
"type": "HelloResult",
"value": {
"selectedMajor": 1,
"selectedMinor": 0,
"workerId": "fly-a",
"incarnationId": "inc-1",
"role": "agent",
"buildDigest": "44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66",
"contractDigest": "cc8321d6375c494d043fdd0260f21bc0ec51dacc9f6abb7f909cdcd3041b78bf",
"capabilities": [
"agent-step-v1"
],
"limits": {
"maxAgents": 4,
"maxPorts": 4
}
},
"reason": "limits.workerThreads is required by the 2026-09-22 workers-v1 amendment"
},
{
"name": "hello result promising more threads than a launcher may allocate",
"type": "HelloResult",
"value": {
"selectedMajor": 1,
"selectedMinor": 0,
"workerId": "fly-a",
"incarnationId": "inc-1",
"role": "agent",
"buildDigest": "44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66",
"contractDigest": "cc8321d6375c494d043fdd0260f21bc0ec51dacc9f6abb7f909cdcd3041b78bf",
"capabilities": [
"agent-step-v1"
],
"limits": {
"maxAgents": 4,
"maxPorts": 4,
"workerThreads": 4097
}
},
"reason": "limits.workerThreads is at most maxWorkerThreads"
},
{
"name": "hello result reporting no threads at all",
"type": "HelloResult",
"value": {
"selectedMajor": 1,
"selectedMinor": 0,
"workerId": "fly-a",
"incarnationId": "inc-1",
"role": "agent",
"buildDigest": "44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66",
"contractDigest": "cc8321d6375c494d043fdd0260f21bc0ec51dacc9f6abb7f909cdcd3041b78bf",
"capabilities": [
"agent-step-v1"
],
"limits": {
"maxAgents": 4,
"maxPorts": 4,
"workerThreads": 0
}
},
"reason": "limits.workerThreads is at least one"
},
{
"name": "hello params with no supported majors",
"type": "HelloParams",
@ -3897,4 +3965,4 @@
"reason": "a commit acknowledges the transition's next boundary"
}
]
}
}

File diff suppressed because one or more lines are too long

View file

@ -1758,12 +1758,13 @@
],
"limits": {
"maxAgents": 4,
"maxPorts": 4
"maxPorts": 4,
"workerThreads": 1
}
},
"note": "",
"canonical": "{\"buildDigest\":\"44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66\",\"capabilities\":[\"agent-step-v1\",\"checkpoint-v1\",\"pixel-observation-v1\"],\"contractDigest\":\"cc8321d6375c494d043fdd0260f21bc0ec51dacc9f6abb7f909cdcd3041b78bf\",\"incarnationId\":\"inc-1\",\"limits\":{\"maxAgents\":4,\"maxPorts\":4},\"role\":\"agent\",\"selectedMajor\":1,\"selectedMinor\":0,\"workerId\":\"fly-a\"}",
"digest": "9f8cbc9dfe7a7532c2e41b53c4ce001973ca95703a81aad30ee2b1c8b640bc13"
"canonical": "{\"buildDigest\":\"44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66\",\"capabilities\":[\"agent-step-v1\",\"checkpoint-v1\",\"pixel-observation-v1\"],\"contractDigest\":\"cc8321d6375c494d043fdd0260f21bc0ec51dacc9f6abb7f909cdcd3041b78bf\",\"incarnationId\":\"inc-1\",\"limits\":{\"maxAgents\":4,\"maxPorts\":4,\"workerThreads\":1},\"role\":\"agent\",\"selectedMajor\":1,\"selectedMinor\":0,\"workerId\":\"fly-a\"}",
"digest": "262ea112e24dcdbdc1a5050b6dd6d031eace6a6a4cdca814913ac89fc9e39666"
},
{
"name": "hello result for an environment",
@ -1782,12 +1783,13 @@
],
"limits": {
"maxAgents": 1,
"maxPorts": 1
"maxPorts": 1,
"workerThreads": 1
}
},
"note": "",
"canonical": "{\"buildDigest\":\"44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66\",\"capabilities\":[\"world-step-v1\",\"checkpoint-v1\"],\"contractDigest\":\"cc8321d6375c494d043fdd0260f21bc0ec51dacc9f6abb7f909cdcd3041b78bf\",\"incarnationId\":\"inc-2\",\"limits\":{\"maxAgents\":1,\"maxPorts\":1},\"role\":\"environment\",\"selectedMajor\":1,\"selectedMinor\":0,\"workerId\":\"world\"}",
"digest": "0dc33a6cbf6bf66c4d28dc41b17b2eb7918b55c65469f4161e009f662f11bca2"
"canonical": "{\"buildDigest\":\"44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66\",\"capabilities\":[\"world-step-v1\",\"checkpoint-v1\"],\"contractDigest\":\"cc8321d6375c494d043fdd0260f21bc0ec51dacc9f6abb7f909cdcd3041b78bf\",\"incarnationId\":\"inc-2\",\"limits\":{\"maxAgents\":1,\"maxPorts\":1,\"workerThreads\":1},\"role\":\"environment\",\"selectedMajor\":1,\"selectedMinor\":0,\"workerId\":\"world\"}",
"digest": "06ced6fff0d810005ab2598cc9fb72a138303daef0787232871c9afa43b0eac0"
},
{
"name": "status result before initialization",

View file

@ -366,6 +366,161 @@ impl DomainType for AudioRef {
}
}
// ---------------------------------------------------------------------------------------------
// Epoch audio sequencing (state-media-v1 section 2)
/// One audio stream's chunk sequence within one epoch.
///
/// The contract's three sentences about sequencing are all here: `firstSample` identifies the
/// sample position relative to the episode's configured audio origin; crash restore preserves
/// that position under a new epoch and the first chunk marks `discontinuity`; within an epoch
/// chunks cannot overlap or go backwards.
///
/// A timeline belongs to one epoch. A restore starts a new one with
/// [`AudioTimeline::restored_at`], which is what makes the first chunk's discontinuity flag
/// checkable rather than advisory.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct AudioTimeline {
stream_id: String,
start_sample: u64,
restored: bool,
next_sample: u64,
accepted: u64,
}
impl AudioTimeline {
/// A fresh episode: the first chunk starts at the configured audio origin. Its
/// discontinuity flag is free, because a reset or a recovery establishes a fresh timeline
/// and does publish a discontinuity.
pub fn fresh(descriptor: &AudioDescriptor, origin: u64) -> AudioTimeline {
AudioTimeline {
stream_id: descriptor.stream_id.clone(),
start_sample: origin,
restored: false,
next_sample: origin,
accepted: 0,
}
}
/// A new epoch after a crash restore: the sample position is preserved, and the first
/// chunk of this epoch must mark `discontinuity`.
pub fn restored_at(descriptor: &AudioDescriptor, sample: u64) -> AudioTimeline {
AudioTimeline {
stream_id: descriptor.stream_id.clone(),
start_sample: sample,
restored: true,
next_sample: sample,
accepted: 0,
}
}
/// Where the next chunk may start. A chunk starting earlier overlaps or goes backwards.
pub fn next_sample(&self) -> u64 {
self.next_sample
}
/// How many chunks this epoch has accepted.
pub fn accepted(&self) -> u64 {
self.accepted
}
/// Validates one chunk's shape and its place in the sequence, then advances the timeline.
///
/// A rejected chunk does not advance anything, so a caller that fails its step does not
/// leave the timeline believing the chunk was played.
pub fn accept(&mut self, chunk: &AudioRef, descriptor: &AudioDescriptor) -> Result<()> {
if chunk.stream_id != self.stream_id {
return err(format!(
"AudioTimeline {}: chunk names stream {:?}",
self.stream_id, chunk.stream_id
));
}
chunk.validate_against(descriptor)?;
if self.accepted == 0 {
if chunk.first_sample != self.start_sample {
return err(format!(
"AudioTimeline {}: the first chunk of this epoch must start at sample {}, not {}",
self.stream_id, self.start_sample, chunk.first_sample
));
}
// Only one direction is stated: the first chunk after a restore marks the
// discontinuity. A fresh epoch's first chunk may mark one too -- sections 6 and 7
// have recovery and episode reset publishing a discontinuity on a fresh timeline --
// so the flag is required after a restore and left free at an origin.
if self.restored && !chunk.discontinuity {
return err(format!(
"AudioTimeline {}: the first chunk after a restore marks discontinuity",
self.stream_id
));
}
} else {
if chunk.first_sample < self.next_sample {
return err(format!(
"AudioTimeline {}: firstSample {} overlaps or goes backwards; the previous chunk ends at {}",
self.stream_id, chunk.first_sample, self.next_sample
));
}
// Derived from the sentence above: within an epoch the only discontinuity a chunk
// can carry is a gap it actually skipped. A chunk that continues the previous one
// exactly is continuous by construction.
if chunk.discontinuity && chunk.first_sample == self.next_sample {
return err(format!(
"AudioTimeline {}: a chunk continuing the previous one is not a discontinuity",
self.stream_id
));
}
}
self.next_sample = chunk
.first_sample
.checked_add(chunk.sample_frames)
.ok_or_else(|| {
crate::scalar::wire_err(format!(
"AudioTimeline {}: firstSample + sampleFrames overflows U64",
self.stream_id
))
})?;
self.accepted += 1;
Ok(())
}
}
// ---------------------------------------------------------------------------------------------
// Persistent assets against transient artifacts (state-media-v1 sections 1 and 3)
/// Checks that a transient artifact carries the bytes of an installed asset.
///
/// Persistent [`AssetRef`](crate::workers::AssetRef) and transient
/// [`ArtifactRef`] are different identities and never convert into one another: an asset names
/// installed release content in a preprovisioned registry, while an artifact names live bytes
/// in one store incarnation and resolves only through an owned handle. Importing an asset
/// produces a **new** artifact identity, which is why this checks content rather than identity.
///
/// The digest is mandatory here: state-media-v1 section 1 makes content digests optional on
/// transient live frames and mandatory on persistent asset import.
pub fn check_imported_asset(
asset: &crate::workers::AssetRef,
imported: &ArtifactRef,
) -> Result<()> {
asset.validate()?;
if imported.byte_length != asset.byte_length {
return err(format!(
"imported asset {}: the artifact is {} bytes, the asset is {}",
asset.id, imported.byte_length, asset.byte_length
));
}
match &imported.digest {
Some(d) if *d == asset.digest => Ok(()),
Some(_) => err(format!(
"imported asset {}: the artifact's digest is not the asset's content",
asset.id
)),
None => err(format!(
"imported asset {}: a persistent asset import must carry a content digest",
asset.id
)),
}
}
/// Reads a bounded, unique-by-`viewId` list of view refs.
pub fn view_list(f: &mut Fields<'_>, key: &'static str) -> Result<Vec<ViewRef>> {
let views = list(f, key, 0, MAX_VIEWS, ViewRef::from_json)?;

View file

@ -239,6 +239,11 @@ pub const LIMITS: &[LimitSchema] = &[
value: crate::workers::MAX_CAPABILITIES as u64,
source: "crate",
},
LimitSchema {
name: "maxWorkerThreads",
value: crate::workers::MAX_WORKER_THREADS,
source: "workers-v1 2",
},
LimitSchema {
name: "maxSupportedMajors",
value: crate::workers::MAX_SUPPORTED_MAJORS as u64,
@ -651,8 +656,8 @@ pub const SCHEMAS: &[TypeSchema] = &[
),
req(
"limits",
"{maxAgents:int,maxPorts:int}",
"1..=4 agents and 1..=4 ports",
"{maxAgents:int,maxPorts:int,workerThreads:int}",
"1..=4 agents, 1..=4 ports, and the launcher allocation this worker runs in",
),
],
},

View file

@ -33,6 +33,9 @@ pub const MAX_ACKNOWLEDGE: usize = 16;
pub const MAX_ENGINE_FRAME_LEN: usize = 64;
/// Negotiated capability ids. Not a stated bound; recorded in the schema set.
pub const MAX_CAPABILITIES: usize = 32;
/// The largest `workerThreads` a launcher may allocate to one worker (`workers-v1` 2).
pub const MAX_WORKER_THREADS: u64 = 4_096;
/// Supported majors in Worker.Hello. Not a stated bound; recorded in the schema set.
pub const MAX_SUPPORTED_MAJORS: usize = 8;
/// Domain error messages are <=512 code points (ipc-v1 section 7).
@ -651,7 +654,7 @@ impl DomainType for AgentInitializeParams {
let seed = i32_field(&mut f, "seed")?;
let initial_input = SensoryInput::from_json(f.value("initialInput")?)?;
let initial_decision_context = TypedValue::from_json(f.value("initialDecisionContext")?)?;
let worker_threads = f.int("workerThreads", 1, 4_096)?;
let worker_threads = f.int("workerThreads", 1, MAX_WORKER_THREADS)?;
f.finish()?;
let p = AgentInitializeParams {
agent_id,
@ -1958,6 +1961,13 @@ pub struct HelloResult {
pub capabilities: Vec<String>,
pub max_agents: u64,
pub max_ports: u64,
/// The thread allocation this worker was launched within.
///
/// `workers-v1` bounds `Agent.Initialize`'s `workerThreads` by "within launcher
/// allocation" and, before the 2026-09-22 amendment, named no wire on which a caller could
/// learn it. This is that wire: the worker reports what its launcher gave it, and a caller
/// that meant to ask for more finds out here rather than after the model exists.
pub worker_threads: u64,
}
impl HelloResult {
@ -1990,13 +2000,14 @@ impl DomainType for HelloResult {
let build_digest = f.string("buildDigest")?.to_owned();
let contract_digest = f.string("contractDigest")?.to_owned();
let capabilities = id_list(&mut f, "capabilities", 0, MAX_CAPABILITIES)?;
let (max_agents, max_ports) = {
let (max_agents, max_ports, worker_threads) = {
let v = f.value("limits")?;
let mut l = Fields::new(v, "HelloResult.limits")?;
let max_agents = l.int("maxAgents", 1, MAX_AGENTS as u64)?;
let max_ports = l.int("maxPorts", 1, MAX_PORTS as u64)?;
let worker_threads = l.int("workerThreads", 1, MAX_WORKER_THREADS)?;
l.finish()?;
(max_agents, max_ports)
(max_agents, max_ports, worker_threads)
};
f.finish()?;
let r = HelloResult {
@ -2008,6 +2019,7 @@ impl DomainType for HelloResult {
capabilities,
max_agents,
max_ports,
worker_threads,
};
r.validate()?;
Ok(r)
@ -2031,6 +2043,7 @@ impl DomainType for HelloResult {
obj(vec![
("maxAgents", Value::from(self.max_agents)),
("maxPorts", Value::from(self.max_ports)),
("workerThreads", Value::from(self.worker_threads)),
]),
),
])

View file

@ -0,0 +1,395 @@
//! MEDIA-01 shape rules: every sentence of `state-media-v1` section 2 that a descriptor, a
//! reference or a chunk sequence can be checked against on its own.
//!
//! These are the contract-level halves of the slice's acceptance bullets -- bad strides, bad
//! lengths, bad producing times and the audio rules -- and the type-level distinction between
//! a persistent `AssetRef` and a transient `ArtifactRef`.
use fly_session_types::ArtifactRef;
use fly_session_types::media::{
AudioDescriptor, AudioRef, AudioTimeline, ViewDescriptor, ViewRef, check_imported_asset,
require_finite_samples,
};
use fly_session_types::scalar::DomainType;
use fly_session_types::workers::AssetRef;
fn artifact(byte_length: u64, content_type: &str) -> ArtifactRef {
ArtifactRef {
store_id: "store-a".into(),
artifact_id: "art-1".into(),
generation: 1,
byte_length,
content_type: content_type.into(),
digest: None,
}
}
fn view_descriptor(width: u64, height: u64, delay: u64) -> ViewDescriptor {
ViewDescriptor {
view_id: "arena".into(),
width,
height,
row_stride: width * 4,
pixel_aspect_numerator: 1,
pixel_aspect_denominator: 1,
observation_delay_steps: delay,
}
}
fn view_ref(descriptor: &ViewDescriptor, produced_step: u64, bytes: u64) -> ViewRef {
ViewRef {
view_id: descriptor.view_id.clone(),
produced_step,
pixels: artifact(bytes, "image/x-rgba8"),
}
}
fn audio_descriptor(sample_rate: u64, channels: u64) -> AudioDescriptor {
AudioDescriptor {
stream_id: "arena".into(),
sample_rate,
channels,
}
}
fn audio_ref(
descriptor: &AudioDescriptor,
first_sample: u64,
frames: u64,
discontinuity: bool,
) -> AudioRef {
AudioRef {
stream_id: descriptor.stream_id.clone(),
first_sample,
sample_frames: frames,
samples: artifact(frames * descriptor.channels * 4, "audio/x-f32le"),
discontinuity,
}
}
// -----------------------------------------------------------------------------------------
// Bad strides
/// `rowStride` is exactly `4 x width`; v1 has no padded rows.
#[test]
fn a_padded_row_stride_is_refused() {
let good = view_descriptor(32, 24, 0);
good.validate().expect("4 x width is the only stride");
let mut padded = good.clone();
padded.row_stride = 32 * 4 + 16;
padded.validate().expect_err("a padded row is not readable in v1");
let mut narrow = good.clone();
narrow.row_stride = 32 * 3;
narrow.validate().expect_err("a stride under 4 x width is refused");
// The same rule through the wire form, where a hand-written descriptor arrives.
let mut json = good.to_json();
json["rowStride"] = serde_json::json!(32 * 4 + 4);
ViewDescriptor::from_json(&json).expect_err("a padded stride is refused when read");
}
/// Dimensions are integers 1..=4096, and `pixelAspect` parts positive integers <=65535.
#[test]
fn dimensions_pixel_aspect_and_delay_have_stated_bounds() {
for (width, height) in [(0, 24), (32, 0), (4097, 24), (32, 4097)] {
let mut d = view_descriptor(32, 24, 0);
d.width = width;
d.height = height;
d.row_stride = width.max(1) * 4;
d.validate().expect_err("dimensions are 1..=4096");
}
view_descriptor(1, 1, 0).validate().expect("1x1 is inside the bounds");
view_descriptor(4096, 4096, 0)
.validate()
.expect("4096x4096 is inside the bounds");
for (numerator, denominator) in [(0, 1), (1, 0), (65_536, 1), (1, 65_536)] {
let mut d = view_descriptor(32, 24, 0);
d.pixel_aspect_numerator = numerator;
d.pixel_aspect_denominator = denominator;
d.validate().expect_err("pixelAspect parts are 1..=65535");
}
let mut d = view_descriptor(32, 24, 9);
d.validate().expect_err("observationDelaySteps is 0..=8");
d.observation_delay_steps = 8;
d.validate().expect("eight steps of delay are allowed");
}
/// Only top-left RGBA8 exists in v1; another format is a media-schema change.
#[test]
fn only_rgba8_is_a_readable_view_format() {
let descriptor = view_descriptor(32, 24, 0);
let mut json = descriptor.to_json();
json["format"] = serde_json::json!("rgb8");
ViewDescriptor::from_json(&json).expect_err("rgb8 is not a v1 format");
json["format"] = serde_json::json!("rgba8");
ViewDescriptor::from_json(&json).expect("rgba8 is the v1 format");
}
// -----------------------------------------------------------------------------------------
// Bad lengths
/// A frame's artifact is exactly `rowStride x height` bytes.
#[test]
fn a_frame_whose_length_is_not_stride_times_height_is_refused() {
let descriptor = view_descriptor(32, 24, 0);
let exact = descriptor.frame_bytes();
assert_eq!(exact, 32 * 4 * 24);
view_ref(&descriptor, 7, exact)
.validate_against(&descriptor, None)
.expect("the exact frame length is accepted");
for wrong in [exact - 1, exact + 1, exact - 32 * 4, exact * 2] {
view_ref(&descriptor, 7, wrong)
.validate_against(&descriptor, None)
.expect_err("only rowStride x height is the frame length");
}
}
// -----------------------------------------------------------------------------------------
// Bad producing times
/// A required sensory view is produced at exactly `max(0, boundary - observationDelaySteps)`.
#[test]
fn a_view_produced_at_the_wrong_boundary_is_refused() {
let descriptor = view_descriptor(32, 24, 2);
let bytes = descriptor.frame_bytes();
assert_eq!(descriptor.required_produced_step(10), 8);
view_ref(&descriptor, 8, bytes)
.validate_against(&descriptor, Some(10))
.expect("the declared delay is exactly two steps");
// One frame later than the delay allows, and one frame older: both are step failures,
// not an arbitrary latest frame.
view_ref(&descriptor, 9, bytes)
.validate_against(&descriptor, Some(10))
.expect_err("an under-delayed frame is refused");
view_ref(&descriptor, 7, bytes)
.validate_against(&descriptor, Some(10))
.expect_err("an extra-delayed frame is refused");
}
/// Bootstrap may repeat `O[0]` until the declared pipeline delay fills, and only until then.
#[test]
fn bootstrap_repeats_the_first_frame_until_the_pipeline_delay_fills() {
let descriptor = view_descriptor(32, 24, 3);
let bytes = descriptor.frame_bytes();
// Boundaries 0..=3 all require the frame produced at 0.
for boundary in 0..=3 {
assert_eq!(descriptor.required_produced_step(boundary), 0);
view_ref(&descriptor, 0, bytes)
.validate_against(&descriptor, Some(boundary))
.expect("O[0] repeats while the pipeline fills");
}
// From boundary 4 the pipeline is full and O[0] is a stale frame.
assert_eq!(descriptor.required_produced_step(4), 1);
view_ref(&descriptor, 0, bytes)
.validate_against(&descriptor, Some(4))
.expect_err("the repetition ends when the delay is filled");
view_ref(&descriptor, 1, bytes)
.validate_against(&descriptor, Some(4))
.expect("boundary 4 requires the frame produced at 1");
}
// -----------------------------------------------------------------------------------------
// Audio shapes
/// sampleRate is 8000..=192000, channels 1..=8 and sampleFrames 0..=192000.
#[test]
fn audio_rate_channels_and_frames_have_stated_bounds() {
audio_descriptor(8_000, 1).validate().expect("8 kHz mono is the floor");
audio_descriptor(192_000, 8).validate().expect("192 kHz 8ch is the ceiling");
audio_descriptor(7_999, 2).validate().expect_err("under 8 kHz is refused");
audio_descriptor(192_001, 2).validate().expect_err("over 192 kHz is refused");
audio_descriptor(48_000, 0).validate().expect_err("zero channels are refused");
audio_descriptor(48_000, 9).validate().expect_err("nine channels are refused");
let descriptor = audio_descriptor(48_000, 2);
let mut chunk = audio_ref(&descriptor, 0, 192_000, false);
chunk.validate().expect("192000 frames is the per-chunk ceiling");
chunk.sample_frames = 192_001;
chunk.validate().expect_err("over 192000 frames in one chunk is refused");
}
/// A chunk's artifact is exactly `sampleFrames x channels x 4` bytes.
#[test]
fn an_audio_chunk_length_is_frames_times_channels_times_four() {
let descriptor = audio_descriptor(48_000, 2);
let chunk = audio_ref(&descriptor, 0, 800, false);
assert_eq!(chunk.samples.byte_length, 800 * 2 * 4);
chunk
.validate_against(&descriptor)
.expect("the exact chunk length is accepted");
let mut wrong = chunk.clone();
wrong.samples = artifact(800 * 2 * 4 - 4, "audio/x-f32le");
wrong
.validate_against(&descriptor)
.expect_err("a short chunk is refused");
// The same frames at another channel count are a different number of bytes.
let mono = audio_descriptor(48_000, 1);
let mut wrong_channels = chunk.clone();
wrong_channels.stream_id = mono.stream_id.clone();
wrong_channels
.validate_against(&mono)
.expect_err("stereo bytes are not a mono chunk");
}
/// Samples are finite f32.
#[test]
fn a_non_finite_sample_is_refused() {
let mut bytes = Vec::new();
for value in [0.0f32, -0.5, 0.75, 1.0] {
bytes.extend_from_slice(&value.to_le_bytes());
}
require_finite_samples(&bytes).expect("finite samples are accepted");
for bad in [f32::NAN, f32::INFINITY, f32::NEG_INFINITY] {
let mut broken = bytes.clone();
broken.extend_from_slice(&bad.to_le_bytes());
require_finite_samples(&broken).expect_err("a non-finite sample is refused");
}
require_finite_samples(&bytes[..5]).expect_err("a partial sample is refused");
}
/// Within an epoch, chunks cannot overlap or go backwards.
#[test]
fn chunks_cannot_overlap_or_go_backwards_within_an_epoch() {
let descriptor = audio_descriptor(48_000, 2);
let mut timeline = AudioTimeline::fresh(&descriptor, 0);
timeline
.accept(&audio_ref(&descriptor, 0, 800, false), &descriptor)
.expect("the first chunk starts at the origin");
assert_eq!(timeline.next_sample(), 800);
timeline
.accept(&audio_ref(&descriptor, 800, 800, false), &descriptor)
.expect("the second chunk continues the first");
assert_eq!(timeline.next_sample(), 1_600);
let mut overlapping = timeline.clone();
overlapping
.accept(&audio_ref(&descriptor, 1_599, 800, false), &descriptor)
.expect_err("a chunk that starts inside the previous one is refused");
let mut backwards = timeline.clone();
backwards
.accept(&audio_ref(&descriptor, 0, 800, false), &descriptor)
.expect_err("a chunk that goes backwards is refused");
// A rejected chunk leaves the timeline where it was.
assert_eq!(overlapping.next_sample(), 1_600);
assert_eq!(overlapping.accepted(), 2);
// A gap is forward, so it is allowed; it is the one place a later chunk may mark a
// discontinuity.
timeline
.accept(&audio_ref(&descriptor, 2_000, 800, true), &descriptor)
.expect("a forward gap is not an overlap");
timeline
.accept(&audio_ref(&descriptor, 2_800, 800, true), &descriptor)
.expect_err("a chunk that continues the previous one is not a discontinuity");
}
/// Crash restore preserves the sample position under a new epoch, and its first chunk marks
/// the discontinuity.
#[test]
fn the_first_chunk_after_a_restore_marks_discontinuity() {
let descriptor = audio_descriptor(48_000, 2);
// The requirement is one-directional. A fresh epoch's first chunk may mark a
// discontinuity -- a recovery or an episode reset establishes a fresh timeline and
// publishes one -- so both flags are accepted at an origin.
let mut reset = AudioTimeline::fresh(&descriptor, 0);
reset
.accept(&audio_ref(&descriptor, 0, 800, true), &descriptor)
.expect("a reset episode's first chunk may mark the discontinuity it published");
let mut fresh = AudioTimeline::fresh(&descriptor, 0);
fresh
.accept(&audio_ref(&descriptor, 0, 800, false), &descriptor)
.expect("the episode's first chunk continues nothing");
// The restored epoch resumes at the preserved position.
let mut restored = AudioTimeline::restored_at(&descriptor, 800);
restored
.accept(&audio_ref(&descriptor, 800, 800, false), &descriptor)
.expect_err("the first chunk after a restore marks discontinuity");
let mut restored = AudioTimeline::restored_at(&descriptor, 800);
restored
.accept(&audio_ref(&descriptor, 0, 800, true), &descriptor)
.expect_err("the restored position is preserved, not reset");
restored
.accept(&audio_ref(&descriptor, 800, 800, true), &descriptor)
.expect("the restored epoch resumes at its preserved sample position");
assert_eq!(restored.next_sample(), 1_600);
restored
.accept(&audio_ref(&descriptor, 1_600, 800, false), &descriptor)
.expect("the chunks after it are ordinary");
}
// -----------------------------------------------------------------------------------------
// Persistent assets against transient artifacts
/// An `AssetRef` and an `ArtifactRef` are different identities with different fields, and
/// neither is readable as the other.
#[test]
fn an_asset_ref_is_not_a_transient_artifact_ref() {
let asset = AssetRef {
id: "counter-arena-backend".into(),
digest: "a".repeat(64),
byte_length: 24,
format: "fly-config-v1".into(),
};
let imported = ArtifactRef {
store_id: "store-a".into(),
artifact_id: "art-9".into(),
generation: 1,
byte_length: 24,
content_type: "application/octet-stream".into(),
digest: Some("a".repeat(64)),
};
// Identity fields do not overlap: the asset has no store and the artifact has no format.
let asset_keys: Vec<String> = asset
.to_json()
.as_object()
.expect("an object")
.keys()
.cloned()
.collect();
let artifact_keys: Vec<String> = imported
.to_json()
.as_object()
.expect("an object")
.keys()
.cloned()
.collect();
assert_eq!(asset_keys, vec!["id", "digest", "byteLength", "format"]);
assert!(artifact_keys.contains(&"storeId".to_owned()));
assert!(artifact_keys.contains(&"artifactId".to_owned()));
assert!(!artifact_keys.contains(&"format".to_owned()));
assert!(!asset_keys.contains(&"storeId".to_owned()));
// Neither reads as the other: a view's pixels are an artifact, never an asset.
ArtifactRef::from_json(&asset.to_json()).expect_err("an asset is not an artifact reference");
AssetRef::from_json(&imported.to_json()).expect_err("an artifact is not an asset reference");
let pixels_as_asset = serde_json::json!({
"viewId": "arena",
"producedStep": "0",
"pixels": asset.to_json(),
});
ViewRef::from_json(&pixels_as_asset).expect_err("a view's pixels cannot be an asset");
// Importing an asset is a content check, not an identity conversion.
check_imported_asset(&asset, &imported).expect("the import carries the asset's content");
let mut no_digest = imported.clone();
no_digest.digest = None;
check_imported_asset(&asset, &no_digest)
.expect_err("a persistent asset import must carry a content digest");
let mut other_content = imported.clone();
other_content.digest = Some("b".repeat(64));
check_imported_asset(&asset, &other_content).expect_err("another digest is another content");
let mut short = imported.clone();
short.byte_length = 23;
check_imported_asset(&asset, &short).expect_err("the import must be the asset's length");
}

View file

@ -11,6 +11,13 @@ description = "The lockstep session coordinator, its phase machine and a synthet
name = "fly_session"
path = "src/lib.rs"
# One binary, one role per subcommand. `implementation.md` section 2 allows worker
# executables to be subcommands of one binary rather than separate crates, and the launcher
# starts this one with `agent` or `environment` for a participant in its own process.
[[bin]]
name = "fly-session"
path = "src/bin/fly-session.rs"
[dependencies]
# The domain contract (scalars, payloads, canonical digests, the trace format) and the bus.
# Everything else this crate needs is std or Tokio.
@ -18,7 +25,9 @@ fly-session-types = { path = "../fly-session-types" }
flybus = { path = "../flybus" }
serde_json = { workspace = true }
tokio = { version = "1", features = ["rt", "sync", "time", "macros"] }
# `rt-multi-thread` is not only for the tests: a worker process and a dedicated-thread
# worker each build their own runtime sized to the launcher's thread allocation.
tokio = { version = "1", features = ["rt", "rt-multi-thread", "sync", "time", "macros", "io-util"] }
[dev-dependencies]
tempfile = "3"

View file

@ -3,10 +3,12 @@
The lockstep session coordinator, its phase machine and a synthetic composition over
[`flybus`](../flybus).
This crate is the SESSION-01 slice of the session-framework implementation guide: the
sequential transaction of `step-v1`, driven over the Flybus router, with small fake workers
standing in for a brain and an emulator. It contains no public controller API, no implicit
best-effort retry, no real emulator and no real brain.
This crate is the SESSION-01 and SESSION-02 slices of the session-framework implementation
guide: the transaction of `step-v1`, driven over the Flybus router, with small fake workers
standing in for a brain and an emulator, run either in the coordinator's process, on dedicated
threads, or as one agent process per fly and one environment process under a launcher. It
contains no public controller API, no implicit best-effort retry, no real emulator and no real
brain.
The domain scalars, method payloads, their validation, the canonical digests and the trace
format all come from [`fly-session-types`](../fly-session-types), the CONTRACT-01 crate. This
@ -38,7 +40,54 @@ Ready(k) ─ Prepare all agents concurrently ───────────
| `task` | The task and executor traits, the deterministic counter task, the identity executor |
| `rpc` | Domain calls: `req-<U64>` serials, incarnation pinning, the retry rule |
| `coordinator` | The transaction, the trace, the failure rules and the publication boundary |
| `harness` | The runnable composition: router, two agents, one arena, one coordinator |
| `launcher` | The supervisor: thread budget, identities, start, health check, reap |
| `metrics` | Latency percentiles and the machine's core and memory counters |
| `measure` | The execution-mode comparison of the guide's section 5 |
| `cli` | The binary's subcommands: `agent`, `environment`, `measure` |
| `harness` | The runnable composition: router, the flies, one arena, one coordinator |
## Execution modes and the launcher
A participant runs in one of three places, and the same composition code starts it in any of
them. The separate-process mode is the SESSION-02 subject; the other two are what it is
compared against.
| Mode | Where each participant runs | Transport |
| --- | --- | --- |
| `InProcess` | A task on the coordinator's runtime | in-memory or Unix socket |
| `Thread` | Its own OS thread, with its own runtime | Unix socket |
| `Process` | Its own process: one per fly, one for the world | Unix socket |
The launcher is the configured supervisor. It owns four things:
- **The thread budget.** A total allocation, one slice of it reserved for the coordinator and
its router, and one allocation per participant. A request the total cannot cover is refused
as `BUSY` before anything starts. `Agent.Initialize` carries exactly the allocation the
launcher handed out, and an agent refuses an Initialize asking for more than its own, which
is what `workers-v1` means by "within launcher allocation". The allocation is on the wire,
not only in the launcher's own record: `HelloResult.limits.workerThreads` reports it, under
the dated 2026-09-22 amendment to `workers-v1` section 2 that this slice added, so a
coordinator that is not also its own launcher can read the bound it has to respect.
- **Identity.** The bus client id, the service name, the worker id and an agent's port binding
are launcher configuration. The launcher says `Worker.Hello` with the identity it configured
and refuses anything that answers as another worker, role, incarnation or thread allocation
-- before the coordinator has pinned a registration. The registration the coordinator pins
is the one that hello returned, never one that was assumed.
- **Health.** `Worker.Status` on the supervisor's own monotonic clock, with the `ipc-v1`
section 6 prototype budgets: probe at two seconds, fail at ten, a separate budget for boot.
A status answer never waits for a mutation, so a busy participant is still a healthy one.
- **Reaping.** `Worker.Shutdown` is the request and the operating system is the guarantee. A
participant that does not stop inside the budget is terminated, and the supervisor reports
which of the two happened. A launcher that is dropped takes its children with it.
A separate-process participant is a subcommand of this crate's one binary, which is what
`implementation.md` section 2 allows instead of separate worker crates:
```sh
fly-session agent --socket S --store-root D --client-id C --service N --threads T ...
fly-session environment --socket S --store-root D --client-id C --service N --threads T ...
fly-session measure --steps 300 --agents 1,2,4
```
## What it implements
@ -62,6 +111,31 @@ Ready(k) ─ Prepare all agents concurrently ───────────
- **The failure rules.** A partial commit fails the epoch; an uncertain Advance is resolved
against its original domain request id and never becomes a second batch; a worker
incarnation change invalidates the epoch.
- **A failure stops the epoch rather than neutralising a player.** Every failure carries the
participant it is attributed to, and failing fences the session: the committed boundary
stops moving, the artifact handles are dropped, and no further transition or publication is
allowed. Lifting the fence is a coherent group restore, which is STATE-01's.
- **The `ipc-v1` section 6 procedure, on the path that reaches it.** A call that goes two
seconds without a terminal reply is *uncertain*, not failed. The coordinator then queries
the same operation -- a fresh bus call carrying the original domain request id and body,
pinned to the same incarnation, with its retained attachments -- absorbing `IN_PROGRESS`
while the original is still running. Only when that ends without a definite answer, or the
incarnation is gone, or the retained result expired, is the epoch failed. A merely slow
participant therefore finishes its step, and `step-v1` section 7's "query/retransmit same
request to same incarnation; never new batch" is the same code path for a slow Advance.
The procedure has two explicit bounds, and they do not mean the same thing. **`resolve`, 8
seconds, is the working limit**: two to notice plus eight to resolve is section 6's ten
seconds without progress. **`resolve_attempts`, 8192, is a guard**, not the limit -- the
procedure pauses 2 ms between attempts, so the guard is over sixteen seconds of pauses
alone, twice the budget, and an attempt whose call expires costs a whole probe on top. At
these values the budget is always what fires. Which one did is recorded in
`Coordinator::last_resolution` and named in the failure's own message, so an exhausted
resolution never has to be explained by arithmetic.
- **A bounded diagnosed outcome.** Those budgets are the coordinator's own, on its own clock,
so a participant that dies or stops answering produces a typed failure naming it rather than
a hang. An expired deadline is `unknown`, never `none`: a caller-side timeout is not
evidence that nothing was mutated.
- **Domain deduplication over bus calls.** Same key, request and body replays its cached
reply with fresh delivery ownership over retained artifacts; a changed body is `CONFLICT`; a
duplicate of a running operation is `IN_PROGRESS` for that bus call while the original
@ -125,23 +199,66 @@ harness.shutdown().await;
- **No state methods.** `State.Capture`, `State.StageRestore` and `State.ActivateRestore` are
STATE-01. The phase machine has their edges (`Capturing`, `Restoring`) and the workers do not
advertise them as implemented methods.
- **One process.** SESSION-01 runs every participant in one process over the same router.
SESSION-02 is the per-fly process split.
- **No audience input.** The admitted pre-step stimulation list exists and is always empty.
- **Pacing is coarse.** The pacing deadline rounds one step to whole nanoseconds for sleeping
only; simulation time stays rational and that rounding never re-enters the accumulator.
## Measurements
`fly-session measure` runs the same composition in each mode at one, two and four agents and
reports the thread allocation, the RPC and critical-path percentiles, the memory peaks and the
router's owner, collection and queue counters. **These are local synthetic timings on one
machine and no host capacity claim follows from any of them**; they exist so the three modes
can be compared with each other. Pacing is off for the run, so the samples are work rather
than sleep, and the run report carries the full table.
Every row runs in a child process of its own. A peak-memory figure is a high-water mark that
never falls, so rows sharing one process would each report where that process had already
been: the column would sort itself by row position rather than by mode, and the mode ranking
would reverse when the rows were reordered. One child per row is what makes the number belong
to the row.
What the numbers said on a four-core development box, at 300 transitions per row:
- A process boundary costs about a fifth of the critical path at the median. Two agents: 10.0
ms p50 in-process, 12.2 ms on threads, 12.1 ms across processes, with p99 at 21.4 / 19.4 /
21.4 ms. The dedicated-thread and separate-process variants are within noise of each other,
so what is being paid for is leaving the coordinator's runtime, not crossing a socket.
- `Worker.Status` -- an RPC answered from a cell with no domain work behind it -- is the
router and transport floor: 0.60 / 0.89 / 1.17 ms p50 for the three modes at two agents.
- Four agents needs six threads, which that box does not have, and every mode's tail widens
together. That is the budget being honest about oversubscription, not a property of the
process split.
- Memory is where the split really shows, but not where the first version of this note said.
The coordinator's own peak is roughly the same in all three modes and is *lowest* in process
mode -- 8.9 / 9.2 / 7.9 MiB at one agent -- because the workers are no longer inside it.
What the split costs is the children: about 5.7 MiB per participant process, so the whole
composition is roughly 9 MiB on threads against 39 MiB across processes at four agents.
- Ownership, collection and queues stayed bounded in every mode and at every agent count: at
most 15 live owners, 11 artifact roots and one queued entry per agent, with the store at
rest holding two sealed frames and 128 bytes. Of 311 frames observed, 309 were collected --
the two still owned are the current and previous boundary. The frame count is taken from the
behaviour trace's observation boundaries rather than calculated from the step count, so a
backend sealing two frames per boundary would show up instead of being hidden.
## Tests
```text
cargo test -p fly-session # unit + both integration suites
cargo test -p fly-session # unit + all three integration suites
cargo run -p fly-session --example session # the runnable synthetic session
cargo build -p fly-session --bin fly-session # the worker binary the launcher starts
cargo run -p fly-session --example processes # the same session in all three modes
```
Every integration test runs over both transports, through the same router code: all but one
are generated twice by `both_transports!`, and
`sequential_concurrent_and_reversed_orders_agree` walks both transports inside one test
because it compares their behaviour traces against each other.
The three integration suites do not all run over both transports, and cannot:
- `tests/session.rs` and `tests/failures.rs` are in-process compositions and run over both,
through the same router code. All but one test in them is generated twice by
`both_transports!`; `sequential_concurrent_and_reversed_orders_agree` walks both transports
inside one test, because it compares their behaviour traces against each other.
- `tests/processes.rs` runs over the Unix socket only, in all three execution modes. A
participant in a process of its own has no in-memory transport to reach the router by, so
the mode is the axis that suite varies and the transport is fixed.
- `tests/session.rs`: one world advance per complete batch; every agent Prepared before the
advance; one task evaluation per transition; every agent committed before the next Prepare or
@ -150,6 +267,14 @@ because it compares their behaviour traces against each other.
transition that just ended; a terminal episode pausing at its own boundary; `Worker.Status`
during a session; and sequential, concurrent and reversed dispatch producing one behaviour
trace.
- `tests/processes.rs`: the SESSION-02 acceptance bullets, each generated once per execution
mode -- a slow participant resolved rather than failed, a delayed one-agent result holding
the world, a worker or helper death with a
bounded diagnosed outcome, an uncertain Advance that creates no second batch, a partial
Commit that permits no next-step play, supervision and identity, and the launcher thread
allocation -- plus the sequential/reversed/parallel trace comparison across all three modes
and the two process-mode section 4 rows: a router restart during a world advance, and an old
worker's reply after a restart.
- `tests/failures.rs`: a duplicate Prepare after a lost reply; a duplicate Commit; the same
batch with altered controls; a lost Advance result; a cached artifact consumed by its first
caller; one Commit failing after another succeeded; a replaced registration; a reply from

View file

@ -0,0 +1,75 @@
//! The same synthetic session in all three execution modes, printing the one behaviour trace
//! they agree on.
//!
//! ```sh
//! cargo run -p fly-session --example processes
//! ```
//!
//! The separate-process run starts one agent process per fly and one environment process
//! through this crate's own binary, so it needs that binary built:
//!
//! ```sh
//! cargo build -p fly-session --bin fly-session
//! ```
use fly_session::harness::{ExecutionMode, HarnessConfig, SessionHarness, Via};
use fly_session::launcher::default_worker_program;
#[tokio::main(flavor = "multi_thread", worker_threads = 4)]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let program = default_worker_program();
println!("worker program: {}", program.display());
let mut agreed: Option<Vec<String>> = None;
for mode in ExecutionMode::all() {
let dir = tempfile::tempdir()?;
let config = HarnessConfig { mode, ..HarnessConfig::default() };
println!(
"\n=== {} : {} threads for {} participants plus the coordinator",
mode.label(),
config.budget()?.total(),
config.agents.len() + 1
);
let mut harness = SessionHarness::start(Via::Unix, dir.path(), config).await?;
harness.coordinator.bootstrap().await?;
let reports = harness.coordinator.run(3).await?;
for report in &reports {
println!(" committed boundary {}", report.boundary);
}
for (worker_id, status) in harness.launcher.health_check_all().await {
match status {
Ok(status) => println!(
" {worker_id}: {:?}, progress {}",
status.state, status.progress_counter
),
Err(e) => println!(" {worker_id}: unhealthy: {e}"),
}
}
let behaviour = harness.coordinator.trace.behavior();
match &agreed {
None => {
println!(" behaviour trace, {} transitions:", behaviour.len());
for line in &behaviour {
println!(" {line}");
}
agreed = Some(behaviour);
}
Some(first) => {
assert_eq!(
&behaviour, first,
"{} produced a different behaviour trace",
mode.label()
);
println!(" behaviour trace: identical to the first run");
}
}
let reaped = harness.launcher.reap_all(&fly_session::types::id("example")).await;
for (worker_id, outcome) in reaped {
println!(" reaped {worker_id}: {outcome:?}");
}
harness.shutdown().await;
drop(dir);
}
println!("\nall three execution modes produced one behaviour trace");
Ok(())
}

View file

@ -203,6 +203,14 @@ pub struct AgentConfig {
pub incarnation_id: Id,
pub tick_duration: RationalNs,
pub warmup_ticks: u64,
/// The thread allocation the launcher started this worker within. `workers-v1` requires
/// `Agent.Initialize`'s `workerThreads` to lie inside it.
pub worker_threads: usize,
/// Records every view this agent read, so a test can see which artifact reached it.
///
/// It is this process's log: an agent with a process of its own writes to its own copy,
/// which the supervisor cannot read. `SessionHarness::sensor_log` says so with `None`.
pub sensors: crate::media::SensorLog,
pub faults: AgentFaults,
}
@ -296,6 +304,15 @@ impl FakeAgentWorker {
format!("view {} is the wrong length", view.view_id),
));
}
// What this agent read, from the bytes it read: the artifact it was given and the
// digest of its content.
self.config.sensors.record(crate::media::SensedView {
boundary: input.boundary,
view_id: view.view_id.clone(),
artifact_id: artifact.reference().artifact_id.clone(),
produced_step: view.produced_step,
digest: digest_of_bytes(&bytes),
});
total += i64::from(bytes.first().copied().unwrap_or_default());
}
if let Some(structured) = &input.structured {
@ -369,6 +386,18 @@ impl FakeAgentWorker {
if params.worker_threads == 0 {
return Err(DomainError::invalid("workerThreads must be >= 1"));
}
// `workers-v1`: workerThreads is "within launcher allocation". This worker was started
// with that allocation, so a request for more than it is a capacity refusal made
// before the model is constructed, not a silent reduction to what is available.
if params.worker_threads > self.config.worker_threads as u64 {
return Err(DomainError::before(
ErrorCode::Busy,
format!(
"Agent.Initialize asks for {} worker threads; the launcher allocated {}",
params.worker_threads, self.config.worker_threads
),
));
}
params.initial_decision_context.validate().map_err(DomainError::invalid)?;
let available = FakeAgentWorker::available_actions(&params.initial_decision_context)?;
// Everything is validated before the model is constructed.
@ -635,6 +664,10 @@ impl WorkerEndpoint for FakeAgentWorker {
self.status.clone()
}
fn worker_threads(&self) -> u64 {
self.config.worker_threads as u64
}
fn methods(&self) -> Vec<&'static str> {
vec!["Agent.Initialize", "Agent.Prepare", "Agent.Commit"]
}

View file

@ -0,0 +1,8 @@
//! This crate's one binary. Every role it can take is a subcommand of it.
//!
//! `implementation.md` section 2: "Worker executables can be subcommands of one binary
//! initially; process boundaries do not require separate repos."
fn main() -> std::process::ExitCode {
fly_session::cli::main()
}

View file

@ -0,0 +1,337 @@
//! The subcommands of this crate's one binary.
//!
//! `implementation.md` section 2 allows worker executables to be subcommands of one binary
//! rather than separate crates, and that is what these are: `agent` and `environment` are the
//! two worker roles a launcher starts as separate processes, and `measure` runs the execution
//! modes against each other.
//!
//! ```text
//! fly-session agent --socket S --store-root D --client-id C --service N --threads T ...
//! fly-session environment --socket S --store-root D --client-id C --service N --threads T ...
//! fly-session measure [--steps N] [--agents 1,2,4] [--modes in-process,thread,process]
//! ```
//!
//! A worker process is told exactly which participant it is. It proves that identity in
//! `Worker.Hello`, so a process started under another one is refused by its own supervisor
//! before the coordinator has pinned anything.
use std::collections::BTreeMap;
use std::path::PathBuf;
use std::process::ExitCode;
use crate::agent::AgentFaults;
use crate::environment::EnvironmentFaults;
use crate::launcher::{
AgentLaunch, EnvironmentLaunch, ExecutionMode, Started, flags, serve_one,
};
use crate::types::*;
const USAGE: &str = "\
fly-session <command> [options]
agent serve one agent worker on a launcher-created endpoint
environment serve the environment worker on a launcher-created endpoint
measure compare the execution modes and print the measurement table
measure-row measure one row and print it as JSON (one child per row)
Worker options (agent and environment):
--socket PATH the launcher's endpoint for this participant
--store-root PATH the router's artifact store root
--client-id ID the configured bus client identity
--service NAME the one service name this worker registers
--threads N the launcher's thread allocation for this worker
--session ID the session this worker belongs to
--incarnation ID this worker's domain incarnation
agent: --agent ID --port ID --tick-numerator N --tick-denominator N
--warmup-ticks N [--prepare-delay-ms N] [--commit-delay-ms N]
[--fail-commit-at-step N]
environment: --worker ID --ports p1,p2 --step-numerator N --step-denominator N
[--advance-delay-ms N] [--omit-view-at-boundary N]
Measure options:
--steps N transitions per run (default 200)
--warmup-steps N transitions run before sampling starts (default 10)
--agents 1,2,4 agent counts to compare (default 1,2,4)
--modes LIST in-process, thread, process (default all three)
--worker-threads N within-agent worker threads (default 1)
measure-row options: --mode NAME --agents N, plus the measure options above. Each row runs in
a process of its own, so its memory peak is its own rather than the peak of the rows before
it.
";
/// The binary's entry point.
pub fn main() -> ExitCode {
let mut args = std::env::args_os().skip(1);
let Some(command) = args.next() else {
eprint!("{USAGE}");
return ExitCode::from(2);
};
let command = command.to_string_lossy().into_owned();
let rest: Vec<String> = args.map(|a| a.to_string_lossy().into_owned()).collect();
let result = match command.as_str() {
"agent" => Options::parse(&rest, &[flags::COMMON, flags::AGENT_ONLY])
.and_then(|o| serve(&command, &o)),
"environment" => Options::parse(&rest, &[flags::COMMON, flags::ENVIRONMENT_ONLY])
.and_then(|o| serve(&command, &o)),
"measure" => Options::parse(&rest, &[flags::MEASURE]).and_then(|o| measure(&o)),
"measure-row" => Options::parse(&rest, &[flags::MEASURE]).and_then(|o| measure_row(&o)),
"--help" | "-h" | "help" => {
print!("{USAGE}");
return ExitCode::SUCCESS;
}
other => Err(format!("unknown command {other:?}\n\n{USAGE}")),
};
match result {
Ok(()) => ExitCode::SUCCESS,
Err(e) => {
eprintln!("fly-session {command}: {e}");
ExitCode::FAILURE
}
}
}
/// `--flag value` options. The launcher builds this argv, so the grammar stays small.
#[derive(Debug, Default)]
struct Options(BTreeMap<String, String>);
impl Options {
/// Reads the options of one command, refusing any flag that command does not have.
///
/// `allowed` comes from [`crate::launcher::flags`], the same constants the launcher
/// writes the argv from. An unknown flag is an error naming it rather than a value that
/// is quietly ignored: a renamed option must fail the launch, not turn into a no-op that
/// no test notices.
fn parse(args: &[String], allowed: &[&[&str]]) -> Result<Options, String> {
let mut out = BTreeMap::new();
let mut iter = args.iter();
while let Some(flag) = iter.next() {
let Some(name) = flag.strip_prefix("--") else {
return Err(format!("expected an option, found {flag:?}"));
};
if !allowed.iter().any(|set| set.contains(&name)) {
return Err(format!("unknown option --{name} for this command"));
}
let value = iter
.next()
.ok_or_else(|| format!("option --{name} needs a value"))?;
if out.insert(name.to_owned(), value.clone()).is_some() {
return Err(format!("option --{name} was given twice"));
}
}
Ok(Options(out))
}
fn required(&self, name: &str) -> Result<&str, String> {
self.0
.get(name)
.map(String::as_str)
.ok_or_else(|| format!("option --{name} is required"))
}
fn optional(&self, name: &str) -> Option<&str> {
self.0.get(name).map(String::as_str)
}
fn id(&self, name: &str) -> Result<Id, String> {
parse_id(self.required(name)?).map_err(|e| format!("--{name}: {e}"))
}
fn u64(&self, name: &str, default: u64) -> Result<u64, String> {
match self.0.get(name) {
None => Ok(default),
Some(value) => value.parse().map_err(|_| format!("--{name}: {value:?} is not a number")),
}
}
fn opt_u64(&self, name: &str) -> Result<Option<u64>, String> {
match self.0.get(name) {
None => Ok(None),
Some(value) => value
.parse()
.map(Some)
.map_err(|_| format!("--{name}: {value:?} is not a number")),
}
}
fn usize(&self, name: &str, default: usize) -> Result<usize, String> {
Ok(self.u64(name, default as u64)? as usize)
}
fn path(&self, name: &str) -> Result<PathBuf, String> {
Ok(PathBuf::from(self.required(name)?))
}
fn rational(&self, numerator: &str, denominator: &str) -> Result<RationalNs, String> {
let n = self.u64(numerator, 0)?;
let d = self.u64(denominator, 1)?;
RationalNs::new(n, d).map_err(|e| format!("--{numerator}/--{denominator}: {}", e.0))
}
}
/// Serves one worker until `Worker.Shutdown`, then exits.
fn serve(role: &str, options: &Options) -> Result<(), String> {
let socket = options.path(flags::SOCKET)?;
let store_root = options.path(flags::STORE_ROOT)?;
let client_id = options.required(flags::CLIENT_ID)?.to_owned();
let service = options.required(flags::SERVICE)?.to_owned();
let threads = options.usize(flags::THREADS, 1)?;
if threads == 0 {
return Err("--threads must be at least 1".to_owned());
}
let session_id = options.id(flags::SESSION)?;
let incarnation_id = options.id(flags::INCARNATION)?;
let what = match role {
"agent" => Started::Agent(AgentLaunch {
session_id,
agent_id: options.id(flags::AGENT)?,
port_id: options.id(flags::PORT)?,
incarnation_id,
tick_duration: options.rational(flags::TICK_NUMERATOR, flags::TICK_DENOMINATOR)?,
warmup_ticks: options.u64(flags::WARMUP_TICKS, 0)?,
worker_threads: threads,
// This process's own log. The supervisor reads what crosses the bus, not this.
sensors: crate::media::SensorLog::new(),
faults: AgentFaults {
fail_commit_at_step: options.opt_u64(flags::FAIL_COMMIT_AT_STEP)?,
prepare_delay_ms: options.u64(flags::PREPARE_DELAY_MS, 0)?,
commit_delay_ms: options.u64(flags::COMMIT_DELAY_MS, 0)?,
},
client_id: client_id.clone(),
service: service.clone(),
}),
_ => Started::Environment(EnvironmentLaunch {
session_id,
worker_id: options.id(flags::WORKER)?,
incarnation_id,
step_duration: options.rational(flags::STEP_NUMERATOR, flags::STEP_DENOMINATOR)?,
ports: parse_ports(options.required(flags::PORTS)?)?,
worker_threads: threads,
observation_delay_steps: options.u64(flags::OBSERVATION_DELAY_STEPS, 0)?,
renders: crate::media::RenderCounter::new(),
faults: EnvironmentFaults {
advance_delay_ms: options.u64(flags::ADVANCE_DELAY_MS, 0)?,
omit_view_at_boundary: options.opt_u64(flags::OMIT_VIEW_AT_BOUNDARY)?,
stale_view_at_boundary: options.opt_u64(flags::STALE_VIEW_AT_BOUNDARY)?,
truncated_view_at_boundary: options
.opt_u64(flags::TRUNCATED_VIEW_AT_BOUNDARY)?,
omit_audio_at_boundary: options.opt_u64(flags::OMIT_AUDIO_AT_BOUNDARY)?,
overlapping_audio_at_boundary: options
.opt_u64(flags::OVERLAPPING_AUDIO_AT_BOUNDARY)?,
},
client_id: client_id.clone(),
service: service.clone(),
}),
};
let runtime = tokio::runtime::Builder::new_multi_thread()
.worker_threads(threads)
.enable_all()
.build()
.map_err(|e| format!("runtime: {e}"))?;
runtime.block_on(async move {
let handle = serve_one(&socket, &client_id, &service, &store_root, &what, threads).await?;
// The worker serves until its supervisor's Worker.Shutdown, which it answers before
// it stops. Exiting is then one event, not a race between a reply and a signal.
handle.join().await;
Ok(())
})
}
fn parse_ports(value: &str) -> Result<Vec<Id>, String> {
value
.split(',')
.filter(|part| !part.is_empty())
.map(|part| parse_id(part).map_err(|e| format!("--ports: {e}")))
.collect()
}
fn measure_config(options: &Options) -> Result<crate::measure::MeasureConfig, String> {
let mut config = crate::measure::MeasureConfig {
steps: options.u64("steps", 200)?,
warmup_steps: options.u64("warmup-steps", 10)?,
worker_threads: options.usize("worker-threads", 1)?,
..crate::measure::MeasureConfig::default()
};
if let Some(list) = options.optional("agents") {
config.agent_counts = list
.split(',')
.filter(|p| !p.is_empty())
.map(|p| p.parse::<usize>().map_err(|_| format!("--agents: {p:?}")))
.collect::<Result<Vec<usize>, String>>()?;
}
if let Some(list) = options.optional("modes") {
config.modes = list
.split(',')
.filter(|p| !p.is_empty())
.map(parse_mode)
.collect::<Result<Vec<ExecutionMode>, String>>()?;
}
Ok(config)
}
fn parse_mode(name: &str) -> Result<ExecutionMode, String> {
match name {
"in-process" => Ok(ExecutionMode::InProcess),
"thread" => Ok(ExecutionMode::Thread),
"process" => Ok(ExecutionMode::Process),
other => Err(format!("unknown mode {other:?}")),
}
}
/// Runs the execution-mode comparison and prints its table.
///
/// One child per row: a peak-memory figure is only that row's if nothing else ran in the
/// process that produced it.
fn measure(options: &Options) -> Result<(), String> {
let config = measure_config(options)?;
let program = std::env::current_exe().map_err(|e| format!("current exe: {e}"))?;
let rows = crate::measure::run(&config, &program)?;
print!("{}", crate::measure::table(&rows));
Ok(())
}
/// Measures exactly one row and prints it as one JSON object. The parent's child.
fn measure_row(options: &Options) -> Result<(), String> {
let config = measure_config(options)?;
let mode = parse_mode(options.required("mode")?)?;
let agents = options.usize("agents", 2)?;
let runtime = tokio::runtime::Builder::new_multi_thread()
.enable_all()
.build()
.map_err(|e| format!("runtime: {e}"))?;
let row = runtime.block_on(crate::measure::one(&config, mode, agents))?;
println!("{}", row.to_json());
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
/// An unknown flag is refused by name. A renamed option must fail the launch rather than
/// be accepted and ignored, which would turn a fault or a delay into a no-op.
#[test]
fn an_unknown_option_is_refused_by_name() {
let args: Vec<String> = ["--session", "demo", "--stale-view-at-boundry", "2"]
.iter()
.map(|s| (*s).to_owned())
.collect();
let error = Options::parse(&args, &[flags::COMMON, flags::ENVIRONMENT_ONLY])
.expect_err("an unknown option is refused");
assert!(error.contains("--stale-view-at-boundry"), "{error}");
}
/// A flag that belongs to another command is refused too: an agent has no render delay.
#[test]
fn an_option_of_another_command_is_refused() {
let args: Vec<String> = ["--observation-delay-steps", "2"]
.iter()
.map(|s| (*s).to_owned())
.collect();
Options::parse(&args, &[flags::COMMON, flags::AGENT_ONLY])
.expect_err("the agent command has no render delay");
Options::parse(&args, &[flags::COMMON, flags::ENVIRONMENT_ONLY])
.expect("the environment command does");
}
}

File diff suppressed because it is too large Load diff

View file

@ -4,19 +4,31 @@
//! one interval, and returns boundary `k+1` with its world time advanced by `stepDuration`. It
//! never advances while waiting for the next request, and it does not free-run during agent
//! initialization.
//!
//! Its native output is real: one immutable RGBA8 frame per boundary through a
//! [`ViewPipeline`](crate::media::ViewPipeline) that honours the declared
//! `observationDelaySteps`, and one audio chunk per transition with an exact sample budget.
//! Nothing here resizes, mixes, composites or encodes anything; that is the presentation
//! layer's work.
use std::collections::BTreeSet;
use std::io::Write;
use crate::media::{self, AudioSource, RenderCounter, ViewPipeline};
use crate::task::{controller_schema_ref, inspection, inspection_schema};
// `crate::types` is this crate's facade over the shared `fly-session-types` crate; the
// glob keeps the contract's own names in sight instead of restating them.
use crate::types::*;
use crate::worker::{BoxFuture, HandlerCtx, HandlerReply, StatusCell, WorkerEndpoint};
/// The arena's view: a 4x4 RGBA8 tile whose bytes carry the counter.
pub const VIEW_WIDTH: u64 = 4;
pub const VIEW_HEIGHT: u64 = 4;
/// The arena's one view: a small native RGBA8 image.
pub const VIEW_ID: &str = "arena";
pub const VIEW_WIDTH: u64 = 32;
pub const VIEW_HEIGHT: u64 = 24;
/// The arena's one audio stream. 48 kHz stereo is a native rate, not a presentation choice.
pub const AUDIO_STREAM_ID: &str = "arena";
pub const SAMPLE_RATE: u64 = 48_000;
pub const CHANNELS: u64 = 2;
/// Deliberate faults a test can ask the environment for.
#[derive(Clone, Debug, Default)]
@ -26,6 +38,16 @@ pub struct EnvironmentFaults {
/// Drop the required sensory view from the result at this boundary, so the coordinator
/// meets a world that advanced with no usable sensory data.
pub omit_view_at_boundary: Option<u64>,
/// Serve the previous boundary's frame at this boundary: an extra-delayed sensory input,
/// which is a step failure rather than an acceptable latest frame.
pub stale_view_at_boundary: Option<u64>,
/// Seal a frame one row short at this boundary, so its artifact length is not
/// `rowStride x height`.
pub truncated_view_at_boundary: Option<u64>,
/// Leave the audio chunk out of the result at this boundary.
pub omit_audio_at_boundary: Option<u64>,
/// Emit an audio chunk that starts before the previous chunk ended.
pub overlapping_audio_at_boundary: Option<u64>,
}
#[derive(Clone, Debug)]
@ -36,6 +58,14 @@ pub struct EnvironmentConfig {
/// The world's fixed reduced step duration. 60 Hz is `1/60` s.
pub step_duration: RationalNs,
pub ports: Vec<Id>,
/// The thread allocation the launcher started this worker within.
pub worker_threads: usize,
/// The view's declared render delay, in steps. Zero is same-boundary output.
pub observation_delay_steps: u64,
/// Counts frames actually rendered, so a test can prove one image was not rendered twice.
///
/// It counts in this process only: a world with a process of its own counts there.
pub renders: RenderCounter,
pub faults: EnvironmentFaults,
}
@ -52,6 +82,10 @@ pub struct CounterEnvironment {
world_time: RationalNs,
advances: u64,
batches: BTreeSet<Id>,
pipeline: Option<ViewPipeline>,
audio: Option<AudioSource>,
/// The frame served at the previous boundary, kept only so a fault can serve it again.
previous_view: Option<(ViewRef, flybus::Artifact)>,
}
impl CounterEnvironment {
@ -67,6 +101,9 @@ impl CounterEnvironment {
world_time: RationalNs::ZERO,
advances: 0,
batches: BTreeSet::new(),
pipeline: None,
audio: None,
previous_view: None,
config,
}
}
@ -101,15 +138,25 @@ impl CounterEnvironment {
}
}
pub fn view_descriptor() -> ViewDescriptor {
/// The arena's native view, with the configured render delay.
pub fn view_descriptor(observation_delay_steps: u64) -> ViewDescriptor {
ViewDescriptor {
view_id: id("arena"),
view_id: id(VIEW_ID),
width: VIEW_WIDTH,
height: VIEW_HEIGHT,
row_stride: VIEW_WIDTH * 4,
pixel_aspect_numerator: 1,
pixel_aspect_denominator: 1,
observation_delay_steps: 0,
observation_delay_steps,
}
}
/// The arena's native audio stream.
pub fn audio_descriptor() -> AudioDescriptor {
AudioDescriptor {
stream_id: id(AUDIO_STREAM_ID),
sample_rate: SAMPLE_RATE,
channels: CHANNELS,
}
}
@ -119,10 +166,11 @@ impl CounterEnvironment {
content_digest: digest_of_bytes(b"counter-arena-content-v1"),
configuration_digest: digest_of_bytes(
format!(
"counter-arena-config-v1\nstep={}/{}\nports={}\n",
"counter-arena-config-v1\nstep={}/{}\nports={}\ndelay={}\n",
self.config.step_duration.numerator,
self.config.step_duration.denominator,
self.config.ports.len()
self.config.ports.len(),
self.config.observation_delay_steps
)
.as_bytes(),
),
@ -137,8 +185,10 @@ impl CounterEnvironment {
})
.collect(),
inspection_schema: inspection_schema(),
views: vec![CounterEnvironment::view_descriptor()],
audio: Vec::new(),
views: vec![CounterEnvironment::view_descriptor(
self.config.observation_delay_steps,
)],
audio: vec![CounterEnvironment::audio_descriptor()],
recovery: Recovery::ExactCheckpoint,
determinism: Determinism::FixedBuild,
};
@ -146,73 +196,76 @@ impl CounterEnvironment {
Ok(descriptor)
}
/// Seals one immutable native frame for the current counter and returns the handle.
async fn render(
&self,
ctx: &HandlerCtx<'_>,
) -> DomainResult<(ViewRef, flybus::Artifact)> {
let descriptor = CounterEnvironment::view_descriptor();
let len = descriptor.byte_length();
let mut writer = ctx
.client
.artifacts()
.allocate(len, "image/x-rgba")
.await
.map_err(|e| {
DomainError::new(
ErrorCode::BackendFailure,
format!("frame allocation failed: {}", e.message),
MutationCertainty::Applied,
)
})?;
// Every pixel carries the counter's low byte, so an agent reading the frame reads the
// world rather than a constant.
let byte = (self.counter & 0xff) as u8;
writer
.write_all(&vec![byte; len as usize])
.map_err(|e| {
DomainError::new(
ErrorCode::BackendFailure,
format!("frame write failed: {e}"),
MutationCertainty::Applied,
)
})?;
let artifact = writer.seal().await.map_err(|e| {
DomainError::new(
ErrorCode::BackendFailure,
format!("frame seal failed: {}", e.message),
MutationCertainty::Applied,
)
})?;
let view = ViewRef {
view_id: descriptor.view_id.clone(),
produced_step: descriptor.required_produced_step(self.boundary),
pixels: artifact.reference().clone(),
};
Ok((view, artifact))
}
/// Renders this boundary's native media and returns the observation with its owned
/// handles. The same immutable object serves the sensory and the broadcast view; nothing
/// is rendered twice and no second copy of the pixels exists.
async fn observation(
&self,
&mut self,
ctx: &HandlerCtx<'_>,
) -> DomainResult<(WorldObservation, Vec<(String, flybus::Artifact)>)> {
let omit = self.config.faults.omit_view_at_boundary == Some(self.boundary);
let (views, attachments) = if omit {
(Vec::new(), Vec::new())
let boundary = self.boundary;
let counter = self.counter;
let pipeline = self
.pipeline
.as_mut()
.ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no view pipeline"))?;
if self.config.faults.truncated_view_at_boundary == Some(boundary) {
pipeline
.render_truncated(ctx.client, boundary, counter)
.await?;
} else {
let (view, artifact) = self.render(ctx).await?;
let name = format!("view.{}", view.view_id);
(vec![view], vec![(name, artifact)])
};
pipeline.render(ctx.client, boundary, counter).await?;
}
let produced = pipeline.at(boundary);
let mut attachments = Vec::new();
let mut views = Vec::new();
if self.config.faults.omit_view_at_boundary == Some(boundary) {
// A world that advanced with no usable sensory data.
} else if self.config.faults.stale_view_at_boundary == Some(boundary) {
if let Some((view, artifact)) = self.previous_view.clone() {
attachments.push((media::view_attachment(&view.view_id), artifact));
views.push(view);
}
} else if let Some((view, artifact)) = produced.clone() {
attachments.push((media::view_attachment(&view.view_id), artifact));
views.push(view);
} else {
return Err(DomainError::new(
ErrorCode::BackendFailure,
"the view pipeline has no frame for this boundary",
MutationCertainty::Applied,
));
}
self.previous_view = produced;
let mut audio = Vec::new();
if boundary > 0 && self.config.faults.omit_audio_at_boundary != Some(boundary) {
let step = self.config.step_duration;
let overlap = self.config.faults.overlapping_audio_at_boundary == Some(boundary);
let source = self
.audio
.as_mut()
.ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no audio source"))?;
let (mut chunk, artifact) = source.produce(ctx.client, &step, counter).await?;
if overlap {
// A chunk that starts inside the previous one: the timeline refuses it rather
// than playing the same samples twice.
chunk.first_sample = chunk.first_sample.saturating_sub(1);
}
attachments.push((media::audio_attachment(&chunk.stream_id), artifact));
audio.push(chunk);
}
let observation = WorldObservation {
boundary: self.boundary,
boundary,
world_time: self.world_time,
engine_frame: Some(self.boundary.to_string()),
engine_frame: Some(boundary.to_string()),
sensory_views: views.clone(),
inspection: inspection(self.counter, self.boundary),
inspection: inspection(counter, boundary),
// The same immutable object serves the broadcast view; nothing is rendered twice.
broadcast_views: views,
audio: Vec::new(),
audio,
};
Ok((observation, attachments))
}
@ -268,6 +321,13 @@ impl CounterEnvironment {
self.counter = 0;
self.world_time = RationalNs::ZERO;
self.batches.clear();
self.pipeline = Some(ViewPipeline::new(
CounterEnvironment::view_descriptor(self.config.observation_delay_steps),
self.config.renders.clone(),
));
// A fresh episode starts at audio origin zero; a restore would resume the preserved
// sample position instead, and its first chunk would mark the discontinuity.
self.audio = Some(AudioSource::new(CounterEnvironment::audio_descriptor(), 0));
self.descriptor = Some(descriptor.clone());
// The world is stopped when O[0] goes out and cannot free-run while the brains boot.
self.status.set_state(WorkerState::Ready);
@ -425,6 +485,10 @@ impl WorkerEndpoint for CounterEnvironment {
self.status.clone()
}
fn worker_threads(&self) -> u64 {
self.config.worker_threads as u64
}
fn methods(&self) -> Vec<&'static str> {
vec!["Environment.Initialize", "Environment.Advance"]
}

View file

@ -1,36 +1,38 @@
//! The runnable synthetic composition: one router, two fake agents, one counter arena and one
//! coordinator, over either transport.
//! The runnable synthetic composition: one router, the configured flies, one counter arena and
//! one coordinator, in whichever execution mode the composition asks for.
//!
//! All participants use router semantics even when colocated, so the in-memory and
//! Unix-socket runs exercise the same code. The caller owns the store root directory, which
//! keeps this module free of a temporary-directory dependency.
//! All participants use router semantics even when colocated, so every mode and both
//! transports exercise the same code. The caller owns the store root directory, which keeps
//! this module free of a temporary-directory dependency.
//!
//! The three execution modes are the SESSION-02 comparison:
//!
//! | Mode | Where each participant runs | Transport |
//! | --- | --- | --- |
//! | [`ExecutionMode::InProcess`] | A task on the coordinator's runtime | either |
//! | [`ExecutionMode::Thread`] | Its own OS thread and runtime | Unix socket |
//! | [`ExecutionMode::Process`] | Its own process, one per fly plus one world | Unix socket |
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use std::path::Path;
use std::sync::Mutex;
use std::sync::atomic::{AtomicU64, Ordering};
use flybus::{
Client, ClientConfig, Grants, Pattern, Policy, Router, RouterConfig, ServiceConfig, Transport,
UnixListenerHandle,
};
use flybus::{Client, Grants, Pattern, Policy, Router, RouterConfig};
use crate::agent::{AgentConfig, AgentFaults, FakeAgentWorker, synthetic_profile};
use crate::agent::{AgentFaults, synthetic_profile};
use crate::coordinator::{AgentSlot, Coordinator};
use crate::environment::{CounterEnvironment, EnvironmentConfig, EnvironmentFaults};
use crate::rpc::WorkerRef;
use crate::environment::EnvironmentFaults;
use crate::media::{RenderCounter, SensorLog};
use crate::launcher::{
AgentLaunch, EnvironmentLaunch, Launcher, ReapOutcome, SUPERVISOR_CLIENT, ThreadBudget,
};
use crate::task::{ActionExecutor, CounterTask, IdentityExecutor, Terminal};
// `crate::types` is this crate's facade over the shared `fly-session-types` crate; the
// glob keeps the contract's own names in sight instead of restating them.
use crate::types::*;
use crate::worker::{StatusCell, WorkerHandle, serve};
use crate::worker::StatusCell;
/// Which transport the session runs over. Both must produce the same behaviour.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum Via {
Memory,
Unix,
}
pub use crate::launcher::{ExecutionMode, Via};
/// One agent in the composition.
#[derive(Clone, Debug)]
@ -41,6 +43,22 @@ pub struct AgentSpec {
/// derivation algorithm is specified before the real agent slice.
pub seed: i32,
pub faults: AgentFaults,
/// The threads this agent asks the launcher for. `Agent.Initialize` carries exactly what
/// the launcher allocated, which `workers-v1` requires it to lie within.
pub worker_threads: usize,
}
impl AgentSpec {
/// One agent on one thread, with no injected fault.
pub fn new(agent_id: &str, port_id: &str, seed: i32) -> AgentSpec {
AgentSpec {
agent_id: id(agent_id),
port_id: id(port_id),
seed,
faults: AgentFaults::default(),
worker_threads: 1,
}
}
}
/// The composition the harness builds.
@ -55,7 +73,18 @@ pub struct HarnessConfig {
pub tick_ms: u64,
pub warmup_ticks: u64,
pub terminal: Terminal,
/// The view's declared render delay, in steps. Zero is same-boundary output.
pub observation_delay_steps: u64,
pub environment_faults: EnvironmentFaults,
/// Where each participant runs.
pub mode: ExecutionMode,
/// The total thread allocation the launcher may hand out. `None` sizes it from the
/// composition and the machine, which is what an ordinary run wants; a test that means to
/// exhaust the budget names a number.
pub thread_budget: Option<usize>,
/// The threads reserved for the coordinator, its router and its store.
pub coordinator_threads: usize,
pub environment_threads: usize,
}
impl Default for HarnessConfig {
@ -65,31 +94,46 @@ impl Default for HarnessConfig {
epoch: id("e1"),
episode_id: id("ep1"),
agents: vec![
AgentSpec {
agent_id: id("fly-a"),
port_id: id("p1"),
seed: 7,
faults: AgentFaults::default(),
},
AgentSpec {
agent_id: id("fly-b"),
port_id: id("p2"),
seed: 11,
faults: AgentFaults::default(),
},
AgentSpec { seed: 7, ..AgentSpec::new("fly-a", "p1", 7) },
AgentSpec { seed: 11, ..AgentSpec::new("fly-b", "p2", 11) },
],
step_hz: 60,
tick_ms: 1,
warmup_ticks: 10,
terminal: Terminal::Never,
observation_delay_steps: 0,
environment_faults: EnvironmentFaults::default(),
mode: ExecutionMode::InProcess,
thread_budget: None,
coordinator_threads: 1,
environment_threads: 1,
}
}
}
impl HarnessConfig {
/// The threads this composition needs at a minimum: the coordinator, the world and every
/// agent's own allocation.
pub fn required_threads(&self) -> usize {
self.coordinator_threads
+ self.environment_threads
+ self.agents.iter().map(|a| a.worker_threads).sum::<usize>()
}
/// The budget the launcher runs under: what was configured, or a budget that covers both
/// this composition and this machine's physical cores.
pub fn budget(&self) -> Result<ThreadBudget, DomainError> {
let total = self
.thread_budget
.unwrap_or_else(|| crate::metrics::physical_cores().max(self.required_threads()));
ThreadBudget::new(total, self.coordinator_threads)
}
}
const ENV_SERVICE: &str = "env.arena";
const ENV_CLIENT: &str = "environment";
const ENV_WORKER: &str = "arena";
const COORDINATOR_CLIENT: &str = "coordinator";
fn agent_service(agent_id: &Id) -> String {
format!("agent.{agent_id}")
@ -105,37 +149,6 @@ fn grants(f: impl FnOnce(&mut Grants)) -> Grants {
g
}
/// Makes a connection for one launcher-bound participant, over the chosen transport.
struct Connector {
router: Router,
via: Via,
store_root: PathBuf,
sockets: PathBuf,
next_socket: AtomicU64,
listeners: Mutex<Vec<UnixListenerHandle>>,
}
impl Connector {
async fn client(&self, id: &str) -> Result<Client, flybus::BusError> {
let transport = match self.via {
Via::Memory => self.router.connect_in_memory_as(id),
Via::Unix => {
let n = self.next_socket.fetch_add(1, Ordering::Relaxed);
let path = self.sockets.join(format!("{id}-{n}.sock"));
let listener = self.router.listen_unix_as(&path, id).await.map_err(|e| {
flybus::BusError::new(flybus::ErrorCode::RouterLost, format!("listen: {e}"))
})?;
let transport = Transport::unix(&path).await.map_err(|e| {
flybus::BusError::new(flybus::ErrorCode::RouterLost, format!("connect: {e}"))
})?;
self.listeners.lock().expect("not poisoned").push(listener);
transport
}
};
Client::connect(transport, ClientConfig::new(id, &self.store_root)).await
}
}
/// What a restarted worker looks like from the outside: a new registration and a new
/// incarnation, both different from the ones the coordinator pinned.
#[derive(Clone, Debug)]
@ -148,16 +161,22 @@ pub struct Restarted {
/// A running synthetic session.
pub struct SessionHarness {
pub coordinator: Coordinator,
pub environment: WorkerHandle,
pub agents: BTreeMap<Id, WorkerHandle>,
pub config: HarnessConfig,
pub via: Via,
connector: Connector,
pub mode: ExecutionMode,
/// The media instrumentation of the participants that live in this process. Both are
/// shared memory, so both are empty for a participant with a process of its own; the
/// accessors below return `None` there rather than zero.
renders: RenderCounter,
sensors: BTreeMap<Id, SensorLog>,
/// The supervisor. It owns every participant's lifetime and thread allocation.
pub launcher: Launcher,
observers: Mutex<Vec<Client>>,
}
impl SessionHarness {
/// Builds the router, the workers and the coordinator. Nothing has stepped yet.
/// Builds the router, launches the workers and builds the coordinator. Nothing has
/// stepped yet.
pub async fn start(
via: Via,
root: &Path,
@ -167,16 +186,29 @@ impl SessionHarness {
let sockets = root.join("sockets");
std::fs::create_dir_all(&sockets).expect("the caller owns a writable directory");
// The launcher's policy: who may connect, and what each may do. Naming a target is not
// authority to use it, so the supervisor calls but never registers or publishes, and a
// worker registers exactly one service and calls nothing.
let mut policy = Policy::closed()
.client(
"coordinator",
COORDINATOR_CLIENT,
grants(|g| {
g.call = vec![Pattern::prefix("agent."), Pattern::prefix("env.")];
g.publish = vec![Pattern::prefix("session.")];
g.manage_topics = vec![Pattern::prefix("session.")];
}),
)
.client(
SUPERVISOR_CLIENT,
grants(|g| {
g.call = vec![Pattern::prefix("agent."), Pattern::prefix("env.")];
}),
)
.client(ENV_CLIENT, grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]))
.client(
&format!("{ENV_CLIENT}-r2"),
grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]),
)
.client("observer", grants(|g| g.subscribe = vec![Pattern::prefix("session.")]));
for spec in &config.agents {
let service = agent_service(&spec.agent_id);
@ -196,66 +228,77 @@ impl SessionHarness {
let router = Router::new(router_config).map_err(|e| {
flybus::BusError::new(flybus::ErrorCode::StoreFailure, format!("router: {e}"))
})?;
let connector = Connector {
router,
via,
store_root,
sockets,
next_socket: AtomicU64::new(0),
listeners: Mutex::new(Vec::new()),
};
let budget = config.budget().map_err(refusal)?;
let mut launcher =
Launcher::start(router, config.mode, via, &store_root, &sockets, budget).await?;
let step_duration = hz(config.step_hz).expect("a positive cadence");
let tick_duration = millis(config.tick_ms).expect("a positive tick");
let renders = RenderCounter::new();
let sensors: BTreeMap<Id, SensorLog> = config
.agents
.iter()
.map(|spec| (spec.agent_id.clone(), SensorLog::new()))
.collect();
// The environment first: it owns the world and the descriptor.
let env_client = connector.client(ENV_CLIENT).await?;
let env_service = env_client.register(ENV_SERVICE, ServiceConfig::default()).await?;
let env_incarnation = env_service.incarnation().to_owned();
let environment = serve(
env_client,
env_service,
CounterEnvironment::new(EnvironmentConfig {
let environment = launcher
.launch_environment(EnvironmentLaunch {
session_id: config.session_id.clone(),
worker_id: id(ENV_WORKER),
incarnation_id: id("arena-inc-1"),
step_duration,
ports: config.agents.iter().map(|a| a.port_id.clone()).collect(),
worker_threads: config.environment_threads,
observation_delay_steps: config.observation_delay_steps,
renders: renders.clone(),
faults: config.environment_faults.clone(),
}),
);
client_id: ENV_CLIENT.to_owned(),
service: ENV_SERVICE.to_owned(),
})
.await
.map_err(refusal)?;
let environment_ref = launcher
.worker(&environment.worker_id)
.expect("just launched")
.worker_ref();
let mut slots = Vec::new();
let mut agents = BTreeMap::new();
for spec in &config.agents {
let service_name = agent_service(&spec.agent_id);
let client = connector.client(&agent_client(&spec.agent_id)).await?;
let service = client.register(&service_name, ServiceConfig::default()).await?;
let incarnation = service.incarnation().to_owned();
let handle = serve(
client,
service,
FakeAgentWorker::new(AgentConfig {
let identity = launcher
.launch_agent(AgentLaunch {
session_id: config.session_id.clone(),
agent_id: spec.agent_id.clone(),
port_id: spec.port_id.clone(),
incarnation_id: parse_id(&format!("{}-inc-1", spec.agent_id))
.expect("an agent id plus a suffix is an Id"),
tick_duration,
warmup_ticks: config.warmup_ticks,
worker_threads: spec.worker_threads,
sensors: sensors[&spec.agent_id].clone(),
faults: spec.faults.clone(),
}),
);
slots.push(AgentSlot::new(
WorkerRef::new(&service_name, &incarnation, &spec.agent_id),
client_id: agent_client(&spec.agent_id),
service: agent_service(&spec.agent_id),
})
.await
.map_err(refusal)?;
let worker_ref = launcher
.worker(&spec.agent_id)
.expect("just launched")
.worker_ref();
let mut slot = AgentSlot::new(
worker_ref,
spec.agent_id.clone(),
spec.port_id.clone(),
synthetic_profile(&spec.agent_id, &tick_duration, config.warmup_ticks),
spec.seed,
));
agents.insert(spec.agent_id.clone(), handle);
);
slot.worker_threads = identity.worker_threads as u64;
slots.push(slot);
}
let coordinator_client = connector.client("coordinator").await?;
let coordinator_client = launcher.connect(COORDINATOR_CLIENT).await?;
let executors: BTreeMap<Id, Box<dyn ActionExecutor>> = config
.agents
.iter()
@ -268,7 +311,7 @@ impl SessionHarness {
config.session_id.clone(),
config.epoch.clone(),
config.episode_id.clone(),
WorkerRef::new(ENV_SERVICE, &env_incarnation, &id(ENV_WORKER)),
environment_ref,
slots,
Box::new(CounterTask::new(&config.epoch, config.terminal)),
executors,
@ -276,27 +319,44 @@ impl SessionHarness {
Ok(SessionHarness {
coordinator,
environment,
agents,
config,
via,
connector,
mode: launcher.mode(),
renders,
sensors,
launcher,
observers: Mutex::new(Vec::new()),
})
}
pub fn router(&self) -> &Router {
&self.connector.router
self.launcher.router()
}
/// The coordinator and its supervisor, borrowed apart.
///
/// A supervisor acts while a transition is in flight -- that is what a supervisor is for
/// -- so the two have to be reachable at the same time.
pub fn parts(&mut self) -> (&mut Coordinator, &mut Launcher) {
(&mut self.coordinator, &mut self.launcher)
}
/// The router's own counters: owners, roots, queued messages and store bytes.
pub fn router_stats(&self) -> flybus::RouterStats {
self.launcher.router().stats()
}
/// A client for `id`, connected the same way every participant is.
///
/// An unconfigured client id is refused by the launcher's policy before it can route, so
/// this is not a way around the composition.
pub async fn client(&self, id: &str) -> Result<Client, flybus::BusError> {
self.connector.client(id).await
self.launcher.connect(id).await
}
/// An extra subscriber, for a test that watches the published boundaries.
pub async fn observer(&self) -> Result<Client, flybus::BusError> {
let client = self.connector.client("observer").await?;
let client = self.launcher.connect("observer").await?;
self.observers.lock().expect("not poisoned").push(client.clone());
Ok(client)
}
@ -306,22 +366,6 @@ impl SessionHarness {
/// The coordinator still pins the old registration, so its next call to that agent fails
/// rather than silently reaching another brain.
pub async fn restart_agent(&mut self, agent_id: &Id) -> Result<Restarted, flybus::BusError> {
let tick_duration = millis(self.config.tick_ms).expect("a positive tick");
if let Some(old) = self.agents.remove(agent_id) {
old.stop().await;
}
let service_name = agent_service(agent_id);
let client = self.connector.client(&format!("{}-r2", agent_client(agent_id))).await?;
let service = loop {
match client.register(&service_name, ServiceConfig::default()).await {
Ok(service) => break service,
Err(e) if e.code == flybus::ErrorCode::Conflict => {
// The old registration is released when its connection finishes closing.
tokio::time::sleep(std::time::Duration::from_millis(5)).await;
}
Err(e) => return Err(e),
}
};
let spec = self
.config
.agents
@ -329,53 +373,115 @@ impl SessionHarness {
.find(|spec| spec.agent_id == *agent_id)
.expect("a configured agent")
.clone();
self.launcher.kill(agent_id).await;
let tick_duration = millis(self.config.tick_ms).expect("a positive tick");
let incarnation_id =
parse_id(&format!("{agent_id}-inc-2")).expect("an agent id plus a suffix is an Id");
let restarted = Restarted {
service: service_name,
service_incarnation: service.incarnation().to_owned(),
incarnation_id: incarnation_id.clone(),
};
let handle = serve(
client,
service,
FakeAgentWorker::new(AgentConfig {
self.launcher
.launch_agent(AgentLaunch {
session_id: self.config.session_id.clone(),
agent_id: agent_id.clone(),
incarnation_id,
port_id: spec.port_id.clone(),
incarnation_id: incarnation_id.clone(),
tick_duration,
warmup_ticks: self.config.warmup_ticks,
faults: spec.faults,
}),
);
self.agents.insert(agent_id.clone(), handle);
Ok(restarted)
worker_threads: spec.worker_threads,
// The same log: a replacement worker in this process keeps writing where its
// predecessor wrote, so a restore's sensory input is visible beside it.
sensors: self.sensors.get(agent_id).cloned().unwrap_or_default(),
faults: spec.faults.clone(),
client_id: format!("{}-r2", agent_client(agent_id)),
service: agent_service(agent_id),
})
.await
.map_err(refusal)?;
let worker = self.launcher.worker(agent_id).expect("just launched");
Ok(Restarted {
service: worker.identity.service.clone(),
service_incarnation: worker.service_incarnation.clone(),
incarnation_id,
})
}
/// The agent worker's progress counter, which is its fake model's mutation count.
pub fn agent_mutations(&self, agent_id: &Id) -> u64 {
self.agents.get(agent_id).map(WorkerHandle::progress_counter).unwrap_or_default()
/// Ends one participant without asking it, as a crash would.
pub async fn kill(&mut self, worker_id: &Id) -> ReapOutcome {
self.launcher.kill(worker_id).await
}
pub fn environment_mutations(&self) -> u64 {
self.environment.progress_counter()
/// The worker id the environment answers to.
pub fn environment_id(&self) -> Id {
id(ENV_WORKER)
}
pub fn agent_status(&self, agent_id: &Id) -> Option<StatusCell> {
self.agents.get(agent_id).map(|handle| handle.status.clone())
}
/// Stops every worker and closes the router.
pub async fn shutdown(self) {
let SessionHarness { coordinator, environment, agents, connector, observers, .. } = self;
drop(coordinator);
environment.stop().await;
for (_, handle) in agents {
handle.stop().await;
/// What one agent read out of its sensory attachments, in order, when this process is
/// where that log lives.
///
/// `None` means "not observable from here", not "nothing was read": an agent with a
/// process of its own records into its own copy. The media path itself crosses a process
/// boundary -- the frame is one artifact in the shared store, reached through owned
/// handles -- but this instrumentation does not, because it is shared memory.
pub fn sensor_log(&self, agent_id: &Id) -> Option<SensorLog> {
match self.mode {
ExecutionMode::Process => None,
_ => self.sensors.get(agent_id).cloned(),
}
}
/// How many native frames the environment rendered, when the world lives in this process.
///
/// `None` for a world with a process of its own, for the same reason as above.
pub fn renders(&self) -> Option<u64> {
match self.mode {
ExecutionMode::Process => None,
_ => Some(self.renders.count()),
}
}
/// The agent worker's progress counter, which is its fake model's mutation count, when
/// this process is where that counter lives.
///
/// `None` means "not observable from here", not "nothing happened": a participant with a
/// process of its own keeps its counter there. [`SessionHarness::progress_of`] reads it
/// over the bus and works in every mode.
pub fn agent_mutations(&self, agent_id: &Id) -> Option<u64> {
self.launcher
.worker(agent_id)
.and_then(crate::launcher::LaunchedWorker::progress_counter)
}
pub fn environment_mutations(&self) -> Option<u64> {
self.agent_mutations(&id(ENV_WORKER))
}
/// One participant's progress counter, read over the bus. Works in every execution mode.
pub async fn progress_of(&mut self, worker_id: &Id) -> Result<u64, DomainError> {
Ok(self.launcher.health_check(worker_id).await?.progress_counter)
}
/// The local status cell of a participant in this process, or `None` for one with a
/// process of its own.
pub fn agent_status(&self, agent_id: &Id) -> Option<StatusCell> {
self.launcher.worker(agent_id).and_then(crate::launcher::LaunchedWorker::status)
}
/// Reaps every participant and closes the router.
pub async fn shutdown(self) {
let SessionHarness { coordinator, mut launcher, observers, .. } = self;
drop(coordinator);
launcher.reap_all(&id("shutdown")).await;
for observer in observers.into_inner().expect("not poisoned") {
observer.close().await;
}
connector.router.shutdown();
launcher.router().shutdown();
}
}
/// A launcher refusal, as a bus error: the harness's one error type stays the bus's.
fn refusal(e: DomainError) -> flybus::BusError {
let code = match e.code {
ErrorCode::Busy => flybus::ErrorCode::QuotaExceeded,
ErrorCode::IdentityMismatch => flybus::ErrorCode::NotAuthorized,
_ => flybus::ErrorCode::RouterLost,
};
flybus::BusError::new(code, e.to_string())
}

File diff suppressed because it is too large Load diff

View file

@ -22,11 +22,16 @@
//! [`step-v1`]: https://example.invalid/step-v1
pub mod agent;
pub mod cli;
pub mod clock;
pub mod coordinator;
pub mod dedup;
pub mod environment;
pub mod harness;
pub mod launcher;
pub mod measure;
pub mod media;
pub mod metrics;
pub mod phase;
pub mod rpc;
pub mod task;
@ -37,5 +42,8 @@ pub mod worker;
pub mod types;
pub use fly_session_types;
pub use coordinator::{Coordinator, DispatchOrder, Injections, SessionFailure, StepReport};
pub use coordinator::{
Coordinator, Deadlines, DispatchOrder, Injections, ResolutionEnd, SessionFailure, StepReport,
};
pub use launcher::{ExecutionMode, Launcher, ReapOutcome, ThreadBudget, Via};
pub use phase::{Phase, PhaseMachine};

View file

@ -0,0 +1,488 @@
//! The execution-mode comparison the implementation guide's section 5 asks for.
//!
//! It runs the same synthetic session in each execution mode, at one, two and four agents,
//! and reports the thread allocation, the RPC and critical-path percentiles, the memory peaks
//! and the router's owner, collection and queue counters.
//!
//! **These are local synthetic timings on one machine, and no host capacity claim follows
//! from any of them.** They exist so the three modes can be compared with each other: the
//! question this slice has to answer is what a process boundary costs, not how fast anything
//! is. Pacing is switched off for the run, so a transition follows the one before it as fast
//! as the participants answer and the samples are work rather than sleep.
//!
//! **Every row runs in a process of its own.** The coordinator's memory figure is a peak --
//! `VmHWM` never falls -- so several rows sharing one process would each report where that
//! process had already been rather than what its own mode costs, and the column would order
//! itself by row position instead of by mode. The parent spawns one `measure-row` child per
//! row and reads its result back, so each figure belongs to the row that produced it.
use std::collections::BTreeMap;
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use serde_json::{Value, json};
use crate::coordinator::DispatchOrder;
use crate::harness::{AgentSpec, HarnessConfig, SessionHarness, Via};
use crate::launcher::{ExecutionMode, flags};
use crate::metrics::{Percentiles, physical_cores};
/// What to compare.
#[derive(Clone, Debug)]
pub struct MeasureConfig {
/// Transitions per run, after the warm-up ones.
pub steps: u64,
/// Transitions run before sampling starts, so first-call costs are not in the samples.
pub warmup_steps: u64,
pub agent_counts: Vec<usize>,
pub modes: Vec<ExecutionMode>,
/// The within-agent worker count each agent asks its launcher for.
pub worker_threads: usize,
}
impl Default for MeasureConfig {
fn default() -> MeasureConfig {
MeasureConfig {
steps: 200,
warmup_steps: 10,
agent_counts: vec![1, 2, 4],
modes: ExecutionMode::all().to_vec(),
worker_threads: 1,
}
}
}
/// The highest value each router counter reached while the transitions ran.
#[derive(Debug, Default)]
struct Peaks {
owners: AtomicU64,
roots: AtomicU64,
queued: AtomicU64,
sealed: AtomicU64,
store_bytes: AtomicU64,
}
impl Peaks {
fn observe(&self, stats: &flybus::RouterStats) {
raise(&self.owners, stats.owners as u64);
raise(&self.roots, stats.artifact_roots);
raise(&self.queued, stats.queued as u64);
raise(&self.sealed, stats.sealed_artifacts as u64);
raise(&self.store_bytes, stats.store_bytes);
}
fn read(&self) -> (usize, u64, usize, usize, u64) {
(
self.owners.load(Ordering::Relaxed) as usize,
self.roots.load(Ordering::Relaxed),
self.queued.load(Ordering::Relaxed) as usize,
self.sealed.load(Ordering::Relaxed) as usize,
self.store_bytes.load(Ordering::Relaxed),
)
}
}
fn raise(slot: &AtomicU64, value: u64) {
slot.fetch_max(value, Ordering::Relaxed);
}
/// One measured composition.
#[derive(Clone, Debug)]
pub struct Row {
pub mode: ExecutionMode,
pub agents: usize,
pub worker_threads: usize,
pub physical_cores: usize,
pub budget_total: usize,
pub budget_used: usize,
pub steps: u64,
/// `Agent.Prepare`, over every agent and every transition.
pub prepare: Percentiles,
pub commit: Percentiles,
pub advance: Percentiles,
/// `Worker.Status`: an RPC with no domain work behind it, so it is the router and
/// transport floor rather than a measure of the worker.
pub status: Percentiles,
/// One whole transition, pacing excluded: the critical path.
pub step: Percentiles,
pub coordinator_peak_rss_kib: u64,
/// The sum of the peak resident sets of the participants with processes of their own.
pub participants_peak_rss_kib: u64,
pub owners_max: usize,
pub owners_final: usize,
pub artifact_roots_max: u64,
pub queued_max: usize,
pub sealed_max: usize,
pub sealed_final: usize,
pub store_bytes_max: u64,
pub store_bytes_final: u64,
/// Frames this run actually observed, counted from the behaviour trace: every sensory
/// view of every transition, plus the one the environment sealed for boundary zero.
///
/// Counted rather than calculated, so a backend that sealed two frames per boundary would
/// show up here instead of being hidden by arithmetic.
pub frames_observed: u64,
}
impl Row {
/// Frames the store collected: observed, minus the ones still owned at the end.
pub fn collected(&self) -> u64 {
self.frames_observed.saturating_sub(self.sealed_final as u64)
}
/// The row as one JSON object, for the child that measured it to hand back.
pub fn to_json(&self) -> Value {
let p = |x: &Percentiles| {
json!({"count": x.count, "p50": x.p50_ns, "p95": x.p95_ns, "p99": x.p99_ns,
"max": x.max_ns})
};
json!({
"mode": self.mode.label(),
"agents": self.agents,
"workerThreads": self.worker_threads,
"physicalCores": self.physical_cores,
"budgetTotal": self.budget_total,
"budgetUsed": self.budget_used,
"steps": self.steps,
"prepare": p(&self.prepare),
"commit": p(&self.commit),
"advance": p(&self.advance),
"status": p(&self.status),
"step": p(&self.step),
"coordinatorPeakRssKib": self.coordinator_peak_rss_kib,
"participantsPeakRssKib": self.participants_peak_rss_kib,
"ownersMax": self.owners_max,
"ownersFinal": self.owners_final,
"artifactRootsMax": self.artifact_roots_max,
"queuedMax": self.queued_max,
"sealedMax": self.sealed_max,
"sealedFinal": self.sealed_final,
"storeBytesMax": self.store_bytes_max,
"storeBytesFinal": self.store_bytes_final,
"framesObserved": self.frames_observed,
})
}
/// Reads back what a `measure-row` child printed.
pub fn from_json(value: &Value) -> Result<Row, String> {
let u = |name: &str| -> Result<u64, String> {
value
.get(name)
.and_then(Value::as_u64)
.ok_or_else(|| format!("measure row: {name} is missing or not a number"))
};
let p = |name: &str| -> Result<Percentiles, String> {
let v = value
.get(name)
.ok_or_else(|| format!("measure row: {name} is missing"))?;
let f = |k: &str| v.get(k).and_then(Value::as_u64).unwrap_or_default();
Ok(Percentiles {
count: f("count") as usize,
p50_ns: f("p50"),
p95_ns: f("p95"),
p99_ns: f("p99"),
max_ns: f("max"),
})
};
let mode = match value.get("mode").and_then(Value::as_str) {
Some("in-process") => ExecutionMode::InProcess,
Some("thread") => ExecutionMode::Thread,
Some("process") => ExecutionMode::Process,
other => return Err(format!("measure row: unknown mode {other:?}")),
};
Ok(Row {
mode,
agents: u("agents")? as usize,
worker_threads: u("workerThreads")? as usize,
physical_cores: u("physicalCores")? as usize,
budget_total: u("budgetTotal")? as usize,
budget_used: u("budgetUsed")? as usize,
steps: u("steps")?,
prepare: p("prepare")?,
commit: p("commit")?,
advance: p("advance")?,
status: p("status")?,
step: p("step")?,
coordinator_peak_rss_kib: u("coordinatorPeakRssKib")?,
participants_peak_rss_kib: u("participantsPeakRssKib")?,
owners_max: u("ownersMax")? as usize,
owners_final: u("ownersFinal")? as usize,
artifact_roots_max: u("artifactRootsMax")?,
queued_max: u("queuedMax")? as usize,
sealed_max: u("sealedMax")? as usize,
sealed_final: u("sealedFinal")? as usize,
store_bytes_max: u("storeBytesMax")?,
store_bytes_final: u("storeBytesFinal")?,
frames_observed: u("framesObserved")?,
})
}
}
/// Runs the comparison, one child process per row.
///
/// `program` is this crate's binary; each row is measured by a `measure-row` invocation of it
/// so that the row's memory peak is its own and not the accumulated peak of the rows before
/// it. The order of the rows therefore cannot change any of their numbers.
pub fn run(config: &MeasureConfig, program: &std::path::Path) -> Result<Vec<Row>, String> {
let mut rows = Vec::new();
for mode in &config.modes {
for agents in &config.agent_counts {
rows.push(row_in_a_child(config, program, *mode, *agents)?);
}
}
Ok(rows)
}
fn row_in_a_child(
config: &MeasureConfig,
program: &std::path::Path,
mode: ExecutionMode,
agents: usize,
) -> Result<Row, String> {
let output = std::process::Command::new(program)
.arg("measure-row")
.arg(format!("--{}", flags::MODE))
.arg(mode.label())
.arg(format!("--{}", flags::AGENTS))
.arg(agents.to_string())
.arg(format!("--{}", flags::STEPS))
.arg(config.steps.to_string())
.arg(format!("--{}", flags::WARMUP_STEPS))
.arg(config.warmup_steps.to_string())
.arg(format!("--{}", flags::WORKER_THREADS))
.arg(config.worker_threads.to_string())
.stdin(std::process::Stdio::null())
.output()
.map_err(|e| format!("measure-row {}: {e}", mode.label()))?;
if !output.status.success() {
return Err(format!(
"measure-row {} {agents}: exited {}: {}",
mode.label(),
output.status,
String::from_utf8_lossy(&output.stderr).trim()
));
}
let text = String::from_utf8_lossy(&output.stdout);
let line = text
.lines()
.rev()
.find(|l| l.trim_start().starts_with('{'))
.ok_or_else(|| format!("measure-row {}: no row on stdout", mode.label()))?;
let value: Value = serde_json::from_str(line)
.map_err(|e| format!("measure-row {}: unreadable row: {e}", mode.label()))?;
Row::from_json(&value)
}
/// Measures exactly one row, in this process. The `measure-row` subcommand's body.
pub async fn one(
config: &MeasureConfig,
mode: ExecutionMode,
agents: usize,
) -> Result<Row, String> {
let dir = std::env::temp_dir().join(format!(
"fly-session-measure-{}-{agents}-{}",
mode.label(),
std::process::id()
));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).map_err(|e| format!("{}: {e}", dir.display()))?;
let result = measure_in(config, mode, agents, &dir).await;
let _ = std::fs::remove_dir_all(&dir);
result
}
async fn measure_in(
config: &MeasureConfig,
mode: ExecutionMode,
agents: usize,
dir: &std::path::Path,
) -> Result<Row, String> {
let specs: Vec<AgentSpec> = (0..agents)
.map(|i| AgentSpec {
worker_threads: config.worker_threads,
..AgentSpec::new(&format!("fly-{}", (b'a' + i as u8) as char), &format!("p{}", i + 1), 7 + i as i32)
})
.collect();
let harness_config = HarnessConfig {
agents: specs,
mode,
..HarnessConfig::default()
};
let budget = harness_config.budget().map_err(|e| e.to_string())?;
let (budget_total, budget_used_floor) = (budget.total(), harness_config.required_threads());
let mut harness = SessionHarness::start(Via::Unix, dir, harness_config)
.await
.map_err(|e| format!("{}: start: {}", mode.label(), e.message))?;
harness.coordinator.dispatch = DispatchOrder::Concurrent;
harness.coordinator.disable_pacing();
harness
.coordinator
.bootstrap()
.await
.map_err(|e| format!("{}: bootstrap: {e}", mode.label()))?;
// The warm-up transitions pay the first-call costs; their samples are then discarded.
harness
.coordinator
.run(config.warmup_steps)
.await
.map_err(|e| format!("{}: warm-up: {e}", mode.label()))?;
harness.coordinator.metrics.clear();
// The router's counters are sampled *while* transitions run, not between them: a queue
// that is empty at every committed boundary says nothing about whether it stayed bounded
// during the transaction, which is the thing section 4 asks about.
let peaks = Arc::new(Peaks::default());
let sampling = Arc::new(AtomicBool::new(true));
let sampler = tokio::spawn({
let router = harness.router().clone();
let peaks = peaks.clone();
let sampling = sampling.clone();
async move {
while sampling.load(Ordering::Relaxed) {
peaks.observe(&router.stats());
tokio::time::sleep(std::time::Duration::from_micros(200)).await;
}
peaks.observe(&router.stats());
}
});
let environment = harness.environment_id();
let mut status = crate::metrics::Metrics::default();
for _ in 0..config.steps {
harness
.coordinator
.step()
.await
.map_err(|e| format!("{}: step: {e}", mode.label()))?;
// One status call per transition: an RPC the worker answers from a cell rather than
// from its endpoint, so it is the router-and-transport floor the domain calls sit on.
let started = std::time::Instant::now();
let _ = harness.launcher.health_check(&environment).await;
status.record("Worker.Status", started.elapsed());
}
sampling.store(false, Ordering::Relaxed);
let _ = sampler.await;
let (owners_max, roots_max, queued_max, sealed_max, store_bytes_max) = peaks.read();
let metrics = &harness.coordinator.metrics;
let zero = Percentiles::default();
let row = Row {
mode,
agents,
worker_threads: config.worker_threads,
physical_cores: physical_cores(),
budget_total,
budget_used: budget_used_floor,
steps: config.steps,
prepare: metrics.percentiles("Agent.Prepare").unwrap_or(zero),
commit: metrics.percentiles("Agent.Commit").unwrap_or(zero),
advance: metrics.percentiles("Environment.Advance").unwrap_or(zero),
status: status.percentiles("Worker.Status").unwrap_or(zero),
step: metrics.percentiles("step").unwrap_or(zero),
coordinator_peak_rss_kib: crate::metrics::peak_rss_kib().unwrap_or_default(),
participants_peak_rss_kib: harness
.launcher
.peak_rss_kib()
.iter()
.filter(|(who, _)| who.as_str() != "coordinator")
.map(|(_, kib)| *kib)
.sum(),
owners_max,
owners_final: harness.router_stats().owners,
artifact_roots_max: roots_max,
queued_max,
sealed_max,
sealed_final: harness.router_stats().sealed_artifacts,
store_bytes_max,
store_bytes_final: harness.router_stats().store_bytes,
// Counted from the behaviour trace: every sensory view of every transition this run
// recorded, plus the frame the environment sealed for boundary zero.
frames_observed: 1 + harness
.coordinator
.trace
.transitions
.iter()
.map(|t| t.behaviour.observation_boundaries.len() as u64)
.sum::<u64>(),
};
harness.shutdown().await;
Ok(row)
}
/// The measurement table, as Markdown.
pub fn table(rows: &[Row]) -> String {
let mut out = String::new();
out.push_str(
"Local synthetic timings on one machine. Not a capacity claim, and not a latency goal.\n\n",
);
if let Some(first) = rows.first() {
out.push_str(&format!(
"Physical cores: {}. Coordinator reservation: 1 thread. Every row was measured in \
a process of its own, so no figure depends on the order of the rows.\n\n",
first.physical_cores
));
}
out.push_str(
"| mode | agents | threads/agent | budget used/total | Prepare p50/p95/p99 us | \
Commit p50/p95/p99 us | Advance p50/p95/p99 us | Status p50/p99 us | step p50/p95/p99 us |\n",
);
out.push_str("| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n");
for r in rows {
out.push_str(&format!(
"| {} | {} | {} | {}/{} | {:.0}/{:.0}/{:.0} | {:.0}/{:.0}/{:.0} | \
{:.0}/{:.0}/{:.0} | {:.0}/{:.0} | {:.0}/{:.0}/{:.0} |\n",
r.mode.label(),
r.agents,
r.worker_threads,
r.budget_used,
r.budget_total,
r.prepare.p50_us(),
r.prepare.p95_us(),
r.prepare.p99_us(),
r.commit.p50_us(),
r.commit.p95_us(),
r.commit.p99_us(),
r.advance.p50_us(),
r.advance.p95_us(),
r.advance.p99_us(),
r.status.p50_us(),
r.status.p99_us(),
r.step.p50_us(),
r.step.p95_us(),
r.step.p99_us(),
));
}
out.push('\n');
out.push_str(
"| mode | agents | coordinator peak RSS KiB | participant peak RSS KiB | owners max/final | \
roots max | queued max | sealed max/final | store bytes max/final | frames observed/collected |\n",
);
out.push_str("| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n");
for r in rows {
out.push_str(&format!(
"| {} | {} | {} | {} | {}/{} | {} | {} | {}/{} | {}/{} | {}/{} |\n",
r.mode.label(),
r.agents,
r.coordinator_peak_rss_kib,
r.participants_peak_rss_kib,
r.owners_max,
r.owners_final,
r.artifact_roots_max,
r.queued_max,
r.sealed_max,
r.sealed_final,
r.store_bytes_max,
r.store_bytes_final,
r.frames_observed,
r.collected(),
));
}
out
}
/// The rows keyed by mode and agent count, for a caller that wants one of them.
pub fn by_composition(rows: &[Row]) -> BTreeMap<(String, usize), Row> {
rows.iter()
.map(|r| ((r.mode.label().to_owned(), r.agents), r.clone()))
.collect()
}

View file

@ -0,0 +1,837 @@
//! Native observations and the presentation handoff: the MEDIA-01 slice.
//!
//! Everything here sits on top of the bus `ArtifactRef` and its ownership rules. There is no
//! second buffer system: an environment allocates, writes and seals one immutable object per
//! boundary, the coordinator forwards that one owned handle to every agent and to publication,
//! and a spectator reads it through an ordinary latest subscription.
//!
//! The module owns four things:
//!
//! 1. Production. [`ViewPipeline`] renders one native frame per boundary and hands out the
//! frame a declared `observationDelaySteps` requires, so a pipeline delay is a real queue
//! rather than a number in a descriptor. [`AudioSource`] produces one chunk per boundary
//! with an exact rational sample budget.
//! 2. Acceptance. [`check_required_views`] and [`AudioTimelines`] are the coordinator's Phase C
//! media checks: a required sensory view must exist at exactly the producing boundary its
//! declared delay implies, and audio chunks cannot overlap or go backwards inside an epoch.
//! 3. Consumption. [`Spectator`] is a presentation-side consumer on a latest subscription with
//! finite credits, and [`detach_frame`] is the renderer that keeps its handle after the
//! message is gone.
//! 4. Identity. [`AssetRegistry`] holds installed persistent content named by `AssetRef`.
//! Importing an asset produces a *new* transient artifact; the two identities never convert.
//!
//! Resizing, overlays, compositing, mixing, encoding and streaming are not here and are not
//! anywhere else in this crate: they belong to the application's presentation layer.
use std::collections::{BTreeMap, VecDeque};
use std::io::Write;
use std::sync::Arc;
use std::sync::atomic::{AtomicU64, Ordering};
use fly_session_types::media::AudioTimeline;
// `crate::types` is this crate's facade over the shared `fly-session-types` crate; the glob
// keeps the contract's own names in sight instead of restating them.
use crate::types::*;
/// The content type of a native RGBA8 frame. Top-left origin, no padded rows.
pub const FRAME_CONTENT_TYPE: &str = "image/x-rgba8";
/// The content type of a native audio chunk: interleaved little-endian f32.
pub const AUDIO_CONTENT_TYPE: &str = "audio/x-f32le";
/// The attachment name one view's pixels travel under.
pub fn view_attachment(view_id: &str) -> String {
format!("view.{view_id}")
}
/// The attachment name one audio stream's samples travel under.
pub fn audio_attachment(stream_id: &str) -> String {
format!("audio.{stream_id}")
}
fn store_error(what: &str, message: &str) -> DomainError {
DomainError::new(
ErrorCode::BackendFailure,
format!("{what}: {message}"),
MutationCertainty::Applied,
)
}
fn media_error(message: impl std::fmt::Display) -> DomainError {
// A world that advanced without usable media leaves the transition's certainty unknown:
// the mutation happened, the observation of it did not.
DomainError::new(ErrorCode::BufferInvalid, message, MutationCertainty::Unknown)
}
// ----------------------------------------------------------------------------------------------
// Production
/// How many frames a producer has actually rendered.
///
/// A shared counter, so a test can prove that forwarding one image to several recipients
/// renders it once.
#[derive(Clone, Debug, Default)]
pub struct RenderCounter(Arc<AtomicU64>);
impl RenderCounter {
pub fn new() -> RenderCounter {
RenderCounter::default()
}
pub fn bump(&self) {
self.0.fetch_add(1, Ordering::SeqCst);
}
pub fn count(&self) -> u64 {
self.0.load(Ordering::SeqCst)
}
}
/// One native frame of the counter arena: a real synthetic pattern, not a constant fill.
///
/// Top-left RGBA8 with `rowStride` exactly `4 x width` and no padded rows, which is the only
/// pixel layout v1 has. The red channel carries the world counter, so a reader that samples
/// one pixel still reads the world; green is a horizontal ramp and blue a vertical ramp with
/// a one-column bar that walks with the boundary, so consecutive frames differ.
pub fn arena_frame(descriptor: &ViewDescriptor, counter: i64, boundary: u64) -> Vec<u8> {
let width = descriptor.width;
let height = descriptor.height;
let stride = descriptor.row_stride as usize;
let mut out = vec![0u8; stride * height as usize];
let counter_byte = (counter & 0xff) as u8;
let bar = boundary % width;
for y in 0..height {
let row = y as usize * stride;
for x in 0..width {
let p = row + x as usize * 4;
let ramp_x = if width > 1 {
(x * 255 / (width - 1)) as u8
} else {
0
};
let ramp_y = if height > 1 {
(y * 255 / (height - 1)) as u8
} else {
0
};
out[p] = counter_byte;
out[p + 1] = ramp_x;
out[p + 2] = if x == bar { 255 } else { ramp_y };
out[p + 3] = 255;
}
}
out
}
/// One view's production pipeline: render at every boundary, deliver with the declared delay.
///
/// `observationDelaySteps` is a real queue here. At boundary `b` the required frame is the one
/// produced at `max(0, b - delay)`, so a declared delay of two repeats `O[0]` at boundaries 0,
/// 1 and 2 -- the bootstrap repetition the contract allows -- and then advances one frame per
/// boundary. Nothing beyond that is retained: an older frame is dropped, so a later boundary
/// cannot be served an arbitrary stale image.
pub struct ViewPipeline {
descriptor: ViewDescriptor,
frames: VecDeque<(u64, flybus::Artifact)>,
renders: RenderCounter,
}
impl ViewPipeline {
pub fn new(descriptor: ViewDescriptor, renders: RenderCounter) -> ViewPipeline {
ViewPipeline {
descriptor,
frames: VecDeque::new(),
renders,
}
}
pub fn descriptor(&self) -> &ViewDescriptor {
&self.descriptor
}
/// Seals one immutable frame for `boundary` and files it under its producing boundary.
pub async fn render(
&mut self,
client: &flybus::Client,
boundary: u64,
counter: i64,
) -> DomainResult<()> {
let bytes = arena_frame(&self.descriptor, counter, boundary);
let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?;
self.renders.bump();
self.frames.push_back((boundary, artifact));
// Keep exactly the frames a declared delay can still require.
while self.frames.len() > self.descriptor.observation_delay_steps as usize + 1 {
self.frames.pop_front();
}
Ok(())
}
/// Seals a frame of the wrong length, which is what a broken backend produces. The
/// reference it returns describes the artifact honestly, so the shape check is the thing
/// under test rather than a lie in the payload.
pub async fn render_truncated(
&mut self,
client: &flybus::Client,
boundary: u64,
counter: i64,
) -> DomainResult<()> {
let mut bytes = arena_frame(&self.descriptor, counter, boundary);
bytes.truncate(bytes.len() - self.descriptor.row_stride as usize);
let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?;
self.renders.bump();
self.frames.push_back((boundary, artifact));
while self.frames.len() > self.descriptor.observation_delay_steps as usize + 2 {
self.frames.pop_front();
}
Ok(())
}
/// The view reference and the owned handle a required sensory view has at `boundary`.
pub fn at(&self, boundary: u64) -> Option<(ViewRef, flybus::Artifact)> {
let produced = self.descriptor.required_produced_step(boundary);
self.frame_produced_at(produced)
}
/// The frame produced at exactly `produced`, if it is still retained.
pub fn frame_produced_at(&self, produced: u64) -> Option<(ViewRef, flybus::Artifact)> {
self.frames
.iter()
.find(|(step, _)| *step == produced)
.map(|(step, artifact)| {
(
ViewRef {
view_id: self.descriptor.view_id.clone(),
produced_step: *step,
pixels: artifact.reference().clone(),
},
artifact.clone(),
)
})
}
/// How many boundaries this pipeline has rendered.
pub fn renders(&self) -> u64 {
self.renders.count()
}
}
/// One audio stream's production: an exact sample budget and a deterministic waveform.
///
/// The number of frames in a step is `sampleRate x stepDuration`, accumulated as a rational so
/// a cadence that does not divide the sample rate never drifts: 8 kHz at 60 Hz produces
/// 133, 133, 134, ... and the sum is exact at every boundary. The waveform is integer-phase
/// arithmetic only, so a `fixed-build` environment produces the same bytes on every run.
pub struct AudioSource {
descriptor: AudioDescriptor,
/// The unconsumed fraction of a frame, over `denominator`.
accumulator: u128,
denominator: u128,
next_sample: u64,
phase: u64,
chunks: u64,
discontinuous: bool,
}
impl AudioSource {
/// A fresh episode, whose first chunk starts at the configured audio origin.
pub fn new(descriptor: AudioDescriptor, origin: u64) -> AudioSource {
AudioSource {
descriptor,
accumulator: 0,
denominator: 1,
next_sample: origin,
phase: 0,
chunks: 0,
discontinuous: false,
}
}
/// A new epoch after a restore: the sample position is preserved and the first chunk of
/// this epoch marks a discontinuity.
pub fn restored_at(descriptor: AudioDescriptor, sample: u64) -> AudioSource {
let mut source = AudioSource::new(descriptor, sample);
source.discontinuous = true;
source
}
pub fn descriptor(&self) -> &AudioDescriptor {
&self.descriptor
}
pub fn next_sample(&self) -> u64 {
self.next_sample
}
pub fn chunks(&self) -> u64 {
self.chunks
}
/// The exact number of sample frames one step of `step` nanoseconds contains.
///
/// The remainder is kept, never rounded: the accumulator is integer arithmetic over the
/// common denominator `stepDenominator x 1e9`.
pub fn frames_for_step(&mut self, step: &RationalNs) -> DomainResult<u64> {
let denominator = u128::from(step.denominator)
.checked_mul(1_000_000_000)
.ok_or_else(|| DomainError::invalid("audio: the step denominator overflows"))?;
if self.denominator != denominator {
// A cadence change would need a new epoch; carrying a remainder across one would
// be a silent resample.
if self.chunks > 0 {
return Err(DomainError::invalid(
"audio: the cadence changed inside an epoch",
));
}
self.denominator = denominator;
}
let per_step = u128::from(self.descriptor.sample_rate)
.checked_mul(u128::from(step.numerator))
.ok_or_else(|| DomainError::invalid("audio: the sample budget overflows"))?;
self.accumulator = self
.accumulator
.checked_add(per_step)
.ok_or_else(|| DomainError::invalid("audio: the sample accumulator overflows"))?;
let frames = self.accumulator / self.denominator;
self.accumulator %= self.denominator;
u64::try_from(frames).map_err(|_| DomainError::invalid("audio: too many frames in a step"))
}
/// Produces one chunk covering exactly one step of world time.
pub async fn produce(
&mut self,
client: &flybus::Client,
step: &RationalNs,
counter: i64,
) -> DomainResult<(AudioRef, flybus::Artifact)> {
let frames = self.frames_for_step(step)?;
let bytes = self.samples(frames, counter);
let artifact = seal(client, AUDIO_CONTENT_TYPE, &bytes).await?;
let chunk = AudioRef {
stream_id: self.descriptor.stream_id.clone(),
first_sample: self.next_sample,
sample_frames: frames,
samples: artifact.reference().clone(),
discontinuity: self.discontinuous && self.chunks == 0,
};
self.next_sample = self
.next_sample
.checked_add(frames)
.ok_or_else(|| DomainError::invalid("audio: the sample position overflows"))?;
self.chunks += 1;
Ok((chunk, artifact))
}
/// A deterministic triangle wave whose pitch follows the world counter, interleaved across
/// the declared channels. Every sample is finite by construction.
fn samples(&mut self, frames: u64, counter: i64) -> Vec<u8> {
let rate = self.descriptor.sample_rate;
let channels = self.descriptor.channels;
let step = 220 + (counter.rem_euclid(8) as u64) * 55;
let mut out = Vec::with_capacity((frames * channels * 4) as usize);
for _ in 0..frames {
self.phase = (self.phase + step) % rate;
let position = self.phase as f32 / rate as f32;
// 1 - 2|2p - 1| is a triangle in [-1, 1] built from exact IEEE operations.
let value = 1.0 - 2.0 * (2.0 * position - 1.0).abs();
for channel in 0..channels {
let scaled = value * 0.25 / (channel + 1) as f32;
out.extend_from_slice(&scaled.to_le_bytes());
}
}
out
}
}
/// Allocates, writes and seals one immutable artifact.
async fn seal(
client: &flybus::Client,
content_type: &str,
bytes: &[u8],
) -> DomainResult<flybus::Artifact> {
let mut writer = client
.artifacts()
.allocate(bytes.len() as u64, content_type)
.await
.map_err(|e| store_error("allocate", &e.message))?;
writer
.write_all(bytes)
.map_err(|e| store_error("write", &e.to_string()))?;
writer
.seal()
.await
.map_err(|e| store_error("seal", &e.message))
}
/// Splits one reply's attachments into the view handles and the audio handles.
///
/// Views are sensory data forwarded to agents; audio is presentation data that is published
/// and never attached to a sensory input.
pub fn split_attachments(
artifacts: BTreeMap<String, flybus::Artifact>,
) -> (
BTreeMap<String, flybus::Artifact>,
BTreeMap<String, flybus::Artifact>,
) {
let mut views = BTreeMap::new();
let mut audio = BTreeMap::new();
for (name, artifact) in artifacts {
if name.starts_with("audio.") {
audio.insert(name, artifact);
} else {
views.insert(name, artifact);
}
}
(views, audio)
}
/// The attachment names one environment's declared media travel under.
pub fn attachment_names(descriptor: &EnvironmentDescriptor) -> Vec<String> {
descriptor
.views
.iter()
.map(|view| view_attachment(&view.view_id))
.chain(
descriptor
.audio
.iter()
.map(|stream| audio_attachment(&stream.stream_id)),
)
.collect()
}
// ----------------------------------------------------------------------------------------------
// Acceptance
/// Every declared view must be present at exactly the producing boundary its delay implies.
///
/// A missing spectator frame is tolerable; a missing required sensory input is not. Neither is
/// one that arrived from an older boundary than the declared delay allows: the transition
/// fails instead of the session substituting whatever frame it happens to hold.
pub fn check_required_views(
descriptor: &EnvironmentDescriptor,
observation: &WorldObservation,
) -> DomainResult<()> {
for view in &descriptor.views {
let want = required_produced_step(view, observation.boundary);
match observation
.sensory_views
.iter()
.find(|given| given.view_id == view.view_id)
{
Some(given) if given.produced_step == want => {}
Some(given) => {
return Err(media_error(format!(
"view {} came from boundary {}, and its declared delay of {} requires {want}",
view.view_id, given.produced_step, view.observation_delay_steps
)));
}
None => {
return Err(media_error(format!(
"required sensory view {} is missing",
view.view_id
)));
}
}
}
Ok(())
}
/// Every declared audio stream produces exactly one chunk per transition.
///
/// The contract states the shape and the ordering of chunks, not whether one has to exist, so
/// this is MEDIA-01's choice and it is deliberate: a session that tolerates a silently missing
/// chunk cannot tell "this world produced no audio for this interval" from "the chunk was
/// lost", and the second is the case the retention rules care about. Boundary 0 has no
/// preceding interval and so carries no chunk.
pub fn check_required_audio(
descriptor: &EnvironmentDescriptor,
observation: &WorldObservation,
) -> DomainResult<()> {
if observation.boundary == 0 {
return Ok(());
}
for stream in &descriptor.audio {
if !observation
.audio
.iter()
.any(|chunk| chunk.stream_id == stream.stream_id)
{
return Err(media_error(format!(
"declared audio stream {} produced no chunk for this transition",
stream.stream_id
)));
}
}
Ok(())
}
/// Every declared audio stream's chunk sequence, one timeline per stream and epoch.
#[derive(Clone, Debug, Default)]
pub struct AudioTimelines(BTreeMap<String, AudioTimeline>);
impl AudioTimelines {
/// Fresh timelines for a new episode: every declared stream starts at origin zero.
pub fn fresh(descriptor: &EnvironmentDescriptor) -> AudioTimelines {
AudioTimelines(
descriptor
.audio
.iter()
.map(|stream| (stream.stream_id.clone(), AudioTimeline::fresh(stream, 0)))
.collect(),
)
}
/// Timelines for a new epoch after a restore: each stream resumes at its preserved sample
/// position, and each one's first chunk must mark a discontinuity.
///
/// A declared stream with no recorded position is an error. Resuming it at sample zero
/// would restart the episode's audio clock silently, which is exactly the best-effort
/// policy the restore rules refuse: crash restore *preserves* the sample position.
pub fn restored(
descriptor: &EnvironmentDescriptor,
positions: &BTreeMap<String, u64>,
) -> DomainResult<AudioTimelines> {
let mut timelines = BTreeMap::new();
for stream in &descriptor.audio {
let at = positions.get(&stream.stream_id).copied().ok_or_else(|| {
DomainError::before(
ErrorCode::IncompatibleState,
format!(
"audio stream {} has no restored sample position",
stream.stream_id
),
)
})?;
timelines.insert(stream.stream_id.clone(), AudioTimeline::restored_at(stream, at));
}
Ok(AudioTimelines(timelines))
}
/// Accepts one observation's chunks. Unknown streams and out-of-sequence chunks fail.
pub fn accept(
&mut self,
descriptor: &EnvironmentDescriptor,
observation: &WorldObservation,
) -> DomainResult<()> {
for chunk in &observation.audio {
let declared = descriptor
.audio_stream(&chunk.stream_id)
.ok_or_else(|| media_error(format!("audio stream {} is not declared", chunk.stream_id)))?;
let timeline = self
.0
.get_mut(&chunk.stream_id)
.ok_or_else(|| media_error(format!("audio stream {} has no timeline", chunk.stream_id)))?;
timeline.accept(chunk, declared).map_err(media_error)?;
}
Ok(())
}
/// Where each stream's next chunk may start.
pub fn positions(&self) -> BTreeMap<String, u64> {
self.0
.iter()
.map(|(id, timeline)| (id.clone(), timeline.next_sample()))
.collect()
}
pub fn accepted(&self, stream_id: &str) -> u64 {
self.0.get(stream_id).map_or(0, AudioTimeline::accepted)
}
}
// ----------------------------------------------------------------------------------------------
// Consumption
/// What one agent actually sensed, recorded where a test can read it.
///
/// The fake agent is the only thing that reads the pixels, so this is how "one shared image
/// reached both agents" is proved from the agents' side rather than from the producer's.
#[derive(Clone, Debug, Default)]
pub struct SensorLog(Arc<std::sync::Mutex<Vec<SensedView>>>);
/// One view an agent read: which boundary it was consumed at, which artifact it was, and the
/// digest of the bytes the agent actually read.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct SensedView {
pub boundary: u64,
pub view_id: Id,
pub artifact_id: String,
pub produced_step: u64,
pub digest: Digest,
}
impl SensorLog {
pub fn new() -> SensorLog {
SensorLog::default()
}
pub fn record(&self, view: SensedView) {
self.0.lock().expect("the sensor log is never poisoned").push(view);
}
pub fn entries(&self) -> Vec<SensedView> {
self.0.lock().expect("the sensor log is never poisoned").clone()
}
/// The artifacts this agent read, in order.
pub fn artifact_ids(&self) -> Vec<String> {
self.entries().into_iter().map(|v| v.artifact_id).collect()
}
}
/// One frame a spectator took off its subscription.
pub struct SpectatorFrame {
pub boundary: u64,
/// Every agent the committed snapshot carries, in publication order. A presentation
/// consumer is a multi-agent consumer: one snapshot holds the whole session.
pub agents: Vec<Id>,
pub sequence: u64,
/// How many undelivered snapshots were coalesced into this one.
pub replaced: u64,
pub view: ViewRef,
pub artifact: flybus::Artifact,
/// Each published chunk with the handle it travelled on.
pub audio: Vec<(AudioRef, flybus::Artifact)>,
}
/// A presentation-side consumer of committed snapshots.
///
/// It subscribes `latest` with finite credits, which is the spectator row of the domain
/// retention table: new snapshots replace its queued value, it never blocks the session, and
/// the only thing a slow one exhausts is its own credits.
pub struct Spectator {
subscription: flybus::Subscription,
held: Vec<flybus::Message>,
seen: u64,
coalesced: u64,
}
impl Spectator {
/// Subscribes to `topic` in latest mode with `credits` in flight.
///
/// A latest subscription always has exactly one queued value; `credits` is its in-flight
/// bound, which the router limits (two by default). Finite credits are the spectator row
/// of the domain retention table: they are the only thing a slow viewer exhausts.
pub async fn attach(
client: &flybus::Client,
topic: &str,
credits: u32,
) -> Result<Spectator, flybus::BusError> {
let subscription = client
.subscribe(
topic,
flybus::SubscriptionConfig::latest().in_flight(credits).replay(true),
)
.await?;
Ok(Spectator {
subscription,
held: Vec::new(),
seen: 0,
coalesced: 0,
})
}
/// Takes the next snapshot, reads its frame and releases the delivery.
pub async fn take_frame(&mut self) -> Option<SpectatorFrame> {
let message = self.subscription.next().await?;
self.seen += 1;
self.coalesced += message.replaced();
let frame = snapshot_frame(&message);
drop(message);
frame
}
/// Takes the next snapshot message itself, for a renderer that keeps its own handle
/// after the message is gone.
pub async fn next_message(&mut self) -> Option<flybus::Message> {
let message = self.subscription.next().await?;
self.seen += 1;
self.coalesced += message.replaced();
Some(message)
}
/// Takes a snapshot without consuming it, which is what a viewer that stops rendering
/// does. Its credits run out and nothing else in the session notices.
pub async fn hold_one(&mut self) -> bool {
match self.subscription.next().await {
Some(message) => {
self.seen += 1;
self.coalesced += message.replaced();
self.held.push(message);
true
}
None => false,
}
}
/// Takes a snapshot without consuming it if one is queued right now.
pub fn try_hold_one(&mut self) -> bool {
match self.subscription.try_next() {
Some(message) => {
self.seen += 1;
self.coalesced += message.replaced();
self.held.push(message);
true
}
None => false,
}
}
/// Releases everything this spectator was holding, returning its credits.
pub fn release(&mut self) {
self.held.clear();
}
pub fn held(&self) -> usize {
self.held.len()
}
pub fn seen(&self) -> u64 {
self.seen
}
/// How many snapshots were replaced in this spectator's queue while it was busy.
pub fn coalesced(&self) -> u64 {
self.coalesced
}
}
/// Reads one committed snapshot's first view and its handle.
pub fn snapshot_frame(message: &flybus::Message) -> Option<SpectatorFrame> {
let payload = message.payload();
let boundary = payload
.get("scope")
.and_then(|s| s.get("step"))
.and_then(serde_json::Value::as_str)
.and_then(|s| s.parse::<u64>().ok())?;
let media = payload.get("media")?;
let views = media.get("views")?.as_array()?;
let view = ViewRef::from_json(views.first()?).ok()?;
// An unreadable chunk, or one whose handle is not attached, makes the whole snapshot
// unreadable rather than a snapshot that quietly has less audio in it than was published.
let mut audio = Vec::new();
for value in media.get("audio")?.as_array()? {
let chunk = AudioRef::from_json(value).ok()?;
let artifact = message.artifact(&audio_attachment(&chunk.stream_id)).ok()?;
audio.push((chunk, artifact));
}
let artifact = message.artifact(&view_attachment(&view.view_id)).ok()?;
let agents = payload
.get("agents")
.and_then(serde_json::Value::as_array)
.map(|agents| {
agents
.iter()
.filter_map(|a| a.get("agentId").and_then(serde_json::Value::as_str))
.map(str::to_owned)
.collect()
})
.unwrap_or_default();
Some(SpectatorFrame {
boundary,
agents,
sequence: message.topic_sequence(),
replaced: message.replaced(),
view,
artifact,
audio,
})
}
/// Takes the frame out of a message and drops the message, as a renderer that finishes later
/// does. The delivery stays alive because the extracted handle still owns it.
pub fn detach_frame(message: flybus::Message) -> Option<(ViewRef, flybus::Artifact)> {
let frame = snapshot_frame(&message)?;
drop(message);
Some((frame.view, frame.artifact))
}
// ----------------------------------------------------------------------------------------------
// Persistent assets
/// The preprovisioned local registry an `AssetRef` names.
///
/// An asset is installed and verified before a run; it is not a path, a URL or something a
/// worker fetches. Nothing in this registry can be addressed by an `ArtifactRef`, and
/// [`AssetRegistry::import`] hands back a fresh transient artifact rather than turning the
/// asset into one.
#[derive(Clone, Debug, Default)]
pub struct AssetRegistry {
installed: BTreeMap<Id, (AssetRef, Vec<u8>)>,
}
impl AssetRegistry {
pub fn new() -> AssetRegistry {
AssetRegistry::default()
}
/// Installs content, verifying that it is the content the reference claims.
pub fn install(&mut self, asset: AssetRef, bytes: Vec<u8>) -> DomainResult<()> {
asset.validate().map_err(DomainError::invalid)?;
if asset.byte_length != bytes.len() as u64 {
return Err(DomainError::before(
ErrorCode::IdentityMismatch,
format!("asset {}: byteLength is not the installed length", asset.id),
));
}
if asset.digest != digest_of_bytes(&bytes) {
return Err(DomainError::before(
ErrorCode::IdentityMismatch,
format!("asset {}: digest is not the installed content", asset.id),
));
}
self.installed.insert(asset.id.clone(), (asset, bytes));
Ok(())
}
/// Resolves installed content. Identity and digest must both match: the same id with
/// another digest is a different asset, not an upgrade.
pub fn resolve(&self, asset: &AssetRef) -> DomainResult<&[u8]> {
match self.installed.get(&asset.id) {
Some((installed, bytes)) if installed == asset => Ok(bytes),
Some(_) => Err(DomainError::before(
ErrorCode::IdentityMismatch,
format!("asset {} is installed with another identity", asset.id),
)),
None => Err(DomainError::before(
ErrorCode::IdentityMismatch,
format!("asset {} is not installed", asset.id),
)),
}
}
pub fn contains(&self, asset: &AssetRef) -> bool {
self.resolve(asset).is_ok()
}
/// Imports installed content into the bus as a fresh immutable artifact.
///
/// The artifact is sealed against the asset's digest, which is mandatory for a persistent
/// asset import, and its identity belongs to the current store incarnation. The asset
/// reference is unchanged and outlives it.
pub async fn import(
&self,
client: &flybus::Client,
asset: &AssetRef,
) -> DomainResult<flybus::Artifact> {
let bytes = self.resolve(asset)?;
let mut writer = client
.artifacts()
.allocate(asset.byte_length, "application/octet-stream")
.await
.map_err(|e| store_error("allocate", &e.message))?;
writer
.write_all(bytes)
.map_err(|e| store_error("write", &e.to_string()))?;
let artifact = writer
.seal_with_digest(Some(asset.digest.clone()))
.await
.map_err(|e| store_error("seal", &e.message))?;
fly_session_types::media::check_imported_asset(asset, artifact.reference())
.map_err(|e| DomainError::before(ErrorCode::IdentityMismatch, e))?;
Ok(artifact)
}
}

View file

@ -0,0 +1,176 @@
//! Latency and resource samples, for the measurements the implementation guide's section 5
//! asks every slice to report.
//!
//! These are local synthetic timings on one machine. No host capacity claim follows from any
//! number this module produces, and nothing here is a gameplay latency goal: the percentiles
//! exist so that the three execution modes can be compared against each other.
use std::collections::BTreeMap;
/// One sample set's order statistics, by nearest rank.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub struct Percentiles {
pub count: usize,
pub p50_ns: u64,
pub p95_ns: u64,
pub p99_ns: u64,
pub max_ns: u64,
}
impl Percentiles {
fn of(sorted: &[u64]) -> Percentiles {
let rank = |p: f64| -> u64 {
if sorted.is_empty() {
return 0;
}
let n = sorted.len() as f64;
let index = (p * n).ceil() as usize;
sorted[index.clamp(1, sorted.len()) - 1]
};
Percentiles {
count: sorted.len(),
p50_ns: rank(0.50),
p95_ns: rank(0.95),
p99_ns: rank(0.99),
max_ns: sorted.last().copied().unwrap_or_default(),
}
}
pub fn p50_us(&self) -> f64 {
self.p50_ns as f64 / 1000.0
}
pub fn p95_us(&self) -> f64 {
self.p95_ns as f64 / 1000.0
}
pub fn p99_us(&self) -> f64 {
self.p99_ns as f64 / 1000.0
}
}
/// Named duration samples. One name is one measured path: a domain method, or the
/// coordinator's whole critical path for a transition.
#[derive(Clone, Debug, Default)]
pub struct Metrics {
samples: BTreeMap<String, Vec<u64>>,
}
impl Metrics {
pub fn record(&mut self, what: &str, elapsed: std::time::Duration) {
let ns = u64::try_from(elapsed.as_nanos()).unwrap_or(u64::MAX);
self.samples.entry(what.to_owned()).or_default().push(ns);
}
pub fn percentiles(&self, what: &str) -> Option<Percentiles> {
let mut values = self.samples.get(what)?.clone();
values.sort_unstable();
Some(Percentiles::of(&values))
}
pub fn names(&self) -> Vec<String> {
self.samples.keys().cloned().collect()
}
pub fn count(&self, what: &str) -> usize {
self.samples.get(what).map(Vec::len).unwrap_or_default()
}
pub fn clear(&mut self) {
self.samples.clear();
}
}
/// This process's peak resident set, in KiB, from its own status file.
pub fn peak_rss_kib() -> Option<u64> {
peak_rss_of("/proc/self/status")
}
/// One child process's peak resident set, in KiB. `None` once the child is gone.
pub fn peak_rss_kib_of(pid: u32) -> Option<u64> {
peak_rss_of(&format!("/proc/{pid}/status"))
}
fn peak_rss_of(path: &str) -> Option<u64> {
let text = std::fs::read_to_string(path).ok()?;
for line in text.lines() {
if let Some(rest) = line.strip_prefix("VmHWM:") {
return rest.split_whitespace().next()?.parse().ok();
}
}
None
}
/// How many physical cores this machine has, counted as distinct (package, core) pairs.
///
/// Falls back to the logical count, which is what a thread budget has to use when the
/// topology cannot be read.
pub fn physical_cores() -> usize {
if let Ok(text) = std::fs::read_to_string("/proc/cpuinfo") {
let mut pairs: std::collections::BTreeSet<(String, String)> =
std::collections::BTreeSet::new();
let (mut package, mut core) = (None, None);
for line in text.lines() {
if line.trim().is_empty() {
if let (Some(p), Some(c)) = (package.take(), core.take()) {
pairs.insert((p, c));
}
continue;
}
if let Some((key, value)) = line.split_once(':') {
match key.trim() {
"physical id" => package = Some(value.trim().to_owned()),
"core id" => core = Some(value.trim().to_owned()),
_ => {}
}
}
}
if let (Some(p), Some(c)) = (package, core) {
pairs.insert((p, c));
}
if !pairs.is_empty() {
return pairs.len();
}
}
logical_cores()
}
/// How many hardware threads this machine reports.
pub fn logical_cores() -> usize {
std::thread::available_parallelism().map(|n| n.get()).unwrap_or(1)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn percentiles_use_nearest_rank() {
let mut m = Metrics::default();
for ns in 1..=100u64 {
m.record("x", std::time::Duration::from_nanos(ns));
}
let p = m.percentiles("x").expect("recorded");
assert_eq!(p.count, 100);
assert_eq!(p.p50_ns, 50);
assert_eq!(p.p95_ns, 95);
assert_eq!(p.p99_ns, 99);
assert_eq!(p.max_ns, 100);
assert!(m.percentiles("y").is_none());
}
#[test]
fn a_single_sample_is_every_percentile() {
let mut m = Metrics::default();
m.record("x", std::time::Duration::from_nanos(7));
let p = m.percentiles("x").expect("recorded");
assert_eq!((p.p50_ns, p.p95_ns, p.p99_ns, p.max_ns), (7, 7, 7, 7));
}
#[test]
fn the_machine_reports_at_least_one_core() {
assert!(physical_cores() >= 1);
assert!(logical_cores() >= 1);
assert!(peak_rss_kib().unwrap_or(1) > 0);
}
}

View file

@ -188,6 +188,12 @@ pub trait WorkerEndpoint: Send + 'static {
fn capabilities(&self) -> Vec<Id>;
fn status_cell(&self) -> StatusCell;
/// The thread allocation this worker's launcher started it within.
///
/// `Worker.Hello` reports it, so a caller bounded by `workers-v1`'s "within launcher
/// allocation" can read the allocation instead of being told it out of band.
fn worker_threads(&self) -> u64;
/// The domain methods this endpoint implements, beyond the common `Worker.*` set.
/// Anything else returns UNSUPPORTED without entering the endpoint.
fn methods(&self) -> Vec<&'static str>;
@ -224,6 +230,22 @@ impl WorkerHandle {
let _ = self.task.await;
self.client.close().await;
}
/// Stops serving without waiting. The connection closes when the last handle to it is
/// dropped, which this does. For a supervisor's `Drop`, where there is no runtime to wait
/// on.
pub fn abort(self) {
self.task.abort();
}
/// Waits until the worker stops serving, which `Worker.Shutdown` makes it do.
///
/// A worker process awaits this and then exits, so the supervisor's `Worker.Shutdown` and
/// the process's exit are the same event rather than two racing ones.
pub async fn join(self) {
let _ = self.task.await;
self.client.close().await;
}
}
/// Registers `service_name` and serves `endpoint` on it until the service ends or Shutdown.
@ -259,7 +281,7 @@ async fn run<E: WorkerEndpoint>(
) {
// Identity and capabilities are fixed for the endpoint's lifetime, so the shell reads them
// once and never takes the endpoint mutex to answer Hello or Status.
let (worker_id, incarnation_id, session_id, role, capabilities, status, methods) = {
let (worker_id, incarnation_id, session_id, role, capabilities, status, methods, threads) = {
let e = endpoint.lock().await;
(
e.worker_id(),
@ -269,6 +291,7 @@ async fn run<E: WorkerEndpoint>(
e.capabilities(),
e.status_cell(),
e.methods(),
e.worker_threads(),
)
};
let mut running: Vec<tokio::task::JoinHandle<()>> = Vec::new();
@ -305,6 +328,7 @@ async fn run<E: WorkerEndpoint>(
&incarnation_id,
role,
&capabilities,
threads,
);
let _ = responder.reply(outcome.to_outcome(), &[]).await;
continue;
@ -606,6 +630,7 @@ fn failure(
failure_outcome(request_id, worker_id, incarnation_id, scope, error)
}
#[allow(clippy::too_many_arguments)]
fn hello(
request: &SessionRpcRequest,
session_id: &Id,
@ -613,6 +638,7 @@ fn hello(
incarnation_id: &Id,
role: Role,
capabilities: &[Id],
worker_threads: u64,
) -> SessionRpcOutcome {
let params: HelloParams = match HelloParams::from_json(&request.params) {
Ok(params) => params,
@ -668,6 +694,7 @@ fn hello(
capabilities: capabilities.to_vec(),
max_agents: MAX_AGENTS as u64,
max_ports: MAX_PORTS as u64,
worker_threads,
};
success(
request,

View file

@ -4,7 +4,7 @@
use std::time::Duration;
use fly_session::harness::{HarnessConfig, SessionHarness, Via};
use fly_session::harness::{ExecutionMode, HarnessConfig, SessionHarness, Via};
use fly_session::types::*;
pub const WAIT: Duration = Duration::from_secs(20);
@ -94,3 +94,55 @@ pub async fn within<T>(what: &str, f: impl std::future::Future<Output = T>) -> T
Err(_) => panic!("{what}: timed out"),
}
}
/// Generates one test per execution mode from an `async fn name(mode: ExecutionMode)`.
///
/// The separate-process mode is the SESSION-02 subject; the other two are the variants it is
/// compared against, and a row that holds in one must hold in all three.
#[macro_export]
macro_rules! all_modes {
($($name:ident),* $(,)?) => {
mod in_process {
$(
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn $name() {
super::$name($crate::common::mode_in_process()).await
}
)*
}
mod dedicated_thread {
$(
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn $name() {
super::$name($crate::common::mode_thread()).await
}
)*
}
mod separate_process {
$(
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn $name() {
super::$name($crate::common::mode_process()).await
}
)*
}
};
}
pub fn mode_in_process() -> ExecutionMode {
ExecutionMode::InProcess
}
pub fn mode_thread() -> ExecutionMode {
ExecutionMode::Thread
}
pub fn mode_process() -> ExecutionMode {
ExecutionMode::Process
}
/// A fixture in one execution mode. The transport is the mode's own: a separate process
/// reaches the router only over a socket.
pub async fn mode_fixture(mode: ExecutionMode, config: HarnessConfig) -> Fixture {
fixture(Via::Unix, HarnessConfig { mode, ..config }).await
}

View file

@ -33,6 +33,15 @@ both_transports!(
const STEPS: u64 = 4;
const INJECT_AT: u64 = 2;
/// The mutation counter of a participant running in this process. These suites are all
/// in-process compositions, so it is always there; SESSION-02's are not, and read it over the
/// bus instead.
fn local_mutations(f: &Fixture, agent_id: &Id) -> u64 {
f.harness
.agent_mutations(agent_id)
.expect("an in-process participant keeps its counter in this process")
}
/// What a run of the standard composition produced.
struct Run {
behaviour: Vec<String>,
@ -51,8 +60,8 @@ async fn run_with(via: Via, injections: Injections) -> Run {
let run = Run {
behaviour: f.harness.coordinator.trace.behavior(),
mutations: vec![
(fly_a(), f.harness.agent_mutations(&fly_a())),
(fly_b(), f.harness.agent_mutations(&fly_b())),
(fly_a(), local_mutations(&f, &fly_a())),
(fly_b(), local_mutations(&f, &fly_b())),
],
counter: f
.harness

View file

@ -0,0 +1,862 @@
//! MEDIA-01 acceptance: native observations on the bus artifact, and the presentation handoff.
//!
//! Every test here is one of the slice's acceptance bullets or one row of the domain retention
//! table in `state-media-v1` section 3. The shape rules themselves are proved against the
//! contract crate in `fly-session-types/tests/media_shapes.rs`; these prove the session's use
//! of them: one shared image, spectators that cannot touch sensory state, a renderer that
//! keeps its handle, and a persistent asset that is not a transient artifact.
mod common;
use std::collections::BTreeMap;
use std::time::Duration;
use serde_json::Map;
use common::{Fixture, default_fixture, fixture, fly_a, fly_b, mode_fixture, within};
use fly_session::environment::{
AUDIO_STREAM_ID, CHANNELS, EnvironmentFaults, SAMPLE_RATE, VIEW_HEIGHT, VIEW_WIDTH,
synthetic_asset,
};
use fly_session::harness::{ExecutionMode, HarnessConfig, Via};
use fly_session::media::{
AssetRegistry, AudioSource, AudioTimelines, SensedView, Spectator, SpectatorFrame,
arena_frame, audio_attachment, detach_frame, view_attachment,
};
use fly_session::phase::Phase;
use fly_session::types::*;
use fly_session_types::media::{AudioTimeline, check_imported_asset, require_finite_samples};
both_transports!(
one_shared_image_reaches_both_agents_through_owned_attachments,
a_spectator_cannot_corrupt_sensory_state,
a_slow_spectator_exhausts_only_its_own_credits,
delayed_rendering_retains_its_handle_after_the_message_drops,
a_declared_render_delay_repeats_o0_until_the_pipeline_fills,
an_extra_delayed_sensory_view_fails_the_step,
a_frame_of_the_wrong_length_fails_the_step,
overlapping_audio_fails_the_step,
one_audio_chunk_per_boundary_with_an_exact_sample_budget,
required_agent_input_is_never_coalesced_while_spectator_snapshots_are,
a_persistent_asset_and_a_transient_artifact_are_different_identities,
a_restored_audio_source_resumes_and_marks_the_discontinuity,
a_missing_audio_chunk_fails_the_step,
restored_timelines_need_every_declared_streams_position,
a_cadence_that_does_not_divide_the_sample_rate_still_lands_on_whole_samples,
);
all_modes!(
the_media_path_works_in_every_execution_mode,
every_media_fault_fires_in_every_execution_mode,
);
const STEPS: u64 = 3;
/// Polls until `ok` holds, so a test never asserts a collection that has not happened yet.
async fn until(what: &str, mut ok: impl FnMut() -> bool) {
let deadline = std::time::Instant::now() + Duration::from_secs(10);
while !ok() {
assert!(std::time::Instant::now() < deadline, "{what}: never happened");
tokio::time::sleep(Duration::from_millis(5)).await;
}
}
/// Drains a latest subscription until the snapshot for `boundary` arrives.
///
/// A latest subscription converges on the newest value rather than delivering every one, so a
/// test that wants a particular committed boundary reads until it gets there.
async fn frame_at(spectator: &mut Spectator, boundary: u64) -> SpectatorFrame {
for _ in 0..64 {
let frame = within("snapshot", spectator.take_frame())
.await
.expect("a snapshot");
if frame.boundary == boundary {
return frame;
}
}
panic!("the spectator never reached boundary {boundary}");
}
/// The views one agent read.
///
/// The sensor log is shared memory, so it is readable only where that agent lives. These tests
/// run in the default in-process composition; the process-mode test below asserts the media
/// path over the bus instead, which is what crosses a process boundary.
fn sensed(f: &Fixture, agent_id: &Id) -> Vec<SensedView> {
f.harness
.sensor_log(agent_id)
.expect("this composition keeps its agents in this process")
.entries()
}
/// How many native frames the world rendered, for a world in this process.
fn renders(f: &Fixture) -> u64 {
f.harness
.renders()
.expect("this composition keeps its world in this process")
}
fn boundaries(entries: &[SensedView]) -> Vec<u64> {
entries.iter().map(|e| e.boundary).collect()
}
fn produced(entries: &[SensedView]) -> Vec<u64> {
entries.iter().map(|e| e.produced_step).collect()
}
/// One image per boundary reaches both agents as an owned attachment, and the same handle is
/// published for presentation. There is no second copy of the pixels anywhere.
async fn one_shared_image_reaches_both_agents_through_owned_attachments(via: Via) {
let mut f = default_fixture(via).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let observer = f.harness.observer().await.unwrap();
let topic = f.harness.coordinator.topics().snapshots.clone();
let mut spectator = Spectator::attach(&observer, &topic, 2).await.unwrap();
within("run", f.harness.coordinator.run(STEPS)).await.unwrap();
// One render per boundary: forwarding the handle to two agents and to publication does not
// render or copy it again.
assert_eq!(
renders(&f),
STEPS + 1,
"one native frame per boundary, whatever the number of recipients"
);
let a = sensed(&f, &fly_a());
let b = sensed(&f, &fly_b());
assert_eq!(boundaries(&a), (0..=STEPS).collect::<Vec<_>>());
assert_eq!(a, b, "both agents read the same artifact and the same bytes");
assert_eq!(produced(&a), (0..=STEPS).collect::<Vec<_>>(), "no declared delay");
// The handle the agents were given is the handle presentation was published.
let published = frame_at(&mut spectator, STEPS).await;
assert_eq!(
published.agents,
vec![fly_a(), fly_b()],
"one snapshot carries the whole multi-agent session"
);
let last = a.last().expect("an entry per boundary");
assert_eq!(published.view.pixels.artifact_id, last.artifact_id);
assert_eq!(published.view.produced_step, last.produced_step);
let bytes = published.artifact.read_all().await.expect("the published frame");
assert_eq!(bytes.len() as u64, VIEW_WIDTH * VIEW_HEIGHT * 4);
assert_eq!(digest_of_bytes(&bytes), last.digest, "the bytes both agents read");
// The coordinator holds exactly one handle per declared view and stream at this boundary.
let handles = f.harness.coordinator.media_handles();
assert_eq!(handles.len(), 2, "one view and one audio chunk: {handles:?}");
assert!(handles.iter().any(|(name, r)| *name == view_attachment("arena")
&& r.artifact_id == last.artifact_id));
f.shutdown().await;
}
/// A spectator reads committed snapshots and cannot touch what the agents sense: it has no
/// authority over the environment, and its own reads leave sensory state exactly as produced.
async fn a_spectator_cannot_corrupt_sensory_state(via: Via) {
let mut f = default_fixture(via).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let observer = f.harness.observer().await.unwrap();
let topic = f.harness.coordinator.topics().snapshots.clone();
let mut spectator = Spectator::attach(&observer, &topic, 2).await.unwrap();
// Naming the environment is not authority to drive it: a subscriber cannot call it.
let refused = observer
.call("env.arena", None, "Environment.Advance", Map::new(), &[])
.await;
assert!(
refused.is_err(),
"a spectator must not be able to call the environment"
);
within("run", f.harness.coordinator.run(STEPS)).await.unwrap();
// The spectator reads the snapshot and releases it while the session keeps stepping.
let frame = within("snapshot", spectator.take_frame()).await.expect("a snapshot");
let seen = frame.artifact.read_all().await.expect("readable");
drop(frame);
within("run more", f.harness.coordinator.run(STEPS)).await.unwrap();
let a = sensed(&f, &fly_a());
let b = sensed(&f, &fly_b());
assert_eq!(boundaries(&a), (0..=2 * STEPS).collect::<Vec<_>>());
assert_eq!(a, b);
assert_eq!(
f.harness.coordinator.committed_boundary(),
Some(2 * STEPS),
"the spectator changed nothing about the session's progress"
);
// What the spectator read was one of the frames the agents encoded, unchanged.
let digest = digest_of_bytes(&seen);
assert!(
a.iter().any(|entry| entry.digest == digest),
"a spectator sees the committed frame, and only reads it"
);
f.shutdown().await;
}
/// A slow spectator exhausts its own credits. New snapshots replace its queued value; the
/// world never waits for it.
async fn a_slow_spectator_exhausts_only_its_own_credits(via: Via) {
let mut f = default_fixture(via).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let observer = f.harness.observer().await.unwrap();
let topic = f.harness.coordinator.topics().snapshots.clone();
// One credit and one queued value: the smallest spectator the bus allows.
let mut slow = Spectator::attach(&observer, &topic, 1).await.unwrap();
let mut fast = Spectator::attach(&observer, &topic, 2).await.unwrap();
// The slow one takes one snapshot and stops rendering, holding its only credit.
assert!(within("first snapshot", slow.hold_one()).await);
assert_eq!(slow.held(), 1);
let steps = 5;
within("run", f.harness.coordinator.run(steps)).await.unwrap();
assert_eq!(f.harness.coordinator.stats().advances, steps);
// With its credit in use it receives nothing more, and its queued value is replaced.
assert!(!slow.try_hold_one(), "a spectator out of credits gets nothing more");
assert_eq!(slow.seen(), 1);
slow.release();
let next = within("after release", slow.hold_one()).await;
assert!(next, "releasing its own delivery returns its own credit");
assert!(
slow.coalesced() > 0,
"new snapshots replaced the value queued for a spectator that was not reading"
);
// The session and the other spectator are untouched.
let latest = frame_at(&mut fast, steps).await;
assert_eq!(
latest.boundary, steps,
"the reading spectator reaches the latest boundary"
);
let a = sensed(&f, &fly_a());
assert_eq!(boundaries(&a), (0..=steps).collect::<Vec<_>>());
f.shutdown().await;
}
/// A renderer keeps its extracted handle after the message is gone, and after the boundary it
/// came from has been replaced everywhere else.
async fn delayed_rendering_retains_its_handle_after_the_message_drops(via: Via) {
let mut f = default_fixture(via).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let observer = f.harness.observer().await.unwrap();
let topic = f.harness.coordinator.topics().snapshots.clone();
let mut spectator = Spectator::attach(&observer, &topic, 2).await.unwrap();
within("step", f.harness.coordinator.run(1)).await.unwrap();
// Take the message, keep only the frame, and drop the message itself.
let message = within("snapshot", spectator.next_message()).await.expect("a snapshot");
let (view, artifact) = detach_frame(message).expect("a frame in the snapshot");
// Everything else moves on: the coordinator drops that boundary's handles, and the topic's
// retained value is replaced twice.
within("more steps", f.harness.coordinator.run(2)).await.unwrap();
let handles = f.harness.coordinator.media_handles();
assert!(
!handles.iter().any(|(_, r)| r.artifact_id == view.pixels.artifact_id),
"the session no longer holds the frame the renderer is still using"
);
// The rendering finishes now, long after its message is gone.
let bytes = artifact.read_all().await.expect("the guard kept the bytes alive");
assert_eq!(bytes.len() as u64, VIEW_WIDTH * VIEW_HEIGHT * 4);
let entries = sensed(&f, &fly_a());
let at_boundary = entries
.iter()
.find(|e| e.produced_step == view.produced_step)
.expect("the agents encoded this frame too");
assert_eq!(
digest_of_bytes(&bytes),
at_boundary.digest,
"the retained handle still reads exactly the frame that was published"
);
f.shutdown().await;
}
/// A declared render delay repeats `O[0]` while the pipeline fills, then advances one frame
/// per boundary. The repetition is the same artifact, not a re-render.
async fn a_declared_render_delay_repeats_o0_until_the_pipeline_fills(via: Via) {
let config = HarnessConfig {
observation_delay_steps: 2,
..HarnessConfig::default()
};
let mut f = fixture(via, config).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
within("run", f.harness.coordinator.run(4)).await.unwrap();
let a = sensed(&f, &fly_a());
assert_eq!(boundaries(&a), vec![0, 1, 2, 3, 4]);
assert_eq!(
produced(&a),
vec![0, 0, 0, 1, 2],
"max(0, boundary - 2) at every boundary"
);
assert_eq!(a[0].artifact_id, a[1].artifact_id);
assert_eq!(a[0].artifact_id, a[2].artifact_id, "O[0] repeats while the delay fills");
assert_ne!(a[2].artifact_id, a[3].artifact_id, "then the pipeline advances");
assert_ne!(a[3].artifact_id, a[4].artifact_id);
// The world still renders once per boundary; the delay is a queue, not a missing frame.
assert_eq!(renders(&f), 5);
f.shutdown().await;
}
/// Beyond the declared delay, an extra-delayed sensory view is a step failure, never an
/// arbitrary latest frame.
async fn an_extra_delayed_sensory_view_fails_the_step(via: Via) {
let config = HarnessConfig {
environment_faults: EnvironmentFaults {
stale_view_at_boundary: Some(2),
..EnvironmentFaults::default()
},
..HarnessConfig::default()
};
let mut f = fixture(via, config).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let failure = within("run", f.harness.coordinator.run(STEPS))
.await
.expect_err("a stale frame fails the transition");
assert_eq!(failure.error.code, ErrorCode::BufferInvalid);
assert_eq!(failure.error.mutation, MutationCertainty::Unknown);
assert_eq!(f.harness.coordinator.phase(), Phase::Failed);
assert_eq!(
f.harness.coordinator.stats().advances,
1,
"the transition that met the stale frame committed nothing"
);
// The agents did not encode the stale frame.
let a = sensed(&f, &fly_a());
assert_eq!(boundaries(&a), vec![0, 1]);
f.shutdown().await;
}
/// A frame whose artifact is not `rowStride x height` bytes fails the step.
async fn a_frame_of_the_wrong_length_fails_the_step(via: Via) {
let config = HarnessConfig {
environment_faults: EnvironmentFaults {
truncated_view_at_boundary: Some(1),
..EnvironmentFaults::default()
},
..HarnessConfig::default()
};
let mut f = fixture(via, config).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let failure = within("run", f.harness.coordinator.run(1))
.await
.expect_err("a short frame fails the transition");
assert_eq!(failure.error.code, ErrorCode::BufferInvalid);
assert_eq!(f.harness.coordinator.phase(), Phase::Failed);
assert_eq!(f.harness.coordinator.stats().advances, 0);
f.shutdown().await;
}
/// Within an epoch, an audio chunk that starts inside the previous one is refused.
async fn overlapping_audio_fails_the_step(via: Via) {
let config = HarnessConfig {
environment_faults: EnvironmentFaults {
overlapping_audio_at_boundary: Some(2),
..EnvironmentFaults::default()
},
..HarnessConfig::default()
};
let mut f = fixture(via, config).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let failure = within("run", f.harness.coordinator.run(STEPS))
.await
.expect_err("an overlapping chunk fails the transition");
assert_eq!(failure.error.code, ErrorCode::BufferInvalid);
assert_eq!(f.harness.coordinator.stats().advances, 1);
f.shutdown().await;
}
/// One chunk per boundary, with an exact sample budget, finite samples and the byte length the
/// descriptor implies. Audio is published for presentation and never enters sensory input.
async fn one_audio_chunk_per_boundary_with_an_exact_sample_budget(via: Via) {
let mut f = default_fixture(via).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let observer = f.harness.observer().await.unwrap();
let topic = f.harness.coordinator.topics().snapshots.clone();
let mut spectator = Spectator::attach(&observer, &topic, 2).await.unwrap();
within("run", f.harness.coordinator.run(STEPS)).await.unwrap();
// 48 kHz at 60 Hz is exactly 800 frames a step, and the positions are contiguous.
let per_step = SAMPLE_RATE / f.harness.config.step_hz;
let positions = f.harness.coordinator.audio_positions();
assert_eq!(positions[AUDIO_STREAM_ID], per_step * STEPS);
let observation = f.harness.coordinator.observation().expect("an observation").clone();
assert_eq!(observation.audio.len(), 1, "one chunk per boundary");
let chunk = &observation.audio[0];
assert_eq!(chunk.sample_frames, per_step);
assert_eq!(chunk.first_sample, per_step * (STEPS - 1));
assert!(!chunk.discontinuity, "an uninterrupted epoch has no discontinuity");
assert_eq!(chunk.samples.byte_length, per_step * CHANNELS * 4);
// Sensory input is pixels only: audio never becomes an agent's input.
assert!(observation.sensory_views.iter().all(|v| v.view_id == "arena"));
let published = frame_at(&mut spectator, STEPS).await;
let (published_chunk, artifact) = published.audio.first().expect("the published chunk");
assert_eq!(published_chunk.samples, chunk.samples);
assert_eq!(
artifact.reference().artifact_id,
chunk.samples.artifact_id,
"the same owned handle is published, not a copy"
);
let bytes = artifact.read_all().await.expect("the published chunk");
assert_eq!(bytes.len() as u64, per_step * CHANNELS * 4);
require_finite_samples(&bytes).expect("native samples are finite f32");
assert_eq!(
published.audio.len(),
1,
"the attachment list names every published chunk"
);
assert_eq!(audio_attachment(AUDIO_STREAM_ID), "audio.arena");
f.shutdown().await;
}
/// A declared stream that produces no chunk for a transition is a step failure, not a silently
/// shorter epoch.
async fn a_missing_audio_chunk_fails_the_step(via: Via) {
let config = HarnessConfig {
environment_faults: EnvironmentFaults {
omit_audio_at_boundary: Some(2),
..EnvironmentFaults::default()
},
..HarnessConfig::default()
};
let mut f = fixture(via, config).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let failure = within("run", f.harness.coordinator.run(STEPS))
.await
.expect_err("a missing chunk fails the transition");
assert_eq!(failure.error.code, ErrorCode::BufferInvalid);
assert_eq!(f.harness.coordinator.stats().advances, 1);
// Boundary 0 has no preceding interval, so its empty audio list is not a missing chunk.
let mut clean = default_fixture(via).await;
within("bootstrap", clean.harness.coordinator.bootstrap()).await.unwrap();
assert!(clean.harness.coordinator.observation().unwrap().audio.is_empty());
clean.shutdown().await;
f.shutdown().await;
}
/// Restored timelines resume every declared stream at its recorded position. A stream with no
/// recorded position is refused, not restarted at zero.
async fn restored_timelines_need_every_declared_streams_position(via: Via) {
let mut f = default_fixture(via).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
within("run", f.harness.coordinator.run(1)).await.unwrap();
let descriptor = f.harness.coordinator.descriptor().expect("a descriptor").clone();
let observation = f.harness.coordinator.observation().expect("an observation").clone();
// A fresh epoch accepts the transition's chunk and advances its position.
let mut fresh = AudioTimelines::fresh(&descriptor);
fresh
.accept(&descriptor, &observation)
.expect("the first chunk of a fresh epoch");
assert_eq!(fresh.accepted(AUDIO_STREAM_ID), 1);
let chunk = observation.audio.first().expect("a chunk").clone();
assert_eq!(
fresh.positions()[AUDIO_STREAM_ID],
chunk.first_sample + chunk.sample_frames
);
// A restore with nothing recorded for a declared stream is an error, not sample zero.
let missing = AudioTimelines::restored(&descriptor, &BTreeMap::new())
.expect_err("a declared stream needs its preserved position");
assert_eq!(missing.code, ErrorCode::IncompatibleState);
// With the position recorded, the restored epoch resumes there and its first chunk must
// mark the discontinuity.
let positions = f.harness.coordinator.audio_positions();
let mut resumed = observation.clone();
let resumed_chunk = resumed.audio.first_mut().expect("a chunk");
resumed_chunk.first_sample = positions[AUDIO_STREAM_ID];
resumed_chunk.discontinuity = false;
let mut restored = AudioTimelines::restored(&descriptor, &positions).expect("positions");
restored
.accept(&descriptor, &resumed)
.expect_err("the first chunk after a restore marks discontinuity");
resumed.audio.first_mut().expect("a chunk").discontinuity = true;
let mut restored = AudioTimelines::restored(&descriptor, &positions).expect("positions");
restored
.accept(&descriptor, &resumed)
.expect("the restored epoch resumes at the preserved position");
assert_eq!(restored.accepted(AUDIO_STREAM_ID), 1);
f.shutdown().await;
}
/// A world cadence that does not divide the sample rate still produces whole samples, with the
/// remainder carried rather than rounded: seven steps of a 7 Hz world are exactly one second.
async fn a_cadence_that_does_not_divide_the_sample_rate_still_lands_on_whole_samples(via: Via) {
let config = HarnessConfig {
step_hz: 7,
..HarnessConfig::default()
};
let mut f = fixture(via, config).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let observer = f.harness.observer().await.unwrap();
let topic = f.harness.coordinator.topics().snapshots.clone();
let mut spectator = Spectator::attach(&observer, &topic, 2).await.unwrap();
// 48000 / 7 is 6857.14..., so no chunk can be the exact share of a second.
let mut frames = Vec::new();
for step in 1..=7 {
within("step", f.harness.coordinator.run(1)).await.unwrap();
let chunk = f
.harness
.coordinator
.observation()
.expect("an observation")
.audio
.first()
.expect("a chunk")
.clone();
assert_eq!(chunk.first_sample, frames.iter().sum::<u64>());
assert!(
chunk.sample_frames == 6_857 || chunk.sample_frames == 6_858,
"step {step} produced {} frames",
chunk.sample_frames
);
assert_eq!(chunk.samples.byte_length, chunk.sample_frames * CHANNELS * 4);
frames.push(chunk.sample_frames);
}
assert_eq!(
frames.iter().sum::<u64>(),
SAMPLE_RATE,
"seven steps of a 7 Hz world are exactly one second of samples: {frames:?}"
);
assert_eq!(f.harness.coordinator.audio_positions()[AUDIO_STREAM_ID], SAMPLE_RATE);
// The published chunk is readable and finite whatever the cadence.
let published = frame_at(&mut spectator, 7).await;
let (chunk, artifact) = published.audio.first().expect("the published chunk");
let bytes = artifact.read_all().await.expect("the published chunk");
assert_eq!(bytes.len() as u64, chunk.sample_frames * CHANNELS * 4);
require_finite_samples(&bytes).expect("native samples are finite f32");
f.shutdown().await;
}
/// The media path itself is mode-agnostic: one native image per boundary, forwarded to every
/// agent as an owned attachment and published once for presentation, whether the participants
/// are tasks on one runtime, threads with their own runtimes, or separate processes.
///
/// What a *test* can see differs by mode, and this test only asserts what crosses a process
/// boundary. An agent validates that each attachment is the artifact its payload names and
/// reads the pixels before it commits, so a session that commits every boundary in process
/// mode has carried one shared image across that boundary through the store, not through
/// shared memory. The sensor log and the render counter are shared memory, so they are
/// asserted where they exist and their absence is asserted where they do not.
async fn the_media_path_works_in_every_execution_mode(mode: ExecutionMode) {
// A declared render delay as well, so the option reaches a world in another process.
let config = HarnessConfig {
observation_delay_steps: 1,
..HarnessConfig::default()
};
let mut f = mode_fixture(mode, config).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let observer = f.harness.observer().await.unwrap();
let topic = f.harness.coordinator.topics().snapshots.clone();
let mut spectator = Spectator::attach(&observer, &topic, 2).await.unwrap();
within("run", f.harness.coordinator.run(STEPS)).await.unwrap();
// Every agent read its attachment and committed, in every mode.
assert_eq!(f.harness.coordinator.committed_boundary(), Some(STEPS));
// The coordinator holds one view handle and one audio handle for this boundary, and the
// observation names exactly those artifacts.
let observation = f.harness.coordinator.observation().expect("an observation").clone();
let handles = f.harness.coordinator.media_handles();
assert_eq!(handles.len(), 2, "one view and one chunk: {handles:?}");
let view = observation.sensory_views.first().expect("a required view");
assert_eq!(
view.produced_step,
STEPS - 1,
"the declared one-step delay reached the world in {mode:?} mode"
);
assert!(handles.iter().any(|(name, r)| *name == view_attachment(&view.view_id)
&& r.artifact_id == view.pixels.artifact_id));
// The same object is what presentation was published, and its bytes read back at the
// declared shape through an ordinary subscription.
let published = frame_at(&mut spectator, STEPS).await;
assert_eq!(published.view.pixels.artifact_id, view.pixels.artifact_id);
assert_eq!(published.view.produced_step, view.produced_step);
let pixels = published.artifact.read_all().await.expect("the published frame");
assert_eq!(pixels.len() as u64, VIEW_WIDTH * VIEW_HEIGHT * 4);
let (chunk, audio) = published.audio.first().expect("the published chunk");
let samples = audio.read_all().await.expect("the published chunk");
assert_eq!(samples.len() as u64, chunk.sample_frames * CHANNELS * 4);
require_finite_samples(&samples).expect("native samples are finite f32");
// The shared-memory instrumentation exists exactly where the participants do.
match (mode, f.harness.sensor_log(&fly_a()), f.harness.renders()) {
(ExecutionMode::Process, sensors, renders) => {
assert!(sensors.is_none() && renders.is_none(), "not observable from here");
}
(_, Some(sensors), Some(renders)) => {
let a = sensors.entries();
let b = f.harness.sensor_log(&fly_b()).expect("in this process").entries();
assert_eq!(a, b, "both agents read the same artifact and the same bytes");
assert_eq!(boundaries(&a), (0..=STEPS).collect::<Vec<_>>());
assert_eq!(
digest_of_bytes(&pixels),
a.last().expect("an entry per boundary").digest,
"the published bytes are the bytes the agents encoded"
);
assert_eq!(renders, STEPS + 1, "one render per boundary");
}
(mode, _, _) => panic!("{mode:?} keeps its participants in this process"),
}
f.shutdown().await;
}
/// Every media fault fires wherever the world runs, which is what proves the wiring rather
/// than assuming it.
///
/// A fault reaches a world in another process only as a `--flag` on its command line, so a
/// renamed or dropped flag would make these faults silently stop firing. Here each one has to
/// fail its transition in every execution mode; the parser refuses an unknown flag, so a
/// mismatch fails the launch instead of turning into a no-op.
async fn every_media_fault_fires_in_every_execution_mode(mode: ExecutionMode) {
let cases: [(&str, EnvironmentFaults, u64); 4] = [
(
"an extra-delayed view",
EnvironmentFaults {
stale_view_at_boundary: Some(2),
..EnvironmentFaults::default()
},
2,
),
(
"a frame of the wrong length",
EnvironmentFaults {
truncated_view_at_boundary: Some(1),
..EnvironmentFaults::default()
},
1,
),
(
"a missing audio chunk",
EnvironmentFaults {
omit_audio_at_boundary: Some(2),
..EnvironmentFaults::default()
},
2,
),
(
"an overlapping audio chunk",
EnvironmentFaults {
overlapping_audio_at_boundary: Some(2),
..EnvironmentFaults::default()
},
2,
),
];
for (what, faults, steps) in cases {
let config = HarnessConfig {
environment_faults: faults,
..HarnessConfig::default()
};
let mut f = mode_fixture(mode, config).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let outcome = within("run", f.harness.coordinator.run(steps)).await;
let failure = match outcome {
Err(failure) => failure,
Ok(reports) => {
panic!("{what} did not fail the transition in {mode:?} mode: {reports:?}")
}
};
assert_eq!(
failure.error.code,
ErrorCode::BufferInvalid,
"{what} in {mode:?} mode: {failure}"
);
assert_eq!(f.harness.coordinator.phase(), Phase::Failed);
assert_eq!(
f.harness.coordinator.stats().advances,
steps - 1,
"{what} in {mode:?} mode committed the boundaries before it and no more"
);
f.shutdown().await;
}
}
/// The retention table: a required agent input is retained through encoding and Commit with no
/// coalescing, while a spectator's snapshots are a latest subscription with finite credits.
async fn required_agent_input_is_never_coalesced_while_spectator_snapshots_are(via: Via) {
let mut f = default_fixture(via).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let observer = f.harness.observer().await.unwrap();
let topic = f.harness.coordinator.topics().snapshots.clone();
let mut spectator = Spectator::attach(&observer, &topic, 1).await.unwrap();
assert!(within("first snapshot", spectator.hold_one()).await);
let steps = 5;
within("run", f.harness.coordinator.run(steps)).await.unwrap();
// The spectator's queue coalesced while it was not reading.
spectator.release();
within("after release", spectator.hold_one()).await;
assert!(spectator.coalesced() > 0);
assert!(
spectator.seen() < steps + 1,
"a latest subscription does not deliver every boundary to a slow reader"
);
// Every agent's required input arrived once per boundary, in order, with nothing dropped
// or replaced.
for agent in [fly_a(), fly_b()] {
let entries = sensed(&f, &agent);
assert_eq!(
boundaries(&entries),
(0..=steps).collect::<Vec<_>>(),
"{agent} encoded every boundary exactly once"
);
}
f.shutdown().await;
}
/// A persistent `AssetRef` names installed content; a transient `ArtifactRef` names live bytes
/// in one store. Importing the asset makes a new artifact, and collecting that artifact leaves
/// the asset installed.
async fn a_persistent_asset_and_a_transient_artifact_are_different_identities(via: Via) {
let mut f = default_fixture(via).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let client = f.harness.observer().await.unwrap();
let body = "counter-arena-backend-v1";
let asset = synthetic_asset("counter-arena-backend", body);
let mut registry = AssetRegistry::new();
registry
.install(asset.clone(), body.as_bytes().to_vec())
.expect("installed content matches its reference");
// Installing content that is not what the reference claims is refused.
let mut wrong = asset.clone();
wrong.byte_length += 1;
registry
.install(wrong, body.as_bytes().to_vec())
.expect_err("an asset reference is its content's identity");
let before = f.harness.router().stats().sealed_artifacts;
let artifact = registry.import(&client, &asset).await.expect("the import");
assert_eq!(
f.harness.router().stats().sealed_artifacts,
before + 1,
"the import is a new object in the store"
);
// The identities are different, and the content is the same.
assert_ne!(artifact.reference().artifact_id, asset.id);
assert_eq!(artifact.reference().byte_length, asset.byte_length);
assert_eq!(artifact.reference().digest.as_deref(), Some(asset.digest.as_str()));
check_imported_asset(&asset, artifact.reference()).expect("the import carries the content");
assert_eq!(
artifact.read_all().await.expect("readable"),
body.as_bytes(),
"the imported artifact is the installed bytes"
);
// The transient artifact is collected with its last handle; the asset is still installed.
drop(artifact);
let router = f.harness.router().clone();
until("the imported artifact is collected", || {
router.stats().sealed_artifacts == before
})
.await;
assert!(registry.contains(&asset));
assert_eq!(
registry.resolve(&asset).expect("still installed"),
body.as_bytes(),
"a persistent asset outlives the bus objects imported from it"
);
f.shutdown().await;
}
/// A restored epoch resumes the preserved sample position, and its first chunk marks the
/// discontinuity that says so. The producer and the validator agree about which epoch it is.
async fn a_restored_audio_source_resumes_and_marks_the_discontinuity(via: Via) {
let f = default_fixture(via).await;
let client = f.harness.observer().await.unwrap();
let descriptor = fly_session::environment::CounterEnvironment::audio_descriptor();
let step = hz(f.harness.config.step_hz).expect("a positive cadence");
let resumed_at = 2_400;
let mut source = AudioSource::restored_at(descriptor.clone(), resumed_at);
let (first, _first_handle) = source
.produce(&client, &step, 3)
.await
.expect("the restored source produces");
assert_eq!(first.first_sample, resumed_at, "the sample position is preserved");
assert!(first.discontinuity, "the first chunk after a restore marks it");
let (second, _second_handle) = source
.produce(&client, &step, 3)
.await
.expect("the next chunk");
assert!(!second.discontinuity, "only the first chunk of the epoch marks it");
assert_eq!(second.first_sample, resumed_at + first.sample_frames);
// The validator accepts exactly this sequence under a restored timeline, and refuses it
// under a fresh one: the flag is what distinguishes the two epochs.
let mut restored = AudioTimeline::restored_at(&descriptor, resumed_at);
restored.accept(&first, &descriptor).expect("the restored epoch");
restored.accept(&second, &descriptor).expect("and its next chunk");
// A fresh episode at the audio origin refuses it: a restore preserves the sample
// position, and the position is what distinguishes the two epochs. The flag is required
// after a restore and free at an origin, so it cannot carry that distinction by itself.
let mut fresh = AudioTimeline::fresh(&descriptor, 0);
fresh
.accept(&first, &descriptor)
.expect_err("a fresh episode starts at its own origin, not a resumed position");
f.shutdown().await;
}
/// The sample budget is exact when the cadence does not divide the sample rate: 8 kHz at 60 Hz
/// is 133, 133, 134 and the total after three steps is exactly 400.
#[test]
fn the_sample_budget_is_exact_when_the_cadence_does_not_divide_the_rate() {
let descriptor = fly_session_types::media::AudioDescriptor {
stream_id: "arena".into(),
sample_rate: 8_000,
channels: 1,
};
let mut source = AudioSource::new(descriptor, 0);
let step = hz(60).expect("a positive cadence");
let frames: Vec<u64> = (0..6)
.map(|_| source.frames_for_step(&step).expect("an exact budget"))
.collect();
assert_eq!(frames, vec![133, 133, 134, 133, 133, 134]);
assert_eq!(frames.iter().sum::<u64>(), 800, "8000 samples in a tenth of a second");
// 48 kHz at 60 Hz divides exactly.
let exact = fly_session_types::media::AudioDescriptor {
stream_id: "arena".into(),
sample_rate: SAMPLE_RATE,
channels: CHANNELS,
};
let mut source = AudioSource::new(exact, 0);
assert_eq!(source.frames_for_step(&step).unwrap(), 800);
}
/// The native frame is a real pattern with no padded rows: `rowStride` is `4 x width`, the
/// rows are top-left first, and consecutive boundaries differ.
#[test]
fn the_native_frame_is_top_left_rgba8_with_no_padding() {
let descriptor = fly_session::environment::CounterEnvironment::view_descriptor(0);
let frame = arena_frame(&descriptor, 5, 3);
assert_eq!(frame.len() as u64, descriptor.row_stride * descriptor.height);
assert_eq!(descriptor.row_stride, descriptor.width * 4);
// Every pixel is opaque and carries the world counter in its red channel.
for pixel in frame.chunks_exact(4) {
assert_eq!(pixel[0], 5);
assert_eq!(pixel[3], 255);
}
assert_ne!(
arena_frame(&descriptor, 5, 3),
arena_frame(&descriptor, 5, 4),
"consecutive boundaries are different images"
);
assert_ne!(
arena_frame(&descriptor, 5, 3),
arena_frame(&descriptor, 6, 3),
"the counter changes the image"
);
}

View file

@ -0,0 +1,823 @@
//! SESSION-02 acceptance: one agent process per fly and one environment process under the
//! coordinator, compared with the in-process and dedicated-thread variants.
//!
//! Every acceptance bullet is one named test here, generated once per execution mode, so a
//! rule that holds in one process holds across a process boundary too. The two process-mode
//! failure rows of section 4 that SESSION-01 could not reach in one process -- a router
//! restart during a world advance, and an old worker's reply after a restart -- are at the
//! end and run in the separate-process mode.
mod common;
use std::collections::BTreeMap;
use std::time::{Duration, Instant};
use common::{at, count, fly_a, fly_b, mode_fixture, within};
use fly_session::agent::AgentFaults;
use fly_session::coordinator::{DispatchOrder, Injections};
use fly_session::environment::EnvironmentFaults;
use fly_session::harness::{ExecutionMode, HarnessConfig, Via};
use fly_session::launcher::{ReapOutcome, ThreadBudget};
use fly_session::ResolutionEnd;
use fly_session::phase::Phase;
use fly_session::types::*;
all_modes!(
a_slow_participant_is_resolved_rather_than_failed,
a_resolution_says_which_of_its_two_bounds_ended_it,
a_delayed_one_agent_result_holds_the_world,
a_worker_death_has_a_bounded_diagnosed_outcome,
a_helper_death_has_a_bounded_diagnosed_outcome,
an_uncertain_advance_never_creates_a_second_batch,
a_partial_commit_never_permits_next_step_play,
every_participant_answers_its_supervisor,
worker_threads_lie_within_the_launcher_allocation,
);
const STEPS: u64 = 4;
fn two_agents(mode: ExecutionMode) -> HarnessConfig {
HarnessConfig { mode, ..HarnessConfig::default() }
}
// -------------------------------------------------------------------------------------------
// Acceptance: sequential, reversed and parallel completion produce equivalent traces
/// `step-v1` section 8, across the process boundary: sequential, concurrent and reversed
/// dispatch, in all three execution modes, produce one behaviour trace.
///
/// This reuses the wave-1 comparator -- the behaviour half of the section 8 trace, with
/// request ids, bus correlation and wall time excluded -- so "a process behaves like a task"
/// is the same assertion that "a reordered dispatch behaves like an ordered one" was.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn sequential_reversed_and_parallel_completion_agree() {
let mut behaviours: BTreeMap<String, Vec<String>> = BTreeMap::new();
for mode in ExecutionMode::all() {
for order in [
DispatchOrder::Sequential,
DispatchOrder::Concurrent,
DispatchOrder::Reversed,
] {
let mut config = two_agents(mode);
// Deliberately unequal completion times, so a concurrent run really does finish
// out of dispatch order whichever side of a process boundary the agents are on.
config.agents[0].faults =
AgentFaults { prepare_delay_ms: 12, ..AgentFaults::default() };
config.agents[1].faults = AgentFaults { commit_delay_ms: 9, ..AgentFaults::default() };
let mut f = mode_fixture(mode, config).await;
f.harness.coordinator.dispatch = order;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
within("run", f.harness.coordinator.run(STEPS)).await.unwrap();
let behaviour = f.harness.coordinator.trace.behavior();
assert_eq!(behaviour.len() as u64, STEPS);
behaviours.insert(format!("{}/{order:?}", mode.label()), behaviour);
f.shutdown().await;
}
}
let mut iter = behaviours.iter();
let (first_name, first) = iter.next().expect("at least one run");
for (name, behaviour) in iter {
assert_eq!(
behaviour, first,
"{name} produced a different behaviour trace from {first_name}"
);
}
}
// -------------------------------------------------------------------------------------------
// ipc-v1 section 6: an uncertain call is resolved, not failed
/// A participant that is merely slow -- slower than the caller's probe, faster than the
/// resolution's budget -- finishes its step. The epoch is not lost, and the resolution adds no
/// second operation.
///
/// This is the `ipc-v1` section 6 procedure on the path that actually reaches it: the probe
/// expires, the coordinator queries the same request id against the same incarnation, the
/// worker answers `IN_PROGRESS` while its original is still running and then replays its
/// cached reply. `step-v1` section 7's Advance row is the same rule, so the world is slow here
/// too and its batch is never re-sent as a new one.
async fn a_slow_participant_is_resolved_rather_than_failed(mode: ExecutionMode) {
// A clean run of the same composition, to compare against.
let clean = {
let mut f = mode_fixture(mode, two_agents(mode)).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
within("run", f.harness.coordinator.run(2)).await.unwrap();
let environment = f.harness.environment_id();
let world = within("progress", f.harness.progress_of(&environment)).await.unwrap();
let out = (f.harness.coordinator.trace.behavior(), world);
f.shutdown().await;
out
};
let mut config = two_agents(mode);
config.agents[1].faults = AgentFaults { prepare_delay_ms: 500, ..AgentFaults::default() };
config.environment_faults =
EnvironmentFaults { advance_delay_ms: 500, ..EnvironmentFaults::default() };
let mut f = mode_fixture(mode, config).await;
// A probe well inside both delays, and a resolution budget well outside them: the point is
// a call that expires and an operation that is nevertheless fine.
f.harness.coordinator.deadlines = fly_session::Deadlines {
probe: Duration::from_millis(120),
resolve: Duration::from_secs(20),
resolve_attempts: 4096,
boot: Duration::from_secs(30),
};
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let reports = within("run", f.harness.coordinator.run(2))
.await
.expect("a slow participant is resolved, not failed");
assert_eq!(reports.len(), 2);
assert_eq!(f.harness.coordinator.phase(), Phase::Ready(2));
assert!(!f.harness.coordinator.is_fenced(), "a slow answer is not a lost epoch");
assert!(
f.harness.coordinator.resolutions >= 2,
"both the slow Prepare and the slow Advance must have run the resolution, not {}",
f.harness.coordinator.resolutions
);
assert!(
f.harness.coordinator.in_progress_replies > 0,
"the resolution must have met the original still running"
);
assert_eq!(
f.harness.coordinator.last_resolution,
Some(ResolutionEnd::Answered),
"the resolution ended by being answered, not by running out of anything"
);
// No second operation anywhere: one advance per transition, one batch id per transition,
// and the same behaviour as the run that never timed out.
assert_eq!(f.harness.coordinator.stats().advances, 2);
let environment = f.harness.environment_id();
let world = within("progress", f.harness.progress_of(&environment)).await.unwrap();
assert_eq!(world, clean.1, "the world moved exactly as often as in the clean run");
assert_eq!(
f.harness.coordinator.trace.behavior(),
clean.0,
"resolving an uncertain call changes no behaviour"
);
let batches: std::collections::BTreeSet<Id> = f
.harness
.coordinator
.trace
.transitions
.iter()
.map(|t| t.behaviour.batch_id.clone())
.collect();
assert_eq!(batches.len(), 2, "one batch id per transition, never a second batch");
// And the agents took exactly the ticks the clean run took: a resolution is a query.
for transition in &f.harness.coordinator.trace.transitions {
for agent in &transition.behaviour.agents {
assert!(agent.ticks_advanced == 16 || agent.ticks_advanced == 17);
}
}
f.shutdown().await;
}
/// The resolution has two bounds, and which one ended it is never left to be guessed.
///
/// `resolve` is the working limit at the default values -- the attempt guard is over sixteen
/// seconds of pauses against an eight-second budget -- so an unresponsive participant runs the
/// budget out. Setting the guard low instead ends the same resolution the other way, and the
/// failure says so both in `last_resolution` and in its own message.
async fn a_resolution_says_which_of_its_two_bounds_ended_it(mode: ExecutionMode) {
// The budget is what ends it at ordinary settings: a generous attempt guard, a short
// budget, and a participant far slower than either.
let mut config = two_agents(mode);
config.agents[1].faults = AgentFaults { prepare_delay_ms: 30_000, ..AgentFaults::default() };
let mut f = mode_fixture(mode, config).await;
f.harness.coordinator.deadlines = fly_session::Deadlines {
probe: Duration::from_millis(50),
resolve: Duration::from_millis(300),
resolve_attempts: 8192,
boot: Duration::from_secs(30),
};
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let started = Instant::now();
let failure = within("step", f.harness.coordinator.step())
.await
.expect_err("a participant that never answers exhausts the resolution");
assert_eq!(f.harness.coordinator.last_resolution, Some(ResolutionEnd::BudgetExpired));
assert!(
failure.error.message.contains("resolution budget"),
"the message names the bound that fired: {failure}"
);
assert_eq!(failure.participant.as_deref(), Some(fly_b().as_str()));
assert_eq!(failure.error.mutation, MutationCertainty::Unknown);
assert!(
started.elapsed() < Duration::from_secs(20),
"the budget, not the 30-second participant, is what ended it"
);
assert!(f.harness.coordinator.is_fenced());
f.shutdown().await;
// The guard is what ends it when it is set below the budget: three attempts against a
// budget the participant could never reach anyway.
let mut config = two_agents(mode);
config.agents[1].faults = AgentFaults { prepare_delay_ms: 30_000, ..AgentFaults::default() };
let mut f = mode_fixture(mode, config).await;
f.harness.coordinator.deadlines = fly_session::Deadlines {
probe: Duration::from_millis(50),
resolve: Duration::from_secs(600),
resolve_attempts: 3,
boot: Duration::from_secs(30),
};
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let failure = within("step", f.harness.coordinator.step())
.await
.expect_err("three attempts are not enough to resolve a silent participant");
assert_eq!(f.harness.coordinator.last_resolution, Some(ResolutionEnd::AttemptsExhausted));
assert!(
failure.error.message.contains("attempt guard") && failure.error.message.contains("3 attempts"),
"the message names the bound that fired and its size: {failure}"
);
assert_eq!(failure.participant.as_deref(), Some(fly_b().as_str()));
f.shutdown().await;
}
// -------------------------------------------------------------------------------------------
// Acceptance: a delayed one-agent result holds the world
/// One agent takes far longer than the other to prepare. No `Environment.Advance` is sent
/// until every agent is Prepared, and the world is still at its old boundary while the
/// coordinator waits.
async fn a_delayed_one_agent_result_holds_the_world(mode: ExecutionMode) {
let mut config = two_agents(mode);
config.agents[1].faults = AgentFaults { prepare_delay_ms: 400, ..AgentFaults::default() };
let mut f = mode_fixture(mode, config).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let environment = f.harness.environment_id();
let before = within("progress", f.harness.progress_of(&environment)).await.unwrap();
let (coordinator, launcher) = f.harness.parts();
// The supervisor watches the world while the transition is in flight. That is what a
// supervisor is for, and `Worker.Status` answers without waiting for a mutation.
let (stepped, held) = tokio::join!(
async { within("step", coordinator.step()).await },
async {
tokio::time::sleep(Duration::from_millis(120)).await;
within("status", launcher.health_check(&environment)).await
}
);
let report = stepped.expect("the transition completes once the slow agent answers");
assert_eq!(report.boundary, 1);
let held = held.expect("the environment answers its supervisor during the wait");
assert_eq!(
held.progress_counter, before,
"the world may not advance while one agent is still preparing"
);
assert_eq!(
held.state,
WorkerState::Ready,
"the environment is at a committed boundary, not advancing"
);
// And the ordering the audit records says the same thing from the coordinator's side.
let audit = f.harness.coordinator.audit.clone();
let advance = at(&audit, "advance:0");
for agent in [fly_a(), fly_b()] {
assert!(
at(&audit, &format!("prepared:{agent}@0")) < advance,
"{agent} must be Prepared before the world advances: {audit:?}"
);
}
assert_eq!(f.harness.coordinator.stats().advances, 1);
f.shutdown().await;
}
// -------------------------------------------------------------------------------------------
// Acceptance: worker or helper death has a bounded diagnosed outcome
/// One agent dies in the middle of its Prepare. The epoch fails with a typed cause naming
/// that agent, within the caller's own budget, and nothing continues on the remainder.
async fn a_worker_death_has_a_bounded_diagnosed_outcome(mode: ExecutionMode) {
let mut config = two_agents(mode);
config.agents[1].faults = AgentFaults { prepare_delay_ms: 5_000, ..AgentFaults::default() };
let mut f = mode_fixture(mode, config).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let started = Instant::now();
let (coordinator, launcher) = f.harness.parts();
let (stepped, reaped) = tokio::join!(
async { within("step", coordinator.step()).await },
async {
tokio::time::sleep(Duration::from_millis(80)).await;
launcher.kill(&fly_b()).await
}
);
assert_eq!(reaped, ReapOutcome::Terminated);
let failure = stepped.expect_err("a dead participant is a failed epoch, not a slow one");
assert!(
started.elapsed() < Duration::from_secs(20),
"the outcome must be bounded, not a hang"
);
assert_eq!(
failure.participant.as_deref(),
Some(fly_b().as_str()),
"the failure names the participant: {failure}"
);
assert_ne!(
failure.error.mutation,
MutationCertainty::None,
"a participant that died mid-call leaves an uncertain mutation, never a clean none"
);
assert_eq!(f.harness.coordinator.phase(), Phase::Failed);
assert!(f.harness.coordinator.is_fenced());
// No partial continuation: no world step, no publication, and no next transition.
assert_eq!(f.harness.coordinator.stats().advances, 0);
assert_eq!(count(&f.harness.coordinator.audit, "publish:1"), 0);
let again = f.harness.coordinator.step().await.expect_err("a fenced epoch takes no step");
assert_eq!(again.error.code, ErrorCode::InvalidPhase);
f.shutdown().await;
}
/// The environment helper dies in the middle of the world advance. Same rule: a typed cause
/// naming it, bounded, and no half-transition afterwards.
async fn a_helper_death_has_a_bounded_diagnosed_outcome(mode: ExecutionMode) {
let config = HarnessConfig {
environment_faults: EnvironmentFaults {
advance_delay_ms: 5_000,
..EnvironmentFaults::default()
},
..two_agents(mode)
};
let mut f = mode_fixture(mode, config).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let environment = f.harness.environment_id();
let started = Instant::now();
let (coordinator, launcher) = f.harness.parts();
let (stepped, reaped) = tokio::join!(
async { within("step", coordinator.step()).await },
async {
tokio::time::sleep(Duration::from_millis(200)).await;
launcher.kill(&environment).await
}
);
assert_eq!(reaped, ReapOutcome::Terminated);
let failure = stepped.expect_err("a dead world is a failed epoch");
assert!(started.elapsed() < Duration::from_secs(20), "bounded, not a hang");
assert_eq!(
failure.participant.as_deref(),
Some(environment.as_str()),
"the failure names the participant: {failure}"
);
assert_ne!(failure.error.mutation, MutationCertainty::None);
assert_eq!(f.harness.coordinator.phase(), Phase::Failed);
assert!(f.harness.coordinator.is_fenced());
assert_eq!(f.harness.coordinator.stats().advances, 0);
// The agents prepared and are not asked to prepare again or to commit anything.
assert_eq!(f.harness.coordinator.stats().commits, 0);
assert_eq!(count(&f.harness.coordinator.audit, "publish:1"), 0);
f.shutdown().await;
}
// -------------------------------------------------------------------------------------------
// Acceptance: an uncertain Advance never creates a second batch
/// The Advance result is lost after the world already stepped. The coordinator resolves the
/// same operation against its original domain request id; the world advances once per
/// transition and the batch is never re-sent as a new one.
async fn an_uncertain_advance_never_creates_a_second_batch(mode: ExecutionMode) {
let clean = {
let mut f = mode_fixture(mode, two_agents(mode)).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
within("run", f.harness.coordinator.run(STEPS)).await.unwrap();
let environment = f.harness.environment_id();
let world = within("progress", f.harness.progress_of(&environment)).await.unwrap();
let out = (f.harness.coordinator.trace.behavior(), world);
f.shutdown().await;
out
};
let mut f = mode_fixture(mode, two_agents(mode)).await;
f.harness.coordinator.injections = Injections {
at_step: 2,
lose_advance_result: true,
..Injections::default()
};
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
within("run", f.harness.coordinator.run(STEPS)).await.unwrap();
let environment = f.harness.environment_id();
let world = within("progress", f.harness.progress_of(&environment)).await.unwrap();
assert_eq!(f.harness.coordinator.stats().advances, STEPS, "one advance per transition");
assert_eq!(
world, clean.1,
"the world moved exactly as often as it did without the loss"
);
assert_eq!(
f.harness.coordinator.trace.behavior(),
clean.0,
"an uncertain Advance changes no behaviour, so it created no second batch"
);
// Every transition has exactly one batch, and every batch id is its own.
let batches: Vec<Id> = f
.harness
.coordinator
.trace
.transitions
.iter()
.map(|t| t.behaviour.batch_id.clone())
.collect();
let unique: std::collections::BTreeSet<Id> = batches.iter().cloned().collect();
assert_eq!(unique.len(), batches.len(), "one batch id per transition: {batches:?}");
let injections = f.harness.coordinator.injection_log.clone();
assert!(
injections.iter().any(|o| o.what == "lost-advance-result" && o.identical),
"the loss must happen after dispatch, so the outcome really is uncertain: {injections:?}"
);
f.shutdown().await;
}
// -------------------------------------------------------------------------------------------
// Acceptance: a partial Commit never permits next-step play
/// One agent's Commit fails after the other's succeeded. The epoch fails naming that agent,
/// the boundary does not move, nothing is published and there is no next transition.
async fn a_partial_commit_never_permits_next_step_play(mode: ExecutionMode) {
let mut config = two_agents(mode);
config.agents[1].faults =
AgentFaults { fail_commit_at_step: Some(1), ..AgentFaults::default() };
let mut f = mode_fixture(mode, config).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
within("step", f.harness.coordinator.step()).await.unwrap();
let environment = f.harness.environment_id();
let failure = within("step", f.harness.coordinator.step())
.await
.expect_err("one failed Commit fails the epoch");
assert_eq!(
failure.participant.as_deref(),
Some(fly_b().as_str()),
"the failure names the agent whose Commit failed: {failure}"
);
assert_eq!(f.harness.coordinator.phase(), Phase::Failed);
assert!(f.harness.coordinator.is_fenced());
// The world moved once inside the failing transition -- the Advance is what the Commit
// follows -- and it moves no further. There is no next-step play on a partial commit.
let world_before = within("progress", f.harness.progress_of(&environment)).await.unwrap();
let again = f.harness.coordinator.step().await.expect_err("no play after a partial commit");
assert_eq!(again.error.code, ErrorCode::InvalidPhase);
let world_after = within("progress", f.harness.progress_of(&environment)).await.unwrap();
assert_eq!(world_after, world_before, "no next world step follows a partial commit");
let status = within("status", f.harness.launcher.health_check(&environment)).await.unwrap();
assert_eq!(
status.current_scope.unwrap().step,
2,
"the world stays at the boundary the failed transition reached"
);
let audit = f.harness.coordinator.audit.clone();
assert_eq!(count(&audit, "publish:2"), 0);
assert_eq!(f.harness.coordinator.committed_boundary(), None);
f.shutdown().await;
}
// -------------------------------------------------------------------------------------------
// Supervision: identity, health and reaping
/// Every participant answers the supervisor with the identity the launcher configured, and
/// stops when it is asked to.
async fn every_participant_answers_its_supervisor(mode: ExecutionMode) {
let mut f = mode_fixture(mode, two_agents(mode)).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
within("run", f.harness.coordinator.run(2)).await.unwrap();
let environment = f.harness.environment_id();
for who in [fly_a(), fly_b(), environment.clone()] {
let worker = f.harness.launcher.worker(&who).expect("a launched participant");
assert_eq!(worker.identity.worker_id, who);
assert_eq!(worker.domain_incarnation, worker.identity.incarnation_id);
assert!(!worker.service_incarnation.is_empty());
let status = within("health", f.harness.launcher.health_check(&who)).await.unwrap();
assert_eq!(status.state, WorkerState::Ready, "{who} is healthy at a boundary");
}
// Every participant reports the allocation its launcher gave it, which is the wire the
// 2026-09-22 `workers-v1` amendment added. The launcher refused anything else at start,
// so a caller reads it here rather than being told it out of band.
for who in [fly_a(), fly_b(), environment.clone()] {
let worker = f.harness.coordinator.agent_ref(&who).cloned().unwrap_or_else(|| {
f.harness.coordinator.environment_ref().clone()
});
let params = serde_json::json!({
"sessionId": "demo",
"expectedWorkerId": who.as_str(),
"role": if who == environment { "environment" } else { "agent" },
"supportedMajors": [1],
});
let result = within(
"hello",
f.harness.coordinator.probe_raw(&worker, "Worker.Hello", None, params),
)
.await
.expect("a worker answers its own identity");
let reported = result["limits"]["workerThreads"].as_u64();
assert_eq!(
reported,
Some(f.harness.launcher.worker(&who).unwrap().identity.worker_threads as u64),
"{who} must report the allocation its launcher gave it"
);
}
// The agents carry their configured port identities; the environment owns the ports.
assert_eq!(
f.harness.launcher.worker(&fly_a()).unwrap().identity.port_id.as_deref(),
Some("p1")
);
assert_eq!(
f.harness.launcher.worker(&fly_b()).unwrap().identity.port_id.as_deref(),
Some("p2")
);
assert!(f.harness.launcher.worker(&environment).unwrap().identity.port_id.is_none());
// A worker that is not the one the caller expects refuses to negotiate at all.
let worker = f.harness.coordinator.agent_ref(&fly_a()).cloned().unwrap();
let wrong = serde_json::json!({
"sessionId": "demo",
"expectedWorkerId": "fly-z",
"role": "agent",
"supportedMajors": [1],
});
let err = within(
"hello",
f.harness.coordinator.probe_raw(&worker, "Worker.Hello", None, wrong),
)
.await
.expect_err("a worker is not whoever a caller says it is");
assert_eq!(err.code, ErrorCode::IdentityMismatch);
// Asking a participant to stop stops it, and the supervisor says which kind of stop it was.
let outcome = f.harness.launcher.reap(&fly_a(), &id("test")).await;
assert_eq!(outcome, ReapOutcome::Stopped, "a live participant answers Worker.Shutdown");
assert_eq!(
f.harness.launcher.reap(&fly_a(), &id("test")).await,
ReapOutcome::AlreadyGone
);
f.shutdown().await;
}
/// `workers-v1`: `Agent.Initialize`'s `workerThreads` lies within the launcher allocation.
///
/// The budget refuses an allocation it cannot cover before anything is started, and an agent
/// refuses an `Agent.Initialize` asking for more threads than its launcher gave it.
async fn worker_threads_lie_within_the_launcher_allocation(mode: ExecutionMode) {
// The budget itself: a total, a coordinator reservation, and a refusal that names both.
let mut budget = ThreadBudget::new(4, 1).unwrap();
assert_eq!(budget.remaining(), 3);
assert_eq!(budget.allocate(&id("arena"), 1).unwrap(), 1);
assert_eq!(budget.allocate(&id("fly-a"), 2).unwrap(), 2);
let refused = budget.allocate(&id("fly-b"), 1).expect_err("the budget is spent");
assert_eq!(refused.code, ErrorCode::Busy);
budget.release(&id("fly-a"));
assert_eq!(budget.allocate(&id("fly-b"), 1).unwrap(), 1);
assert_eq!(budget.allocate(&id("fly-b"), 1).expect_err("already held").code, ErrorCode::Conflict);
// A composition the configured budget cannot cover never starts.
let config = HarnessConfig {
mode,
thread_budget: Some(2),
..HarnessConfig::default()
};
let dir = tempfile::tempdir().expect("a temporary directory");
let refused = fly_session::harness::SessionHarness::start(Via::Unix, dir.path(), config).await;
let refused = refused.err().expect("two threads cannot hold a coordinator, a world and two flies");
assert_eq!(refused.code, flybus::ErrorCode::QuotaExceeded, "{}", refused.message);
drop(dir);
// And the worker's own check: it was launched with one thread, so an Initialize asking
// for eight is refused before the model is constructed.
let mut f = mode_fixture(mode, two_agents(mode)).await;
let worker = f.harness.coordinator.agent_ref(&fly_a()).cloned().unwrap();
let profile = fly_session::agent::synthetic_profile(
&fly_a(),
&millis(1).unwrap(),
f.harness.config.warmup_ticks,
);
let params = serde_json::json!({
"agentId": "fly-a",
"profile": profile.to_json(),
"seed": 7,
"initialInput": {"boundary": "0", "views": [], "structured": null},
"initialDecisionContext": {
"schema": fly_session::task::context_schema().to_json(),
"value": {},
},
"workerThreads": 8,
});
let err = within(
"initialize",
f.harness.coordinator.probe_raw(
&worker,
"Agent.Initialize",
Some(scope_at("demo", "e1", 0)),
params,
),
)
.await
.expect_err("eight threads are not within a one-thread allocation");
assert_eq!(err.code, ErrorCode::Busy);
assert_eq!(err.mutation, MutationCertainty::None, "nothing was constructed");
// The allocation the coordinator actually sends is the one the launcher handed out.
assert_eq!(f.harness.launcher.worker(&fly_a()).unwrap().identity.worker_threads, 1);
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
f.shutdown().await;
}
// -------------------------------------------------------------------------------------------
// Section 4 rows SESSION-01 could not reach in one process
/// Row: "Router restarts during a world advance | Old handles/routes invalid; epoch fails and
/// restores coherently."
///
/// The restore half is STATE-01's. What SESSION-02 establishes is the half before it: the
/// epoch fails with a typed cause naming the participant the coordinator was talking to, the
/// session is fenced, every artifact handle of that store incarnation is gone, and no
/// boundary, publication or further transition follows.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_router_restart_during_a_world_advance_fences_the_epoch() {
let mode = ExecutionMode::Process;
let config = HarnessConfig {
environment_faults: EnvironmentFaults {
advance_delay_ms: 3_000,
..EnvironmentFaults::default()
},
..two_agents(mode)
};
let mut f = mode_fixture(mode, config).await;
// The router is gone in a moment, so the supervisor must not spend its full budget
// asking a participant that can no longer be reached.
f.harness.launcher.set_health_policy(fly_session::launcher::HealthPolicy {
probe: Duration::from_millis(200),
fail: Duration::from_millis(500),
boot: Duration::from_secs(30),
});
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let boundary_before = f.harness.coordinator.observation().unwrap().boundary;
assert!(
f.harness.coordinator.live_view_handles() > 0,
"boundary 0's view is owned before the router goes away"
);
let router = f.harness.router().clone();
let started = Instant::now();
let (coordinator, _launcher) = f.harness.parts();
let (stepped, ()) = tokio::join!(
async { within("step", coordinator.step()).await },
async {
// Mid-advance: the world has been asked to move and has not answered yet.
tokio::time::sleep(Duration::from_millis(250)).await;
router.shutdown();
}
);
let failure = stepped.expect_err("a lost router fails the epoch");
assert!(started.elapsed() < Duration::from_secs(20), "bounded, not a hang");
assert_eq!(
failure.participant.as_deref(),
Some(f.harness.environment_id().as_str()),
"the failure names the participant the coordinator was waiting for: {failure}"
);
assert_ne!(
failure.error.mutation,
MutationCertainty::None,
"the world may have stepped; a lost router is never proof that it did not"
);
assert_eq!(f.harness.coordinator.phase(), Phase::Failed);
assert!(
f.harness.coordinator.is_fenced(),
"old handles and routes are invalid from here on"
);
assert_eq!(
f.harness.coordinator.live_view_handles(),
0,
"the fence drops every artifact handle of the old store incarnation"
);
assert_eq!(f.harness.coordinator.stats().advances, 0, "no boundary was committed");
assert_eq!(count(&f.harness.coordinator.audit, "publish:1"), 0);
assert_eq!(
f.harness.coordinator.observation().unwrap().boundary,
boundary_before,
"the committed observation is still the one from before the advance"
);
// Nothing reconnects into the active epoch: a new call on the old route is refused.
let again = f.harness.coordinator.step().await.expect_err("a fenced epoch takes no step");
assert_eq!(again.error.code, ErrorCode::InvalidPhase);
f.shutdown().await;
}
/// Row: "Old worker replies after restore | Stale epoch/incarnation rejected", with real
/// processes.
///
/// A restarted agent is a new process, a new registration and a new domain incarnation. The
/// coordinator pinned the old registration, so its next call fails rather than reaching the
/// replacement; and the replacement, followed deliberately, refuses an operation from the
/// epoch the old process belonged to.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn an_old_worker_reply_after_a_restart_is_rejected_on_stale_epoch_or_incarnation() {
let mode = ExecutionMode::Process;
let mut f = mode_fixture(mode, two_agents(mode)).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
within("step", f.harness.coordinator.step()).await.unwrap();
let old = f.harness.coordinator.agent_ref(&fly_b()).cloned().unwrap();
let old_pid = f.harness.launcher.worker(&fly_b()).unwrap().pid;
assert!(old_pid.is_some(), "a separate-process agent has a process of its own");
let restarted = f.harness.restart_agent(&fly_b()).await.unwrap();
let new_pid = f.harness.launcher.worker(&fly_b()).unwrap().pid;
assert_ne!(old_pid, new_pid, "a restart is a new process");
assert_ne!(
restarted.service_incarnation, old.bus_incarnation,
"a replacement registration is a new incarnation"
);
// Following the new registration while still pinning the old worker's negotiated
// incarnation is rejected: this is the shape an old worker's reply would arrive in.
let stale = fly_session::rpc::WorkerRef {
service: restarted.service.clone(),
bus_incarnation: restarted.service_incarnation.clone(),
worker_id: fly_b(),
domain_incarnation: old.domain_incarnation.clone(),
};
assert_ne!(old.domain_incarnation, Some(restarted.incarnation_id.clone()));
let err = within("status", f.harness.coordinator.status(&stale))
.await
.expect_err("the replacement is not the incarnation this epoch negotiated");
assert_eq!(err.error.code, ErrorCode::IdentityMismatch);
assert_eq!(err.participant.as_deref(), Some(fly_b().as_str()));
assert_eq!(f.harness.coordinator.phase(), Phase::Failed);
assert!(f.harness.coordinator.is_fenced());
assert_eq!(f.harness.coordinator.stats().advances, 1, "no world step under a lost pin");
f.shutdown().await;
}
/// The other half of the same row, in two parts, because the two refusals are different
/// refusals and each deserves its own exact code.
///
/// A restarted worker is a *fresh* process: it has no epoch at all, so the old epoch's work is
/// refused on phase, not on timeline. The stale-epoch half of the row needs a worker that has
/// an epoch and has left it, which in process mode is the agent that did not restart.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn a_restarted_worker_refuses_an_operation_from_the_old_epoch() {
let mode = ExecutionMode::Process;
let mut f = mode_fixture(mode, two_agents(mode)).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
within("step", f.harness.coordinator.step()).await.unwrap();
// Part one: a live agent process, initialized under epoch e1, meets an operation from
// another epoch. This is the row's stale-epoch half, with a real child process.
let live = f.harness.coordinator.agent_ref(&fly_a()).cloned().unwrap();
let err = within(
"stale epoch",
f.harness.coordinator.probe_raw(
&live,
"Agent.Prepare",
Some(scope_at("demo", "e0", 1)),
prepare_params("fly-a"),
),
)
.await
.expect_err("an old epoch cannot mutate a worker that belongs to this one");
assert_eq!(err.code, ErrorCode::StaleEpoch);
assert_eq!(err.mutation, MutationCertainty::None, "refused before any mutation");
// Part two: the replacement process. It is a fresh worker with no epoch at all, so the
// same request is refused on phase rather than on timeline -- and, either way, nothing
// from the old epoch is applied to a fresh brain.
let restarted = f.harness.restart_agent(&fly_b()).await.unwrap();
let replacement = fly_session::rpc::WorkerRef::new(
&restarted.service,
&restarted.service_incarnation,
&fly_b(),
);
let err = within(
"uninitialized replacement",
f.harness.coordinator.probe_raw(
&replacement,
"Agent.Prepare",
Some(scope_at("demo", "e1", 1)),
prepare_params("fly-b"),
),
)
.await
.expect_err("an uninitialized replacement has no epoch to prepare in");
assert_eq!(
err.code,
ErrorCode::InvalidPhase,
"a fresh process has no epoch to be stale about: {err}"
);
assert_eq!(err.mutation, MutationCertainty::None, "nothing was applied to a fresh brain");
assert_eq!(f.harness.coordinator.stats().advances, 1);
f.shutdown().await;
}
/// A well-formed `Agent.Prepare` body, for a probe whose subject is the scope rather than the
/// payload.
fn prepare_params(agent_id: &str) -> serde_json::Value {
serde_json::json!({
"agentId": agent_id,
"profileDigest": digest_of_bytes(b"whatever"),
"interval": {"numerator": "16666667", "denominator": "1"},
"decisionContextDigest": digest_of_bytes(b"whatever"),
"preStepStimulations": [],
})
}

View file

@ -35,7 +35,7 @@ const STEPS: u64 = 3;
async fn one_world_advance_per_complete_batch(via: Via) {
let mut f = default_fixture(via).await;
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
let before = f.harness.environment_mutations();
let before = f.harness.environment_mutations().expect("an in-process arena");
let reports = within("run", f.harness.coordinator.run(STEPS)).await.unwrap();
assert_eq!(reports.len() as u64, STEPS);
assert_eq!(f.harness.coordinator.stats().advances, STEPS);
@ -46,7 +46,10 @@ async fn one_world_advance_per_complete_batch(via: Via) {
"the world is at exactly one boundary per batch"
);
// The environment's progress counter moves once per advance and not otherwise.
assert_eq!(f.harness.environment_mutations() - before, STEPS);
assert_eq!(
f.harness.environment_mutations().expect("an in-process arena") - before,
STEPS
);
assert_eq!(count(&f.harness.coordinator.audit, "advance:0"), 1);
assert_eq!(f.harness.coordinator.trace.transitions.len() as u64, STEPS);
f.shutdown().await;
@ -227,7 +230,8 @@ async fn bootstrap_cannot_advance_the_world_or_produce_a_reward(via: Via) {
// Warm-up did run, with learning disabled, so the models did mutate.
for agent in [fly_a(), fly_b()] {
assert!(
f.harness.agent_mutations(&agent) >= f.harness.config.warmup_ticks,
f.harness.agent_mutations(&agent).expect("an in-process agent")
>= f.harness.config.warmup_ticks,
"warm-up ticks are real mutations"
);
}
@ -270,7 +274,11 @@ async fn the_committed_snapshot_names_the_boundary_that_just_ended(via: Via) {
// The frame the snapshot names travels as an owned attachment.
if step > 0 {
let frame = message.artifact("view.arena").expect("the published frame");
assert_eq!(frame.reference().byte_length, 4 * 4 * 4);
assert_eq!(
frame.reference().byte_length,
fly_session::environment::VIEW_WIDTH * fly_session::environment::VIEW_HEIGHT * 4,
"the published frame is the environment native frame"
);
}
}
assert_eq!(boundaries[0], (0, false), "boundary 0 has no decision or control");
@ -379,12 +387,7 @@ async fn sequential_concurrent_and_reversed_orders_agree() {
/// specific.
async fn a_single_agent_composition_runs_the_same_transaction(via: Via) {
let config = HarnessConfig {
agents: vec![AgentSpec {
agent_id: id("fly-a"),
port_id: id("p1"),
seed: 7,
faults: AgentFaults::default(),
}],
agents: vec![AgentSpec::new("fly-a", "p1", 7)],
..HarnessConfig::default()
};
let mut f: Fixture = fixture(via, config).await;

View file

@ -714,6 +714,13 @@ async fn caller_disconnect_cleanup_works_before_and_after_consumption() {
.await;
caller.close().await;
drop(pending);
// `Client::close` waits for this client to stop; the router marks the call detached when
// *it* observes the disconnect, in its own connection task. Asserting the reply routing
// before that is a race: under load the reply reaches the router first and is routed to a
// connection that is already closing. Nothing escapes -- teardown releases those roots --
// but `routed` is then true. The contract sentence is about a reply to an already detached
// call, so the test waits for the teardown it is talking about.
e.settle("caller-a disconnected", |s| s.connections == 1).await;
assert!(!responder.reply(obj(json!({})), &[]).await.unwrap());
drop(responder);
e.settle("retained responder retired", |s| {