Compare commits
15 commits
cb9a88c7a3
...
56db91cd9b
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
56db91cd9b | ||
|
|
a301527b09 | ||
|
|
66ec2e5c6b | ||
|
|
f456fe9522 | ||
|
|
ba545d9225 | ||
|
|
a3c1c125cd | ||
|
|
77c8ee4558 | ||
|
|
a48e0ace4c | ||
|
|
079842f818 | ||
|
|
903629db00 | ||
|
|
438eaf3130 | ||
|
|
7b7ebcdf28 | ||
|
|
8d999b2881 | ||
|
|
8afeaf6c92 | ||
|
|
40c6a88949 |
35 changed files with 7242 additions and 397 deletions
|
|
@ -119,7 +119,7 @@ and once over a Unix socket, so `tests/rpc.rs::request_reply_roundtrip` means
|
|||
| Requirement | Status | Code | Test |
|
||||
| --- | --- | --- | --- |
|
||||
| `call-<U64>` with increasing serials per connected client; reused or retired ids are rejected, never executed again | conforms: a syntactically valid id advances the watermark even when admission is refused | `router/state.rs::op_call` (`call_watermark`) | `tests/sol_review_races.rs::rejected_call_id_still_advances_monotonic_watermark`, `tests/rpc.rs::raw_call_ids_and_forged_replies` (both) |
|
||||
| Reconnecting creates a new incarnation rather than reviving old calls | conforms | `router/state.rs::{hello, disconnect}` | `tests/sol_review_races.rs::caller_disconnect_cleanup_works_before_and_after_consumption` |
|
||||
| Reconnecting creates a new incarnation rather than reviving old calls | conforms | `router/state.rs::{hello, disconnect}` | `tests/conformance_routing.rs::unpinned_call_after_incarnation_replacement_reaches_the_new_holder` (both), `tests/sol_review_races.rs::caller_disconnect_cleanup_works_before_and_after_consumption` for the old calls half (its synchronisation was fixed on 2026-09-22; see "A flaky test and what it was measuring") |
|
||||
| An RPC targets one registered service, not a broadcast subject | conforms | `router/state.rs::op_call` | `tests/rpc.rs::request_reply_roundtrip` (both) |
|
||||
| First-dispatch FIFO per caller and service; responses may complete out of order and correlate by callId | conforms | `router/state.rs::{Svc::queue, dispatch_rpc}` | `tests/rpc.rs::fifo_dispatch_and_out_of_order_completion` (both), `tests/conformance_routing.rs::out_of_order_replies_correlate_across_concurrent_callers` (both) |
|
||||
| A service dispatcher can answer status concurrently with a long mutation | conforms | `router/state.rs::dispatch_rpc` (in-flight credits, not one-at-a-time) | `tests/bus_acceptance.rs::a_status_rpc_responds_while_another_handler_is_delayed` (both) |
|
||||
|
|
@ -412,6 +412,45 @@ What the numbers do and do not say:
|
|||
cores against 0.33 to 0.46), because the clients own the copies. A thread that exits between
|
||||
two samples takes its CPU with it, so the router figure is a floor.
|
||||
|
||||
## A flaky test and what it was measuring, 2026-09-22 (MEDIA-01)
|
||||
|
||||
`tests/sol_review_races.rs::caller_disconnect_cleanup_works_before_and_after_consumption`
|
||||
failed intermittently on `main` after the bus slice merged. Reproduced here at
|
||||
**37 failures in 240 runs** (four parallel loops of 60, debug, on the loaded dev VM), always
|
||||
on the same line and always the same way: `responder.reply(...)` returned `routed:true` where
|
||||
the test asserted `false`.
|
||||
|
||||
The mechanism is a synchronisation gap in the test, not a routing defect.
|
||||
`router/state.rs::op_reply` returns `routed:false` only when `call.detached` is set, and for a
|
||||
disconnected caller that flag is set by `router/state.rs::disconnect`, which the router runs
|
||||
when **its** connection task reads EOF. `Client::close` documents what it waits for — "flushes
|
||||
queued releases, closes the connection and waits until the reader has stopped ... the router
|
||||
releases what they owned" — which is the client side only. So after `close()` returns, the
|
||||
router may not have torn the caller's connection down yet, and a reply that reaches it first is
|
||||
routed to a connection that is already closing. Nothing escapes: `disconnect` then releases
|
||||
that connection's roots along with the queued result, which is why the test's own later
|
||||
`settle` calls always passed. Only the `routed` flag, read one step too early, was wrong.
|
||||
|
||||
The fix is in the test: it now waits for the teardown it is talking about
|
||||
(`e.settle("caller-a disconnected", |s| s.connections == 1)`) before asserting the
|
||||
reply-to-a-detached-call sentence of section 6. That is the same bounded
|
||||
poll-until-the-router-settles the rest of the file already uses for router-side consequences;
|
||||
no sleep, no timing constant, and the assertion now has the precondition its contract sentence
|
||||
names. **360 runs after the fix, 0 failures** (240 debug, 120 release).
|
||||
|
||||
Two other intermittent failures were seen in the same sweep and are **not** fixed here, since
|
||||
they belong to the bus slice rather than to this one:
|
||||
|
||||
- `tests/example_demo.rs::the_example_shows_a_counter_rpc_an_observer_and_a_held_frame`,
|
||||
2 failures in 40 standalone runs plus 1 in 12 full-suite runs. It prints
|
||||
"while the frame is held: 1 artifact(s), 2 root(s)" instead of 1 root: the producer's hold
|
||||
release is queued on the control lane and had not been applied when the example read the
|
||||
counts. The same shape of gap, in the guide deliverable's printed output.
|
||||
- `tests/integration.rs::unix_socket::session_over_one_router`, 1 failure in 12 full-suite
|
||||
runs and 0 in 40 standalone runs, at the assertion that the deliberately slow consumer
|
||||
skipped snapshots. Under load it kept up, so the assertion is a timing claim about the
|
||||
machine.
|
||||
|
||||
## Contradictions
|
||||
|
||||
Two, both inside bus-v1, both minor, neither resolved by changing code. Both were referred to
|
||||
|
|
|
|||
|
|
@ -94,10 +94,14 @@ interface HelloResult {
|
|||
workerId: Id; incarnationId: Id; role: "agent" | "environment" | "coordinator";
|
||||
buildDigest: Digest; contractDigest: Digest;
|
||||
capabilities: Id[];
|
||||
limits: { maxAgents: number; maxPorts: number };
|
||||
limits: { maxAgents: number; maxPorts: number; workerThreads: number };
|
||||
}
|
||||
```
|
||||
|
||||
`limits.workerThreads` is the thread allocation the worker's launcher started it within; it
|
||||
is an integer >=1 and its rule belongs to [worker interfaces](workers-v1.md) section 2, whose
|
||||
2026-09-22 amendment added it.
|
||||
|
||||
The bus supplies caller identity; do not accept a forged caller in params. Bind a worker's
|
||||
session authority to the expected coordinator identity/incarnation during negotiation and
|
||||
initialization. Wrong worker/role, no common major or missing required capability refuses
|
||||
|
|
|
|||
|
|
@ -65,6 +65,23 @@ the sample position relative to the episode's configured audio origin, with inte
|
|||
firstSample/sampleRate. Crash restore preserves sample position under a new epoch; first
|
||||
chunk marks discontinuity. Within an epoch, chunks cannot overlap or go backwards.
|
||||
|
||||
**Amendment, 2026-09-22 (MEDIA-01).** Two readings of the paragraphs above, made explicit
|
||||
because they are now enforced:
|
||||
|
||||
- The bootstrap window is exactly the boundaries where `max(0, boundary - observationDelaySteps)`
|
||||
is zero, that is `boundary <= observationDelaySteps`. Inside it the repeated `O[0]` is the
|
||||
**same artifact**, not a fresh render of the same scene; outside it the producing boundary
|
||||
advances one per step, and a frame from any other boundary -- older or newer -- is a step
|
||||
failure. A producer therefore keeps a queue of `observationDelaySteps + 1` frames and nothing
|
||||
more, so there is no older frame available to substitute.
|
||||
- Within an epoch, `discontinuity` marks a range the stream actually skipped. The first chunk
|
||||
after a restore marks it, and a later chunk may mark it when it starts past where the previous
|
||||
chunk ended; a chunk that continues the previous one exactly is continuous by construction and
|
||||
its flag is refused. Without that reading the restore rule is advisory, because a stream could
|
||||
set the flag on every chunk and satisfy it by accident. The requirement is one-directional: a
|
||||
fresh epoch's first chunk **may** mark a discontinuity, because section 6's recovery
|
||||
establishes a fresh timeline and publishes one.
|
||||
|
||||
The environment provides **native game output**. Sensor transformations belong to the agent
|
||||
profile. Resizing for viewers, overlays, composition, audio mixing/resampling, encoding,
|
||||
browser delivery and streaming belong to the application/presentation layer. No bus or
|
||||
|
|
|
|||
|
|
@ -99,6 +99,16 @@ interface AgentInitializeResult {
|
|||
}
|
||||
```
|
||||
|
||||
**Amendment, 2026-09-22 (SESSION-02).** `HelloResult.limits` gains `workerThreads`, an
|
||||
integer >=1 reporting the allocation the launcher started that worker within, because
|
||||
"within launcher allocation" above had no wire-level proof: the launcher passes the number to
|
||||
the worker out of band, and a coordinator that is not also its own launcher had no contract
|
||||
path to it. Hello is where a worker already proves its identity and reports its limits, so the
|
||||
allocation belongs there. A caller asking for more than the worker reports is refused with
|
||||
`BUSY` before the model is constructed, which this section already required; the amendment
|
||||
only makes the number visible to whoever must respect it. It changes `contractDigest`, which
|
||||
[session RPC](ipc-v1.md) section 4 already provides for.
|
||||
|
||||
The profile fixes warm-up/calibration behavior and supported schema versions. Validate inputs
|
||||
and required roles before model construction. Install the initial sensory input, warm the
|
||||
brain with learning disabled, calibrate the fixed readout and establish Ready(0). Do not
|
||||
|
|
|
|||
|
|
@ -684,3 +684,44 @@ then purge; then the stale-doc pass.
|
|||
ground with a new trap (row 54: GO FRONTIER, GO HEAL, GO ROUTE cycling, GO HEAL x204 at net 0);
|
||||
Fable shipped it anyway: the hunt criterion compares within ground both arms reach, and a trap on
|
||||
newly opened ground is a new row, not a regression. Row 54 review started at once.
|
||||
|
||||
## 2026-09-22 - session framework wave 2
|
||||
|
||||
Per-fly processes and native observations landed on top of the wave-1 contract and
|
||||
lockstep session.
|
||||
|
||||
- The session runs in three execution modes that share one coordinator, one worker and
|
||||
one router: in process, one thread per participant, and one process per fly plus one
|
||||
for the environment over Unix sockets. The worker is a subcommand of the existing
|
||||
binary, not a new crate. A launcher owns the thread budget, proves each participant's
|
||||
configured identity and allocation on the wire before the coordinator pins anything,
|
||||
polls health on its own clock, and reaps its children.
|
||||
- A caller deadline that expires no longer fails the epoch on its own. It runs the
|
||||
contract's resolution procedure against the same request and the same incarnation, so
|
||||
a merely slow participant completes its step, and the epoch fails only on a definite
|
||||
refusal, a lost incarnation or an exhausted budget. Which of the two bounds ended a
|
||||
resolution is recorded and named in the failure rather than inferred.
|
||||
- Failures name the participant, and a failed session fences its epoch: the boundary
|
||||
stops, handles are dropped, and no further transition or publication is possible.
|
||||
Worker death, helper death, a router restart mid-advance and stale replies after a
|
||||
restart all have bounded, diagnosed outcomes, each proved in every mode.
|
||||
- The environment now emits native output: one shared RGBA frame per boundary reaching
|
||||
both flies through owned attachments, and one audio chunk per transition on an exact
|
||||
rational sample budget. Observation delay is a real queue, so nothing stale can be
|
||||
substituted. Spectators watch a latest subscription with finite credits and cannot
|
||||
perturb what the flies sense. Audio never enters sensory input.
|
||||
- Every media rule is enforced rather than assumed: strides, dimensions, formats,
|
||||
lengths, producing step, timeline continuity, and the distinction between a persistent
|
||||
asset and a transient artifact. A missing or malformed frame or chunk fails its step.
|
||||
- Three contract silences were closed by dated amendments rather than by convention: the
|
||||
launcher's thread allocation is now on the wire in the worker's hello, the audio
|
||||
discontinuity rule is one-directional, and a mid-step pause is defined.
|
||||
|
||||
Measured on the development box, not capacity claims: with two flies the critical path
|
||||
per transition sat near 10 to 12 ms at the median across all three modes, so a process
|
||||
boundary costs little at the median and shows in the tail. What the split costs is
|
||||
memory, roughly 5.7 MiB per participant process, while the coordinator's own footprint
|
||||
is flat and lowest once the workers leave it.
|
||||
|
||||
Two flaky bus tests predating this work assert timing rather than contract and are being
|
||||
rewritten separately.
|
||||
|
|
|
|||
|
|
@ -43,6 +43,8 @@ export const MAX_ACKNOWLEDGE = 16;
|
|||
export const MAX_ENGINE_FRAME_LEN = 64;
|
||||
/** Not stated by a document; this crate's choice, published in the schema set. */
|
||||
export const MAX_CAPABILITIES = 32;
|
||||
/** The largest `workerThreads` a launcher may allocate to one worker (workers-v1 2). */
|
||||
export const MAX_WORKER_THREADS = 4096;
|
||||
export const MAX_SUPPORTED_MAJORS = 8;
|
||||
export const MAX_MESSAGE_CODE_POINTS = 512;
|
||||
|
||||
|
|
@ -305,7 +307,7 @@ export function readAgentInitializeParams(value: unknown): AgentInitializeParams
|
|||
seed: reader.int('seed', -2_147_483_648, 2_147_483_647),
|
||||
initialInput: readSensoryInput(reader.value('initialInput')),
|
||||
initialDecisionContext: readTypedValue(reader.value('initialDecisionContext')),
|
||||
workerThreads: reader.int('workerThreads', 1, 4_096),
|
||||
workerThreads: reader.int('workerThreads', 1, MAX_WORKER_THREADS),
|
||||
};
|
||||
reader.finish();
|
||||
return params;
|
||||
|
|
@ -776,7 +778,12 @@ export interface HelloResult {
|
|||
buildDigest: Digest;
|
||||
contractDigest: Digest;
|
||||
capabilities: Id[];
|
||||
limits: { maxAgents: number; maxPorts: number };
|
||||
/**
|
||||
* `workerThreads` is the thread allocation this worker was launched within, added by the
|
||||
* 2026-09-22 amendment to workers-v1 section 2: the bound "within launcher allocation" had
|
||||
* no wire on which a caller could learn the allocation.
|
||||
*/
|
||||
limits: { maxAgents: number; maxPorts: number; workerThreads: number };
|
||||
}
|
||||
|
||||
export interface StatusResult {
|
||||
|
|
@ -860,6 +867,7 @@ export function readHelloResult(value: unknown): HelloResult {
|
|||
const limits = {
|
||||
maxAgents: limitsReader.int('maxAgents', 1, MAX_AGENTS),
|
||||
maxPorts: limitsReader.int('maxPorts', 1, MAX_PORTS),
|
||||
workerThreads: limitsReader.int('workerThreads', 1, MAX_WORKER_THREADS),
|
||||
};
|
||||
limitsReader.finish();
|
||||
reader.finish();
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
{
|
||||
"description": "contractDigest is the SHA-256 of the canonical schema set in schema-set.json.",
|
||||
"contractDigest": "7932aef30c4d2d16e428081affc4e0ad187987f5b361138d553e54fd7f843b50",
|
||||
"contractDigest": "d8f29a49b5df05ad8f75f7f5790a3f8cde9c5ad23a685137474c649c3c9da36d",
|
||||
"schemaSetVersion": 1,
|
||||
"schemaSetBytes": 26685,
|
||||
"schemaSetBytes": 26814,
|
||||
"types": 53,
|
||||
"enums": 11,
|
||||
"limits": 25
|
||||
"limits": 26
|
||||
}
|
||||
|
|
|
|||
|
|
@ -2802,7 +2802,8 @@
|
|||
],
|
||||
"limits": {
|
||||
"maxAgents": 4,
|
||||
"maxPorts": 4
|
||||
"maxPorts": 4,
|
||||
"workerThreads": 1
|
||||
}
|
||||
},
|
||||
"reason": "an agent must advertise agent-step-v1"
|
||||
|
|
@ -2823,7 +2824,8 @@
|
|||
],
|
||||
"limits": {
|
||||
"maxAgents": 5,
|
||||
"maxPorts": 4
|
||||
"maxPorts": 4,
|
||||
"workerThreads": 1
|
||||
}
|
||||
},
|
||||
"reason": "the first composition allows four agents"
|
||||
|
|
@ -2844,11 +2846,77 @@
|
|||
],
|
||||
"limits": {
|
||||
"maxAgents": 4,
|
||||
"maxPorts": 4
|
||||
"maxPorts": 4,
|
||||
"workerThreads": 1
|
||||
}
|
||||
},
|
||||
"reason": "v1 selects major 1"
|
||||
},
|
||||
{
|
||||
"name": "hello result without its launcher allocation",
|
||||
"type": "HelloResult",
|
||||
"value": {
|
||||
"selectedMajor": 1,
|
||||
"selectedMinor": 0,
|
||||
"workerId": "fly-a",
|
||||
"incarnationId": "inc-1",
|
||||
"role": "agent",
|
||||
"buildDigest": "44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66",
|
||||
"contractDigest": "cc8321d6375c494d043fdd0260f21bc0ec51dacc9f6abb7f909cdcd3041b78bf",
|
||||
"capabilities": [
|
||||
"agent-step-v1"
|
||||
],
|
||||
"limits": {
|
||||
"maxAgents": 4,
|
||||
"maxPorts": 4
|
||||
}
|
||||
},
|
||||
"reason": "limits.workerThreads is required by the 2026-09-22 workers-v1 amendment"
|
||||
},
|
||||
{
|
||||
"name": "hello result promising more threads than a launcher may allocate",
|
||||
"type": "HelloResult",
|
||||
"value": {
|
||||
"selectedMajor": 1,
|
||||
"selectedMinor": 0,
|
||||
"workerId": "fly-a",
|
||||
"incarnationId": "inc-1",
|
||||
"role": "agent",
|
||||
"buildDigest": "44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66",
|
||||
"contractDigest": "cc8321d6375c494d043fdd0260f21bc0ec51dacc9f6abb7f909cdcd3041b78bf",
|
||||
"capabilities": [
|
||||
"agent-step-v1"
|
||||
],
|
||||
"limits": {
|
||||
"maxAgents": 4,
|
||||
"maxPorts": 4,
|
||||
"workerThreads": 4097
|
||||
}
|
||||
},
|
||||
"reason": "limits.workerThreads is at most maxWorkerThreads"
|
||||
},
|
||||
{
|
||||
"name": "hello result reporting no threads at all",
|
||||
"type": "HelloResult",
|
||||
"value": {
|
||||
"selectedMajor": 1,
|
||||
"selectedMinor": 0,
|
||||
"workerId": "fly-a",
|
||||
"incarnationId": "inc-1",
|
||||
"role": "agent",
|
||||
"buildDigest": "44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66",
|
||||
"contractDigest": "cc8321d6375c494d043fdd0260f21bc0ec51dacc9f6abb7f909cdcd3041b78bf",
|
||||
"capabilities": [
|
||||
"agent-step-v1"
|
||||
],
|
||||
"limits": {
|
||||
"maxAgents": 4,
|
||||
"maxPorts": 4,
|
||||
"workerThreads": 0
|
||||
}
|
||||
},
|
||||
"reason": "limits.workerThreads is at least one"
|
||||
},
|
||||
{
|
||||
"name": "hello params with no supported majors",
|
||||
"type": "HelloParams",
|
||||
|
|
@ -3897,4 +3965,4 @@
|
|||
"reason": "a commit acknowledges the transition's next boundary"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
|
|
|
|||
File diff suppressed because one or more lines are too long
|
|
@ -1758,12 +1758,13 @@
|
|||
],
|
||||
"limits": {
|
||||
"maxAgents": 4,
|
||||
"maxPorts": 4
|
||||
"maxPorts": 4,
|
||||
"workerThreads": 1
|
||||
}
|
||||
},
|
||||
"note": "",
|
||||
"canonical": "{\"buildDigest\":\"44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66\",\"capabilities\":[\"agent-step-v1\",\"checkpoint-v1\",\"pixel-observation-v1\"],\"contractDigest\":\"cc8321d6375c494d043fdd0260f21bc0ec51dacc9f6abb7f909cdcd3041b78bf\",\"incarnationId\":\"inc-1\",\"limits\":{\"maxAgents\":4,\"maxPorts\":4},\"role\":\"agent\",\"selectedMajor\":1,\"selectedMinor\":0,\"workerId\":\"fly-a\"}",
|
||||
"digest": "9f8cbc9dfe7a7532c2e41b53c4ce001973ca95703a81aad30ee2b1c8b640bc13"
|
||||
"canonical": "{\"buildDigest\":\"44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66\",\"capabilities\":[\"agent-step-v1\",\"checkpoint-v1\",\"pixel-observation-v1\"],\"contractDigest\":\"cc8321d6375c494d043fdd0260f21bc0ec51dacc9f6abb7f909cdcd3041b78bf\",\"incarnationId\":\"inc-1\",\"limits\":{\"maxAgents\":4,\"maxPorts\":4,\"workerThreads\":1},\"role\":\"agent\",\"selectedMajor\":1,\"selectedMinor\":0,\"workerId\":\"fly-a\"}",
|
||||
"digest": "262ea112e24dcdbdc1a5050b6dd6d031eace6a6a4cdca814913ac89fc9e39666"
|
||||
},
|
||||
{
|
||||
"name": "hello result for an environment",
|
||||
|
|
@ -1782,12 +1783,13 @@
|
|||
],
|
||||
"limits": {
|
||||
"maxAgents": 1,
|
||||
"maxPorts": 1
|
||||
"maxPorts": 1,
|
||||
"workerThreads": 1
|
||||
}
|
||||
},
|
||||
"note": "",
|
||||
"canonical": "{\"buildDigest\":\"44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66\",\"capabilities\":[\"world-step-v1\",\"checkpoint-v1\"],\"contractDigest\":\"cc8321d6375c494d043fdd0260f21bc0ec51dacc9f6abb7f909cdcd3041b78bf\",\"incarnationId\":\"inc-2\",\"limits\":{\"maxAgents\":1,\"maxPorts\":1},\"role\":\"environment\",\"selectedMajor\":1,\"selectedMinor\":0,\"workerId\":\"world\"}",
|
||||
"digest": "0dc33a6cbf6bf66c4d28dc41b17b2eb7918b55c65469f4161e009f662f11bca2"
|
||||
"canonical": "{\"buildDigest\":\"44575cf5b28512d75644bf54a517dcef304ff809fd511747621b4d64f19aac66\",\"capabilities\":[\"world-step-v1\",\"checkpoint-v1\"],\"contractDigest\":\"cc8321d6375c494d043fdd0260f21bc0ec51dacc9f6abb7f909cdcd3041b78bf\",\"incarnationId\":\"inc-2\",\"limits\":{\"maxAgents\":1,\"maxPorts\":1,\"workerThreads\":1},\"role\":\"environment\",\"selectedMajor\":1,\"selectedMinor\":0,\"workerId\":\"world\"}",
|
||||
"digest": "06ced6fff0d810005ab2598cc9fb72a138303daef0787232871c9afa43b0eac0"
|
||||
},
|
||||
{
|
||||
"name": "status result before initialization",
|
||||
|
|
|
|||
|
|
@ -366,6 +366,161 @@ impl DomainType for AudioRef {
|
|||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
// Epoch audio sequencing (state-media-v1 section 2)
|
||||
|
||||
/// One audio stream's chunk sequence within one epoch.
|
||||
///
|
||||
/// The contract's three sentences about sequencing are all here: `firstSample` identifies the
|
||||
/// sample position relative to the episode's configured audio origin; crash restore preserves
|
||||
/// that position under a new epoch and the first chunk marks `discontinuity`; within an epoch
|
||||
/// chunks cannot overlap or go backwards.
|
||||
///
|
||||
/// A timeline belongs to one epoch. A restore starts a new one with
|
||||
/// [`AudioTimeline::restored_at`], which is what makes the first chunk's discontinuity flag
|
||||
/// checkable rather than advisory.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct AudioTimeline {
|
||||
stream_id: String,
|
||||
start_sample: u64,
|
||||
restored: bool,
|
||||
next_sample: u64,
|
||||
accepted: u64,
|
||||
}
|
||||
|
||||
impl AudioTimeline {
|
||||
/// A fresh episode: the first chunk starts at the configured audio origin. Its
|
||||
/// discontinuity flag is free, because a reset or a recovery establishes a fresh timeline
|
||||
/// and does publish a discontinuity.
|
||||
pub fn fresh(descriptor: &AudioDescriptor, origin: u64) -> AudioTimeline {
|
||||
AudioTimeline {
|
||||
stream_id: descriptor.stream_id.clone(),
|
||||
start_sample: origin,
|
||||
restored: false,
|
||||
next_sample: origin,
|
||||
accepted: 0,
|
||||
}
|
||||
}
|
||||
|
||||
/// A new epoch after a crash restore: the sample position is preserved, and the first
|
||||
/// chunk of this epoch must mark `discontinuity`.
|
||||
pub fn restored_at(descriptor: &AudioDescriptor, sample: u64) -> AudioTimeline {
|
||||
AudioTimeline {
|
||||
stream_id: descriptor.stream_id.clone(),
|
||||
start_sample: sample,
|
||||
restored: true,
|
||||
next_sample: sample,
|
||||
accepted: 0,
|
||||
}
|
||||
}
|
||||
|
||||
/// Where the next chunk may start. A chunk starting earlier overlaps or goes backwards.
|
||||
pub fn next_sample(&self) -> u64 {
|
||||
self.next_sample
|
||||
}
|
||||
|
||||
/// How many chunks this epoch has accepted.
|
||||
pub fn accepted(&self) -> u64 {
|
||||
self.accepted
|
||||
}
|
||||
|
||||
/// Validates one chunk's shape and its place in the sequence, then advances the timeline.
|
||||
///
|
||||
/// A rejected chunk does not advance anything, so a caller that fails its step does not
|
||||
/// leave the timeline believing the chunk was played.
|
||||
pub fn accept(&mut self, chunk: &AudioRef, descriptor: &AudioDescriptor) -> Result<()> {
|
||||
if chunk.stream_id != self.stream_id {
|
||||
return err(format!(
|
||||
"AudioTimeline {}: chunk names stream {:?}",
|
||||
self.stream_id, chunk.stream_id
|
||||
));
|
||||
}
|
||||
chunk.validate_against(descriptor)?;
|
||||
if self.accepted == 0 {
|
||||
if chunk.first_sample != self.start_sample {
|
||||
return err(format!(
|
||||
"AudioTimeline {}: the first chunk of this epoch must start at sample {}, not {}",
|
||||
self.stream_id, self.start_sample, chunk.first_sample
|
||||
));
|
||||
}
|
||||
// Only one direction is stated: the first chunk after a restore marks the
|
||||
// discontinuity. A fresh epoch's first chunk may mark one too -- sections 6 and 7
|
||||
// have recovery and episode reset publishing a discontinuity on a fresh timeline --
|
||||
// so the flag is required after a restore and left free at an origin.
|
||||
if self.restored && !chunk.discontinuity {
|
||||
return err(format!(
|
||||
"AudioTimeline {}: the first chunk after a restore marks discontinuity",
|
||||
self.stream_id
|
||||
));
|
||||
}
|
||||
} else {
|
||||
if chunk.first_sample < self.next_sample {
|
||||
return err(format!(
|
||||
"AudioTimeline {}: firstSample {} overlaps or goes backwards; the previous chunk ends at {}",
|
||||
self.stream_id, chunk.first_sample, self.next_sample
|
||||
));
|
||||
}
|
||||
// Derived from the sentence above: within an epoch the only discontinuity a chunk
|
||||
// can carry is a gap it actually skipped. A chunk that continues the previous one
|
||||
// exactly is continuous by construction.
|
||||
if chunk.discontinuity && chunk.first_sample == self.next_sample {
|
||||
return err(format!(
|
||||
"AudioTimeline {}: a chunk continuing the previous one is not a discontinuity",
|
||||
self.stream_id
|
||||
));
|
||||
}
|
||||
}
|
||||
self.next_sample = chunk
|
||||
.first_sample
|
||||
.checked_add(chunk.sample_frames)
|
||||
.ok_or_else(|| {
|
||||
crate::scalar::wire_err(format!(
|
||||
"AudioTimeline {}: firstSample + sampleFrames overflows U64",
|
||||
self.stream_id
|
||||
))
|
||||
})?;
|
||||
self.accepted += 1;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------------------------
|
||||
// Persistent assets against transient artifacts (state-media-v1 sections 1 and 3)
|
||||
|
||||
/// Checks that a transient artifact carries the bytes of an installed asset.
|
||||
///
|
||||
/// Persistent [`AssetRef`](crate::workers::AssetRef) and transient
|
||||
/// [`ArtifactRef`] are different identities and never convert into one another: an asset names
|
||||
/// installed release content in a preprovisioned registry, while an artifact names live bytes
|
||||
/// in one store incarnation and resolves only through an owned handle. Importing an asset
|
||||
/// produces a **new** artifact identity, which is why this checks content rather than identity.
|
||||
///
|
||||
/// The digest is mandatory here: state-media-v1 section 1 makes content digests optional on
|
||||
/// transient live frames and mandatory on persistent asset import.
|
||||
pub fn check_imported_asset(
|
||||
asset: &crate::workers::AssetRef,
|
||||
imported: &ArtifactRef,
|
||||
) -> Result<()> {
|
||||
asset.validate()?;
|
||||
if imported.byte_length != asset.byte_length {
|
||||
return err(format!(
|
||||
"imported asset {}: the artifact is {} bytes, the asset is {}",
|
||||
asset.id, imported.byte_length, asset.byte_length
|
||||
));
|
||||
}
|
||||
match &imported.digest {
|
||||
Some(d) if *d == asset.digest => Ok(()),
|
||||
Some(_) => err(format!(
|
||||
"imported asset {}: the artifact's digest is not the asset's content",
|
||||
asset.id
|
||||
)),
|
||||
None => err(format!(
|
||||
"imported asset {}: a persistent asset import must carry a content digest",
|
||||
asset.id
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
/// Reads a bounded, unique-by-`viewId` list of view refs.
|
||||
pub fn view_list(f: &mut Fields<'_>, key: &'static str) -> Result<Vec<ViewRef>> {
|
||||
let views = list(f, key, 0, MAX_VIEWS, ViewRef::from_json)?;
|
||||
|
|
|
|||
|
|
@ -239,6 +239,11 @@ pub const LIMITS: &[LimitSchema] = &[
|
|||
value: crate::workers::MAX_CAPABILITIES as u64,
|
||||
source: "crate",
|
||||
},
|
||||
LimitSchema {
|
||||
name: "maxWorkerThreads",
|
||||
value: crate::workers::MAX_WORKER_THREADS,
|
||||
source: "workers-v1 2",
|
||||
},
|
||||
LimitSchema {
|
||||
name: "maxSupportedMajors",
|
||||
value: crate::workers::MAX_SUPPORTED_MAJORS as u64,
|
||||
|
|
@ -651,8 +656,8 @@ pub const SCHEMAS: &[TypeSchema] = &[
|
|||
),
|
||||
req(
|
||||
"limits",
|
||||
"{maxAgents:int,maxPorts:int}",
|
||||
"1..=4 agents and 1..=4 ports",
|
||||
"{maxAgents:int,maxPorts:int,workerThreads:int}",
|
||||
"1..=4 agents, 1..=4 ports, and the launcher allocation this worker runs in",
|
||||
),
|
||||
],
|
||||
},
|
||||
|
|
|
|||
|
|
@ -33,6 +33,9 @@ pub const MAX_ACKNOWLEDGE: usize = 16;
|
|||
pub const MAX_ENGINE_FRAME_LEN: usize = 64;
|
||||
/// Negotiated capability ids. Not a stated bound; recorded in the schema set.
|
||||
pub const MAX_CAPABILITIES: usize = 32;
|
||||
|
||||
/// The largest `workerThreads` a launcher may allocate to one worker (`workers-v1` 2).
|
||||
pub const MAX_WORKER_THREADS: u64 = 4_096;
|
||||
/// Supported majors in Worker.Hello. Not a stated bound; recorded in the schema set.
|
||||
pub const MAX_SUPPORTED_MAJORS: usize = 8;
|
||||
/// Domain error messages are <=512 code points (ipc-v1 section 7).
|
||||
|
|
@ -651,7 +654,7 @@ impl DomainType for AgentInitializeParams {
|
|||
let seed = i32_field(&mut f, "seed")?;
|
||||
let initial_input = SensoryInput::from_json(f.value("initialInput")?)?;
|
||||
let initial_decision_context = TypedValue::from_json(f.value("initialDecisionContext")?)?;
|
||||
let worker_threads = f.int("workerThreads", 1, 4_096)?;
|
||||
let worker_threads = f.int("workerThreads", 1, MAX_WORKER_THREADS)?;
|
||||
f.finish()?;
|
||||
let p = AgentInitializeParams {
|
||||
agent_id,
|
||||
|
|
@ -1958,6 +1961,13 @@ pub struct HelloResult {
|
|||
pub capabilities: Vec<String>,
|
||||
pub max_agents: u64,
|
||||
pub max_ports: u64,
|
||||
/// The thread allocation this worker was launched within.
|
||||
///
|
||||
/// `workers-v1` bounds `Agent.Initialize`'s `workerThreads` by "within launcher
|
||||
/// allocation" and, before the 2026-09-22 amendment, named no wire on which a caller could
|
||||
/// learn it. This is that wire: the worker reports what its launcher gave it, and a caller
|
||||
/// that meant to ask for more finds out here rather than after the model exists.
|
||||
pub worker_threads: u64,
|
||||
}
|
||||
|
||||
impl HelloResult {
|
||||
|
|
@ -1990,13 +2000,14 @@ impl DomainType for HelloResult {
|
|||
let build_digest = f.string("buildDigest")?.to_owned();
|
||||
let contract_digest = f.string("contractDigest")?.to_owned();
|
||||
let capabilities = id_list(&mut f, "capabilities", 0, MAX_CAPABILITIES)?;
|
||||
let (max_agents, max_ports) = {
|
||||
let (max_agents, max_ports, worker_threads) = {
|
||||
let v = f.value("limits")?;
|
||||
let mut l = Fields::new(v, "HelloResult.limits")?;
|
||||
let max_agents = l.int("maxAgents", 1, MAX_AGENTS as u64)?;
|
||||
let max_ports = l.int("maxPorts", 1, MAX_PORTS as u64)?;
|
||||
let worker_threads = l.int("workerThreads", 1, MAX_WORKER_THREADS)?;
|
||||
l.finish()?;
|
||||
(max_agents, max_ports)
|
||||
(max_agents, max_ports, worker_threads)
|
||||
};
|
||||
f.finish()?;
|
||||
let r = HelloResult {
|
||||
|
|
@ -2008,6 +2019,7 @@ impl DomainType for HelloResult {
|
|||
capabilities,
|
||||
max_agents,
|
||||
max_ports,
|
||||
worker_threads,
|
||||
};
|
||||
r.validate()?;
|
||||
Ok(r)
|
||||
|
|
@ -2031,6 +2043,7 @@ impl DomainType for HelloResult {
|
|||
obj(vec![
|
||||
("maxAgents", Value::from(self.max_agents)),
|
||||
("maxPorts", Value::from(self.max_ports)),
|
||||
("workerThreads", Value::from(self.worker_threads)),
|
||||
]),
|
||||
),
|
||||
])
|
||||
|
|
|
|||
395
services/flysim/crates/fly-session-types/tests/media_shapes.rs
Normal file
395
services/flysim/crates/fly-session-types/tests/media_shapes.rs
Normal file
|
|
@ -0,0 +1,395 @@
|
|||
//! MEDIA-01 shape rules: every sentence of `state-media-v1` section 2 that a descriptor, a
|
||||
//! reference or a chunk sequence can be checked against on its own.
|
||||
//!
|
||||
//! These are the contract-level halves of the slice's acceptance bullets -- bad strides, bad
|
||||
//! lengths, bad producing times and the audio rules -- and the type-level distinction between
|
||||
//! a persistent `AssetRef` and a transient `ArtifactRef`.
|
||||
|
||||
use fly_session_types::ArtifactRef;
|
||||
use fly_session_types::media::{
|
||||
AudioDescriptor, AudioRef, AudioTimeline, ViewDescriptor, ViewRef, check_imported_asset,
|
||||
require_finite_samples,
|
||||
};
|
||||
use fly_session_types::scalar::DomainType;
|
||||
use fly_session_types::workers::AssetRef;
|
||||
|
||||
fn artifact(byte_length: u64, content_type: &str) -> ArtifactRef {
|
||||
ArtifactRef {
|
||||
store_id: "store-a".into(),
|
||||
artifact_id: "art-1".into(),
|
||||
generation: 1,
|
||||
byte_length,
|
||||
content_type: content_type.into(),
|
||||
digest: None,
|
||||
}
|
||||
}
|
||||
|
||||
fn view_descriptor(width: u64, height: u64, delay: u64) -> ViewDescriptor {
|
||||
ViewDescriptor {
|
||||
view_id: "arena".into(),
|
||||
width,
|
||||
height,
|
||||
row_stride: width * 4,
|
||||
pixel_aspect_numerator: 1,
|
||||
pixel_aspect_denominator: 1,
|
||||
observation_delay_steps: delay,
|
||||
}
|
||||
}
|
||||
|
||||
fn view_ref(descriptor: &ViewDescriptor, produced_step: u64, bytes: u64) -> ViewRef {
|
||||
ViewRef {
|
||||
view_id: descriptor.view_id.clone(),
|
||||
produced_step,
|
||||
pixels: artifact(bytes, "image/x-rgba8"),
|
||||
}
|
||||
}
|
||||
|
||||
fn audio_descriptor(sample_rate: u64, channels: u64) -> AudioDescriptor {
|
||||
AudioDescriptor {
|
||||
stream_id: "arena".into(),
|
||||
sample_rate,
|
||||
channels,
|
||||
}
|
||||
}
|
||||
|
||||
fn audio_ref(
|
||||
descriptor: &AudioDescriptor,
|
||||
first_sample: u64,
|
||||
frames: u64,
|
||||
discontinuity: bool,
|
||||
) -> AudioRef {
|
||||
AudioRef {
|
||||
stream_id: descriptor.stream_id.clone(),
|
||||
first_sample,
|
||||
sample_frames: frames,
|
||||
samples: artifact(frames * descriptor.channels * 4, "audio/x-f32le"),
|
||||
discontinuity,
|
||||
}
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------------------------
|
||||
// Bad strides
|
||||
|
||||
/// `rowStride` is exactly `4 x width`; v1 has no padded rows.
|
||||
#[test]
|
||||
fn a_padded_row_stride_is_refused() {
|
||||
let good = view_descriptor(32, 24, 0);
|
||||
good.validate().expect("4 x width is the only stride");
|
||||
|
||||
let mut padded = good.clone();
|
||||
padded.row_stride = 32 * 4 + 16;
|
||||
padded.validate().expect_err("a padded row is not readable in v1");
|
||||
|
||||
let mut narrow = good.clone();
|
||||
narrow.row_stride = 32 * 3;
|
||||
narrow.validate().expect_err("a stride under 4 x width is refused");
|
||||
|
||||
// The same rule through the wire form, where a hand-written descriptor arrives.
|
||||
let mut json = good.to_json();
|
||||
json["rowStride"] = serde_json::json!(32 * 4 + 4);
|
||||
ViewDescriptor::from_json(&json).expect_err("a padded stride is refused when read");
|
||||
}
|
||||
|
||||
/// Dimensions are integers 1..=4096, and `pixelAspect` parts positive integers <=65535.
|
||||
#[test]
|
||||
fn dimensions_pixel_aspect_and_delay_have_stated_bounds() {
|
||||
for (width, height) in [(0, 24), (32, 0), (4097, 24), (32, 4097)] {
|
||||
let mut d = view_descriptor(32, 24, 0);
|
||||
d.width = width;
|
||||
d.height = height;
|
||||
d.row_stride = width.max(1) * 4;
|
||||
d.validate().expect_err("dimensions are 1..=4096");
|
||||
}
|
||||
view_descriptor(1, 1, 0).validate().expect("1x1 is inside the bounds");
|
||||
view_descriptor(4096, 4096, 0)
|
||||
.validate()
|
||||
.expect("4096x4096 is inside the bounds");
|
||||
|
||||
for (numerator, denominator) in [(0, 1), (1, 0), (65_536, 1), (1, 65_536)] {
|
||||
let mut d = view_descriptor(32, 24, 0);
|
||||
d.pixel_aspect_numerator = numerator;
|
||||
d.pixel_aspect_denominator = denominator;
|
||||
d.validate().expect_err("pixelAspect parts are 1..=65535");
|
||||
}
|
||||
|
||||
let mut d = view_descriptor(32, 24, 9);
|
||||
d.validate().expect_err("observationDelaySteps is 0..=8");
|
||||
d.observation_delay_steps = 8;
|
||||
d.validate().expect("eight steps of delay are allowed");
|
||||
}
|
||||
|
||||
/// Only top-left RGBA8 exists in v1; another format is a media-schema change.
|
||||
#[test]
|
||||
fn only_rgba8_is_a_readable_view_format() {
|
||||
let descriptor = view_descriptor(32, 24, 0);
|
||||
let mut json = descriptor.to_json();
|
||||
json["format"] = serde_json::json!("rgb8");
|
||||
ViewDescriptor::from_json(&json).expect_err("rgb8 is not a v1 format");
|
||||
json["format"] = serde_json::json!("rgba8");
|
||||
ViewDescriptor::from_json(&json).expect("rgba8 is the v1 format");
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------------------------
|
||||
// Bad lengths
|
||||
|
||||
/// A frame's artifact is exactly `rowStride x height` bytes.
|
||||
#[test]
|
||||
fn a_frame_whose_length_is_not_stride_times_height_is_refused() {
|
||||
let descriptor = view_descriptor(32, 24, 0);
|
||||
let exact = descriptor.frame_bytes();
|
||||
assert_eq!(exact, 32 * 4 * 24);
|
||||
|
||||
view_ref(&descriptor, 7, exact)
|
||||
.validate_against(&descriptor, None)
|
||||
.expect("the exact frame length is accepted");
|
||||
for wrong in [exact - 1, exact + 1, exact - 32 * 4, exact * 2] {
|
||||
view_ref(&descriptor, 7, wrong)
|
||||
.validate_against(&descriptor, None)
|
||||
.expect_err("only rowStride x height is the frame length");
|
||||
}
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------------------------
|
||||
// Bad producing times
|
||||
|
||||
/// A required sensory view is produced at exactly `max(0, boundary - observationDelaySteps)`.
|
||||
#[test]
|
||||
fn a_view_produced_at_the_wrong_boundary_is_refused() {
|
||||
let descriptor = view_descriptor(32, 24, 2);
|
||||
let bytes = descriptor.frame_bytes();
|
||||
assert_eq!(descriptor.required_produced_step(10), 8);
|
||||
|
||||
view_ref(&descriptor, 8, bytes)
|
||||
.validate_against(&descriptor, Some(10))
|
||||
.expect("the declared delay is exactly two steps");
|
||||
// One frame later than the delay allows, and one frame older: both are step failures,
|
||||
// not an arbitrary latest frame.
|
||||
view_ref(&descriptor, 9, bytes)
|
||||
.validate_against(&descriptor, Some(10))
|
||||
.expect_err("an under-delayed frame is refused");
|
||||
view_ref(&descriptor, 7, bytes)
|
||||
.validate_against(&descriptor, Some(10))
|
||||
.expect_err("an extra-delayed frame is refused");
|
||||
}
|
||||
|
||||
/// Bootstrap may repeat `O[0]` until the declared pipeline delay fills, and only until then.
|
||||
#[test]
|
||||
fn bootstrap_repeats_the_first_frame_until_the_pipeline_delay_fills() {
|
||||
let descriptor = view_descriptor(32, 24, 3);
|
||||
let bytes = descriptor.frame_bytes();
|
||||
// Boundaries 0..=3 all require the frame produced at 0.
|
||||
for boundary in 0..=3 {
|
||||
assert_eq!(descriptor.required_produced_step(boundary), 0);
|
||||
view_ref(&descriptor, 0, bytes)
|
||||
.validate_against(&descriptor, Some(boundary))
|
||||
.expect("O[0] repeats while the pipeline fills");
|
||||
}
|
||||
// From boundary 4 the pipeline is full and O[0] is a stale frame.
|
||||
assert_eq!(descriptor.required_produced_step(4), 1);
|
||||
view_ref(&descriptor, 0, bytes)
|
||||
.validate_against(&descriptor, Some(4))
|
||||
.expect_err("the repetition ends when the delay is filled");
|
||||
view_ref(&descriptor, 1, bytes)
|
||||
.validate_against(&descriptor, Some(4))
|
||||
.expect("boundary 4 requires the frame produced at 1");
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------------------------
|
||||
// Audio shapes
|
||||
|
||||
/// sampleRate is 8000..=192000, channels 1..=8 and sampleFrames 0..=192000.
|
||||
#[test]
|
||||
fn audio_rate_channels_and_frames_have_stated_bounds() {
|
||||
audio_descriptor(8_000, 1).validate().expect("8 kHz mono is the floor");
|
||||
audio_descriptor(192_000, 8).validate().expect("192 kHz 8ch is the ceiling");
|
||||
audio_descriptor(7_999, 2).validate().expect_err("under 8 kHz is refused");
|
||||
audio_descriptor(192_001, 2).validate().expect_err("over 192 kHz is refused");
|
||||
audio_descriptor(48_000, 0).validate().expect_err("zero channels are refused");
|
||||
audio_descriptor(48_000, 9).validate().expect_err("nine channels are refused");
|
||||
|
||||
let descriptor = audio_descriptor(48_000, 2);
|
||||
let mut chunk = audio_ref(&descriptor, 0, 192_000, false);
|
||||
chunk.validate().expect("192000 frames is the per-chunk ceiling");
|
||||
chunk.sample_frames = 192_001;
|
||||
chunk.validate().expect_err("over 192000 frames in one chunk is refused");
|
||||
}
|
||||
|
||||
/// A chunk's artifact is exactly `sampleFrames x channels x 4` bytes.
|
||||
#[test]
|
||||
fn an_audio_chunk_length_is_frames_times_channels_times_four() {
|
||||
let descriptor = audio_descriptor(48_000, 2);
|
||||
let chunk = audio_ref(&descriptor, 0, 800, false);
|
||||
assert_eq!(chunk.samples.byte_length, 800 * 2 * 4);
|
||||
chunk
|
||||
.validate_against(&descriptor)
|
||||
.expect("the exact chunk length is accepted");
|
||||
|
||||
let mut wrong = chunk.clone();
|
||||
wrong.samples = artifact(800 * 2 * 4 - 4, "audio/x-f32le");
|
||||
wrong
|
||||
.validate_against(&descriptor)
|
||||
.expect_err("a short chunk is refused");
|
||||
|
||||
// The same frames at another channel count are a different number of bytes.
|
||||
let mono = audio_descriptor(48_000, 1);
|
||||
let mut wrong_channels = chunk.clone();
|
||||
wrong_channels.stream_id = mono.stream_id.clone();
|
||||
wrong_channels
|
||||
.validate_against(&mono)
|
||||
.expect_err("stereo bytes are not a mono chunk");
|
||||
}
|
||||
|
||||
/// Samples are finite f32.
|
||||
#[test]
|
||||
fn a_non_finite_sample_is_refused() {
|
||||
let mut bytes = Vec::new();
|
||||
for value in [0.0f32, -0.5, 0.75, 1.0] {
|
||||
bytes.extend_from_slice(&value.to_le_bytes());
|
||||
}
|
||||
require_finite_samples(&bytes).expect("finite samples are accepted");
|
||||
|
||||
for bad in [f32::NAN, f32::INFINITY, f32::NEG_INFINITY] {
|
||||
let mut broken = bytes.clone();
|
||||
broken.extend_from_slice(&bad.to_le_bytes());
|
||||
require_finite_samples(&broken).expect_err("a non-finite sample is refused");
|
||||
}
|
||||
require_finite_samples(&bytes[..5]).expect_err("a partial sample is refused");
|
||||
}
|
||||
|
||||
/// Within an epoch, chunks cannot overlap or go backwards.
|
||||
#[test]
|
||||
fn chunks_cannot_overlap_or_go_backwards_within_an_epoch() {
|
||||
let descriptor = audio_descriptor(48_000, 2);
|
||||
let mut timeline = AudioTimeline::fresh(&descriptor, 0);
|
||||
timeline
|
||||
.accept(&audio_ref(&descriptor, 0, 800, false), &descriptor)
|
||||
.expect("the first chunk starts at the origin");
|
||||
assert_eq!(timeline.next_sample(), 800);
|
||||
timeline
|
||||
.accept(&audio_ref(&descriptor, 800, 800, false), &descriptor)
|
||||
.expect("the second chunk continues the first");
|
||||
assert_eq!(timeline.next_sample(), 1_600);
|
||||
|
||||
let mut overlapping = timeline.clone();
|
||||
overlapping
|
||||
.accept(&audio_ref(&descriptor, 1_599, 800, false), &descriptor)
|
||||
.expect_err("a chunk that starts inside the previous one is refused");
|
||||
let mut backwards = timeline.clone();
|
||||
backwards
|
||||
.accept(&audio_ref(&descriptor, 0, 800, false), &descriptor)
|
||||
.expect_err("a chunk that goes backwards is refused");
|
||||
// A rejected chunk leaves the timeline where it was.
|
||||
assert_eq!(overlapping.next_sample(), 1_600);
|
||||
assert_eq!(overlapping.accepted(), 2);
|
||||
|
||||
// A gap is forward, so it is allowed; it is the one place a later chunk may mark a
|
||||
// discontinuity.
|
||||
timeline
|
||||
.accept(&audio_ref(&descriptor, 2_000, 800, true), &descriptor)
|
||||
.expect("a forward gap is not an overlap");
|
||||
timeline
|
||||
.accept(&audio_ref(&descriptor, 2_800, 800, true), &descriptor)
|
||||
.expect_err("a chunk that continues the previous one is not a discontinuity");
|
||||
}
|
||||
|
||||
/// Crash restore preserves the sample position under a new epoch, and its first chunk marks
|
||||
/// the discontinuity.
|
||||
#[test]
|
||||
fn the_first_chunk_after_a_restore_marks_discontinuity() {
|
||||
let descriptor = audio_descriptor(48_000, 2);
|
||||
// The requirement is one-directional. A fresh epoch's first chunk may mark a
|
||||
// discontinuity -- a recovery or an episode reset establishes a fresh timeline and
|
||||
// publishes one -- so both flags are accepted at an origin.
|
||||
let mut reset = AudioTimeline::fresh(&descriptor, 0);
|
||||
reset
|
||||
.accept(&audio_ref(&descriptor, 0, 800, true), &descriptor)
|
||||
.expect("a reset episode's first chunk may mark the discontinuity it published");
|
||||
let mut fresh = AudioTimeline::fresh(&descriptor, 0);
|
||||
fresh
|
||||
.accept(&audio_ref(&descriptor, 0, 800, false), &descriptor)
|
||||
.expect("the episode's first chunk continues nothing");
|
||||
|
||||
// The restored epoch resumes at the preserved position.
|
||||
let mut restored = AudioTimeline::restored_at(&descriptor, 800);
|
||||
restored
|
||||
.accept(&audio_ref(&descriptor, 800, 800, false), &descriptor)
|
||||
.expect_err("the first chunk after a restore marks discontinuity");
|
||||
let mut restored = AudioTimeline::restored_at(&descriptor, 800);
|
||||
restored
|
||||
.accept(&audio_ref(&descriptor, 0, 800, true), &descriptor)
|
||||
.expect_err("the restored position is preserved, not reset");
|
||||
restored
|
||||
.accept(&audio_ref(&descriptor, 800, 800, true), &descriptor)
|
||||
.expect("the restored epoch resumes at its preserved sample position");
|
||||
assert_eq!(restored.next_sample(), 1_600);
|
||||
restored
|
||||
.accept(&audio_ref(&descriptor, 1_600, 800, false), &descriptor)
|
||||
.expect("the chunks after it are ordinary");
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------------------------
|
||||
// Persistent assets against transient artifacts
|
||||
|
||||
/// An `AssetRef` and an `ArtifactRef` are different identities with different fields, and
|
||||
/// neither is readable as the other.
|
||||
#[test]
|
||||
fn an_asset_ref_is_not_a_transient_artifact_ref() {
|
||||
let asset = AssetRef {
|
||||
id: "counter-arena-backend".into(),
|
||||
digest: "a".repeat(64),
|
||||
byte_length: 24,
|
||||
format: "fly-config-v1".into(),
|
||||
};
|
||||
let imported = ArtifactRef {
|
||||
store_id: "store-a".into(),
|
||||
artifact_id: "art-9".into(),
|
||||
generation: 1,
|
||||
byte_length: 24,
|
||||
content_type: "application/octet-stream".into(),
|
||||
digest: Some("a".repeat(64)),
|
||||
};
|
||||
|
||||
// Identity fields do not overlap: the asset has no store and the artifact has no format.
|
||||
let asset_keys: Vec<String> = asset
|
||||
.to_json()
|
||||
.as_object()
|
||||
.expect("an object")
|
||||
.keys()
|
||||
.cloned()
|
||||
.collect();
|
||||
let artifact_keys: Vec<String> = imported
|
||||
.to_json()
|
||||
.as_object()
|
||||
.expect("an object")
|
||||
.keys()
|
||||
.cloned()
|
||||
.collect();
|
||||
assert_eq!(asset_keys, vec!["id", "digest", "byteLength", "format"]);
|
||||
assert!(artifact_keys.contains(&"storeId".to_owned()));
|
||||
assert!(artifact_keys.contains(&"artifactId".to_owned()));
|
||||
assert!(!artifact_keys.contains(&"format".to_owned()));
|
||||
assert!(!asset_keys.contains(&"storeId".to_owned()));
|
||||
|
||||
// Neither reads as the other: a view's pixels are an artifact, never an asset.
|
||||
ArtifactRef::from_json(&asset.to_json()).expect_err("an asset is not an artifact reference");
|
||||
AssetRef::from_json(&imported.to_json()).expect_err("an artifact is not an asset reference");
|
||||
let pixels_as_asset = serde_json::json!({
|
||||
"viewId": "arena",
|
||||
"producedStep": "0",
|
||||
"pixels": asset.to_json(),
|
||||
});
|
||||
ViewRef::from_json(&pixels_as_asset).expect_err("a view's pixels cannot be an asset");
|
||||
|
||||
// Importing an asset is a content check, not an identity conversion.
|
||||
check_imported_asset(&asset, &imported).expect("the import carries the asset's content");
|
||||
let mut no_digest = imported.clone();
|
||||
no_digest.digest = None;
|
||||
check_imported_asset(&asset, &no_digest)
|
||||
.expect_err("a persistent asset import must carry a content digest");
|
||||
let mut other_content = imported.clone();
|
||||
other_content.digest = Some("b".repeat(64));
|
||||
check_imported_asset(&asset, &other_content).expect_err("another digest is another content");
|
||||
let mut short = imported.clone();
|
||||
short.byte_length = 23;
|
||||
check_imported_asset(&asset, &short).expect_err("the import must be the asset's length");
|
||||
}
|
||||
|
|
@ -11,6 +11,13 @@ description = "The lockstep session coordinator, its phase machine and a synthet
|
|||
name = "fly_session"
|
||||
path = "src/lib.rs"
|
||||
|
||||
# One binary, one role per subcommand. `implementation.md` section 2 allows worker
|
||||
# executables to be subcommands of one binary rather than separate crates, and the launcher
|
||||
# starts this one with `agent` or `environment` for a participant in its own process.
|
||||
[[bin]]
|
||||
name = "fly-session"
|
||||
path = "src/bin/fly-session.rs"
|
||||
|
||||
[dependencies]
|
||||
# The domain contract (scalars, payloads, canonical digests, the trace format) and the bus.
|
||||
# Everything else this crate needs is std or Tokio.
|
||||
|
|
@ -18,7 +25,9 @@ fly-session-types = { path = "../fly-session-types" }
|
|||
flybus = { path = "../flybus" }
|
||||
|
||||
serde_json = { workspace = true }
|
||||
tokio = { version = "1", features = ["rt", "sync", "time", "macros"] }
|
||||
# `rt-multi-thread` is not only for the tests: a worker process and a dedicated-thread
|
||||
# worker each build their own runtime sized to the launcher's thread allocation.
|
||||
tokio = { version = "1", features = ["rt", "rt-multi-thread", "sync", "time", "macros", "io-util"] }
|
||||
|
||||
[dev-dependencies]
|
||||
tempfile = "3"
|
||||
|
|
|
|||
|
|
@ -3,10 +3,12 @@
|
|||
The lockstep session coordinator, its phase machine and a synthetic composition over
|
||||
[`flybus`](../flybus).
|
||||
|
||||
This crate is the SESSION-01 slice of the session-framework implementation guide: the
|
||||
sequential transaction of `step-v1`, driven over the Flybus router, with small fake workers
|
||||
standing in for a brain and an emulator. It contains no public controller API, no implicit
|
||||
best-effort retry, no real emulator and no real brain.
|
||||
This crate is the SESSION-01 and SESSION-02 slices of the session-framework implementation
|
||||
guide: the transaction of `step-v1`, driven over the Flybus router, with small fake workers
|
||||
standing in for a brain and an emulator, run either in the coordinator's process, on dedicated
|
||||
threads, or as one agent process per fly and one environment process under a launcher. It
|
||||
contains no public controller API, no implicit best-effort retry, no real emulator and no real
|
||||
brain.
|
||||
|
||||
The domain scalars, method payloads, their validation, the canonical digests and the trace
|
||||
format all come from [`fly-session-types`](../fly-session-types), the CONTRACT-01 crate. This
|
||||
|
|
@ -38,7 +40,54 @@ Ready(k) ─ Prepare all agents concurrently ───────────
|
|||
| `task` | The task and executor traits, the deterministic counter task, the identity executor |
|
||||
| `rpc` | Domain calls: `req-<U64>` serials, incarnation pinning, the retry rule |
|
||||
| `coordinator` | The transaction, the trace, the failure rules and the publication boundary |
|
||||
| `harness` | The runnable composition: router, two agents, one arena, one coordinator |
|
||||
| `launcher` | The supervisor: thread budget, identities, start, health check, reap |
|
||||
| `metrics` | Latency percentiles and the machine's core and memory counters |
|
||||
| `measure` | The execution-mode comparison of the guide's section 5 |
|
||||
| `cli` | The binary's subcommands: `agent`, `environment`, `measure` |
|
||||
| `harness` | The runnable composition: router, the flies, one arena, one coordinator |
|
||||
|
||||
## Execution modes and the launcher
|
||||
|
||||
A participant runs in one of three places, and the same composition code starts it in any of
|
||||
them. The separate-process mode is the SESSION-02 subject; the other two are what it is
|
||||
compared against.
|
||||
|
||||
| Mode | Where each participant runs | Transport |
|
||||
| --- | --- | --- |
|
||||
| `InProcess` | A task on the coordinator's runtime | in-memory or Unix socket |
|
||||
| `Thread` | Its own OS thread, with its own runtime | Unix socket |
|
||||
| `Process` | Its own process: one per fly, one for the world | Unix socket |
|
||||
|
||||
The launcher is the configured supervisor. It owns four things:
|
||||
|
||||
- **The thread budget.** A total allocation, one slice of it reserved for the coordinator and
|
||||
its router, and one allocation per participant. A request the total cannot cover is refused
|
||||
as `BUSY` before anything starts. `Agent.Initialize` carries exactly the allocation the
|
||||
launcher handed out, and an agent refuses an Initialize asking for more than its own, which
|
||||
is what `workers-v1` means by "within launcher allocation". The allocation is on the wire,
|
||||
not only in the launcher's own record: `HelloResult.limits.workerThreads` reports it, under
|
||||
the dated 2026-09-22 amendment to `workers-v1` section 2 that this slice added, so a
|
||||
coordinator that is not also its own launcher can read the bound it has to respect.
|
||||
- **Identity.** The bus client id, the service name, the worker id and an agent's port binding
|
||||
are launcher configuration. The launcher says `Worker.Hello` with the identity it configured
|
||||
and refuses anything that answers as another worker, role, incarnation or thread allocation
|
||||
-- before the coordinator has pinned a registration. The registration the coordinator pins
|
||||
is the one that hello returned, never one that was assumed.
|
||||
- **Health.** `Worker.Status` on the supervisor's own monotonic clock, with the `ipc-v1`
|
||||
section 6 prototype budgets: probe at two seconds, fail at ten, a separate budget for boot.
|
||||
A status answer never waits for a mutation, so a busy participant is still a healthy one.
|
||||
- **Reaping.** `Worker.Shutdown` is the request and the operating system is the guarantee. A
|
||||
participant that does not stop inside the budget is terminated, and the supervisor reports
|
||||
which of the two happened. A launcher that is dropped takes its children with it.
|
||||
|
||||
A separate-process participant is a subcommand of this crate's one binary, which is what
|
||||
`implementation.md` section 2 allows instead of separate worker crates:
|
||||
|
||||
```sh
|
||||
fly-session agent --socket S --store-root D --client-id C --service N --threads T ...
|
||||
fly-session environment --socket S --store-root D --client-id C --service N --threads T ...
|
||||
fly-session measure --steps 300 --agents 1,2,4
|
||||
```
|
||||
|
||||
## What it implements
|
||||
|
||||
|
|
@ -62,6 +111,31 @@ Ready(k) ─ Prepare all agents concurrently ───────────
|
|||
- **The failure rules.** A partial commit fails the epoch; an uncertain Advance is resolved
|
||||
against its original domain request id and never becomes a second batch; a worker
|
||||
incarnation change invalidates the epoch.
|
||||
- **A failure stops the epoch rather than neutralising a player.** Every failure carries the
|
||||
participant it is attributed to, and failing fences the session: the committed boundary
|
||||
stops moving, the artifact handles are dropped, and no further transition or publication is
|
||||
allowed. Lifting the fence is a coherent group restore, which is STATE-01's.
|
||||
- **The `ipc-v1` section 6 procedure, on the path that reaches it.** A call that goes two
|
||||
seconds without a terminal reply is *uncertain*, not failed. The coordinator then queries
|
||||
the same operation -- a fresh bus call carrying the original domain request id and body,
|
||||
pinned to the same incarnation, with its retained attachments -- absorbing `IN_PROGRESS`
|
||||
while the original is still running. Only when that ends without a definite answer, or the
|
||||
incarnation is gone, or the retained result expired, is the epoch failed. A merely slow
|
||||
participant therefore finishes its step, and `step-v1` section 7's "query/retransmit same
|
||||
request to same incarnation; never new batch" is the same code path for a slow Advance.
|
||||
|
||||
The procedure has two explicit bounds, and they do not mean the same thing. **`resolve`, 8
|
||||
seconds, is the working limit**: two to notice plus eight to resolve is section 6's ten
|
||||
seconds without progress. **`resolve_attempts`, 8192, is a guard**, not the limit -- the
|
||||
procedure pauses 2 ms between attempts, so the guard is over sixteen seconds of pauses
|
||||
alone, twice the budget, and an attempt whose call expires costs a whole probe on top. At
|
||||
these values the budget is always what fires. Which one did is recorded in
|
||||
`Coordinator::last_resolution` and named in the failure's own message, so an exhausted
|
||||
resolution never has to be explained by arithmetic.
|
||||
- **A bounded diagnosed outcome.** Those budgets are the coordinator's own, on its own clock,
|
||||
so a participant that dies or stops answering produces a typed failure naming it rather than
|
||||
a hang. An expired deadline is `unknown`, never `none`: a caller-side timeout is not
|
||||
evidence that nothing was mutated.
|
||||
- **Domain deduplication over bus calls.** Same key, request and body replays its cached
|
||||
reply with fresh delivery ownership over retained artifacts; a changed body is `CONFLICT`; a
|
||||
duplicate of a running operation is `IN_PROGRESS` for that bus call while the original
|
||||
|
|
@ -125,23 +199,66 @@ harness.shutdown().await;
|
|||
- **No state methods.** `State.Capture`, `State.StageRestore` and `State.ActivateRestore` are
|
||||
STATE-01. The phase machine has their edges (`Capturing`, `Restoring`) and the workers do not
|
||||
advertise them as implemented methods.
|
||||
- **One process.** SESSION-01 runs every participant in one process over the same router.
|
||||
SESSION-02 is the per-fly process split.
|
||||
- **No audience input.** The admitted pre-step stimulation list exists and is always empty.
|
||||
- **Pacing is coarse.** The pacing deadline rounds one step to whole nanoseconds for sleeping
|
||||
only; simulation time stays rational and that rounding never re-enters the accumulator.
|
||||
|
||||
## Measurements
|
||||
|
||||
`fly-session measure` runs the same composition in each mode at one, two and four agents and
|
||||
reports the thread allocation, the RPC and critical-path percentiles, the memory peaks and the
|
||||
router's owner, collection and queue counters. **These are local synthetic timings on one
|
||||
machine and no host capacity claim follows from any of them**; they exist so the three modes
|
||||
can be compared with each other. Pacing is off for the run, so the samples are work rather
|
||||
than sleep, and the run report carries the full table.
|
||||
|
||||
Every row runs in a child process of its own. A peak-memory figure is a high-water mark that
|
||||
never falls, so rows sharing one process would each report where that process had already
|
||||
been: the column would sort itself by row position rather than by mode, and the mode ranking
|
||||
would reverse when the rows were reordered. One child per row is what makes the number belong
|
||||
to the row.
|
||||
|
||||
What the numbers said on a four-core development box, at 300 transitions per row:
|
||||
|
||||
- A process boundary costs about a fifth of the critical path at the median. Two agents: 10.0
|
||||
ms p50 in-process, 12.2 ms on threads, 12.1 ms across processes, with p99 at 21.4 / 19.4 /
|
||||
21.4 ms. The dedicated-thread and separate-process variants are within noise of each other,
|
||||
so what is being paid for is leaving the coordinator's runtime, not crossing a socket.
|
||||
- `Worker.Status` -- an RPC answered from a cell with no domain work behind it -- is the
|
||||
router and transport floor: 0.60 / 0.89 / 1.17 ms p50 for the three modes at two agents.
|
||||
- Four agents needs six threads, which that box does not have, and every mode's tail widens
|
||||
together. That is the budget being honest about oversubscription, not a property of the
|
||||
process split.
|
||||
- Memory is where the split really shows, but not where the first version of this note said.
|
||||
The coordinator's own peak is roughly the same in all three modes and is *lowest* in process
|
||||
mode -- 8.9 / 9.2 / 7.9 MiB at one agent -- because the workers are no longer inside it.
|
||||
What the split costs is the children: about 5.7 MiB per participant process, so the whole
|
||||
composition is roughly 9 MiB on threads against 39 MiB across processes at four agents.
|
||||
- Ownership, collection and queues stayed bounded in every mode and at every agent count: at
|
||||
most 15 live owners, 11 artifact roots and one queued entry per agent, with the store at
|
||||
rest holding two sealed frames and 128 bytes. Of 311 frames observed, 309 were collected --
|
||||
the two still owned are the current and previous boundary. The frame count is taken from the
|
||||
behaviour trace's observation boundaries rather than calculated from the step count, so a
|
||||
backend sealing two frames per boundary would show up instead of being hidden.
|
||||
|
||||
## Tests
|
||||
|
||||
```text
|
||||
cargo test -p fly-session # unit + both integration suites
|
||||
cargo test -p fly-session # unit + all three integration suites
|
||||
cargo run -p fly-session --example session # the runnable synthetic session
|
||||
cargo build -p fly-session --bin fly-session # the worker binary the launcher starts
|
||||
cargo run -p fly-session --example processes # the same session in all three modes
|
||||
```
|
||||
|
||||
Every integration test runs over both transports, through the same router code: all but one
|
||||
are generated twice by `both_transports!`, and
|
||||
`sequential_concurrent_and_reversed_orders_agree` walks both transports inside one test
|
||||
because it compares their behaviour traces against each other.
|
||||
The three integration suites do not all run over both transports, and cannot:
|
||||
|
||||
- `tests/session.rs` and `tests/failures.rs` are in-process compositions and run over both,
|
||||
through the same router code. All but one test in them is generated twice by
|
||||
`both_transports!`; `sequential_concurrent_and_reversed_orders_agree` walks both transports
|
||||
inside one test, because it compares their behaviour traces against each other.
|
||||
- `tests/processes.rs` runs over the Unix socket only, in all three execution modes. A
|
||||
participant in a process of its own has no in-memory transport to reach the router by, so
|
||||
the mode is the axis that suite varies and the transport is fixed.
|
||||
|
||||
- `tests/session.rs`: one world advance per complete batch; every agent Prepared before the
|
||||
advance; one task evaluation per transition; every agent committed before the next Prepare or
|
||||
|
|
@ -150,6 +267,14 @@ because it compares their behaviour traces against each other.
|
|||
transition that just ended; a terminal episode pausing at its own boundary; `Worker.Status`
|
||||
during a session; and sequential, concurrent and reversed dispatch producing one behaviour
|
||||
trace.
|
||||
- `tests/processes.rs`: the SESSION-02 acceptance bullets, each generated once per execution
|
||||
mode -- a slow participant resolved rather than failed, a delayed one-agent result holding
|
||||
the world, a worker or helper death with a
|
||||
bounded diagnosed outcome, an uncertain Advance that creates no second batch, a partial
|
||||
Commit that permits no next-step play, supervision and identity, and the launcher thread
|
||||
allocation -- plus the sequential/reversed/parallel trace comparison across all three modes
|
||||
and the two process-mode section 4 rows: a router restart during a world advance, and an old
|
||||
worker's reply after a restart.
|
||||
- `tests/failures.rs`: a duplicate Prepare after a lost reply; a duplicate Commit; the same
|
||||
batch with altered controls; a lost Advance result; a cached artifact consumed by its first
|
||||
caller; one Commit failing after another succeeded; a replaced registration; a reply from
|
||||
|
|
|
|||
75
services/flysim/crates/fly-session/examples/processes.rs
Normal file
75
services/flysim/crates/fly-session/examples/processes.rs
Normal file
|
|
@ -0,0 +1,75 @@
|
|||
//! The same synthetic session in all three execution modes, printing the one behaviour trace
|
||||
//! they agree on.
|
||||
//!
|
||||
//! ```sh
|
||||
//! cargo run -p fly-session --example processes
|
||||
//! ```
|
||||
//!
|
||||
//! The separate-process run starts one agent process per fly and one environment process
|
||||
//! through this crate's own binary, so it needs that binary built:
|
||||
//!
|
||||
//! ```sh
|
||||
//! cargo build -p fly-session --bin fly-session
|
||||
//! ```
|
||||
|
||||
use fly_session::harness::{ExecutionMode, HarnessConfig, SessionHarness, Via};
|
||||
use fly_session::launcher::default_worker_program;
|
||||
|
||||
#[tokio::main(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn main() -> Result<(), Box<dyn std::error::Error>> {
|
||||
let program = default_worker_program();
|
||||
println!("worker program: {}", program.display());
|
||||
let mut agreed: Option<Vec<String>> = None;
|
||||
|
||||
for mode in ExecutionMode::all() {
|
||||
let dir = tempfile::tempdir()?;
|
||||
let config = HarnessConfig { mode, ..HarnessConfig::default() };
|
||||
println!(
|
||||
"\n=== {} : {} threads for {} participants plus the coordinator",
|
||||
mode.label(),
|
||||
config.budget()?.total(),
|
||||
config.agents.len() + 1
|
||||
);
|
||||
let mut harness = SessionHarness::start(Via::Unix, dir.path(), config).await?;
|
||||
harness.coordinator.bootstrap().await?;
|
||||
let reports = harness.coordinator.run(3).await?;
|
||||
for report in &reports {
|
||||
println!(" committed boundary {}", report.boundary);
|
||||
}
|
||||
for (worker_id, status) in harness.launcher.health_check_all().await {
|
||||
match status {
|
||||
Ok(status) => println!(
|
||||
" {worker_id}: {:?}, progress {}",
|
||||
status.state, status.progress_counter
|
||||
),
|
||||
Err(e) => println!(" {worker_id}: unhealthy: {e}"),
|
||||
}
|
||||
}
|
||||
let behaviour = harness.coordinator.trace.behavior();
|
||||
match &agreed {
|
||||
None => {
|
||||
println!(" behaviour trace, {} transitions:", behaviour.len());
|
||||
for line in &behaviour {
|
||||
println!(" {line}");
|
||||
}
|
||||
agreed = Some(behaviour);
|
||||
}
|
||||
Some(first) => {
|
||||
assert_eq!(
|
||||
&behaviour, first,
|
||||
"{} produced a different behaviour trace",
|
||||
mode.label()
|
||||
);
|
||||
println!(" behaviour trace: identical to the first run");
|
||||
}
|
||||
}
|
||||
let reaped = harness.launcher.reap_all(&fly_session::types::id("example")).await;
|
||||
for (worker_id, outcome) in reaped {
|
||||
println!(" reaped {worker_id}: {outcome:?}");
|
||||
}
|
||||
harness.shutdown().await;
|
||||
drop(dir);
|
||||
}
|
||||
println!("\nall three execution modes produced one behaviour trace");
|
||||
Ok(())
|
||||
}
|
||||
|
|
@ -203,6 +203,14 @@ pub struct AgentConfig {
|
|||
pub incarnation_id: Id,
|
||||
pub tick_duration: RationalNs,
|
||||
pub warmup_ticks: u64,
|
||||
/// The thread allocation the launcher started this worker within. `workers-v1` requires
|
||||
/// `Agent.Initialize`'s `workerThreads` to lie inside it.
|
||||
pub worker_threads: usize,
|
||||
/// Records every view this agent read, so a test can see which artifact reached it.
|
||||
///
|
||||
/// It is this process's log: an agent with a process of its own writes to its own copy,
|
||||
/// which the supervisor cannot read. `SessionHarness::sensor_log` says so with `None`.
|
||||
pub sensors: crate::media::SensorLog,
|
||||
pub faults: AgentFaults,
|
||||
}
|
||||
|
||||
|
|
@ -296,6 +304,15 @@ impl FakeAgentWorker {
|
|||
format!("view {} is the wrong length", view.view_id),
|
||||
));
|
||||
}
|
||||
// What this agent read, from the bytes it read: the artifact it was given and the
|
||||
// digest of its content.
|
||||
self.config.sensors.record(crate::media::SensedView {
|
||||
boundary: input.boundary,
|
||||
view_id: view.view_id.clone(),
|
||||
artifact_id: artifact.reference().artifact_id.clone(),
|
||||
produced_step: view.produced_step,
|
||||
digest: digest_of_bytes(&bytes),
|
||||
});
|
||||
total += i64::from(bytes.first().copied().unwrap_or_default());
|
||||
}
|
||||
if let Some(structured) = &input.structured {
|
||||
|
|
@ -369,6 +386,18 @@ impl FakeAgentWorker {
|
|||
if params.worker_threads == 0 {
|
||||
return Err(DomainError::invalid("workerThreads must be >= 1"));
|
||||
}
|
||||
// `workers-v1`: workerThreads is "within launcher allocation". This worker was started
|
||||
// with that allocation, so a request for more than it is a capacity refusal made
|
||||
// before the model is constructed, not a silent reduction to what is available.
|
||||
if params.worker_threads > self.config.worker_threads as u64 {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::Busy,
|
||||
format!(
|
||||
"Agent.Initialize asks for {} worker threads; the launcher allocated {}",
|
||||
params.worker_threads, self.config.worker_threads
|
||||
),
|
||||
));
|
||||
}
|
||||
params.initial_decision_context.validate().map_err(DomainError::invalid)?;
|
||||
let available = FakeAgentWorker::available_actions(¶ms.initial_decision_context)?;
|
||||
// Everything is validated before the model is constructed.
|
||||
|
|
@ -635,6 +664,10 @@ impl WorkerEndpoint for FakeAgentWorker {
|
|||
self.status.clone()
|
||||
}
|
||||
|
||||
fn worker_threads(&self) -> u64 {
|
||||
self.config.worker_threads as u64
|
||||
}
|
||||
|
||||
fn methods(&self) -> Vec<&'static str> {
|
||||
vec!["Agent.Initialize", "Agent.Prepare", "Agent.Commit"]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,8 @@
|
|||
//! This crate's one binary. Every role it can take is a subcommand of it.
|
||||
//!
|
||||
//! `implementation.md` section 2: "Worker executables can be subcommands of one binary
|
||||
//! initially; process boundaries do not require separate repos."
|
||||
|
||||
fn main() -> std::process::ExitCode {
|
||||
fly_session::cli::main()
|
||||
}
|
||||
337
services/flysim/crates/fly-session/src/cli.rs
Normal file
337
services/flysim/crates/fly-session/src/cli.rs
Normal file
|
|
@ -0,0 +1,337 @@
|
|||
//! The subcommands of this crate's one binary.
|
||||
//!
|
||||
//! `implementation.md` section 2 allows worker executables to be subcommands of one binary
|
||||
//! rather than separate crates, and that is what these are: `agent` and `environment` are the
|
||||
//! two worker roles a launcher starts as separate processes, and `measure` runs the execution
|
||||
//! modes against each other.
|
||||
//!
|
||||
//! ```text
|
||||
//! fly-session agent --socket S --store-root D --client-id C --service N --threads T ...
|
||||
//! fly-session environment --socket S --store-root D --client-id C --service N --threads T ...
|
||||
//! fly-session measure [--steps N] [--agents 1,2,4] [--modes in-process,thread,process]
|
||||
//! ```
|
||||
//!
|
||||
//! A worker process is told exactly which participant it is. It proves that identity in
|
||||
//! `Worker.Hello`, so a process started under another one is refused by its own supervisor
|
||||
//! before the coordinator has pinned anything.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::path::PathBuf;
|
||||
use std::process::ExitCode;
|
||||
|
||||
use crate::agent::AgentFaults;
|
||||
use crate::environment::EnvironmentFaults;
|
||||
use crate::launcher::{
|
||||
AgentLaunch, EnvironmentLaunch, ExecutionMode, Started, flags, serve_one,
|
||||
};
|
||||
use crate::types::*;
|
||||
|
||||
const USAGE: &str = "\
|
||||
fly-session <command> [options]
|
||||
|
||||
agent serve one agent worker on a launcher-created endpoint
|
||||
environment serve the environment worker on a launcher-created endpoint
|
||||
measure compare the execution modes and print the measurement table
|
||||
measure-row measure one row and print it as JSON (one child per row)
|
||||
|
||||
Worker options (agent and environment):
|
||||
--socket PATH the launcher's endpoint for this participant
|
||||
--store-root PATH the router's artifact store root
|
||||
--client-id ID the configured bus client identity
|
||||
--service NAME the one service name this worker registers
|
||||
--threads N the launcher's thread allocation for this worker
|
||||
--session ID the session this worker belongs to
|
||||
--incarnation ID this worker's domain incarnation
|
||||
|
||||
agent: --agent ID --port ID --tick-numerator N --tick-denominator N
|
||||
--warmup-ticks N [--prepare-delay-ms N] [--commit-delay-ms N]
|
||||
[--fail-commit-at-step N]
|
||||
environment: --worker ID --ports p1,p2 --step-numerator N --step-denominator N
|
||||
[--advance-delay-ms N] [--omit-view-at-boundary N]
|
||||
|
||||
Measure options:
|
||||
--steps N transitions per run (default 200)
|
||||
--warmup-steps N transitions run before sampling starts (default 10)
|
||||
--agents 1,2,4 agent counts to compare (default 1,2,4)
|
||||
--modes LIST in-process, thread, process (default all three)
|
||||
--worker-threads N within-agent worker threads (default 1)
|
||||
|
||||
measure-row options: --mode NAME --agents N, plus the measure options above. Each row runs in
|
||||
a process of its own, so its memory peak is its own rather than the peak of the rows before
|
||||
it.
|
||||
";
|
||||
|
||||
/// The binary's entry point.
|
||||
pub fn main() -> ExitCode {
|
||||
let mut args = std::env::args_os().skip(1);
|
||||
let Some(command) = args.next() else {
|
||||
eprint!("{USAGE}");
|
||||
return ExitCode::from(2);
|
||||
};
|
||||
let command = command.to_string_lossy().into_owned();
|
||||
let rest: Vec<String> = args.map(|a| a.to_string_lossy().into_owned()).collect();
|
||||
let result = match command.as_str() {
|
||||
"agent" => Options::parse(&rest, &[flags::COMMON, flags::AGENT_ONLY])
|
||||
.and_then(|o| serve(&command, &o)),
|
||||
"environment" => Options::parse(&rest, &[flags::COMMON, flags::ENVIRONMENT_ONLY])
|
||||
.and_then(|o| serve(&command, &o)),
|
||||
"measure" => Options::parse(&rest, &[flags::MEASURE]).and_then(|o| measure(&o)),
|
||||
"measure-row" => Options::parse(&rest, &[flags::MEASURE]).and_then(|o| measure_row(&o)),
|
||||
"--help" | "-h" | "help" => {
|
||||
print!("{USAGE}");
|
||||
return ExitCode::SUCCESS;
|
||||
}
|
||||
other => Err(format!("unknown command {other:?}\n\n{USAGE}")),
|
||||
};
|
||||
match result {
|
||||
Ok(()) => ExitCode::SUCCESS,
|
||||
Err(e) => {
|
||||
eprintln!("fly-session {command}: {e}");
|
||||
ExitCode::FAILURE
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// `--flag value` options. The launcher builds this argv, so the grammar stays small.
|
||||
#[derive(Debug, Default)]
|
||||
struct Options(BTreeMap<String, String>);
|
||||
|
||||
impl Options {
|
||||
/// Reads the options of one command, refusing any flag that command does not have.
|
||||
///
|
||||
/// `allowed` comes from [`crate::launcher::flags`], the same constants the launcher
|
||||
/// writes the argv from. An unknown flag is an error naming it rather than a value that
|
||||
/// is quietly ignored: a renamed option must fail the launch, not turn into a no-op that
|
||||
/// no test notices.
|
||||
fn parse(args: &[String], allowed: &[&[&str]]) -> Result<Options, String> {
|
||||
let mut out = BTreeMap::new();
|
||||
let mut iter = args.iter();
|
||||
while let Some(flag) = iter.next() {
|
||||
let Some(name) = flag.strip_prefix("--") else {
|
||||
return Err(format!("expected an option, found {flag:?}"));
|
||||
};
|
||||
if !allowed.iter().any(|set| set.contains(&name)) {
|
||||
return Err(format!("unknown option --{name} for this command"));
|
||||
}
|
||||
let value = iter
|
||||
.next()
|
||||
.ok_or_else(|| format!("option --{name} needs a value"))?;
|
||||
if out.insert(name.to_owned(), value.clone()).is_some() {
|
||||
return Err(format!("option --{name} was given twice"));
|
||||
}
|
||||
}
|
||||
Ok(Options(out))
|
||||
}
|
||||
|
||||
fn required(&self, name: &str) -> Result<&str, String> {
|
||||
self.0
|
||||
.get(name)
|
||||
.map(String::as_str)
|
||||
.ok_or_else(|| format!("option --{name} is required"))
|
||||
}
|
||||
|
||||
fn optional(&self, name: &str) -> Option<&str> {
|
||||
self.0.get(name).map(String::as_str)
|
||||
}
|
||||
|
||||
fn id(&self, name: &str) -> Result<Id, String> {
|
||||
parse_id(self.required(name)?).map_err(|e| format!("--{name}: {e}"))
|
||||
}
|
||||
|
||||
fn u64(&self, name: &str, default: u64) -> Result<u64, String> {
|
||||
match self.0.get(name) {
|
||||
None => Ok(default),
|
||||
Some(value) => value.parse().map_err(|_| format!("--{name}: {value:?} is not a number")),
|
||||
}
|
||||
}
|
||||
|
||||
fn opt_u64(&self, name: &str) -> Result<Option<u64>, String> {
|
||||
match self.0.get(name) {
|
||||
None => Ok(None),
|
||||
Some(value) => value
|
||||
.parse()
|
||||
.map(Some)
|
||||
.map_err(|_| format!("--{name}: {value:?} is not a number")),
|
||||
}
|
||||
}
|
||||
|
||||
fn usize(&self, name: &str, default: usize) -> Result<usize, String> {
|
||||
Ok(self.u64(name, default as u64)? as usize)
|
||||
}
|
||||
|
||||
fn path(&self, name: &str) -> Result<PathBuf, String> {
|
||||
Ok(PathBuf::from(self.required(name)?))
|
||||
}
|
||||
|
||||
fn rational(&self, numerator: &str, denominator: &str) -> Result<RationalNs, String> {
|
||||
let n = self.u64(numerator, 0)?;
|
||||
let d = self.u64(denominator, 1)?;
|
||||
RationalNs::new(n, d).map_err(|e| format!("--{numerator}/--{denominator}: {}", e.0))
|
||||
}
|
||||
}
|
||||
|
||||
/// Serves one worker until `Worker.Shutdown`, then exits.
|
||||
fn serve(role: &str, options: &Options) -> Result<(), String> {
|
||||
let socket = options.path(flags::SOCKET)?;
|
||||
let store_root = options.path(flags::STORE_ROOT)?;
|
||||
let client_id = options.required(flags::CLIENT_ID)?.to_owned();
|
||||
let service = options.required(flags::SERVICE)?.to_owned();
|
||||
let threads = options.usize(flags::THREADS, 1)?;
|
||||
if threads == 0 {
|
||||
return Err("--threads must be at least 1".to_owned());
|
||||
}
|
||||
let session_id = options.id(flags::SESSION)?;
|
||||
let incarnation_id = options.id(flags::INCARNATION)?;
|
||||
let what = match role {
|
||||
"agent" => Started::Agent(AgentLaunch {
|
||||
session_id,
|
||||
agent_id: options.id(flags::AGENT)?,
|
||||
port_id: options.id(flags::PORT)?,
|
||||
incarnation_id,
|
||||
tick_duration: options.rational(flags::TICK_NUMERATOR, flags::TICK_DENOMINATOR)?,
|
||||
warmup_ticks: options.u64(flags::WARMUP_TICKS, 0)?,
|
||||
worker_threads: threads,
|
||||
// This process's own log. The supervisor reads what crosses the bus, not this.
|
||||
sensors: crate::media::SensorLog::new(),
|
||||
faults: AgentFaults {
|
||||
fail_commit_at_step: options.opt_u64(flags::FAIL_COMMIT_AT_STEP)?,
|
||||
prepare_delay_ms: options.u64(flags::PREPARE_DELAY_MS, 0)?,
|
||||
commit_delay_ms: options.u64(flags::COMMIT_DELAY_MS, 0)?,
|
||||
},
|
||||
client_id: client_id.clone(),
|
||||
service: service.clone(),
|
||||
}),
|
||||
_ => Started::Environment(EnvironmentLaunch {
|
||||
session_id,
|
||||
worker_id: options.id(flags::WORKER)?,
|
||||
incarnation_id,
|
||||
step_duration: options.rational(flags::STEP_NUMERATOR, flags::STEP_DENOMINATOR)?,
|
||||
ports: parse_ports(options.required(flags::PORTS)?)?,
|
||||
worker_threads: threads,
|
||||
observation_delay_steps: options.u64(flags::OBSERVATION_DELAY_STEPS, 0)?,
|
||||
renders: crate::media::RenderCounter::new(),
|
||||
faults: EnvironmentFaults {
|
||||
advance_delay_ms: options.u64(flags::ADVANCE_DELAY_MS, 0)?,
|
||||
omit_view_at_boundary: options.opt_u64(flags::OMIT_VIEW_AT_BOUNDARY)?,
|
||||
stale_view_at_boundary: options.opt_u64(flags::STALE_VIEW_AT_BOUNDARY)?,
|
||||
truncated_view_at_boundary: options
|
||||
.opt_u64(flags::TRUNCATED_VIEW_AT_BOUNDARY)?,
|
||||
omit_audio_at_boundary: options.opt_u64(flags::OMIT_AUDIO_AT_BOUNDARY)?,
|
||||
overlapping_audio_at_boundary: options
|
||||
.opt_u64(flags::OVERLAPPING_AUDIO_AT_BOUNDARY)?,
|
||||
},
|
||||
client_id: client_id.clone(),
|
||||
service: service.clone(),
|
||||
}),
|
||||
};
|
||||
let runtime = tokio::runtime::Builder::new_multi_thread()
|
||||
.worker_threads(threads)
|
||||
.enable_all()
|
||||
.build()
|
||||
.map_err(|e| format!("runtime: {e}"))?;
|
||||
runtime.block_on(async move {
|
||||
let handle = serve_one(&socket, &client_id, &service, &store_root, &what, threads).await?;
|
||||
// The worker serves until its supervisor's Worker.Shutdown, which it answers before
|
||||
// it stops. Exiting is then one event, not a race between a reply and a signal.
|
||||
handle.join().await;
|
||||
Ok(())
|
||||
})
|
||||
}
|
||||
|
||||
fn parse_ports(value: &str) -> Result<Vec<Id>, String> {
|
||||
value
|
||||
.split(',')
|
||||
.filter(|part| !part.is_empty())
|
||||
.map(|part| parse_id(part).map_err(|e| format!("--ports: {e}")))
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn measure_config(options: &Options) -> Result<crate::measure::MeasureConfig, String> {
|
||||
let mut config = crate::measure::MeasureConfig {
|
||||
steps: options.u64("steps", 200)?,
|
||||
warmup_steps: options.u64("warmup-steps", 10)?,
|
||||
worker_threads: options.usize("worker-threads", 1)?,
|
||||
..crate::measure::MeasureConfig::default()
|
||||
};
|
||||
if let Some(list) = options.optional("agents") {
|
||||
config.agent_counts = list
|
||||
.split(',')
|
||||
.filter(|p| !p.is_empty())
|
||||
.map(|p| p.parse::<usize>().map_err(|_| format!("--agents: {p:?}")))
|
||||
.collect::<Result<Vec<usize>, String>>()?;
|
||||
}
|
||||
if let Some(list) = options.optional("modes") {
|
||||
config.modes = list
|
||||
.split(',')
|
||||
.filter(|p| !p.is_empty())
|
||||
.map(parse_mode)
|
||||
.collect::<Result<Vec<ExecutionMode>, String>>()?;
|
||||
}
|
||||
Ok(config)
|
||||
}
|
||||
|
||||
fn parse_mode(name: &str) -> Result<ExecutionMode, String> {
|
||||
match name {
|
||||
"in-process" => Ok(ExecutionMode::InProcess),
|
||||
"thread" => Ok(ExecutionMode::Thread),
|
||||
"process" => Ok(ExecutionMode::Process),
|
||||
other => Err(format!("unknown mode {other:?}")),
|
||||
}
|
||||
}
|
||||
|
||||
/// Runs the execution-mode comparison and prints its table.
|
||||
///
|
||||
/// One child per row: a peak-memory figure is only that row's if nothing else ran in the
|
||||
/// process that produced it.
|
||||
fn measure(options: &Options) -> Result<(), String> {
|
||||
let config = measure_config(options)?;
|
||||
let program = std::env::current_exe().map_err(|e| format!("current exe: {e}"))?;
|
||||
let rows = crate::measure::run(&config, &program)?;
|
||||
print!("{}", crate::measure::table(&rows));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Measures exactly one row and prints it as one JSON object. The parent's child.
|
||||
fn measure_row(options: &Options) -> Result<(), String> {
|
||||
let config = measure_config(options)?;
|
||||
let mode = parse_mode(options.required("mode")?)?;
|
||||
let agents = options.usize("agents", 2)?;
|
||||
let runtime = tokio::runtime::Builder::new_multi_thread()
|
||||
.enable_all()
|
||||
.build()
|
||||
.map_err(|e| format!("runtime: {e}"))?;
|
||||
let row = runtime.block_on(crate::measure::one(&config, mode, agents))?;
|
||||
println!("{}", row.to_json());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// An unknown flag is refused by name. A renamed option must fail the launch rather than
|
||||
/// be accepted and ignored, which would turn a fault or a delay into a no-op.
|
||||
#[test]
|
||||
fn an_unknown_option_is_refused_by_name() {
|
||||
let args: Vec<String> = ["--session", "demo", "--stale-view-at-boundry", "2"]
|
||||
.iter()
|
||||
.map(|s| (*s).to_owned())
|
||||
.collect();
|
||||
let error = Options::parse(&args, &[flags::COMMON, flags::ENVIRONMENT_ONLY])
|
||||
.expect_err("an unknown option is refused");
|
||||
assert!(error.contains("--stale-view-at-boundry"), "{error}");
|
||||
}
|
||||
|
||||
/// A flag that belongs to another command is refused too: an agent has no render delay.
|
||||
#[test]
|
||||
fn an_option_of_another_command_is_refused() {
|
||||
let args: Vec<String> = ["--observation-delay-steps", "2"]
|
||||
.iter()
|
||||
.map(|s| (*s).to_owned())
|
||||
.collect();
|
||||
Options::parse(&args, &[flags::COMMON, flags::AGENT_ONLY])
|
||||
.expect_err("the agent command has no render delay");
|
||||
Options::parse(&args, &[flags::COMMON, flags::ENVIRONMENT_ONLY])
|
||||
.expect("the environment command does");
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -4,19 +4,31 @@
|
|||
//! one interval, and returns boundary `k+1` with its world time advanced by `stepDuration`. It
|
||||
//! never advances while waiting for the next request, and it does not free-run during agent
|
||||
//! initialization.
|
||||
//!
|
||||
//! Its native output is real: one immutable RGBA8 frame per boundary through a
|
||||
//! [`ViewPipeline`](crate::media::ViewPipeline) that honours the declared
|
||||
//! `observationDelaySteps`, and one audio chunk per transition with an exact sample budget.
|
||||
//! Nothing here resizes, mixes, composites or encodes anything; that is the presentation
|
||||
//! layer's work.
|
||||
|
||||
use std::collections::BTreeSet;
|
||||
use std::io::Write;
|
||||
|
||||
use crate::media::{self, AudioSource, RenderCounter, ViewPipeline};
|
||||
use crate::task::{controller_schema_ref, inspection, inspection_schema};
|
||||
// `crate::types` is this crate's facade over the shared `fly-session-types` crate; the
|
||||
// glob keeps the contract's own names in sight instead of restating them.
|
||||
use crate::types::*;
|
||||
use crate::worker::{BoxFuture, HandlerCtx, HandlerReply, StatusCell, WorkerEndpoint};
|
||||
|
||||
/// The arena's view: a 4x4 RGBA8 tile whose bytes carry the counter.
|
||||
pub const VIEW_WIDTH: u64 = 4;
|
||||
pub const VIEW_HEIGHT: u64 = 4;
|
||||
/// The arena's one view: a small native RGBA8 image.
|
||||
pub const VIEW_ID: &str = "arena";
|
||||
pub const VIEW_WIDTH: u64 = 32;
|
||||
pub const VIEW_HEIGHT: u64 = 24;
|
||||
|
||||
/// The arena's one audio stream. 48 kHz stereo is a native rate, not a presentation choice.
|
||||
pub const AUDIO_STREAM_ID: &str = "arena";
|
||||
pub const SAMPLE_RATE: u64 = 48_000;
|
||||
pub const CHANNELS: u64 = 2;
|
||||
|
||||
/// Deliberate faults a test can ask the environment for.
|
||||
#[derive(Clone, Debug, Default)]
|
||||
|
|
@ -26,6 +38,16 @@ pub struct EnvironmentFaults {
|
|||
/// Drop the required sensory view from the result at this boundary, so the coordinator
|
||||
/// meets a world that advanced with no usable sensory data.
|
||||
pub omit_view_at_boundary: Option<u64>,
|
||||
/// Serve the previous boundary's frame at this boundary: an extra-delayed sensory input,
|
||||
/// which is a step failure rather than an acceptable latest frame.
|
||||
pub stale_view_at_boundary: Option<u64>,
|
||||
/// Seal a frame one row short at this boundary, so its artifact length is not
|
||||
/// `rowStride x height`.
|
||||
pub truncated_view_at_boundary: Option<u64>,
|
||||
/// Leave the audio chunk out of the result at this boundary.
|
||||
pub omit_audio_at_boundary: Option<u64>,
|
||||
/// Emit an audio chunk that starts before the previous chunk ended.
|
||||
pub overlapping_audio_at_boundary: Option<u64>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
|
|
@ -36,6 +58,14 @@ pub struct EnvironmentConfig {
|
|||
/// The world's fixed reduced step duration. 60 Hz is `1/60` s.
|
||||
pub step_duration: RationalNs,
|
||||
pub ports: Vec<Id>,
|
||||
/// The thread allocation the launcher started this worker within.
|
||||
pub worker_threads: usize,
|
||||
/// The view's declared render delay, in steps. Zero is same-boundary output.
|
||||
pub observation_delay_steps: u64,
|
||||
/// Counts frames actually rendered, so a test can prove one image was not rendered twice.
|
||||
///
|
||||
/// It counts in this process only: a world with a process of its own counts there.
|
||||
pub renders: RenderCounter,
|
||||
pub faults: EnvironmentFaults,
|
||||
}
|
||||
|
||||
|
|
@ -52,6 +82,10 @@ pub struct CounterEnvironment {
|
|||
world_time: RationalNs,
|
||||
advances: u64,
|
||||
batches: BTreeSet<Id>,
|
||||
pipeline: Option<ViewPipeline>,
|
||||
audio: Option<AudioSource>,
|
||||
/// The frame served at the previous boundary, kept only so a fault can serve it again.
|
||||
previous_view: Option<(ViewRef, flybus::Artifact)>,
|
||||
}
|
||||
|
||||
impl CounterEnvironment {
|
||||
|
|
@ -67,6 +101,9 @@ impl CounterEnvironment {
|
|||
world_time: RationalNs::ZERO,
|
||||
advances: 0,
|
||||
batches: BTreeSet::new(),
|
||||
pipeline: None,
|
||||
audio: None,
|
||||
previous_view: None,
|
||||
config,
|
||||
}
|
||||
}
|
||||
|
|
@ -101,15 +138,25 @@ impl CounterEnvironment {
|
|||
}
|
||||
}
|
||||
|
||||
pub fn view_descriptor() -> ViewDescriptor {
|
||||
/// The arena's native view, with the configured render delay.
|
||||
pub fn view_descriptor(observation_delay_steps: u64) -> ViewDescriptor {
|
||||
ViewDescriptor {
|
||||
view_id: id("arena"),
|
||||
view_id: id(VIEW_ID),
|
||||
width: VIEW_WIDTH,
|
||||
height: VIEW_HEIGHT,
|
||||
row_stride: VIEW_WIDTH * 4,
|
||||
pixel_aspect_numerator: 1,
|
||||
pixel_aspect_denominator: 1,
|
||||
observation_delay_steps: 0,
|
||||
observation_delay_steps,
|
||||
}
|
||||
}
|
||||
|
||||
/// The arena's native audio stream.
|
||||
pub fn audio_descriptor() -> AudioDescriptor {
|
||||
AudioDescriptor {
|
||||
stream_id: id(AUDIO_STREAM_ID),
|
||||
sample_rate: SAMPLE_RATE,
|
||||
channels: CHANNELS,
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -119,10 +166,11 @@ impl CounterEnvironment {
|
|||
content_digest: digest_of_bytes(b"counter-arena-content-v1"),
|
||||
configuration_digest: digest_of_bytes(
|
||||
format!(
|
||||
"counter-arena-config-v1\nstep={}/{}\nports={}\n",
|
||||
"counter-arena-config-v1\nstep={}/{}\nports={}\ndelay={}\n",
|
||||
self.config.step_duration.numerator,
|
||||
self.config.step_duration.denominator,
|
||||
self.config.ports.len()
|
||||
self.config.ports.len(),
|
||||
self.config.observation_delay_steps
|
||||
)
|
||||
.as_bytes(),
|
||||
),
|
||||
|
|
@ -137,8 +185,10 @@ impl CounterEnvironment {
|
|||
})
|
||||
.collect(),
|
||||
inspection_schema: inspection_schema(),
|
||||
views: vec![CounterEnvironment::view_descriptor()],
|
||||
audio: Vec::new(),
|
||||
views: vec![CounterEnvironment::view_descriptor(
|
||||
self.config.observation_delay_steps,
|
||||
)],
|
||||
audio: vec![CounterEnvironment::audio_descriptor()],
|
||||
recovery: Recovery::ExactCheckpoint,
|
||||
determinism: Determinism::FixedBuild,
|
||||
};
|
||||
|
|
@ -146,73 +196,76 @@ impl CounterEnvironment {
|
|||
Ok(descriptor)
|
||||
}
|
||||
|
||||
/// Seals one immutable native frame for the current counter and returns the handle.
|
||||
async fn render(
|
||||
&self,
|
||||
ctx: &HandlerCtx<'_>,
|
||||
) -> DomainResult<(ViewRef, flybus::Artifact)> {
|
||||
let descriptor = CounterEnvironment::view_descriptor();
|
||||
let len = descriptor.byte_length();
|
||||
let mut writer = ctx
|
||||
.client
|
||||
.artifacts()
|
||||
.allocate(len, "image/x-rgba")
|
||||
.await
|
||||
.map_err(|e| {
|
||||
DomainError::new(
|
||||
ErrorCode::BackendFailure,
|
||||
format!("frame allocation failed: {}", e.message),
|
||||
MutationCertainty::Applied,
|
||||
)
|
||||
})?;
|
||||
// Every pixel carries the counter's low byte, so an agent reading the frame reads the
|
||||
// world rather than a constant.
|
||||
let byte = (self.counter & 0xff) as u8;
|
||||
writer
|
||||
.write_all(&vec![byte; len as usize])
|
||||
.map_err(|e| {
|
||||
DomainError::new(
|
||||
ErrorCode::BackendFailure,
|
||||
format!("frame write failed: {e}"),
|
||||
MutationCertainty::Applied,
|
||||
)
|
||||
})?;
|
||||
let artifact = writer.seal().await.map_err(|e| {
|
||||
DomainError::new(
|
||||
ErrorCode::BackendFailure,
|
||||
format!("frame seal failed: {}", e.message),
|
||||
MutationCertainty::Applied,
|
||||
)
|
||||
})?;
|
||||
let view = ViewRef {
|
||||
view_id: descriptor.view_id.clone(),
|
||||
produced_step: descriptor.required_produced_step(self.boundary),
|
||||
pixels: artifact.reference().clone(),
|
||||
};
|
||||
Ok((view, artifact))
|
||||
}
|
||||
|
||||
/// Renders this boundary's native media and returns the observation with its owned
|
||||
/// handles. The same immutable object serves the sensory and the broadcast view; nothing
|
||||
/// is rendered twice and no second copy of the pixels exists.
|
||||
async fn observation(
|
||||
&self,
|
||||
&mut self,
|
||||
ctx: &HandlerCtx<'_>,
|
||||
) -> DomainResult<(WorldObservation, Vec<(String, flybus::Artifact)>)> {
|
||||
let omit = self.config.faults.omit_view_at_boundary == Some(self.boundary);
|
||||
let (views, attachments) = if omit {
|
||||
(Vec::new(), Vec::new())
|
||||
let boundary = self.boundary;
|
||||
let counter = self.counter;
|
||||
let pipeline = self
|
||||
.pipeline
|
||||
.as_mut()
|
||||
.ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no view pipeline"))?;
|
||||
if self.config.faults.truncated_view_at_boundary == Some(boundary) {
|
||||
pipeline
|
||||
.render_truncated(ctx.client, boundary, counter)
|
||||
.await?;
|
||||
} else {
|
||||
let (view, artifact) = self.render(ctx).await?;
|
||||
let name = format!("view.{}", view.view_id);
|
||||
(vec![view], vec![(name, artifact)])
|
||||
};
|
||||
pipeline.render(ctx.client, boundary, counter).await?;
|
||||
}
|
||||
let produced = pipeline.at(boundary);
|
||||
|
||||
let mut attachments = Vec::new();
|
||||
let mut views = Vec::new();
|
||||
if self.config.faults.omit_view_at_boundary == Some(boundary) {
|
||||
// A world that advanced with no usable sensory data.
|
||||
} else if self.config.faults.stale_view_at_boundary == Some(boundary) {
|
||||
if let Some((view, artifact)) = self.previous_view.clone() {
|
||||
attachments.push((media::view_attachment(&view.view_id), artifact));
|
||||
views.push(view);
|
||||
}
|
||||
} else if let Some((view, artifact)) = produced.clone() {
|
||||
attachments.push((media::view_attachment(&view.view_id), artifact));
|
||||
views.push(view);
|
||||
} else {
|
||||
return Err(DomainError::new(
|
||||
ErrorCode::BackendFailure,
|
||||
"the view pipeline has no frame for this boundary",
|
||||
MutationCertainty::Applied,
|
||||
));
|
||||
}
|
||||
self.previous_view = produced;
|
||||
|
||||
let mut audio = Vec::new();
|
||||
if boundary > 0 && self.config.faults.omit_audio_at_boundary != Some(boundary) {
|
||||
let step = self.config.step_duration;
|
||||
let overlap = self.config.faults.overlapping_audio_at_boundary == Some(boundary);
|
||||
let source = self
|
||||
.audio
|
||||
.as_mut()
|
||||
.ok_or_else(|| DomainError::before(ErrorCode::InvalidPhase, "no audio source"))?;
|
||||
let (mut chunk, artifact) = source.produce(ctx.client, &step, counter).await?;
|
||||
if overlap {
|
||||
// A chunk that starts inside the previous one: the timeline refuses it rather
|
||||
// than playing the same samples twice.
|
||||
chunk.first_sample = chunk.first_sample.saturating_sub(1);
|
||||
}
|
||||
attachments.push((media::audio_attachment(&chunk.stream_id), artifact));
|
||||
audio.push(chunk);
|
||||
}
|
||||
|
||||
let observation = WorldObservation {
|
||||
boundary: self.boundary,
|
||||
boundary,
|
||||
world_time: self.world_time,
|
||||
engine_frame: Some(self.boundary.to_string()),
|
||||
engine_frame: Some(boundary.to_string()),
|
||||
sensory_views: views.clone(),
|
||||
inspection: inspection(self.counter, self.boundary),
|
||||
inspection: inspection(counter, boundary),
|
||||
// The same immutable object serves the broadcast view; nothing is rendered twice.
|
||||
broadcast_views: views,
|
||||
audio: Vec::new(),
|
||||
audio,
|
||||
};
|
||||
Ok((observation, attachments))
|
||||
}
|
||||
|
|
@ -268,6 +321,13 @@ impl CounterEnvironment {
|
|||
self.counter = 0;
|
||||
self.world_time = RationalNs::ZERO;
|
||||
self.batches.clear();
|
||||
self.pipeline = Some(ViewPipeline::new(
|
||||
CounterEnvironment::view_descriptor(self.config.observation_delay_steps),
|
||||
self.config.renders.clone(),
|
||||
));
|
||||
// A fresh episode starts at audio origin zero; a restore would resume the preserved
|
||||
// sample position instead, and its first chunk would mark the discontinuity.
|
||||
self.audio = Some(AudioSource::new(CounterEnvironment::audio_descriptor(), 0));
|
||||
self.descriptor = Some(descriptor.clone());
|
||||
// The world is stopped when O[0] goes out and cannot free-run while the brains boot.
|
||||
self.status.set_state(WorkerState::Ready);
|
||||
|
|
@ -425,6 +485,10 @@ impl WorkerEndpoint for CounterEnvironment {
|
|||
self.status.clone()
|
||||
}
|
||||
|
||||
fn worker_threads(&self) -> u64 {
|
||||
self.config.worker_threads as u64
|
||||
}
|
||||
|
||||
fn methods(&self) -> Vec<&'static str> {
|
||||
vec!["Environment.Initialize", "Environment.Advance"]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,36 +1,38 @@
|
|||
//! The runnable synthetic composition: one router, two fake agents, one counter arena and one
|
||||
//! coordinator, over either transport.
|
||||
//! The runnable synthetic composition: one router, the configured flies, one counter arena and
|
||||
//! one coordinator, in whichever execution mode the composition asks for.
|
||||
//!
|
||||
//! All participants use router semantics even when colocated, so the in-memory and
|
||||
//! Unix-socket runs exercise the same code. The caller owns the store root directory, which
|
||||
//! keeps this module free of a temporary-directory dependency.
|
||||
//! All participants use router semantics even when colocated, so every mode and both
|
||||
//! transports exercise the same code. The caller owns the store root directory, which keeps
|
||||
//! this module free of a temporary-directory dependency.
|
||||
//!
|
||||
//! The three execution modes are the SESSION-02 comparison:
|
||||
//!
|
||||
//! | Mode | Where each participant runs | Transport |
|
||||
//! | --- | --- | --- |
|
||||
//! | [`ExecutionMode::InProcess`] | A task on the coordinator's runtime | either |
|
||||
//! | [`ExecutionMode::Thread`] | Its own OS thread and runtime | Unix socket |
|
||||
//! | [`ExecutionMode::Process`] | Its own process, one per fly plus one world | Unix socket |
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::path::Path;
|
||||
use std::sync::Mutex;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
|
||||
use flybus::{
|
||||
Client, ClientConfig, Grants, Pattern, Policy, Router, RouterConfig, ServiceConfig, Transport,
|
||||
UnixListenerHandle,
|
||||
};
|
||||
use flybus::{Client, Grants, Pattern, Policy, Router, RouterConfig};
|
||||
|
||||
use crate::agent::{AgentConfig, AgentFaults, FakeAgentWorker, synthetic_profile};
|
||||
use crate::agent::{AgentFaults, synthetic_profile};
|
||||
use crate::coordinator::{AgentSlot, Coordinator};
|
||||
use crate::environment::{CounterEnvironment, EnvironmentConfig, EnvironmentFaults};
|
||||
use crate::rpc::WorkerRef;
|
||||
use crate::environment::EnvironmentFaults;
|
||||
use crate::media::{RenderCounter, SensorLog};
|
||||
use crate::launcher::{
|
||||
AgentLaunch, EnvironmentLaunch, Launcher, ReapOutcome, SUPERVISOR_CLIENT, ThreadBudget,
|
||||
};
|
||||
use crate::task::{ActionExecutor, CounterTask, IdentityExecutor, Terminal};
|
||||
// `crate::types` is this crate's facade over the shared `fly-session-types` crate; the
|
||||
// glob keeps the contract's own names in sight instead of restating them.
|
||||
use crate::types::*;
|
||||
use crate::worker::{StatusCell, WorkerHandle, serve};
|
||||
use crate::worker::StatusCell;
|
||||
|
||||
/// Which transport the session runs over. Both must produce the same behaviour.
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub enum Via {
|
||||
Memory,
|
||||
Unix,
|
||||
}
|
||||
pub use crate::launcher::{ExecutionMode, Via};
|
||||
|
||||
/// One agent in the composition.
|
||||
#[derive(Clone, Debug)]
|
||||
|
|
@ -41,6 +43,22 @@ pub struct AgentSpec {
|
|||
/// derivation algorithm is specified before the real agent slice.
|
||||
pub seed: i32,
|
||||
pub faults: AgentFaults,
|
||||
/// The threads this agent asks the launcher for. `Agent.Initialize` carries exactly what
|
||||
/// the launcher allocated, which `workers-v1` requires it to lie within.
|
||||
pub worker_threads: usize,
|
||||
}
|
||||
|
||||
impl AgentSpec {
|
||||
/// One agent on one thread, with no injected fault.
|
||||
pub fn new(agent_id: &str, port_id: &str, seed: i32) -> AgentSpec {
|
||||
AgentSpec {
|
||||
agent_id: id(agent_id),
|
||||
port_id: id(port_id),
|
||||
seed,
|
||||
faults: AgentFaults::default(),
|
||||
worker_threads: 1,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The composition the harness builds.
|
||||
|
|
@ -55,7 +73,18 @@ pub struct HarnessConfig {
|
|||
pub tick_ms: u64,
|
||||
pub warmup_ticks: u64,
|
||||
pub terminal: Terminal,
|
||||
/// The view's declared render delay, in steps. Zero is same-boundary output.
|
||||
pub observation_delay_steps: u64,
|
||||
pub environment_faults: EnvironmentFaults,
|
||||
/// Where each participant runs.
|
||||
pub mode: ExecutionMode,
|
||||
/// The total thread allocation the launcher may hand out. `None` sizes it from the
|
||||
/// composition and the machine, which is what an ordinary run wants; a test that means to
|
||||
/// exhaust the budget names a number.
|
||||
pub thread_budget: Option<usize>,
|
||||
/// The threads reserved for the coordinator, its router and its store.
|
||||
pub coordinator_threads: usize,
|
||||
pub environment_threads: usize,
|
||||
}
|
||||
|
||||
impl Default for HarnessConfig {
|
||||
|
|
@ -65,31 +94,46 @@ impl Default for HarnessConfig {
|
|||
epoch: id("e1"),
|
||||
episode_id: id("ep1"),
|
||||
agents: vec![
|
||||
AgentSpec {
|
||||
agent_id: id("fly-a"),
|
||||
port_id: id("p1"),
|
||||
seed: 7,
|
||||
faults: AgentFaults::default(),
|
||||
},
|
||||
AgentSpec {
|
||||
agent_id: id("fly-b"),
|
||||
port_id: id("p2"),
|
||||
seed: 11,
|
||||
faults: AgentFaults::default(),
|
||||
},
|
||||
AgentSpec { seed: 7, ..AgentSpec::new("fly-a", "p1", 7) },
|
||||
AgentSpec { seed: 11, ..AgentSpec::new("fly-b", "p2", 11) },
|
||||
],
|
||||
step_hz: 60,
|
||||
tick_ms: 1,
|
||||
warmup_ticks: 10,
|
||||
terminal: Terminal::Never,
|
||||
observation_delay_steps: 0,
|
||||
environment_faults: EnvironmentFaults::default(),
|
||||
mode: ExecutionMode::InProcess,
|
||||
thread_budget: None,
|
||||
coordinator_threads: 1,
|
||||
environment_threads: 1,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl HarnessConfig {
|
||||
/// The threads this composition needs at a minimum: the coordinator, the world and every
|
||||
/// agent's own allocation.
|
||||
pub fn required_threads(&self) -> usize {
|
||||
self.coordinator_threads
|
||||
+ self.environment_threads
|
||||
+ self.agents.iter().map(|a| a.worker_threads).sum::<usize>()
|
||||
}
|
||||
|
||||
/// The budget the launcher runs under: what was configured, or a budget that covers both
|
||||
/// this composition and this machine's physical cores.
|
||||
pub fn budget(&self) -> Result<ThreadBudget, DomainError> {
|
||||
let total = self
|
||||
.thread_budget
|
||||
.unwrap_or_else(|| crate::metrics::physical_cores().max(self.required_threads()));
|
||||
ThreadBudget::new(total, self.coordinator_threads)
|
||||
}
|
||||
}
|
||||
|
||||
const ENV_SERVICE: &str = "env.arena";
|
||||
const ENV_CLIENT: &str = "environment";
|
||||
const ENV_WORKER: &str = "arena";
|
||||
const COORDINATOR_CLIENT: &str = "coordinator";
|
||||
|
||||
fn agent_service(agent_id: &Id) -> String {
|
||||
format!("agent.{agent_id}")
|
||||
|
|
@ -105,37 +149,6 @@ fn grants(f: impl FnOnce(&mut Grants)) -> Grants {
|
|||
g
|
||||
}
|
||||
|
||||
/// Makes a connection for one launcher-bound participant, over the chosen transport.
|
||||
struct Connector {
|
||||
router: Router,
|
||||
via: Via,
|
||||
store_root: PathBuf,
|
||||
sockets: PathBuf,
|
||||
next_socket: AtomicU64,
|
||||
listeners: Mutex<Vec<UnixListenerHandle>>,
|
||||
}
|
||||
|
||||
impl Connector {
|
||||
async fn client(&self, id: &str) -> Result<Client, flybus::BusError> {
|
||||
let transport = match self.via {
|
||||
Via::Memory => self.router.connect_in_memory_as(id),
|
||||
Via::Unix => {
|
||||
let n = self.next_socket.fetch_add(1, Ordering::Relaxed);
|
||||
let path = self.sockets.join(format!("{id}-{n}.sock"));
|
||||
let listener = self.router.listen_unix_as(&path, id).await.map_err(|e| {
|
||||
flybus::BusError::new(flybus::ErrorCode::RouterLost, format!("listen: {e}"))
|
||||
})?;
|
||||
let transport = Transport::unix(&path).await.map_err(|e| {
|
||||
flybus::BusError::new(flybus::ErrorCode::RouterLost, format!("connect: {e}"))
|
||||
})?;
|
||||
self.listeners.lock().expect("not poisoned").push(listener);
|
||||
transport
|
||||
}
|
||||
};
|
||||
Client::connect(transport, ClientConfig::new(id, &self.store_root)).await
|
||||
}
|
||||
}
|
||||
|
||||
/// What a restarted worker looks like from the outside: a new registration and a new
|
||||
/// incarnation, both different from the ones the coordinator pinned.
|
||||
#[derive(Clone, Debug)]
|
||||
|
|
@ -148,16 +161,22 @@ pub struct Restarted {
|
|||
/// A running synthetic session.
|
||||
pub struct SessionHarness {
|
||||
pub coordinator: Coordinator,
|
||||
pub environment: WorkerHandle,
|
||||
pub agents: BTreeMap<Id, WorkerHandle>,
|
||||
pub config: HarnessConfig,
|
||||
pub via: Via,
|
||||
connector: Connector,
|
||||
pub mode: ExecutionMode,
|
||||
/// The media instrumentation of the participants that live in this process. Both are
|
||||
/// shared memory, so both are empty for a participant with a process of its own; the
|
||||
/// accessors below return `None` there rather than zero.
|
||||
renders: RenderCounter,
|
||||
sensors: BTreeMap<Id, SensorLog>,
|
||||
/// The supervisor. It owns every participant's lifetime and thread allocation.
|
||||
pub launcher: Launcher,
|
||||
observers: Mutex<Vec<Client>>,
|
||||
}
|
||||
|
||||
impl SessionHarness {
|
||||
/// Builds the router, the workers and the coordinator. Nothing has stepped yet.
|
||||
/// Builds the router, launches the workers and builds the coordinator. Nothing has
|
||||
/// stepped yet.
|
||||
pub async fn start(
|
||||
via: Via,
|
||||
root: &Path,
|
||||
|
|
@ -167,16 +186,29 @@ impl SessionHarness {
|
|||
let sockets = root.join("sockets");
|
||||
std::fs::create_dir_all(&sockets).expect("the caller owns a writable directory");
|
||||
|
||||
// The launcher's policy: who may connect, and what each may do. Naming a target is not
|
||||
// authority to use it, so the supervisor calls but never registers or publishes, and a
|
||||
// worker registers exactly one service and calls nothing.
|
||||
let mut policy = Policy::closed()
|
||||
.client(
|
||||
"coordinator",
|
||||
COORDINATOR_CLIENT,
|
||||
grants(|g| {
|
||||
g.call = vec![Pattern::prefix("agent."), Pattern::prefix("env.")];
|
||||
g.publish = vec![Pattern::prefix("session.")];
|
||||
g.manage_topics = vec![Pattern::prefix("session.")];
|
||||
}),
|
||||
)
|
||||
.client(
|
||||
SUPERVISOR_CLIENT,
|
||||
grants(|g| {
|
||||
g.call = vec![Pattern::prefix("agent."), Pattern::prefix("env.")];
|
||||
}),
|
||||
)
|
||||
.client(ENV_CLIENT, grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]))
|
||||
.client(
|
||||
&format!("{ENV_CLIENT}-r2"),
|
||||
grants(|g| g.register = vec![Pattern::exact(ENV_SERVICE)]),
|
||||
)
|
||||
.client("observer", grants(|g| g.subscribe = vec![Pattern::prefix("session.")]));
|
||||
for spec in &config.agents {
|
||||
let service = agent_service(&spec.agent_id);
|
||||
|
|
@ -196,66 +228,77 @@ impl SessionHarness {
|
|||
let router = Router::new(router_config).map_err(|e| {
|
||||
flybus::BusError::new(flybus::ErrorCode::StoreFailure, format!("router: {e}"))
|
||||
})?;
|
||||
let connector = Connector {
|
||||
router,
|
||||
via,
|
||||
store_root,
|
||||
sockets,
|
||||
next_socket: AtomicU64::new(0),
|
||||
listeners: Mutex::new(Vec::new()),
|
||||
};
|
||||
|
||||
let budget = config.budget().map_err(refusal)?;
|
||||
let mut launcher =
|
||||
Launcher::start(router, config.mode, via, &store_root, &sockets, budget).await?;
|
||||
|
||||
let step_duration = hz(config.step_hz).expect("a positive cadence");
|
||||
let tick_duration = millis(config.tick_ms).expect("a positive tick");
|
||||
let renders = RenderCounter::new();
|
||||
let sensors: BTreeMap<Id, SensorLog> = config
|
||||
.agents
|
||||
.iter()
|
||||
.map(|spec| (spec.agent_id.clone(), SensorLog::new()))
|
||||
.collect();
|
||||
|
||||
// The environment first: it owns the world and the descriptor.
|
||||
let env_client = connector.client(ENV_CLIENT).await?;
|
||||
let env_service = env_client.register(ENV_SERVICE, ServiceConfig::default()).await?;
|
||||
let env_incarnation = env_service.incarnation().to_owned();
|
||||
let environment = serve(
|
||||
env_client,
|
||||
env_service,
|
||||
CounterEnvironment::new(EnvironmentConfig {
|
||||
let environment = launcher
|
||||
.launch_environment(EnvironmentLaunch {
|
||||
session_id: config.session_id.clone(),
|
||||
worker_id: id(ENV_WORKER),
|
||||
incarnation_id: id("arena-inc-1"),
|
||||
step_duration,
|
||||
ports: config.agents.iter().map(|a| a.port_id.clone()).collect(),
|
||||
worker_threads: config.environment_threads,
|
||||
observation_delay_steps: config.observation_delay_steps,
|
||||
renders: renders.clone(),
|
||||
faults: config.environment_faults.clone(),
|
||||
}),
|
||||
);
|
||||
client_id: ENV_CLIENT.to_owned(),
|
||||
service: ENV_SERVICE.to_owned(),
|
||||
})
|
||||
.await
|
||||
.map_err(refusal)?;
|
||||
let environment_ref = launcher
|
||||
.worker(&environment.worker_id)
|
||||
.expect("just launched")
|
||||
.worker_ref();
|
||||
|
||||
let mut slots = Vec::new();
|
||||
let mut agents = BTreeMap::new();
|
||||
for spec in &config.agents {
|
||||
let service_name = agent_service(&spec.agent_id);
|
||||
let client = connector.client(&agent_client(&spec.agent_id)).await?;
|
||||
let service = client.register(&service_name, ServiceConfig::default()).await?;
|
||||
let incarnation = service.incarnation().to_owned();
|
||||
let handle = serve(
|
||||
client,
|
||||
service,
|
||||
FakeAgentWorker::new(AgentConfig {
|
||||
let identity = launcher
|
||||
.launch_agent(AgentLaunch {
|
||||
session_id: config.session_id.clone(),
|
||||
agent_id: spec.agent_id.clone(),
|
||||
port_id: spec.port_id.clone(),
|
||||
incarnation_id: parse_id(&format!("{}-inc-1", spec.agent_id))
|
||||
.expect("an agent id plus a suffix is an Id"),
|
||||
tick_duration,
|
||||
warmup_ticks: config.warmup_ticks,
|
||||
worker_threads: spec.worker_threads,
|
||||
sensors: sensors[&spec.agent_id].clone(),
|
||||
faults: spec.faults.clone(),
|
||||
}),
|
||||
);
|
||||
slots.push(AgentSlot::new(
|
||||
WorkerRef::new(&service_name, &incarnation, &spec.agent_id),
|
||||
client_id: agent_client(&spec.agent_id),
|
||||
service: agent_service(&spec.agent_id),
|
||||
})
|
||||
.await
|
||||
.map_err(refusal)?;
|
||||
let worker_ref = launcher
|
||||
.worker(&spec.agent_id)
|
||||
.expect("just launched")
|
||||
.worker_ref();
|
||||
let mut slot = AgentSlot::new(
|
||||
worker_ref,
|
||||
spec.agent_id.clone(),
|
||||
spec.port_id.clone(),
|
||||
synthetic_profile(&spec.agent_id, &tick_duration, config.warmup_ticks),
|
||||
spec.seed,
|
||||
));
|
||||
agents.insert(spec.agent_id.clone(), handle);
|
||||
);
|
||||
slot.worker_threads = identity.worker_threads as u64;
|
||||
slots.push(slot);
|
||||
}
|
||||
|
||||
let coordinator_client = connector.client("coordinator").await?;
|
||||
let coordinator_client = launcher.connect(COORDINATOR_CLIENT).await?;
|
||||
let executors: BTreeMap<Id, Box<dyn ActionExecutor>> = config
|
||||
.agents
|
||||
.iter()
|
||||
|
|
@ -268,7 +311,7 @@ impl SessionHarness {
|
|||
config.session_id.clone(),
|
||||
config.epoch.clone(),
|
||||
config.episode_id.clone(),
|
||||
WorkerRef::new(ENV_SERVICE, &env_incarnation, &id(ENV_WORKER)),
|
||||
environment_ref,
|
||||
slots,
|
||||
Box::new(CounterTask::new(&config.epoch, config.terminal)),
|
||||
executors,
|
||||
|
|
@ -276,27 +319,44 @@ impl SessionHarness {
|
|||
|
||||
Ok(SessionHarness {
|
||||
coordinator,
|
||||
environment,
|
||||
agents,
|
||||
config,
|
||||
via,
|
||||
connector,
|
||||
mode: launcher.mode(),
|
||||
renders,
|
||||
sensors,
|
||||
launcher,
|
||||
observers: Mutex::new(Vec::new()),
|
||||
})
|
||||
}
|
||||
|
||||
pub fn router(&self) -> &Router {
|
||||
&self.connector.router
|
||||
self.launcher.router()
|
||||
}
|
||||
|
||||
/// The coordinator and its supervisor, borrowed apart.
|
||||
///
|
||||
/// A supervisor acts while a transition is in flight -- that is what a supervisor is for
|
||||
/// -- so the two have to be reachable at the same time.
|
||||
pub fn parts(&mut self) -> (&mut Coordinator, &mut Launcher) {
|
||||
(&mut self.coordinator, &mut self.launcher)
|
||||
}
|
||||
|
||||
/// The router's own counters: owners, roots, queued messages and store bytes.
|
||||
pub fn router_stats(&self) -> flybus::RouterStats {
|
||||
self.launcher.router().stats()
|
||||
}
|
||||
|
||||
/// A client for `id`, connected the same way every participant is.
|
||||
///
|
||||
/// An unconfigured client id is refused by the launcher's policy before it can route, so
|
||||
/// this is not a way around the composition.
|
||||
pub async fn client(&self, id: &str) -> Result<Client, flybus::BusError> {
|
||||
self.connector.client(id).await
|
||||
self.launcher.connect(id).await
|
||||
}
|
||||
|
||||
/// An extra subscriber, for a test that watches the published boundaries.
|
||||
pub async fn observer(&self) -> Result<Client, flybus::BusError> {
|
||||
let client = self.connector.client("observer").await?;
|
||||
let client = self.launcher.connect("observer").await?;
|
||||
self.observers.lock().expect("not poisoned").push(client.clone());
|
||||
Ok(client)
|
||||
}
|
||||
|
|
@ -306,22 +366,6 @@ impl SessionHarness {
|
|||
/// The coordinator still pins the old registration, so its next call to that agent fails
|
||||
/// rather than silently reaching another brain.
|
||||
pub async fn restart_agent(&mut self, agent_id: &Id) -> Result<Restarted, flybus::BusError> {
|
||||
let tick_duration = millis(self.config.tick_ms).expect("a positive tick");
|
||||
if let Some(old) = self.agents.remove(agent_id) {
|
||||
old.stop().await;
|
||||
}
|
||||
let service_name = agent_service(agent_id);
|
||||
let client = self.connector.client(&format!("{}-r2", agent_client(agent_id))).await?;
|
||||
let service = loop {
|
||||
match client.register(&service_name, ServiceConfig::default()).await {
|
||||
Ok(service) => break service,
|
||||
Err(e) if e.code == flybus::ErrorCode::Conflict => {
|
||||
// The old registration is released when its connection finishes closing.
|
||||
tokio::time::sleep(std::time::Duration::from_millis(5)).await;
|
||||
}
|
||||
Err(e) => return Err(e),
|
||||
}
|
||||
};
|
||||
let spec = self
|
||||
.config
|
||||
.agents
|
||||
|
|
@ -329,53 +373,115 @@ impl SessionHarness {
|
|||
.find(|spec| spec.agent_id == *agent_id)
|
||||
.expect("a configured agent")
|
||||
.clone();
|
||||
self.launcher.kill(agent_id).await;
|
||||
let tick_duration = millis(self.config.tick_ms).expect("a positive tick");
|
||||
let incarnation_id =
|
||||
parse_id(&format!("{agent_id}-inc-2")).expect("an agent id plus a suffix is an Id");
|
||||
let restarted = Restarted {
|
||||
service: service_name,
|
||||
service_incarnation: service.incarnation().to_owned(),
|
||||
incarnation_id: incarnation_id.clone(),
|
||||
};
|
||||
let handle = serve(
|
||||
client,
|
||||
service,
|
||||
FakeAgentWorker::new(AgentConfig {
|
||||
self.launcher
|
||||
.launch_agent(AgentLaunch {
|
||||
session_id: self.config.session_id.clone(),
|
||||
agent_id: agent_id.clone(),
|
||||
incarnation_id,
|
||||
port_id: spec.port_id.clone(),
|
||||
incarnation_id: incarnation_id.clone(),
|
||||
tick_duration,
|
||||
warmup_ticks: self.config.warmup_ticks,
|
||||
faults: spec.faults,
|
||||
}),
|
||||
);
|
||||
self.agents.insert(agent_id.clone(), handle);
|
||||
Ok(restarted)
|
||||
worker_threads: spec.worker_threads,
|
||||
// The same log: a replacement worker in this process keeps writing where its
|
||||
// predecessor wrote, so a restore's sensory input is visible beside it.
|
||||
sensors: self.sensors.get(agent_id).cloned().unwrap_or_default(),
|
||||
faults: spec.faults.clone(),
|
||||
client_id: format!("{}-r2", agent_client(agent_id)),
|
||||
service: agent_service(agent_id),
|
||||
})
|
||||
.await
|
||||
.map_err(refusal)?;
|
||||
let worker = self.launcher.worker(agent_id).expect("just launched");
|
||||
Ok(Restarted {
|
||||
service: worker.identity.service.clone(),
|
||||
service_incarnation: worker.service_incarnation.clone(),
|
||||
incarnation_id,
|
||||
})
|
||||
}
|
||||
|
||||
/// The agent worker's progress counter, which is its fake model's mutation count.
|
||||
pub fn agent_mutations(&self, agent_id: &Id) -> u64 {
|
||||
self.agents.get(agent_id).map(WorkerHandle::progress_counter).unwrap_or_default()
|
||||
/// Ends one participant without asking it, as a crash would.
|
||||
pub async fn kill(&mut self, worker_id: &Id) -> ReapOutcome {
|
||||
self.launcher.kill(worker_id).await
|
||||
}
|
||||
|
||||
pub fn environment_mutations(&self) -> u64 {
|
||||
self.environment.progress_counter()
|
||||
/// The worker id the environment answers to.
|
||||
pub fn environment_id(&self) -> Id {
|
||||
id(ENV_WORKER)
|
||||
}
|
||||
|
||||
pub fn agent_status(&self, agent_id: &Id) -> Option<StatusCell> {
|
||||
self.agents.get(agent_id).map(|handle| handle.status.clone())
|
||||
}
|
||||
|
||||
/// Stops every worker and closes the router.
|
||||
pub async fn shutdown(self) {
|
||||
let SessionHarness { coordinator, environment, agents, connector, observers, .. } = self;
|
||||
drop(coordinator);
|
||||
environment.stop().await;
|
||||
for (_, handle) in agents {
|
||||
handle.stop().await;
|
||||
/// What one agent read out of its sensory attachments, in order, when this process is
|
||||
/// where that log lives.
|
||||
///
|
||||
/// `None` means "not observable from here", not "nothing was read": an agent with a
|
||||
/// process of its own records into its own copy. The media path itself crosses a process
|
||||
/// boundary -- the frame is one artifact in the shared store, reached through owned
|
||||
/// handles -- but this instrumentation does not, because it is shared memory.
|
||||
pub fn sensor_log(&self, agent_id: &Id) -> Option<SensorLog> {
|
||||
match self.mode {
|
||||
ExecutionMode::Process => None,
|
||||
_ => self.sensors.get(agent_id).cloned(),
|
||||
}
|
||||
}
|
||||
|
||||
/// How many native frames the environment rendered, when the world lives in this process.
|
||||
///
|
||||
/// `None` for a world with a process of its own, for the same reason as above.
|
||||
pub fn renders(&self) -> Option<u64> {
|
||||
match self.mode {
|
||||
ExecutionMode::Process => None,
|
||||
_ => Some(self.renders.count()),
|
||||
}
|
||||
}
|
||||
|
||||
/// The agent worker's progress counter, which is its fake model's mutation count, when
|
||||
/// this process is where that counter lives.
|
||||
///
|
||||
/// `None` means "not observable from here", not "nothing happened": a participant with a
|
||||
/// process of its own keeps its counter there. [`SessionHarness::progress_of`] reads it
|
||||
/// over the bus and works in every mode.
|
||||
pub fn agent_mutations(&self, agent_id: &Id) -> Option<u64> {
|
||||
self.launcher
|
||||
.worker(agent_id)
|
||||
.and_then(crate::launcher::LaunchedWorker::progress_counter)
|
||||
}
|
||||
|
||||
pub fn environment_mutations(&self) -> Option<u64> {
|
||||
self.agent_mutations(&id(ENV_WORKER))
|
||||
}
|
||||
|
||||
/// One participant's progress counter, read over the bus. Works in every execution mode.
|
||||
pub async fn progress_of(&mut self, worker_id: &Id) -> Result<u64, DomainError> {
|
||||
Ok(self.launcher.health_check(worker_id).await?.progress_counter)
|
||||
}
|
||||
|
||||
/// The local status cell of a participant in this process, or `None` for one with a
|
||||
/// process of its own.
|
||||
pub fn agent_status(&self, agent_id: &Id) -> Option<StatusCell> {
|
||||
self.launcher.worker(agent_id).and_then(crate::launcher::LaunchedWorker::status)
|
||||
}
|
||||
|
||||
/// Reaps every participant and closes the router.
|
||||
pub async fn shutdown(self) {
|
||||
let SessionHarness { coordinator, mut launcher, observers, .. } = self;
|
||||
drop(coordinator);
|
||||
launcher.reap_all(&id("shutdown")).await;
|
||||
for observer in observers.into_inner().expect("not poisoned") {
|
||||
observer.close().await;
|
||||
}
|
||||
connector.router.shutdown();
|
||||
launcher.router().shutdown();
|
||||
}
|
||||
}
|
||||
|
||||
/// A launcher refusal, as a bus error: the harness's one error type stays the bus's.
|
||||
fn refusal(e: DomainError) -> flybus::BusError {
|
||||
let code = match e.code {
|
||||
ErrorCode::Busy => flybus::ErrorCode::QuotaExceeded,
|
||||
ErrorCode::IdentityMismatch => flybus::ErrorCode::NotAuthorized,
|
||||
_ => flybus::ErrorCode::RouterLost,
|
||||
};
|
||||
flybus::BusError::new(code, e.to_string())
|
||||
}
|
||||
|
|
|
|||
1542
services/flysim/crates/fly-session/src/launcher.rs
Normal file
1542
services/flysim/crates/fly-session/src/launcher.rs
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -22,11 +22,16 @@
|
|||
//! [`step-v1`]: https://example.invalid/step-v1
|
||||
|
||||
pub mod agent;
|
||||
pub mod cli;
|
||||
pub mod clock;
|
||||
pub mod coordinator;
|
||||
pub mod dedup;
|
||||
pub mod environment;
|
||||
pub mod harness;
|
||||
pub mod launcher;
|
||||
pub mod measure;
|
||||
pub mod media;
|
||||
pub mod metrics;
|
||||
pub mod phase;
|
||||
pub mod rpc;
|
||||
pub mod task;
|
||||
|
|
@ -37,5 +42,8 @@ pub mod worker;
|
|||
pub mod types;
|
||||
pub use fly_session_types;
|
||||
|
||||
pub use coordinator::{Coordinator, DispatchOrder, Injections, SessionFailure, StepReport};
|
||||
pub use coordinator::{
|
||||
Coordinator, Deadlines, DispatchOrder, Injections, ResolutionEnd, SessionFailure, StepReport,
|
||||
};
|
||||
pub use launcher::{ExecutionMode, Launcher, ReapOutcome, ThreadBudget, Via};
|
||||
pub use phase::{Phase, PhaseMachine};
|
||||
|
|
|
|||
488
services/flysim/crates/fly-session/src/measure.rs
Normal file
488
services/flysim/crates/fly-session/src/measure.rs
Normal file
|
|
@ -0,0 +1,488 @@
|
|||
//! The execution-mode comparison the implementation guide's section 5 asks for.
|
||||
//!
|
||||
//! It runs the same synthetic session in each execution mode, at one, two and four agents,
|
||||
//! and reports the thread allocation, the RPC and critical-path percentiles, the memory peaks
|
||||
//! and the router's owner, collection and queue counters.
|
||||
//!
|
||||
//! **These are local synthetic timings on one machine, and no host capacity claim follows
|
||||
//! from any of them.** They exist so the three modes can be compared with each other: the
|
||||
//! question this slice has to answer is what a process boundary costs, not how fast anything
|
||||
//! is. Pacing is switched off for the run, so a transition follows the one before it as fast
|
||||
//! as the participants answer and the samples are work rather than sleep.
|
||||
//!
|
||||
//! **Every row runs in a process of its own.** The coordinator's memory figure is a peak --
|
||||
//! `VmHWM` never falls -- so several rows sharing one process would each report where that
|
||||
//! process had already been rather than what its own mode costs, and the column would order
|
||||
//! itself by row position instead of by mode. The parent spawns one `measure-row` child per
|
||||
//! row and reads its result back, so each figure belongs to the row that produced it.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
|
||||
|
||||
use serde_json::{Value, json};
|
||||
|
||||
use crate::coordinator::DispatchOrder;
|
||||
use crate::harness::{AgentSpec, HarnessConfig, SessionHarness, Via};
|
||||
use crate::launcher::{ExecutionMode, flags};
|
||||
use crate::metrics::{Percentiles, physical_cores};
|
||||
|
||||
/// What to compare.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct MeasureConfig {
|
||||
/// Transitions per run, after the warm-up ones.
|
||||
pub steps: u64,
|
||||
/// Transitions run before sampling starts, so first-call costs are not in the samples.
|
||||
pub warmup_steps: u64,
|
||||
pub agent_counts: Vec<usize>,
|
||||
pub modes: Vec<ExecutionMode>,
|
||||
/// The within-agent worker count each agent asks its launcher for.
|
||||
pub worker_threads: usize,
|
||||
}
|
||||
|
||||
impl Default for MeasureConfig {
|
||||
fn default() -> MeasureConfig {
|
||||
MeasureConfig {
|
||||
steps: 200,
|
||||
warmup_steps: 10,
|
||||
agent_counts: vec![1, 2, 4],
|
||||
modes: ExecutionMode::all().to_vec(),
|
||||
worker_threads: 1,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The highest value each router counter reached while the transitions ran.
|
||||
#[derive(Debug, Default)]
|
||||
struct Peaks {
|
||||
owners: AtomicU64,
|
||||
roots: AtomicU64,
|
||||
queued: AtomicU64,
|
||||
sealed: AtomicU64,
|
||||
store_bytes: AtomicU64,
|
||||
}
|
||||
|
||||
impl Peaks {
|
||||
fn observe(&self, stats: &flybus::RouterStats) {
|
||||
raise(&self.owners, stats.owners as u64);
|
||||
raise(&self.roots, stats.artifact_roots);
|
||||
raise(&self.queued, stats.queued as u64);
|
||||
raise(&self.sealed, stats.sealed_artifacts as u64);
|
||||
raise(&self.store_bytes, stats.store_bytes);
|
||||
}
|
||||
|
||||
fn read(&self) -> (usize, u64, usize, usize, u64) {
|
||||
(
|
||||
self.owners.load(Ordering::Relaxed) as usize,
|
||||
self.roots.load(Ordering::Relaxed),
|
||||
self.queued.load(Ordering::Relaxed) as usize,
|
||||
self.sealed.load(Ordering::Relaxed) as usize,
|
||||
self.store_bytes.load(Ordering::Relaxed),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
fn raise(slot: &AtomicU64, value: u64) {
|
||||
slot.fetch_max(value, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
/// One measured composition.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct Row {
|
||||
pub mode: ExecutionMode,
|
||||
pub agents: usize,
|
||||
pub worker_threads: usize,
|
||||
pub physical_cores: usize,
|
||||
pub budget_total: usize,
|
||||
pub budget_used: usize,
|
||||
pub steps: u64,
|
||||
/// `Agent.Prepare`, over every agent and every transition.
|
||||
pub prepare: Percentiles,
|
||||
pub commit: Percentiles,
|
||||
pub advance: Percentiles,
|
||||
/// `Worker.Status`: an RPC with no domain work behind it, so it is the router and
|
||||
/// transport floor rather than a measure of the worker.
|
||||
pub status: Percentiles,
|
||||
/// One whole transition, pacing excluded: the critical path.
|
||||
pub step: Percentiles,
|
||||
pub coordinator_peak_rss_kib: u64,
|
||||
/// The sum of the peak resident sets of the participants with processes of their own.
|
||||
pub participants_peak_rss_kib: u64,
|
||||
pub owners_max: usize,
|
||||
pub owners_final: usize,
|
||||
pub artifact_roots_max: u64,
|
||||
pub queued_max: usize,
|
||||
pub sealed_max: usize,
|
||||
pub sealed_final: usize,
|
||||
pub store_bytes_max: u64,
|
||||
pub store_bytes_final: u64,
|
||||
/// Frames this run actually observed, counted from the behaviour trace: every sensory
|
||||
/// view of every transition, plus the one the environment sealed for boundary zero.
|
||||
///
|
||||
/// Counted rather than calculated, so a backend that sealed two frames per boundary would
|
||||
/// show up here instead of being hidden by arithmetic.
|
||||
pub frames_observed: u64,
|
||||
}
|
||||
|
||||
impl Row {
|
||||
/// Frames the store collected: observed, minus the ones still owned at the end.
|
||||
pub fn collected(&self) -> u64 {
|
||||
self.frames_observed.saturating_sub(self.sealed_final as u64)
|
||||
}
|
||||
|
||||
/// The row as one JSON object, for the child that measured it to hand back.
|
||||
pub fn to_json(&self) -> Value {
|
||||
let p = |x: &Percentiles| {
|
||||
json!({"count": x.count, "p50": x.p50_ns, "p95": x.p95_ns, "p99": x.p99_ns,
|
||||
"max": x.max_ns})
|
||||
};
|
||||
json!({
|
||||
"mode": self.mode.label(),
|
||||
"agents": self.agents,
|
||||
"workerThreads": self.worker_threads,
|
||||
"physicalCores": self.physical_cores,
|
||||
"budgetTotal": self.budget_total,
|
||||
"budgetUsed": self.budget_used,
|
||||
"steps": self.steps,
|
||||
"prepare": p(&self.prepare),
|
||||
"commit": p(&self.commit),
|
||||
"advance": p(&self.advance),
|
||||
"status": p(&self.status),
|
||||
"step": p(&self.step),
|
||||
"coordinatorPeakRssKib": self.coordinator_peak_rss_kib,
|
||||
"participantsPeakRssKib": self.participants_peak_rss_kib,
|
||||
"ownersMax": self.owners_max,
|
||||
"ownersFinal": self.owners_final,
|
||||
"artifactRootsMax": self.artifact_roots_max,
|
||||
"queuedMax": self.queued_max,
|
||||
"sealedMax": self.sealed_max,
|
||||
"sealedFinal": self.sealed_final,
|
||||
"storeBytesMax": self.store_bytes_max,
|
||||
"storeBytesFinal": self.store_bytes_final,
|
||||
"framesObserved": self.frames_observed,
|
||||
})
|
||||
}
|
||||
|
||||
/// Reads back what a `measure-row` child printed.
|
||||
pub fn from_json(value: &Value) -> Result<Row, String> {
|
||||
let u = |name: &str| -> Result<u64, String> {
|
||||
value
|
||||
.get(name)
|
||||
.and_then(Value::as_u64)
|
||||
.ok_or_else(|| format!("measure row: {name} is missing or not a number"))
|
||||
};
|
||||
let p = |name: &str| -> Result<Percentiles, String> {
|
||||
let v = value
|
||||
.get(name)
|
||||
.ok_or_else(|| format!("measure row: {name} is missing"))?;
|
||||
let f = |k: &str| v.get(k).and_then(Value::as_u64).unwrap_or_default();
|
||||
Ok(Percentiles {
|
||||
count: f("count") as usize,
|
||||
p50_ns: f("p50"),
|
||||
p95_ns: f("p95"),
|
||||
p99_ns: f("p99"),
|
||||
max_ns: f("max"),
|
||||
})
|
||||
};
|
||||
let mode = match value.get("mode").and_then(Value::as_str) {
|
||||
Some("in-process") => ExecutionMode::InProcess,
|
||||
Some("thread") => ExecutionMode::Thread,
|
||||
Some("process") => ExecutionMode::Process,
|
||||
other => return Err(format!("measure row: unknown mode {other:?}")),
|
||||
};
|
||||
Ok(Row {
|
||||
mode,
|
||||
agents: u("agents")? as usize,
|
||||
worker_threads: u("workerThreads")? as usize,
|
||||
physical_cores: u("physicalCores")? as usize,
|
||||
budget_total: u("budgetTotal")? as usize,
|
||||
budget_used: u("budgetUsed")? as usize,
|
||||
steps: u("steps")?,
|
||||
prepare: p("prepare")?,
|
||||
commit: p("commit")?,
|
||||
advance: p("advance")?,
|
||||
status: p("status")?,
|
||||
step: p("step")?,
|
||||
coordinator_peak_rss_kib: u("coordinatorPeakRssKib")?,
|
||||
participants_peak_rss_kib: u("participantsPeakRssKib")?,
|
||||
owners_max: u("ownersMax")? as usize,
|
||||
owners_final: u("ownersFinal")? as usize,
|
||||
artifact_roots_max: u("artifactRootsMax")?,
|
||||
queued_max: u("queuedMax")? as usize,
|
||||
sealed_max: u("sealedMax")? as usize,
|
||||
sealed_final: u("sealedFinal")? as usize,
|
||||
store_bytes_max: u("storeBytesMax")?,
|
||||
store_bytes_final: u("storeBytesFinal")?,
|
||||
frames_observed: u("framesObserved")?,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// Runs the comparison, one child process per row.
|
||||
///
|
||||
/// `program` is this crate's binary; each row is measured by a `measure-row` invocation of it
|
||||
/// so that the row's memory peak is its own and not the accumulated peak of the rows before
|
||||
/// it. The order of the rows therefore cannot change any of their numbers.
|
||||
pub fn run(config: &MeasureConfig, program: &std::path::Path) -> Result<Vec<Row>, String> {
|
||||
let mut rows = Vec::new();
|
||||
for mode in &config.modes {
|
||||
for agents in &config.agent_counts {
|
||||
rows.push(row_in_a_child(config, program, *mode, *agents)?);
|
||||
}
|
||||
}
|
||||
Ok(rows)
|
||||
}
|
||||
|
||||
fn row_in_a_child(
|
||||
config: &MeasureConfig,
|
||||
program: &std::path::Path,
|
||||
mode: ExecutionMode,
|
||||
agents: usize,
|
||||
) -> Result<Row, String> {
|
||||
let output = std::process::Command::new(program)
|
||||
.arg("measure-row")
|
||||
.arg(format!("--{}", flags::MODE))
|
||||
.arg(mode.label())
|
||||
.arg(format!("--{}", flags::AGENTS))
|
||||
.arg(agents.to_string())
|
||||
.arg(format!("--{}", flags::STEPS))
|
||||
.arg(config.steps.to_string())
|
||||
.arg(format!("--{}", flags::WARMUP_STEPS))
|
||||
.arg(config.warmup_steps.to_string())
|
||||
.arg(format!("--{}", flags::WORKER_THREADS))
|
||||
.arg(config.worker_threads.to_string())
|
||||
.stdin(std::process::Stdio::null())
|
||||
.output()
|
||||
.map_err(|e| format!("measure-row {}: {e}", mode.label()))?;
|
||||
if !output.status.success() {
|
||||
return Err(format!(
|
||||
"measure-row {} {agents}: exited {}: {}",
|
||||
mode.label(),
|
||||
output.status,
|
||||
String::from_utf8_lossy(&output.stderr).trim()
|
||||
));
|
||||
}
|
||||
let text = String::from_utf8_lossy(&output.stdout);
|
||||
let line = text
|
||||
.lines()
|
||||
.rev()
|
||||
.find(|l| l.trim_start().starts_with('{'))
|
||||
.ok_or_else(|| format!("measure-row {}: no row on stdout", mode.label()))?;
|
||||
let value: Value = serde_json::from_str(line)
|
||||
.map_err(|e| format!("measure-row {}: unreadable row: {e}", mode.label()))?;
|
||||
Row::from_json(&value)
|
||||
}
|
||||
|
||||
/// Measures exactly one row, in this process. The `measure-row` subcommand's body.
|
||||
pub async fn one(
|
||||
config: &MeasureConfig,
|
||||
mode: ExecutionMode,
|
||||
agents: usize,
|
||||
) -> Result<Row, String> {
|
||||
let dir = std::env::temp_dir().join(format!(
|
||||
"fly-session-measure-{}-{agents}-{}",
|
||||
mode.label(),
|
||||
std::process::id()
|
||||
));
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
std::fs::create_dir_all(&dir).map_err(|e| format!("{}: {e}", dir.display()))?;
|
||||
let result = measure_in(config, mode, agents, &dir).await;
|
||||
let _ = std::fs::remove_dir_all(&dir);
|
||||
result
|
||||
}
|
||||
|
||||
async fn measure_in(
|
||||
config: &MeasureConfig,
|
||||
mode: ExecutionMode,
|
||||
agents: usize,
|
||||
dir: &std::path::Path,
|
||||
) -> Result<Row, String> {
|
||||
let specs: Vec<AgentSpec> = (0..agents)
|
||||
.map(|i| AgentSpec {
|
||||
worker_threads: config.worker_threads,
|
||||
..AgentSpec::new(&format!("fly-{}", (b'a' + i as u8) as char), &format!("p{}", i + 1), 7 + i as i32)
|
||||
})
|
||||
.collect();
|
||||
let harness_config = HarnessConfig {
|
||||
agents: specs,
|
||||
mode,
|
||||
..HarnessConfig::default()
|
||||
};
|
||||
let budget = harness_config.budget().map_err(|e| e.to_string())?;
|
||||
let (budget_total, budget_used_floor) = (budget.total(), harness_config.required_threads());
|
||||
let mut harness = SessionHarness::start(Via::Unix, dir, harness_config)
|
||||
.await
|
||||
.map_err(|e| format!("{}: start: {}", mode.label(), e.message))?;
|
||||
harness.coordinator.dispatch = DispatchOrder::Concurrent;
|
||||
harness.coordinator.disable_pacing();
|
||||
harness
|
||||
.coordinator
|
||||
.bootstrap()
|
||||
.await
|
||||
.map_err(|e| format!("{}: bootstrap: {e}", mode.label()))?;
|
||||
// The warm-up transitions pay the first-call costs; their samples are then discarded.
|
||||
harness
|
||||
.coordinator
|
||||
.run(config.warmup_steps)
|
||||
.await
|
||||
.map_err(|e| format!("{}: warm-up: {e}", mode.label()))?;
|
||||
harness.coordinator.metrics.clear();
|
||||
|
||||
// The router's counters are sampled *while* transitions run, not between them: a queue
|
||||
// that is empty at every committed boundary says nothing about whether it stayed bounded
|
||||
// during the transaction, which is the thing section 4 asks about.
|
||||
let peaks = Arc::new(Peaks::default());
|
||||
let sampling = Arc::new(AtomicBool::new(true));
|
||||
let sampler = tokio::spawn({
|
||||
let router = harness.router().clone();
|
||||
let peaks = peaks.clone();
|
||||
let sampling = sampling.clone();
|
||||
async move {
|
||||
while sampling.load(Ordering::Relaxed) {
|
||||
peaks.observe(&router.stats());
|
||||
tokio::time::sleep(std::time::Duration::from_micros(200)).await;
|
||||
}
|
||||
peaks.observe(&router.stats());
|
||||
}
|
||||
});
|
||||
|
||||
let environment = harness.environment_id();
|
||||
let mut status = crate::metrics::Metrics::default();
|
||||
for _ in 0..config.steps {
|
||||
harness
|
||||
.coordinator
|
||||
.step()
|
||||
.await
|
||||
.map_err(|e| format!("{}: step: {e}", mode.label()))?;
|
||||
// One status call per transition: an RPC the worker answers from a cell rather than
|
||||
// from its endpoint, so it is the router-and-transport floor the domain calls sit on.
|
||||
let started = std::time::Instant::now();
|
||||
let _ = harness.launcher.health_check(&environment).await;
|
||||
status.record("Worker.Status", started.elapsed());
|
||||
}
|
||||
sampling.store(false, Ordering::Relaxed);
|
||||
let _ = sampler.await;
|
||||
let (owners_max, roots_max, queued_max, sealed_max, store_bytes_max) = peaks.read();
|
||||
|
||||
let metrics = &harness.coordinator.metrics;
|
||||
let zero = Percentiles::default();
|
||||
let row = Row {
|
||||
mode,
|
||||
agents,
|
||||
worker_threads: config.worker_threads,
|
||||
physical_cores: physical_cores(),
|
||||
budget_total,
|
||||
budget_used: budget_used_floor,
|
||||
steps: config.steps,
|
||||
prepare: metrics.percentiles("Agent.Prepare").unwrap_or(zero),
|
||||
commit: metrics.percentiles("Agent.Commit").unwrap_or(zero),
|
||||
advance: metrics.percentiles("Environment.Advance").unwrap_or(zero),
|
||||
status: status.percentiles("Worker.Status").unwrap_or(zero),
|
||||
step: metrics.percentiles("step").unwrap_or(zero),
|
||||
coordinator_peak_rss_kib: crate::metrics::peak_rss_kib().unwrap_or_default(),
|
||||
participants_peak_rss_kib: harness
|
||||
.launcher
|
||||
.peak_rss_kib()
|
||||
.iter()
|
||||
.filter(|(who, _)| who.as_str() != "coordinator")
|
||||
.map(|(_, kib)| *kib)
|
||||
.sum(),
|
||||
owners_max,
|
||||
owners_final: harness.router_stats().owners,
|
||||
artifact_roots_max: roots_max,
|
||||
queued_max,
|
||||
sealed_max,
|
||||
sealed_final: harness.router_stats().sealed_artifacts,
|
||||
store_bytes_max,
|
||||
store_bytes_final: harness.router_stats().store_bytes,
|
||||
// Counted from the behaviour trace: every sensory view of every transition this run
|
||||
// recorded, plus the frame the environment sealed for boundary zero.
|
||||
frames_observed: 1 + harness
|
||||
.coordinator
|
||||
.trace
|
||||
.transitions
|
||||
.iter()
|
||||
.map(|t| t.behaviour.observation_boundaries.len() as u64)
|
||||
.sum::<u64>(),
|
||||
};
|
||||
harness.shutdown().await;
|
||||
Ok(row)
|
||||
}
|
||||
|
||||
/// The measurement table, as Markdown.
|
||||
pub fn table(rows: &[Row]) -> String {
|
||||
let mut out = String::new();
|
||||
out.push_str(
|
||||
"Local synthetic timings on one machine. Not a capacity claim, and not a latency goal.\n\n",
|
||||
);
|
||||
if let Some(first) = rows.first() {
|
||||
out.push_str(&format!(
|
||||
"Physical cores: {}. Coordinator reservation: 1 thread. Every row was measured in \
|
||||
a process of its own, so no figure depends on the order of the rows.\n\n",
|
||||
first.physical_cores
|
||||
));
|
||||
}
|
||||
out.push_str(
|
||||
"| mode | agents | threads/agent | budget used/total | Prepare p50/p95/p99 us | \
|
||||
Commit p50/p95/p99 us | Advance p50/p95/p99 us | Status p50/p99 us | step p50/p95/p99 us |\n",
|
||||
);
|
||||
out.push_str("| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n");
|
||||
for r in rows {
|
||||
out.push_str(&format!(
|
||||
"| {} | {} | {} | {}/{} | {:.0}/{:.0}/{:.0} | {:.0}/{:.0}/{:.0} | \
|
||||
{:.0}/{:.0}/{:.0} | {:.0}/{:.0} | {:.0}/{:.0}/{:.0} |\n",
|
||||
r.mode.label(),
|
||||
r.agents,
|
||||
r.worker_threads,
|
||||
r.budget_used,
|
||||
r.budget_total,
|
||||
r.prepare.p50_us(),
|
||||
r.prepare.p95_us(),
|
||||
r.prepare.p99_us(),
|
||||
r.commit.p50_us(),
|
||||
r.commit.p95_us(),
|
||||
r.commit.p99_us(),
|
||||
r.advance.p50_us(),
|
||||
r.advance.p95_us(),
|
||||
r.advance.p99_us(),
|
||||
r.status.p50_us(),
|
||||
r.status.p99_us(),
|
||||
r.step.p50_us(),
|
||||
r.step.p95_us(),
|
||||
r.step.p99_us(),
|
||||
));
|
||||
}
|
||||
out.push('\n');
|
||||
out.push_str(
|
||||
"| mode | agents | coordinator peak RSS KiB | participant peak RSS KiB | owners max/final | \
|
||||
roots max | queued max | sealed max/final | store bytes max/final | frames observed/collected |\n",
|
||||
);
|
||||
out.push_str("| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n");
|
||||
for r in rows {
|
||||
out.push_str(&format!(
|
||||
"| {} | {} | {} | {} | {}/{} | {} | {} | {}/{} | {}/{} | {}/{} |\n",
|
||||
r.mode.label(),
|
||||
r.agents,
|
||||
r.coordinator_peak_rss_kib,
|
||||
r.participants_peak_rss_kib,
|
||||
r.owners_max,
|
||||
r.owners_final,
|
||||
r.artifact_roots_max,
|
||||
r.queued_max,
|
||||
r.sealed_max,
|
||||
r.sealed_final,
|
||||
r.store_bytes_max,
|
||||
r.store_bytes_final,
|
||||
r.frames_observed,
|
||||
r.collected(),
|
||||
));
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// The rows keyed by mode and agent count, for a caller that wants one of them.
|
||||
pub fn by_composition(rows: &[Row]) -> BTreeMap<(String, usize), Row> {
|
||||
rows.iter()
|
||||
.map(|r| ((r.mode.label().to_owned(), r.agents), r.clone()))
|
||||
.collect()
|
||||
}
|
||||
837
services/flysim/crates/fly-session/src/media.rs
Normal file
837
services/flysim/crates/fly-session/src/media.rs
Normal file
|
|
@ -0,0 +1,837 @@
|
|||
//! Native observations and the presentation handoff: the MEDIA-01 slice.
|
||||
//!
|
||||
//! Everything here sits on top of the bus `ArtifactRef` and its ownership rules. There is no
|
||||
//! second buffer system: an environment allocates, writes and seals one immutable object per
|
||||
//! boundary, the coordinator forwards that one owned handle to every agent and to publication,
|
||||
//! and a spectator reads it through an ordinary latest subscription.
|
||||
//!
|
||||
//! The module owns four things:
|
||||
//!
|
||||
//! 1. Production. [`ViewPipeline`] renders one native frame per boundary and hands out the
|
||||
//! frame a declared `observationDelaySteps` requires, so a pipeline delay is a real queue
|
||||
//! rather than a number in a descriptor. [`AudioSource`] produces one chunk per boundary
|
||||
//! with an exact rational sample budget.
|
||||
//! 2. Acceptance. [`check_required_views`] and [`AudioTimelines`] are the coordinator's Phase C
|
||||
//! media checks: a required sensory view must exist at exactly the producing boundary its
|
||||
//! declared delay implies, and audio chunks cannot overlap or go backwards inside an epoch.
|
||||
//! 3. Consumption. [`Spectator`] is a presentation-side consumer on a latest subscription with
|
||||
//! finite credits, and [`detach_frame`] is the renderer that keeps its handle after the
|
||||
//! message is gone.
|
||||
//! 4. Identity. [`AssetRegistry`] holds installed persistent content named by `AssetRef`.
|
||||
//! Importing an asset produces a *new* transient artifact; the two identities never convert.
|
||||
//!
|
||||
//! Resizing, overlays, compositing, mixing, encoding and streaming are not here and are not
|
||||
//! anywhere else in this crate: they belong to the application's presentation layer.
|
||||
|
||||
use std::collections::{BTreeMap, VecDeque};
|
||||
use std::io::Write;
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicU64, Ordering};
|
||||
|
||||
use fly_session_types::media::AudioTimeline;
|
||||
|
||||
// `crate::types` is this crate's facade over the shared `fly-session-types` crate; the glob
|
||||
// keeps the contract's own names in sight instead of restating them.
|
||||
use crate::types::*;
|
||||
|
||||
/// The content type of a native RGBA8 frame. Top-left origin, no padded rows.
|
||||
pub const FRAME_CONTENT_TYPE: &str = "image/x-rgba8";
|
||||
|
||||
/// The content type of a native audio chunk: interleaved little-endian f32.
|
||||
pub const AUDIO_CONTENT_TYPE: &str = "audio/x-f32le";
|
||||
|
||||
/// The attachment name one view's pixels travel under.
|
||||
pub fn view_attachment(view_id: &str) -> String {
|
||||
format!("view.{view_id}")
|
||||
}
|
||||
|
||||
/// The attachment name one audio stream's samples travel under.
|
||||
pub fn audio_attachment(stream_id: &str) -> String {
|
||||
format!("audio.{stream_id}")
|
||||
}
|
||||
|
||||
fn store_error(what: &str, message: &str) -> DomainError {
|
||||
DomainError::new(
|
||||
ErrorCode::BackendFailure,
|
||||
format!("{what}: {message}"),
|
||||
MutationCertainty::Applied,
|
||||
)
|
||||
}
|
||||
|
||||
fn media_error(message: impl std::fmt::Display) -> DomainError {
|
||||
// A world that advanced without usable media leaves the transition's certainty unknown:
|
||||
// the mutation happened, the observation of it did not.
|
||||
DomainError::new(ErrorCode::BufferInvalid, message, MutationCertainty::Unknown)
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------------------------
|
||||
// Production
|
||||
|
||||
/// How many frames a producer has actually rendered.
|
||||
///
|
||||
/// A shared counter, so a test can prove that forwarding one image to several recipients
|
||||
/// renders it once.
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub struct RenderCounter(Arc<AtomicU64>);
|
||||
|
||||
impl RenderCounter {
|
||||
pub fn new() -> RenderCounter {
|
||||
RenderCounter::default()
|
||||
}
|
||||
|
||||
pub fn bump(&self) {
|
||||
self.0.fetch_add(1, Ordering::SeqCst);
|
||||
}
|
||||
|
||||
pub fn count(&self) -> u64 {
|
||||
self.0.load(Ordering::SeqCst)
|
||||
}
|
||||
}
|
||||
|
||||
/// One native frame of the counter arena: a real synthetic pattern, not a constant fill.
|
||||
///
|
||||
/// Top-left RGBA8 with `rowStride` exactly `4 x width` and no padded rows, which is the only
|
||||
/// pixel layout v1 has. The red channel carries the world counter, so a reader that samples
|
||||
/// one pixel still reads the world; green is a horizontal ramp and blue a vertical ramp with
|
||||
/// a one-column bar that walks with the boundary, so consecutive frames differ.
|
||||
pub fn arena_frame(descriptor: &ViewDescriptor, counter: i64, boundary: u64) -> Vec<u8> {
|
||||
let width = descriptor.width;
|
||||
let height = descriptor.height;
|
||||
let stride = descriptor.row_stride as usize;
|
||||
let mut out = vec![0u8; stride * height as usize];
|
||||
let counter_byte = (counter & 0xff) as u8;
|
||||
let bar = boundary % width;
|
||||
for y in 0..height {
|
||||
let row = y as usize * stride;
|
||||
for x in 0..width {
|
||||
let p = row + x as usize * 4;
|
||||
let ramp_x = if width > 1 {
|
||||
(x * 255 / (width - 1)) as u8
|
||||
} else {
|
||||
0
|
||||
};
|
||||
let ramp_y = if height > 1 {
|
||||
(y * 255 / (height - 1)) as u8
|
||||
} else {
|
||||
0
|
||||
};
|
||||
out[p] = counter_byte;
|
||||
out[p + 1] = ramp_x;
|
||||
out[p + 2] = if x == bar { 255 } else { ramp_y };
|
||||
out[p + 3] = 255;
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// One view's production pipeline: render at every boundary, deliver with the declared delay.
|
||||
///
|
||||
/// `observationDelaySteps` is a real queue here. At boundary `b` the required frame is the one
|
||||
/// produced at `max(0, b - delay)`, so a declared delay of two repeats `O[0]` at boundaries 0,
|
||||
/// 1 and 2 -- the bootstrap repetition the contract allows -- and then advances one frame per
|
||||
/// boundary. Nothing beyond that is retained: an older frame is dropped, so a later boundary
|
||||
/// cannot be served an arbitrary stale image.
|
||||
pub struct ViewPipeline {
|
||||
descriptor: ViewDescriptor,
|
||||
frames: VecDeque<(u64, flybus::Artifact)>,
|
||||
renders: RenderCounter,
|
||||
}
|
||||
|
||||
impl ViewPipeline {
|
||||
pub fn new(descriptor: ViewDescriptor, renders: RenderCounter) -> ViewPipeline {
|
||||
ViewPipeline {
|
||||
descriptor,
|
||||
frames: VecDeque::new(),
|
||||
renders,
|
||||
}
|
||||
}
|
||||
|
||||
pub fn descriptor(&self) -> &ViewDescriptor {
|
||||
&self.descriptor
|
||||
}
|
||||
|
||||
/// Seals one immutable frame for `boundary` and files it under its producing boundary.
|
||||
pub async fn render(
|
||||
&mut self,
|
||||
client: &flybus::Client,
|
||||
boundary: u64,
|
||||
counter: i64,
|
||||
) -> DomainResult<()> {
|
||||
let bytes = arena_frame(&self.descriptor, counter, boundary);
|
||||
let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?;
|
||||
self.renders.bump();
|
||||
self.frames.push_back((boundary, artifact));
|
||||
// Keep exactly the frames a declared delay can still require.
|
||||
while self.frames.len() > self.descriptor.observation_delay_steps as usize + 1 {
|
||||
self.frames.pop_front();
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Seals a frame of the wrong length, which is what a broken backend produces. The
|
||||
/// reference it returns describes the artifact honestly, so the shape check is the thing
|
||||
/// under test rather than a lie in the payload.
|
||||
pub async fn render_truncated(
|
||||
&mut self,
|
||||
client: &flybus::Client,
|
||||
boundary: u64,
|
||||
counter: i64,
|
||||
) -> DomainResult<()> {
|
||||
let mut bytes = arena_frame(&self.descriptor, counter, boundary);
|
||||
bytes.truncate(bytes.len() - self.descriptor.row_stride as usize);
|
||||
let artifact = seal(client, FRAME_CONTENT_TYPE, &bytes).await?;
|
||||
self.renders.bump();
|
||||
self.frames.push_back((boundary, artifact));
|
||||
while self.frames.len() > self.descriptor.observation_delay_steps as usize + 2 {
|
||||
self.frames.pop_front();
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// The view reference and the owned handle a required sensory view has at `boundary`.
|
||||
pub fn at(&self, boundary: u64) -> Option<(ViewRef, flybus::Artifact)> {
|
||||
let produced = self.descriptor.required_produced_step(boundary);
|
||||
self.frame_produced_at(produced)
|
||||
}
|
||||
|
||||
/// The frame produced at exactly `produced`, if it is still retained.
|
||||
pub fn frame_produced_at(&self, produced: u64) -> Option<(ViewRef, flybus::Artifact)> {
|
||||
self.frames
|
||||
.iter()
|
||||
.find(|(step, _)| *step == produced)
|
||||
.map(|(step, artifact)| {
|
||||
(
|
||||
ViewRef {
|
||||
view_id: self.descriptor.view_id.clone(),
|
||||
produced_step: *step,
|
||||
pixels: artifact.reference().clone(),
|
||||
},
|
||||
artifact.clone(),
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
/// How many boundaries this pipeline has rendered.
|
||||
pub fn renders(&self) -> u64 {
|
||||
self.renders.count()
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
/// One audio stream's production: an exact sample budget and a deterministic waveform.
|
||||
///
|
||||
/// The number of frames in a step is `sampleRate x stepDuration`, accumulated as a rational so
|
||||
/// a cadence that does not divide the sample rate never drifts: 8 kHz at 60 Hz produces
|
||||
/// 133, 133, 134, ... and the sum is exact at every boundary. The waveform is integer-phase
|
||||
/// arithmetic only, so a `fixed-build` environment produces the same bytes on every run.
|
||||
pub struct AudioSource {
|
||||
descriptor: AudioDescriptor,
|
||||
/// The unconsumed fraction of a frame, over `denominator`.
|
||||
accumulator: u128,
|
||||
denominator: u128,
|
||||
next_sample: u64,
|
||||
phase: u64,
|
||||
chunks: u64,
|
||||
discontinuous: bool,
|
||||
}
|
||||
|
||||
impl AudioSource {
|
||||
/// A fresh episode, whose first chunk starts at the configured audio origin.
|
||||
pub fn new(descriptor: AudioDescriptor, origin: u64) -> AudioSource {
|
||||
AudioSource {
|
||||
descriptor,
|
||||
accumulator: 0,
|
||||
denominator: 1,
|
||||
next_sample: origin,
|
||||
phase: 0,
|
||||
chunks: 0,
|
||||
discontinuous: false,
|
||||
}
|
||||
}
|
||||
|
||||
/// A new epoch after a restore: the sample position is preserved and the first chunk of
|
||||
/// this epoch marks a discontinuity.
|
||||
pub fn restored_at(descriptor: AudioDescriptor, sample: u64) -> AudioSource {
|
||||
let mut source = AudioSource::new(descriptor, sample);
|
||||
source.discontinuous = true;
|
||||
source
|
||||
}
|
||||
|
||||
pub fn descriptor(&self) -> &AudioDescriptor {
|
||||
&self.descriptor
|
||||
}
|
||||
|
||||
pub fn next_sample(&self) -> u64 {
|
||||
self.next_sample
|
||||
}
|
||||
|
||||
pub fn chunks(&self) -> u64 {
|
||||
self.chunks
|
||||
}
|
||||
|
||||
/// The exact number of sample frames one step of `step` nanoseconds contains.
|
||||
///
|
||||
/// The remainder is kept, never rounded: the accumulator is integer arithmetic over the
|
||||
/// common denominator `stepDenominator x 1e9`.
|
||||
pub fn frames_for_step(&mut self, step: &RationalNs) -> DomainResult<u64> {
|
||||
let denominator = u128::from(step.denominator)
|
||||
.checked_mul(1_000_000_000)
|
||||
.ok_or_else(|| DomainError::invalid("audio: the step denominator overflows"))?;
|
||||
if self.denominator != denominator {
|
||||
// A cadence change would need a new epoch; carrying a remainder across one would
|
||||
// be a silent resample.
|
||||
if self.chunks > 0 {
|
||||
return Err(DomainError::invalid(
|
||||
"audio: the cadence changed inside an epoch",
|
||||
));
|
||||
}
|
||||
self.denominator = denominator;
|
||||
}
|
||||
let per_step = u128::from(self.descriptor.sample_rate)
|
||||
.checked_mul(u128::from(step.numerator))
|
||||
.ok_or_else(|| DomainError::invalid("audio: the sample budget overflows"))?;
|
||||
self.accumulator = self
|
||||
.accumulator
|
||||
.checked_add(per_step)
|
||||
.ok_or_else(|| DomainError::invalid("audio: the sample accumulator overflows"))?;
|
||||
let frames = self.accumulator / self.denominator;
|
||||
self.accumulator %= self.denominator;
|
||||
u64::try_from(frames).map_err(|_| DomainError::invalid("audio: too many frames in a step"))
|
||||
}
|
||||
|
||||
/// Produces one chunk covering exactly one step of world time.
|
||||
pub async fn produce(
|
||||
&mut self,
|
||||
client: &flybus::Client,
|
||||
step: &RationalNs,
|
||||
counter: i64,
|
||||
) -> DomainResult<(AudioRef, flybus::Artifact)> {
|
||||
let frames = self.frames_for_step(step)?;
|
||||
let bytes = self.samples(frames, counter);
|
||||
let artifact = seal(client, AUDIO_CONTENT_TYPE, &bytes).await?;
|
||||
let chunk = AudioRef {
|
||||
stream_id: self.descriptor.stream_id.clone(),
|
||||
first_sample: self.next_sample,
|
||||
sample_frames: frames,
|
||||
samples: artifact.reference().clone(),
|
||||
discontinuity: self.discontinuous && self.chunks == 0,
|
||||
};
|
||||
self.next_sample = self
|
||||
.next_sample
|
||||
.checked_add(frames)
|
||||
.ok_or_else(|| DomainError::invalid("audio: the sample position overflows"))?;
|
||||
self.chunks += 1;
|
||||
Ok((chunk, artifact))
|
||||
}
|
||||
|
||||
/// A deterministic triangle wave whose pitch follows the world counter, interleaved across
|
||||
/// the declared channels. Every sample is finite by construction.
|
||||
fn samples(&mut self, frames: u64, counter: i64) -> Vec<u8> {
|
||||
let rate = self.descriptor.sample_rate;
|
||||
let channels = self.descriptor.channels;
|
||||
let step = 220 + (counter.rem_euclid(8) as u64) * 55;
|
||||
let mut out = Vec::with_capacity((frames * channels * 4) as usize);
|
||||
for _ in 0..frames {
|
||||
self.phase = (self.phase + step) % rate;
|
||||
let position = self.phase as f32 / rate as f32;
|
||||
// 1 - 2|2p - 1| is a triangle in [-1, 1] built from exact IEEE operations.
|
||||
let value = 1.0 - 2.0 * (2.0 * position - 1.0).abs();
|
||||
for channel in 0..channels {
|
||||
let scaled = value * 0.25 / (channel + 1) as f32;
|
||||
out.extend_from_slice(&scaled.to_le_bytes());
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
}
|
||||
|
||||
/// Allocates, writes and seals one immutable artifact.
|
||||
async fn seal(
|
||||
client: &flybus::Client,
|
||||
content_type: &str,
|
||||
bytes: &[u8],
|
||||
) -> DomainResult<flybus::Artifact> {
|
||||
let mut writer = client
|
||||
.artifacts()
|
||||
.allocate(bytes.len() as u64, content_type)
|
||||
.await
|
||||
.map_err(|e| store_error("allocate", &e.message))?;
|
||||
writer
|
||||
.write_all(bytes)
|
||||
.map_err(|e| store_error("write", &e.to_string()))?;
|
||||
writer
|
||||
.seal()
|
||||
.await
|
||||
.map_err(|e| store_error("seal", &e.message))
|
||||
}
|
||||
|
||||
/// Splits one reply's attachments into the view handles and the audio handles.
|
||||
///
|
||||
/// Views are sensory data forwarded to agents; audio is presentation data that is published
|
||||
/// and never attached to a sensory input.
|
||||
pub fn split_attachments(
|
||||
artifacts: BTreeMap<String, flybus::Artifact>,
|
||||
) -> (
|
||||
BTreeMap<String, flybus::Artifact>,
|
||||
BTreeMap<String, flybus::Artifact>,
|
||||
) {
|
||||
let mut views = BTreeMap::new();
|
||||
let mut audio = BTreeMap::new();
|
||||
for (name, artifact) in artifacts {
|
||||
if name.starts_with("audio.") {
|
||||
audio.insert(name, artifact);
|
||||
} else {
|
||||
views.insert(name, artifact);
|
||||
}
|
||||
}
|
||||
(views, audio)
|
||||
}
|
||||
|
||||
/// The attachment names one environment's declared media travel under.
|
||||
pub fn attachment_names(descriptor: &EnvironmentDescriptor) -> Vec<String> {
|
||||
descriptor
|
||||
.views
|
||||
.iter()
|
||||
.map(|view| view_attachment(&view.view_id))
|
||||
.chain(
|
||||
descriptor
|
||||
.audio
|
||||
.iter()
|
||||
.map(|stream| audio_attachment(&stream.stream_id)),
|
||||
)
|
||||
.collect()
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------------------------
|
||||
// Acceptance
|
||||
|
||||
/// Every declared view must be present at exactly the producing boundary its delay implies.
|
||||
///
|
||||
/// A missing spectator frame is tolerable; a missing required sensory input is not. Neither is
|
||||
/// one that arrived from an older boundary than the declared delay allows: the transition
|
||||
/// fails instead of the session substituting whatever frame it happens to hold.
|
||||
pub fn check_required_views(
|
||||
descriptor: &EnvironmentDescriptor,
|
||||
observation: &WorldObservation,
|
||||
) -> DomainResult<()> {
|
||||
for view in &descriptor.views {
|
||||
let want = required_produced_step(view, observation.boundary);
|
||||
match observation
|
||||
.sensory_views
|
||||
.iter()
|
||||
.find(|given| given.view_id == view.view_id)
|
||||
{
|
||||
Some(given) if given.produced_step == want => {}
|
||||
Some(given) => {
|
||||
return Err(media_error(format!(
|
||||
"view {} came from boundary {}, and its declared delay of {} requires {want}",
|
||||
view.view_id, given.produced_step, view.observation_delay_steps
|
||||
)));
|
||||
}
|
||||
None => {
|
||||
return Err(media_error(format!(
|
||||
"required sensory view {} is missing",
|
||||
view.view_id
|
||||
)));
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Every declared audio stream produces exactly one chunk per transition.
|
||||
///
|
||||
/// The contract states the shape and the ordering of chunks, not whether one has to exist, so
|
||||
/// this is MEDIA-01's choice and it is deliberate: a session that tolerates a silently missing
|
||||
/// chunk cannot tell "this world produced no audio for this interval" from "the chunk was
|
||||
/// lost", and the second is the case the retention rules care about. Boundary 0 has no
|
||||
/// preceding interval and so carries no chunk.
|
||||
pub fn check_required_audio(
|
||||
descriptor: &EnvironmentDescriptor,
|
||||
observation: &WorldObservation,
|
||||
) -> DomainResult<()> {
|
||||
if observation.boundary == 0 {
|
||||
return Ok(());
|
||||
}
|
||||
for stream in &descriptor.audio {
|
||||
if !observation
|
||||
.audio
|
||||
.iter()
|
||||
.any(|chunk| chunk.stream_id == stream.stream_id)
|
||||
{
|
||||
return Err(media_error(format!(
|
||||
"declared audio stream {} produced no chunk for this transition",
|
||||
stream.stream_id
|
||||
)));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Every declared audio stream's chunk sequence, one timeline per stream and epoch.
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub struct AudioTimelines(BTreeMap<String, AudioTimeline>);
|
||||
|
||||
impl AudioTimelines {
|
||||
/// Fresh timelines for a new episode: every declared stream starts at origin zero.
|
||||
pub fn fresh(descriptor: &EnvironmentDescriptor) -> AudioTimelines {
|
||||
AudioTimelines(
|
||||
descriptor
|
||||
.audio
|
||||
.iter()
|
||||
.map(|stream| (stream.stream_id.clone(), AudioTimeline::fresh(stream, 0)))
|
||||
.collect(),
|
||||
)
|
||||
}
|
||||
|
||||
/// Timelines for a new epoch after a restore: each stream resumes at its preserved sample
|
||||
/// position, and each one's first chunk must mark a discontinuity.
|
||||
///
|
||||
/// A declared stream with no recorded position is an error. Resuming it at sample zero
|
||||
/// would restart the episode's audio clock silently, which is exactly the best-effort
|
||||
/// policy the restore rules refuse: crash restore *preserves* the sample position.
|
||||
pub fn restored(
|
||||
descriptor: &EnvironmentDescriptor,
|
||||
positions: &BTreeMap<String, u64>,
|
||||
) -> DomainResult<AudioTimelines> {
|
||||
let mut timelines = BTreeMap::new();
|
||||
for stream in &descriptor.audio {
|
||||
let at = positions.get(&stream.stream_id).copied().ok_or_else(|| {
|
||||
DomainError::before(
|
||||
ErrorCode::IncompatibleState,
|
||||
format!(
|
||||
"audio stream {} has no restored sample position",
|
||||
stream.stream_id
|
||||
),
|
||||
)
|
||||
})?;
|
||||
timelines.insert(stream.stream_id.clone(), AudioTimeline::restored_at(stream, at));
|
||||
}
|
||||
Ok(AudioTimelines(timelines))
|
||||
}
|
||||
|
||||
/// Accepts one observation's chunks. Unknown streams and out-of-sequence chunks fail.
|
||||
pub fn accept(
|
||||
&mut self,
|
||||
descriptor: &EnvironmentDescriptor,
|
||||
observation: &WorldObservation,
|
||||
) -> DomainResult<()> {
|
||||
for chunk in &observation.audio {
|
||||
let declared = descriptor
|
||||
.audio_stream(&chunk.stream_id)
|
||||
.ok_or_else(|| media_error(format!("audio stream {} is not declared", chunk.stream_id)))?;
|
||||
let timeline = self
|
||||
.0
|
||||
.get_mut(&chunk.stream_id)
|
||||
.ok_or_else(|| media_error(format!("audio stream {} has no timeline", chunk.stream_id)))?;
|
||||
timeline.accept(chunk, declared).map_err(media_error)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Where each stream's next chunk may start.
|
||||
pub fn positions(&self) -> BTreeMap<String, u64> {
|
||||
self.0
|
||||
.iter()
|
||||
.map(|(id, timeline)| (id.clone(), timeline.next_sample()))
|
||||
.collect()
|
||||
}
|
||||
|
||||
pub fn accepted(&self, stream_id: &str) -> u64 {
|
||||
self.0.get(stream_id).map_or(0, AudioTimeline::accepted)
|
||||
}
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------------------------
|
||||
// Consumption
|
||||
|
||||
/// What one agent actually sensed, recorded where a test can read it.
|
||||
///
|
||||
/// The fake agent is the only thing that reads the pixels, so this is how "one shared image
|
||||
/// reached both agents" is proved from the agents' side rather than from the producer's.
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub struct SensorLog(Arc<std::sync::Mutex<Vec<SensedView>>>);
|
||||
|
||||
/// One view an agent read: which boundary it was consumed at, which artifact it was, and the
|
||||
/// digest of the bytes the agent actually read.
|
||||
#[derive(Clone, Debug, PartialEq, Eq)]
|
||||
pub struct SensedView {
|
||||
pub boundary: u64,
|
||||
pub view_id: Id,
|
||||
pub artifact_id: String,
|
||||
pub produced_step: u64,
|
||||
pub digest: Digest,
|
||||
}
|
||||
|
||||
impl SensorLog {
|
||||
pub fn new() -> SensorLog {
|
||||
SensorLog::default()
|
||||
}
|
||||
|
||||
pub fn record(&self, view: SensedView) {
|
||||
self.0.lock().expect("the sensor log is never poisoned").push(view);
|
||||
}
|
||||
|
||||
pub fn entries(&self) -> Vec<SensedView> {
|
||||
self.0.lock().expect("the sensor log is never poisoned").clone()
|
||||
}
|
||||
|
||||
/// The artifacts this agent read, in order.
|
||||
pub fn artifact_ids(&self) -> Vec<String> {
|
||||
self.entries().into_iter().map(|v| v.artifact_id).collect()
|
||||
}
|
||||
}
|
||||
|
||||
/// One frame a spectator took off its subscription.
|
||||
pub struct SpectatorFrame {
|
||||
pub boundary: u64,
|
||||
/// Every agent the committed snapshot carries, in publication order. A presentation
|
||||
/// consumer is a multi-agent consumer: one snapshot holds the whole session.
|
||||
pub agents: Vec<Id>,
|
||||
pub sequence: u64,
|
||||
/// How many undelivered snapshots were coalesced into this one.
|
||||
pub replaced: u64,
|
||||
pub view: ViewRef,
|
||||
pub artifact: flybus::Artifact,
|
||||
/// Each published chunk with the handle it travelled on.
|
||||
pub audio: Vec<(AudioRef, flybus::Artifact)>,
|
||||
}
|
||||
|
||||
/// A presentation-side consumer of committed snapshots.
|
||||
///
|
||||
/// It subscribes `latest` with finite credits, which is the spectator row of the domain
|
||||
/// retention table: new snapshots replace its queued value, it never blocks the session, and
|
||||
/// the only thing a slow one exhausts is its own credits.
|
||||
pub struct Spectator {
|
||||
subscription: flybus::Subscription,
|
||||
held: Vec<flybus::Message>,
|
||||
seen: u64,
|
||||
coalesced: u64,
|
||||
}
|
||||
|
||||
impl Spectator {
|
||||
/// Subscribes to `topic` in latest mode with `credits` in flight.
|
||||
///
|
||||
/// A latest subscription always has exactly one queued value; `credits` is its in-flight
|
||||
/// bound, which the router limits (two by default). Finite credits are the spectator row
|
||||
/// of the domain retention table: they are the only thing a slow viewer exhausts.
|
||||
pub async fn attach(
|
||||
client: &flybus::Client,
|
||||
topic: &str,
|
||||
credits: u32,
|
||||
) -> Result<Spectator, flybus::BusError> {
|
||||
let subscription = client
|
||||
.subscribe(
|
||||
topic,
|
||||
flybus::SubscriptionConfig::latest().in_flight(credits).replay(true),
|
||||
)
|
||||
.await?;
|
||||
Ok(Spectator {
|
||||
subscription,
|
||||
held: Vec::new(),
|
||||
seen: 0,
|
||||
coalesced: 0,
|
||||
})
|
||||
}
|
||||
|
||||
/// Takes the next snapshot, reads its frame and releases the delivery.
|
||||
pub async fn take_frame(&mut self) -> Option<SpectatorFrame> {
|
||||
let message = self.subscription.next().await?;
|
||||
self.seen += 1;
|
||||
self.coalesced += message.replaced();
|
||||
let frame = snapshot_frame(&message);
|
||||
drop(message);
|
||||
frame
|
||||
}
|
||||
|
||||
/// Takes the next snapshot message itself, for a renderer that keeps its own handle
|
||||
/// after the message is gone.
|
||||
pub async fn next_message(&mut self) -> Option<flybus::Message> {
|
||||
let message = self.subscription.next().await?;
|
||||
self.seen += 1;
|
||||
self.coalesced += message.replaced();
|
||||
Some(message)
|
||||
}
|
||||
|
||||
/// Takes a snapshot without consuming it, which is what a viewer that stops rendering
|
||||
/// does. Its credits run out and nothing else in the session notices.
|
||||
pub async fn hold_one(&mut self) -> bool {
|
||||
match self.subscription.next().await {
|
||||
Some(message) => {
|
||||
self.seen += 1;
|
||||
self.coalesced += message.replaced();
|
||||
self.held.push(message);
|
||||
true
|
||||
}
|
||||
None => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Takes a snapshot without consuming it if one is queued right now.
|
||||
pub fn try_hold_one(&mut self) -> bool {
|
||||
match self.subscription.try_next() {
|
||||
Some(message) => {
|
||||
self.seen += 1;
|
||||
self.coalesced += message.replaced();
|
||||
self.held.push(message);
|
||||
true
|
||||
}
|
||||
None => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Releases everything this spectator was holding, returning its credits.
|
||||
pub fn release(&mut self) {
|
||||
self.held.clear();
|
||||
}
|
||||
|
||||
pub fn held(&self) -> usize {
|
||||
self.held.len()
|
||||
}
|
||||
|
||||
pub fn seen(&self) -> u64 {
|
||||
self.seen
|
||||
}
|
||||
|
||||
/// How many snapshots were replaced in this spectator's queue while it was busy.
|
||||
pub fn coalesced(&self) -> u64 {
|
||||
self.coalesced
|
||||
}
|
||||
}
|
||||
|
||||
/// Reads one committed snapshot's first view and its handle.
|
||||
pub fn snapshot_frame(message: &flybus::Message) -> Option<SpectatorFrame> {
|
||||
let payload = message.payload();
|
||||
let boundary = payload
|
||||
.get("scope")
|
||||
.and_then(|s| s.get("step"))
|
||||
.and_then(serde_json::Value::as_str)
|
||||
.and_then(|s| s.parse::<u64>().ok())?;
|
||||
let media = payload.get("media")?;
|
||||
let views = media.get("views")?.as_array()?;
|
||||
let view = ViewRef::from_json(views.first()?).ok()?;
|
||||
// An unreadable chunk, or one whose handle is not attached, makes the whole snapshot
|
||||
// unreadable rather than a snapshot that quietly has less audio in it than was published.
|
||||
let mut audio = Vec::new();
|
||||
for value in media.get("audio")?.as_array()? {
|
||||
let chunk = AudioRef::from_json(value).ok()?;
|
||||
let artifact = message.artifact(&audio_attachment(&chunk.stream_id)).ok()?;
|
||||
audio.push((chunk, artifact));
|
||||
}
|
||||
let artifact = message.artifact(&view_attachment(&view.view_id)).ok()?;
|
||||
let agents = payload
|
||||
.get("agents")
|
||||
.and_then(serde_json::Value::as_array)
|
||||
.map(|agents| {
|
||||
agents
|
||||
.iter()
|
||||
.filter_map(|a| a.get("agentId").and_then(serde_json::Value::as_str))
|
||||
.map(str::to_owned)
|
||||
.collect()
|
||||
})
|
||||
.unwrap_or_default();
|
||||
Some(SpectatorFrame {
|
||||
boundary,
|
||||
agents,
|
||||
sequence: message.topic_sequence(),
|
||||
replaced: message.replaced(),
|
||||
view,
|
||||
artifact,
|
||||
audio,
|
||||
})
|
||||
}
|
||||
|
||||
/// Takes the frame out of a message and drops the message, as a renderer that finishes later
|
||||
/// does. The delivery stays alive because the extracted handle still owns it.
|
||||
pub fn detach_frame(message: flybus::Message) -> Option<(ViewRef, flybus::Artifact)> {
|
||||
let frame = snapshot_frame(&message)?;
|
||||
drop(message);
|
||||
Some((frame.view, frame.artifact))
|
||||
}
|
||||
|
||||
// ----------------------------------------------------------------------------------------------
|
||||
// Persistent assets
|
||||
|
||||
/// The preprovisioned local registry an `AssetRef` names.
|
||||
///
|
||||
/// An asset is installed and verified before a run; it is not a path, a URL or something a
|
||||
/// worker fetches. Nothing in this registry can be addressed by an `ArtifactRef`, and
|
||||
/// [`AssetRegistry::import`] hands back a fresh transient artifact rather than turning the
|
||||
/// asset into one.
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub struct AssetRegistry {
|
||||
installed: BTreeMap<Id, (AssetRef, Vec<u8>)>,
|
||||
}
|
||||
|
||||
impl AssetRegistry {
|
||||
pub fn new() -> AssetRegistry {
|
||||
AssetRegistry::default()
|
||||
}
|
||||
|
||||
/// Installs content, verifying that it is the content the reference claims.
|
||||
pub fn install(&mut self, asset: AssetRef, bytes: Vec<u8>) -> DomainResult<()> {
|
||||
asset.validate().map_err(DomainError::invalid)?;
|
||||
if asset.byte_length != bytes.len() as u64 {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::IdentityMismatch,
|
||||
format!("asset {}: byteLength is not the installed length", asset.id),
|
||||
));
|
||||
}
|
||||
if asset.digest != digest_of_bytes(&bytes) {
|
||||
return Err(DomainError::before(
|
||||
ErrorCode::IdentityMismatch,
|
||||
format!("asset {}: digest is not the installed content", asset.id),
|
||||
));
|
||||
}
|
||||
self.installed.insert(asset.id.clone(), (asset, bytes));
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Resolves installed content. Identity and digest must both match: the same id with
|
||||
/// another digest is a different asset, not an upgrade.
|
||||
pub fn resolve(&self, asset: &AssetRef) -> DomainResult<&[u8]> {
|
||||
match self.installed.get(&asset.id) {
|
||||
Some((installed, bytes)) if installed == asset => Ok(bytes),
|
||||
Some(_) => Err(DomainError::before(
|
||||
ErrorCode::IdentityMismatch,
|
||||
format!("asset {} is installed with another identity", asset.id),
|
||||
)),
|
||||
None => Err(DomainError::before(
|
||||
ErrorCode::IdentityMismatch,
|
||||
format!("asset {} is not installed", asset.id),
|
||||
)),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn contains(&self, asset: &AssetRef) -> bool {
|
||||
self.resolve(asset).is_ok()
|
||||
}
|
||||
|
||||
/// Imports installed content into the bus as a fresh immutable artifact.
|
||||
///
|
||||
/// The artifact is sealed against the asset's digest, which is mandatory for a persistent
|
||||
/// asset import, and its identity belongs to the current store incarnation. The asset
|
||||
/// reference is unchanged and outlives it.
|
||||
pub async fn import(
|
||||
&self,
|
||||
client: &flybus::Client,
|
||||
asset: &AssetRef,
|
||||
) -> DomainResult<flybus::Artifact> {
|
||||
let bytes = self.resolve(asset)?;
|
||||
let mut writer = client
|
||||
.artifacts()
|
||||
.allocate(asset.byte_length, "application/octet-stream")
|
||||
.await
|
||||
.map_err(|e| store_error("allocate", &e.message))?;
|
||||
writer
|
||||
.write_all(bytes)
|
||||
.map_err(|e| store_error("write", &e.to_string()))?;
|
||||
let artifact = writer
|
||||
.seal_with_digest(Some(asset.digest.clone()))
|
||||
.await
|
||||
.map_err(|e| store_error("seal", &e.message))?;
|
||||
fly_session_types::media::check_imported_asset(asset, artifact.reference())
|
||||
.map_err(|e| DomainError::before(ErrorCode::IdentityMismatch, e))?;
|
||||
Ok(artifact)
|
||||
}
|
||||
}
|
||||
176
services/flysim/crates/fly-session/src/metrics.rs
Normal file
176
services/flysim/crates/fly-session/src/metrics.rs
Normal file
|
|
@ -0,0 +1,176 @@
|
|||
//! Latency and resource samples, for the measurements the implementation guide's section 5
|
||||
//! asks every slice to report.
|
||||
//!
|
||||
//! These are local synthetic timings on one machine. No host capacity claim follows from any
|
||||
//! number this module produces, and nothing here is a gameplay latency goal: the percentiles
|
||||
//! exist so that the three execution modes can be compared against each other.
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
|
||||
/// One sample set's order statistics, by nearest rank.
|
||||
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
|
||||
pub struct Percentiles {
|
||||
pub count: usize,
|
||||
pub p50_ns: u64,
|
||||
pub p95_ns: u64,
|
||||
pub p99_ns: u64,
|
||||
pub max_ns: u64,
|
||||
}
|
||||
|
||||
impl Percentiles {
|
||||
fn of(sorted: &[u64]) -> Percentiles {
|
||||
let rank = |p: f64| -> u64 {
|
||||
if sorted.is_empty() {
|
||||
return 0;
|
||||
}
|
||||
let n = sorted.len() as f64;
|
||||
let index = (p * n).ceil() as usize;
|
||||
sorted[index.clamp(1, sorted.len()) - 1]
|
||||
};
|
||||
Percentiles {
|
||||
count: sorted.len(),
|
||||
p50_ns: rank(0.50),
|
||||
p95_ns: rank(0.95),
|
||||
p99_ns: rank(0.99),
|
||||
max_ns: sorted.last().copied().unwrap_or_default(),
|
||||
}
|
||||
}
|
||||
|
||||
pub fn p50_us(&self) -> f64 {
|
||||
self.p50_ns as f64 / 1000.0
|
||||
}
|
||||
|
||||
pub fn p95_us(&self) -> f64 {
|
||||
self.p95_ns as f64 / 1000.0
|
||||
}
|
||||
|
||||
pub fn p99_us(&self) -> f64 {
|
||||
self.p99_ns as f64 / 1000.0
|
||||
}
|
||||
}
|
||||
|
||||
/// Named duration samples. One name is one measured path: a domain method, or the
|
||||
/// coordinator's whole critical path for a transition.
|
||||
#[derive(Clone, Debug, Default)]
|
||||
pub struct Metrics {
|
||||
samples: BTreeMap<String, Vec<u64>>,
|
||||
}
|
||||
|
||||
impl Metrics {
|
||||
pub fn record(&mut self, what: &str, elapsed: std::time::Duration) {
|
||||
let ns = u64::try_from(elapsed.as_nanos()).unwrap_or(u64::MAX);
|
||||
self.samples.entry(what.to_owned()).or_default().push(ns);
|
||||
}
|
||||
|
||||
pub fn percentiles(&self, what: &str) -> Option<Percentiles> {
|
||||
let mut values = self.samples.get(what)?.clone();
|
||||
values.sort_unstable();
|
||||
Some(Percentiles::of(&values))
|
||||
}
|
||||
|
||||
pub fn names(&self) -> Vec<String> {
|
||||
self.samples.keys().cloned().collect()
|
||||
}
|
||||
|
||||
pub fn count(&self, what: &str) -> usize {
|
||||
self.samples.get(what).map(Vec::len).unwrap_or_default()
|
||||
}
|
||||
|
||||
pub fn clear(&mut self) {
|
||||
self.samples.clear();
|
||||
}
|
||||
}
|
||||
|
||||
/// This process's peak resident set, in KiB, from its own status file.
|
||||
pub fn peak_rss_kib() -> Option<u64> {
|
||||
peak_rss_of("/proc/self/status")
|
||||
}
|
||||
|
||||
/// One child process's peak resident set, in KiB. `None` once the child is gone.
|
||||
pub fn peak_rss_kib_of(pid: u32) -> Option<u64> {
|
||||
peak_rss_of(&format!("/proc/{pid}/status"))
|
||||
}
|
||||
|
||||
fn peak_rss_of(path: &str) -> Option<u64> {
|
||||
let text = std::fs::read_to_string(path).ok()?;
|
||||
for line in text.lines() {
|
||||
if let Some(rest) = line.strip_prefix("VmHWM:") {
|
||||
return rest.split_whitespace().next()?.parse().ok();
|
||||
}
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
/// How many physical cores this machine has, counted as distinct (package, core) pairs.
|
||||
///
|
||||
/// Falls back to the logical count, which is what a thread budget has to use when the
|
||||
/// topology cannot be read.
|
||||
pub fn physical_cores() -> usize {
|
||||
if let Ok(text) = std::fs::read_to_string("/proc/cpuinfo") {
|
||||
let mut pairs: std::collections::BTreeSet<(String, String)> =
|
||||
std::collections::BTreeSet::new();
|
||||
let (mut package, mut core) = (None, None);
|
||||
for line in text.lines() {
|
||||
if line.trim().is_empty() {
|
||||
if let (Some(p), Some(c)) = (package.take(), core.take()) {
|
||||
pairs.insert((p, c));
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if let Some((key, value)) = line.split_once(':') {
|
||||
match key.trim() {
|
||||
"physical id" => package = Some(value.trim().to_owned()),
|
||||
"core id" => core = Some(value.trim().to_owned()),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
if let (Some(p), Some(c)) = (package, core) {
|
||||
pairs.insert((p, c));
|
||||
}
|
||||
if !pairs.is_empty() {
|
||||
return pairs.len();
|
||||
}
|
||||
}
|
||||
logical_cores()
|
||||
}
|
||||
|
||||
/// How many hardware threads this machine reports.
|
||||
pub fn logical_cores() -> usize {
|
||||
std::thread::available_parallelism().map(|n| n.get()).unwrap_or(1)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn percentiles_use_nearest_rank() {
|
||||
let mut m = Metrics::default();
|
||||
for ns in 1..=100u64 {
|
||||
m.record("x", std::time::Duration::from_nanos(ns));
|
||||
}
|
||||
let p = m.percentiles("x").expect("recorded");
|
||||
assert_eq!(p.count, 100);
|
||||
assert_eq!(p.p50_ns, 50);
|
||||
assert_eq!(p.p95_ns, 95);
|
||||
assert_eq!(p.p99_ns, 99);
|
||||
assert_eq!(p.max_ns, 100);
|
||||
assert!(m.percentiles("y").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_single_sample_is_every_percentile() {
|
||||
let mut m = Metrics::default();
|
||||
m.record("x", std::time::Duration::from_nanos(7));
|
||||
let p = m.percentiles("x").expect("recorded");
|
||||
assert_eq!((p.p50_ns, p.p95_ns, p.p99_ns, p.max_ns), (7, 7, 7, 7));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn the_machine_reports_at_least_one_core() {
|
||||
assert!(physical_cores() >= 1);
|
||||
assert!(logical_cores() >= 1);
|
||||
assert!(peak_rss_kib().unwrap_or(1) > 0);
|
||||
}
|
||||
}
|
||||
|
|
@ -188,6 +188,12 @@ pub trait WorkerEndpoint: Send + 'static {
|
|||
fn capabilities(&self) -> Vec<Id>;
|
||||
fn status_cell(&self) -> StatusCell;
|
||||
|
||||
/// The thread allocation this worker's launcher started it within.
|
||||
///
|
||||
/// `Worker.Hello` reports it, so a caller bounded by `workers-v1`'s "within launcher
|
||||
/// allocation" can read the allocation instead of being told it out of band.
|
||||
fn worker_threads(&self) -> u64;
|
||||
|
||||
/// The domain methods this endpoint implements, beyond the common `Worker.*` set.
|
||||
/// Anything else returns UNSUPPORTED without entering the endpoint.
|
||||
fn methods(&self) -> Vec<&'static str>;
|
||||
|
|
@ -224,6 +230,22 @@ impl WorkerHandle {
|
|||
let _ = self.task.await;
|
||||
self.client.close().await;
|
||||
}
|
||||
|
||||
/// Stops serving without waiting. The connection closes when the last handle to it is
|
||||
/// dropped, which this does. For a supervisor's `Drop`, where there is no runtime to wait
|
||||
/// on.
|
||||
pub fn abort(self) {
|
||||
self.task.abort();
|
||||
}
|
||||
|
||||
/// Waits until the worker stops serving, which `Worker.Shutdown` makes it do.
|
||||
///
|
||||
/// A worker process awaits this and then exits, so the supervisor's `Worker.Shutdown` and
|
||||
/// the process's exit are the same event rather than two racing ones.
|
||||
pub async fn join(self) {
|
||||
let _ = self.task.await;
|
||||
self.client.close().await;
|
||||
}
|
||||
}
|
||||
|
||||
/// Registers `service_name` and serves `endpoint` on it until the service ends or Shutdown.
|
||||
|
|
@ -259,7 +281,7 @@ async fn run<E: WorkerEndpoint>(
|
|||
) {
|
||||
// Identity and capabilities are fixed for the endpoint's lifetime, so the shell reads them
|
||||
// once and never takes the endpoint mutex to answer Hello or Status.
|
||||
let (worker_id, incarnation_id, session_id, role, capabilities, status, methods) = {
|
||||
let (worker_id, incarnation_id, session_id, role, capabilities, status, methods, threads) = {
|
||||
let e = endpoint.lock().await;
|
||||
(
|
||||
e.worker_id(),
|
||||
|
|
@ -269,6 +291,7 @@ async fn run<E: WorkerEndpoint>(
|
|||
e.capabilities(),
|
||||
e.status_cell(),
|
||||
e.methods(),
|
||||
e.worker_threads(),
|
||||
)
|
||||
};
|
||||
let mut running: Vec<tokio::task::JoinHandle<()>> = Vec::new();
|
||||
|
|
@ -305,6 +328,7 @@ async fn run<E: WorkerEndpoint>(
|
|||
&incarnation_id,
|
||||
role,
|
||||
&capabilities,
|
||||
threads,
|
||||
);
|
||||
let _ = responder.reply(outcome.to_outcome(), &[]).await;
|
||||
continue;
|
||||
|
|
@ -606,6 +630,7 @@ fn failure(
|
|||
failure_outcome(request_id, worker_id, incarnation_id, scope, error)
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
fn hello(
|
||||
request: &SessionRpcRequest,
|
||||
session_id: &Id,
|
||||
|
|
@ -613,6 +638,7 @@ fn hello(
|
|||
incarnation_id: &Id,
|
||||
role: Role,
|
||||
capabilities: &[Id],
|
||||
worker_threads: u64,
|
||||
) -> SessionRpcOutcome {
|
||||
let params: HelloParams = match HelloParams::from_json(&request.params) {
|
||||
Ok(params) => params,
|
||||
|
|
@ -668,6 +694,7 @@ fn hello(
|
|||
capabilities: capabilities.to_vec(),
|
||||
max_agents: MAX_AGENTS as u64,
|
||||
max_ports: MAX_PORTS as u64,
|
||||
worker_threads,
|
||||
};
|
||||
success(
|
||||
request,
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
|
||||
use std::time::Duration;
|
||||
|
||||
use fly_session::harness::{HarnessConfig, SessionHarness, Via};
|
||||
use fly_session::harness::{ExecutionMode, HarnessConfig, SessionHarness, Via};
|
||||
use fly_session::types::*;
|
||||
|
||||
pub const WAIT: Duration = Duration::from_secs(20);
|
||||
|
|
@ -94,3 +94,55 @@ pub async fn within<T>(what: &str, f: impl std::future::Future<Output = T>) -> T
|
|||
Err(_) => panic!("{what}: timed out"),
|
||||
}
|
||||
}
|
||||
|
||||
/// Generates one test per execution mode from an `async fn name(mode: ExecutionMode)`.
|
||||
///
|
||||
/// The separate-process mode is the SESSION-02 subject; the other two are the variants it is
|
||||
/// compared against, and a row that holds in one must hold in all three.
|
||||
#[macro_export]
|
||||
macro_rules! all_modes {
|
||||
($($name:ident),* $(,)?) => {
|
||||
mod in_process {
|
||||
$(
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn $name() {
|
||||
super::$name($crate::common::mode_in_process()).await
|
||||
}
|
||||
)*
|
||||
}
|
||||
mod dedicated_thread {
|
||||
$(
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn $name() {
|
||||
super::$name($crate::common::mode_thread()).await
|
||||
}
|
||||
)*
|
||||
}
|
||||
mod separate_process {
|
||||
$(
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn $name() {
|
||||
super::$name($crate::common::mode_process()).await
|
||||
}
|
||||
)*
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
pub fn mode_in_process() -> ExecutionMode {
|
||||
ExecutionMode::InProcess
|
||||
}
|
||||
|
||||
pub fn mode_thread() -> ExecutionMode {
|
||||
ExecutionMode::Thread
|
||||
}
|
||||
|
||||
pub fn mode_process() -> ExecutionMode {
|
||||
ExecutionMode::Process
|
||||
}
|
||||
|
||||
/// A fixture in one execution mode. The transport is the mode's own: a separate process
|
||||
/// reaches the router only over a socket.
|
||||
pub async fn mode_fixture(mode: ExecutionMode, config: HarnessConfig) -> Fixture {
|
||||
fixture(Via::Unix, HarnessConfig { mode, ..config }).await
|
||||
}
|
||||
|
|
|
|||
|
|
@ -33,6 +33,15 @@ both_transports!(
|
|||
const STEPS: u64 = 4;
|
||||
const INJECT_AT: u64 = 2;
|
||||
|
||||
/// The mutation counter of a participant running in this process. These suites are all
|
||||
/// in-process compositions, so it is always there; SESSION-02's are not, and read it over the
|
||||
/// bus instead.
|
||||
fn local_mutations(f: &Fixture, agent_id: &Id) -> u64 {
|
||||
f.harness
|
||||
.agent_mutations(agent_id)
|
||||
.expect("an in-process participant keeps its counter in this process")
|
||||
}
|
||||
|
||||
/// What a run of the standard composition produced.
|
||||
struct Run {
|
||||
behaviour: Vec<String>,
|
||||
|
|
@ -51,8 +60,8 @@ async fn run_with(via: Via, injections: Injections) -> Run {
|
|||
let run = Run {
|
||||
behaviour: f.harness.coordinator.trace.behavior(),
|
||||
mutations: vec![
|
||||
(fly_a(), f.harness.agent_mutations(&fly_a())),
|
||||
(fly_b(), f.harness.agent_mutations(&fly_b())),
|
||||
(fly_a(), local_mutations(&f, &fly_a())),
|
||||
(fly_b(), local_mutations(&f, &fly_b())),
|
||||
],
|
||||
counter: f
|
||||
.harness
|
||||
|
|
|
|||
862
services/flysim/crates/fly-session/tests/media.rs
Normal file
862
services/flysim/crates/fly-session/tests/media.rs
Normal file
|
|
@ -0,0 +1,862 @@
|
|||
//! MEDIA-01 acceptance: native observations on the bus artifact, and the presentation handoff.
|
||||
//!
|
||||
//! Every test here is one of the slice's acceptance bullets or one row of the domain retention
|
||||
//! table in `state-media-v1` section 3. The shape rules themselves are proved against the
|
||||
//! contract crate in `fly-session-types/tests/media_shapes.rs`; these prove the session's use
|
||||
//! of them: one shared image, spectators that cannot touch sensory state, a renderer that
|
||||
//! keeps its handle, and a persistent asset that is not a transient artifact.
|
||||
|
||||
mod common;
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::time::Duration;
|
||||
|
||||
use serde_json::Map;
|
||||
|
||||
use common::{Fixture, default_fixture, fixture, fly_a, fly_b, mode_fixture, within};
|
||||
use fly_session::environment::{
|
||||
AUDIO_STREAM_ID, CHANNELS, EnvironmentFaults, SAMPLE_RATE, VIEW_HEIGHT, VIEW_WIDTH,
|
||||
synthetic_asset,
|
||||
};
|
||||
use fly_session::harness::{ExecutionMode, HarnessConfig, Via};
|
||||
use fly_session::media::{
|
||||
AssetRegistry, AudioSource, AudioTimelines, SensedView, Spectator, SpectatorFrame,
|
||||
arena_frame, audio_attachment, detach_frame, view_attachment,
|
||||
};
|
||||
use fly_session::phase::Phase;
|
||||
use fly_session::types::*;
|
||||
use fly_session_types::media::{AudioTimeline, check_imported_asset, require_finite_samples};
|
||||
|
||||
both_transports!(
|
||||
one_shared_image_reaches_both_agents_through_owned_attachments,
|
||||
a_spectator_cannot_corrupt_sensory_state,
|
||||
a_slow_spectator_exhausts_only_its_own_credits,
|
||||
delayed_rendering_retains_its_handle_after_the_message_drops,
|
||||
a_declared_render_delay_repeats_o0_until_the_pipeline_fills,
|
||||
an_extra_delayed_sensory_view_fails_the_step,
|
||||
a_frame_of_the_wrong_length_fails_the_step,
|
||||
overlapping_audio_fails_the_step,
|
||||
one_audio_chunk_per_boundary_with_an_exact_sample_budget,
|
||||
required_agent_input_is_never_coalesced_while_spectator_snapshots_are,
|
||||
a_persistent_asset_and_a_transient_artifact_are_different_identities,
|
||||
a_restored_audio_source_resumes_and_marks_the_discontinuity,
|
||||
a_missing_audio_chunk_fails_the_step,
|
||||
restored_timelines_need_every_declared_streams_position,
|
||||
a_cadence_that_does_not_divide_the_sample_rate_still_lands_on_whole_samples,
|
||||
);
|
||||
|
||||
all_modes!(
|
||||
the_media_path_works_in_every_execution_mode,
|
||||
every_media_fault_fires_in_every_execution_mode,
|
||||
);
|
||||
|
||||
const STEPS: u64 = 3;
|
||||
|
||||
/// Polls until `ok` holds, so a test never asserts a collection that has not happened yet.
|
||||
async fn until(what: &str, mut ok: impl FnMut() -> bool) {
|
||||
let deadline = std::time::Instant::now() + Duration::from_secs(10);
|
||||
while !ok() {
|
||||
assert!(std::time::Instant::now() < deadline, "{what}: never happened");
|
||||
tokio::time::sleep(Duration::from_millis(5)).await;
|
||||
}
|
||||
}
|
||||
|
||||
/// Drains a latest subscription until the snapshot for `boundary` arrives.
|
||||
///
|
||||
/// A latest subscription converges on the newest value rather than delivering every one, so a
|
||||
/// test that wants a particular committed boundary reads until it gets there.
|
||||
async fn frame_at(spectator: &mut Spectator, boundary: u64) -> SpectatorFrame {
|
||||
for _ in 0..64 {
|
||||
let frame = within("snapshot", spectator.take_frame())
|
||||
.await
|
||||
.expect("a snapshot");
|
||||
if frame.boundary == boundary {
|
||||
return frame;
|
||||
}
|
||||
}
|
||||
panic!("the spectator never reached boundary {boundary}");
|
||||
}
|
||||
|
||||
/// The views one agent read.
|
||||
///
|
||||
/// The sensor log is shared memory, so it is readable only where that agent lives. These tests
|
||||
/// run in the default in-process composition; the process-mode test below asserts the media
|
||||
/// path over the bus instead, which is what crosses a process boundary.
|
||||
fn sensed(f: &Fixture, agent_id: &Id) -> Vec<SensedView> {
|
||||
f.harness
|
||||
.sensor_log(agent_id)
|
||||
.expect("this composition keeps its agents in this process")
|
||||
.entries()
|
||||
}
|
||||
|
||||
/// How many native frames the world rendered, for a world in this process.
|
||||
fn renders(f: &Fixture) -> u64 {
|
||||
f.harness
|
||||
.renders()
|
||||
.expect("this composition keeps its world in this process")
|
||||
}
|
||||
|
||||
fn boundaries(entries: &[SensedView]) -> Vec<u64> {
|
||||
entries.iter().map(|e| e.boundary).collect()
|
||||
}
|
||||
|
||||
fn produced(entries: &[SensedView]) -> Vec<u64> {
|
||||
entries.iter().map(|e| e.produced_step).collect()
|
||||
}
|
||||
|
||||
/// One image per boundary reaches both agents as an owned attachment, and the same handle is
|
||||
/// published for presentation. There is no second copy of the pixels anywhere.
|
||||
async fn one_shared_image_reaches_both_agents_through_owned_attachments(via: Via) {
|
||||
let mut f = default_fixture(via).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let observer = f.harness.observer().await.unwrap();
|
||||
let topic = f.harness.coordinator.topics().snapshots.clone();
|
||||
let mut spectator = Spectator::attach(&observer, &topic, 2).await.unwrap();
|
||||
within("run", f.harness.coordinator.run(STEPS)).await.unwrap();
|
||||
|
||||
// One render per boundary: forwarding the handle to two agents and to publication does not
|
||||
// render or copy it again.
|
||||
assert_eq!(
|
||||
renders(&f),
|
||||
STEPS + 1,
|
||||
"one native frame per boundary, whatever the number of recipients"
|
||||
);
|
||||
|
||||
let a = sensed(&f, &fly_a());
|
||||
let b = sensed(&f, &fly_b());
|
||||
assert_eq!(boundaries(&a), (0..=STEPS).collect::<Vec<_>>());
|
||||
assert_eq!(a, b, "both agents read the same artifact and the same bytes");
|
||||
assert_eq!(produced(&a), (0..=STEPS).collect::<Vec<_>>(), "no declared delay");
|
||||
|
||||
// The handle the agents were given is the handle presentation was published.
|
||||
let published = frame_at(&mut spectator, STEPS).await;
|
||||
assert_eq!(
|
||||
published.agents,
|
||||
vec![fly_a(), fly_b()],
|
||||
"one snapshot carries the whole multi-agent session"
|
||||
);
|
||||
let last = a.last().expect("an entry per boundary");
|
||||
assert_eq!(published.view.pixels.artifact_id, last.artifact_id);
|
||||
assert_eq!(published.view.produced_step, last.produced_step);
|
||||
let bytes = published.artifact.read_all().await.expect("the published frame");
|
||||
assert_eq!(bytes.len() as u64, VIEW_WIDTH * VIEW_HEIGHT * 4);
|
||||
assert_eq!(digest_of_bytes(&bytes), last.digest, "the bytes both agents read");
|
||||
|
||||
// The coordinator holds exactly one handle per declared view and stream at this boundary.
|
||||
let handles = f.harness.coordinator.media_handles();
|
||||
assert_eq!(handles.len(), 2, "one view and one audio chunk: {handles:?}");
|
||||
assert!(handles.iter().any(|(name, r)| *name == view_attachment("arena")
|
||||
&& r.artifact_id == last.artifact_id));
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// A spectator reads committed snapshots and cannot touch what the agents sense: it has no
|
||||
/// authority over the environment, and its own reads leave sensory state exactly as produced.
|
||||
async fn a_spectator_cannot_corrupt_sensory_state(via: Via) {
|
||||
let mut f = default_fixture(via).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let observer = f.harness.observer().await.unwrap();
|
||||
let topic = f.harness.coordinator.topics().snapshots.clone();
|
||||
let mut spectator = Spectator::attach(&observer, &topic, 2).await.unwrap();
|
||||
|
||||
// Naming the environment is not authority to drive it: a subscriber cannot call it.
|
||||
let refused = observer
|
||||
.call("env.arena", None, "Environment.Advance", Map::new(), &[])
|
||||
.await;
|
||||
assert!(
|
||||
refused.is_err(),
|
||||
"a spectator must not be able to call the environment"
|
||||
);
|
||||
|
||||
within("run", f.harness.coordinator.run(STEPS)).await.unwrap();
|
||||
// The spectator reads the snapshot and releases it while the session keeps stepping.
|
||||
let frame = within("snapshot", spectator.take_frame()).await.expect("a snapshot");
|
||||
let seen = frame.artifact.read_all().await.expect("readable");
|
||||
drop(frame);
|
||||
within("run more", f.harness.coordinator.run(STEPS)).await.unwrap();
|
||||
|
||||
let a = sensed(&f, &fly_a());
|
||||
let b = sensed(&f, &fly_b());
|
||||
assert_eq!(boundaries(&a), (0..=2 * STEPS).collect::<Vec<_>>());
|
||||
assert_eq!(a, b);
|
||||
assert_eq!(
|
||||
f.harness.coordinator.committed_boundary(),
|
||||
Some(2 * STEPS),
|
||||
"the spectator changed nothing about the session's progress"
|
||||
);
|
||||
// What the spectator read was one of the frames the agents encoded, unchanged.
|
||||
let digest = digest_of_bytes(&seen);
|
||||
assert!(
|
||||
a.iter().any(|entry| entry.digest == digest),
|
||||
"a spectator sees the committed frame, and only reads it"
|
||||
);
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// A slow spectator exhausts its own credits. New snapshots replace its queued value; the
|
||||
/// world never waits for it.
|
||||
async fn a_slow_spectator_exhausts_only_its_own_credits(via: Via) {
|
||||
let mut f = default_fixture(via).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let observer = f.harness.observer().await.unwrap();
|
||||
let topic = f.harness.coordinator.topics().snapshots.clone();
|
||||
// One credit and one queued value: the smallest spectator the bus allows.
|
||||
let mut slow = Spectator::attach(&observer, &topic, 1).await.unwrap();
|
||||
let mut fast = Spectator::attach(&observer, &topic, 2).await.unwrap();
|
||||
|
||||
// The slow one takes one snapshot and stops rendering, holding its only credit.
|
||||
assert!(within("first snapshot", slow.hold_one()).await);
|
||||
assert_eq!(slow.held(), 1);
|
||||
|
||||
let steps = 5;
|
||||
within("run", f.harness.coordinator.run(steps)).await.unwrap();
|
||||
assert_eq!(f.harness.coordinator.stats().advances, steps);
|
||||
|
||||
// With its credit in use it receives nothing more, and its queued value is replaced.
|
||||
assert!(!slow.try_hold_one(), "a spectator out of credits gets nothing more");
|
||||
assert_eq!(slow.seen(), 1);
|
||||
slow.release();
|
||||
let next = within("after release", slow.hold_one()).await;
|
||||
assert!(next, "releasing its own delivery returns its own credit");
|
||||
assert!(
|
||||
slow.coalesced() > 0,
|
||||
"new snapshots replaced the value queued for a spectator that was not reading"
|
||||
);
|
||||
|
||||
// The session and the other spectator are untouched.
|
||||
let latest = frame_at(&mut fast, steps).await;
|
||||
assert_eq!(
|
||||
latest.boundary, steps,
|
||||
"the reading spectator reaches the latest boundary"
|
||||
);
|
||||
let a = sensed(&f, &fly_a());
|
||||
assert_eq!(boundaries(&a), (0..=steps).collect::<Vec<_>>());
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// A renderer keeps its extracted handle after the message is gone, and after the boundary it
|
||||
/// came from has been replaced everywhere else.
|
||||
async fn delayed_rendering_retains_its_handle_after_the_message_drops(via: Via) {
|
||||
let mut f = default_fixture(via).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let observer = f.harness.observer().await.unwrap();
|
||||
let topic = f.harness.coordinator.topics().snapshots.clone();
|
||||
let mut spectator = Spectator::attach(&observer, &topic, 2).await.unwrap();
|
||||
within("step", f.harness.coordinator.run(1)).await.unwrap();
|
||||
|
||||
// Take the message, keep only the frame, and drop the message itself.
|
||||
let message = within("snapshot", spectator.next_message()).await.expect("a snapshot");
|
||||
let (view, artifact) = detach_frame(message).expect("a frame in the snapshot");
|
||||
|
||||
// Everything else moves on: the coordinator drops that boundary's handles, and the topic's
|
||||
// retained value is replaced twice.
|
||||
within("more steps", f.harness.coordinator.run(2)).await.unwrap();
|
||||
let handles = f.harness.coordinator.media_handles();
|
||||
assert!(
|
||||
!handles.iter().any(|(_, r)| r.artifact_id == view.pixels.artifact_id),
|
||||
"the session no longer holds the frame the renderer is still using"
|
||||
);
|
||||
|
||||
// The rendering finishes now, long after its message is gone.
|
||||
let bytes = artifact.read_all().await.expect("the guard kept the bytes alive");
|
||||
assert_eq!(bytes.len() as u64, VIEW_WIDTH * VIEW_HEIGHT * 4);
|
||||
let entries = sensed(&f, &fly_a());
|
||||
let at_boundary = entries
|
||||
.iter()
|
||||
.find(|e| e.produced_step == view.produced_step)
|
||||
.expect("the agents encoded this frame too");
|
||||
assert_eq!(
|
||||
digest_of_bytes(&bytes),
|
||||
at_boundary.digest,
|
||||
"the retained handle still reads exactly the frame that was published"
|
||||
);
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// A declared render delay repeats `O[0]` while the pipeline fills, then advances one frame
|
||||
/// per boundary. The repetition is the same artifact, not a re-render.
|
||||
async fn a_declared_render_delay_repeats_o0_until_the_pipeline_fills(via: Via) {
|
||||
let config = HarnessConfig {
|
||||
observation_delay_steps: 2,
|
||||
..HarnessConfig::default()
|
||||
};
|
||||
let mut f = fixture(via, config).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
within("run", f.harness.coordinator.run(4)).await.unwrap();
|
||||
|
||||
let a = sensed(&f, &fly_a());
|
||||
assert_eq!(boundaries(&a), vec![0, 1, 2, 3, 4]);
|
||||
assert_eq!(
|
||||
produced(&a),
|
||||
vec![0, 0, 0, 1, 2],
|
||||
"max(0, boundary - 2) at every boundary"
|
||||
);
|
||||
assert_eq!(a[0].artifact_id, a[1].artifact_id);
|
||||
assert_eq!(a[0].artifact_id, a[2].artifact_id, "O[0] repeats while the delay fills");
|
||||
assert_ne!(a[2].artifact_id, a[3].artifact_id, "then the pipeline advances");
|
||||
assert_ne!(a[3].artifact_id, a[4].artifact_id);
|
||||
// The world still renders once per boundary; the delay is a queue, not a missing frame.
|
||||
assert_eq!(renders(&f), 5);
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// Beyond the declared delay, an extra-delayed sensory view is a step failure, never an
|
||||
/// arbitrary latest frame.
|
||||
async fn an_extra_delayed_sensory_view_fails_the_step(via: Via) {
|
||||
let config = HarnessConfig {
|
||||
environment_faults: EnvironmentFaults {
|
||||
stale_view_at_boundary: Some(2),
|
||||
..EnvironmentFaults::default()
|
||||
},
|
||||
..HarnessConfig::default()
|
||||
};
|
||||
let mut f = fixture(via, config).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let failure = within("run", f.harness.coordinator.run(STEPS))
|
||||
.await
|
||||
.expect_err("a stale frame fails the transition");
|
||||
assert_eq!(failure.error.code, ErrorCode::BufferInvalid);
|
||||
assert_eq!(failure.error.mutation, MutationCertainty::Unknown);
|
||||
assert_eq!(f.harness.coordinator.phase(), Phase::Failed);
|
||||
assert_eq!(
|
||||
f.harness.coordinator.stats().advances,
|
||||
1,
|
||||
"the transition that met the stale frame committed nothing"
|
||||
);
|
||||
// The agents did not encode the stale frame.
|
||||
let a = sensed(&f, &fly_a());
|
||||
assert_eq!(boundaries(&a), vec![0, 1]);
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// A frame whose artifact is not `rowStride x height` bytes fails the step.
|
||||
async fn a_frame_of_the_wrong_length_fails_the_step(via: Via) {
|
||||
let config = HarnessConfig {
|
||||
environment_faults: EnvironmentFaults {
|
||||
truncated_view_at_boundary: Some(1),
|
||||
..EnvironmentFaults::default()
|
||||
},
|
||||
..HarnessConfig::default()
|
||||
};
|
||||
let mut f = fixture(via, config).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let failure = within("run", f.harness.coordinator.run(1))
|
||||
.await
|
||||
.expect_err("a short frame fails the transition");
|
||||
assert_eq!(failure.error.code, ErrorCode::BufferInvalid);
|
||||
assert_eq!(f.harness.coordinator.phase(), Phase::Failed);
|
||||
assert_eq!(f.harness.coordinator.stats().advances, 0);
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// Within an epoch, an audio chunk that starts inside the previous one is refused.
|
||||
async fn overlapping_audio_fails_the_step(via: Via) {
|
||||
let config = HarnessConfig {
|
||||
environment_faults: EnvironmentFaults {
|
||||
overlapping_audio_at_boundary: Some(2),
|
||||
..EnvironmentFaults::default()
|
||||
},
|
||||
..HarnessConfig::default()
|
||||
};
|
||||
let mut f = fixture(via, config).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let failure = within("run", f.harness.coordinator.run(STEPS))
|
||||
.await
|
||||
.expect_err("an overlapping chunk fails the transition");
|
||||
assert_eq!(failure.error.code, ErrorCode::BufferInvalid);
|
||||
assert_eq!(f.harness.coordinator.stats().advances, 1);
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// One chunk per boundary, with an exact sample budget, finite samples and the byte length the
|
||||
/// descriptor implies. Audio is published for presentation and never enters sensory input.
|
||||
async fn one_audio_chunk_per_boundary_with_an_exact_sample_budget(via: Via) {
|
||||
let mut f = default_fixture(via).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let observer = f.harness.observer().await.unwrap();
|
||||
let topic = f.harness.coordinator.topics().snapshots.clone();
|
||||
let mut spectator = Spectator::attach(&observer, &topic, 2).await.unwrap();
|
||||
within("run", f.harness.coordinator.run(STEPS)).await.unwrap();
|
||||
|
||||
// 48 kHz at 60 Hz is exactly 800 frames a step, and the positions are contiguous.
|
||||
let per_step = SAMPLE_RATE / f.harness.config.step_hz;
|
||||
let positions = f.harness.coordinator.audio_positions();
|
||||
assert_eq!(positions[AUDIO_STREAM_ID], per_step * STEPS);
|
||||
|
||||
let observation = f.harness.coordinator.observation().expect("an observation").clone();
|
||||
assert_eq!(observation.audio.len(), 1, "one chunk per boundary");
|
||||
let chunk = &observation.audio[0];
|
||||
assert_eq!(chunk.sample_frames, per_step);
|
||||
assert_eq!(chunk.first_sample, per_step * (STEPS - 1));
|
||||
assert!(!chunk.discontinuity, "an uninterrupted epoch has no discontinuity");
|
||||
assert_eq!(chunk.samples.byte_length, per_step * CHANNELS * 4);
|
||||
// Sensory input is pixels only: audio never becomes an agent's input.
|
||||
assert!(observation.sensory_views.iter().all(|v| v.view_id == "arena"));
|
||||
|
||||
let published = frame_at(&mut spectator, STEPS).await;
|
||||
let (published_chunk, artifact) = published.audio.first().expect("the published chunk");
|
||||
assert_eq!(published_chunk.samples, chunk.samples);
|
||||
assert_eq!(
|
||||
artifact.reference().artifact_id,
|
||||
chunk.samples.artifact_id,
|
||||
"the same owned handle is published, not a copy"
|
||||
);
|
||||
let bytes = artifact.read_all().await.expect("the published chunk");
|
||||
assert_eq!(bytes.len() as u64, per_step * CHANNELS * 4);
|
||||
require_finite_samples(&bytes).expect("native samples are finite f32");
|
||||
assert_eq!(
|
||||
published.audio.len(),
|
||||
1,
|
||||
"the attachment list names every published chunk"
|
||||
);
|
||||
assert_eq!(audio_attachment(AUDIO_STREAM_ID), "audio.arena");
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// A declared stream that produces no chunk for a transition is a step failure, not a silently
|
||||
/// shorter epoch.
|
||||
async fn a_missing_audio_chunk_fails_the_step(via: Via) {
|
||||
let config = HarnessConfig {
|
||||
environment_faults: EnvironmentFaults {
|
||||
omit_audio_at_boundary: Some(2),
|
||||
..EnvironmentFaults::default()
|
||||
},
|
||||
..HarnessConfig::default()
|
||||
};
|
||||
let mut f = fixture(via, config).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let failure = within("run", f.harness.coordinator.run(STEPS))
|
||||
.await
|
||||
.expect_err("a missing chunk fails the transition");
|
||||
assert_eq!(failure.error.code, ErrorCode::BufferInvalid);
|
||||
assert_eq!(f.harness.coordinator.stats().advances, 1);
|
||||
// Boundary 0 has no preceding interval, so its empty audio list is not a missing chunk.
|
||||
let mut clean = default_fixture(via).await;
|
||||
within("bootstrap", clean.harness.coordinator.bootstrap()).await.unwrap();
|
||||
assert!(clean.harness.coordinator.observation().unwrap().audio.is_empty());
|
||||
clean.shutdown().await;
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// Restored timelines resume every declared stream at its recorded position. A stream with no
|
||||
/// recorded position is refused, not restarted at zero.
|
||||
async fn restored_timelines_need_every_declared_streams_position(via: Via) {
|
||||
let mut f = default_fixture(via).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
within("run", f.harness.coordinator.run(1)).await.unwrap();
|
||||
let descriptor = f.harness.coordinator.descriptor().expect("a descriptor").clone();
|
||||
let observation = f.harness.coordinator.observation().expect("an observation").clone();
|
||||
|
||||
// A fresh epoch accepts the transition's chunk and advances its position.
|
||||
let mut fresh = AudioTimelines::fresh(&descriptor);
|
||||
fresh
|
||||
.accept(&descriptor, &observation)
|
||||
.expect("the first chunk of a fresh epoch");
|
||||
assert_eq!(fresh.accepted(AUDIO_STREAM_ID), 1);
|
||||
let chunk = observation.audio.first().expect("a chunk").clone();
|
||||
assert_eq!(
|
||||
fresh.positions()[AUDIO_STREAM_ID],
|
||||
chunk.first_sample + chunk.sample_frames
|
||||
);
|
||||
|
||||
// A restore with nothing recorded for a declared stream is an error, not sample zero.
|
||||
let missing = AudioTimelines::restored(&descriptor, &BTreeMap::new())
|
||||
.expect_err("a declared stream needs its preserved position");
|
||||
assert_eq!(missing.code, ErrorCode::IncompatibleState);
|
||||
|
||||
// With the position recorded, the restored epoch resumes there and its first chunk must
|
||||
// mark the discontinuity.
|
||||
let positions = f.harness.coordinator.audio_positions();
|
||||
let mut resumed = observation.clone();
|
||||
let resumed_chunk = resumed.audio.first_mut().expect("a chunk");
|
||||
resumed_chunk.first_sample = positions[AUDIO_STREAM_ID];
|
||||
resumed_chunk.discontinuity = false;
|
||||
let mut restored = AudioTimelines::restored(&descriptor, &positions).expect("positions");
|
||||
restored
|
||||
.accept(&descriptor, &resumed)
|
||||
.expect_err("the first chunk after a restore marks discontinuity");
|
||||
resumed.audio.first_mut().expect("a chunk").discontinuity = true;
|
||||
let mut restored = AudioTimelines::restored(&descriptor, &positions).expect("positions");
|
||||
restored
|
||||
.accept(&descriptor, &resumed)
|
||||
.expect("the restored epoch resumes at the preserved position");
|
||||
assert_eq!(restored.accepted(AUDIO_STREAM_ID), 1);
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// A world cadence that does not divide the sample rate still produces whole samples, with the
|
||||
/// remainder carried rather than rounded: seven steps of a 7 Hz world are exactly one second.
|
||||
async fn a_cadence_that_does_not_divide_the_sample_rate_still_lands_on_whole_samples(via: Via) {
|
||||
let config = HarnessConfig {
|
||||
step_hz: 7,
|
||||
..HarnessConfig::default()
|
||||
};
|
||||
let mut f = fixture(via, config).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let observer = f.harness.observer().await.unwrap();
|
||||
let topic = f.harness.coordinator.topics().snapshots.clone();
|
||||
let mut spectator = Spectator::attach(&observer, &topic, 2).await.unwrap();
|
||||
|
||||
// 48000 / 7 is 6857.14..., so no chunk can be the exact share of a second.
|
||||
let mut frames = Vec::new();
|
||||
for step in 1..=7 {
|
||||
within("step", f.harness.coordinator.run(1)).await.unwrap();
|
||||
let chunk = f
|
||||
.harness
|
||||
.coordinator
|
||||
.observation()
|
||||
.expect("an observation")
|
||||
.audio
|
||||
.first()
|
||||
.expect("a chunk")
|
||||
.clone();
|
||||
assert_eq!(chunk.first_sample, frames.iter().sum::<u64>());
|
||||
assert!(
|
||||
chunk.sample_frames == 6_857 || chunk.sample_frames == 6_858,
|
||||
"step {step} produced {} frames",
|
||||
chunk.sample_frames
|
||||
);
|
||||
assert_eq!(chunk.samples.byte_length, chunk.sample_frames * CHANNELS * 4);
|
||||
frames.push(chunk.sample_frames);
|
||||
}
|
||||
assert_eq!(
|
||||
frames.iter().sum::<u64>(),
|
||||
SAMPLE_RATE,
|
||||
"seven steps of a 7 Hz world are exactly one second of samples: {frames:?}"
|
||||
);
|
||||
assert_eq!(f.harness.coordinator.audio_positions()[AUDIO_STREAM_ID], SAMPLE_RATE);
|
||||
|
||||
// The published chunk is readable and finite whatever the cadence.
|
||||
let published = frame_at(&mut spectator, 7).await;
|
||||
let (chunk, artifact) = published.audio.first().expect("the published chunk");
|
||||
let bytes = artifact.read_all().await.expect("the published chunk");
|
||||
assert_eq!(bytes.len() as u64, chunk.sample_frames * CHANNELS * 4);
|
||||
require_finite_samples(&bytes).expect("native samples are finite f32");
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// The media path itself is mode-agnostic: one native image per boundary, forwarded to every
|
||||
/// agent as an owned attachment and published once for presentation, whether the participants
|
||||
/// are tasks on one runtime, threads with their own runtimes, or separate processes.
|
||||
///
|
||||
/// What a *test* can see differs by mode, and this test only asserts what crosses a process
|
||||
/// boundary. An agent validates that each attachment is the artifact its payload names and
|
||||
/// reads the pixels before it commits, so a session that commits every boundary in process
|
||||
/// mode has carried one shared image across that boundary through the store, not through
|
||||
/// shared memory. The sensor log and the render counter are shared memory, so they are
|
||||
/// asserted where they exist and their absence is asserted where they do not.
|
||||
async fn the_media_path_works_in_every_execution_mode(mode: ExecutionMode) {
|
||||
// A declared render delay as well, so the option reaches a world in another process.
|
||||
let config = HarnessConfig {
|
||||
observation_delay_steps: 1,
|
||||
..HarnessConfig::default()
|
||||
};
|
||||
let mut f = mode_fixture(mode, config).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let observer = f.harness.observer().await.unwrap();
|
||||
let topic = f.harness.coordinator.topics().snapshots.clone();
|
||||
let mut spectator = Spectator::attach(&observer, &topic, 2).await.unwrap();
|
||||
within("run", f.harness.coordinator.run(STEPS)).await.unwrap();
|
||||
|
||||
// Every agent read its attachment and committed, in every mode.
|
||||
assert_eq!(f.harness.coordinator.committed_boundary(), Some(STEPS));
|
||||
|
||||
// The coordinator holds one view handle and one audio handle for this boundary, and the
|
||||
// observation names exactly those artifacts.
|
||||
let observation = f.harness.coordinator.observation().expect("an observation").clone();
|
||||
let handles = f.harness.coordinator.media_handles();
|
||||
assert_eq!(handles.len(), 2, "one view and one chunk: {handles:?}");
|
||||
let view = observation.sensory_views.first().expect("a required view");
|
||||
assert_eq!(
|
||||
view.produced_step,
|
||||
STEPS - 1,
|
||||
"the declared one-step delay reached the world in {mode:?} mode"
|
||||
);
|
||||
assert!(handles.iter().any(|(name, r)| *name == view_attachment(&view.view_id)
|
||||
&& r.artifact_id == view.pixels.artifact_id));
|
||||
|
||||
// The same object is what presentation was published, and its bytes read back at the
|
||||
// declared shape through an ordinary subscription.
|
||||
let published = frame_at(&mut spectator, STEPS).await;
|
||||
assert_eq!(published.view.pixels.artifact_id, view.pixels.artifact_id);
|
||||
assert_eq!(published.view.produced_step, view.produced_step);
|
||||
let pixels = published.artifact.read_all().await.expect("the published frame");
|
||||
assert_eq!(pixels.len() as u64, VIEW_WIDTH * VIEW_HEIGHT * 4);
|
||||
let (chunk, audio) = published.audio.first().expect("the published chunk");
|
||||
let samples = audio.read_all().await.expect("the published chunk");
|
||||
assert_eq!(samples.len() as u64, chunk.sample_frames * CHANNELS * 4);
|
||||
require_finite_samples(&samples).expect("native samples are finite f32");
|
||||
|
||||
// The shared-memory instrumentation exists exactly where the participants do.
|
||||
match (mode, f.harness.sensor_log(&fly_a()), f.harness.renders()) {
|
||||
(ExecutionMode::Process, sensors, renders) => {
|
||||
assert!(sensors.is_none() && renders.is_none(), "not observable from here");
|
||||
}
|
||||
(_, Some(sensors), Some(renders)) => {
|
||||
let a = sensors.entries();
|
||||
let b = f.harness.sensor_log(&fly_b()).expect("in this process").entries();
|
||||
assert_eq!(a, b, "both agents read the same artifact and the same bytes");
|
||||
assert_eq!(boundaries(&a), (0..=STEPS).collect::<Vec<_>>());
|
||||
assert_eq!(
|
||||
digest_of_bytes(&pixels),
|
||||
a.last().expect("an entry per boundary").digest,
|
||||
"the published bytes are the bytes the agents encoded"
|
||||
);
|
||||
assert_eq!(renders, STEPS + 1, "one render per boundary");
|
||||
}
|
||||
(mode, _, _) => panic!("{mode:?} keeps its participants in this process"),
|
||||
}
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// Every media fault fires wherever the world runs, which is what proves the wiring rather
|
||||
/// than assuming it.
|
||||
///
|
||||
/// A fault reaches a world in another process only as a `--flag` on its command line, so a
|
||||
/// renamed or dropped flag would make these faults silently stop firing. Here each one has to
|
||||
/// fail its transition in every execution mode; the parser refuses an unknown flag, so a
|
||||
/// mismatch fails the launch instead of turning into a no-op.
|
||||
async fn every_media_fault_fires_in_every_execution_mode(mode: ExecutionMode) {
|
||||
let cases: [(&str, EnvironmentFaults, u64); 4] = [
|
||||
(
|
||||
"an extra-delayed view",
|
||||
EnvironmentFaults {
|
||||
stale_view_at_boundary: Some(2),
|
||||
..EnvironmentFaults::default()
|
||||
},
|
||||
2,
|
||||
),
|
||||
(
|
||||
"a frame of the wrong length",
|
||||
EnvironmentFaults {
|
||||
truncated_view_at_boundary: Some(1),
|
||||
..EnvironmentFaults::default()
|
||||
},
|
||||
1,
|
||||
),
|
||||
(
|
||||
"a missing audio chunk",
|
||||
EnvironmentFaults {
|
||||
omit_audio_at_boundary: Some(2),
|
||||
..EnvironmentFaults::default()
|
||||
},
|
||||
2,
|
||||
),
|
||||
(
|
||||
"an overlapping audio chunk",
|
||||
EnvironmentFaults {
|
||||
overlapping_audio_at_boundary: Some(2),
|
||||
..EnvironmentFaults::default()
|
||||
},
|
||||
2,
|
||||
),
|
||||
];
|
||||
for (what, faults, steps) in cases {
|
||||
let config = HarnessConfig {
|
||||
environment_faults: faults,
|
||||
..HarnessConfig::default()
|
||||
};
|
||||
let mut f = mode_fixture(mode, config).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let outcome = within("run", f.harness.coordinator.run(steps)).await;
|
||||
let failure = match outcome {
|
||||
Err(failure) => failure,
|
||||
Ok(reports) => {
|
||||
panic!("{what} did not fail the transition in {mode:?} mode: {reports:?}")
|
||||
}
|
||||
};
|
||||
assert_eq!(
|
||||
failure.error.code,
|
||||
ErrorCode::BufferInvalid,
|
||||
"{what} in {mode:?} mode: {failure}"
|
||||
);
|
||||
assert_eq!(f.harness.coordinator.phase(), Phase::Failed);
|
||||
assert_eq!(
|
||||
f.harness.coordinator.stats().advances,
|
||||
steps - 1,
|
||||
"{what} in {mode:?} mode committed the boundaries before it and no more"
|
||||
);
|
||||
f.shutdown().await;
|
||||
}
|
||||
}
|
||||
|
||||
/// The retention table: a required agent input is retained through encoding and Commit with no
|
||||
/// coalescing, while a spectator's snapshots are a latest subscription with finite credits.
|
||||
async fn required_agent_input_is_never_coalesced_while_spectator_snapshots_are(via: Via) {
|
||||
let mut f = default_fixture(via).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let observer = f.harness.observer().await.unwrap();
|
||||
let topic = f.harness.coordinator.topics().snapshots.clone();
|
||||
let mut spectator = Spectator::attach(&observer, &topic, 1).await.unwrap();
|
||||
assert!(within("first snapshot", spectator.hold_one()).await);
|
||||
|
||||
let steps = 5;
|
||||
within("run", f.harness.coordinator.run(steps)).await.unwrap();
|
||||
|
||||
// The spectator's queue coalesced while it was not reading.
|
||||
spectator.release();
|
||||
within("after release", spectator.hold_one()).await;
|
||||
assert!(spectator.coalesced() > 0);
|
||||
assert!(
|
||||
spectator.seen() < steps + 1,
|
||||
"a latest subscription does not deliver every boundary to a slow reader"
|
||||
);
|
||||
|
||||
// Every agent's required input arrived once per boundary, in order, with nothing dropped
|
||||
// or replaced.
|
||||
for agent in [fly_a(), fly_b()] {
|
||||
let entries = sensed(&f, &agent);
|
||||
assert_eq!(
|
||||
boundaries(&entries),
|
||||
(0..=steps).collect::<Vec<_>>(),
|
||||
"{agent} encoded every boundary exactly once"
|
||||
);
|
||||
}
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// A persistent `AssetRef` names installed content; a transient `ArtifactRef` names live bytes
|
||||
/// in one store. Importing the asset makes a new artifact, and collecting that artifact leaves
|
||||
/// the asset installed.
|
||||
async fn a_persistent_asset_and_a_transient_artifact_are_different_identities(via: Via) {
|
||||
let mut f = default_fixture(via).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let client = f.harness.observer().await.unwrap();
|
||||
|
||||
let body = "counter-arena-backend-v1";
|
||||
let asset = synthetic_asset("counter-arena-backend", body);
|
||||
let mut registry = AssetRegistry::new();
|
||||
registry
|
||||
.install(asset.clone(), body.as_bytes().to_vec())
|
||||
.expect("installed content matches its reference");
|
||||
// Installing content that is not what the reference claims is refused.
|
||||
let mut wrong = asset.clone();
|
||||
wrong.byte_length += 1;
|
||||
registry
|
||||
.install(wrong, body.as_bytes().to_vec())
|
||||
.expect_err("an asset reference is its content's identity");
|
||||
|
||||
let before = f.harness.router().stats().sealed_artifacts;
|
||||
let artifact = registry.import(&client, &asset).await.expect("the import");
|
||||
assert_eq!(
|
||||
f.harness.router().stats().sealed_artifacts,
|
||||
before + 1,
|
||||
"the import is a new object in the store"
|
||||
);
|
||||
|
||||
// The identities are different, and the content is the same.
|
||||
assert_ne!(artifact.reference().artifact_id, asset.id);
|
||||
assert_eq!(artifact.reference().byte_length, asset.byte_length);
|
||||
assert_eq!(artifact.reference().digest.as_deref(), Some(asset.digest.as_str()));
|
||||
check_imported_asset(&asset, artifact.reference()).expect("the import carries the content");
|
||||
assert_eq!(
|
||||
artifact.read_all().await.expect("readable"),
|
||||
body.as_bytes(),
|
||||
"the imported artifact is the installed bytes"
|
||||
);
|
||||
|
||||
// The transient artifact is collected with its last handle; the asset is still installed.
|
||||
drop(artifact);
|
||||
let router = f.harness.router().clone();
|
||||
until("the imported artifact is collected", || {
|
||||
router.stats().sealed_artifacts == before
|
||||
})
|
||||
.await;
|
||||
assert!(registry.contains(&asset));
|
||||
assert_eq!(
|
||||
registry.resolve(&asset).expect("still installed"),
|
||||
body.as_bytes(),
|
||||
"a persistent asset outlives the bus objects imported from it"
|
||||
);
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// A restored epoch resumes the preserved sample position, and its first chunk marks the
|
||||
/// discontinuity that says so. The producer and the validator agree about which epoch it is.
|
||||
async fn a_restored_audio_source_resumes_and_marks_the_discontinuity(via: Via) {
|
||||
let f = default_fixture(via).await;
|
||||
let client = f.harness.observer().await.unwrap();
|
||||
let descriptor = fly_session::environment::CounterEnvironment::audio_descriptor();
|
||||
let step = hz(f.harness.config.step_hz).expect("a positive cadence");
|
||||
let resumed_at = 2_400;
|
||||
|
||||
let mut source = AudioSource::restored_at(descriptor.clone(), resumed_at);
|
||||
let (first, _first_handle) = source
|
||||
.produce(&client, &step, 3)
|
||||
.await
|
||||
.expect("the restored source produces");
|
||||
assert_eq!(first.first_sample, resumed_at, "the sample position is preserved");
|
||||
assert!(first.discontinuity, "the first chunk after a restore marks it");
|
||||
let (second, _second_handle) = source
|
||||
.produce(&client, &step, 3)
|
||||
.await
|
||||
.expect("the next chunk");
|
||||
assert!(!second.discontinuity, "only the first chunk of the epoch marks it");
|
||||
assert_eq!(second.first_sample, resumed_at + first.sample_frames);
|
||||
|
||||
// The validator accepts exactly this sequence under a restored timeline, and refuses it
|
||||
// under a fresh one: the flag is what distinguishes the two epochs.
|
||||
let mut restored = AudioTimeline::restored_at(&descriptor, resumed_at);
|
||||
restored.accept(&first, &descriptor).expect("the restored epoch");
|
||||
restored.accept(&second, &descriptor).expect("and its next chunk");
|
||||
// A fresh episode at the audio origin refuses it: a restore preserves the sample
|
||||
// position, and the position is what distinguishes the two epochs. The flag is required
|
||||
// after a restore and free at an origin, so it cannot carry that distinction by itself.
|
||||
let mut fresh = AudioTimeline::fresh(&descriptor, 0);
|
||||
fresh
|
||||
.accept(&first, &descriptor)
|
||||
.expect_err("a fresh episode starts at its own origin, not a resumed position");
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// The sample budget is exact when the cadence does not divide the sample rate: 8 kHz at 60 Hz
|
||||
/// is 133, 133, 134 and the total after three steps is exactly 400.
|
||||
#[test]
|
||||
fn the_sample_budget_is_exact_when_the_cadence_does_not_divide_the_rate() {
|
||||
let descriptor = fly_session_types::media::AudioDescriptor {
|
||||
stream_id: "arena".into(),
|
||||
sample_rate: 8_000,
|
||||
channels: 1,
|
||||
};
|
||||
let mut source = AudioSource::new(descriptor, 0);
|
||||
let step = hz(60).expect("a positive cadence");
|
||||
let frames: Vec<u64> = (0..6)
|
||||
.map(|_| source.frames_for_step(&step).expect("an exact budget"))
|
||||
.collect();
|
||||
assert_eq!(frames, vec![133, 133, 134, 133, 133, 134]);
|
||||
assert_eq!(frames.iter().sum::<u64>(), 800, "8000 samples in a tenth of a second");
|
||||
|
||||
// 48 kHz at 60 Hz divides exactly.
|
||||
let exact = fly_session_types::media::AudioDescriptor {
|
||||
stream_id: "arena".into(),
|
||||
sample_rate: SAMPLE_RATE,
|
||||
channels: CHANNELS,
|
||||
};
|
||||
let mut source = AudioSource::new(exact, 0);
|
||||
assert_eq!(source.frames_for_step(&step).unwrap(), 800);
|
||||
}
|
||||
|
||||
/// The native frame is a real pattern with no padded rows: `rowStride` is `4 x width`, the
|
||||
/// rows are top-left first, and consecutive boundaries differ.
|
||||
#[test]
|
||||
fn the_native_frame_is_top_left_rgba8_with_no_padding() {
|
||||
let descriptor = fly_session::environment::CounterEnvironment::view_descriptor(0);
|
||||
let frame = arena_frame(&descriptor, 5, 3);
|
||||
assert_eq!(frame.len() as u64, descriptor.row_stride * descriptor.height);
|
||||
assert_eq!(descriptor.row_stride, descriptor.width * 4);
|
||||
// Every pixel is opaque and carries the world counter in its red channel.
|
||||
for pixel in frame.chunks_exact(4) {
|
||||
assert_eq!(pixel[0], 5);
|
||||
assert_eq!(pixel[3], 255);
|
||||
}
|
||||
assert_ne!(
|
||||
arena_frame(&descriptor, 5, 3),
|
||||
arena_frame(&descriptor, 5, 4),
|
||||
"consecutive boundaries are different images"
|
||||
);
|
||||
assert_ne!(
|
||||
arena_frame(&descriptor, 5, 3),
|
||||
arena_frame(&descriptor, 6, 3),
|
||||
"the counter changes the image"
|
||||
);
|
||||
}
|
||||
823
services/flysim/crates/fly-session/tests/processes.rs
Normal file
823
services/flysim/crates/fly-session/tests/processes.rs
Normal file
|
|
@ -0,0 +1,823 @@
|
|||
//! SESSION-02 acceptance: one agent process per fly and one environment process under the
|
||||
//! coordinator, compared with the in-process and dedicated-thread variants.
|
||||
//!
|
||||
//! Every acceptance bullet is one named test here, generated once per execution mode, so a
|
||||
//! rule that holds in one process holds across a process boundary too. The two process-mode
|
||||
//! failure rows of section 4 that SESSION-01 could not reach in one process -- a router
|
||||
//! restart during a world advance, and an old worker's reply after a restart -- are at the
|
||||
//! end and run in the separate-process mode.
|
||||
|
||||
mod common;
|
||||
|
||||
use std::collections::BTreeMap;
|
||||
use std::time::{Duration, Instant};
|
||||
|
||||
use common::{at, count, fly_a, fly_b, mode_fixture, within};
|
||||
use fly_session::agent::AgentFaults;
|
||||
use fly_session::coordinator::{DispatchOrder, Injections};
|
||||
use fly_session::environment::EnvironmentFaults;
|
||||
use fly_session::harness::{ExecutionMode, HarnessConfig, Via};
|
||||
use fly_session::launcher::{ReapOutcome, ThreadBudget};
|
||||
use fly_session::ResolutionEnd;
|
||||
use fly_session::phase::Phase;
|
||||
use fly_session::types::*;
|
||||
|
||||
all_modes!(
|
||||
a_slow_participant_is_resolved_rather_than_failed,
|
||||
a_resolution_says_which_of_its_two_bounds_ended_it,
|
||||
a_delayed_one_agent_result_holds_the_world,
|
||||
a_worker_death_has_a_bounded_diagnosed_outcome,
|
||||
a_helper_death_has_a_bounded_diagnosed_outcome,
|
||||
an_uncertain_advance_never_creates_a_second_batch,
|
||||
a_partial_commit_never_permits_next_step_play,
|
||||
every_participant_answers_its_supervisor,
|
||||
worker_threads_lie_within_the_launcher_allocation,
|
||||
);
|
||||
|
||||
const STEPS: u64 = 4;
|
||||
|
||||
fn two_agents(mode: ExecutionMode) -> HarnessConfig {
|
||||
HarnessConfig { mode, ..HarnessConfig::default() }
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------------------------
|
||||
// Acceptance: sequential, reversed and parallel completion produce equivalent traces
|
||||
|
||||
/// `step-v1` section 8, across the process boundary: sequential, concurrent and reversed
|
||||
/// dispatch, in all three execution modes, produce one behaviour trace.
|
||||
///
|
||||
/// This reuses the wave-1 comparator -- the behaviour half of the section 8 trace, with
|
||||
/// request ids, bus correlation and wall time excluded -- so "a process behaves like a task"
|
||||
/// is the same assertion that "a reordered dispatch behaves like an ordered one" was.
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn sequential_reversed_and_parallel_completion_agree() {
|
||||
let mut behaviours: BTreeMap<String, Vec<String>> = BTreeMap::new();
|
||||
for mode in ExecutionMode::all() {
|
||||
for order in [
|
||||
DispatchOrder::Sequential,
|
||||
DispatchOrder::Concurrent,
|
||||
DispatchOrder::Reversed,
|
||||
] {
|
||||
let mut config = two_agents(mode);
|
||||
// Deliberately unequal completion times, so a concurrent run really does finish
|
||||
// out of dispatch order whichever side of a process boundary the agents are on.
|
||||
config.agents[0].faults =
|
||||
AgentFaults { prepare_delay_ms: 12, ..AgentFaults::default() };
|
||||
config.agents[1].faults = AgentFaults { commit_delay_ms: 9, ..AgentFaults::default() };
|
||||
let mut f = mode_fixture(mode, config).await;
|
||||
f.harness.coordinator.dispatch = order;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
within("run", f.harness.coordinator.run(STEPS)).await.unwrap();
|
||||
let behaviour = f.harness.coordinator.trace.behavior();
|
||||
assert_eq!(behaviour.len() as u64, STEPS);
|
||||
behaviours.insert(format!("{}/{order:?}", mode.label()), behaviour);
|
||||
f.shutdown().await;
|
||||
}
|
||||
}
|
||||
let mut iter = behaviours.iter();
|
||||
let (first_name, first) = iter.next().expect("at least one run");
|
||||
for (name, behaviour) in iter {
|
||||
assert_eq!(
|
||||
behaviour, first,
|
||||
"{name} produced a different behaviour trace from {first_name}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------------------------
|
||||
// ipc-v1 section 6: an uncertain call is resolved, not failed
|
||||
|
||||
/// A participant that is merely slow -- slower than the caller's probe, faster than the
|
||||
/// resolution's budget -- finishes its step. The epoch is not lost, and the resolution adds no
|
||||
/// second operation.
|
||||
///
|
||||
/// This is the `ipc-v1` section 6 procedure on the path that actually reaches it: the probe
|
||||
/// expires, the coordinator queries the same request id against the same incarnation, the
|
||||
/// worker answers `IN_PROGRESS` while its original is still running and then replays its
|
||||
/// cached reply. `step-v1` section 7's Advance row is the same rule, so the world is slow here
|
||||
/// too and its batch is never re-sent as a new one.
|
||||
async fn a_slow_participant_is_resolved_rather_than_failed(mode: ExecutionMode) {
|
||||
// A clean run of the same composition, to compare against.
|
||||
let clean = {
|
||||
let mut f = mode_fixture(mode, two_agents(mode)).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
within("run", f.harness.coordinator.run(2)).await.unwrap();
|
||||
let environment = f.harness.environment_id();
|
||||
let world = within("progress", f.harness.progress_of(&environment)).await.unwrap();
|
||||
let out = (f.harness.coordinator.trace.behavior(), world);
|
||||
f.shutdown().await;
|
||||
out
|
||||
};
|
||||
|
||||
let mut config = two_agents(mode);
|
||||
config.agents[1].faults = AgentFaults { prepare_delay_ms: 500, ..AgentFaults::default() };
|
||||
config.environment_faults =
|
||||
EnvironmentFaults { advance_delay_ms: 500, ..EnvironmentFaults::default() };
|
||||
let mut f = mode_fixture(mode, config).await;
|
||||
// A probe well inside both delays, and a resolution budget well outside them: the point is
|
||||
// a call that expires and an operation that is nevertheless fine.
|
||||
f.harness.coordinator.deadlines = fly_session::Deadlines {
|
||||
probe: Duration::from_millis(120),
|
||||
resolve: Duration::from_secs(20),
|
||||
resolve_attempts: 4096,
|
||||
boot: Duration::from_secs(30),
|
||||
};
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let reports = within("run", f.harness.coordinator.run(2))
|
||||
.await
|
||||
.expect("a slow participant is resolved, not failed");
|
||||
|
||||
assert_eq!(reports.len(), 2);
|
||||
assert_eq!(f.harness.coordinator.phase(), Phase::Ready(2));
|
||||
assert!(!f.harness.coordinator.is_fenced(), "a slow answer is not a lost epoch");
|
||||
assert!(
|
||||
f.harness.coordinator.resolutions >= 2,
|
||||
"both the slow Prepare and the slow Advance must have run the resolution, not {}",
|
||||
f.harness.coordinator.resolutions
|
||||
);
|
||||
assert!(
|
||||
f.harness.coordinator.in_progress_replies > 0,
|
||||
"the resolution must have met the original still running"
|
||||
);
|
||||
assert_eq!(
|
||||
f.harness.coordinator.last_resolution,
|
||||
Some(ResolutionEnd::Answered),
|
||||
"the resolution ended by being answered, not by running out of anything"
|
||||
);
|
||||
|
||||
// No second operation anywhere: one advance per transition, one batch id per transition,
|
||||
// and the same behaviour as the run that never timed out.
|
||||
assert_eq!(f.harness.coordinator.stats().advances, 2);
|
||||
let environment = f.harness.environment_id();
|
||||
let world = within("progress", f.harness.progress_of(&environment)).await.unwrap();
|
||||
assert_eq!(world, clean.1, "the world moved exactly as often as in the clean run");
|
||||
assert_eq!(
|
||||
f.harness.coordinator.trace.behavior(),
|
||||
clean.0,
|
||||
"resolving an uncertain call changes no behaviour"
|
||||
);
|
||||
let batches: std::collections::BTreeSet<Id> = f
|
||||
.harness
|
||||
.coordinator
|
||||
.trace
|
||||
.transitions
|
||||
.iter()
|
||||
.map(|t| t.behaviour.batch_id.clone())
|
||||
.collect();
|
||||
assert_eq!(batches.len(), 2, "one batch id per transition, never a second batch");
|
||||
// And the agents took exactly the ticks the clean run took: a resolution is a query.
|
||||
for transition in &f.harness.coordinator.trace.transitions {
|
||||
for agent in &transition.behaviour.agents {
|
||||
assert!(agent.ticks_advanced == 16 || agent.ticks_advanced == 17);
|
||||
}
|
||||
}
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// The resolution has two bounds, and which one ended it is never left to be guessed.
|
||||
///
|
||||
/// `resolve` is the working limit at the default values -- the attempt guard is over sixteen
|
||||
/// seconds of pauses against an eight-second budget -- so an unresponsive participant runs the
|
||||
/// budget out. Setting the guard low instead ends the same resolution the other way, and the
|
||||
/// failure says so both in `last_resolution` and in its own message.
|
||||
async fn a_resolution_says_which_of_its_two_bounds_ended_it(mode: ExecutionMode) {
|
||||
// The budget is what ends it at ordinary settings: a generous attempt guard, a short
|
||||
// budget, and a participant far slower than either.
|
||||
let mut config = two_agents(mode);
|
||||
config.agents[1].faults = AgentFaults { prepare_delay_ms: 30_000, ..AgentFaults::default() };
|
||||
let mut f = mode_fixture(mode, config).await;
|
||||
f.harness.coordinator.deadlines = fly_session::Deadlines {
|
||||
probe: Duration::from_millis(50),
|
||||
resolve: Duration::from_millis(300),
|
||||
resolve_attempts: 8192,
|
||||
boot: Duration::from_secs(30),
|
||||
};
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let started = Instant::now();
|
||||
let failure = within("step", f.harness.coordinator.step())
|
||||
.await
|
||||
.expect_err("a participant that never answers exhausts the resolution");
|
||||
assert_eq!(f.harness.coordinator.last_resolution, Some(ResolutionEnd::BudgetExpired));
|
||||
assert!(
|
||||
failure.error.message.contains("resolution budget"),
|
||||
"the message names the bound that fired: {failure}"
|
||||
);
|
||||
assert_eq!(failure.participant.as_deref(), Some(fly_b().as_str()));
|
||||
assert_eq!(failure.error.mutation, MutationCertainty::Unknown);
|
||||
assert!(
|
||||
started.elapsed() < Duration::from_secs(20),
|
||||
"the budget, not the 30-second participant, is what ended it"
|
||||
);
|
||||
assert!(f.harness.coordinator.is_fenced());
|
||||
f.shutdown().await;
|
||||
|
||||
// The guard is what ends it when it is set below the budget: three attempts against a
|
||||
// budget the participant could never reach anyway.
|
||||
let mut config = two_agents(mode);
|
||||
config.agents[1].faults = AgentFaults { prepare_delay_ms: 30_000, ..AgentFaults::default() };
|
||||
let mut f = mode_fixture(mode, config).await;
|
||||
f.harness.coordinator.deadlines = fly_session::Deadlines {
|
||||
probe: Duration::from_millis(50),
|
||||
resolve: Duration::from_secs(600),
|
||||
resolve_attempts: 3,
|
||||
boot: Duration::from_secs(30),
|
||||
};
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let failure = within("step", f.harness.coordinator.step())
|
||||
.await
|
||||
.expect_err("three attempts are not enough to resolve a silent participant");
|
||||
assert_eq!(f.harness.coordinator.last_resolution, Some(ResolutionEnd::AttemptsExhausted));
|
||||
assert!(
|
||||
failure.error.message.contains("attempt guard") && failure.error.message.contains("3 attempts"),
|
||||
"the message names the bound that fired and its size: {failure}"
|
||||
);
|
||||
assert_eq!(failure.participant.as_deref(), Some(fly_b().as_str()));
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------------------------
|
||||
// Acceptance: a delayed one-agent result holds the world
|
||||
|
||||
/// One agent takes far longer than the other to prepare. No `Environment.Advance` is sent
|
||||
/// until every agent is Prepared, and the world is still at its old boundary while the
|
||||
/// coordinator waits.
|
||||
async fn a_delayed_one_agent_result_holds_the_world(mode: ExecutionMode) {
|
||||
let mut config = two_agents(mode);
|
||||
config.agents[1].faults = AgentFaults { prepare_delay_ms: 400, ..AgentFaults::default() };
|
||||
let mut f = mode_fixture(mode, config).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let environment = f.harness.environment_id();
|
||||
let before = within("progress", f.harness.progress_of(&environment)).await.unwrap();
|
||||
|
||||
let (coordinator, launcher) = f.harness.parts();
|
||||
// The supervisor watches the world while the transition is in flight. That is what a
|
||||
// supervisor is for, and `Worker.Status` answers without waiting for a mutation.
|
||||
let (stepped, held) = tokio::join!(
|
||||
async { within("step", coordinator.step()).await },
|
||||
async {
|
||||
tokio::time::sleep(Duration::from_millis(120)).await;
|
||||
within("status", launcher.health_check(&environment)).await
|
||||
}
|
||||
);
|
||||
let report = stepped.expect("the transition completes once the slow agent answers");
|
||||
assert_eq!(report.boundary, 1);
|
||||
let held = held.expect("the environment answers its supervisor during the wait");
|
||||
assert_eq!(
|
||||
held.progress_counter, before,
|
||||
"the world may not advance while one agent is still preparing"
|
||||
);
|
||||
assert_eq!(
|
||||
held.state,
|
||||
WorkerState::Ready,
|
||||
"the environment is at a committed boundary, not advancing"
|
||||
);
|
||||
|
||||
// And the ordering the audit records says the same thing from the coordinator's side.
|
||||
let audit = f.harness.coordinator.audit.clone();
|
||||
let advance = at(&audit, "advance:0");
|
||||
for agent in [fly_a(), fly_b()] {
|
||||
assert!(
|
||||
at(&audit, &format!("prepared:{agent}@0")) < advance,
|
||||
"{agent} must be Prepared before the world advances: {audit:?}"
|
||||
);
|
||||
}
|
||||
assert_eq!(f.harness.coordinator.stats().advances, 1);
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------------------------
|
||||
// Acceptance: worker or helper death has a bounded diagnosed outcome
|
||||
|
||||
/// One agent dies in the middle of its Prepare. The epoch fails with a typed cause naming
|
||||
/// that agent, within the caller's own budget, and nothing continues on the remainder.
|
||||
async fn a_worker_death_has_a_bounded_diagnosed_outcome(mode: ExecutionMode) {
|
||||
let mut config = two_agents(mode);
|
||||
config.agents[1].faults = AgentFaults { prepare_delay_ms: 5_000, ..AgentFaults::default() };
|
||||
let mut f = mode_fixture(mode, config).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let started = Instant::now();
|
||||
|
||||
let (coordinator, launcher) = f.harness.parts();
|
||||
let (stepped, reaped) = tokio::join!(
|
||||
async { within("step", coordinator.step()).await },
|
||||
async {
|
||||
tokio::time::sleep(Duration::from_millis(80)).await;
|
||||
launcher.kill(&fly_b()).await
|
||||
}
|
||||
);
|
||||
assert_eq!(reaped, ReapOutcome::Terminated);
|
||||
let failure = stepped.expect_err("a dead participant is a failed epoch, not a slow one");
|
||||
assert!(
|
||||
started.elapsed() < Duration::from_secs(20),
|
||||
"the outcome must be bounded, not a hang"
|
||||
);
|
||||
assert_eq!(
|
||||
failure.participant.as_deref(),
|
||||
Some(fly_b().as_str()),
|
||||
"the failure names the participant: {failure}"
|
||||
);
|
||||
assert_ne!(
|
||||
failure.error.mutation,
|
||||
MutationCertainty::None,
|
||||
"a participant that died mid-call leaves an uncertain mutation, never a clean none"
|
||||
);
|
||||
assert_eq!(f.harness.coordinator.phase(), Phase::Failed);
|
||||
assert!(f.harness.coordinator.is_fenced());
|
||||
// No partial continuation: no world step, no publication, and no next transition.
|
||||
assert_eq!(f.harness.coordinator.stats().advances, 0);
|
||||
assert_eq!(count(&f.harness.coordinator.audit, "publish:1"), 0);
|
||||
let again = f.harness.coordinator.step().await.expect_err("a fenced epoch takes no step");
|
||||
assert_eq!(again.error.code, ErrorCode::InvalidPhase);
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// The environment helper dies in the middle of the world advance. Same rule: a typed cause
|
||||
/// naming it, bounded, and no half-transition afterwards.
|
||||
async fn a_helper_death_has_a_bounded_diagnosed_outcome(mode: ExecutionMode) {
|
||||
let config = HarnessConfig {
|
||||
environment_faults: EnvironmentFaults {
|
||||
advance_delay_ms: 5_000,
|
||||
..EnvironmentFaults::default()
|
||||
},
|
||||
..two_agents(mode)
|
||||
};
|
||||
let mut f = mode_fixture(mode, config).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let environment = f.harness.environment_id();
|
||||
let started = Instant::now();
|
||||
|
||||
let (coordinator, launcher) = f.harness.parts();
|
||||
let (stepped, reaped) = tokio::join!(
|
||||
async { within("step", coordinator.step()).await },
|
||||
async {
|
||||
tokio::time::sleep(Duration::from_millis(200)).await;
|
||||
launcher.kill(&environment).await
|
||||
}
|
||||
);
|
||||
assert_eq!(reaped, ReapOutcome::Terminated);
|
||||
let failure = stepped.expect_err("a dead world is a failed epoch");
|
||||
assert!(started.elapsed() < Duration::from_secs(20), "bounded, not a hang");
|
||||
assert_eq!(
|
||||
failure.participant.as_deref(),
|
||||
Some(environment.as_str()),
|
||||
"the failure names the participant: {failure}"
|
||||
);
|
||||
assert_ne!(failure.error.mutation, MutationCertainty::None);
|
||||
assert_eq!(f.harness.coordinator.phase(), Phase::Failed);
|
||||
assert!(f.harness.coordinator.is_fenced());
|
||||
assert_eq!(f.harness.coordinator.stats().advances, 0);
|
||||
// The agents prepared and are not asked to prepare again or to commit anything.
|
||||
assert_eq!(f.harness.coordinator.stats().commits, 0);
|
||||
assert_eq!(count(&f.harness.coordinator.audit, "publish:1"), 0);
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------------------------
|
||||
// Acceptance: an uncertain Advance never creates a second batch
|
||||
|
||||
/// The Advance result is lost after the world already stepped. The coordinator resolves the
|
||||
/// same operation against its original domain request id; the world advances once per
|
||||
/// transition and the batch is never re-sent as a new one.
|
||||
async fn an_uncertain_advance_never_creates_a_second_batch(mode: ExecutionMode) {
|
||||
let clean = {
|
||||
let mut f = mode_fixture(mode, two_agents(mode)).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
within("run", f.harness.coordinator.run(STEPS)).await.unwrap();
|
||||
let environment = f.harness.environment_id();
|
||||
let world = within("progress", f.harness.progress_of(&environment)).await.unwrap();
|
||||
let out = (f.harness.coordinator.trace.behavior(), world);
|
||||
f.shutdown().await;
|
||||
out
|
||||
};
|
||||
|
||||
let mut f = mode_fixture(mode, two_agents(mode)).await;
|
||||
f.harness.coordinator.injections = Injections {
|
||||
at_step: 2,
|
||||
lose_advance_result: true,
|
||||
..Injections::default()
|
||||
};
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
within("run", f.harness.coordinator.run(STEPS)).await.unwrap();
|
||||
let environment = f.harness.environment_id();
|
||||
let world = within("progress", f.harness.progress_of(&environment)).await.unwrap();
|
||||
|
||||
assert_eq!(f.harness.coordinator.stats().advances, STEPS, "one advance per transition");
|
||||
assert_eq!(
|
||||
world, clean.1,
|
||||
"the world moved exactly as often as it did without the loss"
|
||||
);
|
||||
assert_eq!(
|
||||
f.harness.coordinator.trace.behavior(),
|
||||
clean.0,
|
||||
"an uncertain Advance changes no behaviour, so it created no second batch"
|
||||
);
|
||||
// Every transition has exactly one batch, and every batch id is its own.
|
||||
let batches: Vec<Id> = f
|
||||
.harness
|
||||
.coordinator
|
||||
.trace
|
||||
.transitions
|
||||
.iter()
|
||||
.map(|t| t.behaviour.batch_id.clone())
|
||||
.collect();
|
||||
let unique: std::collections::BTreeSet<Id> = batches.iter().cloned().collect();
|
||||
assert_eq!(unique.len(), batches.len(), "one batch id per transition: {batches:?}");
|
||||
let injections = f.harness.coordinator.injection_log.clone();
|
||||
assert!(
|
||||
injections.iter().any(|o| o.what == "lost-advance-result" && o.identical),
|
||||
"the loss must happen after dispatch, so the outcome really is uncertain: {injections:?}"
|
||||
);
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------------------------
|
||||
// Acceptance: a partial Commit never permits next-step play
|
||||
|
||||
/// One agent's Commit fails after the other's succeeded. The epoch fails naming that agent,
|
||||
/// the boundary does not move, nothing is published and there is no next transition.
|
||||
async fn a_partial_commit_never_permits_next_step_play(mode: ExecutionMode) {
|
||||
let mut config = two_agents(mode);
|
||||
config.agents[1].faults =
|
||||
AgentFaults { fail_commit_at_step: Some(1), ..AgentFaults::default() };
|
||||
let mut f = mode_fixture(mode, config).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
within("step", f.harness.coordinator.step()).await.unwrap();
|
||||
let environment = f.harness.environment_id();
|
||||
|
||||
let failure = within("step", f.harness.coordinator.step())
|
||||
.await
|
||||
.expect_err("one failed Commit fails the epoch");
|
||||
assert_eq!(
|
||||
failure.participant.as_deref(),
|
||||
Some(fly_b().as_str()),
|
||||
"the failure names the agent whose Commit failed: {failure}"
|
||||
);
|
||||
assert_eq!(f.harness.coordinator.phase(), Phase::Failed);
|
||||
assert!(f.harness.coordinator.is_fenced());
|
||||
|
||||
// The world moved once inside the failing transition -- the Advance is what the Commit
|
||||
// follows -- and it moves no further. There is no next-step play on a partial commit.
|
||||
let world_before = within("progress", f.harness.progress_of(&environment)).await.unwrap();
|
||||
let again = f.harness.coordinator.step().await.expect_err("no play after a partial commit");
|
||||
assert_eq!(again.error.code, ErrorCode::InvalidPhase);
|
||||
let world_after = within("progress", f.harness.progress_of(&environment)).await.unwrap();
|
||||
assert_eq!(world_after, world_before, "no next world step follows a partial commit");
|
||||
let status = within("status", f.harness.launcher.health_check(&environment)).await.unwrap();
|
||||
assert_eq!(
|
||||
status.current_scope.unwrap().step,
|
||||
2,
|
||||
"the world stays at the boundary the failed transition reached"
|
||||
);
|
||||
let audit = f.harness.coordinator.audit.clone();
|
||||
assert_eq!(count(&audit, "publish:2"), 0);
|
||||
assert_eq!(f.harness.coordinator.committed_boundary(), None);
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------------------------
|
||||
// Supervision: identity, health and reaping
|
||||
|
||||
/// Every participant answers the supervisor with the identity the launcher configured, and
|
||||
/// stops when it is asked to.
|
||||
async fn every_participant_answers_its_supervisor(mode: ExecutionMode) {
|
||||
let mut f = mode_fixture(mode, two_agents(mode)).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
within("run", f.harness.coordinator.run(2)).await.unwrap();
|
||||
|
||||
let environment = f.harness.environment_id();
|
||||
for who in [fly_a(), fly_b(), environment.clone()] {
|
||||
let worker = f.harness.launcher.worker(&who).expect("a launched participant");
|
||||
assert_eq!(worker.identity.worker_id, who);
|
||||
assert_eq!(worker.domain_incarnation, worker.identity.incarnation_id);
|
||||
assert!(!worker.service_incarnation.is_empty());
|
||||
let status = within("health", f.harness.launcher.health_check(&who)).await.unwrap();
|
||||
assert_eq!(status.state, WorkerState::Ready, "{who} is healthy at a boundary");
|
||||
}
|
||||
// Every participant reports the allocation its launcher gave it, which is the wire the
|
||||
// 2026-09-22 `workers-v1` amendment added. The launcher refused anything else at start,
|
||||
// so a caller reads it here rather than being told it out of band.
|
||||
for who in [fly_a(), fly_b(), environment.clone()] {
|
||||
let worker = f.harness.coordinator.agent_ref(&who).cloned().unwrap_or_else(|| {
|
||||
f.harness.coordinator.environment_ref().clone()
|
||||
});
|
||||
let params = serde_json::json!({
|
||||
"sessionId": "demo",
|
||||
"expectedWorkerId": who.as_str(),
|
||||
"role": if who == environment { "environment" } else { "agent" },
|
||||
"supportedMajors": [1],
|
||||
});
|
||||
let result = within(
|
||||
"hello",
|
||||
f.harness.coordinator.probe_raw(&worker, "Worker.Hello", None, params),
|
||||
)
|
||||
.await
|
||||
.expect("a worker answers its own identity");
|
||||
let reported = result["limits"]["workerThreads"].as_u64();
|
||||
assert_eq!(
|
||||
reported,
|
||||
Some(f.harness.launcher.worker(&who).unwrap().identity.worker_threads as u64),
|
||||
"{who} must report the allocation its launcher gave it"
|
||||
);
|
||||
}
|
||||
|
||||
// The agents carry their configured port identities; the environment owns the ports.
|
||||
assert_eq!(
|
||||
f.harness.launcher.worker(&fly_a()).unwrap().identity.port_id.as_deref(),
|
||||
Some("p1")
|
||||
);
|
||||
assert_eq!(
|
||||
f.harness.launcher.worker(&fly_b()).unwrap().identity.port_id.as_deref(),
|
||||
Some("p2")
|
||||
);
|
||||
assert!(f.harness.launcher.worker(&environment).unwrap().identity.port_id.is_none());
|
||||
|
||||
// A worker that is not the one the caller expects refuses to negotiate at all.
|
||||
let worker = f.harness.coordinator.agent_ref(&fly_a()).cloned().unwrap();
|
||||
let wrong = serde_json::json!({
|
||||
"sessionId": "demo",
|
||||
"expectedWorkerId": "fly-z",
|
||||
"role": "agent",
|
||||
"supportedMajors": [1],
|
||||
});
|
||||
let err = within(
|
||||
"hello",
|
||||
f.harness.coordinator.probe_raw(&worker, "Worker.Hello", None, wrong),
|
||||
)
|
||||
.await
|
||||
.expect_err("a worker is not whoever a caller says it is");
|
||||
assert_eq!(err.code, ErrorCode::IdentityMismatch);
|
||||
|
||||
// Asking a participant to stop stops it, and the supervisor says which kind of stop it was.
|
||||
let outcome = f.harness.launcher.reap(&fly_a(), &id("test")).await;
|
||||
assert_eq!(outcome, ReapOutcome::Stopped, "a live participant answers Worker.Shutdown");
|
||||
assert_eq!(
|
||||
f.harness.launcher.reap(&fly_a(), &id("test")).await,
|
||||
ReapOutcome::AlreadyGone
|
||||
);
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// `workers-v1`: `Agent.Initialize`'s `workerThreads` lies within the launcher allocation.
|
||||
///
|
||||
/// The budget refuses an allocation it cannot cover before anything is started, and an agent
|
||||
/// refuses an `Agent.Initialize` asking for more threads than its launcher gave it.
|
||||
async fn worker_threads_lie_within_the_launcher_allocation(mode: ExecutionMode) {
|
||||
// The budget itself: a total, a coordinator reservation, and a refusal that names both.
|
||||
let mut budget = ThreadBudget::new(4, 1).unwrap();
|
||||
assert_eq!(budget.remaining(), 3);
|
||||
assert_eq!(budget.allocate(&id("arena"), 1).unwrap(), 1);
|
||||
assert_eq!(budget.allocate(&id("fly-a"), 2).unwrap(), 2);
|
||||
let refused = budget.allocate(&id("fly-b"), 1).expect_err("the budget is spent");
|
||||
assert_eq!(refused.code, ErrorCode::Busy);
|
||||
budget.release(&id("fly-a"));
|
||||
assert_eq!(budget.allocate(&id("fly-b"), 1).unwrap(), 1);
|
||||
assert_eq!(budget.allocate(&id("fly-b"), 1).expect_err("already held").code, ErrorCode::Conflict);
|
||||
|
||||
// A composition the configured budget cannot cover never starts.
|
||||
let config = HarnessConfig {
|
||||
mode,
|
||||
thread_budget: Some(2),
|
||||
..HarnessConfig::default()
|
||||
};
|
||||
let dir = tempfile::tempdir().expect("a temporary directory");
|
||||
let refused = fly_session::harness::SessionHarness::start(Via::Unix, dir.path(), config).await;
|
||||
let refused = refused.err().expect("two threads cannot hold a coordinator, a world and two flies");
|
||||
assert_eq!(refused.code, flybus::ErrorCode::QuotaExceeded, "{}", refused.message);
|
||||
drop(dir);
|
||||
|
||||
// And the worker's own check: it was launched with one thread, so an Initialize asking
|
||||
// for eight is refused before the model is constructed.
|
||||
let mut f = mode_fixture(mode, two_agents(mode)).await;
|
||||
let worker = f.harness.coordinator.agent_ref(&fly_a()).cloned().unwrap();
|
||||
let profile = fly_session::agent::synthetic_profile(
|
||||
&fly_a(),
|
||||
&millis(1).unwrap(),
|
||||
f.harness.config.warmup_ticks,
|
||||
);
|
||||
let params = serde_json::json!({
|
||||
"agentId": "fly-a",
|
||||
"profile": profile.to_json(),
|
||||
"seed": 7,
|
||||
"initialInput": {"boundary": "0", "views": [], "structured": null},
|
||||
"initialDecisionContext": {
|
||||
"schema": fly_session::task::context_schema().to_json(),
|
||||
"value": {},
|
||||
},
|
||||
"workerThreads": 8,
|
||||
});
|
||||
let err = within(
|
||||
"initialize",
|
||||
f.harness.coordinator.probe_raw(
|
||||
&worker,
|
||||
"Agent.Initialize",
|
||||
Some(scope_at("demo", "e1", 0)),
|
||||
params,
|
||||
),
|
||||
)
|
||||
.await
|
||||
.expect_err("eight threads are not within a one-thread allocation");
|
||||
assert_eq!(err.code, ErrorCode::Busy);
|
||||
assert_eq!(err.mutation, MutationCertainty::None, "nothing was constructed");
|
||||
// The allocation the coordinator actually sends is the one the launcher handed out.
|
||||
assert_eq!(f.harness.launcher.worker(&fly_a()).unwrap().identity.worker_threads, 1);
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------------------------------
|
||||
// Section 4 rows SESSION-01 could not reach in one process
|
||||
|
||||
/// Row: "Router restarts during a world advance | Old handles/routes invalid; epoch fails and
|
||||
/// restores coherently."
|
||||
///
|
||||
/// The restore half is STATE-01's. What SESSION-02 establishes is the half before it: the
|
||||
/// epoch fails with a typed cause naming the participant the coordinator was talking to, the
|
||||
/// session is fenced, every artifact handle of that store incarnation is gone, and no
|
||||
/// boundary, publication or further transition follows.
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn a_router_restart_during_a_world_advance_fences_the_epoch() {
|
||||
let mode = ExecutionMode::Process;
|
||||
let config = HarnessConfig {
|
||||
environment_faults: EnvironmentFaults {
|
||||
advance_delay_ms: 3_000,
|
||||
..EnvironmentFaults::default()
|
||||
},
|
||||
..two_agents(mode)
|
||||
};
|
||||
let mut f = mode_fixture(mode, config).await;
|
||||
// The router is gone in a moment, so the supervisor must not spend its full budget
|
||||
// asking a participant that can no longer be reached.
|
||||
f.harness.launcher.set_health_policy(fly_session::launcher::HealthPolicy {
|
||||
probe: Duration::from_millis(200),
|
||||
fail: Duration::from_millis(500),
|
||||
boot: Duration::from_secs(30),
|
||||
});
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let boundary_before = f.harness.coordinator.observation().unwrap().boundary;
|
||||
assert!(
|
||||
f.harness.coordinator.live_view_handles() > 0,
|
||||
"boundary 0's view is owned before the router goes away"
|
||||
);
|
||||
let router = f.harness.router().clone();
|
||||
let started = Instant::now();
|
||||
|
||||
let (coordinator, _launcher) = f.harness.parts();
|
||||
let (stepped, ()) = tokio::join!(
|
||||
async { within("step", coordinator.step()).await },
|
||||
async {
|
||||
// Mid-advance: the world has been asked to move and has not answered yet.
|
||||
tokio::time::sleep(Duration::from_millis(250)).await;
|
||||
router.shutdown();
|
||||
}
|
||||
);
|
||||
let failure = stepped.expect_err("a lost router fails the epoch");
|
||||
assert!(started.elapsed() < Duration::from_secs(20), "bounded, not a hang");
|
||||
assert_eq!(
|
||||
failure.participant.as_deref(),
|
||||
Some(f.harness.environment_id().as_str()),
|
||||
"the failure names the participant the coordinator was waiting for: {failure}"
|
||||
);
|
||||
assert_ne!(
|
||||
failure.error.mutation,
|
||||
MutationCertainty::None,
|
||||
"the world may have stepped; a lost router is never proof that it did not"
|
||||
);
|
||||
assert_eq!(f.harness.coordinator.phase(), Phase::Failed);
|
||||
assert!(
|
||||
f.harness.coordinator.is_fenced(),
|
||||
"old handles and routes are invalid from here on"
|
||||
);
|
||||
assert_eq!(
|
||||
f.harness.coordinator.live_view_handles(),
|
||||
0,
|
||||
"the fence drops every artifact handle of the old store incarnation"
|
||||
);
|
||||
assert_eq!(f.harness.coordinator.stats().advances, 0, "no boundary was committed");
|
||||
assert_eq!(count(&f.harness.coordinator.audit, "publish:1"), 0);
|
||||
assert_eq!(
|
||||
f.harness.coordinator.observation().unwrap().boundary,
|
||||
boundary_before,
|
||||
"the committed observation is still the one from before the advance"
|
||||
);
|
||||
// Nothing reconnects into the active epoch: a new call on the old route is refused.
|
||||
let again = f.harness.coordinator.step().await.expect_err("a fenced epoch takes no step");
|
||||
assert_eq!(again.error.code, ErrorCode::InvalidPhase);
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// Row: "Old worker replies after restore | Stale epoch/incarnation rejected", with real
|
||||
/// processes.
|
||||
///
|
||||
/// A restarted agent is a new process, a new registration and a new domain incarnation. The
|
||||
/// coordinator pinned the old registration, so its next call fails rather than reaching the
|
||||
/// replacement; and the replacement, followed deliberately, refuses an operation from the
|
||||
/// epoch the old process belonged to.
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn an_old_worker_reply_after_a_restart_is_rejected_on_stale_epoch_or_incarnation() {
|
||||
let mode = ExecutionMode::Process;
|
||||
let mut f = mode_fixture(mode, two_agents(mode)).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
within("step", f.harness.coordinator.step()).await.unwrap();
|
||||
|
||||
let old = f.harness.coordinator.agent_ref(&fly_b()).cloned().unwrap();
|
||||
let old_pid = f.harness.launcher.worker(&fly_b()).unwrap().pid;
|
||||
assert!(old_pid.is_some(), "a separate-process agent has a process of its own");
|
||||
let restarted = f.harness.restart_agent(&fly_b()).await.unwrap();
|
||||
let new_pid = f.harness.launcher.worker(&fly_b()).unwrap().pid;
|
||||
assert_ne!(old_pid, new_pid, "a restart is a new process");
|
||||
assert_ne!(
|
||||
restarted.service_incarnation, old.bus_incarnation,
|
||||
"a replacement registration is a new incarnation"
|
||||
);
|
||||
|
||||
// Following the new registration while still pinning the old worker's negotiated
|
||||
// incarnation is rejected: this is the shape an old worker's reply would arrive in.
|
||||
let stale = fly_session::rpc::WorkerRef {
|
||||
service: restarted.service.clone(),
|
||||
bus_incarnation: restarted.service_incarnation.clone(),
|
||||
worker_id: fly_b(),
|
||||
domain_incarnation: old.domain_incarnation.clone(),
|
||||
};
|
||||
assert_ne!(old.domain_incarnation, Some(restarted.incarnation_id.clone()));
|
||||
let err = within("status", f.harness.coordinator.status(&stale))
|
||||
.await
|
||||
.expect_err("the replacement is not the incarnation this epoch negotiated");
|
||||
assert_eq!(err.error.code, ErrorCode::IdentityMismatch);
|
||||
assert_eq!(err.participant.as_deref(), Some(fly_b().as_str()));
|
||||
assert_eq!(f.harness.coordinator.phase(), Phase::Failed);
|
||||
assert!(f.harness.coordinator.is_fenced());
|
||||
assert_eq!(f.harness.coordinator.stats().advances, 1, "no world step under a lost pin");
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// The other half of the same row, in two parts, because the two refusals are different
|
||||
/// refusals and each deserves its own exact code.
|
||||
///
|
||||
/// A restarted worker is a *fresh* process: it has no epoch at all, so the old epoch's work is
|
||||
/// refused on phase, not on timeline. The stale-epoch half of the row needs a worker that has
|
||||
/// an epoch and has left it, which in process mode is the agent that did not restart.
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
async fn a_restarted_worker_refuses_an_operation_from_the_old_epoch() {
|
||||
let mode = ExecutionMode::Process;
|
||||
let mut f = mode_fixture(mode, two_agents(mode)).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
within("step", f.harness.coordinator.step()).await.unwrap();
|
||||
|
||||
// Part one: a live agent process, initialized under epoch e1, meets an operation from
|
||||
// another epoch. This is the row's stale-epoch half, with a real child process.
|
||||
let live = f.harness.coordinator.agent_ref(&fly_a()).cloned().unwrap();
|
||||
let err = within(
|
||||
"stale epoch",
|
||||
f.harness.coordinator.probe_raw(
|
||||
&live,
|
||||
"Agent.Prepare",
|
||||
Some(scope_at("demo", "e0", 1)),
|
||||
prepare_params("fly-a"),
|
||||
),
|
||||
)
|
||||
.await
|
||||
.expect_err("an old epoch cannot mutate a worker that belongs to this one");
|
||||
assert_eq!(err.code, ErrorCode::StaleEpoch);
|
||||
assert_eq!(err.mutation, MutationCertainty::None, "refused before any mutation");
|
||||
|
||||
// Part two: the replacement process. It is a fresh worker with no epoch at all, so the
|
||||
// same request is refused on phase rather than on timeline -- and, either way, nothing
|
||||
// from the old epoch is applied to a fresh brain.
|
||||
let restarted = f.harness.restart_agent(&fly_b()).await.unwrap();
|
||||
let replacement = fly_session::rpc::WorkerRef::new(
|
||||
&restarted.service,
|
||||
&restarted.service_incarnation,
|
||||
&fly_b(),
|
||||
);
|
||||
let err = within(
|
||||
"uninitialized replacement",
|
||||
f.harness.coordinator.probe_raw(
|
||||
&replacement,
|
||||
"Agent.Prepare",
|
||||
Some(scope_at("demo", "e1", 1)),
|
||||
prepare_params("fly-b"),
|
||||
),
|
||||
)
|
||||
.await
|
||||
.expect_err("an uninitialized replacement has no epoch to prepare in");
|
||||
assert_eq!(
|
||||
err.code,
|
||||
ErrorCode::InvalidPhase,
|
||||
"a fresh process has no epoch to be stale about: {err}"
|
||||
);
|
||||
assert_eq!(err.mutation, MutationCertainty::None, "nothing was applied to a fresh brain");
|
||||
assert_eq!(f.harness.coordinator.stats().advances, 1);
|
||||
f.shutdown().await;
|
||||
}
|
||||
|
||||
/// A well-formed `Agent.Prepare` body, for a probe whose subject is the scope rather than the
|
||||
/// payload.
|
||||
fn prepare_params(agent_id: &str) -> serde_json::Value {
|
||||
serde_json::json!({
|
||||
"agentId": agent_id,
|
||||
"profileDigest": digest_of_bytes(b"whatever"),
|
||||
"interval": {"numerator": "16666667", "denominator": "1"},
|
||||
"decisionContextDigest": digest_of_bytes(b"whatever"),
|
||||
"preStepStimulations": [],
|
||||
})
|
||||
}
|
||||
|
|
@ -35,7 +35,7 @@ const STEPS: u64 = 3;
|
|||
async fn one_world_advance_per_complete_batch(via: Via) {
|
||||
let mut f = default_fixture(via).await;
|
||||
within("bootstrap", f.harness.coordinator.bootstrap()).await.unwrap();
|
||||
let before = f.harness.environment_mutations();
|
||||
let before = f.harness.environment_mutations().expect("an in-process arena");
|
||||
let reports = within("run", f.harness.coordinator.run(STEPS)).await.unwrap();
|
||||
assert_eq!(reports.len() as u64, STEPS);
|
||||
assert_eq!(f.harness.coordinator.stats().advances, STEPS);
|
||||
|
|
@ -46,7 +46,10 @@ async fn one_world_advance_per_complete_batch(via: Via) {
|
|||
"the world is at exactly one boundary per batch"
|
||||
);
|
||||
// The environment's progress counter moves once per advance and not otherwise.
|
||||
assert_eq!(f.harness.environment_mutations() - before, STEPS);
|
||||
assert_eq!(
|
||||
f.harness.environment_mutations().expect("an in-process arena") - before,
|
||||
STEPS
|
||||
);
|
||||
assert_eq!(count(&f.harness.coordinator.audit, "advance:0"), 1);
|
||||
assert_eq!(f.harness.coordinator.trace.transitions.len() as u64, STEPS);
|
||||
f.shutdown().await;
|
||||
|
|
@ -227,7 +230,8 @@ async fn bootstrap_cannot_advance_the_world_or_produce_a_reward(via: Via) {
|
|||
// Warm-up did run, with learning disabled, so the models did mutate.
|
||||
for agent in [fly_a(), fly_b()] {
|
||||
assert!(
|
||||
f.harness.agent_mutations(&agent) >= f.harness.config.warmup_ticks,
|
||||
f.harness.agent_mutations(&agent).expect("an in-process agent")
|
||||
>= f.harness.config.warmup_ticks,
|
||||
"warm-up ticks are real mutations"
|
||||
);
|
||||
}
|
||||
|
|
@ -270,7 +274,11 @@ async fn the_committed_snapshot_names_the_boundary_that_just_ended(via: Via) {
|
|||
// The frame the snapshot names travels as an owned attachment.
|
||||
if step > 0 {
|
||||
let frame = message.artifact("view.arena").expect("the published frame");
|
||||
assert_eq!(frame.reference().byte_length, 4 * 4 * 4);
|
||||
assert_eq!(
|
||||
frame.reference().byte_length,
|
||||
fly_session::environment::VIEW_WIDTH * fly_session::environment::VIEW_HEIGHT * 4,
|
||||
"the published frame is the environment native frame"
|
||||
);
|
||||
}
|
||||
}
|
||||
assert_eq!(boundaries[0], (0, false), "boundary 0 has no decision or control");
|
||||
|
|
@ -379,12 +387,7 @@ async fn sequential_concurrent_and_reversed_orders_agree() {
|
|||
/// specific.
|
||||
async fn a_single_agent_composition_runs_the_same_transaction(via: Via) {
|
||||
let config = HarnessConfig {
|
||||
agents: vec![AgentSpec {
|
||||
agent_id: id("fly-a"),
|
||||
port_id: id("p1"),
|
||||
seed: 7,
|
||||
faults: AgentFaults::default(),
|
||||
}],
|
||||
agents: vec![AgentSpec::new("fly-a", "p1", 7)],
|
||||
..HarnessConfig::default()
|
||||
};
|
||||
let mut f: Fixture = fixture(via, config).await;
|
||||
|
|
|
|||
|
|
@ -714,6 +714,13 @@ async fn caller_disconnect_cleanup_works_before_and_after_consumption() {
|
|||
.await;
|
||||
caller.close().await;
|
||||
drop(pending);
|
||||
// `Client::close` waits for this client to stop; the router marks the call detached when
|
||||
// *it* observes the disconnect, in its own connection task. Asserting the reply routing
|
||||
// before that is a race: under load the reply reaches the router first and is routed to a
|
||||
// connection that is already closing. Nothing escapes -- teardown releases those roots --
|
||||
// but `routed` is then true. The contract sentence is about a reply to an already detached
|
||||
// call, so the test waits for the teardown it is talking about.
|
||||
e.settle("caller-a disconnected", |s| s.connections == 1).await;
|
||||
assert!(!responder.reply(obj(json!({})), &[]).await.unwrap());
|
||||
drop(responder);
|
||||
e.settle("retained responder retired", |s| {
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue