The flybus session_over_one_router failure my workspace runs were
counting is already fixed on main: the coalescing branch forces the
coalescing deterministically instead of asserting that a slow consumer
must skip. Merging before the runs so they measure a tree that exists.
One conflict, in the crate README, and it was two sections both newly
added at the same anchor rather than two versions of one thing. Both are
kept: STATE-01's checkpoint and recovery section, then the guidance on
which replies permit a subset, which stays immediately above the
contract-narrowing section where someone adding a check will meet it.
coordinator.rs and the process tests auto-merged. Checked rather than
assumed: the acknowledge equality check is still gone, acknowledge_replies
and last_resolution_attempts are present, STATE-01's capture, durable
and rebase surfaces are present, and both test changes survived.
The sweep behind this branch found one check demanding an exact match
where the contract permits a short answer, and four that were right to
demand one. The question that separates them belongs where the next
check gets written, not only in a run report: is the far side reporting
what it did, or being held to a requirement?
Worker.Acknowledge is the only reply of the first kind here, because
ipc-v1 section 5 makes it idempotent. The commit and batch checks are
the second kind and are named so nobody loosens them later in the name
of tolerance; they are what make a partial commit and an incomplete
batch fail.
Review follow-up on the checkpoint store.
A dropped reply channel and an expired caller budget were both reported as
ReplyLost. They are different facts -- the first means the write is over and its
outcome did not reach here, the second means the save is still going -- so they are
now separate outcomes, and the durable wait has its own budget rather than borrowing
the one that bounds a call to a participant. Both still leave durable metadata where
it was, and for both the resolution asks the store about the same checkpoint.
The writer's two bounds refuse at different moments and the comment claimed
otherwise: the outstanding-capture bound is taken before a capture is requested, and
the byte budget cannot be, because a capture's size is not known until it exists. The
byte check, the decision and the change to the byte total are now one critical
section, the peak is sampled after a superseded job's bytes are gone, and the writer's
own bookkeeping is over a type that holds only the outcomes a writer can produce.
The manifest's coordinator.eventWatermarks is {lastSourceStep, issued}; the fixture
illustrated {lastEventId, lastOrdinal}, and the illustration is what changed, because
an event id is derived from the epoch and cannot be compared across the restore that
gives the session a new one.
checkpoint-envelope-v1 section 3 also now says, under the same dated amendment, that
a required-manifest-field change must bump envelopeVersion once production files
exist: contractDigest is taken over the schema set and does not cover this manifest,
so the envelope version is the only thing that can carry such a change.
STATE-01 over the FLYSESS1 envelope CONTRACT-01 specified.
fly-session gains a `state` module: the durable store with its generations, its
rotation and the commit order of checkpoint-envelope-v1 section 5, where the store
manifest rename is the durable commit point; a compatibility block whose comparison
names the identity that differs rather than one opaque digest; and a bounded writer
that owns its payload handles until the bytes are committed or the job fails.
The writer's queue slot is taken before the first State.Capture, so a saturated
writer refuses a capture rather than queueing it without bound, and the refusal is a
BUSY the stepping session survives. Capture and durability are two events: a capture
completes when an immutable capture exists, and only the store manifest rename moves
the durable mark. A lost save reply is an outcome, and the resolution asks the store
about the same checkpoint instead of saving again.
Both worker roles implement State.Capture, State.StageRestore and
State.ActivateRestore, with once-only restore tokens bound to checkpoint, scope,
payload and incarnation. A restore selects a complete compatible generation, imports
every payload as a fresh artifact, stages the group, validates the coordinator's own
ledgers, and only then activates; a failure anywhere leaves the fence closed and
records every participant that staged as one that must be replaced. The fence lifts
at Failed -> Restoring(k) -> Paused(k) and nowhere else.
The task and the action executor gain the capture/validate_restore/install_restore
interfaces workers-v1 section 4 lists, and the ledger can re-derive the event
identities it issued under another epoch, which is what lets a resumed run's
behaviour trace be compared with an uninterrupted one.
media: check_required_audio now takes the observation's provenance instead of
exempting boundary 0. A chunk is the audio of an interval, and the observation
ActivateRestore installs covers none.
checkpoint-envelope-v1 section 3 gains a dated amendment adding `environment` to the
manifest, the holder of the world's own payload, which the table named for every
other participant; `helperState`, which that table already listed, joins the
required-field set in Rust and TypeScript. The fixture was regenerated by the
existing example; the schema set and contractDigest are unchanged.
state-media-v1 section 5 gains a dated amendment for three readings this slice
enforces: the State RPCs' compatibilityDigest is the participant's, not the
manifest's composition-level block; a restored observation carries no audio chunk;
and a participant that staged into an abandoned install must be replaced.
Two review notes.
The launcher allocation became a required field of HelloResult.limits
with no negative fixture behind it. Three rows now cover it: missing,
above maxWorkerThreads, and zero. Both readers reject all three, and no
derived fixture moved, because invalid.json is not one of them.
The resolution's two bounds disagreed. 512 attempts at a 2 ms pause
give up near 1.5 s, so the attempt count silently pre-empted the 8 s
budget the doc comment advertised. The budget is now the working limit
and says so: the guard is 8192 attempts, over sixteen seconds of pauses
against an eight-second budget, so at the default values the budget is
always what fires. Which one did is no longer arithmetic either --
ResolutionEnd records it, the failure message names the bound and its
size, and the code, the doc comment and the README all state the same
numbers. a_resolution_says_which_of_its_two_bounds_ended_it drives each
bound to the end in every execution mode.
Review fixes for SESSION-02.
An expired caller deadline was becoming a failed epoch without the
ipc-v1 section 6 resolution. That procedure existed and was correct and
had exactly one caller, a test injection, so the deadline this slice
introduced bypassed it and a merely slow participant lost its epoch.
Deadlines is now the two-stage shape section 6 describes -- a probe,
then a bounded resolve budget and attempt count -- call_owned returns a
typed CallOutcome so an expiry is distinguishable from a refusal, and
Prepare, Commit, Advance and the lifecycle calls all query the same
request id against the same incarnation before the epoch can fail. This
is also step-v1 section 7's Advance row, which was imperative about it.
The coordinator peak-RSS column was measuring the measuring process.
VmHWM never falls and every row shared one process, so the column was
cumulative and the mode ranking reversed when the rows were reordered.
Each row now runs in a measure-row child of its own. The corrected
numbers say the opposite of what the first report claimed: the
coordinator's own peak is roughly flat across the modes and lowest in
process mode, and the cost of the split is the children.
workers-v1 section 2 bounded Agent.Initialize's workerThreads by
"within launcher allocation" and named no wire for it. Dated amendment:
HelloResult.limits gains workerThreads, the worker reports what its
launcher gave it, and the launcher refuses one that disagrees. The
schema set, the shared fixtures and the TypeScript package move
together; contractDigest changes, which ipc-v1 section 4 provides for.
Also: the stale-epoch row now reaches the stale-epoch path against a
live agent process and asserts exact codes on both halves; the
router-restart row asserts the handle drop it claimed; frames are
counted from the behaviour trace instead of calculated; the README says
which suites run over which transports; bootstrap is fence-guarded; the
shutdown reason is an Id rather than a silent fallback; and
agent_mutations returns None rather than zero where the counter lives
in another process.
SESSION-02: one agent process per fly and one environment process under
the coordinator over the Unix-socket transport, compared against the
in-process composition and a dedicated-thread variant. The mode is the
only thing that changes; the composition, the coordinator, the workers
and the router are the same code in all three.
The launcher is the configured supervisor. It owns a total thread budget
with one allocation per participant, refused as BUSY before anything
starts when the total cannot cover it; the configured client, service,
worker and port identities, proved in Worker.Hello before the
coordinator pins a registration; Worker.Status health on the
supervisor's own monotonic clock at the ipc-v1 section 6 budgets; and
reaping, where Worker.Shutdown is the request and the operating system
is the guarantee.
The worker executable is a subcommand of this crate's one binary, which
is what implementation.md section 2 allows in place of a separate
worker crate.
The coordinator's fault behaviour: every failure names the participant
it is attributed to, every domain call has a caller-side deadline so a
dead participant is a diagnosed outcome rather than a hang, and failing
fences the epoch -- the boundary stops, the handles drop, and no further
transition or publication is allowed. Agent.Initialize now carries the
launcher's allocation, and an agent refuses one asking for more.
tests/processes.rs proves every acceptance bullet once per execution
mode, and the two section 4 rows SESSION-01 could not reach in one
process: a router restart during a world advance, and an old worker's
reply after a restart. measure compares the three modes at one, two and
four agents; its table is in the crate README, and it is not a capacity
claim.
Review found two Memory-only tests behind a README claim that every integration test runs
twice. Both are now generated by both_transports!: the one-agent composition, and the
old-epoch refusal, which is one of the three tests the old-worker-replies row cites. The
README says which single test walks both transports inside itself instead, because it
compares their traces against each other.
step-v1 section 2 gains the mid-step pause line and a dated amendment saying why it is not
a new edge: a pause requested during a transition is the ordinary
Committing(k) -> Ready(k+1) edge followed by Ready(k+1) -> Paused(k+1), because section 6
requires the transition to finish first, so the only boundary such a pause can land on is
the one the transition just committed.
CONTRACT-01 landed its crate, so the local stand-in module goes away and the domain
scalars, method payloads, their validation, the canonical digests and the trace format
all come from the contract. `src/types.rs` is a facade over that crate plus the few
things a coordinator needs that are not part of the type contract: a session-side
DomainError, the synthetic composition's schema and event-id derivations, and the
coordinator-local ControllerIntent, PortBinding and AgentOutcome that never cross the
bus.
Consequences worth naming:
- Payloads are read and written through DomainType::from_json / to_json instead of
serde derives, so a misspelled required field fails where the contract says it
should. serde, sha2 and ryu-js leave this crate's dependencies with them.
- The step-v1 section 8 trace is the contract's TransitionTrace, with behaviour and
operational metadata already separated; the dispatch-order comparison now runs over
the contract's own behaviour encoding.
- Two things the migration found. IN_PROGRESS is raised strictly before any mutation,
so its certainty is "none", not "unknown"; the local module had it wrong.
WorldObservation::validate_against checks the views a result carries but does not
require every declared view to be present, so requiring them is made explicit in the
coordinator's phase C check, where step-v1 puts it.
- The lost-Advance-result injection now waits for the worker to report the operation
before abandoning the call, so the case it injects really is a loss after dispatch
rather than a cancellation before it.
Gates: cargo test -p fly-session (61 tests, both transports), cargo clippy
--all-targets clean, and the runnable example produces the same behaviour trace over
both transports.
The lockstep coordinator, its phase machine and a synthetic composition, as a new
workspace member. Every worker method is a Flybus RPC to an incarnation-pinned
service; the domain request ids and result caches of ipc-v1 section 5 sit in front of
every mutation.
- coordinator: the step-v1 section 3 transaction in order -- prepare every agent
concurrently, run one executor per agent in sorted agent-id order, assemble all port
controls in descriptor order, send exactly one Environment.Advance, evaluate the task
once, commit every agent concurrently, and move the committed boundary only when all
of them succeeded. Sequential, concurrent and reversed dispatch are selectable and
must agree.
- phase: the section 2 machine as an explicit edge table, Paused and Failed included,
with a committed-boundary predicate that gates pauses, captures and publication.
- clock: checked rational accumulation. A 60 Hz world with a 1 ms model tick runs
16, 17, 17 ticks and comes back to a remainder of exactly zero.
- dedup: operation keys, the cached reply with its own artifact holds, CONFLICT,
IN_PROGRESS, RESULT_EXPIRED, STALE_STEP, the lifecycle bound and Worker.Acknowledge.
- worker: the dispatch shell. Admission order and the result cache live here; the
endpoint mutex is the worker's simulation lock, so one mutation runs at a time while
Worker.Status answers from a separate cell.
- agent, environment, task: a fake model with an explicit seed and a mutation counter
the tests read, a fixed readout stub, a counter arena that seals one immutable frame
per boundary, a deterministic task and the identity executor.
- fly_session_types: the CONTRACT-01 domain types as a local stand-in, reconciled with
the shared crate in a following commit.
Tests run twice, over the in-memory transport and over a Unix socket, through the same
router: the SESSION-01 acceptance bullets and every section 4 failure-injection row
that applies to this slice.
