take_events kept returning Option<EventBatchView> and defaulting through the
question-mark operator, so a batch missing a field read as end of stream and the
ConsumerEvents enum added in the previous round described nothing. It returns
Batch or Unreadable now, and a test publishes a batch with no droppedBefore,
asserts it is reported as unreadable naming the field, and asserts the next real
batch still reads.
The checkpoint-envelope-v1 amendment cites the rule that lets a required
manifest field land with envelopeVersion still 1 while no production file
exists.
Dropping the length check left nothing checking the reply against the
request at all: AcknowledgeResult::validate_against existed with no
caller, so a worker could acknowledge ids this session never asked
about. That is the other half of the rule the README states. A short
list is the worker reporting what it released and is accepted; an id
from outside the request is the worker reporting about someone else's
cache and is refused, named, before any mutation.
The bootstrap-path regression is now covered too. The earlier test calls
acknowledge_replies directly, which guards the check where it lives but
not where it lived, so a length check put back into acknowledge_lifecycle
left it green. The duplicate_lifecycle_acknowledge injection releases the
ids first, out of sight, so the call that method makes and checks is
already the second one -- the shape the section 6 resolution produces.
Verified by putting the old check back: three tests fail with it, none
without.
Also: last_resolution_attempts is cleared with last_resolution, so a
resolution ending before its first attempt no longer reports the
previous count; the guard half of the bound test asserts the fence like
the budget half; the attempts assertion checks a real bound rather than
u32::MAX; and two dead Instant bindings are gone.
The index an agent attested to at Agent.Initialize joins its compatibility
identity and its checkpoint manifest row, so a replacement fly that built
another graph -- the same dataset, the same neuron count, another index --
cannot install a checkpoint taken under the first one. It was accepted before,
because agent_compatibility digested a dataset digest recomputed from a free
function rather than what the worker attested to, and the restored composition
was then published under its predecessor's indexDigest.
The refusal names what the worker is, not just that two digests differ. Dated
amendment to checkpoint-envelope-v1's agents row; no wire type and no schema
text change, so the contract digest is unchanged.
AgentSlot.telemetry was written only by the Agent.Initialize handler, so every
CommittedSnapshot carried warm-up telemetry labelled as boundary k while each
AgentCommitResult.telemetry was validated and dropped. The commit result is now
stored on the slot beside the committed step, and the media test asserts that
published telemetry advances across boundaries instead of merely being nonzero.
A refused snapshot is recorded and sequenced like a refused descriptor revision,
because the repair path exists for the consumer that did not receive it; the
sequence advances with the value rather than with the delivery, so two snapshots
can never share one. The query service counts an answer it could not deliver
rather than discarding the result, and an unreadable event batch is distinct
from the end of the stream.
The flybus session_over_one_router failure my workspace runs were
counting is already fixed on main: the coalescing branch forces the
coalescing deterministically instead of asserting that a slow consumer
must skip. Merging before the runs so they measure a tree that exists.
One conflict, in the crate README, and it was two sections both newly
added at the same anchor rather than two versions of one thing. Both are
kept: STATE-01's checkpoint and recovery section, then the guidance on
which replies permit a subset, which stays immediately above the
contract-narrowing section where someone adding a check will meet it.
coordinator.rs and the process tests auto-merged. Checked rather than
assumed: the acknowledge equality check is still gone, acknowledge_replies
and last_resolution_attempts are present, STATE-01's capture, durable
and rebase surfaces are present, and both test changes survived.
A group restore re-establishes a committed boundary this epoch did not run a
transition into, so its snapshot carries no decision and no controls for any
agent, and a fresh epoch is a new compositionDigest, so the descriptor takes the
next revision rather than republishing revision 1 with different contents.
CommittedSnapshot's rule becomes: null at boundary 0 and at an installed
boundary, always together, and for every agent or none -- a snapshot where one
fly acted and another did not would be two boundaries in one value. Dated
amendment to publishing-v1 section 3, with the schema set, the fixtures and the
TypeScript package moved together and the digest regenerated.
The short-list acknowledgment is a contract rule, so it gets a test that
says so rather than one that depends on a worker being slow.
acknowledge_replies carries the ipc-v1 section 5 sentence in its doc
comment and returns what the worker actually released;
acknowledge_lifecycle calls it, so bootstrap and the test exercise the
same path.
an_acknowledge_that_releases_nothing_is_not_a_failure drives the case
directly, once per execution mode: bootstrap releases every lifecycle
reply, the test asks for those ids again, the worker ignores them and
releases nothing, and the coordinator must accept the empty list, stay
unfenced, stay at its boundary and still play the next transition. It
fails if the equality check returns.
The bound test failed two runs in thirty under load, and not on the
bound it was testing. Both failures were bootstrap: "a worker did not
acknowledge every lifecycle reply".
ipc-v1 section 5 says already released or unknown ids are ignored, and
the contract type already holds the acknowledged list to a subset of
the request. So the second Acknowledge of the same ids answers with an
empty list by design -- and the section 6 resolution produces exactly
that second Acknowledge whenever the first reply is slower than the
probe. Demanding the whole list back turned a safe, contract-sanctioned
retry into a failed epoch, which is a defect in the coordinator rather
than in the test: a slow lifecycle reply would do it to a real session
too.
The test made itself easy to hit by installing a fifty-millisecond
probe before bootstrap, so bootstrap's own lifecycle calls ran under a
budget meant for the step under test. It now bootstraps at ordinary
deadlines and tightens them afterwards.
The two bounds are also separated by construction rather than by clock.
Each half puts the bound it is not testing out of reach -- u32::MAX
attempts against a fifth of a second, three attempts against an hour --
so no scheduling delay can flip which one fires, and the silent
participant is ten minutes slow against a twenty-second test timeout,
so returning at all proves a bound ended it. The wall-clock assertion
is gone and the attempt count is asserted instead, which
last_resolution_attempts now records. One agent per composition, so the
participant the failure names is not a race either.
An undeclared stimulus kind is refused before the model is touched, proved by an
injection through Agent.Commit rather than by a unit call, so the declaration a
descriptor publishes is the thing the worker enforces.
The publishing-v1 section 2 amendment now says in its own words that
Session.GetDescriptor and Session.GetSnapshot are internal and provisional names,
which the later public v2 step may rename or supersede.
Review follow-up on the checkpoint store.
A dropped reply channel and an expired caller budget were both reported as
ReplyLost. They are different facts -- the first means the write is over and its
outcome did not reach here, the second means the save is still going -- so they are
now separate outcomes, and the durable wait has its own budget rather than borrowing
the one that bounds a call to a participant. Both still leave durable metadata where
it was, and for both the resolution asks the store about the same checkpoint.
The writer's two bounds refuse at different moments and the comment claimed
otherwise: the outstanding-capture bound is taken before a capture is requested, and
the byte budget cannot be, because a capture's size is not known until it exists. The
byte check, the decision and the change to the byte total are now one critical
section, the peak is sampled after a superseded job's bytes are gone, and the writer's
own bookkeeping is over a type that holds only the outcomes a writer can produce.
The manifest's coordinator.eventWatermarks is {lastSourceStep, issued}; the fixture
illustrated {lastEventId, lastOrdinal}, and the illustration is what changed, because
an event id is derived from the epoch and cannot be compared across the restore that
gives the session a new one.
checkpoint-envelope-v1 section 3 also now says, under the same dated amendment, that
a required-manifest-field change must bump envelopeVersion once production files
exist: contractDigest is taken over the schema set and does not cover this manifest,
so the envelope version is the only thing that can carry such a change.
STATE-01 over the FLYSESS1 envelope CONTRACT-01 specified.
fly-session gains a `state` module: the durable store with its generations, its
rotation and the commit order of checkpoint-envelope-v1 section 5, where the store
manifest rename is the durable commit point; a compatibility block whose comparison
names the identity that differs rather than one opaque digest; and a bounded writer
that owns its payload handles until the bytes are committed or the job fails.
The writer's queue slot is taken before the first State.Capture, so a saturated
writer refuses a capture rather than queueing it without bound, and the refusal is a
BUSY the stepping session survives. Capture and durability are two events: a capture
completes when an immutable capture exists, and only the store manifest rename moves
the durable mark. A lost save reply is an outcome, and the resolution asks the store
about the same checkpoint instead of saving again.
Both worker roles implement State.Capture, State.StageRestore and
State.ActivateRestore, with once-only restore tokens bound to checkpoint, scope,
payload and incarnation. A restore selects a complete compatible generation, imports
every payload as a fresh artifact, stages the group, validates the coordinator's own
ledgers, and only then activates; a failure anywhere leaves the fence closed and
records every participant that staged as one that must be replaced. The fence lifts
at Failed -> Restoring(k) -> Paused(k) and nowhere else.
The task and the action executor gain the capture/validate_restore/install_restore
interfaces workers-v1 section 4 lists, and the ledger can re-derive the event
identities it issued under another epoch, which is what lets a resumed run's
behaviour trace be compared with an uninterrupted one.
media: check_required_audio now takes the observation's provenance instead of
exempting boundary 0. A chunk is the audio of an interval, and the observation
ActivateRestore installs covers none.
checkpoint-envelope-v1 section 3 gains a dated amendment adding `environment` to the
manifest, the holder of the world's own payload, which the table named for every
other participant; `helperState`, which that table already listed, joins the
required-field set in Rust and TypeScript. The fixture was regenerated by the
existing example; the schema set and contractDigest are unchanged.
state-media-v1 section 5 gains a dated amendment for three readings this slice
enforces: the State RPCs' compatibilityDigest is the participant's, not the
manifest's composition-level block; a restored observation carries no audio chunk;
and a participant that staged into an abandoned install must be replaced.
publishing-v1 over the same bus: a declared delivery policy per topic, named
publication outcomes, the bounded event batch, application-owned state and cues,
a read-only descriptor query service and a fake multi-agent consumer.
The session publishes the contract types rather than an ad-hoc payload, so a
descriptor says what the workers attested to and a snapshot is checked against it
before it is published and again when it is read: every frame comes from the
boundary its declared delay implies, every handle is the artifact its reference
names, and an observer's refusal takes no world step and fences no epoch.
AgentInitializeResult gains graph (datasetDigest, indexDigest, neuronCount,
rateRoles, supportedStimuli), without which no AgentDescriptor field in
publishing-v1 section 3 had a source. Dated amendments to workers-v1 section 2,
publishing-v1 section 2 and state-media-v1 section 3.
The command line between a launcher and a participant in its own process
was coupled by string convention on both sides and checked by neither: the
parser accepted any --flag it did not know, so a renamed or dropped option
would have become a fault that never fires or a delay that is never
applied, with nothing failing.
Every flag now has one name, in launcher::flags, written by the launcher's
argv and read by the parser. Options::parse takes the sets its command
allows and refuses anything outside them by name, so a mismatch fails the
launch. measure and measure-row use the same constants for their own
flags.
Three tests hold the two sides together: every flag a launch writes is one
its command accepts, every media option is written for a separate process,
and an unknown or wrong-command flag is refused by name. The four media
faults now also run end to end in all three execution modes, each failing
its own transition, so the wiring across a process boundary is proved
rather than assumed.
Keep-both everywhere the two slices met. lib.rs takes both module sets.
AgentConfig keeps worker_threads and the sensor log; EnvironmentConfig
keeps worker_threads, the render delay and the render counter.
coordinator.rs keeps the two-stage resolution and its blame() beside the
media split of the Advance reply's attachments, and its imports take both.
harness.rs is main's launcher-based file with this slice's media
instrumentation re-applied on top.
The media instrumentation is shared memory, so it now follows the
launcher's own rule for the progress counter: sensor_log and renders
return None for a participant with a process of its own rather than a
misleading zero. The launcher carries the sensor log and the render
counter to a participant in this process and the render delay and the four
media faults on the command line to one in another process, where the
child builds its own log and counter.
The media path itself is mode-agnostic and is now tested as such: one
image per boundary, forwarded to every agent and published once, asserted
over the bus in all three execution modes, with the shared-memory
assertions made only where those participants live.
Two review notes.
The launcher allocation became a required field of HelloResult.limits
with no negative fixture behind it. Three rows now cover it: missing,
above maxWorkerThreads, and zero. Both readers reject all three, and no
derived fixture moved, because invalid.json is not one of them.
The resolution's two bounds disagreed. 512 attempts at a 2 ms pause
give up near 1.5 s, so the attempt count silently pre-empted the 8 s
budget the doc comment advertised. The budget is now the working limit
and says so: the guard is 8192 attempts, over sixteen seconds of pauses
against an eight-second budget, so at the default values the budget is
always what fires. Which one did is no longer arithmetic either --
ResolutionEnd records it, the failure message names the bound and its
size, and the code, the doc comment and the README all state the same
numbers. a_resolution_says_which_of_its_two_bounds_ended_it drives each
bound to the end in every execution mode.
Review fixes for SESSION-02.
An expired caller deadline was becoming a failed epoch without the
ipc-v1 section 6 resolution. That procedure existed and was correct and
had exactly one caller, a test injection, so the deadline this slice
introduced bypassed it and a merely slow participant lost its epoch.
Deadlines is now the two-stage shape section 6 describes -- a probe,
then a bounded resolve budget and attempt count -- call_owned returns a
typed CallOutcome so an expiry is distinguishable from a refusal, and
Prepare, Commit, Advance and the lifecycle calls all query the same
request id against the same incarnation before the epoch can fail. This
is also step-v1 section 7's Advance row, which was imperative about it.
The coordinator peak-RSS column was measuring the measuring process.
VmHWM never falls and every row shared one process, so the column was
cumulative and the mode ranking reversed when the rows were reordered.
Each row now runs in a measure-row child of its own. The corrected
numbers say the opposite of what the first report claimed: the
coordinator's own peak is roughly flat across the modes and lowest in
process mode, and the cost of the split is the children.
workers-v1 section 2 bounded Agent.Initialize's workerThreads by
"within launcher allocation" and named no wire for it. Dated amendment:
HelloResult.limits gains workerThreads, the worker reports what its
launcher gave it, and the launcher refuses one that disagrees. The
schema set, the shared fixtures and the TypeScript package move
together; contractDigest changes, which ipc-v1 section 4 provides for.
Also: the stale-epoch row now reaches the stale-epoch path against a
live agent process and asserts exact codes on both halves; the
router-restart row asserts the handle drop it claimed; frames are
counted from the behaviour trace instead of calculated; the README says
which suites run over which transports; bootstrap is fence-guarded; the
shutdown reason is an Id rather than a silent fallback; and
agent_mutations returns None rather than zero where the counter lives
in another process.
Review round two.
A restored timeline refuses a declared stream with no recorded sample
position instead of resuming it at zero, which would have restarted the
episode's audio clock silently.
A declared stream that produces no chunk for a transition now fails the
step. The contract does not say a chunk must exist; this slice requires
one, because a silently missing chunk cannot be told from a lost one.
Boundary 0 is exempt: no interval precedes it.
The fresh-epoch discontinuity refusal is dropped rather than written into
the amendment. Only the restore direction is stated, and sections 6 and 7
both have a fresh timeline publishing a discontinuity after a recovery or
a reset, so refusing the flag at an origin contradicted them. The dated
amendment now says the requirement is one-directional.
A snapshot with an unreadable chunk, or one whose handle is not attached,
is now unreadable as a whole rather than quietly carrying less audio than
was published. ViewPipeline::is_bootstrap_repeat is removed.
New tests: a missing chunk fails the step, restored timelines need every
declared stream's position, and a 7 Hz world's seven chunks sum to exactly
one second of 48 kHz samples. The bus-conformance reconnect row also cites
the test that actually replaces an incarnation.
SESSION-02: one agent process per fly and one environment process under
the coordinator over the Unix-socket transport, compared against the
in-process composition and a dedicated-thread variant. The mode is the
only thing that changes; the composition, the coordinator, the workers
and the router are the same code in all three.
The launcher is the configured supervisor. It owns a total thread budget
with one allocation per participant, refused as BUSY before anything
starts when the total cannot cover it; the configured client, service,
worker and port identities, proved in Worker.Hello before the
coordinator pins a registration; Worker.Status health on the
supervisor's own monotonic clock at the ipc-v1 section 6 budgets; and
reaping, where Worker.Shutdown is the request and the operating system
is the guarantee.
The worker executable is a subcommand of this crate's one binary, which
is what implementation.md section 2 allows in place of a separate
worker crate.
The coordinator's fault behaviour: every failure names the participant
it is attributed to, every domain call has a caller-side deadline so a
dead participant is a diagnosed outcome rather than a hang, and failing
fences the epoch -- the boundary stops, the handles drop, and no further
transition or publication is allowed. Agent.Initialize now carries the
launcher's allocation, and an agent refuses one asking for more.
tests/processes.rs proves every acceptance bullet once per execution
mode, and the two section 4 rows SESSION-01 could not reach in one
process: a router restart during a world advance, and an old worker's
reply after a restart. measure compares the three modes at one, two and
four agents; its table is in the crate README, and it is not a capacity
claim.
Puts MEDIA-01 on top of the bus ArtifactRef rather than beside it.
fly-session-types gains the two validators a descriptor and a reference
cannot carry on their own: AudioTimeline, which holds one stream chunk
sequence for one epoch (no overlap, no backwards, the first chunk after a
restore marks the discontinuity), and check_imported_asset, which checks
that a transient artifact carries an installed asset content without ever
converting one identity into the other.
fly-session gains a media module: a ViewPipeline that renders one native
frame per boundary and serves the frame the declared observationDelaySteps
requires (so bootstrap repeats O[0] exactly while the pipeline fills), an
AudioSource with an exact rational sample budget and a deterministic
integer-phase waveform, the Phase C acceptance checks, a latest-subscription
Spectator with finite credits, a SensorLog recording what each agent read,
and an AssetRegistry for installed persistent content.
The counter arena now emits a real 32x24 RGBA8 pattern per boundary and one
audio chunk per transition; the coordinator forwards the one owned view
handle to every agent Commit and publishes the same handle, with the audio
handles, for presentation.
36 new tests: 12 shape rules against the contract crate and 24 session-level
acceptance tests over both transports.
The CONTRACT-01 amendment narrows Worker.Acknowledge ids to the req-<U64> form, which this
crate already read as DomainRequestId. The 1..=16 bound now comes from the contract's
MAX_ACKNOWLEDGE too, so there is one definition of it rather than two.
CONTRACT-01 landed its crate, so the local stand-in module goes away and the domain
scalars, method payloads, their validation, the canonical digests and the trace format
all come from the contract. `src/types.rs` is a facade over that crate plus the few
things a coordinator needs that are not part of the type contract: a session-side
DomainError, the synthetic composition's schema and event-id derivations, and the
coordinator-local ControllerIntent, PortBinding and AgentOutcome that never cross the
bus.
Consequences worth naming:
- Payloads are read and written through DomainType::from_json / to_json instead of
serde derives, so a misspelled required field fails where the contract says it
should. serde, sha2 and ryu-js leave this crate's dependencies with them.
- The step-v1 section 8 trace is the contract's TransitionTrace, with behaviour and
operational metadata already separated; the dispatch-order comparison now runs over
the contract's own behaviour encoding.
- Two things the migration found. IN_PROGRESS is raised strictly before any mutation,
so its certainty is "none", not "unknown"; the local module had it wrong.
WorldObservation::validate_against checks the views a result carries but does not
require every declared view to be present, so requiring them is made explicit in the
coordinator's phase C check, where step-v1 puts it.
- The lost-Advance-result injection now waits for the worker to report the operation
before abandoning the call, so the case it injects really is a loss after dispatch
rather than a cancellation before it.
Gates: cargo test -p fly-session (61 tests, both transports), cargo clippy
--all-targets clean, and the runnable example produces the same behaviour trace over
both transports.
The lockstep coordinator, its phase machine and a synthetic composition, as a new
workspace member. Every worker method is a Flybus RPC to an incarnation-pinned
service; the domain request ids and result caches of ipc-v1 section 5 sit in front of
every mutation.
- coordinator: the step-v1 section 3 transaction in order -- prepare every agent
concurrently, run one executor per agent in sorted agent-id order, assemble all port
controls in descriptor order, send exactly one Environment.Advance, evaluate the task
once, commit every agent concurrently, and move the committed boundary only when all
of them succeeded. Sequential, concurrent and reversed dispatch are selectable and
must agree.
- phase: the section 2 machine as an explicit edge table, Paused and Failed included,
with a committed-boundary predicate that gates pauses, captures and publication.
- clock: checked rational accumulation. A 60 Hz world with a 1 ms model tick runs
16, 17, 17 ticks and comes back to a remainder of exactly zero.
- dedup: operation keys, the cached reply with its own artifact holds, CONFLICT,
IN_PROGRESS, RESULT_EXPIRED, STALE_STEP, the lifecycle bound and Worker.Acknowledge.
- worker: the dispatch shell. Admission order and the result cache live here; the
endpoint mutex is the worker's simulation lock, so one mutation runs at a time while
Worker.Status answers from a separate cell.
- agent, environment, task: a fake model with an explicit seed and a mutation counter
the tests read, a fixed readout stub, a counter arena that seals one immutable frame
per boundary, a deterministic task and the identity executor.
- fly_session_types: the CONTRACT-01 domain types as a local stand-in, reconciled with
the shared crate in a following commit.
Tests run twice, over the in-memory transport and over a Unix socket, through the same
router: the SESSION-01 acceptance bullets and every section 4 failure-injection row
that applies to this slice.
