alex/flybrain
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Merge feat/sf-bus-conformance: bus-v1 conformance audit, the teardown-poll race, BUS-01..03 acceptance tests and the measured example
Some checks are pending
ci / node 22 (test + typecheck) (push) Waiting to run
Details
ci / rust stable (cargo test --workspace --release) (push) Waiting to run
Details
ci / infra/tests/lint.sh (push) Waiting to run
Details
ci / playwright apps/stage (allowed to fail) (push) Waiting to run
Details

Browse source
This commit is contained in:
acamilo 2026-09-22 12:40:29 +00:00
commit 431f67baa0
13 changed files with 1786 additions and 190 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
docs/design/flybus.md
View file

@ -2,7 +2,8 @@
Status: **crate landed, nothing wired onto it**. Written 2026-09-22. Index only; the
authority for the API and the wire format is the crate's own
[README](../../services/flysim/crates/flybus/README.md).
[README](../../services/flysim/crates/flybus/README.md), and the audit of the crate against
the draft is the [conformance report](session-framework/bus-conformance.md).
## What it is

3
docs/design/session-framework/README.md
View file

@ -33,6 +33,9 @@ retroactively to existing [public feed](../../feed-protocol.md),
5. [Session media/state](state-media-v1.md) — observation timing and coherent recovery.
6. [Application/presentation boundary](publishing-v1.md) — snapshots, flexible data and effects.
7. [Implementation guide](implementation.md) — sequenced build tasks and acceptance tests.
8. [Flybus conformance report](bus-conformance.md) — the `flybus` crate audited sentence by
sentence against bus-v1, with the test that proves each row, the measurements and the
draft's own contradictions. A review artifact, not a contract.
For context: [modular-session analysis](../malecns-modular-sessions.md) and
[Melee audit](../melee-framework-audit.md). Each contract owns its named subject; step ordering

465
docs/design/session-framework/bus-conformance.md Normal file
View file

@ -0,0 +1,465 @@
# Flybus v1 conformance report: the `flybus` crate against `bus-v1`
Status: **audit, 2026-09-22**. Subject: `services/flysim/crates/flybus`, the replayed
implementation of [Flybus v1](bus-v1.md) (draft 1, 2026-09-18), with no consumers yet. Scalar
encodings come from [ipc-v1](ipc-v1.md) sections 1 to 3. Acceptance lists come from the
[implementation guide](implementation.md) slices BUS-01, BUS-02 and BUS-03.
Every normative sentence of bus-v1 sections 2 to 11 gets a row. Four statuses:
| Status | Meaning |
| --- | --- |
| conforms | The implementation does what the sentence requires, and a test proves it. |
| deviates-allowed | It differs, and a quoted sentence of the draft permits the difference. |
| deviates-must-fix | It differs and the draft requires otherwise. |
| not-implemented | Not built yet; the row names who owns it. |
Counts over 195 rows: **conforms 178, deviates-allowed 9, deviates-must-fix 1 (fixed),
not-implemented 7**. The audit found the one deviates-must-fix — connection teardown could be
starved for the length of a whole frame by the writer it was waiting for — and it is fixed on
this branch, so the row for it now reads conforms and records the fix (section 9, "cannot be
starved"). Two contradictions inside the draft are recorded at the end and left alone.
Test names below are the functions in `services/flysim/crates/flybus/tests`, 239 of them in
this branch (`cargo test -p flybus`), plus the ignored measurement. Everything marked
"(both)" is generated twice by the `both_transports!` macro, once over the in-memory transport
and once over a Unix socket, so `tests/rpc.rs::request_reply_roundtrip` means
`in_memory::request_reply_roundtrip` and `unix_socket::request_reply_roundtrip`.
## 2. Client API
| Requirement | Status | Code | Test |
| --- | --- | --- | --- |
| One client/connection serves RPC, pub/sub and artifacts | conforms | `client/mod.rs::Client` | `tests/integration.rs::session_over_one_router` (both) |
| The illustrative surface (`connect`, `register`, `call`, `subscribe`, `publish`, `artifacts().allocate`, `seal`, `message.artifact`) | deviates-allowed: "Illustrative Rust surface (not yet implemented)"; `connect` takes the transport, `call` takes no `budget`, `next()` yields `Option` | `client/mod.rs`, `client/handles.rs` | `tests/rpc.rs`, `tests/pubsub.rs`, `tests/artifacts.rs` |
| The artifact store is a storage backend of the bus, not another messaging service | conforms | `store.rs`, reached only through `Artifacts`/`Artifact` | `tests/artifacts.rs::allocate_write_seal_read` (both) |
| Bulk data does not pass through router socket payloads; no separate data-transfer API | conforms: attachments carry `ArtifactRef` only; envelopes are capped at 64 KiB | `wire.rs::ArtifactRef`, `wire.rs::MAX_ENVELOPE_BYTES` | `tests/wire.rs::envelope_size_limits` (both), `tests/perf.rs` |
| `Artifact` is a read-only, cloneable handle | conforms | `client/handles.rs::Artifact` (no write API, `#[derive(Clone)]`) | `tests/conformance_artifacts.rs::extracted_artifact_outlives_the_message_it_came_from` (both) |
| `ArtifactWriter` is unique, not cloneable; sealing consumes its writable lifetime | conforms | `client/handles.rs::ArtifactWriter::seal_with_digest(mut self)` | `tests/artifacts.rs::seal_is_immune_to_live_writable_handles` (both) |
| Mapped slices cannot outlive their handle | conforms by construction: there is no mapping API; `ArtifactFile` owns its `Artifact` | `client/handles.rs::ArtifactFile` | `tests/bus_acceptance.rs::disconnect_releases_logical_ownership_without_mutating_open_bytes` (both) |
| Rust RAII automates releases | conforms | `client/handles.rs::OwnerGuard::drop` | `tests/artifacts.rs::fan_out_shares_one_object_and_the_last_consumer_collects` (both) |
| Other language bindings provide equivalent explicit close/context-manager behaviour | not-implemented (Rust only; section 1 says a binding "may" exist). Owner: a future binding | — | — |
| Garbage collection means reclaiming an unowned artifact, not inspecting game state | conforms | `router/state.rs::drop_roots` | `tests/conformance_artifacts.rs::collection_waits_for_every_retained_owner` (both) |
## 3. Addressing and identities
| Requirement | Status | Code | Test |
| --- | --- | --- | --- |
| Every identity of the table exists: `routerId`, `clientId`/`clientIncarnation`, `connectionId`, `service`/`serviceIncarnation`, `callId`, `topic`/`topicIncarnation`/`topicSequence`, `deliveryId`, `artifactId`/`generation`, `ownerId` | conforms | `router/mod.rs::fresh_tag`, `router/state.rs` (`serial_id` for `conn`/`svc`/`top`/`sub`/`dlv`/`own`/`a`) | `tests/conformance_wire.rs::hello_reports_the_contract_digest_and_valid_limits` (both), `tests/rpc.rs::registration_is_exclusive_and_pinned` (both) |
| A fresh `routerId`/`storeId` per incarnation; old handles fail after restart | conforms | `router/mod.rs::Router::new` | `tests/artifacts.rs::router_restart_invalidates_old_handles` (both) |
| Reconnect creates a new `clientIncarnation`; v1 does not resume a connection's queues or delivery owners | conforms | `router/state.rs::hello` refuses a reused incarnation; `disconnect` releases everything | `tests/sol_review_races.rs::caller_disconnect_cleanup_works_before_and_after_consumption` |
| Identifiers are bounded ASCII; `Id` and `U64` match ipc-v1 | conforms: `^[a-z0-9][a-z0-9._-]{0,63}$`, `"0"\|[1-9][0-9]*` | `wire.rs::is_id`, `wire.rs::parse_u64` | `wire.rs::tests::scalars`, `tests/conformance_wire.rs::malformed_envelope_scalars_are_rejected` (both) |
| Service/topic names are 1..192 of `[a-z0-9._-]` with no empty dot-separated segment | conforms | `wire.rs::is_name` | `wire.rs::tests::scalars` |
| Exact names only; wildcard routing and queue groups deferred | conforms: routing is `HashMap` lookup by exact name. `Pattern::Prefix` is a launcher grant, never a route | `router/state.rs::services`/`topics`, `policy.rs::Pattern` | `tests/pubsub.rs::subscription_and_topic_validation` (both), `tests/rpc.rs::authority_is_enforced` (both) |
| One live registration owns a service name; duplicate registration fails; no implicit round-robin or replacement | conforms (`CONFLICT`) | `router/state.rs::op_register` | `tests/rpc.rs::registration_is_exclusive_and_pinned` (both), `tests/conformance_routing.rs::duplicate_registration_by_owner_itself_is_rejected` (both) |
| Registration returns the incarnation; callers pin it; a change fails with `TARGET_CHANGED` | conforms | `router/state.rs::op_call` | `tests/rpc.rs::registration_is_exclusive_and_pinned` (both), `tests/bus_acceptance.rs::no_automatic_retry_or_failover_onto_a_replacement_registration` (both) |
| An unpinned call reaches whoever holds the name now | conforms | `router/state.rs::op_call` (`expectedIncarnation: null`) | `tests/conformance_routing.rs::unpinned_call_after_incarnation_replacement_reaches_the_new_holder` (both) |
| `Worker.Hello` stays a domain RPC, distinct from transport negotiation | conforms by absence: `bus.hello` carries no role, capability or session field | `router/state.rs::hello` | `tests/wire.rs::hello_negotiation` (both) |
| Service/topic access is configured per participant by the launcher; naming a target is not authority | conforms | `policy.rs::{Policy, Grants}`, checked in every `op_*` | `tests/rpc.rs::authority_is_enforced` (both), `tests/pubsub.rs::subscription_and_topic_validation` (both) |
| Presentation subscribes without gaining authority to invoke Advance | conforms: `subscribe` and `call` are separate grants | `policy.rs::Grants` | `tests/rpc.rs::authority_is_enforced` (both) |
| One live connection per client id, and a client id's last incarnation may not be reused | deviates-allowed (narrowing): section 3 makes `callId` "unique ... for this client incarnation" and section 6 keeps serials "per connected client", which two live connections for one id would make ambiguous. Cost: one small record per client id ever seen | `router/state.rs::{ClientRecord, hello}` | `tests/sol_review_races.rs::pending_connections_are_bounded_and_hello_expires` |
## 4. Wire envelope and framing
| Requirement | Status | Code | Test |
| --- | --- | --- | --- |
| Framing is `u32` little-endian JSON length, then UTF-8 JSON | conforms | `wire.rs::{read_frame, write_frame}` | `tests/conformance_wire.rs::length_prefix_is_little_endian` (both) |
| Envelope fields exactly `protocol`/`major`/`minor`/`id`/`replyTo`/`kind`/`op`/`body`/`attachments` | conforms | `wire.rs::Envelope::{decode, to_value}` | `tests/conformance_wire.rs::unknown_top_level_field_closes_the_connection` (both) |
| `ArtifactRef` fields `storeId`/`artifactId`/`generation`/`byteLength`/`contentType`/`digest` | conforms (`generation`/`byteLength` as U64 strings) | `wire.rs::ArtifactRef` | `tests/conformance_artifacts.rs::stale_store_incarnation_and_generation_are_rejected` (both) |
| `Attachment` is `{name, ref, ownerId}` | conforms | `wire.rs::Attachment` | `tests/conformance_artifacts.rs::allocate_write_seal_open_roundtrip_and_mismatches` (both) |
| Maximum total JSON envelope 65,536 bytes | conforms | `wire.rs::MAX_ENVELOPE_BYTES`, checked on decode and encode | `tests/conformance_wire.rs::frame_at_the_size_ceiling_is_accepted_one_byte_over_is_not` (both) |
| A delivery the router would build over the limit is refused at admission | conforms, and required by "Message length includes this wrapper" (section 5): admission sizes the delivery with the longest ids | `router/state.rs::frame_len` in `op_call`/`op_reply`/`op_publish` | `tests/sol_review_races.rs::unsent_oversized_call_rolls_back_its_local_slot`, `tests/wire.rs::envelope_size_limits` (both) |
| Up to 32 attachments, unique names; `contentType` nonempty ASCII <=127 | conforms | `wire.rs::{MAX_ATTACHMENTS, is_content_type}`, `Envelope::decode` | `tests/conformance_wire.rs::malformed_envelope_scalars_are_rejected` (both), `tests/artifacts.rs::quotas_are_enforced` (both) |
| No pixel/base64/checkpoint bytes in JSON; artifact sizes are independent of envelope size | conforms: bulk bytes only reach the store; `byteLength` is a string in the reference | `store.rs`, `wire.rs::ArtifactRef` | `tests/perf.rs` (1.2 MB frames, envelopes under 1 KiB) |
| Domain schemas enumerate every referenced artifact; generated bindings enforce it | not-implemented for domain schemas (the bus enforces its own attachment list). Owner: CONTRACT-01 / `fly-session-types` | `router/state.rs::check_attachments` enforces the bus half | `tests/conformance_artifacts.rs::forward_requires_the_source_owner_still_live` (both) |
| The router validates attachment declarations/ownership, not domain payload contents | conforms | `router/state.rs::{check_owned, check_attachments}`; `payload`/`outcome` stay opaque | `tests/conformance_artifacts.rs::owner_ids_are_scoped_to_their_connection` (both) |
| Commands have unique monotonically issued `msg-<U64>` ids per connection | conforms (strictly increasing) | `router/state.rs::handle` | `tests/conformance_wire.rs::{non_canonical_command_ids_are_rejected, non_increasing_command_ids_are_rejected}` (both) |
| Replies correlate with `replyTo`; notices and deliveries carry router-generated ids | conforms | `router/state.rs::{reply, outbound}` | `tests/conformance_wire.rs::non_null_reply_to_on_a_command_is_rejected` (both) |
| The router supplies authenticated sender/target metadata; senders cannot forge it in `body` | conforms on launcher-bound transports; the sender's `body` never supplies identity | `router/state.rs::{identity, Call::request_body}` | `tests/rpc.rs::raw_call_ids_and_forged_replies` (both), `tests/sol_review_races.rs::responder_survives_cancel_then_request_drop` |
| Identity is bound out of band before Hello; a mismatching Hello is refused before registration | conforms | `router/mod.rs::{serve_as, listen_unix_as}`, `state.rs::hello` | `tests/wire.rs::hello_refusals` (both) |
| Open/unbound transports are self-asserted, not authentication | deviates-allowed (explicit narrowing): section 1's "one trusted local deployment" and section 4's "The launcher provides expected client/registration privileges". `Policy::open()` is documented as test/trusted-only | `policy.rs::permits_unbound_transport`, `router/state.rs::add_conn` | `tests/sol_review_races.rs::pending_connections_are_bounded_and_hello_expires` |
| Reject duplicate JSON keys, invalid UTF-8, NaN/Infinity, unknown envelope fields, zero/oversize frames, invalid ranges | conforms | `wire.rs::{parse_json_strict, StrictVisitor, Fields::finish}`, `read_frame` | `tests/conformance_wire.rs::{duplicate_json_keys_are_rejected_at_every_depth, invalid_utf8_is_rejected, zero_length_frame_closes_the_connection, oversize_length_prefix_is_rejected_before_reading_body}` (both) |
| Read length before allocating | conforms | `wire.rs::read_frame` checks the prefix before `vec![0u8; len]` | `tests/conformance_wire.rs::oversize_length_prefix_is_rejected_before_reading_body` (both) |
| Handle partial reads/writes; serialise one writer per connection | conforms | `wire.rs::read_frame`, `router/mod.rs::{write_selected, write_loop}` (one writer task) | `tests/conformance_wire.rs::{a_frame_written_one_byte_at_a_time_still_decodes, a_reply_read_one_byte_at_a_time_still_decodes, truncated_frame_disconnects_cleanly}` (both) |
| No ancillary-FD tricks in the first file-backed implementation | conforms | `transport.rs` carries bytes only | — |
| A future memory backend keeps the same client/ownership API | not-implemented (future). Owner: a later storage backend | — | — |
| `bus.hello` body `{clientId, clientIncarnation, supportedMajors}`, no attachments; reply `{routerId, connectionId, selectedMajor, selectedMinor, contractDigest, limits}` | conforms | `router/state.rs::hello`, `limits.rs::Limits::to_json` | `tests/wire.rs::hello_negotiation` (both), `tests/conformance_wire.rs::{hello_refuses_attachments, hello_reports_the_contract_digest_and_valid_limits}` (both) |
| Refuse incompatible majors or identity mismatch before registration | conforms (`VERSION_MISMATCH`, `NOT_AUTHORIZED`) | `router/state.rs::hello` | `tests/conformance_wire.rs::hello_refuses_an_unsupported_major` (both), `tests/wire.rs::hello_refusals` (both) |
| Schema changes change `contractDigest` | conforms: the digest is the SHA-256 of `wire::CONTRACT`, which lists every operation, delivery and notice; the client refuses a router whose digest differs | `wire.rs::{CONTRACT, contract_digest}`, `client/mod.rs::connect` | `tests/conformance_wire.rs::memory_and_unix_negotiate_the_identical_contract` |
| Bodies reject unknown fields and `minor` must be 0 after hello | deviates-allowed (stricter than "unknown envelope fields", forbidden nowhere; section 4's envelope fixes `minor: 0`) | `wire.rs::Fields::finish`, `router/state.rs::handle` | `tests/wire.rs::body_errors_keep_the_connection` (both), `tests/sol_review_races.rs::client_rejects_unknown_reply_fields_and_invalid_direction_rules` |
| The connection reader dispatches replies and requests without blocking on user handlers | conforms: the reader hands `Request`/`Message` to unbounded per-handle channels and returns | `client/reactor.rs::{read_loop, on_delivery}` | `tests/pubsub.rs::saturated_subscriber_does_not_block_control` (both) |
| Artifact I/O and hashing run outside the routing critical section; no routing lock across slow I/O | conforms: `Outcome::Allocate`/`Seal` leave the lock, run on the blocking pool and re-enter; unlinks happen after the guard drops | `router/mod.rs::{read_loop, Inner::with_state}`, `store.rs::seal` | `tests/artifacts.rs::quotas_are_enforced` (both), `tests/perf.rs` (seal p50 1.3 ms, publish admission p50 0.4 ms) |
## 5. Operation registry
| Requirement | Status | Code | Test |
| --- | --- | --- | --- |
| Replies are `{ok:true, value}` or `{ok:false, error:{code, message, dispatch}}` | conforms | `router/state.rs::reply_body`, `client/reactor.rs::parse_reply` | `tests/wire.rs::body_errors_keep_the_connection` (both) |
| All 19 commands of the table exist with the listed bodies and reply values | conforms | `router/state.rs::handle` dispatch table; `wire.rs::CONTRACT` | `tests/conformance_wire.rs` + `tests/{rpc,pubsub,artifacts}.rs` (both) |
| `rpc.responder.release {callId, requestDeliveryId} -> {released}` is added to the registry | deviates-allowed: section 4's "Changes to these draft schemas change contractDigest" (this is a draft), and it is what keeps section 6's "bounded call correlation metadata" bounded when a handler keeps a responder after dropping the request. Recorded as an amendment in bus-v1 section 12 | `router/state.rs::op_responder_release`, `client/handles.rs::ReplyGuard` | `tests/sol_rereview_regressions.rs::{dropping_last_attached_responder_settles_the_call, dropping_last_responder_clone_settles_the_call}` |
| Release batches carry 1..64 ids | conforms | `wire.rs::MAX_BATCH`, `Fields::array` | `tests/artifacts.rs::release_ids_are_watermarked` (both) |
| No attachments on management commands except `rpc.call`, `rpc.reply`, `publish` | conforms | `router/state.rs::handle` | `tests/wire.rs::body_errors_keep_the_connection` (both) |
| `accepted`/`routed`/`removed`/`declared`/`cleared`/`deleted`/`replayLatest` are booleans; `subscribers`/`replaced`/`released` are U64 counts | conforms | `router/state.rs` (`.into()` for bools, `to_string()` for counts) | `tests/pubsub.rs::bounded_fifo_and_atomic_backpressure` (both), `client/reactor.rs::validate_reply_value` |
| Released counts count newly released roots, so an idempotent repeat may report zero | conforms | `router/state.rs::{op_consumed, op_release}` | `tests/artifacts.rs::release_ids_are_watermarked` (both) |
| Queue/credit requests are integers 1..65535 and cannot exceed configured limits | conforms | `wire.rs::MAX_CREDIT`, `op_register`/`op_subscribe` quota checks | `tests/pubsub.rs::subscription_and_topic_validation` (both), `tests/rpc.rs::service_queue_backpressure` (both) |
| Method strings are 1..128 printable ASCII | conforms | `wire.rs::is_method` | `wire.rs::tests::scalars` |
| The call target is a service name; `expectedIncarnation` is the registration id | conforms | `router/state.rs::op_call` (`f.name`, `f.nullable_id`) | `tests/rpc.rs::registration_is_exclusive_and_pinned` (both) |
| Location grants are `{storeId, relativePath}` resolved under the configured root; absolute paths, parent traversal and symlink escapes are rejected | conforms | `store.rs::resolve`, opened `O_NOFOLLOW` | `store.rs::tests::resolve_refuses_escapes`, `tests/conformance_artifacts.rs::issued_locations_are_relative_and_contained` (both) |
| Locations are SDK-private and do not appear in an application's `ArtifactRef`; runtime paths are not committed into schemas | conforms: `Location` only ever appears in `artifact.allocate`/`artifact.open` replies | `wire.rs::Location`, `client/handles.rs::Artifact::open` | `tests/conformance_artifacts.rs::issued_locations_are_relative_and_contained` (both) |
| Deliveries carry exactly the listed bodies (`rpc.request`, `rpc.result`, `topic.message`) | conforms | `router/state.rs::{Call::request_body, Call::result_body, TopicMsg::body}` | `tests/conformance_wire.rs` + `client/reactor.rs::on_delivery` strict parse |
| `caller`/`responder` include clientId and clientIncarnation | conforms | `wire.rs::Identity` | `tests/rpc.rs::request_reply_roundtrip` (both) |
| Delivery attachment ownerIds are replaced by the recipient's deliveryId; source tokens are never delegated | conforms, and the client refuses a delivery whose attachment owner is not its delivery | `router/state.rs::attachments_with_owner`, `client/reactor.rs::attachments` | `tests/conformance_artifacts.rs::owner_ids_are_scoped_to_their_connection` (both) |
| `topicSequence` and counters are U64 strings; the router assigns the ids; the SDK exposes typed payloads plus handles | conforms | `router/state.rs::TopicMsg::body`, `client/handles.rs::Message` | `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) |
| Required bounded notices: route removal, subscription closure, call failure | conforms | `router/state.rs::{remove_service, shutdown, op_responder_release}` | `tests/pubsub.rs::router_shutdown_closes_subscriptions_with_notices` (both), `tests/rpc.rs::service_disconnect_fails_calls` (both) |
| Who gets those notices, and when | deviates-allowed: the draft names the notices but not their audience. `route.removed` goes to callers with open calls on the removed registration, `subscription.closed` only at router shutdown (nothing else ends a subscription without the client's own act), and `connection.closing` is an added notice before every router-initiated close | `router/state.rs::{remove_service, shutdown, violation}` | `tests/wire.rs::malformed_frames_close_the_connection` (both) |
| If notice capacity is exhausted, close the connection rather than lose control-plane correctness | conforms | `router/state.rs::push_control` | `tests/wire.rs::control_lane_exhaustion_closes` (both) |
## 6. RPC behaviour
| Requirement | Status | Code | Test |
| --- | --- | --- | --- |
| `call-<U64>` with increasing serials per connected client; reused or retired ids are rejected, never executed again | conforms: a syntactically valid id advances the watermark even when admission is refused | `router/state.rs::op_call` (`call_watermark`) | `tests/sol_review_races.rs::rejected_call_id_still_advances_monotonic_watermark`, `tests/rpc.rs::raw_call_ids_and_forged_replies` (both) |
| Reconnecting creates a new incarnation rather than reviving old calls | conforms | `router/state.rs::{hello, disconnect}` | `tests/sol_review_races.rs::caller_disconnect_cleanup_works_before_and_after_consumption` |
| An RPC targets one registered service, not a broadcast subject | conforms | `router/state.rs::op_call` | `tests/rpc.rs::request_reply_roundtrip` (both) |
| First-dispatch FIFO per caller and service; responses may complete out of order and correlate by callId | conforms | `router/state.rs::{Svc::queue, dispatch_rpc}` | `tests/rpc.rs::fifo_dispatch_and_out_of_order_completion` (both), `tests/conformance_routing.rs::out_of_order_replies_correlate_across_concurrent_callers` (both) |
| A service dispatcher can answer status concurrently with a long mutation | conforms | `router/state.rs::dispatch_rpc` (in-flight credits, not one-at-a-time) | `tests/bus_acceptance.rs::a_status_rpc_responds_while_another_handler_is_delayed` (both) |
| The router implements no frame barriers or numerical ordering | conforms by absence | `router/state.rs` (no domain fields) | `tests/integration.rs::session_over_one_router` (both) |
| Admission validates route, pinned incarnation, size, quotas and every source artifact owner, and establishes request-delivery roots atomically before accepting | conforms: every check precedes the first mutation, then roots, queue entry and reply | `router/state.rs::op_call` | `tests/conformance_artifacts.rs::rejected_call_creates_no_roots` (both), `tests/artifacts.rs::failed_admission_is_atomic` (both) |
| Rejection establishes no delivery and drops provisional roots | conforms | `router/state.rs::op_call` (validate-then-mutate) | `tests/conformance_artifacts.rs::rejected_call_creates_no_roots` (both) |
| An accepted call is not proof that its handler ran | conforms: `accepted` is admission only; the terminal outcome arrives as `rpc.result` or `call.failed` | `client/mod.rs::call`, `client/handles.rs::PendingCall` | `tests/rpc.rs::service_disconnect_fails_calls` (both) |
| Mark dispatched before any request bytes can reach the target; later transport loss is an unknown outcome | conforms: `Phase::Dispatched` and the delivery id are set when the frame is selected, before a byte is written | `router/state.rs::{next_frame, dispatch_rpc}` | `tests/sol_rereview_regressions.rs::shutdown_cancels_partial_rpc_request_before_reclaiming_attachment` |
| The service replies with its own owned handles; the router establishes caller-result ownership before accepting the reply | conforms | `router/state.rs::op_reply` (`check_attachments`, then `add_roots`, then the phase change) | `tests/rpc.rs::endpoint_cache_replays_artifact_results` (both) |
| Only bounded call correlation metadata is kept until the result is consumed or the caller detaches; not an indefinite result cache | conforms: the record dies with consumption, detachment, disconnect or the last responder release; reply capabilities share the per-client owner bound | `router/state.rs::{remove_call, dispatch_rpc, op_responder_release}` | `tests/sol_review_races.rs::{cancel_after_request_consumption_retires_correlation, dropping_service_with_buffered_requests_retires_every_call}` |
| A second `rpc.reply` for the same call is rejected, not routed twice | conforms (`CALL_GONE`) | `router/state.rs::op_reply` | `tests/rpc.rs::replies_are_single_and_independent_of_the_request_guard` (both) |
| Responding does not release the request's delivery guard | conforms | `client/handles.rs::{Request, Responder}` (separate guards) | `tests/rpc.rs::replies_are_single_and_independent_of_the_request_guard` (both) |
| No automatic retry or failover; never route a retry automatically to a restarted worker | conforms | `router/state.rs::{remove_service, disconnect}` fail calls instead of re-queueing | `tests/bus_acceptance.rs::no_automatic_retry_or_failover_onto_a_replacement_registration` (both) |
| A deadline belongs to the calling client; on timeout it may cancel | conforms: no budget on the wire; `result()` is cancel-safe under `tokio::time::timeout` | `client/handles.rs::PendingCall::{result, cancel}` | `tests/rpc.rs::cancellation_states` (both) |
| Domain retries use a fresh callId with the same domain requestId/body, pinned to the same incarnation; endpoint dedup supplies safe replay; the router does not infer it from method names | conforms | `client/mod.rs::call`; the router carries `payload` opaquely | `tests/bus_acceptance.rs::a_retransmission_repeats_the_domain_request_under_a_fresh_call_id` (both), `tests/rpc.rs::endpoint_cache_replays_artifact_results` (both) |
| Queued cancellation releases its queued artifact roots and returns `cancelled-before-dispatch` | conforms | `router/state.rs::op_cancel` -> `remove_call` -> `drop_roots` | `tests/conformance_routing.rs::cancel_before_dispatch_releases_queued_artifact_roots` (both) |
| After dispatch, return `execution-unknown` and keep the recipient's delivery alive until consumed or disconnected | conforms | `router/state.rs::op_cancel` (`detached = true`, delivery untouched) | `tests/rpc.rs::cancellation_states` (both), `tests/conformance_routing.rs::caller_disconnect_detaches_dispatched_call_but_service_keeps_serving` (both) |
| A later reply to a detached call returns `routed:false` with no caller-result roots; the service still owns any retained result | conforms | `router/state.rs::op_reply` (`call.detached` branch, before `add_roots`) | `tests/conformance_routing.rs::cancel_after_dispatch_then_late_reply_with_artifact_is_not_routed` (both) |
| A terminal result already admitted makes cancellation report `completed`; the client drains and consumes it | conforms | `router/state.rs::op_cancel` (`Phase::Replied`), `client/reactor.rs::on_delivery` consumes an abandoned result | `tests/rpc.rs::{cancellation_states, dropped_call_is_cancelled_and_late_result_consumed}` (both) |
| A retired or unknown correlation reports `call-gone`; those four strings are the complete enum | conforms | `router/state.rs::op_cancel`, `client/handles.rs::CancelState` | `tests/rpc.rs::cancellation_states` (both), `client/reactor.rs::validate_reply_value` |
| No cancel state authorises re-execution; cancelling a future does not abandon incoming delivery ownership | conforms | `client/handles.rs::CallGuard::drop` (best-effort cancel, result still consumed) | `tests/rpc.rs::dropped_call_is_cancelled_and_late_result_consumed` (both), `tests/sol_review_races.rs::reply_racing_cancel_has_only_the_two_contract_outcomes` |
| Endpoint replay caches Artifact handles plus payload, not bare references, and owns holds until eviction | conforms (SDK support; the discipline is the endpoint's) | `client/handles.rs::Artifact::retain`, `ArtifactWriter::seal` returns a hold | `tests/rpc.rs::endpoint_cache_replays_artifact_results` (both), `tests/bus_acceptance.rs::a_lost_result_leaks_no_roots_and_the_endpoint_cache_still_replays` (both) |
| Re-delivery gets new delivery ids pointing to the same immutable bytes | conforms | `router/state.rs::dispatch_rpc` (fresh `dlv-<n>` per delivery) | `tests/rpc.rs::endpoint_cache_replays_artifact_results` (both) |
| An expired domain cache returns `RESULT_EXPIRED` | not-implemented: a domain code, not a transport code. Owner: `fly-session-rpc` (ipc-v1) | — | — |
## 7. Pub/sub semantics
| Requirement | Status | Code | Test |
| --- | --- | --- | --- |
| Topic declaration teaches the router nothing about meaning; no hardcoded frame/brain topics | conforms | `router/state.rs::{Topic, op_declare}` | `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) |
| `latest`: one queued value, replacing only an undelivered one; replacement releases that entry's roots; delivered or in-use messages are never reclaimed early; maxQueued is exactly 1 | conforms | `router/state.rs::{op_publish (latest branch), op_subscribe}` | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both), `tests/conformance_artifacts.rs::latest_mode_holds_at_most_two_roots_delivered_plus_queued` (both) |
| `bounded`: FIFO, no coalescing or silent loss; when capacity is unavailable, reject with `BACKPRESSURE` before admitting any delivery | conforms | `router/state.rs::op_publish` (pre-checks every subscriber) | `tests/pubsub.rs::bounded_fifo_and_atomic_backpressure` (both), `tests/conformance_routing.rs::bounded_overflow_rolls_back_all_artifact_roots` (both) |
| maxInFlight credits return only on `delivery.consumed`, not on socket write completion | conforms | `router/state.rs::release_owner` (credit returned when the owner is released) | `tests/pubsub.rs::credits_return_only_on_consume` (both), `tests/conformance_routing.rs::bounded_credit_waits_for_every_extracted_artifact` (both) |
| A latest subscriber with all credits in use still has one replaceable queued value | conforms | `router/state.rs::{Sub::queue, dispatch_topic}` (queue and credits are separate) | `tests/bus_acceptance.rs::both_transports_produce_equivalent_behaviour_traces` (events 21 and 24: `publish replaced=1`, then `latest seq=3 replaced=1`) |
| Atomic subscriber/retention snapshot at admission; validate and reserve every queue entry and owner budget before accepting | conforms: one mutex, validate-then-mutate | `router/state.rs::op_publish` | `tests/artifacts.rs::failed_admission_is_atomic` (both) |
| A bounded overflow rejects the whole publish: no partial fan-out, no retained-latest update | conforms | `router/state.rs::op_publish` | `tests/conformance_routing.rs::bounded_overflow_rolls_back_all_artifact_roots` (both) |
| On acceptance, one `topicSequence` and roots for every delivery and the optional retained value | conforms; a refused publication spends no sequence number | `router/state.rs::op_publish` (`t.sequence += 1` after the checks) | `tests/pubsub.rs::bounded_fifo_and_atomic_backpressure` (both) |
| Different topics have no total ordering; multiple publishers follow router acceptance order | conforms: per-topic sequence only | `router/state.rs::Topic::sequence` | `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) |
| The publication reply counts accepted subscriptions and replaced queue entries, not consumers that processed data | conforms | `router/state.rs::op_publish` reply | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both) |
| `replaced` on a delivery reports how many undelivered messages were coalesced since that subscription's preceding delivery | conforms | `router/state.rs::{Sub::replaced, dispatch_topic}` (taken at dispatch) | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both) |
| Optional `retained:latest` holds one last message and its artifacts independent of subscribers | conforms | `router/state.rs::op_publish` (retain branch) | `tests/conformance_artifacts.rs::retained_topic_value_holds_a_root_independent_of_subscribers` (both) |
| `replayLatest` enqueues the retained value before subsequent accepted publications; bounded preserves the order, latest may coalesce it | conforms | `router/state.rs::op_subscribe` (replay is enqueued under the subscribe lock) | `tests/conformance_routing.rs::latest_replay_is_ordered_ahead_of_a_racing_publish` (both), `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) |
| Replay uses the original topicSequence, a fresh deliveryId and explicit roots | conforms | `router/state.rs::op_subscribe` (`add_roots`, the same `Arc<TopicMsg>`) | `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) |
| Without retention, a zero-subscriber publication retains no ownership after admission | conforms | `router/state.rs::op_publish` | `tests/pubsub.rs::zero_subscriber_publish_retains_nothing` (both) |
| Clearing a topic releases only its retained root, not active consumers | conforms | `router/state.rs::op_clear` | `tests/conformance_routing.rs::cleared_topic_gives_no_replay_until_a_fresh_publish` (both) |
| Topic count and retained bytes are capped | conforms; `max_retained_bytes` is added to the draft's table because this sentence requires it | `limits.rs::{max_topics, max_retained_bytes}`, `router/state.rs::{op_declare, op_publish}` | `tests/pubsub.rs::topic_and_retention_quotas` (both) |
| No durable replay, automatic redelivery or exactly-once claim | conforms by absence | `router/state.rs` (queues are in memory and die with the connection) | `tests/artifacts.rs::router_restart_invalidates_old_handles` (both) |
| Deleting or redeclaring a topic creates a fresh topicIncarnation; a reset sequence cannot be read as continuation | conforms | `router/state.rs::{op_delete, op_declare}` | `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both) |
| Old subscription deliveries keep their original incarnation and ownership until consumed | conforms | `router/state.rs::drop_subscription` (matches on the incarnation), delivery bodies carry it | `tests/pubsub.rs::unsubscribe_discards_queue_but_not_deliveries` (both) |
| `topic.delete` only with no subscribers | conforms (`CONFLICT`) | `router/state.rs::op_delete` | `tests/pubsub.rs::subscription_and_topic_validation` (both) |
| Bus admission, message consumption and durable storage acknowledgment are three different events | conforms: `publish` returns admission counts, `delivery.consumed` is separate, and there is no storage ack in the bus | `router/state.rs::{op_publish, op_consumed}` | `tests/pubsub.rs::credits_return_only_on_consume` (both) |
| A topic must be declared before publish or subscribe | deviates-allowed: the draft is silent on undeclared topics, while "Topic count and retained bytes are capped" and `topic.declare`'s "conflicting settings fail" both imply a registry a publication cannot create by accident. `NO_TOPIC` names the refusal (amendment, section 12); `topic.delete` of an unknown topic still answers `deleted:false` | `router/state.rs::{op_publish, op_subscribe, op_clear}` | `tests/pubsub.rs::subscription_and_topic_validation` (both) |
## 8. Artifact lifecycle and garbage collection
### 8.1 Immutable object lifecycle
| Requirement | Status | Code | Test |
| --- | --- | --- | --- |
| `ALLOCATED/WRITING -> SEALED -> owned -> COLLECTED`, and an abandoned or disconnected writer goes straight to COLLECTED | conforms | `router/state.rs::{ArtState, abandon_writer, finish_seal}` | `tests/artifacts.rs::{writer_drop_releases_staging, disconnect_releases_all_but_retained}` (both), `tests/conformance_artifacts.rs::disconnect_abandons_an_unsealed_writer` (both) |
| `ArtifactRef` is an identity, not an address or authority; opening needs a current root on that connection | conforms | `router/state.rs::{check_owned, op_open}` | `tests/conformance_artifacts.rs::owner_ids_are_scoped_to_their_connection` (both) |
| storeId is the store incarnation; old handles fail after restart | conforms (`ARTIFACT_GONE`) | `router/state.rs::check_owned` | `tests/artifacts.rs::router_restart_invalidates_old_handles` (both) |
| No artifact id or inode reuse; `generation` is 1 | conforms | `wire.rs::GENERATION`, `router/state.rs::next_artifact` (monotonic) | `tests/conformance_artifacts.rs::stale_store_incarnation_and_generation_are_rejected` (both) |
| Content hashes optional for live frames, mandatory where a domain contract says so | conforms: both paths exist and the router verifies what it is given | `client/handles.rs::ArtifactWriter::seal_with_digest`, `store.rs::copy_exact` | `tests/artifacts.rs::seal_checks_length_and_digest` (both) |
| The first backend is runtime-configured local files, optionally on tmpfs | conforms | `store.rs::Store::create` under `RouterConfig::store_root` | `store.rs::tests::orphans_are_removed_and_live_stores_kept` |
| The producer writes staging storage outside the message stream | conforms | `store.rs::create_staging`, `client/mod.rs::Artifacts::allocate` | `tests/artifacts.rs::allocate_write_seal_read` (both) |
| Seal closes writable handles in the SDK, checks length and digest, then finishes an immutable store-owned object before acknowledging | conforms: a writable descriptor kept after sealing reaches only the unlinked staging inode | `client/handles.rs::seal_with_digest` (drops the file first), `store.rs::seal` (fresh 0444 inode) | `tests/artifacts.rs::seal_is_immune_to_live_writable_handles` (both), `tests/conformance_artifacts.rs::seal_is_immutable_despite_a_stale_writable_handle` (both) |
| A copy into a fresh sealed inode is allowed; account for both allocations during sealing | conforms | `router/state.rs::op_seal` (`store_bytes += len` for the copy, released in `finish_seal`) | `tests/artifacts.rs::quotas_are_enforced` (both) |
| No per-frame fsync for transient media | conforms by absence | `store.rs` | `tests/perf.rs` (seal p50 1.3 ms for 1.2 MB) |
| Consumers resolve a readLocation through `artifact.open` and read it read-only | conforms | `router/state.rs::op_open`, `store.rs::open_read` | `tests/conformance_artifacts.rs::allocate_write_seal_open_roundtrip_and_mismatches` (both) |
| Locations are private grants, not placed in application bodies or public feeds | conforms | `router/state.rs::op_open` reply only | `tests/conformance_artifacts.rs::issued_locations_are_relative_and_contained` (both) |
| All filesystem access stays behind the client Artifact API; no second bulk-transfer server | conforms in the API. The store is only as private as the OS user, which the crate's Limitations section states | `client/handles.rs`, `store.rs` | `store.rs::tests::resolve_refuses_escapes` |
### 8.2 What owns an artifact
| Requirement | Status | Code | Test |
| --- | --- | --- | --- |
| Roots: producer hold/active writer, accepted queued delivery, in-flight delivery, retained latest, explicit hold | conforms, all five | `router/state.rs::{Owner, add_roots, drop_roots}` | `tests/conformance_artifacts.rs::{queued_deliveries_hold_roots_before_dispatch, collection_waits_for_every_retained_owner, retained_topic_value_holds_a_root_independent_of_subscribers}` (both) |
| Seal transfers the unique writer into a hold | conforms; the seal reply reuses the writer's own ownerId to express exactly that transfer | `router/state.rs::finish_seal` | `tests/artifacts.rs::allocate_write_seal_read` (both) |
| Admission creates destination roots before the sender may relinquish source roots | conforms: the SDK holds every source `OwnerGuard` until the router has answered | `client/reactor.rs::OutCommand::keep`, `router/state.rs::{op_call, op_reply, op_publish}` | `tests/conformance_artifacts.rs::forward_requires_the_source_owner_still_live` (both) |
| A timeout must not drop a source guard while an unsent operation might still be admitted; the client keeps the guard until the transport outcome is known | conforms: an unsent command's guards travel with it and are released only when it fails or is answered | `client/reactor.rs::{next_outgoing, fail_all}` | `tests/sol_review_races.rs::unsent_oversized_call_rolls_back_its_local_slot`, `tests/artifacts.rs::abandoned_futures_do_not_leak_owners` (both) |
| Every envelope lists its complete artifact set; duplicates in one delivery count once | conforms | `router/state.rs::{check_attachments, dedup}` | `tests/conformance_artifacts.rs::allocate_write_seal_open_roundtrip_and_mismatches` (both) |
| A retained topic and several consumers can reference the same bytes; the router updates metadata only and never copies bytes for fan-out | conforms | `router/state.rs::op_publish` (`Arc<TopicMsg>` plus root counts) | `tests/artifacts.rs::fan_out_shares_one_object_and_the_last_consumer_collects` (both), `tests/perf.rs` (store peak 3.7 MB for three consumers) |
### 8.3 Consumed means no remaining use
| Requirement | Status | Code | Test |
| --- | --- | --- | --- |
| The incoming message owns a shared DeliveryGuard; extracting an Artifact clones it; dropping the message alone does not consume the delivery | conforms | `client/handles.rs::{OwnerGuard, Message::artifact}` | `tests/artifacts.rs::extracted_artifacts_outlive_their_message` (both), `tests/conformance_artifacts.rs::extracted_artifact_outlives_the_message_it_came_from` (both) |
| Local handle clones need no bus round trip; dropping the last guard queues `delivery.consumed` on a bounded control lane | conforms | `client/handles.rs::OwnerGuard::drop`, `client/reactor.rs::push_control` | `tests/pubsub.rs::credits_return_only_on_consume` (both) |
| Ownership is at delivery granularity; independent retention needs `artifact.retain` before the guard is dropped | conforms | `client/handles.rs::Artifact::retain` | `tests/conformance_artifacts.rs::explicit_retain_outlives_the_original_hold` (both) |
| A domain acknowledgment implicitly drops nothing | conforms by absence: only a guard drop or an explicit release ends ownership | `client/handles.rs::OwnerGuard` | `tests/rpc.rs::endpoint_cache_replays_artifact_results` (both) |
| An allocation or seal grant that arrives after its caller abandoned the future is still processed and released; no owner the application never saw is leaked | conforms: the reactor builds the handle, so an undelivered reply drops it | `client/reactor.rs::{complete, Hook::Owner}` | `tests/artifacts.rs::abandoned_futures_do_not_leak_owners` (both) |
| An in-progress seal has a bounded I/O hold; on producer disconnect it cleans up and never publishes an ownerless object | conforms | `router/state.rs::finish_seal` (`owner_live` check), `router/mod.rs::read_loop` seal task | `tests/conformance_artifacts.rs::disconnect_abandons_an_unsealed_writer` (both) |
| Dropping a response future is not consumption: the client owns queued results until surfaced, discarded or disconnected | conforms | `client/reactor.rs::{CallSlot, on_delivery}` | `tests/rpc.rs::dropped_call_is_cancelled_and_late_result_consumed` (both) |
| Receivers await asynchronous CPU/GPU use before releasing the guard | conforms as far as the API can enforce: the guard lives as long as any `Artifact`/`ArtifactFile` clone | `client/handles.rs::{Artifact, ArtifactFile}` | `tests/conformance_routing.rs::bounded_credit_waits_for_every_extracted_artifact` (both) |
| A pointer from a mapping cannot outlive its Artifact; FFI wrappers enforce it | not-implemented: no mapping and no FFI surface exists. Owner: a future mmap or binding | — | — |
| Release commands are batched, idempotent and scoped to the owning connection | conforms | `router/state.rs::{op_release, op_consumed}`, `client/reactor.rs::take_batch` | `tests/artifacts.rs::owners_are_scoped_to_their_connection` (both) |
| Delivery and hold ids use monotonic per-connection serials with separate watermarks; a retired id is a no-op, a never-issued one an error; no tombstone per frame | conforms | `router/state.rs::{Conn::delivery_issued, Conn::hold_issued, op_consumed, op_release}` | `tests/artifacts.rs::release_ids_are_watermarked` (both) |
| Control-lane exhaustion closes the connection instead of losing releases | conforms on both sides | `router/state.rs::push_control`, `client/reactor.rs::push_control` | `tests/wire.rs::control_lane_exhaustion_closes` (both) |
### 8.4 Crash, disconnect and safe physical reclamation
| Requirement | Status | Code | Test |
| --- | --- | --- | --- |
| On disconnect: unregister services and subscriptions, cancel queued deliveries, release that connection's writers, holds and delivery roots | conforms | `router/state.rs::disconnect` | `tests/artifacts.rs::disconnect_releases_all_but_retained` (both), `tests/conformance_artifacts.rs::{disconnect_releases_an_explicit_hold, disconnect_releases_queued_and_dispatched_deliveries}` (both) |
| Retained topic roots stay router-owned | conforms | `router/state.rs::disconnect` (topics untouched) | `tests/conformance_routing.rs::subscriber_disconnect_releases_queued_and_delivered_artifacts_but_not_retention` (both) |
| Late replies and releases cannot attach to a new connection or service incarnation | conforms: owners are per connection and calls remember their service incarnation | `router/state.rs::{check_owned, op_reply, remove_service}` | `tests/artifacts.rs::owners_are_scoped_to_their_connection` (both), `tests/rpc.rs::raw_call_ids_and_forged_replies` (both) |
| Teardown reclaims a connection's roots without racing the frame it is writing: either a complete frame precedes the reclamation or a partial one is cut and nothing more is appended | conforms, **fixed on this branch** (see the section 9 row on starvation). Teardown now marks the stream closing once, before waiting for the poll already in progress, so only that one poll can still write and every later one is refused | `router/state.rs::WriteGate`, `router/mod.rs::write_selected`, `router/state.rs::close_conn` | `tests/sol_rereview_regressions.rs::{teardown_waits_for_an_active_transport_poll_before_reclaiming, shutdown_does_not_deliver_a_frame_after_releasing_its_owner, protocol_close_cancels_partial_topic_frame_before_reclaiming_attachment, shutdown_cancels_partial_rpc_request_before_reclaiming_attachment, shutdown_cancels_partial_rpc_result_before_reclaiming_attachment}` |
| GC removes the registry entry and unlinks the sealed object after its final root is gone | conforms; the unlink happens after the routing lock is released | `router/state.rs::drop_roots`, `router/mod.rs::Inner::with_state` | `tests/artifacts.rs::fan_out_shares_one_object_and_the_last_consumer_collects` (both) |
| Existing mappings stay valid until the OS closes them; never overwrite the inode or reuse its bytes | conforms: sealed files are 0444, written once and only unlinked | `store.rs::{seal, remove}` | `tests/bus_acceptance.rs::disconnect_releases_logical_ownership_without_mutating_open_bytes` (both) |
| Logical reclamation is not proof of physical release; measurements include OS mappings and client memory | conforms: `RouterStats` is documented as logical, and the measurement reports process RSS | `router/state.rs::RouterStats`, `tests/perf.rs` | `tests/perf.rs` (RSS 11 to 16 MB) |
| No TTL may reclaim a live owned artifact | conforms by absence: nothing in the router expires an owned root | `router/state.rs` | `tests/conformance_artifacts.rs::collection_waits_for_every_retained_owner` (both) |
| Limits may disconnect a consumer but cannot overwrite memory under a renderer | conforms: exhaustion closes the connection, which releases roots; bytes are never rewritten | `router/state.rs::push_control`, `store.rs` | `tests/wire.rs::control_lane_exhaustion_closes` (both) |
| Future pooled shared memory must prove equivalent lifetime/generation safety | not-implemented (deferred by the draft). Owner: a later storage backend | — | — |
| Router restart makes a new routerId/storeId, loses routes, queues and retention, and invalidates old handles | conforms | `router/mod.rs::Router::new`, `store.rs::Store::create` | `tests/artifacts.rs::router_restart_invalidates_old_handles` (both) |
| Orphan files from a stopped router are cleaned without being treated as durable checkpoints | conforms, at the next router start on the same root (a `flock`-free marked directory) | `store.rs::clean_orphans` | `store.rs::tests::orphans_are_removed_and_live_stores_kept` |
| Live sessions fail their epoch and use coherent recovery | not-implemented: a session responsibility. Owner: STATE-01 / step-v1 | — | — |
## 9. Bounds, scheduling and failure reporting
| Requirement | Status | Code | Test |
| --- | --- | --- | --- |
| Every default of the table, exactly: clients/services/topics 64/256/512; subscriptions 128 per client and 1024 total; control envelope 64 KiB; active calls 64; service queued/in-flight 16/16; latest 1/2; bounded 64/16; active owners 256; store 512 MiB and 128 MiB per object; per-client queued envelope bytes 1 MiB; reserved lane 128 frames and 1 MiB | conforms | `limits.rs::Limits::default`, `wire.rs::MAX_ENVELOPE_BYTES` | `tests/conformance_wire.rs::hello_reports_the_contract_digest_and_valid_limits` (both), `tests/pubsub.rs::topic_and_retention_quotas` (both) |
| Limits are configured explicitly and a configuration the router could not honour is refused | conforms | `limits.rs::Limits::validate`, `router/mod.rs::Router::new` | `tests/rpc.rs::active_call_limit` (both), `tests/artifacts.rs::quotas_are_enforced` (both) |
| The per-client queued envelope byte budget counts `bounded` subscriptions only, and latest slots are bounded at subscriptions x 64 KiB instead | conforms to the amended table. The draft's single row was the audit's contradiction 1: this section also forbids a latest spectator from being the reason a publication is refused, so its slot cannot sit in a budget whose overflow rejects one. The coordinator amended the row and gave latest slots their own (bus-v1 section 12, 2026-09-22) | `limits.rs::max_queued_bytes_per_client`, `router/state.rs::op_publish` | `tests/bus_acceptance.rs::a_latest_subscriber_never_refuses_a_publication` (both), `tests/sol_review_races.rs::{retained_replay_obeys_bounded_queue_byte_quota, latest_replay_remains_bounded_outside_the_bounded_byte_pool}` |
| Reserve an owner allowance for lifecycle and results separately from ordinary telemetry | conforms | `limits.rs::reserved_owners_per_client`, `router/state.rs::{ordinary_budget_left, dispatch_rpc}` | `tests/conformance_artifacts.rs::artifact_bounds_and_owner_budget_are_enforced` (both) |
| Memory quotas account for staging, seal copies, queued deliveries and caches | conforms | `router/state.rs::{op_allocate, op_seal, Conn::queued_bytes}` | `tests/artifacts.rs::quotas_are_enforced` (both) |
| Ownership metadata is bounded even when many roots share one artifact | conforms: a root is a counter, and owners are bounded per client | `router/state.rs::{Art::roots, ordinary_budget_left}` | `tests/conformance_artifacts.rs::artifact_bounds_and_owner_budget_are_enforced` (both) |
| Disk-full, allocation failure or hash mismatch returns a typed artifact error and cleans provisional storage and roots | conforms (`QUOTA_EXCEEDED`, `STORE_FAILURE`, `ARTIFACT_MISMATCH`) | `router/state.rs::{op_allocate, finish_allocate, op_seal, finish_seal}`, `store.rs::seal` | `tests/artifacts.rs::seal_checks_length_and_digest` (both), `tests/conformance_artifacts.rs::allocate_write_seal_open_roundtrip_and_mismatches` (both) |
| The router fairly services clients | deviates-allowed: fairness is Tokio's scheduling plus a yield every 32 commands from one connection, and round robin between a connection's services and subscriptions. The draft sets no fairness metric; the crate's Limitations section calls this modest | `router/mod.rs::read_loop`, `router/state.rs::{dispatch_rpc, dispatch_topic}` | `tests/pubsub.rs::saturated_subscriber_does_not_block_control` (both) |
| Replies, release, cancellation and route-health control cannot be starved by telemetry | conforms: control first, then RPC, then topic data, with topic data given a turn after 16 higher-priority frames | `router/state.rs::{next_frame, TOPIC_STARVATION_LIMIT}` | `tests/sol_rereview_regressions.rs::{fair_topic_insertion_preserves_router_envelope_order, sdk_accepts_fair_topic_insertion_through_saturated_control_backlog}`, `tests/pubsub.rs::saturated_subscriber_does_not_block_control` (both) |
| ... and neither can connection teardown be starved by the frame it is waiting for | **was deviates-must-fix, fixed on this branch**. The write gate was a plain mutex held across each synchronous transport poll, so a writer sending a frame one byte per poll re-acquired it hundreds of times while teardown waited for it, and could finish a whole delivery before teardown got in: `teardown_waits_for_an_active_transport_poll_before_reclaiming` failed 6 runs out of 6 in release and about 1 in 5 in debug. The gate now separates "teardown has begun" (a flag set once, without waiting) from "a poll is in progress" (a condvar teardown waits on), so teardown's window is one poll instead of a whole frame, and no byte can follow it. No public signature changed | `router/state.rs::WriteGate`, `router/mod.rs::write_selected` | `tests/sol_rereview_regressions.rs::teardown_waits_for_an_active_transport_poll_before_reclaiming` (rewritten: a 50 KB delivery the resumed writer cannot finish, and a channel instead of a sleep) |
| Preserve FIFO for calls to a target despite lane scheduling | conforms: the service queue is FIFO and lane choice never reorders it | `router/state.rs::dispatch_rpc` | `tests/rpc.rs::fifo_dispatch_and_out_of_order_completion` (both) |
| Classification is an explicit generic envelope operation or policy, not a topic-name heuristic | conforms by construction: `next_frame` chooses a lane from the queue an item sits in (control, then RPC, then topic), and neither it nor `pop_control`/`dispatch_rpc`/`dispatch_topic` reads a service or topic name. A name reaches the scheduler only as opaque bytes inside an already-classified frame | `router/state.rs::{next_frame, pop_control, dispatch_rpc, dispatch_topic}` | `tests/bus_acceptance.rs::a_topic_named_like_a_notice_is_still_classified_as_topic_data` (both), and `tests/sol_rereview_regressions.rs::fair_topic_insertion_preserves_router_envelope_order` for the lane order itself |
| No indefinite wait inside the router on subscriber readiness or artifact I/O | conforms: a slow reader stalls only its own writer task; I/O leaves the lock | `router/mod.rs::{write_loop, read_loop}` | `tests/pubsub.rs::saturated_subscriber_does_not_block_control` (both) |
| Admission is bounded; rejected callers choose their own policy | conforms | `router/state.rs::{op_call, op_publish}` | `tests/rpc.rs::service_queue_backpressure` (both) |
| The thirteen transport error codes exist with those names | conforms | `error.rs::ErrorCode` | `tests/wire.rs::body_errors_keep_the_connection` (both) |
| Three more codes: `CONFLICT`, `NO_TOPIC`, `ARTIFACT_MISMATCH` | deviates-allowed: "Transport errors **include** ..." is not an exhaustive list, and each names a refusal the draft requires but leaves unnamed. Recorded as an amendment in bus-v1 section 12 | `error.rs::ErrorCode` | `tests/pubsub.rs::subscription_and_topic_validation` (both), `tests/artifacts.rs::seal_checks_length_and_digest` (both) |
| Before admission report `not-dispatched`; once dispatch might have occurred report `dispatched` or `unknown` conservatively | conforms | `error.rs::BusError::new` (not-dispatched by default), `router/state.rs` dispatched notices, `client/reactor.rs::fail_all` (unknown) | `tests/rpc.rs::{cancellation_states, service_disconnect_fails_calls}` (both), `tests/sol_review_races.rs::writer_failure_terminates_reader_and_pending_work` |
| Bounded subscriptions can reject a publication; latest spectators cannot hold a session transaction indefinitely | conforms: a latest subscriber never causes `BACKPRESSURE` | `router/state.rs::op_publish` (the latest branch skips every capacity check) | `tests/bus_acceptance.rs::a_latest_subscriber_never_refuses_a_publication` (both: 100 publications of 60 KB into one unconsumed slot, six times the bounded pool, none refused, 98 coalesced), with `tests/pubsub.rs::{bounded_fifo_and_atomic_backpressure, saturated_subscriber_does_not_block_control}` (both) for the bounded half |
| Sustained pinned-artifact quota exhaustion is surfaced as pressure, not solved by freeing live data | conforms: `QUOTA_EXCEEDED`, never eviction | `router/state.rs::{op_allocate, op_seal}` | `tests/artifacts.rs::quotas_are_enforced` (both) |
| Session and application policies choose disconnect, pause or fail; the router does not know which | conforms by absence | `router/state.rs` | `tests/pubsub.rs::saturated_subscriber_does_not_block_control` (both) |
## 10. Native-frame bandwidth check
| Requirement | Status | Code | Test |
| --- | --- | --- | --- |
| 640x480 RGBA is 1,228,800 bytes; 73.728 MB/s at 60 fps is the planning dimension | conforms as measured, not as a claim | `tests/perf.rs::{W, H, HZ}` | `tests/perf.rs` (120 frames in 2.00 s, 0 late) |
| Two agent workers plus one presentation consumer read the same immutable frame object | conforms | `router/state.rs::op_publish` (roots, not copies) | `tests/perf.rs` (three consumers, store peak 3.7 MB = three 1.2 MB objects in flight, not nine) |
| Bus messages contain only references; no 3x byte fan-out through the router | conforms | `wire.rs::Attachment` | `tests/artifacts.rs::fan_out_shares_one_object_and_the_last_consumer_collects` (both), `tests/perf.rs` |
| Readers still incur memory traffic; renderer readback and seal copying remain real costs | conforms, and both are now measured separately | `tests/perf.rs` (`producer copy into staging`, `seal`, `consumer readback`) | `tests/perf.rs` |
| No claim of zero-copy capture or measured host capacity | conforms: the measurement section below says so in those words | `tests/perf.rs` header | — |
| Nothing game-specific, no rendering, publishing or sampling logic in Flybus | conforms: no crate in the workspace depends on flybus yet, and the crate names no game, brain or stream concept | `crates/flybus/**` | `tests/integration.rs::session_over_one_router` (both; the domain lives in the test) |
## 11. Acceptance tests and implementation sequence
| bus-v1 item | Status | Test |
| --- | --- | --- |
| 1. Wire/router: schema, framing, Hello, exclusive routes, pinned incarnations, request/reply, disconnect, bounds; in-memory passes the same tests as Unix sockets | conforms | `tests/conformance_wire.rs` (45), `tests/wire.rs` (12), `tests/rpc.rs` (26), all `both_transports!`; `tests/bus_acceptance.rs::both_transports_produce_equivalent_behaviour_traces` |
| 2. Pub/sub: exact topics, FIFO/bounded rejection, latest coalescing, retained replay and clear, atomic fan-out, fair control/reply delivery under a saturated subscriber | conforms | `tests/pubsub.rs` (20), `tests/conformance_routing.rs` (24) |
| 3. Artifacts: allocate/seal/read; publication before seal fails; fan-out owns one object; the last consumer releases; a retained extracted frame survives a message drop | conforms | `tests/artifacts.rs` (28), `tests/conformance_artifacts.rs` (36) |
| 4. Faults: sender drops after admission, consumer dies mid-read, reply lost, queued frame replaced, subscription closes with in-use deliveries, router restarts, old release arrives; no double-free, use-after-reuse, unbounded tombstones or hidden replay | conforms | `tests/conformance_routing.rs::{caller_disconnect_detaches_dispatched_call_but_service_keeps_serving, subscriber_disconnect_releases_queued_and_delivered_artifacts_but_not_retention}`, `tests/artifacts.rs::{release_ids_are_watermarked, router_restart_invalidates_old_handles}`, `tests/bus_acceptance.rs::{a_lost_result_leaks_no_roots_and_the_endpoint_cache_still_replays, disconnect_releases_logical_ownership_without_mutating_open_bytes}`, `tests/sol_rereview_regressions.rs` (11) |
| 5. RPC cache: an endpoint retains an artifact-bearing result, the original caller consumes it, a domain retry still returns valid bytes, eviction drops the last hold | conforms | `tests/rpc.rs::endpoint_cache_replays_artifact_results` (both), `tests/bus_acceptance.rs::a_retransmission_repeats_the_domain_request_under_a_fresh_call_id` (both) |
| 6. Integration: two parallel fake agents, a complete-batch environment RPC, committed snapshot publication and a deliberately slow presentation consumer on one router | conforms | `tests/integration.rs::session_over_one_router` (both) |
| 7. Performance: 640x480x60 with three consumers, one delayed; p50/p95/p99 RPC latency, router CPU, copy and readback cost separately, RSS, store live and peak bytes, outstanding roots, collection lag, queue lengths, for one, two and four agents | conforms | `tests/perf.rs::frames_at_60hz_with_three_consumers` (`--ignored`); numbers below |
| The first executable example: a counter RPC, a pub/sub observer and a frame artifact held past message consumption, in one small Rust program, no game or browser | conforms | `examples/demo.rs` (`cargo run -p flybus --example demo`), asserted by `tests/example_demo.rs::the_example_shows_a_counter_rpc_an_observer_and_a_held_frame` |
## The implementation guide's acceptance bullets
Every bullet of BUS-01, BUS-02 and BUS-03, with the named test that proves it. A bullet a
pre-existing suite already covered is cited here rather than duplicated; the rest are the
tests in `tests/bus_acceptance.rs`, named after their bullet.
### BUS-01 — router and RPC, in-memory and Unix socket parity
| Bullet | Test |
| --- | --- |
| Partial frames and writes | `tests/conformance_wire.rs::{a_frame_written_one_byte_at_a_time_still_decodes, a_reply_read_one_byte_at_a_time_still_decodes, truncated_frame_disconnects_cleanly, length_prefix_is_little_endian}` (both), `tests/sol_rereview_regressions.rs::{shutdown_cancels_partial_rpc_request_before_reclaiming_attachment, protocol_close_cancels_partial_topic_frame_before_reclaiming_attachment}` |
| Disconnect after request | `tests/rpc.rs::service_disconnect_fails_calls` (both), `tests/conformance_routing.rs::caller_disconnect_detaches_dispatched_call_but_service_keeps_serving` (both), `tests/sol_review_races.rs::caller_disconnect_cleanup_works_before_and_after_consumption` |
| Lost result | **new** `tests/bus_acceptance.rs::a_lost_result_leaks_no_roots_and_the_endpoint_cache_still_replays` (both) |
| Retransmission fixtures | **new** `tests/bus_acceptance.rs::a_retransmission_repeats_the_domain_request_under_a_fresh_call_id` (both), with `tests/rpc.rs::endpoint_cache_replays_artifact_results` (both) |
| No automatic retry | **new** `tests/bus_acceptance.rs::no_automatic_retry_or_failover_onto_a_replacement_registration` (both) |
| Cancel after dispatch reports execution-unknown | `tests/rpc.rs::cancellation_states` (both), `tests/sol_review_races.rs::reply_racing_cancel_has_only_the_two_contract_outcomes` |
| Incarnation replacement is visible | `tests/rpc.rs::registration_is_exclusive_and_pinned` (both), `tests/conformance_routing.rs::{duplicate_registration_by_owner_itself_is_rejected, unpinned_call_after_incarnation_replacement_reaches_the_new_holder}` (both) |
| Out-of-order replies | `tests/rpc.rs::fifo_dispatch_and_out_of_order_completion` (both), `tests/conformance_routing.rs::out_of_order_replies_correlate_across_concurrent_callers` (both) |
| Status RPC responds while another handler is delayed | **new** `tests/bus_acceptance.rs::a_status_rpc_responds_while_another_handler_is_delayed` (both) |
| Saturation is bounded | `tests/rpc.rs::{service_queue_backpressure, active_call_limit}` (both), `tests/wire.rs::control_lane_exhaustion_closes` (both), `tests/sol_review_races.rs::pending_connections_are_bounded_and_hello_expires` |
| Both transports produce equivalent behaviour traces | **new** `tests/bus_acceptance.rs::both_transports_produce_equivalent_behaviour_traces`, over the trace recorder in `tests/common/mod.rs::Trace` |
The recorder keeps behaviour and refuses operational identity: `Trace::record` panics on any
router-issued id, so a trace holds methods, payload fields, counts, sequences, credits, cancel
states and error codes only. The two scenarios (an RPC one and a pub/sub-plus-artifact one)
produce 29 events, identical over both transports; `FLYBUS_TRACE=1` prints them.
### BUS-02 — pub/sub, retention and backpressure
| Bullet | Test |
| --- | --- |
| Overflow rejects a bounded publication before partial fan-out | `tests/pubsub.rs::bounded_fifo_and_atomic_backpressure` (both), `tests/conformance_routing.rs::bounded_overflow_rolls_back_all_artifact_roots` (both) |
| Latest replaces only queued messages | `tests/pubsub.rs::latest_coalesces_only_undelivered_values` (both), `tests/conformance_artifacts.rs::latest_mode_holds_at_most_two_roots_delivered_plus_queued` (both) |
| Delivery consumption returns credits | `tests/pubsub.rs::credits_return_only_on_consume` (both), `tests/conformance_routing.rs::bounded_credit_waits_for_every_extracted_artifact` (both) |
| Unsubscribe preserves already-delivered ownership | `tests/pubsub.rs::unsubscribe_discards_queue_but_not_deliveries` (both), `tests/conformance_routing.rs::unsubscribe_cannot_reach_another_connections_subscription_id` (both) |
| Retained replay is ordered | `tests/pubsub.rs::retained_replay_clear_delete_and_incarnations` (both), `tests/conformance_routing.rs::{latest_replay_is_ordered_ahead_of_a_racing_publish, cleared_topic_gives_no_replay_until_a_fresh_publish}` (both), `tests/sol_review_races.rs::{retained_replay_obeys_bounded_queue_byte_quota, latest_replay_remains_bounded_outside_the_bounded_byte_pool}` |
| Stalled observers cannot starve RPC replies | `tests/pubsub.rs::saturated_subscriber_does_not_block_control` (both), `tests/sol_rereview_regressions.rs::{fair_topic_insertion_preserves_router_envelope_order, sdk_accepts_fair_topic_insertion_through_saturated_control_backlog}` |
All six were already covered, so BUS-02 added no test of its own. The equivalence trace above
carries a pub/sub and artifact scenario, so BUS-02's behaviour is in the transport comparison
too, and `a_latest_subscriber_never_refuses_a_publication` (both) proves the rule that sits
behind "latest replaces only queued messages": the replacement never turns into a refusal.
### BUS-03 — artifact-backed messages and automatic lifetimes
| Bullet | Test |
| --- | --- |
| Last owner collects | `tests/artifacts.rs::fan_out_shares_one_object_and_the_last_consumer_collects` (both), `tests/conformance_artifacts.rs::collection_waits_for_every_retained_owner` (both) |
| Extracted handle survives message drop | `tests/artifacts.rs::extracted_artifacts_outlive_their_message` (both), `tests/conformance_artifacts.rs::{extracted_artifact_outlives_the_message_it_came_from, explicit_retain_outlives_the_original_hold}` (both) |
| Forward before release is safe | `tests/conformance_artifacts.rs::forward_requires_the_source_owner_still_live` (both), `tests/sol_review_races.rs::unsent_oversized_call_rolls_back_its_local_slot`, `tests/integration.rs::session_over_one_router` (both; the coordinator forwards a frame to two agents) |
| Lost replies and cache replay remain valid | **new** `tests/bus_acceptance.rs::a_lost_result_leaks_no_roots_and_the_endpoint_cache_still_replays` (both), with `tests/rpc.rs::endpoint_cache_replays_artifact_results` (both) |
| Disconnect releases logical ownership without mutating still-mapped bytes | **new** `tests/bus_acceptance.rs::disconnect_releases_logical_ownership_without_mutating_open_bytes` (both), with `tests/artifacts.rs::{disconnect_releases_all_but_retained, seal_is_immune_to_live_writable_handles}` (both) |
| Retained latest and queue replacement release the correct roots | `tests/conformance_artifacts.rs::{latest_mode_holds_at_most_two_roots_delivered_plus_queued, retained_topic_value_holds_a_root_independent_of_subscribers, queued_deliveries_hold_roots_before_dispatch}` (both), `tests/conformance_routing.rs::subscriber_disconnect_releases_queued_and_delivered_artifacts_but_not_retention` (both) |
| Measure 640x480 RGBA x 60 with three readers: one stored image, no raw pixels in router messages, bounded CPU/RSS/owners/queues, reader and copy costs recorded | `tests/perf.rs::frames_at_60hz_with_three_consumers`; see the measurement below |
## The crate README's differences from the draft
Each of the ten differences the crate lists, kept with the sentence that allows it or fixed.
| README difference | Verdict |
| --- | --- |
| 1. Extra error codes `CONFLICT`, `NO_TOPIC`, `ARTIFACT_MISMATCH` | Kept. Allowed by section 9: "Transport errors **include** `INVALID_ENVELOPE`, ..." — an inclusive list. Each names a refusal the draft requires without naming its code, so all three are now in the bus-v1 amendment (section 12) |
| 2. Topics must be declared; `topic.clear` of an unknown topic is `NO_TOPIC` while `topic.delete` answers `deleted:false` | Kept. The draft is silent; section 7's "Topic count and retained bytes are capped" and `topic.declare`'s "conflicting settings fail" both presuppose a registry. See the section 7 row |
| 3. When notices are sent | Kept. Section 5 requires the three notices but says nothing about audience or timing: "Required bounded notices are route removal, subscription closure and call failure" |
| 4. Byte budgets: `max_queued_bytes_per_client` counts bounded subscriptions only; `max_retained_bytes` added | Kept, and no longer a difference: the section 9 table now names the bounded pool and bounds latest slots separately (amendment, contradiction 1), and section 7's "Topic count and retained bytes are capped" requires the retained cap |
| 5. Admission sizes a delivery with the router-added ids at their longest, so an inbound envelope near 65,536 bytes can be refused although it fits | Kept, and required: section 5's "Message length includes this wrapper" plus section 4's fixed maximum mean the delivery the router would build must also fit. ipc-v1 section 2's "must not silently truncate a payload" forbids the alternative |
| 6. One live connection per client id; a client id's last incarnation may not be reused; only `*_as` endpoints authenticate the Hello id | Kept. See the section 3 and 4 rows: section 3's per-incarnation callId uniqueness and section 4's launcher-provided privileges |
| 7. The seal reply's ownerId is the writer's own id, now a hold | Kept. Section 8.2: "seal transfers its unique writer" — one owner token, transferred |
| 8. No `budget` argument on calls, and no router executable | Kept. Section 1 calls the executable "optional"; section 6 puts the deadline in the calling client, and the section 2 sketch no longer shows a budget either (amendment, contradiction 2) |
| 9. Wire strictness: unknown fields in management bodies refused, `minor` must be 0 after hello | Kept. Stricter than section 4's "unknown envelope fields", forbidden nowhere, and section 4's envelope literally fixes `minor: 0` |
| 10. `rpc.responder.release` and its terminal `call.failed` | Kept. Section 4 allows draft schema changes ("Changes to these draft schemas change contractDigest"), and it is how section 6's "bounded call correlation metadata" stays bounded when a handler outlives its request delivery. Added to the bus-v1 amendment (section 12) |
## Measured on the dev VM
Two complete release runs of `cargo test --release -p flybus --test perf -- --ignored
--nocapture` on the development VM (4 CPUs), 640x480 RGBA at 60 Hz for 2 s per schedule, three
latest-mode consumers with the third delayed 40 ms per frame, and one, two and four agent
services called every frame. The router runs on its own two-thread Tokio runtime whose threads
carry a distinct name, so its CPU (routing plus the seal copies on its blocking pool) is
measured apart from the clients' four threads in the same process. Each cell is the range over
the two runs.
| Metric | 1 agent | 2 agents | 4 agents |
| --- | --- | --- | --- |
| Frames produced (late) | 120 (0) | 120 (0) | 120 (0 and 3) |
| RPC round trip ms p50/p95/p99 | 0.51-0.60 / 0.84-1.47 / 1.12-2.05 | 0.76-0.78 / 1.34-2.21 / 2.14-6.18 | 1.04-1.08 / 1.67-5.01 / 2.06-10.02 |
| allocate (quota + staging file) ms | 0.96-1.01 / 1.13-1.28 / 1.34-1.49 | 0.98-1.02 / 1.22-3.50 / 1.82-5.61 | 0.80-0.94 / 1.15-5.32 / 1.31-6.60 |
| producer copy into staging ms | 0.37-0.38 / 0.45-0.49 / 0.67-0.75 | 0.37 / 0.45 / 0.51-0.55 | 0.36-0.39 / 0.41-0.49 / 0.48-0.77 |
| seal, router copy to a sealed inode, ms | 1.31-1.35 / 1.65-1.73 / 1.89-1.90 | 1.27-1.42 / 1.69-4.97 / 1.87-7.41 | 1.14-1.30 / 1.64-5.33 / 1.72-7.00 |
| publish admission ms | 0.41-0.42 / 0.61-0.85 / 0.68-1.11 | 0.41-0.43 / 0.83-1.07 / 1.17-3.10 | 0.42-0.49 / 0.73-2.75 / 1.52-4.54 |
| consumer readback of 1.2 MB ms | 0.65-0.75 / 1.19-1.53 / 1.45-1.92 | 0.76-0.91 / 1.45-2.20 / 1.76-6.02 | 0.89-0.99 / 1.67-4.85 / 2.04-6.38 |
| Router CPU (cores, of 2 threads) | 0.175-0.185 | 0.200 | 0.175-0.215 |
| Whole process CPU (cores, of 6 threads) | 0.325-0.335 | 0.410-0.425 | 0.370-0.460 |
| VmRSS / VmHWM | 11.2-13.6 / 13.6-14.4 MB | 11.8-14.1 / 15.0 MB | 15.7-15.8 / 16.4-16.6 MB |
| Store bytes peak / live after drain | 3.7 MB / 0 | 3.7 MB / 0 | 3.7 MB / 0 |
| Outstanding roots peak / live | 5-6 / 0 | 6 / 0 | 5-6 / 0 |
| Queue length peak / live | 1 / 0 | 1 / 0 | 1 / 0 |
| Collection lag after the last frame | 63.5-74.7 ms | 73.6-83.0 ms | 44.1-68.2 ms |
| Frames seen by the three consumers (coalesced) | 120, 120, 49 (0, 0, 70-71) | 120, 120, 49 (0, 0, 71) | 120, 120, 47-49 (0, 0, 70-71) |
What the numbers do and do not say:
- **These are not capacity claims.** Two runs of a synthetic two-second schedule, in one
process, on one shared virtual machine with fewer CPUs than the process has threads, over
temporary local storage, with no capture device, encoder or renderer in the path. They are a
floor on cost, not a ceiling on throughput, and nothing here licenses a sizing decision.
- The second run's tails are three to five times the first run's (RPC p99 10.02 ms against
2.06 ms at four agents, three late frames against none) because other work shared the VM's
four CPUs during it. The p50s barely moved. That spread is the honest width of a measurement
on a shared host, not a property of the bus.
- No byte fan-out: three consumers of a 1.2 MB frame kept the store's peak at 3.7 MB, which is
one sealed object plus the staging and sealing copies of the next frame, not one object per
consumer. Outstanding roots peaked at six.
- Copy costs are real and dominate routing: the producer's copy into staging (p50 0.4 ms), the
router's seal copy into a fresh inode (p50 1.1 to 1.4 ms) and a consumer's readback (p50 0.7
to 1.0 ms) each cost as much as or more than publish admission (p50 0.4 ms).
- The delayed consumer coalesced 70 or 71 of 120 frames and never caused a rejected
publication, which is section 9's rule about latest spectators in one number.
- Every schedule drained completely: live store bytes, roots and queue lengths are zero
afterwards, 44 to 83 ms behind the last frame.
- Router CPU grows with agent count much more slowly than the whole process (0.18 to 0.22
cores against 0.33 to 0.46), because the clients own the copies. A thread that exits between
two samples takes its CPU with it, so the router figure is a floor.
## Contradictions
Two, both inside bus-v1, both minor, neither resolved by changing code. Both were referred to
the coordinator rather than guessed at, and both now carry a dated amendment in bus-v1
section 12; the rows above cite the amended wording. The original reading is kept here because
it is the reason for the amendment.
1. **The per-client queued envelope byte budget versus the latest-mode guarantee.** Section 9's
table has "Per-client ordinary queued envelope bytes | 1 MiB". Section 7 requires that a
latest subscriber always has one replaceable queued value, and section 9 itself says "latest
spectator subscriptions cannot hold a required session transaction indefinitely" — that is, a
latest subscriber must never be the reason a publication is rejected. A byte budget that
covered latest slots and rejected on overflow would violate the second statement; a budget
that excludes them is not the sentence in the table. The crate excludes them, which keeps the
normative sentence and loosens the table row: the worst case becomes subscriptions x 64 KiB
(8 MiB at the default 128 subscriptions per client) instead of 1 MiB. **Resolved in the
spec, not the code** (2026-09-22): the table row now names the bounded pool and latest slots
have their own row, so the implementation conforms as written. No behaviour changed, and
`a_latest_subscriber_never_refuses_a_publication` now proves the guarantee directly.
2. **A call `budget` versus a client-owned deadline.** Section 2's illustrative surface passes a
`budget` into `bus.call(...)`, while section 6 states "A deadline belongs to the calling
client" and gives the router no timeout behaviour, and section 5's `rpc.call` body has no
budget field. The crate follows sections 5 and 6 and has no budget argument, which is safe
because section 2 is labelled illustrative. **Resolved in the spec, not the code**
(2026-09-22): the sketch drops `budget` and shows the deadline at the caller, so the
illustrative surface and the wire contract now agree.
Ambiguities resolved without treating them as contradictions, for the record:
- Section 5's "Reply only by the registered recipient" is read as "by the connection the request
was delivered to", so that section 6's "dispatched calls can still be answered" after
`service.unregister` remains possible. The crate correlates replies by request delivery id on
that connection, not by the live registration.
- Section 9's "Active calls per client 64" is read as calls the caller still awaits: a call
detached by a post-dispatch cancel frees its caller slot while the router keeps the bounded
correlation record until the service consumes or answers it.
## Not implemented, and who owns it
- A standalone router executable (section 1 calls it optional): embed `Router`.
- Bindings in other languages (section 1: a binding "may" exist), and the mapping/FFI pointer
lifetime rules that section 8.3 writes for them.
- A memory storage backend (section 4) and pooled shared memory with generation reuse
(section 8.4), both deferred by the draft.
- Domain-schema enforcement that every referenced artifact is listed in attachments (section 4),
and the domain `RESULT_EXPIRED` outcome (section 6): CONTRACT-01 and `fly-session-rpc`.
- Epoch failure and coherent recovery after a router restart (section 8.4): the session
contract, STATE-01.
- Any consumer at all: no other crate depends on flybus yet, so the feed and control surfaces of
[feed-protocol](../../feed-protocol.md) and [control-api](../../control-api.md) are untouched
and the crate's "Wiring still pending" list still stands.

45
docs/design/session-framework/bus-v1.md
View file

@ -43,7 +43,7 @@ Illustrative Rust surface (not yet implemented):
```rust
let bus = Client::connect(config).await?;
let service = bus.register("agent.fly-a", service_config).await?;
let reply = bus.call(target, "Agent.Prepare", payload, attachments, budget).await?;
let reply = timeout(deadline, bus.call(target, "Agent.Prepare", payload, attachments)).await?;
let subscription = bus.subscribe("session.demo.snapshots", subscription_config).await?;
bus.publish("session.demo.snapshots", payload, attachments).await?;
@ -422,7 +422,8 @@ Configure limits explicitly; these defaults are a prototype starting point, not
| Bounded subscription queued / in-flight deliveries | 64 / 16 |
| Active owners per client | 256 |
| Total artifact storage / per object | 512 MiB / 128 MiB |
| Per-client ordinary queued envelope bytes | 1 MiB |
| Per-client ordinary bounded queued envelope bytes | 1 MiB |
| Latest subscription slots | subscriptions × 64 KiB |
| Reserved management/reply lane | 128 frames and 1 MiB per client |
Reserve an owner allowance for lifecycle/results separately from ordinary telemetry; memory
@ -438,7 +439,8 @@ bounded; rejected callers choose their own retry/fail/pause policy.
Transport errors include `INVALID_ENVELOPE`, `VERSION_MISMATCH`, `NOT_AUTHORIZED`,
`NO_SERVICE`, `TARGET_CHANGED`, `BACKPRESSURE`, `CALL_GONE`, `ARTIFACT_UNSEALED`,
`ARTIFACT_GONE`, `OWNER_INVALID`, `QUOTA_EXCEEDED`, `STORE_FAILURE`, `ROUTER_LOST`.
`ARTIFACT_GONE`, `OWNER_INVALID`, `QUOTA_EXCEEDED`, `STORE_FAILURE`, `ROUTER_LOST`, and the
three of section 12.
Before admission use dispatch:not-dispatched. Once dispatch might have occurred, report
unknown/dispatched conservatively; a caller-side timeout must not imply no mutation.
@ -484,3 +486,40 @@ pipeline may itself exchange large artifacts through this same bus if useful.
The first executable example should show a counter RPC, a pub/sub observer, and a frame
artifact held past message consumption in one small Rust program. No game or browser required.
Distributed simulation ordering remains the [session contract's](step-v1.md) responsibility.
## 12. Amendments
Draft 1 stands as written above. Each amendment below names something the draft requires but
left unnamed, and is dated. The implementation and the sentence-by-sentence audit behind these
entries are in the [conformance report](bus-conformance.md).
**2026-09-22, from the flybus conformance audit.** Three error codes, because section 9's list
is inclusive and these three refusals had no name:
| Code | Reason |
| --- | --- |
| `CONFLICT` | Section 3's duplicate registration, section 5's conflicting topic redeclaration and section 5's `topic.delete` with subscribers are refusals of a live claim, not a missing route, a quota or a bad envelope. |
| `NO_TOPIC` | Publishing to or subscribing to a name nobody declared is a missing topic, and `NO_SERVICE` names the service case only. |
| `ARTIFACT_MISMATCH` | Section 9's "hash mismatch returns a typed artifact error", plus a sealed length that disagrees with the allocation and a reference that disagrees with the artifact it names; `STORE_FAILURE` would blame the store for the caller's claim. |
**2026-09-22, same audit.** One added operation, because section 6 requires bounded call
correlation and gives no way to end it when a handler keeps reply authority after releasing the
request delivery:
| Command | Body / reply value | Semantics |
| --- | --- | --- |
| `rpc.responder.release` | `{callId, requestDeliveryId}` -> `{released}` | The recipient gives up reply authority for a dispatched call. The final release for an attached call retires the correlation and emits `call.failed` with dispatch `dispatched`: `CALL_GONE` while the route is live, `NO_SERVICE` after route loss. Request consumption (section 8.3) stays independent of it. |
Both amendments change `contractDigest`, which section 4 already provides for.
**2026-09-22, coordinator decision on the audit's contradiction 1.** Section 9's table row
"Per-client ordinary queued envelope bytes | 1 MiB" now reads "Per-client ordinary **bounded**
queued envelope bytes", and latest slots get their own row, "subscriptions × 64 KiB", because
section 7's unconditional one-slot guarantee and the structural 1/2 cap outweigh one imprecise
table row: a budget whose overflow rejects a publication cannot contain a subscription that
this same section forbids to reject one.
**2026-09-22, coordinator decision on the audit's contradiction 2.** Section 2's sketch no
longer passes a `budget` into `bus.call` and shows the deadline at the caller instead, because
section 5's wire contract for `rpc.call` has no budget field and section 2 is self-labelled
illustrative.

62
services/flysim/crates/flybus/README.md
View file

@ -10,7 +10,9 @@ ownership; it does not know what the messages mean.
This crate implements the Flybus v1 draft (session-framework design, `bus-v1`, draft 1 of
2026-09-18), using the scalar encodings of its companion `ipc-v1` (`Id`, `U64`, `Digest`).
Where this crate narrows or extends the draft, the difference is listed under
[Differences from the draft](#differences-from-the-draft).
[Differences from the draft](#differences-from-the-draft). Every sentence of the draft's
sections 2 to 11 is audited against this code, with the test that proves it, in
`docs/design/session-framework/bus-conformance.md`.
Nothing in the crate is specific to a game, a brain or a stream. It is a workspace member and
no other crate depends on it yet.
@ -256,11 +258,13 @@ before admission are `not-dispatched`. A command in flight when the connection i
connection rather than drop a release.
- **No waiting on readers.** A client that stops reading stalls only its own writer task. The
router keeps admitting until that client's queues refuse, then returns `BACKPRESSURE` to
publishers of bounded topics. Connection teardown is synchronized with each synchronous
`poll_write`/`poll_flush` call without holding a mutex across an await. Teardown either observes
a complete frame before reclaiming its delivery owner, or marks a partial frame canceled before
cleanup and appends no final notice to the truncated stream. At a frame boundary, normal final
notices are still attempted.
publishers of bounded topics. Connection teardown is ordered against each synchronous
`poll_write`/`poll_flush` call without holding a mutex across an await: it marks the stream
closing once, without waiting, and only then waits for a poll already in progress, so no
later poll reaches the transport however long the writer had been holding it. Teardown either
observes a complete frame before reclaiming its delivery owner, or marks a partial frame
canceled before cleanup and appends no final notice to the truncated stream. At a frame
boundary, normal final notices are still attempted.
- **RPC.** First dispatch is FIFO per service, which includes per caller. A call is marked
dispatched when its bytes are about to be written. Replies correlate by call id and may
arrive in any order. A second reply to one call fails with `CALL_GONE`. A reply to a detached
@ -276,7 +280,8 @@ before admission are `not-dispatched`. A command in flight when the connection i
1. **Extra error codes.** `CONFLICT` covers a duplicate registration, a conflicting topic
redeclaration and deleting a topic that has subscribers. `NO_TOPIC` covers publishing or
subscribing to an undeclared topic. `ARTIFACT_MISMATCH` covers a seal whose length or digest
is wrong, and a reference that disagrees with the artifact it names.
is wrong, and a reference that disagrees with the artifact it names. All three are now
amendments to the draft (bus-v1 section 12), which its inclusive error list allows.
2. **Topics must be declared.** Publish and subscribe fail with `NO_TOPIC` otherwise.
`topic.delete` of an unknown topic returns `deleted:false`. `topic.clear` of an unknown
topic returns `NO_TOPIC`.
@ -286,9 +291,11 @@ before admission are `not-dispatched`. A command in flight when the connection i
- `route.removed` goes to callers with queued or dispatched calls on the removed
registration, not to every client.
- `connection.closing` is an extra notice that precedes every router-initiated close.
4. **Byte budgets.**
4. **Byte budgets.** No longer a difference: the draft's section 9 table was amended on
2026-09-22 to name the bounded pool and to bound latest slots separately.
- `max_queued_bytes_per_client` counts only `bounded` subscriptions. A `latest` slot is bounded
by subscription count times envelope size.
by subscription count times envelope size, because a latest subscriber may never be the
reason a publication is refused.
- `max_retained_bytes` (not in the draft's table) counts the artifact bytes pinned by
retained values, once per topic.
5. **Delivery size.** Admission computes the delivery's size with the router-added ids at their
@ -300,11 +307,14 @@ before admission are `not-dispatched`. A command in flight when the connection i
distinct client id ever seen.
7. **Seal reply.** The reply's `ownerId` is the writer's own id, now an explicit hold.
8. **No `budget` argument on calls, and no router executable.** Timeouts are the caller's
(`tokio::time::timeout` plus `cancel`). The draft's executable is optional; embed `Router`.
(`tokio::time::timeout` plus `cancel`); the draft's section 2 sketch was amended on
2026-09-22 to show the deadline there too. The draft's executable is optional; embed
`Router`.
9. **Wire strictness.** Management bodies reject unknown fields. After hello, envelopes must
carry `minor: 0`.
10. **Reply capability release.** The SDK sends `rpc.responder.release {callId,
requestDeliveryId} -> {released}` when the last local reply capability is dropped. This
requestDeliveryId} -> {released}` when the last local reply capability is dropped, an
operation the draft does not list and now carries as an amendment (bus-v1 section 12). This
keeps request consumption independent from late-reply correlation while bounding that
correlation under the service connection's owner limit. For an attached dispatched call,
final release atomically retires the correlation and caller slot and emits `call.failed`
@ -330,13 +340,16 @@ before admission are `not-dispatched`. A command in flight when the connection i
only when a new router starts on the same root. A directory counts as orphaned when it carries
the store marker and its `flock` is free.
- **Rust only.** There are no other language bindings.
- **Two perf runs.** `tests/perf.rs` (ignored by default) measured 640x480 RGBA frames at
60 Hz over a Unix socket to three latest-mode consumers, one delayed 40 ms per frame. It ran
twice, on a laptop under WSL, release build, router and clients in one process. Allocate,
write and seal took p50 1.2 to 1.3 ms and p99 1.5 to 2.0 ms per 1.2 MB frame. Publish
admission took p50 0.2 ms. The RPC round trip took p50 0.24 / 0.29 to 0.31 / 0.50 ms with
1 / 2 / 4 agents. The whole process used 0.18 to 0.25 cores and about 15 MB RSS. The store
peaked at 3.7 MB. These are two runs, not capacity data.
- **Two perf runs, not capacity data.** `tests/perf.rs` (ignored by default) measures 640x480
RGBA frames at 60 Hz over a Unix socket to three latest-mode consumers, one delayed 40 ms per
frame, with 1, 2 and 4 agent services called every frame. The router gets its own two-thread
runtime with a distinct thread name, so its CPU is separable from the clients' in the same
process. Two release runs on a shared 4-CPU development VM: producer copy into staging p50
0.4 ms, seal copy p50 1.1 to 1.4 ms, consumer readback p50 0.7 to 1.0 ms, publish admission
p50 0.4 ms, RPC round trip p50 0.5 to 1.1 ms; router 0.18 to 0.22 cores, whole process 0.33
to 0.46; 11 to 17 MB RSS; store peak 3.7 MB and zero live after drain. The second run's p99s
were three to five times the first's because other work shared the host. The full table, and
what it does not claim, are in `docs/design/session-framework/bus-conformance.md`.
## Tests
@ -368,3 +381,16 @@ socket, through the same router code:
router restarts.
- `tests/integration.rs`: two agents called in parallel with a forwarded frame, an environment
service, committed snapshot publication, a slow latest consumer and a bounded recorder.
- `tests/bus_acceptance.rs`: the implementation guide's BUS-01/02/03 acceptance bullets that the
suites above do not already prove, one test per bullet - a lost result, a retransmission
fixture, no failover onto a replacement registration, a status RPC answering while another
handler is delayed, and a disconnect that reclaims ownership without touching an open file -
plus `both_transports_produce_equivalent_behaviour_traces`, which replays one RPC scenario
and one pub/sub-and-artifact scenario through the `Trace` recorder in `tests/common/mod.rs`
and requires the two transports to record the same 29 behaviour events. `Trace::record`
panics on a router-issued id, so a trace cannot drift into operational detail.
`FLYBUS_TRACE=1` prints it.
It also holds the two rules an audit reviewer found cited but unproven: a latest subscriber
flooded with 100 publications of 60 KB never refuses one, and a topic named exactly like a
router notice is still delivered as topic data.
- `tests/example_demo.rs`: runs `examples/demo.rs` and asserts every line it prints.

103
services/flysim/crates/flybus/examples/demo.rs
View file

@ -1,24 +1,38 @@
//! A counter RPC, a pub/sub observer and a frame artifact held past its message, in one
//! process over the in-memory transport (bus-v1 section 11).
//! The guide's first deliverable: a counter RPC, a pub/sub observer and a frame artifact held
//! past its message object's lifetime, in one program (bus-v1 section 11, implementation
//! guide section 1). No game, browser or second transport is involved.
//!
//! ```text
//! cargo run -p flybus --example demo
//! ```
//!
//! `tests/example_demo.rs` runs [`run`] and asserts every line it returns.
use std::io::Write;
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::{Duration, Instant};
use flybus::{
Client, ClientConfig, Policy, Retained, Router, RouterConfig, ServiceConfig, SubscriptionConfig,
Client, ClientConfig, Policy, Retained, Router, RouterConfig, ServiceConfig,
SubscriptionConfig,
};
use serde_json::{Map, Value, json};
const W: usize = 160;
const H: usize = 144;
fn obj(v: Value) -> Map<String, Value> {
v.as_object().cloned().unwrap_or_default()
}
#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let root = std::env::temp_dir().join(format!("flybus-demo-{}", std::process::id()));
/// The three parts, in one program, over one router. Returns the lines the example prints.
pub async fn run() -> Result<Vec<String>, Box<dyn std::error::Error>> {
static RUNS: AtomicU64 = AtomicU64::new(0);
let root = std::env::temp_dir().join(format!(
"flybus-demo-{}-{}",
std::process::id(),
RUNS.fetch_add(1, Ordering::Relaxed)
));
let mut config = RouterConfig::new(&root);
config.policy = Policy::open();
let router = Router::new(config)?;
@ -28,14 +42,17 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
ClientConfig::new(id, &root),
)
};
let mut lines = Vec::new();
// A counter service.
// 1. A counter service. An exclusive endpoint, pinned by its caller to the registration
// it discovered, reached through the router like every other operation.
let counter = connect("counter").await?;
let mut svc = counter
.register("example.counter", ServiceConfig::default())
.await?;
tokio::spawn(async move {
let mut total = 0;
let incarnation = svc.incarnation().to_owned();
let service = tokio::spawn(async move {
let mut total = 0i64;
while let Some(req) = svc.next().await {
total += req.payload()["amount"].as_i64().unwrap_or(0);
let _ = req.reply(obj(json!({ "total": total })), &[]).await;
@ -46,54 +63,86 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
let res = app
.call_and_wait(
"example.counter",
None,
Some(&incarnation),
"Counter.Increment",
obj(json!({"amount": 2})),
obj(json!({"amount": 1})),
&[],
)
.await?;
println!("counter total = {}", res.outcome()["total"]);
lines.push(format!("counter total = {}", res.outcome()["total"]));
}
// An observer of a frame topic.
app.declare_topic("world.demo.frame", Retained::None)
.await?;
// 2. A pub/sub observer. A latest-value subscription, so a slow observer coalesces
// instead of holding the producer up.
app.declare_topic("world.demo.frame", Retained::None).await?;
let observer = connect("observer").await?;
let mut frames = observer
.subscribe("world.demo.frame", SubscriptionConfig::latest())
.await?;
// 3. A frame artifact. The bytes live in the store; the message carries a reference and
// the dimensions.
let mut writer = app
.artifacts()
.allocate(160 * 144 * 4, "image/x-rgba")
.allocate((W * H * 4) as u64, "image/x-rgba")
.await?;
writer.write_all(&vec![0x7f; 160 * 144 * 4])?;
writer.write_all(&vec![0x7f; W * H * 4])?;
let frame = writer.seal().await?;
let receipt = app
.publish(
"world.demo.frame",
obj(json!({"width": 160, "height": 144})),
obj(json!({"width": W, "height": H})),
&[("frame", &frame)],
)
.await?;
println!(
lines.push(format!(
"published sequence {} to {} subscriber(s)",
receipt.topic_sequence, receipt.subscribers
);
));
// The producer lets go of its own hold; the delivery keeps the bytes alive.
drop(frame);
let message = frames.next().await.ok_or("subscription closed")?;
let message = frames.next().await.ok_or("the subscription closed")?;
let image = message.artifact("frame")?;
drop(message); // the extracted handle still owns the delivery
let bytes = image.read_all().await?;
println!(
"read {} bytes after the message was dropped; router: {:?}",
bytes.len(),
router.stats()
);
lines.push(format!(
"read {} bytes after the message was dropped",
bytes.len()
));
let held = router.stats();
lines.push(format!(
"while the frame is held: {} artifact(s), {} root(s)",
held.sealed_artifacts, held.artifact_roots
));
drop(image); // the last handle: the delivery is consumed and the frame collected
// Consumption reaches the router on the client's control lane, so collection is not
// instantaneous.
let deadline = Instant::now() + Duration::from_secs(10);
while router.stats().artifacts > 0 {
if Instant::now() > deadline {
return Err("the frame was never collected".into());
}
tokio::time::sleep(Duration::from_millis(1)).await;
}
let collected = router.stats();
lines.push(format!(
"after the last handle: {} artifact(s), {} root(s)",
collected.artifacts, collected.artifact_roots
));
service.abort();
router.shutdown();
std::fs::remove_dir_all(&root)?;
drop((app, observer, counter));
let _ = std::fs::remove_dir_all(&root);
Ok(lines)
}
#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
for line in run().await? {
println!("{line}");
}
Ok(())
}

71
services/flysim/crates/flybus/src/router/mod.rs
View file

@ -364,75 +364,58 @@ async fn write_selected<W: AsyncWrite + Unpin>(
let mut frame = Vec::with_capacity(bytes.len() + 4);
frame.extend_from_slice(&(bytes.len() as u32).to_le_bytes());
frame.extend_from_slice(bytes);
{
let mut gate = signals.write_gate.lock().unwrap_or_else(|e| e.into_inner());
if !gate.begin_frame(frame.len()) {
return SelectedWrite::Closing {
partial: gate.cut_partial(),
let closing = || SelectedWrite::Closing {
partial: signals.write_gate.cut_partial(),
};
}
let interrupted = || io::Error::new(io::ErrorKind::Interrupted, "connection closing");
if !signals.write_gate.begin_frame(frame.len()) {
return closing();
}
let mut written = 0;
while written < frame.len() {
let polled = std::future::poll_fn(|cx| {
let mut gate = signals.write_gate.lock().unwrap_or_else(|e| e.into_inner());
if gate.closing() {
return std::task::Poll::Ready(Err(io::Error::new(
io::ErrorKind::Interrupted,
"connection closing",
)));
if !signals.write_gate.enter_poll() {
return std::task::Poll::Ready(Err(interrupted()));
}
match std::pin::Pin::new(&mut *wr).poll_write(cx, &frame[written..]) {
let polled = std::pin::Pin::new(&mut *wr).poll_write(cx, &frame[written..]);
let wrote = match polled {
std::task::Poll::Ready(Ok(n)) => n,
_ => 0,
};
signals.write_gate.leave_poll(wrote);
match polled {
std::task::Poll::Ready(Ok(0)) => std::task::Poll::Ready(Err(io::Error::new(
io::ErrorKind::WriteZero,
"failed to write router frame",
))),
std::task::Poll::Ready(Ok(n)) => {
gate.wrote(n);
std::task::Poll::Ready(Ok(n))
}
other => other,
}
})
.await;
match polled {
Ok(n) => written += n,
Err(e) if e.kind() == io::ErrorKind::Interrupted => {
let gate = signals.write_gate.lock().unwrap_or_else(|e| e.into_inner());
return SelectedWrite::Closing {
partial: gate.cut_partial(),
};
}
Err(e) if e.kind() == io::ErrorKind::Interrupted => return closing(),
Err(_) => return SelectedWrite::Failed,
}
}
let flushed = std::future::poll_fn(|cx| {
let gate = signals.write_gate.lock().unwrap_or_else(|e| e.into_inner());
if gate.closing() {
return std::task::Poll::Ready(Err(io::Error::new(
io::ErrorKind::Interrupted,
"connection closing",
)));
if !signals.write_gate.enter_poll() {
return std::task::Poll::Ready(Err(interrupted()));
}
std::pin::Pin::new(&mut *wr).poll_flush(cx)
let polled = std::pin::Pin::new(&mut *wr).poll_flush(cx);
signals.write_gate.leave_poll(0);
polled
})
.await;
if let Err(e) = flushed {
if e.kind() == io::ErrorKind::Interrupted {
let gate = signals.write_gate.lock().unwrap_or_else(|e| e.into_inner());
return SelectedWrite::Closing {
partial: gate.cut_partial(),
};
return closing();
}
return SelectedWrite::Failed;
}
signals
.write_gate
.lock()
.unwrap_or_else(|e| e.into_inner())
.finish_frame();
signals.write_gate.finish_frame();
SelectedWrite::Complete
}
@ -452,13 +435,9 @@ async fn write_loop<W: AsyncWrite + Unpin>(
tokio::pin!(write);
let selected = tokio::select! {
result = &mut write => result,
_ = stopped(&mut shutdown) => {
let gate = signals
.write_gate
.lock()
.unwrap_or_else(|e| e.into_inner());
SelectedWrite::Closing { partial: gate.cut_partial() }
}
_ = stopped(&mut shutdown) => SelectedWrite::Closing {
partial: signals.write_gate.cut_partial(),
},
};
match selected {
SelectedWrite::Complete => {}

99
services/flysim/crates/flybus/src/router/state.rs
View file

@ -42,56 +42,98 @@ pub(crate) struct ConnSignals {
pub wake: Notify,
/// Flips to true when the connection is closed; both tasks watch it.
pub shutdown: watch::Sender<bool>,
/// Serializes synchronous transport polls with connection teardown. It is never held
/// across an await.
pub write_gate: std::sync::Mutex<WriteGate>,
/// Orders connection teardown against the synchronous transport polls of the writer.
pub write_gate: WriteGate,
/// The last frames to write before closing: a refusal or `connection.closing` notice,
/// preceded on router shutdown by `subscription.closed` notices.
pub final_frames: std::sync::Mutex<Vec<Vec<u8>>>,
}
/// Orders teardown against the writer's synchronous transport polls.
///
/// Teardown marks the stream closing *before* it waits for a poll already in progress, so at
/// most that one poll can still write and every later one is refused, whichever task reaches
/// the lock first. The earlier design held one mutex across each poll instead, which a writer
/// sending a frame a byte per poll re-acquired hundreds of times while teardown waited for it:
/// teardown could be starved for a whole frame and the frame completed just before its
/// delivery owner was reclaimed. The lock is held only for these bookkeeping steps, never
/// across an await.
#[derive(Default)]
pub(crate) struct WriteGate {
state: std::sync::Mutex<GateState>,
/// Signalled when a poll leaves the transport.
idle: std::sync::Condvar,
}
#[derive(Default)]
struct GateState {
closing: bool,
/// A writer is inside a synchronous transport poll right now.
polling: bool,
frame_len: usize,
written: usize,
cut_partial: bool,
}
impl WriteGate {
pub(crate) fn begin_frame(&mut self, len: usize) -> bool {
if self.closing {
fn lock(&self) -> std::sync::MutexGuard<'_, GateState> {
self.state.lock().unwrap_or_else(|e| e.into_inner())
}
/// Starts one frame. False once teardown has begun.
pub(crate) fn begin_frame(&self, len: usize) -> bool {
let mut g = self.lock();
if g.closing {
return false;
}
self.frame_len = len;
self.written = 0;
g.frame_len = len;
g.written = 0;
true
}
pub(crate) fn wrote(&mut self, len: usize) {
self.written += len;
debug_assert!(self.written <= self.frame_len);
/// Claims the transport for one synchronous poll. False once teardown has begun.
pub(crate) fn enter_poll(&self) -> bool {
let mut g = self.lock();
if g.closing {
return false;
}
debug_assert!(!g.polling, "one writer task polls one connection");
g.polling = true;
true
}
pub(crate) fn finish_frame(&mut self) {
if !self.closing {
debug_assert_eq!(self.written, self.frame_len);
self.frame_len = 0;
self.written = 0;
/// Releases the transport, accounting for what that poll wrote.
pub(crate) fn leave_poll(&self, wrote: usize) {
let mut g = self.lock();
g.polling = false;
g.written += wrote;
debug_assert!(g.written <= g.frame_len);
drop(g);
self.idle.notify_all();
}
pub(crate) fn finish_frame(&self) {
let mut g = self.lock();
if !g.closing {
debug_assert_eq!(g.written, g.frame_len);
g.frame_len = 0;
g.written = 0;
}
}
fn begin_close(&mut self) {
self.closing = true;
self.cut_partial = self.written > 0 && self.written < self.frame_len;
/// Refuses every later poll, then waits for one already in progress and records whether it
/// left a frame half written. The caller may reclaim owners once this returns.
fn begin_close(&self) {
let mut g = self.lock();
g.closing = true;
while g.polling {
g = self.idle.wait(g).unwrap_or_else(|e| e.into_inner());
}
pub(crate) fn closing(&self) -> bool {
self.closing
g.cut_partial = g.written > 0 && g.written < g.frame_len;
}
pub(crate) fn cut_partial(&self) -> bool {
self.cut_partial
self.lock().cut_partial
}
}
@ -100,7 +142,7 @@ impl ConnSignals {
ConnSignals {
wake: Notify::new(),
shutdown: watch::Sender::new(false),
write_gate: std::sync::Mutex::new(WriteGate::default()),
write_gate: WriteGate::default(),
final_frames: std::sync::Mutex::new(Vec::new()),
}
}
@ -705,15 +747,14 @@ impl State {
.lock()
.unwrap_or_else(|e| e.into_inner())
.extend(final_frames);
// A transport poll that is already in progress finishes before this lock is acquired.
// Once acquired, teardown marks the stream closing before reclaiming any owner, and no
// later normal-frame poll is allowed through.
let mut write_gate = signals.write_gate.lock().unwrap_or_else(|e| e.into_inner());
write_gate.begin_close();
// Teardown refuses every later transport poll first, then waits for a poll already in
// progress, and only then reclaims what the connection owned. So a delivery frame is
// either complete before its owner is reclaimed, or left truncated with nothing more
// appended to the stream; the writer can never finish it afterwards.
signals.write_gate.begin_close();
self.disconnect(c);
signals.shutdown.send_replace(true);
signals.wake.notify_one();
drop(write_gate);
self.flush_notices();
}

779
services/flysim/crates/flybus/tests/bus_acceptance.rs Normal file
View file

@ -0,0 +1,779 @@
//! The BUS-01, BUS-02 and BUS-03 acceptance bullets of the implementation guide that the
//! other suites do not already prove, one test per bullet, named after the bullet, plus the
//! transport-equivalence traces.
//!
//! `docs/design/session-framework/bus-conformance.md` maps every bullet of all three lists to
//! the test that proves it; the bullets already covered elsewhere are cited there instead of
//! being repeated here.
mod common;
use std::collections::HashMap;
use std::io::{Read, Write};
use std::time::Duration;
use common::{Env, Trace, Via, code, env, obj, quiet, sealed, within};
use flybus::wire::Kind;
use flybus::{
CancelState, Client, Dispatch, ErrorCode, Retained, Service, ServiceConfig, SubscriptionConfig,
};
use serde_json::json;
/// A service that executes once per domain `requestId`, keeps its result artifact on an
/// explicit hold of its own and answers a repeat from that cache (bus-v1 section 6).
fn spawn_cache_service(client: Client, mut svc: Service) -> tokio::task::JoinHandle<()> {
tokio::spawn(async move {
let mut cache: HashMap<String, flybus::Artifact> = HashMap::new();
let mut executions = 0u64;
while let Some(req) = svc.next().await {
let rid = req.payload()["requestId"]
.as_str()
.unwrap_or_default()
.to_owned();
if !cache.contains_key(&rid) {
executions += 1;
let bytes = format!("state of {rid}").into_bytes();
let mut w = client
.artifacts()
.allocate(bytes.len() as u64, "text/plain")
.await
.unwrap();
w.write_all(&bytes).unwrap();
cache.insert(rid.clone(), w.seal().await.unwrap());
}
let art = cache[&rid].clone();
let _ = req
.reply(
obj(json!({"requestId": rid, "executions": executions})),
&[("state", &art)],
)
.await;
}
})
}
/// One execution per domain `requestId`: the endpoint owns the result artifact and replays it
/// from its own hold. Returns whether the reply was routed to a still-attached caller.
async fn serve_cached(
server: &Client,
req: &flybus::Request,
cache: &mut HashMap<String, flybus::Artifact>,
executions: &mut u64,
) -> bool {
let rid = req.payload()["requestId"].as_str().unwrap().to_owned();
if !cache.contains_key(&rid) {
*executions += 1;
let bytes = format!("state of {rid}").into_bytes();
cache.insert(rid.clone(), sealed(server, &bytes, "text/plain").await);
}
let art = cache[&rid].clone();
req.reply(obj(json!({"executions": *executions})), &[("state", &art)])
.await
.unwrap()
}
// ---------------------------------------------------------------------------------------------
// BUS-01
/// BUS-01: "lost result", and BUS-03: "lost replies and cache replay remain valid".
///
/// The caller never reads its admitted result and then loses its connection. The router keeps
/// no result cache of its own, leaks no root, and the endpoint's own hold still replays the
/// same bytes for a repeat of the domain request.
async fn a_lost_result_leaks_no_roots_and_the_endpoint_cache_still_replays(via: Via) {
let e = env(via).await;
let server = e.client("server").await;
let svc = server
.register("agent.lossy", ServiceConfig::default())
.await
.unwrap();
let handler = spawn_cache_service(server.clone(), svc);
// A caller that admits a call and never reads the result delivery.
let mut raw = e.raw_hello("caller").await;
let accepted = raw
.call(
"rpc.call",
json!({
"callId": "call-1", "target": "agent.lossy", "expectedIncarnation": null,
"method": "Agent.Prepare", "payload": {"requestId": "req-41"}
}),
)
.await
.unwrap();
assert_eq!(accepted["accepted"], json!(true));
// Two roots: the endpoint's cache hold and the caller's result.
e.settle("result admitted", |s| {
s.sealed_artifacts == 1 && s.artifact_roots == 2
})
.await;
drop(raw);
e.settle("the lost result leaks nothing", |s| {
s.calls == 0 && s.artifact_roots == 1 && s.owners == 1 && s.sealed_artifacts == 1
})
.await;
// The domain retry returns the cached artifact, still readable.
let caller = e.client("retry").await;
let res = within(
"cache replay",
caller.call_and_wait(
"agent.lossy",
None,
"Agent.Prepare",
obj(json!({"requestId": "req-41"})),
&[],
),
)
.await
.unwrap();
assert_eq!(
res.outcome()["executions"], 1,
"the lost result was recomputed"
);
let art = res.artifact("state").unwrap();
assert_eq!(art.read_all().await.unwrap(), b"state of req-41");
drop((art, res));
handler.abort();
}
/// BUS-01: "retransmission fixtures". The same domain request body is sent twice under two
/// bus call ids, pinned to one service incarnation; the endpoint executes once (bus-v1
/// section 6, ipc-v1 section 3).
async fn a_retransmission_repeats_the_domain_request_under_a_fresh_call_id(via: Via) {
let e = env(via).await;
let server = e.client("server").await;
let caller = e.client("caller").await;
let mut svc = server
.register("agent.retried", ServiceConfig::default())
.await
.unwrap();
let incarnation = svc.incarnation().to_owned();
// The fixture: one domain body, sent twice, byte for byte.
let body = obj(json!({"requestId": "req-41", "params": {"step": "41"}}));
let first = caller
.call(
"agent.retried",
Some(&incarnation),
"Agent.Prepare",
body.clone(),
&[],
)
.await
.unwrap();
let first_call_id = first.call_id().to_owned();
let attempt = within("first attempt", svc.next()).await.unwrap();
assert_eq!(attempt.payload(), &body);
// The caller gives up. Cancelling after dispatch cannot undo the work.
assert_eq!(first.cancel().await.unwrap(), CancelState::ExecutionUnknown);
// The endpoint finishes anyway and caches the result; the reply reaches nobody.
let mut executions = 0u64;
let mut cache: HashMap<String, flybus::Artifact> = HashMap::new();
assert_eq!(attempt.call_id(), first_call_id);
let routed = serve_cached(&server, &attempt, &mut cache, &mut executions).await;
assert!(!routed, "the detached caller was still reachable");
drop(attempt);
// The retry: a new bus call id, the original domain body, the same pinned incarnation.
let mut second = caller
.call(
"agent.retried",
Some(&incarnation),
"Agent.Prepare",
body.clone(),
&[],
)
.await
.unwrap();
assert_ne!(
second.call_id(),
first_call_id,
"a safe retry uses a fresh bus call id"
);
assert_eq!(second.service_incarnation(), incarnation);
let repeat = within("retry", svc.next()).await.unwrap();
assert_eq!(repeat.payload(), &body, "the domain body changed");
assert_ne!(repeat.call_id(), first_call_id);
assert!(serve_cached(&server, &repeat, &mut cache, &mut executions).await);
drop(repeat);
let res = within("retry result", second.result()).await.unwrap();
assert_eq!(res.outcome()["executions"], 1, "the retry re-executed");
assert_eq!(
res.artifact("state").unwrap().read_all().await.unwrap(),
b"state of req-41"
);
assert_eq!(executions, 1);
drop(res);
drop(cache);
}
/// BUS-01: "no automatic retry/failover". Neither a queued nor a dispatched call is replayed
/// onto a replacement registration, and an old pinned incarnation fails rather than reaching
/// the new holder (bus-v1 sections 3 and 6).
async fn no_automatic_retry_or_failover_onto_a_replacement_registration(via: Via) {
let e = env(via).await;
let first_host = e.client("first").await;
let second_host = e.client("second").await;
let caller = e.client("caller").await;
let mut svc = first_host
.register(
"agent.fly-a",
ServiceConfig {
max_queued: 4,
max_in_flight: 1,
},
)
.await
.unwrap();
let old = svc.incarnation().to_owned();
let mut dispatched = caller
.call("agent.fly-a", Some(&old), "Agent.Prepare", obj(json!({"n": 1})), &[])
.await
.unwrap();
let held = within("request", svc.next()).await.unwrap();
let mut queued = caller
.call("agent.fly-a", Some(&old), "Agent.Prepare", obj(json!({"n": 2})), &[])
.await
.unwrap();
// The worker goes away with one call dispatched and one still queued. Unregister first,
// while the request credit is still held, so the queued call cannot be dispatched.
drop(svc);
let q = within("queued call", queued.result()).await.unwrap_err();
assert_eq!(
(q.code, q.dispatch),
(ErrorCode::NoService, Dispatch::NotDispatched)
);
drop(held);
first_host.close().await;
let d = within("dispatched call", dispatched.result())
.await
.unwrap_err();
assert_eq!(d.dispatch, Dispatch::Dispatched, "{d}");
assert!(
matches!(d.code, ErrorCode::NoService | ErrorCode::CallGone),
"{d}"
);
// A restarted worker takes the name. Nothing is replayed onto it.
let mut replacement = second_host
.register("agent.fly-a", ServiceConfig::default())
.await
.unwrap();
assert_ne!(replacement.incarnation(), old);
quiet("a retry onto the replacement", replacement.next()).await;
let pinned = caller
.call("agent.fly-a", Some(&old), "Agent.Prepare", obj(json!({"n": 3})), &[])
.await
.unwrap_err();
assert_eq!(pinned.code, ErrorCode::TargetChanged);
assert_eq!(pinned.dispatch, Dispatch::NotDispatched);
// Only the caller's own fresh call reaches the new incarnation.
let mut fresh = caller
.call(
"agent.fly-a",
Some(replacement.incarnation()),
"Agent.Prepare",
obj(json!({"n": 1})),
&[],
)
.await
.unwrap();
let req = within("fresh request", replacement.next()).await.unwrap();
assert_eq!(req.payload()["n"], 1);
assert!(req.reply(obj(json!({"ok": true})), &[]).await.unwrap());
assert_eq!(
within("fresh result", fresh.result())
.await
.unwrap()
.outcome()["ok"],
true
);
}
/// BUS-01: "status RPC can respond while another handler is delayed". One service, two calls:
/// the dispatcher answers the status call concurrently with an open mutation, and the mutation
/// completes out of order afterwards (bus-v1 section 6).
async fn a_status_rpc_responds_while_another_handler_is_delayed(via: Via) {
let e = env(via).await;
let server = e.client("server").await;
let caller = e.client("caller").await;
let mut svc = server
.register(
"agent.fly-a",
ServiceConfig {
max_queued: 4,
max_in_flight: 4,
},
)
.await
.unwrap();
let mut advance = caller
.call(
"agent.fly-a",
None,
"Environment.Advance",
obj(json!({"step": "41"})),
&[],
)
.await
.unwrap();
let delayed = within("advance request", svc.next()).await.unwrap();
let mut status = caller
.call("agent.fly-a", None, "Worker.Status", obj(json!({})), &[])
.await
.unwrap();
let status_req = within("status request", svc.next()).await.unwrap();
assert_eq!(status_req.method(), "Worker.Status");
assert!(
status_req
.reply(obj(json!({"phase": "advancing"})), &[])
.await
.unwrap()
);
let answered = within("status result", status.result()).await.unwrap();
assert_eq!(answered.outcome()["phase"], "advancing");
drop((answered, status_req));
assert!(
tokio::time::timeout(Duration::from_millis(150), advance.result())
.await
.is_err(),
"the delayed handler answered early"
);
assert!(
delayed
.reply(obj(json!({"step": "41"})), &[])
.await
.unwrap()
);
let done = within("advance result", advance.result()).await.unwrap();
assert_eq!(done.outcome()["step"], "41");
drop((done, delayed));
e.settle("calls retired", |s| s.calls == 0 && s.owners == 0)
.await;
}
// ---------------------------------------------------------------------------------------------
// Section 9: the two rules an audit reviewer found cited but unproven
/// bus-v1 section 9: "Bounded event subscriptions can reject publication; latest spectator
/// subscriptions cannot hold a required session transaction indefinitely."
///
/// A latest subscriber that never consumes must never be the reason a publication is refused,
/// however many envelope bytes its slot would have accumulated: the slot sits outside the
/// per-client bounded-queue byte pool. 100 publications of 60 KB are six times that pool.
async fn a_latest_subscriber_never_refuses_a_publication(via: Via) {
let e = env(via).await;
let publisher = e.client("publisher").await;
let spectator = e.client("spectator").await;
publisher
.declare_topic("world.demo.frame", Retained::None)
.await
.unwrap();
// One credit, and nothing ever consumes it: after the first delivery every later
// publication meets the single replaceable slot.
let _stuck = spectator
.subscribe(
"world.demo.frame",
SubscriptionConfig::latest().in_flight(1),
)
.await
.unwrap();
e.settle("subscribed", |s| s.subscriptions == 1).await;
let blob = "s".repeat(60_000);
let publish = async |n: u64| {
within(
"publication",
publisher.publish("world.demo.frame", obj(json!({"n": n, "blob": blob})), &[]),
)
.await
.unwrap_or_else(|err| panic!("publication {n} was refused with {err}"))
};
publish(0).await;
e.settle("the only credit is in use", |s| s.owners == 1).await;
let mut replaced_total = 0;
for n in 1..100u64 {
let receipt = publish(n).await;
assert_eq!(receipt.subscribers, 1);
replaced_total += receipt.replaced;
}
// The first of those found an empty slot; the other 98 replaced an undelivered value.
assert_eq!(replaced_total, 98);
let stats = e.stats();
assert_eq!(stats.queued, 1, "the slot never grew: {stats:?}");
assert_eq!(stats.owners, 1, "no credit came back: {stats:?}");
}
/// bus-v1 section 9: "classification is an explicit generic envelope operation/policy, not a
/// topic-name heuristic", and section 3: "The router treats names as opaque addresses."
///
/// A topic whose name is spelled exactly like a router notice is still declared, routed and
/// delivered as topic data: the delivery is a `topic.message`, not the notice it is named
/// after, and its payload arrives untouched.
async fn a_topic_named_like_a_notice_is_still_classified_as_topic_data(via: Via) {
let e = env(via).await;
let publisher = e.client("publisher").await;
let mut raw = e.raw_hello("watcher").await;
let names = ["call.failed", "route.removed", "subscription.closed"];
for name in names {
publisher.declare_topic(name, Retained::None).await.unwrap();
let reply = raw
.call(
"subscribe",
json!({"topic": name, "mode": "bounded", "maxQueued": 4, "maxInFlight": 4, "replayLatest": false}),
)
.await;
assert_eq!(code(&reply), "OK", "{name} could not be subscribed to");
}
for name in names {
publisher
.publish(name, obj(json!({"named": name})), &[])
.await
.unwrap();
let envelope = raw.event().await;
assert_eq!(
(envelope.kind, envelope.op.as_str()),
(Kind::Delivery, "topic.message"),
"the topic name {name} changed how the router classified it"
);
assert_eq!(envelope.body["topic"], json!(name));
assert_eq!(envelope.body["payload"]["named"], json!(name));
}
}
// ---------------------------------------------------------------------------------------------
// BUS-03
/// BUS-03: "disconnect releases logical ownership without mutating still-mapped bytes"
/// (bus-v1 section 8.4). The consumer's connection ends while it still has the sealed file
/// open; the router reclaims every logical root and unlinks the file, and the open handle
/// still reads the original bytes.
async fn disconnect_releases_logical_ownership_without_mutating_open_bytes(via: Via) {
let e = env(via).await;
let producer = e.client("producer").await;
let consumer = e.client("consumer").await;
producer
.declare_topic("world.demo.frame", Retained::None)
.await
.unwrap();
let mut sub = consumer
.subscribe("world.demo.frame", SubscriptionConfig::latest())
.await
.unwrap();
let pixels: Vec<u8> = (0..4096u32).map(|i| (i % 251) as u8).collect();
let frame = sealed(&producer, &pixels, "image/x-rgba").await;
producer
.publish(
"world.demo.frame",
obj(json!({"n": 1})),
&[("frame", &frame)],
)
.await
.unwrap();
drop(frame);
let msg = within("frame", sub.next()).await.unwrap();
let image = msg.artifact("frame").unwrap();
drop(msg);
let mut file = image.open().await.unwrap();
let mut head = vec![0u8; 16];
file.read_exact(&mut head).unwrap();
assert_eq!(head, pixels[..16]);
assert_eq!(e.files("sealed"), 1);
// The connection ends with the file still open.
drop((sub, image));
consumer.close().await;
e.settle("logical ownership released", |s| {
s.owners == 0 && s.artifacts == 0 && s.store_bytes == 0
})
.await;
e.settle_files("sealed", 0).await;
// Reclaiming the registry entry did not touch the inode.
let mut rest = Vec::new();
file.read_to_end(&mut rest).unwrap();
assert_eq!(rest, pixels[16..]);
assert_eq!(file.len(), pixels.len() as u64);
}
// ---------------------------------------------------------------------------------------------
// BUS-01: both transports produce equivalent behaviour traces
/// An RPC scenario: registration, admission, FIFO dispatch, service backpressure, cancel
/// before dispatch, reply and result, and retirement.
async fn rpc_trace(e: &Env, t: &Trace) {
let server = e.client("trace-server").await;
let caller = e.client("trace-caller").await;
let mut svc = server
.register(
"agent.traced",
ServiceConfig {
max_queued: 1,
max_in_flight: 1,
},
)
.await
.unwrap();
t.record("service registered");
let mut first = caller
.call(
"agent.traced",
Some(svc.incarnation()),
"Agent.Prepare",
obj(json!({"n": 1})),
&[],
)
.await
.unwrap();
t.record("call admitted n=1");
let req = within("traced request", svc.next()).await.unwrap();
t.record(format!(
"request method={} n={} caller={}",
req.method(),
req.payload()["n"],
req.caller().client_id
));
let mut queued = caller
.call(
"agent.traced",
Some(svc.incarnation()),
"Agent.Prepare",
obj(json!({"n": 2})),
&[],
)
.await
.unwrap();
t.record("call admitted n=2");
let refused = caller
.call(
"agent.traced",
Some(svc.incarnation()),
"Agent.Prepare",
obj(json!({"n": 3})),
&[],
)
.await
.unwrap_err();
t.record(format!(
"call refused {} {}",
refused.code,
refused.dispatch.as_str()
));
t.record(format!("cancel state={:?}", queued.cancel().await.unwrap()));
let gone = within("cancelled result", queued.result()).await.unwrap_err();
t.record(format!(
"cancelled result {} {}",
gone.code,
gone.dispatch.as_str()
));
t.record(format!(
"reply routed={}",
req.reply(obj(json!({"prepared": 1})), &[]).await.unwrap()
));
let res = within("traced result", first.result()).await.unwrap();
t.record(format!(
"result prepared={} responder={}",
res.outcome()["prepared"],
res.responder().client_id
));
drop((res, req));
let s = e
.settle("traced calls retired", |s| s.calls == 0 && s.owners == 0)
.await;
t.record(format!(
"retired calls={} active={} owners={}",
s.calls, s.active_calls, s.owners
));
}
/// A pub/sub and artifact scenario: retained declaration, a bounded and a latest subscriber,
/// a late replaying subscriber, latest replacement of an undelivered value, an extracted
/// artifact outliving its message, an explicit hold, clear, delete and collection.
async fn pubsub_artifact_trace(e: &Env, t: &Trace) {
let topic = "session.demo.snapshots";
let producer = e.client("trace-producer").await;
let reader = e.client("trace-reader").await;
let spectator = e.client("trace-spectator").await;
let latecomer = e.client("trace-latecomer").await;
let declared = producer.declare_topic(topic, Retained::Latest).await.unwrap();
t.record(format!("topic declared={}", declared.declared));
let mut bounded = reader
.subscribe(topic, SubscriptionConfig::bounded().queued(4).in_flight(4))
.await
.unwrap();
// One credit only: while its message is held, the next publication queues and the one
// after that replaces it.
let mut latest = spectator
.subscribe(topic, SubscriptionConfig::latest().in_flight(1))
.await
.unwrap();
t.record("two subscriptions");
let first = sealed(&producer, b"snapshot-1", "application/octet-stream").await;
let r1 = producer
.publish(topic, obj(json!({"step": "1"})), &[("state", &first)])
.await
.unwrap();
t.record(format!(
"publish seq={} subscribers={} replaced={}",
r1.topic_sequence, r1.subscribers, r1.replaced
));
drop(first);
let m1 = within("bounded 1", bounded.next()).await.unwrap();
t.record(format!(
"bounded seq={} replaced={} step={} attachments=[{}]",
m1.topic_sequence(),
m1.replaced(),
m1.payload()["step"],
m1.attachment_names().collect::<Vec<_>>().join(",")
));
let state = m1.artifact("state").unwrap();
drop(m1);
// The extracted handle keeps the delivery alive past the message object.
let bytes = state.read_all().await.unwrap();
t.record(format!("artifact bytes={}", bytes.len()));
let kept = state.retain().await.unwrap();
drop(state);
t.record("explicit hold taken");
let held = within("latest 1", latest.next()).await.unwrap();
t.record(format!(
"latest seq={} replaced={}",
held.topic_sequence(),
held.replaced()
));
let mut replaying = latecomer
.subscribe(
topic,
SubscriptionConfig::bounded().queued(4).in_flight(4).replay(true),
)
.await
.unwrap();
let replayed = within("replay", replaying.next()).await.unwrap();
t.record(format!(
"replayed seq={} step={}",
replayed.topic_sequence(),
replayed.payload()["step"]
));
drop(replayed);
let second = sealed(&producer, b"snapshot-2", "application/octet-stream").await;
let r2 = producer
.publish(topic, obj(json!({"step": "2"})), &[("state", &second)])
.await
.unwrap();
t.record(format!(
"publish seq={} subscribers={} replaced={}",
r2.topic_sequence, r2.subscribers, r2.replaced
));
drop(second);
for (who, sub) in [("bounded", &mut bounded), ("replaying", &mut replaying)] {
let m = within("second delivery", sub.next()).await.unwrap();
t.record(format!(
"{who} seq={} replaced={} step={}",
m.topic_sequence(),
m.replaced(),
m.payload()["step"]
));
}
// The latest subscriber still holds its only credit, so this replaces its queued value.
let r3 = producer
.publish(topic, obj(json!({"step": "3"})), &[])
.await
.unwrap();
t.record(format!(
"publish seq={} subscribers={} replaced={}",
r3.topic_sequence, r3.subscribers, r3.replaced
));
for (who, sub) in [("bounded", &mut bounded), ("replaying", &mut replaying)] {
let m = within("third delivery", sub.next()).await.unwrap();
t.record(format!(
"{who} seq={} replaced={} step={}",
m.topic_sequence(),
m.replaced(),
m.payload()["step"]
));
}
drop(held);
let coalesced = within("latest 2", latest.next()).await.unwrap();
t.record(format!(
"latest seq={} replaced={} step={}",
coalesced.topic_sequence(),
coalesced.replaced(),
coalesced.payload()["step"]
));
drop(coalesced);
t.record(format!(
"cleared={}",
producer.clear_topic(topic).await.unwrap()
));
drop((bounded, latest, replaying));
e.settle("unsubscribed", |s| s.subscriptions == 0).await;
t.record(format!(
"deleted={}",
producer.delete_topic(topic).await.unwrap()
));
drop(kept);
let s = e
.settle("traced artifacts collected", |s| {
s.artifacts == 0 && s.store_bytes == 0 && s.owners == 0
})
.await;
t.record(format!(
"collected artifacts={} roots={} owners={} retained_bytes={}",
s.artifacts, s.artifact_roots, s.owners, s.retained_bytes
));
e.settle_files("sealed", 0).await;
t.record("store empty");
}
async fn behaviour_trace(via: Via) -> Vec<String> {
let e = env(via).await;
let t = Trace::new();
rpc_trace(&e, &t).await;
pubsub_artifact_trace(&e, &t).await;
t.events()
}
/// BUS-01: "both transports produce equivalent behavior traces for the same scenario". The
/// trace records behaviour only: methods, payload fields, counts, sequences, credits, states
/// and error codes, never a router-issued id, a path or a time.
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn both_transports_produce_equivalent_behaviour_traces() {
let memory = behaviour_trace(Via::Memory).await;
let unix = behaviour_trace(Via::Unix).await;
if std::env::var_os("FLYBUS_TRACE").is_some() {
for (i, event) in memory.iter().enumerate() {
println!("{i:3} {event}");
}
}
assert!(memory.len() >= 25, "a thin trace: {memory:#?}");
assert_eq!(
memory, unix,
"the in-memory and Unix-socket traces disagree\nmemory: {memory:#?}\nunix: {unix:#?}"
);
}
both_transports!(
a_lost_result_leaks_no_roots_and_the_endpoint_cache_still_replays,
a_retransmission_repeats_the_domain_request_under_a_fresh_call_id,
no_automatic_retry_or_failover_onto_a_replacement_registration,
a_status_rpc_responds_while_another_handler_is_delayed,
a_latest_subscriber_never_refuses_a_publication,
a_topic_named_like_a_notice_is_still_classified_as_topic_data,
disconnect_releases_logical_ownership_without_mutating_open_bytes,
);

33
services/flysim/crates/flybus/tests/common/mod.rs
View file

@ -5,8 +5,8 @@
use std::future::Future;
use std::path::{Path, PathBuf};
use std::sync::Mutex;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use flybus::wire::{Envelope, Kind, Location, read_frame};
@ -178,6 +178,37 @@ impl Env {
}
}
/// A behaviour trace: what a scenario did, in order. Methods, payload fields, counts,
/// sequences, credits, states and error codes are behaviour; router-issued ids, paths and
/// times are not, and [`Trace::record`] refuses them. Two transports running the same
/// scenario must record the same events (implementation guide, BUS-01).
#[derive(Clone, Default)]
pub struct Trace(Arc<Mutex<Vec<String>>>);
impl Trace {
pub fn new() -> Trace {
Trace::default()
}
pub fn record(&self, event: impl Into<String>) {
let event = event.into();
for id in [
"msg-", "bus-", "conn-", "svc-", "top-", "sub-", "dlv-", "own-", "call-", "inc-",
"router-", "store-", "/tmp", "a-1",
] {
assert!(
!event.contains(id),
"a trace records behaviour, not the operational id in {event:?}"
);
}
self.0.lock().unwrap().push(event);
}
pub fn events(&self) -> Vec<String> {
self.0.lock().unwrap().clone()
}
}
pub fn obj(v: Value) -> Map<String, Value> {
match v {
Value::Object(m) => m,

26
services/flysim/crates/flybus/tests/example_demo.rs Normal file
View file

@ -0,0 +1,26 @@
//! The guide's example is also a test: `cargo run -p flybus --example demo` prints exactly
//! these lines (bus-v1 section 11, implementation guide section 1).
#[allow(dead_code)]
#[path = "../examples/demo.rs"]
mod demo;
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn the_example_shows_a_counter_rpc_an_observer_and_a_held_frame() {
let lines = demo::run().await.expect("the example ran");
assert_eq!(
lines.iter().map(String::as_str).collect::<Vec<_>>(),
vec![
// A counter service, called three times through the router.
"counter total = 1",
"counter total = 2",
"counter total = 3",
// One observer, one accepted publication, one sequence number.
"published sequence 1 to 1 subscriber(s)",
// 160x144 RGBA, read after the message object was dropped.
"read 92160 bytes after the message was dropped",
"while the frame is held: 1 artifact(s), 1 root(s)",
"after the last handle: 0 artifact(s), 0 root(s)",
]
);
}

233
services/flysim/crates/flybus/tests/perf.rs
View file

@ -1,7 +1,11 @@
//! bus-v1 section 11 item 7, as a measurement rather than a gate: 640x480 RGBA frames at
//! 60 Hz over a Unix socket to three consumers (one delayed), with 1, 2 and 4 agent services
//! pinged every frame. Router and clients share this process, so CPU and RSS are the whole
//! process. Run with:
//! bus-v1 section 11 item 7 and implementation-guide BUS-03, as a measurement rather than a
//! gate: 640x480 RGBA frames at 60 Hz over a Unix socket to three latest-mode consumers (one
//! delayed 40 ms per frame), with 1, 2 and 4 agent services called every frame.
//!
//! The router runs on its own Tokio runtime whose threads carry a distinct name, so its CPU
//! (routing plus the seal copies on its blocking pool) is measured apart from the clients'.
//! Producer copy cost and consumer readback cost are measured separately from routing. Nothing
//! here is a capacity claim: one host, one process, synthetic payloads.
//!
//! ```text
//! cargo test --release -p flybus --test perf -- --ignored --nocapture
@ -10,24 +14,55 @@
mod common;
use std::io::Write;
use std::path::PathBuf;
use std::time::{Duration, Instant};
use common::{Via, env, obj};
use flybus::{Client, Retained, ServiceConfig, SubscriptionConfig};
use common::obj;
use flybus::{
Client, ClientConfig, Policy, Retained, Router, RouterConfig, ServiceConfig,
SubscriptionConfig,
};
use serde_json::json;
const W: usize = 640;
const H: usize = 480;
const HZ: u64 = 60;
const SECONDS: u64 = 2;
const ROUTER_THREAD: &str = "flybus-router";
const ROUTER_WORKERS: usize = 2;
const CLIENT_WORKERS: usize = 4;
/// utime+stime of this process, in seconds (fields 14 and 15 of /proc/self/stat, 100 Hz).
fn proc_cpu_seconds() -> f64 {
// utime + stime, fields 14 and 15 of /proc/self/stat, in clock ticks (100 Hz on Linux).
let stat = std::fs::read_to_string("/proc/self/stat").unwrap_or_default();
let after = stat.rsplit_once(')').map_or("", |(_, rest)| rest);
let f: Vec<&str> = after.split_whitespace().collect();
thread_cpu_seconds(None)
}
/// utime+stime of the threads whose name matches, in seconds; all of them when `name` is
/// `None`. A thread that exits between two samples takes its time with it, so this is a floor
/// for pools that retire idle threads.
fn thread_cpu_seconds(name: Option<&str>) -> f64 {
let mut total = 0.0;
let Ok(dir) = std::fs::read_dir("/proc/self/task") else {
return 0.0;
};
for entry in dir.flatten() {
let Ok(stat) = std::fs::read_to_string(entry.path().join("stat")) else {
continue;
};
let Some((head, rest)) = stat.rsplit_once(')') else {
continue;
};
if let Some(want) = name {
let comm = head.split_once('(').map_or("", |(_, c)| c);
if comm != want {
continue;
}
}
let f: Vec<&str> = rest.split_whitespace().collect();
let ticks = |i: usize| f.get(i).and_then(|v| v.parse::<f64>().ok()).unwrap_or(0.0);
(ticks(11) + ticks(12)) / 100.0
total += (ticks(11) + ticks(12)) / 100.0;
}
total
}
fn proc_status(key: &str) -> String {
@ -45,12 +80,86 @@ fn pct(sorted: &[Duration], p: f64) -> Duration {
sorted[((sorted.len() - 1) as f64 * p).round() as usize]
}
async fn consumer(client: Client, delay: Duration) -> (u64, u64) {
fn ms(d: Duration) -> String {
format!("{:.2}", d.as_secs_f64() * 1000.0)
}
fn percentiles(label: &str, v: &mut [Duration]) -> String {
v.sort();
format!(
"{label} ms p50/p95/p99: {}/{}/{}",
ms(pct(v, 0.5)),
ms(pct(v, 0.95)),
ms(pct(v, 0.99))
)
}
/// The router on its own runtime, reached over a Unix socket.
struct Host {
router: Router,
socket: PathBuf,
store_root: PathBuf,
rt: Option<tokio::runtime::Runtime>,
_dir: tempfile::TempDir,
}
impl Host {
fn start() -> Host {
let dir = tempfile::tempdir().unwrap();
let store_root = dir.path().join("store");
let socket = dir.path().join("bus.sock");
let mut config = RouterConfig::new(&store_root);
config.policy = Policy::open();
let router = Router::new(config).unwrap();
let rt = tokio::runtime::Builder::new_multi_thread()
.worker_threads(ROUTER_WORKERS)
.thread_name(ROUTER_THREAD)
.enable_all()
.build()
.unwrap();
// The listener, and so every connection task, belongs to the router's runtime.
let (ready, started) = std::sync::mpsc::channel();
let (r, s) = (router.clone(), socket.clone());
rt.spawn(async move {
let _listener = r.listen_unix(&s).await.expect("the router listens");
ready.send(()).expect("start() is waiting");
std::future::pending::<()>().await
});
started.recv().expect("the router runtime started its listener");
Host {
router,
socket,
store_root,
rt: Some(rt),
_dir: dir,
}
}
async fn client(&self, id: &str) -> Client {
Client::connect_unix(&self.socket, ClientConfig::new(id, &self.store_root))
.await
.unwrap()
}
fn stop(&mut self) {
self.router.shutdown();
if let Some(rt) = self.rt.take() {
// A runtime cannot be dropped from inside another one.
std::thread::spawn(move || rt.shutdown_timeout(Duration::from_secs(2)))
.join()
.expect("the router runtime stopped");
}
}
}
/// A latest-mode consumer: extracts the frame, drops the message, reads the bytes and then
/// takes `delay` to "render" them. Returns (frames seen, coalesced, readback times).
async fn consumer(client: Client, delay: Duration) -> (u64, u64, Vec<Duration>) {
let mut sub = client
.subscribe("world.demo.frame", SubscriptionConfig::latest())
.await
.unwrap();
let (mut seen, mut replaced) = (0, 0);
let (mut seen, mut replaced, mut readback) = (0, 0, Vec::new());
while let Some(m) = sub.next().await {
if m.payload().get("end").is_some() {
break;
@ -58,29 +167,30 @@ async fn consumer(client: Client, delay: Duration) -> (u64, u64) {
replaced += m.replaced();
let frame = m.artifact("frame").unwrap();
drop(m);
let t = Instant::now();
let bytes = frame.read_all().await.unwrap();
readback.push(t.elapsed());
assert_eq!(bytes.len(), W * H * 4);
tokio::time::sleep(delay).await;
seen += 1;
}
(seen, replaced)
(seen, replaced, readback)
}
async fn run(agents: usize) {
let e = env(Via::Unix).await;
let producer = e.client("producer").await;
async fn run(host: &Host, agents: usize) {
let producer = host.client("producer").await;
producer
.declare_topic("world.demo.frame", Retained::None)
.await
.unwrap();
let mut consumers = Vec::new();
for (i, delay) in [0u64, 0, 40].into_iter().enumerate() {
let c = e.client(&format!("consumer-{i}")).await;
let c = host.client(&format!("consumer-{i}")).await;
consumers.push(tokio::spawn(consumer(c, Duration::from_millis(delay))));
}
let mut services = Vec::new();
for k in 0..agents {
let c = e.client(&format!("agent-{k}")).await;
let c = host.client(&format!("agent-{k}")).await;
let mut svc = c
.register(&format!("agent.a{k}"), ServiceConfig::default())
.await
@ -92,16 +202,20 @@ async fn run(agents: usize) {
}
}));
}
let caller = e.client("coordinator").await;
let caller = host.client("coordinator").await;
// Let every subscription land before the first frame.
e.settle("subscribed", |s| s.subscriptions == 3).await;
while host.router.stats().subscriptions != 3 {
tokio::time::sleep(Duration::from_millis(5)).await;
}
let pixels: Vec<u8> = (0..W * H * 4).map(|i| (i % 253) as u8).collect();
let frames = HZ * SECONDS;
let period = Duration::from_nanos(1_000_000_000 / HZ);
let (mut produce, mut publish, mut rpc) = (Vec::new(), Vec::new(), Vec::new());
let (mut allocate, mut copy, mut seal) = (Vec::new(), Vec::new(), Vec::new());
let (mut publish, mut rpc) = (Vec::new(), Vec::new());
let (mut peak_bytes, mut peak_roots, mut peak_queued, mut late) = (0u64, 0u64, 0usize, 0u32);
let cpu0 = proc_cpu_seconds();
let router_cpu0 = thread_cpu_seconds(Some(ROUTER_THREAD));
let start = Instant::now();
for n in 0..frames {
let deadline = start + period * n as u32;
@ -111,9 +225,13 @@ async fn run(agents: usize) {
.allocate(pixels.len() as u64, "image/x-rgba")
.await
.unwrap();
allocate.push(t.elapsed());
let t = Instant::now();
w.write_all(&pixels).unwrap();
copy.push(t.elapsed());
let t = Instant::now();
let frame = w.seal().await.unwrap();
produce.push(t.elapsed());
seal.push(t.elapsed());
let t = Instant::now();
producer
.publish(
@ -140,7 +258,7 @@ async fn run(agents: usize) {
for c in calls {
rpc.push(c.await.unwrap());
}
let s = e.stats();
let s = host.router.stats();
peak_bytes = peak_bytes.max(s.store_bytes);
peak_roots = peak_roots.max(s.artifact_roots);
peak_queued = peak_queued.max(s.queued);
@ -153,61 +271,72 @@ async fn run(agents: usize) {
}
let wall = start.elapsed().as_secs_f64();
let cpu = proc_cpu_seconds() - cpu0;
let router_cpu = thread_cpu_seconds(Some(ROUTER_THREAD)) - router_cpu0;
let end = Instant::now();
producer
.publish("world.demo.frame", obj(json!({"end": true})), &[])
.await
.unwrap();
let mut results = Vec::new();
let mut readback = Vec::new();
for c in consumers {
results.push(c.await.unwrap());
let (seen, replaced, mut times) = c.await.unwrap();
readback.append(&mut times);
results.push((seen, replaced));
}
while host.router.stats().store_bytes != 0 {
assert!(
end.elapsed() < Duration::from_secs(10),
"the store never drained: {:?}",
host.router.stats()
);
tokio::time::sleep(Duration::from_millis(1)).await;
}
e.settle("collected", |s| s.store_bytes == 0).await;
let collect_lag = end.elapsed();
let live = host.router.stats();
for s in services {
s.abort();
}
for v in [&mut produce, &mut publish, &mut rpc] {
v.sort();
}
let ms = |d: Duration| format!("{:.2}", d.as_secs_f64() * 1000.0);
println!("agents={agents} frames={frames} over {wall:.2}s, late frames {late}");
println!(" {}", percentiles("allocate (quota + staging file)", &mut allocate));
println!(" {}", percentiles("producer copy into staging", &mut copy));
println!(" {}", percentiles("seal (router copy to a sealed inode)", &mut seal));
println!(" {}", percentiles("publish admission", &mut publish));
println!(" {}", percentiles("rpc round trip", &mut rpc));
println!(" {}", percentiles("consumer readback of 1.2 MB", &mut readback));
println!(
" produce (allocate+write+seal copy) ms p50/p95/p99: {}/{}/{}",
ms(pct(&produce, 0.5)),
ms(pct(&produce, 0.95)),
ms(pct(&produce, 0.99))
);
println!(
" publish admission ms p50/p95/p99: {}/{}/{}",
ms(pct(&publish, 0.5)),
ms(pct(&publish, 0.95)),
ms(pct(&publish, 0.99))
);
println!(
" rpc round trip ms p50/p95/p99: {}/{}/{}",
ms(pct(&rpc, 0.5)),
ms(pct(&rpc, 0.95)),
ms(pct(&rpc, 0.99))
);
println!(
" process cpu {:.2} cores; VmRSS {} VmHWM {}",
" cpu cores: router {:.3} of {ROUTER_WORKERS} threads, whole process {:.3} of {} threads on {} cpus",
router_cpu / wall,
cpu / wall,
ROUTER_WORKERS + CLIENT_WORKERS,
std::thread::available_parallelism().map_or(0, |n| n.get())
);
println!(
" VmRSS {} VmHWM {}",
proc_status("VmRSS:"),
proc_status("VmHWM:")
);
println!(
" store peak {:.1} MB, peak roots {peak_roots}, peak queued {peak_queued}, drain+collect {:.1} ms",
" store peak {:.1} MB, live after drain {} B; roots peak {peak_roots}, live {}; queued peak {peak_queued}, live {}",
peak_bytes as f64 / 1e6,
live.store_bytes,
live.artifact_roots,
live.queued
);
println!(
" collection lag after the last frame {:.1} ms",
collect_lag.as_secs_f64() * 1000.0
);
println!(" consumers (frames seen, replaced): {results:?}");
println!(" consumers (frames seen, coalesced): {results:?}");
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[ignore = "measurement; run with --release -- --ignored --nocapture"]
async fn frames_at_60hz_with_three_consumers() {
for agents in [1, 2, 4] {
run(agents).await;
let mut host = Host::start();
run(&host, agents).await;
host.stop();
}
}

32
services/flysim/crates/flybus/tests/sol_rereview_regressions.rs
View file

@ -734,8 +734,17 @@ async fn teardown_waits_for_an_active_transport_poll_before_reclaiming() {
.store_dir()
.join("sealed")
.join(&artifact.reference().artifact_id);
// A deliberately large delivery: the transport under the gate writes one byte per poll,
// so a writer that resumes cannot possibly finish this frame inside the window between
// releasing the held poll and teardown marking the stream closing. Without that, a short
// frame sometimes completes first, which is teardown's other legal arm and would make the
// assertions below a coin toss rather than a test of the ordering.
publisher
.publish("t.poll-gate", obj(json!({})), &[("data", &artifact)])
.publish(
"t.poll-gate",
obj(json!({"blob": "p".repeat(50_000)})),
&[("data", &artifact)],
)
.await
.unwrap();
drop(artifact);
@ -745,11 +754,17 @@ async fn teardown_waits_for_an_active_transport_poll_before_reclaiming() {
let done = Arc::new(AtomicBool::new(false));
let shutdown_done = done.clone();
let shutdown_router = router.clone();
// The thread announces itself before calling shutdown, so the assertions below need no
// sleep: teardown cannot get past the write gate until the held poll returns, which only
// `hold.release()` allows.
let (started_tx, started_rx) = std::sync::mpsc::channel();
let shutdown = std::thread::spawn(move || {
started_tx.send(()).expect("the test is waiting");
shutdown_router.shutdown();
shutdown_done.store(true, Ordering::SeqCst);
});
tokio::time::sleep(Duration::from_millis(50)).await;
started_rx.recv().expect("shutdown thread started");
for _ in 0..64 {
assert!(
!done.load(Ordering::SeqCst),
"teardown completed while poll_write was active"
@ -758,6 +773,8 @@ async fn teardown_waits_for_an_active_transport_poll_before_reclaiming() {
sealed_path.exists(),
"artifact was reclaimed while poll_write was active"
);
tokio::task::yield_now().await;
}
hold.release();
shutdown.join().unwrap();
@ -765,12 +782,23 @@ async fn teardown_waits_for_an_active_transport_poll_before_reclaiming() {
assert_eq!(router.stats().owners, 0);
assert_eq!(router.stats().artifacts, 0);
assert!(!sealed_path.exists());
let mut ops = Vec::new();
while let Some(envelope) = within("poll-gate close", raw.recv()).await {
assert_ne!(
envelope.op, "topic.message",
"delivery completed after teardown reclaimed its owner"
);
ops.push(envelope.op);
}
// The delivery never completes, so only teardown's two shapes are legal: the frame was
// cut short and nothing whatever follows it, or it never began and the stream is still
// frame aligned, in which case the closing notices are all that follow.
assert!(
ops.is_empty()
|| ops == ["subscription.closed".to_owned(), "connection.closing".to_owned()],
"a cut stream carries nothing more and an aligned one exactly the closing notices: \
{ops:?}"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]