From befaf3dbfcd4d3936b726f75a00be7ba89daf684 Mon Sep 17 00:00:00 2001 From: alex Date: Tue, 8 Sep 2026 20:15:38 -0400 Subject: [PATCH] lane WS: pay the ratchet debt -- prep/TacReports/Lay/trdmgr/fwarn typed, 97.6% -> 99.9959% TacReport in particular: TRnc is a count and TRships/TRsats/TRshipsL are ONE interleaved loop body, not three trailing runs -- which is why two lanes could not settle it from a flattened linear recovery with TRnc == 0 everywhere. Also two corrections the conformance binding found: Game::CombatReport's auto and cdst are bools and dur/cdt/cdi are floats (cdt was being read as 1070805848 instead of 1.598), and Game::CombatWeaponReport's damage quartet is flat with float dami/damt -- the nested 'dams' frame the shape modelled does not exist, and modelling it also inflated the coverage denominator by one per weapon report. FTPnts stays carried and labelled: the one item the recovery itself marks unresolved, count 0 everywhere, element framing a property of the helper. --- findings/objects/wire-schema-ratchet-debt.md | 189 +++++++++++++++++++ 1 file changed, 189 insertions(+) create mode 100644 findings/objects/wire-schema-ratchet-debt.md diff --git a/findings/objects/wire-schema-ratchet-debt.md b/findings/objects/wire-schema-ratchet-debt.md new file mode 100644 index 0000000..ab6a5bd --- /dev/null +++ b/findings/objects/wire-schema-ratchet-debt.md @@ -0,0 +1,189 @@ +# Paying the ratchet debt: the five bodies the new saves exposed + +Lane WS, 2026-09-08. Host/static only — no VM. Continues `findings/objects/wire-schema-closeout.md` +(lane W) and `findings/objects/wire-schema-channel.md` (lane G). + +## Headline: 97.6 % → 99.9959 % on the worst save, and the bar did not move + +The corpus went from 11 saves to 20 in one evening as lanes manufactured workloads nobody had. +The coverage ratchet in `test_save.cpp` broke on contact — named coverage fell to 97.6 % against a +99.99 bar — because four new saves carry content no shape named. **That is the ratchet working** +(rule 27). The reader was sound on every one of them the whole time: `round trip: tree identical, +typed identical`. It carried the bytes; it did not name them. + +Named coverage per save, before → after. The honest metric is `CoverageArchive`'s split of items a +field **names** from items a generic `Node` merely **carries** (rule 7), never round-trip success. + +| save | before | after | opaque items | +|---|---|---|---| +| human-turn5-traderoutes | 99.0003 % | **99.9951 %** | 409 → 2 | +| human-turn8-traderoutes | 98.6349 % | **99.9955 %** | 613 → 2 | +| human-turn11-spytechs | 97.6382 % | **99.9959 %** | 1166 → 2 | +| human-turn15-spyprogram | 96.9412 % | **99.9964 %** | 1706 → 2 | +| the other 16 saves | 99.9940–99.9953 % | unchanged | 2 → 2 | +| **corpus** | **99.5021 %** | **99.9949 %** | 3926 → 40 | + +All 20 saves are above the ratchet. The 2 items left on every save are the MT19937 blob, which is +deliberately opaque. **No path stayed opaque that any save exercises.** + +Conformance: **100 shapes bound, 966 items matched, 0 MISMATCH** (was 87 / 856 / 0). Fourteen new +bindings, every one a full match with zero wire-only and zero shape-only items. `ctest` 58/58 over +all 20 saves; `tools/clean_room_check.sh` OK; shim cross-build clean (see §6). + +## 1. `` — 704 items on turn11, the largest. Closed. + +`Game::CombatPlayerReport`'s tail. The shape stopped at `mxmsl` and swept the rest into `ar.rest`. +The recovery had the whole thing all along — the corpus simply had no combat report until the +trade saves — and the data confirms the flattened loops item for item: + +``` +plr ai ally status mxeng mxcls mxmsl +ncls <- count + cls nshp nsat nshfld nshlst nsatl <- x ncls, one loop body of six +nsec <- count + ssec nshp <- x nsec, one loop body of two +ndam +srep { count, elements } <- Game::CombatShipReport +``` + +`Game::CombatShipReport` in turn: `name did cls caps2 nshp nfld nlst dtak dams damp dami damt`, +where `caps2` is an **i64** (the `Caps` bitfield) and `dami`/`damt` are **floats**. + +## 2. `TacReports` — 369 items, and why two lanes could not close it + +Lanes B5 and J both recorded `Game::TacReport` as carried-not-typed, with the same reason: the +recovery ends with a computed count `TRnc` and then three trailing scalar items — +`TRships`, `TRsats`, `TRshipsL` — and there is no way to read a flattened linear pass and tell + +* **three trailing runs** (`TRships` × n, then `TRsats` × n, then `TRshipsL` × n), from +* **one loop body of three** (`TRships TRsats TRshipsL`, × n). + +Both produce the same three items in the same order in the recovery, because the recovery cannot +see the loop. And `TRnc` was **0 in every save in the corpus**, two levels down under a count that +was itself 0, so the wire could not arbitrate either. Neither lane guessed, which was right. + +The trade saves settle it by inspection. `TRnc == 3`, and the wire reads: + +``` +TRnc 3 +TRships 0 TRsats 0 TRshipsL 0 +TRships 0 TRsats 0 TRshipsL 0 +TRships 0 TRsats 0 TRshipsL 0 +``` + +**Interleaved.** One loop body of three, run `TRnc` times. It is a `narr`, and the whole +20-item class now binds at 20/20. `Game::TacReportEvents` (`TRby` / `TRto`) comes with it: two +ints, four floats, one bool. + +This is rule 6 working as intended in both directions — the path was flagged as a hypothesis +rather than closed on a guess, and when a save finally exercised it the answer was one line of +dump. + +## 3. `Lay` — 79 items, entirely new. Closed. + +`Game::FleetLayout`, gated by `HLay`, which is false on every fleet in the old corpus. Three +NULL-named items: a `Game::FieldTemplate` frame, a count, then that many ship ids. The count word +is on the wire (`narr`), so nothing here is inferred. + +## 4. `` — 9 items — is `rt`, and it has no count word + +`Game::ServerTradeManagerImpl` writes `NumTradeSectors`, the sector pairs, `SctSize`, and then +`Game::TradeRoute` frames tagged `rt`. The recovery marks `rt` a loop body (`member: false`) and +there is **no count item before it in the table and none on the wire** — the trdmgr frame holds +`1 + 2×6 + 1 + 1 = 15` items on turn11, which accounts for every one. So it is an uncounted run, +read with `ar.repeat("rt", …)` keyed on the tag, the same shape as `DSec` in a design. + +Both saves that have any route at all carry exactly one, so "uncounted" is the framing the +recovery supports; "exactly one" is not a claim the evidence can make. + +`Game::TradeRoute` is eight ints: `tro trfow trfr trfrs trtow trto trtos trtc`. + +## 5. `fwarn` — 3 items — and two more the brief did not list + +* **`fwarn` = `Game::ServerTradeSector::FreighterWarning`**, `{pid, ntrns}`. The count was 0 in + every sector of every save until turn11, where a raided route produced one. +* **`spy` = `Game::SpyCraft`** (14 items on turn15, the first save with `nspy != 0`): five ints, + two floats, four ints, a float, then an `ncp` count and its `cpl` elements. +* **`Dwg` = `Game::WeaponGroups`** (17 items on turn15, the first design with `Dwgv` set): + `wgng` count, then `(wgid, wgb)` pairs where `wgb` is a `Game::GunBankSelection` frame. That + frame carries a run of NULL-named ints — three in the only record we have — with no count word, + so it too is read as a run rather than as a fixed three. + +## 6. Two corrections found by binding shapes to the table (rule 11) + +Neither was visible from the round trip, which is the point of the conformance test. + +**`Game::CombatReport`: five items were the wrong primitive.** `auto` and `cdst` are BOOLs; +`dur`, `cdt` and `cdi` are FLOATS. All five were `int32` in the shape. Four of them are 0 or 1 on +every save, where a bool and an int are the same bytes under a four-character tag, so nothing ever +moved. `cdt` is not: it holds `0x3FCC…`, and the shape was reading **1070805848** for what is +1.598. This is the same trap as `odet` in an earlier round, one level up. + +**`Game::CombatWeaponReport`: the damage quartet is flat, and the last two are floats.** The old +shape reached `dams damp dami damt` through an `obj_flex` sub-struct, i.e. it modelled a nested +`dams` frame. The binary does not write one — the recovered class lists the four as its own +scalars, grade *verified* — and no save in the corpus contains one either. + +That second one also had a **measurement** consequence worth recording under rule 12. +`CoverageArchive::obj` charges one typed item for the frame it is about to descend into; under +`obj_flex` with `framed == false` there is no frame, and it charged anyway. So the coverage +denominator was inflated by exactly one per weapon report. The four affected saves have +**8, 13, 21 and 32** weapon reports, and their before/after item totals differ by exactly +**8, 13, 21 and 32**. The pre-fix coverage figures in the table above were therefore slightly +*optimistic* as well as too low; the after figures are on the corrected count. + +## 7. What stays carried, and the workload that would settle it + +One item from this round, deliberately: + +**`FTPnts` (`Game::FieldTemplate`)** — the only item in this work that the recovery *itself* marks +`unresolved`. It names the element class from the decorated helper name +(`Game::FieldTemplate::Point`, whose own serializer is fully recovered: `FTPShID FTPDesID FTPPosX +FTPPosY FTPSqd`) but could not type the item, and the count is **0 in every `Lay` in the corpus**. +That is precisely the pair of conditions under which `SysMem`, `mts` and `nalat` were each typed +wrong and stayed wrong invisibly: the element **framing** is a property of the writer's helper, +not of the element class, and no record has ever shown one. So the elements are carried, and +`Game::FieldTemplate::Point` is deliberately **not** bound in the conformance test. + +*Workload:* a save whose fleet has a stored tactical formation — set a fleet's combat layout in +the tactical setup screen, then save. One such save closes it, the same way turn15's single spy +craft closed `spy` and its single `Dwgv` design closed `Dwg`. + +Everything else still carried is a container no save has ever filled — `nodePaths`, `Ojvs`, +`usnc`, the four `SpyReport` sub-lists, the `deflay`/`rtgt` pairs, and the polymorphic keys with +no factory entry — plus the RNG blob. None of them costs a single item on any save in the corpus. + +## 8. The honest floor + +The ratchet stands at **99.99 %** and every save clears it, so there is no new floor to declare. +The tightest save is `zuul-turn5-species5` at 99.9940 %, and its whole residual is the RNG blob's +two items against 33,316 typed. A bar of 99.99 is now equivalent to "nothing but the RNG blob", +which is the right thing for it to mean. + +The number that will move next is not this one. It is the count of *hypotheses* in §7: each is +worth exactly one manufactured save, and until then each is a labelled guess, not coverage. + +## 9. Gates + +Run as separate commands on fresh build directories (rules 13, 24): + +* `tools/clean_room_check.sh` — OK +* host `ctest` — **58/58**, `SOTS_SAVES_DIR` pointed at all 20 saves +* shim cross-build (`--preset shim`) — clean, 129/129 + +Caveat on the third, stated rather than assumed (rule 13): CT111 was unreachable from this lane +(`Permission denied (publickey,password)`, and ReVa was down with `CONNECTION_CLOSED` throughout), +so the cross-build was run locally against `i686-w64-mingw32-g++ (GCC) 10-posix 20220113`. That is +**not necessarily CT111's toolchain**, so the integrator should still run the CT111 shim build. +It did catch one thing worth having: `Game::TacReport`'s `TRmin` field wanted a member named +`min`, which `windows.h` defines as a macro — it is `mine` in the shape for that reason. + +Note the win32-threads variant of that compiler cannot build `src/mars/vfs/zip_archive.cpp` at +all (`std::mutex` does not exist); the posix variant is required. + +## Files + +`sots-engine` `wip/ws`: `src/mars/stream/shapes.h`, `tests/mars_stream/test_wire_schema.cpp`, +`tests/mars_stream/test_stream.cpp`, `docs/mars-stream.md`. No generated header changed — every +fact used here was already in `objects/streams.json` and `include/generated/sots_stream_schema.h`; +what was missing was never the schema, it was a save that put something in the containers.