sots-re/findings/objects/wire-schema-ratchet-debt.md

Paying the ratchet debt: the five bodies the new saves exposed

Lane WS, 2026-09-08. Host/static only — no VM. Continues findings/objects/wire-schema-closeout.md (lane W) and findings/objects/wire-schema-channel.md (lane G).

Headline: 97.6 % → 99.9959 % on the worst save, and the bar did not move

The corpus went from 11 saves to 20 in one evening as lanes manufactured workloads nobody had. The coverage ratchet in test_save.cpp broke on contact — named coverage fell to 97.6 % against a 99.99 bar — because four new saves carry content no shape named. That is the ratchet working (rule 27). The reader was sound on every one of them the whole time: round trip: tree identical, typed identical. It carried the bytes; it did not name them.

Named coverage per save, before → after. The honest metric is CoverageArchive's split of items a field names from items a generic Node merely carries (rule 7), never round-trip success.

save	before	after	opaque items
human-turn5-traderoutes	99.0003 %	99.9951 %	409 → 2
human-turn8-traderoutes	98.6349 %	99.9955 %	613 → 2
human-turn11-spytechs	97.6382 %	99.9959 %	1166 → 2
human-turn15-spyprogram	96.9412 %	99.9964 %	1706 → 2
the other 16 saves	99.9940–99.9953 %	unchanged	2 → 2
corpus	99.5021 %	99.9949 %	3926 → 40

All 20 saves are above the ratchet. The 2 items left on every save are the MT19937 blob, which is deliberately opaque. No path stayed opaque that any save exercises.

Conformance: 100 shapes bound, 966 items matched, 0 MISMATCH (was 87 / 856 / 0). Fourteen new bindings, every one a full match with zero wire-only and zero shape-only items. ctest 58/58 over all 20 saves; tools/clean_room_check.sh OK; shim cross-build clean (see §6).

1. <rest:prep> — 704 items on turn11, the largest. Closed.

Game::CombatPlayerReport's tail. The shape stopped at mxmsl and swept the rest into ar.rest. The recovery had the whole thing all along — the corpus simply had no combat report until the trade saves — and the data confirms the flattened loops item for item:

plr ai ally status mxeng mxcls mxmsl
ncls                                    <- count
  cls nshp nsat nshfld nshlst nsatl     <- x ncls, one loop body of six
nsec                                    <- count
  ssec nshp                             <- x nsec, one loop body of two
ndam
srep { count, elements }                <- Game::CombatShipReport

Game::CombatShipReport in turn: name did cls caps2 nshp nfld nlst dtak dams damp dami damt, where caps2 is an i64 (the Caps bitfield) and dami/damt are floats.

2. TacReports — 369 items, and why two lanes could not close it

Lanes B5 and J both recorded Game::TacReport as carried-not-typed, with the same reason: the recovery ends with a computed count TRnc and then three trailing scalar items — TRships, TRsats, TRshipsL — and there is no way to read a flattened linear pass and tell

• three trailing runs (TRships × n, then TRsats × n, then TRshipsL × n), from
• one loop body of three (TRships TRsats TRshipsL, × n).

Both produce the same three items in the same order in the recovery, because the recovery cannot see the loop. And TRnc was 0 in every save in the corpus, two levels down under a count that was itself 0, so the wire could not arbitrate either. Neither lane guessed, which was right.

The trade saves settle it by inspection. TRnc == 3, and the wire reads:

TRnc 3
TRships 0  TRsats 0  TRshipsL 0
TRships 0  TRsats 0  TRshipsL 0
TRships 0  TRsats 0  TRshipsL 0

Interleaved. One loop body of three, run TRnc times. It is a narr, and the whole 20-item class now binds at 20/20. Game::TacReportEvents (TRby / TRto) comes with it: two ints, four floats, one bool.

This is rule 6 working as intended in both directions — the path was flagged as a hypothesis rather than closed on a guess, and when a save finally exercised it the answer was one line of dump.

3. Lay — 79 items, entirely new. Closed.

Game::FleetLayout, gated by HLay, which is false on every fleet in the old corpus. Three NULL-named items: a Game::FieldTemplate frame, a count, then that many ship ids. The count word is on the wire (narr), so nothing here is inferred.

4. <rest:trdmgr> — 9 items — is rt, and it has no count word

Game::ServerTradeManagerImpl writes NumTradeSectors, the sector pairs, SctSize, and then Game::TradeRoute frames tagged rt. The recovery marks rt a loop body (member: false) and there is no count item before it in the table and none on the wire — the trdmgr frame holds 1 + 2×6 + 1 + 1 = 15 items on turn11, which accounts for every one. So it is an uncounted run, read with ar.repeat("rt", …) keyed on the tag, the same shape as DSec in a design.

Both saves that have any route at all carry exactly one, so "uncounted" is the framing the recovery supports; "exactly one" is not a claim the evidence can make.

Game::TradeRoute is eight ints: tro trfow trfr trfrs trtow trto trtos trtc.

5. fwarn — 3 items — and two more the brief did not list

• fwarn = Game::ServerTradeSector::FreighterWarning, {pid, ntrns}. The count was 0 in every sector of every save until turn11, where a raided route produced one.
• spy = Game::SpyCraft (14 items on turn15, the first save with nspy != 0): five ints, two floats, four ints, a float, then an ncp count and its cpl elements.
• Dwg = Game::WeaponGroups (17 items on turn15, the first design with Dwgv set): wgng count, then (wgid, wgb) pairs where wgb is a Game::GunBankSelection frame. That frame carries a run of NULL-named ints — three in the only record we have — with no count word, so it too is read as a run rather than as a fixed three.

6. Two corrections found by binding shapes to the table (rule 11)

Neither was visible from the round trip, which is the point of the conformance test.

Game::CombatReport: five items were the wrong primitive. auto and cdst are BOOLs; dur, cdt and cdi are FLOATS. All five were int32 in the shape. Four of them are 0 or 1 on every save, where a bool and an int are the same bytes under a four-character tag, so nothing ever moved. cdt is not: it holds 0x3FCC…, and the shape was reading 1070805848 for what is 1.598. This is the same trap as odet in an earlier round, one level up.

Game::CombatWeaponReport: the damage quartet is flat, and the last two are floats. The old shape reached dams damp dami damt through an obj_flex sub-struct, i.e. it modelled a nested dams frame. The binary does not write one — the recovered class lists the four as its own scalars, grade verified — and no save in the corpus contains one either.

That second one also had a measurement consequence worth recording under rule 12. CoverageArchive::obj charges one typed item for the frame it is about to descend into; under obj_flex with framed == false there is no frame, and it charged anyway. So the coverage denominator was inflated by exactly one per weapon report. The four affected saves have 8, 13, 21 and 32 weapon reports, and their before/after item totals differ by exactly 8, 13, 21 and 32. The pre-fix coverage figures in the table above were therefore slightly optimistic as well as too low; the after figures are on the corrected count.

7. What stays carried, and the workload that would settle it

One item from this round, deliberately:

FTPnts (Game::FieldTemplate) — the only item in this work that the recovery itself marks unresolved. It names the element class from the decorated helper name (Game::FieldTemplate::Point, whose own serializer is fully recovered: FTPShID FTPDesID FTPPosX FTPPosY FTPSqd) but could not type the item, and the count is 0 in every Lay in the corpus. That is precisely the pair of conditions under which SysMem, mts and nalat were each typed wrong and stayed wrong invisibly: the element framing is a property of the writer's helper, not of the element class, and no record has ever shown one. So the elements are carried, and Game::FieldTemplate::Point is deliberately not bound in the conformance test.

Workload: a save whose fleet has a stored tactical formation — set a fleet's combat layout in the tactical setup screen, then save. One such save closes it, the same way turn15's single spy craft closed spy and its single Dwgv design closed Dwg.

Everything else still carried is a container no save has ever filled — nodePaths, Ojvs, usnc, the four SpyReport sub-lists, the deflay/rtgt pairs, and the polymorphic keys with no factory entry — plus the RNG blob. None of them costs a single item on any save in the corpus.

8. The honest floor

The ratchet stands at 99.99 % and every save clears it, so there is no new floor to declare. The tightest save is zuul-turn5-species5 at 99.9940 %, and its whole residual is the RNG blob's two items against 33,316 typed. A bar of 99.99 is now equivalent to "nothing but the RNG blob", which is the right thing for it to mean.

The number that will move next is not this one. It is the count of hypotheses in §7: each is worth exactly one manufactured save, and until then each is a labelled guess, not coverage.

9. Gates

Run as separate commands on fresh build directories (rules 13, 24):

• tools/clean_room_check.sh — OK
• host ctest — 58/58, SOTS_SAVES_DIR pointed at all 20 saves
• shim cross-build (--preset shim) — clean, 129/129

Caveat on the third, stated rather than assumed (rule 13): CT111 was unreachable from this lane (Permission denied (publickey,password), and ReVa was down with CONNECTION_CLOSED throughout), so the cross-build was run locally against i686-w64-mingw32-g++ (GCC) 10-posix 20220113. That is not necessarily CT111's toolchain, so the integrator should still run the CT111 shim build. It did catch one thing worth having: Game::TacReport's TRmin field wanted a member named min, which windows.h defines as a macro — it is mine in the shape for that reason.

Note the win32-threads variant of that compiler cannot build src/mars/vfs/zip_archive.cpp at all (std::mutex does not exist); the posix variant is required.

Files

sots-engine wip/ws: src/mars/stream/shapes.h, tests/mars_stream/test_wire_schema.cpp, tests/mars_stream/test_stream.cpp, docs/mars-stream.md. No generated header changed — every fact used here was already in objects/streams.json and include/generated/sots_stream_schema.h; what was missing was never the schema, it was a save that put something in the containers.