sots-re/findings/objects/wire-schema-channel.md

The wire-schema channel: getting lane D's layouts into the engine

Lane G, 2026-09-08. Host/static only — VM140 was held by lane U and the game was never run.

Lane D recovered 386 class layouts from the Mars::IStreamable serializers. That knowledge lived only in the notes repo. This lane built the channel that carries it into sots-engine, wired the engine's save codec to it, and used it to find and fix four real defects in the reader. Engine coverage of a real save went from 37.9 % to 97.1 % of stream items typed (not merely round-tripped), with the byte-identical round trip preserved throughout.

---

1. The design choice: a wire schema, not a struct layout

objects/layouts.json is a memory-layout projection and is the wrong shape for a codec. Two things are lost in serializers.py's build(), and both are the substance of the on-disk format:

• Order. Lab.layout() returns fields in the program order of Write (with base-class and sub-writer calls spliced in at their call site) — that order is the disk order. build() then does fields.sort(key=lambda x: (x["off_abs"], x["va"])). 89 of the 386 classes have at least one field whose offset order differs from its write order, so for those the published record does not describe the stream.
• Repetition. build() merges two writes of the same offset into alt_tags. That is exactly the SVSOJewelsOfTheCrown JEWELLOCATIONID-twice trap, and the merge silently drops the second item.

So this lane added tools/streams.py, a second projection of the same recovery that keeps the order and the repeats — and drops every memory fact: no off, no size, no sizeof, no gaps, no strides. sots-engine is our own C++, not a byte-for-byte decomp; it must read and write the format faithfully and must not inherit the original's ABI in its runtime types. (The shim is the one component that legitimately needs the original ABI, because it reads the running game's memory. Those few offsets already have a home: offset entries in ghidra/addresses.json.)

The on-disk primitive is not layouts.json's kind, which is a memory type. It comes from the stream vftable slot the writer called, or the wrapper helper it called instead:

slot	helper	disk
+0x18	0x8b9d70	string
+0x1c	0x8b9c20	bool (1 byte)
+0x20	0x8b9be0	float32
+0x24	0x8b9d50, 0x8b9d00, 0x816490	int32
+0x28	—	nested frame
+0x30	0x8b9c60	raw (n = 8 → int64)

A member the original holds as int16 or int8 is written by WriteInt and is four bytes on the wire. 0x8b9d00 is a distinct int16 wrapper that widens; 0x816490 is Stream::WriteNetworkObjectId, an id, also int32. Nothing narrow ever reaches the stream.

tools/gen_stream_schema.py then emits include/generated/sots_stream_schema.h — a sibling of gen_addresses.py under the same discipline: generated, provenance header, never hand-edited, regenerated on every merge. 386 classes, 2,042 wire items.

2. Why it is a check, not a generator

The recovered schema is a specification, not a program. Two properties make a codec driven straight off it desynchronise on real data:

• It is unconditional. The recovery is a linear pass over Write and cannot see Write's branches. StarShip writes BQ2 only when hbq is set, and pop/ppop only when hsp is set; the table lists all three unconditionally. The sequence is a superset of any single record.
• Container loops are flattened. A count item is followed by its element items as siblings, not nested inside the container. member == false marks the elements, and that is the only signal.

The engine's hand-written io(Ar&) shapes therefore stay the codec — they can express the conditionals and the nesting that the binary facts cannot supply — and the generated table is what proves they agree with the binary, item for item, in order. That is the same relationship sots_addresses.h already has with the shim: it supplies facts that hand-written code consumes; it does not generate code.

sots-engine grew a fourth archive, SchemaProbe (src/mars/stream/probe.h), which walks a shape's io() with every branch taken — the same "all branches" view the recovery has — and records the item sequence. tests/mars_stream/test_wire_schema.cpp LCS-aligns that against the generated table. 56 shapes bound, 657 items matched, 0 mismatches. A tag both sides name with different disk primitives is a hard build failure; a wire-only item is reported, not fatal.

3. What the check found

Four defects, all present in both save_reader.py and the engine, none of which any round-trip test could have caught:

1. SystemParams field 1 is a std::string, not an int. Both readers modelled it as int. It is the empty string in every save available, and an empty string is four zero bytes — byte-identical to the int 0. A save carrying a non-empty name here would have desynchronised both parsers. The engine now reads it as a string.
2. ObservedTech::odet / ObservedWeapon::odet are bool, not int. The binary calls the bool writer and lane D's own golden table already said bool at +0x08; save_reader.py and the engine both said int. Byte-safe here only by coincidence: with a 4-char tag a bool item and an int item both occupy 12 bytes and the bool's three pad bytes are zeroed. A shorter tag would not have been so forgiving. 68 otch elements per save exercised this and never showed it.
3. SpeciesRatios::nv is a count, not a field. The recovery flags sp and va2 as loop-body writes; the saves agree (nv==1 frames carry one (sp, va2) pair, nv==0 frames carry nothing).
4. ShipRecords::srbd is a count, not a field — the second of two counted sections (srd, src, srb, srl, sri, note no srk). Values 0, 1, 3 and 4 occur across the players available.

Applying all four left the byte-identical round trip intact on all four real saves.

4. Two recorded traps resolved

• Game::ServerTradeManager's Read/Write really are the inherited no-op — and the trdmgr frame really does have content. The resolution is that the call is virtual: the object is a Game::ServerTradeManagerImpl, and that class has a real serializer emitting exactly NumTradeSectors / TradeID / Trade / SctSize, with Game::ServerTradeSector giving the sector body (14 items). The same shape resolves Game::IServerSpyManager → Game::ServerSpyManager. General rule for the next lane: when an interface type's serializer is the inherited stub, look for the concrete *Impl / non-I class.
• Game::ShipDesign::Write makes no stream call, as recorded — and that is still true. But the Des frame's section body is Game::ShipDesignDef::Section, which is recovered (DSec frame → Game::ShipSectionID, DGbnk2 carr, DOpts carr), so most of the design record is typeable anyway. Only DW2, Dtc and Dwgv have no recovered serializer and came from the saves.

5. Coverage

A byte-identical round trip is not a coverage claim: the engine reached it partly by typing fields and partly by carrying whole bodies as generic Nodes, which round-trip trivially because they are copied verbatim. CoverageArchive (also in probe.h) separates the two by walking a populated shape's io() and counting items a field names against items only a Node carried.

save	before	after
turn1-state	37.9 %	97.1 %
turn2-state	38.8 %	97.2 %
turn3-state	39.4 %	97.2 %
zuul-turn5-species5	42.5 %	97.6 %

The bodies typed this round, all against the recovered schema: TechTree (both NumTechs sections — the second is the per-tech state, whose field list and types come from Game::SpyReportTechTree, which streams the same record, and which is where Tfc being a bool came from), Events (EventStorage → TurnEvents → Event, three nesting levels), ShipRecs, sprjs, civr/spe, comms, spy2, spymgr, aid, Ojvs, AIEnf, FNG, trdmgr, and the Des section / gun-bank tree. DW2 is branch-gated on bID: wid (an id) when set, wfn (a family name) when clear.

What is still opaque, and why (turn1 numbers, of 36,644 items):

region	items	why
CD custom-data blocks	744	TurnCommands_v5 is 35 anonymous scalars with 27 trailing zeros — a no-orders snapshot. Typing it needs a save with issued turn commands.
SvSctOb	147	Scenario/encounter objects: 8 EncObj variants keyed by EncID, structure known but each needs its own shape.
DOpts	94	carr<String>; read_elem has no std::string branch yet.
spies2	56	Anonymous container, count 0 in every save — element shape unobservable.
RNG	2	Correctly opaque: the raw MT19937 state block.
Attrib	2	Empty anonymous container in every save.

The test carries a ratchet (pct >= 95.0), so typing a body can never silently regress.

6. Honest limits

• Coverage is measured in items, not bytes, and only over the four saves available.
• Ojvs, aid, Attrib, comms.nmsg, AIEnf.Nas, spymgr.nspy and spy2.rtc/evc/ttc are 0 in every player of every save. Their containers are typed from the recovery; their element shapes are not verified by any save and are marked as such in shapes.h.
• Game::Plague and Game::SpecialProjectImpl are never emitted (NumPlgs2 and NSprj are 0 everywhere), so Plg and Sprj have no ground truth at all. Note the Plg name collision: /Sim/Sys/.../Plg is a plague frame, /Sim/Flt/Ship/Plg is a plain int (always −1).
• The 176 anonymous-tag classes remain usable but nameless; the conformance check aligns them by tag, so a class of all-"." items aligns only as well as its ordering allows.
• sizeof never crossed the channel, so lane D's lower-bound caveat does not apply to anything the engine now believes.

7. Artifacts

• tools/streams.py — the wire projection (objects/streams.json, 386 classes, 2,042 items). uv run python3 tools/streams.py Game::StarShip prints one class readably.
• tools/gen_stream_schema.py — emits sots-engine's include/generated/sots_stream_schema.h.
• sots-engine wip/layouts: src/mars/stream/probe.h (SchemaProbe, CoverageArchive), tests/mars_stream/test_wire_schema.cpp, widened shapes.h.
• ctest 34/34 (was 33/33), tools/clean_room_check.sh OK, test_save skips cleanly with SOTS_SAVES_DIR unset.

8. Open items for the next lane

1. A save with issued turn commands would unblock CD/TurnCommands_v5 — the single biggest remaining blob, and the one that matters for a functional reimplementation of orders.
2. SvSctOb's eight EncObj variants, one shape each, keyed by EncID (3, 4, 5, 9, 17, 10, 20, 1).
3. read_elem needs a std::string branch so carr<std::string> works (DOpts, and any future string list).
4. Re-run tools/streams.py whenever serializers.py improves — the generated header regenerates from it in 0.2 s, and the conformance test will say immediately whether the engine still agrees.
5. save_reader.py still carries all four defects in §3 — Planet.p1 as int, odet as int, and nv / srbd as fields rather than counts. This lane deliberately did not touch it: every fix is byte-neutral on all four saves available, so nothing observable would change, and quietly editing the oracle mid-campaign is worse than recording the divergence. It was left verified as it stands — 36 tests pass, --strict exit 0 on all four saves, and state_checksum.py reports coverage: PROVED (591,376 bytes rebuilt == inflated on turn1; 532,752 on the Zuul save). Whoever owns the reader should apply the four and re-run both suites; the engine is already the corrected reference.
6. The engine's SAVE_FORMAT.md counterpart should record that bool and int items are the same size only when the tag length makes the joint padding agree — that coincidence is what hid the odet defect, and it will not hold for a 3- or 5-character tag.