sots-engine/docs/G-wire-schema.md

G — the wire schema channel

include/generated/sots_stream_schema.h is the second generated channel into this repo, alongside include/generated/sots_addresses.h. Both carry facts about the original binary and nothing else, both are produced by a script in the sots-re notes repo, and neither is ever hand-edited.

sots-re  objects/layouts.json          (the serializer recovery, memory-layout view)
   |     tools/streams.py              -> objects/streams.json   (wire view)
   |     tools/gen_stream_schema.py
   v
sots-engine  include/generated/sots_stream_schema.h

Regenerate after any change on the notes side:

cd ~/sots-re
uv run python3 tools/streams.py
uv run python3 tools/gen_stream_schema.py ../sots-engine/include/generated/sots_stream_schema.h

What crosses, and what deliberately does not

What crosses is the wire schema: for each serializable class, the ordered sequence of items its Write puts on the stream — on-disk tag, on-disk primitive, and how the item is framed.

What does not cross is every memory fact in the recovery: field offsets, sizeof, gaps, container strides. This engine is our own C++, not a byte-for-byte decompilation. It has to read and write the on-disk format faithfully; it must not inherit the original's ABI in its runtime types, and shapes.h is free to lay its structs out however C++ likes.

The shim is the one component that legitimately needs the original's ABI, because it reads the running game's memory. Those offsets have their own home: offset entries in addresses.json, which arrive through sots_addresses.h.

Note the primitive in the table is the disk type. A member the original holds as int16 or int8 is written by Stream::WriteInt and is four bytes on the wire; nothing narrow reaches the stream.

Why it is a check, not a generator

The table is a specification, not a program. The recovery is a linear pass over the game's Write, so:

• It cannot see branches. StarShip writes BQ2 only when hbq is set; the table lists it unconditionally. The sequence is a superset of what any single record contains.
• Container loops are flattened. A count item is followed by its element items as siblings (member == false), not nested inside the container.

A codec driven straight off the table would desynchronise on the first branch. So the hand-written io(Ar&) shapes stay the codec — they express the conditionals and the nesting the binary facts cannot supply — and the table is what proves they agree with the binary.

The two extra archives

src/mars/stream/probe.h adds two archives to the three in archive.h:

• SchemaProbe walks a shape's io() with every branch taken — the same "all branches" view the recovery has — and records the item sequence. tests/mars_stream/test_wire_schema.cpp LCS-aligns that against the generated table. A tag both sides name with different disk primitives is a hard failure; a wire-only item is a conditional or coverage debt and is reported only. Run with SOTS_WIRE_VERBOSE=1 to list every gap.
• CoverageArchive walks a populated shape's io() and separates items a field names (typed) from items only a generic Node carried (opaque). This matters because a byte-identical round trip is not a coverage claim: ar.any and the ar.rest tail round-trip perfectly by copying bytes they do not understand. test_save prints the split and the worst offenders, and carries a ratchet so typing a body can never silently regress.

Adding a shape

1. uv run python3 ~/sots-re/tools/streams.py Game::TheClass — the item list, in disk order.
2. Write the io() in shapes.h. Watch for two things the table shows but does not spell out: an item marked [element] is a loop body, so the item before it is a count, not a field; and an item the game writes conditionally will be listed anyway.
3. Add a check<sh::Shape>("Shape", "Game::TheClass") line to test_wire_schema.cpp.
4. ctest — the conformance test fails on a type disagreement, and test_save fails if the round trip breaks or coverage regresses.

What the table cannot tell you: polymorphic members

Mars::StreamableHelper<Base> is a pointer, and the wire schema only records that a frame goes here — never which derived class wrote it. Game::SVSOSots writes two such lists, each preceded by the key that selects the body (xscn, a scenario name, for xsc; EncID, an int, for EncObj). Those maps are not on the wire at all; they were read out of the game's own factories and are recorded in the notes repo (findings/objects/svsctob-variants.md). The shapes apply the key in both directions — the reader select()s from what it just read, the writer from what it is about to write — so a body goes back out as whatever it came in as, and a key with no shape falls to a generic Node and still round-trips.

The CD custom-data blocks are the same problem one level up, and worse: the key is not even adjacent. CDT lists the block ids and the CD frames follow in that order, so a block's body is selected by the id at the same ordinal in a different frame. SaveGame::io() walks the two in step and select()s from the id's suffix, in both directions; a suffix with no shape (today, .TurnCommands_v5) falls to a generic Node.

Two more things the table's prim column will not save you from:

• A bool and an int item are the same size only when the tag length makes the padding agree. A 4-character tag gives 4 + 4 + 1 → 12 and 4 + 4 + 4 = 12; a 3-character tag gives 8 and 12. That coincidence is what hid odet being a bool for a whole campaign round.
• An empty string is four zero bytes, byte-identical to the int 0, so a string field that is empty in all available data round-trips perfectly while typed as an int. That is what hid Game::SystemParams's name field.
• A container whose count is 0 in every save hides its element type completely. The table names the element class, but the framing is a property of the helper that writes it, not of the element: VectorHelper<Mars::StreamableEnum<T>> writes each element as a frame holding one int, not as a bare int, because StreamableEnum's own writer is a nested object. SysMem, mts and nalat were all typed as int arrays on the strength of the decorated name and all three were wrong in the same way — invisibly, because none of them has ever had an element.