# G — the wire schema channel

`include/generated/sots_stream_schema.h` is the second generated channel into this repo, alongside
`include/generated/sots_addresses.h`. Both carry **facts about the original binary and nothing
else**, both are produced by a script in the `sots-re` notes repo, and neither is ever hand-edited.

```
sots-re  objects/layouts.json          (the serializer recovery, memory-layout view)
   |     tools/streams.py              -> objects/streams.json   (wire view)
   |     tools/gen_stream_schema.py
   v
sots-engine  include/generated/sots_stream_schema.h
```

Regenerate after any change on the notes side:

```sh
cd ~/sots-re
uv run python3 tools/streams.py
uv run python3 tools/gen_stream_schema.py ../sots-engine/include/generated/sots_stream_schema.h
```

## What crosses, and what deliberately does not

What crosses is the **wire schema**: for each serializable class, the ordered sequence of items its
`Write` puts on the stream — on-disk tag, on-disk primitive, and how the item is framed.

What does **not** cross is every memory fact in the recovery: field offsets, `sizeof`, gaps,
container strides. This engine is our own C++, not a byte-for-byte decompilation. It has to read and
write the on-disk *format* faithfully; it must not inherit the original's ABI in its runtime types,
and `shapes.h` is free to lay its structs out however C++ likes.

The shim is the one component that legitimately needs the original's ABI, because it reads the
running game's memory. Those offsets have their own home: `offset` entries in `addresses.json`,
which arrive through `sots_addresses.h`.

Note the primitive in the table is the **disk** type. A member the original holds as `int16` or
`int8` is written by `Stream::WriteInt` and is four bytes on the wire; nothing narrow reaches the
stream.

## Why it is a check, not a generator

The table is a **specification, not a program**. The recovery is a linear pass over the game's
`Write`, so:

* **It cannot see branches.** `StarShip` writes `BQ2` only when `hbq` is set; the table lists it
  unconditionally. The sequence is a *superset* of what any single record contains.
* **Container loops are flattened.** A count item is followed by its element items as siblings
  (`member == false`), not nested inside the container.

A codec driven straight off the table would desynchronise on the first branch. So the hand-written
`io(Ar&)` shapes stay the codec — they express the conditionals and the nesting the binary facts
cannot supply — and the table is what proves they agree with the binary.

## The two extra archives

`src/mars/stream/probe.h` adds two archives to the three in `archive.h`:

* **`SchemaProbe`** walks a shape's `io()` with **every branch taken** — the same "all branches" view
  the recovery has — and records the item sequence. `tests/mars_stream/test_wire_schema.cpp`
  LCS-aligns that against the generated table. A tag both sides name with different disk primitives
  is a **hard failure**; a wire-only item is a conditional or coverage debt and is reported only.
  Run with `SOTS_WIRE_VERBOSE=1` to list every gap.
* **`CoverageArchive`** walks a *populated* shape's `io()` and separates items a field **names**
  (`typed`) from items only a generic `Node` **carried** (`opaque`). This matters because a
  byte-identical round trip is not a coverage claim: `ar.any` and the `ar.rest` tail round-trip
  perfectly by copying bytes they do not understand. `test_save` prints the split and the worst
  offenders, and carries a ratchet so typing a body can never silently regress.

## Adding a shape

1. `uv run python3 ~/sots-re/tools/streams.py Game::TheClass` — the item list, in disk order.
2. Write the `io()` in `shapes.h`. Watch for two things the table shows but does not spell out: an
   item marked `[element]` is a **loop body**, so the item before it is a **count**, not a field; and
   an item the game writes conditionally will be listed anyway.
3. Add a `check<sh::Shape>("Shape", "Game::TheClass")` line to `test_wire_schema.cpp`.
4. `ctest` — the conformance test fails on a type disagreement, and `test_save` fails if the round
   trip breaks or coverage regresses.

## What the table cannot tell you: polymorphic members

`Mars::StreamableHelper<Base>` is a **pointer**, and the wire schema only records that a frame goes
here — never which derived class wrote it. `Game::SVSOSots` writes two such lists, each preceded by
the key that selects the body (`xscn`, a scenario name, for `xsc`; `EncID`, an int, for `EncObj`).
Those maps are not on the wire at all; they were read out of the game's own factories and are
recorded in the notes repo (`findings/objects/svsctob-variants.md`). The shapes apply the key in
both directions — the reader `select()`s from what it just read, the writer from what it is about
to write — so a body goes back out as whatever it came in as, and a key with no shape falls to a
generic `Node` and still round-trips.

The `CD` custom-data blocks are the same problem one level up, and worse: the key is not even
adjacent. `CDT` lists the block ids and the `CD` frames follow in that order, so a block's body is
selected by the id at the **same ordinal** in a different frame. `SaveGame::io()` walks the two in
step and `select()`s from the id's suffix, in both directions; a suffix with no shape (today,
`.TurnCommands_v5`) falls to a generic `Node`.

Two more things the table's `prim` column will not save you from:

* **A `bool` and an `int` item are the same size only when the tag length makes the padding agree.**
  A 4-character tag gives 4 + 4 + 1 → 12 and 4 + 4 + 4 = 12; a 3-character tag gives 8 and 12. That
  coincidence is what hid `odet` being a bool for a whole campaign round.
* **An empty string is four zero bytes**, byte-identical to the int 0, so a string field that is
  empty in all available data round-trips perfectly while typed as an int. That is what hid
  `Game::SystemParams`'s name field.
* **A container whose count is 0 in every save hides its element type completely.** The table names
  the element class, but the framing is a property of the *helper* that writes it, not of the
  element: `VectorHelper<Mars::StreamableEnum<T>>` writes each element as a **frame** holding one
  int, not as a bare int, because `StreamableEnum`'s own writer is a nested object. `SysMem`, `mts`
  and `nalat` were all typed as int arrays on the strength of the decorated name and all three were
  wrong in the same way — invisibly, because none of them has ever had an element.