8e45b43638
Game::StrategyAIAgent::Streamable and the ten shapes under it. The whole writer is unconditional -- the branch the decompiler shows around lnat is an inlined vector destructor whose operator delete is marked noreturn, and both paths converge -- so the recovered sequence and a single record are the same sequence, and all 36 items match with 0 wire-only and 0 shape-only. CD blocks are now selected by the CDT id at the same ordinal, in both directions; the one .TurnCommands_v5 block per save still falls to a Node. Also: Sim's Attrib was not an empty frame, it was an AttribMap holding a count of 0, and typing it closes those two items too. And StreamableEnum<T> writes a frame containing one int, not a bare int, so SysMem/mts/nalat are arrays of one-int frames -- byte-neutral, since all three have count 0 in every save, but the previous typing was wrong. Conformance 74 shapes/769 items -> 86/838, still 0 MISMATCH. Round trip byte-identical on all four saves; ratchet 97.5 -> 99.8. Every container that is empty in all four saves is named as such in the notes; the new unit test populates each one, since nothing else exercises them.
110 lines
6.4 KiB
Markdown
110 lines
6.4 KiB
Markdown
# G — the wire schema channel
|
|
|
|
`include/generated/sots_stream_schema.h` is the second generated channel into this repo, alongside
|
|
`include/generated/sots_addresses.h`. Both carry **facts about the original binary and nothing
|
|
else**, both are produced by a script in the `sots-re` notes repo, and neither is ever hand-edited.
|
|
|
|
```
|
|
sots-re objects/layouts.json (the serializer recovery, memory-layout view)
|
|
| tools/streams.py -> objects/streams.json (wire view)
|
|
| tools/gen_stream_schema.py
|
|
v
|
|
sots-engine include/generated/sots_stream_schema.h
|
|
```
|
|
|
|
Regenerate after any change on the notes side:
|
|
|
|
```sh
|
|
cd ~/sots-re
|
|
uv run python3 tools/streams.py
|
|
uv run python3 tools/gen_stream_schema.py ../sots-engine/include/generated/sots_stream_schema.h
|
|
```
|
|
|
|
## What crosses, and what deliberately does not
|
|
|
|
What crosses is the **wire schema**: for each serializable class, the ordered sequence of items its
|
|
`Write` puts on the stream — on-disk tag, on-disk primitive, and how the item is framed.
|
|
|
|
What does **not** cross is every memory fact in the recovery: field offsets, `sizeof`, gaps,
|
|
container strides. This engine is our own C++, not a byte-for-byte decompilation. It has to read and
|
|
write the on-disk *format* faithfully; it must not inherit the original's ABI in its runtime types,
|
|
and `shapes.h` is free to lay its structs out however C++ likes.
|
|
|
|
The shim is the one component that legitimately needs the original's ABI, because it reads the
|
|
running game's memory. Those offsets have their own home: `offset` entries in `addresses.json`,
|
|
which arrive through `sots_addresses.h`.
|
|
|
|
Note the primitive in the table is the **disk** type. A member the original holds as `int16` or
|
|
`int8` is written by `Stream::WriteInt` and is four bytes on the wire; nothing narrow reaches the
|
|
stream.
|
|
|
|
## Why it is a check, not a generator
|
|
|
|
The table is a **specification, not a program**. The recovery is a linear pass over the game's
|
|
`Write`, so:
|
|
|
|
* **It cannot see branches.** `StarShip` writes `BQ2` only when `hbq` is set; the table lists it
|
|
unconditionally. The sequence is a *superset* of what any single record contains.
|
|
* **Container loops are flattened.** A count item is followed by its element items as siblings
|
|
(`member == false`), not nested inside the container.
|
|
|
|
A codec driven straight off the table would desynchronise on the first branch. So the hand-written
|
|
`io(Ar&)` shapes stay the codec — they express the conditionals and the nesting the binary facts
|
|
cannot supply — and the table is what proves they agree with the binary.
|
|
|
|
## The two extra archives
|
|
|
|
`src/mars/stream/probe.h` adds two archives to the three in `archive.h`:
|
|
|
|
* **`SchemaProbe`** walks a shape's `io()` with **every branch taken** — the same "all branches" view
|
|
the recovery has — and records the item sequence. `tests/mars_stream/test_wire_schema.cpp`
|
|
LCS-aligns that against the generated table. A tag both sides name with different disk primitives
|
|
is a **hard failure**; a wire-only item is a conditional or coverage debt and is reported only.
|
|
Run with `SOTS_WIRE_VERBOSE=1` to list every gap.
|
|
* **`CoverageArchive`** walks a *populated* shape's `io()` and separates items a field **names**
|
|
(`typed`) from items only a generic `Node` **carried** (`opaque`). This matters because a
|
|
byte-identical round trip is not a coverage claim: `ar.any` and the `ar.rest` tail round-trip
|
|
perfectly by copying bytes they do not understand. `test_save` prints the split and the worst
|
|
offenders, and carries a ratchet so typing a body can never silently regress.
|
|
|
|
## Adding a shape
|
|
|
|
1. `uv run python3 ~/sots-re/tools/streams.py Game::TheClass` — the item list, in disk order.
|
|
2. Write the `io()` in `shapes.h`. Watch for two things the table shows but does not spell out: an
|
|
item marked `[element]` is a **loop body**, so the item before it is a **count**, not a field; and
|
|
an item the game writes conditionally will be listed anyway.
|
|
3. Add a `check<sh::Shape>("Shape", "Game::TheClass")` line to `test_wire_schema.cpp`.
|
|
4. `ctest` — the conformance test fails on a type disagreement, and `test_save` fails if the round
|
|
trip breaks or coverage regresses.
|
|
|
|
## What the table cannot tell you: polymorphic members
|
|
|
|
`Mars::StreamableHelper<Base>` is a **pointer**, and the wire schema only records that a frame goes
|
|
here — never which derived class wrote it. `Game::SVSOSots` writes two such lists, each preceded by
|
|
the key that selects the body (`xscn`, a scenario name, for `xsc`; `EncID`, an int, for `EncObj`).
|
|
Those maps are not on the wire at all; they were read out of the game's own factories and are
|
|
recorded in the notes repo (`findings/objects/svsctob-variants.md`). The shapes apply the key in
|
|
both directions — the reader `select()`s from what it just read, the writer from what it is about
|
|
to write — so a body goes back out as whatever it came in as, and a key with no shape falls to a
|
|
generic `Node` and still round-trips.
|
|
|
|
The `CD` custom-data blocks are the same problem one level up, and worse: the key is not even
|
|
adjacent. `CDT` lists the block ids and the `CD` frames follow in that order, so a block's body is
|
|
selected by the id at the **same ordinal** in a different frame. `SaveGame::io()` walks the two in
|
|
step and `select()`s from the id's suffix, in both directions; a suffix with no shape (today,
|
|
`.TurnCommands_v5`) falls to a generic `Node`.
|
|
|
|
Two more things the table's `prim` column will not save you from:
|
|
|
|
* **A `bool` and an `int` item are the same size only when the tag length makes the padding agree.**
|
|
A 4-character tag gives 4 + 4 + 1 → 12 and 4 + 4 + 4 = 12; a 3-character tag gives 8 and 12. That
|
|
coincidence is what hid `odet` being a bool for a whole campaign round.
|
|
* **An empty string is four zero bytes**, byte-identical to the int 0, so a string field that is
|
|
empty in all available data round-trips perfectly while typed as an int. That is what hid
|
|
`Game::SystemParams`'s name field.
|
|
* **A container whose count is 0 in every save hides its element type completely.** The table names
|
|
the element class, but the framing is a property of the *helper* that writes it, not of the
|
|
element: `VectorHelper<Mars::StreamableEnum<T>>` writes each element as a **frame** holding one
|
|
int, not as a bare int, because `StreamableEnum`'s own writer is a nested object. `SysMem`, `mts`
|
|
and `nalat` were all typed as int arrays on the strength of the decorated name and all three were
|
|
wrong in the same way — invisibly, because none of them has ever had an element.
|