5bae56a397
The serializer recovery reaches this repo as a generated wire schema
(include/generated/sots_stream_schema.h, 386 classes / 2042 items): for each
class, the ordered sequence of items its Write puts on the stream. Facts only —
no field offsets, no sizeof, no strides. This engine reads and writes the on-disk
format; it does not inherit the original's memory layout.
The table is a specification, not a program: the recovery is a linear pass over
Write, so it cannot see Write's branches (StarShip's BQ2 is gated on hbq but
listed unconditionally) and it flattens container loops. A codec driven off it
would desynchronise. The hand-written io() shapes stay the codec; SchemaProbe
(probe.h) walks them with every branch taken and test_wire_schema LCS-aligns
that against the table — 56 shapes bound, 657 items matched, 0 mismatches.
Four defects the check found, all invisible to a round-trip test:
- SystemParams field 1 is a string, not an int. It is the empty string in
every save, and an empty string is four zero bytes — byte-identical to the
int 0, so it round-tripped by luck. A named planet would have desynced.
- ObservedTech/ObservedWeapon odet is a bool, not an int. Byte-safe only
because a 4-char tag makes a bool item and an int item both 12 bytes.
- SpeciesRatios nv and ShipRecords srbd are counts, not fields.
CoverageArchive separates items a field names from items a Node merely carries,
because a byte-identical round trip is not a coverage claim. Typed coverage of a
real save goes 37.9% -> 97.1% (97.2/97.2/97.6 on the others) with the round trip
still byte-identical, by typing TechTree (both NumTechs sections), Events,
ShipRecs, sprjs, civr, comms, spy2, spymgr, aid, Ojvs, AIEnf, FNG, trdmgr and
the Des section/gun-bank tree. Ratchet at 95%.
trdmgr resolves a recorded trap: ServerTradeManager's Read/Write really are the
inherited no-op, but the call is virtual and ServerTradeManagerImpl has the real
serializer. Same shape resolves IServerSpyManager -> ServerSpyManager.
ctest 34/34, clean_room_check OK, test_save skips cleanly with SOTS_SAVES_DIR unset.
78 lines
4.1 KiB
Markdown
78 lines
4.1 KiB
Markdown
# G — the wire schema channel
|
|
|
|
`include/generated/sots_stream_schema.h` is the second generated channel into this repo, alongside
|
|
`include/generated/sots_addresses.h`. Both carry **facts about the original binary and nothing
|
|
else**, both are produced by a script in the `sots-re` notes repo, and neither is ever hand-edited.
|
|
|
|
```
|
|
sots-re objects/layouts.json (the serializer recovery, memory-layout view)
|
|
| tools/streams.py -> objects/streams.json (wire view)
|
|
| tools/gen_stream_schema.py
|
|
v
|
|
sots-engine include/generated/sots_stream_schema.h
|
|
```
|
|
|
|
Regenerate after any change on the notes side:
|
|
|
|
```sh
|
|
cd ~/sots-re
|
|
uv run python3 tools/streams.py
|
|
uv run python3 tools/gen_stream_schema.py ../sots-engine/include/generated/sots_stream_schema.h
|
|
```
|
|
|
|
## What crosses, and what deliberately does not
|
|
|
|
What crosses is the **wire schema**: for each serializable class, the ordered sequence of items its
|
|
`Write` puts on the stream — on-disk tag, on-disk primitive, and how the item is framed.
|
|
|
|
What does **not** cross is every memory fact in the recovery: field offsets, `sizeof`, gaps,
|
|
container strides. This engine is our own C++, not a byte-for-byte decompilation. It has to read and
|
|
write the on-disk *format* faithfully; it must not inherit the original's ABI in its runtime types,
|
|
and `shapes.h` is free to lay its structs out however C++ likes.
|
|
|
|
The shim is the one component that legitimately needs the original's ABI, because it reads the
|
|
running game's memory. Those offsets have their own home: `offset` entries in `addresses.json`,
|
|
which arrive through `sots_addresses.h`.
|
|
|
|
Note the primitive in the table is the **disk** type. A member the original holds as `int16` or
|
|
`int8` is written by `Stream::WriteInt` and is four bytes on the wire; nothing narrow reaches the
|
|
stream.
|
|
|
|
## Why it is a check, not a generator
|
|
|
|
The table is a **specification, not a program**. The recovery is a linear pass over the game's
|
|
`Write`, so:
|
|
|
|
* **It cannot see branches.** `StarShip` writes `BQ2` only when `hbq` is set; the table lists it
|
|
unconditionally. The sequence is a *superset* of what any single record contains.
|
|
* **Container loops are flattened.** A count item is followed by its element items as siblings
|
|
(`member == false`), not nested inside the container.
|
|
|
|
A codec driven straight off the table would desynchronise on the first branch. So the hand-written
|
|
`io(Ar&)` shapes stay the codec — they express the conditionals and the nesting the binary facts
|
|
cannot supply — and the table is what proves they agree with the binary.
|
|
|
|
## The two extra archives
|
|
|
|
`src/mars/stream/probe.h` adds two archives to the three in `archive.h`:
|
|
|
|
* **`SchemaProbe`** walks a shape's `io()` with **every branch taken** — the same "all branches" view
|
|
the recovery has — and records the item sequence. `tests/mars_stream/test_wire_schema.cpp`
|
|
LCS-aligns that against the generated table. A tag both sides name with different disk primitives
|
|
is a **hard failure**; a wire-only item is a conditional or coverage debt and is reported only.
|
|
Run with `SOTS_WIRE_VERBOSE=1` to list every gap.
|
|
* **`CoverageArchive`** walks a *populated* shape's `io()` and separates items a field **names**
|
|
(`typed`) from items only a generic `Node` **carried** (`opaque`). This matters because a
|
|
byte-identical round trip is not a coverage claim: `ar.any` and the `ar.rest` tail round-trip
|
|
perfectly by copying bytes they do not understand. `test_save` prints the split and the worst
|
|
offenders, and carries a ratchet so typing a body can never silently regress.
|
|
|
|
## Adding a shape
|
|
|
|
1. `uv run python3 ~/sots-re/tools/streams.py Game::TheClass` — the item list, in disk order.
|
|
2. Write the `io()` in `shapes.h`. Watch for two things the table shows but does not spell out: an
|
|
item marked `[element]` is a **loop body**, so the item before it is a **count**, not a field; and
|
|
an item the game writes conditionally will be listed anyway.
|
|
3. Add a `check<sh::Shape>("Shape", "Game::TheClass")` line to `test_wire_schema.cpp`.
|
|
4. `ctest` — the conformance test fails on a type disagreement, and `test_save` fails if the round
|
|
trip breaks or coverage regresses.
|