188 lines
12 KiB
Markdown
188 lines
12 KiB
Markdown
# The wire-schema channel: getting lane D's layouts into the engine
|
||
|
||
Lane G, 2026-09-08. Host/static only — VM140 was held by lane U and the game was never run.
|
||
|
||
Lane D recovered 386 class layouts from the `Mars::IStreamable` serializers. That knowledge lived
|
||
only in the notes repo. This lane built the channel that carries it into `sots-engine`, wired the
|
||
engine's save codec to it, and used it to find and fix four real defects in the reader. Engine
|
||
coverage of a real save went from **37.9 % to 97.1 %** of stream items *typed* (not merely
|
||
round-tripped), with the byte-identical round trip preserved throughout.
|
||
|
||
---
|
||
|
||
## 1. The design choice: a wire schema, not a struct layout
|
||
|
||
`objects/layouts.json` is a **memory-layout** projection and is the wrong shape for a codec. Two
|
||
things are lost in `serializers.py`'s `build()`, and both are the substance of the on-disk format:
|
||
|
||
* **Order.** `Lab.layout()` returns fields in the program order of `Write` (with base-class and
|
||
sub-writer calls spliced in at their call site) — that order *is* the disk order. `build()` then
|
||
does `fields.sort(key=lambda x: (x["off_abs"], x["va"]))`. **89 of the 386 classes** have at least
|
||
one field whose offset order differs from its write order, so for those the published record does
|
||
not describe the stream.
|
||
* **Repetition.** `build()` merges two writes of the same offset into `alt_tags`. That is exactly the
|
||
`SVSOJewelsOfTheCrown` `JEWELLOCATIONID`-twice trap, and the merge silently drops the second item.
|
||
|
||
So this lane added `tools/streams.py`, a second projection of the same recovery that keeps the order
|
||
and the repeats — and **drops every memory fact**: no `off`, no `size`, no `sizeof`, no `gaps`, no
|
||
`strides`. `sots-engine` is our own C++, not a byte-for-byte decomp; it must read and write the
|
||
*format* faithfully and must not inherit the original's ABI in its runtime types. (The shim is the
|
||
one component that legitimately needs the original ABI, because it reads the running game's memory.
|
||
Those few offsets already have a home: `offset` entries in `ghidra/addresses.json`.)
|
||
|
||
The on-disk primitive is **not** `layouts.json`'s `kind`, which is a memory type. It comes from the
|
||
stream vftable slot the writer called, or the wrapper helper it called instead:
|
||
|
||
| slot | helper | disk |
|
||
|---|---|---|
|
||
| `+0x18` | `0x8b9d70` | string |
|
||
| `+0x1c` | `0x8b9c20` | bool (1 byte) |
|
||
| `+0x20` | `0x8b9be0` | float32 |
|
||
| `+0x24` | `0x8b9d50`, `0x8b9d00`, `0x816490` | **int32** |
|
||
| `+0x28` | — | nested frame |
|
||
| `+0x30` | `0x8b9c60` | raw (`n = 8` → int64) |
|
||
|
||
A member the original holds as `int16` or `int8` is written by `WriteInt` and is **four bytes on the
|
||
wire**. `0x8b9d00` is a distinct `int16` wrapper that widens; `0x816490` is
|
||
`Stream::WriteNetworkObjectId`, an id, also int32. Nothing narrow ever reaches the stream.
|
||
|
||
`tools/gen_stream_schema.py` then emits `include/generated/sots_stream_schema.h` — a sibling of
|
||
`gen_addresses.py` under the same discipline: generated, provenance header, never hand-edited,
|
||
regenerated on every merge. 386 classes, 2,042 wire items.
|
||
|
||
## 2. Why it is a *check*, not a generator
|
||
|
||
**The recovered schema is a specification, not a program.** Two properties make a codec driven
|
||
straight off it desynchronise on real data:
|
||
|
||
* **It is unconditional.** The recovery is a linear pass over `Write` and cannot see `Write`'s
|
||
branches. `StarShip` writes `BQ2` only when `hbq` is set, and `pop`/`ppop` only when `hsp` is set;
|
||
the table lists all three unconditionally. The sequence is a **superset** of any single record.
|
||
* **Container loops are flattened.** A count item is followed by its element items as siblings, not
|
||
nested inside the container. `member == false` marks the elements, and that is the only signal.
|
||
|
||
The engine's hand-written `io(Ar&)` shapes therefore stay the codec — they can express the
|
||
conditionals and the nesting that the binary facts cannot supply — and the generated table is what
|
||
**proves they agree with the binary, item for item, in order**. That is the same relationship
|
||
`sots_addresses.h` already has with the shim: it supplies facts that hand-written code consumes; it
|
||
does not generate code.
|
||
|
||
`sots-engine` grew a fourth archive, `SchemaProbe` (`src/mars/stream/probe.h`), which walks a shape's
|
||
`io()` with **every branch taken** — the same "all branches" view the recovery has — and records the
|
||
item sequence. `tests/mars_stream/test_wire_schema.cpp` LCS-aligns that against the generated table.
|
||
**56 shapes bound, 657 items matched, 0 mismatches.** A tag both sides name with different disk
|
||
primitives is a hard build failure; a wire-only item is reported, not fatal.
|
||
|
||
## 3. What the check found
|
||
|
||
Four defects, all present in **both** `save_reader.py` and the engine, none of which any round-trip
|
||
test could have caught:
|
||
|
||
1. **`SystemParams` field 1 is a `std::string`, not an int.** Both readers modelled it as `int`. It is
|
||
the **empty string in every save available, and an empty string is four zero bytes** — byte-identical
|
||
to the int `0`. A save carrying a non-empty name here would have desynchronised both parsers.
|
||
The engine now reads it as a string.
|
||
2. **`ObservedTech::odet` / `ObservedWeapon::odet` are `bool`, not int.** The binary calls the bool
|
||
writer and lane D's own golden table already said `bool` at `+0x08`; `save_reader.py` and the
|
||
engine both said int. Byte-safe here **only by coincidence**: with a 4-char tag a bool item and an
|
||
int item both occupy 12 bytes and the bool's three pad bytes are zeroed. A shorter tag would not
|
||
have been so forgiving. 68 `otch` elements per save exercised this and never showed it.
|
||
3. **`SpeciesRatios::nv` is a count, not a field.** The recovery flags `sp` and `va2` as loop-body
|
||
writes; the saves agree (`nv==1` frames carry one `(sp, va2)` pair, `nv==0` frames carry nothing).
|
||
4. **`ShipRecords::srbd` is a count, not a field** — the second of two counted sections
|
||
(`srd, src, srb, srl, sri`, note no `srk`). Values 0, 1, 3 and 4 occur across the players available.
|
||
|
||
Applying all four left the byte-identical round trip intact on all four real saves.
|
||
|
||
## 4. Two recorded traps resolved
|
||
|
||
* **`Game::ServerTradeManager`'s Read/Write really are the inherited no-op — and the `trdmgr` frame
|
||
really does have content.** The resolution is that the call is **virtual**: the object is a
|
||
`Game::ServerTradeManagerImpl`, and *that* class has a real serializer emitting exactly
|
||
`NumTradeSectors` / `TradeID` / `Trade` / `SctSize`, with `Game::ServerTradeSector` giving the
|
||
sector body (14 items). The same shape resolves `Game::IServerSpyManager` → `Game::ServerSpyManager`.
|
||
**General rule for the next lane: when an interface type's serializer is the inherited stub, look
|
||
for the concrete `*Impl` / non-`I` class.**
|
||
* **`Game::ShipDesign::Write` makes no stream call**, as recorded — and that is still true. But the
|
||
`Des` frame's *section* body is `Game::ShipDesignDef::Section`, which is recovered
|
||
(`DSec` frame → `Game::ShipSectionID`, `DGbnk2` carr, `DOpts` carr), so most of the design record
|
||
is typeable anyway. Only `DW2`, `Dtc` and `Dwgv` have no recovered serializer and came from the saves.
|
||
|
||
## 5. Coverage
|
||
|
||
A byte-identical round trip is **not** a coverage claim: the engine reached it partly by typing
|
||
fields and partly by carrying whole bodies as generic `Node`s, which round-trip trivially because
|
||
they are copied verbatim. `CoverageArchive` (also in `probe.h`) separates the two by walking a
|
||
*populated* shape's `io()` and counting items a field names against items only a `Node` carried.
|
||
|
||
| save | before | after |
|
||
|---|---|---|
|
||
| turn1-state | 37.9 % | **97.1 %** |
|
||
| turn2-state | 38.8 % | **97.2 %** |
|
||
| turn3-state | 39.4 % | **97.2 %** |
|
||
| zuul-turn5-species5 | 42.5 % | **97.6 %** |
|
||
|
||
The bodies typed this round, all against the recovered schema: `TechTree` (both `NumTechs` sections —
|
||
the second is the per-tech state, whose field list and types come from `Game::SpyReportTechTree`,
|
||
which streams the same record, and which is where `Tfc` being a bool came from), `Events`
|
||
(`EventStorage` → `TurnEvents` → `Event`, three nesting levels), `ShipRecs`, `sprjs`, `civr`/`spe`,
|
||
`comms`, `spy2`, `spymgr`, `aid`, `Ojvs`, `AIEnf`, `FNG`, `trdmgr`, and the `Des` section /
|
||
gun-bank tree. `DW2` is branch-gated on `bID`: `wid` (an id) when set, `wfn` (a family name) when clear.
|
||
|
||
**What is still opaque, and why** (turn1 numbers, of 36,644 items):
|
||
|
||
| region | items | why |
|
||
|---|---|---|
|
||
| `CD` custom-data blocks | 744 | `TurnCommands_v5` is 35 anonymous scalars with 27 trailing zeros — a *no-orders* snapshot. Typing it needs a save with issued turn commands. |
|
||
| `SvSctOb` | 147 | Scenario/encounter objects: 8 `EncObj` variants keyed by `EncID`, structure known but each needs its own shape. |
|
||
| `DOpts` | 94 | `carr<String>`; `read_elem` has no `std::string` branch yet. |
|
||
| `spies2` | 56 | Anonymous container, count 0 in every save — element shape unobservable. |
|
||
| `RNG` | 2 | Correctly opaque: the raw MT19937 state block. |
|
||
| `Attrib` | 2 | Empty anonymous container in every save. |
|
||
|
||
The test carries a **ratchet** (`pct >= 95.0`), so typing a body can never silently regress.
|
||
|
||
## 6. Honest limits
|
||
|
||
* Coverage is measured in **items**, not bytes, and only over the four saves available.
|
||
* `Ojvs`, `aid`, `Attrib`, `comms.nmsg`, `AIEnf.Nas`, `spymgr.nspy` and `spy2.rtc/evc/ttc` are **0 in
|
||
every player of every save**. Their containers are typed from the recovery; their *element* shapes
|
||
are not verified by any save and are marked as such in `shapes.h`.
|
||
* `Game::Plague` and `Game::SpecialProjectImpl` are never emitted (`NumPlgs2` and `NSprj` are 0
|
||
everywhere), so `Plg` and `Sprj` have no ground truth at all. Note the `Plg` name collision:
|
||
`/Sim/Sys/.../Plg` is a plague frame, `/Sim/Flt/Ship/Plg` is a plain int (always −1).
|
||
* The 176 anonymous-tag classes remain usable but nameless; the conformance check aligns them by
|
||
tag, so a class of all-`"."` items aligns only as well as its ordering allows.
|
||
* `sizeof` never crossed the channel, so lane D's lower-bound caveat does not apply to anything the
|
||
engine now believes.
|
||
|
||
## 7. Artifacts
|
||
|
||
* `tools/streams.py` — the wire projection (`objects/streams.json`, 386 classes, 2,042 items).
|
||
`uv run python3 tools/streams.py Game::StarShip` prints one class readably.
|
||
* `tools/gen_stream_schema.py` — emits `sots-engine`'s `include/generated/sots_stream_schema.h`.
|
||
* `sots-engine` `wip/layouts`: `src/mars/stream/probe.h` (`SchemaProbe`, `CoverageArchive`),
|
||
`tests/mars_stream/test_wire_schema.cpp`, widened `shapes.h`.
|
||
* ctest **34/34** (was 33/33), `tools/clean_room_check.sh` **OK**, `test_save` skips cleanly with
|
||
`SOTS_SAVES_DIR` unset.
|
||
|
||
## 8. Open items for the next lane
|
||
|
||
1. **A save with issued turn commands** would unblock `CD`/`TurnCommands_v5` — the single biggest
|
||
remaining blob, and the one that matters for a functional reimplementation of orders.
|
||
2. `SvSctOb`'s eight `EncObj` variants, one shape each, keyed by `EncID` (3, 4, 5, 9, 17, 10, 20, 1).
|
||
3. `read_elem` needs a `std::string` branch so `carr<std::string>` works (`DOpts`, and any future
|
||
string list).
|
||
4. Re-run `tools/streams.py` whenever `serializers.py` improves — the generated header regenerates
|
||
from it in 0.2 s, and the conformance test will say immediately whether the engine still agrees.
|
||
5. **`save_reader.py` still carries all four defects in §3** — `Planet.p1` as int, `odet` as int, and
|
||
`nv` / `srbd` as fields rather than counts. This lane deliberately did **not** touch it: every fix
|
||
is byte-neutral on all four saves available, so nothing observable would change, and quietly
|
||
editing the oracle mid-campaign is worse than recording the divergence. It was left verified as it
|
||
stands — 36 tests pass, `--strict` exit 0 on all four saves, and `state_checksum.py` reports
|
||
`coverage: PROVED` (591,376 bytes rebuilt == inflated on turn1; 532,752 on the Zuul save). Whoever
|
||
owns the reader should apply the four and re-run both suites; the engine is already the corrected
|
||
reference.
|
||
6. The engine's `SAVE_FORMAT.md` counterpart should record that `bool` and `int` items are the same
|
||
size **only** when the tag length makes the joint padding agree — that coincidence is what hid the
|
||
`odet` defect, and it will not hold for a 3- or 5-character tag.
|