2170782a81
SvSctOb is a StreamableHelper<SVScriptObject> -- a polymorphic pointer holding a Game::SVSOSots, which writes two variant lists each dispatched by the key item before it (xscn -> xsc, EncID -> EncObj). Neither map is on the wire; both were read out of the game's factories (see the notes repo). The shapes apply the key in both directions, so a body goes back out as whatever it came in as, and an unmodelled key still round-trips as a Node. 18 new shapes: SVSOSots, the four scenario bodies (traps / crowdefs / indsys / gmtrigger -- indsys really does serialize nothing, its Read and Write are both the shared `ret 4` stub) and the eight encounter bodies the saves exercise. The four factory ids no save carries (7 SystemKiller, 8 PuppetMaster, 14 Locust, 21 Ortgay) are deliberately NOT typed: their serializers are recovered but nothing could check a shape for them. DOpts and SVSOVonNeumann::trev are VectorHelper<Mars::String>, so read_elem / write_elem / SchemaBuilder::carr grew the std::string branch lane G listed as missing. spies2 is VectorHelper<int>: the TYPE is certain from the helper's own decorated name, but the count is 0 in all 28 systems of all four saves, so no element value has ever been observed -- the shape is a hypothesis about behaviour even though it is a fact about type. Same for SysMem and mts. Conformance 56 shapes / 657 items -> 74 / 769, still 0 MISMATCH, and every new binding is 0 wire-only and 0 shape-only. Coverage 97.1/97.2/97.2/97.6 -> 98.0/98.0/98.0/98.4 with the byte-identical round trip preserved; ratchet 95.0 -> 97.5. CD is now the only remaining region of size, and it stays opaque: the recovered 44-item Game::TurnCommands sequence cannot be aligned to the save's 35 items even as a subsequence (item 4 is 8 bytes, so a bool where the recovery says i32; and the 27 trailing ints have only 22 i32 slots to come from), which proves the no-orders diagnosis rather than assuming it. Two unit tests added that need no saves: the string-array element branch (including the empty string, which is four zero bytes and so looks like int 0) and the SvSctOb variant dispatch round trip. ctest 34/34, clean_room_check OK, test_save skips cleanly with SOTS_SAVES_DIR unset. sots_stream_schema.h unchanged: streams.py and gen_stream_schema.py were re-run and the output is byte-identical apart from the provenance line.
98 lines
5.4 KiB
Markdown
98 lines
5.4 KiB
Markdown
# G — the wire schema channel
|
|
|
|
`include/generated/sots_stream_schema.h` is the second generated channel into this repo, alongside
|
|
`include/generated/sots_addresses.h`. Both carry **facts about the original binary and nothing
|
|
else**, both are produced by a script in the `sots-re` notes repo, and neither is ever hand-edited.
|
|
|
|
```
|
|
sots-re objects/layouts.json (the serializer recovery, memory-layout view)
|
|
| tools/streams.py -> objects/streams.json (wire view)
|
|
| tools/gen_stream_schema.py
|
|
v
|
|
sots-engine include/generated/sots_stream_schema.h
|
|
```
|
|
|
|
Regenerate after any change on the notes side:
|
|
|
|
```sh
|
|
cd ~/sots-re
|
|
uv run python3 tools/streams.py
|
|
uv run python3 tools/gen_stream_schema.py ../sots-engine/include/generated/sots_stream_schema.h
|
|
```
|
|
|
|
## What crosses, and what deliberately does not
|
|
|
|
What crosses is the **wire schema**: for each serializable class, the ordered sequence of items its
|
|
`Write` puts on the stream — on-disk tag, on-disk primitive, and how the item is framed.
|
|
|
|
What does **not** cross is every memory fact in the recovery: field offsets, `sizeof`, gaps,
|
|
container strides. This engine is our own C++, not a byte-for-byte decompilation. It has to read and
|
|
write the on-disk *format* faithfully; it must not inherit the original's ABI in its runtime types,
|
|
and `shapes.h` is free to lay its structs out however C++ likes.
|
|
|
|
The shim is the one component that legitimately needs the original's ABI, because it reads the
|
|
running game's memory. Those offsets have their own home: `offset` entries in `addresses.json`,
|
|
which arrive through `sots_addresses.h`.
|
|
|
|
Note the primitive in the table is the **disk** type. A member the original holds as `int16` or
|
|
`int8` is written by `Stream::WriteInt` and is four bytes on the wire; nothing narrow reaches the
|
|
stream.
|
|
|
|
## Why it is a check, not a generator
|
|
|
|
The table is a **specification, not a program**. The recovery is a linear pass over the game's
|
|
`Write`, so:
|
|
|
|
* **It cannot see branches.** `StarShip` writes `BQ2` only when `hbq` is set; the table lists it
|
|
unconditionally. The sequence is a *superset* of what any single record contains.
|
|
* **Container loops are flattened.** A count item is followed by its element items as siblings
|
|
(`member == false`), not nested inside the container.
|
|
|
|
A codec driven straight off the table would desynchronise on the first branch. So the hand-written
|
|
`io(Ar&)` shapes stay the codec — they express the conditionals and the nesting the binary facts
|
|
cannot supply — and the table is what proves they agree with the binary.
|
|
|
|
## The two extra archives
|
|
|
|
`src/mars/stream/probe.h` adds two archives to the three in `archive.h`:
|
|
|
|
* **`SchemaProbe`** walks a shape's `io()` with **every branch taken** — the same "all branches" view
|
|
the recovery has — and records the item sequence. `tests/mars_stream/test_wire_schema.cpp`
|
|
LCS-aligns that against the generated table. A tag both sides name with different disk primitives
|
|
is a **hard failure**; a wire-only item is a conditional or coverage debt and is reported only.
|
|
Run with `SOTS_WIRE_VERBOSE=1` to list every gap.
|
|
* **`CoverageArchive`** walks a *populated* shape's `io()` and separates items a field **names**
|
|
(`typed`) from items only a generic `Node` **carried** (`opaque`). This matters because a
|
|
byte-identical round trip is not a coverage claim: `ar.any` and the `ar.rest` tail round-trip
|
|
perfectly by copying bytes they do not understand. `test_save` prints the split and the worst
|
|
offenders, and carries a ratchet so typing a body can never silently regress.
|
|
|
|
## Adding a shape
|
|
|
|
1. `uv run python3 ~/sots-re/tools/streams.py Game::TheClass` — the item list, in disk order.
|
|
2. Write the `io()` in `shapes.h`. Watch for two things the table shows but does not spell out: an
|
|
item marked `[element]` is a **loop body**, so the item before it is a **count**, not a field; and
|
|
an item the game writes conditionally will be listed anyway.
|
|
3. Add a `check<sh::Shape>("Shape", "Game::TheClass")` line to `test_wire_schema.cpp`.
|
|
4. `ctest` — the conformance test fails on a type disagreement, and `test_save` fails if the round
|
|
trip breaks or coverage regresses.
|
|
|
|
## What the table cannot tell you: polymorphic members
|
|
|
|
`Mars::StreamableHelper<Base>` is a **pointer**, and the wire schema only records that a frame goes
|
|
here — never which derived class wrote it. `Game::SVSOSots` writes two such lists, each preceded by
|
|
the key that selects the body (`xscn`, a scenario name, for `xsc`; `EncID`, an int, for `EncObj`).
|
|
Those maps are not on the wire at all; they were read out of the game's own factories and are
|
|
recorded in the notes repo (`findings/objects/svsctob-variants.md`). The shapes apply the key in
|
|
both directions — the reader `select()`s from what it just read, the writer from what it is about
|
|
to write — so a body goes back out as whatever it came in as, and a key with no shape falls to a
|
|
generic `Node` and still round-trips.
|
|
|
|
Two more things the table's `prim` column will not save you from:
|
|
|
|
* **A `bool` and an `int` item are the same size only when the tag length makes the padding agree.**
|
|
A 4-character tag gives 4 + 4 + 1 → 12 and 4 + 4 + 4 = 12; a 3-character tag gives 8 and 12. That
|
|
coincidence is what hid `odet` being a bool for a whole campaign round.
|
|
* **An empty string is four zero bytes**, byte-identical to the int 0, so a string field that is
|
|
empty in all available data round-trips perfectly while typed as an int. That is what hid
|
|
`Game::SystemParams`'s name field.
|