# Automated struct recovery from the `Mars::IStreamable` serializers Lane D, 2026-09-08. Static/host only — VM140 was held by lane V and the game was never run. The campaign had ~1,600 classes mapped by name and a few dozen by layout. This round makes the layout step mechanical: every serializable class in the exe carries an **enumeration** of its own fields — its `Write(Stream&)`, which walks the members in order with a 4-char name tag — and a decoder for that idiom turns "read one class in an afternoon" into "read them all in 0.35 s". Tools: `tools/rtti_map.py` (RTTI → vftables), `tools/serializers.py` (the recovery), `tools/serializers_golden.py` (the regression set), `tools/serializers_ghidra.py` (write-back). Outputs: `objects/layouts.json`, `objects/layouts.md`, `objects/layouts.h`. --- ## 1. Validation first `uv run python3 tools/serializers.py validate` runs three checks, all against answers this campaign already had **before** the tool existed. Nothing below §1 is claimed until this passes. ### A — field offsets and kinds, vs `struct-recovery.md` §1–§4 and `observedtech-append.md` ``` [OK ] Game::DiplomacyStats 14/14 exact [OK ] Game::FlightPlan 6/6 exact [OK ] Game::FlightPlan::Waypoint 3/3 exact [OK ] Game::IndependenceInfo 5/5 exact [OK ] Game::MoraleEvent 6/6 exact [OK ] Game::NodeRoute 3/3 exact [OK ] Game::ObservedTech 5/5 exact [OK ] Game::ObservedWeapon 5/5 exact [OK ] Game::PlayerReport 11/11 exact [OK ] Game::PopulationGroup 3/3 exact [MISS] Game::ServerPlayer 101/103 exact 2 missing [OK ] Game::ServerSystem 79/79 exact [OK ] Game::ShipBuildOrder 5/5 exact [OK ] Game::StarFleet 16/16 exact [OK ] Game::StarShip 23/23 exact [OK ] Game::StarSystem::OutputRates 7/7 exact [OK ] Game::StarSystem::PlayerView 13/13 exact fields: 305/307 exact, 2 not recovered, 0 WRONG ``` **Zero wrong.** Every offset, every width, every tag the tool emits for these 17 classes matches the hand-recovered tables — including the ones that were traps: `ObservedTech` `+0x24` is *not* reported as a field (it is the string's trailing `_Alval`), `Bats2`/`rcex` come out as int64 and not int, `TShn`/`ETS`/the whole `DiplomacyStats` block come out int16, `ltis` comes out int, `pswd` comes out string, `TRM`/`CstR`/`CstE`/`CstT`/`shrm`/`RefCap`/`RepCap`/`PlayerView.Infra` come out float. The two misses are one failure class, described in §4: a value assembled **across a branch** (`ResTNm` writes `tech ? tech->name : ""`; `tnc` writes `max(member, 1)`). A linear pass sees only the last branch's expression. Neither was papered over with a heuristic. ### B — `sizeof`, from the container-stride divides ``` [OK ] Game::DiplomacyStats got 0x24 [OK ] Game::ObservedWeapon got 0x2c [OK ] Game::MoraleEvent got 0x50 [OK ] Game::PlayerReport got 0x30 [OK ] Game::ObservedTech got 0x2c ``` `sizeof(ObservedTech) = 0x2c` — the value lane X pinned three ways — falls out mechanically, from a completely different place: `VectorHelper::Write` divides the vector's byte span by the element stride, and MSVC's magic-number sequence for that divide is invertible (`M = ceil(2^(32+k)/s)`). `PopulationGroup` and `OutputRates` are `[--]` — they have no `VectorHelper`, so this oracle is silent on them rather than guessing. ### C — on-disk tag order, vs `save_reader.py` 22 of `save_reader.py`'s shapes name the exact `::Write` that produces them. For every one, the tool's recovered tag sequence contains the oracle's tag sequence **in the same relative order**: ``` 22 shapes agree, 0 disagree ``` That includes `Sys` (78 tags), `Player` (104), `CreateParams` (25), `Ship` (22), `Slot` (18), `DipStat` (14), `PlayerView` (14). `save_reader.py` parses the three real saves `--strict` clean with 36 tests and lane C's `state_checksum.py` proves its coverage by re-serialising byte-for-byte, so this is disassembly and the save oracle agreeing independently, at scale. ### D — Read/Write cross-check, run on every class The campaign's own standard ("the Read functions agree on every member address"), automated: the class's `Read` is extracted the same way and its member offsets compared to `Write`'s, matched by tag name. ``` Read/Write cross-check : 437/437 fields agree (0 classes with any disagreement) ``` Two subtleties had to be handled or this check manufactures false alarms, and both are worth carrying: a tag used **twice** in one class (`ServerPlayer` writes both an `int Team` at `0xac` and a `PlayerAlliances Team` at `0x168`) cannot be matched by name, and an **anonymous** tag (`"."`, the writer passed NULL) carries no identity at all, so positional matching turns one side's under-recovery into a cascade of fake conflicts. Both are excluded, and the classes affected are reported as not-cross-checkable rather than as agreeing. --- ## 2. How it works Five things had to be got right; each of the last four was worth 10–170 classes on its own. **The idiom.** Writer side, one shape under two calling conventions: ``` push 0xff ; default-value argument ; lea r,[this+D] | mov r,[this+D] | movzx r,word [this+D] push r push ; -> .rdata "otch", "pswd", ... (or 0 -> "." on disk) push / mov ecx, call WriteString/WriteBool/… | call [[stream]+slot] ``` The push **immediately before the tag push** is always the member. Stream vftable slots, read off the wrappers rather than assumed: `+0x18` string, `+0x1c` bool, `+0x20` float, `+0x24` int, `+0x28` nested, `+0x30` raw bytes (`n = 8` → int64). **Sizes come from the kind, never from the offsets touched** — lane S's rule. `bool` 1, `int16` 2, int/float/enum/handle 4, `int64` 8, `std::string` **0x1c**, `std::vector` **0x10**, both *allocator-last* in this build's STL and therefore invisible to any touch-scan. **(a) RTTI gives the owner and the this-adjustment.** `tools/rtti_map.py` walks type-descriptor ← `COL.pTypeDescriptor` ← `vftable[-1]` for the whole image: 1,924 type descriptors (exactly the inventory's count), 2,172 COLs, 2,172 vftables. It reproduces the known anchors exactly — `ObservedTech` vftable `0x00a2439c` / COL `0x00a81c78`; `ServerSystem`'s IStreamable vftable `0x00a2043c` at COL offset **+8** with Read `0x0075d4b0` / Write `0x00749630`. `COL.offset` is what turns a decompiled offset into an absolute one, and for `ServerPlayer` that is `+0x3a0`, so most of its members are at *negative* decompiled offsets. **(b) The class hierarchy descriptor is the only honest membership test.** A 3-slot vftable is *not* a serializer test: `TacAISquadRule_*`, the CSV row parsers and ~100 other classes have three virtuals. Walking `COL → ClassHierarchyDescriptor → base class array` and requiring `Mars::IStreamable` in the base list removes them all. This dropped a 116-class "empty" bucket to 11 — i.e. it removed 105 classes that were never serializers, instead of reporting them as failures. **(c) `mod=0` memory operands.** `tools/x86disp.py` indexes operands that carry a *displacement*, so `mov eax,[edi]` — a member at offset **0** — is invisible to it. For a plain struct with no vptr, offset 0 is a real field: `StarSystem::OutputRates.SRt` is exactly that, and so is every `handle`/element read through a bare iterator. Synthesising the mod=0 case recovered `SRt` and every `Flt`/`GF`/`SnF`/`MnF` element in `ServerSystem`. **(d) The pointer-member idiom.** `HomeSys`, `PlrID`, `DesID`, `FltID`, `SrnTo` are written as `member->id`, so the pushed value is a load off a *pointer that came from* a member, three to six instructions and a conditional branch earlier. Remembering where a register's pointer came from (and reading the base's this-ness **before** retiring the destination — `mov edi,[edi+0x1c0]` overwrites the this-register with the member it is reading) recovered those five plus `ServerSystem::indi`. **(e) Sub-writers.** Two kinds, both invisible without special handling. A **base-class Write** (`StarMapNode::Write` inside `ServerSystem::Write`, giving `Pos`) and a **private sub-writer of the same class** (`StrategyServer` writes its six id lists in `FUN_00794cd0`, so `PlayerIDs`, `DesignIDs`, `SystemIDs`, `FleetIDs`, `ShipIDs`, `TradeIDs` are simply absent without it — that was the last disagreement in check C). Both are spliced in at the call site, which preserves disk order. A third kind, a **tag-forwarding** writer `f(stream, name, member)` that passes the caller's name through to a stream primitive, is *discovered* (by spotting a tag push that is `[ebp+0xc]`) rather than listed by hand, because a hand list is exactly what silently under-reports on 1,600 classes. **Containers.** A count argument is never a member; it is `(_Mylast - _Myfirst)/stride`. Recovering the subtraction gives the container's offset (`NumFlts` → the `vector` at `0x16c`, `NVO` → the map `_Mysize` at `0x278`), and inverting the divide gives the element stride — which is `sizeof(T)`. --- ## 3. Results at scale `uv run python3 tools/serializers.py all` — 0.35 s for the whole binary. | tier | classes | member fields | meaning | |---|---|---|---| | **verified** | 87 | 542 | every field resolved and sized, and `Read` agrees on every comparable offset | | **clean** | 77 | 328 | every field resolved and sized; no `Read`-side comparison available | | **unnamed** | 176 | 471 | offsets and types resolved, but the writer passes a NULL name so the fields have no on-disk names (combat commands, network messages) | | **partial** | 31 | 341 | see §4 | | **empty** | 15 | 0 | the class's `Write` writes no tagged field | | **total** | **386** | **1,682** | | * **58 classes have a `sizeof` corroborated by a second, independent line of evidence** — 45 from a `VectorHelper` element-stride divide, 13 where the field enumeration's lower bound meets the upper bound implied by every embedding of that type in another class. For the rest the tool reports a **lower bound** and says so: a serializer enumerates *serialised* members, so a non-streamed cache or back-pointer at the tail is invisible to it. That is the honest limit of this method and it is stated everywhere the number appears. * Coverage of the population: 368 non-template classes have `Mars::IStreamable` in their RTTI base list; 348 of them are covered here, plus 38 POD types (`Vector3`, `OutputRates`, …) that have no vftable of their own and are reachable only through a specialised `StreamableHelper` thunk (`push ebp; mov ebp,esp; mov ecx,[ecx+8]; pop ebp; jmp writer`). * This is roughly a **10× increase** in classes with a known layout, and it is reproducible from the exe in under a second. --- ## 4. Failure classes — what defeats the tool, and whether it scales This is the part that says whether the remaining classes are reachable. | # | failure | classes | why it happens | fixable? | |---|---|---|---|---| | 1 | **anonymous tags** | 176 | the writer passes a NULL name, so the field is `"."` on disk. Offsets and types are still recovered; only the *names* are missing, and the Read cross-check cannot run | not by this method — the names do not exist in the binary. These are combat commands and network messages, where the reader is positional | | 2 | **nested member of unknown size** | 19 | an inline sub-object whose own type has neither a `VectorHelper` stride nor a tight embedding bound (`TechTree`, `ShipRecords`, `FleetLayout`) | yes — one constructor or `operator new` size each | | 3 | **container elements only** | 9 | everything the serializer writes is a loop body over a `std::map`/`std::list` node, so the offsets are within the node, not the object (`BuildQueue`, `SpyReport`'s four lists) | partly — needs a node-layout model for the map/list | | 4 | **untyped sub-writer** | 3 | a discovered tag-forwarding writer whose argument type could not be classified, so the member's offset is known but its size is not | yes, per sub-writer | | 5 | **value built across a branch** | 2 fields | `tech ? tech->name : ""`, `max(member, 1)`. The pass is linear, so it sees the last branch only | yes, with a basic-block-aware pass — deliberately **not** patched with a backward-search heuristic, because there is no oracle for the 1,600 classes on which a heuristic would silently be wrong | | 6 | **`std::map` / `std::list` members** | — | recovered as their `_Mysize` int, not as the container. `ServerSystem::NVO` comes out as `int @0x278`; the map itself starts at `0x274` | yes, and cheaply: this build's `std::map` is 0x10 bytes with head@0/size@4, from `NVO@0x274` → `NVE@0x284` | | 7 | **runtime-built tag** | ~1 | `Mars::ParticleSystem::Write` makes six write-side calls and not one tag is a compile-time constant | no | | 8 | **stub `Write`** | 15 | e.g. `Game::ServerTradeManager`'s Read and Write are both the inherited no-op `0x924fb0`; `Game::ShipDesign::Write` makes no stream call at all. These classes are not streamed through `IStreamable` and something else persists them | n/a — a finding, not a failure | **Does it generalise?** Yes, with one caveat that matters. Categories 2, 3, 4, 5 and 6 are 64 classes and every one of them is a bounded, mechanical fix — none is a research problem. Category 1 (176 classes, 46% of the population) is a hard limit, but a *soft* one: those layouts are complete in offsets and types, and only the names are missing, so a reimplementation can use them as-is. Category 7 is genuinely out of reach and is one class. The caveat: **this method covers only what the class serialises.** `ServerSystem` is `>= 0x2d8` by enumeration and the campaign's hand table agrees, but neither can see a non-streamed member past the last serialised one. Every `sizeof` in the output is therefore labelled with how it was obtained, and the 58 corroborated ones are the only ones that should be treated as closed. --- ## 5. Write-back * **`objects/layouts.json`** — the full machine-readable result (fields, offsets, kinds, tags, nested types, container strides, grade, why, Read/Write agreement). * **`objects/layouts.md`** — the same as tables, one section per class. * **`objects/layouts.h`** — 288 C structs, gaps made explicit as padding, `Mars_string` (0x1c) and `Mars_vector` (0x10) declared once with the allocator word in place. * **Ghidra** — `tools/serializers_ghidra.py structs` pushes every struct through `parse-c-structure` (dependencies first, resumable) — **288 pushed, 0 failed** — and `… labels` names the `Write`/`Read` entry points (**328 labels, 0 failed**). `Game_ObservedTech` comes back from Ghidra as `size: 44`. * **`ghidra/addresses.json`** — +201 entries: `_Write` / `_Read` for all 87 verified-tier classes, and `sizeof_` for every corroborated size, each carrying its Read/Write agreement count in the prototype comment. Header regenerated with `tools/gen_addresses.py` (never hand-edited). ## 6. Open items for the next lane 1. **`Game::ShipDesign::Write` (`0x8747a0`) makes no stream call.** `ShipDesign` is an important class and its IStreamable slots are inert — worth knowing how designs actually persist (`ShipDesignDef` is already verified separately, which may be the whole answer). 2. **`std::map`/`std::list` node layouts** would close failure classes 3 and 6 — ~15 classes. 3. **The 19 unsized nested types** need one size each; a constructor or `operator new` argument scan would do it in bulk and would also give sizes for the non-streamed tail. 4. `Game::SVSOJewelsOfTheCrown` writes the tag `JEWELLOCATIONID` **twice**, at `+0x8` and `+0x10`. Not an error in the tool (the Write really does emit it twice) but it means the on-disk record has a duplicated name — a save reader matching that shape by name will get the wrong field.