sots-re/findings/objects/serializer-struct-recovery.md

# Automated struct recovery from the `Mars::IStreamable` serializers

Lane D, 2026-09-08. Static/host only — VM140 was held by lane V and the game was never run.

The campaign had ~1,600 classes mapped by name and a few dozen by layout. This round makes the
layout step mechanical: every serializable class in the exe carries an **enumeration** of its own
fields — its `Write(Stream&)`, which walks the members in order with a 4-char name tag — and a
decoder for that idiom turns "read one class in an afternoon" into "read them all in 0.35 s".

Tools: `tools/rtti_map.py` (RTTI → vftables), `tools/serializers.py` (the recovery),
`tools/serializers_golden.py` (the regression set), `tools/serializers_ghidra.py` (write-back).
Outputs: `objects/layouts.json`, `objects/layouts.md`, `objects/layouts.h`.

---

## 1. Validation first

`uv run python3 tools/serializers.py validate` runs three checks, all against answers this
campaign already had **before** the tool existed. Nothing below §1 is claimed until this passes.

### A — field offsets and kinds, vs `struct-recovery.md` §1–§4 and `observedtech-append.md`

```
  [OK  ] Game::DiplomacyStats                    14/14  exact
  [OK  ] Game::FlightPlan                         6/6   exact
  [OK  ] Game::FlightPlan::Waypoint               3/3   exact
  [OK  ] Game::IndependenceInfo                   5/5   exact
  [OK  ] Game::MoraleEvent                        6/6   exact
  [OK  ] Game::NodeRoute                          3/3   exact
  [OK  ] Game::ObservedTech                       5/5   exact
  [OK  ] Game::ObservedWeapon                     5/5   exact
  [OK  ] Game::PlayerReport                      11/11  exact
  [OK  ] Game::PopulationGroup                    3/3   exact
  [MISS] Game::ServerPlayer                     101/103 exact  2 missing
  [OK  ] Game::ServerSystem                      79/79  exact
  [OK  ] Game::ShipBuildOrder                     5/5   exact
  [OK  ] Game::StarFleet                         16/16  exact
  [OK  ] Game::StarShip                          23/23  exact
  [OK  ] Game::StarSystem::OutputRates            7/7   exact
  [OK  ] Game::StarSystem::PlayerView            13/13  exact

  fields: 305/307 exact, 2 not recovered, 0 WRONG
```

**Zero wrong.** Every offset, every width, every tag the tool emits for these 17 classes matches the
hand-recovered tables — including the ones that were traps: `ObservedTech` `+0x24` is *not* reported
as a field (it is the string's trailing `_Alval`), `Bats2`/`rcex` come out as int64 and not int,
`TShn`/`ETS`/the whole `DiplomacyStats` block come out int16, `ltis` comes out int, `pswd` comes out
string, `TRM`/`CstR`/`CstE`/`CstT`/`shrm`/`RefCap`/`RepCap`/`PlayerView.Infra` come out float.

The two misses are one failure class, described in §4: a value assembled **across a branch**
(`ResTNm` writes `tech ? tech->name : ""`; `tnc` writes `max(member, 1)`). A linear pass sees only
the last branch's expression. Neither was papered over with a heuristic.

### B — `sizeof`, from the container-stride divides

```
  [OK  ] Game::DiplomacyStats  got 0x24     [OK  ] Game::ObservedWeapon  got 0x2c
  [OK  ] Game::MoraleEvent     got 0x50     [OK  ] Game::PlayerReport    got 0x30
  [OK  ] Game::ObservedTech    got 0x2c
```

`sizeof(ObservedTech) = 0x2c` — the value lane X pinned three ways — falls out mechanically, from a
completely different place: `VectorHelper<ObservedTech>::Write` divides the vector's byte span by
the element stride, and MSVC's magic-number sequence for that divide is invertible
(`M = ceil(2^(32+k)/s)`). `PopulationGroup` and `OutputRates` are `[--]` — they have no
`VectorHelper`, so this oracle is silent on them rather than guessing.

### C — on-disk tag order, vs `save_reader.py`

22 of `save_reader.py`'s shapes name the exact `::Write` that produces them. For every one, the
tool's recovered tag sequence contains the oracle's tag sequence **in the same relative order**:

```
  22 shapes agree, 0 disagree
```

That includes `Sys` (78 tags), `Player` (104), `CreateParams` (25), `Ship` (22), `Slot` (18),
`DipStat` (14), `PlayerView` (14). `save_reader.py` parses the three real saves `--strict` clean
with 36 tests and lane C's `state_checksum.py` proves its coverage by re-serialising byte-for-byte,
so this is disassembly and the save oracle agreeing independently, at scale.

### D — Read/Write cross-check, run on every class

The campaign's own standard ("the Read functions agree on every member address"), automated: the
class's `Read` is extracted the same way and its member offsets compared to `Write`'s, matched by
tag name.

```
  Read/Write cross-check : 437/437 fields agree (0 classes with any disagreement)
```

Two subtleties had to be handled or this check manufactures false alarms, and both are worth
carrying: a tag used **twice** in one class (`ServerPlayer` writes both an `int Team` at `0xac` and
a `PlayerAlliances Team` at `0x168`) cannot be matched by name, and an **anonymous** tag (`"."`,
the writer passed NULL) carries no identity at all, so positional matching turns one side's
under-recovery into a cascade of fake conflicts. Both are excluded, and the classes affected are
reported as not-cross-checkable rather than as agreeing.

---

## 2. How it works

Five things had to be got right; each of the last four was worth 10–170 classes on its own.

**The idiom.** Writer side, one shape under two calling conventions:

```
push 0xff                     ; default-value argument
<member expression>           ; lea r,[this+D] | mov r,[this+D] | movzx r,word [this+D]
push r
push <tag>                    ; -> .rdata "otch", "pswd", ...   (or 0 -> "." on disk)
push <stream>  /  mov ecx,<stream>
call WriteString/WriteBool/…  |  call [[stream]+slot]
```

The push **immediately before the tag push** is always the member. Stream vftable slots, read off
the wrappers rather than assumed: `+0x18` string, `+0x1c` bool, `+0x20` float, `+0x24` int,
`+0x28` nested, `+0x30` raw bytes (`n = 8` → int64).

**Sizes come from the kind, never from the offsets touched** — lane S's rule. `bool` 1, `int16` 2,
int/float/enum/handle 4, `int64` 8, `std::string` **0x1c**, `std::vector` **0x10**, both
*allocator-last* in this build's STL and therefore invisible to any touch-scan.

**(a) RTTI gives the owner and the this-adjustment.** `tools/rtti_map.py` walks
type-descriptor ← `COL.pTypeDescriptor` ← `vftable[-1]` for the whole image: 1,924 type
descriptors (exactly the inventory's count), 2,172 COLs, 2,172 vftables. It reproduces the known
anchors exactly — `ObservedTech` vftable `0x00a2439c` / COL `0x00a81c78`; `ServerSystem`'s
IStreamable vftable `0x00a2043c` at COL offset **+8** with Read `0x0075d4b0` / Write `0x00749630`.
`COL.offset` is what turns a decompiled offset into an absolute one, and for `ServerPlayer` that is
`+0x3a0`, so most of its members are at *negative* decompiled offsets.

**(b) The class hierarchy descriptor is the only honest membership test.** A 3-slot vftable is
*not* a serializer test: `TacAISquadRule_*`, the CSV row parsers and ~100 other classes have three
virtuals. Walking `COL → ClassHierarchyDescriptor → base class array` and requiring
`Mars::IStreamable` in the base list removes them all. This dropped a 116-class "empty" bucket to
11 — i.e. it removed 105 classes that were never serializers, instead of reporting them as failures.

**(c) `mod=0` memory operands.** `tools/x86disp.py` indexes operands that carry a *displacement*, so
`mov eax,[edi]` — a member at offset **0** — is invisible to it. For a plain struct with no vptr,
offset 0 is a real field: `StarSystem::OutputRates.SRt` is exactly that, and so is every
`handle`/element read through a bare iterator. Synthesising the mod=0 case recovered `SRt` and every
`Flt`/`GF`/`SnF`/`MnF` element in `ServerSystem`.

**(d) The pointer-member idiom.** `HomeSys`, `PlrID`, `DesID`, `FltID`, `SrnTo` are written as
`member->id`, so the pushed value is a load off a *pointer that came from* a member, three to six
instructions and a conditional branch earlier. Remembering where a register's pointer came from
(and reading the base's this-ness **before** retiring the destination — `mov edi,[edi+0x1c0]`
overwrites the this-register with the member it is reading) recovered those five plus
`ServerSystem::indi`.

**(e) Sub-writers.** Two kinds, both invisible without special handling. A **base-class Write**
(`StarMapNode::Write` inside `ServerSystem::Write`, giving `Pos`) and a **private sub-writer of the
same class** (`StrategyServer` writes its six id lists in `FUN_00794cd0`, so `PlayerIDs`,
`DesignIDs`, `SystemIDs`, `FleetIDs`, `ShipIDs`, `TradeIDs` are simply absent without it — that was
the last disagreement in check C). Both are spliced in at the call site, which preserves disk order.
A third kind, a **tag-forwarding** writer `f(stream, name, member)` that passes the caller's name
through to a stream primitive, is *discovered* (by spotting a tag push that is `[ebp+0xc]`) rather
than listed by hand, because a hand list is exactly what silently under-reports on 1,600 classes.

**Containers.** A count argument is never a member; it is `(_Mylast - _Myfirst)/stride`. Recovering
the subtraction gives the container's offset (`NumFlts` → the `vector` at `0x16c`, `NVO` → the map
`_Mysize` at `0x278`), and inverting the divide gives the element stride — which is `sizeof(T)`.

---

## 3. Results at scale

`uv run python3 tools/serializers.py all` — 0.35 s for the whole binary.

| tier | classes | member fields | meaning |
|---|---|---|---|
| **verified** | 87 | 542 | every field resolved and sized, and `Read` agrees on every comparable offset |
| **clean** | 77 | 328 | every field resolved and sized; no `Read`-side comparison available |
| **unnamed** | 176 | 471 | offsets and types resolved, but the writer passes a NULL name so the fields have no on-disk names (combat commands, network messages) |
| **partial** | 31 | 341 | see §4 |
| **empty** | 15 | 0 | the class's `Write` writes no tagged field |
| **total** | **386** | **1,682** | |

* **58 classes have a `sizeof` corroborated by a second, independent line of evidence** — 45 from a
  `VectorHelper<T>` element-stride divide, 13 where the field enumeration's lower bound meets the
  upper bound implied by every embedding of that type in another class. For the rest the tool
  reports a **lower bound** and says so: a serializer enumerates *serialised* members, so a
  non-streamed cache or back-pointer at the tail is invisible to it. That is the honest limit of
  this method and it is stated everywhere the number appears.
* Coverage of the population: 368 non-template classes have `Mars::IStreamable` in their RTTI base
  list; 348 of them are covered here, plus 38 POD types (`Vector3`, `OutputRates`, …) that have no
  vftable of their own and are reachable only through a specialised `StreamableHelper` thunk
  (`push ebp; mov ebp,esp; mov ecx,[ecx+8]; pop ebp; jmp writer`).
* This is roughly a **10× increase** in classes with a known layout, and it is reproducible from the
  exe in under a second.

---

## 4. Failure classes — what defeats the tool, and whether it scales

This is the part that says whether the remaining classes are reachable.

| # | failure | classes | why it happens | fixable? |
|---|---|---|---|---|
| 1 | **anonymous tags** | 176 | the writer passes a NULL name, so the field is `"."` on disk. Offsets and types are still recovered; only the *names* are missing, and the Read cross-check cannot run | not by this method — the names do not exist in the binary. These are combat commands and network messages, where the reader is positional |
| 2 | **nested member of unknown size** | 19 | an inline sub-object whose own type has neither a `VectorHelper` stride nor a tight embedding bound (`TechTree`, `ShipRecords`, `FleetLayout`) | yes — one constructor or `operator new` size each |
| 3 | **container elements only** | 9 | everything the serializer writes is a loop body over a `std::map`/`std::list` node, so the offsets are within the node, not the object (`BuildQueue`, `SpyReport`'s four lists) | partly — needs a node-layout model for the map/list |
| 4 | **untyped sub-writer** | 3 | a discovered tag-forwarding writer whose argument type could not be classified, so the member's offset is known but its size is not | yes, per sub-writer |
| 5 | **value built across a branch** | 2 fields | `tech ? tech->name : ""`, `max(member, 1)`. The pass is linear, so it sees the last branch only | yes, with a basic-block-aware pass — deliberately **not** patched with a backward-search heuristic, because there is no oracle for the 1,600 classes on which a heuristic would silently be wrong |
| 6 | **`std::map` / `std::list` members** | — | recovered as their `_Mysize` int, not as the container. `ServerSystem::NVO` comes out as `int @0x278`; the map itself starts at `0x274` | yes, and cheaply: this build's `std::map` is 0x10 bytes with head@0/size@4, from `NVO@0x274` → `NVE@0x284` |
| 7 | **runtime-built tag** | ~1 | `Mars::ParticleSystem::Write` makes six write-side calls and not one tag is a compile-time constant | no |
| 8 | **stub `Write`** | 15 | e.g. `Game::ServerTradeManager`'s Read and Write are both the inherited no-op `0x924fb0`; `Game::ShipDesign::Write` makes no stream call at all. These classes are not streamed through `IStreamable` and something else persists them | n/a — a finding, not a failure |

**Does it generalise?** Yes, with one caveat that matters. Categories 2, 3, 4, 5 and 6 are 64
classes and every one of them is a bounded, mechanical fix — none is a research problem. Category 1
(176 classes, 46% of the population) is a hard limit, but a *soft* one: those layouts are complete
in offsets and types, and only the names are missing, so a reimplementation can use them as-is.
Category 7 is genuinely out of reach and is one class.

The caveat: **this method covers only what the class serialises.** `ServerSystem` is `>= 0x2d8` by
enumeration and the campaign's hand table agrees, but neither can see a non-streamed member past the
last serialised one. Every `sizeof` in the output is therefore labelled with how it was obtained,
and the 58 corroborated ones are the only ones that should be treated as closed.

---

## 5. Write-back

* **`objects/layouts.json`** — the full machine-readable result (fields, offsets, kinds, tags,
  nested types, container strides, grade, why, Read/Write agreement).
* **`objects/layouts.md`** — the same as tables, one section per class.
* **`objects/layouts.h`** — 288 C structs, gaps made explicit as padding, `Mars_string` (0x1c) and
  `Mars_vector` (0x10) declared once with the allocator word in place.
* **Ghidra** — `tools/serializers_ghidra.py structs` pushes every struct through
  `parse-c-structure` (dependencies first, resumable) — **288 pushed, 0 failed** — and `… labels`
  names the `Write`/`Read` entry points (**328 labels, 0 failed**). `Game_ObservedTech` comes back from Ghidra as `size: 44`.
* **`ghidra/addresses.json`** — +201 entries: `<Class>_Write` / `<Class>_Read` for all 87
  verified-tier classes, and `sizeof_<Class>` for every corroborated size, each carrying its
  Read/Write agreement count in the prototype comment. Header regenerated with
  `tools/gen_addresses.py` (never hand-edited).

## 6. Open items for the next lane

1. **`Game::ShipDesign::Write` (`0x8747a0`) makes no stream call.** `ShipDesign` is an important
   class and its IStreamable slots are inert — worth knowing how designs actually persist
   (`ShipDesignDef` is already verified separately, which may be the whole answer).
2. **`std::map`/`std::list` node layouts** would close failure classes 3 and 6 — ~15 classes.
3. **The 19 unsized nested types** need one size each; a constructor or `operator new` argument
   scan would do it in bulk and would also give sizes for the non-streamed tail.
4. `Game::SVSOJewelsOfTheCrown` writes the tag `JEWELLOCATIONID` **twice**, at `+0x8` and `+0x10`.
   Not an error in the tool (the Write really does emit it twice) but it means the on-disk record
   has a duplicated name — a save reader matching that shape by name will get the wrong field.