sots-re/findings/objects/serializer-struct-recovery.md

Automated struct recovery from the Mars::IStreamable serializers

Lane D, 2026-09-08. Static/host only — VM140 was held by lane V and the game was never run.

The campaign had ~1,600 classes mapped by name and a few dozen by layout. This round makes the layout step mechanical: every serializable class in the exe carries an enumeration of its own fields — its Write(Stream&), which walks the members in order with a 4-char name tag — and a decoder for that idiom turns "read one class in an afternoon" into "read them all in 0.35 s".

Tools: tools/rtti_map.py (RTTI → vftables), tools/serializers.py (the recovery), tools/serializers_golden.py (the regression set), tools/serializers_ghidra.py (write-back). Outputs: objects/layouts.json, objects/layouts.md, objects/layouts.h.

---

1. Validation first

uv run python3 tools/serializers.py validate runs three checks, all against answers this campaign already had before the tool existed. Nothing below §1 is claimed until this passes.

A — field offsets and kinds, vs struct-recovery.md §1–§4 and observedtech-append.md

  [OK  ] Game::DiplomacyStats                    14/14  exact
  [OK  ] Game::FlightPlan                         6/6   exact
  [OK  ] Game::FlightPlan::Waypoint               3/3   exact
  [OK  ] Game::IndependenceInfo                   5/5   exact
  [OK  ] Game::MoraleEvent                        6/6   exact
  [OK  ] Game::NodeRoute                          3/3   exact
  [OK  ] Game::ObservedTech                       5/5   exact
  [OK  ] Game::ObservedWeapon                     5/5   exact
  [OK  ] Game::PlayerReport                      11/11  exact
  [OK  ] Game::PopulationGroup                    3/3   exact
  [MISS] Game::ServerPlayer                     101/103 exact  2 missing
  [OK  ] Game::ServerSystem                      79/79  exact
  [OK  ] Game::ShipBuildOrder                     5/5   exact
  [OK  ] Game::StarFleet                         16/16  exact
  [OK  ] Game::StarShip                          23/23  exact
  [OK  ] Game::StarSystem::OutputRates            7/7   exact
  [OK  ] Game::StarSystem::PlayerView            13/13  exact

  fields: 305/307 exact, 2 not recovered, 0 WRONG

Zero wrong. Every offset, every width, every tag the tool emits for these 17 classes matches the hand-recovered tables — including the ones that were traps: ObservedTech +0x24 is not reported as a field (it is the string's trailing _Alval), Bats2/rcex come out as int64 and not int, TShn/ETS/the whole DiplomacyStats block come out int16, ltis comes out int, pswd comes out string, TRM/CstR/CstE/CstT/shrm/RefCap/RepCap/PlayerView.Infra come out float.

The two misses are one failure class, described in §4: a value assembled across a branch (ResTNm writes tech ? tech->name : ""; tnc writes max(member, 1)). A linear pass sees only the last branch's expression. Neither was papered over with a heuristic.

B — sizeof, from the container-stride divides

  [OK  ] Game::DiplomacyStats  got 0x24     [OK  ] Game::ObservedWeapon  got 0x2c
  [OK  ] Game::MoraleEvent     got 0x50     [OK  ] Game::PlayerReport    got 0x30
  [OK  ] Game::ObservedTech    got 0x2c

sizeof(ObservedTech) = 0x2c — the value lane X pinned three ways — falls out mechanically, from a completely different place: VectorHelper<ObservedTech>::Write divides the vector's byte span by the element stride, and MSVC's magic-number sequence for that divide is invertible (M = ceil(2^(32+k)/s)). PopulationGroup and OutputRates are [--] — they have no VectorHelper, so this oracle is silent on them rather than guessing.

C — on-disk tag order, vs save_reader.py

22 of save_reader.py's shapes name the exact ::Write that produces them. For every one, the tool's recovered tag sequence contains the oracle's tag sequence in the same relative order:

  22 shapes agree, 0 disagree

That includes Sys (78 tags), Player (104), CreateParams (25), Ship (22), Slot (18), DipStat (14), PlayerView (14). save_reader.py parses the three real saves --strict clean with 36 tests and lane C's state_checksum.py proves its coverage by re-serialising byte-for-byte, so this is disassembly and the save oracle agreeing independently, at scale.

D — Read/Write cross-check, run on every class

The campaign's own standard ("the Read functions agree on every member address"), automated: the class's Read is extracted the same way and its member offsets compared to Write's, matched by tag name.

  Read/Write cross-check : 437/437 fields agree (0 classes with any disagreement)

Two subtleties had to be handled or this check manufactures false alarms, and both are worth carrying: a tag used twice in one class (ServerPlayer writes both an int Team at 0xac and a PlayerAlliances Team at 0x168) cannot be matched by name, and an anonymous tag (".", the writer passed NULL) carries no identity at all, so positional matching turns one side's under-recovery into a cascade of fake conflicts. Both are excluded, and the classes affected are reported as not-cross-checkable rather than as agreeing.

---

2. How it works

Five things had to be got right; each of the last four was worth 10–170 classes on its own.

The idiom. Writer side, one shape under two calling conventions:

push 0xff                     ; default-value argument
<member expression>           ; lea r,[this+D] | mov r,[this+D] | movzx r,word [this+D]
push r
push <tag>                    ; -> .rdata "otch", "pswd", ...   (or 0 -> "." on disk)
push <stream>  /  mov ecx,<stream>
call WriteString/WriteBool/…  |  call [[stream]+slot]

The push immediately before the tag push is always the member. Stream vftable slots, read off the wrappers rather than assumed: +0x18 string, +0x1c bool, +0x20 float, +0x24 int, +0x28 nested, +0x30 raw bytes (n = 8 → int64).

Sizes come from the kind, never from the offsets touched — lane S's rule. bool 1, int16 2, int/float/enum/handle 4, int64 8, std::string 0x1c, std::vector 0x10, both allocator-last in this build's STL and therefore invisible to any touch-scan.

(a) RTTI gives the owner and the this-adjustment. tools/rtti_map.py walks type-descriptor ← COL.pTypeDescriptor ← vftable[-1] for the whole image: 1,924 type descriptors (exactly the inventory's count), 2,172 COLs, 2,172 vftables. It reproduces the known anchors exactly — ObservedTech vftable 0x00a2439c / COL 0x00a81c78; ServerSystem's IStreamable vftable 0x00a2043c at COL offset +8 with Read 0x0075d4b0 / Write 0x00749630. COL.offset is what turns a decompiled offset into an absolute one, and for ServerPlayer that is +0x3a0, so most of its members are at negative decompiled offsets.

(b) The class hierarchy descriptor is the only honest membership test. A 3-slot vftable is not a serializer test: TacAISquadRule_*, the CSV row parsers and ~100 other classes have three virtuals. Walking COL → ClassHierarchyDescriptor → base class array and requiring Mars::IStreamable in the base list removes them all. This dropped a 116-class "empty" bucket to 11 — i.e. it removed 105 classes that were never serializers, instead of reporting them as failures.

(c) mod=0 memory operands. tools/x86disp.py indexes operands that carry a displacement, so mov eax,[edi] — a member at offset 0 — is invisible to it. For a plain struct with no vptr, offset 0 is a real field: StarSystem::OutputRates.SRt is exactly that, and so is every handle/element read through a bare iterator. Synthesising the mod=0 case recovered SRt and every Flt/GF/SnF/MnF element in ServerSystem.

(d) The pointer-member idiom. HomeSys, PlrID, DesID, FltID, SrnTo are written as member->id, so the pushed value is a load off a pointer that came from a member, three to six instructions and a conditional branch earlier. Remembering where a register's pointer came from (and reading the base's this-ness before retiring the destination — mov edi,[edi+0x1c0] overwrites the this-register with the member it is reading) recovered those five plus ServerSystem::indi.

(e) Sub-writers. Two kinds, both invisible without special handling. A base-class Write (StarMapNode::Write inside ServerSystem::Write, giving Pos) and a private sub-writer of the same class (StrategyServer writes its six id lists in FUN_00794cd0, so PlayerIDs, DesignIDs, SystemIDs, FleetIDs, ShipIDs, TradeIDs are simply absent without it — that was the last disagreement in check C). Both are spliced in at the call site, which preserves disk order. A third kind, a tag-forwarding writer f(stream, name, member) that passes the caller's name through to a stream primitive, is discovered (by spotting a tag push that is [ebp+0xc]) rather than listed by hand, because a hand list is exactly what silently under-reports on 1,600 classes.

Containers. A count argument is never a member; it is (_Mylast - _Myfirst)/stride. Recovering the subtraction gives the container's offset (NumFlts → the vector at 0x16c, NVO → the map _Mysize at 0x278), and inverting the divide gives the element stride — which is sizeof(T).

---

3. Results at scale

uv run python3 tools/serializers.py all — 0.35 s for the whole binary.

tier	classes	member fields	meaning
verified	87	542	every field resolved and sized, and Read agrees on every comparable offset
clean	77	328	every field resolved and sized; no Read-side comparison available
unnamed	176	471	offsets and types resolved, but the writer passes a NULL name so the fields have no on-disk names (combat commands, network messages)
partial	31	341	see §4
empty	15	0	the class's Write writes no tagged field
total	386	1,682

• 58 classes have a sizeof corroborated by a second, independent line of evidence — 45 from a VectorHelper<T> element-stride divide, 13 where the field enumeration's lower bound meets the upper bound implied by every embedding of that type in another class. For the rest the tool reports a lower bound and says so: a serializer enumerates serialised members, so a non-streamed cache or back-pointer at the tail is invisible to it. That is the honest limit of this method and it is stated everywhere the number appears.
• Coverage of the population: 368 non-template classes have Mars::IStreamable in their RTTI base list; 348 of them are covered here, plus 38 POD types (Vector3, OutputRates, …) that have no vftable of their own and are reachable only through a specialised StreamableHelper thunk (push ebp; mov ebp,esp; mov ecx,[ecx+8]; pop ebp; jmp writer).
• This is roughly a 10× increase in classes with a known layout, and it is reproducible from the exe in under a second.

---

4. Failure classes — what defeats the tool, and whether it scales

This is the part that says whether the remaining classes are reachable.

#	failure	classes	why it happens	fixable?
1	anonymous tags	176	the writer passes a NULL name, so the field is "." on disk. Offsets and types are still recovered; only the names are missing, and the Read cross-check cannot run	not by this method — the names do not exist in the binary. These are combat commands and network messages, where the reader is positional
2	nested member of unknown size	19	an inline sub-object whose own type has neither a VectorHelper stride nor a tight embedding bound (TechTree, ShipRecords, FleetLayout)	yes — one constructor or operator new size each
3	container elements only	9	everything the serializer writes is a loop body over a std::map/std::list node, so the offsets are within the node, not the object (BuildQueue, SpyReport's four lists)	partly — needs a node-layout model for the map/list
4	untyped sub-writer	3	a discovered tag-forwarding writer whose argument type could not be classified, so the member's offset is known but its size is not	yes, per sub-writer
5	value built across a branch	2 fields	tech ? tech->name : "", max(member, 1). The pass is linear, so it sees the last branch only	yes, with a basic-block-aware pass — deliberately not patched with a backward-search heuristic, because there is no oracle for the 1,600 classes on which a heuristic would silently be wrong
6	std::map / std::list members	—	recovered as their _Mysize int, not as the container. ServerSystem::NVO comes out as int @0x278; the map itself starts at 0x274	yes, and cheaply: this build's std::map is 0x10 bytes with head@0/size@4, from NVO@0x274 → NVE@0x284
7	runtime-built tag	~1	Mars::ParticleSystem::Write makes six write-side calls and not one tag is a compile-time constant	no
8	stub Write	15	e.g. Game::ServerTradeManager's Read and Write are both the inherited no-op 0x924fb0; Game::ShipDesign::Write makes no stream call at all. These classes are not streamed through IStreamable and something else persists them	n/a — a finding, not a failure

Does it generalise? Yes, with one caveat that matters. Categories 2, 3, 4, 5 and 6 are 64 classes and every one of them is a bounded, mechanical fix — none is a research problem. Category 1 (176 classes, 46% of the population) is a hard limit, but a soft one: those layouts are complete in offsets and types, and only the names are missing, so a reimplementation can use them as-is. Category 7 is genuinely out of reach and is one class.

The caveat: this method covers only what the class serialises. ServerSystem is >= 0x2d8 by enumeration and the campaign's hand table agrees, but neither can see a non-streamed member past the last serialised one. Every sizeof in the output is therefore labelled with how it was obtained, and the 58 corroborated ones are the only ones that should be treated as closed.

---

5. Write-back

• objects/layouts.json — the full machine-readable result (fields, offsets, kinds, tags, nested types, container strides, grade, why, Read/Write agreement).
• objects/layouts.md — the same as tables, one section per class.
• objects/layouts.h — 288 C structs, gaps made explicit as padding, Mars_string (0x1c) and Mars_vector (0x10) declared once with the allocator word in place.
• Ghidra — tools/serializers_ghidra.py structs pushes every struct through parse-c-structure (dependencies first, resumable) — 288 pushed, 0 failed — and … labels names the Write/Read entry points (328 labels, 0 failed). Game_ObservedTech comes back from Ghidra as size: 44.
• ghidra/addresses.json — +201 entries: <Class>_Write / <Class>_Read for all 87 verified-tier classes, and sizeof_<Class> for every corroborated size, each carrying its Read/Write agreement count in the prototype comment. Header regenerated with tools/gen_addresses.py (never hand-edited).

6. Open items for the next lane

1. Game::ShipDesign::Write (0x8747a0) makes no stream call. ShipDesign is an important class and its IStreamable slots are inert — worth knowing how designs actually persist (ShipDesignDef is already verified separately, which may be the whole answer).
2. std::map/std::list node layouts would close failure classes 3 and 6 — ~15 classes.
3. The 19 unsized nested types need one size each; a constructor or operator new argument scan would do it in bulk and would also give sizes for the non-streamed tail.
4. Game::SVSOJewelsOfTheCrown writes the tag JEWELLOCATIONID twice, at +0x8 and +0x10. Not an error in the tool (the Write really does emit it twice) but it means the on-disk record has a duplicated name — a save reader matching that shape by name will get the wrong field.