From 3fcfd7f522deef42874e9d76da7cf5c7543fc788 Mon Sep 17 00:00:00 2001 From: alex Date: Tue, 8 Sep 2026 05:55:27 -0400 Subject: [PATCH] board: lane D mass struct recovery; fragment-dir structural fix --- campaign/board.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/campaign/board.md b/campaign/board.md index bd9693a..01f5830 100644 --- a/campaign/board.md +++ b/campaign/board.md @@ -99,3 +99,8 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`). | serializer struct recovery (automated) | objects | verified | high | 90% | 2026-09-08 | **Lane D: struct recovery is now MECHANICAL.** `tools/serializers.py` decodes the `Mars::IStreamable` `Write` idiom for the whole binary in **0.35 s**; `tools/rtti_map.py` walks type-descriptor <- COL <- vftable[-1] (1,924 TDs = the inventory's count, 2,172 COLs/vftables) to attribute each serializer to its class and its COL this-adjustment. **VALIDATION RUN BEFORE ANY NEW CLAIM, four ways: (A) 305/307 field offsets+kinds exact across 17 already-known classes, ZERO WRONG** vs struct-recovery.md 1-4 (ServerSystem 79/79, StarShip 23/23, StarFleet 16/16, ServerPlayer 101/103, PlayerView 13/13, DiplomacyStats 14/14 - including every trap: ObservedTech+0x24 correctly NOT a field, Bats2/rcex int64, TShn/ETS/DipStats int16, ltis int, pswd string, TRM/CstR/E/T/shrm/RefCap/RepCap/PlayerView.Infra float); **(B) sizeof read off the VectorHelper element-stride divide** (invert MSVC's magic `M = ceil(2^(32+k)/s)`) reproduces ObservedTech **0x2c** independently of lane X's three proofs, plus MoraleEvent 0x50, PlayerReport 0x30, DiplomacyStats 0x24, ObservedDesign 0x10; **(C) 22 of save_reader.py's shapes, recovered tag order identical, 22 agree / 0 disagree** (Sys 78 tags, Player 104, CreateParams 25, Ship 22, Slot 18); **(D) automated Read/Write cross-check on every class, 437/437 field offsets agree**. SCALE: **386 classes with a Write, 1,682 member fields** - verified 87 (542 fields) / clean 77 (328) / unnamed 176 (471) / partial 31 / empty 15; **58 sizeofs corroborated** by a second line of evidence, the rest reported as LOWER BOUNDS and labelled as such (a serializer enumerates only *serialised* members - a non-streamed tail is invisible). ~10x more classes with a known layout, reproducible from the exe in under a second. FOUR ENABLERS each worth 10-170 classes: (1) the RTTI **ClassHierarchyDescriptor** is the only honest "is this an IStreamable" test - a 3-slot vftable also matches TacAISquadRule_*, the CSV row parsers and ~100 others, and using it dropped a bogus 116-class "empty" bucket to 11; (2) **mod=0 memory operands** carry no displacement so `tools/x86disp.py` cannot index them - every field at offset 0 was invisible (OutputRates.SRt, every container element read through a bare iterator); (3) the **member->id pointer idiom** behind every handle field (HomeSys/PlrID/DesID/FltID/SrnTo) - and the base's this-ness must be read BEFORE retiring the destination, because `mov edi,[edi+0x1c0]` overwrites the this-register with the member it is reading; (4) **sub-writers**, base-class (StarMapNode inside ServerSystem = `Pos`) and private-same-class (StrategyServer's six id lists live in FUN_00794cd0 - without it PlayerIDs/DesignIDs/SystemIDs/FleetIDs/ShipIDs/TradeIDs are simply absent), spliced at the call site so disk order survives. FAILURE CLASSES, enumerated: 176 anonymous-tag classes (writer passes NULL, so offsets+types are complete but there are NO on-disk names - a hard limit on names, not on layout; combat commands + network messages); 19 unsized nested types; 9 map/list-node-only; 3 untyped sub-writers; 2 fields lost to a value assembled ACROSS A BRANCH (`tech ? tech->name : ""`, `max(v,1)`) - **left unrecovered on purpose rather than patched with a backward-search heuristic that has no oracle on the other 1,600 classes**; std::map/list members recovered only as their `_Mysize` int; 1 runtime-built tag (Mars::ParticleSystem). Everything but the first is a bounded mechanical fix. WRITE-BACK: 288 structures + 328 labels into Ghidra (0 failures), +201 addresses.json entries, header regenerated with gen_addresses.py. `findings/objects/serializer-struct-recovery.md`, `objects/layouts.{json,md,h}` | | Game::ShipDesign is NOT streamed through IStreamable | objects | mapped | high | 100% | 2026-09-08 | Lane D, from the serializer sweep: `ShipDesign`'s IStreamable slots are inert - `Write` 0x008747a0 makes no stream call at all. Same for `Game::ServerTradeManager` (Read and Write are BOTH the inherited no-op 0x924fb0) and 13 others incl. the whole `Mars::Particle*` family and `Mars::DecalFile`. So designs persist by some other path (`ShipDesignDef` is already verified separately and may be the whole answer) - worth confirming before anyone goes looking for a ShipDesign serializer that does not exist | | SVSOJewelsOfTheCrown writes a DUPLICATE tag | objects | mapped | high | 100% | 2026-09-08 | Lane D: `SVSOJewelsOfTheCrown::Write` emits the tag `JEWELLOCATIONID` TWICE, at +0x8 and +0x10. Not a tool error - the Write really does it. Any save reader matching that shape BY NAME will bind the wrong field. Same shape as ServerPlayer's two `Team` members (int @0xac, PlayerAlliances @0x168), which is why the Read/Write cross-check has to exclude duplicated tags rather than report them as conflicts | +| serializer struct recovery (mass) | objects | verified | high | 95% | 2026-09-08 | Lane D industrialised lane S's enumeration rule: the ~179 Streamable serializers ARE complete field enumerations with 4-char tags. 386 classes with a Write, 1,682 member fields, whole binary in 0.35 s. Verified 87 (542 fields) / clean 77 (328) / unnamed 176 (471) / partial 31 (341) / empty 15 - roughly 10x more classes with a known layout. VALIDATED FOUR WAYS BEFORE ANY NEW CLAIM: (A) 305/307 fields exact vs layouts we already had, 2 unrecovered, ZERO WRONG, every known trap reproduced incl. ObservedTech+0x24 correctly NOT a field; (B) sizeof independently inverted from MSVC's container-stride magic M=ceil(2^(32+k)/s) - ObservedTech 0x2c falls out mechanically, plus MoraleEvent 0x50, PlayerReport 0x30, DiplomacyStats 0x24; (C) 22 on-disk tag orders vs save_reader.py, 22 agree 0 disagree; (D) automated Read/Write cross-check, 437/437 offsets agree, 0 conflicts. 58 sizeofs corroborated by a second line of evidence; the REST ARE REPORTED AS LOWER BOUNDS AND LABELLED - a serializer enumerates only SERIALISED members, so a non-streamed tail is invisible. Ghidra: 288 structures + 328 labels, 0 failures | +| serializer recovery: failure classes | objects | mapped | high | 100% | 2026-09-08 | What defeats the tool, i.e. whether it scales to the remaining ~1,600 classes. 176 anonymous tags (writer passes NULL - offsets/types complete, no on-disk names; combat commands + network messages; NOT fixable, the names do not exist in the binary, layouts still usable). 19 nested member of unknown size (fixable, one ctor size each). 9 map/list node bodies. 3 untyped sub-writer. 2 fields VALUE ASSEMBLED ACROSS A BRANCH (`tech ? tech->name : ""`, `max(v,1)`) - DELIBERATELY left unrecovered rather than patched with a backward-search heuristic that has no oracle on the other 1,600 classes. 1 runtime-built tag (Mars::ParticleSystem, not fixable). Four enablers each worth 10-170 classes: RTTI ClassHierarchyDescriptor as the only honest "is this IStreamable" test (a 3-slot vftable also matches TacAISquadRule_* and the row parsers; using it dropped a bogus 116-class "empty" bucket to 11), mod=0 memory operands (x86disp.py cannot index them; they hide every field at offset 0), the member->id pointer idiom behind every handle field, and sub-writers (StrategyServer's six id lists live in FUN_00794cd0) | +| Game::ShipDesign has NO serializer | objects | verified | high | 100% | 2026-09-08 | Lane D: Game::ShipDesign::Write (0x008747a0) MAKES NO STREAM CALL, and ServerTradeManager's Read and Write are both the inherited no-op. Designs persist some other way. Recorded so nobody hunts for a serializer that does not exist | +| SVSOJewelsOfTheCrown duplicate tag | objects | verified | med | 100% | 2026-09-08 | Its Write emits the tag JEWELLOCATIONID TWICE (+0x8 and +0x10). A name-matching save reader binds the WRONG field. Ours must bind by ordinal here, not by name | +| addresses.json fragment dir (structural fix) | meta | verified | high | 100% | 2026-09-08 | Three cross-lane sweeps today all traced to ONE shared file. `ghidra/addresses.d/.json` fragments are now merged by gen_addresses.py after addresses.json, in sorted order; a DUPLICATE NAME ACROSS FILES IS A HARD ERROR (exit 1), never last-wins, because two lanes disagreeing about an address is precisely what must not be papered over. Self-tested both ways (duplicate exit=1, clean exit=0). Integrator folds fragments back into addresses.json once a lane merges. This replaces the "stage by path" rule with a design that does not depend on remembering it |