From f965c8c76900ede64d5fd92be0b7c3bbc297d107 Mon Sep 17 00:00:00 2001 From: alex Date: Tue, 8 Sep 2026 05:09:33 -0400 Subject: [PATCH] lane S: std::string is 0x1c binary-wide; ObservedTech element fully mapped Settles the 0x18-vs-0x1c contradiction lane X raised. 0x1c is right, everywhere, and there is exactly one std::string instantiation in this binary: _Bx@0, _Mysize@0x10, _Myres@0x14, _Alval@0x18. ObservedTech+0x24 is that string's trailing empty-allocator word, not the unaccounted data field it was read as. Three complete enumerations of the element each skip it: ObservedTech::Write 0x00817cf0, the ctor 0x008562a0, and the copy ctor inlined at 0x0079a184. Generalised with a new scanner, tools/strfootprint.py, which recovers every (base, disp, tag) handed to the Mars::Stream string helpers: 65 std::string members off a non-stack base across every serializer in the exe, ZERO with a sibling member inside the 0x1c span, and 51 of the 52 measurable inter-member gaps exactly 0x1c. Corroborated by the vector walk stride (add esi,0x1c @0x00699c29), PostEvent's by-value strings at [ebp+8]/[ebp+0x24] with RET 0x4c, and MoraleEvent 0x50 = name@0x34 + 0x1c. Blast radius: zero recovered struct tables were wrong. Every string-bearing layout already used 0x1c spans and 0x1c gaps -- ServerPlayer::pswd @0x2dc..0x2f7, the row flagged for re-checking, included. Only prose carried the 0x18 number: the loader-prototypes conventions line, the GlobalConst_ParseString prototype, and the ObservedTech element table. struct-recovery S0 additionally had _Mysize/_Myres transposed (size@0x14, res@0x18) while every table in the same file used the correct offsets; fixed. ObservedTech's four on-disk fields are now mapped rather than guessed, by reading the serializer as lane X suggested: +0x04 uint16 otnF, +0x06 uint16 otnL, +0x08 bool odet (ONE BYTE, WriteBool), +0x0c std::string otch (0x1c), +0x28 int owith = 0x2c exactly. That matches save_reader.py's on-disk order already. Game:: ObservedWeapon (0x00817bc0/0x00817b10) is the same element with tag owep. Oracles unaffected and re-run: save_reader 36/36 and --strict exit 0 on all three real saves; state_checksum 38 tests OK, coverage PROVED byte-for-byte on turn1 and turn3. sots-engine wip/strings 32d3e36 syncs the header and corrects two stale "unpinned" comments: clean_room_check OK, host ctest 33/33. Standing rule this produced: never size a struct member from the offsets the code touches. This build's STL puts the empty allocator LAST in both string (0x1c) and vector (0x10), and an empty allocator is never loaded or stored, so a touch-scan undercounts by exactly 4 every time. Size from an enumeration instead. --- campaign/open-questions.md | 2 +- findings/objects/struct-recovery.md | 39 +++- findings/subsystems/loader-prototypes.md | 4 +- findings/subsystems/observedtech-append.md | 169 ++++++++++++++--- ghidra/addresses.json | 92 ++++++++- ghidra/generated/sots_addresses.h | 32 +++- guides/re-windows-2000s-howto.md | 2 +- tools/strfootprint.py | 210 +++++++++++++++++++++ verify/save-reader/SAVE_FORMAT.md | 14 ++ 9 files changed, 526 insertions(+), 38 deletions(-) create mode 100644 tools/strfootprint.py diff --git a/campaign/open-questions.md b/campaign/open-questions.md index 19c3658..330c87c 100644 --- a/campaign/open-questions.md +++ b/campaign/open-questions.md @@ -38,7 +38,7 @@ Each links to the finding that raised it. Promoted to backlog or closed by **re- - **SAVE_FORMAT tag corrections (fix Python reader + spec)** — real on-disk tags: `otnF` (not `ontF`) in Odes/Owep/Otch, `nextid` (not `nextId`) in NdGr2, Design = `FAIDes/DHide/DWep/DName`; `ords`/`wpts` are real tags. Python's positional R() matching hid these. RNG: float mapping `(float)(y*2^-32)`, `next_int` mask, and lazy-vs-eager twist at `left==0` still need binary confirmation. (from [[mars-stream]]) - **RESOLVED: tech `allows` default** — unlisted species = 1.0 (confirmed in `FUN_005822d0` tree-creation roll, strategic-turn-internals §2); `game/data` uses 100. Still open from game/data: what the engine's converter does with the 34 malformed tokens (`force_right o`, `crew false`, `1.0f`, `0-5`, ``90\``); repeated scalars in a block are last-wins per the sequential if/else consumers (loader-prototypes §M3) — confirm on a fixture in compare mode. (from [[game-data]]) - **RESOLVED: SAVE_FORMAT tag names** — all corrections confirmed by bytes and applied to the Python reader + spec (§10). Residual doc debt: `findings/objects/save-editor-structs.md` and `verify/design-rules/SHIP_DESIGN_RULES.md` still quote R1's `ontF/faiDes/nextId` spellings (R1-provenance; annotate rather than rewrite). (from [[SAVE_FORMAT]]) -- **Notes disagree on MSVC-2010 `std::string` layout** — `struct-recovery.md` §0 vs `turn-spine.md` §1.1 give different offsets; pin both from the `_Myres >= 16` SSO branch in `Stream::WriteString` and correct the loser. (from [[re-windows-2000s-howto]]) +- **RESOLVED (2026-09-08, lane S): `std::string` layout and size, once and for all** — `_Bx@0, _Mysize@0x10, _Myres@0x14, _Alval@0x18`, **sizeof 0x1c**, and there is exactly **one** instantiation in this binary. `struct-recovery.md` §0 was right (its §0 prose had `size`/`res` transposed — fixed); `turn-spine.md` §1.1 was wrong (already annotated); `loader-prototypes.md` and one `addresses.json` prototype said 0x18 — fixed. Lane X's `ObservedTech` 0x18 reading was a mis-attribution: `+0x24` is the string's trailing `_Alval`, not a data field, proved by three complete enumerations of the element (`ObservedTech::Write` 0x00817cf0, ctor 0x008562a0, copy ctor 0x0079a184) and generalised over the whole exe by `tools/strfootprint.py` (65 string members, 0 collisions inside the 0x1c span, 51/52 gaps exactly 0x1c). **Zero recovered struct tables were wrong** — `pswd` included. Standing rule that follows: *never size a struct member from the offsets the code touches* — this build's STL puts the empty allocator **last** in both `string` (0x1c) and `vector` (0x10), and an empty allocator is never loaded or stored, so a touch-scan undercounts by 4 every time. See `observedtech-append.md` §9. (from [[re-windows-2000s-howto]], [[observedtech-append]]) - **RESOLVED: formula gaps (all 8)** — see [[formula-gaps]]; game/sim's low-confidence functions can now be pinned. **RESOLVED: std::string** = `_Bx@0,_Mysize@0x10,_Myres@0x14,_Alval@0x18`, sizeof 0x1c (struct-recovery §0 right; turn-spine §1.1 WRONG — annotated). **CORRECTION:** the 116-entry table @0x00a19718 is a name-membership list, not the effects table; use `g_TechIdNames` (196) + `OnTechResearched`. Open: values of the 6-entry AI-tech bonus table (0x00a17888); producer of ServerPlayer +0x224/+0x228/+0x22c beyond the setup-record copy (0x0077b620). (from [[tech-effects]]) - **RESOLVED: RNG semantics** — `Seed` is `thiscall(this, uint32)` RET 4 (MT19937 init + immediate twist); `Twist` takes `this` in ECX only; **twist is LAZY** (`if (left==0) Twist()` inside the draw), `left` lives at `+0x9c4`, confirming the save blob layout. **float mapping, needs a targeted check (not a red flag):** `NextFloat` decompiles as returning `float10`, but that is simply how x86 float returns look (value in `ST(0)`), so it is weak evidence of extended-precision *computation*; a single `y * 2^-32` multiply rounds once either way. B3 should still compare the mapping explicitly (exact constant and whether the draw is `y*2^-32`, `(y>>8)*2^-24`, or divided by 2^32-1), and the x87-vs-SSE float-parity policy from the RE how-to is still needed before the x64 standalone. `RNG_NextInt` signature still unverified. (own Ghidra pass) - **Struct-modelling hazard (found by M2)** — an MSVC-2010 `std::vector` member is **three words**, so a naive C translation put the next pointer at `+0x18` when it is really at `+0x14`; the hook silently reported the unmodelled word (string bytes `"TION"`) as a real field. Any hand-modelled game struct must pin its size with `static_assert` and account for 3-word vectors. (from [[M2]]) diff --git a/findings/objects/struct-recovery.md b/findings/objects/struct-recovery.md index f7d5cb9..3c26421 100644 --- a/findings/objects/struct-recovery.md +++ b/findings/objects/struct-recovery.md @@ -68,8 +68,17 @@ Readers accept legacy tags (`ISuit`, `Income`, `HPop`, `Bats`, `Builds`, `Clr`, `NShps`, `SysID`, `TrdID`, `Caps`, `GtTrf`, `FtSens`, `FtInc`, `Pris`, `NumPlgs`, `lcid`, `morev`, `cme`) by reading them into scratch/NULL — these are pre-1.8 fields, NOT members. -Common Mars/MSVC layouts seen: `std::string` = 0x1c bytes (MSVC10 `_Bx` union@0, size@0x14, res@0x18; -`FUN_008b9d70` does the `res>=16 ? heap : sso` check); `std::vector` = {begin@0, end@4, cap@8}; +Common Mars/MSVC layouts seen: **`std::string` = 0x1c bytes** — `_Bx` union@0 (16-byte SSO buffer, or a +`char*` when `_Myres >= 16`), `_Mysize`@**0x10**, `_Myres`@**0x14**, `_Alval`@0x18 (empty allocator, occupies +a word, never read or written). `FUN_008b9d70` does the `res>=16 ? heap : sso` check. +*(2026-09-08: this line previously transposed the two to `size@0x14, res@0x18`; the 0x10/0x14 offsets are the +verified ones — `Stream::WriteString` `cmp [str+0x14],0x10`, `basic_string::assign` 0x00425550.)* +The allocator is **trailing** in this build's STL, in strings and vectors alike, so it is invisible to any +scan of "which offsets does the code touch" and costs exactly 4 bytes if you size a member that way. See +`findings/subsystems/observedtech-append.md` §9 for the whole-binary audit (`tools/strfootprint.py`): 65 +`std::string` members across every serializer, zero with a sibling field inside the 0x1c span, 51 of 52 +measurable inter-member gaps exactly 0x1c. There is one string layout in this binary. +`std::vector` = {begin@0, end@4, cap@8, `_Alval`@0xc — 0x10 bytes}; `std::map/set` node = {left@0, parent@4, right@8, key@0xc, value@0x10, …, color/isnil bytes at tail}; `std::list` = {head*@0, size@4}. `Mars::NetworkObject` = {vptr@0, int id@4}. @@ -371,6 +380,32 @@ Four `std::list`s: `+4 list` (count `defc2`@+8, items `def`), ### 2.5 `Game::TechTree` — Write `FUN_005890a0` (tags `NumTechs`, `TNm`, `NumBrs`; per-tech body in a sub-writer not decompiled here; logs "TechTree: Tech %d not found saving tech tree"). +### 2.6 `Game::ObservedTech` (0x2c) — Write `0x00817cf0`, Read `0x00817c40` +Elements of `ServerPlayer::otch`, the `vector` at `ServerPlayer+0x274`. Polymorphic: +vftable `0x00a2439c`, RTTI `.?AVObservedTech@Game@@`, slots `{[0] 0x00793610 dtor, [1] Read, [2] Write}`. + +| off | type | save name | notes | +|---|---|---|---| +| 0x00 | vptr | — | `0x00a2439c` | +| 0x04 | `uint16` | `otnF` | turn first observed; widened to int32 on disk | +| 0x06 | `uint16` | `otnL` | turn last observed; widened to int32 on disk | +| 0x08 | `bool` | `odet` | **one byte** (+3 pad); `WriteBool`/`ReadBool` | +| 0x0c..0x27 | `std::string` | `otch` | tech name; `_Mysize`@0x1c, `_Myres`@0x20, `_Alval`@0x24 | +| 0x28 | `int` | `owith` | last member; `0x28 + 4 = 0x2c` = sizeof, no slack | + +`sizeof` = `0x2c` (44), pinned three ways by lane X (magic divide `0x2e8ba2e9 sar 3` at `0x0087239f`, +`imul reg,reg,0x2c`, search stride `add edi,0x2c`); the member map is the serializer's own order. +`Game::ObservedWeapon` (Write `0x00817bc0`, Read `0x00817b10`) is the identical element with tag `owep` +in place of `otch`; `odes` elements are the smaller `otnF otnL odid opid` record. + +Appended by `RecordObservedTech` `0x007ba1a0`, which **de-duplicates by tech name** — a reimplementation +that just `push_back`s will diverge on re-observation. It writes `otnF` and `otnL` from the *same* source +word on first sighting, so first-seen == last-seen initially. + +**Do not read `+0x24` as a field.** It is the name string's trailing `_Alval`. See §0 and +`findings/subsystems/observedtech-append.md` §9 — this is the class that produced the 0x18-vs-0x1c +`std::string` scare, and the resolution is that `0x1c` was right everywhere. + --- ## 3. `Game::StarFleet` (R1 `SimFleetDetails`) — Write `FUN_00701070`, Read `FUN_00702470`; `this` = obj+8 diff --git a/findings/subsystems/loader-prototypes.md b/findings/subsystems/loader-prototypes.md index c83f568..6bd13b8 100644 --- a/findings/subsystems/loader-prototypes.md +++ b/findings/subsystems/loader-prototypes.md @@ -12,7 +12,9 @@ Raw decompiles (before rename) are in `/srv/re-lab/handoff/loader-decompiles/rec decompiles in `/srv/re-lab/handoff/loader-decompiles/verify/.c`. Conventions: `thiscall` = `this` in ECX, args pushed right-to-left, callee pops (`RET n`); `cdecl` = all on stack, caller -pops. `std::string` = MSVC-2010 layout (16-byte SSO buffer / pointer, size @+0x10, capacity @+0x14, 0x18 bytes). +pops. `std::string` = **0x1c bytes**: 16-byte SSO buffer / pointer `_Bx` @+0, `_Mysize` @+0x10, `_Myres` @+0x14, +`_Alval` (empty allocator, never read or written) @+0x18. *Corrected 2026-09-08 — this line said 0x18, which is +the extent of the fields the code touches, not the size of the object; see `observedtech-append.md` §9.* "Case-insensitive" always means MSVCR100 `_stricmp` (ASCII fold). --- diff --git a/findings/subsystems/observedtech-append.md b/findings/subsystems/observedtech-append.md index 0b2a722..b039219 100644 --- a/findings/subsystems/observedtech-append.md +++ b/findings/subsystems/observedtech-append.md @@ -168,30 +168,49 @@ COL `0x00a81c78` → type descriptor `0x00aeede4` → **`.?AVObservedTech@Game@@ `ObservedTech` is polymorphic and its first word is a vptr, not a data field — which the on-disk shape does not tell you. -| offset | size | evidence | -|---|---|---| -| `+0x00` | 4 | vptr `0x00a2439c`; ctor writes it, RTTI-confirmed | -| `+0x04` | 2 | `mov word [eax-0x28],cx` at `0x007ba2b0` (`eax` = `_Mylast`, element = `_Mylast-0x2c`), source `[ebx+0xc]` | -| `+0x06` | 2 | `mov word [eax-0x26],dx` at `0x007ba2c6`, **same source word** `[ebx+0xc]` | -| `+0x08` | 4 | **unaccounted** | -| `+0x0c..+0x23` | 0x18 | `std::string` (the `otch` tech name). Object base is `+0x0c`, MSVC layout `{_Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14}`. Ctor writes `[+0x0c]=0`, `[+0x1c]=0`, `[+0x20]=0xf`; the search loop reads `[+0x1c]` as the length and calls the compare with `this = +0x0c`; the post-append assign uses `lea ecx,[_Mylast-0x20]` = `+0x0c` | -| `+0x24` | 4 | **unaccounted** | -| `+0x28` | 4 | **unaccounted** | +> **Superseded 2026-09-08 by lane S — see §9.** The table as first written called the +> embedded string 0x18 bytes and left `+0x08`, `+0x24`, `+0x28` unaccounted. `+0x24` is not a +> field: it is the string's own trailing allocator word. The corrected map is below; the +> original reasoning is kept in §9 because the way it went wrong is the useful part. -Note the string here is **0x18 bytes**, not the 0x1c the `ServerPlayer::pswd` row in -`struct-recovery.md` implies — three separate reads inside this element agree on -`{buf16, _Mysize@+0x10, _Myres@+0x14}`. Worth re-checking `pswd` against that. +| offset | size | member | evidence | +|---|---|---|---| +| `+0x00` | 4 | vptr `0x00a2439c` | ctor writes it, RTTI-confirmed | +| `+0x04` | 2 | `uint16 otnF` (turn first observed) | `mov word [eax-0x28],cx` at `0x007ba2b0` (`eax` = `_Mylast`, element = `_Mylast-0x2c`), source `[ebx+0xc]`; tag from `ObservedTech::Write` | +| `+0x06` | 2 | `uint16 otnL` (turn last observed) | `mov word [eax-0x26],dx` at `0x007ba2c6`, **same source word** `[ebx+0xc]` — first sighting sets first == last | +| `+0x08` | 1 | `bool odet` | ctor stores a **byte** (`mov [esi+0x8],bl`, `0x008562f4`); copy-ctor copies a byte; serialised with `WriteBool`/`ReadBool` | +| `+0x0c..+0x27` | 0x1c | `std::string otch` (tech name) | object base `+0x0c`, MSVC `{_Bx[16] @0, _Mysize @0x10, _Myres @0x14, _Alval @0x18}`. Ctor writes `[+0x0c]=0`, `[+0x1c]=0`, `[+0x20]=0xf`; the search loop reads `[+0x1c]` as the length and calls compare with `this = +0x0c`; the post-append assign uses `lea ecx,[_Mylast-0x20]` = `+0x0c`. `+0x24` is `_Alval` — never read, never written, by anything | +| `+0x28` | 4 | `int owith` | ctor zeroes it; `ObservedTech::Write` emits it last | -The four on-disk ints (`otnF`, `otnL`, `odet`, `owith`) have to map onto `+0x04`/`+0x06` (two -16-bit fields) and the three 4-byte slots `+0x08`, `+0x24`, `+0x28`. **That mapping is not -determined here and is deliberately not guessed.** The one suggestive observation, flagged as a -*hypothesis only*: `+0x04` and `+0x06` are two adjacent 16-bit fields written from the same -source word on first observation, which is the shape you would expect of a first-seen / -last-seen turn pair — but nothing here proves it. +`4 + 2 + 2 + 1(+3 pad) + 0x1c + 4 = 0x2c` exactly — sizeof is fully accounted for, with no +padding slack and no unaccounted field. -What would settle the mapping: read `ObservedTech`'s `Read`/`Write` serializer, where the -member order is explicit. Lane P's `observed_techs` region byte delta remains a valid live -cross-check and should now come back as exactly 44 per completion. +### Where the mapping came from — the serializer + +`Game::ObservedTech`'s vftable `0x00a2439c` is the usual 3 slots +`{ [0] 0x00793610 scalar deleting dtor, [1] 0x00817c40 Read, [2] 0x00817cf0 Write }`. +`Write` enumerates the entire object, in order, and touches nothing else: + +``` +0x00817cff movzx ecx,word [edi+0x04] push 0xa2b38c "otnF" -> stream vft+0x24 (int) +0x00817d0f movzx ecx,word [edi+0x06] push 0xa2b384 "otnL" -> stream vft+0x24 (int) +0x00817d26 lea eax,[edi+0x08] push 0xa2b36c "odet" -> WriteBool 0x008b9c20 +0x00817d37 lea ecx,[edi+0x0c] push 0xa2b3e8 "otch" -> WriteString 0x008b9d70 +0x00817d46 mov eax,[edi+0x28] push 0xa2b364 "owith" -> stream vft+0x24 (int) +``` + +`Read` (`0x00817c40`) is the exact mirror: `otnF`/`otnL` read as ints and stored back with +16-bit `mov word [ebx],ax`, `odet` through `ReadBool`, `otch` through `ReadString`, `owith` +reached as `add edi,0x28`. That matches the on-disk order `save_reader.py` already had +(`Otch = otnF otnL odet otch(string) owith`), which is a nice independent agreement between +the disassembly and the save oracle. + +`Game::ObservedWeapon` (`Write` `0x00817bc0`, `Read` `0x00817b10`) is the same element shape +with tag `owep` (`0x00a2b3f0`) in place of `otch` — a second instance of the identical 0x2c +layout. + +Lane P's `observed_techs` region byte delta remains a valid live cross-check and should come +back as exactly 44 per completion. ## 5. False-positive rate, honestly @@ -276,3 +295,111 @@ the stride evidence on the first four. `RecordObservedTech`, `vector_ObservedTech_push_back`, `vector_ObservedTech_assign`, `ObservedTech_ctor`, `vector_44B_grow`; `ServerPlayer_off_ObservedTechs` updated from "NOT PINNED" to `verified`. + +**Lane S round (2026-09-08).** Labels: `ObservedTech_Write` `0x00817cf0`, `ObservedTech_Read` +`0x00817c40`, `ObservedTech_scalar_deleting_dtor` `0x00793610`, `ObservedWeapon_Write` +`0x00817bc0`, `ObservedWeapon_Read` `0x00817b10`, `vector_ObservedTech_uninit_copy` `0x0079a150`, +`vector_string_find_by_name` `0x00699bd0`. Prototypes on the five class methods. Plate comments +carrying the full member map on the two serializers, the ctor, the dtor and the copy helper, and +the `sizeof(std::string) = 0x1c` fact on `Stream::WriteString` `0x008b9d70` and on the +`vector` stride site `0x00699bd0` — the two places a future lane is most likely to look. +`addresses.json` +10 entries (`std_string_sizeof`, `ObservedTech_Read/_Write/_dtor/_copy_ctor`, +`ObservedTech_off_TurnFirst/_TurnLast/_Detected/_With`, `ObservedWeapon_Write`), 4 corrected. + +--- + +## 9. `std::string` is 0x1c, not 0x18 — the correction, and why it mattered (lane S, 2026-09-08) + +§4 above originally reported the embedded string as **0x18 bytes**, against the campaign-wide +`_Bx@0, _Mysize@0x10, _Myres@0x14, _Alval@0x18`, sizeof `0x1c`. Lane X flagged it as "worth +re-checking" rather than asserting it, which was the right call: **`0x1c` is correct, and it is +correct everywhere in this binary.** `ObservedTech+0x24` is the string's own trailing allocator +word, not a data member. + +### Why the 0x18 reading looked right + +Everything lane X observed was accurate. The string's *live* fields really do stop at `+0x14` +(`_Bx@0`, `_Mysize@0x10`, `_Myres@0x14`), because `_Alval` is `std::allocator` — an empty +class. It occupies a word of the object but is never loaded or stored, so it is invisible to +any evidence based on **what the code touches**. Sizing a type from its accessed fields +undercounts it by exactly the tail padding. The same trap is live for `std::vector` in this +build, which is `{_Myfirst, _Mylast, _Myend, _Alval}` = `0x10` while only three words are ever +read (`events.md` already records the `_Alval` word at `EventStorage+0x10` and `+0x14`). + +### The three things that settle it inside `ObservedTech` + +Each is a *complete enumeration* of the object, which is the right instrument here — an +enumeration can show a field's **absence**, a touch-scan cannot. + +1. **`ObservedTech::Write` `0x00817cf0`** serialises `+0x04, +0x06, +0x08, +0x0c, +0x28`. + No `+0x24`. (`Read` `0x00817c40` mirrors it.) +2. **`ObservedTech_ctor` `0x008562a0`** initialises `+0x00, +0x04, +0x08, +0x0c(string), +0x28`. + No `+0x24` — while zeroing every other scalar in the object. +3. **The copy constructor**, inlined at `0x0079a184` inside the vector's uninitialised-copy + helper `FUN_0079a150`, copies `+0x04, +0x06, +0x08`, the string at `+0x0c`, and `+0x28`. + No `+0x24`. + +A 4-byte data member that the constructor, the copy constructor and the serializer all ignore +is not a data member. + +### Binary-wide check — `tools/strfootprint.py` + +One class is an anecdote, so the same question was put to the whole binary. Every serializer +hands member pointers to the `Mars::Stream` primitive helpers with a fixed idiom +(`lea r,[base+disp]; push r; push tag; push stream; call helper`), so the scanner recovers +`(base, disp, tag)` for every string site and then asks: does the same function touch any other +offset inside `(N, N+0x1c)` off the same base register? + +``` +string helper call sites : 241 + resolved to a class member offset : 65 + stack temporaries (ebp/esp base) : 30 + offset not reached by a plain lea : 146 + +members with a sibling inside (N, N+0x1c) : 0 + +gap from a string member to the next member on the same base: + +0x1c : 51 + +0x20 : 1 +``` + +**65 string members across every serializer in the exe, zero collisions, and 51 of the 52 +measurable gaps are exactly `0x1c`.** The single `+0x20` is `StrategyServer::KeyPath` at +`+0x134`, whose *Read* side gives `+0x1c` to `+0x150` — the writer simply skips a member. +There is no `0x18` instantiation, no empty-base-optimised variant, and no game-local string +class. One layout, `0x1c`. + +Two exclusions matter or the scan reports noise, and both are why a naive version of this +would have "confirmed" 0x18: + +* **`ebp`/`esp` bases are stack temporaries, not members.** A local string at `[ebp-0x2c]` + shows accesses at `-0x1c` and `-0x18` — which read exactly like two sibling fields inside + the span. All 30 such sites are excluded. +* **`N+0x10` and `N+0x14` are the string's own `_Mysize`/`_Myres`**, touched inline whenever + the compiler expands the `_Myres >= 16 ? _Ptr : _Buf` test at a call site. + +### Independent corroboration outside the scan + +* `FUN_00699bd0` walks a `vector` doing the SSO test at `[esi+0x14]`/`[esi]` — + element base *is* string base — and advances `add esi,0x1c` (`0x00699c29`). The stride of a + vector of strings **is** `sizeof(std::string)`. +* `EventStorage::PostEvent` `0x008862b0` takes two by-value `std::string`s at `[ebp+0x08]` and + `[ebp+0x24]` (spacing `0x1c`) and `RET 0x4c` = `2*0x1c + 5*4`. +* `MoraleEvent` is 0x50 bytes with its name string at `+0x34`: `0x34 + 0x1c = 0x50` exactly. + +### Blast radius — what actually had to change + +Nothing in any recovered struct table. Every string-bearing layout in `struct-recovery.md`, +`save-editor-structs.md`, `events.md` and `schema-gaps-resolved.md` already used `0x1c` spans +and `0x1c` inter-field gaps, and the binary agrees with all of them. `ServerPlayer::pswd` at +`0x2dc..0x2f7` — the row §4 asked to re-check — is **correct**: `Write` `0x008563e0` puts the +next member exactly `0x1c` above it. The corrections were confined to three prose statements +that had propagated the 0x18 number (`loader-prototypes.md` conventions line, the +`GlobalConst_ParseString` prototype in `addresses.json`, and §4 here) plus one transposition in +`struct-recovery.md` §0 that said `size@0x14, res@0x18` where the verified offsets are +`0x10`/`0x14`. + +The lesson worth carrying, and the reason this was worth chasing rather than reconciling: **do +not size a struct member from the offsets the code touches.** Trailing empty-allocator words in +this build's STL are invisible to a touch-scan and cost exactly 4 bytes every time. Size types +from an enumeration — a serializer, a constructor, a copy constructor, or a container stride. diff --git a/ghidra/addresses.json b/ghidra/addresses.json index 885af86..11aef4d 100644 --- a/ghidra/addresses.json +++ b/ghidra/addresses.json @@ -334,7 +334,7 @@ "name": "GlobalConst_ParseString", "addr": "0x008b7670", "convention": "cdecl", - "prototype": "void (std::string* storage, const char* text) /* *storage = text; MSVC2010 std::string 0x18 bytes {+0 char buf[16] | char* ptr when capacity > 15, +0x10 size, +0x14 capacity}; 146 keys (*_NAME, *_SOUND, *_TEXTURENAME, DERELICT_SECTION_nn ...); seen in the M1 trace */", + "prototype": "void (std::string* storage, const char* text) /* *storage = text; MSVC std::string is 0x1c bytes {+0 char buf[16] | char* ptr when capacity > 15, +0x10 _Mysize, +0x14 _Myres, +0x18 _Alval (empty allocator, never read/written)} -- see std_string_sizeof; 146 keys (*_NAME, *_SOUND, *_TEXTURENAME, DERELICT_SECTION_nn ...); seen in the M1 trace */", "status": "verified-by-trace", "source": "sots-engine docs/M1.md" }, @@ -3174,7 +3174,7 @@ "name": "ObservedTech_sizeof", "offset": "0x2c", "convention": "constant", - "prototype": "sizeof(Game::ObservedTech) = 0x2c (44 bytes). Confirmed independently by (a) the compiler's magic division by 44 (mov eax,0x2e8ba2e9; imul; sar edx,3) at 0x0087239f and 0x007b7339, (b) imul reg,reg,0x2c at 0x0087243a, 0x00872468, 0x007b735b, and (c) the iterator advance `add edi,0x2c` in RecordObservedTech's linear search at 0x007ba257. Matches the on-disk lower bound exactly (4 int + one 0x1c std::string = 44), so there is no padding slack.", + "prototype": "sizeof(Game::ObservedTech) = 0x2c (44 bytes). Confirmed independently by (a) the compiler's magic division by 44 (mov eax,0x2e8ba2e9; imul; sar edx,3) at 0x0087239f and 0x007b7339, (b) imul reg,reg,0x2c at 0x0087243a, 0x00872468, 0x007b735b, and (c) the iterator advance `add edi,0x2c` in RecordObservedTech's linear search at 0x007ba257. FULL MEMBER MAP, from ObservedTech_Write 0x00817cf0 / ObservedTech_Read 0x00817c40 (lane S 2026-09-08): +0x00 vptr 0x00a2439c; +0x04 uint16 otnF; +0x06 uint16 otnL; +0x08 bool odet (1 byte, +3 pad); +0x0c std::string otch (0x1c, so _Mysize at +0x1c, _Myres at +0x20, _Alval at +0x24); +0x28 int owith. 4 + 2 + 2 + 4 + 0x1c + 4 = 0x2c exactly, no padding slack and no unaccounted field.", "status": "verified", "source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)" }, @@ -3190,7 +3190,7 @@ "name": "ObservedTech_off_Name", "offset": "0x0c", "convention": "offset", - "prototype": "std::string (save tag `otch`, the tech name) at ObservedTech+0x0c, 0x18 bytes, spanning +0x0c..+0x23. MSVC layout relative to the string object: _Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14 -- i.e. ObservedTech+0x1c is the length and +0x20 the capacity. Evidence: ObservedTech_ctor 0x008562a0 writes [elem+0x0c]=0, [elem+0x1c]=0, [elem+0x20]=0xf; RecordObservedTech's search reads [elem+0x1c] as the length and calls compare with this=elem+0x0c; the post-append assign uses lea ecx,[_Mylast-0x20]. NOTE this string is 0x18 bytes, not the 0x1c implied by the ServerPlayer pswd row in struct-recovery.md. UNACCOUNTED in the element: +0x08, +0x24, +0x28 (4 bytes each) plus two 16-bit fields at +0x04/+0x06 written from one source word at 0x007ba2b0 / 0x007ba2c6 -- the mapping of the four on-disk ints (otnF, otnL, odet, owith) onto those slots is NOT determined.", + "prototype": "std::string (save tag `otch`, the tech name) at ObservedTech+0x0c, 0x1c bytes, spanning +0x0c..+0x27. MSVC layout relative to the string object: _Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14, _Alval @+0x18 -- i.e. ObservedTech+0x1c is the length, +0x20 the capacity, +0x24 the empty-allocator word (never read or written by anything). Evidence: ObservedTech_ctor 0x008562a0 writes [elem+0x0c]=0, [elem+0x1c]=0, [elem+0x20]=0xf; RecordObservedTech's search reads [elem+0x1c] as the length and calls compare with this=elem+0x0c; the post-append assign uses lea ecx,[_Mylast-0x20]. CORRECTION (lane S 2026-09-08): an earlier revision called this string 0x18 bytes and listed +0x24 as an unaccounted data field. It is not one -- three independent whole-object enumerations skip +0x24 entirely: ObservedTech_ctor 0x008562a0, ObservedTech_copy_ctor 0x0079a184, and ObservedTech_Write 0x00817cf0 (which serialises +0x04,+0x06,+0x08,+0x0c,+0x28 and nothing else). sizeof(std::string)=0x1c holds binary-wide; see std_string_sizeof.", "status": "verified", "source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)" }, @@ -3198,7 +3198,7 @@ "name": "RecordObservedTech", "addr": "0x007ba1a0", "convention": "thiscall", - "prototype": "void (this, ServerPlayer* observer, ?, std::string* techName) -- appends to observer->otch. Linear-searches observer->otch (ServerPlayer+0x274) with stride 0x2c comparing each element's name string; if not found, default-constructs an ObservedTech on the stack (0x008562a0) and push_backs it (0x007b7320 @0x007ba288), then writes two 16-bit fields into the new element at +0x04 and +0x06 from param_1+0xc. DIRECT CALLEE of ServerPlayer::OnTechResearched 0x00891790; also called from 0x007be228, 0x007be4e1, 0x007be535. This is the append lane P could not find.", + "prototype": "void (this, ServerPlayer* observer, ?, std::string* techName) -- appends to observer->otch. Linear-searches observer->otch (ServerPlayer+0x274) with stride 0x2c comparing each element's name string; if not found, default-constructs an ObservedTech on the stack (0x008562a0) and push_backs it (0x007b7320 @0x007ba288), then writes otnF (+0x04) and otnL (+0x06), both 16-bit, from the same source word param_1+0xc -- i.e. first-observed turn == last-observed turn on the first sighting, which is what the tag names now confirm. DIRECT CALLEE of ServerPlayer::OnTechResearched 0x00891790; also called from 0x007be228, 0x007be4e1, 0x007be535. This is the append lane P could not find. DE-DUPLICATING: appends only when no existing element carries that tech name, so a reimplementation must not naively push_back on re-observation.", "status": "verified", "source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)" }, @@ -3222,9 +3222,89 @@ "name": "ObservedTech_ctor", "addr": "0x008562a0", "convention": "thiscall", - "prototype": "Game::ObservedTech* (ObservedTech* this) -- default ctor; sets vptr 0x00a2439c and empties the name string.", + "prototype": "Game::ObservedTech* (ObservedTech* this) -- default ctor. Writes exactly: vptr 0x00a2439c at +0x00; dword 0 at +0x04 (otnF+otnL zeroed together); BYTE 0 at +0x08 (`mov [esi+0x8],bl`, 0x008562f4 -- odet is a bool, not an int); the empty string at +0x0c (_Buf[0]=0, _Mysize=0, _Myres=0xf, then assign(\"\") 0x00425550); dword 0 at +0x28 (owith). It never touches +0x24 -- that word is the string's _Alval.", "status": "verified", - "source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)" + "source": "findings/subsystems/observedtech-append.md (lane X; element map corrected + serializer read by lane S 2026-09-08)" + }, + { + "name": "ObservedTech_Write", + "addr": "0x00817cf0", + "convention": "thiscall", + "prototype": "void Game::ObservedTech::Write(Mars::Stream* s) RET 4. Vftable 0x00a2439c slot [2]. Serialises the whole object, in order and with nothing else: `otnF` = movzx word [this+0x04] via stream vft+0x24 (int); `otnL` = movzx word [this+0x06] via vft+0x24; `odet` = WriteBool 0x008b9c20 (&this[+0x08]); `otch` = WriteString 0x008b9d70 (&this[+0x0c]); `owith` = [this+0x28] via vft+0x24. THIS IS THE MEMBER MAP: there is no field at +0x24. Tag pointers 0x00a2b38c/0x00a2b384/0x00a2b36c/0x00a2b3e8/0x00a2b364.", + "status": "verified", + "source": "lane S own disassembly 2026-09-08" + }, + { + "name": "ObservedTech_Read", + "addr": "0x00817c40", + "convention": "thiscall", + "prototype": "void Game::ObservedTech::Read(Mars::Stream* s) RET 4. Vftable 0x00a2439c slot [1]. Mirror of ObservedTech_Write: `otnF`/`otnL` through stream vft+0x10 (int-by-ref) stored back as 16-bit (`mov word [ebx],ax`) at +0x04/+0x06, `odet` = ReadBool 0x008b9c00 (&this[+0x08]), `otch` = ReadString 0x008b9d90 (&this[+0x0c]), `owith` at +0x28 (reached as `add edi,0x28`).", + "status": "verified", + "source": "lane S own disassembly 2026-09-08" + }, + { + "name": "ObservedTech_dtor", + "addr": "0x00793610", + "convention": "thiscall", + "prototype": "ObservedTech* (ObservedTech* this, int flags) RET 4. Vftable 0x00a2439c slot [0], scalar deleting dtor. Inlines ~basic_string on the name at +0x0c (`cmp [esi+0x20],0x10` -> operator delete [esi+0x0c], then _Tidy: [esi+0x20]=0xf, [esi+0x1c]=0, byte [esi+0x0c]=0), restores the base vptr 0x009e22bc, and frees `this` when flags&1. The string is the ONLY member needing destruction -- confirming there is no second string or owned pointer in the element.", + "status": "verified", + "source": "lane S own disassembly 2026-09-08" + }, + { + "name": "ObservedTech_copy_ctor", + "addr": "0x0079a184", + "convention": "site", + "prototype": "site inside the vector uninitialised-copy helper FUN_0079a150(&_Alval, dest, src). The inlined copy constructor writes vptr 0x00a2439c, `mov word [dst+0x04],[src+0x04]`, `mov word [dst+0x06],[src+0x06]`, `mov byte [dst+0x08],[src+0x08]`, the string at +0x0c (via basic_string::assign 0x00425430), and `mov [dst+0x28],[src+0x28]`. Nothing is copied at +0x24 -- second independent proof that +0x24 belongs to the 0x1c string, not to a data member.", + "status": "verified", + "source": "lane S own disassembly 2026-09-08" + }, + { + "name": "ObservedTech_off_TurnFirst", + "offset": "0x04", + "convention": "offset", + "prototype": "uint16 otnF -- turn the tech was first observed. Widened to int on disk by stream vft+0x24. Written together with otnL from one source word at RecordObservedTech 0x007ba2b0.", + "status": "verified", + "source": "lane S: ObservedTech_Write 0x00817cf0 / ObservedTech_Read 0x00817c40" + }, + { + "name": "ObservedTech_off_TurnLast", + "offset": "0x06", + "convention": "offset", + "prototype": "uint16 otnL -- turn the tech was last observed. Widened to int on disk. Written at RecordObservedTech 0x007ba2c6 from the same source word as otnF.", + "status": "verified", + "source": "lane S: ObservedTech_Write 0x00817cf0 / ObservedTech_Read 0x00817c40" + }, + { + "name": "ObservedTech_off_Detected", + "offset": "0x08", + "convention": "offset", + "prototype": "bool odet -- ONE BYTE (ctor `mov [esi+0x8],bl` 0x008562f4; copy-ctor `mov byte` 0x0079a1a0), serialised through WriteBool/ReadBool, so on disk it is 1 byte + 3 NUL joint padding. Note the save reader types `odet` as int; for a 4-character tag a bool item and an int item are the same 12 bytes and the value is identical, so that is benign, not a parse error.", + "status": "verified", + "source": "lane S: ObservedTech_Write 0x00817cf0 / ObservedTech_ctor 0x008562a0" + }, + { + "name": "ObservedTech_off_With", + "offset": "0x28", + "convention": "offset", + "prototype": "int owith -- last member of the element; serialised through stream vft+0x24 with tag 0x00a2b364. +0x28..+0x2b is the tail of the 0x2c-byte element, which is why sizeof is 0x2c with no padding slack.", + "status": "verified", + "source": "lane S: ObservedTech_Write 0x00817cf0 / ObservedTech_Read 0x00817c40" + }, + { + "name": "ObservedWeapon_Write", + "addr": "0x00817bc0", + "convention": "thiscall", + "prototype": "void Game::ObservedWeapon::Write(Mars::Stream* s) RET 4. Byte-for-byte the same shape as ObservedTech_Write with tag `owep` (0x00a2b3f0) in place of `otch`: otnF u16 @+0x04, otnL u16 @+0x06, odet bool @+0x08, owep std::string (0x1c) @+0x0c, owith int @+0x28. Reader is 0x00817b10. Independent second instance of the same 0x2c element shape.", + "status": "verified", + "source": "lane S own disassembly 2026-09-08" + }, + { + "name": "std_string_sizeof", + "offset": "0x1c", + "convention": "constant", + "prototype": "sizeof(std::basic_string) = 0x1c (28) binary-wide, ONE layout only: _Bx (16-byte SSO union, char[16] or char*) @+0x00, _Mysize @+0x10, _Myres @+0x14, _Alval (empty allocator) @+0x18. The allocator word is trailing, never read and never written -- the same allocator-last shape as this build's std::vector {_Myfirst,_Mylast,_Myend,_Alval} = 0x10. Evidence: (a) Stream::WriteString 0x008b9d76 `cmp [str+0x14],0x10` / `mov eax,[str]` and basic_string::assign 0x00425550 (_Mysize +0x10, _Myres +0x14) fix the field offsets; (b) the vector scan loop FUN_00699bd0 advances `add esi,0x1c` (0x00699c29) while doing the SSO test at [esi+0x14]/[esi] -- element base == string base, so the stride IS the sizeof; (c) PostEvent 0x008862b0 takes two by-value strings at [ebp+0x08] and [ebp+0x24] and RET 0x4c = 2*0x1c + 5*4; (d) a whole-binary sweep of the Stream string helpers recovered 65 std::string members off a non-stack base across all serializers -- ZERO have any sibling member inside (N, N+0x1c), and 51 of the 52 that have a next member have it at exactly N+0x1c (the one exception, StrategyServer KeyPath @+0x134, is +0x20 on the Write side and +0x1c on the Read side, i.e. the writer skips a member). There is no 0x18 instantiation, no empty-base variant and no game-local string class.", + "status": "verified", + "source": "lane S 2026-09-08; scanner tools/strfootprint.py" }, { "name": "vector_44B_grow", diff --git a/ghidra/generated/sots_addresses.h b/ghidra/generated/sots_addresses.h index 2adca21..d1674ec 100644 --- a/ghidra/generated/sots_addresses.h +++ b/ghidra/generated/sots_addresses.h @@ -1,5 +1,5 @@ // GENERATED — do not edit. Facts about Sword of the Stars.exe (GOG 1.8.1). -// Source: sots-re ghidra/addresses.json @ 1a58bcd, generated 2026-09-08 by tools/gen_addresses.py +// Source: sots-re ghidra/addresses.json @ d523d27, generated 2026-09-08 by tools/gen_addresses.py // Runtime address = (uintptr_t)GetModuleHandle(NULL) + RVA (the exe is ASLR-relocated). #pragma once #include @@ -89,7 +89,7 @@ constexpr uint32_t GlobalConst_ParseColour = 0x004b7110; constexpr uint32_t GlobalConst_ParseRect = 0x004b7040; // cdecl void (float xyz[3], const char* text) /* '%f %f %f'; 5 keys (HIVER_SPAWN_*_GATE_HEIGHT / _SHIP_OFFSET ...); inferred from the M1 trace */ [verified-by-trace] constexpr uint32_t GlobalConst_ParseVec3 = 0x004b7080; -// cdecl void (std::string* storage, const char* text) /* *storage = text; MSVC2010 std::string 0x18 bytes {+0 char buf[16] | char* ptr when capacity > 15, +0x10 size, +0x14 capacity}; 146 keys (*_NAME, *_SOUND, *_TEXTURENAME, DERELICT_SECTION_nn ...); seen in the M1 trace */ [verified-by-trace] +// cdecl void (std::string* storage, const char* text) /* *storage = text; MSVC std::string is 0x1c bytes {+0 char buf[16] | char* ptr when capacity > 15, +0x10 _Mysize, +0x14 _Myres, +0x18 _Alval (empty allocator, never read/written)} -- see std_string_sizeof; 146 keys (*_NAME, *_SOUND, *_TEXTURENAME, DERELICT_SECTION_nn ...); seen in the M1 trace */ [verified-by-trace] constexpr uint32_t GlobalConst_ParseString = 0x004b7670; // cdecl void (void) /* called once from Application::Initialize; sets g_GlobalConstsLoaded */ [verified] constexpr uint32_t GlobalConsts_LoadAll = 0x004b76d0; @@ -799,20 +799,40 @@ constexpr uint32_t Mars_Vec3_NormaliseEpsilon = 0x005e1ef8; constexpr uint32_t StrategyServer_MoveFleet_straight_leg = 0x003da0f2; // site site inside StrategyServer::MoveFleet /* the move != distance branch: exactly two roundings per component, tmp.c = float32(dir.c * move) then pos.c = float32(pos.c + tmp.c). `move` is reloaded from a float32 slot. The sibling branch (move == distance, an EXACT float compare) copies the destination's three words verbatim with mov, so an arrival never steps onto its destination. */ [verified] constexpr uint32_t StrategyServer_MoveFleet_position_update = 0x003da2ac; -// constant sizeof(Game::ObservedTech) = 0x2c (44 bytes). Confirmed independently by (a) the compiler's magic division by 44 (mov eax,0x2e8ba2e9; imul; sar edx,3) at 0x0087239f and 0x007b7339, (b) imul reg,reg,0x2c at 0x0087243a, 0x00872468, 0x007b735b, and (c) the iterator advance `add edi,0x2c` in RecordObservedTech's linear search at 0x007ba257. Matches the on-disk lower bound exactly (4 int + one 0x1c std::string = 44), so there is no padding slack. [verified] +// constant sizeof(Game::ObservedTech) = 0x2c (44 bytes). Confirmed independently by (a) the compiler's magic division by 44 (mov eax,0x2e8ba2e9; imul; sar edx,3) at 0x0087239f and 0x007b7339, (b) imul reg,reg,0x2c at 0x0087243a, 0x00872468, 0x007b735b, and (c) the iterator advance `add edi,0x2c` in RecordObservedTech's linear search at 0x007ba257. FULL MEMBER MAP, from ObservedTech_Write 0x00817cf0 / ObservedTech_Read 0x00817c40 (lane S 2026-09-08): +0x00 vptr 0x00a2439c; +0x04 uint16 otnF; +0x06 uint16 otnL; +0x08 bool odet (1 byte, +3 pad); +0x0c std::string otch (0x1c, so _Mysize at +0x1c, _Myres at +0x20, _Alval at +0x24); +0x28 int owith. 4 + 2 + 2 + 4 + 0x1c + 4 = 0x2c exactly, no padding slack and no unaccounted field. [verified] constexpr uint32_t ObservedTech_sizeof = 0x0000002c; // data Game::ObservedTech vftable. RTTI COL 0x00a81c78 -> type descriptor 0x00aeede4 = '.?AVObservedTech@Game@@'. Written to element+0x00 by ObservedTech_ctor 0x008562a0 -- so ObservedTech is polymorphic and its first word is the vptr, not a data field. [verified] constexpr uint32_t ObservedTech_vftable = 0x0062439c; -// offset std::string (save tag `otch`, the tech name) at ObservedTech+0x0c, 0x18 bytes, spanning +0x0c..+0x23. MSVC layout relative to the string object: _Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14 -- i.e. ObservedTech+0x1c is the length and +0x20 the capacity. Evidence: ObservedTech_ctor 0x008562a0 writes [elem+0x0c]=0, [elem+0x1c]=0, [elem+0x20]=0xf; RecordObservedTech's search reads [elem+0x1c] as the length and calls compare with this=elem+0x0c; the post-append assign uses lea ecx,[_Mylast-0x20]. NOTE this string is 0x18 bytes, not the 0x1c implied by the ServerPlayer pswd row in struct-recovery.md. UNACCOUNTED in the element: +0x08, +0x24, +0x28 (4 bytes each) plus two 16-bit fields at +0x04/+0x06 written from one source word at 0x007ba2b0 / 0x007ba2c6 -- the mapping of the four on-disk ints (otnF, otnL, odet, owith) onto those slots is NOT determined. [verified] +// offset std::string (save tag `otch`, the tech name) at ObservedTech+0x0c, 0x1c bytes, spanning +0x0c..+0x27. MSVC layout relative to the string object: _Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14, _Alval @+0x18 -- i.e. ObservedTech+0x1c is the length, +0x20 the capacity, +0x24 the empty-allocator word (never read or written by anything). Evidence: ObservedTech_ctor 0x008562a0 writes [elem+0x0c]=0, [elem+0x1c]=0, [elem+0x20]=0xf; RecordObservedTech's search reads [elem+0x1c] as the length and calls compare with this=elem+0x0c; the post-append assign uses lea ecx,[_Mylast-0x20]. CORRECTION (lane S 2026-09-08): an earlier revision called this string 0x18 bytes and listed +0x24 as an unaccounted data field. It is not one -- three independent whole-object enumerations skip +0x24 entirely: ObservedTech_ctor 0x008562a0, ObservedTech_copy_ctor 0x0079a184, and ObservedTech_Write 0x00817cf0 (which serialises +0x04,+0x06,+0x08,+0x0c,+0x28 and nothing else). sizeof(std::string)=0x1c holds binary-wide; see std_string_sizeof. [verified] constexpr uint32_t ObservedTech_off_Name = 0x0000000c; -// thiscall void (this, ServerPlayer* observer, ?, std::string* techName) -- appends to observer->otch. Linear-searches observer->otch (ServerPlayer+0x274) with stride 0x2c comparing each element's name string; if not found, default-constructs an ObservedTech on the stack (0x008562a0) and push_backs it (0x007b7320 @0x007ba288), then writes two 16-bit fields into the new element at +0x04 and +0x06 from param_1+0xc. DIRECT CALLEE of ServerPlayer::OnTechResearched 0x00891790; also called from 0x007be228, 0x007be4e1, 0x007be535. This is the append lane P could not find. [verified] +// thiscall void (this, ServerPlayer* observer, ?, std::string* techName) -- appends to observer->otch. Linear-searches observer->otch (ServerPlayer+0x274) with stride 0x2c comparing each element's name string; if not found, default-constructs an ObservedTech on the stack (0x008562a0) and push_backs it (0x007b7320 @0x007ba288), then writes otnF (+0x04) and otnL (+0x06), both 16-bit, from the same source word param_1+0xc -- i.e. first-observed turn == last-observed turn on the first sighting, which is what the tag names now confirm. DIRECT CALLEE of ServerPlayer::OnTechResearched 0x00891790; also called from 0x007be228, 0x007be4e1, 0x007be535. This is the append lane P could not find. DE-DUPLICATING: appends only when no existing element carries that tech name, so a reimplementation must not naively push_back on re-observation. [verified] constexpr uint32_t RecordObservedTech = 0x003ba1a0; // thiscall void (std::vector* this, ObservedTech* value) RET 4. Stride 0x2c. Calls vector_44B_grow 0x007b5820 when _Mylast==_Myend -- the realloc that moves all three vector words. Exactly one caller (RecordObservedTech), so this instantiation is not COMDAT-ambiguous. [verified] constexpr uint32_t vector_ObservedTech_push_back = 0x003b7320; // thiscall std::vector& operator=(const std::vector&) RET 4. Both callers pass ServerPlayer+0x274 (0x00878091 in the settings copy-out, 0x00892507 in the copy-in). [verified] constexpr uint32_t vector_ObservedTech_assign = 0x00472380; -// thiscall Game::ObservedTech* (ObservedTech* this) -- default ctor; sets vptr 0x00a2439c and empties the name string. [verified] +// thiscall Game::ObservedTech* (ObservedTech* this) -- default ctor. Writes exactly: vptr 0x00a2439c at +0x00; dword 0 at +0x04 (otnF+otnL zeroed together); BYTE 0 at +0x08 (`mov [esi+0x8],bl`, 0x008562f4 -- odet is a bool, not an int); the empty string at +0x0c (_Buf[0]=0, _Mysize=0, _Myres=0xf, then assign("") 0x00425550); dword 0 at +0x28 (owith). It never touches +0x24 -- that word is the string's _Alval. [verified] constexpr uint32_t ObservedTech_ctor = 0x004562a0; +// thiscall void Game::ObservedTech::Write(Mars::Stream* s) RET 4. Vftable 0x00a2439c slot [2]. Serialises the whole object, in order and with nothing else: `otnF` = movzx word [this+0x04] via stream vft+0x24 (int); `otnL` = movzx word [this+0x06] via vft+0x24; `odet` = WriteBool 0x008b9c20 (&this[+0x08]); `otch` = WriteString 0x008b9d70 (&this[+0x0c]); `owith` = [this+0x28] via vft+0x24. THIS IS THE MEMBER MAP: there is no field at +0x24. Tag pointers 0x00a2b38c/0x00a2b384/0x00a2b36c/0x00a2b3e8/0x00a2b364. [verified] +constexpr uint32_t ObservedTech_Write = 0x00417cf0; +// thiscall void Game::ObservedTech::Read(Mars::Stream* s) RET 4. Vftable 0x00a2439c slot [1]. Mirror of ObservedTech_Write: `otnF`/`otnL` through stream vft+0x10 (int-by-ref) stored back as 16-bit (`mov word [ebx],ax`) at +0x04/+0x06, `odet` = ReadBool 0x008b9c00 (&this[+0x08]), `otch` = ReadString 0x008b9d90 (&this[+0x0c]), `owith` at +0x28 (reached as `add edi,0x28`). [verified] +constexpr uint32_t ObservedTech_Read = 0x00417c40; +// thiscall ObservedTech* (ObservedTech* this, int flags) RET 4. Vftable 0x00a2439c slot [0], scalar deleting dtor. Inlines ~basic_string on the name at +0x0c (`cmp [esi+0x20],0x10` -> operator delete [esi+0x0c], then _Tidy: [esi+0x20]=0xf, [esi+0x1c]=0, byte [esi+0x0c]=0), restores the base vptr 0x009e22bc, and frees `this` when flags&1. The string is the ONLY member needing destruction -- confirming there is no second string or owned pointer in the element. [verified] +constexpr uint32_t ObservedTech_dtor = 0x00393610; +// site site inside the vector uninitialised-copy helper FUN_0079a150(&_Alval, dest, src). The inlined copy constructor writes vptr 0x00a2439c, `mov word [dst+0x04],[src+0x04]`, `mov word [dst+0x06],[src+0x06]`, `mov byte [dst+0x08],[src+0x08]`, the string at +0x0c (via basic_string::assign 0x00425430), and `mov [dst+0x28],[src+0x28]`. Nothing is copied at +0x24 -- second independent proof that +0x24 belongs to the 0x1c string, not to a data member. [verified] +constexpr uint32_t ObservedTech_copy_ctor = 0x0039a184; +// offset uint16 otnF -- turn the tech was first observed. Widened to int on disk by stream vft+0x24. Written together with otnL from one source word at RecordObservedTech 0x007ba2b0. [verified] +constexpr uint32_t ObservedTech_off_TurnFirst = 0x00000004; +// offset uint16 otnL -- turn the tech was last observed. Widened to int on disk. Written at RecordObservedTech 0x007ba2c6 from the same source word as otnF. [verified] +constexpr uint32_t ObservedTech_off_TurnLast = 0x00000006; +// offset bool odet -- ONE BYTE (ctor `mov [esi+0x8],bl` 0x008562f4; copy-ctor `mov byte` 0x0079a1a0), serialised through WriteBool/ReadBool, so on disk it is 1 byte + 3 NUL joint padding. Note the save reader types `odet` as int; for a 4-character tag a bool item and an int item are the same 12 bytes and the value is identical, so that is benign, not a parse error. [verified] +constexpr uint32_t ObservedTech_off_Detected = 0x00000008; +// offset int owith -- last member of the element; serialised through stream vft+0x24 with tag 0x00a2b364. +0x28..+0x2b is the tail of the 0x2c-byte element, which is why sizeof is 0x2c with no padding slack. [verified] +constexpr uint32_t ObservedTech_off_With = 0x00000028; +// thiscall void Game::ObservedWeapon::Write(Mars::Stream* s) RET 4. Byte-for-byte the same shape as ObservedTech_Write with tag `owep` (0x00a2b3f0) in place of `otch`: otnF u16 @+0x04, otnL u16 @+0x06, odet bool @+0x08, owep std::string (0x1c) @+0x0c, owith int @+0x28. Reader is 0x00817b10. Independent second instance of the same 0x2c element shape. [verified] +constexpr uint32_t ObservedWeapon_Write = 0x00417bc0; +// constant sizeof(std::basic_string) = 0x1c (28) binary-wide, ONE layout only: _Bx (16-byte SSO union, char[16] or char*) @+0x00, _Mysize @+0x10, _Myres @+0x14, _Alval (empty allocator) @+0x18. The allocator word is trailing, never read and never written -- the same allocator-last shape as this build's std::vector {_Myfirst,_Mylast,_Myend,_Alval} = 0x10. Evidence: (a) Stream::WriteString 0x008b9d76 `cmp [str+0x14],0x10` / `mov eax,[str]` and basic_string::assign 0x00425550 (_Mysize +0x10, _Myres +0x14) fix the field offsets; (b) the vector scan loop FUN_00699bd0 advances `add esi,0x1c` (0x00699c29) while doing the SSO test at [esi+0x14]/[esi] -- element base == string base, so the stride IS the sizeof; (c) PostEvent 0x008862b0 takes two by-value strings at [ebp+0x08] and [ebp+0x24] and RET 0x4c = 2*0x1c + 5*4; (d) a whole-binary sweep of the Stream string helpers recovered 65 std::string members off a non-stack base across all serializers -- ZERO have any sibling member inside (N, N+0x1c), and 51 of the 52 that have a next member have it at exactly N+0x1c (the one exception, StrategyServer KeyPath @+0x134, is +0x20 on the Write side and +0x1c on the Read side, i.e. the writer skips a member). There is no 0x18 instantiation, no empty-base variant and no game-local string class. [verified] +constexpr uint32_t std_string_sizeof = 0x0000001c; // thiscall vector::_Reserve/grow for a 44-byte element type. Shared with FUN_0086dec0, so it may be a COMDAT-folded body -- do NOT assume it is ObservedTech-specific. [mapped] constexpr uint32_t vector_44B_grow = 0x003b5820; diff --git a/guides/re-windows-2000s-howto.md b/guides/re-windows-2000s-howto.md index f83fbb2..afc1fa1 100644 --- a/guides/re-windows-2000s-howto.md +++ b/guides/re-windows-2000s-howto.md @@ -117,7 +117,7 @@ call-outs cite `findings/` and `~/sots-engine/docs/`. **1c. Library triage.** Apply Ghidra's VS2010 x86 FID databases (plus threatrack's) or FLIRT via ApplySig to fence off CRT/STL/D3DX code; anything left is game code. STL template instantiations compiled from headers won't match a library signature — recognise them by shape instead: MSVC-2010 `std::vector` = `{begin, end, cap}`, `std::list` = `{head*, size}`, red-black node = `{left, parent, right, key…, color/isnil tail}`, `std::string` = 0x1c bytes on x86 with the SSO union and `_Mysize`/`_Myres`. -> **Our experience:** our own notes disagree on whether `_Bx` starts at +0 or +4 (allocator stub) in this build (`struct-recovery.md` §0 vs `turn-spine.md` §1.1). The reliable lever is the `_Myres >= 16 ? _Ptr : _Buf` branch in any string consumer (`Stream::WriteString` @ 0x008b9d70), which pins both offsets. Verify per binary; do not trust a table. +> **Our experience — settled 2026-09-08.** `_Bx` starts at **+0** in this build: `_Bx@0, _Mysize@0x10, _Myres@0x14, _Alval@0x18`, sizeof `0x1c`. The lever that pins the field offsets is the `_Myres >= 16 ? _Ptr : _Buf` branch in any string consumer (`Stream::WriteString` @ 0x008b9d70). The lever that pins the **size** is different and easy to miss: `_Alval` is an empty allocator, so it is never loaded or stored and no touch-based analysis can see it. Size the type from an *enumeration* instead — a container stride (`vector` walk `add esi,0x1c` @ 0x00699c29), a by-value argument's frame spacing (`PostEvent` strings at `[ebp+8]`/`[ebp+0x24]`, `RET 0x4c`), or a copy constructor / serializer that lists every member. Sizing from touched fields undercounts by exactly the trailing allocator word — that is how a lane got 0x18 for the same type. Note the allocator is **trailing** in this build's `std::vector` too (`{_Myfirst,_Mylast,_Myend,_Alval}` = 0x10), which is the opposite of the MSVC `_String_val`/`_Vector_val` allocator-first shape you will read about; verify per binary, do not trust a table. **1d. Strings are the map.** Config keys, file names, log format strings and — decisively — **serialization tag names**. If the save format is tagged, each `Read/Write` references dozens of a struct's tag strings in order; rank functions by distinct tag xrefs, decompile, and read `this+offset → tag → type` straight off. This is the single highest-yield lever for the object model, and it is under-documented in the literature (see §4). diff --git a/tools/strfootprint.py b/tools/strfootprint.py new file mode 100644 index 0000000..b2e0ea5 --- /dev/null +++ b/tools/strfootprint.py @@ -0,0 +1,210 @@ +#!/usr/bin/env python3 +"""Whole-binary audit of the std::string member footprint in Sword of the Stars.exe. + +Why this exists +--------------- +`sizeof(std::string)` is load-bearing: it is embedded in ~65 recovered class +layouts, and a 4-byte error in it silently shifts every field that follows. +Lane X's ObservedTech work reported the embedded string as 0x18 bytes, against +the campaign-wide 0x1c. Arguing about it from one class is how you get a +plausible answer instead of a true one, so this settles it from the whole +binary at once. + +Method +------ +Every serialiser in this exe hands a member pointer to one of the Mars::Stream +primitive helpers with the same idiom: + + push 0xff ; default arg + lea ,[+] ; &this->member + push + push ; -> .rdata "otch", "pswd", ... + push + call WriteString / ReadString / WriteBool / ... + +So for each helper call we recover (base register, displacement, tag). Then, +per (function, base register), we take every OTHER displacement the same +function touches off that register. If sizeof(std::string) were 0x18 for some +instantiation, that class would necessarily have a real member somewhere in +(N+4 .. N+0x1b). The audit is: does one exist, anywhere? + +Two things must be excluded or the answer is noise: + * ebp/esp bases -- those are stack temporaries, not class members. (A local + string at [ebp-0x2c] shows _Mysize at -0x1c and _Myres at -0x18, which + looks exactly like two "members" inside the span.) + * N+0x10 and N+0x14 -- the string's OWN _Mysize/_Myres, read inline whenever + the compiler inlines the SSO `_Myres >= 16 ? _Ptr : _Buf` test. + +Result (2026-09-08, lane S): 65 string members off a non-stack base, ZERO with +a sibling inside the 0x1c span, and 51 of the 52 that have a next member have +it at exactly +0x1c. One layout, 0x1c, binary-wide. + +Usage: uv run python3 tools/strfootprint.py [--all] + --all also lists every recovered string member and its gap. +""" +import json +import os +import struct +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +REPO = os.path.dirname(HERE) +sys.path.insert(0, HERE) +from x86disp import decode, Desync, load_pe, EXE, FUNCS # noqa: E402 + +# Mars::Stream primitive wrappers (struct-recovery.md S0). All take +# (stream, tagname, &member, default) and are called from the class Read/Write. +HELPERS = { + 0x008b9d70: "WriteString", 0x008b9d90: "ReadString", + 0x008b9c20: "WriteBool", 0x008b9c00: "ReadBool", + 0x008b9be0: "WriteFloat", 0x008b9bc0: "ReadFloat", + 0x008b9d50: "WriteInt", 0x008b9d20: "ReadInt", + 0x008b9d00: "WriteInt16", 0x008b9cd0: "ReadInt16", + 0x008b9c60: "WriteInt64", 0x008b9c40: "ReadInt64", +} +STRING_HELPERS = ("WriteString", "ReadString") +STR_SIZE = 0x1c +SSO_OWN = (0x10, 0x14) # the string's own _Mysize / _Myres +STACK_BASES = ("ebp", "esp") + + +# ---------------------------------------------------------------- .rdata reader +def _load_rdata(): + data = open(EXE, "rb").read() + pe = struct.unpack_from("= rsz: + return None + o = ro + d + e = _DATA.find(b"\0", o, o + maxn) + if e < 0: + return None + try: + return _DATA[o:e].decode("ascii") + except UnicodeDecodeError: + return None + return None + + +def main(): + show_all = "--all" in sys.argv + _, secs = load_pe(EXE) + sva, eva, buf, _ = secs[0] + funcs = json.load(open(FUNCS)) + starts = sorted((int(k, 16), v[0]) for k, v in funcs.items()) + svas = [s[0] for s in starts] + rd_lo, rd_hi = 0x9DD000, 0xAD9000 + + records, touch = [], {} + for idx, (fva, nm) in enumerate(starts): + if not (sva <= fva < eva): + continue + fend = svas[idx + 1] if idx + 1 < len(starts) else eva + ins, i, end = [], fva - sva, fend - sva + while i < end: + try: + ln, info = decode(buf, i, len(buf)) + except Desync: + break + ins.append((sva + i, ln, info, buf[i:i + ln])) + i += ln + if not any(r[0] == 0xE8 and l == 5 and + (v + 5 + struct.unpack_from(" disp and (d - disp) not in SSO_OWN) + inside = [d for d in others if d < disp + STR_SIZE] + if inside: + violations.append((fva, nm, base, disp, tag, helper, inside)) + if others: + g = others[0] - disp + gaps[g] = gaps.get(g, 0) + 1 + if show_all and (fva, base, disp, helper) not in seen: + seen.add((fva, base, disp, helper)) + nxt = f"+0x{others[0]:x}" if others else "-" + print(f"0x{fva:08x} {nm:26s} {base} +0x{disp:<6x} {tag!r:12s} " + f"{helper:12s} next={nxt}") + + print(f"\nstring helper call sites : {len(strs)}") + print(f" resolved to a class member offset : {len(members)}") + print(f" stack temporaries (ebp/esp base) : {stack}") + print(f" offset not reached by a plain lea : {unresolved}") + print(f"\nmembers with a sibling inside (N, N+0x{STR_SIZE:x}) " + f": {len(violations)} <-- must be 0 for sizeof(std::string)==0x1c") + for v in violations: + print(f" 0x{v[0]:08x} {v[1]} {v[2]}+0x{v[3]:x} {v[4]!r} " + f"inside={[hex(x) for x in v[6]]}") + print("\ngap from a string member to the next member on the same base:") + for g in sorted(gaps): + print(f" +0x{g:<4x} : {gaps[g]}") + return 1 if violations else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/verify/save-reader/SAVE_FORMAT.md b/verify/save-reader/SAVE_FORMAT.md index d2833b1..2941b0a 100644 --- a/verify/save-reader/SAVE_FORMAT.md +++ b/verify/save-reader/SAVE_FORMAT.md @@ -165,6 +165,20 @@ satl satk`). Objective records (Player `odes`/`owep`/`otch`, each a framed VectorHelper of `"."` elements): `odes` = `otnF otnL odid opid`; `owep` = `otnF otnL odet owep`(string) `owith`; `otch` = `otnF otnL odet otch`(string) `owith`. + +> **Confirmed against the binary (2026-09-08, lane S).** `otch` elements are +> `Game::ObservedTech` (`Write` `0x00817cf0`, `Read` `0x00817c40`, `sizeof` `0x2c`) and `owep` +> elements are `Game::ObservedWeapon` (`Write` `0x00817bc0`, `Read` `0x00817b10`) — the same +> element shape with a different string tag. The serializers emit exactly the order this file +> already had. In-memory types, which the on-disk framing does not reveal: `otnF` and `otnL` are +> **`uint16`** widened to int by the stream (`movzx` then stream `vft+0x24`); `odet` is a +> **`bool`** written through `WriteBool`, i.e. 1 byte plus 3 NUL joint-padding bytes, not an +> int32. `save_reader.py` types `odet` as `int`, which is **benign and deliberate**: `odet` is a +> 4-character tag, so a bool item and an int item are both `pad4(4+4+1) == pad4(4+4+4) == 12` +> bytes and the little-endian value is identical (§2's stated "bool and int have identical item +> sizes for a name whose length is a multiple of 4"). Re-typing it would change nothing on +> disk; it is recorded here so the *reimplementation* uses a `bool`, and so nobody later +> "discovers" the same 3 padding bytes as a missing field. `NdGr2` (node grid): `paths{"." n, n×"."{npt npid npfr npto npctm npcby npdtn npdtf npenp npuse nptf}}` then `nextid`(int). `BQ` (system build queue): `ords{"." n, n×"."{desID con conleft sav ordID}}` — count 0 in turn1..3.