diff --git a/campaign/open-questions.md b/campaign/open-questions.md index 19c3658..330c87c 100644 --- a/campaign/open-questions.md +++ b/campaign/open-questions.md @@ -38,7 +38,7 @@ Each links to the finding that raised it. Promoted to backlog or closed by **re- - **SAVE_FORMAT tag corrections (fix Python reader + spec)** — real on-disk tags: `otnF` (not `ontF`) in Odes/Owep/Otch, `nextid` (not `nextId`) in NdGr2, Design = `FAIDes/DHide/DWep/DName`; `ords`/`wpts` are real tags. Python's positional R() matching hid these. RNG: float mapping `(float)(y*2^-32)`, `next_int` mask, and lazy-vs-eager twist at `left==0` still need binary confirmation. (from [[mars-stream]]) - **RESOLVED: tech `allows` default** — unlisted species = 1.0 (confirmed in `FUN_005822d0` tree-creation roll, strategic-turn-internals §2); `game/data` uses 100. Still open from game/data: what the engine's converter does with the 34 malformed tokens (`force_right o`, `crew false`, `1.0f`, `0-5`, ``90\``); repeated scalars in a block are last-wins per the sequential if/else consumers (loader-prototypes §M3) — confirm on a fixture in compare mode. (from [[game-data]]) - **RESOLVED: SAVE_FORMAT tag names** — all corrections confirmed by bytes and applied to the Python reader + spec (§10). Residual doc debt: `findings/objects/save-editor-structs.md` and `verify/design-rules/SHIP_DESIGN_RULES.md` still quote R1's `ontF/faiDes/nextId` spellings (R1-provenance; annotate rather than rewrite). (from [[SAVE_FORMAT]]) -- **Notes disagree on MSVC-2010 `std::string` layout** — `struct-recovery.md` §0 vs `turn-spine.md` §1.1 give different offsets; pin both from the `_Myres >= 16` SSO branch in `Stream::WriteString` and correct the loser. (from [[re-windows-2000s-howto]]) +- **RESOLVED (2026-09-08, lane S): `std::string` layout and size, once and for all** — `_Bx@0, _Mysize@0x10, _Myres@0x14, _Alval@0x18`, **sizeof 0x1c**, and there is exactly **one** instantiation in this binary. `struct-recovery.md` §0 was right (its §0 prose had `size`/`res` transposed — fixed); `turn-spine.md` §1.1 was wrong (already annotated); `loader-prototypes.md` and one `addresses.json` prototype said 0x18 — fixed. Lane X's `ObservedTech` 0x18 reading was a mis-attribution: `+0x24` is the string's trailing `_Alval`, not a data field, proved by three complete enumerations of the element (`ObservedTech::Write` 0x00817cf0, ctor 0x008562a0, copy ctor 0x0079a184) and generalised over the whole exe by `tools/strfootprint.py` (65 string members, 0 collisions inside the 0x1c span, 51/52 gaps exactly 0x1c). **Zero recovered struct tables were wrong** — `pswd` included. Standing rule that follows: *never size a struct member from the offsets the code touches* — this build's STL puts the empty allocator **last** in both `string` (0x1c) and `vector` (0x10), and an empty allocator is never loaded or stored, so a touch-scan undercounts by 4 every time. See `observedtech-append.md` §9. (from [[re-windows-2000s-howto]], [[observedtech-append]]) - **RESOLVED: formula gaps (all 8)** — see [[formula-gaps]]; game/sim's low-confidence functions can now be pinned. **RESOLVED: std::string** = `_Bx@0,_Mysize@0x10,_Myres@0x14,_Alval@0x18`, sizeof 0x1c (struct-recovery §0 right; turn-spine §1.1 WRONG — annotated). **CORRECTION:** the 116-entry table @0x00a19718 is a name-membership list, not the effects table; use `g_TechIdNames` (196) + `OnTechResearched`. Open: values of the 6-entry AI-tech bonus table (0x00a17888); producer of ServerPlayer +0x224/+0x228/+0x22c beyond the setup-record copy (0x0077b620). (from [[tech-effects]]) - **RESOLVED: RNG semantics** — `Seed` is `thiscall(this, uint32)` RET 4 (MT19937 init + immediate twist); `Twist` takes `this` in ECX only; **twist is LAZY** (`if (left==0) Twist()` inside the draw), `left` lives at `+0x9c4`, confirming the save blob layout. **float mapping, needs a targeted check (not a red flag):** `NextFloat` decompiles as returning `float10`, but that is simply how x86 float returns look (value in `ST(0)`), so it is weak evidence of extended-precision *computation*; a single `y * 2^-32` multiply rounds once either way. B3 should still compare the mapping explicitly (exact constant and whether the draw is `y*2^-32`, `(y>>8)*2^-24`, or divided by 2^32-1), and the x87-vs-SSE float-parity policy from the RE how-to is still needed before the x64 standalone. `RNG_NextInt` signature still unverified. (own Ghidra pass) - **Struct-modelling hazard (found by M2)** — an MSVC-2010 `std::vector` member is **three words**, so a naive C translation put the next pointer at `+0x18` when it is really at `+0x14`; the hook silently reported the unmodelled word (string bytes `"TION"`) as a real field. Any hand-modelled game struct must pin its size with `static_assert` and account for 3-word vectors. (from [[M2]]) diff --git a/findings/objects/struct-recovery.md b/findings/objects/struct-recovery.md index f7d5cb9..3c26421 100644 --- a/findings/objects/struct-recovery.md +++ b/findings/objects/struct-recovery.md @@ -68,8 +68,17 @@ Readers accept legacy tags (`ISuit`, `Income`, `HPop`, `Bats`, `Builds`, `Clr`, `NShps`, `SysID`, `TrdID`, `Caps`, `GtTrf`, `FtSens`, `FtInc`, `Pris`, `NumPlgs`, `lcid`, `morev`, `cme`) by reading them into scratch/NULL — these are pre-1.8 fields, NOT members. -Common Mars/MSVC layouts seen: `std::string` = 0x1c bytes (MSVC10 `_Bx` union@0, size@0x14, res@0x18; -`FUN_008b9d70` does the `res>=16 ? heap : sso` check); `std::vector` = {begin@0, end@4, cap@8}; +Common Mars/MSVC layouts seen: **`std::string` = 0x1c bytes** — `_Bx` union@0 (16-byte SSO buffer, or a +`char*` when `_Myres >= 16`), `_Mysize`@**0x10**, `_Myres`@**0x14**, `_Alval`@0x18 (empty allocator, occupies +a word, never read or written). `FUN_008b9d70` does the `res>=16 ? heap : sso` check. +*(2026-09-08: this line previously transposed the two to `size@0x14, res@0x18`; the 0x10/0x14 offsets are the +verified ones — `Stream::WriteString` `cmp [str+0x14],0x10`, `basic_string::assign` 0x00425550.)* +The allocator is **trailing** in this build's STL, in strings and vectors alike, so it is invisible to any +scan of "which offsets does the code touch" and costs exactly 4 bytes if you size a member that way. See +`findings/subsystems/observedtech-append.md` §9 for the whole-binary audit (`tools/strfootprint.py`): 65 +`std::string` members across every serializer, zero with a sibling field inside the 0x1c span, 51 of 52 +measurable inter-member gaps exactly 0x1c. There is one string layout in this binary. +`std::vector` = {begin@0, end@4, cap@8, `_Alval`@0xc — 0x10 bytes}; `std::map/set` node = {left@0, parent@4, right@8, key@0xc, value@0x10, …, color/isnil bytes at tail}; `std::list` = {head*@0, size@4}. `Mars::NetworkObject` = {vptr@0, int id@4}. @@ -371,6 +380,32 @@ Four `std::list`s: `+4 list` (count `defc2`@+8, items `def`), ### 2.5 `Game::TechTree` — Write `FUN_005890a0` (tags `NumTechs`, `TNm`, `NumBrs`; per-tech body in a sub-writer not decompiled here; logs "TechTree: Tech %d not found saving tech tree"). +### 2.6 `Game::ObservedTech` (0x2c) — Write `0x00817cf0`, Read `0x00817c40` +Elements of `ServerPlayer::otch`, the `vector` at `ServerPlayer+0x274`. Polymorphic: +vftable `0x00a2439c`, RTTI `.?AVObservedTech@Game@@`, slots `{[0] 0x00793610 dtor, [1] Read, [2] Write}`. + +| off | type | save name | notes | +|---|---|---|---| +| 0x00 | vptr | — | `0x00a2439c` | +| 0x04 | `uint16` | `otnF` | turn first observed; widened to int32 on disk | +| 0x06 | `uint16` | `otnL` | turn last observed; widened to int32 on disk | +| 0x08 | `bool` | `odet` | **one byte** (+3 pad); `WriteBool`/`ReadBool` | +| 0x0c..0x27 | `std::string` | `otch` | tech name; `_Mysize`@0x1c, `_Myres`@0x20, `_Alval`@0x24 | +| 0x28 | `int` | `owith` | last member; `0x28 + 4 = 0x2c` = sizeof, no slack | + +`sizeof` = `0x2c` (44), pinned three ways by lane X (magic divide `0x2e8ba2e9 sar 3` at `0x0087239f`, +`imul reg,reg,0x2c`, search stride `add edi,0x2c`); the member map is the serializer's own order. +`Game::ObservedWeapon` (Write `0x00817bc0`, Read `0x00817b10`) is the identical element with tag `owep` +in place of `otch`; `odes` elements are the smaller `otnF otnL odid opid` record. + +Appended by `RecordObservedTech` `0x007ba1a0`, which **de-duplicates by tech name** — a reimplementation +that just `push_back`s will diverge on re-observation. It writes `otnF` and `otnL` from the *same* source +word on first sighting, so first-seen == last-seen initially. + +**Do not read `+0x24` as a field.** It is the name string's trailing `_Alval`. See §0 and +`findings/subsystems/observedtech-append.md` §9 — this is the class that produced the 0x18-vs-0x1c +`std::string` scare, and the resolution is that `0x1c` was right everywhere. + --- ## 3. `Game::StarFleet` (R1 `SimFleetDetails`) — Write `FUN_00701070`, Read `FUN_00702470`; `this` = obj+8 diff --git a/findings/subsystems/loader-prototypes.md b/findings/subsystems/loader-prototypes.md index c83f568..6bd13b8 100644 --- a/findings/subsystems/loader-prototypes.md +++ b/findings/subsystems/loader-prototypes.md @@ -12,7 +12,9 @@ Raw decompiles (before rename) are in `/srv/re-lab/handoff/loader-decompiles/rec decompiles in `/srv/re-lab/handoff/loader-decompiles/verify/.c`. Conventions: `thiscall` = `this` in ECX, args pushed right-to-left, callee pops (`RET n`); `cdecl` = all on stack, caller -pops. `std::string` = MSVC-2010 layout (16-byte SSO buffer / pointer, size @+0x10, capacity @+0x14, 0x18 bytes). +pops. `std::string` = **0x1c bytes**: 16-byte SSO buffer / pointer `_Bx` @+0, `_Mysize` @+0x10, `_Myres` @+0x14, +`_Alval` (empty allocator, never read or written) @+0x18. *Corrected 2026-09-08 — this line said 0x18, which is +the extent of the fields the code touches, not the size of the object; see `observedtech-append.md` §9.* "Case-insensitive" always means MSVCR100 `_stricmp` (ASCII fold). --- diff --git a/findings/subsystems/observedtech-append.md b/findings/subsystems/observedtech-append.md index 0b2a722..b039219 100644 --- a/findings/subsystems/observedtech-append.md +++ b/findings/subsystems/observedtech-append.md @@ -168,30 +168,49 @@ COL `0x00a81c78` → type descriptor `0x00aeede4` → **`.?AVObservedTech@Game@@ `ObservedTech` is polymorphic and its first word is a vptr, not a data field — which the on-disk shape does not tell you. -| offset | size | evidence | -|---|---|---| -| `+0x00` | 4 | vptr `0x00a2439c`; ctor writes it, RTTI-confirmed | -| `+0x04` | 2 | `mov word [eax-0x28],cx` at `0x007ba2b0` (`eax` = `_Mylast`, element = `_Mylast-0x2c`), source `[ebx+0xc]` | -| `+0x06` | 2 | `mov word [eax-0x26],dx` at `0x007ba2c6`, **same source word** `[ebx+0xc]` | -| `+0x08` | 4 | **unaccounted** | -| `+0x0c..+0x23` | 0x18 | `std::string` (the `otch` tech name). Object base is `+0x0c`, MSVC layout `{_Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14}`. Ctor writes `[+0x0c]=0`, `[+0x1c]=0`, `[+0x20]=0xf`; the search loop reads `[+0x1c]` as the length and calls the compare with `this = +0x0c`; the post-append assign uses `lea ecx,[_Mylast-0x20]` = `+0x0c` | -| `+0x24` | 4 | **unaccounted** | -| `+0x28` | 4 | **unaccounted** | +> **Superseded 2026-09-08 by lane S — see §9.** The table as first written called the +> embedded string 0x18 bytes and left `+0x08`, `+0x24`, `+0x28` unaccounted. `+0x24` is not a +> field: it is the string's own trailing allocator word. The corrected map is below; the +> original reasoning is kept in §9 because the way it went wrong is the useful part. -Note the string here is **0x18 bytes**, not the 0x1c the `ServerPlayer::pswd` row in -`struct-recovery.md` implies — three separate reads inside this element agree on -`{buf16, _Mysize@+0x10, _Myres@+0x14}`. Worth re-checking `pswd` against that. +| offset | size | member | evidence | +|---|---|---|---| +| `+0x00` | 4 | vptr `0x00a2439c` | ctor writes it, RTTI-confirmed | +| `+0x04` | 2 | `uint16 otnF` (turn first observed) | `mov word [eax-0x28],cx` at `0x007ba2b0` (`eax` = `_Mylast`, element = `_Mylast-0x2c`), source `[ebx+0xc]`; tag from `ObservedTech::Write` | +| `+0x06` | 2 | `uint16 otnL` (turn last observed) | `mov word [eax-0x26],dx` at `0x007ba2c6`, **same source word** `[ebx+0xc]` — first sighting sets first == last | +| `+0x08` | 1 | `bool odet` | ctor stores a **byte** (`mov [esi+0x8],bl`, `0x008562f4`); copy-ctor copies a byte; serialised with `WriteBool`/`ReadBool` | +| `+0x0c..+0x27` | 0x1c | `std::string otch` (tech name) | object base `+0x0c`, MSVC `{_Bx[16] @0, _Mysize @0x10, _Myres @0x14, _Alval @0x18}`. Ctor writes `[+0x0c]=0`, `[+0x1c]=0`, `[+0x20]=0xf`; the search loop reads `[+0x1c]` as the length and calls compare with `this = +0x0c`; the post-append assign uses `lea ecx,[_Mylast-0x20]` = `+0x0c`. `+0x24` is `_Alval` — never read, never written, by anything | +| `+0x28` | 4 | `int owith` | ctor zeroes it; `ObservedTech::Write` emits it last | -The four on-disk ints (`otnF`, `otnL`, `odet`, `owith`) have to map onto `+0x04`/`+0x06` (two -16-bit fields) and the three 4-byte slots `+0x08`, `+0x24`, `+0x28`. **That mapping is not -determined here and is deliberately not guessed.** The one suggestive observation, flagged as a -*hypothesis only*: `+0x04` and `+0x06` are two adjacent 16-bit fields written from the same -source word on first observation, which is the shape you would expect of a first-seen / -last-seen turn pair — but nothing here proves it. +`4 + 2 + 2 + 1(+3 pad) + 0x1c + 4 = 0x2c` exactly — sizeof is fully accounted for, with no +padding slack and no unaccounted field. -What would settle the mapping: read `ObservedTech`'s `Read`/`Write` serializer, where the -member order is explicit. Lane P's `observed_techs` region byte delta remains a valid live -cross-check and should now come back as exactly 44 per completion. +### Where the mapping came from — the serializer + +`Game::ObservedTech`'s vftable `0x00a2439c` is the usual 3 slots +`{ [0] 0x00793610 scalar deleting dtor, [1] 0x00817c40 Read, [2] 0x00817cf0 Write }`. +`Write` enumerates the entire object, in order, and touches nothing else: + +``` +0x00817cff movzx ecx,word [edi+0x04] push 0xa2b38c "otnF" -> stream vft+0x24 (int) +0x00817d0f movzx ecx,word [edi+0x06] push 0xa2b384 "otnL" -> stream vft+0x24 (int) +0x00817d26 lea eax,[edi+0x08] push 0xa2b36c "odet" -> WriteBool 0x008b9c20 +0x00817d37 lea ecx,[edi+0x0c] push 0xa2b3e8 "otch" -> WriteString 0x008b9d70 +0x00817d46 mov eax,[edi+0x28] push 0xa2b364 "owith" -> stream vft+0x24 (int) +``` + +`Read` (`0x00817c40`) is the exact mirror: `otnF`/`otnL` read as ints and stored back with +16-bit `mov word [ebx],ax`, `odet` through `ReadBool`, `otch` through `ReadString`, `owith` +reached as `add edi,0x28`. That matches the on-disk order `save_reader.py` already had +(`Otch = otnF otnL odet otch(string) owith`), which is a nice independent agreement between +the disassembly and the save oracle. + +`Game::ObservedWeapon` (`Write` `0x00817bc0`, `Read` `0x00817b10`) is the same element shape +with tag `owep` (`0x00a2b3f0`) in place of `otch` — a second instance of the identical 0x2c +layout. + +Lane P's `observed_techs` region byte delta remains a valid live cross-check and should come +back as exactly 44 per completion. ## 5. False-positive rate, honestly @@ -276,3 +295,111 @@ the stride evidence on the first four. `RecordObservedTech`, `vector_ObservedTech_push_back`, `vector_ObservedTech_assign`, `ObservedTech_ctor`, `vector_44B_grow`; `ServerPlayer_off_ObservedTechs` updated from "NOT PINNED" to `verified`. + +**Lane S round (2026-09-08).** Labels: `ObservedTech_Write` `0x00817cf0`, `ObservedTech_Read` +`0x00817c40`, `ObservedTech_scalar_deleting_dtor` `0x00793610`, `ObservedWeapon_Write` +`0x00817bc0`, `ObservedWeapon_Read` `0x00817b10`, `vector_ObservedTech_uninit_copy` `0x0079a150`, +`vector_string_find_by_name` `0x00699bd0`. Prototypes on the five class methods. Plate comments +carrying the full member map on the two serializers, the ctor, the dtor and the copy helper, and +the `sizeof(std::string) = 0x1c` fact on `Stream::WriteString` `0x008b9d70` and on the +`vector` stride site `0x00699bd0` — the two places a future lane is most likely to look. +`addresses.json` +10 entries (`std_string_sizeof`, `ObservedTech_Read/_Write/_dtor/_copy_ctor`, +`ObservedTech_off_TurnFirst/_TurnLast/_Detected/_With`, `ObservedWeapon_Write`), 4 corrected. + +--- + +## 9. `std::string` is 0x1c, not 0x18 — the correction, and why it mattered (lane S, 2026-09-08) + +§4 above originally reported the embedded string as **0x18 bytes**, against the campaign-wide +`_Bx@0, _Mysize@0x10, _Myres@0x14, _Alval@0x18`, sizeof `0x1c`. Lane X flagged it as "worth +re-checking" rather than asserting it, which was the right call: **`0x1c` is correct, and it is +correct everywhere in this binary.** `ObservedTech+0x24` is the string's own trailing allocator +word, not a data member. + +### Why the 0x18 reading looked right + +Everything lane X observed was accurate. The string's *live* fields really do stop at `+0x14` +(`_Bx@0`, `_Mysize@0x10`, `_Myres@0x14`), because `_Alval` is `std::allocator` — an empty +class. It occupies a word of the object but is never loaded or stored, so it is invisible to +any evidence based on **what the code touches**. Sizing a type from its accessed fields +undercounts it by exactly the tail padding. The same trap is live for `std::vector` in this +build, which is `{_Myfirst, _Mylast, _Myend, _Alval}` = `0x10` while only three words are ever +read (`events.md` already records the `_Alval` word at `EventStorage+0x10` and `+0x14`). + +### The three things that settle it inside `ObservedTech` + +Each is a *complete enumeration* of the object, which is the right instrument here — an +enumeration can show a field's **absence**, a touch-scan cannot. + +1. **`ObservedTech::Write` `0x00817cf0`** serialises `+0x04, +0x06, +0x08, +0x0c, +0x28`. + No `+0x24`. (`Read` `0x00817c40` mirrors it.) +2. **`ObservedTech_ctor` `0x008562a0`** initialises `+0x00, +0x04, +0x08, +0x0c(string), +0x28`. + No `+0x24` — while zeroing every other scalar in the object. +3. **The copy constructor**, inlined at `0x0079a184` inside the vector's uninitialised-copy + helper `FUN_0079a150`, copies `+0x04, +0x06, +0x08`, the string at `+0x0c`, and `+0x28`. + No `+0x24`. + +A 4-byte data member that the constructor, the copy constructor and the serializer all ignore +is not a data member. + +### Binary-wide check — `tools/strfootprint.py` + +One class is an anecdote, so the same question was put to the whole binary. Every serializer +hands member pointers to the `Mars::Stream` primitive helpers with a fixed idiom +(`lea r,[base+disp]; push r; push tag; push stream; call helper`), so the scanner recovers +`(base, disp, tag)` for every string site and then asks: does the same function touch any other +offset inside `(N, N+0x1c)` off the same base register? + +``` +string helper call sites : 241 + resolved to a class member offset : 65 + stack temporaries (ebp/esp base) : 30 + offset not reached by a plain lea : 146 + +members with a sibling inside (N, N+0x1c) : 0 + +gap from a string member to the next member on the same base: + +0x1c : 51 + +0x20 : 1 +``` + +**65 string members across every serializer in the exe, zero collisions, and 51 of the 52 +measurable gaps are exactly `0x1c`.** The single `+0x20` is `StrategyServer::KeyPath` at +`+0x134`, whose *Read* side gives `+0x1c` to `+0x150` — the writer simply skips a member. +There is no `0x18` instantiation, no empty-base-optimised variant, and no game-local string +class. One layout, `0x1c`. + +Two exclusions matter or the scan reports noise, and both are why a naive version of this +would have "confirmed" 0x18: + +* **`ebp`/`esp` bases are stack temporaries, not members.** A local string at `[ebp-0x2c]` + shows accesses at `-0x1c` and `-0x18` — which read exactly like two sibling fields inside + the span. All 30 such sites are excluded. +* **`N+0x10` and `N+0x14` are the string's own `_Mysize`/`_Myres`**, touched inline whenever + the compiler expands the `_Myres >= 16 ? _Ptr : _Buf` test at a call site. + +### Independent corroboration outside the scan + +* `FUN_00699bd0` walks a `vector` doing the SSO test at `[esi+0x14]`/`[esi]` — + element base *is* string base — and advances `add esi,0x1c` (`0x00699c29`). The stride of a + vector of strings **is** `sizeof(std::string)`. +* `EventStorage::PostEvent` `0x008862b0` takes two by-value `std::string`s at `[ebp+0x08]` and + `[ebp+0x24]` (spacing `0x1c`) and `RET 0x4c` = `2*0x1c + 5*4`. +* `MoraleEvent` is 0x50 bytes with its name string at `+0x34`: `0x34 + 0x1c = 0x50` exactly. + +### Blast radius — what actually had to change + +Nothing in any recovered struct table. Every string-bearing layout in `struct-recovery.md`, +`save-editor-structs.md`, `events.md` and `schema-gaps-resolved.md` already used `0x1c` spans +and `0x1c` inter-field gaps, and the binary agrees with all of them. `ServerPlayer::pswd` at +`0x2dc..0x2f7` — the row §4 asked to re-check — is **correct**: `Write` `0x008563e0` puts the +next member exactly `0x1c` above it. The corrections were confined to three prose statements +that had propagated the 0x18 number (`loader-prototypes.md` conventions line, the +`GlobalConst_ParseString` prototype in `addresses.json`, and §4 here) plus one transposition in +`struct-recovery.md` §0 that said `size@0x14, res@0x18` where the verified offsets are +`0x10`/`0x14`. + +The lesson worth carrying, and the reason this was worth chasing rather than reconciling: **do +not size a struct member from the offsets the code touches.** Trailing empty-allocator words in +this build's STL are invisible to a touch-scan and cost exactly 4 bytes every time. Size types +from an enumeration — a serializer, a constructor, a copy constructor, or a container stride. diff --git a/ghidra/addresses.json b/ghidra/addresses.json index 885af86..11aef4d 100644 --- a/ghidra/addresses.json +++ b/ghidra/addresses.json @@ -334,7 +334,7 @@ "name": "GlobalConst_ParseString", "addr": "0x008b7670", "convention": "cdecl", - "prototype": "void (std::string* storage, const char* text) /* *storage = text; MSVC2010 std::string 0x18 bytes {+0 char buf[16] | char* ptr when capacity > 15, +0x10 size, +0x14 capacity}; 146 keys (*_NAME, *_SOUND, *_TEXTURENAME, DERELICT_SECTION_nn ...); seen in the M1 trace */", + "prototype": "void (std::string* storage, const char* text) /* *storage = text; MSVC std::string is 0x1c bytes {+0 char buf[16] | char* ptr when capacity > 15, +0x10 _Mysize, +0x14 _Myres, +0x18 _Alval (empty allocator, never read/written)} -- see std_string_sizeof; 146 keys (*_NAME, *_SOUND, *_TEXTURENAME, DERELICT_SECTION_nn ...); seen in the M1 trace */", "status": "verified-by-trace", "source": "sots-engine docs/M1.md" }, @@ -3174,7 +3174,7 @@ "name": "ObservedTech_sizeof", "offset": "0x2c", "convention": "constant", - "prototype": "sizeof(Game::ObservedTech) = 0x2c (44 bytes). Confirmed independently by (a) the compiler's magic division by 44 (mov eax,0x2e8ba2e9; imul; sar edx,3) at 0x0087239f and 0x007b7339, (b) imul reg,reg,0x2c at 0x0087243a, 0x00872468, 0x007b735b, and (c) the iterator advance `add edi,0x2c` in RecordObservedTech's linear search at 0x007ba257. Matches the on-disk lower bound exactly (4 int + one 0x1c std::string = 44), so there is no padding slack.", + "prototype": "sizeof(Game::ObservedTech) = 0x2c (44 bytes). Confirmed independently by (a) the compiler's magic division by 44 (mov eax,0x2e8ba2e9; imul; sar edx,3) at 0x0087239f and 0x007b7339, (b) imul reg,reg,0x2c at 0x0087243a, 0x00872468, 0x007b735b, and (c) the iterator advance `add edi,0x2c` in RecordObservedTech's linear search at 0x007ba257. FULL MEMBER MAP, from ObservedTech_Write 0x00817cf0 / ObservedTech_Read 0x00817c40 (lane S 2026-09-08): +0x00 vptr 0x00a2439c; +0x04 uint16 otnF; +0x06 uint16 otnL; +0x08 bool odet (1 byte, +3 pad); +0x0c std::string otch (0x1c, so _Mysize at +0x1c, _Myres at +0x20, _Alval at +0x24); +0x28 int owith. 4 + 2 + 2 + 4 + 0x1c + 4 = 0x2c exactly, no padding slack and no unaccounted field.", "status": "verified", "source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)" }, @@ -3190,7 +3190,7 @@ "name": "ObservedTech_off_Name", "offset": "0x0c", "convention": "offset", - "prototype": "std::string (save tag `otch`, the tech name) at ObservedTech+0x0c, 0x18 bytes, spanning +0x0c..+0x23. MSVC layout relative to the string object: _Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14 -- i.e. ObservedTech+0x1c is the length and +0x20 the capacity. Evidence: ObservedTech_ctor 0x008562a0 writes [elem+0x0c]=0, [elem+0x1c]=0, [elem+0x20]=0xf; RecordObservedTech's search reads [elem+0x1c] as the length and calls compare with this=elem+0x0c; the post-append assign uses lea ecx,[_Mylast-0x20]. NOTE this string is 0x18 bytes, not the 0x1c implied by the ServerPlayer pswd row in struct-recovery.md. UNACCOUNTED in the element: +0x08, +0x24, +0x28 (4 bytes each) plus two 16-bit fields at +0x04/+0x06 written from one source word at 0x007ba2b0 / 0x007ba2c6 -- the mapping of the four on-disk ints (otnF, otnL, odet, owith) onto those slots is NOT determined.", + "prototype": "std::string (save tag `otch`, the tech name) at ObservedTech+0x0c, 0x1c bytes, spanning +0x0c..+0x27. MSVC layout relative to the string object: _Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14, _Alval @+0x18 -- i.e. ObservedTech+0x1c is the length, +0x20 the capacity, +0x24 the empty-allocator word (never read or written by anything). Evidence: ObservedTech_ctor 0x008562a0 writes [elem+0x0c]=0, [elem+0x1c]=0, [elem+0x20]=0xf; RecordObservedTech's search reads [elem+0x1c] as the length and calls compare with this=elem+0x0c; the post-append assign uses lea ecx,[_Mylast-0x20]. CORRECTION (lane S 2026-09-08): an earlier revision called this string 0x18 bytes and listed +0x24 as an unaccounted data field. It is not one -- three independent whole-object enumerations skip +0x24 entirely: ObservedTech_ctor 0x008562a0, ObservedTech_copy_ctor 0x0079a184, and ObservedTech_Write 0x00817cf0 (which serialises +0x04,+0x06,+0x08,+0x0c,+0x28 and nothing else). sizeof(std::string)=0x1c holds binary-wide; see std_string_sizeof.", "status": "verified", "source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)" }, @@ -3198,7 +3198,7 @@ "name": "RecordObservedTech", "addr": "0x007ba1a0", "convention": "thiscall", - "prototype": "void (this, ServerPlayer* observer, ?, std::string* techName) -- appends to observer->otch. Linear-searches observer->otch (ServerPlayer+0x274) with stride 0x2c comparing each element's name string; if not found, default-constructs an ObservedTech on the stack (0x008562a0) and push_backs it (0x007b7320 @0x007ba288), then writes two 16-bit fields into the new element at +0x04 and +0x06 from param_1+0xc. DIRECT CALLEE of ServerPlayer::OnTechResearched 0x00891790; also called from 0x007be228, 0x007be4e1, 0x007be535. This is the append lane P could not find.", + "prototype": "void (this, ServerPlayer* observer, ?, std::string* techName) -- appends to observer->otch. Linear-searches observer->otch (ServerPlayer+0x274) with stride 0x2c comparing each element's name string; if not found, default-constructs an ObservedTech on the stack (0x008562a0) and push_backs it (0x007b7320 @0x007ba288), then writes otnF (+0x04) and otnL (+0x06), both 16-bit, from the same source word param_1+0xc -- i.e. first-observed turn == last-observed turn on the first sighting, which is what the tag names now confirm. DIRECT CALLEE of ServerPlayer::OnTechResearched 0x00891790; also called from 0x007be228, 0x007be4e1, 0x007be535. This is the append lane P could not find. DE-DUPLICATING: appends only when no existing element carries that tech name, so a reimplementation must not naively push_back on re-observation.", "status": "verified", "source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)" }, @@ -3222,9 +3222,89 @@ "name": "ObservedTech_ctor", "addr": "0x008562a0", "convention": "thiscall", - "prototype": "Game::ObservedTech* (ObservedTech* this) -- default ctor; sets vptr 0x00a2439c and empties the name string.", + "prototype": "Game::ObservedTech* (ObservedTech* this) -- default ctor. Writes exactly: vptr 0x00a2439c at +0x00; dword 0 at +0x04 (otnF+otnL zeroed together); BYTE 0 at +0x08 (`mov [esi+0x8],bl`, 0x008562f4 -- odet is a bool, not an int); the empty string at +0x0c (_Buf[0]=0, _Mysize=0, _Myres=0xf, then assign(\"\") 0x00425550); dword 0 at +0x28 (owith). It never touches +0x24 -- that word is the string's _Alval.", "status": "verified", - "source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)" + "source": "findings/subsystems/observedtech-append.md (lane X; element map corrected + serializer read by lane S 2026-09-08)" + }, + { + "name": "ObservedTech_Write", + "addr": "0x00817cf0", + "convention": "thiscall", + "prototype": "void Game::ObservedTech::Write(Mars::Stream* s) RET 4. Vftable 0x00a2439c slot [2]. Serialises the whole object, in order and with nothing else: `otnF` = movzx word [this+0x04] via stream vft+0x24 (int); `otnL` = movzx word [this+0x06] via vft+0x24; `odet` = WriteBool 0x008b9c20 (&this[+0x08]); `otch` = WriteString 0x008b9d70 (&this[+0x0c]); `owith` = [this+0x28] via vft+0x24. THIS IS THE MEMBER MAP: there is no field at +0x24. Tag pointers 0x00a2b38c/0x00a2b384/0x00a2b36c/0x00a2b3e8/0x00a2b364.", + "status": "verified", + "source": "lane S own disassembly 2026-09-08" + }, + { + "name": "ObservedTech_Read", + "addr": "0x00817c40", + "convention": "thiscall", + "prototype": "void Game::ObservedTech::Read(Mars::Stream* s) RET 4. Vftable 0x00a2439c slot [1]. Mirror of ObservedTech_Write: `otnF`/`otnL` through stream vft+0x10 (int-by-ref) stored back as 16-bit (`mov word [ebx],ax`) at +0x04/+0x06, `odet` = ReadBool 0x008b9c00 (&this[+0x08]), `otch` = ReadString 0x008b9d90 (&this[+0x0c]), `owith` at +0x28 (reached as `add edi,0x28`).", + "status": "verified", + "source": "lane S own disassembly 2026-09-08" + }, + { + "name": "ObservedTech_dtor", + "addr": "0x00793610", + "convention": "thiscall", + "prototype": "ObservedTech* (ObservedTech* this, int flags) RET 4. Vftable 0x00a2439c slot [0], scalar deleting dtor. Inlines ~basic_string on the name at +0x0c (`cmp [esi+0x20],0x10` -> operator delete [esi+0x0c], then _Tidy: [esi+0x20]=0xf, [esi+0x1c]=0, byte [esi+0x0c]=0), restores the base vptr 0x009e22bc, and frees `this` when flags&1. The string is the ONLY member needing destruction -- confirming there is no second string or owned pointer in the element.", + "status": "verified", + "source": "lane S own disassembly 2026-09-08" + }, + { + "name": "ObservedTech_copy_ctor", + "addr": "0x0079a184", + "convention": "site", + "prototype": "site inside the vector uninitialised-copy helper FUN_0079a150(&_Alval, dest, src). The inlined copy constructor writes vptr 0x00a2439c, `mov word [dst+0x04],[src+0x04]`, `mov word [dst+0x06],[src+0x06]`, `mov byte [dst+0x08],[src+0x08]`, the string at +0x0c (via basic_string::assign 0x00425430), and `mov [dst+0x28],[src+0x28]`. Nothing is copied at +0x24 -- second independent proof that +0x24 belongs to the 0x1c string, not to a data member.", + "status": "verified", + "source": "lane S own disassembly 2026-09-08" + }, + { + "name": "ObservedTech_off_TurnFirst", + "offset": "0x04", + "convention": "offset", + "prototype": "uint16 otnF -- turn the tech was first observed. Widened to int on disk by stream vft+0x24. Written together with otnL from one source word at RecordObservedTech 0x007ba2b0.", + "status": "verified", + "source": "lane S: ObservedTech_Write 0x00817cf0 / ObservedTech_Read 0x00817c40" + }, + { + "name": "ObservedTech_off_TurnLast", + "offset": "0x06", + "convention": "offset", + "prototype": "uint16 otnL -- turn the tech was last observed. Widened to int on disk. Written at RecordObservedTech 0x007ba2c6 from the same source word as otnF.", + "status": "verified", + "source": "lane S: ObservedTech_Write 0x00817cf0 / ObservedTech_Read 0x00817c40" + }, + { + "name": "ObservedTech_off_Detected", + "offset": "0x08", + "convention": "offset", + "prototype": "bool odet -- ONE BYTE (ctor `mov [esi+0x8],bl` 0x008562f4; copy-ctor `mov byte` 0x0079a1a0), serialised through WriteBool/ReadBool, so on disk it is 1 byte + 3 NUL joint padding. Note the save reader types `odet` as int; for a 4-character tag a bool item and an int item are the same 12 bytes and the value is identical, so that is benign, not a parse error.", + "status": "verified", + "source": "lane S: ObservedTech_Write 0x00817cf0 / ObservedTech_ctor 0x008562a0" + }, + { + "name": "ObservedTech_off_With", + "offset": "0x28", + "convention": "offset", + "prototype": "int owith -- last member of the element; serialised through stream vft+0x24 with tag 0x00a2b364. +0x28..+0x2b is the tail of the 0x2c-byte element, which is why sizeof is 0x2c with no padding slack.", + "status": "verified", + "source": "lane S: ObservedTech_Write 0x00817cf0 / ObservedTech_Read 0x00817c40" + }, + { + "name": "ObservedWeapon_Write", + "addr": "0x00817bc0", + "convention": "thiscall", + "prototype": "void Game::ObservedWeapon::Write(Mars::Stream* s) RET 4. Byte-for-byte the same shape as ObservedTech_Write with tag `owep` (0x00a2b3f0) in place of `otch`: otnF u16 @+0x04, otnL u16 @+0x06, odet bool @+0x08, owep std::string (0x1c) @+0x0c, owith int @+0x28. Reader is 0x00817b10. Independent second instance of the same 0x2c element shape.", + "status": "verified", + "source": "lane S own disassembly 2026-09-08" + }, + { + "name": "std_string_sizeof", + "offset": "0x1c", + "convention": "constant", + "prototype": "sizeof(std::basic_string) = 0x1c (28) binary-wide, ONE layout only: _Bx (16-byte SSO union, char[16] or char*) @+0x00, _Mysize @+0x10, _Myres @+0x14, _Alval (empty allocator) @+0x18. The allocator word is trailing, never read and never written -- the same allocator-last shape as this build's std::vector {_Myfirst,_Mylast,_Myend,_Alval} = 0x10. Evidence: (a) Stream::WriteString 0x008b9d76 `cmp [str+0x14],0x10` / `mov eax,[str]` and basic_string::assign 0x00425550 (_Mysize +0x10, _Myres +0x14) fix the field offsets; (b) the vector scan loop FUN_00699bd0 advances `add esi,0x1c` (0x00699c29) while doing the SSO test at [esi+0x14]/[esi] -- element base == string base, so the stride IS the sizeof; (c) PostEvent 0x008862b0 takes two by-value strings at [ebp+0x08] and [ebp+0x24] and RET 0x4c = 2*0x1c + 5*4; (d) a whole-binary sweep of the Stream string helpers recovered 65 std::string members off a non-stack base across all serializers -- ZERO have any sibling member inside (N, N+0x1c), and 51 of the 52 that have a next member have it at exactly N+0x1c (the one exception, StrategyServer KeyPath @+0x134, is +0x20 on the Write side and +0x1c on the Read side, i.e. the writer skips a member). There is no 0x18 instantiation, no empty-base variant and no game-local string class.", + "status": "verified", + "source": "lane S 2026-09-08; scanner tools/strfootprint.py" }, { "name": "vector_44B_grow", diff --git a/ghidra/generated/sots_addresses.h b/ghidra/generated/sots_addresses.h index 2adca21..d1674ec 100644 --- a/ghidra/generated/sots_addresses.h +++ b/ghidra/generated/sots_addresses.h @@ -1,5 +1,5 @@ // GENERATED — do not edit. Facts about Sword of the Stars.exe (GOG 1.8.1). -// Source: sots-re ghidra/addresses.json @ 1a58bcd, generated 2026-09-08 by tools/gen_addresses.py +// Source: sots-re ghidra/addresses.json @ d523d27, generated 2026-09-08 by tools/gen_addresses.py // Runtime address = (uintptr_t)GetModuleHandle(NULL) + RVA (the exe is ASLR-relocated). #pragma once #include @@ -89,7 +89,7 @@ constexpr uint32_t GlobalConst_ParseColour = 0x004b7110; constexpr uint32_t GlobalConst_ParseRect = 0x004b7040; // cdecl void (float xyz[3], const char* text) /* '%f %f %f'; 5 keys (HIVER_SPAWN_*_GATE_HEIGHT / _SHIP_OFFSET ...); inferred from the M1 trace */ [verified-by-trace] constexpr uint32_t GlobalConst_ParseVec3 = 0x004b7080; -// cdecl void (std::string* storage, const char* text) /* *storage = text; MSVC2010 std::string 0x18 bytes {+0 char buf[16] | char* ptr when capacity > 15, +0x10 size, +0x14 capacity}; 146 keys (*_NAME, *_SOUND, *_TEXTURENAME, DERELICT_SECTION_nn ...); seen in the M1 trace */ [verified-by-trace] +// cdecl void (std::string* storage, const char* text) /* *storage = text; MSVC std::string is 0x1c bytes {+0 char buf[16] | char* ptr when capacity > 15, +0x10 _Mysize, +0x14 _Myres, +0x18 _Alval (empty allocator, never read/written)} -- see std_string_sizeof; 146 keys (*_NAME, *_SOUND, *_TEXTURENAME, DERELICT_SECTION_nn ...); seen in the M1 trace */ [verified-by-trace] constexpr uint32_t GlobalConst_ParseString = 0x004b7670; // cdecl void (void) /* called once from Application::Initialize; sets g_GlobalConstsLoaded */ [verified] constexpr uint32_t GlobalConsts_LoadAll = 0x004b76d0; @@ -799,20 +799,40 @@ constexpr uint32_t Mars_Vec3_NormaliseEpsilon = 0x005e1ef8; constexpr uint32_t StrategyServer_MoveFleet_straight_leg = 0x003da0f2; // site site inside StrategyServer::MoveFleet /* the move != distance branch: exactly two roundings per component, tmp.c = float32(dir.c * move) then pos.c = float32(pos.c + tmp.c). `move` is reloaded from a float32 slot. The sibling branch (move == distance, an EXACT float compare) copies the destination's three words verbatim with mov, so an arrival never steps onto its destination. */ [verified] constexpr uint32_t StrategyServer_MoveFleet_position_update = 0x003da2ac; -// constant sizeof(Game::ObservedTech) = 0x2c (44 bytes). Confirmed independently by (a) the compiler's magic division by 44 (mov eax,0x2e8ba2e9; imul; sar edx,3) at 0x0087239f and 0x007b7339, (b) imul reg,reg,0x2c at 0x0087243a, 0x00872468, 0x007b735b, and (c) the iterator advance `add edi,0x2c` in RecordObservedTech's linear search at 0x007ba257. Matches the on-disk lower bound exactly (4 int + one 0x1c std::string = 44), so there is no padding slack. [verified] +// constant sizeof(Game::ObservedTech) = 0x2c (44 bytes). Confirmed independently by (a) the compiler's magic division by 44 (mov eax,0x2e8ba2e9; imul; sar edx,3) at 0x0087239f and 0x007b7339, (b) imul reg,reg,0x2c at 0x0087243a, 0x00872468, 0x007b735b, and (c) the iterator advance `add edi,0x2c` in RecordObservedTech's linear search at 0x007ba257. FULL MEMBER MAP, from ObservedTech_Write 0x00817cf0 / ObservedTech_Read 0x00817c40 (lane S 2026-09-08): +0x00 vptr 0x00a2439c; +0x04 uint16 otnF; +0x06 uint16 otnL; +0x08 bool odet (1 byte, +3 pad); +0x0c std::string otch (0x1c, so _Mysize at +0x1c, _Myres at +0x20, _Alval at +0x24); +0x28 int owith. 4 + 2 + 2 + 4 + 0x1c + 4 = 0x2c exactly, no padding slack and no unaccounted field. [verified] constexpr uint32_t ObservedTech_sizeof = 0x0000002c; // data Game::ObservedTech vftable. RTTI COL 0x00a81c78 -> type descriptor 0x00aeede4 = '.?AVObservedTech@Game@@'. Written to element+0x00 by ObservedTech_ctor 0x008562a0 -- so ObservedTech is polymorphic and its first word is the vptr, not a data field. [verified] constexpr uint32_t ObservedTech_vftable = 0x0062439c; -// offset std::string (save tag `otch`, the tech name) at ObservedTech+0x0c, 0x18 bytes, spanning +0x0c..+0x23. MSVC layout relative to the string object: _Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14 -- i.e. ObservedTech+0x1c is the length and +0x20 the capacity. Evidence: ObservedTech_ctor 0x008562a0 writes [elem+0x0c]=0, [elem+0x1c]=0, [elem+0x20]=0xf; RecordObservedTech's search reads [elem+0x1c] as the length and calls compare with this=elem+0x0c; the post-append assign uses lea ecx,[_Mylast-0x20]. NOTE this string is 0x18 bytes, not the 0x1c implied by the ServerPlayer pswd row in struct-recovery.md. UNACCOUNTED in the element: +0x08, +0x24, +0x28 (4 bytes each) plus two 16-bit fields at +0x04/+0x06 written from one source word at 0x007ba2b0 / 0x007ba2c6 -- the mapping of the four on-disk ints (otnF, otnL, odet, owith) onto those slots is NOT determined. [verified] +// offset std::string (save tag `otch`, the tech name) at ObservedTech+0x0c, 0x1c bytes, spanning +0x0c..+0x27. MSVC layout relative to the string object: _Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14, _Alval @+0x18 -- i.e. ObservedTech+0x1c is the length, +0x20 the capacity, +0x24 the empty-allocator word (never read or written by anything). Evidence: ObservedTech_ctor 0x008562a0 writes [elem+0x0c]=0, [elem+0x1c]=0, [elem+0x20]=0xf; RecordObservedTech's search reads [elem+0x1c] as the length and calls compare with this=elem+0x0c; the post-append assign uses lea ecx,[_Mylast-0x20]. CORRECTION (lane S 2026-09-08): an earlier revision called this string 0x18 bytes and listed +0x24 as an unaccounted data field. It is not one -- three independent whole-object enumerations skip +0x24 entirely: ObservedTech_ctor 0x008562a0, ObservedTech_copy_ctor 0x0079a184, and ObservedTech_Write 0x00817cf0 (which serialises +0x04,+0x06,+0x08,+0x0c,+0x28 and nothing else). sizeof(std::string)=0x1c holds binary-wide; see std_string_sizeof. [verified] constexpr uint32_t ObservedTech_off_Name = 0x0000000c; -// thiscall void (this, ServerPlayer* observer, ?, std::string* techName) -- appends to observer->otch. Linear-searches observer->otch (ServerPlayer+0x274) with stride 0x2c comparing each element's name string; if not found, default-constructs an ObservedTech on the stack (0x008562a0) and push_backs it (0x007b7320 @0x007ba288), then writes two 16-bit fields into the new element at +0x04 and +0x06 from param_1+0xc. DIRECT CALLEE of ServerPlayer::OnTechResearched 0x00891790; also called from 0x007be228, 0x007be4e1, 0x007be535. This is the append lane P could not find. [verified] +// thiscall void (this, ServerPlayer* observer, ?, std::string* techName) -- appends to observer->otch. Linear-searches observer->otch (ServerPlayer+0x274) with stride 0x2c comparing each element's name string; if not found, default-constructs an ObservedTech on the stack (0x008562a0) and push_backs it (0x007b7320 @0x007ba288), then writes otnF (+0x04) and otnL (+0x06), both 16-bit, from the same source word param_1+0xc -- i.e. first-observed turn == last-observed turn on the first sighting, which is what the tag names now confirm. DIRECT CALLEE of ServerPlayer::OnTechResearched 0x00891790; also called from 0x007be228, 0x007be4e1, 0x007be535. This is the append lane P could not find. DE-DUPLICATING: appends only when no existing element carries that tech name, so a reimplementation must not naively push_back on re-observation. [verified] constexpr uint32_t RecordObservedTech = 0x003ba1a0; // thiscall void (std::vector* this, ObservedTech* value) RET 4. Stride 0x2c. Calls vector_44B_grow 0x007b5820 when _Mylast==_Myend -- the realloc that moves all three vector words. Exactly one caller (RecordObservedTech), so this instantiation is not COMDAT-ambiguous. [verified] constexpr uint32_t vector_ObservedTech_push_back = 0x003b7320; // thiscall std::vector& operator=(const std::vector&) RET 4. Both callers pass ServerPlayer+0x274 (0x00878091 in the settings copy-out, 0x00892507 in the copy-in). [verified] constexpr uint32_t vector_ObservedTech_assign = 0x00472380; -// thiscall Game::ObservedTech* (ObservedTech* this) -- default ctor; sets vptr 0x00a2439c and empties the name string. [verified] +// thiscall Game::ObservedTech* (ObservedTech* this) -- default ctor. Writes exactly: vptr 0x00a2439c at +0x00; dword 0 at +0x04 (otnF+otnL zeroed together); BYTE 0 at +0x08 (`mov [esi+0x8],bl`, 0x008562f4 -- odet is a bool, not an int); the empty string at +0x0c (_Buf[0]=0, _Mysize=0, _Myres=0xf, then assign("") 0x00425550); dword 0 at +0x28 (owith). It never touches +0x24 -- that word is the string's _Alval. [verified] constexpr uint32_t ObservedTech_ctor = 0x004562a0; +// thiscall void Game::ObservedTech::Write(Mars::Stream* s) RET 4. Vftable 0x00a2439c slot [2]. Serialises the whole object, in order and with nothing else: `otnF` = movzx word [this+0x04] via stream vft+0x24 (int); `otnL` = movzx word [this+0x06] via vft+0x24; `odet` = WriteBool 0x008b9c20 (&this[+0x08]); `otch` = WriteString 0x008b9d70 (&this[+0x0c]); `owith` = [this+0x28] via vft+0x24. THIS IS THE MEMBER MAP: there is no field at +0x24. Tag pointers 0x00a2b38c/0x00a2b384/0x00a2b36c/0x00a2b3e8/0x00a2b364. [verified] +constexpr uint32_t ObservedTech_Write = 0x00417cf0; +// thiscall void Game::ObservedTech::Read(Mars::Stream* s) RET 4. Vftable 0x00a2439c slot [1]. Mirror of ObservedTech_Write: `otnF`/`otnL` through stream vft+0x10 (int-by-ref) stored back as 16-bit (`mov word [ebx],ax`) at +0x04/+0x06, `odet` = ReadBool 0x008b9c00 (&this[+0x08]), `otch` = ReadString 0x008b9d90 (&this[+0x0c]), `owith` at +0x28 (reached as `add edi,0x28`). [verified] +constexpr uint32_t ObservedTech_Read = 0x00417c40; +// thiscall ObservedTech* (ObservedTech* this, int flags) RET 4. Vftable 0x00a2439c slot [0], scalar deleting dtor. Inlines ~basic_string on the name at +0x0c (`cmp [esi+0x20],0x10` -> operator delete [esi+0x0c], then _Tidy: [esi+0x20]=0xf, [esi+0x1c]=0, byte [esi+0x0c]=0), restores the base vptr 0x009e22bc, and frees `this` when flags&1. The string is the ONLY member needing destruction -- confirming there is no second string or owned pointer in the element. [verified] +constexpr uint32_t ObservedTech_dtor = 0x00393610; +// site site inside the vector uninitialised-copy helper FUN_0079a150(&_Alval, dest, src). The inlined copy constructor writes vptr 0x00a2439c, `mov word [dst+0x04],[src+0x04]`, `mov word [dst+0x06],[src+0x06]`, `mov byte [dst+0x08],[src+0x08]`, the string at +0x0c (via basic_string::assign 0x00425430), and `mov [dst+0x28],[src+0x28]`. Nothing is copied at +0x24 -- second independent proof that +0x24 belongs to the 0x1c string, not to a data member. [verified] +constexpr uint32_t ObservedTech_copy_ctor = 0x0039a184; +// offset uint16 otnF -- turn the tech was first observed. Widened to int on disk by stream vft+0x24. Written together with otnL from one source word at RecordObservedTech 0x007ba2b0. [verified] +constexpr uint32_t ObservedTech_off_TurnFirst = 0x00000004; +// offset uint16 otnL -- turn the tech was last observed. Widened to int on disk. Written at RecordObservedTech 0x007ba2c6 from the same source word as otnF. [verified] +constexpr uint32_t ObservedTech_off_TurnLast = 0x00000006; +// offset bool odet -- ONE BYTE (ctor `mov [esi+0x8],bl` 0x008562f4; copy-ctor `mov byte` 0x0079a1a0), serialised through WriteBool/ReadBool, so on disk it is 1 byte + 3 NUL joint padding. Note the save reader types `odet` as int; for a 4-character tag a bool item and an int item are the same 12 bytes and the value is identical, so that is benign, not a parse error. [verified] +constexpr uint32_t ObservedTech_off_Detected = 0x00000008; +// offset int owith -- last member of the element; serialised through stream vft+0x24 with tag 0x00a2b364. +0x28..+0x2b is the tail of the 0x2c-byte element, which is why sizeof is 0x2c with no padding slack. [verified] +constexpr uint32_t ObservedTech_off_With = 0x00000028; +// thiscall void Game::ObservedWeapon::Write(Mars::Stream* s) RET 4. Byte-for-byte the same shape as ObservedTech_Write with tag `owep` (0x00a2b3f0) in place of `otch`: otnF u16 @+0x04, otnL u16 @+0x06, odet bool @+0x08, owep std::string (0x1c) @+0x0c, owith int @+0x28. Reader is 0x00817b10. Independent second instance of the same 0x2c element shape. [verified] +constexpr uint32_t ObservedWeapon_Write = 0x00417bc0; +// constant sizeof(std::basic_string) = 0x1c (28) binary-wide, ONE layout only: _Bx (16-byte SSO union, char[16] or char*) @+0x00, _Mysize @+0x10, _Myres @+0x14, _Alval (empty allocator) @+0x18. The allocator word is trailing, never read and never written -- the same allocator-last shape as this build's std::vector {_Myfirst,_Mylast,_Myend,_Alval} = 0x10. Evidence: (a) Stream::WriteString 0x008b9d76 `cmp [str+0x14],0x10` / `mov eax,[str]` and basic_string::assign 0x00425550 (_Mysize +0x10, _Myres +0x14) fix the field offsets; (b) the vector scan loop FUN_00699bd0 advances `add esi,0x1c` (0x00699c29) while doing the SSO test at [esi+0x14]/[esi] -- element base == string base, so the stride IS the sizeof; (c) PostEvent 0x008862b0 takes two by-value strings at [ebp+0x08] and [ebp+0x24] and RET 0x4c = 2*0x1c + 5*4; (d) a whole-binary sweep of the Stream string helpers recovered 65 std::string members off a non-stack base across all serializers -- ZERO have any sibling member inside (N, N+0x1c), and 51 of the 52 that have a next member have it at exactly N+0x1c (the one exception, StrategyServer KeyPath @+0x134, is +0x20 on the Write side and +0x1c on the Read side, i.e. the writer skips a member). There is no 0x18 instantiation, no empty-base variant and no game-local string class. [verified] +constexpr uint32_t std_string_sizeof = 0x0000001c; // thiscall vector::_Reserve/grow for a 44-byte element type. Shared with FUN_0086dec0, so it may be a COMDAT-folded body -- do NOT assume it is ObservedTech-specific. [mapped] constexpr uint32_t vector_44B_grow = 0x003b5820; diff --git a/guides/re-windows-2000s-howto.md b/guides/re-windows-2000s-howto.md index f83fbb2..afc1fa1 100644 --- a/guides/re-windows-2000s-howto.md +++ b/guides/re-windows-2000s-howto.md @@ -117,7 +117,7 @@ call-outs cite `findings/` and `~/sots-engine/docs/`. **1c. Library triage.** Apply Ghidra's VS2010 x86 FID databases (plus threatrack's) or FLIRT via ApplySig to fence off CRT/STL/D3DX code; anything left is game code. STL template instantiations compiled from headers won't match a library signature — recognise them by shape instead: MSVC-2010 `std::vector` = `{begin, end, cap}`, `std::list` = `{head*, size}`, red-black node = `{left, parent, right, key…, color/isnil tail}`, `std::string` = 0x1c bytes on x86 with the SSO union and `_Mysize`/`_Myres`. -> **Our experience:** our own notes disagree on whether `_Bx` starts at +0 or +4 (allocator stub) in this build (`struct-recovery.md` §0 vs `turn-spine.md` §1.1). The reliable lever is the `_Myres >= 16 ? _Ptr : _Buf` branch in any string consumer (`Stream::WriteString` @ 0x008b9d70), which pins both offsets. Verify per binary; do not trust a table. +> **Our experience — settled 2026-09-08.** `_Bx` starts at **+0** in this build: `_Bx@0, _Mysize@0x10, _Myres@0x14, _Alval@0x18`, sizeof `0x1c`. The lever that pins the field offsets is the `_Myres >= 16 ? _Ptr : _Buf` branch in any string consumer (`Stream::WriteString` @ 0x008b9d70). The lever that pins the **size** is different and easy to miss: `_Alval` is an empty allocator, so it is never loaded or stored and no touch-based analysis can see it. Size the type from an *enumeration* instead — a container stride (`vector` walk `add esi,0x1c` @ 0x00699c29), a by-value argument's frame spacing (`PostEvent` strings at `[ebp+8]`/`[ebp+0x24]`, `RET 0x4c`), or a copy constructor / serializer that lists every member. Sizing from touched fields undercounts by exactly the trailing allocator word — that is how a lane got 0x18 for the same type. Note the allocator is **trailing** in this build's `std::vector` too (`{_Myfirst,_Mylast,_Myend,_Alval}` = 0x10), which is the opposite of the MSVC `_String_val`/`_Vector_val` allocator-first shape you will read about; verify per binary, do not trust a table. **1d. Strings are the map.** Config keys, file names, log format strings and — decisively — **serialization tag names**. If the save format is tagged, each `Read/Write` references dozens of a struct's tag strings in order; rank functions by distinct tag xrefs, decompile, and read `this+offset → tag → type` straight off. This is the single highest-yield lever for the object model, and it is under-documented in the literature (see §4). diff --git a/tools/strfootprint.py b/tools/strfootprint.py new file mode 100644 index 0000000..b2e0ea5 --- /dev/null +++ b/tools/strfootprint.py @@ -0,0 +1,210 @@ +#!/usr/bin/env python3 +"""Whole-binary audit of the std::string member footprint in Sword of the Stars.exe. + +Why this exists +--------------- +`sizeof(std::string)` is load-bearing: it is embedded in ~65 recovered class +layouts, and a 4-byte error in it silently shifts every field that follows. +Lane X's ObservedTech work reported the embedded string as 0x18 bytes, against +the campaign-wide 0x1c. Arguing about it from one class is how you get a +plausible answer instead of a true one, so this settles it from the whole +binary at once. + +Method +------ +Every serialiser in this exe hands a member pointer to one of the Mars::Stream +primitive helpers with the same idiom: + + push 0xff ; default arg + lea ,[+] ; &this->member + push + push ; -> .rdata "otch", "pswd", ... + push + call WriteString / ReadString / WriteBool / ... + +So for each helper call we recover (base register, displacement, tag). Then, +per (function, base register), we take every OTHER displacement the same +function touches off that register. If sizeof(std::string) were 0x18 for some +instantiation, that class would necessarily have a real member somewhere in +(N+4 .. N+0x1b). The audit is: does one exist, anywhere? + +Two things must be excluded or the answer is noise: + * ebp/esp bases -- those are stack temporaries, not class members. (A local + string at [ebp-0x2c] shows _Mysize at -0x1c and _Myres at -0x18, which + looks exactly like two "members" inside the span.) + * N+0x10 and N+0x14 -- the string's OWN _Mysize/_Myres, read inline whenever + the compiler inlines the SSO `_Myres >= 16 ? _Ptr : _Buf` test. + +Result (2026-09-08, lane S): 65 string members off a non-stack base, ZERO with +a sibling inside the 0x1c span, and 51 of the 52 that have a next member have +it at exactly +0x1c. One layout, 0x1c, binary-wide. + +Usage: uv run python3 tools/strfootprint.py [--all] + --all also lists every recovered string member and its gap. +""" +import json +import os +import struct +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +REPO = os.path.dirname(HERE) +sys.path.insert(0, HERE) +from x86disp import decode, Desync, load_pe, EXE, FUNCS # noqa: E402 + +# Mars::Stream primitive wrappers (struct-recovery.md S0). All take +# (stream, tagname, &member, default) and are called from the class Read/Write. +HELPERS = { + 0x008b9d70: "WriteString", 0x008b9d90: "ReadString", + 0x008b9c20: "WriteBool", 0x008b9c00: "ReadBool", + 0x008b9be0: "WriteFloat", 0x008b9bc0: "ReadFloat", + 0x008b9d50: "WriteInt", 0x008b9d20: "ReadInt", + 0x008b9d00: "WriteInt16", 0x008b9cd0: "ReadInt16", + 0x008b9c60: "WriteInt64", 0x008b9c40: "ReadInt64", +} +STRING_HELPERS = ("WriteString", "ReadString") +STR_SIZE = 0x1c +SSO_OWN = (0x10, 0x14) # the string's own _Mysize / _Myres +STACK_BASES = ("ebp", "esp") + + +# ---------------------------------------------------------------- .rdata reader +def _load_rdata(): + data = open(EXE, "rb").read() + pe = struct.unpack_from("= rsz: + return None + o = ro + d + e = _DATA.find(b"\0", o, o + maxn) + if e < 0: + return None + try: + return _DATA[o:e].decode("ascii") + except UnicodeDecodeError: + return None + return None + + +def main(): + show_all = "--all" in sys.argv + _, secs = load_pe(EXE) + sva, eva, buf, _ = secs[0] + funcs = json.load(open(FUNCS)) + starts = sorted((int(k, 16), v[0]) for k, v in funcs.items()) + svas = [s[0] for s in starts] + rd_lo, rd_hi = 0x9DD000, 0xAD9000 + + records, touch = [], {} + for idx, (fva, nm) in enumerate(starts): + if not (sva <= fva < eva): + continue + fend = svas[idx + 1] if idx + 1 < len(starts) else eva + ins, i, end = [], fva - sva, fend - sva + while i < end: + try: + ln, info = decode(buf, i, len(buf)) + except Desync: + break + ins.append((sva + i, ln, info, buf[i:i + ln])) + i += ln + if not any(r[0] == 0xE8 and l == 5 and + (v + 5 + struct.unpack_from(" disp and (d - disp) not in SSO_OWN) + inside = [d for d in others if d < disp + STR_SIZE] + if inside: + violations.append((fva, nm, base, disp, tag, helper, inside)) + if others: + g = others[0] - disp + gaps[g] = gaps.get(g, 0) + 1 + if show_all and (fva, base, disp, helper) not in seen: + seen.add((fva, base, disp, helper)) + nxt = f"+0x{others[0]:x}" if others else "-" + print(f"0x{fva:08x} {nm:26s} {base} +0x{disp:<6x} {tag!r:12s} " + f"{helper:12s} next={nxt}") + + print(f"\nstring helper call sites : {len(strs)}") + print(f" resolved to a class member offset : {len(members)}") + print(f" stack temporaries (ebp/esp base) : {stack}") + print(f" offset not reached by a plain lea : {unresolved}") + print(f"\nmembers with a sibling inside (N, N+0x{STR_SIZE:x}) " + f": {len(violations)} <-- must be 0 for sizeof(std::string)==0x1c") + for v in violations: + print(f" 0x{v[0]:08x} {v[1]} {v[2]}+0x{v[3]:x} {v[4]!r} " + f"inside={[hex(x) for x in v[6]]}") + print("\ngap from a string member to the next member on the same base:") + for g in sorted(gaps): + print(f" +0x{g:<4x} : {gaps[g]}") + return 1 if violations else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/verify/save-reader/SAVE_FORMAT.md b/verify/save-reader/SAVE_FORMAT.md index d2833b1..2941b0a 100644 --- a/verify/save-reader/SAVE_FORMAT.md +++ b/verify/save-reader/SAVE_FORMAT.md @@ -165,6 +165,20 @@ satl satk`). Objective records (Player `odes`/`owep`/`otch`, each a framed VectorHelper of `"."` elements): `odes` = `otnF otnL odid opid`; `owep` = `otnF otnL odet owep`(string) `owith`; `otch` = `otnF otnL odet otch`(string) `owith`. + +> **Confirmed against the binary (2026-09-08, lane S).** `otch` elements are +> `Game::ObservedTech` (`Write` `0x00817cf0`, `Read` `0x00817c40`, `sizeof` `0x2c`) and `owep` +> elements are `Game::ObservedWeapon` (`Write` `0x00817bc0`, `Read` `0x00817b10`) — the same +> element shape with a different string tag. The serializers emit exactly the order this file +> already had. In-memory types, which the on-disk framing does not reveal: `otnF` and `otnL` are +> **`uint16`** widened to int by the stream (`movzx` then stream `vft+0x24`); `odet` is a +> **`bool`** written through `WriteBool`, i.e. 1 byte plus 3 NUL joint-padding bytes, not an +> int32. `save_reader.py` types `odet` as `int`, which is **benign and deliberate**: `odet` is a +> 4-character tag, so a bool item and an int item are both `pad4(4+4+1) == pad4(4+4+4) == 12` +> bytes and the little-endian value is identical (§2's stated "bool and int have identical item +> sizes for a name whose length is a multiple of 4"). Re-typing it would change nothing on +> disk; it is recorded here so the *reimplementation* uses a `bool`, and so nobody later +> "discovers" the same 3 padding bytes as a missing field. `NdGr2` (node grid): `paths{"." n, n×"."{npt npid npfr npto npctm npcby npdtn npdtf npenp npuse nptf}}` then `nextid`(int). `BQ` (system build queue): `ords{"." n, n×"."{desID con conleft sav ordID}}` — count 0 in turn1..3.