460cb7ca2b
Ghidra does not index ModRM displacements, so `lea reg,[reg+disp]` -- the MSVC idiom for taking a member's address -- is invisible to find-constant-uses. That blind spot parked ServerPlayer+0x274 and covers every non-trivial member of the ~1,600 classes still to map. tools/x86disp.py: full x86-32 length decoder (prefixes, 1/2/3-byte opcodes, ModRM, SIB, sign-extended disp8, disp32, every immediate form) swept from Ghidra's 41,089 function starts so decodes begin on real instruction boundaries. 2,174,504 instructions, 612,166 displacement sites, 100.0% code coverage, 70 desyncs (0.17%), zero unknown opcodes. Excludes no-base disp32 forms (mod=0/rm=5, sib.base=5) which are absolute globals, not member offsets. Commands: build/query/cohort/func/dis/stats/brute. Works off a gitignored local cache in dumps/ rather than hammering CT111. Validated before use: re-finds lea eax,[ecx+0x29c] in ServerPlayer::GetEventStorage (0x0080db00) and both known OnTechResearched +0x29c sites, plus a new one in ProcessTurn. Positive control: the ServerPlayer serializer scores 50/50 known offsets. sizeof(Game::ObservedTech) = 0x2c (44), proven three ways: the exact magic divide 0x2e8ba2e9 sar 3 at 0x0087239f, imul reg,reg,0x2c at 0x0087243a and 0x007b735b, and the search stride add edi,0x2c at 0x007ba257. Append site: RecordObservedTech+0xdf (0x007ba27f) -- lea ecx,[player+0x274]; call vector_ObservedTech_push_back 0x007b7320 RecordObservedTech (0x007ba1a0) is a direct callee of OnTechResearched and de-duplicates by tech name before appending. The realloc through 0x007b5820 is why lane R's guard saw all three vector words move. Element carries a vptr (RTTI .?AVObservedTech@Game@@) at +0 and a 0x18-byte std::string at +0x0c; the four on-disk ints map onto +0x04/+0x06/+0x08/+0x24/+0x28 in an order this read does NOT determine, and is not guessed. Also corrects harness-audit row 11: ComputeBudget has no store to Budget+0x64 (its only +0x64 accesses are loads off a different base), and ProcessResearch's int* overbudget arg is a ProcessTurn stack local, not Budget+0x64. Agrees with lane R's guard seeing 0 changes in 4284 calls. Honest limits are recorded in the note and the board: this is a recall tool, not an oracle. Class-level precision at 0x274 is ~13% by function, i.e. a ~900x search-space cut that still needs one call-graph check. Cohort ranking must not be used as a hard filter -- it would have discarded the correct answer here. Ghidra writeback: labels + plate comments on RecordObservedTech, vector_ObservedTech_push_back, ObservedTech_ctor, vector_ObservedTech_assign, vector_44B_grow, vftable_ObservedTech.
278 lines
14 KiB
Markdown
278 lines
14 KiB
Markdown
# `vector<ObservedTech> otch` — pinned, and the displacement scanner that pinned it
|
||
|
||
Lane X. Static/host only; VM140 was held by lane F and the game was never run.
|
||
|
||
Two things here: a general-purpose tool (`tools/x86disp.py`) for a class of cross-reference
|
||
Ghidra cannot index, and the `ServerPlayer+0x274` answer it was built to get.
|
||
|
||
---
|
||
|
||
## 1. The gap the tool fills
|
||
|
||
Ghidra indexes **immediate** operands. It does not index **ModRM displacements**. On a
|
||
41,411-function MSVC C++ binary `lea reg,[reg+disp]` is *the* idiom for taking the address of
|
||
a member — it is how every `std::vector`, every `std::string`, every embedded sub-object gets
|
||
passed to a method or a constructor. So `find-constant-uses 0x274` returned 13 unrelated hits
|
||
and none of them was the vector, and lane P correctly concluded the search was a limitation
|
||
rather than evidence of absence.
|
||
|
||
Lane E only found `EventStorage` at `+0x29c` because a whole accessor function happened to
|
||
exist (`lea eax,[ecx+0x29c]; ret`). We were blind to this whole reference class all campaign,
|
||
with ~1,600 classes still to go.
|
||
|
||
## 2. The tool — `tools/x86disp.py`
|
||
|
||
A real x86-32 **length decoder** (legacy prefixes, 1-/2-/3-byte opcodes, ModRM, SIB, disp8
|
||
sign-extended, disp32, every immediate form), swept from Ghidra's function starts so every
|
||
decode begins on a genuine instruction boundary. It indexes every memory operand that carries
|
||
a displacement, then answers "what code touches offset N off some object?" with the containing
|
||
function, instruction address, base register and decoded instruction.
|
||
|
||
Correctness points that would otherwise produce confident garbage, all handled:
|
||
|
||
* `mod=0,rm=5` and `mod=0,rm=4,sib.base=5` are **disp32 with no base** — an absolute global
|
||
address, not a member offset. Excluded from member queries. (Without this, every
|
||
`mov eax,[0x00a2bd88]` in the binary shows up as a "displacement".)
|
||
* `mod=1` disp8 is **sign-extended**: `-0x08` must not be reported as `+0xf8`.
|
||
* `mod=3` is a register operand with no displacement at all.
|
||
* `0x67` address-size prefix means 16-bit ModRM, a different layout entirely — refused rather
|
||
than mis-decoded.
|
||
|
||
Commands: `build`, `query <disp>`, `cohort`, `func <va>`, `dis <va>`, `stats`, `brute`.
|
||
|
||
**Decode quality:**
|
||
|
||
```
|
||
functions swept : 41089
|
||
instructions : 2174504
|
||
disp sites : 612166
|
||
desyncs : 70 (0.17% of functions)
|
||
code coverage : 6142049/6142358 (100.0%)
|
||
```
|
||
|
||
Zero unknown-opcode desyncs — the opcode tables cover everything this binary contains. All 70
|
||
desyncs are truncations at a section/function boundary, not lost sync inside real code.
|
||
|
||
> Gotcha worth carrying: the first build clipped each sweep at `fva + Ghidra's sizeInBytes`
|
||
> and lost 11% of functions to mid-instruction truncation — Ghidra's `sizeInBytes` understates
|
||
> real bodies often enough to matter (it was cutting valid `mov esp,ebp; pop ebp; ret`
|
||
> epilogues in half). Sweeping to the **next function start** instead took coverage 89% → 100%.
|
||
|
||
The tool works off a local cache (`dumps/`, gitignored: the exe, the function list, the index)
|
||
so it never hammers the shared Ghidra box. `tools/cache_functions.py` pulls the function list
|
||
once.
|
||
|
||
## 3. Validation — it rediscovers what we already knew
|
||
|
||
Run before any new claim was made.
|
||
|
||
**GT1 — `ServerPlayer::GetEventStorage`, the one accessor we already had:**
|
||
|
||
```
|
||
$ uv run python3 tools/x86disp.py func 0x0080db00
|
||
ServerPlayer_GetEventStorage @ 0x0080db00 size 7
|
||
0x0080db00 lea eax,[ecx+0x29c] ServerPlayer_GetEventStorage+0x0 (8d819c020000)
|
||
```
|
||
|
||
**GT2 — lane E's two `OnTechResearched` sites, plus one it did not have:**
|
||
|
||
```
|
||
$ uv run python3 tools/x86disp.py query 0x29c --lea
|
||
ProcessTurn @0x00891340
|
||
0x00891713 lea ecx,[esi+0x29c] ProcessTurn+0x3d3 <-- NEW
|
||
OnTechResearched @0x00891790
|
||
0x008919ab lea ecx,[esi+0x29c] OnTechResearched+0x21b <-- known
|
||
0x0089241d lea ecx,[esi+0x29c] OnTechResearched+0xc8d <-- known
|
||
```
|
||
|
||
**GT3 — `EvNxID` at `ServerPlayer+0x2b0`:** 90 sites found; the `ServerPlayer` ones are present.
|
||
This one is also the honest illustration of the precision problem — see §5.
|
||
|
||
**GT4 — positive control for the cohort ranker.** Given the 50 `ServerPlayer` offsets already
|
||
in `struct-recovery.md`, `FUN_0087fac0` scores **50 co-hits out of 50** — it is the
|
||
`ServerPlayer` serializer, and nothing else in the binary comes close. A scanner that could
|
||
not surface that function from a bare offset query would not be worth using.
|
||
|
||
## 4. The answer: `sizeof(ObservedTech)` and the append site
|
||
|
||
### `sizeof(Game::ObservedTech) = 0x2c` (44 bytes) — verified, three independent ways
|
||
|
||
**(a)** `vector<ObservedTech>::operator=` at `0x00872380` divides the vector's byte span by a
|
||
constant using MSVC's magic-number sequence:
|
||
|
||
```
|
||
0x00872398 mov ecx,[edi+0x4] ; src._Mylast
|
||
0x0087239b mov edi,[edi] ; src._Myfirst
|
||
0x0087239d sub ecx,edi ; byte span
|
||
0x0087239f mov eax,0x2e8ba2e9
|
||
0x008723a4 imul ecx
|
||
0x008723a6 sar edx,3
|
||
```
|
||
|
||
`ceil(2^35 / 44) == 0x2e8ba2e9` **exactly**, and emulating the full sequence reproduces
|
||
n = 0,1,2,3,10,71,1000 from spans of 0,44,88,132,440,3124,44000. Divisor = 44, unambiguously.
|
||
|
||
**(b)** The same function then multiplies back by the literal stride:
|
||
|
||
```
|
||
0x0087243a imul ecx,ecx,0x2c
|
||
0x0087243d add ecx,[esi] ; this->_Myfirst + n*44
|
||
0x00872441 mov [esi+0x4],ecx ; this->_Mylast = ...
|
||
```
|
||
|
||
Same literal at `0x00872468` and at `0x007b735b` inside `push_back`.
|
||
|
||
**(c)** The linear search in the append function advances its iterator by `add edi,0x2c`
|
||
(`0x007ba257`).
|
||
|
||
This matches the on-disk lower bound *exactly* — 4 × `int` + one `0x1c` `std::string` = 44 —
|
||
so there is no padding slack anywhere in the element.
|
||
|
||
### The append site
|
||
|
||
```
|
||
FUN_007ba1a0 = RecordObservedTech (direct callee of ServerPlayer::OnTechResearched 0x00891790)
|
||
|
||
0x007ba221 mov edi,[esi+0x274] ; it = observer->otch._Myfirst
|
||
0x007ba227 cmp edi,[esi+0x278] ; ... != _Mylast ?
|
||
0x007ba22d je 0x007ba274 ; empty -> append
|
||
loop: compare each element's name string (this = elem+0x0c, length = [elem+0x1c])
|
||
0x007ba257 add edi,0x2c ; ++it *** stride 44 ***
|
||
0x007ba25a cmp edi,[esi+0x278]
|
||
0x007ba260 jne loop
|
||
0x007ba274 lea ecx,[ebp-0x3c]
|
||
0x007ba277 call 0x008562a0 ; ObservedTech::ObservedTech() on the stack
|
||
0x007ba27e push eax
|
||
0x007ba27f lea ecx,[esi+0x274] ; this = &player->otch <<< THE APPEND
|
||
0x007ba288 call 0x007b7320 ; vector<ObservedTech>::push_back
|
||
```
|
||
|
||
`push_back` (`0x007b7320`) grows via `0x007b5820` when `_Mylast == _Myend` — **that realloc is
|
||
exactly why lane R's guard saw all three of `player+0x274/0x278/0x27c` move on a completion**,
|
||
rather than only `_Mylast`.
|
||
|
||
`RecordObservedTech` is called from `OnTechResearched` (`0x00891790`) and from `0x007be228`,
|
||
`0x007be4e1`, `0x007be535`. It is a **de-duplicating** append: it appends only if no existing
|
||
element already carries that tech name — worth knowing for the reimplementation, since a naive
|
||
`push_back` would diverge on a re-observation.
|
||
|
||
Neither `push_back` nor `operator=` is COMDAT-ambiguous: `0x007b7320` has exactly one caller
|
||
(`RecordObservedTech`, `ecx = player+0x274`) and `0x00872380` has two, both passing
|
||
`ServerPlayer+0x274`. `0x007b5820` **is** shared with `FUN_0086dec0` and may be a folded body,
|
||
so it is labelled `vector_44B_grow`, not as ObservedTech-specific.
|
||
|
||
### Element layout — as far as the evidence actually goes
|
||
|
||
The default ctor at `0x008562a0` writes vtable `0x00a2439c` at `+0x00`. RTTI:
|
||
COL `0x00a81c78` → type descriptor `0x00aeede4` → **`.?AVObservedTech@Game@@`**. So
|
||
`ObservedTech` is polymorphic and its first word is a vptr, not a data field — which the
|
||
on-disk shape does not tell you.
|
||
|
||
| offset | size | evidence |
|
||
|---|---|---|
|
||
| `+0x00` | 4 | vptr `0x00a2439c`; ctor writes it, RTTI-confirmed |
|
||
| `+0x04` | 2 | `mov word [eax-0x28],cx` at `0x007ba2b0` (`eax` = `_Mylast`, element = `_Mylast-0x2c`), source `[ebx+0xc]` |
|
||
| `+0x06` | 2 | `mov word [eax-0x26],dx` at `0x007ba2c6`, **same source word** `[ebx+0xc]` |
|
||
| `+0x08` | 4 | **unaccounted** |
|
||
| `+0x0c..+0x23` | 0x18 | `std::string` (the `otch` tech name). Object base is `+0x0c`, MSVC layout `{_Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14}`. Ctor writes `[+0x0c]=0`, `[+0x1c]=0`, `[+0x20]=0xf`; the search loop reads `[+0x1c]` as the length and calls the compare with `this = +0x0c`; the post-append assign uses `lea ecx,[_Mylast-0x20]` = `+0x0c` |
|
||
| `+0x24` | 4 | **unaccounted** |
|
||
| `+0x28` | 4 | **unaccounted** |
|
||
|
||
Note the string here is **0x18 bytes**, not the 0x1c the `ServerPlayer::pswd` row in
|
||
`struct-recovery.md` implies — three separate reads inside this element agree on
|
||
`{buf16, _Mysize@+0x10, _Myres@+0x14}`. Worth re-checking `pswd` against that.
|
||
|
||
The four on-disk ints (`otnF`, `otnL`, `odet`, `owith`) have to map onto `+0x04`/`+0x06` (two
|
||
16-bit fields) and the three 4-byte slots `+0x08`, `+0x24`, `+0x28`. **That mapping is not
|
||
determined here and is deliberately not guessed.** The one suggestive observation, flagged as a
|
||
*hypothesis only*: `+0x04` and `+0x06` are two adjacent 16-bit fields written from the same
|
||
source word on first observation, which is the shape you would expect of a first-seen /
|
||
last-seen turn pair — but nothing here proves it.
|
||
|
||
What would settle the mapping: read `ObservedTech`'s `Read`/`Write` serializer, where the
|
||
member order is explicit. Lane P's `observed_techs` region byte delta remains a valid live
|
||
cross-check and should now come back as exactly 44 per completion.
|
||
|
||
## 5. False-positive rate, honestly
|
||
|
||
Two different error rates, and the interesting one is not the one you'd expect.
|
||
|
||
**Decode-level error of the naive method is low.** A raw byte scan for `8D` + ModRM + the
|
||
displacement — the thing you'd write without a decoder — is mostly *right*:
|
||
|
||
| query | naive candidates | not actually an instruction | real sites the naive scan **missed** |
|
||
|---|---|---|---|
|
||
| `lea`, disp32 `0x274` | 19 | 0 (0.0%) | **80 of 99** |
|
||
| 11 common opcodes, disp32 `0x274` | 89 | 1 (1.1%) | 11 of 99 |
|
||
| `lea`, disp8 `0x14` | 828 | 1 (0.1%) | **13,784 of 14,611** |
|
||
| 11 common opcodes, disp8 `0x14` | 10,326 | 59 (0.6%) | 4,344 of 14,611 |
|
||
|
||
So the decoder's win is **recall, not decode precision**. A hand-written opcode set misses
|
||
80–94% of the real accesses, because a member is read, written, compared, and float-loaded far
|
||
more often than its address is taken, and you cannot enumerate those opcodes by hand.
|
||
|
||
**Class-level precision is the real problem, and it is poor.** The scanner knows the
|
||
displacement; it cannot know what class the base register holds. `query 0x274` returns 99 sites
|
||
in 45 functions and only about 6 of those functions are actually touching `ServerPlayer::otch`
|
||
— roughly **13% precision by function**. The honest way to state the value is as search-space
|
||
reduction: 41,411 functions → 45, about **900×**, down to a list a human reads in two minutes.
|
||
|
||
**The cohort ranker helps, and has a trap.** Scoring functions by how many *already-known*
|
||
offsets of the same class they touch pulls real class methods to the top (the serializer hits
|
||
50/50). But it would have **thrown away the correct answer**: `RecordObservedTech` touches only
|
||
`0x274` and `0x278` and nothing else on `ServerPlayer`, so any `--min>=1` filter drops it. Use
|
||
cohort as a ranker, never as a filter. This is now written into the tool's own docstring.
|
||
|
||
What actually closed the case was the plain `query 0x274` list **intersected with one call-graph
|
||
lookup** (`OnTechResearched`'s direct callees). Displacement scan for recall, call graph for
|
||
disambiguation — neither alone was enough.
|
||
|
||
## 6. Previously anonymous offsets — attribution results
|
||
|
||
The payoff was smaller than hoped, because most of the audit's anonymous offsets had already
|
||
been named by other lanes since the audit was written.
|
||
|
||
| offset | status before | after |
|
||
|---|---|---|
|
||
| `player+0x274/0x278/0x27c` | "vector grew, append site unknown" | **`RecordObservedTech+0xdf` (0x007ba27f)**, `sizeof` = 0x2c |
|
||
| `player+0x2b0` | already named by lane E | unchanged (`EvNxID`) |
|
||
| `TechTree+0x20` | already `TechTree_off_OrderCounter` | unchanged |
|
||
| `ServerPlayer+0x308` | already `ServerPlayer_off_NodeBore` | unchanged |
|
||
| `Budget+0x64` | harness-audit row 11: "the original writes it" | **not supported.** See below |
|
||
|
||
**`Budget+0x64` (harness-audit row 11) — a correction.** `ServerPlayer::ComputeBudget`
|
||
(`0x00863030`) writes its `Budget*` out-param through `esi`, and every such store lands in
|
||
`+0x00..+0x54` (the 22-int block). The only two `+0x64` accesses in the whole function are
|
||
**loads off a different base register** (`mov ecx,[eax+0x64]` at `0x0086328c`,
|
||
`mov edx,[eax+0x64]` at `0x0086335d`). There is no store to `Budget+0x64` in `ComputeBudget`.
|
||
|
||
Separately, `TechTree::ProcessResearch`'s `int* overbudget` fourth argument is **not**
|
||
`Budget+0x64`: its only caller is `ProcessTurn` (`0x00891340`) and the call at `0x008914a5`
|
||
passes `lea edx,[ebp-0x14]` — a stack local, consumed immediately after the call
|
||
(`mov eax,[ebp-0x14]; cmp eax,0; jle`).
|
||
|
||
This agrees with lane R, whose `budget_object` guard saw `Budget+0x64` change in **0 of 4284
|
||
calls**. Row 11 should be reclassified from "the original writes it and B1 never checked" to
|
||
"nothing has been shown to write it"; the remaining way to settle it is a write watchpoint on
|
||
that word, not another static search.
|
||
|
||
## 7. Verdict on the technique
|
||
|
||
Worth keeping, with its limits stated. It answered a question that had been parked, and the
|
||
`0x29c` validation turned up a `ProcessTurn` `EventStorage` site nobody had. But it is a
|
||
**recall** tool that produces a 20–100 line candidate list per offset, not an oracle: every
|
||
result still needs a call-graph or decompiler check before it is a fact. For the ~1,600
|
||
remaining classes the realistic workflow is `query <offset>` → read the list → confirm with one
|
||
cross-reference call.
|
||
|
||
## 8. Ghidra writeback
|
||
|
||
Labels: `RecordObservedTech` `0x007ba1a0`, `vector_ObservedTech_push_back` `0x007b7320`,
|
||
`ObservedTech_ctor` `0x008562a0`, `vector_ObservedTech_assign` `0x00872380`,
|
||
`vector_44B_grow` `0x007b5820`, `vftable_ObservedTech` `0x00a2439c`. Plate comments carrying
|
||
the stride evidence on the first four.
|
||
|
||
`addresses.json`: `ObservedTech_sizeof`, `ObservedTech_vftable`, `ObservedTech_off_Name`,
|
||
`RecordObservedTech`, `vector_ObservedTech_push_back`, `vector_ObservedTech_assign`,
|
||
`ObservedTech_ctor`, `vector_44B_grow`; `ServerPlayer_off_ObservedTechs` updated from
|
||
"NOT PINNED" to `verified`.
|