sots-re/findings/subsystems/observedtech-append.md

vector<ObservedTech> otch — pinned, and the displacement scanner that pinned it

Lane X. Static/host only; VM140 was held by lane F and the game was never run.

Two things here: a general-purpose tool (tools/x86disp.py) for a class of cross-reference Ghidra cannot index, and the ServerPlayer+0x274 answer it was built to get.

---

1. The gap the tool fills

Ghidra indexes immediate operands. It does not index ModRM displacements. On a 41,411-function MSVC C++ binary lea reg,[reg+disp] is the idiom for taking the address of a member — it is how every std::vector, every std::string, every embedded sub-object gets passed to a method or a constructor. So find-constant-uses 0x274 returned 13 unrelated hits and none of them was the vector, and lane P correctly concluded the search was a limitation rather than evidence of absence.

Lane E only found EventStorage at +0x29c because a whole accessor function happened to exist (lea eax,[ecx+0x29c]; ret). We were blind to this whole reference class all campaign, with ~1,600 classes still to go.

2. The tool — tools/x86disp.py

A real x86-32 length decoder (legacy prefixes, 1-/2-/3-byte opcodes, ModRM, SIB, disp8 sign-extended, disp32, every immediate form), swept from Ghidra's function starts so every decode begins on a genuine instruction boundary. It indexes every memory operand that carries a displacement, then answers "what code touches offset N off some object?" with the containing function, instruction address, base register and decoded instruction.

Correctness points that would otherwise produce confident garbage, all handled:

• mod=0,rm=5 and mod=0,rm=4,sib.base=5 are disp32 with no base — an absolute global address, not a member offset. Excluded from member queries. (Without this, every mov eax,[0x00a2bd88] in the binary shows up as a "displacement".)
• mod=1 disp8 is sign-extended: -0x08 must not be reported as +0xf8.
• mod=3 is a register operand with no displacement at all.
• 0x67 address-size prefix means 16-bit ModRM, a different layout entirely — refused rather than mis-decoded.

Commands: build, query <disp>, cohort, func <va>, dis <va>, stats, brute.

Decode quality:

functions swept : 41089
instructions    : 2174504
disp sites      : 612166
desyncs         : 70 (0.17% of functions)
code coverage   : 6142049/6142358 (100.0%)

Zero unknown-opcode desyncs — the opcode tables cover everything this binary contains. All 70 desyncs are truncations at a section/function boundary, not lost sync inside real code.

Gotcha worth carrying: the first build clipped each sweep at fva + Ghidra's sizeInBytes and lost 11% of functions to mid-instruction truncation — Ghidra's sizeInBytes understates real bodies often enough to matter (it was cutting valid mov esp,ebp; pop ebp; ret epilogues in half). Sweeping to the next function start instead took coverage 89% → 100%.

The tool works off a local cache (dumps/, gitignored: the exe, the function list, the index) so it never hammers the shared Ghidra box. tools/cache_functions.py pulls the function list once.

3. Validation — it rediscovers what we already knew

Run before any new claim was made.

GT1 — ServerPlayer::GetEventStorage, the one accessor we already had:

$ uv run python3 tools/x86disp.py func 0x0080db00
ServerPlayer_GetEventStorage @ 0x0080db00 size 7
  0x0080db00  lea eax,[ecx+0x29c]   ServerPlayer_GetEventStorage+0x0  (8d819c020000)

GT2 — lane E's two OnTechResearched sites, plus one it did not have:

$ uv run python3 tools/x86disp.py query 0x29c --lea
 ProcessTurn  @0x00891340
  0x00891713  lea ecx,[esi+0x29c]   ProcessTurn+0x3d3          <-- NEW
 OnTechResearched  @0x00891790
  0x008919ab  lea ecx,[esi+0x29c]   OnTechResearched+0x21b     <-- known
  0x0089241d  lea ecx,[esi+0x29c]   OnTechResearched+0xc8d     <-- known

GT3 — EvNxID at ServerPlayer+0x2b0: 90 sites found; the ServerPlayer ones are present. This one is also the honest illustration of the precision problem — see §5.

GT4 — positive control for the cohort ranker. Given the 50 ServerPlayer offsets already in struct-recovery.md, FUN_0087fac0 scores 50 co-hits out of 50 — it is the ServerPlayer serializer, and nothing else in the binary comes close. A scanner that could not surface that function from a bare offset query would not be worth using.

4. The answer: sizeof(ObservedTech) and the append site

sizeof(Game::ObservedTech) = 0x2c (44 bytes) — verified, three independent ways

(a) vector<ObservedTech>::operator= at 0x00872380 divides the vector's byte span by a constant using MSVC's magic-number sequence:

0x00872398  mov ecx,[edi+0x4]      ; src._Mylast
0x0087239b  mov edi,[edi]          ; src._Myfirst
0x0087239d  sub ecx,edi            ; byte span
0x0087239f  mov eax,0x2e8ba2e9
0x008723a4  imul ecx
0x008723a6  sar edx,3

ceil(2^35 / 44) == 0x2e8ba2e9 exactly, and emulating the full sequence reproduces n = 0,1,2,3,10,71,1000 from spans of 0,44,88,132,440,3124,44000. Divisor = 44, unambiguously.

(b) The same function then multiplies back by the literal stride:

0x0087243a  imul ecx,ecx,0x2c
0x0087243d  add ecx,[esi]          ; this->_Myfirst + n*44
0x00872441  mov [esi+0x4],ecx      ; this->_Mylast = ...

Same literal at 0x00872468 and at 0x007b735b inside push_back.

(c) The linear search in the append function advances its iterator by add edi,0x2c (0x007ba257).

This matches the on-disk lower bound exactly — 4 × int + one 0x1c std::string = 44 — so there is no padding slack anywhere in the element.

The append site

FUN_007ba1a0 = RecordObservedTech   (direct callee of ServerPlayer::OnTechResearched 0x00891790)

0x007ba221  mov edi,[esi+0x274]     ; it = observer->otch._Myfirst
0x007ba227  cmp edi,[esi+0x278]     ; ... != _Mylast ?
0x007ba22d  je  0x007ba274          ; empty -> append
      loop: compare each element's name string (this = elem+0x0c, length = [elem+0x1c])
0x007ba257  add edi,0x2c            ; ++it   *** stride 44 ***
0x007ba25a  cmp edi,[esi+0x278]
0x007ba260  jne loop
0x007ba274  lea ecx,[ebp-0x3c]
0x007ba277  call 0x008562a0         ; ObservedTech::ObservedTech()  on the stack
0x007ba27e  push eax
0x007ba27f  lea ecx,[esi+0x274]     ; this = &player->otch      <<< THE APPEND
0x007ba288  call 0x007b7320         ; vector<ObservedTech>::push_back

push_back (0x007b7320) grows via 0x007b5820 when _Mylast == _Myend — that realloc is exactly why lane R's guard saw all three of player+0x274/0x278/0x27c move on a completion, rather than only _Mylast.

RecordObservedTech is called from OnTechResearched (0x00891790) and from 0x007be228, 0x007be4e1, 0x007be535. It is a de-duplicating append: it appends only if no existing element already carries that tech name — worth knowing for the reimplementation, since a naive push_back would diverge on a re-observation.

Neither push_back nor operator= is COMDAT-ambiguous: 0x007b7320 has exactly one caller (RecordObservedTech, ecx = player+0x274) and 0x00872380 has two, both passing ServerPlayer+0x274. 0x007b5820 is shared with FUN_0086dec0 and may be a folded body, so it is labelled vector_44B_grow, not as ObservedTech-specific.

Element layout — as far as the evidence actually goes

The default ctor at 0x008562a0 writes vtable 0x00a2439c at +0x00. RTTI: COL 0x00a81c78 → type descriptor 0x00aeede4 → .?AVObservedTech@Game@@. So ObservedTech is polymorphic and its first word is a vptr, not a data field — which the on-disk shape does not tell you.

offset	size	evidence
+0x00	4	vptr 0x00a2439c; ctor writes it, RTTI-confirmed
+0x04	2	mov word [eax-0x28],cx at 0x007ba2b0 (eax = _Mylast, element = _Mylast-0x2c), source [ebx+0xc]
+0x06	2	mov word [eax-0x26],dx at 0x007ba2c6, same source word [ebx+0xc]
+0x08	4	unaccounted
+0x0c..+0x23	0x18	std::string (the otch tech name). Object base is +0x0c, MSVC layout {_Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14}. Ctor writes [+0x0c]=0, [+0x1c]=0, [+0x20]=0xf; the search loop reads [+0x1c] as the length and calls the compare with this = +0x0c; the post-append assign uses lea ecx,[_Mylast-0x20] = +0x0c
+0x24	4	unaccounted
+0x28	4	unaccounted

Note the string here is 0x18 bytes, not the 0x1c the ServerPlayer::pswd row in struct-recovery.md implies — three separate reads inside this element agree on {buf16, _Mysize@+0x10, _Myres@+0x14}. Worth re-checking pswd against that.

The four on-disk ints (otnF, otnL, odet, owith) have to map onto +0x04/+0x06 (two 16-bit fields) and the three 4-byte slots +0x08, +0x24, +0x28. That mapping is not determined here and is deliberately not guessed. The one suggestive observation, flagged as a hypothesis only: +0x04 and +0x06 are two adjacent 16-bit fields written from the same source word on first observation, which is the shape you would expect of a first-seen / last-seen turn pair — but nothing here proves it.

What would settle the mapping: read ObservedTech's Read/Write serializer, where the member order is explicit. Lane P's observed_techs region byte delta remains a valid live cross-check and should now come back as exactly 44 per completion.

5. False-positive rate, honestly

Two different error rates, and the interesting one is not the one you'd expect.

Decode-level error of the naive method is low. A raw byte scan for 8D + ModRM + the displacement — the thing you'd write without a decoder — is mostly right:

query	naive candidates	not actually an instruction	real sites the naive scan missed
lea, disp32 0x274	19	0 (0.0%)	80 of 99
11 common opcodes, disp32 0x274	89	1 (1.1%)	11 of 99
lea, disp8 0x14	828	1 (0.1%)	13,784 of 14,611
11 common opcodes, disp8 0x14	10,326	59 (0.6%)	4,344 of 14,611

So the decoder's win is recall, not decode precision. A hand-written opcode set misses 80–94% of the real accesses, because a member is read, written, compared, and float-loaded far more often than its address is taken, and you cannot enumerate those opcodes by hand.

Class-level precision is the real problem, and it is poor. The scanner knows the displacement; it cannot know what class the base register holds. query 0x274 returns 99 sites in 45 functions and only about 6 of those functions are actually touching ServerPlayer::otch — roughly 13% precision by function. The honest way to state the value is as search-space reduction: 41,411 functions → 45, about 900×, down to a list a human reads in two minutes.

The cohort ranker helps, and has a trap. Scoring functions by how many already-known offsets of the same class they touch pulls real class methods to the top (the serializer hits 50/50). But it would have thrown away the correct answer: RecordObservedTech touches only 0x274 and 0x278 and nothing else on ServerPlayer, so any --min>=1 filter drops it. Use cohort as a ranker, never as a filter. This is now written into the tool's own docstring.

What actually closed the case was the plain query 0x274 list intersected with one call-graph lookup (OnTechResearched's direct callees). Displacement scan for recall, call graph for disambiguation — neither alone was enough.

6. Previously anonymous offsets — attribution results

The payoff was smaller than hoped, because most of the audit's anonymous offsets had already been named by other lanes since the audit was written.

offset	status before	after
player+0x274/0x278/0x27c	"vector grew, append site unknown"	RecordObservedTech+0xdf (0x007ba27f), sizeof = 0x2c
player+0x2b0	already named by lane E	unchanged (EvNxID)
TechTree+0x20	already TechTree_off_OrderCounter	unchanged
ServerPlayer+0x308	already ServerPlayer_off_NodeBore	unchanged
Budget+0x64	harness-audit row 11: "the original writes it"	not supported. See below

Budget+0x64 (harness-audit row 11) — a correction. ServerPlayer::ComputeBudget (0x00863030) writes its Budget* out-param through esi, and every such store lands in +0x00..+0x54 (the 22-int block). The only two +0x64 accesses in the whole function are loads off a different base register (mov ecx,[eax+0x64] at 0x0086328c, mov edx,[eax+0x64] at 0x0086335d). There is no store to Budget+0x64 in ComputeBudget.

Separately, TechTree::ProcessResearch's int* overbudget fourth argument is not Budget+0x64: its only caller is ProcessTurn (0x00891340) and the call at 0x008914a5 passes lea edx,[ebp-0x14] — a stack local, consumed immediately after the call (mov eax,[ebp-0x14]; cmp eax,0; jle).

This agrees with lane R, whose budget_object guard saw Budget+0x64 change in 0 of 4284 calls. Row 11 should be reclassified from "the original writes it and B1 never checked" to "nothing has been shown to write it"; the remaining way to settle it is a write watchpoint on that word, not another static search.

7. Verdict on the technique

Worth keeping, with its limits stated. It answered a question that had been parked, and the 0x29c validation turned up a ProcessTurn EventStorage site nobody had. But it is a recall tool that produces a 20–100 line candidate list per offset, not an oracle: every result still needs a call-graph or decompiler check before it is a fact. For the ~1,600 remaining classes the realistic workflow is query <offset> → read the list → confirm with one cross-reference call.

8. Ghidra writeback

Labels: RecordObservedTech 0x007ba1a0, vector_ObservedTech_push_back 0x007b7320, ObservedTech_ctor 0x008562a0, vector_ObservedTech_assign 0x00872380, vector_44B_grow 0x007b5820, vftable_ObservedTech 0x00a2439c. Plate comments carrying the stride evidence on the first four.

addresses.json: ObservedTech_sizeof, ObservedTech_vftable, ObservedTech_off_Name, RecordObservedTech, vector_ObservedTech_push_back, vector_ObservedTech_assign, ObservedTech_ctor, vector_44B_grow; ServerPlayer_off_ObservedTechs updated from "NOT PINNED" to verified.