sots-re/findings/subsystems/observedtech-append.md

vector<ObservedTech> otch — pinned, and the displacement scanner that pinned it

Lane X. Static/host only; VM140 was held by lane F and the game was never run.

Two things here: a general-purpose tool (tools/x86disp.py) for a class of cross-reference Ghidra cannot index, and the ServerPlayer+0x274 answer it was built to get.

---

1. The gap the tool fills

Ghidra indexes immediate operands. It does not index ModRM displacements. On a 41,411-function MSVC C++ binary lea reg,[reg+disp] is the idiom for taking the address of a member — it is how every std::vector, every std::string, every embedded sub-object gets passed to a method or a constructor. So find-constant-uses 0x274 returned 13 unrelated hits and none of them was the vector, and lane P correctly concluded the search was a limitation rather than evidence of absence.

Lane E only found EventStorage at +0x29c because a whole accessor function happened to exist (lea eax,[ecx+0x29c]; ret). We were blind to this whole reference class all campaign, with ~1,600 classes still to go.

2. The tool — tools/x86disp.py

A real x86-32 length decoder (legacy prefixes, 1-/2-/3-byte opcodes, ModRM, SIB, disp8 sign-extended, disp32, every immediate form), swept from Ghidra's function starts so every decode begins on a genuine instruction boundary. It indexes every memory operand that carries a displacement, then answers "what code touches offset N off some object?" with the containing function, instruction address, base register and decoded instruction.

Correctness points that would otherwise produce confident garbage, all handled:

• mod=0,rm=5 and mod=0,rm=4,sib.base=5 are disp32 with no base — an absolute global address, not a member offset. Excluded from member queries. (Without this, every mov eax,[0x00a2bd88] in the binary shows up as a "displacement".)
• mod=1 disp8 is sign-extended: -0x08 must not be reported as +0xf8.
• mod=3 is a register operand with no displacement at all.
• 0x67 address-size prefix means 16-bit ModRM, a different layout entirely — refused rather than mis-decoded.

Commands: build, query <disp>, cohort, func <va>, dis <va>, stats, brute.

Decode quality:

functions swept : 41089
instructions    : 2174504
disp sites      : 612166
desyncs         : 70 (0.17% of functions)
code coverage   : 6142049/6142358 (100.0%)

Zero unknown-opcode desyncs — the opcode tables cover everything this binary contains. All 70 desyncs are truncations at a section/function boundary, not lost sync inside real code.

Gotcha worth carrying: the first build clipped each sweep at fva + Ghidra's sizeInBytes and lost 11% of functions to mid-instruction truncation — Ghidra's sizeInBytes understates real bodies often enough to matter (it was cutting valid mov esp,ebp; pop ebp; ret epilogues in half). Sweeping to the next function start instead took coverage 89% → 100%.

The tool works off a local cache (dumps/, gitignored: the exe, the function list, the index) so it never hammers the shared Ghidra box. tools/cache_functions.py pulls the function list once.

3. Validation — it rediscovers what we already knew

Run before any new claim was made.

GT1 — ServerPlayer::GetEventStorage, the one accessor we already had:

$ uv run python3 tools/x86disp.py func 0x0080db00
ServerPlayer_GetEventStorage @ 0x0080db00 size 7
  0x0080db00  lea eax,[ecx+0x29c]   ServerPlayer_GetEventStorage+0x0  (8d819c020000)

GT2 — lane E's two OnTechResearched sites, plus one it did not have:

$ uv run python3 tools/x86disp.py query 0x29c --lea
 ProcessTurn  @0x00891340
  0x00891713  lea ecx,[esi+0x29c]   ProcessTurn+0x3d3          <-- NEW
 OnTechResearched  @0x00891790
  0x008919ab  lea ecx,[esi+0x29c]   OnTechResearched+0x21b     <-- known
  0x0089241d  lea ecx,[esi+0x29c]   OnTechResearched+0xc8d     <-- known

GT3 — EvNxID at ServerPlayer+0x2b0: 90 sites found; the ServerPlayer ones are present. This one is also the honest illustration of the precision problem — see §5.

GT4 — positive control for the cohort ranker. Given the 50 ServerPlayer offsets already in struct-recovery.md, FUN_0087fac0 scores 50 co-hits out of 50 — it is the ServerPlayer serializer, and nothing else in the binary comes close. A scanner that could not surface that function from a bare offset query would not be worth using.

4. The answer: sizeof(ObservedTech) and the append site

sizeof(Game::ObservedTech) = 0x2c (44 bytes) — verified, three independent ways

(a) vector<ObservedTech>::operator= at 0x00872380 divides the vector's byte span by a constant using MSVC's magic-number sequence:

0x00872398  mov ecx,[edi+0x4]      ; src._Mylast
0x0087239b  mov edi,[edi]          ; src._Myfirst
0x0087239d  sub ecx,edi            ; byte span
0x0087239f  mov eax,0x2e8ba2e9
0x008723a4  imul ecx
0x008723a6  sar edx,3

ceil(2^35 / 44) == 0x2e8ba2e9 exactly, and emulating the full sequence reproduces n = 0,1,2,3,10,71,1000 from spans of 0,44,88,132,440,3124,44000. Divisor = 44, unambiguously.

(b) The same function then multiplies back by the literal stride:

0x0087243a  imul ecx,ecx,0x2c
0x0087243d  add ecx,[esi]          ; this->_Myfirst + n*44
0x00872441  mov [esi+0x4],ecx      ; this->_Mylast = ...

Same literal at 0x00872468 and at 0x007b735b inside push_back.

(c) The linear search in the append function advances its iterator by add edi,0x2c (0x007ba257).

This matches the on-disk lower bound exactly — 4 × int + one 0x1c std::string = 44 — so there is no padding slack anywhere in the element.

The append site

FUN_007ba1a0 = RecordObservedTech   (direct callee of ServerPlayer::OnTechResearched 0x00891790)

0x007ba221  mov edi,[esi+0x274]     ; it = observer->otch._Myfirst
0x007ba227  cmp edi,[esi+0x278]     ; ... != _Mylast ?
0x007ba22d  je  0x007ba274          ; empty -> append
      loop: compare each element's name string (this = elem+0x0c, length = [elem+0x1c])
0x007ba257  add edi,0x2c            ; ++it   *** stride 44 ***
0x007ba25a  cmp edi,[esi+0x278]
0x007ba260  jne loop
0x007ba274  lea ecx,[ebp-0x3c]
0x007ba277  call 0x008562a0         ; ObservedTech::ObservedTech()  on the stack
0x007ba27e  push eax
0x007ba27f  lea ecx,[esi+0x274]     ; this = &player->otch      <<< THE APPEND
0x007ba288  call 0x007b7320         ; vector<ObservedTech>::push_back

push_back (0x007b7320) grows via 0x007b5820 when _Mylast == _Myend — that realloc is exactly why lane R's guard saw all three of player+0x274/0x278/0x27c move on a completion, rather than only _Mylast.

RecordObservedTech is called from OnTechResearched (0x00891790) and from 0x007be228, 0x007be4e1, 0x007be535. It is a de-duplicating append: it appends only if no existing element already carries that tech name — worth knowing for the reimplementation, since a naive push_back would diverge on a re-observation.

Neither push_back nor operator= is COMDAT-ambiguous: 0x007b7320 has exactly one caller (RecordObservedTech, ecx = player+0x274) and 0x00872380 has two, both passing ServerPlayer+0x274. 0x007b5820 is shared with FUN_0086dec0 and may be a folded body, so it is labelled vector_44B_grow, not as ObservedTech-specific.

Element layout — as far as the evidence actually goes

The default ctor at 0x008562a0 writes vtable 0x00a2439c at +0x00. RTTI: COL 0x00a81c78 → type descriptor 0x00aeede4 → .?AVObservedTech@Game@@. So ObservedTech is polymorphic and its first word is a vptr, not a data field — which the on-disk shape does not tell you.

Superseded 2026-09-08 by lane S — see §9. The table as first written called the embedded string 0x18 bytes and left +0x08, +0x24, +0x28 unaccounted. +0x24 is not a field: it is the string's own trailing allocator word. The corrected map is below; the original reasoning is kept in §9 because the way it went wrong is the useful part.

offset	size	member	evidence
+0x00	4	vptr 0x00a2439c	ctor writes it, RTTI-confirmed
+0x04	2	uint16 otnF (turn first observed)	mov word [eax-0x28],cx at 0x007ba2b0 (eax = _Mylast, element = _Mylast-0x2c), source [ebx+0xc]; tag from ObservedTech::Write
+0x06	2	uint16 otnL (turn last observed)	mov word [eax-0x26],dx at 0x007ba2c6, same source word [ebx+0xc] — first sighting sets first == last
+0x08	1	bool odet	ctor stores a byte (mov [esi+0x8],bl, 0x008562f4); copy-ctor copies a byte; serialised with WriteBool/ReadBool
+0x0c..+0x27	0x1c	std::string otch (tech name)	object base +0x0c, MSVC {_Bx[16] @0, _Mysize @0x10, _Myres @0x14, _Alval @0x18}. Ctor writes [+0x0c]=0, [+0x1c]=0, [+0x20]=0xf; the search loop reads [+0x1c] as the length and calls compare with this = +0x0c; the post-append assign uses lea ecx,[_Mylast-0x20] = +0x0c. +0x24 is _Alval — never read, never written, by anything
+0x28	4	int owith	ctor zeroes it; ObservedTech::Write emits it last

4 + 2 + 2 + 1(+3 pad) + 0x1c + 4 = 0x2c exactly — sizeof is fully accounted for, with no padding slack and no unaccounted field.

Where the mapping came from — the serializer

Game::ObservedTech's vftable 0x00a2439c is the usual 3 slots { [0] 0x00793610 scalar deleting dtor, [1] 0x00817c40 Read, [2] 0x00817cf0 Write }. Write enumerates the entire object, in order, and touches nothing else:

0x00817cff  movzx ecx,word [edi+0x04]   push 0xa2b38c "otnF"   -> stream vft+0x24 (int)
0x00817d0f  movzx ecx,word [edi+0x06]   push 0xa2b384 "otnL"   -> stream vft+0x24 (int)
0x00817d26  lea   eax,[edi+0x08]        push 0xa2b36c "odet"   -> WriteBool  0x008b9c20
0x00817d37  lea   ecx,[edi+0x0c]        push 0xa2b3e8 "otch"   -> WriteString 0x008b9d70
0x00817d46  mov   eax,[edi+0x28]        push 0xa2b364 "owith"  -> stream vft+0x24 (int)

Read (0x00817c40) is the exact mirror: otnF/otnL read as ints and stored back with 16-bit mov word [ebx],ax, odet through ReadBool, otch through ReadString, owith reached as add edi,0x28. That matches the on-disk order save_reader.py already had (Otch = otnF otnL odet otch(string) owith), which is a nice independent agreement between the disassembly and the save oracle.

Game::ObservedWeapon (Write 0x00817bc0, Read 0x00817b10) is the same element shape with tag owep (0x00a2b3f0) in place of otch — a second instance of the identical 0x2c layout.

Lane P's observed_techs region byte delta remains a valid live cross-check and should come back as exactly 44 per completion.

5. False-positive rate, honestly

Two different error rates, and the interesting one is not the one you'd expect.

Decode-level error of the naive method is low. A raw byte scan for 8D + ModRM + the displacement — the thing you'd write without a decoder — is mostly right:

query	naive candidates	not actually an instruction	real sites the naive scan missed
lea, disp32 0x274	19	0 (0.0%)	80 of 99
11 common opcodes, disp32 0x274	89	1 (1.1%)	11 of 99
lea, disp8 0x14	828	1 (0.1%)	13,784 of 14,611
11 common opcodes, disp8 0x14	10,326	59 (0.6%)	4,344 of 14,611

So the decoder's win is recall, not decode precision. A hand-written opcode set misses 80–94% of the real accesses, because a member is read, written, compared, and float-loaded far more often than its address is taken, and you cannot enumerate those opcodes by hand.

Class-level precision is the real problem, and it is poor. The scanner knows the displacement; it cannot know what class the base register holds. query 0x274 returns 99 sites in 45 functions and only about 6 of those functions are actually touching ServerPlayer::otch — roughly 13% precision by function. The honest way to state the value is as search-space reduction: 41,411 functions → 45, about 900×, down to a list a human reads in two minutes.

The cohort ranker helps, and has a trap. Scoring functions by how many already-known offsets of the same class they touch pulls real class methods to the top (the serializer hits 50/50). But it would have thrown away the correct answer: RecordObservedTech touches only 0x274 and 0x278 and nothing else on ServerPlayer, so any --min>=1 filter drops it. Use cohort as a ranker, never as a filter. This is now written into the tool's own docstring.

What actually closed the case was the plain query 0x274 list intersected with one call-graph lookup (OnTechResearched's direct callees). Displacement scan for recall, call graph for disambiguation — neither alone was enough.

6. Previously anonymous offsets — attribution results

The payoff was smaller than hoped, because most of the audit's anonymous offsets had already been named by other lanes since the audit was written.

offset	status before	after
player+0x274/0x278/0x27c	"vector grew, append site unknown"	RecordObservedTech+0xdf (0x007ba27f), sizeof = 0x2c
player+0x2b0	already named by lane E	unchanged (EvNxID)
TechTree+0x20	already TechTree_off_OrderCounter	unchanged
ServerPlayer+0x308	already ServerPlayer_off_NodeBore	unchanged
Budget+0x64	harness-audit row 11: "the original writes it"	not supported. See below

Budget+0x64 (harness-audit row 11) — a correction. ServerPlayer::ComputeBudget (0x00863030) writes its Budget* out-param through esi, and every such store lands in +0x00..+0x54 (the 22-int block). The only two +0x64 accesses in the whole function are loads off a different base register (mov ecx,[eax+0x64] at 0x0086328c, mov edx,[eax+0x64] at 0x0086335d). There is no store to Budget+0x64 in ComputeBudget.

Separately, TechTree::ProcessResearch's int* overbudget fourth argument is not Budget+0x64: its only caller is ProcessTurn (0x00891340) and the call at 0x008914a5 passes lea edx,[ebp-0x14] — a stack local, consumed immediately after the call (mov eax,[ebp-0x14]; cmp eax,0; jle).

This agrees with lane R, whose budget_object guard saw Budget+0x64 change in 0 of 4284 calls. Row 11 should be reclassified from "the original writes it and B1 never checked" to "nothing has been shown to write it"; the remaining way to settle it is a write watchpoint on that word, not another static search.

7. Verdict on the technique

Worth keeping, with its limits stated. It answered a question that had been parked, and the 0x29c validation turned up a ProcessTurn EventStorage site nobody had. But it is a recall tool that produces a 20–100 line candidate list per offset, not an oracle: every result still needs a call-graph or decompiler check before it is a fact. For the ~1,600 remaining classes the realistic workflow is query <offset> → read the list → confirm with one cross-reference call.

8. Ghidra writeback

Labels: RecordObservedTech 0x007ba1a0, vector_ObservedTech_push_back 0x007b7320, ObservedTech_ctor 0x008562a0, vector_ObservedTech_assign 0x00872380, vector_44B_grow 0x007b5820, vftable_ObservedTech 0x00a2439c. Plate comments carrying the stride evidence on the first four.

addresses.json: ObservedTech_sizeof, ObservedTech_vftable, ObservedTech_off_Name, RecordObservedTech, vector_ObservedTech_push_back, vector_ObservedTech_assign, ObservedTech_ctor, vector_44B_grow; ServerPlayer_off_ObservedTechs updated from "NOT PINNED" to verified.

Lane S round (2026-09-08). Labels: ObservedTech_Write 0x00817cf0, ObservedTech_Read 0x00817c40, ObservedTech_scalar_deleting_dtor 0x00793610, ObservedWeapon_Write 0x00817bc0, ObservedWeapon_Read 0x00817b10, vector_ObservedTech_uninit_copy 0x0079a150, vector_string_find_by_name 0x00699bd0. Prototypes on the five class methods. Plate comments carrying the full member map on the two serializers, the ctor, the dtor and the copy helper, and the sizeof(std::string) = 0x1c fact on Stream::WriteString 0x008b9d70 and on the vector<string> stride site 0x00699bd0 — the two places a future lane is most likely to look. addresses.json +10 entries (std_string_sizeof, ObservedTech_Read/_Write/_dtor/_copy_ctor, ObservedTech_off_TurnFirst/_TurnLast/_Detected/_With, ObservedWeapon_Write), 4 corrected.

---

9. std::string is 0x1c, not 0x18 — the correction, and why it mattered (lane S, 2026-09-08)

§4 above originally reported the embedded string as 0x18 bytes, against the campaign-wide _Bx@0, _Mysize@0x10, _Myres@0x14, _Alval@0x18, sizeof 0x1c. Lane X flagged it as "worth re-checking" rather than asserting it, which was the right call: 0x1c is correct, and it is correct everywhere in this binary. ObservedTech+0x24 is the string's own trailing allocator word, not a data member.

Why the 0x18 reading looked right

Everything lane X observed was accurate. The string's live fields really do stop at +0x14 (_Bx@0, _Mysize@0x10, _Myres@0x14), because _Alval is std::allocator<char> — an empty class. It occupies a word of the object but is never loaded or stored, so it is invisible to any evidence based on what the code touches. Sizing a type from its accessed fields undercounts it by exactly the tail padding. The same trap is live for std::vector in this build, which is {_Myfirst, _Mylast, _Myend, _Alval} = 0x10 while only three words are ever read (events.md already records the _Alval word at EventStorage+0x10 and +0x14).

The three things that settle it inside ObservedTech

Each is a complete enumeration of the object, which is the right instrument here — an enumeration can show a field's absence, a touch-scan cannot.

1. ObservedTech::Write 0x00817cf0 serialises +0x04, +0x06, +0x08, +0x0c, +0x28. No +0x24. (Read 0x00817c40 mirrors it.)
2. ObservedTech_ctor 0x008562a0 initialises +0x00, +0x04, +0x08, +0x0c(string), +0x28. No +0x24 — while zeroing every other scalar in the object.
3. The copy constructor, inlined at 0x0079a184 inside the vector's uninitialised-copy helper FUN_0079a150, copies +0x04, +0x06, +0x08, the string at +0x0c, and +0x28. No +0x24.

A 4-byte data member that the constructor, the copy constructor and the serializer all ignore is not a data member.

Binary-wide check — tools/strfootprint.py

One class is an anecdote, so the same question was put to the whole binary. Every serializer hands member pointers to the Mars::Stream primitive helpers with a fixed idiom (lea r,[base+disp]; push r; push tag; push stream; call helper), so the scanner recovers (base, disp, tag) for every string site and then asks: does the same function touch any other offset inside (N, N+0x1c) off the same base register?

string helper call sites            : 241
  resolved to a class member offset : 65
  stack temporaries (ebp/esp base)  : 30
  offset not reached by a plain lea : 146

members with a sibling inside (N, N+0x1c) : 0

gap from a string member to the next member on the same base:
  +0x1c   : 51
  +0x20   : 1

65 string members across every serializer in the exe, zero collisions, and 51 of the 52 measurable gaps are exactly 0x1c. The single +0x20 is StrategyServer::KeyPath at +0x134, whose Read side gives +0x1c to +0x150 — the writer simply skips a member. There is no 0x18 instantiation, no empty-base-optimised variant, and no game-local string class. One layout, 0x1c.

Two exclusions matter or the scan reports noise, and both are why a naive version of this would have "confirmed" 0x18:

• ebp/esp bases are stack temporaries, not members. A local string at [ebp-0x2c] shows accesses at -0x1c and -0x18 — which read exactly like two sibling fields inside the span. All 30 such sites are excluded.
• N+0x10 and N+0x14 are the string's own _Mysize/_Myres, touched inline whenever the compiler expands the _Myres >= 16 ? _Ptr : _Buf test at a call site.

Independent corroboration outside the scan

• FUN_00699bd0 walks a vector<std::string> doing the SSO test at [esi+0x14]/[esi] — element base is string base — and advances add esi,0x1c (0x00699c29). The stride of a vector of strings is sizeof(std::string).
• EventStorage::PostEvent 0x008862b0 takes two by-value std::strings at [ebp+0x08] and [ebp+0x24] (spacing 0x1c) and RET 0x4c = 2*0x1c + 5*4.
• MoraleEvent is 0x50 bytes with its name string at +0x34: 0x34 + 0x1c = 0x50 exactly.

Blast radius — what actually had to change

Nothing in any recovered struct table. Every string-bearing layout in struct-recovery.md, save-editor-structs.md, events.md and schema-gaps-resolved.md already used 0x1c spans and 0x1c inter-field gaps, and the binary agrees with all of them. ServerPlayer::pswd at 0x2dc..0x2f7 — the row §4 asked to re-check — is correct: Write 0x008563e0 puts the next member exactly 0x1c above it. The corrections were confined to three prose statements that had propagated the 0x18 number (loader-prototypes.md conventions line, the GlobalConst_ParseString prototype in addresses.json, and §4 here) plus one transposition in struct-recovery.md §0 that said size@0x14, res@0x18 where the verified offsets are 0x10/0x14.

The lesson worth carrying, and the reason this was worth chasing rather than reconciling: do not size a struct member from the offsets the code touches. Trailing empty-allocator words in this build's STL are invisible to a touch-scan and cost exactly 4 bytes every time. Size types from an enumeration — a serializer, a constructor, a copy constructor, or a container stride.