Compare commits

..

No commits in common. "d523d27f50495c8e7c55aea1bbcef77aca5c591d" and "1a58bcdf90998978d8ce4df59c776ca115b3e8c3" have entirely different histories.

8 changed files with 8 additions and 1240 deletions

View file

@ -68,7 +68,7 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`).
| RollResearchEvent draw (behavioural) | verify | verified | high | 100% | 2026-09-08 | **CLOSED by lane R.** On turn 6 (`IND_TRKSTL`, tech 10094) `research_roll_pending` was true going in; the original drew exactly one word (rng `left` 375->374, `next_index` 249->250) and cleared the flag, and `ours` reproduced both bit-for-bit on the scratch generator. 0 divergences on that call. Needs 5 End Turns from ref-turn2 to reach - the reference turn has no completion at all |
| golden-trace recapture (post-guards) | verify | verified | high | 100% | 2026-09-08 | DONE on the live game, build `recap-7584bad-20260908T0615Z` (NO source change needed - the audit's machinery did all of it). **B3 ProcessResearch: the defect is VISIBLE** - `side.events.after.v.next_id orig=4 ours=3`, one divergent call of 3 on the reference turn and its ONLY divergent field; bit-for-bit the `EvNxID 4->3` that previously needed a 609 KB save diff. Over 5 turns 15 calls / 3 diverged, **RNG matched 15/15**, and the two completion calls miss TWO event ids each. **B1 ComputeBudget: verdict held** - 4284 compared, 0 diverged, exit 0, `budget_object` guard caught 0 undeclared writes (Budget+0x64 never changed value). **MoveFleet: 8 of 45 diverge by 1 ULP of position** (new; see its own row). First guarded captures for OnTechResearched (2 calls), ServerSystem::ProcessTurn (140 calls) and MoveFleet (45). Guards mapped SetResearched live (ConMod[0..2], OutMod, PopMod, ResTNm, TechTree+0x20 order counter) and found an UNDECLARED `vector<ObservedTech> otch` append at player+0x274. Oracle held on every run's first End Turn. Report `findings/subsystems/golden-trace-recapture.md`; engine `docs/R-recapture.md`; traces `verify/traces/recap-*`, reports `verify/results/compare/recap-*`. sots-engine branch `wip/recapture` e50d5e5 (merged with main 82ef52f; ctest 32/32, clean-room OK) |
| MoveFleet position rounding (1 ULP) | verify | verified | high | 100% | 2026-09-08 | **CLOSED by lane M.** Mechanism read off the instruction stream, not fitted: the engine's `Mars_Vec3_Normalize` (0x00422520, 123 callers) narrows to float32 FOUR times - `sumsq = f32(x*x+y*y+z*z)` (products/adds stay in 53-bit regs, only the SUM is stored), `len = f32(sqrt(sumsq))`, `inv = f32(1.0/len)` a RECIPROCAL that is MULTIPLIED through rather than three divides, and `dir.c = f32(delta.c*inv)`; and MoveFleet stores each `dest.c - pos.c` BACK TO A FLOAT32 SLOT before calling it, and takes the leg distance from that same call's return value. `ours` did all of it in double. The position tail (`f32(pos + f32(dir*move))`) was already right, which is exactly why the error was a constant ABSOLUTE ~1.2e-7. Confirmed OFFLINE first (an arrival copies the destination verbatim, so calls 115/155 hand you fleet 34's and fleet 50's exact float32 destinations = 8 fully determined legs; the 5-narrowing model reproduces the ORIGINAL bit-for-bit on all 8, the old double model reproduces `ours` on the 3 divergent ones), then LIVE: control run 8/45 diverged exit 1, fixed run **0/45 diverged exit 0**, with identical args, identical pos.before and identical ORIGINAL pos.after on all 45 calls. Report `findings/subsystems/movefleet-position-rounding.md`, engine `docs/M-movefleet.md`, branch `wip/movefleet` 2aa8cba |
| undeclared ObservedTech append | verify | backlog | — | 0% | 2026-09-08 | NEW (lane R). A tech completion grows `vector<ObservedTech> otch` at ServerPlayer+0x274 (all three vector words move = a realloc). Seen as an undeclared write by BOTH the ProcessResearch `player` guard and the OnTechResearched `player` guard. It is serialized ServerPlayer state and it is in NO coverage note anywhere - a third list append in the same neighbourhood as the event list. B3's replace oracle never saw it because turn 1 of ref-turn2 has no completion | **APPEND SITE NOW NAMED (lane X): `RecordObservedTech` 0x007ba1a0, called from OnTechResearched; stride 0x2c.** Still needs a declared region + a model in ours.
| undeclared ObservedTech append | verify | backlog | — | 0% | 2026-09-08 | NEW (lane R). A tech completion grows `vector<ObservedTech> otch` at ServerPlayer+0x274 (all three vector words move = a realloc). Seen as an undeclared write by BOTH the ProcessResearch `player` guard and the OnTechResearched `player` guard. It is serialized ServerPlayer state and it is in NO coverage note anywhere - a third list append in the same neighbourhood as the event list. B3's replace oracle never saw it because turn 1 of ref-turn2 has no completion |
| unnamed offsets from guard hits | verify | backlog | — | 0% | 2026-09-08 | NEW (lane R). Three spans the guards report every run and no addresses.json entry names: **ServerSystem+0xd8 (1 B)** and **ServerSystem+0x238 (4 B)** - written by the AI home system on every colony turn, alongside the fleet-vector growth; **StarFleet+0xdc (1 B)** - written on every moving MoveFleet call, just past Speed (FPsp2 @0xd8). Cheap wins for the contract |
| waypoint types 2-5 have no coverage | verify | backlog | — | 0% | 2026-09-08 | NEW (lane M, promoted from a coverage line to its own row because it is now the biggest gap in `MoveFleet`). Types 2 (node line), 3 (node route), 4 (gate teleport) and 5 (probabilistic jump) have NEVER fired in any capture, and the node-line step is WRONG BY CONSTRUCTION - `sim::NodeLineStep` and `sim::BuildStutterSegments` are written and host-tested but are NOT wired into the hook, which steps every waypoint type as `speed x dt`. **`ref-turn2` structurally cannot exercise them**: lane M held the VM and tried. The only mover in that save is the AI, which travels straight runs; the player that would travel a node line has `DE 00 CR 00 DN 00` at its home system (screenshot `verify/results/shim/mf-human-home-no-ships.png`), so Move/Manage Fleets are greyed out on every turn. Needs a ship built over several turns, or - much cheaper - a PURPOSE-BUILT SAVE with a fleet already in orbit next to a node line. Same save would unblock the gate-traffic and probabilistic-jump rows. Also owed on that path: `sim::Distance` is still plain double, and `Mars_Vec3_Length` (0x004224b0) says every vector length in the engine is float32-narrowed twice, so the stutter geometry is probably 1 ULP out the same way the position update was - deliberately left alone by lane M because there is no behavioural evidence to correct it against |
| ref-turn2 has no tech completion | meta | verified | high | 100% | 2026-09-08 | TRAP for anyone writing a workload (lane R). The documented one-End-Turn recipe produces **zero** `OnTechResearched` calls - an empty log that still passes. It takes 5 End Turns (to turn 7) to reach a completion. Also: only the FIRST End Turn is reproducible - it hashed to the oracle on all four runs and its research calls reproduce docs/B3.md exactly, but from turn 4 the AI picks a different research target than B3 recorded while the point totals stay nearly identical. Treat anything past turn 1 as *a* run, not *the* run |
@ -85,10 +85,4 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`).
| MoveFleet waypoint types 2-5 | verify | backlog | — | 0% | 2026-09-08 | Still ZERO behavioural coverage after lane M. Not for lack of trying: the only mover in ref-turn2 is the AI (straight runs only), and the player that would travel a node line has DE/CR/DN all 00 at its home system, so Move/Manage Fleets are greyed out every turn - there is nothing to send along the node lines the map draws. Needs a ship built over several turns or a purpose-built save. The type-2 node-line step is still WRONG BY CONSTRUCTION (B4). Also: sim::Distance deliberately left in double (only stutter geometry uses it); Mars_Vec3_Length says it is probably 1 ULP out the same way, but there is zero behavioural evidence to correct it against - do not "fix" it blind |
| P2-P event posting in ours | phase2 | mapped | high | 80% | 2026-09-08 | HOST-VERIFIED, VM RUN QUEUED (lane F holds VM140). next_id reaches 4 in a host reproduction of recap-b3 call 0, fixture rebuilt from raw bytes at the real 0x1c/0x18/0x74 strides and cross-checked against turn3-state.sav with lane C's state_checksum --tree. Count-only (lane E option a): ours never calls the game's PostEvent and REPLACE MODE WRITES NOTHING - a bumped EvNxID with no record behind it would corrupt the very save the oracle hashes. Three design points: the event scan is taken in describe_args BEFORE the original (taken after, ours would dedup against the original's own posts and agree for the wrong reason); dedup risk is MEASURED and reported as events_dedup_risk, not assumed; KeylessEventText resolves keys to "%s" so the shim carries no prose. VERIFIED from the instruction stream: SetResearched 0x00581e10 calls owner vft+0x10 with (flags>>2)&1 and ProcessResearch passes flags=2, so silent=false and the completion event IS posted - previously only inferable from "EvNxID moved by two". ctest 33/33, shim cross-builds on CT111 (lane P could only syntax-check) |
| EVENT_TECHS_UNLOCKED not posted (predicted residual) | verify | backlog | — | 0% | 2026-09-08 | Lane P FLAGGED RATHER THAN GUESSED. Trigger IS pinned (SetResearched's sweep sets state=2 + stamps turnAvailable sticky at -1; tail loop collects state==2 && turnAvailable==currentTurn) but evaluating it needs the unlock cascade ours deliberately does not run. The driver takes the unlock list as an INPUT and is handed nullptr ("no list") - deliberately distinct from an empty list ("computed, empty"). PREDICTED RESIDUAL: next_id short by exactly 1 on every completion call. Posting it "whenever something completed" would score on this save and be WRONG the first time a completion unlocks nothing - the exact false-pass shape this project keeps catching |
| sizeof(ObservedTech) unpinned | objects | verified | high | 100% | 2026-09-08 | **PINNED (lane X).** `sizeof(Game::ObservedTech) = 0x2c (44)` -- three independent proofs: the magic divide `0x2e8ba2e9 sar 3` (= /44, exact) at 0x0087239f, `imul reg,reg,0x2c` at 0x0087243a / 0x007b735b, and the search stride `add edi,0x2c` at 0x007ba257. **Append site = `RecordObservedTech+0xdf` (0x007ba27f): `lea ecx,[player+0x274]; call vector_ObservedTech_push_back 0x007b7320`** -- a de-duplicating append, direct callee of OnTechResearched 0x00891790; the realloc through 0x007b5820 is why all three vector words move. Element: vptr 0x00a2439c at +0 (RTTI `.?AVObservedTech@Game@@`), std::string at +0x0c (0x18 B); +0x04/+0x06 (16-bit) and +0x08/+0x24/+0x28 hold the four on-disk ints in an order NOT yet determined. Built the general tool the row asked for: `tools/x86disp.py`, an x86 displacement xref scanner (100% code coverage, 0.17% desync). `findings/subsystems/observedtech-append.md` |
| lea-displacement xref scanner | meta | verified | high | 100% | 2026-09-08 | `tools/x86disp.py` -- fixes the systemic blind spot that Ghidra does not index ModRM displacements. Full x86-32 length decoder swept from Ghidra's 41,089 function starts: 2,174,504 instructions, 612,166 displacement sites, **100.0% code coverage, 70 desyncs (0.17%), zero unknown opcodes**. Validated against ground truth before use (re-finds `lea eax,[ecx+0x29c]` in GetEventStorage, both OnTechResearched +0x29c sites, and one NEW ProcessTurn site). HONEST LIMITS: it is a **recall** tool, not an oracle -- class-level precision at 0x274 is ~13% by function (99 sites / 45 functions, ~6 real), i.e. a 900x search-space cut that still needs one call-graph check. The naive byte scan it replaces is not wrong so much as **blind**: it misses 80/99 real sites at 0x274 and 13,784/14,611 at disp8 0x14. `cohort` ranking must never be used as a hard filter -- it would have discarded the correct ObservedTech answer. Works off a gitignored local cache in `dumps/`, so it does not hammer CT111 |
| lea-displacement xref scanner | meta | verified | high | 100% | 2026-09-08 | `tools/x86disp.py` - full x86-32 length decoder swept from 41,089 Ghidra function starts: 2,174,504 instructions, 612,166 disp sites, 0 unknown opcodes, 100.0% code coverage, 0.17% desyncs. VALIDATED against ground truth before any new claim (rediscovers GetEventStorage's lea ecx+0x29c, EvNxID +0x2b0; positive control: given 50 known ServerPlayer offsets, FUN_0087fac0 scores 50/50 = the serializer, nothing close). BUILD GOTCHA: clipping sweeps at fva+Ghidra sizeInBytes lost 11% of functions to mid-instruction truncation; sweeping to the NEXT function start took coverage 89% -> 100%. HONEST LIMIT: the win is RECALL not precision - naive lea-only scan MISSES 80 of 99 real 0x274 sites; class-level precision ~13% by function. Value = search space 41,411 -> 45 (~900x), then disambiguate by call graph |
| RANKER TRAP: cohort filter discards correct answers | meta | verified | high | 100% | 2026-09-08 | Lane X's cohort ranker WOULD HAVE DISCARDED THE CORRECT ANSWER. RecordObservedTech touches only 0x274/0x278 and nothing else on ServerPlayer, so every --min>=1 cohort filter drops it. What actually closed the case was plain `query` + ONE call-graph lookup on OnTechResearched's callees. RULE now in the tool docstring: it is a RANKER, NEVER a filter. Displacement scan for recall, call graph for disambiguation; neither alone sufficed |
| ObservedTech struct | objects | verified | high | 100% | 2026-09-08 | sizeof = 0x2c (44) by THREE independent proofs: exact magic divide 0x2e8ba2e9 sar 3 (= ceil(2^35/44), emulated against n=0..1000) at 0x0087239f; imul reg,reg,0x2c at 0x0087243a/0x007b735b; search stride add edi,0x2c at 0x007ba257. APPEND SITE: RecordObservedTech+0xdf (0x007ba27f) `lea ecx,[player+0x274]; call vector_ObservedTech_push_back 0x007b7320`. RecordObservedTech (0x007ba1a0) is a DIRECT CALLEE of OnTechResearched and DE-DUPLICATES BY TECH NAME before appending - a naive push_back in the reimpl WOULD DIVERGE on re-observation. Realloc through 0x007b5820 explains why lane R saw all three vector words move. Element: vptr 0x00a2439c at +0 (RTTI .?AVObservedTech@Game@@). The four on-disk ints map onto +0x04/+0x06 (16-bit) and +0x08/+0x24/+0x28 - NOT DETERMINED, lane X did not guess it; settle via ObservedTech's serializer or lane P's live byte delta (should read exactly 44) |
| std::string size 0x18 vs 0x1c CONTRADICTION | objects | backlog | — | 0% | 2026-09-08 | Lane X reports ObservedTech's embedded std::string at +0x0c occupying 0x18 bytes, NOT the 0x1c our standing MSVC fact asserts (_Bx@0, _Mysize@0x10, _Myres@0x14, _Alval@0x18). 0x1c is used across MANY recovered layouts, so if the real allocator member is elided (empty-base optimisation) in some instantiations we may have LATENT OFF-BY-4 ERRORS in other structs. Foundational - audit before trusting any further string-bearing layout |
| harness-audit row 11 CORRECTED (Budget+0x64) | verify | verified | high | 100% | 2026-09-08 | Lane X: row 11 is NOT SUPPORTED. ComputeBudget writes its Budget* only through esi into +0x00..+0x54; its only two +0x64 accesses are LOADS OFF A DIFFERENT BASE. And ProcessResearch's `int* overbudget` is a ProcessTurn STACK LOCAL (lea edx,[ebp-0x14] at 0x008914a5), not Budget+0x64. Agrees with lane R's 0-of-4284 guard result. Reclassified to "nothing shown to write it"; only a watchpoint settles it definitively |
| sizeof(ObservedTech) unpinned | objects | backlog | — | 0% | 2026-09-08 | Lane P declared the region at ServerPlayer+0x274 but did NOT model the bytes. The append site is in neither OnTechResearched's nor SetResearched's decompilation, and `find-constant-uses` for 0x274 returns only unrelated objects because GHIDRA DOES NOT INDEX `lea` DISPLACEMENTS - a systemic blind spot, not a one-off. The declared region's byte delta on a completion call will MEASURE the struct size. Worth a raw-byte scanner for lea-displacement xrefs as a reusable tool |

View file

@ -284,11 +284,6 @@ said "an arriving call was simply clean"; that is still true and still means not
pinned against the instruction stream.
* `ServerSystem+0xd8` and `ServerSystem+0x238` are unnamed and are written every turn by the
AI home system. `StarFleet+0xdc` likewise, on every move.
> **Follow-up 2026-09-08 (lane X):** the append is now named — `RecordObservedTech` `0x007ba1a0`,
> called directly from `OnTechResearched`; `sizeof(ObservedTech) = 0x2c`; the realloc that moves all
> three words is `vector_44B_grow` `0x007b5820`. See `findings/subsystems/observedtech-append.md`.
> That note also corrects harness-audit row 11: nothing in `ComputeBudget` stores to `Budget+0x64`.
* `vector<ObservedTech> otch` (`ServerPlayer+0x274`) is undeclared everywhere and is save
state.
* Nothing here exercises: replace mode (any hook), a Zuul double roll, a rebellion, an

View file

@ -1,278 +0,0 @@
# `vector<ObservedTech> otch` — pinned, and the displacement scanner that pinned it
Lane X. Static/host only; VM140 was held by lane F and the game was never run.
Two things here: a general-purpose tool (`tools/x86disp.py`) for a class of cross-reference
Ghidra cannot index, and the `ServerPlayer+0x274` answer it was built to get.
---
## 1. The gap the tool fills
Ghidra indexes **immediate** operands. It does not index **ModRM displacements**. On a
41,411-function MSVC C++ binary `lea reg,[reg+disp]` is *the* idiom for taking the address of
a member — it is how every `std::vector`, every `std::string`, every embedded sub-object gets
passed to a method or a constructor. So `find-constant-uses 0x274` returned 13 unrelated hits
and none of them was the vector, and lane P correctly concluded the search was a limitation
rather than evidence of absence.
Lane E only found `EventStorage` at `+0x29c` because a whole accessor function happened to
exist (`lea eax,[ecx+0x29c]; ret`). We were blind to this whole reference class all campaign,
with ~1,600 classes still to go.
## 2. The tool — `tools/x86disp.py`
A real x86-32 **length decoder** (legacy prefixes, 1-/2-/3-byte opcodes, ModRM, SIB, disp8
sign-extended, disp32, every immediate form), swept from Ghidra's function starts so every
decode begins on a genuine instruction boundary. It indexes every memory operand that carries
a displacement, then answers "what code touches offset N off some object?" with the containing
function, instruction address, base register and decoded instruction.
Correctness points that would otherwise produce confident garbage, all handled:
* `mod=0,rm=5` and `mod=0,rm=4,sib.base=5` are **disp32 with no base** — an absolute global
address, not a member offset. Excluded from member queries. (Without this, every
`mov eax,[0x00a2bd88]` in the binary shows up as a "displacement".)
* `mod=1` disp8 is **sign-extended**: `-0x08` must not be reported as `+0xf8`.
* `mod=3` is a register operand with no displacement at all.
* `0x67` address-size prefix means 16-bit ModRM, a different layout entirely — refused rather
than mis-decoded.
Commands: `build`, `query <disp>`, `cohort`, `func <va>`, `dis <va>`, `stats`, `brute`.
**Decode quality:**
```
functions swept : 41089
instructions : 2174504
disp sites : 612166
desyncs : 70 (0.17% of functions)
code coverage : 6142049/6142358 (100.0%)
```
Zero unknown-opcode desyncs — the opcode tables cover everything this binary contains. All 70
desyncs are truncations at a section/function boundary, not lost sync inside real code.
> Gotcha worth carrying: the first build clipped each sweep at `fva + Ghidra's sizeInBytes`
> and lost 11% of functions to mid-instruction truncation — Ghidra's `sizeInBytes` understates
> real bodies often enough to matter (it was cutting valid `mov esp,ebp; pop ebp; ret`
> epilogues in half). Sweeping to the **next function start** instead took coverage 89% → 100%.
The tool works off a local cache (`dumps/`, gitignored: the exe, the function list, the index)
so it never hammers the shared Ghidra box. `tools/cache_functions.py` pulls the function list
once.
## 3. Validation — it rediscovers what we already knew
Run before any new claim was made.
**GT1 — `ServerPlayer::GetEventStorage`, the one accessor we already had:**
```
$ uv run python3 tools/x86disp.py func 0x0080db00
ServerPlayer_GetEventStorage @ 0x0080db00 size 7
0x0080db00 lea eax,[ecx+0x29c] ServerPlayer_GetEventStorage+0x0 (8d819c020000)
```
**GT2 — lane E's two `OnTechResearched` sites, plus one it did not have:**
```
$ uv run python3 tools/x86disp.py query 0x29c --lea
ProcessTurn @0x00891340
0x00891713 lea ecx,[esi+0x29c] ProcessTurn+0x3d3 <-- NEW
OnTechResearched @0x00891790
0x008919ab lea ecx,[esi+0x29c] OnTechResearched+0x21b <-- known
0x0089241d lea ecx,[esi+0x29c] OnTechResearched+0xc8d <-- known
```
**GT3 — `EvNxID` at `ServerPlayer+0x2b0`:** 90 sites found; the `ServerPlayer` ones are present.
This one is also the honest illustration of the precision problem — see §5.
**GT4 — positive control for the cohort ranker.** Given the 50 `ServerPlayer` offsets already
in `struct-recovery.md`, `FUN_0087fac0` scores **50 co-hits out of 50** — it is the
`ServerPlayer` serializer, and nothing else in the binary comes close. A scanner that could
not surface that function from a bare offset query would not be worth using.
## 4. The answer: `sizeof(ObservedTech)` and the append site
### `sizeof(Game::ObservedTech) = 0x2c` (44 bytes) — verified, three independent ways
**(a)** `vector<ObservedTech>::operator=` at `0x00872380` divides the vector's byte span by a
constant using MSVC's magic-number sequence:
```
0x00872398 mov ecx,[edi+0x4] ; src._Mylast
0x0087239b mov edi,[edi] ; src._Myfirst
0x0087239d sub ecx,edi ; byte span
0x0087239f mov eax,0x2e8ba2e9
0x008723a4 imul ecx
0x008723a6 sar edx,3
```
`ceil(2^35 / 44) == 0x2e8ba2e9` **exactly**, and emulating the full sequence reproduces
n = 0,1,2,3,10,71,1000 from spans of 0,44,88,132,440,3124,44000. Divisor = 44, unambiguously.
**(b)** The same function then multiplies back by the literal stride:
```
0x0087243a imul ecx,ecx,0x2c
0x0087243d add ecx,[esi] ; this->_Myfirst + n*44
0x00872441 mov [esi+0x4],ecx ; this->_Mylast = ...
```
Same literal at `0x00872468` and at `0x007b735b` inside `push_back`.
**(c)** The linear search in the append function advances its iterator by `add edi,0x2c`
(`0x007ba257`).
This matches the on-disk lower bound *exactly* — 4 × `int` + one `0x1c` `std::string` = 44 —
so there is no padding slack anywhere in the element.
### The append site
```
FUN_007ba1a0 = RecordObservedTech (direct callee of ServerPlayer::OnTechResearched 0x00891790)
0x007ba221 mov edi,[esi+0x274] ; it = observer->otch._Myfirst
0x007ba227 cmp edi,[esi+0x278] ; ... != _Mylast ?
0x007ba22d je 0x007ba274 ; empty -> append
loop: compare each element's name string (this = elem+0x0c, length = [elem+0x1c])
0x007ba257 add edi,0x2c ; ++it *** stride 44 ***
0x007ba25a cmp edi,[esi+0x278]
0x007ba260 jne loop
0x007ba274 lea ecx,[ebp-0x3c]
0x007ba277 call 0x008562a0 ; ObservedTech::ObservedTech() on the stack
0x007ba27e push eax
0x007ba27f lea ecx,[esi+0x274] ; this = &player->otch <<< THE APPEND
0x007ba288 call 0x007b7320 ; vector<ObservedTech>::push_back
```
`push_back` (`0x007b7320`) grows via `0x007b5820` when `_Mylast == _Myend` — **that realloc is
exactly why lane R's guard saw all three of `player+0x274/0x278/0x27c` move on a completion**,
rather than only `_Mylast`.
`RecordObservedTech` is called from `OnTechResearched` (`0x00891790`) and from `0x007be228`,
`0x007be4e1`, `0x007be535`. It is a **de-duplicating** append: it appends only if no existing
element already carries that tech name — worth knowing for the reimplementation, since a naive
`push_back` would diverge on a re-observation.
Neither `push_back` nor `operator=` is COMDAT-ambiguous: `0x007b7320` has exactly one caller
(`RecordObservedTech`, `ecx = player+0x274`) and `0x00872380` has two, both passing
`ServerPlayer+0x274`. `0x007b5820` **is** shared with `FUN_0086dec0` and may be a folded body,
so it is labelled `vector_44B_grow`, not as ObservedTech-specific.
### Element layout — as far as the evidence actually goes
The default ctor at `0x008562a0` writes vtable `0x00a2439c` at `+0x00`. RTTI:
COL `0x00a81c78` → type descriptor `0x00aeede4` → **`.?AVObservedTech@Game@@`**. So
`ObservedTech` is polymorphic and its first word is a vptr, not a data field — which the
on-disk shape does not tell you.
| offset | size | evidence |
|---|---|---|
| `+0x00` | 4 | vptr `0x00a2439c`; ctor writes it, RTTI-confirmed |
| `+0x04` | 2 | `mov word [eax-0x28],cx` at `0x007ba2b0` (`eax` = `_Mylast`, element = `_Mylast-0x2c`), source `[ebx+0xc]` |
| `+0x06` | 2 | `mov word [eax-0x26],dx` at `0x007ba2c6`, **same source word** `[ebx+0xc]` |
| `+0x08` | 4 | **unaccounted** |
| `+0x0c..+0x23` | 0x18 | `std::string` (the `otch` tech name). Object base is `+0x0c`, MSVC layout `{_Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14}`. Ctor writes `[+0x0c]=0`, `[+0x1c]=0`, `[+0x20]=0xf`; the search loop reads `[+0x1c]` as the length and calls the compare with `this = +0x0c`; the post-append assign uses `lea ecx,[_Mylast-0x20]` = `+0x0c` |
| `+0x24` | 4 | **unaccounted** |
| `+0x28` | 4 | **unaccounted** |
Note the string here is **0x18 bytes**, not the 0x1c the `ServerPlayer::pswd` row in
`struct-recovery.md` implies — three separate reads inside this element agree on
`{buf16, _Mysize@+0x10, _Myres@+0x14}`. Worth re-checking `pswd` against that.
The four on-disk ints (`otnF`, `otnL`, `odet`, `owith`) have to map onto `+0x04`/`+0x06` (two
16-bit fields) and the three 4-byte slots `+0x08`, `+0x24`, `+0x28`. **That mapping is not
determined here and is deliberately not guessed.** The one suggestive observation, flagged as a
*hypothesis only*: `+0x04` and `+0x06` are two adjacent 16-bit fields written from the same
source word on first observation, which is the shape you would expect of a first-seen /
last-seen turn pair — but nothing here proves it.
What would settle the mapping: read `ObservedTech`'s `Read`/`Write` serializer, where the
member order is explicit. Lane P's `observed_techs` region byte delta remains a valid live
cross-check and should now come back as exactly 44 per completion.
## 5. False-positive rate, honestly
Two different error rates, and the interesting one is not the one you'd expect.
**Decode-level error of the naive method is low.** A raw byte scan for `8D` + ModRM + the
displacement — the thing you'd write without a decoder — is mostly *right*:
| query | naive candidates | not actually an instruction | real sites the naive scan **missed** |
|---|---|---|---|
| `lea`, disp32 `0x274` | 19 | 0 (0.0%) | **80 of 99** |
| 11 common opcodes, disp32 `0x274` | 89 | 1 (1.1%) | 11 of 99 |
| `lea`, disp8 `0x14` | 828 | 1 (0.1%) | **13,784 of 14,611** |
| 11 common opcodes, disp8 `0x14` | 10,326 | 59 (0.6%) | 4,344 of 14,611 |
So the decoder's win is **recall, not decode precision**. A hand-written opcode set misses
80–94% of the real accesses, because a member is read, written, compared, and float-loaded far
more often than its address is taken, and you cannot enumerate those opcodes by hand.
**Class-level precision is the real problem, and it is poor.** The scanner knows the
displacement; it cannot know what class the base register holds. `query 0x274` returns 99 sites
in 45 functions and only about 6 of those functions are actually touching `ServerPlayer::otch`
— roughly **13% precision by function**. The honest way to state the value is as search-space
reduction: 41,411 functions → 45, about **900×**, down to a list a human reads in two minutes.
**The cohort ranker helps, and has a trap.** Scoring functions by how many *already-known*
offsets of the same class they touch pulls real class methods to the top (the serializer hits
50/50). But it would have **thrown away the correct answer**: `RecordObservedTech` touches only
`0x274` and `0x278` and nothing else on `ServerPlayer`, so any `--min>=1` filter drops it. Use
cohort as a ranker, never as a filter. This is now written into the tool's own docstring.
What actually closed the case was the plain `query 0x274` list **intersected with one call-graph
lookup** (`OnTechResearched`'s direct callees). Displacement scan for recall, call graph for
disambiguation — neither alone was enough.
## 6. Previously anonymous offsets — attribution results
The payoff was smaller than hoped, because most of the audit's anonymous offsets had already
been named by other lanes since the audit was written.
| offset | status before | after |
|---|---|---|
| `player+0x274/0x278/0x27c` | "vector grew, append site unknown" | **`RecordObservedTech+0xdf` (0x007ba27f)**, `sizeof` = 0x2c |
| `player+0x2b0` | already named by lane E | unchanged (`EvNxID`) |
| `TechTree+0x20` | already `TechTree_off_OrderCounter` | unchanged |
| `ServerPlayer+0x308` | already `ServerPlayer_off_NodeBore` | unchanged |
| `Budget+0x64` | harness-audit row 11: "the original writes it" | **not supported.** See below |
**`Budget+0x64` (harness-audit row 11) — a correction.** `ServerPlayer::ComputeBudget`
(`0x00863030`) writes its `Budget*` out-param through `esi`, and every such store lands in
`+0x00..+0x54` (the 22-int block). The only two `+0x64` accesses in the whole function are
**loads off a different base register** (`mov ecx,[eax+0x64]` at `0x0086328c`,
`mov edx,[eax+0x64]` at `0x0086335d`). There is no store to `Budget+0x64` in `ComputeBudget`.
Separately, `TechTree::ProcessResearch`'s `int* overbudget` fourth argument is **not**
`Budget+0x64`: its only caller is `ProcessTurn` (`0x00891340`) and the call at `0x008914a5`
passes `lea edx,[ebp-0x14]` — a stack local, consumed immediately after the call
(`mov eax,[ebp-0x14]; cmp eax,0; jle`).
This agrees with lane R, whose `budget_object` guard saw `Budget+0x64` change in **0 of 4284
calls**. Row 11 should be reclassified from "the original writes it and B1 never checked" to
"nothing has been shown to write it"; the remaining way to settle it is a write watchpoint on
that word, not another static search.
## 7. Verdict on the technique
Worth keeping, with its limits stated. It answered a question that had been parked, and the
`0x29c` validation turned up a `ProcessTurn` `EventStorage` site nobody had. But it is a
**recall** tool that produces a 20–100 line candidate list per offset, not an oracle: every
result still needs a call-graph or decompiler check before it is a fact. For the ~1,600
remaining classes the realistic workflow is `query <offset>` → read the list → confirm with one
cross-reference call.
## 8. Ghidra writeback
Labels: `RecordObservedTech` `0x007ba1a0`, `vector_ObservedTech_push_back` `0x007b7320`,
`ObservedTech_ctor` `0x008562a0`, `vector_ObservedTech_assign` `0x00872380`,
`vector_44B_grow` `0x007b5820`, `vftable_ObservedTech` `0x00a2439c`. Plate comments carrying
the stride evidence on the first four.
`addresses.json`: `ObservedTech_sizeof`, `ObservedTech_vftable`, `ObservedTech_off_Name`,
`RecordObservedTech`, `vector_ObservedTech_push_back`, `vector_ObservedTech_assign`,
`ObservedTech_ctor`, `vector_44B_grow`; `ServerPlayer_off_ObservedTechs` updated from
"NOT PINNED" to `verified`.

View file

@ -109,15 +109,7 @@ New `addresses.json` entries from this read: `TechNode_off_TurnAvailable` 0x20,
`TechNode_off_TurnResearched` 0x24, `TechNode_off_Order` 0x28, `TechNode_off_Children` 0x04,
`TechEdge_off_CostRP` 0x1c, `TechEdge_off_ChildDef` 0x40, `TechTree_off_OrderCounter` 0x20.
## 5. `vector<ObservedTech> otch` at `ServerPlayer+0x274` — ~~still not pinned~~ **PINNED**
> **RESOLVED 2026-09-08 by lane X** — see `findings/subsystems/observedtech-append.md`.
> `sizeof(Game::ObservedTech) = 0x2c (44)`. The append is `RecordObservedTech+0xdf` (`0x007ba27f`):
> `lea ecx,[player+0x274]; call vector_ObservedTech_push_back 0x007b7320`. `RecordObservedTech`
> (`0x007ba1a0`) is a **direct callee of `OnTechResearched`** and de-duplicates by tech name before
> appending. The suspicion recorded below was right: `find-constant-uses` missed it because the
> encoding is a `lea` displacement. `tools/x86disp.py` now indexes those.
> The rest of this section is left as written, as the record of what was known at the time.
## 5. `vector<ObservedTech> otch` at `ServerPlayer+0x274` — still not pinned
Lane R's guards caught all three vector words moving on every tech completion, in both the
`ProcessResearch` and the `OnTechResearched` player guards, and it is serialized `ServerPlayer`

View file

@ -1454,9 +1454,9 @@
"name": "ServerPlayer_off_ObservedTechs",
"offset": "0x274",
"convention": "offset",
"prototype": "std::vector<ObservedTech> otch -- 3 words {_Myfirst@0x274,_Mylast@0x278,_Myend@0x27c}; save tag otch. sizeof(ObservedTech) = 0x2c (44), PINNED three ways: magic divide 0x2e8ba2e9 sar 3 (=/44) in vector_ObservedTech_assign 0x0087239f, imul reg,reg,0x2c at 0x0087243a / 0x007b735b, and the linear search stride `add edi,0x2c` at 0x007ba257. Append site = RecordObservedTech+0xdf (0x007ba27f): lea ecx,[player+0x274]; call vector_ObservedTech_push_back 0x007b7320. All three words move because push_back reallocs through vector_44B_grow 0x007b5820.",
"status": "verified",
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
"prototype": "std::vector<ObservedTech> otch (3 words {first,last,end}); save tag otch, element {int otnF, otnL, odet; string otch; int owith}. All three words move on every tech completion (a realloc) -- observed live by both the ProcessResearch and the OnTechResearched player guards. NOT PINNED: the append call site and sizeof(ObservedTech); the byte span the observed_techs region reports is what will measure the stride",
"status": "verified-by-save",
"source": "findings/objects/struct-recovery.md#2 (Read/Write serializers) + findings/subsystems/golden-trace-recapture.md (player guard, both completion calls)"
},
{
"name": "ServerPlayer_off_IncMod",
@ -3169,70 +3169,6 @@
"prototype": "site inside StrategyServer::MoveFleet /* the move != distance branch: exactly two roundings per component, tmp.c = float32(dir.c * move) then pos.c = float32(pos.c + tmp.c). `move` is reloaded from a float32 slot. The sibling branch (move == distance, an EXACT float compare) copies the destination's three words verbatim with mov, so an arrival never steps onto its destination. */",
"status": "verified",
"source": "lane M own disassembly pass 2026-09-08"
},
{
"name": "ObservedTech_sizeof",
"offset": "0x2c",
"convention": "constant",
"prototype": "sizeof(Game::ObservedTech) = 0x2c (44 bytes). Confirmed independently by (a) the compiler's magic division by 44 (mov eax,0x2e8ba2e9; imul; sar edx,3) at 0x0087239f and 0x007b7339, (b) imul reg,reg,0x2c at 0x0087243a, 0x00872468, 0x007b735b, and (c) the iterator advance `add edi,0x2c` in RecordObservedTech's linear search at 0x007ba257. Matches the on-disk lower bound exactly (4 int + one 0x1c std::string = 44), so there is no padding slack.",
"status": "verified",
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
},
{
"name": "ObservedTech_vftable",
"addr": "0x00a2439c",
"convention": "data",
"prototype": "Game::ObservedTech vftable. RTTI COL 0x00a81c78 -> type descriptor 0x00aeede4 = '.?AVObservedTech@Game@@'. Written to element+0x00 by ObservedTech_ctor 0x008562a0 -- so ObservedTech is polymorphic and its first word is the vptr, not a data field.",
"status": "verified",
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
},
{
"name": "ObservedTech_off_Name",
"offset": "0x0c",
"convention": "offset",
"prototype": "std::string (save tag `otch`, the tech name) at ObservedTech+0x0c, 0x18 bytes, spanning +0x0c..+0x23. MSVC layout relative to the string object: _Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14 -- i.e. ObservedTech+0x1c is the length and +0x20 the capacity. Evidence: ObservedTech_ctor 0x008562a0 writes [elem+0x0c]=0, [elem+0x1c]=0, [elem+0x20]=0xf; RecordObservedTech's search reads [elem+0x1c] as the length and calls compare with this=elem+0x0c; the post-append assign uses lea ecx,[_Mylast-0x20]. NOTE this string is 0x18 bytes, not the 0x1c implied by the ServerPlayer pswd row in struct-recovery.md. UNACCOUNTED in the element: +0x08, +0x24, +0x28 (4 bytes each) plus two 16-bit fields at +0x04/+0x06 written from one source word at 0x007ba2b0 / 0x007ba2c6 -- the mapping of the four on-disk ints (otnF, otnL, odet, owith) onto those slots is NOT determined.",
"status": "verified",
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
},
{
"name": "RecordObservedTech",
"addr": "0x007ba1a0",
"convention": "thiscall",
"prototype": "void (this, ServerPlayer* observer, ?, std::string* techName) -- appends to observer->otch. Linear-searches observer->otch (ServerPlayer+0x274) with stride 0x2c comparing each element's name string; if not found, default-constructs an ObservedTech on the stack (0x008562a0) and push_backs it (0x007b7320 @0x007ba288), then writes two 16-bit fields into the new element at +0x04 and +0x06 from param_1+0xc. DIRECT CALLEE of ServerPlayer::OnTechResearched 0x00891790; also called from 0x007be228, 0x007be4e1, 0x007be535. This is the append lane P could not find.",
"status": "verified",
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
},
{
"name": "vector_ObservedTech_push_back",
"addr": "0x007b7320",
"convention": "thiscall",
"prototype": "void (std::vector<ObservedTech>* this, ObservedTech* value) RET 4. Stride 0x2c. Calls vector_44B_grow 0x007b5820 when _Mylast==_Myend -- the realloc that moves all three vector words. Exactly one caller (RecordObservedTech), so this instantiation is not COMDAT-ambiguous.",
"status": "verified",
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
},
{
"name": "vector_ObservedTech_assign",
"addr": "0x00872380",
"convention": "thiscall",
"prototype": "std::vector<ObservedTech>& operator=(const std::vector<ObservedTech>&) RET 4. Both callers pass ServerPlayer+0x274 (0x00878091 in the settings copy-out, 0x00892507 in the copy-in).",
"status": "verified",
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
},
{
"name": "ObservedTech_ctor",
"addr": "0x008562a0",
"convention": "thiscall",
"prototype": "Game::ObservedTech* (ObservedTech* this) -- default ctor; sets vptr 0x00a2439c and empties the name string.",
"status": "verified",
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
},
{
"name": "vector_44B_grow",
"addr": "0x007b5820",
"convention": "thiscall",
"prototype": "vector<T>::_Reserve/grow for a 44-byte element type. Shared with FUN_0086dec0, so it may be a COMDAT-folded body -- do NOT assume it is ObservedTech-specific.",
"status": "mapped",
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
}
]
}

View file

@ -1,5 +1,5 @@
// GENERATED — do not edit. Facts about Sword of the Stars.exe (GOG 1.8.1).
// Source: sots-re ghidra/addresses.json @ 1a58bcd, generated 2026-09-08 by tools/gen_addresses.py
// Source: sots-re ghidra/addresses.json @ b0ef139, generated 2026-09-08 by tools/gen_addresses.py
// Runtime address = (uintptr_t)GetModuleHandle(NULL) + RVA (the exe is ASLR-relocated).
#pragma once
#include <cstdint>
@ -369,7 +369,7 @@ constexpr uint32_t ServerPlayer_off_SetupResearchMult = 0x0000022c;
constexpr uint32_t ServerPlayer_off_Sav = 0x00000284;
// offset Tech* current research target (ResT); NULL = none [verified-by-save]
constexpr uint32_t ServerPlayer_off_ResearchTarget = 0x00000294;
// offset std::vector<ObservedTech> otch -- 3 words {_Myfirst@0x274,_Mylast@0x278,_Myend@0x27c}; save tag otch. sizeof(ObservedTech) = 0x2c (44), PINNED three ways: magic divide 0x2e8ba2e9 sar 3 (=/44) in vector_ObservedTech_assign 0x0087239f, imul reg,reg,0x2c at 0x0087243a / 0x007b735b, and the linear search stride `add edi,0x2c` at 0x007ba257. Append site = RecordObservedTech+0xdf (0x007ba27f): lea ecx,[player+0x274]; call vector_ObservedTech_push_back 0x007b7320. All three words move because push_back reallocs through vector_44B_grow 0x007b5820. [verified]
// offset std::vector<ObservedTech> otch (3 words {first,last,end}); save tag otch, element {int otnF, otnL, odet; string otch; int owith}. All three words move on every tech completion (a realloc) -- observed live by both the ProcessResearch and the OnTechResearched player guards. NOT PINNED: the append call site and sizeof(ObservedTech); the byte span the observed_techs region reports is what will measure the stride [verified-by-save]
constexpr uint32_t ServerPlayer_off_ObservedTechs = 0x00000274;
// offset float IncMod [verified-by-save]
constexpr uint32_t ServerPlayer_off_IncMod = 0x0000030c;
@ -799,21 +799,5 @@ constexpr uint32_t Mars_Vec3_NormaliseEpsilon = 0x005e1ef8;
constexpr uint32_t StrategyServer_MoveFleet_straight_leg = 0x003da0f2;
// site site inside StrategyServer::MoveFleet /* the move != distance branch: exactly two roundings per component, tmp.c = float32(dir.c * move) then pos.c = float32(pos.c + tmp.c). `move` is reloaded from a float32 slot. The sibling branch (move == distance, an EXACT float compare) copies the destination's three words verbatim with mov, so an arrival never steps onto its destination. */ [verified]
constexpr uint32_t StrategyServer_MoveFleet_position_update = 0x003da2ac;
// constant sizeof(Game::ObservedTech) = 0x2c (44 bytes). Confirmed independently by (a) the compiler's magic division by 44 (mov eax,0x2e8ba2e9; imul; sar edx,3) at 0x0087239f and 0x007b7339, (b) imul reg,reg,0x2c at 0x0087243a, 0x00872468, 0x007b735b, and (c) the iterator advance `add edi,0x2c` in RecordObservedTech's linear search at 0x007ba257. Matches the on-disk lower bound exactly (4 int + one 0x1c std::string = 44), so there is no padding slack. [verified]
constexpr uint32_t ObservedTech_sizeof = 0x0000002c;
// data Game::ObservedTech vftable. RTTI COL 0x00a81c78 -> type descriptor 0x00aeede4 = '.?AVObservedTech@Game@@'. Written to element+0x00 by ObservedTech_ctor 0x008562a0 -- so ObservedTech is polymorphic and its first word is the vptr, not a data field. [verified]
constexpr uint32_t ObservedTech_vftable = 0x0062439c;
// offset std::string (save tag `otch`, the tech name) at ObservedTech+0x0c, 0x18 bytes, spanning +0x0c..+0x23. MSVC layout relative to the string object: _Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14 -- i.e. ObservedTech+0x1c is the length and +0x20 the capacity. Evidence: ObservedTech_ctor 0x008562a0 writes [elem+0x0c]=0, [elem+0x1c]=0, [elem+0x20]=0xf; RecordObservedTech's search reads [elem+0x1c] as the length and calls compare with this=elem+0x0c; the post-append assign uses lea ecx,[_Mylast-0x20]. NOTE this string is 0x18 bytes, not the 0x1c implied by the ServerPlayer pswd row in struct-recovery.md. UNACCOUNTED in the element: +0x08, +0x24, +0x28 (4 bytes each) plus two 16-bit fields at +0x04/+0x06 written from one source word at 0x007ba2b0 / 0x007ba2c6 -- the mapping of the four on-disk ints (otnF, otnL, odet, owith) onto those slots is NOT determined. [verified]
constexpr uint32_t ObservedTech_off_Name = 0x0000000c;
// thiscall void (this, ServerPlayer* observer, ?, std::string* techName) -- appends to observer->otch. Linear-searches observer->otch (ServerPlayer+0x274) with stride 0x2c comparing each element's name string; if not found, default-constructs an ObservedTech on the stack (0x008562a0) and push_backs it (0x007b7320 @0x007ba288), then writes two 16-bit fields into the new element at +0x04 and +0x06 from param_1+0xc. DIRECT CALLEE of ServerPlayer::OnTechResearched 0x00891790; also called from 0x007be228, 0x007be4e1, 0x007be535. This is the append lane P could not find. [verified]
constexpr uint32_t RecordObservedTech = 0x003ba1a0;
// thiscall void (std::vector<ObservedTech>* this, ObservedTech* value) RET 4. Stride 0x2c. Calls vector_44B_grow 0x007b5820 when _Mylast==_Myend -- the realloc that moves all three vector words. Exactly one caller (RecordObservedTech), so this instantiation is not COMDAT-ambiguous. [verified]
constexpr uint32_t vector_ObservedTech_push_back = 0x003b7320;
// thiscall std::vector<ObservedTech>& operator=(const std::vector<ObservedTech>&) RET 4. Both callers pass ServerPlayer+0x274 (0x00878091 in the settings copy-out, 0x00892507 in the copy-in). [verified]
constexpr uint32_t vector_ObservedTech_assign = 0x00472380;
// thiscall Game::ObservedTech* (ObservedTech* this) -- default ctor; sets vptr 0x00a2439c and empties the name string. [verified]
constexpr uint32_t ObservedTech_ctor = 0x004562a0;
// thiscall vector<T>::_Reserve/grow for a 44-byte element type. Shared with FUN_0086dec0, so it may be a COMDAT-folded body -- do NOT assume it is ObservedTech-specific. [mapped]
constexpr uint32_t vector_44B_grow = 0x003b5820;
} // namespace sots::addr

View file

@ -1,57 +0,0 @@
#!/usr/bin/env python3
"""Cache the full Ghidra function list (address, size, name) to dumps/functions.json.
Ghidra/ReVa is a shared resource -- pull the list once, then work offline.
Usage: uv run python3 tools/cache_functions.py [--refresh]
"""
import json
import os
import subprocess
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
REPO = os.path.dirname(HERE)
OUT = os.path.join(REPO, "dumps", "functions.json")
PROG = "/Sword of the Stars.exe"
def call(tool, args):
r = subprocess.run(
["uv", "run", "python3", os.path.join(HERE, "reva_call.py"), tool, json.dumps(args)],
capture_output=True, text=True, cwd=REPO, timeout=300)
if r.returncode != 0:
sys.exit(f"reva_call {tool} failed: {r.stderr[:400]}")
return r.stdout
def main():
if os.path.exists(OUT) and "--refresh" not in sys.argv:
print(f"{OUT} exists; use --refresh to re-pull")
return
funcs = {}
start = 0
while True:
raw = call("get-functions", {
"programPath": PROG, "startIndex": start, "filterDefaultNames": False})
lines = [ln for ln in raw.splitlines() if ln.strip()]
header = json.loads(lines[0])
for ln in lines[1:]:
try:
f = json.loads(ln)
except json.JSONDecodeError:
continue
funcs[f["address"]] = [f["name"], f.get("sizeInBytes", 0)]
nxt = header.get("nextStartIndex")
total = header.get("totalCount")
print(f" {len(funcs)}/{total}", file=sys.stderr)
if nxt is None or nxt <= start or len(funcs) >= total:
break
start = nxt
os.makedirs(os.path.dirname(OUT), exist_ok=True)
with open(OUT, "w") as fh:
json.dump(funcs, fh)
print(f"wrote {len(funcs)} functions to {OUT}")
if __name__ == "__main__":
main()

View file

@ -1,798 +0,0 @@
#!/usr/bin/env python3
"""x86-32 displacement cross-referencer for the SOTS1 exe.
Why this exists
---------------
Ghidra indexes immediate operands but NOT ModRM displacements. On an MSVC C++
binary `lea reg,[reg+disp]` is *the* idiom for taking the address of a member --
it is how every std::vector / std::string / embedded sub-object is passed to a
method or ctor. `find-constant-uses 0x274` therefore cannot see the append site
for a vector member at +0x274. This tool answers the question Ghidra can't:
"what code takes the address of, or accesses, offset N off some object?"
How it works
------------
x86 is variable-length and not self-synchronising, so a naive byte scan for
"8D 8E <disp32>" produces confident garbage. Instead we run a full instruction
*length* decoder (prefixes / 1-,2-,3-byte opcodes / ModRM / SIB / disp / imm)
seeded from Ghidra's 41k function starts, so every decode begins on a real
instruction boundary. `--brute` runs the naive scan too, purely so the
false-positive rate of the naive method can be *measured* rather than guessed.
Correctness notes that matter (each of these is a silent-garbage source):
* mod=0,rm=5 -> disp32 with NO base: an absolute global address, not a
member offset. Excluded from member queries by default.
* mod=0,rm=4,
sib.base=5 -> disp32, no base, index only. Same: excluded.
* mod=1 -> disp8 SIGN-EXTENDED. -0x08 must not be reported as 0xf8.
* mod=3 -> register operand, no memory, no displacement at all.
* 0x67 addr-size -> 16-bit ModRM, entirely different layout. Skipped, flagged.
Usage
-----
uv run python3 tools/x86disp.py build # decode + cache index
uv run python3 tools/x86disp.py query 0x274 # who touches +0x274
uv run python3 tools/x86disp.py query 0x29c --lea # lea only
uv run python3 tools/x86disp.py func 0x0080db00 # dump one function
uv run python3 tools/x86disp.py stats
"""
import bisect
import json
import os
import struct
import sys
HERE = os.path.dirname(os.path.abspath(__file__))
REPO = os.path.dirname(HERE)
EXE = os.path.join(REPO, "dumps", "sots.exe")
FUNCS = os.path.join(REPO, "dumps", "functions.json")
INDEX = os.path.join(REPO, "dumps", "dispindex.json")
R32 = ["eax", "ecx", "edx", "ebx", "esp", "ebp", "esi", "edi"]
# ---------------------------------------------------------------- opcode maps
# value: (has_modrm, imm_kind)
# imm kinds: 0 none | 'b' 1 | 'w' 2 | 'd' 4 | 'z' 2-if-66-else-4
# 'p' far ptr 6 (4 with 66) | 'a' moffs (addr-size) | 'e' enter 3
# 'g6' F6 group (b if reg<2) | 'g7' F7 group (z if reg<2)
_ONE = {}
def _fill(rng, modrm, imm):
for o in rng:
_ONE[o] = (modrm, imm)
# 00..3F: the eight ALU ops, each /r /r /r /r AL,ib eAX,iz + 2 seg ops
for _base in (0x00, 0x08, 0x10, 0x18, 0x20, 0x28, 0x30, 0x38):
_fill(range(_base, _base + 4), True, 0)
_ONE[_base + 4] = (False, "b")
_ONE[_base + 5] = (False, "z")
_ONE[_base + 6] = (False, 0) # PUSH seg / prefix-or-ascii-adjust
_ONE[_base + 7] = (False, 0)
_fill(range(0x40, 0x60), False, 0) # INC/DEC/PUSH/POP r32
_fill([0x60, 0x61], False, 0) # PUSHA/POPA
_ONE[0x62] = (True, 0) # BOUND
_ONE[0x63] = (True, 0) # ARPL
_ONE[0x68] = (False, "z") # PUSH iz
_ONE[0x69] = (True, "z") # IMUL r,Ev,iz
_ONE[0x6A] = (False, "b") # PUSH ib
_ONE[0x6B] = (True, "b") # IMUL r,Ev,ib
_fill(range(0x6C, 0x70), False, 0) # INS/OUTS
_fill(range(0x70, 0x80), False, "b") # Jcc rel8
_ONE[0x80] = (True, "b")
_ONE[0x81] = (True, "z")
_ONE[0x82] = (True, "b")
_ONE[0x83] = (True, "b")
_fill(range(0x84, 0x90), True, 0) # TEST/XCHG/MOV/MOV-seg/LEA/POP Ev
_fill(range(0x90, 0x9A), False, 0) # NOP/XCHG/CWDE/CDQ
_ONE[0x9A] = (False, "p") # CALLF
_fill(range(0x9B, 0xA0), False, 0)
_fill(range(0xA0, 0xA4), False, "a") # MOV moffs
_fill(range(0xA4, 0xA8), False, 0) # MOVS/CMPS
_ONE[0xA8] = (False, "b")
_ONE[0xA9] = (False, "z")
_fill(range(0xAA, 0xB0), False, 0) # STOS/LODS/SCAS
_fill(range(0xB0, 0xB8), False, "b") # MOV r8,ib
_fill(range(0xB8, 0xC0), False, "z") # MOV r32,iz
_ONE[0xC0] = (True, "b")
_ONE[0xC1] = (True, "b")
_ONE[0xC2] = (False, "w") # RET imm16
_ONE[0xC3] = (False, 0)
_ONE[0xC4] = (True, 0) # LES
_ONE[0xC5] = (True, 0) # LDS
_ONE[0xC6] = (True, "b") # MOV Eb,Ib
_ONE[0xC7] = (True, "z") # MOV Ev,Iz
_ONE[0xC8] = (False, "e") # ENTER iw,ib
_ONE[0xC9] = (False, 0)
_ONE[0xCA] = (False, "w") # RETF imm16
_fill([0xCB, 0xCC], False, 0)
_ONE[0xCD] = (False, "b") # INT ib
_fill([0xCE, 0xCF], False, 0)
_fill(range(0xD0, 0xD4), True, 0) # shift group by 1 / by CL
_ONE[0xD4] = (False, "b") # AAM
_ONE[0xD5] = (False, "b") # AAD
_fill([0xD6, 0xD7], False, 0)
_fill(range(0xD8, 0xE0), True, 0) # x87 -- always ModRM
_fill(range(0xE0, 0xE4), False, "b") # LOOP*/JECXZ
_fill(range(0xE4, 0xE8), False, "b") # IN/OUT ib
_ONE[0xE8] = (False, "z") # CALL rel32
_ONE[0xE9] = (False, "z") # JMP rel32
_ONE[0xEA] = (False, "p") # JMPF
_ONE[0xEB] = (False, "b") # JMP rel8
_fill(range(0xEC, 0xF0), False, 0) # IN/OUT DX
_fill(range(0xF0, 0xF6), False, 0) # LOCK/INT1/REP*/HLT/CMC
_ONE[0xF6] = (True, "g6")
_ONE[0xF7] = (True, "g7")
_fill(range(0xF8, 0xFE), False, 0)
_ONE[0xFE] = (True, 0)
_ONE[0xFF] = (True, 0)
_TWO = {}
def _fill2(rng, modrm, imm):
for o in rng:
_TWO[o] = (modrm, imm)
_fill2(range(0x00, 0x05), True, 0)
_fill2(range(0x05, 0x0D), False, 0)
_TWO[0x0D] = (True, 0)
_TWO[0x0E] = (False, 0)
_TWO[0x0F] = (True, "b") # 3DNow!
_fill2(range(0x10, 0x18), True, 0)
_fill2(range(0x18, 0x20), True, 0) # hint-NOP / prefetch
_fill2(range(0x20, 0x25), True, 0)
_fill2(range(0x28, 0x30), True, 0)
_fill2(range(0x30, 0x38), False, 0)
_fill2(range(0x40, 0x50), True, 0) # CMOVcc
_fill2(range(0x50, 0x70), True, 0) # SSE/MMX
_TWO[0x70] = (True, "b")
_fill2(range(0x71, 0x74), True, "b")
_fill2(range(0x74, 0x77), True, 0)
_TWO[0x77] = (False, 0) # EMMS
_fill2(range(0x78, 0x80), True, 0)
_fill2(range(0x80, 0x90), False, "z") # Jcc rel32
_fill2(range(0x90, 0xA0), True, 0) # SETcc
_fill2([0xA0, 0xA1, 0xA2], False, 0)
_TWO[0xA3] = (True, 0) # BT
_TWO[0xA4] = (True, "b") # SHLD ib
_TWO[0xA5] = (True, 0) # SHLD CL
_fill2([0xA8, 0xA9, 0xAA], False, 0)
_TWO[0xAB] = (True, 0) # BTS
_TWO[0xAC] = (True, "b") # SHRD ib
_fill2([0xAD, 0xAE, 0xAF], True, 0)
_fill2(range(0xB0, 0xBA), True, 0)
_TWO[0xBA] = (True, "b") # group8 BT/BTS/BTR/BTC ib
_fill2(range(0xBB, 0xC0), True, 0)
_fill2([0xC0, 0xC1], True, 0) # XADD
_TWO[0xC2] = (True, "b") # CMPPS
_TWO[0xC3] = (True, 0) # MOVNTI
_fill2([0xC4, 0xC5, 0xC6], True, "b")
_TWO[0xC7] = (True, 0) # group9 CMPXCHG8B
_fill2(range(0xC8, 0xD0), False, 0) # BSWAP
_fill2(range(0xD0, 0x100), True, 0) # MMX/SSE bulk
# minimal mnemonics -- enough to read a report, not a full disassembler
_ALU = ["add", "or", "adc", "sbb", "and", "sub", "xor", "cmp"]
_G1 = _ALU
_G5 = ["inc", "dec", "call", "callf", "jmp", "jmpf", "push", "?"]
_G3 = ["test", "test", "not", "neg", "mul", "imul", "div", "idiv"]
_SHIFT = ["rol", "ror", "rcl", "rcr", "shl", "shr", "shl", "sar"]
def mnemonic(op2, op, reg):
"""Best-effort mnemonic. op2 True => the opcode was 0F-escaped."""
if op2:
if 0x10 <= op <= 0x17 or 0x28 <= op <= 0x2F or 0x51 <= op <= 0x5F:
return "sse"
if 0x40 <= op <= 0x4F:
return "cmov"
if 0x90 <= op <= 0x9F:
return "setcc"
if op in (0xB6, 0xB7):
return "movzx"
if op in (0xBE, 0xBF):
return "movsx"
if op == 0xAF:
return "imul"
if op in (0x6E, 0x6F, 0x7E, 0x7F, 0xD6):
return "movq/movd"
return f"0f{op:02x}"
if op < 0x40 and (op & 7) < 6:
return _ALU[op >> 3]
if op in (0x88, 0x89, 0x8A, 0x8B, 0xC6, 0xC7):
return "mov"
if op == 0x8D:
return "lea"
if op in (0x84, 0x85):
return "test"
if op in (0x86, 0x87):
return "xchg"
if op in (0x80, 0x81, 0x82, 0x83):
return _G1[reg]
if op in (0xC0, 0xC1, 0xD0, 0xD1, 0xD2, 0xD3):
return _SHIFT[reg]
if op in (0xF6, 0xF7):
return _G3[reg]
if op == 0xFF:
return _G5[reg]
if op == 0xFE:
return ["inc", "dec"][reg] if reg < 2 else "?"
if op == 0x8F:
return "pop"
if op in (0x69, 0x6B):
return "imul"
if 0xD8 <= op <= 0xDF:
return "x87"
if op == 0x62:
return "bound"
return f"{op:02x}"
class Desync(Exception):
pass
def decode(buf, i, end):
"""Decode one instruction at buf[i]. Returns (length, info|None).
info = dict(mnem, base, index, scale, disp, dispsize, modrm_reg, is_lea)
for instructions with a memory operand carrying a displacement; None
otherwise. Raises Desync on an unknown/invalid opcode.
"""
start = i
opsize66 = False
addr67 = False
while i < end:
b = buf[i]
if b == 0x66:
opsize66 = True
i += 1
elif b == 0x67:
addr67 = True
i += 1
elif b in (0xF0, 0xF2, 0xF3, 0x2E, 0x36, 0x3E, 0x26, 0x64, 0x65):
i += 1
else:
break
else:
raise Desync("prefix run to end")
if i - start > 14:
raise Desync("prefix flood")
op = buf[i]
i += 1
op2 = op3 = False
if op == 0x0F:
if i >= end:
raise Desync("truncated 0f")
op = buf[i]
i += 1
op2 = True
if op in (0x38, 0x3A):
three_imm = "b" if op == 0x3A else 0
if i >= end:
raise Desync("truncated 0f3x")
op = buf[i]
i += 1
op3 = True
has_modrm, imm = True, three_imm
else:
ent = _TWO.get(op)
if ent is None:
raise Desync(f"unknown 0f{op:02x}")
has_modrm, imm = ent
else:
ent = _ONE.get(op)
if ent is None:
raise Desync(f"unknown {op:02x}")
has_modrm, imm = ent
info = None
if has_modrm:
if i >= end:
raise Desync("truncated modrm")
modrm = buf[i]
i += 1
mod = modrm >> 6
reg = (modrm >> 3) & 7
rm = modrm & 7
if imm == "g6":
imm = "b" if reg < 2 else 0
elif imm == "g7":
imm = "z" if reg < 2 else 0
if mod != 3:
if addr67:
# 16-bit ModRM: different table entirely. Rare in MSVC code and
# never the member-address idiom -- skip rather than mis-decode.
raise Desync("16-bit addressing (0x67)")
base = index = None
scale = 1
if rm == 4:
if i >= end:
raise Desync("truncated sib")
sib = buf[i]
i += 1
scale = 1 << (sib >> 6)
idx = (sib >> 3) & 7
bse = sib & 7
index = None if idx == 4 else R32[idx]
if bse == 5 and mod == 0:
base = None # disp32 absolute + index
else:
base = R32[bse]
elif rm == 5 and mod == 0:
base = None # disp32 absolute
else:
base = R32[rm]
disp = 0
dispsize = 0
if mod == 1:
if i >= end:
raise Desync("truncated disp8")
disp = struct.unpack_from("<b", buf, i)[0] # SIGN-EXTENDED
dispsize = 1
i += 1
elif mod == 2 or base is None:
if i + 4 > end:
raise Desync("truncated disp32")
disp = struct.unpack_from("<i", buf, i)[0]
dispsize = 4
i += 4
if dispsize:
info = {
"mnem": mnemonic(op2, op, reg), "base": base,
"index": index, "scale": scale, "disp": disp,
"dispsize": dispsize, "reg": R32[reg],
"lea": (not op2 and op == 0x8D),
}
else:
reg = 0
n = 0
if imm == "b":
n = 1
elif imm == "w":
n = 2
elif imm == "d":
n = 4
elif imm == "z":
n = 2 if opsize66 else 4
elif imm == "p":
n = 4 if opsize66 else 6
elif imm == "a":
n = 2 if addr67 else 4
elif imm == "e":
n = 3
i += n
if i > end:
raise Desync("truncated imm")
if i == start:
raise Desync("zero length")
return i - start, info
# ------------------------------------------------------------------- PE / IO
def load_pe(path):
"""Return (image_base, [(va_start, va_end, bytes, name)]) for exec sections."""
data = open(path, "rb").read()
pe = struct.unpack_from("<I", data, 0x3C)[0]
assert data[pe:pe + 4] == b"PE\0\0", "not a PE"
nsec = struct.unpack_from("<H", data, pe + 6)[0]
optsz = struct.unpack_from("<H", data, pe + 20)[0]
base = struct.unpack_from("<I", data, pe + 24 + 28)[0]
secs = []
off = pe + 24 + optsz
for k in range(nsec):
s = off + k * 40
name = data[s:s + 8].rstrip(b"\0").decode("latin1")
vsize, vaddr, rsize, raddr = struct.unpack_from("<IIII", data, s + 8)
chars = struct.unpack_from("<I", data, s + 36)[0]
if not (chars & 0x20000000): # IMAGE_SCN_MEM_EXECUTE
continue
n = min(vsize, rsize) if vsize else rsize
secs.append((base + vaddr, base + vaddr + n, data[raddr:raddr + n], name))
return base, secs
def load_funcs():
with open(FUNCS) as fh:
raw = json.load(fh)
fl = sorted((int(a, 16), n, sz) for a, (n, sz) in raw.items())
return fl
# --------------------------------------------------------------------- build
def build():
_, secs = load_pe(EXE)
funcs = load_funcs()
starts = [f[0] for f in funcs]
sites = [] # [va, disp, base, index, scale, mnem, lea, funcidx, hex]
covered = 0
total_code = sum(e - s for s, e, _, _ in secs)
desyncs = 0
decoded_ins = 0
for fi, (fva, fname, fsz) in enumerate(funcs):
sec = next((s for s in secs if s[0] <= fva < s[1]), None)
if sec is None or fsz <= 0:
continue
sva, eva, buf, _ = sec
# Sweep to the NEXT function start, not fva+fsz: Ghidra's sizeInBytes
# understates ~11% of bodies (it clips valid epilogues mid-instruction).
# Sites past fsz are still attributed to this function but flagged, so
# a caller can tell body-proper from possible inter-function padding.
j = bisect.bisect_right(starts, fva)
limit = min(starts[j] if j < len(starts) else eva, eva)
body_end = fva + fsz
i = fva - sva
end = limit - sva
while i < end:
try:
ln, info = decode(buf, i, end)
except Desync:
desyncs += 1
break
decoded_ins += 1
if info and info["base"] is not None and info["disp"] != 0:
va = sva + i
sites.append([va, info["disp"], info["base"], info["index"],
info["scale"], info["mnem"], info["lea"], fi,
buf[i:i + min(ln, 10)].hex(),
1 if va < body_end else 0, info["reg"]])
i += ln
covered += max(0, i - (fva - sva))
idx = {}
for k, s in enumerate(sites):
idx.setdefault(str(s[1]), []).append(k)
out = {
"sites": sites,
"byDisp": idx,
"funcs": [[f[0], f[1], f[2]] for f in funcs],
"stats": {"functions": len(funcs), "instructions": decoded_ins,
"sites": len(sites), "desyncs": desyncs,
"bytesCovered": covered, "codeBytes": total_code},
}
with open(INDEX, "w") as fh:
json.dump(out, fh)
st = out["stats"]
print(f"functions swept : {st['functions']}")
print(f"instructions : {st['instructions']}")
print(f"disp sites : {st['sites']}")
print(f"desyncs : {st['desyncs']} "
f"({100.0 * st['desyncs'] / st['functions']:.2f}% of functions)")
print(f"code coverage : {st['bytesCovered']}/{st['codeBytes']} "
f"({100.0 * st['bytesCovered'] / st['codeBytes']:.1f}%)")
# --------------------------------------------------------------------- query
def load_index():
if not os.path.exists(INDEX):
sys.exit("no index; run: uv run python3 tools/x86disp.py build")
with open(INDEX) as fh:
return json.load(fh)
THIS_REGS = ("ecx", "esi", "edi", "ebx") # typical `this` carriers in MSVC
def fmt(site, funcs):
va, disp, base, index, scale, mnem, lea, fi, hx, inbody, dst = site
fva, fname, _ = funcs[fi]
ea = f"[{base}"
if index:
ea += f"+{index}*{scale}"
ea += f"{'+' if disp >= 0 else '-'}0x{abs(disp):x}]"
txt = f"{mnem} {dst},{ea}" if lea else f"{mnem} {ea}"
return (f" 0x{va:08x} {txt:<32} "
f"{fname}+0x{va - fva:x}{'' if inbody else ' [past-body]'} ({hx})")
def query(argv):
want = int(argv[0], 0)
only_lea = "--lea" in argv
only_this = "--this" in argv
show_all = "--all" in argv
d = load_index()
funcs = d["funcs"]
keys = d["byDisp"].get(str(want), [])
hits = [d["sites"][k] for k in keys]
if only_lea:
hits = [h for h in hits if h[6]]
if only_this:
hits = [h for h in hits if h[2] in THIS_REGS]
hits.sort(key=lambda h: h[0])
ranked, other = [], []
for h in hits:
(ranked if h[2] in THIS_REGS else other).append(h)
print(f"displacement 0x{want:x} ({want}): {len(hits)} site(s)")
print(f"\n== base is a likely `this` ({'/'.join(THIS_REGS)}): {len(ranked)} ==")
byfn = {}
for h in ranked:
byfn.setdefault(h[7], []).append(h)
for fi in sorted(byfn, key=lambda f: funcs[f][0]):
print(f" {funcs[fi][1]} @0x{funcs[fi][0]:08x}")
for h in byfn[fi]:
print(fmt(h, funcs))
if other:
print(f"\n== other base regs (esp/ebp = locals, eax/edx = temps): {len(other)} ==")
if show_all:
for h in other:
print(fmt(h, funcs))
else:
cnt = {}
for h in other:
cnt[h[2]] = cnt.get(h[2], 0) + 1
print(" " + ", ".join(f"{k}:{v}" for k, v in sorted(cnt.items()))
+ " (--all to list)")
def func_dump(argv):
fva = int(argv[0], 0)
d = load_index()
funcs = d["funcs"]
fi = next((i for i, f in enumerate(funcs) if f[0] == fva), None)
if fi is None:
sys.exit(f"no function at 0x{fva:08x}")
print(f"{funcs[fi][1]} @ 0x{fva:08x} size {funcs[fi][2]}")
for s in d["sites"]:
if s[7] == fi:
print(fmt(s, funcs))
def stats():
d = load_index()
st = d["stats"]
for k, v in st.items():
print(f"{k:16}: {v}")
hist = {}
for s in d["sites"]:
if s[2] in THIS_REGS:
hist[s[1]] = hist.get(s[1], 0) + 1
print(f"\ndistinct displacements off ecx/esi/edi/ebx: {len(hist)}")
top = sorted(hist.items(), key=lambda kv: -kv[1])[:15]
print("most common: " + ", ".join(f"0x{k:x}({v})" for k, v in top))
_STORE = {0x89, 0x01, 0x29, 0x39, 0x09, 0x21, 0x31, 0x19, 0x11,
0x88, 0x00, 0x08, 0x10, 0x18, 0x20, 0x28, 0x30, 0x38}
def _reg_form(raw):
"""Render the mod=3 (register-to-register) forms, which are exactly the
ones that carry the arithmetic you care about when recovering a struct
stride: `imul ecx,ecx,0x2c`, `sar edx,3`, `sub ecx,edi`."""
p = 0
while p < len(raw) and raw[p] in (0x66, 0x67, 0xF0, 0xF2, 0xF3,
0x2E, 0x36, 0x3E, 0x26, 0x64, 0x65):
p += 1
op = raw[p]
if op == 0x0F or p + 1 >= len(raw):
return mnemonic(op == 0x0F, raw[p + 1] if op == 0x0F else op, 0)
m = raw[p + 1]
mod, reg, rm = m >> 6, (m >> 3) & 7, m & 7
if mod != 3:
# memory operand: render it and show which register is the other half,
# so a load reads `mov ecx,[edi+4]` not a bare `mov`.
q = p + 2
ea = "["
if rm == 4:
sib = raw[q]; q += 1
bse, ix = sib & 7, (sib >> 3) & 7
ea += ("" if (bse == 5 and mod == 0) else R32[bse])
if ix != 4:
ea += f"+{R32[ix]}*{1 << (sib >> 6)}"
elif not (rm == 5 and mod == 0):
ea += R32[rm]
if mod == 1 and q < len(raw):
dv = struct.unpack_from("<b", raw, q)[0]
ea += f"{'+' if dv >= 0 else '-'}0x{abs(dv):x}"
elif mod == 2 and q + 4 <= len(raw):
dv = struct.unpack_from("<i", raw, q)[0]
ea += f"{'+' if dv >= 0 else '-'}0x{abs(dv):x}"
ea += "]"
nm = mnemonic(False, op, reg)
if op in _STORE:
return f"{nm} {ea},{R32[reg]}"
if op in (0x80, 0x81, 0x83, 0xC6, 0xC7, 0xF6, 0xF7, 0xFF, 0xFE,
0xC0, 0xC1, 0xD0, 0xD1, 0xD2, 0xD3) or op >= 0xD8:
return f"{nm} {ea}"
return f"{nm} {R32[reg]},{ea}"
r, x = R32[reg], R32[rm]
if op in (0x89, 0x01, 0x29, 0x39, 0x09, 0x21, 0x31, 0x19, 0x11, 0x85, 0x87):
a, b = x, r # Ev,Gv -> dst is rm
else:
a, b = r, x # Gv,Ev -> dst is reg
nm = mnemonic(False, op, reg)
if op in (0xC0, 0xC1): # shift group, imm8
return f"{_SHIFT[reg]} {x},{raw[p + 2]}"
if op in (0xD1, 0xD3):
return f"{_SHIFT[reg]} {x},{'cl' if op == 0xD3 else '1'}"
if op == 0x6B: # imul r,rm,imm8 <- the stride!
return f"imul {r},{x},0x{struct.unpack_from('<b', raw, p + 2)[0] & 0xff:x}"
if op == 0x69:
return f"imul {r},{x},0x{struct.unpack_from('<I', raw, p + 2)[0]:x}"
if op in (0x83, 0x81, 0x80):
v = (raw[p + 2] if op != 0x81
else struct.unpack_from("<I", raw, p + 2)[0])
return f"{_G1[reg]} {x},0x{v:x}"
if op in (0xF7, 0xF6):
return f"{_G3[reg]} {x}"
if op == 0xFF:
return f"{_G5[reg]} {x}"
return f"{nm} {a},{b}"
def dis(argv):
"""Linear disassembly window: `dis <va> [count]`. Length-accurate; the
mnemonics are minimal but call/jmp targets are resolved to function names,
which is what you actually need to follow a `lea this,[obj+N]; call` pair."""
va = int(argv[0], 0)
n = int(argv[1]) if len(argv) > 1 else 24
_, secs = load_pe(EXE)
d = load_index()
fmap = {f[0]: f[1] for f in d["funcs"]}
sec = next((s for s in secs if s[0] <= va < s[1]), None)
if sec is None:
sys.exit("address not in an executable section")
sva, eva, buf, _ = sec
i = va - sva
for _ in range(n):
try:
ln, info = decode(buf, i, len(buf))
except Desync as e:
print(f"0x{sva + i:08x} <desync: {e}>")
return
raw = buf[i:i + ln]
cur = sva + i
txt = ""
# resolve the two rel32 forms and the rel8 jumps by hand
if raw[0] in (0xE8, 0xE9) and ln == 5:
tgt = cur + 5 + struct.unpack_from("<i", raw, 1)[0]
txt = (f"{'call' if raw[0] == 0xE8 else 'jmp'} 0x{tgt:08x}"
f" {fmap.get(tgt, '')}")
elif raw[0] == 0x0F and 0x80 <= raw[1] <= 0x8F and ln == 6:
tgt = cur + 6 + struct.unpack_from("<i", raw, 2)[0]
txt = f"jcc 0x{tgt:08x}"
elif raw[0] == 0xEB or 0x70 <= raw[0] <= 0x7F:
tgt = cur + ln + struct.unpack_from("<b", raw, ln - 1)[0]
txt = f"jmp/jcc 0x{tgt:08x}"
elif 0x50 <= raw[0] <= 0x57:
txt = f"push {R32[raw[0] - 0x50]}"
elif 0x58 <= raw[0] <= 0x5F:
txt = f"pop {R32[raw[0] - 0x58]}"
elif raw[0] == 0x6A:
txt = f"push 0x{raw[1]:x}"
elif raw[0] == 0x68:
txt = f"push 0x{struct.unpack_from('<I', raw, 1)[0]:x}"
elif 0xB8 <= raw[0] <= 0xBF and ln == 5:
txt = f"mov {R32[raw[0] - 0xB8]},0x{struct.unpack_from('<I', raw, 1)[0]:x}"
elif raw[0] == 0xC3:
txt = "ret"
elif raw[0] == 0xC2:
txt = f"ret 0x{struct.unpack_from('<H', raw, 1)[0]:x}"
else:
txt = _reg_form(raw)
lbl = fmap.get(cur, "")
print(f"0x{cur:08x} {raw.hex():<18} {txt:<44} {lbl}")
i += ln
def cohort(argv):
"""Rank functions by how many offsets of a KNOWN class layout they touch.
This is the answer to the real precision problem. A bare query for 0x274
returns ~99 sites and the tool cannot know which base register holds a
ServerPlayer. But a function that touches 0x274 *and* 0x29c *and* 0x244
*and* 0x254 is not doing that by coincidence -- those are ServerPlayer's
members. Feed the offsets we have already pinned and the class's own
methods float to the top.
uv run python3 tools/x86disp.py cohort 0x274 --known=0x29c,0x244,0x254,...
WARNING, learned the hard way on the ObservedTech hunt: use this as a
RANKER, never as a hard filter. It finds fat *class methods* (the
ServerPlayer serializer scores 50/50) but it actively hides narrow helpers.
`RecordObservedTech` -- the actual append site -- touches only 0x274 and
0x278 and nothing else on ServerPlayer, so every --min>=1 setting drops it.
Read the full `query` output before trusting a cohort shortlist.
"""
want = int(argv[0], 0)
known = []
minhits = 2
for a in argv[1:]:
if a.startswith("--known="):
known = [int(x, 0) for x in a[8:].split(",")]
elif a.startswith("--min="):
minhits = int(a[6:])
if not known:
sys.exit("need --known=<comma-separated offsets of the same class>")
d = load_index()
funcs = d["funcs"]
kset = set(known)
touched = {} # funcidx -> set(offset)
target = {} # funcidx -> [sites at `want`]
for s in d["sites"]:
if s[2] not in THIS_REGS:
continue
if s[1] in kset:
touched.setdefault(s[7], set()).add(s[1])
if s[1] == want:
target.setdefault(s[7], []).append(s)
rows = []
for fi, sites in target.items():
hits = touched.get(fi, set()) - {want}
if len(hits) >= minhits:
rows.append((len(hits), fi, sites, hits))
rows.sort(key=lambda r: -r[0])
print(f"functions touching 0x{want:x} AND >={minhits} other known offsets "
f"of this class: {len(rows)} of {len(target)} candidates "
f"({100.0 * len(rows) / max(1, len(target)):.0f}% kept)")
for n, fi, sites, hits in rows:
fva, fname, _ = funcs[fi]
hx = ",".join(f"0x{h:x}" for h in sorted(hits))
print(f"\n {fname} @0x{fva:08x} [{n} co-hits: {hx}]")
for s in sites:
print(fmt(s, funcs))
# ------------------------------------------------- naive scan, for FP measure
def brute(argv):
"""Naive 'search the raw bytes' scan -- the thing you'd write without a
decoder. Only exists so we can put a NUMBER on how wrong it is.
`--ops 8d` (default) = lea only. `--ops 8d,89,8b,01,03,39,3b` = the wider
scan you would actually need, since a member is read/written far more often
than its address is taken. The wider the opcode set and the smaller the
displacement, the worse the naive method gets -- that is the point.
"""
want = int(argv[0], 0)
ops = {0x8D}
for a in argv[1:]:
if a.startswith("--ops="):
ops = {int(x, 16) for x in a[6:].split(",")}
_, secs = load_pe(EXE)
d = load_index()
real = {s[0] for s in d["sites"] if s[1] == want}
found = []
d8 = want if -128 <= want <= 127 else None
pat32 = struct.pack("<i", want)
for sva, eva, buf, _ in secs:
for i in range(len(buf) - 6):
if buf[i] not in ops:
continue
m = buf[i + 1]
mod, rm = m >> 6, m & 7
if mod == 3 or mod == 0:
continue
k = i + 2 + (1 if rm == 4 else 0)
if mod == 1 and d8 is not None:
if struct.unpack_from("<b", buf, k)[0] == want:
found.append(sva + i)
elif mod == 2 and buf[k:k + 4] == pat32:
found.append(sva + i)
fp = [a for a in found if a not in real]
tag = ",".join(f"{o:02x}" for o in sorted(ops))
print(f"naive byte-scan (opcodes {tag}) for disp 0x{want:x}: "
f"{len(found)} candidate(s)")
print(f" real instructions at that address : {len(found) - len(fp)}")
print(f" NOT an instruction boundary : {len(fp)} "
f"({100.0 * len(fp) / max(1, len(found)):.1f}% false positive)")
missed = len(real) - (len(found) - len(fp))
print(f" real sites the naive scan MISSED : {missed} of {len(real)}")
if __name__ == "__main__":
if len(sys.argv) < 2:
sys.exit(__doc__)
cmd, rest = sys.argv[1], sys.argv[2:]
{"build": lambda a: build(), "query": query, "func": func_dump,
"stats": lambda a: stats(), "brute": brute, "cohort": cohort,
"dis": dis}[cmd](rest)