lane X: x86 displacement xref scanner; pin sizeof(ObservedTech) and its append site
Ghidra does not index ModRM displacements, so `lea reg,[reg+disp]` -- the MSVC idiom for taking a member's address -- is invisible to find-constant-uses. That blind spot parked ServerPlayer+0x274 and covers every non-trivial member of the ~1,600 classes still to map. tools/x86disp.py: full x86-32 length decoder (prefixes, 1/2/3-byte opcodes, ModRM, SIB, sign-extended disp8, disp32, every immediate form) swept from Ghidra's 41,089 function starts so decodes begin on real instruction boundaries. 2,174,504 instructions, 612,166 displacement sites, 100.0% code coverage, 70 desyncs (0.17%), zero unknown opcodes. Excludes no-base disp32 forms (mod=0/rm=5, sib.base=5) which are absolute globals, not member offsets. Commands: build/query/cohort/func/dis/stats/brute. Works off a gitignored local cache in dumps/ rather than hammering CT111. Validated before use: re-finds lea eax,[ecx+0x29c] in ServerPlayer::GetEventStorage (0x0080db00) and both known OnTechResearched +0x29c sites, plus a new one in ProcessTurn. Positive control: the ServerPlayer serializer scores 50/50 known offsets. sizeof(Game::ObservedTech) = 0x2c (44), proven three ways: the exact magic divide 0x2e8ba2e9 sar 3 at 0x0087239f, imul reg,reg,0x2c at 0x0087243a and 0x007b735b, and the search stride add edi,0x2c at 0x007ba257. Append site: RecordObservedTech+0xdf (0x007ba27f) -- lea ecx,[player+0x274]; call vector_ObservedTech_push_back 0x007b7320 RecordObservedTech (0x007ba1a0) is a direct callee of OnTechResearched and de-duplicates by tech name before appending. The realloc through 0x007b5820 is why lane R's guard saw all three vector words move. Element carries a vptr (RTTI .?AVObservedTech@Game@@) at +0 and a 0x18-byte std::string at +0x0c; the four on-disk ints map onto +0x04/+0x06/+0x08/+0x24/+0x28 in an order this read does NOT determine, and is not guessed. Also corrects harness-audit row 11: ComputeBudget has no store to Budget+0x64 (its only +0x64 accesses are loads off a different base), and ProcessResearch's int* overbudget arg is a ProcessTurn stack local, not Budget+0x64. Agrees with lane R's guard seeing 0 changes in 4284 calls. Honest limits are recorded in the note and the board: this is a recall tool, not an oracle. Class-level precision at 0x274 is ~13% by function, i.e. a ~900x search-space cut that still needs one call-graph check. Cohort ranking must not be used as a hard filter -- it would have discarded the correct answer here. Ghidra writeback: labels + plate comments on RecordObservedTech, vector_ObservedTech_push_back, ObservedTech_ctor, vector_ObservedTech_assign, vector_44B_grow, vftable_ObservedTech.
This commit is contained in:
alex
parent
1a58bcdf90
commit
460cb7ca2b
8 changed files with 1235 additions and 8 deletions
|
|
@ -68,7 +68,7 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`).
|
|||
| RollResearchEvent draw (behavioural) | verify | verified | high | 100% | 2026-09-08 | **CLOSED by lane R.** On turn 6 (`IND_TRKSTL`, tech 10094) `research_roll_pending` was true going in; the original drew exactly one word (rng `left` 375->374, `next_index` 249->250) and cleared the flag, and `ours` reproduced both bit-for-bit on the scratch generator. 0 divergences on that call. Needs 5 End Turns from ref-turn2 to reach - the reference turn has no completion at all |
|
||||
| golden-trace recapture (post-guards) | verify | verified | high | 100% | 2026-09-08 | DONE on the live game, build `recap-7584bad-20260908T0615Z` (NO source change needed - the audit's machinery did all of it). **B3 ProcessResearch: the defect is VISIBLE** - `side.events.after.v.next_id orig=4 ours=3`, one divergent call of 3 on the reference turn and its ONLY divergent field; bit-for-bit the `EvNxID 4->3` that previously needed a 609 KB save diff. Over 5 turns 15 calls / 3 diverged, **RNG matched 15/15**, and the two completion calls miss TWO event ids each. **B1 ComputeBudget: verdict held** - 4284 compared, 0 diverged, exit 0, `budget_object` guard caught 0 undeclared writes (Budget+0x64 never changed value). **MoveFleet: 8 of 45 diverge by 1 ULP of position** (new; see its own row). First guarded captures for OnTechResearched (2 calls), ServerSystem::ProcessTurn (140 calls) and MoveFleet (45). Guards mapped SetResearched live (ConMod[0..2], OutMod, PopMod, ResTNm, TechTree+0x20 order counter) and found an UNDECLARED `vector<ObservedTech> otch` append at player+0x274. Oracle held on every run's first End Turn. Report `findings/subsystems/golden-trace-recapture.md`; engine `docs/R-recapture.md`; traces `verify/traces/recap-*`, reports `verify/results/compare/recap-*`. sots-engine branch `wip/recapture` e50d5e5 (merged with main 82ef52f; ctest 32/32, clean-room OK) |
|
||||
| MoveFleet position rounding (1 ULP) | verify | verified | high | 100% | 2026-09-08 | **CLOSED by lane M.** Mechanism read off the instruction stream, not fitted: the engine's `Mars_Vec3_Normalize` (0x00422520, 123 callers) narrows to float32 FOUR times - `sumsq = f32(x*x+y*y+z*z)` (products/adds stay in 53-bit regs, only the SUM is stored), `len = f32(sqrt(sumsq))`, `inv = f32(1.0/len)` a RECIPROCAL that is MULTIPLIED through rather than three divides, and `dir.c = f32(delta.c*inv)`; and MoveFleet stores each `dest.c - pos.c` BACK TO A FLOAT32 SLOT before calling it, and takes the leg distance from that same call's return value. `ours` did all of it in double. The position tail (`f32(pos + f32(dir*move))`) was already right, which is exactly why the error was a constant ABSOLUTE ~1.2e-7. Confirmed OFFLINE first (an arrival copies the destination verbatim, so calls 115/155 hand you fleet 34's and fleet 50's exact float32 destinations = 8 fully determined legs; the 5-narrowing model reproduces the ORIGINAL bit-for-bit on all 8, the old double model reproduces `ours` on the 3 divergent ones), then LIVE: control run 8/45 diverged exit 1, fixed run **0/45 diverged exit 0**, with identical args, identical pos.before and identical ORIGINAL pos.after on all 45 calls. Report `findings/subsystems/movefleet-position-rounding.md`, engine `docs/M-movefleet.md`, branch `wip/movefleet` 2aa8cba |
|
||||
| undeclared ObservedTech append | verify | backlog | — | 0% | 2026-09-08 | NEW (lane R). A tech completion grows `vector<ObservedTech> otch` at ServerPlayer+0x274 (all three vector words move = a realloc). Seen as an undeclared write by BOTH the ProcessResearch `player` guard and the OnTechResearched `player` guard. It is serialized ServerPlayer state and it is in NO coverage note anywhere - a third list append in the same neighbourhood as the event list. B3's replace oracle never saw it because turn 1 of ref-turn2 has no completion |
|
||||
| undeclared ObservedTech append | verify | backlog | — | 0% | 2026-09-08 | NEW (lane R). A tech completion grows `vector<ObservedTech> otch` at ServerPlayer+0x274 (all three vector words move = a realloc). Seen as an undeclared write by BOTH the ProcessResearch `player` guard and the OnTechResearched `player` guard. It is serialized ServerPlayer state and it is in NO coverage note anywhere - a third list append in the same neighbourhood as the event list. B3's replace oracle never saw it because turn 1 of ref-turn2 has no completion | **APPEND SITE NOW NAMED (lane X): `RecordObservedTech` 0x007ba1a0, called from OnTechResearched; stride 0x2c.** Still needs a declared region + a model in ours.
|
||||
| unnamed offsets from guard hits | verify | backlog | — | 0% | 2026-09-08 | NEW (lane R). Three spans the guards report every run and no addresses.json entry names: **ServerSystem+0xd8 (1 B)** and **ServerSystem+0x238 (4 B)** - written by the AI home system on every colony turn, alongside the fleet-vector growth; **StarFleet+0xdc (1 B)** - written on every moving MoveFleet call, just past Speed (FPsp2 @0xd8). Cheap wins for the contract |
|
||||
| waypoint types 2-5 have no coverage | verify | backlog | — | 0% | 2026-09-08 | NEW (lane M, promoted from a coverage line to its own row because it is now the biggest gap in `MoveFleet`). Types 2 (node line), 3 (node route), 4 (gate teleport) and 5 (probabilistic jump) have NEVER fired in any capture, and the node-line step is WRONG BY CONSTRUCTION - `sim::NodeLineStep` and `sim::BuildStutterSegments` are written and host-tested but are NOT wired into the hook, which steps every waypoint type as `speed x dt`. **`ref-turn2` structurally cannot exercise them**: lane M held the VM and tried. The only mover in that save is the AI, which travels straight runs; the player that would travel a node line has `DE 00 CR 00 DN 00` at its home system (screenshot `verify/results/shim/mf-human-home-no-ships.png`), so Move/Manage Fleets are greyed out on every turn. Needs a ship built over several turns, or - much cheaper - a PURPOSE-BUILT SAVE with a fleet already in orbit next to a node line. Same save would unblock the gate-traffic and probabilistic-jump rows. Also owed on that path: `sim::Distance` is still plain double, and `Mars_Vec3_Length` (0x004224b0) says every vector length in the engine is float32-narrowed twice, so the stutter geometry is probably 1 ULP out the same way the position update was - deliberately left alone by lane M because there is no behavioural evidence to correct it against |
|
||||
| ref-turn2 has no tech completion | meta | verified | high | 100% | 2026-09-08 | TRAP for anyone writing a workload (lane R). The documented one-End-Turn recipe produces **zero** `OnTechResearched` calls - an empty log that still passes. It takes 5 End Turns (to turn 7) to reach a completion. Also: only the FIRST End Turn is reproducible - it hashed to the oracle on all four runs and its research calls reproduce docs/B3.md exactly, but from turn 4 the AI picks a different research target than B3 recorded while the point totals stay nearly identical. Treat anything past turn 1 as *a* run, not *the* run |
|
||||
|
|
@ -85,4 +85,5 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`).
|
|||
| MoveFleet waypoint types 2-5 | verify | backlog | — | 0% | 2026-09-08 | Still ZERO behavioural coverage after lane M. Not for lack of trying: the only mover in ref-turn2 is the AI (straight runs only), and the player that would travel a node line has DE/CR/DN all 00 at its home system, so Move/Manage Fleets are greyed out every turn - there is nothing to send along the node lines the map draws. Needs a ship built over several turns or a purpose-built save. The type-2 node-line step is still WRONG BY CONSTRUCTION (B4). Also: sim::Distance deliberately left in double (only stutter geometry uses it); Mars_Vec3_Length says it is probably 1 ULP out the same way, but there is zero behavioural evidence to correct it against - do not "fix" it blind |
|
||||
| P2-P event posting in ours | phase2 | mapped | high | 80% | 2026-09-08 | HOST-VERIFIED, VM RUN QUEUED (lane F holds VM140). next_id reaches 4 in a host reproduction of recap-b3 call 0, fixture rebuilt from raw bytes at the real 0x1c/0x18/0x74 strides and cross-checked against turn3-state.sav with lane C's state_checksum --tree. Count-only (lane E option a): ours never calls the game's PostEvent and REPLACE MODE WRITES NOTHING - a bumped EvNxID with no record behind it would corrupt the very save the oracle hashes. Three design points: the event scan is taken in describe_args BEFORE the original (taken after, ours would dedup against the original's own posts and agree for the wrong reason); dedup risk is MEASURED and reported as events_dedup_risk, not assumed; KeylessEventText resolves keys to "%s" so the shim carries no prose. VERIFIED from the instruction stream: SetResearched 0x00581e10 calls owner vft+0x10 with (flags>>2)&1 and ProcessResearch passes flags=2, so silent=false and the completion event IS posted - previously only inferable from "EvNxID moved by two". ctest 33/33, shim cross-builds on CT111 (lane P could only syntax-check) |
|
||||
| EVENT_TECHS_UNLOCKED not posted (predicted residual) | verify | backlog | — | 0% | 2026-09-08 | Lane P FLAGGED RATHER THAN GUESSED. Trigger IS pinned (SetResearched's sweep sets state=2 + stamps turnAvailable sticky at -1; tail loop collects state==2 && turnAvailable==currentTurn) but evaluating it needs the unlock cascade ours deliberately does not run. The driver takes the unlock list as an INPUT and is handed nullptr ("no list") - deliberately distinct from an empty list ("computed, empty"). PREDICTED RESIDUAL: next_id short by exactly 1 on every completion call. Posting it "whenever something completed" would score on this save and be WRONG the first time a completion unlocks nothing - the exact false-pass shape this project keeps catching |
|
||||
| sizeof(ObservedTech) unpinned | objects | backlog | — | 0% | 2026-09-08 | Lane P declared the region at ServerPlayer+0x274 but did NOT model the bytes. The append site is in neither OnTechResearched's nor SetResearched's decompilation, and `find-constant-uses` for 0x274 returns only unrelated objects because GHIDRA DOES NOT INDEX `lea` DISPLACEMENTS - a systemic blind spot, not a one-off. The declared region's byte delta on a completion call will MEASURE the struct size. Worth a raw-byte scanner for lea-displacement xrefs as a reusable tool |
|
||||
| sizeof(ObservedTech) unpinned | objects | verified | high | 100% | 2026-09-08 | **PINNED (lane X).** `sizeof(Game::ObservedTech) = 0x2c (44)` -- three independent proofs: the magic divide `0x2e8ba2e9 sar 3` (= /44, exact) at 0x0087239f, `imul reg,reg,0x2c` at 0x0087243a / 0x007b735b, and the search stride `add edi,0x2c` at 0x007ba257. **Append site = `RecordObservedTech+0xdf` (0x007ba27f): `lea ecx,[player+0x274]; call vector_ObservedTech_push_back 0x007b7320`** -- a de-duplicating append, direct callee of OnTechResearched 0x00891790; the realloc through 0x007b5820 is why all three vector words move. Element: vptr 0x00a2439c at +0 (RTTI `.?AVObservedTech@Game@@`), std::string at +0x0c (0x18 B); +0x04/+0x06 (16-bit) and +0x08/+0x24/+0x28 hold the four on-disk ints in an order NOT yet determined. Built the general tool the row asked for: `tools/x86disp.py`, an x86 displacement xref scanner (100% code coverage, 0.17% desync). `findings/subsystems/observedtech-append.md` |
|
||||
| lea-displacement xref scanner | meta | verified | high | 100% | 2026-09-08 | `tools/x86disp.py` -- fixes the systemic blind spot that Ghidra does not index ModRM displacements. Full x86-32 length decoder swept from Ghidra's 41,089 function starts: 2,174,504 instructions, 612,166 displacement sites, **100.0% code coverage, 70 desyncs (0.17%), zero unknown opcodes**. Validated against ground truth before use (re-finds `lea eax,[ecx+0x29c]` in GetEventStorage, both OnTechResearched +0x29c sites, and one NEW ProcessTurn site). HONEST LIMITS: it is a **recall** tool, not an oracle -- class-level precision at 0x274 is ~13% by function (99 sites / 45 functions, ~6 real), i.e. a 900x search-space cut that still needs one call-graph check. The naive byte scan it replaces is not wrong so much as **blind**: it misses 80/99 real sites at 0x274 and 13,784/14,611 at disp8 0x14. `cohort` ranking must never be used as a hard filter -- it would have discarded the correct ObservedTech answer. Works off a gitignored local cache in `dumps/`, so it does not hammer CT111 |
|
||||
|
|
|
|||
|
|
@ -284,6 +284,11 @@ said "an arriving call was simply clean"; that is still true and still means not
|
|||
pinned against the instruction stream.
|
||||
* `ServerSystem+0xd8` and `ServerSystem+0x238` are unnamed and are written every turn by the
|
||||
AI home system. `StarFleet+0xdc` likewise, on every move.
|
||||
> **Follow-up 2026-09-08 (lane X):** the append is now named — `RecordObservedTech` `0x007ba1a0`,
|
||||
> called directly from `OnTechResearched`; `sizeof(ObservedTech) = 0x2c`; the realloc that moves all
|
||||
> three words is `vector_44B_grow` `0x007b5820`. See `findings/subsystems/observedtech-append.md`.
|
||||
> That note also corrects harness-audit row 11: nothing in `ComputeBudget` stores to `Budget+0x64`.
|
||||
|
||||
* `vector<ObservedTech> otch` (`ServerPlayer+0x274`) is undeclared everywhere and is save
|
||||
state.
|
||||
* Nothing here exercises: replace mode (any hook), a Zuul double roll, a rebellion, an
|
||||
|
|
|
|||
278
findings/subsystems/observedtech-append.md
Normal file
278
findings/subsystems/observedtech-append.md
Normal file
|
|
@ -0,0 +1,278 @@
|
|||
# `vector<ObservedTech> otch` — pinned, and the displacement scanner that pinned it
|
||||
|
||||
Lane X. Static/host only; VM140 was held by lane F and the game was never run.
|
||||
|
||||
Two things here: a general-purpose tool (`tools/x86disp.py`) for a class of cross-reference
|
||||
Ghidra cannot index, and the `ServerPlayer+0x274` answer it was built to get.
|
||||
|
||||
---
|
||||
|
||||
## 1. The gap the tool fills
|
||||
|
||||
Ghidra indexes **immediate** operands. It does not index **ModRM displacements**. On a
|
||||
41,411-function MSVC C++ binary `lea reg,[reg+disp]` is *the* idiom for taking the address of
|
||||
a member — it is how every `std::vector`, every `std::string`, every embedded sub-object gets
|
||||
passed to a method or a constructor. So `find-constant-uses 0x274` returned 13 unrelated hits
|
||||
and none of them was the vector, and lane P correctly concluded the search was a limitation
|
||||
rather than evidence of absence.
|
||||
|
||||
Lane E only found `EventStorage` at `+0x29c` because a whole accessor function happened to
|
||||
exist (`lea eax,[ecx+0x29c]; ret`). We were blind to this whole reference class all campaign,
|
||||
with ~1,600 classes still to go.
|
||||
|
||||
## 2. The tool — `tools/x86disp.py`
|
||||
|
||||
A real x86-32 **length decoder** (legacy prefixes, 1-/2-/3-byte opcodes, ModRM, SIB, disp8
|
||||
sign-extended, disp32, every immediate form), swept from Ghidra's function starts so every
|
||||
decode begins on a genuine instruction boundary. It indexes every memory operand that carries
|
||||
a displacement, then answers "what code touches offset N off some object?" with the containing
|
||||
function, instruction address, base register and decoded instruction.
|
||||
|
||||
Correctness points that would otherwise produce confident garbage, all handled:
|
||||
|
||||
* `mod=0,rm=5` and `mod=0,rm=4,sib.base=5` are **disp32 with no base** — an absolute global
|
||||
address, not a member offset. Excluded from member queries. (Without this, every
|
||||
`mov eax,[0x00a2bd88]` in the binary shows up as a "displacement".)
|
||||
* `mod=1` disp8 is **sign-extended**: `-0x08` must not be reported as `+0xf8`.
|
||||
* `mod=3` is a register operand with no displacement at all.
|
||||
* `0x67` address-size prefix means 16-bit ModRM, a different layout entirely — refused rather
|
||||
than mis-decoded.
|
||||
|
||||
Commands: `build`, `query <disp>`, `cohort`, `func <va>`, `dis <va>`, `stats`, `brute`.
|
||||
|
||||
**Decode quality:**
|
||||
|
||||
```
|
||||
functions swept : 41089
|
||||
instructions : 2174504
|
||||
disp sites : 612166
|
||||
desyncs : 70 (0.17% of functions)
|
||||
code coverage : 6142049/6142358 (100.0%)
|
||||
```
|
||||
|
||||
Zero unknown-opcode desyncs — the opcode tables cover everything this binary contains. All 70
|
||||
desyncs are truncations at a section/function boundary, not lost sync inside real code.
|
||||
|
||||
> Gotcha worth carrying: the first build clipped each sweep at `fva + Ghidra's sizeInBytes`
|
||||
> and lost 11% of functions to mid-instruction truncation — Ghidra's `sizeInBytes` understates
|
||||
> real bodies often enough to matter (it was cutting valid `mov esp,ebp; pop ebp; ret`
|
||||
> epilogues in half). Sweeping to the **next function start** instead took coverage 89% → 100%.
|
||||
|
||||
The tool works off a local cache (`dumps/`, gitignored: the exe, the function list, the index)
|
||||
so it never hammers the shared Ghidra box. `tools/cache_functions.py` pulls the function list
|
||||
once.
|
||||
|
||||
## 3. Validation — it rediscovers what we already knew
|
||||
|
||||
Run before any new claim was made.
|
||||
|
||||
**GT1 — `ServerPlayer::GetEventStorage`, the one accessor we already had:**
|
||||
|
||||
```
|
||||
$ uv run python3 tools/x86disp.py func 0x0080db00
|
||||
ServerPlayer_GetEventStorage @ 0x0080db00 size 7
|
||||
0x0080db00 lea eax,[ecx+0x29c] ServerPlayer_GetEventStorage+0x0 (8d819c020000)
|
||||
```
|
||||
|
||||
**GT2 — lane E's two `OnTechResearched` sites, plus one it did not have:**
|
||||
|
||||
```
|
||||
$ uv run python3 tools/x86disp.py query 0x29c --lea
|
||||
ProcessTurn @0x00891340
|
||||
0x00891713 lea ecx,[esi+0x29c] ProcessTurn+0x3d3 <-- NEW
|
||||
OnTechResearched @0x00891790
|
||||
0x008919ab lea ecx,[esi+0x29c] OnTechResearched+0x21b <-- known
|
||||
0x0089241d lea ecx,[esi+0x29c] OnTechResearched+0xc8d <-- known
|
||||
```
|
||||
|
||||
**GT3 — `EvNxID` at `ServerPlayer+0x2b0`:** 90 sites found; the `ServerPlayer` ones are present.
|
||||
This one is also the honest illustration of the precision problem — see §5.
|
||||
|
||||
**GT4 — positive control for the cohort ranker.** Given the 50 `ServerPlayer` offsets already
|
||||
in `struct-recovery.md`, `FUN_0087fac0` scores **50 co-hits out of 50** — it is the
|
||||
`ServerPlayer` serializer, and nothing else in the binary comes close. A scanner that could
|
||||
not surface that function from a bare offset query would not be worth using.
|
||||
|
||||
## 4. The answer: `sizeof(ObservedTech)` and the append site
|
||||
|
||||
### `sizeof(Game::ObservedTech) = 0x2c` (44 bytes) — verified, three independent ways
|
||||
|
||||
**(a)** `vector<ObservedTech>::operator=` at `0x00872380` divides the vector's byte span by a
|
||||
constant using MSVC's magic-number sequence:
|
||||
|
||||
```
|
||||
0x00872398 mov ecx,[edi+0x4] ; src._Mylast
|
||||
0x0087239b mov edi,[edi] ; src._Myfirst
|
||||
0x0087239d sub ecx,edi ; byte span
|
||||
0x0087239f mov eax,0x2e8ba2e9
|
||||
0x008723a4 imul ecx
|
||||
0x008723a6 sar edx,3
|
||||
```
|
||||
|
||||
`ceil(2^35 / 44) == 0x2e8ba2e9` **exactly**, and emulating the full sequence reproduces
|
||||
n = 0,1,2,3,10,71,1000 from spans of 0,44,88,132,440,3124,44000. Divisor = 44, unambiguously.
|
||||
|
||||
**(b)** The same function then multiplies back by the literal stride:
|
||||
|
||||
```
|
||||
0x0087243a imul ecx,ecx,0x2c
|
||||
0x0087243d add ecx,[esi] ; this->_Myfirst + n*44
|
||||
0x00872441 mov [esi+0x4],ecx ; this->_Mylast = ...
|
||||
```
|
||||
|
||||
Same literal at `0x00872468` and at `0x007b735b` inside `push_back`.
|
||||
|
||||
**(c)** The linear search in the append function advances its iterator by `add edi,0x2c`
|
||||
(`0x007ba257`).
|
||||
|
||||
This matches the on-disk lower bound *exactly* — 4 × `int` + one `0x1c` `std::string` = 44 —
|
||||
so there is no padding slack anywhere in the element.
|
||||
|
||||
### The append site
|
||||
|
||||
```
|
||||
FUN_007ba1a0 = RecordObservedTech (direct callee of ServerPlayer::OnTechResearched 0x00891790)
|
||||
|
||||
0x007ba221 mov edi,[esi+0x274] ; it = observer->otch._Myfirst
|
||||
0x007ba227 cmp edi,[esi+0x278] ; ... != _Mylast ?
|
||||
0x007ba22d je 0x007ba274 ; empty -> append
|
||||
loop: compare each element's name string (this = elem+0x0c, length = [elem+0x1c])
|
||||
0x007ba257 add edi,0x2c ; ++it *** stride 44 ***
|
||||
0x007ba25a cmp edi,[esi+0x278]
|
||||
0x007ba260 jne loop
|
||||
0x007ba274 lea ecx,[ebp-0x3c]
|
||||
0x007ba277 call 0x008562a0 ; ObservedTech::ObservedTech() on the stack
|
||||
0x007ba27e push eax
|
||||
0x007ba27f lea ecx,[esi+0x274] ; this = &player->otch <<< THE APPEND
|
||||
0x007ba288 call 0x007b7320 ; vector<ObservedTech>::push_back
|
||||
```
|
||||
|
||||
`push_back` (`0x007b7320`) grows via `0x007b5820` when `_Mylast == _Myend` — **that realloc is
|
||||
exactly why lane R's guard saw all three of `player+0x274/0x278/0x27c` move on a completion**,
|
||||
rather than only `_Mylast`.
|
||||
|
||||
`RecordObservedTech` is called from `OnTechResearched` (`0x00891790`) and from `0x007be228`,
|
||||
`0x007be4e1`, `0x007be535`. It is a **de-duplicating** append: it appends only if no existing
|
||||
element already carries that tech name — worth knowing for the reimplementation, since a naive
|
||||
`push_back` would diverge on a re-observation.
|
||||
|
||||
Neither `push_back` nor `operator=` is COMDAT-ambiguous: `0x007b7320` has exactly one caller
|
||||
(`RecordObservedTech`, `ecx = player+0x274`) and `0x00872380` has two, both passing
|
||||
`ServerPlayer+0x274`. `0x007b5820` **is** shared with `FUN_0086dec0` and may be a folded body,
|
||||
so it is labelled `vector_44B_grow`, not as ObservedTech-specific.
|
||||
|
||||
### Element layout — as far as the evidence actually goes
|
||||
|
||||
The default ctor at `0x008562a0` writes vtable `0x00a2439c` at `+0x00`. RTTI:
|
||||
COL `0x00a81c78` → type descriptor `0x00aeede4` → **`.?AVObservedTech@Game@@`**. So
|
||||
`ObservedTech` is polymorphic and its first word is a vptr, not a data field — which the
|
||||
on-disk shape does not tell you.
|
||||
|
||||
| offset | size | evidence |
|
||||
|---|---|---|
|
||||
| `+0x00` | 4 | vptr `0x00a2439c`; ctor writes it, RTTI-confirmed |
|
||||
| `+0x04` | 2 | `mov word [eax-0x28],cx` at `0x007ba2b0` (`eax` = `_Mylast`, element = `_Mylast-0x2c`), source `[ebx+0xc]` |
|
||||
| `+0x06` | 2 | `mov word [eax-0x26],dx` at `0x007ba2c6`, **same source word** `[ebx+0xc]` |
|
||||
| `+0x08` | 4 | **unaccounted** |
|
||||
| `+0x0c..+0x23` | 0x18 | `std::string` (the `otch` tech name). Object base is `+0x0c`, MSVC layout `{_Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14}`. Ctor writes `[+0x0c]=0`, `[+0x1c]=0`, `[+0x20]=0xf`; the search loop reads `[+0x1c]` as the length and calls the compare with `this = +0x0c`; the post-append assign uses `lea ecx,[_Mylast-0x20]` = `+0x0c` |
|
||||
| `+0x24` | 4 | **unaccounted** |
|
||||
| `+0x28` | 4 | **unaccounted** |
|
||||
|
||||
Note the string here is **0x18 bytes**, not the 0x1c the `ServerPlayer::pswd` row in
|
||||
`struct-recovery.md` implies — three separate reads inside this element agree on
|
||||
`{buf16, _Mysize@+0x10, _Myres@+0x14}`. Worth re-checking `pswd` against that.
|
||||
|
||||
The four on-disk ints (`otnF`, `otnL`, `odet`, `owith`) have to map onto `+0x04`/`+0x06` (two
|
||||
16-bit fields) and the three 4-byte slots `+0x08`, `+0x24`, `+0x28`. **That mapping is not
|
||||
determined here and is deliberately not guessed.** The one suggestive observation, flagged as a
|
||||
*hypothesis only*: `+0x04` and `+0x06` are two adjacent 16-bit fields written from the same
|
||||
source word on first observation, which is the shape you would expect of a first-seen /
|
||||
last-seen turn pair — but nothing here proves it.
|
||||
|
||||
What would settle the mapping: read `ObservedTech`'s `Read`/`Write` serializer, where the
|
||||
member order is explicit. Lane P's `observed_techs` region byte delta remains a valid live
|
||||
cross-check and should now come back as exactly 44 per completion.
|
||||
|
||||
## 5. False-positive rate, honestly
|
||||
|
||||
Two different error rates, and the interesting one is not the one you'd expect.
|
||||
|
||||
**Decode-level error of the naive method is low.** A raw byte scan for `8D` + ModRM + the
|
||||
displacement — the thing you'd write without a decoder — is mostly *right*:
|
||||
|
||||
| query | naive candidates | not actually an instruction | real sites the naive scan **missed** |
|
||||
|---|---|---|---|
|
||||
| `lea`, disp32 `0x274` | 19 | 0 (0.0%) | **80 of 99** |
|
||||
| 11 common opcodes, disp32 `0x274` | 89 | 1 (1.1%) | 11 of 99 |
|
||||
| `lea`, disp8 `0x14` | 828 | 1 (0.1%) | **13,784 of 14,611** |
|
||||
| 11 common opcodes, disp8 `0x14` | 10,326 | 59 (0.6%) | 4,344 of 14,611 |
|
||||
|
||||
So the decoder's win is **recall, not decode precision**. A hand-written opcode set misses
|
||||
80–94% of the real accesses, because a member is read, written, compared, and float-loaded far
|
||||
more often than its address is taken, and you cannot enumerate those opcodes by hand.
|
||||
|
||||
**Class-level precision is the real problem, and it is poor.** The scanner knows the
|
||||
displacement; it cannot know what class the base register holds. `query 0x274` returns 99 sites
|
||||
in 45 functions and only about 6 of those functions are actually touching `ServerPlayer::otch`
|
||||
— roughly **13% precision by function**. The honest way to state the value is as search-space
|
||||
reduction: 41,411 functions → 45, about **900×**, down to a list a human reads in two minutes.
|
||||
|
||||
**The cohort ranker helps, and has a trap.** Scoring functions by how many *already-known*
|
||||
offsets of the same class they touch pulls real class methods to the top (the serializer hits
|
||||
50/50). But it would have **thrown away the correct answer**: `RecordObservedTech` touches only
|
||||
`0x274` and `0x278` and nothing else on `ServerPlayer`, so any `--min>=1` filter drops it. Use
|
||||
cohort as a ranker, never as a filter. This is now written into the tool's own docstring.
|
||||
|
||||
What actually closed the case was the plain `query 0x274` list **intersected with one call-graph
|
||||
lookup** (`OnTechResearched`'s direct callees). Displacement scan for recall, call graph for
|
||||
disambiguation — neither alone was enough.
|
||||
|
||||
## 6. Previously anonymous offsets — attribution results
|
||||
|
||||
The payoff was smaller than hoped, because most of the audit's anonymous offsets had already
|
||||
been named by other lanes since the audit was written.
|
||||
|
||||
| offset | status before | after |
|
||||
|---|---|---|
|
||||
| `player+0x274/0x278/0x27c` | "vector grew, append site unknown" | **`RecordObservedTech+0xdf` (0x007ba27f)**, `sizeof` = 0x2c |
|
||||
| `player+0x2b0` | already named by lane E | unchanged (`EvNxID`) |
|
||||
| `TechTree+0x20` | already `TechTree_off_OrderCounter` | unchanged |
|
||||
| `ServerPlayer+0x308` | already `ServerPlayer_off_NodeBore` | unchanged |
|
||||
| `Budget+0x64` | harness-audit row 11: "the original writes it" | **not supported.** See below |
|
||||
|
||||
**`Budget+0x64` (harness-audit row 11) — a correction.** `ServerPlayer::ComputeBudget`
|
||||
(`0x00863030`) writes its `Budget*` out-param through `esi`, and every such store lands in
|
||||
`+0x00..+0x54` (the 22-int block). The only two `+0x64` accesses in the whole function are
|
||||
**loads off a different base register** (`mov ecx,[eax+0x64]` at `0x0086328c`,
|
||||
`mov edx,[eax+0x64]` at `0x0086335d`). There is no store to `Budget+0x64` in `ComputeBudget`.
|
||||
|
||||
Separately, `TechTree::ProcessResearch`'s `int* overbudget` fourth argument is **not**
|
||||
`Budget+0x64`: its only caller is `ProcessTurn` (`0x00891340`) and the call at `0x008914a5`
|
||||
passes `lea edx,[ebp-0x14]` — a stack local, consumed immediately after the call
|
||||
(`mov eax,[ebp-0x14]; cmp eax,0; jle`).
|
||||
|
||||
This agrees with lane R, whose `budget_object` guard saw `Budget+0x64` change in **0 of 4284
|
||||
calls**. Row 11 should be reclassified from "the original writes it and B1 never checked" to
|
||||
"nothing has been shown to write it"; the remaining way to settle it is a write watchpoint on
|
||||
that word, not another static search.
|
||||
|
||||
## 7. Verdict on the technique
|
||||
|
||||
Worth keeping, with its limits stated. It answered a question that had been parked, and the
|
||||
`0x29c` validation turned up a `ProcessTurn` `EventStorage` site nobody had. But it is a
|
||||
**recall** tool that produces a 20–100 line candidate list per offset, not an oracle: every
|
||||
result still needs a call-graph or decompiler check before it is a fact. For the ~1,600
|
||||
remaining classes the realistic workflow is `query <offset>` → read the list → confirm with one
|
||||
cross-reference call.
|
||||
|
||||
## 8. Ghidra writeback
|
||||
|
||||
Labels: `RecordObservedTech` `0x007ba1a0`, `vector_ObservedTech_push_back` `0x007b7320`,
|
||||
`ObservedTech_ctor` `0x008562a0`, `vector_ObservedTech_assign` `0x00872380`,
|
||||
`vector_44B_grow` `0x007b5820`, `vftable_ObservedTech` `0x00a2439c`. Plate comments carrying
|
||||
the stride evidence on the first four.
|
||||
|
||||
`addresses.json`: `ObservedTech_sizeof`, `ObservedTech_vftable`, `ObservedTech_off_Name`,
|
||||
`RecordObservedTech`, `vector_ObservedTech_push_back`, `vector_ObservedTech_assign`,
|
||||
`ObservedTech_ctor`, `vector_44B_grow`; `ServerPlayer_off_ObservedTechs` updated from
|
||||
"NOT PINNED" to `verified`.
|
||||
|
|
@ -109,7 +109,15 @@ New `addresses.json` entries from this read: `TechNode_off_TurnAvailable` 0x20,
|
|||
`TechNode_off_TurnResearched` 0x24, `TechNode_off_Order` 0x28, `TechNode_off_Children` 0x04,
|
||||
`TechEdge_off_CostRP` 0x1c, `TechEdge_off_ChildDef` 0x40, `TechTree_off_OrderCounter` 0x20.
|
||||
|
||||
## 5. `vector<ObservedTech> otch` at `ServerPlayer+0x274` — still not pinned
|
||||
## 5. `vector<ObservedTech> otch` at `ServerPlayer+0x274` — ~~still not pinned~~ **PINNED**
|
||||
|
||||
> **RESOLVED 2026-09-08 by lane X** — see `findings/subsystems/observedtech-append.md`.
|
||||
> `sizeof(Game::ObservedTech) = 0x2c (44)`. The append is `RecordObservedTech+0xdf` (`0x007ba27f`):
|
||||
> `lea ecx,[player+0x274]; call vector_ObservedTech_push_back 0x007b7320`. `RecordObservedTech`
|
||||
> (`0x007ba1a0`) is a **direct callee of `OnTechResearched`** and de-duplicates by tech name before
|
||||
> appending. The suspicion recorded below was right: `find-constant-uses` missed it because the
|
||||
> encoding is a `lea` displacement. `tools/x86disp.py` now indexes those.
|
||||
> The rest of this section is left as written, as the record of what was known at the time.
|
||||
|
||||
Lane R's guards caught all three vector words moving on every tech completion, in both the
|
||||
`ProcessResearch` and the `OnTechResearched` player guards, and it is serialized `ServerPlayer`
|
||||
|
|
|
|||
|
|
@ -1454,9 +1454,9 @@
|
|||
"name": "ServerPlayer_off_ObservedTechs",
|
||||
"offset": "0x274",
|
||||
"convention": "offset",
|
||||
"prototype": "std::vector<ObservedTech> otch (3 words {first,last,end}); save tag otch, element {int otnF, otnL, odet; string otch; int owith}. All three words move on every tech completion (a realloc) -- observed live by both the ProcessResearch and the OnTechResearched player guards. NOT PINNED: the append call site and sizeof(ObservedTech); the byte span the observed_techs region reports is what will measure the stride",
|
||||
"status": "verified-by-save",
|
||||
"source": "findings/objects/struct-recovery.md#2 (Read/Write serializers) + findings/subsystems/golden-trace-recapture.md (player guard, both completion calls)"
|
||||
"prototype": "std::vector<ObservedTech> otch -- 3 words {_Myfirst@0x274,_Mylast@0x278,_Myend@0x27c}; save tag otch. sizeof(ObservedTech) = 0x2c (44), PINNED three ways: magic divide 0x2e8ba2e9 sar 3 (=/44) in vector_ObservedTech_assign 0x0087239f, imul reg,reg,0x2c at 0x0087243a / 0x007b735b, and the linear search stride `add edi,0x2c` at 0x007ba257. Append site = RecordObservedTech+0xdf (0x007ba27f): lea ecx,[player+0x274]; call vector_ObservedTech_push_back 0x007b7320. All three words move because push_back reallocs through vector_44B_grow 0x007b5820.",
|
||||
"status": "verified",
|
||||
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
|
||||
},
|
||||
{
|
||||
"name": "ServerPlayer_off_IncMod",
|
||||
|
|
@ -3169,6 +3169,70 @@
|
|||
"prototype": "site inside StrategyServer::MoveFleet /* the move != distance branch: exactly two roundings per component, tmp.c = float32(dir.c * move) then pos.c = float32(pos.c + tmp.c). `move` is reloaded from a float32 slot. The sibling branch (move == distance, an EXACT float compare) copies the destination's three words verbatim with mov, so an arrival never steps onto its destination. */",
|
||||
"status": "verified",
|
||||
"source": "lane M own disassembly pass 2026-09-08"
|
||||
},
|
||||
{
|
||||
"name": "ObservedTech_sizeof",
|
||||
"offset": "0x2c",
|
||||
"convention": "constant",
|
||||
"prototype": "sizeof(Game::ObservedTech) = 0x2c (44 bytes). Confirmed independently by (a) the compiler's magic division by 44 (mov eax,0x2e8ba2e9; imul; sar edx,3) at 0x0087239f and 0x007b7339, (b) imul reg,reg,0x2c at 0x0087243a, 0x00872468, 0x007b735b, and (c) the iterator advance `add edi,0x2c` in RecordObservedTech's linear search at 0x007ba257. Matches the on-disk lower bound exactly (4 int + one 0x1c std::string = 44), so there is no padding slack.",
|
||||
"status": "verified",
|
||||
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
|
||||
},
|
||||
{
|
||||
"name": "ObservedTech_vftable",
|
||||
"addr": "0x00a2439c",
|
||||
"convention": "data",
|
||||
"prototype": "Game::ObservedTech vftable. RTTI COL 0x00a81c78 -> type descriptor 0x00aeede4 = '.?AVObservedTech@Game@@'. Written to element+0x00 by ObservedTech_ctor 0x008562a0 -- so ObservedTech is polymorphic and its first word is the vptr, not a data field.",
|
||||
"status": "verified",
|
||||
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
|
||||
},
|
||||
{
|
||||
"name": "ObservedTech_off_Name",
|
||||
"offset": "0x0c",
|
||||
"convention": "offset",
|
||||
"prototype": "std::string (save tag `otch`, the tech name) at ObservedTech+0x0c, 0x18 bytes, spanning +0x0c..+0x23. MSVC layout relative to the string object: _Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14 -- i.e. ObservedTech+0x1c is the length and +0x20 the capacity. Evidence: ObservedTech_ctor 0x008562a0 writes [elem+0x0c]=0, [elem+0x1c]=0, [elem+0x20]=0xf; RecordObservedTech's search reads [elem+0x1c] as the length and calls compare with this=elem+0x0c; the post-append assign uses lea ecx,[_Mylast-0x20]. NOTE this string is 0x18 bytes, not the 0x1c implied by the ServerPlayer pswd row in struct-recovery.md. UNACCOUNTED in the element: +0x08, +0x24, +0x28 (4 bytes each) plus two 16-bit fields at +0x04/+0x06 written from one source word at 0x007ba2b0 / 0x007ba2c6 -- the mapping of the four on-disk ints (otnF, otnL, odet, owith) onto those slots is NOT determined.",
|
||||
"status": "verified",
|
||||
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
|
||||
},
|
||||
{
|
||||
"name": "RecordObservedTech",
|
||||
"addr": "0x007ba1a0",
|
||||
"convention": "thiscall",
|
||||
"prototype": "void (this, ServerPlayer* observer, ?, std::string* techName) -- appends to observer->otch. Linear-searches observer->otch (ServerPlayer+0x274) with stride 0x2c comparing each element's name string; if not found, default-constructs an ObservedTech on the stack (0x008562a0) and push_backs it (0x007b7320 @0x007ba288), then writes two 16-bit fields into the new element at +0x04 and +0x06 from param_1+0xc. DIRECT CALLEE of ServerPlayer::OnTechResearched 0x00891790; also called from 0x007be228, 0x007be4e1, 0x007be535. This is the append lane P could not find.",
|
||||
"status": "verified",
|
||||
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
|
||||
},
|
||||
{
|
||||
"name": "vector_ObservedTech_push_back",
|
||||
"addr": "0x007b7320",
|
||||
"convention": "thiscall",
|
||||
"prototype": "void (std::vector<ObservedTech>* this, ObservedTech* value) RET 4. Stride 0x2c. Calls vector_44B_grow 0x007b5820 when _Mylast==_Myend -- the realloc that moves all three vector words. Exactly one caller (RecordObservedTech), so this instantiation is not COMDAT-ambiguous.",
|
||||
"status": "verified",
|
||||
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
|
||||
},
|
||||
{
|
||||
"name": "vector_ObservedTech_assign",
|
||||
"addr": "0x00872380",
|
||||
"convention": "thiscall",
|
||||
"prototype": "std::vector<ObservedTech>& operator=(const std::vector<ObservedTech>&) RET 4. Both callers pass ServerPlayer+0x274 (0x00878091 in the settings copy-out, 0x00892507 in the copy-in).",
|
||||
"status": "verified",
|
||||
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
|
||||
},
|
||||
{
|
||||
"name": "ObservedTech_ctor",
|
||||
"addr": "0x008562a0",
|
||||
"convention": "thiscall",
|
||||
"prototype": "Game::ObservedTech* (ObservedTech* this) -- default ctor; sets vptr 0x00a2439c and empties the name string.",
|
||||
"status": "verified",
|
||||
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
|
||||
},
|
||||
{
|
||||
"name": "vector_44B_grow",
|
||||
"addr": "0x007b5820",
|
||||
"convention": "thiscall",
|
||||
"prototype": "vector<T>::_Reserve/grow for a 44-byte element type. Shared with FUN_0086dec0, so it may be a COMDAT-folded body -- do NOT assume it is ObservedTech-specific.",
|
||||
"status": "mapped",
|
||||
"source": "findings/subsystems/observedtech-append.md (lane X, tools/x86disp.py displacement scan)"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
// GENERATED — do not edit. Facts about Sword of the Stars.exe (GOG 1.8.1).
|
||||
// Source: sots-re ghidra/addresses.json @ b0ef139, generated 2026-09-08 by tools/gen_addresses.py
|
||||
// Source: sots-re ghidra/addresses.json @ 1a58bcd, generated 2026-09-08 by tools/gen_addresses.py
|
||||
// Runtime address = (uintptr_t)GetModuleHandle(NULL) + RVA (the exe is ASLR-relocated).
|
||||
#pragma once
|
||||
#include <cstdint>
|
||||
|
|
@ -369,7 +369,7 @@ constexpr uint32_t ServerPlayer_off_SetupResearchMult = 0x0000022c;
|
|||
constexpr uint32_t ServerPlayer_off_Sav = 0x00000284;
|
||||
// offset Tech* current research target (ResT); NULL = none [verified-by-save]
|
||||
constexpr uint32_t ServerPlayer_off_ResearchTarget = 0x00000294;
|
||||
// offset std::vector<ObservedTech> otch (3 words {first,last,end}); save tag otch, element {int otnF, otnL, odet; string otch; int owith}. All three words move on every tech completion (a realloc) -- observed live by both the ProcessResearch and the OnTechResearched player guards. NOT PINNED: the append call site and sizeof(ObservedTech); the byte span the observed_techs region reports is what will measure the stride [verified-by-save]
|
||||
// offset std::vector<ObservedTech> otch -- 3 words {_Myfirst@0x274,_Mylast@0x278,_Myend@0x27c}; save tag otch. sizeof(ObservedTech) = 0x2c (44), PINNED three ways: magic divide 0x2e8ba2e9 sar 3 (=/44) in vector_ObservedTech_assign 0x0087239f, imul reg,reg,0x2c at 0x0087243a / 0x007b735b, and the linear search stride `add edi,0x2c` at 0x007ba257. Append site = RecordObservedTech+0xdf (0x007ba27f): lea ecx,[player+0x274]; call vector_ObservedTech_push_back 0x007b7320. All three words move because push_back reallocs through vector_44B_grow 0x007b5820. [verified]
|
||||
constexpr uint32_t ServerPlayer_off_ObservedTechs = 0x00000274;
|
||||
// offset float IncMod [verified-by-save]
|
||||
constexpr uint32_t ServerPlayer_off_IncMod = 0x0000030c;
|
||||
|
|
@ -799,5 +799,21 @@ constexpr uint32_t Mars_Vec3_NormaliseEpsilon = 0x005e1ef8;
|
|||
constexpr uint32_t StrategyServer_MoveFleet_straight_leg = 0x003da0f2;
|
||||
// site site inside StrategyServer::MoveFleet /* the move != distance branch: exactly two roundings per component, tmp.c = float32(dir.c * move) then pos.c = float32(pos.c + tmp.c). `move` is reloaded from a float32 slot. The sibling branch (move == distance, an EXACT float compare) copies the destination's three words verbatim with mov, so an arrival never steps onto its destination. */ [verified]
|
||||
constexpr uint32_t StrategyServer_MoveFleet_position_update = 0x003da2ac;
|
||||
// constant sizeof(Game::ObservedTech) = 0x2c (44 bytes). Confirmed independently by (a) the compiler's magic division by 44 (mov eax,0x2e8ba2e9; imul; sar edx,3) at 0x0087239f and 0x007b7339, (b) imul reg,reg,0x2c at 0x0087243a, 0x00872468, 0x007b735b, and (c) the iterator advance `add edi,0x2c` in RecordObservedTech's linear search at 0x007ba257. Matches the on-disk lower bound exactly (4 int + one 0x1c std::string = 44), so there is no padding slack. [verified]
|
||||
constexpr uint32_t ObservedTech_sizeof = 0x0000002c;
|
||||
// data Game::ObservedTech vftable. RTTI COL 0x00a81c78 -> type descriptor 0x00aeede4 = '.?AVObservedTech@Game@@'. Written to element+0x00 by ObservedTech_ctor 0x008562a0 -- so ObservedTech is polymorphic and its first word is the vptr, not a data field. [verified]
|
||||
constexpr uint32_t ObservedTech_vftable = 0x0062439c;
|
||||
// offset std::string (save tag `otch`, the tech name) at ObservedTech+0x0c, 0x18 bytes, spanning +0x0c..+0x23. MSVC layout relative to the string object: _Bx[16] @+0x00, _Mysize @+0x10, _Myres @+0x14 -- i.e. ObservedTech+0x1c is the length and +0x20 the capacity. Evidence: ObservedTech_ctor 0x008562a0 writes [elem+0x0c]=0, [elem+0x1c]=0, [elem+0x20]=0xf; RecordObservedTech's search reads [elem+0x1c] as the length and calls compare with this=elem+0x0c; the post-append assign uses lea ecx,[_Mylast-0x20]. NOTE this string is 0x18 bytes, not the 0x1c implied by the ServerPlayer pswd row in struct-recovery.md. UNACCOUNTED in the element: +0x08, +0x24, +0x28 (4 bytes each) plus two 16-bit fields at +0x04/+0x06 written from one source word at 0x007ba2b0 / 0x007ba2c6 -- the mapping of the four on-disk ints (otnF, otnL, odet, owith) onto those slots is NOT determined. [verified]
|
||||
constexpr uint32_t ObservedTech_off_Name = 0x0000000c;
|
||||
// thiscall void (this, ServerPlayer* observer, ?, std::string* techName) -- appends to observer->otch. Linear-searches observer->otch (ServerPlayer+0x274) with stride 0x2c comparing each element's name string; if not found, default-constructs an ObservedTech on the stack (0x008562a0) and push_backs it (0x007b7320 @0x007ba288), then writes two 16-bit fields into the new element at +0x04 and +0x06 from param_1+0xc. DIRECT CALLEE of ServerPlayer::OnTechResearched 0x00891790; also called from 0x007be228, 0x007be4e1, 0x007be535. This is the append lane P could not find. [verified]
|
||||
constexpr uint32_t RecordObservedTech = 0x003ba1a0;
|
||||
// thiscall void (std::vector<ObservedTech>* this, ObservedTech* value) RET 4. Stride 0x2c. Calls vector_44B_grow 0x007b5820 when _Mylast==_Myend -- the realloc that moves all three vector words. Exactly one caller (RecordObservedTech), so this instantiation is not COMDAT-ambiguous. [verified]
|
||||
constexpr uint32_t vector_ObservedTech_push_back = 0x003b7320;
|
||||
// thiscall std::vector<ObservedTech>& operator=(const std::vector<ObservedTech>&) RET 4. Both callers pass ServerPlayer+0x274 (0x00878091 in the settings copy-out, 0x00892507 in the copy-in). [verified]
|
||||
constexpr uint32_t vector_ObservedTech_assign = 0x00472380;
|
||||
// thiscall Game::ObservedTech* (ObservedTech* this) -- default ctor; sets vptr 0x00a2439c and empties the name string. [verified]
|
||||
constexpr uint32_t ObservedTech_ctor = 0x004562a0;
|
||||
// thiscall vector<T>::_Reserve/grow for a 44-byte element type. Shared with FUN_0086dec0, so it may be a COMDAT-folded body -- do NOT assume it is ObservedTech-specific. [mapped]
|
||||
constexpr uint32_t vector_44B_grow = 0x003b5820;
|
||||
|
||||
} // namespace sots::addr
|
||||
|
|
|
|||
57
tools/cache_functions.py
Normal file
57
tools/cache_functions.py
Normal file
|
|
@ -0,0 +1,57 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Cache the full Ghidra function list (address, size, name) to dumps/functions.json.
|
||||
|
||||
Ghidra/ReVa is a shared resource -- pull the list once, then work offline.
|
||||
Usage: uv run python3 tools/cache_functions.py [--refresh]
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
REPO = os.path.dirname(HERE)
|
||||
OUT = os.path.join(REPO, "dumps", "functions.json")
|
||||
PROG = "/Sword of the Stars.exe"
|
||||
|
||||
|
||||
def call(tool, args):
|
||||
r = subprocess.run(
|
||||
["uv", "run", "python3", os.path.join(HERE, "reva_call.py"), tool, json.dumps(args)],
|
||||
capture_output=True, text=True, cwd=REPO, timeout=300)
|
||||
if r.returncode != 0:
|
||||
sys.exit(f"reva_call {tool} failed: {r.stderr[:400]}")
|
||||
return r.stdout
|
||||
|
||||
|
||||
def main():
|
||||
if os.path.exists(OUT) and "--refresh" not in sys.argv:
|
||||
print(f"{OUT} exists; use --refresh to re-pull")
|
||||
return
|
||||
funcs = {}
|
||||
start = 0
|
||||
while True:
|
||||
raw = call("get-functions", {
|
||||
"programPath": PROG, "startIndex": start, "filterDefaultNames": False})
|
||||
lines = [ln for ln in raw.splitlines() if ln.strip()]
|
||||
header = json.loads(lines[0])
|
||||
for ln in lines[1:]:
|
||||
try:
|
||||
f = json.loads(ln)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
funcs[f["address"]] = [f["name"], f.get("sizeInBytes", 0)]
|
||||
nxt = header.get("nextStartIndex")
|
||||
total = header.get("totalCount")
|
||||
print(f" {len(funcs)}/{total}", file=sys.stderr)
|
||||
if nxt is None or nxt <= start or len(funcs) >= total:
|
||||
break
|
||||
start = nxt
|
||||
os.makedirs(os.path.dirname(OUT), exist_ok=True)
|
||||
with open(OUT, "w") as fh:
|
||||
json.dump(funcs, fh)
|
||||
print(f"wrote {len(funcs)} functions to {OUT}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
798
tools/x86disp.py
Normal file
798
tools/x86disp.py
Normal file
|
|
@ -0,0 +1,798 @@
|
|||
#!/usr/bin/env python3
|
||||
"""x86-32 displacement cross-referencer for the SOTS1 exe.
|
||||
|
||||
Why this exists
|
||||
---------------
|
||||
Ghidra indexes immediate operands but NOT ModRM displacements. On an MSVC C++
|
||||
binary `lea reg,[reg+disp]` is *the* idiom for taking the address of a member --
|
||||
it is how every std::vector / std::string / embedded sub-object is passed to a
|
||||
method or ctor. `find-constant-uses 0x274` therefore cannot see the append site
|
||||
for a vector member at +0x274. This tool answers the question Ghidra can't:
|
||||
|
||||
"what code takes the address of, or accesses, offset N off some object?"
|
||||
|
||||
How it works
|
||||
------------
|
||||
x86 is variable-length and not self-synchronising, so a naive byte scan for
|
||||
"8D 8E <disp32>" produces confident garbage. Instead we run a full instruction
|
||||
*length* decoder (prefixes / 1-,2-,3-byte opcodes / ModRM / SIB / disp / imm)
|
||||
seeded from Ghidra's 41k function starts, so every decode begins on a real
|
||||
instruction boundary. `--brute` runs the naive scan too, purely so the
|
||||
false-positive rate of the naive method can be *measured* rather than guessed.
|
||||
|
||||
Correctness notes that matter (each of these is a silent-garbage source):
|
||||
* mod=0,rm=5 -> disp32 with NO base: an absolute global address, not a
|
||||
member offset. Excluded from member queries by default.
|
||||
* mod=0,rm=4,
|
||||
sib.base=5 -> disp32, no base, index only. Same: excluded.
|
||||
* mod=1 -> disp8 SIGN-EXTENDED. -0x08 must not be reported as 0xf8.
|
||||
* mod=3 -> register operand, no memory, no displacement at all.
|
||||
* 0x67 addr-size -> 16-bit ModRM, entirely different layout. Skipped, flagged.
|
||||
|
||||
Usage
|
||||
-----
|
||||
uv run python3 tools/x86disp.py build # decode + cache index
|
||||
uv run python3 tools/x86disp.py query 0x274 # who touches +0x274
|
||||
uv run python3 tools/x86disp.py query 0x29c --lea # lea only
|
||||
uv run python3 tools/x86disp.py func 0x0080db00 # dump one function
|
||||
uv run python3 tools/x86disp.py stats
|
||||
"""
|
||||
import bisect
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
REPO = os.path.dirname(HERE)
|
||||
EXE = os.path.join(REPO, "dumps", "sots.exe")
|
||||
FUNCS = os.path.join(REPO, "dumps", "functions.json")
|
||||
INDEX = os.path.join(REPO, "dumps", "dispindex.json")
|
||||
|
||||
R32 = ["eax", "ecx", "edx", "ebx", "esp", "ebp", "esi", "edi"]
|
||||
|
||||
# ---------------------------------------------------------------- opcode maps
|
||||
# value: (has_modrm, imm_kind)
|
||||
# imm kinds: 0 none | 'b' 1 | 'w' 2 | 'd' 4 | 'z' 2-if-66-else-4
|
||||
# 'p' far ptr 6 (4 with 66) | 'a' moffs (addr-size) | 'e' enter 3
|
||||
# 'g6' F6 group (b if reg<2) | 'g7' F7 group (z if reg<2)
|
||||
_ONE = {}
|
||||
|
||||
|
||||
def _fill(rng, modrm, imm):
|
||||
for o in rng:
|
||||
_ONE[o] = (modrm, imm)
|
||||
|
||||
|
||||
# 00..3F: the eight ALU ops, each /r /r /r /r AL,ib eAX,iz + 2 seg ops
|
||||
for _base in (0x00, 0x08, 0x10, 0x18, 0x20, 0x28, 0x30, 0x38):
|
||||
_fill(range(_base, _base + 4), True, 0)
|
||||
_ONE[_base + 4] = (False, "b")
|
||||
_ONE[_base + 5] = (False, "z")
|
||||
_ONE[_base + 6] = (False, 0) # PUSH seg / prefix-or-ascii-adjust
|
||||
_ONE[_base + 7] = (False, 0)
|
||||
_fill(range(0x40, 0x60), False, 0) # INC/DEC/PUSH/POP r32
|
||||
_fill([0x60, 0x61], False, 0) # PUSHA/POPA
|
||||
_ONE[0x62] = (True, 0) # BOUND
|
||||
_ONE[0x63] = (True, 0) # ARPL
|
||||
_ONE[0x68] = (False, "z") # PUSH iz
|
||||
_ONE[0x69] = (True, "z") # IMUL r,Ev,iz
|
||||
_ONE[0x6A] = (False, "b") # PUSH ib
|
||||
_ONE[0x6B] = (True, "b") # IMUL r,Ev,ib
|
||||
_fill(range(0x6C, 0x70), False, 0) # INS/OUTS
|
||||
_fill(range(0x70, 0x80), False, "b") # Jcc rel8
|
||||
_ONE[0x80] = (True, "b")
|
||||
_ONE[0x81] = (True, "z")
|
||||
_ONE[0x82] = (True, "b")
|
||||
_ONE[0x83] = (True, "b")
|
||||
_fill(range(0x84, 0x90), True, 0) # TEST/XCHG/MOV/MOV-seg/LEA/POP Ev
|
||||
_fill(range(0x90, 0x9A), False, 0) # NOP/XCHG/CWDE/CDQ
|
||||
_ONE[0x9A] = (False, "p") # CALLF
|
||||
_fill(range(0x9B, 0xA0), False, 0)
|
||||
_fill(range(0xA0, 0xA4), False, "a") # MOV moffs
|
||||
_fill(range(0xA4, 0xA8), False, 0) # MOVS/CMPS
|
||||
_ONE[0xA8] = (False, "b")
|
||||
_ONE[0xA9] = (False, "z")
|
||||
_fill(range(0xAA, 0xB0), False, 0) # STOS/LODS/SCAS
|
||||
_fill(range(0xB0, 0xB8), False, "b") # MOV r8,ib
|
||||
_fill(range(0xB8, 0xC0), False, "z") # MOV r32,iz
|
||||
_ONE[0xC0] = (True, "b")
|
||||
_ONE[0xC1] = (True, "b")
|
||||
_ONE[0xC2] = (False, "w") # RET imm16
|
||||
_ONE[0xC3] = (False, 0)
|
||||
_ONE[0xC4] = (True, 0) # LES
|
||||
_ONE[0xC5] = (True, 0) # LDS
|
||||
_ONE[0xC6] = (True, "b") # MOV Eb,Ib
|
||||
_ONE[0xC7] = (True, "z") # MOV Ev,Iz
|
||||
_ONE[0xC8] = (False, "e") # ENTER iw,ib
|
||||
_ONE[0xC9] = (False, 0)
|
||||
_ONE[0xCA] = (False, "w") # RETF imm16
|
||||
_fill([0xCB, 0xCC], False, 0)
|
||||
_ONE[0xCD] = (False, "b") # INT ib
|
||||
_fill([0xCE, 0xCF], False, 0)
|
||||
_fill(range(0xD0, 0xD4), True, 0) # shift group by 1 / by CL
|
||||
_ONE[0xD4] = (False, "b") # AAM
|
||||
_ONE[0xD5] = (False, "b") # AAD
|
||||
_fill([0xD6, 0xD7], False, 0)
|
||||
_fill(range(0xD8, 0xE0), True, 0) # x87 -- always ModRM
|
||||
_fill(range(0xE0, 0xE4), False, "b") # LOOP*/JECXZ
|
||||
_fill(range(0xE4, 0xE8), False, "b") # IN/OUT ib
|
||||
_ONE[0xE8] = (False, "z") # CALL rel32
|
||||
_ONE[0xE9] = (False, "z") # JMP rel32
|
||||
_ONE[0xEA] = (False, "p") # JMPF
|
||||
_ONE[0xEB] = (False, "b") # JMP rel8
|
||||
_fill(range(0xEC, 0xF0), False, 0) # IN/OUT DX
|
||||
_fill(range(0xF0, 0xF6), False, 0) # LOCK/INT1/REP*/HLT/CMC
|
||||
_ONE[0xF6] = (True, "g6")
|
||||
_ONE[0xF7] = (True, "g7")
|
||||
_fill(range(0xF8, 0xFE), False, 0)
|
||||
_ONE[0xFE] = (True, 0)
|
||||
_ONE[0xFF] = (True, 0)
|
||||
|
||||
_TWO = {}
|
||||
|
||||
|
||||
def _fill2(rng, modrm, imm):
|
||||
for o in rng:
|
||||
_TWO[o] = (modrm, imm)
|
||||
|
||||
|
||||
_fill2(range(0x00, 0x05), True, 0)
|
||||
_fill2(range(0x05, 0x0D), False, 0)
|
||||
_TWO[0x0D] = (True, 0)
|
||||
_TWO[0x0E] = (False, 0)
|
||||
_TWO[0x0F] = (True, "b") # 3DNow!
|
||||
_fill2(range(0x10, 0x18), True, 0)
|
||||
_fill2(range(0x18, 0x20), True, 0) # hint-NOP / prefetch
|
||||
_fill2(range(0x20, 0x25), True, 0)
|
||||
_fill2(range(0x28, 0x30), True, 0)
|
||||
_fill2(range(0x30, 0x38), False, 0)
|
||||
_fill2(range(0x40, 0x50), True, 0) # CMOVcc
|
||||
_fill2(range(0x50, 0x70), True, 0) # SSE/MMX
|
||||
_TWO[0x70] = (True, "b")
|
||||
_fill2(range(0x71, 0x74), True, "b")
|
||||
_fill2(range(0x74, 0x77), True, 0)
|
||||
_TWO[0x77] = (False, 0) # EMMS
|
||||
_fill2(range(0x78, 0x80), True, 0)
|
||||
_fill2(range(0x80, 0x90), False, "z") # Jcc rel32
|
||||
_fill2(range(0x90, 0xA0), True, 0) # SETcc
|
||||
_fill2([0xA0, 0xA1, 0xA2], False, 0)
|
||||
_TWO[0xA3] = (True, 0) # BT
|
||||
_TWO[0xA4] = (True, "b") # SHLD ib
|
||||
_TWO[0xA5] = (True, 0) # SHLD CL
|
||||
_fill2([0xA8, 0xA9, 0xAA], False, 0)
|
||||
_TWO[0xAB] = (True, 0) # BTS
|
||||
_TWO[0xAC] = (True, "b") # SHRD ib
|
||||
_fill2([0xAD, 0xAE, 0xAF], True, 0)
|
||||
_fill2(range(0xB0, 0xBA), True, 0)
|
||||
_TWO[0xBA] = (True, "b") # group8 BT/BTS/BTR/BTC ib
|
||||
_fill2(range(0xBB, 0xC0), True, 0)
|
||||
_fill2([0xC0, 0xC1], True, 0) # XADD
|
||||
_TWO[0xC2] = (True, "b") # CMPPS
|
||||
_TWO[0xC3] = (True, 0) # MOVNTI
|
||||
_fill2([0xC4, 0xC5, 0xC6], True, "b")
|
||||
_TWO[0xC7] = (True, 0) # group9 CMPXCHG8B
|
||||
_fill2(range(0xC8, 0xD0), False, 0) # BSWAP
|
||||
_fill2(range(0xD0, 0x100), True, 0) # MMX/SSE bulk
|
||||
|
||||
# minimal mnemonics -- enough to read a report, not a full disassembler
|
||||
_ALU = ["add", "or", "adc", "sbb", "and", "sub", "xor", "cmp"]
|
||||
_G1 = _ALU
|
||||
_G5 = ["inc", "dec", "call", "callf", "jmp", "jmpf", "push", "?"]
|
||||
_G3 = ["test", "test", "not", "neg", "mul", "imul", "div", "idiv"]
|
||||
_SHIFT = ["rol", "ror", "rcl", "rcr", "shl", "shr", "shl", "sar"]
|
||||
|
||||
|
||||
def mnemonic(op2, op, reg):
|
||||
"""Best-effort mnemonic. op2 True => the opcode was 0F-escaped."""
|
||||
if op2:
|
||||
if 0x10 <= op <= 0x17 or 0x28 <= op <= 0x2F or 0x51 <= op <= 0x5F:
|
||||
return "sse"
|
||||
if 0x40 <= op <= 0x4F:
|
||||
return "cmov"
|
||||
if 0x90 <= op <= 0x9F:
|
||||
return "setcc"
|
||||
if op in (0xB6, 0xB7):
|
||||
return "movzx"
|
||||
if op in (0xBE, 0xBF):
|
||||
return "movsx"
|
||||
if op == 0xAF:
|
||||
return "imul"
|
||||
if op in (0x6E, 0x6F, 0x7E, 0x7F, 0xD6):
|
||||
return "movq/movd"
|
||||
return f"0f{op:02x}"
|
||||
if op < 0x40 and (op & 7) < 6:
|
||||
return _ALU[op >> 3]
|
||||
if op in (0x88, 0x89, 0x8A, 0x8B, 0xC6, 0xC7):
|
||||
return "mov"
|
||||
if op == 0x8D:
|
||||
return "lea"
|
||||
if op in (0x84, 0x85):
|
||||
return "test"
|
||||
if op in (0x86, 0x87):
|
||||
return "xchg"
|
||||
if op in (0x80, 0x81, 0x82, 0x83):
|
||||
return _G1[reg]
|
||||
if op in (0xC0, 0xC1, 0xD0, 0xD1, 0xD2, 0xD3):
|
||||
return _SHIFT[reg]
|
||||
if op in (0xF6, 0xF7):
|
||||
return _G3[reg]
|
||||
if op == 0xFF:
|
||||
return _G5[reg]
|
||||
if op == 0xFE:
|
||||
return ["inc", "dec"][reg] if reg < 2 else "?"
|
||||
if op == 0x8F:
|
||||
return "pop"
|
||||
if op in (0x69, 0x6B):
|
||||
return "imul"
|
||||
if 0xD8 <= op <= 0xDF:
|
||||
return "x87"
|
||||
if op == 0x62:
|
||||
return "bound"
|
||||
return f"{op:02x}"
|
||||
|
||||
|
||||
class Desync(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def decode(buf, i, end):
|
||||
"""Decode one instruction at buf[i]. Returns (length, info|None).
|
||||
|
||||
info = dict(mnem, base, index, scale, disp, dispsize, modrm_reg, is_lea)
|
||||
for instructions with a memory operand carrying a displacement; None
|
||||
otherwise. Raises Desync on an unknown/invalid opcode.
|
||||
"""
|
||||
start = i
|
||||
opsize66 = False
|
||||
addr67 = False
|
||||
while i < end:
|
||||
b = buf[i]
|
||||
if b == 0x66:
|
||||
opsize66 = True
|
||||
i += 1
|
||||
elif b == 0x67:
|
||||
addr67 = True
|
||||
i += 1
|
||||
elif b in (0xF0, 0xF2, 0xF3, 0x2E, 0x36, 0x3E, 0x26, 0x64, 0x65):
|
||||
i += 1
|
||||
else:
|
||||
break
|
||||
else:
|
||||
raise Desync("prefix run to end")
|
||||
if i - start > 14:
|
||||
raise Desync("prefix flood")
|
||||
|
||||
op = buf[i]
|
||||
i += 1
|
||||
op2 = op3 = False
|
||||
if op == 0x0F:
|
||||
if i >= end:
|
||||
raise Desync("truncated 0f")
|
||||
op = buf[i]
|
||||
i += 1
|
||||
op2 = True
|
||||
if op in (0x38, 0x3A):
|
||||
three_imm = "b" if op == 0x3A else 0
|
||||
if i >= end:
|
||||
raise Desync("truncated 0f3x")
|
||||
op = buf[i]
|
||||
i += 1
|
||||
op3 = True
|
||||
has_modrm, imm = True, three_imm
|
||||
else:
|
||||
ent = _TWO.get(op)
|
||||
if ent is None:
|
||||
raise Desync(f"unknown 0f{op:02x}")
|
||||
has_modrm, imm = ent
|
||||
else:
|
||||
ent = _ONE.get(op)
|
||||
if ent is None:
|
||||
raise Desync(f"unknown {op:02x}")
|
||||
has_modrm, imm = ent
|
||||
|
||||
info = None
|
||||
if has_modrm:
|
||||
if i >= end:
|
||||
raise Desync("truncated modrm")
|
||||
modrm = buf[i]
|
||||
i += 1
|
||||
mod = modrm >> 6
|
||||
reg = (modrm >> 3) & 7
|
||||
rm = modrm & 7
|
||||
if imm == "g6":
|
||||
imm = "b" if reg < 2 else 0
|
||||
elif imm == "g7":
|
||||
imm = "z" if reg < 2 else 0
|
||||
|
||||
if mod != 3:
|
||||
if addr67:
|
||||
# 16-bit ModRM: different table entirely. Rare in MSVC code and
|
||||
# never the member-address idiom -- skip rather than mis-decode.
|
||||
raise Desync("16-bit addressing (0x67)")
|
||||
base = index = None
|
||||
scale = 1
|
||||
if rm == 4:
|
||||
if i >= end:
|
||||
raise Desync("truncated sib")
|
||||
sib = buf[i]
|
||||
i += 1
|
||||
scale = 1 << (sib >> 6)
|
||||
idx = (sib >> 3) & 7
|
||||
bse = sib & 7
|
||||
index = None if idx == 4 else R32[idx]
|
||||
if bse == 5 and mod == 0:
|
||||
base = None # disp32 absolute + index
|
||||
else:
|
||||
base = R32[bse]
|
||||
elif rm == 5 and mod == 0:
|
||||
base = None # disp32 absolute
|
||||
else:
|
||||
base = R32[rm]
|
||||
|
||||
disp = 0
|
||||
dispsize = 0
|
||||
if mod == 1:
|
||||
if i >= end:
|
||||
raise Desync("truncated disp8")
|
||||
disp = struct.unpack_from("<b", buf, i)[0] # SIGN-EXTENDED
|
||||
dispsize = 1
|
||||
i += 1
|
||||
elif mod == 2 or base is None:
|
||||
if i + 4 > end:
|
||||
raise Desync("truncated disp32")
|
||||
disp = struct.unpack_from("<i", buf, i)[0]
|
||||
dispsize = 4
|
||||
i += 4
|
||||
if dispsize:
|
||||
info = {
|
||||
"mnem": mnemonic(op2, op, reg), "base": base,
|
||||
"index": index, "scale": scale, "disp": disp,
|
||||
"dispsize": dispsize, "reg": R32[reg],
|
||||
"lea": (not op2 and op == 0x8D),
|
||||
}
|
||||
else:
|
||||
reg = 0
|
||||
|
||||
n = 0
|
||||
if imm == "b":
|
||||
n = 1
|
||||
elif imm == "w":
|
||||
n = 2
|
||||
elif imm == "d":
|
||||
n = 4
|
||||
elif imm == "z":
|
||||
n = 2 if opsize66 else 4
|
||||
elif imm == "p":
|
||||
n = 4 if opsize66 else 6
|
||||
elif imm == "a":
|
||||
n = 2 if addr67 else 4
|
||||
elif imm == "e":
|
||||
n = 3
|
||||
i += n
|
||||
if i > end:
|
||||
raise Desync("truncated imm")
|
||||
if i == start:
|
||||
raise Desync("zero length")
|
||||
return i - start, info
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- PE / IO
|
||||
def load_pe(path):
|
||||
"""Return (image_base, [(va_start, va_end, bytes, name)]) for exec sections."""
|
||||
data = open(path, "rb").read()
|
||||
pe = struct.unpack_from("<I", data, 0x3C)[0]
|
||||
assert data[pe:pe + 4] == b"PE\0\0", "not a PE"
|
||||
nsec = struct.unpack_from("<H", data, pe + 6)[0]
|
||||
optsz = struct.unpack_from("<H", data, pe + 20)[0]
|
||||
base = struct.unpack_from("<I", data, pe + 24 + 28)[0]
|
||||
secs = []
|
||||
off = pe + 24 + optsz
|
||||
for k in range(nsec):
|
||||
s = off + k * 40
|
||||
name = data[s:s + 8].rstrip(b"\0").decode("latin1")
|
||||
vsize, vaddr, rsize, raddr = struct.unpack_from("<IIII", data, s + 8)
|
||||
chars = struct.unpack_from("<I", data, s + 36)[0]
|
||||
if not (chars & 0x20000000): # IMAGE_SCN_MEM_EXECUTE
|
||||
continue
|
||||
n = min(vsize, rsize) if vsize else rsize
|
||||
secs.append((base + vaddr, base + vaddr + n, data[raddr:raddr + n], name))
|
||||
return base, secs
|
||||
|
||||
|
||||
def load_funcs():
|
||||
with open(FUNCS) as fh:
|
||||
raw = json.load(fh)
|
||||
fl = sorted((int(a, 16), n, sz) for a, (n, sz) in raw.items())
|
||||
return fl
|
||||
|
||||
|
||||
# --------------------------------------------------------------------- build
|
||||
def build():
|
||||
_, secs = load_pe(EXE)
|
||||
funcs = load_funcs()
|
||||
starts = [f[0] for f in funcs]
|
||||
sites = [] # [va, disp, base, index, scale, mnem, lea, funcidx, hex]
|
||||
covered = 0
|
||||
total_code = sum(e - s for s, e, _, _ in secs)
|
||||
desyncs = 0
|
||||
decoded_ins = 0
|
||||
|
||||
for fi, (fva, fname, fsz) in enumerate(funcs):
|
||||
sec = next((s for s in secs if s[0] <= fva < s[1]), None)
|
||||
if sec is None or fsz <= 0:
|
||||
continue
|
||||
sva, eva, buf, _ = sec
|
||||
# Sweep to the NEXT function start, not fva+fsz: Ghidra's sizeInBytes
|
||||
# understates ~11% of bodies (it clips valid epilogues mid-instruction).
|
||||
# Sites past fsz are still attributed to this function but flagged, so
|
||||
# a caller can tell body-proper from possible inter-function padding.
|
||||
j = bisect.bisect_right(starts, fva)
|
||||
limit = min(starts[j] if j < len(starts) else eva, eva)
|
||||
body_end = fva + fsz
|
||||
i = fva - sva
|
||||
end = limit - sva
|
||||
while i < end:
|
||||
try:
|
||||
ln, info = decode(buf, i, end)
|
||||
except Desync:
|
||||
desyncs += 1
|
||||
break
|
||||
decoded_ins += 1
|
||||
if info and info["base"] is not None and info["disp"] != 0:
|
||||
va = sva + i
|
||||
sites.append([va, info["disp"], info["base"], info["index"],
|
||||
info["scale"], info["mnem"], info["lea"], fi,
|
||||
buf[i:i + min(ln, 10)].hex(),
|
||||
1 if va < body_end else 0, info["reg"]])
|
||||
i += ln
|
||||
covered += max(0, i - (fva - sva))
|
||||
|
||||
idx = {}
|
||||
for k, s in enumerate(sites):
|
||||
idx.setdefault(str(s[1]), []).append(k)
|
||||
out = {
|
||||
"sites": sites,
|
||||
"byDisp": idx,
|
||||
"funcs": [[f[0], f[1], f[2]] for f in funcs],
|
||||
"stats": {"functions": len(funcs), "instructions": decoded_ins,
|
||||
"sites": len(sites), "desyncs": desyncs,
|
||||
"bytesCovered": covered, "codeBytes": total_code},
|
||||
}
|
||||
with open(INDEX, "w") as fh:
|
||||
json.dump(out, fh)
|
||||
st = out["stats"]
|
||||
print(f"functions swept : {st['functions']}")
|
||||
print(f"instructions : {st['instructions']}")
|
||||
print(f"disp sites : {st['sites']}")
|
||||
print(f"desyncs : {st['desyncs']} "
|
||||
f"({100.0 * st['desyncs'] / st['functions']:.2f}% of functions)")
|
||||
print(f"code coverage : {st['bytesCovered']}/{st['codeBytes']} "
|
||||
f"({100.0 * st['bytesCovered'] / st['codeBytes']:.1f}%)")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------- query
|
||||
def load_index():
|
||||
if not os.path.exists(INDEX):
|
||||
sys.exit("no index; run: uv run python3 tools/x86disp.py build")
|
||||
with open(INDEX) as fh:
|
||||
return json.load(fh)
|
||||
|
||||
|
||||
THIS_REGS = ("ecx", "esi", "edi", "ebx") # typical `this` carriers in MSVC
|
||||
|
||||
|
||||
def fmt(site, funcs):
|
||||
va, disp, base, index, scale, mnem, lea, fi, hx, inbody, dst = site
|
||||
fva, fname, _ = funcs[fi]
|
||||
ea = f"[{base}"
|
||||
if index:
|
||||
ea += f"+{index}*{scale}"
|
||||
ea += f"{'+' if disp >= 0 else '-'}0x{abs(disp):x}]"
|
||||
txt = f"{mnem} {dst},{ea}" if lea else f"{mnem} {ea}"
|
||||
return (f" 0x{va:08x} {txt:<32} "
|
||||
f"{fname}+0x{va - fva:x}{'' if inbody else ' [past-body]'} ({hx})")
|
||||
|
||||
|
||||
def query(argv):
|
||||
want = int(argv[0], 0)
|
||||
only_lea = "--lea" in argv
|
||||
only_this = "--this" in argv
|
||||
show_all = "--all" in argv
|
||||
d = load_index()
|
||||
funcs = d["funcs"]
|
||||
keys = d["byDisp"].get(str(want), [])
|
||||
hits = [d["sites"][k] for k in keys]
|
||||
if only_lea:
|
||||
hits = [h for h in hits if h[6]]
|
||||
if only_this:
|
||||
hits = [h for h in hits if h[2] in THIS_REGS]
|
||||
hits.sort(key=lambda h: h[0])
|
||||
|
||||
ranked, other = [], []
|
||||
for h in hits:
|
||||
(ranked if h[2] in THIS_REGS else other).append(h)
|
||||
print(f"displacement 0x{want:x} ({want}): {len(hits)} site(s)")
|
||||
print(f"\n== base is a likely `this` ({'/'.join(THIS_REGS)}): {len(ranked)} ==")
|
||||
byfn = {}
|
||||
for h in ranked:
|
||||
byfn.setdefault(h[7], []).append(h)
|
||||
for fi in sorted(byfn, key=lambda f: funcs[f][0]):
|
||||
print(f" {funcs[fi][1]} @0x{funcs[fi][0]:08x}")
|
||||
for h in byfn[fi]:
|
||||
print(fmt(h, funcs))
|
||||
if other:
|
||||
print(f"\n== other base regs (esp/ebp = locals, eax/edx = temps): {len(other)} ==")
|
||||
if show_all:
|
||||
for h in other:
|
||||
print(fmt(h, funcs))
|
||||
else:
|
||||
cnt = {}
|
||||
for h in other:
|
||||
cnt[h[2]] = cnt.get(h[2], 0) + 1
|
||||
print(" " + ", ".join(f"{k}:{v}" for k, v in sorted(cnt.items()))
|
||||
+ " (--all to list)")
|
||||
|
||||
|
||||
def func_dump(argv):
|
||||
fva = int(argv[0], 0)
|
||||
d = load_index()
|
||||
funcs = d["funcs"]
|
||||
fi = next((i for i, f in enumerate(funcs) if f[0] == fva), None)
|
||||
if fi is None:
|
||||
sys.exit(f"no function at 0x{fva:08x}")
|
||||
print(f"{funcs[fi][1]} @ 0x{fva:08x} size {funcs[fi][2]}")
|
||||
for s in d["sites"]:
|
||||
if s[7] == fi:
|
||||
print(fmt(s, funcs))
|
||||
|
||||
|
||||
def stats():
|
||||
d = load_index()
|
||||
st = d["stats"]
|
||||
for k, v in st.items():
|
||||
print(f"{k:16}: {v}")
|
||||
hist = {}
|
||||
for s in d["sites"]:
|
||||
if s[2] in THIS_REGS:
|
||||
hist[s[1]] = hist.get(s[1], 0) + 1
|
||||
print(f"\ndistinct displacements off ecx/esi/edi/ebx: {len(hist)}")
|
||||
top = sorted(hist.items(), key=lambda kv: -kv[1])[:15]
|
||||
print("most common: " + ", ".join(f"0x{k:x}({v})" for k, v in top))
|
||||
|
||||
|
||||
_STORE = {0x89, 0x01, 0x29, 0x39, 0x09, 0x21, 0x31, 0x19, 0x11,
|
||||
0x88, 0x00, 0x08, 0x10, 0x18, 0x20, 0x28, 0x30, 0x38}
|
||||
|
||||
|
||||
def _reg_form(raw):
|
||||
"""Render the mod=3 (register-to-register) forms, which are exactly the
|
||||
ones that carry the arithmetic you care about when recovering a struct
|
||||
stride: `imul ecx,ecx,0x2c`, `sar edx,3`, `sub ecx,edi`."""
|
||||
p = 0
|
||||
while p < len(raw) and raw[p] in (0x66, 0x67, 0xF0, 0xF2, 0xF3,
|
||||
0x2E, 0x36, 0x3E, 0x26, 0x64, 0x65):
|
||||
p += 1
|
||||
op = raw[p]
|
||||
if op == 0x0F or p + 1 >= len(raw):
|
||||
return mnemonic(op == 0x0F, raw[p + 1] if op == 0x0F else op, 0)
|
||||
m = raw[p + 1]
|
||||
mod, reg, rm = m >> 6, (m >> 3) & 7, m & 7
|
||||
if mod != 3:
|
||||
# memory operand: render it and show which register is the other half,
|
||||
# so a load reads `mov ecx,[edi+4]` not a bare `mov`.
|
||||
q = p + 2
|
||||
ea = "["
|
||||
if rm == 4:
|
||||
sib = raw[q]; q += 1
|
||||
bse, ix = sib & 7, (sib >> 3) & 7
|
||||
ea += ("" if (bse == 5 and mod == 0) else R32[bse])
|
||||
if ix != 4:
|
||||
ea += f"+{R32[ix]}*{1 << (sib >> 6)}"
|
||||
elif not (rm == 5 and mod == 0):
|
||||
ea += R32[rm]
|
||||
if mod == 1 and q < len(raw):
|
||||
dv = struct.unpack_from("<b", raw, q)[0]
|
||||
ea += f"{'+' if dv >= 0 else '-'}0x{abs(dv):x}"
|
||||
elif mod == 2 and q + 4 <= len(raw):
|
||||
dv = struct.unpack_from("<i", raw, q)[0]
|
||||
ea += f"{'+' if dv >= 0 else '-'}0x{abs(dv):x}"
|
||||
ea += "]"
|
||||
nm = mnemonic(False, op, reg)
|
||||
if op in _STORE:
|
||||
return f"{nm} {ea},{R32[reg]}"
|
||||
if op in (0x80, 0x81, 0x83, 0xC6, 0xC7, 0xF6, 0xF7, 0xFF, 0xFE,
|
||||
0xC0, 0xC1, 0xD0, 0xD1, 0xD2, 0xD3) or op >= 0xD8:
|
||||
return f"{nm} {ea}"
|
||||
return f"{nm} {R32[reg]},{ea}"
|
||||
r, x = R32[reg], R32[rm]
|
||||
if op in (0x89, 0x01, 0x29, 0x39, 0x09, 0x21, 0x31, 0x19, 0x11, 0x85, 0x87):
|
||||
a, b = x, r # Ev,Gv -> dst is rm
|
||||
else:
|
||||
a, b = r, x # Gv,Ev -> dst is reg
|
||||
nm = mnemonic(False, op, reg)
|
||||
if op in (0xC0, 0xC1): # shift group, imm8
|
||||
return f"{_SHIFT[reg]} {x},{raw[p + 2]}"
|
||||
if op in (0xD1, 0xD3):
|
||||
return f"{_SHIFT[reg]} {x},{'cl' if op == 0xD3 else '1'}"
|
||||
if op == 0x6B: # imul r,rm,imm8 <- the stride!
|
||||
return f"imul {r},{x},0x{struct.unpack_from('<b', raw, p + 2)[0] & 0xff:x}"
|
||||
if op == 0x69:
|
||||
return f"imul {r},{x},0x{struct.unpack_from('<I', raw, p + 2)[0]:x}"
|
||||
if op in (0x83, 0x81, 0x80):
|
||||
v = (raw[p + 2] if op != 0x81
|
||||
else struct.unpack_from("<I", raw, p + 2)[0])
|
||||
return f"{_G1[reg]} {x},0x{v:x}"
|
||||
if op in (0xF7, 0xF6):
|
||||
return f"{_G3[reg]} {x}"
|
||||
if op == 0xFF:
|
||||
return f"{_G5[reg]} {x}"
|
||||
return f"{nm} {a},{b}"
|
||||
|
||||
|
||||
def dis(argv):
|
||||
"""Linear disassembly window: `dis <va> [count]`. Length-accurate; the
|
||||
mnemonics are minimal but call/jmp targets are resolved to function names,
|
||||
which is what you actually need to follow a `lea this,[obj+N]; call` pair."""
|
||||
va = int(argv[0], 0)
|
||||
n = int(argv[1]) if len(argv) > 1 else 24
|
||||
_, secs = load_pe(EXE)
|
||||
d = load_index()
|
||||
fmap = {f[0]: f[1] for f in d["funcs"]}
|
||||
sec = next((s for s in secs if s[0] <= va < s[1]), None)
|
||||
if sec is None:
|
||||
sys.exit("address not in an executable section")
|
||||
sva, eva, buf, _ = sec
|
||||
i = va - sva
|
||||
for _ in range(n):
|
||||
try:
|
||||
ln, info = decode(buf, i, len(buf))
|
||||
except Desync as e:
|
||||
print(f"0x{sva + i:08x} <desync: {e}>")
|
||||
return
|
||||
raw = buf[i:i + ln]
|
||||
cur = sva + i
|
||||
txt = ""
|
||||
# resolve the two rel32 forms and the rel8 jumps by hand
|
||||
if raw[0] in (0xE8, 0xE9) and ln == 5:
|
||||
tgt = cur + 5 + struct.unpack_from("<i", raw, 1)[0]
|
||||
txt = (f"{'call' if raw[0] == 0xE8 else 'jmp'} 0x{tgt:08x}"
|
||||
f" {fmap.get(tgt, '')}")
|
||||
elif raw[0] == 0x0F and 0x80 <= raw[1] <= 0x8F and ln == 6:
|
||||
tgt = cur + 6 + struct.unpack_from("<i", raw, 2)[0]
|
||||
txt = f"jcc 0x{tgt:08x}"
|
||||
elif raw[0] == 0xEB or 0x70 <= raw[0] <= 0x7F:
|
||||
tgt = cur + ln + struct.unpack_from("<b", raw, ln - 1)[0]
|
||||
txt = f"jmp/jcc 0x{tgt:08x}"
|
||||
elif 0x50 <= raw[0] <= 0x57:
|
||||
txt = f"push {R32[raw[0] - 0x50]}"
|
||||
elif 0x58 <= raw[0] <= 0x5F:
|
||||
txt = f"pop {R32[raw[0] - 0x58]}"
|
||||
elif raw[0] == 0x6A:
|
||||
txt = f"push 0x{raw[1]:x}"
|
||||
elif raw[0] == 0x68:
|
||||
txt = f"push 0x{struct.unpack_from('<I', raw, 1)[0]:x}"
|
||||
elif 0xB8 <= raw[0] <= 0xBF and ln == 5:
|
||||
txt = f"mov {R32[raw[0] - 0xB8]},0x{struct.unpack_from('<I', raw, 1)[0]:x}"
|
||||
elif raw[0] == 0xC3:
|
||||
txt = "ret"
|
||||
elif raw[0] == 0xC2:
|
||||
txt = f"ret 0x{struct.unpack_from('<H', raw, 1)[0]:x}"
|
||||
else:
|
||||
txt = _reg_form(raw)
|
||||
lbl = fmap.get(cur, "")
|
||||
print(f"0x{cur:08x} {raw.hex():<18} {txt:<44} {lbl}")
|
||||
i += ln
|
||||
|
||||
|
||||
def cohort(argv):
|
||||
"""Rank functions by how many offsets of a KNOWN class layout they touch.
|
||||
|
||||
This is the answer to the real precision problem. A bare query for 0x274
|
||||
returns ~99 sites and the tool cannot know which base register holds a
|
||||
ServerPlayer. But a function that touches 0x274 *and* 0x29c *and* 0x244
|
||||
*and* 0x254 is not doing that by coincidence -- those are ServerPlayer's
|
||||
members. Feed the offsets we have already pinned and the class's own
|
||||
methods float to the top.
|
||||
|
||||
uv run python3 tools/x86disp.py cohort 0x274 --known=0x29c,0x244,0x254,...
|
||||
|
||||
WARNING, learned the hard way on the ObservedTech hunt: use this as a
|
||||
RANKER, never as a hard filter. It finds fat *class methods* (the
|
||||
ServerPlayer serializer scores 50/50) but it actively hides narrow helpers.
|
||||
`RecordObservedTech` -- the actual append site -- touches only 0x274 and
|
||||
0x278 and nothing else on ServerPlayer, so every --min>=1 setting drops it.
|
||||
Read the full `query` output before trusting a cohort shortlist.
|
||||
"""
|
||||
want = int(argv[0], 0)
|
||||
known = []
|
||||
minhits = 2
|
||||
for a in argv[1:]:
|
||||
if a.startswith("--known="):
|
||||
known = [int(x, 0) for x in a[8:].split(",")]
|
||||
elif a.startswith("--min="):
|
||||
minhits = int(a[6:])
|
||||
if not known:
|
||||
sys.exit("need --known=<comma-separated offsets of the same class>")
|
||||
d = load_index()
|
||||
funcs = d["funcs"]
|
||||
kset = set(known)
|
||||
touched = {} # funcidx -> set(offset)
|
||||
target = {} # funcidx -> [sites at `want`]
|
||||
for s in d["sites"]:
|
||||
if s[2] not in THIS_REGS:
|
||||
continue
|
||||
if s[1] in kset:
|
||||
touched.setdefault(s[7], set()).add(s[1])
|
||||
if s[1] == want:
|
||||
target.setdefault(s[7], []).append(s)
|
||||
rows = []
|
||||
for fi, sites in target.items():
|
||||
hits = touched.get(fi, set()) - {want}
|
||||
if len(hits) >= minhits:
|
||||
rows.append((len(hits), fi, sites, hits))
|
||||
rows.sort(key=lambda r: -r[0])
|
||||
print(f"functions touching 0x{want:x} AND >={minhits} other known offsets "
|
||||
f"of this class: {len(rows)} of {len(target)} candidates "
|
||||
f"({100.0 * len(rows) / max(1, len(target)):.0f}% kept)")
|
||||
for n, fi, sites, hits in rows:
|
||||
fva, fname, _ = funcs[fi]
|
||||
hx = ",".join(f"0x{h:x}" for h in sorted(hits))
|
||||
print(f"\n {fname} @0x{fva:08x} [{n} co-hits: {hx}]")
|
||||
for s in sites:
|
||||
print(fmt(s, funcs))
|
||||
|
||||
|
||||
# ------------------------------------------------- naive scan, for FP measure
|
||||
def brute(argv):
|
||||
"""Naive 'search the raw bytes' scan -- the thing you'd write without a
|
||||
decoder. Only exists so we can put a NUMBER on how wrong it is.
|
||||
|
||||
`--ops 8d` (default) = lea only. `--ops 8d,89,8b,01,03,39,3b` = the wider
|
||||
scan you would actually need, since a member is read/written far more often
|
||||
than its address is taken. The wider the opcode set and the smaller the
|
||||
displacement, the worse the naive method gets -- that is the point.
|
||||
"""
|
||||
want = int(argv[0], 0)
|
||||
ops = {0x8D}
|
||||
for a in argv[1:]:
|
||||
if a.startswith("--ops="):
|
||||
ops = {int(x, 16) for x in a[6:].split(",")}
|
||||
_, secs = load_pe(EXE)
|
||||
d = load_index()
|
||||
real = {s[0] for s in d["sites"] if s[1] == want}
|
||||
found = []
|
||||
d8 = want if -128 <= want <= 127 else None
|
||||
pat32 = struct.pack("<i", want)
|
||||
for sva, eva, buf, _ in secs:
|
||||
for i in range(len(buf) - 6):
|
||||
if buf[i] not in ops:
|
||||
continue
|
||||
m = buf[i + 1]
|
||||
mod, rm = m >> 6, m & 7
|
||||
if mod == 3 or mod == 0:
|
||||
continue
|
||||
k = i + 2 + (1 if rm == 4 else 0)
|
||||
if mod == 1 and d8 is not None:
|
||||
if struct.unpack_from("<b", buf, k)[0] == want:
|
||||
found.append(sva + i)
|
||||
elif mod == 2 and buf[k:k + 4] == pat32:
|
||||
found.append(sva + i)
|
||||
fp = [a for a in found if a not in real]
|
||||
tag = ",".join(f"{o:02x}" for o in sorted(ops))
|
||||
print(f"naive byte-scan (opcodes {tag}) for disp 0x{want:x}: "
|
||||
f"{len(found)} candidate(s)")
|
||||
print(f" real instructions at that address : {len(found) - len(fp)}")
|
||||
print(f" NOT an instruction boundary : {len(fp)} "
|
||||
f"({100.0 * len(fp) / max(1, len(found)):.1f}% false positive)")
|
||||
missed = len(real) - (len(found) - len(fp))
|
||||
print(f" real sites the naive scan MISSED : {missed} of {len(real)}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) < 2:
|
||||
sys.exit(__doc__)
|
||||
cmd, rest = sys.argv[1], sys.argv[2:]
|
||||
{"build": lambda a: build(), "query": query, "func": func_dump,
|
||||
"stats": lambda a: stats(), "brute": brute, "cohort": cohort,
|
||||
"dis": dis}[cmd](rest)
|
||||
Loading…
Add table
Reference in a new issue