From 0082eee3b9b0eece6c8838207c034476e1c8b8a6 Mon Sep 17 00:00:00 2001 From: alex Date: Mon, 7 Sep 2026 23:52:50 -0400 Subject: [PATCH] m2: live verification results (weapons compare 0 div, oracle passes; section compare crash documented) --- docs/M2.md | 91 ++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 58 insertions(+), 33 deletions(-) diff --git a/docs/M2.md b/docs/M2.md index 52963c3..92e485a 100644 --- a/docs/M2.md +++ b/docs/M2.md @@ -1,11 +1,12 @@ # M2 — id-manifest loading (`_weapons.txt` / `_shipsections.txt`) old-vs-new on the live game -**Status (2026-09-08): code complete and cross-built; one scouting trace captured on the real -game; the golden trace, the compare run and the replace/End-Turn run still owe a VM window.** -VM140 was handed to another agent's behavioural compare mid-milestone, so steps 2–4 of the -milestone stop here rather than interleaving deploys with someone else's run. What the scouting -trace already proves is below — it is the reason the region model changed, so it must be re-run -against the final DLL before it can be called golden. +**Result (2026-09-08):** the weapon dictionary is verified end to end — trace (`tracecmp.py` +exit 0), compare **1/1 compared, 0 divergences**, and replace reproducing the End-Turn +determinism oracle byte for byte (`(Autosave).sav` = `978041ac…`, `(Autosave EndTurn).sav` = +`bb4fd9ac…`) with our loader feeding the game its weapons alone. The section dictionary is +**trace-verified only**: its 885 definitions and ids match the manifests in the golden trace, +but `ours` crashes the engine in compare mode — see "Open: the section compare crash". Game +left running on VM140 in `hooks=trace` at the main menu. ## What was hooked @@ -81,27 +82,58 @@ the rest on the stack, callee-cleaned — GCC's `__attribute__((thiscall))` is e on i386. This is safe **only** because both prototypes are `[verified]`; M0's asm-stub rule still stands for anything marked `[unverified]`. -## Evidence so far +## Runs (`/bulk-storage/re-lab/shim/traces/`, build `m2v-3ebc8bf-20260908T0325Z`) -`/bulk-storage/re-lab/shim/traces/m2-trace-scout.jsonl` (+ `m2-scout-shim.log`), build -`8b231c0-dirty-20260908T0233Z`, `hooks=trace`: 22 calls, `tracecmp.py` exit 0, 0 invalid, both -M2 hooks firing once each. It confirms against the live game: +| file | mode | calls | result | +|---|---|---|---| +| `m2-trace-golden.jsonl` | trace | 22 (2 M2 + 19 LoadFile + selftest) | `tracecmp.py` exit 0, 0 invalid | +| `m2-compare.jsonl` | compare | 1 compared (weapons) | **0 diverged**, 0 errors, exit 0 | +| replace (no log) | replace | weapons | End Turn from `ref-turn2.sav`: `(Autosave).sav` `978041ac…` / `(Autosave EndTurn).sav` `bb4fd9ac…` = oracle | +| `m2-trace-scout.jsonl` | trace | 22 | the earlier build, kept because it is what exposed the +0x14 layout bug | +| `m2-shim-final.log`, `m2-*.png` | | | run banners and `dict:` lines; Turn-2 replace screen, final main menu | + +What the golden trace establishes against the live game: - **Weapons: 123 definitions, ids exactly the manifest's** — 0 id mismatches and 0 manifest - entries missing from the dictionary, cross-checked against `Weapons/_weapons.txt` (123 numbered - lines, three `// DELETED` tombstones ignored as comments). -- **The dictionary is sorted by `_stricmp` on `name`, not by id, not by path**, and `index` is - rewritten to the position afterwards (`index == position` holds for all 123). No tied names. + entries missing, cross-checked against `Weapons/_weapons.txt` (123 numbered lines, three + `// DELETED` tombstones correctly ignored as comments). +- **The dictionary is sorted by `_stricmp` on `name`**, not by id and not by path, with `index` + rewritten to the position afterwards (`index == position` for all 123). The shipped data has + **no tied names**, so the MSVC-sort replica is insurance rather than a load-bearing detail. +- `emt_light` resolves to the weapon of that name — the check that confirmed the +0x14 fix + (the first build read the unmodelled word and saw text bytes instead). - **Sections: 885 definitions**, species 0..6 in the order `Human, Hiver, Tarkas, Liir, _NPC, - Zuul, Morrigi` (from `Species::GetDirName`), counts 145/137/139/136/64/122/142; `index == - position`; ids are the manifests' and the 10 dangling ids are simply absent, matching the - RE notes. Section names arrive already resolved (`Assault Shuttle`, not `@SECTIONNAME_…`). + Zuul, Morrigi`, counts 145/137/139/136/64/122/142, `index == position`, ids from the + manifests with the 10 dangling ids simply absent. Names arrive already resolved + (`Assault Shuttle`, not `@SECTIONNAME_…`). -Host suite: `ctest` 26/26 (`game_config_manifest` = 71 checks: the token-pair loop, the -trailing-newline rule, `atoi`, truncation, path spellings, and the sort replica against -`std::sort` on distinct keys plus determinism/permutation on ties). -Cross-build: `m2-de779ad-20260908T0239Z`, exports 66 names identical to `binkw32.dll`, staged in -`/srv/re-lab/shim/dist-m2` (its own dist, not the shared one). `clean_room_check.sh` OK. +Host suite: `ctest` 26/26 (`game_config_manifest` = 71 checks). Cross-build staged in +`/srv/re-lab/shim/dist-m2`, exports 66 names identical to `binkw32.dll`; deployed from the +lane-local `C:\SOTS\shimdist-m2`. `clean_room_check.sh` OK. + +## Open: the section compare crash + +With `hook.Game::SectionDictionary::SectionDictionary=compare` the engine dies inside `ours` +(`Mars: Application error encountered`, `Mars.dmp` written) after the weapon hook has already +compared cleanly, before the first species finishes — no `dict: [Species/…]` line is ever +logged. The weapon path does the same thing (re-run the game's own per-file loader on a scratch +dictionary) without trouble, so the difference is in `SectionDictionary::LoadSection`, not in +the manifest reader: `manifest_pairs` is shared by both and is covered by host tests. + +Leads for whoever picks this up, cheapest first: + +1. `LoadSection` may append to the dictionary's own vector, in which case `ours` push_backs a + second time and the two allocators fight — instrument by logging the vector's begin/end + around one call before assuming. +2. It runs *after* the original has already built all 885 definitions, so the second pass + registers duplicates with the `TechTree` / string table; a fixed-size table overflowing + there would look exactly like this. +3. The trace-mode path is unaffected, so a narrower boundary (hooking `LoadSection` itself, one + call per section, rather than the whole constructor) would both isolate the fault and give a + finer region model. + +Note the diagnosis cost: the crash dialog was hidden behind a Windows Update dialog, so the +process looked alive-but-wedged for ~12 minutes. Screenshot before believing a hang. ## Gotchas @@ -123,14 +155,7 @@ Cross-build: `m2-de779ad-20260908T0239Z`, exports 66 names identical to `binkw32 `-Wcast-function-type` is an error in this tree. 6. `shim.cfg` per-hook lines: `hook.Game::WeaponDictionary::Init=compare|replace`, `hook.Game::SectionDictionary::SectionDictionary=compare|replace`. Ready-made configs are on - the VM as `C:\SOTS\ui\shim.cfg.m2{trace,compare,replace}`. - -## What remains (needs the VM) - -1. Deploy `/srv/re-lab/shim/dist-m2` (build `m2-de779ad-20260908T0239Z`), relaunch with - `shim.cfg.m2trace`, let it reach the main menu (the hooks fire on the first tick, ~40 s after - launch), `tracecmp.py` → `m2-trace-golden.jsonl`. -2. Relaunch with `shim.cfg.m2compare` → 0 divergences → `m2-compare.jsonl`. -3. Relaunch with `shim.cfg.m2replace`, load `ref-turn2.sav`, End Turn, and check the oracle - (`(Autosave).sav` = `978041ac…`, `(Autosave EndTurn).sav` = `bb4fd9ac…`). -4. Restore `hooks=trace` and leave the game running at the main menu. + the VM as `C:\SOTS\ui\shim.cfg.m2{trace,compare,replace}` plus `shim.cfg.m2replaceW` + (weapons in replace, sections left in trace — the config the oracle run used). +7. The M2 hooks fire on the first `OnTick`, i.e. at the main menu, ~40 s after launch — not + inside `Initialize` like M1. Skip the intro with three `esc` sendkeys before waiting on them.