m2: live verification results (weapons compare 0 div, oracle passes; section compare crash documented)

docs/M2.md

@@ -1,11 +1,12 @@
 # M2 — id-manifest loading (`_weapons.txt` / `_shipsections.txt`) old-vs-new on the live game
 
-**Status (2026-09-08): code complete and cross-built; one scouting trace captured on the real
+**Result (2026-09-08):** the weapon dictionary is verified end to end — trace (`tracecmp.py`
-game; the golden trace, the compare run and the replace/End-Turn run still owe a VM window.**
+exit 0), compare **1/1 compared, 0 divergences**, and replace reproducing the End-Turn
-VM140 was handed to another agent's behavioural compare mid-milestone, so steps 2–4 of the
+determinism oracle byte for byte (`(Autosave).sav` = `978041ac…`, `(Autosave EndTurn).sav` =
-milestone stop here rather than interleaving deploys with someone else's run. What the scouting
+`bb4fd9ac…`) with our loader feeding the game its weapons alone. The section dictionary is
-trace already proves is below — it is the reason the region model changed, so it must be re-run
+**trace-verified only**: its 885 definitions and ids match the manifests in the golden trace,
-against the final DLL before it can be called golden.
+but `ours` crashes the engine in compare mode — see "Open: the section compare crash". Game
+left running on VM140 in `hooks=trace` at the main menu.
 
 ## What was hooked
 
@@ -81,27 +82,58 @@ the rest on the stack, callee-cleaned — GCC's `__attribute__((thiscall))` is e
 on i386. This is safe **only** because both prototypes are `[verified]`; M0's asm-stub rule
 still stands for anything marked `[unverified]`.
 
-## Evidence so far
+## Runs (`/bulk-storage/re-lab/shim/traces/`, build `m2v-3ebc8bf-20260908T0325Z`)
 
-`/bulk-storage/re-lab/shim/traces/m2-trace-scout.jsonl` (+ `m2-scout-shim.log`), build
+| file | mode | calls | result |
-`8b231c0-dirty-20260908T0233Z`, `hooks=trace`: 22 calls, `tracecmp.py` exit 0, 0 invalid, both
+|---|---|---|---|
-M2 hooks firing once each. It confirms against the live game:
+| `m2-trace-golden.jsonl` | trace | 22 (2 M2 + 19 LoadFile + selftest) | `tracecmp.py` exit 0, 0 invalid |
+| `m2-compare.jsonl` | compare | 1 compared (weapons) | **0 diverged**, 0 errors, exit 0 |
+| replace (no log) | replace | weapons | End Turn from `ref-turn2.sav`: `(Autosave).sav` `978041ac…` / `(Autosave EndTurn).sav` `bb4fd9ac…` = oracle |
+| `m2-trace-scout.jsonl` | trace | 22 | the earlier build, kept because it is what exposed the +0x14 layout bug |
+| `m2-shim-final.log`, `m2-*.png` | | | run banners and `dict:` lines; Turn-2 replace screen, final main menu |
+
+What the golden trace establishes against the live game:
 
 - **Weapons: 123 definitions, ids exactly the manifest's** — 0 id mismatches and 0 manifest
-  entries missing from the dictionary, cross-checked against `Weapons/_weapons.txt` (123 numbered
+  entries missing, cross-checked against `Weapons/_weapons.txt` (123 numbered lines, three
-  lines, three `// DELETED` tombstones ignored as comments).
+  `// DELETED` tombstones correctly ignored as comments).
-- **The dictionary is sorted by `_stricmp` on `name`, not by id, not by path**, and `index` is
+- **The dictionary is sorted by `_stricmp` on `name`**, not by id and not by path, with `index`
-  rewritten to the position afterwards (`index == position` holds for all 123). No tied names.
+  rewritten to the position afterwards (`index == position` for all 123). The shipped data has
+  **no tied names**, so the MSVC-sort replica is insurance rather than a load-bearing detail.
+- `emt_light` resolves to the weapon of that name — the check that confirmed the +0x14 fix
+  (the first build read the unmodelled word and saw text bytes instead).
 - **Sections: 885 definitions**, species 0..6 in the order `Human, Hiver, Tarkas, Liir, _NPC,
-  Zuul, Morrigi` (from `Species::GetDirName`), counts 145/137/139/136/64/122/142; `index ==
+  Zuul, Morrigi`, counts 145/137/139/136/64/122/142, `index == position`, ids from the
-  position`; ids are the manifests' and the 10 dangling ids are simply absent, matching the
+  manifests with the 10 dangling ids simply absent. Names arrive already resolved
-  RE notes. Section names arrive already resolved (`Assault Shuttle`, not `@SECTIONNAME_…`).
+  (`Assault Shuttle`, not `@SECTIONNAME_…`).
 
-Host suite: `ctest` 26/26 (`game_config_manifest` = 71 checks: the token-pair loop, the
+Host suite: `ctest` 26/26 (`game_config_manifest` = 71 checks). Cross-build staged in
-trailing-newline rule, `atoi`, truncation, path spellings, and the sort replica against
+`/srv/re-lab/shim/dist-m2`, exports 66 names identical to `binkw32.dll`; deployed from the
-`std::sort` on distinct keys plus determinism/permutation on ties).
+lane-local `C:\SOTS\shimdist-m2`. `clean_room_check.sh` OK.
-Cross-build: `m2-de779ad-20260908T0239Z`, exports 66 names identical to `binkw32.dll`, staged in
+
-`/srv/re-lab/shim/dist-m2` (its own dist, not the shared one). `clean_room_check.sh` OK.
+## Open: the section compare crash
+
+With `hook.Game::SectionDictionary::SectionDictionary=compare` the engine dies inside `ours`
+(`Mars: Application error encountered`, `Mars.dmp` written) after the weapon hook has already
+compared cleanly, before the first species finishes — no `dict: [Species/…]` line is ever
+logged. The weapon path does the same thing (re-run the game's own per-file loader on a scratch
+dictionary) without trouble, so the difference is in `SectionDictionary::LoadSection`, not in
+the manifest reader: `manifest_pairs` is shared by both and is covered by host tests.
+
+Leads for whoever picks this up, cheapest first:
+
+1. `LoadSection` may append to the dictionary's own vector, in which case `ours` push_backs a
+   second time and the two allocators fight — instrument by logging the vector's begin/end
+   around one call before assuming.
+2. It runs *after* the original has already built all 885 definitions, so the second pass
+   registers duplicates with the `TechTree` / string table; a fixed-size table overflowing
+   there would look exactly like this.
+3. The trace-mode path is unaffected, so a narrower boundary (hooking `LoadSection` itself, one
+   call per section, rather than the whole constructor) would both isolate the fault and give a
+   finer region model.
+
+Note the diagnosis cost: the crash dialog was hidden behind a Windows Update dialog, so the
+process looked alive-but-wedged for ~12 minutes. Screenshot before believing a hang.
 
 ## Gotchas
 
@@ -123,14 +155,7 @@ Cross-build: `m2-de779ad-20260908T0239Z`, exports 66 names identical to `binkw32
    `-Wcast-function-type` is an error in this tree.
 6. `shim.cfg` per-hook lines: `hook.Game::WeaponDictionary::Init=compare|replace`,
    `hook.Game::SectionDictionary::SectionDictionary=compare|replace`. Ready-made configs are on
-   the VM as `C:\SOTS\ui\shim.cfg.m2{trace,compare,replace}`.
+   the VM as `C:\SOTS\ui\shim.cfg.m2{trace,compare,replace}` plus `shim.cfg.m2replaceW`
-
+   (weapons in replace, sections left in trace — the config the oracle run used).
-## What remains (needs the VM)
+7. The M2 hooks fire on the first `OnTick`, i.e. at the main menu, ~40 s after launch — not
-
+   inside `Initialize` like M1. Skip the intro with three `esc` sendkeys before waiting on them.
-1. Deploy `/srv/re-lab/shim/dist-m2` (build `m2-de779ad-20260908T0239Z`), relaunch with
-   `shim.cfg.m2trace`, let it reach the main menu (the hooks fire on the first tick, ~40 s after
-   launch), `tracecmp.py` → `m2-trace-golden.jsonl`.
-2. Relaunch with `shim.cfg.m2compare` → 0 divergences → `m2-compare.jsonl`.
-3. Relaunch with `shim.cfg.m2replace`, load `ref-turn2.sav`, End Turn, and check the oracle
-   (`(Autosave).sav` = `978041ac…`, `(Autosave EndTurn).sav` = `bb4fd9ac…`).
-4. Restore `hooks=trace` and leave the game running at the main menu.