# M2 — id-manifest loading (`_weapons.txt` / `_shipsections.txt`) old-vs-new on the live game

**Result (2026-09-08):** the weapon dictionary is verified end to end — trace (`tracecmp.py`
exit 0), compare **1/1 compared, 0 divergences**, and replace reproducing the End-Turn
determinism oracle byte for byte (`(Autosave).sav` = `978041ac…`, `(Autosave EndTurn).sav` =
`bb4fd9ac…`) with our loader feeding the game its weapons alone. The section dictionary is
**trace-verified only**: its 885 definitions and ids match the manifests in the golden trace,
but `ours` crashes the engine in compare mode — see "Open: the section compare crash". Game
left running on VM140 in `hooks=trace` at the main menu.

## What was hooked

Two functions, both `[verified]` `__thiscall` in `sots_addresses.h`, so they go through the
`Hook<Descriptor>` template rather than an asm stub (M0's stub caveat covers *unverified*
prototypes; see "thiscall through the template" below):

| hook name (record `hook`) | RVA | prototype |
|---|---|---|
| `Game::WeaponDictionary::Init` | 0x0019a4c0 | `void (WeaponDictionary*)` — `Weapons/_weapons.txt` |
| `Game::SectionDictionary::SectionDictionary` | 0x00176f40 | `SectionDictionary* (SectionDictionary*, TechTree*)` — `Species/<Race>/sections/_shipsections.txt` ×7 |

Both are called once, from `DemoApp::LoadGameData` on the first `OnTick` after start-up (not
inside `Initialize`, unlike M1). Sources: `src/shim/hooks/dictionaries.{h,cpp}`, installed from
`src/shim/main.cpp` next to the M1 hook.

There is **no registry object** to hook: the loaders drive `Script::ReadToken` themselves and
store the id on each definition. So the hookable boundary is the dictionary loader as a whole,
and the "manifest" is a token-pair stream inside it.

### Region model

The definitions do not exist when the hook is entered, so the declared region is the
**dictionary object itself** — 0x1c bytes (weapons) / 0x18 bytes (sections) — with a struct
describer that walks the vector at +8 and emits one entry per definition, in vector order:

| region | describer |
|---|---|
| `dict` (weapons, 0x1c) | `{unk0:i32, tree:ptr, n:u32, defs:[{index,id,path,name}], unk14:ptr, emt_light:str}` |
| `dict` (sections, 0x18) | `{cur:i32, tree:ptr, n:u32, defs:[{index,species,id,token}], unk14:ptr}` |

`id` (`WeaponDef+4` / `SectionDef+8`) and `index` (`+0`) are the words the original writes and
the point of the milestone; `path`/`name`/`token` are the name half of the id↔name assignment,
decoded from the MSVC `std::string`s at `WeaponDef+8`/`+0x40` and `SectionDef+0x17c`.
`emt_light` is emitted **by the weapon name it points at**, not as a pointer: in compare mode
ours builds its own definition objects, so the pointers necessarily differ while the weapon
they name must not. Every pointer chase in a describer is guarded by a `VirtualQuery` check —
the section dictionary is hooked at its *constructor*, so the "before" snapshot is whatever the
fresh heap block held (the scouting trace shows exactly that: `cur` = 71692504, an unreadable
vector, `invalid: true`).

Args: weapons `this` + the manifest path; sections `this`, `tree`, and the seven directory names
read from the game's own `Species::GetDirName`, so a golden log replays offline without the exe.

## `ours` — `src/game/config/manifest_loader.{h,cpp}` (lib `sots_game_config`)

- File bytes come from the game's own `gobio::ReadFile` (mount order / mods honoured), released
  with `RefCounted_Release`; host tests read plain text.
- `manifest_pairs()` reproduces the loaders' loop over `mars::parse::Script`: read a token, and
  **only if that read succeeded** `atoi` it into the running id; read the file-name token; a read
  that is not `Ok` ends the loop. Consequences, all tested: `// DELETED - n` and `// REMOVED - n`
  lines are ordinary comments and never appear; a non-numeric id token is `atoi`'s 0; tokens are
  truncated at 255 bytes (the loaders' 256-byte buffers); and **a manifest whose last line has no
  trailing newline loses its last entry** (the value token touches EOF, so it reads `AtEnd`).
  The shipped files end with CRLF, so nothing is lost in practice.
- `manifest_atoi()` is the CRT's: leading whitespace, optional sign, decimal digits, wrap modulo
  2^32, 0 when there is no digit.
- Path spelling is the loaders': `"Weapons/" + file`, `dir + "/" + file`.
- The pairs are handed to the game's own `LoadWeapon` / `LoadSection` (per-file parsing is M3
  scope); the weapon dictionary is then sorted and re-indexed, and `emt_light` looked up, as the
  original does after its loop. Definitions built during a compare call leak one allocation set
  (start-up only, like M1's long strings).
- `src/game/config/msvc_sort.h` replays MSVC 2010's `std::sort` (median-of-3/9 three-way
  introsort, insertion sort ≤ 32, heap-sort fallback) so that *ties* land where the original puts
  them. The shipped data turns out to have **no tied weapon names**, so any correct sort would
  do; the replica is kept because the tie order is otherwise unspecified and data can change.

## thiscall through the template

`CallConv::Thiscall` was added to `src/shim/trace/hook.h` (+ `SHIM_THISCALL` in `platform.h`):
`this` is the first `Args` element, passed in ECX by both the detour and the trampoline call,
the rest on the stack, callee-cleaned — GCC's `__attribute__((thiscall))` is exactly MSVC's ABI
on i386. This is safe **only** because both prototypes are `[verified]`; M0's asm-stub rule
still stands for anything marked `[unverified]`.

## Runs (`/bulk-storage/re-lab/shim/traces/`, build `m2v-3ebc8bf-20260908T0325Z`)

| file | mode | calls | result |
|---|---|---|---|
| `m2-trace-golden.jsonl` | trace | 22 (2 M2 + 19 LoadFile + selftest) | `tracecmp.py` exit 0, 0 invalid |
| `m2-compare.jsonl` | compare | 1 compared (weapons) | **0 diverged**, 0 errors, exit 0 |
| replace (no log) | replace | weapons | End Turn from `ref-turn2.sav`: `(Autosave).sav` `978041ac…` / `(Autosave EndTurn).sav` `bb4fd9ac…` = oracle |
| `m2-trace-scout.jsonl` | trace | 22 | the earlier build, kept because it is what exposed the +0x14 layout bug |
| `m2-shim-final.log`, `m2-*.png` | | | run banners and `dict:` lines; Turn-2 replace screen, final main menu |

What the golden trace establishes against the live game:

- **Weapons: 123 definitions, ids exactly the manifest's** — 0 id mismatches and 0 manifest
  entries missing, cross-checked against `Weapons/_weapons.txt` (123 numbered lines, three
  `// DELETED` tombstones correctly ignored as comments).
- **The dictionary is sorted by `_stricmp` on `name`**, not by id and not by path, with `index`
  rewritten to the position afterwards (`index == position` for all 123). The shipped data has
  **no tied names**, so the MSVC-sort replica is insurance rather than a load-bearing detail.
- `emt_light` resolves to the weapon of that name — the check that confirmed the +0x14 fix
  (the first build read the unmodelled word and saw text bytes instead).
- **Sections: 885 definitions**, species 0..6 in the order `Human, Hiver, Tarkas, Liir, _NPC,
  Zuul, Morrigi`, counts 145/137/139/136/64/122/142, `index == position`, ids from the
  manifests with the 10 dangling ids simply absent. Names arrive already resolved
  (`Assault Shuttle`, not `@SECTIONNAME_…`).

Host suite: `ctest` 26/26 (`game_config_manifest` = 71 checks). Cross-build staged in
`/srv/re-lab/shim/dist-m2`, exports 66 names identical to `binkw32.dll`; deployed from the
lane-local `C:\SOTS\shimdist-m2`. `clean_room_check.sh` OK.

## Open: the section compare crash

With `hook.Game::SectionDictionary::SectionDictionary=compare` the engine dies inside `ours`
(`Mars: Application error encountered`, `Mars.dmp` written) after the weapon hook has already
compared cleanly, before the first species finishes — no `dict: [Species/…]` line is ever
logged. The weapon path does the same thing (re-run the game's own per-file loader on a scratch
dictionary) without trouble, so the difference is in `SectionDictionary::LoadSection`, not in
the manifest reader: `manifest_pairs` is shared by both and is covered by host tests.

Leads for whoever picks this up, cheapest first:

1. `LoadSection` may append to the dictionary's own vector, in which case `ours` push_backs a
   second time and the two allocators fight — instrument by logging the vector's begin/end
   around one call before assuming.
2. It runs *after* the original has already built all 885 definitions, so the second pass
   registers duplicates with the `TechTree` / string table; a fixed-size table overflowing
   there would look exactly like this.
3. The trace-mode path is unaffected, so a narrower boundary (hooking `LoadSection` itself, one
   call per section, rather than the whole constructor) would both isolate the fault and give a
   finer region model.

Note the diagnosis cost: the crash dialog was hidden behind a Windows Update dialog, so the
process looked alive-but-wedged for ~12 minutes. Screenshot before believing a hang.

## Gotchas

1. **The dictionary structs are one word longer than a naive C translation.** The `std::vector`
   is three words (+8 begin, +c end, +10 cap), so a `void*` written after it lands at **+0x14**,
   not +0x18. The first scouting build made exactly that mistake and reported the *unmodelled*
   +0x14 word as `emt_light` (it holds text bytes — `"TION"`), which is why the region model
   above carries an explicit `unk14` and a `static_assert` on each struct size. The region
   sizes changed as a result, so the scouting trace is **not** the golden trace.
2. The `+0x14` word is not modelled: it is emitted as an opaque pointer (default policy ignores
   pointer values) so a change is visible in a trace without creating false divergences.
3. `SectionDef+0xc` was tried as an id-ish field and dropped: it holds string bytes, and bytes
   from a different heap block would diverge for no reason.
4. Per-call state is not kept between `regions()` and `ours()` (unlike M1): each dictionary is
   loaded exactly once, and `rebind` only re-aims `this` at the scratch copy.
5. Compare mode allocates through the game's own `msvcr100` `operator new`/`delete`
   (`GetProcAddress` on the mangled names) so the vector the section constructor grows can be
   freed by the game's runtime. Cast `GetProcAddress` results through `void*` — MinGW's
   `-Wcast-function-type` is an error in this tree.
6. `shim.cfg` per-hook lines: `hook.Game::WeaponDictionary::Init=compare|replace`,
   `hook.Game::SectionDictionary::SectionDictionary=compare|replace`. Ready-made configs are on
   the VM as `C:\SOTS\ui\shim.cfg.m2{trace,compare,replace}` plus `shim.cfg.m2replaceW`
   (weapons in replace, sections left in trace — the config the oracle run used).
7. The M2 hooks fire on the first `OnTick`, i.e. at the main menu, ~40 s after launch — not
   inside `Initialize` like M1. Skip the intro with three `esc` sendkeys before waiting on them.