# M2 — id-manifest loading (`_weapons.txt` / `_shipsections.txt`) old-vs-new on the live game

**Status (2026-09-08): code complete and cross-built; one scouting trace captured on the real
game; the golden trace, the compare run and the replace/End-Turn run still owe a VM window.**
VM140 was handed to another agent's behavioural compare mid-milestone, so steps 2–4 of the
milestone stop here rather than interleaving deploys with someone else's run. What the scouting
trace already proves is below — it is the reason the region model changed, so it must be re-run
against the final DLL before it can be called golden.

## What was hooked

Two functions, both `[verified]` `__thiscall` in `sots_addresses.h`, so they go through the
`Hook<Descriptor>` template rather than an asm stub (M0's stub caveat covers *unverified*
prototypes; see "thiscall through the template" below):

| hook name (record `hook`) | RVA | prototype |
|---|---|---|
| `Game::WeaponDictionary::Init` | 0x0019a4c0 | `void (WeaponDictionary*)` — `Weapons/_weapons.txt` |
| `Game::SectionDictionary::SectionDictionary` | 0x00176f40 | `SectionDictionary* (SectionDictionary*, TechTree*)` — `Species/<Race>/sections/_shipsections.txt` ×7 |

Both are called once, from `DemoApp::LoadGameData` on the first `OnTick` after start-up (not
inside `Initialize`, unlike M1). Sources: `src/shim/hooks/dictionaries.{h,cpp}`, installed from
`src/shim/main.cpp` next to the M1 hook.

There is **no registry object** to hook: the loaders drive `Script::ReadToken` themselves and
store the id on each definition. So the hookable boundary is the dictionary loader as a whole,
and the "manifest" is a token-pair stream inside it.

### Region model

The definitions do not exist when the hook is entered, so the declared region is the
**dictionary object itself** — 0x1c bytes (weapons) / 0x18 bytes (sections) — with a struct
describer that walks the vector at +8 and emits one entry per definition, in vector order:

| region | describer |
|---|---|
| `dict` (weapons, 0x1c) | `{unk0:i32, tree:ptr, n:u32, defs:[{index,id,path,name}], unk14:ptr, emt_light:str}` |
| `dict` (sections, 0x18) | `{cur:i32, tree:ptr, n:u32, defs:[{index,species,id,token}], unk14:ptr}` |

`id` (`WeaponDef+4` / `SectionDef+8`) and `index` (`+0`) are the words the original writes and
the point of the milestone; `path`/`name`/`token` are the name half of the id↔name assignment,
decoded from the MSVC `std::string`s at `WeaponDef+8`/`+0x40` and `SectionDef+0x17c`.
`emt_light` is emitted **by the weapon name it points at**, not as a pointer: in compare mode
ours builds its own definition objects, so the pointers necessarily differ while the weapon
they name must not. Every pointer chase in a describer is guarded by a `VirtualQuery` check —
the section dictionary is hooked at its *constructor*, so the "before" snapshot is whatever the
fresh heap block held (the scouting trace shows exactly that: `cur` = 71692504, an unreadable
vector, `invalid: true`).

Args: weapons `this` + the manifest path; sections `this`, `tree`, and the seven directory names
read from the game's own `Species::GetDirName`, so a golden log replays offline without the exe.

## `ours` — `src/game/config/manifest_loader.{h,cpp}` (lib `sots_game_config`)

- File bytes come from the game's own `gobio::ReadFile` (mount order / mods honoured), released
  with `RefCounted_Release`; host tests read plain text.
- `manifest_pairs()` reproduces the loaders' loop over `mars::parse::Script`: read a token, and
  **only if that read succeeded** `atoi` it into the running id; read the file-name token; a read
  that is not `Ok` ends the loop. Consequences, all tested: `// DELETED - n` and `// REMOVED - n`
  lines are ordinary comments and never appear; a non-numeric id token is `atoi`'s 0; tokens are
  truncated at 255 bytes (the loaders' 256-byte buffers); and **a manifest whose last line has no
  trailing newline loses its last entry** (the value token touches EOF, so it reads `AtEnd`).
  The shipped files end with CRLF, so nothing is lost in practice.
- `manifest_atoi()` is the CRT's: leading whitespace, optional sign, decimal digits, wrap modulo
  2^32, 0 when there is no digit.
- Path spelling is the loaders': `"Weapons/" + file`, `dir + "/" + file`.
- The pairs are handed to the game's own `LoadWeapon` / `LoadSection` (per-file parsing is M3
  scope); the weapon dictionary is then sorted and re-indexed, and `emt_light` looked up, as the
  original does after its loop. Definitions built during a compare call leak one allocation set
  (start-up only, like M1's long strings).
- `src/game/config/msvc_sort.h` replays MSVC 2010's `std::sort` (median-of-3/9 three-way
  introsort, insertion sort ≤ 32, heap-sort fallback) so that *ties* land where the original puts
  them. The shipped data turns out to have **no tied weapon names**, so any correct sort would
  do; the replica is kept because the tie order is otherwise unspecified and data can change.

## thiscall through the template

`CallConv::Thiscall` was added to `src/shim/trace/hook.h` (+ `SHIM_THISCALL` in `platform.h`):
`this` is the first `Args` element, passed in ECX by both the detour and the trampoline call,
the rest on the stack, callee-cleaned — GCC's `__attribute__((thiscall))` is exactly MSVC's ABI
on i386. This is safe **only** because both prototypes are `[verified]`; M0's asm-stub rule
still stands for anything marked `[unverified]`.

## Evidence so far

`/bulk-storage/re-lab/shim/traces/m2-trace-scout.jsonl` (+ `m2-scout-shim.log`), build
`8b231c0-dirty-20260908T0233Z`, `hooks=trace`: 22 calls, `tracecmp.py` exit 0, 0 invalid, both
M2 hooks firing once each. It confirms against the live game:

- **Weapons: 123 definitions, ids exactly the manifest's** — 0 id mismatches and 0 manifest
  entries missing from the dictionary, cross-checked against `Weapons/_weapons.txt` (123 numbered
  lines, three `// DELETED` tombstones ignored as comments).
- **The dictionary is sorted by `_stricmp` on `name`, not by id, not by path**, and `index` is
  rewritten to the position afterwards (`index == position` holds for all 123). No tied names.
- **Sections: 885 definitions**, species 0..6 in the order `Human, Hiver, Tarkas, Liir, _NPC,
  Zuul, Morrigi` (from `Species::GetDirName`), counts 145/137/139/136/64/122/142; `index ==
  position`; ids are the manifests' and the 10 dangling ids are simply absent, matching the
  RE notes. Section names arrive already resolved (`Assault Shuttle`, not `@SECTIONNAME_…`).

Host suite: `ctest` 26/26 (`game_config_manifest` = 71 checks: the token-pair loop, the
trailing-newline rule, `atoi`, truncation, path spellings, and the sort replica against
`std::sort` on distinct keys plus determinism/permutation on ties).
Cross-build: `m2-de779ad-20260908T0239Z`, exports 66 names identical to `binkw32.dll`, staged in
`/srv/re-lab/shim/dist-m2` (its own dist, not the shared one). `clean_room_check.sh` OK.

## Gotchas

1. **The dictionary structs are one word longer than a naive C translation.** The `std::vector`
   is three words (+8 begin, +c end, +10 cap), so a `void*` written after it lands at **+0x14**,
   not +0x18. The first scouting build made exactly that mistake and reported the *unmodelled*
   +0x14 word as `emt_light` (it holds text bytes — `"TION"`), which is why the region model
   above carries an explicit `unk14` and a `static_assert` on each struct size. The region
   sizes changed as a result, so the scouting trace is **not** the golden trace.
2. The `+0x14` word is not modelled: it is emitted as an opaque pointer (default policy ignores
   pointer values) so a change is visible in a trace without creating false divergences.
3. `SectionDef+0xc` was tried as an id-ish field and dropped: it holds string bytes, and bytes
   from a different heap block would diverge for no reason.
4. Per-call state is not kept between `regions()` and `ours()` (unlike M1): each dictionary is
   loaded exactly once, and `rebind` only re-aims `this` at the scratch copy.
5. Compare mode allocates through the game's own `msvcr100` `operator new`/`delete`
   (`GetProcAddress` on the mangled names) so the vector the section constructor grows can be
   freed by the game's runtime. Cast `GetProcAddress` results through `void*` — MinGW's
   `-Wcast-function-type` is an error in this tree.
6. `shim.cfg` per-hook lines: `hook.Game::WeaponDictionary::Init=compare|replace`,
   `hook.Game::SectionDictionary::SectionDictionary=compare|replace`. Ready-made configs are on
   the VM as `C:\SOTS\ui\shim.cfg.m2{trace,compare,replace}`.

## What remains (needs the VM)

1. Deploy `/srv/re-lab/shim/dist-m2` (build `m2-de779ad-20260908T0239Z`), relaunch with
   `shim.cfg.m2trace`, let it reach the main menu (the hooks fire on the first tick, ~40 s after
   launch), `tracecmp.py` → `m2-trace-golden.jsonl`.
2. Relaunch with `shim.cfg.m2compare` → 0 divergences → `m2-compare.jsonl`.
3. Relaunch with `shim.cfg.m2replace`, load `ref-turn2.sav`, End Turn, and check the oracle
   (`(Autosave).sav` = `978041ac…`, `(Autosave EndTurn).sav` = `bb4fd9ac…`).
4. Restore `hooks=trace` and leave the game running at the main menu.