sots-engine/docs/M2.md

M2 — id-manifest loading (_weapons.txt / _shipsections.txt) old-vs-new on the live game

Status (2026-09-08): code complete and cross-built; one scouting trace captured on the real game; the golden trace, the compare run and the replace/End-Turn run still owe a VM window. VM140 was handed to another agent's behavioural compare mid-milestone, so steps 2–4 of the milestone stop here rather than interleaving deploys with someone else's run. What the scouting trace already proves is below — it is the reason the region model changed, so it must be re-run against the final DLL before it can be called golden.

What was hooked

Two functions, both [verified] __thiscall in sots_addresses.h, so they go through the Hook<Descriptor> template rather than an asm stub (M0's stub caveat covers unverified prototypes; see "thiscall through the template" below):

hook name (record hook)	RVA	prototype
Game::WeaponDictionary::Init	0x0019a4c0	void (WeaponDictionary*) — Weapons/_weapons.txt
Game::SectionDictionary::SectionDictionary	0x00176f40	SectionDictionary* (SectionDictionary*, TechTree*) — Species/<Race>/sections/_shipsections.txt ×7

Both are called once, from DemoApp::LoadGameData on the first OnTick after start-up (not inside Initialize, unlike M1). Sources: src/shim/hooks/dictionaries.{h,cpp}, installed from src/shim/main.cpp next to the M1 hook.

There is no registry object to hook: the loaders drive Script::ReadToken themselves and store the id on each definition. So the hookable boundary is the dictionary loader as a whole, and the "manifest" is a token-pair stream inside it.

Region model

The definitions do not exist when the hook is entered, so the declared region is the dictionary object itself — 0x1c bytes (weapons) / 0x18 bytes (sections) — with a struct describer that walks the vector at +8 and emits one entry per definition, in vector order:

region	describer
dict (weapons, 0x1c)	{unk0:i32, tree:ptr, n:u32, defs:[{index,id,path,name}], unk14:ptr, emt_light:str}
dict (sections, 0x18)	{cur:i32, tree:ptr, n:u32, defs:[{index,species,id,token}], unk14:ptr}

id (WeaponDef+4 / SectionDef+8) and index (+0) are the words the original writes and the point of the milestone; path/name/token are the name half of the id↔name assignment, decoded from the MSVC std::strings at WeaponDef+8/+0x40 and SectionDef+0x17c. emt_light is emitted by the weapon name it points at, not as a pointer: in compare mode ours builds its own definition objects, so the pointers necessarily differ while the weapon they name must not. Every pointer chase in a describer is guarded by a VirtualQuery check — the section dictionary is hooked at its constructor, so the "before" snapshot is whatever the fresh heap block held (the scouting trace shows exactly that: cur = 71692504, an unreadable vector, invalid: true).

Args: weapons this + the manifest path; sections this, tree, and the seven directory names read from the game's own Species::GetDirName, so a golden log replays offline without the exe.

ours — src/game/config/manifest_loader.{h,cpp} (lib sots_game_config)

• File bytes come from the game's own gobio::ReadFile (mount order / mods honoured), released with RefCounted_Release; host tests read plain text.
• manifest_pairs() reproduces the loaders' loop over mars::parse::Script: read a token, and only if that read succeeded atoi it into the running id; read the file-name token; a read that is not Ok ends the loop. Consequences, all tested: // DELETED - n and // REMOVED - n lines are ordinary comments and never appear; a non-numeric id token is atoi's 0; tokens are truncated at 255 bytes (the loaders' 256-byte buffers); and a manifest whose last line has no trailing newline loses its last entry (the value token touches EOF, so it reads AtEnd). The shipped files end with CRLF, so nothing is lost in practice.
• manifest_atoi() is the CRT's: leading whitespace, optional sign, decimal digits, wrap modulo 2^32, 0 when there is no digit.
• Path spelling is the loaders': "Weapons/" + file, dir + "/" + file.
• The pairs are handed to the game's own LoadWeapon / LoadSection (per-file parsing is M3 scope); the weapon dictionary is then sorted and re-indexed, and emt_light looked up, as the original does after its loop. Definitions built during a compare call leak one allocation set (start-up only, like M1's long strings).
• src/game/config/msvc_sort.h replays MSVC 2010's std::sort (median-of-3/9 three-way introsort, insertion sort ≤ 32, heap-sort fallback) so that ties land where the original puts them. The shipped data turns out to have no tied weapon names, so any correct sort would do; the replica is kept because the tie order is otherwise unspecified and data can change.

thiscall through the template

CallConv::Thiscall was added to src/shim/trace/hook.h (+ SHIM_THISCALL in platform.h): this is the first Args element, passed in ECX by both the detour and the trampoline call, the rest on the stack, callee-cleaned — GCC's __attribute__((thiscall)) is exactly MSVC's ABI on i386. This is safe only because both prototypes are [verified]; M0's asm-stub rule still stands for anything marked [unverified].

Evidence so far

/bulk-storage/re-lab/shim/traces/m2-trace-scout.jsonl (+ m2-scout-shim.log), build 8b231c0-dirty-20260908T0233Z, hooks=trace: 22 calls, tracecmp.py exit 0, 0 invalid, both M2 hooks firing once each. It confirms against the live game:

• Weapons: 123 definitions, ids exactly the manifest's — 0 id mismatches and 0 manifest entries missing from the dictionary, cross-checked against Weapons/_weapons.txt (123 numbered lines, three // DELETED tombstones ignored as comments).
• The dictionary is sorted by _stricmp on name, not by id, not by path, and index is rewritten to the position afterwards (index == position holds for all 123). No tied names.
• Sections: 885 definitions, species 0..6 in the order Human, Hiver, Tarkas, Liir, _NPC, Zuul, Morrigi (from Species::GetDirName), counts 145/137/139/136/64/122/142; index == position; ids are the manifests' and the 10 dangling ids are simply absent, matching the RE notes. Section names arrive already resolved (Assault Shuttle, not @SECTIONNAME_…).

Host suite: ctest 26/26 (game_config_manifest = 71 checks: the token-pair loop, the trailing-newline rule, atoi, truncation, path spellings, and the sort replica against std::sort on distinct keys plus determinism/permutation on ties). Cross-build: m2-de779ad-20260908T0239Z, exports 66 names identical to binkw32.dll, staged in /srv/re-lab/shim/dist-m2 (its own dist, not the shared one). clean_room_check.sh OK.

Gotchas

1. The dictionary structs are one word longer than a naive C translation. The std::vector is three words (+8 begin, +c end, +10 cap), so a void* written after it lands at +0x14, not +0x18. The first scouting build made exactly that mistake and reported the unmodelled +0x14 word as emt_light (it holds text bytes — "TION"), which is why the region model above carries an explicit unk14 and a static_assert on each struct size. The region sizes changed as a result, so the scouting trace is not the golden trace.
2. The +0x14 word is not modelled: it is emitted as an opaque pointer (default policy ignores pointer values) so a change is visible in a trace without creating false divergences.
3. SectionDef+0xc was tried as an id-ish field and dropped: it holds string bytes, and bytes from a different heap block would diverge for no reason.
4. Per-call state is not kept between regions() and ours() (unlike M1): each dictionary is loaded exactly once, and rebind only re-aims this at the scratch copy.
5. Compare mode allocates through the game's own msvcr100 operator new/delete (GetProcAddress on the mangled names) so the vector the section constructor grows can be freed by the game's runtime. Cast GetProcAddress results through void* — MinGW's -Wcast-function-type is an error in this tree.
6. shim.cfg per-hook lines: hook.Game::WeaponDictionary::Init=compare|replace, hook.Game::SectionDictionary::SectionDictionary=compare|replace. Ready-made configs are on the VM as C:\SOTS\ui\shim.cfg.m2{trace,compare,replace}.

What remains (needs the VM)

1. Deploy /srv/re-lab/shim/dist-m2 (build m2-de779ad-20260908T0239Z), relaunch with shim.cfg.m2trace, let it reach the main menu (the hooks fire on the first tick, ~40 s after launch), tracecmp.py → m2-trace-golden.jsonl.
2. Relaunch with shim.cfg.m2compare → 0 divergences → m2-compare.jsonl.
3. Relaunch with shim.cfg.m2replace, load ref-turn2.sav, End Turn, and check the oracle ((Autosave).sav = 978041ac…, (Autosave EndTurn).sav = bb4fd9ac…).
4. Restore hooks=trace and leave the game running at the main menu.