sots-engine/docs/M2.md

M2 — id-manifest loading (_weapons.txt / _shipsections.txt) old-vs-new on the live game

Result (2026-09-08): the weapon dictionary is verified end to end — trace (tracecmp.py exit 0), compare 1/1 compared, 0 divergences, and replace reproducing the End-Turn determinism oracle byte for byte ((Autosave).sav = 978041ac…, (Autosave EndTurn).sav = bb4fd9ac…) with our loader feeding the game its weapons alone. The section dictionary is trace-verified only: its 885 definitions and ids match the manifests in the golden trace, but ours crashes the engine in compare mode — see "Open: the section compare crash". Game left running on VM140 in hooks=trace at the main menu.

What was hooked

Two functions, both [verified] __thiscall in sots_addresses.h, so they go through the Hook<Descriptor> template rather than an asm stub (M0's stub caveat covers unverified prototypes; see "thiscall through the template" below):

hook name (record hook)	RVA	prototype
Game::WeaponDictionary::Init	0x0019a4c0	void (WeaponDictionary*) — Weapons/_weapons.txt
Game::SectionDictionary::SectionDictionary	0x00176f40	SectionDictionary* (SectionDictionary*, TechTree*) — Species/<Race>/sections/_shipsections.txt ×7

Both are called once, from DemoApp::LoadGameData on the first OnTick after start-up (not inside Initialize, unlike M1). Sources: src/shim/hooks/dictionaries.{h,cpp}, installed from src/shim/main.cpp next to the M1 hook.

There is no registry object to hook: the loaders drive Script::ReadToken themselves and store the id on each definition. So the hookable boundary is the dictionary loader as a whole, and the "manifest" is a token-pair stream inside it.

Region model

The definitions do not exist when the hook is entered, so the declared region is the dictionary object itself — 0x1c bytes (weapons) / 0x18 bytes (sections) — with a struct describer that walks the vector at +8 and emits one entry per definition, in vector order:

region	describer
dict (weapons, 0x1c)	{unk0:i32, tree:ptr, n:u32, defs:[{index,id,path,name}], unk14:ptr, emt_light:str}
dict (sections, 0x18)	{cur:i32, tree:ptr, n:u32, defs:[{index,species,id,token}], unk14:ptr}

id (WeaponDef+4 / SectionDef+8) and index (+0) are the words the original writes and the point of the milestone; path/name/token are the name half of the id↔name assignment, decoded from the MSVC std::strings at WeaponDef+8/+0x40 and SectionDef+0x17c. emt_light is emitted by the weapon name it points at, not as a pointer: in compare mode ours builds its own definition objects, so the pointers necessarily differ while the weapon they name must not. Every pointer chase in a describer is guarded by a VirtualQuery check — the section dictionary is hooked at its constructor, so the "before" snapshot is whatever the fresh heap block held (the scouting trace shows exactly that: cur = 71692504, an unreadable vector, invalid: true).

Args: weapons this + the manifest path; sections this, tree, and the seven directory names read from the game's own Species::GetDirName, so a golden log replays offline without the exe.

ours — src/game/config/manifest_loader.{h,cpp} (lib sots_game_config)

• File bytes come from the game's own gobio::ReadFile (mount order / mods honoured), released with RefCounted_Release; host tests read plain text.
• manifest_pairs() reproduces the loaders' loop over mars::parse::Script: read a token, and only if that read succeeded atoi it into the running id; read the file-name token; a read that is not Ok ends the loop. Consequences, all tested: // DELETED - n and // REMOVED - n lines are ordinary comments and never appear; a non-numeric id token is atoi's 0; tokens are truncated at 255 bytes (the loaders' 256-byte buffers); and a manifest whose last line has no trailing newline loses its last entry (the value token touches EOF, so it reads AtEnd). The shipped files end with CRLF, so nothing is lost in practice.
• manifest_atoi() is the CRT's: leading whitespace, optional sign, decimal digits, wrap modulo 2^32, 0 when there is no digit.
• Path spelling is the loaders': "Weapons/" + file, dir + "/" + file.
• The pairs are handed to the game's own LoadWeapon / LoadSection (per-file parsing is M3 scope); the weapon dictionary is then sorted and re-indexed, and emt_light looked up, as the original does after its loop. Definitions built during a compare call leak one allocation set (start-up only, like M1's long strings).
• src/game/config/msvc_sort.h replays MSVC 2010's std::sort (median-of-3/9 three-way introsort, insertion sort ≤ 32, heap-sort fallback) so that ties land where the original puts them. The shipped data turns out to have no tied weapon names, so any correct sort would do; the replica is kept because the tie order is otherwise unspecified and data can change.

thiscall through the template

CallConv::Thiscall was added to src/shim/trace/hook.h (+ SHIM_THISCALL in platform.h): this is the first Args element, passed in ECX by both the detour and the trampoline call, the rest on the stack, callee-cleaned — GCC's __attribute__((thiscall)) is exactly MSVC's ABI on i386. This is safe only because both prototypes are [verified]; M0's asm-stub rule still stands for anything marked [unverified].

Runs (/bulk-storage/re-lab/shim/traces/, build m2v-3ebc8bf-20260908T0325Z)

file	mode	calls	result
m2-trace-golden.jsonl	trace	22 (2 M2 + 19 LoadFile + selftest)	tracecmp.py exit 0, 0 invalid
m2-compare.jsonl	compare	1 compared (weapons)	0 diverged, 0 errors, exit 0
replace (no log)	replace	weapons	End Turn from ref-turn2.sav: (Autosave).sav 978041ac… / (Autosave EndTurn).sav bb4fd9ac… = oracle
m2-trace-scout.jsonl	trace	22	the earlier build, kept because it is what exposed the +0x14 layout bug
m2-shim-final.log, m2-*.png			run banners and dict: lines; Turn-2 replace screen, final main menu

What the golden trace establishes against the live game:

• Weapons: 123 definitions, ids exactly the manifest's — 0 id mismatches and 0 manifest entries missing, cross-checked against Weapons/_weapons.txt (123 numbered lines, three // DELETED tombstones correctly ignored as comments).
• The dictionary is sorted by _stricmp on name, not by id and not by path, with index rewritten to the position afterwards (index == position for all 123). The shipped data has no tied names, so the MSVC-sort replica is insurance rather than a load-bearing detail.
• emt_light resolves to the weapon of that name — the check that confirmed the +0x14 fix (the first build read the unmodelled word and saw text bytes instead).
• Sections: 885 definitions, species 0..6 in the order Human, Hiver, Tarkas, Liir, _NPC, Zuul, Morrigi, counts 145/137/139/136/64/122/142, index == position, ids from the manifests with the 10 dangling ids simply absent. Names arrive already resolved (Assault Shuttle, not @SECTIONNAME_…).

Host suite: ctest 26/26 (game_config_manifest = 71 checks). Cross-build staged in /srv/re-lab/shim/dist-m2, exports 66 names identical to binkw32.dll; deployed from the lane-local C:\SOTS\shimdist-m2. clean_room_check.sh OK.

Open: the section compare crash

With hook.Game::SectionDictionary::SectionDictionary=compare the engine dies inside ours (Mars: Application error encountered, Mars.dmp written) after the weapon hook has already compared cleanly, before the first species finishes — no dict: [Species/…] line is ever logged. The weapon path does the same thing (re-run the game's own per-file loader on a scratch dictionary) without trouble, so the difference is in SectionDictionary::LoadSection, not in the manifest reader: manifest_pairs is shared by both and is covered by host tests.

Leads for whoever picks this up, cheapest first:

1. LoadSection may append to the dictionary's own vector, in which case ours push_backs a second time and the two allocators fight — instrument by logging the vector's begin/end around one call before assuming.
2. It runs after the original has already built all 885 definitions, so the second pass registers duplicates with the TechTree / string table; a fixed-size table overflowing there would look exactly like this.
3. The trace-mode path is unaffected, so a narrower boundary (hooking LoadSection itself, one call per section, rather than the whole constructor) would both isolate the fault and give a finer region model.

Note the diagnosis cost: the crash dialog was hidden behind a Windows Update dialog, so the process looked alive-but-wedged for ~12 minutes. Screenshot before believing a hang.

Gotchas

1. The dictionary structs are one word longer than a naive C translation. The std::vector is three words (+8 begin, +c end, +10 cap), so a void* written after it lands at +0x14, not +0x18. The first scouting build made exactly that mistake and reported the unmodelled +0x14 word as emt_light (it holds text bytes — "TION"), which is why the region model above carries an explicit unk14 and a static_assert on each struct size. The region sizes changed as a result, so the scouting trace is not the golden trace.
2. The +0x14 word is not modelled: it is emitted as an opaque pointer (default policy ignores pointer values) so a change is visible in a trace without creating false divergences.
3. SectionDef+0xc was tried as an id-ish field and dropped: it holds string bytes, and bytes from a different heap block would diverge for no reason.
4. Per-call state is not kept between regions() and ours() (unlike M1): each dictionary is loaded exactly once, and rebind only re-aims this at the scratch copy.
5. Compare mode allocates through the game's own msvcr100 operator new/delete (GetProcAddress on the mangled names) so the vector the section constructor grows can be freed by the game's runtime. Cast GetProcAddress results through void* — MinGW's -Wcast-function-type is an error in this tree.
6. shim.cfg per-hook lines: hook.Game::WeaponDictionary::Init=compare|replace, hook.Game::SectionDictionary::SectionDictionary=compare|replace. Ready-made configs are on the VM as C:\SOTS\ui\shim.cfg.m2{trace,compare,replace} plus shim.cfg.m2replaceW (weapons in replace, sections left in trace — the config the oracle run used).
7. The M2 hooks fire on the first OnTick, i.e. at the main menu, ~40 s after launch — not inside Initialize like M1. Skip the intro with three esc sendkeys before waiting on them.