161 lines
11 KiB
Markdown
161 lines
11 KiB
Markdown
# M2 — id-manifest loading (`_weapons.txt` / `_shipsections.txt`) old-vs-new on the live game
|
||
|
||
**Result (2026-09-08):** the weapon dictionary is verified end to end — trace (`tracecmp.py`
|
||
exit 0), compare **1/1 compared, 0 divergences**, and replace reproducing the End-Turn
|
||
determinism oracle byte for byte (`(Autosave).sav` = `978041ac…`, `(Autosave EndTurn).sav` =
|
||
`bb4fd9ac…`) with our loader feeding the game its weapons alone. The section dictionary is
|
||
**trace-verified only**: its 885 definitions and ids match the manifests in the golden trace,
|
||
but `ours` crashes the engine in compare mode — see "Open: the section compare crash". Game
|
||
left running on VM140 in `hooks=trace` at the main menu.
|
||
|
||
## What was hooked
|
||
|
||
Two functions, both `[verified]` `__thiscall` in `sots_addresses.h`, so they go through the
|
||
`Hook<Descriptor>` template rather than an asm stub (M0's stub caveat covers *unverified*
|
||
prototypes; see "thiscall through the template" below):
|
||
|
||
| hook name (record `hook`) | RVA | prototype |
|
||
|---|---|---|
|
||
| `Game::WeaponDictionary::Init` | 0x0019a4c0 | `void (WeaponDictionary*)` — `Weapons/_weapons.txt` |
|
||
| `Game::SectionDictionary::SectionDictionary` | 0x00176f40 | `SectionDictionary* (SectionDictionary*, TechTree*)` — `Species/<Race>/sections/_shipsections.txt` ×7 |
|
||
|
||
Both are called once, from `DemoApp::LoadGameData` on the first `OnTick` after start-up (not
|
||
inside `Initialize`, unlike M1). Sources: `src/shim/hooks/dictionaries.{h,cpp}`, installed from
|
||
`src/shim/main.cpp` next to the M1 hook.
|
||
|
||
There is **no registry object** to hook: the loaders drive `Script::ReadToken` themselves and
|
||
store the id on each definition. So the hookable boundary is the dictionary loader as a whole,
|
||
and the "manifest" is a token-pair stream inside it.
|
||
|
||
### Region model
|
||
|
||
The definitions do not exist when the hook is entered, so the declared region is the
|
||
**dictionary object itself** — 0x1c bytes (weapons) / 0x18 bytes (sections) — with a struct
|
||
describer that walks the vector at +8 and emits one entry per definition, in vector order:
|
||
|
||
| region | describer |
|
||
|---|---|
|
||
| `dict` (weapons, 0x1c) | `{unk0:i32, tree:ptr, n:u32, defs:[{index,id,path,name}], unk14:ptr, emt_light:str}` |
|
||
| `dict` (sections, 0x18) | `{cur:i32, tree:ptr, n:u32, defs:[{index,species,id,token}], unk14:ptr}` |
|
||
|
||
`id` (`WeaponDef+4` / `SectionDef+8`) and `index` (`+0`) are the words the original writes and
|
||
the point of the milestone; `path`/`name`/`token` are the name half of the id↔name assignment,
|
||
decoded from the MSVC `std::string`s at `WeaponDef+8`/`+0x40` and `SectionDef+0x17c`.
|
||
`emt_light` is emitted **by the weapon name it points at**, not as a pointer: in compare mode
|
||
ours builds its own definition objects, so the pointers necessarily differ while the weapon
|
||
they name must not. Every pointer chase in a describer is guarded by a `VirtualQuery` check —
|
||
the section dictionary is hooked at its *constructor*, so the "before" snapshot is whatever the
|
||
fresh heap block held (the scouting trace shows exactly that: `cur` = 71692504, an unreadable
|
||
vector, `invalid: true`).
|
||
|
||
Args: weapons `this` + the manifest path; sections `this`, `tree`, and the seven directory names
|
||
read from the game's own `Species::GetDirName`, so a golden log replays offline without the exe.
|
||
|
||
## `ours` — `src/game/config/manifest_loader.{h,cpp}` (lib `sots_game_config`)
|
||
|
||
- File bytes come from the game's own `gobio::ReadFile` (mount order / mods honoured), released
|
||
with `RefCounted_Release`; host tests read plain text.
|
||
- `manifest_pairs()` reproduces the loaders' loop over `mars::parse::Script`: read a token, and
|
||
**only if that read succeeded** `atoi` it into the running id; read the file-name token; a read
|
||
that is not `Ok` ends the loop. Consequences, all tested: `// DELETED - n` and `// REMOVED - n`
|
||
lines are ordinary comments and never appear; a non-numeric id token is `atoi`'s 0; tokens are
|
||
truncated at 255 bytes (the loaders' 256-byte buffers); and **a manifest whose last line has no
|
||
trailing newline loses its last entry** (the value token touches EOF, so it reads `AtEnd`).
|
||
The shipped files end with CRLF, so nothing is lost in practice.
|
||
- `manifest_atoi()` is the CRT's: leading whitespace, optional sign, decimal digits, wrap modulo
|
||
2^32, 0 when there is no digit.
|
||
- Path spelling is the loaders': `"Weapons/" + file`, `dir + "/" + file`.
|
||
- The pairs are handed to the game's own `LoadWeapon` / `LoadSection` (per-file parsing is M3
|
||
scope); the weapon dictionary is then sorted and re-indexed, and `emt_light` looked up, as the
|
||
original does after its loop. Definitions built during a compare call leak one allocation set
|
||
(start-up only, like M1's long strings).
|
||
- `src/game/config/msvc_sort.h` replays MSVC 2010's `std::sort` (median-of-3/9 three-way
|
||
introsort, insertion sort ≤ 32, heap-sort fallback) so that *ties* land where the original puts
|
||
them. The shipped data turns out to have **no tied weapon names**, so any correct sort would
|
||
do; the replica is kept because the tie order is otherwise unspecified and data can change.
|
||
|
||
## thiscall through the template
|
||
|
||
`CallConv::Thiscall` was added to `src/shim/trace/hook.h` (+ `SHIM_THISCALL` in `platform.h`):
|
||
`this` is the first `Args` element, passed in ECX by both the detour and the trampoline call,
|
||
the rest on the stack, callee-cleaned — GCC's `__attribute__((thiscall))` is exactly MSVC's ABI
|
||
on i386. This is safe **only** because both prototypes are `[verified]`; M0's asm-stub rule
|
||
still stands for anything marked `[unverified]`.
|
||
|
||
## Runs (`/bulk-storage/re-lab/shim/traces/`, build `m2v-3ebc8bf-20260908T0325Z`)
|
||
|
||
| file | mode | calls | result |
|
||
|---|---|---|---|
|
||
| `m2-trace-golden.jsonl` | trace | 22 (2 M2 + 19 LoadFile + selftest) | `tracecmp.py` exit 0, 0 invalid |
|
||
| `m2-compare.jsonl` | compare | 1 compared (weapons) | **0 diverged**, 0 errors, exit 0 |
|
||
| replace (no log) | replace | weapons | End Turn from `ref-turn2.sav`: `(Autosave).sav` `978041ac…` / `(Autosave EndTurn).sav` `bb4fd9ac…` = oracle |
|
||
| `m2-trace-scout.jsonl` | trace | 22 | the earlier build, kept because it is what exposed the +0x14 layout bug |
|
||
| `m2-shim-final.log`, `m2-*.png` | | | run banners and `dict:` lines; Turn-2 replace screen, final main menu |
|
||
|
||
What the golden trace establishes against the live game:
|
||
|
||
- **Weapons: 123 definitions, ids exactly the manifest's** — 0 id mismatches and 0 manifest
|
||
entries missing, cross-checked against `Weapons/_weapons.txt` (123 numbered lines, three
|
||
`// DELETED` tombstones correctly ignored as comments).
|
||
- **The dictionary is sorted by `_stricmp` on `name`**, not by id and not by path, with `index`
|
||
rewritten to the position afterwards (`index == position` for all 123). The shipped data has
|
||
**no tied names**, so the MSVC-sort replica is insurance rather than a load-bearing detail.
|
||
- `emt_light` resolves to the weapon of that name — the check that confirmed the +0x14 fix
|
||
(the first build read the unmodelled word and saw text bytes instead).
|
||
- **Sections: 885 definitions**, species 0..6 in the order `Human, Hiver, Tarkas, Liir, _NPC,
|
||
Zuul, Morrigi`, counts 145/137/139/136/64/122/142, `index == position`, ids from the
|
||
manifests with the 10 dangling ids simply absent. Names arrive already resolved
|
||
(`Assault Shuttle`, not `@SECTIONNAME_…`).
|
||
|
||
Host suite: `ctest` 26/26 (`game_config_manifest` = 71 checks). Cross-build staged in
|
||
`/srv/re-lab/shim/dist-m2`, exports 66 names identical to `binkw32.dll`; deployed from the
|
||
lane-local `C:\SOTS\shimdist-m2`. `clean_room_check.sh` OK.
|
||
|
||
## Open: the section compare crash
|
||
|
||
With `hook.Game::SectionDictionary::SectionDictionary=compare` the engine dies inside `ours`
|
||
(`Mars: Application error encountered`, `Mars.dmp` written) after the weapon hook has already
|
||
compared cleanly, before the first species finishes — no `dict: [Species/…]` line is ever
|
||
logged. The weapon path does the same thing (re-run the game's own per-file loader on a scratch
|
||
dictionary) without trouble, so the difference is in `SectionDictionary::LoadSection`, not in
|
||
the manifest reader: `manifest_pairs` is shared by both and is covered by host tests.
|
||
|
||
Leads for whoever picks this up, cheapest first:
|
||
|
||
1. `LoadSection` may append to the dictionary's own vector, in which case `ours` push_backs a
|
||
second time and the two allocators fight — instrument by logging the vector's begin/end
|
||
around one call before assuming.
|
||
2. It runs *after* the original has already built all 885 definitions, so the second pass
|
||
registers duplicates with the `TechTree` / string table; a fixed-size table overflowing
|
||
there would look exactly like this.
|
||
3. The trace-mode path is unaffected, so a narrower boundary (hooking `LoadSection` itself, one
|
||
call per section, rather than the whole constructor) would both isolate the fault and give a
|
||
finer region model.
|
||
|
||
Note the diagnosis cost: the crash dialog was hidden behind a Windows Update dialog, so the
|
||
process looked alive-but-wedged for ~12 minutes. Screenshot before believing a hang.
|
||
|
||
## Gotchas
|
||
|
||
1. **The dictionary structs are one word longer than a naive C translation.** The `std::vector`
|
||
is three words (+8 begin, +c end, +10 cap), so a `void*` written after it lands at **+0x14**,
|
||
not +0x18. The first scouting build made exactly that mistake and reported the *unmodelled*
|
||
+0x14 word as `emt_light` (it holds text bytes — `"TION"`), which is why the region model
|
||
above carries an explicit `unk14` and a `static_assert` on each struct size. The region
|
||
sizes changed as a result, so the scouting trace is **not** the golden trace.
|
||
2. The `+0x14` word is not modelled: it is emitted as an opaque pointer (default policy ignores
|
||
pointer values) so a change is visible in a trace without creating false divergences.
|
||
3. `SectionDef+0xc` was tried as an id-ish field and dropped: it holds string bytes, and bytes
|
||
from a different heap block would diverge for no reason.
|
||
4. Per-call state is not kept between `regions()` and `ours()` (unlike M1): each dictionary is
|
||
loaded exactly once, and `rebind` only re-aims `this` at the scratch copy.
|
||
5. Compare mode allocates through the game's own `msvcr100` `operator new`/`delete`
|
||
(`GetProcAddress` on the mangled names) so the vector the section constructor grows can be
|
||
freed by the game's runtime. Cast `GetProcAddress` results through `void*` — MinGW's
|
||
`-Wcast-function-type` is an error in this tree.
|
||
6. `shim.cfg` per-hook lines: `hook.Game::WeaponDictionary::Init=compare|replace`,
|
||
`hook.Game::SectionDictionary::SectionDictionary=compare|replace`. Ready-made configs are on
|
||
the VM as `C:\SOTS\ui\shim.cfg.m2{trace,compare,replace}` plus `shim.cfg.m2replaceW`
|
||
(weapons in replace, sections left in trace — the config the oracle run used).
|
||
7. The M2 hooks fire on the first `OnTick`, i.e. at the main menu, ~40 s after launch — not
|
||
inside `Initialize` like M1. Skip the intro with three `esc` sendkeys before waiting on them.
|