136 lines
9 KiB
Markdown
136 lines
9 KiB
Markdown
# M2 — id-manifest loading (`_weapons.txt` / `_shipsections.txt`) old-vs-new on the live game
|
||
|
||
**Status (2026-09-08): code complete and cross-built; one scouting trace captured on the real
|
||
game; the golden trace, the compare run and the replace/End-Turn run still owe a VM window.**
|
||
VM140 was handed to another agent's behavioural compare mid-milestone, so steps 2–4 of the
|
||
milestone stop here rather than interleaving deploys with someone else's run. What the scouting
|
||
trace already proves is below — it is the reason the region model changed, so it must be re-run
|
||
against the final DLL before it can be called golden.
|
||
|
||
## What was hooked
|
||
|
||
Two functions, both `[verified]` `__thiscall` in `sots_addresses.h`, so they go through the
|
||
`Hook<Descriptor>` template rather than an asm stub (M0's stub caveat covers *unverified*
|
||
prototypes; see "thiscall through the template" below):
|
||
|
||
| hook name (record `hook`) | RVA | prototype |
|
||
|---|---|---|
|
||
| `Game::WeaponDictionary::Init` | 0x0019a4c0 | `void (WeaponDictionary*)` — `Weapons/_weapons.txt` |
|
||
| `Game::SectionDictionary::SectionDictionary` | 0x00176f40 | `SectionDictionary* (SectionDictionary*, TechTree*)` — `Species/<Race>/sections/_shipsections.txt` ×7 |
|
||
|
||
Both are called once, from `DemoApp::LoadGameData` on the first `OnTick` after start-up (not
|
||
inside `Initialize`, unlike M1). Sources: `src/shim/hooks/dictionaries.{h,cpp}`, installed from
|
||
`src/shim/main.cpp` next to the M1 hook.
|
||
|
||
There is **no registry object** to hook: the loaders drive `Script::ReadToken` themselves and
|
||
store the id on each definition. So the hookable boundary is the dictionary loader as a whole,
|
||
and the "manifest" is a token-pair stream inside it.
|
||
|
||
### Region model
|
||
|
||
The definitions do not exist when the hook is entered, so the declared region is the
|
||
**dictionary object itself** — 0x1c bytes (weapons) / 0x18 bytes (sections) — with a struct
|
||
describer that walks the vector at +8 and emits one entry per definition, in vector order:
|
||
|
||
| region | describer |
|
||
|---|---|
|
||
| `dict` (weapons, 0x1c) | `{unk0:i32, tree:ptr, n:u32, defs:[{index,id,path,name}], unk14:ptr, emt_light:str}` |
|
||
| `dict` (sections, 0x18) | `{cur:i32, tree:ptr, n:u32, defs:[{index,species,id,token}], unk14:ptr}` |
|
||
|
||
`id` (`WeaponDef+4` / `SectionDef+8`) and `index` (`+0`) are the words the original writes and
|
||
the point of the milestone; `path`/`name`/`token` are the name half of the id↔name assignment,
|
||
decoded from the MSVC `std::string`s at `WeaponDef+8`/`+0x40` and `SectionDef+0x17c`.
|
||
`emt_light` is emitted **by the weapon name it points at**, not as a pointer: in compare mode
|
||
ours builds its own definition objects, so the pointers necessarily differ while the weapon
|
||
they name must not. Every pointer chase in a describer is guarded by a `VirtualQuery` check —
|
||
the section dictionary is hooked at its *constructor*, so the "before" snapshot is whatever the
|
||
fresh heap block held (the scouting trace shows exactly that: `cur` = 71692504, an unreadable
|
||
vector, `invalid: true`).
|
||
|
||
Args: weapons `this` + the manifest path; sections `this`, `tree`, and the seven directory names
|
||
read from the game's own `Species::GetDirName`, so a golden log replays offline without the exe.
|
||
|
||
## `ours` — `src/game/config/manifest_loader.{h,cpp}` (lib `sots_game_config`)
|
||
|
||
- File bytes come from the game's own `gobio::ReadFile` (mount order / mods honoured), released
|
||
with `RefCounted_Release`; host tests read plain text.
|
||
- `manifest_pairs()` reproduces the loaders' loop over `mars::parse::Script`: read a token, and
|
||
**only if that read succeeded** `atoi` it into the running id; read the file-name token; a read
|
||
that is not `Ok` ends the loop. Consequences, all tested: `// DELETED - n` and `// REMOVED - n`
|
||
lines are ordinary comments and never appear; a non-numeric id token is `atoi`'s 0; tokens are
|
||
truncated at 255 bytes (the loaders' 256-byte buffers); and **a manifest whose last line has no
|
||
trailing newline loses its last entry** (the value token touches EOF, so it reads `AtEnd`).
|
||
The shipped files end with CRLF, so nothing is lost in practice.
|
||
- `manifest_atoi()` is the CRT's: leading whitespace, optional sign, decimal digits, wrap modulo
|
||
2^32, 0 when there is no digit.
|
||
- Path spelling is the loaders': `"Weapons/" + file`, `dir + "/" + file`.
|
||
- The pairs are handed to the game's own `LoadWeapon` / `LoadSection` (per-file parsing is M3
|
||
scope); the weapon dictionary is then sorted and re-indexed, and `emt_light` looked up, as the
|
||
original does after its loop. Definitions built during a compare call leak one allocation set
|
||
(start-up only, like M1's long strings).
|
||
- `src/game/config/msvc_sort.h` replays MSVC 2010's `std::sort` (median-of-3/9 three-way
|
||
introsort, insertion sort ≤ 32, heap-sort fallback) so that *ties* land where the original puts
|
||
them. The shipped data turns out to have **no tied weapon names**, so any correct sort would
|
||
do; the replica is kept because the tie order is otherwise unspecified and data can change.
|
||
|
||
## thiscall through the template
|
||
|
||
`CallConv::Thiscall` was added to `src/shim/trace/hook.h` (+ `SHIM_THISCALL` in `platform.h`):
|
||
`this` is the first `Args` element, passed in ECX by both the detour and the trampoline call,
|
||
the rest on the stack, callee-cleaned — GCC's `__attribute__((thiscall))` is exactly MSVC's ABI
|
||
on i386. This is safe **only** because both prototypes are `[verified]`; M0's asm-stub rule
|
||
still stands for anything marked `[unverified]`.
|
||
|
||
## Evidence so far
|
||
|
||
`/bulk-storage/re-lab/shim/traces/m2-trace-scout.jsonl` (+ `m2-scout-shim.log`), build
|
||
`8b231c0-dirty-20260908T0233Z`, `hooks=trace`: 22 calls, `tracecmp.py` exit 0, 0 invalid, both
|
||
M2 hooks firing once each. It confirms against the live game:
|
||
|
||
- **Weapons: 123 definitions, ids exactly the manifest's** — 0 id mismatches and 0 manifest
|
||
entries missing from the dictionary, cross-checked against `Weapons/_weapons.txt` (123 numbered
|
||
lines, three `// DELETED` tombstones ignored as comments).
|
||
- **The dictionary is sorted by `_stricmp` on `name`, not by id, not by path**, and `index` is
|
||
rewritten to the position afterwards (`index == position` holds for all 123). No tied names.
|
||
- **Sections: 885 definitions**, species 0..6 in the order `Human, Hiver, Tarkas, Liir, _NPC,
|
||
Zuul, Morrigi` (from `Species::GetDirName`), counts 145/137/139/136/64/122/142; `index ==
|
||
position`; ids are the manifests' and the 10 dangling ids are simply absent, matching the
|
||
RE notes. Section names arrive already resolved (`Assault Shuttle`, not `@SECTIONNAME_…`).
|
||
|
||
Host suite: `ctest` 26/26 (`game_config_manifest` = 71 checks: the token-pair loop, the
|
||
trailing-newline rule, `atoi`, truncation, path spellings, and the sort replica against
|
||
`std::sort` on distinct keys plus determinism/permutation on ties).
|
||
Cross-build: `m2-de779ad-20260908T0239Z`, exports 66 names identical to `binkw32.dll`, staged in
|
||
`/srv/re-lab/shim/dist-m2` (its own dist, not the shared one). `clean_room_check.sh` OK.
|
||
|
||
## Gotchas
|
||
|
||
1. **The dictionary structs are one word longer than a naive C translation.** The `std::vector`
|
||
is three words (+8 begin, +c end, +10 cap), so a `void*` written after it lands at **+0x14**,
|
||
not +0x18. The first scouting build made exactly that mistake and reported the *unmodelled*
|
||
+0x14 word as `emt_light` (it holds text bytes — `"TION"`), which is why the region model
|
||
above carries an explicit `unk14` and a `static_assert` on each struct size. The region
|
||
sizes changed as a result, so the scouting trace is **not** the golden trace.
|
||
2. The `+0x14` word is not modelled: it is emitted as an opaque pointer (default policy ignores
|
||
pointer values) so a change is visible in a trace without creating false divergences.
|
||
3. `SectionDef+0xc` was tried as an id-ish field and dropped: it holds string bytes, and bytes
|
||
from a different heap block would diverge for no reason.
|
||
4. Per-call state is not kept between `regions()` and `ours()` (unlike M1): each dictionary is
|
||
loaded exactly once, and `rebind` only re-aims `this` at the scratch copy.
|
||
5. Compare mode allocates through the game's own `msvcr100` `operator new`/`delete`
|
||
(`GetProcAddress` on the mangled names) so the vector the section constructor grows can be
|
||
freed by the game's runtime. Cast `GetProcAddress` results through `void*` — MinGW's
|
||
`-Wcast-function-type` is an error in this tree.
|
||
6. `shim.cfg` per-hook lines: `hook.Game::WeaponDictionary::Init=compare|replace`,
|
||
`hook.Game::SectionDictionary::SectionDictionary=compare|replace`. Ready-made configs are on
|
||
the VM as `C:\SOTS\ui\shim.cfg.m2{trace,compare,replace}`.
|
||
|
||
## What remains (needs the VM)
|
||
|
||
1. Deploy `/srv/re-lab/shim/dist-m2` (build `m2-de779ad-20260908T0239Z`), relaunch with
|
||
`shim.cfg.m2trace`, let it reach the main menu (the hooks fire on the first tick, ~40 s after
|
||
launch), `tracecmp.py` → `m2-trace-golden.jsonl`.
|
||
2. Relaunch with `shim.cfg.m2compare` → 0 divergences → `m2-compare.jsonl`.
|
||
3. Relaunch with `shim.cfg.m2replace`, load `ref-turn2.sav`, End Turn, and check the oracle
|
||
(`(Autosave).sav` = `978041ac…`, `(Autosave EndTurn).sav` = `bb4fd9ac…`).
|
||
4. Restore `hooks=trace` and leave the game running at the main menu.
|