# Shim trace / compare infrastructure (`src/shim/trace/`) Hooks installed by the proxy `binkw32.dll` can run in one of four **modes** and write a JSON-Lines log that the harness in `sots-re/verify/harness/compare/` (`tracecmp.py`) validates and diffs. The wire format is `TRACE_FORMAT.md` (v1) in that directory; the C++ emitter here is a byte-exact port of its reference emitter (`mkfixture.py`) and is tested against it. | file | what | |---|---| | `platform.h` | mutex (CRITICAL_SECTION / pthread), monotonic µs, thread id, UTC stamp, `SHIM_CDECL`/`SHIM_STDCALL` | | `sha256.{h,cpp}` | FIPS 180-4 SHA-256, no allocation | | `emitter.{h,cpp}` | typed values (`Tv`), `Record`, `Meta`, `Buf`, `emit_record`/`emit_meta` (spec §3–5) | | `tracer.{h,cpp}` | `Config` (shim.cfg keys), `Tracer` (locked writer, call ids, depth), `Region`/`Snapshot`/`Scratch`, `diff_tv`/`diff_outputs` (spec §6) | | `hook.h` | `Hook`: the mode dispatch around a hooked C function | | `selftest.{h,cpp}` | worked example hook over the shim's own `Fill()`; run once per launch | Host tests: `tests/shim_trace/` (ctest `shim_trace_*`). They build the same sources with `-Wall -Wextra -Werror` on Linux; the shim cross-build links the same static library. ## Modes | mode | what runs | what is logged | caller gets | |---|---|---|---| | `off` | original | nothing | original's result | | `trace` | original | `args`, `ret`, `side` (before/after of every declared region) | original's result | | `compare` | original, then **ours on a copy of the pre-call state** | trace fields + `ours{ret,side}`, `diverged`, `diff` | original's result | | `replace` | ours | `args`, `ret`, `side` (before/after around **ours**), `coverage` | ours' result | `compare` never lets ours touch game memory: the declared regions are snapshotted *before* the original runs, those snapshots are copied into scratch buffers, the arguments are rebound onto the scratch buffers (`rebind`), and ours runs there. The diff compares the original's `after` with the scratch buffers' contents and both return values. `replace` used to log nothing at all, which is why B3's replace-mode divergence could only be found by hashing the autosave. It now writes one record per call — the same regions, measured around `ours` instead of around the original — so a replace run is readable, `--replay`-able, and directly comparable with a compare log's guard findings (docs/harness-audit.md). A throw from ours, or from any describe/regions/rebind callback, is caught inside the hook and becomes `err` on the record (which the harness counts as a divergence). The original is always invoked in `off`/`trace`/`compare`, whatever happened during capture. Nothing propagates across the hook boundary. ## Configuration (`shim.cfg`, next to the DLL) ``` hooks=trace # default mode for every hook; off = install nothing at all hook.Game::CfgVar_RegisterKey=compare # per-hook override, keyed by the record's "hook" name hook.Shim::SelfTest::Fill=off trace.path=C:\SOTS\shim.trace.jsonl # default: shim.trace.jsonl beside the DLL; overwritten per run trace.inline_max=256 # regions <= this many bytes are logged as hex, else sha256 + 32-byte head trace.flush=always # always (default) | lazy (flush only on divergence / err / exit) fpu.force=0x007f # force the x87 control word at the turn gate; off (default) = force nothing fpu.sample_ticks=on # on (default): log the control word to shim.log whenever it CHANGES ``` Unusable values are logged to `shim.log` (`config: key=value rejected (...)`) and ignored. `hooks=off` keeps M0 behaviour: no MinHook, no trace file. If the trace file cannot be opened, every template hook reports mode `off` (the game runs un-instrumented, never half-instrumented). The M0 `Mars::Application::Initialize` hook is an asm stub, not a template hook: it is installed whenever `hooks` is not `off`, and it only logs to `shim.log`. See "thiscall" below. ### `fpu.*` — the x87 control word (`src/shim/fpu_force.cpp`) Four more asm-stub hooks, installed alongside the M0 one. They exist to answer "does anything the turn pipeline computes actually *depend* on x87 intermediate precision?", which decides whether a future x64/SSE port has a double-rounding budget to preserve. * `fpu.force=<16-bit control word>` writes that word with `fldcw` at `StrategyClient::EndTurn` and `StrategyServer::BeginProcessTurn`, and **nowhere else** -- deliberately a single write per turn, because re-forcing at every hook would guarantee the value is present without proving it ever held. Each site logs observed-before, requested, and read-back-after. * `StrategyServer::ProcessTurn` is hooked **sample-only**: its reading is the independent evidence that the forced word survived into the simulation. The template hooks' own `fpu_cw` fields give ~38 more samples per turn from inside phases 4, 6 and 8. * `fpu.sample_ticks` hooks `DemoApp::OnTick` and logs only when the word *moves*, so `shim.log` carries a timeline rather than a single claim. Field layout: bits 0-5 exception masks, bits 8-9 precision control (`00`=24-bit, `10`=53-bit, `11`=64-bit), bits 10-11 rounding control (`00`=nearest, `01`=down, `10`=up, `11`=truncate), bit 12 infinity control (ignored since the 387, so `0x027f` and `0x127f` are the same arithmetic). `Mars::Application::Run` re-arms `0x127f` via `_controlfp` **every frame**, which is why the forcing has to happen inside the turn call chain: a word forced at `EndTurn` is gone by the time `BeginProcessTurn` runs on a later frame. Result of the sweep: `notes-repo:findings/subsystems/fpu-precision-sensitivity.md`. ## Log `shim.trace.jsonl`: line 1 is the `meta` record (`build`, `exe_sha256` of the running exe, `started`, `inline_max`, and a `hooks` policy map with one entry per registered template hook); every later line is one call record. `call_id` is process-global and taken *before* the original runs, so nested hooks get higher ids and `depth` says who is inside whom. `ts` is microseconds since the tracer opened. The file is ASCII, LF-terminated; a crash mid-run leaves at most one truncated last line, which the harness reports as one invalid record (`--skip-invalid` drops it). `shim.log` gets one `trace:` line at startup (path, default mode) and one `selftest:` line (the self-test hook's mode and checksum), and `shutdown: trace records=N` on detach. ## Declaring a hook A hook is a descriptor struct; `Hook` generates the detour. Everything the template needs is static, so a descriptor is a header-only declaration plus a few small functions. ```cpp #include "shim/trace/hook.h" using namespace shim::trace; struct CfgVarRegisterKeyHook { static constexpr const char* name = "Game::CfgVar_RegisterKey"; // the record's "hook" static constexpr CallConv conv = CallConv::Cdecl; // Cdecl | Stdcall using Ret = bool; // void is allowed using Args = std::tuple; // the parameter list // inputs, in declaration order (never compared by the harness; use .named("x")) static void describe_args(std::vector& out, CfgTable* t, const char* key, int value) { out.push_back(tv::ptr(t).named("table")); out.push_back(tv::str(key).named("key")); // raw cp1252 bytes; nullptr -> ptr 0x0 out.push_back(tv::i32(value).named("value")); } static Tv describe_ret(bool r) { return tv::boolean(r); } // omit when Ret is void // declared side-effect regions: name, address, size, optional structured describer static void regions(std::vector& out, CfgTable* t, const char*, int) { out.push_back(Region{"cfg_table", t, sizeof(CfgTable), &describe_table}); } static Tv describe_table(const void* p, std::size_t, unsigned /*inline_max*/) { const CfgTable* t = static_cast(p); // a *copy* of the region Tv s = tv::struct_(); s.add("count", tv::u32(t->count)); return s; // diffs then name the field } // the same call, aimed at the scratch copies (regions in declaration order) static Args rebind(Scratch& s, CfgTable*, const char* key, int value) { return Args(s.as(0), key, value); } static bool ours(CfgTable* t, const char* key, int value); // the reimplementation static HookPolicy policy() { return HookPolicy{}; } // ftol / ftol_kind / ptr_exact / unordered // REQUIRED (compile error if missing): everything the original writes that no Result // region above covers. See docs/harness-audit.md. static void coverage(Coverage& c) { c.unmodelled("erases the consumed key from the caller's map", Risk::Medium, "a red-black tree cannot be snapshotted before the call", "guard:cfg_table"); // ...or, when the regions really are everything: // c.complete("RegisterKey writes only the table slot declared above"); } }; ``` Install (in `InstallHooks`, after `Tracer::open`): ```cpp using H = Hook; H::register_policy(tracer); // BEFORE tracer.open(): lands in meta.hooks ... H::configure(tracer); // reads hook. / hooks from the config MH_CreateHook(target, reinterpret_cast(H::detour()), reinterpret_cast(&H::original)); MH_EnableHook(target); ``` Rules of the road: - **Regions are copies.** `describe` callbacks receive the snapshot buffer, not live memory; a region that contains pointers into other memory is only compared by the pointer values (ignored by default policy) unless you declare the pointed-to memory as another region. - **Region sizes must be known at call time.** A region whose length is only known after the call cannot be snapshotted "before"; declare an upper bound or split the hook. - **A clean compare bounds only what you declared.** Everything else the original writes is invisible to the diff. Two things exist so that is never silent: `coverage()`, which the compiler makes you fill in, and **guard regions** — `Region::kind = Region::Kind::Guard`, a coarse span (usually the whole object) that is watched around the original, never handed to `ours` and never diffed. Any byte in it that moved and that no Result region covers is reported as an undeclared write. Guards are cheap; declare one over every object the hook can reach. - **Push guards LAST.** Scratch holds Result regions only, so a descriptor that remembers a region index as `out.size()` at push time must not interleave guards. `Hook<>` checks the order on every call and turns a violation into an `err` rather than a shifted mapping. - **Prefer `struct` describers** over raw `bytes` for anything with fields: the harness then points at `side.cfg_table.after.v.count` instead of a byte offset. - **Floats**: `tv::f32` stores at float32 width and prints `%.9g`; the diff rounds both sides to float32 first (spec §6). Never describe a `float` as `f64`. - **Big ints**: `i64`/`u64` at or beyond 2^53 in magnitude are emitted as decimal strings. - **`json` values** are your own canonical text (spec §7); the shim compares them as text. - **Policies** only affect the shim's advisory `diff`/`diverged`; `tracecmp.py` recomputes from the meta policy plus CLI flags and warns when the shim's verdict disagrees. - **Re-entrancy**: a hook calling into another hooked function is fine (depth, ids). A hook calling *itself* through ours in compare mode would recurse into the tracer; do not do that. - **Cost**: trace/compare allocate (Tv trees, snapshots). Fine for M1-scale hooks; do not put a per-frame hot path in `trace` without thinking about `trace.flush=lazy`. ### thiscall / unverified prototypes The template's detour is a real C++ function with a fixed prototype. That is only safe when the prototype is known. M0 proved that a C++ wrapper around an `[unverified]` `thiscall` corrupts the game (`docs/M0.md`, gotcha 1). So: - `__thiscall` hooks and any hook whose entry in `sots_addresses.h` is `[unverified]` stay **asm stubs**: `pushfl/pushal`, call a C logger with ECX, `popal/popfl`, `jmp` trampoline. They are trace-only and cannot capture the return value. Their logger may still build a `Record` and hand it to `Tracer::write` (mode `trace`, `ret` null, regions snapshotted by hand). - Once a prototype is verified, `thiscall` can be adapted to the template by declaring the detour as a `Stdcall` hook that takes `this` as its first explicit parameter behind a tiny asm thunk (`push ecx` then jump) — not provided here; do it only with a verified prototype. - `cdecl`/`stdcall` with verified prototypes use the template directly (`CallConv::Cdecl`, `CallConv::Stdcall`). ## Running a report ``` # on the game VM: play; then copy C:\SOTS\shim.trace.jsonl to verify/traces/.jsonl python3 verify/harness/compare/tracecmp.py verify/traces/m1-cfgvar-20260907.jsonl \ --json-out verify/results/compare/m1-cfgvar-20260907.json # exit 0 clean, 1 divergences, 2 invalid record(s) (--skip-invalid tolerates a truncated tail) # --hook NAME one hook only # --tolerance NAME=abs:1e-6 / rel:1e-5 / ulp:2 float policy (overrides meta.hooks) # --ptr exact compare pointer values too # --replay IMPL.jsonl offline: a golden trace vs a host implementation's {call_id, ret, side} lines ``` ## Host tests ``` cmake -S . -B build-host -G Ninja -DCMAKE_BUILD_TYPE=Debug # or: cmake --preset host cmake --build build-host && ctest --test-dir build-host ``` - `shim_trace_sha256` — FIPS vectors, chunked updates, padding edges. - `shim_trace_emitter` — golden strings for §4 escaping, number forms, bytes inline/head, records, meta; writes `out/emitter_oracle.jsonl`, then `tests/shim_trace/oracle_emit.py` rebuilds the same records with `mkfixture.emit_record` and compares byte for byte, and `tracecmp.py` must read it back (exit 1: it carries one injected divergence). - `shim_trace_diff` — every §6 rule (type, ptr, nan/inf, f32 rounding, abs/rel/ulp, bytes offset, set vs list, unordered policy, struct missing/extra, side names), snapshots, config. - `shim_trace_hook` — the self-test hook through every mode: clean log → `tracecmp` exit 0, wrong/throwing ours → exit 1 (and exit 0 with `--hook` on the clean hook), truncated file → exit 2 / 0 with `--skip-invalid`, plus a second thread and the startup `run_once` path. The Python steps use `/usr/bin/python3` and the harness dir from `-DSOTS_TRACECMP_DIR` (default `$HOME/sots-re/verify/harness/compare`) or the `SOTS_TRACECMP_DIR` environment variable; when `tracecmp.py` is not there they print `SKIP` and the test still passes.