# Shim trace / compare infrastructure (`src/shim/trace/`)

Hooks installed by the proxy `binkw32.dll` can run in one of four **modes** and write a
JSON-Lines log that the harness in `sots-re/verify/harness/compare/` (`tracecmp.py`) validates
and diffs. The wire format is `TRACE_FORMAT.md` (v1) in that directory; the C++ emitter here is
a byte-exact port of its reference emitter (`mkfixture.py`) and is tested against it.

| file | what |
|---|---|
| `platform.h` | mutex (CRITICAL_SECTION / pthread), monotonic µs, thread id, UTC stamp, `SHIM_CDECL`/`SHIM_STDCALL` |
| `sha256.{h,cpp}` | FIPS 180-4 SHA-256, no allocation |
| `emitter.{h,cpp}` | typed values (`Tv`), `Record`, `Meta`, `Buf`, `emit_record`/`emit_meta` (spec §3–5) |
| `tracer.{h,cpp}` | `Config` (shim.cfg keys), `Tracer` (locked writer, call ids, depth), `Region`/`Snapshot`/`Scratch`, `diff_tv`/`diff_outputs` (spec §6) |
| `hook.h` | `Hook<Descriptor>`: the mode dispatch around a hooked C function |
| `selftest.{h,cpp}` | worked example hook over the shim's own `Fill()`; run once per launch |

Host tests: `tests/shim_trace/` (ctest `shim_trace_*`). They build the same sources with
`-Wall -Wextra -Werror` on Linux; the shim cross-build links the same static library.

## Modes

| mode | what runs | what is logged | caller gets |
|---|---|---|---|
| `off` | original | nothing | original's result |
| `trace` | original | `args`, `ret`, `side` (before/after of every declared region) | original's result |
| `compare` | original, then **ours on a copy of the pre-call state** | trace fields + `ours{ret,side}`, `diverged`, `diff` | original's result |
| `replace` | ours | nothing | ours' result |

`compare` never lets ours touch game memory: the declared regions are snapshotted *before* the
original runs, those snapshots are copied into scratch buffers, the arguments are rebound onto
the scratch buffers (`rebind`), and ours runs there. The diff compares the original's `after`
with the scratch buffers' contents and both return values.

A throw from ours, or from any describe/regions/rebind callback, is caught inside the hook and
becomes `err` on the record (which the harness counts as a divergence). The original is always
invoked in `off`/`trace`/`compare`, whatever happened during capture. Nothing propagates across
the hook boundary.

## Configuration (`shim.cfg`, next to the DLL)

```
hooks=trace                     # default mode for every hook; off = install nothing at all
hook.Game::CfgVar_RegisterKey=compare   # per-hook override, keyed by the record's "hook" name
hook.Shim::SelfTest::Fill=off
trace.path=C:\SOTS\shim.trace.jsonl     # default: shim.trace.jsonl beside the DLL; overwritten per run
trace.inline_max=256                    # regions <= this many bytes are logged as hex, else sha256 + 32-byte head
trace.flush=always                      # always (default) | lazy (flush only on divergence / err / exit)
```

Unusable values are logged to `shim.log` (`config: key=value rejected (...)`) and ignored.
`hooks=off` keeps M0 behaviour: no MinHook, no trace file. If the trace file cannot be opened,
every template hook reports mode `off` (the game runs un-instrumented, never half-instrumented).

The M0 `Mars::Application::Initialize` hook is an asm stub, not a template hook: it is installed
whenever `hooks` is not `off`, and it only logs to `shim.log`. See "thiscall" below.

## Log

`shim.trace.jsonl`: line 1 is the `meta` record (`build`, `exe_sha256` of the running exe,
`started`, `inline_max`, and a `hooks` policy map with one entry per registered template hook);
every later line is one call record. `call_id` is process-global and taken *before* the original
runs, so nested hooks get higher ids and `depth` says who is inside whom. `ts` is microseconds
since the tracer opened. The file is ASCII, LF-terminated; a crash mid-run leaves at most one
truncated last line, which the harness reports as one invalid record (`--skip-invalid` drops it).

`shim.log` gets one `trace:` line at startup (path, default mode) and one `selftest:` line
(the self-test hook's mode and checksum), and `shutdown: trace records=N` on detach.

## Declaring a hook

A hook is a descriptor struct; `Hook<D>` generates the detour. Everything the template needs is
static, so a descriptor is a header-only declaration plus a few small functions.

```cpp
#include "shim/trace/hook.h"
using namespace shim::trace;

struct CfgVarRegisterKeyHook {
    static constexpr const char* name = "Game::CfgVar_RegisterKey";   // the record's "hook"
    static constexpr CallConv conv = CallConv::Cdecl;                 // Cdecl | Stdcall
    using Ret = bool;                                                 // void is allowed
    using Args = std::tuple<CfgTable*, const char*, int>;             // the parameter list

    // inputs, in declaration order (never compared by the harness; use .named("x"))
    static void describe_args(std::vector<Tv>& out, CfgTable* t, const char* key, int value) {
        out.push_back(tv::ptr(t).named("table"));
        out.push_back(tv::str(key).named("key"));      // raw cp1252 bytes; nullptr -> ptr 0x0
        out.push_back(tv::i32(value).named("value"));
    }
    static Tv describe_ret(bool r) { return tv::boolean(r); }          // omit when Ret is void

    // declared side-effect regions: name, address, size, optional structured describer
    static void regions(std::vector<Region>& out, CfgTable* t, const char*, int) {
        out.push_back(Region{"cfg_table", t, sizeof(CfgTable), &describe_table});
    }
    static Tv describe_table(const void* p, std::size_t, unsigned /*inline_max*/) {
        const CfgTable* t = static_cast<const CfgTable*>(p);   // a *copy* of the region
        Tv s = tv::struct_();
        s.add("count", tv::u32(t->count));
        return s;                                              // diffs then name the field
    }

    // the same call, aimed at the scratch copies (regions in declaration order)
    static Args rebind(Scratch& s, CfgTable*, const char* key, int value) {
        return Args(s.as<CfgTable>(0), key, value);
    }

    static bool ours(CfgTable* t, const char* key, int value);   // the reimplementation
    static HookPolicy policy() { return HookPolicy{}; }          // ftol / ftol_kind / ptr_exact / unordered
};
```

Install (in `InstallHooks`, after `Tracer::open`):

```cpp
using H = Hook<CfgVarRegisterKeyHook>;
H::register_policy(tracer);               // BEFORE tracer.open(): lands in meta.hooks
...
H::configure(tracer);                     // reads hook.<name> / hooks from the config
MH_CreateHook(target, reinterpret_cast<void*>(H::detour()), reinterpret_cast<void**>(&H::original));
MH_EnableHook(target);
```

Rules of the road:

- **Regions are copies.** `describe` callbacks receive the snapshot buffer, not live memory; a
  region that contains pointers into other memory is only compared by the pointer values
  (ignored by default policy) unless you declare the pointed-to memory as another region.
- **Region sizes must be known at call time.** A region whose length is only known after the
  call cannot be snapshotted "before"; declare an upper bound or split the hook.
- **Prefer `struct` describers** over raw `bytes` for anything with fields: the harness then
  points at `side.cfg_table.after.v.count` instead of a byte offset.
- **Floats**: `tv::f32` stores at float32 width and prints `%.9g`; the diff rounds both sides
  to float32 first (spec §6). Never describe a `float` as `f64`.
- **Big ints**: `i64`/`u64` at or beyond 2^53 in magnitude are emitted as decimal strings.
- **`json` values** are your own canonical text (spec §7); the shim compares them as text.
- **Policies** only affect the shim's advisory `diff`/`diverged`; `tracecmp.py` recomputes from
  the meta policy plus CLI flags and warns when the shim's verdict disagrees.
- **Re-entrancy**: a hook calling into another hooked function is fine (depth, ids). A hook
  calling *itself* through ours in compare mode would recurse into the tracer; do not do that.
- **Cost**: trace/compare allocate (Tv trees, snapshots). Fine for M1-scale hooks; do not put
  a per-frame hot path in `trace` without thinking about `trace.flush=lazy`.

### thiscall / unverified prototypes

The template's detour is a real C++ function with a fixed prototype. That is only safe when the
prototype is known. M0 proved that a C++ wrapper around an `[unverified]` `thiscall` corrupts
the game (`docs/M0.md`, gotcha 1). So:

- `__thiscall` hooks and any hook whose entry in `sots_addresses.h` is `[unverified]` stay
  **asm stubs**: `pushfl/pushal`, call a C logger with ECX, `popal/popfl`, `jmp` trampoline.
  They are trace-only and cannot capture the return value. Their logger may still build a
  `Record` and hand it to `Tracer::write` (mode `trace`, `ret` null, regions snapshotted by hand).
- Once a prototype is verified, `thiscall` can be adapted to the template by declaring the
  detour as a `Stdcall` hook that takes `this` as its first explicit parameter behind a tiny asm
  thunk (`push ecx` then jump) — not provided here; do it only with a verified prototype.
- `cdecl`/`stdcall` with verified prototypes use the template directly (`CallConv::Cdecl`,
  `CallConv::Stdcall`).

## Running a report

```
# on the game VM: play; then copy C:\SOTS\shim.trace.jsonl to verify/traces/<run-tag>.jsonl
python3 verify/harness/compare/tracecmp.py verify/traces/m1-cfgvar-20260907.jsonl \
        --json-out verify/results/compare/m1-cfgvar-20260907.json
#   exit 0 clean, 1 divergences, 2 invalid record(s)   (--skip-invalid tolerates a truncated tail)
#   --hook NAME            one hook only
#   --tolerance NAME=abs:1e-6 / rel:1e-5 / ulp:2       float policy (overrides meta.hooks)
#   --ptr exact            compare pointer values too
#   --replay IMPL.jsonl    offline: a golden trace vs a host implementation's {call_id, ret, side} lines
```

## Host tests

```
cmake -S . -B build-host -G Ninja -DCMAKE_BUILD_TYPE=Debug   # or: cmake --preset host
cmake --build build-host && ctest --test-dir build-host
```

- `shim_trace_sha256` — FIPS vectors, chunked updates, padding edges.
- `shim_trace_emitter` — golden strings for §4 escaping, number forms, bytes inline/head, records,
  meta; writes `out/emitter_oracle.jsonl`, then `tests/shim_trace/oracle_emit.py` rebuilds the
  same records with `mkfixture.emit_record` and compares byte for byte, and `tracecmp.py` must
  read it back (exit 1: it carries one injected divergence).
- `shim_trace_diff` — every §6 rule (type, ptr, nan/inf, f32 rounding, abs/rel/ulp, bytes
  offset, set vs list, unordered policy, struct missing/extra, side names), snapshots, config.
- `shim_trace_hook` — the self-test hook through every mode: clean log → `tracecmp` exit 0,
  wrong/throwing ours → exit 1 (and exit 0 with `--hook` on the clean hook), truncated file →
  exit 2 / 0 with `--skip-invalid`, plus a second thread and the startup `run_once` path.

The Python steps use `/usr/bin/python3` and the harness dir from `-DSOTS_TRACECMP_DIR` (default
`$HOME/sots-re/verify/harness/compare`) or the `SOTS_TRACECMP_DIR` environment variable; when
`tracecmp.py` is not there they print `SKIP` and the test still passes.