diff --git a/campaign/board.md b/campaign/board.md index 006c7b8..5b1c284 100644 --- a/campaign/board.md +++ b/campaign/board.md @@ -33,9 +33,17 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`). | save_reader.py | verify | verified | high | 100% | 2026-09-07 | --strict exit 0 on all 3 real saves; 0 resyncs / 0 hint-failures; only raw = RNG blob (2503 B). 29 tests. SAVE_FORMAT.md confirmed | | Ghidra type write-back | meta | verified | high | 100% | 2026-09-07 | structs saved in project (ServerSystem 87f, ServerPlayer 110f, StarFleet, StarShip, StrategyServer partial, 22 nested); 52 serializers + primitives + ~60 spine fns renamed; decompile shows field names | | strategic turn internals (economy/research/colony/movement/diplomacy) | subsystem | mapped | high | 90% | 2026-09-07 | findings/subsystems/strategic-turn-internals.md: budget/RP/trade/bankruptcy formulas, research rolls (unlisted race=1.0, Zuul x2), colony growth, movement, RNG=MT19937; ~600 g_KEY labels in Ghidra | -| P2-M0 shim bootstrap | phase2 | in-progress | — | 0% | 2026-09-07 | sots-engine: binkw32 proxy + MinHook + trace hook on Application::Initialize; game to menu | +| P2-M0 shim bootstrap | phase2 | verified | high | 100% | 2026-09-07 | sots-engine 5f5bc41: proxy binkw32.dll (66 exports identical), MinHook, asm-stub trace hook on Application::Initialize; menu reached; evidence verify/results/shim/. Initialize signature unverified -> asm stubs only | | P2-M1 flat-KV config loader | phase2 | backlog | — | 0% | 2026-09-07 | CfgVar_RegisterKey 0x008b76a0; trace->compare->replace; save byte-identical oracle | | P2-M2 manifest/id registry | phase2 | backlog | — | 0% | 2026-09-07 | _weapons.txt/_shipsections.txt stable ids | | P2-M3 Mars brace-block parser | phase2 | backlog | — | 0% | 2026-09-07 | + .effect TXT reader; compare over full catalog load | | P2-M4 gobio VFS read | phase2 | backlog | — | 0% | 2026-09-07 | ZIP entries + NativeFileSystem override order | -| loader prototypes (Ghidra) | meta | in-progress | — | 0% | 2026-09-07 | R6: cfgvar parser, manifest loader, brace-block parser entry+node model, gobio read -> ghidra/addresses.json | +| loader prototypes (Ghidra) | meta | in-progress | — | 0% | 2026-09-07 | R6 running: cfgvar parser, manifest loader, brace-block entry+node model, gobio read; + Initialize/Run/OnTick signature checks (M0 crash lesson) | +| compare harness | verify | mapped | high | 100% | 2026-09-07 | verify/harness/compare/: TRACE_FORMAT.md, tracecmp.py, replay, oracle bridge | +| engine: mars/parse | engine | verified | high | 100% | 2026-09-07 | sots-engine: brace-block + .effect readers; oracle 1531/1531; cross-compiles i686; wired into host+shim builds | +| engine: mars/text | engine | verified | high | 100% | 2026-09-07 | flat-kv, id-manifest, csv; oracle 64/64 (Strings.csv 5722 rows); ctest green | +| engine: game/sim formulas | engine | mapped | med | 80% | 2026-09-07 | economy/research/colony/movement pure fns, 356 checks; 5 low-confidence formulas fed back to open-questions | +| engine: mars/stream + rng | engine | in-progress | — | 0% | 2026-09-07 | Streamable reader/writer + MT19937; oracle = 3 real saves + reference vectors | +| engine: game/data catalogs | engine | in-progress | — | 0% | 2026-09-07 | typed WeaponDef/ShipSectionDef/TechTree/IdRegistry on mars/parse+text; oracle = data-catalogs JSON | +| engine: mars/vfs (gob) | engine | in-progress | — | 0% | 2026-09-07 | ZIP reader + native override; oracle = unzip listings | +| determinism oracle | verify | in-progress | — | 0% | 2026-09-07 | load same save -> End Turn -> byte-compare autosaves across runs (VM) | diff --git a/campaign/open-questions.md b/campaign/open-questions.md index 01e6524..72eb5d0 100644 --- a/campaign/open-questions.md +++ b/campaign/open-questions.md @@ -30,3 +30,4 @@ Each links to the finding that raised it. Promoted to backlog or closed by **re- - **Ship-design code rules (Ghidra, from SHIP_DESIGN_RULES.md §8)** — hidden default rider designs are built WITHOUT tech gating (where?); is section class-equality enforced or merely unoffered; confirm the turret class-accept table (`standard ← {standard,missile,grapple}`, `strafe ← {standard}`; grapple weakest); hull-class tech injection (`IND_CruisCon`/`IND_DreadCon`) + station exemption; option defaults/auto-upgrade and what `option_cost` multiplies; weapon cost per bank vs per mount; mass/speed formulas; `faiDes`/`dWep` meaning. (from [[SHIP_DESIGN_RULES]]) - **Corrections from turn internals** — `ServerPlayer+0xf9` is the AI flag (not bTurnDone); `FUN_007d7f70` is encounter detection; ProcessTurn phase 2 is trade, not diplomacy; the spine's `FUN_00840fe0` 'income' is special projects — income is `FUN_00863030` ComputeBudget. `TECHBEN_*` data blocks are inert (effects hard-coded via the 116-entry tech-name table @0x00a19718). Open: TRA/TRP producers, `FPsp2` derivation, `PERGATETRAFFIC_*` readers, exact BnkEl/BnkPr expression. (from [[strategic-turn-internals]]) - **`Mars::Application::Initialize` signature (P2-M0 finding)** — a C++ `__thiscall` detour that called the original crashed inside Initialize; the trace hook now uses a convention-agnostic asm stub. Verify the real signature (stack args? EDX use? return value?) before any `compare`/`replace` hook on it. Same caution for every `[unverified]` entry in `ghidra/addresses.json`. (from [[m0]]) +- **Formula gaps surfaced by the game/sim port (feed back to Ghidra)** — which bankruptcy limit carries the 3.3 factor and when the bankruptcy start turn is stamped; the suitability→carrying-capacity hazard curve shape; the trade-points→money system-income tail; the `POPBONUS_INC` population increment; the expense-slider request term; which running total the tech-income bonus / savings aid read; node-line speed clamp at the influence radius; whether `DecayAllResearch` also hits the current target. (from [[game-sim]]) diff --git a/verify/harness/compare/TRACE_FORMAT.md b/verify/harness/compare/TRACE_FORMAT.md new file mode 100644 index 0000000..f114a14 --- /dev/null +++ b/verify/harness/compare/TRACE_FORMAT.md @@ -0,0 +1,198 @@ +# Trace / compare log format (v1) + +Wire format between the shim (`sots-engine/src/shim/`, C++, no JSON library) and the +harness in this directory (`tracecmp.py`, stdlib Python). One file per shim run; golden +traces live in `verify/traces/`, harness reports in `verify/results/compare/`. + +Design goals, in order: (1) trivially emittable from C++ with `fprintf`; (2) every value +carries its C type so the harness can apply the right equality rule; (3) big memory +regions never blow up the log (hash, summarize, cap); (4) the same record shape serves +online `compare` (shim runs original + ours) and offline `replay` (harness diffs a golden +`trace` log against a host-side implementation's output). + +## 1. Container: JSON Lines + +- UTF-8, no BOM. One JSON object per line, terminated by `\n` (LF only). No blank lines, + no comments, no trailing commas. A line that does not parse is an **invalid record**. +- The first line SHOULD be a `meta` record (section 5). Everything else is a **call + record** (section 2). +- Records are appended in emission order; `call_id` (not line order) is the identity. +- Log files are named `.jsonl` (e.g. `verify/traces/m1-cfgvar-20260907.jsonl`). + +## 2. Call record + +```json +{"ts":123456,"hook":"CfgVar_RegisterKey","mode":"trace","call_id":17,"thread":4120, + "args":[{"t":"str","v":"ForceSingleCore"},{"t":"i32","v":1}], + "ret":{"t":"bool","v":true}, + "side":{"cfg_table":{"before":{"t":"bytes","n":4096,"sha256":"…"}, + "after":{"t":"bytes","n":4096,"sha256":"…"}}}} +``` + +| field | type | required | meaning | +|---|---|---|---| +| `ts` | int | yes | microseconds since shim init (QPC-based). Informational; never compared. | +| `hook` | string | yes | hook name — the function's recovered name (`Game::…` / `Mars::…` or `FUN_xxxxxxxx`). | +| `mode` | `"trace"` \| `"compare"` \| `"replace"` | yes | which mode the hook ran in for this call. | +| `call_id` | int ≥ 0 | yes | process-global, monotonically increasing, atomic. Unique within a log. | +| `thread` | int | yes | OS thread id. Informational. | +| `depth` | int | no | re-entrancy depth (0 = outermost). Informational. | +| `args` | array of typed values | yes | inputs, in declaration order. `n` names them when known. Never compared (they are the snapshot both sides ran on). | +| `ret` | typed value or `null` | yes | return value of the **original** (trace/compare) or of **ours** (replace). `null` = void. | +| `side` | object | yes (may be `{}`) | declared side effects: `name -> {"before": tv|null, "after": tv}`. `after` is compared; `before` is informational (a mismatch in `before` between original and ours would be a snapshot bug and is reported as a warning, not a divergence). | +| `ours` | object | compare only | `{"ret": tv|null, "side": {name: {"after": tv}}}` — the reimplementation's outputs on the same snapshot. | +| `diverged` | bool | compare only | the shim's own verdict. The harness recomputes and reports disagreements as warnings. | +| `diff` | array of diff entries | compare only | the shim's own diff (may be `[]` or truncated). Advisory. | +| `err` | string | no | the hook could not capture (exception in ours, snapshot failure…). Counts as a divergence in `compare`. | +| `note` | string | no | free text. Ignored. | + +Unknown top-level keys are a validation **warning** (not an error) so the shim can add +fields without breaking older harnesses. + +`replace` records have no `ours`; `ret`/`side` are the reimplementation's. They are +counted and validated but never diffed (there is nothing to diff against). + +## 3. Typed values (`tv`) + +Every value is an object `{"t": TYPE, "v": VALUE}` plus optional `"n": "name"`. +TYPEs and their equality rule: + +| `t` | `v` | equality | +|---|---|---| +| `bool` | `true`/`false` | exact | +| `i8 i16 i32 i64 u8 u16 u32 u64` | JSON integer. i64/u64 above 2^53 MUST be a decimal **string** (`"v":"18446744073709551615"`) — the harness accepts both int and string. | exact | +| `f32`, `f64` | JSON number, or the string `"nan"`, `"inf"`, `"-inf"`. Emit with `%.9g` (f32) / `%.17g` (f64). | tolerant (section 6) | +| `str` | JSON string carrying **raw bytes** of a `char*`/`std::string` (cp1252 in this engine). Each byte maps to one code point U+0000–U+00FF (see escaping, section 4). | exact | +| `wstr` | JSON string of a `wchar_t*` (UTF-16 code units → `\uXXXX`). | exact | +| `ptr` | hex string `"0x00a36fd0"` | **ignored** by default (ours allocates elsewhere). A hook policy may set `"ptr":"exact"`. | +| `enum` | integer, optional `"name"` sibling with the symbolic name | exact on the integer | +| `null` | `null` | exact (only equals `null`) | +| `bytes` | see below | exact on `sha256` (and on `hex` when both inline) | +| `list` | JSON array of tv | same length, element-wise, ordered | +| `set` | JSON array of tv | unordered multiset: sort both by canonical text, then element-wise | +| `struct` | JSON object `{field: tv}` | same key set, field-wise (recursion) | +| `json` | any canonical-JSON value (section 7) — used by the oracle bridge and host-side tests | structural exact, with the hook's float tolerance applied to JSON numbers that are floats on **both** sides | + +A type mismatch between the two sides (`i32` vs `u32`, `str` vs `wstr`) is a divergence +(`why: "type"`). Type mismatch does not apply inside `json` values, where int vs float +is a divergence unless both are numbers and the tolerance policy is `"numeric"`. + +### `bytes` — binary blobs and memory regions + +```json +{"t":"bytes","n":4096,"sha256":"<64 hex>","hex":"<2n hex>"} +{"t":"bytes","n":1048576,"sha256":"<64 hex>","head":""} +``` + +- `n` (length in bytes) and `sha256` (lowercase hex of the SHA-256 of the raw bytes) are + **always** present, even for `n = 0` (sha256 of the empty string). +- `hex` (lowercase, no separators) is present when `n <= inline_max`. `inline_max` is a + shim setting recorded in `meta` (default **256**). `hex` is the only inline encoding; + base64 is not used (harder to eyeball, no size win worth a second decoder). +- Larger blobs carry `head` (first 32 bytes, hex) for eyeballing only. +- When both sides are inline and differ, the harness reports the first differing byte + offset; when hashed, only that the hash differs. +- A **memory region** is just a `bytes` value captured at a declared `(address, length)`; + the region's name is the key in `side`. Structured regions should be captured as + `struct` of typed values instead so diffs point at the field, not a byte offset. + +## 4. Emitter rules (C++ without a JSON library) + +The emitter is a set of `fprintf`s. What it must guarantee: + +1. **Key order is fixed**: `ts, hook, mode, call_id, thread, depth, args, ret, side, ours, + diverged, diff, err, note`. (The harness does not care, but fixed order makes logs + `diff`-able by eye and grep-able.) +2. **Strings** (`hook`, `str`/`wstr` `v`, `n`, `err`, `note`, struct field names, side + names): write `"`, then for each unit: + - `"` → `\"`, `\` → `\\` + - `\n` → `\n`, `\r` → `\r`, `\t` → `\t` (literal backslash-letter) + - any other unit `< 0x20`, and for `str` any byte `>= 0x7f` → `\u00XX` (lowercase hex of + the byte), for `wstr` any unit `>= 0x7f` → `\uXXXX` + - everything else verbatim. Close with `"`. + This is exactly what Python's `json.dumps(ensure_ascii=True)` produces for a + latin-1-decoded byte string (except Python also short-forms `\b` and `\f`; the harness + accepts both). No raw bytes ≥ 0x80 ever appear in the log, so the file is ASCII-only. +3. **Numbers**: integers with `%lld`/`%llu`; floats with `%.9g` / `%.17g`; non-finite floats + as the strings `"nan"`, `"inf"`, `"-inf"`. Never emit `nan`/`inf` bare (invalid JSON). +4. **Booleans** `true`/`false`; **void** `null`. +5. **No trailing commas.** Build arrays/objects with a "first element" flag. +6. **One `fprintf` chain per record, ending in `"}\n"` and a `fflush`** at least on + `compare` divergences and on shim teardown, so a crash mid-run leaves a usable log + (the harness tolerates a truncated last line: it is reported as one invalid record). +7. **Thread safety**: take a lock around the whole record; `call_id` is + `InterlockedIncrement`-ed *before* the original is called, so nested hooks get + increasing ids and `depth` says who is inside whom. +8. `sha256`: any tiny public-domain SHA-256; hex lowercase. + +## 5. `meta` record (first line) + +```json +{"meta":{"format":1,"build":"sots-engine 0.0.3 g1a2b3c4","exe_sha256":"…", + "started":"2026-09-07T18:00:00Z","inline_max":256, + "hooks":{"CfgVar_RegisterKey":{"ftol":0,"ptr":"ignore"}, + "Mars::ParseBlock":{"ftol":1e-6,"ftol_kind":"rel","unordered":["ret.v.items"]}}}} +``` + +- `format`: this spec's version (1). The harness refuses other versions. +- `hooks`: per-hook **policy**, the same keys `tracecmp.py --tolerance` accepts. CLI flags + override the log's policy; the log's policy overrides harness defaults. +- Everything else is informational and copied into the report. + +## 6. Divergence rules + +A call **diverges** when any of: `err` is set; `ret` differs; any `side[name].after` +differs from `ours.side[name].after`; a `side` name exists on one side only (`why: +"missing"`). Comparison is by the typed-value table above, recursively. Paths in diff +entries: `ret`, `side..after`, then `.v[]` for list/set elements, +`.v.` for struct fields, `.`/`[]` inside `json` values. + +```json +{"path":"side.cfg_table.after.v.ForceSingleCore","why":"exact","orig":{"t":"i32","v":1},"ours":{"t":"i32","v":0}} +``` + +`why` ∈ `exact | ftol | type | len | missing | extra | hash | err`. + +**Float tolerance policy** (per hook; default `ftol = 0`, i.e. exact by value with +`nan == nan`): +- `ftol`: number ≥ 0. +- `ftol_kind`: `"abs"` (default) — `|a-b| <= ftol`; `"rel"` — `|a-b| <= ftol * max(|a|,|b|)`; + `"ulp"` — within `ftol` units-in-last-place of the value's own width (f32/f64). +- `nan`, `inf`, `-inf`: equal only to themselves, whatever the tolerance. +- `f32` values are rounded to float32 on both sides before comparing: `%.9g` round-trips + the float32 exactly but not the double it was widened to. `f64` compares as-is. +- `f32` compared to `f64` is a type divergence; do not mix. + +**Unordered fields**: `set` values are always unordered. A policy may additionally list +`unordered: [, …]` to treat specific `list` paths as sets (path syntax as above, +without the `.v[i]` tail). Multiset semantics: element multiplicity matters. + +**Pointers**: `ptr` is ignored unless the policy says `"ptr":"exact"`. Two `ptr` values +where exactly one is `"0x0"` is always a divergence (`null` vs non-null is semantic). + +## 7. Canonical JSON (for `json` values and the oracle bridge) + +The form a host-side C++ test dump must reproduce byte-for-byte so a plain `diff` (or +`tracecmp.py --replay`) works. It is Python's +`json.dumps(obj, sort_keys=True, ensure_ascii=True, separators=(",", ":"))` after +normalization by `oracle_parsers.canonical()`: + +- objects: keys sorted by code point, no whitespace anywhere (`{"a":1,"b":[1,2]}`); +- strings: byte-preserving — the parsers decode cp1252, the canonicalizer re-encodes to + cp1252 and maps each byte to U+00XX, so the escaping in section 4 applies verbatim + (a C++ dump escapes raw bytes, no charset table needed); +- integers: decimal, no leading `+`/zeros; +- floats: rounded to **float32**, printed with `%.9g`, then `.0` appended when the text has + no `.`, `e` or `n` (so `1.0` stays distinguishable from the int `1`, and `-0` → `-0.0`); + non-finite → the strings `"nan"`, `"inf"`, `"-inf"`; +- booleans `true`/`false`; `null` for None; tuples become arrays; dataclasses become + objects of their fields; +- the file ends with a single `\n`. + +## 8. Replay (offline) input + +`tracecmp.py --replay GOLDEN.jsonl IMPL.jsonl`: `GOLDEN` is a normal log (any mode; usually +`trace`). `IMPL` is JSONL whose records need only `call_id`, `ret`, `side` (`side` may be +`{name: tv}` or `{name: {"after": tv}}`; `hook` optional but checked when present). Each +golden call_id must appear in `IMPL` (`why: "missing"` otherwise); extra call_ids in +`IMPL` are warnings. The diff rules are those of section 6, with `IMPL` playing `ours`. diff --git a/verify/harness/compare/mkfixture.py b/verify/harness/compare/mkfixture.py new file mode 100644 index 0000000..71c3d23 --- /dev/null +++ b/verify/harness/compare/mkfixture.py @@ -0,0 +1,474 @@ +#!/usr/bin/env python3 +"""mkfixture.py -- synthetic TRACE_FORMAT.md logs for testing tracecmp.py. + +usage: mkfixture.py OUTDIR [--seed N] [--calls N] + +Writes to OUTDIR: + trace_clean.jsonl golden `trace` log (meta + N calls over 3 hooks) + compare_clean.jsonl `compare` log, ours == original everywhere + compare_bad.jsonl `compare` log with injected divergences (see INJECTED) + replay_impl_ok.jsonl impl output matching trace_clean (for --replay) + replay_impl_bad.jsonl impl output with one wrong ret and one missing call + invalid.jsonl broken lines: bad JSON, missing field, bad type, dup id + +The records are written by emit_record(), a Python transliteration of the +C++ emitter rules in TRACE_FORMAT.md section 4 (fixed key order, hand-rolled +string escaping, %.9g floats) -- NOT json.dumps -- so the tests prove that an +emitter following those rules produces logs the harness reads back exactly. + +Stdlib only. +""" +from __future__ import annotations + +import argparse +import copy +import hashlib +import os +import random +import struct +import sys +from typing import Any + +KEY_ORDER = ["ts", "hook", "mode", "call_id", "thread", "depth", "args", "ret", "side", + "ours", "diverged", "diff", "err", "note"] +INLINE_MAX = 256 +HOOKS = ["CfgVar_RegisterKey", "Manifest_Load", "Mars::ParseBlock"] + + +# --- typed-value constructors ----------------------------------------------------- + +def i32(v, n=None): + return _tv("i32", v, n) + + +def u32(v, n=None): + return _tv("u32", v, n) + + +def u64(v, n=None): + return _tv("u64", v if v < 2**53 else str(v), n) + + +def f32(v, n=None): + if v != v: + return _tv("f32", "nan", n) + if v in (float("inf"), float("-inf")): + return _tv("f32", "inf" if v > 0 else "-inf", n) + return _tv("f32", round_f32(v), n) + + +def boolean(v, n=None): + return _tv("bool", bool(v), n) + + +def s(v: str, n=None): + """cp1252 byte string -> str tv (each byte one code point).""" + return _tv("str", v.encode("cp1252").decode("latin-1"), n) + + +def ptr(addr: int, n=None): + return _tv("ptr", "0x%08x" % addr, n) + + +def enum(v: int, name: str | None = None, n=None): + tv = _tv("enum", v, n) + if name: + tv["name"] = name + return tv + + +def null(): + return {"t": "null", "v": None} + + +def lst(items, n=None): + return _tv("list", list(items), n) + + +def sset(items, n=None): + return _tv("set", list(items), n) + + +def struct_(fields: dict, n=None): + return _tv("struct", dict(fields), n) + + +def jsonv(v, n=None): + return _tv("json", v, n) + + +def by(data: bytes, inline_max: int = INLINE_MAX): + """bytes tv: n + sha256 always; hex inline when small, else head. (No 'n' + name on bytes -- the key is taken by the length; name the side region instead.)""" + tv: dict = {"t": "bytes", "n": len(data), "sha256": hashlib.sha256(data).hexdigest()} + if len(data) <= inline_max: + tv["hex"] = data.hex() + else: + tv["head"] = data[:32].hex() + return tv + + +def _tv(t, v, n): + tv = {"t": t, "v": v} + if n: + tv["n"] = n + return tv + + +def round_f32(x: float) -> float: + return struct.unpack(" str: + out = ['"'] + for c in u: + o = ord(c) + if c == '"': + out.append('\\"') + elif c == "\\": + out.append("\\\\") + elif c == "\n": + out.append("\\n") + elif c == "\r": + out.append("\\r") + elif c == "\t": + out.append("\\t") + elif o < 0x20 or o >= 0x7F: + out.append("\\u%04x" % o) + else: + out.append(c) + out.append('"') + return "".join(out) + + +def fmt_num(v: Any, t: str | None = None) -> str: + if isinstance(v, bool): + return "true" if v else "false" + if isinstance(v, int): + return "%d" % v + if isinstance(v, float): + if v != v: + return '"nan"' + if v in (float("inf"), float("-inf")): + return '"inf"' if v > 0 else '"-inf"' + return ("%.9g" if t == "f32" else "%.17g") % v + raise TypeError(v) + + +def emit_value(x: Any, t: str | None = None, plain: bool = False) -> str: + """Generic JSON value emitter with the spec's escaping. Nested dicts that + look like typed values are emitted as such unless plain=True (json payloads).""" + if x is None: + return "null" + if isinstance(x, (bool, int, float)): + return fmt_num(x, t) + if isinstance(x, str): + return esc(x) + if isinstance(x, list): + return "[" + ",".join(emit_value(e, plain=plain) for e in x) + "]" + if isinstance(x, dict): + if not plain and "t" in x and ("v" in x or x["t"] == "bytes"): + return emit_tv(x) + return "{" + ",".join(esc(k) + ":" + emit_value(v, plain=plain) for k, v in x.items()) + "}" + raise TypeError(x) + + +def emit_tv(tv: dict) -> str: + t = tv["t"] + parts = ['"t":' + esc(t)] + if t == "bytes": + parts.append('"n":%d' % tv["n"]) + parts.append('"sha256":' + esc(tv["sha256"])) + if "hex" in tv: + parts.append('"hex":' + esc(tv["hex"])) + if "head" in tv: + parts.append('"head":' + esc(tv["head"])) + else: + v = tv["v"] + if t in ("list", "set"): + parts.append('"v":[' + ",".join(emit_tv(e) for e in v) + "]") + elif t == "struct": + parts.append('"v":{' + ",".join(esc(k) + ":" + emit_tv(e) for k, e in v.items()) + "}") + elif t == "json": + parts.append('"v":' + emit_value(v, plain=True)) + elif t in ("f32", "f64"): + parts.append('"v":' + (esc(v) if isinstance(v, str) else fmt_num(v, t))) + elif t == "wstr": + parts.append('"v":' + esc(v, wide=True)) + elif v is None: + parts.append('"v":null') + elif isinstance(v, str): + parts.append('"v":' + esc(v)) + else: + parts.append('"v":' + fmt_num(v)) + if "name" in tv: + parts.append('"name":' + esc(tv["name"])) + if "n" in tv and t != "bytes": + parts.append('"n":' + esc(tv["n"])) + return "{" + ",".join(parts) + "}" + + +def emit_side(side: dict) -> str: + ents = [] + for name, ent in side.items(): + inner = [] + if "before" in ent: + inner.append('"before":' + ("null" if ent["before"] is None else emit_tv(ent["before"]))) + inner.append('"after":' + emit_tv(ent["after"])) + ents.append(esc(name) + ":{" + ",".join(inner) + "}") + return "{" + ",".join(ents) + "}" + + +def emit_record(rec: dict) -> str: + parts = [] + for k in KEY_ORDER: + if k not in rec: + continue + v = rec[k] + if k in ("ts", "call_id", "thread", "depth"): + parts.append(esc(k) + ":" + fmt_num(v)) + elif k in ("hook", "mode", "err", "note"): + parts.append(esc(k) + ":" + esc(v)) + elif k == "args": + parts.append('"args":[' + ",".join(emit_tv(a) for a in v) + "]") + elif k == "ret": + parts.append('"ret":' + ("null" if v is None else emit_tv(v))) + elif k == "side": + parts.append('"side":' + emit_side(v)) + elif k == "ours": + parts.append('"ours":{"ret":' + ("null" if v["ret"] is None else emit_tv(v["ret"])) + + ',"side":' + emit_side(v["side"]) + "}") + elif k == "diverged": + parts.append('"diverged":' + ("true" if v else "false")) + elif k == "diff": + parts.append('"diff":' + emit_value(v)) + else: + raise AssertionError(k) + return "{" + ",".join(parts) + "}\n" + + +def emit_meta(meta: dict) -> str: + return '{"meta":' + emit_value(meta) + "}\n" + + +def write_log(path: str, meta: dict | None, records: list[dict], raw_lines: list[str] | None = None) -> None: + with open(path, "w", encoding="utf-8", newline="\n") as f: + if meta is not None: + f.write(emit_meta(meta)) + for r in records: + f.write(emit_record(r)) + for line in raw_lines or []: + f.write(line if line.endswith("\n") else line + "\n") + + +# --- scenario generation ---------------------------------------------------------------- + +def meta(**hooks) -> dict: + return {"format": 1, "build": "mkfixture 1", "exe_sha256": "0" * 64, + "started": "2026-09-07T00:00:00Z", "inline_max": INLINE_MAX, "hooks": hooks} + + +def gen_calls(n: int, seed: int = 1) -> list[dict]: + """n trace calls cycling over HOOKS, with every tv type represented.""" + rng = random.Random(seed) + recs = [] + for cid in range(n): + hook = HOOKS[cid % len(HOOKS)] + ts = 1000 + cid * 37 + if hook == "CfgVar_RegisterKey": + key = rng.choice(["ForceSingleCore", "Résumé", "Tab\tKey", 'Quote"d', "Back\\slash"]) + blob = bytes(rng.getrandbits(8) for _ in range(rng.choice([0, 16, 512]))) + rec = {"ts": ts, "hook": hook, "mode": "trace", "call_id": cid, "thread": 4120, "depth": 0, + "args": [s(key, "key"), i32(rng.randint(-5, 5), "value"), ptr(0x00a36fd0 + cid * 16, "table")], + "ret": boolean(True), + "side": {"cfg_table": {"before": by(blob), "after": by(blob[::-1])}}} + elif hook == "Manifest_Load": + ids = [rng.randint(1, 400) for _ in range(rng.randint(1, 5))] + rec = {"ts": ts, "hook": hook, "mode": "trace", "call_id": cid, "thread": 4120, + "args": [s("Weapons/_weapons.txt", "path")], + "ret": i32(len(ids), "count"), + "side": {"registry": {"after": struct_({ + "entries": lst(struct_({"id": u32(i), "name": s("w%d.weapon" % i)}) for i in ids), + "deleted": sset(u32(i) for i in rng.sample(range(500, 600), 3)), + "flags": enum(rng.randint(0, 3), "MF_%d" % 0), + "big": u64(2**60 + cid), + "nothing": null()})}}} + else: + vals = [round_f32(rng.uniform(-10, 10)) for _ in range(3)] + rec = {"ts": ts, "hook": hook, "mode": "trace", "call_id": cid, "thread": 4124, "depth": 1, + "args": [s("weapon { damage %g }" % vals[0], "text"), u32(cid, "len")], + "ret": jsonv({"weapon": {"damage": vals[0], "range": vals[1], "name": "Laseré", "count": 3, + "ok": True, "tags": ["a", "b"]}}), + "side": {"scratch": {"after": lst(f32(v) for v in vals)}, + "nanbox": {"after": f32(float("nan"))}}} + recs.append(rec) + return recs + + +def to_compare(recs: list[dict]) -> list[dict]: + """Clone trace records into compare records where ours == original.""" + out = [] + for r in recs: + c = dict(r) + c["mode"] = "compare" + c["ours"] = {"ret": _deep(r["ret"]), "side": {k: {"after": _deep(v["after"])} for k, v in r["side"].items()}} + c["diverged"] = False + c["diff"] = [] + out.append(c) + return out + + +def _deep(x): + return copy.deepcopy(x) + + +def inject_divergences(recs: list[dict]) -> tuple[list[dict], list[dict]]: + """Mutate `ours` on a handful of compare records (the INJECTED cases below). + Returns (records, expectations); expectation = {call_id, diverges_exact, + diverges_tol (abs 1e-3), why, path}. Needs >= 12 records (4 per hook).""" + recs = [_deep(r) for r in recs] + exp = [] + by_hook: dict[str, list[dict]] = {} + for r in recs: + by_hook.setdefault(r["hook"], []).append(r) + + cfg = by_hook["CfgVar_RegisterKey"] + # 1. bool ret flipped + cfg[0]["ours"]["ret"] = boolean(False) + exp.append({"call_id": cfg[0]["call_id"], "diverges_exact": True, "diverges_tol": True, "why": "exact", "path": "ret"}) + # 2. bytes region differs (same length, different content) -> hash + orig = cfg[1]["side"]["cfg_table"]["after"] + n = orig["n"] + if n == 0: # make the region non-empty on both sides so a content diff is possible + n = 16 + cfg[1]["side"]["cfg_table"]["after"] = orig = by(bytes(range(n))) + data = bytes(n) + if data.hex() == orig.get("hex"): + data = bytes([255]) + data[1:] + cfg[1]["ours"]["side"]["cfg_table"]["after"] = by(data) + exp.append({"call_id": cfg[1]["call_id"], "diverges_exact": True, "diverges_tol": True, "why": "hash", + "path": "side.cfg_table.after"}) + # 3. shim lies: says diverged=true on an identical record -> harness warning only + cfg[2]["diverged"] = True + exp.append({"call_id": cfg[2]["call_id"], "diverges_exact": False, "diverges_tol": False, "why": None, + "path": None, "warn": "shim said diverged"}) + + man = by_hook["Manifest_Load"] + # 4. set reordered -> NOT a divergence + st = man[0]["ours"]["side"]["registry"]["after"] + st["v"]["deleted"]["v"] = list(reversed(st["v"]["deleted"]["v"])) + exp.append({"call_id": man[0]["call_id"], "diverges_exact": False, "diverges_tol": False, "why": None, "path": None}) + # 5. list reordered (2+ entries) or entry id changed -> divergence + st = man[1]["ours"]["side"]["registry"]["after"] + ent = st["v"]["entries"]["v"] + if len(ent) >= 2 and ent[0] != ent[-1]: + ent.reverse() + else: + ent[0]["v"]["id"] = u32(ent[0]["v"]["id"]["v"] + 1) + exp.append({"call_id": man[1]["call_id"], "diverges_exact": True, "diverges_tol": True, "why": "exact", + "path": "side.registry.after.v.entries"}) + # 6. struct field missing + del man[2]["ours"]["side"]["registry"]["after"]["v"]["big"] + exp.append({"call_id": man[2]["call_id"], "diverges_exact": True, "diverges_tol": True, "why": "missing", + "path": "side.registry.after.v.big"}) + + mp = by_hook["Mars::ParseBlock"] + # 7. f32 nudged by 1e-5 -> diverges exact, passes abs 1e-3 + sc = mp[0]["ours"]["side"]["scratch"]["after"]["v"] + sc[1] = f32(sc[1]["v"] + 1e-5) + exp.append({"call_id": mp[0]["call_id"], "diverges_exact": True, "diverges_tol": False, "why": "exact", + "path": "side.scratch.after.v[1]"}) + # 8. json float nudged by 0.5 -> diverges both + rv = mp[1]["ours"]["ret"] = _deep(mp[1]["ret"]) + rv["v"]["weapon"]["range"] = rv["v"]["weapon"]["range"] + 0.5 + exp.append({"call_id": mp[1]["call_id"], "diverges_exact": True, "diverges_tol": True, "why": "exact", + "path": "ret.weapon.range"}) + # 9. err record + mp[2]["err"] = "ours threw std::bad_alloc" + del mp[2]["ours"] + exp.append({"call_id": mp[2]["call_id"], "diverges_exact": True, "diverges_tol": True, "why": "err", "path": "call"}) + # 10. ptr differs -> NOT a divergence (default policy); type mismatch i32 vs u32 -> divergence + if len(cfg) > 3: + cfg[3]["ours"]["ret"] = boolean(True) + cfg[3]["ours"]["side"]["cfg_table"]["after"] = cfg[3]["side"]["cfg_table"]["after"] + exp.append({"call_id": cfg[3]["call_id"], "diverges_exact": False, "diverges_tol": False, "why": None, "path": None}) + if len(man) > 3: + man[3]["ours"]["ret"] = u32(man[3]["ret"]["v"]) + exp.append({"call_id": man[3]["call_id"], "diverges_exact": True, "diverges_tol": True, "why": "type", "path": "ret"}) + # the shim's own verdict is truthful for every injected case except #3 + by_id = {r["call_id"]: r for r in recs} + for e in exp: + if "warn" not in e and "diverged" in by_id[e["call_id"]]: + by_id[e["call_id"]]["diverged"] = e["diverges_exact"] + return recs, exp + + +def impl_records(recs: list[dict]) -> list[dict]: + """Implementation-output records (replay input) equal to the golden outputs.""" + return [{"call_id": r["call_id"], "hook": r["hook"], "ret": r["ret"], + "side": {k: v["after"] for k, v in r["side"].items()}} for r in recs] + + +def emit_impl(path: str, recs: list[dict]) -> None: + with open(path, "w", encoding="utf-8", newline="\n") as f: + for r in recs: + side = "{" + ",".join(esc(k) + ":" + emit_tv(v) for k, v in r["side"].items()) + "}" + f.write('{"call_id":%d,"hook":%s,"ret":%s,"side":%s}\n' % ( + r["call_id"], esc(r["hook"]), "null" if r["ret"] is None else emit_tv(r["ret"]), side)) + + +def invalid_lines() -> list[str]: + return [ + '{"ts":1,"hook":"X","mode":"trace","call_id":900,"thread":1,"args":[],"ret":null,"side":{}}', # ok + '{"ts":1,"hook":"X","mode":"trace","call_id":901,"thread":1,"args":[],"ret":null,"side":{}', # bad JSON + '{"ts":1,"hook":"X","mode":"trace","call_id":902,"thread":1,"args":[]}', # missing ret/side + '{"ts":1,"hook":"X","mode":"trace","call_id":903,"thread":1,"args":[{"t":"i32","v":"x"}],"ret":null,"side":{}}', # bad int + '{"ts":1,"hook":"X","mode":"trace","call_id":900,"thread":1,"args":[],"ret":null,"side":{}}', # dup id + '{"ts":1,"hook":"X","mode":"compare","call_id":904,"thread":1,"args":[],"ret":null,"side":{}}', # compare w/o ours + '{"ts":1,"hook":"X","mode":"trace","call_id":905,"thread":1,"args":[{"t":"bytes","n":2,"sha256":"zz"}],"ret":null,"side":{}}', + '{"ts":1,"hook":"X","mode":"trace","call_id":906,"thread":1,"args":[],"ret":{"t":"u8","v":300},"side":{}}', + '{"ts":1,"hook":"X","mode":"trace","call_id":907,"thread":1,"args":[],"ret":null,"side":{},"bogus":1}', # ok + warning + ] + + +def build_all(outdir: str, n: int = 12, seed: int = 1) -> dict: + os.makedirs(outdir, exist_ok=True) + trace = gen_calls(n, seed) + m = meta(**{"Mars::ParseBlock": {"ftol": 0}}) + paths = {k: os.path.join(outdir, k + ".jsonl") for k in + ("trace_clean", "compare_clean", "compare_bad", "replay_impl_ok", "replay_impl_bad", "invalid")} + write_log(paths["trace_clean"], m, trace) + clean = to_compare(trace) + write_log(paths["compare_clean"], m, clean) + bad, exp = inject_divergences(clean) + write_log(paths["compare_bad"], m, bad) + impl = impl_records(trace) + emit_impl(paths["replay_impl_ok"], impl) + impl_bad = _deep(impl) + impl_bad[0]["ret"] = boolean(False) # wrong ret on call 0 + dropped = impl_bad.pop(1)["call_id"] # missing call 1 + impl_bad.append({"call_id": 9999, "hook": "Extra", "ret": None, "side": {}}) # extra -> warning + emit_impl(paths["replay_impl_bad"], impl_bad) + write_log(paths["invalid"], m, [], invalid_lines()) + return {"paths": paths, "trace": trace, "compare_bad": bad, "expect": exp, + "replay_bad": {"wrong": 0, "missing": dropped, "extra": 9999}} + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0]) + ap.add_argument("outdir") + ap.add_argument("--seed", type=int, default=1) + ap.add_argument("--calls", type=int, default=12) + a = ap.parse_args(argv) + r = build_all(a.outdir, a.calls, a.seed) + for k, p in r["paths"].items(): + print(f"{k:16s} {p}") + print(f"injected divergences: {sum(1 for e in r['expect'] if e['diverges_exact'])} " + f"(tolerant: {sum(1 for e in r['expect'] if e['diverges_tol'])})") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/verify/harness/compare/oracle_parsers.py b/verify/harness/compare/oracle_parsers.py new file mode 100644 index 0000000..1b75a8b --- /dev/null +++ b/verify/harness/compare/oracle_parsers.py @@ -0,0 +1,273 @@ +#!/usr/bin/env python3 +"""oracle_parsers.py -- run the proven Python data-file parsers (verify/parsers/) +over a list of files and emit the CANONICAL JSON form (TRACE_FORMAT.md section 7) +that a C++ reimplementation's test dump must reproduce byte-for-byte. + +usage: + oracle_parsers.py --root EXTRACT_DIR [--out-dir DIR] [--jsonl OUT] [--kind KIND] FILE... + oracle_parsers.py --root EXTRACT_DIR --all [--out-dir DIR] [--jsonl OUT] + + --root the .gob extract root; file kinds are chosen from the path relative + to it exactly as verify.py does (verify.kind_of) + --all walk --root and take every file with a known data kind + --out-dir write /.json per file (canonical text + "\\n") + --jsonl write one TRACE_FORMAT record per file (hook "parse:", + call_id = index in the sorted file list, ret = {"t":"json","v":...}) + -- the golden log for `tracecmp.py --replay IMPL.jsonl OUT.jsonl` + --kind force a kind for every FILE (brace, effect, csv, manifest, rows, kv) + With neither --out-dir nor --jsonl, the canonical text of each file goes to stdout. + +Canonical form (what a C++ dump must produce): + * JSON object keys sorted by code point; no whitespace; arrays in parse order + * strings byte-preserving: the raw cp1252 byte b -> code point b; then escaped + as json.dumps(ensure_ascii=True) does: \\" \\\\ \\n \\r \\t \\b \\f, every other + unit < 0x20 or >= 0x7f as \\u00xx (lowercase) + * ints decimal; bools true/false; None -> null; tuples -> arrays; + Manifest -> {"deleted":[...],"entries":[[id,name],...]} + * floats: round to float32, print %.9g, append ".0" if no '.', 'e' or 'n'; + non-finite -> "nan" / "inf" / "-inf" (strings) + * text ends with one "\\n" + * a file the parser rejects canonicalizes to {"_error":true} + +Result shapes are the parsers' own (see their docstrings): mars_data dict with +repeated keys -> lists, "_items" for bare items; effect_txt ordered [key, value] +pairs; flat_kv dict / rows; manifest as above; csv list of rows. + +Stdlib only. +""" +from __future__ import annotations + +import argparse +import dataclasses +import json +import math +import os +import struct +import sys + +HERE = os.path.dirname(os.path.abspath(__file__)) +PARSERS = os.path.normpath(os.path.join(HERE, "..", "..", "parsers")) +sys.path.insert(0, PARSERS) +import effect_txt # noqa: E402 +import flat_kv # noqa: E402 +import manifest # noqa: E402 +import mars_data # noqa: E402 +import verify as _verify # noqa: E402 (kind_of; main is __main__-guarded) + +kind_of = _verify.kind_of +KINDS = ("brace", "effect", "csv", "manifest", "rows", "kv") +SKIP_KINDS = ("hlsl", "prose", "other") + + +# --- canonical form -------------------------------------------------------------------- + +def canon_float(x: float) -> str: + if math.isnan(x): + return '"nan"' + if math.isinf(x): + return '"inf"' if x > 0 else '"-inf"' + x = struct.unpack(" str: + """Byte-preserving: re-encode the parser's cp1252 text so each byte is one + code point, then escape like json.dumps(ensure_ascii=True).""" + try: + b = s.encode("cp1252") + except UnicodeEncodeError: + b = s.encode("latin-1", "replace") + return json.dumps(b.decode("latin-1"), ensure_ascii=True) + + +def canonical(obj): + """Normalize a parser result to plain JSON-able Python (floats stay float, + strings become byte-mapped, tuples -> lists, dataclasses -> dicts).""" + if dataclasses.is_dataclass(obj) and not isinstance(obj, type): + if isinstance(obj, manifest.Manifest): + return {"entries": [[i, canonical(n)] for i, n in obj.entries], + "deleted": list(obj.deleted)} + return canonical(dataclasses.asdict(obj)) + if isinstance(obj, dict): + return {str(k): canonical(v) for k, v in obj.items()} + if isinstance(obj, (list, tuple)): + return [canonical(v) for v in obj] + if isinstance(obj, bool) or obj is None or isinstance(obj, int): + return obj + if isinstance(obj, float): + if math.isnan(obj): + return "nan" + if math.isinf(obj): + return "inf" if obj > 0 else "-inf" + return struct.unpack(" str: + """Canonical text (no trailing newline) of an already-canonical() object.""" + out: list[str] = [] + _emit(obj, out) + return "".join(out) + + +def _emit(x, out: list[str]) -> None: + if x is None: + out.append("null") + elif isinstance(x, bool): + out.append("true" if x else "false") + elif isinstance(x, int): + out.append("%d" % x) + elif isinstance(x, float): + out.append(canon_float(x)) + elif isinstance(x, str): + out.append(json.dumps(x, ensure_ascii=True)) + elif isinstance(x, list): + out.append("[") + for i, e in enumerate(x): + if i: + out.append(",") + _emit(e, out) + out.append("]") + elif isinstance(x, dict): + out.append("{") + for i, k in enumerate(sorted(x)): + if i: + out.append(",") + out.append(json.dumps(k, ensure_ascii=True)) + out.append(":") + _emit(x[k], out) + out.append("}") + else: + raise TypeError(type(x).__name__) + + +# --- parsing dispatch --------------------------------------------------------------------- + +def parse_kind(path: str, kind: str) -> tuple[object, list[str]]: + """-> (result, notes). Raises on parse failure.""" + notes: list[str] = [] + base = kind.split(":")[0] + if base == "brace": + w: list[str] = [] + obj = mars_data.parse_file(path, warnings=w) + notes += w + elif base == "effect": + obj = effect_txt.parse_file(path) + elif base == "csv": + obj = manifest.parse_csv_file(path) + elif base == "manifest": + obj = manifest.parse_manifest_file(path) + if obj.problems: + raise ValueError("; ".join(obj.problems)) + elif base == "rows": + obj = flat_kv.parse_rows_file(path) + elif base == "kv": + txt = manifest.read_text(path) + obj = flat_kv.parse_kv(txt) + d = flat_kv.duplicates(txt) + if d: + notes.append(f"duplicate keys {d}") + else: + raise ValueError(f"no parser for kind {kind!r}") + return obj, notes + + +def oracle_file(path: str, kind: str) -> tuple[dict | list, list[str], str | None]: + """-> (canonical object, notes, error). Never raises for parse errors.""" + try: + obj, notes = parse_kind(path, kind) + except Exception as e: # noqa: BLE001 + return {"_error": True}, [], f"{type(e).__name__}: {e}" + return canonical(obj), notes, None + + +def collect(root: str, files: list[str], all_files: bool, forced_kind: str | None) -> list[tuple[str, str, str]]: + """-> sorted [(abs path, rel, kind)] with data kinds only.""" + items = [] + if all_files: + for dp, _, fn in os.walk(root): + for f in fn: + items.append(os.path.join(dp, f)) + items += files + out = [] + for p in items: + p = os.path.abspath(p) + rel = os.path.relpath(p, root).replace(os.sep, "/") if root else os.path.basename(p) + k = forced_kind or kind_of(rel) + if k.split(":")[0] in SKIP_KINDS: + continue + out.append((p, rel, k)) + out.sort(key=lambda t: t[1]) + return out + + +def jsonl_record(idx: int, rel: str, kind: str, canon, notes: list[str], err: str | None) -> str: + """One TRACE_FORMAT record (json.dumps is spec-conformant for our values).""" + rec = {"ts": idx, "hook": "parse:" + kind.split(":")[0], "mode": "trace", "call_id": idx, "thread": 0, + "args": [{"t": "str", "v": rel.encode("cp1252", "replace").decode("latin-1"), "n": "path"}], + "ret": {"t": "json", "v": canon}, "side": {}} + if err: + rec["err"] = err + if notes: + rec["note"] = "; ".join(notes) + return json.dumps(rec, ensure_ascii=True, separators=(",", ":"), allow_nan=False) + "\n" + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0], formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("files", nargs="*", metavar="FILE") + ap.add_argument("--root", default=None, help=".gob extract root (kinds from relative path)") + ap.add_argument("--all", action="store_true", help="walk --root for every data file") + ap.add_argument("--out-dir") + ap.add_argument("--jsonl") + ap.add_argument("--kind", choices=KINDS) + a = ap.parse_args(argv) + if a.all and not a.root: + ap.error("--all needs --root") + if not a.all and not a.files: + ap.error("no FILE given (or use --all)") + root = os.path.abspath(a.root) if a.root else None + items = collect(root, a.files, a.all, a.kind) + if not items: + print("error: no data files selected", file=sys.stderr) + return 2 + + jf = None + if a.jsonl: + os.makedirs(os.path.dirname(os.path.abspath(a.jsonl)), exist_ok=True) + jf = open(a.jsonl, "w", encoding="utf-8", newline="\n") + jf.write(json.dumps({"meta": {"format": 1, "build": "oracle_parsers", "started": "", "inline_max": 0, + "hooks": {}, "root": root or ""}}, separators=(",", ":")) + "\n") + errors = 0 + for idx, (p, rel, k) in enumerate(items): + canon, notes, err = oracle_file(p, k) + if err: + errors += 1 + print(f"error: {rel}: {err}", file=sys.stderr) + text = dumps(canon) + "\n" + if a.out_dir: + op = os.path.join(a.out_dir, rel + ".json") + os.makedirs(os.path.dirname(op), exist_ok=True) + with open(op, "w", encoding="utf-8", newline="\n") as f: + f.write(text) + if jf: + jf.write(jsonl_record(idx, rel, k, canon, notes, err)) + if not a.out_dir and not jf: + sys.stdout.write(text) + if jf: + jf.close() + print(f"{len(items)} file(s), {errors} parse error(s)" + + (f" -> {a.out_dir}" if a.out_dir else "") + (f" -> {a.jsonl}" if a.jsonl else ""), file=sys.stderr) + return 0 if not errors else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/verify/harness/compare/test_tracecmp.py b/verify/harness/compare/test_tracecmp.py new file mode 100644 index 0000000..af8b356 --- /dev/null +++ b/verify/harness/compare/test_tracecmp.py @@ -0,0 +1,497 @@ +#!/usr/bin/env python3 +"""Tests for tracecmp.py / mkfixture.py / oracle_parsers.py (synthetic data only). + + /usr/bin/python3 -m unittest -v test_tracecmp + /usr/bin/python3 test_tracecmp.py + +Covers: the reference emitter round-trips through json; record/tv validation and +exit code 2; exact and tolerant (abs/rel/ulp, CLI + meta policy, per-hook) diffs; +set/unordered/ptr/bytes/json rules; replay (missing/extra/wrong); --hook, +--json-out; the oracle bridge's canonical form and a bridge->replay round trip. +""" +from __future__ import annotations + +import io +import json +import os +import shutil +import sys +import tempfile +import unittest +from contextlib import redirect_stderr, redirect_stdout + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import mkfixture as mk # noqa: E402 +import oracle_parsers as op # noqa: E402 +import tracecmp as tc # noqa: E402 + +TMP = tempfile.mkdtemp(prefix="tracecmp-test-") +FX = mk.build_all(os.path.join(TMP, "fx"), n=12, seed=7) +P = FX["paths"] + + +def run(*argv) -> tuple[int, dict, str]: + """tracecmp.main with --json-out; -> (exit, report, stdout).""" + out = os.path.join(TMP, "rep%d.json" % (abs(hash(argv)) % 10**8)) + buf = io.StringIO() + with redirect_stdout(buf): + rc = tc.main(list(argv) + ["--json-out", out]) + with open(out) as f: + return rc, json.load(f), buf.getvalue() + + +def write_log(name: str, recs: list[dict], meta: dict | None = None) -> str: + p = os.path.join(TMP, name) + mk.write_log(p, meta if meta is not None else mk.meta(), recs) + return p + + +def rec(cid: int, ret, ours_ret=None, side=None, ours_side=None, hook="H", **kw) -> dict: + r = {"ts": cid, "hook": hook, "mode": "compare", "call_id": cid, "thread": 1, "args": [], + "ret": ret, "side": side or {}, "ours": {"ret": ret if ours_ret is None else ours_ret, + "side": ours_side if ours_side is not None else {k: {"after": v["after"]} for k, v in (side or {}).items()}}, + "diverged": False, "diff": []} + r.update(kw) + return r + + +def diffs_of(*recs, **policy) -> list[dict]: + pol = tc.Policy().merged(policy) + out = [] + for r in recs: + d, _ = tc.compare_record(r, pol) + out.append(d) + return out + + +class EmitterTest(unittest.TestCase): + def test_roundtrip_every_fixture_record(self): + def f32norm(x): + """%.9g round-trips the float32, not the widened double: normalize.""" + if isinstance(x, dict): + if x.get("t") == "f32" and isinstance(x.get("v"), float): + return {**x, "v": mk.round_f32(x["v"])} + return {k: f32norm(v) for k, v in x.items()} + if isinstance(x, list): + return [f32norm(v) for v in x] + return x + for r in FX["trace"] + FX["compare_bad"]: + line = mk.emit_record(r) + self.assertTrue(line.endswith("}\n")) + self.assertTrue(line.isascii()) + back = json.loads(line) + self.assertEqual(f32norm(back), f32norm(r)) + self.assertNotEqual(back, None) + + def test_string_escaping_matches_python_ensure_ascii(self): + for s in ['plain', 'Résumé', 'Tab\tKey', 'Quote"d', 'Back\\slash', 'nl\nx', 'cr\rx', '\x01\x7f\xff', '€']: + u = s.encode("cp1252").decode("latin-1") + self.assertEqual(json.loads(mk.esc(u)), u) + self.assertEqual(mk.esc(u), json.dumps(u, ensure_ascii=True)) + self.assertEqual(mk.esc("\x08\x0c"), '"\\u0008\\u000c"') # long form is also accepted + self.assertEqual(json.loads(mk.esc("\x08\x0c")), "\x08\x0c") + + def test_float_and_bigint_forms(self): + self.assertEqual(mk.emit_tv(mk.f32(1.5)), '{"t":"f32","v":1.5}') + self.assertEqual(mk.emit_tv(mk.f32(float("nan"))), '{"t":"f32","v":"nan"}') + self.assertEqual(mk.emit_tv(mk.f32(float("-inf"))), '{"t":"f32","v":"-inf"}') + self.assertEqual(mk.emit_tv(mk.u64(2**64 - 1)), '{"t":"u64","v":"18446744073709551615"}') + self.assertEqual(mk.emit_tv(mk.null()), '{"t":"null","v":null}') + b = mk.by(b"\x00\xff") + self.assertEqual(json.loads(mk.emit_tv(b)), b) + self.assertEqual(b["hex"], "00ff") + self.assertEqual(mk.by(b"")["sha256"], "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855") + big = mk.by(bytes(range(256)) * 2) + self.assertNotIn("hex", big) + self.assertEqual(len(big["head"]), 64) + + +class ValidationTest(unittest.TestCase): + def test_invalid_log_exit_2(self): + rc, rep, _ = run(P["invalid"]) + self.assertEqual(rc, 2) + self.assertEqual(rep["totals"]["invalid_records"], 7) + self.assertEqual(rep["totals"]["calls"], 2) + errs = " | ".join(e["error"] for e in rep["invalid"]) + for needle in ("not JSON", "missing required field", "i32.v must be", "duplicate call_id 900", + "needs ours", "sha256", "u8.v=300 out of range"): + self.assertIn(needle, errs) + self.assertTrue(any("unknown field 'bogus'" in w for w in rep["warnings"])) + + def test_skip_invalid(self): + rc, rep, _ = run(P["invalid"], "--skip-invalid") + self.assertEqual(rc, 0) + + def test_meta_format_version(self): + p = write_log("badmeta.jsonl", [], meta={"format": 2}) + rc, rep, _ = run(p) + self.assertEqual(rc, 2) + self.assertIn("unsupported format", rep["invalid"][0]["error"]) + + def test_tv_validation_details(self): + def errs(tv): + e = [] + tc.validate_tv(tv, "x", e) + return e + self.assertEqual(errs(mk.i32(5)), []) + self.assertTrue(errs({"t": "i32"})) + self.assertTrue(errs({"t": "nope", "v": 1})) + self.assertTrue(errs({"t": "str", "v": "€"})) # code point > U+00FF + self.assertEqual(errs({"t": "wstr", "v": "€"}), []) + self.assertTrue(errs({"t": "ptr", "v": "a36fd0"})) + self.assertTrue(errs({"t": "bytes", "n": 2, "sha256": "a" * 64, "hex": "00"})) # hex len != 2n + self.assertTrue(errs({"t": "bool", "v": 1})) + self.assertEqual(errs({"t": "i64", "v": "-9223372036854775808"}), []) + self.assertTrue(errs({"t": "i64", "v": "9223372036854775808"})) + self.assertTrue(errs({"t": "f32", "v": "huge"})) + self.assertEqual(errs({"t": "f32", "v": "inf"}), []) + self.assertTrue(errs({"t": "list", "v": [{"t": "i32", "v": "x"}]})) + self.assertTrue(errs({"t": "struct", "v": [1]})) + + def test_replace_and_trace_records_not_compared(self): + r = rec(1, mk.i32(1), mk.i32(2)) + r["mode"] = "replace" + del r["ours"], r["diverged"], r["diff"] + t = rec(2, mk.i32(1), mk.i32(2)) + t["mode"] = "trace" + t["err"] = "capture failed" + rc, rep, _ = run(write_log("modes.jsonl", [r, t])) + self.assertEqual(rc, 0) + self.assertEqual(rep["hooks"]["H"]["modes"], {"replace": 1, "trace": 1}) + self.assertEqual(rep["hooks"]["H"]["errors"], 1) + self.assertEqual(rep["hooks"]["H"]["compared"], 0) + + +class CompareTest(unittest.TestCase): + def test_clean_log_is_clean(self): + rc, rep, out = run(P["compare_clean"]) + self.assertEqual(rc, 0) + self.assertEqual(rep["totals"], {"calls": 12, "compared": 12, "diverged": 0, "invalid_records": 0}) + self.assertIn("| CfgVar_RegisterKey | 4 |", out) + + def test_injected_divergences_exact(self): + rc, rep, out = run(P["compare_bad"], "--first", "20") + self.assertEqual(rc, 1) + got = {cid for s in rep["hooks"].values() for cid in s["diverged_call_ids"]} + want = {e["call_id"] for e in FX["expect"] if e["diverges_exact"]} + self.assertEqual(got, want) + first = {d["call_id"]: d["diff"] for s in rep["hooks"].values() for d in s["diffs"]} + for e in FX["expect"]: + if not e["diverges_exact"]: + continue + d0 = first[e["call_id"]][0] + self.assertEqual(d0["why"], e["why"], (e, d0)) + self.assertTrue(d0["path"].startswith(e["path"]), (e, d0)) + lie = [e for e in FX["expect"] if e.get("warn")][0] + self.assertTrue(any(f"call {lie['call_id']}: shim said diverged=True" in w for w in rep["warnings"])) + self.assertEqual(len(rep["warnings"]), 1) # truthful verdicts everywhere else + + def test_injected_divergences_tolerant(self): + rc, rep, _ = run(P["compare_bad"], "--tolerance", "1e-3") + self.assertEqual(rc, 1) + got = {cid for s in rep["hooks"].values() for cid in s["diverged_call_ids"]} + want = {e["call_id"] for e in FX["expect"] if e["diverges_tol"]} + self.assertEqual(got, want) + + def test_per_hook_tolerance_and_meta_policy(self): + nudged = [e["call_id"] for e in FX["expect"] if e["diverges_exact"] and not e["diverges_tol"]][0] + # per-hook CLI on the wrong hook: still diverges + rc, rep, _ = run(P["compare_bad"], "--tolerance", "Manifest_Load=1e-3", "--hook", "Mars::ParseBlock") + self.assertIn(nudged, rep["hooks"]["Mars::ParseBlock"]["diverged_call_ids"]) + # per-hook CLI on the right hook: passes + rc, rep, _ = run(P["compare_bad"], "--tolerance", "Mars::ParseBlock=1e-3", "--hook", "Mars::ParseBlock") + self.assertNotIn(nudged, rep["hooks"]["Mars::ParseBlock"]["diverged_call_ids"]) + # meta policy in the log itself, no CLI + bad = FX["compare_bad"] + p = write_log("metapol.jsonl", bad, meta=mk.meta(**{"Mars::ParseBlock": {"ftol": 1e-3}})) + rc, rep, _ = run(p, "--hook", "Mars::ParseBlock") + self.assertNotIn(nudged, rep["hooks"]["Mars::ParseBlock"]["diverged_call_ids"]) + # CLI default overrides meta + rc, rep, _ = run(p, "--hook", "Mars::ParseBlock", "--tolerance", "0") + self.assertIn(nudged, rep["hooks"]["Mars::ParseBlock"]["diverged_call_ids"]) + + def test_float_kinds(self): + a, b = 1000.0, 1000.001 + self.assertFalse(tc.floats_equal(a, b, tc.Policy())) + self.assertTrue(tc.floats_equal(a, b, tc.Policy(ftol=1e-2))) + self.assertTrue(tc.floats_equal(a, b, tc.Policy(ftol=1e-5, ftol_kind="rel"))) + self.assertFalse(tc.floats_equal(a, b, tc.Policy(ftol=1e-7, ftol_kind="rel"))) + x = mk.round_f32(1.0) + y = mk.round_f32(1.0 + 2 ** -23) # 1 ulp of f32 + self.assertTrue(tc.floats_equal(x, y, tc.Policy(ftol=1, ftol_kind="ulp"), "f32")) + self.assertFalse(tc.floats_equal(x, y, tc.Policy(ftol=0.5, ftol_kind="ulp"), "f32")) + self.assertFalse(tc.floats_equal(x, y, tc.Policy(ftol=1, ftol_kind="ulp"), "f64")) + nan, inf = float("nan"), float("inf") + self.assertTrue(tc.floats_equal(nan, nan, tc.Policy(ftol=1e9))) + self.assertFalse(tc.floats_equal(nan, 0.0, tc.Policy(ftol=1e9))) + self.assertFalse(tc.floats_equal(inf, -inf, tc.Policy(ftol=1e9))) + self.assertTrue(tc.floats_equal(inf, inf, tc.Policy())) + # via records: f32 vs f64 is a type divergence; "nan" strings compare equal + d = diffs_of(rec(1, {"t": "f32", "v": 1.0}, {"t": "f64", "v": 1.0}), + rec(2, {"t": "f32", "v": "nan"}, {"t": "f32", "v": "nan"})) + self.assertEqual([x["why"] for x in d[0]], ["type"]) + self.assertEqual(d[1], []) + + def test_ints_strings_enums(self): + d = diffs_of(rec(1, mk.u64(2**60), {"t": "u64", "v": str(2**60)}), + rec(2, mk.s("Résumé"), mk.s("Resume")), + rec(3, mk.enum(1, "A"), mk.enum(1, "B")), + rec(4, mk.enum(1), mk.enum(2)), + rec(5, mk.boolean(True), mk.i32(1))) + self.assertEqual(d[0], []) + self.assertEqual(d[1][0]["why"], "exact") + self.assertEqual(d[2], []) # symbolic name is informational + self.assertEqual(d[3][0]["why"], "exact") + self.assertEqual(d[4][0]["why"], "type") + + def test_ptr_policy(self): + r = rec(1, mk.ptr(0x1000), mk.ptr(0x2000)) + self.assertEqual(diffs_of(r)[0], []) + self.assertEqual(diffs_of(r, ptr="exact")[0][0]["why"], "exact") + self.assertEqual(diffs_of(rec(2, mk.ptr(0), mk.ptr(0x2000)))[0][0]["why"], "exact") + rc, rep, _ = run(write_log("ptr.jsonl", [r]), "--ptr", "exact") + self.assertEqual(rc, 1) + + def test_set_and_unordered(self): + a = mk.sset([mk.i32(1), mk.i32(1), mk.i32(2)]) + self.assertEqual(diffs_of(rec(1, a, mk.sset([mk.i32(2), mk.i32(1), mk.i32(1)])))[0], []) + d = diffs_of(rec(2, a, mk.sset([mk.i32(1), mk.i32(2), mk.i32(2)])))[0] + self.assertEqual(d[0]["why"], "exact") # multiset: multiplicity matters + d = diffs_of(rec(3, a, mk.sset([mk.i32(1), mk.i32(2)])))[0] + self.assertEqual(d[0]["why"], "len") + l1, l2 = mk.lst([mk.i32(1), mk.i32(2)]), mk.lst([mk.i32(2), mk.i32(1)]) + self.assertEqual(diffs_of(rec(4, l1, l2))[0][0]["why"], "exact") + self.assertEqual(diffs_of(rec(4, l1, l2), unordered=["ret"])[0], []) + rc, rep, _ = run(write_log("unord.jsonl", [rec(4, l1, l2, hook="Hk")]), "--unordered", "Hk=ret") + self.assertEqual(rc, 0) + rc, rep, _ = run(write_log("unord2.jsonl", [rec(4, l1, l2, hook="Hk")]), "--unordered", "Other=ret") + self.assertEqual(rc, 1) + + def test_struct_and_side(self): + s1 = mk.struct_({"a": mk.i32(1), "b": mk.s("x")}) + s2 = mk.struct_({"a": mk.i32(1), "c": mk.s("x")}) + d = diffs_of(rec(1, s1, s2))[0] + self.assertEqual([(x["path"], x["why"]) for x in d], [("ret.v.b", "missing"), ("ret.v.c", "extra")]) + side = {"r1": {"before": mk.i32(0), "after": mk.i32(1)}, "r2": {"after": mk.i32(5)}} + ours = {"r1": {"after": mk.i32(1)}, "r3": {"after": mk.i32(5)}} + d = diffs_of(rec(2, None, None, side=side, ours_side=ours))[0] + self.assertEqual([(x["path"], x["why"]) for x in d], [("side.r2.after", "missing"), ("side.r3.after", "extra")]) + # 'before' mismatch is a warning, not a divergence + ours = {"r1": {"before": mk.i32(9), "after": mk.i32(1)}, "r2": {"after": mk.i32(5)}} + dd, warns = tc.compare_record(rec(3, None, None, side=side, ours_side=ours), tc.Policy()) + self.assertEqual(dd, []) + self.assertTrue(any("snapshot" in w for w in warns)) + # void vs value + self.assertEqual(diffs_of(rec(4, None, mk.i32(1)))[0][0]["why"], "extra") + + def test_bytes(self): + a, b = mk.by(b"abcdef"), mk.by(b"abcXef") + d = diffs_of(rec(1, a, b))[0] + self.assertEqual(d[0]["why"], "hash") + self.assertEqual(d[0]["first_diff_offset"], 3) + self.assertEqual(diffs_of(rec(2, a, mk.by(b"abcdefg")))[0][0]["why"], "len") + big1, big2 = mk.by(bytes(300)), mk.by(bytes(299) + b"\x01") + d = diffs_of(rec(3, big1, big2))[0] + self.assertEqual(d[0]["why"], "hash") + self.assertNotIn("first_diff_offset", d[0]) + self.assertEqual(diffs_of(rec(4, mk.by(b""), mk.by(b"")))[0], []) + + def test_json_values(self): + j = lambda v: mk.jsonv(v) # noqa: E731 + self.assertEqual(diffs_of(rec(1, j({"a": [1, 2.5, "x", True, None]}), j({"a": [1, 2.5, "x", True, None]})))[0], []) + d = diffs_of(rec(2, j({"a": 1}), j({"a": 1.0})))[0] + self.assertEqual(d[0]["why"], "type") + self.assertEqual(diffs_of(rec(2, j({"a": 1}), j({"a": 1.0})), numeric=True)[0], []) + d = diffs_of(rec(3, j({"a": {"b": [1, 2]}}), j({"a": {"b": [1]}, "c": 0})))[0] + self.assertEqual([(x["path"], x["why"]) for x in d], [("ret.c", "extra"), ("ret.a.b", "len")]) + d = diffs_of(rec(4, j({"f": 1.0}), j({"f": 1.0 + 1e-7})))[0] + self.assertEqual(d[0]["why"], "exact") + self.assertEqual(diffs_of(rec(4, j({"f": 1.0}), j({"f": 1.0 + 1e-7})), ftol=1e-6)[0], []) + self.assertEqual(diffs_of(rec(5, j({"x": True}), j({"x": 1})))[0][0]["why"], "exact") + d = diffs_of(rec(6, j([3, 1, 2]), j([1, 2, 3])), unordered=["ret"])[0] + self.assertEqual(d, []) + + def test_hook_filter_and_json_out(self): + rc, rep, out = run(P["compare_bad"], "--hook", "Manifest_Load") + self.assertEqual(list(rep["hooks"]), ["Manifest_Load"]) + self.assertEqual(rep["totals"]["calls"], 4) + self.assertEqual(rep["kind"], "report") + self.assertEqual(rep["format"], 1) + rc, rep, out = run(P["compare_bad"], "--hook", "NoSuchHook") + self.assertEqual(rc, 0) + self.assertEqual(rep["totals"]["calls"], 0) + + def test_first_n(self): + rc, rep, out = run(P["compare_bad"], "--first", "1") + for s in rep["hooks"].values(): + self.assertLessEqual(len(s["diffs"]), 1) + self.assertIn("other divergent call_ids", out) + + def test_multiple_logs(self): + rc, rep, _ = run(P["compare_clean"], P["compare_bad"]) + self.assertEqual(rep["totals"]["calls"], 24) + self.assertEqual(rc, 1) + + +class ReplayTest(unittest.TestCase): + def test_replay_ok(self): + rc, rep, _ = run(P["trace_clean"], "--replay", P["replay_impl_ok"]) + self.assertEqual(rc, 0) + self.assertEqual(rep["kind"], "replay") + self.assertEqual(rep["totals"]["compared"], 12) + + def test_replay_bad(self): + rc, rep, out = run(P["trace_clean"], "--replay", P["replay_impl_bad"]) + self.assertEqual(rc, 1) + rb = FX["replay_bad"] + got = {cid for s in rep["hooks"].values() for cid in s["diverged_call_ids"]} + self.assertEqual(got, {rb["wrong"], rb["missing"]}) + whys = {d["call_id"]: d["diff"][0]["why"] for s in rep["hooks"].values() for d in s["diffs"]} + self.assertEqual(whys[rb["wrong"]], "exact") + self.assertEqual(whys[rb["missing"]], "missing") + self.assertTrue(any(str(rb["extra"]) in w for w in rep["warnings"])) + + def test_replay_side_shapes_and_hook_mismatch(self): + g = write_log("g.jsonl", [{"ts": 0, "hook": "H", "mode": "trace", "call_id": 5, "thread": 0, "args": [], + "ret": mk.i32(1), "side": {"r": {"after": mk.i32(2)}}}]) + p = os.path.join(TMP, "impl_shapes.jsonl") + with open(p, "w") as f: + f.write(json.dumps({"call_id": 5, "hook": "Other", "ret": mk.i32(1), "side": {"r": {"after": mk.i32(2)}}}) + "\n") + rc, rep, _ = run(g, "--replay", p) + self.assertEqual(rc, 0) + self.assertTrue(any("hook 'Other'" in w for w in rep["warnings"])) + with open(p, "w") as f: + f.write(json.dumps({"call_id": 5, "ret": mk.i32(1), "side": {"r": mk.i32(3)}}) + "\n") + f.write("garbage\n") + rc, rep, _ = run(g, "--replay", p) + self.assertEqual(rc, 2) # invalid impl line wins over the divergence + self.assertEqual(rep["totals"]["diverged"], 1) + rc, rep, _ = run(g, "--replay", p, "--skip-invalid") + self.assertEqual(rc, 1) + + def test_replay_tolerance(self): + g = write_log("gt.jsonl", [{"ts": 0, "hook": "H", "mode": "trace", "call_id": 1, "thread": 0, "args": [], + "ret": mk.f32(2.0), "side": {}}]) + p = os.path.join(TMP, "impl_tol.jsonl") + with open(p, "w") as f: + f.write(json.dumps({"call_id": 1, "ret": mk.f32(2.0 + 1e-6)}) + "\n") + self.assertEqual(run(g, "--replay", p)[0], 1) + self.assertEqual(run(g, "--replay", p, "--tolerance", "H=1e-5")[0], 0) + + +class OracleBridgeTest(unittest.TestCase): + @classmethod + def setUpClass(cls): + cls.root = os.path.join(TMP, "gob") + files = { + "Data/globals.txt": 'FOO 1\nBAR 1.0\nNAME "Résumé"\nCOL "1 2 3"\nNEG -0.0\nBIG 7e+8\nFLAG TRUE // c\n', + "Weapons/w.weapon": 'weapon {\n name "Laser" // x\n dam .5\n dam 2\n mount { size small }\n "bare"\n}\n', + "Weapons/_weapons.txt": "1 w.weapon\n// DELETED - 2\n3 Other.WEAPON\n", + "Weapons/_turrets.txt": 'a 1 "two words" 3.25\nb 2 x 4\n', + "Data/x.csv": '# a,b\n1,"x,y"\n\n2,z\n', + "Effects/e.effect": 'TXT\nNAME "New Emitter"\nMODIFIER\nBEGIN\n TYPE 1\n K 0.1\nEND\nMODIFIER\nBEGIN\n TYPE 2\nEND\n', + "Weapons/broken.weapon": 'weapon { name "x" "unterminated\n', + "Weapons/_shipsections.txt": "1 a.shipsection\n1 b.shipsection\n", + "shaders/x.fx": "float4 main() {}\n", + } + for rel, txt in files.items(): + p = os.path.join(cls.root, rel) + os.makedirs(os.path.dirname(p), exist_ok=True) + with open(p, "wb") as f: + f.write(txt.encode("cp1252")) + + def test_canonical_floats_and_strings(self): + self.assertEqual(op.canon_float(1.0), "1.0") + self.assertEqual(op.canon_float(-0.0), "-0.0") + self.assertEqual(op.canon_float(0.1), "0.100000001") # f32-rounded, %.9g + self.assertEqual(op.canon_float(7e8), "700000000.0") + self.assertEqual(op.canon_float(1e20), "1.00000002e+20") + self.assertEqual(op.canon_float(float("nan")), '"nan"') + self.assertEqual(op.canon_str("Résumé"), '"R\\u00e9sum\\u00e9"') + self.assertEqual(op.canon_str("€"), '"\\u0080"') # cp1252 byte, not U+20AC + self.assertEqual(op.canon_str('a"b\\c\n\x7f'), '"a\\"b\\\\c\\n\\u007f"') + self.assertEqual(op.dumps(op.canonical({"b": [1, 2.5, (3, "x")], "a": {"z": True, "y": None}})), + '{"a":{"y":null,"z":true},"b":[1,2.5,[3,"x"]]}') + # no-float objects equal json.dumps canonical settings + obj = {"k": ["a", {"n": 1, "m": "é"}]} + self.assertEqual(op.dumps(op.canonical(obj)), json.dumps(obj, sort_keys=True, ensure_ascii=True, separators=(",", ":"))) + + def test_per_kind_output(self): + r = self.root + c, notes, err = op.oracle_file(os.path.join(r, "Data/globals.txt"), "kv") + self.assertIsNone(err) + self.assertEqual(op.dumps(c), '{"BAR":1.0,"BIG":700000000.0,"COL":"1 2 3","FLAG":true,"FOO":1,"NAME":"R\\u00e9sum\\u00e9","NEG":-0.0}') + c, notes, err = op.oracle_file(os.path.join(r, "Weapons/w.weapon"), "brace:weapon") + self.assertEqual(op.dumps(c), '{"weapon":{"_items":"bare","dam":[0.5,2],"mount":{"size":"small"},"name":"Laser"}}') + c, notes, err = op.oracle_file(os.path.join(r, "Weapons/_weapons.txt"), "manifest") + self.assertEqual(op.dumps(c), '{"deleted":[2],"entries":[[1,"w.weapon"],[3,"Other.WEAPON"]]}') + c, notes, err = op.oracle_file(os.path.join(r, "Weapons/_turrets.txt"), "rows") + self.assertEqual(op.dumps(c), '[["a",1,"two words",3.25],["b",2,"x",4]]') + c, notes, err = op.oracle_file(os.path.join(r, "Data/x.csv"), "csv") + self.assertEqual(op.dumps(c), '[["1","x,y"],["2","z"]]') + c, notes, err = op.oracle_file(os.path.join(r, "Effects/e.effect"), "effect") + self.assertEqual(op.dumps(c), '[["NAME","New Emitter"],["MODIFIER",[["TYPE",1],["K",0.100000001]]],["MODIFIER",[["TYPE",2]]]]') + c, notes, err = op.oracle_file(os.path.join(r, "Weapons/broken.weapon"), "brace:weapon") + self.assertEqual(c, {"_error": True}) + self.assertIn("MarsSyntaxError", err) + c, notes, err = op.oracle_file(os.path.join(r, "Weapons/_shipsections.txt"), "manifest") + self.assertIn("duplicate id", err) + + def test_cli_and_replay_round_trip(self): + r = self.root + golden = os.path.join(TMP, "oracle", "golden.jsonl") + outdir = os.path.join(TMP, "oracle", "canon") + err = io.StringIO() + with redirect_stderr(err), redirect_stdout(io.StringIO()): + rc = op.main(["--root", r, "--all", "--jsonl", golden, "--out-dir", outdir]) + self.assertEqual(rc, 1) # two files fail to parse + self.assertIn("8 file(s), 2 parse error(s)", err.getvalue()) + self.assertFalse(os.path.exists(os.path.join(outdir, "shaders/x.fx.json"))) # skipped kind + with open(os.path.join(outdir, "Data/globals.txt.json")) as f: + self.assertTrue(f.read().endswith("}\n")) + # golden validates and self-replays clean + rc, rep, _ = run(golden) + self.assertEqual(rc, 0) + self.assertEqual(rep["hooks"]["parse:brace"]["errors"], 1) + self.assertEqual(rep["hooks"]["parse:manifest"]["errors"], 1) + rc, rep, _ = run(golden, "--replay", golden) + self.assertEqual(rc, 0) + self.assertEqual(rep["totals"]["compared"], 8) + # an "implementation" that gets one float wrong and one file's shape wrong + impl = os.path.join(TMP, "oracle", "impl.jsonl") + with open(golden) as f, open(impl, "w") as g: + for line in f: + d = json.loads(line) + if "meta" in d: + continue + if d["args"][0]["v"] == "Data/globals.txt": + d["ret"]["v"]["BAR"] = 1.0000001 + if d["args"][0]["v"] == "Weapons/w.weapon": + d["ret"]["v"]["weapon"]["dam"] = 2 + g.write(json.dumps({"call_id": d["call_id"], "ret": d["ret"], "side": {}}) + "\n") + rc, rep, _ = run(golden, "--replay", impl) + self.assertEqual(rc, 1) + self.assertEqual(rep["totals"]["diverged"], 2) + rc, rep, _ = run(golden, "--replay", impl, "--tolerance", "1e-6") + self.assertEqual(rep["totals"]["diverged"], 1) + paths = [d["diff"][0]["path"] for s in rep["hooks"].values() for d in s["diffs"]] + self.assertEqual(paths, ["ret.weapon.dam"]) + # stdout mode prints one canonical line per file + buf = io.StringIO() + with redirect_stdout(buf), redirect_stderr(io.StringIO()): + op.main(["--root", r, os.path.join(r, "Data/globals.txt"), "--kind", "kv"]) + self.assertEqual(buf.getvalue().count("\n"), 1) + + def test_kind_dispatch_matches_verify(self): + self.assertEqual(op.kind_of("Weapons/_weapons.txt"), "manifest") + self.assertEqual(op.kind_of("Data/globals.txt"), "kv") + self.assertEqual(op.kind_of("Weapons/x.weapon"), "brace:weapon") + self.assertEqual(op.kind_of("Locale/EN/Desc_x.txt"), "prose") + + +def tearDownModule(): + shutil.rmtree(TMP, ignore_errors=True) + + +if __name__ == "__main__": + unittest.main(verbosity=1) diff --git a/verify/harness/compare/tracecmp.py b/verify/harness/compare/tracecmp.py new file mode 100644 index 0000000..0680051 --- /dev/null +++ b/verify/harness/compare/tracecmp.py @@ -0,0 +1,792 @@ +#!/usr/bin/env python3 +"""tracecmp.py -- validate shim trace/compare logs (TRACE_FORMAT.md) and report +divergences; or replay a golden trace against an implementation's output. + +usage: + tracecmp.py LOG.jsonl [LOG2.jsonl ...] report per-hook divergences + tracecmp.py --replay IMPL.jsonl GOLDEN.jsonl [...] diff GOLDEN (ret/side) vs IMPL + records matched by call_id + +flags: + --hook NAME only this hook (repeatable) + --tolerance SPEC float policy; SPEC = [HOOK=][abs:|rel:|ulp:]NUMBER + e.g. 1e-6 | rel:1e-5 | Mars::ParseBlock=ulp:2 (repeatable; + a bare NUMBER without HOOK sets the default for every hook) + --unordered SPEC treat a list path as a set; SPEC = [HOOK=]PATH (repeatable) + --ptr exact|ignore pointer policy for every hook (default: ignore) + --numeric inside `json` values, compare int vs float as numbers + --first N diffs shown per hook (default 5) + --json-out PATH write the full report as JSON (convention: + verify/results/compare/.json) + --skip-invalid drop unparsable/invalid records with a warning instead of + failing the run + +exit: 0 = no divergences; 1 = divergences; 2 = invalid input (or usage). + +Stdlib only. +""" +from __future__ import annotations + +import argparse +import json +import math +import os +import re +import struct +import sys +from dataclasses import dataclass, field, replace as dc_replace +from typing import Any + +FORMAT_VERSION = 1 +MODES = ("trace", "compare", "replace") +INT_TYPES = {"i8", "i16", "i32", "i64", "u8", "u16", "u32", "u64"} +FLOAT_TYPES = {"f32", "f64"} +TV_TYPES = INT_TYPES | FLOAT_TYPES | {"bool", "str", "wstr", "ptr", "enum", "null", + "bytes", "list", "set", "struct", "json"} +RECORD_KEYS = {"ts", "hook", "mode", "call_id", "thread", "depth", "args", "ret", "side", + "ours", "diverged", "diff", "err", "note"} +INT_RANGE = {"i8": (-2**7, 2**7 - 1), "i16": (-2**15, 2**15 - 1), "i32": (-2**31, 2**31 - 1), + "i64": (-2**63, 2**63 - 1), "u8": (0, 2**8 - 1), "u16": (0, 2**16 - 1), + "u32": (0, 2**32 - 1), "u64": (0, 2**64 - 1)} +_HEX_RE = re.compile(r"^[0-9a-f]*$") +_SHA_RE = re.compile(r"^[0-9a-f]{64}$") +_PTR_RE = re.compile(r"^0x[0-9a-fA-F]{1,16}$") +NONFINITE = {"nan": math.nan, "inf": math.inf, "-inf": -math.inf} + + +# --- policy ------------------------------------------------------------------ + +@dataclass +class Policy: + ftol: float = 0.0 + ftol_kind: str = "abs" # abs | rel | ulp + ptr: str = "ignore" # ignore | exact + numeric: bool = False # json: int vs float compare as numbers + unordered: set = field(default_factory=set) + + def merged(self, d: dict) -> "Policy": + p = dc_replace(self, unordered=set(self.unordered)) + if "ftol" in d: + p.ftol = float(d["ftol"]) + if "ftol_kind" in d: + p.ftol_kind = d["ftol_kind"] + if "ptr" in d: + p.ptr = d["ptr"] + if "numeric" in d: + p.numeric = bool(d["numeric"]) + if "unordered" in d: + p.unordered |= set(d["unordered"]) + return p + + +class PolicyTable: + """defaults < log meta per-hook < CLI default < CLI per-hook.""" + + def __init__(self): + self.default: dict = {} + self.meta: dict[str, dict] = {} + self.cli: dict[str, dict] = {} + + def add_meta(self, hooks: dict) -> None: + for h, d in (hooks or {}).items(): + self.meta.setdefault(h, {}).update(d) + + def add_cli(self, hook: str | None, d: dict) -> None: + if hook is None: + self.default.update(d) + else: + self.cli.setdefault(hook, {}).update(d) + + def for_hook(self, hook: str) -> Policy: + p = Policy().merged(self.meta.get(hook, {})) + p = p.merged(self.default) + return p.merged(self.cli.get(hook, {})) + + +def parse_tolerance(spec: str) -> tuple[str | None, dict]: + hook = None + if "=" in spec: + hook, spec = spec.split("=", 1) + kind = "abs" + if ":" in spec: + kind, spec = spec.split(":", 1) + if kind not in ("abs", "rel", "ulp"): + raise ValueError(f"bad tolerance kind {kind!r}") + return hook, {"ftol": float(spec), "ftol_kind": kind} + + +def parse_unordered(spec: str) -> tuple[str | None, dict]: + hook = None + if "=" in spec: + hook, spec = spec.split("=", 1) + return hook, {"unordered": [spec]} + + +# --- validation ---------------------------------------------------------------- + +class Invalid(Exception): + pass + + +def _is_int(x) -> bool: + return isinstance(x, int) and not isinstance(x, bool) + + +def validate_tv(tv: Any, path: str, errs: list[str]) -> None: + if not isinstance(tv, dict): + errs.append(f"{path}: typed value must be an object, got {type(tv).__name__}") + return + t = tv.get("t") + if t not in TV_TYPES: + errs.append(f"{path}: unknown type {t!r}") + return + if "n" in tv and not isinstance(tv["n"], (str, int)): + errs.append(f"{path}: 'n' must be a string (name) or int (bytes length)") + if t == "bytes": + n, sha = tv.get("n"), tv.get("sha256") + if not _is_int(n) or n < 0: + errs.append(f"{path}: bytes.n must be int >= 0") + if not isinstance(sha, str) or not _SHA_RE.match(sha): + errs.append(f"{path}: bytes.sha256 must be 64 lowercase hex chars") + if "hex" in tv: + h = tv["hex"] + if not isinstance(h, str) or not _HEX_RE.match(h) or (_is_int(n) and len(h) != 2 * n): + errs.append(f"{path}: bytes.hex must be 2*n lowercase hex chars") + if "head" in tv and (not isinstance(tv["head"], str) or not _HEX_RE.match(tv["head"])): + errs.append(f"{path}: bytes.head must be lowercase hex") + return + if "v" not in tv: + errs.append(f"{path}: missing 'v'") + return + v = tv["v"] + if t == "bool": + if not isinstance(v, bool): + errs.append(f"{path}: bool.v must be true/false") + elif t in INT_TYPES: + iv = None + if _is_int(v): + iv = v + elif isinstance(v, str) and re.match(r"^-?\d+$", v): + iv = int(v) + else: + errs.append(f"{path}: {t}.v must be an integer (or decimal string)") + if iv is not None: + lo, hi = INT_RANGE[t] + if not lo <= iv <= hi: + errs.append(f"{path}: {t}.v={iv} out of range") + elif t in FLOAT_TYPES: + if isinstance(v, bool) or not (isinstance(v, (int, float)) or v in NONFINITE): + errs.append(f"{path}: {t}.v must be a number or 'nan'/'inf'/'-inf'") + elif t in ("str", "wstr"): + if not isinstance(v, str): + errs.append(f"{path}: {t}.v must be a string") + elif t == "str" and any(ord(c) > 0xFF for c in v): + errs.append(f"{path}: str.v carries a code point > U+00FF (bytes must be \\u00XX)") + elif t == "ptr": + if not isinstance(v, str) or not _PTR_RE.match(v): + errs.append(f"{path}: ptr.v must be a hex string like 0x00a36fd0") + elif t == "enum": + if not _is_int(v): + errs.append(f"{path}: enum.v must be an integer") + elif t == "null": + if v is not None: + errs.append(f"{path}: null.v must be null") + elif t in ("list", "set"): + if not isinstance(v, list): + errs.append(f"{path}: {t}.v must be an array") + else: + for i, e in enumerate(v): + validate_tv(e, f"{path}.v[{i}]", errs) + elif t == "struct": + if not isinstance(v, dict): + errs.append(f"{path}: struct.v must be an object") + else: + for k, e in v.items(): + validate_tv(e, f"{path}.v.{k}", errs) + elif t == "json": + pass # any JSON + + +def validate_side(side: Any, path: str, errs: list[str], need_before: bool = False) -> None: + if not isinstance(side, dict): + errs.append(f"{path}: must be an object") + return + for name, ent in side.items(): + if not isinstance(ent, dict) or "after" not in ent: + errs.append(f"{path}.{name}: must be {{\"before\"?, \"after\"}}") + continue + if ent.get("before") is not None: + validate_tv(ent["before"], f"{path}.{name}.before", errs) + validate_tv(ent["after"], f"{path}.{name}.after", errs) + + +def validate_record(rec: Any) -> tuple[list[str], list[str]]: + """-> (errors, warnings). Errors make the record invalid.""" + errs: list[str] = [] + warns: list[str] = [] + if not isinstance(rec, dict): + return ["record is not an object"], warns + for k in ("ts", "hook", "mode", "call_id", "thread", "args", "ret", "side"): + if k not in rec: + errs.append(f"missing required field {k!r}") + if errs: + return errs, warns + if not _is_int(rec["ts"]) and not isinstance(rec["ts"], float): + errs.append("ts must be a number") + if not isinstance(rec["hook"], str) or not rec["hook"]: + errs.append("hook must be a non-empty string") + if rec["mode"] not in MODES: + errs.append(f"mode must be one of {MODES}") + if not _is_int(rec["call_id"]) or rec["call_id"] < 0: + errs.append("call_id must be int >= 0") + if not _is_int(rec["thread"]): + errs.append("thread must be int") + if "depth" in rec and (not _is_int(rec["depth"]) or rec["depth"] < 0): + errs.append("depth must be int >= 0") + if not isinstance(rec["args"], list): + errs.append("args must be an array") + else: + for i, a in enumerate(rec["args"]): + validate_tv(a, f"args[{i}]", errs) + if rec["ret"] is not None: + validate_tv(rec["ret"], "ret", errs) + validate_side(rec["side"], "side", errs) + if "err" in rec and not isinstance(rec["err"], str): + errs.append("err must be a string") + if "note" in rec and not isinstance(rec["note"], str): + errs.append("note must be a string") + mode = rec["mode"] + if mode == "compare": + ours = rec.get("ours") + if not isinstance(ours, dict) or "ret" not in ours or "side" not in ours: + if "err" not in rec: + errs.append("compare record needs ours={ret, side} (or err)") + else: + if ours["ret"] is not None: + validate_tv(ours["ret"], "ours.ret", errs) + validate_side(ours["side"], "ours.side", errs) + if "diverged" not in rec: + warns.append("compare record without 'diverged'") + elif not isinstance(rec["diverged"], bool): + errs.append("diverged must be a bool") + if "diff" in rec and not isinstance(rec["diff"], list): + errs.append("diff must be an array") + else: + if "ours" in rec: + warns.append(f"{mode} record carries 'ours' (ignored)") + for k in rec: + if k not in RECORD_KEYS: + warns.append(f"unknown field {k!r}") + return errs, warns + + +# --- comparison ------------------------------------------------------------------ + +def _diff(path: str, why: str, a: Any, b: Any, **extra) -> dict: + d = {"path": path, "why": why, "orig": a, "ours": b} + d.update(extra) + return d + + +def _num(v, width: str = "f64") -> float: + """tv float payload -> Python float. f32 values are rounded to float32: + the emitter's %.9g round-trips the float32, not the exact double.""" + if isinstance(v, str): + return NONFINITE[v] + x = float(v) + if width == "f32" and math.isfinite(x): + x = struct.unpack(" int: + fmt, ifmt = ("f", "i") if width == "f32" else ("d", "q") + def to_int(x): + i = struct.unpack("<" + ifmt, struct.pack("<" + fmt, x))[0] + return i if i >= 0 else -(i & (2**(31 if fmt == "f" else 63) - 1)) + return abs(to_int(a) - to_int(b)) + + +def floats_equal(a: float, b: float, pol: Policy, width: str = "f64") -> bool: + if math.isnan(a) or math.isnan(b): + return math.isnan(a) and math.isnan(b) + if math.isinf(a) or math.isinf(b): + return a == b + if a == b: + return True + if pol.ftol <= 0: + return False + if pol.ftol_kind == "abs": + return abs(a - b) <= pol.ftol + if pol.ftol_kind == "rel": + return abs(a - b) <= pol.ftol * max(abs(a), abs(b)) + if pol.ftol_kind == "ulp": + return _ulp_diff(a, b, width) <= pol.ftol + raise ValueError(pol.ftol_kind) + + +def _canon(x) -> str: + return json.dumps(x, sort_keys=True, separators=(",", ":")) + + +def compare_json(a: Any, b: Any, pol: Policy, path: str, out: list) -> None: + if isinstance(a, bool) or isinstance(b, bool) or a is None or b is None or \ + isinstance(a, str) or isinstance(b, str): + if type(a) is not type(b) or a != b: + out.append(_diff(path, "exact", a, b)) + return + if isinstance(a, (int, float)) and isinstance(b, (int, float)): + both_float = isinstance(a, float) and isinstance(b, float) + if type(a) is not type(b) and not pol.numeric: + out.append(_diff(path, "type", a, b)) + return + if both_float or pol.numeric: + if not floats_equal(float(a), float(b), pol): + out.append(_diff(path, "ftol" if pol.ftol > 0 else "exact", a, b)) + elif a != b: + out.append(_diff(path, "exact", a, b)) + return + if isinstance(a, list) and isinstance(b, list): + if path in pol.unordered: + a = sorted(a, key=_canon) + b = sorted(b, key=_canon) + if len(a) != len(b): + out.append(_diff(path, "len", len(a), len(b))) + return + for i, (x, y) in enumerate(zip(a, b)): + compare_json(x, y, pol, f"{path}[{i}]", out) + return + if isinstance(a, dict) and isinstance(b, dict): + ka, kb = set(a), set(b) + for k in sorted(ka - kb): + out.append(_diff(f"{path}.{k}", "missing", a[k], None)) + for k in sorted(kb - ka): + out.append(_diff(f"{path}.{k}", "extra", None, b[k])) + for k in sorted(ka & kb): + compare_json(a[k], b[k], pol, f"{path}.{k}", out) + return + out.append(_diff(path, "type", a, b)) + + +def compare_tv(a: Any, b: Any, pol: Policy, path: str, out: list) -> None: + """Append diff entries for typed values a (orig) vs b (ours).""" + if a is None or b is None: + if a is not b: + out.append(_diff(path, "missing" if b is None else "extra", a, b)) + return + ta, tb = a.get("t"), b.get("t") + if ta != tb: + out.append(_diff(path, "type", a, b)) + return + t = ta + if t == "ptr": + va, vb = a["v"], b["v"] + za, zb = int(va, 16) == 0, int(vb, 16) == 0 + if za != zb: + out.append(_diff(path, "exact", a, b)) + elif pol.ptr == "exact" and int(va, 16) != int(vb, 16): + out.append(_diff(path, "exact", a, b)) + return + if t == "null": + return + if t == "bool" or t == "enum" or t in ("str", "wstr"): + if a["v"] != b["v"]: + out.append(_diff(path, "exact", a, b)) + return + if t in INT_TYPES: + if int(a["v"]) != int(b["v"]): + out.append(_diff(path, "exact", a, b)) + return + if t in FLOAT_TYPES: + if not floats_equal(_num(a["v"], t), _num(b["v"], t), pol, t): + out.append(_diff(path, "ftol" if pol.ftol > 0 else "exact", a, b)) + return + if t == "bytes": + if a.get("n") != b.get("n"): + out.append(_diff(path, "len", a, b)) + return + if a.get("sha256") != b.get("sha256"): + extra = {} + if "hex" in a and "hex" in b: + ha, hb = a["hex"], b["hex"] + off = next((i for i in range(0, min(len(ha), len(hb)), 2) if ha[i:i+2] != hb[i:i+2]), None) + if off is not None: + extra["first_diff_offset"] = off // 2 + out.append(_diff(path, "hash", a, b, **extra)) + elif "hex" in a and "hex" in b and a["hex"] != b["hex"]: + out.append(_diff(path, "hash", a, b, note="same sha256, different hex: corrupt log")) + return + if t in ("list", "set"): + va, vb = a["v"], b["v"] + if t == "set" or path in pol.unordered: + va = sorted(va, key=_canon) + vb = sorted(vb, key=_canon) + if len(va) != len(vb): + out.append(_diff(path, "len", len(va), len(vb))) + return + for i, (x, y) in enumerate(zip(va, vb)): + compare_tv(x, y, pol, f"{path}.v[{i}]", out) + return + if t == "struct": + va, vb = a["v"], b["v"] + for k in sorted(set(va) - set(vb)): + out.append(_diff(f"{path}.v.{k}", "missing", va[k], None)) + for k in sorted(set(vb) - set(va)): + out.append(_diff(f"{path}.v.{k}", "extra", None, vb[k])) + for k in sorted(set(va) & set(vb)): + compare_tv(va[k], vb[k], pol, f"{path}.v.{k}", out) + return + if t == "json": + compare_json(a["v"], b["v"], pol, path, out) + return + raise AssertionError(t) + + +def _side_after(side: dict, name: str) -> Any: + ent = side[name] + if isinstance(ent, dict) and ent.get("t") is None and "after" in ent: + return ent["after"] + return ent # replay input may give the tv directly + + +def compare_outputs(orig_ret, orig_side: dict, ours_ret, ours_side: dict, pol: Policy) -> list[dict]: + diffs: list[dict] = [] + compare_tv(orig_ret, ours_ret, pol, "ret", diffs) + names_a, names_b = set(orig_side), set(ours_side) + for n in sorted(names_a - names_b): + diffs.append(_diff(f"side.{n}.after", "missing", _side_after(orig_side, n), None)) + for n in sorted(names_b - names_a): + diffs.append(_diff(f"side.{n}.after", "extra", None, _side_after(ours_side, n))) + for n in sorted(names_a & names_b): + compare_tv(_side_after(orig_side, n), _side_after(ours_side, n), pol, f"side.{n}.after", diffs) + return diffs + + +def compare_record(rec: dict, pol: Policy) -> tuple[list[dict], list[str]]: + """Recompute a compare record's diff. -> (diffs, warnings)""" + warns: list[str] = [] + if "err" in rec: + return [_diff("call", "err", None, rec["err"])], warns + ours = rec["ours"] + diffs = compare_outputs(rec["ret"], rec["side"], ours["ret"], ours["side"], pol) + # snapshot sanity: 'before' on both sides should be identical + for n in set(rec["side"]) & set(ours["side"]): + ba, bb = rec["side"][n].get("before"), ours["side"][n].get("before") + if ba is not None and bb is not None: + tmp: list = [] + compare_tv(ba, bb, pol, f"side.{n}.before", tmp) + if tmp: + warns.append(f"call {rec['call_id']}: side.{n}.before differs between sides (snapshot bug?)") + if "diverged" in rec and rec["diverged"] != bool(diffs): + warns.append(f"call {rec['call_id']}: shim said diverged={rec['diverged']}, harness found {len(diffs)} diff(s)") + return diffs, warns + + +# --- loading ----------------------------------------------------------------------- + +@dataclass +class Log: + path: str + meta: dict = field(default_factory=dict) + records: list = field(default_factory=list) + invalid: list = field(default_factory=list) # (lineno, message) + warnings: list = field(default_factory=list) # (lineno, message) + + +def load_log(path: str, policies: PolicyTable | None = None) -> Log: + log = Log(path) + seen: dict[int, int] = {} + with open(path, "r", encoding="utf-8") as f: + for lineno, line in enumerate(f, 1): + line = line.rstrip("\n") + if line.endswith("\r"): + log.warnings.append((lineno, "CRLF line ending")) + line = line[:-1] + if not line.strip(): + log.invalid.append((lineno, "blank line")) + continue + try: + rec = json.loads(line) + except ValueError as e: + log.invalid.append((lineno, f"not JSON: {e}")) + continue + if isinstance(rec, dict) and "meta" in rec and len(rec) == 1: + m = rec["meta"] + if not isinstance(m, dict): + log.invalid.append((lineno, "meta must be an object")) + continue + if m.get("format") != FORMAT_VERSION: + log.invalid.append((lineno, f"unsupported format {m.get('format')!r} (want {FORMAT_VERSION})")) + continue + if lineno != 1: + log.warnings.append((lineno, "meta record not on line 1")) + log.meta = m + if policies is not None: + policies.add_meta(m.get("hooks", {})) + continue + errs, warns = validate_record(rec) + for w in warns: + log.warnings.append((lineno, w)) + if errs: + log.invalid.append((lineno, "; ".join(errs))) + continue + cid = rec["call_id"] + if cid in seen: + log.invalid.append((lineno, f"duplicate call_id {cid} (first at line {seen[cid]})")) + continue + seen[cid] = lineno + rec["_line"] = lineno + log.records.append(rec) + return log + + +def load_impl(path: str) -> tuple[dict[int, dict], list[tuple[int, str]]]: + """Implementation output for --replay: records keyed by call_id.""" + out: dict[int, dict] = {} + invalid: list[tuple[int, str]] = [] + with open(path, "r", encoding="utf-8") as f: + for lineno, line in enumerate(f, 1): + line = line.strip() + if not line: + continue + try: + rec = json.loads(line) + except ValueError as e: + invalid.append((lineno, f"not JSON: {e}")) + continue + if isinstance(rec, dict) and "meta" in rec and len(rec) == 1: + continue + if not isinstance(rec, dict) or not _is_int(rec.get("call_id")): + invalid.append((lineno, "impl record needs an integer call_id")) + continue + errs: list[str] = [] + if rec.get("ret") is not None: + validate_tv(rec["ret"], "ret", errs) + side = rec.get("side", {}) + if not isinstance(side, dict): + errs.append("side must be an object") + else: + for n in side: + validate_tv(_side_after(side, n), f"side.{n}", errs) + if errs: + invalid.append((lineno, "; ".join(errs))) + continue + if rec["call_id"] in out: + invalid.append((lineno, f"duplicate call_id {rec['call_id']}")) + continue + rec.setdefault("side", {}) + rec.setdefault("ret", None) + rec["_line"] = lineno + out[rec["call_id"]] = rec + return out, invalid + + +# --- report ------------------------------------------------------------------------ + +def new_hook_stats() -> dict: + return {"calls": 0, "modes": {}, "compared": 0, "diverged": 0, "errors": 0, + "diverged_call_ids": [], "diffs": []} + + +def run_report(logs: list[Log], policies: PolicyTable, hooks: set[str] | None, first: int) -> dict: + per_hook: dict[str, dict] = {} + warnings: list[str] = [] + for log in logs: + for ln, w in log.warnings: + warnings.append(f"{os.path.basename(log.path)}:{ln}: {w}") + for rec in log.records: + h = rec["hook"] + if hooks and h not in hooks: + continue + st = per_hook.setdefault(h, new_hook_stats()) + st["calls"] += 1 + st["modes"][rec["mode"]] = st["modes"].get(rec["mode"], 0) + 1 + if rec["mode"] != "compare": + if "err" in rec: + st["errors"] += 1 + continue + st["compared"] += 1 + diffs, w = compare_record(rec, policies.for_hook(h)) + warnings.extend(f"{os.path.basename(log.path)}:{rec['_line']}: {x}" for x in w) + if diffs: + st["diverged"] += 1 + if "err" in rec: + st["errors"] += 1 + st["diverged_call_ids"].append(rec["call_id"]) + if len(st["diffs"]) < first: + st["diffs"].append({"call_id": rec["call_id"], "line": rec["_line"], + "file": os.path.basename(log.path), "diff": diffs}) + return _finish(per_hook, logs, warnings, "report") + + +def run_replay(golden: list[Log], impl: dict[int, dict], policies: PolicyTable, + hooks: set[str] | None, first: int) -> dict: + per_hook: dict[str, dict] = {} + warnings: list[str] = [] + seen: set[int] = set() + for log in golden: + for ln, w in log.warnings: + warnings.append(f"{os.path.basename(log.path)}:{ln}: {w}") + for rec in log.records: + h = rec["hook"] + if hooks and h not in hooks: + continue + st = per_hook.setdefault(h, new_hook_stats()) + st["calls"] += 1 + st["modes"][rec["mode"]] = st["modes"].get(rec["mode"], 0) + 1 + st["compared"] += 1 + cid = rec["call_id"] + seen.add(cid) + ours = impl.get(cid) + if ours is None: + diffs = [_diff("call", "missing", None, None, note=f"call_id {cid} absent from impl")] + else: + if "hook" in ours and ours["hook"] != h: + warnings.append(f"impl call_id {cid}: hook {ours['hook']!r} != golden {h!r}") + diffs = compare_outputs(rec["ret"], rec["side"], ours["ret"], ours["side"], policies.for_hook(h)) + if diffs: + st["diverged"] += 1 + st["diverged_call_ids"].append(cid) + if len(st["diffs"]) < first: + st["diffs"].append({"call_id": cid, "line": rec["_line"], + "file": os.path.basename(log.path), "diff": diffs}) + extra = sorted(set(impl) - seen) + if extra: + warnings.append(f"impl has {len(extra)} call_id(s) not in golden: {extra[:10]}{'…' if len(extra) > 10 else ''}") + return _finish(per_hook, golden, warnings, "replay") + + +def _finish(per_hook: dict, logs: list[Log], warnings: list[str], kind: str) -> dict: + total_div = sum(s["diverged"] for s in per_hook.values()) + invalid = [(os.path.basename(l.path), ln, m) for l in logs for ln, m in l.invalid] + return { + "kind": kind, + "format": FORMAT_VERSION, + "inputs": [l.path for l in logs], + "meta": [l.meta for l in logs if l.meta], + "hooks": dict(sorted(per_hook.items())), + "totals": {"calls": sum(s["calls"] for s in per_hook.values()), + "compared": sum(s["compared"] for s in per_hook.values()), + "diverged": total_div, + "invalid_records": len(invalid)}, + "invalid": [{"file": f, "line": ln, "error": m} for f, ln, m in invalid], + "warnings": warnings, + } + + +def _short(x: Any, n: int = 60) -> str: + s = _canon(x) + return s if len(s) <= n else s[:n - 1] + "…" + + +def print_report(rep: dict, out=None) -> None: + out = out or sys.stdout + P = lambda s="": print(s, file=out) # noqa: E731 + P(f"## tracecmp {rep['kind']}: {', '.join(os.path.basename(p) for p in rep['inputs'])}") + P() + for m in rep["meta"]: + P(f"- build: {m.get('build', '?')} started: {m.get('started', '?')} inline_max: {m.get('inline_max', '?')}") + t = rep["totals"] + P(f"- calls: {t['calls']} compared: {t['compared']} diverged: {t['diverged']} " + f"invalid records: {t['invalid_records']} warnings: {len(rep['warnings'])}") + P() + P("| hook | calls | modes | compared | diverged | errors |") + P("|---|---|---|---|---|---|") + for h, s in rep["hooks"].items(): + modes = " ".join(f"{k}:{v}" for k, v in sorted(s["modes"].items())) + P(f"| {h} | {s['calls']} | {modes} | {s['compared']} | {s['diverged']} | {s['errors']} |") + for h, s in rep["hooks"].items(): + if not s["diffs"]: + continue + P() + P(f"### {h}: first {len(s['diffs'])} of {s['diverged']} divergent call(s)") + for d in s["diffs"]: + P(f"- call_id {d['call_id']} ({d['file']}:{d['line']})") + for e in d["diff"][:8]: + extra = "".join(f" {k}={v}" for k, v in e.items() if k not in ("path", "why", "orig", "ours")) + P(f" {e['path']} [{e['why']}] orig={_short(e['orig'])} ours={_short(e['ours'])}{extra}") + if len(d["diff"]) > 8: + P(f" … {len(d['diff']) - 8} more") + more = s["diverged_call_ids"][len(s["diffs"]):] + if more: + P(f" other divergent call_ids: {more[:20]}{' …' if len(more) > 20 else ''}") + if rep["invalid"]: + P() + P(f"### invalid records ({len(rep['invalid'])})") + for e in rep["invalid"][:20]: + P(f"- {e['file']}:{e['line']}: {e['error']}") + if rep["warnings"]: + P() + P(f"### warnings ({len(rep['warnings'])})") + for w in rep["warnings"][:20]: + P(f"- {w}") + + +# --- main ---------------------------------------------------------------------------- + +def build_parser() -> argparse.ArgumentParser: + ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0], formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("logs", nargs="+", metavar="LOG", help="trace/compare JSONL (golden logs in --replay)") + ap.add_argument("--replay", metavar="IMPL", help="implementation output JSONL to diff against LOG") + ap.add_argument("--hook", action="append", default=[], help="only this hook (repeatable)") + ap.add_argument("--tolerance", action="append", default=[], metavar="SPEC", help="[HOOK=][abs:|rel:|ulp:]NUMBER") + ap.add_argument("--unordered", action="append", default=[], metavar="SPEC", help="[HOOK=]PATH") + ap.add_argument("--ptr", choices=("ignore", "exact"), default=None) + ap.add_argument("--numeric", action="store_true") + ap.add_argument("--first", type=int, default=5, metavar="N") + ap.add_argument("--json-out", metavar="PATH") + ap.add_argument("--skip-invalid", action="store_true") + ap.add_argument("--quiet", action="store_true", help="no markdown report on stdout") + return ap + + +def main(argv: list[str] | None = None) -> int: + ap = build_parser() + a = ap.parse_args(argv) + policies = PolicyTable() + try: + for s in a.tolerance: + policies.add_cli(*parse_tolerance(s)) + for s in a.unordered: + policies.add_cli(*parse_unordered(s)) + except ValueError as e: + ap.error(str(e)) + if a.ptr: + policies.add_cli(None, {"ptr": a.ptr}) + if a.numeric: + policies.add_cli(None, {"numeric": True}) + hooks = set(a.hook) or None + + logs = [] + for p in a.logs: + if not os.path.exists(p): + print(f"error: no such file {p}", file=sys.stderr) + return 2 + logs.append(load_log(p, policies)) + impl_invalid: list = [] + if a.replay: + if not os.path.exists(a.replay): + print(f"error: no such file {a.replay}", file=sys.stderr) + return 2 + impl, impl_invalid = load_impl(a.replay) + rep = run_replay(logs, impl, policies, hooks, a.first) + rep["impl"] = a.replay + rep["invalid"].extend({"file": os.path.basename(a.replay), "line": ln, "error": m} for ln, m in impl_invalid) + rep["totals"]["invalid_records"] += len(impl_invalid) + else: + rep = run_report(logs, policies, hooks, a.first) + + if a.json_out: + os.makedirs(os.path.dirname(os.path.abspath(a.json_out)), exist_ok=True) + with open(a.json_out, "w", encoding="utf-8") as f: + json.dump(rep, f, indent=1, sort_keys=True) + f.write("\n") + if not a.quiet: + print_report(rep) + if rep["totals"]["invalid_records"] and not a.skip_invalid: + return 2 + return 1 if rep["totals"]["diverged"] else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/verify/results/compare/.keep.md b/verify/results/compare/.keep.md new file mode 100644 index 0000000..03a4984 --- /dev/null +++ b/verify/results/compare/.keep.md @@ -0,0 +1 @@ +# per-run tracecmp reports (--json-out); see verify/harness/compare/