verify: compare harness (trace format, tracecmp, replay, oracle bridge); board catch-up; sim formula gaps
This commit is contained in:
alex
parent
5614940650
commit
4547f6c8fe
8 changed files with 2246 additions and 2 deletions
|
|
@ -33,9 +33,17 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`).
|
|||
| save_reader.py | verify | verified | high | 100% | 2026-09-07 | --strict exit 0 on all 3 real saves; 0 resyncs / 0 hint-failures; only raw = RNG blob (2503 B). 29 tests. SAVE_FORMAT.md confirmed |
|
||||
| Ghidra type write-back | meta | verified | high | 100% | 2026-09-07 | structs saved in project (ServerSystem 87f, ServerPlayer 110f, StarFleet, StarShip, StrategyServer partial, 22 nested); 52 serializers + primitives + ~60 spine fns renamed; decompile shows field names |
|
||||
| strategic turn internals (economy/research/colony/movement/diplomacy) | subsystem | mapped | high | 90% | 2026-09-07 | findings/subsystems/strategic-turn-internals.md: budget/RP/trade/bankruptcy formulas, research rolls (unlisted race=1.0, Zuul x2), colony growth, movement, RNG=MT19937; ~600 g_KEY labels in Ghidra |
|
||||
| P2-M0 shim bootstrap | phase2 | in-progress | — | 0% | 2026-09-07 | sots-engine: binkw32 proxy + MinHook + trace hook on Application::Initialize; game to menu |
|
||||
| P2-M0 shim bootstrap | phase2 | verified | high | 100% | 2026-09-07 | sots-engine 5f5bc41: proxy binkw32.dll (66 exports identical), MinHook, asm-stub trace hook on Application::Initialize; menu reached; evidence verify/results/shim/. Initialize signature unverified -> asm stubs only |
|
||||
| P2-M1 flat-KV config loader | phase2 | backlog | — | 0% | 2026-09-07 | CfgVar_RegisterKey 0x008b76a0; trace->compare->replace; save byte-identical oracle |
|
||||
| P2-M2 manifest/id registry | phase2 | backlog | — | 0% | 2026-09-07 | _weapons.txt/_shipsections.txt stable ids |
|
||||
| P2-M3 Mars brace-block parser | phase2 | backlog | — | 0% | 2026-09-07 | + .effect TXT reader; compare over full catalog load |
|
||||
| P2-M4 gobio VFS read | phase2 | backlog | — | 0% | 2026-09-07 | ZIP entries + NativeFileSystem override order |
|
||||
| loader prototypes (Ghidra) | meta | in-progress | — | 0% | 2026-09-07 | R6: cfgvar parser, manifest loader, brace-block parser entry+node model, gobio read -> ghidra/addresses.json |
|
||||
| loader prototypes (Ghidra) | meta | in-progress | — | 0% | 2026-09-07 | R6 running: cfgvar parser, manifest loader, brace-block entry+node model, gobio read; + Initialize/Run/OnTick signature checks (M0 crash lesson) |
|
||||
| compare harness | verify | mapped | high | 100% | 2026-09-07 | verify/harness/compare/: TRACE_FORMAT.md, tracecmp.py, replay, oracle bridge |
|
||||
| engine: mars/parse | engine | verified | high | 100% | 2026-09-07 | sots-engine: brace-block + .effect readers; oracle 1531/1531; cross-compiles i686; wired into host+shim builds |
|
||||
| engine: mars/text | engine | verified | high | 100% | 2026-09-07 | flat-kv, id-manifest, csv; oracle 64/64 (Strings.csv 5722 rows); ctest green |
|
||||
| engine: game/sim formulas | engine | mapped | med | 80% | 2026-09-07 | economy/research/colony/movement pure fns, 356 checks; 5 low-confidence formulas fed back to open-questions |
|
||||
| engine: mars/stream + rng | engine | in-progress | — | 0% | 2026-09-07 | Streamable reader/writer + MT19937; oracle = 3 real saves + reference vectors |
|
||||
| engine: game/data catalogs | engine | in-progress | — | 0% | 2026-09-07 | typed WeaponDef/ShipSectionDef/TechTree/IdRegistry on mars/parse+text; oracle = data-catalogs JSON |
|
||||
| engine: mars/vfs (gob) | engine | in-progress | — | 0% | 2026-09-07 | ZIP reader + native override; oracle = unzip listings |
|
||||
| determinism oracle | verify | in-progress | — | 0% | 2026-09-07 | load same save -> End Turn -> byte-compare autosaves across runs (VM) |
|
||||
|
|
|
|||
|
|
@ -30,3 +30,4 @@ Each links to the finding that raised it. Promoted to backlog or closed by **re-
|
|||
- **Ship-design code rules (Ghidra, from SHIP_DESIGN_RULES.md §8)** — hidden default rider designs are built WITHOUT tech gating (where?); is section class-equality enforced or merely unoffered; confirm the turret class-accept table (`standard ← {standard,missile,grapple}`, `strafe ← {standard}`; grapple weakest); hull-class tech injection (`IND_CruisCon`/`IND_DreadCon`) + station exemption; option defaults/auto-upgrade and what `option_cost` multiplies; weapon cost per bank vs per mount; mass/speed formulas; `faiDes`/`dWep` meaning. (from [[SHIP_DESIGN_RULES]])
|
||||
- **Corrections from turn internals** — `ServerPlayer+0xf9` is the AI flag (not bTurnDone); `FUN_007d7f70` is encounter detection; ProcessTurn phase 2 is trade, not diplomacy; the spine's `FUN_00840fe0` 'income' is special projects — income is `FUN_00863030` ComputeBudget. `TECHBEN_*` data blocks are inert (effects hard-coded via the 116-entry tech-name table @0x00a19718). Open: TRA/TRP producers, `FPsp2` derivation, `PERGATETRAFFIC_*` readers, exact BnkEl/BnkPr expression. (from [[strategic-turn-internals]])
|
||||
- **`Mars::Application::Initialize` signature (P2-M0 finding)** — a C++ `__thiscall` detour that called the original crashed inside Initialize; the trace hook now uses a convention-agnostic asm stub. Verify the real signature (stack args? EDX use? return value?) before any `compare`/`replace` hook on it. Same caution for every `[unverified]` entry in `ghidra/addresses.json`. (from [[m0]])
|
||||
- **Formula gaps surfaced by the game/sim port (feed back to Ghidra)** — which bankruptcy limit carries the 3.3 factor and when the bankruptcy start turn is stamped; the suitability→carrying-capacity hazard curve shape; the trade-points→money system-income tail; the `POPBONUS_INC` population increment; the expense-slider request term; which running total the tech-income bonus / savings aid read; node-line speed clamp at the influence radius; whether `DecayAllResearch` also hits the current target. (from [[game-sim]])
|
||||
|
|
|
|||
198
verify/harness/compare/TRACE_FORMAT.md
Normal file
198
verify/harness/compare/TRACE_FORMAT.md
Normal file
|
|
@ -0,0 +1,198 @@
|
|||
# Trace / compare log format (v1)
|
||||
|
||||
Wire format between the shim (`sots-engine/src/shim/`, C++, no JSON library) and the
|
||||
harness in this directory (`tracecmp.py`, stdlib Python). One file per shim run; golden
|
||||
traces live in `verify/traces/`, harness reports in `verify/results/compare/`.
|
||||
|
||||
Design goals, in order: (1) trivially emittable from C++ with `fprintf`; (2) every value
|
||||
carries its C type so the harness can apply the right equality rule; (3) big memory
|
||||
regions never blow up the log (hash, summarize, cap); (4) the same record shape serves
|
||||
online `compare` (shim runs original + ours) and offline `replay` (harness diffs a golden
|
||||
`trace` log against a host-side implementation's output).
|
||||
|
||||
## 1. Container: JSON Lines
|
||||
|
||||
- UTF-8, no BOM. One JSON object per line, terminated by `\n` (LF only). No blank lines,
|
||||
no comments, no trailing commas. A line that does not parse is an **invalid record**.
|
||||
- The first line SHOULD be a `meta` record (section 5). Everything else is a **call
|
||||
record** (section 2).
|
||||
- Records are appended in emission order; `call_id` (not line order) is the identity.
|
||||
- Log files are named `<run-tag>.jsonl` (e.g. `verify/traces/m1-cfgvar-20260907.jsonl`).
|
||||
|
||||
## 2. Call record
|
||||
|
||||
```json
|
||||
{"ts":123456,"hook":"CfgVar_RegisterKey","mode":"trace","call_id":17,"thread":4120,
|
||||
"args":[{"t":"str","v":"ForceSingleCore"},{"t":"i32","v":1}],
|
||||
"ret":{"t":"bool","v":true},
|
||||
"side":{"cfg_table":{"before":{"t":"bytes","n":4096,"sha256":"…"},
|
||||
"after":{"t":"bytes","n":4096,"sha256":"…"}}}}
|
||||
```
|
||||
|
||||
| field | type | required | meaning |
|
||||
|---|---|---|---|
|
||||
| `ts` | int | yes | microseconds since shim init (QPC-based). Informational; never compared. |
|
||||
| `hook` | string | yes | hook name — the function's recovered name (`Game::…` / `Mars::…` or `FUN_xxxxxxxx`). |
|
||||
| `mode` | `"trace"` \| `"compare"` \| `"replace"` | yes | which mode the hook ran in for this call. |
|
||||
| `call_id` | int ≥ 0 | yes | process-global, monotonically increasing, atomic. Unique within a log. |
|
||||
| `thread` | int | yes | OS thread id. Informational. |
|
||||
| `depth` | int | no | re-entrancy depth (0 = outermost). Informational. |
|
||||
| `args` | array of typed values | yes | inputs, in declaration order. `n` names them when known. Never compared (they are the snapshot both sides ran on). |
|
||||
| `ret` | typed value or `null` | yes | return value of the **original** (trace/compare) or of **ours** (replace). `null` = void. |
|
||||
| `side` | object | yes (may be `{}`) | declared side effects: `name -> {"before": tv|null, "after": tv}`. `after` is compared; `before` is informational (a mismatch in `before` between original and ours would be a snapshot bug and is reported as a warning, not a divergence). |
|
||||
| `ours` | object | compare only | `{"ret": tv|null, "side": {name: {"after": tv}}}` — the reimplementation's outputs on the same snapshot. |
|
||||
| `diverged` | bool | compare only | the shim's own verdict. The harness recomputes and reports disagreements as warnings. |
|
||||
| `diff` | array of diff entries | compare only | the shim's own diff (may be `[]` or truncated). Advisory. |
|
||||
| `err` | string | no | the hook could not capture (exception in ours, snapshot failure…). Counts as a divergence in `compare`. |
|
||||
| `note` | string | no | free text. Ignored. |
|
||||
|
||||
Unknown top-level keys are a validation **warning** (not an error) so the shim can add
|
||||
fields without breaking older harnesses.
|
||||
|
||||
`replace` records have no `ours`; `ret`/`side` are the reimplementation's. They are
|
||||
counted and validated but never diffed (there is nothing to diff against).
|
||||
|
||||
## 3. Typed values (`tv`)
|
||||
|
||||
Every value is an object `{"t": TYPE, "v": VALUE}` plus optional `"n": "name"`.
|
||||
TYPEs and their equality rule:
|
||||
|
||||
| `t` | `v` | equality |
|
||||
|---|---|---|
|
||||
| `bool` | `true`/`false` | exact |
|
||||
| `i8 i16 i32 i64 u8 u16 u32 u64` | JSON integer. i64/u64 above 2^53 MUST be a decimal **string** (`"v":"18446744073709551615"`) — the harness accepts both int and string. | exact |
|
||||
| `f32`, `f64` | JSON number, or the string `"nan"`, `"inf"`, `"-inf"`. Emit with `%.9g` (f32) / `%.17g` (f64). | tolerant (section 6) |
|
||||
| `str` | JSON string carrying **raw bytes** of a `char*`/`std::string` (cp1252 in this engine). Each byte maps to one code point U+0000–U+00FF (see escaping, section 4). | exact |
|
||||
| `wstr` | JSON string of a `wchar_t*` (UTF-16 code units → `\uXXXX`). | exact |
|
||||
| `ptr` | hex string `"0x00a36fd0"` | **ignored** by default (ours allocates elsewhere). A hook policy may set `"ptr":"exact"`. |
|
||||
| `enum` | integer, optional `"name"` sibling with the symbolic name | exact on the integer |
|
||||
| `null` | `null` | exact (only equals `null`) |
|
||||
| `bytes` | see below | exact on `sha256` (and on `hex` when both inline) |
|
||||
| `list` | JSON array of tv | same length, element-wise, ordered |
|
||||
| `set` | JSON array of tv | unordered multiset: sort both by canonical text, then element-wise |
|
||||
| `struct` | JSON object `{field: tv}` | same key set, field-wise (recursion) |
|
||||
| `json` | any canonical-JSON value (section 7) — used by the oracle bridge and host-side tests | structural exact, with the hook's float tolerance applied to JSON numbers that are floats on **both** sides |
|
||||
|
||||
A type mismatch between the two sides (`i32` vs `u32`, `str` vs `wstr`) is a divergence
|
||||
(`why: "type"`). Type mismatch does not apply inside `json` values, where int vs float
|
||||
is a divergence unless both are numbers and the tolerance policy is `"numeric"`.
|
||||
|
||||
### `bytes` — binary blobs and memory regions
|
||||
|
||||
```json
|
||||
{"t":"bytes","n":4096,"sha256":"<64 hex>","hex":"<2n hex>"}
|
||||
{"t":"bytes","n":1048576,"sha256":"<64 hex>","head":"<first 32 bytes hex>"}
|
||||
```
|
||||
|
||||
- `n` (length in bytes) and `sha256` (lowercase hex of the SHA-256 of the raw bytes) are
|
||||
**always** present, even for `n = 0` (sha256 of the empty string).
|
||||
- `hex` (lowercase, no separators) is present when `n <= inline_max`. `inline_max` is a
|
||||
shim setting recorded in `meta` (default **256**). `hex` is the only inline encoding;
|
||||
base64 is not used (harder to eyeball, no size win worth a second decoder).
|
||||
- Larger blobs carry `head` (first 32 bytes, hex) for eyeballing only.
|
||||
- When both sides are inline and differ, the harness reports the first differing byte
|
||||
offset; when hashed, only that the hash differs.
|
||||
- A **memory region** is just a `bytes` value captured at a declared `(address, length)`;
|
||||
the region's name is the key in `side`. Structured regions should be captured as
|
||||
`struct` of typed values instead so diffs point at the field, not a byte offset.
|
||||
|
||||
## 4. Emitter rules (C++ without a JSON library)
|
||||
|
||||
The emitter is a set of `fprintf`s. What it must guarantee:
|
||||
|
||||
1. **Key order is fixed**: `ts, hook, mode, call_id, thread, depth, args, ret, side, ours,
|
||||
diverged, diff, err, note`. (The harness does not care, but fixed order makes logs
|
||||
`diff`-able by eye and grep-able.)
|
||||
2. **Strings** (`hook`, `str`/`wstr` `v`, `n`, `err`, `note`, struct field names, side
|
||||
names): write `"`, then for each unit:
|
||||
- `"` → `\"`, `\` → `\\`
|
||||
- `\n` → `\n`, `\r` → `\r`, `\t` → `\t` (literal backslash-letter)
|
||||
- any other unit `< 0x20`, and for `str` any byte `>= 0x7f` → `\u00XX` (lowercase hex of
|
||||
the byte), for `wstr` any unit `>= 0x7f` → `\uXXXX`
|
||||
- everything else verbatim. Close with `"`.
|
||||
This is exactly what Python's `json.dumps(ensure_ascii=True)` produces for a
|
||||
latin-1-decoded byte string (except Python also short-forms `\b` and `\f`; the harness
|
||||
accepts both). No raw bytes ≥ 0x80 ever appear in the log, so the file is ASCII-only.
|
||||
3. **Numbers**: integers with `%lld`/`%llu`; floats with `%.9g` / `%.17g`; non-finite floats
|
||||
as the strings `"nan"`, `"inf"`, `"-inf"`. Never emit `nan`/`inf` bare (invalid JSON).
|
||||
4. **Booleans** `true`/`false`; **void** `null`.
|
||||
5. **No trailing commas.** Build arrays/objects with a "first element" flag.
|
||||
6. **One `fprintf` chain per record, ending in `"}\n"` and a `fflush`** at least on
|
||||
`compare` divergences and on shim teardown, so a crash mid-run leaves a usable log
|
||||
(the harness tolerates a truncated last line: it is reported as one invalid record).
|
||||
7. **Thread safety**: take a lock around the whole record; `call_id` is
|
||||
`InterlockedIncrement`-ed *before* the original is called, so nested hooks get
|
||||
increasing ids and `depth` says who is inside whom.
|
||||
8. `sha256`: any tiny public-domain SHA-256; hex lowercase.
|
||||
|
||||
## 5. `meta` record (first line)
|
||||
|
||||
```json
|
||||
{"meta":{"format":1,"build":"sots-engine 0.0.3 g1a2b3c4","exe_sha256":"…",
|
||||
"started":"2026-09-07T18:00:00Z","inline_max":256,
|
||||
"hooks":{"CfgVar_RegisterKey":{"ftol":0,"ptr":"ignore"},
|
||||
"Mars::ParseBlock":{"ftol":1e-6,"ftol_kind":"rel","unordered":["ret.v.items"]}}}}
|
||||
```
|
||||
|
||||
- `format`: this spec's version (1). The harness refuses other versions.
|
||||
- `hooks`: per-hook **policy**, the same keys `tracecmp.py --tolerance` accepts. CLI flags
|
||||
override the log's policy; the log's policy overrides harness defaults.
|
||||
- Everything else is informational and copied into the report.
|
||||
|
||||
## 6. Divergence rules
|
||||
|
||||
A call **diverges** when any of: `err` is set; `ret` differs; any `side[name].after`
|
||||
differs from `ours.side[name].after`; a `side` name exists on one side only (`why:
|
||||
"missing"`). Comparison is by the typed-value table above, recursively. Paths in diff
|
||||
entries: `ret`, `side.<name>.after`, then `.v[<index>]` for list/set elements,
|
||||
`.v.<field>` for struct fields, `.<key>`/`[<i>]` inside `json` values.
|
||||
|
||||
```json
|
||||
{"path":"side.cfg_table.after.v.ForceSingleCore","why":"exact","orig":{"t":"i32","v":1},"ours":{"t":"i32","v":0}}
|
||||
```
|
||||
|
||||
`why` ∈ `exact | ftol | type | len | missing | extra | hash | err`.
|
||||
|
||||
**Float tolerance policy** (per hook; default `ftol = 0`, i.e. exact by value with
|
||||
`nan == nan`):
|
||||
- `ftol`: number ≥ 0.
|
||||
- `ftol_kind`: `"abs"` (default) — `|a-b| <= ftol`; `"rel"` — `|a-b| <= ftol * max(|a|,|b|)`;
|
||||
`"ulp"` — within `ftol` units-in-last-place of the value's own width (f32/f64).
|
||||
- `nan`, `inf`, `-inf`: equal only to themselves, whatever the tolerance.
|
||||
- `f32` values are rounded to float32 on both sides before comparing: `%.9g` round-trips
|
||||
the float32 exactly but not the double it was widened to. `f64` compares as-is.
|
||||
- `f32` compared to `f64` is a type divergence; do not mix.
|
||||
|
||||
**Unordered fields**: `set` values are always unordered. A policy may additionally list
|
||||
`unordered: [<path>, …]` to treat specific `list` paths as sets (path syntax as above,
|
||||
without the `.v[i]` tail). Multiset semantics: element multiplicity matters.
|
||||
|
||||
**Pointers**: `ptr` is ignored unless the policy says `"ptr":"exact"`. Two `ptr` values
|
||||
where exactly one is `"0x0"` is always a divergence (`null` vs non-null is semantic).
|
||||
|
||||
## 7. Canonical JSON (for `json` values and the oracle bridge)
|
||||
|
||||
The form a host-side C++ test dump must reproduce byte-for-byte so a plain `diff` (or
|
||||
`tracecmp.py --replay`) works. It is Python's
|
||||
`json.dumps(obj, sort_keys=True, ensure_ascii=True, separators=(",", ":"))` after
|
||||
normalization by `oracle_parsers.canonical()`:
|
||||
|
||||
- objects: keys sorted by code point, no whitespace anywhere (`{"a":1,"b":[1,2]}`);
|
||||
- strings: byte-preserving — the parsers decode cp1252, the canonicalizer re-encodes to
|
||||
cp1252 and maps each byte to U+00XX, so the escaping in section 4 applies verbatim
|
||||
(a C++ dump escapes raw bytes, no charset table needed);
|
||||
- integers: decimal, no leading `+`/zeros;
|
||||
- floats: rounded to **float32**, printed with `%.9g`, then `.0` appended when the text has
|
||||
no `.`, `e` or `n` (so `1.0` stays distinguishable from the int `1`, and `-0` → `-0.0`);
|
||||
non-finite → the strings `"nan"`, `"inf"`, `"-inf"`;
|
||||
- booleans `true`/`false`; `null` for None; tuples become arrays; dataclasses become
|
||||
objects of their fields;
|
||||
- the file ends with a single `\n`.
|
||||
|
||||
## 8. Replay (offline) input
|
||||
|
||||
`tracecmp.py --replay GOLDEN.jsonl IMPL.jsonl`: `GOLDEN` is a normal log (any mode; usually
|
||||
`trace`). `IMPL` is JSONL whose records need only `call_id`, `ret`, `side` (`side` may be
|
||||
`{name: tv}` or `{name: {"after": tv}}`; `hook` optional but checked when present). Each
|
||||
golden call_id must appear in `IMPL` (`why: "missing"` otherwise); extra call_ids in
|
||||
`IMPL` are warnings. The diff rules are those of section 6, with `IMPL` playing `ours`.
|
||||
474
verify/harness/compare/mkfixture.py
Normal file
474
verify/harness/compare/mkfixture.py
Normal file
|
|
@ -0,0 +1,474 @@
|
|||
#!/usr/bin/env python3
|
||||
"""mkfixture.py -- synthetic TRACE_FORMAT.md logs for testing tracecmp.py.
|
||||
|
||||
usage: mkfixture.py OUTDIR [--seed N] [--calls N]
|
||||
|
||||
Writes to OUTDIR:
|
||||
trace_clean.jsonl golden `trace` log (meta + N calls over 3 hooks)
|
||||
compare_clean.jsonl `compare` log, ours == original everywhere
|
||||
compare_bad.jsonl `compare` log with injected divergences (see INJECTED)
|
||||
replay_impl_ok.jsonl impl output matching trace_clean (for --replay)
|
||||
replay_impl_bad.jsonl impl output with one wrong ret and one missing call
|
||||
invalid.jsonl broken lines: bad JSON, missing field, bad type, dup id
|
||||
|
||||
The records are written by emit_record(), a Python transliteration of the
|
||||
C++ emitter rules in TRACE_FORMAT.md section 4 (fixed key order, hand-rolled
|
||||
string escaping, %.9g floats) -- NOT json.dumps -- so the tests prove that an
|
||||
emitter following those rules produces logs the harness reads back exactly.
|
||||
|
||||
Stdlib only.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import copy
|
||||
import hashlib
|
||||
import os
|
||||
import random
|
||||
import struct
|
||||
import sys
|
||||
from typing import Any
|
||||
|
||||
KEY_ORDER = ["ts", "hook", "mode", "call_id", "thread", "depth", "args", "ret", "side",
|
||||
"ours", "diverged", "diff", "err", "note"]
|
||||
INLINE_MAX = 256
|
||||
HOOKS = ["CfgVar_RegisterKey", "Manifest_Load", "Mars::ParseBlock"]
|
||||
|
||||
|
||||
# --- typed-value constructors -----------------------------------------------------
|
||||
|
||||
def i32(v, n=None):
|
||||
return _tv("i32", v, n)
|
||||
|
||||
|
||||
def u32(v, n=None):
|
||||
return _tv("u32", v, n)
|
||||
|
||||
|
||||
def u64(v, n=None):
|
||||
return _tv("u64", v if v < 2**53 else str(v), n)
|
||||
|
||||
|
||||
def f32(v, n=None):
|
||||
if v != v:
|
||||
return _tv("f32", "nan", n)
|
||||
if v in (float("inf"), float("-inf")):
|
||||
return _tv("f32", "inf" if v > 0 else "-inf", n)
|
||||
return _tv("f32", round_f32(v), n)
|
||||
|
||||
|
||||
def boolean(v, n=None):
|
||||
return _tv("bool", bool(v), n)
|
||||
|
||||
|
||||
def s(v: str, n=None):
|
||||
"""cp1252 byte string -> str tv (each byte one code point)."""
|
||||
return _tv("str", v.encode("cp1252").decode("latin-1"), n)
|
||||
|
||||
|
||||
def ptr(addr: int, n=None):
|
||||
return _tv("ptr", "0x%08x" % addr, n)
|
||||
|
||||
|
||||
def enum(v: int, name: str | None = None, n=None):
|
||||
tv = _tv("enum", v, n)
|
||||
if name:
|
||||
tv["name"] = name
|
||||
return tv
|
||||
|
||||
|
||||
def null():
|
||||
return {"t": "null", "v": None}
|
||||
|
||||
|
||||
def lst(items, n=None):
|
||||
return _tv("list", list(items), n)
|
||||
|
||||
|
||||
def sset(items, n=None):
|
||||
return _tv("set", list(items), n)
|
||||
|
||||
|
||||
def struct_(fields: dict, n=None):
|
||||
return _tv("struct", dict(fields), n)
|
||||
|
||||
|
||||
def jsonv(v, n=None):
|
||||
return _tv("json", v, n)
|
||||
|
||||
|
||||
def by(data: bytes, inline_max: int = INLINE_MAX):
|
||||
"""bytes tv: n + sha256 always; hex inline when small, else head. (No 'n'
|
||||
name on bytes -- the key is taken by the length; name the side region instead.)"""
|
||||
tv: dict = {"t": "bytes", "n": len(data), "sha256": hashlib.sha256(data).hexdigest()}
|
||||
if len(data) <= inline_max:
|
||||
tv["hex"] = data.hex()
|
||||
else:
|
||||
tv["head"] = data[:32].hex()
|
||||
return tv
|
||||
|
||||
|
||||
def _tv(t, v, n):
|
||||
tv = {"t": t, "v": v}
|
||||
if n:
|
||||
tv["n"] = n
|
||||
return tv
|
||||
|
||||
|
||||
def round_f32(x: float) -> float:
|
||||
return struct.unpack("<f", struct.pack("<f", x))[0]
|
||||
|
||||
|
||||
# --- reference emitter (mirrors the C++ rules) --------------------------------------
|
||||
|
||||
def esc(u: str, wide: bool = False) -> str:
|
||||
out = ['"']
|
||||
for c in u:
|
||||
o = ord(c)
|
||||
if c == '"':
|
||||
out.append('\\"')
|
||||
elif c == "\\":
|
||||
out.append("\\\\")
|
||||
elif c == "\n":
|
||||
out.append("\\n")
|
||||
elif c == "\r":
|
||||
out.append("\\r")
|
||||
elif c == "\t":
|
||||
out.append("\\t")
|
||||
elif o < 0x20 or o >= 0x7F:
|
||||
out.append("\\u%04x" % o)
|
||||
else:
|
||||
out.append(c)
|
||||
out.append('"')
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def fmt_num(v: Any, t: str | None = None) -> str:
|
||||
if isinstance(v, bool):
|
||||
return "true" if v else "false"
|
||||
if isinstance(v, int):
|
||||
return "%d" % v
|
||||
if isinstance(v, float):
|
||||
if v != v:
|
||||
return '"nan"'
|
||||
if v in (float("inf"), float("-inf")):
|
||||
return '"inf"' if v > 0 else '"-inf"'
|
||||
return ("%.9g" if t == "f32" else "%.17g") % v
|
||||
raise TypeError(v)
|
||||
|
||||
|
||||
def emit_value(x: Any, t: str | None = None, plain: bool = False) -> str:
|
||||
"""Generic JSON value emitter with the spec's escaping. Nested dicts that
|
||||
look like typed values are emitted as such unless plain=True (json payloads)."""
|
||||
if x is None:
|
||||
return "null"
|
||||
if isinstance(x, (bool, int, float)):
|
||||
return fmt_num(x, t)
|
||||
if isinstance(x, str):
|
||||
return esc(x)
|
||||
if isinstance(x, list):
|
||||
return "[" + ",".join(emit_value(e, plain=plain) for e in x) + "]"
|
||||
if isinstance(x, dict):
|
||||
if not plain and "t" in x and ("v" in x or x["t"] == "bytes"):
|
||||
return emit_tv(x)
|
||||
return "{" + ",".join(esc(k) + ":" + emit_value(v, plain=plain) for k, v in x.items()) + "}"
|
||||
raise TypeError(x)
|
||||
|
||||
|
||||
def emit_tv(tv: dict) -> str:
|
||||
t = tv["t"]
|
||||
parts = ['"t":' + esc(t)]
|
||||
if t == "bytes":
|
||||
parts.append('"n":%d' % tv["n"])
|
||||
parts.append('"sha256":' + esc(tv["sha256"]))
|
||||
if "hex" in tv:
|
||||
parts.append('"hex":' + esc(tv["hex"]))
|
||||
if "head" in tv:
|
||||
parts.append('"head":' + esc(tv["head"]))
|
||||
else:
|
||||
v = tv["v"]
|
||||
if t in ("list", "set"):
|
||||
parts.append('"v":[' + ",".join(emit_tv(e) for e in v) + "]")
|
||||
elif t == "struct":
|
||||
parts.append('"v":{' + ",".join(esc(k) + ":" + emit_tv(e) for k, e in v.items()) + "}")
|
||||
elif t == "json":
|
||||
parts.append('"v":' + emit_value(v, plain=True))
|
||||
elif t in ("f32", "f64"):
|
||||
parts.append('"v":' + (esc(v) if isinstance(v, str) else fmt_num(v, t)))
|
||||
elif t == "wstr":
|
||||
parts.append('"v":' + esc(v, wide=True))
|
||||
elif v is None:
|
||||
parts.append('"v":null')
|
||||
elif isinstance(v, str):
|
||||
parts.append('"v":' + esc(v))
|
||||
else:
|
||||
parts.append('"v":' + fmt_num(v))
|
||||
if "name" in tv:
|
||||
parts.append('"name":' + esc(tv["name"]))
|
||||
if "n" in tv and t != "bytes":
|
||||
parts.append('"n":' + esc(tv["n"]))
|
||||
return "{" + ",".join(parts) + "}"
|
||||
|
||||
|
||||
def emit_side(side: dict) -> str:
|
||||
ents = []
|
||||
for name, ent in side.items():
|
||||
inner = []
|
||||
if "before" in ent:
|
||||
inner.append('"before":' + ("null" if ent["before"] is None else emit_tv(ent["before"])))
|
||||
inner.append('"after":' + emit_tv(ent["after"]))
|
||||
ents.append(esc(name) + ":{" + ",".join(inner) + "}")
|
||||
return "{" + ",".join(ents) + "}"
|
||||
|
||||
|
||||
def emit_record(rec: dict) -> str:
|
||||
parts = []
|
||||
for k in KEY_ORDER:
|
||||
if k not in rec:
|
||||
continue
|
||||
v = rec[k]
|
||||
if k in ("ts", "call_id", "thread", "depth"):
|
||||
parts.append(esc(k) + ":" + fmt_num(v))
|
||||
elif k in ("hook", "mode", "err", "note"):
|
||||
parts.append(esc(k) + ":" + esc(v))
|
||||
elif k == "args":
|
||||
parts.append('"args":[' + ",".join(emit_tv(a) for a in v) + "]")
|
||||
elif k == "ret":
|
||||
parts.append('"ret":' + ("null" if v is None else emit_tv(v)))
|
||||
elif k == "side":
|
||||
parts.append('"side":' + emit_side(v))
|
||||
elif k == "ours":
|
||||
parts.append('"ours":{"ret":' + ("null" if v["ret"] is None else emit_tv(v["ret"])) +
|
||||
',"side":' + emit_side(v["side"]) + "}")
|
||||
elif k == "diverged":
|
||||
parts.append('"diverged":' + ("true" if v else "false"))
|
||||
elif k == "diff":
|
||||
parts.append('"diff":' + emit_value(v))
|
||||
else:
|
||||
raise AssertionError(k)
|
||||
return "{" + ",".join(parts) + "}\n"
|
||||
|
||||
|
||||
def emit_meta(meta: dict) -> str:
|
||||
return '{"meta":' + emit_value(meta) + "}\n"
|
||||
|
||||
|
||||
def write_log(path: str, meta: dict | None, records: list[dict], raw_lines: list[str] | None = None) -> None:
|
||||
with open(path, "w", encoding="utf-8", newline="\n") as f:
|
||||
if meta is not None:
|
||||
f.write(emit_meta(meta))
|
||||
for r in records:
|
||||
f.write(emit_record(r))
|
||||
for line in raw_lines or []:
|
||||
f.write(line if line.endswith("\n") else line + "\n")
|
||||
|
||||
|
||||
# --- scenario generation ----------------------------------------------------------------
|
||||
|
||||
def meta(**hooks) -> dict:
|
||||
return {"format": 1, "build": "mkfixture 1", "exe_sha256": "0" * 64,
|
||||
"started": "2026-09-07T00:00:00Z", "inline_max": INLINE_MAX, "hooks": hooks}
|
||||
|
||||
|
||||
def gen_calls(n: int, seed: int = 1) -> list[dict]:
|
||||
"""n trace calls cycling over HOOKS, with every tv type represented."""
|
||||
rng = random.Random(seed)
|
||||
recs = []
|
||||
for cid in range(n):
|
||||
hook = HOOKS[cid % len(HOOKS)]
|
||||
ts = 1000 + cid * 37
|
||||
if hook == "CfgVar_RegisterKey":
|
||||
key = rng.choice(["ForceSingleCore", "Résumé", "Tab\tKey", 'Quote"d', "Back\\slash"])
|
||||
blob = bytes(rng.getrandbits(8) for _ in range(rng.choice([0, 16, 512])))
|
||||
rec = {"ts": ts, "hook": hook, "mode": "trace", "call_id": cid, "thread": 4120, "depth": 0,
|
||||
"args": [s(key, "key"), i32(rng.randint(-5, 5), "value"), ptr(0x00a36fd0 + cid * 16, "table")],
|
||||
"ret": boolean(True),
|
||||
"side": {"cfg_table": {"before": by(blob), "after": by(blob[::-1])}}}
|
||||
elif hook == "Manifest_Load":
|
||||
ids = [rng.randint(1, 400) for _ in range(rng.randint(1, 5))]
|
||||
rec = {"ts": ts, "hook": hook, "mode": "trace", "call_id": cid, "thread": 4120,
|
||||
"args": [s("Weapons/_weapons.txt", "path")],
|
||||
"ret": i32(len(ids), "count"),
|
||||
"side": {"registry": {"after": struct_({
|
||||
"entries": lst(struct_({"id": u32(i), "name": s("w%d.weapon" % i)}) for i in ids),
|
||||
"deleted": sset(u32(i) for i in rng.sample(range(500, 600), 3)),
|
||||
"flags": enum(rng.randint(0, 3), "MF_%d" % 0),
|
||||
"big": u64(2**60 + cid),
|
||||
"nothing": null()})}}}
|
||||
else:
|
||||
vals = [round_f32(rng.uniform(-10, 10)) for _ in range(3)]
|
||||
rec = {"ts": ts, "hook": hook, "mode": "trace", "call_id": cid, "thread": 4124, "depth": 1,
|
||||
"args": [s("weapon { damage %g }" % vals[0], "text"), u32(cid, "len")],
|
||||
"ret": jsonv({"weapon": {"damage": vals[0], "range": vals[1], "name": "Laseré", "count": 3,
|
||||
"ok": True, "tags": ["a", "b"]}}),
|
||||
"side": {"scratch": {"after": lst(f32(v) for v in vals)},
|
||||
"nanbox": {"after": f32(float("nan"))}}}
|
||||
recs.append(rec)
|
||||
return recs
|
||||
|
||||
|
||||
def to_compare(recs: list[dict]) -> list[dict]:
|
||||
"""Clone trace records into compare records where ours == original."""
|
||||
out = []
|
||||
for r in recs:
|
||||
c = dict(r)
|
||||
c["mode"] = "compare"
|
||||
c["ours"] = {"ret": _deep(r["ret"]), "side": {k: {"after": _deep(v["after"])} for k, v in r["side"].items()}}
|
||||
c["diverged"] = False
|
||||
c["diff"] = []
|
||||
out.append(c)
|
||||
return out
|
||||
|
||||
|
||||
def _deep(x):
|
||||
return copy.deepcopy(x)
|
||||
|
||||
|
||||
def inject_divergences(recs: list[dict]) -> tuple[list[dict], list[dict]]:
|
||||
"""Mutate `ours` on a handful of compare records (the INJECTED cases below).
|
||||
Returns (records, expectations); expectation = {call_id, diverges_exact,
|
||||
diverges_tol (abs 1e-3), why, path}. Needs >= 12 records (4 per hook)."""
|
||||
recs = [_deep(r) for r in recs]
|
||||
exp = []
|
||||
by_hook: dict[str, list[dict]] = {}
|
||||
for r in recs:
|
||||
by_hook.setdefault(r["hook"], []).append(r)
|
||||
|
||||
cfg = by_hook["CfgVar_RegisterKey"]
|
||||
# 1. bool ret flipped
|
||||
cfg[0]["ours"]["ret"] = boolean(False)
|
||||
exp.append({"call_id": cfg[0]["call_id"], "diverges_exact": True, "diverges_tol": True, "why": "exact", "path": "ret"})
|
||||
# 2. bytes region differs (same length, different content) -> hash
|
||||
orig = cfg[1]["side"]["cfg_table"]["after"]
|
||||
n = orig["n"]
|
||||
if n == 0: # make the region non-empty on both sides so a content diff is possible
|
||||
n = 16
|
||||
cfg[1]["side"]["cfg_table"]["after"] = orig = by(bytes(range(n)))
|
||||
data = bytes(n)
|
||||
if data.hex() == orig.get("hex"):
|
||||
data = bytes([255]) + data[1:]
|
||||
cfg[1]["ours"]["side"]["cfg_table"]["after"] = by(data)
|
||||
exp.append({"call_id": cfg[1]["call_id"], "diverges_exact": True, "diverges_tol": True, "why": "hash",
|
||||
"path": "side.cfg_table.after"})
|
||||
# 3. shim lies: says diverged=true on an identical record -> harness warning only
|
||||
cfg[2]["diverged"] = True
|
||||
exp.append({"call_id": cfg[2]["call_id"], "diverges_exact": False, "diverges_tol": False, "why": None,
|
||||
"path": None, "warn": "shim said diverged"})
|
||||
|
||||
man = by_hook["Manifest_Load"]
|
||||
# 4. set reordered -> NOT a divergence
|
||||
st = man[0]["ours"]["side"]["registry"]["after"]
|
||||
st["v"]["deleted"]["v"] = list(reversed(st["v"]["deleted"]["v"]))
|
||||
exp.append({"call_id": man[0]["call_id"], "diverges_exact": False, "diverges_tol": False, "why": None, "path": None})
|
||||
# 5. list reordered (2+ entries) or entry id changed -> divergence
|
||||
st = man[1]["ours"]["side"]["registry"]["after"]
|
||||
ent = st["v"]["entries"]["v"]
|
||||
if len(ent) >= 2 and ent[0] != ent[-1]:
|
||||
ent.reverse()
|
||||
else:
|
||||
ent[0]["v"]["id"] = u32(ent[0]["v"]["id"]["v"] + 1)
|
||||
exp.append({"call_id": man[1]["call_id"], "diverges_exact": True, "diverges_tol": True, "why": "exact",
|
||||
"path": "side.registry.after.v.entries"})
|
||||
# 6. struct field missing
|
||||
del man[2]["ours"]["side"]["registry"]["after"]["v"]["big"]
|
||||
exp.append({"call_id": man[2]["call_id"], "diverges_exact": True, "diverges_tol": True, "why": "missing",
|
||||
"path": "side.registry.after.v.big"})
|
||||
|
||||
mp = by_hook["Mars::ParseBlock"]
|
||||
# 7. f32 nudged by 1e-5 -> diverges exact, passes abs 1e-3
|
||||
sc = mp[0]["ours"]["side"]["scratch"]["after"]["v"]
|
||||
sc[1] = f32(sc[1]["v"] + 1e-5)
|
||||
exp.append({"call_id": mp[0]["call_id"], "diverges_exact": True, "diverges_tol": False, "why": "exact",
|
||||
"path": "side.scratch.after.v[1]"})
|
||||
# 8. json float nudged by 0.5 -> diverges both
|
||||
rv = mp[1]["ours"]["ret"] = _deep(mp[1]["ret"])
|
||||
rv["v"]["weapon"]["range"] = rv["v"]["weapon"]["range"] + 0.5
|
||||
exp.append({"call_id": mp[1]["call_id"], "diverges_exact": True, "diverges_tol": True, "why": "exact",
|
||||
"path": "ret.weapon.range"})
|
||||
# 9. err record
|
||||
mp[2]["err"] = "ours threw std::bad_alloc"
|
||||
del mp[2]["ours"]
|
||||
exp.append({"call_id": mp[2]["call_id"], "diverges_exact": True, "diverges_tol": True, "why": "err", "path": "call"})
|
||||
# 10. ptr differs -> NOT a divergence (default policy); type mismatch i32 vs u32 -> divergence
|
||||
if len(cfg) > 3:
|
||||
cfg[3]["ours"]["ret"] = boolean(True)
|
||||
cfg[3]["ours"]["side"]["cfg_table"]["after"] = cfg[3]["side"]["cfg_table"]["after"]
|
||||
exp.append({"call_id": cfg[3]["call_id"], "diverges_exact": False, "diverges_tol": False, "why": None, "path": None})
|
||||
if len(man) > 3:
|
||||
man[3]["ours"]["ret"] = u32(man[3]["ret"]["v"])
|
||||
exp.append({"call_id": man[3]["call_id"], "diverges_exact": True, "diverges_tol": True, "why": "type", "path": "ret"})
|
||||
# the shim's own verdict is truthful for every injected case except #3
|
||||
by_id = {r["call_id"]: r for r in recs}
|
||||
for e in exp:
|
||||
if "warn" not in e and "diverged" in by_id[e["call_id"]]:
|
||||
by_id[e["call_id"]]["diverged"] = e["diverges_exact"]
|
||||
return recs, exp
|
||||
|
||||
|
||||
def impl_records(recs: list[dict]) -> list[dict]:
|
||||
"""Implementation-output records (replay input) equal to the golden outputs."""
|
||||
return [{"call_id": r["call_id"], "hook": r["hook"], "ret": r["ret"],
|
||||
"side": {k: v["after"] for k, v in r["side"].items()}} for r in recs]
|
||||
|
||||
|
||||
def emit_impl(path: str, recs: list[dict]) -> None:
|
||||
with open(path, "w", encoding="utf-8", newline="\n") as f:
|
||||
for r in recs:
|
||||
side = "{" + ",".join(esc(k) + ":" + emit_tv(v) for k, v in r["side"].items()) + "}"
|
||||
f.write('{"call_id":%d,"hook":%s,"ret":%s,"side":%s}\n' % (
|
||||
r["call_id"], esc(r["hook"]), "null" if r["ret"] is None else emit_tv(r["ret"]), side))
|
||||
|
||||
|
||||
def invalid_lines() -> list[str]:
|
||||
return [
|
||||
'{"ts":1,"hook":"X","mode":"trace","call_id":900,"thread":1,"args":[],"ret":null,"side":{}}', # ok
|
||||
'{"ts":1,"hook":"X","mode":"trace","call_id":901,"thread":1,"args":[],"ret":null,"side":{}', # bad JSON
|
||||
'{"ts":1,"hook":"X","mode":"trace","call_id":902,"thread":1,"args":[]}', # missing ret/side
|
||||
'{"ts":1,"hook":"X","mode":"trace","call_id":903,"thread":1,"args":[{"t":"i32","v":"x"}],"ret":null,"side":{}}', # bad int
|
||||
'{"ts":1,"hook":"X","mode":"trace","call_id":900,"thread":1,"args":[],"ret":null,"side":{}}', # dup id
|
||||
'{"ts":1,"hook":"X","mode":"compare","call_id":904,"thread":1,"args":[],"ret":null,"side":{}}', # compare w/o ours
|
||||
'{"ts":1,"hook":"X","mode":"trace","call_id":905,"thread":1,"args":[{"t":"bytes","n":2,"sha256":"zz"}],"ret":null,"side":{}}',
|
||||
'{"ts":1,"hook":"X","mode":"trace","call_id":906,"thread":1,"args":[],"ret":{"t":"u8","v":300},"side":{}}',
|
||||
'{"ts":1,"hook":"X","mode":"trace","call_id":907,"thread":1,"args":[],"ret":null,"side":{},"bogus":1}', # ok + warning
|
||||
]
|
||||
|
||||
|
||||
def build_all(outdir: str, n: int = 12, seed: int = 1) -> dict:
|
||||
os.makedirs(outdir, exist_ok=True)
|
||||
trace = gen_calls(n, seed)
|
||||
m = meta(**{"Mars::ParseBlock": {"ftol": 0}})
|
||||
paths = {k: os.path.join(outdir, k + ".jsonl") for k in
|
||||
("trace_clean", "compare_clean", "compare_bad", "replay_impl_ok", "replay_impl_bad", "invalid")}
|
||||
write_log(paths["trace_clean"], m, trace)
|
||||
clean = to_compare(trace)
|
||||
write_log(paths["compare_clean"], m, clean)
|
||||
bad, exp = inject_divergences(clean)
|
||||
write_log(paths["compare_bad"], m, bad)
|
||||
impl = impl_records(trace)
|
||||
emit_impl(paths["replay_impl_ok"], impl)
|
||||
impl_bad = _deep(impl)
|
||||
impl_bad[0]["ret"] = boolean(False) # wrong ret on call 0
|
||||
dropped = impl_bad.pop(1)["call_id"] # missing call 1
|
||||
impl_bad.append({"call_id": 9999, "hook": "Extra", "ret": None, "side": {}}) # extra -> warning
|
||||
emit_impl(paths["replay_impl_bad"], impl_bad)
|
||||
write_log(paths["invalid"], m, [], invalid_lines())
|
||||
return {"paths": paths, "trace": trace, "compare_bad": bad, "expect": exp,
|
||||
"replay_bad": {"wrong": 0, "missing": dropped, "extra": 9999}}
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
|
||||
ap.add_argument("outdir")
|
||||
ap.add_argument("--seed", type=int, default=1)
|
||||
ap.add_argument("--calls", type=int, default=12)
|
||||
a = ap.parse_args(argv)
|
||||
r = build_all(a.outdir, a.calls, a.seed)
|
||||
for k, p in r["paths"].items():
|
||||
print(f"{k:16s} {p}")
|
||||
print(f"injected divergences: {sum(1 for e in r['expect'] if e['diverges_exact'])} "
|
||||
f"(tolerant: {sum(1 for e in r['expect'] if e['diverges_tol'])})")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
273
verify/harness/compare/oracle_parsers.py
Normal file
273
verify/harness/compare/oracle_parsers.py
Normal file
|
|
@ -0,0 +1,273 @@
|
|||
#!/usr/bin/env python3
|
||||
"""oracle_parsers.py -- run the proven Python data-file parsers (verify/parsers/)
|
||||
over a list of files and emit the CANONICAL JSON form (TRACE_FORMAT.md section 7)
|
||||
that a C++ reimplementation's test dump must reproduce byte-for-byte.
|
||||
|
||||
usage:
|
||||
oracle_parsers.py --root EXTRACT_DIR [--out-dir DIR] [--jsonl OUT] [--kind KIND] FILE...
|
||||
oracle_parsers.py --root EXTRACT_DIR --all [--out-dir DIR] [--jsonl OUT]
|
||||
|
||||
--root the .gob extract root; file kinds are chosen from the path relative
|
||||
to it exactly as verify.py does (verify.kind_of)
|
||||
--all walk --root and take every file with a known data kind
|
||||
--out-dir write <out-dir>/<rel>.json per file (canonical text + "\\n")
|
||||
--jsonl write one TRACE_FORMAT record per file (hook "parse:<kind>",
|
||||
call_id = index in the sorted file list, ret = {"t":"json","v":...})
|
||||
-- the golden log for `tracecmp.py --replay IMPL.jsonl OUT.jsonl`
|
||||
--kind force a kind for every FILE (brace, effect, csv, manifest, rows, kv)
|
||||
With neither --out-dir nor --jsonl, the canonical text of each file goes to stdout.
|
||||
|
||||
Canonical form (what a C++ dump must produce):
|
||||
* JSON object keys sorted by code point; no whitespace; arrays in parse order
|
||||
* strings byte-preserving: the raw cp1252 byte b -> code point b; then escaped
|
||||
as json.dumps(ensure_ascii=True) does: \\" \\\\ \\n \\r \\t \\b \\f, every other
|
||||
unit < 0x20 or >= 0x7f as \\u00xx (lowercase)
|
||||
* ints decimal; bools true/false; None -> null; tuples -> arrays;
|
||||
Manifest -> {"deleted":[...],"entries":[[id,name],...]}
|
||||
* floats: round to float32, print %.9g, append ".0" if no '.', 'e' or 'n';
|
||||
non-finite -> "nan" / "inf" / "-inf" (strings)
|
||||
* text ends with one "\\n"
|
||||
* a file the parser rejects canonicalizes to {"_error":true}
|
||||
|
||||
Result shapes are the parsers' own (see their docstrings): mars_data dict with
|
||||
repeated keys -> lists, "_items" for bare items; effect_txt ordered [key, value]
|
||||
pairs; flat_kv dict / rows; manifest as above; csv list of rows.
|
||||
|
||||
Stdlib only.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import dataclasses
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
PARSERS = os.path.normpath(os.path.join(HERE, "..", "..", "parsers"))
|
||||
sys.path.insert(0, PARSERS)
|
||||
import effect_txt # noqa: E402
|
||||
import flat_kv # noqa: E402
|
||||
import manifest # noqa: E402
|
||||
import mars_data # noqa: E402
|
||||
import verify as _verify # noqa: E402 (kind_of; main is __main__-guarded)
|
||||
|
||||
kind_of = _verify.kind_of
|
||||
KINDS = ("brace", "effect", "csv", "manifest", "rows", "kv")
|
||||
SKIP_KINDS = ("hlsl", "prose", "other")
|
||||
|
||||
|
||||
# --- canonical form --------------------------------------------------------------------
|
||||
|
||||
def canon_float(x: float) -> str:
|
||||
if math.isnan(x):
|
||||
return '"nan"'
|
||||
if math.isinf(x):
|
||||
return '"inf"' if x > 0 else '"-inf"'
|
||||
x = struct.unpack("<f", struct.pack("<f", x))[0]
|
||||
s = "%.9g" % x
|
||||
if not any(c in s for c in ".en"):
|
||||
s += ".0"
|
||||
return s
|
||||
|
||||
|
||||
def canon_str(s: str) -> str:
|
||||
"""Byte-preserving: re-encode the parser's cp1252 text so each byte is one
|
||||
code point, then escape like json.dumps(ensure_ascii=True)."""
|
||||
try:
|
||||
b = s.encode("cp1252")
|
||||
except UnicodeEncodeError:
|
||||
b = s.encode("latin-1", "replace")
|
||||
return json.dumps(b.decode("latin-1"), ensure_ascii=True)
|
||||
|
||||
|
||||
def canonical(obj):
|
||||
"""Normalize a parser result to plain JSON-able Python (floats stay float,
|
||||
strings become byte-mapped, tuples -> lists, dataclasses -> dicts)."""
|
||||
if dataclasses.is_dataclass(obj) and not isinstance(obj, type):
|
||||
if isinstance(obj, manifest.Manifest):
|
||||
return {"entries": [[i, canonical(n)] for i, n in obj.entries],
|
||||
"deleted": list(obj.deleted)}
|
||||
return canonical(dataclasses.asdict(obj))
|
||||
if isinstance(obj, dict):
|
||||
return {str(k): canonical(v) for k, v in obj.items()}
|
||||
if isinstance(obj, (list, tuple)):
|
||||
return [canonical(v) for v in obj]
|
||||
if isinstance(obj, bool) or obj is None or isinstance(obj, int):
|
||||
return obj
|
||||
if isinstance(obj, float):
|
||||
if math.isnan(obj):
|
||||
return "nan"
|
||||
if math.isinf(obj):
|
||||
return "inf" if obj > 0 else "-inf"
|
||||
return struct.unpack("<f", struct.pack("<f", obj))[0]
|
||||
if isinstance(obj, str):
|
||||
try:
|
||||
return obj.encode("cp1252").decode("latin-1")
|
||||
except UnicodeEncodeError:
|
||||
return obj.encode("latin-1", "replace").decode("latin-1")
|
||||
raise TypeError(f"cannot canonicalize {type(obj).__name__}")
|
||||
|
||||
|
||||
def dumps(obj) -> str:
|
||||
"""Canonical text (no trailing newline) of an already-canonical() object."""
|
||||
out: list[str] = []
|
||||
_emit(obj, out)
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def _emit(x, out: list[str]) -> None:
|
||||
if x is None:
|
||||
out.append("null")
|
||||
elif isinstance(x, bool):
|
||||
out.append("true" if x else "false")
|
||||
elif isinstance(x, int):
|
||||
out.append("%d" % x)
|
||||
elif isinstance(x, float):
|
||||
out.append(canon_float(x))
|
||||
elif isinstance(x, str):
|
||||
out.append(json.dumps(x, ensure_ascii=True))
|
||||
elif isinstance(x, list):
|
||||
out.append("[")
|
||||
for i, e in enumerate(x):
|
||||
if i:
|
||||
out.append(",")
|
||||
_emit(e, out)
|
||||
out.append("]")
|
||||
elif isinstance(x, dict):
|
||||
out.append("{")
|
||||
for i, k in enumerate(sorted(x)):
|
||||
if i:
|
||||
out.append(",")
|
||||
out.append(json.dumps(k, ensure_ascii=True))
|
||||
out.append(":")
|
||||
_emit(x[k], out)
|
||||
out.append("}")
|
||||
else:
|
||||
raise TypeError(type(x).__name__)
|
||||
|
||||
|
||||
# --- parsing dispatch ---------------------------------------------------------------------
|
||||
|
||||
def parse_kind(path: str, kind: str) -> tuple[object, list[str]]:
|
||||
"""-> (result, notes). Raises on parse failure."""
|
||||
notes: list[str] = []
|
||||
base = kind.split(":")[0]
|
||||
if base == "brace":
|
||||
w: list[str] = []
|
||||
obj = mars_data.parse_file(path, warnings=w)
|
||||
notes += w
|
||||
elif base == "effect":
|
||||
obj = effect_txt.parse_file(path)
|
||||
elif base == "csv":
|
||||
obj = manifest.parse_csv_file(path)
|
||||
elif base == "manifest":
|
||||
obj = manifest.parse_manifest_file(path)
|
||||
if obj.problems:
|
||||
raise ValueError("; ".join(obj.problems))
|
||||
elif base == "rows":
|
||||
obj = flat_kv.parse_rows_file(path)
|
||||
elif base == "kv":
|
||||
txt = manifest.read_text(path)
|
||||
obj = flat_kv.parse_kv(txt)
|
||||
d = flat_kv.duplicates(txt)
|
||||
if d:
|
||||
notes.append(f"duplicate keys {d}")
|
||||
else:
|
||||
raise ValueError(f"no parser for kind {kind!r}")
|
||||
return obj, notes
|
||||
|
||||
|
||||
def oracle_file(path: str, kind: str) -> tuple[dict | list, list[str], str | None]:
|
||||
"""-> (canonical object, notes, error). Never raises for parse errors."""
|
||||
try:
|
||||
obj, notes = parse_kind(path, kind)
|
||||
except Exception as e: # noqa: BLE001
|
||||
return {"_error": True}, [], f"{type(e).__name__}: {e}"
|
||||
return canonical(obj), notes, None
|
||||
|
||||
|
||||
def collect(root: str, files: list[str], all_files: bool, forced_kind: str | None) -> list[tuple[str, str, str]]:
|
||||
"""-> sorted [(abs path, rel, kind)] with data kinds only."""
|
||||
items = []
|
||||
if all_files:
|
||||
for dp, _, fn in os.walk(root):
|
||||
for f in fn:
|
||||
items.append(os.path.join(dp, f))
|
||||
items += files
|
||||
out = []
|
||||
for p in items:
|
||||
p = os.path.abspath(p)
|
||||
rel = os.path.relpath(p, root).replace(os.sep, "/") if root else os.path.basename(p)
|
||||
k = forced_kind or kind_of(rel)
|
||||
if k.split(":")[0] in SKIP_KINDS:
|
||||
continue
|
||||
out.append((p, rel, k))
|
||||
out.sort(key=lambda t: t[1])
|
||||
return out
|
||||
|
||||
|
||||
def jsonl_record(idx: int, rel: str, kind: str, canon, notes: list[str], err: str | None) -> str:
|
||||
"""One TRACE_FORMAT record (json.dumps is spec-conformant for our values)."""
|
||||
rec = {"ts": idx, "hook": "parse:" + kind.split(":")[0], "mode": "trace", "call_id": idx, "thread": 0,
|
||||
"args": [{"t": "str", "v": rel.encode("cp1252", "replace").decode("latin-1"), "n": "path"}],
|
||||
"ret": {"t": "json", "v": canon}, "side": {}}
|
||||
if err:
|
||||
rec["err"] = err
|
||||
if notes:
|
||||
rec["note"] = "; ".join(notes)
|
||||
return json.dumps(rec, ensure_ascii=True, separators=(",", ":"), allow_nan=False) + "\n"
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0], formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("files", nargs="*", metavar="FILE")
|
||||
ap.add_argument("--root", default=None, help=".gob extract root (kinds from relative path)")
|
||||
ap.add_argument("--all", action="store_true", help="walk --root for every data file")
|
||||
ap.add_argument("--out-dir")
|
||||
ap.add_argument("--jsonl")
|
||||
ap.add_argument("--kind", choices=KINDS)
|
||||
a = ap.parse_args(argv)
|
||||
if a.all and not a.root:
|
||||
ap.error("--all needs --root")
|
||||
if not a.all and not a.files:
|
||||
ap.error("no FILE given (or use --all)")
|
||||
root = os.path.abspath(a.root) if a.root else None
|
||||
items = collect(root, a.files, a.all, a.kind)
|
||||
if not items:
|
||||
print("error: no data files selected", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
jf = None
|
||||
if a.jsonl:
|
||||
os.makedirs(os.path.dirname(os.path.abspath(a.jsonl)), exist_ok=True)
|
||||
jf = open(a.jsonl, "w", encoding="utf-8", newline="\n")
|
||||
jf.write(json.dumps({"meta": {"format": 1, "build": "oracle_parsers", "started": "", "inline_max": 0,
|
||||
"hooks": {}, "root": root or ""}}, separators=(",", ":")) + "\n")
|
||||
errors = 0
|
||||
for idx, (p, rel, k) in enumerate(items):
|
||||
canon, notes, err = oracle_file(p, k)
|
||||
if err:
|
||||
errors += 1
|
||||
print(f"error: {rel}: {err}", file=sys.stderr)
|
||||
text = dumps(canon) + "\n"
|
||||
if a.out_dir:
|
||||
op = os.path.join(a.out_dir, rel + ".json")
|
||||
os.makedirs(os.path.dirname(op), exist_ok=True)
|
||||
with open(op, "w", encoding="utf-8", newline="\n") as f:
|
||||
f.write(text)
|
||||
if jf:
|
||||
jf.write(jsonl_record(idx, rel, k, canon, notes, err))
|
||||
if not a.out_dir and not jf:
|
||||
sys.stdout.write(text)
|
||||
if jf:
|
||||
jf.close()
|
||||
print(f"{len(items)} file(s), {errors} parse error(s)"
|
||||
+ (f" -> {a.out_dir}" if a.out_dir else "") + (f" -> {a.jsonl}" if a.jsonl else ""), file=sys.stderr)
|
||||
return 0 if not errors else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
497
verify/harness/compare/test_tracecmp.py
Normal file
497
verify/harness/compare/test_tracecmp.py
Normal file
|
|
@ -0,0 +1,497 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Tests for tracecmp.py / mkfixture.py / oracle_parsers.py (synthetic data only).
|
||||
|
||||
/usr/bin/python3 -m unittest -v test_tracecmp
|
||||
/usr/bin/python3 test_tracecmp.py
|
||||
|
||||
Covers: the reference emitter round-trips through json; record/tv validation and
|
||||
exit code 2; exact and tolerant (abs/rel/ulp, CLI + meta policy, per-hook) diffs;
|
||||
set/unordered/ptr/bytes/json rules; replay (missing/extra/wrong); --hook,
|
||||
--json-out; the oracle bridge's canonical form and a bridge->replay round trip.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from contextlib import redirect_stderr, redirect_stdout
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import mkfixture as mk # noqa: E402
|
||||
import oracle_parsers as op # noqa: E402
|
||||
import tracecmp as tc # noqa: E402
|
||||
|
||||
TMP = tempfile.mkdtemp(prefix="tracecmp-test-")
|
||||
FX = mk.build_all(os.path.join(TMP, "fx"), n=12, seed=7)
|
||||
P = FX["paths"]
|
||||
|
||||
|
||||
def run(*argv) -> tuple[int, dict, str]:
|
||||
"""tracecmp.main with --json-out; -> (exit, report, stdout)."""
|
||||
out = os.path.join(TMP, "rep%d.json" % (abs(hash(argv)) % 10**8))
|
||||
buf = io.StringIO()
|
||||
with redirect_stdout(buf):
|
||||
rc = tc.main(list(argv) + ["--json-out", out])
|
||||
with open(out) as f:
|
||||
return rc, json.load(f), buf.getvalue()
|
||||
|
||||
|
||||
def write_log(name: str, recs: list[dict], meta: dict | None = None) -> str:
|
||||
p = os.path.join(TMP, name)
|
||||
mk.write_log(p, meta if meta is not None else mk.meta(), recs)
|
||||
return p
|
||||
|
||||
|
||||
def rec(cid: int, ret, ours_ret=None, side=None, ours_side=None, hook="H", **kw) -> dict:
|
||||
r = {"ts": cid, "hook": hook, "mode": "compare", "call_id": cid, "thread": 1, "args": [],
|
||||
"ret": ret, "side": side or {}, "ours": {"ret": ret if ours_ret is None else ours_ret,
|
||||
"side": ours_side if ours_side is not None else {k: {"after": v["after"]} for k, v in (side or {}).items()}},
|
||||
"diverged": False, "diff": []}
|
||||
r.update(kw)
|
||||
return r
|
||||
|
||||
|
||||
def diffs_of(*recs, **policy) -> list[dict]:
|
||||
pol = tc.Policy().merged(policy)
|
||||
out = []
|
||||
for r in recs:
|
||||
d, _ = tc.compare_record(r, pol)
|
||||
out.append(d)
|
||||
return out
|
||||
|
||||
|
||||
class EmitterTest(unittest.TestCase):
|
||||
def test_roundtrip_every_fixture_record(self):
|
||||
def f32norm(x):
|
||||
"""%.9g round-trips the float32, not the widened double: normalize."""
|
||||
if isinstance(x, dict):
|
||||
if x.get("t") == "f32" and isinstance(x.get("v"), float):
|
||||
return {**x, "v": mk.round_f32(x["v"])}
|
||||
return {k: f32norm(v) for k, v in x.items()}
|
||||
if isinstance(x, list):
|
||||
return [f32norm(v) for v in x]
|
||||
return x
|
||||
for r in FX["trace"] + FX["compare_bad"]:
|
||||
line = mk.emit_record(r)
|
||||
self.assertTrue(line.endswith("}\n"))
|
||||
self.assertTrue(line.isascii())
|
||||
back = json.loads(line)
|
||||
self.assertEqual(f32norm(back), f32norm(r))
|
||||
self.assertNotEqual(back, None)
|
||||
|
||||
def test_string_escaping_matches_python_ensure_ascii(self):
|
||||
for s in ['plain', 'Résumé', 'Tab\tKey', 'Quote"d', 'Back\\slash', 'nl\nx', 'cr\rx', '\x01\x7f\xff', '€']:
|
||||
u = s.encode("cp1252").decode("latin-1")
|
||||
self.assertEqual(json.loads(mk.esc(u)), u)
|
||||
self.assertEqual(mk.esc(u), json.dumps(u, ensure_ascii=True))
|
||||
self.assertEqual(mk.esc("\x08\x0c"), '"\\u0008\\u000c"') # long form is also accepted
|
||||
self.assertEqual(json.loads(mk.esc("\x08\x0c")), "\x08\x0c")
|
||||
|
||||
def test_float_and_bigint_forms(self):
|
||||
self.assertEqual(mk.emit_tv(mk.f32(1.5)), '{"t":"f32","v":1.5}')
|
||||
self.assertEqual(mk.emit_tv(mk.f32(float("nan"))), '{"t":"f32","v":"nan"}')
|
||||
self.assertEqual(mk.emit_tv(mk.f32(float("-inf"))), '{"t":"f32","v":"-inf"}')
|
||||
self.assertEqual(mk.emit_tv(mk.u64(2**64 - 1)), '{"t":"u64","v":"18446744073709551615"}')
|
||||
self.assertEqual(mk.emit_tv(mk.null()), '{"t":"null","v":null}')
|
||||
b = mk.by(b"\x00\xff")
|
||||
self.assertEqual(json.loads(mk.emit_tv(b)), b)
|
||||
self.assertEqual(b["hex"], "00ff")
|
||||
self.assertEqual(mk.by(b"")["sha256"], "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855")
|
||||
big = mk.by(bytes(range(256)) * 2)
|
||||
self.assertNotIn("hex", big)
|
||||
self.assertEqual(len(big["head"]), 64)
|
||||
|
||||
|
||||
class ValidationTest(unittest.TestCase):
|
||||
def test_invalid_log_exit_2(self):
|
||||
rc, rep, _ = run(P["invalid"])
|
||||
self.assertEqual(rc, 2)
|
||||
self.assertEqual(rep["totals"]["invalid_records"], 7)
|
||||
self.assertEqual(rep["totals"]["calls"], 2)
|
||||
errs = " | ".join(e["error"] for e in rep["invalid"])
|
||||
for needle in ("not JSON", "missing required field", "i32.v must be", "duplicate call_id 900",
|
||||
"needs ours", "sha256", "u8.v=300 out of range"):
|
||||
self.assertIn(needle, errs)
|
||||
self.assertTrue(any("unknown field 'bogus'" in w for w in rep["warnings"]))
|
||||
|
||||
def test_skip_invalid(self):
|
||||
rc, rep, _ = run(P["invalid"], "--skip-invalid")
|
||||
self.assertEqual(rc, 0)
|
||||
|
||||
def test_meta_format_version(self):
|
||||
p = write_log("badmeta.jsonl", [], meta={"format": 2})
|
||||
rc, rep, _ = run(p)
|
||||
self.assertEqual(rc, 2)
|
||||
self.assertIn("unsupported format", rep["invalid"][0]["error"])
|
||||
|
||||
def test_tv_validation_details(self):
|
||||
def errs(tv):
|
||||
e = []
|
||||
tc.validate_tv(tv, "x", e)
|
||||
return e
|
||||
self.assertEqual(errs(mk.i32(5)), [])
|
||||
self.assertTrue(errs({"t": "i32"}))
|
||||
self.assertTrue(errs({"t": "nope", "v": 1}))
|
||||
self.assertTrue(errs({"t": "str", "v": "€"})) # code point > U+00FF
|
||||
self.assertEqual(errs({"t": "wstr", "v": "€"}), [])
|
||||
self.assertTrue(errs({"t": "ptr", "v": "a36fd0"}))
|
||||
self.assertTrue(errs({"t": "bytes", "n": 2, "sha256": "a" * 64, "hex": "00"})) # hex len != 2n
|
||||
self.assertTrue(errs({"t": "bool", "v": 1}))
|
||||
self.assertEqual(errs({"t": "i64", "v": "-9223372036854775808"}), [])
|
||||
self.assertTrue(errs({"t": "i64", "v": "9223372036854775808"}))
|
||||
self.assertTrue(errs({"t": "f32", "v": "huge"}))
|
||||
self.assertEqual(errs({"t": "f32", "v": "inf"}), [])
|
||||
self.assertTrue(errs({"t": "list", "v": [{"t": "i32", "v": "x"}]}))
|
||||
self.assertTrue(errs({"t": "struct", "v": [1]}))
|
||||
|
||||
def test_replace_and_trace_records_not_compared(self):
|
||||
r = rec(1, mk.i32(1), mk.i32(2))
|
||||
r["mode"] = "replace"
|
||||
del r["ours"], r["diverged"], r["diff"]
|
||||
t = rec(2, mk.i32(1), mk.i32(2))
|
||||
t["mode"] = "trace"
|
||||
t["err"] = "capture failed"
|
||||
rc, rep, _ = run(write_log("modes.jsonl", [r, t]))
|
||||
self.assertEqual(rc, 0)
|
||||
self.assertEqual(rep["hooks"]["H"]["modes"], {"replace": 1, "trace": 1})
|
||||
self.assertEqual(rep["hooks"]["H"]["errors"], 1)
|
||||
self.assertEqual(rep["hooks"]["H"]["compared"], 0)
|
||||
|
||||
|
||||
class CompareTest(unittest.TestCase):
|
||||
def test_clean_log_is_clean(self):
|
||||
rc, rep, out = run(P["compare_clean"])
|
||||
self.assertEqual(rc, 0)
|
||||
self.assertEqual(rep["totals"], {"calls": 12, "compared": 12, "diverged": 0, "invalid_records": 0})
|
||||
self.assertIn("| CfgVar_RegisterKey | 4 |", out)
|
||||
|
||||
def test_injected_divergences_exact(self):
|
||||
rc, rep, out = run(P["compare_bad"], "--first", "20")
|
||||
self.assertEqual(rc, 1)
|
||||
got = {cid for s in rep["hooks"].values() for cid in s["diverged_call_ids"]}
|
||||
want = {e["call_id"] for e in FX["expect"] if e["diverges_exact"]}
|
||||
self.assertEqual(got, want)
|
||||
first = {d["call_id"]: d["diff"] for s in rep["hooks"].values() for d in s["diffs"]}
|
||||
for e in FX["expect"]:
|
||||
if not e["diverges_exact"]:
|
||||
continue
|
||||
d0 = first[e["call_id"]][0]
|
||||
self.assertEqual(d0["why"], e["why"], (e, d0))
|
||||
self.assertTrue(d0["path"].startswith(e["path"]), (e, d0))
|
||||
lie = [e for e in FX["expect"] if e.get("warn")][0]
|
||||
self.assertTrue(any(f"call {lie['call_id']}: shim said diverged=True" in w for w in rep["warnings"]))
|
||||
self.assertEqual(len(rep["warnings"]), 1) # truthful verdicts everywhere else
|
||||
|
||||
def test_injected_divergences_tolerant(self):
|
||||
rc, rep, _ = run(P["compare_bad"], "--tolerance", "1e-3")
|
||||
self.assertEqual(rc, 1)
|
||||
got = {cid for s in rep["hooks"].values() for cid in s["diverged_call_ids"]}
|
||||
want = {e["call_id"] for e in FX["expect"] if e["diverges_tol"]}
|
||||
self.assertEqual(got, want)
|
||||
|
||||
def test_per_hook_tolerance_and_meta_policy(self):
|
||||
nudged = [e["call_id"] for e in FX["expect"] if e["diverges_exact"] and not e["diverges_tol"]][0]
|
||||
# per-hook CLI on the wrong hook: still diverges
|
||||
rc, rep, _ = run(P["compare_bad"], "--tolerance", "Manifest_Load=1e-3", "--hook", "Mars::ParseBlock")
|
||||
self.assertIn(nudged, rep["hooks"]["Mars::ParseBlock"]["diverged_call_ids"])
|
||||
# per-hook CLI on the right hook: passes
|
||||
rc, rep, _ = run(P["compare_bad"], "--tolerance", "Mars::ParseBlock=1e-3", "--hook", "Mars::ParseBlock")
|
||||
self.assertNotIn(nudged, rep["hooks"]["Mars::ParseBlock"]["diverged_call_ids"])
|
||||
# meta policy in the log itself, no CLI
|
||||
bad = FX["compare_bad"]
|
||||
p = write_log("metapol.jsonl", bad, meta=mk.meta(**{"Mars::ParseBlock": {"ftol": 1e-3}}))
|
||||
rc, rep, _ = run(p, "--hook", "Mars::ParseBlock")
|
||||
self.assertNotIn(nudged, rep["hooks"]["Mars::ParseBlock"]["diverged_call_ids"])
|
||||
# CLI default overrides meta
|
||||
rc, rep, _ = run(p, "--hook", "Mars::ParseBlock", "--tolerance", "0")
|
||||
self.assertIn(nudged, rep["hooks"]["Mars::ParseBlock"]["diverged_call_ids"])
|
||||
|
||||
def test_float_kinds(self):
|
||||
a, b = 1000.0, 1000.001
|
||||
self.assertFalse(tc.floats_equal(a, b, tc.Policy()))
|
||||
self.assertTrue(tc.floats_equal(a, b, tc.Policy(ftol=1e-2)))
|
||||
self.assertTrue(tc.floats_equal(a, b, tc.Policy(ftol=1e-5, ftol_kind="rel")))
|
||||
self.assertFalse(tc.floats_equal(a, b, tc.Policy(ftol=1e-7, ftol_kind="rel")))
|
||||
x = mk.round_f32(1.0)
|
||||
y = mk.round_f32(1.0 + 2 ** -23) # 1 ulp of f32
|
||||
self.assertTrue(tc.floats_equal(x, y, tc.Policy(ftol=1, ftol_kind="ulp"), "f32"))
|
||||
self.assertFalse(tc.floats_equal(x, y, tc.Policy(ftol=0.5, ftol_kind="ulp"), "f32"))
|
||||
self.assertFalse(tc.floats_equal(x, y, tc.Policy(ftol=1, ftol_kind="ulp"), "f64"))
|
||||
nan, inf = float("nan"), float("inf")
|
||||
self.assertTrue(tc.floats_equal(nan, nan, tc.Policy(ftol=1e9)))
|
||||
self.assertFalse(tc.floats_equal(nan, 0.0, tc.Policy(ftol=1e9)))
|
||||
self.assertFalse(tc.floats_equal(inf, -inf, tc.Policy(ftol=1e9)))
|
||||
self.assertTrue(tc.floats_equal(inf, inf, tc.Policy()))
|
||||
# via records: f32 vs f64 is a type divergence; "nan" strings compare equal
|
||||
d = diffs_of(rec(1, {"t": "f32", "v": 1.0}, {"t": "f64", "v": 1.0}),
|
||||
rec(2, {"t": "f32", "v": "nan"}, {"t": "f32", "v": "nan"}))
|
||||
self.assertEqual([x["why"] for x in d[0]], ["type"])
|
||||
self.assertEqual(d[1], [])
|
||||
|
||||
def test_ints_strings_enums(self):
|
||||
d = diffs_of(rec(1, mk.u64(2**60), {"t": "u64", "v": str(2**60)}),
|
||||
rec(2, mk.s("Résumé"), mk.s("Resume")),
|
||||
rec(3, mk.enum(1, "A"), mk.enum(1, "B")),
|
||||
rec(4, mk.enum(1), mk.enum(2)),
|
||||
rec(5, mk.boolean(True), mk.i32(1)))
|
||||
self.assertEqual(d[0], [])
|
||||
self.assertEqual(d[1][0]["why"], "exact")
|
||||
self.assertEqual(d[2], []) # symbolic name is informational
|
||||
self.assertEqual(d[3][0]["why"], "exact")
|
||||
self.assertEqual(d[4][0]["why"], "type")
|
||||
|
||||
def test_ptr_policy(self):
|
||||
r = rec(1, mk.ptr(0x1000), mk.ptr(0x2000))
|
||||
self.assertEqual(diffs_of(r)[0], [])
|
||||
self.assertEqual(diffs_of(r, ptr="exact")[0][0]["why"], "exact")
|
||||
self.assertEqual(diffs_of(rec(2, mk.ptr(0), mk.ptr(0x2000)))[0][0]["why"], "exact")
|
||||
rc, rep, _ = run(write_log("ptr.jsonl", [r]), "--ptr", "exact")
|
||||
self.assertEqual(rc, 1)
|
||||
|
||||
def test_set_and_unordered(self):
|
||||
a = mk.sset([mk.i32(1), mk.i32(1), mk.i32(2)])
|
||||
self.assertEqual(diffs_of(rec(1, a, mk.sset([mk.i32(2), mk.i32(1), mk.i32(1)])))[0], [])
|
||||
d = diffs_of(rec(2, a, mk.sset([mk.i32(1), mk.i32(2), mk.i32(2)])))[0]
|
||||
self.assertEqual(d[0]["why"], "exact") # multiset: multiplicity matters
|
||||
d = diffs_of(rec(3, a, mk.sset([mk.i32(1), mk.i32(2)])))[0]
|
||||
self.assertEqual(d[0]["why"], "len")
|
||||
l1, l2 = mk.lst([mk.i32(1), mk.i32(2)]), mk.lst([mk.i32(2), mk.i32(1)])
|
||||
self.assertEqual(diffs_of(rec(4, l1, l2))[0][0]["why"], "exact")
|
||||
self.assertEqual(diffs_of(rec(4, l1, l2), unordered=["ret"])[0], [])
|
||||
rc, rep, _ = run(write_log("unord.jsonl", [rec(4, l1, l2, hook="Hk")]), "--unordered", "Hk=ret")
|
||||
self.assertEqual(rc, 0)
|
||||
rc, rep, _ = run(write_log("unord2.jsonl", [rec(4, l1, l2, hook="Hk")]), "--unordered", "Other=ret")
|
||||
self.assertEqual(rc, 1)
|
||||
|
||||
def test_struct_and_side(self):
|
||||
s1 = mk.struct_({"a": mk.i32(1), "b": mk.s("x")})
|
||||
s2 = mk.struct_({"a": mk.i32(1), "c": mk.s("x")})
|
||||
d = diffs_of(rec(1, s1, s2))[0]
|
||||
self.assertEqual([(x["path"], x["why"]) for x in d], [("ret.v.b", "missing"), ("ret.v.c", "extra")])
|
||||
side = {"r1": {"before": mk.i32(0), "after": mk.i32(1)}, "r2": {"after": mk.i32(5)}}
|
||||
ours = {"r1": {"after": mk.i32(1)}, "r3": {"after": mk.i32(5)}}
|
||||
d = diffs_of(rec(2, None, None, side=side, ours_side=ours))[0]
|
||||
self.assertEqual([(x["path"], x["why"]) for x in d], [("side.r2.after", "missing"), ("side.r3.after", "extra")])
|
||||
# 'before' mismatch is a warning, not a divergence
|
||||
ours = {"r1": {"before": mk.i32(9), "after": mk.i32(1)}, "r2": {"after": mk.i32(5)}}
|
||||
dd, warns = tc.compare_record(rec(3, None, None, side=side, ours_side=ours), tc.Policy())
|
||||
self.assertEqual(dd, [])
|
||||
self.assertTrue(any("snapshot" in w for w in warns))
|
||||
# void vs value
|
||||
self.assertEqual(diffs_of(rec(4, None, mk.i32(1)))[0][0]["why"], "extra")
|
||||
|
||||
def test_bytes(self):
|
||||
a, b = mk.by(b"abcdef"), mk.by(b"abcXef")
|
||||
d = diffs_of(rec(1, a, b))[0]
|
||||
self.assertEqual(d[0]["why"], "hash")
|
||||
self.assertEqual(d[0]["first_diff_offset"], 3)
|
||||
self.assertEqual(diffs_of(rec(2, a, mk.by(b"abcdefg")))[0][0]["why"], "len")
|
||||
big1, big2 = mk.by(bytes(300)), mk.by(bytes(299) + b"\x01")
|
||||
d = diffs_of(rec(3, big1, big2))[0]
|
||||
self.assertEqual(d[0]["why"], "hash")
|
||||
self.assertNotIn("first_diff_offset", d[0])
|
||||
self.assertEqual(diffs_of(rec(4, mk.by(b""), mk.by(b"")))[0], [])
|
||||
|
||||
def test_json_values(self):
|
||||
j = lambda v: mk.jsonv(v) # noqa: E731
|
||||
self.assertEqual(diffs_of(rec(1, j({"a": [1, 2.5, "x", True, None]}), j({"a": [1, 2.5, "x", True, None]})))[0], [])
|
||||
d = diffs_of(rec(2, j({"a": 1}), j({"a": 1.0})))[0]
|
||||
self.assertEqual(d[0]["why"], "type")
|
||||
self.assertEqual(diffs_of(rec(2, j({"a": 1}), j({"a": 1.0})), numeric=True)[0], [])
|
||||
d = diffs_of(rec(3, j({"a": {"b": [1, 2]}}), j({"a": {"b": [1]}, "c": 0})))[0]
|
||||
self.assertEqual([(x["path"], x["why"]) for x in d], [("ret.c", "extra"), ("ret.a.b", "len")])
|
||||
d = diffs_of(rec(4, j({"f": 1.0}), j({"f": 1.0 + 1e-7})))[0]
|
||||
self.assertEqual(d[0]["why"], "exact")
|
||||
self.assertEqual(diffs_of(rec(4, j({"f": 1.0}), j({"f": 1.0 + 1e-7})), ftol=1e-6)[0], [])
|
||||
self.assertEqual(diffs_of(rec(5, j({"x": True}), j({"x": 1})))[0][0]["why"], "exact")
|
||||
d = diffs_of(rec(6, j([3, 1, 2]), j([1, 2, 3])), unordered=["ret"])[0]
|
||||
self.assertEqual(d, [])
|
||||
|
||||
def test_hook_filter_and_json_out(self):
|
||||
rc, rep, out = run(P["compare_bad"], "--hook", "Manifest_Load")
|
||||
self.assertEqual(list(rep["hooks"]), ["Manifest_Load"])
|
||||
self.assertEqual(rep["totals"]["calls"], 4)
|
||||
self.assertEqual(rep["kind"], "report")
|
||||
self.assertEqual(rep["format"], 1)
|
||||
rc, rep, out = run(P["compare_bad"], "--hook", "NoSuchHook")
|
||||
self.assertEqual(rc, 0)
|
||||
self.assertEqual(rep["totals"]["calls"], 0)
|
||||
|
||||
def test_first_n(self):
|
||||
rc, rep, out = run(P["compare_bad"], "--first", "1")
|
||||
for s in rep["hooks"].values():
|
||||
self.assertLessEqual(len(s["diffs"]), 1)
|
||||
self.assertIn("other divergent call_ids", out)
|
||||
|
||||
def test_multiple_logs(self):
|
||||
rc, rep, _ = run(P["compare_clean"], P["compare_bad"])
|
||||
self.assertEqual(rep["totals"]["calls"], 24)
|
||||
self.assertEqual(rc, 1)
|
||||
|
||||
|
||||
class ReplayTest(unittest.TestCase):
|
||||
def test_replay_ok(self):
|
||||
rc, rep, _ = run(P["trace_clean"], "--replay", P["replay_impl_ok"])
|
||||
self.assertEqual(rc, 0)
|
||||
self.assertEqual(rep["kind"], "replay")
|
||||
self.assertEqual(rep["totals"]["compared"], 12)
|
||||
|
||||
def test_replay_bad(self):
|
||||
rc, rep, out = run(P["trace_clean"], "--replay", P["replay_impl_bad"])
|
||||
self.assertEqual(rc, 1)
|
||||
rb = FX["replay_bad"]
|
||||
got = {cid for s in rep["hooks"].values() for cid in s["diverged_call_ids"]}
|
||||
self.assertEqual(got, {rb["wrong"], rb["missing"]})
|
||||
whys = {d["call_id"]: d["diff"][0]["why"] for s in rep["hooks"].values() for d in s["diffs"]}
|
||||
self.assertEqual(whys[rb["wrong"]], "exact")
|
||||
self.assertEqual(whys[rb["missing"]], "missing")
|
||||
self.assertTrue(any(str(rb["extra"]) in w for w in rep["warnings"]))
|
||||
|
||||
def test_replay_side_shapes_and_hook_mismatch(self):
|
||||
g = write_log("g.jsonl", [{"ts": 0, "hook": "H", "mode": "trace", "call_id": 5, "thread": 0, "args": [],
|
||||
"ret": mk.i32(1), "side": {"r": {"after": mk.i32(2)}}}])
|
||||
p = os.path.join(TMP, "impl_shapes.jsonl")
|
||||
with open(p, "w") as f:
|
||||
f.write(json.dumps({"call_id": 5, "hook": "Other", "ret": mk.i32(1), "side": {"r": {"after": mk.i32(2)}}}) + "\n")
|
||||
rc, rep, _ = run(g, "--replay", p)
|
||||
self.assertEqual(rc, 0)
|
||||
self.assertTrue(any("hook 'Other'" in w for w in rep["warnings"]))
|
||||
with open(p, "w") as f:
|
||||
f.write(json.dumps({"call_id": 5, "ret": mk.i32(1), "side": {"r": mk.i32(3)}}) + "\n")
|
||||
f.write("garbage\n")
|
||||
rc, rep, _ = run(g, "--replay", p)
|
||||
self.assertEqual(rc, 2) # invalid impl line wins over the divergence
|
||||
self.assertEqual(rep["totals"]["diverged"], 1)
|
||||
rc, rep, _ = run(g, "--replay", p, "--skip-invalid")
|
||||
self.assertEqual(rc, 1)
|
||||
|
||||
def test_replay_tolerance(self):
|
||||
g = write_log("gt.jsonl", [{"ts": 0, "hook": "H", "mode": "trace", "call_id": 1, "thread": 0, "args": [],
|
||||
"ret": mk.f32(2.0), "side": {}}])
|
||||
p = os.path.join(TMP, "impl_tol.jsonl")
|
||||
with open(p, "w") as f:
|
||||
f.write(json.dumps({"call_id": 1, "ret": mk.f32(2.0 + 1e-6)}) + "\n")
|
||||
self.assertEqual(run(g, "--replay", p)[0], 1)
|
||||
self.assertEqual(run(g, "--replay", p, "--tolerance", "H=1e-5")[0], 0)
|
||||
|
||||
|
||||
class OracleBridgeTest(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls):
|
||||
cls.root = os.path.join(TMP, "gob")
|
||||
files = {
|
||||
"Data/globals.txt": 'FOO 1\nBAR 1.0\nNAME "Résumé"\nCOL "1 2 3"\nNEG -0.0\nBIG 7e+8\nFLAG TRUE // c\n',
|
||||
"Weapons/w.weapon": 'weapon {\n name "Laser" // x\n dam .5\n dam 2\n mount { size small }\n "bare"\n}\n',
|
||||
"Weapons/_weapons.txt": "1 w.weapon\n// DELETED - 2\n3 Other.WEAPON\n",
|
||||
"Weapons/_turrets.txt": 'a 1 "two words" 3.25\nb 2 x 4\n',
|
||||
"Data/x.csv": '# a,b\n1,"x,y"\n\n2,z\n',
|
||||
"Effects/e.effect": 'TXT\nNAME "New Emitter"\nMODIFIER\nBEGIN\n TYPE 1\n K 0.1\nEND\nMODIFIER\nBEGIN\n TYPE 2\nEND\n',
|
||||
"Weapons/broken.weapon": 'weapon { name "x" "unterminated\n',
|
||||
"Weapons/_shipsections.txt": "1 a.shipsection\n1 b.shipsection\n",
|
||||
"shaders/x.fx": "float4 main() {}\n",
|
||||
}
|
||||
for rel, txt in files.items():
|
||||
p = os.path.join(cls.root, rel)
|
||||
os.makedirs(os.path.dirname(p), exist_ok=True)
|
||||
with open(p, "wb") as f:
|
||||
f.write(txt.encode("cp1252"))
|
||||
|
||||
def test_canonical_floats_and_strings(self):
|
||||
self.assertEqual(op.canon_float(1.0), "1.0")
|
||||
self.assertEqual(op.canon_float(-0.0), "-0.0")
|
||||
self.assertEqual(op.canon_float(0.1), "0.100000001") # f32-rounded, %.9g
|
||||
self.assertEqual(op.canon_float(7e8), "700000000.0")
|
||||
self.assertEqual(op.canon_float(1e20), "1.00000002e+20")
|
||||
self.assertEqual(op.canon_float(float("nan")), '"nan"')
|
||||
self.assertEqual(op.canon_str("Résumé"), '"R\\u00e9sum\\u00e9"')
|
||||
self.assertEqual(op.canon_str("€"), '"\\u0080"') # cp1252 byte, not U+20AC
|
||||
self.assertEqual(op.canon_str('a"b\\c\n\x7f'), '"a\\"b\\\\c\\n\\u007f"')
|
||||
self.assertEqual(op.dumps(op.canonical({"b": [1, 2.5, (3, "x")], "a": {"z": True, "y": None}})),
|
||||
'{"a":{"y":null,"z":true},"b":[1,2.5,[3,"x"]]}')
|
||||
# no-float objects equal json.dumps canonical settings
|
||||
obj = {"k": ["a", {"n": 1, "m": "é"}]}
|
||||
self.assertEqual(op.dumps(op.canonical(obj)), json.dumps(obj, sort_keys=True, ensure_ascii=True, separators=(",", ":")))
|
||||
|
||||
def test_per_kind_output(self):
|
||||
r = self.root
|
||||
c, notes, err = op.oracle_file(os.path.join(r, "Data/globals.txt"), "kv")
|
||||
self.assertIsNone(err)
|
||||
self.assertEqual(op.dumps(c), '{"BAR":1.0,"BIG":700000000.0,"COL":"1 2 3","FLAG":true,"FOO":1,"NAME":"R\\u00e9sum\\u00e9","NEG":-0.0}')
|
||||
c, notes, err = op.oracle_file(os.path.join(r, "Weapons/w.weapon"), "brace:weapon")
|
||||
self.assertEqual(op.dumps(c), '{"weapon":{"_items":"bare","dam":[0.5,2],"mount":{"size":"small"},"name":"Laser"}}')
|
||||
c, notes, err = op.oracle_file(os.path.join(r, "Weapons/_weapons.txt"), "manifest")
|
||||
self.assertEqual(op.dumps(c), '{"deleted":[2],"entries":[[1,"w.weapon"],[3,"Other.WEAPON"]]}')
|
||||
c, notes, err = op.oracle_file(os.path.join(r, "Weapons/_turrets.txt"), "rows")
|
||||
self.assertEqual(op.dumps(c), '[["a",1,"two words",3.25],["b",2,"x",4]]')
|
||||
c, notes, err = op.oracle_file(os.path.join(r, "Data/x.csv"), "csv")
|
||||
self.assertEqual(op.dumps(c), '[["1","x,y"],["2","z"]]')
|
||||
c, notes, err = op.oracle_file(os.path.join(r, "Effects/e.effect"), "effect")
|
||||
self.assertEqual(op.dumps(c), '[["NAME","New Emitter"],["MODIFIER",[["TYPE",1],["K",0.100000001]]],["MODIFIER",[["TYPE",2]]]]')
|
||||
c, notes, err = op.oracle_file(os.path.join(r, "Weapons/broken.weapon"), "brace:weapon")
|
||||
self.assertEqual(c, {"_error": True})
|
||||
self.assertIn("MarsSyntaxError", err)
|
||||
c, notes, err = op.oracle_file(os.path.join(r, "Weapons/_shipsections.txt"), "manifest")
|
||||
self.assertIn("duplicate id", err)
|
||||
|
||||
def test_cli_and_replay_round_trip(self):
|
||||
r = self.root
|
||||
golden = os.path.join(TMP, "oracle", "golden.jsonl")
|
||||
outdir = os.path.join(TMP, "oracle", "canon")
|
||||
err = io.StringIO()
|
||||
with redirect_stderr(err), redirect_stdout(io.StringIO()):
|
||||
rc = op.main(["--root", r, "--all", "--jsonl", golden, "--out-dir", outdir])
|
||||
self.assertEqual(rc, 1) # two files fail to parse
|
||||
self.assertIn("8 file(s), 2 parse error(s)", err.getvalue())
|
||||
self.assertFalse(os.path.exists(os.path.join(outdir, "shaders/x.fx.json"))) # skipped kind
|
||||
with open(os.path.join(outdir, "Data/globals.txt.json")) as f:
|
||||
self.assertTrue(f.read().endswith("}\n"))
|
||||
# golden validates and self-replays clean
|
||||
rc, rep, _ = run(golden)
|
||||
self.assertEqual(rc, 0)
|
||||
self.assertEqual(rep["hooks"]["parse:brace"]["errors"], 1)
|
||||
self.assertEqual(rep["hooks"]["parse:manifest"]["errors"], 1)
|
||||
rc, rep, _ = run(golden, "--replay", golden)
|
||||
self.assertEqual(rc, 0)
|
||||
self.assertEqual(rep["totals"]["compared"], 8)
|
||||
# an "implementation" that gets one float wrong and one file's shape wrong
|
||||
impl = os.path.join(TMP, "oracle", "impl.jsonl")
|
||||
with open(golden) as f, open(impl, "w") as g:
|
||||
for line in f:
|
||||
d = json.loads(line)
|
||||
if "meta" in d:
|
||||
continue
|
||||
if d["args"][0]["v"] == "Data/globals.txt":
|
||||
d["ret"]["v"]["BAR"] = 1.0000001
|
||||
if d["args"][0]["v"] == "Weapons/w.weapon":
|
||||
d["ret"]["v"]["weapon"]["dam"] = 2
|
||||
g.write(json.dumps({"call_id": d["call_id"], "ret": d["ret"], "side": {}}) + "\n")
|
||||
rc, rep, _ = run(golden, "--replay", impl)
|
||||
self.assertEqual(rc, 1)
|
||||
self.assertEqual(rep["totals"]["diverged"], 2)
|
||||
rc, rep, _ = run(golden, "--replay", impl, "--tolerance", "1e-6")
|
||||
self.assertEqual(rep["totals"]["diverged"], 1)
|
||||
paths = [d["diff"][0]["path"] for s in rep["hooks"].values() for d in s["diffs"]]
|
||||
self.assertEqual(paths, ["ret.weapon.dam"])
|
||||
# stdout mode prints one canonical line per file
|
||||
buf = io.StringIO()
|
||||
with redirect_stdout(buf), redirect_stderr(io.StringIO()):
|
||||
op.main(["--root", r, os.path.join(r, "Data/globals.txt"), "--kind", "kv"])
|
||||
self.assertEqual(buf.getvalue().count("\n"), 1)
|
||||
|
||||
def test_kind_dispatch_matches_verify(self):
|
||||
self.assertEqual(op.kind_of("Weapons/_weapons.txt"), "manifest")
|
||||
self.assertEqual(op.kind_of("Data/globals.txt"), "kv")
|
||||
self.assertEqual(op.kind_of("Weapons/x.weapon"), "brace:weapon")
|
||||
self.assertEqual(op.kind_of("Locale/EN/Desc_x.txt"), "prose")
|
||||
|
||||
|
||||
def tearDownModule():
|
||||
shutil.rmtree(TMP, ignore_errors=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=1)
|
||||
792
verify/harness/compare/tracecmp.py
Normal file
792
verify/harness/compare/tracecmp.py
Normal file
|
|
@ -0,0 +1,792 @@
|
|||
#!/usr/bin/env python3
|
||||
"""tracecmp.py -- validate shim trace/compare logs (TRACE_FORMAT.md) and report
|
||||
divergences; or replay a golden trace against an implementation's output.
|
||||
|
||||
usage:
|
||||
tracecmp.py LOG.jsonl [LOG2.jsonl ...] report per-hook divergences
|
||||
tracecmp.py --replay IMPL.jsonl GOLDEN.jsonl [...] diff GOLDEN (ret/side) vs IMPL
|
||||
records matched by call_id
|
||||
|
||||
flags:
|
||||
--hook NAME only this hook (repeatable)
|
||||
--tolerance SPEC float policy; SPEC = [HOOK=][abs:|rel:|ulp:]NUMBER
|
||||
e.g. 1e-6 | rel:1e-5 | Mars::ParseBlock=ulp:2 (repeatable;
|
||||
a bare NUMBER without HOOK sets the default for every hook)
|
||||
--unordered SPEC treat a list path as a set; SPEC = [HOOK=]PATH (repeatable)
|
||||
--ptr exact|ignore pointer policy for every hook (default: ignore)
|
||||
--numeric inside `json` values, compare int vs float as numbers
|
||||
--first N diffs shown per hook (default 5)
|
||||
--json-out PATH write the full report as JSON (convention:
|
||||
verify/results/compare/<run>.json)
|
||||
--skip-invalid drop unparsable/invalid records with a warning instead of
|
||||
failing the run
|
||||
|
||||
exit: 0 = no divergences; 1 = divergences; 2 = invalid input (or usage).
|
||||
|
||||
Stdlib only.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import struct
|
||||
import sys
|
||||
from dataclasses import dataclass, field, replace as dc_replace
|
||||
from typing import Any
|
||||
|
||||
FORMAT_VERSION = 1
|
||||
MODES = ("trace", "compare", "replace")
|
||||
INT_TYPES = {"i8", "i16", "i32", "i64", "u8", "u16", "u32", "u64"}
|
||||
FLOAT_TYPES = {"f32", "f64"}
|
||||
TV_TYPES = INT_TYPES | FLOAT_TYPES | {"bool", "str", "wstr", "ptr", "enum", "null",
|
||||
"bytes", "list", "set", "struct", "json"}
|
||||
RECORD_KEYS = {"ts", "hook", "mode", "call_id", "thread", "depth", "args", "ret", "side",
|
||||
"ours", "diverged", "diff", "err", "note"}
|
||||
INT_RANGE = {"i8": (-2**7, 2**7 - 1), "i16": (-2**15, 2**15 - 1), "i32": (-2**31, 2**31 - 1),
|
||||
"i64": (-2**63, 2**63 - 1), "u8": (0, 2**8 - 1), "u16": (0, 2**16 - 1),
|
||||
"u32": (0, 2**32 - 1), "u64": (0, 2**64 - 1)}
|
||||
_HEX_RE = re.compile(r"^[0-9a-f]*$")
|
||||
_SHA_RE = re.compile(r"^[0-9a-f]{64}$")
|
||||
_PTR_RE = re.compile(r"^0x[0-9a-fA-F]{1,16}$")
|
||||
NONFINITE = {"nan": math.nan, "inf": math.inf, "-inf": -math.inf}
|
||||
|
||||
|
||||
# --- policy ------------------------------------------------------------------
|
||||
|
||||
@dataclass
|
||||
class Policy:
|
||||
ftol: float = 0.0
|
||||
ftol_kind: str = "abs" # abs | rel | ulp
|
||||
ptr: str = "ignore" # ignore | exact
|
||||
numeric: bool = False # json: int vs float compare as numbers
|
||||
unordered: set = field(default_factory=set)
|
||||
|
||||
def merged(self, d: dict) -> "Policy":
|
||||
p = dc_replace(self, unordered=set(self.unordered))
|
||||
if "ftol" in d:
|
||||
p.ftol = float(d["ftol"])
|
||||
if "ftol_kind" in d:
|
||||
p.ftol_kind = d["ftol_kind"]
|
||||
if "ptr" in d:
|
||||
p.ptr = d["ptr"]
|
||||
if "numeric" in d:
|
||||
p.numeric = bool(d["numeric"])
|
||||
if "unordered" in d:
|
||||
p.unordered |= set(d["unordered"])
|
||||
return p
|
||||
|
||||
|
||||
class PolicyTable:
|
||||
"""defaults < log meta per-hook < CLI default < CLI per-hook."""
|
||||
|
||||
def __init__(self):
|
||||
self.default: dict = {}
|
||||
self.meta: dict[str, dict] = {}
|
||||
self.cli: dict[str, dict] = {}
|
||||
|
||||
def add_meta(self, hooks: dict) -> None:
|
||||
for h, d in (hooks or {}).items():
|
||||
self.meta.setdefault(h, {}).update(d)
|
||||
|
||||
def add_cli(self, hook: str | None, d: dict) -> None:
|
||||
if hook is None:
|
||||
self.default.update(d)
|
||||
else:
|
||||
self.cli.setdefault(hook, {}).update(d)
|
||||
|
||||
def for_hook(self, hook: str) -> Policy:
|
||||
p = Policy().merged(self.meta.get(hook, {}))
|
||||
p = p.merged(self.default)
|
||||
return p.merged(self.cli.get(hook, {}))
|
||||
|
||||
|
||||
def parse_tolerance(spec: str) -> tuple[str | None, dict]:
|
||||
hook = None
|
||||
if "=" in spec:
|
||||
hook, spec = spec.split("=", 1)
|
||||
kind = "abs"
|
||||
if ":" in spec:
|
||||
kind, spec = spec.split(":", 1)
|
||||
if kind not in ("abs", "rel", "ulp"):
|
||||
raise ValueError(f"bad tolerance kind {kind!r}")
|
||||
return hook, {"ftol": float(spec), "ftol_kind": kind}
|
||||
|
||||
|
||||
def parse_unordered(spec: str) -> tuple[str | None, dict]:
|
||||
hook = None
|
||||
if "=" in spec:
|
||||
hook, spec = spec.split("=", 1)
|
||||
return hook, {"unordered": [spec]}
|
||||
|
||||
|
||||
# --- validation ----------------------------------------------------------------
|
||||
|
||||
class Invalid(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def _is_int(x) -> bool:
|
||||
return isinstance(x, int) and not isinstance(x, bool)
|
||||
|
||||
|
||||
def validate_tv(tv: Any, path: str, errs: list[str]) -> None:
|
||||
if not isinstance(tv, dict):
|
||||
errs.append(f"{path}: typed value must be an object, got {type(tv).__name__}")
|
||||
return
|
||||
t = tv.get("t")
|
||||
if t not in TV_TYPES:
|
||||
errs.append(f"{path}: unknown type {t!r}")
|
||||
return
|
||||
if "n" in tv and not isinstance(tv["n"], (str, int)):
|
||||
errs.append(f"{path}: 'n' must be a string (name) or int (bytes length)")
|
||||
if t == "bytes":
|
||||
n, sha = tv.get("n"), tv.get("sha256")
|
||||
if not _is_int(n) or n < 0:
|
||||
errs.append(f"{path}: bytes.n must be int >= 0")
|
||||
if not isinstance(sha, str) or not _SHA_RE.match(sha):
|
||||
errs.append(f"{path}: bytes.sha256 must be 64 lowercase hex chars")
|
||||
if "hex" in tv:
|
||||
h = tv["hex"]
|
||||
if not isinstance(h, str) or not _HEX_RE.match(h) or (_is_int(n) and len(h) != 2 * n):
|
||||
errs.append(f"{path}: bytes.hex must be 2*n lowercase hex chars")
|
||||
if "head" in tv and (not isinstance(tv["head"], str) or not _HEX_RE.match(tv["head"])):
|
||||
errs.append(f"{path}: bytes.head must be lowercase hex")
|
||||
return
|
||||
if "v" not in tv:
|
||||
errs.append(f"{path}: missing 'v'")
|
||||
return
|
||||
v = tv["v"]
|
||||
if t == "bool":
|
||||
if not isinstance(v, bool):
|
||||
errs.append(f"{path}: bool.v must be true/false")
|
||||
elif t in INT_TYPES:
|
||||
iv = None
|
||||
if _is_int(v):
|
||||
iv = v
|
||||
elif isinstance(v, str) and re.match(r"^-?\d+$", v):
|
||||
iv = int(v)
|
||||
else:
|
||||
errs.append(f"{path}: {t}.v must be an integer (or decimal string)")
|
||||
if iv is not None:
|
||||
lo, hi = INT_RANGE[t]
|
||||
if not lo <= iv <= hi:
|
||||
errs.append(f"{path}: {t}.v={iv} out of range")
|
||||
elif t in FLOAT_TYPES:
|
||||
if isinstance(v, bool) or not (isinstance(v, (int, float)) or v in NONFINITE):
|
||||
errs.append(f"{path}: {t}.v must be a number or 'nan'/'inf'/'-inf'")
|
||||
elif t in ("str", "wstr"):
|
||||
if not isinstance(v, str):
|
||||
errs.append(f"{path}: {t}.v must be a string")
|
||||
elif t == "str" and any(ord(c) > 0xFF for c in v):
|
||||
errs.append(f"{path}: str.v carries a code point > U+00FF (bytes must be \\u00XX)")
|
||||
elif t == "ptr":
|
||||
if not isinstance(v, str) or not _PTR_RE.match(v):
|
||||
errs.append(f"{path}: ptr.v must be a hex string like 0x00a36fd0")
|
||||
elif t == "enum":
|
||||
if not _is_int(v):
|
||||
errs.append(f"{path}: enum.v must be an integer")
|
||||
elif t == "null":
|
||||
if v is not None:
|
||||
errs.append(f"{path}: null.v must be null")
|
||||
elif t in ("list", "set"):
|
||||
if not isinstance(v, list):
|
||||
errs.append(f"{path}: {t}.v must be an array")
|
||||
else:
|
||||
for i, e in enumerate(v):
|
||||
validate_tv(e, f"{path}.v[{i}]", errs)
|
||||
elif t == "struct":
|
||||
if not isinstance(v, dict):
|
||||
errs.append(f"{path}: struct.v must be an object")
|
||||
else:
|
||||
for k, e in v.items():
|
||||
validate_tv(e, f"{path}.v.{k}", errs)
|
||||
elif t == "json":
|
||||
pass # any JSON
|
||||
|
||||
|
||||
def validate_side(side: Any, path: str, errs: list[str], need_before: bool = False) -> None:
|
||||
if not isinstance(side, dict):
|
||||
errs.append(f"{path}: must be an object")
|
||||
return
|
||||
for name, ent in side.items():
|
||||
if not isinstance(ent, dict) or "after" not in ent:
|
||||
errs.append(f"{path}.{name}: must be {{\"before\"?, \"after\"}}")
|
||||
continue
|
||||
if ent.get("before") is not None:
|
||||
validate_tv(ent["before"], f"{path}.{name}.before", errs)
|
||||
validate_tv(ent["after"], f"{path}.{name}.after", errs)
|
||||
|
||||
|
||||
def validate_record(rec: Any) -> tuple[list[str], list[str]]:
|
||||
"""-> (errors, warnings). Errors make the record invalid."""
|
||||
errs: list[str] = []
|
||||
warns: list[str] = []
|
||||
if not isinstance(rec, dict):
|
||||
return ["record is not an object"], warns
|
||||
for k in ("ts", "hook", "mode", "call_id", "thread", "args", "ret", "side"):
|
||||
if k not in rec:
|
||||
errs.append(f"missing required field {k!r}")
|
||||
if errs:
|
||||
return errs, warns
|
||||
if not _is_int(rec["ts"]) and not isinstance(rec["ts"], float):
|
||||
errs.append("ts must be a number")
|
||||
if not isinstance(rec["hook"], str) or not rec["hook"]:
|
||||
errs.append("hook must be a non-empty string")
|
||||
if rec["mode"] not in MODES:
|
||||
errs.append(f"mode must be one of {MODES}")
|
||||
if not _is_int(rec["call_id"]) or rec["call_id"] < 0:
|
||||
errs.append("call_id must be int >= 0")
|
||||
if not _is_int(rec["thread"]):
|
||||
errs.append("thread must be int")
|
||||
if "depth" in rec and (not _is_int(rec["depth"]) or rec["depth"] < 0):
|
||||
errs.append("depth must be int >= 0")
|
||||
if not isinstance(rec["args"], list):
|
||||
errs.append("args must be an array")
|
||||
else:
|
||||
for i, a in enumerate(rec["args"]):
|
||||
validate_tv(a, f"args[{i}]", errs)
|
||||
if rec["ret"] is not None:
|
||||
validate_tv(rec["ret"], "ret", errs)
|
||||
validate_side(rec["side"], "side", errs)
|
||||
if "err" in rec and not isinstance(rec["err"], str):
|
||||
errs.append("err must be a string")
|
||||
if "note" in rec and not isinstance(rec["note"], str):
|
||||
errs.append("note must be a string")
|
||||
mode = rec["mode"]
|
||||
if mode == "compare":
|
||||
ours = rec.get("ours")
|
||||
if not isinstance(ours, dict) or "ret" not in ours or "side" not in ours:
|
||||
if "err" not in rec:
|
||||
errs.append("compare record needs ours={ret, side} (or err)")
|
||||
else:
|
||||
if ours["ret"] is not None:
|
||||
validate_tv(ours["ret"], "ours.ret", errs)
|
||||
validate_side(ours["side"], "ours.side", errs)
|
||||
if "diverged" not in rec:
|
||||
warns.append("compare record without 'diverged'")
|
||||
elif not isinstance(rec["diverged"], bool):
|
||||
errs.append("diverged must be a bool")
|
||||
if "diff" in rec and not isinstance(rec["diff"], list):
|
||||
errs.append("diff must be an array")
|
||||
else:
|
||||
if "ours" in rec:
|
||||
warns.append(f"{mode} record carries 'ours' (ignored)")
|
||||
for k in rec:
|
||||
if k not in RECORD_KEYS:
|
||||
warns.append(f"unknown field {k!r}")
|
||||
return errs, warns
|
||||
|
||||
|
||||
# --- comparison ------------------------------------------------------------------
|
||||
|
||||
def _diff(path: str, why: str, a: Any, b: Any, **extra) -> dict:
|
||||
d = {"path": path, "why": why, "orig": a, "ours": b}
|
||||
d.update(extra)
|
||||
return d
|
||||
|
||||
|
||||
def _num(v, width: str = "f64") -> float:
|
||||
"""tv float payload -> Python float. f32 values are rounded to float32:
|
||||
the emitter's %.9g round-trips the float32, not the exact double."""
|
||||
if isinstance(v, str):
|
||||
return NONFINITE[v]
|
||||
x = float(v)
|
||||
if width == "f32" and math.isfinite(x):
|
||||
x = struct.unpack("<f", struct.pack("<f", x))[0]
|
||||
return x
|
||||
|
||||
|
||||
def _ulp_diff(a: float, b: float, width: str) -> int:
|
||||
fmt, ifmt = ("f", "i") if width == "f32" else ("d", "q")
|
||||
def to_int(x):
|
||||
i = struct.unpack("<" + ifmt, struct.pack("<" + fmt, x))[0]
|
||||
return i if i >= 0 else -(i & (2**(31 if fmt == "f" else 63) - 1))
|
||||
return abs(to_int(a) - to_int(b))
|
||||
|
||||
|
||||
def floats_equal(a: float, b: float, pol: Policy, width: str = "f64") -> bool:
|
||||
if math.isnan(a) or math.isnan(b):
|
||||
return math.isnan(a) and math.isnan(b)
|
||||
if math.isinf(a) or math.isinf(b):
|
||||
return a == b
|
||||
if a == b:
|
||||
return True
|
||||
if pol.ftol <= 0:
|
||||
return False
|
||||
if pol.ftol_kind == "abs":
|
||||
return abs(a - b) <= pol.ftol
|
||||
if pol.ftol_kind == "rel":
|
||||
return abs(a - b) <= pol.ftol * max(abs(a), abs(b))
|
||||
if pol.ftol_kind == "ulp":
|
||||
return _ulp_diff(a, b, width) <= pol.ftol
|
||||
raise ValueError(pol.ftol_kind)
|
||||
|
||||
|
||||
def _canon(x) -> str:
|
||||
return json.dumps(x, sort_keys=True, separators=(",", ":"))
|
||||
|
||||
|
||||
def compare_json(a: Any, b: Any, pol: Policy, path: str, out: list) -> None:
|
||||
if isinstance(a, bool) or isinstance(b, bool) or a is None or b is None or \
|
||||
isinstance(a, str) or isinstance(b, str):
|
||||
if type(a) is not type(b) or a != b:
|
||||
out.append(_diff(path, "exact", a, b))
|
||||
return
|
||||
if isinstance(a, (int, float)) and isinstance(b, (int, float)):
|
||||
both_float = isinstance(a, float) and isinstance(b, float)
|
||||
if type(a) is not type(b) and not pol.numeric:
|
||||
out.append(_diff(path, "type", a, b))
|
||||
return
|
||||
if both_float or pol.numeric:
|
||||
if not floats_equal(float(a), float(b), pol):
|
||||
out.append(_diff(path, "ftol" if pol.ftol > 0 else "exact", a, b))
|
||||
elif a != b:
|
||||
out.append(_diff(path, "exact", a, b))
|
||||
return
|
||||
if isinstance(a, list) and isinstance(b, list):
|
||||
if path in pol.unordered:
|
||||
a = sorted(a, key=_canon)
|
||||
b = sorted(b, key=_canon)
|
||||
if len(a) != len(b):
|
||||
out.append(_diff(path, "len", len(a), len(b)))
|
||||
return
|
||||
for i, (x, y) in enumerate(zip(a, b)):
|
||||
compare_json(x, y, pol, f"{path}[{i}]", out)
|
||||
return
|
||||
if isinstance(a, dict) and isinstance(b, dict):
|
||||
ka, kb = set(a), set(b)
|
||||
for k in sorted(ka - kb):
|
||||
out.append(_diff(f"{path}.{k}", "missing", a[k], None))
|
||||
for k in sorted(kb - ka):
|
||||
out.append(_diff(f"{path}.{k}", "extra", None, b[k]))
|
||||
for k in sorted(ka & kb):
|
||||
compare_json(a[k], b[k], pol, f"{path}.{k}", out)
|
||||
return
|
||||
out.append(_diff(path, "type", a, b))
|
||||
|
||||
|
||||
def compare_tv(a: Any, b: Any, pol: Policy, path: str, out: list) -> None:
|
||||
"""Append diff entries for typed values a (orig) vs b (ours)."""
|
||||
if a is None or b is None:
|
||||
if a is not b:
|
||||
out.append(_diff(path, "missing" if b is None else "extra", a, b))
|
||||
return
|
||||
ta, tb = a.get("t"), b.get("t")
|
||||
if ta != tb:
|
||||
out.append(_diff(path, "type", a, b))
|
||||
return
|
||||
t = ta
|
||||
if t == "ptr":
|
||||
va, vb = a["v"], b["v"]
|
||||
za, zb = int(va, 16) == 0, int(vb, 16) == 0
|
||||
if za != zb:
|
||||
out.append(_diff(path, "exact", a, b))
|
||||
elif pol.ptr == "exact" and int(va, 16) != int(vb, 16):
|
||||
out.append(_diff(path, "exact", a, b))
|
||||
return
|
||||
if t == "null":
|
||||
return
|
||||
if t == "bool" or t == "enum" or t in ("str", "wstr"):
|
||||
if a["v"] != b["v"]:
|
||||
out.append(_diff(path, "exact", a, b))
|
||||
return
|
||||
if t in INT_TYPES:
|
||||
if int(a["v"]) != int(b["v"]):
|
||||
out.append(_diff(path, "exact", a, b))
|
||||
return
|
||||
if t in FLOAT_TYPES:
|
||||
if not floats_equal(_num(a["v"], t), _num(b["v"], t), pol, t):
|
||||
out.append(_diff(path, "ftol" if pol.ftol > 0 else "exact", a, b))
|
||||
return
|
||||
if t == "bytes":
|
||||
if a.get("n") != b.get("n"):
|
||||
out.append(_diff(path, "len", a, b))
|
||||
return
|
||||
if a.get("sha256") != b.get("sha256"):
|
||||
extra = {}
|
||||
if "hex" in a and "hex" in b:
|
||||
ha, hb = a["hex"], b["hex"]
|
||||
off = next((i for i in range(0, min(len(ha), len(hb)), 2) if ha[i:i+2] != hb[i:i+2]), None)
|
||||
if off is not None:
|
||||
extra["first_diff_offset"] = off // 2
|
||||
out.append(_diff(path, "hash", a, b, **extra))
|
||||
elif "hex" in a and "hex" in b and a["hex"] != b["hex"]:
|
||||
out.append(_diff(path, "hash", a, b, note="same sha256, different hex: corrupt log"))
|
||||
return
|
||||
if t in ("list", "set"):
|
||||
va, vb = a["v"], b["v"]
|
||||
if t == "set" or path in pol.unordered:
|
||||
va = sorted(va, key=_canon)
|
||||
vb = sorted(vb, key=_canon)
|
||||
if len(va) != len(vb):
|
||||
out.append(_diff(path, "len", len(va), len(vb)))
|
||||
return
|
||||
for i, (x, y) in enumerate(zip(va, vb)):
|
||||
compare_tv(x, y, pol, f"{path}.v[{i}]", out)
|
||||
return
|
||||
if t == "struct":
|
||||
va, vb = a["v"], b["v"]
|
||||
for k in sorted(set(va) - set(vb)):
|
||||
out.append(_diff(f"{path}.v.{k}", "missing", va[k], None))
|
||||
for k in sorted(set(vb) - set(va)):
|
||||
out.append(_diff(f"{path}.v.{k}", "extra", None, vb[k]))
|
||||
for k in sorted(set(va) & set(vb)):
|
||||
compare_tv(va[k], vb[k], pol, f"{path}.v.{k}", out)
|
||||
return
|
||||
if t == "json":
|
||||
compare_json(a["v"], b["v"], pol, path, out)
|
||||
return
|
||||
raise AssertionError(t)
|
||||
|
||||
|
||||
def _side_after(side: dict, name: str) -> Any:
|
||||
ent = side[name]
|
||||
if isinstance(ent, dict) and ent.get("t") is None and "after" in ent:
|
||||
return ent["after"]
|
||||
return ent # replay input may give the tv directly
|
||||
|
||||
|
||||
def compare_outputs(orig_ret, orig_side: dict, ours_ret, ours_side: dict, pol: Policy) -> list[dict]:
|
||||
diffs: list[dict] = []
|
||||
compare_tv(orig_ret, ours_ret, pol, "ret", diffs)
|
||||
names_a, names_b = set(orig_side), set(ours_side)
|
||||
for n in sorted(names_a - names_b):
|
||||
diffs.append(_diff(f"side.{n}.after", "missing", _side_after(orig_side, n), None))
|
||||
for n in sorted(names_b - names_a):
|
||||
diffs.append(_diff(f"side.{n}.after", "extra", None, _side_after(ours_side, n)))
|
||||
for n in sorted(names_a & names_b):
|
||||
compare_tv(_side_after(orig_side, n), _side_after(ours_side, n), pol, f"side.{n}.after", diffs)
|
||||
return diffs
|
||||
|
||||
|
||||
def compare_record(rec: dict, pol: Policy) -> tuple[list[dict], list[str]]:
|
||||
"""Recompute a compare record's diff. -> (diffs, warnings)"""
|
||||
warns: list[str] = []
|
||||
if "err" in rec:
|
||||
return [_diff("call", "err", None, rec["err"])], warns
|
||||
ours = rec["ours"]
|
||||
diffs = compare_outputs(rec["ret"], rec["side"], ours["ret"], ours["side"], pol)
|
||||
# snapshot sanity: 'before' on both sides should be identical
|
||||
for n in set(rec["side"]) & set(ours["side"]):
|
||||
ba, bb = rec["side"][n].get("before"), ours["side"][n].get("before")
|
||||
if ba is not None and bb is not None:
|
||||
tmp: list = []
|
||||
compare_tv(ba, bb, pol, f"side.{n}.before", tmp)
|
||||
if tmp:
|
||||
warns.append(f"call {rec['call_id']}: side.{n}.before differs between sides (snapshot bug?)")
|
||||
if "diverged" in rec and rec["diverged"] != bool(diffs):
|
||||
warns.append(f"call {rec['call_id']}: shim said diverged={rec['diverged']}, harness found {len(diffs)} diff(s)")
|
||||
return diffs, warns
|
||||
|
||||
|
||||
# --- loading -----------------------------------------------------------------------
|
||||
|
||||
@dataclass
|
||||
class Log:
|
||||
path: str
|
||||
meta: dict = field(default_factory=dict)
|
||||
records: list = field(default_factory=list)
|
||||
invalid: list = field(default_factory=list) # (lineno, message)
|
||||
warnings: list = field(default_factory=list) # (lineno, message)
|
||||
|
||||
|
||||
def load_log(path: str, policies: PolicyTable | None = None) -> Log:
|
||||
log = Log(path)
|
||||
seen: dict[int, int] = {}
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
for lineno, line in enumerate(f, 1):
|
||||
line = line.rstrip("\n")
|
||||
if line.endswith("\r"):
|
||||
log.warnings.append((lineno, "CRLF line ending"))
|
||||
line = line[:-1]
|
||||
if not line.strip():
|
||||
log.invalid.append((lineno, "blank line"))
|
||||
continue
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except ValueError as e:
|
||||
log.invalid.append((lineno, f"not JSON: {e}"))
|
||||
continue
|
||||
if isinstance(rec, dict) and "meta" in rec and len(rec) == 1:
|
||||
m = rec["meta"]
|
||||
if not isinstance(m, dict):
|
||||
log.invalid.append((lineno, "meta must be an object"))
|
||||
continue
|
||||
if m.get("format") != FORMAT_VERSION:
|
||||
log.invalid.append((lineno, f"unsupported format {m.get('format')!r} (want {FORMAT_VERSION})"))
|
||||
continue
|
||||
if lineno != 1:
|
||||
log.warnings.append((lineno, "meta record not on line 1"))
|
||||
log.meta = m
|
||||
if policies is not None:
|
||||
policies.add_meta(m.get("hooks", {}))
|
||||
continue
|
||||
errs, warns = validate_record(rec)
|
||||
for w in warns:
|
||||
log.warnings.append((lineno, w))
|
||||
if errs:
|
||||
log.invalid.append((lineno, "; ".join(errs)))
|
||||
continue
|
||||
cid = rec["call_id"]
|
||||
if cid in seen:
|
||||
log.invalid.append((lineno, f"duplicate call_id {cid} (first at line {seen[cid]})"))
|
||||
continue
|
||||
seen[cid] = lineno
|
||||
rec["_line"] = lineno
|
||||
log.records.append(rec)
|
||||
return log
|
||||
|
||||
|
||||
def load_impl(path: str) -> tuple[dict[int, dict], list[tuple[int, str]]]:
|
||||
"""Implementation output for --replay: records keyed by call_id."""
|
||||
out: dict[int, dict] = {}
|
||||
invalid: list[tuple[int, str]] = []
|
||||
with open(path, "r", encoding="utf-8") as f:
|
||||
for lineno, line in enumerate(f, 1):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
rec = json.loads(line)
|
||||
except ValueError as e:
|
||||
invalid.append((lineno, f"not JSON: {e}"))
|
||||
continue
|
||||
if isinstance(rec, dict) and "meta" in rec and len(rec) == 1:
|
||||
continue
|
||||
if not isinstance(rec, dict) or not _is_int(rec.get("call_id")):
|
||||
invalid.append((lineno, "impl record needs an integer call_id"))
|
||||
continue
|
||||
errs: list[str] = []
|
||||
if rec.get("ret") is not None:
|
||||
validate_tv(rec["ret"], "ret", errs)
|
||||
side = rec.get("side", {})
|
||||
if not isinstance(side, dict):
|
||||
errs.append("side must be an object")
|
||||
else:
|
||||
for n in side:
|
||||
validate_tv(_side_after(side, n), f"side.{n}", errs)
|
||||
if errs:
|
||||
invalid.append((lineno, "; ".join(errs)))
|
||||
continue
|
||||
if rec["call_id"] in out:
|
||||
invalid.append((lineno, f"duplicate call_id {rec['call_id']}"))
|
||||
continue
|
||||
rec.setdefault("side", {})
|
||||
rec.setdefault("ret", None)
|
||||
rec["_line"] = lineno
|
||||
out[rec["call_id"]] = rec
|
||||
return out, invalid
|
||||
|
||||
|
||||
# --- report ------------------------------------------------------------------------
|
||||
|
||||
def new_hook_stats() -> dict:
|
||||
return {"calls": 0, "modes": {}, "compared": 0, "diverged": 0, "errors": 0,
|
||||
"diverged_call_ids": [], "diffs": []}
|
||||
|
||||
|
||||
def run_report(logs: list[Log], policies: PolicyTable, hooks: set[str] | None, first: int) -> dict:
|
||||
per_hook: dict[str, dict] = {}
|
||||
warnings: list[str] = []
|
||||
for log in logs:
|
||||
for ln, w in log.warnings:
|
||||
warnings.append(f"{os.path.basename(log.path)}:{ln}: {w}")
|
||||
for rec in log.records:
|
||||
h = rec["hook"]
|
||||
if hooks and h not in hooks:
|
||||
continue
|
||||
st = per_hook.setdefault(h, new_hook_stats())
|
||||
st["calls"] += 1
|
||||
st["modes"][rec["mode"]] = st["modes"].get(rec["mode"], 0) + 1
|
||||
if rec["mode"] != "compare":
|
||||
if "err" in rec:
|
||||
st["errors"] += 1
|
||||
continue
|
||||
st["compared"] += 1
|
||||
diffs, w = compare_record(rec, policies.for_hook(h))
|
||||
warnings.extend(f"{os.path.basename(log.path)}:{rec['_line']}: {x}" for x in w)
|
||||
if diffs:
|
||||
st["diverged"] += 1
|
||||
if "err" in rec:
|
||||
st["errors"] += 1
|
||||
st["diverged_call_ids"].append(rec["call_id"])
|
||||
if len(st["diffs"]) < first:
|
||||
st["diffs"].append({"call_id": rec["call_id"], "line": rec["_line"],
|
||||
"file": os.path.basename(log.path), "diff": diffs})
|
||||
return _finish(per_hook, logs, warnings, "report")
|
||||
|
||||
|
||||
def run_replay(golden: list[Log], impl: dict[int, dict], policies: PolicyTable,
|
||||
hooks: set[str] | None, first: int) -> dict:
|
||||
per_hook: dict[str, dict] = {}
|
||||
warnings: list[str] = []
|
||||
seen: set[int] = set()
|
||||
for log in golden:
|
||||
for ln, w in log.warnings:
|
||||
warnings.append(f"{os.path.basename(log.path)}:{ln}: {w}")
|
||||
for rec in log.records:
|
||||
h = rec["hook"]
|
||||
if hooks and h not in hooks:
|
||||
continue
|
||||
st = per_hook.setdefault(h, new_hook_stats())
|
||||
st["calls"] += 1
|
||||
st["modes"][rec["mode"]] = st["modes"].get(rec["mode"], 0) + 1
|
||||
st["compared"] += 1
|
||||
cid = rec["call_id"]
|
||||
seen.add(cid)
|
||||
ours = impl.get(cid)
|
||||
if ours is None:
|
||||
diffs = [_diff("call", "missing", None, None, note=f"call_id {cid} absent from impl")]
|
||||
else:
|
||||
if "hook" in ours and ours["hook"] != h:
|
||||
warnings.append(f"impl call_id {cid}: hook {ours['hook']!r} != golden {h!r}")
|
||||
diffs = compare_outputs(rec["ret"], rec["side"], ours["ret"], ours["side"], policies.for_hook(h))
|
||||
if diffs:
|
||||
st["diverged"] += 1
|
||||
st["diverged_call_ids"].append(cid)
|
||||
if len(st["diffs"]) < first:
|
||||
st["diffs"].append({"call_id": cid, "line": rec["_line"],
|
||||
"file": os.path.basename(log.path), "diff": diffs})
|
||||
extra = sorted(set(impl) - seen)
|
||||
if extra:
|
||||
warnings.append(f"impl has {len(extra)} call_id(s) not in golden: {extra[:10]}{'…' if len(extra) > 10 else ''}")
|
||||
return _finish(per_hook, golden, warnings, "replay")
|
||||
|
||||
|
||||
def _finish(per_hook: dict, logs: list[Log], warnings: list[str], kind: str) -> dict:
|
||||
total_div = sum(s["diverged"] for s in per_hook.values())
|
||||
invalid = [(os.path.basename(l.path), ln, m) for l in logs for ln, m in l.invalid]
|
||||
return {
|
||||
"kind": kind,
|
||||
"format": FORMAT_VERSION,
|
||||
"inputs": [l.path for l in logs],
|
||||
"meta": [l.meta for l in logs if l.meta],
|
||||
"hooks": dict(sorted(per_hook.items())),
|
||||
"totals": {"calls": sum(s["calls"] for s in per_hook.values()),
|
||||
"compared": sum(s["compared"] for s in per_hook.values()),
|
||||
"diverged": total_div,
|
||||
"invalid_records": len(invalid)},
|
||||
"invalid": [{"file": f, "line": ln, "error": m} for f, ln, m in invalid],
|
||||
"warnings": warnings,
|
||||
}
|
||||
|
||||
|
||||
def _short(x: Any, n: int = 60) -> str:
|
||||
s = _canon(x)
|
||||
return s if len(s) <= n else s[:n - 1] + "…"
|
||||
|
||||
|
||||
def print_report(rep: dict, out=None) -> None:
|
||||
out = out or sys.stdout
|
||||
P = lambda s="": print(s, file=out) # noqa: E731
|
||||
P(f"## tracecmp {rep['kind']}: {', '.join(os.path.basename(p) for p in rep['inputs'])}")
|
||||
P()
|
||||
for m in rep["meta"]:
|
||||
P(f"- build: {m.get('build', '?')} started: {m.get('started', '?')} inline_max: {m.get('inline_max', '?')}")
|
||||
t = rep["totals"]
|
||||
P(f"- calls: {t['calls']} compared: {t['compared']} diverged: {t['diverged']} "
|
||||
f"invalid records: {t['invalid_records']} warnings: {len(rep['warnings'])}")
|
||||
P()
|
||||
P("| hook | calls | modes | compared | diverged | errors |")
|
||||
P("|---|---|---|---|---|---|")
|
||||
for h, s in rep["hooks"].items():
|
||||
modes = " ".join(f"{k}:{v}" for k, v in sorted(s["modes"].items()))
|
||||
P(f"| {h} | {s['calls']} | {modes} | {s['compared']} | {s['diverged']} | {s['errors']} |")
|
||||
for h, s in rep["hooks"].items():
|
||||
if not s["diffs"]:
|
||||
continue
|
||||
P()
|
||||
P(f"### {h}: first {len(s['diffs'])} of {s['diverged']} divergent call(s)")
|
||||
for d in s["diffs"]:
|
||||
P(f"- call_id {d['call_id']} ({d['file']}:{d['line']})")
|
||||
for e in d["diff"][:8]:
|
||||
extra = "".join(f" {k}={v}" for k, v in e.items() if k not in ("path", "why", "orig", "ours"))
|
||||
P(f" {e['path']} [{e['why']}] orig={_short(e['orig'])} ours={_short(e['ours'])}{extra}")
|
||||
if len(d["diff"]) > 8:
|
||||
P(f" … {len(d['diff']) - 8} more")
|
||||
more = s["diverged_call_ids"][len(s["diffs"]):]
|
||||
if more:
|
||||
P(f" other divergent call_ids: {more[:20]}{' …' if len(more) > 20 else ''}")
|
||||
if rep["invalid"]:
|
||||
P()
|
||||
P(f"### invalid records ({len(rep['invalid'])})")
|
||||
for e in rep["invalid"][:20]:
|
||||
P(f"- {e['file']}:{e['line']}: {e['error']}")
|
||||
if rep["warnings"]:
|
||||
P()
|
||||
P(f"### warnings ({len(rep['warnings'])})")
|
||||
for w in rep["warnings"][:20]:
|
||||
P(f"- {w}")
|
||||
|
||||
|
||||
# --- main ----------------------------------------------------------------------------
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0], formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("logs", nargs="+", metavar="LOG", help="trace/compare JSONL (golden logs in --replay)")
|
||||
ap.add_argument("--replay", metavar="IMPL", help="implementation output JSONL to diff against LOG")
|
||||
ap.add_argument("--hook", action="append", default=[], help="only this hook (repeatable)")
|
||||
ap.add_argument("--tolerance", action="append", default=[], metavar="SPEC", help="[HOOK=][abs:|rel:|ulp:]NUMBER")
|
||||
ap.add_argument("--unordered", action="append", default=[], metavar="SPEC", help="[HOOK=]PATH")
|
||||
ap.add_argument("--ptr", choices=("ignore", "exact"), default=None)
|
||||
ap.add_argument("--numeric", action="store_true")
|
||||
ap.add_argument("--first", type=int, default=5, metavar="N")
|
||||
ap.add_argument("--json-out", metavar="PATH")
|
||||
ap.add_argument("--skip-invalid", action="store_true")
|
||||
ap.add_argument("--quiet", action="store_true", help="no markdown report on stdout")
|
||||
return ap
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
ap = build_parser()
|
||||
a = ap.parse_args(argv)
|
||||
policies = PolicyTable()
|
||||
try:
|
||||
for s in a.tolerance:
|
||||
policies.add_cli(*parse_tolerance(s))
|
||||
for s in a.unordered:
|
||||
policies.add_cli(*parse_unordered(s))
|
||||
except ValueError as e:
|
||||
ap.error(str(e))
|
||||
if a.ptr:
|
||||
policies.add_cli(None, {"ptr": a.ptr})
|
||||
if a.numeric:
|
||||
policies.add_cli(None, {"numeric": True})
|
||||
hooks = set(a.hook) or None
|
||||
|
||||
logs = []
|
||||
for p in a.logs:
|
||||
if not os.path.exists(p):
|
||||
print(f"error: no such file {p}", file=sys.stderr)
|
||||
return 2
|
||||
logs.append(load_log(p, policies))
|
||||
impl_invalid: list = []
|
||||
if a.replay:
|
||||
if not os.path.exists(a.replay):
|
||||
print(f"error: no such file {a.replay}", file=sys.stderr)
|
||||
return 2
|
||||
impl, impl_invalid = load_impl(a.replay)
|
||||
rep = run_replay(logs, impl, policies, hooks, a.first)
|
||||
rep["impl"] = a.replay
|
||||
rep["invalid"].extend({"file": os.path.basename(a.replay), "line": ln, "error": m} for ln, m in impl_invalid)
|
||||
rep["totals"]["invalid_records"] += len(impl_invalid)
|
||||
else:
|
||||
rep = run_report(logs, policies, hooks, a.first)
|
||||
|
||||
if a.json_out:
|
||||
os.makedirs(os.path.dirname(os.path.abspath(a.json_out)), exist_ok=True)
|
||||
with open(a.json_out, "w", encoding="utf-8") as f:
|
||||
json.dump(rep, f, indent=1, sort_keys=True)
|
||||
f.write("\n")
|
||||
if not a.quiet:
|
||||
print_report(rep)
|
||||
if rep["totals"]["invalid_records"] and not a.skip_invalid:
|
||||
return 2
|
||||
return 1 if rep["totals"]["diverged"] else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
1
verify/results/compare/.keep.md
Normal file
1
verify/results/compare/.keep.md
Normal file
|
|
@ -0,0 +1 @@
|
|||
# per-run tracecmp reports (--json-out); see verify/harness/compare/
|
||||
Loading…
Add table
Reference in a new issue