alex/sots-re
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
sots-re/verify/harness/compare/TRACE_FORMAT.md

12 KiB
Raw Blame History

Trace / compare log format (v1)

Wire format between the shim (sots-engine/src/shim/, C++, no JSON library) and the harness in this directory (tracecmp.py, stdlib Python). One file per shim run; golden traces live in verify/traces/, harness reports in verify/results/compare/.

Design goals, in order: (1) trivially emittable from C++ with fprintf; (2) every value carries its C type so the harness can apply the right equality rule; (3) big memory regions never blow up the log (hash, summarize, cap); (4) the same record shape serves online compare (shim runs original + ours) and offline replay (harness diffs a golden trace log against a host-side implementation's output).

1. Container: JSON Lines

  • UTF-8, no BOM. One JSON object per line, terminated by \n (LF only). No blank lines, no comments, no trailing commas. A line that does not parse is an invalid record.
  • The first line SHOULD be a meta record (section 5). Everything else is a call record (section 2).
  • Records are appended in emission order; call_id (not line order) is the identity.
  • Log files are named <run-tag>.jsonl (e.g. verify/traces/m1-cfgvar-20260907.jsonl).

2. Call record

{"ts":123456,"hook":"CfgVar_RegisterKey","mode":"trace","call_id":17,"thread":4120,
 "args":[{"t":"str","v":"ForceSingleCore"},{"t":"i32","v":1}],
 "ret":{"t":"bool","v":true},
 "side":{"cfg_table":{"before":{"t":"bytes","n":4096,"sha256":"…"},
                      "after":{"t":"bytes","n":4096,"sha256":"…"}}}}
field type required meaning
ts int yes microseconds since shim init (QPC-based). Informational; never compared.
hook string yes hook name — the function's recovered name (Game::… / Mars::… or FUN_xxxxxxxx).
mode "trace" | "compare" | "replace" yes which mode the hook ran in for this call.
call_id int ≥ 0 yes process-global, monotonically increasing, atomic. Unique within a log.
thread int yes OS thread id. Informational.
depth int no re-entrancy depth (0 = outermost). Informational.
args array of typed values yes inputs, in declaration order. n names them when known. Never compared (they are the snapshot both sides ran on).
ret typed value or null yes return value of the original (trace/compare) or of ours (replace). null = void.
side object yes (may be {}) declared side effects: `name -> {"before": tv
ours object compare only `{"ret": tv
diverged bool compare only the shim's own verdict. The harness recomputes and reports disagreements as warnings.
diff array of diff entries compare only the shim's own diff (may be [] or truncated). Advisory.
err string no the hook could not capture (exception in ours, snapshot failure…). Counts as a divergence in compare.
note string no free text. Ignored.

Unknown top-level keys are a validation warning (not an error) so the shim can add fields without breaking older harnesses.

replace records have no ours; ret/side are the reimplementation's. They are counted and validated but never diffed (there is nothing to diff against).

3. Typed values (tv)

Every value is an object {"t": TYPE, "v": VALUE} plus optional "n": "name". TYPEs and their equality rule:

t v equality
bool true/false exact
i8 i16 i32 i64 u8 u16 u32 u64 JSON integer. i64/u64 above 2^53 MUST be a decimal string ("v":"18446744073709551615") — the harness accepts both int and string. exact
f32, f64 JSON number, or the string "nan", "inf", "-inf". Emit with %.9g (f32) / %.17g (f64). tolerant (section 6)
str JSON string carrying raw bytes of a char*/std::string (cp1252 in this engine). Each byte maps to one code point U+0000–U+00FF (see escaping, section 4). exact
wstr JSON string of a wchar_t* (UTF-16 code units → \uXXXX). exact
ptr hex string "0x00a36fd0" ignored by default (ours allocates elsewhere). A hook policy may set "ptr":"exact".
enum integer, optional "name" sibling with the symbolic name exact on the integer
null null exact (only equals null)
bytes see below exact on sha256 (and on hex when both inline)
list JSON array of tv same length, element-wise, ordered
set JSON array of tv unordered multiset: sort both by canonical text, then element-wise
struct JSON object {field: tv} same key set, field-wise (recursion)
json any canonical-JSON value (section 7) — used by the oracle bridge and host-side tests structural exact, with the hook's float tolerance applied to JSON numbers that are floats on both sides

A type mismatch between the two sides (i32 vs u32, str vs wstr) is a divergence (why: "type"). Type mismatch does not apply inside json values, where int vs float is a divergence unless both are numbers and the tolerance policy is "numeric".

bytes — binary blobs and memory regions

{"t":"bytes","n":4096,"sha256":"<64 hex>","hex":"<2n hex>"}
{"t":"bytes","n":1048576,"sha256":"<64 hex>","head":"<first 32 bytes hex>"}
  • n (length in bytes) and sha256 (lowercase hex of the SHA-256 of the raw bytes) are always present, even for n = 0 (sha256 of the empty string).
  • hex (lowercase, no separators) is present when n <= inline_max. inline_max is a shim setting recorded in meta (default 256). hex is the only inline encoding; base64 is not used (harder to eyeball, no size win worth a second decoder).
  • Larger blobs carry head (first 32 bytes, hex) for eyeballing only.
  • When both sides are inline and differ, the harness reports the first differing byte offset; when hashed, only that the hash differs.
  • A memory region is just a bytes value captured at a declared (address, length); the region's name is the key in side. Structured regions should be captured as struct of typed values instead so diffs point at the field, not a byte offset.

4. Emitter rules (C++ without a JSON library)

The emitter is a set of fprintfs. What it must guarantee:

  1. Key order is fixed: ts, hook, mode, call_id, thread, depth, args, ret, side, ours, diverged, diff, err, note. (The harness does not care, but fixed order makes logs diff-able by eye and grep-able.)
  2. Strings (hook, str/wstr v, n, err, note, struct field names, side names): write ", then for each unit:
    • " → \", \ → \\
    • \n → \n, \r → \r, \t → \t (literal backslash-letter)
    • any other unit < 0x20, and for str any byte >= 0x7f → \u00XX (lowercase hex of the byte), for wstr any unit >= 0x7f → \uXXXX
    • everything else verbatim. Close with ". This is exactly what Python's json.dumps(ensure_ascii=True) produces for a latin-1-decoded byte string (except Python also short-forms \b and \f; the harness accepts both). No raw bytes ≥ 0x80 ever appear in the log, so the file is ASCII-only.
  3. Numbers: integers with %lld/%llu; floats with %.9g / %.17g; non-finite floats as the strings "nan", "inf", "-inf". Never emit nan/inf bare (invalid JSON).
  4. Booleans true/false; void null.
  5. No trailing commas. Build arrays/objects with a "first element" flag.
  6. One fprintf chain per record, ending in "}\n" and a fflush at least on compare divergences and on shim teardown, so a crash mid-run leaves a usable log (the harness tolerates a truncated last line: it is reported as one invalid record).
  7. Thread safety: take a lock around the whole record; call_id is InterlockedIncrement-ed before the original is called, so nested hooks get increasing ids and depth says who is inside whom.
  8. sha256: any tiny public-domain SHA-256; hex lowercase.

5. meta record (first line)

{"meta":{"format":1,"build":"sots-engine 0.0.3 g1a2b3c4","exe_sha256":"…",
         "started":"2026-09-07T18:00:00Z","inline_max":256,
         "hooks":{"CfgVar_RegisterKey":{"ftol":0,"ptr":"ignore"},
                  "Mars::ParseBlock":{"ftol":1e-6,"ftol_kind":"rel","unordered":["ret.v.items"]}}}}
  • format: this spec's version (1). The harness refuses other versions.
  • hooks: per-hook policy, the same keys tracecmp.py --tolerance accepts. CLI flags override the log's policy; the log's policy overrides harness defaults.
  • Everything else is informational and copied into the report.

6. Divergence rules

A call diverges when any of: err is set; ret differs; any side[name].after differs from ours.side[name].after; a side name exists on one side only (why: "missing"). Comparison is by the typed-value table above, recursively. Paths in diff entries: ret, side.<name>.after, then .v[<index>] for list/set elements, .v.<field> for struct fields, .<key>/[<i>] inside json values.

{"path":"side.cfg_table.after.v.ForceSingleCore","why":"exact","orig":{"t":"i32","v":1},"ours":{"t":"i32","v":0}}

why ∈ exact | ftol | type | len | missing | extra | hash | err.

Float tolerance policy (per hook; default ftol = 0, i.e. exact by value with nan == nan):

  • ftol: number ≥ 0.
  • ftol_kind: "abs" (default) — |a-b| <= ftol; "rel" — |a-b| <= ftol * max(|a|,|b|); "ulp" — within ftol units-in-last-place of the value's own width (f32/f64).
  • nan, inf, -inf: equal only to themselves, whatever the tolerance.
  • f32 values are rounded to float32 on both sides before comparing: %.9g round-trips the float32 exactly but not the double it was widened to. f64 compares as-is.
  • f32 compared to f64 is a type divergence; do not mix.

Unordered fields: set values are always unordered. A policy may additionally list unordered: [<path>, …] to treat specific list paths as sets (path syntax as above, without the .v[i] tail). Multiset semantics: element multiplicity matters.

Pointers: ptr is ignored unless the policy says "ptr":"exact". Two ptr values where exactly one is "0x0" is always a divergence (null vs non-null is semantic).

7. Canonical JSON (for json values and the oracle bridge)

The form a host-side C++ test dump must reproduce byte-for-byte so a plain diff (or tracecmp.py --replay) works. It is Python's json.dumps(obj, sort_keys=True, ensure_ascii=True, separators=(",", ":")) after normalization by oracle_parsers.canonical():

  • objects: keys sorted by code point, no whitespace anywhere ({"a":1,"b":[1,2]});
  • strings: byte-preserving — the parsers decode cp1252, the canonicalizer re-encodes to cp1252 and maps each byte to U+00XX, so the escaping in section 4 applies verbatim (a C++ dump escapes raw bytes, no charset table needed);
  • integers: decimal, no leading +/zeros;
  • floats: rounded to float32, printed with %.9g, then .0 appended when the text has no ., e or n (so 1.0 stays distinguishable from the int 1, and -0 → -0.0); non-finite → the strings "nan", "inf", "-inf";
  • booleans true/false; null for None; tuples become arrays; dataclasses become objects of their fields;
  • the file ends with a single \n.

8. Replay (offline) input

tracecmp.py --replay GOLDEN.jsonl IMPL.jsonl: GOLDEN is a normal log (any mode; usually trace). IMPL is JSONL whose records need only call_id, ret, side (side may be {name: tv} or {name: {"after": tv}}; hook optional but checked when present). Each golden call_id must appear in IMPL (why: "missing" otherwise); extra call_ids in IMPL are warnings. The diff rules are those of section 6, with IMPL playing ours.