alex/sots-re
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
sots-re/verify/state-checksum/state_checksum.py
alex d46b3f8391 state_checksum: --relabel-new-ids, comparing a pair modulo this turn's new id labelling 
Implements the 2026-09-09 fleet-id-order resolution, section 3 item 1.  Given the
pre-turn save, compute the ids new in each post-turn save, match the client-minted
(node nibble != 0) new fleets by a key that does not mention the id -- (LocID or
FPlan destination, sorted ship-id set) -- build the bijection pi, rewrite every
fleet reference, compare the master id lists as sets, mask /Summary/Checksum with
its reason on the line, and print pi.

Acceptance, both halves:
  bp-pinA vs bp-pinB     IDENTICAL modulo pi = {1970<->1986}   (35 leaves -> 0)
  ad-oracle-A vs -B      REFUSED, then DIVERGED: 94 leaves     (unchanged)

Five guards, every one refusing rather than degrading: only ids absent from the
pre-turn save; only non-zero node nibbles; pi must permute one set; content keys
must correspond one-to-one and be unique per side; and no leaf anywhere may hold a
permuted id at an unmodelled site (matched on raw bytes, not the reader's typed
value).  A refusal rewrites nothing and falls back to the ordinary comparison.

Three corrections to the specification from contact with the data, in
findings/subsystems/relabel-new-ids.md section 4: FtName is an id-attached label
and needs the same treatment as the id; relabelling the Flt[] keys is the wrong
operation (exchange the bodies -- the fleet table is id-ordered and identical in
both saves); a node's new-id set spans object kinds.

Default path proven unchanged: pre- and post-change modules agree on the root
digest, coverage, mask hits and every (path, digest) in the tree over all 43 saves
under two policies, and on 5,602 lines of CLI stdout across every mode.

Also fixes a pre-existing, unrelated test failure: the re-save localisation test
enumerated pairs over sorted filenames and hard-coded the direction 4 -> 0, which
a later corpus addition reversed.  Suite 38 -> 62 tests, all passing.
2026-09-09 04:12:01 -04:00

1352 lines
56 KiB
Python
Raw Blame History

#!/usr/bin/env python3
"""state_checksum.py -- whole-state, diagnostic checksum of a SOTS1 save.
Why this exists
---------------
Every other verification in this project is *per-function*: hook one routine,
compare the regions it declares, print a verdict. That is only as good as the
region declaration, and region declarations have been wrong (B4: three hooks
that reported "0 diverged" while comparing nothing; the harness audit: 23
undeclared side effects). A green verdict over an empty region set is a
failure, not a pass.
This tool is the complement. It does not ask "did the declared outputs match".
It asks **"is the entire simulation state still identical"**, computed from the
one artefact the game emits as a pure function of state -- the save file, which
`findings/subsystems/determinism-oracle.md` proved byte-identical across runs
and processes.
Two properties make it evidence rather than a comforting number:
1. **Provable coverage.** The checksum is computed from a tree that is
re-serialised and compared byte-for-byte against the inflated save
(`--audit`, on by default). If the reconstruction reproduces the stream,
then every byte of the save is a function of the checksum's inputs, so *no*
state change can be invisible to it. Coverage is asserted by construction,
not by a hand-written region list that someone can forget to fill in.
2. **It localises.** The root digest is the fold of a tree of per-subsystem /
per-object digests, and objects are named (`Player[16 "re"]`,
`Sys[112 "Gamma Cephei"]`, `Events/[turn=3]/[id=3]`), so a divergence
reports *which object moved*, not just that the hash moved.
Float parity: see STATE_CHECKSUM.md section 3. Short version -- the digest is
always exact (raw IEEE-754 bits, or bits with -0.0/NaN normalised under
`--floats canonical`). Tolerance is deliberately *not* a hashing mode; it
exists only in the differ (`--ulps`), because a tolerant hash is a
contradiction: quantisation just moves the cliff, it does not remove it.
Usage
-----
state_checksum.py SAVE # root + subsystem digests
state_checksum.py SAVE --tree --depth 3 # digest tree
state_checksum.py A B # localise divergence
state_checksum.py A B --ulps 2 # classify float diffs
state_checksum.py A B --relabel-new-ids PRE # compare modulo new-id labelling
state_checksum.py --chain chain.json S1 S2 ... # verify a recorded chain
state_checksum.py SAVE --json # machine-readable
Stdlib only (plus save_reader.py, which is stdlib only).
"""
from __future__ import annotations
import argparse
import hashlib
import json
import math
import os
import struct
import sys
_HERE = os.path.dirname(os.path.abspath(__file__))
_READER = os.path.join(os.path.dirname(_HERE), "save-reader")
if _READER not in sys.path:
sys.path.insert(0, _READER)
import save_reader as sr # noqa: E402
__all__ = [
"FLOAT_POLICIES", "MASK_PRESETS", "CkNode", "checksum_save", "checksum_bytes",
"checksum_result", "reconstruct", "audit_coverage", "diff", "DiffEntry",
"float_key", "RelabelRefused", "Relabelling", "build_relabelling",
"apply_relabelling", "format_permutation", "id_node",
]
DIGEST_BYTES = 16 # blake2b-128; 2**-64 collision floor at our object counts
SHORT = 16 # hex chars shown in text output
def _reader_fingerprint() -> str:
"""Identity of the save_reader build the digests were computed under.
The digest hashes each leaf's *inferred kind* alongside its bytes, so it is
a function of (save bytes, reader schema) -- not of the bytes alone. That
is fine for comparing two saves parsed by one reader, and dangerous for a
chain recorded months ago. So it is recorded and checked, never assumed.
It is deliberately NOT folded into the digest: a cosmetic reader edit
should not invalidate every recorded root, it should raise a warning.
"""
try:
with open(sr.__file__, "rb") as f:
return hashlib.blake2b(f.read(), digest_size=8).hexdigest()
except OSError: # pragma: no cover
return "unknown"
READER_FINGERPRINT = _reader_fingerprint()
BEGIN = struct.pack("<I", 0xBEEFBEEF)
END = struct.pack("<I", 0x41104110)
# --- policy -------------------------------------------------------------------
FLOAT_POLICIES = ("bits", "canonical")
#: Documented canonicalisation presets (see determinism-oracle.md). A rule is
#: (leaf tag, required path prefix, replacement kind, replacement value) -- the
#: path prefix keeps a mask from silently swallowing a same-named field
#: somewhere else in the tree, and the run reports how many leaves each rule
#: actually hit so an over-broad mask is visible rather than comfortable.
#:
#: Masking is opt-in. By default nothing is masked, so the known re-save delta
#: is *localised* rather than absorbed.
MASK_PRESETS = {
"none": (),
# load -> save of a post-turn autosave: the server clears the per-player
# "turn ended" flag (4 -> 0) and the derived top-level Checksum follows.
"resave": (
("Status", "/Sim/players/Player[", "int", 0),
("Checksum", "/Summary/", "derived", None),
),
}
def float_key(raw: bytes, policy: str) -> bytes:
"""Hashing key for a 4-byte IEEE-754 single under `policy`."""
if policy == "bits":
return raw
if policy != "canonical":
raise ValueError(f"unknown float policy {policy!r}")
(bits,) = struct.unpack("<I", raw)
if bits == 0x80000000: # -0.0 -> +0.0
return b"\0\0\0\0"
if (bits & 0x7F800000) == 0x7F800000 and (bits & 0x007FFFFF):
return struct.pack("<I", 0x7FC00000) # any NaN -> one quiet NaN
return raw
def ulps_apart(a: bytes, b: bytes) -> float:
"""Distance in float32 ULPs between two raw singles. inf if not comparable."""
(x,) = struct.unpack("<f", a)
(y,) = struct.unpack("<f", b)
if math.isnan(x) or math.isnan(y):
return 0.0 if (math.isnan(x) and math.isnan(y)) else math.inf
if math.isinf(x) or math.isinf(y):
return 0.0 if x == y else math.inf
def ordinal(v: bytes) -> int:
(u,) = struct.unpack("<i", v)
return u if u >= 0 else -0x80000000 - u # monotone map across the sign
return float(abs(ordinal(a) - ordinal(b)))
# --- naming -------------------------------------------------------------------
#: lead-int tag -> (frame tag, group name). The save stores object tables as a
#: flat run of (id, frame) sibling pairs; folding each run into a named group
#: turns `Sim`'s 259 children into a subsystem tree.
PAIR_GROUPS = {
"PlayerID": ("Player", "players"),
"SysID": ("Sys", "systems"),
"FltID": ("Flt", "fleets"),
"ShipID": ("Ship", "ships"),
"DesID": ("Des", "designs"),
"TradeID": ("Trade", "trades"),
"ply": ("hist", "history"),
}
#: frame tag -> child tags to use as a human label, first match wins. Tags are
#: as spelled on disk (SAVE_FORMAT.md section 10); `Ship` carries no name field,
#: so a ship is labelled by its id alone.
NAME_FIELDS = {
"Player": ("PlryName",),
"Sys": ("Name",),
"Flt": ("FtName",),
"Des": ("DName",),
}
#: child tags that identify a NULL-named ('.') element frame
ELEM_KEYS = ("EvTurn", "EvEID", "ordID", "npid", "desID", "Wpt", "seno2", "ID", "set")
#: NonComplexArray count tags whose elements are a run of `.` scalars in the
#: *same* frame (SAVE_FORMAT.md section 4, "inline count"). Left flat, a single
#: id insertion shifts every following sibling and the differ reports a hundred
#: spurious moves; folded, it reports one list that gained one id.
INLINE_ID_LISTS = ("PlayerIDs", "DesignIDs", "SystemIDs", "FleetIDs",
"ShipIDs", "TradeIDs")
#: master id lists whose *element order* is an artefact of the order in which
#: the turn's commands were applied, not a decision. Compared as sets ONLY
#: under `--relabel-new-ids`, never by default -- see RELABEL_NOTES.
SET_ID_LISTS = ("ShipIDs", "FleetIDs", "DesignIDs")
#: top-level digest rows printed by the default (non-tree) view. Paths that do
#: not exist in a given save are silently skipped.
SUBSYSTEM_PATHS = (
"/Summary",
"/CreateParams",
"/Sim/RNG",
"/Sim/Attrib",
"/Sim/turnstats",
"/Sim/players",
"/Sim/systems",
"/Sim/fleets",
"/Sim/NdGr2",
"/Sim/trdmgr",
"/Sim/spymgr",
"/Sim/SvSctOb",
"/CDT",
)
#: per-player sub-rollups, appended under each player row in --subsystems
PLAYER_PARTS = ("TechTree", "Events", "designs", "Diplo", "spy2", "civr", "comms", "Ojvs")
# --- the digest tree ----------------------------------------------------------
class CkNode:
"""One node of the checksum tree."""
__slots__ = ("path", "label", "kind", "digest", "offset", "size",
"leaves", "value_bytes", "children", "value", "raw", "masked",
"values")
def __init__(self, path, label, kind, digest, offset, size,
leaves, value_bytes, children, value=None, raw=b"", masked=False,
values=None):
self.values = values # inline id list contents, for set-diffing
self.path = path
self.label = label
self.kind = kind
self.digest = digest
self.offset = offset
self.size = size
self.leaves = leaves # number of scalar leaves below (or 1)
self.value_bytes = value_bytes # bytes of *value* payload below (framing excluded)
self.children = children
self.value = value
self.raw = raw
self.masked = masked
@property
def hexd(self) -> str:
return self.digest.hex()
def short(self) -> str:
return self.digest.hex()[:SHORT]
def find(self, path: str):
if self.path == path:
return self
for c in self.children:
if path == c.path or path.startswith(c.path + "/"):
return c.find(path)
return None
def walk(self):
yield self
for c in self.children:
yield from c.walk()
def as_dict(self, depth: int = 99, _d: int = 0) -> dict:
out = {"path": self.path, "kind": self.kind, "digest": self.hexd,
"offset": self.offset, "size": self.size,
"leaves": self.leaves, "valueBytes": self.value_bytes}
if self.masked:
out["masked"] = True
if self.kind != "complex" and self.value is not None:
out["value"] = self.value
if self.children and _d < depth:
out["children"] = [c.as_dict(depth, _d + 1) for c in self.children]
elif self.children:
out["childCount"] = len(self.children)
return out
def _h(*parts: bytes) -> bytes:
d = hashlib.blake2b(digest_size=DIGEST_BYTES)
for p in parts:
d.update(struct.pack("<I", len(p)))
d.update(p)
return d.digest()
def _label(node: sr.Node, index: int, lead: sr.Node | None,
unique: bool = False) -> str:
"""Human, stable, greppable label for one item.
A key is appended only when it carries information: an object id, an object
name, or an element key. A bare positional index is appended only when the
tag is ambiguous among its siblings (`unique=False`), so a uniquely-named
field reads as `Status`, not `Status[36]`.
"""
base = node.name if node.name is not None else "<frame>"
key = None
if lead is not None and isinstance(lead.value, int):
key = str(lead.value)
if node.kind == "complex":
want = NAME_FIELDS.get(node.name or "")
if want:
for c in node.children:
if c.name in want and isinstance(c.value, str):
key = f'{key} "{c.value}"' if key else f'"{c.value}"'
break
if key is None and (node.name in (".", None)):
for c in node.children:
if c.name in ELEM_KEYS:
key = f"{c.name}={c.value}"
break
if key is None:
if unique:
return base
key = str(index)
return f"{base}[{key}]"
def _name_counts(nodes) -> dict:
counts: dict = {}
for n in nodes:
counts[n.name] = counts.get(n.name, 0) + 1
return counts
def _group_children(nodes: list[sr.Node]):
"""Fold object runs and inline id lists into synthetic groups.
Returns a list of entries:
("item", node, None) -- a plain child
("group", name, [(id_node, frame_node), ...]) -- an object table
("list", name, count_node, [value_nodes]) -- an inline id list
Order and byte coverage are preserved exactly; nothing is dropped, so the
reconstruction audit still proves total coverage.
"""
out, i, seen = [], 0, {}
n = len(nodes)
while i < n:
cur = nodes[i]
spec = PAIR_GROUPS.get(cur.name or "")
if (spec and i + 1 < n and nodes[i + 1].name == spec[0]
and nodes[i + 1].kind == "complex" and cur.kind in ("int", "float")):
frame_tag, gname = spec
pairs = []
while (i + 1 < n and nodes[i].name == cur.name
and nodes[i + 1].name == frame_tag and nodes[i + 1].kind == "complex"):
pairs.append((nodes[i], nodes[i + 1]))
i += 2
seen[gname] = seen.get(gname, 0) + 1
label = gname if seen[gname] == 1 else f"{gname}({seen[gname]})"
out.append(("group", label, pairs))
continue
if (cur.name in INLINE_ID_LISTS and cur.kind == "int"
and isinstance(cur.value, int) and 0 <= cur.value <= n - i - 1
and all(nodes[i + 1 + k].name == "." and nodes[i + 1 + k].kind != "complex"
for k in range(cur.value))):
vals = nodes[i + 1:i + 1 + cur.value]
out.append(("list", cur.name, cur, vals))
i += 1 + cur.value
continue
out.append(("item", cur, None))
i += 1
return out
def _value_key(node: sr.Node, path: str, policy: str, masks, hits: dict):
"""(kind, key_bytes, masked) for a scalar/raw leaf."""
kind = node.kind
for tag, under, mkind, mval in masks:
if node.name != tag or not path.startswith(under):
continue
hits[tag] = hits.get(tag, 0) + 1
if mkind == "derived":
return kind, b"<derived>", True
if mkind == "int":
return kind, struct.pack("<i", mval), True
raise ValueError(f"bad mask spec {(tag, under, mkind, mval)!r}")
raw = node.raw
if kind == "float":
return kind, float_key(raw, policy), False
if kind == "bool" and policy == "canonical":
return kind, b"\x01" if any(raw) else b"\x00", False
return kind, raw, False
def _build(node: sr.Node, path: str, label: str, policy: str, masks, hits: dict,
id_lists_as_sets: bool = False) -> CkNode:
if node.kind == "complex":
entries = _group_children(node.children)
counts = _name_counts(e[1] for e in entries if e[0] == "item")
# positional index is the ordinal among *same-named* siblings, so one
# insertion elsewhere in the frame does not renumber everything after it
ordinals: dict = {}
kids: list[CkNode] = []
for ent in entries:
if ent[0] == "item":
child = ent[1]
k = ordinals.get(child.name, 0)
ordinals[child.name] = k + 1
lab = _label(child, k, None, unique=counts.get(child.name) == 1)
kids.append(_build(child, f"{path}/{lab}", lab, policy, masks, hits,
id_lists_as_sets))
elif ent[0] == "list":
_, lname, cnode, vals = ent
# path-scoped like the mask rules (2.5): only the MASTER lists
# directly under /Sim, never a same-named list nested in an
# object. On the corpus that is the only place they occur --
# asserted here so a future save that nests one is not silently
# set-compared.
as_set = (id_lists_as_sets and lname in SET_ID_LISTS
and path == "/Sim")
lpath = f"{path}/{lname}[]" + ("(set)" if as_set else "")
keys = [_value_key(v, lpath, policy, masks, hits)[1] for v in vals]
values = [v.value for v in vals]
if as_set:
# element order is apply order, which is the residue this
# mode exists to quotient out; the multiset is still exact,
# so a genuine insertion/removal still reports.
order = sorted(range(len(vals)), key=lambda i: (str(type(values[i])), values[i]))
keys = [keys[i] for i in order]
values = [values[i] for i in order]
ldig = _h(b"list-set" if as_set else b"list",
lname.encode("ascii"), cnode.raw, *keys)
off = cnode.offset
last = vals[-1] if vals else cnode
kids.append(CkNode(lpath, f"{lname}[]" + ("(set)" if as_set else ""),
"list", ldig, off,
last.offset + last.size - off,
1 + len(vals),
len(cnode.raw) + sum(len(v.raw) for v in vals),
[], values=values))
else:
_, gname, pairs = ent
gpath = f"{path}/{gname}"
gkids = []
for j, (idn, frame) in enumerate(pairs):
lab = _label(frame, j, idn)
ck = _build(frame, f"{gpath}/{lab}", lab, policy, masks, hits,
id_lists_as_sets)
# the id int is part of the object's identity; fold it in
idk = _h(b"id", (idn.name or "").encode("ascii"), idn.raw)
ck.digest = _h(b"obj", idk, ck.digest)
ck.offset = idn.offset
ck.size = frame.offset + frame.size - idn.offset
ck.value_bytes += len(idn.raw)
ck.leaves += 1
gkids.append(ck)
gdig = _h(b"group", gname.encode("ascii"),
*[k.digest for k in gkids])
off = gkids[0].offset if gkids else node.offset
size = (gkids[-1].offset + gkids[-1].size - off) if gkids else 0
kids.append(CkNode(gpath, gname, "group", gdig, off, size,
sum(k.leaves for k in gkids),
sum(k.value_bytes for k in gkids), gkids))
dig = _h(b"frame", (node.name or "").encode("ascii"), *[k.digest for k in kids])
return CkNode(path, label, "complex", dig, node.offset, node.size,
sum(k.leaves for k in kids) or 0,
sum(k.value_bytes for k in kids), kids)
kind, key, masked = _value_key(node, path, policy, masks, hits)
dig = _h(b"leaf", (node.name or "").encode("ascii"), kind.encode("ascii"), key)
return CkNode(path, label, kind, dig, node.offset, node.size, 1, len(node.raw),
[], value=node.value, raw=node.raw, masked=masked)
# --- coverage proof -----------------------------------------------------------
def reconstruct(node: sr.Node, out: bytearray, base: int) -> None:
"""Re-serialise a reader Node back into `out`.
`base` is the inflated offset `out` starts at, so joint padding (which is
computed against the absolute 4-byte grid) lands where it did originally.
"""
def pad():
while (base + len(out)) & 3:
out.append(0)
if node.name is None and node.kind == "raw":
out += node.raw # unnamed payload / stray bytes
return
if node.kind == "complex":
if node.name is not None:
b = node.name.encode("ascii")
out += struct.pack("<i", len(b)) + b
pad()
out += BEGIN
for c in node.children:
reconstruct(c, out, base)
out += END
return
b = (node.name or "").encode("ascii")
out += struct.pack("<i", len(b)) + b
if node.kind == "string":
out += struct.pack("<i", len(node.raw)) # reader's raw excludes the length prefix
out += node.raw
pad()
def audit_coverage(tree: sr.Node, inflated: bytes) -> dict:
"""Re-serialise the whole tree and compare with the inflated stream."""
out = bytearray()
for c in tree.children:
reconstruct(c, out, 0)
ok = bytes(out) == inflated
first = None
if not ok:
n = min(len(out), len(inflated))
for i in range(n):
if out[i] != inflated[i]:
first = i
break
if first is None:
first = n
return {"ok": ok, "rebuiltBytes": len(out), "inflatedBytes": len(inflated),
"firstDiff": first}
# --- top level ----------------------------------------------------------------
class SaveChecksum:
def __init__(self, path, root: CkNode, coverage: dict, res: sr.SaveResult,
policy: str, mask: str, mask_hits: dict | None = None):
self.path = path
self.root = root
self.coverage = coverage
self.res = res
self.policy = policy
self.mask = mask
#: leaves each mask rule actually replaced. Printed on every masked run
#: so an over-broad mask is visible rather than comfortable.
self.mask_hits = mask_hits or {}
@property
def digest(self) -> str:
return self.root.hexd
def subsystems(self) -> list[tuple[str, CkNode]]:
rows = []
for p in SUBSYSTEM_PATHS:
n = self._by_tag(p)
if n is not None:
rows.append((p, n))
return rows
def _by_tag(self, tagpath: str):
"""Resolve a '/Sim/players'-style path against labelled nodes."""
cur = self.root
for seg in tagpath.strip("/").split("/"):
nxt = None
for c in cur.children:
base = c.label.split("[")[0]
if base == seg or c.label == seg:
nxt = c
break
if nxt is None:
return None
cur = nxt
return cur
def as_dict(self, depth: int = 2) -> dict:
return {
"file": self.path,
"root": self.digest,
"policy": {"floats": self.policy, "mask": self.mask,
"digest": f"blake2b-{DIGEST_BYTES * 8}",
"readerFingerprint": READER_FINGERPRINT},
"coverage": self.coverage,
"readerIssues": {lvl: self.res.count(lvl) for lvl in ("error", "warn", "info")},
"maskHits": self.mask_hits,
"subsystems": [{"path": p, "digest": n.hexd, "leaves": n.leaves,
"valueBytes": n.value_bytes} for p, n in self.subsystems()],
"tree": self.root.as_dict(depth),
}
def checksum_result(res: sr.SaveResult, path: str, cov: dict,
floats: str = "bits", mask: str = "none",
extra_masks: tuple = (),
id_lists_as_sets: bool = False) -> SaveChecksum:
"""Digest an already-parsed save.
Split out of `checksum_bytes` so `--relabel-new-ids` can parse once, prove
coverage against the *unmodified* parse, and only then rewrite leaves.
"""
if floats not in FLOAT_POLICIES:
raise ValueError(f"floats must be one of {FLOAT_POLICIES}")
if mask not in MASK_PRESETS:
raise ValueError(f"mask must be one of {tuple(MASK_PRESETS)}")
masks = tuple(MASK_PRESETS[mask]) + tuple(
m for m in extra_masks if m not in MASK_PRESETS[mask])
hits: dict = {}
kids = []
counts = _name_counts(res.tree.children)
ordinals: dict = {}
for c in res.tree.children:
k = ordinals.get(c.name, 0)
ordinals[c.name] = k + 1
lab = _label(c, k, None, unique=counts.get(c.name) == 1)
kids.append(_build(c, "/" + lab, lab, floats, masks, hits, id_lists_as_sets))
# domain separation: a relabelled root must never be confusable with a
# plain one. The extra part is appended ONLY in the new mode, so every
# root recorded before this flag existed is bit-for-bit unchanged.
pol = [b"save", floats.encode("ascii"), mask.encode("ascii")]
if extra_masks or id_lists_as_sets:
pol.append(b"relabel-new-ids")
root_dig = _h(*pol, *[k.digest for k in kids])
root = CkNode("", "<save>", "complex", root_dig, 0, len(res.inflated),
sum(k.leaves for k in kids), sum(k.value_bytes for k in kids), kids)
return SaveChecksum(path, root, cov, res, floats, mask, hits)
def checksum_bytes(data: bytes, path: str = "<bytes>", floats: str = "bits",
mask: str = "none", audit: bool = True,
strict: bool = False, extra_masks: tuple = (),
id_lists_as_sets: bool = False) -> SaveChecksum:
res = sr.read_bytes(data, strict=strict)
cov = audit_coverage(res.tree, res.inflated) if audit else {"ok": None}
return checksum_result(res, path, cov, floats, mask, extra_masks, id_lists_as_sets)
def checksum_save(path: str, **kw) -> SaveChecksum:
with open(path, "rb") as f:
return checksum_bytes(f.read(), path=path, **kw)
# --- diffing ------------------------------------------------------------------
class DiffEntry:
__slots__ = ("path", "kind", "a", "b", "ulps", "note")
def __init__(self, path, kind, a=None, b=None, ulps=None, note=""):
self.path, self.kind, self.a, self.b = path, kind, a, b
self.ulps, self.note = ulps, note
def __repr__(self):
if self.kind == "value":
u = f" [{self.ulps:g} ulp]" if self.ulps is not None else ""
return f"{self.path}: {self.a!r} -> {self.b!r}{u}{self.note}"
if self.kind == "list":
return (f"{self.path}: removed {self.a!r}, added {self.b!r}{self.note}")
return f"{self.path}: {self.kind}{self.note}"
def as_dict(self):
d = {"path": self.path, "kind": self.kind}
if self.kind in ("value", "list", "list-reordered"):
d["a"], d["b"] = self.a, self.b
if self.ulps is not None:
d["ulps"] = self.ulps
if self.note:
d["note"] = self.note.strip()
return d
def diff(a: CkNode, b: CkNode, out: list | None = None, limit: int = 200) -> list:
"""Descend two digest trees, reporting only where they differ."""
out = [] if out is None else out
if len(out) >= limit:
return out
if a.digest == b.digest:
return out
if a.children or b.children:
an = {c.label: c for c in a.children}
bn = {c.label: c for c in b.children}
# positional first: order is part of the state
if [c.label for c in a.children] != [c.label for c in b.children]:
for lab in an:
if lab not in bn:
out.append(DiffEntry(a.path + "/" + lab, "only-in-A"))
for lab in bn:
if lab not in an:
out.append(DiffEntry(b.path + "/" + lab, "only-in-B"))
if not (set(an) - set(bn)) and not (set(bn) - set(an)):
out.append(DiffEntry(a.path, "reordered"))
for ca in a.children:
cb = bn.get(ca.label)
if cb is not None:
diff(ca, cb, out, limit)
return out
if a.kind == "list" and b.kind == "list":
av, bv = a.values or [], b.values or []
added = [v for v in bv if v not in av]
removed = [v for v in av if v not in bv]
if added or removed:
out.append(DiffEntry(a.path, "list", removed, added,
note=f" ({len(av)} -> {len(bv)} entries)"))
else:
out.append(DiffEntry(a.path, "list-reordered", av, bv))
return out
ulps = None
if a.kind == "float" and b.kind == "float" and len(a.raw) == 4 == len(b.raw):
ulps = ulps_apart(a.raw, b.raw)
note = ""
if a.kind != b.kind:
note = f" (kind {a.kind} -> {b.kind})"
av, bv = a.value, b.value
if a.kind == "raw" or b.kind == "raw":
# the reader's raw value is a truncated hex preview; two different blobs
# can preview identically, so report a digest of the whole blob instead
av = f"<raw {len(a.raw)} B {_h(a.raw).hex()[:SHORT]}>"
bv = f"<raw {len(b.raw)} B {_h(b.raw).hex()[:SHORT]}>"
out.append(DiffEntry(a.path, "value", av, bv, ulps, note))
return out
# --- relabelling new client-minted ids ----------------------------------------
#
# Why this exists
# ---------------
# `findings/resolutions/2026-09-09-fleet-id-order-residue.md` section 3 item 1.
# Lane BP ran two processes with *identical pinned AI seeds* on the same
# pre-turn save and got autosaves differing in 35 of 61,147 leaves -- all 35 one
# transposition: fleets 1970 and 1986 exchange their entire contents. The
# resolver decoded the ids (`id-allocation.md`: `id = (counter << 4) | node`)
# and found the ids are NOT the variable: they are one client's own counters
# 123/124/125, minted in that order in every process. What varies is the order
# in which the fleet-assignment pass VISITS the ship groups that need a new
# fleet, so the same group receives a different id. No decision changes.
#
# This mode makes the comparison blind to that labelling and to nothing else.
#
# The guards matter more than the feature (rule 1: a green verdict on a hook
# that may be comparing nothing is the failure this campaign was built around).
# There are five, and every one of them REFUSES rather than degrades:
#
# G1 only ids ABSENT from the pre-turn save may be relabelled;
# G2 only ids with a non-zero node nibble (client-minted) may be relabelled;
# G3 pi must be a permutation of ONE set -- the new ids of one node -- so the
# two saves must have minted the same ids; different id sets is a real
# divergence (a different number of allocations) and is never absorbed;
# G4 the content keys must correspond one-to-one, and be unique on each side;
# G5 every leaf anywhere in either save that holds a permuted id must sit at
# a reference site this module models. An unmodelled site would mean the
# rewrite is incomplete, which could make two different states compare
# equal. Finding one is a refusal, not a warning.
#
# A refusal never falls back to a partial relabelling. The CLI prints the
# reason and then runs the ORDINARY comparison, which is the conservative
# direction: whatever really differs still reports as DIVERGED.
class RelabelRefused(Exception):
"""A relabelling could not be built as a full, guarded bijection."""
def id_node(i: int) -> int:
"""The node nibble of an object id (`id-allocation.md` section 1).
0 is the server / the sim that owns the board; `PlyrIdx + 1` is a player's
own client. Only non-zero nibbles are ever relabelled here.
"""
return i & 0xF
#: Leaf tags that hold a *fleet* id, with where they are allowed to appear.
#: `FltID` is both the master fleet table's lead int (the slot, never rewritten)
#: and a ship's back-reference to its fleet (rewritten); `Flt` is a system's
#: list of the fleets standing at it (rewritten). Anything else is G5.
FLEET_SLOT_TAG = "FltID"
FLEET_REF_TAGS = ("FltID", "Flt")
RELABEL_MASK = (("Checksum", "/Summary/", "derived", None),)
RELABEL_NOTES = (
"/Summary/Checksum is MASKED: it is derived from the whole state and its "
"inputs are unmodelled (rule 18's open item), so it cannot be recomputed "
"under a relabelling and its move is not independent evidence. Unmask by "
"dropping --relabel-new-ids.",
"Master id lists (%s) are compared as SETS: their element order is the "
"order the turn's commands were applied in, which is the same visit-order "
"residue. Membership is still exact -- an added or removed id still "
"reports." % "/".join(SET_ID_LISTS),
)
class Relabelling:
"""The bijection pi, plus everything a reader needs to audit it."""
__slots__ = ("pi", "nodes", "new_ids", "keys", "rewrites", "moves")
def __init__(self, pi, nodes, new_ids, keys):
self.pi = pi # A-id -> B-id, a permutation of new_ids
self.nodes = nodes # node nibbles it touches
self.new_ids = new_ids # the new non-zero-node fleet ids
self.keys = keys # A-id -> content key, for the report
self.rewrites: dict = {} # tag -> leaves rewritten
self.moves = 0 # fleet bodies exchanged
@property
def moved(self) -> dict:
return {k: v for k, v in self.pi.items() if k != v}
def __bool__(self) -> bool:
return bool(self.moved)
def format_permutation(pi: dict) -> str:
"""`{1970<->1986}`, or cycle notation for anything longer."""
moved = {k: v for k, v in pi.items() if k != v}
if not moved:
return "{} (identity -- no new id changed hands)"
seen, cycles = set(), []
for k in sorted(moved):
if k in seen:
continue
cyc, cur = [k], moved[k]
seen.add(k)
while cur != k:
cyc.append(cur)
seen.add(cur)
cur = moved[cur]
cycles.append(cyc)
parts = ["<->".join(str(x) for x in c) if len(c) == 2
else "->".join(str(x) for x in c + [c[0]]) for c in cycles]
return "{" + ", ".join(parts) + "}"
def _sim_frame(tree: sr.Node) -> sr.Node:
for c in tree.children:
if c.name == "Sim" and c.kind == "complex":
return c
raise RelabelRefused("no /Sim frame in this save -- not a game state")
def _fleet_slots(tree: sr.Node) -> list:
"""[(lead_id_node, frame_node)] for the master fleet table."""
sim = _sim_frame(tree)
ch = sim.children
out = []
for i in range(len(ch) - 1):
if (ch[i].name == FLEET_SLOT_TAG and ch[i].kind == "int"
and ch[i + 1].name == "Flt" and ch[i + 1].kind == "complex"):
out.append((ch[i], ch[i + 1]))
return out
def _fleet_content_key(frame: sr.Node):
"""An identity for a fleet that does NOT mention its own id.
Resolver section 3.1: `(LocID or FPlan destination, sorted ship-id set)`.
`LocID` and the plan destination are tagged apart, so a fleet standing at
system 384 never keys the same as one bound for 384.
"""
loc, dest, ships = None, None, []
for c in frame.children:
if c.name == "LocID" and c.kind == "int":
loc = c.value
elif c.name == "ShipID" and c.kind == "int":
ships.append(c.value)
elif c.name == "FPlan" and c.kind == "complex":
for g in c.children:
if g.name == "pnd" and g.kind == "int":
dest = g.value
if loc:
where = ("at", loc)
elif dest is not None:
where = ("bound-for", dest)
else:
where = ("nowhere", None)
return (where, tuple(sorted(ships)))
def _fleet_name(frame: sr.Node):
for c in frame.children:
if c.name == "FtName":
return c.value
return None
def _id_occurrences(tree: sr.Node, wanted: set) -> list:
"""Every leaf in `tree` holding one of `wanted`, classified by site.
Matching is on the leaf's **raw four bytes**, not on the reader's typed
value, deliberately: the digest is a function of (save bytes, reader
schema), and a leaf the schema happens to type as a float or a raw blob
would be invisible to a value-based scan while still carrying an id the
rewrite must reach. A completeness guard that a typing slip can silently
empty is exactly the "green verdict over an empty region set" of rule 1.
Returns [(site, path, node)] where site is one of:
"slot" -- the master fleet table's lead FltID (identifies the slot)
"idlist" -- an entry of the inline `FleetIDs` master list
"ref:<tag>" -- a reference that must be rewritten under pi
"UNMODELLED"-- anything else; this is guard G5 and it refuses
"""
by_raw = {struct.pack("<i", v): v for v in wanted}
sim = _sim_frame(tree)
slot_leads = {id(n) for n, _ in _fleet_slots(tree)}
idlist_entries = set()
ch = sim.children
for i, c in enumerate(ch):
if c.name in INLINE_ID_LISTS and c.kind == "int" and isinstance(c.value, int):
for k in range(min(c.value, len(ch) - i - 1)):
idlist_entries.add(id(ch[i + 1 + k]))
out = []
def walk(n, path):
for c in n.children:
p = f"{path}/{c.name if c.name is not None else '.'}"
if c.kind == "complex":
walk(c, p)
continue
if c.raw not in by_raw:
continue
if id(c) in slot_leads:
site = "slot"
elif id(c) in idlist_entries:
site = "idlist"
elif c.name in FLEET_REF_TAGS:
site = f"ref:{c.name}"
else:
site = "UNMODELLED"
out.append((site, p, c))
walk(tree, "")
return out
def build_relabelling(pre_tree: sr.Node, a_tree: sr.Node, b_tree: sr.Node) -> Relabelling:
"""Build pi between the ids new in A and the ids new in B, or refuse.
Nothing is mutated here. Every guard is checked before any rewrite can
happen, so a refusal leaves both saves exactly as parsed.
"""
pre_ids = {n.value for n, _ in _fleet_slots(pre_tree)}
A = {n.value: f for n, f in _fleet_slots(a_tree)}
B = {n.value: f for n, f in _fleet_slots(b_tree)}
# G1 + G2: new, and client-minted
new_a = {i for i in A if i not in pre_ids and id_node(i) != 0}
new_b = {i for i in B if i not in pre_ids and id_node(i) != 0}
# G3: pi permutes ONE set. Unequal id sets means the two runs minted
# different counters -- a different number of allocations, which is a real
# divergence and is never something this mode may absorb.
if new_a != new_b:
raise RelabelRefused(
"the new client-minted fleet ids differ between the two saves -- "
f"A minted {sorted(new_a)}, B minted {sorted(new_b)}"
+ (f"; only in A {sorted(new_a - new_b)}" if new_a - new_b else "")
+ (f"; only in B {sorted(new_b - new_a)}" if new_b - new_a else "")
+ ". That is a difference in what was allocated, not in how it "
"was labelled, so no relabelling is defensible")
new_ids = new_a
pi: dict = {}
keys: dict = {}
for node in sorted({id_node(i) for i in new_ids}):
ids = sorted(i for i in new_ids if id_node(i) == node)
ka: dict = {}
kb: dict = {}
for side, tbl, dst in (("A", A, ka), ("B", B, kb)):
for i in ids:
key = _fleet_content_key(tbl[i])
if key in dst:
raise RelabelRefused(
f"two new fleets in {side} share the content key {key!r} "
f"(ids {dst[key]} and {i}) -- the match would be "
"ambiguous, so no bijection can be built")
dst[key] = i
if set(ka) != set(kb):
only_a = [f"{ka[k]}:{k}" for k in sorted(set(ka) - set(kb), key=repr)]
only_b = [f"{kb[k]}:{k}" for k in sorted(set(kb) - set(ka), key=repr)]
raise RelabelRefused(
f"node {node}: the new fleets do not correspond by content. "
f"Unmatched in A: {only_a}; unmatched in B: {only_b}. Their "
"contents differ, which is a real divergence")
for key, i in ka.items():
pi[i] = kb[key]
keys[i] = key
# names are minted from the same per-fleet ordinal counter as the id
# (Sigma/Tau/Upsilon...), so they are id-attached labels, not content.
# They may only be treated that way if they DO track the id in both
# saves; if they do not, something real differs and we refuse.
for i in ids:
na, nb = _fleet_name(A[i]), _fleet_name(B[i])
if na != nb:
raise RelabelRefused(
f"fleet {i} is named {na!r} in A and {nb!r} in B -- the "
"name does not track the id, so it is not an id-attached "
"label here and relabelling would hide a real difference")
rl = Relabelling(pi, sorted({id_node(i) for i in new_ids}), sorted(new_ids), keys)
# G3 again, on the built map: a permutation of exactly the new ids
if sorted(pi) != sorted(pi.values()):
raise RelabelRefused(
f"pi is not a permutation: domain {sorted(pi)} != image "
f"{sorted(pi.values())}")
for k, v in pi.items():
if id_node(k) != id_node(v):
raise RelabelRefused(
f"pi would map {k} (node {id_node(k)}) to {v} (node "
f"{id_node(v)}) -- ids may only be relabelled within one node")
if k in pre_ids or v in pre_ids:
raise RelabelRefused(
f"pi touches {k if k in pre_ids else v}, which is present in "
"the pre-turn save -- only ids created on this turn may be "
"relabelled")
# G5: no unmodelled reference site, in EITHER save
touched = set(pi) | set(pi.values())
for name, tree in (("A", a_tree), ("B", b_tree)):
for site, path, _n in _id_occurrences(tree, touched):
if site == "UNMODELLED":
raise RelabelRefused(
f"{name} holds a relabelled id at {path}, which this tool "
"does not model as a fleet reference. Rewriting only the "
"sites it knows would leave the two states comparable at a "
"leaf that still carries the old label -- add the tag to "
"FLEET_REF_TAGS after checking what it means")
return rl
def apply_relabelling(tree: sr.Node, rl: Relabelling) -> Relabelling:
"""Rewrite `tree` in place so it reads as if it had minted B's labelling.
Two operations, both a function of pi alone:
* the permuted fleets' **bodies** are exchanged between their slots, while
each slot keeps its own id and its id-attached label (`FtName`). Doing
it this way -- rather than renumbering the slots -- means the master
fleet table's element order, the `FleetIDs[]` list and every object
label stay exactly as the file had them, so nothing but the intended
quotient is taken.
* every *reference* to a permuted id is rewritten to pi(it).
Node `offset`/`size` are left stale on moved bodies; they are reporting
aids in `--tree`/`--json`, not digest inputs. The digest is a function of
(tag, kind, value bytes) only.
"""
if not rl.pi:
return rl
slots = {n.value: f for n, f in _fleet_slots(tree)}
inv = {v: k for k, v in rl.pi.items()}
old_children = {i: list(slots[i].children) for i in rl.pi}
own_name = {}
for i in rl.pi:
own_name[i] = [c for c in slots[i].children if c.name == "FtName"]
for dst in rl.pi.values():
src = inv[dst]
if src == dst:
continue
kept = list(own_name[dst])
kids = []
for c in old_children[src]:
if c.name == "FtName" and kept:
kids.append(kept.pop(0))
else:
kids.append(c)
slots[dst].children = kids
rl.moves += 1
by_raw = {struct.pack("<i", v): v for v in rl.pi}
for site, _path, node in _id_occurrences(tree, set(rl.pi)):
if not site.startswith("ref:"):
continue
old = by_raw[node.raw]
new = rl.pi[old]
if new == old: # a fixed point of pi rewrites nothing
continue
node.raw = struct.pack("<i", new)
if isinstance(node.value, int):
node.value = new
rl.rewrites[site[4:]] = rl.rewrites.get(site[4:], 0) + 1
return rl
# --- chain --------------------------------------------------------------------
def build_chain(files: list[str], **kw) -> dict:
"""Record a chain: one entry per save, in turn order."""
entries = []
for f in files:
ck = checksum_save(f, **kw)
t = ck.res.typed.get("summary") or {}
entries.append({
"name": os.path.basename(f),
"turn": t.get("Turn"),
"root": ck.digest,
"coverage": bool(ck.coverage.get("ok")),
"subsystems": {p: n.hexd for p, n in ck.subsystems()},
})
return {"format": "sots-state-chain/1",
"policy": {"floats": kw.get("floats", "bits"),
"mask": kw.get("mask", "none"),
"digest": f"blake2b-{DIGEST_BYTES * 8}",
"readerFingerprint": READER_FINGERPRINT},
"turns": entries}
def verify_chain(chain: dict, files: list[str], **kw) -> tuple[bool, list[str]]:
msgs, ok = [], True
pol = chain.get("policy") or {}
for field, got in (("floats", kw.get("floats", "bits")),
("mask", kw.get("mask", "none"))):
if pol.get(field) not in (None, got):
msgs.append(f"!! chain was recorded with {field}={pol[field]!r}, "
f"verifying with {got!r} -- roots are not comparable")
ok = False
fp = pol.get("readerFingerprint")
if fp and fp != READER_FINGERPRINT:
msgs.append(f"!! chain was recorded under save_reader {fp}, this is "
f"{READER_FINGERPRINT} -- re-record before trusting a DIVERGE")
rec = chain["turns"]
if len(rec) != len(files):
msgs.append(f"chain has {len(rec)} turns, {len(files)} saves given")
ok = False
for i, f in enumerate(files):
if i >= len(rec):
break
ck = checksum_save(f, **kw)
want = rec[i]
if ck.digest == want["root"]:
msgs.append(f"turn {want.get('turn')} MATCH {ck.digest[:SHORT]} {os.path.basename(f)}")
continue
ok = False
msgs.append(f"turn {want.get('turn')} DIVERGE {ck.digest[:SHORT]} != {want['root'][:SHORT]}"
f" {os.path.basename(f)}")
for p, n in ck.subsystems():
w = want["subsystems"].get(p)
if w and w != n.hexd:
msgs.append(f" subsystem {p}: {w[:SHORT]} -> {n.hexd[:SHORT]}")
msgs.append(" (re-run with the recorded save to localise to an object)")
break # first divergent turn is the one that matters
return ok, msgs
# --- CLI ----------------------------------------------------------------------
def _mask_note(ck: "SaveChecksum") -> str:
"""Always say what a mask actually hit -- a mask nobody audits is a hiding place."""
if ck.mask == "none":
return ""
if not ck.mask_hits:
return " [mask matched NOTHING -- check the rule paths]"
return " [masked: " + ", ".join(f"{k}x{v}" for k, v in sorted(ck.mask_hits.items())) + "]"
def _relabelled_pair(pre_path: str, a_path: str, b_path: str,
floats: str = "bits", mask: str = "none",
audit: bool = True, strict: bool = False):
"""Parse three saves, build pi, and digest A relabelled against B.
On a refusal nothing is rewritten and the ORDINARY comparison is returned
instead, so a real divergence survives. Returns (a, b, relabelling|None,
refusal_message|None).
"""
def read(p):
with open(p, "rb") as f:
res = sr.read_bytes(f.read(), strict=strict)
cov = audit_coverage(res.tree, res.inflated) if audit else {"ok": None}
return res, cov
res_p, _ = read(pre_path)
res_a, cov_a = read(a_path)
res_b, cov_b = read(b_path)
try:
rl = build_relabelling(res_p.tree, res_a.tree, res_b.tree)
except RelabelRefused as exc:
a = checksum_result(res_a, a_path, cov_a, floats, mask)
b = checksum_result(res_b, b_path, cov_b, floats, mask)
return a, b, None, str(exc)
apply_relabelling(res_a.tree, rl)
a = checksum_result(res_a, a_path, cov_a, floats, mask,
extra_masks=RELABEL_MASK, id_lists_as_sets=True)
b = checksum_result(res_b, b_path, cov_b, floats, mask,
extra_masks=RELABEL_MASK, id_lists_as_sets=True)
return a, b, rl, None
def _relabel_report(rl, refusal: str | None, pre_path: str) -> list:
if refusal is not None:
return [
f"relabel: pre-turn save {pre_path}",
"relabel: REFUSED -- " + refusal,
"relabel: NO relabelling was applied; the comparison below is the "
"ordinary one, and nothing has been absorbed.",
]
lines = [f"relabel: pre-turn save {pre_path}",
f"relabel: pi = {format_permutation(rl.pi)}"]
for i in rl.new_ids:
where, ships = rl.keys[i]
lines.append(f"relabel: {i} (counter {i >> 4}, node {id_node(i)}) "
f"-> {rl.pi[i]} {where[0]} {where[1]}, ships {list(ships)}")
lines.append(f"relabel: {len(rl.new_ids)} new client-minted fleet id(s) on "
f"node(s) {rl.nodes}; {len(rl.moved)} relabelled, "
f"{rl.moves} fleet body/bodies exchanged, "
+ (", ".join(f"{k}x{v}" for k, v in sorted(rl.rewrites.items()))
or "no references") + " rewritten")
for note in RELABEL_NOTES:
lines.append("relabel: " + note)
return lines
def _relabel_dict(rl, refusal: str | None, pre_path: str) -> dict:
if refusal is not None:
return {"preTurn": pre_path, "applied": False, "refused": refusal}
return {"preTurn": pre_path, "applied": True,
"pi": {str(k): v for k, v in rl.pi.items()},
"moved": {str(k): v for k, v in rl.moved.items()},
"newIds": rl.new_ids, "nodes": rl.nodes,
"contentKeys": {str(k): repr(v) for k, v in rl.keys.items()},
"bodiesExchanged": rl.moves, "referencesRewritten": rl.rewrites,
"notes": list(RELABEL_NOTES)}
def _print_tree(n: CkNode, depth: int, out, indent: int = 0, top: bool = True):
if not top:
pad = " " * indent
extra = f" {n.leaves} leaves, {n.value_bytes} B" if n.kind in ("complex", "group") else ""
val = "" if n.kind in ("complex", "group") else f" = {n.value!r}"
mk = " [masked]" if n.masked else ""
out.append(f"{pad}{n.short()} {n.label}{val}{extra}{mk}")
if indent < depth:
for c in n.children:
_print_tree(c, depth, out, indent + 1, top=False)
def main(argv=None) -> int:
ap = argparse.ArgumentParser(
description="whole-state diagnostic checksum for SOTS1 saves")
ap.add_argument("saves", nargs="*", help="one save (report), two (diff), or N with --chain")
ap.add_argument("--floats", choices=FLOAT_POLICIES, default="bits",
help="float hashing policy (default: bits = exact IEEE-754)")
ap.add_argument("--mask", choices=tuple(MASK_PRESETS), default="none",
help="canonicalisation preset (default: none, so deltas localise)")
ap.add_argument("--tree", action="store_true", help="print the digest tree")
ap.add_argument("--depth", type=int, default=2, help="tree depth (default 2)")
ap.add_argument("--json", action="store_true")
ap.add_argument("--no-audit", action="store_true",
help="skip the byte-for-byte coverage proof (faster, weaker)")
ap.add_argument("--strict", action="store_true", help="reader --strict")
ap.add_argument("--ulps", type=float, default=None,
help="in a diff, flag float differences at or below N ULPs")
ap.add_argument("--limit", type=int, default=200, help="max diff entries")
ap.add_argument("--relabel-new-ids", metavar="PRE-TURN.SAV", default=None,
help="compare two post-turn saves modulo the labelling of "
"the client-minted ids created on this turn, given the "
"pre-turn save they both came from")
ap.add_argument("--chain", metavar="FILE", help="chain JSON to verify against")
ap.add_argument("--record-chain", metavar="FILE", help="write a chain JSON instead")
args = ap.parse_args(argv)
kw = dict(floats=args.floats, mask=args.mask,
audit=not args.no_audit, strict=args.strict)
if not args.saves:
ap.error("no saves given")
if args.relabel_new_ids and (len(args.saves) != 2 or args.chain or args.record_chain):
ap.error("--relabel-new-ids compares exactly two post-turn saves; it is "
"a property of a pair, not of a single save or a chain")
if args.record_chain:
chain = build_chain(args.saves, **kw)
with open(args.record_chain, "w") as f:
json.dump(chain, f, indent=1)
print(f"recorded {len(chain['turns'])} turns -> {args.record_chain}")
for t in chain["turns"]:
cov = "cov-ok" if t["coverage"] else "COV-FAIL"
print(f" turn {t['turn']} {t['root'][:SHORT]} {cov} {t['name']}")
return 0 if all(t["coverage"] for t in chain["turns"]) else 1
if args.chain:
with open(args.chain) as f:
chain = json.load(f)
ok, msgs = verify_chain(chain, args.saves, **kw)
print("\n".join(msgs))
return 0 if ok else 1
if len(args.saves) == 1:
ck = checksum_save(args.saves[0], **kw)
if args.json:
print(json.dumps(ck.as_dict(args.depth if args.tree else 2), indent=1))
return 0 if ck.coverage.get("ok") is not False else 1
cov = ck.coverage
covs = ("coverage: PROVED (%d bytes rebuilt == inflated)" % cov["rebuiltBytes"]
if cov.get("ok") else
"coverage: SKIPPED" if cov.get("ok") is None else
"coverage: FAILED at inflated offset 0x%x" % (cov.get("firstDiff") or 0))
print(f"file: {ck.path}")
print(f"root: {ck.digest}")
print(f"policy: floats={ck.policy} mask={ck.mask} "
f"digest=blake2b-{DIGEST_BYTES * 8} reader={READER_FINGERPRINT}"
f"{_mask_note(ck)}")
print(f"{covs}; {ck.root.leaves} leaves, {ck.root.value_bytes} value bytes")
print(f"reader: {ck.res.count('error')} error, {ck.res.count('warn')} warn")
if args.tree:
lines: list[str] = []
_print_tree(ck.root, args.depth, lines)
print("\n".join(lines))
else:
print("subsystems:")
for p, n in ck.subsystems():
print(f" {n.short()} {p:<22} {n.leaves:>6} leaves {n.value_bytes:>8} B")
return 0 if cov.get("ok") is not False else 1
if len(args.saves) != 2:
ap.error("give one save, two saves, or use --chain / --record-chain")
rl = None
refusal = None
if args.relabel_new_ids:
a, b, rl, refusal = _relabelled_pair(
args.relabel_new_ids, args.saves[0], args.saves[1], **kw)
else:
a = checksum_save(args.saves[0], **kw)
b = checksum_save(args.saves[1], **kw)
entries = diff(a.root, b.root, limit=args.limit)
if args.ulps is not None:
for e in entries:
if e.kind == "value" and e.ulps is not None and e.ulps <= args.ulps:
e.note += f" <= {args.ulps:g} ULP"
if args.json:
out = {"a": a.as_dict(0), "b": b.as_dict(0),
"identical": a.digest == b.digest,
"diffs": [e.as_dict() for e in entries]}
if args.relabel_new_ids:
out["relabel"] = _relabel_dict(rl, refusal, args.relabel_new_ids)
print(json.dumps(out, indent=1))
return 0 if a.digest == b.digest else 1
print(f"A {a.digest} {a.path}")
print(f"B {b.digest} {b.path}")
print(f"policy: floats={a.policy} mask={a.mask} reader={READER_FINGERPRINT}"
f"{_mask_note(a)}")
if args.relabel_new_ids:
for line in _relabel_report(rl, refusal, args.relabel_new_ids):
print(line)
for ck, nm in ((a, "A"), (b, "B")):
if ck.coverage.get("ok") is False:
print(f"!! {nm}: coverage FAILED at 0x{ck.coverage.get('firstDiff') or 0:x} "
f"-- the digest does not bound this file")
if a.digest == b.digest:
if rl is not None:
print(f"IDENTICAL modulo pi = {format_permutation(rl.pi)}")
else:
print("IDENTICAL")
return 0
modulo = f" modulo pi = {format_permutation(rl.pi)}" if rl is not None else ""
print(f"DIVERGED{modulo}: {len(entries)} leaf difference(s)"
f"{' (limit reached)' if len(entries) >= args.limit else ''}")
for e in entries:
print(f" {e}")
if args.ulps is not None:
flt = [e for e in entries if e.kind == "value" and e.ulps is not None]
near = [e for e in flt if e.ulps <= args.ulps]
print(f"floats: {len(flt)} differ, {len(near)} within {args.ulps:g} ULP")
return 1
if __name__ == "__main__":
sys.exit(main())
Reference in a new issue View git blame Copy permalink