dashboard: coverage generator + first render
This commit is contained in:
alex
parent
d0cfcadce8
commit
6095c0388d
3 changed files with 697 additions and 0 deletions
94
campaign/DASHBOARD.md
Normal file
94
campaign/DASHBOARD.md
Normal file
|
|
@ -0,0 +1,94 @@
|
||||||
|
# SotS RE campaign — coverage dashboard
|
||||||
|
|
||||||
|
Generated 2026-09-07 21:35 UTC · `sots-re` @ d0cfcad,2026-09-07 · `sots-engine` @ 66d4bda,2026-09-07 (12 commits) · regenerate with `tools/dashboard.py`
|
||||||
|
|
||||||
|
> **North star:** A functional reimplementation of the engine — behavior-equivalent, NOT byte-for-byte
|
||||||
|
|
||||||
|
## 1. Map coverage (campaign/board.md)
|
||||||
|
|
||||||
|
43 targets · mapped-or-better **28/43** `[███████░░░] 65%` · verified **12/43** `[███░░░░░░░] 28%`
|
||||||
|
|
||||||
|
| Status | Count | % |
|
||||||
|
|---|---:|---:|
|
||||||
|
| verified | 12 | 28% |
|
||||||
|
| mapped | 16 | 37% |
|
||||||
|
| in-progress | 7 | 16% |
|
||||||
|
| backlog | 7 | 16% |
|
||||||
|
| blocked | 1 | 2% |
|
||||||
|
|
||||||
|
| Type | verified | mapped | in-progress | backlog | blocked | total |
|
||||||
|
|---|---:|---:|---:|---:|---:|---:|
|
||||||
|
| objects | 5 | 2 | 0 | 0 | 0 | 7 |
|
||||||
|
| control-flow | 0 | 2 | 0 | 0 | 0 | 2 |
|
||||||
|
| subsystems | 1 | 5 | 1 | 3 | 1 | 11 |
|
||||||
|
| engine | 2 | 1 | 4 | 0 | 0 | 7 |
|
||||||
|
| verify | 2 | 2 | 1 | 0 | 0 | 5 |
|
||||||
|
| phase2 | 1 | 0 | 0 | 4 | 0 | 5 |
|
||||||
|
| meta | 1 | 4 | 1 | 0 | 0 | 6 |
|
||||||
|
|
||||||
|
## 2. Binary understanding
|
||||||
|
|
||||||
|
- RTTI type descriptors: **1,924** (`Game::` 1,404, `Mars::` 194; serializable types 179)
|
||||||
|
- Classes with recovered member layouts: **21** / 179 serializable types `[█░░░░░░░░░] 12%` — heuristic: distinct `Game::X`/`Mars::X` in `##`–`####` headings of `struct-recovery.md` + `schema-gaps-resolved.md`
|
||||||
|
- Functions: **41,411** (parsed from `01-fingerprint.md`); named/annotated in the **address contract** (`ghidra/addresses.json`, not Ghidra's full rename count): **25**, verified **10** `[████░░░░░░] 40%`
|
||||||
|
|
||||||
|
## 3. Data layer
|
||||||
|
|
||||||
|
- Catalogs: **1,595/1,595** files parsed (91 block kinds in `schema_stats.json`), dangling cross-refs **0** (`crosslink.json`)
|
||||||
|
- Oracle `mars-parse`: **1,531/1,531** files agree `[██████████] 100%`
|
||||||
|
- Oracle `mars-text`: **64/64** files agree `[██████████] 100%`
|
||||||
|
- Saves: **3/3** real saves strict-clean — strict exit 0, 0 errors, 0 warnings
|
||||||
|
- Design rules: **127/127** stock designs pass `[██████████] 100%`
|
||||||
|
|
||||||
|
## 4. Engine accrual (sots-engine)
|
||||||
|
|
||||||
|
| Module | LOC | Test files | Checks | Wired | Doc |
|
||||||
|
|---|---:|---:|---:|---|---|
|
||||||
|
| `game/sim` | 1,342 | 8 | 356 | yes | game-sim.md |
|
||||||
|
| `mars/parse` | 750 | 11 | 188 | yes | mars-parse.md |
|
||||||
|
| `mars/text` | 855 | 8 | 205 | yes | mars-text.md |
|
||||||
|
| `shim` | 184 | 0 | 0 | direct (WIN32) | M0.md |
|
||||||
|
| **total** | **3,131** | **27** | **749** | | |
|
||||||
|
|
||||||
|
Board `engine:` rows: verified **2**, mapped 1, in flight 4 (of 7) — verified & merged `[███░░░░░░░] 29%`
|
||||||
|
|
||||||
|
## 5. Phase 2 milestones
|
||||||
|
|
||||||
|
| Milestone | Status | Coverage | Notes |
|
||||||
|
|---|---|---:|---|
|
||||||
|
| P2-M0 shim bootstrap | ✅ verified | 100% | sots-engine 5f5bc41: proxy binkw32.dll (66 exports identical), MinHook, asm-stub trace hoo |
|
||||||
|
| P2-M1 flat-KV config loader | ⬜ backlog | 0% | CfgVar_RegisterKey 0x008b76a0; trace->compare->replace; save byte-identical oracle |
|
||||||
|
| P2-M2 manifest/id registry | ⬜ backlog | 0% | _weapons.txt/_shipsections.txt stable ids |
|
||||||
|
| P2-M3 Mars brace-block parser | ⬜ backlog | 0% | + .effect TXT reader; compare over full catalog load |
|
||||||
|
| P2-M4 gobio VFS read | ⬜ backlog | 0% | ZIP entries + NativeFileSystem override order |
|
||||||
|
|
||||||
|
## 6. Verification ledger
|
||||||
|
|
||||||
|
- ✅ Saves strict: 3/3 (strict exit 0, 0 errors, 0 warnings)
|
||||||
|
- ✅ Design rules: 127/127
|
||||||
|
- ✅ oracle mars-parse 1531/1531 · ✅ oracle mars-text 64/64
|
||||||
|
- ✅ Compare harness present (`verify/harness/compare/`)
|
||||||
|
- ✅ M0 evidence present (`verify/results/shim/m0.log`)
|
||||||
|
- 🔄 Determinism oracle: in-progress
|
||||||
|
|
||||||
|
## 7. Open questions
|
||||||
|
|
||||||
|
Open **20** · resolved/parked 4 · backlog items: Now 4, Next 3, Later 2, Breadth queue 7, Parked 1
|
||||||
|
|
||||||
|
Most recent open:
|
||||||
|
|
||||||
|
- Formula gaps surfaced by the game/sim port (feed back to Ghidra) — which bankruptcy limit carries …
|
||||||
|
- `Mars::Application::Initialize` signature (P2-M0 finding) — a C++ `__thiscall` detour that called …
|
||||||
|
- Corrections from turn internals — `ServerPlayer+0xf9` is the AI flag (not bTurnDone); `FUN_007d7f7…
|
||||||
|
- Ship-design code rules (Ghidra, from SHIP_DESIGN_RULES.md §8) — hidden default rider designs are b…
|
||||||
|
- Strict-parse gaps vs real save (verifier output, round 3) — (1) `ServerSystem`: 7/28 systems carry…
|
||||||
|
|
||||||
|
## 8. Delta since previous dashboard
|
||||||
|
|
||||||
|
- verified targets: 12 → 12 (+0) · mapped-or-better: 28 → 28 (+0)
|
||||||
|
- engine LOC: 3,131 → 3,131 (+0) · test files: 27 → 27 (+0) · checks: 749 → 749 (+0)
|
||||||
|
- addresses verified: 10 → 10 (+0) · recovered layouts: 21 → 21 (+0) · open questions: 20 → 20 (+0)
|
||||||
|
|
||||||
|
---
|
||||||
|
warnings: none
|
||||||
|
<!-- dashboard-metrics {"verified": 12, "mapped_plus": 28, "targets": 43, "loc": 3131, "tests": 27, "checks": 749, "addr_verified": 10, "addr_total": 25, "layouts": 21, "open_q": 20} -->
|
||||||
110
tools/DASHBOARD_README.md
Normal file
110
tools/DASHBOARD_README.md
Normal file
|
|
@ -0,0 +1,110 @@
|
||||||
|
# `tools/dashboard.py` — what every number means
|
||||||
|
|
||||||
|
Generates `campaign/DASHBOARD.md` from the campaign tracker, the findings, the verify
|
||||||
|
artefacts and the `sots-engine` checkout. Pure Python 3 stdlib (`/usr/bin/python3`), no
|
||||||
|
Ghidra, no network. Idempotent: run it after every integration; it overwrites the
|
||||||
|
dashboard and computes the delta against the previous one before doing so.
|
||||||
|
|
||||||
|
```
|
||||||
|
tools/dashboard.py # write campaign/DASHBOARD.md
|
||||||
|
tools/dashboard.py --print # also echo to stdout
|
||||||
|
tools/dashboard.py --no-write # render to stdout only, touch nothing
|
||||||
|
tools/dashboard.py --engine PATH # sots-engine checkout (default ~/sots-engine, or $SOTS_ENGINE)
|
||||||
|
```
|
||||||
|
|
||||||
|
Nothing is fatal: an unreadable or malformed input produces a `?` or a missing row and an
|
||||||
|
entry in the footer `warnings:` line. Check that line first when a number looks wrong.
|
||||||
|
|
||||||
|
## Header
|
||||||
|
|
||||||
|
| Item | Source |
|
||||||
|
|---|---|
|
||||||
|
| Generated timestamp | wall clock, UTC |
|
||||||
|
| `sots-re` commit | `git log -1 --format=%h,%cs` in this repo |
|
||||||
|
| `sots-engine` commit + count | same in the engine checkout; count = `git rev-list --count HEAD` |
|
||||||
|
| North star | bold sentence under `## North star` in `findings/00-strategy.md` |
|
||||||
|
|
||||||
|
## 1. Map coverage
|
||||||
|
|
||||||
|
Source: `campaign/board.md`, every line starting with `|` except the header and separator.
|
||||||
|
Rows are split on `|` for the first six cells only (`Target, Type, Status, Conf, Coverage,
|
||||||
|
Updated`); everything after is the Notes cell, so backticks or pipes in notes cannot break a
|
||||||
|
row. Rows with fewer than seven cells are skipped with a warning; unknown statuses are
|
||||||
|
counted but warned.
|
||||||
|
|
||||||
|
- **mapped-or-better** = `mapped` + `verified` (verified implies mapped).
|
||||||
|
- Type breakdown maps board `Type` values to rows: `object → objects`, `control-flow`,
|
||||||
|
`subsystem → subsystems`, `engine`, `verify`, `phase2`, `meta`. Any other type appears as an
|
||||||
|
`other` row plus a warning.
|
||||||
|
|
||||||
|
## 2. Binary understanding
|
||||||
|
|
||||||
|
| Number | Source / heuristic |
|
||||||
|
|---|---|
|
||||||
|
| RTTI type descriptors | non-empty line count of `findings/objects/rtti-raw.txt` |
|
||||||
|
| `Game::` / `Mars::` classes | line counts of `classes-game.txt` / `classes-mars.txt` |
|
||||||
|
| serializable types | line count of `serializable-types.txt` — used as the "classes of interest" denominator because save-layout recovery targets exactly these |
|
||||||
|
| classes with recovered layouts | **heuristic:** distinct backticked `Game::X` / `Mars::X` names that appear in `##`, `###` or `####` headings of `findings/objects/struct-recovery.md` and `schema-gaps-resolved.md`. Each such heading introduces a member table or a settled layout. Nested/POD helpers with their own heading count; classes mentioned only in prose or table cells do not. Over-counts slightly when a heading names a class whose layout is only partial (e.g. `TechTree`); under-counts classes recovered elsewhere (e.g. `save-editor-structs.md`). |
|
||||||
|
| functions | regex `N functions` in `findings/01-fingerprint.md`; falls back to the hardcoded 41,411 with a warning |
|
||||||
|
| named/annotated | number of `entries[]` in `ghidra/addresses.json`. **This is the address contract handed to sots-engine, not Ghidra's total rename count** (the Ghidra project has ~60 spine functions + 52 serializers + ~600 `g_KEY` labels renamed; those are not exported anywhere parseable). |
|
||||||
|
| addresses verified | entries whose `status` starts with `verified` (`verified`, `verified-by-save`, …) |
|
||||||
|
|
||||||
|
## 3. Data layer
|
||||||
|
|
||||||
|
| Number | Source |
|
||||||
|
|---|---|
|
||||||
|
| files parsed / total | `Total: N parsed, M failed` line in `findings/subsystems/data-parsers.md` (written by `verify/parsers/verify.py`); total = N + M (the 70 skipped HLSL/prose files are not data and are not counted) |
|
||||||
|
| block kinds | number of top-level keys in `verify/results/data-catalogs/schema_stats.json` |
|
||||||
|
| dangling cross-refs | sum of the lengths of every `*_dangling` list in `crosslink.json` (case-mismatch and "no requires" lists are informational and not counted) |
|
||||||
|
| oracle `mars-parse` / `mars-text` | the `**total**` row of the first table in each `sots-engine/docs/mars-*.md` whose header has a `files` column and an `agree`/`oracle match` column; reported as agree/files. Docs without such a row are warned about. |
|
||||||
|
| saves strict | count of `*.sav` in `verify/results/saves/` and the `strict exit E -- X errors, Y warnings` phrase in the first three lines of `turn2-strict-issues.txt`; clean iff exit 0 and 0 errors. The N/N form assumes the issues file speaks for all real saves, as the board `save_reader.py` row states. |
|
||||||
|
| design rules | first `N/N designs` phrase in `verify/design-rules/SHIP_DESIGN_RULES.md` |
|
||||||
|
|
||||||
|
## 4. Engine accrual
|
||||||
|
|
||||||
|
Modules = every directory under `sots-engine/src/` that has its own `CMakeLists.txt`, plus any
|
||||||
|
`src/...` directory whose sources are compiled directly from the root `CMakeLists.txt`
|
||||||
|
(currently `src/shim`, built only under `if(WIN32)`). `third_party/` is never a module.
|
||||||
|
|
||||||
|
| Column | Source |
|
||||||
|
|---|---|
|
||||||
|
| LOC | `wc -l` equivalent over `*.cpp *.h *.c` in the module directory, excluding `third_party` |
|
||||||
|
| Test files | number of files under `sots-engine/tests/<name>/` where `<name>` is the module path with `/` replaced by `_` (`mars/parse → tests/mars_parse`), skipping `build/` |
|
||||||
|
| Checks | lines in those tests' `*.cpp` matching `CHECK*(`, `REQUIRE*(`, `ASSERT*(`, `EXPECT*(`, `check(`, `assert(`. Counts call sites, not executed assertions (a loop of checks counts once). |
|
||||||
|
| Wired | `yes` if `add_subdirectory(src/<mod>)` appears in the root `CMakeLists.txt`; `direct (WIN32)` if its sources are listed directly inside a WIN32 block; `no` otherwise |
|
||||||
|
| Doc | `docs/<name-with-dashes>.md` if it exists, else the first `docs/*.md` whose first five lines mention the module's leaf name (`shim → M0.md`) |
|
||||||
|
|
||||||
|
"Verified & merged vs in flight" uses the board: `engine:*` rows by status; the bar is
|
||||||
|
verified / all engine rows. In flight = `in-progress`.
|
||||||
|
|
||||||
|
## 5. Phase 2 milestones
|
||||||
|
|
||||||
|
Board rows whose Target starts with `P2-M`. Glyphs: ✅ verified or mapped, 🔄 in-progress,
|
||||||
|
⬜ backlog, ⛔ blocked. Notes are truncated to 90 characters.
|
||||||
|
|
||||||
|
## 6. Verification ledger
|
||||||
|
|
||||||
|
One line per evidence source: saves strict and design rules (as in section 3), each parsed
|
||||||
|
oracle, `verify/harness/compare/` directory present, `verify/results/shim/m0.log` present,
|
||||||
|
and the board status of the `determinism oracle` row.
|
||||||
|
|
||||||
|
## 7. Open questions
|
||||||
|
|
||||||
|
`campaign/open-questions.md`: every bullet beginning `- **`. A bullet is *resolved/parked* if
|
||||||
|
its bold text starts with `RESOLVED`, `Resolved` or `(parked)`; everything else is open. "Most
|
||||||
|
recent" = the last five open bullets in file order (the file is append-ordered), newest first,
|
||||||
|
first 100 characters. Backlog counts are numbered or bulleted items under each `## ` heading
|
||||||
|
of `campaign/backlog.md`.
|
||||||
|
|
||||||
|
## 8. Delta
|
||||||
|
|
||||||
|
The previous `DASHBOARD.md` carries a machine-readable footer comment
|
||||||
|
`<!-- dashboard-metrics {...} -->`. It is read before the file is overwritten; the section
|
||||||
|
shows old → new (±) for verified targets, mapped-or-better, engine LOC, test files, checks,
|
||||||
|
addresses verified, recovered layouts and open questions. A dashboard without that comment
|
||||||
|
(or the first run) reports "first run".
|
||||||
|
|
||||||
|
## Adding a number
|
||||||
|
|
||||||
|
Parse it in its own small function that returns `None` on failure and calls `warn(...)`, render
|
||||||
|
it with a `?` fallback, and document it here. Never let a missing file raise.
|
||||||
493
tools/dashboard.py
Executable file
493
tools/dashboard.py
Executable file
|
|
@ -0,0 +1,493 @@
|
||||||
|
#!/usr/bin/python3
|
||||||
|
"""Generate campaign/DASHBOARD.md — coverage dashboard for the SotS RE + reimplementation campaign.
|
||||||
|
|
||||||
|
Pure stdlib. Idempotent: re-run after every integration.
|
||||||
|
tools/dashboard.py write campaign/DASHBOARD.md
|
||||||
|
tools/dashboard.py --print also echo the markdown to stdout
|
||||||
|
tools/dashboard.py --no-write render only (implies --print)
|
||||||
|
tools/dashboard.py --engine P sots-engine checkout (default ~/sots-engine or $SOTS_ENGINE)
|
||||||
|
Every number's source and heuristic is documented in tools/DASHBOARD_README.md.
|
||||||
|
Parsing is tolerant: malformed inputs are reported in the footer "warnings" line, never fatal.
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import datetime
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
|
||||||
|
RE_ROOT = os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)), ".."))
|
||||||
|
WARN = []
|
||||||
|
FUNCTIONS_FALLBACK = 41411 # findings/01-fingerprint.md, in case the number cannot be parsed
|
||||||
|
STATUSES = ["verified", "mapped", "in-progress", "backlog", "blocked"]
|
||||||
|
TYPE_ROWS = [ # (label, board Type values)
|
||||||
|
("objects", ("object",)), ("control-flow", ("control-flow",)), ("subsystems", ("subsystem",)),
|
||||||
|
("engine", ("engine",)), ("verify", ("verify",)), ("phase2", ("phase2",)), ("meta", ("meta",)),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def warn(msg):
|
||||||
|
WARN.append(msg)
|
||||||
|
|
||||||
|
|
||||||
|
def rp(*parts):
|
||||||
|
return os.path.join(RE_ROOT, *parts)
|
||||||
|
|
||||||
|
|
||||||
|
def read(path):
|
||||||
|
try:
|
||||||
|
with open(path, encoding="utf-8", errors="replace") as f:
|
||||||
|
return f.read()
|
||||||
|
except OSError as e:
|
||||||
|
warn(f"cannot read {os.path.relpath(path, RE_ROOT)}: {e.strerror}")
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def count_lines(path):
|
||||||
|
txt = read(path)
|
||||||
|
return sum(1 for ln in txt.splitlines() if ln.strip())
|
||||||
|
|
||||||
|
|
||||||
|
def git(repo, *args):
|
||||||
|
try:
|
||||||
|
return subprocess.check_output(["git", "-C", repo] + list(args), stderr=subprocess.DEVNULL,
|
||||||
|
text=True).strip()
|
||||||
|
except (subprocess.CalledProcessError, OSError):
|
||||||
|
warn(f"git {' '.join(args)} failed in {repo}")
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def bar(num, den, width=10):
|
||||||
|
pct = 0 if not den else 100.0 * num / den
|
||||||
|
filled = 0 if not den else int(round(width * num / den))
|
||||||
|
return f"`[{'█' * filled}{'░' * (width - filled)}] {pct:.0f}%`"
|
||||||
|
|
||||||
|
|
||||||
|
def n(x):
|
||||||
|
return f"{x:,}"
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- board
|
||||||
|
def parse_board(path):
|
||||||
|
rows = []
|
||||||
|
for i, ln in enumerate(read(path).splitlines(), 1):
|
||||||
|
if not ln.startswith("|"):
|
||||||
|
continue
|
||||||
|
cells = ln.split("|", 7) # ['', target, type, status, conf, coverage, updated, 'notes |']
|
||||||
|
if len(cells) < 8:
|
||||||
|
if not re.match(r"^\|\s*(Target|-+)\s*\|", ln):
|
||||||
|
warn(f"board.md line {i}: fewer than 7 cells, skipped")
|
||||||
|
continue
|
||||||
|
target, typ, status, conf, cov, updated = (c.strip() for c in cells[1:7])
|
||||||
|
notes = cells[7].strip().rstrip("|").strip()
|
||||||
|
if target == "Target" or set(target) <= {"-"}:
|
||||||
|
continue
|
||||||
|
if status not in STATUSES:
|
||||||
|
warn(f"board.md line {i}: unknown status '{status}' for '{target}'")
|
||||||
|
rows.append(dict(target=target.strip("`"), type=typ, status=status, conf=conf, cov=cov,
|
||||||
|
updated=updated, notes=notes))
|
||||||
|
if not rows:
|
||||||
|
warn("board.md: no rows parsed")
|
||||||
|
return rows
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- findings
|
||||||
|
def parse_functions(path):
|
||||||
|
m = re.search(r"\*{0,2}([\d,]{5,})\s+functions", read(path))
|
||||||
|
if m:
|
||||||
|
return int(m.group(1).replace(",", "")), "parsed"
|
||||||
|
warn("01-fingerprint.md: function count not found, using hardcoded 41,411")
|
||||||
|
return FUNCTIONS_FALLBACK, "hardcoded"
|
||||||
|
|
||||||
|
|
||||||
|
def recovered_layouts(paths):
|
||||||
|
"""Heuristic: unique `Game::X` / `Mars::X` names appearing in ##/###/#### headings of the
|
||||||
|
struct-recovery docs, excluding purely descriptive headings (no backticked class)."""
|
||||||
|
names = set()
|
||||||
|
for p in paths:
|
||||||
|
for ln in read(p).splitlines():
|
||||||
|
if re.match(r"^#{2,4} ", ln):
|
||||||
|
for m in re.finditer(r"`((?:Game|Mars)::[A-Za-z_][\w:]*)`", ln):
|
||||||
|
names.add(m.group(1))
|
||||||
|
return sorted(names)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_addresses(path):
|
||||||
|
try:
|
||||||
|
d = json.loads(read(path) or "{}")
|
||||||
|
entries = d.get("entries", d if isinstance(d, list) else [])
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
warn(f"addresses.json: {e}")
|
||||||
|
entries = []
|
||||||
|
total = len(entries)
|
||||||
|
verified = sum(1 for e in entries if str(e.get("status", "")).startswith("verified"))
|
||||||
|
return total, verified
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- verify
|
||||||
|
def parse_catalogs():
|
||||||
|
txt = read(rp("findings", "subsystems", "data-parsers.md"))
|
||||||
|
m = re.search(r"Total:\s*([\d,]+)\s+parsed,\s*([\d,]+)\s+failed", txt)
|
||||||
|
parsed = failed = None
|
||||||
|
if m:
|
||||||
|
parsed, failed = (int(x.replace(",", "")) for x in m.groups())
|
||||||
|
else:
|
||||||
|
warn("data-parsers.md: 'Total: N parsed, M failed' line not found")
|
||||||
|
kinds = None
|
||||||
|
try:
|
||||||
|
kinds = len(json.loads(read(rp("verify", "results", "data-catalogs", "schema_stats.json")) or "{}"))
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
warn(f"schema_stats.json: {e}")
|
||||||
|
dangling = None
|
||||||
|
try:
|
||||||
|
x = json.loads(read(rp("verify", "results", "data-catalogs", "crosslink.json")) or "{}")
|
||||||
|
dangling = sum(len(v) for k, v in x.items() if k.endswith("_dangling") and isinstance(v, list))
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
warn(f"crosslink.json: {e}")
|
||||||
|
return parsed, failed, kinds, dangling
|
||||||
|
|
||||||
|
|
||||||
|
def parse_oracles(engine):
|
||||||
|
"""docs/mars-*.md: the `| **total** |` row of a table whose header has a files column and an
|
||||||
|
agree/match column."""
|
||||||
|
out = []
|
||||||
|
for name in sorted(os.listdir(os.path.join(engine, "docs")) if os.path.isdir(os.path.join(engine, "docs")) else []):
|
||||||
|
if not (name.startswith("mars-") and name.endswith(".md")):
|
||||||
|
continue
|
||||||
|
header = None
|
||||||
|
found = False
|
||||||
|
in_table = False
|
||||||
|
for ln in read(os.path.join(engine, "docs", name)).splitlines():
|
||||||
|
if not ln.startswith("|"):
|
||||||
|
in_table = False
|
||||||
|
continue
|
||||||
|
cells = [c.strip().strip("*").strip().lower() for c in ln.strip().strip("|").split("|")]
|
||||||
|
if set("".join(cells)) <= {"-", ":", " "}:
|
||||||
|
continue
|
||||||
|
if not in_table: # first row of a table is its header
|
||||||
|
header, in_table = cells, True
|
||||||
|
continue
|
||||||
|
if cells and cells[0] == "total" and header:
|
||||||
|
def col(pred):
|
||||||
|
for h, c in zip(header, cells):
|
||||||
|
if pred(h):
|
||||||
|
m = re.search(r"[\d,]+", c)
|
||||||
|
if m:
|
||||||
|
return int(m.group(0).replace(",", ""))
|
||||||
|
return None
|
||||||
|
files = col(lambda h: "files" in h)
|
||||||
|
agree = col(lambda h: "agree" in h or "match" in h)
|
||||||
|
if files is not None and agree is not None:
|
||||||
|
out.append((name[:-3], agree, files))
|
||||||
|
found = True
|
||||||
|
break
|
||||||
|
if not found:
|
||||||
|
warn(f"{name}: no oracle total row parsed")
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def parse_saves():
|
||||||
|
d = rp("verify", "results", "saves")
|
||||||
|
saves = [f for f in (os.listdir(d) if os.path.isdir(d) else []) if f.endswith(".sav")]
|
||||||
|
head = "\n".join(read(os.path.join(d, "turn2-strict-issues.txt")).splitlines()[:3])
|
||||||
|
m = re.search(r"strict exit (\d+)\s*--\s*(\d+) errors,\s*(\d+) warnings", head)
|
||||||
|
if m:
|
||||||
|
ok = m.group(1) == "0" and m.group(2) == "0"
|
||||||
|
detail = f"strict exit {m.group(1)}, {m.group(2)} errors, {m.group(3)} warnings"
|
||||||
|
else:
|
||||||
|
ok = "0 resyncs" in head and "0 hint-failures" in head
|
||||||
|
detail = head.splitlines()[0][:80] if head else "issues file missing"
|
||||||
|
if not head:
|
||||||
|
warn("turn2-strict-issues.txt missing")
|
||||||
|
return len(saves), ok, detail
|
||||||
|
|
||||||
|
|
||||||
|
def parse_design_rules():
|
||||||
|
txt = read(rp("verify", "design-rules", "SHIP_DESIGN_RULES.md"))
|
||||||
|
m = re.search(r"(\d+)/(\d+) designs", txt)
|
||||||
|
if m:
|
||||||
|
return int(m.group(1)), int(m.group(2))
|
||||||
|
warn("SHIP_DESIGN_RULES.md: 'N/N designs' not found")
|
||||||
|
return None, None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- engine
|
||||||
|
SRC_EXT = (".cpp", ".h", ".c")
|
||||||
|
|
||||||
|
|
||||||
|
def loc(path):
|
||||||
|
total = 0
|
||||||
|
for root, dirs, files in os.walk(path):
|
||||||
|
dirs[:] = [d for d in dirs if d != "third_party"]
|
||||||
|
for f in files:
|
||||||
|
if f.endswith(SRC_EXT):
|
||||||
|
with open(os.path.join(root, f), "rb") as fh:
|
||||||
|
total += sum(1 for _ in fh)
|
||||||
|
return total
|
||||||
|
|
||||||
|
|
||||||
|
CHECK_RE = re.compile(r"\b(?:CHECK|REQUIRE|ASSERT|EXPECT)\w*\s*\(|\bcheck\(|\bassert\(")
|
||||||
|
|
||||||
|
|
||||||
|
def engine_modules(engine):
|
||||||
|
src = os.path.join(engine, "src")
|
||||||
|
root_cmake = read(os.path.join(engine, "CMakeLists.txt"))
|
||||||
|
mods = set()
|
||||||
|
for root, dirs, files in os.walk(src):
|
||||||
|
dirs[:] = [d for d in dirs if d != "third_party"]
|
||||||
|
if "CMakeLists.txt" in files:
|
||||||
|
mods.add(os.path.relpath(root, engine))
|
||||||
|
# dirs whose sources are compiled directly from the root CMakeLists (e.g. src/shim/main.cpp)
|
||||||
|
for m in re.finditer(r"(?<![\w/])(src/[\w/]+)/[\w.]+\.(?:cpp|c|def)\b", root_cmake):
|
||||||
|
mods.add(m.group(1))
|
||||||
|
out = []
|
||||||
|
for mod in sorted(mods):
|
||||||
|
name = mod[len("src/"):]
|
||||||
|
tname = name.replace("/", "_")
|
||||||
|
tdir = os.path.join(engine, "tests", tname)
|
||||||
|
tfiles, checks = 0, 0
|
||||||
|
if os.path.isdir(tdir):
|
||||||
|
for root, dirs, files in os.walk(tdir):
|
||||||
|
dirs[:] = [d for d in dirs if d not in ("build",)]
|
||||||
|
for f in files:
|
||||||
|
tfiles += 1
|
||||||
|
if f.endswith(".cpp"):
|
||||||
|
checks += sum(1 for ln in read(os.path.join(root, f)).splitlines() if CHECK_RE.search(ln))
|
||||||
|
if f"add_subdirectory({mod})" in root_cmake:
|
||||||
|
wired = "yes"
|
||||||
|
elif re.search(rf"\b{re.escape(mod)}/", root_cmake):
|
||||||
|
wired = "direct (WIN32)" if "if(WIN32)" in root_cmake else "direct"
|
||||||
|
else:
|
||||||
|
wired = "no"
|
||||||
|
doc = ""
|
||||||
|
cand = os.path.join(engine, "docs", name.replace("/", "-") + ".md")
|
||||||
|
if os.path.exists(cand):
|
||||||
|
doc = os.path.basename(cand)
|
||||||
|
else:
|
||||||
|
leaf = name.split("/")[-1]
|
||||||
|
ddir = os.path.join(engine, "docs")
|
||||||
|
for d in sorted(os.listdir(ddir)) if os.path.isdir(ddir) else []:
|
||||||
|
if d.endswith(".md") and re.search(rf"\b{leaf}\b", "\n".join(read(os.path.join(ddir, d)).splitlines()[:5]), re.I):
|
||||||
|
doc = d
|
||||||
|
break
|
||||||
|
out.append(dict(name=name, loc=loc(os.path.join(engine, mod)), tfiles=tfiles, checks=checks,
|
||||||
|
wired=wired, doc=doc or "—"))
|
||||||
|
if not out:
|
||||||
|
warn("sots-engine: no modules found under src/")
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- questions / backlog
|
||||||
|
def parse_questions(path):
|
||||||
|
open_, closed = [], []
|
||||||
|
for ln in read(path).splitlines():
|
||||||
|
m = re.match(r"^- \*\*(.*)", ln)
|
||||||
|
if not m:
|
||||||
|
continue
|
||||||
|
text = m.group(1)
|
||||||
|
if re.match(r"(RESOLVED|Resolved|\(parked\))", text):
|
||||||
|
closed.append(text)
|
||||||
|
else:
|
||||||
|
open_.append(text)
|
||||||
|
return open_, closed
|
||||||
|
|
||||||
|
|
||||||
|
def parse_backlog(path):
|
||||||
|
sec, counts = None, {}
|
||||||
|
for ln in read(path).splitlines():
|
||||||
|
m = re.match(r"^## (.+)", ln)
|
||||||
|
if m:
|
||||||
|
sec = m.group(1).split("(")[0].strip()
|
||||||
|
counts[sec] = 0
|
||||||
|
elif sec and re.match(r"^\s*(\d+\.|[-*]) ", ln):
|
||||||
|
counts[sec] += 1
|
||||||
|
return counts
|
||||||
|
|
||||||
|
|
||||||
|
def north_star(path):
|
||||||
|
txt = read(path)
|
||||||
|
m = re.search(r"## North star.*?\n\*\*(.+?)\*\*", txt, re.S)
|
||||||
|
return " ".join(m.group(1).split()) if m else "functional reimplementation, behavior-equivalent (see findings/00-strategy.md)"
|
||||||
|
|
||||||
|
|
||||||
|
def previous_metrics(path):
|
||||||
|
m = re.search(r"<!-- dashboard-metrics (\{.*?\}) -->", read(path) if os.path.exists(path) else "")
|
||||||
|
if not m:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return json.loads(m.group(1))
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
warn("previous DASHBOARD.md: metrics comment unreadable")
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------- render
|
||||||
|
GLYPH = {"verified": "✅", "mapped": "✅", "in-progress": "🔄", "backlog": "⬜", "blocked": "⛔"}
|
||||||
|
|
||||||
|
|
||||||
|
def render(engine, prev):
|
||||||
|
L = []
|
||||||
|
now = datetime.datetime.now(datetime.timezone.utc).strftime("%Y-%m-%d %H:%M UTC")
|
||||||
|
re_commit = git(RE_ROOT, "log", "-1", "--format=%h,%cs")
|
||||||
|
eng_commit = git(engine, "log", "-1", "--format=%h,%cs")
|
||||||
|
eng_commits = git(engine, "rev-list", "--count", "HEAD") or "?"
|
||||||
|
|
||||||
|
rows = parse_board(rp("campaign", "board.md"))
|
||||||
|
by = {s: sum(1 for r in rows if r["status"] == s) for s in STATUSES}
|
||||||
|
total = len(rows)
|
||||||
|
mapped_plus = by["verified"] + by["mapped"]
|
||||||
|
|
||||||
|
# 1 header
|
||||||
|
L += ["# SotS RE campaign — coverage dashboard", "",
|
||||||
|
f"Generated {now} · `sots-re` @ {re_commit or '?'} · `sots-engine` @ {eng_commit or '?'} "
|
||||||
|
f"({eng_commits} commits) · regenerate with `tools/dashboard.py`", "",
|
||||||
|
f"> **North star:** {north_star(rp('findings', '00-strategy.md'))}", ""]
|
||||||
|
|
||||||
|
# 2 map coverage
|
||||||
|
L += ["## 1. Map coverage (campaign/board.md)", "",
|
||||||
|
f"{total} targets · mapped-or-better **{mapped_plus}/{total}** {bar(mapped_plus, total)} · "
|
||||||
|
f"verified **{by['verified']}/{total}** {bar(by['verified'], total)}", "",
|
||||||
|
"| Status | Count | % |", "|---|---:|---:|"]
|
||||||
|
for s in STATUSES:
|
||||||
|
L.append(f"| {s} | {by[s]} | {0 if not total else round(100 * by[s] / total)}% |")
|
||||||
|
L += ["", "| Type | verified | mapped | in-progress | backlog | blocked | total |", "|---|---:|---:|---:|---:|---:|---:|"]
|
||||||
|
known = set()
|
||||||
|
for label, types in TYPE_ROWS:
|
||||||
|
known.update(types)
|
||||||
|
sub = [r for r in rows if r["type"] in types]
|
||||||
|
c = {s: sum(1 for r in sub if r["status"] == s) for s in STATUSES}
|
||||||
|
L.append(f"| {label} | {c['verified']} | {c['mapped']} | {c['in-progress']} | {c['backlog']} | {c['blocked']} | {len(sub)} |")
|
||||||
|
other = [r for r in rows if r["type"] not in known]
|
||||||
|
if other:
|
||||||
|
warn("board.md: unknown types " + ", ".join(sorted({r['type'] for r in other})))
|
||||||
|
c = {s: sum(1 for r in other if r["status"] == s) for s in STATUSES}
|
||||||
|
L.append(f"| other | {c['verified']} | {c['mapped']} | {c['in-progress']} | {c['backlog']} | {c['blocked']} | {len(other)} |")
|
||||||
|
L.append("")
|
||||||
|
|
||||||
|
# 3 binary understanding
|
||||||
|
rtti = count_lines(rp("findings", "objects", "rtti-raw.txt"))
|
||||||
|
cg = count_lines(rp("findings", "objects", "classes-game.txt"))
|
||||||
|
cm = count_lines(rp("findings", "objects", "classes-mars.txt"))
|
||||||
|
ser = count_lines(rp("findings", "objects", "serializable-types.txt"))
|
||||||
|
layouts = recovered_layouts([rp("findings", "objects", "struct-recovery.md"), rp("findings", "objects", "schema-gaps-resolved.md")])
|
||||||
|
funcs, fsrc = parse_functions(rp("findings", "01-fingerprint.md"))
|
||||||
|
a_total, a_ver = parse_addresses(rp("ghidra", "addresses.json"))
|
||||||
|
L += ["## 2. Binary understanding", "",
|
||||||
|
f"- RTTI type descriptors: **{n(rtti)}** (`Game::` {n(cg)}, `Mars::` {n(cm)}; serializable types {n(ser)})",
|
||||||
|
f"- Classes with recovered member layouts: **{len(layouts)}** / {ser} serializable types "
|
||||||
|
f"{bar(len(layouts), ser)} — heuristic: distinct `Game::X`/`Mars::X` in `##`–`####` headings of "
|
||||||
|
f"`struct-recovery.md` + `schema-gaps-resolved.md`",
|
||||||
|
f"- Functions: **{n(funcs)}** ({fsrc} from `01-fingerprint.md`); named/annotated in the **address contract** "
|
||||||
|
f"(`ghidra/addresses.json`, not Ghidra's full rename count): **{a_total}**, verified **{a_ver}** "
|
||||||
|
f"{bar(a_ver, a_total)}", ""]
|
||||||
|
|
||||||
|
# 4 data layer
|
||||||
|
parsed, failed, kinds, dangling = parse_catalogs()
|
||||||
|
oracles = parse_oracles(engine)
|
||||||
|
nsaves, s_ok, s_detail = parse_saves()
|
||||||
|
d_ok, d_tot = parse_design_rules()
|
||||||
|
cat = f"**{n(parsed)}/{n(parsed + failed)}** files parsed" if parsed is not None else "**?** files parsed"
|
||||||
|
L += ["## 3. Data layer", "",
|
||||||
|
f"- Catalogs: {cat} ({kinds if kinds is not None else '?'} block kinds in `schema_stats.json`), "
|
||||||
|
f"dangling cross-refs **{dangling if dangling is not None else '?'}** (`crosslink.json`)"]
|
||||||
|
for name, agree, files in oracles:
|
||||||
|
L.append(f"- Oracle `{name}`: **{n(agree)}/{n(files)}** files agree {bar(agree, files)}")
|
||||||
|
if not oracles:
|
||||||
|
L.append("- Oracle agreements: none parsed from `sots-engine/docs/mars-*.md`")
|
||||||
|
L += [f"- Saves: **{nsaves}/{nsaves}** real saves strict-clean — {s_detail}" if s_ok else f"- Saves: {nsaves} real saves, strict **NOT clean** — {s_detail}",
|
||||||
|
f"- Design rules: **{d_ok}/{d_tot}** stock designs pass {bar(d_ok or 0, d_tot or 1)}" if d_ok is not None else "- Design rules: ? (unparsed)", ""]
|
||||||
|
|
||||||
|
# 5 engine
|
||||||
|
mods = engine_modules(engine)
|
||||||
|
eng_rows = [r for r in rows if r["type"] == "engine"]
|
||||||
|
e_ver = sum(1 for r in eng_rows if r["status"] == "verified")
|
||||||
|
e_map = sum(1 for r in eng_rows if r["status"] == "mapped")
|
||||||
|
e_fly = sum(1 for r in eng_rows if r["status"] == "in-progress")
|
||||||
|
tot_loc = sum(m["loc"] for m in mods)
|
||||||
|
tot_tf = sum(m["tfiles"] for m in mods)
|
||||||
|
tot_ck = sum(m["checks"] for m in mods)
|
||||||
|
L += ["## 4. Engine accrual (sots-engine)", "",
|
||||||
|
"| Module | LOC | Test files | Checks | Wired | Doc |", "|---|---:|---:|---:|---|---|"]
|
||||||
|
for m in mods:
|
||||||
|
L.append(f"| `{m['name']}` | {n(m['loc'])} | {m['tfiles']} | {m['checks']} | {m['wired']} | {m['doc']} |")
|
||||||
|
L += [f"| **total** | **{n(tot_loc)}** | **{tot_tf}** | **{tot_ck}** | | |", "",
|
||||||
|
f"Board `engine:` rows: verified **{e_ver}**, mapped {e_map}, in flight {e_fly} (of {len(eng_rows)}) — "
|
||||||
|
f"verified & merged {bar(e_ver, len(eng_rows))}", ""]
|
||||||
|
|
||||||
|
# 6 phase 2
|
||||||
|
L += ["## 5. Phase 2 milestones", "", "| Milestone | Status | Coverage | Notes |", "|---|---|---:|---|"]
|
||||||
|
p2 = [r for r in rows if r["target"].startswith("P2-M")]
|
||||||
|
for r in p2:
|
||||||
|
L.append(f"| {r['target']} | {GLYPH.get(r['status'], '?')} {r['status']} | {r['cov']} | {r['notes'][:90]} |")
|
||||||
|
if not p2:
|
||||||
|
warn("board.md: no P2-M* rows")
|
||||||
|
L.append("")
|
||||||
|
|
||||||
|
# 7 verification ledger
|
||||||
|
def status_of(target_re):
|
||||||
|
for r in rows:
|
||||||
|
if re.search(target_re, r["target"]):
|
||||||
|
return r["status"]
|
||||||
|
return "not on board"
|
||||||
|
m0 = os.path.exists(rp("verify", "results", "shim", "m0.log"))
|
||||||
|
harness = os.path.isdir(rp("verify", "harness", "compare"))
|
||||||
|
L += ["## 6. Verification ledger", "",
|
||||||
|
f"- {'✅' if s_ok else '❌'} Saves strict: {nsaves}/{nsaves} ({s_detail})",
|
||||||
|
f"- {'✅' if d_ok == d_tot and d_ok else '❌'} Design rules: {d_ok}/{d_tot}",
|
||||||
|
"- " + (" · ".join(f"{'✅' if a == f else '❌'} oracle {nm} {a}/{f}" for nm, a, f in oracles) if oracles else "❌ oracle parsers: none parsed"),
|
||||||
|
f"- {'✅' if harness else '❌'} Compare harness present (`verify/harness/compare/`)",
|
||||||
|
f"- {'✅' if m0 else '❌'} M0 evidence present (`verify/results/shim/m0.log`)",
|
||||||
|
f"- {GLYPH.get(status_of(r'^determinism oracle'), '⬜')} Determinism oracle: {status_of(r'^determinism oracle')}", ""]
|
||||||
|
|
||||||
|
# 8 open questions
|
||||||
|
oq, closed = parse_questions(rp("campaign", "open-questions.md"))
|
||||||
|
bl = parse_backlog(rp("campaign", "backlog.md"))
|
||||||
|
L += ["## 7. Open questions", "",
|
||||||
|
f"Open **{len(oq)}** · resolved/parked {len(closed)} · backlog items: " +
|
||||||
|
(", ".join(f"{k} {v}" for k, v in bl.items()) if bl else "?"), "", "Most recent open:", ""]
|
||||||
|
for t in oq[-5:][::-1]:
|
||||||
|
L.append(f"- {t[:100].replace('**', '')}{'…' if len(t) > 100 else ''}")
|
||||||
|
L.append("")
|
||||||
|
|
||||||
|
# 9 delta
|
||||||
|
cur = dict(verified=by["verified"], mapped_plus=mapped_plus, targets=total, loc=tot_loc, tests=tot_tf,
|
||||||
|
checks=tot_ck, addr_verified=a_ver, addr_total=a_total, layouts=len(layouts), open_q=len(oq))
|
||||||
|
L += ["## 8. Delta since previous dashboard", ""]
|
||||||
|
if prev:
|
||||||
|
def d(k):
|
||||||
|
if k not in prev:
|
||||||
|
return "n/a"
|
||||||
|
diff = cur[k] - prev[k]
|
||||||
|
return f"{prev[k]:,} → {cur[k]:,} ({diff:+,})"
|
||||||
|
L += [f"- verified targets: {d('verified')} · mapped-or-better: {d('mapped_plus')}",
|
||||||
|
f"- engine LOC: {d('loc')} · test files: {d('tests')} · checks: {d('checks')}",
|
||||||
|
f"- addresses verified: {d('addr_verified')} · recovered layouts: {d('layouts')} · open questions: {d('open_q')}", ""]
|
||||||
|
else:
|
||||||
|
L += ["- first run (no previous `DASHBOARD.md` metrics found)", ""]
|
||||||
|
|
||||||
|
L += ["---", f"warnings: {'; '.join(WARN) if WARN else 'none'}",
|
||||||
|
f"<!-- dashboard-metrics {json.dumps(cur)} -->", ""]
|
||||||
|
return "\n".join(L)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||||
|
ap.add_argument("--print", action="store_true", help="echo the markdown to stdout")
|
||||||
|
ap.add_argument("--no-write", action="store_true", help="do not write campaign/DASHBOARD.md (implies --print)")
|
||||||
|
ap.add_argument("--engine", default=os.environ.get("SOTS_ENGINE", os.path.expanduser("~/sots-engine")))
|
||||||
|
a = ap.parse_args()
|
||||||
|
if not os.path.isdir(a.engine):
|
||||||
|
warn(f"sots-engine not found at {a.engine}")
|
||||||
|
out_path = rp("campaign", "DASHBOARD.md")
|
||||||
|
md = render(a.engine, previous_metrics(out_path))
|
||||||
|
if a.print or a.no_write:
|
||||||
|
sys.stdout.write(md)
|
||||||
|
if not a.no_write:
|
||||||
|
with open(out_path, "w", encoding="utf-8") as f:
|
||||||
|
f.write(md)
|
||||||
|
print(f"wrote {os.path.relpath(out_path, RE_ROOT)} ({len(WARN)} warnings)", file=sys.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Loading…
Add table
Reference in a new issue