c504729341
tools/standalone_report.py drives sots-engine's sots_turn over each consecutive-turn save pair and diffs the result against the game's own post-turn save with state_checksum.py, which localises to named leaves and proves its own coverage by re-serialisation. turn1-state -> turn2-state baseline 209 diverging, after 204, closed 5 turn2-state -> turn3-state baseline 108 diverging, after 103, closed 5 regressed 0 on both `regressed` is reported next to `closed` and never netted off. It earned its place immediately: committing the phase-31 player-status restore turned two agreeing leaves into disagreeing ones, because the phase writes 1 and the file carries 4. The stable-system stand-in feeding the colony pass is a labelled hypothesis and it survived a changed workload -- the same 3 ntdev leaves closed on both pairs, six agreements, zero disagreements. Two things deliberately NOT implemented: the TShn/ltis counters (18 leaves, a `+1` would close them, but "+1 across one observed turn" is a hypothesis, not a reading), and the RNG state write-back (an advanced-but-incomplete generator is wrong in a different way from an untouched one). dashboard.py gains section 6, reading verify/results/standalone/status.json: phases modelled/committed per driver, baseline vs after, closed vs regressed, the subsystem breakdown of what still differs, and the RNG gap. Sections 6-8 renumbered to 7-9; the delta footer tracks the two new counts. DASHBOARD_README.md documents every number. findings/control-flow/standalone-scaffold.md has the ranked blocker list.
131 lines
9.2 KiB
Markdown
131 lines
9.2 KiB
Markdown
# `tools/dashboard.py` — what every number means
|
|
|
|
Generates `campaign/DASHBOARD.md` from the campaign tracker, the findings, the verify
|
|
artefacts and the `sots-engine` checkout. Pure Python 3 stdlib (`/usr/bin/python3`), no
|
|
Ghidra, no network. Idempotent: run it after every integration; it overwrites the
|
|
dashboard and computes the delta against the previous one before doing so.
|
|
|
|
```
|
|
tools/dashboard.py # write campaign/DASHBOARD.md
|
|
tools/dashboard.py --print # also echo to stdout
|
|
tools/dashboard.py --no-write # render to stdout only, touch nothing
|
|
tools/dashboard.py --engine PATH # sots-engine checkout (default ~/sots-engine, or $SOTS_ENGINE)
|
|
```
|
|
|
|
Nothing is fatal: an unreadable or malformed input produces a `?` or a missing row and an
|
|
entry in the footer `warnings:` line. Check that line first when a number looks wrong.
|
|
|
|
## Header
|
|
|
|
| Item | Source |
|
|
|---|---|
|
|
| Generated timestamp | wall clock, UTC |
|
|
| `sots-re` commit | `git log -1 --format=%h,%cs` in this repo |
|
|
| `sots-engine` commit + count | same in the engine checkout; count = `git rev-list --count HEAD` |
|
|
| North star | bold sentence under `## North star` in `findings/00-strategy.md` |
|
|
|
|
## 1. Map coverage
|
|
|
|
Source: `campaign/board.md`, every line starting with `|` except the header and separator.
|
|
Rows are split on `|` for the first six cells only (`Target, Type, Status, Conf, Coverage,
|
|
Updated`); everything after is the Notes cell, so backticks or pipes in notes cannot break a
|
|
row. Rows with fewer than seven cells are skipped with a warning; unknown statuses are
|
|
counted but warned.
|
|
|
|
- **mapped-or-better** = `mapped` + `verified` (verified implies mapped).
|
|
- Type breakdown maps board `Type` values to rows: `object → objects`, `control-flow`,
|
|
`subsystem → subsystems`, `engine`, `verify`, `phase2`, `meta`. Any other type appears as an
|
|
`other` row plus a warning.
|
|
|
|
## 2. Binary understanding
|
|
|
|
| Number | Source / heuristic |
|
|
|---|---|
|
|
| RTTI type descriptors | non-empty line count of `findings/objects/rtti-raw.txt` |
|
|
| `Game::` / `Mars::` classes | line counts of `classes-game.txt` / `classes-mars.txt` |
|
|
| serializable types | line count of `serializable-types.txt` — used as the "classes of interest" denominator because save-layout recovery targets exactly these |
|
|
| classes with recovered layouts | **heuristic:** distinct backticked `Game::X` / `Mars::X` names that appear in `##`, `###` or `####` headings of `findings/objects/struct-recovery.md` and `schema-gaps-resolved.md`. Each such heading introduces a member table or a settled layout. Nested/POD helpers with their own heading count; classes mentioned only in prose or table cells do not. Over-counts slightly when a heading names a class whose layout is only partial (e.g. `TechTree`); under-counts classes recovered elsewhere (e.g. `save-editor-structs.md`). |
|
|
| functions | regex `N functions` in `findings/01-fingerprint.md`; falls back to the hardcoded 41,411 with a warning |
|
|
| named/annotated | number of `entries[]` in `ghidra/addresses.json`. **This is the address contract handed to sots-engine, not Ghidra's total rename count** (the Ghidra project has ~60 spine functions + 52 serializers + ~600 `g_KEY` labels renamed; those are not exported anywhere parseable). |
|
|
| addresses verified | entries whose `status` starts with `verified` (`verified`, `verified-by-save`, …) |
|
|
|
|
## 3. Data layer
|
|
|
|
| Number | Source |
|
|
|---|---|
|
|
| files parsed / total | `Total: N parsed, M failed` line in `findings/subsystems/data-parsers.md` (written by `verify/parsers/verify.py`); total = N + M (the 70 skipped HLSL/prose files are not data and are not counted) |
|
|
| block kinds | number of top-level keys in `verify/results/data-catalogs/schema_stats.json` |
|
|
| dangling cross-refs | sum of the lengths of every `*_dangling` list in `crosslink.json` (case-mismatch and "no requires" lists are informational and not counted) |
|
|
| oracle `mars-parse` / `mars-text` | the `**total**` row of the first table in each `sots-engine/docs/mars-*.md` whose header has a `files` column and an `agree`/`oracle match` column; reported as agree/files. Docs without such a row are warned about. |
|
|
| saves strict | count of `*.sav` in `verify/results/saves/` and the `strict exit E -- X errors, Y warnings` phrase in the first three lines of `turn2-strict-issues.txt`; clean iff exit 0 and 0 errors. The N/N form assumes the issues file speaks for all real saves, as the board `save_reader.py` row states. |
|
|
| design rules | first `N/N designs` phrase in `verify/design-rules/SHIP_DESIGN_RULES.md` |
|
|
|
|
## 4. Engine accrual
|
|
|
|
Modules = every directory under `sots-engine/src/` that has its own `CMakeLists.txt`, plus any
|
|
`src/...` directory whose sources are compiled directly from the root `CMakeLists.txt`
|
|
(currently `src/shim`, built only under `if(WIN32)`). `third_party/` is never a module.
|
|
|
|
| Column | Source |
|
|
|---|---|
|
|
| LOC | `wc -l` equivalent over `*.cpp *.h *.c` in the module directory, excluding `third_party` |
|
|
| Test files | number of files under `sots-engine/tests/<name>/` where `<name>` is the module path with `/` replaced by `_` (`mars/parse → tests/mars_parse`), skipping `build/` |
|
|
| Checks | lines in those tests' `*.cpp` matching `CHECK*(`, `REQUIRE*(`, `ASSERT*(`, `EXPECT*(`, `check(`, `assert(`. Counts call sites, not executed assertions (a loop of checks counts once). |
|
|
| Wired | `yes` if `add_subdirectory(src/<mod>)` appears in the root `CMakeLists.txt`; `direct (WIN32)` if its sources are listed directly inside a WIN32 block; `no` otherwise |
|
|
| Doc | `docs/<name-with-dashes>.md` if it exists, else the first `docs/*.md` whose first five lines mention the module's leaf name (`shim → M0.md`) |
|
|
|
|
"Verified & merged vs in flight" uses the board: `engine:*` rows by status; the bar is
|
|
verified / all engine rows. In flight = `in-progress`.
|
|
|
|
## 5. Phase 2 milestones
|
|
|
|
Board rows whose Target starts with `P2-M`. Glyphs: ✅ verified or mapped, 🔄 in-progress,
|
|
⬜ backlog, ⛔ blocked. Notes are truncated to 90 characters.
|
|
|
|
## 6. Standalone progress
|
|
|
|
Source: `verify/results/standalone/status.json`, written by `tools/standalone_report.py` (which
|
|
drives `sots-engine`'s `sots_turn` over each consecutive-turn save pair and diffs the result
|
|
against the game's own post-turn save with `verify/state-checksum/state_checksum.py`). The file
|
|
is optional: absent, unreadable, or carrying an unexpected `schema` -> the section renders
|
|
"Not measured" plus a footer warning, and the delta row shows `n/a`.
|
|
|
|
| Number | Source |
|
|
|---|---|
|
|
| turn-driver phases modelled / total | `phases.modelled` / `phases.total`. **Total is 44 = the 32 phases of `StrategyServer::ProcessTurn` + the 12 of `ServerPlayer::ProcessTurn`.** The host steps around the drivers are deliberately excluded from the denominator; the 37-phase post-combat tail is a separate row |
|
|
| committed | phases whose result is actually written to the save. A `blocked` phase is *modelled* (it runs and reports) but not *committed*, so `committed <= modelled` always |
|
|
| status breakdown | `verified` (compared against the live game) / `implemented` / `partial` / `blocked` (formula held, an input is not) / `stub` (named no-op) |
|
|
| baseline / after / closed / regressed | `reference.*`. Baseline = leaves that differ between the input save and the oracle, i.e. the distance a standalone that does nothing has to travel. `regressed` is reported next to `closed` and never netted off: a leaf that agreed before the turn and disagrees after it is a phase doing damage |
|
|
| byte match | `reference.byteMatch` — the milestone itself. The progress bar next to it is closed/baseline, not a claim about how much is left |
|
|
| subsystem table | `reference.subsystems`, the first 8 by leaf count |
|
|
| generator | `rng.wordsModelled` and the standing `rng.wordsPerTurnUnattributed` string |
|
|
|
|
The reference pair is the first entry of `PAIRS` in `standalone_report.py`; every pair it ran
|
|
is in `status.json` under `pairs`, and the readable form is `verify/results/standalone/report.txt`.
|
|
|
|
## 7. Verification ledger
|
|
|
|
One line per evidence source: saves strict and design rules (as in section 3), each parsed
|
|
oracle, `verify/harness/compare/` directory present, `verify/results/shim/m0.log` present,
|
|
and the board status of the `determinism oracle` row.
|
|
|
|
## 8. Open questions
|
|
|
|
`campaign/open-questions.md`: every bullet beginning `- **`. A bullet is *resolved/parked* if
|
|
its bold text starts with `RESOLVED`, `Resolved` or `(parked)`; everything else is open. "Most
|
|
recent" = the last five open bullets in file order (the file is append-ordered), newest first,
|
|
first 100 characters. Backlog counts are numbered or bulleted items under each `## ` heading
|
|
of `campaign/backlog.md`.
|
|
|
|
## 9. Delta
|
|
|
|
The previous `DASHBOARD.md` carries a machine-readable footer comment
|
|
`<!-- dashboard-metrics {...} -->`. It is read before the file is overwritten; the section
|
|
shows old → new (±) for verified targets, mapped-or-better, engine LOC, test files, checks,
|
|
addresses verified, recovered layouts, open questions, and the standalone's closed / still-diverging leaf counts. A dashboard without that comment
|
|
(or the first run) reports "first run".
|
|
|
|
## Adding a number
|
|
|
|
Parse it in its own small function that returns `None` on failure and calls `warn(...)`, render
|
|
it with a `?` fallback, and document it here. Never let a missing file raise.
|