merge lane Y: trade-raid RNG modelled under --commit-rng; T36 turn-record recovered (480 fields, 0 mismatches); generator gap measured at 4 and 2 words

docs/Y-standalone-rng-and-tail.md

@@ -0,0 +1,204 @@
+# Lane Y — the predictions, written before the build
+
+Committed before `src/app` was touched, per earned rule 2. Falsification sections included: how each
+prediction could be wrong, and the symptom of each way.
+
+Two things are being predicted: (a) what the standalone's generator does on lane Z's two calibrated
+oracle pairs, and (b) whether a turn-record model built from the wire reproduces the record the game
+already wrote into every save it made.
+
+---
+
+## P-Y1 — the trade-raid model costs exactly 16 words on an 8-player save
+
+Lane Z measured `ServerTradeManager::GenerateTradeRaidEncounters` firing two `Mars::RNG::Chance`
+rolls for every entry of the server's player vector, and showed that no back-edge contains either
+site, so one word per player per site is a **bound**, not an observation. Both probabilities are
+strictly inside (0, 1), so `Chance` takes neither early-out and spends exactly one word.
+
+The reference save's player vector holds **8** entries (`Sim.NumPlrs`; the wire's `players` array is
+that same vector).
+
+> **The standalone, run with `--commit-rng` on `z2-endturn.sav`, consumes exactly 16 generator
+> words**, all of them attributed to the trade-raid phase, and the post-turn `left` is
+> `395 − 16 = 379`. `mt[624]` is unchanged (no block boundary is crossed).
+
+Same claim on `z-t6-endturn.sav`: 16 words, `left` `375 − 16 = 359`.
+
+**Falsification.**
+
+| how it could be wrong | symptom |
+|---|---|
+| the loop is not over the server's player vector | a word count that is not 2 × 8 |
+| `Chance` is `p >= r` or narrows differently | word count still 16 (the count does not depend on the comparison) — this prediction cannot test it, and says so |
+| the NPC-raid gate is not open on this save | 8 words, not 16 |
+| the refugee-raid gate *is* open | 24 words |
+| the engine's `next_float` disagrees with the original | invisible here; only the count is being tested |
+
+---
+
+## P-Y2 — the residual against the oracle is 4 and 2, not 0
+
+The oracle pairs carry a known cost: `z2-endturn` → `z2-autosave` is **20** words, `z-t6-endturn` →
+`z-t6-autosave` is **18**. Lane Z's per-site ledger attributes the difference to four sites the
+standalone does not model:
+
+* 0x0050329d (`NextFloat`) and 0x007929a4 (`NextInt`), both under `StrategyServer::DetectEncounters`
+  — one word each per turn on 3 of 3 measured turns;
+* `TechTree::ProcessResearch`+0x1c8 and `ServerPlayer::RollResearchEvent`+0x2f — 0–1 words each, both
+  downstream of the research allocation, which is downstream of `ComputeBudget`, which is blocked on
+  the population → base-output term.
+
+> **The standalone lands 4 words short on the turn-5 pair and 2 words short on the turn-6 pair**, and
+> the shortfall is exactly the four sites above: 2 detection + 2 research on turn 5, 2 detection + 0
+> research on turn 6. The generator state therefore does **not** match the oracle on either pair, and
+> the answer to "does it land on the oracle's `mt[624]`/`left`" is **no**, by a stated amount.
+
+**Falsification.** A residual other than 4 and 2 means the site ledger and the file oracle disagree
+about this turn, which would be a real contradiction between two of lane Z's three instruments and
+would have to be reported as one. A residual of 0 would mean the trade-raid count is coincidentally
+absorbing the other sites.
+
+**Why the shortfall is not closed by guessing.** Modelling "2 detection words per turn" would fit
+three observations with no derived rule behind it, and lane I bounds that site's cost at
+`0 .. |contacts| × |detectors|`. It is left unmodelled and reported.
+
+---
+
+## P-Y3 — six fields of the turn record are recoverable from the wire
+
+`StrategyServer::FinalizeTurnRecords` (tail phase 36) fills a per-player record and archives it by
+turn. The archive is on the wire: it is `/Sim/turnstats/nply[]/hist/stats[]`, one `stats` element per
+turn per player, and every save carries an element for its own `Frame`.
+
+> **A record built from a save's own state reproduces that save's archive entry for its own turn, on
+> every player of every save in the corpus, for these six fields:**
+>
+> | record field | rule |
+> |---|---|
+> | `trn` | the frame counter |
+> | `col` | the number of owned-system ids |
+> | `sav` | the player's savings |
+> | `inc` | savings minus previous-turn savings |
+> | `tch` | the number of tech-tree states in the completed state |
+> | `pop` | summed over owned systems: population plus the pending population bonus |
+>
+> and it does **not** reproduce `almem`, `tdinc`, `bat`, the acquired/lost system lists, or the
+> per-hull-class ship census, because each of those is written by a phase or an input the standalone
+> does not hold.
+
+**Provenance, stated plainly.** The six rules were found by search over the corpus in Python before
+this was written, so this is not a blind prediction of the rules. What it *is* a prediction of is
+that a C++ transcription over the engine's own typed shapes agrees with them on all 112
+player-records — a test of the transcription and of the shape typing, not of the rules. Two of the
+six (`pop`, `inc`) were ambiguous in the Python search and were separated by the corpus:
+`pop` = Σ(`Pop` + `pbon`) matched 112/112 while Σ(`Pop` + `dcs`) matched 101/112.
+
+**Falsification.** Any player-record where the six computed fields differ from the stored ones.
+Particularly: a save where a player's owned-system list references a system id the save does not
+carry (the sum would silently drop a term).
+
+---
+
+## P-Y4 — committing phase 36 would regress, so it is not committed
+
+The record the standalone can build is the record for the **input** save's turn, not for the turn it
+is about to produce: `sav` after the turn needs `ApplyNetToSavings`, which is blocked, and `almem`
+needs the alliance-mask rebuild, which is a stub.
+
+> **Committing phase 36 would add a new archive element whose `sav`, `inc`, `almem`, `tdinc` and
+> class census are all wrong, turning one container-shaped divergence per player into several
+> leaf-shaped ones. The divergence report would show a large `regressed` count and a small `closed`
+> one.** It is therefore evaluated, reported, and not written. The divergence baseline stays
+> 5 closed / 0 regressed on both pairs.
+
+**Falsification.** If committing it closed more than it broke, the block was too conservative. That
+is measurable with `--commit-blocked` and is reported rather than argued.
+
+---
+
+## P-Y5 — the bankruptcy divisor in the engine is one ulp wrong
+
+`ComputeBankruptcyLimits` divides by `-0.15`. The constant the original loads is the double
+`-0.15000000596046448`, which is `(double)(float)-0.15f`. The two differ, and the difference is
+visible on a large enough maximum income.
+
+> **`ftol(maxIncome / -0.15)` and `ftol(maxIncome / -0.15000000596046448)` differ for some
+> `maxIncome` reachable in a real game**, and the smallest such value is small enough to be found by
+> a linear scan.
+
+This lane does not fix it: the file is another lane's. It is measured and reported.
+
+**Falsification.** If no `maxIncome` in the int32 range separates the two constants, the correction
+is real but inert and should be recorded as such.
+
+---
+
+# Outcomes
+
+Written after the run, against the predictions above. Nothing above was edited.
+
+## P-Y1 — held exactly
+
+16 words on both calibrated pairs, all attributed to the trade-raid phase, `left` 379 and 359. The
+player vector read 8 on both, as the wire says. The falsification table's other branches did not
+fire: not 8 (the NPC gate was open), not 24 (the refugee gate was not).
+
+## P-Y2 — held exactly
+
+Residual 4 on the turn-4→5 pair and 2 on the turn-5→6 pair, matching the per-call-site split
+(2 detection + 2 research, and 2 detection + 0 research). The state block is byte-identical in both
+cases and only `left` differs, so the two instruments disagree about nothing.
+
+**The answer to the milestone question is therefore "no", by 4 words and by 2.**
+
+One thing this pair of numbers does *not* establish, and the prediction should have said so: the
+residual is a subtraction, so it confirms the **sum** of the modelled and unmodelled terms, not each
+term separately. The static bound is what carries the 16; this arithmetic corroborates it.
+
+## P-Y3 — held
+
+480 fields over 80 player-records across 11 saves, 0 mismatches, 0 dangling owned-system ids. The
+same check runs inside every turn and is printed as a phase note.
+
+## P-Y4 — held, and the shape of the result is more useful than the claim
+
+`--commit-blocked` closes 29 and regresses 17 on the first reference pair, and closes 13 and
+regresses 19 on the second. Every regressed leaf is in the unmodelled set: `almem` x8, `inc` x3,
+`sav` x3, three of the ship census. Nothing outside it moved.
+
+So "committing makes it worse" is **not** what the measurement says: on the first pair the turnstats
+subsystem is net +7 and on the second it is net -9. The record is right in six fields and wrong in
+five, and which way the net falls depends on the save. The phase stays blocked, and it becomes a
+clean +24 the moment the alliance mask and the budget land.
+
+## P-Y5 — WRONG in its framing, and the correction is the result
+
+The prediction said the two constants differ "for some maxIncome reachable in a real game" and that
+the smallest such value could be found by a linear scan. The scan found it immediately, and the
+framing -- inherited from the phrase "one ulp out on large empires" -- was the wrong picture
+entirely.
+
+`trunc(m / -0.15)` and `trunc(m / -0.15000000596046448)` differ for **every m divisible by 3**: 1000
+of the first 3000 integers, smallest `m = 3` giving -20 against -19. It is not precision loss at
+scale. `m / 0.15` is an exact integer whenever 3 divides m, and the two constants straddle it.
+
+And the multiples of 3 are only the floor: the disagreement rate is 33.3% up to about 1.2 million,
+66.7% by 1.36 million and 100% at and above 3 million.
+
+Measured on the corpus by inverting the stored limit: **6 of 25 player-records would get a different
+stored value from the engine's current constant than the game wrote.** The defect is live, it is a
+whole unit rather than an ulp, and it is a third of all inputs at small values and all of them at
+large ones -- the opposite shape from "one ulp on large empires", which would have been harmless.
+
+The file belongs to another lane in flight, so it is reported and not fixed here.
+
+## An unpredicted result: the blocked input is recoverable from every save
+
+The elimination limit is injective in the maximum-income sum it is computed from (slope about 6.67),
+so it inverts. Every save in the corpus therefore states, for every player that owns anything, the
+exact sum of its systems' maximum money outputs -- which is the term that blocks `ComputeBudget`,
+both savings phases, the bankruptcy limits and five fields of the turn record.
+
+That was not predicted and is the most useful thing this lane found. The same inversion recovers the
+protection-limit factor as 3.3 from the saves, independently of the game data files.

src/app/CMakeLists.txt

@@ -5,6 +5,8 @@
 
 add_library(sots_app STATIC
   phase_catalog.cpp
+  trade_raid.cpp
+  turn_record.cpp
   turn.cpp
   report.cpp)
 target_include_directories(sots_app PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/..)

src/app/phase_catalog.cpp

@@ -107,11 +107,14 @@ constexpr PhaseDesc kStrategic[] = {
      "draw-free, and the cheapest unimplemented phase in this table"},
     {Driver::Strategic, 29, "S29", "SystemTailFixup", PhaseStatus::Stub, ""},
     {Driver::Strategic, 30, "S30", "BuildTeamPartition", PhaseStatus::Stub, ""},
-    {Driver::Strategic, 31, "S31", "EncounterDetectionAndStatusRestore", PhaseStatus::Blocked,
+    {Driver::Strategic, 31, "S31", "EncounterDetectionAndStatusRestore", PhaseStatus::Partial,
-     "encounter detection is not modelled. The player-status restore that follows it IS -- it "
+     "trade-raid generation runs first here and IS modelled: two chances per player, one word "
-     "writes 1 -- but the value the file carries is 4, so a further writer between this phase "
+     "each, neither site inside a back edge, so the count is a bound -- it is the turn's "
-     "and the autosave is unaccounted. Committing the 1 turned two agreeing leaves into "
+     "dominant generator cost and it is committed under --commit-rng. Detection proper is not "
-     "disagreeing ones on the turn2->turn3 pair, so it is evaluated and reported instead"},
+     "modelled and spends two further words. The player-status restore that follows it IS "
+     "modelled -- it writes 1 -- but the value the file carries is 4, so a further writer "
+     "between this phase and the autosave is unaccounted. Committing the 1 turned two agreeing "
+     "leaves into disagreeing ones on the turn2->turn3 pair, so it is evaluated and reported"},
 };
 
 // ---------------------------------------------------------------------------------------
@@ -209,8 +212,13 @@ constexpr PhaseDesc kTail[] = {
     {Driver::Tail, 33, "T33", "ShipManagerEndOfTurnHooks", PhaseStatus::Stub, ""},
     {Driver::Tail, 34, "T34", "RecordObservedDesigns", PhaseStatus::Stub, ""},
     {Driver::Tail, 35, "T35", "RebuildPlayerReports", PhaseStatus::Stub, ""},
-    {Driver::Tail, 36, "T36", "FinalizeTurnRecords", PhaseStatus::Stub,
+    {Driver::Tail, 36, "T36", "FinalizeTurnRecords", PhaseStatus::Blocked,
-     "fills every player's turn record and archives it by turn; must stay last"},
+     "fills every player's turn record and archives it by turn; must stay last. Six of its "
+     "fields are recoverable from the wire and are reproduced -- turn, colony count, savings, "
+     "the savings delta, the completed-tech count and the summed population -- and the model is "
+     "self-checked every run against the record the input save already carries for its own "
+     "turn. Not committed: five further fields of the same record are unmodelled, and savings "
+     "for the NEW turn comes from a blocked phase"},
 };
 
 PhaseTally Tally(const PhaseDesc* p, std::size_t n) {

src/app/report.cpp

@@ -75,8 +75,13 @@ void PrintSummary(std::FILE* out, const TurnResult& r) {
     std::fprintf(out, "\nthis run\n");
     std::fprintf(out, "  leaves written      %d\n", r.leafWrites);
     std::fprintf(out, "  leaves NOT written by a blocked phase  %d\n", r.wouldWrite);
-    std::fprintf(out, "  generator words consumed  %d  (state %s)\n", r.rngWords,
+    std::fprintf(out, "  generator words consumed  %d  (state %s, %s)\n", r.rngWords,
-                 r.rngLoaded ? "loaded" : "UNREADABLE");
+                 r.rngLoaded ? "loaded" : "UNREADABLE",
+                 r.rngCommitted ? "WRITTEN BACK" : "left untouched");
+    if (!r.rngUnaccounted.empty()) {
+        std::fprintf(out, "  generator words NOT accounted (never netted off the above):\n");
+        for (const auto& u : r.rngUnaccounted) std::fprintf(out, "      - %s\n", u.c_str());
+    }
     for (const auto& w : r.warnings) std::fprintf(out, "  ! %s\n", w.c_str());
 }
 
@@ -96,7 +101,11 @@ bool WriteMetricJson(const std::string& path, const TurnResult& r, const std::st
     f << ",\n";
     f << "  \"run\": {\"leafWrites\": " << r.leafWrites << ", \"blockedLeafWrites\": "
       << r.wouldWrite << ", \"rngWords\": " << r.rngWords << ", \"rngLoaded\": "
-      << (r.rngLoaded ? "true" : "false") << "},\n";
+      << (r.rngLoaded ? "true" : "false") << ", \"rngCommitted\": "
+      << (r.rngCommitted ? "true" : "false") << ", \"rngUnaccounted\": [";
+    for (std::size_t i = 0; i < r.rngUnaccounted.size(); ++i)
+        f << (i ? ", " : "") << '"' << JsonEscape(r.rngUnaccounted[i]) << '"';
+    f << "]},\n";
     f << "  \"phases\": [\n";
     bool first = true;
     for (const auto& rec : r.records) {

src/app/trade_raid.cpp

@@ -0,0 +1,37 @@
+#include "app/trade_raid.h"
+
+namespace sots::app {
+namespace {
+
+// The chance helper with its word cost reported rather than assumed: the two early-outs
+// return WITHOUT drawing, so a probability outside (0, 1) costs nothing. Counting the draws
+// instead of the calls is what keeps the ledger honest if a loaded tuning table ever pushes
+// one of the three odds to 0 or 1.
+bool ChanceCounted(sots::sim::IRandom& rng, float p, int& words) {
+    if (p <= 0.f) return false;
+    if (p >= 1.f) return true;
+    ++words;
+    return p > rng.NextFloat();
+}
+
+}  // namespace
+
+bool Chance(sots::sim::IRandom& rng, float p) {
+    int ignored = 0;
+    return ChanceCounted(rng, p, ignored);
+}
+
+TradeRaidResult RollTradeRaids(sots::sim::IRandom& rng, int players, const TradeRaidOdds& odds,
+                               const TradeRaidGates& gates) {
+    TradeRaidResult r;
+    if (players <= 0) return r;
+    r.players = players;
+    for (int i = 0; i < players; ++i) {
+        if (ChanceCounted(rng, odds.player, r.words)) ++r.playerRaidHits;
+        if (gates.npcRaids && ChanceCounted(rng, odds.npc, r.words)) ++r.npcRaidHits;
+        if (gates.refugeeRaids && ChanceCounted(rng, odds.refugee, r.words)) ++r.refugeeRaidHits;
+    }
+    return r;
+}
+
+}  // namespace sots::app

src/app/trade_raid.h

@@ -0,0 +1,55 @@
+// Trade-raid encounter generation -- the strategic turn's dominant RNG consumer.
+//
+// The original runs this from inside encounter detection, over the server's player vector,
+// rolling up to three chances per player. Two of the three roll on every turn measured; the
+// third is behind a subsystem that was not present. The important property, and the reason
+// this is modelled at all while the rest of encounter detection is not, is that NO BACK EDGE
+// contains any of the three roll sites: the cost is one word per player per firing site, as a
+// bound rather than as an observation.
+//
+// The probabilities are simulation constants that come from the game data. Their in-image
+// defaults are used when no tuning table is loaded; either way all three are strictly inside
+// (0, 1), so the chance helper takes neither of its early-outs and every roll is exactly one
+// word. That is why the WORD COUNT is independent of the probabilities and the RAID DECISIONS
+// are not.
+#pragma once
+
+#include "game/sim/rng.h"
+
+namespace sots::app {
+
+// The chance helper's contract, transcribed: p <= 0 is false with NO draw, p >= 1 is true with
+// NO draw, otherwise exactly one draw and the test is `p > r` -- equality is a failure. A NaN
+// probability falls through both early-outs and draws.
+bool Chance(sots::sim::IRandom& rng, float p);
+
+struct TradeRaidOdds {
+    // In-image defaults. A loaded tuning table overrides them; the word cost does not move.
+    float player = 0.2f;
+    float npc = 0.05f;
+    float refugee = 0.05f;
+};
+
+struct TradeRaidGates {
+    // The NPC-raid roll is behind a single server-level test that does not depend on the
+    // player, so it is all-or-nothing for the whole turn. Measured open on 8 of 8 turns
+    // across two saves; it is not on the wire, so this is a HYPOTHESIS with a stated default.
+    bool npcRaids = true;
+    // The refugee-raid roll needs a subsystem manager that was absent on every measured turn.
+    bool refugeeRaids = false;
+};
+
+struct TradeRaidResult {
+    int players = 0;        // entries of the player vector the loop visited
+    int words = 0;          // generator words consumed
+    int playerRaidHits = 0; // rolls that SUCCEEDED -- each one may cost a further word we do
+    int npcRaidHits = 0;    //   not model (the target-selection draw), so a non-zero total
+    int refugeeRaidHits = 0;//   here is the signal that the ledger is about to fall short
+};
+
+// Roll the raid chances for `players` players, in the original's order: player raid, then NPC
+// raid, then refugee raid, per player, before moving to the next player.
+TradeRaidResult RollTradeRaids(sots::sim::IRandom& rng, int players, const TradeRaidOdds& odds,
+                               const TradeRaidGates& gates);
+
+}  // namespace sots::app

src/app/turn.cpp

@@ -7,6 +7,8 @@
 #include <cstring>
 #include <string>
 
+#include "app/trade_raid.h"
+#include "app/turn_record.h"
 #include "game/sim/colony.h"
 #include "game/sim/economy.h"
 #include "game/sim/numeric.h"
@@ -253,6 +255,111 @@ void RunSystemTurn(Sys& s, int playerCount, SystemTotals& t) {
     (void)playerCount;
 }
 
+// ---------------------------------------------------------------------------------------
+// The tail's last phase: the per-player turn record, and its own self-check
+// ---------------------------------------------------------------------------------------
+
+// The record this phase would archive for the turn just run, and -- separately -- what the
+// same model produces for the turn the INPUT save was written at, where the save already
+// carries the answer. The second is a check of the model that needs no running game: it is
+// the "testable on load" property of this phase.
+struct TurnRecordAudit {
+    int playersChecked = 0;      // players whose input-turn record could be compared
+    int fieldsCompared = 0;
+    int mismatches = 0;
+    int dangling = 0;            // owned-system ids the save's system table does not carry
+    std::vector<std::string> firstMismatches;
+    int missingArchive = 0;      // players with no archive element for the input turn
+};
+
+TurnRecordAudit AuditTurnRecordsAgainstSave(const SaveGame& game) {
+    TurnRecordAudit a;
+    const std::int32_t inputTurn = game.sim.frame;
+    for (std::size_t i = 0; i < game.sim.players.size(); ++i) {
+        if (i >= game.sim.turnstats.players.size()) break;
+        const auto& hist = game.sim.turnstats.players[i].hist;
+        const auto* stored = FindArchivedRecord(hist, inputTurn);
+        if (!stored) {
+            ++a.missingArchive;
+            continue;
+        }
+        int dangling = 0;
+        const TurnRecord built = BuildTurnRecord(game.sim.players[i].player, game.sim.systems,
+                                                 inputTurn, &dangling);
+        a.dangling += dangling;
+        const TurnRecordDiff d = CompareTurnRecord(built, *stored);
+        ++a.playersChecked;
+        a.fieldsCompared += d.compared;
+        a.mismatches += static_cast<int>(d.mismatches.size());
+        for (const auto& m : d.mismatches)
+            if (a.firstMismatches.size() < 8)
+                a.firstMismatches.push_back(fmt("player %zu: %s", i, m.c_str()));
+    }
+    return a;
+}
+
+void RunFinalizeTurnRecords(SaveGame& game, const TurnOptions& opt, PhaseRecord& rec,
+                            const TurnRecordAudit& audit) {
+    rec.invocations = static_cast<int>(game.sim.players.size());
+    // Six fields per player would be written, plus a new archive element per player. Nothing
+    // is committed by default: five further fields of the same element are unmodelled, and
+    // the two the model DOES hold for the new turn -- savings and the income derived from it
+    // -- are downstream of a blocked phase, so every element written would be wrong in a way
+    // the untouched save is not. `--commit-blocked` writes them anyway, so the claim that
+    // committing makes things worse is a measurement rather than an argument.
+    int archived = 0;
+    for (std::size_t i = 0; i < game.sim.players.size(); ++i) {
+        if (i >= game.sim.turnstats.players.size()) break;
+        auto& hist = game.sim.turnstats.players[i].hist;
+        if (FindArchivedRecord(hist, game.sim.frame)) continue;  // the key is the turn
+        if (!opt.commitBlocked) {
+            ++archived;
+            continue;
+        }
+        const TurnRecord built =
+            BuildTurnRecord(game.sim.players[i].player, game.sim.systems, game.sim.frame);
+        mars::stream::shapes::PlayerTurnStats s;
+        s.trn = built.turn;
+        s.pop = built.population;
+        s.col = built.colonies;
+        s.sav = built.savings;
+        s.inc = built.income;
+        s.tch = built.completedTech;
+        // The census is three hull classes wide whether or not the player owns a ship; the
+        // counts are the unmodelled part, the shape is not.
+        for (std::int32_t c = 0; c < 3; ++c) {
+            mars::stream::shapes::ClassStats cs;
+            cs.cls = c;
+            s.classes.push_back(cs);
+        }
+        hist.stats.push_back(s);
+        ++archived;
+        rec.leafWrites += 6;
+    }
+    rec.committed = rec.leafWrites > 0;
+    if (!opt.commitBlocked) rec.wouldWrite = archived * 6;
+    rec.notes.push_back(fmt("%s %d record(s) for turn %d; 6 modelled field(s) each",
+                            opt.commitBlocked ? "ARCHIVED" : "would archive", archived,
+                            game.sim.frame));
+    if (audit.playersChecked)
+        rec.notes.push_back(fmt("SELF-CHECK on the input turn, where the save carries the "
+                                "answer: %d field(s) over %d player(s), %d mismatch(es)%s",
+                                audit.fieldsCompared, audit.playersChecked, audit.mismatches,
+                                audit.dangling ? " (owned-system ids missing from the table!)"
+                                               : ""));
+    for (const auto& m : audit.firstMismatches) rec.notes.push_back(m);
+    if (audit.missingArchive)
+        rec.notes.push_back(fmt("%d player(s) carry no archive element for the input turn",
+                                audit.missingArchive));
+    std::size_t nUnmodelled = 0;
+    const char* const* un = TurnRecord::Unmodelled(nUnmodelled);
+    for (std::size_t i = 0; i < nUnmodelled; ++i) rec.notes.push_back(fmt("NOT modelled: %s", un[i]));
+    rec.notes.push_back("and the two fields the model does hold for the NEW turn -- savings and "
+                        "the income derived from it -- come from a blocked phase, so committing "
+                        "this would replace one container-shaped divergence per player with "
+                        "several wrong leaves");
+}
+
 }  // namespace
 
 // ---------------------------------------------------------------------------------------
@@ -297,6 +404,11 @@ void ApplySaveWriterInvariants(SaveGame& game, TurnResult& r) {
 TurnResult RunStrategicTurn(SaveGame& game, const TurnOptions& opt) {
     TurnResult r;
 
+    // Taken before any phase runs: the turn-record model is checked against the record the
+    // INPUT save already carries for its own turn. None of the six fields it reads is written
+    // by a phase below, but the check is taken first so that stays true by construction.
+    const TurnRecordAudit recordAudit = AuditTurnRecordsAgainstSave(game);
+
     mars::rng::MT19937 gen(1u);
     r.rngLoaded = LoadGenerator(game, gen);
     if (!r.rngLoaded)
@@ -374,6 +486,30 @@ TurnResult RunStrategicTurn(SaveGame& game, const TurnOptions& opt) {
                 break;
             }
             case 31: {  // S31 EncounterDetectionAndStatusRestore
+                // Trade-raid generation runs FIRST inside this phase, before detection
+                // proper, and it is the turn's dominant RNG consumer: two chances per entry
+                // of the player vector, neither inside a back edge, so the count is a bound.
+                if (r.rngLoaded) {
+                    const int before = rng.words();
+                    const TradeRaidResult tr =
+                        RollTradeRaids(rng, static_cast<int>(game.sim.players.size()),
+                                       TradeRaidOdds{}, TradeRaidGates{});
+                    rec.rngWords = rng.words() - before;
+                    rec.notes.push_back(fmt("trade raids: %d player(s) x 2 rolls = %d generator "
+                                            "word(s), one per roll (neither site sits inside a "
+                                            "back edge, so this is a bound)",
+                                            tr.players, rec.rngWords));
+                    rec.notes.push_back("the refugee-raid roll is NOT counted: its subsystem was "
+                                        "absent on all 8 measured turns. If it is ever present "
+                                        "the turn costs one more word per player");
+                    if (tr.playerRaidHits || tr.npcRaidHits)
+                        rec.notes.push_back(fmt("%d roll(s) succeeded in THIS run -- a success "
+                                                "may cost a further target-selection word that "
+                                                "is not modelled. The values drawn here are not "
+                                                "the game's: earlier unmodelled draws shift the "
+                                                "stream, so the hit COUNT is not a prediction",
+                                                tr.playerRaidHits + tr.npcRaidHits));
+                }
                 // The status restore: every player that is not an AI, or whose secondary AI
                 // flag is set, goes back to status 1. The secondary flag is not on the wire,
                 // so the AI test alone is used and the difference is reported.
@@ -447,6 +583,8 @@ TurnResult RunStrategicTurn(SaveGame& game, const TurnOptions& opt) {
             rec.leafWrites = 1;
             rec.committed = true;
             rec.notes.push_back(fmt("ModCount -> %d", game.sim.modCount));
+        } else if (tp[i].index == 36) {
+            RunFinalizeTurnRecords(game, opt, rec, recordAudit);
         }
         r.records.push_back(rec);
     }
@@ -472,14 +610,35 @@ TurnResult RunStrategicTurn(SaveGame& game, const TurnOptions& opt) {
         r.rngWords += rec.rngWords;
     }
 
+    // The RNG ledger, stated the way the campaign states divergence: what is accounted and
+    // what is not, never netted into one number. A measured turn on the reference save costs
+    // 18-22 words; what is modelled here is the trade-raid block and the research-event roll.
+    r.rngUnaccounted.push_back(
+        "encounter detection draws one unit value and one bounded integer per turn on every "
+        "turn measured (2 words), with no derived rule behind the count -- its bound is the "
+        "product of the contact and detector counts, so it is left unmodelled");
+    r.rngUnaccounted.push_back(
+        "the research-allocation draw is downstream of the budget, which is blocked on the "
+        "per-system money output (0 or 1 word)");
+    if (r.rngLoaded && r.rngWords > 0)
+        r.rngUnaccounted.push_back(
+            "a successful raid roll may draw one further word to pick its target; no roll "
+            "succeeded on any measured turn, so the cost of a success is 0 or 1 and undetermined");
+
     if (opt.commitRng && r.rngLoaded) {
         if (!StoreGenerator(game, gen))
             r.warnings.push_back("generator state could not be written back");
+        else
+            r.rngCommitted = true;
     } else if (r.rngWords > 0) {
         r.warnings.push_back(
             "the generator advanced during this run but the save keeps its original state "
-            "(--commit-rng to write it); the turn's full draw count is not yet attributed");
+            "(--commit-rng to write it)");
     }
+    if (r.rngLoaded)
+        r.warnings.push_back(
+            "the modelled words are a LOWER BOUND on the turn's cost, so a committed generator "
+            "is short by the unaccounted sites below and its drawn VALUES are not the game's");
 
     return r;
 }

src/app/turn.h

@@ -44,8 +44,13 @@ struct TurnResult {
     std::vector<PhaseRecord> records;  // spine order, with player phases nested after S13
     int leafWrites = 0;
     int wouldWrite = 0;
-    int rngWords = 0;
+    int rngWords = 0;       // words this run actually consumed -- a LOWER bound on the turn's
     bool rngLoaded = false;
+    bool rngCommitted = false;
+    // Draw sites a real turn spends words at that this model does not reproduce. Kept next to
+    // rngWords and never subtracted from it: an accounted count and an unaccounted list are
+    // two different facts, and folding them into one hides which is which.
+    std::vector<std::string> rngUnaccounted;
     std::vector<std::string> warnings;
 };
 

src/app/turn_record.cpp

@@ -0,0 +1,87 @@
+#include "app/turn_record.h"
+
+#include <cstdio>
+
+namespace sots::app {
+
+namespace {
+constexpr const char* kUnmodelled[] = {
+    "alliance/vision mask -- rebuilt by the alliance-mask phase of the spine, which is a stub",
+    "trade income -- a budget-derived field, blocked behind the per-system money output",
+    "battles fought -- written by the tail's battle-tally phase, which is a stub",
+    "systems acquired / lost this turn -- the two counted lists are never non-empty in the "
+    "corpus, so their element meaning is unobserved",
+    "the per-hull-class ship census -- needs each design's hull size and its class flag, "
+    "neither of which is on the wire; both come from the game data",
+};
+}  // namespace
+
+const char* const* TurnRecord::Unmodelled(std::size_t& count) {
+    count = sizeof(kUnmodelled) / sizeof(kUnmodelled[0]);
+    return kUnmodelled;
+}
+
+TurnRecord BuildTurnRecord(const mars::stream::shapes::Player& p,
+                           const std::vector<mars::stream::shapes::SysEntry>& systems,
+                           std::int32_t frame, int* danglingOwnedSystems) {
+    TurnRecord r;
+    r.turn = frame;
+    r.savings = p.sav;
+    // The turn's income is the change in savings across the turn, not a budget line: the
+    // previous-turn savings word is on the wire and is stamped before this turn's savings are
+    // written, so the subtraction is exact.
+    r.income = p.sav - p.pvSav;
+    r.colonies = static_cast<std::int32_t>(p.owners.size());
+
+    for (const auto& st : p.techTree.state)
+        if (st.st == kTechStateCompleted) ++r.completedTech;
+
+    // The population term is a sum of TWO per-system words: the live population and the
+    // pending population bonus that has not been folded into it yet. Both are signed 32-bit
+    // and the accumulator is 64-bit, so the sum is taken in 64-bit and never wraps.
+    int dangling = 0;
+    for (std::int32_t id : p.owners) {
+        const mars::stream::shapes::Sys* sys = nullptr;
+        for (const auto& e : systems)
+            if (e.sysID == id) {
+                sys = &e.sys;
+                break;
+            }
+        if (!sys) {
+            ++dangling;
+            continue;
+        }
+        r.population += static_cast<std::int64_t>(sys->pop) + static_cast<std::int64_t>(sys->pbon);
+    }
+    if (danglingOwnedSystems) *danglingOwnedSystems = dangling;
+    return r;
+}
+
+const mars::stream::shapes::PlayerTurnStats* FindArchivedRecord(
+    const mars::stream::shapes::PlayerTurnHistory& hist, std::int32_t turn) {
+    for (const auto& s : hist.stats)
+        if (s.trn == turn) return &s;
+    return nullptr;
+}
+
+TurnRecordDiff CompareTurnRecord(const TurnRecord& built,
+                                 const mars::stream::shapes::PlayerTurnStats& stored) {
+    TurnRecordDiff d;
+    auto cmp = [&](const char* field, long long a, long long b) {
+        ++d.compared;
+        if (a != b) {
+            char buf[160];
+            std::snprintf(buf, sizeof buf, "%s: built %lld != stored %lld", field, a, b);
+            d.mismatches.push_back(buf);
+        }
+    };
+    cmp("trn", built.turn, stored.trn);
+    cmp("pop", built.population, stored.pop);
+    cmp("col", built.colonies, stored.col);
+    cmp("sav", built.savings, stored.sav);
+    cmp("inc", built.income, stored.inc);
+    cmp("tch", built.completedTech, stored.tch);
+    return d;
+}
+
+}  // namespace sots::app

src/app/turn_record.h

@@ -0,0 +1,62 @@
+// The per-player turn record -- the last phase of the post-combat tail.
+//
+// Each player carries a small "what my empire looked like this turn" summary that the final
+// tail phase fills and then copies into a history archive keyed by the turn number. The
+// archive IS on the wire: it is the per-player turn-statistics history, one element per turn,
+// and every save carries an element for its own frame. That makes this phase testable without
+// a running game: build the record from a save's own state and compare it with the element the
+// save already holds for that turn.
+//
+// Six fields are recoverable from the wire and are reproduced here. Five are not, and each is
+// named with the reason -- they belong to phases or inputs the standalone does not hold.
+#pragma once
+
+#include <cstdint>
+#include <string>
+#include <vector>
+
+#include "mars/stream/shapes.h"
+
+namespace sots::app {
+
+// The part of a turn record this model can produce.
+struct TurnRecord {
+    std::int32_t turn = 0;          // the frame counter
+    std::int64_t population = 0;    // summed over owned systems
+    std::int32_t colonies = 0;      // owned-system count
+    std::int32_t savings = 0;
+    std::int32_t income = 0;        // savings minus previous-turn savings
+    std::int32_t completedTech = 0; // tech-tree entries in the completed state
+
+    // Fields the archive element also carries that this model does NOT fill, kept as a
+    // published list rather than as silence. Each is blocked on something named.
+    static const char* const* Unmodelled(std::size_t& count);
+};
+
+// The completed state's value in the tech-tree state word. Named rather than spelled inline
+// because it is the one magic number in this file.
+constexpr std::int32_t kTechStateCompleted = 4;
+
+// Build one player's record from the simulation state. `systemsById` is the save's system
+// table indexed the way the player's owned-system ids index it.
+TurnRecord BuildTurnRecord(const mars::stream::shapes::Player& p,
+                           const std::vector<mars::stream::shapes::SysEntry>& systems,
+                           std::int32_t frame,
+                           // set when an owned-system id is not present in the table, which
+                           // would silently drop a term from the population sum
+                           int* danglingOwnedSystems = nullptr);
+
+// What the archive element for `turn` holds, if the save carries one for that turn.
+const mars::stream::shapes::PlayerTurnStats* FindArchivedRecord(
+    const mars::stream::shapes::PlayerTurnHistory& hist, std::int32_t turn);
+
+// A comparison of a built record against a stored one, over the six modelled fields only.
+struct TurnRecordDiff {
+    int compared = 0;
+    std::vector<std::string> mismatches;  // "field: built != stored"
+    bool ok() const { return mismatches.empty(); }
+};
+TurnRecordDiff CompareTurnRecord(const TurnRecord& built,
+                                 const mars::stream::shapes::PlayerTurnStats& stored);
+
+}  // namespace sots::app

tests/app/CMakeLists.txt

@@ -11,7 +11,17 @@ add_executable(app_test_turn test_turn.cpp)
 target_link_libraries(app_test_turn PRIVATE sots_app)
 add_test(NAME app_turn COMMAND app_test_turn)
 
-foreach(_t app_test_catalog app_test_turn)
+# The trade-raid word count is pinned against a scripted generator, so it always runs.
+add_executable(app_test_trade_raid test_trade_raid.cpp)
+target_link_libraries(app_test_trade_raid PRIVATE sots_app)
+add_test(NAME app_trade_raid COMMAND app_test_trade_raid)
+
+# The turn-record model against the record the game itself archived; needs the owner's saves.
+add_executable(app_test_turn_record test_turn_record.cpp)
+target_link_libraries(app_test_turn_record PRIVATE sots_app)
+add_test(NAME app_turn_record COMMAND app_test_turn_record)
+
+foreach(_t app_test_catalog app_test_turn app_test_trade_raid app_test_turn_record)
   target_include_directories(${_t} PRIVATE ${CMAKE_CURRENT_SOURCE_DIR})
   target_compile_options(${_t} PRIVATE -Wall -Wextra -Wpedantic)
 endforeach()

tests/app/test_trade_raid.cpp

@@ -0,0 +1,142 @@
+// The trade-raid block is the strategic turn's dominant generator cost, so its WORD COUNT is
+// load-bearing in a way its decisions are not. These tests pin the count, the two no-draw
+// early-outs of the chance helper, and the roll order -- all against a scripted generator, so
+// nothing here depends on the real MT19937.
+#include <cstdio>
+#include <vector>
+
+#include "app/trade_raid.h"
+#include "mars/rng/mt19937.h"
+
+static int failures = 0;
+#define CHECK(c)                                                     \
+    do {                                                             \
+        if (!(c)) {                                                  \
+            std::printf("FAIL %s:%d  %s\n", __FILE__, __LINE__, #c); \
+            ++failures;                                              \
+        }                                                            \
+    } while (0)
+
+namespace {
+
+// A generator that hands out a scripted sequence and counts what it handed out.
+class Scripted final : public sots::sim::IRandom {
+public:
+    explicit Scripted(std::vector<float> f) : f_(std::move(f)) {}
+    float NextFloat() override {
+        ++draws;
+        return f_.empty() ? 0.f : f_[(cursor_++) % f_.size()];
+    }
+    std::uint32_t NextIntInclusive(std::uint32_t) override {
+        ++draws;
+        return 0;
+    }
+    std::uint32_t NextUInt32() override {
+        ++draws;
+        return 0;
+    }
+    int draws = 0;
+
+private:
+    std::vector<float> f_;
+    std::size_t cursor_ = 0;
+};
+
+}  // namespace
+
+int main() {
+    using namespace sots::app;
+
+    // --- the chance helper's two no-draw early-outs ------------------------------------
+    {
+        Scripted g({0.5f});
+        CHECK(Chance(g, 0.f) == false);
+        CHECK(Chance(g, -1.f) == false);
+        CHECK(Chance(g, 1.f) == true);
+        CHECK(Chance(g, 2.f) == true);
+        CHECK(g.draws == 0);  // four calls, zero words: both early-outs return before drawing
+    }
+    // --- the comparison is `p > r`, so equality FAILS and still costs a word -----------
+    {
+        Scripted g({0.25f, 0.25f});
+        CHECK(Chance(g, 0.25f) == false);  // p == r -> false
+        CHECK(g.draws == 1);
+        CHECK(Chance(g, 0.5f) == true);  // p > r -> true
+        CHECK(g.draws == 2);
+    }
+    // --- the word count is two per player when the NPC gate is open -------------------
+    {
+        Scripted g({0.9f});  // every roll fails; the count must not depend on that
+        const TradeRaidResult r = RollTradeRaids(g, 8, TradeRaidOdds{}, TradeRaidGates{});
+        CHECK(r.players == 8);
+        CHECK(r.words == 16);
+        CHECK(g.draws == 16);
+        CHECK(r.playerRaidHits == 0);
+        CHECK(r.npcRaidHits == 0);
+        CHECK(r.refugeeRaidHits == 0);
+    }
+    // --- ... and the count is unchanged when every roll succeeds -----------------------
+    {
+        Scripted g({0.0f});  // every roll succeeds (p > 0)
+        const TradeRaidResult r = RollTradeRaids(g, 8, TradeRaidOdds{}, TradeRaidGates{});
+        CHECK(r.words == 16);
+        CHECK(r.playerRaidHits == 8);
+        CHECK(r.npcRaidHits == 8);
+    }
+    // --- one per player when the NPC gate is shut, three when the refugee one is open --
+    {
+        Scripted g({0.9f});
+        TradeRaidGates gates;
+        gates.npcRaids = false;
+        CHECK(RollTradeRaids(g, 7, TradeRaidOdds{}, gates).words == 7);
+        gates.npcRaids = true;
+        gates.refugeeRaids = true;
+        Scripted g2({0.9f});
+        CHECK(RollTradeRaids(g2, 7, TradeRaidOdds{}, gates).words == 21);
+    }
+    // --- a degenerate player vector costs nothing --------------------------------------
+    {
+        Scripted g({0.9f});
+        CHECK(RollTradeRaids(g, 0, TradeRaidOdds{}, TradeRaidGates{}).words == 0);
+        CHECK(RollTradeRaids(g, -3, TradeRaidOdds{}, TradeRaidGates{}).words == 0);
+        CHECK(g.draws == 0);
+    }
+    // --- odds pinned at 0 or 1 cost NOTHING, and the counter says so -------------------
+    // This is the case that would silently break a count-the-calls implementation: a tuning
+    // table that pushes an odds value out of (0, 1) removes the draw, not just the outcome.
+    {
+        Scripted g({0.9f});
+        TradeRaidOdds odds;
+        odds.player = 0.f;
+        odds.npc = 1.f;
+        const TradeRaidResult r = RollTradeRaids(g, 8, odds, TradeRaidGates{});
+        CHECK(r.words == 0);
+        CHECK(g.draws == 0);
+        CHECK(r.playerRaidHits == 0);
+        CHECK(r.npcRaidHits == 8);
+    }
+    // --- against the real generator: 16 words is 16 words -----------------------------
+    {
+        mars::rng::MT19937 gen(12345u);
+        class Adapt final : public sots::sim::IRandom {
+        public:
+            explicit Adapt(mars::rng::MT19937& g) : g_(g) {}
+            float NextFloat() override { return g_.next_float(); }
+            std::uint32_t NextIntInclusive(std::uint32_t n) override {
+                return g_.next_int_inclusive(n);
+            }
+            std::uint32_t NextUInt32() override { return g_.next_u32(); }
+
+        private:
+            mars::rng::MT19937& g_;
+        } a(gen);
+        const int before = gen.index();
+        const TradeRaidResult r = RollTradeRaids(a, 8, TradeRaidOdds{}, TradeRaidGates{});
+        CHECK(r.words == 16);
+        CHECK(gen.index() - before == 16);
+    }
+
+    std::printf(failures ? "trade_raid: FAILED (%d)\n" : "trade_raid: ok (%d failures)\n",
+                failures);
+    return failures ? 1 : 0;
+}

tests/app/test_turn_record.cpp

@@ -0,0 +1,113 @@
+// The turn-record model, checked against the record the game itself wrote.
+//
+// The last phase of the post-combat tail archives a per-player summary keyed by turn, and the
+// archive is on the wire: every save carries an element for its own frame. So the model can be
+// checked with no running game and no VM -- build the record from the save's own state and
+// compare it with the element the save already holds.
+//
+// This is the strongest check available to a lane that holds no game: it compares against
+// bytes the original produced. It is NOT the same as `verified` in the phase catalog, which
+// means "compared against the live game", and nothing here is that.
+//
+// Reads $SOTS_SAVES_DIR at run time and skips cleanly when it is unset. No .sav enters this
+// repo.
+#include <cstdio>
+#include <cstdlib>
+#include <string>
+#include <vector>
+
+#include <dirent.h>
+
+#include "app/turn_record.h"
+#include "mars/stream/save.h"
+
+static int failures = 0;
+#define CHECK(c)                                                     \
+    do {                                                             \
+        if (!(c)) {                                                  \
+            std::printf("FAIL %s:%d  %s\n", __FILE__, __LINE__, #c); \
+            ++failures;                                              \
+        }                                                            \
+    } while (0)
+
+int main() {
+    const char* dir = std::getenv("SOTS_SAVES_DIR");
+    if (!dir || !*dir) {
+        std::printf("app_test_turn_record: SOTS_SAVES_DIR unset, skipped\n");
+        return 0;
+    }
+    DIR* d = opendir(dir);
+    if (!d) {
+        std::fprintf(stderr, "app_test_turn_record: cannot open %s\n", dir);
+        return 1;
+    }
+    std::vector<std::string> saves;
+    while (struct dirent* e = readdir(d)) {
+        const std::string n = e->d_name;
+        if (n.size() > 4 && n.compare(n.size() - 4, 4, ".sav") == 0)
+            saves.push_back(std::string(dir) + "/" + n);
+    }
+    closedir(d);
+    if (saves.empty()) {
+        std::printf("app_test_turn_record: no .sav in %s, skipped\n", dir);
+        return 0;
+    }
+
+    int files = 0, players = 0, fields = 0, dangling = 0, noArchive = 0;
+    for (const std::string& path : saves) {
+        mars::stream::SaveDocument doc;
+        try {
+            doc = mars::stream::read_save_file(path);
+        } catch (const std::exception& ex) {
+            std::printf("  %s: unreadable (%s), skipped\n", path.c_str(), ex.what());
+            continue;
+        }
+        if (doc.count(mars::stream::Issue::Error)) {
+            std::printf("  %s: parse errors, skipped\n", path.c_str());
+            continue;
+        }
+        ++files;
+        const auto& sim = doc.game.sim;
+        int filePlayers = 0, fileFields = 0, fileBad = 0;
+        for (std::size_t i = 0; i < sim.players.size(); ++i) {
+            if (i >= sim.turnstats.players.size()) break;
+            const auto* stored =
+                sots::app::FindArchivedRecord(sim.turnstats.players[i].hist, sim.frame);
+            if (!stored) {
+                ++noArchive;
+                continue;
+            }
+            int dang = 0;
+            const sots::app::TurnRecord built =
+                sots::app::BuildTurnRecord(sim.players[i].player, sim.systems, sim.frame, &dang);
+            dangling += dang;
+            // An owned-system id the save's table does not carry would drop a term from the
+            // population sum without any other symptom, so it is a failure, not a note.
+            CHECK(dang == 0);
+            const sots::app::TurnRecordDiff diff = sots::app::CompareTurnRecord(built, *stored);
+            ++filePlayers;
+            fileFields += diff.compared;
+            for (const auto& m : diff.mismatches) {
+                std::printf("FAIL %s player %zu: %s\n", path.c_str(), i, m.c_str());
+                ++failures;
+                ++fileBad;
+            }
+        }
+        players += filePlayers;
+        fields += fileFields;
+        std::printf("  %s: turn %d, %d player-record(s), %d field(s), %d mismatch(es)\n",
+                    path.c_str(), sim.frame, filePlayers, fileFields, fileBad);
+    }
+
+    // The check is worthless if it compared nothing -- a green run over zero records is the
+    // failure mode this campaign has paid for twice.
+    CHECK(files > 0);
+    CHECK(players > 0);
+    CHECK(fields == players * 6);
+
+    std::printf("app_test_turn_record: %d save(s), %d player-record(s), %d field(s) compared, "
+                "%d player(s) with no archive element, %d dangling owned-system id(s), "
+                "%d failure(s)\n",
+                files, players, fields, noArchive, dangling, failures);
+    return failures ? 1 : 0;
+}