sots-re/findings/subsystems/golden-trace-recapture.md

Golden-trace recapture with guards (lane R, 2026-09-08)

Every golden trace went stale when the harness grew Coverage + guard regions (sots-engine 3f0721f+, docs/harness-audit.md). This is the recapture on the live game. Read the coverage, not the verdict. Two of the five hooks now report divergences that their previous "verified" rows did not, and both are real.

• Build: recap-7584bad-20260908T0615Z (sots-engine 7584bad + four shim.cfg.recap* variants; no source change). Staged C:\SOTS\shimdist-recap, VM140.
• Workload: ref-turn2.sav → Launch → End Turn. The B3/misc runs continue to turn 7 (5 End Turns) to reach a research completion, which turn 1 does not contain.
• Oracle held on every run's first End Turn: (Autosave EndTurn).sav = bb4fd9ac89f41e3b…, (Autosave).sav = 978041acd168b56e… — trace and compare mode do not perturb the game.
• Artefacts: verify/traces/recap-*.jsonl[.gz], reports in verify/results/compare/recap-*, shim logs + screenshots in verify/results/shim/recap-*.

1. Headline table (coverage first)

hook	run	calls	compared	diverged	exit	coverage line
TechTree::ProcessResearch	recap-b3-compare (t2→t7)	15	15	3	1	15 guarded calls, 13 undeclared writes in 2 calls, 0 unstated, 0 contradicted
TechTree::ProcessResearch	first End Turn only	3	3	1	1	3 guarded calls, 0 undeclared writes
ServerPlayer::ComputeBudget	recap-b1-compare	4284	4284	0	0	4284 guarded calls, 0 undeclared writes
ServerPlayer::OnTechResearched	recap-misc-compare	2	2	0	(1)¹	2 guarded calls, 6 undeclared writes in 2 calls
ServerSystem::ProcessTurn	recap-misc-compare	140	140	0	(1)¹	140 guarded calls, 15 undeclared writes in 6 calls
StrategyServer::MoveFleet	recap-misc-compare	45	45	8	(1)¹	45 guarded calls, 42 undeclared writes in 15 calls
all five	recap-golden-trace (trace mode)	3630	—	—	0	3630 guarded calls, 9 undeclared writes in 4 calls

¹ the three misc hooks share one log; its exit code 1 comes from MoveFleet alone.

2. TechTree::ProcessResearch — the over-budget event is now visible

Yes. The defect became a divergence, on exactly the call the audit predicted.

On the first End Turn, one call of three diverges, and in exactly one field:

### Game::TechTree::ProcessResearch: first 1 of 1 divergent call(s)
- call_id 0 (recap-b3-compare.jsonl:2)
    side.events.after.v.next_id [exact] orig={"t":"i32","v":4} ours={"t":"i32","v":3}

That is bit-for-bit the same signal B3's save-file oracle produced (docs/B3.md: EvNxID 4 → 3, one EVENT_RESEARCH_OVERBUDGET missing across 40,300 items). The 609 KB save diff has been replaced by one line in the compare report.

Corroboration that this is the right branch (lane E: the post site is 0x00587b97, reachable only from the completion-roll-failed path, gated !wasDone && nowDone && owner, and it sets node->flag = 2 in the same branch):

region	before	after (orig)	after (ours)
node[144].progress	2879	5768	5768 ✓
node[144].flag	1	2	2 ✓
events.next_id (= ServerPlayer+0x2b0, EvNxID)	3	4	3 ✗
rng (left 432→431, mt[624] hash, next_index 192→193)			identical ✓
overbudget (the caller's int*)	0	0	0 ✓

So the node arithmetic, the flag and the single RNG draw are all reproduced; the only thing missing is the event. Alloc for that call was {tech 144 (IND_Waldo), 2889 points}, species 2 — B3's call 0 exactly.

Across five turns: 15 calls, 3 divergent

call	alloc	draws	events next_id orig → ours	guard hits
0	{144, 2889}	1	4 → 3	none
1, 2	{90,0}, {9,0}	0	0 → 0	none
3	{144, 2898}	0 (ceiling ⇒ guaranteed)	7 → 5 (two events)	9
4, 5	zero-spend	0	0 → 0	none
6	{9, 3064}	1 (rolled, failed)	8 → 8	none
7, 8	zero-spend	0	0 → 0	none
9	{9, 3074}	0	12 → 10 (two events)	4
10, 11	zero-spend	0	0 → 0	none
12	{18, 3087}	1 (rolled, failed)	14 → 14	none
13, 14	zero-spend	0	0 → 0	none

• RNG matched on 15 of 15 (mt[624] hash, left and next_index identical on every call) — stronger than B3's 14/15, because in this session no tech-effect callback drew.
• Calls 3 and 9 are the completions. Their non-events divergences are exactly the SetResearched-owned fields B3 declared out of scope (cost_rp/state/turn_available on unlocked children; ours leaves them at INT_MAX/0/−1). Nothing new there.
• Note the dedup rule (lane E): a duplicate post inside a turn bucket returns the existing id without advancing EvNxID, so next_id not moving is not proof nothing was posted. Calls 6 and 12 rolled and failed without completing by cost, so no post is expected there.

What the guards mapped: SetResearched, measured

This is the "cheapest map of what SetResearched actually touches" the audit asked for. Undeclared byte runs inside guard:player / guard:tree_header on the two completion calls, resolved against the recovered ServerPlayer table:

offset	len	field	call 3	call 9
player+0x10c/0x110/0x114	3 each	ConMod[0..2] (construction cost per hull class)	✓
player+0x124	3	OutMod (industrial output multiplier)	✓
player+0x130	3	PopMod		✓
player+0x274 / 0x278 / 0x27c	3 each	vector<ObservedTech> otch — all three vector words move (a realloc)	✓	✓ (+0x278 only, 1 byte)
player+0x294	4	ResTNm — the current research target pointer is cleared	✓	✓
tree_header+0x20	1	the tree's completion-order counter	✓	✓

Two things worth carrying forward:

1. tree_header+0x20 is harness-audit row 9, confirmed on the game: the per-node order word is compared but the counter it is read from moves outside every Result region.
2. otch (vector<ObservedTech>) grows on a tech completion, and no coverage note in either B2 or B3 mentions it. It is a third list append in the same neighbourhood as the event list, and it is save state (ServerPlayer serializer). B3's replace-mode oracle never saw it because turn 1 contains no completion.
3. The player+0x10c…0x124 hits are precisely OnTechResearched's own declared regions (§4) — a clean cross-check that the guard is reporting the callee, not noise.

3. ServerPlayer::ComputeBudget — verdict held, and the guard is quiet

4284 calls, 4284 compared, 0 divergences, tracecmp exit 0. The budget_object guard (Budget+0x00..0x68) caught 0 undeclared writes in 4284 calls. In particular the over-budget int at Budget+0x64 (harness-audit row 11 — the word B1 captured only as an argument, and arguments are never compared) never changed value on any call. That does not prove the original never writes it, only that it never writes a different value here.

Coverage, honestly:

• Only 20 distinct (player, budget-output) states exist across the 4284 calls. 4278 of the calls are the strategy-map UI polling the human player (one pointer accounts for 4115 of them); the End-Turn pass itself is ~8 calls, one per player in server order. The headline number is repetition, not evidence.
• 13 of the 22 named slots are 0 on every single call: tradeIncome, shipCarriedPop, secondaryManager, bonusIncome, systemIncomeNeg, debtInterest, construction, expenses, researchMoneyGiven, savingsGiven, tra, researchPointsGiven, trp. The board's "budget tail coverage" backlog row is confirmed, and it is worse than stated: five of the six input-boundary slots (1,2,3,4,7,11 = systemIncomePos, tradeIncome, shipCarriedPop, secondaryManager, systemIncomeNeg, construction) are also always zero, so they prove nothing twice over. Only systemIncomePos carries a value.
• Nine slots ever moved: savings, systemIncomePos (input), savingsInterest, maintenance, researchMoneyKept, available, researchMoney, researchPoints, totalResearchPoints. Eight of those are genuinely modelled. That is the real width of the B1 result.
• Cross-check with B3: the End-Turn call for the AI Tarkas player yields researchPoints = 2889, which is exactly the allocation ProcessResearch received.
• The B1 replace-mode double-run defect was not exercised: this lane ran compare only.

4. ServerPlayer::OnTechResearched — first guarded capture; the roll branch finally fired

2 calls, 2 compared, 0 divergences. The workload had to run to turn 7; the reference turn contains no completion at all, which is why B2 never captured this hook on it.

call	tech	branch reached	declared regions the original moved
71	10001 IND_Waldo, node 144, silent=false	economy modifiers	modifiers (con_mod[0..2] 1.0 → 0.899999976), research_target (ptr → null)
147	10094 IND_TRKSTL, node 142	design-option mask + the pending research roll	design_masks (b 64 → 2097216), research_target, roll, rng

The RollResearchEvent draw is now behaviourally verified — the board's backlog row. On call 147 research_roll_pending was true going in; the original drew exactly one word (left 375 → 374, next_index 249 → 250) and cleared the flag, and ours reproduced both on the scratch generator bit-for-bit. That closes "region declared, ours reproduces it, but the branch never fired".

Guard player (0x3e0 span) findings — B2's known gap is now measured, not assumed:

offset	len	field	calls
player+0x2b0	1	EvNxID (EventStorage+0x14, EventStorage embedded at ServerPlayer+0x29c)	both
player+0x274/0x278/0x27c	3	vector<ObservedTech> otch — the vector grew	71
player+0x278	1	same vector, end pointer only	147

So the completion events (EVENT_RESEARCH_COMPLETE / _UNDERBUDGET / _TEMPERANCE) are confirmed to post on every completion, and ours posts none. The guard names the byte rather than leaving it to a save hash that docs/B2.md gotcha 4 explicitly told readers to ignore.

Not exercised at all: silent = true, a Zuul boarding-pod grant, a plague cure, a node-bore allocation, the temperance sweep, the arcology re-evaluation, or any of the writes to other objects. Two calls, two branches. This is a thin capture.

5. ServerSystem::ProcessTurn — first guarded capture

140 calls (28 systems × 5 turns), 140 compared, 0 divergences.

The coverage is the finding:

• Only two declared regions ever moved in 140 calls: ntdev (15 times) and rcex (7 times). infra, ibon, pbon, tres, haltv, bats2, bats_mask, rcex_mask and rng were byte-identical before and after on every one of the 140 calls. A 0-divergence verdict here means "two integer counters agreed and nothing else moved" — it does not exercise the decay, the bonus applies, or the battle countdown.
• rng unchanged on all 140 confirms B4's headline over five turns: no colony drew a word, i.e. no rebellion fired. fpu_cw = 0x127f on every call.
• Owners: 3 owned systems (owner 0, 1, 7) × 5 turns = 15 calls; the other 125 are unowned.

Guard system (0x2d8 span) hits, 15 writes on 6 calls:

offset	len	plausible field	when
system+0x1a4	12	inside Pop2 (civilian Population) at 0x1a0 — the population-group vector's three words	sys 4, 15, 16 on turn 1 only
system+0x170	1	inside Fleets (vector<StarFleet*> at 0x16c) — the end pointer	sys 15, every turn
system+0xd8	1	unnamed in the contract	sys 15, every turn
system+0x238	4	unnamed in the contract	sys 15, every turn

System 15 is the AI's home world and it emits one new fleet per turn (see §6: fleets 50, 66, 98, 114 appear one per turn) — so system+0x170 is the build queue pushing the new fleet into the system's fleet vector, harness-audit row 14 observed live. +0xd8 and +0x238 are unidentified and worth an addresses.json entry.

The morale-event append (audit row 3) never fired — no addiction on this save — so that risk is still unmeasured, not cleared.

6. StrategyServer::MoveFleet — 8 of 45 diverge; B4's clean verdict was a one-sample verdict

45 calls, 45 compared, 8 divergent, tracecmp exit 1. Every divergence is a 1-ULP position component.

- call_id 42   side.pos.after.v.y [exact] orig=3.1515913    ours=3.15159106
- call_id 79   side.pos.after.v.y [exact] orig=1.58417809   ours=1.58417821
- call_id 116  side.pos.after.v.y [exact] orig=0.0167649984 ours=0.0167651176

Structure of the run: 30 of the 45 calls are the six waypointless fleets (wpt_type = -1, early-out, no guard hit). 15 calls do real work and every one of them is wpt_type = 1 (a straight run at speed 2 × dt 1.0). Types 2/3/4/5 never occurred, so node-line travel, node routes, gate teleports and the probabilistic jump remain untested, and the generator never moved in this hook.

Of the 15 moving calls, ULP deltas (orig − ours) on the position:

call	fleet	Δx	Δy	Δz
6, 41, 78, 115, 154	34	0	0	0 / 0 / 0 / 0 / +4
42, 79, 116	50	0	+1 / −1 / −64	0
80, 117	66	0	+1 / 0	0
118, 157	98	0	+1 / −1	0 / +1
155, 156, 158	50/66/114	0 / −1 / 0	0 / −1 / 0	0

The 64-ULP case is a near-zero result (0.0167…): the absolute error is ~1.2e-7 in every case, i.e. half an ULP at the magnitude of the inputs (y ≈ 1.6–4.7), not of the result. That is the signature of one rounding too many (or too few) in the position update — sim::AdvanceAlongDirection narrowing an intermediate to float32 where the original keeps the x87 53-bit double, or the reverse. It is not a formula error: x, y and z all diverge in some call, the step length is right, and every ship-range region matched.

Why B4 missed it: B4 had exactly one moving call, fleet 34 — and fleet 34 matches bit-for-bit on four of its five moves here too. The bug only shows once other fleets start moving. A reader of B4's trace would have seen "1 straight move, verified"; the verdict said "MoveFleet verified".

Guard fleet (0x120 span), 42 writes on the 15 moving calls, resolved against the StarFleet table:

offset	len	field	count
fleet+0x10c	1	Flags (bit 0x100 = held this turn)	15 — every moving call
fleet+0xdc	1	unnamed, just past Speed (FPsp2 @0xd8)	13
fleet+0xa0	4	Location* — the fleet's Location object pointer is replaced	8
fleet+0xcc	1	Waypoints vector last pointer — a waypoint was popped	2
fleet+0xdb	2	straddles Speed/0xdc	2
fleet+0xe0	26	FPogn2 @0xe0 … DestPos @0xec — the destination position rewrite	2

The two calls carrying +0xcc and +0xe0 (115 and 155) are the two arrivals in the whole run — and both compared clean, because none of the arrival machinery is declared. B4 said "an arriving call was simply clean"; that is still true and still means nothing.

7. What a reader of the trace would have noticed that no verdict did

• ProcessResearch fires 3 times per End Turn, not 15 — B3's 15 was five turns. Turn 1 of ref-turn2 contains no tech completion at all, which is why OnTechResearched produced zero records on the reference workload and B2's three completions must have come from a longer run. A lane that captures OnTechResearched on the documented one-turn recipe will get an empty log and a green report.
• The multi-turn continuation is not reproducible across sessions. Turn 1 is byte-identical to the oracle every time, and calls 0–3 reproduce docs/B3.md exactly ({144,2889}, {90,0}, {9,0}, {144,2898}), but from turn 4 the AI picks a different research target than B3 recorded ({9,3064} here vs {142,3064} there, {18,3087} vs {9,3086}) while the point totals stay nearly identical. Anything past the first End Turn should be treated as a run, not the run.
• ComputeBudget is called ~4280 times per session by the UI and ~8 times by the turn pass. Any "N compares" headline for that hook is dominated by UI polling of one player.

8. Still owed

• The count-only events region for ProcessResearch recommended by lane E's docs/E-events.md (converts audit row 1 from "known defect" to "checked"). Not wired — this lane's captures predate that module and re-capturing a good trace was not worth it.
• MoveFleet's 1-ULP position divergence needs the rounding in AdvanceAlongDirection pinned against the instruction stream.
• ServerSystem+0xd8 and ServerSystem+0x238 are unnamed and are written every turn by the AI home system. StarFleet+0xdc likewise, on every move.
• vector<ObservedTech> otch (ServerPlayer+0x274) is undeclared everywhere and is save state.
• Nothing here exercises: replace mode (any hook), a Zuul double roll, a rebellion, an addiction sweep, a plague, gate traffic, waypoint types 2–5, silent = true completions, the budget expense/aid/debt tail, or the Budget+0x64 over-budget write.