alex 0029d022f2 resolver: seed exposure is a predicate on the pre-turn save; rule 26 gains corollary (c)

2026-09-09 00:56:38 -04:00

27 KiB

Raw Blame History

Earned rules

Rules this campaign paid for. Each one exists because it was violated first and something wrong got published, or nearly did. Lane briefs should reference this file rather than restating it.

Ordered by how much damage the violation caused.

1. A green verdict is not evidence. Coverage is.

tracecmp exit 0 means "the declared regions agreed". It does not mean the hook compared anything. B4 found three hooks that each printed 0 diverged while comparing nothing:

describe_args ran before regions(), so every logged argument was one call stale;
the StrategyServer has two bases 4 bytes apart — reading the wrong one yields an empty player vector, zero declared regions, and a confident "1 call, 0 diverged";
a Ghidra-base number was used as a raw struct offset (0x64 vs 0x60), enumerating a vector's spare capacity.

Structural fixes now in place: compile-time-required Coverage on every descriptor (a hook without one does not compile), guard regions that localise undeclared writes by offset rather than reporting that a hash moved, replace-mode records, and tracecmp --strict-coverage. A hook claiming "complete" while a guard caught an undeclared write counts as a divergence.

Read the trace, not the verdict. Every hook bug above was visible to a reader and invisible to the exit code.

2. Write the prediction down before the run

Commit the expected result before building, not before running. Lane U committed its prediction before the build was even staged; lane P wrote its expected residual (next_id short by exactly 1 on every completion call) and lane V confirmed it live.

The strong form: a prediction that survives a changed workload tests the model, not the recording. Lane U's call 9 was not the call its prediction was written against — the AI had picked a different tech — and it unlocked three nodes at costs appearing in no earlier report. Numbers that never occurred cannot have been memorised.

Include a falsification section: how the model could be wrong, and the symptom of each way.

3. Static reading finds what behavioural comparison cannot

Both of these compared clean and were still wrong:

ApplyTechEffect had an early return that would have made every real call a no-op;
the RNG divisor was wrong on 0.78% of draws and flipped zero decisions in 10⁶ trials.

Conversely, behavioural compare found the MoveFleet rounding that static reading had missed. They are different instruments. Use both; trust neither alone.

4. Prefer the instruction stream to the decompiler for control flow

An inlined std::vector destructor looks exactly like a branch. The je skips only the operator delete; both arms converge a few hundred bytes later. This has produced two wrong published findings — turn-spine.md's "deferred end-of-turn tail" (there is no such branch) and a near-miss on Streamable::Write.

Any decompiler if that wraps a whole tail is suspect until you have found the converge point in the disassembly.

5. Size structs by enumeration, never by what the code touches

_Alval is std::allocator<char>, an empty class: it occupies a word and is never loaded or stored, so it is invisible to touch-based analysis and undercounts by exactly 4. That produced a std::string-is-0x18 scare against a correct campaign-wide 0x1c.

An enumeration — serializer, constructor, copy constructor, or container stride — can show absence where a touch-scan cannot. Same trap is live for std::vector here: {_Myfirst,_Mylast,_Myend,_Alval} = 0x10, allocator-last, the opposite of the shape the MSVC textbooks describe.

Related: the on-disk primitive is not the memory kind. A member held as int16/int8 is written by WriteInt and is four bytes on the wire.

6. A path no save exercises is a hypothesis

Label it as one. spies2, SysMem and mts were "closed" with count 0 in every save — no element value has ever been observed. Lane A later found the element typing was wrong, and it was invisible precisely because nothing exercised it. The flagging discipline is what made that checkable.

Saves can be manufactured. No species-5 save existed, so lane V built one and closed a row that had been disassembly-only for a day. Prefer building the workload over weakening the claim.

7. Round-trip byte-identity is not coverage

ar.any bodies round-trip trivially by copying bytes nobody understands. Measure the split between items a field names and items a node merely carries. That reframing moved a "100% round-trip" result to an honest 38%, and then to 99.9% by actual work.

save_reader.py types items from its own kind catalog, not the schema passed to it — so on real saves the catalog and the schema agreed with each other and were both wrong. Its failure mode is silent agreement, not a desync.

9. Rankers rank; they never filter

A cohort/frequency ranker would have discarded the correct answer: the function that appends to ObservedTech touches only two offsets on ServerPlayer, so every --min>=1 cohort filter drops it. What closed the case was a plain query plus one call-graph lookup.

Displacement scan for recall, call graph for disambiguation. Neither alone sufficed. And report the false-positive rate honestly — that scanner's class-level precision is ~13% by function; its value is a 900× search-space cut, not accuracy.

10. Derive experiment parameters from the ISA, not from assumed mnemonics

An x87 sensitivity experiment was briefed as 0x027f / 0x127f / 0x137f. In fact 0x027f is 53-bit (differing from 0x127f only in infinity control, ignored since the 387) and 0x137f is 64-bit extended. Run literally it returns "all three identical" — true, and the conclusion drawn from it would have been wrong on both axes, because it tests single precision not at all and rounding not at all. The real probes are 0x007f and 0x1a7f.

Brief lanes to challenge the parameters, not just execute them. Lane F did, and lane G rejected its brief's premise outright and was right to.

11. Correcting the record beats defending it

Claims downgraded on evidence, by later lanes: "RNG matched 15/15" (workload luck, not a property); B1's and B4's coverage after the harness audit; StreamableEnum typing; the deferred end-of-turn tail; "the research roll draws exactly one NextFloat" (the branch it fires draws a second).

If you find an earlier finding wrong, say so plainly and correct it in place. Nothing here is anyone's reputation.

12. Fix the oracle openly, never quietly

Four defects were found in the reference save reader. The right move was not to patch them mid-campaign in silence: they were recorded, then fixed openly with a test per defect and byte-neutrality proven at item granularity (offset sequences identical across ~39,000 offsets per save; typed-value deltas balancing to the byte).

13. Run verification gates as separate commands

Never &&-chain them. Once, a clean-room failure short-circuited the chain, skipping build and test while a separate push still ran — an unverified tree reached main.

On one merge both gates failed and both were real: a raw FUN_xxxxxxxx identifier had reached docs/ in the public-capable engine repo, and the shim cross-build hit -Werror=unused-function on a WIN32-only TU. Host ctest passed cleanly both times and would have hidden both.

If a lane cannot cross-build its own shim TU, it must say so, and the integrator must run the CT111 shim build before pushing.

14. Never hand-resolve a generated header

Always regenerate from ghidra/addresses.json plus the ghidra/addresses.d/<lane>.json fragments. Hand-resolving a merge conflict there silently dropped a new entry and broke the build — twice.

Per-lane fragments exist because a shared single file caused three cross-lane sweeps in one day. Duplicate names across fragments are a hard error, never last-wins: two lanes disagreeing about an address is exactly what must not be papered over.

15. Report thin coverage as loudly as divergences

A green run that establishes twenty facts is not a strong result. ComputeBudget compared 4,284 calls with 0 divergences — but only 20 distinct states, with 4,278 of them the UI polling one player, and 13 of 22 slots zero on every call, including 5 of 6 declared input-boundary slots.

The most useful sections of the best reports here have been the honest lists of what was not covered.

16. Inlined draws are invisible to call-graph sweeps

The combat resolver's NextFloat is inlined, so the only call-graph edge it leaves is caller -> RNG_Twist, which reads as a bare Twist and is not one — it is the lazy twist inside NextFloat. Lane K's "no NextFloat in that subtree" was wrong for exactly this reason.

An image-wide scan for the MT tempering immediates at instruction boundaries finds 14 game functions with inlined draws that no RNG call-graph sweep can see, two of them inside ProcessTurn's closure at depth 4.

Scan for the constants at instruction boundaries, not for calls. Any RNG accounting built from the call graph alone is a lower bound.

17. A truncated range makes a loop look like a sequence

Lane J read a function as straight-line because it dumped the range at Ghidra's reported size, and the outer back-edge fell just outside it. A delegated sweep made the same mistake independently. That changed the draw count of the very site it was there to measure.

This is the same defect lane X hit from the other side: clipping sweeps at fva + sizeInBytes lost 11% of functions to mid-instruction truncation, and sweeping to the next function start took coverage 89% -> 100%. Ghidra's size is also simply wrong sometimes — it gave 7,499 bytes for a body that is 7,641 and ends mid-instruction.

Never trust a function's end. Disassemble to the next function start and find the real boundary.

18. Measure first. The lab exists.

We have the game running under a hook framework, a debugger, 11 curated saves, a byte-identical autosave oracle and a state-checksum tool that localises to named leaves. Use them before deriving anything.

The RNG gap is the case study. Static analysis spent several lanes on it and produced a complete draw-site inventory with an unexplained count — genuinely good work that could not close. The answer came from ten minutes of live hooking: return-address capture on the entry points attributed every word in one turn, and the dominant consumer turned out to hang off a virtual edge that no direct-edge closure could ever have reached.

The general shape: static reading is for explaining what you measured, not for predicting it. Derivation is the fallback when the path cannot be reached, not the default.

A question of the form "what writes this?", "how many times does this run?", "does this branch ever fire?" is a watchpoint or a hook, not a week of reading. Open items that fit that shape right now: the Player.Status writer between tail phase 31 and the autosave; the TShn/ltis writer; Summary.Checksum's inputs; whether a trade-raid roll ever succeeds (a word count cannot separate "no success" from "empty candidate list" — a hook on the callee separates them instantly).

Corollary, from the same session: search the notes before the binary. The dominant RNG consumer was already identified in strategic-turn-internals.md months of lanes earlier. What was missing was never the identification — it was the connection.

19. The instrument can perturb the thing it measures — check, do not assume

Lane H bisected an autosave that differed by 4 bytes from a byte-identical input, across configurations, and found the cause: a MinHook detour on one function changed the game's behaviour. The patched 5 bytes land on a clean prologue boundary with no branch target inside, and the suppressed draw belongs to a function that runs earlier in the turn than the hooked one. The mechanism is still undetermined.

Two things follow. First, the ledger built with a different hook set was checked and came out behaviour-neutral — its numbers stand. Second, that check had never been run before; it came out right by luck, not by design.

So: before trusting a measurement, take the same measurement with the instrument removed. hooks=off in two fresh processes is cheap and it is the control. A hook that changes the autosave has invalidated every number taken with it installed, and you will not notice from inside the run.

Corollary for the oracle: re-run it after any change to the hook set, not only after engine changes.

20. A count cannot separate "did not fire" from "fired and found nothing"

CreateRaidEncounter cost 0 words on every measured turn, which was read as "the roll never succeeds (~11%, unremarkable over three turns)". An entry probe showed it is entered 2x on one turn and 1x on the next, drawing 0 every time — the rolls succeed at about the predicted rate and the candidate list is empty. Also, three quiet turns at that rate is 1-in-720, not unremarkable: when a prior says "unremarkable", compute it.

Same shape elsewhere in the same lane: four tail callees were "never observed firing" and turned out to be entered every single turn and gated inside — a much stronger negative than reachability. And a residual measured at "2 words per turn" on two saves is one word per player that passes a gate, where exactly one player passes on both corpus saves. A constant fitted to two observations is not a constant.

Instrument the entry, not just the cost.

21. A lane never touches the shared working directory — worktree or clone, always

Two incidents, one session. A VM lane ran git checkout -- in the shared sots-engine worktree and discarded another lane's in-flight generated header. An AI lane ran git checkout -b there, which moved the repo's HEAD, so the integrator's concurrent commit landed on the lane's branch instead of main and was orphaned when that branch was deleted. Nothing was lost either time, but only because both lanes reported it.

git worktree add or a separate clone. This is the same failure the addresses.d/README documents for git add, one level up: shared mutable state plus concurrency, with no lock.

The integrator's counterpart: after any concurrent round, check git log --oneline on main before pushing, and confirm the commits you believe you made are the ones that are there.

22. Union-resolving a merge is not textual concatenation

Two independent modules both added a branch to the same if/else if chain and a file to the same CMake source list. Concatenating the conflict halves produced a ) in the middle of the list and an if body with no closing brace — the host build passed (those files are Windows-only) and the shim cross-build failed, which is exactly the gap rule 13 exists for.

When both sides add a member of a construct, the resolution is to merge them into that construct — one list, one chain — not to paste one after the other. Read the resolved region before committing, and let the cross-build be the judge.

23. A live-verified module can still be wrong — thin coverage is how

ComputeBudget was compared against the original on 4,437 live calls with 0 divergences and was still wrong: its interest literals are widened floats in the image ((double)0.01f, (double)0.15f) and then truncated, so a treasury of exactly 50,000 earns 499, not 500. Our code used exact decimals. Sixteen hand-computed test expectations moved by one when it was fixed.

It survived because those 4,437 calls presented only 20 distinct states, none of them on a boundary — the exact thin-coverage caveat recorded against that lane at the time, later vindicated by a different lane doing arithmetic the compare never exercised.

So "live-verified, 0 divergences" is a statement about the states that occurred, never about the function. Two defences, both cheap: report the distinct-state count next to the call count (rule 15), and test the boundaries by hand — a value that lands exactly on a cap, a treasury that divides exactly, an empty container.

A float literal in this image is a widened float, not a double. It has now bitten the money chain, the bankruptcy divisor and the gate constant. Read the four bytes; do not assume the decimal.

24. Never reuse a build directory across trees — a stale binary measures cleanly

The integrator's gate rsync excludes build* to save transfer time, so the remote build directory survived from an earlier tree. rsync preserves mtimes, so cmake --build saw objects newer than sources, relinked nothing, and produced a stale binary that measured perfectly.

The reported figure was 128 diverging leaves. The real figure for that tree was 124. Two lanes' merged work was invisible, and both had independently reported the better number from their own branches — which is the only reason it was caught: two lanes reporting 126 from a 128 baseline while main also read 128 is arithmetically impossible.

The failure mode is the dangerous kind: no error, no warning, and a plausible number. It is the same shape as querying a build directory while a background job owns it (which also looked exactly like nondeterminism in the engine).

So: a fresh build directory per measurement, or rm -rf the build tree before building. And when a merged result does not reproduce a lane's own number, suspect the build before suspecting the lane.

Implementation, both halves — I got this wrong once by doing only the second. Exclude local build directories from the transfer and remove the remote ones:

rsync -a --delete --exclude .git --exclude 'build*' ./ host:/path/   # do not ship local build dirs
ssh host 'cd /path && rm -rf build-host build-shim && cmake --preset host && …'

A CMakeCache.txt records the absolute path it was created in, so a local build directory copied to another machine poisons the build there — CMake Error: … is different than the directory … where CMakeCache.txt was created, and every test fails.

The signature tells you which failure you have: a real regression changes the pass count (57/58); a broken configure changes the total (0% of 48, where a healthy run is 58). If the denominator moved, stop reading the failures and fix the build.

25. Commit with a pathspec — the index is shared, staging by path is not enough

Four times in one session, a lane's git commit swallowed another lane's staged files in the shared sots-re clone. The last one was plain git commit -m ... with no -A anywhere: every lane had staged by path exactly as instructed, and it still happened, because git add <path> puts a file in the repo's one index and the next git commit takes everything in it.

The fix is a form of the command, not more discipline:

git commit -m "message" -- path/one path/two      # commits ONLY those paths

With a pathspec, git commits the named paths from the working tree and leaves the rest of the index untouched — verified: with two files staged, git commit -- a.txt committed only a.txt and left b.txt staged. That is exactly the isolation lanes need, and it costs nothing.

So: lanes commit with a pathspec, always. git add by path stays good hygiene for reviewing a diff, but it is not the protection — the pathspec on commit is.

The deeper fix, if this recurs: give each lane its own clone of sots-re, the way each already gets its own git worktree of sots-engine (which is why that repo has never had this problem). Rule 21 covers the worktree side; this is its counterpart for the shared evidence repo.

26. A control must agree with itself before it exonerates anything

Rule 19 says take the measurement with the instrument removed. It did not say what to do when the control itself varies, and this session that gap produced two wrong readings.

Three lanes ran hooks=off on turn1-state and got three different files. So no single hooks=off run on that workload was ever a control — it could not exonerate or indict anything. Separately, one lane observed its instrumented run coming out byte-identical to a hooks=off run and read that as the instrument being clean; with an outcome set of size k that agreement is a ~1/k coincidence, not evidence.

So: a control must be reproduced in two fresh processes and agree with itself before it is used as a control. If it does not agree, the workload is non-deterministic and the honest move is to pin the source of variation (here: the per-client AI seed) and re-run, not to pick the run that suits.

Corollary, from the same episode: "one of N moved" says nothing about mechanism. Only one of three AI empires visibly varied, which looked like a tiebreak among near-equal candidates. In fact all three streams differed every run; two empires never reach the seed-sensitive code path at all. Before reasoning from which items varied, establish whether the others were even exposed to the mechanism.

(c) A control that agrees certifies a procedure, not a state — and it means "no exposed decision was reached", never "the AI is deterministic here." Three lanes in one day: ad-turn27 → 28 varied in 94 leaves (the AI's build, fleet-assignment and colony choices on a turn where thirteen ships completed, a design was created and a system colonised); as-turn15 → 16 varied in 22 (a research pick, on a turn the AI entered with ResTNm == ''); ar-turn37 → 39, the same lineage as AD ten turns on, agreed byte-for-byte across three processes — on two turns where the target was held then completed, no design was created and two freighters a turn joined existing fleets. The per-process seed was present in all three. It reaches the save only through a decision whose outcome set is not a singleton given the state, and whether such a decision is reached is a predicate on the pre-turn save — rule 28's shape: ResTNm == '' names a pick turn; NumDes moving names a design turn; a batch of completions names a fleet-assignment turn; one consumer (0x0069dbb0) has a 3–40-turn fuse and cannot show inside two turns of a load. So: record the pair as (save, procedure, hashes) and write the exposure facts next to the hashes; never extend a certified pair by a turn without its own two-process agreement (ar-oracle-A-post enters turn 40 with ResTNm == '' and is predicted to vary); and when a control does vary, decode which consumer was reached before reasoning from the leaves — "the AI varies per process" is a fact about a turn, not about the engine.

27. A ratchet is meant to break when the corpus grows — do not move it, and scope your control

The save corpus went from 11 to 19 in one evening as lanes finally manufactured workloads nobody had before. The coverage ratchet broke immediately: named coverage fell to 97.6% against a 99.99 bar, because the new saves carry content no schema names — <rest:prep>, TacReports, Lay, <rest:trdmgr>, fwarn.

That is the ratchet working. The reader is sound on every one of them (round trip: tree identical, typed identical); it simply does not name the new items. The temptation is to lower the bar so the suite goes green. Type the content instead — a ratchet you move on contact is a decoration.

Two integrator lessons from the same episode:

A new save can break a gate without any code changing. Before blaming a merge, run the suite against the corpus as it was. Both times this happened the merge was clean.
Scope that control by an explicit list, not by a name filter. My first control excluded *traderoutes* and still failed, which briefly looked like the merge was at fault — the lane had added more saves under other names while I worked. Pin the control to the exact set you mean.

28. A zero on a gated path is not a negative until you have the predicate

Four lanes measured one call site at zero. Each reported it correctly. One lane read it statically and placed the draw. The campaign nevertheless recorded "the tail does not draw" — and it does.

The site is gated on a fleet standing at a trade-sector node whose owner's bit is set in a mask that reads 252 in every save we own. Nobody had the predicate, so everybody hunted the nearest visible container: trade routes, then freighters, then a deployed spy. The actual switch is a tech (Commerce Raiding, one turn's research) that flips the mask to 253. One lane's stated "next condition" was falsified by its own save, which had a freighter on an active route the whole time with the site still at zero.

So, four practices:

A zero is a negative only once the gate is a predicate on save fields and the corpus has been counted against it. Until then it is "not reached in the states measured" — which is a different sentence and should be written as one.
Name the next workload from the failed conjunct, not from the nearest container. Decode the gate first, then build the state it names. Lane AC disassembled the chain before building anything and found the predicate had nothing to do with what three lanes were chasing.
A field constant across the whole corpus is a coincidence until its writer is found. tscr at 252 everywhere looked like a fact about the game; it was a fact about our saves. Compare rule 8's eighteen fields that split a roster identically because nothing exercised them.
Pair entry probes with the return-address ledger before calling a subtree draw-free. An entry probe on an inner function reading zero says nothing about a draw sited in its caller — which is exactly where the spy half's detection roll lives.

Confirmed live, 2026-09-09. Lane AS measured both instruments on the same turn: the entry probe on SpyManager::Slot13RngCallee read 0, while the return-address ledger recorded the detection roll firing for one word in the caller's body. The zero was never a statement about the subtree — only about which function the call happens to sit in. (The roll is a plain call, not an inlined one; the earlier claim that it was inlined was wrong, and the practice holds either way.)
A fact about one map is not a fact about the game. Practice 3's sibling, and it cost a lane a whole workload plan. One lane reported that "an enemy colony with an asteroid belt does not exist in this game", which was true of its map. All twenty-two corpus saves have a usable belt, two of them already hold four AI-owned ones, and the next lane's very first generated map had two — zero regenerations. Before building a state to satisfy a condition, count the corpus against it: the state you need may already be sitting in the save set.
Read the polarity before reading the count. In the same chain, the jump goes past the roll: with the belt gone there is no draw and the spy is detected anyway. A zero at that site therefore means two opposite things, and a lane that assumed the intuitive polarity would have reported the detection case as "did not fire".

27 KiB Raw Blame History Unescape Escape