247 lines
14 KiB
Markdown
247 lines
14 KiB
Markdown
# Earned rules
|
||
|
||
Rules this campaign paid for. Each one exists because it was violated first and something wrong got
|
||
published, or nearly did. Lane briefs should reference this file rather than restating it.
|
||
|
||
Ordered by how much damage the violation caused.
|
||
|
||
---
|
||
|
||
## 1. A green verdict is not evidence. Coverage is.
|
||
|
||
`tracecmp` exit 0 means "the declared regions agreed". It does **not** mean the hook compared
|
||
anything. B4 found three hooks that each printed `0 diverged` while comparing nothing:
|
||
|
||
- `describe_args` ran **before** `regions()`, so every logged argument was one call stale;
|
||
- the `StrategyServer` has **two bases 4 bytes apart** — reading the wrong one yields an empty
|
||
player vector, zero declared regions, and a confident "1 call, 0 diverged";
|
||
- a Ghidra-base number was used as a raw struct offset (`0x64` vs `0x60`), enumerating a vector's
|
||
spare capacity.
|
||
|
||
Structural fixes now in place: compile-time-required `Coverage` on every descriptor (a hook without
|
||
one does not compile), guard regions that **localise** undeclared writes by offset rather than
|
||
reporting that a hash moved, replace-mode records, and `tracecmp --strict-coverage`. A hook claiming
|
||
"complete" while a guard caught an undeclared write counts as a **divergence**.
|
||
|
||
**Read the trace, not the verdict.** Every hook bug above was visible to a reader and invisible to
|
||
the exit code.
|
||
|
||
## 2. Write the prediction down before the run
|
||
|
||
Commit the expected result *before* building, not before running. Lane U committed its prediction
|
||
before the build was even staged; lane P wrote its expected residual (`next_id` short by exactly 1
|
||
on every completion call) and lane V confirmed it live.
|
||
|
||
The strong form: **a prediction that survives a changed workload tests the model, not the
|
||
recording.** Lane U's call 9 was not the call its prediction was written against — the AI had picked
|
||
a different tech — and it unlocked three nodes at costs appearing in no earlier report. Numbers that
|
||
never occurred cannot have been memorised.
|
||
|
||
Include a falsification section: how the model could be wrong, and the symptom of each way.
|
||
|
||
## 3. Static reading finds what behavioural comparison cannot
|
||
|
||
Both of these compared clean and were still wrong:
|
||
|
||
- `ApplyTechEffect` had an early return that would have made **every real call a no-op**;
|
||
- the RNG divisor was wrong on 0.78% of draws and flipped **zero** decisions in 10⁶ trials.
|
||
|
||
Conversely, behavioural compare found the `MoveFleet` rounding that static reading had missed. They
|
||
are different instruments. Use both; trust neither alone.
|
||
|
||
## 4. Prefer the instruction stream to the decompiler for control flow
|
||
|
||
**An inlined `std::vector` destructor looks exactly like a branch.** The `je` skips only the
|
||
`operator delete`; both arms converge a few hundred bytes later. This has produced two wrong
|
||
published findings — `turn-spine.md`'s "deferred end-of-turn tail" (there is no such branch) and a
|
||
near-miss on `Streamable::Write`.
|
||
|
||
Any decompiler `if` that wraps a whole tail is suspect until you have found the converge point in
|
||
the disassembly.
|
||
|
||
## 5. Size structs by enumeration, never by what the code touches
|
||
|
||
`_Alval` is `std::allocator<char>`, an **empty class**: it occupies a word and is never loaded or
|
||
stored, so it is invisible to touch-based analysis and undercounts by exactly 4. That produced a
|
||
`std::string`-is-0x18 scare against a correct campaign-wide 0x1c.
|
||
|
||
An **enumeration** — serializer, constructor, copy constructor, or container stride — can show
|
||
**absence** where a touch-scan cannot. Same trap is live for `std::vector` here:
|
||
`{_Myfirst,_Mylast,_Myend,_Alval}` = `0x10`, **allocator-last**, the opposite of the shape the MSVC
|
||
textbooks describe.
|
||
|
||
Related: the on-disk primitive is **not** the memory kind. A member held as `int16`/`int8` is
|
||
written by `WriteInt` and is **four bytes on the wire**.
|
||
|
||
## 6. A path no save exercises is a hypothesis
|
||
|
||
Label it as one. `spies2`, `SysMem` and `mts` were "closed" with count 0 in every save — no element
|
||
value has ever been observed. Lane A later found the element typing was wrong, and it was invisible
|
||
*precisely because* nothing exercised it. The flagging discipline is what made that checkable.
|
||
|
||
**Saves can be manufactured.** No species-5 save existed, so lane V built one and closed a row that
|
||
had been disassembly-only for a day. Prefer building the workload over weakening the claim.
|
||
|
||
## 7. Round-trip byte-identity is not coverage
|
||
|
||
`ar.any` bodies round-trip trivially by copying bytes nobody understands. Measure the split between
|
||
items a field **names** and items a node merely **carries**. That reframing moved a "100% round-trip"
|
||
result to an honest 38%, and then to 99.9% by actual work.
|
||
|
||
## 8. Two independent-looking checks sharing a hidden assumption are one check
|
||
|
||
`save_reader.py` types items from its **own** kind catalog, not the schema passed to it — so on real
|
||
saves the catalog and the schema agreed with each other and were **both wrong**. Its failure mode is
|
||
silent agreement, not a desync.
|
||
|
||
## 9. Rankers rank; they never filter
|
||
|
||
A cohort/frequency ranker would have **discarded the correct answer**: the function that appends to
|
||
`ObservedTech` touches only two offsets on `ServerPlayer`, so every `--min>=1` cohort filter drops
|
||
it. What closed the case was a plain query plus one call-graph lookup.
|
||
|
||
Displacement scan for recall, call graph for disambiguation. Neither alone sufficed. And report the
|
||
false-positive rate honestly — that scanner's class-level precision is ~13% by function; its value
|
||
is a 900× search-space cut, not accuracy.
|
||
|
||
## 10. Derive experiment parameters from the ISA, not from assumed mnemonics
|
||
|
||
An x87 sensitivity experiment was briefed as `0x027f / 0x127f / 0x137f`. In fact `0x027f` is 53-bit
|
||
(differing from `0x127f` only in infinity control, ignored since the 387) and `0x137f` is 64-bit
|
||
extended. Run literally it returns "all three identical" — true, and the conclusion drawn from it
|
||
would have been wrong on both axes, because it tests single precision not at all and rounding not at
|
||
all. The real probes are `0x007f` and `0x1a7f`.
|
||
|
||
**Brief lanes to challenge the parameters, not just execute them.** Lane F did, and lane G rejected
|
||
its brief's premise outright and was right to.
|
||
|
||
## 11. Correcting the record beats defending it
|
||
|
||
Claims downgraded on evidence, by later lanes: "RNG matched 15/15" (workload luck, not a property);
|
||
B1's and B4's coverage after the harness audit; `StreamableEnum` typing; the deferred end-of-turn
|
||
tail; "the research roll draws exactly one NextFloat" (the branch it fires draws a second).
|
||
|
||
If you find an earlier finding wrong, say so plainly and correct it in place. Nothing here is
|
||
anyone's reputation.
|
||
|
||
## 12. Fix the oracle openly, never quietly
|
||
|
||
Four defects were found in the reference save reader. The right move was **not** to patch them
|
||
mid-campaign in silence: they were recorded, then fixed openly with a test per defect and
|
||
byte-neutrality proven at item granularity (offset sequences identical across ~39,000 offsets per
|
||
save; typed-value deltas balancing to the byte).
|
||
|
||
## 13. Run verification gates as separate commands
|
||
|
||
Never `&&`-chain them. Once, a clean-room failure short-circuited the chain, skipping build and test
|
||
while a separate push still ran — an unverified tree reached `main`.
|
||
|
||
On one merge **both** gates failed and both were real: a raw `FUN_xxxxxxxx` identifier had reached
|
||
`docs/` in the public-capable engine repo, and the **shim cross-build** hit `-Werror=unused-function`
|
||
on a WIN32-only TU. Host `ctest` passed cleanly both times and would have hidden both.
|
||
|
||
**If a lane cannot cross-build its own shim TU, it must say so, and the integrator must run the
|
||
CT111 shim build before pushing.**
|
||
|
||
## 14. Never hand-resolve a generated header
|
||
|
||
Always regenerate from `ghidra/addresses.json` plus the `ghidra/addresses.d/<lane>.json` fragments.
|
||
Hand-resolving a merge conflict there silently dropped a new entry and broke the build — twice.
|
||
|
||
Per-lane fragments exist because a shared single file caused three cross-lane sweeps in one day.
|
||
Duplicate names across fragments are a **hard error**, never last-wins: two lanes disagreeing about
|
||
an address is exactly what must not be papered over.
|
||
|
||
## 15. Report thin coverage as loudly as divergences
|
||
|
||
A green run that establishes twenty facts is not a strong result. `ComputeBudget` compared 4,284
|
||
calls with 0 divergences — but only **20 distinct states**, with 4,278 of them the UI polling one
|
||
player, and **13 of 22 slots zero on every call**, including 5 of 6 declared input-boundary slots.
|
||
|
||
The most useful sections of the best reports here have been the honest lists of what was *not*
|
||
covered.
|
||
|
||
## 16. Inlined draws are invisible to call-graph sweeps
|
||
|
||
The combat resolver's `NextFloat` is **inlined**, so the only call-graph edge it leaves is
|
||
`caller -> RNG_Twist`, which reads as a bare Twist and is not one — it is the lazy twist *inside*
|
||
`NextFloat`. Lane K's "no NextFloat in that subtree" was wrong for exactly this reason.
|
||
|
||
An image-wide scan for the MT tempering immediates at instruction boundaries finds **14 game
|
||
functions with inlined draws that no RNG call-graph sweep can see**, two of them inside
|
||
`ProcessTurn`'s closure at depth 4.
|
||
|
||
Scan for the **constants at instruction boundaries**, not for calls. Any RNG accounting built from
|
||
the call graph alone is a lower bound.
|
||
|
||
## 17. A truncated range makes a loop look like a sequence
|
||
|
||
Lane J read a function as straight-line because it dumped the range at **Ghidra's reported size**,
|
||
and the outer back-edge fell just outside it. A delegated sweep made the same mistake independently.
|
||
That changed the draw count of the very site it was there to measure.
|
||
|
||
This is the same defect lane X hit from the other side: clipping sweeps at `fva + sizeInBytes` lost
|
||
**11%** of functions to mid-instruction truncation, and sweeping to the *next function start* took
|
||
coverage 89% -> 100%. Ghidra's size is also simply wrong sometimes — it gave 7,499 bytes for a body
|
||
that is 7,641 and ends mid-instruction.
|
||
|
||
**Never trust a function's end. Disassemble to the next function start and find the real boundary.**
|
||
|
||
## 18. Measure first. The lab exists.
|
||
|
||
We have the game running under a hook framework, a debugger, 11 curated saves, a byte-identical
|
||
autosave oracle and a state-checksum tool that localises to named leaves. **Use them before deriving
|
||
anything.**
|
||
|
||
The RNG gap is the case study. Static analysis spent several lanes on it and produced a complete
|
||
draw-site inventory with an unexplained count — genuinely good work that could not close. The answer
|
||
came from ten minutes of live hooking: return-address capture on the entry points attributed every
|
||
word in one turn, and the dominant consumer turned out to hang off a **virtual** edge that no
|
||
direct-edge closure could ever have reached.
|
||
|
||
The general shape: **static reading is for explaining what you measured, not for predicting it.**
|
||
Derivation is the fallback when the path cannot be reached, not the default.
|
||
|
||
A question of the form *"what writes this?"*, *"how many times does this run?"*, *"does this branch
|
||
ever fire?"* is a **watchpoint or a hook**, not a week of reading. Open items that fit that shape
|
||
right now: the `Player.Status` writer between tail phase 31 and the autosave; the `TShn`/`ltis`
|
||
writer; `Summary.Checksum`'s inputs; whether a trade-raid roll ever succeeds (a word count cannot
|
||
separate "no success" from "empty candidate list" — a hook on the callee separates them instantly).
|
||
|
||
Corollary, from the same session: **search the notes before the binary.** The dominant RNG consumer
|
||
was already identified in `strategic-turn-internals.md` months of lanes earlier. What was missing was
|
||
never the identification — it was the connection.
|
||
|
||
## 19. The instrument can perturb the thing it measures — check, do not assume
|
||
|
||
Lane H bisected an autosave that differed by 4 bytes from a byte-identical input, across
|
||
configurations, and found the cause: **a MinHook detour on one function changed the game's
|
||
behaviour.** The patched 5 bytes land on a clean prologue boundary with no branch target inside,
|
||
and the *suppressed* draw belongs to a function that runs earlier in the turn than the hooked one.
|
||
The mechanism is still undetermined.
|
||
|
||
Two things follow. First, the ledger built with a different hook set was checked and came out
|
||
**behaviour-neutral** — its numbers stand. Second, that check had never been run before; it came
|
||
out right by luck, not by design.
|
||
|
||
So: **before trusting a measurement, take the same measurement with the instrument removed.**
|
||
`hooks=off` in two fresh processes is cheap and it is the control. A hook that changes the autosave
|
||
has invalidated every number taken with it installed, and you will not notice from inside the run.
|
||
|
||
Corollary for the oracle: re-run it after any change to the hook set, not only after engine changes.
|
||
|
||
## 20. A count cannot separate "did not fire" from "fired and found nothing"
|
||
|
||
`CreateRaidEncounter` cost 0 words on every measured turn, which was read as "the roll never
|
||
succeeds (~11%, unremarkable over three turns)". An entry probe showed it is **entered 2x on one
|
||
turn and 1x on the next, drawing 0 every time** — the rolls succeed at about the predicted rate and
|
||
the *candidate list is empty*. Also, three quiet turns at that rate is 1-in-720, not unremarkable:
|
||
when a prior says "unremarkable", compute it.
|
||
|
||
Same shape elsewhere in the same lane: four tail callees were "never observed firing" and turned out
|
||
to be **entered every single turn and gated inside** — a much stronger negative than reachability.
|
||
And a residual measured at "2 words per turn" on two saves is **one word per player that passes a
|
||
gate**, where exactly one player passes on both corpus saves. A constant fitted to two observations
|
||
is not a constant.
|
||
|
||
Instrument the **entry**, not just the cost.
|