188 lines
9.9 KiB
Markdown
188 lines
9.9 KiB
Markdown
# Earned rules
|
||
|
||
Rules this campaign paid for. Each one exists because it was violated first and something wrong got
|
||
published, or nearly did. Lane briefs should reference this file rather than restating it.
|
||
|
||
Ordered by how much damage the violation caused.
|
||
|
||
---
|
||
|
||
## 1. A green verdict is not evidence. Coverage is.
|
||
|
||
`tracecmp` exit 0 means "the declared regions agreed". It does **not** mean the hook compared
|
||
anything. B4 found three hooks that each printed `0 diverged` while comparing nothing:
|
||
|
||
- `describe_args` ran **before** `regions()`, so every logged argument was one call stale;
|
||
- the `StrategyServer` has **two bases 4 bytes apart** — reading the wrong one yields an empty
|
||
player vector, zero declared regions, and a confident "1 call, 0 diverged";
|
||
- a Ghidra-base number was used as a raw struct offset (`0x64` vs `0x60`), enumerating a vector's
|
||
spare capacity.
|
||
|
||
Structural fixes now in place: compile-time-required `Coverage` on every descriptor (a hook without
|
||
one does not compile), guard regions that **localise** undeclared writes by offset rather than
|
||
reporting that a hash moved, replace-mode records, and `tracecmp --strict-coverage`. A hook claiming
|
||
"complete" while a guard caught an undeclared write counts as a **divergence**.
|
||
|
||
**Read the trace, not the verdict.** Every hook bug above was visible to a reader and invisible to
|
||
the exit code.
|
||
|
||
## 2. Write the prediction down before the run
|
||
|
||
Commit the expected result *before* building, not before running. Lane U committed its prediction
|
||
before the build was even staged; lane P wrote its expected residual (`next_id` short by exactly 1
|
||
on every completion call) and lane V confirmed it live.
|
||
|
||
The strong form: **a prediction that survives a changed workload tests the model, not the
|
||
recording.** Lane U's call 9 was not the call its prediction was written against — the AI had picked
|
||
a different tech — and it unlocked three nodes at costs appearing in no earlier report. Numbers that
|
||
never occurred cannot have been memorised.
|
||
|
||
Include a falsification section: how the model could be wrong, and the symptom of each way.
|
||
|
||
## 3. Static reading finds what behavioural comparison cannot
|
||
|
||
Both of these compared clean and were still wrong:
|
||
|
||
- `ApplyTechEffect` had an early return that would have made **every real call a no-op**;
|
||
- the RNG divisor was wrong on 0.78% of draws and flipped **zero** decisions in 10⁶ trials.
|
||
|
||
Conversely, behavioural compare found the `MoveFleet` rounding that static reading had missed. They
|
||
are different instruments. Use both; trust neither alone.
|
||
|
||
## 4. Prefer the instruction stream to the decompiler for control flow
|
||
|
||
**An inlined `std::vector` destructor looks exactly like a branch.** The `je` skips only the
|
||
`operator delete`; both arms converge a few hundred bytes later. This has produced two wrong
|
||
published findings — `turn-spine.md`'s "deferred end-of-turn tail" (there is no such branch) and a
|
||
near-miss on `Streamable::Write`.
|
||
|
||
Any decompiler `if` that wraps a whole tail is suspect until you have found the converge point in
|
||
the disassembly.
|
||
|
||
## 5. Size structs by enumeration, never by what the code touches
|
||
|
||
`_Alval` is `std::allocator<char>`, an **empty class**: it occupies a word and is never loaded or
|
||
stored, so it is invisible to touch-based analysis and undercounts by exactly 4. That produced a
|
||
`std::string`-is-0x18 scare against a correct campaign-wide 0x1c.
|
||
|
||
An **enumeration** — serializer, constructor, copy constructor, or container stride — can show
|
||
**absence** where a touch-scan cannot. Same trap is live for `std::vector` here:
|
||
`{_Myfirst,_Mylast,_Myend,_Alval}` = `0x10`, **allocator-last**, the opposite of the shape the MSVC
|
||
textbooks describe.
|
||
|
||
Related: the on-disk primitive is **not** the memory kind. A member held as `int16`/`int8` is
|
||
written by `WriteInt` and is **four bytes on the wire**.
|
||
|
||
## 6. A path no save exercises is a hypothesis
|
||
|
||
Label it as one. `spies2`, `SysMem` and `mts` were "closed" with count 0 in every save — no element
|
||
value has ever been observed. Lane A later found the element typing was wrong, and it was invisible
|
||
*precisely because* nothing exercised it. The flagging discipline is what made that checkable.
|
||
|
||
**Saves can be manufactured.** No species-5 save existed, so lane V built one and closed a row that
|
||
had been disassembly-only for a day. Prefer building the workload over weakening the claim.
|
||
|
||
## 7. Round-trip byte-identity is not coverage
|
||
|
||
`ar.any` bodies round-trip trivially by copying bytes nobody understands. Measure the split between
|
||
items a field **names** and items a node merely **carries**. That reframing moved a "100% round-trip"
|
||
result to an honest 38%, and then to 99.9% by actual work.
|
||
|
||
## 8. Two independent-looking checks sharing a hidden assumption are one check
|
||
|
||
`save_reader.py` types items from its **own** kind catalog, not the schema passed to it — so on real
|
||
saves the catalog and the schema agreed with each other and were **both wrong**. Its failure mode is
|
||
silent agreement, not a desync.
|
||
|
||
## 9. Rankers rank; they never filter
|
||
|
||
A cohort/frequency ranker would have **discarded the correct answer**: the function that appends to
|
||
`ObservedTech` touches only two offsets on `ServerPlayer`, so every `--min>=1` cohort filter drops
|
||
it. What closed the case was a plain query plus one call-graph lookup.
|
||
|
||
Displacement scan for recall, call graph for disambiguation. Neither alone sufficed. And report the
|
||
false-positive rate honestly — that scanner's class-level precision is ~13% by function; its value
|
||
is a 900× search-space cut, not accuracy.
|
||
|
||
## 10. Derive experiment parameters from the ISA, not from assumed mnemonics
|
||
|
||
An x87 sensitivity experiment was briefed as `0x027f / 0x127f / 0x137f`. In fact `0x027f` is 53-bit
|
||
(differing from `0x127f` only in infinity control, ignored since the 387) and `0x137f` is 64-bit
|
||
extended. Run literally it returns "all three identical" — true, and the conclusion drawn from it
|
||
would have been wrong on both axes, because it tests single precision not at all and rounding not at
|
||
all. The real probes are `0x007f` and `0x1a7f`.
|
||
|
||
**Brief lanes to challenge the parameters, not just execute them.** Lane F did, and lane G rejected
|
||
its brief's premise outright and was right to.
|
||
|
||
## 11. Correcting the record beats defending it
|
||
|
||
Claims downgraded on evidence, by later lanes: "RNG matched 15/15" (workload luck, not a property);
|
||
B1's and B4's coverage after the harness audit; `StreamableEnum` typing; the deferred end-of-turn
|
||
tail; "the research roll draws exactly one NextFloat" (the branch it fires draws a second).
|
||
|
||
If you find an earlier finding wrong, say so plainly and correct it in place. Nothing here is
|
||
anyone's reputation.
|
||
|
||
## 12. Fix the oracle openly, never quietly
|
||
|
||
Four defects were found in the reference save reader. The right move was **not** to patch them
|
||
mid-campaign in silence: they were recorded, then fixed openly with a test per defect and
|
||
byte-neutrality proven at item granularity (offset sequences identical across ~39,000 offsets per
|
||
save; typed-value deltas balancing to the byte).
|
||
|
||
## 13. Run verification gates as separate commands
|
||
|
||
Never `&&`-chain them. Once, a clean-room failure short-circuited the chain, skipping build and test
|
||
while a separate push still ran — an unverified tree reached `main`.
|
||
|
||
On one merge **both** gates failed and both were real: a raw `FUN_xxxxxxxx` identifier had reached
|
||
`docs/` in the public-capable engine repo, and the **shim cross-build** hit `-Werror=unused-function`
|
||
on a WIN32-only TU. Host `ctest` passed cleanly both times and would have hidden both.
|
||
|
||
**If a lane cannot cross-build its own shim TU, it must say so, and the integrator must run the
|
||
CT111 shim build before pushing.**
|
||
|
||
## 14. Never hand-resolve a generated header
|
||
|
||
Always regenerate from `ghidra/addresses.json` plus the `ghidra/addresses.d/<lane>.json` fragments.
|
||
Hand-resolving a merge conflict there silently dropped a new entry and broke the build — twice.
|
||
|
||
Per-lane fragments exist because a shared single file caused three cross-lane sweeps in one day.
|
||
Duplicate names across fragments are a **hard error**, never last-wins: two lanes disagreeing about
|
||
an address is exactly what must not be papered over.
|
||
|
||
## 15. Report thin coverage as loudly as divergences
|
||
|
||
A green run that establishes twenty facts is not a strong result. `ComputeBudget` compared 4,284
|
||
calls with 0 divergences — but only **20 distinct states**, with 4,278 of them the UI polling one
|
||
player, and **13 of 22 slots zero on every call**, including 5 of 6 declared input-boundary slots.
|
||
|
||
The most useful sections of the best reports here have been the honest lists of what was *not*
|
||
covered.
|
||
|
||
## 16. Inlined draws are invisible to call-graph sweeps
|
||
|
||
The combat resolver's `NextFloat` is **inlined**, so the only call-graph edge it leaves is
|
||
`caller -> RNG_Twist`, which reads as a bare Twist and is not one — it is the lazy twist *inside*
|
||
`NextFloat`. Lane K's "no NextFloat in that subtree" was wrong for exactly this reason.
|
||
|
||
An image-wide scan for the MT tempering immediates at instruction boundaries finds **14 game
|
||
functions with inlined draws that no RNG call-graph sweep can see**, two of them inside
|
||
`ProcessTurn`'s closure at depth 4.
|
||
|
||
Scan for the **constants at instruction boundaries**, not for calls. Any RNG accounting built from
|
||
the call graph alone is a lower bound.
|
||
|
||
## 17. A truncated range makes a loop look like a sequence
|
||
|
||
Lane J read a function as straight-line because it dumped the range at **Ghidra's reported size**,
|
||
and the outer back-edge fell just outside it. A delegated sweep made the same mistake independently.
|
||
That changed the draw count of the very site it was there to measure.
|
||
|
||
This is the same defect lane X hit from the other side: clipping sweeps at `fva + sizeInBytes` lost
|
||
**11%** of functions to mid-instruction truncation, and sweeping to the *next function start* took
|
||
coverage 89% -> 100%. Ghidra's size is also simply wrong sometimes — it gave 7,499 bytes for a body
|
||
that is 7,641 and ends mid-instruction.
|
||
|
||
**Never trust a function's end. Disassemble to the next function start and find the real boundary.**
|