guides: the 15 earned method rules, each traced to the finding that produced it
This commit is contained in:
parent
dcc4afddd3
commit
136f98b564
2 changed files with 163 additions and 0 deletions
|
|
@ -143,3 +143,4 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`).
|
|||
| unread driver: OnAllCombatDone_Tail | control-flow | backlog | — | 0% | 2026-09-08 | THE BIGGEST UNREAD BLOCK IS NOT IN ProcessTurn AT ALL. No bankruptcy, no turn results, no turn events, no autosave run in the turn driver - they are all in StrategyServer::OnAllCombatDone_Tail 0x007d92a0, a SECOND driver no lane has read. Highest-value next control-flow target |
|
||||
| player_turn hook prepared (boundary narrowed) | phase2 | mapped | high | 70% | 2026-09-08 | Lane T built the descriptor with compile-time coverage() (9 notes), Result regions for the three phases needing nothing but the player, roll_flags + rng as observations ours never writes, guards over the whole ServerPlayer and the TechTree header. DELIBERATELY NARROWED: phases 2/3/6 (savings, aid records, refund) depend on ComputeBudget's slots and ProcessResearch's overBudget, both STACK LOCALS of the original. The three ways to reach them are calling ComputeBudget ourselves (repairs ships - audit #6), reading the nested hooks (audit #5, SELF-FULFILLING), or inferring from the Sav delta. They are GUARDED, NOT CHECKED; formulas written and unit-tested but not in the verdict. docs/T-turn-driver.md carries the prediction incl. which checks are WEAK BY CONSTRUCTION on the reference save |
|
||||
| INTEGRATOR NOTE: two gate failures caught on lane T merge | meta | verified | high | 100% | 2026-09-08 | Both gates failed on merge and both were real, which is the argument for running them SEPARATELY rather than &&-chained: (1) clean-room exit 1 - a raw FUN_00889d60 identifier had reached docs/ in the public-capable engine repo; fixed by using the name the lane itself had already put in addresses.json (ServerPlayer_OnResearchRollSucceeded). (2) The shim CROSS-BUILD failed -Werror=unused-function on player_turn.cpp - the file is WIN32-only and the lane could not build it on the notes host, exactly as it flagged. Host ctest passed both times and would have hidden both. A lane that cannot cross-build its own shim TU must say so, and the integrator must run the CT111 shim build before pushing |
|
||||
| guides/method-rules.md | meta | verified | high | 100% | 2026-09-08 | The 15 rules this campaign PAID FOR, each traced to the finding that produced it, ranked by damage caused. Written because the integrator was pasting the same hard-won constraints into every lane brief, which invites drift and omission. Lane briefs now REFERENCE it instead of restating it. Covers: green verdict != evidence; predict before the run; static vs behavioural find different bugs; inlined dtor looks like a branch; size by enumeration; unexercised path = hypothesis; round-trip != coverage; two checks sharing an assumption are one check; rankers never filter; derive experiment params from the ISA; correct the record; fix the oracle openly; separate gates; never hand-resolve a generated header; report thin coverage as loudly as divergences |
|
||||
|
|
|
|||
162
guides/method-rules.md
Normal file
162
guides/method-rules.md
Normal file
|
|
@ -0,0 +1,162 @@
|
|||
# Earned rules
|
||||
|
||||
Rules this campaign paid for. Each one exists because it was violated first and something wrong got
|
||||
published, or nearly did. Lane briefs should reference this file rather than restating it.
|
||||
|
||||
Ordered by how much damage the violation caused.
|
||||
|
||||
---
|
||||
|
||||
## 1. A green verdict is not evidence. Coverage is.
|
||||
|
||||
`tracecmp` exit 0 means "the declared regions agreed". It does **not** mean the hook compared
|
||||
anything. B4 found three hooks that each printed `0 diverged` while comparing nothing:
|
||||
|
||||
- `describe_args` ran **before** `regions()`, so every logged argument was one call stale;
|
||||
- the `StrategyServer` has **two bases 4 bytes apart** — reading the wrong one yields an empty
|
||||
player vector, zero declared regions, and a confident "1 call, 0 diverged";
|
||||
- a Ghidra-base number was used as a raw struct offset (`0x64` vs `0x60`), enumerating a vector's
|
||||
spare capacity.
|
||||
|
||||
Structural fixes now in place: compile-time-required `Coverage` on every descriptor (a hook without
|
||||
one does not compile), guard regions that **localise** undeclared writes by offset rather than
|
||||
reporting that a hash moved, replace-mode records, and `tracecmp --strict-coverage`. A hook claiming
|
||||
"complete" while a guard caught an undeclared write counts as a **divergence**.
|
||||
|
||||
**Read the trace, not the verdict.** Every hook bug above was visible to a reader and invisible to
|
||||
the exit code.
|
||||
|
||||
## 2. Write the prediction down before the run
|
||||
|
||||
Commit the expected result *before* building, not before running. Lane U committed its prediction
|
||||
before the build was even staged; lane P wrote its expected residual (`next_id` short by exactly 1
|
||||
on every completion call) and lane V confirmed it live.
|
||||
|
||||
The strong form: **a prediction that survives a changed workload tests the model, not the
|
||||
recording.** Lane U's call 9 was not the call its prediction was written against — the AI had picked
|
||||
a different tech — and it unlocked three nodes at costs appearing in no earlier report. Numbers that
|
||||
never occurred cannot have been memorised.
|
||||
|
||||
Include a falsification section: how the model could be wrong, and the symptom of each way.
|
||||
|
||||
## 3. Static reading finds what behavioural comparison cannot
|
||||
|
||||
Both of these compared clean and were still wrong:
|
||||
|
||||
- `ApplyTechEffect` had an early return that would have made **every real call a no-op**;
|
||||
- the RNG divisor was wrong on 0.78% of draws and flipped **zero** decisions in 10⁶ trials.
|
||||
|
||||
Conversely, behavioural compare found the `MoveFleet` rounding that static reading had missed. They
|
||||
are different instruments. Use both; trust neither alone.
|
||||
|
||||
## 4. Prefer the instruction stream to the decompiler for control flow
|
||||
|
||||
**An inlined `std::vector` destructor looks exactly like a branch.** The `je` skips only the
|
||||
`operator delete`; both arms converge a few hundred bytes later. This has produced two wrong
|
||||
published findings — `turn-spine.md`'s "deferred end-of-turn tail" (there is no such branch) and a
|
||||
near-miss on `Streamable::Write`.
|
||||
|
||||
Any decompiler `if` that wraps a whole tail is suspect until you have found the converge point in
|
||||
the disassembly.
|
||||
|
||||
## 5. Size structs by enumeration, never by what the code touches
|
||||
|
||||
`_Alval` is `std::allocator<char>`, an **empty class**: it occupies a word and is never loaded or
|
||||
stored, so it is invisible to touch-based analysis and undercounts by exactly 4. That produced a
|
||||
`std::string`-is-0x18 scare against a correct campaign-wide 0x1c.
|
||||
|
||||
An **enumeration** — serializer, constructor, copy constructor, or container stride — can show
|
||||
**absence** where a touch-scan cannot. Same trap is live for `std::vector` here:
|
||||
`{_Myfirst,_Mylast,_Myend,_Alval}` = `0x10`, **allocator-last**, the opposite of the shape the MSVC
|
||||
textbooks describe.
|
||||
|
||||
Related: the on-disk primitive is **not** the memory kind. A member held as `int16`/`int8` is
|
||||
written by `WriteInt` and is **four bytes on the wire**.
|
||||
|
||||
## 6. A path no save exercises is a hypothesis
|
||||
|
||||
Label it as one. `spies2`, `SysMem` and `mts` were "closed" with count 0 in every save — no element
|
||||
value has ever been observed. Lane A later found the element typing was wrong, and it was invisible
|
||||
*precisely because* nothing exercised it. The flagging discipline is what made that checkable.
|
||||
|
||||
**Saves can be manufactured.** No species-5 save existed, so lane V built one and closed a row that
|
||||
had been disassembly-only for a day. Prefer building the workload over weakening the claim.
|
||||
|
||||
## 7. Round-trip byte-identity is not coverage
|
||||
|
||||
`ar.any` bodies round-trip trivially by copying bytes nobody understands. Measure the split between
|
||||
items a field **names** and items a node merely **carries**. That reframing moved a "100% round-trip"
|
||||
result to an honest 38%, and then to 99.9% by actual work.
|
||||
|
||||
## 8. Two independent-looking checks sharing a hidden assumption are one check
|
||||
|
||||
`save_reader.py` types items from its **own** kind catalog, not the schema passed to it — so on real
|
||||
saves the catalog and the schema agreed with each other and were **both wrong**. Its failure mode is
|
||||
silent agreement, not a desync.
|
||||
|
||||
## 9. Rankers rank; they never filter
|
||||
|
||||
A cohort/frequency ranker would have **discarded the correct answer**: the function that appends to
|
||||
`ObservedTech` touches only two offsets on `ServerPlayer`, so every `--min>=1` cohort filter drops
|
||||
it. What closed the case was a plain query plus one call-graph lookup.
|
||||
|
||||
Displacement scan for recall, call graph for disambiguation. Neither alone sufficed. And report the
|
||||
false-positive rate honestly — that scanner's class-level precision is ~13% by function; its value
|
||||
is a 900× search-space cut, not accuracy.
|
||||
|
||||
## 10. Derive experiment parameters from the ISA, not from assumed mnemonics
|
||||
|
||||
An x87 sensitivity experiment was briefed as `0x027f / 0x127f / 0x137f`. In fact `0x027f` is 53-bit
|
||||
(differing from `0x127f` only in infinity control, ignored since the 387) and `0x137f` is 64-bit
|
||||
extended. Run literally it returns "all three identical" — true, and the conclusion drawn from it
|
||||
would have been wrong on both axes, because it tests single precision not at all and rounding not at
|
||||
all. The real probes are `0x007f` and `0x1a7f`.
|
||||
|
||||
**Brief lanes to challenge the parameters, not just execute them.** Lane F did, and lane G rejected
|
||||
its brief's premise outright and was right to.
|
||||
|
||||
## 11. Correcting the record beats defending it
|
||||
|
||||
Claims downgraded on evidence, by later lanes: "RNG matched 15/15" (workload luck, not a property);
|
||||
B1's and B4's coverage after the harness audit; `StreamableEnum` typing; the deferred end-of-turn
|
||||
tail; "the research roll draws exactly one NextFloat" (the branch it fires draws a second).
|
||||
|
||||
If you find an earlier finding wrong, say so plainly and correct it in place. Nothing here is
|
||||
anyone's reputation.
|
||||
|
||||
## 12. Fix the oracle openly, never quietly
|
||||
|
||||
Four defects were found in the reference save reader. The right move was **not** to patch them
|
||||
mid-campaign in silence: they were recorded, then fixed openly with a test per defect and
|
||||
byte-neutrality proven at item granularity (offset sequences identical across ~39,000 offsets per
|
||||
save; typed-value deltas balancing to the byte).
|
||||
|
||||
## 13. Run verification gates as separate commands
|
||||
|
||||
Never `&&`-chain them. Once, a clean-room failure short-circuited the chain, skipping build and test
|
||||
while a separate push still ran — an unverified tree reached `main`.
|
||||
|
||||
On one merge **both** gates failed and both were real: a raw `FUN_xxxxxxxx` identifier had reached
|
||||
`docs/` in the public-capable engine repo, and the **shim cross-build** hit `-Werror=unused-function`
|
||||
on a WIN32-only TU. Host `ctest` passed cleanly both times and would have hidden both.
|
||||
|
||||
**If a lane cannot cross-build its own shim TU, it must say so, and the integrator must run the
|
||||
CT111 shim build before pushing.**
|
||||
|
||||
## 14. Never hand-resolve a generated header
|
||||
|
||||
Always regenerate from `ghidra/addresses.json` plus the `ghidra/addresses.d/<lane>.json` fragments.
|
||||
Hand-resolving a merge conflict there silently dropped a new entry and broke the build — twice.
|
||||
|
||||
Per-lane fragments exist because a shared single file caused three cross-lane sweeps in one day.
|
||||
Duplicate names across fragments are a **hard error**, never last-wins: two lanes disagreeing about
|
||||
an address is exactly what must not be papered over.
|
||||
|
||||
## 15. Report thin coverage as loudly as divergences
|
||||
|
||||
A green run that establishes twenty facts is not a strong result. `ComputeBudget` compared 4,284
|
||||
calls with 0 divergences — but only **20 distinct states**, with 4,278 of them the UI polling one
|
||||
player, and **13 of 22 slots zero on every call**, including 5 of 6 declared input-boundary slots.
|
||||
|
||||
The most useful sections of the best reports here have been the honest lists of what was *not*
|
||||
covered.
|
||||
Loading…
Add table
Reference in a new issue