327 lines
20 KiB
Markdown
327 lines
20 KiB
Markdown
# B3 — `TechTree::ProcessResearch` old-vs-new, with the RNG state as a declared region
|
||
|
||
**Result (2026-09-08): verified on the live game.** Trace: 3 calls, `tracecmp.py` exit 0.
|
||
Compare: **15 calls, 13 with zero divergence**; the two that diverged are both completion turns
|
||
and every diff is in a field `TechTree::SetResearched` owns — the documented scope boundary —
|
||
plus one extra RNG draw on one of them (below). The RNG post-state matched on 14 of 15 calls,
|
||
including every roll. Replace: our research pass fed the game for a whole turn from
|
||
`ref-turn2.sav` and the resulting 609 KB autosave is **item-for-item identical to the oracle
|
||
except for one missing event** (`EVENT_RESEARCH_OVERBUDGET`), so the End-Turn oracle does **not**
|
||
pass yet — see "The replace-mode gap". Game left at the main menu in `hooks=trace`.
|
||
|
||
Build `b3-b791392-20260908T0353Z`, staged in `/srv/re-lab/shim/dist-b3`, deployed to the VM from
|
||
`C:\SOTS\shimdist-b3`. Evidence in `/srv/re-lab/shim/traces/b3-*`.
|
||
|
||
The point of the target: one call exercises the MT19937, the completion-odds formula and the
|
||
Zuul double roll at once, so a match validates all three together — and its RNG consumption is
|
||
observable, which makes the draw *count* checkable rather than merely plausible.
|
||
|
||
## What was hooked
|
||
|
||
| hook name (record `hook`) | RVA | prototype |
|
||
|---|---|---|
|
||
| `Game::TechTree::ProcessResearch` | 0x001876c0 | `void (TechTree*, Mars::RNG*, vector<{TechDef*,int}>*, int*)` |
|
||
|
||
`__thiscall`, `[verified]`, so it goes through `Hook<>` with `CallConv::Thiscall` (M2's
|
||
addition). Source: `src/shim/hooks/research.{h,cpp}`, installed from `src/shim/main.cpp` after
|
||
the M1/M2 hooks. There is exactly one caller (inside `ServerPlayer::ProcessTurn`), which fires
|
||
once per player per turn.
|
||
|
||
**The second parameter was a `?` in the address contract; it is the `Mars::RNG` object.** The
|
||
call site loads it from `StrategyServer+0x16c` and the function re-bases it with `+4` before
|
||
every draw. That is now in `ghidra/addresses.json` along with the tree/node offsets, the
|
||
generator layout and `TechTree::Cost`, and regenerated into `sots_addresses.h`.
|
||
|
||
### Region model
|
||
|
||
Three kinds of region, all snapshotted before the original runs:
|
||
|
||
| region | size | describer |
|
||
|---|---|---|
|
||
| `rng` | 0x9cc | `{vptr:ptr, mt:bytes(2496 → sha256+head), left:i32, next_index:i64}` |
|
||
| `overbudget` | 4 | `{v:i32}` |
|
||
| `node[i]` | 0x34 each, one per non-null slot | `{def, tech_id, kids_*, unk10, state, cost_rp, progress, turn_available, turn_researched, order, flag, unk30}` |
|
||
|
||
`next` is a heap address, so it is reported as its index into `mt` — which is what it means,
|
||
and what survives being written by a reimplementation. `left` alone already pins the stream
|
||
position (`next == &mt[624 - left]` always), so the index is a cross-check, not the evidence.
|
||
|
||
Args carry the evidence a golden log needs to replay offline: `tree`, `owner`, `species`,
|
||
`node_count`, `rng`, `rng_left_in`, the `alloc` list as `{tech_id, points}`, `overbudget_in`,
|
||
and **`fpu_cw`** — the x87 control word in force for the call (see "Float mapping" below).
|
||
|
||
Declaring *every* node, not just the ones we expect to change, is deliberate: it is what proves
|
||
`ours` neither misses a write nor makes an extra one. The cost is size — a compare record is
|
||
roughly 200 nodes × three snapshots, so the b3 configs turn every other hook off.
|
||
|
||
### The comparison design
|
||
|
||
`ProcessResearch` consumes RNG, so comparing two implementations that draw from different
|
||
streams would diverge for a reason that has nothing to do with the formulas. Instead:
|
||
|
||
1. the generator object is a **declared region**, so its `mt[624]` + `left` are snapshotted
|
||
before the original runs, alongside the tech-tree state;
|
||
2. the original runs and advances the real generator;
|
||
3. `ours` runs on the scratch copies, and seeds a `mars::rng::MT19937` with `load_state()` from
|
||
the *pre-call* snapshot — so both implementations read the identical stream;
|
||
4. `ours` writes its final generator state back into the scratch copy, so the diff compares the
|
||
**post-call RNG state** as well as the outputs.
|
||
|
||
If the post-states match, we consumed the same words in the same order. That is the check with
|
||
teeth: getting the odds right but drawing twice (or not drawing at all) moves `left` and the
|
||
hash. `tests/mars_stream/test_rng.cpp` pins the property offline, including the negative case
|
||
(one draw too few leaves a different state).
|
||
|
||
`next` is rebuilt by `ours` against the **live** generator address so the describer's index
|
||
arithmetic reads the same on both sides; `ours` never dereferences it.
|
||
|
||
## What `ours` covers, and what it deliberately does not
|
||
|
||
`ours` is `sots::sim::ProcessResearchTurn` (new, `src/game/sim/research.{h,cpp}`) plus a thin
|
||
shim adapter. It reproduces exactly the words the hooked function writes itself:
|
||
|
||
* the allocation loop — spend window, spend, `*overbudget`, progress, the roll, the completion
|
||
decision, the over-budget flag, the "completed early" flag, `state = 4` on completion;
|
||
* the decay sweep over every available node.
|
||
|
||
It does **not** reproduce `TechTree::SetResearched`, which the original calls on completion:
|
||
the turn/order stamps, the child-unlock cascade and the owner's tech-effect callback. That is
|
||
its own milestone, and the callback writes live player state that compare mode must never
|
||
touch. So:
|
||
|
||
> **A turn on which a tech completes is expected to diverge**, in the completing node's
|
||
> `turn_researched` / `order` and in the child nodes `SetResearched` unlocks. A turn on which
|
||
> nothing completes — the overwhelmingly common case — must match everywhere.
|
||
> *(Borne out: the two divergent calls of the live compare carry exactly those fields and
|
||
> nothing else. The prediction that the cascade also makes no RNG draw was **half wrong** — one
|
||
> of the two completions consumed an extra word inside the owner's tech-effect callback; see
|
||
> "The extra draw" below.)*
|
||
|
||
It also does **not** post the events the pass raises — and that turned out to matter: the
|
||
over-budget branch sets `node.flag = 2` *and* pushes an `EVENT_RESEARCH_OVERBUDGET` onto the
|
||
owner's event list, and only the flag is modelled. The event list was never a declared region,
|
||
so compare mode is blind to it; the End-Turn oracle is what caught it. See "The replace-mode
|
||
gap".
|
||
|
||
The effective cost of a node comes from the game's own `TechTree::Cost` (read-only: it only
|
||
reads `costRP`, the def and the owner, and calls the read-only cost-multiplier helper). The
|
||
cost multiplier is a separate, medium-confidence formula and not what this milestone measures;
|
||
this is the same delegation M2 makes to `LoadWeapon`. `ours` receives the live tree pointer and
|
||
treats it as read-only — every node it writes is a scratch copy.
|
||
|
||
`ours` also works in `replace` mode, where no `regions`/`rebind` ran: it then reads the tree's
|
||
own node vector and the live generator. The per-call statics carry a flag that is cleared at
|
||
the end of every `ours`, so a replace call can never inherit a stale compare mapping.
|
||
|
||
## Float mapping — the headline finding
|
||
|
||
**A draw is `y / (2^32 − 1)`, not `y × 2^-32`.** The multiplier in the image is the double
|
||
`0x3df0000000001000`, which is exactly `1/4294967295`; the constant next to it is the `+2^32`
|
||
unsigned fix-up applied after a sign-extending integer load. So:
|
||
|
||
* `MT19937::kUnitScale` is now `1.0 / 4294967295.0`;
|
||
* the range is **closed**: `y == 0xffffffff` maps to exactly `1.0`, not to just below it;
|
||
* the value is left in `st(0)` and the caller narrows it — every consumer in the strategic sim
|
||
stores it to a 4-byte float first, which is what `next_float()` models.
|
||
|
||
Honest caveat, because it decides how to read a passing compare: the old and new divisors differ
|
||
by 2^-32 relative, far below a float32 ulp. Measured over 10^6 draws they give a **different
|
||
float 0.78 % of the time**, and they flip an actual research completion decision (roll vs an
|
||
odds of 1/3) **0 times in 10^6** — the expected rate is about one in two billion. So the
|
||
compare cannot prove the divisor; the disassembly and the constant's bit pattern do, and the
|
||
compare's job is the rest.
|
||
|
||
The one thing the binary cannot settle is the x87 **precision-control** field at run time. At
|
||
the MSVC default (53-bit, `cw = 0x027f`) the multiply rounds to double and the caller's store
|
||
rounds again — that is what `next_float()` does. A Direct3D 9 device created without
|
||
`FPU_PRESERVE` leaves 24-bit precision, in which the fix-up and the multiply each round to 24
|
||
bits; `MT19937::float_from_pc24()` models that, and the two differ for **0.094 %** of words —
|
||
last-bit only. The hook records `fpu_cw` on every call, so the first trace settles it; if it
|
||
comes back 24-bit the change is to route `next_float` through `float_from_pc24` and to compute
|
||
the odds the same way, and nothing else in the milestone moves.
|
||
|
||
`float10` in the decompile is just the i386 float return ABI and was not chased.
|
||
|
||
## Bugs found and fixed in our implementation
|
||
|
||
All five are read off the instruction sequence, not tuned to make anything match.
|
||
|
||
1. **The unit divisor** (above): `2^-32` → `1/(2^32 − 1)`.
|
||
2. **`NextInt` is inclusive.** The rejection mask is built from `n` itself, not `n − 1`, and the
|
||
loop re-draws while the masked word is **greater than** `n` — so the result is uniform on
|
||
`[0, n]`, one value wider than we had. The bound is also passed **by pointer**, which is why
|
||
the prototype had stayed `[unverified]`. `next_int(n)` is now `next_int_inclusive(n)` and
|
||
`IRandom::NextInt` is `IRandom::NextIntInclusive`, so every call site had to be re-read.
|
||
3. **`spend` has no floor at zero.** The original is a plain signed `min(points, hi − progress)`;
|
||
ours clamped it at 0, which would have hidden a negative spend (and understated `*overbudget`)
|
||
whenever progress was already past the 150 % ceiling.
|
||
4. **`odds` is a float32.** The original computes `(progress − lo) / hi` on the x87 and stores it
|
||
to a 4-byte slot before the comparison; ours kept it in double. Likewise the roll, the Zuul
|
||
minimum, and the `progress / cost` ratio used for the "completed early" flag.
|
||
5. **Two constants are widened float literals, not decimals.** The decay fraction is
|
||
`(double)0.05f = 0.05000000074505806` and the early-completion threshold is
|
||
`(double)0.8f = 0.800000011920929`. Both sit on a truncation/compare boundary.
|
||
|
||
Two smaller ones in the same pass: the decay guard is `progress != 0`, not `progress > 0`; and
|
||
a node whose cost is still `INT_MAX` is **not** special-cased by the original — it feeds that
|
||
straight into the 5 % multiply, which wipes any progress out. The 50 %/150 % bounds are a
|
||
32-bit multiply that wraps near `INT_MAX` rather than a widening one.
|
||
|
||
## Host tests
|
||
|
||
`ctest` 26/26. New coverage:
|
||
|
||
* `mars_rng_unit` — the standard MT19937 vectors (seed 5489, and the 10000th output) were
|
||
already there; added the unit mapping word by word (`0 → 0`, `0xffffffff → 1.0`, the
|
||
high-bit fix-up path), the measured rarity of the divisor difference, the PC24 variant's
|
||
bound, `cover_mask`, the inclusive integer bound (including that `n` itself is reachable and
|
||
that `n == 0` still consumes a word), and **the compare design end to end**: snapshot →
|
||
original draws → ours seeded from the snapshot reproduces the values and the post-state, with
|
||
a negative case that one draw too few does not.
|
||
* `game_sim_research` — 121 checks. Added the spend window (truncation, the floor/ceiling
|
||
clamps, the `INT_MAX` edge), that the odds are float32, that `spend` goes negative rather
|
||
than clamping, the early-completion boundary at exactly 80 % versus one point below, and
|
||
`ProcessResearchTurn` (order of the passes, the funded node decaying too, hidden slots never
|
||
touched, an out-of-range entry consuming no draw, `overbudget` accumulating).
|
||
|
||
Cross-build: `b3-81218c7-dirty-20260908T0311Z`, exports 66 names identical to `binkw32.dll`,
|
||
staged in `/srv/re-lab/shim/dist-b3`. `tools/clean_room_check.sh` OK.
|
||
|
||
## Gotchas
|
||
|
||
1. **Two different `this` pointers for one object.** `Twist`, `NextFloat` and `NextInt` take
|
||
`&mt` — the object **plus 4** — so *their* `this+0x9c0/+0x9c4` are `next`/`left`, while the
|
||
object's own layout is `{vftable @+0, mt[624] @+4, next @+0x9c4, left @+0x9c8}` = 0x9cc
|
||
bytes. The address contract used to state both readings as if they were one; it now says
|
||
which is which. Getting this wrong shifts every generator field by a word.
|
||
2. The save blob is 0x9c4 bytes = `mt[624]` + `left`, i.e. it **skips** `next`, which sits
|
||
between them in the object. `left` is sufficient because `next == &mt[624 - left]` and the
|
||
original's `Read` recomputes it.
|
||
3. `Region::name` is a `const char*` held for the whole call, so the per-node name strings are
|
||
`resize`d once up front and never grown — a reallocation would dangle every name already
|
||
pushed.
|
||
4. Per-call state is kept in statics between `regions()` → `rebind()` → `ours()` (M1's
|
||
concession). Safe here because the turn pass is single-threaded and `ProcessResearch` has one
|
||
caller and never nests; do not copy the pattern to a re-entrant hook.
|
||
5. A compare record is large (one region per tech node). The staged configs
|
||
(`shim.cfg.b3{trace,compare,replace}`, in the dist) switch every other hook off for that
|
||
reason; `trace.inline_max` stays at 256 so the 2496-byte state block is hashed rather than
|
||
inlined.
|
||
6. `TechDef`'s first word is the tech id and it indexes `TechTree+0x10`; the allocation vector's
|
||
stride is 8 (`{TechDef*, int}`). Both are in the address contract now rather than inferred.
|
||
|
||
## Runs (`/srv/re-lab/shim/traces/`, build `b3-b791392-20260908T0353Z`)
|
||
|
||
| file | mode | calls | result |
|
||
|---|---|---|---|
|
||
| `b3-trace-golden.jsonl` | trace | 3 | `tracecmp.py` exit 0, 0 invalid |
|
||
| `b3-compare.jsonl` | compare | 15 | **13 clean, 2 diverged** (both completion turns; see below) |
|
||
| `b3-replace-autosave.sav` vs `b3-orig-autosave.sav` | replace | 3 | `(Autosave EndTurn).sav` = `bb4fd9ac…` (oracle); `(Autosave).sav` = `fd071d18…` ≠ oracle `978041ac…` |
|
||
| `b3-replace-save-diff.txt` | | | the whole difference: one missing event |
|
||
| `b3-shim.log`, `b3-final-menu.png`, `b3-compare-turn7.png` | | | banners, final state |
|
||
|
||
Procedure: `ref-turn2.sav` → End Turn ×1 (trace), ×5 (compare), ×1 (replace and the oracle
|
||
re-check). Each End Turn produces three calls: the AI Tarkas player (which actually researches),
|
||
the human, and a second Tarkas tree, the last two with a zero-point allocation.
|
||
|
||
### What the compare proves
|
||
|
||
Per call, the trace records the draw count as `left` before minus after:
|
||
|
||
| call | species | alloc `{tech, points}` | draws (orig / ours) | outcome |
|
||
|---|---|---|---|---|
|
||
| 0 | Tarkas | {144, 2889} | 1 / 1 | over-budget event raised, flag 1 → 2 |
|
||
| 1, 2 | Human, Tarkas | {90, 0}, {9, 0} | 0 / 0 | zero spend → odds 0, roll 1, no draw |
|
||
| 3 | Tarkas | {144, 2898} | 0 / 0 | reached the 150 % ceiling → guaranteed, **no draw**, completed |
|
||
| 6 | Tarkas | {142, 3064} | 1 / 1 | rolled and failed (odds −0.078, below 50 % of cost) |
|
||
| 9 | Tarkas | {142, 3074} | **2 / 1** | rolled at odds 0.178 and **completed early** (6138/8000 = 0.767) |
|
||
| 12 | Tarkas | {9, 3086} | 1 / 1 | rolled and failed |
|
||
|
||
Every integer the pass computes matched: progress, the spend cap, `*overbudget`, and the flag on
|
||
every node, including the three interesting branches — the over-budget event flag (call 0), the
|
||
guaranteed completion at the ceiling with no draw at all (call 3), and the "completed early"
|
||
flag 1 → 0 below 80 % of cost (call 9). On call 9 ours drew the same word as the original (the
|
||
completion decision and the early flag both match), so the extra draw is the original's, not a
|
||
missing one of ours.
|
||
|
||
**The two divergences.** Both are completion turns, and both carry only `SetResearched`-owned
|
||
fields: `order` and `turn_researched` on the completing node, and `cost_rp` / `state` /
|
||
`turn_available` on the children it unlocks (three children on call 3, one on call 9). That is
|
||
exactly the set this milestone declared out of scope in advance.
|
||
|
||
**The extra draw (call 9).** The original consumed one word more than ours *after* the completion
|
||
roll. Call 3 also completed a tech and consumed no extra word, so this is not "SetResearched
|
||
always draws" — it is the owner's tech-effect callback for that particular tech. That callback is
|
||
`ServerPlayer::OnTechResearched`, which is the B2 lane's target; B2 should expect to find at
|
||
least one effect that rolls.
|
||
|
||
### The replace-mode gap — one missing event
|
||
|
||
Replace mode ran our pass for a whole turn. `(Autosave EndTurn).sav` (the pre-turn state) is the
|
||
oracle's `bb4fd9ac…`, but the post-turn `(Autosave).sav` is `fd071d18…` and not the oracle's
|
||
`978041ac…`. Dumping both trees and normalising the offsets, the **entire** difference in 40,300
|
||
items is:
|
||
|
||
```
|
||
Player > Events > Events > . > Events: one entry present in the original, absent in ours
|
||
EvEID 3 EvDsc "Research Over Budget"
|
||
EvMsg "Research for Waldo Units has gone overbudget."
|
||
EvImg "EVENT_RESEARCH_OVERBUDGET" EvLoc 0 EvPos {inf,inf,inf} EvAct 1 EvCID 0
|
||
+ the two counters that index it: EvNxID 4 → 3 and the list length 2 → 1
|
||
```
|
||
|
||
Nothing else — not one number the research pass computes — differs. So the formulas are right and
|
||
the milestone under-scoped itself: `docs/B3.md` promised the over-budget *flag* and treated the
|
||
event beside it as part of the same branch without saying it was excluded. It is not modelled,
|
||
`ours` writes `node.flag = 2` and stops there, and compare mode could not see it because the
|
||
player's event list was never a declared region.
|
||
|
||
Two consequences worth carrying forward:
|
||
|
||
* **The oracle catches what a region-scoped compare cannot.** Compare mode was clean on exactly
|
||
the call that produced the missing event, because the event lands somewhere we did not declare.
|
||
A clean compare bounds the state you declared, nothing more.
|
||
* To close this, `ours` needs to post the event through the game's own event API (the same call
|
||
the original makes right after setting the flag), the way M2 delegates to `LoadWeapon` — the
|
||
message text is composed from the tech name, so it cannot be synthesised from our side alone.
|
||
Until then, **replace mode is not oracle-clean** and should not be used as a determinism check
|
||
for this hook.
|
||
|
||
## Corrections this run made to the notes
|
||
|
||
1. **The current research target has state 3, not 2 — so it does not decay.** Every funded node
|
||
in the trace is state 3 (`node[144]`, `node[142]`, `node[9]`), and the decay loop's guard is
|
||
`state == 2`. `formula-gaps.md` Q8 read the loop correctly but concluded the funded node
|
||
decays; it does not, because the tree marks the selected target with a different state. Our
|
||
code was already right (it compares against `Available == 2`); the prose was wrong. The state
|
||
set observed in one tree: 164 hidden (0), 7 parent-researched (1), 23 available (2), 1 current
|
||
target (3), 22 researched (4).
|
||
2. **`fpu_cw` = `0x127f` at call time** (`0x027f` at DLL init): precision control = 10b = **53-bit
|
||
double**, rounding = nearest. Bit 12 is the legacy infinity-control flag and has no effect. So
|
||
the MSVC default path is the right one and `next_float()` — double product, then the caller's
|
||
narrowing to float32 — models the original exactly. `float_from_pc24()` stays as an unused
|
||
contingency; nothing needs to change.
|
||
3. **Nodes per tree = 293**, not the 196 the tech-name table suggested.
|
||
|
||
## Coverage this run did not reach
|
||
|
||
* **No Zuul.** `ref-turn2.sav` has two players, `re` (human) and `Fane Lao` (AI), and the trees
|
||
seen are species 0 (Human) and 2 (Tarkas). The double roll is therefore still verified by
|
||
disassembly and host tests only, never by behaviour. A compare from a save with a species-5
|
||
player would close it; the check is simply that `left` drops by 2 on a roll instead of 1.
|
||
* **The decay sweep never fired.** No node in any traced tree was state 2 with non-zero progress
|
||
— idle partially-researched techs do not occur in this save, and the current target is state 3.
|
||
`DecayAllResearch` is covered by host tests only.
|
||
* Only one lab-accident-free, single-target allocation shape was seen (one entry per call).
|
||
|
||
## If the replace-mode gap is closed
|
||
|
||
1. Add the event post to `ours` (see above), rebuild, restage `/srv/re-lab/shim/dist-b3`.
|
||
2. Deploy from `C:\SOTS\shimdist-b3`, copy `shim.cfg.b3replace` over `C:\SOTS\shim.cfg`,
|
||
relaunch, load `ref-turn2.sav`, End Turn once, and re-check the oracle:
|
||
`(Autosave).sav` = `978041ac…`, `(Autosave EndTurn).sav` = `bb4fd9ac…`.
|
||
3. Restore the default `shim.cfg` (`hooks=trace`) and leave the game at the main menu.
|
||
|
||
Note that replace mode also stops modelling completions the moment one happens on a later turn:
|
||
`ours` does not run the unlock cascade, so a replace run must not be pushed past the first
|
||
completion until `SetResearched` is a milestone of its own.
|