# B3 — `TechTree::ProcessResearch` old-vs-new, with the RNG state as a declared region

**Status (2026-09-08): code complete, cross-built and staged; every VM step still owed.**
VM140 is held by another lane for the whole of this milestone, so nothing was deployed, the
game was not stopped or relaunched, and `C:\SOTS\binkw32.dll` / `C:\SOTS\shimdist` were not
touched. Everything below is offline work plus what the *binary* says; the run list is at the
end. The build lives in its own tree (`/srv/re-lab/build/sots-engine-b3`) and its own dist
(`/srv/re-lab/shim/dist-b3`), not the shared ones.

The point of the target: one call exercises the MT19937, the completion-odds formula and the
Zuul double roll at once, so a match validates all three together — and its RNG consumption is
observable, which makes the draw *count* checkable rather than merely plausible.

## What was hooked

| hook name (record `hook`) | RVA | prototype |
|---|---|---|
| `Game::TechTree::ProcessResearch` | 0x001876c0 | `void (TechTree*, Mars::RNG*, vector<{TechDef*,int}>*, int*)` |

`__thiscall`, `[verified]`, so it goes through `Hook<>` with `CallConv::Thiscall` (M2's
addition). Source: `src/shim/hooks/research.{h,cpp}`, installed from `src/shim/main.cpp` after
the M1/M2 hooks. There is exactly one caller (inside `ServerPlayer::ProcessTurn`), which fires
once per player per turn.

**The second parameter was a `?` in the address contract; it is the `Mars::RNG` object.** The
call site loads it from `StrategyServer+0x16c` and the function re-bases it with `+4` before
every draw. That is now in `ghidra/addresses.json` along with the tree/node offsets, the
generator layout and `TechTree::Cost`, and regenerated into `sots_addresses.h`.

### Region model

Three kinds of region, all snapshotted before the original runs:

| region | size | describer |
|---|---|---|
| `rng` | 0x9cc | `{vptr:ptr, mt:bytes(2496 → sha256+head), left:i32, next_index:i64}` |
| `overbudget` | 4 | `{v:i32}` |
| `node[i]` | 0x34 each, one per non-null slot | `{def, tech_id, kids_*, unk10, state, cost_rp, progress, turn_available, turn_researched, order, flag, unk30}` |

`next` is a heap address, so it is reported as its index into `mt` — which is what it means,
and what survives being written by a reimplementation. `left` alone already pins the stream
position (`next == &mt[624 - left]` always), so the index is a cross-check, not the evidence.

Args carry the evidence a golden log needs to replay offline: `tree`, `owner`, `species`,
`node_count`, `rng`, `rng_left_in`, the `alloc` list as `{tech_id, points}`, `overbudget_in`,
and **`fpu_cw`** — the x87 control word in force for the call (see "Float mapping" below).

Declaring *every* node, not just the ones we expect to change, is deliberate: it is what proves
`ours` neither misses a write nor makes an extra one. The cost is size — a compare record is
roughly 200 nodes × three snapshots, so the b3 configs turn every other hook off.

### The comparison design

`ProcessResearch` consumes RNG, so comparing two implementations that draw from different
streams would diverge for a reason that has nothing to do with the formulas. Instead:

1. the generator object is a **declared region**, so its `mt[624]` + `left` are snapshotted
   before the original runs, alongside the tech-tree state;
2. the original runs and advances the real generator;
3. `ours` runs on the scratch copies, and seeds a `mars::rng::MT19937` with `load_state()` from
   the *pre-call* snapshot — so both implementations read the identical stream;
4. `ours` writes its final generator state back into the scratch copy, so the diff compares the
   **post-call RNG state** as well as the outputs.

If the post-states match, we consumed the same words in the same order. That is the check with
teeth: getting the odds right but drawing twice (or not drawing at all) moves `left` and the
hash. `tests/mars_stream/test_rng.cpp` pins the property offline, including the negative case
(one draw too few leaves a different state).

`next` is rebuilt by `ours` against the **live** generator address so the describer's index
arithmetic reads the same on both sides; `ours` never dereferences it.

## What `ours` covers, and what it deliberately does not

`ours` is `sots::sim::ProcessResearchTurn` (new, `src/game/sim/research.{h,cpp}`) plus a thin
shim adapter. It reproduces exactly the words the hooked function writes itself:

* the allocation loop — spend window, spend, `*overbudget`, progress, the roll, the completion
  decision, the over-budget flag, the "completed early" flag, `state = 4` on completion;
* the decay sweep over every available node.

It does **not** reproduce `TechTree::SetResearched`, which the original calls on completion:
the turn/order stamps, the child-unlock cascade and the owner's tech-effect callback. That is
its own milestone, and the callback writes live player state that compare mode must never
touch. So:

> **A turn on which a tech completes is expected to diverge**, in the completing node's
> `turn_researched` / `order` and in the child nodes `SetResearched` unlocks — and in nothing
> else, including the RNG (the cascade makes no draw; verified statically, though the owner
> callback itself was not audited). A turn on which nothing completes — the overwhelmingly
> common case — must match everywhere.

The effective cost of a node comes from the game's own `TechTree::Cost` (read-only: it only
reads `costRP`, the def and the owner, and calls the read-only cost-multiplier helper). The
cost multiplier is a separate, medium-confidence formula and not what this milestone measures;
this is the same delegation M2 makes to `LoadWeapon`. `ours` receives the live tree pointer and
treats it as read-only — every node it writes is a scratch copy.

`ours` also works in `replace` mode, where no `regions`/`rebind` ran: it then reads the tree's
own node vector and the live generator. The per-call statics carry a flag that is cleared at
the end of every `ours`, so a replace call can never inherit a stale compare mapping.

## Float mapping — the headline finding

**A draw is `y / (2^32 − 1)`, not `y × 2^-32`.** The multiplier in the image is the double
`0x3df0000000001000`, which is exactly `1/4294967295`; the constant next to it is the `+2^32`
unsigned fix-up applied after a sign-extending integer load. So:

* `MT19937::kUnitScale` is now `1.0 / 4294967295.0`;
* the range is **closed**: `y == 0xffffffff` maps to exactly `1.0`, not to just below it;
* the value is left in `st(0)` and the caller narrows it — every consumer in the strategic sim
  stores it to a 4-byte float first, which is what `next_float()` models.

Honest caveat, because it decides how to read a passing compare: the old and new divisors differ
by 2^-32 relative, far below a float32 ulp. Measured over 10^6 draws they give a **different
float 0.78 % of the time**, and they flip an actual research completion decision (roll vs an
odds of 1/3) **0 times in 10^6** — the expected rate is about one in two billion. So the
compare cannot prove the divisor; the disassembly and the constant's bit pattern do, and the
compare's job is the rest.

The one thing the binary cannot settle is the x87 **precision-control** field at run time. At
the MSVC default (53-bit, `cw = 0x027f`) the multiply rounds to double and the caller's store
rounds again — that is what `next_float()` does. A Direct3D 9 device created without
`FPU_PRESERVE` leaves 24-bit precision, in which the fix-up and the multiply each round to 24
bits; `MT19937::float_from_pc24()` models that, and the two differ for **0.094 %** of words —
last-bit only. The hook records `fpu_cw` on every call, so the first trace settles it; if it
comes back 24-bit the change is to route `next_float` through `float_from_pc24` and to compute
the odds the same way, and nothing else in the milestone moves.

`float10` in the decompile is just the i386 float return ABI and was not chased.

## Bugs found and fixed in our implementation

All five are read off the instruction sequence, not tuned to make anything match.

1. **The unit divisor** (above): `2^-32` → `1/(2^32 − 1)`.
2. **`NextInt` is inclusive.** The rejection mask is built from `n` itself, not `n − 1`, and the
   loop re-draws while the masked word is **greater than** `n` — so the result is uniform on
   `[0, n]`, one value wider than we had. The bound is also passed **by pointer**, which is why
   the prototype had stayed `[unverified]`. `next_int(n)` is now `next_int_inclusive(n)` and
   `IRandom::NextInt` is `IRandom::NextIntInclusive`, so every call site had to be re-read.
3. **`spend` has no floor at zero.** The original is a plain signed `min(points, hi − progress)`;
   ours clamped it at 0, which would have hidden a negative spend (and understated `*overbudget`)
   whenever progress was already past the 150 % ceiling.
4. **`odds` is a float32.** The original computes `(progress − lo) / hi` on the x87 and stores it
   to a 4-byte slot before the comparison; ours kept it in double. Likewise the roll, the Zuul
   minimum, and the `progress / cost` ratio used for the "completed early" flag.
5. **Two constants are widened float literals, not decimals.** The decay fraction is
   `(double)0.05f = 0.05000000074505806` and the early-completion threshold is
   `(double)0.8f = 0.800000011920929`. Both sit on a truncation/compare boundary.

Two smaller ones in the same pass: the decay guard is `progress != 0`, not `progress > 0`; and
a node whose cost is still `INT_MAX` is **not** special-cased by the original — it feeds that
straight into the 5 % multiply, which wipes any progress out. The 50 %/150 % bounds are a
32-bit multiply that wraps near `INT_MAX` rather than a widening one.

## Host tests

`ctest` 26/26. New coverage:

* `mars_rng_unit` — the standard MT19937 vectors (seed 5489, and the 10000th output) were
  already there; added the unit mapping word by word (`0 → 0`, `0xffffffff → 1.0`, the
  high-bit fix-up path), the measured rarity of the divisor difference, the PC24 variant's
  bound, `cover_mask`, the inclusive integer bound (including that `n` itself is reachable and
  that `n == 0` still consumes a word), and **the compare design end to end**: snapshot →
  original draws → ours seeded from the snapshot reproduces the values and the post-state, with
  a negative case that one draw too few does not.
* `game_sim_research` — 121 checks. Added the spend window (truncation, the floor/ceiling
  clamps, the `INT_MAX` edge), that the odds are float32, that `spend` goes negative rather
  than clamping, the early-completion boundary at exactly 80 % versus one point below, and
  `ProcessResearchTurn` (order of the passes, the funded node decaying too, hidden slots never
  touched, an out-of-range entry consuming no draw, `overbudget` accumulating).

Cross-build: `b3-81218c7-dirty-20260908T0311Z`, exports 66 names identical to `binkw32.dll`,
staged in `/srv/re-lab/shim/dist-b3`. `tools/clean_room_check.sh` OK.

## Gotchas

1. **Two different `this` pointers for one object.** `Twist`, `NextFloat` and `NextInt` take
   `&mt` — the object **plus 4** — so *their* `this+0x9c0/+0x9c4` are `next`/`left`, while the
   object's own layout is `{vftable @+0, mt[624] @+4, next @+0x9c4, left @+0x9c8}` = 0x9cc
   bytes. The address contract used to state both readings as if they were one; it now says
   which is which. Getting this wrong shifts every generator field by a word.
2. The save blob is 0x9c4 bytes = `mt[624]` + `left`, i.e. it **skips** `next`, which sits
   between them in the object. `left` is sufficient because `next == &mt[624 - left]` and the
   original's `Read` recomputes it.
3. `Region::name` is a `const char*` held for the whole call, so the per-node name strings are
   `resize`d once up front and never grown — a reallocation would dangle every name already
   pushed.
4. Per-call state is kept in statics between `regions()` → `rebind()` → `ours()` (M1's
   concession). Safe here because the turn pass is single-threaded and `ProcessResearch` has one
   caller and never nests; do not copy the pattern to a re-entrant hook.
5. A compare record is large (one region per tech node). The staged configs
   (`shim.cfg.b3{trace,compare,replace}`, in the dist) switch every other hook off for that
   reason; `trace.inline_max` stays at 256 so the 2496-byte state block is hashed rather than
   inlined.
6. `TechDef`'s first word is the tech id and it indexes `TechTree+0x10`; the allocation vector's
   stride is 8 (`{TechDef*, int}`). Both are in the address contract now rather than inferred.

## What remains (needs the VM)

The lane holding VM140 must be finished first; then, in this order:

1. Deploy `/srv/re-lab/shim/dist-b3` (build `b3-81218c7-dirty-20260908T0311Z`): `scp` it to
   `C:\SOTS\shimdist-b3\` and run `deploy.ps1 -Dist C:\SOTS\shimdist-b3` — **a separate staging
   directory from the shared `C:\SOTS\shimdist`**, so the other lane's dist is not overwritten.
2. Copy `shim.cfg.b3trace` over `C:\SOTS\shim.cfg`, relaunch, load `ref-turn2.sav`, press End
   Turn once, and pull `C:\SOTS\shim.trace.jsonl` → `b3-trace-golden.jsonl`.
   `tracecmp.py` must exit 0 with 0 invalid records. Expect one record per player that both
   has a research target and did not suffer a lab accident that turn — the caller gates the
   call on `if (ResT && !RollResearchAccident())` — so a turn with no records at all is a
   setup problem, not a pass.
   **Read off this trace before going further:** `fpu_cw` (expect `0x027f`; `0x007f`/`0x003f`
   means 24-bit precision and the `float_from_pc24` route), and, per record, `rng_left_in`
   minus the after-state's `left` — 1 for a non-Zuul player who rolled, 2 for a Zuul, 0 when
   the spend was zero or the tech was already at the 150 % ceiling.
3. Copy `shim.cfg.b3compare`, relaunch, load `ref-turn2.sav`, End Turn → `b3-compare.jsonl`.
   Expect **0 divergences on every record where no tech completed**. On a record whose
   `ours` shows a completion, the only permitted diffs are `turn_researched` / `order` on that
   node and the child nodes `SetResearched` unlocked; the `rng` region must still match. Any
   other diff is a real finding — report it, do not tune the formula.
   To get more turns, keep pressing End Turn; each turn adds one record per player.
4. If the compare is clean: copy `shim.cfg.b3replace`, relaunch, load `ref-turn2.sav`, End Turn,
   and check the determinism oracle — `(Autosave).sav` = `978041ac…`, `(Autosave EndTurn).sav`
   = `bb4fd9ac…`. This is the strongest single result available: replace mode means our research
   pass fed the game, and the save hash means the whole turn still landed byte for byte.
5. Restore the previous `shim.cfg` (`hooks=trace`) and leave the game at the main menu, as M1/M2
   left it.

Not done, and worth saying: the compare above cannot distinguish the two unit divisors (see
"Float mapping"), and it will not exercise the Zuul path at all unless a Zuul player is in the
reference save — check `species` in the trace args, and if none is 5, run one more compare from
a Zuul save before calling the double roll verified by behaviour rather than by disassembly.