233 lines
15 KiB
Markdown
233 lines
15 KiB
Markdown
# B3 — `TechTree::ProcessResearch` old-vs-new, with the RNG state as a declared region
|
||
|
||
**Status (2026-09-08): code complete, cross-built and staged; every VM step still owed.**
|
||
VM140 is held by another lane for the whole of this milestone, so nothing was deployed, the
|
||
game was not stopped or relaunched, and `C:\SOTS\binkw32.dll` / `C:\SOTS\shimdist` were not
|
||
touched. Everything below is offline work plus what the *binary* says; the run list is at the
|
||
end. The build lives in its own tree (`/srv/re-lab/build/sots-engine-b3`) and its own dist
|
||
(`/srv/re-lab/shim/dist-b3`), not the shared ones.
|
||
|
||
The point of the target: one call exercises the MT19937, the completion-odds formula and the
|
||
Zuul double roll at once, so a match validates all three together — and its RNG consumption is
|
||
observable, which makes the draw *count* checkable rather than merely plausible.
|
||
|
||
## What was hooked
|
||
|
||
| hook name (record `hook`) | RVA | prototype |
|
||
|---|---|---|
|
||
| `Game::TechTree::ProcessResearch` | 0x001876c0 | `void (TechTree*, Mars::RNG*, vector<{TechDef*,int}>*, int*)` |
|
||
|
||
`__thiscall`, `[verified]`, so it goes through `Hook<>` with `CallConv::Thiscall` (M2's
|
||
addition). Source: `src/shim/hooks/research.{h,cpp}`, installed from `src/shim/main.cpp` after
|
||
the M1/M2 hooks. There is exactly one caller (inside `ServerPlayer::ProcessTurn`), which fires
|
||
once per player per turn.
|
||
|
||
**The second parameter was a `?` in the address contract; it is the `Mars::RNG` object.** The
|
||
call site loads it from `StrategyServer+0x16c` and the function re-bases it with `+4` before
|
||
every draw. That is now in `ghidra/addresses.json` along with the tree/node offsets, the
|
||
generator layout and `TechTree::Cost`, and regenerated into `sots_addresses.h`.
|
||
|
||
### Region model
|
||
|
||
Three kinds of region, all snapshotted before the original runs:
|
||
|
||
| region | size | describer |
|
||
|---|---|---|
|
||
| `rng` | 0x9cc | `{vptr:ptr, mt:bytes(2496 → sha256+head), left:i32, next_index:i64}` |
|
||
| `overbudget` | 4 | `{v:i32}` |
|
||
| `node[i]` | 0x34 each, one per non-null slot | `{def, tech_id, kids_*, unk10, state, cost_rp, progress, turn_available, turn_researched, order, flag, unk30}` |
|
||
|
||
`next` is a heap address, so it is reported as its index into `mt` — which is what it means,
|
||
and what survives being written by a reimplementation. `left` alone already pins the stream
|
||
position (`next == &mt[624 - left]` always), so the index is a cross-check, not the evidence.
|
||
|
||
Args carry the evidence a golden log needs to replay offline: `tree`, `owner`, `species`,
|
||
`node_count`, `rng`, `rng_left_in`, the `alloc` list as `{tech_id, points}`, `overbudget_in`,
|
||
and **`fpu_cw`** — the x87 control word in force for the call (see "Float mapping" below).
|
||
|
||
Declaring *every* node, not just the ones we expect to change, is deliberate: it is what proves
|
||
`ours` neither misses a write nor makes an extra one. The cost is size — a compare record is
|
||
roughly 200 nodes × three snapshots, so the b3 configs turn every other hook off.
|
||
|
||
### The comparison design
|
||
|
||
`ProcessResearch` consumes RNG, so comparing two implementations that draw from different
|
||
streams would diverge for a reason that has nothing to do with the formulas. Instead:
|
||
|
||
1. the generator object is a **declared region**, so its `mt[624]` + `left` are snapshotted
|
||
before the original runs, alongside the tech-tree state;
|
||
2. the original runs and advances the real generator;
|
||
3. `ours` runs on the scratch copies, and seeds a `mars::rng::MT19937` with `load_state()` from
|
||
the *pre-call* snapshot — so both implementations read the identical stream;
|
||
4. `ours` writes its final generator state back into the scratch copy, so the diff compares the
|
||
**post-call RNG state** as well as the outputs.
|
||
|
||
If the post-states match, we consumed the same words in the same order. That is the check with
|
||
teeth: getting the odds right but drawing twice (or not drawing at all) moves `left` and the
|
||
hash. `tests/mars_stream/test_rng.cpp` pins the property offline, including the negative case
|
||
(one draw too few leaves a different state).
|
||
|
||
`next` is rebuilt by `ours` against the **live** generator address so the describer's index
|
||
arithmetic reads the same on both sides; `ours` never dereferences it.
|
||
|
||
## What `ours` covers, and what it deliberately does not
|
||
|
||
`ours` is `sots::sim::ProcessResearchTurn` (new, `src/game/sim/research.{h,cpp}`) plus a thin
|
||
shim adapter. It reproduces exactly the words the hooked function writes itself:
|
||
|
||
* the allocation loop — spend window, spend, `*overbudget`, progress, the roll, the completion
|
||
decision, the over-budget flag, the "completed early" flag, `state = 4` on completion;
|
||
* the decay sweep over every available node.
|
||
|
||
It does **not** reproduce `TechTree::SetResearched`, which the original calls on completion:
|
||
the turn/order stamps, the child-unlock cascade and the owner's tech-effect callback. That is
|
||
its own milestone, and the callback writes live player state that compare mode must never
|
||
touch. So:
|
||
|
||
> **A turn on which a tech completes is expected to diverge**, in the completing node's
|
||
> `turn_researched` / `order` and in the child nodes `SetResearched` unlocks — and in nothing
|
||
> else, including the RNG (the cascade makes no draw; verified statically, though the owner
|
||
> callback itself was not audited). A turn on which nothing completes — the overwhelmingly
|
||
> common case — must match everywhere.
|
||
|
||
The effective cost of a node comes from the game's own `TechTree::Cost` (read-only: it only
|
||
reads `costRP`, the def and the owner, and calls the read-only cost-multiplier helper). The
|
||
cost multiplier is a separate, medium-confidence formula and not what this milestone measures;
|
||
this is the same delegation M2 makes to `LoadWeapon`. `ours` receives the live tree pointer and
|
||
treats it as read-only — every node it writes is a scratch copy.
|
||
|
||
`ours` also works in `replace` mode, where no `regions`/`rebind` ran: it then reads the tree's
|
||
own node vector and the live generator. The per-call statics carry a flag that is cleared at
|
||
the end of every `ours`, so a replace call can never inherit a stale compare mapping.
|
||
|
||
## Float mapping — the headline finding
|
||
|
||
**A draw is `y / (2^32 − 1)`, not `y × 2^-32`.** The multiplier in the image is the double
|
||
`0x3df0000000001000`, which is exactly `1/4294967295`; the constant next to it is the `+2^32`
|
||
unsigned fix-up applied after a sign-extending integer load. So:
|
||
|
||
* `MT19937::kUnitScale` is now `1.0 / 4294967295.0`;
|
||
* the range is **closed**: `y == 0xffffffff` maps to exactly `1.0`, not to just below it;
|
||
* the value is left in `st(0)` and the caller narrows it — every consumer in the strategic sim
|
||
stores it to a 4-byte float first, which is what `next_float()` models.
|
||
|
||
Honest caveat, because it decides how to read a passing compare: the old and new divisors differ
|
||
by 2^-32 relative, far below a float32 ulp. Measured over 10^6 draws they give a **different
|
||
float 0.78 % of the time**, and they flip an actual research completion decision (roll vs an
|
||
odds of 1/3) **0 times in 10^6** — the expected rate is about one in two billion. So the
|
||
compare cannot prove the divisor; the disassembly and the constant's bit pattern do, and the
|
||
compare's job is the rest.
|
||
|
||
The one thing the binary cannot settle is the x87 **precision-control** field at run time. At
|
||
the MSVC default (53-bit, `cw = 0x027f`) the multiply rounds to double and the caller's store
|
||
rounds again — that is what `next_float()` does. A Direct3D 9 device created without
|
||
`FPU_PRESERVE` leaves 24-bit precision, in which the fix-up and the multiply each round to 24
|
||
bits; `MT19937::float_from_pc24()` models that, and the two differ for **0.094 %** of words —
|
||
last-bit only. The hook records `fpu_cw` on every call, so the first trace settles it; if it
|
||
comes back 24-bit the change is to route `next_float` through `float_from_pc24` and to compute
|
||
the odds the same way, and nothing else in the milestone moves.
|
||
|
||
`float10` in the decompile is just the i386 float return ABI and was not chased.
|
||
|
||
## Bugs found and fixed in our implementation
|
||
|
||
All five are read off the instruction sequence, not tuned to make anything match.
|
||
|
||
1. **The unit divisor** (above): `2^-32` → `1/(2^32 − 1)`.
|
||
2. **`NextInt` is inclusive.** The rejection mask is built from `n` itself, not `n − 1`, and the
|
||
loop re-draws while the masked word is **greater than** `n` — so the result is uniform on
|
||
`[0, n]`, one value wider than we had. The bound is also passed **by pointer**, which is why
|
||
the prototype had stayed `[unverified]`. `next_int(n)` is now `next_int_inclusive(n)` and
|
||
`IRandom::NextInt` is `IRandom::NextIntInclusive`, so every call site had to be re-read.
|
||
3. **`spend` has no floor at zero.** The original is a plain signed `min(points, hi − progress)`;
|
||
ours clamped it at 0, which would have hidden a negative spend (and understated `*overbudget`)
|
||
whenever progress was already past the 150 % ceiling.
|
||
4. **`odds` is a float32.** The original computes `(progress − lo) / hi` on the x87 and stores it
|
||
to a 4-byte slot before the comparison; ours kept it in double. Likewise the roll, the Zuul
|
||
minimum, and the `progress / cost` ratio used for the "completed early" flag.
|
||
5. **Two constants are widened float literals, not decimals.** The decay fraction is
|
||
`(double)0.05f = 0.05000000074505806` and the early-completion threshold is
|
||
`(double)0.8f = 0.800000011920929`. Both sit on a truncation/compare boundary.
|
||
|
||
Two smaller ones in the same pass: the decay guard is `progress != 0`, not `progress > 0`; and
|
||
a node whose cost is still `INT_MAX` is **not** special-cased by the original — it feeds that
|
||
straight into the 5 % multiply, which wipes any progress out. The 50 %/150 % bounds are a
|
||
32-bit multiply that wraps near `INT_MAX` rather than a widening one.
|
||
|
||
## Host tests
|
||
|
||
`ctest` 26/26. New coverage:
|
||
|
||
* `mars_rng_unit` — the standard MT19937 vectors (seed 5489, and the 10000th output) were
|
||
already there; added the unit mapping word by word (`0 → 0`, `0xffffffff → 1.0`, the
|
||
high-bit fix-up path), the measured rarity of the divisor difference, the PC24 variant's
|
||
bound, `cover_mask`, the inclusive integer bound (including that `n` itself is reachable and
|
||
that `n == 0` still consumes a word), and **the compare design end to end**: snapshot →
|
||
original draws → ours seeded from the snapshot reproduces the values and the post-state, with
|
||
a negative case that one draw too few does not.
|
||
* `game_sim_research` — 121 checks. Added the spend window (truncation, the floor/ceiling
|
||
clamps, the `INT_MAX` edge), that the odds are float32, that `spend` goes negative rather
|
||
than clamping, the early-completion boundary at exactly 80 % versus one point below, and
|
||
`ProcessResearchTurn` (order of the passes, the funded node decaying too, hidden slots never
|
||
touched, an out-of-range entry consuming no draw, `overbudget` accumulating).
|
||
|
||
Cross-build: `b3-81218c7-dirty-20260908T0311Z`, exports 66 names identical to `binkw32.dll`,
|
||
staged in `/srv/re-lab/shim/dist-b3`. `tools/clean_room_check.sh` OK.
|
||
|
||
## Gotchas
|
||
|
||
1. **Two different `this` pointers for one object.** `Twist`, `NextFloat` and `NextInt` take
|
||
`&mt` — the object **plus 4** — so *their* `this+0x9c0/+0x9c4` are `next`/`left`, while the
|
||
object's own layout is `{vftable @+0, mt[624] @+4, next @+0x9c4, left @+0x9c8}` = 0x9cc
|
||
bytes. The address contract used to state both readings as if they were one; it now says
|
||
which is which. Getting this wrong shifts every generator field by a word.
|
||
2. The save blob is 0x9c4 bytes = `mt[624]` + `left`, i.e. it **skips** `next`, which sits
|
||
between them in the object. `left` is sufficient because `next == &mt[624 - left]` and the
|
||
original's `Read` recomputes it.
|
||
3. `Region::name` is a `const char*` held for the whole call, so the per-node name strings are
|
||
`resize`d once up front and never grown — a reallocation would dangle every name already
|
||
pushed.
|
||
4. Per-call state is kept in statics between `regions()` → `rebind()` → `ours()` (M1's
|
||
concession). Safe here because the turn pass is single-threaded and `ProcessResearch` has one
|
||
caller and never nests; do not copy the pattern to a re-entrant hook.
|
||
5. A compare record is large (one region per tech node). The staged configs
|
||
(`shim.cfg.b3{trace,compare,replace}`, in the dist) switch every other hook off for that
|
||
reason; `trace.inline_max` stays at 256 so the 2496-byte state block is hashed rather than
|
||
inlined.
|
||
6. `TechDef`'s first word is the tech id and it indexes `TechTree+0x10`; the allocation vector's
|
||
stride is 8 (`{TechDef*, int}`). Both are in the address contract now rather than inferred.
|
||
|
||
## What remains (needs the VM)
|
||
|
||
The lane holding VM140 must be finished first; then, in this order:
|
||
|
||
1. Deploy `/srv/re-lab/shim/dist-b3` (build `b3-81218c7-dirty-20260908T0311Z`): `scp` it to
|
||
`C:\SOTS\shimdist-b3\` and run `deploy.ps1 -Dist C:\SOTS\shimdist-b3` — **a separate staging
|
||
directory from the shared `C:\SOTS\shimdist`**, so the other lane's dist is not overwritten.
|
||
2. Copy `shim.cfg.b3trace` over `C:\SOTS\shim.cfg`, relaunch, load `ref-turn2.sav`, press End
|
||
Turn once, and pull `C:\SOTS\shim.trace.jsonl` → `b3-trace-golden.jsonl`.
|
||
`tracecmp.py` must exit 0 with 0 invalid records. Expect one record per player that both
|
||
has a research target and did not suffer a lab accident that turn — the caller gates the
|
||
call on `if (ResT && !RollResearchAccident())` — so a turn with no records at all is a
|
||
setup problem, not a pass.
|
||
**Read off this trace before going further:** `fpu_cw` (expect `0x027f`; `0x007f`/`0x003f`
|
||
means 24-bit precision and the `float_from_pc24` route), and, per record, `rng_left_in`
|
||
minus the after-state's `left` — 1 for a non-Zuul player who rolled, 2 for a Zuul, 0 when
|
||
the spend was zero or the tech was already at the 150 % ceiling.
|
||
3. Copy `shim.cfg.b3compare`, relaunch, load `ref-turn2.sav`, End Turn → `b3-compare.jsonl`.
|
||
Expect **0 divergences on every record where no tech completed**. On a record whose
|
||
`ours` shows a completion, the only permitted diffs are `turn_researched` / `order` on that
|
||
node and the child nodes `SetResearched` unlocked; the `rng` region must still match. Any
|
||
other diff is a real finding — report it, do not tune the formula.
|
||
To get more turns, keep pressing End Turn; each turn adds one record per player.
|
||
4. If the compare is clean: copy `shim.cfg.b3replace`, relaunch, load `ref-turn2.sav`, End Turn,
|
||
and check the determinism oracle — `(Autosave).sav` = `978041ac…`, `(Autosave EndTurn).sav`
|
||
= `bb4fd9ac…`. This is the strongest single result available: replace mode means our research
|
||
pass fed the game, and the save hash means the whole turn still landed byte for byte.
|
||
5. Restore the previous `shim.cfg` (`hooks=trace`) and leave the game at the main menu, as M1/M2
|
||
left it.
|
||
|
||
Not done, and worth saying: the compare above cannot distinguish the two unit divisors (see
|
||
"Float mapping"), and it will not exercise the Zuul path at all unless a Zuul player is in the
|
||
reference save — check `species` in the trace args, and if none is 5, run one more compare from
|
||
a Zuul save before calling the double roll verified by behaviour rather than by disassembly.
|