sots-engine/docs/B2.md

363 lines
22 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# B2 — `ServerPlayer::OnTechResearched` old-vs-new, with the player's fields as named regions
**Status (2026-09-08): code complete, cross-built and staged; every VM step still owed.**
VM140 was held by another lane for the whole of this milestone, so nothing was deployed,
the game was not stopped or relaunched, and `C:\SOTS` was not touched. The build lives in
its own tree (`/srv/re-lab/build/sots-engine-b2`) and its own dist
(`/srv/re-lab/shim/dist-b2`), not the shared ones. Everything below is offline work plus
what the *binary* says; the run list is at the end.
The point of the target: it is the single place where 36 hard-coded strategic effects, the
per-species xenotech flag words, the two design-option masks and the node-bore parameters
are all written. One call validates the whole `src/game/effects` layer at once.
## What was hooked
| hook name (record `hook`) | RVA | prototype |
|---|---|---|
| `Game::ServerPlayer::OnTechResearched` | 0x00491790 | `void (ServerPlayer*, TechDef* def, bool silent)` |
`__thiscall`, `[verified]`, vft slot 4, so it goes through `Hook<>` with
`CallConv::Thiscall`. Source: `src/shim/hooks/tech_effects.{h,cpp}` (the descriptor and the
delegation into the game's tech tree) over `src/shim/hooks/tech_effect_fields.{h,cpp}` (the
byte-level field adapter, host-tested — the split B1 made for `budget_inputs`). Installed
from `src/shim/main.cpp` after the B3 hook.
`TechTree::SetResearched` is the only caller. It fires **once per tech completion**, which
on a typical turn is zero times — see "Coverage" below, because that is the hard part of
this milestone.
### Region model
Fourteen regions, one per field group the callback writes, each with a struct describer, so
a divergence reads `side.modifiers.after.v.out_mod` rather than a byte offset in a blob:
| region | offset | what the describer names |
|---|---|---|
| `suit` | +0xb4, 8 B | `suit_tol`, `max_overharvest` |
| `res_mod` | +0xc0, 4 B | `res_mod` |
| `abilities` | +0xfc, 12 B | `reb_ai`, `ai_benefit`, `trade_allowed`, `commerce_raiding`, `view_intel`, `grav_synth`, `advanced_sensors`, `arcology`, + 4 unmodelled bytes |
| `modifiers` | +0x108, 0x54 B | `pddm`, `con_mod[3]`, `sav_mod[3]`, `out_mod`, `pop_mod`, `terra_mod`, `asteroid_mining`, `min_rate`, `per_gate_traffic`, `cast_range/efficiency/threshold`, + 5 unmodelled words |
| `design_masks` | +0x190, 8 B | `design_options_a`, `design_options_b` |
| `translation` | +0x1a4, 4 B | `translation_known` |
| `vaccines` | +0x288, 12 B | `has_vaccine`, `has_immunity`, `node_track_mask` |
| `research_target` | +0x294, 4 B | `research_target`, `research_target_set` |
| `node_bore_ptr` | +0x308, 4 B | the block pointer, as an opaque `ptr` |
| `inc_mod` | +0x30c, 4 B | `inc_mod` |
| `capture_designs` | +0x330, 4 B | `capture_designs` |
| `species_flags` | +0x348, 0x20 B | `species_flags[7]`, `count` |
| `roll` | +0x3b4, 8 B | `research_roll_pending`, `ai_rebellion` |
| `node_bore` | `*(+0x308)`, 12 B | `node_bore_params[3]` — declared **only when the block already exists** |
The unmodelled words are emitted as raw `u32` on purpose: if the original writes one of
them, that is a real finding and shows up as a divergence instead of going unseen. Pointer
words are emitted as `ptr` (ignored by the default policy) because they differ run to run.
`tests/shim_techfx` asserts that no two regions overlap and that each stays inside the
declared span.
### Comparison design
The regions do not cover the whole object, so there is no "scratch `ServerPlayer`" to hand
`ours`. Instead `rebind` passes `self` through unchanged and records the scratch buffer of
each region; `ours` then reads and writes **every field the callback can touch** through
those buffers, and reads only never-written fields (species, the tech-tree pointer) off the
live object, where "live" and "before" are the same value. `techfx::Views` is that
indirection, and it is what makes replace mode (no regions at all) the same code path.
Two things `ours` delegates to the game, in the same spirit as B3 delegating
`TechTree::Cost`:
* **the researched set** — 196 calls to `TechTree::HasResearched`, a verified, read-only
three-compare function. In compare mode the tree is the live, post-original one, which
is exactly the state the original's own tail saw (the completing node is already state 4).
* **the TechId of the definition** — 196 calls to `MasterTechTree::IsTech`, for the reason
in "Findings" below.
## What `ours` deliberately does not do
Compare mode must not touch live game state or consume randomness, so `ours` reproduces the
player-field writes and nothing else. Not reproduced, and each reported in `shim.log`
instead (one `techfx: ours ...` line per completion):
* the three completion events (`EVENT_RESEARCH_COMPLETE` / `_UNDERBUDGET` / `_TEMPERANCE`);
* the pending plague-cure roll — it draws from the strategic generator, so `ours` clears
the two words it guards and logs whether it would have fired;
* the writes to *other* objects: every owned system's AI flag (`CCC_AIVrus` / `CCC_AISlv`),
the arcology civilian-cap re-evaluation, the addiction cure behind the temperance sweep,
and the plague clear across systems and ships;
* `TechTree::SetResearched` for the Zuul boarding-pod grant (it would mutate the tree).
None of those touch a declared region, so **a compare stays clean through them** — but it
also does not check them. That is the honest boundary of this milestone: the compare proves
the player's fields, the log line proves what our layer decided about the rest.
One consequence worth stating: the Zuul grant makes the original call itself recursively.
The inner call rewrites the same tail fields with `IND_BrdPod` researched — and `IND_BrdPod`
is in neither mask table, has no chain branch and no xenotech bit, so the inner call's
writes equal the outer's. The compare is unaffected either way, since `ours` reads the
post-original tree.
## Findings — what the binary says that the notes did not
Read off the instruction stream first; the compare is confirmation, not discovery (B3's
lesson). Ten corrections, all of which changed code.
### 1. Every modifier is float32, and every constant is a widened float literal
The arithmetic is `fld dword [field]; fadd qword [k]; fstp dword [field]`: a float32 field,
combined with a *double* constant, stored back through float32. And every one of those
constants is a float32 value widened to double, not the decimal it looks like:
| written as | the double actually in the image | equals |
|---|---|---|
| 0.05 | 0x3FA99999A0000000 | `(double)0.05f` |
| 0.06 | 0x3FAEB851E0000000 | `(double)0.06f` |
| 0.10 | 0x3FB99999A0000000 | `(double)0.1f` |
| 0.15 | 0x3FC3333340000000 | `(double)0.15f` |
| 0.20 | 0x3FC99999A0000000 | `(double)0.2f` |
| 0.30 | 0x3FD3333340000000 | `(double)0.3f` |
| 0.45 | 0x3FDCCCCCC0000000 | `(double)0.45f` |
| 0.60 | 0x3FE3333340000000 | `(double)0.6f` |
| 0.90 | 0x3FECCCCCC0000000 | `(double)0.9f` |
| 0.35 | 0x3FD6666660000000 | `(double)0.35f` |
| 0.25, 0.5, 0.75, 1.0, 1.5 | exact | unaffected |
Ours held the state in `double` and used exact decimals. Rounding once at the end instead
of at every step drifts: `PlayerEconomyState` now holds `float`s, the apply layer rounds
through float32 at each step (`AddF` / `MulF`), and the table's literals carry an `f`
suffix so they widen to the same double bit pattern. `tests/game_effects` pins the six-tech
terraform chain against the stepwise result, and `tests/shim_techfx` pins the exact bit
patterns at the exact offsets.
Caveat, the same one B3 records: at 53-bit x87 precision control (the MSVC default) the
middle step rounds to double and the store rounds again, which is exactly what
`(float)((double)a + k)` gives on both the i386 target and the host. At 24-bit precision it
would not be. The hook records `fpu_cw` on every call, so the first trace settles it.
### 2. The AI-benefit bonus values are 0.5 — the table is dumped
`g_AITechValueTable` (6 × 12 bytes, `{int techId, float rebellionOdds, float bonus}`) was
"values not dumped" in the notes. It reads:
| tech | rebellion odds | bonus |
|---|---|---|
| CCC_AI, CCC_AIAdmin, CCC_AIFac | 0.1f | **0.5f** each |
| CCC_AIFRCON (10083) | 0.2f | 0 |
| CCC_AIVrus, CCC_AISlv | 0 | 0 |
So `AiBonusValues` now defaults to 0.5/0.5/0.5, and `AiRebellionOdds(id)` carries the odds
column. `ApplyAITechBonus` multiplies the value by `AIBn ? 1.0f : -1.0f` and narrows the
product to float32 before adding, which the apply layer now does too. `CCC_AIFRCON` is in
the table for its odds only — no branch reads its bonus.
### 3. `PrGtTrf` is an integer
The gate techs do `mov ecx,[cfg]; mov edx,[player+0x148]; cmp edx,[ecx]; jl ...` — a signed
**integer** max against an integer config word, not a float compare. Ours modelled the field
and both `PERGATETRAFFIC_*` tuning keys as `double`. Both are `int` now.
### 4. The node-bore parameters live behind a pointer, and the rule is "highest researched"
`ServerPlayer+0x308` is not three inline words: it is a pointer to a separately allocated
3-word block. The updater runs on **every** completion, allocates the block on first use and
`operator delete`s it when no bore drive is researched. The selector tests `DRV_RAD`
{95,60,5}, then `DRV_REND` {65,35,4}, then `DRV_RIP` {45,15,3}, first hit wins — which
confirms "highest wins" and raises it from medium to high confidence. There is **no species
gate**: the notes call these Zuul parameters, but nothing in the code checks the species (the
techs themselves are Zuul-only by data). `PlayerEconomyState` now carries
`hasNodeBoreParams` so "absent" is a state of its own, and the selection is re-derived in
the tail rather than applied by the completing tech.
### 5. A sticky "translation known" mask at +0x1a4 that was not in the notes
`RebuildSpeciesTechFlags` has a **second pass** the catalog does not mention: for each
species except the NPC race, if the species' flag word has bit 0 (level-1 translation), it
ORs that species' bit into `ServerPlayer+0x1a4`. Only ever ORed — never cleared, so it
survives the first pass clearing a flag. Modelled as `translationKnownMask`.
### 6. Three effects are tail checks, not effects of their tech
* **capture designs** — `if (!cdp) { if (HasResearched(SpyBm) && HasResearched(SlvgTech)) cdp = 1; }`
runs on *every* completion. Ours only fired when the completing tech was one of the two,
which is the same outcome in the normal path but wrong for a save where both are already
researched and the flag is not yet set.
* **the design-option masks** — recomputed wholesale from the researched set every time.
* **the node-bore parameters** — see above.
### 7. The completion callback has no already-researched guard
`ApplyTechEffect` returned early when the tech was already researched. The callback cannot
do that: `SetResearched` marks the node state 4 *before* invoking it, so the guard would
make every real call a no-op. `ApplyTechCompletion` is the unguarded form and is what the
hook calls; `ApplyTechEffect` keeps the guard for callers driving the layer themselves.
### 8. `TechDef`'s first word is **not** the TechId
`MasterTechTree::IsTech(def, id)` maps `id - 10000` into the master `TechDef*[196]` table and
compares pointers. `TechTree::HasResearched(id)` does the same map to reach a `TechDef`, and
*then* uses that def's first word as an index into the tree's node vector. So the first word
is a node index in a larger key space, and the TechId can only be got from the identity test.
`ours` therefore scans `IsTech` over 10000..10195 (196 three-compare calls, once per
completion) rather than trusting the word — and the record carries both, so a trace shows
the two key spaces side by side. The valid range is confirmed as exactly 10000..10195, with
197 as the "none" sentinel special-cased before the subtraction.
### 9. The 196-name table is dumped, so the key space is no longer reconstructed
`g_TechIdNames` is 196 × `{const char* name, int}`. Reading it out gives every name in
position order, which turns `tech_id.h` from a partial reconstruction (89 confirmed names,
~55 inferred, ~52 `Unresolved_NNN`) into the table itself. Sanity checks all pass: index 0
is `CCC_AdvSens` (the id the chain's first `push 0x2710` tests), index 30 `IND_HrdStrct`
(0x272e), indices 38..43 the six vaccines (0x2736..0x273b), index 66 `DRV_RAD` (0x2752).
Every one of the seven prefixes previously inferred for indices 100..107 was right, and so
were all 31 names the design-option mask tables would have implied.
Two guesses were **wrong**, and one of them was a bug:
* **The proliferate block has five entries, not six.** It is 158..162 (Human, Hiver,
Tarkas, Liir, Morrigi — no Zuul, like the other 5-entry families), and 163/164 are
`CCC_NDTRKHUM` / `CCC_NDTRKZUL`. Our block ran 158..163, so
`XenoTechId(Proliferate, Morrigi)` returned 10163 — the *Human node-track tech*. Fixed.
* Index 130, the Zuul slot of the level-3 translation block, is `XNC_DOMZUUL`, not a
translation by name. The position is unchanged, so nothing keyed on it moves.
The species order inside every block is Human, Hiver, Tarkas, Liir, [Zuul,] Morrigi, and
the families that omit the Zuul are Incorporate, Addict, Temperance, Accommodate **and
Proliferate** — all of which were medium/low confidence and are now high. The node-track
techs' ids are known for the first time (10163 / 10164), which is what lets the hook set
`NPTrk`: our layer resolves them by name, and before this the name did not resolve.
The two design-option mask tables (`{int techId, uint bit}`) were dumped in the same pass
and every id in them agrees with the name table, so `kDesignOptionIdsA/B` are in the
effects module and `ComputeDesignOptionMasks` has a by-id overload — which is what the hook
needs, since the game keys on ids.
### 10. About half the techs have no TechId at all, and the tail still runs for them
`resolve_tech_id` returns "none" for any definition outside the 196-name table — which is
roughly half the shipped `.tech` files. The callback has no branch for those, but it runs
its whole tail for them all the same: the bore selection, the flag words, the sticky
translation mask, the capture-designs pair test and the temperance sweep. `EffectsOf` is
empty for them, so an early return would have been easy to write and wrong.
`RunCompletionTail` is that tail on its own, and it is what `ours` calls in that case.
### Things the notes had right
The 36-branch chain is an if/else-if over ids 10000..10031 in exactly that order with no
gaps (`push 0x2710` … `push 0x272f`, then the tail's 0x2730..0x2733); at most one branch
runs. `IND_Waldo`/`IND_ExpSys` share a block, `BIO_EnvTail` falls through into
`BIO_TerBac`'s terraform add, and `IND_OrbDry` really does leave `ConMod[0]` alone. Every
magnitude in the catalog is confirmed. The plague-cure masks are bits 0..4 for ids
10038..10042 and 0x0f for 10043 — with one subtlety: the test is "this def **or a
descendant of it**", which our exact-id `PlagueCureMask` cannot express without the tree.
## Host tests
`ctest` 28/28 on the build box.
* `game_effects` — 399 checks (was 285). Added: the float32 state and the widened-literal
identities, the multiplicative tech on top of an additive one, the integer gate-traffic
max, node-bore highest-wins plus re-derivation on an unrelated completion, the
capture-designs tail check from a save where both are already researched, the sticky
translation mask (including that a rebuild does not take a bit back), that
`ApplyTechCompletion` applies twice while `ApplyTechEffect` does not, the recovered AI
bonus and odds, and that the by-name and by-id design-option builders agree bit for bit.
* `shim_techfx` (new) — 1473 checks. A synthetic `ServerPlayer` buffer driven through the
real adapter: the region table (no overlaps, all inside the span, every region named and
described), a full read/write round trip, **all 44 catalogued effects** byte-checked at
the offsets the original writes, hand-written float32 bit patterns for the additive,
multiplicative and exactly-representable cases, the integer gate max, the absent
node-bore block, the species flag words and their count, the research-target clear, and
that the describers emit field names a diff can point at.
Cross-build: `b2-9cd997d-dirty-20260908T0359Z`, exports 66 names identical to
`binkw32.dll`, staged in `/srv/re-lab/shim/dist-b2`. `tools/clean_room_check.sh` OK.
## Coverage — say it up front
**Only a tech that actually completes fires this hook.** The reference save is turn 2 of a
28-star game with 0 starting techs, so a given End Turn may produce no records at all. The
plan, in order of what it proves:
1. **Host-side, already done.** Every catalogued effect is applied to a synthetic player and
byte-checked (`shim_techfx`, 1473 checks). This is the exhaustive coverage; it does not
need the VM and it does not depend on which techs the save happens to finish.
2. **`shim.cfg.b2scout`** (staged alongside the three) turns on B3's `ProcessResearch` hook
in trace mode as well, so one End Turn shows both the completions that fired and every
node's `progress` against its `cost`. That says how many turns are needed before a
completion, instead of guessing.
3. **Trace, then compare, driven by whatever completes.** Press End Turn repeatedly; each
completion is one record. A run that yields zero records is a coverage failure to report,
not a pass — `tracecmp.py` exiting 0 on an empty trace proves nothing.
4. If the reference save cannot be driven to a completion in a reasonable number of turns,
the fallback is a fresh game with research set high (the game-setup research slider) and
a cheap first tech, saved as its own reference. That is a new save, so its determinism
oracle would have to be recorded before it is useful for step 4 of the run list.
## Gotchas
1. **`describe_args` runs before the original**, and therefore before `ours`. Anything
`ours` decides cannot ride on the record; it goes to `shim.log` instead.
2. **The hook can nest**, because the Zuul branch calls `SetResearched` → the callback. The
nested call happens *inside the original*, not inside the hook, so the per-call statics
are only ever written by the outer hook invocation. `ours` never recurses.
3. **The node-bore block cannot be created by `ours`.** In compare mode the region is
declared only when the block already exists, and the block pointer is described as an
opaque `ptr` so the original allocating it is not a divergence. In replace mode `ours`
calls the game's own updater to keep the allocation consistent — which means replace
mode does not exercise our node-bore selection; compare and the host tests do.
4. **Replace mode for this hook is deliberately incomplete.** It writes the player's fields
but raises no events, flags no systems, cures nothing and grants no boarding pods. On a
turn where nothing completes the hook never fires and the determinism oracle should still
hold — that is what step 4 below actually tests. On a turn where something completes, a
changed save hash is expected and is not a finding.
5. A compare record is small (14 regions, none over 0x54 bytes), so unlike B3 the configs
could leave other hooks on. They do not: the b2 configs switch everything else off so a
trace holds nothing but this callback.
6. `ServerPlayer+0x308` is a 32-bit pointer field. Host tests cannot write a real pointer
into a synthetic buffer at that offset (it would spill over `IncMod` at +0x30c), so they
set the view's block pointer directly.
## What remains (needs the VM)
The lane holding VM140 must be finished first; then, in this order:
1. Deploy `/srv/re-lab/shim/dist-b2` (build `b2-9cd997d-dirty-20260908T0359Z`): `scp` it to
`C:\SOTS\shimdist-b2\` and run `deploy.ps1 -Dist C:\SOTS\shimdist-b2` — **a separate
staging directory from the shared `C:\SOTS\shimdist`**, so no other lane's dist is
overwritten.
2. **Scout.** Copy `shim.cfg.b2scout` over `C:\SOTS\shim.cfg`, relaunch, load
`ref-turn2.sav`, press End Turn once, pull `C:\SOTS\shim.trace.jsonl`. Read off it:
* how many `Game::ServerPlayer::OnTechResearched` records there are (expect 0 or 1);
* from the `Game::TechTree::ProcessResearch` records, each player's funded node and its
`progress` vs `cost_rp`, i.e. how many more turns to a completion;
* `fpu_cw` (expect `0x027f`; `0x007f`/`0x003f` means 24-bit precision, which would change
the rounding shape in `AddF`/`MulF` and nothing else).
3. **Golden trace.** Copy `shim.cfg.b2trace`, relaunch, load `ref-turn2.sav`, and press End
Turn as many times as step 2 says are needed. Pull the trace →
`verify/traces/b2-techfx-golden.jsonl`; `tracecmp.py` must exit 0 with 0 invalid records
**and at least one record**. Check per record: `tech_id` is in 10000..10195 and
`tech_name` matches; `def_node_index` differs from `tech_id` (finding 8); the `side`
entries change only the fields that tech's branch should touch, plus the tail's
`design_masks` / `species_flags` / `translation` / `node_bore`.
4. **Compare.** Copy `shim.cfg.b2compare`, relaunch, load `ref-turn2.sav`, same number of
End Turns → `b2-techfx-compare.jsonl`. Expect **0 divergences on every record**. There is
no expected-divergence carve-out here: everything `ours` does not model is outside the
declared regions. Any diff is a real finding — report it, do not tune the table. Read the
matching `techfx: ours ...` lines out of `C:\SOTS\shim.log` and check them against the
completing tech (granted tech only for a Zuul `IND_CruisCon`, plague mask only for a
vaccine, and so on).
5. **Replace (weak check only).** Copy `shim.cfg.b2replace`, relaunch, load `ref-turn2.sav`,
press End Turn **once** (a turn on which nothing completes), and check the determinism
oracle — `(Autosave).sav` = `978041ac…`, `(Autosave EndTurn).sav` = `bb4fd9ac…`. That
proves the hook installs and perturbs nothing. Do **not** expect the oracle to hold on a
turn where a tech completes; see gotcha 4.
6. Restore the previous `shim.cfg` (`hooks=trace`) and leave the game at the main menu, as
M1/M2/B3 leave it.
Not done, and worth saying: no record of this hook has ever been captured, so unlike M2
there is not even a scouting trace to confirm the region model against a live object. The
first thing to check in step 2 is that the `before` snapshots look like plausible player
state (`out_mod`/`pop_mod`/`terra_mod` near 1, `con_mod`/`sav_mod` near 1,
`per_gate_traffic` a small integer) — if a field reads as garbage, the offset is wrong and
the run list stops there.