sots-engine/docs/B4.md

346 lines
24 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# B4 — the colony turn and the fleet movement pass, old vs new
**Status (2026-09-08): code complete, cross-built, staged; every VM step still owed.**
VM140 was held by another lane for the whole of this milestone, so nothing was deployed, the
game was not stopped or relaunched, and `C:\SOTS\binkw32.dll` / `C:\SOTS\shimdist` were not
touched. Everything below is offline work plus what the *binary* says; the run list is at the
end. The build lives in its own tree (`/srv/re-lab/build/sots-engine-b4`) and its own dist
(`/srv/re-lab/shim/dist-b4`), not the shared ones. Build
`b4-final-20260908T0500Z`, exports 66 names identical to `binkw32.dll`. Host suite **30/30**. `tools/clean_room_check.sh` OK.
Ghidra was available this round and was used at the end to write the verified prototypes back
into the shared project (`reva-server` stopped for the run and restarted afterwards).
## What was hooked
| hook name (record `hook`) | RVA | prototype (all now `[verified]`) |
|---|---|---|
| `Game::ServerSystem::ProcessTurn` | 0x003598e0 | `void (ServerSystem*)` — **no stack arguments** |
| `Game::StrategyServer::MoveFleet` | 0x003d9ee0 | `bool (StrategyServer*, StarFleet*, float dt)` |
| `Game::StrategyServer::ProcessFleetMovement` | 0x003da9a0 | `void (StrategyServer*)` |
All three are `__thiscall` and go through `Hook<>` with `CallConv::Thiscall`. Sources:
`src/shim/hooks/colony_turn.{h,cpp}` and `src/shim/hooks/fleet_movement.{h,cpp}`, installed
from `src/shim/main.cpp` after the B1/B2/B3 hooks. The pure halves are
`src/shim/hooks/colony_inputs.{h,cpp}` (lib `shim_colony`, ctest `shim_colony_unit`) and
`src/shim/hooks/movement_inputs.{h,cpp}` (lib `shim_movement`, ctest `shim_movement_unit`) —
the B1 split, so the mapping is exhaustively testable without the VM.
### How the prototypes were verified
Every one was read off the instruction stream (`objdump -d` over the shipped exe) before it was
hooked, because M0's lesson is that a wrong `thiscall` prototype crashes the game:
* `ServerSystem::ProcessTurn` ends in a **plain `ret`** and nothing in the body reads
`[ebp+8]`. The existing decompile shows a second parameter `void* stream`; that is a Ghidra
guess and it is wrong. Hooking it as a one-argument function would have corrupted the stack.
* `MoveFleet` ends in `ret 8`; `[ebp+8]` goes into ESI as the fleet and `[ebp+0xc]` is read
with `fld DWORD` — a 4-byte float. Each of the five call sites pushes its `dt` with
`push ecx; fstp DWORD PTR [esp]`. It returns `AL`, and the caller tests it.
* `ProcessFleetMovement` ends in a plain `ret` with `mov esi,ecx` and no stack reads.
## The declared input boundary — say it out loud
Both colony and movement targets are mostly *dispatchers*. Being honest about that is what
makes the compare mean anything.
**`ServerSystem::ProcessTurn`.** The function body writes: the unowned infrastructure decay,
both pending bonus pools (through `ApplyInfraBonus` / `ApplyPopBonus`), `ntdev`, the
long-stability accrual, `TRes = 0`, `haltv[0..2] = false`, the two per-player countdown
sweeps, and the addiction morale events. Everything else is a callee. So the declared regions
are exactly
> `infra`, `ibon`, `pbon`, `ntdev`, `tres`, `haltv`, `bats2`, `bats_mask`, `rcex`,
> `rcex_mask`, `rng`
and **nothing else about the system is declared** — population, morale, resources, plague,
slaves and rebellion state are simply not regions, so the harness never compares them. They
are recorded in the `inputs` region for the trace instead. Two live values our side asks the
game for rather than re-deriving: `ServerSystem::IsStable` (the stability verdict) and
`ServerSystem::MaxPop` (the imperial capacity the accrual reads) — both read-only, the same
delegation B3 makes to `TechTree::Cost`.
**`MoveFleet`.** Declared: `pos`, `prev_pos`, one region per ship's `range`, the generator,
and the return value. Not declared: the departure hook, the route revalidation, every arrival
handler, the waypoint list, the tanker top-up. **A call that arrives is expected to differ in
all of that**, and none of it is compared.
**`ProcessFleetMovement`.** Declared: each player's gate-traffic word. That is the one thing
our side can honestly reproduce, because the original computes it at the very *end* of the
pass from the post-move fleet state — which is exactly the state `ours` reads. The pass
schedule itself is recorded in the arguments (and `sim::PlanFleetMovement` predicts the call
order so a trace can be checked against it) but is not compared, because reproducing it would
mean running `MoveFleet`.
Neither `MoveFleet` nor `ProcessTurn` offers **replace** mode. Both say so in `shim.log` and
fall back to the original: our side models a slice, and feeding that slice to the game would
strand every arriving fleet or skip a colony's whole turn.
### The RNG region, and why it has teeth here
A colony turn's RNG consumption was swept function by function to call depth one:
`ProcessPlague`, the civilian growth pass and `ProcessSlaves` make **no draws at all**, and
neither does `ProcessTurn` itself. **`ProcessRebellion` is the only consumer**, and its count
is data-dependent — one `RandChance` per iteration of a 64-bit rebel counter, plus a
short-circuiting per-species roll loop, plus one outcome roll.
So the generator is a declared region whose expected post-state on a system with no rebellion
is **bit-identical**, and any movement of it names the system whose rebellion fired. `ours`
seeds a `mars::rng::MT19937` from the pre-call snapshot, consumes the draws its own pass makes
(currently none) and writes the state back, exactly as B3 does. `next` is rebuilt against the
live generator address so the describer's index arithmetic reads the same on both sides.
`MoveFleet` declares the generator too, because a type-5 waypoint draws from it: **one word on
a successful jump, two on a miss** (the second seeds the random scatter direction).
## Corrections found by reading the binary
Twenty-two, none of them tuned to make anything match. The ones that change behaviour:
### Movement
1. **The range grace margin is added, not subtracted.** The notes said
`range = MinRange(fleet) − 0.05`. The constant at `0x00a1d2c0` is `0x3d4ccccd` — `+0.05f`,
sign bit clear — and `MinRange` *adds* its float argument (`fadd DWORD PTR [ebp+8]`). The
correct expression is `range = float32(MinRange(fleet) + 0.05f)`: a grace fudge so a fleet
exactly at its range limit can still reach the target. Our old code stopped fleets 0.1 short.
2. **The out-of-range case zeroes the *range*, not the step.** The original asks for the
minimum range a second time with no margin and, when that is exactly zero, does
`fstp DWORD PTR [ebp+0x8]` — the range slot. `step` is untouched, and `step` is later the
**divisor** of the pass fraction, so zeroing it instead changes (or NaNs) the recursion.
3. **`move` has no floor at zero.** `move = min(min(range, step), distance)` in that order; a
negative minimum range moves the fleet backwards. The floor at zero exists only on each
ship's range in the fuel loop.
4. **`MinRange` seeds its accumulator with `FLT_MAX`,** so an empty fleet is unconstrained
rather than stranded, and it skips no ship — a range-exempt tanker still clamps the fleet.
5. **The probabilistic jump does not stop part-way along the vector.** On a miss the fleet is
placed at `dest + randomUnitVector x v` where `v = float32(roll x CstE)` — *scattered around
the destination* by exactly `v`, at the cost of a second RNG draw. The jump succeeds iff
`!(v > CstT)`, so equality arrives, and the pass fraction is 1.0 either way, so a type-5
waypoint never recurses.
6. **Only waypoint type 3 reports a partial pass fraction.** `IsNodeWaypoint` is a 7-entry jump
table that is true for 3 alone — and the node-*line* case of the movement switch is type
**2**, which it does not accept. Everything else reports a full pass unless the move was
blocked, in which case the fraction is `clamp01(move / step)`.
7. **The recursion threshold is a widened float literal and the test is strict.** The constant
at `0x00a261f0` is an 8-byte double whose value is exactly `(double)0.9999f` =
0.9998999834060669; `fraction == threshold` does **not** recurse, and the recursive `dt` is
`float32((1 − fraction) x dt)`.
8. **The stutter overlap rule is not a midpoint.** When `seg[i].end > seg[i+1].start` the
original sets *both* boundaries to `float32(seg[i].end + 0.5 x (seg[i].end − seg[i+1].start))`
— the mirror of the midpoint about `seg[i].end`, pushing the boundary **forward past both
chords** by half the overlap. Verified down to the ModRM byte, because the FSUB/FADD operand
order is the whole claim. Nothing is dropped and nothing is clipped back, so a chord
swallowed by its predecessor comes out **inverted** (`start > end`) and the step loop skips
it. We reproduce it, bug and all.
9. Smaller ones in the same pass: the chord parameters are clamped to `[0, length]` before the
drop test (without which the intersect routine's `±FLT_MAX` sentinels would poison the
list); the drop threshold is `fabs(start − end) <= 0.01f` (a float32 literal, and inclusive);
the sort is a real `std::sort` keyed on `start` alone, so ties are unordered.
10. **The pass schedule is a pursuit model, not a "departing / in-transit / other" split.**
Every fleet whose current waypoint targets another *fleet* is classified by the relation
between the two owners: relation 0 (no treaty) makes it a pursuer, anything else a
follower. Prey move half a turn, then pursuers move half a turn and a pursuer that arrives
retires itself *and its prey* from the rest of the schedule, then the surviving prey take
their second half, then everything unscheduled takes a full turn (an uncaught pursuer gets
a second half instead), then the followers take a full turn. A fleet that is only a
*follower's* target is not prey and takes a normal full turn.
11. **Gate traffic** is the sum of a **signed int16** at `fleet+0xc0` over fleets whose *front*
waypoint type is 4 or 5, indexed by the owner's own index word, and it is **assigned** to
each player rather than accumulated. Two things worth recording: the original accumulates
by `player->index` but writes back by the player's *position* in the server's vector, which
agree only while `players[j]->index == j`; and the accumulator is a fixed 32 ints with no
bounds check.
12. **The field the notes called `FPogn2` at `fleet+0xec` is `FPdpos`.** By the FlightPlan
layout `FPogn2` is at `+0xe0`; `+0xec` is the destination position, which is what the pass
writes.
13. **`STUTTER_MIN_SPEED == STUTTER_MAX_SPEED == 0.33` in the shipped data** (radius 2), so the
linear ramp collapses to a constant 0.33× inside any influence sphere and 1.0× outside. The
ramp is still implemented because the constants are data-file tunable, but a run against
shipped data cannot distinguish it from a binary in/out multiplier — worth knowing before
reading a clean compare as evidence for the ramp.
### Colony
14. **The population growth curve has no `pop / capacity` term.** The notes had
`g = clamp01((1 − clamp01(pop/cap))^EXP)`. The capacity is never passed into the growth
chain at all; the base of the power is a **suitability** term:
`base = 1 − clamp01(min(|ideal − clamp(suit, 0, 20)|, SuitTol) / SuitTol)`, and the
exponent is clamped into `[0.01f, 1000]` before a real `pow()`. Every modifier that follows
is gated on a strict `> 0` and stored back to a float32. The 50,000,000 cap is not here
either — it lives in the apply.
15. **The over-cap shrink runs only when the colony was *already* over the cap**, is computed
from the *old* population, and a colony that merely grows past the cap simply lands on it.
16. **The second `SYSTEMBONUS_MINTURNS` gate is `ntdev`, not `rbtn`.** Read off the two `cmp`s
at the head of `AccrueSystemBonus`. Our `SystemBonusInputs` field was named
`turnsSinceRebellion`; it is `turnsDeveloping`.
17. **Both bonus-apply helpers reset `ntdev` to zero on a colony that is not the owner's home
system.** That is a real feedback loop: a colony still absorbing a bonus never accumulates
the developing turns the accrual gate wants, so it fails the gate on the same turn.
`ApplyInfraBonus` also snaps `Infra` to **exactly 1.0f** when the pool covers the whole
remainder, and an unowned system *drops* its whole population pool.
18. **The output-rate normaliser pins the trade slider.** It takes a "pinned channel" argument
that every call site leaves null, which selects trade. Only trade is clamped into `[0, 1]`;
the other three are summed in float32 (trade excluded) and rescaled to `1 − trade`; the
all-zero fallback is an equal split over **three** channels, not four. Our version rescaled
all four symmetrically.
19. **The engine's "round" is `fistp`/`fild` — round to nearest, ties to EVEN.** Not
round-half-away-from-zero. 302.5 becomes 302. And `out[0]`, `out[3]` and the construction
points go through the *truncating* helper, not the rounding one.
20. **Every carrying capacity is rounded down to a multiple of ten** by the shared helper, and
`Size x 1e8` is an exact 64-bit *integer* product (the 1e8 is the immediate `0x05f5e100`);
only the modifier chain is floating point. The arcology bonus is 0 for slaves.
21. **The terraforming modifier is inside the point *count*, not only the yield** — a better
modifier needs proportionally fewer points, which is what keeps need and yield consistent —
the sign is `-1` only for `suit > ideal` strictly, and the apply clamps at the ideal from
whichever side it approached. `TerraformPointsNeeded` is a `ceil`, not a truncation.
22. Smaller ones: the slave death rate folds the hazard term into the base *before* the output
term (order matters when every step narrows to float32), an unowned system reports 1.0
rather than 0, the worst plague at the system contributes an **additive** rate term, and
both `SLAVES_MIN/MAX_DEATHS` are disabled by *any* negative value with the result clamped
into `[0, slaves]`. `BuildQueue::ProcessTurn` returns the leftover points by value, a money
refusal **skips** the order rather than stopping the pass, and removal is a separate sweep
that unlinks every order at or below zero.
### Float discipline
Every constant was checked bit by bit, because B2 and B3 were both bitten here.
| what | address | bits | value | kind |
|---|---|---|---|---|
| unowned infra decay | `0x009e9170` | `3f947ae140000000` | 0.019999999552965164 | **widened `0.02f`**, not the decimal |
| range grace | `0x00a1d2c0` | `3d4ccccd` | +0.05000000074505806 | float32 `0.05f`, **positive** |
| recursion threshold | `0x00a261f0` | `3fefff2e40000000` | 0.9998999834060669 | double whose value is `(double)0.9999f` |
| stutter chord drop | `0x009e3e14` | `3c23d70a` | 0.009999999776482582 | float32 `0.01f` |
| stutter overlap half | `0x009e20a0` | `3fe0000000000000` | 0.5 | **true double** |
| hazard band `+0.1` | `0x00a1a438` | `3fb999999999999a` | 0.1 | **true double**, not `(double)0.1f` |
| output-rate threshold | `0x009e22c8` | `3f1a36e2e0000000` | 9.999999747378752e-05 | widened `1e-4f` |
| infra divisor | `0x00a1f930` | `3f014d2f5dbb9cfa` | 3.3e-05 | **true double** |
| infra chain | — | — | `/500`, `x0.01`, `x1.65` | 0.01 and 1.65 are true doubles; the three steps are not folded |
| terraform yield | `0x009e62b8` | `3ff3333340000000` | 1.2000000476837158 | **widened `1.2f`** |
| terraform need | `0x00a1f928` | `3ffccccce0000000` | 1.8000000715255737 | `1.5 x (double)1.2f` |
| growth exponent floor | `0x009e3e14` | `3c23d70a` | 0.01f | shared with the chord drop |
| slave mod base / step | `0x009e3030` / `0x009e20d8` | `3f4ccccd` / `3fc99999a0000000` | 0.800000011920929 / 0.20000000298023224 | float32 `0.8f` / widened `0.2f` |
`numeric.h` gained `F32()` (narrow-and-widen) and `RoundHalfEven()`; `rng.h`'s `IRandom` gained
`NextUInt32()` for the raw word the jump scatter takes.
## What our side runs
* `sots::sim::ProcessColonyTurn` (new, `game/sim/colony.{h,cpp}`) — the dispatcher's own writes,
in the original's order, plus `ColonyCountdowns` / `AddictionPhaseOf` and the corrected
`ApplyPopulationBonus` / `ApplyInfrastructureBonus` / `AccrueSystemBonus`.
* `sots::sim::StepFleet` (the shim adapter) over `ResolveMoveStep`, `AdvanceAlongDirection`,
`ConsumeShipRange`, `PassFraction` and `RollProbabilisticJump`.
* `sots::sim::GateTrafficTotals` and `sots::sim::PlanFleetMovement`.
* `sots::sim::NodeLineStep` and `BuildStutterSegments` are corrected and unit-tested but are
**not** wired into the hook: the node-line step needs the node graph walked from the live
server, which this milestone does not do. A node-line waypoint therefore records its type in
the trace and is left to the original — a declared gap, not a silent one.
## Host tests
`ctest` **30/30** (was 28). New coverage:
* `game_sim_colony` — 200 checks. Rewritten around the corrected growth curve (the suitability
base, the `[0, 20]` clamp, the exponent clamp), the ties-to-even rounding, the pinned-trade
normaliser, the terraform modifier inside the point count, the infra/suit apply clamps, the
additive plague rate and the both-ends slave clamp.
* `game_sim_movement` — 144 checks. The `+0.05` grace, the range-zeroing stranded case, the
absence of a floor on `move`, the empty-fleet `FLT_MAX`, the scatter semantics of a failed
jump and its two draws, the type-3-only pass fraction, the strict recursion threshold, the
forward-pushed overlap boundary (including the inverted segment it produces), the pass
schedule with and without a catch, and the gate-traffic sum.
* `shim_colony_unit` — 62 checks. The snapshot round trip, the input mapping, a reference-save
shaped colony turn, the bonus/accrual interaction with the non-home `ntdev` reset, the
addiction sweep across all four phases plus temperance, the unowned colony, and the
countdown edges (index 15, the clamp, the skip-when-zero).
* `shim_movement_unit` — 56 checks. Straight steps, the exempt-tanker clamp, the arrival snap,
the node-waypoint fraction, the gate teleport, both jump outcomes, a held fleet, the gate
traffic and the schedule.
## Gotchas
1. **`ServerSystem::ProcessTurn` takes no arguments.** The decompile in the RE handoff shows a
second parameter. Trusting it would have pushed a garbage word and corrupted the stack —
exactly M0's failure mode.
2. **`MoveFleet` recurses into itself** for a multi-waypoint leg. `Hook<>` handles the nesting
(depth, call ids) and the inner call gets its own record, but the outer record's "after"
snapshot includes everything the recursion did — read a record's `depth` before comparing
two of them.
3. **Per-call state lives in statics** between `regions()` → `rebind()` → `ours()` (M1's
concession). Safe here because the turn pass is single-threaded, but `MoveFleet` *does* nest
— the compare path is still correct because the template runs `regions`/`rebind`/`ours` for
one call before the original's recursion can start, but do not add state that has to survive
the original's execution.
4. **`Region::name` is a `const char*` held for the whole call**, so the per-ship and per-player
name strings are `reserve`d once up front; a reallocation would dangle every name already
pushed.
5. **`ProcessFleetMovement`'s `ours` re-reads the fleet list.** The gate-traffic total is
computed by the original *after* the passes, so comparing against a pre-call snapshot would
diverge for the wrong reason.
6. A compare record for `ProcessTurn` is small (twelve regions), but the generator's 2496-byte
block is hashed rather than inlined — keep `trace.inline_max` at 256.
7. `MoveFleet` in `compare` is the busiest of the three; on the reference save that is a handful
of records, but on a large map it is every fleet times up to five passes times the recursion
depth. The `b4scout` config exists for exactly that reason: it leaves `MoveFleet` off.
## What remains (needs the VM)
The lane holding VM140 must be finished first; then, in this order:
1. Deploy `/srv/re-lab/shim/dist-b4` (build `b4-final-20260908T0500Z`): `scp` it to
`C:\SOTS\shimdist-b4\` and run `deploy.ps1 -Dist C:\SOTS\shimdist-b4` — **a separate staging
directory from the shared `C:\SOTS\shimdist`**, so no other lane's dist is overwritten.
2. **Scout pass.** Copy `shim.cfg.b4scout` over `C:\SOTS\shim.cfg`, relaunch, load
`ref-turn2.sav`, press End Turn once, pull `C:\SOTS\shim.trace.jsonl` →
`b4-scout.jsonl`. `tracecmp.py` must exit 0 with 0 invalid records. `MoveFleet` is off in
this config, so the file stays small. **Read off it before going further:**
* `fpu_cw` on every record — expect `0x027f`; `0x007f` / `0x003f` means 24-bit x87
precision and the float mapping needs the PC24 route (B3's open question).
* per `ServerSystem::ProcessTurn` record: `args.rng_left_in` minus `side.rng.after.left`.
**Expect 0 on every system.** A non-zero delta names a system whose `ProcessRebellion`
fired, and that system's compare record is then expected to diverge on `rng` and only on
`rng`.
* how many systems report `owned`, `stable`, a non-zero `ibon`/`pbon`, a non-empty
`civilians` list and a non-zero `bats2`/`rcex`. That is the coverage table for step 4.
* per `ProcessFleetMovement` record: the `fleet_state` list — how many fleets exist, how many
have waypoints, and whether any waypoint type is 4 or 5. If none is, gate traffic is
exercised in its zero branch only and must be reported that way.
3. **Trace pass.** Copy `shim.cfg.b4trace` (adds `MoveFleet`), relaunch, load `ref-turn2.sav`,
End Turn → `b4-trace-golden.jsonl`. Check the `MoveFleet` record sequence against
`sim::PlanFleetMovement`'s prediction: the fleet ids and `dt` values should appear in the
pass order documented above. A mismatch is a finding about the schedule, not about the step.
4. **Compare pass.** Copy `shim.cfg.b4compare`, relaunch, load `ref-turn2.sav`, End Turn →
`b4-compare.jsonl`. Expected:
* `ServerSystem::ProcessTurn` — **0 divergences on every system**, on all twelve regions.
The reference save is a turn-2 two-empire game, so `infra`/`ibon`/`pbon` are probably
exercised in their no-op branches and `bats2`/`rcex` in their all-zero branch; say which
regions actually carried a value rather than implying the rest passed.
* `MoveFleet` — 0 divergences on `pos`, `prev_pos` and every `ship[i].range` for a call that
did **not** arrive. A call that arrived is expected to match on those too (the snap is a
verbatim copy) but the RNG and the undeclared arrival state are the original's; a
divergence on `pos` after an arrival is a real finding.
* `ProcessFleetMovement` — 0 divergences on every `gate_traffic[i]`.
Any other diff is a real finding: report it, do not tune the formula.
5. Restore the previous `shim.cfg` (`hooks=trace`) and leave the game at the main menu, as
M1/M2/B1/B3 left it.
**Not done, and worth saying:**
* **Replace mode is not offered** for any of the three hooks, so there is no End-Turn oracle
result for this milestone. That is a deliberate consequence of the input boundary, not an
omission — the strongest available evidence here is the compare plus the RNG post-state.
* **Sub-paths the reference save will not exercise**, and which must therefore be reported as
untested rather than passed: plague, rebellion (and with it every RNG draw in a colony turn),
slaves, terraforming (the reference colony sits at its ideal), the addiction sweep at any
phase, the unowned-system infrastructure decay, a non-home colony's `ntdev` reset, the
probabilistic jump, the gate teleport, node-line travel, and a stranded fleet. A save with a
plague, a rebelling colony, a Hiver gate network and a Zuul node bore would exercise most of
them and is the natural next workload.
* The **node-line step** is modelled and unit-tested but not hooked (see "What our side runs"),
and under shipped data the stutter ramp is a constant anyway (correction 13).
* `ComputeOutputFromRates` was read instruction by instruction and every correction is folded
into `game/sim/colony`, but it is **not hooked**: it repairs damaged ships in orbit as a side
effect (`0x00751590` with its estimate flag clear), so a compare hook cannot run it on a
scratch copy of the system without isolating the ships too. That is its own milestone.