sots-engine/docs/B4.md

B4 — the colony turn and the fleet movement pass, old vs new

Status (2026-09-08): code complete, cross-built, staged; every VM step still owed. VM140 was held by another lane for the whole of this milestone, so nothing was deployed, the game was not stopped or relaunched, and C:\SOTS\binkw32.dll / C:\SOTS\shimdist were not touched. Everything below is offline work plus what the binary says; the run list is at the end. The build lives in its own tree (/srv/re-lab/build/sots-engine-b4) and its own dist (/srv/re-lab/shim/dist-b4), not the shared ones. Build b4-final-20260908T0500Z, exports 66 names identical to binkw32.dll. Host suite 30/30. tools/clean_room_check.sh OK.

Ghidra was available this round and was used at the end to write the verified prototypes back into the shared project (reva-server stopped for the run and restarted afterwards).

What was hooked

hook name (record hook)	RVA	prototype (all now [verified])
Game::ServerSystem::ProcessTurn	0x003598e0	void (ServerSystem*) — no stack arguments
Game::StrategyServer::MoveFleet	0x003d9ee0	bool (StrategyServer*, StarFleet*, float dt)
Game::StrategyServer::ProcessFleetMovement	0x003da9a0	void (StrategyServer*)

All three are __thiscall and go through Hook<> with CallConv::Thiscall. Sources: src/shim/hooks/colony_turn.{h,cpp} and src/shim/hooks/fleet_movement.{h,cpp}, installed from src/shim/main.cpp after the B1/B2/B3 hooks. The pure halves are src/shim/hooks/colony_inputs.{h,cpp} (lib shim_colony, ctest shim_colony_unit) and src/shim/hooks/movement_inputs.{h,cpp} (lib shim_movement, ctest shim_movement_unit) — the B1 split, so the mapping is exhaustively testable without the VM.

How the prototypes were verified

Every one was read off the instruction stream (objdump -d over the shipped exe) before it was hooked, because M0's lesson is that a wrong thiscall prototype crashes the game:

• ServerSystem::ProcessTurn ends in a plain ret and nothing in the body reads [ebp+8]. The existing decompile shows a second parameter void* stream; that is a Ghidra guess and it is wrong. Hooking it as a one-argument function would have corrupted the stack.
• MoveFleet ends in ret 8; [ebp+8] goes into ESI as the fleet and [ebp+0xc] is read with fld DWORD — a 4-byte float. Each of the five call sites pushes its dt with push ecx; fstp DWORD PTR [esp]. It returns AL, and the caller tests it.
• ProcessFleetMovement ends in a plain ret with mov esi,ecx and no stack reads.

The declared input boundary — say it out loud

Both colony and movement targets are mostly dispatchers. Being honest about that is what makes the compare mean anything.

ServerSystem::ProcessTurn. The function body writes: the unowned infrastructure decay, both pending bonus pools (through ApplyInfraBonus / ApplyPopBonus), ntdev, the long-stability accrual, TRes = 0, haltv[0..2] = false, the two per-player countdown sweeps, and the addiction morale events. Everything else is a callee. So the declared regions are exactly

infra, ibon, pbon, ntdev, tres, haltv, bats2, bats_mask, rcex, rcex_mask, rng

and nothing else about the system is declared — population, morale, resources, plague, slaves and rebellion state are simply not regions, so the harness never compares them. They are recorded in the inputs region for the trace instead. Two live values our side asks the game for rather than re-deriving: ServerSystem::IsStable (the stability verdict) and ServerSystem::MaxPop (the imperial capacity the accrual reads) — both read-only, the same delegation B3 makes to TechTree::Cost.

MoveFleet. Declared: pos, prev_pos, one region per ship's range, the generator, and the return value. Not declared: the departure hook, the route revalidation, every arrival handler, the waypoint list, the tanker top-up. A call that arrives is expected to differ in all of that, and none of it is compared.

ProcessFleetMovement. Declared: each player's gate-traffic word. That is the one thing our side can honestly reproduce, because the original computes it at the very end of the pass from the post-move fleet state — which is exactly the state ours reads. The pass schedule itself is recorded in the arguments (and sim::PlanFleetMovement predicts the call order so a trace can be checked against it) but is not compared, because reproducing it would mean running MoveFleet.

Neither MoveFleet nor ProcessTurn offers replace mode. Both say so in shim.log and fall back to the original: our side models a slice, and feeding that slice to the game would strand every arriving fleet or skip a colony's whole turn.

The RNG region, and why it has teeth here

A colony turn's RNG consumption was swept function by function to call depth one: ProcessPlague, the civilian growth pass and ProcessSlaves make no draws at all, and neither does ProcessTurn itself. ProcessRebellion is the only consumer, and its count is data-dependent — one RandChance per iteration of a 64-bit rebel counter, plus a short-circuiting per-species roll loop, plus one outcome roll.

So the generator is a declared region whose expected post-state on a system with no rebellion is bit-identical, and any movement of it names the system whose rebellion fired. ours seeds a mars::rng::MT19937 from the pre-call snapshot, consumes the draws its own pass makes (currently none) and writes the state back, exactly as B3 does. next is rebuilt against the live generator address so the describer's index arithmetic reads the same on both sides.

MoveFleet declares the generator too, because a type-5 waypoint draws from it: one word on a successful jump, two on a miss (the second seeds the random scatter direction).

Corrections found by reading the binary

Twenty-two, none of them tuned to make anything match. The ones that change behaviour:

Movement

1. The range grace margin is added, not subtracted. The notes said range = MinRange(fleet) − 0.05. The constant at 0x00a1d2c0 is 0x3d4ccccd — +0.05f, sign bit clear — and MinRange adds its float argument (fadd DWORD PTR [ebp+8]). The correct expression is range = float32(MinRange(fleet) + 0.05f): a grace fudge so a fleet exactly at its range limit can still reach the target. Our old code stopped fleets 0.1 short.
2. The out-of-range case zeroes the range, not the step. The original asks for the minimum range a second time with no margin and, when that is exactly zero, does fstp DWORD PTR [ebp+0x8] — the range slot. step is untouched, and step is later the divisor of the pass fraction, so zeroing it instead changes (or NaNs) the recursion.
3. move has no floor at zero. move = min(min(range, step), distance) in that order; a negative minimum range moves the fleet backwards. The floor at zero exists only on each ship's range in the fuel loop.
4. MinRange seeds its accumulator with FLT_MAX, so an empty fleet is unconstrained rather than stranded, and it skips no ship — a range-exempt tanker still clamps the fleet.
5. The probabilistic jump does not stop part-way along the vector. On a miss the fleet is placed at dest + randomUnitVector x v where v = float32(roll x CstE) — scattered around the destination by exactly v, at the cost of a second RNG draw. The jump succeeds iff !(v > CstT), so equality arrives, and the pass fraction is 1.0 either way, so a type-5 waypoint never recurses.
6. Only waypoint type 3 reports a partial pass fraction. IsNodeWaypoint is a 7-entry jump table that is true for 3 alone — and the node-line case of the movement switch is type 2, which it does not accept. Everything else reports a full pass unless the move was blocked, in which case the fraction is clamp01(move / step).
7. The recursion threshold is a widened float literal and the test is strict. The constant at 0x00a261f0 is an 8-byte double whose value is exactly (double)0.9999f = 0.9998999834060669; fraction == threshold does not recurse, and the recursive dt is float32((1 − fraction) x dt).
8. The stutter overlap rule is not a midpoint. When seg[i].end > seg[i+1].start the original sets both boundaries to float32(seg[i].end + 0.5 x (seg[i].end − seg[i+1].start)) — the mirror of the midpoint about seg[i].end, pushing the boundary forward past both chords by half the overlap. Verified down to the ModRM byte, because the FSUB/FADD operand order is the whole claim. Nothing is dropped and nothing is clipped back, so a chord swallowed by its predecessor comes out inverted (start > end) and the step loop skips it. We reproduce it, bug and all.
9. Smaller ones in the same pass: the chord parameters are clamped to [0, length] before the drop test (without which the intersect routine's ±FLT_MAX sentinels would poison the list); the drop threshold is fabs(start − end) <= 0.01f (a float32 literal, and inclusive); the sort is a real std::sort keyed on start alone, so ties are unordered.
10. The pass schedule is a pursuit model, not a "departing / in-transit / other" split. Every fleet whose current waypoint targets another fleet is classified by the relation between the two owners: relation 0 (no treaty) makes it a pursuer, anything else a follower. Prey move half a turn, then pursuers move half a turn and a pursuer that arrives retires itself and its prey from the rest of the schedule, then the surviving prey take their second half, then everything unscheduled takes a full turn (an uncaught pursuer gets a second half instead), then the followers take a full turn. A fleet that is only a follower's target is not prey and takes a normal full turn.
11. Gate traffic is the sum of a signed int16 at fleet+0xc0 over fleets whose front waypoint type is 4 or 5, indexed by the owner's own index word, and it is assigned to each player rather than accumulated. Two things worth recording: the original accumulates by player->index but writes back by the player's position in the server's vector, which agree only while players[j]->index == j; and the accumulator is a fixed 32 ints with no bounds check.
12. The field the notes called FPogn2 at fleet+0xec is FPdpos. By the FlightPlan layout FPogn2 is at +0xe0; +0xec is the destination position, which is what the pass writes.
13. STUTTER_MIN_SPEED == STUTTER_MAX_SPEED == 0.33 in the shipped data (radius 2), so the linear ramp collapses to a constant 0.33× inside any influence sphere and 1.0× outside. The ramp is still implemented because the constants are data-file tunable, but a run against shipped data cannot distinguish it from a binary in/out multiplier — worth knowing before reading a clean compare as evidence for the ramp.

Colony

14. The population growth curve has no pop / capacity term. The notes had g = clamp01((1 − clamp01(pop/cap))^EXP). The capacity is never passed into the growth chain at all; the base of the power is a suitability term: base = 1 − clamp01(min(|ideal − clamp(suit, 0, 20)|, SuitTol) / SuitTol), and the exponent is clamped into [0.01f, 1000] before a real pow(). Every modifier that follows is gated on a strict > 0 and stored back to a float32. The 50,000,000 cap is not here either — it lives in the apply.
15. The over-cap shrink runs only when the colony was already over the cap, is computed from the old population, and a colony that merely grows past the cap simply lands on it.
16. The second SYSTEMBONUS_MINTURNS gate is ntdev, not rbtn. Read off the two cmps at the head of AccrueSystemBonus. Our SystemBonusInputs field was named turnsSinceRebellion; it is turnsDeveloping.
17. Both bonus-apply helpers reset ntdev to zero on a colony that is not the owner's home system. That is a real feedback loop: a colony still absorbing a bonus never accumulates the developing turns the accrual gate wants, so it fails the gate on the same turn. ApplyInfraBonus also snaps Infra to exactly 1.0f when the pool covers the whole remainder, and an unowned system drops its whole population pool.
18. The output-rate normaliser pins the trade slider. It takes a "pinned channel" argument that every call site leaves null, which selects trade. Only trade is clamped into [0, 1]; the other three are summed in float32 (trade excluded) and rescaled to 1 − trade; the all-zero fallback is an equal split over three channels, not four. Our version rescaled all four symmetrically.
19. The engine's "round" is fistp/fild — round to nearest, ties to EVEN. Not round-half-away-from-zero. 302.5 becomes 302. And out[0], out[3] and the construction points go through the truncating helper, not the rounding one.
20. Every carrying capacity is rounded down to a multiple of ten by the shared helper, and Size x 1e8 is an exact 64-bit integer product (the 1e8 is the immediate 0x05f5e100); only the modifier chain is floating point. The arcology bonus is 0 for slaves.
21. The terraforming modifier is inside the point count, not only the yield — a better modifier needs proportionally fewer points, which is what keeps need and yield consistent — the sign is -1 only for suit > ideal strictly, and the apply clamps at the ideal from whichever side it approached. TerraformPointsNeeded is a ceil, not a truncation.
22. Smaller ones: the slave death rate folds the hazard term into the base before the output term (order matters when every step narrows to float32), an unowned system reports 1.0 rather than 0, the worst plague at the system contributes an additive rate term, and both SLAVES_MIN/MAX_DEATHS are disabled by any negative value with the result clamped into [0, slaves]. BuildQueue::ProcessTurn returns the leftover points by value, a money refusal skips the order rather than stopping the pass, and removal is a separate sweep that unlinks every order at or below zero.

Float discipline

Every constant was checked bit by bit, because B2 and B3 were both bitten here.

what	address	bits	value	kind
unowned infra decay	0x009e9170	3f947ae140000000	0.019999999552965164	widened 0.02f, not the decimal
range grace	0x00a1d2c0	3d4ccccd	+0.05000000074505806	float32 0.05f, positive
recursion threshold	0x00a261f0	3fefff2e40000000	0.9998999834060669	double whose value is (double)0.9999f
stutter chord drop	0x009e3e14	3c23d70a	0.009999999776482582	float32 0.01f
stutter overlap half	0x009e20a0	3fe0000000000000	0.5	true double
hazard band +0.1	0x00a1a438	3fb999999999999a	0.1	true double, not (double)0.1f
output-rate threshold	0x009e22c8	3f1a36e2e0000000	9.999999747378752e-05	widened 1e-4f
infra divisor	0x00a1f930	3f014d2f5dbb9cfa	3.3e-05	true double
infra chain	—	—	/500, x0.01, x1.65	0.01 and 1.65 are true doubles; the three steps are not folded
terraform yield	0x009e62b8	3ff3333340000000	1.2000000476837158	widened 1.2f
terraform need	0x00a1f928	3ffccccce0000000	1.8000000715255737	1.5 x (double)1.2f
growth exponent floor	0x009e3e14	3c23d70a	0.01f	shared with the chord drop
slave mod base / step	0x009e3030 / 0x009e20d8	3f4ccccd / 3fc99999a0000000	0.800000011920929 / 0.20000000298023224	float32 0.8f / widened 0.2f

numeric.h gained F32() (narrow-and-widen) and RoundHalfEven(); rng.h's IRandom gained NextUInt32() for the raw word the jump scatter takes.

What our side runs

• sots::sim::ProcessColonyTurn (new, game/sim/colony.{h,cpp}) — the dispatcher's own writes, in the original's order, plus ColonyCountdowns / AddictionPhaseOf and the corrected ApplyPopulationBonus / ApplyInfrastructureBonus / AccrueSystemBonus.
• sots::sim::StepFleet (the shim adapter) over ResolveMoveStep, AdvanceAlongDirection, ConsumeShipRange, PassFraction and RollProbabilisticJump.
• sots::sim::GateTrafficTotals and sots::sim::PlanFleetMovement.
• sots::sim::NodeLineStep and BuildStutterSegments are corrected and unit-tested but are not wired into the hook: the node-line step needs the node graph walked from the live server, which this milestone does not do. A node-line waypoint therefore records its type in the trace and is left to the original — a declared gap, not a silent one.

Host tests

ctest 30/30 (was 28). New coverage:

• game_sim_colony — 200 checks. Rewritten around the corrected growth curve (the suitability base, the [0, 20] clamp, the exponent clamp), the ties-to-even rounding, the pinned-trade normaliser, the terraform modifier inside the point count, the infra/suit apply clamps, the additive plague rate and the both-ends slave clamp.
• game_sim_movement — 144 checks. The +0.05 grace, the range-zeroing stranded case, the absence of a floor on move, the empty-fleet FLT_MAX, the scatter semantics of a failed jump and its two draws, the type-3-only pass fraction, the strict recursion threshold, the forward-pushed overlap boundary (including the inverted segment it produces), the pass schedule with and without a catch, and the gate-traffic sum.
• shim_colony_unit — 62 checks. The snapshot round trip, the input mapping, a reference-save shaped colony turn, the bonus/accrual interaction with the non-home ntdev reset, the addiction sweep across all four phases plus temperance, the unowned colony, and the countdown edges (index 15, the clamp, the skip-when-zero).
• shim_movement_unit — 56 checks. Straight steps, the exempt-tanker clamp, the arrival snap, the node-waypoint fraction, the gate teleport, both jump outcomes, a held fleet, the gate traffic and the schedule.

Gotchas

1. ServerSystem::ProcessTurn takes no arguments. The decompile in the RE handoff shows a second parameter. Trusting it would have pushed a garbage word and corrupted the stack — exactly M0's failure mode.
2. MoveFleet recurses into itself for a multi-waypoint leg. Hook<> handles the nesting (depth, call ids) and the inner call gets its own record, but the outer record's "after" snapshot includes everything the recursion did — read a record's depth before comparing two of them.
3. Per-call state lives in statics between regions() → rebind() → ours() (M1's concession). Safe here because the turn pass is single-threaded, but MoveFleet does nest — the compare path is still correct because the template runs regions/rebind/ours for one call before the original's recursion can start, but do not add state that has to survive the original's execution.
4. Region::name is a const char* held for the whole call, so the per-ship and per-player name strings are reserved once up front; a reallocation would dangle every name already pushed.
5. ProcessFleetMovement's ours re-reads the fleet list. The gate-traffic total is computed by the original after the passes, so comparing against a pre-call snapshot would diverge for the wrong reason.
6. A compare record for ProcessTurn is small (twelve regions), but the generator's 2496-byte block is hashed rather than inlined — keep trace.inline_max at 256.
7. MoveFleet in compare is the busiest of the three; on the reference save that is a handful of records, but on a large map it is every fleet times up to five passes times the recursion depth. The b4scout config exists for exactly that reason: it leaves MoveFleet off.

What remains (needs the VM)

The lane holding VM140 must be finished first; then, in this order:

1. Deploy /srv/re-lab/shim/dist-b4 (build b4-final-20260908T0500Z): scp it to C:\SOTS\shimdist-b4\ and run deploy.ps1 -Dist C:\SOTS\shimdist-b4 — a separate staging directory from the shared C:\SOTS\shimdist, so no other lane's dist is overwritten.
2. Scout pass. Copy shim.cfg.b4scout over C:\SOTS\shim.cfg, relaunch, load ref-turn2.sav, press End Turn once, pull C:\SOTS\shim.trace.jsonl → b4-scout.jsonl. tracecmp.py must exit 0 with 0 invalid records. MoveFleet is off in this config, so the file stays small. Read off it before going further:
   • fpu_cw on every record — expect 0x027f; 0x007f / 0x003f means 24-bit x87 precision and the float mapping needs the PC24 route (B3's open question).
   • per ServerSystem::ProcessTurn record: args.rng_left_in minus side.rng.after.left. Expect 0 on every system. A non-zero delta names a system whose ProcessRebellion fired, and that system's compare record is then expected to diverge on rng and only on rng.
   • how many systems report owned, stable, a non-zero ibon/pbon, a non-empty civilians list and a non-zero bats2/rcex. That is the coverage table for step 4.
   • per ProcessFleetMovement record: the fleet_state list — how many fleets exist, how many have waypoints, and whether any waypoint type is 4 or 5. If none is, gate traffic is exercised in its zero branch only and must be reported that way.
3. Trace pass. Copy shim.cfg.b4trace (adds MoveFleet), relaunch, load ref-turn2.sav, End Turn → b4-trace-golden.jsonl. Check the MoveFleet record sequence against sim::PlanFleetMovement's prediction: the fleet ids and dt values should appear in the pass order documented above. A mismatch is a finding about the schedule, not about the step.
4. Compare pass. Copy shim.cfg.b4compare, relaunch, load ref-turn2.sav, End Turn → b4-compare.jsonl. Expected:
   • ServerSystem::ProcessTurn — 0 divergences on every system, on all twelve regions. The reference save is a turn-2 two-empire game, so infra/ibon/pbon are probably exercised in their no-op branches and bats2/rcex in their all-zero branch; say which regions actually carried a value rather than implying the rest passed.
   • MoveFleet — 0 divergences on pos, prev_pos and every ship[i].range for a call that did not arrive. A call that arrived is expected to match on those too (the snap is a verbatim copy) but the RNG and the undeclared arrival state are the original's; a divergence on pos after an arrival is a real finding.
   • ProcessFleetMovement — 0 divergences on every gate_traffic[i]. Any other diff is a real finding: report it, do not tune the formula.
5. Restore the previous shim.cfg (hooks=trace) and leave the game at the main menu, as M1/M2/B1/B3 left it.

Not done, and worth saying:

• Replace mode is not offered for any of the three hooks, so there is no End-Turn oracle result for this milestone. That is a deliberate consequence of the input boundary, not an omission — the strongest available evidence here is the compare plus the RNG post-state.
• Sub-paths the reference save will not exercise, and which must therefore be reported as untested rather than passed: plague, rebellion (and with it every RNG draw in a colony turn), slaves, terraforming (the reference colony sits at its ideal), the addiction sweep at any phase, the unowned-system infrastructure decay, a non-home colony's ntdev reset, the probabilistic jump, the gate teleport, node-line travel, and a stranded fleet. A save with a plague, a rebelling colony, a Hiver gate network and a Zuul node bore would exercise most of them and is the natural next workload.
• The node-line step is modelled and unit-tested but not hooked (see "What our side runs"), and under shipped data the stutter ramp is a constant anyway (correction 13).
• ComputeOutputFromRates was read instruction by instruction and every correction is folded into game/sim/colony, but it is not hooked: it repairs damaged ships in orbit as a side effect (0x00751590 with its estimate flag clear), so a compare hook cannot run it on a scratch copy of the system without isolating the ships too. That is its own milestone.