`sots_turn` loads a save through the engine's own reader, walks the published
phase order of all three turn drivers, runs what we hold, prints what we do
not, and writes the result back through the engine's own writer.
The phase catalog carries all 32 + 12 + 37 phases whether or not they are
implemented, so an unimplemented phase is a named no-op that appears in the run
log rather than a silent absence. 14 of the 44 turn-driver phases are modelled,
7 commit anything, 2 of the 37 tail phases are modelled.
Modelled but NOT committed is a first-class state. A phase whose formula we hold
and whose inputs we do not is evaluated, reported, and left unwritten unless
--commit-blocked is passed. That distinction was earned: committing phase 31's
player-status restore regressed two leaves that had agreed with the oracle
before the turn, because the phase writes 1 and the file carries 4.
Measured against the game's own post-turn saves, leaves localised by
state_checksum.py with coverage proved by re-serialisation:
turn1-state -> turn2-state 209 -> 204 diverging, closed 5, regressed 0
turn2-state -> turn3-state 108 -> 103 diverging, closed 5, regressed 0
Two tests: app_catalog (the tables stay complete and nothing claims to be
verified against a live game) and app_turn (11 saves driven; an untouched load
re-serialises byte-identically, a turn leaves the file re-readable, and no
blocked or stub phase writes anything). Skips cleanly without SOTS_SAVES_DIR.
ctest 38/38, clean-room OK. src/shim untouched. docs/S-standalone.md has the
full gap list.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ARBgSooAfokKUy6wKUKEyZ
A per-turn RNG budget is only as good as the entry-point table, and ours had
three of the seven. Adds the two that are modellable and documents the rest.
float_range(lo, hi) exactly one word. Narrows TWICE -- the scaled product
is stored to a 4-byte float before lo is added, and the
sum is stored again. Evaluating in double and narrowing
once disagrees on a measurable fraction of words, and the
test asserts the two models are distinguishable so the
shortcut cannot creep back.
int_range_bell(lo, hi) AT LEAST TWO words. Triangular, not uniform: the span is
split into h/2 and h - h/2 (truncating toward zero) and
each half drawn inclusively, first half first. The bounds
reach the draw as unsigned, so an inverted range yields a
huge first bound rather than an empty one; reproduced, not
corrected.
Documented but deliberately not modelled: a truncated-normal integer range built
on rejection sampling around a Box-Muller pair. It costs TWO WORDS PER ATTEMPT
and the attempt count is unbounded, and predicting its stream position needs log,
sqrt and cos to agree bit for bit with the original CRT. Nothing in the strategic
turn reaches it. It is recorded so a ledger that meets it does not score its two
words as one draw.
Also recorded in docs/mars-rng.md, because each is a way a word budget goes wrong:
* two calling conventions for one generator -- three entry points take the state
block (the object plus four bytes) and four take the object itself, and one
caller uses both within forty bytes of itself;
* two different divisors in the same image, 1/(2^32 - 1) for the unit draw and
2^-32 (with a +0.5 offset on the word) for the normal path;
* the unit draw is inlined at twenty-eight sites across eleven functions, so any
budget assembled by counting calls is a LOWER BOUND. Exactly one of those
eleven is reachable from the strategic turn driver.
Host ctest 36/36 and tools/clean_room_check.sh run as separate commands, both
clean. No src/shim change, so no cross-build is implicated.
The last opaque block of the save format. Reconciles lane W's provable negative
against lane O's issued-order saves: the two disagreed because three different
things were being compared as one.
Read out of Game::TurnCommands::Write (0x00842540, 764 bytes, no loops of its
own). Two halves:
* a PROLOGUE of six flag-gated groups. Each is a WriteBool on a member,
followed only when set by that command's payload. Write order is NOT offset
order -- each gate sits after its payload in the struct -- so the class is
one of the 89 whose offset-sorted layout view cannot be aligned to the wire.
That, not a branch, is why the sorted view showed an i32 where the save has
a bool at item 4.
* TWENTY-SEVEN std::list<T> members (0x70..0x1a8, stride 0xc, allocator-last),
each written by its own helper as WriteInt(size) then size element records.
All 27 are always written, so an empty list still costs one zero int.
So the recovered 44 items are 17 member writes plus ONE ITEM PER CONTAINER CALL
SITE: the linear recovery keeps the call site, guesses its kind from an element
field it could resolve, and drops the count word. 44 - 17 = 27 = the number of
lists. The "27 trailing ints with only 22 i32 slots" objection is that same 27
seen from both sides -- a kind mismatch in the table, not a structural
impossibility. And 8 prologue items + 27 zero counts = the 35-item block every
no-orders save carries bit-identically.
Item arithmetic closes to the unit on all five distinct workloads (35/38/41/61/
123). One correction to the provisional layout: the fleet-move element ends in a
COUNTED route vector {fleetId, nHops, nHops x systemId}, not a fixed quadruple --
with a four-item element the writer would need 26 lists on one save and 28 on
another, and it has 27.
Conformance: the generic check cannot be used here. Every tag is "." so the LCS
degenerates to a strict positional compare in which any primitive disagreement is
fatal, and the table's tail describes elements where the wire has counts. A
dedicated check states what is checkable instead: the 17-item prologue item for
item (17/17, and SchemaProbe takes every branch, so this is real evidence that
the conditional structure read from the instruction stream is the one the
recovery flattened) and the tail count (27 lists vs 27 table items). The tail is
reported as wire-only, never claimed as matched.
86 shapes / 838 items -> 87 / 856, still 0 MISMATCH.
Twenty-two of the 27 lists are HYPOTHESES: the scalar sequence comes straight off
the helper, but no save exercises them. Nested element bodies not otherwise
modelled here (ShipDesignDef, FleetLayout, WeaponGroups, DefenceLayout,
RaidTargets) are carried as opaque Nodes rather than guessed at, so a wrong body
cannot desynchronise a reader. select() matches the exact ".TurnCommands_v5"
suffix, so a future _v6 falls back to the carried Node.
SECOND DEFECT, present at main and unrelated to this block: zuul-turn23-fleet23
is the first save with a non-empty NVs list, and its element's leading id was
typed positionally as "." where the real tag is PID. The typed round trip on that
save differed at 0x89c14. Rule 6 exactly -- a path no save exercised was a
hypothesis flying as a fact. Fixed, and independently corroborated: the recovered
table for Game::ServerSystem names that item PID, and the Sys row moves from
102 matched / 1 wire-only / 3 shape-only to 103 / 0 / 2.
Coverage (CoverageArchive typed-vs-carried, not round-trip success): all eleven
saves 99.7-99.9% -> 100.0%; opaque items 37/43/63/132 -> 2 everywhere, and those
two are the deliberately-carried MT19937 block. Nothing else in any save we hold
is untyped. Ratchet 99.8 -> 99.99. Round trip byte-identical on all eleven
(newly so on zuul-turn23-fleet23).
test_save now asserts, on every real save, that each TurnCommands_v5 block is
consumed by the prologue plus the 27 lists with nothing left over -- the
item-granular statement a wrong list count or element width breaks first.
clean-room OK; host ctest 36/36; test_save 11 saves 0 failures. src/shim/ not
touched, so no cross-build was needed.
Descriptor + pure adapter + host tests for the per-player turn driver. Not
deployed; the WIN32 half is unbuilt here (no cross-compiler on this host).
The declared boundary is narrower than the function on purpose. Phases 2, 3
and 6 -- the savings apply, the aid records and the research refund -- are pure
functions of ComputeBudget's 22 slots and ProcessResearch's overBudget, and
both live in the original's own stack frame. Reaching them would mean calling
ComputeBudget ourselves (it repairs ships in orbit, audit #6), reading the
nested B1/B3 hooks (audit #5, the self-fulfilling compare), or inferring them
from the Sav delta. So they are guarded, not checked, and the three formulas
are written and unit-tested but not wired into the verdict.
Declared: the phase-7 clear, the RebAI decay, the descending timed-bonus
sweep, plus roll_flags and rng as observations ours never writes. Guards over
the whole ServerPlayer and the TechTree header.
docs/T-turn-driver.md states, before any run: which regions must not diverge,
which checks are weak by construction on the reference save, what falsifies
the ResearchRollPending reading, and the save that would finally fire the
branch nobody has seen.
host ctest 36/36 (was 35/35); clean_room_check OK.
Game::StrategyAIAgent::Streamable and the ten shapes under it. The whole
writer is unconditional -- the branch the decompiler shows around lnat is an
inlined vector destructor whose operator delete is marked noreturn, and both
paths converge -- so the recovered sequence and a single record are the same
sequence, and all 36 items match with 0 wire-only and 0 shape-only.
CD blocks are now selected by the CDT id at the same ordinal, in both
directions; the one .TurnCommands_v5 block per save still falls to a Node.
Also: Sim's Attrib was not an empty frame, it was an AttribMap holding a count
of 0, and typing it closes those two items too. And StreamableEnum<T> writes a
frame containing one int, not a bare int, so SysMem/mts/nalat are arrays of
one-int frames -- byte-neutral, since all three have count 0 in every save,
but the previous typing was wrong.
Conformance 74 shapes/769 items -> 86/838, still 0 MISMATCH. Round trip
byte-identical on all four saves; ratchet 97.5 -> 99.8. Every container that
is empty in all four saves is named as such in the notes; the new unit test
populates each one, since nothing else exercises them.
SvSctOb is a StreamableHelper<SVScriptObject> -- a polymorphic pointer holding a
Game::SVSOSots, which writes two variant lists each dispatched by the key item
before it (xscn -> xsc, EncID -> EncObj). Neither map is on the wire; both were
read out of the game's factories (see the notes repo). The shapes apply the key
in both directions, so a body goes back out as whatever it came in as, and an
unmodelled key still round-trips as a Node. 18 new shapes: SVSOSots, the four
scenario bodies (traps / crowdefs / indsys / gmtrigger -- indsys really does
serialize nothing, its Read and Write are both the shared `ret 4` stub) and the
eight encounter bodies the saves exercise. The four factory ids no save carries
(7 SystemKiller, 8 PuppetMaster, 14 Locust, 21 Ortgay) are deliberately NOT
typed: their serializers are recovered but nothing could check a shape for them.
DOpts and SVSOVonNeumann::trev are VectorHelper<Mars::String>, so read_elem /
write_elem / SchemaBuilder::carr grew the std::string branch lane G listed as
missing. spies2 is VectorHelper<int>: the TYPE is certain from the helper's own
decorated name, but the count is 0 in all 28 systems of all four saves, so no
element value has ever been observed -- the shape is a hypothesis about
behaviour even though it is a fact about type. Same for SysMem and mts.
Conformance 56 shapes / 657 items -> 74 / 769, still 0 MISMATCH, and every new
binding is 0 wire-only and 0 shape-only. Coverage 97.1/97.2/97.2/97.6 ->
98.0/98.0/98.0/98.4 with the byte-identical round trip preserved; ratchet
95.0 -> 97.5. CD is now the only remaining region of size, and it stays opaque:
the recovered 44-item Game::TurnCommands sequence cannot be aligned to the
save's 35 items even as a subsequence (item 4 is 8 bytes, so a bool where the
recovery says i32; and the 27 trailing ints have only 22 i32 slots to come
from), which proves the no-orders diagnosis rather than assuming it.
Two unit tests added that need no saves: the string-array element branch
(including the empty string, which is four zero bytes and so looks like int 0)
and the SvSctOb variant dispatch round trip.
ctest 34/34, clean_room_check OK, test_save skips cleanly with SOTS_SAVES_DIR
unset. sots_stream_schema.h unchanged: streams.py and gen_stream_schema.py were
re-run and the output is byte-identical apart from the provenance line.
The serializer recovery reaches this repo as a generated wire schema
(include/generated/sots_stream_schema.h, 386 classes / 2042 items): for each
class, the ordered sequence of items its Write puts on the stream. Facts only —
no field offsets, no sizeof, no strides. This engine reads and writes the on-disk
format; it does not inherit the original's memory layout.
The table is a specification, not a program: the recovery is a linear pass over
Write, so it cannot see Write's branches (StarShip's BQ2 is gated on hbq but
listed unconditionally) and it flattens container loops. A codec driven off it
would desynchronise. The hand-written io() shapes stay the codec; SchemaProbe
(probe.h) walks them with every branch taken and test_wire_schema LCS-aligns
that against the table — 56 shapes bound, 657 items matched, 0 mismatches.
Four defects the check found, all invisible to a round-trip test:
- SystemParams field 1 is a string, not an int. It is the empty string in
every save, and an empty string is four zero bytes — byte-identical to the
int 0, so it round-tripped by luck. A named planet would have desynced.
- ObservedTech/ObservedWeapon odet is a bool, not an int. Byte-safe only
because a 4-char tag makes a bool item and an int item both 12 bytes.
- SpeciesRatios nv and ShipRecords srbd are counts, not fields.
CoverageArchive separates items a field names from items a Node merely carries,
because a byte-identical round trip is not a coverage claim. Typed coverage of a
real save goes 37.9% -> 97.1% (97.2/97.2/97.6 on the others) with the round trip
still byte-identical, by typing TechTree (both NumTechs sections), Events,
ShipRecs, sprjs, civr, comms, spy2, spymgr, aid, Ojvs, AIEnf, FNG, trdmgr and
the Des section/gun-bank tree. Ratchet at 95%.
trdmgr resolves a recorded trap: ServerTradeManager's Read/Write really are the
inherited no-op, but the call is virtual and ServerTradeManagerImpl has the real
serializer. Same shape resolves IServerSpyManager -> ServerSpyManager.
ctest 34/34, clean_room_check OK, test_save skips cleanly with SOTS_SAVES_DIR unset.
Pure module game/sim/techgraph: PrereqsMet (AND of ORs, empty group fails),
SetResearched (stamps, child-cost sweep, sticky turnAvailable, zero-cost
recursion) and the newly-available collector, all read out of 0x00581e10,
0x0057d8e0 and 0x00587cc3.
Wired into the B3 hook in compare mode only, over the scratch node copies:
four more node write-backs, the EVENT_TECHS_UNLOCKED list (still an input,
still nullptr when it could not be computed), the de-duplicating observed-tech
append and the one RNG word RollResearchEvent draws.
docs/U-unlock.md section 4 is the prediction, written before the build was staged.
TechTree::ProcessResearch's events region now compares a modelled value instead of
reporting a known defect. ours posts the pass's events into its own
sots::events::EventStorage, seeded from a scan of the owner's list taken BEFORE the
original runs, and writes only the counts into the region's scratch copy. The game's
PostEvent is never called and no live byte moves; replace mode still posts nothing,
because a bumped EvNxID with no record behind it would corrupt the oracle's save.
- game/events: PostResearchPassEvents (the decision half, pure) + KeylessEventText
- shim/hooks/event_inputs (new lib shim_events, host-tested): the live<->model adapter,
carrying game pointers as explicit uint32 so a 64-bit host build cannot alias them
- research hook: the wiring, a new observed_techs region for ServerPlayer+0x274, and
turn / events_next_id_in / events_dedup_risk in the args so the count model's own
assumption is measured rather than assumed
- EVENT_TECHS_UNLOCKED is NOT posted: its trigger is pinned but needs SetResearched's
unlock cascade, which ours does not run. The driver takes the unlock list as an input
and is handed 'no list', so a missing input cannot look like a modelled negative.
Predicted residual: next_id short by exactly 1 on a completion call.
ctest 33/33 (shim_events_unit is new), clean_room_check OK. The shim TU is
syntax-checked only: no MinGW cross toolchain on this box.
See docs/P-events-wiring.md for the exact prediction for the next VM run.
The behavioural compare found 8 of 45 StrategyServer::MoveFleet calls diverging by
one ULP on a position component. Read off the instruction stream, the cause is that
the engine's vector normalise narrows to float32 four separate times and we kept
everything in double:
delta.c = f32(dest.c - pos.c) stored back to a float32 slot before normalising
sumsq = f32(x*x + y*y + z*z) products/adds in 53-bit regs, only the SUM stored
len = f32(sqrt(sumsq))
inv = f32(1.0 / len) a reciprocal, MULTIPLIED through, not three divides
dir.c = f32(delta.c * inv)
and the same call returns the leg distance, so it is never recomputed in a wider
precision either. The position tail was already right, which is why the error was a
constant absolute ~1.2e-7 (half an ULP of the inputs) rather than a formula error.
Adds NormalizeVec3 / StraightLeg / StraightLegDistance / AdvanceAlongUnitDirection
and rebuilds AdvanceAlongDirection on them; the movement hook now takes both the
direction and the distance from one StraightLeg call, as the original does. The
arrival test is an exact float compare, so the distance has to be that same float32.
Tests pin float32 BIT PATTERNS, not tolerances: one case per narrowing plus four
independent legs component by component. A CHECK_NEAR would pass against the old
arithmetic.
sim::Distance is left in double on purpose and flagged at its declaration: it now
serves only the node-line/stutter geometry, which very likely needs the same
treatment but has zero behavioural coverage to correct it against.
Live, same VM/save/workload, run twice by this lane:
control recap-7584bad-20260908T0615Z 45 calls, 45 compared, 8 diverged, exit 1
fixed mf-45bdf7d-dirty-20260908T0721Z 45 calls, 45 compared, 0 diverged, exit 0
with identical arguments, identical pos.before and identical ORIGINAL pos.after on
all 45 calls. The control reproduced the eight divergent call_ids exactly.
Coverage unchanged and still thin: all 15 moving calls are the same straight-run
waypoint type; types 2-5 were attempted and could not be reached (the only player
that would travel a node line has no ships on this save). See docs/M-movefleet.md.
ctest 32/32; tools/clean_room_check.sh OK.
Recovers the game's event-posting API so the engine can post events and the
compare harness can see them. Until now the owner's event list was invisible to
every layer: B3's replace-mode oracle failed by exactly one item across 40,300
(an unposted EVENT_RESEARCH_OVERBUDGET) while its compare read clean, and B2's
clean compare bounds the economy fields only.
New pure module src/game/events:
* EventStorage / TurnEvents / PlayerEvent -- the list is bucketed by TURN, not
flat, which the save-editor struct notes had wrong.
* EventStorage::Post reproducing the original's rules, including the four that
change save bytes: the FLT_MAX (not infinity) default position; action 0 with
no subject and no position storing as 2; per-bucket dedup that compares
message/image/location/position/action but NOT summary; and EvNxID starting
at 0 and being promoted to 1 on the first post.
* PruneOldTurns reproduced with its off-by-one: of a leading run of buckets
older than turn-50 it erases n-1, so one stale bucket always survives. The
survivor is serialized, so correcting it would diverge.
* research_events: the five events the research path raises, their EvImg
identifiers and string-table keys, and the 0.8 completion split evaluated
against (double)0.8f rather than the decimal 0.8.
Localized text is deliberately absent: only the EVENTSUM_/EVENTMSG_ keys are
here and the text resolves through a caller-supplied lookup, as the game does.
The four event offsets the B3 hook carried as local literals now come from the
generated header; they are read off instructions rather than inferred from the
save schema.
tests/game_events: 112 checks including a replay of the event list
turn3-state.sav actually holds. ctest 31/31 -> 32/32.
docs/E-events.md carries the proposed region and Coverage wording for the next
B3 recapture.
