sots-engine/docs/M1.md

M1 — flat KEY/value config loader (Mars::GlobalConsts::LoadFile) old-vs-new on the live game

Result (2026-09-07/08): the first real old-vs-new verification. Trace mode: 19 LoadFile calls (1088 constant slots), tracecmp.py exit 0. Compare mode: 19/19 compared, 0 divergences. Replace mode: our loader fed the game its constants alone; loading ref-turn2.sav and pressing End Turn reproduced the determinism oracle byte for byte ((Autosave).sav = 978041ac…, (Autosave EndTurn).sav = bb4fd9ac…). The offline replay test (golden trace through the host build) also passes. Game left running on VM140 in hooks=trace at the main menu.

What was hooked

Mars::GlobalConsts::LoadFile(const char* file, GlobalConstMap* consts) — cdecl, verified, so the Hook<Descriptor> template applies directly (src/shim/hooks/global_consts.{h,cpp}, installed from src/shim/main.cpp via InstallTemplateHook<>). It is called once per data file from LoadAll() inside Application::Initialize, before the intro movie; the files it touches (19, in map order of the file names):

Data/Combat/{actor,assaultshuttle,camera,drone,gravboat,ortgay,planet,sensors,ship,spyship,terrain,vnhomeworld}.txt
Data/encounters.txt  Data/globals.txt  Data/options.txt  Data/Species.txt
Data/Strategy/AI/AIRebellion.txt  Data/Strategy/starmap.txt  Data/Strategy/StrategyVars.txt

ctechvars.txt, shipai.txt, damfx*.txt, playercolors.txt, systemnames.txt are not GlobalConsts files (other readers).

Region model

The map handed to LoadFile is an MSVC 2010 std::map<const char*, GlobalConst*> holding the constants registered for that file (key -> {storage, key, parser, file}; layouts are now in sots_addresses.h). regions() walks it in order before the original runs and declares one region per constant, named by the key, sized and described by the parser the registering stub pushed:

parser (VA)	kind	width	describer
GlobalConst_ParseInt 0x008b7000	int	4	{t:"int", v:i32}
GlobalConst_ParseFloat 0x008b7020	float	4	{t:"float", v:f32}
GlobalConst_ParseFloatScaled 0x008b70e0	fscaled	4	{t:"fscaled", v:f32}
GlobalConst_ParseColour 0x008b7110	colour	16	{t:"colour", r,g,b,a:f32}
GlobalConst_ParseRect 0x008b7040 (new)	rect	16	{t:"rect", x,y,w,h:i32}
GlobalConst_ParseVec3 0x008b7080 (new)	vec3	12	{t:"vec3", x,y,z:f32}
GlobalConst_ParseString 0x008b7670 (new)	string	24	{t:"string", v:str} (decoded MSVC std::string)

Args: file (str), consts (ptr), nkeys (u32) and scale (f64, the scaled-float multiplier) — the last so a golden log replays offline with nothing from the exe. Return: void.

Discoveries the RE notes did not have (all fed back into sots-re/ghidra/addresses.json, status verified-by-trace, and regenerated into include/generated/sots_addresses.h):

• Three more parsers. 146 keys (*_NAME, *_SOUND, *_TEXTURENAME, DERELICT_SECTION_nn, Data/options.txt …) are std::string slots (0x18 bytes: 16-byte SSO buffer / heap pointer, size @+0x10, capacity @+0x14). Five HIVER_SPAWN_* keys are float[3] ("0 0 50.0"). Two COMBATSETUP_CARD_*_RECT keys are int[4].
• The scaled-float constant at 0x00af5210 is a double (pi/180 = 0x3f91df46a2529d39), not a float; reading it as float gives -2.85e-18. The product is computed in double and stored as float: (float)((double)v * scale).
• Data/Strategy/StrategyVars.txt and Data/encounters.txt have no trailing newline, so the original never applies their last key (CIVILIAN_BURDEN_RATIO, HERALD_SPEECH_MAX_INVERVAL): visible in the trace as the only unchanged regions in those files and reproduced by ours ("expected but not found", missing 1 in both).

ours — src/game/config/config_loader.{h,cpp} (lib sots_game_config)

• File bytes come from the game's own gobio::ReadFile (honours the mount order / mods) and are released with RefCounted_Release; the host tests read plain files.
• Pairs come from mars::text::parse_flat_kv (engine-parity: Mars::Script tokenizer, blocks skipped, trailing pair dropped). apply() implements first-occurrence-wins on top of the raw pair list (a slot is consumed on first sight; repeats and unknown keys are logged "[file] KEY not recognized or is multiply defined.", absent slots "[file] KEY expected but not found.", to shim.log with a cfg: prefix).
• Typed writes mirror sscanf: %d (CRT wrap-around accumulator, no write when no digit), %f (one rounding from the text via strtold → float), colour (%d %d %d %d, defaults 255, /255.0 in double, clamp, always writes all four), vec3 / rect (components until the first scan failure), scaled float (scan then multiply, whatever the scan did).
• String slots are written through an ExternWriter: the shim hands the text to the game's own ParseString (a MinGW-built std::string cannot be handed to MSVCR100). In compare mode that runs on the scratch copy of the object, which starts as the image default (empty short string), so nothing belonging to the live object is ever freed; long strings leak one heap block per compare call (startup only).

Runs (/bulk-storage/re-lab/shim/traces/, build 4f0a9db-dirty-20260908T0205Z)

file	mode	calls	result
m1-trace-golden.jsonl	trace	19 LoadFile + 1 selftest	tracecmp.py exit 0, 0 invalid
m1-compare.jsonl	compare	19 compared	0 diverged, 0 errors, exit 0
replace (no log)	replace	19	End Turn from ref-turn2.sav: (Autosave).sav 978041ac… / (Autosave EndTurn).sav bb4fd9ac… = oracle
m1-shim.log			all four runs' banners, cfg: stats per file
m1-*.png			main menu (trace, compare, final), turn 2 / turn 3 in replace

Offline: SOTS_M1_TRACE=<golden> SOTS_DATA_DIR=<extract root> build-host/tests/game_config/game_config_replay → tracecmp.py --replay exit 0 (19 calls, 0 files missing). Host suite: ctest 22/22.

Gotchas

1. shim.cfg per-hook line: hook.Mars::GlobalConsts::LoadFile=compare|replace (default hooks=trace).
2. Z: is per-logon; from SSH there is no share. Push the dist to C:\SOTS\shimdist\ with scp and run deploy.ps1 -Dist C:\SOTS\shimdist. Inline PowerShell over ssh mangles $ and quotes — ship .ps1 files (C:\SOTS\ui\{status,loginfo,lastrun,relaunch,hashes}.ps1).
3. Local cmake is 3.22 (< 3.25 required): use uvx --from cmake cmake with Unix Makefiles.
4. The per-call slot table is kept in statics between regions() → rebind() → ours(); fine because LoadFile is single-threaded at startup, but do not reuse the pattern for re-entrant hooks.
5. Data/Combat/ctechvars.txt style files (a single name { … } block) would set nothing through this loader — they are simply not GlobalConsts files, as the trace confirms.