sots-re/verify/save-reader/SAVE_FORMAT.md

SOTS1 save format — as implemented by save_reader.py

Status: validated against the three real saves (verify/results/saves/turn1..3-state.sav, SotS 1.8): save_reader.py <save> --strict exits 0 on all three with resyncs: 0, hint-failures: 0 and 2503 raw bytes (the opaque RNG blob). Sources: the community editors (R1 Bardez, R2 SOTSedit; save-editor-structs.md), the binary member tables (struct-recovery.md) and the serializer decompiles in findings/objects/schema-gaps-resolved.md (round 3), which settled every item that was marked VERIFY in the previous revision of this file. The synthetic fixture (save_writer_stub.py) round-trips under the same assumptions (test_save_reader.py). The C++ typed writer (which reproduces the real saves byte-identically) later exposed a handful of tag names this file had wrong or left unnamed; they are corrected throughout and listed in §10.

1. Container

File	one gzip member (1f 8b), standard deflate. gzip.decompress
Inflated stream	parsed from offset 0; all offsets in reader output are inflated offsets
Byte order	little-endian everywhere
Text	windows-1252, no NUL terminator
Non-gzip input	accepted as an already-inflated stream (R1's *.sav.inflate.dat)

2. Primitive encodings

tag       := int32 len, len bytes ASCII                (field name; "." when the writer passed NULL)
string    := int32 len, len bytes cp1252               len 0 is legal (Key, MapF, Pwd, Scenario, KeyPath)
int       := 4 bytes (int32)                           handle ids, int16 members widened, enums, counts
float     := 4 bytes IEEE-754 single
bool      := 1 byte, 0/1
int64     := 8 bytes                                   Bats2, rcex, PopC, pop (raw-bytes write path)
raw       := n bytes, length known only to the writer  RNG state (2503 B incl. pad = MT19937 624 words + index)

A named value is tag value pad, where pad is NUL bytes bringing the item to a 4-byte boundary. Padding is joint — confirmed on the real saves: `pad4(4 + len(name)

• len(value)) over the whole item (haltvbool = 12 B,vnhbool = 8 B,Name= 4+4+4+len rounded). The "split" convention (pad after the name and after the value) is kept only so--padding auto` can reject it; it never matched a real file.

mode	layout	item size
joint (real files)	[len][name][value][pad]	pad4(4 + len(name) + len(value))
split (not observed)	[len][name][pad][value][pad]	pad4(4 + len(name)) + pad4(len(value))

Values carry no type byte. Type comes from (a) the schema/catalog, (b) a lookahead plausibility test (a candidate layout is accepted only if a plausible tag, a frame marker, or EOF follows), (c) for 4-byte words with no hint, an int/float classification by bit pattern (|int| <= 100000 → int; finite float with 1e-6 <= |f| < 1e12 → float; else int). The other reading is kept as alt. Limitation: with a name whose length is a multiple of 4, bool and int have identical item sizes; unknown names default to int.

Two rules about string values (both were reader bugs before round 3):

• Only tag bytes are constrained to printable ASCII. A string value may hold any windows-1252 byte, including 0x80–0x9f (system names such as Kor’Voth carry 0x92 = right single quote). A tag the catalog knows to be a string is read without any text test; a value can therefore never veto the layout of the item before it (that veto was the haltv → VFlags/Name/vnh cascade). Guessing a string for an unknown tag still uses a text-plausibility guard, which accepts every cp1252-defined byte.
• An empty string is 4 zero bytes, byte-identical to int 0. The walker reads it as "" when the tag is hinted; when an unhinted walker read a 4-zero-byte word, the schema applier coerces it to "" instead of reporting "expected string, read int".

3. Complex frames

[len][name][pad to 4]  BE EF BE EF   ...children...   10 41 10 41
                       (0xBEEFBEEF)                   (0x41104110 = ~0xBEEFBEEF)

• Every "nested object" (IStreamable via stream vft+0x28) is framed. Frames nest; the reader recovers the tree from the markers alone, without a schema.
• "." convention (confirmed): any write with a NULL name is emitted with the tag "." (len 1). That covers every VectorHelper<T> array ("." = count, then n × "." elements: frames for streamable T, plain values for POD T), the scalars of StarMapParams and StrategyPlayerGameSettings, PlayerColorID's index/r/g/b, and Vector3 bodies. The reader matches such items positionally (R() fields) and reports each one as an info ("tag '.' read positionally as …"); "." is deliberately kept out of the global type catalog because it carries ints, floats and frames alike.
• Vector3 bodies are tagged: 3 × "." float items inside the frame (Pos, PrvPos, FtOrig, FPogn2, FPdpos, Health, MapP planet positions; writer FUN_008a60d0). The reader keeps the 12-raw-byte fallback but it is not exercised by real files.
• A frame may be tagless (BEEFBEEF directly at an item position): accepted by the reader; not observed in the real saves (element frames use ".").
• RNG frame body is one "."-tagged opaque blob read straight to its END marker (2503 B including the 3 joint-padding bytes).
• Markers are only interpreted at item boundaries; a value that happens to equal a marker is not a problem unless the reader is already resynchronising.

4. Arrays

kind	encoding	reader schema
NonComplexArray (inline count)	named int count, then count × element inline in the same frame	NArr
ComplexArray (VectorHelper<T>)	a frame containing: "." int count, then count × "." element	CArr
uncounted repetition	elements repeated until the frame END (or until the lead tag stops recurring)	Repeat
element = leaf wrapper	SysID int + Sys frame; PlayerID + Player; FltID + Flt; ShipID + Ship; DesID + Des; ply + hist	Seq([...])
sparse tables	mnsp then n × (msp, mv); nadct then n × (ads, adt); haltc then n × (haltt, haltv); zdsc then n × (zdsi, zdst); PrNSp then n × (PrSp, PrNum)	NArr(Seq)

Inline-count tags used by the binary: NumPlrs NumSys NumFlts NShips NumActs NumOwn NumDes NumLeg NumNotes NumPR NSprj Nexp NWeapXcl ndeflay rdtc numcreps ninv NumFlts NumGFs NumSnF NumMnF NVO NVE NVs NumPlgs2 PopNG mnsp nadct haltc PrNSp NTH nply ncls seno2 spc zdsc NumIDs PlayerIDs DesignIDs SystemIDs FleetIDs ShipIDs TradeIDs AllExc AllExcCF. Count and element tags inside framed arrays (Players, GOWinPly, sacq, slost, dipstats, preps, odes, owep, otch, cme2, Ojvs, ords, wpts, paths, MapP planets …) are always "." (§3). ords (BuildQueue), wpts (FlightPlan) and paths (NdGr2) are real framed-array tags, not positional items (§10).

Uncounted lists: hist holds n × stats frames until its END; Sim holds 7 × (ISsp string, ISsu float) after the players; the root holds one CD frame per custom-data id.

5. Conditionals the reader honours

gate	consequence
ClrID/indcl/FxCrID frame: "." int index == -1	three "." ints r,g,b follow inside the frame
vnh true	vnd, vnex3, vnpex3 (the only conditional in that region of Sys; Name and VFlags are unconditional)
system-level hindi true	indi frame (IndependenceInfo) after NVs
NVO entry	indi frame is always present after isind (inline member of the map value; isind does not gate it)
PrMax > 0	PrNSp + n × (PrSp, PrNum); with PrMax == 0 the PrisH frame ends right after PrMax
HFPlan true	FPlan frame: wpts{"." n, n×"."{Wpt Tp nrt{nrp nrf nrt}}} FPsp2 FPeta2 FPogn2 FPdpos pnd (turn3 only)
HLay true	Lay frame (opaque)
hbq true	BQ2 frame (same body as BQ: ords{"." n, n×"."{desID con conleft sav ordID}}); hsp true → pop, ppop Population frames
HasAIR true	AIR frame (opaque)
system PID non-null	BQ frame present (reader: optional by name)
SvSctOb pointer non-NULL	SvSctOb frame before zdsc (present in the three real saves; Opt)
custom-data blob non-NULL	one CD frame per CDT id that has data
legacy tags ISuit ARes SysID TrdID Caps GtTrf FtSens FtInc lcid SensMod ExPopSys AIDifficultyID Rand RandEnc NPC	accepted if present (read-side only), never written by 1.8

6. Top-level layout

Written by FUN_00877070; loader FUN_0086abb0; summary-only reader FUN_008773c0.

offset 0:  "Summary"       StrategyGameInfo         GameName Turn NumSys Checksum Players{"."=n, n×"."{Slot{} Rank}}
                                                    Session{TMRS{TSTL TCTL TQTL TQTLE}} MapShape IncMod ResMod
                                                    Alliances Teams Encounters Scenario
           "CreateParams"  StrategyGameCreateParams Name ID RSeed AID Key MapP{"." int, "."{planets}, "." n×"."{ints},
                                                    "."{nodePaths}} MapS MapF NSys REnc SDist SSize SRes SSuit MaxP ASpec
                                                    bAlly NTeam tmgrp PSav PCol PTech IncM ResM scrp{spc n×(spsn sppn sppv)}
           "Sim"           StrategyServer           KeyPath NMSz NMLc NMnx | PlayerIDs DesignIDs SystemIDs FleetIDs
                                                    ShipIDs TradeIDs (each: count, n×".") | ModCount Frame GameID Attrib{}
                                                    RNG{} GameName Map IncMod ResMod EnAl EnTm GOTurn GOWinPly{} NPCm NPCo
                                                    NPCi NPCv NPCa szadj rsadj suadj sprjs{} RandEncAdj cmbtid turnstats{}
                                                    numcreps n×crep{} ninv n×(invs inve invt invtb) AllExc n×(AllExc AllExc)
                                                    AllExc n×(AllExc AllExc) AllExcCF n×(AllExcCFp AllExcCFp)
                                                    NumPlrs n×(PlayerID Player{}) 7×(ISsp ISsu) NumSys n×(SysID Sys{})
                                                    NdGr2{} trdmgr{} spymgr{} NumFlts n×(FltID Flt{}) NumActs n×Act
                                                    [SvSctOb{}] zdsc n×(zdsi zdst)
           "CDT"           CustomDataTable          NumIDs n×ID (strings: Player.<id>.TurnCommands_v5, Player.<id>.AIAgent …)
           n × "CD"        opaque custom-data frames, one per id that has a blob

Slot (SlotDef): IsPlay IsDead IsReq IsRec IsFxNm FxNm IsFxSp FxSp IsFxCr FxCrID{} IsFxBd FxBd IsFxAv FxAv Tag Pwd(string) Team Settings{4×"." int}. turnstats (GameTurnHistory): nply, n × (ply, hist{ply, stats{} …until END}); stats (PlayerTurnStats): pop(int64) sacq{} slost{} (VectorHelper: set ses seop senp seno2 n×seot2) trn almem inc tdinc sav col bat tch(int) ncls 3 × (cls shpt shpl shpk satt satl satk). Des (design header, under NumDes/NumLeg n × (DesID, Des{})): FAIDes(bool) DHide(bool) DWep(int) DName(string), then the DSec section frames (kept generic). Objective records (Player odes/owep/otch, each a framed VectorHelper of "." elements): odes = otnF otnL odid opid; owep = otnF otnL odet owep(string) owith; otch = otnF otnL odet otch(string) owith.

Confirmed against the binary (2026-09-08, lane S). otch elements are Game::ObservedTech (Write 0x00817cf0, Read 0x00817c40, sizeof 0x2c) and owep elements are Game::ObservedWeapon (Write 0x00817bc0, Read 0x00817b10) — the same element shape with a different string tag. The serializers emit exactly the order this file already had. In-memory types, which the on-disk framing does not reveal: otnF and otnL are uint16 widened to int by the stream (movzx then stream vft+0x24); odet is a bool written through WriteBool, i.e. 1 byte plus 3 NUL joint-padding bytes, not an int32. save_reader.py types odet as int, which is benign and deliberate: odet is a 4-character tag, so a bool item and an int item are both pad4(4+4+1) == pad4(4+4+4) == 12 bytes and the little-endian value is identical (§2's stated "bool and int have identical item sizes for a name whose length is a multiple of 4"). Re-typing it would change nothing on disk; it is recorded here so the reimplementation uses a bool, and so nobody later "discovers" the same 3 padding bytes as a missing field. NdGr2 (node grid): paths{"." n, n×"."{npt npid npfr npto npctm npcby npdtn npdtf npenp npuse nptf}} then nextid(int). BQ (system build queue): ords{"." n, n×"."{desID con conleft sav ordID}} — count 0 in turn1..3.

Everything not covered by a shape (TechTree body, Events, ShipRecs, spy2, civr, comms, Ojvs, Attrib, sprjs, SvSctOb, trdmgr, spymgr, CD, …) is kept as the generic {"_name","_off","_items":[{"name","kind","value","off"}...]} form.

7. Struct field orders and types applied

Shapes live in save_reader.py (Summary, Slot, CreateParams, MapP, Sim, TurnStats/PlayerTurnHistory/PlayerTurnStats, Sys, Player, Fleet, Ship, PlayerView, Population/PopG, Morale, MoraleEvent, BuildQueue/BuildOrder, IndependenceInfo, Rts, DipStat, Prep, FlightPlan, Waypoint, PrisonerHold, CdTable, …). A("tag", type) = on-disk tag confirmed in the exe (strict name match); R("name", type) = positional ("."-tagged items, or R1 C# names). Binary corrections applied over R1/R2:

• float: TRM CstR CstE CstT shrm RefCap RepCap, PlayerView Infra; Summary.IncMod/ResMod, Sim.IncMod/ResMod, Sim.szadj/rsadj/suadj, TMRS.TSTL/TCTL/TQTL/TQTLE, CreateParams.SDist/SSize/SRes/SSuit/REnc/IncM/ResM, ISsu
• int64: Bats2 rcex PopC, PlayerTurnStats.pop
• bool: Abdn Dstyd PvMA AIBn haltv EnAl EnTm (R2's "short" readings are the value byte)
• int on disk though int16 in memory: TShn ETS tch col bat and all DiplomacyStats counters
• string: pswd, Key, MapF, Pwd, KeyPath (all may be empty)
• Vector3 (3 × "." float): Pos PrvPos FtOrig FPogn2 FPdpos Health
• Nexp entries carry xid xmin xmax xper(float)
• Team appears twice in Player: an int, later a nested {ALid AL NA CF} frame (typed key Alliances)
• GOWinPly is a framed VectorHelper<uint> ("." count + n × "." int), empty in the real saves
• CreateParams/Summary tag case is as written above (R1's camel-case names were wrong)
• Objective lead tag otnF, node-grid nextid, design header FAIDes DHide DWep DName, and the framed arrays ords/wpts/paths are A() (byte-confirmed, §10); the reader keeps R1's keys faiDes dHide dWep dName in its typed output for stock_designs.py

8. Reader output conventions

• --dump: one line per item, @<inflated offset> name kind value; ? after the kind means the type was guessed, (alt …) shows the other reading of a 4-byte word; frames print { … } with item count and byte size.
• --json: {"padding", "stats", "issues": [...], "data": {...}}; data has the keys summary, createParams, sim, cdTable, customData; every typed struct carries _off; unexpected items are kept under _unexpected / _extra; unknown regions keep the generic form.
• Issue levels: error (schema field missing / type impossible / frame unterminated), warn (resync, hint not plausible, unexpected items, width mismatch, best-effort read at EOF), info (positional "." tag read as an R() name, unnamed small payload before END). --strict fails on error or warn.
• Exit status: 0 clean, 1 errors present, 2 unreadable container or strict failure.

9. Verification checklist (all settled on the real saves)

1. Padding convention — joint (§2).
2. Summary frame: 13 children with the tags in §6 (R1's names were case-variants).
3. Vector3 bodies — tagged, 3 × "." float (§3).
4. Element and count tags inside framed arrays — "." (§3, §4).
5. --padding auto picks joint; resyncs == 0 and hint_failures == 0 on turn1/2/3; the only raw bytes are the RNG blob. Remaining unknowns are the bodies of the opaque frames listed in §6 (kept generic, not a parsing gap).
6. Every tag the reader matches by name is spelled as on disk (§10); the only positional (R()) items left are NULL-named "." items and R1-named fields inside crep, NdGr2.paths elements, MapP and Waypoint.nrt.

10. Tag-name corrections (changelog)

Found by the C++ typed save writer, which has to emit the real tag strings to round-trip the saves byte-identically. The Python reader had matched these fields positionally (R()), so a wrong spelling in the spec never failed a parse — the walker reported each as an info ("tag 'otnF' read positionally as 'ontF'") and the typed output silently used the wrong name. Each entry below was re-verified by inflating the saves and searching the byte stream for [int32 len][ascii] (offsets are inflated offsets, decimal).

was (spec / R1)	is (on disk)	where	evidence
ontF	otnF	first field of every odes/owep/otch element	otnF ×62 in turn2 (first @72288, in otch; @123344 region odes element otnF otnL odid opid); ×60 turn1, ×62 turn3. ontF ×0 in all three
nextId	nextid	last item of NdGr2	nextid ×1: turn1 @564432, turn2 @575500, turn3 @580152 (value 44 in turn2, after paths). nextId ×0
faiDes dHide dWep dName	FAIDes DHide DWep DName	first four items of every Des frame	×43 each in turn2 (first Des frame @67820: FAIDes @67832, DHide @67844, DWep @67856, DName @67868); ×41 turn1, ×43 turn3. Camel-case forms ×0
unnamed/positional	ords	framed VectorHelper inside BQ	×3 in each save (turn2 @531144, 548092, 550044; each BQ @−12); count 0 in all
unnamed/positional	wpts	framed VectorHelper inside FPlan	turn3 only (the only save with HFPlan = 1): FPlan @591476, wpts @591492, element Wpt @591528 Tp @591540 nrt @591552. Absent from turn1/turn2 because no fleet has a flight plan there (HFPlan = 0 ×6 / ×7)
positional	paths	framed VectorHelper inside NdGr2	×1 per save (turn2 @568072, NdGr2 @568056); spelled as R1 had it but now matched by name

Reader changes (save_reader.py): the fields above are A(); ords/wpts/paths lost the flex inline hedge (the game always frames them); the never-occurring dName/faiDes/ dHide entries were dropped from MANUAL_KINDS. Typed-output keys are unchanged (faiDes dHide dWep dName via key=; otnF, nextid, ords, wpts, paths were never consumed under the old names). Effect on the real saves: strict still exits 0 with 0 resyncs / 0 hint-failures; the info count drops from 257/259/259 to 197 on turn1/2/3 — the 60/62/62 otnF "read positionally" notes are gone, and every remaining info is a "." item. test_save_reader.py::TagNameCorrectionsTest pins each correction and fails a frame that uses the old spelling.

11. CD Player.<id>.TurnCommands_v5 — the issued-order block (lane O, 2026-09-08)

Until now every save in the corpus carried the same empty TurnCommands_v5 frame, so the block looked structureless. state_checksum --tree --depth 1 makes that concrete: CD[0] is 3df7d93164fb1d7d, 35 leaves / 122 B, byte-for-byte identical in turn1/2/3-state.sav, zuul-turn5-species5.sav and human-turn3-noderoute.sav. Saves made after issuing orders and before End Turn grow it (all five below are --strict clean and coverage: PROVED):

save	CD[0]	orders issued that turn
any no-orders save	35 leaves / 122 B	—
human-turn2-orders.sav	41 / 143 B	1 fleet move
zuul-turn15-orders.sav	61 / 226 B	research target + research boost + 5 build + 1 fleet move
zuul-turn17-orders2.sav	129 / 498 B	20 build + 1 system-rates + 1 colonize (3 ships)

Body, as read off zuul-turn15-orders.sav / zuul-turn17-orders2.sav (all items "."-tagged, so this is a positional record, not a named one).

Corrected by lane Q, 2026-09-08, from the writer's own instruction stream (0x00842540). The shape below is right about every command it observed, and wrong about two structural things: the fleet-move element ends in a counted route vector, not a fixed {fleetId, 1, destSystemId, 0} quadruple; and the "trailing zero words" are not padding but twenty-seven command lists, of which five have ever been non-empty. Full derivation and the per-list element records are in findings/objects/turncommands-block.md.

int   playerId                 (16 = the human player's PID; matches CDT's "Player.00000016.…")
bool  hasResearchRateCmd    ; if set, float researchRate follows
                               (True in every sample; 0.25 default -> 0.97 after the slider)
bool  hasResearchTargetCmd  ; if set, int techId follows          (191 = WEP_GrnLas)
bool  hasResearchBoostCmd   ; if set, int money then float follow (216383 -- exactly the
                               Imperial Savings delta the boost produced; float 0.9992…)
bool  gate4                 ; if set, bool + int follow           (never seen set)
bool  gate5                 ; if set, three floats follow         (never seen set)
bool  gate6                 ; if set, a CivilianRatios frame follows (never seen set)

27 x { int count ; count x <that list's element record> }   -- ALL 27 always written,
                                                              so an empty list costs one zero int

  list 3  build orders   { int ordinal; int designId; int systemId; int w }
                               (ordinal is the running build-queue index, not 0-based per turn;
                                designId 608 = DE Colonizer, 576 = DE Armor; systemId 384 = the
                                home system that owns the queue)
  list 5  system rates   { int systemId; frame { float SRs; float SRt; float SRsc;
                                                 float SRtf; float SRi; float SRoh; int SRnr } }
                               (SRsc = 1.0 is the Planetary Budget slider pushed fully to
                                Construction; this is the only NAMED sub-frame in the block)
  list 7  colonize       { int shipId; int w }
  list 8  fleet moves    { int fleetId; int nHops; nHops x int systemId }
  list 14 ?              { int fleetId; int 0; bool True }   -- observed once, alongside a move

So the empty block is 1 + 1 + 1 + 5 (playerId, the rate gate and its float, five clear gates) + 27 zero counts = 35 items, which is why every no-orders save is bit-identical here.

Cross-checks that pin the decode: 384/432 resolve to Sys Gallandro/Octans in the same file, 688 to the Flt the UI showed as Alpha Fleet with Dest: Octans, and 216383 is the exact Imperial Savings drop the Boost Research panel caused. The commands are pending, not applied: in zuul-turn15-orders.sav the ServerPlayer still reads ResRate 0.25, ResTNm '', Sav 2,902,722 while the block already carries rate 0.97 and the boost — the block is the client→server queue that ProcessTurn drains.

Not observed: the three unset gates (4, 5, 6 — the last being the empire civilian-settings command), and the twenty-two lists other than 3, 5, 7, 8 and 14. Neither noderoute save carries a fleet move, so the node-route UI does not queue through list 8.

save_reader.py still reads CD frames generically (any), as it does for the AIAgent bodies — see §8 of findings/objects/turncommands-block.md for why that was left alone. The typed decode lives in the engine (src/mars/stream/shapes.h, shapes::TurnCommands).