sots-re/verify/save-reader/SAVE_FORMAT.md

SOTS1 save format — as implemented by save_reader.py

Status: unvalidated against a real save. Everything below is what the reader assumes, derived from the community editors (R1 Bardez, R2 SOTSedit; save-editor-structs.md) corrected by the binary member tables (struct-recovery.md). The synthetic fixture (save_writer_stub.py) round-trips under these assumptions; a real .sav is the first real test. Items marked VERIFY are the ones the verifier should diff against the binary first.

1. Container

File	one gzip member (1f 8b), standard deflate. gzip.decompress
Inflated stream	parsed from offset 0; all offsets in reader output are inflated offsets
Byte order	little-endian everywhere
Text	windows-1252, no NUL terminator
Non-gzip input	accepted as an already-inflated stream (R1's *.sav.inflate.dat)

2. Primitive encodings

tag       := int32 len, len bytes ASCII                (field name; may be "." or "")
string    := int32 len, len bytes cp1252
int       := 4 bytes (int32)                           handle ids, int16 members, enums, counts
float     := 4 bytes IEEE-754 single
bool      := 1 byte, 0/1
int64     := 8 bytes                                   Bats2, rcex, PopC (raw-bytes write path)
raw       := n bytes, length known only to the writer  RNG state (2500 B = MT19937 624 words + index)

A named value is tag value pad, where pad is NUL bytes bringing the item to a 4-byte boundary. The reader implements two padding conventions and auto-detects which one the file uses (--padding auto, default):

mode	layout	item size
joint (default, R1's PaddingSize=4 over 4+len(name)+len(value))	[len][name][value][pad]	pad4(4 + len(name) + len(value))
split	[len][name][pad][value][pad]	pad4(4 + len(name)) + pad4(len(value))

The two differ only when len(name) % 4 != 0 and the value is not a multiple of 4 bytes (bools, odd-length strings): e.g. NPC+bool is 8 bytes in joint, 12 in split. VERIFY (highest priority): find a 3-char bool tag (NPC, Dep, Atq, Srn, cta, hgs, vnh, hbq, hsp) in the real inflated stream and check whether the value byte immediately follows the name. Reader assumption when writing this: joint.

Values carry no type byte. Type comes from (a) the schema/catalog, (b) a lookahead plausibility test (a candidate layout is accepted only if a plausible tag, a frame marker, or EOF follows), (c) for 4-byte words with no hint, an int/float classification by bit pattern (|int| <= 100000 → int; finite float with 1e-6 <= |f| < 1e12 → float; else int). The other reading is kept as alt. Limitation: with a name whose length is a multiple of 4, bool and int have identical item sizes; unknown names default to int.

3. Complex frames

[len][name][pad to 4]  BE EF BE EF   ...children...   10 41 10 41
                       (0xBEEFBEEF)                   (0x41104110 = ~0xBEEFBEEF)

• Every "nested object" (IStreamable via stream vft+0x28) is framed. Frames nest; the reader recovers the tree from the markers alone, without a schema.
• A frame may be tagless (BEEFBEEF directly at an item position): the reader accepts it; not observed in the reference material. VERIFY if seen.
• A frame body may be untagged bytes: Mars::Vector3 (Pos, PrvPos, FtOrig, FPogn2, FPdpos) and ShipHealth (Health) are "3 unnamed floats" per the binary. Reader accepts either 12 raw bytes before the END marker or three tagged floats (any tag, e.g. "."). VERIFY which.
• RNG frame body is treated as opaque bytes up to its END marker.
• Markers are only interpreted at item boundaries; a value that happens to equal a marker is not a problem unless the reader is already resynchronising.

4. Arrays

kind	encoding	reader schema
NonComplexArray (VectorHelper, std::list)	named int count, then count × element inline in the same frame	NArr
ComplexArray	a frame containing: int count, then count × element	CArr
element = leaf wrapper	e.g. SysID int + Sys frame; PlayerID + Player; FltID + Flt; ShipID + Ship; DesID + Des	Seq([...])
sparse tables	mnsp then n × (msp index, mv value); nadct then n × (ads, adt); haltc then n × (haltt, haltv)	NArr(Seq)

Count tags used by the binary: NumPlrs NumSys NumFlts NShips NumActs NumOwn NumDes NumLeg NumNotes NumPR NSprj Nexp NWeapXcl ndeflay rdtc numcreps ninv NumFlts NumGFs NumSnF NumMnF NVO NVE NVs NumPlgs2 PopNG mnsp nadct haltc PrNSp NTH. Count tags inside framed arrays (dipstats, preps, odes, owep, otch, cme2, Ojvs…) are unknown — the reader takes the first child as the count whatever its name. Element frame tags are unknown (R1 suggests ".") — matched by position.

5. Conditionals the reader honours

gate	consequence
ClrID/indcl/fxCrId frame: int index == -1	three ints r,g,b follow inside the frame
vnh true	vnd, vnex3, vnpex3
hindi true	indi frame (IndependenceInfo)
NVO entry isind true	indi frame
HFPlan true	FPlan frame
HLay true	Lay frame (opaque)
hbq true	BQ2 frame; hsp true → pop, ppop Population frames
HasAIR true	AIR frame (opaque)
system PID non-null	BQ frame present (reader: optional by name)
legacy tags ISuit ARes SysID TrdID Caps GtTrf FtSens FtInc lcid SensMod ExPopSys AIDifficultyID	accepted if present, expected absent in 1.8 saves

6. Top-level layout

offset 0:  [7]"Summary" [pad] BEEFBEEF ... 41104110         (R2 confirms the tag "Summary")
           CreateParameters frame  (tag unknown)
           Sim frame               (tag unknown)   KeyPath NMSz NMLc NMnx <id lists> ModCount Frame
                                                    GameID Attrib RNG GameName Map IncMod ResMod EnAl
                                                    EnTm GOTurn GOWinPly NPCm NPCo NPCi NPCv NPCa
                                                    szadj rsadj suadj sprjs RandEncAdj cmbtid turnstats
                                                    numcreps/crep ninv <invs.. AllExc..> NumPlrs/PlayerID/Player
                                                    <ISsp ISsu> NumSys/SysID/Sys NdGr2 trdmgr spymgr
                                                    NumFlts/FltID/Flt NumActs/Act SvSctOb zdsc zdsi zdst
           CdTable                 UNFRAMED at root: cdt frame, cdplayer frame, N × cdai frames

Regions in <...> are parsed generically and kept as raw item lists (sim.idLists, sim.invasionsAndExclusions, sim.species). Everything not covered by a shape (TechTree body, Events, ShipRecs, spy2, civr, comms, Ojvs, SvSctOb, trdmgr, spymgr, CdTable, …) is kept as the generic {"_name","_off","_items":[{"name","kind","value","off"}...]} form.

7. Struct field orders and types applied

Shapes live in save_reader.py (Sys, Player, Fleet, Ship, PlayerView, Population/PopG, Morale, MoraleEvent, BuildQueue/BuildOrder, IndependenceInfo, Rts, DipStat, Prep, FlightPlan, Waypoint, PrisonerHold, Summary, CreateParams, …). A("tag", type) = on-disk tag confirmed in the exe (strict name match); R("name", type) = R1 C# name only (positional). Binary corrections applied over R1:

• float: TRM CstR CstE CstT shrm RefCap RepCap, PlayerView Infra
• int64: Bats2 rcex PopC
• bool: Abdn Dstyd PvMA AIBn (R2's "short" readings are the value byte)
• int on disk though int16 in memory: TShn ETS and all DiplomacyStats counters
• string: pswd
• Vector3 (3 floats): FtOrig
• Nexp entries carry xid xmin xmax xper(float)
• Team appears twice in Player: an int, later a nested {ALid AL NA CF} frame (typed key Alliances)

8. Reader output conventions

• --dump: one line per item, @<inflated offset> name kind value; ? after the kind means the type was guessed, (alt …) shows the other reading of a 4-byte word; frames print { … } with item count and byte size.
• --json: {"padding", "stats", "issues": [...], "data": {...}}; every typed struct carries _off; unexpected items are kept under _unexpected / _extra; unknown regions keep the generic form.
• Issue levels: error (schema field missing / type impossible / frame unterminated), warn (resync, hint not plausible, unexpected items, width mismatch, best-effort read at EOF), info (unnamed small payload before END, positional tag name differing from the R1 name). --strict fails on error or warn.
• Exit status: 0 clean, 1 errors present, 2 unreadable container.

9. What the verifier should check first on a real save

1. Padding convention (§2) — a 3-char bool tag settles it.
2. The Summary frame's 13 children and their actual tag names (R1 names are probably case-variants: NumSys is confirmed by R2 as a tag string).
3. Whether Vector3 bodies are tagged (§3).
4. The tag used for element frames and for counts inside framed arrays (§4).
5. Whether the reader's auto-detected padding, resyncs == 0 and hint_failures == 0 hold; any resync offset points at a layout gap.