alex/sots-re
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
sots-re/verify/save-reader/SAVE_FORMAT.md

18 KiB
Raw Blame History

SOTS1 save format — as implemented by save_reader.py

Status: validated against the three real saves (verify/results/saves/turn1..3-state.sav, SotS 1.8): save_reader.py <save> --strict exits 0 on all three with resyncs: 0, hint-failures: 0 and 2503 raw bytes (the opaque RNG blob). Sources: the community editors (R1 Bardez, R2 SOTSedit; save-editor-structs.md), the binary member tables (struct-recovery.md) and the serializer decompiles in findings/objects/schema-gaps-resolved.md (round 3), which settled every item that was marked VERIFY in the previous revision of this file. The synthetic fixture (save_writer_stub.py) round-trips under the same assumptions (test_save_reader.py). The C++ typed writer (which reproduces the real saves byte-identically) later exposed a handful of tag names this file had wrong or left unnamed; they are corrected throughout and listed in §10.

1. Container
File one gzip member (1f 8b), standard deflate. gzip.decompress
Inflated stream parsed from offset 0; all offsets in reader output are inflated offsets
Byte order little-endian everywhere
Text windows-1252, no NUL terminator
Non-gzip input accepted as an already-inflated stream (R1's *.sav.inflate.dat)

2. Primitive encodings

tag       := int32 len, len bytes ASCII                (field name; "." when the writer passed NULL)
string    := int32 len, len bytes cp1252               len 0 is legal (Key, MapF, Pwd, Scenario, KeyPath)
int       := 4 bytes (int32)                           handle ids, int16 members widened, enums, counts
float     := 4 bytes IEEE-754 single
bool      := 1 byte, 0/1
int64     := 8 bytes                                   Bats2, rcex, PopC, pop (raw-bytes write path)
raw       := n bytes, length known only to the writer  RNG state (2503 B incl. pad = MT19937 624 words + index)

A named value is tag value pad, where pad is NUL bytes bringing the item to a 4-byte boundary. Padding is joint — confirmed on the real saves: `pad4(4 + len(name)

  • len(value)) over the whole item (haltvbool = 12 B,vnhbool = 8 B,Name= 4+4+4+len rounded). The "split" convention (pad after the name and after the value) is kept only so--padding auto` can reject it; it never matched a real file.
mode layout item size
joint (real files) [len][name][value][pad] pad4(4 + len(name) + len(value))
split (not observed) [len][name][pad][value][pad] pad4(4 + len(name)) + pad4(len(value))

Values carry no type byte. Type comes from (a) the schema/catalog, (b) a lookahead plausibility test (a candidate layout is accepted only if a plausible tag, a frame marker, or EOF follows), (c) for 4-byte words with no hint, an int/float classification by bit pattern (|int| <= 100000 → int; finite float with 1e-6 <= |f| < 1e12 → float; else int). The other reading is kept as alt. Limitation: with a name whose length is a multiple of 4, bool and int have identical item sizes; unknown names default to int.

Two rules about string values (both were reader bugs before round 3):

  • Only tag bytes are constrained to printable ASCII. A string value may hold any windows-1252 byte, including 0x80–0x9f (system names such as Kor’Voth carry 0x92 = right single quote). A tag the catalog knows to be a string is read without any text test; a value can therefore never veto the layout of the item before it (that veto was the haltv → VFlags/Name/vnh cascade). Guessing a string for an unknown tag still uses a text-plausibility guard, which accepts every cp1252-defined byte.
  • An empty string is 4 zero bytes, byte-identical to int 0. The walker reads it as "" when the tag is hinted; when an unhinted walker read a 4-zero-byte word, the schema applier coerces it to "" instead of reporting "expected string, read int".

3. Complex frames

[len][name][pad to 4]  BE EF BE EF   ...children...   10 41 10 41
                       (0xBEEFBEEF)                   (0x41104110 = ~0xBEEFBEEF)
  • Every "nested object" (IStreamable via stream vft+0x28) is framed. Frames nest; the reader recovers the tree from the markers alone, without a schema.
  • "." convention (confirmed): any write with a NULL name is emitted with the tag "." (len 1). That covers every VectorHelper<T> array ("." = count, then n × "." elements: frames for streamable T, plain values for POD T), the scalars of StarMapParams and StrategyPlayerGameSettings, PlayerColorID's index/r/g/b, and Vector3 bodies. The reader matches such items positionally (R() fields) and reports each one as an info ("tag '.' read positionally as …"); "." is deliberately kept out of the global type catalog because it carries ints, floats and frames alike.
  • Vector3 bodies are tagged: 3 × "." float items inside the frame (Pos, PrvPos, FtOrig, FPogn2, FPdpos, Health, MapP planet positions; writer FUN_008a60d0). The reader keeps the 12-raw-byte fallback but it is not exercised by real files.
  • A frame may be tagless (BEEFBEEF directly at an item position): accepted by the reader; not observed in the real saves (element frames use ".").
  • RNG frame body is one "."-tagged opaque blob read straight to its END marker (2503 B including the 3 joint-padding bytes).
  • Markers are only interpreted at item boundaries; a value that happens to equal a marker is not a problem unless the reader is already resynchronising.

4. Arrays

kind encoding reader schema
NonComplexArray (inline count) named int count, then count × element inline in the same frame NArr
ComplexArray (VectorHelper<T>) a frame containing: "." int count, then count × "." element CArr
uncounted repetition elements repeated until the frame END (or until the lead tag stops recurring) Repeat
element = leaf wrapper SysID int + Sys frame; PlayerID + Player; FltID + Flt; ShipID + Ship; DesID + Des; ply + hist Seq([...])
sparse tables mnsp then n × (msp, mv); nadct then n × (ads, adt); haltc then n × (haltt, haltv); zdsc then n × (zdsi, zdst); PrNSp then n × (PrSp, PrNum) NArr(Seq)

Inline-count tags used by the binary: NumPlrs NumSys NumFlts NShips NumActs NumOwn NumDes NumLeg NumNotes NumPR NSprj Nexp NWeapXcl ndeflay rdtc numcreps ninv NumFlts NumGFs NumSnF NumMnF NVO NVE NVs NumPlgs2 PopNG mnsp nadct haltc PrNSp NTH nply ncls seno2 spc zdsc NumIDs PlayerIDs DesignIDs SystemIDs FleetIDs ShipIDs TradeIDs AllExc AllExcCF. Count and element tags inside framed arrays (Players, GOWinPly, sacq, slost, dipstats, preps, odes, owep, otch, cme2, Ojvs, ords, wpts, paths, MapP planets …) are always "." (§3). ords (BuildQueue), wpts (FlightPlan) and paths (NdGr2) are real framed-array tags, not positional items (§10).

Uncounted lists: hist holds n × stats frames until its END; Sim holds 7 × (ISsp string, ISsu float) after the players; the root holds one CD frame per custom-data id.

5. Conditionals the reader honours

gate consequence
ClrID/indcl/FxCrID frame: "." int index == -1 three "." ints r,g,b follow inside the frame
vnh true vnd, vnex3, vnpex3 (the only conditional in that region of Sys; Name and VFlags are unconditional)
system-level hindi true indi frame (IndependenceInfo) after NVs
NVO entry indi frame is always present after isind (inline member of the map value; isind does not gate it)
PrMax > 0 PrNSp + n × (PrSp, PrNum); with PrMax == 0 the PrisH frame ends right after PrMax
HFPlan true FPlan frame: wpts{"." n, n×"."{Wpt Tp nrt{nrp nrf nrt}}} FPsp2 FPeta2 FPogn2 FPdpos pnd (turn3 only)
HLay true Lay frame (opaque)
hbq true BQ2 frame (same body as BQ: ords{"." n, n×"."{desID con conleft sav ordID}}); hsp true → pop, ppop Population frames
HasAIR true AIR frame (opaque)
system PID non-null BQ frame present (reader: optional by name)
SvSctOb pointer non-NULL SvSctOb frame before zdsc (present in the three real saves; Opt)
custom-data blob non-NULL one CD frame per CDT id that has data
legacy tags ISuit ARes SysID TrdID Caps GtTrf FtSens FtInc lcid SensMod ExPopSys AIDifficultyID Rand RandEnc NPC accepted if present (read-side only), never written by 1.8

6. Top-level layout

Written by FUN_00877070; loader FUN_0086abb0; summary-only reader FUN_008773c0.

offset 0:  "Summary"       StrategyGameInfo         GameName Turn NumSys Checksum Players{"."=n, n×"."{Slot{} Rank}}
                                                    Session{TMRS{TSTL TCTL TQTL TQTLE}} MapShape IncMod ResMod
                                                    Alliances Teams Encounters Scenario
           "CreateParams"  StrategyGameCreateParams Name ID RSeed AID Key MapP{"." int, "."{planets}, "." n×"."{ints},
                                                    "."{nodePaths}} MapS MapF NSys REnc SDist SSize SRes SSuit MaxP ASpec
                                                    bAlly NTeam tmgrp PSav PCol PTech IncM ResM scrp{spc n×(spsn sppn sppv)}
           "Sim"           StrategyServer           KeyPath NMSz NMLc NMnx | PlayerIDs DesignIDs SystemIDs FleetIDs
                                                    ShipIDs TradeIDs (each: count, n×".") | ModCount Frame GameID Attrib{}
                                                    RNG{} GameName Map IncMod ResMod EnAl EnTm GOTurn GOWinPly{} NPCm NPCo
                                                    NPCi NPCv NPCa szadj rsadj suadj sprjs{} RandEncAdj cmbtid turnstats{}
                                                    numcreps n×crep{} ninv n×(invs inve invt invtb) AllExc n×(AllExc AllExc)
                                                    AllExc n×(AllExc AllExc) AllExcCF n×(AllExcCFp AllExcCFp)
                                                    NumPlrs n×(PlayerID Player{}) 7×(ISsp ISsu) NumSys n×(SysID Sys{})
                                                    NdGr2{} trdmgr{} spymgr{} NumFlts n×(FltID Flt{}) NumActs n×Act
                                                    [SvSctOb{}] zdsc n×(zdsi zdst)
           "CDT"           CustomDataTable          NumIDs n×ID (strings: Player.<id>.TurnCommands_v5, Player.<id>.AIAgent …)
           n × "CD"        opaque custom-data frames, one per id that has a blob

Slot (SlotDef): IsPlay IsDead IsReq IsRec IsFxNm FxNm IsFxSp FxSp IsFxCr FxCrID{} IsFxBd FxBd IsFxAv FxAv Tag Pwd(string) Team Settings{4×"." int}. turnstats (GameTurnHistory): nply, n × (ply, hist{ply, stats{} …until END}); stats (PlayerTurnStats): pop(int64) sacq{} slost{} (VectorHelper: set ses seop senp seno2 n×seot2) trn almem inc tdinc sav col bat tch(int) ncls 3 × (cls shpt shpl shpk satt satl satk). Des (design header, under NumDes/NumLeg n × (DesID, Des{})): FAIDes(bool) DHide(bool) DWep(int) DName(string), then the DSec section frames (kept generic). Objective records (Player odes/owep/otch, each a framed VectorHelper of "." elements): odes = otnF otnL odid opid; owep = otnF otnL odet owep(string) owith; otch = otnF otnL odet otch(string) owith. NdGr2 (node grid): paths{"." n, n×"."{npt npid npfr npto npctm npcby npdtn npdtf npenp npuse nptf}} then nextid(int). BQ (system build queue): ords{"." n, n×"."{desID con conleft sav ordID}} — count 0 in turn1..3.

Everything not covered by a shape (TechTree body, Events, ShipRecs, spy2, civr, comms, Ojvs, Attrib, sprjs, SvSctOb, trdmgr, spymgr, CD, …) is kept as the generic {"_name","_off","_items":[{"name","kind","value","off"}...]} form.

7. Struct field orders and types applied

Shapes live in save_reader.py (Summary, Slot, CreateParams, MapP, Sim, TurnStats/PlayerTurnHistory/PlayerTurnStats, Sys, Player, Fleet, Ship, PlayerView, Population/PopG, Morale, MoraleEvent, BuildQueue/BuildOrder, IndependenceInfo, Rts, DipStat, Prep, FlightPlan, Waypoint, PrisonerHold, CdTable, …). A("tag", type) = on-disk tag confirmed in the exe (strict name match); R("name", type) = positional ("."-tagged items, or R1 C# names). Binary corrections applied over R1/R2:

  • float: TRM CstR CstE CstT shrm RefCap RepCap, PlayerView Infra; Summary.IncMod/ResMod, Sim.IncMod/ResMod, Sim.szadj/rsadj/suadj, TMRS.TSTL/TCTL/TQTL/TQTLE, CreateParams.SDist/SSize/SRes/SSuit/REnc/IncM/ResM, ISsu
  • int64: Bats2 rcex PopC, PlayerTurnStats.pop
  • bool: Abdn Dstyd PvMA AIBn haltv EnAl EnTm (R2's "short" readings are the value byte)
  • int on disk though int16 in memory: TShn ETS tch col bat and all DiplomacyStats counters
  • string: pswd, Key, MapF, Pwd, KeyPath (all may be empty)
  • Vector3 (3 × "." float): Pos PrvPos FtOrig FPogn2 FPdpos Health
  • Nexp entries carry xid xmin xmax xper(float)
  • Team appears twice in Player: an int, later a nested {ALid AL NA CF} frame (typed key Alliances)
  • GOWinPly is a framed VectorHelper<uint> ("." count + n × "." int), empty in the real saves
  • CreateParams/Summary tag case is as written above (R1's camel-case names were wrong)
  • Objective lead tag otnF, node-grid nextid, design header FAIDes DHide DWep DName, and the framed arrays ords/wpts/paths are A() (byte-confirmed, §10); the reader keeps R1's keys faiDes dHide dWep dName in its typed output for stock_designs.py

8. Reader output conventions

  • --dump: one line per item, @<inflated offset> name kind value; ? after the kind means the type was guessed, (alt …) shows the other reading of a 4-byte word; frames print { … } with item count and byte size.
  • --json: {"padding", "stats", "issues": [...], "data": {...}}; data has the keys summary, createParams, sim, cdTable, customData; every typed struct carries _off; unexpected items are kept under _unexpected / _extra; unknown regions keep the generic form.
  • Issue levels: error (schema field missing / type impossible / frame unterminated), warn (resync, hint not plausible, unexpected items, width mismatch, best-effort read at EOF), info (positional "." tag read as an R() name, unnamed small payload before END). --strict fails on error or warn.
  • Exit status: 0 clean, 1 errors present, 2 unreadable container or strict failure.

9. Verification checklist (all settled on the real saves)

  1. Padding convention — joint (§2).
  2. Summary frame: 13 children with the tags in §6 (R1's names were case-variants).
  3. Vector3 bodies — tagged, 3 × "." float (§3).
  4. Element and count tags inside framed arrays — "." (§3, §4).
  5. --padding auto picks joint; resyncs == 0 and hint_failures == 0 on turn1/2/3; the only raw bytes are the RNG blob. Remaining unknowns are the bodies of the opaque frames listed in §6 (kept generic, not a parsing gap).
  6. Every tag the reader matches by name is spelled as on disk (§10); the only positional (R()) items left are NULL-named "." items and R1-named fields inside crep, NdGr2.paths elements, MapP and Waypoint.nrt.

10. Tag-name corrections (changelog)

Found by the C++ typed save writer, which has to emit the real tag strings to round-trip the saves byte-identically. The Python reader had matched these fields positionally (R()), so a wrong spelling in the spec never failed a parse — the walker reported each as an info ("tag 'otnF' read positionally as 'ontF'") and the typed output silently used the wrong name. Each entry below was re-verified by inflating the saves and searching the byte stream for [int32 len][ascii] (offsets are inflated offsets, decimal).

was (spec / R1) is (on disk) where evidence
ontF otnF first field of every odes/owep/otch element otnF ×62 in turn2 (first @72288, in otch; @123344 region odes element otnF otnL odid opid); ×60 turn1, ×62 turn3. ontF ×0 in all three
nextId nextid last item of NdGr2 nextid ×1: turn1 @564432, turn2 @575500, turn3 @580152 (value 44 in turn2, after paths). nextId ×0
faiDes dHide dWep dName FAIDes DHide DWep DName first four items of every Des frame ×43 each in turn2 (first Des frame @67820: FAIDes @67832, DHide @67844, DWep @67856, DName @67868); ×41 turn1, ×43 turn3. Camel-case forms ×0
unnamed/positional ords framed VectorHelper inside BQ ×3 in each save (turn2 @531144, 548092, 550044; each BQ @−12); count 0 in all
unnamed/positional wpts framed VectorHelper inside FPlan turn3 only (the only save with HFPlan = 1): FPlan @591476, wpts @591492, element Wpt @591528 Tp @591540 nrt @591552. Absent from turn1/turn2 because no fleet has a flight plan there (HFPlan = 0 ×6 / ×7)
positional paths framed VectorHelper inside NdGr2 ×1 per save (turn2 @568072, NdGr2 @568056); spelled as R1 had it but now matched by name

Reader changes (save_reader.py): the fields above are A(); ords/wpts/paths lost the flex inline hedge (the game always frames them); the never-occurring dName/faiDes/ dHide entries were dropped from MANUAL_KINDS. Typed-output keys are unchanged (faiDes dHide dWep dName via key=; otnF, nextid, ords, wpts, paths were never consumed under the old names). Effect on the real saves: strict still exits 0 with 0 resyncs / 0 hint-failures; the info count drops from 257/259/259 to 197 on turn1/2/3 — the 60/62/62 otnF "read positionally" notes are gone, and every remaining info is a "." item. test_save_reader.py::TagNameCorrectionsTest pins each correction and fails a frame that uses the old spelling.