alex/sots-re
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
sots-re/verify/save-reader/SAVE_FORMAT.md

14 KiB
Raw Blame History

SOTS1 save format — as implemented by save_reader.py

Status: validated against the three real saves (verify/results/saves/turn1..3-state.sav, SotS 1.8): save_reader.py <save> --strict exits 0 on all three with resyncs: 0, hint-failures: 0 and 2503 raw bytes (the opaque RNG blob). Sources: the community editors (R1 Bardez, R2 SOTSedit; save-editor-structs.md), the binary member tables (struct-recovery.md) and the serializer decompiles in findings/objects/schema-gaps-resolved.md (round 3), which settled every item that was marked VERIFY in the previous revision of this file. The synthetic fixture (save_writer_stub.py) round-trips under the same assumptions (test_save_reader.py).

1. Container
File one gzip member (1f 8b), standard deflate. gzip.decompress
Inflated stream parsed from offset 0; all offsets in reader output are inflated offsets
Byte order little-endian everywhere
Text windows-1252, no NUL terminator
Non-gzip input accepted as an already-inflated stream (R1's *.sav.inflate.dat)

2. Primitive encodings

tag       := int32 len, len bytes ASCII                (field name; "." when the writer passed NULL)
string    := int32 len, len bytes cp1252               len 0 is legal (Key, MapF, Pwd, Scenario, KeyPath)
int       := 4 bytes (int32)                           handle ids, int16 members widened, enums, counts
float     := 4 bytes IEEE-754 single
bool      := 1 byte, 0/1
int64     := 8 bytes                                   Bats2, rcex, PopC, pop (raw-bytes write path)
raw       := n bytes, length known only to the writer  RNG state (2503 B incl. pad = MT19937 624 words + index)

A named value is tag value pad, where pad is NUL bytes bringing the item to a 4-byte boundary. Padding is joint — confirmed on the real saves: `pad4(4 + len(name)

  • len(value)) over the whole item (haltvbool = 12 B,vnhbool = 8 B,Name= 4+4+4+len rounded). The "split" convention (pad after the name and after the value) is kept only so--padding auto` can reject it; it never matched a real file.
mode layout item size
joint (real files) [len][name][value][pad] pad4(4 + len(name) + len(value))
split (not observed) [len][name][pad][value][pad] pad4(4 + len(name)) + pad4(len(value))

Values carry no type byte. Type comes from (a) the schema/catalog, (b) a lookahead plausibility test (a candidate layout is accepted only if a plausible tag, a frame marker, or EOF follows), (c) for 4-byte words with no hint, an int/float classification by bit pattern (|int| <= 100000 → int; finite float with 1e-6 <= |f| < 1e12 → float; else int). The other reading is kept as alt. Limitation: with a name whose length is a multiple of 4, bool and int have identical item sizes; unknown names default to int.

Two rules about string values (both were reader bugs before round 3):

  • Only tag bytes are constrained to printable ASCII. A string value may hold any windows-1252 byte, including 0x80–0x9f (system names such as Kor’Voth carry 0x92 = right single quote). A tag the catalog knows to be a string is read without any text test; a value can therefore never veto the layout of the item before it (that veto was the haltv → VFlags/Name/vnh cascade). Guessing a string for an unknown tag still uses a text-plausibility guard, which accepts every cp1252-defined byte.
  • An empty string is 4 zero bytes, byte-identical to int 0. The walker reads it as "" when the tag is hinted; when an unhinted walker read a 4-zero-byte word, the schema applier coerces it to "" instead of reporting "expected string, read int".

3. Complex frames

[len][name][pad to 4]  BE EF BE EF   ...children...   10 41 10 41
                       (0xBEEFBEEF)                   (0x41104110 = ~0xBEEFBEEF)
  • Every "nested object" (IStreamable via stream vft+0x28) is framed. Frames nest; the reader recovers the tree from the markers alone, without a schema.
  • "." convention (confirmed): any write with a NULL name is emitted with the tag "." (len 1). That covers every VectorHelper<T> array ("." = count, then n × "." elements: frames for streamable T, plain values for POD T), the scalars of StarMapParams and StrategyPlayerGameSettings, PlayerColorID's index/r/g/b, and Vector3 bodies. The reader matches such items positionally (R() fields) and reports each one as an info ("tag '.' read positionally as …"); "." is deliberately kept out of the global type catalog because it carries ints, floats and frames alike.
  • Vector3 bodies are tagged: 3 × "." float items inside the frame (Pos, PrvPos, FtOrig, FPogn2, FPdpos, Health, MapP planet positions; writer FUN_008a60d0). The reader keeps the 12-raw-byte fallback but it is not exercised by real files.
  • A frame may be tagless (BEEFBEEF directly at an item position): accepted by the reader; not observed in the real saves (element frames use ".").
  • RNG frame body is one "."-tagged opaque blob read straight to its END marker (2503 B including the 3 joint-padding bytes).
  • Markers are only interpreted at item boundaries; a value that happens to equal a marker is not a problem unless the reader is already resynchronising.

4. Arrays

kind encoding reader schema
NonComplexArray (inline count) named int count, then count × element inline in the same frame NArr
ComplexArray (VectorHelper<T>) a frame containing: "." int count, then count × "." element CArr
uncounted repetition elements repeated until the frame END (or until the lead tag stops recurring) Repeat
element = leaf wrapper SysID int + Sys frame; PlayerID + Player; FltID + Flt; ShipID + Ship; DesID + Des; ply + hist Seq([...])
sparse tables mnsp then n × (msp, mv); nadct then n × (ads, adt); haltc then n × (haltt, haltv); zdsc then n × (zdsi, zdst); PrNSp then n × (PrSp, PrNum) NArr(Seq)

Inline-count tags used by the binary: NumPlrs NumSys NumFlts NShips NumActs NumOwn NumDes NumLeg NumNotes NumPR NSprj Nexp NWeapXcl ndeflay rdtc numcreps ninv NumFlts NumGFs NumSnF NumMnF NVO NVE NVs NumPlgs2 PopNG mnsp nadct haltc PrNSp NTH nply ncls seno2 spc zdsc NumIDs PlayerIDs DesignIDs SystemIDs FleetIDs ShipIDs TradeIDs AllExc AllExcCF. Count and element tags inside framed arrays (Players, GOWinPly, sacq, slost, dipstats, preps, odes, owep, otch, cme2, Ojvs, MapP planets …) are always "." (§3).

Uncounted lists: hist holds n × stats frames until its END; Sim holds 7 × (ISsp string, ISsu float) after the players; the root holds one CD frame per custom-data id.

5. Conditionals the reader honours

gate consequence
ClrID/indcl/FxCrID frame: "." int index == -1 three "." ints r,g,b follow inside the frame
vnh true vnd, vnex3, vnpex3 (the only conditional in that region of Sys; Name and VFlags are unconditional)
system-level hindi true indi frame (IndependenceInfo) after NVs
NVO entry indi frame is always present after isind (inline member of the map value; isind does not gate it)
PrMax > 0 PrNSp + n × (PrSp, PrNum); with PrMax == 0 the PrisH frame ends right after PrMax
HFPlan true FPlan frame
HLay true Lay frame (opaque)
hbq true BQ2 frame; hsp true → pop, ppop Population frames
HasAIR true AIR frame (opaque)
system PID non-null BQ frame present (reader: optional by name)
SvSctOb pointer non-NULL SvSctOb frame before zdsc (present in the three real saves; Opt)
custom-data blob non-NULL one CD frame per CDT id that has data
legacy tags ISuit ARes SysID TrdID Caps GtTrf FtSens FtInc lcid SensMod ExPopSys AIDifficultyID Rand RandEnc NPC accepted if present (read-side only), never written by 1.8

6. Top-level layout

Written by FUN_00877070; loader FUN_0086abb0; summary-only reader FUN_008773c0.

offset 0:  "Summary"       StrategyGameInfo         GameName Turn NumSys Checksum Players{"."=n, n×"."{Slot{} Rank}}
                                                    Session{TMRS{TSTL TCTL TQTL TQTLE}} MapShape IncMod ResMod
                                                    Alliances Teams Encounters Scenario
           "CreateParams"  StrategyGameCreateParams Name ID RSeed AID Key MapP{"." int, "."{planets}, "." n×"."{ints},
                                                    "."{nodePaths}} MapS MapF NSys REnc SDist SSize SRes SSuit MaxP ASpec
                                                    bAlly NTeam tmgrp PSav PCol PTech IncM ResM scrp{spc n×(spsn sppn sppv)}
           "Sim"           StrategyServer           KeyPath NMSz NMLc NMnx | PlayerIDs DesignIDs SystemIDs FleetIDs
                                                    ShipIDs TradeIDs (each: count, n×".") | ModCount Frame GameID Attrib{}
                                                    RNG{} GameName Map IncMod ResMod EnAl EnTm GOTurn GOWinPly{} NPCm NPCo
                                                    NPCi NPCv NPCa szadj rsadj suadj sprjs{} RandEncAdj cmbtid turnstats{}
                                                    numcreps n×crep{} ninv n×(invs inve invt invtb) AllExc n×(AllExc AllExc)
                                                    AllExc n×(AllExc AllExc) AllExcCF n×(AllExcCFp AllExcCFp)
                                                    NumPlrs n×(PlayerID Player{}) 7×(ISsp ISsu) NumSys n×(SysID Sys{})
                                                    NdGr2{} trdmgr{} spymgr{} NumFlts n×(FltID Flt{}) NumActs n×Act
                                                    [SvSctOb{}] zdsc n×(zdsi zdst)
           "CDT"           CustomDataTable          NumIDs n×ID (strings: Player.<id>.TurnCommands_v5, Player.<id>.AIAgent …)
           n × "CD"        opaque custom-data frames, one per id that has a blob

Slot (SlotDef): IsPlay IsDead IsReq IsRec IsFxNm FxNm IsFxSp FxSp IsFxCr FxCrID{} IsFxBd FxBd IsFxAv FxAv Tag Pwd(string) Team Settings{4×"." int}. turnstats (GameTurnHistory): nply, n × (ply, hist{ply, stats{} …until END}); stats (PlayerTurnStats): pop(int64) sacq{} slost{} (VectorHelper: set ses seop senp seno2 n×seot2) trn almem inc tdinc sav col bat tch(int) ncls 3 × (cls shpt shpl shpk satt satl satk).

Everything not covered by a shape (TechTree body, Events, ShipRecs, spy2, civr, comms, Ojvs, Attrib, sprjs, SvSctOb, trdmgr, spymgr, CD, …) is kept as the generic {"_name","_off","_items":[{"name","kind","value","off"}...]} form.

7. Struct field orders and types applied

Shapes live in save_reader.py (Summary, Slot, CreateParams, MapP, Sim, TurnStats/PlayerTurnHistory/PlayerTurnStats, Sys, Player, Fleet, Ship, PlayerView, Population/PopG, Morale, MoraleEvent, BuildQueue/BuildOrder, IndependenceInfo, Rts, DipStat, Prep, FlightPlan, Waypoint, PrisonerHold, CdTable, …). A("tag", type) = on-disk tag confirmed in the exe (strict name match); R("name", type) = positional ("."-tagged items, or R1 C# names). Binary corrections applied over R1/R2:

  • float: TRM CstR CstE CstT shrm RefCap RepCap, PlayerView Infra; Summary.IncMod/ResMod, Sim.IncMod/ResMod, Sim.szadj/rsadj/suadj, TMRS.TSTL/TCTL/TQTL/TQTLE, CreateParams.SDist/SSize/SRes/SSuit/REnc/IncM/ResM, ISsu
  • int64: Bats2 rcex PopC, PlayerTurnStats.pop
  • bool: Abdn Dstyd PvMA AIBn haltv EnAl EnTm (R2's "short" readings are the value byte)
  • int on disk though int16 in memory: TShn ETS tch col bat and all DiplomacyStats counters
  • string: pswd, Key, MapF, Pwd, KeyPath (all may be empty)
  • Vector3 (3 × "." float): Pos PrvPos FtOrig FPogn2 FPdpos Health
  • Nexp entries carry xid xmin xmax xper(float)
  • Team appears twice in Player: an int, later a nested {ALid AL NA CF} frame (typed key Alliances)
  • GOWinPly is a framed VectorHelper<uint> ("." count + n × "." int), empty in the real saves
  • CreateParams/Summary tag case is as written above (R1's camel-case names were wrong)

8. Reader output conventions

  • --dump: one line per item, @<inflated offset> name kind value; ? after the kind means the type was guessed, (alt …) shows the other reading of a 4-byte word; frames print { … } with item count and byte size.
  • --json: {"padding", "stats", "issues": [...], "data": {...}}; data has the keys summary, createParams, sim, cdTable, customData; every typed struct carries _off; unexpected items are kept under _unexpected / _extra; unknown regions keep the generic form.
  • Issue levels: error (schema field missing / type impossible / frame unterminated), warn (resync, hint not plausible, unexpected items, width mismatch, best-effort read at EOF), info (positional "." tag read as an R() name, unnamed small payload before END). --strict fails on error or warn.
  • Exit status: 0 clean, 1 errors present, 2 unreadable container or strict failure.

9. Verification checklist (all settled on the real saves)

  1. Padding convention — joint (§2).
  2. Summary frame: 13 children with the tags in §6 (R1's names were case-variants).
  3. Vector3 bodies — tagged, 3 × "." float (§3).
  4. Element and count tags inside framed arrays — "." (§3, §4).
  5. --padding auto picks joint; resyncs == 0 and hint_failures == 0 on turn1/2/3; the only raw bytes are the RNG blob. Remaining unknowns are the bodies of the opaque frames listed in §6 (kept generic, not a parsing gap).