209 lines
14 KiB
Markdown
209 lines
14 KiB
Markdown
# SOTS1 save format — as implemented by `save_reader.py`
|
||
|
||
Status: **validated against the three real saves** (`verify/results/saves/turn1..3-state.sav`,
|
||
SotS 1.8): `save_reader.py <save> --strict` exits 0 on all three with `resyncs: 0`,
|
||
`hint-failures: 0` and 2503 raw bytes (the opaque `RNG` blob). Sources: the community
|
||
editors (R1 Bardez, R2 SOTSedit; `save-editor-structs.md`), the binary member tables
|
||
(`struct-recovery.md`) and the serializer decompiles in
|
||
`findings/objects/schema-gaps-resolved.md` (round 3), which settled every item that was
|
||
marked VERIFY in the previous revision of this file. The synthetic fixture
|
||
(`save_writer_stub.py`) round-trips under the same assumptions (`test_save_reader.py`).
|
||
|
||
## 1. Container
|
||
|
||
| | |
|
||
|---|---|
|
||
| File | one gzip member (`1f 8b`), standard deflate. `gzip.decompress` |
|
||
| Inflated stream | parsed from offset 0; all offsets in reader output are inflated offsets |
|
||
| Byte order | little-endian everywhere |
|
||
| Text | windows-1252, no NUL terminator |
|
||
| Non-gzip input | accepted as an already-inflated stream (R1's `*.sav.inflate.dat`) |
|
||
|
||
## 2. Primitive encodings
|
||
|
||
```
|
||
tag := int32 len, len bytes ASCII (field name; "." when the writer passed NULL)
|
||
string := int32 len, len bytes cp1252 len 0 is legal (Key, MapF, Pwd, Scenario, KeyPath)
|
||
int := 4 bytes (int32) handle ids, int16 members widened, enums, counts
|
||
float := 4 bytes IEEE-754 single
|
||
bool := 1 byte, 0/1
|
||
int64 := 8 bytes Bats2, rcex, PopC, pop (raw-bytes write path)
|
||
raw := n bytes, length known only to the writer RNG state (2503 B incl. pad = MT19937 624 words + index)
|
||
```
|
||
|
||
A **named value** is `tag value pad`, where `pad` is NUL bytes bringing the item to a
|
||
4-byte boundary. **Padding is joint** — confirmed on the real saves: `pad4(4 + len(name)
|
||
+ len(value))` over the whole item (`haltv` bool = 12 B, `vnh` bool = 8 B, `Name` =
|
||
4+4+4+len rounded). The "split" convention (pad after the name and after the value) is
|
||
kept only so `--padding auto` can reject it; it never matched a real file.
|
||
|
||
| mode | layout | item size |
|
||
|---|---|---|
|
||
| `joint` (real files) | `[len][name][value][pad]` | `pad4(4 + len(name) + len(value))` |
|
||
| `split` (not observed) | `[len][name][pad][value][pad]` | `pad4(4 + len(name)) + pad4(len(value))` |
|
||
|
||
Values carry **no type byte**. Type comes from (a) the schema/catalog, (b) a lookahead
|
||
plausibility test (a candidate layout is accepted only if a plausible tag, a frame marker,
|
||
or EOF follows), (c) for 4-byte words with no hint, an int/float classification by bit
|
||
pattern (`|int| <= 100000` → int; finite float with `1e-6 <= |f| < 1e12` → float; else
|
||
int). The other reading is kept as `alt`. Limitation: with a name whose length is a
|
||
multiple of 4, `bool` and `int` have identical item sizes; unknown names default to `int`.
|
||
|
||
Two rules about **string values** (both were reader bugs before round 3):
|
||
|
||
* Only **tag** bytes are constrained to printable ASCII. A string **value** may hold any
|
||
windows-1252 byte, including 0x80–0x9f (system names such as `Kor’Voth` carry 0x92 =
|
||
right single quote). A tag the catalog knows to be a string is read without any text
|
||
test; a value can therefore never veto the layout of the item before it (that veto was
|
||
the `haltv` → `VFlags`/`Name`/`vnh` cascade). Guessing a string for an *unknown* tag
|
||
still uses a text-plausibility guard, which accepts every cp1252-defined byte.
|
||
* An **empty string is 4 zero bytes**, byte-identical to int 0. The walker reads it as
|
||
`""` when the tag is hinted; when an unhinted walker read a 4-zero-byte word, the
|
||
schema applier coerces it to `""` instead of reporting "expected string, read int".
|
||
|
||
## 3. Complex frames
|
||
|
||
```
|
||
[len][name][pad to 4] BE EF BE EF ...children... 10 41 10 41
|
||
(0xBEEFBEEF) (0x41104110 = ~0xBEEFBEEF)
|
||
```
|
||
|
||
* Every "nested object" (IStreamable via stream vft+0x28) is framed. Frames nest; the
|
||
reader recovers the tree from the markers alone, without a schema.
|
||
* **`"."` convention (confirmed):** any write with a NULL name is emitted with the tag
|
||
`"."` (len 1). That covers every `VectorHelper<T>` array (`"."` = count, then n × `"."`
|
||
elements: frames for streamable T, plain values for POD T), the scalars of
|
||
`StarMapParams` and `StrategyPlayerGameSettings`, `PlayerColorID`'s index/r/g/b, and
|
||
`Vector3` bodies. The reader matches such items positionally (`R()` fields) and reports
|
||
each one as an *info* ("tag '.' read positionally as …"); `"."` is deliberately kept out
|
||
of the global type catalog because it carries ints, floats and frames alike.
|
||
* **Vector3 bodies are tagged**: 3 × `"."` float items inside the frame (`Pos`, `PrvPos`,
|
||
`FtOrig`, `FPogn2`, `FPdpos`, `Health`, `MapP` planet positions; writer `FUN_008a60d0`).
|
||
The reader keeps the 12-raw-byte fallback but it is not exercised by real files.
|
||
* A frame may be **tagless** (BEEFBEEF directly at an item position): accepted by the
|
||
reader; not observed in the real saves (element frames use `"."`).
|
||
* `RNG` frame body is one `"."`-tagged opaque blob read straight to its END marker
|
||
(2503 B including the 3 joint-padding bytes).
|
||
* Markers are only interpreted at item boundaries; a value that happens to equal a marker
|
||
is not a problem unless the reader is already resynchronising.
|
||
|
||
## 4. Arrays
|
||
|
||
| kind | encoding | reader schema |
|
||
|---|---|---|
|
||
| NonComplexArray (inline count) | named int count, then count × element **inline in the same frame** | `NArr` |
|
||
| ComplexArray (`VectorHelper<T>`) | a frame containing: `"."` int count, then count × `"."` element | `CArr` |
|
||
| uncounted repetition | elements repeated until the frame END (or until the lead tag stops recurring) | `Repeat` |
|
||
| element = leaf wrapper | `SysID` int + `Sys` frame; `PlayerID` + `Player`; `FltID` + `Flt`; `ShipID` + `Ship`; `DesID` + `Des`; `ply` + `hist` | `Seq([...])` |
|
||
| sparse tables | `mnsp` then n × (`msp`, `mv`); `nadct` then n × (`ads`, `adt`); `haltc` then n × (`haltt`, `haltv`); `zdsc` then n × (`zdsi`, `zdst`); `PrNSp` then n × (`PrSp`, `PrNum`) | `NArr(Seq)` |
|
||
|
||
Inline-count tags used by the binary: `NumPlrs NumSys NumFlts NShips NumActs NumOwn NumDes
|
||
NumLeg NumNotes NumPR NSprj Nexp NWeapXcl ndeflay rdtc numcreps ninv NumFlts NumGFs NumSnF
|
||
NumMnF NVO NVE NVs NumPlgs2 PopNG mnsp nadct haltc PrNSp NTH nply ncls seno2 spc zdsc NumIDs
|
||
PlayerIDs DesignIDs SystemIDs FleetIDs ShipIDs TradeIDs AllExc AllExcCF`. Count and element
|
||
tags inside framed arrays (`Players`, `GOWinPly`, `sacq`, `slost`, `dipstats`, `preps`,
|
||
`odes`, `owep`, `otch`, `cme2`, `Ojvs`, `MapP` planets …) are always `"."` (§3).
|
||
|
||
Uncounted lists: `hist` holds n × `stats` frames until its END; `Sim` holds 7 × (`ISsp`
|
||
string, `ISsu` float) after the players; the root holds one `CD` frame per custom-data id.
|
||
|
||
## 5. Conditionals the reader honours
|
||
|
||
| gate | consequence |
|
||
|---|---|
|
||
| `ClrID`/`indcl`/`FxCrID` frame: `"."` int index == -1 | three `"."` ints r,g,b follow inside the frame |
|
||
| `vnh` true | `vnd`, `vnex3`, `vnpex3` (the only conditional in that region of `Sys`; `Name` and `VFlags` are unconditional) |
|
||
| system-level `hindi` true | `indi` frame (IndependenceInfo) after `NVs` |
|
||
| `NVO` entry | `indi` frame is **always** present after `isind` (inline member of the map value; `isind` does not gate it) |
|
||
| `PrMax > 0` | `PrNSp` + n × (`PrSp`, `PrNum`); with `PrMax == 0` the `PrisH` frame ends right after `PrMax` |
|
||
| `HFPlan` true | `FPlan` frame |
|
||
| `HLay` true | `Lay` frame (opaque) |
|
||
| `hbq` true | `BQ2` frame; `hsp` true → `pop`, `ppop` Population frames |
|
||
| `HasAIR` true | `AIR` frame (opaque) |
|
||
| system `PID` non-null | `BQ` frame present (reader: optional by name) |
|
||
| `SvSctOb` pointer non-NULL | `SvSctOb` frame before `zdsc` (present in the three real saves; `Opt`) |
|
||
| custom-data blob non-NULL | one `CD` frame per `CDT` id that has data |
|
||
| legacy tags `ISuit ARes SysID TrdID Caps GtTrf FtSens FtInc lcid SensMod ExPopSys AIDifficultyID Rand RandEnc NPC` | accepted if present (read-side only), never written by 1.8 |
|
||
|
||
## 6. Top-level layout
|
||
|
||
Written by `FUN_00877070`; loader `FUN_0086abb0`; summary-only reader `FUN_008773c0`.
|
||
|
||
```
|
||
offset 0: "Summary" StrategyGameInfo GameName Turn NumSys Checksum Players{"."=n, n×"."{Slot{} Rank}}
|
||
Session{TMRS{TSTL TCTL TQTL TQTLE}} MapShape IncMod ResMod
|
||
Alliances Teams Encounters Scenario
|
||
"CreateParams" StrategyGameCreateParams Name ID RSeed AID Key MapP{"." int, "."{planets}, "." n×"."{ints},
|
||
"."{nodePaths}} MapS MapF NSys REnc SDist SSize SRes SSuit MaxP ASpec
|
||
bAlly NTeam tmgrp PSav PCol PTech IncM ResM scrp{spc n×(spsn sppn sppv)}
|
||
"Sim" StrategyServer KeyPath NMSz NMLc NMnx | PlayerIDs DesignIDs SystemIDs FleetIDs
|
||
ShipIDs TradeIDs (each: count, n×".") | ModCount Frame GameID Attrib{}
|
||
RNG{} GameName Map IncMod ResMod EnAl EnTm GOTurn GOWinPly{} NPCm NPCo
|
||
NPCi NPCv NPCa szadj rsadj suadj sprjs{} RandEncAdj cmbtid turnstats{}
|
||
numcreps n×crep{} ninv n×(invs inve invt invtb) AllExc n×(AllExc AllExc)
|
||
AllExc n×(AllExc AllExc) AllExcCF n×(AllExcCFp AllExcCFp)
|
||
NumPlrs n×(PlayerID Player{}) 7×(ISsp ISsu) NumSys n×(SysID Sys{})
|
||
NdGr2{} trdmgr{} spymgr{} NumFlts n×(FltID Flt{}) NumActs n×Act
|
||
[SvSctOb{}] zdsc n×(zdsi zdst)
|
||
"CDT" CustomDataTable NumIDs n×ID (strings: Player.<id>.TurnCommands_v5, Player.<id>.AIAgent …)
|
||
n × "CD" opaque custom-data frames, one per id that has a blob
|
||
```
|
||
|
||
`Slot` (SlotDef): `IsPlay IsDead IsReq IsRec IsFxNm FxNm IsFxSp FxSp IsFxCr FxCrID{} IsFxBd FxBd
|
||
IsFxAv FxAv Tag Pwd(string) Team Settings{4×"." int}`.
|
||
`turnstats` (GameTurnHistory): `nply`, n × (`ply`, `hist{ply, stats{} …until END}`);
|
||
`stats` (PlayerTurnStats): `pop`(int64) `sacq{}` `slost{}` (VectorHelper<SystemEvent>: `set ses seop
|
||
senp seno2 n×seot2`) `trn almem inc tdinc sav col bat tch`(int) `ncls` 3 × (`cls shpt shpl shpk satt
|
||
satl satk`).
|
||
|
||
Everything not covered by a shape (TechTree body, Events, ShipRecs, spy2, civr, comms, Ojvs,
|
||
Attrib, sprjs, SvSctOb, trdmgr, spymgr, CD, …) is kept as the generic
|
||
`{"_name","_off","_items":[{"name","kind","value","off"}...]}` form.
|
||
|
||
## 7. Struct field orders and types applied
|
||
|
||
Shapes live in `save_reader.py` (`Summary`, `Slot`, `CreateParams`, `MapP`, `Sim`,
|
||
`TurnStats`/`PlayerTurnHistory`/`PlayerTurnStats`, `Sys`, `Player`, `Fleet`, `Ship`,
|
||
`PlayerView`, `Population`/`PopG`, `Morale`, `MoraleEvent`, `BuildQueue`/`BuildOrder`,
|
||
`IndependenceInfo`, `Rts`, `DipStat`, `Prep`, `FlightPlan`, `Waypoint`, `PrisonerHold`,
|
||
`CdTable`, …). `A("tag", type)` = on-disk tag confirmed in the exe (strict name match);
|
||
`R("name", type)` = positional (`"."`-tagged items, or R1 C# names). Binary corrections
|
||
applied over R1/R2:
|
||
|
||
* float: `TRM CstR CstE CstT shrm RefCap RepCap`, PlayerView `Infra`; `Summary.IncMod/ResMod`,
|
||
`Sim.IncMod/ResMod`, `Sim.szadj/rsadj/suadj`, `TMRS.TSTL/TCTL/TQTL/TQTLE`,
|
||
`CreateParams.SDist/SSize/SRes/SSuit/REnc/IncM/ResM`, `ISsu`
|
||
* int64: `Bats2 rcex PopC`, `PlayerTurnStats.pop`
|
||
* bool: `Abdn Dstyd PvMA AIBn haltv EnAl EnTm` (R2's "short" readings are the value byte)
|
||
* int on disk though int16 in memory: `TShn ETS tch col bat` and all DiplomacyStats counters
|
||
* string: `pswd`, `Key`, `MapF`, `Pwd`, `KeyPath` (all may be empty)
|
||
* Vector3 (3 × `"."` float): `Pos PrvPos FtOrig FPogn2 FPdpos Health`
|
||
* `Nexp` entries carry `xid xmin xmax xper(float)`
|
||
* `Team` appears twice in Player: an int, later a nested `{ALid AL NA CF}` frame (typed key `Alliances`)
|
||
* `GOWinPly` is a framed `VectorHelper<uint>` (`"."` count + n × `"."` int), empty in the real saves
|
||
* `CreateParams`/`Summary` tag case is as written above (R1's camel-case names were wrong)
|
||
|
||
## 8. Reader output conventions
|
||
|
||
* `--dump`: one line per item, `@<inflated offset> name kind value`; `?` after the kind
|
||
means the type was guessed, `(alt …)` shows the other reading of a 4-byte word; frames
|
||
print `{` … `}` with item count and byte size.
|
||
* `--json`: `{"padding", "stats", "issues": [...], "data": {...}}`; `data` has the keys
|
||
`summary`, `createParams`, `sim`, `cdTable`, `customData`; every typed struct carries
|
||
`_off`; unexpected items are kept under `_unexpected` / `_extra`; unknown regions keep
|
||
the generic form.
|
||
* Issue levels: `error` (schema field missing / type impossible / frame unterminated),
|
||
`warn` (resync, hint not plausible, unexpected items, width mismatch, best-effort read
|
||
at EOF), `info` (positional `"."` tag read as an R() name, unnamed small payload before
|
||
END). `--strict` fails on error or warn.
|
||
* Exit status: 0 clean, 1 errors present, 2 unreadable container or strict failure.
|
||
|
||
## 9. Verification checklist (all settled on the real saves)
|
||
|
||
1. Padding convention — **joint** (§2).
|
||
2. `Summary` frame: 13 children with the tags in §6 (R1's names were case-variants).
|
||
3. Vector3 bodies — **tagged**, 3 × `"."` float (§3).
|
||
4. Element and count tags inside framed arrays — **`"."`** (§3, §4).
|
||
5. `--padding auto` picks joint; `resyncs == 0` and `hint_failures == 0` on turn1/2/3;
|
||
the only raw bytes are the `RNG` blob. Remaining unknowns are the bodies of the
|
||
opaque frames listed in §6 (kept generic, not a parsing gap).
|