# SOTS1 save format — as implemented by `save_reader.py`

Status: **validated against the three real saves** (`verify/results/saves/turn1..3-state.sav`,
SotS 1.8): `save_reader.py <save> --strict` exits 0 on all three with `resyncs: 0`,
`hint-failures: 0` and 2503 raw bytes (the opaque `RNG` blob). Sources: the community
editors (R1 Bardez, R2 SOTSedit; `save-editor-structs.md`), the binary member tables
(`struct-recovery.md`) and the serializer decompiles in
`findings/objects/schema-gaps-resolved.md` (round 3), which settled every item that was
marked VERIFY in the previous revision of this file. The synthetic fixture
(`save_writer_stub.py`) round-trips under the same assumptions (`test_save_reader.py`).
The C++ typed writer (which reproduces the real saves byte-identically) later exposed a
handful of tag names this file had wrong or left unnamed; they are corrected throughout
and listed in §10.

## 1. Container

| | |
|---|---|
| File | one gzip member (`1f 8b`), standard deflate. `gzip.decompress` |
| Inflated stream | parsed from offset 0; all offsets in reader output are inflated offsets |
| Byte order | little-endian everywhere |
| Text | windows-1252, no NUL terminator |
| Non-gzip input | accepted as an already-inflated stream (R1's `*.sav.inflate.dat`) |

## 2. Primitive encodings

```
tag       := int32 len, len bytes ASCII                (field name; "." when the writer passed NULL)
string    := int32 len, len bytes cp1252               len 0 is legal (Key, MapF, Pwd, Scenario, KeyPath)
int       := 4 bytes (int32)                           handle ids, int16 members widened, enums, counts
float     := 4 bytes IEEE-754 single
bool      := 1 byte, 0/1
int64     := 8 bytes                                   Bats2, rcex, PopC, pop (raw-bytes write path)
raw       := n bytes, length known only to the writer  RNG state (2503 B incl. pad = MT19937 624 words + index)
```

A **named value** is `tag value pad`, where `pad` is NUL bytes bringing the item to a
4-byte boundary. **Padding is joint** — confirmed on the real saves: `pad4(4 + len(name)
+ len(value))` over the whole item (`haltv` bool = 12 B, `vnh` bool = 8 B, `Name` =
4+4+4+len rounded). The "split" convention (pad after the name and after the value) is
kept only so `--padding auto` can reject it; it never matched a real file.

| mode | layout | item size |
|---|---|---|
| `joint` (real files) | `[len][name][value][pad]` | `pad4(4 + len(name) + len(value))` |
| `split` (not observed) | `[len][name][pad][value][pad]` | `pad4(4 + len(name)) + pad4(len(value))` |

Values carry **no type byte**. Type comes from (a) the schema/catalog, (b) a lookahead
plausibility test (a candidate layout is accepted only if a plausible tag, a frame marker,
or EOF follows), (c) for 4-byte words with no hint, an int/float classification by bit
pattern (`|int| <= 100000` → int; finite float with `1e-6 <= |f| < 1e12` → float; else
int). The other reading is kept as `alt`. Limitation: with a name whose length is a
multiple of 4, `bool` and `int` have identical item sizes; unknown names default to `int`.

Two rules about **string values** (both were reader bugs before round 3):

* Only **tag** bytes are constrained to printable ASCII. A string **value** may hold any
  windows-1252 byte, including 0x80–0x9f (system names such as `Kor’Voth` carry 0x92 =
  right single quote). A tag the catalog knows to be a string is read without any text
  test; a value can therefore never veto the layout of the item before it (that veto was
  the `haltv` → `VFlags`/`Name`/`vnh` cascade). Guessing a string for an *unknown* tag
  still uses a text-plausibility guard, which accepts every cp1252-defined byte.
* An **empty string is 4 zero bytes**, byte-identical to int 0. The walker reads it as
  `""` when the tag is hinted; when an unhinted walker read a 4-zero-byte word, the
  schema applier coerces it to `""` instead of reporting "expected string, read int".

## 3. Complex frames

```
[len][name][pad to 4]  BE EF BE EF   ...children...   10 41 10 41
                       (0xBEEFBEEF)                   (0x41104110 = ~0xBEEFBEEF)
```

* Every "nested object" (IStreamable via stream vft+0x28) is framed. Frames nest; the
  reader recovers the tree from the markers alone, without a schema.
* **`"."` convention (confirmed):** any write with a NULL name is emitted with the tag
  `"."` (len 1). That covers every `VectorHelper<T>` array (`"."` = count, then n × `"."`
  elements: frames for streamable T, plain values for POD T), the scalars of
  `StarMapParams` and `StrategyPlayerGameSettings`, `PlayerColorID`'s index/r/g/b, and
  `Vector3` bodies. The reader matches such items positionally (`R()` fields) and reports
  each one as an *info* ("tag '.' read positionally as …"); `"."` is deliberately kept out
  of the global type catalog because it carries ints, floats and frames alike.
* **Vector3 bodies are tagged**: 3 × `"."` float items inside the frame (`Pos`, `PrvPos`,
  `FtOrig`, `FPogn2`, `FPdpos`, `Health`, `MapP` planet positions; writer `FUN_008a60d0`).
  The reader keeps the 12-raw-byte fallback but it is not exercised by real files.
* A frame may be **tagless** (BEEFBEEF directly at an item position): accepted by the
  reader; not observed in the real saves (element frames use `"."`).
* `RNG` frame body is one `"."`-tagged opaque blob read straight to its END marker
  (2503 B including the 3 joint-padding bytes).
* Markers are only interpreted at item boundaries; a value that happens to equal a marker
  is not a problem unless the reader is already resynchronising.

## 4. Arrays

| kind | encoding | reader schema |
|---|---|---|
| NonComplexArray (inline count) | named int count, then count × element **inline in the same frame** | `NArr` |
| ComplexArray (`VectorHelper<T>`) | a frame containing: `"."` int count, then count × `"."` element | `CArr` |
| uncounted repetition | elements repeated until the frame END (or until the lead tag stops recurring) | `Repeat` |
| element = leaf wrapper | `SysID` int + `Sys` frame; `PlayerID` + `Player`; `FltID` + `Flt`; `ShipID` + `Ship`; `DesID` + `Des`; `ply` + `hist` | `Seq([...])` |
| sparse tables | `mnsp` then n × (`msp`, `mv`); `nadct` then n × (`ads`, `adt`); `haltc` then n × (`haltt`, `haltv`); `zdsc` then n × (`zdsi`, `zdst`); `PrNSp` then n × (`PrSp`, `PrNum`) | `NArr(Seq)` |

Inline-count tags used by the binary: `NumPlrs NumSys NumFlts NShips NumActs NumOwn NumDes
NumLeg NumNotes NumPR NSprj Nexp NWeapXcl ndeflay rdtc numcreps ninv NumFlts NumGFs NumSnF
NumMnF NVO NVE NVs NumPlgs2 PopNG mnsp nadct haltc PrNSp NTH nply ncls seno2 spc zdsc NumIDs
PlayerIDs DesignIDs SystemIDs FleetIDs ShipIDs TradeIDs AllExc AllExcCF`. Count and element
tags inside framed arrays (`Players`, `GOWinPly`, `sacq`, `slost`, `dipstats`, `preps`,
`odes`, `owep`, `otch`, `cme2`, `Ojvs`, `ords`, `wpts`, `paths`, `MapP` planets …) are
always `"."` (§3). `ords` (BuildQueue), `wpts` (FlightPlan) and `paths` (NdGr2) are real
framed-array tags, not positional items (§10).

Uncounted lists: `hist` holds n × `stats` frames until its END; `Sim` holds 7 × (`ISsp`
string, `ISsu` float) after the players; the root holds one `CD` frame per custom-data id.

## 5. Conditionals the reader honours

| gate | consequence |
|---|---|
| `ClrID`/`indcl`/`FxCrID` frame: `"."` int index == -1 | three `"."` ints r,g,b follow inside the frame |
| `vnh` true | `vnd`, `vnex3`, `vnpex3` (the only conditional in that region of `Sys`; `Name` and `VFlags` are unconditional) |
| system-level `hindi` true | `indi` frame (IndependenceInfo) after `NVs` |
| `NVO` entry | `indi` frame is **always** present after `isind` (inline member of the map value; `isind` does not gate it) |
| `PrMax > 0` | `PrNSp` + n × (`PrSp`, `PrNum`); with `PrMax == 0` the `PrisH` frame ends right after `PrMax` |
| `HFPlan` true | `FPlan` frame: `wpts{"." n, n×"."{Wpt Tp nrt{nrp nrf nrt}}} FPsp2 FPeta2 FPogn2 FPdpos pnd` (turn3 only) |
| `HLay` true | `Lay` frame (opaque) |
| `hbq` true | `BQ2` frame (same body as `BQ`: `ords{"." n, n×"."{desID con conleft sav ordID}}`); `hsp` true → `pop`, `ppop` Population frames |
| `HasAIR` true | `AIR` frame (opaque) |
| system `PID` non-null | `BQ` frame present (reader: optional by name) |
| `SvSctOb` pointer non-NULL | `SvSctOb` frame before `zdsc` (present in the three real saves; `Opt`) |
| custom-data blob non-NULL | one `CD` frame per `CDT` id that has data |
| legacy tags `ISuit ARes SysID TrdID Caps GtTrf FtSens FtInc lcid SensMod ExPopSys AIDifficultyID Rand RandEnc NPC` | accepted if present (read-side only), never written by 1.8 |

## 6. Top-level layout

Written by `FUN_00877070`; loader `FUN_0086abb0`; summary-only reader `FUN_008773c0`.

```
offset 0:  "Summary"       StrategyGameInfo         GameName Turn NumSys Checksum Players{"."=n, n×"."{Slot{} Rank}}
                                                    Session{TMRS{TSTL TCTL TQTL TQTLE}} MapShape IncMod ResMod
                                                    Alliances Teams Encounters Scenario
           "CreateParams"  StrategyGameCreateParams Name ID RSeed AID Key MapP{"." int, "."{planets}, "." n×"."{ints},
                                                    "."{nodePaths}} MapS MapF NSys REnc SDist SSize SRes SSuit MaxP ASpec
                                                    bAlly NTeam tmgrp PSav PCol PTech IncM ResM scrp{spc n×(spsn sppn sppv)}
           "Sim"           StrategyServer           KeyPath NMSz NMLc NMnx | PlayerIDs DesignIDs SystemIDs FleetIDs
                                                    ShipIDs TradeIDs (each: count, n×".") | ModCount Frame GameID Attrib{}
                                                    RNG{} GameName Map IncMod ResMod EnAl EnTm GOTurn GOWinPly{} NPCm NPCo
                                                    NPCi NPCv NPCa szadj rsadj suadj sprjs{} RandEncAdj cmbtid turnstats{}
                                                    numcreps n×crep{} ninv n×(invs inve invt invtb) AllExc n×(AllExc AllExc)
                                                    AllExc n×(AllExc AllExc) AllExcCF n×(AllExcCFp AllExcCFp)
                                                    NumPlrs n×(PlayerID Player{}) 7×(ISsp ISsu) NumSys n×(SysID Sys{})
                                                    NdGr2{} trdmgr{} spymgr{} NumFlts n×(FltID Flt{}) NumActs n×Act
                                                    [SvSctOb{}] zdsc n×(zdsi zdst)
           "CDT"           CustomDataTable          NumIDs n×ID (strings: Player.<id>.TurnCommands_v5, Player.<id>.AIAgent …)
           n × "CD"        opaque custom-data frames, one per id that has a blob
```

`Slot` (SlotDef): `IsPlay IsDead IsReq IsRec IsFxNm FxNm IsFxSp FxSp IsFxCr FxCrID{} IsFxBd FxBd
IsFxAv FxAv Tag Pwd(string) Team Settings{4×"." int}`.
`turnstats` (GameTurnHistory): `nply`, n × (`ply`, `hist{ply, stats{} …until END}`);
`stats` (PlayerTurnStats): `pop`(int64) `sacq{}` `slost{}` (VectorHelper<SystemEvent>: `set ses seop
senp seno2 n×seot2`) `trn almem inc tdinc sav col bat tch`(int) `ncls` 3 × (`cls shpt shpl shpk satt
satl satk`).
`Des` (design header, under `NumDes`/`NumLeg` n × (`DesID`, `Des{}`)): `FAIDes`(bool) `DHide`(bool)
`DWep`(int) `DName`(string), then the `DSec` section frames (kept generic).
Objective records (Player `odes`/`owep`/`otch`, each a framed VectorHelper of `"."` elements):
`odes` = `otnF otnL odid opid`; `owep` = `otnF otnL odet owep`(string) `owith`; `otch` = `otnF otnL odet
otch`(string) `owith`.
`NdGr2` (node grid): `paths{"." n, n×"."{npt npid npfr npto npctm npcby npdtn npdtf npenp npuse nptf}}`
then `nextid`(int).
`BQ` (system build queue): `ords{"." n, n×"."{desID con conleft sav ordID}}` — count 0 in turn1..3.

Everything not covered by a shape (TechTree body, Events, ShipRecs, spy2, civr, comms, Ojvs,
Attrib, sprjs, SvSctOb, trdmgr, spymgr, CD, …) is kept as the generic
`{"_name","_off","_items":[{"name","kind","value","off"}...]}` form.

## 7. Struct field orders and types applied

Shapes live in `save_reader.py` (`Summary`, `Slot`, `CreateParams`, `MapP`, `Sim`,
`TurnStats`/`PlayerTurnHistory`/`PlayerTurnStats`, `Sys`, `Player`, `Fleet`, `Ship`,
`PlayerView`, `Population`/`PopG`, `Morale`, `MoraleEvent`, `BuildQueue`/`BuildOrder`,
`IndependenceInfo`, `Rts`, `DipStat`, `Prep`, `FlightPlan`, `Waypoint`, `PrisonerHold`,
`CdTable`, …). `A("tag", type)` = on-disk tag confirmed in the exe (strict name match);
`R("name", type)` = positional (`"."`-tagged items, or R1 C# names). Binary corrections
applied over R1/R2:

* float: `TRM CstR CstE CstT shrm RefCap RepCap`, PlayerView `Infra`; `Summary.IncMod/ResMod`,
  `Sim.IncMod/ResMod`, `Sim.szadj/rsadj/suadj`, `TMRS.TSTL/TCTL/TQTL/TQTLE`,
  `CreateParams.SDist/SSize/SRes/SSuit/REnc/IncM/ResM`, `ISsu`
* int64: `Bats2 rcex PopC`, `PlayerTurnStats.pop`
* bool: `Abdn Dstyd PvMA AIBn haltv EnAl EnTm` (R2's "short" readings are the value byte)
* int on disk though int16 in memory: `TShn ETS tch col bat` and all DiplomacyStats counters
* string: `pswd`, `Key`, `MapF`, `Pwd`, `KeyPath` (all may be empty)
* Vector3 (3 × `"."` float): `Pos PrvPos FtOrig FPogn2 FPdpos Health`
* `Nexp` entries carry `xid xmin xmax xper(float)`
* `Team` appears twice in Player: an int, later a nested `{ALid AL NA CF}` frame (typed key `Alliances`)
* `GOWinPly` is a framed `VectorHelper<uint>` (`"."` count + n × `"."` int), empty in the real saves
* `CreateParams`/`Summary` tag case is as written above (R1's camel-case names were wrong)
* Objective lead tag `otnF`, node-grid `nextid`, design header `FAIDes DHide DWep DName`, and the
  framed arrays `ords`/`wpts`/`paths` are `A()` (byte-confirmed, §10); the reader keeps R1's
  keys `faiDes dHide dWep dName` in its typed output for `stock_designs.py`

## 8. Reader output conventions

* `--dump`: one line per item, `@<inflated offset> name kind value`; `?` after the kind
  means the type was guessed, `(alt …)` shows the other reading of a 4-byte word; frames
  print `{` … `}` with item count and byte size.
* `--json`: `{"padding", "stats", "issues": [...], "data": {...}}`; `data` has the keys
  `summary`, `createParams`, `sim`, `cdTable`, `customData`; every typed struct carries
  `_off`; unexpected items are kept under `_unexpected` / `_extra`; unknown regions keep
  the generic form.
* Issue levels: `error` (schema field missing / type impossible / frame unterminated),
  `warn` (resync, hint not plausible, unexpected items, width mismatch, best-effort read
  at EOF), `info` (positional `"."` tag read as an R() name, unnamed small payload before
  END). `--strict` fails on error or warn.
* Exit status: 0 clean, 1 errors present, 2 unreadable container or strict failure.

## 9. Verification checklist (all settled on the real saves)

1. Padding convention — **joint** (§2).
2. `Summary` frame: 13 children with the tags in §6 (R1's names were case-variants).
3. Vector3 bodies — **tagged**, 3 × `"."` float (§3).
4. Element and count tags inside framed arrays — **`"."`** (§3, §4).
5. `--padding auto` picks joint; `resyncs == 0` and `hint_failures == 0` on turn1/2/3;
   the only raw bytes are the `RNG` blob. Remaining unknowns are the bodies of the
   opaque frames listed in §6 (kept generic, not a parsing gap).
6. Every tag the reader matches by name is spelled as on disk (§10); the only positional
   (`R()`) items left are NULL-named `"."` items and R1-named fields inside `crep`,
   `NdGr2.paths` elements, `MapP` and `Waypoint.nrt`.

## 10. Tag-name corrections (changelog)

Found by the C++ typed save writer, which has to emit the real tag strings to round-trip
the saves byte-identically. The Python reader had matched these fields **positionally**
(`R()`), so a wrong spelling in the spec never failed a parse — the walker reported each as
an *info* ("tag 'otnF' read positionally as 'ontF'") and the typed output silently used the
wrong name. Each entry below was re-verified by inflating the saves and searching the
byte stream for `[int32 len][ascii]` (offsets are inflated offsets, decimal).

| was (spec / R1) | is (on disk) | where | evidence |
|---|---|---|---|
| `ontF` | **`otnF`** | first field of every `odes`/`owep`/`otch` element | `otnF` ×62 in turn2 (first @72288, in `otch`; @123344 region `odes` element `otnF otnL odid opid`); ×60 turn1, ×62 turn3. `ontF` ×0 in all three |
| `nextId` | **`nextid`** | last item of `NdGr2` | `nextid` ×1: turn1 @564432, turn2 @575500, turn3 @580152 (value 44 in turn2, after `paths`). `nextId` ×0 |
| `faiDes dHide dWep dName` | **`FAIDes DHide DWep DName`** | first four items of every `Des` frame | ×43 each in turn2 (first Des frame @67820: `FAIDes` @67832, `DHide` @67844, `DWep` @67856, `DName` @67868); ×41 turn1, ×43 turn3. Camel-case forms ×0 |
| unnamed/positional | **`ords`** | framed VectorHelper inside `BQ` | ×3 in each save (turn2 @531144, 548092, 550044; each `BQ` @−12); count 0 in all |
| unnamed/positional | **`wpts`** | framed VectorHelper inside `FPlan` | turn3 only (the only save with `HFPlan` = 1): `FPlan` @591476, `wpts` @591492, element `Wpt` @591528 `Tp` @591540 `nrt` @591552. Absent from turn1/turn2 because no fleet has a flight plan there (`HFPlan` = 0 ×6 / ×7) |
| positional | **`paths`** | framed VectorHelper inside `NdGr2` | ×1 per save (turn2 @568072, `NdGr2` @568056); spelled as R1 had it but now matched by name |

Reader changes (`save_reader.py`): the fields above are `A()`; `ords`/`wpts`/`paths` lost
the `flex` inline hedge (the game always frames them); the never-occurring `dName`/`faiDes`/
`dHide` entries were dropped from `MANUAL_KINDS`. Typed-output keys are unchanged
(`faiDes dHide dWep dName` via `key=`; `otnF`, `nextid`, `ords`, `wpts`, `paths` were never
consumed under the old names). Effect on the real saves: strict still exits 0 with 0
resyncs / 0 hint-failures; the info count drops from 257/259/259 to 197 on turn1/2/3 — the
60/62/62 `otnF` "read positionally" notes are gone, and every remaining info is a `"."`
item. `test_save_reader.py::TagNameCorrectionsTest` pins each correction and fails a
frame that uses the old spelling.