169 lines
9.1 KiB
Markdown
169 lines
9.1 KiB
Markdown
# SOTS1 save format — as implemented by `save_reader.py`
|
||
|
||
Status: **unvalidated against a real save.** Everything below is what the
|
||
reader *assumes*, derived from the community editors (R1 Bardez, R2 SOTSedit;
|
||
`save-editor-structs.md`) corrected by the binary member tables
|
||
(`struct-recovery.md`). The synthetic fixture (`save_writer_stub.py`) round-trips
|
||
under these assumptions; a real `.sav` is the first real test. Items marked
|
||
**VERIFY** are the ones the verifier should diff against the binary first.
|
||
|
||
## 1. Container
|
||
|
||
| | |
|
||
|---|---|
|
||
| File | one gzip member (`1f 8b`), standard deflate. `gzip.decompress` |
|
||
| Inflated stream | parsed from offset 0; all offsets in reader output are inflated offsets |
|
||
| Byte order | little-endian everywhere |
|
||
| Text | windows-1252, no NUL terminator |
|
||
| Non-gzip input | accepted as an already-inflated stream (R1's `*.sav.inflate.dat`) |
|
||
|
||
## 2. Primitive encodings
|
||
|
||
```
|
||
tag := int32 len, len bytes ASCII (field name; may be "." or "")
|
||
string := int32 len, len bytes cp1252
|
||
int := 4 bytes (int32) handle ids, int16 members, enums, counts
|
||
float := 4 bytes IEEE-754 single
|
||
bool := 1 byte, 0/1
|
||
int64 := 8 bytes Bats2, rcex, PopC (raw-bytes write path)
|
||
raw := n bytes, length known only to the writer RNG state (2500 B = MT19937 624 words + index)
|
||
```
|
||
|
||
A **named value** is `tag value pad`, where `pad` is NUL bytes bringing the item
|
||
to a 4-byte boundary. The reader implements two padding conventions and
|
||
auto-detects which one the file uses (`--padding auto`, default):
|
||
|
||
| mode | layout | item size |
|
||
|---|---|---|
|
||
| `joint` (default, R1's `PaddingSize=4` over `4+len(name)+len(value)`) | `[len][name][value][pad]` | `pad4(4 + len(name) + len(value))` |
|
||
| `split` | `[len][name][pad][value][pad]` | `pad4(4 + len(name)) + pad4(len(value))` |
|
||
|
||
The two differ only when `len(name) % 4 != 0` **and** the value is not a
|
||
multiple of 4 bytes (bools, odd-length strings): e.g. `NPC`+bool is 8 bytes in
|
||
`joint`, 12 in `split`. **VERIFY (highest priority):** find a 3-char bool tag
|
||
(`NPC`, `Dep`, `Atq`, `Srn`, `cta`, `hgs`, `vnh`, `hbq`, `hsp`) in the real
|
||
inflated stream and check whether the value byte immediately follows the name.
|
||
Reader assumption when writing this: `joint`.
|
||
|
||
Values carry **no type byte**. Type comes from (a) the schema/catalog, (b) a
|
||
lookahead plausibility test (a candidate layout is accepted only if a plausible
|
||
tag, a frame marker, or EOF follows), (c) for 4-byte words with no hint, an
|
||
int/float classification by bit pattern (`|int| <= 100000` → int; finite float
|
||
with `1e-6 <= |f| < 1e12` → float; else int). The other reading is kept as `alt`.
|
||
Limitation: with a name whose length is a multiple of 4, `bool` and `int` have
|
||
identical item sizes; unknown names default to `int`.
|
||
|
||
## 3. Complex frames
|
||
|
||
```
|
||
[len][name][pad to 4] BE EF BE EF ...children... 10 41 10 41
|
||
(0xBEEFBEEF) (0x41104110 = ~0xBEEFBEEF)
|
||
```
|
||
|
||
* Every "nested object" (IStreamable via stream vft+0x28) is framed. Frames
|
||
nest; the reader recovers the tree from the markers alone, without a schema.
|
||
* A frame may be **tagless** (BEEFBEEF directly at an item position): the reader
|
||
accepts it; not observed in the reference material. **VERIFY** if seen.
|
||
* A frame body may be **untagged bytes**: `Mars::Vector3` (`Pos`, `PrvPos`,
|
||
`FtOrig`, `FPogn2`, `FPdpos`) and `ShipHealth` (`Health`) are "3 unnamed
|
||
floats" per the binary. Reader accepts either 12 raw bytes before the END
|
||
marker or three tagged floats (any tag, e.g. `"."`). **VERIFY** which.
|
||
* `RNG` frame body is treated as opaque bytes up to its END marker.
|
||
* Markers are only interpreted at item boundaries; a value that happens to equal
|
||
a marker is not a problem unless the reader is already resynchronising.
|
||
|
||
## 4. Arrays
|
||
|
||
| kind | encoding | reader schema |
|
||
|---|---|---|
|
||
| NonComplexArray (`VectorHelper`, `std::list`) | named int count, then count × element **inline in the same frame** | `NArr` |
|
||
| ComplexArray | a frame containing: int count, then count × element | `CArr` |
|
||
| element = leaf wrapper | e.g. `SysID` int + `Sys` frame; `PlayerID` + `Player`; `FltID` + `Flt`; `ShipID` + `Ship`; `DesID` + `Des` | `Seq([...])` |
|
||
| sparse tables | `mnsp` then n × (`msp` index, `mv` value); `nadct` then n × (`ads`, `adt`); `haltc` then n × (`haltt`, `haltv`) | `NArr(Seq)` |
|
||
|
||
Count tags used by the binary: `NumPlrs NumSys NumFlts NShips NumActs NumOwn NumDes
|
||
NumLeg NumNotes NumPR NSprj Nexp NWeapXcl ndeflay rdtc numcreps ninv NumFlts NumGFs
|
||
NumSnF NumMnF NVO NVE NVs NumPlgs2 PopNG mnsp nadct haltc PrNSp NTH`. Count tags
|
||
inside framed arrays (`dipstats`, `preps`, `odes`, `owep`, `otch`, `cme2`, `Ojvs`…)
|
||
are unknown — the reader takes the first child as the count whatever its name.
|
||
Element frame tags are unknown (R1 suggests `"."`) — matched by position.
|
||
|
||
## 5. Conditionals the reader honours
|
||
|
||
| gate | consequence |
|
||
|---|---|
|
||
| `ClrID`/`indcl`/`fxCrId` frame: int index == -1 | three ints r,g,b follow inside the frame |
|
||
| `vnh` true | `vnd`, `vnex3`, `vnpex3` |
|
||
| `hindi` true | `indi` frame (IndependenceInfo) |
|
||
| `NVO` entry `isind` true | `indi` frame |
|
||
| `HFPlan` true | `FPlan` frame |
|
||
| `HLay` true | `Lay` frame (opaque) |
|
||
| `hbq` true | `BQ2` frame; `hsp` true → `pop`, `ppop` Population frames |
|
||
| `HasAIR` true | `AIR` frame (opaque) |
|
||
| system `PID` non-null | `BQ` frame present (reader: optional by name) |
|
||
| legacy tags `ISuit ARes SysID TrdID Caps GtTrf FtSens FtInc lcid SensMod ExPopSys AIDifficultyID` | accepted if present, expected absent in 1.8 saves |
|
||
|
||
## 6. Top-level layout
|
||
|
||
```
|
||
offset 0: [7]"Summary" [pad] BEEFBEEF ... 41104110 (R2 confirms the tag "Summary")
|
||
CreateParameters frame (tag unknown)
|
||
Sim frame (tag unknown) KeyPath NMSz NMLc NMnx <id lists> ModCount Frame
|
||
GameID Attrib RNG GameName Map IncMod ResMod EnAl
|
||
EnTm GOTurn GOWinPly NPCm NPCo NPCi NPCv NPCa
|
||
szadj rsadj suadj sprjs RandEncAdj cmbtid turnstats
|
||
numcreps/crep ninv <invs.. AllExc..> NumPlrs/PlayerID/Player
|
||
<ISsp ISsu> NumSys/SysID/Sys NdGr2 trdmgr spymgr
|
||
NumFlts/FltID/Flt NumActs/Act SvSctOb zdsc zdsi zdst
|
||
CdTable UNFRAMED at root: cdt frame, cdplayer frame, N × cdai frames
|
||
```
|
||
|
||
Regions in `<...>` are parsed generically and kept as raw item lists
|
||
(`sim.idLists`, `sim.invasionsAndExclusions`, `sim.species`). Everything not
|
||
covered by a shape (TechTree body, Events, ShipRecs, spy2, civr, comms, Ojvs,
|
||
SvSctOb, trdmgr, spymgr, CdTable, …) is kept as the generic
|
||
`{"_name","_off","_items":[{"name","kind","value","off"}...]}` form.
|
||
|
||
## 7. Struct field orders and types applied
|
||
|
||
Shapes live in `save_reader.py` (`Sys`, `Player`, `Fleet`, `Ship`, `PlayerView`,
|
||
`Population`/`PopG`, `Morale`, `MoraleEvent`, `BuildQueue`/`BuildOrder`,
|
||
`IndependenceInfo`, `Rts`, `DipStat`, `Prep`, `FlightPlan`, `Waypoint`,
|
||
`PrisonerHold`, `Summary`, `CreateParams`, …). `A("tag", type)` = on-disk tag
|
||
confirmed in the exe (strict name match); `R("name", type)` = R1 C# name only
|
||
(positional). Binary corrections applied over R1:
|
||
|
||
* float: `TRM CstR CstE CstT shrm RefCap RepCap`, PlayerView `Infra`
|
||
* int64: `Bats2 rcex PopC`
|
||
* bool: `Abdn Dstyd PvMA AIBn` (R2's "short" readings are the value byte)
|
||
* int on disk though int16 in memory: `TShn ETS` and all DiplomacyStats counters
|
||
* string: `pswd`
|
||
* Vector3 (3 floats): `FtOrig`
|
||
* `Nexp` entries carry `xid xmin xmax xper(float)`
|
||
* `Team` appears twice in Player: an int, later a nested `{ALid AL NA CF}` frame
|
||
(typed key `Alliances`)
|
||
|
||
## 8. Reader output conventions
|
||
|
||
* `--dump`: one line per item, `@<inflated offset> name kind value`; `?` after
|
||
the kind means the type was guessed, `(alt …)` shows the other reading of a
|
||
4-byte word; frames print `{` … `}` with item count and byte size.
|
||
* `--json`: `{"padding", "stats", "issues": [...], "data": {...}}`; every typed
|
||
struct carries `_off`; unexpected items are kept under `_unexpected` /
|
||
`_extra`; unknown regions keep the generic form.
|
||
* Issue levels: `error` (schema field missing / type impossible / frame
|
||
unterminated), `warn` (resync, hint not plausible, unexpected items,
|
||
width mismatch, best-effort read at EOF), `info` (unnamed small payload
|
||
before END, positional tag name differing from the R1 name). `--strict`
|
||
fails on error or warn.
|
||
* Exit status: 0 clean, 1 errors present, 2 unreadable container.
|
||
|
||
## 9. What the verifier should check first on a real save
|
||
|
||
1. Padding convention (§2) — a 3-char bool tag settles it.
|
||
2. The `Summary` frame's 13 children and their actual tag names (R1 names are
|
||
probably case-variants: `NumSys` is confirmed by R2 as a tag string).
|
||
3. Whether Vector3 bodies are tagged (§3).
|
||
4. The tag used for element frames and for counts inside framed arrays (§4).
|
||
5. Whether the reader's auto-detected padding, `resyncs == 0` and
|
||
`hint_failures == 0` hold; any resync offset points at a layout gap.
|