122 lines
7.3 KiB
Markdown
122 lines
7.3 KiB
Markdown
# mars::stream — Streamable serialization (save files)
|
||
|
||
`src/mars/stream/` reimplements the engine's self-describing "Streamable" stream as used by
|
||
`.sav` files: a generic walker that recovers the item tree from any stream, a writer that
|
||
emits the same framing, and typed shapes for the confirmed top-level structures. The format
|
||
facts come from the RE repo's confirmed description (`verify/save-reader/SAVE_FORMAT.md`);
|
||
the reference Python reader is the oracle the C++ is tested against.
|
||
|
||
## The format in one screen
|
||
|
||
```
|
||
container one gzip member; everything below is the inflated stream, little-endian
|
||
item [int32 len][name bytes][value][NUL pad] pad brings the WHOLE item to 4 bytes
|
||
("joint" padding; a NULL name is written as ".")
|
||
scalars int32 | float32 | bool (1 byte) | int64 | string = [int32 len][cp1252 bytes]
|
||
(an empty string is 4 zero bytes — byte-identical to int 0; no type bytes anywhere)
|
||
frame [len][name][pad] BE EF BE EF ...items... 10 41 10 41 (0x41104110 = ~0xBEEFBEEF)
|
||
arrays VectorHelper<T>: a frame holding "." count + n × "." elements (frames or scalars)
|
||
inline arrays: a named count followed by n × element in the same frame
|
||
Vector3 a frame of 3 × "." float
|
||
root Summary → CreateParams → Sim → CDT → n × CD
|
||
RNG Sim.RNG { "." raw[2503] }: MT19937 mt[624] + left (0x9c4) + 3 pad bytes (see mars-rng.md)
|
||
```
|
||
|
||
## Modules
|
||
|
||
| file | contents |
|
||
|---|---|
|
||
| `bytes.h` | LE read/write helpers, `pad4`, the two markers |
|
||
| `node.h` | `Node{name, tagged, kind, raw, hinted, offset, size, children}`, `Issue`, `Stats` |
|
||
| `gzip.h/.cpp` | `gunzip`, `inflate_container` (passthrough for already-inflated data), `gzip` — via vendored miniz |
|
||
| `reader.h/.cpp` | `Walker` — the generic tokenizer; `read_tree()` |
|
||
| `writer.h/.cpp` | `Writer` — scalar/frame/raw emitters with joint padding; `Writer::node()` re-emits a tree |
|
||
| `dump.h/.cpp` | `dump_tree()` text form (identical to the reference `--dump`), Python float repr, JSON quoting |
|
||
| `schema.h` | `Desc`/`Hint`/`Registry` — the type hints the walker consults |
|
||
| `archive.h` | `ReadArchive`, `WriteArchive`, `SchemaBuilder` — drive a shape's `io()` field list |
|
||
| `shapes.h` | the typed shapes (Summary, CreateParams/MapP, Sim, Player, Sys, Fleet, Ship, …) |
|
||
| `save.h/.cpp` | `read_save_file/bytes()` → `SaveDocument{inflated, tree, game, issues, stats}`, `write_save()`, `write_tree()`, `save_registry()` |
|
||
| `savedump_main.cpp` | `sots_savedump SAVE [--dump] [--strict] [--roundtrip] [--rewrite OUT]` |
|
||
|
||
## Walker (generic reader)
|
||
|
||
The walker needs no schema. Frames are hard synchronisation points; between markers each
|
||
item's scalar layout is chosen by
|
||
|
||
1. a **hint** — positional (`Desc::prefix`, from the shape describing the frame) or by tag
|
||
(`Desc::by_name`, then the global tag → kind catalog), or
|
||
2. **guessing**: try `[word, bool, string, int64]` and keep the first layout after which
|
||
another plausible item, a marker or EOF follows (two items of lookahead; string values
|
||
only text-checked while guessing, since a known string may hold any cp1252 byte); a bare
|
||
4-byte word is int if `|i| <= 100000`, float if finite with `1e-6 <= |f| < 1e12`, else int.
|
||
|
||
Nothing readable is lost: when no layout fits, the bytes up to the next marker or plausible
|
||
tag become a `raw` node (a warning), a small unnamed payload before an END is a `raw` node
|
||
(info), and the walk resumes. Every node keeps its exact value bytes and offset, so
|
||
`write_tree(read_tree(x)) == x` for any input the walker accepts, damaged or not. These rules
|
||
are the reference reader's; the test suite checks the two implementations tokenise the three
|
||
real saves identically (every line of the dump, including the guessed/hinted marks).
|
||
|
||
Hints are not hand-maintained: `save_registry()` runs every shape's `io()` under
|
||
`SchemaBuilder`, which records prefixes, by-name tables and the global catalog (a tag
|
||
claimed with two different kinds is dropped from the catalog; optional legacy tags register
|
||
at lowest priority; `"."` is never a hint key because it carries ints, floats and frames alike).
|
||
|
||
## Shapes and archives
|
||
|
||
A shape is a struct with `kStreamName` and a `template<class Ar> void io(Ar&)` listing its
|
||
fields in disk order:
|
||
|
||
```cpp
|
||
struct Summary {
|
||
static constexpr const char* kStreamName = "Summary";
|
||
std::string gameName; int32_t turn = 0; ...
|
||
template <class Ar> void io(Ar& ar) {
|
||
ar.str(A("GameName"), gameName); // A(): confirmed on-disk tag, matched by name
|
||
ar.i32(A("Turn"), turn);
|
||
ar.carr(A("Players"), players); // VectorHelper: "." count + "." frames
|
||
...
|
||
ar.rest(extra); // anything the shape does not describe, kept generic
|
||
}
|
||
};
|
||
```
|
||
|
||
`R("idx")` fields are positional (the game wrote a NULL name — `"."` on disk — or the
|
||
reference name differs from the disk spelling, e.g. `faiDes` → `FAIDes`, `ontF` → `otnF`,
|
||
`nextId` → `nextid`); the second argument of `R` is the tag to write. Conditionals are plain
|
||
`ar.when(vnh, ...)`, optional legacy tags `ar.opt_i32(A("ARes"), aRes)`, uncounted lists
|
||
`ar.repeat("stats", ...)`, inline arrays `ar.narr(A("NumSys"), systems, elem)`. Bodies the
|
||
format keeps opaque (`TechTree`, `Events`, `ShipRecs`, `spy2`, `civr`, `comms`, `Ojvs`,
|
||
`Attrib`, `sprjs`, `SvSctOb`, `trdmgr`, `spymgr`, `CD`, the RNG blob) are `Node` members and
|
||
re-emit verbatim.
|
||
|
||
`ReadArchive` mirrors the reference applier: a confirmed tag that is missing is an error,
|
||
unexpected items before it a warning, a positional item read under another tag an info;
|
||
kinds are coerced with the same width/zero rules (4 zero bytes are the empty string). The
|
||
same `io()` drives `WriteArchive`, so a shape loaded from a real save writes back
|
||
**byte-identically** — the whole file, typed shapes plus retained generic bodies.
|
||
|
||
## Tests (`tests/mars_stream/build_and_run.sh`, plain g++)
|
||
|
||
* `test_stream.cpp` — hand-built byte fixtures: primitive encodings and joint padding (bytes
|
||
written out by hand and compared with the `Writer`), framing/nesting/tagless/empty frames,
|
||
`"."` arrays, resync (two fixtures cross-checked with the reference reader), cp1252 values,
|
||
hints from the registry, the raw RNG frame, typed Summary/Ship/Fleet write → read → write,
|
||
gzip, Python float repr and JSON quoting.
|
||
* `test_rng.cpp` — MT19937 reference vectors and state (de)serialization.
|
||
* `test_save.cpp` — **real saves** when `SOTS_SAVES_DIR` is set (skips otherwise, saves never
|
||
enter the repo): 0 errors/warnings/resyncs/hint failures, only the 2503-byte RNG blob raw,
|
||
typed values consistent (Summary ↔ Sim ↔ CreateParams counts, names, 7 species), RNG state
|
||
reachable from `seed(RSeed)`, tree and typed round trips byte-identical; with
|
||
`SOTS_SAVE_READER` set, `oracle/compare.py` diffs the C++ dump against the reference
|
||
`--dump` (exact and canonical agreement, summary values).
|
||
|
||
Result on the owner's three saves (SotS 1.8, turns 1–3): 100% exact line agreement, both
|
||
round trips identical, summaries agree.
|
||
|
||
## Not done / open
|
||
|
||
* Only joint padding is implemented (the split convention never matched a real file).
|
||
* Shapes for the opaque bodies listed above (kept generic on purpose, as in the reference).
|
||
* The float-vs-int guess for unhinted words inside opaque bodies is a heuristic; the typed
|
||
layer never depends on it (it coerces from the raw bytes).
|