mars::stream — Streamable serialization (save files)

src/mars/stream/ reimplements the engine's self-describing "Streamable" stream as used by .sav files: a generic walker that recovers the item tree from any stream, a writer that emits the same framing, and typed shapes for the confirmed top-level structures. The format facts come from the RE repo's confirmed description (verify/save-reader/SAVE_FORMAT.md); the reference Python reader is the oracle the C++ is tested against.

The format in one screen

container   one gzip member; everything below is the inflated stream, little-endian
item        [int32 len][name bytes][value][NUL pad]      pad brings the WHOLE item to 4 bytes
                                                         ("joint" padding; a NULL name is written as ".")
scalars     int32 | float32 | bool (1 byte) | int64 | string = [int32 len][cp1252 bytes]
            (an empty string is 4 zero bytes — byte-identical to int 0; no type bytes anywhere)
frame       [len][name][pad]  BE EF BE EF  ...items...  10 41 10 41      (0x41104110 = ~0xBEEFBEEF)
arrays      VectorHelper<T>: a frame holding "." count + n × "." elements (frames or scalars)
            inline arrays:   a named count followed by n × element in the same frame
Vector3     a frame of 3 × "." float
root        Summary → CreateParams → Sim → CDT → n × CD
RNG         Sim.RNG { "." raw[2503] }: MT19937 mt[624] + left (0x9c4) + 3 pad bytes (see mars-rng.md)

Modules

file	contents
`bytes.h`	LE read/write helpers, `pad4`, the two markers
`node.h`	`Node{name, tagged, kind, raw, hinted, offset, size, children}`, `Issue`, `Stats`
`gzip.h/.cpp`	`gunzip`, `inflate_container` (passthrough for already-inflated data), `gzip` — via vendored miniz
`reader.h/.cpp`	`Walker` — the generic tokenizer; `read_tree()`
`writer.h/.cpp`	`Writer` — scalar/frame/raw emitters with joint padding; `Writer::node()` re-emits a tree
`dump.h/.cpp`	`dump_tree()` text form (identical to the reference `--dump`), Python float repr, JSON quoting
`schema.h`	`Desc`/`Hint`/`Registry` — the type hints the walker consults
`archive.h`	`ReadArchive`, `WriteArchive`, `SchemaBuilder` — drive a shape's `io()` field list
`shapes.h`	the typed shapes (Summary, CreateParams/MapP, Sim, Player, Sys, Fleet, Ship, …)
`save.h/.cpp`	`read_save_file/bytes()` → `SaveDocument{inflated, tree, game, issues, stats}`, `write_save()`, `write_tree()`, `save_registry()`
`savedump_main.cpp`	`sots_savedump SAVE [--dump] [--strict] [--roundtrip] [--rewrite OUT]`

Walker (generic reader)

The walker needs no schema. Frames are hard synchronisation points; between markers each item's scalar layout is chosen by

a hint — positional (Desc::prefix, from the shape describing the frame) or by tag (Desc::by_name, then the global tag → kind catalog), or
guessing: try [word, bool, string, int64] and keep the first layout after which another plausible item, a marker or EOF follows (two items of lookahead; string values only text-checked while guessing, since a known string may hold any cp1252 byte); a bare 4-byte word is int if |i| <= 100000, float if finite with 1e-6 <= |f| < 1e12, else int.

Nothing readable is lost: when no layout fits, the bytes up to the next marker or plausible tag become a raw node (a warning), a small unnamed payload before an END is a raw node (info), and the walk resumes. Every node keeps its exact value bytes and offset, so write_tree(read_tree(x)) == x for any input the walker accepts, damaged or not. These rules are the reference reader's; the test suite checks the two implementations tokenise the three real saves identically (every line of the dump, including the guessed/hinted marks).

Hints are not hand-maintained: save_registry() runs every shape's io() under SchemaBuilder, which records prefixes, by-name tables and the global catalog (a tag claimed with two different kinds is dropped from the catalog; optional legacy tags register at lowest priority; "." is never a hint key because it carries ints, floats and frames alike).

Shapes and archives

A shape is a struct with kStreamName and a template<class Ar> void io(Ar&) listing its fields in disk order:

struct Summary {
    static constexpr const char* kStreamName = "Summary";
    std::string gameName; int32_t turn = 0; ...
    template <class Ar> void io(Ar& ar) {
        ar.str(A("GameName"), gameName);      // A(): confirmed on-disk tag, matched by name
        ar.i32(A("Turn"), turn);
        ar.carr(A("Players"), players);       // VectorHelper: "." count + "." frames
        ...
        ar.rest(extra);                       // anything the shape does not describe, kept generic
    }
};

R("idx") fields are positional (the game wrote a NULL name — "." on disk — or the reference name differs from the disk spelling, e.g. faiDes → FAIDes, ontF → otnF, nextId → nextid); the second argument of R is the tag to write. Conditionals are plain ar.when(vnh, ...), optional legacy tags ar.opt_i32(A("ARes"), aRes), uncounted lists ar.repeat("stats", ...), inline arrays ar.narr(A("NumSys"), systems, elem). Bodies the format keeps opaque (TechTree, Events, ShipRecs, spy2, civr, comms, Ojvs, Attrib, sprjs, SvSctOb, trdmgr, spymgr, CD, the RNG blob) are Node members and re-emit verbatim.

ReadArchive mirrors the reference applier: a confirmed tag that is missing is an error, unexpected items before it a warning, a positional item read under another tag an info; kinds are coerced with the same width/zero rules (4 zero bytes are the empty string). The same io() drives WriteArchive, so a shape loaded from a real save writes back byte-identically — the whole file, typed shapes plus retained generic bodies.

Tests (`tests/mars_stream/build_and_run.sh`, plain g++)

test_stream.cpp — hand-built byte fixtures: primitive encodings and joint padding (bytes written out by hand and compared with the Writer), framing/nesting/tagless/empty frames, "." arrays, resync (two fixtures cross-checked with the reference reader), cp1252 values, hints from the registry, the raw RNG frame, typed Summary/Ship/Fleet write → read → write, gzip, Python float repr and JSON quoting.
test_rng.cpp — MT19937 reference vectors and state (de)serialization.
test_save.cpp — real saves when SOTS_SAVES_DIR is set (skips otherwise, saves never enter the repo): 0 errors/warnings/resyncs/hint failures, only the 2503-byte RNG blob raw, typed values consistent (Summary ↔ Sim ↔ CreateParams counts, names, 7 species), RNG state reachable from seed(RSeed), tree and typed round trips byte-identical; with SOTS_SAVE_READER set, oracle/compare.py diffs the C++ dump against the reference --dump (exact and canonical agreement, summary values).

Result on the owner's three saves (SotS 1.8, turns 1–3): 100% exact line agreement, both round trips identical, summaries agree.

Not done / open

Only joint padding is implemented (the split convention never matched a real file).
Shapes for the opaque bodies listed above (kept generic on purpose, as in the reference).
The float-vs-int guess for unhinted words inside opaque bodies is a heuristic; the typed layer never depends on it (it coerces from the raw bytes).

7.3 KiB Raw Blame History Unescape Escape