sots-engine/docs/mars-stream.md

mars::stream — Streamable serialization (save files)

src/mars/stream/ reimplements the engine's self-describing "Streamable" stream as used by .sav files: a generic walker that recovers the item tree from any stream, a writer that emits the same framing, and typed shapes for the confirmed top-level structures. The format facts come from the RE repo's confirmed description (verify/save-reader/SAVE_FORMAT.md); the reference Python reader is the oracle the C++ is tested against.

The format in one screen

container   one gzip member; everything below is the inflated stream, little-endian
item        [int32 len][name bytes][value][NUL pad]      pad brings the WHOLE item to 4 bytes
                                                         ("joint" padding; a NULL name is written as ".")
scalars     int32 | float32 | bool (1 byte) | int64 | string = [int32 len][cp1252 bytes]
            (an empty string is 4 zero bytes — byte-identical to int 0; no type bytes anywhere)
frame       [len][name][pad]  BE EF BE EF  ...items...  10 41 10 41      (0x41104110 = ~0xBEEFBEEF)
arrays      VectorHelper<T>: a frame holding "." count + n × "." elements (frames or scalars)
            inline arrays:   a named count followed by n × element in the same frame
Vector3     a frame of 3 × "." float
root        Summary → CreateParams → Sim → CDT → n × CD
RNG         Sim.RNG { "." raw[2503] }: MT19937 mt[624] + left (0x9c4) + 3 pad bytes (see mars-rng.md)

Modules

file	contents
bytes.h	LE read/write helpers, pad4, the two markers
node.h	Node{name, tagged, kind, raw, hinted, offset, size, children}, Issue, Stats
gzip.h/.cpp	gunzip, inflate_container (passthrough for already-inflated data), gzip — via vendored miniz
reader.h/.cpp	Walker — the generic tokenizer; read_tree()
writer.h/.cpp	Writer — scalar/frame/raw emitters with joint padding; Writer::node() re-emits a tree
dump.h/.cpp	dump_tree() text form (identical to the reference --dump), Python float repr, JSON quoting
schema.h	Desc/Hint/Registry — the type hints the walker consults
archive.h	ReadArchive, WriteArchive, SchemaBuilder — drive a shape's io() field list
shapes.h	the typed shapes (Summary, CreateParams/MapP, Sim, Player, Sys, Fleet, Ship, …)
save.h/.cpp	read_save_file/bytes() → SaveDocument{inflated, tree, game, issues, stats}, write_save(), write_tree(), save_registry()
savedump_main.cpp	sots_savedump SAVE [--dump] [--strict] [--roundtrip] [--rewrite OUT]

Walker (generic reader)

The walker needs no schema. Frames are hard synchronisation points; between markers each item's scalar layout is chosen by

1. a hint — positional (Desc::prefix, from the shape describing the frame) or by tag (Desc::by_name, then the global tag → kind catalog), or
2. guessing: try [word, bool, string, int64] and keep the first layout after which another plausible item, a marker or EOF follows (two items of lookahead; string values only text-checked while guessing, since a known string may hold any cp1252 byte); a bare 4-byte word is int if |i| <= 100000, float if finite with 1e-6 <= |f| < 1e12, else int.

Nothing readable is lost: when no layout fits, the bytes up to the next marker or plausible tag become a raw node (a warning), a small unnamed payload before an END is a raw node (info), and the walk resumes. Every node keeps its exact value bytes and offset, so write_tree(read_tree(x)) == x for any input the walker accepts, damaged or not. These rules are the reference reader's; the test suite checks the two implementations tokenise the three real saves identically (every line of the dump, including the guessed/hinted marks).

Hints are not hand-maintained: save_registry() runs every shape's io() under SchemaBuilder, which records prefixes, by-name tables and the global catalog (a tag claimed with two different kinds is dropped from the catalog; optional legacy tags register at lowest priority; "." is never a hint key because it carries ints, floats and frames alike).

Shapes and archives

A shape is a struct with kStreamName and a template<class Ar> void io(Ar&) listing its fields in disk order:

struct Summary {
    static constexpr const char* kStreamName = "Summary";
    std::string gameName; int32_t turn = 0; ...
    template <class Ar> void io(Ar& ar) {
        ar.str(A("GameName"), gameName);      // A(): confirmed on-disk tag, matched by name
        ar.i32(A("Turn"), turn);
        ar.carr(A("Players"), players);       // VectorHelper: "." count + "." frames
        ...
        ar.rest(extra);                       // anything the shape does not describe, kept generic
    }
};

R("idx") fields are positional (the game wrote a NULL name — "." on disk — or the reference name differs from the disk spelling, e.g. faiDes → FAIDes, ontF → otnF, nextId → nextid); the second argument of R is the tag to write. Conditionals are plain ar.when(vnh, ...), optional legacy tags ar.opt_i32(A("ARes"), aRes), uncounted lists ar.repeat("stats", ...), inline arrays ar.narr(A("NumSys"), systems, elem). Bodies still kept opaque are Node members and re-emit verbatim. That list has shrunk to two kinds: the RNG blob, which is deliberately opaque, and containers no save has ever filled — nodePaths, Ojvs, usnc, FTPnts, the four SpyReport sub-lists, the deflay/rtgt pairs, and the polymorphic keys with no factory entry (SvSctOb variants, CD block suffixes). Those are hypotheses under rule 6, not modelling debt: the element framing is a property of the writer's helper and no record has ever shown one. On every save in the corpus the only opaque items left are the RNG blob's two.

ReadArchive mirrors the reference applier: a confirmed tag that is missing is an error, unexpected items before it a warning, a positional item read under another tag an info; kinds are coerced with the same width/zero rules (4 zero bytes are the empty string). The same io() drives WriteArchive, so a shape loaded from a real save writes back byte-identically — the whole file, typed shapes plus retained generic bodies.

Tests (tests/mars_stream/build_and_run.sh, plain g++)

• test_stream.cpp — hand-built byte fixtures: primitive encodings and joint padding (bytes written out by hand and compared with the Writer), framing/nesting/tagless/empty frames, "." arrays, resync (two fixtures cross-checked with the reference reader), cp1252 values, hints from the registry, the raw RNG frame, typed Summary/Ship/Fleet write → read → write, gzip, Python float repr and JSON quoting.
• test_rng.cpp — MT19937 reference vectors and state (de)serialization.
• test_save.cpp — real saves when SOTS_SAVES_DIR is set (skips otherwise, saves never enter the repo): 0 errors/warnings/resyncs/hint failures, only the 2503-byte RNG blob raw, typed values consistent (Summary ↔ Sim ↔ CreateParams counts, names, 7 species), RNG state reachable from seed(RSeed), tree and typed round trips byte-identical; with SOTS_SAVE_READER set, oracle/compare.py diffs the C++ dump against the reference --dump (exact and canonical agreement, summary values).

Result on the owner's 20-save corpus (SotS 1.8): 100% exact line agreement, both round trips identical, summaries agree. Named coverage is 99.994–99.996% per save; the residual is the two items of the RNG blob and nothing else.

Not done / open

• Only joint padding is implemented (the split convention never matched a real file).
• Element shapes for the never-filled containers listed above — each needs a save that puts something in it before the framing can be typed rather than guessed.
• The float-vs-int guess for unhinted words inside opaque bodies is a heuristic; the typed layer never depends on it (it coerces from the raw bytes).