# mars::stream — Streamable serialization (save files)

`src/mars/stream/` reimplements the engine's self-describing "Streamable" stream as used by
`.sav` files: a generic walker that recovers the item tree from any stream, a writer that
emits the same framing, and typed shapes for the confirmed top-level structures. The format
facts come from the RE repo's confirmed description (`verify/save-reader/SAVE_FORMAT.md`);
the reference Python reader is the oracle the C++ is tested against.

## The format in one screen

```
container   one gzip member; everything below is the inflated stream, little-endian
item        [int32 len][name bytes][value][NUL pad]      pad brings the WHOLE item to 4 bytes
                                                         ("joint" padding; a NULL name is written as ".")
scalars     int32 | float32 | bool (1 byte) | int64 | string = [int32 len][cp1252 bytes]
            (an empty string is 4 zero bytes — byte-identical to int 0; no type bytes anywhere)
frame       [len][name][pad]  BE EF BE EF  ...items...  10 41 10 41      (0x41104110 = ~0xBEEFBEEF)
arrays      VectorHelper<T>: a frame holding "." count + n × "." elements (frames or scalars)
            inline arrays:   a named count followed by n × element in the same frame
Vector3     a frame of 3 × "." float
root        Summary → CreateParams → Sim → CDT → n × CD
RNG         Sim.RNG { "." raw[2503] }: MT19937 mt[624] + left (0x9c4) + 3 pad bytes (see mars-rng.md)
```

## Modules

| file | contents |
|---|---|
| `bytes.h` | LE read/write helpers, `pad4`, the two markers |
| `node.h` | `Node{name, tagged, kind, raw, hinted, offset, size, children}`, `Issue`, `Stats` |
| `gzip.h/.cpp` | `gunzip`, `inflate_container` (passthrough for already-inflated data), `gzip` — via vendored miniz |
| `reader.h/.cpp` | `Walker` — the generic tokenizer; `read_tree()` |
| `writer.h/.cpp` | `Writer` — scalar/frame/raw emitters with joint padding; `Writer::node()` re-emits a tree |
| `dump.h/.cpp` | `dump_tree()` text form (identical to the reference `--dump`), Python float repr, JSON quoting |
| `schema.h` | `Desc`/`Hint`/`Registry` — the type hints the walker consults |
| `archive.h` | `ReadArchive`, `WriteArchive`, `SchemaBuilder` — drive a shape's `io()` field list |
| `shapes.h` | the typed shapes (Summary, CreateParams/MapP, Sim, Player, Sys, Fleet, Ship, …) |
| `save.h/.cpp` | `read_save_file/bytes()` → `SaveDocument{inflated, tree, game, issues, stats}`, `write_save()`, `write_tree()`, `save_registry()` |
| `savedump_main.cpp` | `sots_savedump SAVE [--dump] [--strict] [--roundtrip] [--rewrite OUT]` |

## Walker (generic reader)

The walker needs no schema. Frames are hard synchronisation points; between markers each
item's scalar layout is chosen by

1. a **hint** — positional (`Desc::prefix`, from the shape describing the frame) or by tag
   (`Desc::by_name`, then the global tag → kind catalog), or
2. **guessing**: try `[word, bool, string, int64]` and keep the first layout after which
   another plausible item, a marker or EOF follows (two items of lookahead; string values
   only text-checked while guessing, since a known string may hold any cp1252 byte); a bare
   4-byte word is int if `|i| <= 100000`, float if finite with `1e-6 <= |f| < 1e12`, else int.

Nothing readable is lost: when no layout fits, the bytes up to the next marker or plausible
tag become a `raw` node (a warning), a small unnamed payload before an END is a `raw` node
(info), and the walk resumes. Every node keeps its exact value bytes and offset, so
`write_tree(read_tree(x)) == x` for any input the walker accepts, damaged or not. These rules
are the reference reader's; the test suite checks the two implementations tokenise the three
real saves identically (every line of the dump, including the guessed/hinted marks).

Hints are not hand-maintained: `save_registry()` runs every shape's `io()` under
`SchemaBuilder`, which records prefixes, by-name tables and the global catalog (a tag
claimed with two different kinds is dropped from the catalog; optional legacy tags register
at lowest priority; `"."` is never a hint key because it carries ints, floats and frames alike).

## Shapes and archives

A shape is a struct with `kStreamName` and a `template<class Ar> void io(Ar&)` listing its
fields in disk order:

```cpp
struct Summary {
    static constexpr const char* kStreamName = "Summary";
    std::string gameName; int32_t turn = 0; ...
    template <class Ar> void io(Ar& ar) {
        ar.str(A("GameName"), gameName);      // A(): confirmed on-disk tag, matched by name
        ar.i32(A("Turn"), turn);
        ar.carr(A("Players"), players);       // VectorHelper: "." count + "." frames
        ...
        ar.rest(extra);                       // anything the shape does not describe, kept generic
    }
};
```

`R("idx")` fields are positional (the game wrote a NULL name — `"."` on disk — or the
reference name differs from the disk spelling, e.g. `faiDes` → `FAIDes`, `ontF` → `otnF`,
`nextId` → `nextid`); the second argument of `R` is the tag to write. Conditionals are plain
`ar.when(vnh, ...)`, optional legacy tags `ar.opt_i32(A("ARes"), aRes)`, uncounted lists
`ar.repeat("stats", ...)`, inline arrays `ar.narr(A("NumSys"), systems, elem)`. Bodies still
kept opaque are `Node` members and re-emit verbatim. That list has shrunk to two kinds: the
**RNG blob**, which is deliberately opaque, and **containers no save has ever filled** —
`nodePaths`, `Ojvs`, `usnc`, `FTPnts`, the four `SpyReport` sub-lists, the `deflay`/`rtgt`
pairs, and the polymorphic keys with no factory entry (`SvSctOb` variants, `CD` block
suffixes). Those are hypotheses under rule 6, not modelling debt: the element framing is a
property of the writer's helper and no record has ever shown one. On every save in the
corpus the only opaque items left are the RNG blob's two.

`ReadArchive` mirrors the reference applier: a confirmed tag that is missing is an error,
unexpected items before it a warning, a positional item read under another tag an info;
kinds are coerced with the same width/zero rules (4 zero bytes are the empty string). The
same `io()` drives `WriteArchive`, so a shape loaded from a real save writes back
**byte-identically** — the whole file, typed shapes plus retained generic bodies.

## Tests (`tests/mars_stream/build_and_run.sh`, plain g++)

* `test_stream.cpp` — hand-built byte fixtures: primitive encodings and joint padding (bytes
  written out by hand and compared with the `Writer`), framing/nesting/tagless/empty frames,
  `"."` arrays, resync (two fixtures cross-checked with the reference reader), cp1252 values,
  hints from the registry, the raw RNG frame, typed Summary/Ship/Fleet write → read → write,
  gzip, Python float repr and JSON quoting.
* `test_rng.cpp` — MT19937 reference vectors and state (de)serialization.
* `test_save.cpp` — **real saves** when `SOTS_SAVES_DIR` is set (skips otherwise, saves never
  enter the repo): 0 errors/warnings/resyncs/hint failures, only the 2503-byte RNG blob raw,
  typed values consistent (Summary ↔ Sim ↔ CreateParams counts, names, 7 species), RNG state
  reachable from `seed(RSeed)`, tree and typed round trips byte-identical; with
  `SOTS_SAVE_READER` set, `oracle/compare.py` diffs the C++ dump against the reference
  `--dump` (exact and canonical agreement, summary values).

Result on the owner's 20-save corpus (SotS 1.8): 100% exact line agreement, both round trips
identical, summaries agree. Named coverage is 99.994–99.996% per save; the residual is the
two items of the RNG blob and nothing else.

## Not done / open

* Only joint padding is implemented (the split convention never matched a real file).
* Element shapes for the never-filled containers listed above — each needs a save that puts
  something in it before the framing can be typed rather than guessed.
* The float-vs-int guess for unhinted words inside opaque bodies is a heuristic; the typed
  layer never depends on it (it coerces from the raw bytes).