7f04bcc88b
The corpus went 11 -> 19 saves and named coverage fell to 97.6% against a 99.99
bar. Rule 27 says type the content. Typed, all from the recovered wire schema
plus the records the new trade/spy saves finally carry:
Game::CombatPlayerReport the <rest:prep> tail -- ncls run of six per ship
class, nsec run of two per section, ndam, srep
Game::CombatShipReport srep elements (caps2 is an i64; dami/damt f32)
Game::TacReport TRnc is a COUNT and TRships/TRsats/TRshipsL are one
loop body, INTERLEAVED on the wire -- not three
trailing runs. Two lanes could not settle this
because TRnc was 0 in every save until tonight.
Game::TacReportEvents TRby / TRto
Game::FleetLayout Lay: a FieldTemplate frame then a count and its ids
Game::TradeRoute rt -- a container write with NO count word, read as
an uncounted run keyed on the tag
Game::ServerTradeSector::FreighterWarning fwarn elements
Game::SpyCraft spy elements
Game::WeaponGroups Dwg, and Game::GunBankSelection under it
Corrections found on the way, both silent until a shape was bound to the table:
* Game::CombatReport: auto and cdst are BOOLs, dur/cdt/cdi are FLOATS. All
five were ints here. Four are 0/1 everywhere so the bytes never moved; cdt
is not, and was being read as 1070805848 instead of 1.598.
* Game::CombatWeaponReport: the damage quartet is flat and dami/damt are
floats. The old shape reached them through obj_flex, modelling a nested
`dams` frame the binary does not write. That also made CoverageArchive
charge one phantom typed item per weapon report -- exactly 8/13/21/32 on the
four affected saves -- so the pre-fix coverage figures were slightly
optimistic as well as too low.
FTPnts stays carried: it is the one item the recovery itself marks unresolved,
its count is 0 in every save, and element framing is a property of the helper --
the SysMem / mts / nalat trap. The workload that settles it is a save with a
stored fleet tactical formation.
Named coverage, per save, before -> after:
human-turn5-traderoutes 99.0003% -> 99.9951%
human-turn8-traderoutes 98.6349% -> 99.9955%
human-turn11-spytechs 97.6382% -> 99.9959%
human-turn15-spyprogram 96.9412% -> 99.9964%
the other 16 saves unchanged, 99.9940-99.9953%
corpus 99.5021% -> 99.9949%
All 20 saves are above the ratchet; the only opaque items left anywhere are the
two of the RNG blob. 14 new shapes bound to the wire schema, every one a full
match: 100 shapes, 966 items, 0 MISMATCH. ctest 58/58 over all 20 saves, both
round trips byte-identical, 0 errors and 0 warnings.
128 lines
7.9 KiB
Markdown
128 lines
7.9 KiB
Markdown
# mars::stream — Streamable serialization (save files)
|
||
|
||
`src/mars/stream/` reimplements the engine's self-describing "Streamable" stream as used by
|
||
`.sav` files: a generic walker that recovers the item tree from any stream, a writer that
|
||
emits the same framing, and typed shapes for the confirmed top-level structures. The format
|
||
facts come from the RE repo's confirmed description (`verify/save-reader/SAVE_FORMAT.md`);
|
||
the reference Python reader is the oracle the C++ is tested against.
|
||
|
||
## The format in one screen
|
||
|
||
```
|
||
container one gzip member; everything below is the inflated stream, little-endian
|
||
item [int32 len][name bytes][value][NUL pad] pad brings the WHOLE item to 4 bytes
|
||
("joint" padding; a NULL name is written as ".")
|
||
scalars int32 | float32 | bool (1 byte) | int64 | string = [int32 len][cp1252 bytes]
|
||
(an empty string is 4 zero bytes — byte-identical to int 0; no type bytes anywhere)
|
||
frame [len][name][pad] BE EF BE EF ...items... 10 41 10 41 (0x41104110 = ~0xBEEFBEEF)
|
||
arrays VectorHelper<T>: a frame holding "." count + n × "." elements (frames or scalars)
|
||
inline arrays: a named count followed by n × element in the same frame
|
||
Vector3 a frame of 3 × "." float
|
||
root Summary → CreateParams → Sim → CDT → n × CD
|
||
RNG Sim.RNG { "." raw[2503] }: MT19937 mt[624] + left (0x9c4) + 3 pad bytes (see mars-rng.md)
|
||
```
|
||
|
||
## Modules
|
||
|
||
| file | contents |
|
||
|---|---|
|
||
| `bytes.h` | LE read/write helpers, `pad4`, the two markers |
|
||
| `node.h` | `Node{name, tagged, kind, raw, hinted, offset, size, children}`, `Issue`, `Stats` |
|
||
| `gzip.h/.cpp` | `gunzip`, `inflate_container` (passthrough for already-inflated data), `gzip` — via vendored miniz |
|
||
| `reader.h/.cpp` | `Walker` — the generic tokenizer; `read_tree()` |
|
||
| `writer.h/.cpp` | `Writer` — scalar/frame/raw emitters with joint padding; `Writer::node()` re-emits a tree |
|
||
| `dump.h/.cpp` | `dump_tree()` text form (identical to the reference `--dump`), Python float repr, JSON quoting |
|
||
| `schema.h` | `Desc`/`Hint`/`Registry` — the type hints the walker consults |
|
||
| `archive.h` | `ReadArchive`, `WriteArchive`, `SchemaBuilder` — drive a shape's `io()` field list |
|
||
| `shapes.h` | the typed shapes (Summary, CreateParams/MapP, Sim, Player, Sys, Fleet, Ship, …) |
|
||
| `save.h/.cpp` | `read_save_file/bytes()` → `SaveDocument{inflated, tree, game, issues, stats}`, `write_save()`, `write_tree()`, `save_registry()` |
|
||
| `savedump_main.cpp` | `sots_savedump SAVE [--dump] [--strict] [--roundtrip] [--rewrite OUT]` |
|
||
|
||
## Walker (generic reader)
|
||
|
||
The walker needs no schema. Frames are hard synchronisation points; between markers each
|
||
item's scalar layout is chosen by
|
||
|
||
1. a **hint** — positional (`Desc::prefix`, from the shape describing the frame) or by tag
|
||
(`Desc::by_name`, then the global tag → kind catalog), or
|
||
2. **guessing**: try `[word, bool, string, int64]` and keep the first layout after which
|
||
another plausible item, a marker or EOF follows (two items of lookahead; string values
|
||
only text-checked while guessing, since a known string may hold any cp1252 byte); a bare
|
||
4-byte word is int if `|i| <= 100000`, float if finite with `1e-6 <= |f| < 1e12`, else int.
|
||
|
||
Nothing readable is lost: when no layout fits, the bytes up to the next marker or plausible
|
||
tag become a `raw` node (a warning), a small unnamed payload before an END is a `raw` node
|
||
(info), and the walk resumes. Every node keeps its exact value bytes and offset, so
|
||
`write_tree(read_tree(x)) == x` for any input the walker accepts, damaged or not. These rules
|
||
are the reference reader's; the test suite checks the two implementations tokenise the three
|
||
real saves identically (every line of the dump, including the guessed/hinted marks).
|
||
|
||
Hints are not hand-maintained: `save_registry()` runs every shape's `io()` under
|
||
`SchemaBuilder`, which records prefixes, by-name tables and the global catalog (a tag
|
||
claimed with two different kinds is dropped from the catalog; optional legacy tags register
|
||
at lowest priority; `"."` is never a hint key because it carries ints, floats and frames alike).
|
||
|
||
## Shapes and archives
|
||
|
||
A shape is a struct with `kStreamName` and a `template<class Ar> void io(Ar&)` listing its
|
||
fields in disk order:
|
||
|
||
```cpp
|
||
struct Summary {
|
||
static constexpr const char* kStreamName = "Summary";
|
||
std::string gameName; int32_t turn = 0; ...
|
||
template <class Ar> void io(Ar& ar) {
|
||
ar.str(A("GameName"), gameName); // A(): confirmed on-disk tag, matched by name
|
||
ar.i32(A("Turn"), turn);
|
||
ar.carr(A("Players"), players); // VectorHelper: "." count + "." frames
|
||
...
|
||
ar.rest(extra); // anything the shape does not describe, kept generic
|
||
}
|
||
};
|
||
```
|
||
|
||
`R("idx")` fields are positional (the game wrote a NULL name — `"."` on disk — or the
|
||
reference name differs from the disk spelling, e.g. `faiDes` → `FAIDes`, `ontF` → `otnF`,
|
||
`nextId` → `nextid`); the second argument of `R` is the tag to write. Conditionals are plain
|
||
`ar.when(vnh, ...)`, optional legacy tags `ar.opt_i32(A("ARes"), aRes)`, uncounted lists
|
||
`ar.repeat("stats", ...)`, inline arrays `ar.narr(A("NumSys"), systems, elem)`. Bodies still
|
||
kept opaque are `Node` members and re-emit verbatim. That list has shrunk to two kinds: the
|
||
**RNG blob**, which is deliberately opaque, and **containers no save has ever filled** —
|
||
`nodePaths`, `Ojvs`, `usnc`, `FTPnts`, the four `SpyReport` sub-lists, the `deflay`/`rtgt`
|
||
pairs, and the polymorphic keys with no factory entry (`SvSctOb` variants, `CD` block
|
||
suffixes). Those are hypotheses under rule 6, not modelling debt: the element framing is a
|
||
property of the writer's helper and no record has ever shown one. On every save in the
|
||
corpus the only opaque items left are the RNG blob's two.
|
||
|
||
`ReadArchive` mirrors the reference applier: a confirmed tag that is missing is an error,
|
||
unexpected items before it a warning, a positional item read under another tag an info;
|
||
kinds are coerced with the same width/zero rules (4 zero bytes are the empty string). The
|
||
same `io()` drives `WriteArchive`, so a shape loaded from a real save writes back
|
||
**byte-identically** — the whole file, typed shapes plus retained generic bodies.
|
||
|
||
## Tests (`tests/mars_stream/build_and_run.sh`, plain g++)
|
||
|
||
* `test_stream.cpp` — hand-built byte fixtures: primitive encodings and joint padding (bytes
|
||
written out by hand and compared with the `Writer`), framing/nesting/tagless/empty frames,
|
||
`"."` arrays, resync (two fixtures cross-checked with the reference reader), cp1252 values,
|
||
hints from the registry, the raw RNG frame, typed Summary/Ship/Fleet write → read → write,
|
||
gzip, Python float repr and JSON quoting.
|
||
* `test_rng.cpp` — MT19937 reference vectors and state (de)serialization.
|
||
* `test_save.cpp` — **real saves** when `SOTS_SAVES_DIR` is set (skips otherwise, saves never
|
||
enter the repo): 0 errors/warnings/resyncs/hint failures, only the 2503-byte RNG blob raw,
|
||
typed values consistent (Summary ↔ Sim ↔ CreateParams counts, names, 7 species), RNG state
|
||
reachable from `seed(RSeed)`, tree and typed round trips byte-identical; with
|
||
`SOTS_SAVE_READER` set, `oracle/compare.py` diffs the C++ dump against the reference
|
||
`--dump` (exact and canonical agreement, summary values).
|
||
|
||
Result on the owner's 20-save corpus (SotS 1.8): 100% exact line agreement, both round trips
|
||
identical, summaries agree. Named coverage is 99.994–99.996% per save; the residual is the
|
||
two items of the RNG blob and nothing else.
|
||
|
||
## Not done / open
|
||
|
||
* Only joint padding is implemented (the split convention never matched a real file).
|
||
* Element shapes for the never-filled containers listed above — each needs a save that puts
|
||
something in it before the framing can be typed rather than guessed.
|
||
* The float-vs-int guess for unhinted words inside opaque bodies is a heuristic; the typed
|
||
layer never depends on it (it coerces from the raw bytes).
|