# `mars::parse` — the engine's data-file readers `src/mars/parse/` reimplements the two text readers the original engine uses for its catalogs: the recursive **brace-block** key/value format (weapons, ship sections, the tech tree, combat missions, badge/light defs, GUI scripts, scenario and other block-form `.txt`) and the separate **`.effect`** format. C++17, no dependencies, no exceptions across the API, bytes passed through untouched (the data is cp1252). Behaviour is specified by the proven Python readers in the RE repo (`sots-re/verify/parsers/mars_data.py`, `effect_txt.py`) and the findings in `findings/subsystems/data-parsers.md`. The C++ is written from that understanding, not transliterated; the Python readers serve as the test oracle. ``` src/mars/parse/ result.h Diagnostic {line, message}; Result value.h/.cpp classify / as_int / as_double / as_bool / iequals blocks.h/.cpp brace-block reader: parse_blocks(text, Options) -> Result effect.h/.cpp .effect reader: parse_effect(text) -> Result CMakeLists.txt static library `mars_parse` (include root: src/) tests/mars_parse/ build_and_run.sh plain g++ build; unit tests; real-data oracle test test_*.cpp hand-written samples, one test per quirk canon.h/.cpp canonical JSON rendering (matches oracle/dump.py) oracle_check.cpp parses every real file, diffs against the oracle oracle/dump.py runs the RE-repo readers, emits canonical JSON per file ``` ## 1. Brace-block format — grammar as implemented ``` body := (block | pair | item)* block := NAME '{' body '}' pair := NAME value item := QUOTED -- a quoted string with no key | NAME -- a bareword directly before '}' or EOF value := QUOTED | BAREWORD comment := '//' ... end-of-line -- only at a token boundary ``` Tokens (whitespace-driven, not line-based): | token | rule | |---|---| | whitespace | space, tab, CR, LF, VT, FF | | `{` `}` | single-character tokens; they also terminate a bareword (`weapon{` is two tokens) | | QUOTED | `"` … `"`; no escapes at all — backslashes and `//` inside are literal; may span lines; an empty `""` is a valid value | | BAREWORD | maximal run of anything that is not whitespace, `{`, `}` or `"` — so `abc"def"` is bareword `abc` followed by a quoted token, and `3//x` is one bareword | | comment | `//` at the start of a token runs to end of line | The file's top level is itself a body: catalog files hold one or more named blocks, scenario `.txt` files mix top-level pairs and `player{}` blocks. Disambiguation of a bareword `N`: look at the next token. `{` → a block named `N`; `}` or EOF → an item; anything else → a pair `N value` (the value may be bare or quoted). A quoted token in key position is always an item. ### Leniency (what the shipped data needs) | condition | default (`Options::strict = false`) | `strict = true` | |---|---|---| | EOF while blocks are open | close every open block, one `Diagnostic` per open block in `Document::warnings` | error at the EOF line | | `}` at top level | ignored, warning, parsing continues | error | | unterminated `"` | error | error | | `{` with no name before it | error | error | The two recoveries are exactly what 12 shipped shipsections require (11 never close their outer block, `Human/CrPropaganda.shipsection` has one `}` too many). They load in the game, so the engine performs the same repairs. Nothing in the shipped data hits the two hard errors. ### Node model ```cpp struct Entry { // one thing in a block, in file order enum class Kind { Pair, Item, Block } kind; std::string key; // Pair/Block: as written (case preserved) std::string value; // Pair: value text; Item: the item text bool quoted; // value/item was in double quotes std::unique_ptr block; // Block only int line; // 1-based }; struct Node { // a named block; the root has name "" std::string name; std::vector entries; int line; const Entry* first(key); std::vector all(key); // pairs + blocks const std::string* first_value(key); std::vector values(key); const Node* first_block(key); std::vector blocks(key); std::vector items(); bool has(key); }; struct Document { Node root; std::vector warnings; }; Result parse_blocks(std::string_view, Options = {}); ``` Design points, each tied to a data fact: - **Keys are matched case-insensitively** (`first`/`all`/… use `iequals`), the original spelling is kept. The data mixes `Requires`/`requires`, `badge`/`Badge`; identifiers in values are also case-mixed but that is the consumer's concern (`iequals` is public for it). - **Repeated keys are not merged**: `all("requires")` returns them in file order. 970 `requires`, 3,721 `bank`, and the 19 banks that repeat `turretsize` all rely on order being kept. - **Pairs and blocks share the key namespace**: `first`/`all` see both because 153 shipsections carry `option DRV_X` beside `option { option A option B }`. `values()`/`blocks()` filter by kind when a consumer wants only one. - **Values are raw text**. `50`, `.5`, `7e+8`, `TRUE`, `@WEAPON_X` and `"8"` all stay as written, with `quoted` recording the difference. Conversion is explicit via `value.h`: `classify()` reports Int / Float / Bool / Text using the C-float shapes the data uses (`[+-]?d+`, `[+-]?(d+.d*|.d+|d+)([eE][+-]?d+)?`, `true`/`false` any case); `as_int`/`as_double`/`as_bool` convert. A quoted string is never a number to the oracle, and consumers should honour `quoted` the same way. - **Items**: bare quoted strings (`systemnames.txt` lists) and a lone trailing bareword are `Kind::Item`, reachable through `items()`. ## 2. `.effect` format — grammar as implemented Line-based, not brace-block. ``` line 1: TXT magic (surrounding whitespace ignored) KEY value [value ...] scalar entry; values are "quoted" or bare KEY followed (after blank/comment lines) by BEGIN ... END a nested group ``` - Lines split on LF; a trailing CR is dropped (three shipped files are CRLF). - `//` starts a comment unless inside double quotes; blank lines are skipped; indentation is cosmetic. - Tokens: `"…"` (may hold spaces — `NAME "New Emitter"`) or a run of non-whitespace. A `"` that never closes on its line is just part of a bare token. - A line with a single token is a group key and **must** be followed by `BEGIN`. `BEGIN` with no pending key, `END` with no open group, a group key followed by anything but `BEGIN`, a trailing group key, and unclosed groups at EOF are all errors. No leniency has been added: all 415 shipped files are balanced, and there is no evidence yet of what the engine would do otherwise. - Order is preserved and repeated keys are kept as separate entries because it is semantic: `PARTICLEDATATYPE n` precedes the curves that belong to it and `MODIFIER` repeats once per type. ```cpp struct EffectValue { std::string text; bool quoted; }; struct EffectEntry { std::string key; // as written (UPPERCASE in the data) std::vector values; // scalar line: 1+ values std::unique_ptr group; // group line int line; bool is_group() const; std::string_view value() const; // first value }; struct EffectGroup { std::vector entries; first(key); all(key); }; struct EffectFile { EffectGroup root; }; Result parse_effect(std::string_view); ``` ## 3. Tests `tests/mars_parse/build_and_run.sh` (plain `g++ -std=c++17 -Wall -Wextra -Wpedantic -Werror`): 1. **Unit tests** — 39 cases with hand-written samples: nesting, same-line blocks and glued braces, quoted values with spaces/backslashes/`//`, empty `""`, comments at token boundaries, repeated keys in order, case-insensitive lookup, quoted items and trailing barewords, `option` scalar+block, `@TOKEN` verbatim, EOF-closes-blocks and stray-`}` in lenient and strict mode, unterminated string and nameless `{` errors, CRLF and line numbers, multi-line strings, high bytes, and every effect error path. 2. **Real-data oracle test** — runs when `SOTS_DATA_DIR` points at an extracted `sots.gob` tree. `oracle/dump.py` runs the RE-repo readers (`SOTS_RE_PARSERS`, default `~/sots-re/verify/parsers`; exit 3 → the C++ side runs count-only) and writes a canonical JSON per file; `oracle_check` parses every file with `mars_parse`, renders the same canonical form (`canon.cpp`) and requires byte equality, plus equal lenient-warning counts. The canonical form is the oracle's own dict shape (keys lower-cased and sorted, repeats → lists, items under `_items`, bare pair values typed), with floats rendered by both sides as `"\x01" + printf("%.17g")` so formatting is shared, and cp1252 bytes rendered one `\u00xx` per byte. A one-character change in an oracle file is detected (verified). Skips cleanly with `SOTS_DATA_DIR` unset. Nothing from the game is committed; `tests/mars_parse/build/` (which holds the oracle JSON) is git-ignored. ### Oracle results (2026-09-07, owner's `gob-extract`) | kind | files | parsed | oracle match | need lenient recovery | |---|---|---|---|---| | shipsection | 875 | 875 | 875 | 12 | | weapon | 207 | 207 | 207 | 0 | | effect | 415 | 415 | 415 | — | | tech (MasterTechList) | 1 | 1 | 1 | 0 | | combat | 3 | 3 | 3 | 0 | | script | 4 | 4 | 4 | 0 | | def | 2 | 2 | 2 | 0 | | block-form txt (Scenarios + 7 Data/Models files) | 24 | 24 | 24 | 0 | | **total** | **1,531** | **1,531** | **1,531 (100%)** | 12 | The 12 strict-mode rejects are exactly the list in `data-parsers.md` (Hiver 6, Liir 4, Morrigi 1 unclosed; Human `CrPropaganda` stray `}`), and the per-file warning counts equal the oracle's. The remaining 64 of the Python suite's 1,595 files are CSV / flat key-value / manifest / positional tables — different readers, outside this module. ## 4. Open questions - **Engine leniency beyond the two observed repairs.** Unterminated strings, a nameless `{`, and malformed `.effect` nesting are errors here because no shipped file exercises them; the original may be more forgiving. Worth a shim-side probe before the loader is wired to user mods. - **`//` glued to a bareword** (`3//x`) is one token, matching the oracle. Whether the engine's tokenizer cuts a comment mid-word is unverified; the data never depends on it. - **Whitespace set.** The C++ uses the C `isspace` set; the Python oracle's `\s` also covers U+00A0 and a few control characters. The data contains neither (checked: all 1,531 files are pure ASCII), so this cannot be observed from data. - **Typed values.** The oracle types barewords (`50` → int) at parse time; this module keeps text and offers converters. If a consumer needs oracle-identical typing it should use `classify()` and honour `quoted`.