`mars::parse` — the engine's data-file readers

src/mars/parse/ reimplements the two text readers the original engine uses for its catalogs: the recursive brace-block key/value format (weapons, ship sections, the tech tree, combat missions, badge/light defs, GUI scripts, scenario and other block-form .txt) and the separate .effect format. C++17, no dependencies, no exceptions across the API, bytes passed through untouched (the data is cp1252).

Behaviour is specified by the proven Python readers in the RE repo (sots-re/verify/parsers/mars_data.py, effect_txt.py) and the findings in findings/subsystems/data-parsers.md. The C++ is written from that understanding, not transliterated; the Python readers serve as the test oracle.

src/mars/parse/
  result.h    Diagnostic {line, message}; Result<T>
  value.h/.cpp  classify / as_int / as_double / as_bool / iequals
  blocks.h/.cpp brace-block reader: parse_blocks(text, Options) -> Result<Document>
  effect.h/.cpp .effect reader:     parse_effect(text)          -> Result<EffectFile>
  CMakeLists.txt  static library `mars_parse` (include root: src/)
tests/mars_parse/
  build_and_run.sh   plain g++ build; unit tests; real-data oracle test
  test_*.cpp         hand-written samples, one test per quirk
  canon.h/.cpp       canonical JSON rendering (matches oracle/dump.py)
  oracle_check.cpp   parses every real file, diffs against the oracle
  oracle/dump.py     runs the RE-repo readers, emits canonical JSON per file

1. Brace-block format — grammar as implemented

body    := (block | pair | item)*
block   := NAME '{' body '}'
pair    := NAME value
item    := QUOTED                    -- a quoted string with no key
         | NAME                      -- a bareword directly before '}' or EOF
value   := QUOTED | BAREWORD
comment := '//' ... end-of-line      -- only at a token boundary

Tokens (whitespace-driven, not line-based):

token	rule
whitespace	space, tab, CR, LF, VT, FF
`{` `}`	single-character tokens; they also terminate a bareword (`weapon{` is two tokens)
QUOTED	`"` … `"`; no escapes at all — backslashes and `//` inside are literal; may span lines; an empty `""` is a valid value
BAREWORD	maximal run of anything that is not whitespace, `{`, `}` or `"` — so `abc"def"` is bareword `abc` followed by a quoted token, and `3//x` is one bareword
comment	`//` at the start of a token runs to end of line

The file's top level is itself a body: catalog files hold one or more named blocks, scenario .txt files mix top-level pairs and player{} blocks.

Disambiguation of a bareword N: look at the next token. { → a block named N; } or EOF → an item; anything else → a pair N value (the value may be bare or quoted). A quoted token in key position is always an item.

Leniency (what the shipped data needs)

condition	default (`Options::strict = false`)	`strict = true`
EOF while blocks are open	close every open block, one `Diagnostic` per open block in `Document::warnings`	error at the EOF line
`}` at top level	ignored, warning, parsing continues	error
unterminated `"`	error	error
`{` with no name before it	error	error

The two recoveries are exactly what 12 shipped shipsections require (11 never close their outer block, Human/CrPropaganda.shipsection has one } too many). They load in the game, so the engine performs the same repairs. Nothing in the shipped data hits the two hard errors.

Node model

struct Entry {                      // one thing in a block, in file order
    enum class Kind { Pair, Item, Block } kind;
    std::string key;                // Pair/Block: as written (case preserved)
    std::string value;              // Pair: value text; Item: the item text
    bool quoted;                    // value/item was in double quotes
    std::unique_ptr<Node> block;    // Block only
    int line;                       // 1-based
};
struct Node {                       // a named block; the root has name ""
    std::string name;
    std::vector<Entry> entries;
    int line;
    const Entry* first(key);  std::vector<const Entry*> all(key);      // pairs + blocks
    const std::string* first_value(key);  std::vector<std::string_view> values(key);
    const Node* first_block(key);        std::vector<const Node*> blocks(key);
    std::vector<std::string_view> items();  bool has(key);
};
struct Document { Node root; std::vector<Diagnostic> warnings; };
Result<Document> parse_blocks(std::string_view, Options = {});

Design points, each tied to a data fact:

Keys are matched case-insensitively (first/all/… use iequals), the original spelling is kept. The data mixes Requires/requires, badge/Badge; identifiers in values are also case-mixed but that is the consumer's concern (iequals is public for it).
Repeated keys are not merged: all("requires") returns them in file order. 970 requires, 3,721 bank, and the 19 banks that repeat turretsize all rely on order being kept.
Pairs and blocks share the key namespace: first/all see both because 153 shipsections carry option DRV_X beside option { option A option B }. values()/blocks() filter by kind when a consumer wants only one.
Values are raw text. 50, .5, 7e+8, TRUE, @WEAPON_X and "8" all stay as written, with quoted recording the difference. Conversion is explicit via value.h: classify() reports Int / Float / Bool / Text using the C-float shapes the data uses ([+-]?d+, [+-]?(d+.d*|.d+|d+)([eE][+-]?d+)?, true/false any case); as_int/as_double/as_bool convert. A quoted string is never a number to the oracle, and consumers should honour quoted the same way.
Items: bare quoted strings (systemnames.txt lists) and a lone trailing bareword are Kind::Item, reachable through items().

2. `.effect` format — grammar as implemented

Line-based, not brace-block.

line 1:  TXT                          magic (surrounding whitespace ignored)
         KEY value [value ...]        scalar entry; values are "quoted" or bare
         KEY                          followed (after blank/comment lines) by
             BEGIN ... END            a nested group

Lines split on LF; a trailing CR is dropped (three shipped files are CRLF).
// starts a comment unless inside double quotes; blank lines are skipped; indentation is cosmetic.
Tokens: "…" (may hold spaces — NAME "New Emitter") or a run of non-whitespace. A " that never closes on its line is just part of a bare token.
A line with a single token is a group key and must be followed by BEGIN. BEGIN with no pending key, END with no open group, a group key followed by anything but BEGIN, a trailing group key, and unclosed groups at EOF are all errors. No leniency has been added: all 415 shipped files are balanced, and there is no evidence yet of what the engine would do otherwise.
Order is preserved and repeated keys are kept as separate entries because it is semantic: PARTICLEDATATYPE n precedes the curves that belong to it and MODIFIER repeats once per type.

struct EffectValue { std::string text; bool quoted; };
struct EffectEntry {
    std::string key;                       // as written (UPPERCASE in the data)
    std::vector<EffectValue> values;       // scalar line: 1+ values
    std::unique_ptr<EffectGroup> group;    // group line
    int line;
    bool is_group() const;  std::string_view value() const;   // first value
};
struct EffectGroup { std::vector<EffectEntry> entries; first(key); all(key); };
struct EffectFile  { EffectGroup root; };
Result<EffectFile> parse_effect(std::string_view);

3. Tests

tests/mars_parse/build_and_run.sh (plain g++ -std=c++17 -Wall -Wextra -Wpedantic -Werror):

Unit tests — 39 cases with hand-written samples: nesting, same-line blocks and glued braces, quoted values with spaces/backslashes///, empty "", comments at token boundaries, repeated keys in order, case-insensitive lookup, quoted items and trailing barewords, option scalar+block, @TOKEN verbatim, EOF-closes-blocks and stray-} in lenient and strict mode, unterminated string and nameless { errors, CRLF and line numbers, multi-line strings, high bytes, and every effect error path.
Real-data oracle test — runs when SOTS_DATA_DIR points at an extracted sots.gob tree. oracle/dump.py runs the RE-repo readers (SOTS_RE_PARSERS, default ~/sots-re/verify/parsers; exit 3 → the C++ side runs count-only) and writes a canonical JSON per file; oracle_check parses every file with mars_parse, renders the same canonical form (canon.cpp) and requires byte equality, plus equal lenient-warning counts.

The canonical form is the oracle's own dict shape (keys lower-cased and sorted, repeats → lists, items under _items, bare pair values typed), with floats rendered by both sides as "\x01" + printf("%.17g") so formatting is shared, and cp1252 bytes rendered one \u00xx per byte. A one-character change in an oracle file is detected (verified).

Skips cleanly with SOTS_DATA_DIR unset. Nothing from the game is committed; tests/mars_parse/build/ (which holds the oracle JSON) is git-ignored.

Oracle results (2026-09-07, owner's `gob-extract`)

kind	files	parsed	oracle match	need lenient recovery
shipsection	875	875	875	12
weapon	207	207	207	0
effect	415	415	415	—
tech (MasterTechList)	1	1	1	0
combat	3	3	3	0
script	4	4	4	0
def	2	2	2	0
block-form txt (Scenarios + 7 Data/Models files)	24	24	24	0
total	1,531	1,531	1,531 (100%)	12

The 12 strict-mode rejects are exactly the list in data-parsers.md (Hiver 6, Liir 4, Morrigi 1 unclosed; Human CrPropaganda stray }), and the per-file warning counts equal the oracle's. The remaining 64 of the Python suite's 1,595 files are CSV / flat key-value / manifest / positional tables — different readers, outside this module.

4. Open questions

Engine leniency beyond the two observed repairs. Unterminated strings, a nameless {, and malformed .effect nesting are errors here because no shipped file exercises them; the original may be more forgiving. Worth a shim-side probe before the loader is wired to user mods.
// glued to a bareword (3//x) is one token, matching the oracle. Whether the engine's tokenizer cuts a comment mid-word is unverified; the data never depends on it.
Whitespace set. The C++ uses the C isspace set; the Python oracle's \s also covers U+00A0 and a few control characters. The data contains neither (checked: all 1,531 files are pure ASCII), so this cannot be observed from data.
Typed values. The oracle types barewords (50 → int) at parse time; this module keeps text and offers converters. If a consumer needs oracle-identical typing it should use classify() and honour quoted.

11 KiB Raw Blame History

mars::parse — the engine's data-file readers