222 lines
11 KiB
Markdown
222 lines
11 KiB
Markdown
# `mars::parse` — the engine's data-file readers
|
|
|
|
`src/mars/parse/` reimplements the two text readers the original engine uses
|
|
for its catalogs: the recursive **brace-block** key/value format (weapons, ship
|
|
sections, the tech tree, combat missions, badge/light defs, GUI scripts,
|
|
scenario and other block-form `.txt`) and the separate **`.effect`** format.
|
|
C++17, no dependencies, no exceptions across the API, bytes passed through
|
|
untouched (the data is cp1252).
|
|
|
|
Behaviour is specified by the proven Python readers in the RE repo
|
|
(`sots-re/verify/parsers/mars_data.py`, `effect_txt.py`) and the findings in
|
|
`findings/subsystems/data-parsers.md`. The C++ is written from that
|
|
understanding, not transliterated; the Python readers serve as the test oracle.
|
|
|
|
```
|
|
src/mars/parse/
|
|
result.h Diagnostic {line, message}; Result<T>
|
|
value.h/.cpp classify / as_int / as_double / as_bool / iequals
|
|
blocks.h/.cpp brace-block reader: parse_blocks(text, Options) -> Result<Document>
|
|
effect.h/.cpp .effect reader: parse_effect(text) -> Result<EffectFile>
|
|
CMakeLists.txt static library `mars_parse` (include root: src/)
|
|
tests/mars_parse/
|
|
build_and_run.sh plain g++ build; unit tests; real-data oracle test
|
|
test_*.cpp hand-written samples, one test per quirk
|
|
canon.h/.cpp canonical JSON rendering (matches oracle/dump.py)
|
|
oracle_check.cpp parses every real file, diffs against the oracle
|
|
oracle/dump.py runs the RE-repo readers, emits canonical JSON per file
|
|
```
|
|
|
|
## 1. Brace-block format — grammar as implemented
|
|
|
|
```
|
|
body := (block | pair | item)*
|
|
block := NAME '{' body '}'
|
|
pair := NAME value
|
|
item := QUOTED -- a quoted string with no key
|
|
| NAME -- a bareword directly before '}' or EOF
|
|
value := QUOTED | BAREWORD
|
|
comment := '//' ... end-of-line -- only at a token boundary
|
|
```
|
|
|
|
Tokens (whitespace-driven, not line-based):
|
|
|
|
| token | rule |
|
|
|---|---|
|
|
| whitespace | space, tab, CR, LF, VT, FF |
|
|
| `{` `}` | single-character tokens; they also terminate a bareword (`weapon{` is two tokens) |
|
|
| QUOTED | `"` … `"`; no escapes at all — backslashes and `//` inside are literal; may span lines; an empty `""` is a valid value |
|
|
| BAREWORD | maximal run of anything that is not whitespace, `{`, `}` or `"` — so `abc"def"` is bareword `abc` followed by a quoted token, and `3//x` is one bareword |
|
|
| comment | `//` at the start of a token runs to end of line |
|
|
|
|
The file's top level is itself a body: catalog files hold one or more named
|
|
blocks, scenario `.txt` files mix top-level pairs and `player{}` blocks.
|
|
|
|
Disambiguation of a bareword `N`: look at the next token. `{` → a block named
|
|
`N`; `}` or EOF → an item; anything else → a pair `N value` (the value may be
|
|
bare or quoted). A quoted token in key position is always an item.
|
|
|
|
### Leniency (what the shipped data needs)
|
|
|
|
| condition | default (`Options::strict = false`) | `strict = true` |
|
|
|---|---|---|
|
|
| EOF while blocks are open | close every open block, one `Diagnostic` per open block in `Document::warnings` | error at the EOF line |
|
|
| `}` at top level | ignored, warning, parsing continues | error |
|
|
| unterminated `"` | error | error |
|
|
| `{` with no name before it | error | error |
|
|
|
|
The two recoveries are exactly what 12 shipped shipsections require (11 never
|
|
close their outer block, `Human/CrPropaganda.shipsection` has one `}` too many).
|
|
They load in the game, so the engine performs the same repairs. Nothing in the
|
|
shipped data hits the two hard errors.
|
|
|
|
### Node model
|
|
|
|
```cpp
|
|
struct Entry { // one thing in a block, in file order
|
|
enum class Kind { Pair, Item, Block } kind;
|
|
std::string key; // Pair/Block: as written (case preserved)
|
|
std::string value; // Pair: value text; Item: the item text
|
|
bool quoted; // value/item was in double quotes
|
|
std::unique_ptr<Node> block; // Block only
|
|
int line; // 1-based
|
|
};
|
|
struct Node { // a named block; the root has name ""
|
|
std::string name;
|
|
std::vector<Entry> entries;
|
|
int line;
|
|
const Entry* first(key); std::vector<const Entry*> all(key); // pairs + blocks
|
|
const std::string* first_value(key); std::vector<std::string_view> values(key);
|
|
const Node* first_block(key); std::vector<const Node*> blocks(key);
|
|
std::vector<std::string_view> items(); bool has(key);
|
|
};
|
|
struct Document { Node root; std::vector<Diagnostic> warnings; };
|
|
Result<Document> parse_blocks(std::string_view, Options = {});
|
|
```
|
|
|
|
Design points, each tied to a data fact:
|
|
|
|
- **Keys are matched case-insensitively** (`first`/`all`/… use `iequals`), the
|
|
original spelling is kept. The data mixes `Requires`/`requires`,
|
|
`badge`/`Badge`; identifiers in values are also case-mixed but that is the
|
|
consumer's concern (`iequals` is public for it).
|
|
- **Repeated keys are not merged**: `all("requires")` returns them in file
|
|
order. 970 `requires`, 3,721 `bank`, and the 19 banks that repeat
|
|
`turretsize` all rely on order being kept.
|
|
- **Pairs and blocks share the key namespace**: `first`/`all` see both because
|
|
153 shipsections carry `option DRV_X` beside `option { option A option B }`.
|
|
`values()`/`blocks()` filter by kind when a consumer wants only one.
|
|
- **Values are raw text**. `50`, `.5`, `7e+8`, `TRUE`, `@WEAPON_X` and `"8"`
|
|
all stay as written, with `quoted` recording the difference. Conversion is
|
|
explicit via `value.h`: `classify()` reports Int / Float / Bool / Text using
|
|
the C-float shapes the data uses (`[+-]?d+`, `[+-]?(d+.d*|.d+|d+)([eE][+-]?d+)?`,
|
|
`true`/`false` any case); `as_int`/`as_double`/`as_bool` convert. A quoted
|
|
string is never a number to the oracle, and consumers should honour `quoted`
|
|
the same way.
|
|
- **Items**: bare quoted strings (`systemnames.txt` lists) and a lone trailing
|
|
bareword are `Kind::Item`, reachable through `items()`.
|
|
|
|
## 2. `.effect` format — grammar as implemented
|
|
|
|
Line-based, not brace-block.
|
|
|
|
```
|
|
line 1: TXT magic (surrounding whitespace ignored)
|
|
KEY value [value ...] scalar entry; values are "quoted" or bare
|
|
KEY followed (after blank/comment lines) by
|
|
BEGIN ... END a nested group
|
|
```
|
|
|
|
- Lines split on LF; a trailing CR is dropped (three shipped files are CRLF).
|
|
- `//` starts a comment unless inside double quotes; blank lines are skipped;
|
|
indentation is cosmetic.
|
|
- Tokens: `"…"` (may hold spaces — `NAME "New Emitter"`) or a run of
|
|
non-whitespace. A `"` that never closes on its line is just part of a bare
|
|
token.
|
|
- A line with a single token is a group key and **must** be followed by
|
|
`BEGIN`. `BEGIN` with no pending key, `END` with no open group, a group key
|
|
followed by anything but `BEGIN`, a trailing group key, and unclosed groups
|
|
at EOF are all errors. No leniency has been added: all 415 shipped files are
|
|
balanced, and there is no evidence yet of what the engine would do otherwise.
|
|
- Order is preserved and repeated keys are kept as separate entries because it
|
|
is semantic: `PARTICLEDATATYPE n` precedes the curves that belong to it and
|
|
`MODIFIER` repeats once per type.
|
|
|
|
```cpp
|
|
struct EffectValue { std::string text; bool quoted; };
|
|
struct EffectEntry {
|
|
std::string key; // as written (UPPERCASE in the data)
|
|
std::vector<EffectValue> values; // scalar line: 1+ values
|
|
std::unique_ptr<EffectGroup> group; // group line
|
|
int line;
|
|
bool is_group() const; std::string_view value() const; // first value
|
|
};
|
|
struct EffectGroup { std::vector<EffectEntry> entries; first(key); all(key); };
|
|
struct EffectFile { EffectGroup root; };
|
|
Result<EffectFile> parse_effect(std::string_view);
|
|
```
|
|
|
|
## 3. Tests
|
|
|
|
`tests/mars_parse/build_and_run.sh` (plain `g++ -std=c++17 -Wall -Wextra
|
|
-Wpedantic -Werror`):
|
|
|
|
1. **Unit tests** — 39 cases with hand-written samples: nesting, same-line
|
|
blocks and glued braces, quoted values with spaces/backslashes/`//`, empty
|
|
`""`, comments at token boundaries, repeated keys in order, case-insensitive
|
|
lookup, quoted items and trailing barewords, `option` scalar+block, `@TOKEN`
|
|
verbatim, EOF-closes-blocks and stray-`}` in lenient and strict mode,
|
|
unterminated string and nameless `{` errors, CRLF and line numbers,
|
|
multi-line strings, high bytes, and every effect error path.
|
|
2. **Real-data oracle test** — runs when `SOTS_DATA_DIR` points at an
|
|
extracted `sots.gob` tree. `oracle/dump.py` runs the RE-repo readers
|
|
(`SOTS_RE_PARSERS`, default `~/sots-re/verify/parsers`; exit 3 → the C++
|
|
side runs count-only) and writes a canonical JSON per file; `oracle_check`
|
|
parses every file with `mars_parse`, renders the same canonical form
|
|
(`canon.cpp`) and requires byte equality, plus equal lenient-warning counts.
|
|
|
|
The canonical form is the oracle's own dict shape (keys lower-cased and
|
|
sorted, repeats → lists, items under `_items`, bare pair values typed), with
|
|
floats rendered by both sides as `"\x01" + printf("%.17g")` so formatting is
|
|
shared, and cp1252 bytes rendered one `\u00xx` per byte. A one-character
|
|
change in an oracle file is detected (verified).
|
|
|
|
Skips cleanly with `SOTS_DATA_DIR` unset. Nothing from the game is committed;
|
|
`tests/mars_parse/build/` (which holds the oracle JSON) is git-ignored.
|
|
|
|
### Oracle results (2026-09-07, owner's `gob-extract`)
|
|
|
|
| kind | files | parsed | oracle match | need lenient recovery |
|
|
|---|---|---|---|---|
|
|
| shipsection | 875 | 875 | 875 | 12 |
|
|
| weapon | 207 | 207 | 207 | 0 |
|
|
| effect | 415 | 415 | 415 | — |
|
|
| tech (MasterTechList) | 1 | 1 | 1 | 0 |
|
|
| combat | 3 | 3 | 3 | 0 |
|
|
| script | 4 | 4 | 4 | 0 |
|
|
| def | 2 | 2 | 2 | 0 |
|
|
| block-form txt (Scenarios + 7 Data/Models files) | 24 | 24 | 24 | 0 |
|
|
| **total** | **1,531** | **1,531** | **1,531 (100%)** | 12 |
|
|
|
|
The 12 strict-mode rejects are exactly the list in `data-parsers.md`
|
|
(Hiver 6, Liir 4, Morrigi 1 unclosed; Human `CrPropaganda` stray `}`), and the
|
|
per-file warning counts equal the oracle's. The remaining 64 of the Python
|
|
suite's 1,595 files are CSV / flat key-value / manifest / positional tables —
|
|
different readers, outside this module.
|
|
|
|
## 4. Open questions
|
|
|
|
- **Engine leniency beyond the two observed repairs.** Unterminated strings,
|
|
a nameless `{`, and malformed `.effect` nesting are errors here because no
|
|
shipped file exercises them; the original may be more forgiving. Worth a
|
|
shim-side probe before the loader is wired to user mods.
|
|
- **`//` glued to a bareword** (`3//x`) is one token, matching the oracle.
|
|
Whether the engine's tokenizer cuts a comment mid-word is unverified; the
|
|
data never depends on it.
|
|
- **Whitespace set.** The C++ uses the C `isspace` set; the Python oracle's
|
|
`\s` also covers U+00A0 and a few control characters. The data contains
|
|
neither (checked: all 1,531 files are pure ASCII), so this cannot be
|
|
observed from data.
|
|
- **Typed values.** The oracle types barewords (`50` → int) at parse time;
|
|
this module keeps text and offers converters. If a consumer needs
|
|
oracle-identical typing it should use `classify()` and honour `quoted`.
|