alex/sots-engine
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

mars/parse + mars/text: engine parity (Mars::Script tokenizer, first-wins keys, trailing-pair drop); oracles 1531/1531, 64/64

Browse source
This commit is contained in:
alex 2026-09-07 18:10:18 -04:00
parent 9e110b43ba
commit 9ccde259aa
13 changed files with 955 additions and 481 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

210
docs/mars-parse.md
View file

@ -1,85 +1,107 @@
# `mars::parse` — the engine's data-file readers # `mars::parse` — the engine's data-file readers
`src/mars/parse/` reimplements the two text readers the original engine uses `src/mars/parse/` reimplements the two text readers the original engine uses
for its catalogs: the recursive **brace-block** key/value format (weapons, ship for its catalogs: the **brace-block** key/value format (weapons, ship
sections, the tech tree, combat missions, badge/light defs, GUI scripts, sections, the tech tree, combat missions, badge/light defs, GUI scripts,
scenario and other block-form `.txt`) and the separate **`.effect`** format. scenario and other block-form `.txt`) and the separate **`.effect`** format.
C++17, no dependencies, no exceptions across the API, bytes passed through C++17, no dependencies, no exceptions across the API, bytes passed through
untouched (the data is cp1252). untouched (the data is cp1252).
Behaviour is specified by the proven Python readers in the RE repo Behaviour is specified by the original engine's `Mars::Script` tokenizer as
(`sots-re/verify/parsers/mars_data.py`, `effect_txt.py`) and the findings in recovered in the RE repo (`findings/subsystems/loader-prototypes.md` §M3) and
`findings/subsystems/data-parsers.md`. The C++ is written from that mirrored by the RE repo's Python readers (`sots-re/verify/parsers/mars_data.py`,
understanding, not transliterated; the Python readers serve as the test oracle. `effect_txt.py`), which serve as the test oracle. The C++ is written from that
understanding, not transliterated.
``` ```
src/mars/parse/ src/mars/parse/
result.h Diagnostic {line, message}; Result<T> result.h Diagnostic {line, message}; Result<T>
value.h/.cpp classify / as_int / as_double / as_bool / iequals value.h/.cpp classify / as_int / as_double / as_bool / iequals
script.h header-only pull tokenizer with the engine's semantics
(read_token / next / skip_block); shared with mars::text
blocks.h/.cpp brace-block reader: parse_blocks(text, Options) -> Result<Document> blocks.h/.cpp brace-block reader: parse_blocks(text, Options) -> Result<Document>
effect.h/.cpp .effect reader: parse_effect(text) -> Result<EffectFile> effect.h/.cpp .effect reader: parse_effect(text) -> Result<EffectFile>
CMakeLists.txt static library `mars_parse` (include root: src/) CMakeLists.txt static library `mars_parse` (include root: src/)
tests/mars_parse/ tests/mars_parse/
build_and_run.sh plain g++ build; unit tests; real-data oracle test build_and_run.sh plain g++ build; unit tests; real-data oracle test
test_*.cpp hand-written samples, one test per quirk test_*.cpp hand-written samples, one test per rule
canon.h/.cpp canonical JSON rendering (matches oracle/dump.py) canon.h/.cpp canonical JSON rendering (matches oracle/dump.py)
oracle_check.cpp parses every real file, diffs against the oracle oracle_check.cpp parses every real file, diffs against the oracle
oracle/dump.py runs the RE-repo readers, emits canonical JSON per file oracle/dump.py runs the RE-repo readers, emits canonical JSON per file
``` ```
## 1. Brace-block format — grammar as implemented ## 1. The tokenizer — `Script` (script.h)
``` The original has no tree. Every loader (`WeaponDef::ParseScript`,
body := (block | pair | item)* `SectionDef::ParseScript`, `MasterTechTree::ParseTech`, the GUI script
block := NAME '{' body '}' callbacks, `GlobalConsts::LoadFile`) drives one pull tokenizer with three
pair := NAME value operations and interprets the steps itself:
item := QUOTED -- a quoted string with no key
| NAME -- a bareword directly before '}' or EOF
value := QUOTED | BAREWORD
comment := '//' ... end-of-line -- only at a token boundary
```
Tokens (whitespace-driven, not line-based): | operation | what it does |
| token | rule |
|---|---| |---|---|
| whitespace | space, tab, CR, LF, VT, FF | | `read_token(Raw&)` | one token. `Ok` = read, input remains; `AtEnd` = read, the token touches the end of input; `NoInput` = nothing to read |
| `{` `}` | single-character tokens; they also terminate a bareword (`weapon{` is two tokens) | | `next(ScriptToken&)` | reads a key; if it is `}` the step is `Close`. Otherwise reads a value; `Open` if the value is `{`, else `Pair`. Returns the status of the **value** read |
| QUOTED | `"` … `"`; no escapes at all — backslashes and `//` inside are literal; may span lines; an empty `""` is a valid value | | `skip_block(depth)` | consumes tokens counting whole `{`/`}` tokens until depth 0 |
| BAREWORD | maximal run of anything that is not whitespace, `{`, `}` or `"` — so `abc"def"` is bareword `abc` followed by a quoted token, and `3//x` is one bareword |
| comment | `//` at the start of a token runs to end of line |
Token rules:
| rule | detail |
|---|---|
| whitespace | space, tab, CR, LF — nothing else (`\v`, `\f`, NBSP are token bytes) |
| bareword | a run of non-whitespace. **Braces are not delimiters**: `{` and `}` count only as whole tokens, so `weapon{` is one word and `abc"def"` is one word |
| quoted | a token starting with `"`, `'` or a backtick ends at the next occurrence of that same character; the quotes are stripped, no escape processing, whitespace and newlines inside are kept; an unterminated quote runs to end of input. Text glued after a closing quote starts the next token |
| comment | a token whose extracted text starts with `//` — bare or quoted — is a comment: the rest of the line is skipped and reading continues. `3//x` is one word; `"0 0 0"// junk` is the value `0 0 0` then a comment |
| length | text is capped at 1023 bytes (the whole token is still consumed) |
| brace test | exact compare on the stripped text, so a quoted `"{"` / `"}"` acts as a brace |
`Script` is public so a per-consumer loader can be written against the same
step stream when a catalog needs the original's own key handling.
## 2. Brace-block trees — `parse_blocks`
`parse_blocks` runs the loaders' `while (next() == Ok)` loop and records the
steps: `Pair` → `Entry::Kind::Pair` (the key may be quoted — `key_quoted`),
`Open` → `Entry::Kind::Block` with a recursive body, `Close` → the block ends.
The file's top level is itself a body: catalog files hold one or more named The file's top level is itself a body: catalog files hold one or more named
blocks, scenario `.txt` files mix top-level pairs and `player{}` blocks. blocks, scenario `.txt` files mix top-level pairs and `player{}` blocks.
Disambiguation of a bareword `N`: look at the next token. `{` → a block named There is no "item" concept: a quoted string in key position is a key
`N`; `}` or EOF → an item; anything else → a pair `N value` (the value may be (`systemnames.txt` lists become pairs of consecutive names), and a lone word
bare or quoted). A quoted token in key position is always an item. before `}` takes the `}` as its value. That is what the original sees.
### Leniency (what the shipped data needs) ### End of input and other tolerances
| condition | default (`Options::strict = false`) | `strict = true` | Everything the original merely tolerates is recorded as a `Diagnostic` in
|---|---|---| `Document::warnings`; with `Options::strict` each one is an error instead.
| EOF while blocks are open | close every open block, one `Diagnostic` per open block in `Document::warnings` | error at the EOF line |
| `}` at top level | ignored, warning, parsing continues | error |
| unterminated `"` | error | error |
| `{` with no name before it | error | error |
The two recoveries are exactly what 12 shipped shipsections require (11 never | condition | what the original does (and this reader) |
close their outer block, `Human/CrPropaganda.shipsection` has one `}` too many). |---|---|
They load in the game, so the engine performs the same repairs. Nothing in the | input ends inside a block | the block simply ends (one warning per open block) |
shipped data hits the two hard errors. | `}` at top level | ignored |
| final `KEY value` whose value token touches the end of input (no trailing newline) | the step returns `AtEnd` and the pair is **dropped** |
| final KEY with no value; final `NAME {` touching the end | dropped |
| unterminated quote | runs to end of input (and, touching it, that pair is dropped) |
| `{` in key position | an ordinary key; kept |
| `}` that closes a block and touches the end of input | a plain close — no warning; the original stops there either way and the outcome is identical |
The shipped data exercises: 12 shipsections (11 unclosed outer blocks, Human
`CrPropaganda` with one `}` too many) and `Data/Strategy/systemnames.txt`,
whose `hiver` and `liir` lists have an odd number of names — the last name
pairs with the `}`, so `tarkas`/`liir` nest inside `hiver` and `morrigi`
inside `liir`, and two blocks are left open at the end. 27 catalog files end
with a `}` as their last byte (plain close). No brace file drops a pair.
### Node model ### Node model
```cpp ```cpp
struct Entry { // one thing in a block, in file order struct Entry { // one step in a block, in file order
enum class Kind { Pair, Item, Block } kind; enum class Kind { Pair, Block } kind;
std::string key; // Pair/Block: as written (case preserved) std::string key; // as written (case preserved, quotes stripped)
std::string value; // Pair: value text; Item: the item text std::string value; // Pair: value text
bool quoted; // value/item was in double quotes bool quoted; // value was quoted
bool key_quoted; // key was quoted
std::unique_ptr<Node> block; // Block only std::unique_ptr<Node> block; // Block only
int line; // 1-based int line; // 1-based, counts '\n'
}; };
struct Node { // a named block; the root has name "" struct Node { // a named block; the root has name ""
std::string name; std::string name;
@ -88,7 +110,7 @@ struct Node { // a named block; the root has name ""
const Entry* first(key); std::vector<const Entry*> all(key); // pairs + blocks const Entry* first(key); std::vector<const Entry*> all(key); // pairs + blocks
const std::string* first_value(key); std::vector<std::string_view> values(key); const std::string* first_value(key); std::vector<std::string_view> values(key);
const Node* first_block(key); std::vector<const Node*> blocks(key); const Node* first_block(key); std::vector<const Node*> blocks(key);
std::vector<std::string_view> items(); bool has(key); bool has(key);
}; };
struct Document { Node root; std::vector<Diagnostic> warnings; }; struct Document { Node root; std::vector<Diagnostic> warnings; };
Result<Document> parse_blocks(std::string_view, Options = {}); Result<Document> parse_blocks(std::string_view, Options = {});
@ -96,10 +118,9 @@ Result<Document> parse_blocks(std::string_view, Options = {});
Design points, each tied to a data fact: Design points, each tied to a data fact:
- **Keys are matched case-insensitively** (`first`/`all`/… use `iequals`), the - **Keys are matched case-insensitively** (`first`/`all`/… use `iequals`, the
original spelling is kept. The data mixes `Requires`/`requires`, original uses `_stricmp`), the original spelling is kept. The data mixes
`badge`/`Badge`; identifiers in values are also case-mixed but that is the `Requires`/`requires`, `badge`/`Badge`.
consumer's concern (`iequals` is public for it).
- **Repeated keys are not merged**: `all("requires")` returns them in file - **Repeated keys are not merged**: `all("requires")` returns them in file
order. 970 `requires`, 3,721 `bank`, and the 19 banks that repeat order. 970 `requires`, 3,721 `bank`, and the 19 banks that repeat
`turretsize` all rely on order being kept. `turretsize` all rely on order being kept.
@ -109,16 +130,14 @@ Design points, each tied to a data fact:
- **Values are raw text**. `50`, `.5`, `7e+8`, `TRUE`, `@WEAPON_X` and `"8"` - **Values are raw text**. `50`, `.5`, `7e+8`, `TRUE`, `@WEAPON_X` and `"8"`
all stay as written, with `quoted` recording the difference. Conversion is all stay as written, with `quoted` recording the difference. Conversion is
explicit via `value.h`: `classify()` reports Int / Float / Bool / Text using explicit via `value.h`: `classify()` reports Int / Float / Bool / Text using
the C-float shapes the data uses (`[+-]?d+`, `[+-]?(d+.d*|.d+|d+)([eE][+-]?d+)?`, the C-float shapes the data uses; `as_int`/`as_double`/`as_bool` convert.
`true`/`false` any case); `as_int`/`as_double`/`as_bool` convert. A quoted (The original uses `atoi`/`atof` per key, so quoting never changes typing
string is never a number to the oracle, and consumers should honour `quoted` there; the oracle's canonical form types only barewords, and consumers
the same way. wanting engine typing should convert regardless of `quoted`.)
- **Items**: bare quoted strings (`systemnames.txt` lists) and a lone trailing
bareword are `Kind::Item`, reachable through `items()`.
## 2. `.effect` format — grammar as implemented ## 3. `.effect` format — grammar as implemented
Line-based, not brace-block. Line-based, not brace-block (a different engine class, `Mars::TextFileStream`).
``` ```
line 1: TXT magic (surrounding whitespace ignored) line 1: TXT magic (surrounding whitespace ignored)
@ -156,37 +175,41 @@ struct EffectFile { EffectGroup root; };
Result<EffectFile> parse_effect(std::string_view); Result<EffectFile> parse_effect(std::string_view);
``` ```
## 3. Tests ## 4. Tests
`tests/mars_parse/build_and_run.sh` (plain `g++ -std=c++17 -Wall -Wextra `tests/mars_parse/build_and_run.sh` (plain `g++ -std=c++17 -Wall -Wextra
-Wpedantic -Werror`): -Wpedantic -Werror`):
1. **Unit tests** — 39 cases with hand-written samples: nesting, same-line 1. **Unit tests** — 51 cases with hand-written samples: the `Script` status
blocks and glued braces, quoted values with spaces/backslashes/`//`, empty codes, `next()` step shapes, `skip_block`, the 1023-byte cap; nesting,
`""`, comments at token boundaries, repeated keys in order, case-insensitive same-line blocks, braces glued to words, the three quote characters,
lookup, quoted items and trailing barewords, `option` scalar+block, `@TOKEN` quotes inside words, quoted braces, the four-character whitespace set,
verbatim, EOF-closes-blocks and stray-`}` in lenient and strict mode, comments bare / glued / quoted / at end of input, repeated keys in order,
unterminated string and nameless `{` errors, CRLF and line numbers, case-insensitive lookup, quoted keys and odd-count lists, lone word before
multi-line strings, high bytes, and every effect error path. `}`, `option` scalar+block, `@TOKEN` verbatim, EOF-ends-blocks, stray `}`,
`}` as last byte, dropped final pair / key / block header, unterminated
quote, `{` as key — each in lenient and strict mode where it applies —
CRLF and line numbers, multi-line strings, high bytes, and every effect
error path.
2. **Real-data oracle test** — runs when `SOTS_DATA_DIR` points at an 2. **Real-data oracle test** — runs when `SOTS_DATA_DIR` points at an
extracted `sots.gob` tree. `oracle/dump.py` runs the RE-repo readers extracted `sots.gob` tree. `oracle/dump.py` runs the RE-repo readers
(`SOTS_RE_PARSERS`, default `~/sots-re/verify/parsers`; exit 3 → the C++ (`SOTS_RE_PARSERS`, default `~/sots-re/verify/parsers`; exit 3 → the C++
side runs count-only) and writes a canonical JSON per file; `oracle_check` side runs count-only) and writes a canonical JSON per file; `oracle_check`
parses every file with `mars_parse`, renders the same canonical form parses every file with `mars_parse`, renders the same canonical form
(`canon.cpp`) and requires byte equality, plus equal lenient-warning counts. (`canon.cpp`) and requires byte equality, plus equal warning counts.
The canonical form is the oracle's own dict shape (keys lower-cased and The canonical form is the oracle's own dict shape (keys lower-cased and
sorted, repeats → lists, items under `_items`, bare pair values typed), with sorted, repeats → lists, bare pair values typed), with floats rendered by both
floats rendered by both sides as `"\x01" + printf("%.17g")` so formatting is sides as `"\x01" + printf("%.17g")` so formatting is shared, and cp1252 bytes
shared, and cp1252 bytes rendered one `\u00xx` per byte. A one-character rendered one `\u00xx` per byte. A one-character change in an oracle file is
change in an oracle file is detected (verified). detected (verified).
Skips cleanly with `SOTS_DATA_DIR` unset. Nothing from the game is committed; Skips cleanly with `SOTS_DATA_DIR` unset. Nothing from the game is committed;
`tests/mars_parse/build/` (which holds the oracle JSON) is git-ignored. `tests/mars_parse/build/` (which holds the oracle JSON) is git-ignored.
### Oracle results (2026-09-07, owner's `gob-extract`) ### Oracle results (2026-09-07, engine-parity rules, owner's `gob-extract`)
| kind | files | parsed | oracle match | need lenient recovery | | kind | files | parsed | oracle match | tolerances taken |
|---|---|---|---|---| |---|---|---|---|---|
| shipsection | 875 | 875 | 875 | 12 | | shipsection | 875 | 875 | 875 | 12 |
| weapon | 207 | 207 | 207 | 0 | | weapon | 207 | 207 | 207 | 0 |
@ -195,28 +218,25 @@ Skips cleanly with `SOTS_DATA_DIR` unset. Nothing from the game is committed;
| combat | 3 | 3 | 3 | 0 | | combat | 3 | 3 | 3 | 0 |
| script | 4 | 4 | 4 | 0 | | script | 4 | 4 | 4 | 0 |
| def | 2 | 2 | 2 | 0 | | def | 2 | 2 | 2 | 0 |
| block-form txt (Scenarios + 7 Data/Models files) | 24 | 24 | 24 | 0 | | block-form txt (Scenarios + 7 Data/Models files) | 24 | 24 | 24 | 1 (`systemnames.txt`) |
| **total** | **1,531** | **1,531** | **1,531 (100%)** | 12 | | **total** | **1,531** | **1,531** | **1,531 (100%)** | 13 |
The 12 strict-mode rejects are exactly the list in `data-parsers.md` Against the previous (tree-grammar) reading, exactly one file's canonical
(Hiver 6, Liir 4, Morrigi 1 unclosed; Human `CrPropaganda` stray `}`), and the output changed: `Data/Strategy/systemnames.txt` (quoted names become pairs;
per-file warning counts equal the oracle's. The remaining 64 of the Python odd-count lists nest the following blocks). Every weapon, shipsection, tech,
suite's 1,595 files are CSV / flat key-value / manifest / positional tables — combat, def, script and scenario file renders byte-identically, so nothing the
different readers, outside this module. cross-links or catalogs consume moved. The remaining 64 of the Python suite's
1,595 files are CSV / flat key-value / manifest / positional tables — see
`docs/mars-text.md`.
## 4. Open questions ## 5. Open questions
- **Engine leniency beyond the two observed repairs.** Unterminated strings, - **`systemnames.txt` consumer.** The generic step stream pairs its names up;
a nameless `{`, and malformed `.effect` nesting are errors here because no the original's system-name loader "handles that format itself" (RE notes)
shipped file exercises them; the original may be more forgiving. Worth a and its exact handling of the odd-count lists is not recovered. A dedicated
shim-side probe before the loader is wired to user mods. reader over `Script` will be needed for name generation parity.
- **`//` glued to a bareword** (`3//x`) is one token, matching the oracle. - **`skip_block` granularity.** Assumed to count raw `{`/`}` tokens (so a
Whether the engine's tokenizer cuts a comment mid-word is unverified; the `}` in value position inside a skipped block still closes it, unlike
data never depends on it. `next()`). No shipped file distinguishes the two.
- **Whitespace set.** The C++ uses the C `isspace` set; the Python oracle's - **Malformed `.effect` nesting** stays an error here because no shipped file
`\s` also covers U+00A0 and a few control characters. The data contains exercises it; the original may be more forgiving.
neither (checked: all 1,531 files are pure ASCII), so this cannot be
observed from data.
- **Typed values.** The oracle types barewords (`50` → int) at parse time;
this module keeps text and offers converters. If a consumer needs
oracle-identical typing it should use `classify()` and honour `quoted`.

152
docs/mars-text.md
View file

@ -7,9 +7,12 @@ CSVs. Library target: `mars_text` (static, `src/mars/text/CMakeLists.txt`).
Include as `mars/text/<header>.h`. C++17, no dependencies, no exceptions cross Include as `mars/text/<header>.h`. C++17, no dependencies, no exceptions cross
the API; every reader is total and reports oddities as `Problem`s. the API; every reader is total and reports oddities as `Problem`s.
The behaviour spec is the RE repo's proven Python readers The behaviour spec for the `KEY value` tables is the original engine's
(`sots-re/verify/parsers/flat_kv.py`, `manifest.py`), which parse all 1,595 `GlobalConsts` loader as recovered in the RE repo
shipped data files. This module is checked against them file by file (below). (`findings/subsystems/loader-prototypes.md`, §M1 with the tokenizer of §M3);
the RE repo's Python readers (`sots-re/verify/parsers/flat_kv.py`,
`manifest.py`) implement the same rules and serve as the oracle this module
is checked against file by file (below).
## API ## API
@ -29,12 +32,12 @@ struct Color { double r, g, b, a; int components; };
std::optional<Color> parse_color(std::string_view "r g b [a]"); std::optional<Color> parse_color(std::string_view "r g b [a]");
// flat_kv.h // flat_kv.h
struct KvEntry { int line; std::string key /*original spelling*/; std::vector<Token> values; }; struct KvEntry { int line; std::string key /*original spelling*/; Token value; };
class FlatKV { entries(); find(key) /*case-insensitive, last wins*/; has(key); class FlatKV { entries(); find(key) /*case-insensitive, FIRST wins*/; has(key);
get_int/get_float/get_bool/get_string/get_color(key); duplicates(); }; get_int/get_float/get_bool/get_string/get_color(key); duplicates(); };
struct Row { int line; std::vector<Token> tokens; }; using Rows = std::vector<Row>; struct Row { int line; std::vector<Token> tokens; }; using Rows = std::vector<Row>;
Result<FlatKV> parse_flat_kv(std::string_view); Result<FlatKV> parse_flat_kv(std::string_view); // engine loader loop (mars/parse/script.h)
Result<Rows> parse_rows(std::string_view); Result<Rows> parse_rows(std::string_view); // line-based
std::string_view strip_comment(std::string_view line); std::vector<Token> split_tokens(std::string_view line); std::string_view strip_comment(std::string_view line); std::vector<Token> split_tokens(std::string_view line);
// manifest.h // manifest.h
@ -52,48 +55,74 @@ Result<std::vector<CsvRow>> split_csv_records(std::string_view); // raw RFC-418
Input is the file's raw bytes (`std::string_view`). The files are cp1252 with Input is the file's raw bytes (`std::string_view`). The files are cp1252 with
mixed CRLF/LF; bytes ≥ 0x80 are passed through untouched and are never mixed CRLF/LF; bytes ≥ 0x80 are passed through untouched and are never
whitespace or case-folded. Whitespace is ASCII only (space, `\t`, `\r`, `\n`, whitespace or case-folded.
`\v`, `\f`).
## Behaviour ## Behaviour
### Flat `KEY value` tables (`parse_flat_kv`) and row tables (`parse_rows`) ### Flat `KEY value` tables (`parse_flat_kv`)
Per line: strip a trailing `//` comment (quote-aware: `//` inside `"..."` is This is the engine's `GlobalConsts::LoadFile` loop, verbatim in shape. The
text, and an unclosed `"` disables comment stripping for the rest of the file is **not** read by lines: it is stepped through with the same pull
line), trim, skip if empty, then tokenise: tokenizer the brace-block catalogs use (`mars/parse/script.h`,
`Script::next()`), and each step is one of
- `"..."` is one token (may be empty, may contain spaces/tabs/`//`); no escape | step | what the loader does |
processing, so `"Data\\x.txt"` keeps both backslashes. |---|---|
- Otherwise a run of non-whitespace is one bareword; a `"` that has no | `KEY value` | one token each. `value` is consumed by the key's registered parser; a colour must therefore be quoted (`"48 29 2"`), and `LIST 1 2 three` is the two pairs `LIST=1`, `2=three` |
closing quote, or sits mid-word, is an ordinary character. | `NAME {` | the block is skipped to its matching `}` (`Problem::Kind::SkippedBlock`) |
| `}` | ignored |
`parse_flat_kv` takes the first token as the key and the remaining tokens as Tokenizer rules (script.h): whitespace is space/tab/CR/LF only; a bareword
the value: none (key-only line), one (scalar) or several (a list — the runs to whitespace, so braces glued to a word are part of it; `"`, `'` and
reference reader returns a Python list here; none of the 20 shipped tables backtick open a quoted token that ends at the same character, no escapes,
actually do this). `parse_rows` returns every line's token list. quotes stripped, an unterminated quote runs to end of input
(`UnterminatedQuote`); a token whose text starts with `//` is a comment to
end of line (so `3//x` is one word, `"0 0 0"// junk` is the value `0 0 0`
followed by a comment); tokens are capped at 1023 bytes.
Keys are matched **case-insensitively** and the **first occurrence wins**:
the loader erases a key from its expected-set once consumed, so a later
duplicate is logged "multiply defined" and ignored. `find()` returns the first
entry; every pair stays in `entries()` in file order; the later line is
reported as `DuplicateKey`; `duplicates()` lists repeated (case-folded) keys
with their line numbers. Unknown keys are the caller's business (the engine
ignores them); a registered key absent from the file keeps its default.
**End of input**: the loader stops at the first step that does not complete.
A final `KEY value` whose value token touches the end of the file — no
trailing newline or space — is therefore **dropped** (`DroppedTrailingPair`),
as is a final key with no value. Two shipped files lose a key this way:
`Data/Strategy/StrategyVars.txt` (`CIVILIAN_BURDEN_RATIO 0.5`) and
`Data/encounters.txt` (`HERALD_SPEECH_MAX_INVERVAL 45`) — the game runs on
those keys' compiled-in defaults.
Typing is lazy: `Token::kind()` classifies a bareword exactly like the Typing is lazy: `Token::kind()` classifies a bareword exactly like the
reference `coerce()`: `[+-]?\d+` → Int, `[+-]?(\d+\.\d*|\.\d+|\d+)([eE][+-]?\d+)?` reference `coerce()`: `[+-]?\d+` → Int, `[+-]?(\d+\.\d*|\.\d+|\d+)([eE][+-]?\d+)?`
→ Float (`.5`, `-.8`, `1.`, `7e+8`), `true`/`false` any case → Bool, else → Float (`.5`, `-.8`, `1.`, `7e+8`), `true`/`false` any case → Bool, else
Bareword (`0x10`, `inf`, `nan`, `1,5` are barewords). A quoted token is always Bareword (`0x10`, `inf`, `nan`, `1,5` are barewords). A quoted token is always
a String — `"8"` never becomes a number. `as_float()` also accepts Int shapes; a String — `"8"` never becomes a number. `as_float()` also accepts Int shapes;
`as_int()` refuses values that do not fit in 64 bits. `as_int()` refuses values that do not fit in 64 bits. (The engine types by the
key's registered parser — `sscanf` `%d`/`%f` — not by the text; see the RE
Keys: lookup is ASCII case-insensitive (`find("mars_default_color")` finds notes. `Token` keeps the text so a consumer can apply the registered parser.)
`MARS_DEFAULT_COLOR`); the entry keeps the original spelling and line. When a
key repeats, `find()` returns the **last** line — the same "later line wins"
default as the reference `parse_kv(on_dup='last')`. All entries stay in
`entries()`; `duplicates()` lists repeated (case-folded) keys with their line
numbers. The shipped tables contain no duplicates, exact or case-folded, so
the case-folding in `duplicates()` is a deliberate superset of the reference's
exact-key check.
Colours are quoted `"r g b"` / `"r g b a"` strings (tabs inside are fine, as in Colours are quoted `"r g b"` / `"r g b a"` strings (tabs inside are fine, as in
`globals.txt`); `get_color` / `parse_color` require 3 or 4 numeric components. `globals.txt`); `get_color` / `parse_color` require 3 or 4 numeric components.
Problems: `UnbalancedQuote` (odd number of `"` on a line) is reported but the `KvEntry::line` counts `\n` only (the engine has no notion of lines; the
line is still parsed. None occur in shipped data. number is for diagnostics).
### Row tables (`parse_rows`)
Line-based (these files are read by other engine code whose reader has not
been recovered; the behaviour is the reference reader's). Per line: strip a
trailing `//` comment (quote-aware: `//` inside `"..."` is text, and an
unclosed `"` disables comment stripping for the rest of the line), trim, skip
if empty, then tokenise: `"..."` is one token (may be empty, may contain
spaces/tabs/`//`; no escapes), otherwise a run of non-whitespace is one
bareword; a `"` that has no closing quote, or sits mid-word, is an ordinary
character. Whitespace here is the C set (space, `\t`, `\r`, `\n`, `\v`, `\f`).
`UnbalancedQuote` (odd number of `"` on a line) is reported but the line is
still parsed. None occur in shipped data.
### Id manifests (`parse_manifest`) ### Id manifests (`parse_manifest`)
@ -135,13 +164,14 @@ end the search.
## Oracle results ## Oracle results
`tests/mars_text/build_and_run.sh` compiles with `g++ -std=c++17 -Wall -Wextra `tests/mars_text/build_and_run.sh` compiles with `g++ -std=c++17 -Wall -Wextra
-Werror`, runs the unit tests (165 checks, hand-written samples for every -Werror`, runs the unit tests (201 checks, hand-written samples for every
quirk above), the real-data facts test, and — when `SOTS_DATA_DIR` is set — rule above, including a dedicated engine-step suite for the kv loader), the
dumps every file `mars_text` owns through both `dump_json` (C++) and real-data facts test, and — when `SOTS_DATA_DIR` is set — dumps every file
`oracle/dump.py` (the Python readers) and compares them structurally, `mars_text` owns through both `dump_json` (C++) and `oracle/dump.py` (the
type-aware (`compare.py`). Python readers) and compares them structurally, type-aware (`compare.py`).
Run 2026-09-07 against the extracted `sots.gob` + `sots_local_en.gob` tree: Run 2026-09-07 (engine-parity rules) against the extracted `sots.gob` +
`sots_local_en.gob` tree:
| kind | files | agree | | kind | files | agree |
|---|---|---| |---|---|---|
@ -151,23 +181,33 @@ Run 2026-09-07 against the extracted `sots.gob` + `sots_local_en.gob` tree:
| csv (AI tables, scenarios, `RealSpace.csv`, `SpriteTable.csv`, `Strings.csv`, `SpeechEvents.csv`, asteroids, music, sound_ui) | 28 | 28 | | csv (AI tables, scenarios, `RealSpace.csv`, `SpriteTable.csv`, `Strings.csv`, `SpeechEvents.csv`, asteroids, music, sound_ui) | 28 | 28 |
| **total** | **64** | **64 (100%)** | | **total** | **64** | **64 (100%)** |
Compared per file: the full key → value map (exact keys, last wins, typed Compared per file: the full key → value map (case-insensitive keys under
values), duplicate-key report, every row's typed tokens, manifest entries / their first spelling, first wins, typed values), duplicate-key report, every
tombstones / problems (kind, line, id), CSV header and every stripped cell. row's typed tokens, manifest entries / tombstones / problems (kind, line, id),
Real-data facts also asserted: `globals.txt` 364 keys, `MARS_DEFAULT_COLOR` = CSV header and every stripped cell. Real-data facts also asserted:
(255,177,39); `_turrets.txt` 42 × 8 tokens; `_weapons.txt` 123 ids, deleted `globals.txt` 364 keys, no duplicates, `MARS_DEFAULT_COLOR` = (255,177,39);
36/58/59; Human `_shipsections.txt` 145 ids with `dewar.shipsection` → 98; `StrategyVars.txt` 96 keys with `CIVILIAN_BURDEN_RATIO` dropped and
`Strings.csv` 5,722 raw records → 5,200 data rows → 5,196 distinct keys, `encounters.txt` with `HERALD_SPEECH_MAX_INVERVAL` dropped (one
exactly one multi-line raw cell; `aitechpri.csv` 0 rows with a 7-name schema. `DroppedTrailingPair` each); `_turrets.txt` 42 × 8 tokens; `_weapons.txt` 123
ids, deleted 36/58/59; Human `_shipsections.txt` 145 ids with
`dewar.shipsection` → 98; `Strings.csv` 5,722 raw records → 5,200 data rows →
5,196 distinct keys, exactly one multi-line raw cell; `aitechpri.csv` 0 rows
with a 7-name schema.
What changed when the kv reader moved from the earlier line-based, last-wins
reading to the engine's rules: no shipped kv file contains a duplicate key
(exact or case-folded), so first-wins flips no value; the only observable
difference is the two dropped trailing pairs above.
## Deliberate divergences from the reference ## Deliberate divergences from the reference
None affect shipped data (verified above); listed so nobody chases them: None affect shipped data (verified above); listed so nobody chases them:
- Whitespace is ASCII-only. The Python readers operate on decoded `str`, so - Row/manifest/CSV whitespace is ASCII-only. The Python readers operate on
they would also treat cp1252 `0xA0` (NBSP) and bytes `0x1C`–`0x1F` as decoded `str`, so they would also treat cp1252 `0xA0` (NBSP) and bytes
whitespace / line breaks. The relevant files contain no such bytes. `0x1C`–`0x1F` as whitespace / line breaks. The relevant files contain no
- `duplicates()` folds key case; the reference compares exact keys. such bytes. (The kv reader and the Python `Script` both use the engine's
exact four-character set.)
- Ints are 64-bit; the reference has unbounded ints. An overflowing bareword - Ints are 64-bit; the reference has unbounded ints. An overflowing bareword
classifies as `Int` but `as_int()` returns nullopt. classifies as `Int` but `as_int()` returns nullopt.
- The reference `csv` module rejects a NUL byte; this reader treats it as data. - The reference `csv` module rejects a NUL byte; this reader treats it as data.
@ -176,8 +216,8 @@ None affect shipped data (verified above); listed so nobody chases them:
## Open questions ## Open questions
- The engine's own tolerance is unknown for the cases where the reference is - The `SkipBlock` used for `NAME {` inside a kv file is assumed to count raw
strict (manifest entry with a trailing comment, a key that repeats). We copy `{`/`}` tokens (`Script::skip_block`); no shipped kv file contains a block,
the reference; both are absent from shipped data. so this cannot be observed from data.
- Multi-token `KEY a b c` values never occur in shipped tables; the API keeps - The engine's own tolerance is unknown for a manifest entry with a trailing
them as a token list rather than guessing a type. comment; we copy the reference (absent from shipped data).

232
src/mars/parse/blocks.cpp
View file

@ -2,6 +2,7 @@
#include <utility> #include <utility>
#include "script.h"
#include "value.h" #include "value.h"
namespace mars::parse { namespace mars::parse {
@ -17,14 +18,14 @@ Entry::~Entry() = default;
const Entry* Node::first(std::string_view key) const { const Entry* Node::first(std::string_view key) const {
for (const Entry& e : entries) for (const Entry& e : entries)
if (!e.is_item() && iequals(e.key, key)) return &e; if (iequals(e.key, key)) return &e;
return nullptr; return nullptr;
} }
std::vector<const Entry*> Node::all(std::string_view key) const { std::vector<const Entry*> Node::all(std::string_view key) const {
std::vector<const Entry*> out; std::vector<const Entry*> out;
for (const Entry& e : entries) for (const Entry& e : entries)
if (!e.is_item() && iequals(e.key, key)) out.push_back(&e); if (iequals(e.key, key)) out.push_back(&e);
return out; return out;
} }
@ -54,108 +55,26 @@ std::vector<const Node*> Node::blocks(std::string_view key) const {
return out; return out;
} }
std::vector<std::string_view> Node::items() const { // ---- parser -----------------------------------------------------------------
std::vector<std::string_view> out;
for (const Entry& e : entries)
if (e.is_item()) out.push_back(e.value);
return out;
}
// ---- tokenizer --------------------------------------------------------------
namespace { namespace {
enum class Tok { Open, Close, Quoted, Bare, End };
struct Token {
Tok kind = Tok::End;
std::string_view text; // Quoted: contents without the quotes
int line = 0;
};
bool is_space(unsigned char c) {
return c == ' ' || c == '\t' || c == '\n' || c == '\r' || c == '\v' || c == '\f';
}
class Lexer {
public:
explicit Lexer(std::string_view s) : s_(s) {}
// Returns false (with `err` set) only for an unterminated string.
bool next(Token& out, Diagnostic& err) {
skip_trivia();
out.line = line_;
if (pos_ >= s_.size()) {
out.kind = Tok::End;
out.text = {};
return true;
}
char c = s_[pos_];
if (c == '{') { ++pos_; out.kind = Tok::Open; out.text = s_.substr(pos_ - 1, 1); return true; }
if (c == '}') { ++pos_; out.kind = Tok::Close; out.text = s_.substr(pos_ - 1, 1); return true; }
if (c == '"') {
std::size_t close = s_.find('"', pos_ + 1);
if (close == std::string_view::npos) {
err = {line_, "unterminated string"};
return false;
}
out.kind = Tok::Quoted;
out.text = s_.substr(pos_ + 1, close - pos_ - 1);
for (char q : out.text) if (q == '\n') ++line_; // strings may span lines
pos_ = close + 1;
return true;
}
std::size_t start = pos_;
while (pos_ < s_.size()) {
unsigned char u = static_cast<unsigned char>(s_[pos_]);
if (is_space(u) || u == '{' || u == '}' || u == '"') break;
++pos_;
}
out.kind = Tok::Bare;
out.text = s_.substr(start, pos_ - start);
return true;
}
private:
void skip_trivia() {
while (pos_ < s_.size()) {
unsigned char c = static_cast<unsigned char>(s_[pos_]);
if (is_space(c)) {
if (c == '\n') ++line_;
++pos_;
} else if (c == '/' && pos_ + 1 < s_.size() && s_[pos_ + 1] == '/') {
while (pos_ < s_.size() && s_[pos_] != '\n') ++pos_;
} else {
break;
}
}
}
std::string_view s_;
std::size_t pos_ = 0;
int line_ = 1;
};
// ---- parser -----------------------------------------------------------------
class Parser { class Parser {
public: public:
Parser(std::string_view text, Options opt) : lex_(text), opt_(opt) {} Parser(std::string_view text, Options opt) : script_(text), opt_(opt) {}
Result<Document> run() { Result<Document> run() {
Document doc; Document doc;
if (!advance()) return err_; if (body(doc.root, 0) == Step::Error) return err_;
if (!body(doc.root, 0)) return err_;
doc.warnings = std::move(warnings_); doc.warnings = std::move(warnings_);
return doc; return doc;
} }
private: private:
// Fills `cur_` with the next token. False on lexer error. enum class Step { Closed, Eof, Error };
bool advance() { return lex_.next(cur_, err_); }
// Lenient recovery point: warn, or fail under strict. // Every tolerance the loaders exhibit is recorded here; strict rejects it.
bool recover(int line, std::string message) { bool tolerate(int line, std::string message) {
if (opt_.strict) { if (opt_.strict) {
err_ = {line, std::move(message)}; err_ = {line, std::move(message)};
return false; return false;
@ -164,88 +83,101 @@ private:
return true; return true;
} }
bool fail(int line, std::string message) { static std::string quote(std::string_view s) { return "'" + std::string(s) + "'"; }
err_ = {line, std::move(message)};
return false;
}
// Parses entries into `node` until the matching '}' (depth > 0) or EOF. // Runs the loaders' step loop for one block body. Closed: a `}` ended it.
// On return `cur_` is the token after the block. // Eof: the input ran out (the caller, if inside a block, reports that).
bool body(Node& node, int depth) { Step body(Node& node, int depth) {
for (;;) { for (;;) {
switch (cur_.kind) { ScriptToken t;
case Tok::End: const ReadStatus rc = script_.next(t);
if (depth > 0 &&
!recover(cur_.line, "end of file inside block '" + node.name +
"' (depth " + std::to_string(depth) + ")"))
return false;
return true; // EOF closes every open block
case Tok::Close: if (t.key_unterminated && !tolerate(t.key_line, "unterminated quote runs to end of input"))
return Step::Error;
if (t.value_unterminated && !tolerate(t.value_line, "unterminated quote runs to end of input"))
return Step::Error;
if (rc != ReadStatus::Ok) {
// The original returns here with the step incomplete; whatever
// was read is discarded. Report what that means for this file.
if (t.key_status == ReadStatus::AtEnd && t.key == "}") {
// A closing brace as the very last byte: the block ends
// exactly as it would have with a newline after it.
if (depth == 0) {
if (!tolerate(t.key_line, "stray '}' at top level")) return Step::Error;
return Step::Eof;
}
return Step::Closed;
}
if (t.key_status == ReadStatus::AtEnd) {
if (!tolerate(t.key_line, "trailing key " + quote(t.key) +
" touches end of input: dropped"))
return Step::Error;
} else if (t.key_status == ReadStatus::Ok) {
std::string what;
if (t.value_status == ReadStatus::NoInput)
what = "trailing key " + quote(t.key) + " has no value: dropped";
else if (t.type == ScriptToken::Type::Open)
what = "block header " + quote(t.key) + " { touches end of input: dropped";
else
what = "final pair " + quote(t.key) + " " + quote(t.value) +
" touches end of input (no trailing newline): dropped";
if (!tolerate(t.key_line, what)) return Step::Error;
}
return Step::Eof;
}
switch (t.type) {
case ScriptToken::Type::Close:
if (depth == 0) { if (depth == 0) {
if (!recover(cur_.line, "stray '}' at top level")) return false; if (!tolerate(t.key_line, "stray '}' at top level")) return Step::Error;
if (!advance()) return false;
continue; continue;
} }
return advance(); return Step::Closed;
case Tok::Open: case ScriptToken::Type::Open: {
return fail(cur_.line, "'{' without a block name"); if (t.key == "{" && !tolerate(t.key_line, "'{' used as a block name")) return Step::Error;
case Tok::Quoted: {
Entry e; Entry e;
e.kind = Entry::Kind::Item; e.kind = Entry::Kind::Block;
e.value = std::string(cur_.text); e.key = std::string(t.key);
e.quoted = true; e.key_quoted = t.key_quoted;
e.line = cur_.line; e.line = t.key_line;
e.block = std::make_unique<Node>();
e.block->name = e.key;
e.block->line = t.value_line;
const Step inner = body(*e.block, depth + 1);
if (inner == Step::Error) return inner;
node.entries.push_back(std::move(e)); node.entries.push_back(std::move(e));
if (!advance()) return false; if (inner == Step::Eof) {
if (!tolerate(script_.line(), "end of input inside block " + quote(t.key) +
" (depth " + std::to_string(depth + 1) + ")"))
return Step::Error;
return Step::Eof;
}
continue; continue;
} }
case Tok::Bare: { case ScriptToken::Type::Pair: {
Token name = cur_; if (t.key == "{" && !tolerate(t.key_line, "'{' used as a key")) return Step::Error;
if (!advance()) return false;
if (cur_.kind == Tok::End || cur_.kind == Tok::Close) {
// A lone word right before the block closes: a flag item.
Entry e;
e.kind = Entry::Kind::Item;
e.value = std::string(name.text);
e.line = name.line;
node.entries.push_back(std::move(e));
continue;
}
if (cur_.kind == Tok::Open) {
Entry e;
e.kind = Entry::Kind::Block;
e.key = std::string(name.text);
e.line = name.line;
e.block = std::make_unique<Node>();
e.block->name = e.key;
e.block->line = cur_.line;
if (!advance()) return false;
if (!body(*e.block, depth + 1)) return false;
node.entries.push_back(std::move(e));
continue;
}
// Bare or Quoted: a key/value pair.
Entry e; Entry e;
e.kind = Entry::Kind::Pair; e.kind = Entry::Kind::Pair;
e.key = std::string(name.text); e.key = std::string(t.key);
e.value = std::string(cur_.text); e.value = std::string(t.value);
e.quoted = cur_.kind == Tok::Quoted; e.quoted = t.value_quoted;
e.line = name.line; e.key_quoted = t.key_quoted;
e.line = t.key_line;
node.entries.push_back(std::move(e)); node.entries.push_back(std::move(e));
if (!advance()) return false;
continue; continue;
} }
case ScriptToken::Type::None:
break; // not produced when rc == Ok
} }
} }
} }
Lexer lex_; Script script_;
Options opt_; Options opt_;
Token cur_;
Diagnostic err_; Diagnostic err_;
std::vector<Diagnostic> warnings_; std::vector<Diagnostic> warnings_;
}; };

61
src/mars/parse/blocks.h
View file

@ -4,23 +4,35 @@
// *.shipsection, *.tech, *.combat, *.def, *.script and the block-form *.txt // *.shipsection, *.tech, *.combat, *.def, *.script and the block-form *.txt
// files (scenarios, tutorial, credits, system names, sky defs, ...). // files (scenarios, tutorial, credits, system names, sky defs, ...).
// //
// body := (block | pair | item)* // The original engine has no tree: every loader pulls `Script::next()` steps
// block := NAME '{' body '}' // (see script.h) and interprets them itself. This module runs exactly that
// pair := NAME value // step loop and records what it yields, so the tree below is the step stream
// item := QUOTED | NAME (a NAME directly before '}' or EOF) // made addressable, nothing more:
// value := QUOTED | BAREWORD
// comment := '//' ... end of line
// //
// The tokenizer is whitespace-driven, not line-based: a block may open on the // step := KEY value -> Entry::Kind::Pair (key may itself be quoted)
// same line as a preceding pair, a name may sit on the same line as its brace, // | NAME { -> Entry::Kind::Block (body until the matching `}`)
// and a quoted string may span lines. There is no escape syntax -- backslashes // | } -> closes the block
// and '//' inside quotes are literal. Bytes pass through untouched (cp1252).
// //
// Leniency (the shipped data needs both; the engine accepts them): // Tokenizer: whitespace-only delimiting (`{`/`}` count only as whole tokens),
// * end of file closes every open block -> Diagnostic in Document::warnings // `"` `'` backtick quotes without escapes, `//` comments, 1023-byte tokens --
// * a stray '}' at top level is ignored -> Diagnostic in Document::warnings // the rules are listed in script.h.
// With Options::strict both become errors. Three things are always errors: //
// an unterminated quoted string, a '{' with no name before it, and nothing else. // End-of-input behaviour is what the loaders' `while (next() == Ok)` loops
// produce, and each case is reported in Document::warnings:
// * input ends inside a block -> the block simply ends
// * a `}` at top level -> ignored
// * a final `KEY value` whose value touches the end of input (no trailing
// newline) -> that pair is DROPPED
// * a final KEY with no value -> dropped
// * a final `NAME {` touching the end -> dropped
// * an unterminated quote -> runs to the end of input
// * `{` in key position -> an ordinary key (kept)
// A `}` that closes a block and touches the end of input is a plain close
// (the original stops there either way; the outcome is identical).
// With Options::strict every warning is an error instead.
//
// No escape syntax -- backslashes and `//` inside quotes are literal. Bytes
// pass through untouched (cp1252).
#pragma once #pragma once
#include <memory> #include <memory>
@ -34,16 +46,17 @@ namespace mars::parse {
struct Node; struct Node;
// One thing inside a block, in file order. // One step inside a block, in file order.
struct Entry { struct Entry {
enum class Kind { Pair, Item, Block }; enum class Kind { Pair, Block };
Kind kind = Kind::Pair; Kind kind = Kind::Pair;
std::string key; // Pair/Block: the name as written (original case). Item: empty. std::string key; // the name as written (original case, quotes stripped)
std::string value; // Pair: the value text; Item: the item text. Block: empty. std::string value; // Pair: the value text. Block: empty.
bool quoted = false; // value/item was written inside double quotes bool quoted = false; // value was written inside quotes
bool key_quoted = false; // key was written inside quotes (systemnames.txt lists)
std::unique_ptr<Node> block; // Block only std::unique_ptr<Node> block; // Block only
int line = 0; // 1-based line of the key (or of the item) int line = 0; // 1-based line of the key
Entry(); Entry();
Entry(Entry&&) noexcept; Entry(Entry&&) noexcept;
@ -53,7 +66,6 @@ struct Entry {
Entry& operator=(const Entry&) = delete; Entry& operator=(const Entry&) = delete;
bool is_pair() const { return kind == Kind::Pair; } bool is_pair() const { return kind == Kind::Pair; }
bool is_item() const { return kind == Kind::Item; }
bool is_block() const { return kind == Kind::Block; } bool is_block() const { return kind == Kind::Block; }
}; };
@ -73,17 +85,16 @@ struct Node {
std::vector<std::string_view> values(std::string_view key) const; std::vector<std::string_view> values(std::string_view key) const;
const Node* first_block(std::string_view key) const; const Node* first_block(std::string_view key) const;
std::vector<const Node*> blocks(std::string_view key) const; std::vector<const Node*> blocks(std::string_view key) const;
std::vector<std::string_view> items() const; // bare quoted / trailing barewords
bool has(std::string_view key) const { return first(key) != nullptr; } bool has(std::string_view key) const { return first(key) != nullptr; }
}; };
struct Document { struct Document {
Node root; // the file's top level is itself a body Node root; // the file's top level is itself a body
std::vector<Diagnostic> warnings; // lenient recoveries taken std::vector<Diagnostic> warnings; // end-of-input / stray-brace cases listed above
}; };
struct Options { struct Options {
bool strict = false; // reject files the engine would repair bool strict = false; // reject files the engine would tolerate (every warning -> error)
}; };
Result<Document> parse_blocks(std::string_view text, Options options = {}); Result<Document> parse_blocks(std::string_view text, Options options = {});

182
src/mars/parse/script.h Normal file
View file

@ -0,0 +1,182 @@
// mars::parse::Script -- the engine's pull tokenizer for every brace-block
// and flat KEY/value text file (weapons, ship sections, tech tree, GUI
// scripts, scenario .txt, and the GlobalConsts tables under Data/).
//
// Header-only so both mars_parse (block trees) and mars_text (flat tables)
// share one implementation of the rules without linking each other.
//
// The original reads a file token by token; there is no line structure and no
// tree. Three operations exist and consumers drive them directly:
//
// read_token one whitespace-delimited token
// next the `KEY value` / `NAME {` / `}` step every loader loops on
// skip_block consume a block the caller does not understand
//
// Tokenizer rules (exact):
// * whitespace is ' ', '\t', '\r', '\n' -- nothing else;
// * a bareword runs until whitespace. Braces are NOT delimiters: `{` and `}`
// are only recognised when they are whole tokens, so `name{` is one word;
// * a token starting with `"`, `'` or a backtick is quoted; it ends at the
// next occurrence of the same character. No escape processing, the quotes
// are stripped, whitespace inside is kept, an unterminated quote runs to
// the end of the input (`Raw::unterminated` reports it);
// * a token whose text starts with `//` is a comment: the rest of that line
// is skipped and the next token is read. The check is made on the
// extracted text, so `3//x` is one word and a quoted token whose content
// starts with `//` is also a comment;
// * token text is silently truncated to kMaxTokenLength bytes (the scan
// still consumes the whole token);
// * no `/* */`, no `=`, no BOM handling.
//
// Status codes mirror the original: Ok = a token was read and input remains;
// AtEnd = a token was read and it touches the end of the input; NoInput = no
// token could be read. `next()` returns the status of its *value* read, which
// is why a final pair whose value touches the end of input is dropped by every
// consumer (they loop `while (next() == Ok)`).
#pragma once
#include <cstddef>
#include <string_view>
namespace mars::parse {
enum class ReadStatus { Ok = 0, NoInput = 1, AtEnd = 2 };
struct ScriptToken {
enum class Type { None = 0, Pair = 1, Open = 2, Close = 3 };
Type type = Type::None;
std::string_view key; // text with quotes stripped (Close: "}")
std::string_view value; // Pair: the value; Open: "{"; else empty
bool key_quoted = false;
bool value_quoted = false;
bool key_unterminated = false; // quoted token that ran to end of input
bool value_unterminated = false;
int key_line = 0; // 1-based line where each token started
int value_line = 0;
ReadStatus key_status = ReadStatus::NoInput; // status of each read;
ReadStatus value_status = ReadStatus::NoInput; // next() returns value_status
};
class Script {
public:
static constexpr std::size_t kMaxTokenLength = 1023;
struct Raw {
std::string_view text;
bool quoted = false;
bool unterminated = false;
int line = 0;
};
explicit Script(std::string_view text) : s_(text) {}
std::size_t cursor() const { return pos_; }
int line() const { return line_; }
bool at_end() const { return pos_ >= s_.size(); }
static bool is_space(char c) { return c == ' ' || c == '\t' || c == '\r' || c == '\n'; }
// One token. `out` is reset first; on NoInput it stays empty.
ReadStatus read_token(Raw& out) {
out = Raw{};
for (;;) {
skip_space();
if (pos_ >= s_.size()) return ReadStatus::NoInput;
const int line = line_;
const char c = s_[pos_];
std::string_view text;
bool quoted = false, unterminated = false;
if (c == '"' || c == '\'' || c == '`') {
quoted = true;
const std::size_t close = s_.find(c, pos_ + 1);
const std::size_t end = close == std::string_view::npos ? s_.size() : close;
text = s_.substr(pos_ + 1, end - pos_ - 1);
unterminated = close == std::string_view::npos;
advance_to(unterminated ? s_.size() : close + 1);
} else {
std::size_t end = pos_;
while (end < s_.size() && !is_space(s_[end])) ++end;
text = s_.substr(pos_, end - pos_);
pos_ = end;
}
if (text.size() >= 2 && text[0] == '/' && text[1] == '/') {
skip_line(); // comment: to after the next \r/\n run, then read again
continue;
}
if (text.size() > kMaxTokenLength) text = text.substr(0, kMaxTokenLength);
out.text = text;
out.quoted = quoted;
out.unterminated = unterminated;
out.line = line;
return pos_ >= s_.size() ? ReadStatus::AtEnd : ReadStatus::Ok;
}
}
// The loaders' step: a key, then (unless the key is `}`) a value.
// Returns the status of the last read; the token is only complete when
// that is Ok. The text of whatever was read is filled in regardless.
ReadStatus next(ScriptToken& out) {
out = ScriptToken{};
Raw k;
out.key_status = read_token(k);
out.key = k.text;
out.key_quoted = k.quoted;
out.key_unterminated = k.unterminated;
out.key_line = k.line;
if (out.key_status != ReadStatus::Ok) return out.key_status;
if (k.text == "}") { // exact compare on the stripped text
out.type = ScriptToken::Type::Close;
return ReadStatus::Ok;
}
Raw v;
out.value_status = read_token(v);
out.value = v.text;
out.value_quoted = v.quoted;
out.value_unterminated = v.unterminated;
out.value_line = v.line;
out.type = v.text == "{" ? ScriptToken::Type::Open : ScriptToken::Type::Pair;
return out.value_status;
}
// Consume tokens, counting `{` / `}`, until `depth` reaches zero.
// Ok when the block closed; otherwise the status that ended the input.
ReadStatus skip_block(int depth) {
Raw t;
while (depth > 0) {
ReadStatus rc = read_token(t);
if (rc == ReadStatus::NoInput) return rc;
if (t.text == "{") ++depth;
else if (t.text == "}") --depth;
if (rc != ReadStatus::Ok) return depth == 0 ? ReadStatus::Ok : rc;
}
return ReadStatus::Ok;
}
private:
void skip_space() {
while (pos_ < s_.size() && is_space(s_[pos_])) {
if (s_[pos_] == '\n') ++line_;
++pos_;
}
}
void skip_line() {
while (pos_ < s_.size() && s_[pos_] != '\r' && s_[pos_] != '\n') ++pos_;
while (pos_ < s_.size() && (s_[pos_] == '\r' || s_[pos_] == '\n')) {
if (s_[pos_] == '\n') ++line_;
++pos_;
}
}
void advance_to(std::size_t p) {
for (; pos_ < p; ++pos_)
if (s_[pos_] == '\n') ++line_;
}
std::string_view s_;
std::size_t pos_ = 0;
int line_ = 1;
};
} // namespace mars::parse

113
src/mars/text/flat_kv.cpp
View file

@ -1,5 +1,6 @@
#include "mars/text/flat_kv.h" #include "mars/text/flat_kv.h"
#include "mars/parse/script.h"
#include "mars/text/lines.h" #include "mars/text/lines.h"
namespace mars::text { namespace mars::text {
@ -48,64 +49,46 @@ bool has_unbalanced_quote(std::string_view line) {
return quotes % 2 == 1; return quotes % 2 == 1;
} }
// Shared line pass: yields (line number, tokens) for every non-blank line. std::string quote(std::string_view s) { return "'" + std::string(s) + "'"; }
template <class Fn>
void tokenised_lines(std::string_view text, std::vector<Problem>& problems, Fn&& fn) {
for_each_line(text, [&](int lineno, std::string_view raw) {
std::string_view line = strip(strip_comment(raw));
if (line.empty()) return;
if (has_unbalanced_quote(line))
problems.push_back(Problem{Problem::Kind::UnbalancedQuote, lineno, -1,
"line " + std::to_string(lineno) + ": unbalanced quote"});
fn(lineno, split_tokens(line));
});
}
} // namespace } // namespace
// ---- FlatKV -------------------------------------------------------------- // ---- FlatKV --------------------------------------------------------------
void FlatKV::add(KvEntry entry) { void FlatKV::add(KvEntry entry) {
last_[fold_case(entry.key)] = entries_.size(); first_.emplace(fold_case(entry.key), entries_.size()); // keeps the first index
entries_.push_back(std::move(entry)); entries_.push_back(std::move(entry));
} }
const KvEntry* FlatKV::find(std::string_view key) const { const KvEntry* FlatKV::find(std::string_view key) const {
auto it = last_.find(fold_case(key)); auto it = first_.find(fold_case(key));
return it == last_.end() ? nullptr : &entries_[it->second]; return it == first_.end() ? nullptr : &entries_[it->second];
} }
namespace {
const Token* first_token(const FlatKV& kv, std::string_view key) {
const KvEntry* e = kv.find(key);
return (e && !e->values.empty()) ? &e->values.front() : nullptr;
}
} // namespace
std::optional<std::int64_t> FlatKV::get_int(std::string_view key) const { std::optional<std::int64_t> FlatKV::get_int(std::string_view key) const {
const Token* t = first_token(*this, key); const KvEntry* e = find(key);
return t ? t->as_int() : std::nullopt; return e ? e->value.as_int() : std::nullopt;
} }
std::optional<double> FlatKV::get_float(std::string_view key) const { std::optional<double> FlatKV::get_float(std::string_view key) const {
const Token* t = first_token(*this, key); const KvEntry* e = find(key);
return t ? t->as_float() : std::nullopt; return e ? e->value.as_float() : std::nullopt;
} }
std::optional<bool> FlatKV::get_bool(std::string_view key) const { std::optional<bool> FlatKV::get_bool(std::string_view key) const {
const Token* t = first_token(*this, key); const KvEntry* e = find(key);
return t ? t->as_bool() : std::nullopt; return e ? e->value.as_bool() : std::nullopt;
} }
std::optional<std::string_view> FlatKV::get_string(std::string_view key) const { std::optional<std::string_view> FlatKV::get_string(std::string_view key) const {
const Token* t = first_token(*this, key); const KvEntry* e = find(key);
if (!t) return std::nullopt; if (!e) return std::nullopt;
return std::string_view(t->text); return std::string_view(e->value.text);
} }
std::optional<Color> FlatKV::get_color(std::string_view key) const { std::optional<Color> FlatKV::get_color(std::string_view key) const {
const Token* t = first_token(*this, key); const KvEntry* e = find(key);
return t ? parse_color(t->text) : std::nullopt; return e ? parse_color(e->value.text) : std::nullopt;
} }
std::vector<std::pair<std::string, std::vector<int>>> FlatKV::duplicates() const { std::vector<std::pair<std::string, std::vector<int>>> FlatKV::duplicates() const {
@ -123,16 +106,55 @@ std::vector<std::pair<std::string, std::vector<int>>> FlatKV::duplicates() const
return out; return out;
} }
// The GlobalConsts loader loop: step with Script::next(); a pair is consumed
// on first sight, a block is skipped, a stray `}` is ignored, and the loop
// ends on the first step that does not complete (which drops that step).
Result<FlatKV> parse_flat_kv(std::string_view text) { Result<FlatKV> parse_flat_kv(std::string_view text) {
using mars::parse::ReadStatus;
using mars::parse::ScriptToken;
Result<FlatKV> r; Result<FlatKV> r;
tokenised_lines(text, r.problems, [&](int lineno, std::vector<Token> toks) { mars::parse::Script script(text);
KvEntry e; for (;;) {
e.line = lineno; ScriptToken t;
e.key = std::move(toks.front().text); const ReadStatus rc = script.next(t);
toks.erase(toks.begin()); if (t.key_unterminated)
e.values = std::move(toks); r.problems.push_back(Problem{Problem::Kind::UnterminatedQuote, t.key_line, -1,
r.value.add(std::move(e)); "line " + std::to_string(t.key_line) + ": unterminated quote"});
}); if (t.value_unterminated)
r.problems.push_back(Problem{Problem::Kind::UnterminatedQuote, t.value_line, -1,
"line " + std::to_string(t.value_line) + ": unterminated quote"});
if (rc != ReadStatus::Ok) {
const bool key_read = t.key_status != ReadStatus::NoInput;
if (key_read && t.key != "}") {
std::string what = t.value_status == ReadStatus::NoInput || t.key_status == ReadStatus::AtEnd
? "trailing key " + quote(t.key) + " without a value: dropped"
: "final pair " + quote(t.key) + " " + quote(t.value) +
" touches end of input (no trailing newline): dropped";
r.problems.push_back(Problem{Problem::Kind::DroppedTrailingPair, t.key_line, -1,
"line " + std::to_string(t.key_line) + ": " + what});
}
break;
}
if (t.type == ScriptToken::Type::Pair) {
if (r.value.has(t.key)) {
r.problems.push_back(Problem{Problem::Kind::DuplicateKey, t.key_line, -1,
"line " + std::to_string(t.key_line) + ": " + quote(t.key) +
" multiply defined (first occurrence kept)"});
}
KvEntry e;
e.line = t.key_line;
e.key = std::string(t.key);
e.value = Token{std::string(t.value), t.value_quoted};
r.value.add(std::move(e));
} else if (t.type == ScriptToken::Type::Open) {
r.problems.push_back(Problem{Problem::Kind::SkippedBlock, t.key_line, -1,
"line " + std::to_string(t.key_line) + ": block " + quote(t.key) +
" skipped"});
if (script.skip_block(1) != ReadStatus::Ok) break;
}
// Close at top level: ignored
}
return r; return r;
} }
@ -140,8 +162,13 @@ Result<FlatKV> parse_flat_kv(std::string_view text) {
Result<Rows> parse_rows(std::string_view text) { Result<Rows> parse_rows(std::string_view text) {
Result<Rows> r; Result<Rows> r;
tokenised_lines(text, r.problems, [&](int lineno, std::vector<Token> toks) { for_each_line(text, [&](int lineno, std::string_view raw) {
r.value.push_back(Row{lineno, std::move(toks)}); std::string_view line = strip(strip_comment(raw));
if (line.empty()) return;
if (has_unbalanced_quote(line))
r.problems.push_back(Problem{Problem::Kind::UnbalancedQuote, lineno, -1,
"line " + std::to_string(lineno) + ": unbalanced quote"});
r.value.push_back(Row{lineno, split_tokens(line)});
}); });
return r; return r;
} }

43
src/mars/text/flat_kv.h
View file

@ -1,14 +1,26 @@
// mars::text -- the flat "KEY value" tuning tables and whitespace-positional // mars::text -- the flat "KEY value" tuning tables (Data/**/*.txt, read by the
// row tables (Data/**/*.txt, Weapons/_turrets.txt, Badges/BadgeTable.txt ...). // engine's GlobalConsts loader) and the whitespace-positional row tables
// (Weapons/_turrets.txt, Badges/BadgeTable.txt ...).
// //
// Line grammar (both shapes): // KEY value tables (parse_flat_kv) follow the engine's loader exactly. The
// file is not read by lines: the same pull tokenizer as the brace-block
// catalogs (mars/parse/script.h) steps through it and every step is
// KEY value one token each -- so `COLOR "48 29 2"` must be quoted, and
// `LIST 1 2 3` would be the pairs LIST=1 and 2=3
// NAME { a block; skipped up to its matching `}`
// } ignored
// Keys are matched case-insensitively and the FIRST occurrence wins: the
// loader erases a key once it has consumed it, so a later duplicate is
// "multiply defined" and ignored (Problem::Kind::DuplicateKey). A final pair
// whose value touches the end of the file (no trailing newline) is dropped
// (DroppedTrailingPair) -- two shipped files lose their last key this way.
// `//` comments, `"`/`'`/backtick quotes without escapes, 1023-byte tokens:
// see script.h.
//
// Row tables (parse_rows) are line-based:
// line := token* ('//' comment)? -- '//' inside "..." is not a comment // line := token* ('//' comment)? -- '//' inside "..." is not a comment
// token := '"' [^"]* '"' | non-whitespace+ -- a quoted token may be empty ("") // token := '"' [^"]* '"' | non-whitespace+ -- a quoted token may be empty ("")
// -- an unpaired '"' is just a bareword char // -- an unpaired '"' is just a bareword char
// KEY value tables take the first token of a line as the key and the rest as
// its value(s); a key alone is allowed (empty value). Keys are looked up
// case-insensitively; when a key repeats, the last line wins (earlier entries
// stay visible through entries() / duplicates()).
#pragma once #pragma once
#include <optional> #include <optional>
@ -25,21 +37,22 @@ namespace mars::text {
struct KvEntry { struct KvEntry {
int line = 0; int line = 0;
std::string key; // original spelling std::string key; // original spelling (quotes stripped if it was quoted)
std::vector<Token> values; // 0 = key only, 1 = scalar, n = list Token value; // exactly one token
}; };
class FlatKV { class FlatKV {
public: public:
// Every pair in file order, later duplicates included.
const std::vector<KvEntry>& entries() const { return entries_; } const std::vector<KvEntry>& entries() const { return entries_; }
std::size_t size() const { return entries_.size(); } std::size_t size() const { return entries_.size(); }
// Case-insensitive lookup; the last entry with that key. // Case-insensitive lookup; the FIRST entry with that key (engine rule).
const KvEntry* find(std::string_view key) const; const KvEntry* find(std::string_view key) const;
bool has(std::string_view key) const { return find(key) != nullptr; } bool has(std::string_view key) const { return find(key) != nullptr; }
// Convenience accessors on the first value token of the entry. // Convenience accessors on the entry's value token.
// nullopt when the key is missing, has no value, or the token has the wrong shape. // nullopt when the key is missing or the token has the wrong shape.
std::optional<std::int64_t> get_int(std::string_view key) const; std::optional<std::int64_t> get_int(std::string_view key) const;
std::optional<double> get_float(std::string_view key) const; std::optional<double> get_float(std::string_view key) const;
std::optional<bool> get_bool(std::string_view key) const; std::optional<bool> get_bool(std::string_view key) const;
@ -53,7 +66,7 @@ public:
private: private:
std::vector<KvEntry> entries_; std::vector<KvEntry> entries_;
std::unordered_map<std::string, std::size_t> last_; // folded key -> index of last entry std::unordered_map<std::string, std::size_t> first_; // folded key -> index of first entry
}; };
struct Row { struct Row {
@ -65,8 +78,8 @@ using Rows = std::vector<Row>;
Result<FlatKV> parse_flat_kv(std::string_view text); Result<FlatKV> parse_flat_kv(std::string_view text);
Result<Rows> parse_rows(std::string_view text); Result<Rows> parse_rows(std::string_view text);
// Building blocks, exposed for tests and for one-off line readers. // Line-based building blocks used by parse_rows, exposed for one-off readers.
std::string_view strip_comment(std::string_view line); // drop a trailing // comment (quote-aware) std::string_view strip_comment(std::string_view line); // drop a trailing // comment (quote-aware)
std::vector<Token> split_tokens(std::string_view line); // tokenise per the grammar above std::vector<Token> split_tokens(std::string_view line); // tokenise per the row grammar above
} // namespace mars::text } // namespace mars::text

14
src/mars/text/result.h
View file

@ -12,11 +12,15 @@ namespace mars::text {
struct Problem { struct Problem {
enum class Kind { enum class Kind {
Unrecognised, // manifest: line is neither entry, comment nor tombstone Unrecognised, // manifest: line is neither entry, comment nor tombstone
DuplicateId, // manifest: id assigned twice (id = the repeated id) DuplicateId, // manifest: id assigned twice (id = the repeated id)
DeletedAndAssigned, // manifest: id is both a tombstone and an entry (line = 0) DeletedAndAssigned, // manifest: id is both a tombstone and an entry (line = 0)
UnbalancedQuote, // flat kv / rows: odd number of '"' on a line UnbalancedQuote, // rows: odd number of '"' on a line
UnterminatedQuotedCell // csv: end of input inside a quoted cell UnterminatedQuotedCell, // csv: end of input inside a quoted cell
DuplicateKey, // flat kv: key already defined (this later line is ignored)
DroppedTrailingPair, // flat kv: final pair touches end of input (no newline): dropped
UnterminatedQuote, // flat kv: a quote that runs to end of input
SkippedBlock // flat kv: a `name {` block was skipped
}; };
Kind kind; Kind kind;
int line = 0; // 1-based; 0 when the problem is not tied to a line int line = 0; // 1-based; 0 when the problem is not tied to a line

3
tests/mars_parse/canon.cpp
View file

@ -108,9 +108,6 @@ Json node_to_json(const Node& n) {
Json obj = Json::object(); Json obj = Json::object();
for (const Entry& e : n.entries) { for (const Entry& e : n.entries) {
switch (e.kind) { switch (e.kind) {
case Entry::Kind::Item:
obj.add("_items", Json::string(e.value));
break;
case Entry::Kind::Pair: case Entry::Kind::Pair:
obj.add(lower(e.key), e.quoted ? Json::string(e.value) : typed(e.value)); obj.add(lower(e.key), e.quoted ? Json::string(e.value) : typed(e.value));
break; break;

266
tests/mars_parse/test_blocks.cpp
View file

@ -3,6 +3,7 @@
#include "canon.h" #include "canon.h"
#include "mars/parse/blocks.h" #include "mars/parse/blocks.h"
#include "mars/parse/script.h"
#include "test_main.h" #include "test_main.h"
using namespace mars::parse; using namespace mars::parse;
@ -23,8 +24,74 @@ std::string canon_of(std::string_view text, Options o = {}) {
return canon::blocks_json(must_parse(text, o).root); return canon::blocks_json(must_parse(text, o).root);
} }
Options strict_opts() {
Options o;
o.strict = true;
return o;
}
} // namespace } // namespace
// ---- Script: the engine's pull tokenizer -------------------------------------
TEST(script_read_token_statuses) {
Script s("a b\n");
Script::Raw t;
CHECK(s.read_token(t) == ReadStatus::Ok);
CHECK_EQ(std::string(t.text), "a");
CHECK(s.read_token(t) == ReadStatus::Ok); // "b" is followed by the newline
CHECK_EQ(std::string(t.text), "b");
CHECK(s.read_token(t) == ReadStatus::NoInput);
CHECK(t.text.empty());
Script e("a b");
CHECK(e.read_token(t) == ReadStatus::Ok);
CHECK(e.read_token(t) == ReadStatus::AtEnd); // "b" touches the end of input
CHECK_EQ(std::string(t.text), "b");
CHECK(e.read_token(t) == ReadStatus::NoInput);
}
TEST(script_next_returns_the_value_read_status) {
Script s("k v\nname {\n}\nlast 1");
ScriptToken t;
CHECK(s.next(t) == ReadStatus::Ok);
CHECK(t.type == ScriptToken::Type::Pair);
CHECK_EQ(std::string(t.key), "k");
CHECK_EQ(std::string(t.value), "v");
CHECK(s.next(t) == ReadStatus::Ok);
CHECK(t.type == ScriptToken::Type::Open);
CHECK_EQ(std::string(t.key), "name");
CHECK(s.next(t) == ReadStatus::Ok);
CHECK(t.type == ScriptToken::Type::Close);
CHECK(s.next(t) == ReadStatus::AtEnd); // the step was read but is incomplete
CHECK_EQ(std::string(t.key), "last"); // ... its text is still available
CHECK_EQ(std::string(t.value), "1");
CHECK(s.next(t) == ReadStatus::NoInput);
}
TEST(script_skip_block_counts_whole_brace_tokens_only) {
Script s("x 1 a{ } b } after 2\n");
ScriptToken t;
CHECK(s.next(t) == ReadStatus::Ok); // x 1
CHECK(s.skip_block(1) == ReadStatus::Ok); // a{ is a word; first whole } closes
CHECK(s.next(t) == ReadStatus::Ok);
CHECK_EQ(std::string(t.key), "b"); // b } -> pair b="}"
CHECK_EQ(std::string(t.value), "}");
Script u("a { b {");
CHECK(u.skip_block(1) != ReadStatus::Ok); // input ends inside the block
}
TEST(script_token_text_is_capped_at_1023_bytes) {
const std::string text = std::string(1100, 'x') + " 1\n";
Script s(text);
ScriptToken t;
CHECK(s.next(t) == ReadStatus::Ok);
CHECK_EQ(t.key.size(), Script::kMaxTokenLength);
CHECK_EQ(std::string(t.value), "1"); // the whole word was consumed
}
// ---- tree ---------------------------------------------------------------------
TEST(basic_block_pairs_and_nesting) { TEST(basic_block_pairs_and_nesting) {
Document d = must_parse( Document d = must_parse(
"weapon\n{\n\tname @WEAPON_X\n\tcost 50\n\tbank { mount { node N1 } }\n}\n"); "weapon\n{\n\tname @WEAPON_X\n\tcost 50\n\tbank { mount { node N1 } }\n}\n");
@ -54,17 +121,33 @@ TEST(quoted_values_keep_spaces_backslashes_and_slashes) {
CHECK_EQ(*d.root.first_value("d"), ""); CHECK_EQ(*d.root.first_value("d"), "");
CHECK(d.root.first("a")->quoted); CHECK(d.root.first("a")->quoted);
CHECK(d.root.first("d")->quoted); CHECK(d.root.first("d")->quoted);
CHECK(!d.root.first("a")->key_quoted);
CHECK_EQ(d.root.entries.size(), std::size_t{4}); CHECK_EQ(d.root.entries.size(), std::size_t{4});
} }
TEST(single_quote_and_backtick_also_quote) {
Document d = must_parse("a 'x y'\nb `p q`\nc \"it's\"\nd 'say \"hi\"'\n");
CHECK_EQ(*d.root.first_value("a"), "x y");
CHECK(d.root.first("a")->quoted);
CHECK_EQ(*d.root.first_value("b"), "p q");
CHECK_EQ(*d.root.first_value("c"), "it's"); // only the opening character closes
CHECK_EQ(*d.root.first_value("d"), "say \"hi\"");
CHECK(d.warnings.empty());
}
TEST(comments_are_stripped) { TEST(comments_are_stripped) {
std::string s = canon_of("// leading\nx 1 // trailing\n//y 2\nz 3\n"); std::string s = canon_of("// leading\nx 1 // trailing\n//y 2\nz 3\n");
CHECK_EQ(s, R"({"x":1,"z":3})"); CHECK_EQ(s, R"({"x":1,"z":3})");
// '//' only starts a comment at a token boundary: glued to a bareword it is // The comment test is made on the extracted token: glued to a word it is
// part of the word (the reference reader behaves this way; the shipped data // part of the word ...
// never relies on either reading).
CHECK_EQ(canon_of("z 3//glued\n"), R"({"z":"3//glued"})"); CHECK_EQ(canon_of("z 3//glued\n"), R"({"z":"3//glued"})");
CHECK_EQ(canon_of("z 3 //not glued\n"), R"({"z":3})"); CHECK_EQ(canon_of("z 3 //not glued\n"), R"({"z":3})");
// ... and a quoted token whose content starts with // is a comment too.
CHECK_EQ(canon_of("\"// whole line\" junk\na 1\n"), R"({"a":1})");
// a closing quote glued to // : the quoted token ends, then the comment starts
CHECK_EQ(canon_of("c \"0 0 0\"// \" 92 76 20\"\n"), R"({"c":"0 0 0"})");
// comment on the last line without a newline: the pair before it survives
CHECK_EQ(canon_of("a 1 // end"), R"({"a":1})");
} }
TEST(repeated_keys_stay_in_order) { TEST(repeated_keys_stay_in_order) {
@ -88,22 +171,35 @@ TEST(keys_are_case_insensitive_but_case_is_kept) {
CHECK_EQ(canon_of("tech { Requires X requires Y }"), R"({"tech":{"requires":["X","Y"]}})"); CHECK_EQ(canon_of("tech { Requires X requires Y }"), R"({"tech":{"requires":["X","Y"]}})");
} }
TEST(bare_quoted_items_and_trailing_bareword) { TEST(quoted_strings_in_key_position_are_keys) {
Document d = must_parse("names { \"Sol\" \"Alpha Centauri\" }\nflags { on off }"); // systemnames.txt lists: the engine pairs them up like any KEY value step
auto items = d.root.first_block("names")->items(); Document d = must_parse("names { \"Sol\" \"Alpha Centauri\" \"Vega\" \"Deneb\" }\n");
CHECK_EQ(items.size(), std::size_t{2}); const Node* n = d.root.first_block("names");
CHECK_EQ(std::string(items[1]), "Alpha Centauri"); CHECK_EQ(n->entries.size(), std::size_t{2});
// "on off": 'on' takes 'off' as its value; nothing is left over. CHECK_EQ(n->entries[0].key, "Sol");
const Node* f = d.root.first_block("flags"); CHECK(n->entries[0].key_quoted);
CHECK_EQ(*f->first_value("on"), "off"); CHECK_EQ(n->entries[0].value, "Alpha Centauri");
// A lone word right before '}' is an item, not a key. CHECK(n->entries[0].quoted);
Document e = must_parse("flags { a 1 solo }"); CHECK_EQ(*n->first_value("vega"), "Deneb");
CHECK_EQ(e.root.first_block("flags")->items().size(), std::size_t{1}); CHECK_EQ(canon_of("n { \"x\" \"y\" }\n"), R"({"n":{"x":"y"}})");
CHECK_EQ(std::string(e.root.first_block("flags")->items()[0]), "solo"); }
CHECK_EQ(canon_of("flags { a 1 solo }"), R"({"flags":{"_items":"solo","a":1}})");
CHECK_EQ(canon_of("n { \"x\" \"y\" }"), R"({"n":{"_items":["x","y"]}})"); TEST(odd_item_count_pairs_the_last_item_with_the_brace) {
// lone bareword at EOF, top level // The `}` becomes a value, the block stays open and swallows what follows
CHECK_EQ(canon_of("a 1 trailing"), R"({"_items":"trailing","a":1})"); // (hiver/liir lists in the shipped systemnames.txt).
auto r = parse_blocks("n { \"A\" \"B\" \"C\" }\nm { \"D\" \"E\" }\n");
CHECK(r.ok());
CHECK_EQ(canon::blocks_json(r->root), R"({"n":{"a":"B","c":"}","m":{"d":"E"}}})");
CHECK_EQ(r->warnings.size(), std::size_t{1}); // n never closes
}
TEST(lone_word_before_a_brace_takes_the_brace_as_its_value) {
auto r = parse_blocks("flags { a 1 solo }\n");
CHECK(r.ok());
CHECK_EQ(canon::blocks_json(r->root), R"({"flags":{"a":1,"solo":"}"}})");
CHECK_EQ(r->warnings.size(), std::size_t{1}); // flags is left open
// "on off": 'on' takes 'off' as its value
CHECK_EQ(canon_of("flags { on off }\n"), R"({"flags":{"on":"off"}})");
} }
TEST(option_scalar_and_block_share_a_key) { TEST(option_scalar_and_block_share_a_key) {
@ -123,22 +219,45 @@ TEST(option_scalar_and_block_share_a_key) {
} }
TEST(tokenizer_is_not_line_based) { TEST(tokenizer_is_not_line_based) {
// block opens on the same line as a preceding pair; name glued to brace; // block opens on the same line as a preceding pair; several entries per
// several entries per line; entries split across lines // line; entries split across lines
std::string s = canon_of( std::string s = canon_of(
"bank { turretsize small mount { node N }\n}\n" "bank { turretsize small mount { node N }\n}\n"
"weapon{ name\n\nX }\n" "weapon { name\n\nX }\n"
"a\n{\nb\n1\n}"); "a\n{\nb\n1\n}\n");
CHECK_EQ(s, R"({"a":{"b":1},"bank":{"mount":{"node":"N"},"turretsize":"small"},"weapon":{"name":"X"}})"); CHECK_EQ(s, R"({"a":{"b":1},"bank":{"mount":{"node":"N"},"turretsize":"small"},"weapon":{"name":"X"}})");
} }
TEST(braces_split_barewords) { TEST(braces_are_not_delimiters) {
CHECK_EQ(canon_of("a{b 1}c 2"), R"({"a":{"b":1},"c":2})"); // Only a whole-token { or } is a brace: glued to a word it is part of the word.
// a{b -> key, 1}c -> value, then `2` is a trailing key without a value (dropped)
auto r = parse_blocks("a{b 1}c 2\n");
CHECK(r.ok());
CHECK_EQ(canon::blocks_json(r->root), R"({"a{b":"1}c"})");
CHECK_EQ(r->warnings.size(), std::size_t{1});
auto w = parse_blocks("weapon{ name X }\n");
CHECK_EQ(canon::blocks_json(w->root), R"({"weapon{":"name","x":"}"})");
CHECK(w->root.first_block("weapon") == nullptr);
CHECK_EQ(canon_of("weapon { name X }\n"), R"({"weapon":{"name":"X"}})");
} }
TEST(quotes_split_barewords) { TEST(quotes_do_not_split_barewords) {
// abc"def" -> bareword abc, then quoted def (a pair abc="def") // abc"def" is one word; a quote only opens a string at the start of a token
CHECK_EQ(canon_of("abc\"def\""), R"({"abc":"def"})"); CHECK_EQ(canon_of("abc\"def\" 1\n"), R"({"abc\"def\"":1})");
// a closing quote followed directly by text: the next token starts there
CHECK_EQ(canon_of("k \"v\"x 1\n"), R"({"k":"v","x":1})");
}
TEST(quoted_braces_count_as_braces) {
// Next() compares the stripped text, so "{" and "}" open and close blocks.
CHECK_EQ(canon_of("a \"{\" x 1 \"}\"\n"), R"({"a":{"x":1}})");
}
TEST(whitespace_is_space_tab_cr_lf_only) {
Document d = must_parse("a\v1 2\nb\f3 4\n");
CHECK(d.root.first("a\v1") != nullptr);
CHECK_EQ(*d.root.first_value("a\v1"), "2");
CHECK_EQ(*d.root.first_value("b\f3"), "4");
} }
TEST(eof_closes_open_blocks_lenient) { TEST(eof_closes_open_blocks_lenient) {
@ -153,13 +272,26 @@ TEST(eof_closes_open_blocks_lenient) {
} }
TEST(eof_inside_block_is_an_error_when_strict) { TEST(eof_inside_block_is_an_error_when_strict) {
Options strict; auto r = parse_blocks("shipsection {\n model M\n", strict_opts());
strict.strict = true;
auto r = parse_blocks("shipsection {\n model M\n", strict);
CHECK(!r.ok()); CHECK(!r.ok());
CHECK_EQ(r.error().line, 3); CHECK_EQ(r.error().line, 3);
} }
TEST(closing_brace_touching_eof_is_a_plain_close) {
auto r = parse_blocks("a { x 1 }");
CHECK(r.ok());
CHECK(r->warnings.empty());
CHECK_EQ(canon::blocks_json(r->root), R"({"a":{"x":1}})");
auto two = parse_blocks("a { b { x 1 } }");
CHECK(two->warnings.empty());
auto one_short = parse_blocks("a { b { x 1 }");
CHECK_EQ(one_short->warnings.size(), std::size_t{1}); // a is left open
CHECK(one_short->root.first_block("a")->first_block("b") != nullptr);
auto stray = parse_blocks("a { x 1 }\n}");
CHECK_EQ(stray->warnings.size(), std::size_t{1}); // stray at top level
CHECK(parse_blocks("a { x 1 }", strict_opts()).ok());
}
TEST(stray_top_level_close_is_ignored_lenient) { TEST(stray_top_level_close_is_ignored_lenient) {
auto r = parse_blocks("a { x 1 }\n}\nb { y 2 }\n"); auto r = parse_blocks("a { x 1 }\n}\nb { y 2 }\n");
CHECK(r.ok()); CHECK(r.ok());
@ -171,28 +303,63 @@ TEST(stray_top_level_close_is_ignored_lenient) {
} }
TEST(stray_top_level_close_is_an_error_when_strict) { TEST(stray_top_level_close_is_an_error_when_strict) {
Options strict; auto r = parse_blocks("a { x 1 }\n}\n", strict_opts());
strict.strict = true;
auto r = parse_blocks("a { x 1 }\n}\n", strict);
CHECK(!r.ok()); CHECK(!r.ok());
CHECK_EQ(r.error().line, 2); CHECK_EQ(r.error().line, 2);
} }
TEST(unterminated_string_is_always_an_error) { TEST(final_pair_without_trailing_newline_is_dropped) {
auto r = parse_blocks("a 1\nb 2");
CHECK(r.ok());
CHECK_EQ(canon::blocks_json(r->root), R"({"a":1})");
CHECK_EQ(r->warnings.size(), std::size_t{1});
CHECK_EQ(r->warnings[0].line, 2);
CHECK(!parse_blocks("a 1\nb 2", strict_opts()).ok());
// any trailing whitespace saves it
CHECK_EQ(canon_of("a 1\nb 2 "), R"({"a":1,"b":2})");
CHECK_EQ(canon_of("a 1\nb 2\r\n"), R"({"a":1,"b":2})");
// a quoted value whose closing quote is the last byte is dropped too
CHECK_EQ(canon_of("a 1\nb \"2\""), R"({"a":1})");
// inside a block the pair is dropped and the block is left open
auto in_block = parse_blocks("s {\n a 1\n b 2");
CHECK_EQ(canon::blocks_json(in_block->root), R"({"s":{"a":1}})");
CHECK_EQ(in_block->warnings.size(), std::size_t{2});
}
TEST(trailing_key_without_value_is_dropped) {
auto r = parse_blocks("a 1\nb\n");
CHECK_EQ(canon::blocks_json(r->root), R"({"a":1})");
CHECK_EQ(r->warnings.size(), std::size_t{1});
auto t = parse_blocks("a 1\nb"); // key itself touches the end
CHECK_EQ(canon::blocks_json(t->root), R"({"a":1})");
CHECK_EQ(t->warnings.size(), std::size_t{1});
auto h = parse_blocks("a 1\nb {"); // block header touching the end
CHECK_EQ(canon::blocks_json(h->root), R"({"a":1})");
CHECK_EQ(h->warnings.size(), std::size_t{1});
CHECK_EQ(canon_of("a 1 trailing\n"), R"({"a":1})");
}
TEST(unterminated_quote_runs_to_end_of_input) {
// The value swallows the rest of the file and then touches EOF -> dropped.
auto r = parse_blocks("a \"never closed\n b 1\n"); auto r = parse_blocks("a \"never closed\n b 1\n");
CHECK(!r.ok()); CHECK(r.ok());
CHECK_EQ(r.error().line, 1); CHECK_EQ(canon::blocks_json(r->root), "{}");
Options strict; CHECK_EQ(r->warnings.size(), std::size_t{2}); // unterminated + dropped
strict.strict = true; CHECK(!parse_blocks("a \"never closed", strict_opts()).ok());
CHECK(!parse_blocks("a \"never closed", strict).ok()); // the text is still visible through the tokenizer
Script s("a \"x\ny");
ScriptToken t;
CHECK(s.next(t) == ReadStatus::AtEnd);
CHECK(t.value_unterminated);
CHECK_EQ(std::string(t.value), "x\ny");
} }
TEST(open_brace_without_name_is_always_an_error) { TEST(open_brace_in_key_position_is_a_key) {
auto r = parse_blocks("a 1\n{ b 2 }"); auto r = parse_blocks("a 1\n{ b 2 }\n");
CHECK(!r.ok()); CHECK(r.ok());
CHECK_EQ(r.error().line, 2); CHECK_EQ(canon::blocks_json(r->root), R"({"2":"}","a":1,"{":"b"})");
// ... also when it follows a complete pair on the same line CHECK_EQ(r->warnings.size(), std::size_t{1});
CHECK(!parse_blocks("a 1 { b 2 }").ok()); CHECK(!parse_blocks("a 1\n{ b 2 }\n", strict_opts()).ok());
} }
TEST(crlf_and_line_numbers) { TEST(crlf_and_line_numbers) {
@ -217,6 +384,7 @@ TEST(empty_and_comment_only_input) {
Document e = must_parse(" // nothing here\n\n"); Document e = must_parse(" // nothing here\n\n");
CHECK(e.root.entries.empty()); CHECK(e.root.entries.empty());
CHECK_EQ(canon_of(""), "{}"); CHECK_EQ(canon_of(""), "{}");
CHECK_EQ(canon_of("// no newline at all"), "{}");
} }
TEST(top_level_pairs_and_blocks_mix) { TEST(top_level_pairs_and_blocks_mix) {
@ -227,7 +395,7 @@ TEST(top_level_pairs_and_blocks_mix) {
TEST(numbers_are_typed_only_in_canonical_form_and_only_when_bare) { TEST(numbers_are_typed_only_in_canonical_form_and_only_when_bare) {
// The Node keeps raw text; the canonical form types barewords like the oracle // The Node keeps raw text; the canonical form types barewords like the oracle
std::string s = canon_of("i 7 f .5 e 7e+8 t TRUE q \"8\" n -.8 d 5."); std::string s = canon_of("i 7 f .5 e 7e+8 t TRUE q \"8\" n -.8 d 5.\n");
CHECK_EQ(s, "{\"d\":\"\\u00015\",\"e\":\"\\u0001700000000\",\"f\":\"\\u00010.5\"," CHECK_EQ(s, "{\"d\":\"\\u00015\",\"e\":\"\\u0001700000000\",\"f\":\"\\u00010.5\","
"\"i\":7,\"n\":\"\\u0001-0.80000000000000004\",\"q\":\"8\",\"t\":true}"); "\"i\":7,\"n\":\"\\u0001-0.80000000000000004\",\"q\":\"8\",\"t\":true}");
} }
@ -241,7 +409,7 @@ TEST(high_bytes_pass_through) {
} }
TEST(deep_nesting_returns_to_the_right_parent) { TEST(deep_nesting_returns_to_the_right_parent) {
Document d = must_parse("a { b { c { d 1 } e 2 } f 3 } g 4"); Document d = must_parse("a { b { c { d 1 } e 2 } f 3 } g 4\n");
const Node* a = d.root.first_block("a"); const Node* a = d.root.first_block("a");
CHECK_EQ(*a->first_block("b")->first_block("c")->first_value("d"), "1"); CHECK_EQ(*a->first_block("b")->first_block("c")->first_value("d"), "1");
CHECK_EQ(*a->first_block("b")->first_value("e"), "2"); CHECK_EQ(*a->first_block("b")->first_value("e"), "2");
@ -251,7 +419,7 @@ TEST(deep_nesting_returns_to_the_right_parent) {
} }
TEST(lookup_misses_return_null_or_empty) { TEST(lookup_misses_return_null_or_empty) {
Document d = must_parse("a 1"); Document d = must_parse("a 1\n");
CHECK(d.root.first("zzz") == nullptr); CHECK(d.root.first("zzz") == nullptr);
CHECK(d.root.first_value("zzz") == nullptr); CHECK(d.root.first_value("zzz") == nullptr);
CHECK(d.root.first_block("a") == nullptr); // a is a pair, not a block CHECK(d.root.first_block("a") == nullptr); // a is a pair, not a block

26
tests/mars_text/dump_json.cpp
View file

@ -7,6 +7,7 @@
#include <iterator> #include <iterator>
#include <map> #include <map>
#include <string> #include <string>
#include <vector>
#include "mars/text/csv.h" #include "mars/text/csv.h"
#include "mars/text/flat_kv.h" #include "mars/text/flat_kv.h"
@ -64,24 +65,21 @@ void json_tokens(const std::vector<Token>& toks, std::string& out) {
out.push_back(']'); out.push_back(']');
} }
// kv value: null / scalar / list, matching the reference reader's shape
void json_kv_value(const std::vector<Token>& toks, std::string& out) {
if (toks.empty()) out += "null";
else if (toks.size() == 1) json_token(toks[0], out);
else json_tokens(toks, out);
}
std::string dump_kv(const FlatKV& kv) { std::string dump_kv(const FlatKV& kv) {
std::map<std::string, const KvEntry*> last; // exact key, last wins // engine rule: keys compared case-insensitively, first occurrence wins,
for (const KvEntry& e : kv.entries()) last[e.key] = &e; // reported under its first spelling (the reference reader's dict shape)
std::map<std::string, const KvEntry*> first_seen;
std::vector<const KvEntry*> order;
for (const KvEntry& e : kv.entries())
if (first_seen.emplace(fold_case(e.key), &e).second) order.push_back(&e);
std::string out = "{\"values\":{"; std::string out = "{\"values\":{";
bool first = true; bool first = true;
for (const auto& [key, e] : last) { for (const KvEntry* e : order) {
if (!first) out.push_back(','); if (!first) out.push_back(',');
first = false; first = false;
json_string(key, out); json_string(e->key, out);
out.push_back(':'); out.push_back(':');
json_kv_value(e->values, out); json_token(e->value, out);
} }
out += "},\"duplicates\":{"; out += "},\"duplicates\":{";
first = true; first = true;
@ -114,6 +112,10 @@ const char* problem_kind(Problem::Kind k) {
case Problem::Kind::DeletedAndAssigned: return "deleted_and_assigned"; case Problem::Kind::DeletedAndAssigned: return "deleted_and_assigned";
case Problem::Kind::UnbalancedQuote: return "unbalanced_quote"; case Problem::Kind::UnbalancedQuote: return "unbalanced_quote";
case Problem::Kind::UnterminatedQuotedCell: return "unterminated_quoted_cell"; case Problem::Kind::UnterminatedQuotedCell: return "unterminated_quoted_cell";
case Problem::Kind::DuplicateKey: return "duplicate_key";
case Problem::Kind::DroppedTrailingPair: return "dropped_trailing_pair";
case Problem::Kind::UnterminatedQuote: return "unterminated_quote";
case Problem::Kind::SkippedBlock: return "skipped_block";
} }
return "?"; return "?";
} }

15
tests/mars_text/realdata_test.cpp
View file

@ -56,12 +56,21 @@ int main() {
auto endgame = r.value.get_color("ENDGAME_FILL_COLOR"); auto endgame = r.value.get_color("ENDGAME_FILL_COLOR");
CHECK(endgame && endgame->components == 3); CHECK(endgame && endgame->components == 3);
} }
// StrategyVars.txt: typed access // StrategyVars.txt: typed access; the file has no trailing newline, so the
// engine drops its last pair (CIVILIAN_BURDEN_RATIO keeps the image default)
{ {
auto r = parse_flat_kv(read_file(base + "Data/Strategy/StrategyVars.txt")); auto r = parse_flat_kv(read_file(base + "Data/Strategy/StrategyVars.txt"));
CHECK(r.ok()); CHECK(r.value.size() == 96);
CHECK(r.value.size() == 97);
CHECK(r.value.get_float("SLAVES_DEATH_RATE").has_value()); CHECK(r.value.get_float("SLAVES_DEATH_RATE").has_value());
CHECK(!r.value.has("CIVILIAN_BURDEN_RATIO"));
CHECK(r.problems.size() == 1 && r.problems[0].kind == Problem::Kind::DroppedTrailingPair);
}
// encounters.txt: same, HERALD_SPEECH_MAX_INVERVAL is never read
{
auto r = parse_flat_kv(read_file(base + "Data/encounters.txt"));
CHECK(!r.value.has("HERALD_SPEECH_MAX_INVERVAL"));
CHECK(r.problems.size() == 1 && r.problems[0].kind == Problem::Kind::DroppedTrailingPair);
CHECK(r.value.duplicates().empty());
} }
// _turrets.txt: 42 rows, 8 columns, quoted model in the last column // _turrets.txt: 42 rows, 8 columns, quoted model in the last column
{ {

119
tests/mars_text/unit_tests.cpp
View file

@ -121,14 +121,13 @@ static void test_flat_kv() {
"EMPTY \"\"\n" "EMPTY \"\"\n"
"COLOR \"255\t177\t\t39\"\n" "COLOR \"255\t177\t\t39\"\n"
"FLAG true\n" "FLAG true\n"
"NAME_ONLY\n"
"LIST 1 2 three\n"
"path \"Data\\\\x.txt\"\n" "path \"Data\\\\x.txt\"\n"
"SLAVES_DEATH_RATE 0.10\n" "slaves_death_rate 0.10\n"
"BADQ \"oops\n"; "ENDGAME \"0 0 0\"// \" 92 76 20\"\n"
"LAST 7\n";
auto r = parse_flat_kv(text); auto r = parse_flat_kv(text);
const FlatKV& kv = r.value; const FlatKV& kv = r.value;
CHECK_EQ(kv.size(), 11u); CHECK_EQ(kv.size(), 10u); // every pair in file order, the duplicate included
// typed access, case-insensitive lookup, original spelling preserved // typed access, case-insensitive lookup, original spelling preserved
CHECK_EQ(kv.get_int("max_ships").value(), 12); CHECK_EQ(kv.get_int("max_ships").value(), 12);
@ -136,49 +135,118 @@ static void test_flat_kv() {
CHECK_EQ(kv.find("MAX_SHIPS")->line, 4); CHECK_EQ(kv.find("MAX_SHIPS")->line, 4);
CHECK(!kv.get_int("TITLE")); CHECK(!kv.get_int("TITLE"));
CHECK_EQ(kv.get_string("TITLE").value(), std::string_view("Sword // of the Stars")); CHECK_EQ(kv.get_string("TITLE").value(), std::string_view("Sword // of the Stars"));
CHECK(kv.find("TITLE")->values[0].quoted); CHECK(kv.find("TITLE")->value.quoted);
CHECK_EQ(kv.get_string("EMPTY").value(), std::string_view("")); CHECK_EQ(kv.get_string("EMPTY").value(), std::string_view(""));
auto c = kv.get_color("COLOR"); auto c = kv.get_color("COLOR");
CHECK(c && c->r == 255 && c->g == 177 && c->b == 39); CHECK(c && c->r == 255 && c->g == 177 && c->b == 39);
CHECK_EQ(kv.get_bool("flag").value(), true); CHECK_EQ(kv.get_bool("flag").value(), true);
CHECK(kv.has("NAME_ONLY"));
CHECK(kv.find("NAME_ONLY")->values.empty());
CHECK(!kv.get_string("NAME_ONLY"));
CHECK_EQ(kv.find("LIST")->values.size(), 3u);
CHECK(kv.find("LIST")->values[2].kind() == ValueKind::Bareword);
CHECK_EQ(kv.get_int("LIST").value(), 1); // first token
CHECK_EQ(kv.get_string("path").value(), std::string_view("Data\\\\x.txt")); // no escape processing CHECK_EQ(kv.get_string("path").value(), std::string_view("Data\\\\x.txt")); // no escape processing
CHECK_EQ(kv.get_string("ENDGAME").value(), std::string_view("0 0 0")); // closing quote glued to //
CHECK_EQ(kv.get_int("LAST").value(), 7);
CHECK(!kv.has("missing")); CHECK(!kv.has("missing"));
CHECK(!kv.get_int("missing")); CHECK(!kv.get_int("missing"));
// duplicates: last wins for lookup, both entries kept // duplicates: FIRST occurrence wins (engine erases a key once consumed);
CHECK_EQ(kv.get_float("slaves_death_rate").value(), 0.10); // the later line is reported and stays visible in entries()/duplicates()
CHECK_EQ(kv.get_float("slaves_death_rate").value(), 0.05);
CHECK_EQ(kv.find("SLAVES_DEATH_RATE")->line, 3);
auto dups = kv.duplicates(); auto dups = kv.duplicates();
CHECK_EQ(dups.size(), 1u); CHECK_EQ(dups.size(), 1u);
CHECK(dups[0].first == "slaves_death_rate"); CHECK(dups[0].first == "slaves_death_rate");
CHECK((dups[0].second == std::vector<int>{3, 12})); CHECK((dups[0].second == std::vector<int>{3, 10}));
// unbalanced quote: line still parsed, warning reported
CHECK_EQ(r.problems.size(), 1u); CHECK_EQ(r.problems.size(), 1u);
CHECK(r.problems[0].kind == Problem::Kind::UnbalancedQuote && r.problems[0].line == 13); CHECK(r.problems[0].kind == Problem::Kind::DuplicateKey && r.problems[0].line == 10);
CHECK_EQ(kv.get_string("BADQ").value(), std::string_view("\"oops"));
// key case only differs: a duplicate for us // key case only differs: the same key to the engine (_stricmp)
auto r2 = parse_flat_kv("Foo 1\nFOO 2\n"); auto r2 = parse_flat_kv("Foo 1\nFOO 2\n");
CHECK_EQ(r2.value.get_int("foo").value(), 2); CHECK_EQ(r2.value.get_int("foo").value(), 1);
CHECK_EQ(r2.value.duplicates().size(), 1u); CHECK_EQ(r2.value.duplicates().size(), 1u);
CHECK(r2.problems.size() == 1 && r2.problems[0].kind == Problem::Kind::DuplicateKey);
// empty / comment-only input // empty / comment-only input
CHECK_EQ(parse_flat_kv("").value.size(), 0u); CHECK_EQ(parse_flat_kv("").value.size(), 0u);
CHECK_EQ(parse_flat_kv("// nothing\n\n \n").value.size(), 0u); CHECK_EQ(parse_flat_kv("// nothing\n\n \n").value.size(), 0u);
// line numbers with mixed terminators CHECK(parse_flat_kv("// no newline").ok());
auto r3 = parse_flat_kv("A 1\rB 2\r\nC 3\nD 4"); // line numbers count '\n' only (mixed terminators)
CHECK_EQ(r3.value.find("D")->line, 4); auto r3 = parse_flat_kv("A 1\rB 2\r\nC 3\nD 4\n");
CHECK_EQ(r3.value.find("B")->line, 1);
CHECK_EQ(r3.value.find("D")->line, 3);
// high bytes pass through untouched // high bytes pass through untouched
auto r4 = parse_flat_kv("K \"caf\xe9 \x93q\x94\""); auto r4 = parse_flat_kv("K \"caf\xe9 \x93q\x94\"\n");
CHECK_EQ(r4.value.get_string("K").value(), std::string_view("caf\xe9 \x93q\x94")); CHECK_EQ(r4.value.get_string("K").value(), std::string_view("caf\xe9 \x93q\x94"));
} }
// The loader is the engine's token stepper, not a line reader.
static void test_flat_kv_engine_steps() {
// one value token per key: extra tokens start the next pair
auto r = parse_flat_kv("LIST 1 2 three\n");
CHECK_EQ(r.value.size(), 2u);
CHECK_EQ(r.value.get_int("LIST").value(), 1);
CHECK_EQ(r.value.get_string("2").value(), std::string_view("three"));
// a key alone on a line takes the next line's key as its value
auto k = parse_flat_kv("A\nB 2\nC 3\n");
CHECK_EQ(k.value.get_string("A").value(), std::string_view("B"));
CHECK_EQ(k.value.get_string("2").value(), std::string_view("C"));
CHECK(!k.value.has("B"));
CHECK(k.problems.size() == 1 && k.problems[0].kind == Problem::Kind::DroppedTrailingPair); // "3"
// braces are not delimiters
auto b = parse_flat_kv("A{ 1\n");
CHECK(b.value.has("A{") && !b.value.has("A"));
// the final pair is dropped when its value touches the end of input
auto d = parse_flat_kv("A 1\nB 2");
CHECK(d.value.has("A") && !d.value.has("B"));
CHECK_EQ(d.value.size(), 1u);
CHECK(d.problems.size() == 1 && d.problems[0].kind == Problem::Kind::DroppedTrailingPair &&
d.problems[0].line == 2);
CHECK(parse_flat_kv("A 1\nB \"2\"").problems.size() == 1); // closing quote as last byte: same
CHECK(parse_flat_kv("A 1\nB 2 ").value.has("B")); // any trailing whitespace saves it
CHECK(parse_flat_kv("A 1\nB 2 // c").value.has("B")); // ... or a comment
auto lone = parse_flat_kv("A 1\nB\n");
CHECK(!lone.value.has("B") && lone.problems.size() == 1);
auto lone2 = parse_flat_kv("A 1\nB");
CHECK(!lone2.value.has("B") && lone2.problems.size() == 1);
// quotes: ", ' and backtick, no escapes; a quote inside a word is literal
auto q = parse_flat_kv("A 'x y'\nB `p q`\nC it's\nD \"say \\\"hi\" 5\n");
CHECK_EQ(q.value.get_string("A").value(), std::string_view("x y"));
CHECK(q.value.find("A")->value.quoted);
CHECK_EQ(q.value.get_string("B").value(), std::string_view("p q"));
CHECK_EQ(q.value.get_string("C").value(), std::string_view("it's"));
CHECK_EQ(q.value.get_string("D").value(), std::string_view("say \\")); // backslash does not escape
CHECK_EQ(q.value.get_int("hi\"").value(), 5); // the rest is a new key
CHECK(q.ok());
// an unterminated quote runs to the end of input (and so the pair is dropped)
auto u = parse_flat_kv("A \"never\nB 2\n");
CHECK_EQ(u.value.size(), 0u);
CHECK_EQ(u.problems.size(), 2u);
CHECK(u.problems[0].kind == Problem::Kind::UnterminatedQuote);
CHECK(u.problems[1].kind == Problem::Kind::DroppedTrailingPair);
// comments: the test is on the extracted token
auto c = parse_flat_kv("A 3//x\n\"// whole line\" junk\nB 2\n");
CHECK_EQ(c.value.get_string("A").value(), std::string_view("3//x"));
CHECK(c.value.find("A")->value.kind() == ValueKind::Bareword);
CHECK_EQ(c.value.get_int("B").value(), 2);
CHECK_EQ(c.value.size(), 2u);
// a block is skipped; a stray } is ignored
auto blk = parse_flat_kv("A 1\nblk {\n X 2\n sub { Y 3 }\n}\n}\nB 4\n");
CHECK_EQ(blk.value.size(), 2u);
CHECK(blk.value.has("A") && blk.value.has("B") && !blk.value.has("X"));
CHECK(blk.problems.size() == 1 && blk.problems[0].kind == Problem::Kind::SkippedBlock);
// tokens are capped at 1023 bytes
std::string longkey(1100, 'k');
auto cap = parse_flat_kv(longkey + " 1\n");
CHECK_EQ(cap.value.entries()[0].key.size(), 1023u);
CHECK(cap.value.has(longkey.substr(0, 1023)));
// whitespace is space, tab, CR, LF only
auto ws = parse_flat_kv("A\v1 2\n");
CHECK(ws.value.has("A\v1"));
}
static void test_rows() { static void test_rows() {
const char* text = const char* text =
"// size weapon-size class health \"model\"\r\n" "// size weapon-size class health \"model\"\r\n"
@ -329,6 +397,7 @@ int main() {
test_color(); test_color();
test_strip_comment_and_tokens(); test_strip_comment_and_tokens();
test_flat_kv(); test_flat_kv();
test_flat_kv_engine_steps();
test_rows(); test_rows();
test_manifest(); test_manifest();
test_csv_records(); test_csv_records();