88 lines
6.7 KiB
Markdown
88 lines
6.7 KiB
Markdown
# Reader parity with the original engine — 2026-09-07
|
||
|
||
Both reader stacks — the C++ modules in `sots-engine` (`src/mars/parse`,
|
||
`src/mars/text`) and these Python oracles — now implement the loader
|
||
semantics Ghidra confirmed in `findings/subsystems/loader-prototypes.md`
|
||
(§M1 `GlobalConsts::LoadFile`, §M3 `Mars::Script`). This file lists each rule,
|
||
where the evidence is, and what it does to the shipped data. "Before" is the
|
||
previous tree-grammar / line-based reading; all runs are over the owner's
|
||
`gob-extract` (1,595 data files).
|
||
|
||
## Rules
|
||
|
||
| # | rule | evidence | before | after |
|
||
|---|---|---|---|---|
|
||
| 1 | Flat kv: keys compared with `_stricmp`, **first occurrence wins**; later duplicates are "multiply defined" and ignored | §M1 `LoadFile` pseudo-code (`consts->erase(node)` after the parser runs; the "not recognized or is multiply defined" log) | `parse_kv(on_dup='last')`, exact-key dict | `on_dup='first'` default, keys folded, first spelling kept; `warnings` records duplicates |
|
||
| 2 | Flat kv is read with the `Mars::Script` stepper, not by lines: `KEY value` = two tokens, `NAME {` skipped via `SkipBlock(1)`, `}` ignored | §M1 loop (`Script::Next`, `tok.type` 1/2/3), leniency bullets ("the value is one token", "`{}` blocks … follow the `Mars::Script` tokenizer rules (M3)") | line split, first token = key, rest = value list | `flat_kv._steps()` over `mars_data.Script` |
|
||
| 3 | Final `KEY value` whose value touches EOF is **dropped** (`Next` returns the status of the value read; loops stop on non-zero) | §M3 `Next()` pseudo-code and consequence bullet; §M1 comment "EOF (or last pair without trailing newline)" | kept | dropped, warning `final pair … touches end of input` |
|
||
| 4 | Whitespace = space, tab, CR, LF only | §M3 tokenizer rules, bullet 1 | Python `\s` (also `\v \f \x1c-\x1f \xa0`), C++ `isspace` set | exact four characters |
|
||
| 5 | Braces are **not** delimiters; `{`/`}` only as whole tokens (`strcmp` on the buffer) | §M3 bullet 3 ("barewords end only at whitespace… `weapon{` is one token"), `Next()` `strcmp` | `{ } "` cut a bareword | bareword runs to whitespace; quoted `"{"`/`"}"` act as braces |
|
||
| 6 | Quotes `"`, `'`, backtick; closes at the same character; no escapes; unterminated runs to EOF | §M3 bullet 2 | `"` only; unterminated = hard error | all three; unterminated tolerated (warning) |
|
||
| 7 | A token whose extracted text starts with `//` is a comment (bare or quoted); glued `3//x` is one word | §M3 comment bullet | `//` recognised only at a token boundary in the raw text (same result for bare; quoted-`//` was a string) | test on the extracted token |
|
||
| 8 | EOF ends parsing anywhere (unclosed blocks fine); stray top-level `}` ignored | §M3 consequence bullets 1–2 | same (already lenient) | same; warnings kept |
|
||
| 9 | A quoted string in key position is a key; its value is the next token | §M3 consequence bullet 4 (systemnames) | `_items` list | pair with `key_quoted` |
|
||
| 10 | Token text capped at 1023 bytes | §M3 bullet "output is truncated silently at outMax-1" | none | cap in both readers (longest shipped token is 95 bytes) |
|
||
|
||
Where the doc is silent we kept prior behaviour and say so: text glued after
|
||
a closing quote starts a new token (`"0 0 0"// x` in `globals.txt:230`
|
||
yields `0 0 0` then a comment — the only such spot in the data); `SkipBlock`
|
||
is assumed to count raw brace tokens; a `}` that closes a block and is the
|
||
last byte of the file is treated as a plain close (the original stops there
|
||
either way — identical outcome, no warning; 27 catalog files end that way);
|
||
the row tables (`_turrets.txt` etc.) and `.effect` files are read by other
|
||
engine classes and were not changed.
|
||
|
||
## Effect on the shipped data
|
||
|
||
### Flat kv (20 files, `Data/**/*.txt`)
|
||
|
||
- **Duplicates: none.** No shipped kv file repeats a key, exactly or
|
||
case-folded, so first-wins flips no value.
|
||
- **Dropped trailing pairs: 2 files, 1 key each** (file has no trailing newline):
|
||
- `Data/Strategy/StrategyVars.txt` line 118: `CIVILIAN_BURDEN_RATIO 0.5` — never read; the game runs on the compiled-in default.
|
||
- `Data/encounters.txt` line 225: `HERALD_SPEECH_MAX_INVERVAL 45` — same.
|
||
- Multi-token values, `{}` blocks, `'`/backtick quotes, unterminated quotes:
|
||
none in shipped kv files (single quotes appear only inside `//` comments).
|
||
- `globals.txt`: 364 keys, unchanged; line 230 (`"0 0 0"// …`) reads the same.
|
||
|
||
### Brace-block (1,116 files)
|
||
|
||
- Canonical output changed for **one file**: `Data/Strategy/systemnames.txt`.
|
||
Its lists are bare quoted names; under `Next()` they pair up
|
||
(`"Procyon" "Deneb"` → key `procyon` = `Deneb`). `hiver` (87 names) and
|
||
`liir` (81) are odd, so their last name takes `}` as its value and the
|
||
block stays open: `tarkas` and `liir` nest inside `hiver`, `morrigi` inside
|
||
`liir`; two blocks are left open at EOF (2 warnings). The original's
|
||
system-name loader interprets that stream itself; a dedicated consumer is
|
||
needed for name-generation parity (open question in `docs/mars-parse.md`).
|
||
- Every `.weapon`, `.shipsection`, `.tech`, `.combat`, `.def`, `.script`,
|
||
scenario and other block-form `.txt` renders **byte-identically** to before;
|
||
`verify.py`'s artifacts (`tech_tree.json`, `weapons.json`,
|
||
`shipsections.json`, `strings.json`, `schema_stats.json`, `crosslink.json`,
|
||
`tech_tree.dot`) are unchanged.
|
||
- No brace file drops a pair (all end in `}` or a newline); 27 end with `}`
|
||
as the last byte (plain close); no unterminated quotes, no quoted-`//`
|
||
tokens, no `'`/backtick-quoted tokens, longest token 95 bytes
|
||
(`skydefs.txt`).
|
||
- `.effect` files (415): untouched reader, unchanged.
|
||
|
||
## Oracle results
|
||
|
||
| suite | before | after |
|
||
|---|---|---|
|
||
| `verify.py` | 1,595 parsed / 0 failed; 12 tolerances (12 shipsections) | 1,595 / 0; 15 tolerances (12 shipsections + `systemnames.txt` + 2 kv dropped pairs); artifacts identical |
|
||
| engine `tests/mars_parse/build_and_run.sh` (vs. updated oracle) | 1,531 / 1,531 | **1,531 / 1,531**, 51 unit tests, warning counts equal (13 files) |
|
||
| engine `tests/mars_text/build_and_run.sh` (vs. updated oracle) | 64 / 64 | **64 / 64**, 201 unit checks, 205 real-data checks |
|
||
|
||
## Files
|
||
|
||
- Python (this dir): `mars_data.py` (new `Script` tokenizer + step-loop
|
||
parser; `_items` gone), `flat_kv.py` (`parse_kv` over `Script`, first-wins,
|
||
`warnings=`), `verify.py` (kv warnings reported; counts unchanged).
|
||
- Engine worktree `~/sots-engine-wt/engine-parity` (branch `wip/engine-parity`):
|
||
`src/mars/parse/script.h` (new, header-only, shared), `blocks.{h,cpp}`,
|
||
`src/mars/text/flat_kv.{h,cpp}`, `result.h` (new Problem kinds),
|
||
`tests/mars_parse/{test_blocks,canon}.cpp`,
|
||
`tests/mars_text/{unit_tests,realdata_test,dump_json}.cpp`,
|
||
`docs/mars-parse.md`, `docs/mars-text.md`. No CMake or build-script edits;
|
||
no game data or decompiler text.
|