# Reader parity with the original engine — 2026-09-07

Both reader stacks — the C++ modules in `sots-engine` (`src/mars/parse`,
`src/mars/text`) and these Python oracles — now implement the loader
semantics Ghidra confirmed in `findings/subsystems/loader-prototypes.md`
(§M1 `GlobalConsts::LoadFile`, §M3 `Mars::Script`). This file lists each rule,
where the evidence is, and what it does to the shipped data. "Before" is the
previous tree-grammar / line-based reading; all runs are over the owner's
`gob-extract` (1,595 data files).

## Rules

| # | rule | evidence | before | after |
|---|---|---|---|---|
| 1 | Flat kv: keys compared with `_stricmp`, **first occurrence wins**; later duplicates are "multiply defined" and ignored | §M1 `LoadFile` pseudo-code (`consts->erase(node)` after the parser runs; the "not recognized or is multiply defined" log) | `parse_kv(on_dup='last')`, exact-key dict | `on_dup='first'` default, keys folded, first spelling kept; `warnings` records duplicates |
| 2 | Flat kv is read with the `Mars::Script` stepper, not by lines: `KEY value` = two tokens, `NAME {` skipped via `SkipBlock(1)`, `}` ignored | §M1 loop (`Script::Next`, `tok.type` 1/2/3), leniency bullets ("the value is one token", "`{}` blocks … follow the `Mars::Script` tokenizer rules (M3)") | line split, first token = key, rest = value list | `flat_kv._steps()` over `mars_data.Script` |
| 3 | Final `KEY value` whose value touches EOF is **dropped** (`Next` returns the status of the value read; loops stop on non-zero) | §M3 `Next()` pseudo-code and consequence bullet; §M1 comment "EOF (or last pair without trailing newline)" | kept | dropped, warning `final pair … touches end of input` |
| 4 | Whitespace = space, tab, CR, LF only | §M3 tokenizer rules, bullet 1 | Python `\s` (also `\v \f \x1c-\x1f \xa0`), C++ `isspace` set | exact four characters |
| 5 | Braces are **not** delimiters; `{`/`}` only as whole tokens (`strcmp` on the buffer) | §M3 bullet 3 ("barewords end only at whitespace… `weapon{` is one token"), `Next()` `strcmp` | `{ } "` cut a bareword | bareword runs to whitespace; quoted `"{"`/`"}"` act as braces |
| 6 | Quotes `"`, `'`, backtick; closes at the same character; no escapes; unterminated runs to EOF | §M3 bullet 2 | `"` only; unterminated = hard error | all three; unterminated tolerated (warning) |
| 7 | A token whose extracted text starts with `//` is a comment (bare or quoted); glued `3//x` is one word | §M3 comment bullet | `//` recognised only at a token boundary in the raw text (same result for bare; quoted-`//` was a string) | test on the extracted token |
| 8 | EOF ends parsing anywhere (unclosed blocks fine); stray top-level `}` ignored | §M3 consequence bullets 1–2 | same (already lenient) | same; warnings kept |
| 9 | A quoted string in key position is a key; its value is the next token | §M3 consequence bullet 4 (systemnames) | `_items` list | pair with `key_quoted` |
| 10 | Token text capped at 1023 bytes | §M3 bullet "output is truncated silently at outMax-1" | none | cap in both readers (longest shipped token is 95 bytes) |

Where the doc is silent we kept prior behaviour and say so: text glued after
a closing quote starts a new token (`"0 0 0"// x` in `globals.txt:230`
yields `0 0 0` then a comment — the only such spot in the data); `SkipBlock`
is assumed to count raw brace tokens; a `}` that closes a block and is the
last byte of the file is treated as a plain close (the original stops there
either way — identical outcome, no warning; 27 catalog files end that way);
the row tables (`_turrets.txt` etc.) and `.effect` files are read by other
engine classes and were not changed.

## Effect on the shipped data

### Flat kv (20 files, `Data/**/*.txt`)

- **Duplicates: none.** No shipped kv file repeats a key, exactly or
  case-folded, so first-wins flips no value.
- **Dropped trailing pairs: 2 files, 1 key each** (file has no trailing newline):
  - `Data/Strategy/StrategyVars.txt` line 118: `CIVILIAN_BURDEN_RATIO 0.5` — never read; the game runs on the compiled-in default.
  - `Data/encounters.txt` line 225: `HERALD_SPEECH_MAX_INVERVAL 45` — same.
- Multi-token values, `{}` blocks, `'`/backtick quotes, unterminated quotes:
  none in shipped kv files (single quotes appear only inside `//` comments).
- `globals.txt`: 364 keys, unchanged; line 230 (`"0 0 0"// …`) reads the same.

### Brace-block (1,116 files)

- Canonical output changed for **one file**: `Data/Strategy/systemnames.txt`.
  Its lists are bare quoted names; under `Next()` they pair up
  (`"Procyon" "Deneb"` → key `procyon` = `Deneb`). `hiver` (87 names) and
  `liir` (81) are odd, so their last name takes `}` as its value and the
  block stays open: `tarkas` and `liir` nest inside `hiver`, `morrigi` inside
  `liir`; two blocks are left open at EOF (2 warnings). The original's
  system-name loader interprets that stream itself; a dedicated consumer is
  needed for name-generation parity (open question in `docs/mars-parse.md`).
- Every `.weapon`, `.shipsection`, `.tech`, `.combat`, `.def`, `.script`,
  scenario and other block-form `.txt` renders **byte-identically** to before;
  `verify.py`'s artifacts (`tech_tree.json`, `weapons.json`,
  `shipsections.json`, `strings.json`, `schema_stats.json`, `crosslink.json`,
  `tech_tree.dot`) are unchanged.
- No brace file drops a pair (all end in `}` or a newline); 27 end with `}`
  as the last byte (plain close); no unterminated quotes, no quoted-`//`
  tokens, no `'`/backtick-quoted tokens, longest token 95 bytes
  (`skydefs.txt`).
- `.effect` files (415): untouched reader, unchanged.

## Oracle results

| suite | before | after |
|---|---|---|
| `verify.py` | 1,595 parsed / 0 failed; 12 tolerances (12 shipsections) | 1,595 / 0; 15 tolerances (12 shipsections + `systemnames.txt` + 2 kv dropped pairs); artifacts identical |
| engine `tests/mars_parse/build_and_run.sh` (vs. updated oracle) | 1,531 / 1,531 | **1,531 / 1,531**, 51 unit tests, warning counts equal (13 files) |
| engine `tests/mars_text/build_and_run.sh` (vs. updated oracle) | 64 / 64 | **64 / 64**, 201 unit checks, 205 real-data checks |

## Files

- Python (this dir): `mars_data.py` (new `Script` tokenizer + step-loop
  parser; `_items` gone), `flat_kv.py` (`parse_kv` over `Script`, first-wins,
  `warnings=`), `verify.py` (kv warnings reported; counts unchanged).
- Engine worktree `~/sots-engine-wt/engine-parity` (branch `wip/engine-parity`):
  `src/mars/parse/script.h` (new, header-only, shared), `blocks.{h,cpp}`,
  `src/mars/text/flat_kv.{h,cpp}`, `result.h` (new Problem kinds),
  `tests/mars_parse/{test_blocks,canon}.cpp`,
  `tests/mars_text/{unit_tests,realdata_test,dump_json}.cpp`,
  `docs/mars-parse.md`, `docs/mars-text.md`. No CMake or build-script edits;
  no game data or decompiler text.