parsers: engine parity (first-wins, Mars::Script tokenizer rules, trailing-pair drop); untrack pyc

verify/parsers/PARITY_NOTES.md

@@ -0,0 +1,88 @@
+# Reader parity with the original engine — 2026-09-07
+
+Both reader stacks — the C++ modules in `sots-engine` (`src/mars/parse`,
+`src/mars/text`) and these Python oracles — now implement the loader
+semantics Ghidra confirmed in `findings/subsystems/loader-prototypes.md`
+(§M1 `GlobalConsts::LoadFile`, §M3 `Mars::Script`). This file lists each rule,
+where the evidence is, and what it does to the shipped data. "Before" is the
+previous tree-grammar / line-based reading; all runs are over the owner's
+`gob-extract` (1,595 data files).
+
+## Rules
+
+| # | rule | evidence | before | after |
+|---|---|---|---|---|
+| 1 | Flat kv: keys compared with `_stricmp`, **first occurrence wins**; later duplicates are "multiply defined" and ignored | §M1 `LoadFile` pseudo-code (`consts->erase(node)` after the parser runs; the "not recognized or is multiply defined" log) | `parse_kv(on_dup='last')`, exact-key dict | `on_dup='first'` default, keys folded, first spelling kept; `warnings` records duplicates |
+| 2 | Flat kv is read with the `Mars::Script` stepper, not by lines: `KEY value` = two tokens, `NAME {` skipped via `SkipBlock(1)`, `}` ignored | §M1 loop (`Script::Next`, `tok.type` 1/2/3), leniency bullets ("the value is one token", "`{}` blocks … follow the `Mars::Script` tokenizer rules (M3)") | line split, first token = key, rest = value list | `flat_kv._steps()` over `mars_data.Script` |
+| 3 | Final `KEY value` whose value touches EOF is **dropped** (`Next` returns the status of the value read; loops stop on non-zero) | §M3 `Next()` pseudo-code and consequence bullet; §M1 comment "EOF (or last pair without trailing newline)" | kept | dropped, warning `final pair … touches end of input` |
+| 4 | Whitespace = space, tab, CR, LF only | §M3 tokenizer rules, bullet 1 | Python `\s` (also `\v \f \x1c-\x1f \xa0`), C++ `isspace` set | exact four characters |
+| 5 | Braces are **not** delimiters; `{`/`}` only as whole tokens (`strcmp` on the buffer) | §M3 bullet 3 ("barewords end only at whitespace… `weapon{` is one token"), `Next()` `strcmp` | `{ } "` cut a bareword | bareword runs to whitespace; quoted `"{"`/`"}"` act as braces |
+| 6 | Quotes `"`, `'`, backtick; closes at the same character; no escapes; unterminated runs to EOF | §M3 bullet 2 | `"` only; unterminated = hard error | all three; unterminated tolerated (warning) |
+| 7 | A token whose extracted text starts with `//` is a comment (bare or quoted); glued `3//x` is one word | §M3 comment bullet | `//` recognised only at a token boundary in the raw text (same result for bare; quoted-`//` was a string) | test on the extracted token |
+| 8 | EOF ends parsing anywhere (unclosed blocks fine); stray top-level `}` ignored | §M3 consequence bullets 1–2 | same (already lenient) | same; warnings kept |
+| 9 | A quoted string in key position is a key; its value is the next token | §M3 consequence bullet 4 (systemnames) | `_items` list | pair with `key_quoted` |
+| 10 | Token text capped at 1023 bytes | §M3 bullet "output is truncated silently at outMax-1" | none | cap in both readers (longest shipped token is 95 bytes) |
+
+Where the doc is silent we kept prior behaviour and say so: text glued after
+a closing quote starts a new token (`"0 0 0"// x` in `globals.txt:230`
+yields `0 0 0` then a comment — the only such spot in the data); `SkipBlock`
+is assumed to count raw brace tokens; a `}` that closes a block and is the
+last byte of the file is treated as a plain close (the original stops there
+either way — identical outcome, no warning; 27 catalog files end that way);
+the row tables (`_turrets.txt` etc.) and `.effect` files are read by other
+engine classes and were not changed.
+
+## Effect on the shipped data
+
+### Flat kv (20 files, `Data/**/*.txt`)
+
+- **Duplicates: none.** No shipped kv file repeats a key, exactly or
+  case-folded, so first-wins flips no value.
+- **Dropped trailing pairs: 2 files, 1 key each** (file has no trailing newline):
+  - `Data/Strategy/StrategyVars.txt` line 118: `CIVILIAN_BURDEN_RATIO 0.5` — never read; the game runs on the compiled-in default.
+  - `Data/encounters.txt` line 225: `HERALD_SPEECH_MAX_INVERVAL 45` — same.
+- Multi-token values, `{}` blocks, `'`/backtick quotes, unterminated quotes:
+  none in shipped kv files (single quotes appear only inside `//` comments).
+- `globals.txt`: 364 keys, unchanged; line 230 (`"0 0 0"// …`) reads the same.
+
+### Brace-block (1,116 files)
+
+- Canonical output changed for **one file**: `Data/Strategy/systemnames.txt`.
+  Its lists are bare quoted names; under `Next()` they pair up
+  (`"Procyon" "Deneb"` → key `procyon` = `Deneb`). `hiver` (87 names) and
+  `liir` (81) are odd, so their last name takes `}` as its value and the
+  block stays open: `tarkas` and `liir` nest inside `hiver`, `morrigi` inside
+  `liir`; two blocks are left open at EOF (2 warnings). The original's
+  system-name loader interprets that stream itself; a dedicated consumer is
+  needed for name-generation parity (open question in `docs/mars-parse.md`).
+- Every `.weapon`, `.shipsection`, `.tech`, `.combat`, `.def`, `.script`,
+  scenario and other block-form `.txt` renders **byte-identically** to before;
+  `verify.py`'s artifacts (`tech_tree.json`, `weapons.json`,
+  `shipsections.json`, `strings.json`, `schema_stats.json`, `crosslink.json`,
+  `tech_tree.dot`) are unchanged.
+- No brace file drops a pair (all end in `}` or a newline); 27 end with `}`
+  as the last byte (plain close); no unterminated quotes, no quoted-`//`
+  tokens, no `'`/backtick-quoted tokens, longest token 95 bytes
+  (`skydefs.txt`).
+- `.effect` files (415): untouched reader, unchanged.
+
+## Oracle results
+
+| suite | before | after |
+|---|---|---|
+| `verify.py` | 1,595 parsed / 0 failed; 12 tolerances (12 shipsections) | 1,595 / 0; 15 tolerances (12 shipsections + `systemnames.txt` + 2 kv dropped pairs); artifacts identical |
+| engine `tests/mars_parse/build_and_run.sh` (vs. updated oracle) | 1,531 / 1,531 | **1,531 / 1,531**, 51 unit tests, warning counts equal (13 files) |
+| engine `tests/mars_text/build_and_run.sh` (vs. updated oracle) | 64 / 64 | **64 / 64**, 201 unit checks, 205 real-data checks |
+
+## Files
+
+- Python (this dir): `mars_data.py` (new `Script` tokenizer + step-loop
+  parser; `_items` gone), `flat_kv.py` (`parse_kv` over `Script`, first-wins,
+  `warnings=`), `verify.py` (kv warnings reported; counts unchanged).
+- Engine worktree `~/sots-engine-wt/engine-parity` (branch `wip/engine-parity`):
+  `src/mars/parse/script.h` (new, header-only, shared), `blocks.{h,cpp}`,
+  `src/mars/text/flat_kv.{h,cpp}`, `result.h` (new Problem kinds),
+  `tests/mars_parse/{test_blocks,canon}.cpp`,
+  `tests/mars_text/{unit_tests,realdata_test,dump_json}.cpp`,
+  `docs/mars-parse.md`, `docs/mars-text.md`. No CMake or build-script edits;
+  no game data or decompiler text.

verify/parsers/flat_kv.py

@@ -4,11 +4,22 @@ whitespace-positional tables under Data/, Weapons/, Badges/, Avatars/, GUI/.
 Two shapes exist:
 
   parse_kv(text)    KEY value            -> {KEY: value}
-                    one constant per line; value is a bareword or a
-                    "quoted string"; '//' comments; colors are quoted
-                    "r g b" (use color()).  Files: Data/globals.txt,
-                    Data/species.txt, Data/Strategy/StrategyVars.txt,
-                    Data/Combat/*.txt (most), Data/encounters.txt, ...
+                    the engine's GlobalConsts loader (loader-prototypes.md
+                    M1): the file is stepped through with the Mars::Script
+                    tokenizer (mars_data.Script), NOT read by lines.
+                      KEY value   one token each: `COLOR "48 29 2"` must be
+                                  quoted; `LIST 1 2 3` is LIST=1 and 2=3
+                      NAME {      a block, skipped to its matching `}`
+                      }           ignored
+                    Keys match case-insensitively (_stricmp) and the FIRST
+                    occurrence wins: a key is consumed on first sight, later
+                    duplicates are "multiply defined" and ignored. A final
+                    pair whose value touches EOF (no trailing newline) is
+                    DROPPED -- StrategyVars.txt and encounters.txt each lose
+                    their last key this way in the shipped data.
+                    Files: Data/globals.txt, Data/species.txt,
+                    Data/Strategy/StrategyVars.txt, Data/Combat/*.txt,
+                    Data/encounters.txt, ...
 
   parse_rows(text)  tok tok tok ...      -> [[tok, ...], ...]
                     one record per line, whitespace separated, quoted
@@ -17,10 +28,7 @@ Two shapes exist:
                     Data/Combat/damfx*.txt, Data/Strategy/playercolors.txt,
                     Badges/BadgeTable.txt, Avatars/AvatarTable.txt,
                     GUI/WeaponIconPlacements.txt
-
-Quirks handled: '//' inside a quoted value is not a comment; a quoted
-value may be empty (""); keys repeat in a few files (kept as list);
-duplicate-key detection is exposed via parse_kv(..., on_dup=).
+                    (read by other engine code; line-based, unchanged)
 
 Stdlib only.
 """
@@ -29,7 +37,7 @@ from __future__ import annotations
 import re
 from typing import Any
 
-from mars_data import coerce
+from mars_data import AT_END, NO_INPUT, OK, Script, ScriptToken, coerce
 
 __all__ = ["parse_kv", "parse_rows", "duplicates", "color", "strip_comment",
            "split_tokens", "parse_kv_file", "parse_rows_file"]
@@ -37,6 +45,8 @@ __all__ = ["parse_kv", "parse_rows", "duplicates", "color", "strip_comment",
 _TOK_RE = re.compile(r'"([^"]*)"|(\S+)')
 
 
+# --- row tables (line-based) ------------------------------------------------
+
 def strip_comment(line: str) -> str:
     """Remove a trailing // comment, ignoring // inside double quotes."""
     in_q = False
@@ -74,38 +84,64 @@ def parse_rows(text: str, *, typed: bool = True) -> list[list[Any]]:
     return rows
 
 
-def _pairs(text: str, typed: bool):
-    for lineno, raw in enumerate(text.splitlines(), 1):
-        line = strip_comment(raw).strip()
-        if not line:
-            continue
-        toks = split_tokens(line)
-        key = toks[0][0]
-        vals = [coerce(t) if (typed and not q) else t for t, q in toks[1:]]
-        val: Any = None if not vals else (vals[0] if len(vals) == 1 else vals)
-        yield lineno, key, val
+# --- KEY value tables (the GlobalConsts loader loop) -------------------------
+
+def _steps(text: str, warnings: list):
+    """Yield (line, key, value_text, value_quoted) for every complete pair
+    the loader would consume; record every tolerance in `warnings`."""
+    sc = Script(text)
+    while True:
+        rc, t = sc.next()
+        if t.key_unterminated:
+            warnings.append(f"line {t.key_line}: unterminated quote runs to end of input")
+        if t.value_unterminated:
+            warnings.append(f"line {t.value_line}: unterminated quote runs to end of input")
+        if rc != OK:
+            if t.key_status != NO_INPUT and t.key != "}":
+                if t.value_status == NO_INPUT or t.key_status == AT_END:
+                    warnings.append(f"line {t.key_line}: trailing key '{t.key}' without a value: dropped")
+                else:
+                    warnings.append(f"line {t.key_line}: final pair '{t.key}' '{t.value}' touches end of "
+                                    "input (no trailing newline): dropped")
+            return
+        if t.type == ScriptToken.PAIR:
+            yield t.key_line, t.key, t.value, t.value_quoted
+        elif t.type == ScriptToken.OPEN:
+            warnings.append(f"line {t.key_line}: block '{t.key}' skipped")
+            if sc.skip_block(1) != OK:
+                return
+        # CLOSE at top level: ignored
 
 
-def parse_kv(text: str, *, typed: bool = True, on_dup: str = "last") -> dict:
-    """KEY value per line -> dict.  A value made of several unquoted tokens
-    is kept as a list.  on_dup: 'last' (later line wins), 'first', 'error'.
-    Use duplicates() to find repeated keys."""
+def parse_kv(text: str, *, typed: bool = True, on_dup: str = "first",
+             warnings: list | None = None) -> dict:
+    """KEY value per line -> dict keyed by the first-seen spelling.
+    on_dup: 'first' (engine: first occurrence wins), 'last', 'error'.
+    Pass warnings=[] to collect dropped pairs, duplicates and skipped blocks."""
+    w = warnings if warnings is not None else []
     d: dict = {}
-    for lineno, key, val in _pairs(text, typed):
-        if key in d:
+    spelling: dict[str, str] = {}   # folded key -> first spelling
+    for lineno, key, txt, quoted in _steps(text, w):
+        val = coerce(txt) if (typed and not quoted) else txt
+        lk = key.lower()
+        if lk in spelling:
             if on_dup == "error":
                 raise ValueError(f"line {lineno}: duplicate key {key}")
+            w.append(f"line {lineno}: '{key}' multiply defined (first occurrence kept)")
             if on_dup == "first":
                 continue
+            d[spelling[lk]] = val
+            continue
+        spelling[lk] = key
         d[key] = val
     return d
 
 
 def duplicates(text: str) -> dict[str, list[int]]:
-    """key -> line numbers, for keys that appear more than once."""
+    """folded key -> line numbers, for keys that appear more than once."""
     seen: dict[str, list[int]] = {}
-    for lineno, key, _ in _pairs(text, False):
-        seen.setdefault(key, []).append(lineno)
+    for lineno, key, _, _ in _steps(text, []):
+        seen.setdefault(key.lower(), []).append(lineno)
     return {k: v for k, v in seen.items() if len(v) > 1}
 
 

verify/parsers/mars_data.py

@@ -1,94 +1,182 @@
-"""mars_data.py -- reader for the Mars engine's brace-block key/value format.
+"""mars_data.py -- reader for the Mars engine's brace-block key/value format,
+built on a tokenizer with the engine's own `Mars::Script` semantics.
 
 Covers: *.weapon, *.shipsection, *.tech, *.combat, *.def, *.script and the
 block-form *.txt files (scenarios, tutorial, credits, systemnames, skydefs,
-ctechvars, shipai).
+ctechvars, shipai). flat_kv.py reuses `Script` for the Data/**/*.txt tables.
 
-Grammar (as observed in every shipped SOTS1 file):
+The engine (findings/subsystems/loader-prototypes.md, M3) has no tree: every
+loader pulls `Script::Next()` steps and interprets them itself.
 
-    body    := (block | pair | item)*
-    block   := NAME '{' body '}'
-    pair    := NAME value
-    item    := QUOTED                 # bare quoted string inside a block
-    value   := QUOTED | BAREWORD
-    comment := '//' .* EOL
+    step := KEY value        (one token each; the key may itself be quoted)
+          | NAME {           (a block, until the matching `}`)
+          | }
 
-A file's top level is itself a body (scenario .txt files mix top-level pairs
-and player{} blocks; catalog files hold one or many named blocks).
+`parse()` runs that step loop and records it as nested dicts, so the result
+is the step stream and nothing more.
 
-Result shape: plain dicts.  A key seen once maps to its value; a key seen
-more than once maps to a list (use get_list() when you want a list always).
-Bare quoted items are collected under the key "_items".
+Tokenizer (exact engine rules):
+  * whitespace is ' ', '\\t', '\\r', '\\n' only
+  * a bareword runs until whitespace -- braces are NOT delimiters, `{`/`}`
+    count only as whole tokens (`name{` is one word)
+  * a token starting with ", ' or ` is quoted; it ends at the same character;
+    no escapes; the quotes are stripped; an unterminated quote runs to EOF
+  * a token whose text starts with // is a comment (rest of line skipped);
+    checked on the extracted text, so `3//x` is one word and a quoted token
+    whose content starts with // is a comment too
+  * token text is truncated to 1023 bytes
+  * read_token status: 0 ok, 1 nothing read, 2 token touches end of input;
+    next() returns the status of its VALUE read
 
-Quirks handled (all seen in the real data, see parsers-report.md):
-  * keys are case-insensitive to the engine ("Requires"/"requires",
-    "badge"/"Badge") -> keys are lower-cased unless keep_case=True
-  * a block may open on the same line as a preceding pair
-    ("turretsize small   mount {") and a block name may sit on the same line
-    as its brace ("weapon {")
-  * backslashes inside quoted strings are literal (Windows paths); there is
-    no escape syntax
-  * '//' inside a quoted string is not a comment
-  * CRLF and LF line endings, cp1252 bytes (decoded losslessly)
-  * numbers use C float syntax: ".5", "-.8", "7e+8"
+End-of-input consequences (each recorded in `warnings`; strict=True raises):
+  * input ends inside a block          -> block simply ends
+  * `}` at top level                   -> ignored
+  * final `KEY value` whose value touches EOF (no trailing newline) -> DROPPED
+  * final KEY without a value / final `NAME {` -> dropped
+  * unterminated quote                 -> runs to EOF
+  * `{` in key position                -> ordinary key (kept)
+  A `}` that closes a block and touches EOF is a plain close (identical
+  outcome to the engine, which stops there either way).
+
+Result shape: plain dicts. A key seen once maps to its value; a key seen more
+than once maps to a list (use get_list() when you want a list always).
+Keys are lower-cased unless keep_case=True (the engine matches them with
+_stricmp). Bareword values are typed unless typed=False.
 
 Stdlib only.
 """
 from __future__ import annotations
 
 import re
-from typing import Any, Iterator
+from typing import Any
 
-__all__ = ["parse", "parse_file", "coerce", "get_list", "MarsSyntaxError"]
+__all__ = ["Script", "ScriptToken", "MAX_TOKEN", "OK", "NO_INPUT", "AT_END",
+           "parse", "parse_file", "coerce", "get_list", "MarsSyntaxError"]
 
 
 class MarsSyntaxError(ValueError):
     pass
 
 
-# --- tokenizer -------------------------------------------------------------
+# --- tokenizer (Mars::Script) -----------------------------------------------
 
-_TOKEN_RE = re.compile(
-    r"""
-    (?P<ws>\s+)
-  | (?P<comment>//[^\n]*)
-  | (?P<open>\{)
-  | (?P<close>\})
-  | (?P<quoted>"[^"]*")
-  | (?P<bad_quote>")
-  | (?P<bare>[^\s{}"]+)
-    """,
-    re.VERBOSE,
-)
+_WS = " \t\r\n"
+_QUOTES = "\"'`"
+MAX_TOKEN = 1023
+OK, NO_INPUT, AT_END = 0, 1, 2
 
 
-def _tokenize(text: str) -> Iterator[tuple[str, str, int]]:
-    """Yield (kind, value, line).  kind in {open, close, quoted, bare}."""
-    line = 1
-    pos = 0
-    n = len(text)
-    while pos < n:
-        m = _TOKEN_RE.match(text, pos)
-        if m is None:  # pragma: no cover - regex is exhaustive
-            raise MarsSyntaxError(f"line {line}: cannot tokenize {text[pos:pos+20]!r}")
-        kind = m.lastgroup
-        tok = m.group()
-        pos = m.end()
-        if kind == "ws":
-            line += tok.count("\n")
-            continue
-        if kind == "comment":
-            continue
-        if kind == "bad_quote":
-            raise MarsSyntaxError(f"line {line}: unterminated string")
-        if kind == "quoted":
-            yield kind, tok[1:-1], line
-            line += tok.count("\n")
-        else:
-            yield kind, tok, line
+class _Raw:
+    __slots__ = ("text", "quoted", "unterminated", "line")
+
+    def __init__(self, text: str, quoted: bool, unterminated: bool, line: int):
+        self.text = text
+        self.quoted = quoted
+        self.unterminated = unterminated
+        self.line = line
 
 
-# --- parser ----------------------------------------------------------------
+class ScriptToken:
+    NONE, PAIR, OPEN, CLOSE = 0, 1, 2, 3
+    __slots__ = ("type", "key", "value", "key_quoted", "value_quoted",
+                 "key_unterminated", "value_unterminated", "key_line", "value_line",
+                 "key_status", "value_status")
+
+    def __init__(self):
+        self.type = self.NONE
+        self.key = ""
+        self.value = ""
+        self.key_quoted = self.value_quoted = False
+        self.key_unterminated = self.value_unterminated = False
+        self.key_line = self.value_line = 0
+        self.key_status = self.value_status = NO_INPUT
+
+
+class Script:
+    """Pull tokenizer: read_token / next / skip_block, as in the engine."""
+
+    def __init__(self, text: str):
+        self.s = text
+        self.pos = 0
+        self.line = 1
+
+    def _advance(self, p: int) -> None:
+        self.line += self.s.count("\n", self.pos, p)
+        self.pos = p
+
+    def read_token(self):
+        """-> (status, _Raw | None)."""
+        s = self.s
+        n = len(s)
+        while True:
+            p = self.pos
+            while p < n and s[p] in _WS:
+                p += 1
+            self._advance(p)
+            if p >= n:
+                return NO_INPUT, None
+            line = self.line
+            c = s[p]
+            if c in _QUOTES:
+                close = s.find(c, p + 1)
+                if close < 0:
+                    text, unterminated = s[p + 1:], True
+                    self._advance(n)
+                else:
+                    text, unterminated = s[p + 1:close], False
+                    self._advance(close + 1)
+                quoted = True
+            else:
+                e = p
+                while e < n and s[e] not in _WS:
+                    e += 1
+                text, quoted, unterminated = s[p:e], False, False
+                self.pos = e
+            if text.startswith("//"):
+                p = self.pos
+                while p < n and s[p] not in "\r\n":
+                    p += 1
+                while p < n and s[p] in "\r\n":
+                    p += 1
+                self._advance(p)
+                continue
+            raw = _Raw(text[:MAX_TOKEN], quoted, unterminated, line)
+            return (AT_END if self.pos >= n else OK), raw
+
+    def next(self):
+        """-> (status, ScriptToken). Complete only when status == OK."""
+        t = ScriptToken()
+        rc, k = self.read_token()
+        t.key_status = rc
+        if k is not None:
+            t.key, t.key_quoted, t.key_unterminated, t.key_line = k.text, k.quoted, k.unterminated, k.line
+        if rc != OK:
+            return rc, t
+        if k.text == "}":
+            t.type = ScriptToken.CLOSE
+            return OK, t
+        rc, v = self.read_token()
+        t.value_status = rc
+        if v is not None:
+            t.value, t.value_quoted, t.value_unterminated, t.value_line = v.text, v.quoted, v.unterminated, v.line
+        t.type = ScriptToken.OPEN if t.value == "{" else ScriptToken.PAIR
+        return rc, t
+
+    def skip_block(self, depth: int) -> int:
+        while depth > 0:
+            rc, t = self.read_token()
+            if rc == NO_INPUT:
+                return rc
+            if t.text == "{":
+                depth += 1
+            elif t.text == "}":
+                depth -= 1
+            if rc != OK:
+                return OK if depth == 0 else rc
+        return OK
+
+
+# --- values ---------------------------------------------------------------
 
 _INT_RE = re.compile(r"[+-]?\d+$")
 _FLOAT_RE = re.compile(r"[+-]?(\d+\.\d*|\.\d+|\d+)([eE][+-]?\d+)?$")
@@ -127,69 +215,75 @@ def get_list(d: dict, key: str) -> list:
     return v if isinstance(v, list) else [v]
 
 
+# --- parser (the loaders' step loop) --------------------------------------
+
+_CLOSED, _EOF = 0, 1
+
+
 class _Parser:
     def __init__(self, text: str, typed: bool, keep_case: bool, strict: bool, warnings: list | None):
-        self.toks = list(_tokenize(text))
-        self.i = 0
+        self.script = Script(text)
         self.typed = typed
         self.keep_case = keep_case
         self.strict = strict
         self.warnings = warnings if warnings is not None else []
 
-    def _warn(self, msg: str) -> None:
+    def _warn(self, line: int, msg: str) -> None:
+        msg = f"line {line}: {msg}"
         if self.strict:
             raise MarsSyntaxError(msg)
         self.warnings.append(msg)
 
-    def _peek(self):
-        return self.toks[self.i] if self.i < len(self.toks) else None
-
-    def _next(self):
-        t = self.toks[self.i]
-        self.i += 1
-        return t
-
     def _key(self, name: str) -> str:
         return name if self.keep_case else name.lower()
 
-    def body(self, depth: int) -> dict:
+    def body(self, depth: int):
         d: dict = {}
+        sc = self.script
         while True:
-            t = self._peek()
-            if t is None:
-                if depth:
-                    # 11 shipped shipsections never close their outer block;
-                    # the engine treats EOF as closing every open block.
-                    self._warn(f"end of file inside block (depth {depth})")
-                return d
-            kind, val, line = t
-            if kind == "close":
-                self._next()
-                if not depth:
-                    # CrPropaganda.shipsection has one '}' too many.
-                    self._warn(f"line {line}: stray '}}' at top level")
+            rc, t = sc.next()
+            if t.key_unterminated:
+                self._warn(t.key_line, "unterminated quote runs to end of input")
+            if t.value_unterminated:
+                self._warn(t.value_line, "unterminated quote runs to end of input")
+            if rc != OK:
+                if t.key_status == AT_END and t.key == "}":
+                    if depth == 0:
+                        self._warn(t.key_line, "stray '}' at top level")
+                        return d, _EOF
+                    return d, _CLOSED
+                if t.key_status == AT_END:
+                    self._warn(t.key_line, f"trailing key '{t.key}' touches end of input: dropped")
+                elif t.key_status == OK:
+                    if t.value_status == NO_INPUT:
+                        self._warn(t.key_line, f"trailing key '{t.key}' has no value: dropped")
+                    elif t.type == ScriptToken.OPEN:
+                        self._warn(t.key_line, f"block header '{t.key}' {{ touches end of input: dropped")
+                    else:
+                        self._warn(t.key_line, f"final pair '{t.key}' '{t.value}' touches end of input "
+                                               "(no trailing newline): dropped")
+                return d, _EOF
+            if t.type == ScriptToken.CLOSE:
+                if depth == 0:
+                    self._warn(t.key_line, "stray '}' at top level")
                     continue
-                return d
-            if kind == "open":
-                raise MarsSyntaxError(f"line {line}: '{{' without a block name")
-            self._next()
-            if kind == "quoted":
-                # bare string item (systemnames.txt lists) -- never a key
-                _add(d, "_items", val)
+                return d, _CLOSED
+            if t.type == ScriptToken.OPEN:
+                if t.key == "{":
+                    self._warn(t.key_line, "'{' used as a block name")
+                sub, how = self.body(depth + 1)
+                _add(d, self._key(t.key), sub)
+                if how == _EOF:
+                    self._warn(sc.line, f"end of input inside block '{t.key}' (depth {depth + 1})")
+                    return d, _EOF
                 continue
-            nxt = self._peek()
-            if nxt is None or nxt[0] == "close":
-                # lone bareword at end of block: treat as flag item
-                _add(d, "_items", val)
-                continue
-            if nxt[0] == "open":
-                self._next()
-                _add(d, self._key(val), self.body(depth + 1))
-                continue
-            nkind, nval, _ = self._next()
-            if nkind == "bare" and self.typed:
-                nval = coerce(nval)
-            _add(d, self._key(val), nval)
+            # PAIR
+            if t.key == "{":
+                self._warn(t.key_line, "'{' used as a key")
+            val: Any = t.value
+            if not t.value_quoted and self.typed:
+                val = coerce(val)
+            _add(d, self._key(t.key), val)
 
 
 def parse(text: str, *, typed: bool = True, keep_case: bool = False,
@@ -198,11 +292,10 @@ def parse(text: str, *, typed: bool = True, keep_case: bool = False,
 
     typed      -- convert bareword numbers/bools (quoted strings stay str)
     keep_case  -- keep key case instead of lower-casing
-    strict     -- raise on unbalanced braces instead of recovering the way
-                  the engine does (EOF closes open blocks, stray top-level
-                  '}' ignored); pass warnings=[] to collect the recoveries
+    strict     -- raise on anything the engine merely tolerates (see module
+                  doc); pass warnings=[] to collect those cases instead
     """
-    return _Parser(text, typed, keep_case, strict, warnings).body(0)
+    return _Parser(text, typed, keep_case, strict, warnings).body(0)[0]
 
 
 def parse_file(path, **kw) -> dict:

verify/parsers/verify.py

@@ -139,10 +139,10 @@ def main(root: str, out: str) -> int:
                 obj = flat_kv.parse_rows_file(p)
             elif k == "kv":
                 txt = manifest.read_text(p)
-                obj = flat_kv.parse_kv(txt)
-                d = flat_kv.duplicates(txt)
-                if d:
-                    warns.append((rel, [f"duplicate keys {d}"]))
+                w = []
+                obj = flat_kv.parse_kv(txt, warnings=w)   # engine rules: first occurrence wins
+                if w:
+                    warns.append((rel, w))
             else:
                 continue
             parsed[rel] = obj
@@ -170,7 +170,7 @@ def main(root: str, out: str) -> int:
             P(f"- `{rel}`: {e}")
     if warns:
         P("")
-        P("Lenient recoveries (engine-compatible; strict=True would reject these):")
+        P("Engine tolerances taken (what the original loader does with these files; strict=True would reject):")
         for rel, w in warns:
             P(f"- `{rel}`: {'; '.join(w)}")
     P("")