alex/sots-re
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

parsers: engine parity (first-wins, Mars::Script tokenizer rules, trailing-pair drop); untrack pyc

Browse source
This commit is contained in:
alex 2026-09-07 18:10:17 -04:00
parent 50f608f997
commit 31153cd278
8 changed files with 365 additions and 148 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

88
verify/parsers/PARITY_NOTES.md Normal file
View file

@ -0,0 +1,88 @@
# Reader parity with the original engine — 2026-09-07
Both reader stacks — the C++ modules in `sots-engine` (`src/mars/parse`,
`src/mars/text`) and these Python oracles — now implement the loader
semantics Ghidra confirmed in `findings/subsystems/loader-prototypes.md`
(§M1 `GlobalConsts::LoadFile`, §M3 `Mars::Script`). This file lists each rule,
where the evidence is, and what it does to the shipped data. "Before" is the
previous tree-grammar / line-based reading; all runs are over the owner's
`gob-extract` (1,595 data files).
## Rules
| # | rule | evidence | before | after |
|---|---|---|---|---|
| 1 | Flat kv: keys compared with `_stricmp`, **first occurrence wins**; later duplicates are "multiply defined" and ignored | §M1 `LoadFile` pseudo-code (`consts->erase(node)` after the parser runs; the "not recognized or is multiply defined" log) | `parse_kv(on_dup='last')`, exact-key dict | `on_dup='first'` default, keys folded, first spelling kept; `warnings` records duplicates |
| 2 | Flat kv is read with the `Mars::Script` stepper, not by lines: `KEY value` = two tokens, `NAME {` skipped via `SkipBlock(1)`, `}` ignored | §M1 loop (`Script::Next`, `tok.type` 1/2/3), leniency bullets ("the value is one token", "`{}` blocks … follow the `Mars::Script` tokenizer rules (M3)") | line split, first token = key, rest = value list | `flat_kv._steps()` over `mars_data.Script` |
| 3 | Final `KEY value` whose value touches EOF is **dropped** (`Next` returns the status of the value read; loops stop on non-zero) | §M3 `Next()` pseudo-code and consequence bullet; §M1 comment "EOF (or last pair without trailing newline)" | kept | dropped, warning `final pair … touches end of input` |
| 4 | Whitespace = space, tab, CR, LF only | §M3 tokenizer rules, bullet 1 | Python `\s` (also `\v \f \x1c-\x1f \xa0`), C++ `isspace` set | exact four characters |
| 5 | Braces are **not** delimiters; `{`/`}` only as whole tokens (`strcmp` on the buffer) | §M3 bullet 3 ("barewords end only at whitespace… `weapon{` is one token"), `Next()` `strcmp` | `{ } "` cut a bareword | bareword runs to whitespace; quoted `"{"`/`"}"` act as braces |
| 6 | Quotes `"`, `'`, backtick; closes at the same character; no escapes; unterminated runs to EOF | §M3 bullet 2 | `"` only; unterminated = hard error | all three; unterminated tolerated (warning) |
| 7 | A token whose extracted text starts with `//` is a comment (bare or quoted); glued `3//x` is one word | §M3 comment bullet | `//` recognised only at a token boundary in the raw text (same result for bare; quoted-`//` was a string) | test on the extracted token |
| 8 | EOF ends parsing anywhere (unclosed blocks fine); stray top-level `}` ignored | §M3 consequence bullets 1–2 | same (already lenient) | same; warnings kept |
| 9 | A quoted string in key position is a key; its value is the next token | §M3 consequence bullet 4 (systemnames) | `_items` list | pair with `key_quoted` |
| 10 | Token text capped at 1023 bytes | §M3 bullet "output is truncated silently at outMax-1" | none | cap in both readers (longest shipped token is 95 bytes) |
Where the doc is silent we kept prior behaviour and say so: text glued after
a closing quote starts a new token (`"0 0 0"// x` in `globals.txt:230`
yields `0 0 0` then a comment — the only such spot in the data); `SkipBlock`
is assumed to count raw brace tokens; a `}` that closes a block and is the
last byte of the file is treated as a plain close (the original stops there
either way — identical outcome, no warning; 27 catalog files end that way);
the row tables (`_turrets.txt` etc.) and `.effect` files are read by other
engine classes and were not changed.
## Effect on the shipped data
### Flat kv (20 files, `Data/**/*.txt`)
- **Duplicates: none.** No shipped kv file repeats a key, exactly or
case-folded, so first-wins flips no value.
- **Dropped trailing pairs: 2 files, 1 key each** (file has no trailing newline):
- `Data/Strategy/StrategyVars.txt` line 118: `CIVILIAN_BURDEN_RATIO 0.5` — never read; the game runs on the compiled-in default.
- `Data/encounters.txt` line 225: `HERALD_SPEECH_MAX_INVERVAL 45` — same.
- Multi-token values, `{}` blocks, `'`/backtick quotes, unterminated quotes:
none in shipped kv files (single quotes appear only inside `//` comments).
- `globals.txt`: 364 keys, unchanged; line 230 (`"0 0 0"// …`) reads the same.
### Brace-block (1,116 files)
- Canonical output changed for **one file**: `Data/Strategy/systemnames.txt`.
Its lists are bare quoted names; under `Next()` they pair up
(`"Procyon" "Deneb"` → key `procyon` = `Deneb`). `hiver` (87 names) and
`liir` (81) are odd, so their last name takes `}` as its value and the
block stays open: `tarkas` and `liir` nest inside `hiver`, `morrigi` inside
`liir`; two blocks are left open at EOF (2 warnings). The original's
system-name loader interprets that stream itself; a dedicated consumer is
needed for name-generation parity (open question in `docs/mars-parse.md`).
- Every `.weapon`, `.shipsection`, `.tech`, `.combat`, `.def`, `.script`,
scenario and other block-form `.txt` renders **byte-identically** to before;
`verify.py`'s artifacts (`tech_tree.json`, `weapons.json`,
`shipsections.json`, `strings.json`, `schema_stats.json`, `crosslink.json`,
`tech_tree.dot`) are unchanged.
- No brace file drops a pair (all end in `}` or a newline); 27 end with `}`
as the last byte (plain close); no unterminated quotes, no quoted-`//`
tokens, no `'`/backtick-quoted tokens, longest token 95 bytes
(`skydefs.txt`).
- `.effect` files (415): untouched reader, unchanged.
## Oracle results
| suite | before | after |
|---|---|---|
| `verify.py` | 1,595 parsed / 0 failed; 12 tolerances (12 shipsections) | 1,595 / 0; 15 tolerances (12 shipsections + `systemnames.txt` + 2 kv dropped pairs); artifacts identical |
| engine `tests/mars_parse/build_and_run.sh` (vs. updated oracle) | 1,531 / 1,531 | **1,531 / 1,531**, 51 unit tests, warning counts equal (13 files) |
| engine `tests/mars_text/build_and_run.sh` (vs. updated oracle) | 64 / 64 | **64 / 64**, 201 unit checks, 205 real-data checks |
## Files
- Python (this dir): `mars_data.py` (new `Script` tokenizer + step-loop
parser; `_items` gone), `flat_kv.py` (`parse_kv` over `Script`, first-wins,
`warnings=`), `verify.py` (kv warnings reported; counts unchanged).
- Engine worktree `~/sots-engine-wt/engine-parity` (branch `wip/engine-parity`):
`src/mars/parse/script.h` (new, header-only, shared), `blocks.{h,cpp}`,
`src/mars/text/flat_kv.{h,cpp}`, `result.h` (new Problem kinds),
`tests/mars_parse/{test_blocks,canon}.cpp`,
`tests/mars_text/{unit_tests,realdata_test,dump_json}.cpp`,
`docs/mars-parse.md`, `docs/mars-text.md`. No CMake or build-script edits;
no game data or decompiler text.

BIN
verify/parsers/__pycache__/effect_txt.cpython-310.pyc
View file

Binary file not shown.

BIN
verify/parsers/__pycache__/flat_kv.cpython-310.pyc
View file

Binary file not shown.

BIN
verify/parsers/__pycache__/manifest.cpython-310.pyc
View file

Binary file not shown.

BIN
verify/parsers/__pycache__/mars_data.cpython-310.pyc
View file

Binary file not shown.

94
verify/parsers/flat_kv.py
View file

@ -4,11 +4,22 @@ whitespace-positional tables under Data/, Weapons/, Badges/, Avatars/, GUI/.
Two shapes exist: Two shapes exist:
parse_kv(text) KEY value -> {KEY: value} parse_kv(text) KEY value -> {KEY: value}
one constant per line; value is a bareword or a the engine's GlobalConsts loader (loader-prototypes.md
"quoted string"; '//' comments; colors are quoted M1): the file is stepped through with the Mars::Script
"r g b" (use color()). Files: Data/globals.txt, tokenizer (mars_data.Script), NOT read by lines.
Data/species.txt, Data/Strategy/StrategyVars.txt, KEY value one token each: `COLOR "48 29 2"` must be
Data/Combat/*.txt (most), Data/encounters.txt, ... quoted; `LIST 1 2 3` is LIST=1 and 2=3
NAME { a block, skipped to its matching `}`
} ignored
Keys match case-insensitively (_stricmp) and the FIRST
occurrence wins: a key is consumed on first sight, later
duplicates are "multiply defined" and ignored. A final
pair whose value touches EOF (no trailing newline) is
DROPPED -- StrategyVars.txt and encounters.txt each lose
their last key this way in the shipped data.
Files: Data/globals.txt, Data/species.txt,
Data/Strategy/StrategyVars.txt, Data/Combat/*.txt,
Data/encounters.txt, ...
parse_rows(text) tok tok tok ... -> [[tok, ...], ...] parse_rows(text) tok tok tok ... -> [[tok, ...], ...]
one record per line, whitespace separated, quoted one record per line, whitespace separated, quoted
@ -17,10 +28,7 @@ Two shapes exist:
Data/Combat/damfx*.txt, Data/Strategy/playercolors.txt, Data/Combat/damfx*.txt, Data/Strategy/playercolors.txt,
Badges/BadgeTable.txt, Avatars/AvatarTable.txt, Badges/BadgeTable.txt, Avatars/AvatarTable.txt,
GUI/WeaponIconPlacements.txt GUI/WeaponIconPlacements.txt
(read by other engine code; line-based, unchanged)
Quirks handled: '//' inside a quoted value is not a comment; a quoted
value may be empty (""); keys repeat in a few files (kept as list);
duplicate-key detection is exposed via parse_kv(..., on_dup=).
Stdlib only. Stdlib only.
""" """
@ -29,7 +37,7 @@ from __future__ import annotations
import re import re
from typing import Any from typing import Any
from mars_data import coerce from mars_data import AT_END, NO_INPUT, OK, Script, ScriptToken, coerce
__all__ = ["parse_kv", "parse_rows", "duplicates", "color", "strip_comment", __all__ = ["parse_kv", "parse_rows", "duplicates", "color", "strip_comment",
"split_tokens", "parse_kv_file", "parse_rows_file"] "split_tokens", "parse_kv_file", "parse_rows_file"]
@ -37,6 +45,8 @@ __all__ = ["parse_kv", "parse_rows", "duplicates", "color", "strip_comment",
_TOK_RE = re.compile(r'"([^"]*)"|(\S+)') _TOK_RE = re.compile(r'"([^"]*)"|(\S+)')
# --- row tables (line-based) ------------------------------------------------
def strip_comment(line: str) -> str: def strip_comment(line: str) -> str:
"""Remove a trailing // comment, ignoring // inside double quotes.""" """Remove a trailing // comment, ignoring // inside double quotes."""
in_q = False in_q = False
@ -74,38 +84,64 @@ def parse_rows(text: str, *, typed: bool = True) -> list[list[Any]]:
return rows return rows
def _pairs(text: str, typed: bool): # --- KEY value tables (the GlobalConsts loader loop) -------------------------
for lineno, raw in enumerate(text.splitlines(), 1):
line = strip_comment(raw).strip() def _steps(text: str, warnings: list):
if not line: """Yield (line, key, value_text, value_quoted) for every complete pair
continue the loader would consume; record every tolerance in `warnings`."""
toks = split_tokens(line) sc = Script(text)
key = toks[0][0] while True:
vals = [coerce(t) if (typed and not q) else t for t, q in toks[1:]] rc, t = sc.next()
val: Any = None if not vals else (vals[0] if len(vals) == 1 else vals) if t.key_unterminated:
yield lineno, key, val warnings.append(f"line {t.key_line}: unterminated quote runs to end of input")
if t.value_unterminated:
warnings.append(f"line {t.value_line}: unterminated quote runs to end of input")
if rc != OK:
if t.key_status != NO_INPUT and t.key != "}":
if t.value_status == NO_INPUT or t.key_status == AT_END:
warnings.append(f"line {t.key_line}: trailing key '{t.key}' without a value: dropped")
else:
warnings.append(f"line {t.key_line}: final pair '{t.key}' '{t.value}' touches end of "
"input (no trailing newline): dropped")
return
if t.type == ScriptToken.PAIR:
yield t.key_line, t.key, t.value, t.value_quoted
elif t.type == ScriptToken.OPEN:
warnings.append(f"line {t.key_line}: block '{t.key}' skipped")
if sc.skip_block(1) != OK:
return
# CLOSE at top level: ignored
def parse_kv(text: str, *, typed: bool = True, on_dup: str = "last") -> dict: def parse_kv(text: str, *, typed: bool = True, on_dup: str = "first",
"""KEY value per line -> dict. A value made of several unquoted tokens warnings: list | None = None) -> dict:
is kept as a list. on_dup: 'last' (later line wins), 'first', 'error'. """KEY value per line -> dict keyed by the first-seen spelling.
Use duplicates() to find repeated keys.""" on_dup: 'first' (engine: first occurrence wins), 'last', 'error'.
Pass warnings=[] to collect dropped pairs, duplicates and skipped blocks."""
w = warnings if warnings is not None else []
d: dict = {} d: dict = {}
for lineno, key, val in _pairs(text, typed): spelling: dict[str, str] = {} # folded key -> first spelling
if key in d: for lineno, key, txt, quoted in _steps(text, w):
val = coerce(txt) if (typed and not quoted) else txt
lk = key.lower()
if lk in spelling:
if on_dup == "error": if on_dup == "error":
raise ValueError(f"line {lineno}: duplicate key {key}") raise ValueError(f"line {lineno}: duplicate key {key}")
w.append(f"line {lineno}: '{key}' multiply defined (first occurrence kept)")
if on_dup == "first": if on_dup == "first":
continue continue
d[spelling[lk]] = val
continue
spelling[lk] = key
d[key] = val d[key] = val
return d return d
def duplicates(text: str) -> dict[str, list[int]]: def duplicates(text: str) -> dict[str, list[int]]:
"""key -> line numbers, for keys that appear more than once.""" """folded key -> line numbers, for keys that appear more than once."""
seen: dict[str, list[int]] = {} seen: dict[str, list[int]] = {}
for lineno, key, _ in _pairs(text, False): for lineno, key, _, _ in _steps(text, []):
seen.setdefault(key, []).append(lineno) seen.setdefault(key.lower(), []).append(lineno)
return {k: v for k, v in seen.items() if len(v) > 1} return {k: v for k, v in seen.items() if len(v) > 1}

321
verify/parsers/mars_data.py
View file

@ -1,94 +1,182 @@
"""mars_data.py -- reader for the Mars engine's brace-block key/value format. """mars_data.py -- reader for the Mars engine's brace-block key/value format,
built on a tokenizer with the engine's own `Mars::Script` semantics.
Covers: *.weapon, *.shipsection, *.tech, *.combat, *.def, *.script and the Covers: *.weapon, *.shipsection, *.tech, *.combat, *.def, *.script and the
block-form *.txt files (scenarios, tutorial, credits, systemnames, skydefs, block-form *.txt files (scenarios, tutorial, credits, systemnames, skydefs,
ctechvars, shipai). ctechvars, shipai). flat_kv.py reuses `Script` for the Data/**/*.txt tables.
Grammar (as observed in every shipped SOTS1 file): The engine (findings/subsystems/loader-prototypes.md, M3) has no tree: every
loader pulls `Script::Next()` steps and interprets them itself.
body := (block | pair | item)* step := KEY value (one token each; the key may itself be quoted)
block := NAME '{' body '}' | NAME { (a block, until the matching `}`)
pair := NAME value | }
item := QUOTED # bare quoted string inside a block
value := QUOTED | BAREWORD
comment := '//' .* EOL
A file's top level is itself a body (scenario .txt files mix top-level pairs `parse()` runs that step loop and records it as nested dicts, so the result
and player{} blocks; catalog files hold one or many named blocks). is the step stream and nothing more.
Result shape: plain dicts. A key seen once maps to its value; a key seen Tokenizer (exact engine rules):
more than once maps to a list (use get_list() when you want a list always). * whitespace is ' ', '\\t', '\\r', '\\n' only
Bare quoted items are collected under the key "_items". * a bareword runs until whitespace -- braces are NOT delimiters, `{`/`}`
count only as whole tokens (`name{` is one word)
* a token starting with ", ' or ` is quoted; it ends at the same character;
no escapes; the quotes are stripped; an unterminated quote runs to EOF
* a token whose text starts with // is a comment (rest of line skipped);
checked on the extracted text, so `3//x` is one word and a quoted token
whose content starts with // is a comment too
* token text is truncated to 1023 bytes
* read_token status: 0 ok, 1 nothing read, 2 token touches end of input;
next() returns the status of its VALUE read
Quirks handled (all seen in the real data, see parsers-report.md): End-of-input consequences (each recorded in `warnings`; strict=True raises):
* keys are case-insensitive to the engine ("Requires"/"requires", * input ends inside a block -> block simply ends
"badge"/"Badge") -> keys are lower-cased unless keep_case=True * `}` at top level -> ignored
* a block may open on the same line as a preceding pair * final `KEY value` whose value touches EOF (no trailing newline) -> DROPPED
("turretsize small mount {") and a block name may sit on the same line * final KEY without a value / final `NAME {` -> dropped
as its brace ("weapon {") * unterminated quote -> runs to EOF
* backslashes inside quoted strings are literal (Windows paths); there is * `{` in key position -> ordinary key (kept)
no escape syntax A `}` that closes a block and touches EOF is a plain close (identical
* '//' inside a quoted string is not a comment outcome to the engine, which stops there either way).
* CRLF and LF line endings, cp1252 bytes (decoded losslessly)
* numbers use C float syntax: ".5", "-.8", "7e+8" Result shape: plain dicts. A key seen once maps to its value; a key seen more
than once maps to a list (use get_list() when you want a list always).
Keys are lower-cased unless keep_case=True (the engine matches them with
_stricmp). Bareword values are typed unless typed=False.
Stdlib only. Stdlib only.
""" """
from __future__ import annotations from __future__ import annotations
import re import re
from typing import Any, Iterator from typing import Any
__all__ = ["parse", "parse_file", "coerce", "get_list", "MarsSyntaxError"] __all__ = ["Script", "ScriptToken", "MAX_TOKEN", "OK", "NO_INPUT", "AT_END",
"parse", "parse_file", "coerce", "get_list", "MarsSyntaxError"]
class MarsSyntaxError(ValueError): class MarsSyntaxError(ValueError):
pass pass
# --- tokenizer ------------------------------------------------------------- # --- tokenizer (Mars::Script) -----------------------------------------------
_TOKEN_RE = re.compile( _WS = " \t\r\n"
r""" _QUOTES = "\"'`"
(?P<ws>\s+) MAX_TOKEN = 1023
| (?P<comment>//[^\n]*) OK, NO_INPUT, AT_END = 0, 1, 2
| (?P<open>\{)
| (?P<close>\})
| (?P<quoted>"[^"]*")
| (?P<bad_quote>")
| (?P<bare>[^\s{}"]+)
""",
re.VERBOSE,
)
def _tokenize(text: str) -> Iterator[tuple[str, str, int]]: class _Raw:
"""Yield (kind, value, line). kind in {open, close, quoted, bare}.""" __slots__ = ("text", "quoted", "unterminated", "line")
line = 1
pos = 0 def __init__(self, text: str, quoted: bool, unterminated: bool, line: int):
n = len(text) self.text = text
while pos < n: self.quoted = quoted
m = _TOKEN_RE.match(text, pos) self.unterminated = unterminated
if m is None: # pragma: no cover - regex is exhaustive self.line = line
raise MarsSyntaxError(f"line {line}: cannot tokenize {text[pos:pos+20]!r}")
kind = m.lastgroup
tok = m.group()
pos = m.end()
if kind == "ws":
line += tok.count("\n")
continue
if kind == "comment":
continue
if kind == "bad_quote":
raise MarsSyntaxError(f"line {line}: unterminated string")
if kind == "quoted":
yield kind, tok[1:-1], line
line += tok.count("\n")
else:
yield kind, tok, line
# --- parser ---------------------------------------------------------------- class ScriptToken:
NONE, PAIR, OPEN, CLOSE = 0, 1, 2, 3
__slots__ = ("type", "key", "value", "key_quoted", "value_quoted",
"key_unterminated", "value_unterminated", "key_line", "value_line",
"key_status", "value_status")
def __init__(self):
self.type = self.NONE
self.key = ""
self.value = ""
self.key_quoted = self.value_quoted = False
self.key_unterminated = self.value_unterminated = False
self.key_line = self.value_line = 0
self.key_status = self.value_status = NO_INPUT
class Script:
"""Pull tokenizer: read_token / next / skip_block, as in the engine."""
def __init__(self, text: str):
self.s = text
self.pos = 0
self.line = 1
def _advance(self, p: int) -> None:
self.line += self.s.count("\n", self.pos, p)
self.pos = p
def read_token(self):
"""-> (status, _Raw | None)."""
s = self.s
n = len(s)
while True:
p = self.pos
while p < n and s[p] in _WS:
p += 1
self._advance(p)
if p >= n:
return NO_INPUT, None
line = self.line
c = s[p]
if c in _QUOTES:
close = s.find(c, p + 1)
if close < 0:
text, unterminated = s[p + 1:], True
self._advance(n)
else:
text, unterminated = s[p + 1:close], False
self._advance(close + 1)
quoted = True
else:
e = p
while e < n and s[e] not in _WS:
e += 1
text, quoted, unterminated = s[p:e], False, False
self.pos = e
if text.startswith("//"):
p = self.pos
while p < n and s[p] not in "\r\n":
p += 1
while p < n and s[p] in "\r\n":
p += 1
self._advance(p)
continue
raw = _Raw(text[:MAX_TOKEN], quoted, unterminated, line)
return (AT_END if self.pos >= n else OK), raw
def next(self):
"""-> (status, ScriptToken). Complete only when status == OK."""
t = ScriptToken()
rc, k = self.read_token()
t.key_status = rc
if k is not None:
t.key, t.key_quoted, t.key_unterminated, t.key_line = k.text, k.quoted, k.unterminated, k.line
if rc != OK:
return rc, t
if k.text == "}":
t.type = ScriptToken.CLOSE
return OK, t
rc, v = self.read_token()
t.value_status = rc
if v is not None:
t.value, t.value_quoted, t.value_unterminated, t.value_line = v.text, v.quoted, v.unterminated, v.line
t.type = ScriptToken.OPEN if t.value == "{" else ScriptToken.PAIR
return rc, t
def skip_block(self, depth: int) -> int:
while depth > 0:
rc, t = self.read_token()
if rc == NO_INPUT:
return rc
if t.text == "{":
depth += 1
elif t.text == "}":
depth -= 1
if rc != OK:
return OK if depth == 0 else rc
return OK
# --- values ---------------------------------------------------------------
_INT_RE = re.compile(r"[+-]?\d+$") _INT_RE = re.compile(r"[+-]?\d+$")
_FLOAT_RE = re.compile(r"[+-]?(\d+\.\d*|\.\d+|\d+)([eE][+-]?\d+)?$") _FLOAT_RE = re.compile(r"[+-]?(\d+\.\d*|\.\d+|\d+)([eE][+-]?\d+)?$")
@ -127,69 +215,75 @@ def get_list(d: dict, key: str) -> list:
return v if isinstance(v, list) else [v] return v if isinstance(v, list) else [v]
# --- parser (the loaders' step loop) --------------------------------------
_CLOSED, _EOF = 0, 1
class _Parser: class _Parser:
def __init__(self, text: str, typed: bool, keep_case: bool, strict: bool, warnings: list | None): def __init__(self, text: str, typed: bool, keep_case: bool, strict: bool, warnings: list | None):
self.toks = list(_tokenize(text)) self.script = Script(text)
self.i = 0
self.typed = typed self.typed = typed
self.keep_case = keep_case self.keep_case = keep_case
self.strict = strict self.strict = strict
self.warnings = warnings if warnings is not None else [] self.warnings = warnings if warnings is not None else []
def _warn(self, msg: str) -> None: def _warn(self, line: int, msg: str) -> None:
msg = f"line {line}: {msg}"
if self.strict: if self.strict:
raise MarsSyntaxError(msg) raise MarsSyntaxError(msg)
self.warnings.append(msg) self.warnings.append(msg)
def _peek(self):
return self.toks[self.i] if self.i < len(self.toks) else None
def _next(self):
t = self.toks[self.i]
self.i += 1
return t
def _key(self, name: str) -> str: def _key(self, name: str) -> str:
return name if self.keep_case else name.lower() return name if self.keep_case else name.lower()
def body(self, depth: int) -> dict: def body(self, depth: int):
d: dict = {} d: dict = {}
sc = self.script
while True: while True:
t = self._peek() rc, t = sc.next()
if t is None: if t.key_unterminated:
if depth: self._warn(t.key_line, "unterminated quote runs to end of input")
# 11 shipped shipsections never close their outer block; if t.value_unterminated:
# the engine treats EOF as closing every open block. self._warn(t.value_line, "unterminated quote runs to end of input")
self._warn(f"end of file inside block (depth {depth})") if rc != OK:
return d if t.key_status == AT_END and t.key == "}":
kind, val, line = t if depth == 0:
if kind == "close": self._warn(t.key_line, "stray '}' at top level")
self._next() return d, _EOF
if not depth: return d, _CLOSED
# CrPropaganda.shipsection has one '}' too many. if t.key_status == AT_END:
self._warn(f"line {line}: stray '}}' at top level") self._warn(t.key_line, f"trailing key '{t.key}' touches end of input: dropped")
elif t.key_status == OK:
if t.value_status == NO_INPUT:
self._warn(t.key_line, f"trailing key '{t.key}' has no value: dropped")
elif t.type == ScriptToken.OPEN:
self._warn(t.key_line, f"block header '{t.key}' {{ touches end of input: dropped")
else:
self._warn(t.key_line, f"final pair '{t.key}' '{t.value}' touches end of input "
"(no trailing newline): dropped")
return d, _EOF
if t.type == ScriptToken.CLOSE:
if depth == 0:
self._warn(t.key_line, "stray '}' at top level")
continue continue
return d return d, _CLOSED
if kind == "open": if t.type == ScriptToken.OPEN:
raise MarsSyntaxError(f"line {line}: '{{' without a block name") if t.key == "{":
self._next() self._warn(t.key_line, "'{' used as a block name")
if kind == "quoted": sub, how = self.body(depth + 1)
# bare string item (systemnames.txt lists) -- never a key _add(d, self._key(t.key), sub)
_add(d, "_items", val) if how == _EOF:
self._warn(sc.line, f"end of input inside block '{t.key}' (depth {depth + 1})")
return d, _EOF
continue continue
nxt = self._peek() # PAIR
if nxt is None or nxt[0] == "close": if t.key == "{":
# lone bareword at end of block: treat as flag item self._warn(t.key_line, "'{' used as a key")
_add(d, "_items", val) val: Any = t.value
continue if not t.value_quoted and self.typed:
if nxt[0] == "open": val = coerce(val)
self._next() _add(d, self._key(t.key), val)
_add(d, self._key(val), self.body(depth + 1))
continue
nkind, nval, _ = self._next()
if nkind == "bare" and self.typed:
nval = coerce(nval)
_add(d, self._key(val), nval)
def parse(text: str, *, typed: bool = True, keep_case: bool = False, def parse(text: str, *, typed: bool = True, keep_case: bool = False,
@ -198,11 +292,10 @@ def parse(text: str, *, typed: bool = True, keep_case: bool = False,
typed -- convert bareword numbers/bools (quoted strings stay str) typed -- convert bareword numbers/bools (quoted strings stay str)
keep_case -- keep key case instead of lower-casing keep_case -- keep key case instead of lower-casing
strict -- raise on unbalanced braces instead of recovering the way strict -- raise on anything the engine merely tolerates (see module
the engine does (EOF closes open blocks, stray top-level doc); pass warnings=[] to collect those cases instead
'}' ignored); pass warnings=[] to collect the recoveries
""" """
return _Parser(text, typed, keep_case, strict, warnings).body(0) return _Parser(text, typed, keep_case, strict, warnings).body(0)[0]
def parse_file(path, **kw) -> dict: def parse_file(path, **kw) -> dict:

10
verify/parsers/verify.py
View file

@ -139,10 +139,10 @@ def main(root: str, out: str) -> int:
obj = flat_kv.parse_rows_file(p) obj = flat_kv.parse_rows_file(p)
elif k == "kv": elif k == "kv":
txt = manifest.read_text(p) txt = manifest.read_text(p)
obj = flat_kv.parse_kv(txt) w = []
d = flat_kv.duplicates(txt) obj = flat_kv.parse_kv(txt, warnings=w) # engine rules: first occurrence wins
if d: if w:
warns.append((rel, [f"duplicate keys {d}"])) warns.append((rel, w))
else: else:
continue continue
parsed[rel] = obj parsed[rel] = obj
@ -170,7 +170,7 @@ def main(root: str, out: str) -> int:
P(f"- `{rel}`: {e}") P(f"- `{rel}`: {e}")
if warns: if warns:
P("") P("")
P("Lenient recoveries (engine-compatible; strict=True would reject these):") P("Engine tolerances taken (what the original loader does with these files; strict=True would reject):")
for rel, w in warns: for rel, w in warns:
P(f"- `{rel}`: {'; '.join(w)}") P(f"- `{rel}`: {'; '.join(w)}")
P("") P("")