sots-re/verify/parsers/PARITY_NOTES.md

Reader parity with the original engine — 2026-09-07

Both reader stacks — the C++ modules in sots-engine (src/mars/parse, src/mars/text) and these Python oracles — now implement the loader semantics Ghidra confirmed in findings/subsystems/loader-prototypes.md (§M1 GlobalConsts::LoadFile, §M3 Mars::Script). This file lists each rule, where the evidence is, and what it does to the shipped data. "Before" is the previous tree-grammar / line-based reading; all runs are over the owner's gob-extract (1,595 data files).

Rules

#	rule	evidence	before	after
1	Flat kv: keys compared with _stricmp, first occurrence wins; later duplicates are "multiply defined" and ignored	§M1 LoadFile pseudo-code (consts->erase(node) after the parser runs; the "not recognized or is multiply defined" log)	parse_kv(on_dup='last'), exact-key dict	on_dup='first' default, keys folded, first spelling kept; warnings records duplicates
2	Flat kv is read with the Mars::Script stepper, not by lines: KEY value = two tokens, NAME { skipped via SkipBlock(1), } ignored	§M1 loop (Script::Next, tok.type 1/2/3), leniency bullets ("the value is one token", "{} blocks … follow the Mars::Script tokenizer rules (M3)")	line split, first token = key, rest = value list	flat_kv._steps() over mars_data.Script
3	Final KEY value whose value touches EOF is dropped (Next returns the status of the value read; loops stop on non-zero)	§M3 Next() pseudo-code and consequence bullet; §M1 comment "EOF (or last pair without trailing newline)"	kept	dropped, warning final pair … touches end of input
4	Whitespace = space, tab, CR, LF only	§M3 tokenizer rules, bullet 1	Python \s (also \v \f \x1c-\x1f \xa0), C++ isspace set	exact four characters
5	Braces are not delimiters; {/} only as whole tokens (strcmp on the buffer)	§M3 bullet 3 ("barewords end only at whitespace… weapon{ is one token"), Next() strcmp	{ } " cut a bareword	bareword runs to whitespace; quoted "{"/"}" act as braces
6	Quotes ", ', backtick; closes at the same character; no escapes; unterminated runs to EOF	§M3 bullet 2	" only; unterminated = hard error	all three; unterminated tolerated (warning)
7	A token whose extracted text starts with // is a comment (bare or quoted); glued 3//x is one word	§M3 comment bullet	// recognised only at a token boundary in the raw text (same result for bare; quoted-// was a string)	test on the extracted token
8	EOF ends parsing anywhere (unclosed blocks fine); stray top-level } ignored	§M3 consequence bullets 1–2	same (already lenient)	same; warnings kept
9	A quoted string in key position is a key; its value is the next token	§M3 consequence bullet 4 (systemnames)	_items list	pair with key_quoted
10	Token text capped at 1023 bytes	§M3 bullet "output is truncated silently at outMax-1"	none	cap in both readers (longest shipped token is 95 bytes)

Where the doc is silent we kept prior behaviour and say so: text glued after a closing quote starts a new token ("0 0 0"// x in globals.txt:230 yields 0 0 0 then a comment — the only such spot in the data); SkipBlock is assumed to count raw brace tokens; a } that closes a block and is the last byte of the file is treated as a plain close (the original stops there either way — identical outcome, no warning; 27 catalog files end that way); the row tables (_turrets.txt etc.) and .effect files are read by other engine classes and were not changed.

Effect on the shipped data

Flat kv (20 files, Data/**/*.txt)

• Duplicates: none. No shipped kv file repeats a key, exactly or case-folded, so first-wins flips no value.
• Dropped trailing pairs: 2 files, 1 key each (file has no trailing newline):
  • Data/Strategy/StrategyVars.txt line 118: CIVILIAN_BURDEN_RATIO 0.5 — never read; the game runs on the compiled-in default.
  • Data/encounters.txt line 225: HERALD_SPEECH_MAX_INVERVAL 45 — same.
• Multi-token values, {} blocks, '/backtick quotes, unterminated quotes: none in shipped kv files (single quotes appear only inside // comments).
• globals.txt: 364 keys, unchanged; line 230 ("0 0 0"// …) reads the same.

Brace-block (1,116 files)

• Canonical output changed for one file: Data/Strategy/systemnames.txt. Its lists are bare quoted names; under Next() they pair up ("Procyon" "Deneb" → key procyon = Deneb). hiver (87 names) and liir (81) are odd, so their last name takes } as its value and the block stays open: tarkas and liir nest inside hiver, morrigi inside liir; two blocks are left open at EOF (2 warnings). The original's system-name loader interprets that stream itself; a dedicated consumer is needed for name-generation parity (open question in docs/mars-parse.md).
• Every .weapon, .shipsection, .tech, .combat, .def, .script, scenario and other block-form .txt renders byte-identically to before; verify.py's artifacts (tech_tree.json, weapons.json, shipsections.json, strings.json, schema_stats.json, crosslink.json, tech_tree.dot) are unchanged.
• No brace file drops a pair (all end in } or a newline); 27 end with } as the last byte (plain close); no unterminated quotes, no quoted-// tokens, no '/backtick-quoted tokens, longest token 95 bytes (skydefs.txt).
• .effect files (415): untouched reader, unchanged.

Oracle results

suite	before	after
verify.py	1,595 parsed / 0 failed; 12 tolerances (12 shipsections)	1,595 / 0; 15 tolerances (12 shipsections + systemnames.txt + 2 kv dropped pairs); artifacts identical
engine tests/mars_parse/build_and_run.sh (vs. updated oracle)	1,531 / 1,531	1,531 / 1,531, 51 unit tests, warning counts equal (13 files)
engine tests/mars_text/build_and_run.sh (vs. updated oracle)	64 / 64	64 / 64, 201 unit checks, 205 real-data checks

Files

• Python (this dir): mars_data.py (new Script tokenizer + step-loop parser; _items gone), flat_kv.py (parse_kv over Script, first-wins, warnings=), verify.py (kv warnings reported; counts unchanged).
• Engine worktree ~/sots-engine-wt/engine-parity (branch wip/engine-parity): src/mars/parse/script.h (new, header-only, shared), blocks.{h,cpp}, src/mars/text/flat_kv.{h,cpp}, result.h (new Problem kinds), tests/mars_parse/{test_blocks,canon}.cpp, tests/mars_text/{unit_tests,realdata_test,dump_json}.cpp, docs/mars-parse.md, docs/mars-text.md. No CMake or build-script edits; no game data or decompiler text.