sots-engine/docs/game-data.md

game::data — the typed catalogs

src/game/data/ turns the parsed data files into the definition objects the game builds at startup: weapons, ship sections, the turret table, the stable id registries, the tech tree, the string table, and a Catalog that loads a whole data root, resolves names and cross-checks every reference. It sits on mars::parse (brace blocks) and mars::text (rows, manifests, CSV); library target game_data (static, src/game/data/CMakeLists.txt), include as game/data/<header>.h. C++17, no dependencies, no exceptions across the API.

Behaviour is specified by the RE findings (data-model.md §4, data-parsers.md, SHIP_DESIGN_RULES.md §1–3); the normalised JSON catalogs the reference Python tooling emits are the oracle, and this module is checked against them value by value (§5).

src/game/data/
  common.h/.cpp      Species (engine index order), Problem, Loaded<T>, case folding
  block.h/.cpp       Block/Attr: copyable snapshot of a parsed block, typed reads
  fields.h           internal: typed reads that record Problems
  weapon.h/.cpp      WeaponDef, RangeTable, BoltDef, FireControl, Ratings; load_weapon
  shipsection.h/.cpp ShipSectionDef, OptionGroup, BankDef, MountDef, NetForceLimits; load_shipsection
  turrets.h/.cpp     TurretTable (_turrets.txt), IdRegistry (_weapons/_shipsections.txt)
  techtree.h/.cpp    TechTree, TechNode, AllowsEdge, parse_allows
  strings.h/.cpp     StringTable (Strings.csv)
  catalog.h/.cpp     Catalog: load_catalog(root), lookups, cross_check()
tests/game_data/
  build_and_run.sh   plain g++ build; unit tests; real-data facts; oracle comparison
  test_*.cpp         hand-written samples for every loader and the option normalisation
  realdata_test.cpp  counts / spot facts / cross-check against the published numbers
  dump_catalog.cpp   whole catalog -> JSON (typed fields + raw blocks)
  oracle/compare.py  field-by-field comparison with the reference catalogs

1. Common conventions

• Loaders are total. load_x(const mars::parse::Document&) returns Loaded<T> { optional<T> value; vector<Problem> problems; }. Only a file with no usable top-level block (or a hard syntax error in the parse_x text convenience) yields no value; everything else loads and reports. Problem { kind, file, line, key, message } kinds: MissingBlock, MissingKey, BadValue, Unparsed, Duplicate, IoError, Syntax (the parser's lenient recoveries are reported as Syntax "recovered: …").
• Typed fields are optional. A key that is absent is nullopt; a key whose value does not have the expected shape is nullopt plus a BadValue problem, and the text stays reachable in raw. Ints need the int shape; doubles accept int or C-float; bools accept true/false (any case) or an int (hidden 1, autonomous 1, nodesign 1).
• Last occurrence wins when a scalar key repeats (mass 1800 mass 1000 on two engine sections, crew on four Zuul sections, turretsize in 19 banks). Repeatable keys (requires, exclude, compatible_section, option, bank, mount, thruster, allows, inc/dec, section, filename) are vectors in file order.
• Text kept as written, matched folded. turretclass PlanetMissile, section_type Engine, requires WEP_HCLAS are stored verbatim; every lookup, enum derivation and cross-link comparison is ASCII case-insensitive, which is how the data itself is consistent.
• raw. Every def keeps a Block copy of its whole block: ordered attrs (key, text, quoted, line), blocks, items, with find (last wins) / all / block / blocks_named / str / strs and typed Attr::as_int/as_double/as_bool. The long tail of section role flags and capacities (refinery, police, colonizer_pop, mining_rate, sounds, anim{}) and the non-bolt weapon behaviour blocks live there.
• Species are indexed in the engine's own order — Human 0, Hiver 1, Tarkas 2, Liir 3, _NPC 4, Zuul 5, Morrigi 6 (the save-file species table).

2. Schemas as implemented

WeaponDef (weapon{}; Weapons/*.weapon, Species/<Race>/weapons/*.weapon)

field	key	type	notes
stem, file, scope, id	—		catalog identity; scope NPC for anything under Species/; id from _weapons.txt (player only)
display_name	—	optional string	name resolved through the string table (@X → X); a non-@ name is its own display name
name	name	string	@WEAPON_* token, required
weapon_class, weapon_family, weapon_damage_type	weaponclass (required), weaponfamily, weapondamagetype	string
requires	requires*	vector	AND of techs; 30 weapons have none
compatible_section* / exclusive_species		vector / string	rider carriers; zuul on the two grapples
cost, burst_volleys, range, range_planet, muzzle_speed, hpbonus, dam_est		optional int
turret_size, turret_class	turretsize, turretclass (both required)	string
track_speed_mod, recharge_time, volley_period, volley_duration, buildup_delay, solution_tolerance		optional double
hidden, pinpoint, blindfire, secondary_pd		optional bool
model1..3, muzzle_effect, muzzle_sound, icon_file, icon_rect		string
fc	fc_requires_los … fc_targets_expire	11 optional bools	fire-control flags
ratings	rating_frate/dam/acc/range	optional double	UI stars; fractional on 18 weapons
behavior_kind	first of bolt beam torpedo rider missile chainlightning col mine disintegrator grapple projectedshield mirv nodecannon siege mesonprojector spyship wraith present	string	behavior() returns that block from raw
bolt	bolt{}	optional BoltDef	rangetable, planet (dam_pop/infra/terra), mass, beam_origin, beam_length, ricochet_mod, effect, impact_effect, expire_effect
rangetable	<behaviour>.rangetable{}	optional RangeTable	present for bolt (66) and torpedo (19): three bands point_blank/effective/maximum of {range, deviation, damage} from `pb_/eff_/max_range[_dev
planet_damage	<behaviour>.dam_pop/dam_infra/dam_terra	PlanetDamage	whichever behaviour block is present

ShipSectionDef (shipsection{}; Species/<Race>/sections/*.shipsection)

field	key	type	notes
race, species, stem, file, id	—		race is the directory name; id from the race's _shipsections.txt
display_name, description, unlocked_by	—		SECTIONNAME_<stem> / SECTIONDESC_<stem>; techs whose ship{section} names the stem (tech order)
model (required), dam_model		string
requires*		vector	may name GRP_<group>
section_type_text → section_type	section_type	string → SectionType {None, Command, Mission, Engine, Other}	41 sections have none (riders, NPC hulls)
section_class_text → section_class	section_class	string → SectionClass {None, Destroyer, Cruiser, Dreadnought, Other}
design_class, entity_class		string	station, rider, …
health, mass (both required)		optional double	NPC masses are written as floats
cost, cpoints, crew, command_cost, maintenance_cost, command_quota		optional int
socket_fore/aft, dam_socket_fore/aft		string	geometry only; has_sockets() is the hull-vs-standalone test (design rule A3)
options	option T and option{ option A … }*	vector of OptionGroup { members, scalar, line }	one normalised list in file order: a scalar entry is a one-member group with scalar = true, a block keeps its members. 229 scalar entries in 153 files, 1,226 blocks
optiondef	optiondef{ option … }	optional OptionGroup	the shield-level group on the 12 shield sections; kept separate because it is a separate key
banks	bank{}*	vector of BankDef	turret_class, turret_size (last if repeated; repeated_turret_spec flags the 19 banks that do), weapon (fixed weapon file on 404 NPC banks), show_turrets, invincible, mounts (MountDef { node, min/max_azimuth, min/max_inclination, home_azimuth/inclination })
ftlspeed, nodespeed, range, scanrange, tactical_sensor_range		optional double
engine_techera		string	fission / fusion / antimatter
netforcelimits	netforcelimits{}	optional NetForceLimits	force_forward/right/up, torque_yaw/pitch/roll, speed, rotspeed
thrusters	thruster{}*	vector	node, effect, idle_effect
exclude*	exclude "STEM"	vector	ramscoop command sections
explicit_command_section, explicit_engine_section, explicit_section		string, string, optional bool	station triples
autonomous, nodesign		optional bool	written as 1 or true

TurretTable (Weapons/_turrets.txt) and IdRegistry

TurretRow { mount_size, weapon_size, turret_class, health, track_speed, azimuth_scale, inclination_scale, model } from the 8-token positional rows (a row with any other width is Unparsed and skipped). find(mount, weapon, class), has_weapon_pair(weapon_size, class), has_bank_pair(mount_size, class), rows_for_bank(...) — all case-insensitive.

IdRegistry wraps mars::text::Manifest: IdEntry { id, name, stem, line }, deleted() tombstones, find(id) (last assignment wins), id_of(name or stem) case-insensitive (DEWar.SHIPSECTION, dewar, DEWar.shipsection all resolve), is_deleted(id). Manifest problems map to Unparsed / Duplicate.

TechTree (TechTree/MasterTechList.tech)

TechNode { name, family (as written, 155 nodes), family_inferred (upper name prefix), type, threat, group (as written), option_cost, unlock_explicitly, requires, benefits_inc, benefits_dec, sections, weapon_files, allows (edge indices), display_name, description, raw }.

AllowsEdge { from, to, rp, pct[7], pct_written[7], unparsed, text, line } — parse_allows splits the string on whitespace: token 0 is the child, RP:n (any case) the cost, <Species>:n a percentage for that species (any of the seven names, any case); anything else lands in unparsed and is reported. Unlisted species default to 100 — this is the engine default the RE notes describe (data-parsers.md, "believed 100"); pct_written records which values the file actually spelled out so a consumer can tell the two apart. _NPC is never written and is therefore always 100.

groups: upper-cased group value → member names (first-seen order); group_members("TORPS") and group_members("GRP_Torps") both work; requirement_exists(token) is true for a tech name or a non-empty group. strategy{ inc/dec } is kept as the TECHBEN_* token lists — no semantics are attached here. edges_to / edges_from / roots() are graph helpers.

StringTable (Locale/EN/Strings.csv)

Column 0 → column 1 of every data row of the commented CSV; keys folded, the later row wins on a repeat (the four trailing-space duplicates), resolve() strips a leading @.

Catalog

load_catalog(root) reads Weapons/*.weapon + _weapons.txt + _turrets.txt, every Species/<dir>/sections/ (+ _shipsections.txt) and Species/<dir>/weapons/, TechTree/MasterTechList.tech and, if present, Locale/EN/Strings.csv; missing mandatory files are IoError problems, the rest still loads. Weapons are ordered player-first then by folded stem, sections by (race, folded stem); ids, display names, descriptions and unlocked_by are filled in; lookups: weapon(stem) (player catalog wins on a stem clash), weapon_by_file, weapon_by_id, section(race, stem), section_by_id, sections_named(stem) (all races), tech_node.

cross_check() → CrossCheck with one vector per link kind. Dangling (counted by dangling_count(), clean() = 0): weapon/section requires → tech (a GRP_x needs a non-empty group), section option/optiondef members → tech, tech ship{section} → any race's section, tech weapon{filename} → a loaded weapon file, tech requires → tech/group, allows child → tech (and unparsed edges), bank{weapon} → weapon file, manifest id → file and file → manifest id, weapon and bank (size, class) → a turret row, @name → string. Informational: case-only mismatches, weapons without requires, missing TECHNAME_/TECHDESC_ and SECTIONNAME_/SECTIONDESC_ (per distinct stem), roots.

3. Normalisation decisions

decision	why
option scalar and block forms merge into one ordered list of groups	the engine treats a bare option T as a one-member choice (design rule D1); order is preserved through the entries' position in the block so the save-file DOpts order (group order) can be reproduced
optiondef stays a separate optional group	different key, different meaning (shield level), only 12 files
repeated scalar → last value, flagged on banks	the design-rule validator takes last; the engine's choice is unknown (open question), so the fact is kept (repeated_turret_spec, raw.all(key)) rather than hidden
numeric fields that do not parse → nullopt + BadValue, text kept in raw	34 values in the shipped data are typos (force_right o ×22, crew false ×9, trackspeed_mod 1.0f, min_inclination 0-5, max_inclination 90\); guessing what the engine's converter makes of them would be fiction, so they are reported instead
bools accept 0/1	hidden 1, autonomous 1, nodesign 1 coexist with true in the data
section_type/section_class enums beside the text	the data spells them in two cases; behaviour code wants the enum, the oracle comparison wants the text
species percentages stored for all seven species, default 100	matches the RE description of the engine default and gives _NPC (which has a catalog but never appears in allows) a defined value
family_inferred from the name prefix	half the nodes lack family, and where present it is sometimes the display branch (WEP_Dsrptr has family "TRP")
NPC weapons are scope NPC with no id	they have no manifest; NPC banks reference them by file path (bank{weapon})

4. Tests

tests/game_data/build_and_run.sh (plain g++ -std=c++17 -Wall -Wextra -Wpedantic -Werror):

1. Unit tests — 27 cases on hand-written samples: every typed field of a weapon, bolt + rangetable, torpedo/beam behaviour detection, missing keys and bad values as problems, fatal no-block / syntax error, lenient recovery, last-wins and list order; section fields, the option normalisation across both forms, optiondef, banks with repeated turretsize, mounts, netforcelimits, thrusters, exclude, enum parsing, long-tail access through raw; turret rows (comment lines, empty model, short row), case-insensitive fit lookups; id registry with a tombstone, stem/name lookup, duplicate id; parse_allows full / partial / bad tokens, tree nodes, groups, GRP_ resolution, strategy links, ship / weapon links, roots, duplicate names; a miniature data root written to a temp directory exercising load_catalog, every lookup, and every cross_check list (each one seeded with exactly one dangling case); a missing root.
2. Real-data facts (SOTS_DATA_DIR set, else SKIP) — counts and published facts (below).
3. Oracle comparison — dump_catalog writes every def with its typed fields and its raw block rendered in the reference reader's dict shape; oracle/compare.py compares with weapons.json, shipsections.json, tech_tree.json, strings.json, crosslink.json.

Canonicalisation in compare.py: raw bodies are compared exactly and type-aware (this re-proves the block snapshot); typed fields are compared against the oracle value under the same key with last of a repeated key, repeatable keys always lists, numbers numerically, bools accept the oracle's 0/1, absent = null/""/[]; option lists are normalised on the oracle side the same way the loader does it; allows edges compare {from, to, rp, pct-as-written} and the effective 7-species percentages against pct.get(race, 100); the cross-check report is compared list by list with crosslink.json (manifest gaps folded, scalar options recomputed from the dump). Strings are cp1252 in the files and are decoded to Unicode by the dumper, since the oracle JSON holds decoded text.

Results (2026-09-07, owner's gob-extract)

what	ours	published
weapons	207 (123 player, 84 NPC)	207
ship sections	875 (Human 144, Hiver 137, Morrigi 141, Liir 135, Tarkas 132, Zuul 122, _NPC 64)	875
tech nodes / allows edges / groups	293 / 354 / 9	293 / 354 / 9
turret rows / weapon ids / deleted	42 / 123 / {36, 58, 59}	same
strings	5,196	5,196
banks / mounts / scalar options / repeated-spec banks / banks without size	3,721 / 7,375 / 229 / 19 / 2	same
lenient parse recoveries	12	12
oracle comparison	229,042 values, 0 differences	—
cross-check dangling	13 = 10 manifest ids without a file (Tarkas 3, 4, 16, 32, 43, 108, 109; DEWar in Human 98, Liir 86, Morrigi 39) + 3 NPC weapon name tokens absent from strings	same
every other link	0 dangling; 18 / 7 case-only mismatches; 30 weapons without requires; 67 stems without SECTIONDESC_; 12 roots	same

Divergences: none in values. The 34 untyped tokens listed in §3 are BadValue problems on our side and plain strings/bools in the oracle; compare.py requires that set to match exactly (it is spelled out in KNOWN_UNTYPED), so a change in either direction is a failure.

Skips cleanly with SOTS_DATA_DIR unset. tests/game_data/build/ (which holds the dump) is git-ignored; nothing from the game is committed.

5. Open questions

• Repeated scalar keys — first or last value (two engine sections' mass, four crew, 19 bank turretsize/turretclass)? Last is implemented; raw.all() keeps both.
• Converter leniency — what the engine makes of force_right o, crew false, trackspeed_mod 1.0f, min_inclination 0-5, max_inclination 90\. Probably 0, 0, 1.0, 0, 90 if it uses C-style prefix parsing; not assumed here.
• allows default of 100 for unlisted species is taken from the RE notes ("believed"), not proven from the binary.
• section_type for typeless sections — Other/None here; whether the engine treats a missing type as "mission" for rider hulls is code.
• tech ship{section} is kept as data (sections, unlocked_by) and is not used as a build gate, per SHIP_DESIGN_RULES.md §4.
• Turret fit rules (standard banks accepting missiles, strafe accepting standard, grapples) belong to the design validator, not this module; TurretTable only answers "is there a row".