sots-engine/docs/game-data.md

# `game::data` — the typed catalogs

`src/game/data/` turns the parsed data files into the definition objects the
game builds at startup: weapons, ship sections, the turret table, the stable
id registries, the tech tree, the string table, and a `Catalog` that loads a
whole data root, resolves names and cross-checks every reference. It sits on
`mars::parse` (brace blocks) and `mars::text` (rows, manifests, CSV); library
target `game_data` (static, `src/game/data/CMakeLists.txt`), include as
`game/data/<header>.h`. C++17, no dependencies, no exceptions across the API.

Behaviour is specified by the RE findings (`data-model.md` §4,
`data-parsers.md`, `SHIP_DESIGN_RULES.md` §1–3); the normalised JSON catalogs
the reference Python tooling emits are the oracle, and this module is checked
against them value by value (§5).

```
src/game/data/
  common.h/.cpp      Species (engine index order), Problem, Loaded<T>, case folding
  block.h/.cpp       Block/Attr: copyable snapshot of a parsed block, typed reads
  fields.h           internal: typed reads that record Problems
  weapon.h/.cpp      WeaponDef, RangeTable, BoltDef, FireControl, Ratings; load_weapon
  shipsection.h/.cpp ShipSectionDef, OptionGroup, BankDef, MountDef, NetForceLimits; load_shipsection
  turrets.h/.cpp     TurretTable (_turrets.txt), IdRegistry (_weapons/_shipsections.txt)
  techtree.h/.cpp    TechTree, TechNode, AllowsEdge, parse_allows
  strings.h/.cpp     StringTable (Strings.csv)
  catalog.h/.cpp     Catalog: load_catalog(root), lookups, cross_check()
tests/game_data/
  build_and_run.sh   plain g++ build; unit tests; real-data facts; oracle comparison
  test_*.cpp         hand-written samples for every loader and the option normalisation
  realdata_test.cpp  counts / spot facts / cross-check against the published numbers
  dump_catalog.cpp   whole catalog -> JSON (typed fields + raw blocks)
  oracle/compare.py  field-by-field comparison with the reference catalogs
```

## 1. Common conventions

- **Loaders are total.** `load_x(const mars::parse::Document&)` returns
  `Loaded<T> { optional<T> value; vector<Problem> problems; }`. Only a file
  with no usable top-level block (or a hard syntax error in the `parse_x`
  text convenience) yields no value; everything else loads and reports.
  `Problem { kind, file, line, key, message }` kinds: `MissingBlock`,
  `MissingKey`, `BadValue`, `Unparsed`, `Duplicate`, `IoError`, `Syntax`
  (the parser's lenient recoveries are reported as `Syntax` "recovered: …").
- **Typed fields are `optional`.** A key that is absent is `nullopt`; a key
  whose value does not have the expected shape is `nullopt` *plus* a
  `BadValue` problem, and the text stays reachable in `raw`. Ints need the
  int shape; doubles accept int or C-float; bools accept `true`/`false` (any
  case) or an int (`hidden 1`, `autonomous 1`, `nodesign 1`).
- **Last occurrence wins** when a scalar key repeats (`mass 1800 mass 1000`
  on two engine sections, `crew` on four Zuul sections, `turretsize` in 19
  banks). Repeatable keys (`requires`, `exclude`, `compatible_section`,
  `option`, `bank`, `mount`, `thruster`, `allows`, `inc`/`dec`, `section`,
  `filename`) are vectors in file order.
- **Text kept as written, matched folded.** `turretclass PlanetMissile`,
  `section_type Engine`, `requires WEP_HCLAS` are stored verbatim; every
  lookup, enum derivation and cross-link comparison is ASCII
  case-insensitive, which is how the data itself is consistent.
- **`raw`.** Every def keeps a `Block` copy of its whole block: ordered
  `attrs` (key, text, quoted, line), `blocks`, `items`, with `find` (last
  wins) / `all` / `block` / `blocks_named` / `str` / `strs` and typed
  `Attr::as_int/as_double/as_bool`. The long tail of section role flags and
  capacities (`refinery`, `police`, `colonizer_pop`, `mining_rate`, sounds,
  `anim{}`) and the non-`bolt` weapon behaviour blocks live there.
- **Species** are indexed in the engine's own order — `Human 0, Hiver 1,
  Tarkas 2, Liir 3, _NPC 4, Zuul 5, Morrigi 6` (the save-file species table).

## 2. Schemas as implemented

### `WeaponDef` (`weapon{}`; `Weapons/*.weapon`, `Species/<Race>/weapons/*.weapon`)

| field | key | type | notes |
|---|---|---|---|
| `stem`, `file`, `scope`, `id` | — | | catalog identity; `scope` NPC for anything under `Species/`; `id` from `_weapons.txt` (player only) |
| `display_name` | — | optional string | `name` resolved through the string table (`@X` → `X`); a non-`@` name is its own display name |
| `name` | `name` | string | `@WEAPON_*` token, required |
| `weapon_class`, `weapon_family`, `weapon_damage_type` | `weaponclass` (required), `weaponfamily`, `weapondamagetype` | string | |
| `requires` | `requires`* | vector | AND of techs; 30 weapons have none |
| `compatible_section`* / `exclusive_species` | | vector / string | rider carriers; `zuul` on the two grapples |
| `cost`, `burst_volleys`, `range`, `range_planet`, `muzzle_speed`, `hpbonus`, `dam_est` | | optional int | |
| `turret_size`, `turret_class` | `turretsize`, `turretclass` (both required) | string | |
| `track_speed_mod`, `recharge_time`, `volley_period`, `volley_duration`, `buildup_delay`, `solution_tolerance` | | optional double | |
| `hidden`, `pinpoint`, `blindfire`, `secondary_pd` | | optional bool | |
| `model1..3`, `muzzle_effect`, `muzzle_sound`, `icon_file`, `icon_rect` | | string | |
| `fc` | `fc_requires_los … fc_targets_expire` | 11 optional bools | fire-control flags |
| `ratings` | `rating_frate/dam/acc/range` | optional double | UI stars; fractional on 18 weapons |
| `behavior_kind` | first of `bolt beam torpedo rider missile chainlightning col mine disintegrator grapple projectedshield mirv nodecannon siege mesonprojector spyship wraith` present | string | `behavior()` returns that block from `raw` |
| `bolt` | `bolt{}` | optional `BoltDef` | `rangetable`, `planet` (`dam_pop/infra/terra`), `mass`, `beam_origin`, `beam_length`, `ricochet_mod`, `effect`, `impact_effect`, `expire_effect` |
| `rangetable` | `<behaviour>.rangetable{}` | optional `RangeTable` | present for `bolt` (66) and `torpedo` (19): three bands `point_blank/effective/maximum` of `{range, deviation, damage}` from `pb_/eff_/max_range[_dev|_dam]` |
| `planet_damage` | `<behaviour>.dam_pop/dam_infra/dam_terra` | `PlanetDamage` | whichever behaviour block is present |

### `ShipSectionDef` (`shipsection{}`; `Species/<Race>/sections/*.shipsection`)

| field | key | type | notes |
|---|---|---|---|
| `race`, `species`, `stem`, `file`, `id` | — | | `race` is the directory name; `id` from the race's `_shipsections.txt` |
| `display_name`, `description`, `unlocked_by` | — | | `SECTIONNAME_<stem>` / `SECTIONDESC_<stem>`; techs whose `ship{section}` names the stem (tech order) |
| `model` (required), `dam_model` | | string | |
| `requires`* | | vector | may name `GRP_<group>` |
| `section_type_text` → `section_type` | `section_type` | string → `SectionType {None, Command, Mission, Engine, Other}` | 41 sections have none (riders, NPC hulls) |
| `section_class_text` → `section_class` | `section_class` | string → `SectionClass {None, Destroyer, Cruiser, Dreadnought, Other}` | |
| `design_class`, `entity_class` | | string | `station`, `rider`, … |
| `health`, `mass` (both required) | | optional double | NPC masses are written as floats |
| `cost`, `cpoints`, `crew`, `command_cost`, `maintenance_cost`, `command_quota` | | optional int | |
| `socket_fore/aft`, `dam_socket_fore/aft` | | string | geometry only; `has_sockets()` is the hull-vs-standalone test (design rule A3) |
| `options` | `option T` and `option{ option A … }`* | vector of `OptionGroup { members, scalar, line }` | **one normalised list in file order**: a scalar entry is a one-member group with `scalar = true`, a block keeps its members. 229 scalar entries in 153 files, 1,226 blocks |
| `optiondef` | `optiondef{ option … }` | optional `OptionGroup` | the shield-level group on the 12 shield sections; kept separate because it is a separate key |
| `banks` | `bank{}`* | vector of `BankDef` | `turret_class`, `turret_size` (last if repeated; `repeated_turret_spec` flags the 19 banks that do), `weapon` (fixed weapon file on 404 NPC banks), `show_turrets`, `invincible`, `mounts` (`MountDef { node, min/max_azimuth, min/max_inclination, home_azimuth/inclination }`) |
| `ftlspeed`, `nodespeed`, `range`, `scanrange`, `tactical_sensor_range` | | optional double | |
| `engine_techera` | | string | `fission / fusion / antimatter` |
| `netforcelimits` | `netforcelimits{}` | optional `NetForceLimits` | `force_forward/right/up`, `torque_yaw/pitch/roll`, `speed`, `rotspeed` |
| `thrusters` | `thruster{}`* | vector | `node`, `effect`, `idle_effect` |
| `exclude`* | `exclude "STEM"` | vector | ramscoop command sections |
| `explicit_command_section`, `explicit_engine_section`, `explicit_section` | | string, string, optional bool | station triples |
| `autonomous`, `nodesign` | | optional bool | written as `1` or `true` |

### `TurretTable` (`Weapons/_turrets.txt`) and `IdRegistry`

`TurretRow { mount_size, weapon_size, turret_class, health, track_speed,
azimuth_scale, inclination_scale, model }` from the 8-token positional rows
(a row with any other width is `Unparsed` and skipped). `find(mount, weapon,
class)`, `has_weapon_pair(weapon_size, class)`, `has_bank_pair(mount_size,
class)`, `rows_for_bank(...)` — all case-insensitive.

`IdRegistry` wraps `mars::text::Manifest`: `IdEntry { id, name, stem, line }`,
`deleted()` tombstones, `find(id)` (last assignment wins), `id_of(name or
stem)` case-insensitive (`DEWar.SHIPSECTION`, `dewar`, `DEWar.shipsection`
all resolve), `is_deleted(id)`. Manifest problems map to `Unparsed` /
`Duplicate`.

### `TechTree` (`TechTree/MasterTechList.tech`)

`TechNode { name, family (as written, 155 nodes), family_inferred (upper
name prefix), type, threat, group (as written), option_cost,
unlock_explicitly, requires, benefits_inc, benefits_dec, sections,
weapon_files, allows (edge indices), display_name, description, raw }`.

`AllowsEdge { from, to, rp, pct[7], pct_written[7], unparsed, text, line }` —
`parse_allows` splits the string on whitespace: token 0 is the child,
`RP:n` (any case) the cost, `<Species>:n` a percentage for that species
(any of the seven names, any case); anything else lands in `unparsed` and is
reported. **Unlisted species default to 100** — this is the engine default
the RE notes describe (`data-parsers.md`, "believed 100"); `pct_written`
records which values the file actually spelled out so a consumer can tell
the two apart. `_NPC` is never written and is therefore always 100.

`groups`: upper-cased `group` value → member names (first-seen order);
`group_members("TORPS")` and `group_members("GRP_Torps")` both work;
`requirement_exists(token)` is true for a tech name or a non-empty group.
`strategy{ inc/dec }` is kept as the `TECHBEN_*` token lists — no semantics
are attached here. `edges_to / edges_from / roots()` are graph helpers.

### `StringTable` (`Locale/EN/Strings.csv`)

Column 0 → column 1 of every data row of the commented CSV; keys folded, the
later row wins on a repeat (the four trailing-space duplicates), `resolve()`
strips a leading `@`.

### `Catalog`

`load_catalog(root)` reads `Weapons/*.weapon` + `_weapons.txt` +
`_turrets.txt`, every `Species/<dir>/sections/` (+ `_shipsections.txt`) and
`Species/<dir>/weapons/`, `TechTree/MasterTechList.tech` and, if present,
`Locale/EN/Strings.csv`; missing mandatory files are `IoError` problems, the
rest still loads. Weapons are ordered player-first then by folded stem,
sections by (race, folded stem); ids, display names, descriptions and
`unlocked_by` are filled in; lookups: `weapon(stem)` (player catalog wins on
a stem clash), `weapon_by_file`, `weapon_by_id`, `section(race, stem)`,
`section_by_id`, `sections_named(stem)` (all races), `tech_node`.

`cross_check()` → `CrossCheck` with one vector per link kind. Dangling
(counted by `dangling_count()`, `clean()` = 0): weapon/section `requires` →
tech (a `GRP_x` needs a non-empty group), section `option`/`optiondef`
members → tech, tech `ship{section}` → any race's section, tech
`weapon{filename}` → a loaded weapon file, tech `requires` → tech/group,
`allows` child → tech (and unparsed edges), `bank{weapon}` → weapon file,
manifest id → file and file → manifest id, weapon and bank
(size, class) → a turret row, `@name` → string. Informational: case-only
mismatches, weapons without `requires`, missing `TECHNAME_/TECHDESC_` and
`SECTIONNAME_/SECTIONDESC_` (per distinct stem), roots.

## 3. Normalisation decisions

| decision | why |
|---|---|
| `option` scalar and block forms merge into one ordered list of groups | the engine treats a bare `option T` as a one-member choice (design rule D1); order is preserved through the entries' position in the block so the save-file `DOpts` order (group order) can be reproduced |
| `optiondef` stays a separate optional group | different key, different meaning (shield level), only 12 files |
| repeated scalar → last value, flagged on banks | the design-rule validator takes last; the engine's choice is unknown (open question), so the fact is kept (`repeated_turret_spec`, `raw.all(key)`) rather than hidden |
| numeric fields that do not parse → `nullopt` + `BadValue`, text kept in `raw` | 34 values in the shipped data are typos (`force_right o` ×22, `crew false` ×9, `trackspeed_mod 1.0f`, `min_inclination 0-5`, `max_inclination 90\`); guessing what the engine's converter makes of them would be fiction, so they are reported instead |
| bools accept 0/1 | `hidden 1`, `autonomous 1`, `nodesign 1` coexist with `true` in the data |
| `section_type`/`section_class` enums beside the text | the data spells them in two cases; behaviour code wants the enum, the oracle comparison wants the text |
| species percentages stored for all seven species, default 100 | matches the RE description of the engine default and gives `_NPC` (which has a catalog but never appears in `allows`) a defined value |
| `family_inferred` from the name prefix | half the nodes lack `family`, and where present it is sometimes the *display* branch (`WEP_Dsrptr` has `family "TRP"`) |
| NPC weapons are `scope NPC` with no id | they have no manifest; NPC banks reference them by file path (`bank{weapon}`) |

## 4. Tests

`tests/game_data/build_and_run.sh` (plain `g++ -std=c++17 -Wall -Wextra
-Wpedantic -Werror`):

1. **Unit tests** — 27 cases on hand-written samples: every typed field of a
   weapon, `bolt` + `rangetable`, torpedo/beam behaviour detection, missing
   keys and bad values as problems, fatal no-block / syntax error, lenient
   recovery, last-wins and list order; section fields, the option
   normalisation across both forms, `optiondef`, banks with repeated
   `turretsize`, mounts, `netforcelimits`, thrusters, `exclude`, enum
   parsing, long-tail access through `raw`; turret rows (comment lines,
   empty model, short row), case-insensitive fit lookups; id registry with a
   tombstone, stem/name lookup, duplicate id; `parse_allows` full / partial
   / bad tokens, tree nodes, groups, `GRP_` resolution, strategy links, ship
   / weapon links, roots, duplicate names; a miniature data root written to
   a temp directory exercising `load_catalog`, every lookup, and every
   `cross_check` list (each one seeded with exactly one dangling case); a
   missing root.
2. **Real-data facts** (`SOTS_DATA_DIR` set, else SKIP) — counts and
   published facts (below).
3. **Oracle comparison** — `dump_catalog` writes every def with its typed
   fields and its `raw` block rendered in the reference reader's dict shape;
   `oracle/compare.py` compares with `weapons.json`, `shipsections.json`,
   `tech_tree.json`, `strings.json`, `crosslink.json`.

Canonicalisation in `compare.py`: `raw` bodies are compared exactly and
type-aware (this re-proves the block snapshot); typed fields are compared
against the oracle value under the same key with *last of a repeated key*,
*repeatable keys always lists*, *numbers numerically*, *bools accept the
oracle's 0/1*, *absent = null/""/[]*; option lists are normalised on the
oracle side the same way the loader does it; `allows` edges compare
`{from, to, rp, pct-as-written}` and the effective 7-species percentages
against `pct.get(race, 100)`; the cross-check report is compared list by
list with `crosslink.json` (manifest gaps folded, scalar options recomputed
from the dump). Strings are cp1252 in the files and are decoded to Unicode
by the dumper, since the oracle JSON holds decoded text.

### Results (2026-09-07, owner's `gob-extract`)

| what | ours | published |
|---|---|---|
| weapons | 207 (123 player, 84 NPC) | 207 |
| ship sections | 875 (Human 144, Hiver 137, Morrigi 141, Liir 135, Tarkas 132, Zuul 122, _NPC 64) | 875 |
| tech nodes / `allows` edges / groups | 293 / 354 / 9 | 293 / 354 / 9 |
| turret rows / weapon ids / deleted | 42 / 123 / {36, 58, 59} | same |
| strings | 5,196 | 5,196 |
| banks / mounts / scalar options / repeated-spec banks / banks without size | 3,721 / 7,375 / 229 / 19 / 2 | same |
| lenient parse recoveries | 12 | 12 |
| **oracle comparison** | **229,042 values, 0 differences** | — |
| cross-check dangling | 13 = 10 manifest ids without a file (Tarkas 3, 4, 16, 32, 43, 108, 109; `DEWar` in Human 98, Liir 86, Morrigi 39) + 3 NPC weapon name tokens absent from strings | same |
| every other link | 0 dangling; 18 / 7 case-only mismatches; 30 weapons without `requires`; 67 stems without `SECTIONDESC_`; 12 roots | same |

**Divergences: none in values.** The 34 untyped tokens listed in §3 are
`BadValue` problems on our side and plain strings/bools in the oracle;
`compare.py` requires that set to match exactly (it is spelled out in
`KNOWN_UNTYPED`), so a change in either direction is a failure.

Skips cleanly with `SOTS_DATA_DIR` unset. `tests/game_data/build/` (which
holds the dump) is git-ignored; nothing from the game is committed.

## 5. Open questions

- **Repeated scalar keys** — first or last value (two engine sections'
  `mass`, four `crew`, 19 bank `turretsize/turretclass`)? Last is
  implemented; `raw.all()` keeps both.
- **Converter leniency** — what the engine makes of `force_right o`,
  `crew false`, `trackspeed_mod 1.0f`, `min_inclination 0-5`,
  `max_inclination 90\`. Probably `0`, `0`, `1.0`, `0`, `90` if it uses
  C-style prefix parsing; not assumed here.
- **`allows` default of 100** for unlisted species is taken from the RE
  notes ("believed"), not proven from the binary.
- **`section_type` for typeless sections** — `Other`/`None` here; whether
  the engine treats a missing type as "mission" for rider hulls is code.
- **`tech ship{section}`** is kept as data (`sections`, `unlocked_by`) and
  is *not* used as a build gate, per `SHIP_DESIGN_RULES.md` §4.
- **Turret fit rules** (standard banks accepting missiles, strafe accepting
  standard, grapples) belong to the design validator, not this module;
  `TurretTable` only answers "is there a row".