data-model: SOTS content is data-driven; parser-first reimpl lever (from .gob harvest)

2026-09-07 14:57:31 -04:00 · 2026-09-07 14:57:31 -04:00 · ef8d557499
commit ef8d557499
parent ff8ca6668c
2 changed files with 258 additions and 0 deletions
--- a/campaign/backlog.md
+++ b/campaign/backlog.md
@ -4,6 +4,10 @@ Object model + control flow are mapped concurrently. Priority favors targets tha
 most other work. Groomed by **re-quartermaster**.

 ## Now
+1. **Data readers first (reimpl lever)** — implement the Mars brace-block parser + flat-KV +
+   CSV readers. Unlocks tech tree, weapons (207), ship sections (875), races, AI tables,
+   scenarios, tuning — nearly free. Also becomes the first `verify/parsers/`. See
+   `findings/subsystems/data-model.md`.
 1. **RTTI class inventory** — pull the full class/vftable list from ReVa; this seeds the real
   object-model target names (currently placeholders on the board).
 2. **empire/player object** — the root of the game state; cross-ref the save-editor structs.
--- a/findings/subsystems/data-model.md
+++ b/findings/subsystems/data-model.md
@ -0,0 +1,254 @@
+# SOTS1 .gob Data-Model Inventory
+
+Harvest of the data-driven object model from the two game archives. Both `.gob`
+files are **renamed uncompressed ZIPs** (`unzip -l` / `unzip` work directly).
+
+- `/srv/re-lab/sots-game/sots.gob` (1.45 GB) — main asset/data archive (8,354 entries)
+- `/srv/re-lab/sots-game/sots_local_en.gob` (599 MB) — English localization + voice (2,037 entries)
+- Extracted textual data (855 KB, 1,665 files): `/srv/re-lab/gob-extract/`
+- Raw `unzip -l` listings: `/srv/re-lab/handoff/list_sots.gob.txt`, `list_sots_local_en.gob.txt`
+
+## TL;DR
+Almost the entire *content model* of SOTS is data, not code. The engine ("Mars")
+provides a single recursive **brace-block, whitespace key/value** parser; the game
+("Game") loads catalogs of these blocks into definition objects at startup. Weapons
+(207), ship sections (875, per-species), the full tech tree (~293 nodes, one file),
+races, turrets, badges/lights, scenarios, combat missions, planet visuals, and all
+tuning constants (economy, combat, AI) are external files. Code owns *behavior*
+(how a `weaponclass bullet` flies and applies `dam_pop`), the data owns *which
+objects exist and their stats*. `@TOKEN`/`SOTS_*` string keys resolve against
+`Locale/EN/Strings.csv` (5,722 rows) — the model is fully localization-indirected.
+
+---
+
+## 1. File-type histograms
+
+### sots.gob (by extension)
+| ext | count | meaning |
+|---|---|---|
+| x | 4302 | DirectX `.X` meshes (ships, turrets, planets) — binary, NOT extracted |
+| tga | 1439 | textures / icons — NOT extracted |
+| shipsection | 885 | **ship module definitions** (data) |
+| dds | 482 | compressed textures — NOT extracted |
+| effect | 415 | combat/visual effect defs (data, brace-block) |
+| wav | 357 | sound — NOT extracted |
+| weapon | 207 | **weapon definitions** (data) |
+| txt | 59 | tuning-constant tables + list manifests (data) |
+| fx / fxh | 58 / 5 | HLSL shaders / headers |
+| csv | 26 | AI tables, star coords, string tables, scenarios (data) |
+| script | 4 | brace-block config (GUI/planet/tech-display) |
+| combat | 3 | combat-mission (encounter) defs |
+| def | 2 | ShipBadges, ShipLights |
+| tech | 1 | **MasterTechList.tech — the entire tech tree** |
+| decal/jpg/bmp | 5/1/1 | misc art |
+
+### sots_local_en.gob
+| ext | count | meaning |
+|---|---|---|
+| wav | 2002 | voice-over / speech — NOT extracted |
+| txt | 8 | race descriptions, chat translations, credits |
+| tga | 4 | localized art |
+| csv | 2 | `Strings.csv` (UI strings), `SpeechEvents.csv` |
+
+Extracted (textual only): 875 shipsection, 415 effect, 207 weapon, 67 txt, 58 fx,
+28 csv, 5 fxh, 4 script, 3 combat, 2 def, 1 tech.
+
+## 2. Top-level directory tree
+
+### sots.gob
+```
+Species/    5522  per-race art + sections/ (the ship-section catalogs)
+Effects/     599  .effect visual defs
+Models/      506  planets, asteroids, skysphere (+ PlanetResources.script)
+TechTree/    423  one .tech file + per-tech .tga icons
+GUI/         396  UI layout, SpriteTable.csv, WeaponIcons
+Sounds/      369  SFX (+ sound_ui.csv)
+Badges/      169  emblem art + BadgeTable.txt
+Weapons/     132  .weapon defs + _weapons/_turrets manifests + art/
+Avatars/     111  leader portraits + AvatarTable.txt
+Data/         44  *** GLOBAL TUNING: Combat/, Strategy/, globals, species, encounters ***
+Render/       38  render config
+Scenarios/    26  campaign/scenario defs (.txt + .csv fleet tables)
+Decals/       14
+CombatMissions/3  encounter templates (.combat)
+```
+### sots_local_en.gob
+```
+Sounds/  2018   Locale/  10   GUI/  6   Data/  1
+```
+
+---
+
+## 3. The core data format
+
+One convention dominates (engine's Mars parser). **Named brace blocks**, nested,
+with `key <whitespace> value` lines; `value` may be a bareword, number, or
+`"quoted string with spaces"`; `//` line comments; `@TOKEN` and `SOTS_*`/`WEAPON_*`
+references are localization keys resolved via `Strings.csv`.
+
+```
+weapon
+{
+    name  @WEAPON_BAL_GAUSS
+    cost  50
+    bank { turretclass standard  mount { node LightGunNode01  min_azimuth -130 } }
+}
+```
+
+Three secondary formats:
+- **Flat KEY value tuning tables** (`Data/**/*.txt`): `SLAVES_DEATH_RATE 0.05` — one
+  constant per line, colors as `"r g b"`. These are global singletons, not catalogs.
+- **CSV** (`Data/Strategy/**`, `Scenarios/*.csv`, locale): `#`-commented header,
+  comma-delimited; used for AI decision tables, star coordinates, string tables.
+- **Numbered manifest .txt** (`_weapons.txt`, `_shipsections.txt`): `<stable-int-id>
+  <filename>` — assigns the **network/savegame IDs**; explicit `// DELETED - 36`
+  and "always use new IDs" comment => IDs are a persistent wire contract.
+
+---
+
+## 4. High-value catalogs (schema + excerpt)
+
+### 4.1 Tech tree — `TechTree/MasterTechList.tech` (62 KB, ~293 nodes)
+Single file, list of `tech { }` blocks. This is the whole research graph.
+
+Schema per node:
+| key | meaning |
+|---|---|
+| `name "IND_Waldo"` | unique tech ID (FAMILY_Short) |
+| `family "IND"` | tech family/branch (roots mostly) |
+| `type P` | project type flag (P seen) |
+| `threat N` | AI/threat weighting |
+| `allows "CHILD_ID  RP:cost  Human:%  Zuul:% Hiver:% Tarkas:% Liir:% Morrigi:%"` | **directed edge**: unlocks CHILD at research-point cost, with per-species availability probability (0 = race can never get it, 100 = always). Multiple per node = the DAG. |
+| `strategy { inc/dec TECHBEN_* }` | economic/empire modifiers granted (e.g. `TECHBEN_INDOUTPUT`, `TECHBEN_HULLSTR`, `TECHBEN_SHIPCONCOST`) |
+| `ship { section DEHammerhead ... }` | **ship sections unlocked** by this tech (links tech -> shipsection catalog) |
+
+Families seen: IND, NRG/EWP, SLD, DRV, TRP, WHD/WAR, BAL, BIO, CCC, DRN, XNC.
+The per-species `Human:0 ... Morrigi:100` weighting encodes the game's famous
+race-unique-tech mechanic entirely in data (e.g. only Human gets `DRV_Node`, only
+Tarka `DRV_Hyper`, only Liir `DRV_StrWrp`).
+
+### 4.2 Weapons — `Weapons/*.weapon` (207) + `_weapons.txt` manifest
+```
+weapon {
+  name @WEAPON_BAL_GAUSS         weaponclass bullet    weaponfamily gauss
+  requires WEP_GsDrvr            cost 50
+  turretsize small  turretclass standard  trackspeed_mod 1.0
+  burst_volleys 1   recharge_time 5   muzzle_speed 300
+  range 455   range_planet 1075
+  model1/2/3 barrel_*.X          muzzle_effect/impact_effect effects/*.effect
+  fc_requires_los / fc_manual_target / fc_controllable ...   (fire-control flags)
+  bolt { rangetable { pb_range/eff_range/max_range + _dev + _dam }   <- damage falloff
+         dam_pop 3500  dam_infra .00005  dam_terra 0 }               <- vs pop/infra/terrain
+  rating_frate/dam/acc/range   <- UI star ratings
+}
+```
+- `weaponclass` (drives engine behavior): beam, bullet, missile, energy, torpedo,
+  laser, col(ony), mine, rider, emitter, grapple, etc. (38 distinct).
+- `weaponfamily` (AI grouping / mutual exclusion): gauss, energycannon, missile,
+  torpedo, conventionalbeam, laser, heavybeam, conventionalmine, emitter.
+- `requires <TECH_ID>` ties every weapon back to the tech tree.
+- `_weapons.txt` gives each weapon a **stable integer ID (1..123+)** for net/save.
+
+### 4.3 Ship sections — `Species/<Race>/sections/*.shipsection` (875)
+Per-race catalogs (Human 144, Hiver 137, Morrigi 141, Liir 135, Tarkas 132, Zuul
+122, _NPC 64). A "ship" = 3 sections (command/mission/engine) snapped by sockets.
+```
+shipsection {
+  model Species/Human/art/sections/CruiserAIC.X
+  requires CCC_AIFrCon           section_type command    section_class Cruiser
+  health 2800  mass 2500  cost 60000  cpoints 900  crew 3
+  socket_fore/socket_aft <node>          <- how sections join
+  option { option IND_PlyAlloy ... }     <- tech-gated build options/upgrades
+  netforcelimits { force_forward/right/up  torque_yaw/pitch/roll  speed rotspeed }
+  bank { turretclass std turretsize medium
+         mount { node MediumGunNode01  min/max_azimuth  min/max_inclination } }
+}
+```
+- `section_type`: command (164), mission (502), engine (108). => the 3-part hull.
+- `section_class`: destroyer / cruiser / dreadnought (the 3 hull sizes). Name
+  prefixes DE/CR/DN encode class.
+- Engine sections add `ftlspeed`, `nodespeed`, `engine_techera`, thrusters.
+- `bank`/`mount` = hardpoints (turret class+size + gimbal limits) => this is the
+  data that constrains which `.weapon` can be fitted where.
+
+### 4.4 Turrets — `Weapons/_turrets.txt` (positional table)
+`size  weapon-size  class  health  track-speed  azimuth%  inclination%  "model.x"`
+Cross-references weapon `turretsize`/`turretclass` and section `bank` slots.
+
+### 4.5 Races / species
+- `Data/species.txt` — per-race scalars: `SENSORMOD_<RACE>`, `SPECIESCOLOR_<RACE>`.
+- Deeper per-race behavior is expressed *distributed*: their `sections/` catalogs,
+  the `Human:/Zuul:/...` columns in the tech tree, `affinity_*` AI tables, and
+  locale `Desc<Race>.txt`. There is no single "race stats" file — race identity is
+  emergent from these tables.
+- `Data/Strategy/playercolors.txt`, `starcolors.txt`, `systemnames.txt` — cosmetic/naming.
+
+### 4.6 Planets / economy
+- **Economy tuning** lives in flat KEY-value tables, not per-planet rows:
+  `Data/Strategy/StrategyVars.txt` (117 lines: bankruptcy, trade routes
+  `TRADE_ROUTE_REQ_*`, slaves `SLAVES_*`, system bonuses, starmap sizing),
+  `Data/globals.txt` (435 lines, mostly UI but some sim), `Data/Combat/planet.txt`
+  (planetary-defense combat params).
+- `Models/Planets/PlanetResources.script` — per-race x planet-quality (Ideal/…) ->
+  texture sets; visual, not economic.
+- `Data/Strategy/RealSpace.csv` — `system,x,y,z,Name` real-star coordinates (~348).
+- Planet *economic model* (population growth, output, terraforming) is largely in
+  code, tuned by the StrategyVars/globals constants — the weakest data coverage.
+
+### 4.7 AI decision tables — `Data/Strategy/AI/*.csv`
+Clean, comment-documented CSVs the AI consults:
+- `affinity_section.csv` — `section,species,purpose` (which sections serve which role)
+- `affinity_weapon.csv` — `weapon-family,purpose` (e.g. `gauss,reflect_cm`)
+- `weapon_replacements.csv` — `weapon,replaces-weapon` final substitution pass
+- `aitechpri.csv` — `tech, <6 per-race priorities 1-255>` (lower = research sooner)
+- `aitechgrp/aitechmode` , `stock_design_names`, `stock_player_names`,
+  `rebelai_names`, `stock_diplomacy_messages`, `raider_sections`.
+
+### 4.8 Scenarios & combat missions
+- `Scenarios/*.txt` (18) — `name/desc/objs/rules` (all `@`-tokens), `numplayers`,
+  `numsystems`, `mapshape`, `script_object`, repeated `player { recommended }`.
+  Backed by `Scenarios/*_Fleets.csv`, `_FleetTemplates.csv`, `_Techs.csv`,
+  `_ShipRequests.csv` — data-driven fleet/tech grants for campaigns.
+- `CombatMissions/Encounter/{DeepSpace,InSystem,NodeSpace}.combat` — encounter
+  templates: `summary/combatenv/endcondition{elimination,time}/orbit{type,radius}`.
+
+### 4.9 Cosmetic catalogs
+`Species/ShipBadges.def` (badge {name,w,h,depth,orientation}), `ShipLights.def`
+(light {name,effect,period,offset,duration}), `Badges/BadgeTable.txt`,
+`Avatars/AvatarTable.txt`, `GUI/SpriteTable.csv`, `Effects/*.effect` (415).
+
+### 4.10 Localization — `sots_local_en.gob`
+`Locale/EN/Strings.csv` (5,722 rows: `Key,String,Size,Notes`) resolves every
+`@TOKEN`/`SOTS_*`/`WEAPON_*` in the data. `SpeechEvents.csv`, `Desc<Race>.txt`,
+`ChatTrans.txt`. Confirms the whole model is name-key indirected for i18n.
+
+---
+
+## 5. Data-defined vs code-defined (map to `Game::` classes)
+
+| Object model area | Where | Likely class |
+|---|---|---|
+| Tech graph (nodes, edges, RP costs, race gating, benefits) | **DATA** `MasterTechList.tech` | `Game::TechDef` / TechTree; `TECHBEN_*` enum in code |
+| Which ship sections/weapons a tech unlocks | **DATA** (`ship{}` / `requires`) | linkage tables |
+| Weapon stats, damage falloff, fire-control | **DATA** `*.weapon` | `Game::WeaponDef` |
+| Weapon *flight/behavior* per `weaponclass` | **CODE** | weapon subclasses keyed by `weaponclass` string |
+| Ship sections: stats, sockets, hardpoints, thrust | **DATA** `*.shipsection` | `Game::ShipSectionDef` |
+| Ship = command+mission+engine assembly rules | **DATA** (socket/section_type) | `Game::ShipDesign` |
+| Turret sizes/tracking/models | **DATA** `_turrets.txt` | `Game::TurretDef` |
+| Stable net/save IDs | **DATA** `_weapons.txt`/`_shipsections.txt` | id registry |
+| Race sensor/color scalars | **DATA** `species.txt` | `Game::Species` |
+| Race economy/behavior specifics | **CODE + tuning consts** | per-race code paths |
+| Economy: trade/bankruptcy/slaves/system bonus consts | **DATA** `StrategyVars.txt` | global config singleton |
+| Planet growth/output/terraform simulation | **CODE** (tuned by consts) | `Game::Planet` |
+| Star map coords / names | **DATA** `RealSpace.csv`, `systemnames.txt` | map generator |
+| AI weapon/section/tech preferences | **DATA** `Strategy/AI/*.csv` | AI advisor |
+| Combat sim constants (ship/planet/drone/sensors…) | **DATA** `Data/Combat/*.txt` | combat engine config |
+| Scenarios / encounters | **DATA** `.txt`+`.csv` / `.combat` | scenario loader |
+| Strings / speech | **DATA** locale gob | string table |
+
+**Bottom line for reimplementation:** stand up the Mars brace-block parser + the
+flat-KV and CSV readers first; you then get the tech tree, all weapons, all ship
+sections/turrets, races, scenarios, AI tables and every tuning constant essentially
+for free. The remaining code work is *behavior*: weapon flight per `weaponclass`,
+the planet/economy simulation (data only supplies its constants), per-race special
+logic, and the combat/strategy engines that consume these catalogs.