diff --git a/campaign/backlog.md b/campaign/backlog.md index e9ad02f..4062ecc 100644 --- a/campaign/backlog.md +++ b/campaign/backlog.md @@ -4,6 +4,10 @@ Object model + control flow are mapped concurrently. Priority favors targets tha most other work. Groomed by **re-quartermaster**. ## Now +1. **Data readers first (reimpl lever)** — implement the Mars brace-block parser + flat-KV + + CSV readers. Unlocks tech tree, weapons (207), ship sections (875), races, AI tables, + scenarios, tuning — nearly free. Also becomes the first `verify/parsers/`. See + `findings/subsystems/data-model.md`. 1. **RTTI class inventory** — pull the full class/vftable list from ReVa; this seeds the real object-model target names (currently placeholders on the board). 2. **empire/player object** — the root of the game state; cross-ref the save-editor structs. diff --git a/findings/subsystems/data-model.md b/findings/subsystems/data-model.md new file mode 100644 index 0000000..386683f --- /dev/null +++ b/findings/subsystems/data-model.md @@ -0,0 +1,254 @@ +# SOTS1 .gob Data-Model Inventory + +Harvest of the data-driven object model from the two game archives. Both `.gob` +files are **renamed uncompressed ZIPs** (`unzip -l` / `unzip` work directly). + +- `/srv/re-lab/sots-game/sots.gob` (1.45 GB) — main asset/data archive (8,354 entries) +- `/srv/re-lab/sots-game/sots_local_en.gob` (599 MB) — English localization + voice (2,037 entries) +- Extracted textual data (855 KB, 1,665 files): `/srv/re-lab/gob-extract/` +- Raw `unzip -l` listings: `/srv/re-lab/handoff/list_sots.gob.txt`, `list_sots_local_en.gob.txt` + +## TL;DR +Almost the entire *content model* of SOTS is data, not code. The engine ("Mars") +provides a single recursive **brace-block, whitespace key/value** parser; the game +("Game") loads catalogs of these blocks into definition objects at startup. Weapons +(207), ship sections (875, per-species), the full tech tree (~293 nodes, one file), +races, turrets, badges/lights, scenarios, combat missions, planet visuals, and all +tuning constants (economy, combat, AI) are external files. Code owns *behavior* +(how a `weaponclass bullet` flies and applies `dam_pop`), the data owns *which +objects exist and their stats*. `@TOKEN`/`SOTS_*` string keys resolve against +`Locale/EN/Strings.csv` (5,722 rows) — the model is fully localization-indirected. + +--- + +## 1. File-type histograms + +### sots.gob (by extension) +| ext | count | meaning | +|---|---|---| +| x | 4302 | DirectX `.X` meshes (ships, turrets, planets) — binary, NOT extracted | +| tga | 1439 | textures / icons — NOT extracted | +| shipsection | 885 | **ship module definitions** (data) | +| dds | 482 | compressed textures — NOT extracted | +| effect | 415 | combat/visual effect defs (data, brace-block) | +| wav | 357 | sound — NOT extracted | +| weapon | 207 | **weapon definitions** (data) | +| txt | 59 | tuning-constant tables + list manifests (data) | +| fx / fxh | 58 / 5 | HLSL shaders / headers | +| csv | 26 | AI tables, star coords, string tables, scenarios (data) | +| script | 4 | brace-block config (GUI/planet/tech-display) | +| combat | 3 | combat-mission (encounter) defs | +| def | 2 | ShipBadges, ShipLights | +| tech | 1 | **MasterTechList.tech — the entire tech tree** | +| decal/jpg/bmp | 5/1/1 | misc art | + +### sots_local_en.gob +| ext | count | meaning | +|---|---|---| +| wav | 2002 | voice-over / speech — NOT extracted | +| txt | 8 | race descriptions, chat translations, credits | +| tga | 4 | localized art | +| csv | 2 | `Strings.csv` (UI strings), `SpeechEvents.csv` | + +Extracted (textual only): 875 shipsection, 415 effect, 207 weapon, 67 txt, 58 fx, +28 csv, 5 fxh, 4 script, 3 combat, 2 def, 1 tech. + +## 2. Top-level directory tree + +### sots.gob +``` +Species/ 5522 per-race art + sections/ (the ship-section catalogs) +Effects/ 599 .effect visual defs +Models/ 506 planets, asteroids, skysphere (+ PlanetResources.script) +TechTree/ 423 one .tech file + per-tech .tga icons +GUI/ 396 UI layout, SpriteTable.csv, WeaponIcons +Sounds/ 369 SFX (+ sound_ui.csv) +Badges/ 169 emblem art + BadgeTable.txt +Weapons/ 132 .weapon defs + _weapons/_turrets manifests + art/ +Avatars/ 111 leader portraits + AvatarTable.txt +Data/ 44 *** GLOBAL TUNING: Combat/, Strategy/, globals, species, encounters *** +Render/ 38 render config +Scenarios/ 26 campaign/scenario defs (.txt + .csv fleet tables) +Decals/ 14 +CombatMissions/3 encounter templates (.combat) +``` +### sots_local_en.gob +``` +Sounds/ 2018 Locale/ 10 GUI/ 6 Data/ 1 +``` + +--- + +## 3. The core data format + +One convention dominates (engine's Mars parser). **Named brace blocks**, nested, +with `key value` lines; `value` may be a bareword, number, or +`"quoted string with spaces"`; `//` line comments; `@TOKEN` and `SOTS_*`/`WEAPON_*` +references are localization keys resolved via `Strings.csv`. + +``` +weapon +{ + name @WEAPON_BAL_GAUSS + cost 50 + bank { turretclass standard mount { node LightGunNode01 min_azimuth -130 } } +} +``` + +Three secondary formats: +- **Flat KEY value tuning tables** (`Data/**/*.txt`): `SLAVES_DEATH_RATE 0.05` — one + constant per line, colors as `"r g b"`. These are global singletons, not catalogs. +- **CSV** (`Data/Strategy/**`, `Scenarios/*.csv`, locale): `#`-commented header, + comma-delimited; used for AI decision tables, star coordinates, string tables. +- **Numbered manifest .txt** (`_weapons.txt`, `_shipsections.txt`): ` + ` — assigns the **network/savegame IDs**; explicit `// DELETED - 36` + and "always use new IDs" comment => IDs are a persistent wire contract. + +--- + +## 4. High-value catalogs (schema + excerpt) + +### 4.1 Tech tree — `TechTree/MasterTechList.tech` (62 KB, ~293 nodes) +Single file, list of `tech { }` blocks. This is the whole research graph. + +Schema per node: +| key | meaning | +|---|---| +| `name "IND_Waldo"` | unique tech ID (FAMILY_Short) | +| `family "IND"` | tech family/branch (roots mostly) | +| `type P` | project type flag (P seen) | +| `threat N` | AI/threat weighting | +| `allows "CHILD_ID RP:cost Human:% Zuul:% Hiver:% Tarkas:% Liir:% Morrigi:%"` | **directed edge**: unlocks CHILD at research-point cost, with per-species availability probability (0 = race can never get it, 100 = always). Multiple per node = the DAG. | +| `strategy { inc/dec TECHBEN_* }` | economic/empire modifiers granted (e.g. `TECHBEN_INDOUTPUT`, `TECHBEN_HULLSTR`, `TECHBEN_SHIPCONCOST`) | +| `ship { section DEHammerhead ... }` | **ship sections unlocked** by this tech (links tech -> shipsection catalog) | + +Families seen: IND, NRG/EWP, SLD, DRV, TRP, WHD/WAR, BAL, BIO, CCC, DRN, XNC. +The per-species `Human:0 ... Morrigi:100` weighting encodes the game's famous +race-unique-tech mechanic entirely in data (e.g. only Human gets `DRV_Node`, only +Tarka `DRV_Hyper`, only Liir `DRV_StrWrp`). + +### 4.2 Weapons — `Weapons/*.weapon` (207) + `_weapons.txt` manifest +``` +weapon { + name @WEAPON_BAL_GAUSS weaponclass bullet weaponfamily gauss + requires WEP_GsDrvr cost 50 + turretsize small turretclass standard trackspeed_mod 1.0 + burst_volleys 1 recharge_time 5 muzzle_speed 300 + range 455 range_planet 1075 + model1/2/3 barrel_*.X muzzle_effect/impact_effect effects/*.effect + fc_requires_los / fc_manual_target / fc_controllable ... (fire-control flags) + bolt { rangetable { pb_range/eff_range/max_range + _dev + _dam } <- damage falloff + dam_pop 3500 dam_infra .00005 dam_terra 0 } <- vs pop/infra/terrain + rating_frate/dam/acc/range <- UI star ratings +} +``` +- `weaponclass` (drives engine behavior): beam, bullet, missile, energy, torpedo, + laser, col(ony), mine, rider, emitter, grapple, etc. (38 distinct). +- `weaponfamily` (AI grouping / mutual exclusion): gauss, energycannon, missile, + torpedo, conventionalbeam, laser, heavybeam, conventionalmine, emitter. +- `requires ` ties every weapon back to the tech tree. +- `_weapons.txt` gives each weapon a **stable integer ID (1..123+)** for net/save. + +### 4.3 Ship sections — `Species//sections/*.shipsection` (875) +Per-race catalogs (Human 144, Hiver 137, Morrigi 141, Liir 135, Tarkas 132, Zuul +122, _NPC 64). A "ship" = 3 sections (command/mission/engine) snapped by sockets. +``` +shipsection { + model Species/Human/art/sections/CruiserAIC.X + requires CCC_AIFrCon section_type command section_class Cruiser + health 2800 mass 2500 cost 60000 cpoints 900 crew 3 + socket_fore/socket_aft <- how sections join + option { option IND_PlyAlloy ... } <- tech-gated build options/upgrades + netforcelimits { force_forward/right/up torque_yaw/pitch/roll speed rotspeed } + bank { turretclass std turretsize medium + mount { node MediumGunNode01 min/max_azimuth min/max_inclination } } +} +``` +- `section_type`: command (164), mission (502), engine (108). => the 3-part hull. +- `section_class`: destroyer / cruiser / dreadnought (the 3 hull sizes). Name + prefixes DE/CR/DN encode class. +- Engine sections add `ftlspeed`, `nodespeed`, `engine_techera`, thrusters. +- `bank`/`mount` = hardpoints (turret class+size + gimbal limits) => this is the + data that constrains which `.weapon` can be fitted where. + +### 4.4 Turrets — `Weapons/_turrets.txt` (positional table) +`size weapon-size class health track-speed azimuth% inclination% "model.x"` +Cross-references weapon `turretsize`/`turretclass` and section `bank` slots. + +### 4.5 Races / species +- `Data/species.txt` — per-race scalars: `SENSORMOD_`, `SPECIESCOLOR_`. +- Deeper per-race behavior is expressed *distributed*: their `sections/` catalogs, + the `Human:/Zuul:/...` columns in the tech tree, `affinity_*` AI tables, and + locale `Desc.txt`. There is no single "race stats" file — race identity is + emergent from these tables. +- `Data/Strategy/playercolors.txt`, `starcolors.txt`, `systemnames.txt` — cosmetic/naming. + +### 4.6 Planets / economy +- **Economy tuning** lives in flat KEY-value tables, not per-planet rows: + `Data/Strategy/StrategyVars.txt` (117 lines: bankruptcy, trade routes + `TRADE_ROUTE_REQ_*`, slaves `SLAVES_*`, system bonuses, starmap sizing), + `Data/globals.txt` (435 lines, mostly UI but some sim), `Data/Combat/planet.txt` + (planetary-defense combat params). +- `Models/Planets/PlanetResources.script` — per-race x planet-quality (Ideal/…) -> + texture sets; visual, not economic. +- `Data/Strategy/RealSpace.csv` — `system,x,y,z,Name` real-star coordinates (~348). +- Planet *economic model* (population growth, output, terraforming) is largely in + code, tuned by the StrategyVars/globals constants — the weakest data coverage. + +### 4.7 AI decision tables — `Data/Strategy/AI/*.csv` +Clean, comment-documented CSVs the AI consults: +- `affinity_section.csv` — `section,species,purpose` (which sections serve which role) +- `affinity_weapon.csv` — `weapon-family,purpose` (e.g. `gauss,reflect_cm`) +- `weapon_replacements.csv` — `weapon,replaces-weapon` final substitution pass +- `aitechpri.csv` — `tech, <6 per-race priorities 1-255>` (lower = research sooner) +- `aitechgrp/aitechmode` , `stock_design_names`, `stock_player_names`, + `rebelai_names`, `stock_diplomacy_messages`, `raider_sections`. + +### 4.8 Scenarios & combat missions +- `Scenarios/*.txt` (18) — `name/desc/objs/rules` (all `@`-tokens), `numplayers`, + `numsystems`, `mapshape`, `script_object`, repeated `player { recommended }`. + Backed by `Scenarios/*_Fleets.csv`, `_FleetTemplates.csv`, `_Techs.csv`, + `_ShipRequests.csv` — data-driven fleet/tech grants for campaigns. +- `CombatMissions/Encounter/{DeepSpace,InSystem,NodeSpace}.combat` — encounter + templates: `summary/combatenv/endcondition{elimination,time}/orbit{type,radius}`. + +### 4.9 Cosmetic catalogs +`Species/ShipBadges.def` (badge {name,w,h,depth,orientation}), `ShipLights.def` +(light {name,effect,period,offset,duration}), `Badges/BadgeTable.txt`, +`Avatars/AvatarTable.txt`, `GUI/SpriteTable.csv`, `Effects/*.effect` (415). + +### 4.10 Localization — `sots_local_en.gob` +`Locale/EN/Strings.csv` (5,722 rows: `Key,String,Size,Notes`) resolves every +`@TOKEN`/`SOTS_*`/`WEAPON_*` in the data. `SpeechEvents.csv`, `Desc.txt`, +`ChatTrans.txt`. Confirms the whole model is name-key indirected for i18n. + +--- + +## 5. Data-defined vs code-defined (map to `Game::` classes) + +| Object model area | Where | Likely class | +|---|---|---| +| Tech graph (nodes, edges, RP costs, race gating, benefits) | **DATA** `MasterTechList.tech` | `Game::TechDef` / TechTree; `TECHBEN_*` enum in code | +| Which ship sections/weapons a tech unlocks | **DATA** (`ship{}` / `requires`) | linkage tables | +| Weapon stats, damage falloff, fire-control | **DATA** `*.weapon` | `Game::WeaponDef` | +| Weapon *flight/behavior* per `weaponclass` | **CODE** | weapon subclasses keyed by `weaponclass` string | +| Ship sections: stats, sockets, hardpoints, thrust | **DATA** `*.shipsection` | `Game::ShipSectionDef` | +| Ship = command+mission+engine assembly rules | **DATA** (socket/section_type) | `Game::ShipDesign` | +| Turret sizes/tracking/models | **DATA** `_turrets.txt` | `Game::TurretDef` | +| Stable net/save IDs | **DATA** `_weapons.txt`/`_shipsections.txt` | id registry | +| Race sensor/color scalars | **DATA** `species.txt` | `Game::Species` | +| Race economy/behavior specifics | **CODE + tuning consts** | per-race code paths | +| Economy: trade/bankruptcy/slaves/system bonus consts | **DATA** `StrategyVars.txt` | global config singleton | +| Planet growth/output/terraform simulation | **CODE** (tuned by consts) | `Game::Planet` | +| Star map coords / names | **DATA** `RealSpace.csv`, `systemnames.txt` | map generator | +| AI weapon/section/tech preferences | **DATA** `Strategy/AI/*.csv` | AI advisor | +| Combat sim constants (ship/planet/drone/sensors…) | **DATA** `Data/Combat/*.txt` | combat engine config | +| Scenarios / encounters | **DATA** `.txt`+`.csv` / `.combat` | scenario loader | +| Strings / speech | **DATA** locale gob | string table | + +**Bottom line for reimplementation:** stand up the Mars brace-block parser + the +flat-KV and CSV readers first; you then get the tech tree, all weapons, all ship +sections/turrets, races, scenarios, AI tables and every tuning constant essentially +for free. The remaining code work is *behavior*: weapon flight per `weaponclass`, +the planet/economy simulation (data only supplies its constants), per-race special +logic, and the combat/strategy engines that consume these catalogs.