alex 7139dee219 verify: mars data parsers proven 100% on 1595 files; catalogs; tech-tree graph

2026-09-07 15:22:55 -04:00

27 KiB

Raw Blame History

SOTS1 data-format parsers — report

Round-two deliverable: stdlib-only Python 3 readers for every textual data format shipped in sots.gob / sots_local_en.gob, proven against all 1,665 extracted files, with cross-link checks and normalized JSON artifacts.

Location: spicy:/bulk-storage/re-lab/handoff/parsers/ (this file is its sibling). Input: spicy:/bulk-storage/re-lab/gob-extract/. Python: 3.x, no third-party deps.

python3 parsers/verify.py <gob-extract-dir> <out-dir>     # re-runs everything, prints the appendix below

Headline

1,595 / 1,595 data files parse (100%); the 70 skipped files are HLSL shaders (.fx/.fxh) and prose (Desc<Race>.txt, ChatTrans.txt) — not data.
12 shipsections are syntactically broken in the shipped archive (11 never close their outer {, one has an extra }). They load in-game, so the engine's parser is lenient in exactly two ways: EOF closes all open blocks, and a stray top-level } is ignored. mars_data mirrors that by default and records a warning; strict=True rejects them.
Cross-links are clean where it matters: 0 dangling weapon→tech, section→tech, tech→section, tech→weapon-file, tech→tech, option→tech, bank→weapon-file, allows→tech, AI/scenario CSV→catalog references.
Real inconsistencies found (data bugs / engine-behaviour evidence): keys and identifiers are matched case-insensitively everywhere (25 refs only resolve that way), 10 _shipsections.txt ids point at files that do not exist, 3 NPC weapon name tokens are missing from Strings.csv, 4 Strings.csv keys are duplicated with a trailing space, and the three aitech*.csv AI tables are empty templates.

The readers

`mars_data.py` — the Mars brace-block format

parse(text, typed=True, keep_case=False, strict=False, warnings=None) / parse_file(path). Tokenizer-based (not line-based), so it handles every layout the files use. Grammar as observed:

body   := (NAME '{' body '}' | NAME value | "bare quoted item")*
value  := "quoted" | bareword          comment := // to end of line

Returns plain dicts. Repeated keys become lists (get_list(d, k) normalizes); bare quoted strings inside a block go under "_items"; the file's top level is itself a body. Barewords that look like C numbers (.5, -.8, 7e+8) or true/false are typed; quoted strings never are (numplayers "8" stays a str).

Used for: .weapon (207), .shipsection (875), .tech (1), .combat (3), .def (2), .script (4), and the 24 block-form .txt (scenarios, tutorial, credits, systemnames, skydefs, ctechvars, shipai).

Quirks handled (all present in shipped files):

quirk	where	handling
keys case-insensitive: `Requires`/`requires`, `badge`/`Badge`, `Planet_Impact_Effect`	tech, defs, weapons	keys lower-cased by default
identifier values case-insensitive: `WEP_HCLAS` vs tech `WEP_HCLas`, `Ind_AstMon`, `SECTIONNAME_DEHammerHead` vs file `DEHammerhead`, manifest `DEWar.SHIPSECTION`	18 weapon + 7 section `requires`, string keys, manifests	all cross-links compare `.lower()`
block opens on the same line as a pair: `turretsize small mount {`	`_NPC/_Wreckage_E.shipsection`	tokenizer, not line-based
block name and brace on one line: `weapon {`	MasterTechList.tech (67×)	same
outer block never closed (EOF at depth 1)	11 shipsections (Hiver 6, Liir 4, Morrigi 1)	lenient close + warning
one `}` too many	`Human/CrPropaganda.shipsection`	ignored + warning
`//` inside quoted strings; backslash Windows paths; `""` empty values	globals.txt, GUI scripts	no escape processing, comment scan is quote-aware
bare quoted list items with no key	`Data/Strategy/systemnames.txt`	`_items` list
scalar and block under the same key: `option DRV_PlsmFoc` beside `option { option A ... }`	153 shipsections (229 occurrences)	both merge into one list; consumers accept str-or-dict members
repeated `turretsize`/`turretclass` inside one `bank{}`	19 banks (Zuul DN*, …)	kept as list; verify takes last
mixed CRLF/LF, cp1252 bytes	everywhere	decode cp1252

No /* */ comments, no =, no escapes, no BOMs were found.

`flat_kv.py` — `KEY value` tuning tables and positional rows

parse_kv(text, typed=True, on_dup='last') → {KEY: value}; a value of several unquoted tokens becomes a list; duplicates(text) lists repeated keys (none in shipped files). color("r g b") → tuple. parse_rows(text) → list of token rows for the positional tables (_turrets.txt, _defaultweapons.txt, damfx*.txt, playercolors.txt, BadgeTable.txt, AvatarTable.txt, WeaponIconPlacements.txt). Comment stripping is quote-aware (ENDGAME_FILL_COLOR "0 0 0" // 48 29 2" is a real line).

`manifest.py` — id manifests and CSVs

parse_manifest(text) → Manifest(entries=[(id, filename)], deleted=[ids], problems=[…]); validates id uniqueness, honours // DELETED - n, and by_name() folds case. parse_csv(text) drops #/// comment rows and blank rows, strips cells, honours RFC-4180 quoting (Strings.csv has one multi-line cell and quoted commas; stock_diplomacy_messages.csv quotes every cell including its "# species" header). parse_csv_with_header() recovers a # <a>,<b> schema row when present.

`effect_txt.py` — `Effects/*.effect`

Not brace-block, contrary to the round-one write-up. A TXT magic line, then KEY value lines and KEY + indented BEGIN … END groups. Order is semantic (PARTICLEDATATYPE n precedes the CREATION/VARIATION/OVERLIFE curves that belong to it; MODIFIER repeats once per type), so each level is returned as an ordered [key, value] list; to_dict() gives the unordered view. 415/415 parse; BEGIN/END always balanced.

`verify.py`

Parses every file with the reader its type calls for, computes key-frequency schema stats per block path, runs the cross-link suite, writes the artifacts, and prints the appendix below. Exit code 0 = no parse failures.

Cross-link results (summary; full detail in appendix and `crosslink.json`)

link	refs	dangling	notes
weapon `requires` → tech	189 (177 weapons; 12 list 2+ techs = AND)	0	18 case-only mismatches; 13 player weapons have no `requires` (missiles, mirv warheads, boarding shuttles, spyship, wraith)
shipsection `requires` → tech	970	0	7 case-only (`Ind_AstMon`, `DRN_BTLRdrs`)
shipsection `option`/`optiondef` → tech	3,842	0
tech `ship{section}` → shipsection	149	0	section names are race-agnostic; matched against the union of race catalogs
tech `weapon{filename}` → file	67	0
tech `requires` → tech or `GRP_x`	154	0	9 groups (`group TORPS` ↔ `requires GRP_TORPS`): PRJCTR 4, PD 3, HVYBEAM 3, SPINAL 2, TORPS 12, MINES 7, BIOMISSILE 5, JAMMING 3, SHIELDS 6
tech `allows` edges	354	0	every edge parses to `child RP:n [Race:%…]`; 12 roots (`*_ROOT`) are never a target
`_weapons.txt` ↔ `Weapons/*.weapon`	123 ids, deleted 36/58/59	0 either way	`Species/_NPC/weapons/` (84) has no manifest — referenced by path from NPC `bank{weapon}` (404 refs, 0 dangling)
`_shipsections.txt` ↔ files	885 ids	10 ids with no file	Tarkas ids 3,4,16,32,43,108,109 (`CRAbsorber, CRAIC, CRDeflector, DEAbsorber, DEDeflector, DEDisruptor, CRDisruptor`); `DEWar` in Human 98, Liir 86, Morrigi 39. These are reserved wire ids, consistent with the "never reuse ids" header.
`TECHNAME_/TECHDESC_`	293	0 / 0
`SECTIONNAME_/SECTIONDESC_` (267 stems)		0 / 67	62 of the missing DESC are `_NPC`-only, 5 are hidden player stems (`_assaultshuttle`, `_biomissile`, `_boardingpod`, `_nodemissile`, `_spy`)
every `@TOKEN` in brace files	378	3	`@WEAPON_NPC_REFUGEEDRONE`, `@WEAPON_SILQUEEN_CANNON`, `@WEAPON_VNSYSK_CUTTINGBEAM` (NPC weapons; would render as raw token)
`_turrets.txt` vs weapon/bank (size,class)	42 rows	0 / 0	only when compared case-insensitively (`Large`, `Missile`, `Standard`, `COL`, `PlanetMissile` appear)
`_defaultweapons.txt` → weapon file	31	0
AI `affinity_section/raider_sections` → section, `weapon_replacements` → weapon, `affinity_weapon` → weaponfamily	200 / – / 6 / 9	0
`aitechpri/aitechgrp/aitechmode.csv`	0 rows	–	comment-only templates: AI tech priorities are code-owned
Scenario `Techs.csv` → tech, `FleetTemplates.csv` → section	127 / 87	0

Schema facts worth knowing (from `schema_stats.json`)

weapon: 207 instances, 60 distinct scalar keys; behaviour sub-block is one of bolt(66, always with rangetable), beam(52), torpedo(19, with rangetable), rider(17), missile(13), chainlightning(10), col(9), mine(7), disintegrator(3), grapple, projectedshield, mirv(2 each), nodecannon, siege, mesonprojector, spyship, wraith(1 each). Always present: name weaponclass turretsize turretclass burst_volleys recharge_time.
shipsection: 875 instances, 87 distinct keys; always model health mass; bank(3,721) → mount(7,375); thruster(1,105); netforcelimits(875); option(1,226 block instances); anim(247). section_type values are case-mixed (engine/Engine, command/Command) and 41 sections have none (riders, NPC hulls); section_class has 16 with none.
tech: 293 nodes; keys name(293) allows(354) threat(169) family(155) requires(154) group(45) type(39, always P) option_cost(18) unlock_explicitly(6); sub-blocks ship(71) weapon(67) strategy(61). family is written on only 155 nodes — tech_tree.json adds family_inferred from the name prefix.
Strings.csv: 5,200 data rows → 5,196 keys (4 duplicated via trailing space, identical text). Key prefixes: AIDIP 755, EVENTMSG 415, TECHNAME/TECHDESC 293 each, SECTIONNAME 283, WEAPON 139, TECHBEN 25 …

Artifacts (`parsers/out/`)

file	size	content
`tech_tree.json`	217 KB	293 nodes (name, display name/desc, family, family_inferred, type, threat, group, option_cost, requires, TECHBEN inc/dec, sections, weapons, allows) + 354 edges `{from, to, rp, pct{Race:%}}` + groups
`weapons.json`	345 KB	207 weapons: full parsed body + `stem, file, scope(player/NPC), id, display_name`
`shipsections.json`	2.7 MB	875 sections: full body + `race, stem, id, display_name, description, unlocked_by[]`
`strings.json`	396 KB	5,196 token → text
`schema_stats.json`	33 KB	per block path: instance count, key frequencies, sub-block frequencies
`crosslink.json`	27 KB	machine-readable form of the cross-link findings
`tech_tree.dot`	54 KB	Graphviz digraph, nodes coloured by family, edges labelled RP + race % (no graphviz on this box; `dot -Tsvg` elsewhere)
`report.md`		verbatim `verify.py` output (appended below)

Caveat on edges[].pct: a race absent from an allows string has no override in the data. The default (believed 100%) is engine code, not asserted by the artifact.

Appendix — `verify.py` output

Parse results

reader	file kind	files	parsed
mars_data	brace:combat	3	3
mars_data	brace:def	2	2
mars_data	brace:script	4	4
mars_data	brace:shipsection	875	875
mars_data	brace:tech	1	1
mars_data	brace:txt	24	24
mars_data	brace:weapon	207	207
manifest.parse_csv	csv	28	28
effect_txt	effect	415	415
flat_kv.parse_kv	kv	20	20
manifest.parse_manifest	manifest	8	8
flat_kv.parse_rows	rows	8	8

Total: 1595 parsed, 0 failed (skipped: 70 HLSL/prose files that are not data).

Lenient recoveries (engine-compatible; strict=True would reject these):

Species/Morrigi/sections/DECommand.shipsection: end of file inside block (depth 1)
Species/Liir/sections/CRAbsorber.shipsection: end of file inside block (depth 1)
Species/Liir/sections/CRCommand.shipsection: end of file inside block (depth 1)
Species/Liir/sections/CRFireControl.shipsection: end of file inside block (depth 1)
Species/Liir/sections/CRShield.shipsection: end of file inside block (depth 1)
Species/Human/sections/CrPropaganda.shipsection: line 105: stray '}' at top level
Species/Hiver/sections/DEAntiMatter.shipsection: end of file inside block (depth 1)
Species/Hiver/sections/DEDeflector.shipsection: end of file inside block (depth 1)
Species/Hiver/sections/DEDisruptor.shipsection: end of file inside block (depth 1)
Species/Hiver/sections/DEJammer.shipsection: end of file inside block (depth 1)
Species/Hiver/sections/DEShield.shipsection: end of file inside block (depth 1)
Species/Hiver/sections/DESquadronCnc.shipsection: end of file inside block (depth 1)

Schema stats (key frequency per block type)

Full table in schema_stats.json. Block paths with instance counts and the keys seen in them (count = number of block instances carrying the key):

`weapon.weapon` (207 instances)

keys: name:207, weaponclass:207, turretsize:207, turretclass:207, burst_volleys:207, recharge_time:207, cost:204, icon_file:204, icon_rect:204, muzzle_sound:203, muzzle_sound_minrange:200, muzzle_effect:190, requires:189, dam_est:179, volley_period:176, rating_frate:176, rating_dam:176, rating_acc:176, rating_range:176, fc_requires_los:173, fc_requires_inrange:173, fc_requires_enemycolony:173, fc_manual_target:173, fc_manual_toggle:173, fc_manual_launch:173, fc_controllable:173, fc_holdsfire:173, range:167, range_planet:167, hpbonus:152, model1:143, muzzle_speed:140, weaponfamily:130, volley_duration:107, solution_tolerance:89, turretmodel1:67, trackspeed_mod:55, model2:43, pinpoint:41, buildup_delay:40, blindfire:37, weapondamagetype:21, munitionsize:15, model3:14, fc_explicit_target:14, fc_exclusive_launch:14, fc_targets_expire:14, turretmodel2:14, target_queue_size:13, compatible_section:10, hidden:9, lock_period:4, buildup_sound:3, buildup_sound_minrange:3, exclusive_species:2, secondary_pd:2, turretmodel3:2, buildup_effect:1, centered_muzzle_effect:1, anim:1

sub-blocks: bolt:66, beam:52, torpedo:19, rider:17, missile:13, chainlightning:10, col:9, mine:7, disintegrator:3, grapple:2, projectedshield:2, mirv:2, nodecannon:1, siege:1, mesonprojector:1, spyship:1, wraith:1

`shipsection.shipsection` (875 instances)

keys: requires:970, mass:877, model:875, health:875, section_class:859, section_type:834, cost:831, cpoints:824, crew:644, preview_ofs:549, socket_aft:543, socket_fore:487, option:229, entity_class:189, range:182, engine_sound:178, autonomous:178, ftlspeed:168, dam_model:154, engine_idle_sound:133, design_class:126, engine_sound_minrange:125, engine_techera:115, command_cost:111, maintenance_cost:98, nodespeed:75, node_cannon_fling:68, dam_socket_aft:68, dam_socket_fore:67, section_sound:65, section_sound_repeat:65, nodesign:63, section_lod_type:60, view_dist:60, explicit_section:60, defence_platform:49, death_effect:39, tacticalsensorrange:38, scanrange:35, command_quota:34, station:33, death_area_impact_effect:30, death_dam:30, death_damradius:30, explicit_command_section:30, explicit_engine_section:30, refueling_capacity:21, split_traffic_volume:20, repair_capacity:18, aicontrol:17, rebelai_scanrange:17, rebelai_tacticalsensorrange:17, colonizer_pop:12, colonizer_infra:12, colonizer_terra:12, refinery:12, freighter:10, monitor:8, shield_model_offset:7, construction_capacity:7, huge:6, mining_capacity:6, mining_rate:6, spytender:6, propaganda:6, ewar:6, science:6, spy:6, exclude:6, freighterq:5, police:5, tradingpost:5, prisoner_capacity:4, node_bore:3, section_sound_minrange:3, gravboat_bonus:3, gateship:2, ramscoop:2, construction_devourer:1, construction_refresh:1, construction_salvagerate:1, mining_trap:1, colony_trap:1, imperial_population:1, civilian_population:1, protectorate:1, node_missile:1

sub-blocks: bank:3721, option:1226, thruster:1105, netforcelimits:875, anim:247, optiondef:12

`tech.tech` (293 instances)

keys: allows:354, name:293, threat:169, family:155, requires:154, group:45, type:39, option_cost:18, unlock_explicitly:6

sub-blocks: ship:71, weapon:67, strategy:61

All block paths: combat(3), combat.combatenv(3), combat.endcondition(3), combat.orbit(17), combat.planet(1), combat.summary(3), def(2), def.badge(27), def.light(196), script(4), script.distanceline(1), script.fleetline(1), script.gateline(1), script.hiver(1), script.hiver.dead(1), script.hiver.good(1), script.hiver.ideal(1), script.hiver.ideal_2(1), script.hiver.poor(1), script.human(1), script.human.dead(1), script.human.good(1), script.human.ideal(1), script.human.ideal_2(1), script.human.poor(1), script.liir(1), script.liir.dead(1), script.liir.good(1), script.liir.ideal(1), script.liir.ideal_2(1), script.liir.poor(1), script.morrigi(1), script.morrigi.dead(1), script.morrigi.good(1), script.morrigi.ideal(1), script.morrigi.ideal_2(1), script.morrigi.poor(1), script.moveline(1), script.nodeline(1), script.sprite(11), script.tarkas(1), script.tarkas.dead(1), script.tarkas.good(1), script.tarkas.ideal(1), script.tarkas.ideal_2(1), script.tarkas.poor(1), script.techcolors(1), script.techtreebackground(1), script.techtreemodelinfo(12), script.tradeline(1), script.zuul(1), script.zuul.dead(1), script.zuul.good(1), script.zuul.ideal(1), script.zuul.ideal_2(1), script.zuul.poor(1), shipsection(875), shipsection.shipsection(875), shipsection.shipsection.anim(247), shipsection.shipsection.bank(3721), shipsection.shipsection.bank.mount(7375), shipsection.shipsection.netforcelimits(875), shipsection.shipsection.option(1226), shipsection.shipsection.optiondef(12), shipsection.shipsection.thruster(1105), tech(1), tech.tech(293), tech.tech.ship(71), tech.tech.strategy(61), tech.tech.weapon(67), weapon(207), weapon.weapon(207), weapon.weapon.beam(52), weapon.weapon.bolt(66), weapon.weapon.bolt.rangetable(66), weapon.weapon.chainlightning(10), weapon.weapon.col(9), weapon.weapon.disintegrator(3), weapon.weapon.grapple(2), weapon.weapon.mesonprojector(1), weapon.weapon.mine(7), weapon.weapon.mirv(2), weapon.weapon.missile(13), weapon.weapon.nodecannon(1), weapon.weapon.projectedshield(2), weapon.weapon.rider(17), weapon.weapon.siege(1), weapon.weapon.spyship(1), weapon.weapon.torpedo(19), weapon.weapon.torpedo.rangetable(19), weapon.weapon.wraith(1)

Cross-link results

weapon requires -> tech: 189 refs in 177 weapons (12 weapons list 2+ techs), 0 dangling, 18 case-mismatched; 30 weapons have no requires (NPC: 17, player: ['_mis_mirv_warhead.weapon', '_mis_planet_mirv_warhead.weapon', 'bal_grapple.weapon', 'brd_prisonershuttle.weapon', 'brd_shuttle.weapon', 'brd_tarkahunter.weapon', 'mis.weapon', 'mis_defplat.weapon', 'mis_planet.weapon', 'mis_planet_hvy.weapon', 'mis_planet_mirv.weapon', 'spyship.weapon', 'wraith.weapon']).
- case: Weapons/bem_beamer_uv.weapon requires WEP_UVBmr (tech is WEP_UvBmr)
- case: Weapons/bem_part.weapon requires WEP_prtBm (tech is WEP_PrtBm)
- case: Weapons/bem_part_spinal.weapon requires WEP_prtBm (tech is WEP_PrtBm)
- case: Weapons/can_plasma.weapon requires WEP_plsmcan (tech is WEP_PlsmCan)
- case: Weapons/hvy_bem_hclas.weapon requires WEP_HCLAS (tech is WEP_HCLas)
- case: Weapons/hvy_bem_hclas_free.weapon requires WEP_HCLAS (tech is WEP_HCLas)
- case: Weapons/hvy_bem_lancer.weapon requires WEP_lancer (tech is WEP_Lancer)
- case: Weapons/hvy_bem_lancer_free.weapon requires WEP_lancer (tech is WEP_Lancer)
- case: Weapons/las_green.weapon requires WEP_grnlas (tech is WEP_GrnLas)
- case: Weapons/las_uv.weapon requires WEP_UVlas (tech is WEP_UvLas)
- case: Weapons/las_xray.weapon requires WEP_xrylas (tech is WEP_XryLas)
- case: Weapons/trp_plasma.weapon requires WEP_plsmtrp (tech is WEP_PlsmTrp)
- case: Species/_NPC/weapons/Herald_bem_beamer_uv.weapon requires WEP_UVBmr (tech is WEP_UvBmr)
- case: Species/_NPC/weapons/Refugee_sml.weapon requires WEP_UVBmr (tech is WEP_UvBmr)
- case: Species/_NPC/weapons/SiliciodQueenCannon.weapon requires WEP_plsmcan (tech is WEP_PlsmCan)
- case: Species/_NPC/weapons/SwarmerCannon.weapon requires WEP_plsmcan (tech is WEP_PlsmCan)
- case: Species/_NPC/weapons/Wreckage_bem_part.weapon requires WEP_prtBm (tech is WEP_PrtBm)
- case: Species/_NPC/weapons/Wreckage_las_xray.weapon requires WEP_xrylas (tech is WEP_XryLas)
shipsection requires -> tech: 970 refs, 0 dangling, 7 case-mismatched.
- case: Species/Zuul/sections/_AsteroidMonitor.shipsection requires Ind_AstMon
- case: Species/Tarkas/sections/DNCarrier.shipsection requires DRN_BTLRdrs
- case: Species/Tarkas/sections/_AsteroidMonitor.shipsection requires Ind_AstMon
- case: Species/Morrigi/sections/_AsteroidMonitor.shipsection requires Ind_AstMon
- case: Species/Liir/sections/_AsteroidMonitor.shipsection requires Ind_AstMon
- case: Species/Human/sections/_AsteroidMonitor.shipsection requires Ind_AstMon
- case: Species/Hiver/sections/_AsteroidMonitor.shipsection requires Ind_AstMon
shipsection option{option T}/optiondef -> tech: 3842 refs, 0 dangling. Two forms coexist: option { option A option B } (a mutually-exclusive choice group) and a bare section-level option T (229 occurrences in 153 files, e.g. option DRV_PlsmFoc on engine sections) -- both merge under the key option, so consumers must accept str-or-dict list members.
tech ship{section} -> shipsection: 149 refs, 0 dangling (matched against the union of all race catalogs, case-insensitive).
tech weapon{filename} -> file: 67 refs, 0 dangling.
tech requires -> tech/GRP_: 154 refs, 0 dangling. Groups: {'PRJCTR': 4, 'PD': 3, 'HVYBEAM': 3, 'SPINAL': 2, 'TORPS': 12, 'MINES': 7, 'BIOMISSILE': 5, 'JAMMING': 3, 'SHIELDS': 6}.
tech allows edges: 354, 0 point at unknown techs, 0 unparsable.
techs never allowed by anything (roots/orphans): 12: IND_ROOT, EWP_ROOT, SLD_Root, NRG_Root, TRP_ROOT, WHD_ROOT, BAL_ROOT, DRV_ROOT, BIO_ROOT, CCC_ROOT, DRN_ROOT, XNC_ROOT
duplicate tech names: none
id manifests <-> files:
- Weapons: 123 ids (deleted [36, 58, 59]); listed-but-no-file 0; file-but-unlisted 0
- Human: 145 ids (deleted none); listed-but-no-file 1; file-but-unlisted 0
  - MISSING FILE for id 98: dewar.shipsection
- Zuul: 122 ids (deleted none); listed-but-no-file 0; file-but-unlisted 0
- Hiver: 137 ids (deleted none); listed-but-no-file 0; file-but-unlisted 0
- Tarkas: 139 ids (deleted none); listed-but-no-file 7; file-but-unlisted 0
  - MISSING FILE for id 3: crabsorber.shipsection
  - MISSING FILE for id 4: craic.shipsection
  - MISSING FILE for id 16: crdeflector.shipsection
  - MISSING FILE for id 109: crdisruptor.shipsection
  - MISSING FILE for id 32: deabsorber.shipsection
  - MISSING FILE for id 43: dedeflector.shipsection
  - MISSING FILE for id 108: dedisruptor.shipsection
- Liir: 136 ids (deleted none); listed-but-no-file 1; file-but-unlisted 0
  - MISSING FILE for id 86: dewar.shipsection
- Morrigi: 142 ids (deleted none); listed-but-no-file 1; file-but-unlisted 0
  - MISSING FILE for id 39: dewar.shipsection
- _NPC: 64 ids (deleted none); listed-but-no-file 0; file-but-unlisted 0
- Species/_NPC/weapons/*.weapon (84 files) have no manifest at all; they are referenced by filename from _NPC shipsection bank{weapon} lines.
localization:
- Strings.csv: 5200 data rows -> 5196 keys. 4 keys occur twice because one copy carries a trailing space (parse_csv strips cells; the later row wins):
  - SECTIONNAME_CRGravboat: 'Gravboat' then 'Gravboat'
  - SECTIONNAME_CRFusion_Carver: 'Fusion Void Carver' then 'Fusion Void Carver'
  - SECTIONNAME_CRAntimatter_Carver: 'AM Void Carver' then 'AM Void Carver'
  - SECTIONNAME_CRAntimatter_Mastery: 'AM Void Mastery' then 'AM Void Mastery'
- TECHNAME_/TECHDESC_ for 293 techs: 0 / 0 missing. [] []
- SECTIONNAME_/SECTIONDESC_ for 267 distinct section stems: 0 / 67 missing.
  - missing SECTIONNAME_: 0 are _NPC-only stems (never shown in the design UI); player-race stems: 0 []
  - missing SECTIONDESC_: 62 are _NPC-only stems (never shown in the design UI); player-race stems: 5 ['_assaultshuttle', '_biomissile', '_boardingpod', '_nodemissile', '_spy']
- weapon name @TOKEN: 3 unresolved of 207; 0 weapons carry no name.
  - UNRESOLVED Species/_NPC/weapons/Drone_shuttle.weapon name @WEAPON_NPC_REFUGEEDRONE
  - UNRESOLVED Species/_NPC/weapons/SiliciodQueenCannon.weapon name @WEAPON_SILQUEEN_CANNON
  - UNRESOLVED Species/_NPC/weapons/VNSYSK_Beam.weapon name @WEAPON_VNSYSK_CUTTINGBEAM
- all @TOKEN refs in brace-block files: 378 refs, 3 unresolved.
  - UNRESOLVED Species/_NPC/weapons/Drone_shuttle.weapon @WEAPON_NPC_REFUGEEDRONE
  - UNRESOLVED Species/_NPC/weapons/SiliciodQueenCannon.weapon @WEAPON_SILQUEEN_CANNON
  - UNRESOLVED Species/_NPC/weapons/VNSYSK_Beam.weapon @WEAPON_VNSYSK_CUTTINGBEAM
_turrets.txt (42 rows; size/class values compared case-insensitively -- the data mixes Large/large, Missile/missile, Standard/standard):
- weapon (turretsize,turretclass) pairs with no turret row: none
- section bank (turretsize,turretclass) pairs with no turret row: none
- banks with no turretsize at all (NPC fixed-weapon banks): 2; banks that repeat turretsize/turretclass inside one bank{} (last value taken): 19
shipsection bank{weapon <file>} -> file: 404 refs, 0 dangling.
_defaultweapons.txt: 31 rows, 0 name a missing weapon file.
Data/Strategy/AI/aitechpri.csv: 0 data rows -- the shipped file is a comment-only template (schema documented in its header, no entries); the AI's tech priorities must therefore come from code.
Data/Strategy/AI/aitechgrp.csv: 0 data rows -- the shipped file is a comment-only template (schema documented in its header, no entries); the AI's tech priorities must therefore come from code.
Data/Strategy/AI/aitechmode.csv: 0 data rows -- the shipped file is a comment-only template (schema documented in its header, no entries); the AI's tech priorities must therefore come from code.
AI/affinity_section.csv: 200 rows; unknown sections: none
AI/raider_sections.csv: unknown sections: none
AI/weapon_replacements.csv: 6 rows; unknown weapon stems: none
AI/affinity_weapon.csv: families ['conventionalbeam', 'conventionalmine', 'emitter', 'energycannon', 'gauss', 'heavybeam', 'laser', 'missile', 'torpedo']; not a weaponfamily in any .weapon: none. weaponfamily values in data: {'energycannon': 19, 'heavybeam': 7, 'missile': 14, 'gauss': 27, 'laser': 12, 'conventionalbeam': 21, 'emitter': 9, 'conventionalmine': 6, 'torpedo': 15}
Scenarios/HiverInvasion_FleetTemplates.csv: 81 rows; unknown sections: none
Scenarios/HiverInvasion_Techs.csv: 67 rows; unknown techs: none
Scenarios/UpstartApes_EmpireTechs.csv: 60 rows; unknown techs: none
Scenarios/UpstartApes_GiftFleetTemplates.csv: 6 rows; unknown sections: none

Artifacts

tech_tree.json (225 KB)
weapons.json (345 KB)
shipsections.json (2752 KB)
strings.json (396 KB)
schema_stats.json (33 KB)
crosslink.json (24 KB)
tech_tree.dot (54 KB)
tech_tree.json: 293 nodes, 354 edges; weapons.json: 207; shipsections.json: 875; strings.json: 5196 keys

27 KiB Raw Blame History Unescape Escape

SOTS1 data-format parsers — report

Headline

The readers

mars_data.py — the Mars brace-block format

flat_kv.py — KEY value tuning tables and positional rows

manifest.py — id manifests and CSVs

effect_txt.py — Effects/*.effect

verify.py

Cross-link results (summary; full detail in appendix and crosslink.json)

Schema facts worth knowing (from schema_stats.json)

Artifacts (parsers/out/)

Appendix — verify.py output