diff --git a/campaign/board.md b/campaign/board.md index cbdbbcf..fb24e4a 100644 --- a/campaign/board.md +++ b/campaign/board.md @@ -9,11 +9,11 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`). | `.gob` format | subsystem | mapped | high | 100% | 2026-09-07 | renamed uncompressed ZIP (community-known) | | `Mars::AppStartup` (entry) | control-flow | mapped | high | 100% | 2026-09-07 | entry 0x00925794 -> WinMain 0x0089dd30 -> DemoApp ctor 0x0089c950 -> Mars::Application::Initialize 0x008a0e50 (config/affinity, D3D9, window, sound thread, OnStartup -> net thread). findings/control-flow/turn-spine.md | | main loop / tick dispatch | control-flow | mapped | high | 90% | 2026-09-07 | Application::Run 0x0089f5b0: FrameTimer, PanelManager, DemoApp::OnUpdate 0x00898800 / OnTick 0x0089a640 / OnRender 0x00899210. Turn pipeline: EndTurn 0x00783be0 -> BeginProcessTurn 0x007d98e0 -> StrategyServer::ProcessTurn 0x007dc6c0 -> RunCombatRound 0x007cbe80 -> SETurnResults -> ResumePlaying 0x007ddc90. Lockstep on every machine | -| `Game::ClientPlayer` / `AIPlayer` | object | verified | high | 85% | 2026-09-07 | ServerPlayer layout confirmed vs real saves (names, species, home, savings, designs coherent across turns 1-3). Gaps: none in player block; findings/objects/struct-recovery.md | -| `Game::StarSystem` | object | verified | high | 95% | 2026-09-07 | ServerSystem layout CONFIRMED: the 'conditional VFlags' gap was a reader bug (cp1252 in Name); haltv is a bool; NVO nodes write indi unconditionally. findings/objects/schema-gaps-resolved.md | +| `Game::ClientPlayer` / `AIPlayer` | object | verified | high | 100% | 2026-09-07 | ServerPlayer confirmed strict vs real saves (turns 1-3) | +| `Game::StarSystem` | object | verified | high | 100% | 2026-09-07 | ServerSystem confirmed strict vs real saves; NVO.indi unconditional; haltv bool; cp1252 names | | `Game::Planet` / `DOPlanet` | object | mapped | high | 100% | 2026-09-07 | Planet : Actor is a RENDER actor, not streamed; all colony state is in ServerSystem (+PlayerView in NVs map) | | `Game::ShipDesignDef` / `DesignList` | object | in-progress | low | 30% | 2026-09-07 | R5 breadth: design/assembly rules + validator derived from section/weapon/turret data, tested vs stock designs in real saves (verify/design-rules/); vft@0x009fef64 leaf | -| `Game::FleetLayout` | object | verified | high | 85% | 2026-09-07 | StarFleet/StarShip confirmed vs real saves; gap: PrisonerHold `prisoners` entries lack `PrNSp` (conditional/absent). Round-4 target | +| `Game::FleetLayout` | object | verified | high | 100% | 2026-09-07 | StarFleet/StarShip confirmed strict vs real saves; PrisonerHold PrNSp gated on PrMax>0 | | Mars entity chain | object | mapped | high | 100% | 2026-09-07 | EntityBase@0x00a36fd0 (10) -> PhysicalEntity@0x00a3710c (12) -> TargetableEntity -> Actor@0x009e2114 (36); mixins RigidBody/Body/NetworkObject/HandleObject/IScriptable | | `Game::GameCombatSim` + `CombatCommand*` | subsystem | backlog | — | 0% | 2026-09-07 | tactical combat (~40 command types) | | D3D9 device init | subsystem | backlog | — | 0% | 2026-09-07 | `Mars::DrawDevice`/`Camera` → Direct3DCreate9/CreateDevice | @@ -27,9 +27,9 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`). | networking (SNM/FNM + GameSpy) | subsystem | backlog | — | 0% | 2026-09-07 | ~60 SNM strategy msgs, FNM file-xfer/host-migration; lockstep sim | | battle-load: thread contention | subsystem | in-progress | med | 50% | 2026-09-07 | time-boxed profile running; on landing: record verdict as a finding and DROP from active backlog (not on the reimpl critical path) | | UI screen & flow map | meta | mapped | high | 100% | 2026-09-07 | findings/subsystems/ui-screen-map.md - 36 screens; screens are C++ on Mars controls (NOT data); turn state machine recovered | -| `Game::StrategyServer` (sim block) | object | mapped | high | 95% | 2026-09-07 | member table complete (47 rows) incl. ID-list sub-writer FUN_00794cd0, GOWinPly/sprjs/RNG/creps/turnstats/invasions; PlayerTurnStats + Invasion structs in Ghidra. Verify: strict pass pending reader patch | +| `Game::StrategyServer` (sim block) | object | verified | high | 100% | 2026-09-07 | full member table (47 rows) confirmed vs real saves under strict parse incl. ID lists, GOWinPly, invasions, zdsc pairs, SvSctOb (present in all 3 saves) | | stream primitive API | subsystem | mapped | high | 100% | 2026-09-07 | IStreamable vft: +0x18 string, +0x1c bool, +0x20 float, +0x24 int, +0x28 nested, +0x30 raw; FUN_00816490 = NetworkObject handle id | | real save for verification | verify | verified | high | 100% | 2026-09-07 | 6 saves, 3 distinct turn states in verify/results/saves/ (byte-deterministic per state). Game runs on VM140 via DXVK 3.1 + lavapipe; recipe findings/subsystems/running-the-game.md | -| save_reader.py | verify | verified | high | 90% | 2026-09-07 | parses all 3 real saves end-to-end (padding=joint confirmed); --strict exit 2 with a precise gap list: verify/results/saves/turn2-strict-issues.txt | +| save_reader.py | verify | verified | high | 100% | 2026-09-07 | --strict exit 0 on all 3 real saves; 0 resyncs / 0 hint-failures; only raw = RNG blob (2503 B). 29 tests. SAVE_FORMAT.md confirmed | | Ghidra type write-back | meta | verified | high | 100% | 2026-09-07 | structs saved in project (ServerSystem 87f, ServerPlayer 110f, StarFleet, StarShip, StrategyServer partial, 22 nested); 52 serializers + primitives + ~60 spine fns renamed; decompile shows field names | | strategic turn internals (economy/research/colony/movement/diplomacy) | subsystem | in-progress | — | 0% | 2026-09-07 | R5 breadth: formulas + constant->code map from ServerPlayer/ServerSystem::ProcessTurn, TechTree::ProcessResearch, MoveFleet | diff --git a/campaign/open-questions.md b/campaign/open-questions.md index 3fb30f9..c748ed5 100644 --- a/campaign/open-questions.md +++ b/campaign/open-questions.md @@ -25,3 +25,4 @@ Each links to the finding that raised it. Promoted to backlog or closed by **re- - **Save framing ambiguities (settle on first real save)** — padding joint `[len][name][value][pad]` vs split; bool vs int for names with len%4==0 (no type byte); on-disk tags for Summary/CreateParameters and count/element tags inside framed arrays unknown (positional for now). A real 3-char bool tag (`NPC`,`Dep`,`hsp`) settles padding. (from [[SAVE_FORMAT]]) - **Strict-parse gaps vs real save (verifier output, round 3)** — (1) `ServerSystem`: 7/28 systems carry `VFlags` where `Name`/`vnh` were expected — a conditional layout (uncolonized/special systems?); (2) `halt[].haltv` is a 31/35-byte struct, not a bool; (3) optional `indi` (IndependenceInfo?) precedes `NVE`/`PID` in some systems; (4) `PrisonerHold.prisoners[]` entries lack `PrNSp`; (5) `/createParams/key` is int; (6) `turnstats.hist.stats.tch` is int; (7) optional trailing `zdsi`/`zdst`. Padding = **joint** (settled). 7 resyncs / 2728 raw bytes ≈ RNG blob. (from [[turn2-strict-issues]]) - **RESOLVED: strict-parse gaps** — 5/8 were reader bugs (ASCII plausibility test applied to string values; empty string == int 0), 3 real: `PrNSp` only when `PrMax>0`; `indi` unconditional in `NVO` nodes (system-level gated by `hindi`); `zdsi/zdst` pairs under `zdsc`. `Key` is an empty string; `tch` int. Summary/CreateParams tag lists confirmed. Reader patch in flight. (from [[schema-gaps-resolved]]) +- **RESOLVED: save framing** — padding is joint; `"."` = NULL tag (VectorHelper count+elements, Vector3 bodies); empty strings are 4 zero bytes; `CDT` is a tagged frame + opaque `CD` frames. All three real saves parse `--strict` clean. Note: `SvSctOb` IS present in all saves (patch doc §7 was wrong). (from [[SAVE_FORMAT]]) diff --git a/verify/design-rules/data/HiverInvasion_FleetTemplates.csv b/verify/design-rules/data/HiverInvasion_FleetTemplates.csv new file mode 100644 index 0000000..9efbe38 --- /dev/null +++ b/verify/design-rules/data/HiverInvasion_FleetTemplates.csv @@ -0,0 +1,88 @@ +# fleet-template-name,command,mission,engine,count + +Princess Prime Fleet,DNBattleBridge,DNGuard,DNLongRangeFusion,1 +Princess Prime Fleet,DNAssault,DNBarrage,DNLongRangeFusion,2 +Princess Prime Fleet,CRAssault,CRBarrage,CRLongRangeFusion,5 +Princess Prime Fleet,CRBattleBridge,CRArmor,CRLongRangeFusion,5 +Princess Prime Fleet,CRAssault,CRRefinery,CRLongRangeFusion,4 +Princess Prime Fleet,CRHammerhead,CRBiomeColonizer,CRLongRangeFusion,8 +Princess Prime Fleet,CRHammerhead,CRRepairandSalvage,CRLongRangeFusion,6 +Princess Prime Fleet,CRAssault,CRStrikeForceCNC,CRLongRangeFusion,2 +Princess Prime Fleet,CRDeepScan,CRArmor,CRLongRangeFusion,2 +Princess Prime Fleet,CRAssault,CRGate,CRLongRangeFusion,5 +Princess Prime Fleet,DEHammerhead,DEArmor,DELongRangeFusion,15 +Princess Prime Fleet,DEHammerhead,DETorpedo,DELongRangeFusion,10 +Princess Prime Fleet,DEHammerhead,DESpinalMount,DELongRangeFusion,10 +Princess Prime Fleet,DEHammerhead,DEPointDefence,DELongRangeFusion,5 +Princess Prime Fleet,DEDeepScan,DEJammer,DELongRangeFusion,5 +Princess Prime Fleet,DEHammerhead,DEAssaultShuttle,DELongRangeFusion,15 + +Princess Beta Fleet,DNAssault,DNArmadaCNC,DNLongRangeFusion,1 +Princess Beta Fleet,CRAssault,CRBarrage,CRLongRangeFusion,5 +Princess Beta Fleet,CRBattleBridge,CRArmor,CRLongRangeFusion,5 +Princess Beta Fleet,CRHammerhead,CRArmor,CRLongRangeFusion,5 +Princess Beta Fleet,CRAssault,CRRefinery,CRLongRangeFusion,3 +Princess Beta Fleet,CRHammerhead,CRBiomeColonizer,CRLongRangeFusion,8 +Princess Beta Fleet,CRHammerhead,CRRepairandSalvage,CRLongRangeFusion,4 +Princess Beta Fleet,CRAssault,CRStrikeForceCNC,CRLongRangeFusion,2 +Princess Beta Fleet,CRDeepScan,CRArmor,CRLongRangeFusion,2 +Princess Beta Fleet,CRAssault,CRGate,CRLongRangeFusion,3 +Princess Beta Fleet,DEHammerhead,DEArmor,DELongRangeFusion,18 +Princess Beta Fleet,DEHammerhead,DETorpedo,DELongRangeFusion,10 +Princess Beta Fleet,DEHammerhead,DESpinalMount,DELongRangeFusion,10 +Princess Beta Fleet,DEHammerhead,DEPointDefence,DELongRangeFusion,8 +Princess Beta Fleet,DEDeepScan,DEJammer,DELongRangeFusion,5 +Princess Beta Fleet,DEHammerhead,DEAssaultShuttle,DELongRangeFusion,15 +Princess Beta Fleet,DECommand,DEColonizer,DELongRangeFusion,10 +Princess Beta Fleet,DECommand,DETanker,DELongRangeFusion,5 +Princess Beta Fleet,DEHammerhead,DEGate,DELongRangeFusion,5 + +Guardian Fleet One,CRAssault,CRBarrage,CRLongRangeFusion,3 +Guardian Fleet One,CRBattleBridge,CRArmor,CRLongRangeFusion,3 +Guardian Fleet One,CRHammerhead,CRArmor,CRLongRangeFusion,5 +Guardian Fleet One,CRAssault,CRRefinery,CRLongRangeFusion,3 +Guardian Fleet One,CRHammerhead,CRBiomeColonizer,CRLongRangeFusion,4 +Guardian Fleet One,CRHammerhead,CRRepairandSalvage,CRLongRangeFusion,2 +Guardian Fleet One,CRAssault,CRStrikeForceCNC,CRLongRangeFusion,2 +Guardian Fleet One,CRDeepScan,CRArmor,CRLongRangeFusion,2 +Guardian Fleet One,CRAssault,CRGate,CRLongRangeFusion,3 +Guardian Fleet One,DEHammerhead,DEArmor,DELongRangeFusion,20 +Guardian Fleet One,DEHammerhead,DETorpedo,DELongRangeFusion,10 +Guardian Fleet One,DEHammerhead,DESpinalMount,DELongRangeFusion,10 +Guardian Fleet One,DEHammerhead,DEPointDefence,DELongRangeFusion,5 +Guardian Fleet One,DEDeepScan,DEJammer,DELongRangeFusion,5 +Guardian Fleet One,DEHammerhead,DEAssaultShuttle,DELongRangeFusion,20 +Guardian Fleet One,DECommand,DEColonizer,DELongRangeFusion,10 +Guardian Fleet One,DECommand,DETanker,DELongRangeFusion,10 +Guardian Fleet One,DEHammerhead,DEGate,DELongRangeFusion,5 + +Scout Fleet Alpha,CRHammerhead,CRArmor,CRLongRangeFusion,1 +Scout Fleet Alpha,CRAssault,CRRefinery,CRLongRangeFusion,1 +Scout Fleet Alpha,CRHammerhead,CRRepairandSalvage,CRLongRangeFusion,1 +Scout Fleet Alpha,CRAssault,CRStrikeForceCNC,CRLongRangeFusion,1 +Scout Fleet Alpha,DEHammerhead,DEArmor,DELongRangeFusion,20 +Scout Fleet Alpha,DEHammerhead,DETorpedo,DELongRangeFusion,5 +Scout Fleet Alpha,DEHammerhead,DESpinalMount,DELongRangeFusion,5 +Scout Fleet Alpha,DEHammerhead,DEPointDefence,DELongRangeFusion,3 +Scout Fleet Alpha,DEDeepScan,DEJammer,DELongRangeFusion,5 +Scout Fleet Alpha,DEHammerhead,DEAssaultShuttle,DELongRangeFusion,10 +Scout Fleet Alpha,DECommand,DEColonizer,DELongRangeFusion,10 +Scout Fleet Alpha,DECommand,DETanker,DELongRangeFusion,10 +Scout Fleet Alpha,DEHammerhead,DEGate,DELongRangeFusion,10 + +Mining Fleet One,CRHammerhead,CRStrikeForceCNC,CRLongRangeFusion,1 +Mining Fleet One,CRAssault,CRMining,CRLongRangeFusion,3 +Mining Fleet One,CRBattleBridge,CRMining,CRLongRangeFusion,3 +Mining Fleet One,CRAssault,CRRefinery,CRLongRangeFusion,1 +Mining Fleet One,DEDeepScan,DEJammer,DELongRangeFusion,2 +Mining Fleet One,DEHammerhead,DEGate,DELongRangeFusion,5 + +Explorer Fleet One,CRRamScoop,CRStrikeForceCNC,CRFusion,1 +Explorer Fleet One,CRRamScoop,CRArmor,CRFusion,5 +Explorer Fleet One,CRRamScoop,CRBiomeColonizer,CRFusion,3 +Explorer Fleet One,CRRamScoop,CRRefinery,CRFusion,1 +Explorer Fleet One,DERamScoop,DEArmor,DEFusion,15 +Explorer Fleet One,DERamScoop,DETorpedo,DEFusion,5 +Explorer Fleet One,DERamScoop,DEPointDefence,DEFusion,3 +Explorer Fleet One,DERamScoop,DEGate,DEFusion,10 +Explorer Fleet One,DERamScoop,DEColonizer,DEFusion,4 diff --git a/verify/design-rules/data/UpstartApes_GiftFleetTemplates.csv b/verify/design-rules/data/UpstartApes_GiftFleetTemplates.csv new file mode 100644 index 0000000..636f41e --- /dev/null +++ b/verify/design-rules/data/UpstartApes_GiftFleetTemplates.csv @@ -0,0 +1,12 @@ +# fleet-template-name,command,mission,engine,count + +Gift Fleet A,DECommand,DEArmor,DEFusion,6 + +Gift Fleet B,CRDeepScan,CRExtendedRange,CRFusion,2 + +Gift Fleet C,CRAssault,CRBarrage,CRFusion,2 + +Gift Fleet D,DEHammerHead,DEArmor,DEFusion,12 +Gift Fleet D,DEDeepScan,DESquadronCNC,DEFusion,1 + +Gift Fleet E,CRDeepScan,CRExtendedRange,CRFusion,10 diff --git a/verify/design-rules/data/_defaultweapons.txt b/verify/design-rules/data/_defaultweapons.txt new file mode 100644 index 0000000..ffb8301 --- /dev/null +++ b/verify/design-rules/data/_defaultweapons.txt @@ -0,0 +1,34 @@ +// Selection of default weapons for turrets in the absence of tech and player information. +// + +standard tiny "las_pd.weapon" +standard small "las_red.weapon" +standard medium "las_red.weapon" +standard large "can_plasma.weapon" +missile medium "mis.weapon" +missile large "mis.weapon" +planetmissile large "mis_defplat.weapon" +beam large "hvy_bem_lancer.weapon" +spinal large "bem_part_spinal.weapon" +torpedo large "trp_plasma.weapon" +mine large "min_nuke.weapon" +missilerider large "bio_plague.weapon" +nodemissilerider large "mis_node.weapon" +assaultshuttlerider large "brd_shuttle.weapon" +siege large "bal_siege.weapon" +prisonershuttlerider large "brd_prisonershuttle.weapon" +projector large "can_prjplasma.weapon" +freebeam large "hvy_bem_lancer_free.weapon" +boardingpod large "brd_boardingpod.weapon" +wraith large "wraith.weapon" +nodecannon large "nodecannon.weapon" +dronerider large "brd_drone.weapon" +col large "col_flock.weapon" +spyship large "spyship.weapon" +tarkahunter large "brd_tarkahunter.weapon" +grapple medium "bal_grapple.weapon" +grapple large "bal_grapple.weapon" +strafe small "las_red.weapon" +strafe medium "las_red.weapon" +strafe large "las_red.weapon" +railcannon large "bal_railcannon.weapon" diff --git a/verify/design-rules/data/_turrets.txt b/verify/design-rules/data/_turrets.txt new file mode 100644 index 0000000..0c60e45 --- /dev/null +++ b/verify/design-rules/data/_turrets.txt @@ -0,0 +1,44 @@ +// size weapon-size class health track-speed constrain-azimuth-% constrain-inclination-% "modelname" + +small tiny standard 20 360 1.00 1.00 "turret_pd.x" +small small standard 120 320 1.00 1.00 "turret_s1standard.x" +medium small standard 350 180 1.00 1.00 "turret_m2standard.x" +large small standard 800 120 1.00 1.00 "turret_l3standard.x" +medium medium standard 500 140 1.00 1.00 "turret_m1standard.x" +large medium standard 950 100 1.00 1.00 "turret_l2standard.x" +large large standard 1400 80 1.00 1.00 "turret_l1standard.x" +small small strafe 120 320 1.00 1.00 "turret_s1standard.x" +medium small strafe 350 180 1.00 1.00 "turret_m2standard.x" +large small strafe 800 120 1.00 1.00 "turret_l3standard.x" +medium medium strafe 500 140 1.00 1.00 "turret_m1standard.x" +large medium strafe 950 100 1.00 1.00 "turret_l2standard.x" +large large strafe 1400 80 1.00 1.00 "turret_l1standard.x" +small small grapple 120 320 1.00 1.00 "turret_s1standard.x" +medium small grapple 350 180 1.00 1.00 "turret_m2standard.x" +large small grapple 800 120 1.00 1.00 "turret_l3standard.x" +medium medium grapple 500 140 1.00 1.00 "turret_m1standard.x" +large medium grapple 950 100 1.00 1.00 "turret_l2standard.x" +large large grapple 1400 80 1.00 1.00 "turret_l1standard.x" +medium medium missile 400 100 1.00 0.25 "turret_m1missile.x" +large medium missile 950 100 1.00 0.25 "turret_l2missile.x" +large large missile 1000 100 1.00 0.25 "turret_l1missile.x" +large medium planetmissile 1000 100 1.00 1.00 "turret_l2icbm.x" +large large beam 1500 80 1.00 1.00 "" +large large freebeam 1700 20 1.00 1.00 "turret_l1freebeam.x" +large large spinal 800 100 1.00 1.00 "" +large large torpedo 800 100 1.00 1.00 "" +large large mine 800 100 1.00 1.00 "" +large large missilerider 800 100 1.00 1.00 "" +large large assaultshuttlerider 800 100 1.00 1.00 "" +large large prisonershuttlerider 800 100 1.00 1.00 "" +large large nodemissilerider 800 100 1.00 1.00 "" +large large siege 800 100 1.00 1.00 "" +large large projector 2600 100 1.00 1.00 "turret_l1projector.x" +large large boardingpod 800 100 1.00 1.00 "" +large large wraith 800 100 1.00 1.00 "" +large large nodecannon 800 100 1.00 1.00 "" +large large dronerider 800 100 1.00 1.00 "" +large large col 800 100 1.00 1.00 "" +large large spyship 800 100 1.00 1.00 "" +large large railcannon 1600 100 1.00 1.00 "" +large large tarkahunter 800 100 1.00 1.00 "" diff --git a/verify/design-rules/design_rules.py b/verify/design-rules/design_rules.py new file mode 100644 index 0000000..147d4d8 --- /dev/null +++ b/verify/design-rules/design_rules.py @@ -0,0 +1,757 @@ +"""design_rules.py -- SOTS1 (2006) ship-design assembly rules, derived from data. + +Stdlib only. Loads the normalized catalogs written by verify/parsers/verify.py +(shipsections.json, weapons.json, tech_tree.json) plus the two positional +tables that live next to this file (data/_turrets.txt, data/_defaultweapons.txt) +and exposes: + + cat = Catalog.load() # default paths, cached + validate(design, cat=None) -> [Violation] + derive_stats(design, cat=None) -> dict + applied_techs(section_use, sdef, cat) # == the save's DOpts list + design_from_save(species_idx, slots, ..) # save-reader record -> Design + +Design (plain dict; every key optional except race/mission): + + { + "name": "Armor", + "race": "Human", # catalog species (Human/Hiver/Tarkas/Liir/Zuul/Morrigi/_NPC) + "hidden": False, # engine-generated hidden design (rider defaults) + "command": {"section": "DECommand", "weapons": ["bal_gauss"], "options": []}, + "mission": {"section": "DEArmor", "weapons": ["bal_gauss", "bal_gauss", "mis"], "options": []}, + "engine": {"section": "DEFission", "weapons": ["bal_gauss"], "options": []}, + "known_techs": ["DRV_Fissn", ...], # omit/None -> tech gating is skipped + } + +`weapons` holds one entry per `bank{}` block of the section, in file order; an +entry is a weapon stem ("bal_gauss"), a weapon file path, an int id from +_weapons.txt, or None (empty). `options` lists the chosen option techs. + +Rule ids (see SHIP_DESIGN_RULES.md for the evidence behind each): + A* structure/slots, B* banks & weapon fit, C* tech gating, D* options, + E* derived stats. Violation.level is "error" (design is invalid), + "warn" (valid for the engine but not something the designer UI offers / + something only code can settle) or "info". +""" +from __future__ import annotations + +import csv +import json +import os +import re +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any, Iterable + +__all__ = [ + "Catalog", "Violation", "validate", "derive_stats", "applied_techs", + "design_from_save", "weapon_fits_bank", "SPECIES_INDEX", "SLOTS", + "HULL_CLASS_TECH", "BANK_CLASS_ACCEPTS", +] + +HERE = Path(__file__).resolve().parent +DEFAULT_CATALOG_DIR = HERE.parent / "results" / "data-catalogs" +DEFAULT_DATA_DIR = HERE / "data" + +# Order of the species table in a real save (sim.species ISsp list); the +# design record's DSec[0] is an index into it. +SPECIES_INDEX = ["Human", "Hiver", "Tarkas", "Liir", "_NPC", "Zuul", "Morrigi"] + +SLOTS = ("command", "mission", "engine") + +# Hull-size techs implied by section_class (evidence: the save's DOpts list +# carries them for every cruiser/dreadnought section that is not a station). +HULL_CLASS_TECH = {"cruiser": "IND_CruisCon", "dreadnought": "IND_DreadCon"} + +# Which weapon turretclass a bank turretclass accepts besides an exact match. +# standard <- missile : stock designs put `mis` (medium/missile) in the +# medium/standard bank of DEArmor and DEDefencePlatform +# standard <- grapple : bal_grapple/bal_disruptorwhip are turretclass grapple +# but no section has a grapple bank; _turrets.txt has +# grapple rows for medium/large only +# strafe <- standard: no weapon has turretclass strafe; the NPC +# _HeraldDefender strafe bank carries a standard-class +# beamer; _turrets.txt strafe rows mirror standard rows +BANK_CLASS_ACCEPTS = { + "standard": {"standard", "missile", "grapple"}, + "strafe": {"standard"}, +} + + +# -------------------------------------------------------------------------- +# helpers + +def _last(v): + """Scalar from a catalog value; repeated keys were folded into lists.""" + if isinstance(v, list): + return v[-1] if v else None + return v + + +def _list(v) -> list: + if v is None: + return [] + return v if isinstance(v, list) else [v] + + +def _low(v) -> str | None: + return None if v is None else str(v).lower() + + +def _num(v, default=0): + v = _last(v) + if isinstance(v, bool): + return default + return v if isinstance(v, (int, float)) else default + + +# -------------------------------------------------------------------------- +# catalog model + +@dataclass +class BankDef: + index: int + size: str | None # small/medium/large (lower) + cls: str | None # standard/missile/... (lower) + mounts: int + fixed_weapon: str | None # NPC banks: weapon file bound in data + showturrets: bool = True + + +@dataclass +class OptionGroup: + techs: tuple # mutually exclusive choices, file order + kind: str = "option" # "option" | "optiondef" + + +@dataclass +class SectionDef: + race: str + stem: str + file: str + id: int | None + type: str | None # command/mission/engine (lower) or None + cls: str | None # destroyer/cruiser/dreadnought (lower) or None + socket_fore: str | None + socket_aft: str | None + requires: list + option_groups: list + banks: list + raw: dict + nodesign: bool = False + explicit_section: bool = False + explicit_command_section: str | None = None + explicit_engine_section: str | None = None + exclude: list = field(default_factory=list) + design_class: str | None = None + entity_class: str | None = None + + @property + def has_sockets(self) -> bool: + return self.socket_fore is not None or self.socket_aft is not None + + @property + def is_station(self) -> bool: + return _low(self.design_class) == "station" + + def hull_class_tech(self) -> str | None: + if self.is_station: + return None + return HULL_CLASS_TECH.get(self.cls or "") + + def stat(self, key, default=None): + return _last(self.raw.get(key, default)) + + +@dataclass +class WeaponDef: + stem: str + file: str + scope: str # player / NPC + id: int | None + size: str | None + cls: str | None + weaponclass: str | None + requires: list + cost: float + exclusive_species: str | None + compatible_section: list + hidden: bool + raw: dict + + +@dataclass +class TurretRow: + size: str + weapon_size: str + cls: str + health: float + track_speed: float + az: float + inc: float + model: str + + +@dataclass +class Violation: + rule: str + level: str + msg: str + slot: str | None = None + bank: int | None = None + + def __str__(self): + where = f" [{self.slot}{'' if self.bank is None else ' bank ' + str(self.bank)}]" if self.slot else "" + return f"{self.level.upper():5} {self.rule}{where}: {self.msg}" + + +class Catalog: + _cache: dict = {} + + def __init__(self, sections_json, weapons_json, tech_json, turrets_txt, defaults_txt): + self.sections: dict[str, dict[str, SectionDef]] = {} # race -> stem.lower -> def + self.sections_by_id: dict[str, dict[int, SectionDef]] = {} + self.weapons: dict[str, WeaponDef] = {} # stem.lower -> def + self.weapons_by_file: dict[str, WeaponDef] = {} + self.weapons_by_id: dict[int, WeaponDef] = {} + self.techs: dict[str, dict] = {} # name.lower -> node + self.tech_names: dict[str, str] = {} # name.lower -> canonical + self.groups: dict[str, set] = {} # group.lower -> {tech.lower} + self.unobtainable: dict[str, set] = {} # tech.lower -> {race.lower} + self.turrets: dict[tuple, TurretRow] = {} # (size, wsize, cls) -> row + self.default_weapons: dict[tuple, str] = {} # (cls, size) -> weapon file + self._load_sections(sections_json) + self._load_weapons(weapons_json) + self._load_techs(tech_json) + self._load_turrets(turrets_txt) + self._load_defaults(defaults_txt) + + # -- loading ------------------------------------------------------------ + @classmethod + def load(cls, catalog_dir=None, data_dir=None) -> "Catalog": + catalog_dir = Path(catalog_dir or os.environ.get("SOTS_CATALOG_DIR") or DEFAULT_CATALOG_DIR) + data_dir = Path(data_dir or DEFAULT_DATA_DIR) + key = (str(catalog_dir), str(data_dir)) + if key not in cls._cache: + cls._cache[key] = cls( + catalog_dir / "shipsections.json", catalog_dir / "weapons.json", + catalog_dir / "tech_tree.json", data_dir / "_turrets.txt", + data_dir / "_defaultweapons.txt") + return cls._cache[key] + + def _load_sections(self, path): + data = json.load(open(path, encoding="utf-8")) + for s in data["sections"]: + groups = [] + for o in _list(s.get("option")): + if isinstance(o, dict): + groups.append(OptionGroup(tuple(str(t) for t in _list(o.get("option"))))) + else: + groups.append(OptionGroup((str(o),))) + for o in _list(s.get("optiondef")): + if isinstance(o, dict): + groups.append(OptionGroup(tuple(str(t) for t in _list(o.get("option"))), "optiondef")) + banks = [] + for i, b in enumerate(_list(s.get("bank"))): + if not isinstance(b, dict): + continue + st = _last(b.get("showturrets")) + banks.append(BankDef( + index=i, size=_low(_last(b.get("turretsize"))), cls=_low(_last(b.get("turretclass"))), + mounts=len(_list(b.get("mount"))), fixed_weapon=_last(b.get("weapon")), + showturrets=(st is None or bool(st)))) + sd = SectionDef( + race=s["race"], stem=s["stem"], file=s["file"], id=s.get("id"), + type=_low(_last(s.get("section_type"))), cls=_low(_last(s.get("section_class"))), + socket_fore=_last(s.get("socket_fore")), socket_aft=_last(s.get("socket_aft")), + requires=[str(r) for r in _list(s.get("requires"))], option_groups=groups, banks=banks, raw=s, + nodesign=bool(_last(s.get("nodesign", False))), + explicit_section=bool(_last(s.get("explicit_section", False))), + explicit_command_section=_last(s.get("explicit_command_section")), + explicit_engine_section=_last(s.get("explicit_engine_section")), + exclude=[str(x) for x in _list(s.get("exclude"))], + design_class=_last(s.get("design_class")), entity_class=_last(s.get("entity_class"))) + self.sections.setdefault(sd.race, {})[sd.stem.lower()] = sd + if sd.id is not None: + self.sections_by_id.setdefault(sd.race, {})[int(sd.id)] = sd + + def _load_weapons(self, path): + data = json.load(open(path, encoding="utf-8")) + for w in data["weapons"]: + wd = WeaponDef( + stem=w["stem"], file=w["file"], scope=w.get("scope", "player"), id=w.get("id"), + size=_low(_last(w.get("turretsize"))), cls=_low(_last(w.get("turretclass"))), + weaponclass=_low(_last(w.get("weaponclass"))), + requires=[str(r) for r in _list(w.get("requires"))], cost=_num(w.get("cost"), 0), + exclusive_species=_low(_last(w.get("exclusive_species"))), + compatible_section=[str(c) for c in _list(w.get("compatible_section"))], + hidden=bool(_last(w.get("hidden", False))), raw=w) + self.weapons[wd.stem.lower()] = wd + self.weapons_by_file[wd.file.lower()] = wd + self.weapons_by_file[os.path.basename(wd.file).lower()] = wd + if wd.id is not None: + self.weapons_by_id[int(wd.id)] = wd + + def _load_techs(self, path): + data = json.load(open(path, encoding="utf-8")) + self.races = list(data.get("races", [])) + for n in data["nodes"]: + self.techs[n["name"].lower()] = n + self.tech_names[n["name"].lower()] = n["name"] + if n.get("group"): + self.groups.setdefault(str(n["group"]).lower(), set()).add(n["name"].lower()) + for g, members in data.get("groups", {}).items(): + self.groups.setdefault(g.lower(), set()).update(m.lower() for m in members) + # race availability: a tech is unobtainable for a race when every edge + # into it writes 0% for that race (absent race -> engine default, + # believed 100%). + incoming: dict[str, list] = {} + for e in data["edges"]: + incoming.setdefault(e["to"].lower(), []).append(e) + for t, edges in incoming.items(): + for race in self.races: + if all(e.get("pct", {}).get(race) == 0 for e in edges): + self.unobtainable.setdefault(t, set()).add(race.lower()) + + def _load_turrets(self, path): + for row in _rows(path): + if len(row) < 8: + continue + size, wsize, cls = (str(row[0]).lower(), str(row[1]).lower(), str(row[2]).lower()) + self.turrets[(size, wsize, cls)] = TurretRow( + size, wsize, cls, float(row[3]), float(row[4]), float(row[5]), float(row[6]), str(row[7])) + + def _load_defaults(self, path): + if not Path(path).exists(): + return + for row in _rows(path): + if len(row) >= 3: + self.default_weapons[(str(row[0]).lower(), str(row[1]).lower())] = str(row[2]) + + # -- lookups -------------------------------------------------------------- + def race_key(self, race: str) -> str | None: + for r in self.sections: + if r.lower() == str(race).lower(): + return r + return None + + def section(self, race: str, ref) -> SectionDef | None: + r = self.race_key(race) + if r is None or ref is None: + return None + if isinstance(ref, int): + return self.sections_by_id.get(r, {}).get(ref) + ref = str(ref) + stem = os.path.basename(ref) + stem = re.sub(r"\.shipsection$", "", stem, flags=re.I) + return self.sections[r].get(stem.lower()) + + def weapon(self, ref) -> WeaponDef | None: + if ref is None: + return None + if isinstance(ref, int): + return self.weapons_by_id.get(ref) + ref = str(ref) + low = ref.lower() + if low in self.weapons: + return self.weapons[low] + if low in self.weapons_by_file: + return self.weapons_by_file[low] + stem = re.sub(r"\.weapon$", "", os.path.basename(low)) + return self.weapons.get(stem) or self.weapons_by_file.get(stem + ".weapon") + + def tech(self, name: str) -> dict | None: + return self.techs.get(str(name).lower()) + + def tech_known(self, name: str, known: set) -> bool: + """`known` is a set of lower-cased tech names. GRP_ is satisfied by + any known tech of group .""" + low = str(name).lower() + if low.startswith("grp_"): + return bool(self.groups.get(low[4:], set()) & known) + return low in known + + +def _rows(path) -> list[list[str]]: + """Whitespace-positional rows with // comments and quoted tokens.""" + out = [] + tok = re.compile(r'"([^"]*)"|(\S+)') + for raw in Path(path).read_bytes().decode("cp1252").splitlines(): + line = _strip_comment(raw).strip() + if not line: + continue + out.append([m.group(1) if m.group(1) is not None else m.group(2) for m in tok.finditer(line)]) + return out + + +def _strip_comment(line: str) -> str: + in_q = False + for i, c in enumerate(line): + if c == '"': + in_q = not in_q + elif c == "/" and not in_q and line.startswith("//", i): + return line[:i] + return line + + +# -------------------------------------------------------------------------- +# turret fit + +def turret_row(cat: Catalog, bank: BankDef, weapon: WeaponDef) -> TurretRow | None: + """The _turrets.txt row a weapon in a bank would use: keyed by the weapon's + class first (missile-in-standard picks the missile turret model), then by + the bank's class (strafe banks have their own rows, no weapon is class + strafe).""" + for cls in (weapon.cls, bank.cls): + r = cat.turrets.get((bank.size, weapon.size, cls)) + if r: + return r + return None + + +def weapon_fits_bank(cat: Catalog, bank: BankDef, weapon: WeaponDef) -> tuple[bool, str]: + if bank.size is None or bank.cls is None: + return (weapon.file.lower() == str(bank.fixed_weapon or "").lower(), + "bank has no turretsize/turretclass (fixed NPC weapon only)") + if not (weapon.cls == bank.cls or weapon.cls in BANK_CLASS_ACCEPTS.get(bank.cls, ())): + return False, f"turretclass {weapon.cls} not accepted by {bank.cls} bank" + if turret_row(cat, bank, weapon) is None: + return False, f"no _turrets.txt row for ({bank.size} bank, {weapon.size} weapon, {weapon.cls}/{bank.cls})" + return True, "ok" + + +# -------------------------------------------------------------------------- +# design access + +def _slot(design: dict, slot: str) -> dict | None: + v = design.get(slot) + if v is None: + return None + if isinstance(v, str): + return {"section": v} + return v + + +def _known_set(design: dict) -> set | None: + k = design.get("known_techs") + if k is None: + return None + return {str(t).lower() for t in k} + + +def applied_techs(use: dict, sdef: SectionDef, cat: Catalog | None = None) -> list[str]: + """The techs a section instance is built with -- exactly what the save + stores per section in DOpts: hull-class tech (cruiser/dreadnought, not + stations), then `requires` in file order, then the chosen option techs in + option-group order.""" + out = [] + hct = sdef.hull_class_tech() + if hct: + out.append(hct) + out.extend(sdef.requires) + chosen = {str(o).lower(): str(o) for o in _list((use or {}).get("options"))} + for g in sdef.option_groups: + for t in g.techs: + if t.lower() in chosen: + out.append(chosen[t.lower()]) + return out + + +# -------------------------------------------------------------------------- +# validation + +def validate(design: dict, cat: Catalog | None = None) -> list[Violation]: + cat = cat or Catalog.load() + v: list[Violation] = [] + race = design.get("race") + hidden = bool(design.get("hidden")) + known = _known_set(design) + rk = cat.race_key(race) if race else None + if rk is None: + v.append(Violation("A5", "error", f"unknown species catalog {race!r}")) + return v + + # -- resolve sections ----------------------------------------------------- + uses: dict[str, dict | None] = {s: _slot(design, s) for s in SLOTS} + defs: dict[str, SectionDef | None] = {} + for slot in SLOTS: + use = uses[slot] + if use is None: + defs[slot] = None + continue + sd = cat.section(rk, use.get("section")) + if sd is None: + v.append(Violation("A5", "error", f"section {use.get('section')!r} not in {rk} catalog", slot)) + defs[slot] = sd + if defs["mission"] is None and uses["mission"] is not None: + return v + m = defs["mission"] + if m is None: + v.append(Violation("A2", "error", "mission slot is empty")) + return v + + # A3 hull ship vs standalone + standalone = not m.has_sockets + if m.explicit_command_section or m.explicit_engine_section: + for slot, want in (("command", m.explicit_command_section), ("engine", m.explicit_engine_section)): + got = defs[slot] + if want and (got is None or got.stem.lower() != want.lower()): + v.append(Violation("A8", "error", f"{m.stem} requires {slot} section {want}, got {got.stem if got else None}", slot)) + elif standalone: + for slot in ("command", "engine"): + if uses[slot] is not None: + v.append(Violation("A3", "error", f"{m.stem} has no sockets (standalone hull) but {slot} slot is filled", slot)) + else: + for slot in ("command", "engine"): + if defs[slot] is None: + v.append(Violation("A3", "error", f"{m.stem} has sockets; {slot} section required", slot)) + + present = [(s, d) for s, d in defs.items() if d is not None] + + # A4 slot/type, A9 nodesign, A8 explicit partner sections + for slot, sd in present: + if sd.type is None: + if slot == "mission" and standalone and (hidden or _low(sd.design_class) == "rider"): + v.append(Violation("A4", "info", f"{sd.stem} has no section_type (rider hull in a hidden design)", slot)) + else: + v.append(Violation("A4", "error", f"{sd.stem} has no section_type and cannot fill the {slot} slot", slot)) + elif sd.type != slot: + v.append(Violation("A4", "error", f"{sd.stem} is a {sd.type} section, placed in {slot} slot", slot)) + if sd.nodesign and not hidden: + v.append(Violation("A9", "warn", f"{sd.stem} is nodesign (not offered by the design UI)", slot)) + if sd.explicit_section and slot != "mission": + owner = m.explicit_command_section if slot == "command" else m.explicit_engine_section + if not owner or owner.lower() != sd.stem.lower(): + v.append(Violation("A8", "error", f"{sd.stem} is an explicit_section usable only with its owning station", slot)) + + # A6 class equality + classes = {sd.cls for _, sd in present if sd.cls} + if len(classes) > 1: + v.append(Violation("A6", "error", f"mixed section_class: {sorted(classes)}")) + + # A7 exclude + stems = {sd.stem.lower(): slot for slot, sd in present} + for slot, sd in present: + for ex in sd.exclude: + if ex.lower() in stems: + v.append(Violation("A7", "error", f"{sd.stem} excludes {ex} (in {stems[ex.lower()]} slot)", slot)) + + # -- banks & weapons --------------------------------------------------------- + for slot, sd in present: + use = uses[slot] or {} + weapons = _list(use.get("weapons")) + if len(weapons) != len(sd.banks): + v.append(Violation("B1", "error", f"{sd.stem} has {len(sd.banks)} banks, design lists {len(weapons)} weapons", slot)) + for i, bank in enumerate(sd.banks): + ref = weapons[i] if i < len(weapons) else None + wd = cat.weapon(ref) if ref is not None else None + if ref is not None and wd is None: + v.append(Violation("B5", "error", f"unknown weapon {ref!r}", slot, i)) + continue + if bank.fixed_weapon: + fixed = cat.weapon(bank.fixed_weapon) + if wd is None: + wd = fixed + elif fixed is not None and wd.file.lower() != fixed.file.lower(): + v.append(Violation("B3", "error", f"bank is bound to {os.path.basename(bank.fixed_weapon)}, design has {wd.stem}", slot, i)) + continue + if wd is None: + v.append(Violation("B2", "error", f"bank {i} ({bank.size} {bank.cls}, {bank.mounts} mounts) has no weapon", slot, i)) + continue + ok, why = weapon_fits_bank(cat, bank, wd) + if not ok: + v.append(Violation("B4", "error", f"{wd.stem} ({wd.size} {wd.cls}) does not fit bank {i} ({bank.size} {bank.cls}): {why}", slot, i)) + if wd.scope != "player" and rk != "_NPC" and not bank.fixed_weapon: + v.append(Violation("B5", "error", f"{wd.stem} is an NPC-only weapon", slot, i)) + if wd.exclusive_species and wd.exclusive_species != rk.lower(): + v.append(Violation("B6", "error", f"{wd.stem} is exclusive to {wd.exclusive_species}", slot, i)) + if wd.hidden and not hidden and not bank.fixed_weapon: + v.append(Violation("B8", "error", f"{wd.stem} is a hidden (engine-only) weapon", slot, i)) + if wd.compatible_section: + if not any(cat.section(rk, cs) for cs in wd.compatible_section): + v.append(Violation("B7", "error", f"{wd.stem} needs a rider section {wd.compatible_section} that {rk} lacks", slot, i)) + # tech gating for weapons + if known is not None and not bank.fixed_weapon: + for t in wd.requires: + if not cat.tech_known(t, known): + v.append(Violation("C2", "error", f"{wd.stem} requires {t}", slot, i)) + + # -- options ---------------------------------------------------------------- + for slot, sd in present: + use = uses[slot] or {} + chosen = [str(o) for o in _list(use.get("options"))] + seen_groups: dict[int, str] = {} + for o in chosen: + gi = next((i for i, g in enumerate(sd.option_groups) if o.lower() in {t.lower() for t in g.techs}), None) + if gi is None: + v.append(Violation("D1", "error", f"{o} is not an option of {sd.stem}", slot)) + elif gi in seen_groups: + v.append(Violation("D1", "error", f"{o} and {seen_groups[gi]} are in the same option group of {sd.stem}", slot)) + else: + seen_groups[gi] = o + if known is not None and cat.tech(o) is not None and not cat.tech_known(o, known): + v.append(Violation("C3", "error", f"option {o} is not researched", slot)) + + # -- tech gating --------------------------------------------------------------- + for slot, sd in present: + for t in sd.requires: + if cat.tech(t) is None and not t.lower().startswith("grp_"): + v.append(Violation("C1", "warn", f"{sd.stem} requires unknown tech {t}", slot)) + elif known is not None and not cat.tech_known(t, known): + v.append(Violation("C1", "error", f"{sd.stem} requires {t}", slot)) + if rk.lower() in cat.unobtainable.get(t.lower(), ()): + v.append(Violation("C5", "error", f"{sd.stem} requires {t}, which {rk} can never research", slot)) + hct = sd.hull_class_tech() + if hct and known is not None and not cat.tech_known(hct, known): + v.append(Violation("C4", "error", f"{sd.stem} is a {sd.cls} section; {hct} not researched", slot)) + return v + + +# -------------------------------------------------------------------------- +# derived stats + +_CAPACITY_KEYS = ( + "refueling_capacity", "repair_capacity", "construction_capacity", "mining_capacity", "mining_rate", + "prisoner_capacity", "colonizer_pop", "colonizer_infra", "colonizer_terra", "scanrange", + "tacticalsensorrange", "range", "split_traffic_volume", "refinery", "freighter", "freighterq", + "police", "tradingpost", "monitor", "spy", "spytender", "science", "propaganda", "ewar", "gateship", + "ramscoop", "node_bore", "node_missile", "gravboat_bonus", "protectorate", "huge", "defence_platform", +) + +_OPTION_COST_NOTE = ("cost multiplies section cost by the product of the chosen options' " + "tech option_cost values (assumption -- option_cost is the only cost-shaped " + "field on option techs; the multiplier's base and stacking order are code)") + + +def derive_stats(design: dict, cat: Catalog | None = None) -> dict: + cat = cat or Catalog.load() + rk = cat.race_key(design.get("race", "")) or design.get("race") + out: dict[str, Any] = { + "race": rk, "name": design.get("name"), "sections": {}, "hull_class": None, + "mass": 0.0, "section_cost": 0.0, "section_cost_with_options": 0.0, "weapon_cost_per_bank": 0.0, + "weapon_cost_per_mount": 0.0, "health_total": 0.0, "crew": 0, "cpoints": 0.0, "turrets": 0, + "banks": 0, "command_cost": 0, "command_quota": None, "maintenance_cost": 0, + "ftlspeed": None, "nodespeed": None, "engine_techera": None, "netforcelimits": None, + "capacities": {}, "applied_techs": {}, "notes": [ + "health is tracked per section by the engine (ShipHealth = 3 floats); health_total is only a sum", + _OPTION_COST_NOTE, + "weapon cost: one turret per mount is the visual model; whether cost is per bank or per mount is code (both reported)", + "tactical speed/turn: engine section netforcelimits are reported; the combat integrator (mass vs force) is code", + ], + } + for slot in SLOTS: + use = _slot(design, slot) + if use is None: + continue + sd = cat.section(rk, use.get("section")) + if sd is None: + continue + opt_mult = 1.0 + for o in _list(use.get("options")): + n = cat.tech(o) + if n and n.get("option_cost") is not None: + opt_mult *= float(n["option_cost"]) + sec = { + "section": sd.stem, "type": sd.type, "class": sd.cls, "mass": _num(sd.stat("mass")), + "cost": _num(sd.stat("cost")), "option_cost_multiplier": opt_mult, "health": _num(sd.stat("health")), + "crew": _num(sd.stat("crew")), "cpoints": _num(sd.stat("cpoints")), "banks": [], + } + weapons = _list(use.get("weapons")) + for i, bank in enumerate(sd.banks): + ref = weapons[i] if i < len(weapons) else None + wd = cat.weapon(ref) if ref is not None else (cat.weapon(bank.fixed_weapon) if bank.fixed_weapon else None) + row = turret_row(cat, bank, wd) if wd else None + sec["banks"].append({ + "size": bank.size, "class": bank.cls, "mounts": bank.mounts, "weapon": wd.stem if wd else None, + "weapon_cost": wd.cost if wd else 0, "turret_model": row.model if row else None, + "turret_health": row.health if row else None, "turret_track_speed": row.track_speed if row else None, + }) + out["banks"] += 1 + out["turrets"] += bank.mounts + if wd: + out["weapon_cost_per_bank"] += wd.cost + out["weapon_cost_per_mount"] += wd.cost * max(bank.mounts, 1) + out["sections"][slot] = sec + out["mass"] += sec["mass"] + out["section_cost"] += sec["cost"] + out["section_cost_with_options"] += sec["cost"] * opt_mult + out["health_total"] += sec["health"] + out["crew"] += int(sec["crew"] or 0) + out["cpoints"] += sec["cpoints"] + out["command_cost"] += int(_num(sd.stat("command_cost"))) + out["maintenance_cost"] += int(_num(sd.stat("maintenance_cost"))) + if sd.stat("command_quota") is not None: + out["command_quota"] = (out["command_quota"] or 0) + int(_num(sd.stat("command_quota"))) + out["hull_class"] = out["hull_class"] or sd.cls + out["applied_techs"][slot] = applied_techs(use, sd, cat) + for k in _CAPACITY_KEYS: + if k in sd.raw: + out["capacities"][k] = _last(sd.raw[k]) + if sd.type == "engine" or (slot == "mission" and out["ftlspeed"] is None and sd.stat("ftlspeed") is not None): + out["ftlspeed"] = sd.stat("ftlspeed") + out["nodespeed"] = sd.stat("nodespeed") + out["engine_techera"] = sd.stat("engine_techera") + out["netforcelimits"] = sd.raw.get("netforcelimits") + out["total_cost_estimate"] = out["section_cost_with_options"] + out["weapon_cost_per_bank"] + return out + + +# -------------------------------------------------------------------------- +# save-record conversion + +def design_from_save(name: str, slots: Iterable[dict], cat: Catalog | None = None, *, + hidden: bool = False, known_techs=None, race: str | None = None) -> dict: + """Build a Design from a save-reader design record. + + `slots` is the DSec list in save order (command, mission, engine, +2 unused), + each as {"species": int, "section_id": int, "weapons": [wid|file|None per bank], + "options": [tech...]} with species/section_id (0, 0) meaning empty. The + species index is resolved through SPECIES_INDEX; if the design mixes + species the result carries "mixed_species": [...] for the validator. + """ + cat = cat or Catalog.load() + d: dict = {"name": name, "hidden": hidden, "race": race} + species_seen = [] + for slot, rec in zip(SLOTS, slots): + if not rec or (rec.get("species", 0) == 0 and rec.get("section_id", 0) == 0): + d[slot] = None + continue + sp = SPECIES_INDEX[rec["species"]] if rec.get("species") is not None else race + species_seen.append(sp) + sd = cat.section(sp, int(rec["section_id"])) + d[slot] = { + "section": sd.stem if sd else f"#{rec['section_id']}", + "weapons": list(rec.get("weapons", [])), + "options": list(rec.get("options", [])), + "save_opts": list(rec.get("options", [])), + } + if sd is not None: + # DOpts holds hull-class tech + requires + chosen options; the + # design's own `options` are the chosen ones only. + structural = {t.lower() for t in sd.requires} + hct = sd.hull_class_tech() + if hct: + structural.add(hct.lower()) + d[slot]["options"] = [o for o in rec.get("options", []) if o.lower() not in structural] + if d["race"] is None: + d["race"] = species_seen[0] if species_seen else "Human" + if len(set(species_seen)) > 1: + d["mixed_species"] = species_seen + if known_techs is not None: + d["known_techs"] = list(known_techs) + return d + + +if __name__ == "__main__": # tiny smoke demo + cat = Catalog.load() + demo = { + "name": "Armor", "race": "Human", + "command": {"section": "DECommand", "weapons": ["bal_gauss"]}, + "mission": {"section": "DEArmor", "weapons": ["bal_gauss", "bal_gauss", "mis"]}, + "engine": {"section": "DEFission", "weapons": ["bal_gauss"]}, + "known_techs": ["DRV_Fissn", "DRV_Node", "WEP_GsDrvr"], + } + for viol in validate(demo, cat) or ["no violations"]: + print(viol) + st = derive_stats(demo, cat) + print({k: st[k] for k in ("mass", "section_cost", "weapon_cost_per_bank", "crew", "ftlspeed", "nodespeed", "turrets")}) diff --git a/verify/results/saves/turn2-strict-issues.txt b/verify/results/saves/turn2-strict-issues.txt index 7e4b783..76c7816 100644 --- a/verify/results/saves/turn2-strict-issues.txt +++ b/verify/results/saves/turn2-strict-issues.txt @@ -1,53 +1,6 @@ -# save_reader --strict against real save turn2-state.sav (padding=joint, 7 resyncs, 2728 raw bytes) - 16x [warn] /Sim/turnstats/hist/stats :: 'tch': hinted type string not plausible, read as int - 5x [warn] /Sim/Sys :: 'haltv': no value layout fits; skipped 31 bytes to tag - 2x [warn] /Sim/Sys :: 'haltv': no value layout fits; skipped 35 bytes to tag - 1x [error] /createParams/key :: expected string, read int - 1x [error] /sim/fleets[0]/Flt/ships[0]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended) - 1x [error] /sim/fleets[0]/Flt/ships[1]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended) - 1x [error] /sim/fleets[0]/Flt/ships[2]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended) - 1x [error] /sim/fleets[0]/Flt/ships[3]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended) - 1x [error] /sim/fleets[0]/Flt/ships[4]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended) - 1x [error] /sim/fleets[0]/Flt/ships[5]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended) - 1x [error] /sim/fleets[0]/Flt/ships[6]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended) - 1x [error] /sim/fleets[0]/Flt/ships[7]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended) - 1x [error] /sim/fleets[0]/Flt/ships[8]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended) - 1x [error] /sim/fleets[0]/Flt/ships[9]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended) - 1x [error] /sim/fleets[1]/Flt/ships[0]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended) - 1x [error] /sim/fleets[2]/Flt/ships[0]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended) - 1x [error] /sim/fleets[3]/Flt/ships[0]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended) - 1x [error] /sim/fleets[4]/Flt/ships[0]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended) - 1x [error] /sim/fleets[5]/Flt/ships[0]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended) - 1x [error] /sim/fleets[6]/Flt/ships[0]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended) - 1x [error] /sim/systems[10]/Sys/Name :: expected 'Name', found 'VFlags'; field missing - 1x [error] /sim/systems[10]/Sys/halt[2]/haltv :: expected bool, read raw - 1x [error] /sim/systems[10]/Sys/vnh :: expected 'vnh', found 'VFlags'; field missing - 1x [error] /sim/systems[14]/Sys/Name :: expected 'Name', found 'VFlags'; field missing - 1x [error] /sim/systems[14]/Sys/halt[2]/haltv :: expected bool, read raw - 1x [error] /sim/systems[14]/Sys/vnh :: expected 'vnh', found 'VFlags'; field missing - 1x [error] /sim/systems[16]/Sys/Name :: expected 'Name', found 'VFlags'; field missing - 1x [error] /sim/systems[16]/Sys/halt[2]/haltv :: expected bool, read raw - 1x [error] /sim/systems[16]/Sys/vnh :: expected 'vnh', found 'VFlags'; field missing - 1x [error] /sim/systems[18]/Sys/Name :: expected 'Name', found 'VFlags'; field missing - 1x [error] /sim/systems[18]/Sys/halt[2]/haltv :: expected bool, read raw - 1x [error] /sim/systems[18]/Sys/vnh :: expected 'vnh', found 'VFlags'; field missing - 1x [error] /sim/systems[24]/Sys/Name :: expected 'Name', found 'VFlags'; field missing - 1x [error] /sim/systems[24]/Sys/halt[2]/haltv :: expected bool, read raw - 1x [error] /sim/systems[24]/Sys/vnh :: expected 'vnh', found 'VFlags'; field missing - 1x [error] /sim/systems[25]/Sys/Name :: expected 'Name', found 'VFlags'; field missing - 1x [error] /sim/systems[25]/Sys/halt[2]/haltv :: expected bool, read raw - 1x [error] /sim/systems[25]/Sys/vnh :: expected 'vnh', found 'VFlags'; field missing - 1x [error] /sim/systems[8]/Sys/Name :: expected 'Name', found 'VFlags'; field missing - 1x [error] /sim/systems[8]/Sys/halt[2]/haltv :: expected bool, read raw - 1x [error] /sim/systems[8]/Sys/vnh :: expected 'vnh', found 'VFlags'; field missing - 1x [error] /sim/zdsi :: missing field 'zdsi' (frame ended) - 1x [error] /sim/zdst :: missing field 'zdst' (frame ended) - 1x [warn] /sim/systems[11]/Sys/nve :: 1 unexpected item(s) before 'NVE': 'indi' - 1x [warn] /sim/systems[15]/Sys/nve :: 1 unexpected item(s) before 'NVE': 'indi' - 1x [warn] /sim/systems[18]/Sys/nve :: 1 unexpected item(s) before 'NVE': 'indi' - 1x [warn] /sim/systems[1]/Sys/nve :: 1 unexpected item(s) before 'NVE': 'indi' - 1x [warn] /sim/systems[22]/Sys/nve :: 1 unexpected item(s) before 'NVE': 'indi' - 1x [warn] /sim/systems[25]/Sys/colonies[1]/PID :: 1 unexpected item(s) before 'PID': 'indi' - 1x [warn] /sim/systems[25]/Sys/nve :: 1 unexpected item(s) before 'NVE': 'indi' - 1x [warn] /sim/systems[27]/Sys/nve :: 1 unexpected item(s) before 'NVE': 'indi' - 1x [warn] /sim/systems[4]/Sys/nve :: 1 unexpected item(s) before 'NVE': 'indi' +# save_reader --strict against real save turn2-state.sav (padding=joint, 0 resyncs, 0 hint-failures, 2503 raw bytes = opaque RNG blob) +# result: strict exit 0 -- 0 errors, 0 warnings, 259 info +# after findings/objects/schema-gaps-resolved.md §10 (reader: string values never ASCII-tested, len-0 strings; +# schema: NVO.indi unconditional, PrNSp iff PrMax>0, zdsc pairs, turnstats/tch int, Summary/CreateParams tags, CDT framed) +# same result on turn1-state.sav (257 info) and turn3-state.sav (259 info) +259x [info] tag '.' read positionally as (NULL-named items: Summary Settings/FxCrID, CreateParams MapP planets, Sim id lists, Vector3 bodies) diff --git a/verify/save-reader/SAVE_FORMAT.md b/verify/save-reader/SAVE_FORMAT.md index 9f9b44f..809a864 100644 --- a/verify/save-reader/SAVE_FORMAT.md +++ b/verify/save-reader/SAVE_FORMAT.md @@ -1,11 +1,13 @@ # SOTS1 save format — as implemented by `save_reader.py` -Status: **unvalidated against a real save.** Everything below is what the -reader *assumes*, derived from the community editors (R1 Bardez, R2 SOTSedit; -`save-editor-structs.md`) corrected by the binary member tables -(`struct-recovery.md`). The synthetic fixture (`save_writer_stub.py`) round-trips -under these assumptions; a real `.sav` is the first real test. Items marked -**VERIFY** are the ones the verifier should diff against the binary first. +Status: **validated against the three real saves** (`verify/results/saves/turn1..3-state.sav`, +SotS 1.8): `save_reader.py --strict` exits 0 on all three with `resyncs: 0`, +`hint-failures: 0` and 2503 raw bytes (the opaque `RNG` blob). Sources: the community +editors (R1 Bardez, R2 SOTSedit; `save-editor-structs.md`), the binary member tables +(`struct-recovery.md`) and the serializer decompiles in +`findings/objects/schema-gaps-resolved.md` (round 3), which settled every item that was +marked VERIFY in the previous revision of this file. The synthetic fixture +(`save_writer_stub.py`) round-trips under the same assumptions (`test_save_reader.py`). ## 1. Container @@ -20,38 +22,44 @@ under these assumptions; a real `.sav` is the first real test. Items marked ## 2. Primitive encodings ``` -tag := int32 len, len bytes ASCII (field name; may be "." or "") -string := int32 len, len bytes cp1252 -int := 4 bytes (int32) handle ids, int16 members, enums, counts +tag := int32 len, len bytes ASCII (field name; "." when the writer passed NULL) +string := int32 len, len bytes cp1252 len 0 is legal (Key, MapF, Pwd, Scenario, KeyPath) +int := 4 bytes (int32) handle ids, int16 members widened, enums, counts float := 4 bytes IEEE-754 single bool := 1 byte, 0/1 -int64 := 8 bytes Bats2, rcex, PopC (raw-bytes write path) -raw := n bytes, length known only to the writer RNG state (2500 B = MT19937 624 words + index) +int64 := 8 bytes Bats2, rcex, PopC, pop (raw-bytes write path) +raw := n bytes, length known only to the writer RNG state (2503 B incl. pad = MT19937 624 words + index) ``` -A **named value** is `tag value pad`, where `pad` is NUL bytes bringing the item -to a 4-byte boundary. The reader implements two padding conventions and -auto-detects which one the file uses (`--padding auto`, default): +A **named value** is `tag value pad`, where `pad` is NUL bytes bringing the item to a +4-byte boundary. **Padding is joint** — confirmed on the real saves: `pad4(4 + len(name) ++ len(value))` over the whole item (`haltv` bool = 12 B, `vnh` bool = 8 B, `Name` = +4+4+4+len rounded). The "split" convention (pad after the name and after the value) is +kept only so `--padding auto` can reject it; it never matched a real file. | mode | layout | item size | |---|---|---| -| `joint` (default, R1's `PaddingSize=4` over `4+len(name)+len(value)`) | `[len][name][value][pad]` | `pad4(4 + len(name) + len(value))` | -| `split` | `[len][name][pad][value][pad]` | `pad4(4 + len(name)) + pad4(len(value))` | +| `joint` (real files) | `[len][name][value][pad]` | `pad4(4 + len(name) + len(value))` | +| `split` (not observed) | `[len][name][pad][value][pad]` | `pad4(4 + len(name)) + pad4(len(value))` | -The two differ only when `len(name) % 4 != 0` **and** the value is not a -multiple of 4 bytes (bools, odd-length strings): e.g. `NPC`+bool is 8 bytes in -`joint`, 12 in `split`. **VERIFY (highest priority):** find a 3-char bool tag -(`NPC`, `Dep`, `Atq`, `Srn`, `cta`, `hgs`, `vnh`, `hbq`, `hsp`) in the real -inflated stream and check whether the value byte immediately follows the name. -Reader assumption when writing this: `joint`. +Values carry **no type byte**. Type comes from (a) the schema/catalog, (b) a lookahead +plausibility test (a candidate layout is accepted only if a plausible tag, a frame marker, +or EOF follows), (c) for 4-byte words with no hint, an int/float classification by bit +pattern (`|int| <= 100000` → int; finite float with `1e-6 <= |f| < 1e12` → float; else +int). The other reading is kept as `alt`. Limitation: with a name whose length is a +multiple of 4, `bool` and `int` have identical item sizes; unknown names default to `int`. -Values carry **no type byte**. Type comes from (a) the schema/catalog, (b) a -lookahead plausibility test (a candidate layout is accepted only if a plausible -tag, a frame marker, or EOF follows), (c) for 4-byte words with no hint, an -int/float classification by bit pattern (`|int| <= 100000` → int; finite float -with `1e-6 <= |f| < 1e12` → float; else int). The other reading is kept as `alt`. -Limitation: with a name whose length is a multiple of 4, `bool` and `int` have -identical item sizes; unknown names default to `int`. +Two rules about **string values** (both were reader bugs before round 3): + +* Only **tag** bytes are constrained to printable ASCII. A string **value** may hold any + windows-1252 byte, including 0x80–0x9f (system names such as `Kor’Voth` carry 0x92 = + right single quote). A tag the catalog knows to be a string is read without any text + test; a value can therefore never veto the layout of the item before it (that veto was + the `haltv` → `VFlags`/`Name`/`vnh` cascade). Guessing a string for an *unknown* tag + still uses a text-plausibility guard, which accepts every cp1252-defined byte. +* An **empty string is 4 zero bytes**, byte-identical to int 0. The walker reads it as + `""` when the tag is hinted; when an unhinted walker read a 4-zero-byte word, the + schema applier coerces it to `""` instead of reporting "expected string, read int". ## 3. Complex frames @@ -60,110 +68,142 @@ identical item sizes; unknown names default to `int`. (0xBEEFBEEF) (0x41104110 = ~0xBEEFBEEF) ``` -* Every "nested object" (IStreamable via stream vft+0x28) is framed. Frames - nest; the reader recovers the tree from the markers alone, without a schema. -* A frame may be **tagless** (BEEFBEEF directly at an item position): the reader - accepts it; not observed in the reference material. **VERIFY** if seen. -* A frame body may be **untagged bytes**: `Mars::Vector3` (`Pos`, `PrvPos`, - `FtOrig`, `FPogn2`, `FPdpos`) and `ShipHealth` (`Health`) are "3 unnamed - floats" per the binary. Reader accepts either 12 raw bytes before the END - marker or three tagged floats (any tag, e.g. `"."`). **VERIFY** which. -* `RNG` frame body is treated as opaque bytes up to its END marker. -* Markers are only interpreted at item boundaries; a value that happens to equal - a marker is not a problem unless the reader is already resynchronising. +* Every "nested object" (IStreamable via stream vft+0x28) is framed. Frames nest; the + reader recovers the tree from the markers alone, without a schema. +* **`"."` convention (confirmed):** any write with a NULL name is emitted with the tag + `"."` (len 1). That covers every `VectorHelper` array (`"."` = count, then n × `"."` + elements: frames for streamable T, plain values for POD T), the scalars of + `StarMapParams` and `StrategyPlayerGameSettings`, `PlayerColorID`'s index/r/g/b, and + `Vector3` bodies. The reader matches such items positionally (`R()` fields) and reports + each one as an *info* ("tag '.' read positionally as …"); `"."` is deliberately kept out + of the global type catalog because it carries ints, floats and frames alike. +* **Vector3 bodies are tagged**: 3 × `"."` float items inside the frame (`Pos`, `PrvPos`, + `FtOrig`, `FPogn2`, `FPdpos`, `Health`, `MapP` planet positions; writer `FUN_008a60d0`). + The reader keeps the 12-raw-byte fallback but it is not exercised by real files. +* A frame may be **tagless** (BEEFBEEF directly at an item position): accepted by the + reader; not observed in the real saves (element frames use `"."`). +* `RNG` frame body is one `"."`-tagged opaque blob read straight to its END marker + (2503 B including the 3 joint-padding bytes). +* Markers are only interpreted at item boundaries; a value that happens to equal a marker + is not a problem unless the reader is already resynchronising. ## 4. Arrays | kind | encoding | reader schema | |---|---|---| -| NonComplexArray (`VectorHelper`, `std::list`) | named int count, then count × element **inline in the same frame** | `NArr` | -| ComplexArray | a frame containing: int count, then count × element | `CArr` | -| element = leaf wrapper | e.g. `SysID` int + `Sys` frame; `PlayerID` + `Player`; `FltID` + `Flt`; `ShipID` + `Ship`; `DesID` + `Des` | `Seq([...])` | -| sparse tables | `mnsp` then n × (`msp` index, `mv` value); `nadct` then n × (`ads`, `adt`); `haltc` then n × (`haltt`, `haltv`) | `NArr(Seq)` | +| NonComplexArray (inline count) | named int count, then count × element **inline in the same frame** | `NArr` | +| ComplexArray (`VectorHelper`) | a frame containing: `"."` int count, then count × `"."` element | `CArr` | +| uncounted repetition | elements repeated until the frame END (or until the lead tag stops recurring) | `Repeat` | +| element = leaf wrapper | `SysID` int + `Sys` frame; `PlayerID` + `Player`; `FltID` + `Flt`; `ShipID` + `Ship`; `DesID` + `Des`; `ply` + `hist` | `Seq([...])` | +| sparse tables | `mnsp` then n × (`msp`, `mv`); `nadct` then n × (`ads`, `adt`); `haltc` then n × (`haltt`, `haltv`); `zdsc` then n × (`zdsi`, `zdst`); `PrNSp` then n × (`PrSp`, `PrNum`) | `NArr(Seq)` | -Count tags used by the binary: `NumPlrs NumSys NumFlts NShips NumActs NumOwn NumDes -NumLeg NumNotes NumPR NSprj Nexp NWeapXcl ndeflay rdtc numcreps ninv NumFlts NumGFs -NumSnF NumMnF NVO NVE NVs NumPlgs2 PopNG mnsp nadct haltc PrNSp NTH`. Count tags -inside framed arrays (`dipstats`, `preps`, `odes`, `owep`, `otch`, `cme2`, `Ojvs`…) -are unknown — the reader takes the first child as the count whatever its name. -Element frame tags are unknown (R1 suggests `"."`) — matched by position. +Inline-count tags used by the binary: `NumPlrs NumSys NumFlts NShips NumActs NumOwn NumDes +NumLeg NumNotes NumPR NSprj Nexp NWeapXcl ndeflay rdtc numcreps ninv NumFlts NumGFs NumSnF +NumMnF NVO NVE NVs NumPlgs2 PopNG mnsp nadct haltc PrNSp NTH nply ncls seno2 spc zdsc NumIDs +PlayerIDs DesignIDs SystemIDs FleetIDs ShipIDs TradeIDs AllExc AllExcCF`. Count and element +tags inside framed arrays (`Players`, `GOWinPly`, `sacq`, `slost`, `dipstats`, `preps`, +`odes`, `owep`, `otch`, `cme2`, `Ojvs`, `MapP` planets …) are always `"."` (§3). + +Uncounted lists: `hist` holds n × `stats` frames until its END; `Sim` holds 7 × (`ISsp` +string, `ISsu` float) after the players; the root holds one `CD` frame per custom-data id. ## 5. Conditionals the reader honours | gate | consequence | |---|---| -| `ClrID`/`indcl`/`fxCrId` frame: int index == -1 | three ints r,g,b follow inside the frame | -| `vnh` true | `vnd`, `vnex3`, `vnpex3` | -| `hindi` true | `indi` frame (IndependenceInfo) | -| `NVO` entry `isind` true | `indi` frame | +| `ClrID`/`indcl`/`FxCrID` frame: `"."` int index == -1 | three `"."` ints r,g,b follow inside the frame | +| `vnh` true | `vnd`, `vnex3`, `vnpex3` (the only conditional in that region of `Sys`; `Name` and `VFlags` are unconditional) | +| system-level `hindi` true | `indi` frame (IndependenceInfo) after `NVs` | +| `NVO` entry | `indi` frame is **always** present after `isind` (inline member of the map value; `isind` does not gate it) | +| `PrMax > 0` | `PrNSp` + n × (`PrSp`, `PrNum`); with `PrMax == 0` the `PrisH` frame ends right after `PrMax` | | `HFPlan` true | `FPlan` frame | | `HLay` true | `Lay` frame (opaque) | | `hbq` true | `BQ2` frame; `hsp` true → `pop`, `ppop` Population frames | | `HasAIR` true | `AIR` frame (opaque) | | system `PID` non-null | `BQ` frame present (reader: optional by name) | -| legacy tags `ISuit ARes SysID TrdID Caps GtTrf FtSens FtInc lcid SensMod ExPopSys AIDifficultyID` | accepted if present, expected absent in 1.8 saves | +| `SvSctOb` pointer non-NULL | `SvSctOb` frame before `zdsc` (present in the three real saves; `Opt`) | +| custom-data blob non-NULL | one `CD` frame per `CDT` id that has data | +| legacy tags `ISuit ARes SysID TrdID Caps GtTrf FtSens FtInc lcid SensMod ExPopSys AIDifficultyID Rand RandEnc NPC` | accepted if present (read-side only), never written by 1.8 | ## 6. Top-level layout +Written by `FUN_00877070`; loader `FUN_0086abb0`; summary-only reader `FUN_008773c0`. + ``` -offset 0: [7]"Summary" [pad] BEEFBEEF ... 41104110 (R2 confirms the tag "Summary") - CreateParameters frame (tag unknown) - Sim frame (tag unknown) KeyPath NMSz NMLc NMnx ModCount Frame - GameID Attrib RNG GameName Map IncMod ResMod EnAl - EnTm GOTurn GOWinPly NPCm NPCo NPCi NPCv NPCa - szadj rsadj suadj sprjs RandEncAdj cmbtid turnstats - numcreps/crep ninv NumPlrs/PlayerID/Player - NumSys/SysID/Sys NdGr2 trdmgr spymgr - NumFlts/FltID/Flt NumActs/Act SvSctOb zdsc zdsi zdst - CdTable UNFRAMED at root: cdt frame, cdplayer frame, N × cdai frames +offset 0: "Summary" StrategyGameInfo GameName Turn NumSys Checksum Players{"."=n, n×"."{Slot{} Rank}} + Session{TMRS{TSTL TCTL TQTL TQTLE}} MapShape IncMod ResMod + Alliances Teams Encounters Scenario + "CreateParams" StrategyGameCreateParams Name ID RSeed AID Key MapP{"." int, "."{planets}, "." n×"."{ints}, + "."{nodePaths}} MapS MapF NSys REnc SDist SSize SRes SSuit MaxP ASpec + bAlly NTeam tmgrp PSav PCol PTech IncM ResM scrp{spc n×(spsn sppn sppv)} + "Sim" StrategyServer KeyPath NMSz NMLc NMnx | PlayerIDs DesignIDs SystemIDs FleetIDs + ShipIDs TradeIDs (each: count, n×".") | ModCount Frame GameID Attrib{} + RNG{} GameName Map IncMod ResMod EnAl EnTm GOTurn GOWinPly{} NPCm NPCo + NPCi NPCv NPCa szadj rsadj suadj sprjs{} RandEncAdj cmbtid turnstats{} + numcreps n×crep{} ninv n×(invs inve invt invtb) AllExc n×(AllExc AllExc) + AllExc n×(AllExc AllExc) AllExcCF n×(AllExcCFp AllExcCFp) + NumPlrs n×(PlayerID Player{}) 7×(ISsp ISsu) NumSys n×(SysID Sys{}) + NdGr2{} trdmgr{} spymgr{} NumFlts n×(FltID Flt{}) NumActs n×Act + [SvSctOb{}] zdsc n×(zdsi zdst) + "CDT" CustomDataTable NumIDs n×ID (strings: Player..TurnCommands_v5, Player..AIAgent …) + n × "CD" opaque custom-data frames, one per id that has a blob ``` -Regions in `<...>` are parsed generically and kept as raw item lists -(`sim.idLists`, `sim.invasionsAndExclusions`, `sim.species`). Everything not -covered by a shape (TechTree body, Events, ShipRecs, spy2, civr, comms, Ojvs, -SvSctOb, trdmgr, spymgr, CdTable, …) is kept as the generic +`Slot` (SlotDef): `IsPlay IsDead IsReq IsRec IsFxNm FxNm IsFxSp FxSp IsFxCr FxCrID{} IsFxBd FxBd +IsFxAv FxAv Tag Pwd(string) Team Settings{4×"." int}`. +`turnstats` (GameTurnHistory): `nply`, n × (`ply`, `hist{ply, stats{} …until END}`); +`stats` (PlayerTurnStats): `pop`(int64) `sacq{}` `slost{}` (VectorHelper: `set ses seop +senp seno2 n×seot2`) `trn almem inc tdinc sav col bat tch`(int) `ncls` 3 × (`cls shpt shpl shpk satt +satl satk`). + +Everything not covered by a shape (TechTree body, Events, ShipRecs, spy2, civr, comms, Ojvs, +Attrib, sprjs, SvSctOb, trdmgr, spymgr, CD, …) is kept as the generic `{"_name","_off","_items":[{"name","kind","value","off"}...]}` form. ## 7. Struct field orders and types applied -Shapes live in `save_reader.py` (`Sys`, `Player`, `Fleet`, `Ship`, `PlayerView`, -`Population`/`PopG`, `Morale`, `MoraleEvent`, `BuildQueue`/`BuildOrder`, -`IndependenceInfo`, `Rts`, `DipStat`, `Prep`, `FlightPlan`, `Waypoint`, -`PrisonerHold`, `Summary`, `CreateParams`, …). `A("tag", type)` = on-disk tag -confirmed in the exe (strict name match); `R("name", type)` = R1 C# name only -(positional). Binary corrections applied over R1: +Shapes live in `save_reader.py` (`Summary`, `Slot`, `CreateParams`, `MapP`, `Sim`, +`TurnStats`/`PlayerTurnHistory`/`PlayerTurnStats`, `Sys`, `Player`, `Fleet`, `Ship`, +`PlayerView`, `Population`/`PopG`, `Morale`, `MoraleEvent`, `BuildQueue`/`BuildOrder`, +`IndependenceInfo`, `Rts`, `DipStat`, `Prep`, `FlightPlan`, `Waypoint`, `PrisonerHold`, +`CdTable`, …). `A("tag", type)` = on-disk tag confirmed in the exe (strict name match); +`R("name", type)` = positional (`"."`-tagged items, or R1 C# names). Binary corrections +applied over R1/R2: -* float: `TRM CstR CstE CstT shrm RefCap RepCap`, PlayerView `Infra` -* int64: `Bats2 rcex PopC` -* bool: `Abdn Dstyd PvMA AIBn` (R2's "short" readings are the value byte) -* int on disk though int16 in memory: `TShn ETS` and all DiplomacyStats counters -* string: `pswd` -* Vector3 (3 floats): `FtOrig` +* float: `TRM CstR CstE CstT shrm RefCap RepCap`, PlayerView `Infra`; `Summary.IncMod/ResMod`, + `Sim.IncMod/ResMod`, `Sim.szadj/rsadj/suadj`, `TMRS.TSTL/TCTL/TQTL/TQTLE`, + `CreateParams.SDist/SSize/SRes/SSuit/REnc/IncM/ResM`, `ISsu` +* int64: `Bats2 rcex PopC`, `PlayerTurnStats.pop` +* bool: `Abdn Dstyd PvMA AIBn haltv EnAl EnTm` (R2's "short" readings are the value byte) +* int on disk though int16 in memory: `TShn ETS tch col bat` and all DiplomacyStats counters +* string: `pswd`, `Key`, `MapF`, `Pwd`, `KeyPath` (all may be empty) +* Vector3 (3 × `"."` float): `Pos PrvPos FtOrig FPogn2 FPdpos Health` * `Nexp` entries carry `xid xmin xmax xper(float)` -* `Team` appears twice in Player: an int, later a nested `{ALid AL NA CF}` frame - (typed key `Alliances`) +* `Team` appears twice in Player: an int, later a nested `{ALid AL NA CF}` frame (typed key `Alliances`) +* `GOWinPly` is a framed `VectorHelper` (`"."` count + n × `"."` int), empty in the real saves +* `CreateParams`/`Summary` tag case is as written above (R1's camel-case names were wrong) ## 8. Reader output conventions -* `--dump`: one line per item, `@ name kind value`; `?` after - the kind means the type was guessed, `(alt …)` shows the other reading of a - 4-byte word; frames print `{` … `}` with item count and byte size. -* `--json`: `{"padding", "stats", "issues": [...], "data": {...}}`; every typed - struct carries `_off`; unexpected items are kept under `_unexpected` / - `_extra`; unknown regions keep the generic form. -* Issue levels: `error` (schema field missing / type impossible / frame - unterminated), `warn` (resync, hint not plausible, unexpected items, - width mismatch, best-effort read at EOF), `info` (unnamed small payload - before END, positional tag name differing from the R1 name). `--strict` - fails on error or warn. -* Exit status: 0 clean, 1 errors present, 2 unreadable container. +* `--dump`: one line per item, `@ name kind value`; `?` after the kind + means the type was guessed, `(alt …)` shows the other reading of a 4-byte word; frames + print `{` … `}` with item count and byte size. +* `--json`: `{"padding", "stats", "issues": [...], "data": {...}}`; `data` has the keys + `summary`, `createParams`, `sim`, `cdTable`, `customData`; every typed struct carries + `_off`; unexpected items are kept under `_unexpected` / `_extra`; unknown regions keep + the generic form. +* Issue levels: `error` (schema field missing / type impossible / frame unterminated), + `warn` (resync, hint not plausible, unexpected items, width mismatch, best-effort read + at EOF), `info` (positional `"."` tag read as an R() name, unnamed small payload before + END). `--strict` fails on error or warn. +* Exit status: 0 clean, 1 errors present, 2 unreadable container or strict failure. -## 9. What the verifier should check first on a real save +## 9. Verification checklist (all settled on the real saves) -1. Padding convention (§2) — a 3-char bool tag settles it. -2. The `Summary` frame's 13 children and their actual tag names (R1 names are - probably case-variants: `NumSys` is confirmed by R2 as a tag string). -3. Whether Vector3 bodies are tagged (§3). -4. The tag used for element frames and for counts inside framed arrays (§4). -5. Whether the reader's auto-detected padding, `resyncs == 0` and - `hint_failures == 0` hold; any resync offset points at a layout gap. +1. Padding convention — **joint** (§2). +2. `Summary` frame: 13 children with the tags in §6 (R1's names were case-variants). +3. Vector3 bodies — **tagged**, 3 × `"."` float (§3). +4. Element and count tags inside framed arrays — **`"."`** (§3, §4). +5. `--padding auto` picks joint; `resyncs == 0` and `hint_failures == 0` on turn1/2/3; + the only raw bytes are the `RNG` blob. Remaining unknowns are the bodies of the + opaque frames listed in §6 (kept generic, not a parsing gap). diff --git a/verify/save-reader/save_reader.py b/verify/save-reader/save_reader.py index 4d5ae74..a55ae11 100644 --- a/verify/save-reader/save_reader.py +++ b/verify/save-reader/save_reader.py @@ -24,18 +24,25 @@ has to guess it. This module does both: malformed regions become `raw` nodes; parsing continues at the next frame marker or plausible tag. Nothing in the schema is required for the walk to succeed. - * Shapes the recovered struct field orders (Summary, CreateParameters, - Sim -> Player / Sys / Flt / Ship ...) applied on top of the - generic tree to produce typed dicts. Fields marked A() carry - binary-confirmed on-disk names (struct-recovery.md) and are - matched by name; fields marked R() carry only the community - editor's C# name and are matched by position. + * Shapes the recovered struct field orders (Summary, CreateParams, + Sim -> Player / Sys / Flt / Ship ..., CDT) applied on top of + the generic tree to produce typed dicts. Fields marked A() + carry binary-confirmed on-disk names (struct-recovery.md, + schema-gaps-resolved.md) and are matched by name; fields marked + R() are matched by position (NULL-named "." items, or names + known only from the community editor). -Padding convention: the community reference is ambiguous about whether the -NUL padding is computed over the whole item ("joint": [len][name][value][pad]) -or separately after the name and after the value ("split"). Both are -implemented; --padding auto (default) walks the file both ways and keeps the -cleaner parse. REAL-SAVE VALIDATION IS PENDING -- see SAVE_FORMAT.md. +Padding convention: the NUL padding is computed over the whole item ("joint": +[len][name][value][pad]) -- confirmed on the real saves. The alternative +("split": pad after the name and after the value) is kept for --padding auto, +which walks the file both ways and keeps the cleaner parse. + +Two facts about string VALUES matter for the plausibility test: they may be +empty (len 0 = four zero bytes, byte-identical to int 0) and they may hold any +windows-1252 byte (system names use 0x92 = right single quote). Only TAG bytes +are constrained to printable ASCII; a value never vetoes a layout. Items the +game writes with a NULL name carry the tag ".". See SAVE_FORMAT.md and +findings/objects/schema-gaps-resolved.md. usage: save_reader.py SAVE [--dump] [--json] [--strict] [--padding MODE] [--out FILE] [--inflate FILE] @@ -55,7 +62,7 @@ from typing import Any, Callable, Optional __all__ = [ "BEGIN_MARK", "END_MARK", "SaveFormatError", "Issue", "Node", "Walker", - "Shape", "Seq", "CArr", "NArr", "If", "Opt", "Until", "Rest", "A", "R", + "Shape", "Seq", "CArr", "NArr", "If", "Opt", "Until", "Repeat", "Rest", "A", "R", "ROOT", "apply_schema", "read_save", "read_bytes", "inflate", "dump_tree", "plain", "pad4", ] @@ -210,11 +217,18 @@ class NArr: class If: - """Conditional group: parsed iff out[key] == equals (default True).""" - __slots__ = ("key", "inner", "equals") + """Conditional group: parsed iff out[key] == equals (default True), or, + with greater_than=N, iff out[key] is a number > N (e.g. PrMax > 0).""" + __slots__ = ("key", "inner", "equals", "greater_than") - def __init__(self, key, inner, equals=True): - self.key, self.inner, self.equals = key, inner, equals + def __init__(self, key, inner, equals=True, greater_than=None): + self.key, self.inner, self.equals, self.greater_than = key, inner, equals, greater_than + + def holds(self, out: dict) -> bool: + v = out.get(self.key) + if self.greater_than is not None: + return isinstance(v, (int, float)) and not isinstance(v, bool) and v > self.greater_than + return v == self.equals class Opt: @@ -233,6 +247,22 @@ class Until: self.key, self.stop = key, stop +class Repeat: + """Uncounted repetition: consume `elem` (a Field or Seq whose first field is + an A() tag) again and again while the next item carries that tag. Used for + lists the writer emits with no count (turnstats `stats` frames until the + frame END, the 7 `ISsp`/`ISsu` pairs, the root `CD` frames).""" + __slots__ = ("elem", "key", "lead") + + def __init__(self, elem, key=None): + self.elem = elem + first = elem.fields[0] if isinstance(elem, Seq) else elem + if not isinstance(first, Field): + raise TypeError("Repeat needs a Field or a Seq starting with a Field") + self.lead = first.name + self.key = key or first.key + + class Rest: """Collect all remaining items generically.""" __slots__ = ("key",) @@ -294,6 +324,9 @@ def _collect_names(fields, d): elif isinstance(f, If): inner = f.inner _collect_names(inner.fields if isinstance(inner, Seq) else [inner], d) + elif isinstance(f, Repeat): + e = f.elem + _collect_names(e.fields if isinstance(e, Seq) else [e], d) # --- schema: recovered struct layouts ----------------------------------------- @@ -301,58 +334,59 @@ def _collect_names(fields, d): # from struct-recovery.md (binary). A() names are on-disk tags confirmed in the # exe; R() names are R1's C# field names (disk spelling unknown -> positional). -NestedInt = Shape(None, [R("value", "int"), Rest()]) - -PlayerColor = Shape("ClrID", [ +# Game::PlayerColorID: "." int index; iff -1 the three "." ints r,g,b follow. +# Framed under ClrID (Player), indcl (IndependenceInfo) and FxCrID (SlotDef). +PlayerColor = Shape(None, [ R("idx", "int"), If("idx", Seq([R("r", "int"), R("g", "int"), R("b", "int")]), equals=-1), Rest(), ]) -# -- Summary (R1 §3) ----------------------------------------------------------- -PlayerSettings = Shape("Settings", [ - R("initialTreasury", "int"), R("initialColonies", "int"), - R("initialTechnologies", "int"), R("difficulty", "int"), Rest(), +# -- Summary = StrategyGameInfo::Write @0x00829960 (schema-gaps-resolved.md §5.2) +# Every tag below is confirmed in the binary. Items whose writer passes a NULL +# name are on disk as "." and are matched positionally (R()). +PlayerSettings = Shape("Settings", [ # StrategyPlayerGameSettings: 4 x "." int + R("treasury", "int"), R("colonies", "int"), R("techs", "int"), R("difficulty", "int"), Rest(), ]) -PlayerSlot = Shape("Slot", [ - R("isPlay", "bool"), R("isDead", "bool"), R("isReq", "bool"), R("isRec", "bool"), - R("isFxNm", "bool"), R("fxNm", "string"), - R("isFxSp", "bool"), R("fxSp", "int"), - R("isFxCr", "bool"), R("fxCrId", PlayerColor), - R("isFxBd", "bool"), R("fxBd", "string"), - R("isFxAv", "bool"), R("fxAv", "string"), - R("tag", "int"), R("pwd", "int"), R("team", "int"), - R("settings", PlayerSettings), Rest(), +Slot = Shape("Slot", [ # SlotDef::Write @0x008276d0 + A("IsPlay", "bool"), A("IsDead", "bool"), A("IsReq", "bool"), A("IsRec", "bool"), + A("IsFxNm", "bool"), A("FxNm", "string"), A("IsFxSp", "bool"), A("FxSp", "int"), + A("IsFxCr", "bool"), A("FxCrID", PlayerColor), A("IsFxBd", "bool"), A("FxBd", "string"), + A("IsFxAv", "bool"), A("FxAv", "string"), A("Tag", "int"), A("Pwd", "string"), A("Team", "int"), + A("Settings", PlayerSettings), Rest(), ]) -PlayerSlotWrapper = Shape("PlayerSlotWrapper", [R("slot", PlayerSlot), R("rank", "int"), Rest()]) -Tmrs = Shape("Tmrs", [R("tstl", "int"), R("tctl", "int"), R("tqtl", "int"), R("tqtle", "int"), Rest()]) -Session = Shape("Session", [R("tmrs", Tmrs), Rest()]) +PlayerInfo = Shape(None, [A("Slot", Slot), A("Rank", "int"), Rest()]) # element tag "." +Tmrs = Shape("TMRS", [A("TSTL", "float"), A("TCTL", "float"), A("TQTL", "float"), A("TQTLE", "float"), Rest()]) +Session = Shape("Session", [A("TMRS", Tmrs), Rest()]) Summary = Shape("Summary", [ - R("gameName", "string"), R("turn", "int"), R("numSys", "int"), R("checkSum", "int"), - R("players", CArr(PlayerSlotWrapper)), R("session", Session), - R("mapShape", "int"), R("incMod", "int"), R("resMod", "int"), - R("alliances", "bool"), R("teams", "bool"), R("encounters", "bool"), - R("scenario", "string"), Rest(), + A("GameName", "string"), A("Turn", "int"), A("NumSys", "int"), A("Checksum", "int"), + A("Players", CArr(PlayerInfo)), A("Session", Session), A("MapShape", "int"), + A("IncMod", "float"), A("ResMod", "float"), A("Alliances", "bool"), A("Teams", "bool"), + A("Encounters", "bool"), A("Scenario", "string"), Rest(), ]) -# -- CreateParameters (R1 §4) --------------------------------------------------- -Planet = Shape("Planet", [ - R("coordinates", "vec3"), R("unknown1", "int"), R("unknown2", "int"), - R("unknown3", "int"), R("unknown4", "int"), Rest(), +# -- CreateParams = StrategyGameCreateParams::Write @0x0082ae40 (§5.1) ---------- +Planet = Shape(None, [ # SystemParams element, all tags "." + R("pos", "vec3"), R("p1", "int"), R("p2", "int"), R("p3", "int"), R("p4", "float"), Rest(), ]) -MapPNpc = Shape("Npc", [R("unknown1", "int"), R("unknown2", "int"), Rest()]) -MapP = Shape("MapP", [ - R("unknown1", "int"), R("planetArray", CArr(Planet)), - R("players", NArr(CArr("int"))), R("npcArray", CArr(MapPNpc)), Rest(), +MapP = Shape("MapP", [ # StarMapParams::Write @0x00727a10, all tags "." + R("mapType", "int"), + R("planets", CArr(Planet)), # VectorHelper + R("players", NArr(CArr("int"))), # "." count, n x "." frame{ "." count, n x "." int } + R("nodePaths", CArr("any")), # VectorHelper, empty in the real saves + Rest(), ]) -Scrp = Shape("Scrp", [R("spc", "int"), Rest()]) -CreateParams = Shape("CreateParameters", [ - R("name", "string"), R("id", "int"), R("rSeed", "int"), R("aid", "int"), R("key", "string"), - R("mapP", MapP), R("mapS", "int"), R("mapF", "int"), R("nSys", "int"), R("rEnc", "float"), - R("sDist", "int"), R("sSize", "float"), R("sRes", "float"), R("sSuit", "int"), R("maxP", "int"), - R("aSpec", "int"), R("bAlly", "bool"), R("nTeam", "int"), R("tmgrp", "bool"), - R("pSav", "int"), R("pCol", "int"), R("pTech", "int"), R("incM", "float"), R("resM", "float"), - R("scrp", Scrp), Rest(), +Scrp = Shape("scrp", [ # StrategyScriptParams::Write @0x0082ad40 + A("spc", NArr(Seq([A("spsn", "string"), A("sppn", "string"), A("sppv", "string")])), key="params"), + Rest(), +]) +CreateParams = Shape("CreateParams", [ + A("Name", "string"), A("ID", "int"), A("RSeed", "int"), A("AID", "int"), A("Key", "string"), # Key may be "" + A("MapP", MapP), A("MapS", "int"), A("MapF", "string"), A("NSys", "int"), A("REnc", "float"), + A("SDist", "float"), A("SSize", "float"), A("SRes", "float"), A("SSuit", "float"), A("MaxP", "int"), + A("ASpec", "int"), A("bAlly", "bool"), A("NTeam", "int"), A("tmgrp", "bool"), + A("PSav", "int"), A("PCol", "int"), A("PTech", "int"), A("IncM", "float"), A("ResM", "float"), + A("scrp", Scrp), Rest(), ]) # -- shared sim types (struct-recovery §1.4, §1.5, §1.6, §1.7) ------------------ @@ -417,9 +451,11 @@ Sys = Shape("Sys", [ A("NumGFs", NArr(A("GF", "int")), key="gates"), A("NumSnF", NArr(A("SnF", "int")), key="stations"), A("NumMnF", NArr(A("MnF", "int")), key="monitors"), + # NVO entry: indi is an inline member of the map value and is written for + # every colony, NOT gated by isind (ServerSystem::Write @0x00749630). A("NVO", NArr(Seq([ A("PID", "int"), A("TShn", "int"), A("OID", "int"), A("isind", "bool"), - If("isind", A("indi", IndependenceInfo)), + A("indi", IndependenceInfo), ])), key="colonies"), A("NVE", NArr(Seq([A("EPid", "int"), A("ETS", "int"), A("Eid", "int")])), key="nve"), A("NVs", NArr(Seq([R("pid", "int"), R("pview", PlayerView)])), key="views"), @@ -513,8 +549,10 @@ FlightPlan = Shape("FPlan", [ R("wpts", CArr(Waypoint), flex=True), A("FPsp2", "float"), A("FPeta2", "int"), A("FPogn2", "vec3"), A("FPdpos", "vec3"), A("pnd", "int"), Rest(), ]) -PrisonerHold = Shape("PrisH", [ - A("PrMax", "int"), A("PrNSp", NArr(Seq([A("PrSp", "int"), A("PrNum", "int")])), key="prisoners"), Rest(), +PrisonerHold = Shape("PrisH", [ # PrisonerHold::Write @0x0056ec00 + A("PrMax", "int"), + If("PrMax", A("PrNSp", NArr(Seq([A("PrSp", "int"), A("PrNum", "int")])), key="prisoners"), greater_than=0), + Rest(), ]) Ship = Shape("Ship", [ A("DesID", "int"), A("FltID", "int"), A("PlrID", "int"), A("Range", "float"), @@ -561,34 +599,65 @@ NodePath = Shape(None, [ ]) NodeGrid = Shape("NdGr2", [R("paths", CArr(NodePath), flex=True), R("nextId", "int"), Rest()]) -# -- Sim block (struct-recovery §5 tag order + R1 §5) ---------------------------- +# -- turn statistics (schema-gaps-resolved.md §6) -------------------------------- +SystemEvent = Shape(None, [ # SystemEvent::Write @0x008189a0, element tag "." + A("set", "int"), A("ses", "int"), A("seop", "int"), A("senp", "int"), + A("seno2", NArr(A("seot2", "int")), key="others"), Rest(), +]) +ClassStats = Seq([A("cls", "int"), A("shpt", "int"), A("shpl", "int"), A("shpk", "int"), + A("satt", "int"), A("satl", "int"), A("satk", "int")]) +PlayerTurnStats = Shape("stats", [ # PlayerTurnStats::Write @0x0082c290 + A("pop", "int64"), A("sacq", CArr(SystemEvent)), A("slost", CArr(SystemEvent)), + A("trn", "int"), A("almem", "int"), A("inc", "int"), A("tdinc", "int"), A("sav", "int"), + A("col", "int"), A("bat", "int"), A("tch", "int"), # tch: INT (int16 widened) + A("ncls", NArr(ClassStats), key="classes"), Rest(), +]) +PlayerTurnHistory = Shape("hist", [ # PlayerTurnHistory::Write @0x0082c4a0 + A("ply", "int"), Repeat(A("stats", PlayerTurnStats), key="stats"), # no count: until frame END +]) +TurnStats = Shape("turnstats", [ # GameTurnHistory::Write @0x0082c5a0 + A("nply", NArr(Seq([A("ply", "int"), A("hist", PlayerTurnHistory)])), key="players"), Rest(), +]) + +# -- Sim block = StrategyServer::Write @0x0079fa70 (schema-gaps-resolved.md §8) ---- +Invasion = Seq([A("invs", "int"), A("inve", "int"), A("invt", "int"), A("invtb", "int")]) +IdList = NArr(R(".", "int")) # FUN_00794cd0: count, n x "." int Sim = Shape("Sim", [ A("KeyPath", "string"), A("NMSz", "int"), A("NMLc", "int"), A("NMnx", "int"), - Until("idLists", "ModCount"), # PlayerIDs/DesignIDs/SystemIDs/FleetIDs/ShipIDs/TradeIDs + A("PlayerIDs", IdList, key="playerIds"), A("DesignIDs", IdList, key="designIds"), + A("SystemIDs", IdList, key="systemIds"), A("FleetIDs", IdList, key="fleetIds"), + A("ShipIDs", IdList, key="shipIds"), A("TradeIDs", IdList, key="tradeIds"), A("ModCount", "int"), A("Frame", "int"), A("GameID", "int"), Opt("AIDifficultyID", "int"), - A("Attrib", "any"), A("RNG", "any"), A("GameName", "string"), - A("Map", "int"), A("IncMod", "int"), A("ResMod", "int"), A("EnAl", "int"), A("EnTm", "int"), - A("GOTurn", "int"), A("GOWinPly", NestedInt), - A("NPCm", "int"), A("NPCo", "int"), A("NPCi", "int"), A("NPCv", "int"), A("NPCa", "int"), - A("szadj", "int"), A("rsadj", "int"), A("suadj", "int"), - A("sprjs", "any"), A("RandEncAdj", "float"), A("cmbtid", "int"), A("turnstats", "any"), + A("Attrib", "any"), Opt("Rand", "int"), A("RNG", "any"), A("GameName", "string"), + A("Map", "int"), A("IncMod", "float"), A("ResMod", "float"), Opt("RandEnc", "bool"), + A("EnAl", "bool"), A("EnTm", "bool"), + A("GOTurn", "int"), A("GOWinPly", CArr("int")), # VectorHelper + A("NPCm", "int"), A("NPCo", "int"), A("NPCi", "int"), A("NPCv", "int"), A("NPCa", "int"), Opt("NPC", "int"), + A("szadj", "float"), A("rsadj", "float"), A("suadj", "float"), + A("sprjs", "any"), A("RandEncAdj", "float"), A("cmbtid", "int"), A("turnstats", TurnStats), A("numcreps", NArr(A("crep", Crep)), key="combatReports"), - A("ninv", "int"), - Until("invasionsAndExclusions", "NumPlrs"), # invs/inve/invt/invtb, AllExc x6, AllExcCF, AllExcCFp x2 + A("ninv", NArr(Invasion), key="invasions"), + A("AllExc", NArr(Seq([A("AllExc", "int"), A("AllExc", "int")])), key="exclusions3"), + A("AllExc", NArr(Seq([A("AllExc", "int"), A("AllExc", "int")])), key="exclusions2"), + A("AllExcCF", NArr(Seq([A("AllExcCFp", "int"), A("AllExcCFp", "int")])), key="exclusionsCF"), A("NumPlrs", NArr(Seq([A("PlayerID", "int"), A("Player", Player)])), key="players"), - Until("species", "NumSys"), # ISsp / ISsu + Repeat(Seq([A("ISsp", "string"), A("ISsu", "float")]), key="species"), # 7 pairs, no count A("NumSys", NArr(Seq([A("SysID", "int"), A("Sys", Sys)])), key="systems"), A("NdGr2", NodeGrid), A("trdmgr", "any"), A("spymgr", "any"), A("NumFlts", NArr(Seq([A("FltID", "int"), A("Flt", Fleet)])), key="fleets"), A("NumActs", NArr(A("Act", "int")), key="acts"), - A("SvSctOb", "any"), - A("zdsc", "int"), A("zdsi", "int"), A("zdst", "int"), + Opt("SvSctOb", "any"), # only if pointer != NULL + A("zdsc", NArr(Seq([A("zdsi", "int"), A("zdst", "int")])), key="zoneDefence"), Rest(), ]) -# -- file root: Summary, CreateParameters, Sim, then the unframed CdTable --------- +# -- file root (FUN_00877070): Summary, CreateParams, Sim, CDT, then one opaque +# CD frame per custom-data id that has a blob (Player..TurnCommands_v5 / .AIAgent). +CdTable = Shape("CDT", [A("NumIDs", NArr(A("ID", "string")), key="ids"), Rest()]) ROOT = Seq([ - R("summary", Summary), R("createParams", CreateParams), R("sim", Sim), Rest("cdTable"), + A("Summary", Summary, key="summary"), A("CreateParams", CreateParams, key="createParams"), + A("Sim", Sim, key="sim"), A("CDT", CdTable, key="cdTable"), + Repeat(A("CD", "any"), key="customData"), ]) # Hand-maintained kinds for names that only occur inside generic ("any") regions. @@ -599,7 +668,7 @@ MANUAL_KINDS = { "faiDes": "bool", "dHide": "bool", # objectives / comms / research / encounters (R1 §5, §6, §11) "cmp": "bool", "dsc": "string", "xcsn": "string", "nm": "string", "ntg": "string", - "tch": "string", "wep": "string", "gmch": "int", "drad": "float", "cst": "float", + "wep": "string", "gmch": "int", "drad": "float", "cst": "float", "aOdd": "float", "aInc": "float", "rMd": "float", "sctSize": "float", "smx": "float", "nPrvVa": "float", "crPce": "bool", "maintHf": "bool", "caps2": "int64", "tRsld": "bool", "tRsldd": "bool", "tRsldc": "bool", "tRsldi": "bool", "tRsldr": "bool", @@ -607,7 +676,10 @@ MANUAL_KINDS = { } -def _register(shape, kinds: dict, conflicts: set, shapes: dict, seen: set): +def _register(shape, kinds: dict, conflicts: set, shapes: dict, seen: set, weak: bool = False): + """Collect global (name -> kind) hints and (name -> shape) frames. + weak=False registers A()/R() fields (conflicting kinds are dropped later); + weak=True registers Opt() legacy tags only where nothing else claims the name.""" if id(shape) in seen: return seen.add(id(shape)) @@ -615,37 +687,43 @@ def _register(shape, kinds: dict, conflicts: set, shapes: dict, seen: set): shapes.setdefault(shape.name, shape) fields = shape.fields if isinstance(shape, (Shape, Seq)) else [] for f in fields: - items = [f] + items, is_opt = [f], False if isinstance(f, Opt): - items = [f.field] + items, is_opt = [f.field], True elif isinstance(f, If): items = f.inner.fields if isinstance(f.inner, Seq) else [f.inner] + elif isinstance(f, Repeat): + items = f.elem.fields if isinstance(f.elem, Seq) else [f.elem] elif not isinstance(f, Field): continue for fld in items: if not isinstance(fld, Field): continue - _register_type(fld.name, fld.type, kinds, conflicts, shapes, seen) + _register_type(fld.name, fld.type, kinds, conflicts, shapes, seen, weak, is_opt) -def _register_type(name, t, kinds, conflicts, shapes, seen): +def _register_type(name, t, kinds, conflicts, shapes, seen, weak=False, is_opt=False): if isinstance(t, str): if t in PRIMITIVES: - if name in kinds and kinds[name] != t: - conflicts.add(name) - kinds.setdefault(name, t) + if weak: + if is_opt and name not in conflicts: + kinds.setdefault(name, t) + elif not is_opt: + if name in kinds and kinds[name] != t: + conflicts.add(name) + kinds.setdefault(name, t) elif isinstance(t, Shape): if name and not t.name: shapes.setdefault(name, t) - _register(t, kinds, conflicts, shapes, seen) + _register(t, kinds, conflicts, shapes, seen, weak) elif isinstance(t, Seq): - _register(t, kinds, conflicts, shapes, seen) + _register(t, kinds, conflicts, shapes, seen, weak) elif isinstance(t, (CArr, NArr)): e = t.elem if isinstance(e, Field): - _register_type(e.name, e.type, kinds, conflicts, shapes, seen) + _register_type(e.name, e.type, kinds, conflicts, shapes, seen, weak) elif isinstance(e, (Shape, Seq)): - _register(e, kinds, conflicts, shapes, seen) + _register(e, kinds, conflicts, shapes, seen, weak) if isinstance(t, CArr) and name: shapes.setdefault(name, t) @@ -655,6 +733,8 @@ def _build_catalog(): _register(ROOT, kinds, conflicts, shapes, set()) for k in conflicts: # ambiguous across contexts -> no global hint kinds.pop(k, None) + _register(ROOT, kinds, conflicts, shapes, set(), weak=True) # legacy Opt() tags, lowest priority + kinds.pop(".", None) # the NULL-name tag carries ints, floats, frames -- never hint it for k, v in MANUAL_KINDS.items(): kinds[k] = v return kinds, shapes @@ -667,10 +747,18 @@ RAW_FRAMES = {"RNG"} # --- generic walker ----------------------------------------------------------- +_CP1252_UNDEFINED = frozenset((0x81, 0x8d, 0x8f, 0x90, 0x9d)) + + def _text_plausible(b: bytes) -> bool: + """Guard used only while GUESSING that an unknown tag holds a string. Every + byte windows-1252 defines counts as text -- including 0x80-0x9f (system + names like "Kor\\x92Voth" use 0x92 = right single quote). Never apply this + to a tag that is known to be a string.""" if not b: return True - ok = sum(1 for c in b if 0x20 <= c < 0x7f or c >= 0xa0 or c in (9, 10, 13)) + ok = sum(1 for c in b if (0x20 <= c < 0x7f) or (c >= 0x80 and c not in _CP1252_UNDEFINED) + or c in (9, 10, 13)) return ok * 10 >= len(b) * 9 @@ -745,8 +833,10 @@ class Walker: return False return self.zero(vp + size, end) - def try_scalar(self, tag_start: int, name_end: int, kind: str): - """Read a candidate scalar; (kind, value, raw, end) or None if impossible.""" + def try_scalar(self, tag_start: int, name_end: int, kind: str, known: bool = False): + """Read a candidate scalar; (kind, value, raw, end) or None if impossible. + known=True means the kind comes from the schema/catalog: a string value + is then accepted whatever bytes it holds (only tags must be ASCII).""" d, n = self.d, self.n vp = self.value_pos(tag_start, name_end) if kind == "string": @@ -759,10 +849,10 @@ class Walker: if vp + 4 + ln > n: self._eof_limited = True return None - raw = d[vp + 4:vp + 4 + ln] - if not _text_plausible(raw): + raw = d[vp + 4:vp + 4 + ln] # ln == 0 (empty string) is legal + if not known and not _text_plausible(raw): return None - size, value = 4 + ln, raw.decode(ENCODING) + size, value = 4 + ln, raw.decode(ENCODING, errors="replace") else: size = 4 if kind == "word" else SIZES[kind] if vp + size > n: @@ -810,6 +900,13 @@ class Walker: return True if depth <= 0: return True + # a tag the catalog knows to be a string is read as one without any + # text test, so a non-ASCII value (cp1252 0x92 in "Kor'Voth") can never + # veto the layout of the item that precedes it + if self.kinds.get(name) == "string": + r = self.try_scalar(p, ne, "string", known=True) + if r is not None and self.plausible_item(r[3], depth - 1, allow_empty): + return True for kind in self.GUESS_ORDER: r = self.try_scalar(p, ne, kind) if r is None: @@ -943,7 +1040,7 @@ class Walker: if mode == "eof" and not self._eof_limited: break for i, cand in enumerate(order): - r = self.try_scalar(p, ne, cand) + r = self.try_scalar(p, ne, cand, known=(i == 0 and kind is not None)) if r is None: continue if mode == "eof" or self.plausible_item(r[3], 2, allow_empty=(mode == "empty-ok")): @@ -1108,12 +1205,20 @@ class Applier: if nxt is not None and nxt.name is not None and nxt.name.lower() == f.field.name.lower(): self.apply_field(f.field, cur, out, path) elif isinstance(f, If): - if out.get(f.key) == f.equals: + if f.holds(out): inner = f.inner if isinstance(inner, Seq): self.apply_fields(inner.fields, cur, out, path) else: self.apply_field(inner, cur, out, path) + elif isinstance(f, Repeat): + items = [] + while not cur.done and cur.peek().name == f.lead: + before = cur.i + items.append(self.convert_elem(f.elem, cur, f"{path}/{f.key}[{len(items)}]")) + if cur.i == before: + break + out[f.key] = items elif isinstance(f, Until): items = [] while not cur.done: @@ -1288,6 +1393,11 @@ class Applier: if k in ("int", "float") and len(raw) == 4: self.issue("warn", path, node, "int64 expected, 4-byte word read (width mismatch)") return struct.unpack(" list[str]: f"issues: {res.count('error')} error, {res.count('warn')} warn, {res.count('info')} info"] s = t.get("summary") or {} if isinstance(s, dict): - lines.append(f"summary: game={s.get('gameName')!r} turn={s.get('turn')} numSys={s.get('numSys')} " - f"players={len(s.get('players') or [])} scenario={s.get('scenario')!r}") + lines.append(f"summary: game={s.get('GameName')!r} turn={s.get('Turn')} numSys={s.get('NumSys')} " + f"players={len(s.get('Players') or [])} scenario={s.get('Scenario')!r}") sim = t.get("sim") or {} if isinstance(sim, dict): lines.append(f"sim: gameName={sim.get('GameName')!r} players={len(sim.get('players') or [])} " diff --git a/verify/save-reader/save_writer_stub.py b/verify/save-reader/save_writer_stub.py index a80bbf4..f6e467d 100644 --- a/verify/save-reader/save_writer_stub.py +++ b/verify/save-reader/save_writer_stub.py @@ -208,17 +208,18 @@ class Fixture: if f.field.name in OPT_EMIT: self.emit_field(f.field, out, ctx) elif isinstance(f, sr.If): - if out.get(f.key) == f.equals: + if f.holds(out): inner = f.inner if isinstance(inner, sr.Seq): self.emit_fields(inner.fields, out, ctx) else: self.emit_field(inner, out, ctx) + elif isinstance(f, sr.Repeat): + out[f.key] = [self.emit_elem(f.elem, ctx) for _ in range(ARRAY_LEN)] elif isinstance(f, sr.Until): out[f.key] = self.filler_items() elif isinstance(f, sr.Rest): - if top: - out[f.key] = self.cd_table() + pass # nothing extra is written else: raise TypeError(f) @@ -265,7 +266,7 @@ class Fixture: def emit_carr(self, name, carr: sr.CArr, ctx) -> list: self.w.begin(name) - self.w.int("Count", ARRAY_LEN) + self.w.int(".", ARRAY_LEN) # VectorHelper count tag is "." items = [self.emit_elem(carr.elem, ctx) for _ in range(ARRAY_LEN)] self.w.end() return items @@ -280,31 +281,15 @@ class Fixture: self.emit_field(elem, sub, ctx) return sub.get(elem.key) if isinstance(elem, sr.Shape): - return self.emit_shape(".", elem) # R1: element tags are often "." + return self.emit_shape(".", elem) # VectorHelper element tag is "." if isinstance(elem, sr.CArr): return self.emit_carr(".", elem, ctx) + if elem == "any": + return self.generic_frame(".") if elem in sr.PRIMITIVES: - return self.emit_scalar("Item", elem, ctx) + return self.emit_scalar(".", elem, ctx) raise TypeError(elem) - def cd_table(self) -> list: - """Unframed 4th section: a framed string array, an unknown-shaped player - block and two AI blocks -- all opaque to the schema.""" - w = self.w - items = [] - w.begin("cdt") - w.int("Count", 2) - w.string("Item", "cd-a") - w.string("Item", "cd-b") - w.end() - items.append({"_name": "cdt", "_items": [_plain_scalar("Count", "int", 2), - _plain_scalar("Item", "string", "cd-a"), - _plain_scalar("Item", "string", "cd-b")]}) - items.append(self.generic_frame("cdplayer")) - items.append(self.generic_frame("cdai")) - items.append(self.generic_frame("cdai")) - return items - def build_fixture(padding="joint", seed=1): """-> (inflated bytes, expected typed dict)""" diff --git a/verify/save-reader/test_save_reader.py b/verify/save-reader/test_save_reader.py index dc3766c..d1d947a 100644 --- a/verify/save-reader/test_save_reader.py +++ b/verify/save-reader/test_save_reader.py @@ -3,8 +3,9 @@ /usr/bin/python3 -m unittest -v test_save_reader -NOTE: these prove the reader against its OWN framing assumptions. Validation -against a real Sword of the Stars save is still pending. +NOTE: these prove the reader against its OWN framing assumptions (the synthetic +writer). The real-save check is `save_reader.py verify/results/saves/turn*-state.sav +--strict` (exit 0 on all three as of the schema-gaps-resolved.md patch). """ from __future__ import annotations @@ -248,7 +249,11 @@ class SchemaRoundTripTest(unittest.TestCase): self.assertIsInstance(ship["RefCap"], float) self.assertIn("BQ2", ship) self.assertIn("pop", ship) - colors = [p["slot"]["fxCrId"] for p in res.typed["summary"]["players"]] + self.assertEqual(len(res.typed["cdTable"]["ids"]), 2) + self.assertEqual(len(res.typed["customData"]), 2) # Repeat(CD) at the root + self.assertEqual(len(sim["species"]), 2) # Repeat(ISsp/ISsu) without a count + self.assertEqual(len(sim["turnstats"]["players"][0]["hist"]["stats"]), 2) + colors = [p["Slot"]["FxCrID"] for p in res.typed["summary"]["Players"]] custom = [c for c in colors if c["idx"] == -1] self.assertTrue(custom and all("r" in c for c in custom)) # -1 -> RGB follows self.assertTrue(all("r" not in c for c in colors if c["idx"] != -1)) @@ -304,18 +309,104 @@ class SchemaRoundTripTest(unittest.TestCase): self.assertTrue(any(i.level == "warn" for i in res.issues)) def test_coerce_retypes_words(self): - """A schema float stored as a word the walker guessed 'int' is re-read from raw bytes.""" + """A 4-byte word the walker guessed one way is re-read from its raw bytes + when the schema (matched positionally, so the walker had no hint) says otherwise.""" w = sw.SaveWriter() - w.begin("Tmrs") - w.int("tstl", 0x7F7FFFFF) - w.float("tctl", 60.0) # 0x42700000, walker may call it float; schema says int - w.int("tqtl", 0x7F7FFFFF) - w.int("tqtle", 0) + w.begin("Wd") + w.float("zzza", 60.0) # 0x42700000: guessed float; schema says int + w.int("zzzb", 0x7F7FFFFF) # guessed int; schema says float (FLT_MAX) w.end() - schema = sr.Seq([sr.R("t", sr.Tmrs)]) - res = sr.read_bytes(w.bytes(), padding="joint", schema=schema) - self.assertEqual(res.typed["t"]["tctl"], 0x42700000) - self.assertEqual(res.typed["t"]["tstl"], 0x7F7FFFFF) + shape = sr.Shape("Wd", [sr.R("a", "int"), sr.R("b", "float"), sr.Rest()]) + res = sr.read_bytes(w.bytes(), padding="joint", schema=sr.Seq([sr.R("t", shape)])) + self.assertEqual(res.typed["t"]["a"], 0x42700000) + self.assertEqual(res.typed["t"]["b"], 3.4028234663852886e+38) + self.assertEqual(res.count("error"), 0) + + def test_non_ascii_string_value_never_vetoes_layout(self): + """Gap 1+2: a 1-byte bool followed by a bool and a cp1252 string with a + byte in 0x80-0x9f (0x92 = right single quote) must parse exactly like an + ASCII name -- the text test applies to tags, never to string values.""" + for name in ("Kor’Voth", "Kaa’Vaalu", "Plain"): + w = sw.SaveWriter() + w.begin("Tst") # unknown frame: hints come from the catalog by name + w.int("haltc", 1) + w.int("haltt", 0) + w.bool("haltv", False) # 5-char tag + 1 byte -> 2 pad bytes (joint) + w.bool("vnh", False) + w.string("Name", name) # encodes 0x92 for the quote + w.int("VFlags", 0) + w.end() + data = w.bytes() + if "’" in name: + self.assertIn(b"\x92", data) + wk, root = walk(data) + got = [(c.name, c.kind, c.value) for c in root.children[0].children] + self.assertEqual(got, [("haltc", "int", 1), ("haltt", "int", 0), ("haltv", "bool", False), + ("vnh", "bool", False), ("Name", "string", name), ("VFlags", "int", 0)]) + self.assertEqual(wk.stats["resyncs"], 0) + self.assertEqual(wk.stats["hint_failures"], 0) + # same guarantee for an unhinted (guessed) tag holding non-ASCII text + w = sw.SaveWriter() + w.bool("zzq", True) + w.string("zzs", "Kor’Voth") + w.int("zzz", 5) + wk, root = walk(w.bytes()) + self.assertEqual([(c.kind, c.value) for c in root.children], + [("bool", True), ("string", "Kor’Voth"), ("int", 5)]) + self.assertEqual(wk.stats["resyncs"], 0) + + def test_empty_string_values(self): + """Gap 5: an empty string is 4 zero bytes, byte-identical to int 0. It + must be read as "" both when the tag is hinted (walker) and when the + walker had no hint and guessed int (applier coerce).""" + w = sw.SaveWriter() + w.begin("CreateParams") + w.string("Name", "g") + w.int("ID", 1); w.int("RSeed", 2); w.int("AID", 3) + w.string("Key", "") # 03 00 00 00 'Key' 00 00 00 00 + 1 pad + w.end() + data = w.bytes() + self.assertIn(b"\x03\x00\x00\x00Key\x00\x00\x00\x00\x00", data) + wk, root = walk(data) # hinted: CreateParams is in the catalog + key = root.children[0].children[4] + self.assertEqual((key.name, key.kind, key.value, key.hinted), ("Key", "string", "", True)) + self.assertEqual(wk.issues, []) + # unhinted: an unknown frame name -> walker guesses int 0; positional schema wants a string + w = sw.SaveWriter() + w.begin("Unknown") + w.string("zzk", "") + w.string("zzm", "") + w.end() + shape = sr.Shape("Unknown", [sr.R("k", "string"), sr.R("m", "string"), sr.Rest()]) + res = sr.read_bytes(w.bytes(), padding="joint", schema=sr.Seq([sr.R("u", shape)]), strict=True) + self.assertEqual((res.typed["u"]["k"], res.typed["u"]["m"]), ("", "")) + self.assertEqual(res.count("error"), 0) + + def test_prisoner_hold_gating(self): + """Gap 4: PrNSp and the (PrSp, PrNum) pairs are written only when PrMax > 0.""" + def prish(pr_max, pairs): + w = sw.SaveWriter() + w.begin("PrisH") + w.int("PrMax", pr_max) + if pr_max > 0: + w.int("PrNSp", len(pairs)) + for sp, num in pairs: + w.int("PrSp", sp) + w.int("PrNum", num) + w.end() + return sr.read_bytes(w.bytes(), padding="joint", strict=True, + schema=sr.Seq([sr.R("p", sr.PrisonerHold)])).typed["p"] + empty = prish(0, []) + self.assertEqual(empty["PrMax"], 0) + self.assertNotIn("prisoners", empty) + full = prish(40, [(1, 12), (3, 5)]) + self.assertEqual(full["PrMax"], 40) + self.assertEqual(full["prisoners"], [{"PrSp": 1, "PrNum": 12}, {"PrSp": 3, "PrNum": 5}]) + # a PrMax>0 frame with the pairs missing is a real schema error + w = sw.SaveWriter() + w.begin("PrisH"); w.int("PrMax", 40); w.end() + with self.assertRaises(sr.SaveFormatError): + sr.read_bytes(w.bytes(), padding="joint", strict=True, schema=sr.Seq([sr.R("p", sr.PrisonerHold)])) class CliTest(unittest.TestCase):