verified: save_reader strict-clean on 3 real saves; all core layouts confirmed; SAVE_FORMAT confirmed
This commit is contained in:
alex
parent
5cea675b45
commit
426df0232a
12 changed files with 1410 additions and 295 deletions
|
|
@ -9,11 +9,11 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`).
|
|||
| `.gob` format | subsystem | mapped | high | 100% | 2026-09-07 | renamed uncompressed ZIP (community-known) |
|
||||
| `Mars::AppStartup` (entry) | control-flow | mapped | high | 100% | 2026-09-07 | entry 0x00925794 -> WinMain 0x0089dd30 -> DemoApp ctor 0x0089c950 -> Mars::Application::Initialize 0x008a0e50 (config/affinity, D3D9, window, sound thread, OnStartup -> net thread). findings/control-flow/turn-spine.md |
|
||||
| main loop / tick dispatch | control-flow | mapped | high | 90% | 2026-09-07 | Application::Run 0x0089f5b0: FrameTimer, PanelManager, DemoApp::OnUpdate 0x00898800 / OnTick 0x0089a640 / OnRender 0x00899210. Turn pipeline: EndTurn 0x00783be0 -> BeginProcessTurn 0x007d98e0 -> StrategyServer::ProcessTurn 0x007dc6c0 -> RunCombatRound 0x007cbe80 -> SETurnResults -> ResumePlaying 0x007ddc90. Lockstep on every machine |
|
||||
| `Game::ClientPlayer` / `AIPlayer` | object | verified | high | 85% | 2026-09-07 | ServerPlayer layout confirmed vs real saves (names, species, home, savings, designs coherent across turns 1-3). Gaps: none in player block; findings/objects/struct-recovery.md |
|
||||
| `Game::StarSystem` | object | verified | high | 95% | 2026-09-07 | ServerSystem layout CONFIRMED: the 'conditional VFlags' gap was a reader bug (cp1252 in Name); haltv is a bool; NVO nodes write indi unconditionally. findings/objects/schema-gaps-resolved.md |
|
||||
| `Game::ClientPlayer` / `AIPlayer` | object | verified | high | 100% | 2026-09-07 | ServerPlayer confirmed strict vs real saves (turns 1-3) |
|
||||
| `Game::StarSystem` | object | verified | high | 100% | 2026-09-07 | ServerSystem confirmed strict vs real saves; NVO.indi unconditional; haltv bool; cp1252 names |
|
||||
| `Game::Planet` / `DOPlanet` | object | mapped | high | 100% | 2026-09-07 | Planet : Actor is a RENDER actor, not streamed; all colony state is in ServerSystem (+PlayerView in NVs map) |
|
||||
| `Game::ShipDesignDef` / `DesignList` | object | in-progress | low | 30% | 2026-09-07 | R5 breadth: design/assembly rules + validator derived from section/weapon/turret data, tested vs stock designs in real saves (verify/design-rules/); vft@0x009fef64 leaf |
|
||||
| `Game::FleetLayout` | object | verified | high | 85% | 2026-09-07 | StarFleet/StarShip confirmed vs real saves; gap: PrisonerHold `prisoners` entries lack `PrNSp` (conditional/absent). Round-4 target |
|
||||
| `Game::FleetLayout` | object | verified | high | 100% | 2026-09-07 | StarFleet/StarShip confirmed strict vs real saves; PrisonerHold PrNSp gated on PrMax>0 |
|
||||
| Mars entity chain | object | mapped | high | 100% | 2026-09-07 | EntityBase@0x00a36fd0 (10) -> PhysicalEntity@0x00a3710c (12) -> TargetableEntity -> Actor@0x009e2114 (36); mixins RigidBody/Body/NetworkObject/HandleObject/IScriptable |
|
||||
| `Game::GameCombatSim` + `CombatCommand*` | subsystem | backlog | — | 0% | 2026-09-07 | tactical combat (~40 command types) |
|
||||
| D3D9 device init | subsystem | backlog | — | 0% | 2026-09-07 | `Mars::DrawDevice`/`Camera` → Direct3DCreate9/CreateDevice |
|
||||
|
|
@ -27,9 +27,9 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`).
|
|||
| networking (SNM/FNM + GameSpy) | subsystem | backlog | — | 0% | 2026-09-07 | ~60 SNM strategy msgs, FNM file-xfer/host-migration; lockstep sim |
|
||||
| battle-load: thread contention | subsystem | in-progress | med | 50% | 2026-09-07 | time-boxed profile running; on landing: record verdict as a finding and DROP from active backlog (not on the reimpl critical path) |
|
||||
| UI screen & flow map | meta | mapped | high | 100% | 2026-09-07 | findings/subsystems/ui-screen-map.md - 36 screens; screens are C++ on Mars controls (NOT data); turn state machine recovered |
|
||||
| `Game::StrategyServer` (sim block) | object | mapped | high | 95% | 2026-09-07 | member table complete (47 rows) incl. ID-list sub-writer FUN_00794cd0, GOWinPly/sprjs/RNG/creps/turnstats/invasions; PlayerTurnStats + Invasion structs in Ghidra. Verify: strict pass pending reader patch |
|
||||
| `Game::StrategyServer` (sim block) | object | verified | high | 100% | 2026-09-07 | full member table (47 rows) confirmed vs real saves under strict parse incl. ID lists, GOWinPly, invasions, zdsc pairs, SvSctOb (present in all 3 saves) |
|
||||
| stream primitive API | subsystem | mapped | high | 100% | 2026-09-07 | IStreamable vft: +0x18 string, +0x1c bool, +0x20 float, +0x24 int, +0x28 nested, +0x30 raw; FUN_00816490 = NetworkObject handle id |
|
||||
| real save for verification | verify | verified | high | 100% | 2026-09-07 | 6 saves, 3 distinct turn states in verify/results/saves/ (byte-deterministic per state). Game runs on VM140 via DXVK 3.1 + lavapipe; recipe findings/subsystems/running-the-game.md |
|
||||
| save_reader.py | verify | verified | high | 90% | 2026-09-07 | parses all 3 real saves end-to-end (padding=joint confirmed); --strict exit 2 with a precise gap list: verify/results/saves/turn2-strict-issues.txt |
|
||||
| save_reader.py | verify | verified | high | 100% | 2026-09-07 | --strict exit 0 on all 3 real saves; 0 resyncs / 0 hint-failures; only raw = RNG blob (2503 B). 29 tests. SAVE_FORMAT.md confirmed |
|
||||
| Ghidra type write-back | meta | verified | high | 100% | 2026-09-07 | structs saved in project (ServerSystem 87f, ServerPlayer 110f, StarFleet, StarShip, StrategyServer partial, 22 nested); 52 serializers + primitives + ~60 spine fns renamed; decompile shows field names |
|
||||
| strategic turn internals (economy/research/colony/movement/diplomacy) | subsystem | in-progress | — | 0% | 2026-09-07 | R5 breadth: formulas + constant->code map from ServerPlayer/ServerSystem::ProcessTurn, TechTree::ProcessResearch, MoveFleet |
|
||||
|
|
|
|||
|
|
@ -25,3 +25,4 @@ Each links to the finding that raised it. Promoted to backlog or closed by **re-
|
|||
- **Save framing ambiguities (settle on first real save)** — padding joint `[len][name][value][pad]` vs split; bool vs int for names with len%4==0 (no type byte); on-disk tags for Summary/CreateParameters and count/element tags inside framed arrays unknown (positional for now). A real 3-char bool tag (`NPC`,`Dep`,`hsp`) settles padding. (from [[SAVE_FORMAT]])
|
||||
- **Strict-parse gaps vs real save (verifier output, round 3)** — (1) `ServerSystem`: 7/28 systems carry `VFlags` where `Name`/`vnh` were expected — a conditional layout (uncolonized/special systems?); (2) `halt[].haltv` is a 31/35-byte struct, not a bool; (3) optional `indi` (IndependenceInfo?) precedes `NVE`/`PID` in some systems; (4) `PrisonerHold.prisoners[]` entries lack `PrNSp`; (5) `/createParams/key` is int; (6) `turnstats.hist.stats.tch` is int; (7) optional trailing `zdsi`/`zdst`. Padding = **joint** (settled). 7 resyncs / 2728 raw bytes ≈ RNG blob. (from [[turn2-strict-issues]])
|
||||
- **RESOLVED: strict-parse gaps** — 5/8 were reader bugs (ASCII plausibility test applied to string values; empty string == int 0), 3 real: `PrNSp` only when `PrMax>0`; `indi` unconditional in `NVO` nodes (system-level gated by `hindi`); `zdsi/zdst` pairs under `zdsc`. `Key` is an empty string; `tch` int. Summary/CreateParams tag lists confirmed. Reader patch in flight. (from [[schema-gaps-resolved]])
|
||||
- **RESOLVED: save framing** — padding is joint; `"."` = NULL tag (VectorHelper count+elements, Vector3 bodies); empty strings are 4 zero bytes; `CDT` is a tagged frame + opaque `CD` frames. All three real saves parse `--strict` clean. Note: `SvSctOb` IS present in all saves (patch doc §7 was wrong). (from [[SAVE_FORMAT]])
|
||||
|
|
|
|||
88
verify/design-rules/data/HiverInvasion_FleetTemplates.csv
Normal file
88
verify/design-rules/data/HiverInvasion_FleetTemplates.csv
Normal file
|
|
@ -0,0 +1,88 @@
|
|||
# fleet-template-name,command,mission,engine,count
|
||||
|
||||
Princess Prime Fleet,DNBattleBridge,DNGuard,DNLongRangeFusion,1
|
||||
Princess Prime Fleet,DNAssault,DNBarrage,DNLongRangeFusion,2
|
||||
Princess Prime Fleet,CRAssault,CRBarrage,CRLongRangeFusion,5
|
||||
Princess Prime Fleet,CRBattleBridge,CRArmor,CRLongRangeFusion,5
|
||||
Princess Prime Fleet,CRAssault,CRRefinery,CRLongRangeFusion,4
|
||||
Princess Prime Fleet,CRHammerhead,CRBiomeColonizer,CRLongRangeFusion,8
|
||||
Princess Prime Fleet,CRHammerhead,CRRepairandSalvage,CRLongRangeFusion,6
|
||||
Princess Prime Fleet,CRAssault,CRStrikeForceCNC,CRLongRangeFusion,2
|
||||
Princess Prime Fleet,CRDeepScan,CRArmor,CRLongRangeFusion,2
|
||||
Princess Prime Fleet,CRAssault,CRGate,CRLongRangeFusion,5
|
||||
Princess Prime Fleet,DEHammerhead,DEArmor,DELongRangeFusion,15
|
||||
Princess Prime Fleet,DEHammerhead,DETorpedo,DELongRangeFusion,10
|
||||
Princess Prime Fleet,DEHammerhead,DESpinalMount,DELongRangeFusion,10
|
||||
Princess Prime Fleet,DEHammerhead,DEPointDefence,DELongRangeFusion,5
|
||||
Princess Prime Fleet,DEDeepScan,DEJammer,DELongRangeFusion,5
|
||||
Princess Prime Fleet,DEHammerhead,DEAssaultShuttle,DELongRangeFusion,15
|
||||
|
||||
Princess Beta Fleet,DNAssault,DNArmadaCNC,DNLongRangeFusion,1
|
||||
Princess Beta Fleet,CRAssault,CRBarrage,CRLongRangeFusion,5
|
||||
Princess Beta Fleet,CRBattleBridge,CRArmor,CRLongRangeFusion,5
|
||||
Princess Beta Fleet,CRHammerhead,CRArmor,CRLongRangeFusion,5
|
||||
Princess Beta Fleet,CRAssault,CRRefinery,CRLongRangeFusion,3
|
||||
Princess Beta Fleet,CRHammerhead,CRBiomeColonizer,CRLongRangeFusion,8
|
||||
Princess Beta Fleet,CRHammerhead,CRRepairandSalvage,CRLongRangeFusion,4
|
||||
Princess Beta Fleet,CRAssault,CRStrikeForceCNC,CRLongRangeFusion,2
|
||||
Princess Beta Fleet,CRDeepScan,CRArmor,CRLongRangeFusion,2
|
||||
Princess Beta Fleet,CRAssault,CRGate,CRLongRangeFusion,3
|
||||
Princess Beta Fleet,DEHammerhead,DEArmor,DELongRangeFusion,18
|
||||
Princess Beta Fleet,DEHammerhead,DETorpedo,DELongRangeFusion,10
|
||||
Princess Beta Fleet,DEHammerhead,DESpinalMount,DELongRangeFusion,10
|
||||
Princess Beta Fleet,DEHammerhead,DEPointDefence,DELongRangeFusion,8
|
||||
Princess Beta Fleet,DEDeepScan,DEJammer,DELongRangeFusion,5
|
||||
Princess Beta Fleet,DEHammerhead,DEAssaultShuttle,DELongRangeFusion,15
|
||||
Princess Beta Fleet,DECommand,DEColonizer,DELongRangeFusion,10
|
||||
Princess Beta Fleet,DECommand,DETanker,DELongRangeFusion,5
|
||||
Princess Beta Fleet,DEHammerhead,DEGate,DELongRangeFusion,5
|
||||
|
||||
Guardian Fleet One,CRAssault,CRBarrage,CRLongRangeFusion,3
|
||||
Guardian Fleet One,CRBattleBridge,CRArmor,CRLongRangeFusion,3
|
||||
Guardian Fleet One,CRHammerhead,CRArmor,CRLongRangeFusion,5
|
||||
Guardian Fleet One,CRAssault,CRRefinery,CRLongRangeFusion,3
|
||||
Guardian Fleet One,CRHammerhead,CRBiomeColonizer,CRLongRangeFusion,4
|
||||
Guardian Fleet One,CRHammerhead,CRRepairandSalvage,CRLongRangeFusion,2
|
||||
Guardian Fleet One,CRAssault,CRStrikeForceCNC,CRLongRangeFusion,2
|
||||
Guardian Fleet One,CRDeepScan,CRArmor,CRLongRangeFusion,2
|
||||
Guardian Fleet One,CRAssault,CRGate,CRLongRangeFusion,3
|
||||
Guardian Fleet One,DEHammerhead,DEArmor,DELongRangeFusion,20
|
||||
Guardian Fleet One,DEHammerhead,DETorpedo,DELongRangeFusion,10
|
||||
Guardian Fleet One,DEHammerhead,DESpinalMount,DELongRangeFusion,10
|
||||
Guardian Fleet One,DEHammerhead,DEPointDefence,DELongRangeFusion,5
|
||||
Guardian Fleet One,DEDeepScan,DEJammer,DELongRangeFusion,5
|
||||
Guardian Fleet One,DEHammerhead,DEAssaultShuttle,DELongRangeFusion,20
|
||||
Guardian Fleet One,DECommand,DEColonizer,DELongRangeFusion,10
|
||||
Guardian Fleet One,DECommand,DETanker,DELongRangeFusion,10
|
||||
Guardian Fleet One,DEHammerhead,DEGate,DELongRangeFusion,5
|
||||
|
||||
Scout Fleet Alpha,CRHammerhead,CRArmor,CRLongRangeFusion,1
|
||||
Scout Fleet Alpha,CRAssault,CRRefinery,CRLongRangeFusion,1
|
||||
Scout Fleet Alpha,CRHammerhead,CRRepairandSalvage,CRLongRangeFusion,1
|
||||
Scout Fleet Alpha,CRAssault,CRStrikeForceCNC,CRLongRangeFusion,1
|
||||
Scout Fleet Alpha,DEHammerhead,DEArmor,DELongRangeFusion,20
|
||||
Scout Fleet Alpha,DEHammerhead,DETorpedo,DELongRangeFusion,5
|
||||
Scout Fleet Alpha,DEHammerhead,DESpinalMount,DELongRangeFusion,5
|
||||
Scout Fleet Alpha,DEHammerhead,DEPointDefence,DELongRangeFusion,3
|
||||
Scout Fleet Alpha,DEDeepScan,DEJammer,DELongRangeFusion,5
|
||||
Scout Fleet Alpha,DEHammerhead,DEAssaultShuttle,DELongRangeFusion,10
|
||||
Scout Fleet Alpha,DECommand,DEColonizer,DELongRangeFusion,10
|
||||
Scout Fleet Alpha,DECommand,DETanker,DELongRangeFusion,10
|
||||
Scout Fleet Alpha,DEHammerhead,DEGate,DELongRangeFusion,10
|
||||
|
||||
Mining Fleet One,CRHammerhead,CRStrikeForceCNC,CRLongRangeFusion,1
|
||||
Mining Fleet One,CRAssault,CRMining,CRLongRangeFusion,3
|
||||
Mining Fleet One,CRBattleBridge,CRMining,CRLongRangeFusion,3
|
||||
Mining Fleet One,CRAssault,CRRefinery,CRLongRangeFusion,1
|
||||
Mining Fleet One,DEDeepScan,DEJammer,DELongRangeFusion,2
|
||||
Mining Fleet One,DEHammerhead,DEGate,DELongRangeFusion,5
|
||||
|
||||
Explorer Fleet One,CRRamScoop,CRStrikeForceCNC,CRFusion,1
|
||||
Explorer Fleet One,CRRamScoop,CRArmor,CRFusion,5
|
||||
Explorer Fleet One,CRRamScoop,CRBiomeColonizer,CRFusion,3
|
||||
Explorer Fleet One,CRRamScoop,CRRefinery,CRFusion,1
|
||||
Explorer Fleet One,DERamScoop,DEArmor,DEFusion,15
|
||||
Explorer Fleet One,DERamScoop,DETorpedo,DEFusion,5
|
||||
Explorer Fleet One,DERamScoop,DEPointDefence,DEFusion,3
|
||||
Explorer Fleet One,DERamScoop,DEGate,DEFusion,10
|
||||
Explorer Fleet One,DERamScoop,DEColonizer,DEFusion,4
|
||||
|
12
verify/design-rules/data/UpstartApes_GiftFleetTemplates.csv
Normal file
12
verify/design-rules/data/UpstartApes_GiftFleetTemplates.csv
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
# fleet-template-name,command,mission,engine,count
|
||||
|
||||
Gift Fleet A,DECommand,DEArmor,DEFusion,6
|
||||
|
||||
Gift Fleet B,CRDeepScan,CRExtendedRange,CRFusion,2
|
||||
|
||||
Gift Fleet C,CRAssault,CRBarrage,CRFusion,2
|
||||
|
||||
Gift Fleet D,DEHammerHead,DEArmor,DEFusion,12
|
||||
Gift Fleet D,DEDeepScan,DESquadronCNC,DEFusion,1
|
||||
|
||||
Gift Fleet E,CRDeepScan,CRExtendedRange,CRFusion,10
|
||||
|
34
verify/design-rules/data/_defaultweapons.txt
Normal file
34
verify/design-rules/data/_defaultweapons.txt
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
// Selection of default weapons for turrets in the absence of tech and player information.
|
||||
// <mount-class> <mount-size> <weapon-file>
|
||||
|
||||
standard tiny "las_pd.weapon"
|
||||
standard small "las_red.weapon"
|
||||
standard medium "las_red.weapon"
|
||||
standard large "can_plasma.weapon"
|
||||
missile medium "mis.weapon"
|
||||
missile large "mis.weapon"
|
||||
planetmissile large "mis_defplat.weapon"
|
||||
beam large "hvy_bem_lancer.weapon"
|
||||
spinal large "bem_part_spinal.weapon"
|
||||
torpedo large "trp_plasma.weapon"
|
||||
mine large "min_nuke.weapon"
|
||||
missilerider large "bio_plague.weapon"
|
||||
nodemissilerider large "mis_node.weapon"
|
||||
assaultshuttlerider large "brd_shuttle.weapon"
|
||||
siege large "bal_siege.weapon"
|
||||
prisonershuttlerider large "brd_prisonershuttle.weapon"
|
||||
projector large "can_prjplasma.weapon"
|
||||
freebeam large "hvy_bem_lancer_free.weapon"
|
||||
boardingpod large "brd_boardingpod.weapon"
|
||||
wraith large "wraith.weapon"
|
||||
nodecannon large "nodecannon.weapon"
|
||||
dronerider large "brd_drone.weapon"
|
||||
col large "col_flock.weapon"
|
||||
spyship large "spyship.weapon"
|
||||
tarkahunter large "brd_tarkahunter.weapon"
|
||||
grapple medium "bal_grapple.weapon"
|
||||
grapple large "bal_grapple.weapon"
|
||||
strafe small "las_red.weapon"
|
||||
strafe medium "las_red.weapon"
|
||||
strafe large "las_red.weapon"
|
||||
railcannon large "bal_railcannon.weapon"
|
||||
44
verify/design-rules/data/_turrets.txt
Normal file
44
verify/design-rules/data/_turrets.txt
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
// size weapon-size class health track-speed constrain-azimuth-% constrain-inclination-% "modelname"
|
||||
|
||||
small tiny standard 20 360 1.00 1.00 "turret_pd.x"
|
||||
small small standard 120 320 1.00 1.00 "turret_s1standard.x"
|
||||
medium small standard 350 180 1.00 1.00 "turret_m2standard.x"
|
||||
large small standard 800 120 1.00 1.00 "turret_l3standard.x"
|
||||
medium medium standard 500 140 1.00 1.00 "turret_m1standard.x"
|
||||
large medium standard 950 100 1.00 1.00 "turret_l2standard.x"
|
||||
large large standard 1400 80 1.00 1.00 "turret_l1standard.x"
|
||||
small small strafe 120 320 1.00 1.00 "turret_s1standard.x"
|
||||
medium small strafe 350 180 1.00 1.00 "turret_m2standard.x"
|
||||
large small strafe 800 120 1.00 1.00 "turret_l3standard.x"
|
||||
medium medium strafe 500 140 1.00 1.00 "turret_m1standard.x"
|
||||
large medium strafe 950 100 1.00 1.00 "turret_l2standard.x"
|
||||
large large strafe 1400 80 1.00 1.00 "turret_l1standard.x"
|
||||
small small grapple 120 320 1.00 1.00 "turret_s1standard.x"
|
||||
medium small grapple 350 180 1.00 1.00 "turret_m2standard.x"
|
||||
large small grapple 800 120 1.00 1.00 "turret_l3standard.x"
|
||||
medium medium grapple 500 140 1.00 1.00 "turret_m1standard.x"
|
||||
large medium grapple 950 100 1.00 1.00 "turret_l2standard.x"
|
||||
large large grapple 1400 80 1.00 1.00 "turret_l1standard.x"
|
||||
medium medium missile 400 100 1.00 0.25 "turret_m1missile.x"
|
||||
large medium missile 950 100 1.00 0.25 "turret_l2missile.x"
|
||||
large large missile 1000 100 1.00 0.25 "turret_l1missile.x"
|
||||
large medium planetmissile 1000 100 1.00 1.00 "turret_l2icbm.x"
|
||||
large large beam 1500 80 1.00 1.00 ""
|
||||
large large freebeam 1700 20 1.00 1.00 "turret_l1freebeam.x"
|
||||
large large spinal 800 100 1.00 1.00 ""
|
||||
large large torpedo 800 100 1.00 1.00 ""
|
||||
large large mine 800 100 1.00 1.00 ""
|
||||
large large missilerider 800 100 1.00 1.00 ""
|
||||
large large assaultshuttlerider 800 100 1.00 1.00 ""
|
||||
large large prisonershuttlerider 800 100 1.00 1.00 ""
|
||||
large large nodemissilerider 800 100 1.00 1.00 ""
|
||||
large large siege 800 100 1.00 1.00 ""
|
||||
large large projector 2600 100 1.00 1.00 "turret_l1projector.x"
|
||||
large large boardingpod 800 100 1.00 1.00 ""
|
||||
large large wraith 800 100 1.00 1.00 ""
|
||||
large large nodecannon 800 100 1.00 1.00 ""
|
||||
large large dronerider 800 100 1.00 1.00 ""
|
||||
large large col 800 100 1.00 1.00 ""
|
||||
large large spyship 800 100 1.00 1.00 ""
|
||||
large large railcannon 1600 100 1.00 1.00 ""
|
||||
large large tarkahunter 800 100 1.00 1.00 ""
|
||||
757
verify/design-rules/design_rules.py
Normal file
757
verify/design-rules/design_rules.py
Normal file
|
|
@ -0,0 +1,757 @@
|
|||
"""design_rules.py -- SOTS1 (2006) ship-design assembly rules, derived from data.
|
||||
|
||||
Stdlib only. Loads the normalized catalogs written by verify/parsers/verify.py
|
||||
(shipsections.json, weapons.json, tech_tree.json) plus the two positional
|
||||
tables that live next to this file (data/_turrets.txt, data/_defaultweapons.txt)
|
||||
and exposes:
|
||||
|
||||
cat = Catalog.load() # default paths, cached
|
||||
validate(design, cat=None) -> [Violation]
|
||||
derive_stats(design, cat=None) -> dict
|
||||
applied_techs(section_use, sdef, cat) # == the save's DOpts list
|
||||
design_from_save(species_idx, slots, ..) # save-reader record -> Design
|
||||
|
||||
Design (plain dict; every key optional except race/mission):
|
||||
|
||||
{
|
||||
"name": "Armor",
|
||||
"race": "Human", # catalog species (Human/Hiver/Tarkas/Liir/Zuul/Morrigi/_NPC)
|
||||
"hidden": False, # engine-generated hidden design (rider defaults)
|
||||
"command": {"section": "DECommand", "weapons": ["bal_gauss"], "options": []},
|
||||
"mission": {"section": "DEArmor", "weapons": ["bal_gauss", "bal_gauss", "mis"], "options": []},
|
||||
"engine": {"section": "DEFission", "weapons": ["bal_gauss"], "options": []},
|
||||
"known_techs": ["DRV_Fissn", ...], # omit/None -> tech gating is skipped
|
||||
}
|
||||
|
||||
`weapons` holds one entry per `bank{}` block of the section, in file order; an
|
||||
entry is a weapon stem ("bal_gauss"), a weapon file path, an int id from
|
||||
_weapons.txt, or None (empty). `options` lists the chosen option techs.
|
||||
|
||||
Rule ids (see SHIP_DESIGN_RULES.md for the evidence behind each):
|
||||
A* structure/slots, B* banks & weapon fit, C* tech gating, D* options,
|
||||
E* derived stats. Violation.level is "error" (design is invalid),
|
||||
"warn" (valid for the engine but not something the designer UI offers /
|
||||
something only code can settle) or "info".
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
|
||||
__all__ = [
|
||||
"Catalog", "Violation", "validate", "derive_stats", "applied_techs",
|
||||
"design_from_save", "weapon_fits_bank", "SPECIES_INDEX", "SLOTS",
|
||||
"HULL_CLASS_TECH", "BANK_CLASS_ACCEPTS",
|
||||
]
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
DEFAULT_CATALOG_DIR = HERE.parent / "results" / "data-catalogs"
|
||||
DEFAULT_DATA_DIR = HERE / "data"
|
||||
|
||||
# Order of the species table in a real save (sim.species ISsp list); the
|
||||
# design record's DSec[0] is an index into it.
|
||||
SPECIES_INDEX = ["Human", "Hiver", "Tarkas", "Liir", "_NPC", "Zuul", "Morrigi"]
|
||||
|
||||
SLOTS = ("command", "mission", "engine")
|
||||
|
||||
# Hull-size techs implied by section_class (evidence: the save's DOpts list
|
||||
# carries them for every cruiser/dreadnought section that is not a station).
|
||||
HULL_CLASS_TECH = {"cruiser": "IND_CruisCon", "dreadnought": "IND_DreadCon"}
|
||||
|
||||
# Which weapon turretclass a bank turretclass accepts besides an exact match.
|
||||
# standard <- missile : stock designs put `mis` (medium/missile) in the
|
||||
# medium/standard bank of DEArmor and DEDefencePlatform
|
||||
# standard <- grapple : bal_grapple/bal_disruptorwhip are turretclass grapple
|
||||
# but no section has a grapple bank; _turrets.txt has
|
||||
# grapple rows for medium/large only
|
||||
# strafe <- standard: no weapon has turretclass strafe; the NPC
|
||||
# _HeraldDefender strafe bank carries a standard-class
|
||||
# beamer; _turrets.txt strafe rows mirror standard rows
|
||||
BANK_CLASS_ACCEPTS = {
|
||||
"standard": {"standard", "missile", "grapple"},
|
||||
"strafe": {"standard"},
|
||||
}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# helpers
|
||||
|
||||
def _last(v):
|
||||
"""Scalar from a catalog value; repeated keys were folded into lists."""
|
||||
if isinstance(v, list):
|
||||
return v[-1] if v else None
|
||||
return v
|
||||
|
||||
|
||||
def _list(v) -> list:
|
||||
if v is None:
|
||||
return []
|
||||
return v if isinstance(v, list) else [v]
|
||||
|
||||
|
||||
def _low(v) -> str | None:
|
||||
return None if v is None else str(v).lower()
|
||||
|
||||
|
||||
def _num(v, default=0):
|
||||
v = _last(v)
|
||||
if isinstance(v, bool):
|
||||
return default
|
||||
return v if isinstance(v, (int, float)) else default
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# catalog model
|
||||
|
||||
@dataclass
|
||||
class BankDef:
|
||||
index: int
|
||||
size: str | None # small/medium/large (lower)
|
||||
cls: str | None # standard/missile/... (lower)
|
||||
mounts: int
|
||||
fixed_weapon: str | None # NPC banks: weapon file bound in data
|
||||
showturrets: bool = True
|
||||
|
||||
|
||||
@dataclass
|
||||
class OptionGroup:
|
||||
techs: tuple # mutually exclusive choices, file order
|
||||
kind: str = "option" # "option" | "optiondef"
|
||||
|
||||
|
||||
@dataclass
|
||||
class SectionDef:
|
||||
race: str
|
||||
stem: str
|
||||
file: str
|
||||
id: int | None
|
||||
type: str | None # command/mission/engine (lower) or None
|
||||
cls: str | None # destroyer/cruiser/dreadnought (lower) or None
|
||||
socket_fore: str | None
|
||||
socket_aft: str | None
|
||||
requires: list
|
||||
option_groups: list
|
||||
banks: list
|
||||
raw: dict
|
||||
nodesign: bool = False
|
||||
explicit_section: bool = False
|
||||
explicit_command_section: str | None = None
|
||||
explicit_engine_section: str | None = None
|
||||
exclude: list = field(default_factory=list)
|
||||
design_class: str | None = None
|
||||
entity_class: str | None = None
|
||||
|
||||
@property
|
||||
def has_sockets(self) -> bool:
|
||||
return self.socket_fore is not None or self.socket_aft is not None
|
||||
|
||||
@property
|
||||
def is_station(self) -> bool:
|
||||
return _low(self.design_class) == "station"
|
||||
|
||||
def hull_class_tech(self) -> str | None:
|
||||
if self.is_station:
|
||||
return None
|
||||
return HULL_CLASS_TECH.get(self.cls or "")
|
||||
|
||||
def stat(self, key, default=None):
|
||||
return _last(self.raw.get(key, default))
|
||||
|
||||
|
||||
@dataclass
|
||||
class WeaponDef:
|
||||
stem: str
|
||||
file: str
|
||||
scope: str # player / NPC
|
||||
id: int | None
|
||||
size: str | None
|
||||
cls: str | None
|
||||
weaponclass: str | None
|
||||
requires: list
|
||||
cost: float
|
||||
exclusive_species: str | None
|
||||
compatible_section: list
|
||||
hidden: bool
|
||||
raw: dict
|
||||
|
||||
|
||||
@dataclass
|
||||
class TurretRow:
|
||||
size: str
|
||||
weapon_size: str
|
||||
cls: str
|
||||
health: float
|
||||
track_speed: float
|
||||
az: float
|
||||
inc: float
|
||||
model: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class Violation:
|
||||
rule: str
|
||||
level: str
|
||||
msg: str
|
||||
slot: str | None = None
|
||||
bank: int | None = None
|
||||
|
||||
def __str__(self):
|
||||
where = f" [{self.slot}{'' if self.bank is None else ' bank ' + str(self.bank)}]" if self.slot else ""
|
||||
return f"{self.level.upper():5} {self.rule}{where}: {self.msg}"
|
||||
|
||||
|
||||
class Catalog:
|
||||
_cache: dict = {}
|
||||
|
||||
def __init__(self, sections_json, weapons_json, tech_json, turrets_txt, defaults_txt):
|
||||
self.sections: dict[str, dict[str, SectionDef]] = {} # race -> stem.lower -> def
|
||||
self.sections_by_id: dict[str, dict[int, SectionDef]] = {}
|
||||
self.weapons: dict[str, WeaponDef] = {} # stem.lower -> def
|
||||
self.weapons_by_file: dict[str, WeaponDef] = {}
|
||||
self.weapons_by_id: dict[int, WeaponDef] = {}
|
||||
self.techs: dict[str, dict] = {} # name.lower -> node
|
||||
self.tech_names: dict[str, str] = {} # name.lower -> canonical
|
||||
self.groups: dict[str, set] = {} # group.lower -> {tech.lower}
|
||||
self.unobtainable: dict[str, set] = {} # tech.lower -> {race.lower}
|
||||
self.turrets: dict[tuple, TurretRow] = {} # (size, wsize, cls) -> row
|
||||
self.default_weapons: dict[tuple, str] = {} # (cls, size) -> weapon file
|
||||
self._load_sections(sections_json)
|
||||
self._load_weapons(weapons_json)
|
||||
self._load_techs(tech_json)
|
||||
self._load_turrets(turrets_txt)
|
||||
self._load_defaults(defaults_txt)
|
||||
|
||||
# -- loading ------------------------------------------------------------
|
||||
@classmethod
|
||||
def load(cls, catalog_dir=None, data_dir=None) -> "Catalog":
|
||||
catalog_dir = Path(catalog_dir or os.environ.get("SOTS_CATALOG_DIR") or DEFAULT_CATALOG_DIR)
|
||||
data_dir = Path(data_dir or DEFAULT_DATA_DIR)
|
||||
key = (str(catalog_dir), str(data_dir))
|
||||
if key not in cls._cache:
|
||||
cls._cache[key] = cls(
|
||||
catalog_dir / "shipsections.json", catalog_dir / "weapons.json",
|
||||
catalog_dir / "tech_tree.json", data_dir / "_turrets.txt",
|
||||
data_dir / "_defaultweapons.txt")
|
||||
return cls._cache[key]
|
||||
|
||||
def _load_sections(self, path):
|
||||
data = json.load(open(path, encoding="utf-8"))
|
||||
for s in data["sections"]:
|
||||
groups = []
|
||||
for o in _list(s.get("option")):
|
||||
if isinstance(o, dict):
|
||||
groups.append(OptionGroup(tuple(str(t) for t in _list(o.get("option")))))
|
||||
else:
|
||||
groups.append(OptionGroup((str(o),)))
|
||||
for o in _list(s.get("optiondef")):
|
||||
if isinstance(o, dict):
|
||||
groups.append(OptionGroup(tuple(str(t) for t in _list(o.get("option"))), "optiondef"))
|
||||
banks = []
|
||||
for i, b in enumerate(_list(s.get("bank"))):
|
||||
if not isinstance(b, dict):
|
||||
continue
|
||||
st = _last(b.get("showturrets"))
|
||||
banks.append(BankDef(
|
||||
index=i, size=_low(_last(b.get("turretsize"))), cls=_low(_last(b.get("turretclass"))),
|
||||
mounts=len(_list(b.get("mount"))), fixed_weapon=_last(b.get("weapon")),
|
||||
showturrets=(st is None or bool(st))))
|
||||
sd = SectionDef(
|
||||
race=s["race"], stem=s["stem"], file=s["file"], id=s.get("id"),
|
||||
type=_low(_last(s.get("section_type"))), cls=_low(_last(s.get("section_class"))),
|
||||
socket_fore=_last(s.get("socket_fore")), socket_aft=_last(s.get("socket_aft")),
|
||||
requires=[str(r) for r in _list(s.get("requires"))], option_groups=groups, banks=banks, raw=s,
|
||||
nodesign=bool(_last(s.get("nodesign", False))),
|
||||
explicit_section=bool(_last(s.get("explicit_section", False))),
|
||||
explicit_command_section=_last(s.get("explicit_command_section")),
|
||||
explicit_engine_section=_last(s.get("explicit_engine_section")),
|
||||
exclude=[str(x) for x in _list(s.get("exclude"))],
|
||||
design_class=_last(s.get("design_class")), entity_class=_last(s.get("entity_class")))
|
||||
self.sections.setdefault(sd.race, {})[sd.stem.lower()] = sd
|
||||
if sd.id is not None:
|
||||
self.sections_by_id.setdefault(sd.race, {})[int(sd.id)] = sd
|
||||
|
||||
def _load_weapons(self, path):
|
||||
data = json.load(open(path, encoding="utf-8"))
|
||||
for w in data["weapons"]:
|
||||
wd = WeaponDef(
|
||||
stem=w["stem"], file=w["file"], scope=w.get("scope", "player"), id=w.get("id"),
|
||||
size=_low(_last(w.get("turretsize"))), cls=_low(_last(w.get("turretclass"))),
|
||||
weaponclass=_low(_last(w.get("weaponclass"))),
|
||||
requires=[str(r) for r in _list(w.get("requires"))], cost=_num(w.get("cost"), 0),
|
||||
exclusive_species=_low(_last(w.get("exclusive_species"))),
|
||||
compatible_section=[str(c) for c in _list(w.get("compatible_section"))],
|
||||
hidden=bool(_last(w.get("hidden", False))), raw=w)
|
||||
self.weapons[wd.stem.lower()] = wd
|
||||
self.weapons_by_file[wd.file.lower()] = wd
|
||||
self.weapons_by_file[os.path.basename(wd.file).lower()] = wd
|
||||
if wd.id is not None:
|
||||
self.weapons_by_id[int(wd.id)] = wd
|
||||
|
||||
def _load_techs(self, path):
|
||||
data = json.load(open(path, encoding="utf-8"))
|
||||
self.races = list(data.get("races", []))
|
||||
for n in data["nodes"]:
|
||||
self.techs[n["name"].lower()] = n
|
||||
self.tech_names[n["name"].lower()] = n["name"]
|
||||
if n.get("group"):
|
||||
self.groups.setdefault(str(n["group"]).lower(), set()).add(n["name"].lower())
|
||||
for g, members in data.get("groups", {}).items():
|
||||
self.groups.setdefault(g.lower(), set()).update(m.lower() for m in members)
|
||||
# race availability: a tech is unobtainable for a race when every edge
|
||||
# into it writes 0% for that race (absent race -> engine default,
|
||||
# believed 100%).
|
||||
incoming: dict[str, list] = {}
|
||||
for e in data["edges"]:
|
||||
incoming.setdefault(e["to"].lower(), []).append(e)
|
||||
for t, edges in incoming.items():
|
||||
for race in self.races:
|
||||
if all(e.get("pct", {}).get(race) == 0 for e in edges):
|
||||
self.unobtainable.setdefault(t, set()).add(race.lower())
|
||||
|
||||
def _load_turrets(self, path):
|
||||
for row in _rows(path):
|
||||
if len(row) < 8:
|
||||
continue
|
||||
size, wsize, cls = (str(row[0]).lower(), str(row[1]).lower(), str(row[2]).lower())
|
||||
self.turrets[(size, wsize, cls)] = TurretRow(
|
||||
size, wsize, cls, float(row[3]), float(row[4]), float(row[5]), float(row[6]), str(row[7]))
|
||||
|
||||
def _load_defaults(self, path):
|
||||
if not Path(path).exists():
|
||||
return
|
||||
for row in _rows(path):
|
||||
if len(row) >= 3:
|
||||
self.default_weapons[(str(row[0]).lower(), str(row[1]).lower())] = str(row[2])
|
||||
|
||||
# -- lookups --------------------------------------------------------------
|
||||
def race_key(self, race: str) -> str | None:
|
||||
for r in self.sections:
|
||||
if r.lower() == str(race).lower():
|
||||
return r
|
||||
return None
|
||||
|
||||
def section(self, race: str, ref) -> SectionDef | None:
|
||||
r = self.race_key(race)
|
||||
if r is None or ref is None:
|
||||
return None
|
||||
if isinstance(ref, int):
|
||||
return self.sections_by_id.get(r, {}).get(ref)
|
||||
ref = str(ref)
|
||||
stem = os.path.basename(ref)
|
||||
stem = re.sub(r"\.shipsection$", "", stem, flags=re.I)
|
||||
return self.sections[r].get(stem.lower())
|
||||
|
||||
def weapon(self, ref) -> WeaponDef | None:
|
||||
if ref is None:
|
||||
return None
|
||||
if isinstance(ref, int):
|
||||
return self.weapons_by_id.get(ref)
|
||||
ref = str(ref)
|
||||
low = ref.lower()
|
||||
if low in self.weapons:
|
||||
return self.weapons[low]
|
||||
if low in self.weapons_by_file:
|
||||
return self.weapons_by_file[low]
|
||||
stem = re.sub(r"\.weapon$", "", os.path.basename(low))
|
||||
return self.weapons.get(stem) or self.weapons_by_file.get(stem + ".weapon")
|
||||
|
||||
def tech(self, name: str) -> dict | None:
|
||||
return self.techs.get(str(name).lower())
|
||||
|
||||
def tech_known(self, name: str, known: set) -> bool:
|
||||
"""`known` is a set of lower-cased tech names. GRP_<x> is satisfied by
|
||||
any known tech of group <x>."""
|
||||
low = str(name).lower()
|
||||
if low.startswith("grp_"):
|
||||
return bool(self.groups.get(low[4:], set()) & known)
|
||||
return low in known
|
||||
|
||||
|
||||
def _rows(path) -> list[list[str]]:
|
||||
"""Whitespace-positional rows with // comments and quoted tokens."""
|
||||
out = []
|
||||
tok = re.compile(r'"([^"]*)"|(\S+)')
|
||||
for raw in Path(path).read_bytes().decode("cp1252").splitlines():
|
||||
line = _strip_comment(raw).strip()
|
||||
if not line:
|
||||
continue
|
||||
out.append([m.group(1) if m.group(1) is not None else m.group(2) for m in tok.finditer(line)])
|
||||
return out
|
||||
|
||||
|
||||
def _strip_comment(line: str) -> str:
|
||||
in_q = False
|
||||
for i, c in enumerate(line):
|
||||
if c == '"':
|
||||
in_q = not in_q
|
||||
elif c == "/" and not in_q and line.startswith("//", i):
|
||||
return line[:i]
|
||||
return line
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# turret fit
|
||||
|
||||
def turret_row(cat: Catalog, bank: BankDef, weapon: WeaponDef) -> TurretRow | None:
|
||||
"""The _turrets.txt row a weapon in a bank would use: keyed by the weapon's
|
||||
class first (missile-in-standard picks the missile turret model), then by
|
||||
the bank's class (strafe banks have their own rows, no weapon is class
|
||||
strafe)."""
|
||||
for cls in (weapon.cls, bank.cls):
|
||||
r = cat.turrets.get((bank.size, weapon.size, cls))
|
||||
if r:
|
||||
return r
|
||||
return None
|
||||
|
||||
|
||||
def weapon_fits_bank(cat: Catalog, bank: BankDef, weapon: WeaponDef) -> tuple[bool, str]:
|
||||
if bank.size is None or bank.cls is None:
|
||||
return (weapon.file.lower() == str(bank.fixed_weapon or "").lower(),
|
||||
"bank has no turretsize/turretclass (fixed NPC weapon only)")
|
||||
if not (weapon.cls == bank.cls or weapon.cls in BANK_CLASS_ACCEPTS.get(bank.cls, ())):
|
||||
return False, f"turretclass {weapon.cls} not accepted by {bank.cls} bank"
|
||||
if turret_row(cat, bank, weapon) is None:
|
||||
return False, f"no _turrets.txt row for ({bank.size} bank, {weapon.size} weapon, {weapon.cls}/{bank.cls})"
|
||||
return True, "ok"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# design access
|
||||
|
||||
def _slot(design: dict, slot: str) -> dict | None:
|
||||
v = design.get(slot)
|
||||
if v is None:
|
||||
return None
|
||||
if isinstance(v, str):
|
||||
return {"section": v}
|
||||
return v
|
||||
|
||||
|
||||
def _known_set(design: dict) -> set | None:
|
||||
k = design.get("known_techs")
|
||||
if k is None:
|
||||
return None
|
||||
return {str(t).lower() for t in k}
|
||||
|
||||
|
||||
def applied_techs(use: dict, sdef: SectionDef, cat: Catalog | None = None) -> list[str]:
|
||||
"""The techs a section instance is built with -- exactly what the save
|
||||
stores per section in DOpts: hull-class tech (cruiser/dreadnought, not
|
||||
stations), then `requires` in file order, then the chosen option techs in
|
||||
option-group order."""
|
||||
out = []
|
||||
hct = sdef.hull_class_tech()
|
||||
if hct:
|
||||
out.append(hct)
|
||||
out.extend(sdef.requires)
|
||||
chosen = {str(o).lower(): str(o) for o in _list((use or {}).get("options"))}
|
||||
for g in sdef.option_groups:
|
||||
for t in g.techs:
|
||||
if t.lower() in chosen:
|
||||
out.append(chosen[t.lower()])
|
||||
return out
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# validation
|
||||
|
||||
def validate(design: dict, cat: Catalog | None = None) -> list[Violation]:
|
||||
cat = cat or Catalog.load()
|
||||
v: list[Violation] = []
|
||||
race = design.get("race")
|
||||
hidden = bool(design.get("hidden"))
|
||||
known = _known_set(design)
|
||||
rk = cat.race_key(race) if race else None
|
||||
if rk is None:
|
||||
v.append(Violation("A5", "error", f"unknown species catalog {race!r}"))
|
||||
return v
|
||||
|
||||
# -- resolve sections -----------------------------------------------------
|
||||
uses: dict[str, dict | None] = {s: _slot(design, s) for s in SLOTS}
|
||||
defs: dict[str, SectionDef | None] = {}
|
||||
for slot in SLOTS:
|
||||
use = uses[slot]
|
||||
if use is None:
|
||||
defs[slot] = None
|
||||
continue
|
||||
sd = cat.section(rk, use.get("section"))
|
||||
if sd is None:
|
||||
v.append(Violation("A5", "error", f"section {use.get('section')!r} not in {rk} catalog", slot))
|
||||
defs[slot] = sd
|
||||
if defs["mission"] is None and uses["mission"] is not None:
|
||||
return v
|
||||
m = defs["mission"]
|
||||
if m is None:
|
||||
v.append(Violation("A2", "error", "mission slot is empty"))
|
||||
return v
|
||||
|
||||
# A3 hull ship vs standalone
|
||||
standalone = not m.has_sockets
|
||||
if m.explicit_command_section or m.explicit_engine_section:
|
||||
for slot, want in (("command", m.explicit_command_section), ("engine", m.explicit_engine_section)):
|
||||
got = defs[slot]
|
||||
if want and (got is None or got.stem.lower() != want.lower()):
|
||||
v.append(Violation("A8", "error", f"{m.stem} requires {slot} section {want}, got {got.stem if got else None}", slot))
|
||||
elif standalone:
|
||||
for slot in ("command", "engine"):
|
||||
if uses[slot] is not None:
|
||||
v.append(Violation("A3", "error", f"{m.stem} has no sockets (standalone hull) but {slot} slot is filled", slot))
|
||||
else:
|
||||
for slot in ("command", "engine"):
|
||||
if defs[slot] is None:
|
||||
v.append(Violation("A3", "error", f"{m.stem} has sockets; {slot} section required", slot))
|
||||
|
||||
present = [(s, d) for s, d in defs.items() if d is not None]
|
||||
|
||||
# A4 slot/type, A9 nodesign, A8 explicit partner sections
|
||||
for slot, sd in present:
|
||||
if sd.type is None:
|
||||
if slot == "mission" and standalone and (hidden or _low(sd.design_class) == "rider"):
|
||||
v.append(Violation("A4", "info", f"{sd.stem} has no section_type (rider hull in a hidden design)", slot))
|
||||
else:
|
||||
v.append(Violation("A4", "error", f"{sd.stem} has no section_type and cannot fill the {slot} slot", slot))
|
||||
elif sd.type != slot:
|
||||
v.append(Violation("A4", "error", f"{sd.stem} is a {sd.type} section, placed in {slot} slot", slot))
|
||||
if sd.nodesign and not hidden:
|
||||
v.append(Violation("A9", "warn", f"{sd.stem} is nodesign (not offered by the design UI)", slot))
|
||||
if sd.explicit_section and slot != "mission":
|
||||
owner = m.explicit_command_section if slot == "command" else m.explicit_engine_section
|
||||
if not owner or owner.lower() != sd.stem.lower():
|
||||
v.append(Violation("A8", "error", f"{sd.stem} is an explicit_section usable only with its owning station", slot))
|
||||
|
||||
# A6 class equality
|
||||
classes = {sd.cls for _, sd in present if sd.cls}
|
||||
if len(classes) > 1:
|
||||
v.append(Violation("A6", "error", f"mixed section_class: {sorted(classes)}"))
|
||||
|
||||
# A7 exclude
|
||||
stems = {sd.stem.lower(): slot for slot, sd in present}
|
||||
for slot, sd in present:
|
||||
for ex in sd.exclude:
|
||||
if ex.lower() in stems:
|
||||
v.append(Violation("A7", "error", f"{sd.stem} excludes {ex} (in {stems[ex.lower()]} slot)", slot))
|
||||
|
||||
# -- banks & weapons ---------------------------------------------------------
|
||||
for slot, sd in present:
|
||||
use = uses[slot] or {}
|
||||
weapons = _list(use.get("weapons"))
|
||||
if len(weapons) != len(sd.banks):
|
||||
v.append(Violation("B1", "error", f"{sd.stem} has {len(sd.banks)} banks, design lists {len(weapons)} weapons", slot))
|
||||
for i, bank in enumerate(sd.banks):
|
||||
ref = weapons[i] if i < len(weapons) else None
|
||||
wd = cat.weapon(ref) if ref is not None else None
|
||||
if ref is not None and wd is None:
|
||||
v.append(Violation("B5", "error", f"unknown weapon {ref!r}", slot, i))
|
||||
continue
|
||||
if bank.fixed_weapon:
|
||||
fixed = cat.weapon(bank.fixed_weapon)
|
||||
if wd is None:
|
||||
wd = fixed
|
||||
elif fixed is not None and wd.file.lower() != fixed.file.lower():
|
||||
v.append(Violation("B3", "error", f"bank is bound to {os.path.basename(bank.fixed_weapon)}, design has {wd.stem}", slot, i))
|
||||
continue
|
||||
if wd is None:
|
||||
v.append(Violation("B2", "error", f"bank {i} ({bank.size} {bank.cls}, {bank.mounts} mounts) has no weapon", slot, i))
|
||||
continue
|
||||
ok, why = weapon_fits_bank(cat, bank, wd)
|
||||
if not ok:
|
||||
v.append(Violation("B4", "error", f"{wd.stem} ({wd.size} {wd.cls}) does not fit bank {i} ({bank.size} {bank.cls}): {why}", slot, i))
|
||||
if wd.scope != "player" and rk != "_NPC" and not bank.fixed_weapon:
|
||||
v.append(Violation("B5", "error", f"{wd.stem} is an NPC-only weapon", slot, i))
|
||||
if wd.exclusive_species and wd.exclusive_species != rk.lower():
|
||||
v.append(Violation("B6", "error", f"{wd.stem} is exclusive to {wd.exclusive_species}", slot, i))
|
||||
if wd.hidden and not hidden and not bank.fixed_weapon:
|
||||
v.append(Violation("B8", "error", f"{wd.stem} is a hidden (engine-only) weapon", slot, i))
|
||||
if wd.compatible_section:
|
||||
if not any(cat.section(rk, cs) for cs in wd.compatible_section):
|
||||
v.append(Violation("B7", "error", f"{wd.stem} needs a rider section {wd.compatible_section} that {rk} lacks", slot, i))
|
||||
# tech gating for weapons
|
||||
if known is not None and not bank.fixed_weapon:
|
||||
for t in wd.requires:
|
||||
if not cat.tech_known(t, known):
|
||||
v.append(Violation("C2", "error", f"{wd.stem} requires {t}", slot, i))
|
||||
|
||||
# -- options ----------------------------------------------------------------
|
||||
for slot, sd in present:
|
||||
use = uses[slot] or {}
|
||||
chosen = [str(o) for o in _list(use.get("options"))]
|
||||
seen_groups: dict[int, str] = {}
|
||||
for o in chosen:
|
||||
gi = next((i for i, g in enumerate(sd.option_groups) if o.lower() in {t.lower() for t in g.techs}), None)
|
||||
if gi is None:
|
||||
v.append(Violation("D1", "error", f"{o} is not an option of {sd.stem}", slot))
|
||||
elif gi in seen_groups:
|
||||
v.append(Violation("D1", "error", f"{o} and {seen_groups[gi]} are in the same option group of {sd.stem}", slot))
|
||||
else:
|
||||
seen_groups[gi] = o
|
||||
if known is not None and cat.tech(o) is not None and not cat.tech_known(o, known):
|
||||
v.append(Violation("C3", "error", f"option {o} is not researched", slot))
|
||||
|
||||
# -- tech gating ---------------------------------------------------------------
|
||||
for slot, sd in present:
|
||||
for t in sd.requires:
|
||||
if cat.tech(t) is None and not t.lower().startswith("grp_"):
|
||||
v.append(Violation("C1", "warn", f"{sd.stem} requires unknown tech {t}", slot))
|
||||
elif known is not None and not cat.tech_known(t, known):
|
||||
v.append(Violation("C1", "error", f"{sd.stem} requires {t}", slot))
|
||||
if rk.lower() in cat.unobtainable.get(t.lower(), ()):
|
||||
v.append(Violation("C5", "error", f"{sd.stem} requires {t}, which {rk} can never research", slot))
|
||||
hct = sd.hull_class_tech()
|
||||
if hct and known is not None and not cat.tech_known(hct, known):
|
||||
v.append(Violation("C4", "error", f"{sd.stem} is a {sd.cls} section; {hct} not researched", slot))
|
||||
return v
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# derived stats
|
||||
|
||||
_CAPACITY_KEYS = (
|
||||
"refueling_capacity", "repair_capacity", "construction_capacity", "mining_capacity", "mining_rate",
|
||||
"prisoner_capacity", "colonizer_pop", "colonizer_infra", "colonizer_terra", "scanrange",
|
||||
"tacticalsensorrange", "range", "split_traffic_volume", "refinery", "freighter", "freighterq",
|
||||
"police", "tradingpost", "monitor", "spy", "spytender", "science", "propaganda", "ewar", "gateship",
|
||||
"ramscoop", "node_bore", "node_missile", "gravboat_bonus", "protectorate", "huge", "defence_platform",
|
||||
)
|
||||
|
||||
_OPTION_COST_NOTE = ("cost multiplies section cost by the product of the chosen options' "
|
||||
"tech option_cost values (assumption -- option_cost is the only cost-shaped "
|
||||
"field on option techs; the multiplier's base and stacking order are code)")
|
||||
|
||||
|
||||
def derive_stats(design: dict, cat: Catalog | None = None) -> dict:
|
||||
cat = cat or Catalog.load()
|
||||
rk = cat.race_key(design.get("race", "")) or design.get("race")
|
||||
out: dict[str, Any] = {
|
||||
"race": rk, "name": design.get("name"), "sections": {}, "hull_class": None,
|
||||
"mass": 0.0, "section_cost": 0.0, "section_cost_with_options": 0.0, "weapon_cost_per_bank": 0.0,
|
||||
"weapon_cost_per_mount": 0.0, "health_total": 0.0, "crew": 0, "cpoints": 0.0, "turrets": 0,
|
||||
"banks": 0, "command_cost": 0, "command_quota": None, "maintenance_cost": 0,
|
||||
"ftlspeed": None, "nodespeed": None, "engine_techera": None, "netforcelimits": None,
|
||||
"capacities": {}, "applied_techs": {}, "notes": [
|
||||
"health is tracked per section by the engine (ShipHealth = 3 floats); health_total is only a sum",
|
||||
_OPTION_COST_NOTE,
|
||||
"weapon cost: one turret per mount is the visual model; whether cost is per bank or per mount is code (both reported)",
|
||||
"tactical speed/turn: engine section netforcelimits are reported; the combat integrator (mass vs force) is code",
|
||||
],
|
||||
}
|
||||
for slot in SLOTS:
|
||||
use = _slot(design, slot)
|
||||
if use is None:
|
||||
continue
|
||||
sd = cat.section(rk, use.get("section"))
|
||||
if sd is None:
|
||||
continue
|
||||
opt_mult = 1.0
|
||||
for o in _list(use.get("options")):
|
||||
n = cat.tech(o)
|
||||
if n and n.get("option_cost") is not None:
|
||||
opt_mult *= float(n["option_cost"])
|
||||
sec = {
|
||||
"section": sd.stem, "type": sd.type, "class": sd.cls, "mass": _num(sd.stat("mass")),
|
||||
"cost": _num(sd.stat("cost")), "option_cost_multiplier": opt_mult, "health": _num(sd.stat("health")),
|
||||
"crew": _num(sd.stat("crew")), "cpoints": _num(sd.stat("cpoints")), "banks": [],
|
||||
}
|
||||
weapons = _list(use.get("weapons"))
|
||||
for i, bank in enumerate(sd.banks):
|
||||
ref = weapons[i] if i < len(weapons) else None
|
||||
wd = cat.weapon(ref) if ref is not None else (cat.weapon(bank.fixed_weapon) if bank.fixed_weapon else None)
|
||||
row = turret_row(cat, bank, wd) if wd else None
|
||||
sec["banks"].append({
|
||||
"size": bank.size, "class": bank.cls, "mounts": bank.mounts, "weapon": wd.stem if wd else None,
|
||||
"weapon_cost": wd.cost if wd else 0, "turret_model": row.model if row else None,
|
||||
"turret_health": row.health if row else None, "turret_track_speed": row.track_speed if row else None,
|
||||
})
|
||||
out["banks"] += 1
|
||||
out["turrets"] += bank.mounts
|
||||
if wd:
|
||||
out["weapon_cost_per_bank"] += wd.cost
|
||||
out["weapon_cost_per_mount"] += wd.cost * max(bank.mounts, 1)
|
||||
out["sections"][slot] = sec
|
||||
out["mass"] += sec["mass"]
|
||||
out["section_cost"] += sec["cost"]
|
||||
out["section_cost_with_options"] += sec["cost"] * opt_mult
|
||||
out["health_total"] += sec["health"]
|
||||
out["crew"] += int(sec["crew"] or 0)
|
||||
out["cpoints"] += sec["cpoints"]
|
||||
out["command_cost"] += int(_num(sd.stat("command_cost")))
|
||||
out["maintenance_cost"] += int(_num(sd.stat("maintenance_cost")))
|
||||
if sd.stat("command_quota") is not None:
|
||||
out["command_quota"] = (out["command_quota"] or 0) + int(_num(sd.stat("command_quota")))
|
||||
out["hull_class"] = out["hull_class"] or sd.cls
|
||||
out["applied_techs"][slot] = applied_techs(use, sd, cat)
|
||||
for k in _CAPACITY_KEYS:
|
||||
if k in sd.raw:
|
||||
out["capacities"][k] = _last(sd.raw[k])
|
||||
if sd.type == "engine" or (slot == "mission" and out["ftlspeed"] is None and sd.stat("ftlspeed") is not None):
|
||||
out["ftlspeed"] = sd.stat("ftlspeed")
|
||||
out["nodespeed"] = sd.stat("nodespeed")
|
||||
out["engine_techera"] = sd.stat("engine_techera")
|
||||
out["netforcelimits"] = sd.raw.get("netforcelimits")
|
||||
out["total_cost_estimate"] = out["section_cost_with_options"] + out["weapon_cost_per_bank"]
|
||||
return out
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# save-record conversion
|
||||
|
||||
def design_from_save(name: str, slots: Iterable[dict], cat: Catalog | None = None, *,
|
||||
hidden: bool = False, known_techs=None, race: str | None = None) -> dict:
|
||||
"""Build a Design from a save-reader design record.
|
||||
|
||||
`slots` is the DSec list in save order (command, mission, engine, +2 unused),
|
||||
each as {"species": int, "section_id": int, "weapons": [wid|file|None per bank],
|
||||
"options": [tech...]} with species/section_id (0, 0) meaning empty. The
|
||||
species index is resolved through SPECIES_INDEX; if the design mixes
|
||||
species the result carries "mixed_species": [...] for the validator.
|
||||
"""
|
||||
cat = cat or Catalog.load()
|
||||
d: dict = {"name": name, "hidden": hidden, "race": race}
|
||||
species_seen = []
|
||||
for slot, rec in zip(SLOTS, slots):
|
||||
if not rec or (rec.get("species", 0) == 0 and rec.get("section_id", 0) == 0):
|
||||
d[slot] = None
|
||||
continue
|
||||
sp = SPECIES_INDEX[rec["species"]] if rec.get("species") is not None else race
|
||||
species_seen.append(sp)
|
||||
sd = cat.section(sp, int(rec["section_id"]))
|
||||
d[slot] = {
|
||||
"section": sd.stem if sd else f"#{rec['section_id']}",
|
||||
"weapons": list(rec.get("weapons", [])),
|
||||
"options": list(rec.get("options", [])),
|
||||
"save_opts": list(rec.get("options", [])),
|
||||
}
|
||||
if sd is not None:
|
||||
# DOpts holds hull-class tech + requires + chosen options; the
|
||||
# design's own `options` are the chosen ones only.
|
||||
structural = {t.lower() for t in sd.requires}
|
||||
hct = sd.hull_class_tech()
|
||||
if hct:
|
||||
structural.add(hct.lower())
|
||||
d[slot]["options"] = [o for o in rec.get("options", []) if o.lower() not in structural]
|
||||
if d["race"] is None:
|
||||
d["race"] = species_seen[0] if species_seen else "Human"
|
||||
if len(set(species_seen)) > 1:
|
||||
d["mixed_species"] = species_seen
|
||||
if known_techs is not None:
|
||||
d["known_techs"] = list(known_techs)
|
||||
return d
|
||||
|
||||
|
||||
if __name__ == "__main__": # tiny smoke demo
|
||||
cat = Catalog.load()
|
||||
demo = {
|
||||
"name": "Armor", "race": "Human",
|
||||
"command": {"section": "DECommand", "weapons": ["bal_gauss"]},
|
||||
"mission": {"section": "DEArmor", "weapons": ["bal_gauss", "bal_gauss", "mis"]},
|
||||
"engine": {"section": "DEFission", "weapons": ["bal_gauss"]},
|
||||
"known_techs": ["DRV_Fissn", "DRV_Node", "WEP_GsDrvr"],
|
||||
}
|
||||
for viol in validate(demo, cat) or ["no violations"]:
|
||||
print(viol)
|
||||
st = derive_stats(demo, cat)
|
||||
print({k: st[k] for k in ("mass", "section_cost", "weapon_cost_per_bank", "crew", "ftlspeed", "nodespeed", "turrets")})
|
||||
|
|
@ -1,53 +1,6 @@
|
|||
# save_reader --strict against real save turn2-state.sav (padding=joint, 7 resyncs, 2728 raw bytes)
|
||||
16x [warn] /Sim/turnstats/hist/stats :: 'tch': hinted type string not plausible, read as int
|
||||
5x [warn] /Sim/Sys :: 'haltv': no value layout fits; skipped 31 bytes to tag
|
||||
2x [warn] /Sim/Sys :: 'haltv': no value layout fits; skipped 35 bytes to tag
|
||||
1x [error] /createParams/key :: expected string, read int
|
||||
1x [error] /sim/fleets[0]/Flt/ships[0]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended)
|
||||
1x [error] /sim/fleets[0]/Flt/ships[1]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended)
|
||||
1x [error] /sim/fleets[0]/Flt/ships[2]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended)
|
||||
1x [error] /sim/fleets[0]/Flt/ships[3]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended)
|
||||
1x [error] /sim/fleets[0]/Flt/ships[4]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended)
|
||||
1x [error] /sim/fleets[0]/Flt/ships[5]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended)
|
||||
1x [error] /sim/fleets[0]/Flt/ships[6]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended)
|
||||
1x [error] /sim/fleets[0]/Flt/ships[7]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended)
|
||||
1x [error] /sim/fleets[0]/Flt/ships[8]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended)
|
||||
1x [error] /sim/fleets[0]/Flt/ships[9]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended)
|
||||
1x [error] /sim/fleets[1]/Flt/ships[0]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended)
|
||||
1x [error] /sim/fleets[2]/Flt/ships[0]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended)
|
||||
1x [error] /sim/fleets[3]/Flt/ships[0]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended)
|
||||
1x [error] /sim/fleets[4]/Flt/ships[0]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended)
|
||||
1x [error] /sim/fleets[5]/Flt/ships[0]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended)
|
||||
1x [error] /sim/fleets[6]/Flt/ships[0]/Ship/PrisH/prisoners :: missing field 'PrNSp' (frame ended)
|
||||
1x [error] /sim/systems[10]/Sys/Name :: expected 'Name', found 'VFlags'; field missing
|
||||
1x [error] /sim/systems[10]/Sys/halt[2]/haltv :: expected bool, read raw
|
||||
1x [error] /sim/systems[10]/Sys/vnh :: expected 'vnh', found 'VFlags'; field missing
|
||||
1x [error] /sim/systems[14]/Sys/Name :: expected 'Name', found 'VFlags'; field missing
|
||||
1x [error] /sim/systems[14]/Sys/halt[2]/haltv :: expected bool, read raw
|
||||
1x [error] /sim/systems[14]/Sys/vnh :: expected 'vnh', found 'VFlags'; field missing
|
||||
1x [error] /sim/systems[16]/Sys/Name :: expected 'Name', found 'VFlags'; field missing
|
||||
1x [error] /sim/systems[16]/Sys/halt[2]/haltv :: expected bool, read raw
|
||||
1x [error] /sim/systems[16]/Sys/vnh :: expected 'vnh', found 'VFlags'; field missing
|
||||
1x [error] /sim/systems[18]/Sys/Name :: expected 'Name', found 'VFlags'; field missing
|
||||
1x [error] /sim/systems[18]/Sys/halt[2]/haltv :: expected bool, read raw
|
||||
1x [error] /sim/systems[18]/Sys/vnh :: expected 'vnh', found 'VFlags'; field missing
|
||||
1x [error] /sim/systems[24]/Sys/Name :: expected 'Name', found 'VFlags'; field missing
|
||||
1x [error] /sim/systems[24]/Sys/halt[2]/haltv :: expected bool, read raw
|
||||
1x [error] /sim/systems[24]/Sys/vnh :: expected 'vnh', found 'VFlags'; field missing
|
||||
1x [error] /sim/systems[25]/Sys/Name :: expected 'Name', found 'VFlags'; field missing
|
||||
1x [error] /sim/systems[25]/Sys/halt[2]/haltv :: expected bool, read raw
|
||||
1x [error] /sim/systems[25]/Sys/vnh :: expected 'vnh', found 'VFlags'; field missing
|
||||
1x [error] /sim/systems[8]/Sys/Name :: expected 'Name', found 'VFlags'; field missing
|
||||
1x [error] /sim/systems[8]/Sys/halt[2]/haltv :: expected bool, read raw
|
||||
1x [error] /sim/systems[8]/Sys/vnh :: expected 'vnh', found 'VFlags'; field missing
|
||||
1x [error] /sim/zdsi :: missing field 'zdsi' (frame ended)
|
||||
1x [error] /sim/zdst :: missing field 'zdst' (frame ended)
|
||||
1x [warn] /sim/systems[11]/Sys/nve :: 1 unexpected item(s) before 'NVE': 'indi'
|
||||
1x [warn] /sim/systems[15]/Sys/nve :: 1 unexpected item(s) before 'NVE': 'indi'
|
||||
1x [warn] /sim/systems[18]/Sys/nve :: 1 unexpected item(s) before 'NVE': 'indi'
|
||||
1x [warn] /sim/systems[1]/Sys/nve :: 1 unexpected item(s) before 'NVE': 'indi'
|
||||
1x [warn] /sim/systems[22]/Sys/nve :: 1 unexpected item(s) before 'NVE': 'indi'
|
||||
1x [warn] /sim/systems[25]/Sys/colonies[1]/PID :: 1 unexpected item(s) before 'PID': 'indi'
|
||||
1x [warn] /sim/systems[25]/Sys/nve :: 1 unexpected item(s) before 'NVE': 'indi'
|
||||
1x [warn] /sim/systems[27]/Sys/nve :: 1 unexpected item(s) before 'NVE': 'indi'
|
||||
1x [warn] /sim/systems[4]/Sys/nve :: 1 unexpected item(s) before 'NVE': 'indi'
|
||||
# save_reader --strict against real save turn2-state.sav (padding=joint, 0 resyncs, 0 hint-failures, 2503 raw bytes = opaque RNG blob)
|
||||
# result: strict exit 0 -- 0 errors, 0 warnings, 259 info
|
||||
# after findings/objects/schema-gaps-resolved.md §10 (reader: string values never ASCII-tested, len-0 strings;
|
||||
# schema: NVO.indi unconditional, PrNSp iff PrMax>0, zdsc pairs, turnstats/tch int, Summary/CreateParams tags, CDT framed)
|
||||
# same result on turn1-state.sav (257 info) and turn3-state.sav (259 info)
|
||||
259x [info] tag '.' read positionally as <R() name> (NULL-named items: Summary Settings/FxCrID, CreateParams MapP planets, Sim id lists, Vector3 bodies)
|
||||
|
|
|
|||
|
|
@ -1,11 +1,13 @@
|
|||
# SOTS1 save format — as implemented by `save_reader.py`
|
||||
|
||||
Status: **unvalidated against a real save.** Everything below is what the
|
||||
reader *assumes*, derived from the community editors (R1 Bardez, R2 SOTSedit;
|
||||
`save-editor-structs.md`) corrected by the binary member tables
|
||||
(`struct-recovery.md`). The synthetic fixture (`save_writer_stub.py`) round-trips
|
||||
under these assumptions; a real `.sav` is the first real test. Items marked
|
||||
**VERIFY** are the ones the verifier should diff against the binary first.
|
||||
Status: **validated against the three real saves** (`verify/results/saves/turn1..3-state.sav`,
|
||||
SotS 1.8): `save_reader.py <save> --strict` exits 0 on all three with `resyncs: 0`,
|
||||
`hint-failures: 0` and 2503 raw bytes (the opaque `RNG` blob). Sources: the community
|
||||
editors (R1 Bardez, R2 SOTSedit; `save-editor-structs.md`), the binary member tables
|
||||
(`struct-recovery.md`) and the serializer decompiles in
|
||||
`findings/objects/schema-gaps-resolved.md` (round 3), which settled every item that was
|
||||
marked VERIFY in the previous revision of this file. The synthetic fixture
|
||||
(`save_writer_stub.py`) round-trips under the same assumptions (`test_save_reader.py`).
|
||||
|
||||
## 1. Container
|
||||
|
||||
|
|
@ -20,38 +22,44 @@ under these assumptions; a real `.sav` is the first real test. Items marked
|
|||
## 2. Primitive encodings
|
||||
|
||||
```
|
||||
tag := int32 len, len bytes ASCII (field name; may be "." or "")
|
||||
string := int32 len, len bytes cp1252
|
||||
int := 4 bytes (int32) handle ids, int16 members, enums, counts
|
||||
tag := int32 len, len bytes ASCII (field name; "." when the writer passed NULL)
|
||||
string := int32 len, len bytes cp1252 len 0 is legal (Key, MapF, Pwd, Scenario, KeyPath)
|
||||
int := 4 bytes (int32) handle ids, int16 members widened, enums, counts
|
||||
float := 4 bytes IEEE-754 single
|
||||
bool := 1 byte, 0/1
|
||||
int64 := 8 bytes Bats2, rcex, PopC (raw-bytes write path)
|
||||
raw := n bytes, length known only to the writer RNG state (2500 B = MT19937 624 words + index)
|
||||
int64 := 8 bytes Bats2, rcex, PopC, pop (raw-bytes write path)
|
||||
raw := n bytes, length known only to the writer RNG state (2503 B incl. pad = MT19937 624 words + index)
|
||||
```
|
||||
|
||||
A **named value** is `tag value pad`, where `pad` is NUL bytes bringing the item
|
||||
to a 4-byte boundary. The reader implements two padding conventions and
|
||||
auto-detects which one the file uses (`--padding auto`, default):
|
||||
A **named value** is `tag value pad`, where `pad` is NUL bytes bringing the item to a
|
||||
4-byte boundary. **Padding is joint** — confirmed on the real saves: `pad4(4 + len(name)
|
||||
+ len(value))` over the whole item (`haltv` bool = 12 B, `vnh` bool = 8 B, `Name` =
|
||||
4+4+4+len rounded). The "split" convention (pad after the name and after the value) is
|
||||
kept only so `--padding auto` can reject it; it never matched a real file.
|
||||
|
||||
| mode | layout | item size |
|
||||
|---|---|---|
|
||||
| `joint` (default, R1's `PaddingSize=4` over `4+len(name)+len(value)`) | `[len][name][value][pad]` | `pad4(4 + len(name) + len(value))` |
|
||||
| `split` | `[len][name][pad][value][pad]` | `pad4(4 + len(name)) + pad4(len(value))` |
|
||||
| `joint` (real files) | `[len][name][value][pad]` | `pad4(4 + len(name) + len(value))` |
|
||||
| `split` (not observed) | `[len][name][pad][value][pad]` | `pad4(4 + len(name)) + pad4(len(value))` |
|
||||
|
||||
The two differ only when `len(name) % 4 != 0` **and** the value is not a
|
||||
multiple of 4 bytes (bools, odd-length strings): e.g. `NPC`+bool is 8 bytes in
|
||||
`joint`, 12 in `split`. **VERIFY (highest priority):** find a 3-char bool tag
|
||||
(`NPC`, `Dep`, `Atq`, `Srn`, `cta`, `hgs`, `vnh`, `hbq`, `hsp`) in the real
|
||||
inflated stream and check whether the value byte immediately follows the name.
|
||||
Reader assumption when writing this: `joint`.
|
||||
Values carry **no type byte**. Type comes from (a) the schema/catalog, (b) a lookahead
|
||||
plausibility test (a candidate layout is accepted only if a plausible tag, a frame marker,
|
||||
or EOF follows), (c) for 4-byte words with no hint, an int/float classification by bit
|
||||
pattern (`|int| <= 100000` → int; finite float with `1e-6 <= |f| < 1e12` → float; else
|
||||
int). The other reading is kept as `alt`. Limitation: with a name whose length is a
|
||||
multiple of 4, `bool` and `int` have identical item sizes; unknown names default to `int`.
|
||||
|
||||
Values carry **no type byte**. Type comes from (a) the schema/catalog, (b) a
|
||||
lookahead plausibility test (a candidate layout is accepted only if a plausible
|
||||
tag, a frame marker, or EOF follows), (c) for 4-byte words with no hint, an
|
||||
int/float classification by bit pattern (`|int| <= 100000` → int; finite float
|
||||
with `1e-6 <= |f| < 1e12` → float; else int). The other reading is kept as `alt`.
|
||||
Limitation: with a name whose length is a multiple of 4, `bool` and `int` have
|
||||
identical item sizes; unknown names default to `int`.
|
||||
Two rules about **string values** (both were reader bugs before round 3):
|
||||
|
||||
* Only **tag** bytes are constrained to printable ASCII. A string **value** may hold any
|
||||
windows-1252 byte, including 0x80–0x9f (system names such as `Kor’Voth` carry 0x92 =
|
||||
right single quote). A tag the catalog knows to be a string is read without any text
|
||||
test; a value can therefore never veto the layout of the item before it (that veto was
|
||||
the `haltv` → `VFlags`/`Name`/`vnh` cascade). Guessing a string for an *unknown* tag
|
||||
still uses a text-plausibility guard, which accepts every cp1252-defined byte.
|
||||
* An **empty string is 4 zero bytes**, byte-identical to int 0. The walker reads it as
|
||||
`""` when the tag is hinted; when an unhinted walker read a 4-zero-byte word, the
|
||||
schema applier coerces it to `""` instead of reporting "expected string, read int".
|
||||
|
||||
## 3. Complex frames
|
||||
|
||||
|
|
@ -60,110 +68,142 @@ identical item sizes; unknown names default to `int`.
|
|||
(0xBEEFBEEF) (0x41104110 = ~0xBEEFBEEF)
|
||||
```
|
||||
|
||||
* Every "nested object" (IStreamable via stream vft+0x28) is framed. Frames
|
||||
nest; the reader recovers the tree from the markers alone, without a schema.
|
||||
* A frame may be **tagless** (BEEFBEEF directly at an item position): the reader
|
||||
accepts it; not observed in the reference material. **VERIFY** if seen.
|
||||
* A frame body may be **untagged bytes**: `Mars::Vector3` (`Pos`, `PrvPos`,
|
||||
`FtOrig`, `FPogn2`, `FPdpos`) and `ShipHealth` (`Health`) are "3 unnamed
|
||||
floats" per the binary. Reader accepts either 12 raw bytes before the END
|
||||
marker or three tagged floats (any tag, e.g. `"."`). **VERIFY** which.
|
||||
* `RNG` frame body is treated as opaque bytes up to its END marker.
|
||||
* Markers are only interpreted at item boundaries; a value that happens to equal
|
||||
a marker is not a problem unless the reader is already resynchronising.
|
||||
* Every "nested object" (IStreamable via stream vft+0x28) is framed. Frames nest; the
|
||||
reader recovers the tree from the markers alone, without a schema.
|
||||
* **`"."` convention (confirmed):** any write with a NULL name is emitted with the tag
|
||||
`"."` (len 1). That covers every `VectorHelper<T>` array (`"."` = count, then n × `"."`
|
||||
elements: frames for streamable T, plain values for POD T), the scalars of
|
||||
`StarMapParams` and `StrategyPlayerGameSettings`, `PlayerColorID`'s index/r/g/b, and
|
||||
`Vector3` bodies. The reader matches such items positionally (`R()` fields) and reports
|
||||
each one as an *info* ("tag '.' read positionally as …"); `"."` is deliberately kept out
|
||||
of the global type catalog because it carries ints, floats and frames alike.
|
||||
* **Vector3 bodies are tagged**: 3 × `"."` float items inside the frame (`Pos`, `PrvPos`,
|
||||
`FtOrig`, `FPogn2`, `FPdpos`, `Health`, `MapP` planet positions; writer `FUN_008a60d0`).
|
||||
The reader keeps the 12-raw-byte fallback but it is not exercised by real files.
|
||||
* A frame may be **tagless** (BEEFBEEF directly at an item position): accepted by the
|
||||
reader; not observed in the real saves (element frames use `"."`).
|
||||
* `RNG` frame body is one `"."`-tagged opaque blob read straight to its END marker
|
||||
(2503 B including the 3 joint-padding bytes).
|
||||
* Markers are only interpreted at item boundaries; a value that happens to equal a marker
|
||||
is not a problem unless the reader is already resynchronising.
|
||||
|
||||
## 4. Arrays
|
||||
|
||||
| kind | encoding | reader schema |
|
||||
|---|---|---|
|
||||
| NonComplexArray (`VectorHelper`, `std::list`) | named int count, then count × element **inline in the same frame** | `NArr` |
|
||||
| ComplexArray | a frame containing: int count, then count × element | `CArr` |
|
||||
| element = leaf wrapper | e.g. `SysID` int + `Sys` frame; `PlayerID` + `Player`; `FltID` + `Flt`; `ShipID` + `Ship`; `DesID` + `Des` | `Seq([...])` |
|
||||
| sparse tables | `mnsp` then n × (`msp` index, `mv` value); `nadct` then n × (`ads`, `adt`); `haltc` then n × (`haltt`, `haltv`) | `NArr(Seq)` |
|
||||
| NonComplexArray (inline count) | named int count, then count × element **inline in the same frame** | `NArr` |
|
||||
| ComplexArray (`VectorHelper<T>`) | a frame containing: `"."` int count, then count × `"."` element | `CArr` |
|
||||
| uncounted repetition | elements repeated until the frame END (or until the lead tag stops recurring) | `Repeat` |
|
||||
| element = leaf wrapper | `SysID` int + `Sys` frame; `PlayerID` + `Player`; `FltID` + `Flt`; `ShipID` + `Ship`; `DesID` + `Des`; `ply` + `hist` | `Seq([...])` |
|
||||
| sparse tables | `mnsp` then n × (`msp`, `mv`); `nadct` then n × (`ads`, `adt`); `haltc` then n × (`haltt`, `haltv`); `zdsc` then n × (`zdsi`, `zdst`); `PrNSp` then n × (`PrSp`, `PrNum`) | `NArr(Seq)` |
|
||||
|
||||
Count tags used by the binary: `NumPlrs NumSys NumFlts NShips NumActs NumOwn NumDes
|
||||
NumLeg NumNotes NumPR NSprj Nexp NWeapXcl ndeflay rdtc numcreps ninv NumFlts NumGFs
|
||||
NumSnF NumMnF NVO NVE NVs NumPlgs2 PopNG mnsp nadct haltc PrNSp NTH`. Count tags
|
||||
inside framed arrays (`dipstats`, `preps`, `odes`, `owep`, `otch`, `cme2`, `Ojvs`…)
|
||||
are unknown — the reader takes the first child as the count whatever its name.
|
||||
Element frame tags are unknown (R1 suggests `"."`) — matched by position.
|
||||
Inline-count tags used by the binary: `NumPlrs NumSys NumFlts NShips NumActs NumOwn NumDes
|
||||
NumLeg NumNotes NumPR NSprj Nexp NWeapXcl ndeflay rdtc numcreps ninv NumFlts NumGFs NumSnF
|
||||
NumMnF NVO NVE NVs NumPlgs2 PopNG mnsp nadct haltc PrNSp NTH nply ncls seno2 spc zdsc NumIDs
|
||||
PlayerIDs DesignIDs SystemIDs FleetIDs ShipIDs TradeIDs AllExc AllExcCF`. Count and element
|
||||
tags inside framed arrays (`Players`, `GOWinPly`, `sacq`, `slost`, `dipstats`, `preps`,
|
||||
`odes`, `owep`, `otch`, `cme2`, `Ojvs`, `MapP` planets …) are always `"."` (§3).
|
||||
|
||||
Uncounted lists: `hist` holds n × `stats` frames until its END; `Sim` holds 7 × (`ISsp`
|
||||
string, `ISsu` float) after the players; the root holds one `CD` frame per custom-data id.
|
||||
|
||||
## 5. Conditionals the reader honours
|
||||
|
||||
| gate | consequence |
|
||||
|---|---|
|
||||
| `ClrID`/`indcl`/`fxCrId` frame: int index == -1 | three ints r,g,b follow inside the frame |
|
||||
| `vnh` true | `vnd`, `vnex3`, `vnpex3` |
|
||||
| `hindi` true | `indi` frame (IndependenceInfo) |
|
||||
| `NVO` entry `isind` true | `indi` frame |
|
||||
| `ClrID`/`indcl`/`FxCrID` frame: `"."` int index == -1 | three `"."` ints r,g,b follow inside the frame |
|
||||
| `vnh` true | `vnd`, `vnex3`, `vnpex3` (the only conditional in that region of `Sys`; `Name` and `VFlags` are unconditional) |
|
||||
| system-level `hindi` true | `indi` frame (IndependenceInfo) after `NVs` |
|
||||
| `NVO` entry | `indi` frame is **always** present after `isind` (inline member of the map value; `isind` does not gate it) |
|
||||
| `PrMax > 0` | `PrNSp` + n × (`PrSp`, `PrNum`); with `PrMax == 0` the `PrisH` frame ends right after `PrMax` |
|
||||
| `HFPlan` true | `FPlan` frame |
|
||||
| `HLay` true | `Lay` frame (opaque) |
|
||||
| `hbq` true | `BQ2` frame; `hsp` true → `pop`, `ppop` Population frames |
|
||||
| `HasAIR` true | `AIR` frame (opaque) |
|
||||
| system `PID` non-null | `BQ` frame present (reader: optional by name) |
|
||||
| legacy tags `ISuit ARes SysID TrdID Caps GtTrf FtSens FtInc lcid SensMod ExPopSys AIDifficultyID` | accepted if present, expected absent in 1.8 saves |
|
||||
| `SvSctOb` pointer non-NULL | `SvSctOb` frame before `zdsc` (present in the three real saves; `Opt`) |
|
||||
| custom-data blob non-NULL | one `CD` frame per `CDT` id that has data |
|
||||
| legacy tags `ISuit ARes SysID TrdID Caps GtTrf FtSens FtInc lcid SensMod ExPopSys AIDifficultyID Rand RandEnc NPC` | accepted if present (read-side only), never written by 1.8 |
|
||||
|
||||
## 6. Top-level layout
|
||||
|
||||
Written by `FUN_00877070`; loader `FUN_0086abb0`; summary-only reader `FUN_008773c0`.
|
||||
|
||||
```
|
||||
offset 0: [7]"Summary" [pad] BEEFBEEF ... 41104110 (R2 confirms the tag "Summary")
|
||||
CreateParameters frame (tag unknown)
|
||||
Sim frame (tag unknown) KeyPath NMSz NMLc NMnx <id lists> ModCount Frame
|
||||
GameID Attrib RNG GameName Map IncMod ResMod EnAl
|
||||
EnTm GOTurn GOWinPly NPCm NPCo NPCi NPCv NPCa
|
||||
szadj rsadj suadj sprjs RandEncAdj cmbtid turnstats
|
||||
numcreps/crep ninv <invs.. AllExc..> NumPlrs/PlayerID/Player
|
||||
<ISsp ISsu> NumSys/SysID/Sys NdGr2 trdmgr spymgr
|
||||
NumFlts/FltID/Flt NumActs/Act SvSctOb zdsc zdsi zdst
|
||||
CdTable UNFRAMED at root: cdt frame, cdplayer frame, N × cdai frames
|
||||
offset 0: "Summary" StrategyGameInfo GameName Turn NumSys Checksum Players{"."=n, n×"."{Slot{} Rank}}
|
||||
Session{TMRS{TSTL TCTL TQTL TQTLE}} MapShape IncMod ResMod
|
||||
Alliances Teams Encounters Scenario
|
||||
"CreateParams" StrategyGameCreateParams Name ID RSeed AID Key MapP{"." int, "."{planets}, "." n×"."{ints},
|
||||
"."{nodePaths}} MapS MapF NSys REnc SDist SSize SRes SSuit MaxP ASpec
|
||||
bAlly NTeam tmgrp PSav PCol PTech IncM ResM scrp{spc n×(spsn sppn sppv)}
|
||||
"Sim" StrategyServer KeyPath NMSz NMLc NMnx | PlayerIDs DesignIDs SystemIDs FleetIDs
|
||||
ShipIDs TradeIDs (each: count, n×".") | ModCount Frame GameID Attrib{}
|
||||
RNG{} GameName Map IncMod ResMod EnAl EnTm GOTurn GOWinPly{} NPCm NPCo
|
||||
NPCi NPCv NPCa szadj rsadj suadj sprjs{} RandEncAdj cmbtid turnstats{}
|
||||
numcreps n×crep{} ninv n×(invs inve invt invtb) AllExc n×(AllExc AllExc)
|
||||
AllExc n×(AllExc AllExc) AllExcCF n×(AllExcCFp AllExcCFp)
|
||||
NumPlrs n×(PlayerID Player{}) 7×(ISsp ISsu) NumSys n×(SysID Sys{})
|
||||
NdGr2{} trdmgr{} spymgr{} NumFlts n×(FltID Flt{}) NumActs n×Act
|
||||
[SvSctOb{}] zdsc n×(zdsi zdst)
|
||||
"CDT" CustomDataTable NumIDs n×ID (strings: Player.<id>.TurnCommands_v5, Player.<id>.AIAgent …)
|
||||
n × "CD" opaque custom-data frames, one per id that has a blob
|
||||
```
|
||||
|
||||
Regions in `<...>` are parsed generically and kept as raw item lists
|
||||
(`sim.idLists`, `sim.invasionsAndExclusions`, `sim.species`). Everything not
|
||||
covered by a shape (TechTree body, Events, ShipRecs, spy2, civr, comms, Ojvs,
|
||||
SvSctOb, trdmgr, spymgr, CdTable, …) is kept as the generic
|
||||
`Slot` (SlotDef): `IsPlay IsDead IsReq IsRec IsFxNm FxNm IsFxSp FxSp IsFxCr FxCrID{} IsFxBd FxBd
|
||||
IsFxAv FxAv Tag Pwd(string) Team Settings{4×"." int}`.
|
||||
`turnstats` (GameTurnHistory): `nply`, n × (`ply`, `hist{ply, stats{} …until END}`);
|
||||
`stats` (PlayerTurnStats): `pop`(int64) `sacq{}` `slost{}` (VectorHelper<SystemEvent>: `set ses seop
|
||||
senp seno2 n×seot2`) `trn almem inc tdinc sav col bat tch`(int) `ncls` 3 × (`cls shpt shpl shpk satt
|
||||
satl satk`).
|
||||
|
||||
Everything not covered by a shape (TechTree body, Events, ShipRecs, spy2, civr, comms, Ojvs,
|
||||
Attrib, sprjs, SvSctOb, trdmgr, spymgr, CD, …) is kept as the generic
|
||||
`{"_name","_off","_items":[{"name","kind","value","off"}...]}` form.
|
||||
|
||||
## 7. Struct field orders and types applied
|
||||
|
||||
Shapes live in `save_reader.py` (`Sys`, `Player`, `Fleet`, `Ship`, `PlayerView`,
|
||||
`Population`/`PopG`, `Morale`, `MoraleEvent`, `BuildQueue`/`BuildOrder`,
|
||||
`IndependenceInfo`, `Rts`, `DipStat`, `Prep`, `FlightPlan`, `Waypoint`,
|
||||
`PrisonerHold`, `Summary`, `CreateParams`, …). `A("tag", type)` = on-disk tag
|
||||
confirmed in the exe (strict name match); `R("name", type)` = R1 C# name only
|
||||
(positional). Binary corrections applied over R1:
|
||||
Shapes live in `save_reader.py` (`Summary`, `Slot`, `CreateParams`, `MapP`, `Sim`,
|
||||
`TurnStats`/`PlayerTurnHistory`/`PlayerTurnStats`, `Sys`, `Player`, `Fleet`, `Ship`,
|
||||
`PlayerView`, `Population`/`PopG`, `Morale`, `MoraleEvent`, `BuildQueue`/`BuildOrder`,
|
||||
`IndependenceInfo`, `Rts`, `DipStat`, `Prep`, `FlightPlan`, `Waypoint`, `PrisonerHold`,
|
||||
`CdTable`, …). `A("tag", type)` = on-disk tag confirmed in the exe (strict name match);
|
||||
`R("name", type)` = positional (`"."`-tagged items, or R1 C# names). Binary corrections
|
||||
applied over R1/R2:
|
||||
|
||||
* float: `TRM CstR CstE CstT shrm RefCap RepCap`, PlayerView `Infra`
|
||||
* int64: `Bats2 rcex PopC`
|
||||
* bool: `Abdn Dstyd PvMA AIBn` (R2's "short" readings are the value byte)
|
||||
* int on disk though int16 in memory: `TShn ETS` and all DiplomacyStats counters
|
||||
* string: `pswd`
|
||||
* Vector3 (3 floats): `FtOrig`
|
||||
* float: `TRM CstR CstE CstT shrm RefCap RepCap`, PlayerView `Infra`; `Summary.IncMod/ResMod`,
|
||||
`Sim.IncMod/ResMod`, `Sim.szadj/rsadj/suadj`, `TMRS.TSTL/TCTL/TQTL/TQTLE`,
|
||||
`CreateParams.SDist/SSize/SRes/SSuit/REnc/IncM/ResM`, `ISsu`
|
||||
* int64: `Bats2 rcex PopC`, `PlayerTurnStats.pop`
|
||||
* bool: `Abdn Dstyd PvMA AIBn haltv EnAl EnTm` (R2's "short" readings are the value byte)
|
||||
* int on disk though int16 in memory: `TShn ETS tch col bat` and all DiplomacyStats counters
|
||||
* string: `pswd`, `Key`, `MapF`, `Pwd`, `KeyPath` (all may be empty)
|
||||
* Vector3 (3 × `"."` float): `Pos PrvPos FtOrig FPogn2 FPdpos Health`
|
||||
* `Nexp` entries carry `xid xmin xmax xper(float)`
|
||||
* `Team` appears twice in Player: an int, later a nested `{ALid AL NA CF}` frame
|
||||
(typed key `Alliances`)
|
||||
* `Team` appears twice in Player: an int, later a nested `{ALid AL NA CF}` frame (typed key `Alliances`)
|
||||
* `GOWinPly` is a framed `VectorHelper<uint>` (`"."` count + n × `"."` int), empty in the real saves
|
||||
* `CreateParams`/`Summary` tag case is as written above (R1's camel-case names were wrong)
|
||||
|
||||
## 8. Reader output conventions
|
||||
|
||||
* `--dump`: one line per item, `@<inflated offset> name kind value`; `?` after
|
||||
the kind means the type was guessed, `(alt …)` shows the other reading of a
|
||||
4-byte word; frames print `{` … `}` with item count and byte size.
|
||||
* `--json`: `{"padding", "stats", "issues": [...], "data": {...}}`; every typed
|
||||
struct carries `_off`; unexpected items are kept under `_unexpected` /
|
||||
`_extra`; unknown regions keep the generic form.
|
||||
* Issue levels: `error` (schema field missing / type impossible / frame
|
||||
unterminated), `warn` (resync, hint not plausible, unexpected items,
|
||||
width mismatch, best-effort read at EOF), `info` (unnamed small payload
|
||||
before END, positional tag name differing from the R1 name). `--strict`
|
||||
fails on error or warn.
|
||||
* Exit status: 0 clean, 1 errors present, 2 unreadable container.
|
||||
* `--dump`: one line per item, `@<inflated offset> name kind value`; `?` after the kind
|
||||
means the type was guessed, `(alt …)` shows the other reading of a 4-byte word; frames
|
||||
print `{` … `}` with item count and byte size.
|
||||
* `--json`: `{"padding", "stats", "issues": [...], "data": {...}}`; `data` has the keys
|
||||
`summary`, `createParams`, `sim`, `cdTable`, `customData`; every typed struct carries
|
||||
`_off`; unexpected items are kept under `_unexpected` / `_extra`; unknown regions keep
|
||||
the generic form.
|
||||
* Issue levels: `error` (schema field missing / type impossible / frame unterminated),
|
||||
`warn` (resync, hint not plausible, unexpected items, width mismatch, best-effort read
|
||||
at EOF), `info` (positional `"."` tag read as an R() name, unnamed small payload before
|
||||
END). `--strict` fails on error or warn.
|
||||
* Exit status: 0 clean, 1 errors present, 2 unreadable container or strict failure.
|
||||
|
||||
## 9. What the verifier should check first on a real save
|
||||
## 9. Verification checklist (all settled on the real saves)
|
||||
|
||||
1. Padding convention (§2) — a 3-char bool tag settles it.
|
||||
2. The `Summary` frame's 13 children and their actual tag names (R1 names are
|
||||
probably case-variants: `NumSys` is confirmed by R2 as a tag string).
|
||||
3. Whether Vector3 bodies are tagged (§3).
|
||||
4. The tag used for element frames and for counts inside framed arrays (§4).
|
||||
5. Whether the reader's auto-detected padding, `resyncs == 0` and
|
||||
`hint_failures == 0` hold; any resync offset points at a layout gap.
|
||||
1. Padding convention — **joint** (§2).
|
||||
2. `Summary` frame: 13 children with the tags in §6 (R1's names were case-variants).
|
||||
3. Vector3 bodies — **tagged**, 3 × `"."` float (§3).
|
||||
4. Element and count tags inside framed arrays — **`"."`** (§3, §4).
|
||||
5. `--padding auto` picks joint; `resyncs == 0` and `hint_failures == 0` on turn1/2/3;
|
||||
the only raw bytes are the `RNG` blob. Remaining unknowns are the bodies of the
|
||||
opaque frames listed in §6 (kept generic, not a parsing gap).
|
||||
|
|
|
|||
|
|
@ -24,18 +24,25 @@ has to guess it. This module does both:
|
|||
malformed regions become `raw` nodes; parsing continues at the
|
||||
next frame marker or plausible tag. Nothing in the schema is
|
||||
required for the walk to succeed.
|
||||
* Shapes the recovered struct field orders (Summary, CreateParameters,
|
||||
Sim -> Player / Sys / Flt / Ship ...) applied on top of the
|
||||
generic tree to produce typed dicts. Fields marked A() carry
|
||||
binary-confirmed on-disk names (struct-recovery.md) and are
|
||||
matched by name; fields marked R() carry only the community
|
||||
editor's C# name and are matched by position.
|
||||
* Shapes the recovered struct field orders (Summary, CreateParams,
|
||||
Sim -> Player / Sys / Flt / Ship ..., CDT) applied on top of
|
||||
the generic tree to produce typed dicts. Fields marked A()
|
||||
carry binary-confirmed on-disk names (struct-recovery.md,
|
||||
schema-gaps-resolved.md) and are matched by name; fields marked
|
||||
R() are matched by position (NULL-named "." items, or names
|
||||
known only from the community editor).
|
||||
|
||||
Padding convention: the community reference is ambiguous about whether the
|
||||
NUL padding is computed over the whole item ("joint": [len][name][value][pad])
|
||||
or separately after the name and after the value ("split"). Both are
|
||||
implemented; --padding auto (default) walks the file both ways and keeps the
|
||||
cleaner parse. REAL-SAVE VALIDATION IS PENDING -- see SAVE_FORMAT.md.
|
||||
Padding convention: the NUL padding is computed over the whole item ("joint":
|
||||
[len][name][value][pad]) -- confirmed on the real saves. The alternative
|
||||
("split": pad after the name and after the value) is kept for --padding auto,
|
||||
which walks the file both ways and keeps the cleaner parse.
|
||||
|
||||
Two facts about string VALUES matter for the plausibility test: they may be
|
||||
empty (len 0 = four zero bytes, byte-identical to int 0) and they may hold any
|
||||
windows-1252 byte (system names use 0x92 = right single quote). Only TAG bytes
|
||||
are constrained to printable ASCII; a value never vetoes a layout. Items the
|
||||
game writes with a NULL name carry the tag ".". See SAVE_FORMAT.md and
|
||||
findings/objects/schema-gaps-resolved.md.
|
||||
|
||||
usage: save_reader.py SAVE [--dump] [--json] [--strict] [--padding MODE]
|
||||
[--out FILE] [--inflate FILE]
|
||||
|
|
@ -55,7 +62,7 @@ from typing import Any, Callable, Optional
|
|||
|
||||
__all__ = [
|
||||
"BEGIN_MARK", "END_MARK", "SaveFormatError", "Issue", "Node", "Walker",
|
||||
"Shape", "Seq", "CArr", "NArr", "If", "Opt", "Until", "Rest", "A", "R",
|
||||
"Shape", "Seq", "CArr", "NArr", "If", "Opt", "Until", "Repeat", "Rest", "A", "R",
|
||||
"ROOT", "apply_schema", "read_save", "read_bytes", "inflate", "dump_tree",
|
||||
"plain", "pad4",
|
||||
]
|
||||
|
|
@ -210,11 +217,18 @@ class NArr:
|
|||
|
||||
|
||||
class If:
|
||||
"""Conditional group: parsed iff out[key] == equals (default True)."""
|
||||
__slots__ = ("key", "inner", "equals")
|
||||
"""Conditional group: parsed iff out[key] == equals (default True), or,
|
||||
with greater_than=N, iff out[key] is a number > N (e.g. PrMax > 0)."""
|
||||
__slots__ = ("key", "inner", "equals", "greater_than")
|
||||
|
||||
def __init__(self, key, inner, equals=True):
|
||||
self.key, self.inner, self.equals = key, inner, equals
|
||||
def __init__(self, key, inner, equals=True, greater_than=None):
|
||||
self.key, self.inner, self.equals, self.greater_than = key, inner, equals, greater_than
|
||||
|
||||
def holds(self, out: dict) -> bool:
|
||||
v = out.get(self.key)
|
||||
if self.greater_than is not None:
|
||||
return isinstance(v, (int, float)) and not isinstance(v, bool) and v > self.greater_than
|
||||
return v == self.equals
|
||||
|
||||
|
||||
class Opt:
|
||||
|
|
@ -233,6 +247,22 @@ class Until:
|
|||
self.key, self.stop = key, stop
|
||||
|
||||
|
||||
class Repeat:
|
||||
"""Uncounted repetition: consume `elem` (a Field or Seq whose first field is
|
||||
an A() tag) again and again while the next item carries that tag. Used for
|
||||
lists the writer emits with no count (turnstats `stats` frames until the
|
||||
frame END, the 7 `ISsp`/`ISsu` pairs, the root `CD` frames)."""
|
||||
__slots__ = ("elem", "key", "lead")
|
||||
|
||||
def __init__(self, elem, key=None):
|
||||
self.elem = elem
|
||||
first = elem.fields[0] if isinstance(elem, Seq) else elem
|
||||
if not isinstance(first, Field):
|
||||
raise TypeError("Repeat needs a Field or a Seq starting with a Field")
|
||||
self.lead = first.name
|
||||
self.key = key or first.key
|
||||
|
||||
|
||||
class Rest:
|
||||
"""Collect all remaining items generically."""
|
||||
__slots__ = ("key",)
|
||||
|
|
@ -294,6 +324,9 @@ def _collect_names(fields, d):
|
|||
elif isinstance(f, If):
|
||||
inner = f.inner
|
||||
_collect_names(inner.fields if isinstance(inner, Seq) else [inner], d)
|
||||
elif isinstance(f, Repeat):
|
||||
e = f.elem
|
||||
_collect_names(e.fields if isinstance(e, Seq) else [e], d)
|
||||
|
||||
|
||||
# --- schema: recovered struct layouts -----------------------------------------
|
||||
|
|
@ -301,58 +334,59 @@ def _collect_names(fields, d):
|
|||
# from struct-recovery.md (binary). A() names are on-disk tags confirmed in the
|
||||
# exe; R() names are R1's C# field names (disk spelling unknown -> positional).
|
||||
|
||||
NestedInt = Shape(None, [R("value", "int"), Rest()])
|
||||
|
||||
PlayerColor = Shape("ClrID", [
|
||||
# Game::PlayerColorID: "." int index; iff -1 the three "." ints r,g,b follow.
|
||||
# Framed under ClrID (Player), indcl (IndependenceInfo) and FxCrID (SlotDef).
|
||||
PlayerColor = Shape(None, [
|
||||
R("idx", "int"),
|
||||
If("idx", Seq([R("r", "int"), R("g", "int"), R("b", "int")]), equals=-1),
|
||||
Rest(),
|
||||
])
|
||||
|
||||
# -- Summary (R1 §3) -----------------------------------------------------------
|
||||
PlayerSettings = Shape("Settings", [
|
||||
R("initialTreasury", "int"), R("initialColonies", "int"),
|
||||
R("initialTechnologies", "int"), R("difficulty", "int"), Rest(),
|
||||
# -- Summary = StrategyGameInfo::Write @0x00829960 (schema-gaps-resolved.md §5.2)
|
||||
# Every tag below is confirmed in the binary. Items whose writer passes a NULL
|
||||
# name are on disk as "." and are matched positionally (R()).
|
||||
PlayerSettings = Shape("Settings", [ # StrategyPlayerGameSettings: 4 x "." int
|
||||
R("treasury", "int"), R("colonies", "int"), R("techs", "int"), R("difficulty", "int"), Rest(),
|
||||
])
|
||||
PlayerSlot = Shape("Slot", [
|
||||
R("isPlay", "bool"), R("isDead", "bool"), R("isReq", "bool"), R("isRec", "bool"),
|
||||
R("isFxNm", "bool"), R("fxNm", "string"),
|
||||
R("isFxSp", "bool"), R("fxSp", "int"),
|
||||
R("isFxCr", "bool"), R("fxCrId", PlayerColor),
|
||||
R("isFxBd", "bool"), R("fxBd", "string"),
|
||||
R("isFxAv", "bool"), R("fxAv", "string"),
|
||||
R("tag", "int"), R("pwd", "int"), R("team", "int"),
|
||||
R("settings", PlayerSettings), Rest(),
|
||||
Slot = Shape("Slot", [ # SlotDef::Write @0x008276d0
|
||||
A("IsPlay", "bool"), A("IsDead", "bool"), A("IsReq", "bool"), A("IsRec", "bool"),
|
||||
A("IsFxNm", "bool"), A("FxNm", "string"), A("IsFxSp", "bool"), A("FxSp", "int"),
|
||||
A("IsFxCr", "bool"), A("FxCrID", PlayerColor), A("IsFxBd", "bool"), A("FxBd", "string"),
|
||||
A("IsFxAv", "bool"), A("FxAv", "string"), A("Tag", "int"), A("Pwd", "string"), A("Team", "int"),
|
||||
A("Settings", PlayerSettings), Rest(),
|
||||
])
|
||||
PlayerSlotWrapper = Shape("PlayerSlotWrapper", [R("slot", PlayerSlot), R("rank", "int"), Rest()])
|
||||
Tmrs = Shape("Tmrs", [R("tstl", "int"), R("tctl", "int"), R("tqtl", "int"), R("tqtle", "int"), Rest()])
|
||||
Session = Shape("Session", [R("tmrs", Tmrs), Rest()])
|
||||
PlayerInfo = Shape(None, [A("Slot", Slot), A("Rank", "int"), Rest()]) # element tag "."
|
||||
Tmrs = Shape("TMRS", [A("TSTL", "float"), A("TCTL", "float"), A("TQTL", "float"), A("TQTLE", "float"), Rest()])
|
||||
Session = Shape("Session", [A("TMRS", Tmrs), Rest()])
|
||||
Summary = Shape("Summary", [
|
||||
R("gameName", "string"), R("turn", "int"), R("numSys", "int"), R("checkSum", "int"),
|
||||
R("players", CArr(PlayerSlotWrapper)), R("session", Session),
|
||||
R("mapShape", "int"), R("incMod", "int"), R("resMod", "int"),
|
||||
R("alliances", "bool"), R("teams", "bool"), R("encounters", "bool"),
|
||||
R("scenario", "string"), Rest(),
|
||||
A("GameName", "string"), A("Turn", "int"), A("NumSys", "int"), A("Checksum", "int"),
|
||||
A("Players", CArr(PlayerInfo)), A("Session", Session), A("MapShape", "int"),
|
||||
A("IncMod", "float"), A("ResMod", "float"), A("Alliances", "bool"), A("Teams", "bool"),
|
||||
A("Encounters", "bool"), A("Scenario", "string"), Rest(),
|
||||
])
|
||||
|
||||
# -- CreateParameters (R1 §4) ---------------------------------------------------
|
||||
Planet = Shape("Planet", [
|
||||
R("coordinates", "vec3"), R("unknown1", "int"), R("unknown2", "int"),
|
||||
R("unknown3", "int"), R("unknown4", "int"), Rest(),
|
||||
# -- CreateParams = StrategyGameCreateParams::Write @0x0082ae40 (§5.1) ----------
|
||||
Planet = Shape(None, [ # SystemParams element, all tags "."
|
||||
R("pos", "vec3"), R("p1", "int"), R("p2", "int"), R("p3", "int"), R("p4", "float"), Rest(),
|
||||
])
|
||||
MapPNpc = Shape("Npc", [R("unknown1", "int"), R("unknown2", "int"), Rest()])
|
||||
MapP = Shape("MapP", [
|
||||
R("unknown1", "int"), R("planetArray", CArr(Planet)),
|
||||
R("players", NArr(CArr("int"))), R("npcArray", CArr(MapPNpc)), Rest(),
|
||||
MapP = Shape("MapP", [ # StarMapParams::Write @0x00727a10, all tags "."
|
||||
R("mapType", "int"),
|
||||
R("planets", CArr(Planet)), # VectorHelper<SystemParams>
|
||||
R("players", NArr(CArr("int"))), # "." count, n x "." frame{ "." count, n x "." int }
|
||||
R("nodePaths", CArr("any")), # VectorHelper<SimpleNodePath>, empty in the real saves
|
||||
Rest(),
|
||||
])
|
||||
Scrp = Shape("Scrp", [R("spc", "int"), Rest()])
|
||||
CreateParams = Shape("CreateParameters", [
|
||||
R("name", "string"), R("id", "int"), R("rSeed", "int"), R("aid", "int"), R("key", "string"),
|
||||
R("mapP", MapP), R("mapS", "int"), R("mapF", "int"), R("nSys", "int"), R("rEnc", "float"),
|
||||
R("sDist", "int"), R("sSize", "float"), R("sRes", "float"), R("sSuit", "int"), R("maxP", "int"),
|
||||
R("aSpec", "int"), R("bAlly", "bool"), R("nTeam", "int"), R("tmgrp", "bool"),
|
||||
R("pSav", "int"), R("pCol", "int"), R("pTech", "int"), R("incM", "float"), R("resM", "float"),
|
||||
R("scrp", Scrp), Rest(),
|
||||
Scrp = Shape("scrp", [ # StrategyScriptParams::Write @0x0082ad40
|
||||
A("spc", NArr(Seq([A("spsn", "string"), A("sppn", "string"), A("sppv", "string")])), key="params"),
|
||||
Rest(),
|
||||
])
|
||||
CreateParams = Shape("CreateParams", [
|
||||
A("Name", "string"), A("ID", "int"), A("RSeed", "int"), A("AID", "int"), A("Key", "string"), # Key may be ""
|
||||
A("MapP", MapP), A("MapS", "int"), A("MapF", "string"), A("NSys", "int"), A("REnc", "float"),
|
||||
A("SDist", "float"), A("SSize", "float"), A("SRes", "float"), A("SSuit", "float"), A("MaxP", "int"),
|
||||
A("ASpec", "int"), A("bAlly", "bool"), A("NTeam", "int"), A("tmgrp", "bool"),
|
||||
A("PSav", "int"), A("PCol", "int"), A("PTech", "int"), A("IncM", "float"), A("ResM", "float"),
|
||||
A("scrp", Scrp), Rest(),
|
||||
])
|
||||
|
||||
# -- shared sim types (struct-recovery §1.4, §1.5, §1.6, §1.7) ------------------
|
||||
|
|
@ -417,9 +451,11 @@ Sys = Shape("Sys", [
|
|||
A("NumGFs", NArr(A("GF", "int")), key="gates"),
|
||||
A("NumSnF", NArr(A("SnF", "int")), key="stations"),
|
||||
A("NumMnF", NArr(A("MnF", "int")), key="monitors"),
|
||||
# NVO entry: indi is an inline member of the map value and is written for
|
||||
# every colony, NOT gated by isind (ServerSystem::Write @0x00749630).
|
||||
A("NVO", NArr(Seq([
|
||||
A("PID", "int"), A("TShn", "int"), A("OID", "int"), A("isind", "bool"),
|
||||
If("isind", A("indi", IndependenceInfo)),
|
||||
A("indi", IndependenceInfo),
|
||||
])), key="colonies"),
|
||||
A("NVE", NArr(Seq([A("EPid", "int"), A("ETS", "int"), A("Eid", "int")])), key="nve"),
|
||||
A("NVs", NArr(Seq([R("pid", "int"), R("pview", PlayerView)])), key="views"),
|
||||
|
|
@ -513,8 +549,10 @@ FlightPlan = Shape("FPlan", [
|
|||
R("wpts", CArr(Waypoint), flex=True), A("FPsp2", "float"), A("FPeta2", "int"),
|
||||
A("FPogn2", "vec3"), A("FPdpos", "vec3"), A("pnd", "int"), Rest(),
|
||||
])
|
||||
PrisonerHold = Shape("PrisH", [
|
||||
A("PrMax", "int"), A("PrNSp", NArr(Seq([A("PrSp", "int"), A("PrNum", "int")])), key="prisoners"), Rest(),
|
||||
PrisonerHold = Shape("PrisH", [ # PrisonerHold::Write @0x0056ec00
|
||||
A("PrMax", "int"),
|
||||
If("PrMax", A("PrNSp", NArr(Seq([A("PrSp", "int"), A("PrNum", "int")])), key="prisoners"), greater_than=0),
|
||||
Rest(),
|
||||
])
|
||||
Ship = Shape("Ship", [
|
||||
A("DesID", "int"), A("FltID", "int"), A("PlrID", "int"), A("Range", "float"),
|
||||
|
|
@ -561,34 +599,65 @@ NodePath = Shape(None, [
|
|||
])
|
||||
NodeGrid = Shape("NdGr2", [R("paths", CArr(NodePath), flex=True), R("nextId", "int"), Rest()])
|
||||
|
||||
# -- Sim block (struct-recovery §5 tag order + R1 §5) ----------------------------
|
||||
# -- turn statistics (schema-gaps-resolved.md §6) --------------------------------
|
||||
SystemEvent = Shape(None, [ # SystemEvent::Write @0x008189a0, element tag "."
|
||||
A("set", "int"), A("ses", "int"), A("seop", "int"), A("senp", "int"),
|
||||
A("seno2", NArr(A("seot2", "int")), key="others"), Rest(),
|
||||
])
|
||||
ClassStats = Seq([A("cls", "int"), A("shpt", "int"), A("shpl", "int"), A("shpk", "int"),
|
||||
A("satt", "int"), A("satl", "int"), A("satk", "int")])
|
||||
PlayerTurnStats = Shape("stats", [ # PlayerTurnStats::Write @0x0082c290
|
||||
A("pop", "int64"), A("sacq", CArr(SystemEvent)), A("slost", CArr(SystemEvent)),
|
||||
A("trn", "int"), A("almem", "int"), A("inc", "int"), A("tdinc", "int"), A("sav", "int"),
|
||||
A("col", "int"), A("bat", "int"), A("tch", "int"), # tch: INT (int16 widened)
|
||||
A("ncls", NArr(ClassStats), key="classes"), Rest(),
|
||||
])
|
||||
PlayerTurnHistory = Shape("hist", [ # PlayerTurnHistory::Write @0x0082c4a0
|
||||
A("ply", "int"), Repeat(A("stats", PlayerTurnStats), key="stats"), # no count: until frame END
|
||||
])
|
||||
TurnStats = Shape("turnstats", [ # GameTurnHistory::Write @0x0082c5a0
|
||||
A("nply", NArr(Seq([A("ply", "int"), A("hist", PlayerTurnHistory)])), key="players"), Rest(),
|
||||
])
|
||||
|
||||
# -- Sim block = StrategyServer::Write @0x0079fa70 (schema-gaps-resolved.md §8) ----
|
||||
Invasion = Seq([A("invs", "int"), A("inve", "int"), A("invt", "int"), A("invtb", "int")])
|
||||
IdList = NArr(R(".", "int")) # FUN_00794cd0: count, n x "." int
|
||||
Sim = Shape("Sim", [
|
||||
A("KeyPath", "string"), A("NMSz", "int"), A("NMLc", "int"), A("NMnx", "int"),
|
||||
Until("idLists", "ModCount"), # PlayerIDs/DesignIDs/SystemIDs/FleetIDs/ShipIDs/TradeIDs
|
||||
A("PlayerIDs", IdList, key="playerIds"), A("DesignIDs", IdList, key="designIds"),
|
||||
A("SystemIDs", IdList, key="systemIds"), A("FleetIDs", IdList, key="fleetIds"),
|
||||
A("ShipIDs", IdList, key="shipIds"), A("TradeIDs", IdList, key="tradeIds"),
|
||||
A("ModCount", "int"), A("Frame", "int"), A("GameID", "int"), Opt("AIDifficultyID", "int"),
|
||||
A("Attrib", "any"), A("RNG", "any"), A("GameName", "string"),
|
||||
A("Map", "int"), A("IncMod", "int"), A("ResMod", "int"), A("EnAl", "int"), A("EnTm", "int"),
|
||||
A("GOTurn", "int"), A("GOWinPly", NestedInt),
|
||||
A("NPCm", "int"), A("NPCo", "int"), A("NPCi", "int"), A("NPCv", "int"), A("NPCa", "int"),
|
||||
A("szadj", "int"), A("rsadj", "int"), A("suadj", "int"),
|
||||
A("sprjs", "any"), A("RandEncAdj", "float"), A("cmbtid", "int"), A("turnstats", "any"),
|
||||
A("Attrib", "any"), Opt("Rand", "int"), A("RNG", "any"), A("GameName", "string"),
|
||||
A("Map", "int"), A("IncMod", "float"), A("ResMod", "float"), Opt("RandEnc", "bool"),
|
||||
A("EnAl", "bool"), A("EnTm", "bool"),
|
||||
A("GOTurn", "int"), A("GOWinPly", CArr("int")), # VectorHelper<uint>
|
||||
A("NPCm", "int"), A("NPCo", "int"), A("NPCi", "int"), A("NPCv", "int"), A("NPCa", "int"), Opt("NPC", "int"),
|
||||
A("szadj", "float"), A("rsadj", "float"), A("suadj", "float"),
|
||||
A("sprjs", "any"), A("RandEncAdj", "float"), A("cmbtid", "int"), A("turnstats", TurnStats),
|
||||
A("numcreps", NArr(A("crep", Crep)), key="combatReports"),
|
||||
A("ninv", "int"),
|
||||
Until("invasionsAndExclusions", "NumPlrs"), # invs/inve/invt/invtb, AllExc x6, AllExcCF, AllExcCFp x2
|
||||
A("ninv", NArr(Invasion), key="invasions"),
|
||||
A("AllExc", NArr(Seq([A("AllExc", "int"), A("AllExc", "int")])), key="exclusions3"),
|
||||
A("AllExc", NArr(Seq([A("AllExc", "int"), A("AllExc", "int")])), key="exclusions2"),
|
||||
A("AllExcCF", NArr(Seq([A("AllExcCFp", "int"), A("AllExcCFp", "int")])), key="exclusionsCF"),
|
||||
A("NumPlrs", NArr(Seq([A("PlayerID", "int"), A("Player", Player)])), key="players"),
|
||||
Until("species", "NumSys"), # ISsp / ISsu
|
||||
Repeat(Seq([A("ISsp", "string"), A("ISsu", "float")]), key="species"), # 7 pairs, no count
|
||||
A("NumSys", NArr(Seq([A("SysID", "int"), A("Sys", Sys)])), key="systems"),
|
||||
A("NdGr2", NodeGrid), A("trdmgr", "any"), A("spymgr", "any"),
|
||||
A("NumFlts", NArr(Seq([A("FltID", "int"), A("Flt", Fleet)])), key="fleets"),
|
||||
A("NumActs", NArr(A("Act", "int")), key="acts"),
|
||||
A("SvSctOb", "any"),
|
||||
A("zdsc", "int"), A("zdsi", "int"), A("zdst", "int"),
|
||||
Opt("SvSctOb", "any"), # only if pointer != NULL
|
||||
A("zdsc", NArr(Seq([A("zdsi", "int"), A("zdst", "int")])), key="zoneDefence"),
|
||||
Rest(),
|
||||
])
|
||||
|
||||
# -- file root: Summary, CreateParameters, Sim, then the unframed CdTable ---------
|
||||
# -- file root (FUN_00877070): Summary, CreateParams, Sim, CDT, then one opaque
|
||||
# CD frame per custom-data id that has a blob (Player.<id>.TurnCommands_v5 / .AIAgent).
|
||||
CdTable = Shape("CDT", [A("NumIDs", NArr(A("ID", "string")), key="ids"), Rest()])
|
||||
ROOT = Seq([
|
||||
R("summary", Summary), R("createParams", CreateParams), R("sim", Sim), Rest("cdTable"),
|
||||
A("Summary", Summary, key="summary"), A("CreateParams", CreateParams, key="createParams"),
|
||||
A("Sim", Sim, key="sim"), A("CDT", CdTable, key="cdTable"),
|
||||
Repeat(A("CD", "any"), key="customData"),
|
||||
])
|
||||
|
||||
# Hand-maintained kinds for names that only occur inside generic ("any") regions.
|
||||
|
|
@ -599,7 +668,7 @@ MANUAL_KINDS = {
|
|||
"faiDes": "bool", "dHide": "bool",
|
||||
# objectives / comms / research / encounters (R1 §5, §6, §11)
|
||||
"cmp": "bool", "dsc": "string", "xcsn": "string", "nm": "string", "ntg": "string",
|
||||
"tch": "string", "wep": "string", "gmch": "int", "drad": "float", "cst": "float",
|
||||
"wep": "string", "gmch": "int", "drad": "float", "cst": "float",
|
||||
"aOdd": "float", "aInc": "float", "rMd": "float", "sctSize": "float", "smx": "float",
|
||||
"nPrvVa": "float", "crPce": "bool", "maintHf": "bool", "caps2": "int64",
|
||||
"tRsld": "bool", "tRsldd": "bool", "tRsldc": "bool", "tRsldi": "bool", "tRsldr": "bool",
|
||||
|
|
@ -607,7 +676,10 @@ MANUAL_KINDS = {
|
|||
}
|
||||
|
||||
|
||||
def _register(shape, kinds: dict, conflicts: set, shapes: dict, seen: set):
|
||||
def _register(shape, kinds: dict, conflicts: set, shapes: dict, seen: set, weak: bool = False):
|
||||
"""Collect global (name -> kind) hints and (name -> shape) frames.
|
||||
weak=False registers A()/R() fields (conflicting kinds are dropped later);
|
||||
weak=True registers Opt() legacy tags only where nothing else claims the name."""
|
||||
if id(shape) in seen:
|
||||
return
|
||||
seen.add(id(shape))
|
||||
|
|
@ -615,37 +687,43 @@ def _register(shape, kinds: dict, conflicts: set, shapes: dict, seen: set):
|
|||
shapes.setdefault(shape.name, shape)
|
||||
fields = shape.fields if isinstance(shape, (Shape, Seq)) else []
|
||||
for f in fields:
|
||||
items = [f]
|
||||
items, is_opt = [f], False
|
||||
if isinstance(f, Opt):
|
||||
items = [f.field]
|
||||
items, is_opt = [f.field], True
|
||||
elif isinstance(f, If):
|
||||
items = f.inner.fields if isinstance(f.inner, Seq) else [f.inner]
|
||||
elif isinstance(f, Repeat):
|
||||
items = f.elem.fields if isinstance(f.elem, Seq) else [f.elem]
|
||||
elif not isinstance(f, Field):
|
||||
continue
|
||||
for fld in items:
|
||||
if not isinstance(fld, Field):
|
||||
continue
|
||||
_register_type(fld.name, fld.type, kinds, conflicts, shapes, seen)
|
||||
_register_type(fld.name, fld.type, kinds, conflicts, shapes, seen, weak, is_opt)
|
||||
|
||||
|
||||
def _register_type(name, t, kinds, conflicts, shapes, seen):
|
||||
def _register_type(name, t, kinds, conflicts, shapes, seen, weak=False, is_opt=False):
|
||||
if isinstance(t, str):
|
||||
if t in PRIMITIVES:
|
||||
if name in kinds and kinds[name] != t:
|
||||
conflicts.add(name)
|
||||
kinds.setdefault(name, t)
|
||||
if weak:
|
||||
if is_opt and name not in conflicts:
|
||||
kinds.setdefault(name, t)
|
||||
elif not is_opt:
|
||||
if name in kinds and kinds[name] != t:
|
||||
conflicts.add(name)
|
||||
kinds.setdefault(name, t)
|
||||
elif isinstance(t, Shape):
|
||||
if name and not t.name:
|
||||
shapes.setdefault(name, t)
|
||||
_register(t, kinds, conflicts, shapes, seen)
|
||||
_register(t, kinds, conflicts, shapes, seen, weak)
|
||||
elif isinstance(t, Seq):
|
||||
_register(t, kinds, conflicts, shapes, seen)
|
||||
_register(t, kinds, conflicts, shapes, seen, weak)
|
||||
elif isinstance(t, (CArr, NArr)):
|
||||
e = t.elem
|
||||
if isinstance(e, Field):
|
||||
_register_type(e.name, e.type, kinds, conflicts, shapes, seen)
|
||||
_register_type(e.name, e.type, kinds, conflicts, shapes, seen, weak)
|
||||
elif isinstance(e, (Shape, Seq)):
|
||||
_register(e, kinds, conflicts, shapes, seen)
|
||||
_register(e, kinds, conflicts, shapes, seen, weak)
|
||||
if isinstance(t, CArr) and name:
|
||||
shapes.setdefault(name, t)
|
||||
|
||||
|
|
@ -655,6 +733,8 @@ def _build_catalog():
|
|||
_register(ROOT, kinds, conflicts, shapes, set())
|
||||
for k in conflicts: # ambiguous across contexts -> no global hint
|
||||
kinds.pop(k, None)
|
||||
_register(ROOT, kinds, conflicts, shapes, set(), weak=True) # legacy Opt() tags, lowest priority
|
||||
kinds.pop(".", None) # the NULL-name tag carries ints, floats, frames -- never hint it
|
||||
for k, v in MANUAL_KINDS.items():
|
||||
kinds[k] = v
|
||||
return kinds, shapes
|
||||
|
|
@ -667,10 +747,18 @@ RAW_FRAMES = {"RNG"}
|
|||
|
||||
# --- generic walker -----------------------------------------------------------
|
||||
|
||||
_CP1252_UNDEFINED = frozenset((0x81, 0x8d, 0x8f, 0x90, 0x9d))
|
||||
|
||||
|
||||
def _text_plausible(b: bytes) -> bool:
|
||||
"""Guard used only while GUESSING that an unknown tag holds a string. Every
|
||||
byte windows-1252 defines counts as text -- including 0x80-0x9f (system
|
||||
names like "Kor\\x92Voth" use 0x92 = right single quote). Never apply this
|
||||
to a tag that is known to be a string."""
|
||||
if not b:
|
||||
return True
|
||||
ok = sum(1 for c in b if 0x20 <= c < 0x7f or c >= 0xa0 or c in (9, 10, 13))
|
||||
ok = sum(1 for c in b if (0x20 <= c < 0x7f) or (c >= 0x80 and c not in _CP1252_UNDEFINED)
|
||||
or c in (9, 10, 13))
|
||||
return ok * 10 >= len(b) * 9
|
||||
|
||||
|
||||
|
|
@ -745,8 +833,10 @@ class Walker:
|
|||
return False
|
||||
return self.zero(vp + size, end)
|
||||
|
||||
def try_scalar(self, tag_start: int, name_end: int, kind: str):
|
||||
"""Read a candidate scalar; (kind, value, raw, end) or None if impossible."""
|
||||
def try_scalar(self, tag_start: int, name_end: int, kind: str, known: bool = False):
|
||||
"""Read a candidate scalar; (kind, value, raw, end) or None if impossible.
|
||||
known=True means the kind comes from the schema/catalog: a string value
|
||||
is then accepted whatever bytes it holds (only tags must be ASCII)."""
|
||||
d, n = self.d, self.n
|
||||
vp = self.value_pos(tag_start, name_end)
|
||||
if kind == "string":
|
||||
|
|
@ -759,10 +849,10 @@ class Walker:
|
|||
if vp + 4 + ln > n:
|
||||
self._eof_limited = True
|
||||
return None
|
||||
raw = d[vp + 4:vp + 4 + ln]
|
||||
if not _text_plausible(raw):
|
||||
raw = d[vp + 4:vp + 4 + ln] # ln == 0 (empty string) is legal
|
||||
if not known and not _text_plausible(raw):
|
||||
return None
|
||||
size, value = 4 + ln, raw.decode(ENCODING)
|
||||
size, value = 4 + ln, raw.decode(ENCODING, errors="replace")
|
||||
else:
|
||||
size = 4 if kind == "word" else SIZES[kind]
|
||||
if vp + size > n:
|
||||
|
|
@ -810,6 +900,13 @@ class Walker:
|
|||
return True
|
||||
if depth <= 0:
|
||||
return True
|
||||
# a tag the catalog knows to be a string is read as one without any
|
||||
# text test, so a non-ASCII value (cp1252 0x92 in "Kor'Voth") can never
|
||||
# veto the layout of the item that precedes it
|
||||
if self.kinds.get(name) == "string":
|
||||
r = self.try_scalar(p, ne, "string", known=True)
|
||||
if r is not None and self.plausible_item(r[3], depth - 1, allow_empty):
|
||||
return True
|
||||
for kind in self.GUESS_ORDER:
|
||||
r = self.try_scalar(p, ne, kind)
|
||||
if r is None:
|
||||
|
|
@ -943,7 +1040,7 @@ class Walker:
|
|||
if mode == "eof" and not self._eof_limited:
|
||||
break
|
||||
for i, cand in enumerate(order):
|
||||
r = self.try_scalar(p, ne, cand)
|
||||
r = self.try_scalar(p, ne, cand, known=(i == 0 and kind is not None))
|
||||
if r is None:
|
||||
continue
|
||||
if mode == "eof" or self.plausible_item(r[3], 2, allow_empty=(mode == "empty-ok")):
|
||||
|
|
@ -1108,12 +1205,20 @@ class Applier:
|
|||
if nxt is not None and nxt.name is not None and nxt.name.lower() == f.field.name.lower():
|
||||
self.apply_field(f.field, cur, out, path)
|
||||
elif isinstance(f, If):
|
||||
if out.get(f.key) == f.equals:
|
||||
if f.holds(out):
|
||||
inner = f.inner
|
||||
if isinstance(inner, Seq):
|
||||
self.apply_fields(inner.fields, cur, out, path)
|
||||
else:
|
||||
self.apply_field(inner, cur, out, path)
|
||||
elif isinstance(f, Repeat):
|
||||
items = []
|
||||
while not cur.done and cur.peek().name == f.lead:
|
||||
before = cur.i
|
||||
items.append(self.convert_elem(f.elem, cur, f"{path}/{f.key}[{len(items)}]"))
|
||||
if cur.i == before:
|
||||
break
|
||||
out[f.key] = items
|
||||
elif isinstance(f, Until):
|
||||
items = []
|
||||
while not cur.done:
|
||||
|
|
@ -1288,6 +1393,11 @@ class Applier:
|
|||
if k in ("int", "float") and len(raw) == 4:
|
||||
self.issue("warn", path, node, "int64 expected, 4-byte word read (width mismatch)")
|
||||
return struct.unpack("<i", raw)[0]
|
||||
elif kind == "string":
|
||||
# an empty string is "len 0" = 4 zero bytes, byte-identical to int 0;
|
||||
# an unhinted walker reads it as a word -- that IS the empty string
|
||||
if k in ("int", "float") and raw == b"\0\0\0\0":
|
||||
return ""
|
||||
self.issue("error", path, node, f"expected {kind}, read {k}")
|
||||
return node.value
|
||||
|
||||
|
|
@ -1376,8 +1486,8 @@ def _summary_lines(res: SaveResult, path: str) -> list[str]:
|
|||
f"issues: {res.count('error')} error, {res.count('warn')} warn, {res.count('info')} info"]
|
||||
s = t.get("summary") or {}
|
||||
if isinstance(s, dict):
|
||||
lines.append(f"summary: game={s.get('gameName')!r} turn={s.get('turn')} numSys={s.get('numSys')} "
|
||||
f"players={len(s.get('players') or [])} scenario={s.get('scenario')!r}")
|
||||
lines.append(f"summary: game={s.get('GameName')!r} turn={s.get('Turn')} numSys={s.get('NumSys')} "
|
||||
f"players={len(s.get('Players') or [])} scenario={s.get('Scenario')!r}")
|
||||
sim = t.get("sim") or {}
|
||||
if isinstance(sim, dict):
|
||||
lines.append(f"sim: gameName={sim.get('GameName')!r} players={len(sim.get('players') or [])} "
|
||||
|
|
|
|||
|
|
@ -208,17 +208,18 @@ class Fixture:
|
|||
if f.field.name in OPT_EMIT:
|
||||
self.emit_field(f.field, out, ctx)
|
||||
elif isinstance(f, sr.If):
|
||||
if out.get(f.key) == f.equals:
|
||||
if f.holds(out):
|
||||
inner = f.inner
|
||||
if isinstance(inner, sr.Seq):
|
||||
self.emit_fields(inner.fields, out, ctx)
|
||||
else:
|
||||
self.emit_field(inner, out, ctx)
|
||||
elif isinstance(f, sr.Repeat):
|
||||
out[f.key] = [self.emit_elem(f.elem, ctx) for _ in range(ARRAY_LEN)]
|
||||
elif isinstance(f, sr.Until):
|
||||
out[f.key] = self.filler_items()
|
||||
elif isinstance(f, sr.Rest):
|
||||
if top:
|
||||
out[f.key] = self.cd_table()
|
||||
pass # nothing extra is written
|
||||
else:
|
||||
raise TypeError(f)
|
||||
|
||||
|
|
@ -265,7 +266,7 @@ class Fixture:
|
|||
|
||||
def emit_carr(self, name, carr: sr.CArr, ctx) -> list:
|
||||
self.w.begin(name)
|
||||
self.w.int("Count", ARRAY_LEN)
|
||||
self.w.int(".", ARRAY_LEN) # VectorHelper count tag is "."
|
||||
items = [self.emit_elem(carr.elem, ctx) for _ in range(ARRAY_LEN)]
|
||||
self.w.end()
|
||||
return items
|
||||
|
|
@ -280,31 +281,15 @@ class Fixture:
|
|||
self.emit_field(elem, sub, ctx)
|
||||
return sub.get(elem.key)
|
||||
if isinstance(elem, sr.Shape):
|
||||
return self.emit_shape(".", elem) # R1: element tags are often "."
|
||||
return self.emit_shape(".", elem) # VectorHelper element tag is "."
|
||||
if isinstance(elem, sr.CArr):
|
||||
return self.emit_carr(".", elem, ctx)
|
||||
if elem == "any":
|
||||
return self.generic_frame(".")
|
||||
if elem in sr.PRIMITIVES:
|
||||
return self.emit_scalar("Item", elem, ctx)
|
||||
return self.emit_scalar(".", elem, ctx)
|
||||
raise TypeError(elem)
|
||||
|
||||
def cd_table(self) -> list:
|
||||
"""Unframed 4th section: a framed string array, an unknown-shaped player
|
||||
block and two AI blocks -- all opaque to the schema."""
|
||||
w = self.w
|
||||
items = []
|
||||
w.begin("cdt")
|
||||
w.int("Count", 2)
|
||||
w.string("Item", "cd-a")
|
||||
w.string("Item", "cd-b")
|
||||
w.end()
|
||||
items.append({"_name": "cdt", "_items": [_plain_scalar("Count", "int", 2),
|
||||
_plain_scalar("Item", "string", "cd-a"),
|
||||
_plain_scalar("Item", "string", "cd-b")]})
|
||||
items.append(self.generic_frame("cdplayer"))
|
||||
items.append(self.generic_frame("cdai"))
|
||||
items.append(self.generic_frame("cdai"))
|
||||
return items
|
||||
|
||||
|
||||
def build_fixture(padding="joint", seed=1):
|
||||
"""-> (inflated bytes, expected typed dict)"""
|
||||
|
|
|
|||
|
|
@ -3,8 +3,9 @@
|
|||
|
||||
/usr/bin/python3 -m unittest -v test_save_reader
|
||||
|
||||
NOTE: these prove the reader against its OWN framing assumptions. Validation
|
||||
against a real Sword of the Stars save is still pending.
|
||||
NOTE: these prove the reader against its OWN framing assumptions (the synthetic
|
||||
writer). The real-save check is `save_reader.py verify/results/saves/turn*-state.sav
|
||||
--strict` (exit 0 on all three as of the schema-gaps-resolved.md patch).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
|
|
@ -248,7 +249,11 @@ class SchemaRoundTripTest(unittest.TestCase):
|
|||
self.assertIsInstance(ship["RefCap"], float)
|
||||
self.assertIn("BQ2", ship)
|
||||
self.assertIn("pop", ship)
|
||||
colors = [p["slot"]["fxCrId"] for p in res.typed["summary"]["players"]]
|
||||
self.assertEqual(len(res.typed["cdTable"]["ids"]), 2)
|
||||
self.assertEqual(len(res.typed["customData"]), 2) # Repeat(CD) at the root
|
||||
self.assertEqual(len(sim["species"]), 2) # Repeat(ISsp/ISsu) without a count
|
||||
self.assertEqual(len(sim["turnstats"]["players"][0]["hist"]["stats"]), 2)
|
||||
colors = [p["Slot"]["FxCrID"] for p in res.typed["summary"]["Players"]]
|
||||
custom = [c for c in colors if c["idx"] == -1]
|
||||
self.assertTrue(custom and all("r" in c for c in custom)) # -1 -> RGB follows
|
||||
self.assertTrue(all("r" not in c for c in colors if c["idx"] != -1))
|
||||
|
|
@ -304,18 +309,104 @@ class SchemaRoundTripTest(unittest.TestCase):
|
|||
self.assertTrue(any(i.level == "warn" for i in res.issues))
|
||||
|
||||
def test_coerce_retypes_words(self):
|
||||
"""A schema float stored as a word the walker guessed 'int' is re-read from raw bytes."""
|
||||
"""A 4-byte word the walker guessed one way is re-read from its raw bytes
|
||||
when the schema (matched positionally, so the walker had no hint) says otherwise."""
|
||||
w = sw.SaveWriter()
|
||||
w.begin("Tmrs")
|
||||
w.int("tstl", 0x7F7FFFFF)
|
||||
w.float("tctl", 60.0) # 0x42700000, walker may call it float; schema says int
|
||||
w.int("tqtl", 0x7F7FFFFF)
|
||||
w.int("tqtle", 0)
|
||||
w.begin("Wd")
|
||||
w.float("zzza", 60.0) # 0x42700000: guessed float; schema says int
|
||||
w.int("zzzb", 0x7F7FFFFF) # guessed int; schema says float (FLT_MAX)
|
||||
w.end()
|
||||
schema = sr.Seq([sr.R("t", sr.Tmrs)])
|
||||
res = sr.read_bytes(w.bytes(), padding="joint", schema=schema)
|
||||
self.assertEqual(res.typed["t"]["tctl"], 0x42700000)
|
||||
self.assertEqual(res.typed["t"]["tstl"], 0x7F7FFFFF)
|
||||
shape = sr.Shape("Wd", [sr.R("a", "int"), sr.R("b", "float"), sr.Rest()])
|
||||
res = sr.read_bytes(w.bytes(), padding="joint", schema=sr.Seq([sr.R("t", shape)]))
|
||||
self.assertEqual(res.typed["t"]["a"], 0x42700000)
|
||||
self.assertEqual(res.typed["t"]["b"], 3.4028234663852886e+38)
|
||||
self.assertEqual(res.count("error"), 0)
|
||||
|
||||
def test_non_ascii_string_value_never_vetoes_layout(self):
|
||||
"""Gap 1+2: a 1-byte bool followed by a bool and a cp1252 string with a
|
||||
byte in 0x80-0x9f (0x92 = right single quote) must parse exactly like an
|
||||
ASCII name -- the text test applies to tags, never to string values."""
|
||||
for name in ("Kor’Voth", "Kaa’Vaalu", "Plain"):
|
||||
w = sw.SaveWriter()
|
||||
w.begin("Tst") # unknown frame: hints come from the catalog by name
|
||||
w.int("haltc", 1)
|
||||
w.int("haltt", 0)
|
||||
w.bool("haltv", False) # 5-char tag + 1 byte -> 2 pad bytes (joint)
|
||||
w.bool("vnh", False)
|
||||
w.string("Name", name) # encodes 0x92 for the quote
|
||||
w.int("VFlags", 0)
|
||||
w.end()
|
||||
data = w.bytes()
|
||||
if "’" in name:
|
||||
self.assertIn(b"\x92", data)
|
||||
wk, root = walk(data)
|
||||
got = [(c.name, c.kind, c.value) for c in root.children[0].children]
|
||||
self.assertEqual(got, [("haltc", "int", 1), ("haltt", "int", 0), ("haltv", "bool", False),
|
||||
("vnh", "bool", False), ("Name", "string", name), ("VFlags", "int", 0)])
|
||||
self.assertEqual(wk.stats["resyncs"], 0)
|
||||
self.assertEqual(wk.stats["hint_failures"], 0)
|
||||
# same guarantee for an unhinted (guessed) tag holding non-ASCII text
|
||||
w = sw.SaveWriter()
|
||||
w.bool("zzq", True)
|
||||
w.string("zzs", "Kor’Voth")
|
||||
w.int("zzz", 5)
|
||||
wk, root = walk(w.bytes())
|
||||
self.assertEqual([(c.kind, c.value) for c in root.children],
|
||||
[("bool", True), ("string", "Kor’Voth"), ("int", 5)])
|
||||
self.assertEqual(wk.stats["resyncs"], 0)
|
||||
|
||||
def test_empty_string_values(self):
|
||||
"""Gap 5: an empty string is 4 zero bytes, byte-identical to int 0. It
|
||||
must be read as "" both when the tag is hinted (walker) and when the
|
||||
walker had no hint and guessed int (applier coerce)."""
|
||||
w = sw.SaveWriter()
|
||||
w.begin("CreateParams")
|
||||
w.string("Name", "g")
|
||||
w.int("ID", 1); w.int("RSeed", 2); w.int("AID", 3)
|
||||
w.string("Key", "") # 03 00 00 00 'Key' 00 00 00 00 + 1 pad
|
||||
w.end()
|
||||
data = w.bytes()
|
||||
self.assertIn(b"\x03\x00\x00\x00Key\x00\x00\x00\x00\x00", data)
|
||||
wk, root = walk(data) # hinted: CreateParams is in the catalog
|
||||
key = root.children[0].children[4]
|
||||
self.assertEqual((key.name, key.kind, key.value, key.hinted), ("Key", "string", "", True))
|
||||
self.assertEqual(wk.issues, [])
|
||||
# unhinted: an unknown frame name -> walker guesses int 0; positional schema wants a string
|
||||
w = sw.SaveWriter()
|
||||
w.begin("Unknown")
|
||||
w.string("zzk", "")
|
||||
w.string("zzm", "")
|
||||
w.end()
|
||||
shape = sr.Shape("Unknown", [sr.R("k", "string"), sr.R("m", "string"), sr.Rest()])
|
||||
res = sr.read_bytes(w.bytes(), padding="joint", schema=sr.Seq([sr.R("u", shape)]), strict=True)
|
||||
self.assertEqual((res.typed["u"]["k"], res.typed["u"]["m"]), ("", ""))
|
||||
self.assertEqual(res.count("error"), 0)
|
||||
|
||||
def test_prisoner_hold_gating(self):
|
||||
"""Gap 4: PrNSp and the (PrSp, PrNum) pairs are written only when PrMax > 0."""
|
||||
def prish(pr_max, pairs):
|
||||
w = sw.SaveWriter()
|
||||
w.begin("PrisH")
|
||||
w.int("PrMax", pr_max)
|
||||
if pr_max > 0:
|
||||
w.int("PrNSp", len(pairs))
|
||||
for sp, num in pairs:
|
||||
w.int("PrSp", sp)
|
||||
w.int("PrNum", num)
|
||||
w.end()
|
||||
return sr.read_bytes(w.bytes(), padding="joint", strict=True,
|
||||
schema=sr.Seq([sr.R("p", sr.PrisonerHold)])).typed["p"]
|
||||
empty = prish(0, [])
|
||||
self.assertEqual(empty["PrMax"], 0)
|
||||
self.assertNotIn("prisoners", empty)
|
||||
full = prish(40, [(1, 12), (3, 5)])
|
||||
self.assertEqual(full["PrMax"], 40)
|
||||
self.assertEqual(full["prisoners"], [{"PrSp": 1, "PrNum": 12}, {"PrSp": 3, "PrNum": 5}])
|
||||
# a PrMax>0 frame with the pairs missing is a real schema error
|
||||
w = sw.SaveWriter()
|
||||
w.begin("PrisH"); w.int("PrMax", 40); w.end()
|
||||
with self.assertRaises(sr.SaveFormatError):
|
||||
sr.read_bytes(w.bytes(), padding="joint", strict=True, schema=sr.Seq([sr.R("p", sr.PrisonerHold)]))
|
||||
|
||||
|
||||
class CliTest(unittest.TestCase):
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue