diff --git a/campaign/board.md b/campaign/board.md index 96def78..9464ea2 100644 --- a/campaign/board.md +++ b/campaign/board.md @@ -7,8 +7,8 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`). |---|---|---|---|---|---|---| | RTTI class inventory | meta | mapped | high | 100% | 2026-09-07 | `findings/objects/00-inventory.md` — 1924 types, engine=Mars | | `.gob` format | subsystem | mapped | high | 100% | 2026-09-07 | renamed uncompressed ZIP (community-known) | -| `Mars::AppStartup` (entry) | control-flow | backlog | — | 0% | 2026-09-07 | bootstrap → main loop | -| main loop / tick dispatch | control-flow | in-progress | — | 0% | 2026-09-07 | turn SM known from names: strategic round -> SETurnEndPending -> SEProcessTurn -> querying round -> combat round -> SNMAllCombatDone. Ghidra: StrategyMapScreen ctor, ScreenBar::ToggleList, CombatScreen load | +| `Mars::AppStartup` (entry) | control-flow | mapped | high | 100% | 2026-09-07 | entry 0x00925794 -> WinMain 0x0089dd30 -> DemoApp ctor 0x0089c950 -> Mars::Application::Initialize 0x008a0e50 (config/affinity, D3D9, window, sound thread, OnStartup -> net thread). findings/control-flow/turn-spine.md | +| main loop / tick dispatch | control-flow | mapped | high | 90% | 2026-09-07 | Application::Run 0x0089f5b0: FrameTimer, PanelManager, DemoApp::OnUpdate 0x00898800 / OnTick 0x0089a640 / OnRender 0x00899210. Turn pipeline: EndTurn 0x00783be0 -> BeginProcessTurn 0x007d98e0 -> StrategyServer::ProcessTurn 0x007dc6c0 -> RunCombatRound 0x007cbe80 -> SETurnResults -> ResumePlaying 0x007ddc90. Lockstep on every machine | | `Game::ClientPlayer` / `AIPlayer` | object | mapped | high | 80% | 2026-09-07 | empire STATE = Game::ServerPlayer (~110 members, abs 0x28..0x3dc; IStreamable +0x3a0; read FUN_008804d0 / write FUN_008563e0). ClientPlayer = client view. findings/objects/struct-recovery.md. Verify vs real save pending | | `Game::StarSystem` | object | mapped | high | 80% | 2026-09-07 | Game::ServerSystem = star system + colony (~90 members, abs 0x18..0x2d4; read FUN_0075d4b0 / write FUN_00749630). PlayerView per-player fog view. Verify vs real save pending | | `Game::Planet` / `DOPlanet` | object | mapped | high | 100% | 2026-09-07 | Planet : Actor is a RENDER actor, not streamed; all colony state is in ServerSystem (+PlayerView in NVs map) | @@ -25,10 +25,10 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`). | string / config intel | meta | mapped | high | 100% | 2026-09-07 | findings/subsystems/strings-and-config.md | | Mars brace-block parser | subsystem | verified | high | 100% | 2026-09-07 | verify/parsers/ (mars_data, flat_kv, manifest, effect_txt, verify.py): 1595/1595 files parse, 0 dangling cross-links; catalogs in verify/results/data-catalogs/ | | networking (SNM/FNM + GameSpy) | subsystem | backlog | — | 0% | 2026-09-07 | ~60 SNM strategy msgs, FNM file-xfer/host-migration; lockstep sim | -| battle-load: thread contention | subsystem | backlog | — | 0% | 2026-09-07 | root cause hunt: CreateThread sites FUN_00902350, FUN_00736e30, FUN_008a0e50; streaming-sound thread; lockstep sync waits | +| battle-load: thread contention | subsystem | mapped | med | 50% | 2026-09-07 | 3 CreateThread sites CLEARED (net watchdog 0x00901f40, TIME_CRITICAL audio streaming 0x008ef1d0, starmap blob mesh 0x00735bb0). Sim+combat load are on the MAIN thread -> candidates: audio thread CS (g_musicCS) or D3D9 driver threads. Next: dynamic profile under x32dbg once the game runs | | UI screen & flow map | meta | mapped | high | 100% | 2026-09-07 | findings/subsystems/ui-screen-map.md - 36 screens; screens are C++ on Mars controls (NOT data); turn state machine recovered | | `Game::StrategyServer` (sim block) | object | mapped | high | 80% | 2026-09-07 | top-level sim state; read FUN_007d27a0 / write FUN_0079fa70; IStreamable +0 | | stream primitive API | subsystem | mapped | high | 100% | 2026-09-07 | IStreamable vft: +0x18 string, +0x1c bool, +0x20 float, +0x24 int, +0x28 nested, +0x30 raw; FUN_00816490 = NetworkObject handle id | | real save for verification | verify | in-progress | — | 0% | 2026-09-07 | need the game running (DXVK+lavapipe on VM140) -> autosave -> parse with recovered layouts | | save_reader.py | verify | in-progress | — | 0% | 2026-09-07 | Streamable reader (gzip, name-tagged, BEEFBEEF framing) from reference + recovered layouts | -| Ghidra type write-back | meta | in-progress | — | 0% | 2026-09-07 | create struct datatypes + rename serializers in the sots project so ReVa sessions inherit the map | +| Ghidra type write-back | meta | verified | high | 100% | 2026-09-07 | structs saved in project (ServerSystem 87f, ServerPlayer 110f, StarFleet, StarShip, StrategyServer partial, 22 nested); 52 serializers + primitives + ~60 spine fns renamed; decompile shows field names | diff --git a/campaign/open-questions.md b/campaign/open-questions.md index 5ef434e..d4032e8 100644 --- a/campaign/open-questions.md +++ b/campaign/open-questions.md @@ -20,3 +20,5 @@ Each links to the finding that raised it. Promoted to backlog or closed by **re- - **Engine parser leniency** — 12 shipped shipsections are syntactically broken (unclosed `{`, extra `}`) yet load; keys and identifiers are case-insensitive. Reimplementation must match this leniency. `.effect` is its own `TXT`/`BEGIN-END` format, not brace-block (corrects round one). (from [[data-parsers]]) - **Struct recovery leftovers** — `PlayerColorID` exact on-disk width (writer `FUN_008b9cb0` undecompiled); `ServerSystem+0x10` owner type; `TechTree` per-tech body; `CdPlayer` block. R2's `OID = PID*16` is an id-allocation pattern, not in this code. (from [[struct-recovery]]) - **Resolved (R1/R2 contradictions)** — `Bats2`/`rcex` are int64 (R2 wrong); `Abdn`/`Dstyd` bools, `ltis` int; `TRM`/`CstR/E/T`/`shrm`/`RefCap`/`RepCap`/PlayerView `Infra` are floats; `pswd` string; `TShn`/`ETS`/diplomacy counters int16 in memory, int32 on disk; `Nexp` carries `xid/xmin/xmax/xper`; `FtOrig` is Vector3. (from [[struct-recovery]]) +- **Battle-load, narrowed** — sim/combat load run on the main thread; the only other threads are net watchdog, TIME_CRITICAL audio streaming (`g_musicCS`), and a star-map mesh builder. Hypothesis: audio-thread critical-section contention or D3D9 runtime/driver threads on many cores. Needs a dynamic profile (x32dbg / ETW) under the software-GPU stack. (from [[turn-spine]]) +- **Spine leftovers** — `StrategyServer` struct partial (41 fields); static-initialiser region 0x009be000–0x009c1400 undisassembled; `Mars::Stream` vftable not located; several small ProcessTurn phase fns unnamed. (from [[turn-spine]]) diff --git a/findings/control-flow/.keep.md b/findings/control-flow/.keep.md deleted file mode 100644 index 0c44fd4..0000000 --- a/findings/control-flow/.keep.md +++ /dev/null @@ -1 +0,0 @@ -# placeholder — populated during the campaign diff --git a/findings/control-flow/turn-spine.md b/findings/control-flow/turn-spine.md new file mode 100644 index 0000000..201d954 --- /dev/null +++ b/findings/control-flow/turn-spine.md @@ -0,0 +1,250 @@ +# Ghidra write-back + turn/main-loop spine — Sword of the Stars (2006) + +Program `sots` / "Sword of the Stars.exe", ImageBase 0x00400000. All addresses are VAs. Ghidra 12.0.4 headless, +project `/srv/re-lab/ghidra-projects/sots`; all changes below are **saved in the project** (four headless runs, each +"Save succeeded"). Scripts (CT111 `/root/`, copies in `/srv/re-lab/handoff/scripts/`): `SpineRecon.java`..`SpineRecon4.java` +(read-only recon), `WriteBack.java` (types + renames + labels), `FixThis.java` (class-struct merge + serializer `this` +views), `FixPlayer.java`. Raw decompiles used for the analysis: CT111 `/tmp/spine/*.c`; the verification decompiles are in +`/srv/re-lab/handoff/ghidra-verify/`. + +Inputs: `findings/objects/struct-recovery.md` (member tables), `findings/objects/ghidra-recon.md` (affinity, vftables), +`findings/subsystems/ui-screen-map.md` §3 (SE*/SNM* names). + +--- + +## 1. Type write-back + +### 1.1 Where the types live (important for anyone continuing in Ghidra) +Ghidra resolves the `this` type of a `__thiscall` function in class namespace `Game::X` to the structure +**`/Game/X`** (`VariableUtilities.findOrCreateClassStruct`). The recovered structs were therefore merged *into those +class structs*, so every function placed in the class namespace decompiles with field names automatically: + +| Ghidra data type | size | fields | note | +|---|---|---|---| +| `/Game/ServerSystem` | 0x2d8 | 87 | abs offsets from struct-recovery §1 (+ base vptrs/netId/owner) | +| `/Game/ServerPlayer` | 0x3e0 | 110 | §2; IStreamable sub-object at +0x3a0 | +| `/Game/StarFleet` | 0x120 | 21 | §3 | +| `/Game/StarShip` | 0xb0 | 26 | §4 | +| `/Game/StrategyServer` | 0x320 | 41 | **partial**: offsets from `StrategyServer::Write` + the spine functions (Frame/turn @+0xc, Systems @+0x44, Players @+0x54, Fleets @+0x64, Acts @+0x74, OnEventCallback @+0x170, StrategyEvents list @+0x2b8, bProcessingTurn @+0x2f1, …) | +| `/Game/Population` 0x14, `PopulationGroup` 0x18, `Morale` 0x20, `MoraleEvent` 0x50, `IndependenceInfo` 0x70, `PlayerColorID` 4, `ShipBuildOrder` 0x18, `DiplomacyStats` 0x24, `PlayerReport` 0x30, `PlayerAlliances` 0x10, `NodeRoute` 0x10, `FlightPlan` 0x38, `FlightPlan/Waypoint` 0x1c, `ShipHealth` 0x10, `PrisonerHold` 0x18, `EventStorage` 0x1c, `CivilianRatios` 0x2c, `ShipRecords` 0x44, `SpyReport` 0x34, `/Game/StarSystem/OutputRates` 0x1c, `/Game/StarSystem/PlayerView` 0x9c, `/Mars/Vector3` 12 | | | nested types from §1.1–§2.5 / §3.1 | +| `/SOTS/std::string` 0x1c, `std::vector` 12, `std::list` 8, `std::map` 8, `Game::FleetLayout` 0x24 | | | helpers. `std::string` layout **verified** from `Stream::WriteString` (`if (s->_Myres > 15) p = s->_Ptr`): `_Bx`@0 (16), `_Mysize`@0x10, `_Myres`@0x14, `_Alval`@0x18 | +| `/SOTS/Game::ServerSystem_ser8`, `StarFleet_ser8`, `StarShip_ser8`, `ServerPlayer_ser928` | | | **serializer views**: the same fields shifted by the IStreamable COL offset (8 / 0x3a0), because inside `X::Read/Write` `this = object + COL`. Applied (custom storage, `this` in ECX) to the eight Read/Write functions. For ServerPlayer only the +0x3a0.. tail is visible; the rest stays as `this + -0x378` etc. | +| `/Game/BuildQueue`, `TechTree`, `ShipDesign`, `Tech`, `CommMessageContainer`, `FleetNameGenerator`, `AIRebellion`, `AIEncounterFlags`, `ServerNodeGraph`, `ServerTradeManager`, `IServerSpyManager`, `AttribMap`, `SVScriptObject`, `StrategyEvent`, `/Mars/Stream` | 0 | | opaque, pointer targets only | + +Every struct carries a description citing `struct-recovery.md`; gaps are left undefined. One extra member was added from +the spine work: `ServerPlayer+0xf9 bTurnDone_nonser` (tested by `StrategyServer::OnPlayerEndTurn`, not serialized), and +`ServerPlayer+0x164 Status` is the field `SNMSetPlayerStatus` writes (4 = "done", 1 = playing). + +### 1.2 Renames (all with plate comments citing the finding doc) +*Serializers* (52): `Game::ServerSystem::Read/Write` (0x0075d4b0/0x00749630), `Game::ServerPlayer::Read/Write` +(0x008804d0/0x008563e0), `Game::StarFleet::Read/Write` (0x00702470/0x00701070), `Game::StarShip::Read/Write` +(0x00853fa0/0x008291f0), `Game::StrategyServer::Read/Write` (0x007d27a0/0x0079fa70), `Game::StarMapNode::Read/Write` +(0x00727790/0x00727820), `Game::StarSystem::PlayerView::Read/Write`, `::OutputRates::Read/Write`, `Population`, +`PopulationGroup`, `IndependenceInfo`, `Morale`, `MoraleEvent`, `ShipBuildOrder`, `PlayerNotes`, `SpyReport`, +`PlayerReport`, `DiplomacyStats::Write`, `FlightPlan`, `FlightPlan::Waypoint`, `NodeRoute`, `PrisonerHold`, +`EventStorage::Write`, `PlayerAlliances::Write`, `ShipHealth::Write`, `PlayerColorID::Write`, `Mars::Vector3::Write`, +`TechTree::Write`, `ShipRecords::Write`, `CivilianRatios::Write` — exactly the addresses in struct-recovery §0. + +*Stream primitives* (`Mars::Stream::…`): `WriteString` 0x008b9d70, `WriteBool` 0x008b9c20, `WriteFloat` 0x008b9be0, +`WriteInt` 0x008b9d50, `WriteInt16AsInt` 0x008b9d00, `WriteInt8AsInt` 0x008b9cb0, `WriteInt64` 0x008b9c60, +`WriteHandleId` 0x00816490, `ReadFloat` 0x008b9bc0, `ReadInt` 0x008b9d20, `ReadBool` 0x008b9c00, `ReadString` +0x008b9d90, `ReadInt64` 0x008b9c40, `ReadInt16` 0x008b9cd0, `ReadHandle` 0x008164d0. Each plate comment lists the Stream +vftable slot map (+0x10 ReadIntRef, +0x14 ReadNested, +0x18 String, +0x1c Bool, +0x20 Float, +0x24 Int, +0x28 Nested, ++0x30 Raw). The `Mars::Stream` vftable itself was not located this round (the wrappers are the only anchor). + +*Affinity / app*: `Process_PinAffinity` 0x0089ee70; `Mars::Application::Initialize` 0x008a0e50 (+ secondary label +`AppStartup_ReadConfig` at the same address, as requested); `WinMain` 0x0089dd30; `Mars::Application::Run` 0x0089f5b0; +`Mars::Application::PumpMessages` 0x0089f1c0; `Mars::Application::Application` 0x008a0170; `CreateAppWindow` 0x0089fe70; +`Mars::FrameTimer::Update` 0x0090c700; `Mars::TimerList::Dispatch` 0x008e5ac0; `Mars::AppStartup::OnConfigToken` +0x0089f4d0; `Mars::ConfigParser::ParseFile` 0x008cd820; `Game::DemoApp::{DemoApp 0x0089c950, OnStartup 0x0089d610, +OnShutdown 0x0089dfb0, OnUpdate 0x00898800, OnTick 0x0089a640, OnRender 0x00899210, ShouldSleepWhenInactive 0x008986a0, +CreateStrategyGame 0x00898b00}`. Globals: `g_pApplication` 0x00b2d540, `g_pDemoApp` 0x00b2d0bc. + +*Threads, game creation, turn spine*: see §2/§3 (every function named there was renamed; ~60 functions). + +*Network message registry*: 82 registry entries labelled `NetMsgReg_` (EOL comment = id/factory), the 21 unnamed +factory functions renamed `Game::::Create`, `Mars::NetMessageRegistry::Register` 0x008d2290, +`g_NetMsgRegistryById` 0x00b2ddf8. EOL comments on the three `CreateThread` sites. + +### 1.3 Verification (re-decompile with the new types) +`Game::ServerSystem::Write` (0x00749630), `this` typed `ServerSystem_ser8*` — field names appear for every tag +(`/srv/re-lab/handoff/ghidra-verify/verify_ServerSystem_Write.c`, 125 `this->` references; Read: 205): +```c +void __thiscall Game::ServerSystem::Write(ServerSystem_ser8 *this, void *stream) + ... + StarMapNode::Write(stream); + Mars::Stream::WriteFloat(pvVar3,"R",&this->R,0xffffffff,uVar5); + Mars::Stream::WriteFloat(pvVar3,"G",&this->G,0xffffffff); + Mars::Stream::WriteFloat(pvVar3,"B",&this->B,0xffffffff); + Mars::Stream::WriteFloat(pvVar3,"A",&this->A,0xffffffff); + Mars::Stream::WriteInt(pvVar3,"Idx",&this->Idx,0xffffffff); + Mars::Stream::WriteInt(pvVar3,"Size",&this->Size,0xffffffff); + Mars::Stream::WriteFloat(pvVar3,"Suit",&this->Suit,0xffffffff); + Mars::Stream::WriteInt(pvVar3,"Res",&this->Res,0xffffffff); + Mars::Stream::WriteInt(pvVar3,"ARes2",&this->ARes2,0xffffffff); + Mars::Stream::WriteInt(pvVar3,"MRes",&this->MRes,0xffffffff); + Mars::Stream::WriteBool(pvVar3,"NoRebAI",&this->NoRebAI,0xffffffff); + Mars::Stream::WriteInt(pvVar3,"TRes",&this->TRes,0xffffffff); + Mars::Stream::WriteInt(pvVar3,"Pop",&this->Pop,0xffffffff); + local_1c = &this->Pop2; local_24 = Mars::StreamableHelper::vftable; + (**(code **)(*(int *)pvVar3 + 0x28))("Pop2",&local_24); + Mars::Stream::WriteFloat(pvVar3,"Infra",&this->Infra,0xffffffff); + Mars::Stream::WriteInt(pvVar3,"PvPop",&this->PvPop,0xffffffff); + ... +``` +`Game::StrategyServer::Write` (COL 0, plain `StrategyServer*`): `WriteString(stream,"KeyPath",&this->KeyPath)`, +`local_60 = (this->NodeMapLines._Mylast - _Myfirst)/0x14` (NMSz), `WriteInt(stream,"ModCount",&this->ModCount)`, +`WriteInt(stream,"Frame",&this->Frame)`, `("GameID",this->GameID)`, `this->Attrib`, `&this->GameName`, `this->Map`, +`&this->IncMod`, `&this->ResMod` … (69 refs). Spine functions after the merge: `BeginProcessTurn`: +`this->Frame = this->Frame + 1; this->bProcessingTurn = true; log("Begin processing turn %d", this->Frame)`; +`OnPlayerEndTurn`: `piVar10 = this->Players._Myfirst … if (this->OnEventCallback == NULL) … (*this->OnEventCallback)(id,0x27,&ev)`; +`ServerSystem::ProcessTurn`: `if (this->PID == NULL) this->Infra -= …; if (this->indi) iVar4 = this->indi->indsp; this->ntdev = 0`. +Every save tag in the decompile lines up with the field of the same name — the offset tables in struct-recovery.md are +confirmed by the decompiler itself. + +--- + +## 2. Turn / main-loop spine + +### 2.1 Entry → init → main loop +``` +entry 0x00925794 → ___tmainCRTStartup 0x00925501 → WinMain 0x0089dd30 (hInst=0x400000, cmdline, nShow) + ├ single-instance mutex "Kerberos_SwordOfTheStars_Mutex" (bypass: /concurrent); FindWindow+SetForegroundWindow otherwise + ├ new Game::DemoApp (0x1b8) 0x0089c950 → Mars::Application::Application 0x008a0170 (g_pApplication = this) + ├ Mars::Application::Initialize 0x008a0e50 ("AppStartup_ReadConfig") + │ CPU/ForceSingleCore > 0 → Process_PinAffinity 0x0089ee70 (SetProcessAffinityMask 1<vft+0x60() (UI/input update) + app->vft+0x18 = DemoApp::OnUpdate 0x00898800 (Mars network pump FUN_00902ad0; current game object(+0x158)->Update; strategy game (+0x150) → FUN_007879c0) + if (!focused && app->vft+0xc ShouldSleepWhenInactive) Sleep(20) + _controlfp(...); every _DAT_00a36860 s → Mars::TimerList::Dispatch 0x008e5ac0 + ok = app->vft+0x1c = DemoApp::OnTick 0x0089a640 (sound/console/panels + top-level state machine: loads/unloads strategy game @+0x150 and combat @+0x158; 0 when quitting) + if (ok && DrawDevice) { dev->Begin; if (!dev->IsLost) { dev->+8; app->vft+0x20 = DemoApp::OnRender 0x00899210 (PanelManager->vft+0x64 draw / movie player); dev->+0xc; dev->Present } } + } while (Mars::Application::PumpMessages 0x0089f1c0 (PeekMessage/Translate/Dispatch, 0 on WM_QUIT) && ok); +``` +The strategy game is created from the lobby via `DemoApp::CreateStrategyGame` 0x00898b00 → `StrategyApp::CreateGame` +0x00888e80 (GameOptions/AIProcessMinTime, DefaultAutoRefuel, starcolors.txt, `TurnCommands_v5`) → `StrategyServer` +ctor 0x007d78d0 (+0x170 OnEventCallback from a ctor argument), `StrategyServer::InitGame` 0x007c8d90 (SEInitGame, +SEAddDesign, SynchronizePlayer), `StrategyServer::LoadGame` 0x007dd530 ("loaded from file"), star-map UI +(FUN_00778f40 → … → `StarMapPanelBase` ctor 0x00741cd0 ← THREAD 3). + +### 2.2 Event plumbing (how `SE*` reach the client) +`Game::SE*` are `Game::StrategyEvent` subclasses with a single virtual (dtor) — there is **no dynamic_cast dispatch**; +the server hands each event to a callback with an **integer event type**: `StrategyServer+0x170 OnEventCallback +(playerNetId, int type, StrategyEvent** ev)`; `StrategyServer::BroadcastEvent` 0x00789710 loops over `Players`. +Missing callback → "StrategyServer: OnEvent() called, but no callback function specified." (31 sites). Types seen: +`0x21 SETurnEndPending` (client side, `StrategyClient::RaiseEvent` 0x00783ee0), `0x24 SEProcessTurn`, `0x26 +SEResumePlaying`, `0x27 SELastPlaying`. The client counterpart is the `StrategyClient` (vftable 0x00a2298c). + +`SNM*` are network messages registered by static initialisers (`.text` 0x009be0cc–0x009c138c, code Ghidra had not +disassembled) via `Mars::NetMessageRegistry::Register` 0x008d2290 → `g_NetMsgRegistryById[id]` 0x00b2ddf8. SNM entry +address = 0x00b2bc94 + 12·id (`{name, id, factory}`), so a handler comparing `msg->GetType()->id` against +`DAT_00b2xxxx` is comparing against `entry+4`. Ids that matter for the turn: + +| id | message | id | message | +|---|---|---|---| +| 0x29 | SNMUpdate (host → all: every player's TurnCommands) | 0x3f | SNMEndTurn (client → host) | +| 0x2a/0x2b | SNMHostCombat / Reply | 0x40 | SNMUpdateComplete | +| 0x2c/0x2d/0x2e | SNMLaunchCombat / JoinCombat / JoinCombatReply | 0x43/0x44 | SNMResumePlaying / …Received | +| 0x2f | SNMAllCombatDone | 0x3d / 0x3e | SNMRunAI / SNMKillAI | +| 0x30 / 0x31 | SNMEncounterQueryResults / SNMEncounterResults | 0x53 | SNMTurnInfo | +| 0x32 | SNMSetPlayerStatus | 0x5b | SNMQueryEndTurnDone | +| 0x3c | SNMDoEncounterQuery | 0x52 | SNMSetTimers | + +(full table of 82 ids: `NetMsgReg_*` labels in Ghidra; ids 0x00–0x0f are the Mars transport `TNM*`/`FNM*`/`NM*`.) + +### 2.3 The turn, end to end (addresses = renamed functions) +``` +[strategic round] UI End Turn (FUN_005e4f80 / FUN_00579310) + → Game::StrategyClient::EndTurn 0x00783be0 (marks +0x57/+0x132, QPC timestamp, EndTurnDelay 0x0076ab90) + → RaiseEvent(0x21, SETurnEndPending) 0x00783ee0 [SETurnEndPending: "Done", cancellable → CancelEndTurn 0x007856f0 → SETurnEndCancelled] + → SendEndTurn 0x00783980 → SNMEndTurn (factory 0x0088c7b0; TurnCommands @+4, AIEncounterFlags @+0x1b8) + StrategyClient::Update 0x007842b0: strategy timer → SETurnTimeExpired / EndTurnForced 0x00783d30 / SNMQueryEndTurnDone + +[host] Game::StrategyNetworkClient::OnMessage 0x00784640 case 0x3f SNMEndTurn + → StrategyServer::OnPlayerEndTurn 0x007d9af0 (if exactly one human still not done → SELastPlaying 0x27 to him) + → StrategyServer::StorePlayerTurnCommands 0x007893c0 + host StrategyNetworkServer::Update 0x007cda40 → when all done broadcasts SNMUpdate (0x29) carrying ALL TurnCommands + +[every machine — lock-step sim] OnMessage case 0x29 SNMUpdate ("Processing turn update, but the local player is not done" guard) + → StrategyServer::BeginProcessTurn 0x007d98e0 Frame++, bProcessingTurn=1, "Begin processing turn %d", reset fleet transients, BroadcastEvent(0x24 SEProcessTurn) + → StrategyServer::ApplyTurnCommands 0x007b18b0 "set for turn processing": fleet orders, builds, research, alliances (EVENT_ALLIANCE_*) + → FUN_007ad0f0, FUN_0078f6a0, FUN_0081ff40 (small, unnamed) + → StrategyServer::ProcessTurn 0x007dc6c0 (dt = 1.0f) ← §2.4 + → FUN_008cfe20(server->GameID), FUN_007cf540; if local clients: StrategyApp::SyncLocalClients(1) 0x00815fd0 → StrategyServer::SynchronizePlayer 0x007c6220 per local client + → client state (+0x58) = 5; SNMUpdateComplete (0x40) to host + +[querying / combat round — host] StrategyNetworkServer::RunCombatRound 0x007cbe80 + → per encounter: SendEncounterQuery 0x007bfe60 → SNMDoEncounterQuery (0x3c) → clients: state 4 (C1 query UI, FUN_00847320) + → SNMEncounterQueryResults (0x30) → "Notifying %s to host encounter %d" SNMHostCombat (0x2a) / Reply (0x2b) / SNMLaunchCombat (0x2c) / SNMJoinCombat (0x2d) + (client side: StrategyClient::Update logs "All(%d) clients connected to combat server", "Launching combat for encounter %d"; combat itself = DemoApp+0x158 object driven from DemoApp::OnTick) + → SNMEncounterResults (0x31) … → all resolved: SNMAllCombatDone (0x2f) "All combat complete, waiting for clients to process results" + +[every machine] OnMessage case 0x2f SNMAllCombatDone + → FUN_00789330, FUN_007c0600 (1089 B, post-combat bookkeeping), StrategyServer::ApplyEncounterResults 0x007d4400 (EVENT_STATION_ENABLED/DISABLED …) + → DispatchTurnResults 0x007cd2a0 → SendTurnResultsToPlayers 0x007c5850 → per player SETurnResults::Create 0x007a7ae0 + fill 0x007c24d0 + dispatch 0x0079ac10 [SETurnResults] + → FUN_00761310, StrategyApp::SyncLocalClients(0) + → StrategyServer::GenerateTurnEvents 0x007dc640 → BuildTurnEvents 0x007db780 (EventStorage::TurnEvents per player, SynchronizePlayer) [SETurnEvents] + → autosave "(Autosave EndTurn)" (FUN_00895210 / "Auto saving of game failed."), FUN_0076c610, FUN_00769340, FUN_00781cf0 + → state = 6, "StrategyNetworkClient: All combat complete msg received. New turn begins..." + +[resume] host StrategyNetworkServer::SendResumePlaying 0x00794770 → SNMResumePlaying (0x43) + → OnMessage case 0x43 → StrategyServer::ResumePlaying 0x007ddc90: every player with Status(+0x164)==0 gets SEResumePlaying (0x26); reply SNMResumePlayingReceived (0x44); state = 1 (playing) + → (SELastPlaying / SEResumePlaying / SETurnEvents drive the UI_LAST_PLAYING "new turn" sound and the S16 events list) + +[AI] SNMRunAI (0x3d) → StrategyApp::RunAI 0x008706f0 ("RunAI: No StrategyServer created") → RaiseAIPrepareTurn 0x00815f20 → SEAIPrepareTurn; AI players' TurnCommands enter the same SNMEndTurn/SNMUpdate path. +``` +Client state machine (`StrategyNetworkClient+0x58`): 1 playing → (End Turn) → 5 turn processed / waiting for combat → +4 querying (per encounter) → 6 all-combat-done → 1. + +### 2.4 `Game::StrategyServer::ProcessTurn` 0x007dc6c0 — phases (decompiled with the new types, `ghidra-verify/verify_ProcessTurn.c`) +1. **Per-system pre-pass** over `Systems`: colony lookup (FUN_007437e0), morale-event push (FUN_00752a10), string-table text (FUN_00743530/FUN_008c97f0), `SESystemAbandoned` via FUN_007b9df0. +2. FUN_0086b300 + FUN_007adc80 (alliance/diplomacy upkeep; no strings). +3. Per-player pre-pass over `Players` (small). +4. **Movement**: fleet snapshot (FUN_00794ad0 / FUN_007b9b90), `ProcessNodeSpaceTravel` 0x007a0e20 (EVENT_FLEET_MULTIPOINT_NONODE, EVENT_LOSTINNODESPACE_NOBORE/ENGINES), **`ProcessFleetMovement` 0x007da9a0 → `MoveFleet` 0x007d9ee0 per fleet ("Destination of fleet doesn't exist. Stopping fleet.", cancels ship actions) → `SEFleetArrived`; `OnFleetArrived` 0x007ccb10 (EVENT_FLEET_ARRIVED)**. +5. Per-fleet → per-ship FUN_00814ea0 (ship upkeep). +6. **Per-system `ServerSystem::ProcessTurn` 0x007598e0**: infra decay if unowned, independence, pop/morale growth, `ProcessPlague` 0x00756a90 (EVENT_PLAGUE_OUTBREAK/CURED, COLONY_DESTROYEDBYPLAGUE), **`ProcessBuildQueue` 0x00752500 → `BuildQueue::ProcessTurn` 0x00890d50 → `SEBuildCompleted`** (with ShipBuildOrderDef), resources/terraforming (FUN_0074b230/FUN_00754220), `ProcessSlaves` 0x007537b0 (EVENT_SLAVES_DEAD), `ProcessRebellion` 0x007583b0 (EVENT_SYSTEM_REBELLION_CONTINUES). +7. FUN_0078a7c0. +8. **Per-player `ServerPlayer::ProcessTurn` 0x00891340 (dt)**: income/savings (FUN_00840fe0), **research**: if `ResT` set → `RollResearchAccident` 0x00889dc0 ("ACCIDENT!!"/"All okay.", EVENT_LABACCIDENT_*) then `TechTree::ProcessResearch` 0x005876c0 (EVENT_RESEARCH_OVERBUDGET, EVENT_TECHS_UNLOCKED), else EVENT_NO_RESEARCH; special projects FUN_00863cf0. +9. `ProcessMissions` 0x007999a0, `ProcessStations` 0x007ae480 (EVENT_STATIONS_SCUTTLED), `ProcessDefenceSats` 0x007af0b0 (EVENT_DEFSATS_SCUTTLED). +10. Per-ship flag pass: flag 4 → FUN_0080caf0(1); flag 0x400000 → FUN_00815230. +11. **Encounter detection**: second snapshot FUN_00794ad0 → `local_98` = pending encounters. **If none**: end-of-turn tail runs now — `ProcessAid` 0x007ad100 (EVENT_GIVE_SAVINGS/RESEARCH), `ProcessSpecialProjects` 0x007a3310 (EVENT_SPRJTECHOFFER_STARTED), `ProcessSurrenders` 0x007d0d10 (EVENT_PLAYER_SURRENDERED_, EVENT_SYSTEM_SURRENDERED), per-player FUN_00818530(0), script hooks `SvSctOb->vft+0x10(6)` / `(0x1c)`, FUN_0086a8d0, FUN_0078ab30, FUN_00799380, FUN_0078aa70, per-system FUN_00743ec0, FUN_007b4c00, FUN_007d7f70 (2.4 KB, per-team 0x74-byte records), and every player with `+0xf9==0 || +0xfa!=0` gets `Status(+0x164) = 1` (back to playing). **If encounters exist** the tail is deferred to the `SNMAllCombatDone` handler (§2.3), i.e. the turn's results/events are only produced after the combat round. + +--- + +## 3. Threads (the three `CreateThread` sites) + +| # | site | creator | thread proc | what it does | +|---|---|---|---|---| +| 1 | 0x0090242d | `Mars::NetworkManager::Create` 0x00902350 ← `Mars::Network::Startup` 0x00902470 ← `DemoApp::OnStartup` | `Mars::NetworkManager::WatchdogThreadProc` 0x00901f40 | `Sleep(10)` loop under `g_netCS` 0x00b2e5f8: pings host link (+0x20) and listener (+0x98), detects timeouts — "Network(%f): No response from host %s." / "...from client %s." — and drops the link. Pure liveness watchdog; never touches the sim. Globals `g_pNetworkManager` 0x00b2e61c, `g_hNetworkWatchdogThread` 0x00b2e620. | +| 2 | 0x008a14ef | `Mars::Application::Initialize` 0x008a0e50 ("Initializing Streaming sound update thread...") | `Mars::SoundSystem::StreamingUpdateThreadProc` 0x008ef1d0, **SetThreadPriority 15 = TIME_CRITICAL** | `WaitForMultipleObjects({app+0x84 wake, app+0x88 quit})`; under `g_musicCS` 0x00b2e4a8: if the current music stream (`g_pCurrentMusicStream` 0x00b2e4c4) has ended/faded (float +0x388 ≤ 0) → stop it and open the next queued track (`MusicPlayer::OpenMusicFile` 0x008ef040, "[%s] cannot open music file for playback"), else `StreamingSound::FillBuffer` 0x0091b5a0 (DirectSound GetCurrentPosition/Lock/decode; "Couldn't restore buffer"); then FUN_008b65d0 (sound-system update). Music/DirectSound streaming only — but it is the **only elevated-priority thread** in the process. | +| 3 | 0x00736e84 | `Game::BackgroundWorker::Start` 0x00736e30, first statement of `Game::StarMapPanelBase` ctor 0x00741cd0 (Render/StarMapBlobs_Solid.fx, StarMapBlobs_Glow.fx, POLMAP_* colours, Skysphere) ← `PoliticalMapPanel` ctor 0x007424b0 ← strategy-map screen ctor FUN_005e9780 ← … ← `StrategyApp::CreateGame` | `Game::BackgroundWorker::ThreadProc` 0x00735bb0 | `Sleep(10)` poll loop; struct `{CRITICAL_SECTION@0; HANDLE thread@0x18; job*@0x54; flags@0x58..0x5c (0x5b quit, 0x5c exited)}`; when a job is posted runs `StarMapBlobs::BuildBlobMesh_Job` 0x00732ab0 = political-map territory "blob" mesh (point cloud → centroid/spread → implicit-surface polygoniser FUN_008fc160 with callbacks FUN_00722010/FUN_0071ea60 → mesh FUN_008fa5b0), then sets done. **Strategic-map overlay only — not the battle loader** (prior recon's "prime suspect" is cleared). | + +Consequences for the battle-load / many-core lead: the combat load and the whole sim run on the **main thread** (DemoApp::OnTick +state machine → combat object at `DemoApp+0x158`, `OnUpdate` → `+0x158->vft+4`). The only application threads are two 10 ms +pollers and a TIME_CRITICAL audio streamer; everything else multi-threaded is inside D3D9/DirectSound/driver. If the +slowdown is contention, the candidates inside the game's own code are the `g_musicCS` critical section (held by the +TIME_CRITICAL thread while decoding) and the `Sleep(10)` pollers — not a sim worker. + +--- + +## 4. Caveats / open items +- Names are inferred from strings/callers (no symbols). `StrategyNetworkClient::OnMessage` (0x00784640) is the combined + host+client message handler (host branches test `+0x54 = StrategyServer*`); the "StrategyNetworkServer" functions live + on the host object driven from `DemoApp::OnUpdate`. +- `StrategyServer` struct is partial (41 fields); `ServerPlayer::Read/Write` show only the +0x3a0.. tail by name. +- Not named: FUN_007ad0f0/FUN_0078f6a0/FUN_0081ff40/FUN_007cf540 (between ApplyTurnCommands and ProcessTurn), + FUN_00789330/FUN_007c0600 (AllCombatDone pre-pass), FUN_0086b300/FUN_007adc80/FUN_0078a7c0/FUN_007d7f70 (turn phases + without strings), the client-side event sink behind `StrategyClient::RaiseEvent`, and the `Mars::Stream` vftable. +- The static-initialiser region 0x009be000–0x009c1400 is still undisassembled code in the project (only labels were + added); running "Disassemble" there would expose all 82 registrations as functions. +- reva-server was stopped for the headless runs and restarted at the end. No git commit was made. diff --git a/ghidra/.keep.md b/ghidra/.keep.md deleted file mode 100644 index 56c0d60..0000000 --- a/ghidra/.keep.md +++ /dev/null @@ -1 +0,0 @@ -# ghidra/ — placeholder, populate as work lands diff --git a/ghidra/scripts/FixPlayer.java b/ghidra/scripts/FixPlayer.java new file mode 100644 index 0000000..8418063 --- /dev/null +++ b/ghidra/scripts/FixPlayer.java @@ -0,0 +1,37 @@ +import ghidra.app.script.GhidraScript; +import ghidra.program.model.address.*; +import ghidra.program.model.listing.*; +import ghidra.program.model.symbol.*; +import ghidra.program.model.data.*; +import ghidra.program.model.lang.*; +import ghidra.app.decompiler.*; +import java.util.*; +import java.io.*; + +// ServerPlayer::Read/Write had no parameters: create this (ECX) = ServerPlayer_ser928*, stream (stack+4); verify decompile. +public class FixPlayer extends GhidraScript { + @Override + public void run() throws Exception { + DataTypeManager dtm = currentProgram.getDataTypeManager(); + List l = new ArrayList(); dtm.findDataTypes("Game::ServerPlayer_ser928", l); + DataType view = l.get(0); + List l2 = new ArrayList(); dtm.findDataTypes("Stream", l2); DataType stream = null; for (DataType d : l2) if (d.getCategoryPath().getPath().equals("/Mars")) stream = d; + Register ecx = currentProgram.getRegister("ECX"); + PrintWriter log = new PrintWriter(new FileWriter("/tmp/spine/fixplayer.log")); + for (long a : new long[]{0x008563e0L, 0x008804d0L}) { + Function f = getFunctionAt(toAddr(a)); + List ps = new ArrayList(); + ps.add(new ParameterImpl("this", new PointerDataType(view), ecx, currentProgram)); + ps.add(new ParameterImpl("stream", new PointerDataType(stream == null ? VoidDataType.dataType : stream), 4, currentProgram)); + f.setCustomVariableStorage(true); + f.replaceParameters(ps, Function.FunctionUpdateType.CUSTOM_STORAGE, true, SourceType.USER_DEFINED); + log.println(f.getName(true) + " params=" + f.getParameterCount() + " " + f.getPrototypeString(false, false)); + } + DecompInterface decomp = new DecompInterface(); decomp.openProgram(currentProgram); + Function f = getFunctionAt(toAddr(0x008563e0L)); + DecompileResults res = decomp.decompileFunction(f, 240, monitor); + PrintWriter w = new PrintWriter(new FileWriter("/tmp/spine/verify_ServerPlayer_Write.c")); + w.println(res.getDecompiledFunction().getC()); w.close(); + decomp.dispose(); log.close(); println("fixplayer done"); + } +} diff --git a/ghidra/scripts/FixThis.java b/ghidra/scripts/FixThis.java new file mode 100644 index 0000000..0981630 --- /dev/null +++ b/ghidra/scripts/FixThis.java @@ -0,0 +1,95 @@ +import ghidra.app.script.GhidraScript; +import ghidra.program.model.address.*; +import ghidra.program.model.mem.*; +import ghidra.program.model.listing.*; +import ghidra.program.model.symbol.*; +import ghidra.program.model.data.*; +import ghidra.app.decompiler.*; +import java.util.*; +import java.util.regex.*; +import java.io.*; + +// Fix-up: make the class structs Ghidra uses for __thiscall 'this' (VariableUtilities.findOrCreateClassStruct) BE the recovered +// structs; give the COL!=0 serializers custom storage with the shifted view; re-run verification decompiles. +public class FixThis extends GhidraScript { + DataTypeManager dtm; SymbolTable st; PrintWriter log; DecompInterface decomp; Memory mem; + static final String[][] CLASSES = { // {namespace path, /SOTS type name} + {"Game::ServerSystem","Game::ServerSystem"},{"Game::ServerPlayer","Game::ServerPlayer"},{"Game::StarFleet","Game::StarFleet"},{"Game::StarShip","Game::StarShip"},{"Game::StrategyServer","Game::StrategyServer"}, + {"Game::Population","Game::Population"},{"Game::PopulationGroup","Game::PopulationGroup"},{"Game::Morale","Game::Morale"},{"Game::MoraleEvent","Game::MoraleEvent"},{"Game::IndependenceInfo","Game::IndependenceInfo"}, + {"Game::StarSystem::OutputRates","Game::StarSystem::OutputRates"},{"Game::StarSystem::PlayerView","Game::StarSystem::PlayerView"},{"Game::ShipBuildOrder","Game::ShipBuildOrder"},{"Game::DiplomacyStats","Game::DiplomacyStats"}, + {"Game::PlayerReport","Game::PlayerReport"},{"Game::PlayerAlliances","Game::PlayerAlliances"},{"Game::NodeRoute","Game::NodeRoute"},{"Game::FlightPlan::Waypoint","Game::Waypoint"},{"Game::FlightPlan","Game::FlightPlan"}, + {"Game::ShipHealth","Game::ShipHealth"},{"Game::PrisonerHold","Game::PrisonerHold"},{"Game::EventStorage","Game::EventStorage"},{"Game::CivilianRatios","Game::CivilianRatios"},{"Game::ShipRecords","Game::ShipRecords"}, + {"Game::SpyReport","Game::SpyReport"},{"Game::PlayerColorID","Game::PlayerColorID"},{"Mars::Vector3","Mars::Vector3"}, + {"Game::BuildQueue","Game::BuildQueue"},{"Game::TechTree","Game::TechTree"},{"Mars::Stream","Mars::Stream"},{"Game::StrategyEvent","Game::StrategyEvent"}}; + // serializers whose this = object + COL offset : {addr, class type name, col} + static final Object[][] SHIFTED = {{0x00749630L,"Game::ServerSystem",8},{0x0075d4b0L,"Game::ServerSystem",8},{0x00701070L,"Game::StarFleet",8},{0x00702470L,"Game::StarFleet",8}, + {0x008291f0L,"Game::StarShip",8},{0x00853fa0L,"Game::StarShip",8},{0x008563e0L,"Game::ServerPlayer",0x3a0},{0x008804d0L,"Game::ServerPlayer",0x3a0}}; + + DataType findSots(String name) { List l = new ArrayList(); dtm.findDataTypes(name, l); for (DataType d : l) if (d.getCategoryPath().getPath().equals("/SOTS")) return d; return l.isEmpty() ? null : l.get(0); } + String cstr(Address a, int max) { + try { byte[] b = new byte[max]; int got = mem.getBytes(a, b); int i = 0; + while (i < got && b[i] != 0 && (b[i]&0xff) >= 0x20 && (b[i]&0xff) < 0x7f) i++; + if (i >= 1 && i < got && b[i] == 0) return new String(b, 0, i, "ISO-8859-1"); } catch (Exception e) {} + return null; + } + void verify(long addr, String outName) throws Exception { + Function f = getFunctionAt(toAddr(addr)); + DecompileResults res = decomp.decompileFunction(f, 240, monitor); + PrintWriter w = new PrintWriter(new FileWriter("/tmp/spine/" + outName)); + if (res == null || !res.decompileCompleted()) { w.println("[decompile failed]"); w.close(); return; } + String c = res.getDecompiledFunction().getC(); + Matcher m = Pattern.compile("&?(DAT|PTR_s_|s_[A-Za-z0-9_]*|PTR_DAT|u_[A-Za-z0-9_]*)_([0-9a-f]{8})").matcher(c); + StringBuffer sb = new StringBuffer(); + while (m.find()) { Address a = toAddr(Long.parseLong(m.group(2), 16)); String s = cstr(a, 64); String rep = m.group(0); if (s != null && s.length() <= 60) rep = "\"" + s + "\""; m.appendReplacement(sb, Matcher.quoteReplacement(rep)); } + m.appendTail(sb); w.println(sb); w.close(); log.println(" verify " + outName + " this-> refs: " + (sb.toString().split("this->").length - 1)); + } + + @Override + public void run() throws Exception { + dtm = currentProgram.getDataTypeManager(); st = currentProgram.getSymbolTable(); mem = currentProgram.getMemory(); + log = new PrintWriter(new FileWriter("/tmp/spine/fixthis.log")); + Map cls = new HashMap(); + for (String[] c : CLASSES) { + String path = c[0], tname = c[1]; + Namespace cur = currentProgram.getGlobalNamespace(); + for (String part : path.split("::")) { Namespace n = st.getNamespace(part, cur); if (n == null) n = st.createClass(cur, part, SourceType.USER_DEFINED); cur = n; } + if (!(cur instanceof GhidraClass)) { cur = st.convertNamespaceToClass(cur); } + Structure cs = VariableUtilities.findOrCreateClassStruct((GhidraClass) cur, dtm); + DataType mine = findSots(tname); + log.println("class " + path + " -> class struct " + cs.getPathName() + " (len " + cs.getLength() + ") mine=" + (mine == null ? "null" : mine.getPathName() + " len " + mine.getLength())); + if (mine == null || mine == cs || mine.isEquivalent(cs) && mine.getPathName().equals(cs.getPathName())) { cls.put(tname, cs); continue; } + if (mine instanceof Structure) { + cs.replaceWith(mine); cs.setDescription(mine.getDescription()); + dtm.replaceDataType(mine, cs, false); // repoint every use of the /SOTS copy to the class struct and drop the copy + log.println(" merged; class struct now len " + cs.getLength() + " with " + cs.getNumDefinedComponents() + " fields"); + } + cls.put(tname, cs); + } + // shifted serializer views (rebuild from the class structs) + Map views = new HashMap(); + for (Object[] s : SHIFTED) { + String tname = (String) s[1]; int col = ((Number) s[2]).intValue(); String key = tname + col; + Structure v = views.get(key); + if (v == null) { + Structure src = cls.get(tname); + StructureDataType nv = new StructureDataType(new CategoryPath("/SOTS"), tname + "_ser" + col, src.getLength() - col, dtm); + nv.setDescription("Serializer view of " + tname + ": this = object + 0x" + Integer.toHexString(col) + " (IStreamable sub-object); only for " + tname + "::Read/Write. Fields below the sub-object offset are NOT visible here."); + for (DataTypeComponent c : src.getDefinedComponents()) { if (c.getOffset() < col) continue; try { nv.replaceAtOffset(c.getOffset() - col, c.getDataType(), c.getLength(), c.getFieldName(), c.getComment()); } catch (Exception e) {} } + v = (Structure) dtm.addDataType(nv, DataTypeConflictHandler.REPLACE_HANDLER); views.put(key, v); + } + Function f = getFunctionAt(toAddr((Long) s[0])); + try { + f.setCustomVariableStorage(true); + Parameter p = f.getParameter(0); + p.setDataType(new PointerDataType(v), SourceType.USER_DEFINED); + if (f.getParameterCount() > 1) f.getParameter(1).setName("stream", SourceType.USER_DEFINED); + f.setComment((f.getComment() == null ? "" : f.getComment() + "\n") + "NOTE: this = object + 0x" + Integer.toHexString(col) + " (IStreamable sub-object); typed as " + v.getName() + " (custom storage)."); + log.println(" " + f.getName(true) + ": this -> " + v.getName() + "* (custom storage) param0=" + p.getName() + " storage=" + p.getVariableStorage()); + } catch (Exception e) { log.println(" !! " + f.getName(true) + ": " + e.getMessage()); } + } + decomp = new DecompInterface(); decomp.openProgram(currentProgram); + verify(0x00749630L, "verify_ServerSystem_Write.c"); verify(0x0075d4b0L, "verify_ServerSystem_Read.c"); verify(0x0079fa70L, "verify_StrategyServer_Write.c"); verify(0x00701070L, "verify_StarFleet_Write.c"); + verify(0x008563e0L, "verify_ServerPlayer_Write.c"); verify(0x007d98e0L, "verify_BeginProcessTurn.c"); verify(0x007dc6c0L, "verify_ProcessTurn.c"); verify(0x007598e0L, "verify_ServerSystem_ProcessTurn.c"); verify(0x007d9af0L, "verify_OnPlayerEndTurn.c"); + decomp.dispose(); log.close(); println("fixthis done"); + } +} diff --git a/ghidra/scripts/SpineRecon.java b/ghidra/scripts/SpineRecon.java new file mode 100644 index 0000000..4d0ae7e --- /dev/null +++ b/ghidra/scripts/SpineRecon.java @@ -0,0 +1,190 @@ +import ghidra.app.script.GhidraScript; +import ghidra.program.model.address.*; +import ghidra.program.model.mem.*; +import ghidra.program.model.listing.*; +import ghidra.program.model.symbol.*; +import ghidra.app.decompiler.*; +import java.util.*; +import java.util.regex.*; +import java.io.*; + +// Pass 1 (read-only): anchors for the turn/main-loop spine. +// A. strings containing SE*/SNM*/AppStartup/etc. -> xrefs -> containing functions (+ their callers) +// B. symbols (RTTI/vftables) containing those names +// C. entry point + callees (WinMain hunt), CreateThread sites + thread procs +// D. std::string layout check (FUN_008b9d70) +public class SpineRecon extends GhidraScript { + DecompInterface decomp; Memory mem; PrintWriter out; ReferenceManager rm; SymbolTable st; + static final String[] NEEDLES = { + "SETurnEndPending","SNMEndTurn","SEProcessTurn","SNMDoEncounterQuery","SNMAllCombatDone","SNMResumePlaying", + "SEAIPrepareTurn","SNMRunAI","SEFleetArrived","SEBuildCompleted","SETurnResults","SETurnEvents","AppStartup", + "SETurnTimeExpired","SEResumePlaying","SNMEncounterQueryResults","SNMHostCombat","SNMLaunchCombat","SNMEncounterResults", + "SETurnEndCancelled","SNMQueryEndTurnDone","SELastPlaying","PROCESSING_TITLE","Initializing Game","LOBBYSTATUS_STRATEGY_ROUND", + "SENewTurn","SEStartTurn","SEEndTurn","StrategyServer","TurnCommands","ProcessTurn","EndTurn","MainLoop","Tick","AppMain" + }; + static final String[] DECOMP = {"00902350","00736e30","008a0e50","008b9d70","008b9d90"}; + + String cstr(Address a, int max) { + try { + byte[] b = new byte[max]; int got = mem.getBytes(a, b); + int i = 0; while (i < got && b[i] != 0 && (b[i]&0xff) >= 0x20 && (b[i]&0xff) < 0x7f) i++; + if (i >= 1 && i < got && b[i] == 0) return new String(b, 0, i, "ISO-8859-1"); + } catch (Exception e) {} + return null; + } + String decompRes(Function f) { + try { + DecompileResults res = decomp.decompileFunction(f, 180, monitor); + if (res == null || !res.decompileCompleted()) return "[decompile failed]"; + String c = res.getDecompiledFunction().getC(); + Matcher m = Pattern.compile("&?(DAT|PTR_s_|s_[A-Za-z0-9_]*|PTR_DAT|u_[A-Za-z0-9_]*)_([0-9a-f]{8})").matcher(c); + StringBuffer sb = new StringBuffer(); + while (m.find()) { + Address a = toAddr(Long.parseLong(m.group(2), 16)); + String s = cstr(a, 64); String rep = m.group(0); + if (s != null && s.length() <= 60) rep = "\"" + s + "\""; + m.appendReplacement(sb, Matcher.quoteReplacement(rep)); + } + m.appendTail(sb); return sb.toString(); + } catch (Exception e) { return "[exception " + e.getMessage() + "]"; } + } + void dumpFunc(Function f, String tag) throws IOException { + String n = String.format("%08x", f.getEntryPoint().getOffset()); + File fl = new File("/tmp/spine/" + n + ".c"); if (fl.exists()) return; + PrintWriter w = new PrintWriter(new FileWriter(fl)); + w.println("// " + f.getName() + " @ " + f.getEntryPoint() + " size=" + f.getBody().getNumAddresses() + " tag=" + tag); + StringBuilder cs = new StringBuilder(); + for (Function c : f.getCallingFunctions(monitor)) cs.append(c.getName() + "@" + c.getEntryPoint() + " "); + w.println("// CALLERS: " + cs); + StringBuilder ce = new StringBuilder(); + for (Function c : f.getCalledFunctions(monitor)) ce.append(c.getName() + " "); + w.println("// CALLEES: " + ce); + w.println(decompRes(f)); w.close(); + } + String funcDesc(Function f) { + StringBuilder cs = new StringBuilder(); + int n = 0; + for (Function c : f.getCallingFunctions(monitor)) { if (n++ < 8) cs.append(c.getName() + " "); } + return f.getName() + "@" + f.getEntryPoint() + " sz=" + f.getBody().getNumAddresses() + " callers(" + n + "):[" + cs.toString().trim() + "]"; + } + + @Override + public void run() throws Exception { + mem = currentProgram.getMemory(); rm = currentProgram.getReferenceManager(); st = currentProgram.getSymbolTable(); + decomp = new DecompInterface(); decomp.openProgram(currentProgram); + new File("/tmp/spine").mkdirs(); + out = new PrintWriter(new FileWriter("/tmp/spine/recon.txt")); + Set toDump = new LinkedHashSet(); + + // A. strings + out.println("==== A. STRING ANCHORS ===="); + for (MemoryBlock blk : mem.getBlocks()) { + if (!blk.isInitialized() || blk.isExecute()) continue; + if (!blk.getName().equals(".rdata") && !blk.getName().equals(".data")) continue; + int len = (int) blk.getSize(); byte[] buf = new byte[len]; blk.getBytes(blk.getStart(), buf); + int start = 0; + for (int i = 0; i < len; i++) { + if (buf[i] != 0) continue; + int slen = i - start; + if (slen >= 4 && slen <= 120) { + boolean ascii = true; + for (int k = start; k < i; k++) { int c = buf[k] & 0xff; if (c < 0x20 || c > 0x7e) { ascii = false; break; } } + if (ascii) { + String s = new String(buf, start, slen, "ISO-8859-1"); + for (String nd : NEEDLES) { + if (!s.contains(nd)) continue; + Address a = blk.getStart().add(start); + StringBuilder sb = new StringBuilder(); + ReferenceIterator ri = rm.getReferencesTo(a); int n = 0; + while (ri.hasNext()) { + Reference r = ri.next(); n++; + Function f = getFunctionContaining(r.getFromAddress()); + if (f == null) { sb.append("\n ?" + r.getFromAddress()); continue; } + sb.append("\n " + r.getFromAddress() + " in " + funcDesc(f)); + if (nd.startsWith("SE") || nd.startsWith("SNM") || nd.equals("AppStartup") || nd.equals("PROCESSING_TITLE") || nd.equals("Initializing Game")) toDump.add(f); + } + out.println(a + " \"" + s + "\" [" + nd + "] refs=" + n + sb); + break; + } + } + } + start = i + 1; + } + } + // B. symbols + out.println("\n==== B. SYMBOLS ===="); + SymbolIterator si = st.getAllSymbols(true); + while (si.hasNext()) { + Symbol s = si.next(); String n = s.getName(true); + boolean hit = false; + for (String nd : NEEDLES) if (n.contains(nd)) { hit = true; break; } + if (!hit) continue; + if (n.contains("StreamableHelper") || n.contains("VectorHelper")) continue; + out.println(s.getAddress() + " " + n + " [" + s.getSymbolType() + "]"); + if (n.endsWith("::vftable")) { + Address a = s.getAddress(); + for (int i = 0; i < 12; i++) { + try { + long p = mem.getInt(a.add(i*4)) & 0xffffffffL; Address fa = toAddr(p); + Function f = getFunctionAt(fa); + if (f == null) { if (i > 0) break; out.println(" [" + i + "] " + fa + " (nofunc)"); continue; } + out.println(" [" + i + "] " + funcDesc(f)); + if (f.getBody().getNumAddresses() > 40) toDump.add(f); + } catch (Exception e) { break; } + } + } + } + // C. entry + CreateThread + out.println("\n==== C. ENTRY / THREADS ===="); + for (Symbol s : st.getSymbols("entry")) { + Function f = getFunctionAt(s.getAddress()); + out.println("entry sym @ " + s.getAddress() + " func=" + (f == null ? "null" : funcDesc(f))); + if (f != null) { + toDump.add(f); + for (Function c : f.getCalledFunctions(monitor)) { + out.println(" entry callee: " + funcDesc(c) + " params=" + c.getParameterCount()); + if (c.getBody().getNumAddresses() > 200 && !c.getName().startsWith("_")) toDump.add(c); + } + } + } + for (Symbol s : st.getSymbols("CreateThread")) { + out.println("CreateThread sym @ " + s.getAddress() + " type=" + s.getSymbolType()); + ReferenceIterator ri = rm.getReferencesTo(s.getAddress()); + while (ri.hasNext()) { + Reference r = ri.next(); Function f = getFunctionContaining(r.getFromAddress()); + out.println(" ref " + r.getFromAddress() + " " + r.getReferenceType() + " in " + (f == null ? "?" : funcDesc(f))); + if (f != null) toDump.add(f); + // thunk? follow refs to the thunk + if (f != null && f.isThunk() || (f != null && f.getBody().getNumAddresses() < 12)) { + ReferenceIterator ri2 = rm.getReferencesTo(f.getEntryPoint()); + while (ri2.hasNext()) { Reference r2 = ri2.next(); Function f2 = getFunctionContaining(r2.getFromAddress()); + out.println(" thunk-ref " + r2.getFromAddress() + " in " + (f2 == null ? "?" : funcDesc(f2))); if (f2 != null) toDump.add(f2); } + } + } + } + // thread procs: look at the CreateThread call sites listed in prior recon; find the pushed function pointer + long[] sites = {0x0090242dL, 0x008a14efL, 0x00736e84L}; + for (long sa : sites) { + Address a = toAddr(sa); + out.println("site " + a + ":"); + Instruction ins = getInstructionAt(a); + int back = 0; + Instruction p = ins; + while (p != null && back < 12) { + p = p.getPrevious(); back++; + if (p == null) break; + StringBuilder refs = new StringBuilder(); + for (Reference r : p.getReferencesFrom()) { + Function tf = getFunctionAt(r.getToAddress()); + if (tf != null) { refs.append(" -> FUNC " + funcDesc(tf)); toDump.add(tf); } + } + out.println(" " + p.getAddress() + " " + p + refs); + } + } + for (String d : DECOMP) { Function f = getFunctionAt(toAddr(Long.parseLong(d, 16))); if (f != null) toDump.add(f); } + // D. dump + out.println("\n==== D. DUMPED ===="); + for (Function f : toDump) { out.println(f.getName() + " @ " + f.getEntryPoint()); dumpFunc(f, "spine"); } + out.close(); decomp.dispose(); println("done"); + } +} diff --git a/ghidra/scripts/SpineRecon2.java b/ghidra/scripts/SpineRecon2.java new file mode 100644 index 0000000..5205ede --- /dev/null +++ b/ghidra/scripts/SpineRecon2.java @@ -0,0 +1,132 @@ +import ghidra.app.script.GhidraScript; +import ghidra.program.model.address.*; +import ghidra.program.model.mem.*; +import ghidra.program.model.listing.*; +import ghidra.program.model.symbol.*; +import ghidra.app.decompiler.*; +import java.util.*; +import java.util.regex.*; +import java.io.*; + +// Pass 2 (read-only): event dispatch + main loop + WinMain chain + worker threads. +public class SpineRecon2 extends GhidraScript { + DecompInterface decomp; Memory mem; PrintWriter out; ReferenceManager rm; SymbolTable st; + Set toDump = new LinkedHashSet(); + static final String[] CLASSES = {"SETurnEndPending","SETurnEndCancelled","SETurnTimeExpired","SEProcessTurn","SEAIPrepareTurn","SEFleetArrived", + "SEBuildCompleted","SETurnResults","SETurnEvents","SEResumePlaying","SELastPlaying","SNMEndTurn","SNMQueryEndTurnDone","SNMDoEncounterQuery", + "SNMEncounterQueryResults","SNMHostCombat","SNMLaunchCombat","SNMEncounterResults","SNMAllCombatDone","SNMResumePlaying","SNMResumePlayingReceived", + "SNMRunAI","StrategyEvent","StrategyServer","TurnCommands","AppStartup"}; + static final long[] STRS = {0x00a23c08L,0x00a32698L,0x00a258a8L,0x00a261a8L,0x00a21978L}; + static final String[] APIS = {"PeekMessageA","GetMessageA","DispatchMessageA","TranslateMessage","WinMain","GetStartupInfoA","ShowWindow","QueryPerformanceCounter","timeGetTime"}; + static final long[] DECOMP = {0x00925501L,0x0089dd30L,0x00741cd0L,0x00732ab0L,0x00902470L,0x0091b5a0L,0x008ef040L,0x008a7230L,0x008cd820L,0x008fe730L,0x00761300L,0x00761380L,0x007861d0L,0x00786d50L,0x007825f0L,0x008559e0L,0x0079e500L}; + + String cstr(Address a, int max) { + try { byte[] b = new byte[max]; int got = mem.getBytes(a, b); int i = 0; + while (i < got && b[i] != 0 && (b[i]&0xff) >= 0x20 && (b[i]&0xff) < 0x7f) i++; + if (i >= 1 && i < got && b[i] == 0) return new String(b, 0, i, "ISO-8859-1"); } catch (Exception e) {} + return null; + } + String decompRes(Function f) { + try { DecompileResults res = decomp.decompileFunction(f, 240, monitor); + if (res == null || !res.decompileCompleted()) return "[decompile failed]"; + String c = res.getDecompiledFunction().getC(); + Matcher m = Pattern.compile("&?(DAT|PTR_s_|s_[A-Za-z0-9_]*|PTR_DAT|u_[A-Za-z0-9_]*)_([0-9a-f]{8})").matcher(c); + StringBuffer sb = new StringBuffer(); + while (m.find()) { Address a = toAddr(Long.parseLong(m.group(2), 16)); String s = cstr(a, 64); String rep = m.group(0); + if (s != null && s.length() <= 60) rep = "\"" + s + "\""; m.appendReplacement(sb, Matcher.quoteReplacement(rep)); } + m.appendTail(sb); return sb.toString(); + } catch (Exception e) { return "[exception " + e.getMessage() + "]"; } + } + void dumpFunc(Function f) throws IOException { + String n = String.format("%08x", f.getEntryPoint().getOffset()); + File fl = new File("/tmp/spine/" + n + ".c"); if (fl.exists()) return; + PrintWriter w = new PrintWriter(new FileWriter(fl)); + w.println("// " + f.getName() + " @ " + f.getEntryPoint() + " size=" + f.getBody().getNumAddresses()); + StringBuilder cs = new StringBuilder(); for (Function c : f.getCallingFunctions(monitor)) cs.append(c.getName() + "@" + c.getEntryPoint() + " "); + w.println("// CALLERS: " + cs); + StringBuilder ce = new StringBuilder(); for (Function c : f.getCalledFunctions(monitor)) ce.append(c.getName() + " "); + w.println("// CALLEES: " + ce); + w.println(decompRes(f)); w.close(); + } + String funcDesc(Function f) { + StringBuilder cs = new StringBuilder(); int n = 0; + for (Function c : f.getCallingFunctions(monitor)) { if (n++ < 6) cs.append(c.getName() + " "); } + return f.getName() + "@" + f.getEntryPoint() + " sz=" + f.getBody().getNumAddresses() + " callers(" + n + "):[" + cs.toString().trim() + "]"; + } + void refsTo(Address a, String label, boolean dump, int maxDump) { + ReferenceIterator ri = rm.getReferencesTo(a); int n = 0; StringBuilder sb = new StringBuilder(); + Set fs = new LinkedHashSet(); + while (ri.hasNext()) { Reference r = ri.next(); n++; Function f = getFunctionContaining(r.getFromAddress()); + if (f == null) { sb.append("\n ?" + r.getFromAddress() + " " + r.getReferenceType()); continue; } + sb.append("\n " + r.getFromAddress() + " " + r.getReferenceType() + " in " + funcDesc(f)); fs.add(f); } + out.println(a + " " + label + " refs=" + n + sb); + if (dump) { int k = 0; for (Function f : fs) { if (k++ >= maxDump) break; toDump.add(f); } } + } + void callersUp(Function f, int depth, String indent, Set seen) { + if (depth == 0 || f == null || seen.contains(f)) return; seen.add(f); + for (Function c : f.getCallingFunctions(monitor)) { out.println(indent + "<- " + funcDesc(c)); toDump.add(c); callersUp(c, depth - 1, indent + " ", seen); } + } + + @Override + public void run() throws Exception { + mem = currentProgram.getMemory(); rm = currentProgram.getReferenceManager(); st = currentProgram.getSymbolTable(); + decomp = new DecompInterface(); decomp.openProgram(currentProgram); + new File("/tmp/spine").mkdirs(); + out = new PrintWriter(new FileWriter("/tmp/spine/recon2.txt")); + + out.println("==== 1. STRING XREFS ===="); + for (long s : STRS) { Address a = toAddr(s); refsTo(a, "\"" + cstr(a, 80) + "\"", true, 6); } + + out.println("\n==== 2. CLASS VFTABLE / TYPEDESC XREFS ===="); + SymbolIterator si = st.getAllSymbols(true); + List syms = new ArrayList(); + while (si.hasNext()) { Symbol s = si.next(); String n = s.getName(true); + if (n.contains("Helper")) continue; + if (!(n.endsWith("::vftable") || n.endsWith("::RTTI_Type_Descriptor"))) continue; + for (String c : CLASSES) if (n.equals("Game::" + c + "::vftable") || n.equals("Game::" + c + "::RTTI_Type_Descriptor") || n.equals("Mars::" + c + "::vftable") || n.equals("Mars::" + c + "::RTTI_Type_Descriptor")) { syms.add(s); break; } } + for (Symbol s : syms) refsTo(s.getAddress(), s.getName(true), true, 4); + + out.println("\n==== 3. WIN32 API XREFS ===="); + for (String api : APIS) for (Symbol s : st.getSymbols(api)) { + if (s.getSymbolType() != SymbolType.FUNCTION && s.getSymbolType() != SymbolType.LABEL) continue; + refsTo(s.getAddress(), api + "(" + s.getSymbolType() + ")", api.startsWith("PeekMessage") || api.startsWith("GetMessage") || api.startsWith("DispatchMessage") || api.equals("GetStartupInfoA"), 6); + // follow thunks + ReferenceIterator ri = rm.getReferencesTo(s.getAddress()); + while (ri.hasNext()) { Reference r = ri.next(); Function f = getFunctionContaining(r.getFromAddress()); + if (f != null && (f.isThunk() || f.getBody().getNumAddresses() <= 6)) refsTo(f.getEntryPoint(), " thunk " + f.getName(), api.startsWith("PeekMessage") || api.startsWith("DispatchMessage"), 6); } + } + + out.println("\n==== 4. CALLERS UP from Init (FUN_008a0e50) and FUN_00741cd0 / FUN_00902470 ===="); + for (long fa : new long[]{0x008a0e50L, 0x00741cd0L, 0x00902470L}) { Function f = getFunctionAt(toAddr(fa)); out.println("ROOT " + funcDesc(f)); callersUp(f, 5, " ", new HashSet()); } + + out.println("\n==== 5. APP CLASSES (vftables with 'App' / 'Application' / 'Game' in name) ===="); + si = st.getAllSymbols(true); + while (si.hasNext()) { Symbol s = si.next(); String n = s.getName(true); + if (!n.endsWith("::vftable") || n.contains("Helper")) continue; + if (!(n.contains("App") || n.contains("Application") || n.contains("StarsGame") || n.contains("GameApp") || n.contains("StrategyClient") || n.contains("StrategyGame") || n.contains("GameServer") || n.contains("IApplication"))) continue; + Address a = s.getAddress(); StringBuilder sb = new StringBuilder(a + " " + n + " :"); + for (int i = 0; i < 24; i++) { try { long p = mem.getInt(a.add(i*4)) & 0xffffffffL; Function f = getFunctionAt(toAddr(p)); if (f == null) break; sb.append(" [" + i + "]" + f.getName() + "(" + f.getBody().getNumAddresses() + ")"); } catch (Exception e) { break; } } + out.println(sb); + } + // DAT_00b2d540 (app singleton) writers + out.println("\n==== 6. APP SINGLETON DAT_00b2d540 refs (writes) ===="); + { ReferenceIterator ri = rm.getReferencesTo(toAddr(0x00b2d540L)); int n = 0; + while (ri.hasNext()) { Reference r = ri.next(); if (!r.getReferenceType().isWrite()) continue; Function f = getFunctionContaining(r.getFromAddress()); out.println(" W " + r.getFromAddress() + " in " + (f == null ? "?" : funcDesc(f))); if (f != null) toDump.add(f); if (++n > 20) break; } } + + out.println("\n==== 7. what is at 0x009bf127 (s_SNMEndTurn ref) ===="); + { Address a = toAddr(0x009bf100L); MemoryBlock b = mem.getBlock(a); out.println("block=" + (b == null ? "none" : b.getName() + " exec=" + b.isExecute())); + Function f = getFunctionContaining(a); out.println("func=" + (f == null ? "none" : f.getName())); + if (getInstructionAt(a) == null) disassemble(a); + Instruction ins = getInstructionAt(a); int k = 0; + while (ins != null && k++ < 40) { StringBuilder refs = new StringBuilder(); for (Reference r : ins.getReferencesFrom()) { String s2 = cstr(r.getToAddress(), 40); Function tf = getFunctionAt(r.getToAddress()); refs.append(s2 != null ? " \"" + s2 + "\"" : (tf != null ? " ->" + tf.getName() : "")); } + out.println(" " + ins.getAddress() + " " + ins + refs); ins = ins.getNext(); } + // where do undefined-code refs come from: refs TO 0x009bf120 region + refsTo(toAddr(0x009bf110L), "0x009bf110", false, 0); + } + + for (long d : DECOMP) { Function f = getFunctionAt(toAddr(d)); if (f != null) toDump.add(f); } + out.println("\n==== 8. DUMPED ===="); + for (Function f : toDump) { out.println(f.getName() + " @ " + f.getEntryPoint() + " sz=" + f.getBody().getNumAddresses()); dumpFunc(f); } + out.close(); decomp.dispose(); println("done"); + } +} diff --git a/ghidra/scripts/SpineRecon3.java b/ghidra/scripts/SpineRecon3.java new file mode 100644 index 0000000..4b5bedb --- /dev/null +++ b/ghidra/scripts/SpineRecon3.java @@ -0,0 +1,84 @@ +import ghidra.app.script.GhidraScript; +import ghidra.program.model.address.*; +import ghidra.program.model.mem.*; +import ghidra.program.model.listing.*; +import ghidra.program.model.symbol.*; +import ghidra.app.decompiler.*; +import java.util.*; +import java.util.regex.*; +import java.io.*; + +// Pass 3 (read-only): main loop + turn-processing driver: decompile candidates, callers-up chains for SE* creators. +public class SpineRecon3 extends GhidraScript { + DecompInterface decomp; Memory mem; PrintWriter out; ReferenceManager rm; SymbolTable st; + Set toDump = new LinkedHashSet(); + static final long[] DECOMP = {0x0089f5b0L,0x0089dfb0L,0x00898b00L,0x00789710L,0x007b18b0L,0x007ad0f0L,0x0078f6a0L,0x0081ff40L,0x007dc6c0L,0x007cf540L, + 0x007dc640L,0x007da9a0L,0x007c5850L,0x007c5b00L,0x00789500L,0x00752500L,0x00815fd0L,0x007dd2f0L,0x008d2290L,0x0088c7b0L,0x0089c950L,0x00898ae0L,0x00898af0L, + 0x00761310L,0x00789330L,0x007c0600L,0x007d4400L,0x0076c610L,0x00769340L,0x00781cf0L,0x0089a3b0L,0x0089a640L,0x0089a480L,0x00899790L,0x0089eec0L,0x0089f1b0L}; + static final long[] UPCHAIN = {0x007da9a0L,0x00752500L,0x00789500L,0x007c5850L,0x007c5b00L,0x007dc640L,0x007c8d90L,0x007b18b0L,0x007ad0f0L,0x0083dbf0L,0x0078fac0L,0x007cbe80L,0x007bfe60L,0x00794770L,0x00784640L,0x0089f5b0L,0x00898b00L}; + + String cstr(Address a, int max) { + try { byte[] b = new byte[max]; int got = mem.getBytes(a, b); int i = 0; + while (i < got && b[i] != 0 && (b[i]&0xff) >= 0x20 && (b[i]&0xff) < 0x7f) i++; + if (i >= 1 && i < got && b[i] == 0) return new String(b, 0, i, "ISO-8859-1"); } catch (Exception e) {} + return null; + } + String decompRes(Function f) { + try { DecompileResults res = decomp.decompileFunction(f, 240, monitor); + if (res == null || !res.decompileCompleted()) return "[decompile failed]"; + String c = res.getDecompiledFunction().getC(); + Matcher m = Pattern.compile("&?(DAT|PTR_s_|s_[A-Za-z0-9_]*|PTR_DAT|u_[A-Za-z0-9_]*)_([0-9a-f]{8})").matcher(c); + StringBuffer sb = new StringBuffer(); + while (m.find()) { Address a = toAddr(Long.parseLong(m.group(2), 16)); String s = cstr(a, 100); String rep = m.group(0); + if (s != null && s.length() <= 96) rep = "\"" + s + "\""; m.appendReplacement(sb, Matcher.quoteReplacement(rep)); } + m.appendTail(sb); return sb.toString(); + } catch (Exception e) { return "[exception " + e.getMessage() + "]"; } + } + void dumpFunc(Function f) throws IOException { + String n = String.format("%08x", f.getEntryPoint().getOffset()); + File fl = new File("/tmp/spine/" + n + ".c"); if (fl.exists()) return; + PrintWriter w = new PrintWriter(new FileWriter(fl)); + w.println("// " + f.getName() + " @ " + f.getEntryPoint() + " size=" + f.getBody().getNumAddresses()); + StringBuilder cs = new StringBuilder(); for (Function c : f.getCallingFunctions(monitor)) cs.append(c.getName() + "@" + c.getEntryPoint() + " "); + w.println("// CALLERS: " + cs); + StringBuilder ce = new StringBuilder(); for (Function c : f.getCalledFunctions(monitor)) ce.append(c.getName() + " "); + w.println("// CALLEES: " + ce); + w.println(decompRes(f)); w.close(); + } + String funcDesc(Function f) { + StringBuilder cs = new StringBuilder(); int n = 0; + for (Function c : f.getCallingFunctions(monitor)) { if (n++ < 6) cs.append(c.getName() + " "); } + return f.getName() + "@" + f.getEntryPoint() + " sz=" + f.getBody().getNumAddresses() + " callers(" + n + "):[" + cs.toString().trim() + "]"; + } + // strings referenced by a function, in address order + String strsOf(Function f) { + TreeMap refs = new TreeMap(); + AddressIterator ai = rm.getReferenceSourceIterator(f.getBody(), true); + while (ai.hasNext()) { Address from = ai.next(); for (Reference r : rm.getReferencesFrom(from)) { Address to = r.getToAddress(); if (!mem.contains(to)) continue; + String sv = cstr(to, 90); if (sv != null && sv.length() >= 4) refs.put(from, sv); } } + StringBuilder sb = new StringBuilder(); for (String s : refs.values()) sb.append("\"" + s + "\" "); + return sb.toString(); + } + void callersUp(Function f, int depth, String indent, Set seen) { + if (depth == 0 || f == null || seen.contains(f)) return; seen.add(f); + for (Function c : f.getCallingFunctions(monitor)) { if (c.getName().startsWith("Unwind")) continue; out.println(indent + "<- " + funcDesc(c) + " strs: " + strsOf(c)); callersUp(c, depth - 1, indent + " ", seen); } + } + + @Override + public void run() throws Exception { + mem = currentProgram.getMemory(); rm = currentProgram.getReferenceManager(); st = currentProgram.getSymbolTable(); + decomp = new DecompInterface(); decomp.openProgram(currentProgram); + new File("/tmp/spine").mkdirs(); + out = new PrintWriter(new FileWriter("/tmp/spine/recon3.txt")); + out.println("==== 1. CALLER CHAINS (with strings) ===="); + for (long fa : UPCHAIN) { Function f = getFunctionAt(toAddr(fa)); if (f == null) continue; out.println("ROOT " + funcDesc(f) + " strs: " + strsOf(f)); callersUp(f, 4, " ", new HashSet()); } + out.println("\n==== 2. CALLEES of key drivers with strings ===="); + for (long fa : new long[]{0x0089f5b0L,0x0089dfb0L,0x007b18b0L,0x007dc640L,0x007db780L,0x007c8d90L,0x0089d610L,0x00898b00L,0x00888e80L}) { + Function f = getFunctionAt(toAddr(fa)); if (f == null) continue; out.println("DRIVER " + funcDesc(f)); + for (Function c : f.getCalledFunctions(monitor)) { if (c.getBody().getNumAddresses() < 60) continue; out.println(" -> " + funcDesc(c) + " strs: " + strsOf(c)); } } + for (long d : DECOMP) { Function f = getFunctionAt(toAddr(d)); if (f != null) toDump.add(f); } + out.println("\n==== 3. DUMPED ===="); + for (Function f : toDump) { out.println(f.getName() + " @ " + f.getEntryPoint() + " sz=" + f.getBody().getNumAddresses()); dumpFunc(f); } + out.close(); decomp.dispose(); println("done"); + } +} diff --git a/ghidra/scripts/SpineRecon4.java b/ghidra/scripts/SpineRecon4.java new file mode 100644 index 0000000..341e4c1 --- /dev/null +++ b/ghidra/scripts/SpineRecon4.java @@ -0,0 +1,119 @@ +import ghidra.app.script.GhidraScript; +import ghidra.program.model.address.*; +import ghidra.program.model.mem.*; +import ghidra.program.model.listing.*; +import ghidra.program.model.symbol.*; +import ghidra.program.model.scalar.*; +import ghidra.app.decompiler.*; +import java.util.*; +import java.util.regex.*; +import java.io.*; + +// Pass 4 (read-only): network message registry table, ProcessTurn callee strings, +0x170 callback writers, misc decompiles. +public class SpineRecon4 extends GhidraScript { + DecompInterface decomp; Memory mem; PrintWriter out; ReferenceManager rm; SymbolTable st; + Set toDump = new LinkedHashSet(); + static final long[] DECOMP = {0x00891340L,0x007598e0L,0x00898800L,0x0089a640L,0x00899210L,0x0090c700L,0x008e5ac0L,0x00783be0L,0x00783980L,0x007856f0L,0x007cd2a0L,0x007d4400L,0x00752500L,0x0078a7c0L,0x007ad100L,0x007d7f70L,0x00814ea0L,0x0086b300L,0x007adc80L,0x007a0e20L,0x007999a0L,0x007ae480L,0x007af0b0L,0x008986a0L,0x0089cc70L,0x00898690L,0x00723df0L}; + static final long[] CALLEE_STRS = {0x007dc6c0L,0x007598e0L,0x00891340L,0x007da9a0L,0x007d4400L,0x007cda40L,0x007842b0L,0x00741cd0L,0x00732ab0L}; + + String cstr(Address a, int max) { + try { byte[] b = new byte[max]; int got = mem.getBytes(a, b); int i = 0; + while (i < got && b[i] != 0 && (b[i]&0xff) >= 0x20 && (b[i]&0xff) < 0x7f) i++; + if (i >= 1 && i < got && b[i] == 0) return new String(b, 0, i, "ISO-8859-1"); } catch (Exception e) {} + return null; + } + String decompRes(Function f) { + try { DecompileResults res = decomp.decompileFunction(f, 240, monitor); + if (res == null || !res.decompileCompleted()) return "[decompile failed]"; + String c = res.getDecompiledFunction().getC(); + Matcher m = Pattern.compile("&?(DAT|PTR_s_|s_[A-Za-z0-9_]*|PTR_DAT|u_[A-Za-z0-9_]*)_([0-9a-f]{8})").matcher(c); + StringBuffer sb = new StringBuffer(); + while (m.find()) { Address a = toAddr(Long.parseLong(m.group(2), 16)); String s = cstr(a, 100); String rep = m.group(0); + if (s != null && s.length() <= 96) rep = "\"" + s + "\""; m.appendReplacement(sb, Matcher.quoteReplacement(rep)); } + m.appendTail(sb); return sb.toString(); + } catch (Exception e) { return "[exception " + e.getMessage() + "]"; } + } + void dumpFunc(Function f) throws IOException { + String n = String.format("%08x", f.getEntryPoint().getOffset()); + File fl = new File("/tmp/spine/" + n + ".c"); if (fl.exists()) return; + PrintWriter w = new PrintWriter(new FileWriter(fl)); + w.println("// " + f.getName() + " @ " + f.getEntryPoint() + " size=" + f.getBody().getNumAddresses()); + StringBuilder cs = new StringBuilder(); for (Function c : f.getCallingFunctions(monitor)) cs.append(c.getName() + "@" + c.getEntryPoint() + " "); + w.println("// CALLERS: " + cs); + StringBuilder ce = new StringBuilder(); for (Function c : f.getCalledFunctions(monitor)) ce.append(c.getName() + " "); + w.println("// CALLEES: " + ce); + w.println(decompRes(f)); w.close(); + } + String strsOf(Function f) { + TreeMap refs = new TreeMap(); + AddressIterator ai = rm.getReferenceSourceIterator(f.getBody(), true); + while (ai.hasNext()) { Address from = ai.next(); for (Reference r : rm.getReferencesFrom(from)) { Address to = r.getToAddress(); if (!mem.contains(to)) continue; + String sv = cstr(to, 90); if (sv != null && sv.length() >= 4) refs.put(from, sv); + Symbol s = st.getPrimarySymbol(to); if (s != null && s.getName(true).endsWith("::vftable") && !s.getName(true).contains("Helper")) refs.put(from, "<" + s.getName(true) + ">"); } } + StringBuilder sb = new StringBuilder(); for (String s : refs.values()) sb.append(s.startsWith("<") ? s + " " : "\"" + s + "\" "); + return sb.toString(); + } + + @Override + public void run() throws Exception { + mem = currentProgram.getMemory(); rm = currentProgram.getReferenceManager(); st = currentProgram.getSymbolTable(); + decomp = new DecompInterface(); decomp.openProgram(currentProgram); + new File("/tmp/spine").mkdirs(); + out = new PrintWriter(new FileWriter("/tmp/spine/recon4.txt")); + + out.println("==== 1. NETWORK MESSAGE REGISTRY (static initializers calling FUN_008d2290) ===="); + // scan .text for MOV ECX,imm32 ; CALL 0x008d2290 and read the two preceding pushes + Address lo = toAddr(0x009b0000L), hi = toAddr(0x009c8000L); + Address a = lo; int found = 0; + byte[] buf = new byte[(int)(hi.subtract(lo))]; mem.getBytes(lo, buf); + TreeMap byId = new TreeMap(); + for (int i = 0; i + 10 < buf.length; i++) { + if ((buf[i]&0xff) == 0xB9 && (buf[i+5]&0xff) == 0xE8) { + long rel = (buf[i+6]&0xffL) | ((buf[i+7]&0xffL)<<8) | ((buf[i+8]&0xffL)<<16) | ((buf[i+9]&0xffL)<<24); + long target = lo.getOffset() + i + 10 + (int)rel; + if (target != 0x008d2290L) continue; + long entry = (buf[i+1]&0xffL) | ((buf[i+2]&0xffL)<<8) | ((buf[i+3]&0xffL)<<16) | ((buf[i+4]&0xffL)<<24); + // walk back: PUSH name (68 imm32), PUSH id (6a imm8 | 68 imm32), PUSH factory (68 imm32) + int p = i - 5; long name = 0, id = -1, factory = 0; + if ((buf[p]&0xff) == 0x68) { name = (buf[p+1]&0xffL) | ((buf[p+2]&0xffL)<<8) | ((buf[p+3]&0xffL)<<16) | ((buf[p+4]&0xffL)<<24); } + else continue; + int q; + if ((buf[p-2]&0xff) == 0x6a) { id = buf[p-1]&0xff; q = p - 2; } + else if ((buf[p-5]&0xff) == 0x68) { id = (buf[p-4]&0xffL) | ((buf[p-3]&0xffL)<<8) | ((buf[p-2]&0xffL)<<16) | ((buf[p-1]&0xffL)<<24); q = p - 5; } + else continue; + if ((buf[q-5]&0xff) == 0x68) factory = (buf[q-4]&0xffL) | ((buf[q-3]&0xffL)<<8) | ((buf[q-2]&0xffL)<<16) | ((buf[q-1]&0xffL)<<24); + String nm = cstr(toAddr(name), 64); + Function ff = getFunctionAt(toAddr(factory)); + String line = String.format("id=0x%02x entry=%08x name=%-32s factory=%08x %s init@%08x", id, entry, nm, factory, ff == null ? "(nofunc)" : ff.getName() + "(sz " + ff.getBody().getNumAddresses() + ")", lo.getOffset() + i); + byId.put((int)id, line); found++; + } + } + for (String l : byId.values()) out.println(l); + out.println("total " + found); + + out.println("\n==== 2. WRITERS of StrategyServer+0x170 (callback) : scan for 'mov [reg+0x170], ' in functions referencing StrategyServer ===="); + // brute: instructions with scalar 0x170 operand as memory displacement, being a store + for (long fa : new long[]{0x007d78d0L, 0x00888e80L, 0x007d20d0L}) { + Function f = getFunctionAt(toAddr(fa)); if (f == null) continue; + InstructionIterator ii = currentProgram.getListing().getInstructions(f.getBody(), true); + while (ii.hasNext()) { Instruction ins = ii.next(); String s = ins.toString(); if (s.contains("0x170]")) out.println(" " + f.getName() + " " + ins.getAddress() + " " + s); } + } + // also global search on refs to all functions passed as pointer near 0x170... skip. Check refs to the candidate callback functions: StrategyClient vft[5] FUN_00774fc0 and others in 0x0077xxxx + for (long fa : new long[]{0x00774fc0L, 0x00774f30L, 0x00775100L, 0x00779a70L}) { + Function f = getFunctionAt(toAddr(fa)); if (f == null) continue; StringBuilder sb = new StringBuilder(); + ReferenceIterator ri = rm.getReferencesTo(f.getEntryPoint()); while (ri.hasNext()) { Reference r = ri.next(); Function c = getFunctionContaining(r.getFromAddress()); sb.append(r.getFromAddress() + "(" + r.getReferenceType() + (c == null ? "" : " in " + c.getName()) + ") "); } + out.println(" refs to " + f.getName() + ": " + sb); + } + + out.println("\n==== 3. CALLEE STRINGS ===="); + for (long fa : CALLEE_STRS) { + Function f = getFunctionAt(toAddr(fa)); if (f == null) continue; out.println("DRIVER " + f.getName() + "@" + f.getEntryPoint() + " sz=" + f.getBody().getNumAddresses() + " strs: " + strsOf(f)); + for (Function c : f.getCalledFunctions(monitor)) { if (c.getBody().getNumAddresses() < 40) continue; String s = strsOf(c); StringBuilder cs = new StringBuilder(); int n = 0; for (Function cc : c.getCallingFunctions(monitor)) if (n++ < 4) cs.append(cc.getName() + " "); + out.println(" -> " + c.getName() + "@" + c.getEntryPoint() + " sz=" + c.getBody().getNumAddresses() + " callers(" + n + "):[" + cs.toString().trim() + "] strs: " + s); } + } + for (long d : DECOMP) { Function f = getFunctionAt(toAddr(d)); if (f != null) toDump.add(f); } + out.println("\n==== 4. DUMPED ===="); + for (Function f : toDump) { out.println(f.getName() + " @ " + f.getEntryPoint() + " sz=" + f.getBody().getNumAddresses()); dumpFunc(f); } + out.close(); decomp.dispose(); println("done"); + } +} diff --git a/ghidra/scripts/WriteBack.java b/ghidra/scripts/WriteBack.java new file mode 100644 index 0000000..9e5b39b --- /dev/null +++ b/ghidra/scripts/WriteBack.java @@ -0,0 +1,363 @@ +import ghidra.app.script.GhidraScript; +import ghidra.program.model.address.*; +import ghidra.program.model.mem.*; +import ghidra.program.model.listing.*; +import ghidra.program.model.symbol.*; +import ghidra.program.model.data.*; +import ghidra.app.decompiler.*; +import java.util.*; +import java.util.regex.*; +import java.io.*; + +// Write-back: struct data types from findings/objects/struct-recovery.md, function renames (serializers, affinity, +// turn/main-loop spine), plate comments, net-message registry labels; then verification decompiles. +public class WriteBack extends GhidraScript { + DataTypeManager dtm; SymbolTable st; PrintWriter log; DecompInterface decomp; Memory mem; + Map types = new HashMap(); + static final String DOC = "Source: /srv/re-lab/notes/findings/objects/struct-recovery.md (save-field xref recovery, 2026-09-07)"; + static final String DOC2 = "Source: /srv/re-lab/handoff/ghidra-writeback-and-spine.md (turn/main-loop spine, 2026-09-07)"; + CategoryPath CAT = new CategoryPath("/SOTS"); + + // ---------- type helpers ---------- + DataType prim(String k) { + if (k.equals("int")) return IntegerDataType.dataType; + if (k.equals("uint")) return UnsignedIntegerDataType.dataType; + if (k.equals("short")) return ShortDataType.dataType; + if (k.equals("i64")) return LongLongDataType.dataType; + if (k.equals("float")) return FloatDataType.dataType; + if (k.equals("double")) return DoubleDataType.dataType; + if (k.equals("bool")) return BooleanDataType.dataType; + if (k.equals("i8")) return SignedByteDataType.dataType; + if (k.equals("u8")) return ByteDataType.dataType; + if (k.equals("ptr")) return new PointerDataType(VoidDataType.dataType); + if (k.equals("pfn")) return new PointerDataType(VoidDataType.dataType); + if (k.endsWith("*")) { DataType t = types.get(k.substring(0, k.length()-1)); return new PointerDataType(t == null ? VoidDataType.dataType : t); } + Matcher m = Pattern.compile("^(.+)\\[(\\d+)\\]$").matcher(k); + if (m.matches()) { DataType b = prim(m.group(1)); int n = Integer.parseInt(m.group(2)); return new ArrayDataType(b, n, b.getLength()); } + DataType t = types.get(k); if (t != null) return t; + throw new RuntimeException("unknown type " + k); + } + Structure mk(String name, int size) throws Exception { + String nm = DataUtilities.isValidDataTypeName(name) ? name : name.replace("::", "_"); + StructureDataType s = new StructureDataType(CAT, nm, size, dtm); + Structure r = (Structure) dtm.addDataType(s, DataTypeConflictHandler.REPLACE_HANDLER); + types.put(name, r); log.println(" type " + r.getPathName() + " size=" + r.getLength()); + return r; + } + Structure opaque(String name) throws Exception { // forward-declared class, only used through pointers + String nm = DataUtilities.isValidDataTypeName(name) ? name : name.replace("::", "_"); + StructureDataType s = new StructureDataType(CAT, nm, 0, dtm); s.setDescription("opaque (only referenced by pointer)"); + Structure r = (Structure) dtm.addDataType(s, DataTypeConflictHandler.REPLACE_HANDLER); types.put(name, r); return r; + } + void fields(Structure s, Object[][] rows) { + for (Object[] r : rows) { + int off = ((Number) r[0]).intValue(); DataType dt = prim((String) r[1]); String name = (String) r[2]; String cmt = r.length > 3 ? (String) r[3] : null; + if (off + dt.getLength() > s.getLength()) { log.println(" !! " + s.getName() + "." + name + " @0x" + Integer.toHexString(off) + " exceeds size"); continue; } + try { s.replaceAtOffset(off, dt, dt.getLength(), name, cmt); } + catch (Exception e) { log.println(" !! " + s.getName() + "." + name + " @0x" + Integer.toHexString(off) + ": " + e.getMessage()); } + } + } + Structure fill(String name, int size, String desc, Object[][] rows) throws Exception { + Structure s = types.containsKey(name) ? (Structure) types.get(name) : mk(name, size); + if (s.getLength() != size) { s = mk(name, size); } + s.setDescription(desc + "\n" + DOC); fields(s, rows); return s; + } + // shifted copy for serializer 'this' (= object + colOff) + Structure serView(String name, int colOff) throws Exception { + Structure src = (Structure) types.get(name); + Structure v = mk(name + "_ser" + colOff, src.getLength() - colOff); + v.setDescription("Serializer view of " + name + ": this = object + 0x" + Integer.toHexString(colOff) + " (IStreamable sub-object). Use only to read " + name + "::Read/Write."); + for (DataTypeComponent c : src.getDefinedComponents()) { if (c.getOffset() < colOff) continue; try { v.replaceAtOffset(c.getOffset() - colOff, c.getDataType(), c.getLength(), c.getFieldName(), c.getComment()); } catch (Exception e) {} } + return v; + } + + // ---------- symbol helpers ---------- + Namespace ns(String path) throws Exception { + Namespace cur = currentProgram.getGlobalNamespace(); + for (String part : path.split("::")) { + Namespace n = st.getNamespace(part, cur); + if (n == null) n = st.createClass(cur, part, SourceType.USER_DEFINED); + cur = n; + } + return cur; + } + void fn(long addr, String qualified, String plate) { + try { + Function f = getFunctionAt(toAddr(addr)); + if (f == null) { log.println(" !! no function at " + Long.toHexString(addr)); return; } + int i = qualified.lastIndexOf("::"); + String name = i < 0 ? qualified : qualified.substring(i + 2); + Namespace n = i < 0 ? currentProgram.getGlobalNamespace() : ns(qualified.substring(0, i)); + f.setParentNamespace(n); + f.setName(name, SourceType.USER_DEFINED); + if (plate != null) f.setComment(plate); + log.println(" fn " + toAddr(addr) + " -> " + qualified); + } catch (Exception e) { log.println(" !! rename " + Long.toHexString(addr) + " " + qualified + ": " + e.getMessage()); } + } + void lbl(long addr, String name, String cmt) { + try { Address a = toAddr(addr); createLabel(a, name, true, SourceType.USER_DEFINED); if (cmt != null) setEOLComment(a, cmt); log.println(" label " + a + " " + name); } + catch (Exception e) { log.println(" !! label " + Long.toHexString(addr) + ": " + e.getMessage()); } + } + void retypeThis(long addr, DataType ptr) { + try { Function f = getFunctionAt(toAddr(addr)); + if (f.getParameterCount() > 0) f.getParameter(0).setDataType(ptr, SourceType.USER_DEFINED); + else { List ps = new ArrayList(); ps.add(new ParameterImpl("this", ptr, currentProgram)); ps.add(new ParameterImpl("stream", new PointerDataType(VoidDataType.dataType), currentProgram)); + f.replaceParameters(ps, Function.FunctionUpdateType.DYNAMIC_STORAGE_ALL_PARAMS, true, SourceType.USER_DEFINED); f.setCallingConvention("__thiscall"); } + log.println(" retyped this of " + f.getName() + " -> " + ptr.getName()); + } catch (Exception e) { log.println(" !! retype " + Long.toHexString(addr) + ": " + e.getMessage()); } + } + String cstr(Address a, int max) { + try { byte[] b = new byte[max]; int got = mem.getBytes(a, b); int i = 0; + while (i < got && b[i] != 0 && (b[i]&0xff) >= 0x20 && (b[i]&0xff) < 0x7f) i++; + if (i >= 1 && i < got && b[i] == 0) return new String(b, 0, i, "ISO-8859-1"); } catch (Exception e) {} + return null; + } + void verify(long addr, String outName) throws Exception { + Function f = getFunctionAt(toAddr(addr)); + DecompileResults res = decomp.decompileFunction(f, 240, monitor); + PrintWriter w = new PrintWriter(new FileWriter("/tmp/spine/" + outName)); + if (res == null || !res.decompileCompleted()) { w.println("[decompile failed]"); w.close(); return; } + String c = res.getDecompiledFunction().getC(); + Matcher m = Pattern.compile("&?(DAT|PTR_s_|s_[A-Za-z0-9_]*|PTR_DAT|u_[A-Za-z0-9_]*)_([0-9a-f]{8})").matcher(c); + StringBuffer sb = new StringBuffer(); + while (m.find()) { Address a = toAddr(Long.parseLong(m.group(2), 16)); String s = cstr(a, 64); String rep = m.group(0); if (s != null && s.length() <= 60) rep = "\"" + s + "\""; m.appendReplacement(sb, Matcher.quoteReplacement(rep)); } + m.appendTail(sb); w.println(sb); w.close(); log.println(" verify decompile written " + outName + " (" + sb.length() + " chars)"); + } + + @Override + public void run() throws Exception { + dtm = currentProgram.getDataTypeManager(); st = currentProgram.getSymbolTable(); mem = currentProgram.getMemory(); + new File("/tmp/spine").mkdirs(); + log = new PrintWriter(new FileWriter("/tmp/spine/writeback.log")); + log.println("==== TYPES ===="); + // --- std / Mars primitives --- + fill("Mars::Vector3", 12, "3 floats", new Object[][]{{0,"float","x"},{4,"float","y"},{8,"float","z"}}); + Structure bx = mk("std::string::_Bxty", 16); bx.setDescription("SSO buffer / heap pointer union"); + fields(bx, new Object[][]{{0,"ptr","_Ptr","heap pointer when _Myres >= 16 (else the 16-byte SSO buffer starts here)"}}); + fill("std::string", 0x1c, "MSVC10 basic_string: _Bx@0 (SSO buf / _Ptr), _Mysize@0x10, _Myres@0x14, _Alval@0x18. Verified from Stream::WriteString (FUN_008b9d70): if (str->_Myres > 15) p = str->_Ptr.", + new Object[][]{{0,"std::string::_Bxty","_Bx"},{0x10,"uint","_Mysize"},{0x14,"uint","_Myres"},{0x18,"u8","_Alval"}}); + fill("std::vector", 12, "MSVC10 vector: _Myfirst/_Mylast/_Myend (element type noted in field comments)", new Object[][]{{0,"ptr","_Myfirst"},{4,"ptr","_Mylast"},{8,"ptr","_Myend"}}); + fill("std::list", 8, "MSVC10 list: _Myhead (sentinel node: next@0,prev@4,value@8), _Mysize", new Object[][]{{0,"ptr","_Myhead"},{4,"uint","_Mysize"}}); + fill("std::map", 8, "MSVC10 map/set: _Myhead (node: left@0,parent@4,right@8,key@0xc,value@0x10.. ,isnil byte at tail), _Mysize", new Object[][]{{0,"ptr","_Myhead"},{4,"uint","_Mysize"}}); + // opaque classes referenced by pointer + for (String o : new String[]{"Game::BuildQueue","Game::TechTree","Game::ShipDesign","Game::CommMessageContainer","Game::Tech","Game::FleetNameGenerator","Game::AIRebellion","Game::AIEncounterFlags", + "Game::ServerNodeGraph","Game::ServerTradeManager","Game::IServerSpyManager","Game::AttribMap","Game::SVScriptObject","Game::StarSystem","Game::Plague","Game::DefenceLayout","Game::SpecialProjectImpl","Game::StrategyEvent","Mars::Stream"}) opaque(o); + // main structs first as empty shells so cross pointers resolve + Structure sys = mk("Game::ServerSystem", 0x2d8); Structure ply = mk("Game::ServerPlayer", 0x3e0); Structure flt = mk("Game::StarFleet", 0x120); Structure shp = mk("Game::StarShip", 0xb0); Structure srv = mk("Game::StrategyServer", 0x320); + // --- nested --- + fill("Game::PlayerColorID", 4, "index (-1 = custom rgb) + r,g,b. Write FUN_0053c080", new Object[][]{{0,"i8","index"},{1,"u8","r"},{2,"u8","g"},{3,"u8","b"}}); + fill("Game::PopulationGroup", 0x18, "Read FUN_00536a80 / Write FUN_00536af0", new Object[][]{{0,"ptr","vptr"},{4,"int","PopT"},{8,"int","PopS"},{0x10,"i64","PopC"}}); + fill("Game::Population", 0x14, "vector; Read FUN_005390c0 / Write FUN_00537ef0 (PopNG count, PopG entries)", new Object[][]{{0,"ptr","vptr"},{4,"std::vector","groups","vector (stride 0x18)"}}); + fill("Game::Morale", 0x20, "int[7]; disk: mnsp then (msp,mv) sparse pairs. Read FUN_00744dd0 / Write FUN_00744ea0", new Object[][]{{0,"ptr","vptr"},{4,"int[7]","values"}}); + fill("Game::MoraleEvent", 0x50, "Read FUN_007490b0 / Write FUN_007491b0", new Object[][]{{0,"ptr","vptr"},{4,"int","mid"},{8,"int","mtr"},{0xc,"int","mn"},{0x10,"int","mtp"},{0x14,"Game::Morale","mfx"},{0x34,"std::string","mdsc"}}); + fill("Game::IndependenceInfo", 0x70, "Read FUN_00748df0 / Write FUN_00748ee0", new Object[][]{{0,"ptr","vptr"},{4,"int","indsp"},{8,"Game::PlayerColorID","indcl"},{0x1c,"std::string","indnm"},{0x38,"std::string","indav"},{0x54,"std::string","indba"}}); + fill("Game::StarSystem::OutputRates", 0x1c, "POD; Read FUN_007472a0 / Write FUN_00745190. Disk order SRs,SRt,SRsc,SRtf,SRi,SRoh,SRnr", new Object[][]{{0,"float","SRt"},{4,"float","SRsc"},{8,"float","SRtf"},{0xc,"float","SRi"},{0x10,"float","SRoh"},{0x14,"float","SRs"},{0x18,"int","SRnr"}}); + fill("Game::StarSystem::PlayerView", 0x9c, "per-player seen snapshot (map value in ServerSystem.NVs). Read FUN_00752af0 / Write FUN_007492d0", new Object[][]{{0,"ptr","vptr","0x00a201ac"},{8,"int","VTrn"},{0xc,"int","Pop"},{0x10,"Game::Population","Pop2"},{0x24,"float","Infra"},{0x28,"float","Suit"},{0x2c,"int","Res"},{0x30,"int","ARes2"},{0x34,"int","MRes"},{0x38,"bool","NoRebAI"},{0x3c,"int","pbon"},{0x40,"Game::Population","pbon2"},{0x54,"float","ibon"},{0x58,"int","TerrFl"}}); + fill("Game::ShipBuildOrder", 0x18, "build-queue entry. Read FUN_00813770 / Write FUN_00813800", new Object[][]{{0,"ptr","vptr"},{4,"int","desID"},{8,"int","con"},{0xc,"int","sav"},{0x10,"int","conleft"},{0x14,"int","ordID"}}); + fill("Game::DiplomacyStats", 0x24, "int16 counters (int32 on disk). Write FUN_00818cb0", new Object[][]{{0,"ptr","vptr"},{4,"int","other"},{8,"short","lastnap"},{0xa,"short","lastnapbty"},{0xc,"short","bknnap"},{0xe,"short","btynap"},{0x10,"short","lastally"},{0x12,"short","lastallybty"},{0x14,"short","bknally"},{0x16,"short","btyally"},{0x18,"short","lastcf"},{0x1a,"short","lastcfbty"},{0x1c,"short","bkncf"},{0x1e,"short","btycf"},{0x20,"short","deadhome"}}); + fill("Game::PlayerReport", 0x30, "preps entry. Read FUN_008200a0 / Write FUN_00817480", new Object[][]{{0,"ptr","vptr"},{4,"int","oid"},{8,"int","pid"},{0xc,"int","flds"},{0x10,"int","sav"},{0x14,"int","home"},{0x18,"int","ncol"},{0x1c,"int","mpwr"},{0x20,"int","mcls"},{0x24,"int","mmsl"},{0x28,"int","nshp"},{0x2c,"int","nsat"}}); + fill("Game::PlayerAlliances", 0x10, "Write FUN_006d2e10 (disk tag Team)", new Object[][]{{0,"int","ALid"},{4,"int","AL"},{8,"int","NA"},{0xc,"int","CF"}}); + fill("Game::NodeRoute", 0x10, "Read FUN_006e2260 / Write FUN_006e22e0", new Object[][]{{0,"ptr","vptr"},{4,"int","nrp"},{8,"int","nrf"},{0xc,"int","nrt"}}); + fill("Game::Waypoint", 0x1c, "Read FUN_00701860 / Write FUN_00700ed0", new Object[][]{{0,"ptr","vptr"},{4,"int","Wpt"},{8,"int","Tp"},{0xc,"Game::NodeRoute","nrt"}}); + fill("Game::FlightPlan", 0x38, "Read FUN_00704c70 / Write FUN_00700f60", new Object[][]{{0,"ptr","vptr"},{4,"std::vector","wpts","vector (stride 0x1c)"},{0x14,"float","FPsp2"},{0x18,"int","FPeta2"},{0x1c,"Mars::Vector3","FPogn2"},{0x28,"Mars::Vector3","FPdpos"},{0x34,"int","pnd"}}); + fill("Game::ShipHealth", 0x10, "3 unnamed floats on disk. Write FUN_00813e50", new Object[][]{{0,"ptr","vptr"},{4,"float[3]","hp"}}); + fill("Game::PrisonerHold", 0x18, "Read FUN_0056eb00 / Write FUN_0056ec00; counts[0]=PrMax, [2..8] per species", new Object[][]{{0,"ptr","vptr"},{0x14,"int*","counts"}}); + fill("Game::EventStorage", 0x1c, "Write FUN_00825cc0", new Object[][]{{0,"ptr","vptr"},{4,"std::vector","Events","vector"},{0x14,"int","EvNxID"}}); + fill("Game::CivilianRatios", 0x2c, "Write FUN_0082c740 (body not recovered)", new Object[][]{{0,"ptr","vptr"}}); + fill("Game::ShipRecords", 0x44, "Write FUN_008176a0 (body not recovered)", new Object[][]{{0,"ptr","vptr"}}); + fill("Game::FleetLayout", 0x24, "two vectors; HLay gate = either non-empty", new Object[][]{{0,"ptr","vptr"},{4,"std::vector","v0"},{0x14,"std::vector","v1"}}); + fill("Game::SpyReport", 0x34, "four std::lists. Read FUN_008843d0 / Write FUN_00828ec0", new Object[][]{{0,"ptr","vptr"},{4,"std::list","defences","SpyReportDefences (count defc2)"},{0x10,"std::list","trade","SpyReportTrade (rtc)"},{0x1c,"std::list","events","SpyReportEvents (evc)"},{0x28,"std::list","techtree","SpyReportTechTree (ttc)"}}); + // --- ServerSystem --- + fill("Game::ServerSystem", 0x2d8, "Game::ServerSystem : StarSystem : StarMapNode (+NetworkObject@0, IStreamable@8, HandleObject@0xc). Serializer this = obj+8. Read FUN_0075d4b0 / Write FUN_00749630.", new Object[][]{ + {0,"ptr","vptr_StarSystem","0x00a200e4"},{4,"int","netId","NetworkObject id (handle id written by Stream::WriteHandleId)"},{8,"ptr","vptr_IStreamable","0x00a2043c; serializer this"},{0xc,"ptr","vptr_HandleObject"},{0x10,"Game::StrategyServer*","owner","owner->Players used as player table"}, + {0x18,"Mars::Vector3","Pos"},{0x4c,"float","R"},{0x50,"float","G"},{0x54,"float","B"},{0x58,"float","A"},{0x5c,"int","Idx"},{0x60,"int","Size"},{0x64,"float","Suit"},{0x68,"int","Res"},{0x6c,"int","ARes2"},{0x70,"int","MRes"},{0x74,"int","TRes"}, + {0x78,"bool[3]","haltv"},{0x7c,"float","OutMod"},{0x80,"int","TAcq"},{0x84,"int","TFAcq"},{0x88,"Game::StarSystem::OutputRates","Rts"},{0xa4,"Game::BuildQueue*","BQ"},{0xa8,"std::string","Name"}, + {0xc4,"bool","Abdn"},{0xc5,"bool","Dstyd"},{0xc6,"bool","vnh"},{0xc7,"bool","vnd"},{0xc8,"bool","vnex3"},{0xc9,"bool","vnpex3"},{0xcc,"int","VFlags"},{0xd0,"int","EFlags"},{0xd4,"int","AFlags"},{0xd8,"int","FFlags"},{0xdc,"int","GFlags"},{0xe0,"int","MnRFlags"},{0xe4,"int","RfRFlags"},{0xe8,"int","ClkFlags"}, + {0xf0,"i64","Bats2"},{0xf8,"i64","rcex"},{0x100,"Game::ServerPlayer*","PID","owner player (handle id on disk)"},{0x104,"Game::Population","dcs"},{0x118,"float","dsu"},{0x11c,"Game::Morale","cm"},{0x13c,"std::vector","cme2","vector"},{0x14c,"Game::Morale","PvCM"}, + {0x16c,"std::vector","Flts","vector (NumFlts/Flt)"},{0x17c,"float","RepCur"},{0x180,"float","RepMax"},{0x184,"int","EggScio"},{0x188,"bool","NoRebAI"},{0x189,"bool","PvNoRebAI"},{0x18c,"int","Pop"},{0x190,"float","Infra"},{0x194,"int","pbon"},{0x198,"float","ibon"},{0x19c,"int","TerrFl"}, + {0x1a0,"Game::Population","Pop2"},{0x1b4,"Game::Population","pbon2"},{0x1c8,"Game::IndependenceInfo*","indi","hindi = (indi != NULL)"},{0x1cc,"std::vector","spies2","vector"},{0x1dc,"int","rbfl"},{0x1e0,"bool","hsrg"},{0x1e4,"int[7]","adt","addiction table: nadct + sparse (ads,adt)"}, + {0x200,"int","PvPop"},{0x204,"float","PvInfra"},{0x208,"float","PvSuit"},{0x20c,"int","PvRes"},{0x210,"int","PvARes2"},{0x214,"int","PvMRes"},{0x218,"Game::Population","PvPop2"},{0x238,"Game::StarFleet*","DefF"},{0x23c,"Game::StarFleet*","DefSF"}, + {0x240,"std::vector","GFs","gates (NumGFs/GF)"},{0x250,"std::vector","SnF","stations (NumSnF/SnF)"},{0x260,"std::vector","MnF","monitors (NumMnF/MnF)"},{0x274,"std::map","NVO","colonies: key player idx -> {TShn i16@+2, OID@+4, isind@+8, indi IndependenceInfo@+0xc}"},{0x284,"std::map","NVE","key player idx -> {ETS i16@+2, Eid@+4}"},{0x294,"std::map","NVs","key player idx -> PlayerView (pview)"}, + {0x2a8,"std::vector","Plgs","vector (NumPlgs2/PlgT/Plg)"},{0x2b8,"int","TnsOH"},{0x2bc,"int","TDst"},{0x2c4,"int","ntdev"},{0x2c8,"int","ltis"},{0x2cc,"int","rbtn"},{0x2d0,"int","rbfr"},{0x2d4,"int","rbwn"}}); + // --- ServerPlayer --- + fill("Game::ServerPlayer", 0x3e0, "Game::ServerPlayer : StrategyPlayer (NetworkObject@0). IStreamable sub-object at +0x3a0 (serializer this = obj+0x3a0, most member offsets negative there). Read FUN_008804d0 / Write FUN_008563e0.", new Object[][]{ + {0,"ptr","vptr","0x00a327a4"},{4,"int","netId"},{0x28,"int","PlyrIdx"},{0x2c,"Game::ServerSystem*","HomeSys"},{0x30,"std::vector","Own","vector (NumOwn/OwnId)"},{0x40,"std::string","PlryName"},{0x5c,"int","Species","0 Human .. 6 Morrigi"},{0x60,"Game::PlayerColorID","ClrID"}, + {0x74,"std::string","Bdg"},{0x90,"std::string","Avt"},{0xac,"int","Team"},{0xb0,"float","IdealSuit"},{0xb4,"float","SuitTol"},{0xb8,"float","MaxOH"},{0xbc,"float","ResRate"},{0xc0,"float","ResMod"},{0xc4,"float","ResScl"},{0xd0,"float","TRM"},{0xd4,"int","TRA"},{0xd8,"int","TRP"}, + {0xe4,"std::vector","Des","vector (NumDes/DesID/Des)"},{0xf4,"Game::TechTree*","TechTree"},{0xf8,"bool","Elim"},{0xf9,"bool","bTurnDone_nonser","not serialized; tested by StrategyServer::OnPlayerEndTurn"},{0xfb,"bool","NPC"},{0xfc,"bool","RebAI"},{0xfd,"bool","ReqCL"},{0xfe,"bool","AIBn"},{0xff,"bool","CnTrd"},{0x100,"bool","CnRad"},{0x101,"bool","CnVItl"},{0x102,"bool","hgs"},{0x103,"bool","hadvs"},{0x104,"bool","harcc"}, + {0x108,"float","pddm"},{0x10c,"float[3]","ConMod"},{0x118,"float[3]","SavMod"},{0x124,"float","OutMod"},{0x128,"float","RebOutMod"},{0x12c,"float","ScOutMod"},{0x130,"float","PopMod"},{0x134,"float","TerraMod"},{0x138,"bool","AMine"},{0x13c,"float","MinPure"},{0x140,"float","MinRate"},{0x144,"int","NGts"},{0x148,"int","PrGtTrf"},{0x14c,"int","GTraf"}, + {0x150,"float","CstR"},{0x154,"float","CstE"},{0x158,"float","CstT"},{0x15c,"int","Maint"},{0x160,"float","shrm"},{0x164,"int","Status","also set by SNMSetPlayerStatus; 4 = turn done"},{0x168,"Game::PlayerAlliances","Team2","disk tag Team {ALid,AL,NA,CF}"},{0x178,"std::vector","Leg","vector (NumLeg)"},{0x188,"int","PvSav"},{0x18c,"bool","PvMA"}, + {0x19c,"int","HasDisc"},{0x1a0,"int","HasDiscSp"},{0x1a4,"int","HasDiscCl"},{0x1a8,"int","HasEnc"},{0x1ac,"int","HasEng"},{0x1b0,"Game::ShipRecords","ShipRecs"},{0x1f4,"std::vector","Ojvs","vector"},{0x204,"std::vector","Nexp","vector<{int xid,xmin,xmax; float xper}> (16 B)"},{0x214,"std::vector","WeapXcl","vector"}, + {0x230,"std::vector","dipstats","vector"},{0x240,"Game::CommMessageContainer*","comms"},{0x244,"std::vector","preps","vector"},{0x254,"std::vector","odes","vector"},{0x264,"std::vector","owep","vector"},{0x274,"std::vector","otch","vector"}, + {0x284,"int","Sav"},{0x288,"int","HasImm"},{0x28c,"int","HasVac"},{0x290,"int","NPTrk"},{0x294,"Game::Tech*","ResT","current research (ResTNm = tech->name)"},{0x298,"Game::FleetNameGenerator*","FNG"},{0x29c,"Game::EventStorage","Events"},{0x2b8,"std::list","Nts","list (NumNotes/Nts)"}, + {0x2c4,"int","BnkWrn"},{0x2c8,"int","BnkTrn"},{0x2cc,"int","BnkEl"},{0x2d0,"int","BnkPr"},{0x2d8,"int","plcy"},{0x2dc,"std::string","pswd"},{0x2f8,"bool","Srn"},{0x2fc,"ptr","SrnTo","handle id on disk"},{0x300,"int","lboid"},{0x304,"int","lcid2"},{0x30c,"float","IncMod"},{0x310,"std::vector","aid","vector"}, + {0x320,"std::vector","deflay","vector (ndeflay/deflay)"},{0x330,"bool","cdp"},{0x334,"Game::SpyReport*","spy2"},{0x338,"std::vector","rdt","vector (stride 0x20)"},{0x368,"int","aidf"},{0x370,"Game::CivilianRatios","civr"},{0x39c,"int","tnc","written as max(v,1)"}, + {0x3a0,"ptr","vptr_IStreamable","0x00a32794; serializer this"},{0x3a4,"std::vector","PR","vector<{float PRm; int PRBt}>"},{0x3b4,"bool","ResErrRoll"},{0x3b5,"bool","cta"},{0x3b8,"Game::AIRebellion*","AIR","HasAIR = (AIR != NULL)"},{0x3bc,"Game::AIEncounterFlags*","AIEnf"},{0x3c0,"std::vector","Sprj","vector (NSprj/SprjT/Sprj)"},{0x3d0,"int","NextPrjID"},{0x3d4,"int","lret"},{0x3dc,"int","nmeid"}}); + // --- StarFleet --- + fill("Game::StarFleet", 0x120, "Game::StarFleet : StarMapNode (+NetworkObject@0, IStreamable@8, HandleObject@0xc). Serializer this = obj+8. Read FUN_00702470 / Write FUN_00701070.", new Object[][]{ + {0,"ptr","vptr","0x00a1d608"},{4,"int","netId"},{8,"ptr","vptr_IStreamable","0x00a1d5f8"},{0xc,"ptr","vptr_HandleObject"},{0x10,"ptr","owner"},{0x18,"Mars::Vector3","Pos"},{0x4c,"Mars::Vector3","PrvPos"},{0x58,"Game::ServerPlayer*","PID"},{0x5c,"std::string","FtName"},{0x78,"bool","Perm"}, + {0x7c,"Game::FleetLayout","Lay","HLay gate"},{0xa0,"Game::StarSystem*","LocID"},{0xa4,"std::vector","Ships","vector (NShips/ShipID/Ship)"},{0xc4,"Game::FlightPlan","FPlan","HFPlan gate = wpts non-empty"},{0xfc,"int","FtTrans"},{0x100,"Mars::Vector3","FtOrig"},{0x10c,"int","FtFlg"},{0x110,"int","Ftae"},{0x114,"int","Ftpae"},{0x118,"int","FtEnc"},{0x11c,"int","FtMS"}}); + // --- StarShip --- + fill("Game::StarShip", 0xb0, "Game::StarShip (NetworkObject@0, IStreamable@8). Serializer this = obj+8. Read FUN_00853fa0 / Write FUN_008291f0.", new Object[][]{ + {0,"ptr","vptr","0x00a31418"},{4,"int","netId"},{8,"ptr","vptr_IStreamable","0x00a31408"},{0x10,"Game::ServerPlayer*","PlrID"},{0x14,"Game::ShipDesign*","Des","DesID = design->+0xa4"},{0x20,"float","Range"},{0x24,"Game::ShipHealth","Health"},{0x34,"int","MineCap"},{0x38,"std::vector","TH","vector<{float th,thm}> (NTH/TH/THM)"}, + {0x48,"int","Plg"},{0x4c,"int","Act"},{0x50,"bool","Dep"},{0x51,"bool","Atq"},{0x5c,"int","LCT"},{0x60,"int","tsd"},{0x64,"Game::StarFleet*","FltID"},{0x68,"int","ConCap"},{0x6c,"float","RefCap"},{0x70,"float","RepCap"},{0x7c,"int","EncID"},{0x80,"Game::PrisonerHold","PrisH"},{0x98,"Game::BuildQueue*","BQ2","hbq gate"},{0x9c,"Game::Population*","pop","hsp gate"},{0xa0,"Game::Population*","ppop"},{0xa8,"int","atsp"},{0xac,"int","tblt"}}); + // --- StrategyServer (partial; from Write FUN_0079fa70 + spine work) --- + fill("Game::StrategyServer", 0x320, "Game::StrategyServer (IStreamable@0 = serializer this, primary vftable 0x00a26034 @+4). Offsets from Write FUN_0079fa70 and the turn-spine functions; PARTIAL.", new Object[][]{ + {0,"ptr","vptr_IStreamable","0x00a26084"},{4,"ptr","vptr","0x00a26034"},{8,"int","ModCount"},{0xc,"int","Frame","turn number; ++ in BeginProcessTurn"},{0x10,"int","GOTurn"},{0x14,"int","GameID"},{0x28,"std::string","GameName"}, + {0x44,"std::vector","Systems","vector (NumSys/SysID/Sys)"},{0x54,"std::vector","Players","vector (NumPlrs/PlayerID/Player)"},{0x64,"std::vector","Fleets","vector (NumFlts/FltID/Flt)"},{0x74,"std::vector","Acts","vector (NumActs/Act handle ids)"}, + {0x8c,"std::vector","NodeMapLines","stride 0x14; NMSz = count"},{0x9c,"int","NMLc"},{0xbc,"int","Map"},{0xc0,"float","IncMod"},{0xc4,"float","ResMod"},{0xc8,"bool","EnAl"},{0xc9,"bool","EnTm"},{0xfc,"float[7]","ISsu","per-species (ISsp name, ISsu value)"},{0x134,"std::string","KeyPath"}, + {0x154,"Game::ServerNodeGraph*","NdGr2"},{0x158,"Game::ServerTradeManager*","trdmgr"},{0x15c,"Game::IServerSpyManager*","spymgr"},{0x164,"Game::AttribMap*","Attrib"},{0x170,"pfn","OnEventCallback","(*cb)(playerNetId, int eventType, StrategyEvent** ev); set in ctor FUN_007d78d0 from ctor arg; NULL -> 'OnEvent() called, but no callback function specified'"}, + {0x1a0,"float","RandEncAdj"},{0x1b0,"ptr","listener","optional observer; vft+0x10(code,&args) called on player/fleet removal"},{0x1b4,"Game::SVScriptObject*","SvSctOb"},{0x1b8,"int","NPCm"},{0x1bc,"int","NPCo"},{0x1c0,"int","NPCi"},{0x1c4,"int","NPCv"},{0x1c8,"int","NPCa"},{0x1cc,"float","szadj"},{0x1d0,"float","rsadj"},{0x1d4,"float","suadj"},{0x1f8,"int","cmbtid"}, + {0x2b8,"std::list","StrategyEvents","list pending events (node: next,prev,vptr@8,..,targetId@0x10,id2@0x14)"},{0x2ec,"bool","bRecordEvents","gate for the per-player removal lists at 0x2d4/0x2e0"},{0x2f1,"bool","bProcessingTurn","set 1 in BeginProcessTurn"},{0x318,"std::map","zds","zdsc/zdsi/zdst"}}); + Structure sysV = serView("Game::ServerSystem", 8); Structure fltV = serView("Game::StarFleet", 8); Structure shpV = serView("Game::StarShip", 8); + + log.println("\n==== FUNCTION RENAMES: serializers ===="); + Object[][] ser = { + {0x0075d4b0L,"Game::ServerSystem::Read"},{0x00749630L,"Game::ServerSystem::Write"},{0x008804d0L,"Game::ServerPlayer::Read"},{0x008563e0L,"Game::ServerPlayer::Write"}, + {0x00702470L,"Game::StarFleet::Read"},{0x00701070L,"Game::StarFleet::Write"},{0x00853fa0L,"Game::StarShip::Read"},{0x008291f0L,"Game::StarShip::Write"}, + {0x007d27a0L,"Game::StrategyServer::Read"},{0x0079fa70L,"Game::StrategyServer::Write"},{0x00727790L,"Game::StarMapNode::Read"},{0x00727820L,"Game::StarMapNode::Write"}, + {0x00752af0L,"Game::StarSystem::PlayerView::Read"},{0x007492d0L,"Game::StarSystem::PlayerView::Write"},{0x007472a0L,"Game::StarSystem::OutputRates::Read"},{0x00745190L,"Game::StarSystem::OutputRates::Write"}, + {0x005390c0L,"Game::Population::Read"},{0x00537ef0L,"Game::Population::Write"},{0x00536a80L,"Game::PopulationGroup::Read"},{0x00536af0L,"Game::PopulationGroup::Write"}, + {0x00748df0L,"Game::IndependenceInfo::Read"},{0x00748ee0L,"Game::IndependenceInfo::Write"},{0x00744dd0L,"Game::Morale::Read"},{0x00744ea0L,"Game::Morale::Write"},{0x007490b0L,"Game::MoraleEvent::Read"},{0x007491b0L,"Game::MoraleEvent::Write"}, + {0x00813770L,"Game::ShipBuildOrder::Read"},{0x00813800L,"Game::ShipBuildOrder::Write"},{0x00813250L,"Game::PlayerNotes::Read"},{0x008132b0L,"Game::PlayerNotes::Write"},{0x008843d0L,"Game::SpyReport::Read"},{0x00828ec0L,"Game::SpyReport::Write"}, + {0x008200a0L,"Game::PlayerReport::Read"},{0x00817480L,"Game::PlayerReport::Write"},{0x00818cb0L,"Game::DiplomacyStats::Write"},{0x00704c70L,"Game::FlightPlan::Read"},{0x00700f60L,"Game::FlightPlan::Write"}, + {0x00701860L,"Game::FlightPlan::Waypoint::Read"},{0x00700ed0L,"Game::FlightPlan::Waypoint::Write"},{0x006e2260L,"Game::NodeRoute::Read"},{0x006e22e0L,"Game::NodeRoute::Write"},{0x0056eb00L,"Game::PrisonerHold::Read"},{0x0056ec00L,"Game::PrisonerHold::Write"}, + {0x00825cc0L,"Game::EventStorage::Write"},{0x006d2e10L,"Game::PlayerAlliances::Write"},{0x00813e50L,"Game::ShipHealth::Write"},{0x0053c080L,"Game::PlayerColorID::Write"},{0x008a60d0L,"Mars::Vector3::Write"},{0x005890a0L,"Game::TechTree::Write"},{0x008176a0L,"Game::ShipRecords::Write"},{0x0082c740L,"Game::CivilianRatios::Write"}}; + for (Object[] r : ser) fn((Long) r[0], (String) r[1], "IStreamable serializer (vftable slot [1]=Read, [2]=Write). Member table: " + DOC); + log.println("\n==== FUNCTION RENAMES: stream primitives ===="); + String P = "Stream primitive wrapper. Stream vftable slots: +0x10 ReadIntRef(name,&v)->found, +0x14 ReadNested(name,helper|NULL=skip), +0x18 String, +0x1c Bool, +0x20 Float, +0x24 Int(name,v,default=-1), +0x28 Nested(name,StreamableHelper*), +0x30 RawBytes(name,ptr,n). " + DOC; + Object[][] prims = {{0x008b9d70L,"Mars::Stream::WriteString","(Stream*, name, std::string*) -> vft+0x18; does the _Myres>=16 heap/SSO select"},{0x008b9c20L,"Mars::Stream::WriteBool","(Stream*, name, bool*) -> vft+0x1c"},{0x008b9be0L,"Mars::Stream::WriteFloat","(Stream*, name, float*) -> vft+0x20"}, + {0x008b9d50L,"Mars::Stream::WriteInt","(Stream*, name, int*) -> vft+0x24"},{0x008b9d00L,"Mars::Stream::WriteInt16AsInt","(Stream*, name, int16*) widened -> vft+0x24"},{0x008b9cb0L,"Mars::Stream::WriteInt8AsInt","(Stream*, name, int8*) -> vft+0x24"},{0x008b9c60L,"Mars::Stream::WriteInt64","(Stream*, name, int64*) -> vft+0x30 raw 8 bytes"}, + {0x00816490L,"Mars::Stream::WriteHandleId","(Stream*, name, NetworkObject*) writes obj ? obj->id(+4) : 0"},{0x008b9bc0L,"Mars::Stream::ReadFloat",""},{0x008b9d20L,"Mars::Stream::ReadInt",""},{0x008b9c00L,"Mars::Stream::ReadBool",""},{0x008b9d90L,"Mars::Stream::ReadString","vft+4 (name, buf, 0x400) then assigns std::string"}, + {0x008b9c40L,"Mars::Stream::ReadInt64",""},{0x008b9cd0L,"Mars::Stream::ReadInt16",""},{0x008164d0L,"Mars::Stream::ReadHandle","(Stream*, name) -> handle id -> object* lookup"}}; + for (Object[] r : prims) fn((Long) r[0], (String) r[1], ((String) r[2]) + "\n" + P); + + log.println("\n==== FUNCTION RENAMES: affinity / app / main loop ===="); + fn(0x0089ee70L, "Process_PinAffinity", "Pins the whole process to ONE logical CPU: SetProcessAffinityMask(GetCurrentProcess(), 1 << coreIndex(ESI)). Logs 'Limiting process affinity to CPU-%i'. Called from Mars::Application::Initialize when config CPU/ForceSingleCore > 0. Only affinity/topology API in the binary. Source: findings/objects/ghidra-recon.md"); + fn(0x008a0e50L, "Mars::Application::Initialize", "AppStartup_ReadConfig: reads CPU/ForceSingleCore (-> Process_PinAffinity), startup config via Mars::AppStartup parser, display.cfg, audio.cfg, COM, Direct3DCreate9, window (CreateAppWindow), DrawDevice, timers, spawns the streaming-sound update thread (SoundStreamingThreadProc, THREAD_PRIORITY_TIME_CRITICAL), GlobalConsts/textures/effects/sprites/keymap/font, SimplePainter, PanelManager, then IApplication::OnStartup() (vft+0x10 = Game::DemoApp::OnStartup). this = g_pApplication (DemoApp). Source: findings/objects/ghidra-recon.md + " + DOC2); + lbl(0x008a0e50L, "AppStartup_ReadConfig", null); + fn(0x0089dd30L, "WinMain", "WinMain(hInst, hPrev, lpCmdLine, nShow) called from ___tmainCRTStartup. Tokenises the command line, creates mutex Kerberos_SwordOfTheStars_Mutex (single instance unless /concurrent), new Game::DemoApp (0x1b8 bytes) -> g_pDemoApp, Mars::Application::Initialize, then Mars::Application::Run (main loop). " + DOC2); + fn(0x0089c950L, "Game::DemoApp::DemoApp", "DemoApp ctor (IApplication impl, vftable 0x00a36004). Calls Mars::Application ctor which sets g_pApplication."); + fn(0x008a0170L, "Mars::Application::Application", "Mars::Application ctor: reads app config, sets g_pApplication (DAT_00b2d540) = this."); + fn(0x0089f5b0L, "Mars::Application::Run", "MAIN LOOP. do { FrameTimer::Update; PanelManager->vft+0x60 (UI update); app->vft+0x18 OnUpdate (DemoApp::OnUpdate: network pump + game update); if (!focused && app->vft+0xc) Sleep(20); _controlfp; periodic timer-callback flush (TimerList::Dispatch) every _DAT_00a36860 s; ok = app->vft+0x1c OnTick (DemoApp::OnTick: sound/console/panel/state machine; returns 0 when quitting); if (ok && DrawDevice) { dev->BeginFrame; if (!dev->IsLost) { dev->+8; app->vft+0x20 OnRender; dev->+0xc; dev->Present } } } while (PumpMessages() && ok). " + DOC2); + fn(0x0089f1c0L, "Mars::Application::PumpMessages", "PeekMessage/TranslateMessage/DispatchMessage loop; returns 0 on WM_QUIT (0x12). Keyboard-translation gate: PanelManager(+0x78)->vft+0x58."); + fn(0x0090c700L, "Mars::FrameTimer::Update", "QueryPerformanceCounter-based frame timer: dt(+0x14), total(+0x18 double), fps(+0xc) every +0 ticks."); + fn(0x008e5ac0L, "Mars::TimerList::Dispatch", "walks a std::map of timers calling the +0x20 callback; used by the main loop at a fixed period."); + fn(0x0089fe70L, "Mars::Application::CreateAppWindow", "RegisterClass/CreateWindow/ShowWindow for Kerberos_SwordOfTheStars_WndCls."); + fn(0x0089f4d0L, "Mars::AppStartup::OnConfigToken", "AppStartup vftable[1]: startup-config token handler (key 'conlevel' -> console level FUN_008ba280; other keys -> app->+4->vft+0x24)."); + fn(0x008cd820L, "Mars::ConfigParser::ParseFile", "generic tokenising config-file parser with callback object (used for the startup config with Mars::AppStartup)."); + fn(0x0089d610L, "Game::DemoApp::OnStartup", "IApplication vft+0x10: version banner 'Sword of the Stars%s (%s %s)', profiles, Mars::Network::Startup (creates network watchdog thread), ParticleSystemManager, Mesh, Model, StringTable (Locale//Strings.csv), species, GUIResources, SoundSystem, UI sounds, SpeechEvents, GUIAppearances, PlayerColor/Badge/Avatar dictionaries, PlanetPainter, then main menu / '/join'. " + DOC2); + fn(0x0089dfb0L, "Game::DemoApp::OnShutdown", "IApplication vft+0x14: destroys all game/UI subsystems."); + fn(0x00898800L, "Game::DemoApp::OnUpdate", "IApplication vft+0x18 (called first each main-loop iteration): Mars::Network update (FUN_00902ad0), then current game object(+0x158)->vft+4 Update, then FUN_007879c0 if a strategy game (+0x150) exists."); + fn(0x0089a640L, "Game::DemoApp::OnTick", "IApplication vft+0x1c: sound/console/UI tick and the top-level game-state machine (loads/unloads the strategy game at +0x150 / combat at +0x158); returns 0 when quitting (+0x1b2)."); + fn(0x00899210L, "Game::DemoApp::OnRender", "IApplication vft+0x20: PanelManager->vft+0x64 draw, or the movie/splash player at +0x108."); + fn(0x008986a0L, "Game::DemoApp::ShouldSleepWhenInactive", "IApplication vft+0xc"); + + log.println("\n==== FUNCTION RENAMES: threads ===="); + fn(0x00902470L, "Mars::Network::Startup", "called from DemoApp::OnStartup; -> NetworkManager::Create"); + fn(0x00902350L, "Mars::NetworkManager::Create", "THREAD SITE 1 (0x0090242d): InitializeCriticalSection(g_netCS 0x00b2e5f8), new NetworkManager(0x150), CreateThread(NetworkWatchdogThreadProc). Globals: 0x00b2e61c manager, 0x00b2e620 thread handle, 0x00b2e618 running flag. " + DOC2); + fn(0x00901f40L, "Mars::NetworkManager::WatchdogThreadProc", "THREAD 1 body: loop { Sleep(10); Enter(g_netCS); pump host link(+0x20)/listener(+0x98); if host link timed out -> log 'Network(%f): No response from host %s' and drop; for each client link (+0x70 vector, stride 0xc) -> 'No response from client %s', collect & drop; Leave } until manager NULL. Pure timeout watchdog, not the game sim. " + DOC2); + fn(0x008fe730L, "Mars::NetworkManager::NetworkManager", "ctor (0x150 bytes)"); + fn(0x008ef1d0L, "Mars::SoundSystem::StreamingUpdateThreadProc", "THREAD 2 body (created in Mars::Application::Initialize at 0x008a14ef, priority 15): WaitForMultipleObjects(app+0x84 wake event, app+0x88 quit event); under g_musicCS (0x00b2e4a8): if current music stream (0x00b2e4c4) finished/faded (+0x388 float <= 0) -> stop it and open the next queued music file (MusicPlayer::OpenMusicFile from the 0x00b2e4d0 list); else StreamingSound::FillBuffer; then FUN_008b65d0 (sound system update). Streams music/DirectSound buffers only. " + DOC2); + fn(0x0091b5a0L, "Mars::StreamingSound::FillBuffer", "DirectSound streaming buffer refill (GetCurrentPosition / Lock / decode / Unlock); 'Couldn't restore buffer'."); + fn(0x008ef040L, "Mars::MusicPlayer::OpenMusicFile", "opens a music file (FUN_008b66e0) as the current stream 0x00b2e4c4; '[%s] cannot open music file for playback'."); + fn(0x00736e30L, "Game::BackgroundWorker::Start", "THREAD SITE 3 (0x00736e84): embedded struct {CRITICAL_SECTION cs@0; HANDLE thread@0x18; ...; job* @0x54; flags @0x58..0x5c (0x5b = quit request, 0x5c = exited)}; InitializeCriticalSection + CreateThread(BackgroundWorker::ThreadProc, this). Owner: StarMapPanelBase (strategic star-map renderer). " + DOC2); + fn(0x00735bb0L, "Game::BackgroundWorker::ThreadProc", "THREAD 3 body: loop { Sleep(10); Enter(cs); if quit(+0x5b) {set exited(+0x5c); Leave; break}; Leave; Enter; job = (+0x54 && !+0x59) ? +0x54 : NULL; Leave; if (job) { StarMapBlobs::BuildBlobMesh_Job(job+0x3c, job+4); Enter; +0x58=0,+0x59=1 (done); Leave } }. Background political-map 'blob' (territory overlay) mesh builder for the star map; NOT on the battle path. " + DOC2); + fn(0x00732ab0L, "Game::StarMapBlobs::BuildBlobMesh_Job", "worker job: sums point cloud (job+0x28 vector), centroid/spread (sqrt), computes a blob radius (job+0x14) then FUN_008fc160 (implicit-surface / metaball polygoniser with callbacks FUN_00722010/FUN_0071ea60) and FUN_008fa5b0 (mesh build) into the output vertex list (param_1). Used by the political map overlay (Render/StarMapBlobs_*.fx)."); + fn(0x00741cd0L, "Game::StarMapPanelBase::StarMapPanelBase", "ctor of the strategic star-map view base class (Render/StarMapBlobs_Solid.fx, StarMapBlobs_Glow.fx, POLMAP_* colours, Skysphere); first statement starts the BackgroundWorker thread (this embedded at +0). Derived: Game::PoliticalMapPanel (ctor FUN_007424b0), created by the strategy-map screen ctor FUN_005e9780."); + + log.println("\n==== FUNCTION RENAMES: game creation / strategy server ===="); + fn(0x00898b00L, "Game::DemoApp::CreateStrategyGame", "-> StrategyApp::CreateGame"); + fn(0x00888e80L, "Game::StrategyApp::CreateGame", "reads GameOptions (AIProcessMinTime, DefaultAutoRefuel), Data/Strategy/starcolors.txt, TurnCommands_v5 stream tag, builds the StrategyServer (ctor FUN_007d78d0) + StrategyServer::InitGame, loads (FUN_007dd530 'loaded from file') and the star-map UI (FUN_00778f40 -> ... -> StarMapPanelBase)."); + fn(0x007d78d0L, "Game::StrategyServer::StrategyServer", "ctor (vftables 0x00a26084 IStreamable @0, 0x00a26034 @4). Stores the OnEvent callback at +0x170 from a ctor argument (EBX)."); + fn(0x007c8d90L, "Game::StrategyServer::InitGame", "raises SEInitGame / SEAddDesign for every player, then SynchronizePlayer for each."); + fn(0x007c6220L, "Game::StrategyServer::SynchronizePlayer", "(playerNetId, bool full, bool, byte): pushes the server-side view (systems/fleets/designs/notes/...) to one player's client via the OnEvent callback (SE* events). Logs 'Can't synchronize player %d(id)'. Called by SyncLocalClients, InitGame, GenerateTurnEvents."); + fn(0x00815fd0L, "Game::StrategyApp::SyncLocalClients", "for each local client (+0xc vector): StrategyServer::SynchronizePlayer(client->+0x148 (player id), arg, 0, 0). 'SyncLocalClients() failed, no StrategyServer exists'."); + fn(0x007dd530L, "Game::StrategyServer::LoadGame", "'loaded from file'"); + fn(0x008d2290L, "Mars::NetMessageRegistry::Register", "(this=registry entry {name@0,id@4,factory@8}, name, id, factory); g_NetMsgRegistryById[id] = entry (0x00b2ddf8, 256 slots). Called from static initialisers in .text 0x009be0cc..0x009c138c (unanalysed code). Entry(id) = 0x00b2bc94 + 12*id for the SNM* family."); + lbl(0x00b2ddf8L, "g_NetMsgRegistryById", "NetMessageRegistry entry* [256], indexed by message id"); + lbl(0x00b2d540L, "g_pApplication", "Mars::Application* (the DemoApp)"); + lbl(0x00b2d0bcL, "g_pDemoApp", "Game::DemoApp* created in WinMain"); + lbl(0x00b2e61cL, "g_pNetworkManager", null); lbl(0x00b2e620L, "g_hNetworkWatchdogThread", null); lbl(0x00b2e5f8L, "g_netCS", "CRITICAL_SECTION guarding the network manager"); + lbl(0x00b2e4a8L, "g_musicCS", "CRITICAL_SECTION guarding the streaming music player"); lbl(0x00b2e4c4L, "g_pCurrentMusicStream", null); + + log.println("\n==== FUNCTION RENAMES: turn spine ===="); + fn(0x00784640L, "Game::StrategyNetworkClient::OnMessage", "NETWORK MESSAGE DISPATCH (client AND host side; host checks +0x54 = StrategyServer*). Compares msg->GetType()->id with the registry entries: 0x3f SNMEndTurn -> host: StrategyServer::OnPlayerEndTurn + StorePlayerTurnCommands; 0x29 SNMUpdate (all players' TurnCommands) -> BeginProcessTurn, ApplyTurnCommands, ProcessTurn (deterministic local sim on EVERY machine), SyncLocalClients(1), state=5; 0x3c SNMDoEncounterQuery -> state=4; 0x2f SNMAllCombatDone -> ApplyEncounterResults (-> SETurnResults), SyncLocalClients(0), GenerateTurnEvents (-> SETurnEvents), autosave, state=6 'New turn begins'; 0x43 SNMResumePlaying -> StrategyServer::ResumePlaying (SEResumePlaying), reply SNMResumePlayingReceived, state=1; 0x3d SNMRunAI -> StrategyApp::RunAI; 0x32 SNMSetPlayerStatus -> player->Status(+0x164). " + DOC2); + fn(0x00783be0L, "Game::StrategyClient::EndTurn", "CLIENT End Turn: marks +0x57/+0x132, records QPC time, EndTurnDelay, raises SETurnEndPending (event type 0x21) via RaiseEvent, then SendEndTurn (SNMEndTurn). Callers: UI (FUN_005e4f80, FUN_00579310)."); + fn(0x00783d30L, "Game::StrategyClient::EndTurnForced", "variant used by StrategyClient::Update (turn timer) and FUN_00783ee0; raises SETurnEndPending + SNMEndTurn."); + fn(0x00783980L, "Game::StrategyClient::SendEndTurn", "builds SNMEndTurn (TurnCommands + AIEncounterFlags) and sends it to the host."); + fn(0x007856f0L, "Game::StrategyClient::CancelEndTurn", "raises SETurnEndCancelled."); + fn(0x007842b0L, "Game::StrategyClient::Update", "per-frame client update: strategy turn timer -> SETurnTimeExpired / EndTurnForced / SNMQueryEndTurnDone; combat join/launch bookkeeping ('All(%d) clients connected to combat server', 'Launching combat for encounter %d')."); + fn(0x00783ee0L, "Game::StrategyClient::RaiseEvent", "(int type, StrategyEvent** ev) -> client-side event sink (UI)."); + fn(0x0088c7b0L, "Game::SNMEndTurn::Create", "registry factory for message id 0x3f (object 0x1cc bytes: TurnCommands @+4, AIEncounterFlags @+0x1b8)."); + fn(0x007d9af0L, "Game::StrategyServer::OnPlayerEndTurn", "host: on SNMEndTurn from a player; if >1 human still playing and exactly one not done -> raise SELastPlaying (0x27) to that player via OnEventCallback."); + fn(0x007893c0L, "Game::StrategyServer::StorePlayerTurnCommands", "host: keeps the player's TurnCommands until SNMUpdate is broadcast."); + fn(0x00789710L, "Game::StrategyServer::BroadcastEvent", "(int eventType, StrategyEvent** ev): for every player in Players -> OnEventCallback(player->netId, type, ev). 'OnEvent() called, but no callback function specified' if NULL."); + fn(0x007d98e0L, "Game::StrategyServer::BeginProcessTurn", "Frame(+0xc)++, bProcessingTurn(+0x2f1)=1, log 'Begin processing turn %d', clears per-system/per-fleet transient state (fleets: +0x114=+0x110; +0x110=0), BroadcastEvent(0x24 = SEProcessTurn). " + DOC2); + fn(0x007b18b0L, "Game::StrategyServer::ApplyTurnCommands", "applies every player's TurnCommands from SNMUpdate ('set for turn processing'): fleet orders, builds, research, alliances (EVENT_ALLIANCE_*), diplomacy."); + fn(0x007dc6c0L, "Game::StrategyServer::ProcessTurn", "TURN PROCESSING (arg = 1.0f time step). Phases in order: [1] per-system pre-turn (SESystemAbandoned / morale events); [2] FUN_0086b300 + FUN_007adc80 (alliance/diplomacy upkeep); [3] per-player pre-pass; [4] fleet snapshot (FUN_00794ad0/FUN_007b9b90), node-space travel (ProcessNodeSpaceTravel: EVENT_LOSTINNODESPACE_*), MOVEMENT ProcessFleetMovement -> MoveFleet -> SEFleetArrived; [5] per-fleet per-ship FUN_00814ea0 (ship upkeep); [6] per-system ServerSystem::ProcessTurn (pop/morale, BUILD queue -> BuildQueue::ProcessTurn -> SEBuildCompleted, plague, rebellion, slaves); [7] FUN_0078a7c0; [8] per-player ServerPlayer::ProcessTurn (income/savings, RESEARCH: TechTree::ProcessResearch, lab accidents, EVENT_NO_RESEARCH); [9] ProcessMissions, ProcessStations (EVENT_STATIONS_SCUTTLED), ProcessDefenceSats (EVENT_DEFSATS_SCUTTLED); [10] per-ship flag pass (FUN_00814da0 4 / 0x400000); [11] encounter detection (FUN_00794ad0 second snapshot -> local_98 != 0 means encounters pending); if NO encounters: ProcessAid (EVENT_GIVE_*), ProcessSpecialProjects (EVENT_SPRJTECHOFFER_STARTED), ProcessSurrenders (EVENT_PLAYER_SURRENDERED_/EVENT_SYSTEM_SURRENDERED), per-player FUN_00818530, FUN_0086a8d0, FUN_0078ab30, FUN_00799380, FUN_0078aa70, per-system FUN_00743ec0, FUN_007b4c00, FUN_007d7f70 (end-of-turn bookkeeping over the per-player 0x74-byte records). Turn RESULTS/EVENTS are raised later from the SNMAllCombatDone handler (ApplyEncounterResults -> SETurnResults; GenerateTurnEvents -> SETurnEvents). " + DOC2); + fn(0x007da9a0L, "Game::StrategyServer::ProcessFleetMovement", "iterates fleets with flight plans, calls MoveFleet(fleet, dt) (several passes: normal, in-transit, arrival), then OnFleetArrived (EVENT_FLEET_ARRIVED)."); + fn(0x007d9ee0L, "Game::StrategyServer::MoveFleet", "(StarFleet*, float dt): advances a fleet along its FlightPlan; on arrival raises SEFleetArrived; 'Destination of fleet doesn't exist. Stopping fleet.'; cancels ship actions on departure."); + fn(0x007ccb10L, "Game::StrategyServer::OnFleetArrived", "EVENT_FLEET_ARRIVED bookkeeping"); + fn(0x007a0e20L, "Game::StrategyServer::ProcessNodeSpaceTravel", "EVENT_FLEET_MULTIPOINT_NONODE / EVENT_LOSTINNODESPACE_NOBORE / _ENGINES"); + fn(0x007598e0L, "Game::ServerSystem::ProcessTurn", "per-system turn: population/infrastructure growth, morale (FUN_00752a10), ProcessPlague, ProcessBuildQueue, terraforming/resources, slaves (ProcessSlaves, EVENT_SLAVES_DEAD), rebellion (ProcessRebellion, EVENT_SYSTEM_REBELLION_CONTINUES)."); + fn(0x00752500L, "Game::ServerSystem::ProcessBuildQueue", "-> BuildQueue::ProcessTurn"); + fn(0x00890d50L, "Game::BuildQueue::ProcessTurn", "advances ShipBuildOrders; completed orders -> SEBuildCompleted (with ShipBuildOrderDef). Callers: ServerSystem::ProcessBuildQueue, FUN_00789500 (ship-borne build queues)."); + fn(0x00756a90L, "Game::ServerSystem::ProcessPlague", "EVENT_PLAGUE_OUTBREAK / EVENT_COLONY_DESTROYEDBYPLAGUE / EVENT_PLAGUE_CURED"); + fn(0x007583b0L, "Game::ServerSystem::ProcessRebellion", "EVENT_SYSTEM_REBELLION_CONTINUES"); + fn(0x007537b0L, "Game::ServerSystem::ProcessSlaves", "EVENT_SLAVES_DEAD"); + fn(0x00891340L, "Game::ServerPlayer::ProcessTurn", "per-player turn: savings/income (FUN_00840fe0), research: if ResT set -> RollResearchAccident then TechTree::ProcessResearch (EVENT_RESEARCH_OVERBUDGET / EVENT_TECHS_UNLOCKED) else EVENT_NO_RESEARCH; special projects (FUN_00863cf0)."); + fn(0x005876c0L, "Game::TechTree::ProcessResearch", "EVENT_RESEARCH_OVERBUDGET / EVENT_TECHS_UNLOCKED"); + fn(0x00889dc0L, "Game::ServerPlayer::RollResearchAccident", "'ACCIDENT!!' / 'All okay.' EVENT_LABACCIDENT_SMALL/MEDIUM/LARGE"); + fn(0x007ad100L, "Game::StrategyServer::ProcessAid", "EVENT_GIVE_SAVINGS / EVENT_GIVE_RESEARCH"); + fn(0x007a3310L, "Game::StrategyServer::ProcessSpecialProjects", "EVENT_SPRJTECHOFFER_STARTED"); + fn(0x007d0d10L, "Game::StrategyServer::ProcessSurrenders", "EVENT_PLAYER_SURRENDERED_ / EVENT_SYSTEM_SURRENDERED"); + fn(0x007af0b0L, "Game::StrategyServer::ProcessDefenceSats", "EVENT_DEFSATS_SCUTTLED"); + fn(0x007ae480L, "Game::StrategyServer::ProcessStations", "EVENT_STATIONS_SCUTTLED"); + fn(0x007999a0L, "Game::StrategyServer::ProcessMissions", "'mission'"); + fn(0x007cbe80L, "Game::StrategyNetworkServer::RunCombatRound", "host combat round: for each pending encounter -> SendEncounterQuery (SNMDoEncounterQuery) / 'Notifying %s to host encounter %d' (SNMHostCombat) ...; when all encounters resolved -> SNMAllCombatDone 'All combat complete, waiting for clients to process results'."); + fn(0x007cda40L, "Game::StrategyNetworkServer::Update", "host per-frame: lobby/slot messages, combat round driving (RunCombatRound), SNMTurnInfo, SNMLaunchCombat, SNMResumePlaying (SendResumePlaying)."); + fn(0x007bfe60L, "Game::StrategyNetworkServer::SendEncounterQuery", "builds SNMDoEncounterQuery"); + fn(0x00794770L, "Game::StrategyNetworkServer::SendResumePlaying", "builds SNMResumePlaying"); + fn(0x007d4400L, "Game::StrategyServer::ApplyEncounterResults", "after SNMAllCombatDone: applies combat outcomes (EVENT_STATION_ENABLED/DISABLED ...) then DispatchTurnResults -> SETurnResults per player."); + fn(0x007cd2a0L, "Game::StrategyServer::DispatchTurnResults", "-> SendTurnResultsToPlayers"); + fn(0x007c5850L, "Game::StrategyServer::SendTurnResultsToPlayers", "(perPlayerResults[0x11c stride], n): per player: SETurnResults::Create, fill (FUN_007c24d0), dispatch (FUN_0079ac10)."); + fn(0x007a7ae0L, "Game::SETurnResults::Create", ""); + fn(0x007dc640L, "Game::StrategyServer::GenerateTurnEvents", "-> BuildTurnEvents (SETurnEvents per player) + FUN_00792a20/FUN_007c5610."); + fn(0x007db780L, "Game::StrategyServer::BuildTurnEvents", "builds the per-player SETurnEvents (EventStorage::TurnEvents) from the turn's event log; also SEResetMap/SEAddPlayer/SEInitTrade/SESyncDesign; calls SynchronizePlayer."); + fn(0x007ddc90L, "Game::StrategyServer::ResumePlaying", "on SNMResumePlaying: for every player with Status(+0x164)==0 -> OnEventCallback(netId, 0x26 = SEResumePlaying)."); + fn(0x008706f0L, "Game::StrategyApp::RunAI", "on SNMRunAI: 'RunAI: No StrategyServer created' guard; runs the AI turn (SEAIPrepareTurn via FUN_00815f20) for AI players."); + fn(0x00815f20L, "Game::StrategyApp::RaiseAIPrepareTurn", "raises SEAIPrepareTurn"); + setEOLComment(toAddr(0x0090242dL), "CreateThread -> Mars::NetworkManager::WatchdogThreadProc (thread 1: network timeout watchdog)"); + setEOLComment(toAddr(0x008a14efL), "CreateThread -> Mars::SoundSystem::StreamingUpdateThreadProc (thread 2: music/DirectSound streaming, priority 15)"); + setEOLComment(toAddr(0x00736e84L), "CreateThread -> Game::BackgroundWorker::ThreadProc (thread 3: star-map political-blob mesh builder)"); + + log.println("\n==== NET MESSAGE REGISTRY LABELS ===="); + Address lo = toAddr(0x009b0000L), hi = toAddr(0x009c8000L); byte[] buf = new byte[(int)(hi.subtract(lo))]; mem.getBytes(lo, buf); int n = 0; + for (int i = 0; i + 10 < buf.length; i++) { + if ((buf[i]&0xff) != 0xB9 || (buf[i+5]&0xff) != 0xE8) continue; + long rel = (buf[i+6]&0xffL) | ((buf[i+7]&0xffL)<<8) | ((buf[i+8]&0xffL)<<16) | ((buf[i+9]&0xffL)<<24); + if (lo.getOffset() + i + 10 + (int)rel != 0x008d2290L) continue; + long entry = (buf[i+1]&0xffL) | ((buf[i+2]&0xffL)<<8) | ((buf[i+3]&0xffL)<<16) | ((buf[i+4]&0xffL)<<24); + int p = i - 5; if ((buf[p]&0xff) != 0x68) continue; + long name = (buf[p+1]&0xffL) | ((buf[p+2]&0xffL)<<8) | ((buf[p+3]&0xffL)<<16) | ((buf[p+4]&0xffL)<<24); + long id; int q; if ((buf[p-2]&0xff) == 0x6a) { id = buf[p-1]&0xff; q = p-2; } else if ((buf[p-5]&0xff) == 0x68) { id = (buf[p-4]&0xffL) | ((buf[p-3]&0xffL)<<8) | ((buf[p-2]&0xffL)<<16) | ((buf[p-1]&0xffL)<<24); q = p-5; } else continue; + long factory = (buf[q-5]&0xff) == 0x68 ? ((buf[q-4]&0xffL) | ((buf[q-3]&0xffL)<<8) | ((buf[q-2]&0xffL)<<16) | ((buf[q-1]&0xffL)<<24)) : 0; + String nm = cstr(toAddr(name), 64); if (nm == null) continue; + try { createLabel(toAddr(entry), "NetMsgReg_" + nm, true, SourceType.USER_DEFINED); setEOLComment(toAddr(entry), String.format("NetMessageRegistry entry: name=%s id=0x%02x factory=0x%08x (registered at 0x%08x)", nm, id, factory, lo.getOffset() + i)); } catch (Exception e) {} + Function ff = getFunctionAt(toAddr(factory)); + if (ff != null && ff.getName().startsWith("FUN_")) { try { ff.setParentNamespace(ns("Game::" + nm)); ff.setName("Create", SourceType.USER_DEFINED); ff.setComment(String.format("NetMessage factory for %s (id 0x%02x); registered via Mars::NetMessageRegistry::Register", nm, id)); } catch (Exception e) { log.println(" !! factory " + nm + ": " + e.getMessage()); } } + n++; + } + log.println(" labelled " + n + " registry entries"); + + log.println("\n==== VERIFY ===="); + decomp = new DecompInterface(); decomp.openProgram(currentProgram); + retypeThis(0x00749630L, new PointerDataType(sysV)); retypeThis(0x0075d4b0L, new PointerDataType(sysV)); + retypeThis(0x00701070L, new PointerDataType(fltV)); retypeThis(0x00702470L, new PointerDataType(fltV)); + retypeThis(0x008291f0L, new PointerDataType(shpV)); retypeThis(0x00853fa0L, new PointerDataType(shpV)); + retypeThis(0x0079fa70L, new PointerDataType(srv)); retypeThis(0x007d27a0L, new PointerDataType(srv)); + retypeThis(0x007d98e0L, new PointerDataType(srv)); retypeThis(0x007dc6c0L, new PointerDataType(srv)); retypeThis(0x00789710L, new PointerDataType(srv)); retypeThis(0x007d9af0L, new PointerDataType(srv)); retypeThis(0x007598e0L, new PointerDataType(sys)); retypeThis(0x00891340L, new PointerDataType(ply)); + verify(0x00749630L, "verify_ServerSystem_Write.c"); verify(0x0079fa70L, "verify_StrategyServer_Write.c"); verify(0x00701070L, "verify_StarFleet_Write.c"); verify(0x007d98e0L, "verify_BeginProcessTurn.c"); verify(0x007dc6c0L, "verify_ProcessTurn.c"); + decomp.dispose(); log.close(); println("writeback done"); + } +} diff --git a/save-reader/SAVE_FORMAT.md b/save-reader/SAVE_FORMAT.md new file mode 100644 index 0000000..9f9b44f --- /dev/null +++ b/save-reader/SAVE_FORMAT.md @@ -0,0 +1,169 @@ +# SOTS1 save format — as implemented by `save_reader.py` + +Status: **unvalidated against a real save.** Everything below is what the +reader *assumes*, derived from the community editors (R1 Bardez, R2 SOTSedit; +`save-editor-structs.md`) corrected by the binary member tables +(`struct-recovery.md`). The synthetic fixture (`save_writer_stub.py`) round-trips +under these assumptions; a real `.sav` is the first real test. Items marked +**VERIFY** are the ones the verifier should diff against the binary first. + +## 1. Container + +| | | +|---|---| +| File | one gzip member (`1f 8b`), standard deflate. `gzip.decompress` | +| Inflated stream | parsed from offset 0; all offsets in reader output are inflated offsets | +| Byte order | little-endian everywhere | +| Text | windows-1252, no NUL terminator | +| Non-gzip input | accepted as an already-inflated stream (R1's `*.sav.inflate.dat`) | + +## 2. Primitive encodings + +``` +tag := int32 len, len bytes ASCII (field name; may be "." or "") +string := int32 len, len bytes cp1252 +int := 4 bytes (int32) handle ids, int16 members, enums, counts +float := 4 bytes IEEE-754 single +bool := 1 byte, 0/1 +int64 := 8 bytes Bats2, rcex, PopC (raw-bytes write path) +raw := n bytes, length known only to the writer RNG state (2500 B = MT19937 624 words + index) +``` + +A **named value** is `tag value pad`, where `pad` is NUL bytes bringing the item +to a 4-byte boundary. The reader implements two padding conventions and +auto-detects which one the file uses (`--padding auto`, default): + +| mode | layout | item size | +|---|---|---| +| `joint` (default, R1's `PaddingSize=4` over `4+len(name)+len(value)`) | `[len][name][value][pad]` | `pad4(4 + len(name) + len(value))` | +| `split` | `[len][name][pad][value][pad]` | `pad4(4 + len(name)) + pad4(len(value))` | + +The two differ only when `len(name) % 4 != 0` **and** the value is not a +multiple of 4 bytes (bools, odd-length strings): e.g. `NPC`+bool is 8 bytes in +`joint`, 12 in `split`. **VERIFY (highest priority):** find a 3-char bool tag +(`NPC`, `Dep`, `Atq`, `Srn`, `cta`, `hgs`, `vnh`, `hbq`, `hsp`) in the real +inflated stream and check whether the value byte immediately follows the name. +Reader assumption when writing this: `joint`. + +Values carry **no type byte**. Type comes from (a) the schema/catalog, (b) a +lookahead plausibility test (a candidate layout is accepted only if a plausible +tag, a frame marker, or EOF follows), (c) for 4-byte words with no hint, an +int/float classification by bit pattern (`|int| <= 100000` → int; finite float +with `1e-6 <= |f| < 1e12` → float; else int). The other reading is kept as `alt`. +Limitation: with a name whose length is a multiple of 4, `bool` and `int` have +identical item sizes; unknown names default to `int`. + +## 3. Complex frames + +``` +[len][name][pad to 4] BE EF BE EF ...children... 10 41 10 41 + (0xBEEFBEEF) (0x41104110 = ~0xBEEFBEEF) +``` + +* Every "nested object" (IStreamable via stream vft+0x28) is framed. Frames + nest; the reader recovers the tree from the markers alone, without a schema. +* A frame may be **tagless** (BEEFBEEF directly at an item position): the reader + accepts it; not observed in the reference material. **VERIFY** if seen. +* A frame body may be **untagged bytes**: `Mars::Vector3` (`Pos`, `PrvPos`, + `FtOrig`, `FPogn2`, `FPdpos`) and `ShipHealth` (`Health`) are "3 unnamed + floats" per the binary. Reader accepts either 12 raw bytes before the END + marker or three tagged floats (any tag, e.g. `"."`). **VERIFY** which. +* `RNG` frame body is treated as opaque bytes up to its END marker. +* Markers are only interpreted at item boundaries; a value that happens to equal + a marker is not a problem unless the reader is already resynchronising. + +## 4. Arrays + +| kind | encoding | reader schema | +|---|---|---| +| NonComplexArray (`VectorHelper`, `std::list`) | named int count, then count × element **inline in the same frame** | `NArr` | +| ComplexArray | a frame containing: int count, then count × element | `CArr` | +| element = leaf wrapper | e.g. `SysID` int + `Sys` frame; `PlayerID` + `Player`; `FltID` + `Flt`; `ShipID` + `Ship`; `DesID` + `Des` | `Seq([...])` | +| sparse tables | `mnsp` then n × (`msp` index, `mv` value); `nadct` then n × (`ads`, `adt`); `haltc` then n × (`haltt`, `haltv`) | `NArr(Seq)` | + +Count tags used by the binary: `NumPlrs NumSys NumFlts NShips NumActs NumOwn NumDes +NumLeg NumNotes NumPR NSprj Nexp NWeapXcl ndeflay rdtc numcreps ninv NumFlts NumGFs +NumSnF NumMnF NVO NVE NVs NumPlgs2 PopNG mnsp nadct haltc PrNSp NTH`. Count tags +inside framed arrays (`dipstats`, `preps`, `odes`, `owep`, `otch`, `cme2`, `Ojvs`…) +are unknown — the reader takes the first child as the count whatever its name. +Element frame tags are unknown (R1 suggests `"."`) — matched by position. + +## 5. Conditionals the reader honours + +| gate | consequence | +|---|---| +| `ClrID`/`indcl`/`fxCrId` frame: int index == -1 | three ints r,g,b follow inside the frame | +| `vnh` true | `vnd`, `vnex3`, `vnpex3` | +| `hindi` true | `indi` frame (IndependenceInfo) | +| `NVO` entry `isind` true | `indi` frame | +| `HFPlan` true | `FPlan` frame | +| `HLay` true | `Lay` frame (opaque) | +| `hbq` true | `BQ2` frame; `hsp` true → `pop`, `ppop` Population frames | +| `HasAIR` true | `AIR` frame (opaque) | +| system `PID` non-null | `BQ` frame present (reader: optional by name) | +| legacy tags `ISuit ARes SysID TrdID Caps GtTrf FtSens FtInc lcid SensMod ExPopSys AIDifficultyID` | accepted if present, expected absent in 1.8 saves | + +## 6. Top-level layout + +``` +offset 0: [7]"Summary" [pad] BEEFBEEF ... 41104110 (R2 confirms the tag "Summary") + CreateParameters frame (tag unknown) + Sim frame (tag unknown) KeyPath NMSz NMLc NMnx ModCount Frame + GameID Attrib RNG GameName Map IncMod ResMod EnAl + EnTm GOTurn GOWinPly NPCm NPCo NPCi NPCv NPCa + szadj rsadj suadj sprjs RandEncAdj cmbtid turnstats + numcreps/crep ninv NumPlrs/PlayerID/Player + NumSys/SysID/Sys NdGr2 trdmgr spymgr + NumFlts/FltID/Flt NumActs/Act SvSctOb zdsc zdsi zdst + CdTable UNFRAMED at root: cdt frame, cdplayer frame, N × cdai frames +``` + +Regions in `<...>` are parsed generically and kept as raw item lists +(`sim.idLists`, `sim.invasionsAndExclusions`, `sim.species`). Everything not +covered by a shape (TechTree body, Events, ShipRecs, spy2, civr, comms, Ojvs, +SvSctOb, trdmgr, spymgr, CdTable, …) is kept as the generic +`{"_name","_off","_items":[{"name","kind","value","off"}...]}` form. + +## 7. Struct field orders and types applied + +Shapes live in `save_reader.py` (`Sys`, `Player`, `Fleet`, `Ship`, `PlayerView`, +`Population`/`PopG`, `Morale`, `MoraleEvent`, `BuildQueue`/`BuildOrder`, +`IndependenceInfo`, `Rts`, `DipStat`, `Prep`, `FlightPlan`, `Waypoint`, +`PrisonerHold`, `Summary`, `CreateParams`, …). `A("tag", type)` = on-disk tag +confirmed in the exe (strict name match); `R("name", type)` = R1 C# name only +(positional). Binary corrections applied over R1: + +* float: `TRM CstR CstE CstT shrm RefCap RepCap`, PlayerView `Infra` +* int64: `Bats2 rcex PopC` +* bool: `Abdn Dstyd PvMA AIBn` (R2's "short" readings are the value byte) +* int on disk though int16 in memory: `TShn ETS` and all DiplomacyStats counters +* string: `pswd` +* Vector3 (3 floats): `FtOrig` +* `Nexp` entries carry `xid xmin xmax xper(float)` +* `Team` appears twice in Player: an int, later a nested `{ALid AL NA CF}` frame + (typed key `Alliances`) + +## 8. Reader output conventions + +* `--dump`: one line per item, `@ name kind value`; `?` after + the kind means the type was guessed, `(alt …)` shows the other reading of a + 4-byte word; frames print `{` … `}` with item count and byte size. +* `--json`: `{"padding", "stats", "issues": [...], "data": {...}}`; every typed + struct carries `_off`; unexpected items are kept under `_unexpected` / + `_extra`; unknown regions keep the generic form. +* Issue levels: `error` (schema field missing / type impossible / frame + unterminated), `warn` (resync, hint not plausible, unexpected items, + width mismatch, best-effort read at EOF), `info` (unnamed small payload + before END, positional tag name differing from the R1 name). `--strict` + fails on error or warn. +* Exit status: 0 clean, 1 errors present, 2 unreadable container. + +## 9. What the verifier should check first on a real save + +1. Padding convention (§2) — a 3-char bool tag settles it. +2. The `Summary` frame's 13 children and their actual tag names (R1 names are + probably case-variants: `NumSys` is confirmed by R2 as a tag string). +3. Whether Vector3 bodies are tagged (§3). +4. The tag used for element frames and for counts inside framed arrays (§4). +5. Whether the reader's auto-detected padding, `resyncs == 0` and + `hint_failures == 0` hold; any resync offset points at a layout gap. diff --git a/save-reader/__pycache__/save_reader.cpython-310.pyc b/save-reader/__pycache__/save_reader.cpython-310.pyc new file mode 100644 index 0000000..778a9ce Binary files /dev/null and b/save-reader/__pycache__/save_reader.cpython-310.pyc differ diff --git a/save-reader/__pycache__/save_writer_stub.cpython-310.pyc b/save-reader/__pycache__/save_writer_stub.cpython-310.pyc new file mode 100644 index 0000000..c66e26a Binary files /dev/null and b/save-reader/__pycache__/save_writer_stub.cpython-310.pyc differ diff --git a/save-reader/__pycache__/test_save_reader.cpython-310.pyc b/save-reader/__pycache__/test_save_reader.cpython-310.pyc new file mode 100644 index 0000000..cbf37ca Binary files /dev/null and b/save-reader/__pycache__/test_save_reader.cpython-310.pyc differ diff --git a/save-reader/fixture_joint.sav b/save-reader/fixture_joint.sav new file mode 100644 index 0000000..274c761 Binary files /dev/null and b/save-reader/fixture_joint.sav differ diff --git a/save-reader/fixture_split.sav b/save-reader/fixture_split.sav new file mode 100644 index 0000000..14b8b51 Binary files /dev/null and b/save-reader/fixture_split.sav differ diff --git a/save-reader/ref/parsers/__pycache__/effect_txt.cpython-310.pyc b/save-reader/ref/parsers/__pycache__/effect_txt.cpython-310.pyc new file mode 100644 index 0000000..dc1e5ed Binary files /dev/null and b/save-reader/ref/parsers/__pycache__/effect_txt.cpython-310.pyc differ diff --git a/save-reader/ref/parsers/__pycache__/flat_kv.cpython-310.pyc b/save-reader/ref/parsers/__pycache__/flat_kv.cpython-310.pyc new file mode 100644 index 0000000..bbe3d5c Binary files /dev/null and b/save-reader/ref/parsers/__pycache__/flat_kv.cpython-310.pyc differ diff --git a/save-reader/ref/parsers/__pycache__/manifest.cpython-310.pyc b/save-reader/ref/parsers/__pycache__/manifest.cpython-310.pyc new file mode 100644 index 0000000..643da22 Binary files /dev/null and b/save-reader/ref/parsers/__pycache__/manifest.cpython-310.pyc differ diff --git a/save-reader/ref/parsers/__pycache__/mars_data.cpython-310.pyc b/save-reader/ref/parsers/__pycache__/mars_data.cpython-310.pyc new file mode 100644 index 0000000..f80f5ef Binary files /dev/null and b/save-reader/ref/parsers/__pycache__/mars_data.cpython-310.pyc differ diff --git a/save-reader/ref/parsers/effect_txt.py b/save-reader/ref/parsers/effect_txt.py new file mode 100644 index 0000000..1ccffab --- /dev/null +++ b/save-reader/ref/parsers/effect_txt.py @@ -0,0 +1,98 @@ +"""effect_txt.py -- reader for Effects/*.effect (particle-effect definitions). + +NOT the brace-block format. Layout: + + TXT # magic first line + KEY value # scalar (number, TRUE/FALSE, "quoted") + KEY # group: KEY on its own line, then + BEGIN + ...nested KEY value / groups... + END + +Order matters: 'PARTICLEDATATYPE n' is followed by the CREATION / +VARIATION / OVERLIFE curves that belong to that datatype, and 'MODIFIER' +repeats once per type. So each level is returned as an ordered list of +[key, value] pairs (value = scalar or nested list). to_dict() gives a +dict view (repeats -> lists) when order is not needed. + +Quirks handled: one file has CRLF; 'NAME "New Emitter"' values contain +spaces; indentation is cosmetic (tabs); the format is line-based. + +Stdlib only. +""" +from __future__ import annotations + +from typing import Any + +from flat_kv import split_tokens, strip_comment +from mars_data import coerce + +__all__ = ["parse", "parse_file", "to_dict", "EffectSyntaxError"] + +Pairs = list # list[[key, value]] + + +class EffectSyntaxError(ValueError): + pass + + +def parse(text: str, *, typed: bool = True) -> Pairs: + lines = text.splitlines() + if not lines or lines[0].strip() != "TXT": + raise EffectSyntaxError("missing TXT magic") + stack: list[Pairs] = [[]] + pending_key: str | None = None + for lineno, raw in enumerate(lines[1:], 2): + line = strip_comment(raw).strip() + if not line: + continue + if line == "BEGIN": + if pending_key is None: + raise EffectSyntaxError(f"line {lineno}: BEGIN without a key") + grp: Pairs = [] + stack[-1].append([pending_key, grp]) + stack.append(grp) + pending_key = None + continue + if line == "END": + if len(stack) == 1: + raise EffectSyntaxError(f"line {lineno}: END without BEGIN") + stack.pop() + continue + if pending_key is not None: + raise EffectSyntaxError(f"line {lineno}: key {pending_key!r} not followed by BEGIN") + toks = split_tokens(line) + key = toks[0][0] + if len(toks) == 1: + pending_key = key + continue + vals = [coerce(t) if (typed and not q) else t for t, q in toks[1:]] + stack[-1].append([key, vals[0] if len(vals) == 1 else vals]) + if len(stack) != 1: + raise EffectSyntaxError(f"{len(stack) - 1} unclosed BEGIN group(s)") + if pending_key is not None: + raise EffectSyntaxError(f"trailing key {pending_key!r} without BEGIN") + return stack[0] + + +def to_dict(pairs: Pairs) -> dict: + d: dict = {} + for key, val in pairs: + if isinstance(val, list) and val and isinstance(val[0], list) and len(val[0]) == 2 and isinstance(val[0][0], str): + val = to_dict(val) + if key in d: + if not isinstance(d[key], list) or not getattr(d[key], "_rep", False): + d[key] = _Rep([d[key]]) + d[key].append(val) + else: + d[key] = val + return d + + +class _Rep(list): + _rep = True + + +def parse_file(path, **kw) -> Pairs: + with open(path, "rb") as f: + return parse(f.read().decode("cp1252"), **kw) diff --git a/save-reader/ref/parsers/flat_kv.py b/save-reader/ref/parsers/flat_kv.py new file mode 100644 index 0000000..1ce67cd --- /dev/null +++ b/save-reader/ref/parsers/flat_kv.py @@ -0,0 +1,127 @@ +"""flat_kv.py -- readers for the flat 'KEY value' tuning tables and the +whitespace-positional tables under Data/, Weapons/, Badges/, Avatars/, GUI/. + +Two shapes exist: + + parse_kv(text) KEY value -> {KEY: value} + one constant per line; value is a bareword or a + "quoted string"; '//' comments; colors are quoted + "r g b" (use color()). Files: Data/globals.txt, + Data/species.txt, Data/Strategy/StrategyVars.txt, + Data/Combat/*.txt (most), Data/encounters.txt, ... + + parse_rows(text) tok tok tok ... -> [[tok, ...], ...] + one record per line, whitespace separated, quoted + tokens may contain spaces; '//' comments. Files: + Weapons/_turrets.txt, Weapons/_defaultweapons.txt, + Data/Combat/damfx*.txt, Data/Strategy/playercolors.txt, + Badges/BadgeTable.txt, Avatars/AvatarTable.txt, + GUI/WeaponIconPlacements.txt + +Quirks handled: '//' inside a quoted value is not a comment; a quoted +value may be empty (""); keys repeat in a few files (kept as list); +duplicate-key detection is exposed via parse_kv(..., on_dup=). + +Stdlib only. +""" +from __future__ import annotations + +import re +from typing import Any + +from mars_data import coerce + +__all__ = ["parse_kv", "parse_rows", "duplicates", "color", "strip_comment", + "split_tokens", "parse_kv_file", "parse_rows_file"] + +_TOK_RE = re.compile(r'"([^"]*)"|(\S+)') + + +def strip_comment(line: str) -> str: + """Remove a trailing // comment, ignoring // inside double quotes.""" + in_q = False + i = 0 + n = len(line) + while i < n: + c = line[i] + if c == '"': + in_q = not in_q + elif c == "/" and not in_q and line.startswith("//", i): + return line[:i] + i += 1 + return line + + +def split_tokens(line: str) -> list[tuple[str, bool]]: + """Split a line into (token, was_quoted) pairs.""" + out = [] + for m in _TOK_RE.finditer(line): + if m.group(1) is not None: + out.append((m.group(1), True)) + else: + out.append((m.group(2), False)) + return out + + +def parse_rows(text: str, *, typed: bool = True) -> list[list[Any]]: + rows = [] + for raw in text.splitlines(): + line = strip_comment(raw).strip() + if not line: + continue + toks = split_tokens(line) + rows.append([coerce(t) if (typed and not q) else t for t, q in toks]) + return rows + + +def _pairs(text: str, typed: bool): + for lineno, raw in enumerate(text.splitlines(), 1): + line = strip_comment(raw).strip() + if not line: + continue + toks = split_tokens(line) + key = toks[0][0] + vals = [coerce(t) if (typed and not q) else t for t, q in toks[1:]] + val: Any = None if not vals else (vals[0] if len(vals) == 1 else vals) + yield lineno, key, val + + +def parse_kv(text: str, *, typed: bool = True, on_dup: str = "last") -> dict: + """KEY value per line -> dict. A value made of several unquoted tokens + is kept as a list. on_dup: 'last' (later line wins), 'first', 'error'. + Use duplicates() to find repeated keys.""" + d: dict = {} + for lineno, key, val in _pairs(text, typed): + if key in d: + if on_dup == "error": + raise ValueError(f"line {lineno}: duplicate key {key}") + if on_dup == "first": + continue + d[key] = val + return d + + +def duplicates(text: str) -> dict[str, list[int]]: + """key -> line numbers, for keys that appear more than once.""" + seen: dict[str, list[int]] = {} + for lineno, key, _ in _pairs(text, False): + seen.setdefault(key, []).append(lineno) + return {k: v for k, v in seen.items() if len(v) > 1} + + +def color(value: str) -> tuple: + """'r g b' or 'r g b a' -> tuple of numbers.""" + return tuple(coerce(t) for t in value.split()) + + +def _read(path) -> str: + with open(path, "rb") as f: + return f.read().decode("cp1252") + + +def parse_kv_file(path, **kw) -> dict: + return parse_kv(_read(path), **kw) + + +def parse_rows_file(path, **kw) -> list: + return parse_rows(_read(path), **kw) diff --git a/save-reader/ref/parsers/manifest.py b/save-reader/ref/parsers/manifest.py new file mode 100644 index 0000000..ab4f2eb --- /dev/null +++ b/save-reader/ref/parsers/manifest.py @@ -0,0 +1,121 @@ +"""manifest.py -- the numbered id manifests and the '#'-commented CSVs. + +parse_manifest(text) -> Manifest + Weapons/_weapons.txt and Species//sections/_shipsections.txt: + one per line + // DELETED - retired id (still reserved) + Ids are the persistent network / savegame ids. Filenames are matched + case-insensitively (the shipped manifests have 'DEWar.SHIPSECTION', + 'CRAIC.Shipsection' etc. against lower-case files -- Windows FS). + +parse_csv(text) -> list[list[str]] + Rows with '#' or '//' as first non-blank char are comments; blank rows + dropped; RFC-4180 quoting honoured (Strings.csv has one multi-line cell + and quoted commas). Header rows that start with '#' (aitechpri.csv, + "# species" in stock_diplomacy_messages.csv) are returned separately + via parse_csv_with_header(). + +Stdlib only. +""" +from __future__ import annotations + +import csv +import io +import re +from dataclasses import dataclass, field + +__all__ = ["Manifest", "parse_manifest", "parse_manifest_file", + "parse_csv", "parse_csv_file", "parse_csv_with_header", "read_text"] + +_DELETED_RE = re.compile(r"//\s*DELETED\s*-\s*(\d+)", re.I) +_ENTRY_RE = re.compile(r"^\s*(\d+)\s+(\S+)\s*$") + + +@dataclass +class Manifest: + entries: list[tuple[int, str]] = field(default_factory=list) # (id, filename) + deleted: list[int] = field(default_factory=list) + problems: list[str] = field(default_factory=list) + + def by_id(self) -> dict[int, str]: + return dict(self.entries) + + def by_name(self) -> dict[str, int]: + """lower-cased filename -> id""" + return {n.lower(): i for i, n in self.entries} + + +def parse_manifest(text: str) -> Manifest: + m = Manifest() + seen: dict[int, int] = {} + for lineno, raw in enumerate(text.splitlines(), 1): + line = raw.strip() + if not line: + continue + d = _DELETED_RE.search(line) + if d: + m.deleted.append(int(d.group(1))) + continue + if line.startswith("//"): + continue + e = _ENTRY_RE.match(line) + if not e: + m.problems.append(f"line {lineno}: unrecognised {line!r}") + continue + i, name = int(e.group(1)), e.group(2) + if i in seen: + m.problems.append(f"line {lineno}: duplicate id {i} (first at line {seen[i]})") + seen[i] = lineno + m.entries.append((i, name)) + for i in m.deleted: + if i in seen: + m.problems.append(f"id {i} is both DELETED and assigned") + return m + + +def read_text(path) -> str: + with open(path, "rb") as f: + return f.read().decode("cp1252") + + +def parse_manifest_file(path) -> Manifest: + return parse_manifest(read_text(path)) + + +def _is_comment(row: list[str]) -> bool: + if not row: + return True + first = row[0].lstrip() + if first.startswith("#") or first.startswith("//"): + return True + return all(c.strip() == "" for c in row) + + +def parse_csv(text: str, *, strip: bool = True) -> list[list[str]]: + rows = [] + for row in csv.reader(io.StringIO(text, newline="")): + if _is_comment(row): + continue + rows.append([c.strip() for c in row] if strip else row) + return rows + + +def parse_csv_with_header(text: str) -> tuple[list[str] | None, list[list[str]]]: + """Return (header, rows). Header = the first '#'-prefixed row that + contains a comma (e.g. '# ,,...'), with the '#' and + any '<>' stripped; None when there is no such row.""" + header = None + for row in csv.reader(io.StringIO(text, newline="")): + if not row: + continue + first = row[0].lstrip() + if first.startswith("#") and len(row) > 1: + header = [c.strip().lstrip("#").strip().strip("<>") for c in row] + break + if not _is_comment(row): + break + return header, parse_csv(text) + + +def parse_csv_file(path, **kw): + return parse_csv(read_text(path), **kw) diff --git a/save-reader/ref/parsers/mars_data.py b/save-reader/ref/parsers/mars_data.py new file mode 100644 index 0000000..fe5dcd9 --- /dev/null +++ b/save-reader/ref/parsers/mars_data.py @@ -0,0 +1,211 @@ +"""mars_data.py -- reader for the Mars engine's brace-block key/value format. + +Covers: *.weapon, *.shipsection, *.tech, *.combat, *.def, *.script and the +block-form *.txt files (scenarios, tutorial, credits, systemnames, skydefs, +ctechvars, shipai). + +Grammar (as observed in every shipped SOTS1 file): + + body := (block | pair | item)* + block := NAME '{' body '}' + pair := NAME value + item := QUOTED # bare quoted string inside a block + value := QUOTED | BAREWORD + comment := '//' .* EOL + +A file's top level is itself a body (scenario .txt files mix top-level pairs +and player{} blocks; catalog files hold one or many named blocks). + +Result shape: plain dicts. A key seen once maps to its value; a key seen +more than once maps to a list (use get_list() when you want a list always). +Bare quoted items are collected under the key "_items". + +Quirks handled (all seen in the real data, see parsers-report.md): + * keys are case-insensitive to the engine ("Requires"/"requires", + "badge"/"Badge") -> keys are lower-cased unless keep_case=True + * a block may open on the same line as a preceding pair + ("turretsize small mount {") and a block name may sit on the same line + as its brace ("weapon {") + * backslashes inside quoted strings are literal (Windows paths); there is + no escape syntax + * '//' inside a quoted string is not a comment + * CRLF and LF line endings, cp1252 bytes (decoded losslessly) + * numbers use C float syntax: ".5", "-.8", "7e+8" + +Stdlib only. +""" +from __future__ import annotations + +import re +from typing import Any, Iterator + +__all__ = ["parse", "parse_file", "coerce", "get_list", "MarsSyntaxError"] + + +class MarsSyntaxError(ValueError): + pass + + +# --- tokenizer ------------------------------------------------------------- + +_TOKEN_RE = re.compile( + r""" + (?P\s+) + | (?P//[^\n]*) + | (?P\{) + | (?P\}) + | (?P"[^"]*") + | (?P") + | (?P[^\s{}"]+) + """, + re.VERBOSE, +) + + +def _tokenize(text: str) -> Iterator[tuple[str, str, int]]: + """Yield (kind, value, line). kind in {open, close, quoted, bare}.""" + line = 1 + pos = 0 + n = len(text) + while pos < n: + m = _TOKEN_RE.match(text, pos) + if m is None: # pragma: no cover - regex is exhaustive + raise MarsSyntaxError(f"line {line}: cannot tokenize {text[pos:pos+20]!r}") + kind = m.lastgroup + tok = m.group() + pos = m.end() + if kind == "ws": + line += tok.count("\n") + continue + if kind == "comment": + continue + if kind == "bad_quote": + raise MarsSyntaxError(f"line {line}: unterminated string") + if kind == "quoted": + yield kind, tok[1:-1], line + line += tok.count("\n") + else: + yield kind, tok, line + + +# --- parser ---------------------------------------------------------------- + +_INT_RE = re.compile(r"[+-]?\d+$") +_FLOAT_RE = re.compile(r"[+-]?(\d+\.\d*|\.\d+|\d+)([eE][+-]?\d+)?$") + + +def coerce(tok: str) -> Any: + """Bareword -> int / float / bool when it looks like one, else str.""" + if _INT_RE.match(tok): + return int(tok) + if _FLOAT_RE.match(tok): + return float(tok) + low = tok.lower() + if low == "true": + return True + if low == "false": + return False + return tok + + +def _add(d: dict, key: str, value: Any) -> None: + if key in d: + cur = d[key] + if isinstance(cur, list): + cur.append(value) + else: + d[key] = [cur, value] + else: + d[key] = value + + +def get_list(d: dict, key: str) -> list: + """Always return a list for a key (missing -> [], single -> [x]).""" + v = d.get(key) + if v is None: + return [] + return v if isinstance(v, list) else [v] + + +class _Parser: + def __init__(self, text: str, typed: bool, keep_case: bool, strict: bool, warnings: list | None): + self.toks = list(_tokenize(text)) + self.i = 0 + self.typed = typed + self.keep_case = keep_case + self.strict = strict + self.warnings = warnings if warnings is not None else [] + + def _warn(self, msg: str) -> None: + if self.strict: + raise MarsSyntaxError(msg) + self.warnings.append(msg) + + def _peek(self): + return self.toks[self.i] if self.i < len(self.toks) else None + + def _next(self): + t = self.toks[self.i] + self.i += 1 + return t + + def _key(self, name: str) -> str: + return name if self.keep_case else name.lower() + + def body(self, depth: int) -> dict: + d: dict = {} + while True: + t = self._peek() + if t is None: + if depth: + # 11 shipped shipsections never close their outer block; + # the engine treats EOF as closing every open block. + self._warn(f"end of file inside block (depth {depth})") + return d + kind, val, line = t + if kind == "close": + self._next() + if not depth: + # CrPropaganda.shipsection has one '}' too many. + self._warn(f"line {line}: stray '}}' at top level") + continue + return d + if kind == "open": + raise MarsSyntaxError(f"line {line}: '{{' without a block name") + self._next() + if kind == "quoted": + # bare string item (systemnames.txt lists) -- never a key + _add(d, "_items", val) + continue + nxt = self._peek() + if nxt is None or nxt[0] == "close": + # lone bareword at end of block: treat as flag item + _add(d, "_items", val) + continue + if nxt[0] == "open": + self._next() + _add(d, self._key(val), self.body(depth + 1)) + continue + nkind, nval, _ = self._next() + if nkind == "bare" and self.typed: + nval = coerce(nval) + _add(d, self._key(val), nval) + + +def parse(text: str, *, typed: bool = True, keep_case: bool = False, + strict: bool = False, warnings: list | None = None) -> dict: + """Parse brace-block text into nested dicts. + + typed -- convert bareword numbers/bools (quoted strings stay str) + keep_case -- keep key case instead of lower-casing + strict -- raise on unbalanced braces instead of recovering the way + the engine does (EOF closes open blocks, stray top-level + '}' ignored); pass warnings=[] to collect the recoveries + """ + return _Parser(text, typed, keep_case, strict, warnings).body(0) + + +def parse_file(path, **kw) -> dict: + with open(path, "rb") as f: + raw = f.read() + return parse(raw.decode("cp1252"), **kw) diff --git a/save-reader/ref/parsers/verify.py b/save-reader/ref/parsers/verify.py new file mode 100644 index 0000000..cb4b6d3 --- /dev/null +++ b/save-reader/ref/parsers/verify.py @@ -0,0 +1,634 @@ +"""verify.py -- parse every shipped SOTS1 data file, cross-link the catalogs, +and emit normalized JSON artifacts. + +usage: python3 verify.py + +Prints a markdown report to stdout; writes to : + tech_tree.json weapons.json shipsections.json strings.json + schema_stats.json crosslink.json tech_tree.dot +""" +from __future__ import annotations + +import collections +import json +import os +import re +import sys + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import effect_txt +import flat_kv +import manifest +import mars_data +from mars_data import get_list + +RACES = ["Human", "Zuul", "Hiver", "Tarkas", "Liir", "Morrigi"] +PLAYABLE = RACES +ALL_RACE_DIRS = RACES + ["_NPC"] + +BRACE_TXT = {"Data/tutorial.txt", "Data/credits.txt", "Data/Strategy/systemnames.txt", + "Data/Combat/ctechvars.txt", "Data/Combat/shipai.txt", + "Models/Skysphere/skydefs.txt", "Models/Skysphere/NodeSpace-skydefs.txt"} +ROWS_TXT = {"Weapons/_turrets.txt", "Weapons/_defaultweapons.txt", "Data/Combat/damfx.txt", + "Data/Combat/damfx_levels.txt", "Data/Strategy/playercolors.txt", + "Badges/BadgeTable.txt", "Avatars/AvatarTable.txt", "GUI/WeaponIconPlacements.txt"} +PROSE = {"Locale/EN/ChatTrans.txt"} + + +def kind_of(rel: str) -> str: + ext = rel.rsplit(".", 1)[-1].lower() + base = os.path.basename(rel) + if ext in ("weapon", "shipsection", "tech", "combat", "def", "script"): + return "brace:" + ext + if ext == "effect": + return "effect" + if ext == "csv": + return "csv" + if ext in ("fx", "fxh"): + return "hlsl" + if ext == "txt": + if base in ("_weapons.txt", "_shipsections.txt"): + return "manifest" + if rel.startswith("Scenarios/") or rel in BRACE_TXT: + return "brace:txt" + if rel in ROWS_TXT: + return "rows" + if rel.startswith("Locale/EN/Desc") or rel in PROSE: + return "prose" + return "kv" + return "other" + + +def walk(root): + for dp, _, fn in os.walk(root): + for f in sorted(fn): + p = os.path.join(dp, f) + yield p, os.path.relpath(p, root).replace(os.sep, "/") + + +# --- schema stats ------------------------------------------------------------ + +def schema_walk(node, path, stats): + """Count key occurrences per block path, and which keys are blocks.""" + st = stats.setdefault(path, {"blocks": 0, "keys": collections.Counter(), "sub": collections.Counter()}) + st["blocks"] += 1 + for k, v in node.items(): + vals = v if isinstance(v, list) else [v] + for x in vals: + if isinstance(x, dict): + st["sub"][k] += 1 + schema_walk(x, path + "." + k, stats) + else: + st["keys"][k] += 1 + + +# --- tech tree --------------------------------------------------------------- + +_RP_RE = re.compile(r"^RP:(\d+)$", re.I) +_PCT_RE = re.compile(r"^(\w+):(\d+)$") + + +def parse_allows(s: str): + toks = s.split() + child = toks[0] + rp = None + pct = {} + extra = [] + for t in toks[1:]: + m = _RP_RE.match(t) + if m: + rp = int(m.group(1)) + continue + m = _PCT_RE.match(t) + if m and m.group(1) in RACES: + pct[m.group(1)] = int(m.group(2)) + continue + extra.append(t) + return child, rp, pct, extra + + +def main(root: str, out: str) -> int: + os.makedirs(out, exist_ok=True) + rep = [] + P = rep.append + + # ---- 1. parse everything --------------------------------------------- + ok = collections.Counter() + fail = collections.Counter() + fails = [] + warns = [] + parsed = {} # rel -> object + for p, rel in walk(root): + k = kind_of(rel) + try: + if k.startswith("brace"): + w = [] + obj = mars_data.parse_file(p, warnings=w) + # strict re-parse to record the recovery + if w: + warns.append((rel, w)) + elif k == "effect": + obj = effect_txt.parse_file(p) + elif k == "csv": + obj = manifest.parse_csv_file(p) + elif k == "manifest": + obj = manifest.parse_manifest_file(p) + if obj.problems: + raise ValueError("; ".join(obj.problems)) + elif k == "rows": + obj = flat_kv.parse_rows_file(p) + elif k == "kv": + txt = manifest.read_text(p) + obj = flat_kv.parse_kv(txt) + d = flat_kv.duplicates(txt) + if d: + warns.append((rel, [f"duplicate keys {d}"])) + else: + continue + parsed[rel] = obj + ok[k] += 1 + except Exception as e: # noqa: BLE001 + fail[k] += 1 + fails.append((rel, repr(e))) + + P("## Parse results") + P("") + P("| reader | file kind | files | parsed | failed |") + P("|---|---|---|---|---|") + reader_of = {"brace": "mars_data", "effect": "effect_txt", "csv": "manifest.parse_csv", + "manifest": "manifest.parse_manifest", "rows": "flat_kv.parse_rows", "kv": "flat_kv.parse_kv"} + for k in sorted(set(ok) | set(fail)): + P(f"| {reader_of[k.split(':')[0]]} | {k} | {ok[k] + fail[k]} | {ok[k]} | {fail[k]} |") + P("") + P(f"Total: {sum(ok.values())} parsed, {sum(fail.values())} failed " + f"(skipped: {sum(1 for _, r in walk(root) if kind_of(r) in ('hlsl', 'prose', 'other'))} " + f"HLSL/prose files that are not data).") + if fails: + P("") + P("Failures:") + for rel, e in fails: + P(f"- `{rel}`: {e}") + if warns: + P("") + P("Lenient recoveries (engine-compatible; strict=True would reject these):") + for rel, w in warns: + P(f"- `{rel}`: {'; '.join(w)}") + P("") + + # ---- 2. schema stats --------------------------------------------------- + stats = {} + for rel, obj in parsed.items(): + k = kind_of(rel) + if k in ("brace:weapon", "brace:shipsection", "brace:tech", "brace:combat", "brace:def", "brace:script"): + schema_walk(obj, k.split(":")[1], stats) + schema_json = {path: {"blocks": st["blocks"], + "keys": dict(st["keys"].most_common()), + "subblocks": dict(st["sub"].most_common())} + for path, st in sorted(stats.items())} + json.dump(schema_json, open(os.path.join(out, "schema_stats.json"), "w"), indent=1) + + P("## Schema stats (key frequency per block type)") + P("") + P("Full table in `schema_stats.json`. Block paths with instance counts and the") + P("keys seen in them (count = number of block instances carrying the key):") + P("") + for path in ["weapon.weapon", "shipsection.shipsection", "tech.tech"]: + st = stats[path] + P(f"### `{path}` ({st['blocks']} instances)") + P("") + P("keys: " + ", ".join(f"{k}:{n}" for k, n in st["keys"].most_common())) + P("") + P("sub-blocks: " + ", ".join(f"{k}:{n}" for k, n in st["sub"].most_common())) + P("") + P("All block paths: " + ", ".join(f"`{p}`({st['blocks']})" for p, st in sorted(stats.items()))) + P("") + + # ---- 3. build catalogs ------------------------------------------------- + techs = parsed["TechTree/MasterTechList.tech"]["tech"] + tech_by = {t["name"].lower(): t for t in techs} + groups = collections.defaultdict(list) + for t in techs: + if "group" in t: + groups[str(t["group"]).upper()].append(t["name"]) + + strings_rows = parsed["Locale/EN/Strings.csv"] + strings = {} + string_dups = [] + for r in strings_rows: + k, v = r[0], (r[1] if len(r) > 1 else "") + if k in strings: + string_dups.append((k, strings[k], v)) + strings[k] = v + strings_lc = {k.lower(): v for k, v in strings.items()} + + def s(key): + return strings_lc.get(key.lower()) + + weapons = {} # stem -> record + for rel, obj in parsed.items(): + if kind_of(rel) != "brace:weapon": + continue + stem = os.path.basename(rel)[:-7] + w = dict(obj["weapon"]) + weapons[stem.lower()] = {"stem": stem, "file": rel, + "scope": "NPC" if rel.startswith("Species/_NPC") else "player", + "id": None, **w} + wman = parsed["Weapons/_weapons.txt"] + for i, name in wman.entries: + key = name.lower()[:-7] + if key in weapons and weapons[key]["scope"] == "player": + weapons[key]["id"] = i + + sections = {} # (race, stem) -> record + for rel, obj in parsed.items(): + if kind_of(rel) != "brace:shipsection": + continue + race = rel.split("/")[1] + stem = os.path.basename(rel)[:-12] + sections[(race, stem.lower())] = {"race": race, "stem": stem, "file": rel, "id": None, **obj["shipsection"]} + for race in ALL_RACE_DIRS: + m = parsed[f"Species/{race}/sections/_shipsections.txt"] + for i, name in m.entries: + key = (race, name.lower()[:-12]) + if key in sections: + sections[key]["id"] = i + section_stems = collections.defaultdict(list) # stem.lower -> [races] + for (race, st), rec in sections.items(): + section_stems[st].append(race) + + # ---- 4. cross-links ---------------------------------------------------- + X = {} + P("## Cross-link results") + P("") + + def tech_exists(name): + n = name.lower() + if n in tech_by: + return True + if n.startswith("grp_") and n[4:].upper() in groups: + return True + return False + + # weapon.requires -> tech (repeated `requires` lines = AND of techs) + dang = [] + case_mismatch = [] + nref = 0 + multi = 0 + for w in weapons.values(): + reqs = get_list(w, "requires") + multi += len(reqs) > 1 + for r in reqs: + nref += 1 + r = str(r) + if not tech_exists(r): + dang.append((w["file"], r)) + elif r.lower() in tech_by and tech_by[r.lower()]["name"] != r: + case_mismatch.append((w["file"], r)) + no_req = [w["file"] for w in weapons.values() if "requires" not in w] + X["weapon_requires_dangling"] = dang + X["weapon_requires_case_mismatch"] = case_mismatch + X["weapon_without_requires"] = no_req + P(f"- weapon `requires` -> tech: {nref} refs in {len(weapons) - len(no_req)} weapons ({multi} weapons list 2+ techs), " + f"{len(dang)} dangling, {len(case_mismatch)} case-mismatched; {len(no_req)} weapons have no `requires` " + f"(NPC: {sum(1 for f in no_req if f.startswith('Species/_NPC'))}, player: " + f"{[os.path.basename(f) for f in no_req if not f.startswith('Species/_NPC')]}).") + for f, r in dang: + P(f" - DANGLING `{f}` requires `{r}`") + for f, r in case_mismatch: + P(f" - case: `{f}` requires `{r}` (tech is `{tech_by[r.lower()]['name']}`)") + + # shipsection.requires / option -> tech + dang = [] + case_mm = [] + opt_dang = [] + scalar_opts = [] + nreq = 0 + nopt = 0 + for rec in sections.values(): + for r in get_list(rec, "requires"): + nreq += 1 + if not tech_exists(str(r)): + dang.append((rec["file"], r)) + elif str(r).lower() in tech_by and tech_by[str(r).lower()]["name"] != r: + case_mm.append((rec["file"], r)) + for blk_key in ("option", "optiondef"): + for blk in get_list(rec, blk_key): + # a few files write a bare `option TECH` at section level + # instead of wrapping it in option { } + opts = get_list(blk, "option") if isinstance(blk, dict) else [blk] + if not isinstance(blk, dict): + scalar_opts.append((rec["file"], blk)) + for o in opts: + nopt += 1 + if not tech_exists(str(o)): + opt_dang.append((rec["file"], o)) + X["shipsection_requires_dangling"] = dang + X["shipsection_requires_case_mismatch"] = case_mm + X["shipsection_option_dangling"] = opt_dang + P(f"- shipsection `requires` -> tech: {nreq} refs, {len(dang)} dangling, {len(case_mm)} case-mismatched.") + for f, r in dang: + P(f" - DANGLING `{f}` requires `{r}`") + for f, r in case_mm: + P(f" - case: `{f}` requires `{r}`") + X["shipsection_scalar_option"] = scalar_opts + P(f"- shipsection `option{{option T}}`/`optiondef` -> tech: {nopt} refs, {len(opt_dang)} dangling. " + f"Two forms coexist: `option {{ option A option B }}` (a mutually-exclusive choice group) and a bare " + f"section-level `option T` ({len(scalar_opts)} occurrences in {len(set(f for f, _ in scalar_opts))} files, " + f"e.g. `option DRV_PlsmFoc` on engine sections) -- both merge under the key `option`, so consumers must " + f"accept str-or-dict list members.") + for f, r in sorted(set(opt_dang)): + P(f" - DANGLING `{f}` option `{r}`") + + # tech.ship.section -> shipsection + dang = [] + nsec = 0 + for t in techs: + for blk in get_list(t, "ship"): + for sname in get_list(blk, "section"): + nsec += 1 + if str(sname).lower() not in section_stems: + dang.append((t["name"], sname)) + X["tech_ship_section_dangling"] = dang + P(f"- tech `ship{{section}}` -> shipsection: {nsec} refs, {len(dang)} dangling " + f"(matched against the union of all race catalogs, case-insensitive).") + for t, sname in dang: + P(f" - DANGLING tech `{t}` unlocks section `{sname}`") + + # tech.weapon.filename -> file + disk = {rel.lower() for _, rel in walk(root)} + dang = [(t["name"], w["filename"]) for t in techs for w in get_list(t, "weapon") if w["filename"].lower() not in disk] + X["tech_weapon_filename_dangling"] = dang + nw = sum(len(get_list(t, "weapon")) for t in techs) + P(f"- tech `weapon{{filename}}` -> file: {nw} refs, {len(dang)} dangling.") + + # tech.requires / allows -> tech + dang_req = [(t["name"], r) for t in techs for r in get_list(t, "requires") if not tech_exists(str(r))] + edges = [] + dang_allow = [] + bad_allow = [] + for t in techs: + for a in get_list(t, "allows"): + child, rp, pct, extra = parse_allows(a) + if extra or rp is None: + bad_allow.append((t["name"], a)) + if child.lower() not in tech_by: + dang_allow.append((t["name"], child)) + edges.append({"from": t["name"], "to": child, "rp": rp, "pct": pct}) + X["tech_requires_dangling"] = dang_req + X["tech_allows_dangling"] = dang_allow + X["tech_allows_unparsed"] = bad_allow + P(f"- tech `requires` -> tech/GRP_: {sum(len(get_list(t, 'requires')) for t in techs)} refs, {len(dang_req)} dangling. " + f"Groups: {dict((g, len(v)) for g, v in groups.items())}.") + for t, r in dang_req: + P(f" - DANGLING tech `{t}` requires `{r}`") + P(f"- tech `allows` edges: {len(edges)}, {len(dang_allow)} point at unknown techs, {len(bad_allow)} unparsable.") + for t, c in dang_allow: + P(f" - DANGLING tech `{t}` allows `{c}`") + roots = [t["name"] for t in techs if not any(e["to"].lower() == t["name"].lower() for e in edges)] + P(f"- techs never allowed by anything (roots/orphans): {len(roots)}: {', '.join(roots)}") + dup_names = [n for n, c in collections.Counter(t["name"].lower() for t in techs).items() if c > 1] + P(f"- duplicate tech names: {dup_names or 'none'}") + + # manifests <-> files + P("- id manifests <-> files:") + man_rep = {} + wfiles = {os.path.basename(rel).lower() for rel in parsed if rel.startswith("Weapons/") and rel.endswith(".weapon")} + listed = {n.lower() for _, n in wman.entries} + man_rep["Weapons"] = {"ids": len(wman.entries), "deleted": wman.deleted, + "listed_but_no_file": sorted(listed - wfiles), "file_but_unlisted": sorted(wfiles - listed)} + for race in ALL_RACE_DIRS: + m = parsed[f"Species/{race}/sections/_shipsections.txt"] + files = {os.path.basename(rel).lower() for rel in parsed if rel.startswith(f"Species/{race}/sections/") and rel.endswith(".shipsection")} + listed = {n.lower() for _, n in m.entries} + man_rep[race] = {"ids": len(m.entries), "deleted": m.deleted, + "listed_but_no_file": sorted(listed - files), "file_but_unlisted": sorted(files - listed)} + X["manifests"] = man_rep + for k, v in man_rep.items(): + P(f" - `{k}`: {v['ids']} ids (deleted {v['deleted'] or 'none'}); " + f"listed-but-no-file {len(v['listed_but_no_file'])}; file-but-unlisted {len(v['file_but_unlisted'])}") + for n in v["listed_but_no_file"]: + P(f" - MISSING FILE for id {[i for i, nn in (wman if k == 'Weapons' else parsed[f'Species/{k}/sections/_shipsections.txt']).entries if nn.lower() == n][0]}: `{n}`") + for n in v["file_but_unlisted"]: + P(f" - UNLISTED file `{n}` (no network/save id)") + npc_weapons_unlisted = sorted(w["file"] for w in weapons.values() if w["scope"] == "NPC") + P(f" - `Species/_NPC/weapons/*.weapon` ({len(npc_weapons_unlisted)} files) have no manifest at all; " + f"they are referenced by filename from `_NPC` shipsection `bank{{weapon}}` lines.") + + # strings + P("- localization:") + P(f" - `Strings.csv`: {len(strings_rows)} data rows -> {len(strings)} keys. {len(string_dups)} keys occur twice " + f"because one copy carries a trailing space (parse_csv strips cells; the later row wins):") + for k, a, b in string_dups: + P(f" - `{k}`: {a!r} then {b!r}") + miss_tn = [t["name"] for t in techs if s("TECHNAME_" + t["name"]) is None] + miss_td = [t["name"] for t in techs if s("TECHDESC_" + t["name"]) is None] + P(f" - TECHNAME_/TECHDESC_ for {len(techs)} techs: {len(miss_tn)} / {len(miss_td)} missing. {miss_tn} {miss_td}") + stems = sorted(section_stems) + miss_sn = [st for st in stems if s("SECTIONNAME_" + st) is None] + miss_sd = [st for st in stems if s("SECTIONDESC_" + st) is None] + P(f" - SECTIONNAME_/SECTIONDESC_ for {len(stems)} distinct section stems: {len(miss_sn)} / {len(miss_sd)} missing.") + for label, miss in (("SECTIONNAME_", miss_sn), ("SECTIONDESC_", miss_sd)): + npc_only = [st for st in miss if section_stems[st] == ["_NPC"]] + other = [st for st in miss if st not in npc_only] + P(f" - missing {label}: {len(npc_only)} are `_NPC`-only stems (never shown in the design UI); " + f"player-race stems: {len(other)} {other}") + miss_wn = [(w["file"], w.get("name")) for w in weapons.values() + if isinstance(w.get("name"), str) and w["name"].startswith("@") and s(w["name"][1:]) is None] + unnamed = [w["file"] for w in weapons.values() if "name" not in w] + P(f" - weapon `name @TOKEN`: {len(miss_wn)} unresolved of {sum(1 for w in weapons.values() if 'name' in w)}; " + f"{len(unnamed)} weapons carry no `name`.") + for f, n in miss_wn: + P(f" - UNRESOLVED `{f}` name `{n}`") + # every @token anywhere in brace files + at_missing = collections.Counter() + at_total = 0 + for rel, obj in parsed.items(): + if not kind_of(rel).startswith("brace"): + continue + for tok in re.findall(r"@([A-Za-z0-9_]+)", manifest.read_text(os.path.join(root, rel))): + at_total += 1 + if s(tok) is None: + at_missing[(rel, tok)] += 1 + P(f" - all `@TOKEN` refs in brace-block files: {at_total} refs, {len(at_missing)} unresolved.") + for (rel, tok), n in sorted(at_missing.items()): + P(f" - UNRESOLVED `{rel}` `@{tok}`") + X["strings"] = {"missing_techname": miss_tn, "missing_techdesc": miss_td, + "missing_sectionname": miss_sn, "missing_sectiondesc": miss_sd, + "unresolved_weapon_name": miss_wn, "unresolved_at_tokens": sorted(f"{r}:@{t}" for r, t in at_missing)} + + # turrets + turrets = parsed["Weapons/_turrets.txt"] + + def last_lc(d, key): + v = get_list(d, key) + return str(v[-1]).lower() if v else None + + tpairs = {(str(r[1]).lower(), str(r[2]).lower()) for r in turrets} # (weapon-size, class) + tslots = {(str(r[0]).lower(), str(r[2]).lower()) for r in turrets} # (mount size, class) + wpairs = collections.Counter((last_lc(w, "turretsize"), last_lc(w, "turretclass")) for w in weapons.values()) + w_unfit = sorted((p, n) for p, n in wpairs.items() if p not in tpairs) + bpairs = collections.Counter() + nobank = 0 + dupkeys = 0 + for rec in sections.values(): + for b in get_list(rec, "bank"): + if "turretsize" not in b: + nobank += 1 + continue + if isinstance(b.get("turretsize"), list) or isinstance(b.get("turretclass"), list): + dupkeys += 1 + bpairs[(last_lc(b, "turretsize"), last_lc(b, "turretclass"))] += 1 + b_unfit = sorted((p, n) for p, n in bpairs.items() if p not in tslots) + X["turrets"] = {"turret_rows": len(turrets), "weapon_size_class_pairs_without_turret": w_unfit, + "bank_size_class_pairs_without_turret": b_unfit, + "banks_without_turretsize": nobank, "banks_with_repeated_size_or_class": dupkeys} + P(f"- `_turrets.txt` ({len(turrets)} rows; size/class values compared case-insensitively -- the data mixes " + f"`Large`/`large`, `Missile`/`missile`, `Standard`/`standard`):") + P(f" - weapon (turretsize,turretclass) pairs with no turret row: {w_unfit or 'none'}") + P(f" - section bank (turretsize,turretclass) pairs with no turret row: {b_unfit or 'none'}") + P(f" - banks with no turretsize at all (NPC fixed-weapon banks): {nobank}; banks that repeat " + f"turretsize/turretclass inside one bank{{}} (last value taken): {dupkeys}") + + # NPC bank{weapon} refs + dang = [] + n = 0 + for rec in sections.values(): + for b in get_list(rec, "bank"): + for wf in get_list(b, "weapon"): + n += 1 + if str(wf).lower() not in disk: + dang.append((rec["file"], wf)) + X["bank_weapon_dangling"] = dang + P(f"- shipsection `bank{{weapon }}` -> file: {n} refs, {len(dang)} dangling.") + for f, w in dang: + P(f" - DANGLING `{f}` -> `{w}`") + + # default weapons + dw = parsed["Weapons/_defaultweapons.txt"] + dang = [r for r in dw if ("weapons/" + str(r[2])).lower() not in disk] + P(f"- `_defaultweapons.txt`: {len(dw)} rows, {len(dang)} name a missing weapon file. {dang or ''}") + + # AI tables + def csv_col(rel, col): + return [r[col] for r in parsed[rel] if len(r) > col and r[col]] + ai = {} + for rel in ("Data/Strategy/AI/aitechpri.csv", "Data/Strategy/AI/aitechgrp.csv", "Data/Strategy/AI/aitechmode.csv"): + rows = parsed[rel] + bad = [t for t in csv_col(rel, 0) if t.lower() not in tech_by] + ai[rel] = bad + if not rows: + P(f"- `{rel}`: 0 data rows -- the shipped file is a comment-only template (schema documented in its " + f"header, no entries); the AI's tech priorities must therefore come from code.") + else: + P(f"- `{rel}`: {len(rows)} rows; col0 not a tech: {bad or 'none'}") + bad = [x for x in csv_col("Data/Strategy/AI/affinity_section.csv", 0) if x.lower() not in section_stems] + ai["affinity_section_unknown"] = bad + P(f"- `AI/affinity_section.csv`: {len(parsed['Data/Strategy/AI/affinity_section.csv'])} rows; unknown sections: {bad or 'none'}") + bad = [x for x in csv_col("Data/Strategy/AI/raider_sections.csv", 0) if x.lower() not in section_stems] + P(f"- `AI/raider_sections.csv`: unknown sections: {bad or 'none'}") + wr = parsed["Data/Strategy/AI/weapon_replacements.csv"] + bad = [x for r in wr for x in r if x and x.lower() not in weapons] + ai["weapon_replacements_unknown"] = bad + P(f"- `AI/weapon_replacements.csv`: {len(wr)} rows; unknown weapon stems: {bad or 'none'}") + fams = collections.Counter(str(w.get("weaponfamily")) for w in weapons.values() if "weaponfamily" in w) + aw = csv_col("Data/Strategy/AI/affinity_weapon.csv", 0) + bad = [x for x in aw if x not in fams] + P(f"- `AI/affinity_weapon.csv`: families {sorted(set(aw))}; not a weaponfamily in any .weapon: {bad or 'none'}. " + f"weaponfamily values in data: {dict(fams)}") + # scenarios + for rel in sorted(parsed): + if rel.startswith("Scenarios/") and rel.endswith("Templates.csv"): + bad = [(r[0], x) for r in parsed[rel] for x in r[1:4] if x.lower() not in section_stems] + P(f"- `{rel}`: {len(parsed[rel])} rows; unknown sections: {bad or 'none'}") + if rel.startswith("Scenarios/") and rel.endswith("Techs.csv"): + bad = [x for x in csv_col(rel, 0) if x.lower() not in tech_by] + P(f"- `{rel}`: {len(parsed[rel])} rows; unknown techs: {bad or 'none'}") + X["ai"] = ai + json.dump(X, open(os.path.join(out, "crosslink.json"), "w"), indent=1) + P("") + + # ---- 5. artifacts ------------------------------------------------------ + nodes = [] + for t in techs: + strat = get_list(t, "strategy") + inc = [x for b in strat for x in get_list(b, "inc")] + dec = [x for b in strat for x in get_list(b, "dec")] + nodes.append({ + "name": t["name"], + "display_name": s("TECHNAME_" + t["name"]), + "description": s("TECHDESC_" + t["name"]), + "family": t.get("family"), + "family_inferred": t["name"].split("_", 1)[0].upper(), + "type": t.get("type"), + "threat": t.get("threat"), + "group": t.get("group"), + "option_cost": t.get("option_cost"), + "requires": [str(r) for r in get_list(t, "requires")], + "benefits_inc": inc, + "benefits_dec": dec, + "sections": [str(x) for b in get_list(t, "ship") for x in get_list(b, "section")], + "weapons": [w["filename"] for w in get_list(t, "weapon")], + "allows": [e["to"] for e in edges if e["from"] == t["name"]], + }) + tech_tree = { + "_about": "SOTS1 MasterTechList.tech normalized. family is only written on ~half the nodes; " + "family_inferred is the name prefix (IND/WEP/DRV/...). edges[].pct: per-race availability % as written; " + "a race absent from pct has no override in the file (the engine default -- believed to be 100 -- " + "is code-owned, not asserted here). rp = research-point cost of the edge. " + "requires may name GRP_, satisfied by any tech with group .", + "races": RACES, + "groups": dict(groups), + "nodes": nodes, + "edges": edges, + } + json.dump(tech_tree, open(os.path.join(out, "tech_tree.json"), "w"), indent=1) + + def norm_weapon(w): + d = dict(w) + d["display_name"] = s(w["name"][1:]) if isinstance(w.get("name"), str) and w["name"].startswith("@") else w.get("name") + return d + json.dump({"_about": "All *.weapon files (Weapons/ = player catalog with ids from _weapons.txt; Species/_NPC/weapons = NPC, no ids). " + "Keys lower-cased; repeated keys -> lists; bareword numbers typed.", + "weapons": [norm_weapon(w) for w in sorted(weapons.values(), key=lambda w: (w["scope"], w["stem"].lower()))]}, + open(os.path.join(out, "weapons.json"), "w"), indent=1) + + def norm_section(r): + d = dict(r) + d["display_name"] = s("SECTIONNAME_" + r["stem"]) + d["description"] = s("SECTIONDESC_" + r["stem"]) + d["unlocked_by"] = [t["name"] for t in techs for b in get_list(t, "ship") if r["stem"].lower() in [str(x).lower() for x in get_list(b, "section")]] + return d + json.dump({"_about": "All Species//sections/*.shipsection; id from the race's _shipsections.txt (null = unlisted). " + "Keys lower-cased; repeated keys (bank, option, thruster, requires) -> lists.", + "sections": [norm_section(r) for r in sorted(sections.values(), key=lambda r: (r["race"], r["stem"].lower()))]}, + open(os.path.join(out, "shipsections.json"), "w"), indent=1) + + json.dump(strings, open(os.path.join(out, "strings.json"), "w"), indent=1, ensure_ascii=False) + + with open(os.path.join(out, "tech_tree.dot"), "w") as f: + f.write("digraph sots_tech {\n rankdir=LR; node [shape=box, fontsize=9];\n") + fam_color = {"IND": "#f4d03f", "NRG": "#e74c3c", "SLD": "#3498db", "DRV": "#9b59b6", "TRP": "#e67e22", + "WAR": "#c0392b", "BAL": "#7f8c8d", "BIO": "#2ecc71", "CCC": "#1abc9c", "DRN": "#95a5a6", "XNC": "#d35400"} + for n in nodes: + col = fam_color.get(str(n["family"]), "#ffffff") + label = n["display_name"] or n["name"] + f.write(f' "{n["name"]}" [label="{label}\\n{n["name"]}", style=filled, fillcolor="{col}"];\n') + for e in edges: + lab = f"{e['rp']}" if e["rp"] is not None else "" + if e["pct"]: + lab += "\\n" + " ".join(f"{r[:2]}{v}" for r, v in e["pct"].items()) + f.write(f' "{e["from"]}" -> "{e["to"]}" [label="{lab}", fontsize=7];\n') + f.write("}\n") + + P("## Artifacts") + P("") + for fn in ("tech_tree.json", "weapons.json", "shipsections.json", "strings.json", "schema_stats.json", "crosslink.json", "tech_tree.dot"): + P(f"- `{fn}` ({os.path.getsize(os.path.join(out, fn)) // 1024} KB)") + P(f"- tech_tree.json: {len(nodes)} nodes, {len(edges)} edges; weapons.json: {len(weapons)}; " + f"shipsections.json: {len(sections)}; strings.json: {len(strings)} keys") + print("\n".join(rep)) + return 0 if not fails else 1 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1], sys.argv[2])) diff --git a/save-reader/ref/save-editor-structs.md b/save-reader/ref/save-editor-structs.md new file mode 100644 index 0000000..40abf50 --- /dev/null +++ b/save-reader/ref/save-editor-structs.md @@ -0,0 +1,550 @@ +# Sword of the Stars 1 (SOTS1) — Save-File Struct Reference (Community RE) + +Harvested from two community save-game editors for cross-checking against binary-recovered +serializable types. SOTS1 uses a self-describing "Streamable" serialization system: **field +order + type matter**, and most values are preceded by their own name string. + +Target game version: **SOTS1 v1.8+** (both editors target the 1.8.x line; repo2 says "1.80 +19"). + +## Sources +- **[R1] BardezAnAvatar/Sots.Sots1.SavedGameEditor** — C#. A *complete, ordered, typed* + Streamable parser. Migrated from SourceForge `sots-sge`. This is the authoritative layout + evidence (read/write methods reproduce exact on-disk order). Key files under + `Bardez.Project.SwordOfTheStars.DataStructures/`: + - `BaseSaveStructures.cs` (primitives + framing), `SharedSaveStructures.cs` (coords/colors), + `SaveGameDataStructure.cs` (top level), `SummarySaveStructures.cs`, + `CreateParametersSaveStructures.cs`, `SimulationSaveStructures.cs` (11,974 lines — the bulk), + `CdTableSaveStructures.cs` (combat/AI table). IO: `Bardez.Project.SwordOfTheStars.IO/{Gzip,SaveFileIO}.cs`. +- **[R2] ghbplayer/SOTSedit** — C#/WPF. A *name-tag scanner* (does NOT model layout; it searches + the decompressed blob for length-prefixed field-name strings and reads the value that follows). + Value comes from `Parse.cs` + the field catalog `SOTSEdit.cfg` (friendly-name ⇄ serialized-name + ⇄ type mapping) and race/semantic hacks. Written 2011 "to learn C#"; author calls the parser weak. + +Both editors independently confirm the same primitives, framing markers, race IDs, and color IDs. + +--- + +## 1. File format & serialization mechanics + +### 1.1 Container +- **Whole `.sav` file = gzip stream.** Decompress first (`GZipStream`, standard gzip). [R1 Gzip.cs, R2 gzip.cs] +- Editors work on the **decompressed** byte stream (R1 names its test artifacts `*.sav.inflate.dat`). +- No separate magic/version header is decoded by either editor beyond the top-level Summary block; + version is implied by scenario/field presence, not a numeric version field. Endianness: **little-endian** + throughout (`BitConverter` on x86). + +### 1.2 Decompressed top-level order [R1 SaveGameData.ReadFromStream] +``` +SaveGameData: + 1. SummarySaveStruct summary + 2. CreateParametersSaveStruct createParams + 3. SimSaveStruct sim <-- the giant one (players/systems/fleets/etc.) + 4. CdTable cdTable <-- combat / AI state table +``` + +### 1.3 Primitive encodings [R1 BaseSaveStructures.cs] +Text encoding is **windows-1252** (R1) / ASCII (R2). + +- **StringStruct** (raw string): `Int32 length` + `length` bytes (no NUL terminator counted). + R2 calls this a "BStr": 4-byte little-endian length prefix + ASCII bytes. +- **Padding rule (critical):** every *basic* value struct is **NUL-padded to a 4-byte boundary**. + `PaddingSize = 4`. Padding is computed over `(sizeof(Int32 desc-length) + description.Length + valueBytes)`. +- **Named value fields** (`BasicSaveStruct` subclasses) are each laid out as: + `StringStruct description` (a field-name tag, often the non-descriptive `"."`) → then the value → + then NUL padding to 4 bytes. Concrete leaf types: + | Struct | Payload after description tag | + |---|---| + | `Int32SaveStruct` | 4-byte Int32 | + | `Int64SaveStruct` | 8-byte Int64 | + | `FloatSaveStruct` | 4-byte IEEE Single | + | `BooleanSaveStruct` | 1 byte (0/1), padded to 4 | + | `StringSaveStruct` | nested StringStruct (len+bytes) | + | `ByteArraySaveStruct` | raw bytes (length externally known) | + + So the on-disk shape of a named scalar is: `[len][name-ascii][pad] [value] [pad]`. R2 exploits exactly + this: it locates a field by searching for `[len][name]` and reads the value immediately after. + +### 1.4 Complex-struct framing (the "BEEFBEEF" envelope) [R1 ComplexSaveStruct] +Every **complex** structure is framed: +``` +StringStruct description (NUL-padded to 4) +UInt32 0xBEEFBEEF (begin marker) +... body (ordered child fields) ... +UInt32 0x41104110 (end marker = bitwise NOT of 0xBEEFBEEF) +``` +`0xBEEFBEEF` / `~0xBEEFBEEF (0x41104110)` bracket every complex object — a reliable resync/validation +signature when scanning the binary. (`ISotsStructure` leaf types are NOT framed; only `ComplexSaveStruct`.) + +### 1.5 Array conventions [R1 BaseSaveStructures.cs] +- **ComplexArraySaveStruct**: framed (has description + BEEFBEEF), body = `Int32SaveStruct count` + then `count` × T. +- **NonComplexArraySaveStruct**: NOT framed; body = `Int32SaveStruct count` then `count` × T. + (Distinguishing which arrays are framed vs. not is itself layout evidence — see per-struct notes.) + +### 1.6 Conditional & polymorphic reads (watch for these in the binary) +- **Optional-by-flag:** a boolean/int gate precedes an optional sub-object. + - `SimPlayerColorSaveStruct`: `Int32 colorIndex`; **iff `colorIndex == -1`**, an `RgbColorInt32` + (custom RGB) follows. Otherwise palette index only. + - `SimPlayerDesignDw2SaveStruct` (weapon slot): `Boolean bId`; if true → `Int32 wId`, else → + `StringSaveStruct wfn` (weapon full resource path); then `Int32 dId`. + - `SimFleetShipDetails`: `Boolean hbq` gates `bq`; `Boolean hsp` gates `sp`. Fleet flight-plan/lay + gated by `hfPlan` / `hLay` booleans. + - `SimSystemDetailNvo.isInd` gates independent-colony sub-block; `SimSystemDetailsIndi.hindi`, + `SimSystemDetailsVonNeumann.vnh` similar boolean gates. +- **Polymorphism by string tag:** `SimSvSctObXscn` reads `StringSaveStruct xcsn`, then switches: + `"crowdefs"`, `"gmtrigger"`, `"traps"`, `"indsys"`/default → different body subclass. +- **Polymorphism by fixed position:** `SimScSctObEncObjArray` (grand-menace/encounter objects) reads a + count then dispatches subclass **by index 0..8** in fixed order: + `0 Infest, 1 Dsn, 2 AsteroidMonitor, 3 TD, 4 WD, 5 Hives, 6 Rsuc, 7 Dfts, 8 Ini2`. + +### 1.7 Write-time quirks worth knowing (R2) +- R2 edits in place and cannot safely change string length (it truncates/space-pads to the original + length). R1 rewrites the whole stream and re-pads. If the binary stores string lengths, the game + reads them dynamically (R1 proves round-trip works when re-padded). +- R2 planet OID/PID hack: on-disk **`OID = PID * 16`** (R2 divides by 16 to show a "PlayerID"). + i.e. the raw owner id field is the player index shifted left 4 bits. + +--- + +## 2. Enums / ID tables (agreed by both editors) + +### 2.1 Species / race ID [R2 Parse.cs addRace(); R1 PlayerSlot.FxSp comment] +| ID | Species | +|---|---| +| 0 | Human | +| 1 | Hiver | +| 2 | Tarka(s) | +| 3 | Liir | +| 4 | `_NPC` / AI-rebellion / grand-menace player (R1: "??? AI Rebellion") | +| 5 | Zuul | +| 6 | Morrigi | + +### 2.2 Player color ID (palette index) [R1 PlayerSlot.FxCrId & SimPlayerColor] +`01 Red, 02 Yellow, 03 Blue, 04 Pink/Magenta, 05 Orange, 06 Green, 07 Aqua, 08 Gray, +09 Dark Green, 10 Purple`. Value **-1 ⇒ custom RGB triplet follows** (see §1.6). + +### 2.3 Difficulty [R1 PlayerSettings.Difficulty] +`0 Easy, 1 Normal, 2 Difficult`. + +### 2.4 Sentinel values seen in fields +`0x7FFFFFFF (Int32.MaxValue)` used as "tag"/unset (PlayerSlot.tag); +`team = -1 (0xFFFFFFFF)` = no team; R2: value `-1` = field absent in this save. +No named C# enums exist for tech IDs / weapon families — techs and weapons are **string resource +names** (e.g. tech `tNm`, weapon `wfn`), not numeric enums. Weapon *family* enumeration lives only in +the CdAi combat block as `aiSitWepFams` (opaque int set). + +--- + +## 3. Summary block [R1 SummarySaveStructures.cs] (complex) + +### SummarySaveStruct (fields in on-disk order) +1. `StringSaveStruct gameName` +2. `Int32 turn` +3. `Int32 numSys` (system count) +4. `Int32 checkSum` +5. `ComplexArray players` +6. `SessionSaveStruct session` +7. `Int32 mapShape` +8. `Int32 incMod` (income modifier) +9. `Int32 resMod` (research modifier) +10. `Boolean alliances` +11. `Boolean teams` +12. `Boolean encounters` +13. `StringSaveStruct scenario` + +### PlayerSlotWrapper (complex): `{ PlayerSlotSaveStruct slot; Int32 rank; }` + +### PlayerSlotSaveStruct (complex) — new-game slot definition, ordered: +`Boolean isPlay, isDead, isReq, isRec, isFxNm` → `String fxNm` (fixed name) → +`Boolean isFxSp` → `Int32 fxSp` (species, §2.1) → `Boolean isFxCr` → +`NestedInt32 fxCrId` (color, §2.2) → `Boolean isFxBd` → `String fxBd` (badge) → +`Boolean isFxAv` → `String fxAv` (avatar) → `Int32 tag` (often 0x7FFFFFFF) → +`Int32 pwd` (password) → `Int32 team` (-1=none) → `PlayerSettingsSaveStruct settings`. + +### PlayerSettingsSaveStruct (complex): +`Int32 initialTreasury, initialColonies, initialTechnologies, difficulty (§2.3)`. + +### SessionSaveStruct (complex) → `TmrsSaveStruct tmrs`: +`Int32 tstl (0x7F7FFFFF), tctl (0x42700000), tqtl (0x7F7FFFFF), tqtle (0)` — timer limits (float bit-patterns stored as int). + +--- + +## 4. CreateParameters block [R1 CreateParametersSaveStructures.cs] (complex) + +### CreateParametersSaveStruct (ordered): +`String name; Int32 id; Int32 rSeed (random seed); Int32 aid; String key;` +`MapPSaveStruct mapP;` `Int32 mapS; Int32 mapF; Int32 nSys;` `Float rEnc (random-encounter rate);` +`Int32 sDist;` `Float sSize; Float sRes;` `Int32 sSuit; Int32 maxP; Int32 aSpec;` +`Boolean bAlly; Int32 nTeam; Boolean tmgrp;` +`Int32 pSav (start savings); Int32 pCol (start colonies); Int32 pTech (start techs);` +`Float incM; Float resM; ScrpSaveStruct scrp;` + +### MapPSaveStruct (complex) — initial map/galaxy generation: +1. `Int32 unknown1` +2. `ComplexArray planetArray` +3. `NonComplexArray> players` (per-player int arrays; "non-complex array of players") +4. `ComplexArray npcArray` (≈ players − 1; independents/NPCs) + +### PlanetSaveStruct (complex) — initial star node geometry: +`SpatialCoordinate coordinates (x,y,z floats)`, `Int32 unknown1..4` +(values seen: `0x7FFFFFFF`, `0x7F7FFFFF`). NOTE: this is the *map-generation* planet record; the +*live* planet/colony state lives in `SimSystemDetailsSaveStruct` (§8). + +### MapPNpc (complex): `Int32 unknown1, unknown2`. ### ScrpSaveStruct (complex): `Int32 spc`. + +### Shared value types [R1 SharedSaveStructures.cs] +- `SpatialCoordinateSaveStruct` (complex): `Float x, y, z`. +- `RgbColorFloat` (leaf): `Float r,g,b`. `RgbaColorFloat` (leaf): `RgbColorFloat rgb; Float a`. +- `RgbColorInt32` (leaf): `Int32 r,g,b`. + +--- + +## 5. Simulation block — top level [R1 SimulationSaveStructures.cs] + +### SimSaveStruct (complex) — the master world state, ordered: +``` +String keyPath +Int32 nMsz, nMlc, nMnx (next-id / size counters) +NonComplexArray playerIds, designIds, systemIds, fleetIds, shipIds, tradeIds +Int32 modCount, frame, gameId +AttributeSaveStruct attribute +RngSaveStruct rng (ByteArray unknownData ~2500 bytes: RNG state) +String gameName +Int32 map, incMod, resMod, enAl, enTm, gOTurn +NestedInt32 gOWinPly +Int32 npcm, npco, npci, npcv, npca, szad, rsad, suad +ComplexArray sprjs (shared/special research projects) +Float randEncAdjustment +Int32 cmbtid (next combat id) +ComplexArray turnstats (per-turn per-player history) +NonComplexArray creps (combat reports) +Int32 ninv, allExc1, allExc2, allExcCF +NonComplexArray players <-- EMPIRES (§6) +SimSpeciesArraySaveStruct species (galaxy species list) +NonComplexArray systems <-- STAR SYSTEMS (§8) +SimNodeGrid2 ndgr2 (node-line / warp graph) +SimTradeManager trdmgr (§9) +SimSpyManager spymgr (Int32 xsid, nspy) +NonComplexArray flt <-- FLEETS (§10) +NonComplexArray acts +SimSvSctOb svSctOb (scenario/encounter objects, §11) +Int32 zdsc, zdsi, zdst (Zuul/system-destroyer counters) +``` + +### Small shared sim types +- `SimPopGSaveStruct` (complex): `Int32 popT; Int32 popS; Int64 popC` (pop type / species / civ count). +- `ResearchSaveStruct` (leaf): `NonComplexArray us; String nm; String ntg`. + `ResearchOptionalSaveStruct`: `Int32 usc, usp`. +- `TurnPly` (leaf): `Int32 ply; PlyHistSaveStruct hist`. +- `PlyHistSaveStruct` (complex): `Int32 ply; PlyHistStatsSaveStruct[] stats`. +- `PlyHistStatsSaveStruct` (complex): `Int64 pop; ComplexArray sacq, slost;` + `Int32 trn, almem, inc, tdinc, sav, col, bat, tch; NonComplexArray cls`. +- `PlyHistStatsSacq` (complex): `Int32 set, ses, seop, senp; NonComplexArray seo`. +- `PlyHistClsSaveStruct` (leaf): `Int32 cls, shpt, shpl, shpk, satt, satl, satk` (ship/sat built/lost/killed by class). + +--- + +## 6. Player / Empire [R1 SimulationSaveStructures.cs] + +### SimPlayerSaveStruct (leaf wrapper): `Int32 playerId; SimPlayerDetailsSaveStruct details`. + +### SimPlayerDetailsSaveStruct (complex) — the empire record, on-disk order: +``` +SimPlayerTechTree techTree <-- TECH TREE (§7) +Int32 homeSystem, playerIndex +String playerName +Int32 species (§2.1) +SimPlayerColorSaveStruct colorId (palette idx or -1 + RGB, §1.6/§2.2) +String badge, avatar +Int32 team, sav (savings) +Float idealSuit, suitTolerance, maxOH +Float resRate, resModifier, resScl (research) +Int32 trm, trp, tra +Float outMod, rebOutMod, scOutMod, incMod, popMod, terraMod (economy multipliers) +Boolean aMine; Float minPure, minRate; Int32 ngts, prGtTrf, gTraf +Int32 cstR, cstE, cstT, maint, shrm, status, elim +Boolean npc, rebAi, reqCL +SimPlayerTeamSaveStruct teamStruct (Int32 alid, al, na, cf) +Int32 hasVac, hasImm, npTrak, hasDisc, hasDiscSp, hasDiscCl, hasEnc, hasEng +SimPlayerEventsSaveStruct events (Int32 evNxId; ComplexArray) +NestedInt32 fngNum +Int32 pvSav, pvMA, aibN +Boolean cnTrd, cnRad, hgs, hadvs, harcc, cnVItl +Float pddm +Int32 bankWrn, bankTrn, bankPr, bankEl +SimPlayerShipRecsEventsSaveStruct shipRecs +Int32 nextPrjId, plcy, pswd, lret, nmeid +Boolean cdp +SimPlayerSpySaveStruct spy2 (Int32 defc2, rtc, evc, ttc) +SimPlayerCivrSaveStruct civR (Float smx; ComplexArray) +Int32 aidf +Boolean srn; Int32 srcTo, lboid, lcid2 +String resTnm +Boolean resErrRoll +SimPlayerModsSaveStruct conMods (3× SimPlayerConModSaveStruct {Float conMod, savMod}) +NonComplexArray ownerIds +NonComplexArray designs <-- SHIP DESIGNS (§7.2) +NonComplexArray droneDesigns +NonComplexArray notes (Int32 ntSys; String ntTxt; Int32 ntTrn) +NonComplexArray pr (Float prm; Int32 prbt) +Boolean hasAiRebellion, cta +NestedInt32 aienf +NonComplexArray nSprj +Int32 nexp, nWeapXcl +ComplexArray ovjs (objectives: Int32 id; Bool cmp; Int32 spr; String dsc; Int32 nid) +ComplexArray dipStats (diplomacy; see below) +ComplexArray comms (Int32 msgt; SimPlayerCommMsg msg) +ComplexArray preps +ComplexArray odes +ComplexArray owep +ComplexArray otch +NestedInt32 aid +Int32 ndeflay, rdtc, tnc +``` +- `SimPlayerDipStat` (complex): `Int32 other; SimPlayerDipStatDetail nap, ally, cf; Int32 deadhome`. + `SimPlayerDipStatDetail` (leaf): `Int32 last_, last_bty, bkn_, bty_`. +- `SimPlayerCommMsg` (complex): `Int32 cid2, snd, rcp, exp, sent, rcpt, sys`. +- `SimPlayerPrepSaveStruct` (complex): `Int32 oid, pid, flds, sav, home, ncol, mpwr, mcls, mmsl, nshp, nsat`. +- `SimPlayerOdes/Owep/Otch` (complex, "old design/weapon/tech" build history): + `Odes {Int32 ontF, otnL, odid, opid}`, `Owep {Int32 ontF, otnL, odet; String owep; Int32 owith}`, + `Otch {Int32 ontF, otnL, odet; String otch; Int32 owith}`. +- `SimPlayerDetailsSpecialProjectT` (leaf): `Int32 sprjT; SimPlayerDetailsSpecialProjectDetails sprj`. + `...Details` (complex): `Int32 stp; SpecialProjectSpi spi; tail (polymorphic AsMon | Tech)`. + `...Spi` (complex): `Int32 sPid, sts; Float cst; Int32 mxC; String name; Int32 trns`. + tail `AsMon`: `Int32 rDn, sys, rMn, rMx; Float rMd`; tail `Tech`: `Int32 rDn; Float aOdd, aInc; String tch; Int32 rCst`. + +--- + +## 7. Tech tree & ship designs + +### 7.1 Tech tree [R1] +- `SimPlayerTechTree` (complex): `NonComplexArray tree; NonComplexArray techs`. +- `SimPlayerTechTreeBranch` (leaf): `String tNm` (tech name); `NonComplexArray branches` (child tech names). +- `SimPlayerTechTreeTech` (leaf), ordered: + `String tNm` (tech name) → `Int32 st` → `Int32 tResCost` (research cost) → + `Int32 tResDone` (progress) → `Int32 tAcq` (turn acquired) → `Int32 tiAcq` (turns-to-acquire) → + `Int32 tbd` → `Boolean tfc` → `Int32 tUnlck` (unlocked flag). + **Techs are identified by string name, not a numeric enum.** + +### 7.2 Ship designs [R1] +- `SimPlayerDesignEntrySaveStruct` (leaf): `Int32 designId; SimPlayerDesignSaveStruct design`. +- `SimPlayerDesignSaveStruct` (complex), ordered: + `Boolean faiDes; Boolean dHide; Int32 dWep; String dName;` `SimPlayerDesignSectionArray sections;` + `Int32 dtc; NonComplexArray dwgv`. +- `SimPlayerDesignSectionArray` (leaf): `Int32 count` + `count` × `SimPlayerDesignSectionEntrySaveStruct`. +- `SimPlayerDesignSectionEntrySaveStruct` (complex): `SimPlayerDesignSectionSaveStruct dsec` (Int32 unknown1, unknown2); + `ComplexArray dgbnk2;` `ComplexArray dOpts`. +- `SimPlayerDesignUnknown1SaveStruct` (complex) → `SimPlayerDesignDw2SaveStruct dw2` (weapon slot, + conditional — see §1.6): `Boolean bId; (bId? Int32 wId : String wfn); Int32 dId`. +- `SimPlayerDesignDwg` (complex): `NonComplexArray wgng`; + `SimPlayerDesignGng` (leaf): `Int32 wgid; SimPlayerDesignDwgWgb wgb` (complex: `Int32 unknown1..3`). + +--- + +## 8. Star systems & planets [R1] + +- `SimSystemSaveStruct` (leaf): `Int32 sysId; SimSystemDetailsSaveStruct details`. +- **`SimSystemDetailsSaveStruct` (complex)** — the live system+colony record, on-disk order: +``` +SpatialCoordinate pos +RgbaColorFloat starColor +Int32 idx +Int32 size (1-10) +Float suit (climate hazard) +Int32 res, aRes, mRes (resources / asteroid / extra) +Boolean noRebAi +Int32 tRes, pop +ComplexArray popG +Float infra +Int32 pvPop; ComplexArray pvPopG; Float pvInfra, pvSuit; Int32 pvRes, pvARes2, pvMRes; Bool pvNoRebAi (previous-turn snapshot) +SimSystemDetailRtsSaveStruct rts (Float sRs, sRt, sRsc, sRtf, sRi, sRoh, sRnr — IO allocations) +Int32 abdn; Boolean dstyd; Int32 tnsOh +Float outMod, repCur, repMax +Int32 ntdev, pbon +ComplexArray pbon2 +Float ibon +Int32 ltis, rbfl, rbtn, rbfr, rbwn, hsrg +NonComplexArray halt (Int32 haltt; Bool haltv) +SimSystemDetailsVonNeumann vnm (Bool vnh gate → details: Bool vnd, vnex3, vnpex3) +String name +SimSystemDetailFlags1 flags1 (Int32 vFlags, eFlags, aFlags, fFlags, gFlags) +Int64 bats2 (recent battles; larger=more recent) +Int64 rcex +SimSystemDetailFlags2 flags2 (Int32 mnRFlags, rfRFlags, clkFlags) +Int32 eggScio, terrFl, tAcq, tfAcq, tDst +ComplexArray dcs; Int32 dsu +ComplexArray cm, pvcm (SimSystemDetailCm: Int32 msp, mv) +ComplexArray cme2 (Int32 mid, mtrT, mn, mtp; ComplexArray mfx; String mdsc) +ComplexArray spies +Int32 pid (owner player id), defF, defSf +SimSystemDetailBq bq (ComplexArray ords; Ord: Int32 desId, con, conleft, sav, ordId) +NonComplexArray adct (Int32 ads, adt) +Int32 numPlgs2 +NonComplexArray flts, gfs, snF, mnF (fleets / gates / stations / monitors present) +NonComplexArray nvos (colonies; see below) +NonComplexArray nve (Int32 ePid, ets, eid) +NonComplexArray nvs (Int32 pid; SimSystemDetailVsPView pview) +SimSystemDetailsIndi indi (Bool hindi gate → indsp, SimPlayerColor indcl, String indnm/indav/indba) +``` +- `SimSystemDetailNvo` (leaf): `Int32 pid, tShn, oId; Boolean isInd; SimSystemDetailNvoIndi indi` + (independent-colony sub-block gated by `isInd`). +- `SimSystemDetailVsPView` (complex — per-player *seen* snapshot of a colony): `Int32 vTrn, pop;` + `ComplexArray pop2; Int32 infra; Float suit; Int32 res, aRes2, mRes; Bool noRebAi;` + `Int32 pbon; ComplexArray pbon2; Float ibon; Int32 terrFl; Bool footer`. + +### 8.1 R2 ⇄ R1 cross-map for planet/colony fields (verifier gold) +R2 finds these serialized tags anywhere in the "Planets" region (between markers `NumSys`…`NdGr2`). +They correspond to fields inside R1's `SimSystemDetailsSaveStruct` / `...VsPView`: + +| R2 serialized tag | type | meaning | R1 field | +|---|---|---|---| +| `Idx` | int | planet/system id | `idx` | +| `Name` | string | name | `name` | +| `Size` | int | 1-10 | `size` | +| `Suit` | float | climate hazard | `suit` | +| `Res` / `ARes2` / `MRes` | int | resources | `res` / `aRes` / `mRes` | +| `Infra` | float | infrastructure | `infra` | +| `ibon` | float | infra bonus | `ibon` | +| `Pop` | int | imperial pop | `pop` | +| `pbon` | int | imperial pop bonus | `pbon` | +| `PopC` | long | civilian pop | (SimPopG `popC` Int64) | +| `OID` | int | owner id (**= PID×16**) | `pid` (owner) | +| `PID` | int | derived player id | (OID/16) | +| `SRt/SRsc/SRtf/SRi/SRoh` | int | IO trade/ship/terraform/infra/overharvest | `rts.sRt/sRsc/sRtf/sRi/sRoh` | +| `Abdn` | short | abandon order | `abdn` | +| `Dstyd` | short | star annihilated | `dstyd` | +| `ltis` | short | last-time-seen | `ltis` | +| `VFlags/EFlags/AFlags/FFlags/GFlags` | int | state flags | `flags1.*` | +| `Bats2` | int | recent combat | `bats2` (Int64 in R1) | +| `nadct` | int | addicted (1=yes) | (in `adct` array) | +| `NumFlts/NumGFs/NumSnF/NumMnF` | int | fleets/gates/stations/monitors | `flts/gfs/snF/mnF` counts | + +Note the **type disagreements** (verifier flags): R2 reads `Abdn`, `Dstyd`, `ltis` as **short (Int16)** +while R1 models them as framed `Int32SaveStruct`; R2 reads `Bats2` as int while R1 uses Int64. R2's +name-scan reads the value bytes directly after the tag+pad, so R2's width is the more literal +on-value-bytes claim for those specific fields; treat as "value is small, low bytes are the datum." + +--- + +## 9. Trade & node grid [R1] +- `SimNodeGrid2` (complex): `ComplexArray paths; Int32 nextId`. + `SimNodeGridPath` (complex): `Int32 npt, npid, npfr(from), npto(to), npctm, npcby, npdtn, npdtf, npenp, npuse, nptf`. +- `SimTradeManager` (complex): `NonComplexArray tradeSectors; Float sctSize; List rt`. + `SimTradeSector` (leaf): `Int32 tradeId; SimTradeSectorTradeSaveStruct trade`. + `SimTradeSectorTradeSaveStruct` (complex): `SpatialCoordinate pos; Int32 tradeSectorGridId;` + `SimTradeSectorTradeCtrSaveStruct tsctr(3 floats); Int32 tssec, tsct, tscr, ptssec, ptsct, ptscr;` + `ComplexArray<...Fwarn {Int32 pId, ntrns}> fwarn; NonComplexArray systems, tsflt`. + `SimTradeSectorRt` (complex): `Int32 tro, trfow, trfr, trfrs, trtow, trto, trtos, trtc`. + +--- + +## 10. Fleets & ships [R1] +- `SimFleet` (leaf): `Int32 fltId; SimFleetDetails flt`. +- **`SimFleetDetails` (complex)**, ordered: + `SpatialCoordinate pos; Int32 pId, locId; FlightPlanContainer fplan (Bool hfPlan gate);` + `String ftName; Int32 ftTrans; SimFleetOrigin ftOrig (Int32 ×3);` + `Int32 ftFlag, ftae, ftpae, ftEnc, ftMs, perm; SpatialCoordinate prvPos;` + `LayContainer lay (Bool hLay gate); NonComplexArray ships`. +- `SimFleetDetailsFlightPlan` (complex): `ComplexArray wpts; Float fPsp2; Int32 fPeta2;` + `Fpogn2 (3 floats); SpatialCoordinate fPdpos; Int32 pnd`. `Wpt`: `Int32 wpt, tp; Nrt {Int32 nrp,nrf,nrt}`. +- `SimFleetShip` (leaf): `Int32 shipId; SimFleetShipDetails ship`. +- **`SimFleetShipDetails` (complex)**, ordered: + `Int32 desId (design id), fltId, plrId; Float range; SimFleetShipHealth health (3 floats: command/mission/drive);` + `Int32 conCap, refCap, repCap, mineCap, plg, act; Boolean dep, atq; Int32 encId;` + `SimFleetShipPrish prish (Int32 prMax; NonComplexArray prSp); Int32 lct, tsd, atsp, tblt;` + `Boolean hbq; SimFleetShipDetailsBq bq (gated by hbq);` + `Boolean hsp; SimFleetShipDetailsSp sp (gated by hsp; two SpPop pop/pPop);` + `NonComplexArray th (Float th, thm)`. + `SimFleetShipDetailsBqOrd`: `Int32 desId, con, conleft, sav, ordId` (same shape as system BqOrd). + +--- + +## 11. Combat reports & scenario/encounter objects [R1] +- `SimCrepSaveStruct` (combat report, complex): `Int32 cid, trn; SpatialCoordinate pos; Int32 sid, auto, dur, cow, cdst, cpk, cpt, cdt, cdi;` + `ComplexArray prep; ComplexArray wrep`. + `SimCrepPrepSaveStruct`: `Int32 plr; Bool ai; Int32 ally, status, mxeng, mxcls, mxmsl;` + `NonComplexArray cls; NonComplexArray sec; Int32 ndam; ComplexArray srep`. + `SimCrepPrepSrepSaveStruct`: `String name; Int32 did, cls; Int64 caps2; Int32 nshp, nfld, nlst, dtak; SimDamsSaveStruct dams`. + `SimDamsSaveStruct`: `Int32 dams, damp, dami, damt`. `SimCrepWrepSaveStruct`: `String wep; SimDams dams`. +- `SimSvSctOb` (complex): `ScnObjStruct scn; NonComplexArray xscn; SimScSctObEncObjArray encObjs`. + `SimSvSctObXscn` = polymorphic-by-string (§1.6): `traps` (ComplexArray), + `gmtrigger` (Int32 gmch), `crowdefs` (Int32 sys; NonComplexArray dsys,des; Float drad), `indsys`/default (empty). +- **`SimScSctObEncObjArray`** = grand-menace/encounter table, dispatched **by fixed index 0-8** (§1.6). + Each subclass carries that menace's state, e.g.: + - `EncInfest` (Hiver infestation): `NonComplexArray asg; ComplexArray infests; Int32 deshive, deslarva`. + - `EncHives`: `Int32 qDesignId; ComplexArray hives; ComplexArray queens; ComplexArray sysMem`. + - `EncAsteroidMonitor`, `EncTD`, `EncWD`, `EncRsuc`, `EncDfts` (Von Neumann; large), `EncIni2`. + +--- + +## 12. CdTable — combat / AI persistence block [R1 CdTableSaveStructures.cs] + +### CdTable (leaf, top of the 4th file-section): `ComplexArray cdt; CdPlayer cdplayer; CdAi[] cdai`. + +### CdPlayer (complex) — **entirely reverse-unlabeled** (fields named `unknown1..35`); shape is known: +`Int32 unknown1(=16); Bool unknown2; Float unknown3; Bool unknown4; Int32 unknown4p5; Bool unknown5..8;` +`Int32 unknown9,10; NonComplexArray unknown11;` +`Int32 unknown12..14; NonComplexArray unknown15,16; NonComplexArray unknown17;` +`NonComplexArray unknown18; Int32 unknown19..21;` +`NonComplexArray unknown22; Int32 unknown23..35`. + +### CdAi (complex) — per-AI-player planner state, ordered: +`AttributeSaveStruct aiAttr; NestedInt32 aiTurnPris; CdAiSit aiSit; NestedInt32 aiPlyHat;` +`ComplexArray prs2; Int32 dsh, nbStab, nmBlst; CdAiAidng aidng;` +`Int32 aiHivJ, sdFlT; NestedInt32 nalat; Int32 lnat, lat;` +`NonComplexArray aiSys; NonComplexArray cmbR; NonComplexArray cl;` +`NonComplexArray prv; NonComplexArray tecs; Int32 fct;` +`ComplexArray apr`. +- `CdAiSit` (complex): `NestedInt32 aiSitSecs; NestedInt32 aiSitWepFams` (**weapon-family set** lives here — opaque ints). +- `CdAiCmbr` (complex): `Int32 crTrnK; Bool crPce; NestedInt32 crSys; CdAiCmbrCrplSv2 crplSv2`. + `...CrplSv2`: `Int32 rpBon, rpBonT, savBonus; Bool maintHf; ComplexArray tacReports`. + `TacReport`: `TrStruct trBy, trTo; TacReportDamage damageStruct; NonComplexArray ships`. + `TrStruct`: `Int32 treHd, treHi, treD, treDp, treDi, treDt, treB`. + `TacReportDamage` (leaf): `Int32 tRid, tRal, tRbal, tRlas, tRmis, tRmin, tRnrg, tRbio, tRbrd; Bool tRsld, tRsldd, tRsldc, tRsldi, tRsldr` + — **damage-by-weapon-family breakdown**: bal(listic)/las(er)/mis(sile)/min(e)/nrg(=energy)/bio/brd(=boarding); sld=shields. + `TacReportShips` (leaf): `Int32 tRships, tRsldr, tRshipL`. + +--- + +## 13. Coverage gaps & contradictions + +**Coverage (what's decoded):** +- R1 decodes essentially the *entire* file top-to-bottom: summary, create-params/map-gen, full sim + (players, tech, designs, systems/colonies incl. per-player fog-of-war snapshots, fleets, ships, + trade, node grid, combat reports, grand-menace/encounter objects) and the CdTable AI block. This is + the most complete community model and the primary Rosetta source. +- R2 decodes only Summary, Player Settings, Players, Species, and Planets (colony) fields, by tag + search — but adds *friendly semantics* and confirms the value-bytes width of several planet fields. + +**Known-unknown fields (labelled `unknown*` in R1 — do NOT treat names as authoritative):** +- All of `CdPlayer` (`unknown1..35`) and `CdAiAidngDnId`, `SimFleetOrigin`, `SimFleetDetailsFlightPlanFpogn2`, + `SimTradeSectorTradeCtr`, `SimPlayerDesignDwgWgb`, `SimSystemDetailNvoIndcl*` bodies. +- `SimFleetShipHealth` three floats guessed as command/mission/drive. +- `PlanetSaveStruct.unknown1..4` (map-gen) unexplained. + +**Not decoded / thin:** +- Tactical/real-time combat geometry: only *reports/summaries* are stored (SimCrep*, CdAiCmbr TacReport). + Per-ship in-battle positions/velocities are not in these editors (likely not in the sim save at all). +- `SimSystemDetailSpy` body is empty in R1 (marked "needs to be populated"); spy detail unresolved. +- RNG state (`RngSaveStruct.unknownData`) is an opaque ~2500-byte blob. +- R1 source comments flag `SimSystemDetailSpiesArray`, `SimSvSctObXscnXsc` and + `SimScSctObEncObjDetails` as incomplete/"wrong" in places — verify encounter bodies against binary. + +**Contradictions between R1 and R2 (reconcile against binary):** +1. **Field widths on planet flags:** R2 reads `Abdn`, `Dstyd`, `ltis` as Int16 and `Bats2` as Int32; + R1 models `abdn/ltis` as Int32 and `bats2` as Int64. → The datum is small; check the true stored + width in the binary struct. +2. **PID vs OID:** R2 asserts `OID = PID*16` (owner id is player index << 4); R1 stores a single `pid` + owner field and does not model the ×16 relationship. → Confirm whether the binary owner field is a + raw index or a shifted/tagged handle. +3. **`_NPC` species id 4:** R2 names it `_NPC`; R1 comment guesses "AI Rebellion". Same numeric id 4, + different label — likely a shared "non-player/rogue" species slot. +4. R2 treats Players and PlayerSettings as separate flat tab regions bounded by marker strings + (`HomeSys`…`ISsp`, `Slot`…`Session`); R1 shows these are actually nested (settings inside the + Summary PlayerSlot, live player data inside SimPlayerDetails). R2's region boundaries + (`Summary`,`Slot`,`Session`,`HomeSys`,`ISsp`,`NumSys`,`NdGr2`,`PlayerIDs`,`DesignIDs`) are useful + **section-marker strings** to locate blocks in the raw binary. + +**High-value binary-scan signatures:** +`0xBEEFBEEF` / `0x41104110` complex-struct brackets; length-prefixed ASCII field-name tags +(`[int32 len][name]`) preceding every named scalar; section marker strings above. diff --git a/save-reader/ref/struct-recovery.md b/save-reader/ref/struct-recovery.md new file mode 100644 index 0000000..f7d5cb9 --- /dev/null +++ b/save-reader/ref/struct-recovery.md @@ -0,0 +1,461 @@ +# Struct recovery via save-field-name xrefs — Sword of the Stars (2006) + +Program `sots` / "Sword of the Stars.exe", ImageBase 0x00400000, 32-bit MSVC. All addresses are VAs. +Method: every on-disk field is tagged with its name string; the name strings live in `.rdata`; the +functions that reference dozens of a struct's names in sequence are its `IStreamable::Read`/`Write`. +Decompiled those, mapped `this+offset` → name → type. Scripts (on CT111 `/root/`): `SerFind.java` +(string→xref→function ranking), `SerDump.java`/`SerDump2.java` (decompile with `DAT_` → literal +substitution), `VtOwner.java`/`VtOwner2.java` (find the owning vftable + RTTI Complete-Object-Locator +offset), `SubWrite.java` (sub-struct Read/Write by class name). Raw decompiles: CT111 `/tmp/serdump/*.c`. + +Reference cross-checked: `save-editor-structs.md` (R1 = Bardez editor, R2 = SOTSedit). + +--- + +## 0. Serialization runtime facts (needed to read the tables) + +### IStreamable vftable shape +Every streamable class has a 3-slot vftable `{ [0] scalar-deleting dtor, [1] Read(Stream&), [2] Write(Stream&) }` +(`Mars::IStreamable::vftable` @ 0x009e22bc = `{0x4f7230, 0x924fb0, 0x924fb0}`). `Mars::StreamableHelper` / +`Mars::VectorHelper` are thin adaptors: slot [1]/[2] call the object's virtual Read/Write (or, for +POD types like Vector3, a free function). + +### this-adjustment (IMPORTANT for offsets) +The serializer is called through the class's **IStreamable sub-vftable**, whose RTTI COL `offset` field +gives the sub-object offset. The vftable slots point straight at the functions (no adjustor thunks; +prologues verified in the exe: `55 8b ec 6a ff 68 …`), so inside each function `this` = object + COL +offset. **Absolute member offset = decompiled offset + COL offset.** Tables below give both. + +| Class | IStreamable vftable (COL offset) | Read | Write | Primary vftable | +|---|---|---|---|---| +| `Game::ServerSystem` (: StarSystem : StarMapNode) | 0x00a2043c (**+8**) | `FUN_0075d4b0` | `FUN_00749630` | (StarSystem primary 0x00a200e4) | +| `Game::StarSystem` / `ClientSystem` (StarMapNode part) | 0x00a200d4 / 0x00a20144 (+8) | `FUN_00727790` | `FUN_00727820` | 0x00a200e4 / 0x00a20154 | +| `Game::StarMapNode` | 0x00a1e620 (+8) | `FUN_00727790` | `FUN_00727820` | 0x00a1e630 | +| `Game::ServerPlayer` (: StrategyPlayer) | 0x00a32794 (**+0x3a0 = 928**) | `FUN_008804d0` | `FUN_008563e0` | 0x00a327a4 (COL 0, 8 slots) | +| `Game::StarShip` | 0x00a31408 (**+8**) | `FUN_00853fa0` | `FUN_008291f0` | 0x00a31418 (2 slots) | +| `Game::StarFleet` (: StarMapNode) | 0x00a1d5f8 (**+8**) | `FUN_00702470` | `FUN_00701070` | 0x00a1d608 | +| `Game::StrategyServer` (whole sim block) | 0x00a26084 (+0) | `FUN_007d27a0` | `FUN_0079fa70` | 0x00a26034 (COL 4) | +| `Game::StarSystem::PlayerView` | 0x00a201ac (+0) | `FUN_00752af0` | `FUN_007492d0` | — | +| `Game::StarSystem::OutputRates` (POD, via helper) | helper 0x00a1f884 | `FUN_007472a0` | `FUN_00745190` | — | +| `Game::Population` | 0x009f90f0 (+0) | `FUN_005390c0` | `FUN_00537ef0` | — | +| `Game::PopulationGroup` | 0x009f8d50 (+0) | `FUN_00536a80` | `FUN_00536af0` | — | +| `Game::IndependenceInfo` | 0x00a2005c (+0) | `FUN_00748df0` | `FUN_00748ee0` | — | +| `Game::Morale` / `MoraleEvent` | 0x00a1f7c8 / 0x00a2003c | `FUN_00744dd0` / `FUN_007490b0` | `FUN_00744ea0` / `FUN_007491b0` | — | +| `Game::ShipBuildOrder(Def)` | 0x00a0c160 / 0x00a0ad08 | `FUN_00813770` | `FUN_00813800` | — | +| `Game::PlayerNotes` | 0x00a21948 | `FUN_00813250` | `FUN_008132b0` | — | +| `Game::SpyReport` | 0x00a32b2c | `FUN_008843d0` | `FUN_00828ec0` | — | +| `Game::PlayerReport` (preps) | 0x00a21440 | `FUN_008200a0` | `FUN_00817480` | — | +| `Game::DiplomacyStats` | 0x00a21430 | — | `FUN_00818cb0` | — | +| `Game::FlightPlan` / `::Waypoint` / `NodeRoute` | 0x00a1d50c / 0x00a1d39c / 0x00a1cbdc | `FUN_00704c70` / `FUN_00701860` / `FUN_006e2260` | `FUN_00700f60` / `FUN_00700ed0` / `FUN_006e22e0` | — | +| `Game::PrisonerHold` | 0x009fe130 | `FUN_0056eb00` | `FUN_0056ec00` | — | +| `Game::EventStorage` / `PlayerAlliances` / `ShipHealth` / `PlayerColorID` / `Mars::Vector3` | — | — | `FUN_00825cc0` / `FUN_006d2e10` / `FUN_00813e50` / `FUN_0053c080` / `FUN_008a60d0` | — | + +### Stream primitive API (writer side; `Stream` object vftable, `this` = stream) +| call | meaning | wrapper used by serializers | +|---|---|---| +| vft+0x18 `(name, std::string*)` | write string | `FUN_008b9d70(stream,name,std::string*)` | +| vft+0x1c `(name, byte)` | write bool | `FUN_008b9c20(stream,name,bool*)` | +| vft+0x20 `(name, float)` | write float | `FUN_008b9be0(stream,name,float*)` | +| vft+0x24 `(name, int, default=-1)` | write int32 | `FUN_008b9d50(stream,name,int*)`; `FUN_008b9d00(stream,name,int16*)` (widens short→int) | +| vft+0x28 `(name, IStreamable-helper*)` | write nested object (BEEFBEEF frame) | inline `StreamableHelper{vft, 0, T*}` | +| vft+0x30 `(name, ptr, nbytes)` | write raw bytes | used for 8-byte Int64s | +| `FUN_00816490(stream,name,obj*)` | write **handle id** = `obj ? obj->id(+4) : 0` | NetworkObject id at +4 | +| `FUN_008b9c60(stream,name,int64*)` | write int64 (PopC) | | +Reader side mirrors: `FUN_008b9bc0` float, `FUN_008b9d20` int, `FUN_008b9c00` bool, `FUN_008b9d90` string, +`FUN_008b9c40` int64, `FUN_008b9cd0` short, `FUN_008164d0(stream,name)` handle→object* lookup, +stream vft+0x10 int-by-ref (returns found flag), vft+0x14 nested object (NULL helper = skip/legacy). +Readers accept legacy tags (`ISuit`, `Income`, `HPop`, `Bats`, `Builds`, `Clr`, `SensMod`, `ExPopSys`, +`NShps`, `SysID`, `TrdID`, `Caps`, `GtTrf`, `FtSens`, `FtInc`, `Pris`, `NumPlgs`, `lcid`, `morev`, `cme`) +by reading them into scratch/NULL — these are pre-1.8 fields, NOT members. + +Common Mars/MSVC layouts seen: `std::string` = 0x1c bytes (MSVC10 `_Bx` union@0, size@0x14, res@0x18; +`FUN_008b9d70` does the `res>=16 ? heap : sso` check); `std::vector` = {begin@0, end@4, cap@8}; +`std::map/set` node = {left@0, parent@4, right@8, key@0xc, value@0x10, …, color/isnil bytes at tail}; +`std::list` = {head*@0, size@4}. `Mars::NetworkObject` = {vptr@0, int id@4}. + +--- + +## 1. `Game::ServerSystem` (= live star system + colony record; R1 `SimSystemDetailsSaveStruct`) + +Serializers: Write `FUN_00749630` @ 0x00749630 (3453 B), Read `FUN_0075d4b0` @ 0x0075d4b0 (8320 B, has +legacy branches). Both begin with `StarMapNode::Write/Read` (`FUN_00727820`/`FUN_00727790`) which +emits `Pos`. `this` = obj+8. Base layout: `+0` primary vptr (StarSystem 0x00a200e4), `+4` NetworkObject id, +`+8` IStreamable vptr, `+0xc` HandleObject vptr, `+0x10` owner pointer (`*(+0x10)->+0x50` = player-object +table indexed by map key), `+0x18` Pos. + +| abs off | rel(this+8) | type | save name | notes | +|---|---|---|---|---| +| 0x18 | 0x10 | `Mars::Vector3` (3 floats) | `Pos` | via StarMapNode; on disk 3 unnamed floats | +| 0x4c | 0x44 | float | `R` | starColor.r | +| 0x50 | 0x48 | float | `G` | | +| 0x54 | 0x4c | float | `B` | | +| 0x58 | 0x50 | float | `A` | | +| 0x5c | 0x54 | int | `Idx` | system index | +| 0x60 | 0x58 | int | `Size` | 1–10 | +| 0x64 | 0x5c | float | `Suit` | climate hazard (legacy `ISuit` discarded) | +| 0x68 | 0x60 | int | `Res` | | +| 0x6c | 0x64 | int | `ARes2` | (legacy `ARes` read then overwritten) | +| 0x70 | 0x68 | int | `MRes` | | +| 0x74 | 0x6c | int | `TRes` | | +| 0x78..0x7a | 0x70..0x72 | bool[3] | `haltv` | written as `haltc`=3, then 3×(`haltt`=i, `haltv`=v[i]) | +| 0x7c | 0x74 | float | `OutMod` | | +| 0x80 | 0x78 | int | `TAcq` | | +| 0x84 | 0x7c | int | `TFAcq` | | +| 0x88..0xa3 | 0x80 | `StarSystem::OutputRates` (0x1c) | `Rts` | see §1.1; nested object | +| 0xa4 | 0x9c | `BuildQueue*` | `BQ` | written only if owner (`PID`) non-null | +| 0xa8..0xc3 | 0xa0 | `std::string` | `Name` | | +| 0xc4 | 0xbc | bool | `Abdn` | **bool in memory** (R2 "short" is just the value byte; R1 Int32 is the framing) | +| 0xc5 | 0xbd | bool | `Dstyd` | | +| 0xc6 | 0xbe | bool | `vnh` | gate: if true → `vnd`,`vnex3`,`vnpex3` | +| 0xc7 | 0xbf | bool | `vnd` | | +| 0xc8 | 0xc0 | bool | `vnex3` | | +| 0xc9 | 0xc1 | bool | `vnpex3` | | +| 0xcc | 0xc4 | int | `VFlags` | written by value (int) | +| 0xd0 | 0xc8 | int | `EFlags` | | +| 0xd4 | 0xcc | int | `AFlags` | | +| 0xd8 | 0xd0 | int | `FFlags` | | +| 0xdc | 0xd4 | int | `GFlags` | | +| 0xe0 | 0xd8 | int | `MnRFlags` | | +| 0xe4 | 0xdc | int | `RfRFlags` | | +| 0xe8 | 0xe0 | int | `ClkFlags` | | +| 0xf0 | 0xe8 | **int64** | `Bats2` | raw 8 bytes (R1 correct; R2 Int32 reads low half). Legacy `Bats` int | +| 0xf8 | 0xf0 | **int64** | `rcex` | raw 8 bytes | +| 0x100 | 0xf8 | `ServerPlayer*` | `PID` | owner; written as handle id (`->+4`) | +| 0x104..0x117 | 0xfc | `Population` (0x14) | `dcs` | nested | +| 0x118 | 0x110 | float | `dsu` | | +| 0x11c..0x13b | 0x114 | `Morale` (0x20: vptr + int[7]) | `cm` | nested | +| 0x13c..0x147 | 0x134 | `vector` | `cme2` | VectorHelper | +| 0x14c..0x16b | 0x144 | `Morale` | `PvCM` | | +| 0x16c..0x177 | 0x164 | `vector` | `NumFlts` + n×`Flt` | ids via handle | +| 0x17c | 0x174 | float | `RepCur` | | +| 0x180 | 0x178 | float | `RepMax` | | +| 0x184 | 0x17c | int | `EggScio` | | +| 0x188 | 0x180 | bool | `NoRebAI` | | +| 0x189 | 0x181 | bool | `PvNoRebAI` | | +| 0x18c | 0x184 | int | `Pop` | imperial pop | +| 0x190 | 0x188 | float | `Infra` | | +| 0x194 | 0x18c | int | `pbon` | | +| 0x198 | 0x190 | float | `ibon` | | +| 0x19c | 0x194 | int | `TerrFl` | | +| 0x1a0..0x1b3 | 0x198 | `Population` | `Pop2` | civilian pop groups (R1 `popG`) | +| 0x1b4..0x1c7 | 0x1ac | `Population` | `pbon2` | | +| 0x1c8 | 0x1c0 | `IndependenceInfo*` | `hindi` + `indi` | `hindi` = ptr!=NULL | +| 0x1cc..0x1d7 | 0x1c4 | `vector` | `spies2` | VectorHelper | +| 0x1dc | 0x1d4 | int | `rbfl` | written by value | +| 0x1e0 | 0x1d8 | bool | `hsrg` | | +| 0x1e4..0x1ff | 0x1dc | int[7] | `nadct`,(`ads`=i,`adt`=v) | addiction table: count of non-zero entries then sparse (index,value) pairs | +| 0x200 | 0x1f8 | int | `PvPop` | previous-turn snapshot block | +| 0x204 | 0x1fc | float | `PvInfra` | | +| 0x208 | 0x200 | float | `PvSuit` | | +| 0x20c | 0x204 | int | `PvRes` | | +| 0x210 | 0x208 | int | `PvARes2` | | +| 0x214 | 0x20c | int | `PvMRes` | | +| 0x218..0x22b | 0x210 | `Population` | `PvPop2` | | +| 0x238 | 0x230 | `StarFleet*` | `DefF` | handle id | +| 0x23c | 0x234 | `StarFleet*` | `DefSF` | handle id | +| 0x240..0x24b | 0x238 | `vector` | `NumGFs` + n×`GF` | gates | +| 0x250..0x25b | 0x248 | `vector` | `NumSnF` + n×`SnF` | stations | +| 0x260..0x26b | 0x258 | `vector` | `NumMnF` + n×`MnF` | monitors | +| 0x274 / 0x278 | 0x26c / 0x270 | `std::map` head / size | `NVO` + entries | colonies, see §1.2 | +| 0x284 / 0x288 | 0x27c / 0x280 | `std::map` head / size | `NVE` + entries | §1.2 | +| 0x294 / 0x298 | 0x28c / 0x290 | `std::map` head / size | `NVs` + entries | per-player `pview`, §1.3 | +| 0x2a8..0x2b3 | 0x2a0 | `vector` | `NumPlgs2` + n×(`PlgT`=plg->+4, `Plg` obj) | | +| 0x2b8 | 0x2b0 | int | `TnsOH` | | +| 0x2bc | 0x2b4 | int | `TDst` | | +| 0x2c4 | 0x2bc | int | `ntdev` | | +| 0x2c8 | 0x2c0 | int | `ltis` | **int** (R2 "short" wrong width) | +| 0x2cc | 0x2c4 | int | `rbtn` | | +| 0x2d0 | 0x2c8 | int | `rbfr` | by value | +| 0x2d4 | 0x2cc | int | `rbwn` | | + +Object size ≥ 0x2d8. On-disk order = R1 §8 exactly (Pos, RGBA, Idx, Size, Suit, Res, ARes2, MRes, NoRebAI, +TRes, Pop, Pop2, Infra, PvPop, PvPop2, PvInfra, PvSuit, PvRes, PvARes2, PvMRes, PvNoRebAI, Rts, Abdn, Dstyd, +TnsOH, OutMod, RepCur, RepMax, ntdev, pbon, pbon2, ibon, ltis, rbfl, rbtn, rbfr, rbwn, hsrg, halt*, vn*, Name, +*Flags, Bats2, rcex, Mn/Rf/ClkFlags, EggScio, TerrFl, TAcq, TFAcq, TDst, dcs, dsu, cm, PvCM, cme2, spies2, PID, +DefF, DefSF, BQ, nadct/ads/adt, NumPlgs2…, NumFlts/GFs/SnF/MnF, NVO, NVE, NVs, hindi/indi). + +Members that are only ever read with a NULL/scratch target (not stored): `ISuit`, `Income`, `HPop`, `Builds` +(+`Con`,`Sav`,`ConLeft`,`OrID`,`DesID` — old inline build queue), `Slvs`, `dct`, `cme`, `Bats`, `NumPlgs`. + +### 1.1 `Game::StarSystem::OutputRates` (POD, 0x1c) — Write `FUN_00745190` +Memory order ≠ disk order: `+0x00 float SRt`, `+0x04 SRsc`, `+0x08 SRtf`, `+0x0c SRi`, `+0x10 SRoh`, +`+0x14 SRs`, `+0x18 int SRnr`. Disk order: SRs, SRt, SRsc, SRtf, SRi, SRoh, SRnr. Reader: if `SRs` tag is +absent, reads 5 unnamed floats (legacy). + +### 1.2 Colony maps `NVO` / `NVE` (std::map keyed by player-table index) +Write emits `NVO`=size, then per node: `PID` = handle id of `owner->+0x50[key]` (player object table), +`TShn` = **int16** at node+0x12 (value+2), `OID` = int at node+0x14 (value+4) by value, `isind` bool at +node+0x18 (value+8), `indi` = inline `IndependenceInfo` at node+0x1c (value+0xc, 0x70 bytes). Node isnil +byte at +0x8d ⇒ value size 0x7c. `NVE` nodes: `EPid` handle (key→player), `ETS` int16 @ node+0x12, `Eid` +int @ node+0x14 (isnil @ +0x19 ⇒ value 8 bytes). +`OID` is a stored int, distinct from `PID`; the R2 claim "OID = PID×16" is an id-allocation pattern, not +a derivation in this code (open question — check the HandleObject id allocator). + +### 1.3 `Game::StarSystem::PlayerView` (per-player seen snapshot) — Write `FUN_007492d0`, Read `FUN_00752af0` +Stored inline as map value at node+0x10 (`NVs`; node isnil @ +0xad ⇒ value ≈ 0x9c). +`+0 vptr (0x00a201ac)`, `+8 int VTrn`, `+0xc int Pop`, `+0x10 Population Pop2 (0x14)`, `+0x24 float Infra`, +`+0x28 float Suit`, `+0x2c int Res`, `+0x30 int ARes2`, `+0x34 int MRes`, `+0x38 bool NoRebAI`, `+0x3c int pbon`, +`+0x40 Population pbon2`, `+0x54 float ibon`, `+0x58 int TerrFl`; trailer bool `footer`=1. Reader also accepts +legacy `ARes`, `PvPop/PvInfra/PvSuit/PvRes/PvARes/PvARes2/PvMRes/PvNoRebAI` into the same slots. +(R1 lists `Int32 infra` — it is a **float**.) + +### 1.4 `Game::Population` (0x14) / `Game::PopulationGroup` (0x18 stride) +Population: `+0 vptr`, `+4/+8/+0xc vector`; Write emits `PopNG` = count of groups with +PopC>0 (or ≥0 with low word ≠0), then each as nested `PopG`. PopulationGroup: `+4 int PopT`, `+8 int PopS`, +`+0x10 int64 PopC` (R1 `popT,popS,popC` ✓). + +### 1.5 `Game::IndependenceInfo` (0x70) — Write `FUN_00748ee0` +`+4 int indsp`, `+8 PlayerColorID indcl` (nested), `+0x1c string indnm`, `+0x38 string indav`, `+0x54 string indba`. + +### 1.6 `Game::Morale` / `Game::MoraleEvent` +Morale (0x20): `+0 vptr`, `+4 int[7]`; disk: `mnsp`=n then n×(`msp`=index, `mv`=value) (reader tolerates +missing `mnsp` → 7 fixed entries, skipping index 4). MoraleEvent: `+4 mid`, `+8 mtr`, `+0xc mn`, `+0x10 mtp` +(ints), `+0x14 Morale mfx`, `+0x34 string mdsc`. + +### 1.7 `Game::ShipBuildOrder` (build-queue entry) — Write `FUN_00813800` +`+4 int desID`, `+8 int con`, `+0xc int sav`, `+0x10 int conleft`, `+0x14 int ordID`; disk order desID, con, +conleft, sav, ordID (R1 ✓). + +### Where is "Planet"? +`Game::Planet : Actor` (vft 0x009ef144) is a **render/scene actor** and is not streamed. The colony/planet +state the save calls "planet" (R2's Idx/Name/Size/Suit/Res/Infra/Pop/OID…) is entirely in `ServerSystem` +above plus `PlayerView`. The CreateParameters `PlanetSaveStruct` (x,y,z + 4 ints) is map-gen input +(`StarMapParams`), not touched here. + +--- + +## 2. `Game::ServerPlayer` (empire; R1 `SimPlayerDetailsSaveStruct`) + +Serializers: Write `FUN_008563e0` @ 0x008563e0 (4040 B), Read `FUN_008804d0` @ 0x008804d0 (7647 B). +IStreamable sub-object at **+0x3a0** (COL offset 928); `this` = obj+0x3a0, so decompiled offsets are +negative for most members. Primary vptr @+0 (0x00a327a4, StrategyPlayer shape), NetworkObject id @+4. + +| abs off | rel(this+0x3a0) | type | save name | notes | +|---|---|---|---|---| +| 0x28 | -0x378 | int | `PlyrIdx` | | +| 0x2c | -0x374 | `ServerSystem*` | `HomeSys` | handle id | +| 0x30/0x34 | -0x370/-0x36c | `vector` | `NumOwn` + n×`OwnId` | handle ids (R1 `ownerIds`) | +| 0x40..0x5b | -0x360 | `std::string` | `PlryName` | | +| 0x5c | -0x344 | int | `Species` | 0 Human … 6 Morrigi | +| 0x60 | -0x340 | `PlayerColorID` (4 B) | `ClrID` | nested; §2.1 (legacy `Clr` int skipped) | +| 0x74..0x8f | -0x32c | `std::string` | `Bdg` | badge | +| 0x90..0xab | -0x310 | `std::string` | `Avt` | avatar | +| 0xac | -0x2f4 | int | `Team` | | +| 0xb0 | -0x2f0 | float | `IdealSuit` | | +| 0xb4 | -0x2ec | float | `SuitTol` | | +| 0xb8 | -0x2e8 | float | `MaxOH` | | +| 0xbc | -0x2e4 | float | `ResRate` | | +| 0xc0 | -0x2e0 | float | `ResMod` | | +| 0xc4 | -0x2dc | float | `ResScl` | | +| 0xd0 | -0x2d0 | **float** | `TRM` | R1 says Int32 — it is float | +| 0xd4 | -0x2cc | int | `TRA` | note memory order TRA before TRP | +| 0xd8 | -0x2c8 | int | `TRP` | | +| 0xe4/0xe8 | -0x2bc/-0x2b8 | `vector` | `NumDes` + n×(`DesID`=d->+0xa4, `Des` obj) | current designs | +| 0xf4 | -0x2ac | `TechTree*` | `TechTree` | first field on disk | +| 0xf8 | -0x2a8 | bool | `Elim` | | +| 0xfb | -0x2a5 | bool | `NPC` | | +| 0xfc | -0x2a4 | bool | `RebAI` | | +| 0xfd | -0x2a3 | bool | `ReqCL` | | +| 0xfe | -0x2a2 | bool | `AIBn` | | +| 0xff | -0x2a1 | bool | `CnTrd` | | +| 0x100 | -0x2a0 | bool | `CnRad` | | +| 0x101 | -0x29f | bool | `CnVItl` | | +| 0x102 | -0x29e | bool | `hgs` | | +| 0x103 | -0x29d | bool | `hadvs` | | +| 0x104 | -0x29c | bool | `harcc` | | +| 0x108 | -0x298 | float | `pddm` | | +| 0x10c..0x117 | -0x294 | float[3] | `ConMod` ×3 | interleaved on disk as 3×(ConMod[i], SavMod[i]) | +| 0x118..0x123 | -0x288 | float[3] | `SavMod` ×3 | | +| 0x124 | -0x27c | float | `OutMod` | | +| 0x128 | -0x278 | float | `RebOutMod` | | +| 0x12c | -0x274 | float | `ScOutMod` | | +| 0x130 | -0x270 | float | `PopMod` | (legacy `SensMod` float, `ExPopSys` int skipped between IncMod/PopMod/TerraMod) | +| 0x134 | -0x26c | float | `TerraMod` | | +| 0x138 | -0x268 | bool | `AMine` | | +| 0x13c | -0x264 | float | `MinPure` | | +| 0x140 | -0x260 | float | `MinRate` | | +| 0x144 | -0x25c | int | `NGts` | | +| 0x148 | -0x258 | int | `PrGtTrf` | | +| 0x14c | -0x254 | int | `GTraf` | | +| 0x150 | -0x250 | **float** | `CstR` | R1 Int32 → float | +| 0x154 | -0x24c | **float** | `CstE` | | +| 0x158 | -0x248 | **float** | `CstT` | | +| 0x15c | -0x244 | int | `Maint` | (legacy `NShps` skipped) | +| 0x160 | -0x240 | **float** | `shrm` | | +| 0x164 | -0x23c | int | `Status` | by value | +| 0x168..0x177 | -0x238 | `PlayerAlliances` {int ALid, AL, NA, CF} | `Team` (2nd) | nested (R1 `teamStruct`) | +| 0x178/0x17c | -0x228/-0x224 | `vector` | `NumLeg` + n×(`DesID`,`Des`) | legacy/drone designs (R1 `droneDesigns`) | +| 0x188 | -0x218 | int | `PvSav` | | +| 0x18c | -0x214 | bool | `PvMA` | | +| 0x19c | -0x204 | int | `HasDisc` | by value | +| 0x1a0 | -0x200 | int | `HasDiscSp` | | +| 0x1a4 | -0x1fc | int | `HasDiscCl` | | +| 0x1a8 | -0x1f8 | int | `HasEnc` | | +| 0x1ac | -0x1f4 | int | `HasEng` | | +| 0x1b0 | -0x1f0 | `ShipRecords` (inline) | `ShipRecs` | Write `FUN_008176a0` | +| 0x1f4 | -0x1ac | `vector` | `Ojvs` | VectorHelper | +| 0x204/0x208 | -0x19c/-0x198 | `vector<{int xid,xmin,xmax; float xper}>` (16 B) | `Nexp` + n×(`xid`,`xmin`,`xmax`,`xper`) | R1 misses the per-entry body | +| 0x214/0x218 | -0x18c/-0x188 | `vector` | `NWeapXcl` + n×`WeapXcl` | | +| 0x230 | -0x170 | `vector` | `dipstats` | §2.2 | +| 0x240 | -0x160 | `CommMessageContainer*` | `comms` | | +| 0x244 | -0x15c | `vector` | `preps` | §2.3 | +| 0x254 | -0x14c | `vector` | `odes` | | +| 0x264 | -0x13c | `vector` | `owep` | | +| 0x274 | -0x12c | `vector` | `otch` | | +| 0x284 | -0x11c | int | `Sav` | savings | +| 0x288 | -0x118 | int | `HasImm` | by value | +| 0x28c | -0x114 | int | `HasVac` | | +| 0x290 | -0x110 | int | `NPTrk` | | +| 0x294 | -0x10c | `Tech*` (current research) | `ResTNm` | writes `tech ? tech->name(+4) : ""` | +| 0x298 | -0x108 | `FleetNameGenerator*` | `FNG` | | +| 0x29c | -0x104 | `EventStorage` (inline) | `Events` | {`EvNxID`@+0x14, `Events` vector@+4} | +| 0x2b8/0x2bc | -0xe8/-0xe4 | `std::list` head/size | `NumNotes` + n×`Nts` | node value at +0x10: `NtSys`@+4,`NtTxt` str@+8,`NtTrn`@+0x24 | +| 0x2c4 | -0xdc | int | `BnkWrn` | by value | +| 0x2c8 | -0xd8 | int | `BnkTrn` | | +| 0x2cc | -0xd4 | int | `BnkEl` | | +| 0x2d0 | -0xd0 | int | `BnkPr` | | +| 0x2d8 | -0xc8 | int | `plcy` | by value | +| 0x2dc..0x2f7 | -0xc4 | **`std::string`** | `pswd` | R1 says Int32 — it is a string | +| 0x2f8 | -0xa8 | bool | `Srn` | | +| 0x2fc | -0xa4 | `obj*` | `SrnTo` | handle id (R1 `srcTo`) | +| 0x300 | -0xa0 | int | `lboid` | | +| 0x304 | -0x9c | int | `lcid2` | by value (legacy `lcid`) | +| 0x30c | -0x94 | float | `IncMod` | | +| 0x310 | -0x90 | `vector` | `aid` | | +| 0x320/0x324 | -0x80/-0x7c | `vector` | `ndeflay` + n×`deflay` | | +| 0x330 | -0x70 | bool | `cdp` | | +| 0x334 | -0x6c | `SpyReport*` | `spy2` | §2.4 | +| 0x338/0x33c | -0x68/-0x64 | `vector` (0x20 stride) | `rdtc` + n×`rdt` | | +| 0x368 | -0x38 | int | `aidf` | by value | +| 0x370 | -0x30 | `CivilianRatios` (inline) | `civr` | Write `FUN_0082c740` | +| 0x39c | -4 | int | `tnc` | written as max(v,1) | +| **0x3a0** | 0 | vptr | — | IStreamable sub-vftable 0x00a32794 | +| 0x3a4/0x3a8 | +4/+8 | `vector<{float PRm; int PRBt}>` | `NumPR` + n×(`PRm`,`PRBt`) | | +| 0x3b4 | +0x14 | bool | `ResErrRoll` | | +| 0x3b5 | +0x15 | bool | `cta` | | +| 0x3b8 | +0x18 | `AIRebellion*` | `HasAIR` + `AIR` | gate = ptr!=NULL | +| 0x3bc | +0x1c | `AIEncounterFlags*` | `AIEnf` | | +| 0x3c0/0x3c4 | +0x20/+0x24 | `vector` | `NSprj` + n×(`SprjT`=p->+0x3c, `Sprj`) | | +| 0x3d0 | +0x30 | int | `NextPrjID` | | +| 0x3d4 | +0x34 | int | `lret` | | +| 0x3dc | +0x3c | int | `nmeid` | | + +Object size ≥ 0x3e0. Disk order = R1 §6 (TechTree, HomeSys, PlyrIdx, PlryName, Species, ClrID, Bdg, Avt, +Team, Sav, IdealSuit, SuitTol, MaxOH, ResRate, ResMod, ResScl, TRM, TRP, TRA, OutMod, RebOutMod, ScOutMod, +IncMod, PopMod, TerraMod, AMine, MinPure, MinRate, NGts, PrGtTrf, GTraf, CstR/E/T, Maint, shrm, Status, Elim, +NPC, RebAI, ReqCL, Team{ALid,AL,NA,CF}, HasVac, HasImm, NPTrk, HasDisc, HasDiscSp, HasDiscCl, HasEnc, HasEng, +Events, FNG, PvSav, PvMA, AIBn, CnTrd, CnRad, hgs, hadvs, harcc, CnVItl, pddm, BnkWrn/Trn/Pr/El, ShipRecs, +NextPrjID, plcy, pswd, lret, nmeid, cdp, spy2, civr, aidf, Srn, SrnTo, lboid, lcid2, ResTNm, ResErrRoll, +3×(ConMod,SavMod), NumOwn/OwnId, NumDes/DesID/Des, NumLeg/DesID/Des, NumNotes/Nts, NumPR/PRm/PRBt, HasAIR/AIR, +cta, AIEnf, NSprj/SprjT/Sprj, Nexp/xid/xmin/xmax/xper, NWeapXcl/WeapXcl, Ojvs, dipstats, comms, preps, odes, +owep, otch, aid, ndeflay/deflay, rdtc/rdt, tnc). Note: on-disk `Sav` comes right after `Team` though it +lives at 0x284 in memory. + +### 2.1 `Game::PlayerColorID` (4 bytes) — Write `FUN_0053c080` +`+0 int8 index; +1,+2,+3 uint8 r,g,b`; writer emits index via `FUN_008b9cb0` (char→int on disk), and +**iff index == -1** the three r,g,b bytes (matches R1 §1.6). Same struct used by `IndependenceInfo.indcl`. + +### 2.2 `Game::DiplomacyStats` (0x24) — Write `FUN_00818cb0` +`+4 int other`; then **int16** fields at +8 `lastnap`, +0xa `lastnapbty`, +0xc `bknnap`, +0xe `btynap`, ++0x10 `lastally`, +0x12 `lastallybty`, +0x14 `bknally`, +0x16 `btyally`, +0x18 `lastcf`, +0x1a `lastcfbty`, ++0x1c `bkncf`, +0x1e `btycf`, +0x20 `deadhome` (all widened to int32 on disk; R1 `nap/ally/cf{last_,last_bty,bkn_,bty_}` ✓). + +### 2.3 `Game::PlayerReport` (preps, 0x30) — Write `FUN_00817480` +ints `+4 oid, +8 pid, +0xc flds, +0x10 sav, +0x14 home, +0x18 ncol, +0x1c mpwr, +0x20 mcls, +0x24 mmsl, +0x28 nshp, +0x2c nsat` (R1 ✓). + +### 2.4 `Game::SpyReport` — Write `FUN_00828ec0` +Four `std::list`s: `+4 list` (count `defc2`@+8, items `def`), `+0x10 list` +(`rtc`@+0x14, `strd`), `+0x1c list` (`evc`@+0x20, `evs`), `+0x28 list` +(`ttc`@+0x2c, `tt`). R1 only kept the four counts. + +### 2.5 `Game::TechTree` — Write `FUN_005890a0` (tags `NumTechs`, `TNm`, `NumBrs`; per-tech body in a +sub-writer not decompiled here; logs "TechTree: Tech %d not found saving tech tree"). + +--- + +## 3. `Game::StarFleet` (R1 `SimFleetDetails`) — Write `FUN_00701070`, Read `FUN_00702470`; `this` = obj+8 + +| abs | rel | type | save name | notes | +|---|---|---|---|---| +| 0x18 | 0x10 | Vector3 | `Pos` | via StarMapNode | +| 0x4c | 0x44 | Vector3 | `PrvPos` | | +| 0x58 | 0x50 | `ServerPlayer*` | `PID` | handle | +| 0x5c..0x77 | 0x54 | string | `FtName` | | +| 0x78 | 0x70 | bool | `Perm` | | +| 0x7c | 0x74 | `FleetLayout` (inline, ~0x24) | `HLay` + `Lay` | gate = either of its two vectors (+4/+8, +0x14/+0x18) non-empty | +| 0xa0 | 0x98 | `StarSystem*` | `LocID` | handle (legacy `SysID`,`TrdID` ints skipped) | +| 0xa4/0xa8 | 0x9c/0xa0 | `vector` | `NShips` + n×(`ShipID` handle, `Ship` obj) | | +| 0xc4..0xfb | 0xbc | `FlightPlan` (inline, 0x38) | `HFPlan` + `FPlan` | gate = wpts non-empty; §3.1 | +| 0xfc | 0xf4 | int | `FtTrans` | by value (legacy `Caps` int, `GtTrf` short, `FtSens` float, `FtInc` int skipped) | +| 0x100 | 0xf8 | Vector3 | `FtOrig` | (R1 "3 ints" → 3 floats) | +| 0x10c | 0x104 | int | `FtFlg` | | +| 0x110 | 0x108 | int | `Ftae` | | +| 0x114 | 0x10c | int | `Ftpae` | | +| 0x118 | 0x110 | int | `FtEnc` | | +| 0x11c | 0x114 | int | `FtMS` | | + +### 3.1 `Game::FlightPlan` (0x38) / `Waypoint` / `NodeRoute` +FlightPlan: `+0 vptr`, `+4 vector wpts`, `+0x14 float FPsp2`, `+0x18 int FPeta2`, `+0x1c Vector3 FPogn2`, +`+0x28 Vector3 FPdpos`, `+0x34 int pnd`. Waypoint: `+4 int Wpt`, `+8 int Tp`, `+0xc NodeRoute nrt`. +NodeRoute: `+4 nrp`, `+8 nrf`, `+0xc nrt` (ints). Reader also handles legacy `NumWpt`/`path`/`FPognid`. + +## 4. `Game::StarShip` (R1 `SimFleetShipDetails`) — Write `FUN_008291f0`, Read `FUN_00853fa0`; `this` = obj+8 + +| abs | rel | type | save name | notes | +|---|---|---|---|---| +| 0x10 | 0x8 | `ServerPlayer*` | `PlrID` | handle | +| 0x14 | 0xc | `ShipDesign*` | `DesID` | writes `design->+0xa4` (design id) | +| 0x20 | 0x18 | float | `Range` | | +| 0x24..0x33 | 0x1c | `ShipHealth` {vptr; float[3]} | `Health` | 3 unnamed floats (R1 guess command/mission/drive) | +| 0x34 | 0x2c | int | `MineCap` | | +| 0x38/0x3c | 0x30/0x34 | `vector<{float th, thm}>` | `NTH` + n×(`TH`,`THM`) | | +| 0x48 | 0x40 | int | `Plg` | | +| 0x4c | 0x44 | int | `Act` | | +| 0x50 | 0x48 | bool | `Dep` | | +| 0x51 | 0x49 | bool | `Atq` | | +| 0x5c | 0x54 | int | `LCT` | | +| 0x60 | 0x58 | int | `tsd` | | +| 0x64 | 0x5c | `StarFleet*` | `FltID` | handle | +| 0x68 | 0x60 | int | `ConCap` | | +| 0x6c | 0x64 | **float** | `RefCap` | R1 Int32 → float | +| 0x70 | 0x68 | **float** | `RepCap` | R1 Int32 → float | +| 0x7c | 0x74 | int | `EncID` | | +| 0x80 | 0x78 | `PrisonerHold` (inline) | `PrisH` | `+0x14 int*` → `[0]=PrMax`, `[2..8]` per-species counts; disk `PrMax`,`PrNSp`,(`PrSp`=idx,`PrNum`) | +| 0x98 | 0x90 | `BuildQueue*` | `hbq` + `BQ2` | gate = ptr!=NULL | +| 0x9c | 0x94 | `Population*` | `hsp` + `pop` | gate = ptr!=NULL | +| 0xa0 | 0x98 | `Population*` | `ppop` | | +| 0xa8 | 0xa0 | int | `atsp` | by value | +| 0xac | 0xa4 | int | `tblt` | | + +--- + +## 5. Top-level sim block — `Game::StrategyServer` Write `FUN_0079fa70` / Read `FUN_007d27a0` +Tag order confirms R1 §5: KeyPath, NMSz, NMLc, NMnx, ModCount, Frame, GameID, [AIDifficultyID legacy], +Attrib, RNG, GameName, Map, IncMod, ResMod, EnAl, EnTm, GOTurn, GOWinPly, NPCm/o/i/v/a, szadj, rsadj, suadj, +sprjs, RandEncAdj, cmbtid, turnstats, numcreps/crep, ninv/invs/inve/invt/invtb, AllExc×6, AllExcCF, +AllExcCFp×2, NumPlrs/PlayerID/Player, ISsp, ISsu, NumSys/SysID/Sys, NdGr2, trdmgr, spymgr, NumFlts/FltID/Flt, +NumActs/Act, SvSctOb, zdsc, zdsi, zdst. (Offsets not tabulated — out of scope this round.) + +--- + +## 6. Corrections to the community reference (R1/R2) +- Types: `TRM`, `CstR/E/T`, `shrm`, ship `RefCap/RepCap`, PlayerView `Infra` are **floats**; `pswd` is a + **string**; `PvMA`, `AIBn` are bools; `FtOrig` is a Vector3 (3 floats). +- Widths: `Bats2`/`rcex` are true int64 (R2 int is wrong); `Abdn`/`Dstyd` are bools, `ltis` an int + (R2 "short" is an artefact of reading value bytes); `TShn`, `ETS`, all DiplomacyStats counters are + int16 in memory but int32 on disk. +- Missing in R1: `Nexp` entries carry (`xid`,`xmin`,`xmax`,`xper`); SpyReport lists have bodies; Morale + is a sparse (msp,mv) table; `nadct` is followed by sparse (`ads`,`adt`) pairs over a 7-int table. +- `Plg`/`Act`/`EncID`/`OID`/flags are written by value → plain ints (not handles). + +## 7. Confidence & open questions +- **High**: all offsets/types in §1–§4 (direct from Write functions; Read functions agree on every + member address; COL offsets from RTTI; no adjustor thunks). +- **Medium**: nested struct sizes inferred from neighbouring offsets (Population 0x14, Morale 0x20, + FlightPlan 0x38, PlayerView ≈0x9c, IndependenceInfo 0x70); which members belong to `StarSystem` vs + `ServerSystem` (serializer is `ServerSystem`'s; `ClientSystem` shares the StarSystem vftables and only + streams `Pos`). +- **Open**: `FUN_008b9cb0` exact on-disk width for `PlayerColorID` (R1 says int32 — plausible); + `ServerSystem+0x10` owner type (StrategyServer? its `+0x50` is a player-object table); `OID` + allocation (R2's ×16); per-tech body of `TechTree::Write`; `CdPlayer` block not attempted (names are + R1's `unknownN`, nothing to xref). No types were written back into Ghidra (notes only). diff --git a/save-reader/save_reader.py b/save-reader/save_reader.py new file mode 100644 index 0000000..4d5ae74 --- /dev/null +++ b/save-reader/save_reader.py @@ -0,0 +1,1442 @@ +#!/usr/bin/env python3 +"""save_reader.py -- reader for Sword of the Stars (2006, v1.8) .sav files. + +The game serializes its world through a self-describing "Streamable" stream: + + container whole file is a gzip stream (an already-inflated blob is + accepted too, e.g. R1's *.sav.inflate.dat test artifacts) + named value [int32 len][name bytes] [value] [NUL pad to 4] + complex frame [int32 len][name bytes][NUL pad to 4] 0xBEEFBEEF + ... child items ... + 0x41104110 (= ~0xBEEFBEEF) + arrays a named int32 count followed by count x element; the + "ComplexArray" flavour wraps count+elements in a frame + +Everything is little-endian; text is windows-1252. Values carry NO type +byte -- only their name -- so a reader either knows the type from a schema or +has to guess it. This module does both: + + * Walker generic, resync-capable tokenizer. Walks the inflated bytes + as a tree of Node(name, kind, value, offset). Uses the + BEEFBEEF/41104110 frames as hard synchronisation points, a + catalog of known (context, name) -> type hints, and a + lookahead plausibility test for unknown names. Unknown or + malformed regions become `raw` nodes; parsing continues at the + next frame marker or plausible tag. Nothing in the schema is + required for the walk to succeed. + * Shapes the recovered struct field orders (Summary, CreateParameters, + Sim -> Player / Sys / Flt / Ship ...) applied on top of the + generic tree to produce typed dicts. Fields marked A() carry + binary-confirmed on-disk names (struct-recovery.md) and are + matched by name; fields marked R() carry only the community + editor's C# name and are matched by position. + +Padding convention: the community reference is ambiguous about whether the +NUL padding is computed over the whole item ("joint": [len][name][value][pad]) +or separately after the name and after the value ("split"). Both are +implemented; --padding auto (default) walks the file both ways and keeps the +cleaner parse. REAL-SAVE VALIDATION IS PENDING -- see SAVE_FORMAT.md. + +usage: save_reader.py SAVE [--dump] [--json] [--strict] [--padding MODE] + [--out FILE] [--inflate FILE] + +Stdlib only. +""" +from __future__ import annotations + +import argparse +import gzip +import json +import math +import struct +import sys +import zlib +from typing import Any, Callable, Optional + +__all__ = [ + "BEGIN_MARK", "END_MARK", "SaveFormatError", "Issue", "Node", "Walker", + "Shape", "Seq", "CArr", "NArr", "If", "Opt", "Until", "Rest", "A", "R", + "ROOT", "apply_schema", "read_save", "read_bytes", "inflate", "dump_tree", + "plain", "pad4", +] + +BEGIN_MARK = 0xBEEFBEEF +END_MARK = 0x41104110 +BEGIN_BYTES = struct.pack(" classified as int, not float + + +def pad4(n: int) -> int: + return (n + 3) & ~3 + + +class SaveFormatError(ValueError): + def __init__(self, msg: str, offset: Optional[int] = None): + self.offset = offset + if offset is not None: + msg = f"{msg} (at inflated offset 0x{offset:x})" + super().__init__(msg) + + +class Issue: + __slots__ = ("level", "path", "offset", "msg") + + def __init__(self, level: str, path: str, offset: int, msg: str): + self.level, self.path, self.offset, self.msg = level, path, offset, msg + + def __repr__(self) -> str: + return f"[{self.level}] @0x{self.offset:x} {self.path}: {self.msg}" + + def as_dict(self) -> dict: + return {"level": self.level, "path": self.path, "offset": self.offset, "msg": self.msg} + + +class Node: + """One item of the generic tree. kind is one of PRIMITIVES, 'complex' or 'raw'.""" + __slots__ = ("name", "kind", "value", "offset", "size", "children", "raw", "alt", "hinted") + + def __init__(self, name, kind, value=None, offset=0, size=0, children=None, + raw=b"", alt=None, hinted=False): + self.name = name # str, "" for an empty tag, None for a tagless frame + self.kind = kind + self.value = value + self.offset = offset # inflated offset of the item's tag (or frame marker) + self.size = size # bytes consumed incl. tag, padding and frame markers + self.children = children if children is not None else [] + self.raw = raw # value bytes (scalars / raw) for re-typing + self.alt = alt # alternative reading of a 4-byte word + self.hinted = hinted # type came from the catalog/schema, not a guess + + @property + def is_complex(self) -> bool: + return self.kind == "complex" + + def __repr__(self) -> str: + return f"Node({self.name!r}, {self.kind}, {self.value!r} @0x{self.offset:x})" + + +# --- container --------------------------------------------------------------- + +def inflate(data: bytes) -> bytes: + """gzip -> bytes. Data without the gzip magic is returned unchanged.""" + if data[:2] == b"\x1f\x8b": + try: + return gzip.decompress(data) + except (EOFError, OSError, zlib.error) as e: + raise SaveFormatError(f"gzip container is damaged: {e}") from None + return data + + +# --- schema description objects --------------------------------------------- +# Field types: a primitive name, "vec3" (frame of 3 floats, named or not), +# "any" (keep generic), "raw", a Shape (framed struct), Seq (unframed group), +# CArr (framed count+elements), NArr (unframed count+elements). + +class Field: + __slots__ = ("name", "type", "key", "auth", "flex") + + def __init__(self, name, type_, key=None, auth=False, flex=False): + self.name = name # expected on-disk tag (auth) or R1 name (not auth) + self.type = type_ + self.key = key or name # key in the typed dict + self.auth = auth + self.flex = flex # Shape/CArr may also appear inline (unframed) + + def __repr__(self): + return f"{'A' if self.auth else 'R'}({self.name!r})" + + +def A(name, type_, key=None, flex=False) -> Field: + """Field whose on-disk name is binary-confirmed (struct-recovery.md).""" + return Field(name, type_, key, auth=True, flex=flex) + + +def R(name, type_, key=None, flex=False) -> Field: + """Field known only by the community editor's C# name (matched by position).""" + return Field(name, type_, key, auth=False, flex=flex) + + +class Seq: + """Unframed ordered group of fields (R1 'leaf' structs, array element bodies).""" + __slots__ = ("fields",) + + def __init__(self, fields): + self.fields = list(fields) + + +class Shape: + """Framed struct: name tag + BEEFBEEF + fields + 41104110.""" + __slots__ = ("name", "fields", "_prefix", "_by_name") + + def __init__(self, name, fields): + self.name = name + self.fields = list(fields) + self._prefix = None + self._by_name = None + + def prefix(self) -> list: + """Positional child hints up to the first variable-length construct.""" + if self._prefix is None: + self._prefix = _prefix_of(self.fields) + return self._prefix + + def by_name(self) -> dict: + if self._by_name is None: + d = {} + _collect_names(self.fields, d) + self._by_name = d + return self._by_name + + +class CArr: + __slots__ = ("elem",) + + def __init__(self, elem): + self.elem = elem # primitive, Shape, Seq, or Field + + +class NArr: + __slots__ = ("elem",) + + def __init__(self, elem): + self.elem = elem + + +class If: + """Conditional group: parsed iff out[key] == equals (default True).""" + __slots__ = ("key", "inner", "equals") + + def __init__(self, key, inner, equals=True): + self.key, self.inner, self.equals = key, inner, equals + + +class Opt: + """Optional named item: consumed only when the next tag matches.""" + __slots__ = ("field",) + + def __init__(self, name, type_, key=None): + self.field = Field(name, type_, key, auth=True) + + +class Until: + """Collect generic items until a tag named stop appears (exclusive).""" + __slots__ = ("key", "stop") + + def __init__(self, key, stop): + self.key, self.stop = key, stop + + +class Rest: + """Collect all remaining items generically.""" + __slots__ = ("key",) + + def __init__(self, key="_rest"): + self.key = key + + +def _prefix_of(fields) -> list: + out = [] + for f in fields: + if isinstance(f, Field): + if f.flex: + break + t = f.type + if isinstance(t, Seq): + sub = _prefix_of(t.fields) + out.extend(sub) + if len(sub) != _count_items(t.fields): + break + elif isinstance(t, NArr): + out.append("int") + break + elif t in ("any", "raw"): + out.append(None) + else: + out.append(t) # primitive kind, "vec3", Shape, CArr + else: + break # If / Opt / Until / Rest end positional certainty + return out + + +def _count_items(fields) -> int: + n = 0 + for f in fields: + if isinstance(f, Field) and isinstance(f.type, Seq): + n += _count_items(f.type.fields) + else: + n += 1 + return n + + +def _collect_names(fields, d): + for f in fields: + if isinstance(f, Field): + d.setdefault(f.name, f.type) + d.setdefault(f.name.lower(), f.type) + t = f.type + if isinstance(t, Seq): + _collect_names(t.fields, d) + elif isinstance(t, NArr): + e = t.elem + if isinstance(e, Seq): + _collect_names(e.fields, d) + elif isinstance(e, Field): + _collect_names([e], d) + elif isinstance(f, Opt): + _collect_names([f.field], d) + elif isinstance(f, If): + inner = f.inner + _collect_names(inner.fields if isinstance(inner, Seq) else [inner], d) + + +# --- schema: recovered struct layouts ----------------------------------------- +# Sources: save-editor-structs.md (R1/R2 community editors) with the corrections +# from struct-recovery.md (binary). A() names are on-disk tags confirmed in the +# exe; R() names are R1's C# field names (disk spelling unknown -> positional). + +NestedInt = Shape(None, [R("value", "int"), Rest()]) + +PlayerColor = Shape("ClrID", [ + R("idx", "int"), + If("idx", Seq([R("r", "int"), R("g", "int"), R("b", "int")]), equals=-1), + Rest(), +]) + +# -- Summary (R1 §3) ----------------------------------------------------------- +PlayerSettings = Shape("Settings", [ + R("initialTreasury", "int"), R("initialColonies", "int"), + R("initialTechnologies", "int"), R("difficulty", "int"), Rest(), +]) +PlayerSlot = Shape("Slot", [ + R("isPlay", "bool"), R("isDead", "bool"), R("isReq", "bool"), R("isRec", "bool"), + R("isFxNm", "bool"), R("fxNm", "string"), + R("isFxSp", "bool"), R("fxSp", "int"), + R("isFxCr", "bool"), R("fxCrId", PlayerColor), + R("isFxBd", "bool"), R("fxBd", "string"), + R("isFxAv", "bool"), R("fxAv", "string"), + R("tag", "int"), R("pwd", "int"), R("team", "int"), + R("settings", PlayerSettings), Rest(), +]) +PlayerSlotWrapper = Shape("PlayerSlotWrapper", [R("slot", PlayerSlot), R("rank", "int"), Rest()]) +Tmrs = Shape("Tmrs", [R("tstl", "int"), R("tctl", "int"), R("tqtl", "int"), R("tqtle", "int"), Rest()]) +Session = Shape("Session", [R("tmrs", Tmrs), Rest()]) +Summary = Shape("Summary", [ + R("gameName", "string"), R("turn", "int"), R("numSys", "int"), R("checkSum", "int"), + R("players", CArr(PlayerSlotWrapper)), R("session", Session), + R("mapShape", "int"), R("incMod", "int"), R("resMod", "int"), + R("alliances", "bool"), R("teams", "bool"), R("encounters", "bool"), + R("scenario", "string"), Rest(), +]) + +# -- CreateParameters (R1 §4) --------------------------------------------------- +Planet = Shape("Planet", [ + R("coordinates", "vec3"), R("unknown1", "int"), R("unknown2", "int"), + R("unknown3", "int"), R("unknown4", "int"), Rest(), +]) +MapPNpc = Shape("Npc", [R("unknown1", "int"), R("unknown2", "int"), Rest()]) +MapP = Shape("MapP", [ + R("unknown1", "int"), R("planetArray", CArr(Planet)), + R("players", NArr(CArr("int"))), R("npcArray", CArr(MapPNpc)), Rest(), +]) +Scrp = Shape("Scrp", [R("spc", "int"), Rest()]) +CreateParams = Shape("CreateParameters", [ + R("name", "string"), R("id", "int"), R("rSeed", "int"), R("aid", "int"), R("key", "string"), + R("mapP", MapP), R("mapS", "int"), R("mapF", "int"), R("nSys", "int"), R("rEnc", "float"), + R("sDist", "int"), R("sSize", "float"), R("sRes", "float"), R("sSuit", "int"), R("maxP", "int"), + R("aSpec", "int"), R("bAlly", "bool"), R("nTeam", "int"), R("tmgrp", "bool"), + R("pSav", "int"), R("pCol", "int"), R("pTech", "int"), R("incM", "float"), R("resM", "float"), + R("scrp", Scrp), Rest(), +]) + +# -- shared sim types (struct-recovery §1.4, §1.5, §1.6, §1.7) ------------------ +PopG = Shape("PopG", [A("PopT", "int"), A("PopS", "int"), A("PopC", "int64"), Rest()]) +Population = Shape(None, [A("PopNG", NArr(A("PopG", PopG)), key="groups"), Rest()]) +Morale = Shape(None, [A("mnsp", NArr(Seq([A("msp", "int"), A("mv", "int")])), key="entries"), Rest()]) +MoraleEvent = Shape(None, [ + A("mid", "int"), A("mtr", "int"), A("mn", "int"), A("mtp", "int"), + A("mfx", Morale), A("mdsc", "string"), Rest(), +]) +BuildOrder = Shape(None, [ + A("desID", "int"), A("con", "int"), A("conleft", "int"), A("sav", "int"), A("ordID", "int"), Rest(), +]) +BuildQueue = Shape("BQ", [R("ords", CArr(BuildOrder), flex=True), Rest()]) +IndependenceInfo = Shape("indi", [ + A("indsp", "int"), A("indcl", PlayerColor), A("indnm", "string"), + A("indav", "string"), A("indba", "string"), Rest(), +]) +Rts = Shape("Rts", [ + A("SRs", "float"), A("SRt", "float"), A("SRsc", "float"), A("SRtf", "float"), + A("SRi", "float"), A("SRoh", "float"), A("SRnr", "int"), Rest(), +]) +PlayerView = Shape("pview", [ + A("VTrn", "int"), A("Pop", "int"), A("Pop2", Population), A("Infra", "float"), + A("Suit", "float"), A("Res", "int"), Opt("ARes", "int"), A("ARes2", "int"), A("MRes", "int"), + A("NoRebAI", "bool"), A("pbon", "int"), A("pbon2", Population), A("ibon", "float"), + A("TerrFl", "int"), A("footer", "bool"), Rest(), +]) + +# -- star system (struct-recovery §1, disk order) -------------------------------- +Sys = Shape("Sys", [ + A("Pos", "vec3"), + A("R", "float"), A("G", "float"), A("B", "float"), A("A", "float"), + A("Idx", "int"), A("Size", "int"), Opt("ISuit", "float"), A("Suit", "float"), + A("Res", "int"), Opt("ARes", "int"), A("ARes2", "int"), A("MRes", "int"), + A("NoRebAI", "bool"), A("TRes", "int"), A("Pop", "int"), A("Pop2", Population), + A("Infra", "float"), + A("PvPop", "int"), A("PvPop2", Population), A("PvInfra", "float"), A("PvSuit", "float"), + A("PvRes", "int"), A("PvARes2", "int"), A("PvMRes", "int"), A("PvNoRebAI", "bool"), + A("Rts", Rts), + A("Abdn", "bool"), A("Dstyd", "bool"), A("TnsOH", "int"), + A("OutMod", "float"), A("RepCur", "float"), A("RepMax", "float"), + A("ntdev", "int"), A("pbon", "int"), A("pbon2", Population), A("ibon", "float"), + A("ltis", "int"), A("rbfl", "int"), A("rbtn", "int"), A("rbfr", "int"), A("rbwn", "int"), + A("hsrg", "bool"), + A("haltc", NArr(Seq([A("haltt", "int"), A("haltv", "bool")])), key="halt"), + A("vnh", "bool"), + If("vnh", Seq([A("vnd", "bool"), A("vnex3", "bool"), A("vnpex3", "bool")])), + A("Name", "string"), + A("VFlags", "int"), A("EFlags", "int"), A("AFlags", "int"), A("FFlags", "int"), A("GFlags", "int"), + A("Bats2", "int64"), A("rcex", "int64"), + A("MnRFlags", "int"), A("RfRFlags", "int"), A("ClkFlags", "int"), + A("EggScio", "int"), A("TerrFl", "int"), A("TAcq", "int"), A("TFAcq", "int"), A("TDst", "int"), + A("dcs", Population), A("dsu", "float"), + A("cm", Morale), A("PvCM", Morale), A("cme2", CArr(MoraleEvent), flex=True), + A("spies2", "any"), + A("PID", "int"), A("DefF", "int"), A("DefSF", "int"), + Opt("BQ", BuildQueue), + A("nadct", NArr(Seq([A("ads", "int"), A("adt", "int")])), key="adct"), + A("NumPlgs2", NArr(Seq([A("PlgT", "int"), A("Plg", "any")])), key="plagues"), + A("NumFlts", NArr(A("Flt", "int")), key="fleets"), + A("NumGFs", NArr(A("GF", "int")), key="gates"), + A("NumSnF", NArr(A("SnF", "int")), key="stations"), + A("NumMnF", NArr(A("MnF", "int")), key="monitors"), + A("NVO", NArr(Seq([ + A("PID", "int"), A("TShn", "int"), A("OID", "int"), A("isind", "bool"), + If("isind", A("indi", IndependenceInfo)), + ])), key="colonies"), + A("NVE", NArr(Seq([A("EPid", "int"), A("ETS", "int"), A("Eid", "int")])), key="nve"), + A("NVs", NArr(Seq([R("pid", "int"), R("pview", PlayerView)])), key="views"), + A("hindi", "bool"), + If("hindi", A("indi", IndependenceInfo)), + Rest(), +]) + +# -- player / empire (struct-recovery §2, disk order) ---------------------------- +Alliances = Shape("Team", [A("ALid", "int"), A("AL", "int"), A("NA", "int"), A("CF", "int"), Rest()]) +DipDetail = Seq # helper alias for readability +DipStat = Shape(None, [ + A("other", "int"), + A("lastnap", "int"), A("lastnapbty", "int"), A("bknnap", "int"), A("btynap", "int"), + A("lastally", "int"), A("lastallybty", "int"), A("bknally", "int"), A("btyally", "int"), + A("lastcf", "int"), A("lastcfbty", "int"), A("bkncf", "int"), A("btycf", "int"), + A("deadhome", "int"), Rest(), +]) +Prep = Shape(None, [ + A("oid", "int"), A("pid", "int"), A("flds", "int"), A("sav", "int"), A("home", "int"), + A("ncol", "int"), A("mpwr", "int"), A("mcls", "int"), A("mmsl", "int"), + A("nshp", "int"), A("nsat", "int"), Rest(), +]) +Odes = Shape(None, [R("ontF", "int"), R("otnL", "int"), R("odid", "int"), R("opid", "int"), Rest()]) +Owep = Shape(None, [R("ontF", "int"), R("otnL", "int"), R("odet", "int"), R("owep", "string"), R("owith", "int"), Rest()]) +Otch = Shape(None, [R("ontF", "int"), R("otnL", "int"), R("odet", "int"), R("otch", "string"), R("owith", "int"), Rest()]) +Note = Shape("Nts", [A("NtSys", "int"), A("NtTxt", "string"), A("NtTrn", "int"), Rest()]) +Design = Shape("Des", [ + R("faiDes", "bool"), R("dHide", "bool"), R("dWep", "int"), R("dName", "string"), Rest("sections"), +]) +ConMods = Shape("ConMods", [ + A("ConMod", "float", key="ConMod0"), A("SavMod", "float", key="SavMod0"), + A("ConMod", "float", key="ConMod1"), A("SavMod", "float", key="SavMod1"), + A("ConMod", "float", key="ConMod2"), A("SavMod", "float", key="SavMod2"), +]) +Player = Shape("Player", [ + A("TechTree", "any"), + A("HomeSys", "int"), A("PlyrIdx", "int"), A("PlryName", "string"), A("Species", "int"), + A("ClrID", PlayerColor), A("Bdg", "string"), A("Avt", "string"), A("Team", "int"), A("Sav", "int"), + A("IdealSuit", "float"), A("SuitTol", "float"), A("MaxOH", "float"), + A("ResRate", "float"), A("ResMod", "float"), A("ResScl", "float"), + A("TRM", "float"), A("TRP", "int"), A("TRA", "int"), + A("OutMod", "float"), A("RebOutMod", "float"), A("ScOutMod", "float"), A("IncMod", "float"), + Opt("SensMod", "float"), Opt("ExPopSys", "int"), + A("PopMod", "float"), A("TerraMod", "float"), + A("AMine", "bool"), A("MinPure", "float"), A("MinRate", "float"), + A("NGts", "int"), A("PrGtTrf", "int"), A("GTraf", "int"), + A("CstR", "float"), A("CstE", "float"), A("CstT", "float"), + A("Maint", "int"), A("shrm", "float"), A("Status", "int"), + A("Elim", "bool"), A("NPC", "bool"), A("RebAI", "bool"), A("ReqCL", "bool"), + A("Team", Alliances, key="Alliances"), + A("HasVac", "int"), A("HasImm", "int"), A("NPTrk", "int"), A("HasDisc", "int"), + A("HasDiscSp", "int"), A("HasDiscCl", "int"), A("HasEnc", "int"), A("HasEng", "int"), + A("Events", "any"), A("FNG", "any"), + A("PvSav", "int"), A("PvMA", "bool"), A("AIBn", "bool"), + A("CnTrd", "bool"), A("CnRad", "bool"), A("hgs", "bool"), A("hadvs", "bool"), A("harcc", "bool"), + A("CnVItl", "bool"), A("pddm", "float"), + A("BnkWrn", "int"), A("BnkTrn", "int"), A("BnkPr", "int"), A("BnkEl", "int"), + A("ShipRecs", "any"), + A("NextPrjID", "int"), A("plcy", "int"), A("pswd", "string"), A("lret", "int"), A("nmeid", "int"), + A("cdp", "bool"), A("spy2", "any"), A("civr", "any"), A("aidf", "int"), + A("Srn", "bool"), A("SrnTo", "int"), A("lboid", "int"), Opt("lcid", "int"), A("lcid2", "int"), + A("ResTNm", "string"), A("ResErrRoll", "bool"), + R("conMods", ConMods, flex=True), + A("NumOwn", NArr(A("OwnId", "int")), key="owners"), + A("NumDes", NArr(Seq([A("DesID", "int"), A("Des", Design)])), key="designs"), + A("NumLeg", NArr(Seq([A("DesID", "int"), A("Des", Design)])), key="legacyDesigns"), + A("NumNotes", NArr(A("Nts", Note)), key="notes"), + A("NumPR", NArr(Seq([A("PRm", "float"), A("PRBt", "int")])), key="pr"), + A("HasAIR", "bool"), If("HasAIR", A("AIR", "any")), + A("cta", "bool"), A("AIEnf", "any"), + A("NSprj", NArr(Seq([A("SprjT", "int"), A("Sprj", "any")])), key="specialProjects"), + A("Nexp", NArr(Seq([A("xid", "int"), A("xmin", "int"), A("xmax", "int"), A("xper", "float")])), key="nexp"), + A("NWeapXcl", NArr(A("WeapXcl", "int")), key="weapXcl"), + A("Ojvs", "any"), + A("dipstats", CArr(DipStat)), + A("comms", "any"), + A("preps", CArr(Prep)), + A("odes", CArr(Odes)), A("owep", CArr(Owep)), A("otch", CArr(Otch)), + A("aid", "any"), + A("ndeflay", NArr(A("deflay", "any")), key="defLayouts"), + A("rdtc", NArr(A("rdt", "any")), key="raidTargets"), + A("tnc", "int"), + Rest(), +]) + +# -- fleets & ships (struct-recovery §3, §4) -------------------------------------- +NodeRoute = Shape("nrt", [A("nrp", "int"), A("nrf", "int"), A("nrt", "int"), Rest()]) +Waypoint = Shape(None, [A("Wpt", "int"), A("Tp", "int"), R("nrt", NodeRoute, flex=True), Rest()]) +FlightPlan = Shape("FPlan", [ + R("wpts", CArr(Waypoint), flex=True), A("FPsp2", "float"), A("FPeta2", "int"), + A("FPogn2", "vec3"), A("FPdpos", "vec3"), A("pnd", "int"), Rest(), +]) +PrisonerHold = Shape("PrisH", [ + A("PrMax", "int"), A("PrNSp", NArr(Seq([A("PrSp", "int"), A("PrNum", "int")])), key="prisoners"), Rest(), +]) +Ship = Shape("Ship", [ + A("DesID", "int"), A("FltID", "int"), A("PlrID", "int"), A("Range", "float"), + A("Health", "vec3"), + A("ConCap", "int"), A("RefCap", "float"), A("RepCap", "float"), A("MineCap", "int"), + A("Plg", "int"), A("Act", "int"), A("Dep", "bool"), A("Atq", "bool"), A("EncID", "int"), + A("PrisH", PrisonerHold), + A("LCT", "int"), A("tsd", "int"), A("atsp", "int"), A("tblt", "int"), + A("hbq", "bool"), If("hbq", A("BQ2", BuildQueue)), + A("hsp", "bool"), If("hsp", Seq([A("pop", Population), A("ppop", Population)])), + A("NTH", NArr(Seq([A("TH", "float"), A("THM", "float")])), key="thrusters"), + Rest(), +]) +Fleet = Shape("Flt", [ + A("Pos", "vec3"), A("PID", "int"), A("LocID", "int"), Opt("SysID", "int"), Opt("TrdID", "int"), + A("HFPlan", "bool"), If("HFPlan", A("FPlan", FlightPlan)), + A("FtName", "string"), + Opt("Caps", "int"), Opt("GtTrf", "int"), Opt("FtSens", "float"), Opt("FtInc", "int"), + A("FtTrans", "int"), A("FtOrig", "vec3"), + A("FtFlg", "int"), A("Ftae", "int"), A("Ftpae", "int"), A("FtEnc", "int"), A("FtMS", "int"), + A("Perm", "bool"), A("PrvPos", "vec3"), + A("HLay", "bool"), If("HLay", A("Lay", "any")), + A("NShips", NArr(Seq([A("ShipID", "int"), A("Ship", Ship)])), key="ships"), + Rest(), +]) + +# -- combat reports, node grid (R1 §9, §11) ------------------------------------ +Dams = Shape("dams", [R("dams", "int"), R("damp", "int"), R("dami", "int"), R("damt", "int"), Rest()]) +Wrep = Shape(None, [R("wep", "string"), R("dams", Dams, flex=True), Rest()]) +CrepPrep = Shape(None, [ + R("plr", "int"), R("ai", "bool"), R("ally", "int"), R("status", "int"), + R("mxeng", "int"), R("mxcls", "int"), R("mxmsl", "int"), Rest(), +]) +Crep = Shape("crep", [ + R("cid", "int"), R("trn", "int"), R("pos", "vec3"), R("sid", "int"), R("auto", "int"), + R("dur", "int"), R("cow", "int"), R("cdst", "int"), R("cpk", "int"), R("cpt", "int"), + R("cdt", "int"), R("cdi", "int"), R("prep", CArr(CrepPrep), flex=True), + R("wrep", CArr(Wrep), flex=True), Rest(), +]) +NodePath = Shape(None, [ + R("npt", "int"), R("npid", "int"), R("npfr", "int"), R("npto", "int"), R("npctm", "int"), + R("npcby", "int"), R("npdtn", "int"), R("npdtf", "int"), R("npenp", "int"), + R("npuse", "int"), R("nptf", "int"), Rest(), +]) +NodeGrid = Shape("NdGr2", [R("paths", CArr(NodePath), flex=True), R("nextId", "int"), Rest()]) + +# -- Sim block (struct-recovery §5 tag order + R1 §5) ---------------------------- +Sim = Shape("Sim", [ + A("KeyPath", "string"), A("NMSz", "int"), A("NMLc", "int"), A("NMnx", "int"), + Until("idLists", "ModCount"), # PlayerIDs/DesignIDs/SystemIDs/FleetIDs/ShipIDs/TradeIDs + A("ModCount", "int"), A("Frame", "int"), A("GameID", "int"), Opt("AIDifficultyID", "int"), + A("Attrib", "any"), A("RNG", "any"), A("GameName", "string"), + A("Map", "int"), A("IncMod", "int"), A("ResMod", "int"), A("EnAl", "int"), A("EnTm", "int"), + A("GOTurn", "int"), A("GOWinPly", NestedInt), + A("NPCm", "int"), A("NPCo", "int"), A("NPCi", "int"), A("NPCv", "int"), A("NPCa", "int"), + A("szadj", "int"), A("rsadj", "int"), A("suadj", "int"), + A("sprjs", "any"), A("RandEncAdj", "float"), A("cmbtid", "int"), A("turnstats", "any"), + A("numcreps", NArr(A("crep", Crep)), key="combatReports"), + A("ninv", "int"), + Until("invasionsAndExclusions", "NumPlrs"), # invs/inve/invt/invtb, AllExc x6, AllExcCF, AllExcCFp x2 + A("NumPlrs", NArr(Seq([A("PlayerID", "int"), A("Player", Player)])), key="players"), + Until("species", "NumSys"), # ISsp / ISsu + A("NumSys", NArr(Seq([A("SysID", "int"), A("Sys", Sys)])), key="systems"), + A("NdGr2", NodeGrid), A("trdmgr", "any"), A("spymgr", "any"), + A("NumFlts", NArr(Seq([A("FltID", "int"), A("Flt", Fleet)])), key="fleets"), + A("NumActs", NArr(A("Act", "int")), key="acts"), + A("SvSctOb", "any"), + A("zdsc", "int"), A("zdsi", "int"), A("zdst", "int"), + Rest(), +]) + +# -- file root: Summary, CreateParameters, Sim, then the unframed CdTable --------- +ROOT = Seq([ + R("summary", Summary), R("createParams", CreateParams), R("sim", Sim), Rest("cdTable"), +]) + +# Hand-maintained kinds for names that only occur inside generic ("any") regions. +MANUAL_KINDS = { + # tech tree / designs (R1 §7) + "TNm": "string", "tfc": "bool", "tResCost": "int", "tResDone": "int", "tAcq": "int", + "tiAcq": "int", "tUnlck": "int", "dName": "string", "wfn": "string", "bId": "bool", + "faiDes": "bool", "dHide": "bool", + # objectives / comms / research / encounters (R1 §5, §6, §11) + "cmp": "bool", "dsc": "string", "xcsn": "string", "nm": "string", "ntg": "string", + "tch": "string", "wep": "string", "gmch": "int", "drad": "float", "cst": "float", + "aOdd": "float", "aInc": "float", "rMd": "float", "sctSize": "float", "smx": "float", + "nPrvVa": "float", "crPce": "bool", "maintHf": "bool", "caps2": "int64", + "tRsld": "bool", "tRsldd": "bool", "tRsldc": "bool", "tRsldi": "bool", "tRsldr": "bool", + "prm": "float", +} + + +def _register(shape, kinds: dict, conflicts: set, shapes: dict, seen: set): + if id(shape) in seen: + return + seen.add(id(shape)) + if isinstance(shape, Shape) and shape.name: + shapes.setdefault(shape.name, shape) + fields = shape.fields if isinstance(shape, (Shape, Seq)) else [] + for f in fields: + items = [f] + if isinstance(f, Opt): + items = [f.field] + elif isinstance(f, If): + items = f.inner.fields if isinstance(f.inner, Seq) else [f.inner] + elif not isinstance(f, Field): + continue + for fld in items: + if not isinstance(fld, Field): + continue + _register_type(fld.name, fld.type, kinds, conflicts, shapes, seen) + + +def _register_type(name, t, kinds, conflicts, shapes, seen): + if isinstance(t, str): + if t in PRIMITIVES: + if name in kinds and kinds[name] != t: + conflicts.add(name) + kinds.setdefault(name, t) + elif isinstance(t, Shape): + if name and not t.name: + shapes.setdefault(name, t) + _register(t, kinds, conflicts, shapes, seen) + elif isinstance(t, Seq): + _register(t, kinds, conflicts, shapes, seen) + elif isinstance(t, (CArr, NArr)): + e = t.elem + if isinstance(e, Field): + _register_type(e.name, e.type, kinds, conflicts, shapes, seen) + elif isinstance(e, (Shape, Seq)): + _register(e, kinds, conflicts, shapes, seen) + if isinstance(t, CArr) and name: + shapes.setdefault(name, t) + + +def _build_catalog(): + kinds, conflicts, shapes = {}, set(), {} + _register(ROOT, kinds, conflicts, shapes, set()) + for k in conflicts: # ambiguous across contexts -> no global hint + kinds.pop(k, None) + for k, v in MANUAL_KINDS.items(): + kinds[k] = v + return kinds, shapes + + +GLOBAL_KINDS, GLOBAL_SHAPES = _build_catalog() +# Frames whose body is an opaque byte blob (read straight to the END marker). +RAW_FRAMES = {"RNG"} + + +# --- generic walker ----------------------------------------------------------- + +def _text_plausible(b: bytes) -> bool: + if not b: + return True + ok = sum(1 for c in b if 0x20 <= c < 0x7f or c >= 0xa0 or c in (9, 10, 13)) + return ok * 10 >= len(b) * 9 + + +def _classify_word(raw: bytes): + """Unknown 4-byte word -> (kind, value, alt).""" + i = struct.unpack(" root Node.""" + + GUESS_ORDER = ("word", "bool", "string", "int64") + + def __init__(self, data: bytes, padding: str = "joint", schema=ROOT, + kinds: Optional[dict] = None, shapes: Optional[dict] = None): + if padding not in ("joint", "split"): + raise ValueError("padding must be 'joint' or 'split'") + self.d = data + self.n = len(data) + self.padding = padding + self.schema = schema + self.kinds = GLOBAL_KINDS if kinds is None else kinds + self.shapes = GLOBAL_SHAPES if shapes is None else shapes + self.issues: list[Issue] = [] + self._eof_limited = False + self.stats = {"items": 0, "frames": 0, "resyncs": 0, "hint_failures": 0, + "guessed": 0, "raw_bytes": 0, "best_effort": 0} + + # -- low level --------------------------------------------------------------- + def issue(self, level, off, msg, path=""): + self.issues.append(Issue(level, path, off, msg)) + + def u32(self, p: int) -> int: + return struct.unpack_from(" bool: + return not any(self.d[a:b]) + + def tag_at(self, p: int, allow_empty: bool = False): + """(name, end) if a plausible name tag starts at p, else None.""" + if p + 4 > self.n: + return None + ln = struct.unpack_from(" MAX_TAG_LEN or p + 4 + ln > self.n: + return None + b = self.d[p + 4:p + 4 + ln] + if not all(0x20 <= c < 0x7f for c in b): + return None + return b.decode("ascii"), p + 4 + ln + + def value_pos(self, tag_start: int, name_end: int) -> int: + if self.padding == "joint": + return name_end + return tag_start + pad4(name_end - tag_start) + + def item_end(self, tag_start: int, name_end: int, size: int) -> int: + if self.padding == "joint": + return tag_start + pad4(name_end - tag_start + size) + vp = self.value_pos(tag_start, name_end) + return vp + pad4(size) + + def pads_zero(self, tag_start, name_end, vp, size, end) -> bool: + if self.padding == "split" and not self.zero(name_end, vp): + return False + return self.zero(vp + size, end) + + def try_scalar(self, tag_start: int, name_end: int, kind: str): + """Read a candidate scalar; (kind, value, raw, end) or None if impossible.""" + d, n = self.d, self.n + vp = self.value_pos(tag_start, name_end) + if kind == "string": + if vp + 4 > n: + self._eof_limited = True + return None + ln = struct.unpack_from(" MAX_STRING_LEN: + return None + if vp + 4 + ln > n: + self._eof_limited = True + return None + raw = d[vp + 4:vp + 4 + ln] + if not _text_plausible(raw): + return None + size, value = 4 + ln, raw.decode(ENCODING) + else: + size = 4 if kind == "word" else SIZES[kind] + if vp + size > n: + self._eof_limited = True + return None + raw = d[vp:vp + size] + if kind == "bool": + if raw[0] > 1: + return None + value = raw[0] == 1 + elif kind == "int": + value = struct.unpack(" n: + self._eof_limited = True + return None + if not self.pads_zero(tag_start, name_end, vp, size, end): + return None + return kind, value, raw, end + + def plausible_item(self, p: int, depth: int, allow_empty: bool = False) -> bool: + """Does an item plausibly start at p? Looks `depth` items ahead. + Sets self._eof_limited when a verdict was forced by running out of data.""" + n = self.n + if p == n: + return True + if p + 4 > n: + self._eof_limited = True + return False + w = self.u32(p) + if w == END_MARK or w == BEGIN_MARK: + return True + t = self.tag_at(p, allow_empty) + if t is None: + return False + name, ne = t + a = pad4(ne) + if a + 4 <= n and self.u32(a) == BEGIN_MARK and self.zero(ne, a): + return True + if depth <= 0: + return True + for kind in self.GUESS_ORDER: + r = self.try_scalar(p, ne, kind) + if r is None: + continue + if self.plausible_item(r[3], depth - 1, allow_empty): + return True + return False + + def resync(self, p: int, depth: int): + """Scan forward for the next END marker or plausible tag. -> (what, q).""" + d, n = self.d, self.n + q = p + 1 + while q + 4 <= n: + w = self.u32(q) + if w == END_MARK and depth > 0: + return "end", q + if w == BEGIN_MARK: + return "begin", q + if self.tag_at(q) is not None and self.plausible_item(q, 2, allow_empty=False): + return "tag", q + q += 1 + return "eof", n + + # -- hints ------------------------------------------------------------------- + def child_hint(self, frame_shape, index: int, name: Optional[str]): + """-> (scalar kind or None, sub-shape or None) for child #index named name.""" + kind = sub = None + if isinstance(frame_shape, Shape): + pre = frame_shape.prefix() + if index < len(pre): + kind, sub = _split_hint(pre[index]) + if kind is None and sub is None and name: + t = frame_shape.by_name().get(name) or frame_shape.by_name().get(name.lower()) + if t is not None: + kind, sub = _split_hint(t) + elif isinstance(frame_shape, CArr): + if index == 0: + kind = "int" + else: + kind, sub = _split_hint(frame_shape.elem) + elif frame_shape == "raw": + kind = "raw" + if kind is None and sub is None and name: + # global catalog: exact case only (case-folding was found to + # mislabel unknown names, e.g. 'a' vs star-colour 'A') + k = self.kinds.get(name) + if k: + kind = k + s = self.shapes.get(name) + if s is not None: + sub = s + if name in RAW_FRAMES: + sub = "raw" + return kind, sub + + # -- walking ----------------------------------------------------------------- + def walk(self) -> Node: + children, end = self.walk_frame(0, None, 0, self.schema, "") + root = Node(None, "complex", None, 0, end, children) + return root + + def walk_frame(self, p: int, ctx, depth: int, shape, path: str): + """Read items until the frame's END marker (or EOF). -> (children, pos).""" + d, n = self.d, self.n + children: list[Node] = [] + index = 0 + self.stats["frames"] += depth > 0 + while True: + if p >= n: + if depth > 0: + self.issue("error", p, f"frame {ctx!r} not terminated before EOF", path) + return children, p + if p + 4 > n: + children.append(self._raw_node(None, p, n, "trailing bytes", "warn", path)) + if depth > 0: + self.issue("error", n, f"frame {ctx!r} not terminated before EOF", path) + return children, n + w = self.u32(p) + if w == END_MARK: + if depth > 0: + return children, p + 4 + self.issue("warn", p, "stray END marker at top level", path) + children.append(self._raw_node(None, p, p + 4, "stray END marker", "warn", path)) + p += 4 + continue + node, p = self.read_item(p, ctx, index, depth, shape, path) + children.append(node) + index += 1 + + def read_item(self, p: int, ctx, index: int, depth: int, frame_shape, path: str): + d, n = self.d, self.n + self.stats["items"] += 1 + w = self.u32(p) + if w == BEGIN_MARK: # tagless frame + _, sub = self.child_hint(frame_shape, index, None) + children, end = self.walk_frame(p + 4, None, depth + 1, sub, f"{path}/<{index}>") + return Node(None, "complex", None, p, end - p, children), end + + t = self.tag_at(p, allow_empty=True) + if t is None: + return self._unreadable(p, depth, path) + name, ne = t + kind, sub = self.child_hint(frame_shape, index, name) + a = pad4(ne) + if a + 4 <= n and self.u32(a) == BEGIN_MARK and self.zero(ne, a): + children, end = self.walk_frame(a + 4, name, depth + 1, sub, f"{path}/{name}") + return Node(name, "complex", None, p, end - p, children, hinted=sub is not None), end + + # scalar: hinted kind first, then the guess order + if kind == "raw": + vp = self.value_pos(p, ne) + q = d.find(END_BYTES, vp) if depth > 0 else -1 + if q < 0: + q = n + node = Node(name, "raw", {"len": q - vp, "hex": d[vp:min(q, vp + 32)].hex()}, p, q - p, + raw=d[vp:q], hinted=True) + self.stats["raw_bytes"] += q - vp + return node, q + order = [] + if kind in ("int", "float"): + order.append("word") + elif kind in ("bool", "string", "int64"): + order.append(kind) + order += [k for k in self.GUESS_ORDER if k not in order] + # pass 1: continuation must be a non-empty tag / marker; pass 2 tolerates + # empty ("") tags; pass 3 (only when lookahead was defeated by EOF) takes + # whatever fits the remaining bytes. + self._eof_limited = False + chosen = None + for mode in ("strict", "empty-ok", "eof"): + if mode == "eof" and not self._eof_limited: + break + for i, cand in enumerate(order): + r = self.try_scalar(p, ne, cand) + if r is None: + continue + if mode == "eof" or self.plausible_item(r[3], 2, allow_empty=(mode == "empty-ok")): + chosen = (i, r, mode) + break + if chosen: + break + if chosen: + i, (ckind, value, raw, end), mode = chosen + alt = None + if ckind == "word": + if kind in ("int", "float"): + ckind = kind + value = struct.unpack(" 0: + q = p + while q + 4 <= n and q - p <= SMALL_RAW: + if self.u32(q) == END_MARK: + return self._raw_node(None, p, q, "unnamed payload", "info", path), q + q += 4 + what, q = self.resync(p, depth) + self.stats["resyncs"] += 1 + return self._raw_node(None, p, q, f"unreadable; resynced to {what}", "warn", path), q + + def _raw_node(self, name, p, q, why, level, path) -> Node: + d = self.d + self.issue(level, p, f"{why}: {q - p} raw bytes", path) + self.stats["raw_bytes"] += q - p + return Node(name, "raw", {"len": q - p, "hex": d[p:min(q, p + 32)].hex()}, p, q - p, raw=d[p:q]) + + +def _split_hint(t): + if isinstance(t, str): + if t in PRIMITIVES or t == "raw": + return t, None + return None, None # "vec3", "any" + if isinstance(t, Field): + return _split_hint(t.type) + if isinstance(t, (Shape, CArr)): + return None, t + return None, None + + +# --- generic tree helpers ----------------------------------------------------- + +def plain(node: Node): + """Node -> JSON-able generic representation.""" + if node.is_complex: + return {"_name": node.name, "_off": node.offset, "_items": [plain(c) for c in node.children]} + d = {"name": node.name, "kind": node.kind, "value": node.value, "off": node.offset} + if node.alt is not None: + d["alt"] = node.alt + if not node.hinted and node.kind != "raw": + d["guessed"] = True + return d + + +def dump_tree(node: Node, indent: int = 0, out=None, max_depth: int = 999) -> list[str]: + lines = [] if out is None else out + pad = " " * indent + for c in node.children: + nm = c.name if c.name is not None else "" + if c.is_complex: + lines.append(f"@{c.offset:08x} {pad}{nm} {{ # {len(c.children)} items, {c.size} bytes") + if indent < max_depth: + dump_tree(c, indent + 1, lines, max_depth) + lines.append(f"@{c.offset + c.size - 4:08x} {pad}}}") + elif c.kind == "raw": + v = c.value + lines.append(f"@{c.offset:08x} {pad}{nm} raw[{v['len']}] {v['hex']}{'...' if v['len'] > 32 else ''}") + else: + q = "" if c.hinted else "?" + val = json.dumps(c.value) if isinstance(c.value, str) else c.value + alt = "" + if c.alt is not None and not c.hinted: + alt = f" (alt {c.alt!r})" + lines.append(f"@{c.offset:08x} {pad}{nm} {c.kind}{q} {val}{alt}") + return lines + + +# --- schema application ------------------------------------------------------- + +class _Cursor: + __slots__ = ("nodes", "i") + + def __init__(self, nodes): + self.nodes, self.i = nodes, 0 + + def peek(self): + return self.nodes[self.i] if self.i < len(self.nodes) else None + + def take(self): + n = self.nodes[self.i] + self.i += 1 + return n + + def find(self, name, ci=False): + for j in range(self.i, len(self.nodes)): + nm = self.nodes[j].name + if nm == name or (ci and nm is not None and nm.lower() == name.lower()): + return j + return None + + @property + def done(self): + return self.i >= len(self.nodes) + + +class Applier: + def __init__(self): + self.issues: list[Issue] = [] + + def issue(self, level, path, node, msg): + off = node.offset if isinstance(node, Node) else (node or 0) + self.issues.append(Issue(level, path, off, msg)) + + # -- entry ------------------------------------------------------------------- + def apply(self, schema, root: Node) -> dict: + out = {} + cur = _Cursor(root.children) + self.apply_fields(schema.fields if isinstance(schema, (Shape, Seq)) else [], cur, out, "") + if not cur.done: + out["_extra"] = [plain(n) for n in cur.nodes[cur.i:]] + self.issue("warn", "", cur.peek(), f"{len(cur.nodes) - cur.i} unexpected trailing items") + return out + + # -- fields ------------------------------------------------------------------ + def apply_fields(self, fields, cur: _Cursor, out: dict, path: str): + for f in fields: + if isinstance(f, Field): + self.apply_field(f, cur, out, path) + elif isinstance(f, Opt): + nxt = cur.peek() + if nxt is not None and nxt.name is not None and nxt.name.lower() == f.field.name.lower(): + self.apply_field(f.field, cur, out, path) + elif isinstance(f, If): + if out.get(f.key) == f.equals: + inner = f.inner + if isinstance(inner, Seq): + self.apply_fields(inner.fields, cur, out, path) + else: + self.apply_field(inner, cur, out, path) + elif isinstance(f, Until): + items = [] + while not cur.done: + nxt = cur.peek() + if nxt.name is not None and nxt.name.lower() == f.stop.lower(): + break + items.append(plain(cur.take())) + out[f.key] = items + elif isinstance(f, Rest): + if not cur.done: + out[f.key] = [plain(n) for n in cur.nodes[cur.i:]] + cur.i = len(cur.nodes) + else: + raise TypeError(f"bad schema element {f!r}") + + def apply_field(self, f: Field, cur: _Cursor, out: dict, path: str): + fpath = f"{path}/{f.key}" + t = f.type + if isinstance(t, NArr): + cnt_node = self.take_named(f, cur, out, fpath) + if cnt_node is None: + return + count = self.coerce(cnt_node, "int", fpath) + items = [] + remaining = len(cur.nodes) - cur.i + if not isinstance(count, int) or count < 0 or count > remaining: + self.issue("error", fpath, cnt_node, + f"array count {count!r} exceeds the {remaining} item(s) left in the frame") + count = max(0, min(count if isinstance(count, int) else 0, remaining)) + for i in range(count): + if cur.done: + self.issue("error", fpath, cur.nodes[-1] if cur.nodes else None, + f"array truncated: {i} of {count} elements present") + break + before = cur.i + items.append(self.convert_elem(t.elem, cur, f"{fpath}[{i}]")) + if cur.i == before: + self.issue("error", fpath, cur.peek(), f"array element {i} consumed nothing; stopping") + break + out[f.key] = items + return + if isinstance(t, Seq): + self.apply_fields(t.fields, cur, out, path) + return + if isinstance(t, (Shape, CArr)) and f.flex: + nxt = cur.peek() + if nxt is not None and not nxt.is_complex: + # inline (unframed) variant + if isinstance(t, Shape): + sub = {} + self.apply_fields(t.fields, cur, sub, fpath) + out[f.key] = sub + else: + self.apply_field(Field(f.name, NArr(t.elem), f.key, f.auth), cur, out, path) + return + node = self.take_named(f, cur, out, fpath) + if node is None: + return + out[f.key] = self.convert(node, t, fpath) + + def take_named(self, f: Field, cur: _Cursor, out: dict, fpath: str): + node = cur.peek() + if node is None: + self.issue("error" if f.auth else "warn", fpath, None, f"missing field {f.name!r} (frame ended)") + out[f.key] = None + return None + if f.auth: + if node.name != f.name: + j = cur.find(f.name) + if j is None: + self.issue("error", fpath, node, f"expected {f.name!r}, found {node.name!r}; field missing") + out[f.key] = None + return None + skipped = cur.nodes[cur.i:j] + out.setdefault("_unexpected", []).extend(plain(n) for n in skipped) + self.issue("warn", fpath, node, + f"{len(skipped)} unexpected item(s) before {f.name!r}: " + + ", ".join(repr(n.name) for n in skipped[:6])) + cur.i = j + node = cur.peek() + elif node.name is not None and node.name.lower() != f.name.lower(): + self.issue("info", fpath, node, f"tag {node.name!r} read positionally as {f.name!r}") + return cur.take() + + # -- conversion -------------------------------------------------------------- + def convert_elem(self, elem, cur: _Cursor, path: str): + if isinstance(elem, Seq): + sub = {} + self.apply_fields(elem.fields, cur, sub, path) + return sub + if isinstance(elem, Field): + sub = {} + self.apply_field(elem, cur, sub, path) + return sub.get(elem.key) + node = cur.take() + return self.convert(node, elem, path) + + def convert(self, node: Node, t, path: str): + if isinstance(t, str): + if t in PRIMITIVES: + if node.is_complex: + self.issue("error", path, node, f"expected {t}, found frame {node.name!r}") + return plain(node) + return self.coerce(node, t, path) + if t == "vec3": + return self.vec3(node, path) + return plain(node) # "any" / "raw" + if isinstance(t, Shape): + if not node.is_complex: + self.issue("error", path, node, f"expected frame, found {node.kind} {node.name!r}") + return plain(node) + sub = {"_off": node.offset} + cur = _Cursor(node.children) + self.apply_fields(t.fields, cur, sub, path) + if not cur.done: + sub["_extra"] = [plain(n) for n in cur.nodes[cur.i:]] + self.issue("warn", path, cur.peek(), f"{len(cur.nodes) - cur.i} unexpected item(s) at end of frame") + return sub + if isinstance(t, CArr): + if not node.is_complex: + self.issue("error", path, node, f"expected framed array, found {node.kind} {node.name!r}") + return plain(node) + cur = _Cursor(node.children) + if cur.done: + return [] + count = self.coerce(cur.take(), "int", path) + items = [] + remaining = len(cur.nodes) - cur.i + if not isinstance(count, int) or count < 0 or count > remaining: + self.issue("error", path, node, f"framed-array count {count!r} exceeds {remaining} item(s)") + count = max(0, min(count if isinstance(count, int) else 0, remaining)) + for i in range(count): + if cur.done: + self.issue("error", path, node, f"framed array truncated: {i} of {count}") + break + before = cur.i + items.append(self.convert_elem(t.elem, cur, f"{path}[{i}]")) + if cur.i == before: + self.issue("error", path, cur.peek(), f"array element {i} consumed nothing; stopping") + break + if not cur.done: + self.issue("warn", path, cur.peek(), f"{len(cur.nodes) - cur.i} item(s) after framed array elements") + items.append({"_extra": [plain(n) for n in cur.nodes[cur.i:]]}) + return items + if isinstance(t, Seq): + raise TypeError("Seq cannot be converted from a single node") + raise TypeError(f"unknown field type {t!r}") + + def coerce(self, node: Node, kind: str, path: str): + k, raw = node.kind, node.raw + if k == kind: + return node.value + if kind == "int": + if k in ("float",) and len(raw) == 4: + return struct.unpack(" 1: + self.issue("warn", path, node, f"bool expected, word {raw.hex()} read") + return raw[0] != 0 + elif kind == "int64": + if k in ("int", "float") and len(raw) == 4: + self.issue("warn", path, node, "int64 expected, 4-byte word read (width mismatch)") + return struct.unpack(" (typed dict, issues).""" + ap = Applier() + return ap.apply(schema, root), ap.issues + + +# --- top-level API ------------------------------------------------------------ + +class SaveResult: + def __init__(self, inflated, padding, tree, typed, issues, stats): + self.inflated = inflated + self.padding = padding + self.tree = tree + self.typed = typed + self.issues = issues + self.stats = stats + + def count(self, level: str) -> int: + return sum(1 for i in self.issues if i.level == level) + + +def _walk_mode(data: bytes, padding: str, schema): + w = Walker(data, padding, schema) + tree = w.walk() + return w, tree + + +def read_bytes(data: bytes, padding: str = "auto", strict: bool = False, schema=ROOT) -> SaveResult: + """Parse a .sav (or already-inflated) byte string.""" + inflated = inflate(data) + if len(inflated) < 8: + raise SaveFormatError("stream too short to be a save", 0) + if padding == "auto": + best = None + for mode in ("joint", "split"): + w, tree = _walk_mode(inflated, mode, schema) + score = (w.stats["resyncs"], w.stats["hint_failures"], w.stats["raw_bytes"], + sum(1 for i in w.issues if i.level != "info")) + if best is None or score < best[0]: + best = (score, mode, w, tree) + _, padding, w, tree = best + else: + w, tree = _walk_mode(inflated, padding, schema) + typed, sissues = apply_schema(tree, schema) + issues = w.issues + sissues + res = SaveResult(inflated, padding, tree, typed, issues, dict(w.stats)) + if strict: + bad = [i for i in issues if i.level in ("error", "warn")] + if bad: + head = "\n ".join(repr(i) for i in bad[:20]) + raise SaveFormatError(f"strict: {len(bad)} schema/framing problem(s):\n {head}", bad[0].offset) + return res + + +def read_save(path: str, **kw) -> SaveResult: + with open(path, "rb") as f: + return read_bytes(f.read(), **kw) + + +# --- CLI ------------------------------------------------------------------------ + +def _summary_lines(res: SaveResult, path: str) -> list[str]: + t = res.typed + st = res.stats + lines = [f"file: {path} inflated: {len(res.inflated)} bytes padding: {res.padding}", + f"items: {st['items']} frames: {st['frames']} guessed: {st['guessed']} " + f"hint-failures: {st['hint_failures']} resyncs: {st['resyncs']} raw-bytes: {st['raw_bytes']}", + f"issues: {res.count('error')} error, {res.count('warn')} warn, {res.count('info')} info"] + s = t.get("summary") or {} + if isinstance(s, dict): + lines.append(f"summary: game={s.get('gameName')!r} turn={s.get('turn')} numSys={s.get('numSys')} " + f"players={len(s.get('players') or [])} scenario={s.get('scenario')!r}") + sim = t.get("sim") or {} + if isinstance(sim, dict): + lines.append(f"sim: gameName={sim.get('GameName')!r} players={len(sim.get('players') or [])} " + f"systems={len(sim.get('systems') or [])} fleets={len(sim.get('fleets') or [])}") + for p in sim.get("players") or []: + pl = p.get("Player") if isinstance(p, dict) else None + if isinstance(pl, dict): + lines.append(f" player {p.get('PlayerID')}: {pl.get('PlryName')!r} species={pl.get('Species')} " + f"home={pl.get('HomeSys')} sav={pl.get('Sav')} designs={len(pl.get('designs') or [])}") + return lines + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(description="Sword of the Stars (2006) save reader") + ap.add_argument("save", help=".sav (gzip) or already-inflated stream") + ap.add_argument("--dump", action="store_true", help="print the generic tree (text, or JSON with --json)") + ap.add_argument("--json", action="store_true", help="emit JSON (typed view, or tree with --dump)") + ap.add_argument("--strict", action="store_true", help="fail on any schema/framing deviation") + ap.add_argument("--padding", choices=("auto", "joint", "split"), default="auto") + ap.add_argument("--out", help="write output here instead of stdout") + ap.add_argument("--inflate", metavar="FILE", help="also write the inflated stream to FILE") + ap.add_argument("--issues", type=int, default=30, help="max issues to print (default 30)") + ap.add_argument("--max-depth", type=int, default=999, help="tree dump depth limit") + args = ap.parse_args(argv) + + try: + res = read_save(args.save, padding=args.padding, strict=args.strict) + except SaveFormatError as e: + print(f"error: {e}", file=sys.stderr) + return 2 + if args.inflate: + with open(args.inflate, "wb") as f: + f.write(res.inflated) + + if args.dump and args.json: + text = json.dumps({"padding": res.padding, "stats": res.stats, + "issues": [i.as_dict() for i in res.issues], + "tree": plain(res.tree)["_items"]}, indent=1) + elif args.dump: + text = "\n".join(dump_tree(res.tree, max_depth=args.max_depth)) + elif args.json: + text = json.dumps({"padding": res.padding, "stats": res.stats, + "issues": [i.as_dict() for i in res.issues], + "data": res.typed}, indent=1) + else: + lines = _summary_lines(res, args.save) + shown = [i for i in res.issues if i.level != "info"][:args.issues] + if shown: + lines.append("issues (errors/warnings, first %d):" % args.issues) + lines += [" " + repr(i) for i in shown] + text = "\n".join(lines) + + if args.out: + with open(args.out, "w", encoding="utf-8") as f: + f.write(text + "\n") + else: + sys.stdout.write(text + "\n") + return 1 if res.count("error") else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/save-reader/save_writer_stub.py b/save-reader/save_writer_stub.py new file mode 100644 index 0000000..a80bbf4 --- /dev/null +++ b/save-reader/save_writer_stub.py @@ -0,0 +1,330 @@ +#!/usr/bin/env python3 +"""save_writer_stub.py -- synthetic SOTS1 save builder for testing save_reader. + +This is NOT a game-compatible writer. It emits the same *framing* the reader +assumes (name-tagged values, NUL padding, BEEFBEEF/41104110 frames, gzip) and +drives itself off the reader's schema so every declared struct gets exercised. +Values are deterministic counters, so the expected typed dict is known up +front and the round trip can be asserted. Real-save validation is pending. + + python3 save_writer_stub.py out.sav [--padding joint|split] [--inflated] + +Stdlib only. +""" +from __future__ import annotations + +import argparse +import gzip +import random +import struct +import sys + +import save_reader as sr + +__all__ = ["SaveWriter", "Fixture", "build_fixture"] + + +class SaveWriter: + """Byte builder for the Streamable framing. padding = 'joint' | 'split'.""" + + def __init__(self, padding: str = "joint"): + if padding not in ("joint", "split"): + raise ValueError(padding) + self.padding = padding + self.buf = bytearray() + self.depth = 0 + + # -- primitives ---------------------------------------------------------------- + def _tag(self, name: str) -> int: + b = name.encode("ascii") + self.buf += struct.pack(" int: + """Named value. Returns the offset of the value bytes.""" + start = len(self.buf) + self._tag(name) + if self.padding == "split": + self._pad() + vp = len(self.buf) + self.buf += payload + self._pad() # joint: pad over tag+payload; split: pad payload + assert len(self.buf) == sr.pad4(len(self.buf)) and start % 4 == 0 + return vp + + def int(self, name, v): + return self.item(name, struct.pack(" tagless frame).""" + if name is not None: + self._tag(name) + self._pad() + self.buf += sr.BEGIN_BYTES + self.depth += 1 + + def end(self): + assert self.depth > 0 + self.buf += sr.END_BYTES + self.depth -= 1 + + def vec3(self, name, x, y, z, named: bool = False): + self.begin(name) + if named: + self.float(".", x) + self.float(".", y) + self.float(".", z) + else: + self.buf += struct.pack("<3f", x, y, z) + self.end() + + def bytes(self) -> bytes: + assert self.depth == 0, "unbalanced frames" + return bytes(self.buf) + + def gzip(self) -> bytes: + return gzip.compress(self.bytes(), mtime=0) + + +# --- schema-driven fixture ---------------------------------------------------- + +GATES_TRUE = {"vnh", "hindi", "isind", "HFPlan", "HLay", "hbq", "hsp", "HasAIR"} +OPT_EMIT = {"BQ"} # optional items the fixture does write +ARRAY_LEN = 2 + + +def _plain_scalar(name, kind, value): + return {"name": name, "kind": kind, "value": value} + + +class Fixture: + """Emits a synthetic save from sr.ROOT and records the expected typed dict.""" + + def __init__(self, padding="joint", seed=1): + self.w = SaveWriter(padding) + self.counter = 0 + self.rng = random.Random(seed) + self.expected = None + + # -- value policy ---------------------------------------------------------------- + def next(self) -> int: + self.counter += 1 + return self.counter + + def value(self, name, kind, ctx): + c = self.next() + if kind == "int": + if name == "idx" and ctx is sr.PlayerColor: + self.color_toggle = not getattr(self, "color_toggle", False) + return -1 if self.color_toggle else 3 # alternate custom-RGB / palette + return c if c % 7 else -c + if kind == "float": + return c + 0.25 + if kind == "bool": + return True if name in GATES_TRUE else bool(c % 2) + if kind == "int64": + return c * (1 << 33) + 7 + if kind == "string": + return "str%d%s" % (c, "x" * (c % 4)) + ("é" if c % 5 == 0 else "") + raise ValueError(kind) + + def emit_scalar(self, name, kind, ctx): + v = self.value(name, kind, ctx) + getattr(self.w, kind)(name, v) + return v + + # -- generic filler ---------------------------------------------------------------- + def generic_frame(self, name, with_blob=False): + """Frame with content the schema knows nothing about. Returns plain().""" + w = self.w + w.begin(name) + items = [] + if with_blob: + blob = bytes(self.rng.getrandbits(8) for _ in range(2500)) + while sr.END_BYTES in blob or sr.BEGIN_BYTES in blob: + blob = bytes(self.rng.getrandbits(8) for _ in range(2500)) + vp = w.raw("State", blob) + end = len(w.buf) + items.append(_plain_scalar("State", "raw", {"len": end - vp, "hex": w.buf[vp:vp + 32].hex()})) + else: + c = self.next() + w.int("ga", c) + items.append(_plain_scalar("ga", "int", c)) + w.bool("cmp", True) # 'cmp' is in the catalog -> bool + items.append(_plain_scalar("cmp", "bool", True)) + w.string("gs", "gen%d" % c) + items.append(_plain_scalar("gs", "string", "gen%d" % c)) + w.begin("gn") + w.float("gflt", c + 0.5) + w.end() + items.append({"_name": "gn", "_items": [_plain_scalar("gflt", "float", c + 0.5)]}) + w.begin(None) + w.int("ti", 42) + w.end() + items.append({"_name": None, "_items": [_plain_scalar("ti", "int", 42)]}) + w.end() + return {"_name": name, "_items": items} + + def filler_items(self): + c = self.next() + self.w.int("Filler", c) + out = [_plain_scalar("Filler", "int", c)] + out.append(self.generic_frame("FillerF")) + return out + + # -- schema walk ------------------------------------------------------------------- + def build(self) -> bytes: + self.expected = {} + self.emit_fields(sr.ROOT.fields, self.expected, None, top=True) + return self.w.bytes() + + def emit_fields(self, fields, out: dict, ctx, top=False): + for f in fields: + if isinstance(f, sr.Field): + self.emit_field(f, out, ctx) + elif isinstance(f, sr.Opt): + if f.field.name in OPT_EMIT: + self.emit_field(f.field, out, ctx) + elif isinstance(f, sr.If): + if out.get(f.key) == f.equals: + inner = f.inner + if isinstance(inner, sr.Seq): + self.emit_fields(inner.fields, out, ctx) + else: + self.emit_field(inner, out, ctx) + elif isinstance(f, sr.Until): + out[f.key] = self.filler_items() + elif isinstance(f, sr.Rest): + if top: + out[f.key] = self.cd_table() + else: + raise TypeError(f) + + def emit_field(self, f: sr.Field, out: dict, ctx): + t = f.type + if isinstance(t, sr.NArr): + self.w.int(f.name, ARRAY_LEN) + out[f.key] = [self.emit_elem(t.elem, ctx) for _ in range(ARRAY_LEN)] + elif isinstance(t, sr.Seq): + self.emit_fields(t.fields, out, ctx) + elif isinstance(t, sr.Shape): + if f.flex and self.counter % 2: + sub = {} + self.emit_fields(t.fields, sub, t) # inline variant + out[f.key] = sub + else: + # authoritative fields use their disk tag; R1-only fields + # borrow the Shape's (guessed) frame name where it has one + out[f.key] = self.emit_shape(f.name if f.auth else (t.name or f.name), t) + elif isinstance(t, sr.CArr): + if f.flex and self.counter % 2: + self.w.int(f.name, ARRAY_LEN) # inline variant == NArr + out[f.key] = [self.emit_elem(t.elem, ctx) for _ in range(ARRAY_LEN)] + else: + out[f.key] = self.emit_carr(f.name, t, ctx) + elif t == "vec3": + c = self.next() + v = [c + 0.5, c + 1.5, c + 2.5] + self.w.vec3(f.name, *v, named=bool(c % 3 == 0)) + out[f.key] = v + elif t == "any": + out[f.key] = self.generic_frame(f.name, with_blob=(f.name in sr.RAW_FRAMES)) + elif t in sr.PRIMITIVES: + out[f.key] = self.emit_scalar(f.name, t, ctx) + else: + raise TypeError(t) + + def emit_shape(self, name, shape: sr.Shape) -> dict: + self.w.begin(name) + sub = {} + self.emit_fields(shape.fields, sub, shape) + self.w.end() + return sub + + def emit_carr(self, name, carr: sr.CArr, ctx) -> list: + self.w.begin(name) + self.w.int("Count", ARRAY_LEN) + items = [self.emit_elem(carr.elem, ctx) for _ in range(ARRAY_LEN)] + self.w.end() + return items + + def emit_elem(self, elem, ctx): + if isinstance(elem, sr.Seq): + sub = {} + self.emit_fields(elem.fields, sub, ctx) + return sub + if isinstance(elem, sr.Field): + sub = {} + self.emit_field(elem, sub, ctx) + return sub.get(elem.key) + if isinstance(elem, sr.Shape): + return self.emit_shape(".", elem) # R1: element tags are often "." + if isinstance(elem, sr.CArr): + return self.emit_carr(".", elem, ctx) + if elem in sr.PRIMITIVES: + return self.emit_scalar("Item", elem, ctx) + raise TypeError(elem) + + def cd_table(self) -> list: + """Unframed 4th section: a framed string array, an unknown-shaped player + block and two AI blocks -- all opaque to the schema.""" + w = self.w + items = [] + w.begin("cdt") + w.int("Count", 2) + w.string("Item", "cd-a") + w.string("Item", "cd-b") + w.end() + items.append({"_name": "cdt", "_items": [_plain_scalar("Count", "int", 2), + _plain_scalar("Item", "string", "cd-a"), + _plain_scalar("Item", "string", "cd-b")]}) + items.append(self.generic_frame("cdplayer")) + items.append(self.generic_frame("cdai")) + items.append(self.generic_frame("cdai")) + return items + + +def build_fixture(padding="joint", seed=1): + """-> (inflated bytes, expected typed dict)""" + fx = Fixture(padding, seed) + data = fx.build() + return data, fx.expected + + +def main(argv=None) -> int: + ap = argparse.ArgumentParser(description="write a synthetic SOTS1 save for reader tests") + ap.add_argument("out") + ap.add_argument("--padding", choices=("joint", "split"), default="joint") + ap.add_argument("--inflated", action="store_true", help="write the raw stream, not gzip") + args = ap.parse_args(argv) + data, _ = build_fixture(args.padding) + with open(args.out, "wb") as f: + f.write(data if args.inflated else gzip.compress(data, mtime=0)) + print(f"wrote {args.out}: {len(data)} inflated bytes, padding={args.padding}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/save-reader/test_save_reader.py b/save-reader/test_save_reader.py new file mode 100644 index 0000000..dc3766c --- /dev/null +++ b/save-reader/test_save_reader.py @@ -0,0 +1,370 @@ +#!/usr/bin/env python3 +"""Unit tests for save_reader.py using the synthetic fixture only. + + /usr/bin/python3 -m unittest -v test_save_reader + +NOTE: these prove the reader against its OWN framing assumptions. Validation +against a real Sword of the Stars save is still pending. +""" +from __future__ import annotations + +import gzip +import io +import json +import os +import struct +import sys +import tempfile +import unittest +from contextlib import redirect_stdout + +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +import save_reader as sr +import save_writer_stub as sw + + +def norm(x): + """Drop offsets / guess annotations so typed output compares to expectations.""" + if isinstance(x, dict): + return {k: norm(v) for k, v in x.items() if k not in ("_off", "off", "guessed", "alt")} + if isinstance(x, list): + return [norm(v) for v in x] + return x + + +def walk(data, padding="joint", schema=None): + w = sr.Walker(data, padding, schema) + return w, w.walk() + + +class PrimitivesTest(unittest.TestCase): + def check_prims(self, padding): + w = sw.SaveWriter(padding) + # names of every length mod 4, every primitive width + w.int("Idx", 7) + w.int("Size", -3) + w.float("Suit", 0.75) + w.bool("NPC", True) # 3-char name + 1 byte: the padding-mode-sensitive case + w.bool("Abdn", False) + w.int64("Bats2", (5 << 33) + 9) + w.string("Name", "Sol") + w.string("PlryName", "") + w.string("Bdg", "café") # cp1252 byte 0xE9 + w.int("TRA", 100) + data = w.bytes() + self.assertEqual(len(data) % 4, 0) + wk, root = walk(data, padding) + kinds = [(n.name, n.kind, n.value) for n in root.children] + self.assertEqual(kinds, [ + ("Idx", "int", 7), ("Size", "int", -3), ("Suit", "float", 0.75), + ("NPC", "bool", True), ("Abdn", "bool", False), ("Bats2", "int64", (5 << 33) + 9), + ("Name", "string", "Sol"), ("PlryName", "string", ""), ("Bdg", "string", "café"), + ("TRA", "int", 100)]) + self.assertEqual(wk.stats["resyncs"], 0) + self.assertEqual(wk.issues, []) + # offsets are contiguous and 4-aligned + pos = 0 + for n in root.children: + self.assertEqual(n.offset, pos) + self.assertEqual(pos % 4, 0) + pos += n.size + self.assertEqual(pos, len(data)) + + def test_joint(self): + self.check_prims("joint") + + def test_split(self): + self.check_prims("split") + + def test_padding_layout_joint(self): + w = sw.SaveWriter("joint") + w.bool("NPC", True) + # [03 00 00 00]['N''P''C'][01] -> 8 bytes, no padding needed + self.assertEqual(w.bytes(), b"\x03\x00\x00\x00NPC\x01") + w = sw.SaveWriter("joint") + w.int("Idx", 1) + # 4 + 3 + 4 = 11 -> pad 1 + self.assertEqual(w.bytes(), b"\x03\x00\x00\x00Idx\x01\x00\x00\x00\x00") + + def test_padding_layout_split(self): + w = sw.SaveWriter("split") + w.bool("NPC", True) + self.assertEqual(w.bytes(), b"\x03\x00\x00\x00NPC\x00\x01\x00\x00\x00") + + def test_unknown_names_are_guessed(self): + w = sw.SaveWriter("joint") + w.int("zzq", 12) # 3-char names: bool/int/int64 sizes all differ + w.bool("zzb", True) + w.int64("zzl", 1 << 40) + w.string("zzs", "hello") + w.float("zzf", 3.5) + w.int("zzz", 99) + _, root = walk(w.bytes()) + got = [(n.name, n.kind, n.value, n.hinted) for n in root.children] + self.assertEqual(got, [("zzq", "int", 12, False), ("zzb", "bool", True, False), + ("zzl", "int64", 1 << 40, False), ("zzs", "string", "hello", False), + ("zzf", "float", 3.5, False), ("zzz", "int", 99, False)]) + + def test_word_ambiguity_is_reported_as_alt(self): + w = sw.SaveWriter("joint") + w.float("zzzz", 1.0) # 0x3F800000: unknown name -> float with int alt + _, root = walk(w.bytes()) + n = root.children[0] + self.assertEqual((n.kind, n.value, n.alt), ("float", 1.0, 0x3F800000)) + + def test_catalog_hint_overrides_guess(self): + w = sw.SaveWriter("joint") + w.float("Suit", 0.0) # bits 0 would be guessed int; catalog says float + w.bool("Elim", True) # 4-char name + bool is size-ambiguous with int + _, root = walk(w.bytes()) + self.assertEqual([(n.kind, n.value, n.hinted) for n in root.children], + [("float", 0.0, True), ("bool", True, True)]) + + +class FramingTest(unittest.TestCase): + def test_nested_frames_and_markers(self): + w = sw.SaveWriter() + w.begin("Outer") + w.int("a", 1) + w.begin("Inner") + w.string("s", "x") + w.end() + w.begin(None) # tagless frame + w.int("t", 2) + w.end() + w.end() + data = w.bytes() + self.assertTrue(data.startswith(b"\x05\x00\x00\x00Outer\x00\x00\x00" + sr.BEGIN_BYTES)) + self.assertTrue(data.endswith(sr.END_BYTES + sr.END_BYTES)) + wk, root = walk(data) + outer = root.children[0] + self.assertEqual((outer.name, outer.kind, outer.size), ("Outer", "complex", len(data))) + names = [(c.name, c.kind) for c in outer.children] + self.assertEqual(names, [("a", "int"), ("Inner", "complex"), (None, "complex")]) + self.assertEqual(outer.children[2].children[0].value, 2) + self.assertEqual(wk.issues, []) + + def test_unnamed_vec3_body(self): + w = sw.SaveWriter() + w.vec3("Pos", 1.5, -2.0, 1e6) + w.vec3("Pos", 1.5, -2.0, 1e6, named=True) + wk, root = walk(w.bytes()) + raw = root.children[0].children + self.assertEqual(len(raw), 1) + self.assertEqual(raw[0].kind, "raw") + self.assertEqual(struct.unpack("<3f", raw[0].raw), (1.5, -2.0, 1e6)) + self.assertEqual([c.kind for c in root.children[1].children], ["float"] * 3) + self.assertTrue(all(i.level == "info" for i in wk.issues)) + ap = sr.Applier() + self.assertEqual(ap.vec3(root.children[0], ""), [1.5, -2.0, 1e6]) + self.assertEqual(ap.vec3(root.children[1], ""), [1.5, -2.0, 1e6]) + + def test_resync_through_garbage(self): + """A frame whose body is unknown binary must not derail the parse.""" + import random + rnd = random.Random(7) + blob = bytes(rnd.getrandbits(8) for _ in range(2500)) + w = sw.SaveWriter() + w.int("before", 1) + w.begin("Mystery") + w.raw("State", blob) # named blob, unknown name -> no layout fits + w.end() + w.begin("Junk") + w.buf += blob[:301] # untagged garbage, unaligned length + w._pad() + w.end() + w.int("after", 2) + wk, root = walk(w.bytes()) + self.assertEqual([c.name for c in root.children], ["before", "Mystery", "Junk", "after"]) + self.assertEqual(root.children[3].value, 2) + self.assertEqual(root.children[1].children[0].kind, "raw") + self.assertEqual(root.children[2].children[0].kind, "raw") + self.assertEqual(wk.stats["resyncs"], 2) + self.assertTrue(any(i.level == "warn" for i in wk.issues)) + + def test_raw_frame_hint(self): + """RNG is declared opaque: its body is read to the END marker without warnings.""" + w = sw.SaveWriter() + w.begin("RNG") + w.raw("State", bytes(range(256)) * 9 + b"\x00" * 196) + w.end() + wk, root = walk(w.bytes()) + n = root.children[0].children[0] + self.assertEqual((n.name, n.kind, n.hinted), ("State", "raw", True)) + self.assertEqual(n.value["len"], 2500 + 3) # joint padding bytes are included + self.assertEqual(wk.stats["resyncs"], 0) + + def test_truncated_stream(self): + w = sw.SaveWriter() + w.begin("Summary") + w.int("turn", 5) + w.string("gameName", "abcdef") + w.end() + data = w.bytes()[:-9] + wk, root = walk(data) + self.assertEqual(root.children[0].children[0].value, 5) + self.assertTrue(any("not terminated" in i.msg for i in wk.issues)) + self.assertTrue(all(i.offset <= len(data) for i in wk.issues)) + + def test_damaged_gzip(self): + data = gzip.compress(b"\x03\x00\x00\x00Idx\x01\x00\x00\x00\x00")[:-6] + with self.assertRaises(sr.SaveFormatError): + sr.read_bytes(data) + + def test_inflate_passthrough(self): + raw = b"\x03\x00\x00\x00Idx\x01\x00\x00\x00\x00" + self.assertEqual(sr.inflate(raw), raw) + self.assertEqual(sr.inflate(gzip.compress(raw)), raw) + + +class SchemaRoundTripTest(unittest.TestCase): + def roundtrip(self, padding, request="auto"): + data, expected = sw.build_fixture(padding) + res = sr.read_bytes(gzip.compress(data), padding=request, strict=True) + self.assertEqual(res.padding, padding) + self.assertEqual(res.count("error"), 0) + self.assertEqual(res.count("warn"), 0) + self.assertEqual(res.stats["resyncs"], 0) + self.assertEqual(norm(res.typed), expected) + return res + + def test_roundtrip_joint(self): + res = self.roundtrip("joint") + sim = res.typed["sim"] + self.assertEqual(len(sim["players"]), 2) + self.assertEqual(len(sim["systems"]), 2) + sysd = sim["systems"][0]["Sys"] + self.assertIsInstance(sysd["Bats2"], int) + self.assertGreater(sysd["Bats2"], 1 << 33) + self.assertIsInstance(sysd["Abdn"], bool) + self.assertEqual(len(sysd["Pos"]), 3) + self.assertIn("indi", sysd) # hindi gate honoured + ply = sim["players"][0]["Player"] + self.assertIsInstance(ply["pswd"], str) + self.assertIsInstance(ply["TRM"], float) + self.assertEqual(len(ply["nexp"]), 2) + self.assertEqual(set(ply["nexp"][0]), {"xid", "xmin", "xmax", "xper"}) + ship = sim["fleets"][0]["Flt"]["ships"][0]["Ship"] + self.assertIsInstance(ship["RefCap"], float) + self.assertIn("BQ2", ship) + self.assertIn("pop", ship) + colors = [p["slot"]["fxCrId"] for p in res.typed["summary"]["players"]] + custom = [c for c in colors if c["idx"] == -1] + self.assertTrue(custom and all("r" in c for c in custom)) # -1 -> RGB follows + self.assertTrue(all("r" not in c for c in colors if c["idx"] != -1)) + + def test_roundtrip_split(self): + self.roundtrip("split") + + def test_explicit_padding_mode(self): + self.roundtrip("joint", "joint") + self.roundtrip("split", "split") + + def test_wrong_padding_mode_is_noisy(self): + data, _ = sw.build_fixture("split") + res = sr.read_bytes(data, padding="joint") + self.assertGreater(res.stats["resyncs"] + res.stats["hint_failures"] + res.count("error"), 0) + + def test_json_serializable(self): + data, _ = sw.build_fixture("joint") + res = sr.read_bytes(data) + json.dumps(res.typed) + json.dumps(sr.plain(res.tree)) + json.dumps([i.as_dict() for i in res.issues]) + + def test_strict_rejects_missing_authoritative_field(self): + w = sw.SaveWriter() + w.begin("Sys") + w.vec3("Pos", 0, 0, 0) + w.float("R", 1); w.float("G", 1); w.float("B", 1); w.float("A", 1) + w.int("Idx", 3) + w.int("Suit", 0) # Size missing; Suit written as int + w.end() + data = w.bytes() + schema = sr.Seq([sr.R("sys", sr.Sys)]) + res = sr.read_bytes(data, padding="joint", schema=schema) + self.assertIsNone(res.typed["sys"]["Size"]) + self.assertTrue(any(i.level == "error" and "Size" in i.msg for i in res.issues)) + with self.assertRaises(sr.SaveFormatError): + sr.read_bytes(data, padding="joint", schema=schema, strict=True) + + def test_lenient_skips_unexpected_items(self): + w = sw.SaveWriter() + w.begin("PopG") + w.int("PopT", 1) + w.int("Legacy", 5) # not in the schema + w.int("PopS", 2) + w.int64("PopC", 3) + w.end() + schema = sr.Seq([sr.R("g", sr.PopG)]) + res = sr.read_bytes(w.bytes(), padding="joint", schema=schema) + g = res.typed["g"] + self.assertEqual((g["PopT"], g["PopS"], g["PopC"]), (1, 2, 3)) + self.assertEqual(g["_unexpected"][0]["name"], "Legacy") + self.assertTrue(any(i.level == "warn" for i in res.issues)) + + def test_coerce_retypes_words(self): + """A schema float stored as a word the walker guessed 'int' is re-read from raw bytes.""" + w = sw.SaveWriter() + w.begin("Tmrs") + w.int("tstl", 0x7F7FFFFF) + w.float("tctl", 60.0) # 0x42700000, walker may call it float; schema says int + w.int("tqtl", 0x7F7FFFFF) + w.int("tqtle", 0) + w.end() + schema = sr.Seq([sr.R("t", sr.Tmrs)]) + res = sr.read_bytes(w.bytes(), padding="joint", schema=schema) + self.assertEqual(res.typed["t"]["tctl"], 0x42700000) + self.assertEqual(res.typed["t"]["tstl"], 0x7F7FFFFF) + + +class CliTest(unittest.TestCase): + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + data, _ = sw.build_fixture("joint") + self.path = os.path.join(self.tmp.name, "fixture.sav") + with open(self.path, "wb") as f: + f.write(gzip.compress(data)) + + def tearDown(self): + self.tmp.cleanup() + + def run_cli(self, *args): + buf = io.StringIO() + with redirect_stdout(buf): + rc = sr.main([self.path, *args]) + return rc, buf.getvalue() + + def test_summary(self): + rc, out = self.run_cli() + self.assertEqual(rc, 0) + self.assertIn("padding: joint", out) + self.assertIn("summary: game=", out) + + def test_dump(self): + rc, out = self.run_cli("--dump") + self.assertEqual(rc, 0) + self.assertIn("@00000000 Summary {", out) + self.assertIn("Bats2 int64", out) + + def test_json_and_inflate(self): + outp = os.path.join(self.tmp.name, "out.json") + infl = os.path.join(self.tmp.name, "inflated.bin") + rc, _ = self.run_cli("--json", "--strict", "--out", outp, "--inflate", infl) + self.assertEqual(rc, 0) + with open(outp) as f: + doc = json.load(f) + self.assertIn("summary", doc["data"]) + self.assertEqual(doc["padding"], "joint") + with open(infl, "rb") as f: + self.assertTrue(f.read().startswith(b"\x07\x00\x00\x00Summary")) + + def test_dump_json(self): + rc, out = self.run_cli("--dump", "--json") + self.assertEqual(rc, 0) + doc = json.loads(out) + self.assertEqual(doc["tree"][0]["_name"], "Summary") + + +if __name__ == "__main__": + unittest.main()