control-flow: full app/turn spine with addresses; types written into Ghidra; threads cleared for battle-load
This commit is contained in:
alex
parent
47463c7905
commit
bb01d8e655
32 changed files with 5789 additions and 6 deletions
|
|
@ -7,8 +7,8 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`).
|
|||
|---|---|---|---|---|---|---|
|
||||
| RTTI class inventory | meta | mapped | high | 100% | 2026-09-07 | `findings/objects/00-inventory.md` — 1924 types, engine=Mars |
|
||||
| `.gob` format | subsystem | mapped | high | 100% | 2026-09-07 | renamed uncompressed ZIP (community-known) |
|
||||
| `Mars::AppStartup` (entry) | control-flow | backlog | — | 0% | 2026-09-07 | bootstrap → main loop |
|
||||
| main loop / tick dispatch | control-flow | in-progress | — | 0% | 2026-09-07 | turn SM known from names: strategic round -> SETurnEndPending -> SEProcessTurn -> querying round -> combat round -> SNMAllCombatDone. Ghidra: StrategyMapScreen ctor, ScreenBar::ToggleList, CombatScreen load |
|
||||
| `Mars::AppStartup` (entry) | control-flow | mapped | high | 100% | 2026-09-07 | entry 0x00925794 -> WinMain 0x0089dd30 -> DemoApp ctor 0x0089c950 -> Mars::Application::Initialize 0x008a0e50 (config/affinity, D3D9, window, sound thread, OnStartup -> net thread). findings/control-flow/turn-spine.md |
|
||||
| main loop / tick dispatch | control-flow | mapped | high | 90% | 2026-09-07 | Application::Run 0x0089f5b0: FrameTimer, PanelManager, DemoApp::OnUpdate 0x00898800 / OnTick 0x0089a640 / OnRender 0x00899210. Turn pipeline: EndTurn 0x00783be0 -> BeginProcessTurn 0x007d98e0 -> StrategyServer::ProcessTurn 0x007dc6c0 -> RunCombatRound 0x007cbe80 -> SETurnResults -> ResumePlaying 0x007ddc90. Lockstep on every machine |
|
||||
| `Game::ClientPlayer` / `AIPlayer` | object | mapped | high | 80% | 2026-09-07 | empire STATE = Game::ServerPlayer (~110 members, abs 0x28..0x3dc; IStreamable +0x3a0; read FUN_008804d0 / write FUN_008563e0). ClientPlayer = client view. findings/objects/struct-recovery.md. Verify vs real save pending |
|
||||
| `Game::StarSystem` | object | mapped | high | 80% | 2026-09-07 | Game::ServerSystem = star system + colony (~90 members, abs 0x18..0x2d4; read FUN_0075d4b0 / write FUN_00749630). PlayerView per-player fog view. Verify vs real save pending |
|
||||
| `Game::Planet` / `DOPlanet` | object | mapped | high | 100% | 2026-09-07 | Planet : Actor is a RENDER actor, not streamed; all colony state is in ServerSystem (+PlayerView in NVs map) |
|
||||
|
|
@ -25,10 +25,10 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`).
|
|||
| string / config intel | meta | mapped | high | 100% | 2026-09-07 | findings/subsystems/strings-and-config.md |
|
||||
| Mars brace-block parser | subsystem | verified | high | 100% | 2026-09-07 | verify/parsers/ (mars_data, flat_kv, manifest, effect_txt, verify.py): 1595/1595 files parse, 0 dangling cross-links; catalogs in verify/results/data-catalogs/ |
|
||||
| networking (SNM/FNM + GameSpy) | subsystem | backlog | — | 0% | 2026-09-07 | ~60 SNM strategy msgs, FNM file-xfer/host-migration; lockstep sim |
|
||||
| battle-load: thread contention | subsystem | backlog | — | 0% | 2026-09-07 | root cause hunt: CreateThread sites FUN_00902350, FUN_00736e30, FUN_008a0e50; streaming-sound thread; lockstep sync waits |
|
||||
| battle-load: thread contention | subsystem | mapped | med | 50% | 2026-09-07 | 3 CreateThread sites CLEARED (net watchdog 0x00901f40, TIME_CRITICAL audio streaming 0x008ef1d0, starmap blob mesh 0x00735bb0). Sim+combat load are on the MAIN thread -> candidates: audio thread CS (g_musicCS) or D3D9 driver threads. Next: dynamic profile under x32dbg once the game runs |
|
||||
| UI screen & flow map | meta | mapped | high | 100% | 2026-09-07 | findings/subsystems/ui-screen-map.md - 36 screens; screens are C++ on Mars controls (NOT data); turn state machine recovered |
|
||||
| `Game::StrategyServer` (sim block) | object | mapped | high | 80% | 2026-09-07 | top-level sim state; read FUN_007d27a0 / write FUN_0079fa70; IStreamable +0 |
|
||||
| stream primitive API | subsystem | mapped | high | 100% | 2026-09-07 | IStreamable vft: +0x18 string, +0x1c bool, +0x20 float, +0x24 int, +0x28 nested, +0x30 raw; FUN_00816490 = NetworkObject handle id |
|
||||
| real save for verification | verify | in-progress | — | 0% | 2026-09-07 | need the game running (DXVK+lavapipe on VM140) -> autosave -> parse with recovered layouts |
|
||||
| save_reader.py | verify | in-progress | — | 0% | 2026-09-07 | Streamable reader (gzip, name-tagged, BEEFBEEF framing) from reference + recovered layouts |
|
||||
| Ghidra type write-back | meta | in-progress | — | 0% | 2026-09-07 | create struct datatypes + rename serializers in the sots project so ReVa sessions inherit the map |
|
||||
| Ghidra type write-back | meta | verified | high | 100% | 2026-09-07 | structs saved in project (ServerSystem 87f, ServerPlayer 110f, StarFleet, StarShip, StrategyServer partial, 22 nested); 52 serializers + primitives + ~60 spine fns renamed; decompile shows field names |
|
||||
|
|
|
|||
|
|
@ -20,3 +20,5 @@ Each links to the finding that raised it. Promoted to backlog or closed by **re-
|
|||
- **Engine parser leniency** — 12 shipped shipsections are syntactically broken (unclosed `{`, extra `}`) yet load; keys and identifiers are case-insensitive. Reimplementation must match this leniency. `.effect` is its own `TXT`/`BEGIN-END` format, not brace-block (corrects round one). (from [[data-parsers]])
|
||||
- **Struct recovery leftovers** — `PlayerColorID` exact on-disk width (writer `FUN_008b9cb0` undecompiled); `ServerSystem+0x10` owner type; `TechTree` per-tech body; `CdPlayer` block. R2's `OID = PID*16` is an id-allocation pattern, not in this code. (from [[struct-recovery]])
|
||||
- **Resolved (R1/R2 contradictions)** — `Bats2`/`rcex` are int64 (R2 wrong); `Abdn`/`Dstyd` bools, `ltis` int; `TRM`/`CstR/E/T`/`shrm`/`RefCap`/`RepCap`/PlayerView `Infra` are floats; `pswd` string; `TShn`/`ETS`/diplomacy counters int16 in memory, int32 on disk; `Nexp` carries `xid/xmin/xmax/xper`; `FtOrig` is Vector3. (from [[struct-recovery]])
|
||||
- **Battle-load, narrowed** — sim/combat load run on the main thread; the only other threads are net watchdog, TIME_CRITICAL audio streaming (`g_musicCS`), and a star-map mesh builder. Hypothesis: audio-thread critical-section contention or D3D9 runtime/driver threads on many cores. Needs a dynamic profile (x32dbg / ETW) under the software-GPU stack. (from [[turn-spine]])
|
||||
- **Spine leftovers** — `StrategyServer` struct partial (41 fields); static-initialiser region 0x009be000–0x009c1400 undisassembled; `Mars::Stream` vftable not located; several small ProcessTurn phase fns unnamed. (from [[turn-spine]])
|
||||
|
|
|
|||
|
|
@ -1 +0,0 @@
|
|||
# placeholder — populated during the campaign
|
||||
250
findings/control-flow/turn-spine.md
Normal file
250
findings/control-flow/turn-spine.md
Normal file
|
|
@ -0,0 +1,250 @@
|
|||
# Ghidra write-back + turn/main-loop spine — Sword of the Stars (2006)
|
||||
|
||||
Program `sots` / "Sword of the Stars.exe", ImageBase 0x00400000. All addresses are VAs. Ghidra 12.0.4 headless,
|
||||
project `/srv/re-lab/ghidra-projects/sots`; all changes below are **saved in the project** (four headless runs, each
|
||||
"Save succeeded"). Scripts (CT111 `/root/`, copies in `/srv/re-lab/handoff/scripts/`): `SpineRecon.java`..`SpineRecon4.java`
|
||||
(read-only recon), `WriteBack.java` (types + renames + labels), `FixThis.java` (class-struct merge + serializer `this`
|
||||
views), `FixPlayer.java`. Raw decompiles used for the analysis: CT111 `/tmp/spine/*.c`; the verification decompiles are in
|
||||
`/srv/re-lab/handoff/ghidra-verify/`.
|
||||
|
||||
Inputs: `findings/objects/struct-recovery.md` (member tables), `findings/objects/ghidra-recon.md` (affinity, vftables),
|
||||
`findings/subsystems/ui-screen-map.md` §3 (SE*/SNM* names).
|
||||
|
||||
---
|
||||
|
||||
## 1. Type write-back
|
||||
|
||||
### 1.1 Where the types live (important for anyone continuing in Ghidra)
|
||||
Ghidra resolves the `this` type of a `__thiscall` function in class namespace `Game::X` to the structure
|
||||
**`/Game/X`** (`VariableUtilities.findOrCreateClassStruct`). The recovered structs were therefore merged *into those
|
||||
class structs*, so every function placed in the class namespace decompiles with field names automatically:
|
||||
|
||||
| Ghidra data type | size | fields | note |
|
||||
|---|---|---|---|
|
||||
| `/Game/ServerSystem` | 0x2d8 | 87 | abs offsets from struct-recovery §1 (+ base vptrs/netId/owner) |
|
||||
| `/Game/ServerPlayer` | 0x3e0 | 110 | §2; IStreamable sub-object at +0x3a0 |
|
||||
| `/Game/StarFleet` | 0x120 | 21 | §3 |
|
||||
| `/Game/StarShip` | 0xb0 | 26 | §4 |
|
||||
| `/Game/StrategyServer` | 0x320 | 41 | **partial**: offsets from `StrategyServer::Write` + the spine functions (Frame/turn @+0xc, Systems @+0x44, Players @+0x54, Fleets @+0x64, Acts @+0x74, OnEventCallback @+0x170, StrategyEvents list @+0x2b8, bProcessingTurn @+0x2f1, …) |
|
||||
| `/Game/Population` 0x14, `PopulationGroup` 0x18, `Morale` 0x20, `MoraleEvent` 0x50, `IndependenceInfo` 0x70, `PlayerColorID` 4, `ShipBuildOrder` 0x18, `DiplomacyStats` 0x24, `PlayerReport` 0x30, `PlayerAlliances` 0x10, `NodeRoute` 0x10, `FlightPlan` 0x38, `FlightPlan/Waypoint` 0x1c, `ShipHealth` 0x10, `PrisonerHold` 0x18, `EventStorage` 0x1c, `CivilianRatios` 0x2c, `ShipRecords` 0x44, `SpyReport` 0x34, `/Game/StarSystem/OutputRates` 0x1c, `/Game/StarSystem/PlayerView` 0x9c, `/Mars/Vector3` 12 | | | nested types from §1.1–§2.5 / §3.1 |
|
||||
| `/SOTS/std::string` 0x1c, `std::vector` 12, `std::list` 8, `std::map` 8, `Game::FleetLayout` 0x24 | | | helpers. `std::string` layout **verified** from `Stream::WriteString` (`if (s->_Myres > 15) p = s->_Ptr`): `_Bx`@0 (16), `_Mysize`@0x10, `_Myres`@0x14, `_Alval`@0x18 |
|
||||
| `/SOTS/Game::ServerSystem_ser8`, `StarFleet_ser8`, `StarShip_ser8`, `ServerPlayer_ser928` | | | **serializer views**: the same fields shifted by the IStreamable COL offset (8 / 0x3a0), because inside `X::Read/Write` `this = object + COL`. Applied (custom storage, `this` in ECX) to the eight Read/Write functions. For ServerPlayer only the +0x3a0.. tail is visible; the rest stays as `this + -0x378` etc. |
|
||||
| `/Game/BuildQueue`, `TechTree`, `ShipDesign`, `Tech`, `CommMessageContainer`, `FleetNameGenerator`, `AIRebellion`, `AIEncounterFlags`, `ServerNodeGraph`, `ServerTradeManager`, `IServerSpyManager`, `AttribMap`, `SVScriptObject`, `StrategyEvent`, `/Mars/Stream` | 0 | | opaque, pointer targets only |
|
||||
|
||||
Every struct carries a description citing `struct-recovery.md`; gaps are left undefined. One extra member was added from
|
||||
the spine work: `ServerPlayer+0xf9 bTurnDone_nonser` (tested by `StrategyServer::OnPlayerEndTurn`, not serialized), and
|
||||
`ServerPlayer+0x164 Status` is the field `SNMSetPlayerStatus` writes (4 = "done", 1 = playing).
|
||||
|
||||
### 1.2 Renames (all with plate comments citing the finding doc)
|
||||
*Serializers* (52): `Game::ServerSystem::Read/Write` (0x0075d4b0/0x00749630), `Game::ServerPlayer::Read/Write`
|
||||
(0x008804d0/0x008563e0), `Game::StarFleet::Read/Write` (0x00702470/0x00701070), `Game::StarShip::Read/Write`
|
||||
(0x00853fa0/0x008291f0), `Game::StrategyServer::Read/Write` (0x007d27a0/0x0079fa70), `Game::StarMapNode::Read/Write`
|
||||
(0x00727790/0x00727820), `Game::StarSystem::PlayerView::Read/Write`, `::OutputRates::Read/Write`, `Population`,
|
||||
`PopulationGroup`, `IndependenceInfo`, `Morale`, `MoraleEvent`, `ShipBuildOrder`, `PlayerNotes`, `SpyReport`,
|
||||
`PlayerReport`, `DiplomacyStats::Write`, `FlightPlan`, `FlightPlan::Waypoint`, `NodeRoute`, `PrisonerHold`,
|
||||
`EventStorage::Write`, `PlayerAlliances::Write`, `ShipHealth::Write`, `PlayerColorID::Write`, `Mars::Vector3::Write`,
|
||||
`TechTree::Write`, `ShipRecords::Write`, `CivilianRatios::Write` — exactly the addresses in struct-recovery §0.
|
||||
|
||||
*Stream primitives* (`Mars::Stream::…`): `WriteString` 0x008b9d70, `WriteBool` 0x008b9c20, `WriteFloat` 0x008b9be0,
|
||||
`WriteInt` 0x008b9d50, `WriteInt16AsInt` 0x008b9d00, `WriteInt8AsInt` 0x008b9cb0, `WriteInt64` 0x008b9c60,
|
||||
`WriteHandleId` 0x00816490, `ReadFloat` 0x008b9bc0, `ReadInt` 0x008b9d20, `ReadBool` 0x008b9c00, `ReadString`
|
||||
0x008b9d90, `ReadInt64` 0x008b9c40, `ReadInt16` 0x008b9cd0, `ReadHandle` 0x008164d0. Each plate comment lists the Stream
|
||||
vftable slot map (+0x10 ReadIntRef, +0x14 ReadNested, +0x18 String, +0x1c Bool, +0x20 Float, +0x24 Int, +0x28 Nested,
|
||||
+0x30 Raw). The `Mars::Stream` vftable itself was not located this round (the wrappers are the only anchor).
|
||||
|
||||
*Affinity / app*: `Process_PinAffinity` 0x0089ee70; `Mars::Application::Initialize` 0x008a0e50 (+ secondary label
|
||||
`AppStartup_ReadConfig` at the same address, as requested); `WinMain` 0x0089dd30; `Mars::Application::Run` 0x0089f5b0;
|
||||
`Mars::Application::PumpMessages` 0x0089f1c0; `Mars::Application::Application` 0x008a0170; `CreateAppWindow` 0x0089fe70;
|
||||
`Mars::FrameTimer::Update` 0x0090c700; `Mars::TimerList::Dispatch` 0x008e5ac0; `Mars::AppStartup::OnConfigToken`
|
||||
0x0089f4d0; `Mars::ConfigParser::ParseFile` 0x008cd820; `Game::DemoApp::{DemoApp 0x0089c950, OnStartup 0x0089d610,
|
||||
OnShutdown 0x0089dfb0, OnUpdate 0x00898800, OnTick 0x0089a640, OnRender 0x00899210, ShouldSleepWhenInactive 0x008986a0,
|
||||
CreateStrategyGame 0x00898b00}`. Globals: `g_pApplication` 0x00b2d540, `g_pDemoApp` 0x00b2d0bc.
|
||||
|
||||
*Threads, game creation, turn spine*: see §2/§3 (every function named there was renamed; ~60 functions).
|
||||
|
||||
*Network message registry*: 82 registry entries labelled `NetMsgReg_<name>` (EOL comment = id/factory), the 21 unnamed
|
||||
factory functions renamed `Game::<Msg>::Create`, `Mars::NetMessageRegistry::Register` 0x008d2290,
|
||||
`g_NetMsgRegistryById` 0x00b2ddf8. EOL comments on the three `CreateThread` sites.
|
||||
|
||||
### 1.3 Verification (re-decompile with the new types)
|
||||
`Game::ServerSystem::Write` (0x00749630), `this` typed `ServerSystem_ser8*` — field names appear for every tag
|
||||
(`/srv/re-lab/handoff/ghidra-verify/verify_ServerSystem_Write.c`, 125 `this->` references; Read: 205):
|
||||
```c
|
||||
void __thiscall Game::ServerSystem::Write(ServerSystem_ser8 *this, void *stream)
|
||||
...
|
||||
StarMapNode::Write(stream);
|
||||
Mars::Stream::WriteFloat(pvVar3,"R",&this->R,0xffffffff,uVar5);
|
||||
Mars::Stream::WriteFloat(pvVar3,"G",&this->G,0xffffffff);
|
||||
Mars::Stream::WriteFloat(pvVar3,"B",&this->B,0xffffffff);
|
||||
Mars::Stream::WriteFloat(pvVar3,"A",&this->A,0xffffffff);
|
||||
Mars::Stream::WriteInt(pvVar3,"Idx",&this->Idx,0xffffffff);
|
||||
Mars::Stream::WriteInt(pvVar3,"Size",&this->Size,0xffffffff);
|
||||
Mars::Stream::WriteFloat(pvVar3,"Suit",&this->Suit,0xffffffff);
|
||||
Mars::Stream::WriteInt(pvVar3,"Res",&this->Res,0xffffffff);
|
||||
Mars::Stream::WriteInt(pvVar3,"ARes2",&this->ARes2,0xffffffff);
|
||||
Mars::Stream::WriteInt(pvVar3,"MRes",&this->MRes,0xffffffff);
|
||||
Mars::Stream::WriteBool(pvVar3,"NoRebAI",&this->NoRebAI,0xffffffff);
|
||||
Mars::Stream::WriteInt(pvVar3,"TRes",&this->TRes,0xffffffff);
|
||||
Mars::Stream::WriteInt(pvVar3,"Pop",&this->Pop,0xffffffff);
|
||||
local_1c = &this->Pop2; local_24 = Mars::StreamableHelper<class_Game::Population>::vftable;
|
||||
(**(code **)(*(int *)pvVar3 + 0x28))("Pop2",&local_24);
|
||||
Mars::Stream::WriteFloat(pvVar3,"Infra",&this->Infra,0xffffffff);
|
||||
Mars::Stream::WriteInt(pvVar3,"PvPop",&this->PvPop,0xffffffff);
|
||||
...
|
||||
```
|
||||
`Game::StrategyServer::Write` (COL 0, plain `StrategyServer*`): `WriteString(stream,"KeyPath",&this->KeyPath)`,
|
||||
`local_60 = (this->NodeMapLines._Mylast - _Myfirst)/0x14` (NMSz), `WriteInt(stream,"ModCount",&this->ModCount)`,
|
||||
`WriteInt(stream,"Frame",&this->Frame)`, `("GameID",this->GameID)`, `this->Attrib`, `&this->GameName`, `this->Map`,
|
||||
`&this->IncMod`, `&this->ResMod` … (69 refs). Spine functions after the merge: `BeginProcessTurn`:
|
||||
`this->Frame = this->Frame + 1; this->bProcessingTurn = true; log("Begin processing turn %d", this->Frame)`;
|
||||
`OnPlayerEndTurn`: `piVar10 = this->Players._Myfirst … if (this->OnEventCallback == NULL) … (*this->OnEventCallback)(id,0x27,&ev)`;
|
||||
`ServerSystem::ProcessTurn`: `if (this->PID == NULL) this->Infra -= …; if (this->indi) iVar4 = this->indi->indsp; this->ntdev = 0`.
|
||||
Every save tag in the decompile lines up with the field of the same name — the offset tables in struct-recovery.md are
|
||||
confirmed by the decompiler itself.
|
||||
|
||||
---
|
||||
|
||||
## 2. Turn / main-loop spine
|
||||
|
||||
### 2.1 Entry → init → main loop
|
||||
```
|
||||
entry 0x00925794 → ___tmainCRTStartup 0x00925501 → WinMain 0x0089dd30 (hInst=0x400000, cmdline, nShow)
|
||||
├ single-instance mutex "Kerberos_SwordOfTheStars_Mutex" (bypass: /concurrent); FindWindow+SetForegroundWindow otherwise
|
||||
├ new Game::DemoApp (0x1b8) 0x0089c950 → Mars::Application::Application 0x008a0170 (g_pApplication = this)
|
||||
├ Mars::Application::Initialize 0x008a0e50 ("AppStartup_ReadConfig")
|
||||
│ CPU/ForceSingleCore > 0 → Process_PinAffinity 0x0089ee70 (SetProcessAffinityMask 1<<core)
|
||||
│ "/startup:" cmdline → Mars::ConfigParser::ParseFile 0x008cd820 with Mars::AppStartup (vft[1] = OnConfigToken 0x0089f4d0; "conlevel")
|
||||
│ display.cfg, audio.cfg, CoInitialize, Direct3DCreate9(0x20|0x1f), CreateAppWindow 0x0089fe70, DrawDevice 0x008df760, ClipCursor
|
||||
│ timers → CreateEvent×2 + CreateThread(SoundSystem::StreamingUpdateThreadProc) prio 15 ← THREAD 2
|
||||
│ GlobalConsts, textures, effect pool, sprites, keymap, font, SimplePainter, PanelManager 0x0090f1e0
|
||||
│ IApplication::OnStartup (vft+0x10) = Game::DemoApp::OnStartup 0x0089d610
|
||||
│ banner "Sword of the Stars 1.8.1 (Wed Dec 13 03:38:31 2017)", profiles, Mars::Network::Startup 0x00902470 → NetworkManager::Create 0x00902350 ← THREAD 1
|
||||
│ ParticleSystemManager, Mesh, Model, StringTable, GUIResources, SoundSystem, sound_ui.csv, SpeechEvents, GUIAppearances, colour/badge/avatar tables, main menu / "/join"
|
||||
└ Mars::Application::Run 0x0089f5b0 ← MAIN LOOP
|
||||
do {
|
||||
Mars::FrameTimer::Update 0x0090c700
|
||||
PanelManager->vft+0x60() (UI/input update)
|
||||
app->vft+0x18 = DemoApp::OnUpdate 0x00898800 (Mars network pump FUN_00902ad0; current game object(+0x158)->Update; strategy game (+0x150) → FUN_007879c0)
|
||||
if (!focused && app->vft+0xc ShouldSleepWhenInactive) Sleep(20)
|
||||
_controlfp(...); every _DAT_00a36860 s → Mars::TimerList::Dispatch 0x008e5ac0
|
||||
ok = app->vft+0x1c = DemoApp::OnTick 0x0089a640 (sound/console/panels + top-level state machine: loads/unloads strategy game @+0x150 and combat @+0x158; 0 when quitting)
|
||||
if (ok && DrawDevice) { dev->Begin; if (!dev->IsLost) { dev->+8; app->vft+0x20 = DemoApp::OnRender 0x00899210 (PanelManager->vft+0x64 draw / movie player); dev->+0xc; dev->Present } }
|
||||
} while (Mars::Application::PumpMessages 0x0089f1c0 (PeekMessage/Translate/Dispatch, 0 on WM_QUIT) && ok);
|
||||
```
|
||||
The strategy game is created from the lobby via `DemoApp::CreateStrategyGame` 0x00898b00 → `StrategyApp::CreateGame`
|
||||
0x00888e80 (GameOptions/AIProcessMinTime, DefaultAutoRefuel, starcolors.txt, `TurnCommands_v5`) → `StrategyServer`
|
||||
ctor 0x007d78d0 (+0x170 OnEventCallback from a ctor argument), `StrategyServer::InitGame` 0x007c8d90 (SEInitGame,
|
||||
SEAddDesign, SynchronizePlayer), `StrategyServer::LoadGame` 0x007dd530 ("loaded from file"), star-map UI
|
||||
(FUN_00778f40 → … → `StarMapPanelBase` ctor 0x00741cd0 ← THREAD 3).
|
||||
|
||||
### 2.2 Event plumbing (how `SE*` reach the client)
|
||||
`Game::SE*` are `Game::StrategyEvent` subclasses with a single virtual (dtor) — there is **no dynamic_cast dispatch**;
|
||||
the server hands each event to a callback with an **integer event type**: `StrategyServer+0x170 OnEventCallback
|
||||
(playerNetId, int type, StrategyEvent** ev)`; `StrategyServer::BroadcastEvent` 0x00789710 loops over `Players`.
|
||||
Missing callback → "StrategyServer: OnEvent() called, but no callback function specified." (31 sites). Types seen:
|
||||
`0x21 SETurnEndPending` (client side, `StrategyClient::RaiseEvent` 0x00783ee0), `0x24 SEProcessTurn`, `0x26
|
||||
SEResumePlaying`, `0x27 SELastPlaying`. The client counterpart is the `StrategyClient` (vftable 0x00a2298c).
|
||||
|
||||
`SNM*` are network messages registered by static initialisers (`.text` 0x009be0cc–0x009c138c, code Ghidra had not
|
||||
disassembled) via `Mars::NetMessageRegistry::Register` 0x008d2290 → `g_NetMsgRegistryById[id]` 0x00b2ddf8. SNM entry
|
||||
address = 0x00b2bc94 + 12·id (`{name, id, factory}`), so a handler comparing `msg->GetType()->id` against
|
||||
`DAT_00b2xxxx` is comparing against `entry+4`. Ids that matter for the turn:
|
||||
|
||||
| id | message | id | message |
|
||||
|---|---|---|---|
|
||||
| 0x29 | SNMUpdate (host → all: every player's TurnCommands) | 0x3f | SNMEndTurn (client → host) |
|
||||
| 0x2a/0x2b | SNMHostCombat / Reply | 0x40 | SNMUpdateComplete |
|
||||
| 0x2c/0x2d/0x2e | SNMLaunchCombat / JoinCombat / JoinCombatReply | 0x43/0x44 | SNMResumePlaying / …Received |
|
||||
| 0x2f | SNMAllCombatDone | 0x3d / 0x3e | SNMRunAI / SNMKillAI |
|
||||
| 0x30 / 0x31 | SNMEncounterQueryResults / SNMEncounterResults | 0x53 | SNMTurnInfo |
|
||||
| 0x32 | SNMSetPlayerStatus | 0x5b | SNMQueryEndTurnDone |
|
||||
| 0x3c | SNMDoEncounterQuery | 0x52 | SNMSetTimers |
|
||||
|
||||
(full table of 82 ids: `NetMsgReg_*` labels in Ghidra; ids 0x00–0x0f are the Mars transport `TNM*`/`FNM*`/`NM*`.)
|
||||
|
||||
### 2.3 The turn, end to end (addresses = renamed functions)
|
||||
```
|
||||
[strategic round] UI End Turn (FUN_005e4f80 / FUN_00579310)
|
||||
→ Game::StrategyClient::EndTurn 0x00783be0 (marks +0x57/+0x132, QPC timestamp, EndTurnDelay 0x0076ab90)
|
||||
→ RaiseEvent(0x21, SETurnEndPending) 0x00783ee0 [SETurnEndPending: "Done", cancellable → CancelEndTurn 0x007856f0 → SETurnEndCancelled]
|
||||
→ SendEndTurn 0x00783980 → SNMEndTurn (factory 0x0088c7b0; TurnCommands @+4, AIEncounterFlags @+0x1b8)
|
||||
StrategyClient::Update 0x007842b0: strategy timer → SETurnTimeExpired / EndTurnForced 0x00783d30 / SNMQueryEndTurnDone
|
||||
|
||||
[host] Game::StrategyNetworkClient::OnMessage 0x00784640 case 0x3f SNMEndTurn
|
||||
→ StrategyServer::OnPlayerEndTurn 0x007d9af0 (if exactly one human still not done → SELastPlaying 0x27 to him)
|
||||
→ StrategyServer::StorePlayerTurnCommands 0x007893c0
|
||||
host StrategyNetworkServer::Update 0x007cda40 → when all done broadcasts SNMUpdate (0x29) carrying ALL TurnCommands
|
||||
|
||||
[every machine — lock-step sim] OnMessage case 0x29 SNMUpdate ("Processing turn update, but the local player is not done" guard)
|
||||
→ StrategyServer::BeginProcessTurn 0x007d98e0 Frame++, bProcessingTurn=1, "Begin processing turn %d", reset fleet transients, BroadcastEvent(0x24 SEProcessTurn)
|
||||
→ StrategyServer::ApplyTurnCommands 0x007b18b0 "set for turn processing": fleet orders, builds, research, alliances (EVENT_ALLIANCE_*)
|
||||
→ FUN_007ad0f0, FUN_0078f6a0, FUN_0081ff40 (small, unnamed)
|
||||
→ StrategyServer::ProcessTurn 0x007dc6c0 (dt = 1.0f) ← §2.4
|
||||
→ FUN_008cfe20(server->GameID), FUN_007cf540; if local clients: StrategyApp::SyncLocalClients(1) 0x00815fd0 → StrategyServer::SynchronizePlayer 0x007c6220 per local client
|
||||
→ client state (+0x58) = 5; SNMUpdateComplete (0x40) to host
|
||||
|
||||
[querying / combat round — host] StrategyNetworkServer::RunCombatRound 0x007cbe80
|
||||
→ per encounter: SendEncounterQuery 0x007bfe60 → SNMDoEncounterQuery (0x3c) → clients: state 4 (C1 query UI, FUN_00847320)
|
||||
→ SNMEncounterQueryResults (0x30) → "Notifying %s to host encounter %d" SNMHostCombat (0x2a) / Reply (0x2b) / SNMLaunchCombat (0x2c) / SNMJoinCombat (0x2d)
|
||||
(client side: StrategyClient::Update logs "All(%d) clients connected to combat server", "Launching combat for encounter %d"; combat itself = DemoApp+0x158 object driven from DemoApp::OnTick)
|
||||
→ SNMEncounterResults (0x31) … → all resolved: SNMAllCombatDone (0x2f) "All combat complete, waiting for clients to process results"
|
||||
|
||||
[every machine] OnMessage case 0x2f SNMAllCombatDone
|
||||
→ FUN_00789330, FUN_007c0600 (1089 B, post-combat bookkeeping), StrategyServer::ApplyEncounterResults 0x007d4400 (EVENT_STATION_ENABLED/DISABLED …)
|
||||
→ DispatchTurnResults 0x007cd2a0 → SendTurnResultsToPlayers 0x007c5850 → per player SETurnResults::Create 0x007a7ae0 + fill 0x007c24d0 + dispatch 0x0079ac10 [SETurnResults]
|
||||
→ FUN_00761310, StrategyApp::SyncLocalClients(0)
|
||||
→ StrategyServer::GenerateTurnEvents 0x007dc640 → BuildTurnEvents 0x007db780 (EventStorage::TurnEvents per player, SynchronizePlayer) [SETurnEvents]
|
||||
→ autosave "(Autosave EndTurn)" (FUN_00895210 / "Auto saving of game failed."), FUN_0076c610, FUN_00769340, FUN_00781cf0
|
||||
→ state = 6, "StrategyNetworkClient: All combat complete msg received. New turn begins..."
|
||||
|
||||
[resume] host StrategyNetworkServer::SendResumePlaying 0x00794770 → SNMResumePlaying (0x43)
|
||||
→ OnMessage case 0x43 → StrategyServer::ResumePlaying 0x007ddc90: every player with Status(+0x164)==0 gets SEResumePlaying (0x26); reply SNMResumePlayingReceived (0x44); state = 1 (playing)
|
||||
→ (SELastPlaying / SEResumePlaying / SETurnEvents drive the UI_LAST_PLAYING "new turn" sound and the S16 events list)
|
||||
|
||||
[AI] SNMRunAI (0x3d) → StrategyApp::RunAI 0x008706f0 ("RunAI: No StrategyServer created") → RaiseAIPrepareTurn 0x00815f20 → SEAIPrepareTurn; AI players' TurnCommands enter the same SNMEndTurn/SNMUpdate path.
|
||||
```
|
||||
Client state machine (`StrategyNetworkClient+0x58`): 1 playing → (End Turn) → 5 turn processed / waiting for combat →
|
||||
4 querying (per encounter) → 6 all-combat-done → 1.
|
||||
|
||||
### 2.4 `Game::StrategyServer::ProcessTurn` 0x007dc6c0 — phases (decompiled with the new types, `ghidra-verify/verify_ProcessTurn.c`)
|
||||
1. **Per-system pre-pass** over `Systems`: colony lookup (FUN_007437e0), morale-event push (FUN_00752a10), string-table text (FUN_00743530/FUN_008c97f0), `SESystemAbandoned` via FUN_007b9df0.
|
||||
2. FUN_0086b300 + FUN_007adc80 (alliance/diplomacy upkeep; no strings).
|
||||
3. Per-player pre-pass over `Players` (small).
|
||||
4. **Movement**: fleet snapshot (FUN_00794ad0 / FUN_007b9b90), `ProcessNodeSpaceTravel` 0x007a0e20 (EVENT_FLEET_MULTIPOINT_NONODE, EVENT_LOSTINNODESPACE_NOBORE/ENGINES), **`ProcessFleetMovement` 0x007da9a0 → `MoveFleet` 0x007d9ee0 per fleet ("Destination of fleet doesn't exist. Stopping fleet.", cancels ship actions) → `SEFleetArrived`; `OnFleetArrived` 0x007ccb10 (EVENT_FLEET_ARRIVED)**.
|
||||
5. Per-fleet → per-ship FUN_00814ea0 (ship upkeep).
|
||||
6. **Per-system `ServerSystem::ProcessTurn` 0x007598e0**: infra decay if unowned, independence, pop/morale growth, `ProcessPlague` 0x00756a90 (EVENT_PLAGUE_OUTBREAK/CURED, COLONY_DESTROYEDBYPLAGUE), **`ProcessBuildQueue` 0x00752500 → `BuildQueue::ProcessTurn` 0x00890d50 → `SEBuildCompleted`** (with ShipBuildOrderDef), resources/terraforming (FUN_0074b230/FUN_00754220), `ProcessSlaves` 0x007537b0 (EVENT_SLAVES_DEAD), `ProcessRebellion` 0x007583b0 (EVENT_SYSTEM_REBELLION_CONTINUES).
|
||||
7. FUN_0078a7c0.
|
||||
8. **Per-player `ServerPlayer::ProcessTurn` 0x00891340 (dt)**: income/savings (FUN_00840fe0), **research**: if `ResT` set → `RollResearchAccident` 0x00889dc0 ("ACCIDENT!!"/"All okay.", EVENT_LABACCIDENT_*) then `TechTree::ProcessResearch` 0x005876c0 (EVENT_RESEARCH_OVERBUDGET, EVENT_TECHS_UNLOCKED), else EVENT_NO_RESEARCH; special projects FUN_00863cf0.
|
||||
9. `ProcessMissions` 0x007999a0, `ProcessStations` 0x007ae480 (EVENT_STATIONS_SCUTTLED), `ProcessDefenceSats` 0x007af0b0 (EVENT_DEFSATS_SCUTTLED).
|
||||
10. Per-ship flag pass: flag 4 → FUN_0080caf0(1); flag 0x400000 → FUN_00815230.
|
||||
11. **Encounter detection**: second snapshot FUN_00794ad0 → `local_98` = pending encounters. **If none**: end-of-turn tail runs now — `ProcessAid` 0x007ad100 (EVENT_GIVE_SAVINGS/RESEARCH), `ProcessSpecialProjects` 0x007a3310 (EVENT_SPRJTECHOFFER_STARTED), `ProcessSurrenders` 0x007d0d10 (EVENT_PLAYER_SURRENDERED_, EVENT_SYSTEM_SURRENDERED), per-player FUN_00818530(0), script hooks `SvSctOb->vft+0x10(6)` / `(0x1c)`, FUN_0086a8d0, FUN_0078ab30, FUN_00799380, FUN_0078aa70, per-system FUN_00743ec0, FUN_007b4c00, FUN_007d7f70 (2.4 KB, per-team 0x74-byte records), and every player with `+0xf9==0 || +0xfa!=0` gets `Status(+0x164) = 1` (back to playing). **If encounters exist** the tail is deferred to the `SNMAllCombatDone` handler (§2.3), i.e. the turn's results/events are only produced after the combat round.
|
||||
|
||||
---
|
||||
|
||||
## 3. Threads (the three `CreateThread` sites)
|
||||
|
||||
| # | site | creator | thread proc | what it does |
|
||||
|---|---|---|---|---|
|
||||
| 1 | 0x0090242d | `Mars::NetworkManager::Create` 0x00902350 ← `Mars::Network::Startup` 0x00902470 ← `DemoApp::OnStartup` | `Mars::NetworkManager::WatchdogThreadProc` 0x00901f40 | `Sleep(10)` loop under `g_netCS` 0x00b2e5f8: pings host link (+0x20) and listener (+0x98), detects timeouts — "Network(%f): No response from host %s." / "...from client %s." — and drops the link. Pure liveness watchdog; never touches the sim. Globals `g_pNetworkManager` 0x00b2e61c, `g_hNetworkWatchdogThread` 0x00b2e620. |
|
||||
| 2 | 0x008a14ef | `Mars::Application::Initialize` 0x008a0e50 ("Initializing Streaming sound update thread...") | `Mars::SoundSystem::StreamingUpdateThreadProc` 0x008ef1d0, **SetThreadPriority 15 = TIME_CRITICAL** | `WaitForMultipleObjects({app+0x84 wake, app+0x88 quit})`; under `g_musicCS` 0x00b2e4a8: if the current music stream (`g_pCurrentMusicStream` 0x00b2e4c4) has ended/faded (float +0x388 ≤ 0) → stop it and open the next queued track (`MusicPlayer::OpenMusicFile` 0x008ef040, "[%s] cannot open music file for playback"), else `StreamingSound::FillBuffer` 0x0091b5a0 (DirectSound GetCurrentPosition/Lock/decode; "Couldn't restore buffer"); then FUN_008b65d0 (sound-system update). Music/DirectSound streaming only — but it is the **only elevated-priority thread** in the process. |
|
||||
| 3 | 0x00736e84 | `Game::BackgroundWorker::Start` 0x00736e30, first statement of `Game::StarMapPanelBase` ctor 0x00741cd0 (Render/StarMapBlobs_Solid.fx, StarMapBlobs_Glow.fx, POLMAP_* colours, Skysphere) ← `PoliticalMapPanel` ctor 0x007424b0 ← strategy-map screen ctor FUN_005e9780 ← … ← `StrategyApp::CreateGame` | `Game::BackgroundWorker::ThreadProc` 0x00735bb0 | `Sleep(10)` poll loop; struct `{CRITICAL_SECTION@0; HANDLE thread@0x18; job*@0x54; flags@0x58..0x5c (0x5b quit, 0x5c exited)}`; when a job is posted runs `StarMapBlobs::BuildBlobMesh_Job` 0x00732ab0 = political-map territory "blob" mesh (point cloud → centroid/spread → implicit-surface polygoniser FUN_008fc160 with callbacks FUN_00722010/FUN_0071ea60 → mesh FUN_008fa5b0), then sets done. **Strategic-map overlay only — not the battle loader** (prior recon's "prime suspect" is cleared). |
|
||||
|
||||
Consequences for the battle-load / many-core lead: the combat load and the whole sim run on the **main thread** (DemoApp::OnTick
|
||||
state machine → combat object at `DemoApp+0x158`, `OnUpdate` → `+0x158->vft+4`). The only application threads are two 10 ms
|
||||
pollers and a TIME_CRITICAL audio streamer; everything else multi-threaded is inside D3D9/DirectSound/driver. If the
|
||||
slowdown is contention, the candidates inside the game's own code are the `g_musicCS` critical section (held by the
|
||||
TIME_CRITICAL thread while decoding) and the `Sleep(10)` pollers — not a sim worker.
|
||||
|
||||
---
|
||||
|
||||
## 4. Caveats / open items
|
||||
- Names are inferred from strings/callers (no symbols). `StrategyNetworkClient::OnMessage` (0x00784640) is the combined
|
||||
host+client message handler (host branches test `+0x54 = StrategyServer*`); the "StrategyNetworkServer" functions live
|
||||
on the host object driven from `DemoApp::OnUpdate`.
|
||||
- `StrategyServer` struct is partial (41 fields); `ServerPlayer::Read/Write` show only the +0x3a0.. tail by name.
|
||||
- Not named: FUN_007ad0f0/FUN_0078f6a0/FUN_0081ff40/FUN_007cf540 (between ApplyTurnCommands and ProcessTurn),
|
||||
FUN_00789330/FUN_007c0600 (AllCombatDone pre-pass), FUN_0086b300/FUN_007adc80/FUN_0078a7c0/FUN_007d7f70 (turn phases
|
||||
without strings), the client-side event sink behind `StrategyClient::RaiseEvent`, and the `Mars::Stream` vftable.
|
||||
- The static-initialiser region 0x009be000–0x009c1400 is still undisassembled code in the project (only labels were
|
||||
added); running "Disassemble" there would expose all 82 registrations as functions.
|
||||
- reva-server was stopped for the headless runs and restarted at the end. No git commit was made.
|
||||
|
|
@ -1 +0,0 @@
|
|||
# ghidra/ — placeholder, populate as work lands
|
||||
37
ghidra/scripts/FixPlayer.java
Normal file
37
ghidra/scripts/FixPlayer.java
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
import ghidra.app.script.GhidraScript;
|
||||
import ghidra.program.model.address.*;
|
||||
import ghidra.program.model.listing.*;
|
||||
import ghidra.program.model.symbol.*;
|
||||
import ghidra.program.model.data.*;
|
||||
import ghidra.program.model.lang.*;
|
||||
import ghidra.app.decompiler.*;
|
||||
import java.util.*;
|
||||
import java.io.*;
|
||||
|
||||
// ServerPlayer::Read/Write had no parameters: create this (ECX) = ServerPlayer_ser928*, stream (stack+4); verify decompile.
|
||||
public class FixPlayer extends GhidraScript {
|
||||
@Override
|
||||
public void run() throws Exception {
|
||||
DataTypeManager dtm = currentProgram.getDataTypeManager();
|
||||
List<DataType> l = new ArrayList<DataType>(); dtm.findDataTypes("Game::ServerPlayer_ser928", l);
|
||||
DataType view = l.get(0);
|
||||
List<DataType> l2 = new ArrayList<DataType>(); dtm.findDataTypes("Stream", l2); DataType stream = null; for (DataType d : l2) if (d.getCategoryPath().getPath().equals("/Mars")) stream = d;
|
||||
Register ecx = currentProgram.getRegister("ECX");
|
||||
PrintWriter log = new PrintWriter(new FileWriter("/tmp/spine/fixplayer.log"));
|
||||
for (long a : new long[]{0x008563e0L, 0x008804d0L}) {
|
||||
Function f = getFunctionAt(toAddr(a));
|
||||
List<Variable> ps = new ArrayList<Variable>();
|
||||
ps.add(new ParameterImpl("this", new PointerDataType(view), ecx, currentProgram));
|
||||
ps.add(new ParameterImpl("stream", new PointerDataType(stream == null ? VoidDataType.dataType : stream), 4, currentProgram));
|
||||
f.setCustomVariableStorage(true);
|
||||
f.replaceParameters(ps, Function.FunctionUpdateType.CUSTOM_STORAGE, true, SourceType.USER_DEFINED);
|
||||
log.println(f.getName(true) + " params=" + f.getParameterCount() + " " + f.getPrototypeString(false, false));
|
||||
}
|
||||
DecompInterface decomp = new DecompInterface(); decomp.openProgram(currentProgram);
|
||||
Function f = getFunctionAt(toAddr(0x008563e0L));
|
||||
DecompileResults res = decomp.decompileFunction(f, 240, monitor);
|
||||
PrintWriter w = new PrintWriter(new FileWriter("/tmp/spine/verify_ServerPlayer_Write.c"));
|
||||
w.println(res.getDecompiledFunction().getC()); w.close();
|
||||
decomp.dispose(); log.close(); println("fixplayer done");
|
||||
}
|
||||
}
|
||||
95
ghidra/scripts/FixThis.java
Normal file
95
ghidra/scripts/FixThis.java
Normal file
|
|
@ -0,0 +1,95 @@
|
|||
import ghidra.app.script.GhidraScript;
|
||||
import ghidra.program.model.address.*;
|
||||
import ghidra.program.model.mem.*;
|
||||
import ghidra.program.model.listing.*;
|
||||
import ghidra.program.model.symbol.*;
|
||||
import ghidra.program.model.data.*;
|
||||
import ghidra.app.decompiler.*;
|
||||
import java.util.*;
|
||||
import java.util.regex.*;
|
||||
import java.io.*;
|
||||
|
||||
// Fix-up: make the class structs Ghidra uses for __thiscall 'this' (VariableUtilities.findOrCreateClassStruct) BE the recovered
|
||||
// structs; give the COL!=0 serializers custom storage with the shifted view; re-run verification decompiles.
|
||||
public class FixThis extends GhidraScript {
|
||||
DataTypeManager dtm; SymbolTable st; PrintWriter log; DecompInterface decomp; Memory mem;
|
||||
static final String[][] CLASSES = { // {namespace path, /SOTS type name}
|
||||
{"Game::ServerSystem","Game::ServerSystem"},{"Game::ServerPlayer","Game::ServerPlayer"},{"Game::StarFleet","Game::StarFleet"},{"Game::StarShip","Game::StarShip"},{"Game::StrategyServer","Game::StrategyServer"},
|
||||
{"Game::Population","Game::Population"},{"Game::PopulationGroup","Game::PopulationGroup"},{"Game::Morale","Game::Morale"},{"Game::MoraleEvent","Game::MoraleEvent"},{"Game::IndependenceInfo","Game::IndependenceInfo"},
|
||||
{"Game::StarSystem::OutputRates","Game::StarSystem::OutputRates"},{"Game::StarSystem::PlayerView","Game::StarSystem::PlayerView"},{"Game::ShipBuildOrder","Game::ShipBuildOrder"},{"Game::DiplomacyStats","Game::DiplomacyStats"},
|
||||
{"Game::PlayerReport","Game::PlayerReport"},{"Game::PlayerAlliances","Game::PlayerAlliances"},{"Game::NodeRoute","Game::NodeRoute"},{"Game::FlightPlan::Waypoint","Game::Waypoint"},{"Game::FlightPlan","Game::FlightPlan"},
|
||||
{"Game::ShipHealth","Game::ShipHealth"},{"Game::PrisonerHold","Game::PrisonerHold"},{"Game::EventStorage","Game::EventStorage"},{"Game::CivilianRatios","Game::CivilianRatios"},{"Game::ShipRecords","Game::ShipRecords"},
|
||||
{"Game::SpyReport","Game::SpyReport"},{"Game::PlayerColorID","Game::PlayerColorID"},{"Mars::Vector3","Mars::Vector3"},
|
||||
{"Game::BuildQueue","Game::BuildQueue"},{"Game::TechTree","Game::TechTree"},{"Mars::Stream","Mars::Stream"},{"Game::StrategyEvent","Game::StrategyEvent"}};
|
||||
// serializers whose this = object + COL offset : {addr, class type name, col}
|
||||
static final Object[][] SHIFTED = {{0x00749630L,"Game::ServerSystem",8},{0x0075d4b0L,"Game::ServerSystem",8},{0x00701070L,"Game::StarFleet",8},{0x00702470L,"Game::StarFleet",8},
|
||||
{0x008291f0L,"Game::StarShip",8},{0x00853fa0L,"Game::StarShip",8},{0x008563e0L,"Game::ServerPlayer",0x3a0},{0x008804d0L,"Game::ServerPlayer",0x3a0}};
|
||||
|
||||
DataType findSots(String name) { List<DataType> l = new ArrayList<DataType>(); dtm.findDataTypes(name, l); for (DataType d : l) if (d.getCategoryPath().getPath().equals("/SOTS")) return d; return l.isEmpty() ? null : l.get(0); }
|
||||
String cstr(Address a, int max) {
|
||||
try { byte[] b = new byte[max]; int got = mem.getBytes(a, b); int i = 0;
|
||||
while (i < got && b[i] != 0 && (b[i]&0xff) >= 0x20 && (b[i]&0xff) < 0x7f) i++;
|
||||
if (i >= 1 && i < got && b[i] == 0) return new String(b, 0, i, "ISO-8859-1"); } catch (Exception e) {}
|
||||
return null;
|
||||
}
|
||||
void verify(long addr, String outName) throws Exception {
|
||||
Function f = getFunctionAt(toAddr(addr));
|
||||
DecompileResults res = decomp.decompileFunction(f, 240, monitor);
|
||||
PrintWriter w = new PrintWriter(new FileWriter("/tmp/spine/" + outName));
|
||||
if (res == null || !res.decompileCompleted()) { w.println("[decompile failed]"); w.close(); return; }
|
||||
String c = res.getDecompiledFunction().getC();
|
||||
Matcher m = Pattern.compile("&?(DAT|PTR_s_|s_[A-Za-z0-9_]*|PTR_DAT|u_[A-Za-z0-9_]*)_([0-9a-f]{8})").matcher(c);
|
||||
StringBuffer sb = new StringBuffer();
|
||||
while (m.find()) { Address a = toAddr(Long.parseLong(m.group(2), 16)); String s = cstr(a, 64); String rep = m.group(0); if (s != null && s.length() <= 60) rep = "\"" + s + "\""; m.appendReplacement(sb, Matcher.quoteReplacement(rep)); }
|
||||
m.appendTail(sb); w.println(sb); w.close(); log.println(" verify " + outName + " this-> refs: " + (sb.toString().split("this->").length - 1));
|
||||
}
|
||||
|
||||
@Override
|
||||
public void run() throws Exception {
|
||||
dtm = currentProgram.getDataTypeManager(); st = currentProgram.getSymbolTable(); mem = currentProgram.getMemory();
|
||||
log = new PrintWriter(new FileWriter("/tmp/spine/fixthis.log"));
|
||||
Map<String, Structure> cls = new HashMap<String, Structure>();
|
||||
for (String[] c : CLASSES) {
|
||||
String path = c[0], tname = c[1];
|
||||
Namespace cur = currentProgram.getGlobalNamespace();
|
||||
for (String part : path.split("::")) { Namespace n = st.getNamespace(part, cur); if (n == null) n = st.createClass(cur, part, SourceType.USER_DEFINED); cur = n; }
|
||||
if (!(cur instanceof GhidraClass)) { cur = st.convertNamespaceToClass(cur); }
|
||||
Structure cs = VariableUtilities.findOrCreateClassStruct((GhidraClass) cur, dtm);
|
||||
DataType mine = findSots(tname);
|
||||
log.println("class " + path + " -> class struct " + cs.getPathName() + " (len " + cs.getLength() + ") mine=" + (mine == null ? "null" : mine.getPathName() + " len " + mine.getLength()));
|
||||
if (mine == null || mine == cs || mine.isEquivalent(cs) && mine.getPathName().equals(cs.getPathName())) { cls.put(tname, cs); continue; }
|
||||
if (mine instanceof Structure) {
|
||||
cs.replaceWith(mine); cs.setDescription(mine.getDescription());
|
||||
dtm.replaceDataType(mine, cs, false); // repoint every use of the /SOTS copy to the class struct and drop the copy
|
||||
log.println(" merged; class struct now len " + cs.getLength() + " with " + cs.getNumDefinedComponents() + " fields");
|
||||
}
|
||||
cls.put(tname, cs);
|
||||
}
|
||||
// shifted serializer views (rebuild from the class structs)
|
||||
Map<String, Structure> views = new HashMap<String, Structure>();
|
||||
for (Object[] s : SHIFTED) {
|
||||
String tname = (String) s[1]; int col = ((Number) s[2]).intValue(); String key = tname + col;
|
||||
Structure v = views.get(key);
|
||||
if (v == null) {
|
||||
Structure src = cls.get(tname);
|
||||
StructureDataType nv = new StructureDataType(new CategoryPath("/SOTS"), tname + "_ser" + col, src.getLength() - col, dtm);
|
||||
nv.setDescription("Serializer view of " + tname + ": this = object + 0x" + Integer.toHexString(col) + " (IStreamable sub-object); only for " + tname + "::Read/Write. Fields below the sub-object offset are NOT visible here.");
|
||||
for (DataTypeComponent c : src.getDefinedComponents()) { if (c.getOffset() < col) continue; try { nv.replaceAtOffset(c.getOffset() - col, c.getDataType(), c.getLength(), c.getFieldName(), c.getComment()); } catch (Exception e) {} }
|
||||
v = (Structure) dtm.addDataType(nv, DataTypeConflictHandler.REPLACE_HANDLER); views.put(key, v);
|
||||
}
|
||||
Function f = getFunctionAt(toAddr((Long) s[0]));
|
||||
try {
|
||||
f.setCustomVariableStorage(true);
|
||||
Parameter p = f.getParameter(0);
|
||||
p.setDataType(new PointerDataType(v), SourceType.USER_DEFINED);
|
||||
if (f.getParameterCount() > 1) f.getParameter(1).setName("stream", SourceType.USER_DEFINED);
|
||||
f.setComment((f.getComment() == null ? "" : f.getComment() + "\n") + "NOTE: this = object + 0x" + Integer.toHexString(col) + " (IStreamable sub-object); typed as " + v.getName() + " (custom storage).");
|
||||
log.println(" " + f.getName(true) + ": this -> " + v.getName() + "* (custom storage) param0=" + p.getName() + " storage=" + p.getVariableStorage());
|
||||
} catch (Exception e) { log.println(" !! " + f.getName(true) + ": " + e.getMessage()); }
|
||||
}
|
||||
decomp = new DecompInterface(); decomp.openProgram(currentProgram);
|
||||
verify(0x00749630L, "verify_ServerSystem_Write.c"); verify(0x0075d4b0L, "verify_ServerSystem_Read.c"); verify(0x0079fa70L, "verify_StrategyServer_Write.c"); verify(0x00701070L, "verify_StarFleet_Write.c");
|
||||
verify(0x008563e0L, "verify_ServerPlayer_Write.c"); verify(0x007d98e0L, "verify_BeginProcessTurn.c"); verify(0x007dc6c0L, "verify_ProcessTurn.c"); verify(0x007598e0L, "verify_ServerSystem_ProcessTurn.c"); verify(0x007d9af0L, "verify_OnPlayerEndTurn.c");
|
||||
decomp.dispose(); log.close(); println("fixthis done");
|
||||
}
|
||||
}
|
||||
190
ghidra/scripts/SpineRecon.java
Normal file
190
ghidra/scripts/SpineRecon.java
Normal file
|
|
@ -0,0 +1,190 @@
|
|||
import ghidra.app.script.GhidraScript;
|
||||
import ghidra.program.model.address.*;
|
||||
import ghidra.program.model.mem.*;
|
||||
import ghidra.program.model.listing.*;
|
||||
import ghidra.program.model.symbol.*;
|
||||
import ghidra.app.decompiler.*;
|
||||
import java.util.*;
|
||||
import java.util.regex.*;
|
||||
import java.io.*;
|
||||
|
||||
// Pass 1 (read-only): anchors for the turn/main-loop spine.
|
||||
// A. strings containing SE*/SNM*/AppStartup/etc. -> xrefs -> containing functions (+ their callers)
|
||||
// B. symbols (RTTI/vftables) containing those names
|
||||
// C. entry point + callees (WinMain hunt), CreateThread sites + thread procs
|
||||
// D. std::string layout check (FUN_008b9d70)
|
||||
public class SpineRecon extends GhidraScript {
|
||||
DecompInterface decomp; Memory mem; PrintWriter out; ReferenceManager rm; SymbolTable st;
|
||||
static final String[] NEEDLES = {
|
||||
"SETurnEndPending","SNMEndTurn","SEProcessTurn","SNMDoEncounterQuery","SNMAllCombatDone","SNMResumePlaying",
|
||||
"SEAIPrepareTurn","SNMRunAI","SEFleetArrived","SEBuildCompleted","SETurnResults","SETurnEvents","AppStartup",
|
||||
"SETurnTimeExpired","SEResumePlaying","SNMEncounterQueryResults","SNMHostCombat","SNMLaunchCombat","SNMEncounterResults",
|
||||
"SETurnEndCancelled","SNMQueryEndTurnDone","SELastPlaying","PROCESSING_TITLE","Initializing Game","LOBBYSTATUS_STRATEGY_ROUND",
|
||||
"SENewTurn","SEStartTurn","SEEndTurn","StrategyServer","TurnCommands","ProcessTurn","EndTurn","MainLoop","Tick","AppMain"
|
||||
};
|
||||
static final String[] DECOMP = {"00902350","00736e30","008a0e50","008b9d70","008b9d90"};
|
||||
|
||||
String cstr(Address a, int max) {
|
||||
try {
|
||||
byte[] b = new byte[max]; int got = mem.getBytes(a, b);
|
||||
int i = 0; while (i < got && b[i] != 0 && (b[i]&0xff) >= 0x20 && (b[i]&0xff) < 0x7f) i++;
|
||||
if (i >= 1 && i < got && b[i] == 0) return new String(b, 0, i, "ISO-8859-1");
|
||||
} catch (Exception e) {}
|
||||
return null;
|
||||
}
|
||||
String decompRes(Function f) {
|
||||
try {
|
||||
DecompileResults res = decomp.decompileFunction(f, 180, monitor);
|
||||
if (res == null || !res.decompileCompleted()) return "[decompile failed]";
|
||||
String c = res.getDecompiledFunction().getC();
|
||||
Matcher m = Pattern.compile("&?(DAT|PTR_s_|s_[A-Za-z0-9_]*|PTR_DAT|u_[A-Za-z0-9_]*)_([0-9a-f]{8})").matcher(c);
|
||||
StringBuffer sb = new StringBuffer();
|
||||
while (m.find()) {
|
||||
Address a = toAddr(Long.parseLong(m.group(2), 16));
|
||||
String s = cstr(a, 64); String rep = m.group(0);
|
||||
if (s != null && s.length() <= 60) rep = "\"" + s + "\"";
|
||||
m.appendReplacement(sb, Matcher.quoteReplacement(rep));
|
||||
}
|
||||
m.appendTail(sb); return sb.toString();
|
||||
} catch (Exception e) { return "[exception " + e.getMessage() + "]"; }
|
||||
}
|
||||
void dumpFunc(Function f, String tag) throws IOException {
|
||||
String n = String.format("%08x", f.getEntryPoint().getOffset());
|
||||
File fl = new File("/tmp/spine/" + n + ".c"); if (fl.exists()) return;
|
||||
PrintWriter w = new PrintWriter(new FileWriter(fl));
|
||||
w.println("// " + f.getName() + " @ " + f.getEntryPoint() + " size=" + f.getBody().getNumAddresses() + " tag=" + tag);
|
||||
StringBuilder cs = new StringBuilder();
|
||||
for (Function c : f.getCallingFunctions(monitor)) cs.append(c.getName() + "@" + c.getEntryPoint() + " ");
|
||||
w.println("// CALLERS: " + cs);
|
||||
StringBuilder ce = new StringBuilder();
|
||||
for (Function c : f.getCalledFunctions(monitor)) ce.append(c.getName() + " ");
|
||||
w.println("// CALLEES: " + ce);
|
||||
w.println(decompRes(f)); w.close();
|
||||
}
|
||||
String funcDesc(Function f) {
|
||||
StringBuilder cs = new StringBuilder();
|
||||
int n = 0;
|
||||
for (Function c : f.getCallingFunctions(monitor)) { if (n++ < 8) cs.append(c.getName() + " "); }
|
||||
return f.getName() + "@" + f.getEntryPoint() + " sz=" + f.getBody().getNumAddresses() + " callers(" + n + "):[" + cs.toString().trim() + "]";
|
||||
}
|
||||
|
||||
@Override
|
||||
public void run() throws Exception {
|
||||
mem = currentProgram.getMemory(); rm = currentProgram.getReferenceManager(); st = currentProgram.getSymbolTable();
|
||||
decomp = new DecompInterface(); decomp.openProgram(currentProgram);
|
||||
new File("/tmp/spine").mkdirs();
|
||||
out = new PrintWriter(new FileWriter("/tmp/spine/recon.txt"));
|
||||
Set<Function> toDump = new LinkedHashSet<Function>();
|
||||
|
||||
// A. strings
|
||||
out.println("==== A. STRING ANCHORS ====");
|
||||
for (MemoryBlock blk : mem.getBlocks()) {
|
||||
if (!blk.isInitialized() || blk.isExecute()) continue;
|
||||
if (!blk.getName().equals(".rdata") && !blk.getName().equals(".data")) continue;
|
||||
int len = (int) blk.getSize(); byte[] buf = new byte[len]; blk.getBytes(blk.getStart(), buf);
|
||||
int start = 0;
|
||||
for (int i = 0; i < len; i++) {
|
||||
if (buf[i] != 0) continue;
|
||||
int slen = i - start;
|
||||
if (slen >= 4 && slen <= 120) {
|
||||
boolean ascii = true;
|
||||
for (int k = start; k < i; k++) { int c = buf[k] & 0xff; if (c < 0x20 || c > 0x7e) { ascii = false; break; } }
|
||||
if (ascii) {
|
||||
String s = new String(buf, start, slen, "ISO-8859-1");
|
||||
for (String nd : NEEDLES) {
|
||||
if (!s.contains(nd)) continue;
|
||||
Address a = blk.getStart().add(start);
|
||||
StringBuilder sb = new StringBuilder();
|
||||
ReferenceIterator ri = rm.getReferencesTo(a); int n = 0;
|
||||
while (ri.hasNext()) {
|
||||
Reference r = ri.next(); n++;
|
||||
Function f = getFunctionContaining(r.getFromAddress());
|
||||
if (f == null) { sb.append("\n ?" + r.getFromAddress()); continue; }
|
||||
sb.append("\n " + r.getFromAddress() + " in " + funcDesc(f));
|
||||
if (nd.startsWith("SE") || nd.startsWith("SNM") || nd.equals("AppStartup") || nd.equals("PROCESSING_TITLE") || nd.equals("Initializing Game")) toDump.add(f);
|
||||
}
|
||||
out.println(a + " \"" + s + "\" [" + nd + "] refs=" + n + sb);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
start = i + 1;
|
||||
}
|
||||
}
|
||||
// B. symbols
|
||||
out.println("\n==== B. SYMBOLS ====");
|
||||
SymbolIterator si = st.getAllSymbols(true);
|
||||
while (si.hasNext()) {
|
||||
Symbol s = si.next(); String n = s.getName(true);
|
||||
boolean hit = false;
|
||||
for (String nd : NEEDLES) if (n.contains(nd)) { hit = true; break; }
|
||||
if (!hit) continue;
|
||||
if (n.contains("StreamableHelper") || n.contains("VectorHelper")) continue;
|
||||
out.println(s.getAddress() + " " + n + " [" + s.getSymbolType() + "]");
|
||||
if (n.endsWith("::vftable")) {
|
||||
Address a = s.getAddress();
|
||||
for (int i = 0; i < 12; i++) {
|
||||
try {
|
||||
long p = mem.getInt(a.add(i*4)) & 0xffffffffL; Address fa = toAddr(p);
|
||||
Function f = getFunctionAt(fa);
|
||||
if (f == null) { if (i > 0) break; out.println(" [" + i + "] " + fa + " (nofunc)"); continue; }
|
||||
out.println(" [" + i + "] " + funcDesc(f));
|
||||
if (f.getBody().getNumAddresses() > 40) toDump.add(f);
|
||||
} catch (Exception e) { break; }
|
||||
}
|
||||
}
|
||||
}
|
||||
// C. entry + CreateThread
|
||||
out.println("\n==== C. ENTRY / THREADS ====");
|
||||
for (Symbol s : st.getSymbols("entry")) {
|
||||
Function f = getFunctionAt(s.getAddress());
|
||||
out.println("entry sym @ " + s.getAddress() + " func=" + (f == null ? "null" : funcDesc(f)));
|
||||
if (f != null) {
|
||||
toDump.add(f);
|
||||
for (Function c : f.getCalledFunctions(monitor)) {
|
||||
out.println(" entry callee: " + funcDesc(c) + " params=" + c.getParameterCount());
|
||||
if (c.getBody().getNumAddresses() > 200 && !c.getName().startsWith("_")) toDump.add(c);
|
||||
}
|
||||
}
|
||||
}
|
||||
for (Symbol s : st.getSymbols("CreateThread")) {
|
||||
out.println("CreateThread sym @ " + s.getAddress() + " type=" + s.getSymbolType());
|
||||
ReferenceIterator ri = rm.getReferencesTo(s.getAddress());
|
||||
while (ri.hasNext()) {
|
||||
Reference r = ri.next(); Function f = getFunctionContaining(r.getFromAddress());
|
||||
out.println(" ref " + r.getFromAddress() + " " + r.getReferenceType() + " in " + (f == null ? "?" : funcDesc(f)));
|
||||
if (f != null) toDump.add(f);
|
||||
// thunk? follow refs to the thunk
|
||||
if (f != null && f.isThunk() || (f != null && f.getBody().getNumAddresses() < 12)) {
|
||||
ReferenceIterator ri2 = rm.getReferencesTo(f.getEntryPoint());
|
||||
while (ri2.hasNext()) { Reference r2 = ri2.next(); Function f2 = getFunctionContaining(r2.getFromAddress());
|
||||
out.println(" thunk-ref " + r2.getFromAddress() + " in " + (f2 == null ? "?" : funcDesc(f2))); if (f2 != null) toDump.add(f2); }
|
||||
}
|
||||
}
|
||||
}
|
||||
// thread procs: look at the CreateThread call sites listed in prior recon; find the pushed function pointer
|
||||
long[] sites = {0x0090242dL, 0x008a14efL, 0x00736e84L};
|
||||
for (long sa : sites) {
|
||||
Address a = toAddr(sa);
|
||||
out.println("site " + a + ":");
|
||||
Instruction ins = getInstructionAt(a);
|
||||
int back = 0;
|
||||
Instruction p = ins;
|
||||
while (p != null && back < 12) {
|
||||
p = p.getPrevious(); back++;
|
||||
if (p == null) break;
|
||||
StringBuilder refs = new StringBuilder();
|
||||
for (Reference r : p.getReferencesFrom()) {
|
||||
Function tf = getFunctionAt(r.getToAddress());
|
||||
if (tf != null) { refs.append(" -> FUNC " + funcDesc(tf)); toDump.add(tf); }
|
||||
}
|
||||
out.println(" " + p.getAddress() + " " + p + refs);
|
||||
}
|
||||
}
|
||||
for (String d : DECOMP) { Function f = getFunctionAt(toAddr(Long.parseLong(d, 16))); if (f != null) toDump.add(f); }
|
||||
// D. dump
|
||||
out.println("\n==== D. DUMPED ====");
|
||||
for (Function f : toDump) { out.println(f.getName() + " @ " + f.getEntryPoint()); dumpFunc(f, "spine"); }
|
||||
out.close(); decomp.dispose(); println("done");
|
||||
}
|
||||
}
|
||||
132
ghidra/scripts/SpineRecon2.java
Normal file
132
ghidra/scripts/SpineRecon2.java
Normal file
|
|
@ -0,0 +1,132 @@
|
|||
import ghidra.app.script.GhidraScript;
|
||||
import ghidra.program.model.address.*;
|
||||
import ghidra.program.model.mem.*;
|
||||
import ghidra.program.model.listing.*;
|
||||
import ghidra.program.model.symbol.*;
|
||||
import ghidra.app.decompiler.*;
|
||||
import java.util.*;
|
||||
import java.util.regex.*;
|
||||
import java.io.*;
|
||||
|
||||
// Pass 2 (read-only): event dispatch + main loop + WinMain chain + worker threads.
|
||||
public class SpineRecon2 extends GhidraScript {
|
||||
DecompInterface decomp; Memory mem; PrintWriter out; ReferenceManager rm; SymbolTable st;
|
||||
Set<Function> toDump = new LinkedHashSet<Function>();
|
||||
static final String[] CLASSES = {"SETurnEndPending","SETurnEndCancelled","SETurnTimeExpired","SEProcessTurn","SEAIPrepareTurn","SEFleetArrived",
|
||||
"SEBuildCompleted","SETurnResults","SETurnEvents","SEResumePlaying","SELastPlaying","SNMEndTurn","SNMQueryEndTurnDone","SNMDoEncounterQuery",
|
||||
"SNMEncounterQueryResults","SNMHostCombat","SNMLaunchCombat","SNMEncounterResults","SNMAllCombatDone","SNMResumePlaying","SNMResumePlayingReceived",
|
||||
"SNMRunAI","StrategyEvent","StrategyServer","TurnCommands","AppStartup"};
|
||||
static final long[] STRS = {0x00a23c08L,0x00a32698L,0x00a258a8L,0x00a261a8L,0x00a21978L};
|
||||
static final String[] APIS = {"PeekMessageA","GetMessageA","DispatchMessageA","TranslateMessage","WinMain","GetStartupInfoA","ShowWindow","QueryPerformanceCounter","timeGetTime"};
|
||||
static final long[] DECOMP = {0x00925501L,0x0089dd30L,0x00741cd0L,0x00732ab0L,0x00902470L,0x0091b5a0L,0x008ef040L,0x008a7230L,0x008cd820L,0x008fe730L,0x00761300L,0x00761380L,0x007861d0L,0x00786d50L,0x007825f0L,0x008559e0L,0x0079e500L};
|
||||
|
||||
String cstr(Address a, int max) {
|
||||
try { byte[] b = new byte[max]; int got = mem.getBytes(a, b); int i = 0;
|
||||
while (i < got && b[i] != 0 && (b[i]&0xff) >= 0x20 && (b[i]&0xff) < 0x7f) i++;
|
||||
if (i >= 1 && i < got && b[i] == 0) return new String(b, 0, i, "ISO-8859-1"); } catch (Exception e) {}
|
||||
return null;
|
||||
}
|
||||
String decompRes(Function f) {
|
||||
try { DecompileResults res = decomp.decompileFunction(f, 240, monitor);
|
||||
if (res == null || !res.decompileCompleted()) return "[decompile failed]";
|
||||
String c = res.getDecompiledFunction().getC();
|
||||
Matcher m = Pattern.compile("&?(DAT|PTR_s_|s_[A-Za-z0-9_]*|PTR_DAT|u_[A-Za-z0-9_]*)_([0-9a-f]{8})").matcher(c);
|
||||
StringBuffer sb = new StringBuffer();
|
||||
while (m.find()) { Address a = toAddr(Long.parseLong(m.group(2), 16)); String s = cstr(a, 64); String rep = m.group(0);
|
||||
if (s != null && s.length() <= 60) rep = "\"" + s + "\""; m.appendReplacement(sb, Matcher.quoteReplacement(rep)); }
|
||||
m.appendTail(sb); return sb.toString();
|
||||
} catch (Exception e) { return "[exception " + e.getMessage() + "]"; }
|
||||
}
|
||||
void dumpFunc(Function f) throws IOException {
|
||||
String n = String.format("%08x", f.getEntryPoint().getOffset());
|
||||
File fl = new File("/tmp/spine/" + n + ".c"); if (fl.exists()) return;
|
||||
PrintWriter w = new PrintWriter(new FileWriter(fl));
|
||||
w.println("// " + f.getName() + " @ " + f.getEntryPoint() + " size=" + f.getBody().getNumAddresses());
|
||||
StringBuilder cs = new StringBuilder(); for (Function c : f.getCallingFunctions(monitor)) cs.append(c.getName() + "@" + c.getEntryPoint() + " ");
|
||||
w.println("// CALLERS: " + cs);
|
||||
StringBuilder ce = new StringBuilder(); for (Function c : f.getCalledFunctions(monitor)) ce.append(c.getName() + " ");
|
||||
w.println("// CALLEES: " + ce);
|
||||
w.println(decompRes(f)); w.close();
|
||||
}
|
||||
String funcDesc(Function f) {
|
||||
StringBuilder cs = new StringBuilder(); int n = 0;
|
||||
for (Function c : f.getCallingFunctions(monitor)) { if (n++ < 6) cs.append(c.getName() + " "); }
|
||||
return f.getName() + "@" + f.getEntryPoint() + " sz=" + f.getBody().getNumAddresses() + " callers(" + n + "):[" + cs.toString().trim() + "]";
|
||||
}
|
||||
void refsTo(Address a, String label, boolean dump, int maxDump) {
|
||||
ReferenceIterator ri = rm.getReferencesTo(a); int n = 0; StringBuilder sb = new StringBuilder();
|
||||
Set<Function> fs = new LinkedHashSet<Function>();
|
||||
while (ri.hasNext()) { Reference r = ri.next(); n++; Function f = getFunctionContaining(r.getFromAddress());
|
||||
if (f == null) { sb.append("\n ?" + r.getFromAddress() + " " + r.getReferenceType()); continue; }
|
||||
sb.append("\n " + r.getFromAddress() + " " + r.getReferenceType() + " in " + funcDesc(f)); fs.add(f); }
|
||||
out.println(a + " " + label + " refs=" + n + sb);
|
||||
if (dump) { int k = 0; for (Function f : fs) { if (k++ >= maxDump) break; toDump.add(f); } }
|
||||
}
|
||||
void callersUp(Function f, int depth, String indent, Set<Function> seen) {
|
||||
if (depth == 0 || f == null || seen.contains(f)) return; seen.add(f);
|
||||
for (Function c : f.getCallingFunctions(monitor)) { out.println(indent + "<- " + funcDesc(c)); toDump.add(c); callersUp(c, depth - 1, indent + " ", seen); }
|
||||
}
|
||||
|
||||
@Override
|
||||
public void run() throws Exception {
|
||||
mem = currentProgram.getMemory(); rm = currentProgram.getReferenceManager(); st = currentProgram.getSymbolTable();
|
||||
decomp = new DecompInterface(); decomp.openProgram(currentProgram);
|
||||
new File("/tmp/spine").mkdirs();
|
||||
out = new PrintWriter(new FileWriter("/tmp/spine/recon2.txt"));
|
||||
|
||||
out.println("==== 1. STRING XREFS ====");
|
||||
for (long s : STRS) { Address a = toAddr(s); refsTo(a, "\"" + cstr(a, 80) + "\"", true, 6); }
|
||||
|
||||
out.println("\n==== 2. CLASS VFTABLE / TYPEDESC XREFS ====");
|
||||
SymbolIterator si = st.getAllSymbols(true);
|
||||
List<Symbol> syms = new ArrayList<Symbol>();
|
||||
while (si.hasNext()) { Symbol s = si.next(); String n = s.getName(true);
|
||||
if (n.contains("Helper")) continue;
|
||||
if (!(n.endsWith("::vftable") || n.endsWith("::RTTI_Type_Descriptor"))) continue;
|
||||
for (String c : CLASSES) if (n.equals("Game::" + c + "::vftable") || n.equals("Game::" + c + "::RTTI_Type_Descriptor") || n.equals("Mars::" + c + "::vftable") || n.equals("Mars::" + c + "::RTTI_Type_Descriptor")) { syms.add(s); break; } }
|
||||
for (Symbol s : syms) refsTo(s.getAddress(), s.getName(true), true, 4);
|
||||
|
||||
out.println("\n==== 3. WIN32 API XREFS ====");
|
||||
for (String api : APIS) for (Symbol s : st.getSymbols(api)) {
|
||||
if (s.getSymbolType() != SymbolType.FUNCTION && s.getSymbolType() != SymbolType.LABEL) continue;
|
||||
refsTo(s.getAddress(), api + "(" + s.getSymbolType() + ")", api.startsWith("PeekMessage") || api.startsWith("GetMessage") || api.startsWith("DispatchMessage") || api.equals("GetStartupInfoA"), 6);
|
||||
// follow thunks
|
||||
ReferenceIterator ri = rm.getReferencesTo(s.getAddress());
|
||||
while (ri.hasNext()) { Reference r = ri.next(); Function f = getFunctionContaining(r.getFromAddress());
|
||||
if (f != null && (f.isThunk() || f.getBody().getNumAddresses() <= 6)) refsTo(f.getEntryPoint(), " thunk " + f.getName(), api.startsWith("PeekMessage") || api.startsWith("DispatchMessage"), 6); }
|
||||
}
|
||||
|
||||
out.println("\n==== 4. CALLERS UP from Init (FUN_008a0e50) and FUN_00741cd0 / FUN_00902470 ====");
|
||||
for (long fa : new long[]{0x008a0e50L, 0x00741cd0L, 0x00902470L}) { Function f = getFunctionAt(toAddr(fa)); out.println("ROOT " + funcDesc(f)); callersUp(f, 5, " ", new HashSet<Function>()); }
|
||||
|
||||
out.println("\n==== 5. APP CLASSES (vftables with 'App' / 'Application' / 'Game' in name) ====");
|
||||
si = st.getAllSymbols(true);
|
||||
while (si.hasNext()) { Symbol s = si.next(); String n = s.getName(true);
|
||||
if (!n.endsWith("::vftable") || n.contains("Helper")) continue;
|
||||
if (!(n.contains("App") || n.contains("Application") || n.contains("StarsGame") || n.contains("GameApp") || n.contains("StrategyClient") || n.contains("StrategyGame") || n.contains("GameServer") || n.contains("IApplication"))) continue;
|
||||
Address a = s.getAddress(); StringBuilder sb = new StringBuilder(a + " " + n + " :");
|
||||
for (int i = 0; i < 24; i++) { try { long p = mem.getInt(a.add(i*4)) & 0xffffffffL; Function f = getFunctionAt(toAddr(p)); if (f == null) break; sb.append(" [" + i + "]" + f.getName() + "(" + f.getBody().getNumAddresses() + ")"); } catch (Exception e) { break; } }
|
||||
out.println(sb);
|
||||
}
|
||||
// DAT_00b2d540 (app singleton) writers
|
||||
out.println("\n==== 6. APP SINGLETON DAT_00b2d540 refs (writes) ====");
|
||||
{ ReferenceIterator ri = rm.getReferencesTo(toAddr(0x00b2d540L)); int n = 0;
|
||||
while (ri.hasNext()) { Reference r = ri.next(); if (!r.getReferenceType().isWrite()) continue; Function f = getFunctionContaining(r.getFromAddress()); out.println(" W " + r.getFromAddress() + " in " + (f == null ? "?" : funcDesc(f))); if (f != null) toDump.add(f); if (++n > 20) break; } }
|
||||
|
||||
out.println("\n==== 7. what is at 0x009bf127 (s_SNMEndTurn ref) ====");
|
||||
{ Address a = toAddr(0x009bf100L); MemoryBlock b = mem.getBlock(a); out.println("block=" + (b == null ? "none" : b.getName() + " exec=" + b.isExecute()));
|
||||
Function f = getFunctionContaining(a); out.println("func=" + (f == null ? "none" : f.getName()));
|
||||
if (getInstructionAt(a) == null) disassemble(a);
|
||||
Instruction ins = getInstructionAt(a); int k = 0;
|
||||
while (ins != null && k++ < 40) { StringBuilder refs = new StringBuilder(); for (Reference r : ins.getReferencesFrom()) { String s2 = cstr(r.getToAddress(), 40); Function tf = getFunctionAt(r.getToAddress()); refs.append(s2 != null ? " \"" + s2 + "\"" : (tf != null ? " ->" + tf.getName() : "")); }
|
||||
out.println(" " + ins.getAddress() + " " + ins + refs); ins = ins.getNext(); }
|
||||
// where do undefined-code refs come from: refs TO 0x009bf120 region
|
||||
refsTo(toAddr(0x009bf110L), "0x009bf110", false, 0);
|
||||
}
|
||||
|
||||
for (long d : DECOMP) { Function f = getFunctionAt(toAddr(d)); if (f != null) toDump.add(f); }
|
||||
out.println("\n==== 8. DUMPED ====");
|
||||
for (Function f : toDump) { out.println(f.getName() + " @ " + f.getEntryPoint() + " sz=" + f.getBody().getNumAddresses()); dumpFunc(f); }
|
||||
out.close(); decomp.dispose(); println("done");
|
||||
}
|
||||
}
|
||||
84
ghidra/scripts/SpineRecon3.java
Normal file
84
ghidra/scripts/SpineRecon3.java
Normal file
|
|
@ -0,0 +1,84 @@
|
|||
import ghidra.app.script.GhidraScript;
|
||||
import ghidra.program.model.address.*;
|
||||
import ghidra.program.model.mem.*;
|
||||
import ghidra.program.model.listing.*;
|
||||
import ghidra.program.model.symbol.*;
|
||||
import ghidra.app.decompiler.*;
|
||||
import java.util.*;
|
||||
import java.util.regex.*;
|
||||
import java.io.*;
|
||||
|
||||
// Pass 3 (read-only): main loop + turn-processing driver: decompile candidates, callers-up chains for SE* creators.
|
||||
public class SpineRecon3 extends GhidraScript {
|
||||
DecompInterface decomp; Memory mem; PrintWriter out; ReferenceManager rm; SymbolTable st;
|
||||
Set<Function> toDump = new LinkedHashSet<Function>();
|
||||
static final long[] DECOMP = {0x0089f5b0L,0x0089dfb0L,0x00898b00L,0x00789710L,0x007b18b0L,0x007ad0f0L,0x0078f6a0L,0x0081ff40L,0x007dc6c0L,0x007cf540L,
|
||||
0x007dc640L,0x007da9a0L,0x007c5850L,0x007c5b00L,0x00789500L,0x00752500L,0x00815fd0L,0x007dd2f0L,0x008d2290L,0x0088c7b0L,0x0089c950L,0x00898ae0L,0x00898af0L,
|
||||
0x00761310L,0x00789330L,0x007c0600L,0x007d4400L,0x0076c610L,0x00769340L,0x00781cf0L,0x0089a3b0L,0x0089a640L,0x0089a480L,0x00899790L,0x0089eec0L,0x0089f1b0L};
|
||||
static final long[] UPCHAIN = {0x007da9a0L,0x00752500L,0x00789500L,0x007c5850L,0x007c5b00L,0x007dc640L,0x007c8d90L,0x007b18b0L,0x007ad0f0L,0x0083dbf0L,0x0078fac0L,0x007cbe80L,0x007bfe60L,0x00794770L,0x00784640L,0x0089f5b0L,0x00898b00L};
|
||||
|
||||
String cstr(Address a, int max) {
|
||||
try { byte[] b = new byte[max]; int got = mem.getBytes(a, b); int i = 0;
|
||||
while (i < got && b[i] != 0 && (b[i]&0xff) >= 0x20 && (b[i]&0xff) < 0x7f) i++;
|
||||
if (i >= 1 && i < got && b[i] == 0) return new String(b, 0, i, "ISO-8859-1"); } catch (Exception e) {}
|
||||
return null;
|
||||
}
|
||||
String decompRes(Function f) {
|
||||
try { DecompileResults res = decomp.decompileFunction(f, 240, monitor);
|
||||
if (res == null || !res.decompileCompleted()) return "[decompile failed]";
|
||||
String c = res.getDecompiledFunction().getC();
|
||||
Matcher m = Pattern.compile("&?(DAT|PTR_s_|s_[A-Za-z0-9_]*|PTR_DAT|u_[A-Za-z0-9_]*)_([0-9a-f]{8})").matcher(c);
|
||||
StringBuffer sb = new StringBuffer();
|
||||
while (m.find()) { Address a = toAddr(Long.parseLong(m.group(2), 16)); String s = cstr(a, 100); String rep = m.group(0);
|
||||
if (s != null && s.length() <= 96) rep = "\"" + s + "\""; m.appendReplacement(sb, Matcher.quoteReplacement(rep)); }
|
||||
m.appendTail(sb); return sb.toString();
|
||||
} catch (Exception e) { return "[exception " + e.getMessage() + "]"; }
|
||||
}
|
||||
void dumpFunc(Function f) throws IOException {
|
||||
String n = String.format("%08x", f.getEntryPoint().getOffset());
|
||||
File fl = new File("/tmp/spine/" + n + ".c"); if (fl.exists()) return;
|
||||
PrintWriter w = new PrintWriter(new FileWriter(fl));
|
||||
w.println("// " + f.getName() + " @ " + f.getEntryPoint() + " size=" + f.getBody().getNumAddresses());
|
||||
StringBuilder cs = new StringBuilder(); for (Function c : f.getCallingFunctions(monitor)) cs.append(c.getName() + "@" + c.getEntryPoint() + " ");
|
||||
w.println("// CALLERS: " + cs);
|
||||
StringBuilder ce = new StringBuilder(); for (Function c : f.getCalledFunctions(monitor)) ce.append(c.getName() + " ");
|
||||
w.println("// CALLEES: " + ce);
|
||||
w.println(decompRes(f)); w.close();
|
||||
}
|
||||
String funcDesc(Function f) {
|
||||
StringBuilder cs = new StringBuilder(); int n = 0;
|
||||
for (Function c : f.getCallingFunctions(monitor)) { if (n++ < 6) cs.append(c.getName() + " "); }
|
||||
return f.getName() + "@" + f.getEntryPoint() + " sz=" + f.getBody().getNumAddresses() + " callers(" + n + "):[" + cs.toString().trim() + "]";
|
||||
}
|
||||
// strings referenced by a function, in address order
|
||||
String strsOf(Function f) {
|
||||
TreeMap<Address,String> refs = new TreeMap<Address,String>();
|
||||
AddressIterator ai = rm.getReferenceSourceIterator(f.getBody(), true);
|
||||
while (ai.hasNext()) { Address from = ai.next(); for (Reference r : rm.getReferencesFrom(from)) { Address to = r.getToAddress(); if (!mem.contains(to)) continue;
|
||||
String sv = cstr(to, 90); if (sv != null && sv.length() >= 4) refs.put(from, sv); } }
|
||||
StringBuilder sb = new StringBuilder(); for (String s : refs.values()) sb.append("\"" + s + "\" ");
|
||||
return sb.toString();
|
||||
}
|
||||
void callersUp(Function f, int depth, String indent, Set<Function> seen) {
|
||||
if (depth == 0 || f == null || seen.contains(f)) return; seen.add(f);
|
||||
for (Function c : f.getCallingFunctions(monitor)) { if (c.getName().startsWith("Unwind")) continue; out.println(indent + "<- " + funcDesc(c) + " strs: " + strsOf(c)); callersUp(c, depth - 1, indent + " ", seen); }
|
||||
}
|
||||
|
||||
@Override
|
||||
public void run() throws Exception {
|
||||
mem = currentProgram.getMemory(); rm = currentProgram.getReferenceManager(); st = currentProgram.getSymbolTable();
|
||||
decomp = new DecompInterface(); decomp.openProgram(currentProgram);
|
||||
new File("/tmp/spine").mkdirs();
|
||||
out = new PrintWriter(new FileWriter("/tmp/spine/recon3.txt"));
|
||||
out.println("==== 1. CALLER CHAINS (with strings) ====");
|
||||
for (long fa : UPCHAIN) { Function f = getFunctionAt(toAddr(fa)); if (f == null) continue; out.println("ROOT " + funcDesc(f) + " strs: " + strsOf(f)); callersUp(f, 4, " ", new HashSet<Function>()); }
|
||||
out.println("\n==== 2. CALLEES of key drivers with strings ====");
|
||||
for (long fa : new long[]{0x0089f5b0L,0x0089dfb0L,0x007b18b0L,0x007dc640L,0x007db780L,0x007c8d90L,0x0089d610L,0x00898b00L,0x00888e80L}) {
|
||||
Function f = getFunctionAt(toAddr(fa)); if (f == null) continue; out.println("DRIVER " + funcDesc(f));
|
||||
for (Function c : f.getCalledFunctions(monitor)) { if (c.getBody().getNumAddresses() < 60) continue; out.println(" -> " + funcDesc(c) + " strs: " + strsOf(c)); } }
|
||||
for (long d : DECOMP) { Function f = getFunctionAt(toAddr(d)); if (f != null) toDump.add(f); }
|
||||
out.println("\n==== 3. DUMPED ====");
|
||||
for (Function f : toDump) { out.println(f.getName() + " @ " + f.getEntryPoint() + " sz=" + f.getBody().getNumAddresses()); dumpFunc(f); }
|
||||
out.close(); decomp.dispose(); println("done");
|
||||
}
|
||||
}
|
||||
119
ghidra/scripts/SpineRecon4.java
Normal file
119
ghidra/scripts/SpineRecon4.java
Normal file
|
|
@ -0,0 +1,119 @@
|
|||
import ghidra.app.script.GhidraScript;
|
||||
import ghidra.program.model.address.*;
|
||||
import ghidra.program.model.mem.*;
|
||||
import ghidra.program.model.listing.*;
|
||||
import ghidra.program.model.symbol.*;
|
||||
import ghidra.program.model.scalar.*;
|
||||
import ghidra.app.decompiler.*;
|
||||
import java.util.*;
|
||||
import java.util.regex.*;
|
||||
import java.io.*;
|
||||
|
||||
// Pass 4 (read-only): network message registry table, ProcessTurn callee strings, +0x170 callback writers, misc decompiles.
|
||||
public class SpineRecon4 extends GhidraScript {
|
||||
DecompInterface decomp; Memory mem; PrintWriter out; ReferenceManager rm; SymbolTable st;
|
||||
Set<Function> toDump = new LinkedHashSet<Function>();
|
||||
static final long[] DECOMP = {0x00891340L,0x007598e0L,0x00898800L,0x0089a640L,0x00899210L,0x0090c700L,0x008e5ac0L,0x00783be0L,0x00783980L,0x007856f0L,0x007cd2a0L,0x007d4400L,0x00752500L,0x0078a7c0L,0x007ad100L,0x007d7f70L,0x00814ea0L,0x0086b300L,0x007adc80L,0x007a0e20L,0x007999a0L,0x007ae480L,0x007af0b0L,0x008986a0L,0x0089cc70L,0x00898690L,0x00723df0L};
|
||||
static final long[] CALLEE_STRS = {0x007dc6c0L,0x007598e0L,0x00891340L,0x007da9a0L,0x007d4400L,0x007cda40L,0x007842b0L,0x00741cd0L,0x00732ab0L};
|
||||
|
||||
String cstr(Address a, int max) {
|
||||
try { byte[] b = new byte[max]; int got = mem.getBytes(a, b); int i = 0;
|
||||
while (i < got && b[i] != 0 && (b[i]&0xff) >= 0x20 && (b[i]&0xff) < 0x7f) i++;
|
||||
if (i >= 1 && i < got && b[i] == 0) return new String(b, 0, i, "ISO-8859-1"); } catch (Exception e) {}
|
||||
return null;
|
||||
}
|
||||
String decompRes(Function f) {
|
||||
try { DecompileResults res = decomp.decompileFunction(f, 240, monitor);
|
||||
if (res == null || !res.decompileCompleted()) return "[decompile failed]";
|
||||
String c = res.getDecompiledFunction().getC();
|
||||
Matcher m = Pattern.compile("&?(DAT|PTR_s_|s_[A-Za-z0-9_]*|PTR_DAT|u_[A-Za-z0-9_]*)_([0-9a-f]{8})").matcher(c);
|
||||
StringBuffer sb = new StringBuffer();
|
||||
while (m.find()) { Address a = toAddr(Long.parseLong(m.group(2), 16)); String s = cstr(a, 100); String rep = m.group(0);
|
||||
if (s != null && s.length() <= 96) rep = "\"" + s + "\""; m.appendReplacement(sb, Matcher.quoteReplacement(rep)); }
|
||||
m.appendTail(sb); return sb.toString();
|
||||
} catch (Exception e) { return "[exception " + e.getMessage() + "]"; }
|
||||
}
|
||||
void dumpFunc(Function f) throws IOException {
|
||||
String n = String.format("%08x", f.getEntryPoint().getOffset());
|
||||
File fl = new File("/tmp/spine/" + n + ".c"); if (fl.exists()) return;
|
||||
PrintWriter w = new PrintWriter(new FileWriter(fl));
|
||||
w.println("// " + f.getName() + " @ " + f.getEntryPoint() + " size=" + f.getBody().getNumAddresses());
|
||||
StringBuilder cs = new StringBuilder(); for (Function c : f.getCallingFunctions(monitor)) cs.append(c.getName() + "@" + c.getEntryPoint() + " ");
|
||||
w.println("// CALLERS: " + cs);
|
||||
StringBuilder ce = new StringBuilder(); for (Function c : f.getCalledFunctions(monitor)) ce.append(c.getName() + " ");
|
||||
w.println("// CALLEES: " + ce);
|
||||
w.println(decompRes(f)); w.close();
|
||||
}
|
||||
String strsOf(Function f) {
|
||||
TreeMap<Address,String> refs = new TreeMap<Address,String>();
|
||||
AddressIterator ai = rm.getReferenceSourceIterator(f.getBody(), true);
|
||||
while (ai.hasNext()) { Address from = ai.next(); for (Reference r : rm.getReferencesFrom(from)) { Address to = r.getToAddress(); if (!mem.contains(to)) continue;
|
||||
String sv = cstr(to, 90); if (sv != null && sv.length() >= 4) refs.put(from, sv);
|
||||
Symbol s = st.getPrimarySymbol(to); if (s != null && s.getName(true).endsWith("::vftable") && !s.getName(true).contains("Helper")) refs.put(from, "<" + s.getName(true) + ">"); } }
|
||||
StringBuilder sb = new StringBuilder(); for (String s : refs.values()) sb.append(s.startsWith("<") ? s + " " : "\"" + s + "\" ");
|
||||
return sb.toString();
|
||||
}
|
||||
|
||||
@Override
|
||||
public void run() throws Exception {
|
||||
mem = currentProgram.getMemory(); rm = currentProgram.getReferenceManager(); st = currentProgram.getSymbolTable();
|
||||
decomp = new DecompInterface(); decomp.openProgram(currentProgram);
|
||||
new File("/tmp/spine").mkdirs();
|
||||
out = new PrintWriter(new FileWriter("/tmp/spine/recon4.txt"));
|
||||
|
||||
out.println("==== 1. NETWORK MESSAGE REGISTRY (static initializers calling FUN_008d2290) ====");
|
||||
// scan .text for MOV ECX,imm32 ; CALL 0x008d2290 and read the two preceding pushes
|
||||
Address lo = toAddr(0x009b0000L), hi = toAddr(0x009c8000L);
|
||||
Address a = lo; int found = 0;
|
||||
byte[] buf = new byte[(int)(hi.subtract(lo))]; mem.getBytes(lo, buf);
|
||||
TreeMap<Integer,String> byId = new TreeMap<Integer,String>();
|
||||
for (int i = 0; i + 10 < buf.length; i++) {
|
||||
if ((buf[i]&0xff) == 0xB9 && (buf[i+5]&0xff) == 0xE8) {
|
||||
long rel = (buf[i+6]&0xffL) | ((buf[i+7]&0xffL)<<8) | ((buf[i+8]&0xffL)<<16) | ((buf[i+9]&0xffL)<<24);
|
||||
long target = lo.getOffset() + i + 10 + (int)rel;
|
||||
if (target != 0x008d2290L) continue;
|
||||
long entry = (buf[i+1]&0xffL) | ((buf[i+2]&0xffL)<<8) | ((buf[i+3]&0xffL)<<16) | ((buf[i+4]&0xffL)<<24);
|
||||
// walk back: PUSH name (68 imm32), PUSH id (6a imm8 | 68 imm32), PUSH factory (68 imm32)
|
||||
int p = i - 5; long name = 0, id = -1, factory = 0;
|
||||
if ((buf[p]&0xff) == 0x68) { name = (buf[p+1]&0xffL) | ((buf[p+2]&0xffL)<<8) | ((buf[p+3]&0xffL)<<16) | ((buf[p+4]&0xffL)<<24); }
|
||||
else continue;
|
||||
int q;
|
||||
if ((buf[p-2]&0xff) == 0x6a) { id = buf[p-1]&0xff; q = p - 2; }
|
||||
else if ((buf[p-5]&0xff) == 0x68) { id = (buf[p-4]&0xffL) | ((buf[p-3]&0xffL)<<8) | ((buf[p-2]&0xffL)<<16) | ((buf[p-1]&0xffL)<<24); q = p - 5; }
|
||||
else continue;
|
||||
if ((buf[q-5]&0xff) == 0x68) factory = (buf[q-4]&0xffL) | ((buf[q-3]&0xffL)<<8) | ((buf[q-2]&0xffL)<<16) | ((buf[q-1]&0xffL)<<24);
|
||||
String nm = cstr(toAddr(name), 64);
|
||||
Function ff = getFunctionAt(toAddr(factory));
|
||||
String line = String.format("id=0x%02x entry=%08x name=%-32s factory=%08x %s init@%08x", id, entry, nm, factory, ff == null ? "(nofunc)" : ff.getName() + "(sz " + ff.getBody().getNumAddresses() + ")", lo.getOffset() + i);
|
||||
byId.put((int)id, line); found++;
|
||||
}
|
||||
}
|
||||
for (String l : byId.values()) out.println(l);
|
||||
out.println("total " + found);
|
||||
|
||||
out.println("\n==== 2. WRITERS of StrategyServer+0x170 (callback) : scan for 'mov [reg+0x170], ' in functions referencing StrategyServer ====");
|
||||
// brute: instructions with scalar 0x170 operand as memory displacement, being a store
|
||||
for (long fa : new long[]{0x007d78d0L, 0x00888e80L, 0x007d20d0L}) {
|
||||
Function f = getFunctionAt(toAddr(fa)); if (f == null) continue;
|
||||
InstructionIterator ii = currentProgram.getListing().getInstructions(f.getBody(), true);
|
||||
while (ii.hasNext()) { Instruction ins = ii.next(); String s = ins.toString(); if (s.contains("0x170]")) out.println(" " + f.getName() + " " + ins.getAddress() + " " + s); }
|
||||
}
|
||||
// also global search on refs to all functions passed as pointer near 0x170... skip. Check refs to the candidate callback functions: StrategyClient vft[5] FUN_00774fc0 and others in 0x0077xxxx
|
||||
for (long fa : new long[]{0x00774fc0L, 0x00774f30L, 0x00775100L, 0x00779a70L}) {
|
||||
Function f = getFunctionAt(toAddr(fa)); if (f == null) continue; StringBuilder sb = new StringBuilder();
|
||||
ReferenceIterator ri = rm.getReferencesTo(f.getEntryPoint()); while (ri.hasNext()) { Reference r = ri.next(); Function c = getFunctionContaining(r.getFromAddress()); sb.append(r.getFromAddress() + "(" + r.getReferenceType() + (c == null ? "" : " in " + c.getName()) + ") "); }
|
||||
out.println(" refs to " + f.getName() + ": " + sb);
|
||||
}
|
||||
|
||||
out.println("\n==== 3. CALLEE STRINGS ====");
|
||||
for (long fa : CALLEE_STRS) {
|
||||
Function f = getFunctionAt(toAddr(fa)); if (f == null) continue; out.println("DRIVER " + f.getName() + "@" + f.getEntryPoint() + " sz=" + f.getBody().getNumAddresses() + " strs: " + strsOf(f));
|
||||
for (Function c : f.getCalledFunctions(monitor)) { if (c.getBody().getNumAddresses() < 40) continue; String s = strsOf(c); StringBuilder cs = new StringBuilder(); int n = 0; for (Function cc : c.getCallingFunctions(monitor)) if (n++ < 4) cs.append(cc.getName() + " ");
|
||||
out.println(" -> " + c.getName() + "@" + c.getEntryPoint() + " sz=" + c.getBody().getNumAddresses() + " callers(" + n + "):[" + cs.toString().trim() + "] strs: " + s); }
|
||||
}
|
||||
for (long d : DECOMP) { Function f = getFunctionAt(toAddr(d)); if (f != null) toDump.add(f); }
|
||||
out.println("\n==== 4. DUMPED ====");
|
||||
for (Function f : toDump) { out.println(f.getName() + " @ " + f.getEntryPoint() + " sz=" + f.getBody().getNumAddresses()); dumpFunc(f); }
|
||||
out.close(); decomp.dispose(); println("done");
|
||||
}
|
||||
}
|
||||
363
ghidra/scripts/WriteBack.java
Normal file
363
ghidra/scripts/WriteBack.java
Normal file
|
|
@ -0,0 +1,363 @@
|
|||
import ghidra.app.script.GhidraScript;
|
||||
import ghidra.program.model.address.*;
|
||||
import ghidra.program.model.mem.*;
|
||||
import ghidra.program.model.listing.*;
|
||||
import ghidra.program.model.symbol.*;
|
||||
import ghidra.program.model.data.*;
|
||||
import ghidra.app.decompiler.*;
|
||||
import java.util.*;
|
||||
import java.util.regex.*;
|
||||
import java.io.*;
|
||||
|
||||
// Write-back: struct data types from findings/objects/struct-recovery.md, function renames (serializers, affinity,
|
||||
// turn/main-loop spine), plate comments, net-message registry labels; then verification decompiles.
|
||||
public class WriteBack extends GhidraScript {
|
||||
DataTypeManager dtm; SymbolTable st; PrintWriter log; DecompInterface decomp; Memory mem;
|
||||
Map<String, DataType> types = new HashMap<String, DataType>();
|
||||
static final String DOC = "Source: /srv/re-lab/notes/findings/objects/struct-recovery.md (save-field xref recovery, 2026-09-07)";
|
||||
static final String DOC2 = "Source: /srv/re-lab/handoff/ghidra-writeback-and-spine.md (turn/main-loop spine, 2026-09-07)";
|
||||
CategoryPath CAT = new CategoryPath("/SOTS");
|
||||
|
||||
// ---------- type helpers ----------
|
||||
DataType prim(String k) {
|
||||
if (k.equals("int")) return IntegerDataType.dataType;
|
||||
if (k.equals("uint")) return UnsignedIntegerDataType.dataType;
|
||||
if (k.equals("short")) return ShortDataType.dataType;
|
||||
if (k.equals("i64")) return LongLongDataType.dataType;
|
||||
if (k.equals("float")) return FloatDataType.dataType;
|
||||
if (k.equals("double")) return DoubleDataType.dataType;
|
||||
if (k.equals("bool")) return BooleanDataType.dataType;
|
||||
if (k.equals("i8")) return SignedByteDataType.dataType;
|
||||
if (k.equals("u8")) return ByteDataType.dataType;
|
||||
if (k.equals("ptr")) return new PointerDataType(VoidDataType.dataType);
|
||||
if (k.equals("pfn")) return new PointerDataType(VoidDataType.dataType);
|
||||
if (k.endsWith("*")) { DataType t = types.get(k.substring(0, k.length()-1)); return new PointerDataType(t == null ? VoidDataType.dataType : t); }
|
||||
Matcher m = Pattern.compile("^(.+)\\[(\\d+)\\]$").matcher(k);
|
||||
if (m.matches()) { DataType b = prim(m.group(1)); int n = Integer.parseInt(m.group(2)); return new ArrayDataType(b, n, b.getLength()); }
|
||||
DataType t = types.get(k); if (t != null) return t;
|
||||
throw new RuntimeException("unknown type " + k);
|
||||
}
|
||||
Structure mk(String name, int size) throws Exception {
|
||||
String nm = DataUtilities.isValidDataTypeName(name) ? name : name.replace("::", "_");
|
||||
StructureDataType s = new StructureDataType(CAT, nm, size, dtm);
|
||||
Structure r = (Structure) dtm.addDataType(s, DataTypeConflictHandler.REPLACE_HANDLER);
|
||||
types.put(name, r); log.println(" type " + r.getPathName() + " size=" + r.getLength());
|
||||
return r;
|
||||
}
|
||||
Structure opaque(String name) throws Exception { // forward-declared class, only used through pointers
|
||||
String nm = DataUtilities.isValidDataTypeName(name) ? name : name.replace("::", "_");
|
||||
StructureDataType s = new StructureDataType(CAT, nm, 0, dtm); s.setDescription("opaque (only referenced by pointer)");
|
||||
Structure r = (Structure) dtm.addDataType(s, DataTypeConflictHandler.REPLACE_HANDLER); types.put(name, r); return r;
|
||||
}
|
||||
void fields(Structure s, Object[][] rows) {
|
||||
for (Object[] r : rows) {
|
||||
int off = ((Number) r[0]).intValue(); DataType dt = prim((String) r[1]); String name = (String) r[2]; String cmt = r.length > 3 ? (String) r[3] : null;
|
||||
if (off + dt.getLength() > s.getLength()) { log.println(" !! " + s.getName() + "." + name + " @0x" + Integer.toHexString(off) + " exceeds size"); continue; }
|
||||
try { s.replaceAtOffset(off, dt, dt.getLength(), name, cmt); }
|
||||
catch (Exception e) { log.println(" !! " + s.getName() + "." + name + " @0x" + Integer.toHexString(off) + ": " + e.getMessage()); }
|
||||
}
|
||||
}
|
||||
Structure fill(String name, int size, String desc, Object[][] rows) throws Exception {
|
||||
Structure s = types.containsKey(name) ? (Structure) types.get(name) : mk(name, size);
|
||||
if (s.getLength() != size) { s = mk(name, size); }
|
||||
s.setDescription(desc + "\n" + DOC); fields(s, rows); return s;
|
||||
}
|
||||
// shifted copy for serializer 'this' (= object + colOff)
|
||||
Structure serView(String name, int colOff) throws Exception {
|
||||
Structure src = (Structure) types.get(name);
|
||||
Structure v = mk(name + "_ser" + colOff, src.getLength() - colOff);
|
||||
v.setDescription("Serializer view of " + name + ": this = object + 0x" + Integer.toHexString(colOff) + " (IStreamable sub-object). Use only to read " + name + "::Read/Write.");
|
||||
for (DataTypeComponent c : src.getDefinedComponents()) { if (c.getOffset() < colOff) continue; try { v.replaceAtOffset(c.getOffset() - colOff, c.getDataType(), c.getLength(), c.getFieldName(), c.getComment()); } catch (Exception e) {} }
|
||||
return v;
|
||||
}
|
||||
|
||||
// ---------- symbol helpers ----------
|
||||
Namespace ns(String path) throws Exception {
|
||||
Namespace cur = currentProgram.getGlobalNamespace();
|
||||
for (String part : path.split("::")) {
|
||||
Namespace n = st.getNamespace(part, cur);
|
||||
if (n == null) n = st.createClass(cur, part, SourceType.USER_DEFINED);
|
||||
cur = n;
|
||||
}
|
||||
return cur;
|
||||
}
|
||||
void fn(long addr, String qualified, String plate) {
|
||||
try {
|
||||
Function f = getFunctionAt(toAddr(addr));
|
||||
if (f == null) { log.println(" !! no function at " + Long.toHexString(addr)); return; }
|
||||
int i = qualified.lastIndexOf("::");
|
||||
String name = i < 0 ? qualified : qualified.substring(i + 2);
|
||||
Namespace n = i < 0 ? currentProgram.getGlobalNamespace() : ns(qualified.substring(0, i));
|
||||
f.setParentNamespace(n);
|
||||
f.setName(name, SourceType.USER_DEFINED);
|
||||
if (plate != null) f.setComment(plate);
|
||||
log.println(" fn " + toAddr(addr) + " -> " + qualified);
|
||||
} catch (Exception e) { log.println(" !! rename " + Long.toHexString(addr) + " " + qualified + ": " + e.getMessage()); }
|
||||
}
|
||||
void lbl(long addr, String name, String cmt) {
|
||||
try { Address a = toAddr(addr); createLabel(a, name, true, SourceType.USER_DEFINED); if (cmt != null) setEOLComment(a, cmt); log.println(" label " + a + " " + name); }
|
||||
catch (Exception e) { log.println(" !! label " + Long.toHexString(addr) + ": " + e.getMessage()); }
|
||||
}
|
||||
void retypeThis(long addr, DataType ptr) {
|
||||
try { Function f = getFunctionAt(toAddr(addr));
|
||||
if (f.getParameterCount() > 0) f.getParameter(0).setDataType(ptr, SourceType.USER_DEFINED);
|
||||
else { List<Variable> ps = new ArrayList<Variable>(); ps.add(new ParameterImpl("this", ptr, currentProgram)); ps.add(new ParameterImpl("stream", new PointerDataType(VoidDataType.dataType), currentProgram));
|
||||
f.replaceParameters(ps, Function.FunctionUpdateType.DYNAMIC_STORAGE_ALL_PARAMS, true, SourceType.USER_DEFINED); f.setCallingConvention("__thiscall"); }
|
||||
log.println(" retyped this of " + f.getName() + " -> " + ptr.getName());
|
||||
} catch (Exception e) { log.println(" !! retype " + Long.toHexString(addr) + ": " + e.getMessage()); }
|
||||
}
|
||||
String cstr(Address a, int max) {
|
||||
try { byte[] b = new byte[max]; int got = mem.getBytes(a, b); int i = 0;
|
||||
while (i < got && b[i] != 0 && (b[i]&0xff) >= 0x20 && (b[i]&0xff) < 0x7f) i++;
|
||||
if (i >= 1 && i < got && b[i] == 0) return new String(b, 0, i, "ISO-8859-1"); } catch (Exception e) {}
|
||||
return null;
|
||||
}
|
||||
void verify(long addr, String outName) throws Exception {
|
||||
Function f = getFunctionAt(toAddr(addr));
|
||||
DecompileResults res = decomp.decompileFunction(f, 240, monitor);
|
||||
PrintWriter w = new PrintWriter(new FileWriter("/tmp/spine/" + outName));
|
||||
if (res == null || !res.decompileCompleted()) { w.println("[decompile failed]"); w.close(); return; }
|
||||
String c = res.getDecompiledFunction().getC();
|
||||
Matcher m = Pattern.compile("&?(DAT|PTR_s_|s_[A-Za-z0-9_]*|PTR_DAT|u_[A-Za-z0-9_]*)_([0-9a-f]{8})").matcher(c);
|
||||
StringBuffer sb = new StringBuffer();
|
||||
while (m.find()) { Address a = toAddr(Long.parseLong(m.group(2), 16)); String s = cstr(a, 64); String rep = m.group(0); if (s != null && s.length() <= 60) rep = "\"" + s + "\""; m.appendReplacement(sb, Matcher.quoteReplacement(rep)); }
|
||||
m.appendTail(sb); w.println(sb); w.close(); log.println(" verify decompile written " + outName + " (" + sb.length() + " chars)");
|
||||
}
|
||||
|
||||
@Override
|
||||
public void run() throws Exception {
|
||||
dtm = currentProgram.getDataTypeManager(); st = currentProgram.getSymbolTable(); mem = currentProgram.getMemory();
|
||||
new File("/tmp/spine").mkdirs();
|
||||
log = new PrintWriter(new FileWriter("/tmp/spine/writeback.log"));
|
||||
log.println("==== TYPES ====");
|
||||
// --- std / Mars primitives ---
|
||||
fill("Mars::Vector3", 12, "3 floats", new Object[][]{{0,"float","x"},{4,"float","y"},{8,"float","z"}});
|
||||
Structure bx = mk("std::string::_Bxty", 16); bx.setDescription("SSO buffer / heap pointer union");
|
||||
fields(bx, new Object[][]{{0,"ptr","_Ptr","heap pointer when _Myres >= 16 (else the 16-byte SSO buffer starts here)"}});
|
||||
fill("std::string", 0x1c, "MSVC10 basic_string<char>: _Bx@0 (SSO buf / _Ptr), _Mysize@0x10, _Myres@0x14, _Alval@0x18. Verified from Stream::WriteString (FUN_008b9d70): if (str->_Myres > 15) p = str->_Ptr.",
|
||||
new Object[][]{{0,"std::string::_Bxty","_Bx"},{0x10,"uint","_Mysize"},{0x14,"uint","_Myres"},{0x18,"u8","_Alval"}});
|
||||
fill("std::vector", 12, "MSVC10 vector<T>: _Myfirst/_Mylast/_Myend (element type noted in field comments)", new Object[][]{{0,"ptr","_Myfirst"},{4,"ptr","_Mylast"},{8,"ptr","_Myend"}});
|
||||
fill("std::list", 8, "MSVC10 list<T>: _Myhead (sentinel node: next@0,prev@4,value@8), _Mysize", new Object[][]{{0,"ptr","_Myhead"},{4,"uint","_Mysize"}});
|
||||
fill("std::map", 8, "MSVC10 map/set: _Myhead (node: left@0,parent@4,right@8,key@0xc,value@0x10.. ,isnil byte at tail), _Mysize", new Object[][]{{0,"ptr","_Myhead"},{4,"uint","_Mysize"}});
|
||||
// opaque classes referenced by pointer
|
||||
for (String o : new String[]{"Game::BuildQueue","Game::TechTree","Game::ShipDesign","Game::CommMessageContainer","Game::Tech","Game::FleetNameGenerator","Game::AIRebellion","Game::AIEncounterFlags",
|
||||
"Game::ServerNodeGraph","Game::ServerTradeManager","Game::IServerSpyManager","Game::AttribMap","Game::SVScriptObject","Game::StarSystem","Game::Plague","Game::DefenceLayout","Game::SpecialProjectImpl","Game::StrategyEvent","Mars::Stream"}) opaque(o);
|
||||
// main structs first as empty shells so cross pointers resolve
|
||||
Structure sys = mk("Game::ServerSystem", 0x2d8); Structure ply = mk("Game::ServerPlayer", 0x3e0); Structure flt = mk("Game::StarFleet", 0x120); Structure shp = mk("Game::StarShip", 0xb0); Structure srv = mk("Game::StrategyServer", 0x320);
|
||||
// --- nested ---
|
||||
fill("Game::PlayerColorID", 4, "index (-1 = custom rgb) + r,g,b. Write FUN_0053c080", new Object[][]{{0,"i8","index"},{1,"u8","r"},{2,"u8","g"},{3,"u8","b"}});
|
||||
fill("Game::PopulationGroup", 0x18, "Read FUN_00536a80 / Write FUN_00536af0", new Object[][]{{0,"ptr","vptr"},{4,"int","PopT"},{8,"int","PopS"},{0x10,"i64","PopC"}});
|
||||
fill("Game::Population", 0x14, "vector<PopulationGroup>; Read FUN_005390c0 / Write FUN_00537ef0 (PopNG count, PopG entries)", new Object[][]{{0,"ptr","vptr"},{4,"std::vector","groups","vector<PopulationGroup> (stride 0x18)"}});
|
||||
fill("Game::Morale", 0x20, "int[7]; disk: mnsp then (msp,mv) sparse pairs. Read FUN_00744dd0 / Write FUN_00744ea0", new Object[][]{{0,"ptr","vptr"},{4,"int[7]","values"}});
|
||||
fill("Game::MoraleEvent", 0x50, "Read FUN_007490b0 / Write FUN_007491b0", new Object[][]{{0,"ptr","vptr"},{4,"int","mid"},{8,"int","mtr"},{0xc,"int","mn"},{0x10,"int","mtp"},{0x14,"Game::Morale","mfx"},{0x34,"std::string","mdsc"}});
|
||||
fill("Game::IndependenceInfo", 0x70, "Read FUN_00748df0 / Write FUN_00748ee0", new Object[][]{{0,"ptr","vptr"},{4,"int","indsp"},{8,"Game::PlayerColorID","indcl"},{0x1c,"std::string","indnm"},{0x38,"std::string","indav"},{0x54,"std::string","indba"}});
|
||||
fill("Game::StarSystem::OutputRates", 0x1c, "POD; Read FUN_007472a0 / Write FUN_00745190. Disk order SRs,SRt,SRsc,SRtf,SRi,SRoh,SRnr", new Object[][]{{0,"float","SRt"},{4,"float","SRsc"},{8,"float","SRtf"},{0xc,"float","SRi"},{0x10,"float","SRoh"},{0x14,"float","SRs"},{0x18,"int","SRnr"}});
|
||||
fill("Game::StarSystem::PlayerView", 0x9c, "per-player seen snapshot (map value in ServerSystem.NVs). Read FUN_00752af0 / Write FUN_007492d0", new Object[][]{{0,"ptr","vptr","0x00a201ac"},{8,"int","VTrn"},{0xc,"int","Pop"},{0x10,"Game::Population","Pop2"},{0x24,"float","Infra"},{0x28,"float","Suit"},{0x2c,"int","Res"},{0x30,"int","ARes2"},{0x34,"int","MRes"},{0x38,"bool","NoRebAI"},{0x3c,"int","pbon"},{0x40,"Game::Population","pbon2"},{0x54,"float","ibon"},{0x58,"int","TerrFl"}});
|
||||
fill("Game::ShipBuildOrder", 0x18, "build-queue entry. Read FUN_00813770 / Write FUN_00813800", new Object[][]{{0,"ptr","vptr"},{4,"int","desID"},{8,"int","con"},{0xc,"int","sav"},{0x10,"int","conleft"},{0x14,"int","ordID"}});
|
||||
fill("Game::DiplomacyStats", 0x24, "int16 counters (int32 on disk). Write FUN_00818cb0", new Object[][]{{0,"ptr","vptr"},{4,"int","other"},{8,"short","lastnap"},{0xa,"short","lastnapbty"},{0xc,"short","bknnap"},{0xe,"short","btynap"},{0x10,"short","lastally"},{0x12,"short","lastallybty"},{0x14,"short","bknally"},{0x16,"short","btyally"},{0x18,"short","lastcf"},{0x1a,"short","lastcfbty"},{0x1c,"short","bkncf"},{0x1e,"short","btycf"},{0x20,"short","deadhome"}});
|
||||
fill("Game::PlayerReport", 0x30, "preps entry. Read FUN_008200a0 / Write FUN_00817480", new Object[][]{{0,"ptr","vptr"},{4,"int","oid"},{8,"int","pid"},{0xc,"int","flds"},{0x10,"int","sav"},{0x14,"int","home"},{0x18,"int","ncol"},{0x1c,"int","mpwr"},{0x20,"int","mcls"},{0x24,"int","mmsl"},{0x28,"int","nshp"},{0x2c,"int","nsat"}});
|
||||
fill("Game::PlayerAlliances", 0x10, "Write FUN_006d2e10 (disk tag Team)", new Object[][]{{0,"int","ALid"},{4,"int","AL"},{8,"int","NA"},{0xc,"int","CF"}});
|
||||
fill("Game::NodeRoute", 0x10, "Read FUN_006e2260 / Write FUN_006e22e0", new Object[][]{{0,"ptr","vptr"},{4,"int","nrp"},{8,"int","nrf"},{0xc,"int","nrt"}});
|
||||
fill("Game::Waypoint", 0x1c, "Read FUN_00701860 / Write FUN_00700ed0", new Object[][]{{0,"ptr","vptr"},{4,"int","Wpt"},{8,"int","Tp"},{0xc,"Game::NodeRoute","nrt"}});
|
||||
fill("Game::FlightPlan", 0x38, "Read FUN_00704c70 / Write FUN_00700f60", new Object[][]{{0,"ptr","vptr"},{4,"std::vector","wpts","vector<Waypoint> (stride 0x1c)"},{0x14,"float","FPsp2"},{0x18,"int","FPeta2"},{0x1c,"Mars::Vector3","FPogn2"},{0x28,"Mars::Vector3","FPdpos"},{0x34,"int","pnd"}});
|
||||
fill("Game::ShipHealth", 0x10, "3 unnamed floats on disk. Write FUN_00813e50", new Object[][]{{0,"ptr","vptr"},{4,"float[3]","hp"}});
|
||||
fill("Game::PrisonerHold", 0x18, "Read FUN_0056eb00 / Write FUN_0056ec00; counts[0]=PrMax, [2..8] per species", new Object[][]{{0,"ptr","vptr"},{0x14,"int*","counts"}});
|
||||
fill("Game::EventStorage", 0x1c, "Write FUN_00825cc0", new Object[][]{{0,"ptr","vptr"},{4,"std::vector","Events","vector<TurnEvents>"},{0x14,"int","EvNxID"}});
|
||||
fill("Game::CivilianRatios", 0x2c, "Write FUN_0082c740 (body not recovered)", new Object[][]{{0,"ptr","vptr"}});
|
||||
fill("Game::ShipRecords", 0x44, "Write FUN_008176a0 (body not recovered)", new Object[][]{{0,"ptr","vptr"}});
|
||||
fill("Game::FleetLayout", 0x24, "two vectors; HLay gate = either non-empty", new Object[][]{{0,"ptr","vptr"},{4,"std::vector","v0"},{0x14,"std::vector","v1"}});
|
||||
fill("Game::SpyReport", 0x34, "four std::lists. Read FUN_008843d0 / Write FUN_00828ec0", new Object[][]{{0,"ptr","vptr"},{4,"std::list","defences","SpyReportDefences (count defc2)"},{0x10,"std::list","trade","SpyReportTrade (rtc)"},{0x1c,"std::list","events","SpyReportEvents (evc)"},{0x28,"std::list","techtree","SpyReportTechTree (ttc)"}});
|
||||
// --- ServerSystem ---
|
||||
fill("Game::ServerSystem", 0x2d8, "Game::ServerSystem : StarSystem : StarMapNode (+NetworkObject@0, IStreamable@8, HandleObject@0xc). Serializer this = obj+8. Read FUN_0075d4b0 / Write FUN_00749630.", new Object[][]{
|
||||
{0,"ptr","vptr_StarSystem","0x00a200e4"},{4,"int","netId","NetworkObject id (handle id written by Stream::WriteHandleId)"},{8,"ptr","vptr_IStreamable","0x00a2043c; serializer this"},{0xc,"ptr","vptr_HandleObject"},{0x10,"Game::StrategyServer*","owner","owner->Players used as player table"},
|
||||
{0x18,"Mars::Vector3","Pos"},{0x4c,"float","R"},{0x50,"float","G"},{0x54,"float","B"},{0x58,"float","A"},{0x5c,"int","Idx"},{0x60,"int","Size"},{0x64,"float","Suit"},{0x68,"int","Res"},{0x6c,"int","ARes2"},{0x70,"int","MRes"},{0x74,"int","TRes"},
|
||||
{0x78,"bool[3]","haltv"},{0x7c,"float","OutMod"},{0x80,"int","TAcq"},{0x84,"int","TFAcq"},{0x88,"Game::StarSystem::OutputRates","Rts"},{0xa4,"Game::BuildQueue*","BQ"},{0xa8,"std::string","Name"},
|
||||
{0xc4,"bool","Abdn"},{0xc5,"bool","Dstyd"},{0xc6,"bool","vnh"},{0xc7,"bool","vnd"},{0xc8,"bool","vnex3"},{0xc9,"bool","vnpex3"},{0xcc,"int","VFlags"},{0xd0,"int","EFlags"},{0xd4,"int","AFlags"},{0xd8,"int","FFlags"},{0xdc,"int","GFlags"},{0xe0,"int","MnRFlags"},{0xe4,"int","RfRFlags"},{0xe8,"int","ClkFlags"},
|
||||
{0xf0,"i64","Bats2"},{0xf8,"i64","rcex"},{0x100,"Game::ServerPlayer*","PID","owner player (handle id on disk)"},{0x104,"Game::Population","dcs"},{0x118,"float","dsu"},{0x11c,"Game::Morale","cm"},{0x13c,"std::vector","cme2","vector<MoraleEvent>"},{0x14c,"Game::Morale","PvCM"},
|
||||
{0x16c,"std::vector","Flts","vector<StarFleet*> (NumFlts/Flt)"},{0x17c,"float","RepCur"},{0x180,"float","RepMax"},{0x184,"int","EggScio"},{0x188,"bool","NoRebAI"},{0x189,"bool","PvNoRebAI"},{0x18c,"int","Pop"},{0x190,"float","Infra"},{0x194,"int","pbon"},{0x198,"float","ibon"},{0x19c,"int","TerrFl"},
|
||||
{0x1a0,"Game::Population","Pop2"},{0x1b4,"Game::Population","pbon2"},{0x1c8,"Game::IndependenceInfo*","indi","hindi = (indi != NULL)"},{0x1cc,"std::vector","spies2","vector<int>"},{0x1dc,"int","rbfl"},{0x1e0,"bool","hsrg"},{0x1e4,"int[7]","adt","addiction table: nadct + sparse (ads,adt)"},
|
||||
{0x200,"int","PvPop"},{0x204,"float","PvInfra"},{0x208,"float","PvSuit"},{0x20c,"int","PvRes"},{0x210,"int","PvARes2"},{0x214,"int","PvMRes"},{0x218,"Game::Population","PvPop2"},{0x238,"Game::StarFleet*","DefF"},{0x23c,"Game::StarFleet*","DefSF"},
|
||||
{0x240,"std::vector","GFs","gates (NumGFs/GF)"},{0x250,"std::vector","SnF","stations (NumSnF/SnF)"},{0x260,"std::vector","MnF","monitors (NumMnF/MnF)"},{0x274,"std::map","NVO","colonies: key player idx -> {TShn i16@+2, OID@+4, isind@+8, indi IndependenceInfo@+0xc}"},{0x284,"std::map","NVE","key player idx -> {ETS i16@+2, Eid@+4}"},{0x294,"std::map","NVs","key player idx -> PlayerView (pview)"},
|
||||
{0x2a8,"std::vector","Plgs","vector<Plague*> (NumPlgs2/PlgT/Plg)"},{0x2b8,"int","TnsOH"},{0x2bc,"int","TDst"},{0x2c4,"int","ntdev"},{0x2c8,"int","ltis"},{0x2cc,"int","rbtn"},{0x2d0,"int","rbfr"},{0x2d4,"int","rbwn"}});
|
||||
// --- ServerPlayer ---
|
||||
fill("Game::ServerPlayer", 0x3e0, "Game::ServerPlayer : StrategyPlayer (NetworkObject@0). IStreamable sub-object at +0x3a0 (serializer this = obj+0x3a0, most member offsets negative there). Read FUN_008804d0 / Write FUN_008563e0.", new Object[][]{
|
||||
{0,"ptr","vptr","0x00a327a4"},{4,"int","netId"},{0x28,"int","PlyrIdx"},{0x2c,"Game::ServerSystem*","HomeSys"},{0x30,"std::vector","Own","vector<ServerPlayer*> (NumOwn/OwnId)"},{0x40,"std::string","PlryName"},{0x5c,"int","Species","0 Human .. 6 Morrigi"},{0x60,"Game::PlayerColorID","ClrID"},
|
||||
{0x74,"std::string","Bdg"},{0x90,"std::string","Avt"},{0xac,"int","Team"},{0xb0,"float","IdealSuit"},{0xb4,"float","SuitTol"},{0xb8,"float","MaxOH"},{0xbc,"float","ResRate"},{0xc0,"float","ResMod"},{0xc4,"float","ResScl"},{0xd0,"float","TRM"},{0xd4,"int","TRA"},{0xd8,"int","TRP"},
|
||||
{0xe4,"std::vector","Des","vector<ShipDesign*> (NumDes/DesID/Des)"},{0xf4,"Game::TechTree*","TechTree"},{0xf8,"bool","Elim"},{0xf9,"bool","bTurnDone_nonser","not serialized; tested by StrategyServer::OnPlayerEndTurn"},{0xfb,"bool","NPC"},{0xfc,"bool","RebAI"},{0xfd,"bool","ReqCL"},{0xfe,"bool","AIBn"},{0xff,"bool","CnTrd"},{0x100,"bool","CnRad"},{0x101,"bool","CnVItl"},{0x102,"bool","hgs"},{0x103,"bool","hadvs"},{0x104,"bool","harcc"},
|
||||
{0x108,"float","pddm"},{0x10c,"float[3]","ConMod"},{0x118,"float[3]","SavMod"},{0x124,"float","OutMod"},{0x128,"float","RebOutMod"},{0x12c,"float","ScOutMod"},{0x130,"float","PopMod"},{0x134,"float","TerraMod"},{0x138,"bool","AMine"},{0x13c,"float","MinPure"},{0x140,"float","MinRate"},{0x144,"int","NGts"},{0x148,"int","PrGtTrf"},{0x14c,"int","GTraf"},
|
||||
{0x150,"float","CstR"},{0x154,"float","CstE"},{0x158,"float","CstT"},{0x15c,"int","Maint"},{0x160,"float","shrm"},{0x164,"int","Status","also set by SNMSetPlayerStatus; 4 = turn done"},{0x168,"Game::PlayerAlliances","Team2","disk tag Team {ALid,AL,NA,CF}"},{0x178,"std::vector","Leg","vector<ShipDesign*> (NumLeg)"},{0x188,"int","PvSav"},{0x18c,"bool","PvMA"},
|
||||
{0x19c,"int","HasDisc"},{0x1a0,"int","HasDiscSp"},{0x1a4,"int","HasDiscCl"},{0x1a8,"int","HasEnc"},{0x1ac,"int","HasEng"},{0x1b0,"Game::ShipRecords","ShipRecs"},{0x1f4,"std::vector","Ojvs","vector<Objective>"},{0x204,"std::vector","Nexp","vector<{int xid,xmin,xmax; float xper}> (16 B)"},{0x214,"std::vector","WeapXcl","vector<int>"},
|
||||
{0x230,"std::vector","dipstats","vector<DiplomacyStats>"},{0x240,"Game::CommMessageContainer*","comms"},{0x244,"std::vector","preps","vector<PlayerReport>"},{0x254,"std::vector","odes","vector<ObservedDesign>"},{0x264,"std::vector","owep","vector<ObservedWeapon>"},{0x274,"std::vector","otch","vector<ObservedTech>"},
|
||||
{0x284,"int","Sav"},{0x288,"int","HasImm"},{0x28c,"int","HasVac"},{0x290,"int","NPTrk"},{0x294,"Game::Tech*","ResT","current research (ResTNm = tech->name)"},{0x298,"Game::FleetNameGenerator*","FNG"},{0x29c,"Game::EventStorage","Events"},{0x2b8,"std::list","Nts","list<PlayerNotes> (NumNotes/Nts)"},
|
||||
{0x2c4,"int","BnkWrn"},{0x2c8,"int","BnkTrn"},{0x2cc,"int","BnkEl"},{0x2d0,"int","BnkPr"},{0x2d8,"int","plcy"},{0x2dc,"std::string","pswd"},{0x2f8,"bool","Srn"},{0x2fc,"ptr","SrnTo","handle id on disk"},{0x300,"int","lboid"},{0x304,"int","lcid2"},{0x30c,"float","IncMod"},{0x310,"std::vector","aid","vector<PlayerAid>"},
|
||||
{0x320,"std::vector","deflay","vector<DefenceLayout*> (ndeflay/deflay)"},{0x330,"bool","cdp"},{0x334,"Game::SpyReport*","spy2"},{0x338,"std::vector","rdt","vector<RaidTargets> (stride 0x20)"},{0x368,"int","aidf"},{0x370,"Game::CivilianRatios","civr"},{0x39c,"int","tnc","written as max(v,1)"},
|
||||
{0x3a0,"ptr","vptr_IStreamable","0x00a32794; serializer this"},{0x3a4,"std::vector","PR","vector<{float PRm; int PRBt}>"},{0x3b4,"bool","ResErrRoll"},{0x3b5,"bool","cta"},{0x3b8,"Game::AIRebellion*","AIR","HasAIR = (AIR != NULL)"},{0x3bc,"Game::AIEncounterFlags*","AIEnf"},{0x3c0,"std::vector","Sprj","vector<SpecialProjectImpl*> (NSprj/SprjT/Sprj)"},{0x3d0,"int","NextPrjID"},{0x3d4,"int","lret"},{0x3dc,"int","nmeid"}});
|
||||
// --- StarFleet ---
|
||||
fill("Game::StarFleet", 0x120, "Game::StarFleet : StarMapNode (+NetworkObject@0, IStreamable@8, HandleObject@0xc). Serializer this = obj+8. Read FUN_00702470 / Write FUN_00701070.", new Object[][]{
|
||||
{0,"ptr","vptr","0x00a1d608"},{4,"int","netId"},{8,"ptr","vptr_IStreamable","0x00a1d5f8"},{0xc,"ptr","vptr_HandleObject"},{0x10,"ptr","owner"},{0x18,"Mars::Vector3","Pos"},{0x4c,"Mars::Vector3","PrvPos"},{0x58,"Game::ServerPlayer*","PID"},{0x5c,"std::string","FtName"},{0x78,"bool","Perm"},
|
||||
{0x7c,"Game::FleetLayout","Lay","HLay gate"},{0xa0,"Game::StarSystem*","LocID"},{0xa4,"std::vector","Ships","vector<StarShip*> (NShips/ShipID/Ship)"},{0xc4,"Game::FlightPlan","FPlan","HFPlan gate = wpts non-empty"},{0xfc,"int","FtTrans"},{0x100,"Mars::Vector3","FtOrig"},{0x10c,"int","FtFlg"},{0x110,"int","Ftae"},{0x114,"int","Ftpae"},{0x118,"int","FtEnc"},{0x11c,"int","FtMS"}});
|
||||
// --- StarShip ---
|
||||
fill("Game::StarShip", 0xb0, "Game::StarShip (NetworkObject@0, IStreamable@8). Serializer this = obj+8. Read FUN_00853fa0 / Write FUN_008291f0.", new Object[][]{
|
||||
{0,"ptr","vptr","0x00a31418"},{4,"int","netId"},{8,"ptr","vptr_IStreamable","0x00a31408"},{0x10,"Game::ServerPlayer*","PlrID"},{0x14,"Game::ShipDesign*","Des","DesID = design->+0xa4"},{0x20,"float","Range"},{0x24,"Game::ShipHealth","Health"},{0x34,"int","MineCap"},{0x38,"std::vector","TH","vector<{float th,thm}> (NTH/TH/THM)"},
|
||||
{0x48,"int","Plg"},{0x4c,"int","Act"},{0x50,"bool","Dep"},{0x51,"bool","Atq"},{0x5c,"int","LCT"},{0x60,"int","tsd"},{0x64,"Game::StarFleet*","FltID"},{0x68,"int","ConCap"},{0x6c,"float","RefCap"},{0x70,"float","RepCap"},{0x7c,"int","EncID"},{0x80,"Game::PrisonerHold","PrisH"},{0x98,"Game::BuildQueue*","BQ2","hbq gate"},{0x9c,"Game::Population*","pop","hsp gate"},{0xa0,"Game::Population*","ppop"},{0xa8,"int","atsp"},{0xac,"int","tblt"}});
|
||||
// --- StrategyServer (partial; from Write FUN_0079fa70 + spine work) ---
|
||||
fill("Game::StrategyServer", 0x320, "Game::StrategyServer (IStreamable@0 = serializer this, primary vftable 0x00a26034 @+4). Offsets from Write FUN_0079fa70 and the turn-spine functions; PARTIAL.", new Object[][]{
|
||||
{0,"ptr","vptr_IStreamable","0x00a26084"},{4,"ptr","vptr","0x00a26034"},{8,"int","ModCount"},{0xc,"int","Frame","turn number; ++ in BeginProcessTurn"},{0x10,"int","GOTurn"},{0x14,"int","GameID"},{0x28,"std::string","GameName"},
|
||||
{0x44,"std::vector","Systems","vector<ServerSystem*> (NumSys/SysID/Sys)"},{0x54,"std::vector","Players","vector<ServerPlayer*> (NumPlrs/PlayerID/Player)"},{0x64,"std::vector","Fleets","vector<StarFleet*> (NumFlts/FltID/Flt)"},{0x74,"std::vector","Acts","vector<obj*> (NumActs/Act handle ids)"},
|
||||
{0x8c,"std::vector","NodeMapLines","stride 0x14; NMSz = count"},{0x9c,"int","NMLc"},{0xbc,"int","Map"},{0xc0,"float","IncMod"},{0xc4,"float","ResMod"},{0xc8,"bool","EnAl"},{0xc9,"bool","EnTm"},{0xfc,"float[7]","ISsu","per-species (ISsp name, ISsu value)"},{0x134,"std::string","KeyPath"},
|
||||
{0x154,"Game::ServerNodeGraph*","NdGr2"},{0x158,"Game::ServerTradeManager*","trdmgr"},{0x15c,"Game::IServerSpyManager*","spymgr"},{0x164,"Game::AttribMap*","Attrib"},{0x170,"pfn","OnEventCallback","(*cb)(playerNetId, int eventType, StrategyEvent** ev); set in ctor FUN_007d78d0 from ctor arg; NULL -> 'OnEvent() called, but no callback function specified'"},
|
||||
{0x1a0,"float","RandEncAdj"},{0x1b0,"ptr","listener","optional observer; vft+0x10(code,&args) called on player/fleet removal"},{0x1b4,"Game::SVScriptObject*","SvSctOb"},{0x1b8,"int","NPCm"},{0x1bc,"int","NPCo"},{0x1c0,"int","NPCi"},{0x1c4,"int","NPCv"},{0x1c8,"int","NPCa"},{0x1cc,"float","szadj"},{0x1d0,"float","rsadj"},{0x1d4,"float","suadj"},{0x1f8,"int","cmbtid"},
|
||||
{0x2b8,"std::list","StrategyEvents","list<StrategyEvent*> pending events (node: next,prev,vptr@8,..,targetId@0x10,id2@0x14)"},{0x2ec,"bool","bRecordEvents","gate for the per-player removal lists at 0x2d4/0x2e0"},{0x2f1,"bool","bProcessingTurn","set 1 in BeginProcessTurn"},{0x318,"std::map","zds","zdsc/zdsi/zdst"}});
|
||||
Structure sysV = serView("Game::ServerSystem", 8); Structure fltV = serView("Game::StarFleet", 8); Structure shpV = serView("Game::StarShip", 8);
|
||||
|
||||
log.println("\n==== FUNCTION RENAMES: serializers ====");
|
||||
Object[][] ser = {
|
||||
{0x0075d4b0L,"Game::ServerSystem::Read"},{0x00749630L,"Game::ServerSystem::Write"},{0x008804d0L,"Game::ServerPlayer::Read"},{0x008563e0L,"Game::ServerPlayer::Write"},
|
||||
{0x00702470L,"Game::StarFleet::Read"},{0x00701070L,"Game::StarFleet::Write"},{0x00853fa0L,"Game::StarShip::Read"},{0x008291f0L,"Game::StarShip::Write"},
|
||||
{0x007d27a0L,"Game::StrategyServer::Read"},{0x0079fa70L,"Game::StrategyServer::Write"},{0x00727790L,"Game::StarMapNode::Read"},{0x00727820L,"Game::StarMapNode::Write"},
|
||||
{0x00752af0L,"Game::StarSystem::PlayerView::Read"},{0x007492d0L,"Game::StarSystem::PlayerView::Write"},{0x007472a0L,"Game::StarSystem::OutputRates::Read"},{0x00745190L,"Game::StarSystem::OutputRates::Write"},
|
||||
{0x005390c0L,"Game::Population::Read"},{0x00537ef0L,"Game::Population::Write"},{0x00536a80L,"Game::PopulationGroup::Read"},{0x00536af0L,"Game::PopulationGroup::Write"},
|
||||
{0x00748df0L,"Game::IndependenceInfo::Read"},{0x00748ee0L,"Game::IndependenceInfo::Write"},{0x00744dd0L,"Game::Morale::Read"},{0x00744ea0L,"Game::Morale::Write"},{0x007490b0L,"Game::MoraleEvent::Read"},{0x007491b0L,"Game::MoraleEvent::Write"},
|
||||
{0x00813770L,"Game::ShipBuildOrder::Read"},{0x00813800L,"Game::ShipBuildOrder::Write"},{0x00813250L,"Game::PlayerNotes::Read"},{0x008132b0L,"Game::PlayerNotes::Write"},{0x008843d0L,"Game::SpyReport::Read"},{0x00828ec0L,"Game::SpyReport::Write"},
|
||||
{0x008200a0L,"Game::PlayerReport::Read"},{0x00817480L,"Game::PlayerReport::Write"},{0x00818cb0L,"Game::DiplomacyStats::Write"},{0x00704c70L,"Game::FlightPlan::Read"},{0x00700f60L,"Game::FlightPlan::Write"},
|
||||
{0x00701860L,"Game::FlightPlan::Waypoint::Read"},{0x00700ed0L,"Game::FlightPlan::Waypoint::Write"},{0x006e2260L,"Game::NodeRoute::Read"},{0x006e22e0L,"Game::NodeRoute::Write"},{0x0056eb00L,"Game::PrisonerHold::Read"},{0x0056ec00L,"Game::PrisonerHold::Write"},
|
||||
{0x00825cc0L,"Game::EventStorage::Write"},{0x006d2e10L,"Game::PlayerAlliances::Write"},{0x00813e50L,"Game::ShipHealth::Write"},{0x0053c080L,"Game::PlayerColorID::Write"},{0x008a60d0L,"Mars::Vector3::Write"},{0x005890a0L,"Game::TechTree::Write"},{0x008176a0L,"Game::ShipRecords::Write"},{0x0082c740L,"Game::CivilianRatios::Write"}};
|
||||
for (Object[] r : ser) fn((Long) r[0], (String) r[1], "IStreamable serializer (vftable slot [1]=Read, [2]=Write). Member table: " + DOC);
|
||||
log.println("\n==== FUNCTION RENAMES: stream primitives ====");
|
||||
String P = "Stream primitive wrapper. Stream vftable slots: +0x10 ReadIntRef(name,&v)->found, +0x14 ReadNested(name,helper|NULL=skip), +0x18 String, +0x1c Bool, +0x20 Float, +0x24 Int(name,v,default=-1), +0x28 Nested(name,StreamableHelper*), +0x30 RawBytes(name,ptr,n). " + DOC;
|
||||
Object[][] prims = {{0x008b9d70L,"Mars::Stream::WriteString","(Stream*, name, std::string*) -> vft+0x18; does the _Myres>=16 heap/SSO select"},{0x008b9c20L,"Mars::Stream::WriteBool","(Stream*, name, bool*) -> vft+0x1c"},{0x008b9be0L,"Mars::Stream::WriteFloat","(Stream*, name, float*) -> vft+0x20"},
|
||||
{0x008b9d50L,"Mars::Stream::WriteInt","(Stream*, name, int*) -> vft+0x24"},{0x008b9d00L,"Mars::Stream::WriteInt16AsInt","(Stream*, name, int16*) widened -> vft+0x24"},{0x008b9cb0L,"Mars::Stream::WriteInt8AsInt","(Stream*, name, int8*) -> vft+0x24"},{0x008b9c60L,"Mars::Stream::WriteInt64","(Stream*, name, int64*) -> vft+0x30 raw 8 bytes"},
|
||||
{0x00816490L,"Mars::Stream::WriteHandleId","(Stream*, name, NetworkObject*) writes obj ? obj->id(+4) : 0"},{0x008b9bc0L,"Mars::Stream::ReadFloat",""},{0x008b9d20L,"Mars::Stream::ReadInt",""},{0x008b9c00L,"Mars::Stream::ReadBool",""},{0x008b9d90L,"Mars::Stream::ReadString","vft+4 (name, buf, 0x400) then assigns std::string"},
|
||||
{0x008b9c40L,"Mars::Stream::ReadInt64",""},{0x008b9cd0L,"Mars::Stream::ReadInt16",""},{0x008164d0L,"Mars::Stream::ReadHandle","(Stream*, name) -> handle id -> object* lookup"}};
|
||||
for (Object[] r : prims) fn((Long) r[0], (String) r[1], ((String) r[2]) + "\n" + P);
|
||||
|
||||
log.println("\n==== FUNCTION RENAMES: affinity / app / main loop ====");
|
||||
fn(0x0089ee70L, "Process_PinAffinity", "Pins the whole process to ONE logical CPU: SetProcessAffinityMask(GetCurrentProcess(), 1 << coreIndex(ESI)). Logs 'Limiting process affinity to CPU-%i'. Called from Mars::Application::Initialize when config CPU/ForceSingleCore > 0. Only affinity/topology API in the binary. Source: findings/objects/ghidra-recon.md");
|
||||
fn(0x008a0e50L, "Mars::Application::Initialize", "AppStartup_ReadConfig: reads CPU/ForceSingleCore (-> Process_PinAffinity), startup config via Mars::AppStartup parser, display.cfg, audio.cfg, COM, Direct3DCreate9, window (CreateAppWindow), DrawDevice, timers, spawns the streaming-sound update thread (SoundStreamingThreadProc, THREAD_PRIORITY_TIME_CRITICAL), GlobalConsts/textures/effects/sprites/keymap/font, SimplePainter, PanelManager, then IApplication::OnStartup() (vft+0x10 = Game::DemoApp::OnStartup). this = g_pApplication (DemoApp). Source: findings/objects/ghidra-recon.md + " + DOC2);
|
||||
lbl(0x008a0e50L, "AppStartup_ReadConfig", null);
|
||||
fn(0x0089dd30L, "WinMain", "WinMain(hInst, hPrev, lpCmdLine, nShow) called from ___tmainCRTStartup. Tokenises the command line, creates mutex Kerberos_SwordOfTheStars_Mutex (single instance unless /concurrent), new Game::DemoApp (0x1b8 bytes) -> g_pDemoApp, Mars::Application::Initialize, then Mars::Application::Run (main loop). " + DOC2);
|
||||
fn(0x0089c950L, "Game::DemoApp::DemoApp", "DemoApp ctor (IApplication impl, vftable 0x00a36004). Calls Mars::Application ctor which sets g_pApplication.");
|
||||
fn(0x008a0170L, "Mars::Application::Application", "Mars::Application ctor: reads app config, sets g_pApplication (DAT_00b2d540) = this.");
|
||||
fn(0x0089f5b0L, "Mars::Application::Run", "MAIN LOOP. do { FrameTimer::Update; PanelManager->vft+0x60 (UI update); app->vft+0x18 OnUpdate (DemoApp::OnUpdate: network pump + game update); if (!focused && app->vft+0xc) Sleep(20); _controlfp; periodic timer-callback flush (TimerList::Dispatch) every _DAT_00a36860 s; ok = app->vft+0x1c OnTick (DemoApp::OnTick: sound/console/panel/state machine; returns 0 when quitting); if (ok && DrawDevice) { dev->BeginFrame; if (!dev->IsLost) { dev->+8; app->vft+0x20 OnRender; dev->+0xc; dev->Present } } } while (PumpMessages() && ok). " + DOC2);
|
||||
fn(0x0089f1c0L, "Mars::Application::PumpMessages", "PeekMessage/TranslateMessage/DispatchMessage loop; returns 0 on WM_QUIT (0x12). Keyboard-translation gate: PanelManager(+0x78)->vft+0x58.");
|
||||
fn(0x0090c700L, "Mars::FrameTimer::Update", "QueryPerformanceCounter-based frame timer: dt(+0x14), total(+0x18 double), fps(+0xc) every +0 ticks.");
|
||||
fn(0x008e5ac0L, "Mars::TimerList::Dispatch", "walks a std::map of timers calling the +0x20 callback; used by the main loop at a fixed period.");
|
||||
fn(0x0089fe70L, "Mars::Application::CreateAppWindow", "RegisterClass/CreateWindow/ShowWindow for Kerberos_SwordOfTheStars_WndCls.");
|
||||
fn(0x0089f4d0L, "Mars::AppStartup::OnConfigToken", "AppStartup vftable[1]: startup-config token handler (key 'conlevel' -> console level FUN_008ba280; other keys -> app->+4->vft+0x24).");
|
||||
fn(0x008cd820L, "Mars::ConfigParser::ParseFile", "generic tokenising config-file parser with callback object (used for the startup config with Mars::AppStartup).");
|
||||
fn(0x0089d610L, "Game::DemoApp::OnStartup", "IApplication vft+0x10: version banner 'Sword of the Stars%s (%s %s)', profiles, Mars::Network::Startup (creates network watchdog thread), ParticleSystemManager, Mesh, Model, StringTable (Locale/<loc>/Strings.csv), species, GUIResources, SoundSystem, UI sounds, SpeechEvents, GUIAppearances, PlayerColor/Badge/Avatar dictionaries, PlanetPainter, then main menu / '/join'. " + DOC2);
|
||||
fn(0x0089dfb0L, "Game::DemoApp::OnShutdown", "IApplication vft+0x14: destroys all game/UI subsystems.");
|
||||
fn(0x00898800L, "Game::DemoApp::OnUpdate", "IApplication vft+0x18 (called first each main-loop iteration): Mars::Network update (FUN_00902ad0), then current game object(+0x158)->vft+4 Update, then FUN_007879c0 if a strategy game (+0x150) exists.");
|
||||
fn(0x0089a640L, "Game::DemoApp::OnTick", "IApplication vft+0x1c: sound/console/UI tick and the top-level game-state machine (loads/unloads the strategy game at +0x150 / combat at +0x158); returns 0 when quitting (+0x1b2).");
|
||||
fn(0x00899210L, "Game::DemoApp::OnRender", "IApplication vft+0x20: PanelManager->vft+0x64 draw, or the movie/splash player at +0x108.");
|
||||
fn(0x008986a0L, "Game::DemoApp::ShouldSleepWhenInactive", "IApplication vft+0xc");
|
||||
|
||||
log.println("\n==== FUNCTION RENAMES: threads ====");
|
||||
fn(0x00902470L, "Mars::Network::Startup", "called from DemoApp::OnStartup; -> NetworkManager::Create");
|
||||
fn(0x00902350L, "Mars::NetworkManager::Create", "THREAD SITE 1 (0x0090242d): InitializeCriticalSection(g_netCS 0x00b2e5f8), new NetworkManager(0x150), CreateThread(NetworkWatchdogThreadProc). Globals: 0x00b2e61c manager, 0x00b2e620 thread handle, 0x00b2e618 running flag. " + DOC2);
|
||||
fn(0x00901f40L, "Mars::NetworkManager::WatchdogThreadProc", "THREAD 1 body: loop { Sleep(10); Enter(g_netCS); pump host link(+0x20)/listener(+0x98); if host link timed out -> log 'Network(%f): No response from host %s' and drop; for each client link (+0x70 vector, stride 0xc) -> 'No response from client %s', collect & drop; Leave } until manager NULL. Pure timeout watchdog, not the game sim. " + DOC2);
|
||||
fn(0x008fe730L, "Mars::NetworkManager::NetworkManager", "ctor (0x150 bytes)");
|
||||
fn(0x008ef1d0L, "Mars::SoundSystem::StreamingUpdateThreadProc", "THREAD 2 body (created in Mars::Application::Initialize at 0x008a14ef, priority 15): WaitForMultipleObjects(app+0x84 wake event, app+0x88 quit event); under g_musicCS (0x00b2e4a8): if current music stream (0x00b2e4c4) finished/faded (+0x388 float <= 0) -> stop it and open the next queued music file (MusicPlayer::OpenMusicFile from the 0x00b2e4d0 list); else StreamingSound::FillBuffer; then FUN_008b65d0 (sound system update). Streams music/DirectSound buffers only. " + DOC2);
|
||||
fn(0x0091b5a0L, "Mars::StreamingSound::FillBuffer", "DirectSound streaming buffer refill (GetCurrentPosition / Lock / decode / Unlock); 'Couldn't restore buffer'.");
|
||||
fn(0x008ef040L, "Mars::MusicPlayer::OpenMusicFile", "opens a music file (FUN_008b66e0) as the current stream 0x00b2e4c4; '[%s] cannot open music file for playback'.");
|
||||
fn(0x00736e30L, "Game::BackgroundWorker::Start", "THREAD SITE 3 (0x00736e84): embedded struct {CRITICAL_SECTION cs@0; HANDLE thread@0x18; ...; job* @0x54; flags @0x58..0x5c (0x5b = quit request, 0x5c = exited)}; InitializeCriticalSection + CreateThread(BackgroundWorker::ThreadProc, this). Owner: StarMapPanelBase (strategic star-map renderer). " + DOC2);
|
||||
fn(0x00735bb0L, "Game::BackgroundWorker::ThreadProc", "THREAD 3 body: loop { Sleep(10); Enter(cs); if quit(+0x5b) {set exited(+0x5c); Leave; break}; Leave; Enter; job = (+0x54 && !+0x59) ? +0x54 : NULL; Leave; if (job) { StarMapBlobs::BuildBlobMesh_Job(job+0x3c, job+4); Enter; +0x58=0,+0x59=1 (done); Leave } }. Background political-map 'blob' (territory overlay) mesh builder for the star map; NOT on the battle path. " + DOC2);
|
||||
fn(0x00732ab0L, "Game::StarMapBlobs::BuildBlobMesh_Job", "worker job: sums point cloud (job+0x28 vector<Vector3>), centroid/spread (sqrt), computes a blob radius (job+0x14) then FUN_008fc160 (implicit-surface / metaball polygoniser with callbacks FUN_00722010/FUN_0071ea60) and FUN_008fa5b0 (mesh build) into the output vertex list (param_1). Used by the political map overlay (Render/StarMapBlobs_*.fx).");
|
||||
fn(0x00741cd0L, "Game::StarMapPanelBase::StarMapPanelBase", "ctor of the strategic star-map view base class (Render/StarMapBlobs_Solid.fx, StarMapBlobs_Glow.fx, POLMAP_* colours, Skysphere); first statement starts the BackgroundWorker thread (this embedded at +0). Derived: Game::PoliticalMapPanel (ctor FUN_007424b0), created by the strategy-map screen ctor FUN_005e9780.");
|
||||
|
||||
log.println("\n==== FUNCTION RENAMES: game creation / strategy server ====");
|
||||
fn(0x00898b00L, "Game::DemoApp::CreateStrategyGame", "-> StrategyApp::CreateGame");
|
||||
fn(0x00888e80L, "Game::StrategyApp::CreateGame", "reads GameOptions (AIProcessMinTime, DefaultAutoRefuel), Data/Strategy/starcolors.txt, TurnCommands_v5 stream tag, builds the StrategyServer (ctor FUN_007d78d0) + StrategyServer::InitGame, loads (FUN_007dd530 'loaded from file') and the star-map UI (FUN_00778f40 -> ... -> StarMapPanelBase).");
|
||||
fn(0x007d78d0L, "Game::StrategyServer::StrategyServer", "ctor (vftables 0x00a26084 IStreamable @0, 0x00a26034 @4). Stores the OnEvent callback at +0x170 from a ctor argument (EBX).");
|
||||
fn(0x007c8d90L, "Game::StrategyServer::InitGame", "raises SEInitGame / SEAddDesign for every player, then SynchronizePlayer for each.");
|
||||
fn(0x007c6220L, "Game::StrategyServer::SynchronizePlayer", "(playerNetId, bool full, bool, byte): pushes the server-side view (systems/fleets/designs/notes/...) to one player's client via the OnEvent callback (SE* events). Logs 'Can't synchronize player %d(id)'. Called by SyncLocalClients, InitGame, GenerateTurnEvents.");
|
||||
fn(0x00815fd0L, "Game::StrategyApp::SyncLocalClients", "for each local client (+0xc vector): StrategyServer::SynchronizePlayer(client->+0x148 (player id), arg, 0, 0). 'SyncLocalClients() failed, no StrategyServer exists'.");
|
||||
fn(0x007dd530L, "Game::StrategyServer::LoadGame", "'loaded from file'");
|
||||
fn(0x008d2290L, "Mars::NetMessageRegistry::Register", "(this=registry entry {name@0,id@4,factory@8}, name, id, factory); g_NetMsgRegistryById[id] = entry (0x00b2ddf8, 256 slots). Called from static initialisers in .text 0x009be0cc..0x009c138c (unanalysed code). Entry(id) = 0x00b2bc94 + 12*id for the SNM* family.");
|
||||
lbl(0x00b2ddf8L, "g_NetMsgRegistryById", "NetMessageRegistry entry* [256], indexed by message id");
|
||||
lbl(0x00b2d540L, "g_pApplication", "Mars::Application* (the DemoApp)");
|
||||
lbl(0x00b2d0bcL, "g_pDemoApp", "Game::DemoApp* created in WinMain");
|
||||
lbl(0x00b2e61cL, "g_pNetworkManager", null); lbl(0x00b2e620L, "g_hNetworkWatchdogThread", null); lbl(0x00b2e5f8L, "g_netCS", "CRITICAL_SECTION guarding the network manager");
|
||||
lbl(0x00b2e4a8L, "g_musicCS", "CRITICAL_SECTION guarding the streaming music player"); lbl(0x00b2e4c4L, "g_pCurrentMusicStream", null);
|
||||
|
||||
log.println("\n==== FUNCTION RENAMES: turn spine ====");
|
||||
fn(0x00784640L, "Game::StrategyNetworkClient::OnMessage", "NETWORK MESSAGE DISPATCH (client AND host side; host checks +0x54 = StrategyServer*). Compares msg->GetType()->id with the registry entries: 0x3f SNMEndTurn -> host: StrategyServer::OnPlayerEndTurn + StorePlayerTurnCommands; 0x29 SNMUpdate (all players' TurnCommands) -> BeginProcessTurn, ApplyTurnCommands, ProcessTurn (deterministic local sim on EVERY machine), SyncLocalClients(1), state=5; 0x3c SNMDoEncounterQuery -> state=4; 0x2f SNMAllCombatDone -> ApplyEncounterResults (-> SETurnResults), SyncLocalClients(0), GenerateTurnEvents (-> SETurnEvents), autosave, state=6 'New turn begins'; 0x43 SNMResumePlaying -> StrategyServer::ResumePlaying (SEResumePlaying), reply SNMResumePlayingReceived, state=1; 0x3d SNMRunAI -> StrategyApp::RunAI; 0x32 SNMSetPlayerStatus -> player->Status(+0x164). " + DOC2);
|
||||
fn(0x00783be0L, "Game::StrategyClient::EndTurn", "CLIENT End Turn: marks +0x57/+0x132, records QPC time, EndTurnDelay, raises SETurnEndPending (event type 0x21) via RaiseEvent, then SendEndTurn (SNMEndTurn). Callers: UI (FUN_005e4f80, FUN_00579310).");
|
||||
fn(0x00783d30L, "Game::StrategyClient::EndTurnForced", "variant used by StrategyClient::Update (turn timer) and FUN_00783ee0; raises SETurnEndPending + SNMEndTurn.");
|
||||
fn(0x00783980L, "Game::StrategyClient::SendEndTurn", "builds SNMEndTurn (TurnCommands + AIEncounterFlags) and sends it to the host.");
|
||||
fn(0x007856f0L, "Game::StrategyClient::CancelEndTurn", "raises SETurnEndCancelled.");
|
||||
fn(0x007842b0L, "Game::StrategyClient::Update", "per-frame client update: strategy turn timer -> SETurnTimeExpired / EndTurnForced / SNMQueryEndTurnDone; combat join/launch bookkeeping ('All(%d) clients connected to combat server', 'Launching combat for encounter %d').");
|
||||
fn(0x00783ee0L, "Game::StrategyClient::RaiseEvent", "(int type, StrategyEvent** ev) -> client-side event sink (UI).");
|
||||
fn(0x0088c7b0L, "Game::SNMEndTurn::Create", "registry factory for message id 0x3f (object 0x1cc bytes: TurnCommands @+4, AIEncounterFlags @+0x1b8).");
|
||||
fn(0x007d9af0L, "Game::StrategyServer::OnPlayerEndTurn", "host: on SNMEndTurn from a player; if >1 human still playing and exactly one not done -> raise SELastPlaying (0x27) to that player via OnEventCallback.");
|
||||
fn(0x007893c0L, "Game::StrategyServer::StorePlayerTurnCommands", "host: keeps the player's TurnCommands until SNMUpdate is broadcast.");
|
||||
fn(0x00789710L, "Game::StrategyServer::BroadcastEvent", "(int eventType, StrategyEvent** ev): for every player in Players -> OnEventCallback(player->netId, type, ev). 'OnEvent() called, but no callback function specified' if NULL.");
|
||||
fn(0x007d98e0L, "Game::StrategyServer::BeginProcessTurn", "Frame(+0xc)++, bProcessingTurn(+0x2f1)=1, log 'Begin processing turn %d', clears per-system/per-fleet transient state (fleets: +0x114=+0x110; +0x110=0), BroadcastEvent(0x24 = SEProcessTurn). " + DOC2);
|
||||
fn(0x007b18b0L, "Game::StrategyServer::ApplyTurnCommands", "applies every player's TurnCommands from SNMUpdate ('set for turn processing'): fleet orders, builds, research, alliances (EVENT_ALLIANCE_*), diplomacy.");
|
||||
fn(0x007dc6c0L, "Game::StrategyServer::ProcessTurn", "TURN PROCESSING (arg = 1.0f time step). Phases in order: [1] per-system pre-turn (SESystemAbandoned / morale events); [2] FUN_0086b300 + FUN_007adc80 (alliance/diplomacy upkeep); [3] per-player pre-pass; [4] fleet snapshot (FUN_00794ad0/FUN_007b9b90), node-space travel (ProcessNodeSpaceTravel: EVENT_LOSTINNODESPACE_*), MOVEMENT ProcessFleetMovement -> MoveFleet -> SEFleetArrived; [5] per-fleet per-ship FUN_00814ea0 (ship upkeep); [6] per-system ServerSystem::ProcessTurn (pop/morale, BUILD queue -> BuildQueue::ProcessTurn -> SEBuildCompleted, plague, rebellion, slaves); [7] FUN_0078a7c0; [8] per-player ServerPlayer::ProcessTurn (income/savings, RESEARCH: TechTree::ProcessResearch, lab accidents, EVENT_NO_RESEARCH); [9] ProcessMissions, ProcessStations (EVENT_STATIONS_SCUTTLED), ProcessDefenceSats (EVENT_DEFSATS_SCUTTLED); [10] per-ship flag pass (FUN_00814da0 4 / 0x400000); [11] encounter detection (FUN_00794ad0 second snapshot -> local_98 != 0 means encounters pending); if NO encounters: ProcessAid (EVENT_GIVE_*), ProcessSpecialProjects (EVENT_SPRJTECHOFFER_STARTED), ProcessSurrenders (EVENT_PLAYER_SURRENDERED_/EVENT_SYSTEM_SURRENDERED), per-player FUN_00818530, FUN_0086a8d0, FUN_0078ab30, FUN_00799380, FUN_0078aa70, per-system FUN_00743ec0, FUN_007b4c00, FUN_007d7f70 (end-of-turn bookkeeping over the per-player 0x74-byte records). Turn RESULTS/EVENTS are raised later from the SNMAllCombatDone handler (ApplyEncounterResults -> SETurnResults; GenerateTurnEvents -> SETurnEvents). " + DOC2);
|
||||
fn(0x007da9a0L, "Game::StrategyServer::ProcessFleetMovement", "iterates fleets with flight plans, calls MoveFleet(fleet, dt) (several passes: normal, in-transit, arrival), then OnFleetArrived (EVENT_FLEET_ARRIVED).");
|
||||
fn(0x007d9ee0L, "Game::StrategyServer::MoveFleet", "(StarFleet*, float dt): advances a fleet along its FlightPlan; on arrival raises SEFleetArrived; 'Destination of fleet doesn't exist. Stopping fleet.'; cancels ship actions on departure.");
|
||||
fn(0x007ccb10L, "Game::StrategyServer::OnFleetArrived", "EVENT_FLEET_ARRIVED bookkeeping");
|
||||
fn(0x007a0e20L, "Game::StrategyServer::ProcessNodeSpaceTravel", "EVENT_FLEET_MULTIPOINT_NONODE / EVENT_LOSTINNODESPACE_NOBORE / _ENGINES");
|
||||
fn(0x007598e0L, "Game::ServerSystem::ProcessTurn", "per-system turn: population/infrastructure growth, morale (FUN_00752a10), ProcessPlague, ProcessBuildQueue, terraforming/resources, slaves (ProcessSlaves, EVENT_SLAVES_DEAD), rebellion (ProcessRebellion, EVENT_SYSTEM_REBELLION_CONTINUES).");
|
||||
fn(0x00752500L, "Game::ServerSystem::ProcessBuildQueue", "-> BuildQueue::ProcessTurn");
|
||||
fn(0x00890d50L, "Game::BuildQueue::ProcessTurn", "advances ShipBuildOrders; completed orders -> SEBuildCompleted (with ShipBuildOrderDef). Callers: ServerSystem::ProcessBuildQueue, FUN_00789500 (ship-borne build queues).");
|
||||
fn(0x00756a90L, "Game::ServerSystem::ProcessPlague", "EVENT_PLAGUE_OUTBREAK / EVENT_COLONY_DESTROYEDBYPLAGUE / EVENT_PLAGUE_CURED");
|
||||
fn(0x007583b0L, "Game::ServerSystem::ProcessRebellion", "EVENT_SYSTEM_REBELLION_CONTINUES");
|
||||
fn(0x007537b0L, "Game::ServerSystem::ProcessSlaves", "EVENT_SLAVES_DEAD");
|
||||
fn(0x00891340L, "Game::ServerPlayer::ProcessTurn", "per-player turn: savings/income (FUN_00840fe0), research: if ResT set -> RollResearchAccident then TechTree::ProcessResearch (EVENT_RESEARCH_OVERBUDGET / EVENT_TECHS_UNLOCKED) else EVENT_NO_RESEARCH; special projects (FUN_00863cf0).");
|
||||
fn(0x005876c0L, "Game::TechTree::ProcessResearch", "EVENT_RESEARCH_OVERBUDGET / EVENT_TECHS_UNLOCKED");
|
||||
fn(0x00889dc0L, "Game::ServerPlayer::RollResearchAccident", "'ACCIDENT!!' / 'All okay.' EVENT_LABACCIDENT_SMALL/MEDIUM/LARGE");
|
||||
fn(0x007ad100L, "Game::StrategyServer::ProcessAid", "EVENT_GIVE_SAVINGS / EVENT_GIVE_RESEARCH");
|
||||
fn(0x007a3310L, "Game::StrategyServer::ProcessSpecialProjects", "EVENT_SPRJTECHOFFER_STARTED");
|
||||
fn(0x007d0d10L, "Game::StrategyServer::ProcessSurrenders", "EVENT_PLAYER_SURRENDERED_ / EVENT_SYSTEM_SURRENDERED");
|
||||
fn(0x007af0b0L, "Game::StrategyServer::ProcessDefenceSats", "EVENT_DEFSATS_SCUTTLED");
|
||||
fn(0x007ae480L, "Game::StrategyServer::ProcessStations", "EVENT_STATIONS_SCUTTLED");
|
||||
fn(0x007999a0L, "Game::StrategyServer::ProcessMissions", "'mission'");
|
||||
fn(0x007cbe80L, "Game::StrategyNetworkServer::RunCombatRound", "host combat round: for each pending encounter -> SendEncounterQuery (SNMDoEncounterQuery) / 'Notifying %s to host encounter %d' (SNMHostCombat) ...; when all encounters resolved -> SNMAllCombatDone 'All combat complete, waiting for clients to process results'.");
|
||||
fn(0x007cda40L, "Game::StrategyNetworkServer::Update", "host per-frame: lobby/slot messages, combat round driving (RunCombatRound), SNMTurnInfo, SNMLaunchCombat, SNMResumePlaying (SendResumePlaying).");
|
||||
fn(0x007bfe60L, "Game::StrategyNetworkServer::SendEncounterQuery", "builds SNMDoEncounterQuery");
|
||||
fn(0x00794770L, "Game::StrategyNetworkServer::SendResumePlaying", "builds SNMResumePlaying");
|
||||
fn(0x007d4400L, "Game::StrategyServer::ApplyEncounterResults", "after SNMAllCombatDone: applies combat outcomes (EVENT_STATION_ENABLED/DISABLED ...) then DispatchTurnResults -> SETurnResults per player.");
|
||||
fn(0x007cd2a0L, "Game::StrategyServer::DispatchTurnResults", "-> SendTurnResultsToPlayers");
|
||||
fn(0x007c5850L, "Game::StrategyServer::SendTurnResultsToPlayers", "(perPlayerResults[0x11c stride], n): per player: SETurnResults::Create, fill (FUN_007c24d0), dispatch (FUN_0079ac10).");
|
||||
fn(0x007a7ae0L, "Game::SETurnResults::Create", "");
|
||||
fn(0x007dc640L, "Game::StrategyServer::GenerateTurnEvents", "-> BuildTurnEvents (SETurnEvents per player) + FUN_00792a20/FUN_007c5610.");
|
||||
fn(0x007db780L, "Game::StrategyServer::BuildTurnEvents", "builds the per-player SETurnEvents (EventStorage::TurnEvents) from the turn's event log; also SEResetMap/SEAddPlayer/SEInitTrade/SESyncDesign; calls SynchronizePlayer.");
|
||||
fn(0x007ddc90L, "Game::StrategyServer::ResumePlaying", "on SNMResumePlaying: for every player with Status(+0x164)==0 -> OnEventCallback(netId, 0x26 = SEResumePlaying).");
|
||||
fn(0x008706f0L, "Game::StrategyApp::RunAI", "on SNMRunAI: 'RunAI: No StrategyServer created' guard; runs the AI turn (SEAIPrepareTurn via FUN_00815f20) for AI players.");
|
||||
fn(0x00815f20L, "Game::StrategyApp::RaiseAIPrepareTurn", "raises SEAIPrepareTurn");
|
||||
setEOLComment(toAddr(0x0090242dL), "CreateThread -> Mars::NetworkManager::WatchdogThreadProc (thread 1: network timeout watchdog)");
|
||||
setEOLComment(toAddr(0x008a14efL), "CreateThread -> Mars::SoundSystem::StreamingUpdateThreadProc (thread 2: music/DirectSound streaming, priority 15)");
|
||||
setEOLComment(toAddr(0x00736e84L), "CreateThread -> Game::BackgroundWorker::ThreadProc (thread 3: star-map political-blob mesh builder)");
|
||||
|
||||
log.println("\n==== NET MESSAGE REGISTRY LABELS ====");
|
||||
Address lo = toAddr(0x009b0000L), hi = toAddr(0x009c8000L); byte[] buf = new byte[(int)(hi.subtract(lo))]; mem.getBytes(lo, buf); int n = 0;
|
||||
for (int i = 0; i + 10 < buf.length; i++) {
|
||||
if ((buf[i]&0xff) != 0xB9 || (buf[i+5]&0xff) != 0xE8) continue;
|
||||
long rel = (buf[i+6]&0xffL) | ((buf[i+7]&0xffL)<<8) | ((buf[i+8]&0xffL)<<16) | ((buf[i+9]&0xffL)<<24);
|
||||
if (lo.getOffset() + i + 10 + (int)rel != 0x008d2290L) continue;
|
||||
long entry = (buf[i+1]&0xffL) | ((buf[i+2]&0xffL)<<8) | ((buf[i+3]&0xffL)<<16) | ((buf[i+4]&0xffL)<<24);
|
||||
int p = i - 5; if ((buf[p]&0xff) != 0x68) continue;
|
||||
long name = (buf[p+1]&0xffL) | ((buf[p+2]&0xffL)<<8) | ((buf[p+3]&0xffL)<<16) | ((buf[p+4]&0xffL)<<24);
|
||||
long id; int q; if ((buf[p-2]&0xff) == 0x6a) { id = buf[p-1]&0xff; q = p-2; } else if ((buf[p-5]&0xff) == 0x68) { id = (buf[p-4]&0xffL) | ((buf[p-3]&0xffL)<<8) | ((buf[p-2]&0xffL)<<16) | ((buf[p-1]&0xffL)<<24); q = p-5; } else continue;
|
||||
long factory = (buf[q-5]&0xff) == 0x68 ? ((buf[q-4]&0xffL) | ((buf[q-3]&0xffL)<<8) | ((buf[q-2]&0xffL)<<16) | ((buf[q-1]&0xffL)<<24)) : 0;
|
||||
String nm = cstr(toAddr(name), 64); if (nm == null) continue;
|
||||
try { createLabel(toAddr(entry), "NetMsgReg_" + nm, true, SourceType.USER_DEFINED); setEOLComment(toAddr(entry), String.format("NetMessageRegistry entry: name=%s id=0x%02x factory=0x%08x (registered at 0x%08x)", nm, id, factory, lo.getOffset() + i)); } catch (Exception e) {}
|
||||
Function ff = getFunctionAt(toAddr(factory));
|
||||
if (ff != null && ff.getName().startsWith("FUN_")) { try { ff.setParentNamespace(ns("Game::" + nm)); ff.setName("Create", SourceType.USER_DEFINED); ff.setComment(String.format("NetMessage factory for %s (id 0x%02x); registered via Mars::NetMessageRegistry::Register", nm, id)); } catch (Exception e) { log.println(" !! factory " + nm + ": " + e.getMessage()); } }
|
||||
n++;
|
||||
}
|
||||
log.println(" labelled " + n + " registry entries");
|
||||
|
||||
log.println("\n==== VERIFY ====");
|
||||
decomp = new DecompInterface(); decomp.openProgram(currentProgram);
|
||||
retypeThis(0x00749630L, new PointerDataType(sysV)); retypeThis(0x0075d4b0L, new PointerDataType(sysV));
|
||||
retypeThis(0x00701070L, new PointerDataType(fltV)); retypeThis(0x00702470L, new PointerDataType(fltV));
|
||||
retypeThis(0x008291f0L, new PointerDataType(shpV)); retypeThis(0x00853fa0L, new PointerDataType(shpV));
|
||||
retypeThis(0x0079fa70L, new PointerDataType(srv)); retypeThis(0x007d27a0L, new PointerDataType(srv));
|
||||
retypeThis(0x007d98e0L, new PointerDataType(srv)); retypeThis(0x007dc6c0L, new PointerDataType(srv)); retypeThis(0x00789710L, new PointerDataType(srv)); retypeThis(0x007d9af0L, new PointerDataType(srv)); retypeThis(0x007598e0L, new PointerDataType(sys)); retypeThis(0x00891340L, new PointerDataType(ply));
|
||||
verify(0x00749630L, "verify_ServerSystem_Write.c"); verify(0x0079fa70L, "verify_StrategyServer_Write.c"); verify(0x00701070L, "verify_StarFleet_Write.c"); verify(0x007d98e0L, "verify_BeginProcessTurn.c"); verify(0x007dc6c0L, "verify_ProcessTurn.c");
|
||||
decomp.dispose(); log.close(); println("writeback done");
|
||||
}
|
||||
}
|
||||
169
save-reader/SAVE_FORMAT.md
Normal file
169
save-reader/SAVE_FORMAT.md
Normal file
|
|
@ -0,0 +1,169 @@
|
|||
# SOTS1 save format — as implemented by `save_reader.py`
|
||||
|
||||
Status: **unvalidated against a real save.** Everything below is what the
|
||||
reader *assumes*, derived from the community editors (R1 Bardez, R2 SOTSedit;
|
||||
`save-editor-structs.md`) corrected by the binary member tables
|
||||
(`struct-recovery.md`). The synthetic fixture (`save_writer_stub.py`) round-trips
|
||||
under these assumptions; a real `.sav` is the first real test. Items marked
|
||||
**VERIFY** are the ones the verifier should diff against the binary first.
|
||||
|
||||
## 1. Container
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| File | one gzip member (`1f 8b`), standard deflate. `gzip.decompress` |
|
||||
| Inflated stream | parsed from offset 0; all offsets in reader output are inflated offsets |
|
||||
| Byte order | little-endian everywhere |
|
||||
| Text | windows-1252, no NUL terminator |
|
||||
| Non-gzip input | accepted as an already-inflated stream (R1's `*.sav.inflate.dat`) |
|
||||
|
||||
## 2. Primitive encodings
|
||||
|
||||
```
|
||||
tag := int32 len, len bytes ASCII (field name; may be "." or "")
|
||||
string := int32 len, len bytes cp1252
|
||||
int := 4 bytes (int32) handle ids, int16 members, enums, counts
|
||||
float := 4 bytes IEEE-754 single
|
||||
bool := 1 byte, 0/1
|
||||
int64 := 8 bytes Bats2, rcex, PopC (raw-bytes write path)
|
||||
raw := n bytes, length known only to the writer RNG state (2500 B = MT19937 624 words + index)
|
||||
```
|
||||
|
||||
A **named value** is `tag value pad`, where `pad` is NUL bytes bringing the item
|
||||
to a 4-byte boundary. The reader implements two padding conventions and
|
||||
auto-detects which one the file uses (`--padding auto`, default):
|
||||
|
||||
| mode | layout | item size |
|
||||
|---|---|---|
|
||||
| `joint` (default, R1's `PaddingSize=4` over `4+len(name)+len(value)`) | `[len][name][value][pad]` | `pad4(4 + len(name) + len(value))` |
|
||||
| `split` | `[len][name][pad][value][pad]` | `pad4(4 + len(name)) + pad4(len(value))` |
|
||||
|
||||
The two differ only when `len(name) % 4 != 0` **and** the value is not a
|
||||
multiple of 4 bytes (bools, odd-length strings): e.g. `NPC`+bool is 8 bytes in
|
||||
`joint`, 12 in `split`. **VERIFY (highest priority):** find a 3-char bool tag
|
||||
(`NPC`, `Dep`, `Atq`, `Srn`, `cta`, `hgs`, `vnh`, `hbq`, `hsp`) in the real
|
||||
inflated stream and check whether the value byte immediately follows the name.
|
||||
Reader assumption when writing this: `joint`.
|
||||
|
||||
Values carry **no type byte**. Type comes from (a) the schema/catalog, (b) a
|
||||
lookahead plausibility test (a candidate layout is accepted only if a plausible
|
||||
tag, a frame marker, or EOF follows), (c) for 4-byte words with no hint, an
|
||||
int/float classification by bit pattern (`|int| <= 100000` → int; finite float
|
||||
with `1e-6 <= |f| < 1e12` → float; else int). The other reading is kept as `alt`.
|
||||
Limitation: with a name whose length is a multiple of 4, `bool` and `int` have
|
||||
identical item sizes; unknown names default to `int`.
|
||||
|
||||
## 3. Complex frames
|
||||
|
||||
```
|
||||
[len][name][pad to 4] BE EF BE EF ...children... 10 41 10 41
|
||||
(0xBEEFBEEF) (0x41104110 = ~0xBEEFBEEF)
|
||||
```
|
||||
|
||||
* Every "nested object" (IStreamable via stream vft+0x28) is framed. Frames
|
||||
nest; the reader recovers the tree from the markers alone, without a schema.
|
||||
* A frame may be **tagless** (BEEFBEEF directly at an item position): the reader
|
||||
accepts it; not observed in the reference material. **VERIFY** if seen.
|
||||
* A frame body may be **untagged bytes**: `Mars::Vector3` (`Pos`, `PrvPos`,
|
||||
`FtOrig`, `FPogn2`, `FPdpos`) and `ShipHealth` (`Health`) are "3 unnamed
|
||||
floats" per the binary. Reader accepts either 12 raw bytes before the END
|
||||
marker or three tagged floats (any tag, e.g. `"."`). **VERIFY** which.
|
||||
* `RNG` frame body is treated as opaque bytes up to its END marker.
|
||||
* Markers are only interpreted at item boundaries; a value that happens to equal
|
||||
a marker is not a problem unless the reader is already resynchronising.
|
||||
|
||||
## 4. Arrays
|
||||
|
||||
| kind | encoding | reader schema |
|
||||
|---|---|---|
|
||||
| NonComplexArray (`VectorHelper`, `std::list`) | named int count, then count × element **inline in the same frame** | `NArr` |
|
||||
| ComplexArray | a frame containing: int count, then count × element | `CArr` |
|
||||
| element = leaf wrapper | e.g. `SysID` int + `Sys` frame; `PlayerID` + `Player`; `FltID` + `Flt`; `ShipID` + `Ship`; `DesID` + `Des` | `Seq([...])` |
|
||||
| sparse tables | `mnsp` then n × (`msp` index, `mv` value); `nadct` then n × (`ads`, `adt`); `haltc` then n × (`haltt`, `haltv`) | `NArr(Seq)` |
|
||||
|
||||
Count tags used by the binary: `NumPlrs NumSys NumFlts NShips NumActs NumOwn NumDes
|
||||
NumLeg NumNotes NumPR NSprj Nexp NWeapXcl ndeflay rdtc numcreps ninv NumFlts NumGFs
|
||||
NumSnF NumMnF NVO NVE NVs NumPlgs2 PopNG mnsp nadct haltc PrNSp NTH`. Count tags
|
||||
inside framed arrays (`dipstats`, `preps`, `odes`, `owep`, `otch`, `cme2`, `Ojvs`…)
|
||||
are unknown — the reader takes the first child as the count whatever its name.
|
||||
Element frame tags are unknown (R1 suggests `"."`) — matched by position.
|
||||
|
||||
## 5. Conditionals the reader honours
|
||||
|
||||
| gate | consequence |
|
||||
|---|---|
|
||||
| `ClrID`/`indcl`/`fxCrId` frame: int index == -1 | three ints r,g,b follow inside the frame |
|
||||
| `vnh` true | `vnd`, `vnex3`, `vnpex3` |
|
||||
| `hindi` true | `indi` frame (IndependenceInfo) |
|
||||
| `NVO` entry `isind` true | `indi` frame |
|
||||
| `HFPlan` true | `FPlan` frame |
|
||||
| `HLay` true | `Lay` frame (opaque) |
|
||||
| `hbq` true | `BQ2` frame; `hsp` true → `pop`, `ppop` Population frames |
|
||||
| `HasAIR` true | `AIR` frame (opaque) |
|
||||
| system `PID` non-null | `BQ` frame present (reader: optional by name) |
|
||||
| legacy tags `ISuit ARes SysID TrdID Caps GtTrf FtSens FtInc lcid SensMod ExPopSys AIDifficultyID` | accepted if present, expected absent in 1.8 saves |
|
||||
|
||||
## 6. Top-level layout
|
||||
|
||||
```
|
||||
offset 0: [7]"Summary" [pad] BEEFBEEF ... 41104110 (R2 confirms the tag "Summary")
|
||||
CreateParameters frame (tag unknown)
|
||||
Sim frame (tag unknown) KeyPath NMSz NMLc NMnx <id lists> ModCount Frame
|
||||
GameID Attrib RNG GameName Map IncMod ResMod EnAl
|
||||
EnTm GOTurn GOWinPly NPCm NPCo NPCi NPCv NPCa
|
||||
szadj rsadj suadj sprjs RandEncAdj cmbtid turnstats
|
||||
numcreps/crep ninv <invs.. AllExc..> NumPlrs/PlayerID/Player
|
||||
<ISsp ISsu> NumSys/SysID/Sys NdGr2 trdmgr spymgr
|
||||
NumFlts/FltID/Flt NumActs/Act SvSctOb zdsc zdsi zdst
|
||||
CdTable UNFRAMED at root: cdt frame, cdplayer frame, N × cdai frames
|
||||
```
|
||||
|
||||
Regions in `<...>` are parsed generically and kept as raw item lists
|
||||
(`sim.idLists`, `sim.invasionsAndExclusions`, `sim.species`). Everything not
|
||||
covered by a shape (TechTree body, Events, ShipRecs, spy2, civr, comms, Ojvs,
|
||||
SvSctOb, trdmgr, spymgr, CdTable, …) is kept as the generic
|
||||
`{"_name","_off","_items":[{"name","kind","value","off"}...]}` form.
|
||||
|
||||
## 7. Struct field orders and types applied
|
||||
|
||||
Shapes live in `save_reader.py` (`Sys`, `Player`, `Fleet`, `Ship`, `PlayerView`,
|
||||
`Population`/`PopG`, `Morale`, `MoraleEvent`, `BuildQueue`/`BuildOrder`,
|
||||
`IndependenceInfo`, `Rts`, `DipStat`, `Prep`, `FlightPlan`, `Waypoint`,
|
||||
`PrisonerHold`, `Summary`, `CreateParams`, …). `A("tag", type)` = on-disk tag
|
||||
confirmed in the exe (strict name match); `R("name", type)` = R1 C# name only
|
||||
(positional). Binary corrections applied over R1:
|
||||
|
||||
* float: `TRM CstR CstE CstT shrm RefCap RepCap`, PlayerView `Infra`
|
||||
* int64: `Bats2 rcex PopC`
|
||||
* bool: `Abdn Dstyd PvMA AIBn` (R2's "short" readings are the value byte)
|
||||
* int on disk though int16 in memory: `TShn ETS` and all DiplomacyStats counters
|
||||
* string: `pswd`
|
||||
* Vector3 (3 floats): `FtOrig`
|
||||
* `Nexp` entries carry `xid xmin xmax xper(float)`
|
||||
* `Team` appears twice in Player: an int, later a nested `{ALid AL NA CF}` frame
|
||||
(typed key `Alliances`)
|
||||
|
||||
## 8. Reader output conventions
|
||||
|
||||
* `--dump`: one line per item, `@<inflated offset> name kind value`; `?` after
|
||||
the kind means the type was guessed, `(alt …)` shows the other reading of a
|
||||
4-byte word; frames print `{` … `}` with item count and byte size.
|
||||
* `--json`: `{"padding", "stats", "issues": [...], "data": {...}}`; every typed
|
||||
struct carries `_off`; unexpected items are kept under `_unexpected` /
|
||||
`_extra`; unknown regions keep the generic form.
|
||||
* Issue levels: `error` (schema field missing / type impossible / frame
|
||||
unterminated), `warn` (resync, hint not plausible, unexpected items,
|
||||
width mismatch, best-effort read at EOF), `info` (unnamed small payload
|
||||
before END, positional tag name differing from the R1 name). `--strict`
|
||||
fails on error or warn.
|
||||
* Exit status: 0 clean, 1 errors present, 2 unreadable container.
|
||||
|
||||
## 9. What the verifier should check first on a real save
|
||||
|
||||
1. Padding convention (§2) — a 3-char bool tag settles it.
|
||||
2. The `Summary` frame's 13 children and their actual tag names (R1 names are
|
||||
probably case-variants: `NumSys` is confirmed by R2 as a tag string).
|
||||
3. Whether Vector3 bodies are tagged (§3).
|
||||
4. The tag used for element frames and for counts inside framed arrays (§4).
|
||||
5. Whether the reader's auto-detected padding, `resyncs == 0` and
|
||||
`hint_failures == 0` hold; any resync offset points at a layout gap.
|
||||
BIN
save-reader/__pycache__/save_reader.cpython-310.pyc
Normal file
BIN
save-reader/__pycache__/save_reader.cpython-310.pyc
Normal file
Binary file not shown.
BIN
save-reader/__pycache__/save_writer_stub.cpython-310.pyc
Normal file
BIN
save-reader/__pycache__/save_writer_stub.cpython-310.pyc
Normal file
Binary file not shown.
BIN
save-reader/__pycache__/test_save_reader.cpython-310.pyc
Normal file
BIN
save-reader/__pycache__/test_save_reader.cpython-310.pyc
Normal file
Binary file not shown.
BIN
save-reader/fixture_joint.sav
Normal file
BIN
save-reader/fixture_joint.sav
Normal file
Binary file not shown.
BIN
save-reader/fixture_split.sav
Normal file
BIN
save-reader/fixture_split.sav
Normal file
Binary file not shown.
BIN
save-reader/ref/parsers/__pycache__/effect_txt.cpython-310.pyc
Normal file
BIN
save-reader/ref/parsers/__pycache__/effect_txt.cpython-310.pyc
Normal file
Binary file not shown.
BIN
save-reader/ref/parsers/__pycache__/flat_kv.cpython-310.pyc
Normal file
BIN
save-reader/ref/parsers/__pycache__/flat_kv.cpython-310.pyc
Normal file
Binary file not shown.
BIN
save-reader/ref/parsers/__pycache__/manifest.cpython-310.pyc
Normal file
BIN
save-reader/ref/parsers/__pycache__/manifest.cpython-310.pyc
Normal file
Binary file not shown.
BIN
save-reader/ref/parsers/__pycache__/mars_data.cpython-310.pyc
Normal file
BIN
save-reader/ref/parsers/__pycache__/mars_data.cpython-310.pyc
Normal file
Binary file not shown.
98
save-reader/ref/parsers/effect_txt.py
Normal file
98
save-reader/ref/parsers/effect_txt.py
Normal file
|
|
@ -0,0 +1,98 @@
|
|||
"""effect_txt.py -- reader for Effects/*.effect (particle-effect definitions).
|
||||
|
||||
NOT the brace-block format. Layout:
|
||||
|
||||
TXT # magic first line
|
||||
KEY value # scalar (number, TRUE/FALSE, "quoted")
|
||||
KEY # group: KEY on its own line, then
|
||||
BEGIN
|
||||
...nested KEY value / groups...
|
||||
END
|
||||
|
||||
Order matters: 'PARTICLEDATATYPE n' is followed by the CREATION /
|
||||
VARIATION / OVERLIFE curves that belong to that datatype, and 'MODIFIER'
|
||||
repeats once per type. So each level is returned as an ordered list of
|
||||
[key, value] pairs (value = scalar or nested list). to_dict() gives a
|
||||
dict view (repeats -> lists) when order is not needed.
|
||||
|
||||
Quirks handled: one file has CRLF; 'NAME "New Emitter"' values contain
|
||||
spaces; indentation is cosmetic (tabs); the format is line-based.
|
||||
|
||||
Stdlib only.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from flat_kv import split_tokens, strip_comment
|
||||
from mars_data import coerce
|
||||
|
||||
__all__ = ["parse", "parse_file", "to_dict", "EffectSyntaxError"]
|
||||
|
||||
Pairs = list # list[[key, value]]
|
||||
|
||||
|
||||
class EffectSyntaxError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
def parse(text: str, *, typed: bool = True) -> Pairs:
|
||||
lines = text.splitlines()
|
||||
if not lines or lines[0].strip() != "TXT":
|
||||
raise EffectSyntaxError("missing TXT magic")
|
||||
stack: list[Pairs] = [[]]
|
||||
pending_key: str | None = None
|
||||
for lineno, raw in enumerate(lines[1:], 2):
|
||||
line = strip_comment(raw).strip()
|
||||
if not line:
|
||||
continue
|
||||
if line == "BEGIN":
|
||||
if pending_key is None:
|
||||
raise EffectSyntaxError(f"line {lineno}: BEGIN without a key")
|
||||
grp: Pairs = []
|
||||
stack[-1].append([pending_key, grp])
|
||||
stack.append(grp)
|
||||
pending_key = None
|
||||
continue
|
||||
if line == "END":
|
||||
if len(stack) == 1:
|
||||
raise EffectSyntaxError(f"line {lineno}: END without BEGIN")
|
||||
stack.pop()
|
||||
continue
|
||||
if pending_key is not None:
|
||||
raise EffectSyntaxError(f"line {lineno}: key {pending_key!r} not followed by BEGIN")
|
||||
toks = split_tokens(line)
|
||||
key = toks[0][0]
|
||||
if len(toks) == 1:
|
||||
pending_key = key
|
||||
continue
|
||||
vals = [coerce(t) if (typed and not q) else t for t, q in toks[1:]]
|
||||
stack[-1].append([key, vals[0] if len(vals) == 1 else vals])
|
||||
if len(stack) != 1:
|
||||
raise EffectSyntaxError(f"{len(stack) - 1} unclosed BEGIN group(s)")
|
||||
if pending_key is not None:
|
||||
raise EffectSyntaxError(f"trailing key {pending_key!r} without BEGIN")
|
||||
return stack[0]
|
||||
|
||||
|
||||
def to_dict(pairs: Pairs) -> dict:
|
||||
d: dict = {}
|
||||
for key, val in pairs:
|
||||
if isinstance(val, list) and val and isinstance(val[0], list) and len(val[0]) == 2 and isinstance(val[0][0], str):
|
||||
val = to_dict(val)
|
||||
if key in d:
|
||||
if not isinstance(d[key], list) or not getattr(d[key], "_rep", False):
|
||||
d[key] = _Rep([d[key]])
|
||||
d[key].append(val)
|
||||
else:
|
||||
d[key] = val
|
||||
return d
|
||||
|
||||
|
||||
class _Rep(list):
|
||||
_rep = True
|
||||
|
||||
|
||||
def parse_file(path, **kw) -> Pairs:
|
||||
with open(path, "rb") as f:
|
||||
return parse(f.read().decode("cp1252"), **kw)
|
||||
127
save-reader/ref/parsers/flat_kv.py
Normal file
127
save-reader/ref/parsers/flat_kv.py
Normal file
|
|
@ -0,0 +1,127 @@
|
|||
"""flat_kv.py -- readers for the flat 'KEY value' tuning tables and the
|
||||
whitespace-positional tables under Data/, Weapons/, Badges/, Avatars/, GUI/.
|
||||
|
||||
Two shapes exist:
|
||||
|
||||
parse_kv(text) KEY value -> {KEY: value}
|
||||
one constant per line; value is a bareword or a
|
||||
"quoted string"; '//' comments; colors are quoted
|
||||
"r g b" (use color()). Files: Data/globals.txt,
|
||||
Data/species.txt, Data/Strategy/StrategyVars.txt,
|
||||
Data/Combat/*.txt (most), Data/encounters.txt, ...
|
||||
|
||||
parse_rows(text) tok tok tok ... -> [[tok, ...], ...]
|
||||
one record per line, whitespace separated, quoted
|
||||
tokens may contain spaces; '//' comments. Files:
|
||||
Weapons/_turrets.txt, Weapons/_defaultweapons.txt,
|
||||
Data/Combat/damfx*.txt, Data/Strategy/playercolors.txt,
|
||||
Badges/BadgeTable.txt, Avatars/AvatarTable.txt,
|
||||
GUI/WeaponIconPlacements.txt
|
||||
|
||||
Quirks handled: '//' inside a quoted value is not a comment; a quoted
|
||||
value may be empty (""); keys repeat in a few files (kept as list);
|
||||
duplicate-key detection is exposed via parse_kv(..., on_dup=).
|
||||
|
||||
Stdlib only.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
from mars_data import coerce
|
||||
|
||||
__all__ = ["parse_kv", "parse_rows", "duplicates", "color", "strip_comment",
|
||||
"split_tokens", "parse_kv_file", "parse_rows_file"]
|
||||
|
||||
_TOK_RE = re.compile(r'"([^"]*)"|(\S+)')
|
||||
|
||||
|
||||
def strip_comment(line: str) -> str:
|
||||
"""Remove a trailing // comment, ignoring // inside double quotes."""
|
||||
in_q = False
|
||||
i = 0
|
||||
n = len(line)
|
||||
while i < n:
|
||||
c = line[i]
|
||||
if c == '"':
|
||||
in_q = not in_q
|
||||
elif c == "/" and not in_q and line.startswith("//", i):
|
||||
return line[:i]
|
||||
i += 1
|
||||
return line
|
||||
|
||||
|
||||
def split_tokens(line: str) -> list[tuple[str, bool]]:
|
||||
"""Split a line into (token, was_quoted) pairs."""
|
||||
out = []
|
||||
for m in _TOK_RE.finditer(line):
|
||||
if m.group(1) is not None:
|
||||
out.append((m.group(1), True))
|
||||
else:
|
||||
out.append((m.group(2), False))
|
||||
return out
|
||||
|
||||
|
||||
def parse_rows(text: str, *, typed: bool = True) -> list[list[Any]]:
|
||||
rows = []
|
||||
for raw in text.splitlines():
|
||||
line = strip_comment(raw).strip()
|
||||
if not line:
|
||||
continue
|
||||
toks = split_tokens(line)
|
||||
rows.append([coerce(t) if (typed and not q) else t for t, q in toks])
|
||||
return rows
|
||||
|
||||
|
||||
def _pairs(text: str, typed: bool):
|
||||
for lineno, raw in enumerate(text.splitlines(), 1):
|
||||
line = strip_comment(raw).strip()
|
||||
if not line:
|
||||
continue
|
||||
toks = split_tokens(line)
|
||||
key = toks[0][0]
|
||||
vals = [coerce(t) if (typed and not q) else t for t, q in toks[1:]]
|
||||
val: Any = None if not vals else (vals[0] if len(vals) == 1 else vals)
|
||||
yield lineno, key, val
|
||||
|
||||
|
||||
def parse_kv(text: str, *, typed: bool = True, on_dup: str = "last") -> dict:
|
||||
"""KEY value per line -> dict. A value made of several unquoted tokens
|
||||
is kept as a list. on_dup: 'last' (later line wins), 'first', 'error'.
|
||||
Use duplicates() to find repeated keys."""
|
||||
d: dict = {}
|
||||
for lineno, key, val in _pairs(text, typed):
|
||||
if key in d:
|
||||
if on_dup == "error":
|
||||
raise ValueError(f"line {lineno}: duplicate key {key}")
|
||||
if on_dup == "first":
|
||||
continue
|
||||
d[key] = val
|
||||
return d
|
||||
|
||||
|
||||
def duplicates(text: str) -> dict[str, list[int]]:
|
||||
"""key -> line numbers, for keys that appear more than once."""
|
||||
seen: dict[str, list[int]] = {}
|
||||
for lineno, key, _ in _pairs(text, False):
|
||||
seen.setdefault(key, []).append(lineno)
|
||||
return {k: v for k, v in seen.items() if len(v) > 1}
|
||||
|
||||
|
||||
def color(value: str) -> tuple:
|
||||
"""'r g b' or 'r g b a' -> tuple of numbers."""
|
||||
return tuple(coerce(t) for t in value.split())
|
||||
|
||||
|
||||
def _read(path) -> str:
|
||||
with open(path, "rb") as f:
|
||||
return f.read().decode("cp1252")
|
||||
|
||||
|
||||
def parse_kv_file(path, **kw) -> dict:
|
||||
return parse_kv(_read(path), **kw)
|
||||
|
||||
|
||||
def parse_rows_file(path, **kw) -> list:
|
||||
return parse_rows(_read(path), **kw)
|
||||
121
save-reader/ref/parsers/manifest.py
Normal file
121
save-reader/ref/parsers/manifest.py
Normal file
|
|
@ -0,0 +1,121 @@
|
|||
"""manifest.py -- the numbered id manifests and the '#'-commented CSVs.
|
||||
|
||||
parse_manifest(text) -> Manifest
|
||||
Weapons/_weapons.txt and Species/<Race>/sections/_shipsections.txt:
|
||||
<int-id> <filename> one per line
|
||||
// DELETED - <id> retired id (still reserved)
|
||||
Ids are the persistent network / savegame ids. Filenames are matched
|
||||
case-insensitively (the shipped manifests have 'DEWar.SHIPSECTION',
|
||||
'CRAIC.Shipsection' etc. against lower-case files -- Windows FS).
|
||||
|
||||
parse_csv(text) -> list[list[str]]
|
||||
Rows with '#' or '//' as first non-blank char are comments; blank rows
|
||||
dropped; RFC-4180 quoting honoured (Strings.csv has one multi-line cell
|
||||
and quoted commas). Header rows that start with '#' (aitechpri.csv,
|
||||
"# species" in stock_diplomacy_messages.csv) are returned separately
|
||||
via parse_csv_with_header().
|
||||
|
||||
Stdlib only.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import io
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
__all__ = ["Manifest", "parse_manifest", "parse_manifest_file",
|
||||
"parse_csv", "parse_csv_file", "parse_csv_with_header", "read_text"]
|
||||
|
||||
_DELETED_RE = re.compile(r"//\s*DELETED\s*-\s*(\d+)", re.I)
|
||||
_ENTRY_RE = re.compile(r"^\s*(\d+)\s+(\S+)\s*$")
|
||||
|
||||
|
||||
@dataclass
|
||||
class Manifest:
|
||||
entries: list[tuple[int, str]] = field(default_factory=list) # (id, filename)
|
||||
deleted: list[int] = field(default_factory=list)
|
||||
problems: list[str] = field(default_factory=list)
|
||||
|
||||
def by_id(self) -> dict[int, str]:
|
||||
return dict(self.entries)
|
||||
|
||||
def by_name(self) -> dict[str, int]:
|
||||
"""lower-cased filename -> id"""
|
||||
return {n.lower(): i for i, n in self.entries}
|
||||
|
||||
|
||||
def parse_manifest(text: str) -> Manifest:
|
||||
m = Manifest()
|
||||
seen: dict[int, int] = {}
|
||||
for lineno, raw in enumerate(text.splitlines(), 1):
|
||||
line = raw.strip()
|
||||
if not line:
|
||||
continue
|
||||
d = _DELETED_RE.search(line)
|
||||
if d:
|
||||
m.deleted.append(int(d.group(1)))
|
||||
continue
|
||||
if line.startswith("//"):
|
||||
continue
|
||||
e = _ENTRY_RE.match(line)
|
||||
if not e:
|
||||
m.problems.append(f"line {lineno}: unrecognised {line!r}")
|
||||
continue
|
||||
i, name = int(e.group(1)), e.group(2)
|
||||
if i in seen:
|
||||
m.problems.append(f"line {lineno}: duplicate id {i} (first at line {seen[i]})")
|
||||
seen[i] = lineno
|
||||
m.entries.append((i, name))
|
||||
for i in m.deleted:
|
||||
if i in seen:
|
||||
m.problems.append(f"id {i} is both DELETED and assigned")
|
||||
return m
|
||||
|
||||
|
||||
def read_text(path) -> str:
|
||||
with open(path, "rb") as f:
|
||||
return f.read().decode("cp1252")
|
||||
|
||||
|
||||
def parse_manifest_file(path) -> Manifest:
|
||||
return parse_manifest(read_text(path))
|
||||
|
||||
|
||||
def _is_comment(row: list[str]) -> bool:
|
||||
if not row:
|
||||
return True
|
||||
first = row[0].lstrip()
|
||||
if first.startswith("#") or first.startswith("//"):
|
||||
return True
|
||||
return all(c.strip() == "" for c in row)
|
||||
|
||||
|
||||
def parse_csv(text: str, *, strip: bool = True) -> list[list[str]]:
|
||||
rows = []
|
||||
for row in csv.reader(io.StringIO(text, newline="")):
|
||||
if _is_comment(row):
|
||||
continue
|
||||
rows.append([c.strip() for c in row] if strip else row)
|
||||
return rows
|
||||
|
||||
|
||||
def parse_csv_with_header(text: str) -> tuple[list[str] | None, list[list[str]]]:
|
||||
"""Return (header, rows). Header = the first '#'-prefixed row that
|
||||
contains a comma (e.g. '# <tech>,<human-pri>,...'), with the '#' and
|
||||
any '<>' stripped; None when there is no such row."""
|
||||
header = None
|
||||
for row in csv.reader(io.StringIO(text, newline="")):
|
||||
if not row:
|
||||
continue
|
||||
first = row[0].lstrip()
|
||||
if first.startswith("#") and len(row) > 1:
|
||||
header = [c.strip().lstrip("#").strip().strip("<>") for c in row]
|
||||
break
|
||||
if not _is_comment(row):
|
||||
break
|
||||
return header, parse_csv(text)
|
||||
|
||||
|
||||
def parse_csv_file(path, **kw):
|
||||
return parse_csv(read_text(path), **kw)
|
||||
211
save-reader/ref/parsers/mars_data.py
Normal file
211
save-reader/ref/parsers/mars_data.py
Normal file
|
|
@ -0,0 +1,211 @@
|
|||
"""mars_data.py -- reader for the Mars engine's brace-block key/value format.
|
||||
|
||||
Covers: *.weapon, *.shipsection, *.tech, *.combat, *.def, *.script and the
|
||||
block-form *.txt files (scenarios, tutorial, credits, systemnames, skydefs,
|
||||
ctechvars, shipai).
|
||||
|
||||
Grammar (as observed in every shipped SOTS1 file):
|
||||
|
||||
body := (block | pair | item)*
|
||||
block := NAME '{' body '}'
|
||||
pair := NAME value
|
||||
item := QUOTED # bare quoted string inside a block
|
||||
value := QUOTED | BAREWORD
|
||||
comment := '//' .* EOL
|
||||
|
||||
A file's top level is itself a body (scenario .txt files mix top-level pairs
|
||||
and player{} blocks; catalog files hold one or many named blocks).
|
||||
|
||||
Result shape: plain dicts. A key seen once maps to its value; a key seen
|
||||
more than once maps to a list (use get_list() when you want a list always).
|
||||
Bare quoted items are collected under the key "_items".
|
||||
|
||||
Quirks handled (all seen in the real data, see parsers-report.md):
|
||||
* keys are case-insensitive to the engine ("Requires"/"requires",
|
||||
"badge"/"Badge") -> keys are lower-cased unless keep_case=True
|
||||
* a block may open on the same line as a preceding pair
|
||||
("turretsize small mount {") and a block name may sit on the same line
|
||||
as its brace ("weapon {")
|
||||
* backslashes inside quoted strings are literal (Windows paths); there is
|
||||
no escape syntax
|
||||
* '//' inside a quoted string is not a comment
|
||||
* CRLF and LF line endings, cp1252 bytes (decoded losslessly)
|
||||
* numbers use C float syntax: ".5", "-.8", "7e+8"
|
||||
|
||||
Stdlib only.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any, Iterator
|
||||
|
||||
__all__ = ["parse", "parse_file", "coerce", "get_list", "MarsSyntaxError"]
|
||||
|
||||
|
||||
class MarsSyntaxError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
# --- tokenizer -------------------------------------------------------------
|
||||
|
||||
_TOKEN_RE = re.compile(
|
||||
r"""
|
||||
(?P<ws>\s+)
|
||||
| (?P<comment>//[^\n]*)
|
||||
| (?P<open>\{)
|
||||
| (?P<close>\})
|
||||
| (?P<quoted>"[^"]*")
|
||||
| (?P<bad_quote>")
|
||||
| (?P<bare>[^\s{}"]+)
|
||||
""",
|
||||
re.VERBOSE,
|
||||
)
|
||||
|
||||
|
||||
def _tokenize(text: str) -> Iterator[tuple[str, str, int]]:
|
||||
"""Yield (kind, value, line). kind in {open, close, quoted, bare}."""
|
||||
line = 1
|
||||
pos = 0
|
||||
n = len(text)
|
||||
while pos < n:
|
||||
m = _TOKEN_RE.match(text, pos)
|
||||
if m is None: # pragma: no cover - regex is exhaustive
|
||||
raise MarsSyntaxError(f"line {line}: cannot tokenize {text[pos:pos+20]!r}")
|
||||
kind = m.lastgroup
|
||||
tok = m.group()
|
||||
pos = m.end()
|
||||
if kind == "ws":
|
||||
line += tok.count("\n")
|
||||
continue
|
||||
if kind == "comment":
|
||||
continue
|
||||
if kind == "bad_quote":
|
||||
raise MarsSyntaxError(f"line {line}: unterminated string")
|
||||
if kind == "quoted":
|
||||
yield kind, tok[1:-1], line
|
||||
line += tok.count("\n")
|
||||
else:
|
||||
yield kind, tok, line
|
||||
|
||||
|
||||
# --- parser ----------------------------------------------------------------
|
||||
|
||||
_INT_RE = re.compile(r"[+-]?\d+$")
|
||||
_FLOAT_RE = re.compile(r"[+-]?(\d+\.\d*|\.\d+|\d+)([eE][+-]?\d+)?$")
|
||||
|
||||
|
||||
def coerce(tok: str) -> Any:
|
||||
"""Bareword -> int / float / bool when it looks like one, else str."""
|
||||
if _INT_RE.match(tok):
|
||||
return int(tok)
|
||||
if _FLOAT_RE.match(tok):
|
||||
return float(tok)
|
||||
low = tok.lower()
|
||||
if low == "true":
|
||||
return True
|
||||
if low == "false":
|
||||
return False
|
||||
return tok
|
||||
|
||||
|
||||
def _add(d: dict, key: str, value: Any) -> None:
|
||||
if key in d:
|
||||
cur = d[key]
|
||||
if isinstance(cur, list):
|
||||
cur.append(value)
|
||||
else:
|
||||
d[key] = [cur, value]
|
||||
else:
|
||||
d[key] = value
|
||||
|
||||
|
||||
def get_list(d: dict, key: str) -> list:
|
||||
"""Always return a list for a key (missing -> [], single -> [x])."""
|
||||
v = d.get(key)
|
||||
if v is None:
|
||||
return []
|
||||
return v if isinstance(v, list) else [v]
|
||||
|
||||
|
||||
class _Parser:
|
||||
def __init__(self, text: str, typed: bool, keep_case: bool, strict: bool, warnings: list | None):
|
||||
self.toks = list(_tokenize(text))
|
||||
self.i = 0
|
||||
self.typed = typed
|
||||
self.keep_case = keep_case
|
||||
self.strict = strict
|
||||
self.warnings = warnings if warnings is not None else []
|
||||
|
||||
def _warn(self, msg: str) -> None:
|
||||
if self.strict:
|
||||
raise MarsSyntaxError(msg)
|
||||
self.warnings.append(msg)
|
||||
|
||||
def _peek(self):
|
||||
return self.toks[self.i] if self.i < len(self.toks) else None
|
||||
|
||||
def _next(self):
|
||||
t = self.toks[self.i]
|
||||
self.i += 1
|
||||
return t
|
||||
|
||||
def _key(self, name: str) -> str:
|
||||
return name if self.keep_case else name.lower()
|
||||
|
||||
def body(self, depth: int) -> dict:
|
||||
d: dict = {}
|
||||
while True:
|
||||
t = self._peek()
|
||||
if t is None:
|
||||
if depth:
|
||||
# 11 shipped shipsections never close their outer block;
|
||||
# the engine treats EOF as closing every open block.
|
||||
self._warn(f"end of file inside block (depth {depth})")
|
||||
return d
|
||||
kind, val, line = t
|
||||
if kind == "close":
|
||||
self._next()
|
||||
if not depth:
|
||||
# CrPropaganda.shipsection has one '}' too many.
|
||||
self._warn(f"line {line}: stray '}}' at top level")
|
||||
continue
|
||||
return d
|
||||
if kind == "open":
|
||||
raise MarsSyntaxError(f"line {line}: '{{' without a block name")
|
||||
self._next()
|
||||
if kind == "quoted":
|
||||
# bare string item (systemnames.txt lists) -- never a key
|
||||
_add(d, "_items", val)
|
||||
continue
|
||||
nxt = self._peek()
|
||||
if nxt is None or nxt[0] == "close":
|
||||
# lone bareword at end of block: treat as flag item
|
||||
_add(d, "_items", val)
|
||||
continue
|
||||
if nxt[0] == "open":
|
||||
self._next()
|
||||
_add(d, self._key(val), self.body(depth + 1))
|
||||
continue
|
||||
nkind, nval, _ = self._next()
|
||||
if nkind == "bare" and self.typed:
|
||||
nval = coerce(nval)
|
||||
_add(d, self._key(val), nval)
|
||||
|
||||
|
||||
def parse(text: str, *, typed: bool = True, keep_case: bool = False,
|
||||
strict: bool = False, warnings: list | None = None) -> dict:
|
||||
"""Parse brace-block text into nested dicts.
|
||||
|
||||
typed -- convert bareword numbers/bools (quoted strings stay str)
|
||||
keep_case -- keep key case instead of lower-casing
|
||||
strict -- raise on unbalanced braces instead of recovering the way
|
||||
the engine does (EOF closes open blocks, stray top-level
|
||||
'}' ignored); pass warnings=[] to collect the recoveries
|
||||
"""
|
||||
return _Parser(text, typed, keep_case, strict, warnings).body(0)
|
||||
|
||||
|
||||
def parse_file(path, **kw) -> dict:
|
||||
with open(path, "rb") as f:
|
||||
raw = f.read()
|
||||
return parse(raw.decode("cp1252"), **kw)
|
||||
634
save-reader/ref/parsers/verify.py
Normal file
634
save-reader/ref/parsers/verify.py
Normal file
|
|
@ -0,0 +1,634 @@
|
|||
"""verify.py -- parse every shipped SOTS1 data file, cross-link the catalogs,
|
||||
and emit normalized JSON artifacts.
|
||||
|
||||
usage: python3 verify.py <gob-extract-dir> <out-dir>
|
||||
|
||||
Prints a markdown report to stdout; writes to <out-dir>:
|
||||
tech_tree.json weapons.json shipsections.json strings.json
|
||||
schema_stats.json crosslink.json tech_tree.dot
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import collections
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import effect_txt
|
||||
import flat_kv
|
||||
import manifest
|
||||
import mars_data
|
||||
from mars_data import get_list
|
||||
|
||||
RACES = ["Human", "Zuul", "Hiver", "Tarkas", "Liir", "Morrigi"]
|
||||
PLAYABLE = RACES
|
||||
ALL_RACE_DIRS = RACES + ["_NPC"]
|
||||
|
||||
BRACE_TXT = {"Data/tutorial.txt", "Data/credits.txt", "Data/Strategy/systemnames.txt",
|
||||
"Data/Combat/ctechvars.txt", "Data/Combat/shipai.txt",
|
||||
"Models/Skysphere/skydefs.txt", "Models/Skysphere/NodeSpace-skydefs.txt"}
|
||||
ROWS_TXT = {"Weapons/_turrets.txt", "Weapons/_defaultweapons.txt", "Data/Combat/damfx.txt",
|
||||
"Data/Combat/damfx_levels.txt", "Data/Strategy/playercolors.txt",
|
||||
"Badges/BadgeTable.txt", "Avatars/AvatarTable.txt", "GUI/WeaponIconPlacements.txt"}
|
||||
PROSE = {"Locale/EN/ChatTrans.txt"}
|
||||
|
||||
|
||||
def kind_of(rel: str) -> str:
|
||||
ext = rel.rsplit(".", 1)[-1].lower()
|
||||
base = os.path.basename(rel)
|
||||
if ext in ("weapon", "shipsection", "tech", "combat", "def", "script"):
|
||||
return "brace:" + ext
|
||||
if ext == "effect":
|
||||
return "effect"
|
||||
if ext == "csv":
|
||||
return "csv"
|
||||
if ext in ("fx", "fxh"):
|
||||
return "hlsl"
|
||||
if ext == "txt":
|
||||
if base in ("_weapons.txt", "_shipsections.txt"):
|
||||
return "manifest"
|
||||
if rel.startswith("Scenarios/") or rel in BRACE_TXT:
|
||||
return "brace:txt"
|
||||
if rel in ROWS_TXT:
|
||||
return "rows"
|
||||
if rel.startswith("Locale/EN/Desc") or rel in PROSE:
|
||||
return "prose"
|
||||
return "kv"
|
||||
return "other"
|
||||
|
||||
|
||||
def walk(root):
|
||||
for dp, _, fn in os.walk(root):
|
||||
for f in sorted(fn):
|
||||
p = os.path.join(dp, f)
|
||||
yield p, os.path.relpath(p, root).replace(os.sep, "/")
|
||||
|
||||
|
||||
# --- schema stats ------------------------------------------------------------
|
||||
|
||||
def schema_walk(node, path, stats):
|
||||
"""Count key occurrences per block path, and which keys are blocks."""
|
||||
st = stats.setdefault(path, {"blocks": 0, "keys": collections.Counter(), "sub": collections.Counter()})
|
||||
st["blocks"] += 1
|
||||
for k, v in node.items():
|
||||
vals = v if isinstance(v, list) else [v]
|
||||
for x in vals:
|
||||
if isinstance(x, dict):
|
||||
st["sub"][k] += 1
|
||||
schema_walk(x, path + "." + k, stats)
|
||||
else:
|
||||
st["keys"][k] += 1
|
||||
|
||||
|
||||
# --- tech tree ---------------------------------------------------------------
|
||||
|
||||
_RP_RE = re.compile(r"^RP:(\d+)$", re.I)
|
||||
_PCT_RE = re.compile(r"^(\w+):(\d+)$")
|
||||
|
||||
|
||||
def parse_allows(s: str):
|
||||
toks = s.split()
|
||||
child = toks[0]
|
||||
rp = None
|
||||
pct = {}
|
||||
extra = []
|
||||
for t in toks[1:]:
|
||||
m = _RP_RE.match(t)
|
||||
if m:
|
||||
rp = int(m.group(1))
|
||||
continue
|
||||
m = _PCT_RE.match(t)
|
||||
if m and m.group(1) in RACES:
|
||||
pct[m.group(1)] = int(m.group(2))
|
||||
continue
|
||||
extra.append(t)
|
||||
return child, rp, pct, extra
|
||||
|
||||
|
||||
def main(root: str, out: str) -> int:
|
||||
os.makedirs(out, exist_ok=True)
|
||||
rep = []
|
||||
P = rep.append
|
||||
|
||||
# ---- 1. parse everything ---------------------------------------------
|
||||
ok = collections.Counter()
|
||||
fail = collections.Counter()
|
||||
fails = []
|
||||
warns = []
|
||||
parsed = {} # rel -> object
|
||||
for p, rel in walk(root):
|
||||
k = kind_of(rel)
|
||||
try:
|
||||
if k.startswith("brace"):
|
||||
w = []
|
||||
obj = mars_data.parse_file(p, warnings=w)
|
||||
# strict re-parse to record the recovery
|
||||
if w:
|
||||
warns.append((rel, w))
|
||||
elif k == "effect":
|
||||
obj = effect_txt.parse_file(p)
|
||||
elif k == "csv":
|
||||
obj = manifest.parse_csv_file(p)
|
||||
elif k == "manifest":
|
||||
obj = manifest.parse_manifest_file(p)
|
||||
if obj.problems:
|
||||
raise ValueError("; ".join(obj.problems))
|
||||
elif k == "rows":
|
||||
obj = flat_kv.parse_rows_file(p)
|
||||
elif k == "kv":
|
||||
txt = manifest.read_text(p)
|
||||
obj = flat_kv.parse_kv(txt)
|
||||
d = flat_kv.duplicates(txt)
|
||||
if d:
|
||||
warns.append((rel, [f"duplicate keys {d}"]))
|
||||
else:
|
||||
continue
|
||||
parsed[rel] = obj
|
||||
ok[k] += 1
|
||||
except Exception as e: # noqa: BLE001
|
||||
fail[k] += 1
|
||||
fails.append((rel, repr(e)))
|
||||
|
||||
P("## Parse results")
|
||||
P("")
|
||||
P("| reader | file kind | files | parsed | failed |")
|
||||
P("|---|---|---|---|---|")
|
||||
reader_of = {"brace": "mars_data", "effect": "effect_txt", "csv": "manifest.parse_csv",
|
||||
"manifest": "manifest.parse_manifest", "rows": "flat_kv.parse_rows", "kv": "flat_kv.parse_kv"}
|
||||
for k in sorted(set(ok) | set(fail)):
|
||||
P(f"| {reader_of[k.split(':')[0]]} | {k} | {ok[k] + fail[k]} | {ok[k]} | {fail[k]} |")
|
||||
P("")
|
||||
P(f"Total: {sum(ok.values())} parsed, {sum(fail.values())} failed "
|
||||
f"(skipped: {sum(1 for _, r in walk(root) if kind_of(r) in ('hlsl', 'prose', 'other'))} "
|
||||
f"HLSL/prose files that are not data).")
|
||||
if fails:
|
||||
P("")
|
||||
P("Failures:")
|
||||
for rel, e in fails:
|
||||
P(f"- `{rel}`: {e}")
|
||||
if warns:
|
||||
P("")
|
||||
P("Lenient recoveries (engine-compatible; strict=True would reject these):")
|
||||
for rel, w in warns:
|
||||
P(f"- `{rel}`: {'; '.join(w)}")
|
||||
P("")
|
||||
|
||||
# ---- 2. schema stats ---------------------------------------------------
|
||||
stats = {}
|
||||
for rel, obj in parsed.items():
|
||||
k = kind_of(rel)
|
||||
if k in ("brace:weapon", "brace:shipsection", "brace:tech", "brace:combat", "brace:def", "brace:script"):
|
||||
schema_walk(obj, k.split(":")[1], stats)
|
||||
schema_json = {path: {"blocks": st["blocks"],
|
||||
"keys": dict(st["keys"].most_common()),
|
||||
"subblocks": dict(st["sub"].most_common())}
|
||||
for path, st in sorted(stats.items())}
|
||||
json.dump(schema_json, open(os.path.join(out, "schema_stats.json"), "w"), indent=1)
|
||||
|
||||
P("## Schema stats (key frequency per block type)")
|
||||
P("")
|
||||
P("Full table in `schema_stats.json`. Block paths with instance counts and the")
|
||||
P("keys seen in them (count = number of block instances carrying the key):")
|
||||
P("")
|
||||
for path in ["weapon.weapon", "shipsection.shipsection", "tech.tech"]:
|
||||
st = stats[path]
|
||||
P(f"### `{path}` ({st['blocks']} instances)")
|
||||
P("")
|
||||
P("keys: " + ", ".join(f"{k}:{n}" for k, n in st["keys"].most_common()))
|
||||
P("")
|
||||
P("sub-blocks: " + ", ".join(f"{k}:{n}" for k, n in st["sub"].most_common()))
|
||||
P("")
|
||||
P("All block paths: " + ", ".join(f"`{p}`({st['blocks']})" for p, st in sorted(stats.items())))
|
||||
P("")
|
||||
|
||||
# ---- 3. build catalogs -------------------------------------------------
|
||||
techs = parsed["TechTree/MasterTechList.tech"]["tech"]
|
||||
tech_by = {t["name"].lower(): t for t in techs}
|
||||
groups = collections.defaultdict(list)
|
||||
for t in techs:
|
||||
if "group" in t:
|
||||
groups[str(t["group"]).upper()].append(t["name"])
|
||||
|
||||
strings_rows = parsed["Locale/EN/Strings.csv"]
|
||||
strings = {}
|
||||
string_dups = []
|
||||
for r in strings_rows:
|
||||
k, v = r[0], (r[1] if len(r) > 1 else "")
|
||||
if k in strings:
|
||||
string_dups.append((k, strings[k], v))
|
||||
strings[k] = v
|
||||
strings_lc = {k.lower(): v for k, v in strings.items()}
|
||||
|
||||
def s(key):
|
||||
return strings_lc.get(key.lower())
|
||||
|
||||
weapons = {} # stem -> record
|
||||
for rel, obj in parsed.items():
|
||||
if kind_of(rel) != "brace:weapon":
|
||||
continue
|
||||
stem = os.path.basename(rel)[:-7]
|
||||
w = dict(obj["weapon"])
|
||||
weapons[stem.lower()] = {"stem": stem, "file": rel,
|
||||
"scope": "NPC" if rel.startswith("Species/_NPC") else "player",
|
||||
"id": None, **w}
|
||||
wman = parsed["Weapons/_weapons.txt"]
|
||||
for i, name in wman.entries:
|
||||
key = name.lower()[:-7]
|
||||
if key in weapons and weapons[key]["scope"] == "player":
|
||||
weapons[key]["id"] = i
|
||||
|
||||
sections = {} # (race, stem) -> record
|
||||
for rel, obj in parsed.items():
|
||||
if kind_of(rel) != "brace:shipsection":
|
||||
continue
|
||||
race = rel.split("/")[1]
|
||||
stem = os.path.basename(rel)[:-12]
|
||||
sections[(race, stem.lower())] = {"race": race, "stem": stem, "file": rel, "id": None, **obj["shipsection"]}
|
||||
for race in ALL_RACE_DIRS:
|
||||
m = parsed[f"Species/{race}/sections/_shipsections.txt"]
|
||||
for i, name in m.entries:
|
||||
key = (race, name.lower()[:-12])
|
||||
if key in sections:
|
||||
sections[key]["id"] = i
|
||||
section_stems = collections.defaultdict(list) # stem.lower -> [races]
|
||||
for (race, st), rec in sections.items():
|
||||
section_stems[st].append(race)
|
||||
|
||||
# ---- 4. cross-links ----------------------------------------------------
|
||||
X = {}
|
||||
P("## Cross-link results")
|
||||
P("")
|
||||
|
||||
def tech_exists(name):
|
||||
n = name.lower()
|
||||
if n in tech_by:
|
||||
return True
|
||||
if n.startswith("grp_") and n[4:].upper() in groups:
|
||||
return True
|
||||
return False
|
||||
|
||||
# weapon.requires -> tech (repeated `requires` lines = AND of techs)
|
||||
dang = []
|
||||
case_mismatch = []
|
||||
nref = 0
|
||||
multi = 0
|
||||
for w in weapons.values():
|
||||
reqs = get_list(w, "requires")
|
||||
multi += len(reqs) > 1
|
||||
for r in reqs:
|
||||
nref += 1
|
||||
r = str(r)
|
||||
if not tech_exists(r):
|
||||
dang.append((w["file"], r))
|
||||
elif r.lower() in tech_by and tech_by[r.lower()]["name"] != r:
|
||||
case_mismatch.append((w["file"], r))
|
||||
no_req = [w["file"] for w in weapons.values() if "requires" not in w]
|
||||
X["weapon_requires_dangling"] = dang
|
||||
X["weapon_requires_case_mismatch"] = case_mismatch
|
||||
X["weapon_without_requires"] = no_req
|
||||
P(f"- weapon `requires` -> tech: {nref} refs in {len(weapons) - len(no_req)} weapons ({multi} weapons list 2+ techs), "
|
||||
f"{len(dang)} dangling, {len(case_mismatch)} case-mismatched; {len(no_req)} weapons have no `requires` "
|
||||
f"(NPC: {sum(1 for f in no_req if f.startswith('Species/_NPC'))}, player: "
|
||||
f"{[os.path.basename(f) for f in no_req if not f.startswith('Species/_NPC')]}).")
|
||||
for f, r in dang:
|
||||
P(f" - DANGLING `{f}` requires `{r}`")
|
||||
for f, r in case_mismatch:
|
||||
P(f" - case: `{f}` requires `{r}` (tech is `{tech_by[r.lower()]['name']}`)")
|
||||
|
||||
# shipsection.requires / option -> tech
|
||||
dang = []
|
||||
case_mm = []
|
||||
opt_dang = []
|
||||
scalar_opts = []
|
||||
nreq = 0
|
||||
nopt = 0
|
||||
for rec in sections.values():
|
||||
for r in get_list(rec, "requires"):
|
||||
nreq += 1
|
||||
if not tech_exists(str(r)):
|
||||
dang.append((rec["file"], r))
|
||||
elif str(r).lower() in tech_by and tech_by[str(r).lower()]["name"] != r:
|
||||
case_mm.append((rec["file"], r))
|
||||
for blk_key in ("option", "optiondef"):
|
||||
for blk in get_list(rec, blk_key):
|
||||
# a few files write a bare `option TECH` at section level
|
||||
# instead of wrapping it in option { }
|
||||
opts = get_list(blk, "option") if isinstance(blk, dict) else [blk]
|
||||
if not isinstance(blk, dict):
|
||||
scalar_opts.append((rec["file"], blk))
|
||||
for o in opts:
|
||||
nopt += 1
|
||||
if not tech_exists(str(o)):
|
||||
opt_dang.append((rec["file"], o))
|
||||
X["shipsection_requires_dangling"] = dang
|
||||
X["shipsection_requires_case_mismatch"] = case_mm
|
||||
X["shipsection_option_dangling"] = opt_dang
|
||||
P(f"- shipsection `requires` -> tech: {nreq} refs, {len(dang)} dangling, {len(case_mm)} case-mismatched.")
|
||||
for f, r in dang:
|
||||
P(f" - DANGLING `{f}` requires `{r}`")
|
||||
for f, r in case_mm:
|
||||
P(f" - case: `{f}` requires `{r}`")
|
||||
X["shipsection_scalar_option"] = scalar_opts
|
||||
P(f"- shipsection `option{{option T}}`/`optiondef` -> tech: {nopt} refs, {len(opt_dang)} dangling. "
|
||||
f"Two forms coexist: `option {{ option A option B }}` (a mutually-exclusive choice group) and a bare "
|
||||
f"section-level `option T` ({len(scalar_opts)} occurrences in {len(set(f for f, _ in scalar_opts))} files, "
|
||||
f"e.g. `option DRV_PlsmFoc` on engine sections) -- both merge under the key `option`, so consumers must "
|
||||
f"accept str-or-dict list members.")
|
||||
for f, r in sorted(set(opt_dang)):
|
||||
P(f" - DANGLING `{f}` option `{r}`")
|
||||
|
||||
# tech.ship.section -> shipsection
|
||||
dang = []
|
||||
nsec = 0
|
||||
for t in techs:
|
||||
for blk in get_list(t, "ship"):
|
||||
for sname in get_list(blk, "section"):
|
||||
nsec += 1
|
||||
if str(sname).lower() not in section_stems:
|
||||
dang.append((t["name"], sname))
|
||||
X["tech_ship_section_dangling"] = dang
|
||||
P(f"- tech `ship{{section}}` -> shipsection: {nsec} refs, {len(dang)} dangling "
|
||||
f"(matched against the union of all race catalogs, case-insensitive).")
|
||||
for t, sname in dang:
|
||||
P(f" - DANGLING tech `{t}` unlocks section `{sname}`")
|
||||
|
||||
# tech.weapon.filename -> file
|
||||
disk = {rel.lower() for _, rel in walk(root)}
|
||||
dang = [(t["name"], w["filename"]) for t in techs for w in get_list(t, "weapon") if w["filename"].lower() not in disk]
|
||||
X["tech_weapon_filename_dangling"] = dang
|
||||
nw = sum(len(get_list(t, "weapon")) for t in techs)
|
||||
P(f"- tech `weapon{{filename}}` -> file: {nw} refs, {len(dang)} dangling.")
|
||||
|
||||
# tech.requires / allows -> tech
|
||||
dang_req = [(t["name"], r) for t in techs for r in get_list(t, "requires") if not tech_exists(str(r))]
|
||||
edges = []
|
||||
dang_allow = []
|
||||
bad_allow = []
|
||||
for t in techs:
|
||||
for a in get_list(t, "allows"):
|
||||
child, rp, pct, extra = parse_allows(a)
|
||||
if extra or rp is None:
|
||||
bad_allow.append((t["name"], a))
|
||||
if child.lower() not in tech_by:
|
||||
dang_allow.append((t["name"], child))
|
||||
edges.append({"from": t["name"], "to": child, "rp": rp, "pct": pct})
|
||||
X["tech_requires_dangling"] = dang_req
|
||||
X["tech_allows_dangling"] = dang_allow
|
||||
X["tech_allows_unparsed"] = bad_allow
|
||||
P(f"- tech `requires` -> tech/GRP_: {sum(len(get_list(t, 'requires')) for t in techs)} refs, {len(dang_req)} dangling. "
|
||||
f"Groups: {dict((g, len(v)) for g, v in groups.items())}.")
|
||||
for t, r in dang_req:
|
||||
P(f" - DANGLING tech `{t}` requires `{r}`")
|
||||
P(f"- tech `allows` edges: {len(edges)}, {len(dang_allow)} point at unknown techs, {len(bad_allow)} unparsable.")
|
||||
for t, c in dang_allow:
|
||||
P(f" - DANGLING tech `{t}` allows `{c}`")
|
||||
roots = [t["name"] for t in techs if not any(e["to"].lower() == t["name"].lower() for e in edges)]
|
||||
P(f"- techs never allowed by anything (roots/orphans): {len(roots)}: {', '.join(roots)}")
|
||||
dup_names = [n for n, c in collections.Counter(t["name"].lower() for t in techs).items() if c > 1]
|
||||
P(f"- duplicate tech names: {dup_names or 'none'}")
|
||||
|
||||
# manifests <-> files
|
||||
P("- id manifests <-> files:")
|
||||
man_rep = {}
|
||||
wfiles = {os.path.basename(rel).lower() for rel in parsed if rel.startswith("Weapons/") and rel.endswith(".weapon")}
|
||||
listed = {n.lower() for _, n in wman.entries}
|
||||
man_rep["Weapons"] = {"ids": len(wman.entries), "deleted": wman.deleted,
|
||||
"listed_but_no_file": sorted(listed - wfiles), "file_but_unlisted": sorted(wfiles - listed)}
|
||||
for race in ALL_RACE_DIRS:
|
||||
m = parsed[f"Species/{race}/sections/_shipsections.txt"]
|
||||
files = {os.path.basename(rel).lower() for rel in parsed if rel.startswith(f"Species/{race}/sections/") and rel.endswith(".shipsection")}
|
||||
listed = {n.lower() for _, n in m.entries}
|
||||
man_rep[race] = {"ids": len(m.entries), "deleted": m.deleted,
|
||||
"listed_but_no_file": sorted(listed - files), "file_but_unlisted": sorted(files - listed)}
|
||||
X["manifests"] = man_rep
|
||||
for k, v in man_rep.items():
|
||||
P(f" - `{k}`: {v['ids']} ids (deleted {v['deleted'] or 'none'}); "
|
||||
f"listed-but-no-file {len(v['listed_but_no_file'])}; file-but-unlisted {len(v['file_but_unlisted'])}")
|
||||
for n in v["listed_but_no_file"]:
|
||||
P(f" - MISSING FILE for id {[i for i, nn in (wman if k == 'Weapons' else parsed[f'Species/{k}/sections/_shipsections.txt']).entries if nn.lower() == n][0]}: `{n}`")
|
||||
for n in v["file_but_unlisted"]:
|
||||
P(f" - UNLISTED file `{n}` (no network/save id)")
|
||||
npc_weapons_unlisted = sorted(w["file"] for w in weapons.values() if w["scope"] == "NPC")
|
||||
P(f" - `Species/_NPC/weapons/*.weapon` ({len(npc_weapons_unlisted)} files) have no manifest at all; "
|
||||
f"they are referenced by filename from `_NPC` shipsection `bank{{weapon}}` lines.")
|
||||
|
||||
# strings
|
||||
P("- localization:")
|
||||
P(f" - `Strings.csv`: {len(strings_rows)} data rows -> {len(strings)} keys. {len(string_dups)} keys occur twice "
|
||||
f"because one copy carries a trailing space (parse_csv strips cells; the later row wins):")
|
||||
for k, a, b in string_dups:
|
||||
P(f" - `{k}`: {a!r} then {b!r}")
|
||||
miss_tn = [t["name"] for t in techs if s("TECHNAME_" + t["name"]) is None]
|
||||
miss_td = [t["name"] for t in techs if s("TECHDESC_" + t["name"]) is None]
|
||||
P(f" - TECHNAME_/TECHDESC_ for {len(techs)} techs: {len(miss_tn)} / {len(miss_td)} missing. {miss_tn} {miss_td}")
|
||||
stems = sorted(section_stems)
|
||||
miss_sn = [st for st in stems if s("SECTIONNAME_" + st) is None]
|
||||
miss_sd = [st for st in stems if s("SECTIONDESC_" + st) is None]
|
||||
P(f" - SECTIONNAME_/SECTIONDESC_ for {len(stems)} distinct section stems: {len(miss_sn)} / {len(miss_sd)} missing.")
|
||||
for label, miss in (("SECTIONNAME_", miss_sn), ("SECTIONDESC_", miss_sd)):
|
||||
npc_only = [st for st in miss if section_stems[st] == ["_NPC"]]
|
||||
other = [st for st in miss if st not in npc_only]
|
||||
P(f" - missing {label}: {len(npc_only)} are `_NPC`-only stems (never shown in the design UI); "
|
||||
f"player-race stems: {len(other)} {other}")
|
||||
miss_wn = [(w["file"], w.get("name")) for w in weapons.values()
|
||||
if isinstance(w.get("name"), str) and w["name"].startswith("@") and s(w["name"][1:]) is None]
|
||||
unnamed = [w["file"] for w in weapons.values() if "name" not in w]
|
||||
P(f" - weapon `name @TOKEN`: {len(miss_wn)} unresolved of {sum(1 for w in weapons.values() if 'name' in w)}; "
|
||||
f"{len(unnamed)} weapons carry no `name`.")
|
||||
for f, n in miss_wn:
|
||||
P(f" - UNRESOLVED `{f}` name `{n}`")
|
||||
# every @token anywhere in brace files
|
||||
at_missing = collections.Counter()
|
||||
at_total = 0
|
||||
for rel, obj in parsed.items():
|
||||
if not kind_of(rel).startswith("brace"):
|
||||
continue
|
||||
for tok in re.findall(r"@([A-Za-z0-9_]+)", manifest.read_text(os.path.join(root, rel))):
|
||||
at_total += 1
|
||||
if s(tok) is None:
|
||||
at_missing[(rel, tok)] += 1
|
||||
P(f" - all `@TOKEN` refs in brace-block files: {at_total} refs, {len(at_missing)} unresolved.")
|
||||
for (rel, tok), n in sorted(at_missing.items()):
|
||||
P(f" - UNRESOLVED `{rel}` `@{tok}`")
|
||||
X["strings"] = {"missing_techname": miss_tn, "missing_techdesc": miss_td,
|
||||
"missing_sectionname": miss_sn, "missing_sectiondesc": miss_sd,
|
||||
"unresolved_weapon_name": miss_wn, "unresolved_at_tokens": sorted(f"{r}:@{t}" for r, t in at_missing)}
|
||||
|
||||
# turrets
|
||||
turrets = parsed["Weapons/_turrets.txt"]
|
||||
|
||||
def last_lc(d, key):
|
||||
v = get_list(d, key)
|
||||
return str(v[-1]).lower() if v else None
|
||||
|
||||
tpairs = {(str(r[1]).lower(), str(r[2]).lower()) for r in turrets} # (weapon-size, class)
|
||||
tslots = {(str(r[0]).lower(), str(r[2]).lower()) for r in turrets} # (mount size, class)
|
||||
wpairs = collections.Counter((last_lc(w, "turretsize"), last_lc(w, "turretclass")) for w in weapons.values())
|
||||
w_unfit = sorted((p, n) for p, n in wpairs.items() if p not in tpairs)
|
||||
bpairs = collections.Counter()
|
||||
nobank = 0
|
||||
dupkeys = 0
|
||||
for rec in sections.values():
|
||||
for b in get_list(rec, "bank"):
|
||||
if "turretsize" not in b:
|
||||
nobank += 1
|
||||
continue
|
||||
if isinstance(b.get("turretsize"), list) or isinstance(b.get("turretclass"), list):
|
||||
dupkeys += 1
|
||||
bpairs[(last_lc(b, "turretsize"), last_lc(b, "turretclass"))] += 1
|
||||
b_unfit = sorted((p, n) for p, n in bpairs.items() if p not in tslots)
|
||||
X["turrets"] = {"turret_rows": len(turrets), "weapon_size_class_pairs_without_turret": w_unfit,
|
||||
"bank_size_class_pairs_without_turret": b_unfit,
|
||||
"banks_without_turretsize": nobank, "banks_with_repeated_size_or_class": dupkeys}
|
||||
P(f"- `_turrets.txt` ({len(turrets)} rows; size/class values compared case-insensitively -- the data mixes "
|
||||
f"`Large`/`large`, `Missile`/`missile`, `Standard`/`standard`):")
|
||||
P(f" - weapon (turretsize,turretclass) pairs with no turret row: {w_unfit or 'none'}")
|
||||
P(f" - section bank (turretsize,turretclass) pairs with no turret row: {b_unfit or 'none'}")
|
||||
P(f" - banks with no turretsize at all (NPC fixed-weapon banks): {nobank}; banks that repeat "
|
||||
f"turretsize/turretclass inside one bank{{}} (last value taken): {dupkeys}")
|
||||
|
||||
# NPC bank{weapon} refs
|
||||
dang = []
|
||||
n = 0
|
||||
for rec in sections.values():
|
||||
for b in get_list(rec, "bank"):
|
||||
for wf in get_list(b, "weapon"):
|
||||
n += 1
|
||||
if str(wf).lower() not in disk:
|
||||
dang.append((rec["file"], wf))
|
||||
X["bank_weapon_dangling"] = dang
|
||||
P(f"- shipsection `bank{{weapon <file>}}` -> file: {n} refs, {len(dang)} dangling.")
|
||||
for f, w in dang:
|
||||
P(f" - DANGLING `{f}` -> `{w}`")
|
||||
|
||||
# default weapons
|
||||
dw = parsed["Weapons/_defaultweapons.txt"]
|
||||
dang = [r for r in dw if ("weapons/" + str(r[2])).lower() not in disk]
|
||||
P(f"- `_defaultweapons.txt`: {len(dw)} rows, {len(dang)} name a missing weapon file. {dang or ''}")
|
||||
|
||||
# AI tables
|
||||
def csv_col(rel, col):
|
||||
return [r[col] for r in parsed[rel] if len(r) > col and r[col]]
|
||||
ai = {}
|
||||
for rel in ("Data/Strategy/AI/aitechpri.csv", "Data/Strategy/AI/aitechgrp.csv", "Data/Strategy/AI/aitechmode.csv"):
|
||||
rows = parsed[rel]
|
||||
bad = [t for t in csv_col(rel, 0) if t.lower() not in tech_by]
|
||||
ai[rel] = bad
|
||||
if not rows:
|
||||
P(f"- `{rel}`: 0 data rows -- the shipped file is a comment-only template (schema documented in its "
|
||||
f"header, no entries); the AI's tech priorities must therefore come from code.")
|
||||
else:
|
||||
P(f"- `{rel}`: {len(rows)} rows; col0 not a tech: {bad or 'none'}")
|
||||
bad = [x for x in csv_col("Data/Strategy/AI/affinity_section.csv", 0) if x.lower() not in section_stems]
|
||||
ai["affinity_section_unknown"] = bad
|
||||
P(f"- `AI/affinity_section.csv`: {len(parsed['Data/Strategy/AI/affinity_section.csv'])} rows; unknown sections: {bad or 'none'}")
|
||||
bad = [x for x in csv_col("Data/Strategy/AI/raider_sections.csv", 0) if x.lower() not in section_stems]
|
||||
P(f"- `AI/raider_sections.csv`: unknown sections: {bad or 'none'}")
|
||||
wr = parsed["Data/Strategy/AI/weapon_replacements.csv"]
|
||||
bad = [x for r in wr for x in r if x and x.lower() not in weapons]
|
||||
ai["weapon_replacements_unknown"] = bad
|
||||
P(f"- `AI/weapon_replacements.csv`: {len(wr)} rows; unknown weapon stems: {bad or 'none'}")
|
||||
fams = collections.Counter(str(w.get("weaponfamily")) for w in weapons.values() if "weaponfamily" in w)
|
||||
aw = csv_col("Data/Strategy/AI/affinity_weapon.csv", 0)
|
||||
bad = [x for x in aw if x not in fams]
|
||||
P(f"- `AI/affinity_weapon.csv`: families {sorted(set(aw))}; not a weaponfamily in any .weapon: {bad or 'none'}. "
|
||||
f"weaponfamily values in data: {dict(fams)}")
|
||||
# scenarios
|
||||
for rel in sorted(parsed):
|
||||
if rel.startswith("Scenarios/") and rel.endswith("Templates.csv"):
|
||||
bad = [(r[0], x) for r in parsed[rel] for x in r[1:4] if x.lower() not in section_stems]
|
||||
P(f"- `{rel}`: {len(parsed[rel])} rows; unknown sections: {bad or 'none'}")
|
||||
if rel.startswith("Scenarios/") and rel.endswith("Techs.csv"):
|
||||
bad = [x for x in csv_col(rel, 0) if x.lower() not in tech_by]
|
||||
P(f"- `{rel}`: {len(parsed[rel])} rows; unknown techs: {bad or 'none'}")
|
||||
X["ai"] = ai
|
||||
json.dump(X, open(os.path.join(out, "crosslink.json"), "w"), indent=1)
|
||||
P("")
|
||||
|
||||
# ---- 5. artifacts ------------------------------------------------------
|
||||
nodes = []
|
||||
for t in techs:
|
||||
strat = get_list(t, "strategy")
|
||||
inc = [x for b in strat for x in get_list(b, "inc")]
|
||||
dec = [x for b in strat for x in get_list(b, "dec")]
|
||||
nodes.append({
|
||||
"name": t["name"],
|
||||
"display_name": s("TECHNAME_" + t["name"]),
|
||||
"description": s("TECHDESC_" + t["name"]),
|
||||
"family": t.get("family"),
|
||||
"family_inferred": t["name"].split("_", 1)[0].upper(),
|
||||
"type": t.get("type"),
|
||||
"threat": t.get("threat"),
|
||||
"group": t.get("group"),
|
||||
"option_cost": t.get("option_cost"),
|
||||
"requires": [str(r) for r in get_list(t, "requires")],
|
||||
"benefits_inc": inc,
|
||||
"benefits_dec": dec,
|
||||
"sections": [str(x) for b in get_list(t, "ship") for x in get_list(b, "section")],
|
||||
"weapons": [w["filename"] for w in get_list(t, "weapon")],
|
||||
"allows": [e["to"] for e in edges if e["from"] == t["name"]],
|
||||
})
|
||||
tech_tree = {
|
||||
"_about": "SOTS1 MasterTechList.tech normalized. family is only written on ~half the nodes; "
|
||||
"family_inferred is the name prefix (IND/WEP/DRV/...). edges[].pct: per-race availability % as written; "
|
||||
"a race absent from pct has no override in the file (the engine default -- believed to be 100 -- "
|
||||
"is code-owned, not asserted here). rp = research-point cost of the edge. "
|
||||
"requires may name GRP_<group>, satisfied by any tech with group <group>.",
|
||||
"races": RACES,
|
||||
"groups": dict(groups),
|
||||
"nodes": nodes,
|
||||
"edges": edges,
|
||||
}
|
||||
json.dump(tech_tree, open(os.path.join(out, "tech_tree.json"), "w"), indent=1)
|
||||
|
||||
def norm_weapon(w):
|
||||
d = dict(w)
|
||||
d["display_name"] = s(w["name"][1:]) if isinstance(w.get("name"), str) and w["name"].startswith("@") else w.get("name")
|
||||
return d
|
||||
json.dump({"_about": "All *.weapon files (Weapons/ = player catalog with ids from _weapons.txt; Species/_NPC/weapons = NPC, no ids). "
|
||||
"Keys lower-cased; repeated keys -> lists; bareword numbers typed.",
|
||||
"weapons": [norm_weapon(w) for w in sorted(weapons.values(), key=lambda w: (w["scope"], w["stem"].lower()))]},
|
||||
open(os.path.join(out, "weapons.json"), "w"), indent=1)
|
||||
|
||||
def norm_section(r):
|
||||
d = dict(r)
|
||||
d["display_name"] = s("SECTIONNAME_" + r["stem"])
|
||||
d["description"] = s("SECTIONDESC_" + r["stem"])
|
||||
d["unlocked_by"] = [t["name"] for t in techs for b in get_list(t, "ship") if r["stem"].lower() in [str(x).lower() for x in get_list(b, "section")]]
|
||||
return d
|
||||
json.dump({"_about": "All Species/<race>/sections/*.shipsection; id from the race's _shipsections.txt (null = unlisted). "
|
||||
"Keys lower-cased; repeated keys (bank, option, thruster, requires) -> lists.",
|
||||
"sections": [norm_section(r) for r in sorted(sections.values(), key=lambda r: (r["race"], r["stem"].lower()))]},
|
||||
open(os.path.join(out, "shipsections.json"), "w"), indent=1)
|
||||
|
||||
json.dump(strings, open(os.path.join(out, "strings.json"), "w"), indent=1, ensure_ascii=False)
|
||||
|
||||
with open(os.path.join(out, "tech_tree.dot"), "w") as f:
|
||||
f.write("digraph sots_tech {\n rankdir=LR; node [shape=box, fontsize=9];\n")
|
||||
fam_color = {"IND": "#f4d03f", "NRG": "#e74c3c", "SLD": "#3498db", "DRV": "#9b59b6", "TRP": "#e67e22",
|
||||
"WAR": "#c0392b", "BAL": "#7f8c8d", "BIO": "#2ecc71", "CCC": "#1abc9c", "DRN": "#95a5a6", "XNC": "#d35400"}
|
||||
for n in nodes:
|
||||
col = fam_color.get(str(n["family"]), "#ffffff")
|
||||
label = n["display_name"] or n["name"]
|
||||
f.write(f' "{n["name"]}" [label="{label}\\n{n["name"]}", style=filled, fillcolor="{col}"];\n')
|
||||
for e in edges:
|
||||
lab = f"{e['rp']}" if e["rp"] is not None else ""
|
||||
if e["pct"]:
|
||||
lab += "\\n" + " ".join(f"{r[:2]}{v}" for r, v in e["pct"].items())
|
||||
f.write(f' "{e["from"]}" -> "{e["to"]}" [label="{lab}", fontsize=7];\n')
|
||||
f.write("}\n")
|
||||
|
||||
P("## Artifacts")
|
||||
P("")
|
||||
for fn in ("tech_tree.json", "weapons.json", "shipsections.json", "strings.json", "schema_stats.json", "crosslink.json", "tech_tree.dot"):
|
||||
P(f"- `{fn}` ({os.path.getsize(os.path.join(out, fn)) // 1024} KB)")
|
||||
P(f"- tech_tree.json: {len(nodes)} nodes, {len(edges)} edges; weapons.json: {len(weapons)}; "
|
||||
f"shipsections.json: {len(sections)}; strings.json: {len(strings)} keys")
|
||||
print("\n".join(rep))
|
||||
return 0 if not fails else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1], sys.argv[2]))
|
||||
550
save-reader/ref/save-editor-structs.md
Normal file
550
save-reader/ref/save-editor-structs.md
Normal file
|
|
@ -0,0 +1,550 @@
|
|||
# Sword of the Stars 1 (SOTS1) — Save-File Struct Reference (Community RE)
|
||||
|
||||
Harvested from two community save-game editors for cross-checking against binary-recovered
|
||||
serializable types. SOTS1 uses a self-describing "Streamable" serialization system: **field
|
||||
order + type matter**, and most values are preceded by their own name string.
|
||||
|
||||
Target game version: **SOTS1 v1.8+** (both editors target the 1.8.x line; repo2 says "1.80 +19").
|
||||
|
||||
## Sources
|
||||
- **[R1] BardezAnAvatar/Sots.Sots1.SavedGameEditor** — C#. A *complete, ordered, typed*
|
||||
Streamable parser. Migrated from SourceForge `sots-sge`. This is the authoritative layout
|
||||
evidence (read/write methods reproduce exact on-disk order). Key files under
|
||||
`Bardez.Project.SwordOfTheStars.DataStructures/`:
|
||||
- `BaseSaveStructures.cs` (primitives + framing), `SharedSaveStructures.cs` (coords/colors),
|
||||
`SaveGameDataStructure.cs` (top level), `SummarySaveStructures.cs`,
|
||||
`CreateParametersSaveStructures.cs`, `SimulationSaveStructures.cs` (11,974 lines — the bulk),
|
||||
`CdTableSaveStructures.cs` (combat/AI table). IO: `Bardez.Project.SwordOfTheStars.IO/{Gzip,SaveFileIO}.cs`.
|
||||
- **[R2] ghbplayer/SOTSedit** — C#/WPF. A *name-tag scanner* (does NOT model layout; it searches
|
||||
the decompressed blob for length-prefixed field-name strings and reads the value that follows).
|
||||
Value comes from `Parse.cs` + the field catalog `SOTSEdit.cfg` (friendly-name ⇄ serialized-name
|
||||
⇄ type mapping) and race/semantic hacks. Written 2011 "to learn C#"; author calls the parser weak.
|
||||
|
||||
Both editors independently confirm the same primitives, framing markers, race IDs, and color IDs.
|
||||
|
||||
---
|
||||
|
||||
## 1. File format & serialization mechanics
|
||||
|
||||
### 1.1 Container
|
||||
- **Whole `.sav` file = gzip stream.** Decompress first (`GZipStream`, standard gzip). [R1 Gzip.cs, R2 gzip.cs]
|
||||
- Editors work on the **decompressed** byte stream (R1 names its test artifacts `*.sav.inflate.dat`).
|
||||
- No separate magic/version header is decoded by either editor beyond the top-level Summary block;
|
||||
version is implied by scenario/field presence, not a numeric version field. Endianness: **little-endian**
|
||||
throughout (`BitConverter` on x86).
|
||||
|
||||
### 1.2 Decompressed top-level order [R1 SaveGameData.ReadFromStream]
|
||||
```
|
||||
SaveGameData:
|
||||
1. SummarySaveStruct summary
|
||||
2. CreateParametersSaveStruct createParams
|
||||
3. SimSaveStruct sim <-- the giant one (players/systems/fleets/etc.)
|
||||
4. CdTable cdTable <-- combat / AI state table
|
||||
```
|
||||
|
||||
### 1.3 Primitive encodings [R1 BaseSaveStructures.cs]
|
||||
Text encoding is **windows-1252** (R1) / ASCII (R2).
|
||||
|
||||
- **StringStruct** (raw string): `Int32 length` + `length` bytes (no NUL terminator counted).
|
||||
R2 calls this a "BStr": 4-byte little-endian length prefix + ASCII bytes.
|
||||
- **Padding rule (critical):** every *basic* value struct is **NUL-padded to a 4-byte boundary**.
|
||||
`PaddingSize = 4`. Padding is computed over `(sizeof(Int32 desc-length) + description.Length + valueBytes)`.
|
||||
- **Named value fields** (`BasicSaveStruct` subclasses) are each laid out as:
|
||||
`StringStruct description` (a field-name tag, often the non-descriptive `"."`) → then the value →
|
||||
then NUL padding to 4 bytes. Concrete leaf types:
|
||||
| Struct | Payload after description tag |
|
||||
|---|---|
|
||||
| `Int32SaveStruct` | 4-byte Int32 |
|
||||
| `Int64SaveStruct` | 8-byte Int64 |
|
||||
| `FloatSaveStruct` | 4-byte IEEE Single |
|
||||
| `BooleanSaveStruct` | 1 byte (0/1), padded to 4 |
|
||||
| `StringSaveStruct` | nested StringStruct (len+bytes) |
|
||||
| `ByteArraySaveStruct` | raw bytes (length externally known) |
|
||||
|
||||
So the on-disk shape of a named scalar is: `[len][name-ascii][pad] [value] [pad]`. R2 exploits exactly
|
||||
this: it locates a field by searching for `[len][name]` and reads the value immediately after.
|
||||
|
||||
### 1.4 Complex-struct framing (the "BEEFBEEF" envelope) [R1 ComplexSaveStruct]
|
||||
Every **complex** structure is framed:
|
||||
```
|
||||
StringStruct description (NUL-padded to 4)
|
||||
UInt32 0xBEEFBEEF (begin marker)
|
||||
... body (ordered child fields) ...
|
||||
UInt32 0x41104110 (end marker = bitwise NOT of 0xBEEFBEEF)
|
||||
```
|
||||
`0xBEEFBEEF` / `~0xBEEFBEEF (0x41104110)` bracket every complex object — a reliable resync/validation
|
||||
signature when scanning the binary. (`ISotsStructure` leaf types are NOT framed; only `ComplexSaveStruct`.)
|
||||
|
||||
### 1.5 Array conventions [R1 BaseSaveStructures.cs]
|
||||
- **ComplexArraySaveStruct<T>**: framed (has description + BEEFBEEF), body = `Int32SaveStruct count`
|
||||
then `count` × T.
|
||||
- **NonComplexArraySaveStruct<T>**: NOT framed; body = `Int32SaveStruct count` then `count` × T.
|
||||
(Distinguishing which arrays are framed vs. not is itself layout evidence — see per-struct notes.)
|
||||
|
||||
### 1.6 Conditional & polymorphic reads (watch for these in the binary)
|
||||
- **Optional-by-flag:** a boolean/int gate precedes an optional sub-object.
|
||||
- `SimPlayerColorSaveStruct`: `Int32 colorIndex`; **iff `colorIndex == -1`**, an `RgbColorInt32`
|
||||
(custom RGB) follows. Otherwise palette index only.
|
||||
- `SimPlayerDesignDw2SaveStruct` (weapon slot): `Boolean bId`; if true → `Int32 wId`, else →
|
||||
`StringSaveStruct wfn` (weapon full resource path); then `Int32 dId`.
|
||||
- `SimFleetShipDetails`: `Boolean hbq` gates `bq`; `Boolean hsp` gates `sp`. Fleet flight-plan/lay
|
||||
gated by `hfPlan` / `hLay` booleans.
|
||||
- `SimSystemDetailNvo.isInd` gates independent-colony sub-block; `SimSystemDetailsIndi.hindi`,
|
||||
`SimSystemDetailsVonNeumann.vnh` similar boolean gates.
|
||||
- **Polymorphism by string tag:** `SimSvSctObXscn` reads `StringSaveStruct xcsn`, then switches:
|
||||
`"crowdefs"`, `"gmtrigger"`, `"traps"`, `"indsys"`/default → different body subclass.
|
||||
- **Polymorphism by fixed position:** `SimScSctObEncObjArray` (grand-menace/encounter objects) reads a
|
||||
count then dispatches subclass **by index 0..8** in fixed order:
|
||||
`0 Infest, 1 Dsn, 2 AsteroidMonitor, 3 TD, 4 WD, 5 Hives, 6 Rsuc, 7 Dfts, 8 Ini2`.
|
||||
|
||||
### 1.7 Write-time quirks worth knowing (R2)
|
||||
- R2 edits in place and cannot safely change string length (it truncates/space-pads to the original
|
||||
length). R1 rewrites the whole stream and re-pads. If the binary stores string lengths, the game
|
||||
reads them dynamically (R1 proves round-trip works when re-padded).
|
||||
- R2 planet OID/PID hack: on-disk **`OID = PID * 16`** (R2 divides by 16 to show a "PlayerID").
|
||||
i.e. the raw owner id field is the player index shifted left 4 bits.
|
||||
|
||||
---
|
||||
|
||||
## 2. Enums / ID tables (agreed by both editors)
|
||||
|
||||
### 2.1 Species / race ID [R2 Parse.cs addRace(); R1 PlayerSlot.FxSp comment]
|
||||
| ID | Species |
|
||||
|---|---|
|
||||
| 0 | Human |
|
||||
| 1 | Hiver |
|
||||
| 2 | Tarka(s) |
|
||||
| 3 | Liir |
|
||||
| 4 | `_NPC` / AI-rebellion / grand-menace player (R1: "??? AI Rebellion") |
|
||||
| 5 | Zuul |
|
||||
| 6 | Morrigi |
|
||||
|
||||
### 2.2 Player color ID (palette index) [R1 PlayerSlot.FxCrId & SimPlayerColor]
|
||||
`01 Red, 02 Yellow, 03 Blue, 04 Pink/Magenta, 05 Orange, 06 Green, 07 Aqua, 08 Gray,
|
||||
09 Dark Green, 10 Purple`. Value **-1 ⇒ custom RGB triplet follows** (see §1.6).
|
||||
|
||||
### 2.3 Difficulty [R1 PlayerSettings.Difficulty]
|
||||
`0 Easy, 1 Normal, 2 Difficult`.
|
||||
|
||||
### 2.4 Sentinel values seen in fields
|
||||
`0x7FFFFFFF (Int32.MaxValue)` used as "tag"/unset (PlayerSlot.tag);
|
||||
`team = -1 (0xFFFFFFFF)` = no team; R2: value `-1` = field absent in this save.
|
||||
No named C# enums exist for tech IDs / weapon families — techs and weapons are **string resource
|
||||
names** (e.g. tech `tNm`, weapon `wfn`), not numeric enums. Weapon *family* enumeration lives only in
|
||||
the CdAi combat block as `aiSitWepFams` (opaque int set).
|
||||
|
||||
---
|
||||
|
||||
## 3. Summary block [R1 SummarySaveStructures.cs] (complex)
|
||||
|
||||
### SummarySaveStruct (fields in on-disk order)
|
||||
1. `StringSaveStruct gameName`
|
||||
2. `Int32 turn`
|
||||
3. `Int32 numSys` (system count)
|
||||
4. `Int32 checkSum`
|
||||
5. `ComplexArray<PlayerSlotWrapper> players`
|
||||
6. `SessionSaveStruct session`
|
||||
7. `Int32 mapShape`
|
||||
8. `Int32 incMod` (income modifier)
|
||||
9. `Int32 resMod` (research modifier)
|
||||
10. `Boolean alliances`
|
||||
11. `Boolean teams`
|
||||
12. `Boolean encounters`
|
||||
13. `StringSaveStruct scenario`
|
||||
|
||||
### PlayerSlotWrapper (complex): `{ PlayerSlotSaveStruct slot; Int32 rank; }`
|
||||
|
||||
### PlayerSlotSaveStruct (complex) — new-game slot definition, ordered:
|
||||
`Boolean isPlay, isDead, isReq, isRec, isFxNm` → `String fxNm` (fixed name) →
|
||||
`Boolean isFxSp` → `Int32 fxSp` (species, §2.1) → `Boolean isFxCr` →
|
||||
`NestedInt32 fxCrId` (color, §2.2) → `Boolean isFxBd` → `String fxBd` (badge) →
|
||||
`Boolean isFxAv` → `String fxAv` (avatar) → `Int32 tag` (often 0x7FFFFFFF) →
|
||||
`Int32 pwd` (password) → `Int32 team` (-1=none) → `PlayerSettingsSaveStruct settings`.
|
||||
|
||||
### PlayerSettingsSaveStruct (complex):
|
||||
`Int32 initialTreasury, initialColonies, initialTechnologies, difficulty (§2.3)`.
|
||||
|
||||
### SessionSaveStruct (complex) → `TmrsSaveStruct tmrs`:
|
||||
`Int32 tstl (0x7F7FFFFF), tctl (0x42700000), tqtl (0x7F7FFFFF), tqtle (0)` — timer limits (float bit-patterns stored as int).
|
||||
|
||||
---
|
||||
|
||||
## 4. CreateParameters block [R1 CreateParametersSaveStructures.cs] (complex)
|
||||
|
||||
### CreateParametersSaveStruct (ordered):
|
||||
`String name; Int32 id; Int32 rSeed (random seed); Int32 aid; String key;`
|
||||
`MapPSaveStruct mapP;` `Int32 mapS; Int32 mapF; Int32 nSys;` `Float rEnc (random-encounter rate);`
|
||||
`Int32 sDist;` `Float sSize; Float sRes;` `Int32 sSuit; Int32 maxP; Int32 aSpec;`
|
||||
`Boolean bAlly; Int32 nTeam; Boolean tmgrp;`
|
||||
`Int32 pSav (start savings); Int32 pCol (start colonies); Int32 pTech (start techs);`
|
||||
`Float incM; Float resM; ScrpSaveStruct scrp;`
|
||||
|
||||
### MapPSaveStruct (complex) — initial map/galaxy generation:
|
||||
1. `Int32 unknown1`
|
||||
2. `ComplexArray<PlanetSaveStruct> planetArray`
|
||||
3. `NonComplexArray<ComplexArray<Int32>> players` (per-player int arrays; "non-complex array of players")
|
||||
4. `ComplexArray<MapPNpc> npcArray` (≈ players − 1; independents/NPCs)
|
||||
|
||||
### PlanetSaveStruct (complex) — initial star node geometry:
|
||||
`SpatialCoordinate coordinates (x,y,z floats)`, `Int32 unknown1..4`
|
||||
(values seen: `0x7FFFFFFF`, `0x7F7FFFFF`). NOTE: this is the *map-generation* planet record; the
|
||||
*live* planet/colony state lives in `SimSystemDetailsSaveStruct` (§8).
|
||||
|
||||
### MapPNpc (complex): `Int32 unknown1, unknown2`. ### ScrpSaveStruct (complex): `Int32 spc`.
|
||||
|
||||
### Shared value types [R1 SharedSaveStructures.cs]
|
||||
- `SpatialCoordinateSaveStruct` (complex): `Float x, y, z`.
|
||||
- `RgbColorFloat` (leaf): `Float r,g,b`. `RgbaColorFloat` (leaf): `RgbColorFloat rgb; Float a`.
|
||||
- `RgbColorInt32` (leaf): `Int32 r,g,b`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Simulation block — top level [R1 SimulationSaveStructures.cs]
|
||||
|
||||
### SimSaveStruct (complex) — the master world state, ordered:
|
||||
```
|
||||
String keyPath
|
||||
Int32 nMsz, nMlc, nMnx (next-id / size counters)
|
||||
NonComplexArray<Int32> playerIds, designIds, systemIds, fleetIds, shipIds, tradeIds
|
||||
Int32 modCount, frame, gameId
|
||||
AttributeSaveStruct attribute
|
||||
RngSaveStruct rng (ByteArray unknownData ~2500 bytes: RNG state)
|
||||
String gameName
|
||||
Int32 map, incMod, resMod, enAl, enTm, gOTurn
|
||||
NestedInt32 gOWinPly
|
||||
Int32 npcm, npco, npci, npcv, npca, szad, rsad, suad
|
||||
ComplexArray<ResearchSaveStruct> sprjs (shared/special research projects)
|
||||
Float randEncAdjustment
|
||||
Int32 cmbtid (next combat id)
|
||||
ComplexArray<TurnPly> turnstats (per-turn per-player history)
|
||||
NonComplexArray<SimCrepSaveStruct> creps (combat reports)
|
||||
Int32 ninv, allExc1, allExc2, allExcCF
|
||||
NonComplexArray<SimPlayerSaveStruct> players <-- EMPIRES (§6)
|
||||
SimSpeciesArraySaveStruct species (galaxy species list)
|
||||
NonComplexArray<SimSystemSaveStruct> systems <-- STAR SYSTEMS (§8)
|
||||
SimNodeGrid2 ndgr2 (node-line / warp graph)
|
||||
SimTradeManager trdmgr (§9)
|
||||
SimSpyManager spymgr (Int32 xsid, nspy)
|
||||
NonComplexArray<SimFleet> flt <-- FLEETS (§10)
|
||||
NonComplexArray<Int32> acts
|
||||
SimSvSctOb svSctOb (scenario/encounter objects, §11)
|
||||
Int32 zdsc, zdsi, zdst (Zuul/system-destroyer counters)
|
||||
```
|
||||
|
||||
### Small shared sim types
|
||||
- `SimPopGSaveStruct` (complex): `Int32 popT; Int32 popS; Int64 popC` (pop type / species / civ count).
|
||||
- `ResearchSaveStruct` (leaf): `NonComplexArray<ResearchOptionalSaveStruct> us; String nm; String ntg`.
|
||||
`ResearchOptionalSaveStruct`: `Int32 usc, usp`.
|
||||
- `TurnPly` (leaf): `Int32 ply; PlyHistSaveStruct hist`.
|
||||
- `PlyHistSaveStruct` (complex): `Int32 ply; PlyHistStatsSaveStruct[] stats`.
|
||||
- `PlyHistStatsSaveStruct` (complex): `Int64 pop; ComplexArray<PlyHistStatsSacq> sacq, slost;`
|
||||
`Int32 trn, almem, inc, tdinc, sav, col, bat, tch; NonComplexArray<PlyHistClsSaveStruct> cls`.
|
||||
- `PlyHistStatsSacq` (complex): `Int32 set, ses, seop, senp; NonComplexArray<Int32> seo`.
|
||||
- `PlyHistClsSaveStruct` (leaf): `Int32 cls, shpt, shpl, shpk, satt, satl, satk` (ship/sat built/lost/killed by class).
|
||||
|
||||
---
|
||||
|
||||
## 6. Player / Empire [R1 SimulationSaveStructures.cs]
|
||||
|
||||
### SimPlayerSaveStruct (leaf wrapper): `Int32 playerId; SimPlayerDetailsSaveStruct details`.
|
||||
|
||||
### SimPlayerDetailsSaveStruct (complex) — the empire record, on-disk order:
|
||||
```
|
||||
SimPlayerTechTree techTree <-- TECH TREE (§7)
|
||||
Int32 homeSystem, playerIndex
|
||||
String playerName
|
||||
Int32 species (§2.1)
|
||||
SimPlayerColorSaveStruct colorId (palette idx or -1 + RGB, §1.6/§2.2)
|
||||
String badge, avatar
|
||||
Int32 team, sav (savings)
|
||||
Float idealSuit, suitTolerance, maxOH
|
||||
Float resRate, resModifier, resScl (research)
|
||||
Int32 trm, trp, tra
|
||||
Float outMod, rebOutMod, scOutMod, incMod, popMod, terraMod (economy multipliers)
|
||||
Boolean aMine; Float minPure, minRate; Int32 ngts, prGtTrf, gTraf
|
||||
Int32 cstR, cstE, cstT, maint, shrm, status, elim
|
||||
Boolean npc, rebAi, reqCL
|
||||
SimPlayerTeamSaveStruct teamStruct (Int32 alid, al, na, cf)
|
||||
Int32 hasVac, hasImm, npTrak, hasDisc, hasDiscSp, hasDiscCl, hasEnc, hasEng
|
||||
SimPlayerEventsSaveStruct events (Int32 evNxId; ComplexArray<SimPlayerEvent>)
|
||||
NestedInt32 fngNum
|
||||
Int32 pvSav, pvMA, aibN
|
||||
Boolean cnTrd, cnRad, hgs, hadvs, harcc, cnVItl
|
||||
Float pddm
|
||||
Int32 bankWrn, bankTrn, bankPr, bankEl
|
||||
SimPlayerShipRecsEventsSaveStruct shipRecs
|
||||
Int32 nextPrjId, plcy, pswd, lret, nmeid
|
||||
Boolean cdp
|
||||
SimPlayerSpySaveStruct spy2 (Int32 defc2, rtc, evc, ttc)
|
||||
SimPlayerCivrSaveStruct civR (Float smx; ComplexArray<SimPlayerCivrSpeSpVa {Int32 sp, va2}>)
|
||||
Int32 aidf
|
||||
Boolean srn; Int32 srcTo, lboid, lcid2
|
||||
String resTnm
|
||||
Boolean resErrRoll
|
||||
SimPlayerModsSaveStruct conMods (3× SimPlayerConModSaveStruct {Float conMod, savMod})
|
||||
NonComplexArray<Int32> ownerIds
|
||||
NonComplexArray<SimPlayerDesignEntrySaveStruct> designs <-- SHIP DESIGNS (§7.2)
|
||||
NonComplexArray<SimPlayerDesignEntrySaveStruct> droneDesigns
|
||||
NonComplexArray<SimPlayerNote> notes (Int32 ntSys; String ntTxt; Int32 ntTrn)
|
||||
NonComplexArray<SimPlayerPr> pr (Float prm; Int32 prbt)
|
||||
Boolean hasAiRebellion, cta
|
||||
NestedInt32 aienf
|
||||
NonComplexArray<SimPlayerDetailsSpecialProjectT> nSprj
|
||||
Int32 nexp, nWeapXcl
|
||||
ComplexArray<SimPlayerDetailsOjv> ovjs (objectives: Int32 id; Bool cmp; Int32 spr; String dsc; Int32 nid)
|
||||
ComplexArray<SimPlayerDipStat> dipStats (diplomacy; see below)
|
||||
ComplexArray<SimPlayerComm> comms (Int32 msgt; SimPlayerCommMsg msg)
|
||||
ComplexArray<SimPlayerPrepSaveStruct> preps
|
||||
ComplexArray<SimPlayerOdesSaveStruct> odes
|
||||
ComplexArray<SimPlayerOwepSaveStruct> owep
|
||||
ComplexArray<SimPlayerOtchSaveStruct> otch
|
||||
NestedInt32 aid
|
||||
Int32 ndeflay, rdtc, tnc
|
||||
```
|
||||
- `SimPlayerDipStat` (complex): `Int32 other; SimPlayerDipStatDetail nap, ally, cf; Int32 deadhome`.
|
||||
`SimPlayerDipStatDetail` (leaf): `Int32 last_, last_bty, bkn_, bty_`.
|
||||
- `SimPlayerCommMsg` (complex): `Int32 cid2, snd, rcp, exp, sent, rcpt, sys`.
|
||||
- `SimPlayerPrepSaveStruct` (complex): `Int32 oid, pid, flds, sav, home, ncol, mpwr, mcls, mmsl, nshp, nsat`.
|
||||
- `SimPlayerOdes/Owep/Otch` (complex, "old design/weapon/tech" build history):
|
||||
`Odes {Int32 ontF, otnL, odid, opid}`, `Owep {Int32 ontF, otnL, odet; String owep; Int32 owith}`,
|
||||
`Otch {Int32 ontF, otnL, odet; String otch; Int32 owith}`.
|
||||
- `SimPlayerDetailsSpecialProjectT` (leaf): `Int32 sprjT; SimPlayerDetailsSpecialProjectDetails sprj`.
|
||||
`...Details` (complex): `Int32 stp; SpecialProjectSpi spi; tail (polymorphic AsMon | Tech)`.
|
||||
`...Spi` (complex): `Int32 sPid, sts; Float cst; Int32 mxC; String name; Int32 trns`.
|
||||
tail `AsMon`: `Int32 rDn, sys, rMn, rMx; Float rMd`; tail `Tech`: `Int32 rDn; Float aOdd, aInc; String tch; Int32 rCst`.
|
||||
|
||||
---
|
||||
|
||||
## 7. Tech tree & ship designs
|
||||
|
||||
### 7.1 Tech tree [R1]
|
||||
- `SimPlayerTechTree` (complex): `NonComplexArray<SimPlayerTechTreeBranch> tree; NonComplexArray<SimPlayerTechTreeTech> techs`.
|
||||
- `SimPlayerTechTreeBranch` (leaf): `String tNm` (tech name); `NonComplexArray<String> branches` (child tech names).
|
||||
- `SimPlayerTechTreeTech` (leaf), ordered:
|
||||
`String tNm` (tech name) → `Int32 st` → `Int32 tResCost` (research cost) →
|
||||
`Int32 tResDone` (progress) → `Int32 tAcq` (turn acquired) → `Int32 tiAcq` (turns-to-acquire) →
|
||||
`Int32 tbd` → `Boolean tfc` → `Int32 tUnlck` (unlocked flag).
|
||||
**Techs are identified by string name, not a numeric enum.**
|
||||
|
||||
### 7.2 Ship designs [R1]
|
||||
- `SimPlayerDesignEntrySaveStruct` (leaf): `Int32 designId; SimPlayerDesignSaveStruct design`.
|
||||
- `SimPlayerDesignSaveStruct` (complex), ordered:
|
||||
`Boolean faiDes; Boolean dHide; Int32 dWep; String dName;` `SimPlayerDesignSectionArray sections;`
|
||||
`Int32 dtc; NonComplexArray<SimPlayerDesignDwg> dwgv`.
|
||||
- `SimPlayerDesignSectionArray` (leaf): `Int32 count` + `count` × `SimPlayerDesignSectionEntrySaveStruct`.
|
||||
- `SimPlayerDesignSectionEntrySaveStruct` (complex): `SimPlayerDesignSectionSaveStruct dsec` (Int32 unknown1, unknown2);
|
||||
`ComplexArray<SimPlayerDesignUnknown1SaveStruct> dgbnk2;` `ComplexArray<String> dOpts`.
|
||||
- `SimPlayerDesignUnknown1SaveStruct` (complex) → `SimPlayerDesignDw2SaveStruct dw2` (weapon slot,
|
||||
conditional — see §1.6): `Boolean bId; (bId? Int32 wId : String wfn); Int32 dId`.
|
||||
- `SimPlayerDesignDwg` (complex): `NonComplexArray<SimPlayerDesignGng> wgng`;
|
||||
`SimPlayerDesignGng` (leaf): `Int32 wgid; SimPlayerDesignDwgWgb wgb` (complex: `Int32 unknown1..3`).
|
||||
|
||||
---
|
||||
|
||||
## 8. Star systems & planets [R1]
|
||||
|
||||
- `SimSystemSaveStruct` (leaf): `Int32 sysId; SimSystemDetailsSaveStruct details`.
|
||||
- **`SimSystemDetailsSaveStruct` (complex)** — the live system+colony record, on-disk order:
|
||||
```
|
||||
SpatialCoordinate pos
|
||||
RgbaColorFloat starColor
|
||||
Int32 idx
|
||||
Int32 size (1-10)
|
||||
Float suit (climate hazard)
|
||||
Int32 res, aRes, mRes (resources / asteroid / extra)
|
||||
Boolean noRebAi
|
||||
Int32 tRes, pop
|
||||
ComplexArray<SimPopGSaveStruct> popG
|
||||
Float infra
|
||||
Int32 pvPop; ComplexArray<SimPopG> pvPopG; Float pvInfra, pvSuit; Int32 pvRes, pvARes2, pvMRes; Bool pvNoRebAi (previous-turn snapshot)
|
||||
SimSystemDetailRtsSaveStruct rts (Float sRs, sRt, sRsc, sRtf, sRi, sRoh, sRnr — IO allocations)
|
||||
Int32 abdn; Boolean dstyd; Int32 tnsOh
|
||||
Float outMod, repCur, repMax
|
||||
Int32 ntdev, pbon
|
||||
ComplexArray<SimPopG> pbon2
|
||||
Float ibon
|
||||
Int32 ltis, rbfl, rbtn, rbfr, rbwn, hsrg
|
||||
NonComplexArray<SimSystemDetailHalt> halt (Int32 haltt; Bool haltv)
|
||||
SimSystemDetailsVonNeumann vnm (Bool vnh gate → details: Bool vnd, vnex3, vnpex3)
|
||||
String name
|
||||
SimSystemDetailFlags1 flags1 (Int32 vFlags, eFlags, aFlags, fFlags, gFlags)
|
||||
Int64 bats2 (recent battles; larger=more recent)
|
||||
Int64 rcex
|
||||
SimSystemDetailFlags2 flags2 (Int32 mnRFlags, rfRFlags, clkFlags)
|
||||
Int32 eggScio, terrFl, tAcq, tfAcq, tDst
|
||||
ComplexArray<SimPopG> dcs; Int32 dsu
|
||||
ComplexArray<SimSystemDetailCm> cm, pvcm (SimSystemDetailCm: Int32 msp, mv)
|
||||
ComplexArray<SimSystemDetailCme2> cme2 (Int32 mid, mtrT, mn, mtp; ComplexArray<Cm> mfx; String mdsc)
|
||||
ComplexArray<SimSystemDetailSpy> spies
|
||||
Int32 pid (owner player id), defF, defSf
|
||||
SimSystemDetailBq bq (ComplexArray<SimSystemDetailBqOrd> ords; Ord: Int32 desId, con, conleft, sav, ordId)
|
||||
NonComplexArray<SimSystemDetailAdct> adct (Int32 ads, adt)
|
||||
Int32 numPlgs2
|
||||
NonComplexArray<Int32> flts, gfs, snF, mnF (fleets / gates / stations / monitors present)
|
||||
NonComplexArray<SimSystemDetailNvo> nvos (colonies; see below)
|
||||
NonComplexArray<SimSystemDetailVe> nve (Int32 ePid, ets, eid)
|
||||
NonComplexArray<SimSystemDetailVs> nvs (Int32 pid; SimSystemDetailVsPView pview)
|
||||
SimSystemDetailsIndi indi (Bool hindi gate → indsp, SimPlayerColor indcl, String indnm/indav/indba)
|
||||
```
|
||||
- `SimSystemDetailNvo` (leaf): `Int32 pid, tShn, oId; Boolean isInd; SimSystemDetailNvoIndi indi`
|
||||
(independent-colony sub-block gated by `isInd`).
|
||||
- `SimSystemDetailVsPView` (complex — per-player *seen* snapshot of a colony): `Int32 vTrn, pop;`
|
||||
`ComplexArray<SimPopG> pop2; Int32 infra; Float suit; Int32 res, aRes2, mRes; Bool noRebAi;`
|
||||
`Int32 pbon; ComplexArray<SimPopG> pbon2; Float ibon; Int32 terrFl; Bool footer`.
|
||||
|
||||
### 8.1 R2 ⇄ R1 cross-map for planet/colony fields (verifier gold)
|
||||
R2 finds these serialized tags anywhere in the "Planets" region (between markers `NumSys`…`NdGr2`).
|
||||
They correspond to fields inside R1's `SimSystemDetailsSaveStruct` / `...VsPView`:
|
||||
|
||||
| R2 serialized tag | type | meaning | R1 field |
|
||||
|---|---|---|---|
|
||||
| `Idx` | int | planet/system id | `idx` |
|
||||
| `Name` | string | name | `name` |
|
||||
| `Size` | int | 1-10 | `size` |
|
||||
| `Suit` | float | climate hazard | `suit` |
|
||||
| `Res` / `ARes2` / `MRes` | int | resources | `res` / `aRes` / `mRes` |
|
||||
| `Infra` | float | infrastructure | `infra` |
|
||||
| `ibon` | float | infra bonus | `ibon` |
|
||||
| `Pop` | int | imperial pop | `pop` |
|
||||
| `pbon` | int | imperial pop bonus | `pbon` |
|
||||
| `PopC` | long | civilian pop | (SimPopG `popC` Int64) |
|
||||
| `OID` | int | owner id (**= PID×16**) | `pid` (owner) |
|
||||
| `PID` | int | derived player id | (OID/16) |
|
||||
| `SRt/SRsc/SRtf/SRi/SRoh` | int | IO trade/ship/terraform/infra/overharvest | `rts.sRt/sRsc/sRtf/sRi/sRoh` |
|
||||
| `Abdn` | short | abandon order | `abdn` |
|
||||
| `Dstyd` | short | star annihilated | `dstyd` |
|
||||
| `ltis` | short | last-time-seen | `ltis` |
|
||||
| `VFlags/EFlags/AFlags/FFlags/GFlags` | int | state flags | `flags1.*` |
|
||||
| `Bats2` | int | recent combat | `bats2` (Int64 in R1) |
|
||||
| `nadct` | int | addicted (1=yes) | (in `adct` array) |
|
||||
| `NumFlts/NumGFs/NumSnF/NumMnF` | int | fleets/gates/stations/monitors | `flts/gfs/snF/mnF` counts |
|
||||
|
||||
Note the **type disagreements** (verifier flags): R2 reads `Abdn`, `Dstyd`, `ltis` as **short (Int16)**
|
||||
while R1 models them as framed `Int32SaveStruct`; R2 reads `Bats2` as int while R1 uses Int64. R2's
|
||||
name-scan reads the value bytes directly after the tag+pad, so R2's width is the more literal
|
||||
on-value-bytes claim for those specific fields; treat as "value is small, low bytes are the datum."
|
||||
|
||||
---
|
||||
|
||||
## 9. Trade & node grid [R1]
|
||||
- `SimNodeGrid2` (complex): `ComplexArray<SimNodeGridPath> paths; Int32 nextId`.
|
||||
`SimNodeGridPath` (complex): `Int32 npt, npid, npfr(from), npto(to), npctm, npcby, npdtn, npdtf, npenp, npuse, nptf`.
|
||||
- `SimTradeManager` (complex): `NonComplexArray<SimTradeSector> tradeSectors; Float sctSize; List<SimTradeSectorRt> rt`.
|
||||
`SimTradeSector` (leaf): `Int32 tradeId; SimTradeSectorTradeSaveStruct trade`.
|
||||
`SimTradeSectorTradeSaveStruct` (complex): `SpatialCoordinate pos; Int32 tradeSectorGridId;`
|
||||
`SimTradeSectorTradeCtrSaveStruct tsctr(3 floats); Int32 tssec, tsct, tscr, ptssec, ptsct, ptscr;`
|
||||
`ComplexArray<...Fwarn {Int32 pId, ntrns}> fwarn; NonComplexArray<Int32> systems, tsflt`.
|
||||
`SimTradeSectorRt` (complex): `Int32 tro, trfow, trfr, trfrs, trtow, trto, trtos, trtc`.
|
||||
|
||||
---
|
||||
|
||||
## 10. Fleets & ships [R1]
|
||||
- `SimFleet` (leaf): `Int32 fltId; SimFleetDetails flt`.
|
||||
- **`SimFleetDetails` (complex)**, ordered:
|
||||
`SpatialCoordinate pos; Int32 pId, locId; FlightPlanContainer fplan (Bool hfPlan gate);`
|
||||
`String ftName; Int32 ftTrans; SimFleetOrigin ftOrig (Int32 ×3);`
|
||||
`Int32 ftFlag, ftae, ftpae, ftEnc, ftMs, perm; SpatialCoordinate prvPos;`
|
||||
`LayContainer lay (Bool hLay gate); NonComplexArray<SimFleetShip> ships`.
|
||||
- `SimFleetDetailsFlightPlan` (complex): `ComplexArray<Wpt> wpts; Float fPsp2; Int32 fPeta2;`
|
||||
`Fpogn2 (3 floats); SpatialCoordinate fPdpos; Int32 pnd`. `Wpt`: `Int32 wpt, tp; Nrt {Int32 nrp,nrf,nrt}`.
|
||||
- `SimFleetShip` (leaf): `Int32 shipId; SimFleetShipDetails ship`.
|
||||
- **`SimFleetShipDetails` (complex)**, ordered:
|
||||
`Int32 desId (design id), fltId, plrId; Float range; SimFleetShipHealth health (3 floats: command/mission/drive);`
|
||||
`Int32 conCap, refCap, repCap, mineCap, plg, act; Boolean dep, atq; Int32 encId;`
|
||||
`SimFleetShipPrish prish (Int32 prMax; NonComplexArray<Int32> prSp); Int32 lct, tsd, atsp, tblt;`
|
||||
`Boolean hbq; SimFleetShipDetailsBq bq (gated by hbq);`
|
||||
`Boolean hsp; SimFleetShipDetailsSp sp (gated by hsp; two SpPop pop/pPop);`
|
||||
`NonComplexArray<SimFleetShipDetailsTh> th (Float th, thm)`.
|
||||
`SimFleetShipDetailsBqOrd`: `Int32 desId, con, conleft, sav, ordId` (same shape as system BqOrd).
|
||||
|
||||
---
|
||||
|
||||
## 11. Combat reports & scenario/encounter objects [R1]
|
||||
- `SimCrepSaveStruct` (combat report, complex): `Int32 cid, trn; SpatialCoordinate pos; Int32 sid, auto, dur, cow, cdst, cpk, cpt, cdt, cdi;`
|
||||
`ComplexArray<SimCrepPrepSaveStruct> prep; ComplexArray<SimCrepWrepSaveStruct> wrep`.
|
||||
`SimCrepPrepSaveStruct`: `Int32 plr; Bool ai; Int32 ally, status, mxeng, mxcls, mxmsl;`
|
||||
`NonComplexArray<Cls> cls; NonComplexArray<Sec> sec; Int32 ndam; ComplexArray<Srep> srep`.
|
||||
`SimCrepPrepSrepSaveStruct`: `String name; Int32 did, cls; Int64 caps2; Int32 nshp, nfld, nlst, dtak; SimDamsSaveStruct dams`.
|
||||
`SimDamsSaveStruct`: `Int32 dams, damp, dami, damt`. `SimCrepWrepSaveStruct`: `String wep; SimDams dams`.
|
||||
- `SimSvSctOb` (complex): `ScnObjStruct scn; NonComplexArray<SimSvSctObXscn> xscn; SimScSctObEncObjArray encObjs`.
|
||||
`SimSvSctObXscn` = polymorphic-by-string (§1.6): `traps` (ComplexArray<TrapDetails {Int32 sys,pid,trenc,trgenc}>),
|
||||
`gmtrigger` (Int32 gmch), `crowdefs` (Int32 sys; NonComplexArray<Int32> dsys,des; Float drad), `indsys`/default (empty).
|
||||
- **`SimScSctObEncObjArray`** = grand-menace/encounter table, dispatched **by fixed index 0-8** (§1.6).
|
||||
Each subclass carries that menace's state, e.g.:
|
||||
- `EncInfest` (Hiver infestation): `NonComplexArray<Asg> asg; ComplexArray<Infest> infests; Int32 deshive, deslarva`.
|
||||
- `EncHives`: `Int32 qDesignId; ComplexArray<Hive {Int32 hiveId,queenId,nextQ}> hives; ComplexArray<Queen {Int32 queenId,qDstId}> queens; ComplexArray<NestedInt32> sysMem`.
|
||||
- `EncAsteroidMonitor`, `EncTD`, `EncWD`, `EncRsuc`, `EncDfts` (Von Neumann; large), `EncIni2`.
|
||||
|
||||
---
|
||||
|
||||
## 12. CdTable — combat / AI persistence block [R1 CdTableSaveStructures.cs]
|
||||
|
||||
### CdTable (leaf, top of the 4th file-section): `ComplexArray<String> cdt; CdPlayer cdplayer; CdAi[] cdai`.
|
||||
|
||||
### CdPlayer (complex) — **entirely reverse-unlabeled** (fields named `unknown1..35`); shape is known:
|
||||
`Int32 unknown1(=16); Bool unknown2; Float unknown3; Bool unknown4; Int32 unknown4p5; Bool unknown5..8;`
|
||||
`Int32 unknown9,10; NonComplexArray<CdPlayerUnknown11Item {Int32 unknownId, const1, const2, value1}> unknown11;`
|
||||
`Int32 unknown12..14; NonComplexArray<Int32> unknown15,16; NonComplexArray<Unknown17Item{Int32 ×2}> unknown17;`
|
||||
`NonComplexArray<Int32> unknown18; Int32 unknown19..21;`
|
||||
`NonComplexArray<Unknown22Item{Int32×2,Bool}> unknown22; Int32 unknown23..35`.
|
||||
|
||||
### CdAi (complex) — per-AI-player planner state, ordered:
|
||||
`AttributeSaveStruct aiAttr; NestedInt32 aiTurnPris; CdAiSit aiSit; NestedInt32 aiPlyHat;`
|
||||
`ComplexArray<CdAiPrsUnknown {Int32 pid,trn}> prs2; Int32 dsh, nbStab, nmBlst; CdAiAidng aidng;`
|
||||
`Int32 aiHivJ, sdFlT; NestedInt32 nalat; Int32 lnat, lat;`
|
||||
`NonComplexArray<CdAiSys> aiSys; NonComplexArray<CdAiCmbr> cmbR; NonComplexArray<CdAiCl {Int32 clTn,clSyId,clPlId}> cl;`
|
||||
`NonComplexArray<CdAiPrv {Int32 nPrvId; Float nPrvVa}> prv; NonComplexArray<Int32> tecs; Int32 fct;`
|
||||
`ComplexArray<CdAiApr {Int32 sid,tn0,tn1}> apr`.
|
||||
- `CdAiSit` (complex): `NestedInt32 aiSitSecs; NestedInt32 aiSitWepFams` (**weapon-family set** lives here — opaque ints).
|
||||
- `CdAiCmbr` (complex): `Int32 crTrnK; Bool crPce; NestedInt32 crSys; CdAiCmbrCrplSv2 crplSv2`.
|
||||
`...CrplSv2`: `Int32 rpBon, rpBonT, savBonus; Bool maintHf; ComplexArray<TacReport> tacReports`.
|
||||
`TacReport`: `TrStruct trBy, trTo; TacReportDamage damageStruct; NonComplexArray<TacReportShips> ships`.
|
||||
`TrStruct`: `Int32 treHd, treHi, treD, treDp, treDi, treDt, treB`.
|
||||
`TacReportDamage` (leaf): `Int32 tRid, tRal, tRbal, tRlas, tRmis, tRmin, tRnrg, tRbio, tRbrd; Bool tRsld, tRsldd, tRsldc, tRsldi, tRsldr`
|
||||
— **damage-by-weapon-family breakdown**: bal(listic)/las(er)/mis(sile)/min(e)/nrg(=energy)/bio/brd(=boarding); sld=shields.
|
||||
`TacReportShips` (leaf): `Int32 tRships, tRsldr, tRshipL`.
|
||||
|
||||
---
|
||||
|
||||
## 13. Coverage gaps & contradictions
|
||||
|
||||
**Coverage (what's decoded):**
|
||||
- R1 decodes essentially the *entire* file top-to-bottom: summary, create-params/map-gen, full sim
|
||||
(players, tech, designs, systems/colonies incl. per-player fog-of-war snapshots, fleets, ships,
|
||||
trade, node grid, combat reports, grand-menace/encounter objects) and the CdTable AI block. This is
|
||||
the most complete community model and the primary Rosetta source.
|
||||
- R2 decodes only Summary, Player Settings, Players, Species, and Planets (colony) fields, by tag
|
||||
search — but adds *friendly semantics* and confirms the value-bytes width of several planet fields.
|
||||
|
||||
**Known-unknown fields (labelled `unknown*` in R1 — do NOT treat names as authoritative):**
|
||||
- All of `CdPlayer` (`unknown1..35`) and `CdAiAidngDnId`, `SimFleetOrigin`, `SimFleetDetailsFlightPlanFpogn2`,
|
||||
`SimTradeSectorTradeCtr`, `SimPlayerDesignDwgWgb`, `SimSystemDetailNvoIndcl*` bodies.
|
||||
- `SimFleetShipHealth` three floats guessed as command/mission/drive.
|
||||
- `PlanetSaveStruct.unknown1..4` (map-gen) unexplained.
|
||||
|
||||
**Not decoded / thin:**
|
||||
- Tactical/real-time combat geometry: only *reports/summaries* are stored (SimCrep*, CdAiCmbr TacReport).
|
||||
Per-ship in-battle positions/velocities are not in these editors (likely not in the sim save at all).
|
||||
- `SimSystemDetailSpy` body is empty in R1 (marked "needs to be populated"); spy detail unresolved.
|
||||
- RNG state (`RngSaveStruct.unknownData`) is an opaque ~2500-byte blob.
|
||||
- R1 source comments flag `SimSystemDetailSpiesArray`, `SimSvSctObXscnXsc` and
|
||||
`SimScSctObEncObjDetails` as incomplete/"wrong" in places — verify encounter bodies against binary.
|
||||
|
||||
**Contradictions between R1 and R2 (reconcile against binary):**
|
||||
1. **Field widths on planet flags:** R2 reads `Abdn`, `Dstyd`, `ltis` as Int16 and `Bats2` as Int32;
|
||||
R1 models `abdn/ltis` as Int32 and `bats2` as Int64. → The datum is small; check the true stored
|
||||
width in the binary struct.
|
||||
2. **PID vs OID:** R2 asserts `OID = PID*16` (owner id is player index << 4); R1 stores a single `pid`
|
||||
owner field and does not model the ×16 relationship. → Confirm whether the binary owner field is a
|
||||
raw index or a shifted/tagged handle.
|
||||
3. **`_NPC` species id 4:** R2 names it `_NPC`; R1 comment guesses "AI Rebellion". Same numeric id 4,
|
||||
different label — likely a shared "non-player/rogue" species slot.
|
||||
4. R2 treats Players and PlayerSettings as separate flat tab regions bounded by marker strings
|
||||
(`HomeSys`…`ISsp`, `Slot`…`Session`); R1 shows these are actually nested (settings inside the
|
||||
Summary PlayerSlot, live player data inside SimPlayerDetails). R2's region boundaries
|
||||
(`Summary`,`Slot`,`Session`,`HomeSys`,`ISsp`,`NumSys`,`NdGr2`,`PlayerIDs`,`DesignIDs`) are useful
|
||||
**section-marker strings** to locate blocks in the raw binary.
|
||||
|
||||
**High-value binary-scan signatures:**
|
||||
`0xBEEFBEEF` / `0x41104110` complex-struct brackets; length-prefixed ASCII field-name tags
|
||||
(`[int32 len][name]`) preceding every named scalar; section marker strings above.
|
||||
461
save-reader/ref/struct-recovery.md
Normal file
461
save-reader/ref/struct-recovery.md
Normal file
|
|
@ -0,0 +1,461 @@
|
|||
# Struct recovery via save-field-name xrefs — Sword of the Stars (2006)
|
||||
|
||||
Program `sots` / "Sword of the Stars.exe", ImageBase 0x00400000, 32-bit MSVC. All addresses are VAs.
|
||||
Method: every on-disk field is tagged with its name string; the name strings live in `.rdata`; the
|
||||
functions that reference dozens of a struct's names in sequence are its `IStreamable::Read`/`Write`.
|
||||
Decompiled those, mapped `this+offset` → name → type. Scripts (on CT111 `/root/`): `SerFind.java`
|
||||
(string→xref→function ranking), `SerDump.java`/`SerDump2.java` (decompile with `DAT_` → literal
|
||||
substitution), `VtOwner.java`/`VtOwner2.java` (find the owning vftable + RTTI Complete-Object-Locator
|
||||
offset), `SubWrite.java` (sub-struct Read/Write by class name). Raw decompiles: CT111 `/tmp/serdump/*.c`.
|
||||
|
||||
Reference cross-checked: `save-editor-structs.md` (R1 = Bardez editor, R2 = SOTSedit).
|
||||
|
||||
---
|
||||
|
||||
## 0. Serialization runtime facts (needed to read the tables)
|
||||
|
||||
### IStreamable vftable shape
|
||||
Every streamable class has a 3-slot vftable `{ [0] scalar-deleting dtor, [1] Read(Stream&), [2] Write(Stream&) }`
|
||||
(`Mars::IStreamable::vftable` @ 0x009e22bc = `{0x4f7230, 0x924fb0, 0x924fb0}`). `Mars::StreamableHelper<T>` /
|
||||
`Mars::VectorHelper<T>` are thin adaptors: slot [1]/[2] call the object's virtual Read/Write (or, for
|
||||
POD types like Vector3, a free function).
|
||||
|
||||
### this-adjustment (IMPORTANT for offsets)
|
||||
The serializer is called through the class's **IStreamable sub-vftable**, whose RTTI COL `offset` field
|
||||
gives the sub-object offset. The vftable slots point straight at the functions (no adjustor thunks;
|
||||
prologues verified in the exe: `55 8b ec 6a ff 68 …`), so inside each function `this` = object + COL
|
||||
offset. **Absolute member offset = decompiled offset + COL offset.** Tables below give both.
|
||||
|
||||
| Class | IStreamable vftable (COL offset) | Read | Write | Primary vftable |
|
||||
|---|---|---|---|---|
|
||||
| `Game::ServerSystem` (: StarSystem : StarMapNode) | 0x00a2043c (**+8**) | `FUN_0075d4b0` | `FUN_00749630` | (StarSystem primary 0x00a200e4) |
|
||||
| `Game::StarSystem` / `ClientSystem` (StarMapNode part) | 0x00a200d4 / 0x00a20144 (+8) | `FUN_00727790` | `FUN_00727820` | 0x00a200e4 / 0x00a20154 |
|
||||
| `Game::StarMapNode` | 0x00a1e620 (+8) | `FUN_00727790` | `FUN_00727820` | 0x00a1e630 |
|
||||
| `Game::ServerPlayer` (: StrategyPlayer) | 0x00a32794 (**+0x3a0 = 928**) | `FUN_008804d0` | `FUN_008563e0` | 0x00a327a4 (COL 0, 8 slots) |
|
||||
| `Game::StarShip` | 0x00a31408 (**+8**) | `FUN_00853fa0` | `FUN_008291f0` | 0x00a31418 (2 slots) |
|
||||
| `Game::StarFleet` (: StarMapNode) | 0x00a1d5f8 (**+8**) | `FUN_00702470` | `FUN_00701070` | 0x00a1d608 |
|
||||
| `Game::StrategyServer` (whole sim block) | 0x00a26084 (+0) | `FUN_007d27a0` | `FUN_0079fa70` | 0x00a26034 (COL 4) |
|
||||
| `Game::StarSystem::PlayerView` | 0x00a201ac (+0) | `FUN_00752af0` | `FUN_007492d0` | — |
|
||||
| `Game::StarSystem::OutputRates` (POD, via helper) | helper 0x00a1f884 | `FUN_007472a0` | `FUN_00745190` | — |
|
||||
| `Game::Population` | 0x009f90f0 (+0) | `FUN_005390c0` | `FUN_00537ef0` | — |
|
||||
| `Game::PopulationGroup` | 0x009f8d50 (+0) | `FUN_00536a80` | `FUN_00536af0` | — |
|
||||
| `Game::IndependenceInfo` | 0x00a2005c (+0) | `FUN_00748df0` | `FUN_00748ee0` | — |
|
||||
| `Game::Morale` / `MoraleEvent` | 0x00a1f7c8 / 0x00a2003c | `FUN_00744dd0` / `FUN_007490b0` | `FUN_00744ea0` / `FUN_007491b0` | — |
|
||||
| `Game::ShipBuildOrder(Def)` | 0x00a0c160 / 0x00a0ad08 | `FUN_00813770` | `FUN_00813800` | — |
|
||||
| `Game::PlayerNotes` | 0x00a21948 | `FUN_00813250` | `FUN_008132b0` | — |
|
||||
| `Game::SpyReport` | 0x00a32b2c | `FUN_008843d0` | `FUN_00828ec0` | — |
|
||||
| `Game::PlayerReport` (preps) | 0x00a21440 | `FUN_008200a0` | `FUN_00817480` | — |
|
||||
| `Game::DiplomacyStats` | 0x00a21430 | — | `FUN_00818cb0` | — |
|
||||
| `Game::FlightPlan` / `::Waypoint` / `NodeRoute` | 0x00a1d50c / 0x00a1d39c / 0x00a1cbdc | `FUN_00704c70` / `FUN_00701860` / `FUN_006e2260` | `FUN_00700f60` / `FUN_00700ed0` / `FUN_006e22e0` | — |
|
||||
| `Game::PrisonerHold` | 0x009fe130 | `FUN_0056eb00` | `FUN_0056ec00` | — |
|
||||
| `Game::EventStorage` / `PlayerAlliances` / `ShipHealth` / `PlayerColorID` / `Mars::Vector3` | — | — | `FUN_00825cc0` / `FUN_006d2e10` / `FUN_00813e50` / `FUN_0053c080` / `FUN_008a60d0` | — |
|
||||
|
||||
### Stream primitive API (writer side; `Stream` object vftable, `this` = stream)
|
||||
| call | meaning | wrapper used by serializers |
|
||||
|---|---|---|
|
||||
| vft+0x18 `(name, std::string*)` | write string | `FUN_008b9d70(stream,name,std::string*)` |
|
||||
| vft+0x1c `(name, byte)` | write bool | `FUN_008b9c20(stream,name,bool*)` |
|
||||
| vft+0x20 `(name, float)` | write float | `FUN_008b9be0(stream,name,float*)` |
|
||||
| vft+0x24 `(name, int, default=-1)` | write int32 | `FUN_008b9d50(stream,name,int*)`; `FUN_008b9d00(stream,name,int16*)` (widens short→int) |
|
||||
| vft+0x28 `(name, IStreamable-helper*)` | write nested object (BEEFBEEF frame) | inline `StreamableHelper<T>{vft, 0, T*}` |
|
||||
| vft+0x30 `(name, ptr, nbytes)` | write raw bytes | used for 8-byte Int64s |
|
||||
| `FUN_00816490(stream,name,obj*)` | write **handle id** = `obj ? obj->id(+4) : 0` | NetworkObject id at +4 |
|
||||
| `FUN_008b9c60(stream,name,int64*)` | write int64 (PopC) | |
|
||||
Reader side mirrors: `FUN_008b9bc0` float, `FUN_008b9d20` int, `FUN_008b9c00` bool, `FUN_008b9d90` string,
|
||||
`FUN_008b9c40` int64, `FUN_008b9cd0` short, `FUN_008164d0(stream,name)` handle→object* lookup,
|
||||
stream vft+0x10 int-by-ref (returns found flag), vft+0x14 nested object (NULL helper = skip/legacy).
|
||||
Readers accept legacy tags (`ISuit`, `Income`, `HPop`, `Bats`, `Builds`, `Clr`, `SensMod`, `ExPopSys`,
|
||||
`NShps`, `SysID`, `TrdID`, `Caps`, `GtTrf`, `FtSens`, `FtInc`, `Pris`, `NumPlgs`, `lcid`, `morev`, `cme`)
|
||||
by reading them into scratch/NULL — these are pre-1.8 fields, NOT members.
|
||||
|
||||
Common Mars/MSVC layouts seen: `std::string` = 0x1c bytes (MSVC10 `_Bx` union@0, size@0x14, res@0x18;
|
||||
`FUN_008b9d70` does the `res>=16 ? heap : sso` check); `std::vector<T>` = {begin@0, end@4, cap@8};
|
||||
`std::map/set` node = {left@0, parent@4, right@8, key@0xc, value@0x10, …, color/isnil bytes at tail};
|
||||
`std::list` = {head*@0, size@4}. `Mars::NetworkObject` = {vptr@0, int id@4}.
|
||||
|
||||
---
|
||||
|
||||
## 1. `Game::ServerSystem` (= live star system + colony record; R1 `SimSystemDetailsSaveStruct`)
|
||||
|
||||
Serializers: Write `FUN_00749630` @ 0x00749630 (3453 B), Read `FUN_0075d4b0` @ 0x0075d4b0 (8320 B, has
|
||||
legacy branches). Both begin with `StarMapNode::Write/Read` (`FUN_00727820`/`FUN_00727790`) which
|
||||
emits `Pos`. `this` = obj+8. Base layout: `+0` primary vptr (StarSystem 0x00a200e4), `+4` NetworkObject id,
|
||||
`+8` IStreamable vptr, `+0xc` HandleObject vptr, `+0x10` owner pointer (`*(+0x10)->+0x50` = player-object
|
||||
table indexed by map key), `+0x18` Pos.
|
||||
|
||||
| abs off | rel(this+8) | type | save name | notes |
|
||||
|---|---|---|---|---|
|
||||
| 0x18 | 0x10 | `Mars::Vector3` (3 floats) | `Pos` | via StarMapNode; on disk 3 unnamed floats |
|
||||
| 0x4c | 0x44 | float | `R` | starColor.r |
|
||||
| 0x50 | 0x48 | float | `G` | |
|
||||
| 0x54 | 0x4c | float | `B` | |
|
||||
| 0x58 | 0x50 | float | `A` | |
|
||||
| 0x5c | 0x54 | int | `Idx` | system index |
|
||||
| 0x60 | 0x58 | int | `Size` | 1–10 |
|
||||
| 0x64 | 0x5c | float | `Suit` | climate hazard (legacy `ISuit` discarded) |
|
||||
| 0x68 | 0x60 | int | `Res` | |
|
||||
| 0x6c | 0x64 | int | `ARes2` | (legacy `ARes` read then overwritten) |
|
||||
| 0x70 | 0x68 | int | `MRes` | |
|
||||
| 0x74 | 0x6c | int | `TRes` | |
|
||||
| 0x78..0x7a | 0x70..0x72 | bool[3] | `haltv` | written as `haltc`=3, then 3×(`haltt`=i, `haltv`=v[i]) |
|
||||
| 0x7c | 0x74 | float | `OutMod` | |
|
||||
| 0x80 | 0x78 | int | `TAcq` | |
|
||||
| 0x84 | 0x7c | int | `TFAcq` | |
|
||||
| 0x88..0xa3 | 0x80 | `StarSystem::OutputRates` (0x1c) | `Rts` | see §1.1; nested object |
|
||||
| 0xa4 | 0x9c | `BuildQueue*` | `BQ` | written only if owner (`PID`) non-null |
|
||||
| 0xa8..0xc3 | 0xa0 | `std::string` | `Name` | |
|
||||
| 0xc4 | 0xbc | bool | `Abdn` | **bool in memory** (R2 "short" is just the value byte; R1 Int32 is the framing) |
|
||||
| 0xc5 | 0xbd | bool | `Dstyd` | |
|
||||
| 0xc6 | 0xbe | bool | `vnh` | gate: if true → `vnd`,`vnex3`,`vnpex3` |
|
||||
| 0xc7 | 0xbf | bool | `vnd` | |
|
||||
| 0xc8 | 0xc0 | bool | `vnex3` | |
|
||||
| 0xc9 | 0xc1 | bool | `vnpex3` | |
|
||||
| 0xcc | 0xc4 | int | `VFlags` | written by value (int) |
|
||||
| 0xd0 | 0xc8 | int | `EFlags` | |
|
||||
| 0xd4 | 0xcc | int | `AFlags` | |
|
||||
| 0xd8 | 0xd0 | int | `FFlags` | |
|
||||
| 0xdc | 0xd4 | int | `GFlags` | |
|
||||
| 0xe0 | 0xd8 | int | `MnRFlags` | |
|
||||
| 0xe4 | 0xdc | int | `RfRFlags` | |
|
||||
| 0xe8 | 0xe0 | int | `ClkFlags` | |
|
||||
| 0xf0 | 0xe8 | **int64** | `Bats2` | raw 8 bytes (R1 correct; R2 Int32 reads low half). Legacy `Bats` int |
|
||||
| 0xf8 | 0xf0 | **int64** | `rcex` | raw 8 bytes |
|
||||
| 0x100 | 0xf8 | `ServerPlayer*` | `PID` | owner; written as handle id (`->+4`) |
|
||||
| 0x104..0x117 | 0xfc | `Population` (0x14) | `dcs` | nested |
|
||||
| 0x118 | 0x110 | float | `dsu` | |
|
||||
| 0x11c..0x13b | 0x114 | `Morale` (0x20: vptr + int[7]) | `cm` | nested |
|
||||
| 0x13c..0x147 | 0x134 | `vector<MoraleEvent>` | `cme2` | VectorHelper |
|
||||
| 0x14c..0x16b | 0x144 | `Morale` | `PvCM` | |
|
||||
| 0x16c..0x177 | 0x164 | `vector<StarFleet*>` | `NumFlts` + n×`Flt` | ids via handle |
|
||||
| 0x17c | 0x174 | float | `RepCur` | |
|
||||
| 0x180 | 0x178 | float | `RepMax` | |
|
||||
| 0x184 | 0x17c | int | `EggScio` | |
|
||||
| 0x188 | 0x180 | bool | `NoRebAI` | |
|
||||
| 0x189 | 0x181 | bool | `PvNoRebAI` | |
|
||||
| 0x18c | 0x184 | int | `Pop` | imperial pop |
|
||||
| 0x190 | 0x188 | float | `Infra` | |
|
||||
| 0x194 | 0x18c | int | `pbon` | |
|
||||
| 0x198 | 0x190 | float | `ibon` | |
|
||||
| 0x19c | 0x194 | int | `TerrFl` | |
|
||||
| 0x1a0..0x1b3 | 0x198 | `Population` | `Pop2` | civilian pop groups (R1 `popG`) |
|
||||
| 0x1b4..0x1c7 | 0x1ac | `Population` | `pbon2` | |
|
||||
| 0x1c8 | 0x1c0 | `IndependenceInfo*` | `hindi` + `indi` | `hindi` = ptr!=NULL |
|
||||
| 0x1cc..0x1d7 | 0x1c4 | `vector<int>` | `spies2` | VectorHelper<int> |
|
||||
| 0x1dc | 0x1d4 | int | `rbfl` | written by value |
|
||||
| 0x1e0 | 0x1d8 | bool | `hsrg` | |
|
||||
| 0x1e4..0x1ff | 0x1dc | int[7] | `nadct`,(`ads`=i,`adt`=v) | addiction table: count of non-zero entries then sparse (index,value) pairs |
|
||||
| 0x200 | 0x1f8 | int | `PvPop` | previous-turn snapshot block |
|
||||
| 0x204 | 0x1fc | float | `PvInfra` | |
|
||||
| 0x208 | 0x200 | float | `PvSuit` | |
|
||||
| 0x20c | 0x204 | int | `PvRes` | |
|
||||
| 0x210 | 0x208 | int | `PvARes2` | |
|
||||
| 0x214 | 0x20c | int | `PvMRes` | |
|
||||
| 0x218..0x22b | 0x210 | `Population` | `PvPop2` | |
|
||||
| 0x238 | 0x230 | `StarFleet*` | `DefF` | handle id |
|
||||
| 0x23c | 0x234 | `StarFleet*` | `DefSF` | handle id |
|
||||
| 0x240..0x24b | 0x238 | `vector<obj*>` | `NumGFs` + n×`GF` | gates |
|
||||
| 0x250..0x25b | 0x248 | `vector<obj*>` | `NumSnF` + n×`SnF` | stations |
|
||||
| 0x260..0x26b | 0x258 | `vector<obj*>` | `NumMnF` + n×`MnF` | monitors |
|
||||
| 0x274 / 0x278 | 0x26c / 0x270 | `std::map` head / size | `NVO` + entries | colonies, see §1.2 |
|
||||
| 0x284 / 0x288 | 0x27c / 0x280 | `std::map` head / size | `NVE` + entries | §1.2 |
|
||||
| 0x294 / 0x298 | 0x28c / 0x290 | `std::map` head / size | `NVs` + entries | per-player `pview`, §1.3 |
|
||||
| 0x2a8..0x2b3 | 0x2a0 | `vector<Plague*>` | `NumPlgs2` + n×(`PlgT`=plg->+4, `Plg` obj) | |
|
||||
| 0x2b8 | 0x2b0 | int | `TnsOH` | |
|
||||
| 0x2bc | 0x2b4 | int | `TDst` | |
|
||||
| 0x2c4 | 0x2bc | int | `ntdev` | |
|
||||
| 0x2c8 | 0x2c0 | int | `ltis` | **int** (R2 "short" wrong width) |
|
||||
| 0x2cc | 0x2c4 | int | `rbtn` | |
|
||||
| 0x2d0 | 0x2c8 | int | `rbfr` | by value |
|
||||
| 0x2d4 | 0x2cc | int | `rbwn` | |
|
||||
|
||||
Object size ≥ 0x2d8. On-disk order = R1 §8 exactly (Pos, RGBA, Idx, Size, Suit, Res, ARes2, MRes, NoRebAI,
|
||||
TRes, Pop, Pop2, Infra, PvPop, PvPop2, PvInfra, PvSuit, PvRes, PvARes2, PvMRes, PvNoRebAI, Rts, Abdn, Dstyd,
|
||||
TnsOH, OutMod, RepCur, RepMax, ntdev, pbon, pbon2, ibon, ltis, rbfl, rbtn, rbfr, rbwn, hsrg, halt*, vn*, Name,
|
||||
*Flags, Bats2, rcex, Mn/Rf/ClkFlags, EggScio, TerrFl, TAcq, TFAcq, TDst, dcs, dsu, cm, PvCM, cme2, spies2, PID,
|
||||
DefF, DefSF, BQ, nadct/ads/adt, NumPlgs2…, NumFlts/GFs/SnF/MnF, NVO, NVE, NVs, hindi/indi).
|
||||
|
||||
Members that are only ever read with a NULL/scratch target (not stored): `ISuit`, `Income`, `HPop`, `Builds`
|
||||
(+`Con`,`Sav`,`ConLeft`,`OrID`,`DesID` — old inline build queue), `Slvs`, `dct`, `cme`, `Bats`, `NumPlgs`.
|
||||
|
||||
### 1.1 `Game::StarSystem::OutputRates` (POD, 0x1c) — Write `FUN_00745190`
|
||||
Memory order ≠ disk order: `+0x00 float SRt`, `+0x04 SRsc`, `+0x08 SRtf`, `+0x0c SRi`, `+0x10 SRoh`,
|
||||
`+0x14 SRs`, `+0x18 int SRnr`. Disk order: SRs, SRt, SRsc, SRtf, SRi, SRoh, SRnr. Reader: if `SRs` tag is
|
||||
absent, reads 5 unnamed floats (legacy).
|
||||
|
||||
### 1.2 Colony maps `NVO` / `NVE` (std::map keyed by player-table index)
|
||||
Write emits `NVO`=size, then per node: `PID` = handle id of `owner->+0x50[key]` (player object table),
|
||||
`TShn` = **int16** at node+0x12 (value+2), `OID` = int at node+0x14 (value+4) by value, `isind` bool at
|
||||
node+0x18 (value+8), `indi` = inline `IndependenceInfo` at node+0x1c (value+0xc, 0x70 bytes). Node isnil
|
||||
byte at +0x8d ⇒ value size 0x7c. `NVE` nodes: `EPid` handle (key→player), `ETS` int16 @ node+0x12, `Eid`
|
||||
int @ node+0x14 (isnil @ +0x19 ⇒ value 8 bytes).
|
||||
`OID` is a stored int, distinct from `PID`; the R2 claim "OID = PID×16" is an id-allocation pattern, not
|
||||
a derivation in this code (open question — check the HandleObject id allocator).
|
||||
|
||||
### 1.3 `Game::StarSystem::PlayerView` (per-player seen snapshot) — Write `FUN_007492d0`, Read `FUN_00752af0`
|
||||
Stored inline as map value at node+0x10 (`NVs`; node isnil @ +0xad ⇒ value ≈ 0x9c).
|
||||
`+0 vptr (0x00a201ac)`, `+8 int VTrn`, `+0xc int Pop`, `+0x10 Population Pop2 (0x14)`, `+0x24 float Infra`,
|
||||
`+0x28 float Suit`, `+0x2c int Res`, `+0x30 int ARes2`, `+0x34 int MRes`, `+0x38 bool NoRebAI`, `+0x3c int pbon`,
|
||||
`+0x40 Population pbon2`, `+0x54 float ibon`, `+0x58 int TerrFl`; trailer bool `footer`=1. Reader also accepts
|
||||
legacy `ARes`, `PvPop/PvInfra/PvSuit/PvRes/PvARes/PvARes2/PvMRes/PvNoRebAI` into the same slots.
|
||||
(R1 lists `Int32 infra` — it is a **float**.)
|
||||
|
||||
### 1.4 `Game::Population` (0x14) / `Game::PopulationGroup` (0x18 stride)
|
||||
Population: `+0 vptr`, `+4/+8/+0xc vector<PopulationGroup>`; Write emits `PopNG` = count of groups with
|
||||
PopC>0 (or ≥0 with low word ≠0), then each as nested `PopG`. PopulationGroup: `+4 int PopT`, `+8 int PopS`,
|
||||
`+0x10 int64 PopC` (R1 `popT,popS,popC` ✓).
|
||||
|
||||
### 1.5 `Game::IndependenceInfo` (0x70) — Write `FUN_00748ee0`
|
||||
`+4 int indsp`, `+8 PlayerColorID indcl` (nested), `+0x1c string indnm`, `+0x38 string indav`, `+0x54 string indba`.
|
||||
|
||||
### 1.6 `Game::Morale` / `Game::MoraleEvent`
|
||||
Morale (0x20): `+0 vptr`, `+4 int[7]`; disk: `mnsp`=n then n×(`msp`=index, `mv`=value) (reader tolerates
|
||||
missing `mnsp` → 7 fixed entries, skipping index 4). MoraleEvent: `+4 mid`, `+8 mtr`, `+0xc mn`, `+0x10 mtp`
|
||||
(ints), `+0x14 Morale mfx`, `+0x34 string mdsc`.
|
||||
|
||||
### 1.7 `Game::ShipBuildOrder` (build-queue entry) — Write `FUN_00813800`
|
||||
`+4 int desID`, `+8 int con`, `+0xc int sav`, `+0x10 int conleft`, `+0x14 int ordID`; disk order desID, con,
|
||||
conleft, sav, ordID (R1 ✓).
|
||||
|
||||
### Where is "Planet"?
|
||||
`Game::Planet : Actor` (vft 0x009ef144) is a **render/scene actor** and is not streamed. The colony/planet
|
||||
state the save calls "planet" (R2's Idx/Name/Size/Suit/Res/Infra/Pop/OID…) is entirely in `ServerSystem`
|
||||
above plus `PlayerView`. The CreateParameters `PlanetSaveStruct` (x,y,z + 4 ints) is map-gen input
|
||||
(`StarMapParams`), not touched here.
|
||||
|
||||
---
|
||||
|
||||
## 2. `Game::ServerPlayer` (empire; R1 `SimPlayerDetailsSaveStruct`)
|
||||
|
||||
Serializers: Write `FUN_008563e0` @ 0x008563e0 (4040 B), Read `FUN_008804d0` @ 0x008804d0 (7647 B).
|
||||
IStreamable sub-object at **+0x3a0** (COL offset 928); `this` = obj+0x3a0, so decompiled offsets are
|
||||
negative for most members. Primary vptr @+0 (0x00a327a4, StrategyPlayer shape), NetworkObject id @+4.
|
||||
|
||||
| abs off | rel(this+0x3a0) | type | save name | notes |
|
||||
|---|---|---|---|---|
|
||||
| 0x28 | -0x378 | int | `PlyrIdx` | |
|
||||
| 0x2c | -0x374 | `ServerSystem*` | `HomeSys` | handle id |
|
||||
| 0x30/0x34 | -0x370/-0x36c | `vector<ServerPlayer*>` | `NumOwn` + n×`OwnId` | handle ids (R1 `ownerIds`) |
|
||||
| 0x40..0x5b | -0x360 | `std::string` | `PlryName` | |
|
||||
| 0x5c | -0x344 | int | `Species` | 0 Human … 6 Morrigi |
|
||||
| 0x60 | -0x340 | `PlayerColorID` (4 B) | `ClrID` | nested; §2.1 (legacy `Clr` int skipped) |
|
||||
| 0x74..0x8f | -0x32c | `std::string` | `Bdg` | badge |
|
||||
| 0x90..0xab | -0x310 | `std::string` | `Avt` | avatar |
|
||||
| 0xac | -0x2f4 | int | `Team` | |
|
||||
| 0xb0 | -0x2f0 | float | `IdealSuit` | |
|
||||
| 0xb4 | -0x2ec | float | `SuitTol` | |
|
||||
| 0xb8 | -0x2e8 | float | `MaxOH` | |
|
||||
| 0xbc | -0x2e4 | float | `ResRate` | |
|
||||
| 0xc0 | -0x2e0 | float | `ResMod` | |
|
||||
| 0xc4 | -0x2dc | float | `ResScl` | |
|
||||
| 0xd0 | -0x2d0 | **float** | `TRM` | R1 says Int32 — it is float |
|
||||
| 0xd4 | -0x2cc | int | `TRA` | note memory order TRA before TRP |
|
||||
| 0xd8 | -0x2c8 | int | `TRP` | |
|
||||
| 0xe4/0xe8 | -0x2bc/-0x2b8 | `vector<ShipDesign*>` | `NumDes` + n×(`DesID`=d->+0xa4, `Des` obj) | current designs |
|
||||
| 0xf4 | -0x2ac | `TechTree*` | `TechTree` | first field on disk |
|
||||
| 0xf8 | -0x2a8 | bool | `Elim` | |
|
||||
| 0xfb | -0x2a5 | bool | `NPC` | |
|
||||
| 0xfc | -0x2a4 | bool | `RebAI` | |
|
||||
| 0xfd | -0x2a3 | bool | `ReqCL` | |
|
||||
| 0xfe | -0x2a2 | bool | `AIBn` | |
|
||||
| 0xff | -0x2a1 | bool | `CnTrd` | |
|
||||
| 0x100 | -0x2a0 | bool | `CnRad` | |
|
||||
| 0x101 | -0x29f | bool | `CnVItl` | |
|
||||
| 0x102 | -0x29e | bool | `hgs` | |
|
||||
| 0x103 | -0x29d | bool | `hadvs` | |
|
||||
| 0x104 | -0x29c | bool | `harcc` | |
|
||||
| 0x108 | -0x298 | float | `pddm` | |
|
||||
| 0x10c..0x117 | -0x294 | float[3] | `ConMod` ×3 | interleaved on disk as 3×(ConMod[i], SavMod[i]) |
|
||||
| 0x118..0x123 | -0x288 | float[3] | `SavMod` ×3 | |
|
||||
| 0x124 | -0x27c | float | `OutMod` | |
|
||||
| 0x128 | -0x278 | float | `RebOutMod` | |
|
||||
| 0x12c | -0x274 | float | `ScOutMod` | |
|
||||
| 0x130 | -0x270 | float | `PopMod` | (legacy `SensMod` float, `ExPopSys` int skipped between IncMod/PopMod/TerraMod) |
|
||||
| 0x134 | -0x26c | float | `TerraMod` | |
|
||||
| 0x138 | -0x268 | bool | `AMine` | |
|
||||
| 0x13c | -0x264 | float | `MinPure` | |
|
||||
| 0x140 | -0x260 | float | `MinRate` | |
|
||||
| 0x144 | -0x25c | int | `NGts` | |
|
||||
| 0x148 | -0x258 | int | `PrGtTrf` | |
|
||||
| 0x14c | -0x254 | int | `GTraf` | |
|
||||
| 0x150 | -0x250 | **float** | `CstR` | R1 Int32 → float |
|
||||
| 0x154 | -0x24c | **float** | `CstE` | |
|
||||
| 0x158 | -0x248 | **float** | `CstT` | |
|
||||
| 0x15c | -0x244 | int | `Maint` | (legacy `NShps` skipped) |
|
||||
| 0x160 | -0x240 | **float** | `shrm` | |
|
||||
| 0x164 | -0x23c | int | `Status` | by value |
|
||||
| 0x168..0x177 | -0x238 | `PlayerAlliances` {int ALid, AL, NA, CF} | `Team` (2nd) | nested (R1 `teamStruct`) |
|
||||
| 0x178/0x17c | -0x228/-0x224 | `vector<ShipDesign*>` | `NumLeg` + n×(`DesID`,`Des`) | legacy/drone designs (R1 `droneDesigns`) |
|
||||
| 0x188 | -0x218 | int | `PvSav` | |
|
||||
| 0x18c | -0x214 | bool | `PvMA` | |
|
||||
| 0x19c | -0x204 | int | `HasDisc` | by value |
|
||||
| 0x1a0 | -0x200 | int | `HasDiscSp` | |
|
||||
| 0x1a4 | -0x1fc | int | `HasDiscCl` | |
|
||||
| 0x1a8 | -0x1f8 | int | `HasEnc` | |
|
||||
| 0x1ac | -0x1f4 | int | `HasEng` | |
|
||||
| 0x1b0 | -0x1f0 | `ShipRecords` (inline) | `ShipRecs` | Write `FUN_008176a0` |
|
||||
| 0x1f4 | -0x1ac | `vector<Objective>` | `Ojvs` | VectorHelper |
|
||||
| 0x204/0x208 | -0x19c/-0x198 | `vector<{int xid,xmin,xmax; float xper}>` (16 B) | `Nexp` + n×(`xid`,`xmin`,`xmax`,`xper`) | R1 misses the per-entry body |
|
||||
| 0x214/0x218 | -0x18c/-0x188 | `vector<int>` | `NWeapXcl` + n×`WeapXcl` | |
|
||||
| 0x230 | -0x170 | `vector<DiplomacyStats>` | `dipstats` | §2.2 |
|
||||
| 0x240 | -0x160 | `CommMessageContainer*` | `comms` | |
|
||||
| 0x244 | -0x15c | `vector<PlayerReport>` | `preps` | §2.3 |
|
||||
| 0x254 | -0x14c | `vector<ObservedDesign>` | `odes` | |
|
||||
| 0x264 | -0x13c | `vector<ObservedWeapon>` | `owep` | |
|
||||
| 0x274 | -0x12c | `vector<ObservedTech>` | `otch` | |
|
||||
| 0x284 | -0x11c | int | `Sav` | savings |
|
||||
| 0x288 | -0x118 | int | `HasImm` | by value |
|
||||
| 0x28c | -0x114 | int | `HasVac` | |
|
||||
| 0x290 | -0x110 | int | `NPTrk` | |
|
||||
| 0x294 | -0x10c | `Tech*` (current research) | `ResTNm` | writes `tech ? tech->name(+4) : ""` |
|
||||
| 0x298 | -0x108 | `FleetNameGenerator*` | `FNG` | |
|
||||
| 0x29c | -0x104 | `EventStorage` (inline) | `Events` | {`EvNxID`@+0x14, `Events` vector<TurnEvents>@+4} |
|
||||
| 0x2b8/0x2bc | -0xe8/-0xe4 | `std::list<PlayerNotes>` head/size | `NumNotes` + n×`Nts` | node value at +0x10: `NtSys`@+4,`NtTxt` str@+8,`NtTrn`@+0x24 |
|
||||
| 0x2c4 | -0xdc | int | `BnkWrn` | by value |
|
||||
| 0x2c8 | -0xd8 | int | `BnkTrn` | |
|
||||
| 0x2cc | -0xd4 | int | `BnkEl` | |
|
||||
| 0x2d0 | -0xd0 | int | `BnkPr` | |
|
||||
| 0x2d8 | -0xc8 | int | `plcy` | by value |
|
||||
| 0x2dc..0x2f7 | -0xc4 | **`std::string`** | `pswd` | R1 says Int32 — it is a string |
|
||||
| 0x2f8 | -0xa8 | bool | `Srn` | |
|
||||
| 0x2fc | -0xa4 | `obj*` | `SrnTo` | handle id (R1 `srcTo`) |
|
||||
| 0x300 | -0xa0 | int | `lboid` | |
|
||||
| 0x304 | -0x9c | int | `lcid2` | by value (legacy `lcid`) |
|
||||
| 0x30c | -0x94 | float | `IncMod` | |
|
||||
| 0x310 | -0x90 | `vector<PlayerAid>` | `aid` | |
|
||||
| 0x320/0x324 | -0x80/-0x7c | `vector<DefenceLayout*>` | `ndeflay` + n×`deflay` | |
|
||||
| 0x330 | -0x70 | bool | `cdp` | |
|
||||
| 0x334 | -0x6c | `SpyReport*` | `spy2` | §2.4 |
|
||||
| 0x338/0x33c | -0x68/-0x64 | `vector<RaidTargets>` (0x20 stride) | `rdtc` + n×`rdt` | |
|
||||
| 0x368 | -0x38 | int | `aidf` | by value |
|
||||
| 0x370 | -0x30 | `CivilianRatios` (inline) | `civr` | Write `FUN_0082c740` |
|
||||
| 0x39c | -4 | int | `tnc` | written as max(v,1) |
|
||||
| **0x3a0** | 0 | vptr | — | IStreamable sub-vftable 0x00a32794 |
|
||||
| 0x3a4/0x3a8 | +4/+8 | `vector<{float PRm; int PRBt}>` | `NumPR` + n×(`PRm`,`PRBt`) | |
|
||||
| 0x3b4 | +0x14 | bool | `ResErrRoll` | |
|
||||
| 0x3b5 | +0x15 | bool | `cta` | |
|
||||
| 0x3b8 | +0x18 | `AIRebellion*` | `HasAIR` + `AIR` | gate = ptr!=NULL |
|
||||
| 0x3bc | +0x1c | `AIEncounterFlags*` | `AIEnf` | |
|
||||
| 0x3c0/0x3c4 | +0x20/+0x24 | `vector<SpecialProjectImpl*>` | `NSprj` + n×(`SprjT`=p->+0x3c, `Sprj`) | |
|
||||
| 0x3d0 | +0x30 | int | `NextPrjID` | |
|
||||
| 0x3d4 | +0x34 | int | `lret` | |
|
||||
| 0x3dc | +0x3c | int | `nmeid` | |
|
||||
|
||||
Object size ≥ 0x3e0. Disk order = R1 §6 (TechTree, HomeSys, PlyrIdx, PlryName, Species, ClrID, Bdg, Avt,
|
||||
Team, Sav, IdealSuit, SuitTol, MaxOH, ResRate, ResMod, ResScl, TRM, TRP, TRA, OutMod, RebOutMod, ScOutMod,
|
||||
IncMod, PopMod, TerraMod, AMine, MinPure, MinRate, NGts, PrGtTrf, GTraf, CstR/E/T, Maint, shrm, Status, Elim,
|
||||
NPC, RebAI, ReqCL, Team{ALid,AL,NA,CF}, HasVac, HasImm, NPTrk, HasDisc, HasDiscSp, HasDiscCl, HasEnc, HasEng,
|
||||
Events, FNG, PvSav, PvMA, AIBn, CnTrd, CnRad, hgs, hadvs, harcc, CnVItl, pddm, BnkWrn/Trn/Pr/El, ShipRecs,
|
||||
NextPrjID, plcy, pswd, lret, nmeid, cdp, spy2, civr, aidf, Srn, SrnTo, lboid, lcid2, ResTNm, ResErrRoll,
|
||||
3×(ConMod,SavMod), NumOwn/OwnId, NumDes/DesID/Des, NumLeg/DesID/Des, NumNotes/Nts, NumPR/PRm/PRBt, HasAIR/AIR,
|
||||
cta, AIEnf, NSprj/SprjT/Sprj, Nexp/xid/xmin/xmax/xper, NWeapXcl/WeapXcl, Ojvs, dipstats, comms, preps, odes,
|
||||
owep, otch, aid, ndeflay/deflay, rdtc/rdt, tnc). Note: on-disk `Sav` comes right after `Team` though it
|
||||
lives at 0x284 in memory.
|
||||
|
||||
### 2.1 `Game::PlayerColorID` (4 bytes) — Write `FUN_0053c080`
|
||||
`+0 int8 index; +1,+2,+3 uint8 r,g,b`; writer emits index via `FUN_008b9cb0` (char→int on disk), and
|
||||
**iff index == -1** the three r,g,b bytes (matches R1 §1.6). Same struct used by `IndependenceInfo.indcl`.
|
||||
|
||||
### 2.2 `Game::DiplomacyStats` (0x24) — Write `FUN_00818cb0`
|
||||
`+4 int other`; then **int16** fields at +8 `lastnap`, +0xa `lastnapbty`, +0xc `bknnap`, +0xe `btynap`,
|
||||
+0x10 `lastally`, +0x12 `lastallybty`, +0x14 `bknally`, +0x16 `btyally`, +0x18 `lastcf`, +0x1a `lastcfbty`,
|
||||
+0x1c `bkncf`, +0x1e `btycf`, +0x20 `deadhome` (all widened to int32 on disk; R1 `nap/ally/cf{last_,last_bty,bkn_,bty_}` ✓).
|
||||
|
||||
### 2.3 `Game::PlayerReport` (preps, 0x30) — Write `FUN_00817480`
|
||||
ints `+4 oid, +8 pid, +0xc flds, +0x10 sav, +0x14 home, +0x18 ncol, +0x1c mpwr, +0x20 mcls, +0x24 mmsl, +0x28 nshp, +0x2c nsat` (R1 ✓).
|
||||
|
||||
### 2.4 `Game::SpyReport` — Write `FUN_00828ec0`
|
||||
Four `std::list`s: `+4 list<SpyReportDefences>` (count `defc2`@+8, items `def`), `+0x10 list<SpyReportTrade>`
|
||||
(`rtc`@+0x14, `strd`), `+0x1c list<SpyReportEvents>` (`evc`@+0x20, `evs`), `+0x28 list<SpyReportTechTree>`
|
||||
(`ttc`@+0x2c, `tt`). R1 only kept the four counts.
|
||||
|
||||
### 2.5 `Game::TechTree` — Write `FUN_005890a0` (tags `NumTechs`, `TNm`, `NumBrs`; per-tech body in a
|
||||
sub-writer not decompiled here; logs "TechTree: Tech %d not found saving tech tree").
|
||||
|
||||
---
|
||||
|
||||
## 3. `Game::StarFleet` (R1 `SimFleetDetails`) — Write `FUN_00701070`, Read `FUN_00702470`; `this` = obj+8
|
||||
|
||||
| abs | rel | type | save name | notes |
|
||||
|---|---|---|---|---|
|
||||
| 0x18 | 0x10 | Vector3 | `Pos` | via StarMapNode |
|
||||
| 0x4c | 0x44 | Vector3 | `PrvPos` | |
|
||||
| 0x58 | 0x50 | `ServerPlayer*` | `PID` | handle |
|
||||
| 0x5c..0x77 | 0x54 | string | `FtName` | |
|
||||
| 0x78 | 0x70 | bool | `Perm` | |
|
||||
| 0x7c | 0x74 | `FleetLayout` (inline, ~0x24) | `HLay` + `Lay` | gate = either of its two vectors (+4/+8, +0x14/+0x18) non-empty |
|
||||
| 0xa0 | 0x98 | `StarSystem*` | `LocID` | handle (legacy `SysID`,`TrdID` ints skipped) |
|
||||
| 0xa4/0xa8 | 0x9c/0xa0 | `vector<StarShip*>` | `NShips` + n×(`ShipID` handle, `Ship` obj) | |
|
||||
| 0xc4..0xfb | 0xbc | `FlightPlan` (inline, 0x38) | `HFPlan` + `FPlan` | gate = wpts non-empty; §3.1 |
|
||||
| 0xfc | 0xf4 | int | `FtTrans` | by value (legacy `Caps` int, `GtTrf` short, `FtSens` float, `FtInc` int skipped) |
|
||||
| 0x100 | 0xf8 | Vector3 | `FtOrig` | (R1 "3 ints" → 3 floats) |
|
||||
| 0x10c | 0x104 | int | `FtFlg` | |
|
||||
| 0x110 | 0x108 | int | `Ftae` | |
|
||||
| 0x114 | 0x10c | int | `Ftpae` | |
|
||||
| 0x118 | 0x110 | int | `FtEnc` | |
|
||||
| 0x11c | 0x114 | int | `FtMS` | |
|
||||
|
||||
### 3.1 `Game::FlightPlan` (0x38) / `Waypoint` / `NodeRoute`
|
||||
FlightPlan: `+0 vptr`, `+4 vector<Waypoint> wpts`, `+0x14 float FPsp2`, `+0x18 int FPeta2`, `+0x1c Vector3 FPogn2`,
|
||||
`+0x28 Vector3 FPdpos`, `+0x34 int pnd`. Waypoint: `+4 int Wpt`, `+8 int Tp`, `+0xc NodeRoute nrt`.
|
||||
NodeRoute: `+4 nrp`, `+8 nrf`, `+0xc nrt` (ints). Reader also handles legacy `NumWpt`/`path`/`FPognid`.
|
||||
|
||||
## 4. `Game::StarShip` (R1 `SimFleetShipDetails`) — Write `FUN_008291f0`, Read `FUN_00853fa0`; `this` = obj+8
|
||||
|
||||
| abs | rel | type | save name | notes |
|
||||
|---|---|---|---|---|
|
||||
| 0x10 | 0x8 | `ServerPlayer*` | `PlrID` | handle |
|
||||
| 0x14 | 0xc | `ShipDesign*` | `DesID` | writes `design->+0xa4` (design id) |
|
||||
| 0x20 | 0x18 | float | `Range` | |
|
||||
| 0x24..0x33 | 0x1c | `ShipHealth` {vptr; float[3]} | `Health` | 3 unnamed floats (R1 guess command/mission/drive) |
|
||||
| 0x34 | 0x2c | int | `MineCap` | |
|
||||
| 0x38/0x3c | 0x30/0x34 | `vector<{float th, thm}>` | `NTH` + n×(`TH`,`THM`) | |
|
||||
| 0x48 | 0x40 | int | `Plg` | |
|
||||
| 0x4c | 0x44 | int | `Act` | |
|
||||
| 0x50 | 0x48 | bool | `Dep` | |
|
||||
| 0x51 | 0x49 | bool | `Atq` | |
|
||||
| 0x5c | 0x54 | int | `LCT` | |
|
||||
| 0x60 | 0x58 | int | `tsd` | |
|
||||
| 0x64 | 0x5c | `StarFleet*` | `FltID` | handle |
|
||||
| 0x68 | 0x60 | int | `ConCap` | |
|
||||
| 0x6c | 0x64 | **float** | `RefCap` | R1 Int32 → float |
|
||||
| 0x70 | 0x68 | **float** | `RepCap` | R1 Int32 → float |
|
||||
| 0x7c | 0x74 | int | `EncID` | |
|
||||
| 0x80 | 0x78 | `PrisonerHold` (inline) | `PrisH` | `+0x14 int*` → `[0]=PrMax`, `[2..8]` per-species counts; disk `PrMax`,`PrNSp`,(`PrSp`=idx,`PrNum`) |
|
||||
| 0x98 | 0x90 | `BuildQueue*` | `hbq` + `BQ2` | gate = ptr!=NULL |
|
||||
| 0x9c | 0x94 | `Population*` | `hsp` + `pop` | gate = ptr!=NULL |
|
||||
| 0xa0 | 0x98 | `Population*` | `ppop` | |
|
||||
| 0xa8 | 0xa0 | int | `atsp` | by value |
|
||||
| 0xac | 0xa4 | int | `tblt` | |
|
||||
|
||||
---
|
||||
|
||||
## 5. Top-level sim block — `Game::StrategyServer` Write `FUN_0079fa70` / Read `FUN_007d27a0`
|
||||
Tag order confirms R1 §5: KeyPath, NMSz, NMLc, NMnx, ModCount, Frame, GameID, [AIDifficultyID legacy],
|
||||
Attrib, RNG, GameName, Map, IncMod, ResMod, EnAl, EnTm, GOTurn, GOWinPly, NPCm/o/i/v/a, szadj, rsadj, suadj,
|
||||
sprjs, RandEncAdj, cmbtid, turnstats, numcreps/crep, ninv/invs/inve/invt/invtb, AllExc×6, AllExcCF,
|
||||
AllExcCFp×2, NumPlrs/PlayerID/Player, ISsp, ISsu, NumSys/SysID/Sys, NdGr2, trdmgr, spymgr, NumFlts/FltID/Flt,
|
||||
NumActs/Act, SvSctOb, zdsc, zdsi, zdst. (Offsets not tabulated — out of scope this round.)
|
||||
|
||||
---
|
||||
|
||||
## 6. Corrections to the community reference (R1/R2)
|
||||
- Types: `TRM`, `CstR/E/T`, `shrm`, ship `RefCap/RepCap`, PlayerView `Infra` are **floats**; `pswd` is a
|
||||
**string**; `PvMA`, `AIBn` are bools; `FtOrig` is a Vector3 (3 floats).
|
||||
- Widths: `Bats2`/`rcex` are true int64 (R2 int is wrong); `Abdn`/`Dstyd` are bools, `ltis` an int
|
||||
(R2 "short" is an artefact of reading value bytes); `TShn`, `ETS`, all DiplomacyStats counters are
|
||||
int16 in memory but int32 on disk.
|
||||
- Missing in R1: `Nexp` entries carry (`xid`,`xmin`,`xmax`,`xper`); SpyReport lists have bodies; Morale
|
||||
is a sparse (msp,mv) table; `nadct` is followed by sparse (`ads`,`adt`) pairs over a 7-int table.
|
||||
- `Plg`/`Act`/`EncID`/`OID`/flags are written by value → plain ints (not handles).
|
||||
|
||||
## 7. Confidence & open questions
|
||||
- **High**: all offsets/types in §1–§4 (direct from Write functions; Read functions agree on every
|
||||
member address; COL offsets from RTTI; no adjustor thunks).
|
||||
- **Medium**: nested struct sizes inferred from neighbouring offsets (Population 0x14, Morale 0x20,
|
||||
FlightPlan 0x38, PlayerView ≈0x9c, IndependenceInfo 0x70); which members belong to `StarSystem` vs
|
||||
`ServerSystem` (serializer is `ServerSystem`'s; `ClientSystem` shares the StarSystem vftables and only
|
||||
streams `Pos`).
|
||||
- **Open**: `FUN_008b9cb0` exact on-disk width for `PlayerColorID` (R1 says int32 — plausible);
|
||||
`ServerSystem+0x10` owner type (StrategyServer? its `+0x50` is a player-object table); `OID`
|
||||
allocation (R2's ×16); per-tech body of `TechTree::Write`; `CdPlayer` block not attempted (names are
|
||||
R1's `unknownN`, nothing to xref). No types were written back into Ghidra (notes only).
|
||||
1442
save-reader/save_reader.py
Normal file
1442
save-reader/save_reader.py
Normal file
File diff suppressed because it is too large
Load diff
330
save-reader/save_writer_stub.py
Normal file
330
save-reader/save_writer_stub.py
Normal file
|
|
@ -0,0 +1,330 @@
|
|||
#!/usr/bin/env python3
|
||||
"""save_writer_stub.py -- synthetic SOTS1 save builder for testing save_reader.
|
||||
|
||||
This is NOT a game-compatible writer. It emits the same *framing* the reader
|
||||
assumes (name-tagged values, NUL padding, BEEFBEEF/41104110 frames, gzip) and
|
||||
drives itself off the reader's schema so every declared struct gets exercised.
|
||||
Values are deterministic counters, so the expected typed dict is known up
|
||||
front and the round trip can be asserted. Real-save validation is pending.
|
||||
|
||||
python3 save_writer_stub.py out.sav [--padding joint|split] [--inflated]
|
||||
|
||||
Stdlib only.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gzip
|
||||
import random
|
||||
import struct
|
||||
import sys
|
||||
|
||||
import save_reader as sr
|
||||
|
||||
__all__ = ["SaveWriter", "Fixture", "build_fixture"]
|
||||
|
||||
|
||||
class SaveWriter:
|
||||
"""Byte builder for the Streamable framing. padding = 'joint' | 'split'."""
|
||||
|
||||
def __init__(self, padding: str = "joint"):
|
||||
if padding not in ("joint", "split"):
|
||||
raise ValueError(padding)
|
||||
self.padding = padding
|
||||
self.buf = bytearray()
|
||||
self.depth = 0
|
||||
|
||||
# -- primitives ----------------------------------------------------------------
|
||||
def _tag(self, name: str) -> int:
|
||||
b = name.encode("ascii")
|
||||
self.buf += struct.pack("<i", len(b)) + b
|
||||
return len(b)
|
||||
|
||||
def _pad(self):
|
||||
self.buf += b"\0" * (sr.pad4(len(self.buf)) - len(self.buf))
|
||||
|
||||
def item(self, name: str, payload: bytes) -> int:
|
||||
"""Named value. Returns the offset of the value bytes."""
|
||||
start = len(self.buf)
|
||||
self._tag(name)
|
||||
if self.padding == "split":
|
||||
self._pad()
|
||||
vp = len(self.buf)
|
||||
self.buf += payload
|
||||
self._pad() # joint: pad over tag+payload; split: pad payload
|
||||
assert len(self.buf) == sr.pad4(len(self.buf)) and start % 4 == 0
|
||||
return vp
|
||||
|
||||
def int(self, name, v):
|
||||
return self.item(name, struct.pack("<i", v))
|
||||
|
||||
def uint(self, name, v):
|
||||
return self.item(name, struct.pack("<I", v))
|
||||
|
||||
def float(self, name, v):
|
||||
return self.item(name, struct.pack("<f", v))
|
||||
|
||||
def bool(self, name, v):
|
||||
return self.item(name, b"\1" if v else b"\0")
|
||||
|
||||
def int64(self, name, v):
|
||||
return self.item(name, struct.pack("<q", v))
|
||||
|
||||
def string(self, name, s):
|
||||
b = s.encode("cp1252")
|
||||
return self.item(name, struct.pack("<i", len(b)) + b)
|
||||
|
||||
def raw(self, name, data: bytes):
|
||||
return self.item(name, data)
|
||||
|
||||
def begin(self, name):
|
||||
"""Open a frame (name=None -> tagless frame)."""
|
||||
if name is not None:
|
||||
self._tag(name)
|
||||
self._pad()
|
||||
self.buf += sr.BEGIN_BYTES
|
||||
self.depth += 1
|
||||
|
||||
def end(self):
|
||||
assert self.depth > 0
|
||||
self.buf += sr.END_BYTES
|
||||
self.depth -= 1
|
||||
|
||||
def vec3(self, name, x, y, z, named: bool = False):
|
||||
self.begin(name)
|
||||
if named:
|
||||
self.float(".", x)
|
||||
self.float(".", y)
|
||||
self.float(".", z)
|
||||
else:
|
||||
self.buf += struct.pack("<3f", x, y, z)
|
||||
self.end()
|
||||
|
||||
def bytes(self) -> bytes:
|
||||
assert self.depth == 0, "unbalanced frames"
|
||||
return bytes(self.buf)
|
||||
|
||||
def gzip(self) -> bytes:
|
||||
return gzip.compress(self.bytes(), mtime=0)
|
||||
|
||||
|
||||
# --- schema-driven fixture ----------------------------------------------------
|
||||
|
||||
GATES_TRUE = {"vnh", "hindi", "isind", "HFPlan", "HLay", "hbq", "hsp", "HasAIR"}
|
||||
OPT_EMIT = {"BQ"} # optional items the fixture does write
|
||||
ARRAY_LEN = 2
|
||||
|
||||
|
||||
def _plain_scalar(name, kind, value):
|
||||
return {"name": name, "kind": kind, "value": value}
|
||||
|
||||
|
||||
class Fixture:
|
||||
"""Emits a synthetic save from sr.ROOT and records the expected typed dict."""
|
||||
|
||||
def __init__(self, padding="joint", seed=1):
|
||||
self.w = SaveWriter(padding)
|
||||
self.counter = 0
|
||||
self.rng = random.Random(seed)
|
||||
self.expected = None
|
||||
|
||||
# -- value policy ----------------------------------------------------------------
|
||||
def next(self) -> int:
|
||||
self.counter += 1
|
||||
return self.counter
|
||||
|
||||
def value(self, name, kind, ctx):
|
||||
c = self.next()
|
||||
if kind == "int":
|
||||
if name == "idx" and ctx is sr.PlayerColor:
|
||||
self.color_toggle = not getattr(self, "color_toggle", False)
|
||||
return -1 if self.color_toggle else 3 # alternate custom-RGB / palette
|
||||
return c if c % 7 else -c
|
||||
if kind == "float":
|
||||
return c + 0.25
|
||||
if kind == "bool":
|
||||
return True if name in GATES_TRUE else bool(c % 2)
|
||||
if kind == "int64":
|
||||
return c * (1 << 33) + 7
|
||||
if kind == "string":
|
||||
return "str%d%s" % (c, "x" * (c % 4)) + ("é" if c % 5 == 0 else "")
|
||||
raise ValueError(kind)
|
||||
|
||||
def emit_scalar(self, name, kind, ctx):
|
||||
v = self.value(name, kind, ctx)
|
||||
getattr(self.w, kind)(name, v)
|
||||
return v
|
||||
|
||||
# -- generic filler ----------------------------------------------------------------
|
||||
def generic_frame(self, name, with_blob=False):
|
||||
"""Frame with content the schema knows nothing about. Returns plain()."""
|
||||
w = self.w
|
||||
w.begin(name)
|
||||
items = []
|
||||
if with_blob:
|
||||
blob = bytes(self.rng.getrandbits(8) for _ in range(2500))
|
||||
while sr.END_BYTES in blob or sr.BEGIN_BYTES in blob:
|
||||
blob = bytes(self.rng.getrandbits(8) for _ in range(2500))
|
||||
vp = w.raw("State", blob)
|
||||
end = len(w.buf)
|
||||
items.append(_plain_scalar("State", "raw", {"len": end - vp, "hex": w.buf[vp:vp + 32].hex()}))
|
||||
else:
|
||||
c = self.next()
|
||||
w.int("ga", c)
|
||||
items.append(_plain_scalar("ga", "int", c))
|
||||
w.bool("cmp", True) # 'cmp' is in the catalog -> bool
|
||||
items.append(_plain_scalar("cmp", "bool", True))
|
||||
w.string("gs", "gen%d" % c)
|
||||
items.append(_plain_scalar("gs", "string", "gen%d" % c))
|
||||
w.begin("gn")
|
||||
w.float("gflt", c + 0.5)
|
||||
w.end()
|
||||
items.append({"_name": "gn", "_items": [_plain_scalar("gflt", "float", c + 0.5)]})
|
||||
w.begin(None)
|
||||
w.int("ti", 42)
|
||||
w.end()
|
||||
items.append({"_name": None, "_items": [_plain_scalar("ti", "int", 42)]})
|
||||
w.end()
|
||||
return {"_name": name, "_items": items}
|
||||
|
||||
def filler_items(self):
|
||||
c = self.next()
|
||||
self.w.int("Filler", c)
|
||||
out = [_plain_scalar("Filler", "int", c)]
|
||||
out.append(self.generic_frame("FillerF"))
|
||||
return out
|
||||
|
||||
# -- schema walk -------------------------------------------------------------------
|
||||
def build(self) -> bytes:
|
||||
self.expected = {}
|
||||
self.emit_fields(sr.ROOT.fields, self.expected, None, top=True)
|
||||
return self.w.bytes()
|
||||
|
||||
def emit_fields(self, fields, out: dict, ctx, top=False):
|
||||
for f in fields:
|
||||
if isinstance(f, sr.Field):
|
||||
self.emit_field(f, out, ctx)
|
||||
elif isinstance(f, sr.Opt):
|
||||
if f.field.name in OPT_EMIT:
|
||||
self.emit_field(f.field, out, ctx)
|
||||
elif isinstance(f, sr.If):
|
||||
if out.get(f.key) == f.equals:
|
||||
inner = f.inner
|
||||
if isinstance(inner, sr.Seq):
|
||||
self.emit_fields(inner.fields, out, ctx)
|
||||
else:
|
||||
self.emit_field(inner, out, ctx)
|
||||
elif isinstance(f, sr.Until):
|
||||
out[f.key] = self.filler_items()
|
||||
elif isinstance(f, sr.Rest):
|
||||
if top:
|
||||
out[f.key] = self.cd_table()
|
||||
else:
|
||||
raise TypeError(f)
|
||||
|
||||
def emit_field(self, f: sr.Field, out: dict, ctx):
|
||||
t = f.type
|
||||
if isinstance(t, sr.NArr):
|
||||
self.w.int(f.name, ARRAY_LEN)
|
||||
out[f.key] = [self.emit_elem(t.elem, ctx) for _ in range(ARRAY_LEN)]
|
||||
elif isinstance(t, sr.Seq):
|
||||
self.emit_fields(t.fields, out, ctx)
|
||||
elif isinstance(t, sr.Shape):
|
||||
if f.flex and self.counter % 2:
|
||||
sub = {}
|
||||
self.emit_fields(t.fields, sub, t) # inline variant
|
||||
out[f.key] = sub
|
||||
else:
|
||||
# authoritative fields use their disk tag; R1-only fields
|
||||
# borrow the Shape's (guessed) frame name where it has one
|
||||
out[f.key] = self.emit_shape(f.name if f.auth else (t.name or f.name), t)
|
||||
elif isinstance(t, sr.CArr):
|
||||
if f.flex and self.counter % 2:
|
||||
self.w.int(f.name, ARRAY_LEN) # inline variant == NArr
|
||||
out[f.key] = [self.emit_elem(t.elem, ctx) for _ in range(ARRAY_LEN)]
|
||||
else:
|
||||
out[f.key] = self.emit_carr(f.name, t, ctx)
|
||||
elif t == "vec3":
|
||||
c = self.next()
|
||||
v = [c + 0.5, c + 1.5, c + 2.5]
|
||||
self.w.vec3(f.name, *v, named=bool(c % 3 == 0))
|
||||
out[f.key] = v
|
||||
elif t == "any":
|
||||
out[f.key] = self.generic_frame(f.name, with_blob=(f.name in sr.RAW_FRAMES))
|
||||
elif t in sr.PRIMITIVES:
|
||||
out[f.key] = self.emit_scalar(f.name, t, ctx)
|
||||
else:
|
||||
raise TypeError(t)
|
||||
|
||||
def emit_shape(self, name, shape: sr.Shape) -> dict:
|
||||
self.w.begin(name)
|
||||
sub = {}
|
||||
self.emit_fields(shape.fields, sub, shape)
|
||||
self.w.end()
|
||||
return sub
|
||||
|
||||
def emit_carr(self, name, carr: sr.CArr, ctx) -> list:
|
||||
self.w.begin(name)
|
||||
self.w.int("Count", ARRAY_LEN)
|
||||
items = [self.emit_elem(carr.elem, ctx) for _ in range(ARRAY_LEN)]
|
||||
self.w.end()
|
||||
return items
|
||||
|
||||
def emit_elem(self, elem, ctx):
|
||||
if isinstance(elem, sr.Seq):
|
||||
sub = {}
|
||||
self.emit_fields(elem.fields, sub, ctx)
|
||||
return sub
|
||||
if isinstance(elem, sr.Field):
|
||||
sub = {}
|
||||
self.emit_field(elem, sub, ctx)
|
||||
return sub.get(elem.key)
|
||||
if isinstance(elem, sr.Shape):
|
||||
return self.emit_shape(".", elem) # R1: element tags are often "."
|
||||
if isinstance(elem, sr.CArr):
|
||||
return self.emit_carr(".", elem, ctx)
|
||||
if elem in sr.PRIMITIVES:
|
||||
return self.emit_scalar("Item", elem, ctx)
|
||||
raise TypeError(elem)
|
||||
|
||||
def cd_table(self) -> list:
|
||||
"""Unframed 4th section: a framed string array, an unknown-shaped player
|
||||
block and two AI blocks -- all opaque to the schema."""
|
||||
w = self.w
|
||||
items = []
|
||||
w.begin("cdt")
|
||||
w.int("Count", 2)
|
||||
w.string("Item", "cd-a")
|
||||
w.string("Item", "cd-b")
|
||||
w.end()
|
||||
items.append({"_name": "cdt", "_items": [_plain_scalar("Count", "int", 2),
|
||||
_plain_scalar("Item", "string", "cd-a"),
|
||||
_plain_scalar("Item", "string", "cd-b")]})
|
||||
items.append(self.generic_frame("cdplayer"))
|
||||
items.append(self.generic_frame("cdai"))
|
||||
items.append(self.generic_frame("cdai"))
|
||||
return items
|
||||
|
||||
|
||||
def build_fixture(padding="joint", seed=1):
|
||||
"""-> (inflated bytes, expected typed dict)"""
|
||||
fx = Fixture(padding, seed)
|
||||
data = fx.build()
|
||||
return data, fx.expected
|
||||
|
||||
|
||||
def main(argv=None) -> int:
|
||||
ap = argparse.ArgumentParser(description="write a synthetic SOTS1 save for reader tests")
|
||||
ap.add_argument("out")
|
||||
ap.add_argument("--padding", choices=("joint", "split"), default="joint")
|
||||
ap.add_argument("--inflated", action="store_true", help="write the raw stream, not gzip")
|
||||
args = ap.parse_args(argv)
|
||||
data, _ = build_fixture(args.padding)
|
||||
with open(args.out, "wb") as f:
|
||||
f.write(data if args.inflated else gzip.compress(data, mtime=0))
|
||||
print(f"wrote {args.out}: {len(data)} inflated bytes, padding={args.padding}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
370
save-reader/test_save_reader.py
Normal file
370
save-reader/test_save_reader.py
Normal file
|
|
@ -0,0 +1,370 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Unit tests for save_reader.py using the synthetic fixture only.
|
||||
|
||||
/usr/bin/python3 -m unittest -v test_save_reader
|
||||
|
||||
NOTE: these prove the reader against its OWN framing assumptions. Validation
|
||||
against a real Sword of the Stars save is still pending.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import gzip
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import struct
|
||||
import sys
|
||||
import tempfile
|
||||
import unittest
|
||||
from contextlib import redirect_stdout
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
|
||||
import save_reader as sr
|
||||
import save_writer_stub as sw
|
||||
|
||||
|
||||
def norm(x):
|
||||
"""Drop offsets / guess annotations so typed output compares to expectations."""
|
||||
if isinstance(x, dict):
|
||||
return {k: norm(v) for k, v in x.items() if k not in ("_off", "off", "guessed", "alt")}
|
||||
if isinstance(x, list):
|
||||
return [norm(v) for v in x]
|
||||
return x
|
||||
|
||||
|
||||
def walk(data, padding="joint", schema=None):
|
||||
w = sr.Walker(data, padding, schema)
|
||||
return w, w.walk()
|
||||
|
||||
|
||||
class PrimitivesTest(unittest.TestCase):
|
||||
def check_prims(self, padding):
|
||||
w = sw.SaveWriter(padding)
|
||||
# names of every length mod 4, every primitive width
|
||||
w.int("Idx", 7)
|
||||
w.int("Size", -3)
|
||||
w.float("Suit", 0.75)
|
||||
w.bool("NPC", True) # 3-char name + 1 byte: the padding-mode-sensitive case
|
||||
w.bool("Abdn", False)
|
||||
w.int64("Bats2", (5 << 33) + 9)
|
||||
w.string("Name", "Sol")
|
||||
w.string("PlryName", "")
|
||||
w.string("Bdg", "café") # cp1252 byte 0xE9
|
||||
w.int("TRA", 100)
|
||||
data = w.bytes()
|
||||
self.assertEqual(len(data) % 4, 0)
|
||||
wk, root = walk(data, padding)
|
||||
kinds = [(n.name, n.kind, n.value) for n in root.children]
|
||||
self.assertEqual(kinds, [
|
||||
("Idx", "int", 7), ("Size", "int", -3), ("Suit", "float", 0.75),
|
||||
("NPC", "bool", True), ("Abdn", "bool", False), ("Bats2", "int64", (5 << 33) + 9),
|
||||
("Name", "string", "Sol"), ("PlryName", "string", ""), ("Bdg", "string", "café"),
|
||||
("TRA", "int", 100)])
|
||||
self.assertEqual(wk.stats["resyncs"], 0)
|
||||
self.assertEqual(wk.issues, [])
|
||||
# offsets are contiguous and 4-aligned
|
||||
pos = 0
|
||||
for n in root.children:
|
||||
self.assertEqual(n.offset, pos)
|
||||
self.assertEqual(pos % 4, 0)
|
||||
pos += n.size
|
||||
self.assertEqual(pos, len(data))
|
||||
|
||||
def test_joint(self):
|
||||
self.check_prims("joint")
|
||||
|
||||
def test_split(self):
|
||||
self.check_prims("split")
|
||||
|
||||
def test_padding_layout_joint(self):
|
||||
w = sw.SaveWriter("joint")
|
||||
w.bool("NPC", True)
|
||||
# [03 00 00 00]['N''P''C'][01] -> 8 bytes, no padding needed
|
||||
self.assertEqual(w.bytes(), b"\x03\x00\x00\x00NPC\x01")
|
||||
w = sw.SaveWriter("joint")
|
||||
w.int("Idx", 1)
|
||||
# 4 + 3 + 4 = 11 -> pad 1
|
||||
self.assertEqual(w.bytes(), b"\x03\x00\x00\x00Idx\x01\x00\x00\x00\x00")
|
||||
|
||||
def test_padding_layout_split(self):
|
||||
w = sw.SaveWriter("split")
|
||||
w.bool("NPC", True)
|
||||
self.assertEqual(w.bytes(), b"\x03\x00\x00\x00NPC\x00\x01\x00\x00\x00")
|
||||
|
||||
def test_unknown_names_are_guessed(self):
|
||||
w = sw.SaveWriter("joint")
|
||||
w.int("zzq", 12) # 3-char names: bool/int/int64 sizes all differ
|
||||
w.bool("zzb", True)
|
||||
w.int64("zzl", 1 << 40)
|
||||
w.string("zzs", "hello")
|
||||
w.float("zzf", 3.5)
|
||||
w.int("zzz", 99)
|
||||
_, root = walk(w.bytes())
|
||||
got = [(n.name, n.kind, n.value, n.hinted) for n in root.children]
|
||||
self.assertEqual(got, [("zzq", "int", 12, False), ("zzb", "bool", True, False),
|
||||
("zzl", "int64", 1 << 40, False), ("zzs", "string", "hello", False),
|
||||
("zzf", "float", 3.5, False), ("zzz", "int", 99, False)])
|
||||
|
||||
def test_word_ambiguity_is_reported_as_alt(self):
|
||||
w = sw.SaveWriter("joint")
|
||||
w.float("zzzz", 1.0) # 0x3F800000: unknown name -> float with int alt
|
||||
_, root = walk(w.bytes())
|
||||
n = root.children[0]
|
||||
self.assertEqual((n.kind, n.value, n.alt), ("float", 1.0, 0x3F800000))
|
||||
|
||||
def test_catalog_hint_overrides_guess(self):
|
||||
w = sw.SaveWriter("joint")
|
||||
w.float("Suit", 0.0) # bits 0 would be guessed int; catalog says float
|
||||
w.bool("Elim", True) # 4-char name + bool is size-ambiguous with int
|
||||
_, root = walk(w.bytes())
|
||||
self.assertEqual([(n.kind, n.value, n.hinted) for n in root.children],
|
||||
[("float", 0.0, True), ("bool", True, True)])
|
||||
|
||||
|
||||
class FramingTest(unittest.TestCase):
|
||||
def test_nested_frames_and_markers(self):
|
||||
w = sw.SaveWriter()
|
||||
w.begin("Outer")
|
||||
w.int("a", 1)
|
||||
w.begin("Inner")
|
||||
w.string("s", "x")
|
||||
w.end()
|
||||
w.begin(None) # tagless frame
|
||||
w.int("t", 2)
|
||||
w.end()
|
||||
w.end()
|
||||
data = w.bytes()
|
||||
self.assertTrue(data.startswith(b"\x05\x00\x00\x00Outer\x00\x00\x00" + sr.BEGIN_BYTES))
|
||||
self.assertTrue(data.endswith(sr.END_BYTES + sr.END_BYTES))
|
||||
wk, root = walk(data)
|
||||
outer = root.children[0]
|
||||
self.assertEqual((outer.name, outer.kind, outer.size), ("Outer", "complex", len(data)))
|
||||
names = [(c.name, c.kind) for c in outer.children]
|
||||
self.assertEqual(names, [("a", "int"), ("Inner", "complex"), (None, "complex")])
|
||||
self.assertEqual(outer.children[2].children[0].value, 2)
|
||||
self.assertEqual(wk.issues, [])
|
||||
|
||||
def test_unnamed_vec3_body(self):
|
||||
w = sw.SaveWriter()
|
||||
w.vec3("Pos", 1.5, -2.0, 1e6)
|
||||
w.vec3("Pos", 1.5, -2.0, 1e6, named=True)
|
||||
wk, root = walk(w.bytes())
|
||||
raw = root.children[0].children
|
||||
self.assertEqual(len(raw), 1)
|
||||
self.assertEqual(raw[0].kind, "raw")
|
||||
self.assertEqual(struct.unpack("<3f", raw[0].raw), (1.5, -2.0, 1e6))
|
||||
self.assertEqual([c.kind for c in root.children[1].children], ["float"] * 3)
|
||||
self.assertTrue(all(i.level == "info" for i in wk.issues))
|
||||
ap = sr.Applier()
|
||||
self.assertEqual(ap.vec3(root.children[0], ""), [1.5, -2.0, 1e6])
|
||||
self.assertEqual(ap.vec3(root.children[1], ""), [1.5, -2.0, 1e6])
|
||||
|
||||
def test_resync_through_garbage(self):
|
||||
"""A frame whose body is unknown binary must not derail the parse."""
|
||||
import random
|
||||
rnd = random.Random(7)
|
||||
blob = bytes(rnd.getrandbits(8) for _ in range(2500))
|
||||
w = sw.SaveWriter()
|
||||
w.int("before", 1)
|
||||
w.begin("Mystery")
|
||||
w.raw("State", blob) # named blob, unknown name -> no layout fits
|
||||
w.end()
|
||||
w.begin("Junk")
|
||||
w.buf += blob[:301] # untagged garbage, unaligned length
|
||||
w._pad()
|
||||
w.end()
|
||||
w.int("after", 2)
|
||||
wk, root = walk(w.bytes())
|
||||
self.assertEqual([c.name for c in root.children], ["before", "Mystery", "Junk", "after"])
|
||||
self.assertEqual(root.children[3].value, 2)
|
||||
self.assertEqual(root.children[1].children[0].kind, "raw")
|
||||
self.assertEqual(root.children[2].children[0].kind, "raw")
|
||||
self.assertEqual(wk.stats["resyncs"], 2)
|
||||
self.assertTrue(any(i.level == "warn" for i in wk.issues))
|
||||
|
||||
def test_raw_frame_hint(self):
|
||||
"""RNG is declared opaque: its body is read to the END marker without warnings."""
|
||||
w = sw.SaveWriter()
|
||||
w.begin("RNG")
|
||||
w.raw("State", bytes(range(256)) * 9 + b"\x00" * 196)
|
||||
w.end()
|
||||
wk, root = walk(w.bytes())
|
||||
n = root.children[0].children[0]
|
||||
self.assertEqual((n.name, n.kind, n.hinted), ("State", "raw", True))
|
||||
self.assertEqual(n.value["len"], 2500 + 3) # joint padding bytes are included
|
||||
self.assertEqual(wk.stats["resyncs"], 0)
|
||||
|
||||
def test_truncated_stream(self):
|
||||
w = sw.SaveWriter()
|
||||
w.begin("Summary")
|
||||
w.int("turn", 5)
|
||||
w.string("gameName", "abcdef")
|
||||
w.end()
|
||||
data = w.bytes()[:-9]
|
||||
wk, root = walk(data)
|
||||
self.assertEqual(root.children[0].children[0].value, 5)
|
||||
self.assertTrue(any("not terminated" in i.msg for i in wk.issues))
|
||||
self.assertTrue(all(i.offset <= len(data) for i in wk.issues))
|
||||
|
||||
def test_damaged_gzip(self):
|
||||
data = gzip.compress(b"\x03\x00\x00\x00Idx\x01\x00\x00\x00\x00")[:-6]
|
||||
with self.assertRaises(sr.SaveFormatError):
|
||||
sr.read_bytes(data)
|
||||
|
||||
def test_inflate_passthrough(self):
|
||||
raw = b"\x03\x00\x00\x00Idx\x01\x00\x00\x00\x00"
|
||||
self.assertEqual(sr.inflate(raw), raw)
|
||||
self.assertEqual(sr.inflate(gzip.compress(raw)), raw)
|
||||
|
||||
|
||||
class SchemaRoundTripTest(unittest.TestCase):
|
||||
def roundtrip(self, padding, request="auto"):
|
||||
data, expected = sw.build_fixture(padding)
|
||||
res = sr.read_bytes(gzip.compress(data), padding=request, strict=True)
|
||||
self.assertEqual(res.padding, padding)
|
||||
self.assertEqual(res.count("error"), 0)
|
||||
self.assertEqual(res.count("warn"), 0)
|
||||
self.assertEqual(res.stats["resyncs"], 0)
|
||||
self.assertEqual(norm(res.typed), expected)
|
||||
return res
|
||||
|
||||
def test_roundtrip_joint(self):
|
||||
res = self.roundtrip("joint")
|
||||
sim = res.typed["sim"]
|
||||
self.assertEqual(len(sim["players"]), 2)
|
||||
self.assertEqual(len(sim["systems"]), 2)
|
||||
sysd = sim["systems"][0]["Sys"]
|
||||
self.assertIsInstance(sysd["Bats2"], int)
|
||||
self.assertGreater(sysd["Bats2"], 1 << 33)
|
||||
self.assertIsInstance(sysd["Abdn"], bool)
|
||||
self.assertEqual(len(sysd["Pos"]), 3)
|
||||
self.assertIn("indi", sysd) # hindi gate honoured
|
||||
ply = sim["players"][0]["Player"]
|
||||
self.assertIsInstance(ply["pswd"], str)
|
||||
self.assertIsInstance(ply["TRM"], float)
|
||||
self.assertEqual(len(ply["nexp"]), 2)
|
||||
self.assertEqual(set(ply["nexp"][0]), {"xid", "xmin", "xmax", "xper"})
|
||||
ship = sim["fleets"][0]["Flt"]["ships"][0]["Ship"]
|
||||
self.assertIsInstance(ship["RefCap"], float)
|
||||
self.assertIn("BQ2", ship)
|
||||
self.assertIn("pop", ship)
|
||||
colors = [p["slot"]["fxCrId"] for p in res.typed["summary"]["players"]]
|
||||
custom = [c for c in colors if c["idx"] == -1]
|
||||
self.assertTrue(custom and all("r" in c for c in custom)) # -1 -> RGB follows
|
||||
self.assertTrue(all("r" not in c for c in colors if c["idx"] != -1))
|
||||
|
||||
def test_roundtrip_split(self):
|
||||
self.roundtrip("split")
|
||||
|
||||
def test_explicit_padding_mode(self):
|
||||
self.roundtrip("joint", "joint")
|
||||
self.roundtrip("split", "split")
|
||||
|
||||
def test_wrong_padding_mode_is_noisy(self):
|
||||
data, _ = sw.build_fixture("split")
|
||||
res = sr.read_bytes(data, padding="joint")
|
||||
self.assertGreater(res.stats["resyncs"] + res.stats["hint_failures"] + res.count("error"), 0)
|
||||
|
||||
def test_json_serializable(self):
|
||||
data, _ = sw.build_fixture("joint")
|
||||
res = sr.read_bytes(data)
|
||||
json.dumps(res.typed)
|
||||
json.dumps(sr.plain(res.tree))
|
||||
json.dumps([i.as_dict() for i in res.issues])
|
||||
|
||||
def test_strict_rejects_missing_authoritative_field(self):
|
||||
w = sw.SaveWriter()
|
||||
w.begin("Sys")
|
||||
w.vec3("Pos", 0, 0, 0)
|
||||
w.float("R", 1); w.float("G", 1); w.float("B", 1); w.float("A", 1)
|
||||
w.int("Idx", 3)
|
||||
w.int("Suit", 0) # Size missing; Suit written as int
|
||||
w.end()
|
||||
data = w.bytes()
|
||||
schema = sr.Seq([sr.R("sys", sr.Sys)])
|
||||
res = sr.read_bytes(data, padding="joint", schema=schema)
|
||||
self.assertIsNone(res.typed["sys"]["Size"])
|
||||
self.assertTrue(any(i.level == "error" and "Size" in i.msg for i in res.issues))
|
||||
with self.assertRaises(sr.SaveFormatError):
|
||||
sr.read_bytes(data, padding="joint", schema=schema, strict=True)
|
||||
|
||||
def test_lenient_skips_unexpected_items(self):
|
||||
w = sw.SaveWriter()
|
||||
w.begin("PopG")
|
||||
w.int("PopT", 1)
|
||||
w.int("Legacy", 5) # not in the schema
|
||||
w.int("PopS", 2)
|
||||
w.int64("PopC", 3)
|
||||
w.end()
|
||||
schema = sr.Seq([sr.R("g", sr.PopG)])
|
||||
res = sr.read_bytes(w.bytes(), padding="joint", schema=schema)
|
||||
g = res.typed["g"]
|
||||
self.assertEqual((g["PopT"], g["PopS"], g["PopC"]), (1, 2, 3))
|
||||
self.assertEqual(g["_unexpected"][0]["name"], "Legacy")
|
||||
self.assertTrue(any(i.level == "warn" for i in res.issues))
|
||||
|
||||
def test_coerce_retypes_words(self):
|
||||
"""A schema float stored as a word the walker guessed 'int' is re-read from raw bytes."""
|
||||
w = sw.SaveWriter()
|
||||
w.begin("Tmrs")
|
||||
w.int("tstl", 0x7F7FFFFF)
|
||||
w.float("tctl", 60.0) # 0x42700000, walker may call it float; schema says int
|
||||
w.int("tqtl", 0x7F7FFFFF)
|
||||
w.int("tqtle", 0)
|
||||
w.end()
|
||||
schema = sr.Seq([sr.R("t", sr.Tmrs)])
|
||||
res = sr.read_bytes(w.bytes(), padding="joint", schema=schema)
|
||||
self.assertEqual(res.typed["t"]["tctl"], 0x42700000)
|
||||
self.assertEqual(res.typed["t"]["tstl"], 0x7F7FFFFF)
|
||||
|
||||
|
||||
class CliTest(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self.tmp = tempfile.TemporaryDirectory()
|
||||
data, _ = sw.build_fixture("joint")
|
||||
self.path = os.path.join(self.tmp.name, "fixture.sav")
|
||||
with open(self.path, "wb") as f:
|
||||
f.write(gzip.compress(data))
|
||||
|
||||
def tearDown(self):
|
||||
self.tmp.cleanup()
|
||||
|
||||
def run_cli(self, *args):
|
||||
buf = io.StringIO()
|
||||
with redirect_stdout(buf):
|
||||
rc = sr.main([self.path, *args])
|
||||
return rc, buf.getvalue()
|
||||
|
||||
def test_summary(self):
|
||||
rc, out = self.run_cli()
|
||||
self.assertEqual(rc, 0)
|
||||
self.assertIn("padding: joint", out)
|
||||
self.assertIn("summary: game=", out)
|
||||
|
||||
def test_dump(self):
|
||||
rc, out = self.run_cli("--dump")
|
||||
self.assertEqual(rc, 0)
|
||||
self.assertIn("@00000000 Summary {", out)
|
||||
self.assertIn("Bats2 int64", out)
|
||||
|
||||
def test_json_and_inflate(self):
|
||||
outp = os.path.join(self.tmp.name, "out.json")
|
||||
infl = os.path.join(self.tmp.name, "inflated.bin")
|
||||
rc, _ = self.run_cli("--json", "--strict", "--out", outp, "--inflate", infl)
|
||||
self.assertEqual(rc, 0)
|
||||
with open(outp) as f:
|
||||
doc = json.load(f)
|
||||
self.assertIn("summary", doc["data"])
|
||||
self.assertEqual(doc["padding"], "joint")
|
||||
with open(infl, "rb") as f:
|
||||
self.assertTrue(f.read().startswith(b"\x07\x00\x00\x00Summary"))
|
||||
|
||||
def test_dump_json(self):
|
||||
rc, out = self.run_cli("--dump", "--json")
|
||||
self.assertEqual(rc, 0)
|
||||
doc = json.loads(out)
|
||||
self.assertEqual(doc["tree"][0]["_name"], "Summary")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
Loading…
Add table
Reference in a new issue