diff --git a/campaign/board.md b/campaign/board.md index 0bac816..e9924be 100644 --- a/campaign/board.md +++ b/campaign/board.md @@ -9,19 +9,20 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`). | `.gob` format | subsystem | mapped | high | 100% | 2026-09-07 | renamed uncompressed ZIP (community-known) | | `Mars::AppStartup` (entry) | control-flow | backlog | — | 0% | 2026-09-07 | bootstrap → main loop | | main loop / tick dispatch | control-flow | backlog | — | 0% | 2026-09-07 | how subsystems are pumped per frame/turn | -| `Game::ClientPlayer` / `AIPlayer` | object | backlog | — | 0% | 2026-09-07 | the empire object (SOTS calls it "player"); root of game state | -| `Game::StarSystem` | object | backlog | — | 0% | 2026-09-07 | + `PlayerView::StarSystem`; cross-ref save structs | -| `Game::Planet` / `DOPlanet` | object | backlog | — | 0% | 2026-09-07 | colony/economy data | -| `Game::ShipDesignDef` / `DesignList` | object | backlog | — | 0% | 2026-09-07 | ship design model | +| `Game::ClientPlayer` / `AIPlayer` | object | in-progress | low | 30% | 2026-09-07 | vft ClientPlayer@0x00a32b68 (8) : StrategyPlayer, NetworkObject; AIPlayer@0x00a199c4 (11) : AIObject, IStreamable. fields TBD | +| `Game::StarSystem` | object | in-progress | low | 30% | 2026-09-07 | vft@0x00a200e4 (8) : StarMapNode + NetworkObject/HandleObject/IStreamable. fields TBD | +| `Game::Planet` / `DOPlanet` | object | in-progress | low | 30% | 2026-09-07 | vft@0x009ef144 (36) : Actor->TargetableEntity->PhysicalEntity->EntityBase; DOPlanet@0x00a28cfc (7). fields TBD | +| `Game::ShipDesignDef` / `DesignList` | object | in-progress | low | 30% | 2026-09-07 | vft@0x009fef64 (3), leaf (0 bases); mostly data-defined via .shipsection. fields TBD | | `Game::FleetLayout` | object | backlog | — | 0% | 2026-09-07 | fleet composition | -| Mars entity chain | object | backlog | — | 0% | 2026-09-07 | EntityBase→PhysicalEntity→TargetableEntity→Actor | +| Mars entity chain | object | mapped | high | 100% | 2026-09-07 | EntityBase@0x00a36fd0 (10) -> PhysicalEntity@0x00a3710c (12) -> TargetableEntity -> Actor@0x009e2114 (36); mixins RigidBody/Body/NetworkObject/HandleObject/IScriptable | | `Game::GameCombatSim` + `CombatCommand*` | subsystem | backlog | — | 0% | 2026-09-07 | tactical combat (~40 command types) | | D3D9 device init | subsystem | backlog | — | 0% | 2026-09-07 | `Mars::DrawDevice`/`Camera` → Direct3DCreate9/CreateDevice | | `Mars::Buffer::gobio` | subsystem | in-progress | — | 0% | 2026-09-07 | .gob I/O; anchors the verify/ parser | -| serializable-types vs save structs | verify | in-progress | — | 0% | 2026-09-07 | cross-ref `serializable-types.txt` with community save editors | -| battle-load path | subsystem | in-progress | — | 0% | 2026-09-07 | SMOKING GUN: ForceSingleCore -> SetProcessAffinityMask; Ghidra agent pinning the fn | -| class hierarchy + key vftables | meta | in-progress | — | — | 2026-09-07 | Ghidra recon: inheritance + vftable sizes for core classes | +| serializable-types vs save structs | verify | mapped | high | 100% | 2026-09-07 | findings/objects/save-editor-structs.md (~130 structs, exact order); verifier: cross-check vs binary next | +| battle-load path | subsystem | mapped | med | 60% | 2026-09-07 | mitigation = blunt affinity pin FUN_0089ee70 (SetProcessAffinityMask); NO topology detection -> no adaptive branch; cause = emergent multi-core contention | +| class hierarchy + key vftables | meta | mapped | high | 100% | 2026-09-07 | findings/objects/ghidra-recon.md - RTTI Base_Class_Array read directly; vftables for 8 core classes | | data-model (.gob data files) | subsystem | mapped | high | 100% | 2026-09-07 | findings/subsystems/data-model.md - tech/weapons/sections/races/AI all data-driven | | string / config intel | meta | mapped | high | 100% | 2026-09-07 | findings/subsystems/strings-and-config.md | | Mars brace-block parser | subsystem | backlog | — | 0% | 2026-09-07 | one recursive KV parser; reimpl lever + feeds verify/parsers | | networking (SNM/FNM + GameSpy) | subsystem | backlog | — | 0% | 2026-09-07 | ~60 SNM strategy msgs, FNM file-xfer/host-migration; lockstep sim | +| battle-load: thread contention | subsystem | backlog | — | 0% | 2026-09-07 | root cause hunt: CreateThread sites FUN_00902350, FUN_00736e30, FUN_008a0e50; streaming-sound thread; lockstep sync waits | diff --git a/campaign/open-questions.md b/campaign/open-questions.md index 701f444..0a44a95 100644 --- a/campaign/open-questions.md +++ b/campaign/open-questions.md @@ -12,3 +12,6 @@ Each links to the finding that raised it. Promoted to backlog or closed by **re- state coverage may be partial. Treat as strong hints, verify each field against real saves. - **Lockstep determinism** — `@SYNC ERROR DETECTED: Frame %d` indicates a deterministic lockstep sim. Big for reimplementation + old-vs-new differential (determinism makes equivalence checkable frame-by-frame). Confirm scope. (from [[strings-and-config]]) - **Mods override mechanism** — `.gob` = `gobio::ZipFileSystem` with a `NativeFileSystem` loose-file override; this is how `./Mods` wins over archived files. Verify load order. (from [[strings-and-config]]) +- **Battle-load root cause** — the shipped fix is a blunt process-affinity pin (`FUN_0089ee70`) with NO CPU-topology detection, so there is no adaptive branch. The real cause is emergent multi-core contention: which thread(s)? Investigate the 3 `CreateThread` sites (`FUN_00902350`, `FUN_00736e30`, startup `FUN_008a0e50`), the streaming-sound thread, and lockstep sync waits. (from [[ghidra-recon]]) +- **Save-struct contradictions (R1 vs R2)** — field widths (`Abdn`/`Dstyd`/`ltis` Int16 vs Int32; `Bats2`), R2's `OID = PID*16` owner-handle claim, species id 4 (`_NPC` vs 'AI Rebellion'). Resolve against the binary's Streamable read code. (from [[save-editor-structs]]) +- **Unlabeled save blocks** — `CdPlayer` (unknown1..35), empty `SimSystemDetailSpy`, opaque ~2500 B RNG blob. Analyst targets once the Streamable readers are located. (from [[save-editor-structs]]) diff --git a/findings/objects/ghidra-recon.md b/findings/objects/ghidra-recon.md new file mode 100644 index 0000000..8889625 --- /dev/null +++ b/findings/objects/ghidra-recon.md @@ -0,0 +1,209 @@ +# Ghidra Recon — Sword of the Stars (2006), "Sword of the Stars.exe" + +ImageBase 0x00400000, 32-bit MSVC-2010 DX9. Program `sots` in `/srv/re-lab/ghidra-projects`. +All addresses below are Ghidra/virtual addresses (VA = ImageBase + RVA). +Method: headless GhidraScripts (ReVa MCP was down this run). reva-server stopped for runs, restarted after. + +--- + +## 1. Battle-load / many-core lead: ForceSingleCore + +### Key strings +| String | Address | +|---|---| +| `ForceSingleCore` | `0x00a36eec` (key literal), config-tag blob at `0x00a36ee8` | +| `Limiting process affinity to CPU-%i...` | `0x00a36880` | +| `SetProcessAffinityMask` (data ref name) | `0x00ad8202` | + +### The affinity worker — `FUN_0089ee70` @ `0x0089ee70` +```c +void FUN_0089ee70(void) { // core index arrives in ESI from caller + int unaff_ESI; + FUN_008ba1e0("Limiting process affinity to CPU-%i...", unaff_ESI + 1); + HANDLE hProcess = GetCurrentProcess(); + BOOL ok = SetProcessAffinityMask(hProcess, 1 << ((byte)unaff_ESI & 0x1f)); // pin to ONE core + if (ok) FUN_008ba1e0(" OK.\n"); + else FUN_008ba1e0(" FAILED!\n"); +} +``` +This is the whole mitigation: it pins the entire process to a single logical core via a +one-bit affinity mask. `SetProcessAffinityMask` is the ONLY affinity/topology API the +binary uses (see below). + +### Where the config is read & branches — `FUN_008a0e50` @ `0x008a0e50` +This is the application startup/init routine (also creates the window, D3D device, reads +display.cfg/audio.cfg, spawns a thread). Relevant excerpt: +```c +// ref site 0x008a0ee6 +uVar3 = FUN_00898bc0(&DAT_00a36ee8, "ForceSingleCore", local_64c, local_630); // read config key +FUN_008e7160(uVar3); +... +iVar4 = FUN_008e7140(local_64c); // convert the read value -> int +if (0 < iVar4) { // <-- the many-core branch: any value > 0 + FUN_0089ee70(); // pin process to a single core +} +``` +`FUN_00898bc0` = generic config/ini lookup (key -> value string); `FUN_008e7140` +extracts an integer from the parsed value. If the integer is > 0 the process is bound +to one core (the core index is the value carried in ESI into `FUN_0089ee70`, and the +`"CPU-%i"` log uses index+1). + +### Read on the many-core slowdown +The binary performs **no runtime CPU-topology detection at all** — there is no +`GetSystemInfo`, `GetNativeSystemInfo`, `GetLogicalProcessorInformation(Ex)`, +`GetActiveProcessorCount`, `GetProcessAffinityMask`, or `SetThreadAffinityMask` import +anywhere. So there is **no adaptive "if many cores do X else Y" branch** in the code. +`ForceSingleCore` is a blunt, opt-in override read once at startup that forces the whole +process onto one core. + +Implication for the battle-load bug: the slowdown is not gated by a core-count code path +we can flip; it is emergent multi-core behavior (thread contention / lock convoy / false +sharing / scheduler thrash, or the D3D9 driver's own worker threads) that gets worse as +the OS spreads the process's threads across more cores. Pinning to one core removes the +cross-core traffic and masks it. Next place to dig for the actual contention is the +thread creation sites: + +### Threading entry points (CreateThread callers) — follow-ups +| Caller | Address | Note | +|---|---|---| +| `FUN_00902350` | `0x00902350` | creates a thread at `0x0090242d` | +| `FUN_008a0e50` | `0x008a0e50` | startup routine, creates a thread at `0x008a14ef` | +| `FUN_00736e30` | `0x00736e30` | creates a thread at `0x00736e84` — likely a game/loader worker; prime suspect for the battle-load path | + +`SetProcessAffinityMask` external thunk: `EXTERNAL:00000052`; `CreateThread`: `EXTERNAL:00000019`. + +--- + +## 2. vftable dumps (primary vftable per class) + +MSVC layout note: each class has a `..::vftable_meta_ptr` label (0 methods — that slot +holds the RTTI Complete-Object-Locator pointer, sitting one word *before* the real table) +immediately followed by `..::vftable`, the actual function-pointer array. Classes with +multiple `vftable` groups use multiple inheritance (one sub-table per polymorphic base). +Only the *primary* (largest / class-named) table is listed here; secondary tables are +small MI thunks (`IStreamable` etc.). Repeated addresses = shared/thunk stubs: +`0x0080c5a0` and `0x005f8ac0` are stub/return-nullptr style thunks reused across many +slots; `0x00496e00` a shared pure/empty stub; `0x0085b290` a common getter. + +### Game::ClientPlayer — vftable @ `0x00a32b68`, 8 methods +``` +[0]0x00882630 [1]0x008f6630 [2]0x008188f0 [3]0x00818910 +[4]0x0085b290 [5]0x008801d0 [6]0x0080c5a0 [7]0x0080c5a0 +``` + +### Game::AIPlayer — vftable @ `0x00a199c4`, 11 methods +``` +[0]0x004f7230 [1]0x005f8ac0 [2]0x005f8ac0 [3]0x00723ed0 [4]0x00692350 [5]0x00496e00 +[6]0x00496e00 [7]0x00496e00 [8]0x00496e00 [9]0x00496e00 [10]0x006922e0 +``` +(secondary MI tables: VectorHelper @0x00a1a700, StreamableHelper @0x00a1a730, each 3 methods.) + +### Game::StarSystem — primary vftable @ `0x00a200e4`, 8 methods +``` +[0]0x00753160 [1]0x00725130 [2]0x00751b30 [3]0x00751b40 +[4]0x00751b40 [5]0x005f8ac0 [6]0x0085b290 [7]0x004f41d0 +``` +(also a 3-method table @0x00a200d4; StreamableHelper @0x00a23cc4.) + +### Game::Planet — primary vftable @ `0x009ef144`, 36 methods +``` +[0]0x008a1860 [1]0x004e8960 [2]0x004e72b0 [3]0x0080c5a0 [4]0x004d75a0 [5]0x004d75d0 +[6]0x004e16e0 [7]0x004d9cb0 [8]0x004e5e80 [9]0x0080c5a0 [10]0x005f8ac0 [11]0x00423710 +[12]0x004229a0 [13]0x004f59c0 [14]0x005f8ac0 [15]0x004ddbf0 [16]0x0085b290 [17]0x00425770 +[18]0x00425340 [19]0x00424b70 [20]0x004267f0 [21]0x00426810 [22]0x00426830 [23]0x00422bd0 +[24]0x0080d1f0 [25]0x004242b0 [26]0x00496e00 [27]0x00496e00 [28]0x00496e00 [29]0x004d9c20 +[30]0x0080c5a0 [31]0x0080c5a0 [32]0x005f8ac0 [33]0x0085b290 [34]0x00422b70 [35]0x005dbb90 +``` +(Planet also: Creator vftable @0x009ee234 (1); two 2-method tables @0x009ef12c/0x009ef138. DOPlanet is a separate class @0x00a28cfc, 7 methods.) + +### Game::ShipDesignDef — vftable @ `0x009fef64`, 3 methods +``` +[0]0x00827240 [1]0x00827390 [2]0x0057c250 +``` +(no deep hierarchy — RTTI shows 0 bases; the many `Mars::StreamableHelper` +and `VectorHelper<...GunBank>` tables are serialization helpers for its member containers.) + +### Game::Actor — primary vftable @ `0x009e2114`, 36 methods +``` +[0]0x008a1860 [1]0x00426d10 [2]0x004267d0 [3]0x0080c5a0 [4]0x00426c80 [5]0x00426c90 +[6]0x0080c5a0 [7]0x005f8ac0 [8]0x0080c5a0 [9]0x0080c5a0 [10]0x005f8ac0 [11]0x00423710 +[12]0x004229a0 [13]0x004f59c0 [14]0x00422af0 [15]0x004267e0 [16]0x0085b290 [17]0x00425770 +[18]0x00425340 [19]0x00424b70 [20]0x004267f0 [21]0x00426810 [22]0x00426830 [23]0x00422bd0 +[24]0x0080d1f0 [25]0x004242b0 [26]0x00496e00 [27]0x005dbbc0 [28]0x00496e00 [29]0x00422830 +[30]0x0080c5a0 [31]0x0080c5a0 [32]0x005f8ac0 [33]0x0085b290 [34]0x00422b70 [35]0x005dbb90 +``` +(Actor also: Creator @0x009e1fa8 (1); two 2-method tables @0x009e20f8/0x009e2104. +Interfaces IActorReference @0x009e604c (4), IActorEvents @0x009e90ac (2).) +**Note:** Planet's primary table shares slots 11-13,16-25,30-35 verbatim with Actor's, +and both are 36-wide — confirming Planet : Actor (Planet overrides slots 1-8,14,15,27-29). + +### Mars::EntityBase — primary vftable @ `0x00a36fd0`, 10 methods +``` +[0]0x008a1860 [1]0x008a1da0 [2]0x008a1c30 [3]0x0080c5a0 [4]0x0080c5a0 +[5]0x0080c5a0 [6]0x0080c5a0 [7]0x005f8ac0 [8]0x0080c5a0 [9]0x0080c5a0 +``` +(Creator @0x00a36fb0 (1); two 2-method MI tables @0x00a36fb8/0x00a36fc4.) + +### Mars::PhysicalEntity — primary vftable @ `0x00a3710c`, 12 methods +``` +[0]0x008a1860 [1]0x00423340 [2]0x008a24d0 [3]0x0080c5a0 [4]0x0080c5a0 [5]0x0080c5a0 +[6]0x0080c5a0 [7]0x005f8ac0 [8]0x0080c5a0 [9]0x0080c5a0 [10]0x005f8ac0 [11]0x004227a0 +``` +(Creator @0x00a36f50 (1); two 2-method MI tables @0x00a370f4/0x00a37100.) +**Note:** slot [0]=0x008a1860 shared with EntityBase's slot [0] → PhysicalEntity : EntityBase +(12 = 10 + 2 added slots [1],[2],[11] with 1 EntityBase slot overridden). + +--- + +## 3. RTTI inheritance graph (from Base_Class_Array structures) + +RTTI is **fully populated** — namespaces are NOT empty; the "known empty-namespace issue" +does **not** apply to this program. Chains read directly from each class's +`RTTI_Class_Hierarchy_Descriptor` -> `RTTI_Base_Class_Array` -> BaseClassDescriptors -> +TypeDescriptors. Listed most-derived → root; `mdisp` = byte offset of that base subobject. + +**ClientPlayer** (3 bases): `ClientPlayer` → `StrategyPlayer` → `Mars::NetworkObject`. + +**AIPlayer** (3): `AIPlayer` → `AIObject` → `Mars::IStreamable`. + +**StarSystem** (5): `StarSystem` → `StarMapNode`; + `Mars::NetworkObject`(@0), +`Mars::HandleObject`(@12), `Mars::IStreamable`(@8). + +**ShipDesignDef** (1): `ShipDesignDef` — no base classes (leaf/standalone data-def class). + +**Mars::EntityBase** (7): `EntityBase` → `IScheduledEvent` → +`ListDLI::Link`(@4) → `DLink`(@4); ++ `IScriptable`(@16), `NetworkObject`(@20), `HandleObject`(@28). + +**Mars::PhysicalEntity** (11): `PhysicalEntity` → `EntityBase` → `IScheduledEvent` → +`ListDLI<...>::Link`(@4) → `DLink`(@4); + `IScriptable`(@16), `NetworkObject`(@20), +`HandleObject`(@28), `RigidBody`(@44) → `Body`(@44), `SceneObject::Owner`(@212). + +**Game::Actor** (13): `Actor` → `TargetableEntity` → `Mars::PhysicalEntity` → +`Mars::EntityBase` → `IScheduledEvent` → `ListDLI<...>::Link`(@4) → `DLink`(@4); ++ `IScriptable`(@16), `NetworkObject`(@20), `HandleObject`(@28), `RigidBody`(@44) → +`Body`(@44), `SceneObject::Owner`(@212). + +**Game::Planet** (14): `Planet` → `Actor` → `TargetableEntity` → `PhysicalEntity` → +`EntityBase` → `IScheduledEvent` → `ListDLI<...>::Link`(@4) → `DLink`(@4); ++ `IScriptable`(@16), `NetworkObject`(@20), `HandleObject`(@28), `RigidBody`(@44) → +`Body`(@44), `SceneObject::Owner`(@212). + +### Consolidated single-inheritance spine (Mars physics/entity → Game gameplay): +``` +Body → RigidBody (Mars physics; mixed in at offset 44) +DLink → ListDLI::Link (Mars intrusive list node; offset 4) +IScheduledEvent IScriptable NetworkObject HandleObject IStreamable (Mars interfaces) + \______________ all mixed into ______________/ +Mars::EntityBase → Mars::PhysicalEntity → Game::TargetableEntity → Game::Actor → Game::Planet +StrategyPlayer → ... , ClientPlayer : StrategyPlayer, NetworkObject +AIObject → AIPlayer : AIObject, IStreamable +StarMapNode → StarSystem : StarMapNode (+NetworkObject/HandleObject/IStreamable) +``` + +--- + +## Artifacts / reproduction +Scripts (pushed to CT111 `/root/`): `CoreRecon.java`, `VtRecon.java`, `RttiHier.java`. +Run pattern: `systemctl stop reva-server; sleep 3;` then +`analyzeHeadless /srv/re-lab/ghidra-projects sots -process "Sword of the Stars.exe" -noanalysis -scriptPath /root -postScript