From 994b182f7ae00c50d37362f28b87bb3461fc78e0 Mon Sep 17 00:00:00 2001 From: alex Date: Mon, 7 Sep 2026 14:58:36 -0400 Subject: [PATCH] strings/config: battle-load smoking gun (affinity), lockstep sim, net map, Mods override --- campaign/board.md | 8 +- campaign/open-questions.md | 2 + findings/subsystems/strings-and-config.md | 153 ++++++++++++++++++++++ 3 files changed, 160 insertions(+), 3 deletions(-) create mode 100644 findings/subsystems/strings-and-config.md diff --git a/campaign/board.md b/campaign/board.md index 784d3ac..0bac816 100644 --- a/campaign/board.md +++ b/campaign/board.md @@ -19,7 +19,9 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`). | D3D9 device init | subsystem | backlog | — | 0% | 2026-09-07 | `Mars::DrawDevice`/`Camera` → Direct3DCreate9/CreateDevice | | `Mars::Buffer::gobio` | subsystem | in-progress | — | 0% | 2026-09-07 | .gob I/O; anchors the verify/ parser | | serializable-types vs save structs | verify | in-progress | — | 0% | 2026-09-07 | cross-ref `serializable-types.txt` with community save editors | -| battle-load path | subsystem | in-progress | — | 0% | 2026-09-07 | primary bug (many-core slowdown); LAA already on | +| battle-load path | subsystem | in-progress | — | 0% | 2026-09-07 | SMOKING GUN: ForceSingleCore -> SetProcessAffinityMask; Ghidra agent pinning the fn | | class hierarchy + key vftables | meta | in-progress | — | — | 2026-09-07 | Ghidra recon: inheritance + vftable sizes for core classes | -| data-model (.gob data files) | subsystem | in-progress | — | — | 2026-09-07 | harvest tech/ship/weapon/race tables from archives | -| string / config intel | meta | in-progress | — | — | 2026-09-07 | ini keys, paths, scripting, net msgs, battle-load strings | +| data-model (.gob data files) | subsystem | mapped | high | 100% | 2026-09-07 | findings/subsystems/data-model.md - tech/weapons/sections/races/AI all data-driven | +| string / config intel | meta | mapped | high | 100% | 2026-09-07 | findings/subsystems/strings-and-config.md | +| Mars brace-block parser | subsystem | backlog | — | 0% | 2026-09-07 | one recursive KV parser; reimpl lever + feeds verify/parsers | +| networking (SNM/FNM + GameSpy) | subsystem | backlog | — | 0% | 2026-09-07 | ~60 SNM strategy msgs, FNM file-xfer/host-migration; lockstep sim | diff --git a/campaign/open-questions.md b/campaign/open-questions.md index 38e9be9..701f444 100644 --- a/campaign/open-questions.md +++ b/campaign/open-questions.md @@ -10,3 +10,5 @@ Each links to the finding that raised it. Promoted to backlog or closed by **re- (from [[00-strategy]]) - **Save-editor struct coverage** — community editors cover players/tech/systems; fleet/combat state coverage may be partial. Treat as strong hints, verify each field against real saves. +- **Lockstep determinism** — `@SYNC ERROR DETECTED: Frame %d` indicates a deterministic lockstep sim. Big for reimplementation + old-vs-new differential (determinism makes equivalence checkable frame-by-frame). Confirm scope. (from [[strings-and-config]]) +- **Mods override mechanism** — `.gob` = `gobio::ZipFileSystem` with a `NativeFileSystem` loose-file override; this is how `./Mods` wins over archived files. Verify load order. (from [[strings-and-config]]) diff --git a/findings/subsystems/strings-and-config.md b/findings/subsystems/strings-and-config.md new file mode 100644 index 0000000..9cd52ff --- /dev/null +++ b/findings/subsystems/strings-and-config.md @@ -0,0 +1,153 @@ +# SOTS1 — String Intelligence (Sword of the Stars.exe) + +**Binary:** `/srv/re-lab/sots-game/Sword of the Stars.exe` (7,898,624 bytes) +**Type:** PE32 i386, GUI, 5 sections, MSVC-2010 (MSVCP100/MSVCR100), DirectX-9. +**Extraction:** `strings -n 5` (ASCII) + `strings -e l -n 5` (UTF-16). +Counts: **28,719 unique ASCII strings**, only **8 unique UTF-16** (this build stores nearly everything as ASCII/UTF-8; localized text lives in the `.gob` archives, not the exe). + +--- + +## 0. TL;DR — highest-value findings + +- **Battle-load / multicore bug is right here in the open:** the exe ships a **`ForceSingleCore`** config key (also literal `FORCESINGLE`) that drives **`SetProcessAffinityMask`**, logging **`Limiting process affinity to CPU-%i...`**. These three strings sit adjacent in `.rdata` (offsets 6,510,208 / 6,511,852 / 6,523,472). This is the classic SOTS1 fix for the combat-load hang on multi-core CPUs. +- **Build stamp:** version **`1.8.1`**, compiled **`Wed Dec 13 03:38:31 2017`** (this is the GOG 2018 rerelease of the original 2006 game). +- **PDB leak:** `E:\sots.clean\Distribution\Demo\Sword of the Stars - Retail.pdb` — confirms build tree layout and that this is the "Retail" config off the `sots.clean` branch. +- **No embedded scripting VM** (no Lua/Python). Game logic is **data-driven** via `Data/**/*.txt` + `*.csv` + `*.tech`; `*.script` files are GUI/HUD layout definitions, not a bytecode language. +- **Namespaces:** engine code lives under **`Mars`** (the engine) and game code under **`Game`** (RTTI decorations like `.?AV...@Game@@` / `@Mars@@` are everywhere and are a goldmine for the class map). +- **Networking = GameSpy SDK** (IRC-style Chat/Peer + NatNeg + master server) over **WSOCK32**, plus two in-house message families: **`SNM*`** (Strategy Net Messages, ~60 types) and **`FNM*`** (File Network Messages — chunked reliable transfer used for host migration / savegame sync). + +--- + +## 1. Config keys / sots.ini / registry + +SOTS does **not** embed literal `[Section]` header strings; it reads config through the Win32 profile API (`GetPrivateProfileStringA`, `GetPrivateProfileSectionA`, `WritePrivateProfileStringA/SectionA`) and the registry (`RegOpenKeyExA` on `Software\Sword of the Stars`). + +**Config files / stores (samples):** +- `sots.ini`, `sots.cfg` +- Registry key `Software\Sword of the Stars` (also `Software\GameSpy\GameSpy 3D\Registration`) +- `Profiles/Profile_*.ini`, `Profiles/*.profile`, `Profiles/lastprofile.dat` +- `CurrentProfile`, `ProfileID`, `ProfileName`, `lastprofile` + +**Full user-facing knob list (mined config keys):** + +*CPU / performance (BUG-relevant):* +- `ForceSingleCore` / `FORCESINGLE` + +*Video / render:* +- `adapter`, `width`, `height`, `fullscreen`, `windowed`, `fullscreenWidth`, `fullscreenHeight`, `antialiasing`, `brightness`, `vertexColors` + +*Audio:* +- `MusicVolume` (plus per-sound `*_sound_minrange`/`*_sound_maxrange`, `SOUND_START_DELAY`) + +*Gameplay defaults (game-setup options):* +- `MaxPlayers` ("must be an integer between 2 and 8"), max colonies (1–10) +- `EnableAlliances`, `EnablePlayerPasswords`, `EnableTeams`, `EnableTeamsGrouped` +- `DefaultAutoRefuel`, `DefaultAutoResolveCombat`, `DefaultHideFleetLines`, `DefaultShowAxes`, `DefaultShowTacticalInfo`, `DefaultShowTacticalOverlay`, `DefaultSubdueStars` +- `AutoPause` + +*Networking:* +- `HostName`, `HostPort`, `hostname`, `hostport`, `host_species`, `MaxTxMessageSize` + +--- + +## 2. File paths, extensions & asset pipeline + +**Extension histogram (top): ** `.X` (DirectX mesh, dominant model format), `.fx` (D3D9 effect/shader), `.tga`/`.dds`/`.bmp`/`.jpg` (textures), `.txt` + `.csv` (data tables), `.wav`/`.bik` (audio/Bink video), `.tech` (tech tree), `.script`/`.def`/`.effect`/`.shipsection` (custom defs), `.gob` (archive), `.sav`/`.dat`/`.ini`/`.cfg`. + +**Directory layout (from ~261 path strings):** +- `Data/` — `globals.txt`, `Species.txt`, `encounters.txt`, `music.csv`, `Combat/*.txt` (`ship.txt`, `planet.txt`, `terrain.txt`, `sensors.txt`, `camera.txt`, `drone.txt`, `spyship.txt`, `gravboat.txt`, `shipai.txt`, `damfx.txt`, `ctechvars.txt`…), `Strategy/playercolors.txt`, `Strategy/ai/affinity_weapon.csv`, `Strategy/ai/affinity_section.csv` +- `Species//...` — Human, Hiver, Tarkas, Liir, Zuul, Morrigi, `_NPC` (art/debris, sections, `_Spy.shipsection`) +- `Models/` — `Planets/`, `Shields/` (`sld_*` per hull class/tech), `Asteroids/Asteroids.csv`, `Trade/`, `Skysphere/` +- `Weapons/` — `_weapons.txt`, `_turrets.txt`, `_defaultweapons.txt`, `art/*.X` +- `TechTree/` — `MasterTechList.tech`, `tech.fx`, `techBases.fx`, `TechTreeDisplay.script` +- `Scenarios/` — `Gathering`, `ProgressionWars`, `UpstartApes*`, `BrokenEmpire`, `CivilWar`, `EndFlesh`, `HiverInvasion*`, `CrownJewels`, `Tournament`, `Antiquarians` (each a `.txt` + supporting `.csv`s) +- `GUI/` — `Combat/*.script` (HUD), `StarMap/Lines.script`, `Mouse/mouse_*.tga`, `SplashArt/`, `LoadScreen_Anim0*.tga` +- `Render/` — many `.fx` (`DefaultScene`, `StandardMaterial`, `GasCloud`, `CloakBadges`, `NodeDrive`, `StratStars`, `CombatStars`, `Miniship`, `StarMapBlobs_*`…) +- `Sounds/`, `Movies/` (`sotsintro.bik`), `Locale//Desc*.txt`, `Avatars/AvatarTable.txt`, `Badges/BadgeTable.txt` + +**`.gob` archive format:** the `gobio` namespace (`gobio::IFileSystem`, `NativeFileSystem`, `ZipFileSystem`, `Buffer`/`IBuffer`, `gobio_D3DXInclude`) shows **.gob is a ZIP-based virtual filesystem** with a native-directory fallback (so loose files can override archived ones — useful for asset extraction/modding). Runtime archives: `sots.gob` (1.5 GB) and `sots_local_en.gob` (628 MB, localization). + +--- + +## 3. Scripting + +**No embedded Lua/Python/bytecode VM.** Grep for `lua_`/`luaL_`/`Py_`/`.pyc`/`tolua` returned only incidental substrings. The engine is fully data-driven: +- `*.script` files are **GUI/HUD layout descriptions** parsed by the engine (`GUI/Combat/CombatHUD.script`, `SensorHUD.script`, `NoHUD.script`, `Models/Planets/PlanetResources.script`, `TechTree/TechTreeDisplay.script`, `GUI/StarMap/Lines.script`). Errors: `Error creating extra script %s.`, `Failed to open script %s!`, `StringTable: %s not found in script.` +- Scenario/AI behavior is described in `.txt`/`.csv` tables (`ScenarioKeyFile: Unrecognized script object id: %s`, `Script described %d slots, but max players is %d.`). "Script" here = declarative data, not code. + +--- + +## 4. Format strings & error/log messages (subsystem map) + +~991 printf-style and ~2,498 error/assert strings. They name subsystems cleanly: + +- **Combat sim:** `CombatPlayerAI::Update`, `(Combat Player %i) ...`, `Reserves::PickReserve`, `ShipBase::Damage/Load`, `CombatFieldTemplate`, `SpawnShips`, `ICarrier`/`Carrier runway`, `MissileBase`, `Bolt`/`Mine`/`ChainLightning`, `@SYNC ERROR DETECTED: Frame %d` (determinism check). +- **Strategy sim:** `StrategyClient`/`StrategyServer`/`StrategyApp`, `BuildQueue::*`, `HandleEvent: Cannot init/term/sync fleet|ship|design ...`, `Alliances:`, `SpecialProject:`, `PLAGUE:`, `Sensors: %f seconds to update sensors for all players.` +- **Tech:** `MasterTechTree`, `TechTree::GetSync()/GetInit()`, `Unable to find mesh for tech %s.` +- **Renderer / D3D9:** `Could not create D3D Device!`, `Created D3D device with full features` / `with fallback (%i)`, `*** Losing/Restoring D3D Device Objects ***`, `Adapter: %d: %s`, `Resolution: %dx%d @ %d-Hz`, `Antialiasing: %s`, `D3D available (SDK version %08x).` +- **Serialization:** the `Mars::StreamableHelper<...>` / `VectorHelper<...>` templates (see §8) wrap every `*Params` struct — this is the save/network serialization layer. +- **Crash handling:** `MiniDumpWriteDump` (via `DBGHELP.DLL`) — the game writes minidumps. + +--- + +## 5. Network + +**Transport:** `WSOCK32.dll`; `IPHLPAPI.DLL` (NAT/adapter discovery). + +**GameSpy SDK** (matchmaking + chat): +- Master/query servers: `%s.master.gamespy.com`, `%s.available.gamespy.com`, `%s.ms%d.gamespy.com`; `GameSpyHTTP/1.0`; registry `Software\GameSpy\GameSpy 3D\Registration`; `gamespy_splash_A.tga`. +- **NAT negotiation:** `NatNeg error: %d (progress %d).`, `NatNeg init error: %d.`, extensive `Network: ... NAT negotiation ...` logging. +- **IRC-style Chat/Peer protocol** (format strings): `PRIVMSG %s :%s`, `JOIN %s %s`, `PART %s :%s`, `MODE %s ...`, `NICK %s`, `TOPIC`, `NOTICE`, `USER`, `LOGIN`, plus GameSpy key exchange (`GETKEY`, `SETCKEY`, `CRYPT des %d %s`) and browser tokens (`\hostname\gamemode`, `#GSP!%s`). + +**In-house P2P group layer** ("Network:" logger): host/client group model with **host migration** — `Network: New group created. %s is host.`, `DoHost()/DoConnect()/DoDisconnect()`, `HostMigration: Begin/Setup migration...`, `HostMigration failed: ...`, `Rejoin: ...`. + +**`SNM*` — Strategy Net Message types** (~60; the strategy-layer protocol enum). Samples: `SNMJoin`/`SNMJoinReply`, `SNMEndTurn`, `SNMHostCombat`/`SNMLaunchCombat`/`SNMJoinCombat`, `SNMEncounterQuery*`/`SNMEncounterResults`, `SNMChat`/`SNMLobbyChat`, `SNMAlliance*`, `SNMSlot*` (lobby slot mgmt + passwords: `SNMSlotReqPW`/`SNMSlotPWAttempt`/`SNMSlotPWAccepted`), `SNMSetPlayerStatus`, `SNMRunAI`/`SNMKillAI`/`SNMMakePlayerAI`, `SNMSetTimers`, `SNMRefreshSlots`. + +**`FNM*` — File Network Messages** (`Mars` namespace): `FNMBase`, `FNMChunk`/`FNMChunkReply`, `FNMDone`/`FNMDoneReply`, `FNMSendNotify`/`FNMSendReply`, `FNMError` — a chunked reliable **file-transfer** protocol (drives `FileTransfer: Cannot open %s to send to %s.` and the host-migration savegame push). + +--- + +## 6. Version / build + +- **`1.8.1`** (game version; `Invalid version number string (must conform to '0.0.0'...)`). +- Build timestamp: **`Wed Dec 13 03:38:31 2017`**. +- Window class / mutex: `Kerberos_SwordOfTheStars_WndCls`, `Kerberos_SwordOfTheStars_Mutex`. +- Title format: `Sword of the Stars%s (%s %s)`; MOTD URL `www.kerberos-productions.com/motd`. +- **PDB:** `E:\sots.clean\Distribution\Demo\Sword of the Stars - Retail.pdb`. +- Toolchain: MSVC 2010 (`MSVCP100.dll`, `MSVCR100.dll`). Middleware DLLs: `d3d9.dll`, `d3dx9_42.dll`, `DSOUND.dll`, `binkw32.dll` (Bink video), `WINMM.dll`, `DBGHELP.DLL`. +- Editions gated in-exe: `SOTS_COLLECTORSEDITION_ENABLED_TITLE`, `SOTS_COMPLETECOLLECTION_ENABLED_TITLE`, `SOTS_ARGOSNAVALYARD_TITLE`. + +--- + +## 7. BUG watch — CPU / thread / core / affinity (battle-load hang) + +Smoking-gun strings, with byte offsets in the file: + +| Offset | String | Meaning | +|---|---|---| +| 6,510,208 | `Limiting process affinity to CPU-%i...` | log emitted when single-core mode engages | +| 6,511,852 | `ForceSingleCore` | ini/cfg key that triggers it | +| 6,523,472 | `FORCESINGLE` | uppercased variant (likely token/registry) | +| 7,172,098 | `SetProcessAffinityMask` | KERNEL32 import performing the pin | +| 5,266,079 / 5,266,317 | `F ;Cpu` | fragment near the CPU-detect block | + +Supporting threading imports/strings: `CreateThread`, `CreateMutexA`, `GetCurrentThreadId`, `InterlockedExchange`, `InterlockedCompareExchange`, `IsProcessorFeaturePresent`, `SetThreadPriority`, `_configthreadlocale`, `Initializing Streaming sound update thread...`, `std::_Lockit`. The **audio streaming thread** and the combat sim are the likely contenders in the multi-core race; `ForceSingleCore`→`SetProcessAffinityMask(CPU-i)` is the shipped mitigation. Determinism guard `@SYNC ERROR DETECTED: Frame %d` shows the sim expects lockstep — worth correlating with the load path. + +*(Note: most `.?AV...Affinity@Game@@` and `AITAttack*` hits are gameplay "diplomatic affinity" / AI, NOT CPU affinity — don't confuse them.)* + +--- + +## 8. Data model (tables / params structs) + +No SQL/SQLite. Two shapes: + +**A. CSV/TXT tables** with a schema-ish loader: `[%s,%i] CSV row parsing failed.`, `Field '%s' precedes 'name' field and will be ignored in %s.`, `AsteroidEdictDB: Table is empty!`, `Sprite Table: Sprite %s multiply defined/not found.`, `StringTable`/`GlobalConsts: %s being registered after loading const table.`, `Weapon not found: "%s" - Was it added to the index file?`. There is also XML/DTD validation (MSXML) for some data. + +**B. Binary-serialized `*Params` structs** (save/network via `Mars::StreamableHelper` / `VectorHelper`). The full RTTI list gives the effective "record types": +`StrategyGameParams`, `StrategyGameCreateParams`, `StrategyGameLoadParams`, `StrategyPlayerParams`, `StrategySessionParams`, `StrategySimParams`, `StrategyTimerParams`, `StrategyScriptParams`, `StrategyHostParams`, `StrategyLaunchParams`, `SystemParams`, `StarMapParams`, `NodePointParams`, `ShipParams` (with nested `Section`, `GunBank`), `CombatEncounterParams`, `CombatLaunchParams`, `EncounterParams`, `LaunchParams`, `SyncParams`, `LocustEncounterParams`, `RefugeesEncounterParams`, `SwarmEncounterParams`, `VNHomeEncounterParams`, `VonNeumannEncounterParams`. These are the serialized game-state schema — the right targets for savegame/netpacket structure work. + +**Engine/game split:** RTTI shows engine classes under **`@Mars@@`** (Frame, ModelFrame, PhysicalEntity, ListFramework, gobio, FNM*, Streamable) and game logic under **`@Game@@`** (all the strategy/combat classes). Use this to partition the code map for the Ghidra agent. + +--- + +*Extraction scripts left in CT111: `/tmp/extract.py`, `/tmp/g2.sh`, `/tmp/g3.sh`, `/tmp/g4.sh`.*