diff --git a/findings/00-strategy.md b/findings/00-strategy.md index b257043..624657c 100644 --- a/findings/00-strategy.md +++ b/findings/00-strategy.md @@ -5,59 +5,79 @@ GOG Complete Collection v1.8.1, a 32-bit DirectX 9 MSVC C++ Windows game. Retail no source was released. Personal, owned copy. Lab: Ghidra 12 + ReVa (AI-assisted, MCP) in CT111; Win10 VM140 with x64dbg / Cheat Engine / -RenderDoc / apitrace / DXVK; Samba share `/srv/re-lab`. See trikilli `services/re-lab.md` for infra. +RenderDoc / apitrace / DXVK; Samba share `/srv/re-lab`. Infra: trikilli `services/re-lab.md`. + +## North star (decided 2026-09-07) +**A functional reimplementation of the engine — behavior-equivalent, NOT byte-for-byte** — reached +by broad engine understanding first. Think OpenRCT2 / OpenMW / DevilutionX: a re-written engine that +reads the *original* assets and reproduces behavior, not a byte-identical source rebuild. Byte-matching +decompilation is explicitly **out of scope**. Bug-fixes and mods are welcome byproducts and useful +correctness checks along the way, not the primary goal. ## Goals (priority order) -1. Understand the engine and game logic. -2. Enable modding and bug-fixing — the slow tactical-battle load on many-core CPUs; the 2 GB/4 GB memory ceiling. -3. Preservation — keep it runnable on modern Windows. -4. Interop tools — save/mod editors, file-format parsers. +1. **Broad engine understanding** — map the whole architecture, not just hot spots: main loop & tick, + subsystem boundaries, the core object model (empires, systems, fleets, tech), turn/AI logic, + the renderer, and save/load + asset I/O. +2. Functional reimplementation, built incrementally on that understanding (see method below). +3. Bug-fixing / modding as validation and early wins (battle-load slowness; 2 GB/4 GB ceiling). +4. Preservation and interop tools (save/mod editors, format parsers). ## Approach decision Five styles evaluated: | Style | Verdict | Why | |---|---|---| -| Matching (byte-identical) decomp | **Reject** | Optimized MSVC C++ (inlining, STL, RTTI); no PC-C++ prior art; no payoff until ~complete | -| Functional reimplementation | **Defer** | Right end-state, wrong first move; if ever, OpenRCT2 incremental shim-DLL method, ship engine only never assets | -| Ghidra/ReVa annotated RE | **Core** | Highest for goal 1; our lab is built for it; substrate for everything else | -| Binary patching / modding | **Do first** | Highest for goals 2 & 3; both flagship bugs already have community mitigations | -| File-format / interop RE | **Opportunistic** | Safest legally; partly already done by the community | +| Matching (byte-identical) decomp | **Reject** | Optimized MSVC C++ (inlining, STL, RTTI); no PC-C++ prior art; and not our goal | +| **Functional reimplementation** | **North star** | Behavior-equivalent engine reading original assets; incremental, community-scalable | +| Ghidra/ReVa annotated RE | **Core near-term** | Builds the broad understanding every later step needs; our lab is built for it | +| Binary patching / modding | **Byproduct** | Fast wins + behavioral baselines; not the driver | +| File-format / interop RE | **Opportunistic** | Safe, partly already done by the community | + +## Reimplementation method (the bridge from RE → new engine) +Follow the **OpenRCT2 incremental model**: as understanding of a subsystem solidifies, reimplement it +in modern C++ in a **shim DLL that the original `sots.exe` calls into**, verifying *behavior* (not bytes) +against the original at each step; grow the reimplemented surface until the original code is fully +displaced. This keeps a runnable, testable artifact at every stage instead of a big-bang rewrite. +Golden rule for any future distribution: **ship engine code only, never original assets — users bring +their owned copy.** Clean-room boundaries where it matters. ## Known ground (community — don't re-derive) -- **Archives:** `sots.gob` and expansion `.gob`s are **renamed, uncompressed ZIPs** — 7-Zip opens them. No archive RE needed. -- **Modding is file-override:** drop a file at the same relative path under `./Mods/`; `sots.ini` load order prefers it over the `.gob`. Textures = DDS; models = custom format with community Xporter v1.19. -- **Save format partly RE'd:** BardezAnAvatar/Sots.Sots1.SavedGameEditor (C#, v1.8+, player/tech-tree/system structs), ghbplayer/SOTSedit, a SourceForge editor. Use as a Rosetta Stone for the in-memory object model, not just on-disk saves. -- **Bug mitigations to validate on our GOG copy (behavioral baseline):** - - LAA/4GB: flip the Large-Address-Aware PE flag (safe on GOG; the Steam build breaks with checksum "error 51"). - - Battle load: `[CPU] ForceSingleCore=1` in `sots.ini` — the engine is single-threaded. Goal is to find *why* many cores hurt, not just set the flag. -- **Compiler:** confirm from the binary, don't assume — read the PE **Rich header** + CRT imports (`msvcr71/80/90`). Candidates VS2003/2005/2008. MSVC C++ ⇒ **RTTI present** (Ghidra's Windows x86 PE RTTI Analyzer recovers vftables + class hierarchy — big head start); MSVC name mangling (Ghidra demangles); expect heavily inlined STL. +- **Archives:** `sots.gob` and expansion `.gob`s are **renamed, uncompressed ZIPs** — 7-Zip opens them. +- **Modding is file-override:** files under `./Mods/` at matching relative paths win per `sots.ini` load order. Textures = DDS; models = custom format with community Xporter v1.19. +- **Save format partly RE'd:** BardezAnAvatar/Sots.Sots1.SavedGameEditor (C#, v1.8+, player/tech/system structs), ghbplayer/SOTSedit. A Rosetta Stone for the in-memory object model — accelerates goal 1. +- **Compiler:** confirm from the binary — PE **Rich header** + CRT imports (`msvcr71/80/90`). MSVC C++ ⇒ **RTTI present** (Ghidra Windows x86 PE RTTI Analyzer recovers vftables + class hierarchy — a major head start toward the object model); MSVC name mangling; heavily inlined STL. +- **Bug baselines to validate:** LAA/4GB PE flag (safe on GOG; Steam build breaks, "error 51"); `[CPU] ForceSingleCore=1` (engine is single-threaded). ## Phased plan -- **Phase 0 — baseline & fingerprint.** Unpack `sots.gob`; inventory data + `sots.ini` keys. Ghidra auto-analysis + **RTTI analyzer** + demangle; read Rich header / CRT imports to pin the compiler. Validate LAA + `ForceSingleCore` to get a known-good behavioral baseline. Commit Ghidra project + notes. -- **Phase 1 — annotated RE of the three hot paths** (core program): - 1. **Battle-load path** (top priority). Timestamp the tactical-load transition under apitrace; break in x64dbg, walk up into Ghidra. Diff `ForceSingleCore=0` vs `=1` to localize core-count-sensitive code (`GetSystemInfo`/`dwNumberOfProcessors`, `CreateThread` loops, `QueryPerformanceCounter` spin-waits, per-core allocation). - 2. **Memory-allocation path** (LAA). Find the global allocator/arena; instrument peak use in the large-battle crash; decide whether LAA alone suffices or a deeper allocator shim is warranted. - 3. **D3D9 device init.** RenderDoc/apitrace `Direct3DCreate9`→`CreateDevice`; xref `d3d9.dll` imports to the renderer bootstrap + present loop. Anchor for future shims and compat fixes. - Cross-reference recovered structs against the existing save-editor field defs to name core objects fast. -- **Phase 2 — ship fixes as patches/shims.** Package LAA + `sots.ini` tweaks as a *patcher*, not a patched binary. Prototype the battle-load fix as an asm/hex patch or DLL wrapper (ASI loader / DXVK-style shim) — reversible and legally clean. -- **Phase 3 — interop tools.** Extend the save-format work into a maintained editor + a published format spec (format facts are safe to publish). Add `.gob`/DDS/model helpers only where community tooling is thin. -- **Deferred:** matching decomp (skip); full reimplementation (someday, engine-only, users bring assets). +- **Phase 0 — baseline & fingerprint.** Unpack `sots.gob`; inventory data + `sots.ini`. Ghidra auto-analysis + + **RTTI analyzer** + demangle; pin the compiler (Rich header / CRT). Import the save-editor struct + definitions as a naming seed. Validate LAA + `ForceSingleCore` for behavioral baselines. Commit project + notes. +- **Phase 1 — broad architecture map** (core near-term work). Recover and document, breadth-first: + the entry/init path and **main loop**; the **object model** (empires, star systems, fleets, tech tree — + cross-referenced to the save structs); the **turn/AI** update; the **renderer** (D3D9 init → present loop); + **save/load** and **asset/gob I/O**. Produce a living architecture doc + a class/subsystem map in the repo. + Good early anchors (they cut across the above): the battle-load path, the allocator/arena, D3D9 device init. +- **Phase 2 — reimplement incrementally.** Start with the most self-contained, well-understood subsystem + (likely a file-format parser or a leaf subsystem) as a shim-DLL reimplementation; verify behavior; expand. +- **Phase 3 — fixes, tools, preservation** (ongoing byproducts). Patcher for LAA + battle-load; save editor; + published format specs. +- **Deferred / out of scope:** byte-matching decompilation. ## First concrete targets -Battle-load path · memory-allocation path · D3D9 device init. +Compiler fingerprint · object-model structs (seeded from save editor) · main loop · D3D9 init · +battle-load path · allocator. Breadth first — these anchor the wider map. ## Key sources -- SOTS1 modding & file structure — https://swordofthestars.fandom.com/wiki/Introduction_to_modding , https://swordofthestars.fandom.com/wiki/Community_Mods +- Modding & file structure — https://swordofthestars.fandom.com/wiki/Introduction_to_modding , https://swordofthestars.fandom.com/wiki/Community_Mods - Fixes/compat — https://www.pcgamingwiki.com/wiki/Sword_of_the_Stars -- Battle-load / threading — https://steamcommunity.com/app/42890/discussions/0/615086038683223062 - Save-format head start — https://github.com/BardezAnAvatar/Sots.Sots1.SavedGameEditor , https://github.com/ghbplayer/SOTSedit -- MSVC C++ RE technique — https://dennisbabkin.com/blog/?t=reverse-engineer-virtual-functions-vs-cpp-compiler-vtable-purecall-cfg , https://www.christophbrill.de/en/posts/ghidra_msvc_win32/ +- MSVC C++ RE — https://dennisbabkin.com/blog/?t=reverse-engineer-virtual-functions-vs-cpp-compiler-vtable-purecall-cfg , https://www.christophbrill.de/en/posts/ghidra_msvc_win32/ - Incremental reimplementation model — https://desosa.nl/projects/openrct2/2020/03/02/openrct2,-porting-rollercoaster-tycoon-into-2020.html ## Open uncertainties -- Exact compiler/CRT of the v1.8.1 binary unconfirmed — resolve from Rich header + CRT imports first. -- Many-core battle-load root cause is inferred (single-threaded engine + core-scaling logic) — confirm by the Phase-1 diff. -- Community save editors cover what their authors needed (tech tree, players, systems); fleet/combat-state coverage may be partial. +- Exact compiler/CRT unconfirmed — resolve from Rich header + CRT imports first. +- Many-core battle-load root cause inferred (single-threaded engine + core-scaling) — confirm empirically. +- Save-editor struct coverage may be partial (strong hint set, not a full spec). +- Reimplementation is a long arc; the value gate is the *understanding* (goal 1), which stands on its own. -_Research 2026-09-07._ +_Strategy set 2026-09-07. Direction: broad understanding → functional reimplementation (not byte-matching)._