sots-re/findings/00-strategy.md

SOTS1 — RE strategy

Founding strategy for reverse-engineering Sword of the Stars (2006) — original + expansions, GOG Complete Collection v1.8.1, a 32-bit DirectX 9 MSVC C++ Windows game. Retail binary only; no source was released. Personal, owned copy.

Lab: Ghidra 12 + ReVa (AI-assisted, MCP) in CT111; Win10 VM140 with x64dbg / Cheat Engine / RenderDoc / apitrace / DXVK; Samba share /srv/re-lab. Infra: trikilli services/re-lab.md.

North star (decided 2026-09-07)

A functional reimplementation of the engine — behavior-equivalent, NOT byte-for-byte — reached by broad engine understanding first. Think OpenRCT2 / OpenMW / DevilutionX: a re-written engine that reads the original assets and reproduces behavior, not a byte-identical source rebuild. Byte-matching decompilation is explicitly out of scope. Bug-fixes and mods are welcome byproducts and useful correctness checks along the way, not the primary goal.

Goals (priority order)

1. Broad engine understanding — map the whole architecture, not just hot spots: main loop & tick, subsystem boundaries, the core object model (empires, systems, fleets, tech), turn/AI logic, the renderer, and save/load + asset I/O.
2. Functional reimplementation, built incrementally on that understanding (see method below).
3. Bug-fixing / modding as validation and early wins (battle-load slowness; 2 GB/4 GB ceiling).
4. Preservation and interop tools (save/mod editors, format parsers).

Approach decision

Five styles evaluated:

Style	Verdict	Why
Matching (byte-identical) decomp	Reject	Optimized MSVC C++ (inlining, STL, RTTI); no PC-C++ prior art; and not our goal
Functional reimplementation	North star	Behavior-equivalent engine reading original assets; incremental, community-scalable
Ghidra/ReVa annotated RE	Core near-term	Builds the broad understanding every later step needs; our lab is built for it
Binary patching / modding	Byproduct	Fast wins + behavioral baselines; not the driver
File-format / interop RE	Opportunistic	Safe, partly already done by the community

Reimplementation method (the bridge from RE → new engine)

Follow the OpenRCT2 incremental model: as understanding of a subsystem solidifies, reimplement it in modern C++ in a shim DLL that the original sots.exe calls into, verifying behavior (not bytes) against the original at each step; grow the reimplemented surface until the original code is fully displaced. This keeps a runnable, testable artifact at every stage instead of a big-bang rewrite. Golden rule for any future distribution: ship engine code only, never original assets — users bring their owned copy. Clean-room boundaries where it matters.

Known ground (community — don't re-derive)

• Archives: sots.gob and expansion .gobs are renamed, uncompressed ZIPs — 7-Zip opens them.
• Modding is file-override: files under ./Mods/ at matching relative paths win per sots.ini load order. Textures = DDS; models = custom format with community Xporter v1.19.
• Save format partly RE'd: BardezAnAvatar/Sots.Sots1.SavedGameEditor (C#, v1.8+, player/tech/system structs), ghbplayer/SOTSedit. A Rosetta Stone for the in-memory object model — accelerates goal 1.
• Compiler: confirm from the binary — PE Rich header + CRT imports (msvcr71/80/90). MSVC C++ ⇒ RTTI present (Ghidra Windows x86 PE RTTI Analyzer recovers vftables + class hierarchy — a major head start toward the object model); MSVC name mangling; heavily inlined STL.
• Bug baselines to validate: LAA/4GB PE flag (safe on GOG; Steam build breaks, "error 51"); [CPU] ForceSingleCore=1 (engine is single-threaded).

Phased plan

• Phase 0 — baseline & fingerprint. Unpack sots.gob; inventory data + sots.ini. Ghidra auto-analysis
  • RTTI analyzer + demangle; pin the compiler (Rich header / CRT). Import the save-editor struct definitions as a naming seed. Validate LAA + ForceSingleCore for behavioral baselines. Commit project + notes.
• Phase 1 — broad architecture map (core near-term work). Recover and document, breadth-first: the entry/init path and main loop; the object model (empires, star systems, fleets, tech tree — cross-referenced to the save structs); the turn/AI update; the renderer (D3D9 init → present loop); save/load and asset/gob I/O. Produce a living architecture doc + a class/subsystem map in the repo. Good early anchors (they cut across the above): the battle-load path, the allocator/arena, D3D9 device init.
• Phase 2 — reimplement incrementally. Start with the most self-contained, well-understood subsystem (likely a file-format parser or a leaf subsystem) as a shim-DLL reimplementation; verify behavior; expand.
• Phase 3 — fixes, tools, preservation (ongoing byproducts). Patcher for LAA + battle-load; save editor; published format specs.
• Deferred / out of scope: byte-matching decompilation.

First concrete targets

Compiler fingerprint · object-model structs (seeded from save editor) · main loop · D3D9 init · battle-load path · allocator. Breadth first — these anchor the wider map.

Key sources

• Modding & file structure — https://swordofthestars.fandom.com/wiki/Introduction_to_modding , https://swordofthestars.fandom.com/wiki/Community_Mods
• Fixes/compat — https://www.pcgamingwiki.com/wiki/Sword_of_the_Stars
• Save-format head start — https://github.com/BardezAnAvatar/Sots.Sots1.SavedGameEditor , https://github.com/ghbplayer/SOTSedit
• MSVC C++ RE — https://dennisbabkin.com/blog/?t=reverse-engineer-virtual-functions-vs-cpp-compiler-vtable-purecall-cfg , https://www.christophbrill.de/en/posts/ghidra_msvc_win32/
• Incremental reimplementation model — https://desosa.nl/projects/openrct2/2020/03/02/openrct2,-porting-rollercoaster-tycoon-into-2020.html

Open uncertainties

• Exact compiler/CRT unconfirmed — resolve from Rich header + CRT imports first.
• Many-core battle-load root cause inferred (single-threaded engine + core-scaling) — confirm empirically.
• Save-editor struct coverage may be partial (strong hint set, not a full spec).
• Reimplementation is a long arc; the value gate is the understanding (goal 1), which stands on its own.

Strategy set 2026-09-07. Direction: broad understanding → functional reimplementation (not byte-matching).

Phase 2 — decided 2026-09-07

• Two repos. This repo = evidence + planning (private, binary-derived). alex/sots-engine = the from-scratch engine source, public-capable from day one: clean-room (own code only; no game data, decompiles, saves), MIT, C++17/CMake, assets external via $SOTS_DATA_DIR. Tracking for both stays here. Binary facts cross over only via ghidra/addresses.json → tools/gen_addresses.py → sots-engine/include/generated/sots_addresses.h.
• Straight to the C++ shim (no Python prototype): a proxy binkw32.dll (MinGW i686 on CT111, MinHook) with per-hook off|trace|compare|replace modes; compare runs original + ours on a snapshot and diffs; deterministic saves are the strongest equivalence oracle. The shim is the engine's first frontend (src/shim/); a standalone src/app/ comes when enough engine exists.
• Slow accrual: scaffold only what each milestone needs. Order: M0 bootstrap → M1 flat-KV config loader → M2 id manifests → M3 Mars brace-block parser → M4 gobio VFS → serializers/RNG → turn pipeline.
• Battle-load bug is parked (not on the critical path); resume recipe in the backlog.