97 lines
7.5 KiB
Markdown
97 lines
7.5 KiB
Markdown
# SOTS1 — RE strategy
|
|
|
|
Founding strategy for reverse-engineering **Sword of the Stars (2006)** — original + expansions,
|
|
GOG Complete Collection v1.8.1, a 32-bit DirectX 9 MSVC C++ Windows game. Retail binary only;
|
|
no source was released. Personal, owned copy.
|
|
|
|
Lab: Ghidra 12 + ReVa (AI-assisted, MCP) in CT111; Win10 VM140 with x64dbg / Cheat Engine /
|
|
RenderDoc / apitrace / DXVK; Samba share `/srv/re-lab`. Infra: trikilli `services/re-lab.md`.
|
|
|
|
## North star (decided 2026-09-07)
|
|
**A functional reimplementation of the engine — behavior-equivalent, NOT byte-for-byte** — reached
|
|
by broad engine understanding first. Think OpenRCT2 / OpenMW / DevilutionX: a re-written engine that
|
|
reads the *original* assets and reproduces behavior, not a byte-identical source rebuild. Byte-matching
|
|
decompilation is explicitly **out of scope**. Bug-fixes and mods are welcome byproducts and useful
|
|
correctness checks along the way, not the primary goal.
|
|
|
|
## Goals (priority order)
|
|
1. **Broad engine understanding** — map the whole architecture, not just hot spots: main loop & tick,
|
|
subsystem boundaries, the core object model (empires, systems, fleets, tech), turn/AI logic,
|
|
the renderer, and save/load + asset I/O.
|
|
2. Functional reimplementation, built incrementally on that understanding (see method below).
|
|
3. Bug-fixing / modding as validation and early wins (battle-load slowness; 2 GB/4 GB ceiling).
|
|
4. Preservation and interop tools (save/mod editors, format parsers).
|
|
|
|
## Approach decision
|
|
Five styles evaluated:
|
|
|
|
| Style | Verdict | Why |
|
|
|---|---|---|
|
|
| Matching (byte-identical) decomp | **Reject** | Optimized MSVC C++ (inlining, STL, RTTI); no PC-C++ prior art; and not our goal |
|
|
| **Functional reimplementation** | **North star** | Behavior-equivalent engine reading original assets; incremental, community-scalable |
|
|
| Ghidra/ReVa annotated RE | **Core near-term** | Builds the broad understanding every later step needs; our lab is built for it |
|
|
| Binary patching / modding | **Byproduct** | Fast wins + behavioral baselines; not the driver |
|
|
| File-format / interop RE | **Opportunistic** | Safe, partly already done by the community |
|
|
|
|
## Reimplementation method (the bridge from RE → new engine)
|
|
Follow the **OpenRCT2 incremental model**: as understanding of a subsystem solidifies, reimplement it
|
|
in modern C++ in a **shim DLL that the original `sots.exe` calls into**, verifying *behavior* (not bytes)
|
|
against the original at each step; grow the reimplemented surface until the original code is fully
|
|
displaced. This keeps a runnable, testable artifact at every stage instead of a big-bang rewrite.
|
|
Golden rule for any future distribution: **ship engine code only, never original assets — users bring
|
|
their owned copy.** Clean-room boundaries where it matters.
|
|
|
|
## Known ground (community — don't re-derive)
|
|
- **Archives:** `sots.gob` and expansion `.gob`s are **renamed, uncompressed ZIPs** — 7-Zip opens them.
|
|
- **Modding is file-override:** files under `./Mods/` at matching relative paths win per `sots.ini` load order. Textures = DDS; models = custom format with community Xporter v1.19.
|
|
- **Save format partly RE'd:** BardezAnAvatar/Sots.Sots1.SavedGameEditor (C#, v1.8+, player/tech/system structs), ghbplayer/SOTSedit. A Rosetta Stone for the in-memory object model — accelerates goal 1.
|
|
- **Compiler:** confirm from the binary — PE **Rich header** + CRT imports (`msvcr71/80/90`). MSVC C++ ⇒ **RTTI present** (Ghidra Windows x86 PE RTTI Analyzer recovers vftables + class hierarchy — a major head start toward the object model); MSVC name mangling; heavily inlined STL.
|
|
- **Bug baselines to validate:** LAA/4GB PE flag (safe on GOG; Steam build breaks, "error 51"); `[CPU] ForceSingleCore=1` (engine is single-threaded).
|
|
|
|
## Phased plan
|
|
- **Phase 0 — baseline & fingerprint.** Unpack `sots.gob`; inventory data + `sots.ini`. Ghidra auto-analysis
|
|
+ **RTTI analyzer** + demangle; pin the compiler (Rich header / CRT). Import the save-editor struct
|
|
definitions as a naming seed. Validate LAA + `ForceSingleCore` for behavioral baselines. Commit project + notes.
|
|
- **Phase 1 — broad architecture map** (core near-term work). Recover and document, breadth-first:
|
|
the entry/init path and **main loop**; the **object model** (empires, star systems, fleets, tech tree —
|
|
cross-referenced to the save structs); the **turn/AI** update; the **renderer** (D3D9 init → present loop);
|
|
**save/load** and **asset/gob I/O**. Produce a living architecture doc + a class/subsystem map in the repo.
|
|
Good early anchors (they cut across the above): the battle-load path, the allocator/arena, D3D9 device init.
|
|
- **Phase 2 — reimplement incrementally.** Start with the most self-contained, well-understood subsystem
|
|
(likely a file-format parser or a leaf subsystem) as a shim-DLL reimplementation; verify behavior; expand.
|
|
- **Phase 3 — fixes, tools, preservation** (ongoing byproducts). Patcher for LAA + battle-load; save editor;
|
|
published format specs.
|
|
- **Deferred / out of scope:** byte-matching decompilation.
|
|
|
|
## First concrete targets
|
|
Compiler fingerprint · object-model structs (seeded from save editor) · main loop · D3D9 init ·
|
|
battle-load path · allocator. Breadth first — these anchor the wider map.
|
|
|
|
## Key sources
|
|
- Modding & file structure — https://swordofthestars.fandom.com/wiki/Introduction_to_modding , https://swordofthestars.fandom.com/wiki/Community_Mods
|
|
- Fixes/compat — https://www.pcgamingwiki.com/wiki/Sword_of_the_Stars
|
|
- Save-format head start — https://github.com/BardezAnAvatar/Sots.Sots1.SavedGameEditor , https://github.com/ghbplayer/SOTSedit
|
|
- MSVC C++ RE — https://dennisbabkin.com/blog/?t=reverse-engineer-virtual-functions-vs-cpp-compiler-vtable-purecall-cfg , https://www.christophbrill.de/en/posts/ghidra_msvc_win32/
|
|
- Incremental reimplementation model — https://desosa.nl/projects/openrct2/2020/03/02/openrct2,-porting-rollercoaster-tycoon-into-2020.html
|
|
|
|
## Open uncertainties
|
|
- Exact compiler/CRT unconfirmed — resolve from Rich header + CRT imports first.
|
|
- Many-core battle-load root cause inferred (single-threaded engine + core-scaling) — confirm empirically.
|
|
- Save-editor struct coverage may be partial (strong hint set, not a full spec).
|
|
- Reimplementation is a long arc; the value gate is the *understanding* (goal 1), which stands on its own.
|
|
|
|
_Strategy set 2026-09-07. Direction: broad understanding → functional reimplementation (not byte-matching)._
|
|
|
|
## Phase 2 — decided 2026-09-07
|
|
- **Two repos.** This repo = evidence + planning (private, binary-derived). **`alex/sots-engine`** =
|
|
the from-scratch engine source, **public-capable from day one**: clean-room (own code only; no game
|
|
data, decompiles, saves), MIT, C++17/CMake, assets external via `$SOTS_DATA_DIR`. Tracking for both
|
|
stays here. Binary facts cross over only via `ghidra/addresses.json` → `tools/gen_addresses.py` →
|
|
`sots-engine/include/generated/sots_addresses.h`.
|
|
- **Straight to the C++ shim** (no Python prototype): a proxy `binkw32.dll` (MinGW i686 on CT111,
|
|
MinHook) with per-hook `off|trace|compare|replace` modes; compare runs original + ours on a
|
|
snapshot and diffs; deterministic saves are the strongest equivalence oracle. The shim is the
|
|
engine's first frontend (`src/shim/`); a standalone `src/app/` comes when enough engine exists.
|
|
- **Slow accrual:** scaffold only what each milestone needs. Order: M0 bootstrap → M1 flat-KV config
|
|
loader → M2 id manifests → M3 Mars brace-block parser → M4 gobio VFS → serializers/RNG → turn pipeline.
|
|
- **Battle-load bug is parked** (not on the critical path); resume recipe in the backlog.
|