alex/sots-re
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

lane W: SvSctOb variant factories; fix the four save_reader.py defects openly

Browse source 
Part 1 (notes side): findings/objects/svsctob-variants.md records the two maps
that are nowhere on the wire and were read out of the game -- EncID -> class
from a 23-entry dword jump table at 0x0052bf60 indexed by EncID-1 (0x0052bf00),
and xscn -> class from an exhaustive four-way _stricmp chain at 0x005a7050.
Twelve live EncIDs, four scenario names, and "indsys" =
Game::SVSOIndependentSystems whose Read and Write are both the shared `ret 4`
stub at 0x005f8ac0, so its empty frame is correct output rather than a
truncation. Also: SVSOSots::Read accepts NPCPlr and hastraps, which Write never
emits -- read-only backward compatibility, not a hole in the recovery. And a
correction to the recovery itself: SVSOCrowDefenders writes `dsys` INSIDE the
ndsys loop; layouts.json calls it a plain member and no save can settle it
because both counts are 0 everywhere. 13 addresses in ghidra/addresses.d/
lane-w.json; gen_addresses.py merges to 640 with no duplicate name.

Part 2: lane G found four defects in both readers and deliberately did not
patch the oracle mid-campaign. Fixed now, with tests, and byte-neutral.
  1. Game::SystemParams field 1 is a string, not an int (empty string == four
     zero bytes == int 0, so it round-tripped by luck).
  2. ObservedTech/ObservedWeapon odet is a bool, not an int (byte-safe only
     because a 4-char tag makes both items 12 bytes; 3 chars would not).
  3. SpeciesRatios nv is a count, not a field.
  4. ShipRecords srbd is a count, not a field -- and this one is behaviourally
     confirmed, not inferred: srbd takes 0, 1, 3 and 4 across the players and
     every non-zero count is followed by exactly srbd x 5 scalars.
Note that 3 and 4 were an ABSENCE in save_reader.py, not an error: ShipRecs and
civr were both A(..., "any"), so the fix had to add the shapes rather than
retype a field.

Byte-neutrality: every item's inflated offset is unchanged on all four saves
(38,933 / 39,843 / 40,300 / 35,771 offsets, sequences identical), so no item
boundary moved. state_checksum.py still reports coverage: PROVED on all four
with the same rebuilt byte counts. The /CreateParams and /Sim/players digests do
change, because they hash typed VALUES and two fixes change what a value is --
and the value-byte deltas balance exactly: odet items x 3 plus p1 items x 4.
--strict exit 0 on all four saves; tests 36 -> 48.

findings/objects/wire-schema-closeout.md carries the whole account, including
the proof that CD/TurnCommands_v5 cannot be typed without a save that has
issued orders.
This commit is contained in:
lane-w 2026-09-08 07:05:09 -04:00
parent a0dd707261
commit 2308b6edb9
5 changed files with 711 additions and 5 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

147
findings/objects/svsctob-variants.md Normal file
View file

@ -0,0 +1,147 @@
# `SvSctOb`: the script-object tree, and the two factories that key it
Lane W, 2026-09-08. Host/static only — VM140 was held by lane U and the game was never run.
`Game::StrategyServer` writes one member tagged `SvSctOb`, typed
`Mars::StreamableHelper<Game::SVScriptObject>` — a **polymorphic pointer**. In every save we
hold it points at a `Game::SVSOSots`, whose serializer (`Write` 0x0059ddf0, `Read` 0x005a7a40,
vftable 0x00A063C4) emits, in disk order:
```
ScnID int
ScnObj StreamableHelper<SVSOScenarioBase> -- conditional; NULL, so absent, in all four saves
numx int count
x numx { xscn string -- a scenario NAME
xsc StreamableHelper<SVScriptObject> }
NEncObjs int count
x NEncObjs { EncID int -- an encounter ID
EncObj StreamableHelper<SVScriptObject> }
```
Both `xsc` and `EncObj` are polymorphic and **dispatched by the key item that immediately
precedes them**. Neither map is anywhere on the wire: a reader that does not know them can only
carry the bodies as opaque blobs, which is exactly what `sots-engine` was doing (147 items on
turn1). Both maps were read out of the game.
## 1. `EncID` → class: a 23-entry jump table
`Game::SVSOSots::Read` calls a get-or-create helper at **0x005a4450** (it scans the already
loaded `(id, obj)` pair vector at `this+0x0C..0x10` and only on a miss calls the real factory).
The factory is **0x0052bf00**:
```
0052bf00 push ebp; mov ebp,esp; mov eax,[ebp+8]
0052bf06 dec eax
0052bf07 cmp eax,0x16
0052bf0a ja 0052bf5b ; -> xor eax,eax; ret (NULL)
0052bf0c jmp dword ptr [eax*4 + 0x0052bf60]
```
so the encoding is a **dword jump table at 0x0052bf60, 23 entries, indexed by `EncID - 1`**.
Each thunk runs a ctor, and the class name comes from the `mov dword ptr [obj], <vftable>` store
in that ctor — a demangled-RTTI name, not a guess.
| EncID | ctor | class | in our saves |
|---|---|---|---|
| 1 | 0x0052bb90 → 0x0052b790 | `Game::SVSOVonNeumann` | yes |
| 3 | 0x0051aba0 → 0x0051a870 | `Game::SVSOSwarm` | yes |
| 4 | 0x0051a270 | `Game::SVSODerelict` | yes |
| 5 | 0x0052ae10 | `Game::SVSOMonitor` | yes |
| 7 | 0x0050e0f0 | `Game::SVSOSystemKiller` | no |
| 8 | 0x0050de50 | `Game::SVSOPuppetMaster` | no |
| 9 | 0x0051a820 | `Game::SVSOSlaversRefuel` | yes |
| 10 | 0x0051ae20 | `Game::SVSOSwarmQueen` | yes |
| 14 | 0x0052a030 → 0x00529080 | `Game::SVSOLocust` | no |
| 17 | 0x00524190 | `Game::SVSOCrowRuins` | yes |
| 20 | 0x005295a0 → 0x005293f0 | `Game::SVSORefugees` | yes |
| 21 | 0x004f5210 → 0x004f5170 | `Game::SVSOOrtgay` | no |
Ids 2, 6, 11–13, 15, 16, 18, 19, 22, 23 are dead slots that return NULL; so is anything outside
1..23 (the `ja`). **There is no named enum** — the ids are bare integers and the jump table is the
only encoding of the map anywhere in the binary.
The eight ids the saves carry were independently identified by tag-sequence match against the
recovered serializers *before* the table was read, and the two agree exactly on all eight.
## 2. `xscn` → class: a four-way `_stricmp` chain
`0x005a7050`, falling through to `"Error creating extra script %s.\n"`. Flat if-chain, no table,
no registration list, and **exhaustive** — there are four scenario names in the whole game.
| `xscn` | test at | ctor | class |
|---|---|---|---|
| `traps` | 0x005a7085 | 0x0052cbb0 → 0x0052c940 | `Game::SVSOTraps` |
| `crowdefs` | 0x005a70a4 | 0x0052b3c0 | `Game::SVSOCrowDefenders` |
| `indsys` | 0x005a70be | 0x0075b530 | `Game::SVSOIndependentSystems` |
| `gmtrigger` | 0x005a70d8 | 0x004f5630 | `Game::SVSOGrandMenaceTrigger` |
Corroborated by the new-game seeder at 0x005a7d70, which registers exactly `traps`, `crowdefs`,
`indsys`, and — only when `ScnObj == NULL` — `gmtrigger`. That is the four-entry list our saves
carry, in that order.
### `indsys` serializes nothing, and that is a fact, not an absence of evidence
`Game::SVSOIndependentSystems` has no entry in `objects/layouts.json`, which on its own only says
the recovery never found a serializer. Its vftable is **0x00A20314** (from the `C7 06 14 03 A2 00`
store at 0x0075b585). Comparing slots against `Game::SVSOCrowDefenders`'s vftable at 0x009F5904
(slot0 dtor, slot1 `Read`, slot2 `Write` — both of which match `layouts.json`):
```
slot0 dtor = 0x0075afb0
slot1 Read = 0x005f8ac0
slot2 Write = 0x005f8ac0
```
and 0x005f8ac0 is `C2 04 00` — a bare `ret 4` stub shared across the binary. So the class uses the
inherited no-op for both directions and the `indsys` frame is **genuinely zero-length on disk**.
Its 0x1c4-byte body (two 7-element arrays of 0x20-byte records) is runtime state rebuilt each
session. The empty frame in the saves is the correct output, not a truncation.
## 3. Two loop-nesting questions the bytes could not answer
Both counts involved are 0 in every save, so only the binary settles them.
* **`Game::SVSOCrowDefenders::Write` (0x004f8c90) — the recovery is wrong about `dsys`.** Tag
strings at .rdata 0x009f0270. Emission order:
`sys`; `ndsys` count then a **loop writing `dsys`**; `ndes` count then a loop writing `des`;
`drad`. `objects/layouts.json` records `dsys` as a plain member. It is a loop element.
`des` as an `ndes` element was already right.
* **`Game::SVSODerelict::Write` (0x004fc2b0) — the recovery is right.** `NDsn` count then a loop
of `(DsnID, Dwght)`; `NAsg` count then a loop of `(Eflt, Esys)`. Two fields per iteration in
each, confirmed by the 8-byte element strides.
`objects/streams.json` still carries the `dsys` misclassification. It was left alone rather than
special-cased into `serializers.py`'s loop heuristic: the conformance check aligns by tag and
disk primitive, so the engine's corrected shape and the uncorrected table still agree item for
item, and the correction is recorded here and in `shapes.h`. A future `serializers.py` improvement
should pick it up.
## 4. A third trap resolved: `Read` accepts tags `Write` never emits
`Game::SVSOSots::Read` (0x005a7a40) reads `NPCPlr` (int, first) and `hastraps` (bool) plus a
following `traps` object. `Write` emits none of them. They are read-only backward compatibility
for an older save format; a writer that omits them is correct, and ours does. Worth generalising:
**a tag in `Read` with no counterpart in `Write` is not a hole in the recovery.**
## 5. What the engine does with it
`sots-engine` `wip/wire` types the whole tree: `ScriptObjects` (= `Game::SVSOSots`), the four
scenario bodies, and the eight encounter bodies the saves exercise. The key is applied in both
directions — the reader `select()`s from the `xscn`/`EncID` it just read, the writer from the one
it is about to write — so a body round-trips as whatever it came in as.
The four ids with a factory entry but no occurrence in any save (**7 SystemKiller, 8 PuppetMaster,
14 Locust, 21 Ortgay**) are deliberately **not** modelled. Their serializers are recovered and
shapes for them would probably be right, but nothing could check them; they fall to a generic
`Node` and round-trip verbatim. Same for `ScnObj`, whose pointer is NULL in every save.
`SvSctOb` went from 147 / 156 / 156 / 126 opaque items to **0 on all four saves**, with the
byte-identical round trip preserved. All 18 new shapes bind clean against the generated wire
schema: 0 mismatches, 0 wire-only, 0 shape-only.
## 6. Addresses
`ghidra/addresses.d/lane-w.json` — 13 entries (both factories, the jump table, the get-or-create
helper, the seeder, `SVSOSots` Read/Write/vftable, the `ret 4` stub, the
`SVSOIndependentSystems` vftable, and the three `Write`s the nesting answers came from).
`gen_addresses.py` merges to 640 entries with no duplicate name.

213
findings/objects/wire-schema-closeout.md Normal file
View file

@ -0,0 +1,213 @@
# Closing the opaque sections, and correcting the Python oracle
Lane W, 2026-09-08. Host/static only — VM140 was held by lane U and the game was never run.
Continues `findings/objects/wire-schema-channel.md` (lane G).
## Headline: named coverage 97.1 % → 98.0 %
The honest metric is `CoverageArchive`'s split of items a field **names** from items a generic
`Node` merely **carries** — not round-trip success, which `ar.any` bodies achieve trivially by
copying bytes nobody understands. The byte-identical round trip held throughout.
| save | before | after | still opaque after |
|---|---|---|---|
| turn1-state | 97.1 % | **98.0 %** | `CD` 744, `RNG` 2, `Attrib` 2 |
| turn2-state | 97.2 % | **98.0 %** | `CD` 744, `RNG` 2, `Attrib` 2 |
| turn3-state | 97.2 % | **98.0 %** | `CD` 744, `RNG` 2, `Attrib` 2 |
| zuul-turn5-species5 | 97.6 % | **98.4 %** | `CD` 517, `RNG` 2, `Attrib` 2 |
Conformance: **74 shapes bound, 769 items matched, 0 MISMATCH** (was 56 / 657 / 0). The ratchet in
`test_save.cpp` moved 95.0 → 97.5. `ctest` 34/34; `tools/clean_room_check.sh` OK; `test_save`
still skips cleanly with `SOTS_SAVES_DIR` unset.
`CD` is now the *only* remaining region of any size, and §4 says why it cannot be closed here.
## 1. `SvSctOb` — closed, 147 items → 0
Its own note: `findings/objects/svsctob-variants.md`. Summary: the body is `Game::SVSOSots`, which
writes two variant lists dispatched by a preceding key (`xscn` for `xsc`, `EncID` for `EncObj`).
Neither map is on the wire; both were read out of the game — the ids from a 23-entry dword jump
table at 0x0052bf60 indexed by `EncID - 1`, the names from an exhaustive four-way `_stricmp` chain
at 0x005a7050. Eighteen shapes bound, all clean. The four factory ids no save exercises
(7 SystemKiller, 8 PuppetMaster, 14 Locust, 21 Ortgay) are deliberately left carried, not typed.
## 2. `DOpts` — closed, 94 items → 0. Certain, from the helper's own decorated name.
`Game::ShipDesignDef::Section`'s `DOpts` is `?$VectorHelper@VString@Mars@@@Mars` — a
`Mars::VectorHelper<Mars::String>`. The element type is not inferred from the bytes; it is in the
mangled helper type the writer instantiated. The engine's `read_elem`/`write_elem` grew a
`std::string` branch (the one thing lane G listed as missing) and `carr<std::string>` now works.
The same branch closed `Game::SVSOVonNeumann::trev`, also `VectorHelper<Mars::String>`.
Note the empty-string trap this had to survive: an empty string is four zero bytes, so an empty
`DOpts` element is byte-identical to the int 0. The new unit test writes
`{"OPT_Armour", "", "OPT_Shield"}` specifically to exercise it.
## 3. `spies2` — closed, 56 items → 0, but read this before believing the element layout
`Game::ServerSystem`'s `spies2` is `?$VectorHelper@H@Mars` — `Mars::VectorHelper<int>`. So it is a
framed array of NULL-named ints, and that is a **certainty about the type**, from the same
decorated-name evidence as `DOpts`, not a guess from the bytes.
What is **not** exercised is any element. The count is 0 in all 28 systems of all four saves, so
the 56 items closed here are 28 × (the frame + its `.` count) and **no `spies2` element value has
ever been observed on disk**. If a future save carries spies, the shape says they are plain ints;
that claim rests entirely on `VectorHelper<int>` and would be falsified by the first non-empty one.
Labelling it: the *type* is certain, the *behaviour* is a hypothesis.
`Game::SVSOSwarmQueen::SysMem` and `Game::SVSOVonNeumann::mts` are in the same position —
`?$VectorHelper@U?$StreamableEnum@I@Mars@@@Mars`, i.e. `VectorHelper<StreamableEnum<uint>>`, typed
as int arrays, count 0 everywhere.
## 4. `CD` — NOT closed, and here is the evidence that it cannot be, without a workload
`CD` is two different problems wearing one tag. `CDT` lists the ids in order and one `CD` frame
follows per id:
* **`Player.<id>.TurnCommands_v5`** → `Game::TurnCommands`. One block, 35 items on turn1.
* **`Player.<id>.AIAgent`** → `Game::StrategyAIAgent::Streamable`. Three blocks, ~236 items each,
and these are the bulk of the 744.
### 4a. `TurnCommands`: the recovered sequence provably does not align to a no-orders save
`Game::TurnCommands` is recovered complete at **44 items, every tag `"."`**. The turn1 block has
**35**. Lane G called it a no-orders snapshot. That is right, and it is now stronger than a
description — the two sequences **cannot be aligned at all**, not even as a subsequence:
* Positionally they diverge at **item 4**: the save's item 4 is 8 bytes, which for a 1-character
tag can only be a `bool` (4 + 1 + 1 → 8; an int is 4 + 1 + 4 → 12). The recovery says `i32`.
Item 16 is a plain 12-byte int in the save where the recovery says a frame.
* As a subsequence it fails on counting. The save is `int, bool, float, bool, bool, bool, bool,
bool` then **27 consecutive ints**. After the recovery's item 15 there are only **22** `i32`
slots (indices 17–20, 22–27, 29–31, 33–40, 42); the rest are frames, a `narr`, and bools.
27 > 22, so no order-preserving assignment exists.
So the writer takes branches that a no-orders turn does not, and with every tag `"."` there is no
name to bind against. **`TurnCommands` needs a save with issued orders** — which needs the VM.
Anything typed from here would be a guess dressed as a layout, so nothing was typed.
### 4b. `AIAgent`: statically tractable, deliberately not attempted
`Game::StrategyAIAgent::Streamable` is recovered at 36 items with **named** tags (`AIAttr`,
`AITurnPris`, `AISit`, `AIPlyHat`, `prs2`, `dsh`, `AIDNG`, `AISys`, `CmbR`, `apr`, …), so unlike
`TurnCommands` it *could* be bound by name. Two reasons it was left: the recovery is incomplete
(8 unresolved items) and roughly ten nested sub-objects would each need their own shape and their
own conformance binding. It is the single largest remaining block and a clean, self-contained job
for a following lane — but it is AI cache state, not orders, so it is not on the reimplementation
critical path the way 4a is.
### 4c. `RNG` (2) and `Attrib` (2)
`RNG` is correctly opaque — the raw MT19937 state block — and was left alone, as instructed.
`Attrib` is `StreamableHelper<Game::AttribMap>` and is an empty frame in every save; there is
nothing to type.
## 5. The four defects in `save_reader.py`, fixed openly
Lane G found four real defects in both readers, fixed the engine, and deliberately left the Python
oracle alone rather than quietly editing it mid-campaign. That judgement was right; this is the
follow-through. All four are now fixed in `verify/save-reader/save_reader.py`, each with a test,
and **all four are byte-neutral on all four saves**.
| # | defect | fix |
|---|---|---|
| 1 | `Game::SystemParams` field 1 is a **string**, not an int | `Planet`'s `p1` retyped |
| 2 | `ObservedTech`/`ObservedWeapon` `odet` is a **bool**, not an int | `Owep`/`Otch` retyped |
| 3 | `Game::SpeciesRatios` `nv` is a **count** | new `SpeciesRatios`/`CivilianRatios` shapes |
| 4 | `Game::ShipRecords` `srbd` is a **count** | new `ShipRecords` shape |
A wrinkle worth recording: for 3 and 4 the reader did not have the wrong layout, it had **no**
layout — `ShipRecs` and `civr` were both `A(..., "any")`. So the "defect present in both readers"
was, on the Python side, an absence rather than an error. Fixing it meant *adding* the typed shapes
with the count semantics spelled out, which is a bigger change than a retype and is why the
byte-neutrality evidence below matters more than usual.
### The byte-neutrality evidence
Every item's inflated offset is unchanged. Dumping all four saves before and after and comparing
the offset column:
```
turn1-state: offset sequence IDENTICAL (38,933 items)
turn2-state: offset sequence IDENTICAL (39,843 items)
turn3-state: offset sequence IDENTICAL (40,300 items)
zuul-turn5-species5: offset sequence IDENTICAL (35,771 items)
```
so no item boundary moved; only type labels changed. `state_checksum.py` still reports
**`coverage: PROVED`** on all four (591,376 / 603,360 / 609,080 / 532,752 bytes rebuilt == inflated,
unchanged), with `0 error, 0 warn`.
The subsystem digests for `/CreateParams` and `/Sim/players` do change, and **that is the point** —
they are digests of *typed values*, and two of the fixes change what a value is. The value-byte
totals account for the change exactly:
| save | Δ value bytes | `odet` items × 3 (int 4 B → bool 1 B) | `p1` items × 4 (int 4 B → "" 0 B) |
|---|---|---|---|
| turn1 | −292 | 60 × 3 = 180 | 28 × 4 = 112 |
| turn2 | −295 | 61 × 3 = 183 | 28 × 4 = 112 |
| turn3 | −295 | 61 × 3 = 183 | 28 × 4 = 112 |
| zuul | −238 | 42 × 3 = 126 | 28 × 4 = 112 |
Every row balances to the byte. Nothing else in the digest moved.
`--strict` exits 0 on all four saves. Tests went **36 → 48**, all passing.
### One finding that fell out of the fix
`srbd` is not merely inferred from the recovery — the saves **exercise** it. It takes the values
0, 1, 3 and 4 across the players of the four saves, and every non-zero count is followed by exactly
`srbd × 5` scalars (turn3 player 4: `srbd == 4`, then 20 items in four `srd/src/srb/srl/sri`
groups). Read as a field, those trailing scalars have no explanation at all; the new test asserts
exactly that, and the field reading raises. So defect 4 is behaviourally confirmed, not a
hypothesis. Defect 3 is confirmed the same way but more weakly: `nv` is only ever 0 or 1 in these
saves, so the count reading is confirmed for one pair and the multi-pair case is synthetic.
### The tests
`test_save_reader.py::WireSchemaDefectsTest`, 12 tests. Each defect gets the case the real saves
happen not to contain (a **named** system, a **bool-valued** `odet`, **two** species pairs, a
**non-empty** design-record section), plus a test showing *why* it stayed invisible — an empty
string and the int 0 are the same four bytes; a bool item and an int item are the same 12 bytes
only because `odet` is 4 characters long, and the same test shows a 3-character tag would not have
been forgiving.
One nuance the tests made explicit and that belongs on the record: in `save_reader.py` the generic
walker types `"."` items from its **own kind catalog**, not from the schema passed in. On the real
saves the catalog said int for `p1` and the old schema said int, so the two agreed with each other
and were both wrong — the failure mode there is a silent agreement, not the framing desync the
schema-driven engine would have suffered.
## 6. Artifacts
* `sots-engine` `wip/wire`: `src/mars/stream/shapes.h` (18 new SVSO shapes, `DOpts` →
`vector<string>`, `spies2` → `vector<int32_t>`, `SvSctOb` → `opt_obj<ScriptObjects>`),
`src/mars/stream/archive.h` (`std::string` element branch in `read_elem` / `write_elem` /
`SchemaBuilder::carr`), `tests/mars_stream/test_wire_schema.cpp` (+18 bindings),
`tests/mars_stream/test_stream.cpp` (+2 unit tests, no saves needed),
`tests/mars_stream/test_save.cpp` (ratchet 95.0 → 97.5).
* `sots-re`: `verify/save-reader/save_reader.py` + `test_save_reader.py`,
`ghidra/addresses.d/lane-w.json` (13 entries), `findings/objects/svsctob-variants.md`, this note.
* `include/generated/sots_stream_schema.h` was **not** regenerated: `tools/streams.py` and
`tools/gen_stream_schema.py` were re-run and the output is byte-identical to the committed header
apart from the provenance line, because nothing on the notes side changed the recovery.
* `ghidra/generated/sots_addresses.h` was **not** regenerated either. Lane W's addresses are in the
fragment; `gen_addresses.py` merges to 640 entries with no duplicate name (checked against a
scratch path). Regenerating in place would have swept lane U's in-flight `lane-u.json` into this
lane's branch, and the engine consumes none of these addresses.
## 7. Open items
1. **A save with issued turn commands** remains the blocker for `TurnCommands_v5` (§4a), and it is
now a proven blocker rather than a suspected one.
2. `Game::StrategyAIAgent::Streamable` (§4b) — the largest remaining block, bindable by name,
~10 nested shapes, no VM needed.
3. The four unexercised `EncID`s (7, 8, 14, 21) and `ScnObj` stay carried until a save or a
scenario game produces one.
4. `serializers.py` should reclassify `Game::SVSOCrowDefenders`'s `dsys` as a loop element of
`ndsys` (see `svsctob-variants.md` §3); `objects/streams.json` still carries the old reading and
the conformance check does not notice, because it aligns by tag and disk primitive.
5. `spies2`, `SysMem`, `mts`, `trev`, `Ojvs`, `aid`, `Attrib`, `comms.nmsg`, `AIEnf.Nas`,
`spymgr.nspy` and `spy2.rtc/evc/ttc` are 0 in every save. Their containers are typed; their
element behaviour is not verified by anything.

108
ghidra/addresses.d/lane-w.json Normal file
View file

@ -0,0 +1,108 @@
{
"entries": [
{
"name": "SVScriptObject_FactoryByEncID",
"addr": "0x0052bf00",
"convention": "cdecl",
"prototype": "Game::SVScriptObject* (int encID) // The EncObj factory. `dec eax; cmp eax,0x16; ja <null>; jmp dword [eax*4 + 0x0052bf60]` -- a 23-entry dword jump table indexed by encID-1. Live ids: 1 VonNeumann, 3 Swarm, 4 Derelict, 5 Monitor, 7 SystemKiller, 8 PuppetMaster, 9 SlaversRefuel, 10 SwarmQueen, 14 Locust, 17 CrowRuins, 20 Refugees, 21 Ortgay. Ids 2, 6, 11-13, 15, 16, 18, 19, 22, 23 and everything outside 1..23 return NULL. Class names read off the vftable store in each ctor",
"status": "verified",
"source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)"
},
{
"name": "SVScriptObject_EncIDJumpTable",
"addr": "0x0052bf60",
"convention": "data",
"prototype": "void* [23] // the jump table SVScriptObject_FactoryByEncID indexes with encID-1",
"status": "verified",
"source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)"
},
{
"name": "SVSOSots_GetOrCreateEncObj",
"addr": "0x005a4450",
"convention": "thiscall",
"prototype": "Game::SVScriptObject* (Game::SVSOSots* this, int encID) // scans the loaded (id, obj) pair vector at this+0x0C..0x10 first and only calls SVScriptObject_FactoryByEncID on a miss",
"status": "verified",
"source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)"
},
{
"name": "SVScriptObject_FactoryByScenarioName",
"addr": "0x005a7050",
"convention": "cdecl",
"prototype": "Game::SVScriptObject* (const char* xscn) // the xsc factory: a flat four-way _stricmp chain, EXHAUSTIVE. \"traps\" -> Game::SVSOTraps (ctor 0x0052cbb0), \"crowdefs\" -> Game::SVSOCrowDefenders (0x0052b3c0), \"indsys\" -> Game::SVSOIndependentSystems (0x0075b530), \"gmtrigger\" -> Game::SVSOGrandMenaceTrigger (0x004f5630). Falls through with \"Error creating extra script %s.\"",
"status": "verified",
"source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)"
},
{
"name": "SVSOSots_SeedNewGameScripts",
"addr": "0x005a7d70",
"convention": "thiscall",
"prototype": "void (Game::SVSOSots* this) // registers exactly traps, crowdefs, indsys and -- only when ScnObj == NULL -- gmtrigger. Corroborates that the four scenario names are the whole set",
"status": "verified",
"source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)"
},
{
"name": "SVSOSots_Write",
"addr": "0x0059ddf0",
"convention": "thiscall",
"prototype": "void (Game::SVSOSots* this, Mars::IStream* s) // slot 2 of vftable 0x00A063C4. Emits ScnID, ScnObj (only when non-NULL), numx x (xscn, xsc), NEncObjs x (EncID, EncObj)",
"status": "verified",
"source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)"
},
{
"name": "SVSOSots_Read",
"addr": "0x005a7a40",
"convention": "thiscall",
"prototype": "void (Game::SVSOSots* this, Mars::IStream* s) // slot 1. Accepts two tags Write never emits -- NPCPlr (int, first) and hastraps (bool) with a following traps object -- read-only backward compatibility; a writer that omits them is correct",
"status": "verified",
"source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)"
},
{
"name": "SVSOSots_vftable",
"addr": "0x00a063c4",
"convention": "data",
"prototype": "Game::SVSOSots vftable (slot0 dtor, slot1 Read 0x005a7a40, slot2 Write 0x0059ddf0)",
"status": "verified",
"source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)"
},
{
"name": "SVSOIndependentSystems_vftable",
"addr": "0x00a20314",
"convention": "data",
"prototype": "Game::SVSOIndependentSystems vftable. slot0 dtor 0x0075afb0; slot1 Read AND slot2 Write are both 0x005f8ac0, the shared `ret 4` no-op stub -- so the \"indsys\" frame is genuinely empty on disk and the class's 0x1c4-byte body is runtime-only state",
"status": "verified",
"source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)"
},
{
"name": "Streamable_NoOpStub",
"addr": "0x005f8ac0",
"convention": "thiscall",
"prototype": "void (void*, Mars::IStream*) // `C2 04 00` -- a bare RET 4 shared as the inherited Read/Write for classes that serialize nothing",
"status": "verified",
"source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)"
},
{
"name": "SVSOCrowDefenders_Write",
"addr": "0x004f8c90",
"convention": "thiscall",
"prototype": "void (Game::SVSOCrowDefenders* this, Mars::IStream* s) // sys; ndsys count then a loop writing dsys; ndes count then a loop writing des; drad. NOTE: `dsys` is INSIDE the ndsys loop -- objects/layouts.json records it as a plain member, which is wrong, and no save can settle it because both counts are 0 everywhere",
"status": "verified",
"source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)"
},
{
"name": "SVSOMonitor_Write",
"addr": "0x004fd810",
"convention": "thiscall",
"prototype": "void (Game::SVSOMonitor* this, Mars::IStream* s) // calls SVSODerelict::Write (0x004fc2b0) as its first act -- Monitor derives from Derelict, which is why its tag run starts NDsn/DsnID/Dwght + NAsg/Eflt/Esys before nt/scnm/spwt/rsmd/dsgn",
"status": "verified",
"source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)"
},
{
"name": "SVSODerelict_Write",
"addr": "0x004fc2b0",
"convention": "thiscall",
"prototype": "void (Game::SVSODerelict* this, Mars::IStream* s) // NDsn count then a loop of (DsnID, Dwght); NAsg count then a loop of (Eflt, Esys). Two fields per iteration in each, confirmed by the 8-byte element strides",
"status": "verified",
"source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)"
}
]
}

41
verify/save-reader/save_reader.py
View file

@ -367,7 +367,12 @@ Summary = Shape("Summary", [
# -- CreateParams = StrategyGameCreateParams::Write @0x0082ae40 (§5.1) ----------
Planet = Shape(None, [ # SystemParams element, all tags "."
R("pos", "vec3"), R("p1", "int"), R("p2", "int"), R("p3", "int"), R("p4", "float"), Rest(),
# p1 is a STRING, not an int (Game::SystemParams wire schema: frame, str, i32,
# i32, f32 -- lane G's wire-schema conformance check, findings/objects/
# wire-schema-channel.md §3.1). It is the empty string in every save we hold,
# and an empty string is four zero bytes -- byte-identical to the int 0 -- so
# this round-tripped by luck. A named system would desynchronise the reader.
R("pos", "vec3"), R("p1", "string"), R("p2", "int"), R("p3", "int"), R("p4", "float"), Rest(),
])
MapP = Shape("MapP", [ # StarMapParams::Write @0x00727a10, all tags "."
R("mapType", "int"),
@ -483,9 +488,35 @@ Prep = Shape(None, [
# is `otnF` on disk (R1 had `ontF`; positional matching hid the typo -- caught by
# the C++ round-trip writer and confirmed in the bytes, SAVE_FORMAT.md §10).
Odes = Shape(None, [A("otnF", "int"), A("otnL", "int"), A("odid", "int"), A("opid", "int"), Rest()])
Owep = Shape(None, [A("otnF", "int"), A("otnL", "int"), A("odet", "int"), A("owep", "string"), A("owith", "int"), Rest()])
Otch = Shape(None, [A("otnF", "int"), A("otnL", "int"), A("odet", "int"), A("otch", "string"), A("owith", "int"), Rest()])
# `odet` is a BOOL, not an int: Game::ObservedWeapon / Game::ObservedTech call the
# bool writer (wire-schema-channel.md §3.2). It is byte-safe here only because the
# tag is 4 characters, which makes a bool item and an int item both 12 bytes; a 3-
# or 5-character tag would not have been so forgiving.
Owep = Shape(None, [A("otnF", "int"), A("otnL", "int"), A("odet", "bool"), A("owep", "string"), A("owith", "int"), Rest()])
Otch = Shape(None, [A("otnF", "int"), A("otnL", "int"), A("odet", "bool"), A("otch", "string"), A("owith", "int"), Rest()])
Note = Shape("Nts", [A("NtSys", "int"), A("NtTxt", "string"), A("NtTrn", "int"), Rest()])
# Game::ShipRecords. TWO COUNTED SECTIONS, and `srbd` is the second COUNT, not a
# field (wire-schema-channel.md §3.4). The recovery lists srd/src/srb/srl/sri
# right after `srbd` as loop-body writes, and the saves EXERCISE it: srbd takes the
# values 0, 1, 3 and 4 across the players of the four saves we hold, and every
# non-zero count is followed by exactly srbd x 5 scalars (e.g. turn3 player 4 has
# srbd == 4 and 20 items). So this one is behaviourally confirmed, not inferred.
DesignRecord = Seq([
A("srd", "int"), A("src", "int"), A("srb", "int"), A("srl", "int"), A("sri", "int"),
])
ShipRecords = Shape("ShipRecs", [
A("srnc", NArr(Seq([A("srb", "int"), A("srl", "int"), A("srk", "int"), A("sri", "int")])),
key="records"),
A("srbd", NArr(DesignRecord), key="designRecords"),
Rest(),
])
# Game::SpeciesRatios. `nv` is the COUNT of (sp, va2) pairs, not a field
# (wire-schema-channel.md §3.3): the recovery marks sp and va2 as loop-body writes
# and the saves agree -- nv==1 frames carry one pair, nv==0 frames carry nothing.
SpeciesRatios = Shape("spe", [
A("nv", NArr(Seq([A("sp", "int"), A("va2", "int")])), key="ratios"), Rest(),
])
CivilianRatios = Shape("civr", [A("smx", "float"), A("spe", SpeciesRatios), Rest()])
# Design header: on-disk tags are FAIDes/DHide/DWep/DName (R1's camel-case names
# were wrong). The typed-dict keys keep R1's spelling because
# verify/design-rules/stock_designs.py consumes them.
@ -521,9 +552,9 @@ Player = Shape("Player", [
A("CnTrd", "bool"), A("CnRad", "bool"), A("hgs", "bool"), A("hadvs", "bool"), A("harcc", "bool"),
A("CnVItl", "bool"), A("pddm", "float"),
A("BnkWrn", "int"), A("BnkTrn", "int"), A("BnkPr", "int"), A("BnkEl", "int"),
A("ShipRecs", "any"),
A("ShipRecs", ShipRecords, key="shipRecs"),
A("NextPrjID", "int"), A("plcy", "int"), A("pswd", "string"), A("lret", "int"), A("nmeid", "int"),
A("cdp", "bool"), A("spy2", "any"), A("civr", "any"), A("aidf", "int"),
A("cdp", "bool"), A("spy2", "any"), A("civr", CivilianRatios), A("aidf", "int"),
A("Srn", "bool"), A("SrnTo", "int"), A("lboid", "int"), Opt("lcid", "int"), A("lcid2", "int"),
A("ResTNm", "string"), A("ResErrRoll", "bool"),
R("conMods", ConMods, flex=True),

207
verify/save-reader/test_save_reader.py
View file

@ -573,6 +573,213 @@ class TagNameCorrectionsTest(unittest.TestCase):
self.assertEqual((plans[0]["wpts"][0]["Wpt"], plans[0]["wpts"][0]["nrt"]["nrp"]), (272, -1))
class WireSchemaDefectsTest(unittest.TestCase):
"""The four defects lane G's SchemaProbe found in BOTH readers, and which no
round-trip test could catch (findings/objects/wire-schema-channel.md §3).
Each `test_*_would_have_caught_it` builds the case the real saves happen not
to contain. Each `test_*_is_byte_neutral_here` shows why the defect stayed
invisible: on the data we actually hold the wrong type occupies the same
bytes, so the round trip was clean and wrong at the same time.
"""
REAL = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "results", "saves")
# --- 1. Game::SystemParams field 1 is a string, not an int ------------------
@staticmethod
def system_params(name):
"""One MapP frame with a single SystemParams element carrying `name`."""
w = sw.SaveWriter()
w.begin("MapP")
w.int(".", 0) # mapType
w.begin(".") # VectorHelper<SystemParams>
w.int(".", 1)
w.begin(".") # the element, all tags "."
w.vec3(".", 1.0, 2.0, 3.0)
w.string(".", name) # <- the field in question
w.int(".", 7)
w.int(".", 8)
w.float(".", 0.5)
w.end()
w.end()
w.int(".", 0) # players: bare count, no groups
w.begin("."); w.int(".", 0); w.end() # nodePaths: framed count
w.end()
return w.bytes()
def test_system_params_p1_is_a_string(self):
schema = sr.Seq([sr.A("MapP", sr.MapP)])
res = sr.read_bytes(self.system_params(""), padding="joint", strict=True, schema=schema)
planet = res.typed["MapP"]["planets"][0]
self.assertEqual(planet["p1"], "")
self.assertIsInstance(planet["p1"], str)
def test_a_named_system_reads_back_intact(self):
"""The case no save we hold contains. A non-empty name is a string of a
length the int reading cannot express, and every following field still has
to land -- this is the read the campaign had no answer for."""
data = self.system_params("Beta Hydri")
res = sr.read_bytes(data, padding="joint", strict=True, schema=sr.Seq([sr.A("MapP", sr.MapP)]))
planet = res.typed["MapP"]["planets"][0]
self.assertEqual((planet["p1"], planet["p2"], planet["p3"], planet["p4"]),
("Beta Hydri", 7, 8, 0.5))
def test_the_old_int_type_disagrees_with_these_bytes(self):
"""The catching assertion: with `p1` typed int, applying the schema to a
SystemParams element is a hard error, named or not. (The reader survived
the real saves only because its own kind catalog also said int there, so
schema and walker agreed with each other and both were wrong.)"""
old = sr.Shape(None, [sr.R("pos", "vec3"), sr.R("p1", "int"), sr.R("p2", "int"),
sr.R("p3", "int"), sr.R("p4", "float"), sr.Rest()])
old_map = sr.Shape("MapP", [sr.R("mapType", "int"), sr.R("planets", sr.CArr(old)),
sr.R("players", sr.NArr(sr.CArr("int"))),
sr.R("nodePaths", sr.CArr("any")), sr.Rest()])
for name in ("", "Beta Hydri"):
with self.assertRaises(sr.SaveFormatError):
sr.read_bytes(self.system_params(name), padding="joint", strict=True,
schema=sr.Seq([sr.A("MapP", old_map)]))
def test_empty_name_is_why_it_stayed_hidden(self):
"""An empty string is four zero bytes -- byte-identical to the int 0 -- so
on every save we hold, the wrong type cost nothing."""
s = sw.SaveWriter(); s.string(".", "")
i = sw.SaveWriter(); i.int(".", 0)
self.assertEqual(s.bytes(), i.bytes())
# only the empty one is: a string payload is [int32 len][bytes], so a name
# is 4 + len bytes where the int is 4, and the item stops agreeing as soon
# as the name pushes it past the next multiple of four.
named = sw.SaveWriter(); named.string(".", "Beta Hydri")
self.assertNotEqual(len(named.bytes()), len(i.bytes()))
# --- 2. ObservedTech / ObservedWeapon `odet` is a bool, not an int ----------
def test_odet_is_declared_bool(self):
for shape in (sr.Owep, sr.Otch):
kinds = {f.name: f.type for f in shape.fields if isinstance(f, sr.Field)}
self.assertEqual(kinds["odet"], "bool")
self.assertEqual(sr.GLOBAL_KINDS["odet"], "bool")
def test_odet_reads_back_as_a_bool_not_an_int(self):
w = sw.SaveWriter()
w.begin("otch"); w.int(".", 1)
w.begin("."); w.int("otnF", 1); w.int("otnL", 1); w.bool("odet", True)
w.string("otch", "WEP_RedLas"); w.int("owith", 1); w.end()
w.end()
res = sr.read_bytes(w.bytes(), padding="joint", strict=True,
schema=sr.Seq([sr.A("otch", sr.CArr(sr.Otch))]))
v = res.typed["otch"][0]["odet"]
self.assertIsInstance(v, bool) # `assertEqual(v, 1)` passes either way
self.assertIs(v, True)
def test_a_four_char_tag_is_the_only_reason_odet_was_byte_safe(self):
"""bool and int items are the same size ONLY when the tag length makes the
joint padding agree. 4 chars: 4+4+1 -> 12 and 4+4+4 = 12. 3 chars:
4+3+1 -> 8 but 4+3+4 -> 12, and the parse desyncs."""
four = sw.SaveWriter(); four.bool("odet", True); four.int("owith", 1)
as_int = sw.SaveWriter(); as_int.int("odet", 1); as_int.int("owith", 1)
self.assertEqual(len(four.bytes()), len(as_int.bytes())) # the coincidence
three = sw.SaveWriter(); three.bool("det", True); three.int("owith", 1)
three_int = sw.SaveWriter(); three_int.int("det", 1); three_int.int("owith", 1)
self.assertNotEqual(len(three.bytes()), len(three_int.bytes())) # not a property
# --- 3. Game::SpeciesRatios `nv` is a count -------------------------------
@staticmethod
def civr(pairs):
w = sw.SaveWriter()
w.begin("civr"); w.float("smx", 0.5)
w.begin("spe"); w.int("nv", len(pairs))
for sp, va2 in pairs:
w.int("sp", sp); w.int("va2", va2)
w.end(); w.end()
return w.bytes()
def test_nv_is_a_count_not_a_field(self):
schema = sr.Seq([sr.A("civr", sr.CivilianRatios)])
res = sr.read_bytes(self.civr([(0, 100)]), padding="joint", strict=True, schema=schema)
self.assertEqual(norm(res.typed["civr"]["spe"]["ratios"]), [{"sp": 0, "va2": 100}])
res0 = sr.read_bytes(self.civr([]), padding="joint", strict=True, schema=schema)
self.assertEqual(res0.typed["civr"]["spe"]["ratios"], [])
def test_two_species_would_have_broken_the_field_reading(self):
"""nv is 0 or 1 in every save we hold, so a field reading survives. With
two pairs the second is unexplained and strict parsing fails."""
data = self.civr([(0, 60), (5, 40)])
res = sr.read_bytes(data, padding="joint", strict=True,
schema=sr.Seq([sr.A("civr", sr.CivilianRatios)]))
self.assertEqual(norm(res.typed["civr"]["spe"]["ratios"]),
[{"sp": 0, "va2": 60}, {"sp": 5, "va2": 40}])
wrong = sr.Shape("spe", [sr.A("nv", "int"), sr.A("sp", "int"), sr.A("va2", "int")])
wrong_civr = sr.Shape("civr", [sr.A("smx", "float"), sr.A("spe", wrong)])
with self.assertRaises(sr.SaveFormatError):
sr.read_bytes(data, padding="joint", strict=True,
schema=sr.Seq([sr.A("civr", wrong_civr)]))
# --- 4. Game::ShipRecords `srbd` is a count -------------------------------
@staticmethod
def ship_recs(recs, designs):
w = sw.SaveWriter()
w.begin("ShipRecs")
w.int("srnc", len(recs))
for srb, srl, srk, sri in recs:
w.int("srb", srb); w.int("srl", srl); w.int("srk", srk); w.int("sri", sri)
w.int("srbd", len(designs))
for srd, src, srb, srl, sri in designs:
w.int("srd", srd); w.int("src", src); w.int("srb", srb)
w.int("srl", srl); w.int("sri", sri)
w.end()
return w.bytes()
def test_srbd_is_a_count_not_a_field(self):
data = self.ship_recs([(0, 0, 0, 0)], [(18, 0, 2, 0, 2)])
res = sr.read_bytes(data, padding="joint", strict=True,
schema=sr.Seq([sr.A("ShipRecs", sr.ShipRecords)]))
sr_ = res.typed["ShipRecs"]
self.assertEqual(norm(sr_["records"]), [{"srb": 0, "srl": 0, "srk": 0, "sri": 0}])
self.assertEqual(norm(sr_["designRecords"]),
[{"srd": 18, "src": 0, "srb": 2, "srl": 0, "sri": 2}])
def test_srbd_as_a_field_cannot_explain_the_trailing_records(self):
data = self.ship_recs([(0, 0, 0, 0)], [(18, 0, 2, 0, 2)])
wrong = sr.Shape("ShipRecs", [
sr.A("srnc", sr.NArr(sr.Seq([sr.A("srb", "int"), sr.A("srl", "int"),
sr.A("srk", "int"), sr.A("sri", "int")]))),
sr.A("srbd", "int"),
])
with self.assertRaises(sr.SaveFormatError):
sr.read_bytes(data, padding="joint", strict=True,
schema=sr.Seq([sr.A("ShipRecs", wrong)]))
# --- the four corrections against the real saves ---------------------------
@unittest.skipUnless(os.path.exists(os.path.join(REAL, "turn3-state.sav")), "real saves not present")
def test_real_saves_agree_with_the_corrected_types(self):
res = sr.read_save(os.path.join(self.REAL, "turn3-state.sav"), padding="joint", strict=True)
# 1. every SystemParams name is the empty STRING (never the int 0)
planets = res.typed["createParams"]["MapP"]["planets"]
self.assertTrue(planets)
for p in planets:
self.assertIsInstance(p["p1"], str)
self.assertEqual(p["p1"], "")
players = [p["Player"] for p in res.typed["sim"]["players"]]
# 2. every odet is a bool
odets = [o["odet"] for pl in players for key in ("otch", "owep") for o in pl.get(key, [])]
self.assertTrue(odets)
for v in odets:
self.assertIsInstance(v, bool)
# 3. nv counts (sp, va2) pairs: 1 -> one pair, 0 -> none
seen = set()
for pl in players:
ratios = pl["civr"]["spe"]["ratios"]
seen.add(len(ratios))
for r in ratios:
self.assertEqual(set(r) - {"_off"}, {"sp", "va2"})
self.assertEqual(seen, {0, 1})
# 4. srbd counts 5-scalar design records; turn3 exercises 0, 1 and 4
counts = sorted({len(pl["shipRecs"]["designRecords"]) for pl in players})
self.assertEqual(counts, [0, 1, 4])
for pl in players:
for d in pl["shipRecs"]["designRecords"]:
self.assertEqual(set(d) - {"_off"}, {"srd", "src", "srb", "srl", "sri"})
class CliTest(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()