diff --git a/findings/objects/svsctob-variants.md b/findings/objects/svsctob-variants.md new file mode 100644 index 0000000..4205e92 --- /dev/null +++ b/findings/objects/svsctob-variants.md @@ -0,0 +1,147 @@ +# `SvSctOb`: the script-object tree, and the two factories that key it + +Lane W, 2026-09-08. Host/static only — VM140 was held by lane U and the game was never run. + +`Game::StrategyServer` writes one member tagged `SvSctOb`, typed +`Mars::StreamableHelper` — a **polymorphic pointer**. In every save we +hold it points at a `Game::SVSOSots`, whose serializer (`Write` 0x0059ddf0, `Read` 0x005a7a40, +vftable 0x00A063C4) emits, in disk order: + +``` +ScnID int +ScnObj StreamableHelper -- conditional; NULL, so absent, in all four saves +numx int count + x numx { xscn string -- a scenario NAME + xsc StreamableHelper } +NEncObjs int count + x NEncObjs { EncID int -- an encounter ID + EncObj StreamableHelper } +``` + +Both `xsc` and `EncObj` are polymorphic and **dispatched by the key item that immediately +precedes them**. Neither map is anywhere on the wire: a reader that does not know them can only +carry the bodies as opaque blobs, which is exactly what `sots-engine` was doing (147 items on +turn1). Both maps were read out of the game. + +## 1. `EncID` → class: a 23-entry jump table + +`Game::SVSOSots::Read` calls a get-or-create helper at **0x005a4450** (it scans the already +loaded `(id, obj)` pair vector at `this+0x0C..0x10` and only on a miss calls the real factory). +The factory is **0x0052bf00**: + +``` +0052bf00 push ebp; mov ebp,esp; mov eax,[ebp+8] +0052bf06 dec eax +0052bf07 cmp eax,0x16 +0052bf0a ja 0052bf5b ; -> xor eax,eax; ret (NULL) +0052bf0c jmp dword ptr [eax*4 + 0x0052bf60] +``` + +so the encoding is a **dword jump table at 0x0052bf60, 23 entries, indexed by `EncID - 1`**. +Each thunk runs a ctor, and the class name comes from the `mov dword ptr [obj], ` store +in that ctor — a demangled-RTTI name, not a guess. + +| EncID | ctor | class | in our saves | +|---|---|---|---| +| 1 | 0x0052bb90 → 0x0052b790 | `Game::SVSOVonNeumann` | yes | +| 3 | 0x0051aba0 → 0x0051a870 | `Game::SVSOSwarm` | yes | +| 4 | 0x0051a270 | `Game::SVSODerelict` | yes | +| 5 | 0x0052ae10 | `Game::SVSOMonitor` | yes | +| 7 | 0x0050e0f0 | `Game::SVSOSystemKiller` | no | +| 8 | 0x0050de50 | `Game::SVSOPuppetMaster` | no | +| 9 | 0x0051a820 | `Game::SVSOSlaversRefuel` | yes | +| 10 | 0x0051ae20 | `Game::SVSOSwarmQueen` | yes | +| 14 | 0x0052a030 → 0x00529080 | `Game::SVSOLocust` | no | +| 17 | 0x00524190 | `Game::SVSOCrowRuins` | yes | +| 20 | 0x005295a0 → 0x005293f0 | `Game::SVSORefugees` | yes | +| 21 | 0x004f5210 → 0x004f5170 | `Game::SVSOOrtgay` | no | + +Ids 2, 6, 11–13, 15, 16, 18, 19, 22, 23 are dead slots that return NULL; so is anything outside +1..23 (the `ja`). **There is no named enum** — the ids are bare integers and the jump table is the +only encoding of the map anywhere in the binary. + +The eight ids the saves carry were independently identified by tag-sequence match against the +recovered serializers *before* the table was read, and the two agree exactly on all eight. + +## 2. `xscn` → class: a four-way `_stricmp` chain + +`0x005a7050`, falling through to `"Error creating extra script %s.\n"`. Flat if-chain, no table, +no registration list, and **exhaustive** — there are four scenario names in the whole game. + +| `xscn` | test at | ctor | class | +|---|---|---|---| +| `traps` | 0x005a7085 | 0x0052cbb0 → 0x0052c940 | `Game::SVSOTraps` | +| `crowdefs` | 0x005a70a4 | 0x0052b3c0 | `Game::SVSOCrowDefenders` | +| `indsys` | 0x005a70be | 0x0075b530 | `Game::SVSOIndependentSystems` | +| `gmtrigger` | 0x005a70d8 | 0x004f5630 | `Game::SVSOGrandMenaceTrigger` | + +Corroborated by the new-game seeder at 0x005a7d70, which registers exactly `traps`, `crowdefs`, +`indsys`, and — only when `ScnObj == NULL` — `gmtrigger`. That is the four-entry list our saves +carry, in that order. + +### `indsys` serializes nothing, and that is a fact, not an absence of evidence + +`Game::SVSOIndependentSystems` has no entry in `objects/layouts.json`, which on its own only says +the recovery never found a serializer. Its vftable is **0x00A20314** (from the `C7 06 14 03 A2 00` +store at 0x0075b585). Comparing slots against `Game::SVSOCrowDefenders`'s vftable at 0x009F5904 +(slot0 dtor, slot1 `Read`, slot2 `Write` — both of which match `layouts.json`): + +``` +slot0 dtor = 0x0075afb0 +slot1 Read = 0x005f8ac0 +slot2 Write = 0x005f8ac0 +``` + +and 0x005f8ac0 is `C2 04 00` — a bare `ret 4` stub shared across the binary. So the class uses the +inherited no-op for both directions and the `indsys` frame is **genuinely zero-length on disk**. +Its 0x1c4-byte body (two 7-element arrays of 0x20-byte records) is runtime state rebuilt each +session. The empty frame in the saves is the correct output, not a truncation. + +## 3. Two loop-nesting questions the bytes could not answer + +Both counts involved are 0 in every save, so only the binary settles them. + +* **`Game::SVSOCrowDefenders::Write` (0x004f8c90) — the recovery is wrong about `dsys`.** Tag + strings at .rdata 0x009f0270. Emission order: + `sys`; `ndsys` count then a **loop writing `dsys`**; `ndes` count then a loop writing `des`; + `drad`. `objects/layouts.json` records `dsys` as a plain member. It is a loop element. + `des` as an `ndes` element was already right. +* **`Game::SVSODerelict::Write` (0x004fc2b0) — the recovery is right.** `NDsn` count then a loop + of `(DsnID, Dwght)`; `NAsg` count then a loop of `(Eflt, Esys)`. Two fields per iteration in + each, confirmed by the 8-byte element strides. + +`objects/streams.json` still carries the `dsys` misclassification. It was left alone rather than +special-cased into `serializers.py`'s loop heuristic: the conformance check aligns by tag and +disk primitive, so the engine's corrected shape and the uncorrected table still agree item for +item, and the correction is recorded here and in `shapes.h`. A future `serializers.py` improvement +should pick it up. + +## 4. A third trap resolved: `Read` accepts tags `Write` never emits + +`Game::SVSOSots::Read` (0x005a7a40) reads `NPCPlr` (int, first) and `hastraps` (bool) plus a +following `traps` object. `Write` emits none of them. They are read-only backward compatibility +for an older save format; a writer that omits them is correct, and ours does. Worth generalising: +**a tag in `Read` with no counterpart in `Write` is not a hole in the recovery.** + +## 5. What the engine does with it + +`sots-engine` `wip/wire` types the whole tree: `ScriptObjects` (= `Game::SVSOSots`), the four +scenario bodies, and the eight encounter bodies the saves exercise. The key is applied in both +directions — the reader `select()`s from the `xscn`/`EncID` it just read, the writer from the one +it is about to write — so a body round-trips as whatever it came in as. + +The four ids with a factory entry but no occurrence in any save (**7 SystemKiller, 8 PuppetMaster, +14 Locust, 21 Ortgay**) are deliberately **not** modelled. Their serializers are recovered and +shapes for them would probably be right, but nothing could check them; they fall to a generic +`Node` and round-trip verbatim. Same for `ScnObj`, whose pointer is NULL in every save. + +`SvSctOb` went from 147 / 156 / 156 / 126 opaque items to **0 on all four saves**, with the +byte-identical round trip preserved. All 18 new shapes bind clean against the generated wire +schema: 0 mismatches, 0 wire-only, 0 shape-only. + +## 6. Addresses + +`ghidra/addresses.d/lane-w.json` — 13 entries (both factories, the jump table, the get-or-create +helper, the seeder, `SVSOSots` Read/Write/vftable, the `ret 4` stub, the +`SVSOIndependentSystems` vftable, and the three `Write`s the nesting answers came from). +`gen_addresses.py` merges to 640 entries with no duplicate name. diff --git a/findings/objects/wire-schema-closeout.md b/findings/objects/wire-schema-closeout.md new file mode 100644 index 0000000..76ec355 --- /dev/null +++ b/findings/objects/wire-schema-closeout.md @@ -0,0 +1,213 @@ +# Closing the opaque sections, and correcting the Python oracle + +Lane W, 2026-09-08. Host/static only — VM140 was held by lane U and the game was never run. +Continues `findings/objects/wire-schema-channel.md` (lane G). + +## Headline: named coverage 97.1 % → 98.0 % + +The honest metric is `CoverageArchive`'s split of items a field **names** from items a generic +`Node` merely **carries** — not round-trip success, which `ar.any` bodies achieve trivially by +copying bytes nobody understands. The byte-identical round trip held throughout. + +| save | before | after | still opaque after | +|---|---|---|---| +| turn1-state | 97.1 % | **98.0 %** | `CD` 744, `RNG` 2, `Attrib` 2 | +| turn2-state | 97.2 % | **98.0 %** | `CD` 744, `RNG` 2, `Attrib` 2 | +| turn3-state | 97.2 % | **98.0 %** | `CD` 744, `RNG` 2, `Attrib` 2 | +| zuul-turn5-species5 | 97.6 % | **98.4 %** | `CD` 517, `RNG` 2, `Attrib` 2 | + +Conformance: **74 shapes bound, 769 items matched, 0 MISMATCH** (was 56 / 657 / 0). The ratchet in +`test_save.cpp` moved 95.0 → 97.5. `ctest` 34/34; `tools/clean_room_check.sh` OK; `test_save` +still skips cleanly with `SOTS_SAVES_DIR` unset. + +`CD` is now the *only* remaining region of any size, and §4 says why it cannot be closed here. + +## 1. `SvSctOb` — closed, 147 items → 0 + +Its own note: `findings/objects/svsctob-variants.md`. Summary: the body is `Game::SVSOSots`, which +writes two variant lists dispatched by a preceding key (`xscn` for `xsc`, `EncID` for `EncObj`). +Neither map is on the wire; both were read out of the game — the ids from a 23-entry dword jump +table at 0x0052bf60 indexed by `EncID - 1`, the names from an exhaustive four-way `_stricmp` chain +at 0x005a7050. Eighteen shapes bound, all clean. The four factory ids no save exercises +(7 SystemKiller, 8 PuppetMaster, 14 Locust, 21 Ortgay) are deliberately left carried, not typed. + +## 2. `DOpts` — closed, 94 items → 0. Certain, from the helper's own decorated name. + +`Game::ShipDesignDef::Section`'s `DOpts` is `?$VectorHelper@VString@Mars@@@Mars` — a +`Mars::VectorHelper`. The element type is not inferred from the bytes; it is in the +mangled helper type the writer instantiated. The engine's `read_elem`/`write_elem` grew a +`std::string` branch (the one thing lane G listed as missing) and `carr` now works. + +The same branch closed `Game::SVSOVonNeumann::trev`, also `VectorHelper`. + +Note the empty-string trap this had to survive: an empty string is four zero bytes, so an empty +`DOpts` element is byte-identical to the int 0. The new unit test writes +`{"OPT_Armour", "", "OPT_Shield"}` specifically to exercise it. + +## 3. `spies2` — closed, 56 items → 0, but read this before believing the element layout + +`Game::ServerSystem`'s `spies2` is `?$VectorHelper@H@Mars` — `Mars::VectorHelper`. So it is a +framed array of NULL-named ints, and that is a **certainty about the type**, from the same +decorated-name evidence as `DOpts`, not a guess from the bytes. + +What is **not** exercised is any element. The count is 0 in all 28 systems of all four saves, so +the 56 items closed here are 28 × (the frame + its `.` count) and **no `spies2` element value has +ever been observed on disk**. If a future save carries spies, the shape says they are plain ints; +that claim rests entirely on `VectorHelper` and would be falsified by the first non-empty one. +Labelling it: the *type* is certain, the *behaviour* is a hypothesis. + +`Game::SVSOSwarmQueen::SysMem` and `Game::SVSOVonNeumann::mts` are in the same position — +`?$VectorHelper@U?$StreamableEnum@I@Mars@@@Mars`, i.e. `VectorHelper>`, typed +as int arrays, count 0 everywhere. + +## 4. `CD` — NOT closed, and here is the evidence that it cannot be, without a workload + +`CD` is two different problems wearing one tag. `CDT` lists the ids in order and one `CD` frame +follows per id: + +* **`Player..TurnCommands_v5`** → `Game::TurnCommands`. One block, 35 items on turn1. +* **`Player..AIAgent`** → `Game::StrategyAIAgent::Streamable`. Three blocks, ~236 items each, + and these are the bulk of the 744. + +### 4a. `TurnCommands`: the recovered sequence provably does not align to a no-orders save + +`Game::TurnCommands` is recovered complete at **44 items, every tag `"."`**. The turn1 block has +**35**. Lane G called it a no-orders snapshot. That is right, and it is now stronger than a +description — the two sequences **cannot be aligned at all**, not even as a subsequence: + +* Positionally they diverge at **item 4**: the save's item 4 is 8 bytes, which for a 1-character + tag can only be a `bool` (4 + 1 + 1 → 8; an int is 4 + 1 + 4 → 12). The recovery says `i32`. + Item 16 is a plain 12-byte int in the save where the recovery says a frame. +* As a subsequence it fails on counting. The save is `int, bool, float, bool, bool, bool, bool, + bool` then **27 consecutive ints**. After the recovery's item 15 there are only **22** `i32` + slots (indices 17–20, 22–27, 29–31, 33–40, 42); the rest are frames, a `narr`, and bools. + 27 > 22, so no order-preserving assignment exists. + +So the writer takes branches that a no-orders turn does not, and with every tag `"."` there is no +name to bind against. **`TurnCommands` needs a save with issued orders** — which needs the VM. +Anything typed from here would be a guess dressed as a layout, so nothing was typed. + +### 4b. `AIAgent`: statically tractable, deliberately not attempted + +`Game::StrategyAIAgent::Streamable` is recovered at 36 items with **named** tags (`AIAttr`, +`AITurnPris`, `AISit`, `AIPlyHat`, `prs2`, `dsh`, `AIDNG`, `AISys`, `CmbR`, `apr`, …), so unlike +`TurnCommands` it *could* be bound by name. Two reasons it was left: the recovery is incomplete +(8 unresolved items) and roughly ten nested sub-objects would each need their own shape and their +own conformance binding. It is the single largest remaining block and a clean, self-contained job +for a following lane — but it is AI cache state, not orders, so it is not on the reimplementation +critical path the way 4a is. + +### 4c. `RNG` (2) and `Attrib` (2) + +`RNG` is correctly opaque — the raw MT19937 state block — and was left alone, as instructed. +`Attrib` is `StreamableHelper` and is an empty frame in every save; there is +nothing to type. + +## 5. The four defects in `save_reader.py`, fixed openly + +Lane G found four real defects in both readers, fixed the engine, and deliberately left the Python +oracle alone rather than quietly editing it mid-campaign. That judgement was right; this is the +follow-through. All four are now fixed in `verify/save-reader/save_reader.py`, each with a test, +and **all four are byte-neutral on all four saves**. + +| # | defect | fix | +|---|---|---| +| 1 | `Game::SystemParams` field 1 is a **string**, not an int | `Planet`'s `p1` retyped | +| 2 | `ObservedTech`/`ObservedWeapon` `odet` is a **bool**, not an int | `Owep`/`Otch` retyped | +| 3 | `Game::SpeciesRatios` `nv` is a **count** | new `SpeciesRatios`/`CivilianRatios` shapes | +| 4 | `Game::ShipRecords` `srbd` is a **count** | new `ShipRecords` shape | + +A wrinkle worth recording: for 3 and 4 the reader did not have the wrong layout, it had **no** +layout — `ShipRecs` and `civr` were both `A(..., "any")`. So the "defect present in both readers" +was, on the Python side, an absence rather than an error. Fixing it meant *adding* the typed shapes +with the count semantics spelled out, which is a bigger change than a retype and is why the +byte-neutrality evidence below matters more than usual. + +### The byte-neutrality evidence + +Every item's inflated offset is unchanged. Dumping all four saves before and after and comparing +the offset column: + +``` +turn1-state: offset sequence IDENTICAL (38,933 items) +turn2-state: offset sequence IDENTICAL (39,843 items) +turn3-state: offset sequence IDENTICAL (40,300 items) +zuul-turn5-species5: offset sequence IDENTICAL (35,771 items) +``` + +so no item boundary moved; only type labels changed. `state_checksum.py` still reports +**`coverage: PROVED`** on all four (591,376 / 603,360 / 609,080 / 532,752 bytes rebuilt == inflated, +unchanged), with `0 error, 0 warn`. + +The subsystem digests for `/CreateParams` and `/Sim/players` do change, and **that is the point** — +they are digests of *typed values*, and two of the fixes change what a value is. The value-byte +totals account for the change exactly: + +| save | Δ value bytes | `odet` items × 3 (int 4 B → bool 1 B) | `p1` items × 4 (int 4 B → "" 0 B) | +|---|---|---|---| +| turn1 | −292 | 60 × 3 = 180 | 28 × 4 = 112 | +| turn2 | −295 | 61 × 3 = 183 | 28 × 4 = 112 | +| turn3 | −295 | 61 × 3 = 183 | 28 × 4 = 112 | +| zuul | −238 | 42 × 3 = 126 | 28 × 4 = 112 | + +Every row balances to the byte. Nothing else in the digest moved. + +`--strict` exits 0 on all four saves. Tests went **36 → 48**, all passing. + +### One finding that fell out of the fix + +`srbd` is not merely inferred from the recovery — the saves **exercise** it. It takes the values +0, 1, 3 and 4 across the players of the four saves, and every non-zero count is followed by exactly +`srbd × 5` scalars (turn3 player 4: `srbd == 4`, then 20 items in four `srd/src/srb/srl/sri` +groups). Read as a field, those trailing scalars have no explanation at all; the new test asserts +exactly that, and the field reading raises. So defect 4 is behaviourally confirmed, not a +hypothesis. Defect 3 is confirmed the same way but more weakly: `nv` is only ever 0 or 1 in these +saves, so the count reading is confirmed for one pair and the multi-pair case is synthetic. + +### The tests + +`test_save_reader.py::WireSchemaDefectsTest`, 12 tests. Each defect gets the case the real saves +happen not to contain (a **named** system, a **bool-valued** `odet`, **two** species pairs, a +**non-empty** design-record section), plus a test showing *why* it stayed invisible — an empty +string and the int 0 are the same four bytes; a bool item and an int item are the same 12 bytes +only because `odet` is 4 characters long, and the same test shows a 3-character tag would not have +been forgiving. + +One nuance the tests made explicit and that belongs on the record: in `save_reader.py` the generic +walker types `"."` items from its **own kind catalog**, not from the schema passed in. On the real +saves the catalog said int for `p1` and the old schema said int, so the two agreed with each other +and were both wrong — the failure mode there is a silent agreement, not the framing desync the +schema-driven engine would have suffered. + +## 6. Artifacts + +* `sots-engine` `wip/wire`: `src/mars/stream/shapes.h` (18 new SVSO shapes, `DOpts` → + `vector`, `spies2` → `vector`, `SvSctOb` → `opt_obj`), + `src/mars/stream/archive.h` (`std::string` element branch in `read_elem` / `write_elem` / + `SchemaBuilder::carr`), `tests/mars_stream/test_wire_schema.cpp` (+18 bindings), + `tests/mars_stream/test_stream.cpp` (+2 unit tests, no saves needed), + `tests/mars_stream/test_save.cpp` (ratchet 95.0 → 97.5). +* `sots-re`: `verify/save-reader/save_reader.py` + `test_save_reader.py`, + `ghidra/addresses.d/lane-w.json` (13 entries), `findings/objects/svsctob-variants.md`, this note. +* `include/generated/sots_stream_schema.h` was **not** regenerated: `tools/streams.py` and + `tools/gen_stream_schema.py` were re-run and the output is byte-identical to the committed header + apart from the provenance line, because nothing on the notes side changed the recovery. +* `ghidra/generated/sots_addresses.h` was **not** regenerated either. Lane W's addresses are in the + fragment; `gen_addresses.py` merges to 640 entries with no duplicate name (checked against a + scratch path). Regenerating in place would have swept lane U's in-flight `lane-u.json` into this + lane's branch, and the engine consumes none of these addresses. + +## 7. Open items + +1. **A save with issued turn commands** remains the blocker for `TurnCommands_v5` (§4a), and it is + now a proven blocker rather than a suspected one. +2. `Game::StrategyAIAgent::Streamable` (§4b) — the largest remaining block, bindable by name, + ~10 nested shapes, no VM needed. +3. The four unexercised `EncID`s (7, 8, 14, 21) and `ScnObj` stay carried until a save or a + scenario game produces one. +4. `serializers.py` should reclassify `Game::SVSOCrowDefenders`'s `dsys` as a loop element of + `ndsys` (see `svsctob-variants.md` §3); `objects/streams.json` still carries the old reading and + the conformance check does not notice, because it aligns by tag and disk primitive. +5. `spies2`, `SysMem`, `mts`, `trev`, `Ojvs`, `aid`, `Attrib`, `comms.nmsg`, `AIEnf.Nas`, + `spymgr.nspy` and `spy2.rtc/evc/ttc` are 0 in every save. Their containers are typed; their + element behaviour is not verified by anything. diff --git a/ghidra/addresses.d/lane-w.json b/ghidra/addresses.d/lane-w.json new file mode 100644 index 0000000..d3cb49b --- /dev/null +++ b/ghidra/addresses.d/lane-w.json @@ -0,0 +1,108 @@ +{ + "entries": [ + { + "name": "SVScriptObject_FactoryByEncID", + "addr": "0x0052bf00", + "convention": "cdecl", + "prototype": "Game::SVScriptObject* (int encID) // The EncObj factory. `dec eax; cmp eax,0x16; ja ; jmp dword [eax*4 + 0x0052bf60]` -- a 23-entry dword jump table indexed by encID-1. Live ids: 1 VonNeumann, 3 Swarm, 4 Derelict, 5 Monitor, 7 SystemKiller, 8 PuppetMaster, 9 SlaversRefuel, 10 SwarmQueen, 14 Locust, 17 CrowRuins, 20 Refugees, 21 Ortgay. Ids 2, 6, 11-13, 15, 16, 18, 19, 22, 23 and everything outside 1..23 return NULL. Class names read off the vftable store in each ctor", + "status": "verified", + "source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)" + }, + { + "name": "SVScriptObject_EncIDJumpTable", + "addr": "0x0052bf60", + "convention": "data", + "prototype": "void* [23] // the jump table SVScriptObject_FactoryByEncID indexes with encID-1", + "status": "verified", + "source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)" + }, + { + "name": "SVSOSots_GetOrCreateEncObj", + "addr": "0x005a4450", + "convention": "thiscall", + "prototype": "Game::SVScriptObject* (Game::SVSOSots* this, int encID) // scans the loaded (id, obj) pair vector at this+0x0C..0x10 first and only calls SVScriptObject_FactoryByEncID on a miss", + "status": "verified", + "source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)" + }, + { + "name": "SVScriptObject_FactoryByScenarioName", + "addr": "0x005a7050", + "convention": "cdecl", + "prototype": "Game::SVScriptObject* (const char* xscn) // the xsc factory: a flat four-way _stricmp chain, EXHAUSTIVE. \"traps\" -> Game::SVSOTraps (ctor 0x0052cbb0), \"crowdefs\" -> Game::SVSOCrowDefenders (0x0052b3c0), \"indsys\" -> Game::SVSOIndependentSystems (0x0075b530), \"gmtrigger\" -> Game::SVSOGrandMenaceTrigger (0x004f5630). Falls through with \"Error creating extra script %s.\"", + "status": "verified", + "source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)" + }, + { + "name": "SVSOSots_SeedNewGameScripts", + "addr": "0x005a7d70", + "convention": "thiscall", + "prototype": "void (Game::SVSOSots* this) // registers exactly traps, crowdefs, indsys and -- only when ScnObj == NULL -- gmtrigger. Corroborates that the four scenario names are the whole set", + "status": "verified", + "source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)" + }, + { + "name": "SVSOSots_Write", + "addr": "0x0059ddf0", + "convention": "thiscall", + "prototype": "void (Game::SVSOSots* this, Mars::IStream* s) // slot 2 of vftable 0x00A063C4. Emits ScnID, ScnObj (only when non-NULL), numx x (xscn, xsc), NEncObjs x (EncID, EncObj)", + "status": "verified", + "source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)" + }, + { + "name": "SVSOSots_Read", + "addr": "0x005a7a40", + "convention": "thiscall", + "prototype": "void (Game::SVSOSots* this, Mars::IStream* s) // slot 1. Accepts two tags Write never emits -- NPCPlr (int, first) and hastraps (bool) with a following traps object -- read-only backward compatibility; a writer that omits them is correct", + "status": "verified", + "source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)" + }, + { + "name": "SVSOSots_vftable", + "addr": "0x00a063c4", + "convention": "data", + "prototype": "Game::SVSOSots vftable (slot0 dtor, slot1 Read 0x005a7a40, slot2 Write 0x0059ddf0)", + "status": "verified", + "source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)" + }, + { + "name": "SVSOIndependentSystems_vftable", + "addr": "0x00a20314", + "convention": "data", + "prototype": "Game::SVSOIndependentSystems vftable. slot0 dtor 0x0075afb0; slot1 Read AND slot2 Write are both 0x005f8ac0, the shared `ret 4` no-op stub -- so the \"indsys\" frame is genuinely empty on disk and the class's 0x1c4-byte body is runtime-only state", + "status": "verified", + "source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)" + }, + { + "name": "Streamable_NoOpStub", + "addr": "0x005f8ac0", + "convention": "thiscall", + "prototype": "void (void*, Mars::IStream*) // `C2 04 00` -- a bare RET 4 shared as the inherited Read/Write for classes that serialize nothing", + "status": "verified", + "source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)" + }, + { + "name": "SVSOCrowDefenders_Write", + "addr": "0x004f8c90", + "convention": "thiscall", + "prototype": "void (Game::SVSOCrowDefenders* this, Mars::IStream* s) // sys; ndsys count then a loop writing dsys; ndes count then a loop writing des; drad. NOTE: `dsys` is INSIDE the ndsys loop -- objects/layouts.json records it as a plain member, which is wrong, and no save can settle it because both counts are 0 everywhere", + "status": "verified", + "source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)" + }, + { + "name": "SVSOMonitor_Write", + "addr": "0x004fd810", + "convention": "thiscall", + "prototype": "void (Game::SVSOMonitor* this, Mars::IStream* s) // calls SVSODerelict::Write (0x004fc2b0) as its first act -- Monitor derives from Derelict, which is why its tag run starts NDsn/DsnID/Dwght + NAsg/Eflt/Esys before nt/scnm/spwt/rsmd/dsgn", + "status": "verified", + "source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)" + }, + { + "name": "SVSODerelict_Write", + "addr": "0x004fc2b0", + "convention": "thiscall", + "prototype": "void (Game::SVSODerelict* this, Mars::IStream* s) // NDsn count then a loop of (DsnID, Dwght); NAsg count then a loop of (Eflt, Esys). Two fields per iteration in each, confirmed by the 8-byte element strides", + "status": "verified", + "source": "findings/objects/svsctob-variants.md (lane W 2026-09-08)" + } + ] +} diff --git a/verify/save-reader/save_reader.py b/verify/save-reader/save_reader.py index dcee455..7236ce6 100644 --- a/verify/save-reader/save_reader.py +++ b/verify/save-reader/save_reader.py @@ -367,7 +367,12 @@ Summary = Shape("Summary", [ # -- CreateParams = StrategyGameCreateParams::Write @0x0082ae40 (§5.1) ---------- Planet = Shape(None, [ # SystemParams element, all tags "." - R("pos", "vec3"), R("p1", "int"), R("p2", "int"), R("p3", "int"), R("p4", "float"), Rest(), + # p1 is a STRING, not an int (Game::SystemParams wire schema: frame, str, i32, + # i32, f32 -- lane G's wire-schema conformance check, findings/objects/ + # wire-schema-channel.md §3.1). It is the empty string in every save we hold, + # and an empty string is four zero bytes -- byte-identical to the int 0 -- so + # this round-tripped by luck. A named system would desynchronise the reader. + R("pos", "vec3"), R("p1", "string"), R("p2", "int"), R("p3", "int"), R("p4", "float"), Rest(), ]) MapP = Shape("MapP", [ # StarMapParams::Write @0x00727a10, all tags "." R("mapType", "int"), @@ -483,9 +488,35 @@ Prep = Shape(None, [ # is `otnF` on disk (R1 had `ontF`; positional matching hid the typo -- caught by # the C++ round-trip writer and confirmed in the bytes, SAVE_FORMAT.md §10). Odes = Shape(None, [A("otnF", "int"), A("otnL", "int"), A("odid", "int"), A("opid", "int"), Rest()]) -Owep = Shape(None, [A("otnF", "int"), A("otnL", "int"), A("odet", "int"), A("owep", "string"), A("owith", "int"), Rest()]) -Otch = Shape(None, [A("otnF", "int"), A("otnL", "int"), A("odet", "int"), A("otch", "string"), A("owith", "int"), Rest()]) +# `odet` is a BOOL, not an int: Game::ObservedWeapon / Game::ObservedTech call the +# bool writer (wire-schema-channel.md §3.2). It is byte-safe here only because the +# tag is 4 characters, which makes a bool item and an int item both 12 bytes; a 3- +# or 5-character tag would not have been so forgiving. +Owep = Shape(None, [A("otnF", "int"), A("otnL", "int"), A("odet", "bool"), A("owep", "string"), A("owith", "int"), Rest()]) +Otch = Shape(None, [A("otnF", "int"), A("otnL", "int"), A("odet", "bool"), A("otch", "string"), A("owith", "int"), Rest()]) Note = Shape("Nts", [A("NtSys", "int"), A("NtTxt", "string"), A("NtTrn", "int"), Rest()]) +# Game::ShipRecords. TWO COUNTED SECTIONS, and `srbd` is the second COUNT, not a +# field (wire-schema-channel.md §3.4). The recovery lists srd/src/srb/srl/sri +# right after `srbd` as loop-body writes, and the saves EXERCISE it: srbd takes the +# values 0, 1, 3 and 4 across the players of the four saves we hold, and every +# non-zero count is followed by exactly srbd x 5 scalars (e.g. turn3 player 4 has +# srbd == 4 and 20 items). So this one is behaviourally confirmed, not inferred. +DesignRecord = Seq([ + A("srd", "int"), A("src", "int"), A("srb", "int"), A("srl", "int"), A("sri", "int"), +]) +ShipRecords = Shape("ShipRecs", [ + A("srnc", NArr(Seq([A("srb", "int"), A("srl", "int"), A("srk", "int"), A("sri", "int")])), + key="records"), + A("srbd", NArr(DesignRecord), key="designRecords"), + Rest(), +]) +# Game::SpeciesRatios. `nv` is the COUNT of (sp, va2) pairs, not a field +# (wire-schema-channel.md §3.3): the recovery marks sp and va2 as loop-body writes +# and the saves agree -- nv==1 frames carry one pair, nv==0 frames carry nothing. +SpeciesRatios = Shape("spe", [ + A("nv", NArr(Seq([A("sp", "int"), A("va2", "int")])), key="ratios"), Rest(), +]) +CivilianRatios = Shape("civr", [A("smx", "float"), A("spe", SpeciesRatios), Rest()]) # Design header: on-disk tags are FAIDes/DHide/DWep/DName (R1's camel-case names # were wrong). The typed-dict keys keep R1's spelling because # verify/design-rules/stock_designs.py consumes them. @@ -521,9 +552,9 @@ Player = Shape("Player", [ A("CnTrd", "bool"), A("CnRad", "bool"), A("hgs", "bool"), A("hadvs", "bool"), A("harcc", "bool"), A("CnVItl", "bool"), A("pddm", "float"), A("BnkWrn", "int"), A("BnkTrn", "int"), A("BnkPr", "int"), A("BnkEl", "int"), - A("ShipRecs", "any"), + A("ShipRecs", ShipRecords, key="shipRecs"), A("NextPrjID", "int"), A("plcy", "int"), A("pswd", "string"), A("lret", "int"), A("nmeid", "int"), - A("cdp", "bool"), A("spy2", "any"), A("civr", "any"), A("aidf", "int"), + A("cdp", "bool"), A("spy2", "any"), A("civr", CivilianRatios), A("aidf", "int"), A("Srn", "bool"), A("SrnTo", "int"), A("lboid", "int"), Opt("lcid", "int"), A("lcid2", "int"), A("ResTNm", "string"), A("ResErrRoll", "bool"), R("conMods", ConMods, flex=True), diff --git a/verify/save-reader/test_save_reader.py b/verify/save-reader/test_save_reader.py index 1b6c144..b1b91e9 100644 --- a/verify/save-reader/test_save_reader.py +++ b/verify/save-reader/test_save_reader.py @@ -573,6 +573,213 @@ class TagNameCorrectionsTest(unittest.TestCase): self.assertEqual((plans[0]["wpts"][0]["Wpt"], plans[0]["wpts"][0]["nrt"]["nrp"]), (272, -1)) +class WireSchemaDefectsTest(unittest.TestCase): + """The four defects lane G's SchemaProbe found in BOTH readers, and which no + round-trip test could catch (findings/objects/wire-schema-channel.md §3). + + Each `test_*_would_have_caught_it` builds the case the real saves happen not + to contain. Each `test_*_is_byte_neutral_here` shows why the defect stayed + invisible: on the data we actually hold the wrong type occupies the same + bytes, so the round trip was clean and wrong at the same time. + """ + + REAL = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))), "results", "saves") + + # --- 1. Game::SystemParams field 1 is a string, not an int ------------------ + @staticmethod + def system_params(name): + """One MapP frame with a single SystemParams element carrying `name`.""" + w = sw.SaveWriter() + w.begin("MapP") + w.int(".", 0) # mapType + w.begin(".") # VectorHelper + w.int(".", 1) + w.begin(".") # the element, all tags "." + w.vec3(".", 1.0, 2.0, 3.0) + w.string(".", name) # <- the field in question + w.int(".", 7) + w.int(".", 8) + w.float(".", 0.5) + w.end() + w.end() + w.int(".", 0) # players: bare count, no groups + w.begin("."); w.int(".", 0); w.end() # nodePaths: framed count + w.end() + return w.bytes() + + def test_system_params_p1_is_a_string(self): + schema = sr.Seq([sr.A("MapP", sr.MapP)]) + res = sr.read_bytes(self.system_params(""), padding="joint", strict=True, schema=schema) + planet = res.typed["MapP"]["planets"][0] + self.assertEqual(planet["p1"], "") + self.assertIsInstance(planet["p1"], str) + + def test_a_named_system_reads_back_intact(self): + """The case no save we hold contains. A non-empty name is a string of a + length the int reading cannot express, and every following field still has + to land -- this is the read the campaign had no answer for.""" + data = self.system_params("Beta Hydri") + res = sr.read_bytes(data, padding="joint", strict=True, schema=sr.Seq([sr.A("MapP", sr.MapP)])) + planet = res.typed["MapP"]["planets"][0] + self.assertEqual((planet["p1"], planet["p2"], planet["p3"], planet["p4"]), + ("Beta Hydri", 7, 8, 0.5)) + + def test_the_old_int_type_disagrees_with_these_bytes(self): + """The catching assertion: with `p1` typed int, applying the schema to a + SystemParams element is a hard error, named or not. (The reader survived + the real saves only because its own kind catalog also said int there, so + schema and walker agreed with each other and both were wrong.)""" + old = sr.Shape(None, [sr.R("pos", "vec3"), sr.R("p1", "int"), sr.R("p2", "int"), + sr.R("p3", "int"), sr.R("p4", "float"), sr.Rest()]) + old_map = sr.Shape("MapP", [sr.R("mapType", "int"), sr.R("planets", sr.CArr(old)), + sr.R("players", sr.NArr(sr.CArr("int"))), + sr.R("nodePaths", sr.CArr("any")), sr.Rest()]) + for name in ("", "Beta Hydri"): + with self.assertRaises(sr.SaveFormatError): + sr.read_bytes(self.system_params(name), padding="joint", strict=True, + schema=sr.Seq([sr.A("MapP", old_map)])) + + def test_empty_name_is_why_it_stayed_hidden(self): + """An empty string is four zero bytes -- byte-identical to the int 0 -- so + on every save we hold, the wrong type cost nothing.""" + s = sw.SaveWriter(); s.string(".", "") + i = sw.SaveWriter(); i.int(".", 0) + self.assertEqual(s.bytes(), i.bytes()) + # only the empty one is: a string payload is [int32 len][bytes], so a name + # is 4 + len bytes where the int is 4, and the item stops agreeing as soon + # as the name pushes it past the next multiple of four. + named = sw.SaveWriter(); named.string(".", "Beta Hydri") + self.assertNotEqual(len(named.bytes()), len(i.bytes())) + + # --- 2. ObservedTech / ObservedWeapon `odet` is a bool, not an int ---------- + def test_odet_is_declared_bool(self): + for shape in (sr.Owep, sr.Otch): + kinds = {f.name: f.type for f in shape.fields if isinstance(f, sr.Field)} + self.assertEqual(kinds["odet"], "bool") + self.assertEqual(sr.GLOBAL_KINDS["odet"], "bool") + + def test_odet_reads_back_as_a_bool_not_an_int(self): + w = sw.SaveWriter() + w.begin("otch"); w.int(".", 1) + w.begin("."); w.int("otnF", 1); w.int("otnL", 1); w.bool("odet", True) + w.string("otch", "WEP_RedLas"); w.int("owith", 1); w.end() + w.end() + res = sr.read_bytes(w.bytes(), padding="joint", strict=True, + schema=sr.Seq([sr.A("otch", sr.CArr(sr.Otch))])) + v = res.typed["otch"][0]["odet"] + self.assertIsInstance(v, bool) # `assertEqual(v, 1)` passes either way + self.assertIs(v, True) + + def test_a_four_char_tag_is_the_only_reason_odet_was_byte_safe(self): + """bool and int items are the same size ONLY when the tag length makes the + joint padding agree. 4 chars: 4+4+1 -> 12 and 4+4+4 = 12. 3 chars: + 4+3+1 -> 8 but 4+3+4 -> 12, and the parse desyncs.""" + four = sw.SaveWriter(); four.bool("odet", True); four.int("owith", 1) + as_int = sw.SaveWriter(); as_int.int("odet", 1); as_int.int("owith", 1) + self.assertEqual(len(four.bytes()), len(as_int.bytes())) # the coincidence + three = sw.SaveWriter(); three.bool("det", True); three.int("owith", 1) + three_int = sw.SaveWriter(); three_int.int("det", 1); three_int.int("owith", 1) + self.assertNotEqual(len(three.bytes()), len(three_int.bytes())) # not a property + + # --- 3. Game::SpeciesRatios `nv` is a count ------------------------------- + @staticmethod + def civr(pairs): + w = sw.SaveWriter() + w.begin("civr"); w.float("smx", 0.5) + w.begin("spe"); w.int("nv", len(pairs)) + for sp, va2 in pairs: + w.int("sp", sp); w.int("va2", va2) + w.end(); w.end() + return w.bytes() + + def test_nv_is_a_count_not_a_field(self): + schema = sr.Seq([sr.A("civr", sr.CivilianRatios)]) + res = sr.read_bytes(self.civr([(0, 100)]), padding="joint", strict=True, schema=schema) + self.assertEqual(norm(res.typed["civr"]["spe"]["ratios"]), [{"sp": 0, "va2": 100}]) + res0 = sr.read_bytes(self.civr([]), padding="joint", strict=True, schema=schema) + self.assertEqual(res0.typed["civr"]["spe"]["ratios"], []) + + def test_two_species_would_have_broken_the_field_reading(self): + """nv is 0 or 1 in every save we hold, so a field reading survives. With + two pairs the second is unexplained and strict parsing fails.""" + data = self.civr([(0, 60), (5, 40)]) + res = sr.read_bytes(data, padding="joint", strict=True, + schema=sr.Seq([sr.A("civr", sr.CivilianRatios)])) + self.assertEqual(norm(res.typed["civr"]["spe"]["ratios"]), + [{"sp": 0, "va2": 60}, {"sp": 5, "va2": 40}]) + wrong = sr.Shape("spe", [sr.A("nv", "int"), sr.A("sp", "int"), sr.A("va2", "int")]) + wrong_civr = sr.Shape("civr", [sr.A("smx", "float"), sr.A("spe", wrong)]) + with self.assertRaises(sr.SaveFormatError): + sr.read_bytes(data, padding="joint", strict=True, + schema=sr.Seq([sr.A("civr", wrong_civr)])) + + # --- 4. Game::ShipRecords `srbd` is a count ------------------------------- + @staticmethod + def ship_recs(recs, designs): + w = sw.SaveWriter() + w.begin("ShipRecs") + w.int("srnc", len(recs)) + for srb, srl, srk, sri in recs: + w.int("srb", srb); w.int("srl", srl); w.int("srk", srk); w.int("sri", sri) + w.int("srbd", len(designs)) + for srd, src, srb, srl, sri in designs: + w.int("srd", srd); w.int("src", src); w.int("srb", srb) + w.int("srl", srl); w.int("sri", sri) + w.end() + return w.bytes() + + def test_srbd_is_a_count_not_a_field(self): + data = self.ship_recs([(0, 0, 0, 0)], [(18, 0, 2, 0, 2)]) + res = sr.read_bytes(data, padding="joint", strict=True, + schema=sr.Seq([sr.A("ShipRecs", sr.ShipRecords)])) + sr_ = res.typed["ShipRecs"] + self.assertEqual(norm(sr_["records"]), [{"srb": 0, "srl": 0, "srk": 0, "sri": 0}]) + self.assertEqual(norm(sr_["designRecords"]), + [{"srd": 18, "src": 0, "srb": 2, "srl": 0, "sri": 2}]) + + def test_srbd_as_a_field_cannot_explain_the_trailing_records(self): + data = self.ship_recs([(0, 0, 0, 0)], [(18, 0, 2, 0, 2)]) + wrong = sr.Shape("ShipRecs", [ + sr.A("srnc", sr.NArr(sr.Seq([sr.A("srb", "int"), sr.A("srl", "int"), + sr.A("srk", "int"), sr.A("sri", "int")]))), + sr.A("srbd", "int"), + ]) + with self.assertRaises(sr.SaveFormatError): + sr.read_bytes(data, padding="joint", strict=True, + schema=sr.Seq([sr.A("ShipRecs", wrong)])) + + # --- the four corrections against the real saves --------------------------- + @unittest.skipUnless(os.path.exists(os.path.join(REAL, "turn3-state.sav")), "real saves not present") + def test_real_saves_agree_with_the_corrected_types(self): + res = sr.read_save(os.path.join(self.REAL, "turn3-state.sav"), padding="joint", strict=True) + # 1. every SystemParams name is the empty STRING (never the int 0) + planets = res.typed["createParams"]["MapP"]["planets"] + self.assertTrue(planets) + for p in planets: + self.assertIsInstance(p["p1"], str) + self.assertEqual(p["p1"], "") + players = [p["Player"] for p in res.typed["sim"]["players"]] + # 2. every odet is a bool + odets = [o["odet"] for pl in players for key in ("otch", "owep") for o in pl.get(key, [])] + self.assertTrue(odets) + for v in odets: + self.assertIsInstance(v, bool) + # 3. nv counts (sp, va2) pairs: 1 -> one pair, 0 -> none + seen = set() + for pl in players: + ratios = pl["civr"]["spe"]["ratios"] + seen.add(len(ratios)) + for r in ratios: + self.assertEqual(set(r) - {"_off"}, {"sp", "va2"}) + self.assertEqual(seen, {0, 1}) + # 4. srbd counts 5-scalar design records; turn3 exercises 0, 1 and 4 + counts = sorted({len(pl["shipRecs"]["designRecords"]) for pl in players}) + self.assertEqual(counts, [0, 1, 4]) + for pl in players: + for d in pl["shipRecs"]["designRecords"]: + self.assertEqual(set(d) - {"_off"}, {"srd", "src", "srb", "srl", "sri"}) + + class CliTest(unittest.TestCase): def setUp(self): self.tmp = tempfile.TemporaryDirectory()