2308b6edb9
Part 1 (notes side): findings/objects/svsctob-variants.md records the two maps
that are nowhere on the wire and were read out of the game -- EncID -> class
from a 23-entry dword jump table at 0x0052bf60 indexed by EncID-1 (0x0052bf00),
and xscn -> class from an exhaustive four-way _stricmp chain at 0x005a7050.
Twelve live EncIDs, four scenario names, and "indsys" =
Game::SVSOIndependentSystems whose Read and Write are both the shared `ret 4`
stub at 0x005f8ac0, so its empty frame is correct output rather than a
truncation. Also: SVSOSots::Read accepts NPCPlr and hastraps, which Write never
emits -- read-only backward compatibility, not a hole in the recovery. And a
correction to the recovery itself: SVSOCrowDefenders writes `dsys` INSIDE the
ndsys loop; layouts.json calls it a plain member and no save can settle it
because both counts are 0 everywhere. 13 addresses in ghidra/addresses.d/
lane-w.json; gen_addresses.py merges to 640 with no duplicate name.
Part 2: lane G found four defects in both readers and deliberately did not
patch the oracle mid-campaign. Fixed now, with tests, and byte-neutral.
1. Game::SystemParams field 1 is a string, not an int (empty string == four
zero bytes == int 0, so it round-tripped by luck).
2. ObservedTech/ObservedWeapon odet is a bool, not an int (byte-safe only
because a 4-char tag makes both items 12 bytes; 3 chars would not).
3. SpeciesRatios nv is a count, not a field.
4. ShipRecords srbd is a count, not a field -- and this one is behaviourally
confirmed, not inferred: srbd takes 0, 1, 3 and 4 across the players and
every non-zero count is followed by exactly srbd x 5 scalars.
Note that 3 and 4 were an ABSENCE in save_reader.py, not an error: ShipRecs and
civr were both A(..., "any"), so the fix had to add the shapes rather than
retype a field.
Byte-neutrality: every item's inflated offset is unchanged on all four saves
(38,933 / 39,843 / 40,300 / 35,771 offsets, sequences identical), so no item
boundary moved. state_checksum.py still reports coverage: PROVED on all four
with the same rebuilt byte counts. The /CreateParams and /Sim/players digests do
change, because they hash typed VALUES and two fixes change what a value is --
and the value-byte deltas balance exactly: odet items x 3 plus p1 items x 4.
--strict exit 0 on all four saves; tests 36 -> 48.
findings/objects/wire-schema-closeout.md carries the whole account, including
the proof that CD/TurnCommands_v5 cannot be typed without a save that has
issued orders.
213 lines
13 KiB
Markdown
213 lines
13 KiB
Markdown
# Closing the opaque sections, and correcting the Python oracle
|
||
|
||
Lane W, 2026-09-08. Host/static only — VM140 was held by lane U and the game was never run.
|
||
Continues `findings/objects/wire-schema-channel.md` (lane G).
|
||
|
||
## Headline: named coverage 97.1 % → 98.0 %
|
||
|
||
The honest metric is `CoverageArchive`'s split of items a field **names** from items a generic
|
||
`Node` merely **carries** — not round-trip success, which `ar.any` bodies achieve trivially by
|
||
copying bytes nobody understands. The byte-identical round trip held throughout.
|
||
|
||
| save | before | after | still opaque after |
|
||
|---|---|---|---|
|
||
| turn1-state | 97.1 % | **98.0 %** | `CD` 744, `RNG` 2, `Attrib` 2 |
|
||
| turn2-state | 97.2 % | **98.0 %** | `CD` 744, `RNG` 2, `Attrib` 2 |
|
||
| turn3-state | 97.2 % | **98.0 %** | `CD` 744, `RNG` 2, `Attrib` 2 |
|
||
| zuul-turn5-species5 | 97.6 % | **98.4 %** | `CD` 517, `RNG` 2, `Attrib` 2 |
|
||
|
||
Conformance: **74 shapes bound, 769 items matched, 0 MISMATCH** (was 56 / 657 / 0). The ratchet in
|
||
`test_save.cpp` moved 95.0 → 97.5. `ctest` 34/34; `tools/clean_room_check.sh` OK; `test_save`
|
||
still skips cleanly with `SOTS_SAVES_DIR` unset.
|
||
|
||
`CD` is now the *only* remaining region of any size, and §4 says why it cannot be closed here.
|
||
|
||
## 1. `SvSctOb` — closed, 147 items → 0
|
||
|
||
Its own note: `findings/objects/svsctob-variants.md`. Summary: the body is `Game::SVSOSots`, which
|
||
writes two variant lists dispatched by a preceding key (`xscn` for `xsc`, `EncID` for `EncObj`).
|
||
Neither map is on the wire; both were read out of the game — the ids from a 23-entry dword jump
|
||
table at 0x0052bf60 indexed by `EncID - 1`, the names from an exhaustive four-way `_stricmp` chain
|
||
at 0x005a7050. Eighteen shapes bound, all clean. The four factory ids no save exercises
|
||
(7 SystemKiller, 8 PuppetMaster, 14 Locust, 21 Ortgay) are deliberately left carried, not typed.
|
||
|
||
## 2. `DOpts` — closed, 94 items → 0. Certain, from the helper's own decorated name.
|
||
|
||
`Game::ShipDesignDef::Section`'s `DOpts` is `?$VectorHelper@VString@Mars@@@Mars` — a
|
||
`Mars::VectorHelper<Mars::String>`. The element type is not inferred from the bytes; it is in the
|
||
mangled helper type the writer instantiated. The engine's `read_elem`/`write_elem` grew a
|
||
`std::string` branch (the one thing lane G listed as missing) and `carr<std::string>` now works.
|
||
|
||
The same branch closed `Game::SVSOVonNeumann::trev`, also `VectorHelper<Mars::String>`.
|
||
|
||
Note the empty-string trap this had to survive: an empty string is four zero bytes, so an empty
|
||
`DOpts` element is byte-identical to the int 0. The new unit test writes
|
||
`{"OPT_Armour", "", "OPT_Shield"}` specifically to exercise it.
|
||
|
||
## 3. `spies2` — closed, 56 items → 0, but read this before believing the element layout
|
||
|
||
`Game::ServerSystem`'s `spies2` is `?$VectorHelper@H@Mars` — `Mars::VectorHelper<int>`. So it is a
|
||
framed array of NULL-named ints, and that is a **certainty about the type**, from the same
|
||
decorated-name evidence as `DOpts`, not a guess from the bytes.
|
||
|
||
What is **not** exercised is any element. The count is 0 in all 28 systems of all four saves, so
|
||
the 56 items closed here are 28 × (the frame + its `.` count) and **no `spies2` element value has
|
||
ever been observed on disk**. If a future save carries spies, the shape says they are plain ints;
|
||
that claim rests entirely on `VectorHelper<int>` and would be falsified by the first non-empty one.
|
||
Labelling it: the *type* is certain, the *behaviour* is a hypothesis.
|
||
|
||
`Game::SVSOSwarmQueen::SysMem` and `Game::SVSOVonNeumann::mts` are in the same position —
|
||
`?$VectorHelper@U?$StreamableEnum@I@Mars@@@Mars`, i.e. `VectorHelper<StreamableEnum<uint>>`, typed
|
||
as int arrays, count 0 everywhere.
|
||
|
||
## 4. `CD` — NOT closed, and here is the evidence that it cannot be, without a workload
|
||
|
||
`CD` is two different problems wearing one tag. `CDT` lists the ids in order and one `CD` frame
|
||
follows per id:
|
||
|
||
* **`Player.<id>.TurnCommands_v5`** → `Game::TurnCommands`. One block, 35 items on turn1.
|
||
* **`Player.<id>.AIAgent`** → `Game::StrategyAIAgent::Streamable`. Three blocks, ~236 items each,
|
||
and these are the bulk of the 744.
|
||
|
||
### 4a. `TurnCommands`: the recovered sequence provably does not align to a no-orders save
|
||
|
||
`Game::TurnCommands` is recovered complete at **44 items, every tag `"."`**. The turn1 block has
|
||
**35**. Lane G called it a no-orders snapshot. That is right, and it is now stronger than a
|
||
description — the two sequences **cannot be aligned at all**, not even as a subsequence:
|
||
|
||
* Positionally they diverge at **item 4**: the save's item 4 is 8 bytes, which for a 1-character
|
||
tag can only be a `bool` (4 + 1 + 1 → 8; an int is 4 + 1 + 4 → 12). The recovery says `i32`.
|
||
Item 16 is a plain 12-byte int in the save where the recovery says a frame.
|
||
* As a subsequence it fails on counting. The save is `int, bool, float, bool, bool, bool, bool,
|
||
bool` then **27 consecutive ints**. After the recovery's item 15 there are only **22** `i32`
|
||
slots (indices 17–20, 22–27, 29–31, 33–40, 42); the rest are frames, a `narr`, and bools.
|
||
27 > 22, so no order-preserving assignment exists.
|
||
|
||
So the writer takes branches that a no-orders turn does not, and with every tag `"."` there is no
|
||
name to bind against. **`TurnCommands` needs a save with issued orders** — which needs the VM.
|
||
Anything typed from here would be a guess dressed as a layout, so nothing was typed.
|
||
|
||
### 4b. `AIAgent`: statically tractable, deliberately not attempted
|
||
|
||
`Game::StrategyAIAgent::Streamable` is recovered at 36 items with **named** tags (`AIAttr`,
|
||
`AITurnPris`, `AISit`, `AIPlyHat`, `prs2`, `dsh`, `AIDNG`, `AISys`, `CmbR`, `apr`, …), so unlike
|
||
`TurnCommands` it *could* be bound by name. Two reasons it was left: the recovery is incomplete
|
||
(8 unresolved items) and roughly ten nested sub-objects would each need their own shape and their
|
||
own conformance binding. It is the single largest remaining block and a clean, self-contained job
|
||
for a following lane — but it is AI cache state, not orders, so it is not on the reimplementation
|
||
critical path the way 4a is.
|
||
|
||
### 4c. `RNG` (2) and `Attrib` (2)
|
||
|
||
`RNG` is correctly opaque — the raw MT19937 state block — and was left alone, as instructed.
|
||
`Attrib` is `StreamableHelper<Game::AttribMap>` and is an empty frame in every save; there is
|
||
nothing to type.
|
||
|
||
## 5. The four defects in `save_reader.py`, fixed openly
|
||
|
||
Lane G found four real defects in both readers, fixed the engine, and deliberately left the Python
|
||
oracle alone rather than quietly editing it mid-campaign. That judgement was right; this is the
|
||
follow-through. All four are now fixed in `verify/save-reader/save_reader.py`, each with a test,
|
||
and **all four are byte-neutral on all four saves**.
|
||
|
||
| # | defect | fix |
|
||
|---|---|---|
|
||
| 1 | `Game::SystemParams` field 1 is a **string**, not an int | `Planet`'s `p1` retyped |
|
||
| 2 | `ObservedTech`/`ObservedWeapon` `odet` is a **bool**, not an int | `Owep`/`Otch` retyped |
|
||
| 3 | `Game::SpeciesRatios` `nv` is a **count** | new `SpeciesRatios`/`CivilianRatios` shapes |
|
||
| 4 | `Game::ShipRecords` `srbd` is a **count** | new `ShipRecords` shape |
|
||
|
||
A wrinkle worth recording: for 3 and 4 the reader did not have the wrong layout, it had **no**
|
||
layout — `ShipRecs` and `civr` were both `A(..., "any")`. So the "defect present in both readers"
|
||
was, on the Python side, an absence rather than an error. Fixing it meant *adding* the typed shapes
|
||
with the count semantics spelled out, which is a bigger change than a retype and is why the
|
||
byte-neutrality evidence below matters more than usual.
|
||
|
||
### The byte-neutrality evidence
|
||
|
||
Every item's inflated offset is unchanged. Dumping all four saves before and after and comparing
|
||
the offset column:
|
||
|
||
```
|
||
turn1-state: offset sequence IDENTICAL (38,933 items)
|
||
turn2-state: offset sequence IDENTICAL (39,843 items)
|
||
turn3-state: offset sequence IDENTICAL (40,300 items)
|
||
zuul-turn5-species5: offset sequence IDENTICAL (35,771 items)
|
||
```
|
||
|
||
so no item boundary moved; only type labels changed. `state_checksum.py` still reports
|
||
**`coverage: PROVED`** on all four (591,376 / 603,360 / 609,080 / 532,752 bytes rebuilt == inflated,
|
||
unchanged), with `0 error, 0 warn`.
|
||
|
||
The subsystem digests for `/CreateParams` and `/Sim/players` do change, and **that is the point** —
|
||
they are digests of *typed values*, and two of the fixes change what a value is. The value-byte
|
||
totals account for the change exactly:
|
||
|
||
| save | Δ value bytes | `odet` items × 3 (int 4 B → bool 1 B) | `p1` items × 4 (int 4 B → "" 0 B) |
|
||
|---|---|---|---|
|
||
| turn1 | −292 | 60 × 3 = 180 | 28 × 4 = 112 |
|
||
| turn2 | −295 | 61 × 3 = 183 | 28 × 4 = 112 |
|
||
| turn3 | −295 | 61 × 3 = 183 | 28 × 4 = 112 |
|
||
| zuul | −238 | 42 × 3 = 126 | 28 × 4 = 112 |
|
||
|
||
Every row balances to the byte. Nothing else in the digest moved.
|
||
|
||
`--strict` exits 0 on all four saves. Tests went **36 → 48**, all passing.
|
||
|
||
### One finding that fell out of the fix
|
||
|
||
`srbd` is not merely inferred from the recovery — the saves **exercise** it. It takes the values
|
||
0, 1, 3 and 4 across the players of the four saves, and every non-zero count is followed by exactly
|
||
`srbd × 5` scalars (turn3 player 4: `srbd == 4`, then 20 items in four `srd/src/srb/srl/sri`
|
||
groups). Read as a field, those trailing scalars have no explanation at all; the new test asserts
|
||
exactly that, and the field reading raises. So defect 4 is behaviourally confirmed, not a
|
||
hypothesis. Defect 3 is confirmed the same way but more weakly: `nv` is only ever 0 or 1 in these
|
||
saves, so the count reading is confirmed for one pair and the multi-pair case is synthetic.
|
||
|
||
### The tests
|
||
|
||
`test_save_reader.py::WireSchemaDefectsTest`, 12 tests. Each defect gets the case the real saves
|
||
happen not to contain (a **named** system, a **bool-valued** `odet`, **two** species pairs, a
|
||
**non-empty** design-record section), plus a test showing *why* it stayed invisible — an empty
|
||
string and the int 0 are the same four bytes; a bool item and an int item are the same 12 bytes
|
||
only because `odet` is 4 characters long, and the same test shows a 3-character tag would not have
|
||
been forgiving.
|
||
|
||
One nuance the tests made explicit and that belongs on the record: in `save_reader.py` the generic
|
||
walker types `"."` items from its **own kind catalog**, not from the schema passed in. On the real
|
||
saves the catalog said int for `p1` and the old schema said int, so the two agreed with each other
|
||
and were both wrong — the failure mode there is a silent agreement, not the framing desync the
|
||
schema-driven engine would have suffered.
|
||
|
||
## 6. Artifacts
|
||
|
||
* `sots-engine` `wip/wire`: `src/mars/stream/shapes.h` (18 new SVSO shapes, `DOpts` →
|
||
`vector<string>`, `spies2` → `vector<int32_t>`, `SvSctOb` → `opt_obj<ScriptObjects>`),
|
||
`src/mars/stream/archive.h` (`std::string` element branch in `read_elem` / `write_elem` /
|
||
`SchemaBuilder::carr`), `tests/mars_stream/test_wire_schema.cpp` (+18 bindings),
|
||
`tests/mars_stream/test_stream.cpp` (+2 unit tests, no saves needed),
|
||
`tests/mars_stream/test_save.cpp` (ratchet 95.0 → 97.5).
|
||
* `sots-re`: `verify/save-reader/save_reader.py` + `test_save_reader.py`,
|
||
`ghidra/addresses.d/lane-w.json` (13 entries), `findings/objects/svsctob-variants.md`, this note.
|
||
* `include/generated/sots_stream_schema.h` was **not** regenerated: `tools/streams.py` and
|
||
`tools/gen_stream_schema.py` were re-run and the output is byte-identical to the committed header
|
||
apart from the provenance line, because nothing on the notes side changed the recovery.
|
||
* `ghidra/generated/sots_addresses.h` was **not** regenerated either. Lane W's addresses are in the
|
||
fragment; `gen_addresses.py` merges to 640 entries with no duplicate name (checked against a
|
||
scratch path). Regenerating in place would have swept lane U's in-flight `lane-u.json` into this
|
||
lane's branch, and the engine consumes none of these addresses.
|
||
|
||
## 7. Open items
|
||
|
||
1. **A save with issued turn commands** remains the blocker for `TurnCommands_v5` (§4a), and it is
|
||
now a proven blocker rather than a suspected one.
|
||
2. `Game::StrategyAIAgent::Streamable` (§4b) — the largest remaining block, bindable by name,
|
||
~10 nested shapes, no VM needed.
|
||
3. The four unexercised `EncID`s (7, 8, 14, 21) and `ScnObj` stay carried until a save or a
|
||
scenario game produces one.
|
||
4. `serializers.py` should reclassify `Game::SVSOCrowDefenders`'s `dsys` as a loop element of
|
||
`ndsys` (see `svsctob-variants.md` §3); `objects/streams.json` still carries the old reading and
|
||
the conformance check does not notice, because it aligns by tag and disk primitive.
|
||
5. `spies2`, `SysMem`, `mts`, `trev`, `Ojvs`, `aid`, `Attrib`, `comms.nmsg`, `AIEnf.Nas`,
|
||
`spymgr.nspy` and `spy2.rtc/evc/ttc` are 0 in every save. Their containers are typed; their
|
||
element behaviour is not verified by anything.
|