# Closing the opaque sections, and correcting the Python oracle

Lane W, 2026-09-08. Host/static only — VM140 was held by lane U and the game was never run.
Continues `findings/objects/wire-schema-channel.md` (lane G).

## Headline: named coverage 97.1 % → 98.0 %

The honest metric is `CoverageArchive`'s split of items a field **names** from items a generic
`Node` merely **carries** — not round-trip success, which `ar.any` bodies achieve trivially by
copying bytes nobody understands. The byte-identical round trip held throughout.

| save | before | after | still opaque after |
|---|---|---|---|
| turn1-state | 97.1 % | **98.0 %** | `CD` 744, `RNG` 2, `Attrib` 2 |
| turn2-state | 97.2 % | **98.0 %** | `CD` 744, `RNG` 2, `Attrib` 2 |
| turn3-state | 97.2 % | **98.0 %** | `CD` 744, `RNG` 2, `Attrib` 2 |
| zuul-turn5-species5 | 97.6 % | **98.4 %** | `CD` 517, `RNG` 2, `Attrib` 2 |

Conformance: **74 shapes bound, 769 items matched, 0 MISMATCH** (was 56 / 657 / 0). The ratchet in
`test_save.cpp` moved 95.0 → 97.5. `ctest` 34/34; `tools/clean_room_check.sh` OK; `test_save`
still skips cleanly with `SOTS_SAVES_DIR` unset.

`CD` is now the *only* remaining region of any size, and §4 says why it cannot be closed here.

## 1. `SvSctOb` — closed, 147 items → 0

Its own note: `findings/objects/svsctob-variants.md`. Summary: the body is `Game::SVSOSots`, which
writes two variant lists dispatched by a preceding key (`xscn` for `xsc`, `EncID` for `EncObj`).
Neither map is on the wire; both were read out of the game — the ids from a 23-entry dword jump
table at 0x0052bf60 indexed by `EncID - 1`, the names from an exhaustive four-way `_stricmp` chain
at 0x005a7050. Eighteen shapes bound, all clean. The four factory ids no save exercises
(7 SystemKiller, 8 PuppetMaster, 14 Locust, 21 Ortgay) are deliberately left carried, not typed.

## 2. `DOpts` — closed, 94 items → 0. Certain, from the helper's own decorated name.

`Game::ShipDesignDef::Section`'s `DOpts` is `?$VectorHelper@VString@Mars@@@Mars` — a
`Mars::VectorHelper<Mars::String>`. The element type is not inferred from the bytes; it is in the
mangled helper type the writer instantiated. The engine's `read_elem`/`write_elem` grew a
`std::string` branch (the one thing lane G listed as missing) and `carr<std::string>` now works.

The same branch closed `Game::SVSOVonNeumann::trev`, also `VectorHelper<Mars::String>`.

Note the empty-string trap this had to survive: an empty string is four zero bytes, so an empty
`DOpts` element is byte-identical to the int 0. The new unit test writes
`{"OPT_Armour", "", "OPT_Shield"}` specifically to exercise it.

## 3. `spies2` — closed, 56 items → 0, but read this before believing the element layout

`Game::ServerSystem`'s `spies2` is `?$VectorHelper@H@Mars` — `Mars::VectorHelper<int>`. So it is a
framed array of NULL-named ints, and that is a **certainty about the type**, from the same
decorated-name evidence as `DOpts`, not a guess from the bytes.

What is **not** exercised is any element. The count is 0 in all 28 systems of all four saves, so
the 56 items closed here are 28 × (the frame + its `.` count) and **no `spies2` element value has
ever been observed on disk**. If a future save carries spies, the shape says they are plain ints;
that claim rests entirely on `VectorHelper<int>` and would be falsified by the first non-empty one.
Labelling it: the *type* is certain, the *behaviour* is a hypothesis.

`Game::SVSOSwarmQueen::SysMem` and `Game::SVSOVonNeumann::mts` are in the same position —
`?$VectorHelper@U?$StreamableEnum@I@Mars@@@Mars`, i.e. `VectorHelper<StreamableEnum<uint>>`, typed
as int arrays, count 0 everywhere.

## 4. `CD` — NOT closed, and here is the evidence that it cannot be, without a workload

`CD` is two different problems wearing one tag. `CDT` lists the ids in order and one `CD` frame
follows per id:

* **`Player.<id>.TurnCommands_v5`** → `Game::TurnCommands`. One block, 35 items on turn1.
* **`Player.<id>.AIAgent`** → `Game::StrategyAIAgent::Streamable`. Three blocks, ~236 items each,
  and these are the bulk of the 744.

### 4a. `TurnCommands`: the recovered sequence provably does not align to a no-orders save

`Game::TurnCommands` is recovered complete at **44 items, every tag `"."`**. The turn1 block has
**35**. Lane G called it a no-orders snapshot. That is right, and it is now stronger than a
description — the two sequences **cannot be aligned at all**, not even as a subsequence:

* Positionally they diverge at **item 4**: the save's item 4 is 8 bytes, which for a 1-character
  tag can only be a `bool` (4 + 1 + 1 → 8; an int is 4 + 1 + 4 → 12). The recovery says `i32`.
  Item 16 is a plain 12-byte int in the save where the recovery says a frame.
* As a subsequence it fails on counting. The save is `int, bool, float, bool, bool, bool, bool,
  bool` then **27 consecutive ints**. After the recovery's item 15 there are only **22** `i32`
  slots (indices 17–20, 22–27, 29–31, 33–40, 42); the rest are frames, a `narr`, and bools.
  27 > 22, so no order-preserving assignment exists.

So the writer takes branches that a no-orders turn does not, and with every tag `"."` there is no
name to bind against. **`TurnCommands` needs a save with issued orders** — which needs the VM.
Anything typed from here would be a guess dressed as a layout, so nothing was typed.

### 4b. `AIAgent`: statically tractable, deliberately not attempted

`Game::StrategyAIAgent::Streamable` is recovered at 36 items with **named** tags (`AIAttr`,
`AITurnPris`, `AISit`, `AIPlyHat`, `prs2`, `dsh`, `AIDNG`, `AISys`, `CmbR`, `apr`, …), so unlike
`TurnCommands` it *could* be bound by name. Two reasons it was left: the recovery is incomplete
(8 unresolved items) and roughly ten nested sub-objects would each need their own shape and their
own conformance binding. It is the single largest remaining block and a clean, self-contained job
for a following lane — but it is AI cache state, not orders, so it is not on the reimplementation
critical path the way 4a is.

### 4c. `RNG` (2) and `Attrib` (2)

`RNG` is correctly opaque — the raw MT19937 state block — and was left alone, as instructed.
`Attrib` is `StreamableHelper<Game::AttribMap>` and is an empty frame in every save; there is
nothing to type.

## 5. The four defects in `save_reader.py`, fixed openly

Lane G found four real defects in both readers, fixed the engine, and deliberately left the Python
oracle alone rather than quietly editing it mid-campaign. That judgement was right; this is the
follow-through. All four are now fixed in `verify/save-reader/save_reader.py`, each with a test,
and **all four are byte-neutral on all four saves**.

| # | defect | fix |
|---|---|---|
| 1 | `Game::SystemParams` field 1 is a **string**, not an int | `Planet`'s `p1` retyped |
| 2 | `ObservedTech`/`ObservedWeapon` `odet` is a **bool**, not an int | `Owep`/`Otch` retyped |
| 3 | `Game::SpeciesRatios` `nv` is a **count** | new `SpeciesRatios`/`CivilianRatios` shapes |
| 4 | `Game::ShipRecords` `srbd` is a **count** | new `ShipRecords` shape |

A wrinkle worth recording: for 3 and 4 the reader did not have the wrong layout, it had **no**
layout — `ShipRecs` and `civr` were both `A(..., "any")`. So the "defect present in both readers"
was, on the Python side, an absence rather than an error. Fixing it meant *adding* the typed shapes
with the count semantics spelled out, which is a bigger change than a retype and is why the
byte-neutrality evidence below matters more than usual.

### The byte-neutrality evidence

Every item's inflated offset is unchanged. Dumping all four saves before and after and comparing
the offset column:

```
turn1-state:          offset sequence IDENTICAL (38,933 items)
turn2-state:          offset sequence IDENTICAL (39,843 items)
turn3-state:          offset sequence IDENTICAL (40,300 items)
zuul-turn5-species5:  offset sequence IDENTICAL (35,771 items)
```

so no item boundary moved; only type labels changed. `state_checksum.py` still reports
**`coverage: PROVED`** on all four (591,376 / 603,360 / 609,080 / 532,752 bytes rebuilt == inflated,
unchanged), with `0 error, 0 warn`.

The subsystem digests for `/CreateParams` and `/Sim/players` do change, and **that is the point** —
they are digests of *typed values*, and two of the fixes change what a value is. The value-byte
totals account for the change exactly:

| save | Δ value bytes | `odet` items × 3 (int 4 B → bool 1 B) | `p1` items × 4 (int 4 B → "" 0 B) |
|---|---|---|---|
| turn1 | −292 | 60 × 3 = 180 | 28 × 4 = 112 |
| turn2 | −295 | 61 × 3 = 183 | 28 × 4 = 112 |
| turn3 | −295 | 61 × 3 = 183 | 28 × 4 = 112 |
| zuul | −238 | 42 × 3 = 126 | 28 × 4 = 112 |

Every row balances to the byte. Nothing else in the digest moved.

`--strict` exits 0 on all four saves. Tests went **36 → 48**, all passing.

### One finding that fell out of the fix

`srbd` is not merely inferred from the recovery — the saves **exercise** it. It takes the values
0, 1, 3 and 4 across the players of the four saves, and every non-zero count is followed by exactly
`srbd × 5` scalars (turn3 player 4: `srbd == 4`, then 20 items in four `srd/src/srb/srl/sri`
groups). Read as a field, those trailing scalars have no explanation at all; the new test asserts
exactly that, and the field reading raises. So defect 4 is behaviourally confirmed, not a
hypothesis. Defect 3 is confirmed the same way but more weakly: `nv` is only ever 0 or 1 in these
saves, so the count reading is confirmed for one pair and the multi-pair case is synthetic.

### The tests

`test_save_reader.py::WireSchemaDefectsTest`, 12 tests. Each defect gets the case the real saves
happen not to contain (a **named** system, a **bool-valued** `odet`, **two** species pairs, a
**non-empty** design-record section), plus a test showing *why* it stayed invisible — an empty
string and the int 0 are the same four bytes; a bool item and an int item are the same 12 bytes
only because `odet` is 4 characters long, and the same test shows a 3-character tag would not have
been forgiving.

One nuance the tests made explicit and that belongs on the record: in `save_reader.py` the generic
walker types `"."` items from its **own kind catalog**, not from the schema passed in. On the real
saves the catalog said int for `p1` and the old schema said int, so the two agreed with each other
and were both wrong — the failure mode there is a silent agreement, not the framing desync the
schema-driven engine would have suffered.

## 6. Artifacts

* `sots-engine` `wip/wire`: `src/mars/stream/shapes.h` (18 new SVSO shapes, `DOpts` →
  `vector<string>`, `spies2` → `vector<int32_t>`, `SvSctOb` → `opt_obj<ScriptObjects>`),
  `src/mars/stream/archive.h` (`std::string` element branch in `read_elem` / `write_elem` /
  `SchemaBuilder::carr`), `tests/mars_stream/test_wire_schema.cpp` (+18 bindings),
  `tests/mars_stream/test_stream.cpp` (+2 unit tests, no saves needed),
  `tests/mars_stream/test_save.cpp` (ratchet 95.0 → 97.5).
* `sots-re`: `verify/save-reader/save_reader.py` + `test_save_reader.py`,
  `ghidra/addresses.d/lane-w.json` (13 entries), `findings/objects/svsctob-variants.md`, this note.
* `include/generated/sots_stream_schema.h` was **not** regenerated: `tools/streams.py` and
  `tools/gen_stream_schema.py` were re-run and the output is byte-identical to the committed header
  apart from the provenance line, because nothing on the notes side changed the recovery.
* `ghidra/generated/sots_addresses.h` was **not** regenerated either. Lane W's addresses are in the
  fragment; `gen_addresses.py` merges to 640 entries with no duplicate name (checked against a
  scratch path). Regenerating in place would have swept lane U's in-flight `lane-u.json` into this
  lane's branch, and the engine consumes none of these addresses.

## 7. Open items

1. **A save with issued turn commands** remains the blocker for `TurnCommands_v5` (§4a), and it is
   now a proven blocker rather than a suspected one.
2. `Game::StrategyAIAgent::Streamable` (§4b) — the largest remaining block, bindable by name,
   ~10 nested shapes, no VM needed.
3. The four unexercised `EncID`s (7, 8, 14, 21) and `ScnObj` stay carried until a save or a
   scenario game produces one.
4. `serializers.py` should reclassify `Game::SVSOCrowDefenders`'s `dsys` as a loop element of
   `ndsys` (see `svsctob-variants.md` §3); `objects/streams.json` still carries the old reading and
   the conformance check does not notice, because it aligns by tag and disk primitive.
5. `spies2`, `SysMem`, `mts`, `trev`, `Ojvs`, `aid`, `Attrib`, `comms.nmsg`, `AIEnf.Nas`,
   `spymgr.nspy` and `spy2.rtc/evc/ttc` are 0 in every save. Their containers are typed; their
   element behaviour is not verified by anything.