2308b6edb9
Part 1 (notes side): findings/objects/svsctob-variants.md records the two maps
that are nowhere on the wire and were read out of the game -- EncID -> class
from a 23-entry dword jump table at 0x0052bf60 indexed by EncID-1 (0x0052bf00),
and xscn -> class from an exhaustive four-way _stricmp chain at 0x005a7050.
Twelve live EncIDs, four scenario names, and "indsys" =
Game::SVSOIndependentSystems whose Read and Write are both the shared `ret 4`
stub at 0x005f8ac0, so its empty frame is correct output rather than a
truncation. Also: SVSOSots::Read accepts NPCPlr and hastraps, which Write never
emits -- read-only backward compatibility, not a hole in the recovery. And a
correction to the recovery itself: SVSOCrowDefenders writes `dsys` INSIDE the
ndsys loop; layouts.json calls it a plain member and no save can settle it
because both counts are 0 everywhere. 13 addresses in ghidra/addresses.d/
lane-w.json; gen_addresses.py merges to 640 with no duplicate name.
Part 2: lane G found four defects in both readers and deliberately did not
patch the oracle mid-campaign. Fixed now, with tests, and byte-neutral.
1. Game::SystemParams field 1 is a string, not an int (empty string == four
zero bytes == int 0, so it round-tripped by luck).
2. ObservedTech/ObservedWeapon odet is a bool, not an int (byte-safe only
because a 4-char tag makes both items 12 bytes; 3 chars would not).
3. SpeciesRatios nv is a count, not a field.
4. ShipRecords srbd is a count, not a field -- and this one is behaviourally
confirmed, not inferred: srbd takes 0, 1, 3 and 4 across the players and
every non-zero count is followed by exactly srbd x 5 scalars.
Note that 3 and 4 were an ABSENCE in save_reader.py, not an error: ShipRecs and
civr were both A(..., "any"), so the fix had to add the shapes rather than
retype a field.
Byte-neutrality: every item's inflated offset is unchanged on all four saves
(38,933 / 39,843 / 40,300 / 35,771 offsets, sequences identical), so no item
boundary moved. state_checksum.py still reports coverage: PROVED on all four
with the same rebuilt byte counts. The /CreateParams and /Sim/players digests do
change, because they hash typed VALUES and two fixes change what a value is --
and the value-byte deltas balance exactly: odet items x 3 plus p1 items x 4.
--strict exit 0 on all four saves; tests 36 -> 48.
findings/objects/wire-schema-closeout.md carries the whole account, including
the proof that CD/TurnCommands_v5 cannot be typed without a save that has
issued orders.
13 KiB
Closing the opaque sections, and correcting the Python oracle
Lane W, 2026-09-08. Host/static only — VM140 was held by lane U and the game was never run.
Continues findings/objects/wire-schema-channel.md (lane G).
Headline: named coverage 97.1 % → 98.0 %
The honest metric is CoverageArchive's split of items a field names from items a generic
Node merely carries — not round-trip success, which ar.any bodies achieve trivially by
copying bytes nobody understands. The byte-identical round trip held throughout.
| save | before | after | still opaque after |
|---|---|---|---|
| turn1-state | 97.1 % | 98.0 % | CD 744, RNG 2, Attrib 2 |
| turn2-state | 97.2 % | 98.0 % | CD 744, RNG 2, Attrib 2 |
| turn3-state | 97.2 % | 98.0 % | CD 744, RNG 2, Attrib 2 |
| zuul-turn5-species5 | 97.6 % | 98.4 % | CD 517, RNG 2, Attrib 2 |
Conformance: 74 shapes bound, 769 items matched, 0 MISMATCH (was 56 / 657 / 0). The ratchet in
test_save.cpp moved 95.0 → 97.5. ctest 34/34; tools/clean_room_check.sh OK; test_save
still skips cleanly with SOTS_SAVES_DIR unset.
CD is now the only remaining region of any size, and §4 says why it cannot be closed here.
1. SvSctOb — closed, 147 items → 0
Its own note: findings/objects/svsctob-variants.md. Summary: the body is Game::SVSOSots, which
writes two variant lists dispatched by a preceding key (xscn for xsc, EncID for EncObj).
Neither map is on the wire; both were read out of the game — the ids from a 23-entry dword jump
table at 0x0052bf60 indexed by EncID - 1, the names from an exhaustive four-way _stricmp chain
at 0x005a7050. Eighteen shapes bound, all clean. The four factory ids no save exercises
(7 SystemKiller, 8 PuppetMaster, 14 Locust, 21 Ortgay) are deliberately left carried, not typed.
2. DOpts — closed, 94 items → 0. Certain, from the helper's own decorated name.
Game::ShipDesignDef::Section's DOpts is ?$VectorHelper@VString@Mars@@@Mars — a
Mars::VectorHelper<Mars::String>. The element type is not inferred from the bytes; it is in the
mangled helper type the writer instantiated. The engine's read_elem/write_elem grew a
std::string branch (the one thing lane G listed as missing) and carr<std::string> now works.
The same branch closed Game::SVSOVonNeumann::trev, also VectorHelper<Mars::String>.
Note the empty-string trap this had to survive: an empty string is four zero bytes, so an empty
DOpts element is byte-identical to the int 0. The new unit test writes
{"OPT_Armour", "", "OPT_Shield"} specifically to exercise it.
3. spies2 — closed, 56 items → 0, but read this before believing the element layout
Game::ServerSystem's spies2 is ?$VectorHelper@H@Mars — Mars::VectorHelper<int>. So it is a
framed array of NULL-named ints, and that is a certainty about the type, from the same
decorated-name evidence as DOpts, not a guess from the bytes.
What is not exercised is any element. The count is 0 in all 28 systems of all four saves, so
the 56 items closed here are 28 × (the frame + its . count) and no spies2 element value has
ever been observed on disk. If a future save carries spies, the shape says they are plain ints;
that claim rests entirely on VectorHelper<int> and would be falsified by the first non-empty one.
Labelling it: the type is certain, the behaviour is a hypothesis.
Game::SVSOSwarmQueen::SysMem and Game::SVSOVonNeumann::mts are in the same position —
?$VectorHelper@U?$StreamableEnum@I@Mars@@@Mars, i.e. VectorHelper<StreamableEnum<uint>>, typed
as int arrays, count 0 everywhere.
4. CD — NOT closed, and here is the evidence that it cannot be, without a workload
CD is two different problems wearing one tag. CDT lists the ids in order and one CD frame
follows per id:
Player.<id>.TurnCommands_v5→Game::TurnCommands. One block, 35 items on turn1.Player.<id>.AIAgent→Game::StrategyAIAgent::Streamable. Three blocks, ~236 items each, and these are the bulk of the 744.
4a. TurnCommands: the recovered sequence provably does not align to a no-orders save
Game::TurnCommands is recovered complete at 44 items, every tag ".". The turn1 block has
35. Lane G called it a no-orders snapshot. That is right, and it is now stronger than a
description — the two sequences cannot be aligned at all, not even as a subsequence:
- Positionally they diverge at item 4: the save's item 4 is 8 bytes, which for a 1-character
tag can only be a
bool(4 + 1 + 1 → 8; an int is 4 + 1 + 4 → 12). The recovery saysi32. Item 16 is a plain 12-byte int in the save where the recovery says a frame. - As a subsequence it fails on counting. The save is
int, bool, float, bool, bool, bool, bool, boolthen 27 consecutive ints. After the recovery's item 15 there are only 22i32slots (indices 17–20, 22–27, 29–31, 33–40, 42); the rest are frames, anarr, and bools. 27 > 22, so no order-preserving assignment exists.
So the writer takes branches that a no-orders turn does not, and with every tag "." there is no
name to bind against. TurnCommands needs a save with issued orders — which needs the VM.
Anything typed from here would be a guess dressed as a layout, so nothing was typed.
4b. AIAgent: statically tractable, deliberately not attempted
Game::StrategyAIAgent::Streamable is recovered at 36 items with named tags (AIAttr,
AITurnPris, AISit, AIPlyHat, prs2, dsh, AIDNG, AISys, CmbR, apr, …), so unlike
TurnCommands it could be bound by name. Two reasons it was left: the recovery is incomplete
(8 unresolved items) and roughly ten nested sub-objects would each need their own shape and their
own conformance binding. It is the single largest remaining block and a clean, self-contained job
for a following lane — but it is AI cache state, not orders, so it is not on the reimplementation
critical path the way 4a is.
4c. RNG (2) and Attrib (2)
RNG is correctly opaque — the raw MT19937 state block — and was left alone, as instructed.
Attrib is StreamableHelper<Game::AttribMap> and is an empty frame in every save; there is
nothing to type.
5. The four defects in save_reader.py, fixed openly
Lane G found four real defects in both readers, fixed the engine, and deliberately left the Python
oracle alone rather than quietly editing it mid-campaign. That judgement was right; this is the
follow-through. All four are now fixed in verify/save-reader/save_reader.py, each with a test,
and all four are byte-neutral on all four saves.
| # | defect | fix |
|---|---|---|
| 1 | Game::SystemParams field 1 is a string, not an int |
Planet's p1 retyped |
| 2 | ObservedTech/ObservedWeapon odet is a bool, not an int |
Owep/Otch retyped |
| 3 | Game::SpeciesRatios nv is a count |
new SpeciesRatios/CivilianRatios shapes |
| 4 | Game::ShipRecords srbd is a count |
new ShipRecords shape |
A wrinkle worth recording: for 3 and 4 the reader did not have the wrong layout, it had no
layout — ShipRecs and civr were both A(..., "any"). So the "defect present in both readers"
was, on the Python side, an absence rather than an error. Fixing it meant adding the typed shapes
with the count semantics spelled out, which is a bigger change than a retype and is why the
byte-neutrality evidence below matters more than usual.
The byte-neutrality evidence
Every item's inflated offset is unchanged. Dumping all four saves before and after and comparing the offset column:
turn1-state: offset sequence IDENTICAL (38,933 items)
turn2-state: offset sequence IDENTICAL (39,843 items)
turn3-state: offset sequence IDENTICAL (40,300 items)
zuul-turn5-species5: offset sequence IDENTICAL (35,771 items)
so no item boundary moved; only type labels changed. state_checksum.py still reports
coverage: PROVED on all four (591,376 / 603,360 / 609,080 / 532,752 bytes rebuilt == inflated,
unchanged), with 0 error, 0 warn.
The subsystem digests for /CreateParams and /Sim/players do change, and that is the point —
they are digests of typed values, and two of the fixes change what a value is. The value-byte
totals account for the change exactly:
| save | Δ value bytes | odet items × 3 (int 4 B → bool 1 B) |
p1 items × 4 (int 4 B → "" 0 B) |
|---|---|---|---|
| turn1 | −292 | 60 × 3 = 180 | 28 × 4 = 112 |
| turn2 | −295 | 61 × 3 = 183 | 28 × 4 = 112 |
| turn3 | −295 | 61 × 3 = 183 | 28 × 4 = 112 |
| zuul | −238 | 42 × 3 = 126 | 28 × 4 = 112 |
Every row balances to the byte. Nothing else in the digest moved.
--strict exits 0 on all four saves. Tests went 36 → 48, all passing.
One finding that fell out of the fix
srbd is not merely inferred from the recovery — the saves exercise it. It takes the values
0, 1, 3 and 4 across the players of the four saves, and every non-zero count is followed by exactly
srbd × 5 scalars (turn3 player 4: srbd == 4, then 20 items in four srd/src/srb/srl/sri
groups). Read as a field, those trailing scalars have no explanation at all; the new test asserts
exactly that, and the field reading raises. So defect 4 is behaviourally confirmed, not a
hypothesis. Defect 3 is confirmed the same way but more weakly: nv is only ever 0 or 1 in these
saves, so the count reading is confirmed for one pair and the multi-pair case is synthetic.
The tests
test_save_reader.py::WireSchemaDefectsTest, 12 tests. Each defect gets the case the real saves
happen not to contain (a named system, a bool-valued odet, two species pairs, a
non-empty design-record section), plus a test showing why it stayed invisible — an empty
string and the int 0 are the same four bytes; a bool item and an int item are the same 12 bytes
only because odet is 4 characters long, and the same test shows a 3-character tag would not have
been forgiving.
One nuance the tests made explicit and that belongs on the record: in save_reader.py the generic
walker types "." items from its own kind catalog, not from the schema passed in. On the real
saves the catalog said int for p1 and the old schema said int, so the two agreed with each other
and were both wrong — the failure mode there is a silent agreement, not the framing desync the
schema-driven engine would have suffered.
6. Artifacts
sots-enginewip/wire:src/mars/stream/shapes.h(18 new SVSO shapes,DOpts→vector<string>,spies2→vector<int32_t>,SvSctOb→opt_obj<ScriptObjects>),src/mars/stream/archive.h(std::stringelement branch inread_elem/write_elem/SchemaBuilder::carr),tests/mars_stream/test_wire_schema.cpp(+18 bindings),tests/mars_stream/test_stream.cpp(+2 unit tests, no saves needed),tests/mars_stream/test_save.cpp(ratchet 95.0 → 97.5).sots-re:verify/save-reader/save_reader.py+test_save_reader.py,ghidra/addresses.d/lane-w.json(13 entries),findings/objects/svsctob-variants.md, this note.include/generated/sots_stream_schema.hwas not regenerated:tools/streams.pyandtools/gen_stream_schema.pywere re-run and the output is byte-identical to the committed header apart from the provenance line, because nothing on the notes side changed the recovery.ghidra/generated/sots_addresses.hwas not regenerated either. Lane W's addresses are in the fragment;gen_addresses.pymerges to 640 entries with no duplicate name (checked against a scratch path). Regenerating in place would have swept lane U's in-flightlane-u.jsoninto this lane's branch, and the engine consumes none of these addresses.
7. Open items
- A save with issued turn commands remains the blocker for
TurnCommands_v5(§4a), and it is now a proven blocker rather than a suspected one. Game::StrategyAIAgent::Streamable(§4b) — the largest remaining block, bindable by name, ~10 nested shapes, no VM needed.- The four unexercised
EncIDs (7, 8, 14, 21) andScnObjstay carried until a save or a scenario game produces one. serializers.pyshould reclassifyGame::SVSOCrowDefenders'sdsysas a loop element ofndsys(seesvsctob-variants.md§3);objects/streams.jsonstill carries the old reading and the conformance check does not notice, because it aligns by tag and disk primitive.spies2,SysMem,mts,trev,Ojvs,aid,Attrib,comms.nmsg,AIEnf.Nas,spymgr.nspyandspy2.rtc/evc/ttcare 0 in every save. Their containers are typed; their element behaviour is not verified by anything.