f9b744e2ed
findings/objects/turncommands-block.md -- why lane W's 44-item recovery and the
save's 35/38/41/61/123 items disagreed, in mechanism terms, plus the per-list
element records read off all 22 distinct helper writers.
Three separate causes, only one of which was a branch:
* layouts.md is OFFSET-SORTED and Game::TurnCommands is one of the 89 classes
whose offset order is not write order, because each gate bool sits after the
payload it gates. Align saves against streams.json, never layouts.md.
* the writer's tail is 27 std::list members, each written by its own helper as
WriteInt(size) then size element records. The linear recovery keeps one item
per CALL SITE, guesses its kind from an element field and drops the count.
44 - 17 members = 27 = the list count.
* 8 prologue items + 27 zero counts = the 35-item empty block, which is why
every earlier save was bit-identical here.
Corrects lane O's section 11 in place: the fleet-move element is
{fleetId, nHops, nHops x systemId} -- a counted route vector, not a fixed
{fleetId, 1, destSystemId, 0}. Only the three-item reading makes the item totals
close (a four-item element needs 26 lists on human-turn2 and 28 on zuul-turn15;
the writer has 27). Also records that neither noderoute save carries a fleet
move, so the node-route UI does not queue through that list.
22 of the 27 lists are labelled hypotheses: read off the instruction stream, but
no save exercises them and no element value has ever been observed. List 14 is
observed but not understood -- issuing one fleet move also queues
{sameFleetId, 0, true} there.
save_reader.py deliberately left alone, with the reasoning written down: it reads
CD generically for the AIAgent bodies too, state_checksum's digest tree is built
from the generic tree so typing CD would add no named leaf to any diff, dispatch
by CDT ordinal would need new machinery in the oracle, and rule 8 is pointed at
exactly this move -- a layout mirrored into both readers from one reading is not
two checks. coverage: PROVED, 0 error 0 warn on all eleven saves, unchanged.
ghidra/addresses.d/lane-q.json: 6 entries (the writer, its Read pair, and the
four observed list helpers). Validated by generating to a scratch path; the
tracked header was not regenerated.
315 lines
17 KiB
Markdown
315 lines
17 KiB
Markdown
# `TurnCommands_v5` — the pending order queue, and why the recovery could not be aligned to it
|
||
|
||
Lane Q, 2026-09-08. Static + host only; VM140 was not needed.
|
||
|
||
Closes the last opaque block of the save format. Named coverage on all eleven saves goes to
|
||
**100.0%**, with the two items of the MT19937 blob the only things left carried as Nodes.
|
||
|
||
---
|
||
|
||
## 1. The reconciliation
|
||
|
||
Lane W's negative result was real work and its arithmetic was right. Its conclusion — "a save with
|
||
issued orders is required" — was also right. But the *reason* the two sequences would not align is
|
||
not that the writer takes branches a no-orders turn does not (it does, but that is the smaller
|
||
half). It is that **three different things were being compared as if they were one**.
|
||
|
||
`Game::TurnCommands::Write` is at `0x00842540` and is 764 bytes. Read as instructions, it has two
|
||
halves and no loops of its own:
|
||
|
||
**(1) A prologue of six flag-gated groups.** Each group is a `WriteBool` on a member, followed —
|
||
only when that bool is set — by that command's payload:
|
||
|
||
| gate | payload | meaning |
|
||
|---|---|---|
|
||
| — | `i32 @0x04` | player id, always written |
|
||
| `bool @0x0c` | `f32 @0x08` | research rate (the empire savings/research slider) |
|
||
| `bool @0x14` | `i32 @0x10` | research target tech id |
|
||
| `bool @0x20` | `i32 @0x18`, `f32 @0x1c` | research boost: savings spent, and a fraction |
|
||
| `bool @0x2c` | `bool @0x24`, `i32 @0x28` | never observed set |
|
||
| `bool @0x3c` | `f32 @0x30`, `f32 @0x34`, `f32 @0x38` | never observed set |
|
||
| `bool @0x6c` | `StreamableHelper<Game::CivilianRatios>` frame on `@0x40` | never observed set |
|
||
|
||
**(2) Twenty-seven `std::list<T>` members** at `0x70 … 0x1a8`, stride `0x0c`
|
||
(`{_Myhead, _Mysize, _Alval}` — allocator-last, as rule 5 warns). Each is handed to its own
|
||
free-function writer, `helper(stream, &list)`, and every helper has the same shape:
|
||
|
||
```
|
||
WriteInt(list._Mysize) ; the count -- ALWAYS written, even for an empty list
|
||
for each node: <that list's element record>
|
||
```
|
||
|
||
That is the whole class. Now the three ways the comparison went wrong:
|
||
|
||
### 1a. The sequence lane W aligned against was offset-sorted, not write-ordered
|
||
|
||
`objects/layouts.md` sorts fields by `off_abs`. `Game::TurnCommands` is one of the **89 of 386
|
||
classes** where offset order is not write order — the writer emits `0x04, 0x0c, 0x08, 0x14, 0x10,
|
||
0x20, 0x18, 0x1c, …` because each gate bool precedes its payload in the stream while sitting after
|
||
it in the struct. In the sorted view, item 4 is the `i32 @0x10`; in the writer, item 4 is the
|
||
`bool @0x14`. Lane W's "the save's item 4 is 8 bytes, which can only be a bool where the recovery
|
||
says `i32`" is therefore an artifact of the sorted view, not a contradiction. The **write-ordered**
|
||
table (`objects/streams.json`, which is what the engine's wire-schema channel consumes) already has
|
||
a `bool` in that slot, and matches the save.
|
||
|
||
This is the same trap the wire-schema channel was built to avoid, hit from the other side. It is
|
||
worth restating as an operational rule: **never align a save against `layouts.md`; align against
|
||
`streams.json`.**
|
||
|
||
### 1b. The tail is 27 container *call sites*, not 27 scalars
|
||
|
||
The recovery is a linear pass. It cannot express a loop, so for each of the 27 list writers it
|
||
emits **one item**, guessing a kind from an element field it could resolve, and **drops the count
|
||
word entirely**. That is where the recovered 44 comes from:
|
||
|
||
```
|
||
44 = 17 member writes (all branches, as the recovery always presents them)
|
||
+ 27 container call sites (one item each, element kind guessed, count dropped)
|
||
```
|
||
|
||
`44 − 17 = 27`, and there are exactly 27 list members. Lane W's second objection — "27 consecutive
|
||
ints on disk but only 22 `i32` slots after index 15" — is that same 27 seen from both sides. The
|
||
five slots that are not `i32` in the table (a `Game::StarSystem::OutputRates` frame, a `narr`, two
|
||
`bool`s, a `Game::Population` frame, a `Game::RaidTargets` frame) are the recovery describing an
|
||
**element**, while the wire at that position always shows a **count**. It was a kind mismatch in
|
||
the table, not a structural impossibility in the save.
|
||
|
||
Two of those five are exactly right about which list they name, which is the tell: the table's
|
||
5th tail item is `OutputRates` and list 5 *is* the planetary-budget list; its 8th tail item is
|
||
`narr` and list 8 *is* the only one whose element ends in a nested counted vector.
|
||
|
||
### 1c. The empty block is 8 + 27, and that is why every old save was bit-identical
|
||
|
||
A no-orders turn sets only the research-rate gate. So the block is
|
||
|
||
```
|
||
i32 playerId, bool(true), f32 rate, bool×5 (all false) = 8 items
|
||
27 × WriteInt(0) = 27 items
|
||
-- 35 items, 122 B
|
||
```
|
||
|
||
which is `CD[0]` digest `3df7d93164fb1d7d` in `turn1/2/3-state.sav`, `zuul-turn5-species5.sav`,
|
||
`human-turn3-noderoute.sav`, `zuul-turn17-rollpending.sav` and `zuul-turn23-fleet23.sav`. Nothing
|
||
could be learned from those saves because 27 of their 35 items are the *absence* of every command
|
||
list.
|
||
|
||
### 1d. The arithmetic checks on every save
|
||
|
||
| save | prologue | list contents | total | observed |
|
||
|---|---|---|---|---|
|
||
| any no-orders save | 8 | 27 counts | 35 | 35 |
|
||
| `zuul-turn16-noderoute` | 11 | 27 counts | 38 | 38 |
|
||
| `human-turn2-orders` | 8 | 27 + 3 (move) + 3 (list 14) | 41 | 41 |
|
||
| `zuul-turn15-orders` | 11 | 27 + 5×4 (build) + 3 (move) | 61 | 61 |
|
||
| `zuul-turn17-orders2` | 8 | 27 + 20×4 (build) + 2 (rates) + 3×2 (colonize) | 123 | 123 |
|
||
|
||
(`state_checksum --tree` counts 129 *leaves* for the last one because the `OutputRates` sub-frame
|
||
contributes 7 leaves in place of 1 item.)
|
||
|
||
---
|
||
|
||
## 2. Corrections to lane O's provisional layout
|
||
|
||
Lane O's §11 was derived from bytes and UI correlation and is right about every command it
|
||
observed. Two things are corrected from the writer:
|
||
|
||
**The fleet-move element is not `{fleetId, 1, destSystemId, 0}`.** The helper at `0x0083e550`
|
||
writes `WriteInt(fleetId)` and then a **nested counted int vector**: `n = (v._Mylast −
|
||
v._Myfirst) >> 2`, then `n` ints. So the wire is `{fleetId, hopCount, hopCount × systemId}` — three
|
||
items for a one-hop order, and *longer for a multi-hop route*. Lane O read the `1` as a constant
|
||
and the next list's zero count as a trailing `0`. Only the three-item reading makes the item totals
|
||
close: with a four-item element `human-turn2-orders` needs 26 lists and `zuul-turn15-orders` needs
|
||
28, and the writer has 27.
|
||
|
||
This also means a route order is directly visible in the block, which is what the `noderoute` saves
|
||
were built to catch — and worth recording, neither of them carries one (both decode to zero fleet
|
||
moves), so the node-route UI does not queue through this list.
|
||
|
||
**There are 27 lists, not "five plus trailing zeros".** Lane O's "int x2 = 0" before the build
|
||
count and its "int 0" separators are lists 1, 2, 4, 6 and 9…27 with count 0. The five observed are
|
||
list 3 (build), 5 (system rates), 7 (colonize), 8 (fleet move) and 14.
|
||
|
||
`verify/save-reader/SAVE_FORMAT.md` §11 has been corrected in place.
|
||
|
||
---
|
||
|
||
## 3. The 27 lists
|
||
|
||
Element records read off each helper. RTTI resolves every nested frame's helper vftable to a class,
|
||
which is where the names come from.
|
||
|
||
| # | member | helper | element record | status |
|
||
|---|---|---|---|---|
|
||
| 1 | `+0x70` | `0x0082e310` | `StreamableHelper<ShipDesignDef>` frame, `i32` | hypothesis |
|
||
| 2 | `+0x7c` | `0x00822ca0` | `i32` | hypothesis |
|
||
| 3 | `+0x88` | `0x00822870` | `i32 ordinal, i32 designId, i32 systemId, i32 w` | **observed** |
|
||
| 4 | `+0x94` | `0x008228f0` | `i32 ×3` | hypothesis |
|
||
| 5 | `+0xa0` | `0x0082e3d0` | `i32 systemId`, `StarSystem::OutputRates` frame | **observed** |
|
||
| 6 | `+0xac` | `0x0082e490` | `PlayerNotes` frame | hypothesis |
|
||
| 7 | `+0xb8` | `0x00822960` | `i32 shipId, i32 w` | **observed** |
|
||
| 8 | `+0xc4` | `0x0083e550` | `i32 fleetId`, counted `i32` route | **observed** |
|
||
| 9 | `+0xd0` | `0x008229c0` | `i32, bool, i32, i32, f32` | hypothesis |
|
||
| 10 | `+0xdc` | `0x0083e5f0` | `i32, i32`, counted `i32` | hypothesis |
|
||
| 11 | `+0xe8` | `0x00822a50` | `i32, bool` | hypothesis |
|
||
| 12 | `+0xf4` | `0x0082e530` | `i32`, `FleetLayout` frame | hypothesis |
|
||
| 13 | `+0x100` | `0x00822ab0` | `i32`, `string` | hypothesis |
|
||
| 14 | `+0x10c` | `0x00822b10` | `i32, i32, bool` | **observed** (see below) |
|
||
| 15 | `+0x118` | `0x00822a50` | `i32, bool` | hypothesis |
|
||
| 16 | `+0x124` | `0x00822a50` | `i32, bool` | hypothesis |
|
||
| 17 | `+0x130` | `0x00822b80` | `i32 ×3` | hypothesis |
|
||
| 18 | `+0x13c` | `0x00822b80` | `i32 ×3` | hypothesis |
|
||
| 19 | `+0x148` | `0x00822bf0` | `i32` | hypothesis |
|
||
| 20 | `+0x154` | `0x00822bf0` | `i32` | hypothesis |
|
||
| 21 | `+0x160` | `0x0082e5f0` | `i32, i32`, `VectorHelper<StreamableEnum<uint>>` | hypothesis |
|
||
| 22 | `+0x16c` | `0x0082e6c0` | `i32`, `WeaponGroups` frame | hypothesis |
|
||
| 23 | `+0x178` | `0x0082e780` | `i32`, `Population` frame | hypothesis |
|
||
| 24 | `+0x184` | `0x00822c40` | `i32`, `f32` | hypothesis |
|
||
| 25 | `+0x190` | `0x0082e840` | `i32`, `DefenceLayout` frame | hypothesis |
|
||
| 26 | `+0x19c` | `0x0082e900` | `RaidTargets` frame | hypothesis |
|
||
| 27 | `+0x1a8` | `0x00822ca0` | `i32` | hypothesis |
|
||
|
||
Six member slots share a helper with an earlier slot (15/16 with 11, 18 with 17, 20 with 19, 27
|
||
with 2), so there are 22 distinct helper functions for 27 call sites.
|
||
|
||
**Rule 6 applies to twenty-two of these rows.** The scalar sequences are read directly out of the
|
||
instruction stream, which is stronger than a guess, but no save exercises them and no element value
|
||
has ever been observed. Where an element holds a nested object whose body is not otherwise modelled
|
||
by the engine's shapes (lists 1, 12, 22, 25, 26), the frame is **carried as an opaque Node** rather
|
||
than decoded — a wrong body cannot then desynchronise a reader if one of those lists ever turns up
|
||
populated.
|
||
|
||
**List 14 is observed but not understood.** `human-turn2-orders.sav` — a turn whose only UI action
|
||
was one fleet move — carries one list-14 element `{1456, 0, true}`, and `1456` is the same fleet id
|
||
list 8 carries. So issuing a move queues a second, separate command against the same fleet. What it
|
||
is remains open; it is named `list14` in the engine rather than guessed at.
|
||
|
||
**What lane O's saves could not produce**, and therefore what stays a hypothesis: the three unset
|
||
prologue gates (`@0x2c`, `@0x3c`, `@0x6c` — the last being a whole `CivilianRatios` frame, i.e. the
|
||
empire civilian-settings command), and lists 1, 2, 4, 6, 9–13 and 15–27.
|
||
|
||
---
|
||
|
||
## 4. Is the block version-tagged or otherwise variable?
|
||
|
||
No version word is written. The id string carries the version instead
|
||
(`Player.<id>.TurnCommands_v5`), and the shape is fixed: 27 lists always, six gates always. All the
|
||
variability is the gates and the list lengths, so a single static shape is correct — with one
|
||
guard. The engine's `CustomDataBlock::select()` matches the **exact** suffix `.TurnCommands_v5`;
|
||
anything else, including a hypothetical `_v6`, falls back to the carried Node rather than being
|
||
decoded with a stale layout. There is a unit test for that.
|
||
|
||
---
|
||
|
||
## 5. Conformance, and why the generic check could not be used
|
||
|
||
Every item in the class is written with a NULL name, so on disk every tag is `.`. The wire-schema
|
||
test's LCS degenerates on such a sequence: with all tags equal, the gap branches are unreachable and
|
||
the walk becomes a strict positional comparison in which any primitive disagreement is a fatal
|
||
`MISMATCH`. That is correct behaviour when the two sequences describe the same thing — and here they
|
||
do not, for the reason in §1b. Binding the shape with the generic `check<T>()` would have failed the
|
||
build, and "make it pass" would have meant deforming the shape to match a table that is wrong.
|
||
|
||
So `test_wire_schema.cpp` gets a dedicated check that states what is actually checkable:
|
||
|
||
* **the 17-item prologue, item for item and primitive for primitive** — `17/17`, zero mismatches.
|
||
`SchemaProbe` takes every branch, which is the same all-branches view the recovery has, so the
|
||
conditional structure read out of the instruction stream must reproduce the recovered sequence
|
||
exactly. It does. That is the independent evidence for §1a, and it is not a weak test: get any
|
||
gate/payload pairing wrong and the primitives desynchronise immediately.
|
||
* **the tail count** — the shape has exactly 27 top-level counted lists and the table has exactly
|
||
27 tail items. `27 == 27` is the entire content of that half of the table.
|
||
|
||
The 27 tail items are reported as `wire-only`, never claimed as matched.
|
||
|
||
Totals: **87 shapes bound, 856 items matched, 0 MISMATCH** (was 86 / 838 / 0).
|
||
|
||
---
|
||
|
||
## 6. A second defect this work turned up: `NVs` element tags
|
||
|
||
Not `TurnCommands`, but found by running the whole corpus and worth reporting loudly.
|
||
|
||
`zuul-turn23-fleet23.sav` is the **first save in the campaign with a non-empty `NVs` list** on a
|
||
`ServerSystem`. Its element's leading id had been typed positionally as `R("pid")` — disk tag `.` —
|
||
on the assumption every anonymous element carries `.`. The real tag is `PID`, so the engine's typed
|
||
round trip on that save differed at offset `0x89c14`, at the very first element. This was already
|
||
broken at engine `main` (`c883a32`), before any of lane Q's changes.
|
||
|
||
Rule 6 exactly: a path no save exercised was a hypothesis flying as a fact, and the first workload
|
||
that touched it falsified it. Fixed to `A("PID")`. The fix is independently corroborated — the
|
||
recovered wire table for `Game::ServerSystem` names that item `PID`, and the conformance row moves
|
||
from `102 matched / 1 wire-only / 3 shape-only` to `103 / 0 / 2`.
|
||
|
||
---
|
||
|
||
## 7. Coverage, before and after
|
||
|
||
`CoverageArchive`'s typed-vs-carried split (not round-trip success), all eleven saves:
|
||
|
||
| save | before | after | opaque before → after |
|
||
|---|---|---|---|
|
||
| `turn1-state` | 99.9% | **100.0%** | 37 → 2 |
|
||
| `turn2-state` | 99.9% | **100.0%** | 37 → 2 |
|
||
| `turn3-state` | 99.9% | **100.0%** | 37 → 2 |
|
||
| `zuul-turn5-species5` | 99.9% | **100.0%** | 37 → 2 |
|
||
| `human-turn2-orders` | 99.9% | **100.0%** | 43 → 2 |
|
||
| `human-turn3-noderoute` | 99.9% | **100.0%** | 37 → 2 |
|
||
| `zuul-turn15-orders` | 99.8% | **100.0%** | 63 → 2 |
|
||
| `zuul-turn16-noderoute` | 99.9% | **100.0%** | 40 → 2 |
|
||
| `zuul-turn17-orders2` | 99.7% | **100.0%** | 132 → 2 |
|
||
| `zuul-turn17-rollpending` | 99.9% | **100.0%** | 37 → 2 |
|
||
| `zuul-turn23-fleet23` | 99.9% | **100.0%** | 37 → 2 |
|
||
|
||
The two remaining opaque items on every save are the `RNG` MT19937 block, which is deliberately
|
||
carried. **Nothing else in any save we hold is untyped.** The ratchet moves 99.8 → 99.99.
|
||
|
||
Note `zuul-turn17-orders2` was *below* the old 99.8 ratchet: lane O's largest order save had already
|
||
pushed the opaque count high enough to fail the coverage gate, which is the gate working.
|
||
|
||
Round trip stayed byte-identical on all eleven saves before and after (and became byte-identical on
|
||
`zuul-turn23-fleet23`, where it was not).
|
||
|
||
---
|
||
|
||
## 8. `save_reader.py` was deliberately left alone, and why
|
||
|
||
The Python oracle reads `CD` frames as `Repeat(A("CD", "any"))` — it types **neither** custom-data
|
||
body, `AIAgent` included. Lane A judged that an absence rather than a defect and did not touch it.
|
||
That judgement still holds, and the case for mirroring is weaker than it looks:
|
||
|
||
* the block is already **fully accounted for** by the oracle's generic walk —
|
||
`state_checksum --strict` reports `coverage: PROVED`, `0 error 0 warn`, on all eleven saves,
|
||
before and after;
|
||
* `state_checksum`'s digest tree is built from the **generic** node tree, not the typed dict, so
|
||
typing `CD` would not put a single named leaf into a diff;
|
||
* nothing consumes the typed dict for `CD`;
|
||
* the dispatch would need new machinery in the oracle (selection by `CDT` ordinal, which the schema
|
||
DSL has no construct for) — new risk in the instrument the campaign trusts most;
|
||
* and rule 8 is pointed at exactly this move. A layout mirrored into both readers from one reading
|
||
does not become two checks; its failure mode is silent agreement.
|
||
|
||
So the independent evidence for this layout is **not** "two readers agree". It is the disassembly,
|
||
the 17/17 prologue conformance against a table produced by a different tool, and the item arithmetic
|
||
in §1d closing to the unit on five distinct workloads.
|
||
|
||
What was added instead, where real saves are actually available: `test_save.cpp` now asserts, on
|
||
every save, that each `TurnCommands_v5` block is consumed by the prologue plus the 27 lists with
|
||
**nothing left over** (`extra.empty()`). That is the item-granular statement, and a wrong list count
|
||
or a wrong element width breaks it first.
|
||
|
||
---
|
||
|
||
## 9. Gates
|
||
|
||
Run as separate commands, as rule 13 requires.
|
||
|
||
* `tools/clean_room_check.sh` — **OK**
|
||
* host `ctest` — **36/36**
|
||
* `test_save` with `SOTS_SAVES_DIR` — **11 saves, 0 failures**, round trip byte-identical on all
|
||
* `state_checksum.py --strict` — `coverage: PROVED`, `0 error 0 warn`, on all eleven saves
|
||
* `test_wire_schema` — 87 shapes, 856 items matched, **0 MISMATCH**
|
||
|
||
`src/shim/` was not touched, so no CT111 cross-build was needed.
|
||
|
||
New addresses are in `ghidra/addresses.d/lane-q.json` (6 entries). The fragment was validated by
|
||
generating to a scratch path, never over the tracked header.
|