sots-re/findings/objects/ship-design-catalogue.md
alex a4aba6a9fb lane D2: the ship-design catalogue -- how designs persist, hull size, and the 0x400 flag
Blocker #4 on lane Y's path to a byte-matching turn.

HOW DESIGNS PERSIST, correcting a published finding. Game::ShipDesign::Write is
0x008325e0 and makes four stream calls. The recorded address 0x008747a0 is in
NO vftable at all (lane V2's inversion), so "ShipDesign::Write makes no stream
call" was a misattribution, not a fact about the class. Game::ShipDesign
derives from Game::ShipDesignDef and inherits IStreamable second, so its writer
is reached through an adjustor thunk -- which is what the slot-indexed
serializer sweep found instead. A design persists as two serializers, base and
derived: ShipDesignDef::Write 0x00827390 emits FAIDes/DHide/DWep/DName then
exactly three DSec frames (+0x4c command, +0x24 mission, +0x74 engine),
ShipDesign::Write appends Dtc, the Dwgv flag and a conditional Dwg frame.

THREE sections, not five. The campaign's "slots 3-4 reserved and always empty"
was save_reader.py's trailing Rest("sections") sweeping Dtc and Dwgv into the
section list, and stock_designs.py decoding them as two empty sections -- rule 8
in its exact form, reader and consumer agreeing with each other and both wrong.
Two independent enumerations say three: the writer's straight-line body, and the
ctor's eh_vector_constructor_iterator(this+0x24, 0x28, 3) closing at
0x9c = sizeof(ShipDesignDef).

DWep and Dwgv are BOOLs, not ints -- byte-indistinguishable from ints at a
four-character tag, the same class of defect as ObservedTech.odet.

THE 0x400 FLAG IS `defence_platform`, read off the .shipsection parser's own bit
setter at 0x005749b7. NOT lane B5's 0x400: that one is a fleet flag, on the wire
as FtFlg. The full role-flag table is in the finding. HULL SIZE is section_class
through a three-name stricmp table (Destroyer/Cruiser/Dreadnought -> 0/1/2),
absent or unrecognised meaning 0 with a log line rather than an error. Both
words are recomputed from the data files by ShipDesign::UpdateDerivedStats
0x0087e7c0 and neither is on the wire. Corroborated by the default hull-health
table the same bit picks: 500/3000/15000 without it, 100/500/1000 with.

MEASURED: the census rebuilt from each save's own state matches the record the
game archived, 480 leaves / 0 mismatched over 11 saves and 503 designs, computed
independently in Python and in C++. COVERAGE REPORTED AS LOUDLY: only 32 of the
480 leaves are nonzero, and three of the six census leaves (both cruiser rows,
dreadnought platforms) are unexercised by every save in the corpus.

Closed 0 / regressed 0 against the standalone's divergence list, reported
separately: the census leaves live in src/app's turn record, which lane A2 holds
this cycle, so this lane evaluated and reported rather than writing.

Oracles fixed openly (rule 12): save_reader.py's Des shape, 49/49 with three
corrected tests and one added that pins "exactly three DSec" against real saves;
stock_designs.json regenerated, whose diff is only raw_slots 5->3 and dWep
int->bool across all 127 designs with every other field identical;
test_design_rules.py still 32/32 with the same ground truth.

19 addresses in ghidra/addresses.d/lane-d2.json, no collision; the generated
header was validated to a scratch path, never written in place.
2026-09-08 12:45:36 -04:00

326 lines
18 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# The ship-design catalogue: how designs persist, hull size, and the `0x400` flag
Lane D2, 2026-09-08. Static reading against the exe, checked against all 11 saves.
Blocker #4 on lane Y's path to a byte-matching turn. Three lanes had brushed against the first
question and each left it open; this closes it, and the answer corrects a published finding.
---
## 1. How ship designs persist — the serializer three lanes could not find
**`Game::ShipDesign::Write` exists, is `0x008325e0`, and makes four stream calls.** The published
finding — "`Game::ShipDesign::Write` (`0x008747a0`) makes no stream call at all" — is **wrong in its
address**. `0x008747a0` is not this class's writer and is not a serializer of any kind: lane V2's
exact vtable inversion puts it in **no vftable at all** (`vtable_map.py who 0x008747a0` →
`is in no vftable`), and its body pops a parser stack. The serializer sweep attributed it to
`ShipDesign` by a slot mapping that did not hold for this class.
The class is not the shape the earlier note assumed either. RTTI `0x00a894a8`:
```
Game::ShipDesign : Game::ShipDesignDef, Mars::IStreamable, Mars::RefCounted, Mars::NetworkObject
```
`ShipDesignDef` is the **base**, and it is *not* an `IStreamable` — it declares its own three
virtuals (`Read`, `Write`, dtor, vftable `0x009fef64`) in a different order from `IStreamable`'s.
`ShipDesign` overrides all three; the `IStreamable` sub-object at `+0x9c` (vftable `0x00a32710`)
holds three adjustor thunks back to the primary vftable `0x00a32720`. That thunk layer is why a
slot-indexed sweep mis-keyed the class, and it is a **general warning for the serializer tool**: a
class that inherits an interface *second* reaches its own writer through a thunk, and the thunk is
what a slot walk finds.
So a design persists as **two serializers, a base and a derived one**, exactly the `*Impl` shape
lane G generalised — except the concrete class is a *derived class*, not a separate `Impl` class.
Lane V2's caution was right: check the shape, do not assume it.
### The full `Des` record, both writers, in disk order
| # | tag | primitive | member | writer |
|---|---|---|---|---|
| 1 | `FAIDes` | **bool** | `+0x4` | `ShipDesignDef::Write` `0x00827390` |
| 2 | `DHide` | **bool** | `+0x5` | " |
| 3 | `DWep` | **bool** | `+0x6` | " |
| 4 | `DName` | string | `+0x8` (`Mars::string`, `0x1c`) | " |
| 5 | `DSec` | frame | `+0x4c` — **command** | " (via `StreamableHelper<ShipDesignDef::Section>`) |
| 6 | `DSec` | frame | `+0x24` — **mission** | " |
| 7 | `DSec` | frame | `+0x74` — **engine** | " |
| 8 | `Dtc` | int | `+0x134` | `ShipDesign::Write` `0x008325e0` |
| 9 | `Dwgv` | **bool** | `+0x16c` | " |
| 10 | `Dwg` | frame, **only when `Dwgv`** | `+0x170`, `StreamableHelper<Game::WeaponGroups>` | " |
`ShipDesign::Read` `0x008324f0` mirrors it, with one extra: it reads an int `DRefCnt` into a **null
destination** — read and discarded — that the writer never emits. A network stream writes it; the
save stream does not, and the tag-addressed reader tolerates the absence.
### Three corrections this forces
**(a) Three sections, not five.** `SHIP_DESIGN_RULES.md` §1 said "five section slots, three used;
slots 3-4 are absent in every design (reserved)". There are **three**, and there is no reserved
slot. Two independent enumerations (rule 5):
* the writer emits exactly three `DSec` frames — a straight-line body, no loop;
* the constructor `0x00874c70` runs `eh_vector_constructor_iterator(this+0x24, stride 0x28,
count 3)`, and `0x24 + 3*0x28 = 0x9c`, which is exactly where `ShipDesign`'s own `IStreamable`
vptr sits. `sizeof(Game::ShipDesignDef) == 0x9c`.
The "two extra slots" were **`Dtc` and `Dwgv`**, swept into the section list by the reference Python
reader's trailing `Rest("sections")` and then decoded by `stock_designs.py` as two empty sections.
This is rule 8 in its exact form: the reader and its consumer agreed with each other and were both
wrong, and nothing downstream could see it because an empty slot is a no-op for every rule. Fixed
openly (§6).
**(b) `DWep` and `Dwgv` are bools, not ints.** Both writers call the bool primitive. With their
four-character tags a bool item and an int item are the same size on the wire and the values 0 and 1
are the same bytes, so **no save can distinguish them** — the same class of defect as lane G's
`ObservedTech.odet`, and byte-neutral for the same reason. Corrected in both readers; the engine's
typed round trip stays byte-identical on all 11 saves.
**(c) Offset order ≠ write order, again.** The section array is `[+0x24, +0x4c, +0x74]`; the wire is
`[+0x4c, +0x24, +0x74]`. Reading the array in memory order gives **mission, command, engine**;
reading the wire gives **command, mission, engine**. Both orders matter below — the wire order is
what a save reader needs, the memory order is what decides hull size.
---
## 2. Hull size and the `0x400` flag
Both live on `ShipDesign`, **neither is on the wire**, and both are recomputed from the data files
by `ShipDesign::UpdateDerivedStats` `0x0087e7c0` whenever a design changes or is loaded.
### `0x400` = `defence_platform`
`ShipSectionDef` carries a **64-bit role-flag word** at `+0x298`/`+0x29c`. The `.shipsection` parser
`0x005744e0` sets it one bit at a time: each boolean role key is a call to
`SetRoleFlagBit(&flags, loMask, hiMask, value)` `0x0056e780`. At `0x005749b7`, immediately after the
`strcmp` against the literal `defence_platform` at `0x009fea10`:
```
push ecx ; the parsed value
push 0x0 ; hi mask
lea edx,[esi+0x298]
push 0x400 ; lo mask
push edx
call 0x0056e780
```
**`0x400` is `defence_platform`.** The rest of the word, from the same scan: `refueling_capacity`
`0x2`, `repair_capacity` `0x4`, `refinery` `0x8`, `mining_capacity` `0x10`, `scanrange` /
`rebelai_scanrange` `0x20`, `gateship` `0x40`, `ewar` `0x800`, `ramscoop` `0x1000`, `aicontrol`
`0x2000`, `command_quota` `0x10000`, `node_bore` `0x20000`, `prisoner_capacity` `0x40000`,
`freighter` `0x80000`, `gravboat_bonus` `0x200000`, `construction_capacity` `0x400000`, `science`
`0x1000000`, `tradingpost` `0x2000000`, `freighterQ` `0x8000000`, `police` `0x10000000`, `spy`
`0x40000000`, `spytender` `0x80000000`; high dword `colony_trap` `0x1`, `mining_trap` `0x2`,
`monitor` `0x4`, `propaganda` `0x10`.
The design's word is the **OR** across its resolved sections
(`AggregateSectionStats` `0x00826af0`, `out[4] |= GetRoleFlags(section)`), so one flagged section
flags the whole design. In the shipped data the 49 sections carrying `defence_platform` are exactly
the `{DE,CR,DN}DefencePlatform`, `{CR,DN}TorpedoPlatform`, `{CR,DN}DronePlatform`,
`CRDeepScanPlatform` families and `_VNeumannSatellite` — all standalone mission sections — so no
shipped design mixes a flagged section with unflagged ones. That the rule is an OR is therefore
**inferred from the instruction and untestable on the corpus** (rule 6); it is unit-tested in the
engine on a synthetic design instead.
**This is not lane B5's `0x400`.** A *fleet* gets flag `0x400` when the retreat pipeline creates it,
and that flag reaches the wire as `FtFlg` — `turn1-state.sav`'s first fleet has `FtFlg 1024`. It is
a different word on a different object. The numeral is a coincidence of two bit layouts, and the
brief was right to say "check rather than assume".
### Hull size = `section_class`
`ShipSectionDef+0x260` is the parsed `section_class`, and the parse is a three-name table:
```
ParseShipClassName 0x0056e1c0: _stricmp against "Destroyer", "Cruiser", "Dreadnought"
stores the index it stopped at -> 0, 1, 2
```
Case-insensitive, which the data needs — the shipped catalogs write both `cruiser` and `Cruiser`.
The wrapper `0x0056e250` logs `" [%s] unrecognized ship class"` and stores **0** on no match, and
a section with no `section_class` key at all never reaches the parser and keeps the constructed 0.
So **absent or unrecognised is a destroyer, not an error**. 16 shipped sections have no
`section_class` (the `_Biomissile` / `_BoardingPod` families and three `_NPC` stems).
The design's hull size is `AggregateSectionStats`'s `out[0x1d] = sectionDef->+0x260`, copied to
`ShipDesign+0x12c`. That is an **assignment inside the per-section loop, not a max and not an OR**,
so the *last resolved section in memory slot order* wins — i.e. engine, else command, else mission.
Rule A6 makes every shipped design class-homogeneous, so first-wins and last-wins agree on **all 503
design records in the 11 saves** (measured, §4); the assignment order is implemented anyway because
it is what the original does, and it is unit-tested on a hand-built mixed-class design.
### One corroboration the census does not need
The same `0x400` bit picks the design's **default hull health** when no section overrides it at
`+0x2ac` (`0x00826af0`, `out[2]`):
| hull size | without `0x400` | with `0x400` |
|---|---|---|
| 0 destroyer | 500 | 100 |
| 1 cruiser | 3000 | 500 |
| 2 dreadnought | 15000 | 1000 |
A defence platform is a fifth to a fifteenth of the ship's hull. That is an independent reading of
the same two words and it agrees: `0x400` separates *platforms* from *ships*, and `+0x12c` is the
three-step hull-size ladder.
---
## 3. The census — what the two words feed
`ShipCensusByHullClass` `0x00818a50` takes a player and an `int[8]`:
```
for each fleet in server.fleets where fleet->owner(+0x58) == player:
for each ship in fleet->ships(+0xa4..+0xa8):
design = ship->+0x14
if (design->+0xb8 & 0x400) == 0: ++out[0]; ++out[2 + design->+0x12c]
else: ++out[1]; ++out[5 + design->+0x12c]
```
`out[0]`/`out[1]` are the two grand totals, which the caller computes and **discards**. The six
breakdowns land at `turnRecord+0x2a..+0x34` and reach the wire as the three `cls` groups' **`shpt`**
(ships) and **`satt`** (satellites — the game's own word for a defence platform).
Note what the loop does *not* do: it never looks at the ship's own `PlrID`, only at the fleet's
owner, and it never looks at the ship's health or state. Every ship in an owned fleet counts.
---
## 4. Measurement: 480 census leaves, 0 mismatched
Rebuilt from the save's own state plus the section catalog, compared against the record the game
archived for that save's own frame (the record is rebuilt at end of turn *and* on load, so it never
has to survive a round trip).
| saves | designs classified | leaves compared | **mismatched** |
|---|---|---|---|
| 11 | 503 | **480** | **0** |
Computed twice, independently: once in Python off `save_reader.py` + `design_rules.py`, once in C++
off the engine's own `shapes` reader and `game::data::Catalog`. The two disagree about how many
sections a design has (the Python reader's five vs the engine's three, §1a) and still produce the
same 480 numbers — so they are not the same check wearing two hats.
**Coverage, reported as loudly as the result (rule 15).** Only **32 of the 480 leaves are nonzero in
the archive**. A zero leaf agrees for free.
| leaf | nonzero of 80 |
|---|---|
| cls 0 `shpt` — destroyer ships | **18** |
| cls 0 `satt` — destroyer platforms | **3** |
| cls 1 `shpt` — cruiser ships | 0 |
| cls 1 `satt` — cruiser platforms | 0 |
| cls 2 `shpt` — dreadnought ships | **11** |
| cls 2 `satt` — dreadnought platforms | 0 |
**Three of the six census leaves are unexercised, not verified.** No save in the corpus contains a
cruiser of any kind or a dreadnought-class platform, so the cruiser column and the dreadnought
platform cell are hypotheses (rule 6). They are cheap to exercise — a save taken after
`IND_CruisCon` is researched and one cruiser is built would close the whole cruiser column, and
`zuul-turn23-fleet23.sav`'s 27-destroyer fleet shows the destroyer path is genuinely loaded.
Other honest residuals from the same run:
* 0 ships whose design id is missing from the save's design tables;
* 0 designs none of whose sections resolve against the catalog;
* 0 designs where first-resolved and last-resolved section disagree on hull size (rule A6 predicts
0, and this is the measurement that says so rather than the assumption).
---
## 5. What this closes, and what it does not
Against lane Y's baseline — turn1→turn2 **209→204**, turn2→turn3 **108→103**, 5 closed / 0
regressed, with the 204 broken down `/Sim/players` 82, `/Sim/systems` 80, `/Sim/turnstats` 24,
`/Sim/SvSctOb` 8, plus 10 singletons:
**closed: 0. regressed: 0.** Reported separately, and both are zero *in the standalone's divergence
list*, because this lane wrote no phase.
That is the honest number and it needs the reason spelled out. The census leaves live under
`/Sim/turnstats`, and the turn record is built by `src/app/turn_record.cpp`, which **lane A2 holds**
this cycle (the alliance mask `almem` is a field of the same record). Committing a census field into
that file now is exactly the cross-lane collision the fragment discipline exists to prevent. So this
lane did what lane S2's convention prescribes for a phase whose formula is held but whose write-out
is not this lane's: **evaluate and report, do not write**.
What is now unblocked, quantified:
* **6 of the 24 `/Sim/turnstats` leaves per pair** — `shpt` ×3 and `satt` ×3 — have a formula that
reproduces the archived value on 480 of 480 observations. Wiring them is one call to
`game::design::ShipCensus` inside `BuildTurnRecord`, and the earlier `--commit-blocked` run's
"three leaves of the ship census" among its 17 regressions were exactly these, wrong because the
record committed zeros. They should now commit clean.
* **The ship records** (`Player.ShipRecs`) and **tail phase T24** both need the same two words per
design; they are now available as `DesignStats::hull_size` / `DesignStats::defence_platform` and
as the standalone `classify_design()`.
* The turn record's other four unmodelled fields are **not** touched: `almem` is lane A2's,
`tdinc` waits on per-system money output, `bat` is a tail stub, and `sacq`/`slost` have count 0 in
every save (rule 6).
Nothing here was compared against the live game. `verified` in the phase catalog means "compared
against the running game", and this is not that — it is a comparison against bytes the original
wrote, which is the strongest instrument a lane holding no VM has.
---
## 6. Fixes to the oracles, made openly (rule 12)
Two readers carried the same wrong record shape; both are fixed, with a test per defect and
byte-neutrality shown.
**`verify/save-reader/save_reader.py`** — the `Des` shape ended in `Rest("sections")`. Now: three
`DSec` frames as a `Repeat`, then named `Dtc` (int) and `Dwgv` (bool), then the conditional `Dwg`;
`DWep` retyped int → bool. All 11 saves still read strict-clean; the reader's own suite is **49/49**
(was 48; three existing tests asserted the old shape and were corrected, one test added that pins
"exactly three `DSec`" against the real saves).
**`verify/design-rules/stock_designs.json`** regenerated from the same three saves. The diff is
*only* `raw_slots` 5 → 3 and `dWep` `0/1` → `false/true`; every other field of all 127 designs is
byte-for-byte the same, and `test_design_rules.py` is **32/32** with the same ground truth
(127/127 structural, 197/197 `DOpts`, 121/127 gating with the six Tarkas hidden-rider warns).
**`sots-engine` `src/mars/stream/shapes.h`** — `Design::dWep` and `Design::dwgv` retyped to bool,
the conditional `Dwg` node added, and the comment that said the derived writer made no stream call
replaced with the recovered layout. The typed round trip is still **byte-identical on all 11 saves**
at 100% named coverage.
The engine's `stock_designs.cpp` fixture loader now accepts 3–5 `raw_slots` so an old fixture still
loads. `kSlotCount` is deliberately **left at 5**: the extra two array entries are inert and
touching the enum reaches `rules.cpp` and another lane's tests for no behavioural gain. The comment
in `design.h` is corrected instead — the record is three sections, and the fifth-slot slack is this
engine's own, not the game's.
---
## 7. Open, and what would close it
1. **The cruiser column and the dreadnought-platform cell** are unexercised (§4). One save with a
cruiser in it and one with a DN defence platform close three leaves. Cheapest of everything on
this page.
2. **`Dtc`** (`+0x134`) is on the wire and is stamped by at least one caller immediately before the
recompute; its meaning is not read here. It is 1 on the stock designs.
3. **`Dwgv`/`Dwg`** — weapon groups — are false in every save. The conditional branch is implemented
from the instruction and is unexercised.
4. **The other 24 role-flag bits** are decoded (§2) but only `0x400` is consumed by anything this
lane read. `0x80000` (`freighter`) and `0x8000000` (`freighterQ`) drive a *second* classifier
`0x0081a430`/`0x0082c7c0` over the same two words — worth reading if a freighter count turns up
missing elsewhere.
5. **The remaining ~24 fields of the aggregator's output struct** (`+0x18`..`+0x64` →
`ShipDesign+0xc0`..`+0x118`) are the rest of the derived-stat block — mass, cost, capacities,
speeds. `SHIP_DESIGN_RULES.md` §6 derives most of them from the data with the formulas marked as
assumptions; `0x00826af0` is where every one of those assumptions can be settled, field by field,
and it is now a named function.
---
## Addresses
`ghidra/addresses.d/lane-d2.json` (19 entries, no collision with any existing fragment).
Key ones: `ShipDesignDef::Write` `0x00827390`, `ShipDesign::Write` `0x008325e0`,
`ShipDesign::UpdateDerivedStats` `0x0087e7c0`, `AggregateSectionStats` `0x00826af0`,
`ShipSectionDef::GetRoleFlags` `0x0056ee80`, `ParseShipClassName` `0x0056e1c0`,
`SetRoleFlagBit` `0x0056e780`, `ShipCensusByHullClass` `0x00818a50`;
offsets `ShipDesign+0xb8` (role flags), `+0x12c` (hull size), `ShipSectionDef+0x260`
(`section_class`), `+0x298` (role flags).