lane G: wire-schema channel — layouts.json -> generated stream schema for sots-engine
objects/layouts.json is a memory-layout projection: build() sorts fields by off_abs (89 of 386 classes have offset order != write order) and merges duplicate offsets into alt_tags, which is exactly the JewelsOfTheCrown double-tag trap. Both losses are the substance of the on-disk format. tools/streams.py is a second projection of the same recovery that keeps the program order Lab.layout() already computes and the repeated tags, and drops every memory fact — no off, size, sizeof, gaps or strides. The engine must read and write the format, not inherit the original's ABI. tools/gen_stream_schema.py emits sots-engine's include/generated/sots_stream_schema.h under the same discipline as gen_addresses.py: generated, provenance header, never hand-edited. 386 classes, 2042 wire items.
This commit is contained in:
alex
parent
545c715359
commit
48fcc3ff3a
5 changed files with 17087 additions and 0 deletions
|
|
@ -44,6 +44,7 @@ Status flow: `backlog → in-progress → mapped → verified` (or `blocked`).
|
|||
| engine: mars/text | engine | verified | high | 100% | 2026-09-07 | flat-kv, id-manifest, csv; oracle 64/64 (Strings.csv 5722 rows); ctest green |
|
||||
| engine: game/sim formulas | engine | verified | high | 100% | 2026-09-08 | sim-pin merged: all 5 low-confidence formulas pinned (3.3 on protection limit, old-state bankruptcy decisions, hazard curve, money tail, pop bonus); tests 356->466; + game/effects (196 TechIds, 44 typed effects, 254 checks) |
|
||||
| engine: mars/stream + rng | engine | verified | high | 100% | 2026-09-07 | merging: 100% exact dump agreement on 3 saves; typed shapes round-trip byte-identical whole file; RNG = seed(RSeed)+2 twists confirmed; 4 tag-name fixes for SAVE_FORMAT |
|
||||
| wire-schema channel (layouts -> engine) | engine | verified | high | 97% | 2026-09-08 | **Lane G.** Lane D's serializer recovery now reaches `sots-engine` as a GENERATED WIRE SCHEMA, not struct layouts: `tools/streams.py` -> `objects/streams.json` (386 classes / 2,042 items, disk order preserved, duplicate tags preserved, **every memory fact dropped** - no off/size/sizeof/gaps/strides) -> `tools/gen_stream_schema.py` -> `include/generated/sots_stream_schema.h`. WHY layouts.json was the wrong input: `build()` sorts fields by `off_abs` (**89 of 386 classes** have offset order != write order) and merges duplicate offsets into `alt_tags` (that IS the JewelsOfTheCrown double-tag trap). The table is a SPEC, not a program - the recovery is a linear pass so it cannot see `Write`'s branches and lists conditional fields (StarShip `BQ2`/`hbq`) unconditionally, and flattens container loops; a codec driven off it desynchronises. So hand-written `io()` stays the codec and a new `SchemaProbe` archive + `test_wire_schema.cpp` CHECK it: **56 shapes bound, 657 items matched, 0 mismatches**. FOUND 4 REAL DEFECTS present in BOTH `save_reader.py` and the engine, invisible to any round-trip test: `SystemParams` field 1 is a **string** not an int (empty string == 4 zero bytes == int 0, so it round-tripped by luck; a named planet would have desynced both readers); `ObservedTech/ObservedWeapon.odet` is a **bool** not an int (byte-safe only because a 4-char tag makes bool and int items both 12 bytes); `SpeciesRatios.nv` and `ShipRecords.srbd` are **counts**, not fields. TWO TRAPS RESOLVED: `ServerTradeManager`'s no-op Read/Write is real but the call is **virtual** - `ServerTradeManagerImpl` has the real serializer (`NumTradeSectors`/`TradeID`/`Trade`/`SctSize`), same shape resolves `IServerSpyManager` -> `ServerSpyManager`; general rule: **when an interface's serializer is the inherited stub, look for the concrete `*Impl`**. `ShipDesign::Write` really does make no stream call, but `ShipDesignDef::Section` is recovered so most of `Des` is typeable anyway. COVERAGE, measured by a new `CoverageArchive` that separates items a field NAMES from items a `Node` merely CARRIES (a byte-identical round trip is not a coverage claim): **37.9% -> 97.1%** typed on turn1 (97.2/97.2/97.6 on the others), round trip still byte-identical on all 4 saves. Typed this round: `TechTree` (both NumTechs sections - the 2nd is per-tech state, field list from `SpyReportTechTree` which streams the same record), `Events` (3 nesting levels), `ShipRecs`, `sprjs`, `civr`, `comms`, `spy2`, `spymgr`, `aid`, `Ojvs`, `AIEnf`, `FNG`, `trdmgr`, `Des` sections + gun banks. STILL OPAQUE: `CD` custom data 744 items (TurnCommands_v5 is a no-orders snapshot - **needs a save with issued turn commands**), `SvSctOb` 147 (8 EncObj variants), `DOpts` 94 (`read_elem` has no std::string branch), `spies2` 56, `RNG` 2 (correctly opaque). Ratchet at 95% so typing can't regress. ctest **34/34**, clean-room OK, `test_save` skips cleanly unset. `findings/objects/wire-schema-channel.md`; sots-engine branch `wip/layouts` |
|
||||
| engine: game/data catalogs | engine | verified | high | 100% | 2026-09-07 | merged: WeaponDef/ShipSectionDef/TurretTable/IdRegistry/TechTree/StringTable + cross_check; oracle 229,042 values 0 diffs; crosslink set reproduced exactly; 34 malformed shipped tokens pinned |
|
||||
| engine: mars/vfs (gob) | engine | verified | high | 100% | 2026-09-07 | merged: ZIP reader + native override; 8352+2035 entries = unzip -l; all 10268 files CRC-clean; byte-equal spot checks; ctest 11/11 |
|
||||
| determinism oracle | verify | verified | high | 100% | 2026-09-07 | BYTE-IDENTICAL across 5 runs incl. cross-process: (Autosave).sav 978041ac…, (Autosave EndTurn).sav bb4fd9ac…; gzip MTIME=0; only loaded-post-turn re-save differs (Player.Status 4->0, Summary.Checksum). findings/subsystems/determinism-oracle.md RE-CONFIRMED 2026-09-08 on engine `cef889e` (lane M's movement fix included) by lane F: `bb4fd9ac…` / `978041ac…` unchanged, and reproduced a further 3x under forced control words that leave the arithmetic alone (0x027f/0x127f/0x137f). |
|
||||
|
|
|
|||
177
findings/objects/wire-schema-channel.md
Normal file
177
findings/objects/wire-schema-channel.md
Normal file
|
|
@ -0,0 +1,177 @@
|
|||
# The wire-schema channel: getting lane D's layouts into the engine
|
||||
|
||||
Lane G, 2026-09-08. Host/static only — VM140 was held by lane U and the game was never run.
|
||||
|
||||
Lane D recovered 386 class layouts from the `Mars::IStreamable` serializers. That knowledge lived
|
||||
only in the notes repo. This lane built the channel that carries it into `sots-engine`, wired the
|
||||
engine's save codec to it, and used it to find and fix four real defects in the reader. Engine
|
||||
coverage of a real save went from **37.9 % to 97.1 %** of stream items *typed* (not merely
|
||||
round-tripped), with the byte-identical round trip preserved throughout.
|
||||
|
||||
---
|
||||
|
||||
## 1. The design choice: a wire schema, not a struct layout
|
||||
|
||||
`objects/layouts.json` is a **memory-layout** projection and is the wrong shape for a codec. Two
|
||||
things are lost in `serializers.py`'s `build()`, and both are the substance of the on-disk format:
|
||||
|
||||
* **Order.** `Lab.layout()` returns fields in the program order of `Write` (with base-class and
|
||||
sub-writer calls spliced in at their call site) — that order *is* the disk order. `build()` then
|
||||
does `fields.sort(key=lambda x: (x["off_abs"], x["va"]))`. **89 of the 386 classes** have at least
|
||||
one field whose offset order differs from its write order, so for those the published record does
|
||||
not describe the stream.
|
||||
* **Repetition.** `build()` merges two writes of the same offset into `alt_tags`. That is exactly the
|
||||
`SVSOJewelsOfTheCrown` `JEWELLOCATIONID`-twice trap, and the merge silently drops the second item.
|
||||
|
||||
So this lane added `tools/streams.py`, a second projection of the same recovery that keeps the order
|
||||
and the repeats — and **drops every memory fact**: no `off`, no `size`, no `sizeof`, no `gaps`, no
|
||||
`strides`. `sots-engine` is our own C++, not a byte-for-byte decomp; it must read and write the
|
||||
*format* faithfully and must not inherit the original's ABI in its runtime types. (The shim is the
|
||||
one component that legitimately needs the original ABI, because it reads the running game's memory.
|
||||
Those few offsets already have a home: `offset` entries in `ghidra/addresses.json`.)
|
||||
|
||||
The on-disk primitive is **not** `layouts.json`'s `kind`, which is a memory type. It comes from the
|
||||
stream vftable slot the writer called, or the wrapper helper it called instead:
|
||||
|
||||
| slot | helper | disk |
|
||||
|---|---|---|
|
||||
| `+0x18` | `0x8b9d70` | string |
|
||||
| `+0x1c` | `0x8b9c20` | bool (1 byte) |
|
||||
| `+0x20` | `0x8b9be0` | float32 |
|
||||
| `+0x24` | `0x8b9d50`, `0x8b9d00`, `0x816490` | **int32** |
|
||||
| `+0x28` | — | nested frame |
|
||||
| `+0x30` | `0x8b9c60` | raw (`n = 8` → int64) |
|
||||
|
||||
A member the original holds as `int16` or `int8` is written by `WriteInt` and is **four bytes on the
|
||||
wire**. `0x8b9d00` is a distinct `int16` wrapper that widens; `0x816490` is
|
||||
`Stream::WriteNetworkObjectId`, an id, also int32. Nothing narrow ever reaches the stream.
|
||||
|
||||
`tools/gen_stream_schema.py` then emits `include/generated/sots_stream_schema.h` — a sibling of
|
||||
`gen_addresses.py` under the same discipline: generated, provenance header, never hand-edited,
|
||||
regenerated on every merge. 386 classes, 2,042 wire items.
|
||||
|
||||
## 2. Why it is a *check*, not a generator
|
||||
|
||||
**The recovered schema is a specification, not a program.** Two properties make a codec driven
|
||||
straight off it desynchronise on real data:
|
||||
|
||||
* **It is unconditional.** The recovery is a linear pass over `Write` and cannot see `Write`'s
|
||||
branches. `StarShip` writes `BQ2` only when `hbq` is set, and `pop`/`ppop` only when `hsp` is set;
|
||||
the table lists all three unconditionally. The sequence is a **superset** of any single record.
|
||||
* **Container loops are flattened.** A count item is followed by its element items as siblings, not
|
||||
nested inside the container. `member == false` marks the elements, and that is the only signal.
|
||||
|
||||
The engine's hand-written `io(Ar&)` shapes therefore stay the codec — they can express the
|
||||
conditionals and the nesting that the binary facts cannot supply — and the generated table is what
|
||||
**proves they agree with the binary, item for item, in order**. That is the same relationship
|
||||
`sots_addresses.h` already has with the shim: it supplies facts that hand-written code consumes; it
|
||||
does not generate code.
|
||||
|
||||
`sots-engine` grew a fourth archive, `SchemaProbe` (`src/mars/stream/probe.h`), which walks a shape's
|
||||
`io()` with **every branch taken** — the same "all branches" view the recovery has — and records the
|
||||
item sequence. `tests/mars_stream/test_wire_schema.cpp` LCS-aligns that against the generated table.
|
||||
**56 shapes bound, 657 items matched, 0 mismatches.** A tag both sides name with different disk
|
||||
primitives is a hard build failure; a wire-only item is reported, not fatal.
|
||||
|
||||
## 3. What the check found
|
||||
|
||||
Four defects, all present in **both** `save_reader.py` and the engine, none of which any round-trip
|
||||
test could have caught:
|
||||
|
||||
1. **`SystemParams` field 1 is a `std::string`, not an int.** Both readers modelled it as `int`. It is
|
||||
the **empty string in every save available, and an empty string is four zero bytes** — byte-identical
|
||||
to the int `0`. A save carrying a non-empty name here would have desynchronised both parsers.
|
||||
The engine now reads it as a string.
|
||||
2. **`ObservedTech::odet` / `ObservedWeapon::odet` are `bool`, not int.** The binary calls the bool
|
||||
writer and lane D's own golden table already said `bool` at `+0x08`; `save_reader.py` and the
|
||||
engine both said int. Byte-safe here **only by coincidence**: with a 4-char tag a bool item and an
|
||||
int item both occupy 12 bytes and the bool's three pad bytes are zeroed. A shorter tag would not
|
||||
have been so forgiving. 68 `otch` elements per save exercised this and never showed it.
|
||||
3. **`SpeciesRatios::nv` is a count, not a field.** The recovery flags `sp` and `va2` as loop-body
|
||||
writes; the saves agree (`nv==1` frames carry one `(sp, va2)` pair, `nv==0` frames carry nothing).
|
||||
4. **`ShipRecords::srbd` is a count, not a field** — the second of two counted sections
|
||||
(`srd, src, srb, srl, sri`, note no `srk`). Values 0, 1, 3 and 4 occur across the players available.
|
||||
|
||||
Applying all four left the byte-identical round trip intact on all four real saves.
|
||||
|
||||
## 4. Two recorded traps resolved
|
||||
|
||||
* **`Game::ServerTradeManager`'s Read/Write really are the inherited no-op — and the `trdmgr` frame
|
||||
really does have content.** The resolution is that the call is **virtual**: the object is a
|
||||
`Game::ServerTradeManagerImpl`, and *that* class has a real serializer emitting exactly
|
||||
`NumTradeSectors` / `TradeID` / `Trade` / `SctSize`, with `Game::ServerTradeSector` giving the
|
||||
sector body (14 items). The same shape resolves `Game::IServerSpyManager` → `Game::ServerSpyManager`.
|
||||
**General rule for the next lane: when an interface type's serializer is the inherited stub, look
|
||||
for the concrete `*Impl` / non-`I` class.**
|
||||
* **`Game::ShipDesign::Write` makes no stream call**, as recorded — and that is still true. But the
|
||||
`Des` frame's *section* body is `Game::ShipDesignDef::Section`, which is recovered
|
||||
(`DSec` frame → `Game::ShipSectionID`, `DGbnk2` carr, `DOpts` carr), so most of the design record
|
||||
is typeable anyway. Only `DW2`, `Dtc` and `Dwgv` have no recovered serializer and came from the saves.
|
||||
|
||||
## 5. Coverage
|
||||
|
||||
A byte-identical round trip is **not** a coverage claim: the engine reached it partly by typing
|
||||
fields and partly by carrying whole bodies as generic `Node`s, which round-trip trivially because
|
||||
they are copied verbatim. `CoverageArchive` (also in `probe.h`) separates the two by walking a
|
||||
*populated* shape's `io()` and counting items a field names against items only a `Node` carried.
|
||||
|
||||
| save | before | after |
|
||||
|---|---|---|
|
||||
| turn1-state | 37.9 % | **97.1 %** |
|
||||
| turn2-state | 38.8 % | **97.2 %** |
|
||||
| turn3-state | 39.4 % | **97.2 %** |
|
||||
| zuul-turn5-species5 | 42.5 % | **97.6 %** |
|
||||
|
||||
The bodies typed this round, all against the recovered schema: `TechTree` (both `NumTechs` sections —
|
||||
the second is the per-tech state, whose field list and types come from `Game::SpyReportTechTree`,
|
||||
which streams the same record, and which is where `Tfc` being a bool came from), `Events`
|
||||
(`EventStorage` → `TurnEvents` → `Event`, three nesting levels), `ShipRecs`, `sprjs`, `civr`/`spe`,
|
||||
`comms`, `spy2`, `spymgr`, `aid`, `Ojvs`, `AIEnf`, `FNG`, `trdmgr`, and the `Des` section /
|
||||
gun-bank tree. `DW2` is branch-gated on `bID`: `wid` (an id) when set, `wfn` (a family name) when clear.
|
||||
|
||||
**What is still opaque, and why** (turn1 numbers, of 36,644 items):
|
||||
|
||||
| region | items | why |
|
||||
|---|---|---|
|
||||
| `CD` custom-data blocks | 744 | `TurnCommands_v5` is 35 anonymous scalars with 27 trailing zeros — a *no-orders* snapshot. Typing it needs a save with issued turn commands. |
|
||||
| `SvSctOb` | 147 | Scenario/encounter objects: 8 `EncObj` variants keyed by `EncID`, structure known but each needs its own shape. |
|
||||
| `DOpts` | 94 | `carr<String>`; `read_elem` has no `std::string` branch yet. |
|
||||
| `spies2` | 56 | Anonymous container, count 0 in every save — element shape unobservable. |
|
||||
| `RNG` | 2 | Correctly opaque: the raw MT19937 state block. |
|
||||
| `Attrib` | 2 | Empty anonymous container in every save. |
|
||||
|
||||
The test carries a **ratchet** (`pct >= 95.0`), so typing a body can never silently regress.
|
||||
|
||||
## 6. Honest limits
|
||||
|
||||
* Coverage is measured in **items**, not bytes, and only over the four saves available.
|
||||
* `Ojvs`, `aid`, `Attrib`, `comms.nmsg`, `AIEnf.Nas`, `spymgr.nspy` and `spy2.rtc/evc/ttc` are **0 in
|
||||
every player of every save**. Their containers are typed from the recovery; their *element* shapes
|
||||
are not verified by any save and are marked as such in `shapes.h`.
|
||||
* `Game::Plague` and `Game::SpecialProjectImpl` are never emitted (`NumPlgs2` and `NSprj` are 0
|
||||
everywhere), so `Plg` and `Sprj` have no ground truth at all. Note the `Plg` name collision:
|
||||
`/Sim/Sys/.../Plg` is a plague frame, `/Sim/Flt/Ship/Plg` is a plain int (always −1).
|
||||
* The 176 anonymous-tag classes remain usable but nameless; the conformance check aligns them by
|
||||
tag, so a class of all-`"."` items aligns only as well as its ordering allows.
|
||||
* `sizeof` never crossed the channel, so lane D's lower-bound caveat does not apply to anything the
|
||||
engine now believes.
|
||||
|
||||
## 7. Artifacts
|
||||
|
||||
* `tools/streams.py` — the wire projection (`objects/streams.json`, 386 classes, 2,042 items).
|
||||
`uv run python3 tools/streams.py Game::StarShip` prints one class readably.
|
||||
* `tools/gen_stream_schema.py` — emits `sots-engine`'s `include/generated/sots_stream_schema.h`.
|
||||
* `sots-engine` `wip/layouts`: `src/mars/stream/probe.h` (`SchemaProbe`, `CoverageArchive`),
|
||||
`tests/mars_stream/test_wire_schema.cpp`, widened `shapes.h`.
|
||||
* ctest **34/34** (was 33/33), `tools/clean_room_check.sh` **OK**, `test_save` skips cleanly with
|
||||
`SOTS_SAVES_DIR` unset.
|
||||
|
||||
## 8. Open items for the next lane
|
||||
|
||||
1. **A save with issued turn commands** would unblock `CD`/`TurnCommands_v5` — the single biggest
|
||||
remaining blob, and the one that matters for a functional reimplementation of orders.
|
||||
2. `SvSctOb`'s eight `EncObj` variants, one shape each, keyed by `EncID` (3, 4, 5, 9, 17, 10, 20, 1).
|
||||
3. `read_elem` needs a `std::string` branch so `carr<std::string>` works (`DOpts`, and any future
|
||||
string list).
|
||||
4. Re-run `tools/streams.py` whenever `serializers.py` improves — the generated header regenerates
|
||||
from it in 0.2 s, and the conformance test will say immediately whether the engine still agrees.
|
||||
16518
objects/streams.json
Normal file
16518
objects/streams.json
Normal file
File diff suppressed because it is too large
Load diff
210
tools/gen_stream_schema.py
Normal file
210
tools/gen_stream_schema.py
Normal file
|
|
@ -0,0 +1,210 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Emit include/generated/sots_stream_schema.h for sots-engine from objects/streams.json.
|
||||
|
||||
Sibling of `tools/gen_addresses.py`, same discipline: a generated file carrying
|
||||
**facts only**, a provenance header, never hand-edited, regenerated on every
|
||||
merge.
|
||||
|
||||
What crosses the channel is the *wire schema* -- for each serializable class, the
|
||||
ordered sequence of items its `Write` puts on the stream: on-disk tag, on-disk
|
||||
primitive, and how the item is framed. What deliberately does **not** cross is
|
||||
every memory fact in `objects/layouts.json`: field offsets, `sizeof`, gaps and
|
||||
container strides. `sots-engine` is our own C++, not a byte-for-byte decomp; it
|
||||
must read and write the *format* faithfully and must not inherit the original's
|
||||
ABI in its runtime types. (The shim is the one component that legitimately
|
||||
needs the original ABI, because it reads the running game's memory -- those few
|
||||
offsets already have a home, as `offset` entries in `ghidra/addresses.json`.)
|
||||
|
||||
The header is a *specification*, not a program. See the note on
|
||||
`SOTS_WIRE_UNCONDITIONAL` below and findings/objects/wire-schema-channel.md:
|
||||
a linear pass over `Write` cannot see its branches, so the recovered sequence is
|
||||
a **superset** of what any one record contains. The engine's hand-written
|
||||
`io()` shapes remain the codec; this table is what proves they agree with the
|
||||
binary, item for item, in order.
|
||||
|
||||
Usage
|
||||
-----
|
||||
uv run python3 tools/streams.py # -> objects/streams.json
|
||||
uv run python3 tools/gen_stream_schema.py \
|
||||
../sots-engine/include/generated/sots_stream_schema.h
|
||||
"""
|
||||
import datetime
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
ROOT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..")
|
||||
|
||||
PRIM = {None: "Unknown", "i32": "I32", "i64": "I64", "f32": "F32",
|
||||
"bool": "Bool", "str": "Str", "frame": "Frame", "raw": "Raw"}
|
||||
SHAPE = {"scalar": "Scalar", "frame": "Frame", "carr": "CArr",
|
||||
"narr": "NArr", "raw": "Raw"}
|
||||
|
||||
|
||||
def cstr(s):
|
||||
return '"' + s.replace("\\", "\\\\").replace('"', '\\"') + '"'
|
||||
|
||||
|
||||
def main():
|
||||
streams = json.load(open(os.path.join(ROOT, "objects", "streams.json")))
|
||||
layouts = json.load(open(os.path.join(ROOT, "objects", "layouts.json")))
|
||||
try:
|
||||
rev = subprocess.check_output(
|
||||
["git", "-C", ROOT, "rev-parse", "--short", "HEAD"]).decode().strip()
|
||||
except Exception:
|
||||
rev = "unknown"
|
||||
|
||||
out = [
|
||||
"// GENERATED -- do not edit. Wire schema of Sword of the Stars.exe (GOG 1.8.1).",
|
||||
f"// Source: sots-re objects/streams.json @ {rev}, generated "
|
||||
f"{datetime.date.today()} by tools/gen_stream_schema.py",
|
||||
"//",
|
||||
"// For each serializable class, the ordered sequence of items its Mars::IStreamable",
|
||||
"// Write() puts on the stream. Recovered mechanically from the serializers; the",
|
||||
"// order is the program order of Write, with base-class and sub-writer calls spliced",
|
||||
"// in at their call site, which is the on-disk order.",
|
||||
"//",
|
||||
"// FACTS ONLY. No field offsets, no sizeof, no struct strides: this header describes",
|
||||
"// the on-disk FORMAT, never the original's memory layout. The engine's own types are",
|
||||
"// free to be laid out however they like.",
|
||||
"//",
|
||||
"// SOTS_WIRE_UNCONDITIONAL -- read this before treating the table as a program.",
|
||||
"// The recovery is a linear pass over Write, so it cannot see Write's branches. A",
|
||||
"// field the game emits only under a condition (StarShip's BQ2, gated by hbq) is",
|
||||
"// listed unconditionally here. The sequence is therefore a SUPERSET of what any",
|
||||
"// single record contains, and a codec driven straight off it would desynchronise.",
|
||||
"// Container loops are flattened the same way: a count item is followed by its",
|
||||
"// element items as siblings, with `member == false` marking the elements, rather",
|
||||
"// than nested inside the container. Use this table to CHECK a hand-written codec,",
|
||||
"// not to generate one.",
|
||||
"//",
|
||||
"// `grade` is the recovery tier (verified / clean / unnamed / partial / empty) and",
|
||||
"// read_agree/read_comparable is the class's Read-vs-Write offset cross-check.",
|
||||
"#pragma once",
|
||||
"#include <cstdint>",
|
||||
"#include <cstddef>",
|
||||
"",
|
||||
"namespace sots::wire {",
|
||||
"",
|
||||
"// On-disk primitive, from the Stream vftable slot the writer called:",
|
||||
"// +0x18 str +0x1c bool +0x20 f32 +0x24 i32 +0x28 frame +0x30 raw (n=8 -> i64)",
|
||||
"// NOTE this is the DISK type. A member held as int16 or int8 in the original is",
|
||||
"// written by WriteInt and is I32 on the wire.",
|
||||
"enum class Prim : uint8_t { Unknown, I32, I64, F32, Bool, Str, Frame, Raw };",
|
||||
"",
|
||||
"// How the item is framed on the stream.",
|
||||
"// Scalar [len][tag][value], padded to 4",
|
||||
"// Frame [len][tag][pad] BEEFBEEF ... 41104110",
|
||||
"// CArr a Frame whose first item is a \".\" count, then that many elements",
|
||||
"// NArr a bare count item at this level, then that many elements follow it",
|
||||
"// Raw opaque payload whose length only the writer knows",
|
||||
"enum class Shape : uint8_t { Scalar, Frame, CArr, NArr, Raw };",
|
||||
"",
|
||||
"struct Field {",
|
||||
" const char* tag; // on-disk tag; \".\" when the writer passed NULL",
|
||||
" Prim prim;",
|
||||
" Shape shape;",
|
||||
" const char* of; // frame contents class name, or nullptr",
|
||||
" bool member; // false: a container element / loop-body write",
|
||||
" bool computed; // not a member: a count or a constant the writer derived",
|
||||
" bool unresolved; // the recovery could not type this item",
|
||||
"};",
|
||||
"",
|
||||
"struct Class {",
|
||||
" const char* name;",
|
||||
" const Field* fields;",
|
||||
" uint16_t count;",
|
||||
" uint16_t anon; // items whose tag is \".\"",
|
||||
" const char* grade;",
|
||||
" uint16_t read_agree;",
|
||||
" uint16_t read_comparable;",
|
||||
"};",
|
||||
"",
|
||||
]
|
||||
|
||||
names = sorted(streams)
|
||||
for cls in names:
|
||||
c = streams[cls]
|
||||
ident = "k_" + "".join(ch if ch.isalnum() else "_" for ch in cls)
|
||||
if not c["fields"]:
|
||||
out.append(f"inline constexpr Field {ident}[1] = "
|
||||
"{{nullptr, Prim::Unknown, Shape::Scalar, nullptr, false, false, false}};"
|
||||
f" // {cls}: Write emits no item")
|
||||
continue
|
||||
out.append(f"inline constexpr Field {ident}[] = {{ // {cls}")
|
||||
for f in c["fields"]:
|
||||
of = cstr(f["of"]) if f.get("of") else "nullptr"
|
||||
out.append(" {%s, Prim::%s, Shape::%s, %s, %s, %s, %s}," % (
|
||||
cstr(f["tag"]), PRIM[f["prim"]], SHAPE[f["shape"]], of,
|
||||
"true" if f["member"] else "false",
|
||||
"true" if f.get("computed") else "false",
|
||||
"true" if f.get("unresolved") else "false"))
|
||||
out.append("};")
|
||||
out.append("")
|
||||
out.append("inline constexpr Class kClasses[] = {")
|
||||
for cls in names:
|
||||
c = streams[cls]
|
||||
L = layouts.get(cls, {})
|
||||
ident = "k_" + "".join(ch if ch.isalnum() else "_" for ch in cls)
|
||||
n = len(c["fields"])
|
||||
out.append(" {%s, %s, %d, %d, %s, %d, %d}," % (
|
||||
cstr(cls), ident, n, c["anon"],
|
||||
cstr(L.get("grade") or "unknown"),
|
||||
L.get("read_agree") or 0, L.get("read_comparable") or 0))
|
||||
out.append("};")
|
||||
out.append(f"inline constexpr size_t kClassCount = {len(names)};")
|
||||
out += [
|
||||
"",
|
||||
"// Linear lookup by class name. The table is sorted, but it is small and this is",
|
||||
"// only ever called from tests and tools.",
|
||||
"inline const Class* find(const char* name) {",
|
||||
" for (const Class& c : kClasses) {",
|
||||
" const char* a = c.name;",
|
||||
" const char* b = name;",
|
||||
" while (*a && *a == *b) { ++a; ++b; }",
|
||||
" if (!*a && !*b) return &c;",
|
||||
" }",
|
||||
" return nullptr;",
|
||||
"}",
|
||||
"",
|
||||
"inline const char* prim_name(Prim p) {",
|
||||
" switch (p) {",
|
||||
" case Prim::Unknown: return \"?\";",
|
||||
" case Prim::I32: return \"i32\";",
|
||||
" case Prim::I64: return \"i64\";",
|
||||
" case Prim::F32: return \"f32\";",
|
||||
" case Prim::Bool: return \"bool\";",
|
||||
" case Prim::Str: return \"str\";",
|
||||
" case Prim::Frame: return \"frame\";",
|
||||
" case Prim::Raw: return \"raw\";",
|
||||
" }",
|
||||
" return \"?\";",
|
||||
"}",
|
||||
"",
|
||||
"inline const char* shape_name(Shape s) {",
|
||||
" switch (s) {",
|
||||
" case Shape::Scalar: return \"scalar\";",
|
||||
" case Shape::Frame: return \"frame\";",
|
||||
" case Shape::CArr: return \"carr\";",
|
||||
" case Shape::NArr: return \"narr\";",
|
||||
" case Shape::Raw: return \"raw\";",
|
||||
" }",
|
||||
" return \"?\";",
|
||||
"}",
|
||||
"",
|
||||
"} // namespace sots::wire",
|
||||
"",
|
||||
]
|
||||
|
||||
dest = sys.argv[1] if len(sys.argv) > 1 else os.path.join(
|
||||
ROOT, "objects", "generated", "sots_stream_schema.h")
|
||||
os.makedirs(os.path.dirname(dest), exist_ok=True)
|
||||
open(dest, "w").write("\n".join(out))
|
||||
nf = sum(len(c["fields"]) for c in streams.values())
|
||||
print(f"wrote {dest} ({len(names)} classes, {nf} wire items)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main() or 0)
|
||||
181
tools/streams.py
Normal file
181
tools/streams.py
Normal file
|
|
@ -0,0 +1,181 @@
|
|||
#!/usr/bin/env python3
|
||||
"""The *wire* projection of the serializer recovery -> objects/streams.json.
|
||||
|
||||
`objects/layouts.json` is a **memory-layout** view: `build()` sorts fields by
|
||||
`off_abs` and merges two writes of the same offset into `alt_tags`. That is the
|
||||
right shape for a decompiler struct, and the wrong shape for a codec, because it
|
||||
destroys the two things the on-disk format is made of:
|
||||
|
||||
* **order** -- the stream is a sequence, and its order is the *program* order
|
||||
of `Write` (with base-class / sub-writer calls spliced in at their call
|
||||
site). Offset order is not the same thing: 89 of the 386 classes have at
|
||||
least one field whose offset order differs from its write order.
|
||||
* **repetition** -- `SVSOJewelsOfTheCrown::Write` emits `JEWELLOCATIONID`
|
||||
twice, from two different members. The offset merge turns the second one
|
||||
into an `alt_tags` note and the record loses a field.
|
||||
|
||||
`Lab.layout()` already returns program order; this tool just refuses to throw it
|
||||
away. It also drops every memory fact (`off`, `size`, `sizeof`, `gaps`,
|
||||
`strides`, `vftable`): a reimplementation must read and write the *format*, and
|
||||
must not inherit the original's ABI. What survives is exactly the wire schema --
|
||||
ordinal, on-disk tag, on-disk primitive, and for framed fields the class name of
|
||||
the frame's contents.
|
||||
|
||||
The disk primitive is **not** `layouts.json`'s `kind`, which is a memory type: an
|
||||
`int16` member and an `int8` member are both written by `Stream::WriteInt` and
|
||||
are four bytes on disk. It is derived from the stream vftable slot the writer
|
||||
called, or from the wrapper helper it called instead:
|
||||
|
||||
+0x18 (24) string +0x1c (28) bool +0x20 (32) float
|
||||
+0x24 (36) int +0x28 (40) frame +0x30 (48) raw (n=8 -> int64)
|
||||
|
||||
Usage
|
||||
-----
|
||||
uv run python3 tools/streams.py # -> objects/streams.json
|
||||
uv run python3 tools/streams.py Game::StarShip # one class, readable
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
sys.path.insert(0, HERE)
|
||||
import serializers as S # noqa: E402
|
||||
|
||||
OUT = os.path.join(os.path.dirname(HERE), "objects")
|
||||
|
||||
# Wrapper helpers, by the primitive they write. `serializers.HELPERS` has the
|
||||
# same VAs; naming them here keeps this tool readable and independent of the
|
||||
# spelling used there.
|
||||
HELPER_PRIM = {
|
||||
0x8B9D50: "i32", # WriteInt wrapper
|
||||
0x8B9D00: "i32", # WriteInt16 wrapper -- widened to int32 on the wire
|
||||
0x8B9C20: "bool",
|
||||
0x8B9BE0: "f32",
|
||||
0x8B9D70: "str",
|
||||
0x8B9C60: "i64", # raw, n = 8
|
||||
0x816490: "i32", # Stream::WriteNetworkObjectId -- an id, int32 on disk
|
||||
}
|
||||
SLOT_PRIM = {24: "str", 28: "bool", 32: "f32", 36: "i32", 40: "frame", 48: "raw"}
|
||||
|
||||
|
||||
def disk_prim(f):
|
||||
"""On-disk primitive for one recovered write, or None if undecidable."""
|
||||
h = f.get("helper")
|
||||
if h is not None:
|
||||
p = HELPER_PRIM.get(h)
|
||||
if p:
|
||||
return p
|
||||
slot = f.get("slot")
|
||||
if slot in SLOT_PRIM:
|
||||
p = SLOT_PRIM[slot]
|
||||
if p == "raw" and f.get("size") == 8:
|
||||
return "i64"
|
||||
return p
|
||||
return None
|
||||
|
||||
|
||||
def shape_of(f, prim):
|
||||
"""How the *stream* frames this field.
|
||||
|
||||
scalar one item: [tag][value]
|
||||
frame one framed item: [tag] BEEFBEEF ... 41104110
|
||||
carr framed array: a frame whose first item is a "." count
|
||||
narr a bare int count at this level, then that many elements follow
|
||||
raw opaque payload whose length only the writer knows
|
||||
"""
|
||||
kind = f.get("kind", "")
|
||||
if kind == "vector":
|
||||
# A VectorHelper written through the nested slot is framed (count
|
||||
# inside); one written through the int wrapper is a bare count with the
|
||||
# elements following at the same level.
|
||||
return "carr" if f.get("slot") == 40 else "narr"
|
||||
if prim == "frame":
|
||||
return "frame"
|
||||
if prim == "raw":
|
||||
return "raw"
|
||||
return "scalar"
|
||||
|
||||
|
||||
def project(lab, cls, info):
|
||||
"""One class -> its ordered wire schema."""
|
||||
fields, ok, unknown = [], True, 0
|
||||
for f in lab.layout(info["write"]):
|
||||
# Container element writes and loop-body reads carry `this: false`; they
|
||||
# describe the *element* type, which the frame's own class already
|
||||
# names. Computed values (a count, a constant) are not members but they
|
||||
# ARE items on the wire, so they are kept, tagged as such.
|
||||
prim = disk_prim(f)
|
||||
rec = {
|
||||
"tag": f.get("tag", "."),
|
||||
"prim": prim,
|
||||
"shape": shape_of(f, prim),
|
||||
"member": bool(f.get("this")),
|
||||
}
|
||||
if f.get("inner"):
|
||||
rec["of"] = f["inner"]
|
||||
if f.get("kind") in ("const",):
|
||||
rec["computed"] = True
|
||||
if f.get("unresolved") or prim is None:
|
||||
rec["unresolved"] = True
|
||||
unknown += 1
|
||||
ok = False
|
||||
fields.append(rec)
|
||||
anon = sum(1 for f in fields if f["tag"] == ".")
|
||||
return {
|
||||
"class": cls,
|
||||
"fields": fields,
|
||||
"n": len(fields),
|
||||
"anon": anon,
|
||||
"unresolved": unknown,
|
||||
"named": anon == 0 and bool(fields),
|
||||
"complete": ok and bool(fields),
|
||||
}
|
||||
|
||||
|
||||
def run():
|
||||
lab = S.Lab()
|
||||
out = {}
|
||||
for cls, info in lab.infos.items():
|
||||
out[cls] = project(lab, cls, info)
|
||||
# Helper-only POD types (Vector3, OutputRates, ...) have no vftable of their
|
||||
# own and are reachable only through a specialised StreamableHelper thunk.
|
||||
# They are frame contents for other classes, so the codec needs them.
|
||||
for t, h in lab.helper_writers().items():
|
||||
if t in out or not h["write"]:
|
||||
continue
|
||||
out[t] = project(lab, t, {"write": h["write"]})
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
if len(sys.argv) > 1 and sys.argv[1] != "-":
|
||||
lab = S.Lab()
|
||||
name = sys.argv[1]
|
||||
info = lab.infos.get(name) or lab.helper_writers().get(name)
|
||||
if not info:
|
||||
sys.exit(f"no serializer for {name!r}")
|
||||
p = project(lab, name, info)
|
||||
print(f"{name} ({p['n']} items, {p['anon']} anonymous)")
|
||||
for i, f in enumerate(p["fields"]):
|
||||
extra = f" <{f['of']}>" if f.get("of") else ""
|
||||
flag = " UNRESOLVED" if f.get("unresolved") else ""
|
||||
flag += "" if f["member"] else " [element]"
|
||||
flag += " [computed]" if f.get("computed") else ""
|
||||
print(f" {i:3d} {f['tag']!r:<20} {str(f['prim']):<6} "
|
||||
f"{f['shape']:<6}{extra}{flag}")
|
||||
return 0
|
||||
out = run()
|
||||
os.makedirs(OUT, exist_ok=True)
|
||||
dest = os.path.join(OUT, "streams.json")
|
||||
with open(dest, "w") as fh:
|
||||
json.dump(out, fh, indent=1, sort_keys=True)
|
||||
named = sum(1 for v in out.values() if v["named"])
|
||||
items = sum(v["n"] for v in out.values())
|
||||
print(f"wrote {dest}: {len(out)} classes, {items} wire items, "
|
||||
f"{named} fully named")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main() or 0)
|
||||
Loading…
Add table
Reference in a new issue