alex/sots-re
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

RB: replay a recorded turn's commands -- ModCount is reachable, and the rates frame's memory order is not its wire order

Browse source 
sots_turn --turn-commands puts /Sim/ModCount on the original's 24 with zero residual, closing
the one leaf that has been unreachable from a save all campaign. Canonical pair 108 -> 62,
closed 46, regressed 0, fresh build directory.

The .tcb capture format (line-oriented, parser-free, '?' for a field the instrument could not
read, per-client AI seeds), a converter from lane L4's shim dump, and an adapter from lane CB's
JSON capture -- CB's stays the capture of record, .tcb stays the engine's input, and the two
paths produce byte-identical replays.

A falsified prediction paid for itself: the first run regressed two leaves because list 5's
element is decoded in MEMORY order, and the memory order of the rates frame is NOT its wire
order. Memory member 1 is wire member SRsc; six members unread. Lane CB's decoder has the same
defect and should drop its list-5 record.

Two new addresses (the second and third gate-loop heads) in ghidra/addresses.d/lane-rb.json.
This commit is contained in:
alex 2026-09-08 19:01:35 -04:00
parent af67d31394
commit a4a1d373f1
6 changed files with 836 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

280
findings/subsystems/turn-command-replay.md Normal file
View file

@ -0,0 +1,280 @@
# Replaying a recorded turn: what a command stream reproduces, and what it does not
Lane RB, 2026-09-08. Host work, no VM. Engine worktree `wip/rb` off `main` `4f25f1e`; predictions
committed as `sots-engine` `docs/RB-predictions.md` at `1e6474b`, **before** the module existed
(rule 2). Every number below was taken on a **fresh `build-host`** created with `rm -rf` (rule 24).
Consumes: `ai-order-emission.md` (AI4, the apply order and the cost table), `ai-order-capture.md`
(L4, the live dumps), `turncommands-block.md` (Q, the wire shape),
`turn1-to-turn2-nondeterminism.md` (L5). Produces the first end-to-end **turn record** the
campaign holds.
---
## 0. Lead: `ModCount` is reachable, and it is the only leaf a stream closes on the reference turn
`/Sim/ModCount` has been unreachable from a save for the whole campaign, because it counts a
thing a save does not contain. Our standalone wrote **14** where the original writes **24**: the
two driver bumps and nothing else. Replaying the turn's recorded command blocks puts it on **24**,
exactly, with no fitting and no fudge term.
Canonical pair `turn2-state.sav -> turn3-state.sav`, fresh build:
| | leaves diverging | closed | regressed |
|---|---:|---:|---:|
| do nothing | 108 | — | — |
| standalone, no stream | 63 | 45 | 0 |
| **standalone + recorded stream** | **62** | **46** | **0** |
The one leaf the stream closes is `/Sim/ModCount`, and its twenty-four decompose with **zero
residual**:
```
2 turn drivers (S00, T00) -- ours already
4 research-rate gates, one per submitting block (16, 32, 496, 512)
1 list 5 system rates
1 list 3 build order
1 list 10 (unnamed)
2 list 14 fleet task, modes 0 and 1 against ONE fleet
1 list 8 fleet move
0 list 23 population -- the 17..27 half is free
--
12 delta, on a save that carried 12
```
**Nothing else closes, and that is the honest result rather than a disappointment.** Of the ten
commands, three have a handler this engine holds and all three are no-ops on this workload — the
AI re-issues the research rate and the planetary-budget sliders the save already carries. The
other seven need subsystems we do not have (§4).
The secondary pair is where the stream does work, and it produces the campaign's first **matched
triple** — an input save, the stream captured from the run that consumed it, and *that run's own
autosave*:
| oracle | closed by the stream | still diverging |
|---|---:|---|
| the recording's own autosave (`l4-turn1to2-instrumented-autosave.sav`) | **7** | — |
| the historical `turn2-state.sav`, from a different process | **6** | player 512's `ResTNm` |
Both at **regressed 0**. The seven are `ModCount` (0 → 12) plus `ResRate` and `ResTNm` on players
32, 496 and 512. Against the historical oracle only six close, and the one that does not is
**exactly** the leaf lanes L4 and L5 showed is decided per-process: the recording picked
`XNC_TrnsMorr2`, `turn2-state.sav` holds `BIO_GnMod`. That was written down as a prediction before
the run and it is the strongest thing in this lane — a replay that had closed it would have meant
something was copying the oracle instead of the capture.
---
## 1. The on-disk form: `.tcb`
Line-oriented, whitespace-separated, `#` comments, magic first. A shim can emit it with `fprintf`
and a lane can read it without a tool.
```
tcb 1
meta source l4-turn2to3-aiorders.txt
seed <netId> <32-bit word> -- one per AI client; absent means "not recorded"
block <idx> <playerId> -- the batch SLOT, ascending; playerId 0 = a slot no client wrote
gate <idx> rate|target|boost|group4|group5|civilian <payload...>
list <idx> <listNo> <count> -- required for every non-empty list
elem <idx> <listNo> <elemIdx> <field>... -- fields in WIRE order
```
A field is one token: `iN` int, `fN` float, `b0`/`b1` bool, `s:TEXT` string, `vN` a counted vector
whose length is known and whose values are not, `vN:a,b,c` one that is fully read, and `?` a scalar
the instrument could not reach.
Three parts of that are not decoration.
**`?` and bare `vN` are the point.** A dump reads a fixed window of each element and cannot follow
a pointer, so a route arrives as "one element, contents unknown". That is a different fact from
"no element" and from "an element of zeros": the command still costs its bump and its effect still
cannot be applied. Recording the ignorance is what lets the counter be right while the state is
honestly left alone. The reference turn has **four** such elements out of ten.
**A declared count that disagrees with the elements present is an ERROR, not a warning.** A capture
that lost an element would otherwise produce a counter quietly one short, and nothing downstream
could tell that from a turn that really issued one fewer command. Thirteen malformed captures are
in the engine's test as rejection cases.
**`seed` is carried even though nothing consumes it.** Lane L1 established that each AI client
seeds one MT19937 with a single word at construction, so a decision is a function of *(board,
seed)* and a capture that records the seeds can be **re-derived** rather than replayed. The field
exists now so a capture taken today is still the right file when `game/ai` lands. A reference save
whose seeds were never logged — `turn2-state.sav` — is not reproducible by any process, the
original included, and its commands can only ever be replayed.
### 1.1 Two capture tools, one reader — and a defect one of them shares with my first attempt
Lane CB was building `tools/turncommands_capture.py` in parallel, emitting **JSON**. That is the
better *capture of record* and it should stay: raw element words as ground truth, heap vectors and
strings the deep dump followed, the AI seeds, the container self-check, and — the part neither of
my files had — the input save's hash **bound to the output autosaves' hashes**, so a capture cannot
be silently used against the wrong save.
Rather than a second format in the engine, `tools/tcb_from_json.py` joins them: CB's JSON stays the
capture of record, `.tcb` stays the engine's parser-free input, and one narrow script knows both.
Fed CB's tool's own output over the L4 log, the adapter produces a replay **byte-identical** to the
one from `aiorders_to_tcb.py`. Honest bound on that agreement: both decoders read the *same log*,
so it checks the two decoders against each other and not the log.
**And it caught a defect in CB's decoder, which is the same one I shipped and measured (§3):** its
list-5 record maps the element's memory words straight onto the frame's wire order
(`{systemId, ship, terraform, sciences, …}`). That is wrong by at least one position. The adapter
overrides it to `?` and says why in its own docstring; **CB should drop the list-5 record from
`RECORDS` rather than rely on the adapter to mask it**, because the JSON is the capture of record
and a wrong typing in it will outlive this note. CB's `raw_words` are unaffected and remain right.
Two things CB's dump has that L4's does not, and that the `.tcb` format already has fields for:
`aiseed` (the per-client construction seeds) and `aivec` (the heap vectors — which turn list 8's
route from `v1` into `v1:<hop>` and remove one whole row from the gap list in §4).
**The converter.** `tools/aiorders_to_tcb.py` turns lane L4's shim dump into a `.tcb` mechanically:
it applies the per-list field mapping, undoes the one list whose writer runs backwards, reinterprets
the words the record says are floats, turns a vector's begin/end pair into a length, and writes `?`
where the window could not reach. Both existing captures are converted and checked in at
`verify/results/turncommands/l4-turn{1to2,2to3}.tcb`. **Lane CB does not need to write an emitter:**
the existing `aiorders=on` hook plus this converter already produces the file. If CB does emit
`.tcb` directly, the seed and `name` records are the two things the current dump has no field for.
---
## 2. Apply order, and how it was verified
The batch is a flat run of thirty steps: twenty-seven per-**list** steps, each looping over *every*
player's block before the next step begins, with three per-**player** gate loops spliced in.
```
lists 6 11 20 19 17 18 5 23 24
gate loop A { group5 (free), research target (bump), research rate (bump) } 0x0088fdb0
lists 1 4 3 21 2 22 9 10 12 13 14 15 16 7 8 25 27 26
gate loop B { research boost (bump) } 0x008907b1
gate loop C { group 4 (bump) } 0x0089080a
```
So **one player's list-6 commands are applied before another player's list-11 commands**, the list
sequence starts at 6 and is not 1..27, and the member offsets it walks are not ascending either.
**How it was verified, and what the verification cannot show.**
1. **Address monotonicity, nine of thirty positions.** Six appliers are inlined into the batch and
each writes `ModCount` in place, so a watchpoint run recovered their addresses; the three gate
loop heads are known too. In schedule order those nine are `0x0088fdb0`, `0x0088fe0a`,
`0x008902fe`, `0x008903b9`, `0x0089046c`, `0x008905c8`, `0x008907b1`, `0x008907bc`,
`0x0089080a` — **strictly increasing**, at steps 9, 9, 18, 19, 20, 23, 28, 28, 29. That chain is
an independent re-derivation. The other twenty-one lists' handlers are out of line and this lane
has no record of their call-site addresses inside the batch, so their relative order is
**inherited** from AI4's read of the `add edi, imm` chain, not re-derived. The test says nine.
2. **Bijection and non-sortedness**, asserted: every list exactly once, every located gate exactly
once, and neither the list sequence nor the offset sequence ascending. That is what rules out the
two obvious wrong implementations — `for (list = 1..27)` and walking the block in memory order.
3. **The civilian-ratios gate is absent from the schedule on purpose.** It has no located applier
anywhere in the routine, so including it would be claiming a cost of zero for something whose
cost is unknown. It is caught separately and makes the whole count report itself as a lower
bound.
**Stated plainly: apply order is unfalsifiable on every workload the campaign holds.** Both captures
put every non-empty list on one player and every command on one system, so any permutation produces
the same save and the same count. The order is implemented for the workload that will need it, and
tested against the instruction stream rather than against an outcome. The workload that would make
it falsifiable is **two players commanding the same object in one turn** — the cheapest is a
two-human `/concurrent` game (lane G2's Tier 0), where both clients order fleets at one system.
---
## 3. A falsified prediction, and the finding it paid for
**RB-P3 predicted `regressed 0`. The first run regressed two leaves**, and the cause is a real fact
about the original.
The converter mapped list 5's dumped words straight onto the wire order of the rates frame
(`SRs, SRt, SRsc, SRtf, SRi, SRoh, SRnr`). The replay then wrote the AI's single non-zero slider
into `SRt`, and `Sys[288 "Ke'Dolarra"]` came out with `SRt 1.0 / SRsc 0.0` against an oracle
holding `SRt 0.0 / SRsc 1.0`.
**The memory field order of `Game::StarSystem::OutputRates` is not its wire order.** What is known
precisely:
* the only non-zero word in every dumped element of list 5, on both turns, is at **memory index 2**
of the element (index 0 is the system id, so it is the frame's **second** member);
* the same command **on the wire**, in a save that carries issued orders, puts its only non-zero in
**`SRsc`**, the frame's **third** member (lane Q, `SAVE_FORMAT.md` §11, cross-checked to the
Planetary Budget slider pushed fully to Construction);
* `turn2-state.sav` and `turn3-state.sav` both hold `Sys[288] Rts = {SRs 0, SRt 0, SRsc 1.0, SRtf 0,
SRi 0, SRoh 0, SRnr 0}` — so the AI's command re-issues the state the save already holds, which is
why a *correct* applier is invisible and an incorrect one is immediately visible.
So **memory member 1 is wire member `SRsc`**: one correspondence pinned, six unread. One non-zero
slider cannot determine a permutation of seven, and the converter no longer pretends otherwise — it
emits the system id and seven `?`, and the replayer counts the command and declines it.
Two experiments settle it, both cheap:
* **one UI run** — push two *different* sliders to two *different* values on one system, End Turn
with `aiorders=on`, and read the permutation straight off the element;
* **cheaper, and no VM at all** — a save taken after issuing rates carries the same command on the
**wire**, where every field is NAMED. `zuul-turn17-orders2.sav` has one. A `.tcb` converted from a
save's own `TurnCommands_v5` block needs no memory mapping, and would also give the `.tcb` format
a second, independent producer. **This is the highest-value next step on the capture side** and it
is pure host work.
Two notes on how this was caught, because they generalise. The plain closed/regressed measurement
found it, before the control did — but the control (RB-P4, `--replay-count-only` vs a full replay
must be byte-identical when every modelled handler is a no-op) would have found it too, and it is
what now stands guard: on the canonical pair those two runs are byte-identical, which is the
evidence that the handlers we *do* run write where they claim to rather than agreeing with the
oracle by luck. And the reason a wrong write was visible at all is that the command re-issues
existing state: **a command that re-states the board is the best possible test of an applier**, and
the corpus is full of them.
---
## 4. What a *complete* replay needs that this lane does not have
This is the real Rung B backlog. Each row is the reason one command in the reference turn is counted
but not applied.
| # | needed for | what is missing | shape |
|---|---|---|---|
| 1 | **list 3 build, list 1 design** | **ship construction.** No phase in this engine builds a ship. The command carries a queue ordinal, a design id and a system; the effect is `srb`/`sri`, `Maint`, the savings debit, a hull id and `ShipIDs`. Nine leaves on the canonical pair. | engine work; lane B6/E2 have the map |
| 2 | **list 1, list 3, list 8, list 10, list 14** | **the client's id allocator.** The build order names design **18** and the fleet order names fleet **34** — objects that do not exist in the input save, allocated *client-side before submission*, while the server's own master counter (`NMnx`) issued 1712 and 1776 the same turn. **Two id spaces**, and the small one is part of the wire protocol. A reimplementation that allocates on apply produces a structurally correct save with every AI-created id wrong. | **a watchpoint**, not a week of reading: break on the write that produces 18 and 34 |
| 3 | **list 8 fleet move** | **the route's hops.** The capture records the route's length and not its contents, because the dump does not follow the vector. One more indirection in the dumper. Also needs (2): the route belongs to a fleet the save does not contain. | one dumper change |
| 4 | **list 5 system rates** | **the memory field order of the rates frame** (§3). One correspondence pinned, six unread. | one UI run, or one save-sourced capture |
| 5 | **list 10** | **a name and a meaning.** `{systemId, fleetId, counted vector}`; "assign these ships to this fleet at this system" fits and has never been tested. The vector's contents are unread. | a hook on `0x0088bed0` |
| 6 | **list 14 fleet task** | **what the two modes do.** Two elements per AI fleet order, modes 0 then 1; the interface deposits one, mode 0 only. The *cost* is settled; the *effect* is not read at all. | a hook on the inlined applier |
| 7 | **list 23 population** | **the `Population` body**, 24 bytes behind a vftable that the capture window does not follow. Free in `ModCount`, so it has never been forced. | one dumper change |
| 8 | **the research-target gate** | **the techId → tech-name map.** The wire carries an integer (144, 90, 288); the save carries a name. The client resolves it *off the command* and passes a `char*`. The ids are not `index * 16` and are not indices into anything we hold. Today the capture can carry the observed name and the replay TRANSCRIBES it — reported in its own column, because that is not a reimplementation. **Three data points now exist**: 144 → `IND_Waldo`, 90 → `DRV_PlsFiss`, 288 → `XNC_TrnsMorr2`. | a watchpoint at the gate payload, or a read of `0x006c8580` |
| 9 | **the research-boost gate** | savings spent to advance research; both halves unmodelled. Never observed set on an AI turn. | engine work |
| 10 | **the group-4 and group-5 gates** | no read semantics; group 5 is Hiver-only and no save carries it. | rule 6 — a manufactured Hiver workload |
| 11 | **the civilian-ratios gate** | **no applier located anywhere in the batch.** Its `ModCount` cost is *unknown*, not zero, and any turn that sets it reports a lower bound. | an image-wide search for the consumer |
| 12 | **the load-time batch** | the process applies a batch at LOAD as well as at End Turn (`seq=1`, `n=1`, the local client's block alone, with the rate gate SET). Whether it charges `ModCount` is **untested**, and this lane excludes it by construction. If it does charge, a save loaded and immediately re-saved reads one higher. | one run: load, save, compare |
| 13 | **the whole thing, on any interesting board** | **two turns, one AI empire, 28 stars, no contact.** Lists 2, 4, 6, 9, 11–13, 15–22 and 24–27 have never been non-empty. Apply order is unfalsifiable (§2). Nothing here generalises past a very quiet game. | manufacture the workload |
Two things NOT on that list, deliberately. `Summary.Checksum` moves whenever anything else does and
its inputs are unread — no command replay will touch it until the rest is right. And nothing in
either capture draws a generator word, so the RNG frame is untouched by the stream; the coordinator's
note that the replay interval begins at `BeginProcessTurn` rather than `ProcessTurn` is already
satisfied here — the batch is drained *before* the first phase runs, which is where the End-Turn
dispatcher calls it (`ApplyAllTurnCommands` at `+0x00784904`, `ProcessTurn` at `+0x0078491c`,
`OnAllCombatDone_Tail` at `+0x00784d07`) — but the **hive-registration draws lane L1 found inside
`BeginProcessTurn` are a separate interval and this lane does not model them.**
---
## 5. What this lane did not do
1. **No VM time and no new instrument.** Every live number here is lane L4's, re-read through a
converter. Nothing new ran under a hook.
2. **The CT111 shim cross-build was not run** — CT111 refused the key from this host. `src/game/ai`
is in the cross build, so per rule 13 **the integrator must run it before pushing**. The two
32-bit-specific hazards were audited by hand and one was real and is fixed: `strtoul` plus a
`> 0xffffffffUL` test is tautological where `unsigned long` is 32 bits, which would both accept
out-of-range input silently and trip `-Wextra`/`-Werror`; both sites now use `strtoull`. `%zu`
appears in the new code and also in `src/app/main.cpp` today, so it is not a new exposure.
3. **The apply order was not tested against an outcome** and cannot be on this corpus (§2).
4. **`ToTurnCommandBlock` reuses the existing, already-verified cost model** rather than counting a
second time — deliberately, so there is one implementation of the arithmetic and not two that
can agree with each other while both being wrong (rule 8).
5. **The `.tcb` format has exactly one producer.** Until the save-sourced converter of §3 exists,
every capture comes through one script and a bug in it is invisible.
6. **Seeds are carried and never used.** No capture in the corpus has any.

18
ghidra/addresses.d/lane-rb.json Normal file
View file

@ -0,0 +1,18 @@
{
"entries": [
{
"name": "StrategySim_ApplyTurnCommandBatch_GateLoopB",
"addr": "0x008907b1",
"convention": "label",
"status": "verified",
"prototype": "the SECOND of the three per-player gate loops inside StrategySim::ApplyTurnCommandBatch. `esi = block+0x20`; it tests the research-boost gate at +0x20 and applies its {spend, fraction} payload through 0x00820560, bumping ModCount inline at 0x008907bc. It runs AFTER all twenty-seven list loops, not with the other gates -- the six prologue gates are split across three loops at three points in the routine, which is why a port that applies them together as a prologue gets the order wrong"
},
{
"name": "StrategySim_ApplyTurnCommandBatch_GateLoopC",
"addr": "0x0089080a",
"convention": "label",
"status": "verified",
"prototype": "the THIRD and last per-player gate loop inside StrategySim::ApplyTurnCommandBatch. `esi = block+0x24`; it tests the group-4 gate at +0x2c and applies its {bool, int} payload through 0x00821b90. It is the final step of the whole batch. Together with the loop-A head at 0x0088fdb0 and the loop-B head at 0x008907b1, and the six inlined ModCount bump sites, this gives nine positions of the thirty-step apply schedule an address-monotonicity check -- the only part of the sequence that can be re-derived rather than inherited from the read of the `add edi, imm` chain"
}
]
}

250
tools/aiorders_to_tcb.py Normal file
View file

@ -0,0 +1,250 @@
#!/usr/bin/env python3
"""Turn a live `aiorders` shim dump into a `.tcb` turn-command capture.
The shim's dump is a memory view: for each submitted block it prints the six prologue gates,
the twenty-seven list lengths, and a fixed 48-byte window of every element. That is the right
thing for an instrument to emit -- it commits to nothing -- but it is not a turn record, because
the fields are in MEMORY order (and one list's writer runs them backwards), the floats are still
integers, and a payload behind a pointer is simply absent.
This is the mechanical step in between. It applies the per-list field mapping, reinterprets the
words the element record says are floats, converts a vector's begin/end pair into a length, and
writes `?` wherever the dump window could not reach. The output is what `sots_turn
--turn-commands` reads.
tools/aiorders_to_tcb.py LOG [-o OUT] [--batch SEQ] [--name PID=TECHNAME]... [--seed ID=HEX]...
Two things this tool deliberately will not do.
* It will not invent a payload. A route whose hops live behind a pointer becomes `vN` -- a
vector of known length and unknown contents -- and never `v1:0`. The replayer counts such a
command and refuses to apply it, which is the whole point: a command that was issued and a
command whose effect we can reproduce are different facts.
* It will not pick a batch for you when the choice is ambiguous. A process applies a command
batch at LOAD as well as at End Turn; replaying the load-time one against a save that was
written after it would double-count. The default is the LAST batch in the log, which is the
End-Turn one, and `--batch` overrides it.
`--name PID=TECHNAME` records the tech NAME an instrument observed a research-target gate resolve
to. The wire carries an integer id and the save carries a name; the client resolves one to the
other off the command and that map is unread, so the name can only be recorded, never computed.
`--seed NETID=VALUE` records an AI client's construction seed.
"""
import argparse
import re
import struct
import sys
# Per list: how the 48-byte memory window maps onto the element's wire fields.
#
# `order` is the sequence of word indices to emit; `kind` says how to read each one. The build
# list is the only one that reverses -- its writer emits +0x14, +0x10, +0x0c, +0x08, descending --
# and that is per-list, not a rule. Every other observed list writes ascending.
#
# i an int word
# f a word to reinterpret as a float
# b a word whose low byte is a bool
# v a std::vector: this word and the next hold begin/end, and the length is their difference
# over four. The CONTENTS are behind the pointer and the dump does not follow it, so the
# field is emitted with a length and no values.
# ? a word the element record says is part of an object this window cannot read
LIST_MAP = {
1: [("?", 0)], # a ShipDesignDef object
3: [("i", 3), ("i", 2), ("i", 1), ("i", 0)], # DESCENDING: ordinal, design, system, w
# List 5 carries a system id and then the seven planetary-budget sliders. The id is certain;
# THE SLIDERS ARE NOT, and this used to map them straight onto the wire order and was wrong.
# A replay built on that mapping wrote the AI's one non-zero slider into the wrong member and
# regressed two leaves on the canonical pair -- the command re-issues the rates the save
# already holds, so a correct applier is a no-op and an incorrect one is immediately visible.
#
# What is known: the only non-zero word in every dumped element of this list is at index 2,
# and the same command on the WIRE (where the frame is named) puts its only non-zero in
# `SRsc`, the THIRD member. So memory member 1 is wire member `SRsc` and the frame's memory
# order is not its wire order -- one correspondence pinned, six unread. The `?` below is
# that ignorance, and the replayer counts such a command and refuses to apply it.
#
# The experiment that settles it is one UI run: push two DIFFERENT sliders to two different
# values, capture the block, and read the permutation straight off. Cheaper still, a save
# taken after issuing rates carries the same command on the wire with every field NAMED, so a
# capture converted from a save's own TurnCommands block needs no memory mapping at all.
5: [("i", 0)] + [("?", k) for k in range(1, 8)],
7: [("i", 0), ("i", 1)],
8: [("i", 0), ("v", 1)],
10: [("i", 0), ("i", 1), ("v", 2)],
14: [("i", 0), ("i", 1), ("b", 2)],
23: [("i", 0), ("?", 1)], # a Population object
}
BLK_RE = re.compile(
r"^aiblk seq=(\d+) blk=(\d+)/(\d+) at=\S+ pid=(-?\d+) "
r"rate=(\d):(\S+) target=(\d):(-?\d+) boost=(\d):(-?\d+),(\S+) g4=(\d):(-?\d+),(-?\d+) "
r"f3=(\d):(\S+),(\S+),(\S+) civ=(\d)")
LISTS_RE = re.compile(r"^ailists seq=(\d+) blk=(\d+) pid=(-?\d+) nonEmpty=\d+ sizes\(1\.\.27\)=\[([^\]]*)\]")
ELEM_RE = re.compile(r"^aielem blk=(\d+) pid=(-?\d+) list=(\d+) idx=(\d+) .*? hex=\[([^\]]*)\]")
def f32(word):
"""A dumped word, as the float32 it is, printed so it round-trips exactly."""
return "%.9g" % struct.unpack("<f", struct.pack("<I", word & 0xffffffff))[0]
def parse_log(path):
"""-> {seq: {"n": int, "blocks": {idx: block}}}"""
batches = {}
def batch(seq):
return batches.setdefault(seq, {"n": 0, "blocks": {}})
with open(path, encoding="utf-8", errors="replace") as f:
for line in f:
m = BLK_RE.match(line)
if m:
seq, idx, n = int(m.group(1)), int(m.group(2)), int(m.group(3))
b = batch(seq)
b["n"] = max(b["n"], n)
b["blocks"][idx] = {
"pid": int(m.group(4)),
"rate": (m.group(5) == "1", m.group(6)),
"target": (m.group(7) == "1", int(m.group(8))),
"boost": (m.group(9) == "1", int(m.group(10)), m.group(11)),
"g4": (m.group(12) == "1", int(m.group(13)), int(m.group(14))),
"f3": (m.group(15) == "1", m.group(16), m.group(17), m.group(18)),
"civ": m.group(19) == "1",
"sizes": [0] * 27,
"elems": {},
}
continue
m = LISTS_RE.match(line)
if m:
seq, idx = int(m.group(1)), int(m.group(2))
blk = batch(seq)["blocks"].get(idx)
if blk is not None:
blk["sizes"] = [int(x) for x in m.group(4).split()]
continue
m = ELEM_RE.match(line)
if m:
# An `aielem` line carries no seq, so it belongs to the batch whose block header
# it followed -- which is the most recent one seen.
seq = max(batches) if batches else 0
idx, listno, elemidx = int(m.group(1)), int(m.group(3)), int(m.group(4))
words = [int(x, 16) for x in m.group(5).split()]
blk = batch(seq)["blocks"].get(idx)
if blk is not None:
blk["elems"].setdefault(listno, {})[elemidx] = words
return batches
def element_fields(listno, words):
"""The wire fields of one element, as `.tcb` tokens."""
spec = LIST_MAP.get(listno)
if spec is None:
# An unmapped list: record that the element exists and nothing about it. The command
# still costs whatever its list costs; the replayer will decline to apply it.
return ["?"]
out = []
for kind, k in spec:
if k >= len(words):
out.append("?")
continue
w = words[k]
if kind == "i":
out.append("i%d" % struct.unpack("<i", struct.pack("<I", w))[0])
elif kind == "f":
out.append("f%s" % f32(w))
elif kind == "b":
out.append("b%d" % (1 if (w & 0xff) else 0))
elif kind == "v":
if k + 1 >= len(words):
out.append("?")
else:
out.append("v%d" % ((words[k + 1] - w) // 4))
else:
out.append("?")
return out
def emit(batch, seq, source, names, seeds):
lines = ["tcb 1",
"meta source %s" % source,
"meta batch seq=%d n=%d" % (seq, batch["n"]),
"meta note the load-time batch is excluded; this is the End-Turn submission"]
for netid, value in seeds:
lines.append("seed %s %s" % (netid, value))
for idx in sorted(batch["blocks"]):
b = batch["blocks"][idx]
lines.append("block %d %d" % (idx, b["pid"]))
if b["rate"][0]:
lines.append("gate %d rate %s" % (idx, b["rate"][1]))
if b["target"][0]:
name = names.get(b["pid"])
lines.append("gate %d target %d%s" % (idx, b["target"][1],
(" name %s" % name) if name else ""))
if b["boost"][0]:
lines.append("gate %d boost %d %s" % (idx, b["boost"][1], b["boost"][2]))
if b["g4"][0]:
lines.append("gate %d group4 %d %d" % (idx, b["g4"][1], b["g4"][2]))
if b["f3"][0]:
lines.append("gate %d group5 %s %s %s" % (idx, b["f3"][1], b["f3"][2], b["f3"][3]))
if b["civ"]:
lines.append("gate %d civilian" % idx)
for listno in range(1, 28):
n = b["sizes"][listno - 1]
if not n:
continue
lines.append("list %d %d %d" % (idx, listno, n))
have = b["elems"].get(listno, {})
for e in range(n):
words = have.get(e)
fields = element_fields(listno, words) if words is not None else ["?"]
lines.append("elem %d %d %d %s" % (idx, listno, e, " ".join(fields)))
return "\n".join(lines) + "\n"
def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("log")
ap.add_argument("-o", "--out")
ap.add_argument("--batch", type=int, help="which seq to convert (default: the last)")
ap.add_argument("--name", action="append", default=[], metavar="PID=TECHNAME",
help="the tech name a research-target gate was observed to resolve to")
ap.add_argument("--seed", action="append", default=[], metavar="NETID=VALUE",
help="an AI client's construction seed")
a = ap.parse_args(argv)
batches = parse_log(a.log)
if not batches:
print("no aiblk records in %s" % a.log, file=sys.stderr)
return 2
seq = a.batch if a.batch is not None else max(batches)
if seq not in batches:
print("no batch seq=%d; the log holds %s" % (seq, sorted(batches)), file=sys.stderr)
return 2
names = {}
for n in a.name:
pid, _, tech = n.partition("=")
if not tech:
print("--name wants PID=TECHNAME", file=sys.stderr)
return 2
names[int(pid)] = tech
seeds = []
for s in a.seed:
netid, _, value = s.partition("=")
if not value:
print("--seed wants NETID=VALUE", file=sys.stderr)
return 2
seeds.append((netid, value))
text = emit(batches[seq], seq, a.log.split("/")[-1], names, seeds)
if a.out:
with open(a.out, "w") as f:
f.write(text)
print("wrote %s (batch seq=%d, %d blocks)" % (a.out, seq, len(batches[seq]["blocks"])))
else:
sys.stdout.write(text)
return 0
if __name__ == "__main__":
raise SystemExit(main())

232
tools/tcb_from_json.py Normal file
View file

@ -0,0 +1,232 @@
#!/usr/bin/env python3
"""Convert a lane-CB JSON turn-command capture into the `.tcb` file `sots_turn` reads.
TWO FORMATS ON PURPOSE, AND THIS IS THE JOIN.
`tools/turncommands_capture.py` (lane CB) produces the **capture of record**: raw element words as
ground truth, heap vectors and strings the deep dump followed, the input save's hash bound to the
output autosaves' hashes, the per-client AI seeds, and the container self-check. That is what an
experiment should leave behind and none of it belongs in an engine's input file.
`.tcb` is the **engine's input**: line-oriented, no parser, nothing but the commands and their
provenance. `sots_turn --turn-commands` reads it and nothing else, so the engine never grows a
JSON reader and never has an opinion about how a capture was taken.
This script is the only thing that has to know both, and it is deliberately the narrow part:
it takes `decoded.wire` where lane CB's decoder produced one, falls back to raw words where it did
not, and writes `?` for every field neither could reach.
tools/tcb_from_json.py CAPTURE.json [-o OUT.tcb] [--batch SEQ]
ONE FIELD MAPPING IS KNOWN WRONG AND IS OVERRIDDEN HERE, WITH ITS REASON.
List 5 (system rates) is decoded by lane CB as `{systemId, ship, terraform, sciences, ...}` --
the element's memory words mapped straight onto the frame's WIRE order. Lane RB measured that and
it is wrong: replaying it wrote the AI's one non-zero slider into `SRt` and regressed two leaves
on the canonical pair, where the oracle holds `SRsc = 1.0`. The only non-zero word of every list-5
element ever dumped is at memory index 2, and the same command on the wire -- where the frame is
NAMED -- puts its only non-zero in `SRsc`, the third member. So memory member 1 is wire member
`SRsc`, and the frame's memory order is not its wire order: ONE correspondence pinned, six unread.
Rather than carry a mapping that is known to be off by at least one, this converter emits the
system id and seven `?`. The replayer then counts the command -- the count is right either way --
and refuses to apply it, which is the correct behaviour for a payload nobody has read.
Settling it is one run: push two DIFFERENT sliders to two DIFFERENT values and read the
permutation off the element. Cheaper, a save taken after issuing rates carries the same command on
the wire with every field named, and needs no memory mapping at all.
"""
import argparse
import json
import struct
import sys
# Lists whose decoded `wire` this converter trusts. List 5 is deliberately absent (see above);
# lists with no entry are carried as a single `?`, which counts the command and applies nothing.
TRUSTED = {
3: "iiii", # ordinal, designId, systemId, trailing
7: "ii", # shipId, trailing
14: "iib", # fleetId, mode, flag
}
# Lists whose element leads with scalars and then a counted vector the deep dump may have read.
VECTOR_TAIL = {
8: ("i", 1), # fleetId, then the route
10: ("ii", 2), # systemId, fleetId, then a counted vector
}
UNMAPPED_HEAD = {
5: (1, 7), # one trusted leading int (the system id) and seven unread fields
23: (1, 1), # the system id and a Population body behind a vftable
}
def as_int(v):
if isinstance(v, bool):
return 1 if v else 0
if isinstance(v, float):
return int(v)
return int(v)
def tok_i(v):
return "i%d" % as_int(v)
def fields_for(list_no, elem):
"""The `.tcb` field tokens for one element."""
decoded = elem.get("decoded") or {}
wire = decoded.get("wire")
raw = elem.get("raw_words") or []
vectors = {v["at_word"]: v for v in (elem.get("vectors") or [])}
if list_no in UNMAPPED_HEAD:
lead, unread = UNMAPPED_HEAD[list_no]
head = [tok_i(raw[i]) for i in range(min(lead, len(raw)))]
return head + ["?"] * unread if head else ["?"]
if list_no in TRUSTED:
spec = TRUSTED[list_no]
if not wire or len(wire) < len(spec):
return ["?"]
out = []
for i, kind in enumerate(spec):
v = wire[i]
out.append("b%d" % (1 if v else 0) if kind == "b" else tok_i(v))
return out
if list_no in VECTOR_TAIL:
spec, vec_word = VECTOR_TAIL[list_no]
if len(raw) < len(spec):
return ["?"]
out = [tok_i(struct.unpack("<i", struct.pack("<I", raw[i] & 0xffffffff))[0])
for i in range(len(spec))]
v = vectors.get(vec_word)
if v is None:
# The dump did not follow the pointer. Its LENGTH is still derivable from the
# begin/end pair, and a length with no values is exactly the honest field: the
# command is counted and not applied.
if len(raw) > vec_word + 1:
out.append("v%d" % max(0, (raw[vec_word + 1] - raw[vec_word]) // 4))
else:
out.append("?")
elif v.get("truncated"):
out.append("v%d" % v["count"])
else:
vals = [struct.unpack("<i", struct.pack("<I", w & 0xffffffff))[0] for w in v["words"]]
out.append("v%d:%s" % (len(vals), ",".join(str(x) for x in vals)) if vals
else "v0:")
return out
return ["?"]
def emit(cap, batch):
src = cap.get("workload") or cap.get("note") or "lane-CB capture"
lines = ["tcb 1",
"meta source %s" % str(src).replace("\n", " "),
"meta lane %s" % cap.get("lane", "?"),
"meta guest %s" % cap.get("guest", "?"),
"meta build %s" % cap.get("build", "?"),
"meta captured %s" % cap.get("captured_utc", "?"),
"meta batch seq=%s n=%s" % (batch.get("seq"), batch.get("n"))]
binding = cap.get("binding") or {}
if binding.get("input"):
lines.append("meta input %s %s" % (binding["input"].get("file"),
binding["input"].get("sha256")))
for o in binding.get("outputs") or []:
lines.append("meta output %s %s%s" % (o.get("file"), o.get("sha256"),
"" if o.get("matches_oracle") is None
else (" matches_oracle=%s" % o["matches_oracle"])))
if cap.get("WARNING_INCOMPLETE"):
lines.append("meta warning heap-payloads-absent")
# The seeds. Without these the capture is a log file: the AI is one MT19937 per client seeded
# with one word, so the block is the AI's answer and the seed is its input.
for s in cap.get("ai_seeds") or []:
val = s.get("used") or s.get("observed")
if val is None:
continue
lines.append("seed %s %s%s" % (s.get("netId"), val,
""))
for blk in batch.get("blocks") or []:
lines.append("block %d %d" % (blk["index"], blk.get("playerId", blk.get("pid", 0))))
idx = blk["index"]
gates = blk.get("gates") or {}
def payload(name):
g = gates.get(name) or {}
return g.get("payload") if g.get("set") else None
p = payload("researchRate")
if p is not None:
lines.append("gate %d rate %s" % (idx, p))
p = payload("researchTarget")
if p is not None:
name = (blk.get("researchTargetName") or "").strip()
lines.append("gate %d target %s%s" % (idx, p, (" name %s" % name) if name else ""))
p = payload("researchBoost")
if p is not None:
spend, _, frac = p.partition(",")
lines.append("gate %d boost %s %s" % (idx, spend, frac or "0"))
p = payload("group4")
if p is not None:
flag, _, val = p.partition(",")
lines.append("gate %d group4 %s %s" % (idx, flag, val or "0"))
p = payload("group5")
if p is not None:
a, b, c = (p.split(",") + ["0", "0", "0"])[:3]
lines.append("gate %d group5 %s %s %s" % (idx, a, b, c))
if (gates.get("civilianRatios") or {}).get("set"):
lines.append("gate %d civilian" % idx)
for lt in sorted(blk.get("lists") or [], key=lambda l: l["list"]):
n = lt.get("size", 0)
if not n:
continue
lines.append("list %d %d %d" % (idx, lt["list"], n))
by_index = {e["index"]: e for e in lt.get("elements") or []}
for e in range(n):
elem = by_index.get(e)
f = fields_for(lt["list"], elem) if elem else ["?"]
lines.append("elem %d %d %d %s" % (idx, lt["list"], e, " ".join(f)))
return "\n".join(lines) + "\n"
def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("capture")
ap.add_argument("-o", "--out")
ap.add_argument("--batch", type=int,
help="which batch seq to convert (default: the last, which is the End-Turn "
"submission; a process also applies a batch at LOAD and replaying that "
"one against a save written after it would double-count)")
a = ap.parse_args(argv)
with open(a.capture) as f:
cap = json.load(f)
batches = cap.get("batches") or []
if not batches:
print("no batches in %s" % a.capture, file=sys.stderr)
return 2
if a.batch is not None:
picked = [b for b in batches if b.get("seq") == a.batch]
if not picked:
print("no batch seq=%d; the capture holds %s"
% (a.batch, [b.get("seq") for b in batches]), file=sys.stderr)
return 2
batch = picked[0]
else:
batch = batches[-1]
text = emit(cap, batch)
if a.out:
with open(a.out, "w") as f:
f.write(text)
print("wrote %s (batch seq=%s, %d blocks)"
% (a.out, batch.get("seq"), len(batch.get("blocks") or [])))
else:
sys.stdout.write(text)
return 0
if __name__ == "__main__":
raise SystemExit(main())

27
verify/results/turncommands/l4-turn1to2.tcb Normal file
View file

@ -0,0 +1,27 @@
tcb 1
meta source l4-turn1to2-aiorders.txt
meta batch seq=2 n=8
meta note the load-time batch is excluded; this is the End-Turn submission
block 0 16
gate 0 rate 0.25
block 1 32
gate 1 rate 0.8
gate 1 target 144 name IND_Waldo
list 1 1 1
elem 1 1 0 ?
list 1 3 1
elem 1 3 0 i1 i18 i288 i0
list 1 5 1
elem 1 5 0 i288 ? ? ? ? ? ? ?
list 1 23 1
elem 1 23 0 i288 ?
block 2 496
gate 2 rate 0.8
gate 2 target 90 name DRV_PlsFiss
block 3 512
gate 3 rate 0.8
gate 3 target 288 name XNC_TrnsMorr2
block 4 0
block 5 0
block 6 0
block 7 0

29
verify/results/turncommands/l4-turn2to3.tcb Normal file
View file

@ -0,0 +1,29 @@
tcb 1
meta source l4-turn2to3-aiorders.txt
meta batch seq=2 n=8
meta note the load-time batch is excluded; this is the End-Turn submission
block 0 16
gate 0 rate 0.25
block 1 32
gate 1 rate 0.8
list 1 3 1
elem 1 3 0 i2 i18 i288 i0
list 1 5 1
elem 1 5 0 i288 ? ? ? ? ? ? ?
list 1 8 1
elem 1 8 0 i34 v1
list 1 10 1
elem 1 10 0 i288 i34 v1
list 1 14 2
elem 1 14 0 i34 i0 b1
elem 1 14 1 i34 i1 b1
list 1 23 1
elem 1 23 0 i288 ?
block 2 496
gate 2 rate 0.8
block 3 512
gate 3 rate 0.8
block 4 0
block 5 0
block 6 0
block 7 0