diff --git a/findings/subsystems/turn-command-replay.md b/findings/subsystems/turn-command-replay.md new file mode 100644 index 0000000..0381865 --- /dev/null +++ b/findings/subsystems/turn-command-replay.md @@ -0,0 +1,280 @@ +# Replaying a recorded turn: what a command stream reproduces, and what it does not + +Lane RB, 2026-09-08. Host work, no VM. Engine worktree `wip/rb` off `main` `4f25f1e`; predictions +committed as `sots-engine` `docs/RB-predictions.md` at `1e6474b`, **before** the module existed +(rule 2). Every number below was taken on a **fresh `build-host`** created with `rm -rf` (rule 24). + +Consumes: `ai-order-emission.md` (AI4, the apply order and the cost table), `ai-order-capture.md` +(L4, the live dumps), `turncommands-block.md` (Q, the wire shape), +`turn1-to-turn2-nondeterminism.md` (L5). Produces the first end-to-end **turn record** the +campaign holds. + +--- + +## 0. Lead: `ModCount` is reachable, and it is the only leaf a stream closes on the reference turn + +`/Sim/ModCount` has been unreachable from a save for the whole campaign, because it counts a +thing a save does not contain. Our standalone wrote **14** where the original writes **24**: the +two driver bumps and nothing else. Replaying the turn's recorded command blocks puts it on **24**, +exactly, with no fitting and no fudge term. + +Canonical pair `turn2-state.sav -> turn3-state.sav`, fresh build: + +| | leaves diverging | closed | regressed | +|---|---:|---:|---:| +| do nothing | 108 | — | — | +| standalone, no stream | 63 | 45 | 0 | +| **standalone + recorded stream** | **62** | **46** | **0** | + +The one leaf the stream closes is `/Sim/ModCount`, and its twenty-four decompose with **zero +residual**: + +``` +2 turn drivers (S00, T00) -- ours already +4 research-rate gates, one per submitting block (16, 32, 496, 512) +1 list 5 system rates +1 list 3 build order +1 list 10 (unnamed) +2 list 14 fleet task, modes 0 and 1 against ONE fleet +1 list 8 fleet move +0 list 23 population -- the 17..27 half is free +-- +12 delta, on a save that carried 12 +``` + +**Nothing else closes, and that is the honest result rather than a disappointment.** Of the ten +commands, three have a handler this engine holds and all three are no-ops on this workload — the +AI re-issues the research rate and the planetary-budget sliders the save already carries. The +other seven need subsystems we do not have (§4). + +The secondary pair is where the stream does work, and it produces the campaign's first **matched +triple** — an input save, the stream captured from the run that consumed it, and *that run's own +autosave*: + +| oracle | closed by the stream | still diverging | +|---|---:|---| +| the recording's own autosave (`l4-turn1to2-instrumented-autosave.sav`) | **7** | — | +| the historical `turn2-state.sav`, from a different process | **6** | player 512's `ResTNm` | + +Both at **regressed 0**. The seven are `ModCount` (0 → 12) plus `ResRate` and `ResTNm` on players +32, 496 and 512. Against the historical oracle only six close, and the one that does not is +**exactly** the leaf lanes L4 and L5 showed is decided per-process: the recording picked +`XNC_TrnsMorr2`, `turn2-state.sav` holds `BIO_GnMod`. That was written down as a prediction before +the run and it is the strongest thing in this lane — a replay that had closed it would have meant +something was copying the oracle instead of the capture. + +--- + +## 1. The on-disk form: `.tcb` + +Line-oriented, whitespace-separated, `#` comments, magic first. A shim can emit it with `fprintf` +and a lane can read it without a tool. + +``` +tcb 1 +meta source l4-turn2to3-aiorders.txt +seed <32-bit word> -- one per AI client; absent means "not recorded" +block -- the batch SLOT, ascending; playerId 0 = a slot no client wrote +gate rate|target|boost|group4|group5|civilian +list -- required for every non-empty list +elem ... -- fields in WIRE order +``` + +A field is one token: `iN` int, `fN` float, `b0`/`b1` bool, `s:TEXT` string, `vN` a counted vector +whose length is known and whose values are not, `vN:a,b,c` one that is fully read, and `?` a scalar +the instrument could not reach. + +Three parts of that are not decoration. + +**`?` and bare `vN` are the point.** A dump reads a fixed window of each element and cannot follow +a pointer, so a route arrives as "one element, contents unknown". That is a different fact from +"no element" and from "an element of zeros": the command still costs its bump and its effect still +cannot be applied. Recording the ignorance is what lets the counter be right while the state is +honestly left alone. The reference turn has **four** such elements out of ten. + +**A declared count that disagrees with the elements present is an ERROR, not a warning.** A capture +that lost an element would otherwise produce a counter quietly one short, and nothing downstream +could tell that from a turn that really issued one fewer command. Thirteen malformed captures are +in the engine's test as rejection cases. + +**`seed` is carried even though nothing consumes it.** Lane L1 established that each AI client +seeds one MT19937 with a single word at construction, so a decision is a function of *(board, +seed)* and a capture that records the seeds can be **re-derived** rather than replayed. The field +exists now so a capture taken today is still the right file when `game/ai` lands. A reference save +whose seeds were never logged — `turn2-state.sav` — is not reproducible by any process, the +original included, and its commands can only ever be replayed. + +### 1.1 Two capture tools, one reader — and a defect one of them shares with my first attempt + +Lane CB was building `tools/turncommands_capture.py` in parallel, emitting **JSON**. That is the +better *capture of record* and it should stay: raw element words as ground truth, heap vectors and +strings the deep dump followed, the AI seeds, the container self-check, and — the part neither of +my files had — the input save's hash **bound to the output autosaves' hashes**, so a capture cannot +be silently used against the wrong save. + +Rather than a second format in the engine, `tools/tcb_from_json.py` joins them: CB's JSON stays the +capture of record, `.tcb` stays the engine's parser-free input, and one narrow script knows both. +Fed CB's tool's own output over the L4 log, the adapter produces a replay **byte-identical** to the +one from `aiorders_to_tcb.py`. Honest bound on that agreement: both decoders read the *same log*, +so it checks the two decoders against each other and not the log. + +**And it caught a defect in CB's decoder, which is the same one I shipped and measured (§3):** its +list-5 record maps the element's memory words straight onto the frame's wire order +(`{systemId, ship, terraform, sciences, …}`). That is wrong by at least one position. The adapter +overrides it to `?` and says why in its own docstring; **CB should drop the list-5 record from +`RECORDS` rather than rely on the adapter to mask it**, because the JSON is the capture of record +and a wrong typing in it will outlive this note. CB's `raw_words` are unaffected and remain right. + +Two things CB's dump has that L4's does not, and that the `.tcb` format already has fields for: +`aiseed` (the per-client construction seeds) and `aivec` (the heap vectors — which turn list 8's +route from `v1` into `v1:` and remove one whole row from the gap list in §4). + +**The converter.** `tools/aiorders_to_tcb.py` turns lane L4's shim dump into a `.tcb` mechanically: +it applies the per-list field mapping, undoes the one list whose writer runs backwards, reinterprets +the words the record says are floats, turns a vector's begin/end pair into a length, and writes `?` +where the window could not reach. Both existing captures are converted and checked in at +`verify/results/turncommands/l4-turn{1to2,2to3}.tcb`. **Lane CB does not need to write an emitter:** +the existing `aiorders=on` hook plus this converter already produces the file. If CB does emit +`.tcb` directly, the seed and `name` records are the two things the current dump has no field for. + +--- + +## 2. Apply order, and how it was verified + +The batch is a flat run of thirty steps: twenty-seven per-**list** steps, each looping over *every* +player's block before the next step begins, with three per-**player** gate loops spliced in. + +``` +lists 6 11 20 19 17 18 5 23 24 +gate loop A { group5 (free), research target (bump), research rate (bump) } 0x0088fdb0 +lists 1 4 3 21 2 22 9 10 12 13 14 15 16 7 8 25 27 26 +gate loop B { research boost (bump) } 0x008907b1 +gate loop C { group 4 (bump) } 0x0089080a +``` + +So **one player's list-6 commands are applied before another player's list-11 commands**, the list +sequence starts at 6 and is not 1..27, and the member offsets it walks are not ascending either. + +**How it was verified, and what the verification cannot show.** + +1. **Address monotonicity, nine of thirty positions.** Six appliers are inlined into the batch and + each writes `ModCount` in place, so a watchpoint run recovered their addresses; the three gate + loop heads are known too. In schedule order those nine are `0x0088fdb0`, `0x0088fe0a`, + `0x008902fe`, `0x008903b9`, `0x0089046c`, `0x008905c8`, `0x008907b1`, `0x008907bc`, + `0x0089080a` — **strictly increasing**, at steps 9, 9, 18, 19, 20, 23, 28, 28, 29. That chain is + an independent re-derivation. The other twenty-one lists' handlers are out of line and this lane + has no record of their call-site addresses inside the batch, so their relative order is + **inherited** from AI4's read of the `add edi, imm` chain, not re-derived. The test says nine. +2. **Bijection and non-sortedness**, asserted: every list exactly once, every located gate exactly + once, and neither the list sequence nor the offset sequence ascending. That is what rules out the + two obvious wrong implementations — `for (list = 1..27)` and walking the block in memory order. +3. **The civilian-ratios gate is absent from the schedule on purpose.** It has no located applier + anywhere in the routine, so including it would be claiming a cost of zero for something whose + cost is unknown. It is caught separately and makes the whole count report itself as a lower + bound. + +**Stated plainly: apply order is unfalsifiable on every workload the campaign holds.** Both captures +put every non-empty list on one player and every command on one system, so any permutation produces +the same save and the same count. The order is implemented for the workload that will need it, and +tested against the instruction stream rather than against an outcome. The workload that would make +it falsifiable is **two players commanding the same object in one turn** — the cheapest is a +two-human `/concurrent` game (lane G2's Tier 0), where both clients order fleets at one system. + +--- + +## 3. A falsified prediction, and the finding it paid for + +**RB-P3 predicted `regressed 0`. The first run regressed two leaves**, and the cause is a real fact +about the original. + +The converter mapped list 5's dumped words straight onto the wire order of the rates frame +(`SRs, SRt, SRsc, SRtf, SRi, SRoh, SRnr`). The replay then wrote the AI's single non-zero slider +into `SRt`, and `Sys[288 "Ke'Dolarra"]` came out with `SRt 1.0 / SRsc 0.0` against an oracle +holding `SRt 0.0 / SRsc 1.0`. + +**The memory field order of `Game::StarSystem::OutputRates` is not its wire order.** What is known +precisely: + +* the only non-zero word in every dumped element of list 5, on both turns, is at **memory index 2** + of the element (index 0 is the system id, so it is the frame's **second** member); +* the same command **on the wire**, in a save that carries issued orders, puts its only non-zero in + **`SRsc`**, the frame's **third** member (lane Q, `SAVE_FORMAT.md` §11, cross-checked to the + Planetary Budget slider pushed fully to Construction); +* `turn2-state.sav` and `turn3-state.sav` both hold `Sys[288] Rts = {SRs 0, SRt 0, SRsc 1.0, SRtf 0, + SRi 0, SRoh 0, SRnr 0}` — so the AI's command re-issues the state the save already holds, which is + why a *correct* applier is invisible and an incorrect one is immediately visible. + +So **memory member 1 is wire member `SRsc`**: one correspondence pinned, six unread. One non-zero +slider cannot determine a permutation of seven, and the converter no longer pretends otherwise — it +emits the system id and seven `?`, and the replayer counts the command and declines it. + +Two experiments settle it, both cheap: + +* **one UI run** — push two *different* sliders to two *different* values on one system, End Turn + with `aiorders=on`, and read the permutation straight off the element; +* **cheaper, and no VM at all** — a save taken after issuing rates carries the same command on the + **wire**, where every field is NAMED. `zuul-turn17-orders2.sav` has one. A `.tcb` converted from a + save's own `TurnCommands_v5` block needs no memory mapping, and would also give the `.tcb` format + a second, independent producer. **This is the highest-value next step on the capture side** and it + is pure host work. + +Two notes on how this was caught, because they generalise. The plain closed/regressed measurement +found it, before the control did — but the control (RB-P4, `--replay-count-only` vs a full replay +must be byte-identical when every modelled handler is a no-op) would have found it too, and it is +what now stands guard: on the canonical pair those two runs are byte-identical, which is the +evidence that the handlers we *do* run write where they claim to rather than agreeing with the +oracle by luck. And the reason a wrong write was visible at all is that the command re-issues +existing state: **a command that re-states the board is the best possible test of an applier**, and +the corpus is full of them. + +--- + +## 4. What a *complete* replay needs that this lane does not have + +This is the real Rung B backlog. Each row is the reason one command in the reference turn is counted +but not applied. + +| # | needed for | what is missing | shape | +|---|---|---|---| +| 1 | **list 3 build, list 1 design** | **ship construction.** No phase in this engine builds a ship. The command carries a queue ordinal, a design id and a system; the effect is `srb`/`sri`, `Maint`, the savings debit, a hull id and `ShipIDs`. Nine leaves on the canonical pair. | engine work; lane B6/E2 have the map | +| 2 | **list 1, list 3, list 8, list 10, list 14** | **the client's id allocator.** The build order names design **18** and the fleet order names fleet **34** — objects that do not exist in the input save, allocated *client-side before submission*, while the server's own master counter (`NMnx`) issued 1712 and 1776 the same turn. **Two id spaces**, and the small one is part of the wire protocol. A reimplementation that allocates on apply produces a structurally correct save with every AI-created id wrong. | **a watchpoint**, not a week of reading: break on the write that produces 18 and 34 | +| 3 | **list 8 fleet move** | **the route's hops.** The capture records the route's length and not its contents, because the dump does not follow the vector. One more indirection in the dumper. Also needs (2): the route belongs to a fleet the save does not contain. | one dumper change | +| 4 | **list 5 system rates** | **the memory field order of the rates frame** (§3). One correspondence pinned, six unread. | one UI run, or one save-sourced capture | +| 5 | **list 10** | **a name and a meaning.** `{systemId, fleetId, counted vector}`; "assign these ships to this fleet at this system" fits and has never been tested. The vector's contents are unread. | a hook on `0x0088bed0` | +| 6 | **list 14 fleet task** | **what the two modes do.** Two elements per AI fleet order, modes 0 then 1; the interface deposits one, mode 0 only. The *cost* is settled; the *effect* is not read at all. | a hook on the inlined applier | +| 7 | **list 23 population** | **the `Population` body**, 24 bytes behind a vftable that the capture window does not follow. Free in `ModCount`, so it has never been forced. | one dumper change | +| 8 | **the research-target gate** | **the techId → tech-name map.** The wire carries an integer (144, 90, 288); the save carries a name. The client resolves it *off the command* and passes a `char*`. The ids are not `index * 16` and are not indices into anything we hold. Today the capture can carry the observed name and the replay TRANSCRIBES it — reported in its own column, because that is not a reimplementation. **Three data points now exist**: 144 → `IND_Waldo`, 90 → `DRV_PlsFiss`, 288 → `XNC_TrnsMorr2`. | a watchpoint at the gate payload, or a read of `0x006c8580` | +| 9 | **the research-boost gate** | savings spent to advance research; both halves unmodelled. Never observed set on an AI turn. | engine work | +| 10 | **the group-4 and group-5 gates** | no read semantics; group 5 is Hiver-only and no save carries it. | rule 6 — a manufactured Hiver workload | +| 11 | **the civilian-ratios gate** | **no applier located anywhere in the batch.** Its `ModCount` cost is *unknown*, not zero, and any turn that sets it reports a lower bound. | an image-wide search for the consumer | +| 12 | **the load-time batch** | the process applies a batch at LOAD as well as at End Turn (`seq=1`, `n=1`, the local client's block alone, with the rate gate SET). Whether it charges `ModCount` is **untested**, and this lane excludes it by construction. If it does charge, a save loaded and immediately re-saved reads one higher. | one run: load, save, compare | +| 13 | **the whole thing, on any interesting board** | **two turns, one AI empire, 28 stars, no contact.** Lists 2, 4, 6, 9, 11–13, 15–22 and 24–27 have never been non-empty. Apply order is unfalsifiable (§2). Nothing here generalises past a very quiet game. | manufacture the workload | + +Two things NOT on that list, deliberately. `Summary.Checksum` moves whenever anything else does and +its inputs are unread — no command replay will touch it until the rest is right. And nothing in +either capture draws a generator word, so the RNG frame is untouched by the stream; the coordinator's +note that the replay interval begins at `BeginProcessTurn` rather than `ProcessTurn` is already +satisfied here — the batch is drained *before* the first phase runs, which is where the End-Turn +dispatcher calls it (`ApplyAllTurnCommands` at `+0x00784904`, `ProcessTurn` at `+0x0078491c`, +`OnAllCombatDone_Tail` at `+0x00784d07`) — but the **hive-registration draws lane L1 found inside +`BeginProcessTurn` are a separate interval and this lane does not model them.** + +--- + +## 5. What this lane did not do + +1. **No VM time and no new instrument.** Every live number here is lane L4's, re-read through a + converter. Nothing new ran under a hook. +2. **The CT111 shim cross-build was not run** — CT111 refused the key from this host. `src/game/ai` + is in the cross build, so per rule 13 **the integrator must run it before pushing**. The two + 32-bit-specific hazards were audited by hand and one was real and is fixed: `strtoul` plus a + `> 0xffffffffUL` test is tautological where `unsigned long` is 32 bits, which would both accept + out-of-range input silently and trip `-Wextra`/`-Werror`; both sites now use `strtoull`. `%zu` + appears in the new code and also in `src/app/main.cpp` today, so it is not a new exposure. +3. **The apply order was not tested against an outcome** and cannot be on this corpus (§2). +4. **`ToTurnCommandBlock` reuses the existing, already-verified cost model** rather than counting a + second time — deliberately, so there is one implementation of the arithmetic and not two that + can agree with each other while both being wrong (rule 8). +5. **The `.tcb` format has exactly one producer.** Until the save-sourced converter of §3 exists, + every capture comes through one script and a bug in it is invisible. +6. **Seeds are carried and never used.** No capture in the corpus has any. diff --git a/ghidra/addresses.d/lane-rb.json b/ghidra/addresses.d/lane-rb.json new file mode 100644 index 0000000..30d6bce --- /dev/null +++ b/ghidra/addresses.d/lane-rb.json @@ -0,0 +1,18 @@ +{ + "entries": [ + { + "name": "StrategySim_ApplyTurnCommandBatch_GateLoopB", + "addr": "0x008907b1", + "convention": "label", + "status": "verified", + "prototype": "the SECOND of the three per-player gate loops inside StrategySim::ApplyTurnCommandBatch. `esi = block+0x20`; it tests the research-boost gate at +0x20 and applies its {spend, fraction} payload through 0x00820560, bumping ModCount inline at 0x008907bc. It runs AFTER all twenty-seven list loops, not with the other gates -- the six prologue gates are split across three loops at three points in the routine, which is why a port that applies them together as a prologue gets the order wrong" + }, + { + "name": "StrategySim_ApplyTurnCommandBatch_GateLoopC", + "addr": "0x0089080a", + "convention": "label", + "status": "verified", + "prototype": "the THIRD and last per-player gate loop inside StrategySim::ApplyTurnCommandBatch. `esi = block+0x24`; it tests the group-4 gate at +0x2c and applies its {bool, int} payload through 0x00821b90. It is the final step of the whole batch. Together with the loop-A head at 0x0088fdb0 and the loop-B head at 0x008907b1, and the six inlined ModCount bump sites, this gives nine positions of the thirty-step apply schedule an address-monotonicity check -- the only part of the sequence that can be re-derived rather than inherited from the read of the `add edi, imm` chain" + } + ] +} diff --git a/tools/aiorders_to_tcb.py b/tools/aiorders_to_tcb.py new file mode 100644 index 0000000..83677fb --- /dev/null +++ b/tools/aiorders_to_tcb.py @@ -0,0 +1,250 @@ +#!/usr/bin/env python3 +"""Turn a live `aiorders` shim dump into a `.tcb` turn-command capture. + +The shim's dump is a memory view: for each submitted block it prints the six prologue gates, +the twenty-seven list lengths, and a fixed 48-byte window of every element. That is the right +thing for an instrument to emit -- it commits to nothing -- but it is not a turn record, because +the fields are in MEMORY order (and one list's writer runs them backwards), the floats are still +integers, and a payload behind a pointer is simply absent. + +This is the mechanical step in between. It applies the per-list field mapping, reinterprets the +words the element record says are floats, converts a vector's begin/end pair into a length, and +writes `?` wherever the dump window could not reach. The output is what `sots_turn +--turn-commands` reads. + + tools/aiorders_to_tcb.py LOG [-o OUT] [--batch SEQ] [--name PID=TECHNAME]... [--seed ID=HEX]... + +Two things this tool deliberately will not do. + + * It will not invent a payload. A route whose hops live behind a pointer becomes `vN` -- a + vector of known length and unknown contents -- and never `v1:0`. The replayer counts such a + command and refuses to apply it, which is the whole point: a command that was issued and a + command whose effect we can reproduce are different facts. + * It will not pick a batch for you when the choice is ambiguous. A process applies a command + batch at LOAD as well as at End Turn; replaying the load-time one against a save that was + written after it would double-count. The default is the LAST batch in the log, which is the + End-Turn one, and `--batch` overrides it. + +`--name PID=TECHNAME` records the tech NAME an instrument observed a research-target gate resolve +to. The wire carries an integer id and the save carries a name; the client resolves one to the +other off the command and that map is unread, so the name can only be recorded, never computed. +`--seed NETID=VALUE` records an AI client's construction seed. +""" +import argparse +import re +import struct +import sys + +# Per list: how the 48-byte memory window maps onto the element's wire fields. +# +# `order` is the sequence of word indices to emit; `kind` says how to read each one. The build +# list is the only one that reverses -- its writer emits +0x14, +0x10, +0x0c, +0x08, descending -- +# and that is per-list, not a rule. Every other observed list writes ascending. +# +# i an int word +# f a word to reinterpret as a float +# b a word whose low byte is a bool +# v a std::vector: this word and the next hold begin/end, and the length is their difference +# over four. The CONTENTS are behind the pointer and the dump does not follow it, so the +# field is emitted with a length and no values. +# ? a word the element record says is part of an object this window cannot read +LIST_MAP = { + 1: [("?", 0)], # a ShipDesignDef object + 3: [("i", 3), ("i", 2), ("i", 1), ("i", 0)], # DESCENDING: ordinal, design, system, w + # List 5 carries a system id and then the seven planetary-budget sliders. The id is certain; + # THE SLIDERS ARE NOT, and this used to map them straight onto the wire order and was wrong. + # A replay built on that mapping wrote the AI's one non-zero slider into the wrong member and + # regressed two leaves on the canonical pair -- the command re-issues the rates the save + # already holds, so a correct applier is a no-op and an incorrect one is immediately visible. + # + # What is known: the only non-zero word in every dumped element of this list is at index 2, + # and the same command on the WIRE (where the frame is named) puts its only non-zero in + # `SRsc`, the THIRD member. So memory member 1 is wire member `SRsc` and the frame's memory + # order is not its wire order -- one correspondence pinned, six unread. The `?` below is + # that ignorance, and the replayer counts such a command and refuses to apply it. + # + # The experiment that settles it is one UI run: push two DIFFERENT sliders to two different + # values, capture the block, and read the permutation straight off. Cheaper still, a save + # taken after issuing rates carries the same command on the wire with every field NAMED, so a + # capture converted from a save's own TurnCommands block needs no memory mapping at all. + 5: [("i", 0)] + [("?", k) for k in range(1, 8)], + 7: [("i", 0), ("i", 1)], + 8: [("i", 0), ("v", 1)], + 10: [("i", 0), ("i", 1), ("v", 2)], + 14: [("i", 0), ("i", 1), ("b", 2)], + 23: [("i", 0), ("?", 1)], # a Population object +} + +BLK_RE = re.compile( + r"^aiblk seq=(\d+) blk=(\d+)/(\d+) at=\S+ pid=(-?\d+) " + r"rate=(\d):(\S+) target=(\d):(-?\d+) boost=(\d):(-?\d+),(\S+) g4=(\d):(-?\d+),(-?\d+) " + r"f3=(\d):(\S+),(\S+),(\S+) civ=(\d)") +LISTS_RE = re.compile(r"^ailists seq=(\d+) blk=(\d+) pid=(-?\d+) nonEmpty=\d+ sizes\(1\.\.27\)=\[([^\]]*)\]") +ELEM_RE = re.compile(r"^aielem blk=(\d+) pid=(-?\d+) list=(\d+) idx=(\d+) .*? hex=\[([^\]]*)\]") + + +def f32(word): + """A dumped word, as the float32 it is, printed so it round-trips exactly.""" + return "%.9g" % struct.unpack(" {seq: {"n": int, "blocks": {idx: block}}}""" + batches = {} + + def batch(seq): + return batches.setdefault(seq, {"n": 0, "blocks": {}}) + + with open(path, encoding="utf-8", errors="replace") as f: + for line in f: + m = BLK_RE.match(line) + if m: + seq, idx, n = int(m.group(1)), int(m.group(2)), int(m.group(3)) + b = batch(seq) + b["n"] = max(b["n"], n) + b["blocks"][idx] = { + "pid": int(m.group(4)), + "rate": (m.group(5) == "1", m.group(6)), + "target": (m.group(7) == "1", int(m.group(8))), + "boost": (m.group(9) == "1", int(m.group(10)), m.group(11)), + "g4": (m.group(12) == "1", int(m.group(13)), int(m.group(14))), + "f3": (m.group(15) == "1", m.group(16), m.group(17), m.group(18)), + "civ": m.group(19) == "1", + "sizes": [0] * 27, + "elems": {}, + } + continue + m = LISTS_RE.match(line) + if m: + seq, idx = int(m.group(1)), int(m.group(2)) + blk = batch(seq)["blocks"].get(idx) + if blk is not None: + blk["sizes"] = [int(x) for x in m.group(4).split()] + continue + m = ELEM_RE.match(line) + if m: + # An `aielem` line carries no seq, so it belongs to the batch whose block header + # it followed -- which is the most recent one seen. + seq = max(batches) if batches else 0 + idx, listno, elemidx = int(m.group(1)), int(m.group(3)), int(m.group(4)) + words = [int(x, 16) for x in m.group(5).split()] + blk = batch(seq)["blocks"].get(idx) + if blk is not None: + blk["elems"].setdefault(listno, {})[elemidx] = words + return batches + + +def element_fields(listno, words): + """The wire fields of one element, as `.tcb` tokens.""" + spec = LIST_MAP.get(listno) + if spec is None: + # An unmapped list: record that the element exists and nothing about it. The command + # still costs whatever its list costs; the replayer will decline to apply it. + return ["?"] + out = [] + for kind, k in spec: + if k >= len(words): + out.append("?") + continue + w = words[k] + if kind == "i": + out.append("i%d" % struct.unpack("= len(words): + out.append("?") + else: + out.append("v%d" % ((words[k + 1] - w) // 4)) + else: + out.append("?") + return out + + +def emit(batch, seq, source, names, seeds): + lines = ["tcb 1", + "meta source %s" % source, + "meta batch seq=%d n=%d" % (seq, batch["n"]), + "meta note the load-time batch is excluded; this is the End-Turn submission"] + for netid, value in seeds: + lines.append("seed %s %s" % (netid, value)) + for idx in sorted(batch["blocks"]): + b = batch["blocks"][idx] + lines.append("block %d %d" % (idx, b["pid"])) + if b["rate"][0]: + lines.append("gate %d rate %s" % (idx, b["rate"][1])) + if b["target"][0]: + name = names.get(b["pid"]) + lines.append("gate %d target %d%s" % (idx, b["target"][1], + (" name %s" % name) if name else "")) + if b["boost"][0]: + lines.append("gate %d boost %d %s" % (idx, b["boost"][1], b["boost"][2])) + if b["g4"][0]: + lines.append("gate %d group4 %d %d" % (idx, b["g4"][1], b["g4"][2])) + if b["f3"][0]: + lines.append("gate %d group5 %s %s %s" % (idx, b["f3"][1], b["f3"][2], b["f3"][3])) + if b["civ"]: + lines.append("gate %d civilian" % idx) + for listno in range(1, 28): + n = b["sizes"][listno - 1] + if not n: + continue + lines.append("list %d %d %d" % (idx, listno, n)) + have = b["elems"].get(listno, {}) + for e in range(n): + words = have.get(e) + fields = element_fields(listno, words) if words is not None else ["?"] + lines.append("elem %d %d %d %s" % (idx, listno, e, " ".join(fields))) + return "\n".join(lines) + "\n" + + +def main(argv=None): + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("log") + ap.add_argument("-o", "--out") + ap.add_argument("--batch", type=int, help="which seq to convert (default: the last)") + ap.add_argument("--name", action="append", default=[], metavar="PID=TECHNAME", + help="the tech name a research-target gate was observed to resolve to") + ap.add_argument("--seed", action="append", default=[], metavar="NETID=VALUE", + help="an AI client's construction seed") + a = ap.parse_args(argv) + + batches = parse_log(a.log) + if not batches: + print("no aiblk records in %s" % a.log, file=sys.stderr) + return 2 + seq = a.batch if a.batch is not None else max(batches) + if seq not in batches: + print("no batch seq=%d; the log holds %s" % (seq, sorted(batches)), file=sys.stderr) + return 2 + + names = {} + for n in a.name: + pid, _, tech = n.partition("=") + if not tech: + print("--name wants PID=TECHNAME", file=sys.stderr) + return 2 + names[int(pid)] = tech + seeds = [] + for s in a.seed: + netid, _, value = s.partition("=") + if not value: + print("--seed wants NETID=VALUE", file=sys.stderr) + return 2 + seeds.append((netid, value)) + + text = emit(batches[seq], seq, a.log.split("/")[-1], names, seeds) + if a.out: + with open(a.out, "w") as f: + f.write(text) + print("wrote %s (batch seq=%d, %d blocks)" % (a.out, seq, len(batches[seq]["blocks"]))) + else: + sys.stdout.write(text) + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tools/tcb_from_json.py b/tools/tcb_from_json.py new file mode 100644 index 0000000..2fe2428 --- /dev/null +++ b/tools/tcb_from_json.py @@ -0,0 +1,232 @@ +#!/usr/bin/env python3 +"""Convert a lane-CB JSON turn-command capture into the `.tcb` file `sots_turn` reads. + +TWO FORMATS ON PURPOSE, AND THIS IS THE JOIN. + +`tools/turncommands_capture.py` (lane CB) produces the **capture of record**: raw element words as +ground truth, heap vectors and strings the deep dump followed, the input save's hash bound to the +output autosaves' hashes, the per-client AI seeds, and the container self-check. That is what an +experiment should leave behind and none of it belongs in an engine's input file. + +`.tcb` is the **engine's input**: line-oriented, no parser, nothing but the commands and their +provenance. `sots_turn --turn-commands` reads it and nothing else, so the engine never grows a +JSON reader and never has an opinion about how a capture was taken. + +This script is the only thing that has to know both, and it is deliberately the narrow part: +it takes `decoded.wire` where lane CB's decoder produced one, falls back to raw words where it did +not, and writes `?` for every field neither could reach. + + tools/tcb_from_json.py CAPTURE.json [-o OUT.tcb] [--batch SEQ] + +ONE FIELD MAPPING IS KNOWN WRONG AND IS OVERRIDDEN HERE, WITH ITS REASON. + +List 5 (system rates) is decoded by lane CB as `{systemId, ship, terraform, sciences, ...}` -- +the element's memory words mapped straight onto the frame's WIRE order. Lane RB measured that and +it is wrong: replaying it wrote the AI's one non-zero slider into `SRt` and regressed two leaves +on the canonical pair, where the oracle holds `SRsc = 1.0`. The only non-zero word of every list-5 +element ever dumped is at memory index 2, and the same command on the wire -- where the frame is +NAMED -- puts its only non-zero in `SRsc`, the third member. So memory member 1 is wire member +`SRsc`, and the frame's memory order is not its wire order: ONE correspondence pinned, six unread. + +Rather than carry a mapping that is known to be off by at least one, this converter emits the +system id and seven `?`. The replayer then counts the command -- the count is right either way -- +and refuses to apply it, which is the correct behaviour for a payload nobody has read. + +Settling it is one run: push two DIFFERENT sliders to two DIFFERENT values and read the +permutation off the element. Cheaper, a save taken after issuing rates carries the same command on +the wire with every field named, and needs no memory mapping at all. +""" +import argparse +import json +import struct +import sys + +# Lists whose decoded `wire` this converter trusts. List 5 is deliberately absent (see above); +# lists with no entry are carried as a single `?`, which counts the command and applies nothing. +TRUSTED = { + 3: "iiii", # ordinal, designId, systemId, trailing + 7: "ii", # shipId, trailing + 14: "iib", # fleetId, mode, flag +} +# Lists whose element leads with scalars and then a counted vector the deep dump may have read. +VECTOR_TAIL = { + 8: ("i", 1), # fleetId, then the route + 10: ("ii", 2), # systemId, fleetId, then a counted vector +} +UNMAPPED_HEAD = { + 5: (1, 7), # one trusted leading int (the system id) and seven unread fields + 23: (1, 1), # the system id and a Population body behind a vftable +} + + +def as_int(v): + if isinstance(v, bool): + return 1 if v else 0 + if isinstance(v, float): + return int(v) + return int(v) + + +def tok_i(v): + return "i%d" % as_int(v) + + +def fields_for(list_no, elem): + """The `.tcb` field tokens for one element.""" + decoded = elem.get("decoded") or {} + wire = decoded.get("wire") + raw = elem.get("raw_words") or [] + vectors = {v["at_word"]: v for v in (elem.get("vectors") or [])} + + if list_no in UNMAPPED_HEAD: + lead, unread = UNMAPPED_HEAD[list_no] + head = [tok_i(raw[i]) for i in range(min(lead, len(raw)))] + return head + ["?"] * unread if head else ["?"] + + if list_no in TRUSTED: + spec = TRUSTED[list_no] + if not wire or len(wire) < len(spec): + return ["?"] + out = [] + for i, kind in enumerate(spec): + v = wire[i] + out.append("b%d" % (1 if v else 0) if kind == "b" else tok_i(v)) + return out + + if list_no in VECTOR_TAIL: + spec, vec_word = VECTOR_TAIL[list_no] + if len(raw) < len(spec): + return ["?"] + out = [tok_i(struct.unpack(" vec_word + 1: + out.append("v%d" % max(0, (raw[vec_word + 1] - raw[vec_word]) // 4)) + else: + out.append("?") + elif v.get("truncated"): + out.append("v%d" % v["count"]) + else: + vals = [struct.unpack("