diff --git a/findings/subsystems/id-allocation.md b/findings/subsystems/id-allocation.md new file mode 100644 index 0000000..6e74cc5 --- /dev/null +++ b/findings/subsystems/id-allocation.md @@ -0,0 +1,309 @@ +# Object id allocation: one allocator, sixteen node counters, and why the client's ids are free + +Lane ID, 2026-09-08. **Host work only — no VM run was needed and none was taken.** VM140 was held +and released untouched; its `SavedGames` set is byte-identical to the oracle (§8). + +Closes row 2 of `turn-command-replay.md` §4 — "the client's id allocator" — which lane RB flagged as +"a watchpoint, not a week of reading". It turned out to be neither: it is arithmetic on ids the +corpus already held, confirmed against the instruction stream and cross-checked on twenty saves. + +Predictions were written before any of the work below and are quoted verbatim where they were +falsified (§3.2). Consumes: `combat-retreat-pipeline.md` (B5, `IDMap::AllocateID`), +`turn-command-replay.md` (RB), `ai-order-emission.md` (AI4), `struct-recovery.md`. + +--- + +## 0. Lead: there is no second allocator + +RB's gap list says "**two id spaces**, and the small one is part of the wire protocol". There are +two id *spaces*, and RB is right that a reimplementation which allocates on apply gets every +AI-created id wrong. But there is **one allocator, one implementation, and one formula** — B5's +`IDMap::AllocateID` 0x008b8ae0, `id = (counter << 4) | (node & 0xF)`. + +What differs is **which node's counter** is used, and that is a property of the *sim object*, not of +the code: + +> `Game::StrategyServer` and `Game::StrategyClient` are **both** `Game::StrategySim` +> (RTTI: `StrategyClient` bases `[StrategyClient, StrategySim]` at offset 0; `StrategyServer` bases +> `[StrategyServer, IStreamable, StrategySim]` with vftables at offset 0 **and** offset 4 — that is +> the "two bases 4 bytes apart" of earned rule 1, and the +4 one is the `StrategySim` subobject). +> +> `StrategySim` owns an `IDMap` at **`StrategySim + 0x80`** — i.e. `StrategyServer + 0x84`, +> `StrategyClient + 0x80`. Every sim allocates from **its own** map, on **its own** local node +> index at `IDMap + 0x18`. + +So the object-creation code is base-class code that runs unchanged on both sides. The whole +client-allocates/server-honours protocol is one branch in `StrategySim::CreateDesign` 0x008827e0: + +``` +008828b3 mov eax,[ebp+0x10] ; explicitId, the third argument +008828b6 test eax,eax +008828b8 jne 0x8828c5 ; non-zero -> use the id the command carried +008828ba lea ecx,[edi+0x80] ; else: this sim's own IDMap +008828c0 call 0x8b8b70 ; IDMap::AllocateOnLocalNode +008828c5 ... call 0x8b9350 ; IDMap::Insert(map, design+0xa0, id) +``` + +The client calls it with `explicitId = 0` and mints; the server calls it with the id off the wire +and mints nothing. + +## 1. The arithmetic, which is the whole result + +B5's formula, run backwards on RB's numbers: + +| id | | counter | node | +|---:|---|---:|---:| +| **18** design | *client-allocated* | **1** | **2** | +| **34** fleet | *client-allocated* | **2** | **2** | +| 1712 | from the master counter | 107 | 0 | +| 1728 | ship in the input save | 108 | 0 | +| 1776 | | 111 | 0 | +| 272, 288 systems; 32, 496, 512 players | | 17, 18, 2, 31, 32 | 0 | + +Every id that comes out of a save carries node nibble **0**. Both client-allocated ids carry node +nibble **2**, with counters **1** and **2** — the first two ids that node ever issued. + +### 1.1 On twenty saves, four games, three distinct client nodes + +Nibble histogram over `FltID`/`DesID`/`ShipID`/`SysID`/`PlayerID` in the whole corpus +(`verify/results/saves/*.sav`, read with `verify/save-reader/save_reader.py --dump`): + +| save | ids | nibbles | the non-zero ones | +|---|---:|---|---| +| `turn1-state` | 98 | `{0: 98}` | — | +| `turn2-state` | 102 | `{0: 101, 2: 1}` | 18 | +| `turn3-state` | 104 | `{0: 102, 2: 2}` | 18, 34 | +| `cb-turn2to3-autosave` | 104 | `{0: 102, 2: 2}` | 18, 34 | +| `human-turn3-noderoute` | 90 | `{0: 86, 2: 4}` | 18, 34, 50, 66 | +| `human-turn8-traderoutes` | 151 | `{0: 141, 2: 10}` | 18 … 242 | +| `human-turn15-spyprogram` | 295 | `{0: 252, **1: 1**, 2: 42}` | **17**, 18 … 818 | +| `zuul-turn15-orders` | 150 | `{0: 139, 2: 10, **3: 1**}` | 18 … 162, **19** | +| `zuul-turn17-orders2` | 167 | `{0: 151, 2: 14, **3: 2**}` | 18 … 226, **19, 35** | +| `zuul-turn23-fleet23` | 222 | `{0: 198, 2: 22, 3: 2}` | … | + +Three facts fall straight out, and none of them needed a hook. + +1. **Nodes 1, 2 and 3 all occur.** "The client" is not singular. Each node's counters run + **1, 2, 3, …** contiguously from the start of the game — node 2's first id is `18` in *every* + save in the corpus, node 3's first is `19`, node 1's first is `17`. +2. **The partition is absolute.** ~2,600 ids across four games, zero collisions between spaces. + That is the entire purpose of the nibble and of `IDMap::Initialize`'s `numNodes <= 0x10` cap. +3. **Node = owning player's index + 1.** Every node-nibble object belongs to exactly one player: + + | save | id | owner `PID` | that player's `PlyrIdx` | node | + |---|---:|---:|---:|---:| + | `turn3-state` | design 18, fleet 34 | 32 | 1 | 2 | + | `human-turn15` | design 17 | 16 | 0 | 1 | + | `human-turn15` | design 18 | 32 | 1 | 2 | + | `zuul-turn17` | designs 18, 34, … | 32 | 1 | 2 | + | `zuul-turn17` | designs 19, 35 | 496 | 2 | 3 | + +## 2. The node index, read from the instruction stream + +`StrategyServer::InitGameForPlayer` 0x007c8d90 builds the CreateGame message for one player's +client. Four instructions do the assignment: + +``` +007c8ddf mov esi,[eax+0x28] ; player->PlyrIdx +... +007c8ecb cmp esi,0xffffffff +007c8ece je 0x7c8ed3 +007c8ed0 inc esi ; localNode = PlyrIdx + 1 +007c8ed1 jmp 0x7c8ed5 +007c8ed3 xor esi,esi ; PlyrIdx == -1 -> localNode = 0 +007c8ed5 ... [ebx+0x90] - [ebx+0x8c], /20 ; numNodes = the server map's node count +007c8ef2 mov [ebp-0x134],ecx ; msg->numNodes +007c8ef8 push esi +007c8ef9 lea ecx,[ebx+0x84] ; the SERVER's IDMap +007c8eff mov [ebp-0x130],esi ; msg->localNode +007c8f05 call 0x8b8b80 ; IDMap::GetNodeCounter(server map, localNode) +007c8f1a mov [ebp-0x12c],eax ; msg->startId +``` + +The message is `{numNodes, localNode = PlyrIdx + 1, startId = the server's counter for that node}`. +The client receives it as **case 0** of `StrategyClient::RaiseEvent` 0x00783ee0 (jump table +0x00784200, handler 0x00783f05) and `StrategySim::OnCreateGame` 0x00776f20 calls +`IDMap::Initialize` on its own map at `+0x80` with exactly those three words. + +So `PlyrIdx + 1` is not an inference from the saves any more. The saves and the disassembly agree, +which is the only reason either is worth reporting. + +## 3. Seeding: what a save carries, and what it cannot + +`IDMap::Initialize(numNodes, localNode, startId)` 0x008b9ad0, `ret 0xc`: + +* refuses `numNodes > 0x10` — **sixteen nodes, and that is the four-bit nibble**; +* refuses `localNode` outside `[0, numNodes)`; +* resizes the 0x14-stride node vector, then **zeroes every node's counter** (0x008b9b20); +* writes `startId` into **`nodes[localNode].counter` alone** (0x008b9b38); +* stores `localNode` at `this->+0x18` (0x008b9b3d). + +### 3.1 The save's `NM*` tags are now named + +`StrategyServer::Read` 0x007d27a0 reads three ints and passes them straight in: + +``` +NMSz -> numNodes NMLc -> localNode NMnx -> startId +``` + +That **confirms and extends B5's labelled hypothesis** ("the counter is almost certainly NMnx"). +It is NMnx, it is the counter *for node NMLc only*, and the two tags beside it are the node count +and the local node index. Corpus: + +| save | NMSz | NMLc | NMnx | ModCount | Frame | +|---|---:|---:|---:|---:|---:| +| `turn1-state` | 16 | 0 | 106 | 0 | 1 | +| `turn2-state` | 16 | 0 | 109 | 12 | 2 | +| `turn3-state` | 16 | 0 | **111** | 24 | 3 | + +`StrategyServer::LoadGame` 0x007dd530 sets a fresh game up with `Initialize(16, 0, 0)`, which is +where `NMSz 16` / `NMLc 0` come from and why the host's ids all carry nibble 0. + +**A correction to `turn-command-replay.md` §4 row 2** (earned rule 11). It says the server "issued +1712 and 1776 the same turn", and that design 18 and fleet 34 are "objects that do not exist in the +input save". Neither is quite right on the canonical pair: + +* 1712 is counter **107**, issued during turn **1→2** (NMnx 106 → 109 covers counters 107, 108, 109 + = ids 1712, 1728, 1744). 1776 is counter 111, issued during turn **2→3** (109 → 111). +* **Design 18 is present in `turn2-state.sav`.** It is in the master `DesignIDs` list and in player + 32's block, and it was allocated during turn 1→2. Only **fleet 34** is new on the canonical pair. + So the turn-2→3 build order names an id the input save already holds, and a faithful replay of + that pair has to mint exactly **one** id, not two. That is a materially smaller gap than the row + claims. + +Everything else in the row stands, including the load-bearing part: allocate on apply and you get +`1792` where the original has `34`. + +### 3.2 A falsified prediction, and the open item it leaves + +Prediction ID-P3 said, in full: + +> If NMLc = 0 holds, then the node-2 counter that issued 18 and 34 is NOT in the save, is NOT +> restored by Initialize (which zeroes it), and therefore **restarts at zero on every load**. + +NMLc is 0, and the first two clauses are read directly out of the code: no save carries a per-node +array, `Initialize` zeroes the rest, `IDMap::Insert` 0x008b9350 **never touches a counter** (read to +the end: it logs a duplicate, refuses id 0, bounds-checks `id & 0xF`, writes the id into `obj->+4` +and inserts into `nodes[n]`'s map — no counter write), and `InitGameForPlayer` seeds a client from +`GetNodeCounter`, which therefore returns 0 for every client node in a freshly loaded game. + +**The third clause is falsified by a save already in the corpus.** `cb-turn2to3-autosave.sav` is a +fresh process that *loaded* `turn2-state.sav` (which contains design 18 = node 2, counter 1) and +pressed End Turn. The fleet it created is **34 — counter 2**, not 18. If the client's counter had +started at 0 the fleet would have been 18, colliding with a design that already exists. + +Two explanations survive, and this lane cannot separate them without a hook: + +* **(A)** something restores the server's node-2 counter to 1 during the load, and the client is + seeded with it. I read `Initialize`, `Insert`, `AllocateID`, `FindNode`, `GetNodeCounter` and + `InitGameForPlayer` and found no such write. Absence in six functions is not absence. +* **(B)** the counter really does start at 0 and the client made **one earlier allocation that turn + which never reached the save** — a local design object, a task, an order — taking counter 1, so + the fleet got counter 2. This fits the read code exactly and requires nothing unread. + +**The probe is one hook and it is cheap** (earned rule 18, and rule 20 — instrument the *entry*): +detour `IDMap::AllocateID` 0x008b8ae0 and log `(this, nodeIndex, resulting id, return address)` for +one `turn2-state → turn3-state` End Turn. (A) predicts the client's very first allocation that turn +returns 34; (B) predicts it returns 18 from a site that is not the fleet creator. **A count alone +cannot separate them** — that is exactly the shape rule 20 exists for. + +Whichever it is, the consequence for a reimplementation is the same and is stated in §5. + +### 3.3 The collision hazard, stated as a hypothesis + +If (B) is right, then a client's counter is a pure function of **allocation order within the +process** and a save/load loses it. Load a save in which a client has already allocated *n* objects +and let it allocate again, and it re-issues `(1<<4)|k` — an id the map already holds. `IDMap::Insert` +logs `IDMap: Object already exists with id %d.` and **inserts anyway** (0x008b9367: the log is +followed by fall-through, not a return). That would be an original defect, not ours. + +Labelled hypothesis. The same one-hook probe settles it, and `human-turn15-spyprogram.sav` — node 2 +at counter 51 — is the workload that would make it loud. + +## 4. Layout, for the record + +``` +IDMap (StrategySim + 0x80) + +0x08 / +0x0c / +0x10 vector, STRIDE 0x14 (allocator +0x10, rule 5) + +0x18 int localNodeIndex +NodeEntry 0x14 bytes + +0x00 .. +0x0f std::map id -> object, for THIS node + +0x10 int counter pre-incremented; 0 is never issued +``` + +`AllocateID(node)`: 0 if `node == -1` or the node does not exist; else `++nodes[node].counter` +(pre-increment, and on wrap to 0 it logs and increments again, so `INVALID_NETWORK_ID` is never +handed out); `id = (counter << 4) | (node & 0xF)`, with an overflow log if the counter no longer +round-trips. + +## 5. What a faithful replay has to do + +1. **A per-node counter array, not one counter.** Sixteen of them. +2. **Node 0 for the sim that owns the board; node `PlyrIdx + 1` for each player's client.** +3. **Restore only node `NMLc`'s counter from `NMnx`.** There is nothing else in the save to + restore, and inventing a per-node array in the save format would diverge from the original. +4. **Mint the id at command *emission*, not at apply**, and carry it in the command — the applier + must honour a non-zero id and mint only on zero (`StrategySim::CreateDesign`'s branch). +5. **Do not model the client counter as a function of the save.** It is a function of allocation + order in the process, and until §3.2 is settled a replay of the canonical pair should take the + client's ids from the capture rather than derive them. + +For the canonical pair specifically, (4) plus "design 18 is already in the input save" means the +replay needs exactly one minted id, `34`, and it is the *second* thing node 2 allocates that turn. + +## 6. What this lane did NOT do + +1. **No hook, no VM run, no new instrument.** Every number is from the checked-in corpus and the + image. That is also the limit: §3.2 is open precisely because nothing ran. +2. **The second `LoadTechFile` loop and the message *send* were not read.** I read + `InitGameForPlayer` up to the point where the message is filled, and `OnCreateGame` from the + point where it is received; the transport between them is assumed, not read. It is a single + process, so nothing observable depends on it — but it is not read. +3. **`FUN_008a9030`'s counter at `+0x23c` is a different mechanism** (a Mars scene object with its + own `IDMap` at `+0x68`, `Initialize(1, 0, 0)`, and a separate per-object counter gated on + `obj->+0x20 & 0xF == 2`). It is not game state and it is not this. Recorded so the next lane + that greps for `& 0xF` does not chase it. +4. **Multiplayer is untested.** Every save in the corpus is `NMLc 0`. A real client machine would + save with `NMLc = k` and its own counter, and *the server's* copy of that counter is the thing + §3.2 is about. Nothing here has been exercised across two machines. +5. **`ShipID` allocation was not traced to its site.** Ships in the corpus are all node 0, so they + are minted server-side; I did not read which function does it. + +## 7. Reproducing every number here + +```sh +cd ~/sots-re +uv run python3 verify/save-reader/save_reader.py --dump verify/results/saves/turn3-state.sav \ + | grep -E 'NMSz|NMLc|NMnx' +# nibble histogram: see the one-screen script in section 1.1's method note below +uv run python3 dumps/b6dis.py 0x008b9ad0 # IDMap::Initialize +uv run python3 dumps/b6dis.py 0x007c8d90 # InitGameForPlayer; the +1 is at 0x007c8ed0 +``` + +The histogram is `re.match(r'\s*@[0-9a-f]+\s+(\S+)\s+int\??\s+(-?\d+)', line)` over the dump, +keeping the five id tags and counting `value & 15`. No tool was added for it — it is four lines and +a tool would only hide what it does. + +## 8. VM140 released + +The game was never launched and nothing was written. `C:\SOTS\SavedGames` verified over SSH before +release — **8 files**, and the three oracle hashes match: + +| file | bytes | sha256[0:16] | | +|---|---:|---|---| +| `(Autosave).sav` | 67,219 | `978041acd168b56e` | ✅ oracle | +| `(Autosave EndTurn).sav` | 66,732 | `bb4fd9ac89f41e3b` | ✅ oracle | +| `(Autosave Backup).sav` | 67,219 | `978041acd168b56e` | ✅ oracle | +| `MyGameverify1rtD.sav` | 66,732 | `bb4fd9ac89f41e3b` | | +| `MyGameverify1rtE.sav` | 66,732 | `bb4fd9ac89f41e3b` | | +| `MyGameverify1verify1.sav` | 66,732 | `bb4fd9ac89f41e3b` | | +| `ref-turn2.sav` | 66,739 | `ab4ac2d7e2977260` | | +| `zuul-turn5.sav` | 59,131 | `48559ab5b719b332` | | + +Note for the next VM lane: a filename with parentheses cannot be hashed through +`ssh … 'certutil -hashfile "…(Autosave).sav"'` — `cmd.exe` eats the parens and returns +"Check the spelling", which reads exactly like a missing file. Use +`powershell -NoProfile -c "(Get-FileHash -LiteralPath '…' -Algorithm SHA256).Hash"`. My first +attempt reported the three oracle files as absent and they were never absent. + +**Please mark VM140 free.** diff --git a/findings/subsystems/techid-name-map.md b/findings/subsystems/techid-name-map.md new file mode 100644 index 0000000..91435f4 --- /dev/null +++ b/findings/subsystems/techid-name-map.md @@ -0,0 +1,146 @@ +# `techId` -> tech name: an offline lookup, not a capture to transcribe + +Lane ID, 2026-09-08. Host work only. + +Closes row 8 of `turn-command-replay.md` §4 and the open value that kept Rung C-set's "we can name +all *k*" from closing. + +> **282 is `XNC_TrnsHum2`.** + +And it was **already in our own notes**: lane L4 observed it live in run R2 +(`ai-order-capture.md`, the two-capture table — "player 512 target: `XNC_TrnsHum2`, techId 282"). +Nobody connected it, and lane CB then produced 282 as an unmapped value. Earned rule 18's corollary, +paid for again: **search the notes before the binary.** + +--- + +## 1. The rule + +> **`techId` is the 0-based index of the tech's name in the master tech list sorted by +> `_stricmp` — case-insensitively.** + +293 nodes in stock `TechTree/MasterTechList.tech`, so `techId` runs 0..292. That is a complete, +offline, data-derived mapping: **no capture, no transcription, no VM.** The replayer can stop +carrying the observed name in its own column. + +Every observed point, including the two the rule was not fitted to: + +| techId | name | how it was observed | +|---:|---|---| +| 90 | `DRV_PlsFiss` | RB / L4 capture, player 496 | +| 144 | `IND_Waldo` | RB / L4 capture, player 32 | +| **282** | **`XNC_TrnsHum2`** | L4 `ai-order-capture.md` run R2, player 512 — **independent of the fit** | +| 288 | `XNC_TrnsMorr2` | RB / L4 capture, player 512 | + +The rule was fitted on the three RB reported and then hit 282 on the nose against a live +observation taken days earlier by a different lane on a different run. It also reproduces L4's +structural claim about the family: the six-member `XNC_Trns2` set is exactly +**{282 Hum2, 284 Hvr2, 286 Lir2, 288 Morr2, 290 Trk2, 292 Zuul2}**, interleaved with the `…3` +variants — which is why L4 saw two picks "six apart". + +## 2. Read from the instruction stream, not fitted + +`MasterTechTree::MasterTechTree` 0x0058b870, immediately after the `LoadTechFile` loop: + +``` +0058b9df lea eax,[ebx+0x14] ; the PARSE-ORDER list +0058b9e3 lea ecx,[ebx+0x24] +0058b9e6 call 0x8b3210 ; vector::operator= -- +0x24 is a COPY of +0x14 +0058b9eb..0058b9ff + push pred / push count=(last-first)/4 / push last / push first + call 0x583b10 ; MSVC std::sort::_Sort +0058ba19 xor eax,eax +0058ba23 mov edx,[ebx+0x24] +0058ba26 mov ecx,[edx+eax*4] ; sortedList[i] +0058ba29 mov [ecx],eax ; **def->techId = i** +0058ba31 inc eax +0058ba37 jl 0x58ba23 +``` + +Three things this settles that the fit could not. + +**It is a sorted COPY.** `+0x14` keeps parse order; `+0x24` is the sorted one and the id is its +index. That is why the parse-order pointer-table dump in `strategic-turn-internals.md` +(`turn3/techtable.txt`, which starts `IND_Waldo, IND_RefCoat, …`) does not match the ids, and why +RB correctly concluded the ids "are not indices into anything we hold" — we held the wrong list. + +**The comparator is `_stricmp`, so the order is case-insensitive.** It is inlined into +`_Unguarded_partition` FUN_00581130 at 0x00581190 and 0x005811b8 as +`_stricmp(a->name, b->name) < 0`, reading a `std::string` at `TechDef+0x04` (SSO-tested at `+0x14` +against `0x10`); the import at `0x009dd328` resolves to **`MSVCR100.dll _stricmp`**. + +This matters. **48 of the 293 ids differ between byte-wise ASCII order and case-insensitive order** +— the `CCC_AI*`/`CCC_Adv*` run at 21–28, `DRN_COL`/`DRN_Cmbt`, `DRV_NodFoc`/`DRV_Node`, +`DRV_ROOT`/`DRV_Rad…`, `IND_DSCon`/`IND_Decon`, `IND_ROOT`/`IND_RefCoat`, `WEP_APRtech`/`WEP_Ac…`, +`WEP_KKMsl`/`WEP_KelTrp`, `WEP_MWMsl`/`WEP_Mas…`, `WEP_Nukes`/`WEP_NukMine`, `WEP_Pls*`. **None of +the four observed points discriminates**, so this half is disassembly-only and the corpus cannot +confirm it. Flagged as such (earned rule 6): if a future capture ever reports a techId in one of +those 48 slots, it is a real test and it should be reported as one. No two names collide under +case-folding (checked over all 293), so `std::sort`'s instability cannot bite. + +**Why sorted at all.** The same `+0x24` vector is the binary-search index: `FUN_0057f120` +(`std::lower_bound`) resolves every `requires` / `allows` name reference against it at 0x0058baed. +The id is the index into a lookup table that had to be sorted anyway. + +## 3. The two tech id spaces, which is what made this look hard + +| | space | base | size | where | +|---|---|---:|---:|---| +| **wire / command** | index into the `_stricmp`-sorted master list | 0 | 293 | `TechDef+0x00`; the research-target gate payload | +| **`TechID` enum** | index into `MasterTechTree->resolved[]` | **10000** | **196** | a `{name, id}` table in `.rdata`, `CCC_AdvSens = 10001` | + +`MasterTechTree::GetTechDef` 0x0057d610 and `IsTech` 0x0057d5d0 take the **enum** id: +`if (id == 0xc5) return 0; id -= 10000; if ((unsigned)id > 0xc3) return 0; return resolved[id];` +(`0xc5` = 197 is the "no tech" sentinel.) + +`TechTree::HasResearched` 0x0057d810 is the bridge and shows both in four instructions: + +``` +eax = enumId - 10000; eax = master->resolved[eax]; ; ENUM space -> TechDef* +eax = *eax; ; TechDef+0x00 = the WIRE techId +eax = this->+0x10[eax]; ; per-player node vector, indexed by WIRE id +return eax && eax->+0x14 == 4; +``` + +The enum space exists so that C++ code can name a specific tech; the wire space is a dense index +over *all* techs. `TechTree::IsResearchable` 0x0057e820 — called by the research-target gate applier +at 0x0088ff49 with the payload word straight off the command — bounds-checks against the length of +the 293-entry vector, which is what proves the payload is in the wire space. + +## 4. The table, and how to regenerate it + +`tools/techid_table.py` derives the whole mapping from the data file (or from the checked-in parse +oracle) and self-checks the four observed points: + +```sh +uv run python3 tools/techid_table.py # prints all 293 +uv run python3 tools/techid_table.py 282 288 144 90 # or just some +``` + +It is deliberately short: the rule is one `sorted(key=str.lower)` and a tool that obscured that +would be worse than no tool. It reads `$SOTS_GOB_DIR/TechTree/MasterTechList.tech` if that is set +(the same variable the engine's `realdata_test` uses) and otherwise the checked-in parse oracle, so +it needs no data tree. **It refuses to print a table whose four observed points do not agree** — +a short parse shifts every id after the gap, and the observed points are the only check it has. + +Honest bound: the raw-`.tech` regex path is exercised here only against a synthetic three-block +fixture (both `tech {` and `tech` + newline + `{`); the numbers in this note all come through the +parse-oracle path. + +## 5. What this lane did NOT do + +1. **No live confirmation of its own.** The 282 point is lane L4's observation, re-read. Nothing new + ran. The strongest available further test is free: any future capture reporting a techId in the + 48 sort-ambiguous slots. +2. **Only stock data.** A mod adding a tech file renumbers **every id after the insertion point**, + because the id is a sorted index over the union. That is a property of the original and a real + hazard for anyone replaying a modded game; untested here. +3. **The second `LoadTechFile` loop in the ctor (0x0058ba7e) was not read.** If it can add nodes + *after* 0x0058ba37, ids would be assigned before those nodes exist. Stock data has exactly one + tech file (`data-model.md`) and the 293-entry fit holds, so it does not bite here — but it is + unread, and it is the one way the rule could be incomplete. +4. **The 196-entry `.rdata` enum table was not dumped in full.** It is at the address behind + `MasterTechTree::ResolveTechIds` 0x00581c10; `scratchpad/techfx/techtable196.txt` has an old + dump of it (`CCC_AdvSens = 10001`, `IND_Waldo = 10002`, …). Note that + `nvo-tshn-visible-owner.md` calls `CCC_AdvSens` "tech id 10000"; the table says 10001. One of + the two is off by one and this lane did not resolve which. diff --git a/ghidra/addresses.d/lane-id.json b/ghidra/addresses.d/lane-id.json new file mode 100644 index 0000000..4624c72 --- /dev/null +++ b/ghidra/addresses.d/lane-id.json @@ -0,0 +1,92 @@ +{ + "entries": [ + { + "name": "IDMap_Initialize", + "addr": "0x008b9ad0", + "convention": "thiscall", + "prototype": "bool (IDMap* this, int numNodes, int localNode, int startId) // ret 0xc. THE SEEDING ENTRY POINT, and it names the save's three NM* tags. Calls FUN_008b8f80 (clear), then REFUSES numNodes > 0x10 ('IDMap: Cannot support %d nodes.') -- SIXTEEN is the hard cap, and it is the low nibble of IDMap_AllocateID's id. Refuses localNode outside [0, numNodes) ('IDMap: Node %d does not exist, valid nodes are 0-%d.'). On success: resize the 0x14-stride node vector at this->+0x08 via FUN_008b99d0, ZERO EVERY NODE'S COUNTER (the loop at 0x008b9b20 stores 0 to +eax+0x10, eax += 0x14), then write startId into nodes[localNode].counter ALONE (0x008b9b38), then this->+0x18 = localNode (0x008b9b3d). SO: exactly one node's counter is ever seeded; every other node starts at 0, and nothing else in the class ever restores one -- IDMap_Insert 0x008b9350 does NOT touch a counter. Four call sites: StrategyServer_Read +0x105 = Initialize(NMSz, NMLc, NMnx) on S+0x84 -- WHICH NAMES THE TAGS: NMSz = node count, NMLc = LOCAL NODE INDEX, NMnx = that node's counter; StrategyServer_LoadGame +0x107 = Initialize(16, 0, 0); StrategySim_OnCreateGame +0x45 = Initialize(msg->+4, msg->+8, msg->+0xc) on the CLIENT's map at +0x80; and FUN_008a9f80 +0x1ce = Initialize(1, 0, 0) on an unrelated Mars scene object's map at +0x68", + "status": "verified", + "source": "findings/subsystems/id-allocation.md" + }, + { + "name": "IDMap_AllocateOnLocalNode", + "addr": "0x008b8b70", + "convention": "thiscall", + "prototype": "int (IDMap* this) // 10 B: push [ecx+0x18] (the local node index); call IDMap_AllocateID; ret. The IDMap-relative form of the same thunk IDMap_AllocateLocalID 0x0080f710 wraps from the StrategySim side (that one is `ecx += 0x80` then jmp here). Five direct call sites, all StrategySim methods reached on BOTH the server and a client: StrategySim_CreateDesign +0xe0, FUN_008713a0 +0x3df, SystemBuildQueue_AttachBuiltShip 0x0088e7f0 +0x42, TradeManager_SpawnEncounterSquadron 0x0088f070 +0x216 and +0x28f", + "status": "verified", + "source": "findings/subsystems/id-allocation.md" + }, + { + "name": "IDMap_GetNodeCounter", + "addr": "0x008b8b80", + "convention": "thiscall", + "prototype": "int (IDMap* this, int nodeIndex) // 29 B, ret 4: node = IDMap_FindNode(nodeIndex); return node ? node->+0x10 : 0. A READ of a per-node counter with no side effect. Its only interesting caller is StrategyServer_InitGameForPlayer 0x007c8f05, which uses it to SEED a client: the CreateGame message carries the server's current counter for the node that client is about to own. Since a save restores only ONE node's counter (Initialize zeroes the rest), this returns 0 for every client node in a freshly loaded game", + "status": "verified", + "source": "findings/subsystems/id-allocation.md" + }, + { + "name": "IDMap_FindNode", + "addr": "0x008b8a70", + "convention": "thiscall", + "prototype": "void* (IDMap* this, int nodeIndex) // 101 B, ret 4. Bounds-checks nodeIndex against (this->+0x0c - this->+0x08)/0x14 -- the 0x66666667 / sar 3 divide-by-20 that pins the NodeEntry stride at 0x14 -- logs 'IDMap: Node %d does not exist, valid nodes are 0-%d.' and returns 0 when out of range; else returns _Myfirst + nodeIndex*0x14. Shared by IDMap_AllocateID and IDMap_GetNodeCounter", + "status": "verified", + "source": "findings/subsystems/id-allocation.md" + }, + { + "name": "StrategySim_OnCreateGame", + "addr": "0x00776f20", + "convention": "thiscall", + "prototype": "void (StrategySim* this, CreateGameMsg* msg) // ret 4. CASE 0 of StrategyClient::RaiseEvent 0x00783ee0's 0x2c-entry jump table at 0x00784200 (the handler body is at 0x00783f05). FIRST ACT: IDMap_Initialize(this+0x80, msg->+0x04 numNodes, msg->+0x08 localNode, msg->+0x0c startId) -- THIS IS WHERE A CLIENT'S ID SPACE IS SET UP, and the only path by which an IDMap ever gets a local node index other than 0. Then copies the rest of the message into the sim: +0x10 -> this->+0x08, +0x48 -> this->+0x10, +0x14 -> this->+0xb8, the two floats at +0x18/+0x1c -> this->+0xbc/+0xc0, the bools at +0x20/+0x21 -> this->+0xc4/+0xc5, +0x6c -> this->+0x154, +0x4c -> this->+0xf8, then the vectors from this+0x40 on", + "status": "verified", + "source": "findings/subsystems/id-allocation.md" + }, + { + "name": "StrategyServer_InitGameForPlayer", + "addr": "0x007c8d90", + "convention": "thiscall", + "prototype": "void (StrategyServer* this /*the S frame*/, int playerObjectId, int arg2) // THE NODE-INDEX ASSIGNMENT, in four instructions. Resolves the player through this->+0x84 (the id-to-object lookup FUN_008b9240) and takes esi = player->PlyrIdx(+0x28). Then at 0x007c8ecb: `cmp esi,-1 / je L / inc esi / jmp / L: xor esi,esi` -- localNode = (PlyrIdx == -1) ? 0 : PlyrIdx + 1. NODE 0 IS THE SERVER'S; PLAYER k GETS NODE k+1. It then fills the CreateGame message on the stack at [ebp-0x138]: +0x00 vtable 0x00a252c0, +0x04 numNodes = (server IDMap node vector length, the /20 divide at 0x007c8ee1 over ebx+0x8c/+0x90), +0x08 localNode = that esi, +0x0c startId = IDMap_GetNodeCounter(S+0x84, localNode). Sends it, and StrategySim_OnCreateGame is what receives it. See findings/subsystems/id-allocation.md section 3 for why startId is always 0 for a client after a save load", + "status": "verified", + "source": "findings/subsystems/id-allocation.md" + }, + { + "name": "StrategyServer_LoadGame", + "addr": "0x007dd530", + "convention": "thiscall", + "prototype": "void (StrategyServer* this, ...) // At +0x107 it calls IDMap_Initialize(this+0x84, 16, 0, 0) -- SIXTEEN NODES, LOCAL NODE 0, COUNTER 0. That is where the corpus's NMSz 16 / NMLc 0 come from, and it is why every id created by the host carries low nibble 0", + "status": "verified", + "source": "findings/subsystems/id-allocation.md" + }, + { + "name": "StrategySim_CreateDesign", + "addr": "0x008827e0", + "convention": "thiscall", + "prototype": "void (StrategySim* this, ServerPlayer* owner, void* params, int explicitId, char, char, char, char) // ret 0x18. THE CLIENT-ALLOCATES / SERVER-HONOURS SPLIT, in one branch. operator new(0x1a8) -> ctor FUN_00874c70 -> FUN_0057c6d0(params); design->+0x130 = owner, design->+0x134 = this->+0x08. Then at 0x008828b3: `if (explicitId != 0) use it; else id = IDMap_AllocateOnLocalNode(this+0x80)`, followed by IDMap_Insert(this+0x80, design+0xa0, id). BECAUSE StrategyClient AND StrategyServer BOTH DERIVE FROM StrategySim, this is the SAME code on both sides: the client runs it with explicitId 0 and mints an id on ITS node, puts that id in the turn command, and the server runs it again with explicitId set and mints nothing. A reimplementation that allocates on apply produces every AI-created id wrong", + "status": "verified", + "source": "findings/subsystems/id-allocation.md" + }, + { + "name": "MasterTechTree_SortAndNumber", + "addr": "0x0058b9df", + "convention": "label", + "status": "verified", + "prototype": "THE techId -> NAME MAP, in eight instructions inside MasterTechTree_ctor 0x0058b870, immediately after the LoadTechFile loop. 0x0058b9df: vector::operator=(this+0x24, this+0x14) -- the sorted list at +0x24 is a COPY of the parse-order list at +0x14, which is why a parse-order dump does not match the ids. 0x0058b9ff: std::sort(first=[this+0x24], last=[this+0x28], ideal=(last-first)/4, pred) = FUN_00583b10, MSVC _Sort (the _ISORT_MAX 0x20 test and the _Ideal 3/4 halving are both there); the predicate is INLINED in FUN_00581130 at 0x00581190 and 0x005811b8 as `_stricmp(a->name, b->name) < 0` on TechDef+0x04 (a std::string, SSO-tested at +0x14 against 0x10) -- MSVCR100 _stricmp, so the order is CASE-INSENSITIVE, not byte-wise. 0x0058ba19..0x0058ba37: `for (i = 0; i < n; ++i) sortedList[i]->+0x00 = i;` -- THE WIRE techId IS LITERALLY THE 0-BASED INDEX INTO THE _stricmp-SORTED MASTER LIST. The same sorted vector is then the binary-search index: FUN_0057f120 (std::lower_bound) resolves every `requires`/`allows` name against it at 0x0058baed. NOT the 10000-based TechID enum, which is a separate 196-entry .rdata table", + "source": "findings/subsystems/techid-name-map.md" + }, + { + "name": "TechTree_IsResearchable", + "addr": "0x0057e820", + "convention": "thiscall", + "prototype": "bool (TechTree* this /*per-player*/, int techId) // THE WIRE-SPACE BOUNDS CHECK, and it is what proves the space. Rejects techId < 0 and techId >= (this->+0x14 - this->+0x10)/4 -- the per-player node vector, 293 entries in stock data. Then esi = masterList[techId] from this->+0x04 -> +0x24/+0x28 (the SORTED vector, MasterTechTree_SortAndNumber) and edx = this->+0x10[techId] (the player's node). Reads the name as a std::string at masterEntry+0x04 with the SSO test at +0x18 -- so TechDef is {+0x00 int techId, +0x04 std::string name, ...}. Called by the research-target gate applier at 0x0088ff49 with the techId straight off the command payload ([block-0x28])", + "status": "verified", + "source": "findings/subsystems/techid-name-map.md" + }, + { + "name": "MSVC_Sort", + "addr": "0x00583b10", + "convention": "cdecl", + "prototype": "void (T** first, T** last, int ideal, Pred pred) // MSVC std::sort's _Sort: the `(last-first)/4 <= 0x20` insertion-sort cutoff at 0x00583b26, the `ideal -= ideal/2 + ideal/4` heap-sort fallback counter, and _Unguarded_partition FUN_00581130. Identified here because MasterTechTree_ctor uses it to build the tech id space; the same body will be reached from anywhere else that sorts a pointer vector", + "status": "verified", + "source": "findings/subsystems/techid-name-map.md" + } + ] +} diff --git a/tools/techid_table.py b/tools/techid_table.py new file mode 100644 index 0000000..92b1cbe --- /dev/null +++ b/tools/techid_table.py @@ -0,0 +1,92 @@ +#!/usr/bin/env python3 +"""techId -> tech name, derived offline from the tech tree data. + +The wire techId is the 0-based index of the tech's name in the master tech list sorted with +_stricmp -- case-insensitively. Read from MasterTechTree's constructor: it copies the parse-order +list to a second vector, std::sort()s that copy with an inlined `_stricmp(a->name, b->name) < 0`, +then walks it writing `def->techId = i`. See findings/subsystems/techid-name-map.md. + +This is NOT the 10000-based TechID enum, which is a separate 196-entry .rdata table. + +Usage: + tools/techid_table.py # all of them + tools/techid_table.py 282 288 144 90 # just these +""" +import glob +import json +import os +import re +import sys + +ROOT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..") + +# Four (techId, name) pairs observed live, by two lanes on different runs. 282 is the one the +# rule was NOT fitted to -- it comes from lane L4's run R2 (findings/subsystems/ai-order-capture.md) +# and is the check that this file is right rather than merely self-consistent. +OBSERVED = {90: "DRV_PlsFiss", 144: "IND_Waldo", 282: "XNC_TrnsHum2", 288: "XNC_TrnsMorr2"} + +# Where the 293 names can come from, in order of preference: the raw game file under +# $SOTS_GOB_DIR (the same variable the engine's realdata_test uses), then the checked-in parse +# oracle in sots-engine's tests, which needs no data tree at all. +SOURCES = [p for p in [ + os.path.join(os.environ["SOTS_GOB_DIR"], "TechTree", "MasterTechList.tech") + if os.environ.get("SOTS_GOB_DIR") else None, + os.path.join(ROOT, "..", "sots-engine", "tests", "mars_parse", "build", "oracle", + "TechTree", "MasterTechList.tech.json"), +] if p] + + +def names_from_tech_file(path): + """Pull every `tech { name "X" ... }` block name out of the raw .tech file.""" + text = open(path, encoding="latin-1").read() + return re.findall(r'^\s*tech\b[^\n]*\n(?:[^\n]*\n)*?\s*name\s+"([^"]+)"', text, re.M) + + +def names_from_oracle(path): + return [t["name"] for t in json.load(open(path))["tech"]] + + +def load_names(): + for pat in SOURCES: + for path in sorted(glob.glob(pat)): + if os.path.exists(path): + return (names_from_oracle(path) if path.endswith(".json") + else names_from_tech_file(path)), path + for path in sorted(glob.glob(os.path.join(ROOT, "**", "MasterTechList.tech"), recursive=True)): + return names_from_tech_file(path), path + sys.exit("no MasterTechList.tech (or its parse oracle) found; see SOURCES in this file") + + +def table(): + names, src = load_names() + if len(set(n.lower() for n in names)) != len(names): + sys.exit("case-folded duplicate tech names: the sorted order is not total, stop") + # _stricmp order. str.lower reproduces it exactly for these pure-ASCII names. + return sorted(names, key=str.lower), src + + +def main(argv): + ids, src = table() + bad = [(i, want, ids[i] if i < len(ids) else "") + for i, want in OBSERVED.items() if i >= len(ids) or ids[i] != want] + if bad: + print(f"# {len(ids)} names parsed from {src}", file=sys.stderr) + for i, want, got in bad: + print(f"MISMATCH techId {i}: observed {want!r}, derived {got!r}", file=sys.stderr) + print("The observed points are the only check this file has. Do not trust the table " + "until they agree -- a short parse (a missed block) shifts EVERY id after it.", + file=sys.stderr) + return 1 + print(f"# {len(ids)} techs from {os.path.relpath(src, ROOT)}; " + f"{len(OBSERVED)}/{len(OBSERVED)} observed points agree", file=sys.stderr) + wanted = [int(a) for a in argv] if argv else range(len(ids)) + for i in wanted: + if 0 <= i < len(ids): + print(f"{i:4d} {ids[i]}{' <- observed' if i in OBSERVED else ''}") + else: + print(f"{i:4d} ") + return 0 + + +if __name__ == "__main__": + sys.exit(main(sys.argv[1:]))