alex/sots-re
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

lane ID: one id allocator on sixteen node counters, and techId is a sorted index

Browse source 
The client-side allocator is not a second allocator. StrategyServer and StrategyClient
are both StrategySim, which owns an IDMap at +0x80; each sim allocates from its own map
on its own local node index. StrategyServer::InitGameForPlayer sets that index to
PlyrIdx + 1 (node 0 is the server's) and seeds the client from the server's counter for
that node. IDMap::Initialize names the save's NM* tags: NMSz nodes, NMLc local node,
NMnx that node's counter -- confirming B5's labelled hypothesis and adding the other two.
Cross-checked on 20 saves: nodes 1, 2 and 3 all occur, counters run from 1 per node, and
2,600 ids collide zero times. Corrects turn-command-replay.md row 2: design 18 IS in
turn2-state.sav, so the canonical pair needs one minted id, not two. One open item, with
the one-hook probe named: a reloaded save produced fleet 34 rather than 18, and nothing I
read restores a client counter.

techId is the 0-based index into the master tech list sorted by _stricmp -- read out of
MasterTechTree's ctor, which sorts a copy of the parse-order list and then writes
def->techId = i. 282 is XNC_TrnsHum2, which lane L4 had already observed live and nobody
connected. tools/techid_table.py derives all 293 offline and refuses to print unless the
four observed points agree.
This commit is contained in:
alex 2026-09-08 20:22:24 -04:00
parent f69e078247
commit 975c19a57d
4 changed files with 639 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

309
findings/subsystems/id-allocation.md Normal file
View file

@ -0,0 +1,309 @@
# Object id allocation: one allocator, sixteen node counters, and why the client's ids are free
Lane ID, 2026-09-08. **Host work only — no VM run was needed and none was taken.** VM140 was held
and released untouched; its `SavedGames` set is byte-identical to the oracle (§8).
Closes row 2 of `turn-command-replay.md` §4 — "the client's id allocator" — which lane RB flagged as
"a watchpoint, not a week of reading". It turned out to be neither: it is arithmetic on ids the
corpus already held, confirmed against the instruction stream and cross-checked on twenty saves.
Predictions were written before any of the work below and are quoted verbatim where they were
falsified (§3.2). Consumes: `combat-retreat-pipeline.md` (B5, `IDMap::AllocateID`),
`turn-command-replay.md` (RB), `ai-order-emission.md` (AI4), `struct-recovery.md`.
---
## 0. Lead: there is no second allocator
RB's gap list says "**two id spaces**, and the small one is part of the wire protocol". There are
two id *spaces*, and RB is right that a reimplementation which allocates on apply gets every
AI-created id wrong. But there is **one allocator, one implementation, and one formula** — B5's
`IDMap::AllocateID` 0x008b8ae0, `id = (counter << 4) | (node & 0xF)`.
What differs is **which node's counter** is used, and that is a property of the *sim object*, not of
the code:
> `Game::StrategyServer` and `Game::StrategyClient` are **both** `Game::StrategySim`
> (RTTI: `StrategyClient` bases `[StrategyClient, StrategySim]` at offset 0; `StrategyServer` bases
> `[StrategyServer, IStreamable, StrategySim]` with vftables at offset 0 **and** offset 4 — that is
> the "two bases 4 bytes apart" of earned rule 1, and the +4 one is the `StrategySim` subobject).
>
> `StrategySim` owns an `IDMap` at **`StrategySim + 0x80`** — i.e. `StrategyServer + 0x84`,
> `StrategyClient + 0x80`. Every sim allocates from **its own** map, on **its own** local node
> index at `IDMap + 0x18`.
So the object-creation code is base-class code that runs unchanged on both sides. The whole
client-allocates/server-honours protocol is one branch in `StrategySim::CreateDesign` 0x008827e0:
```
008828b3 mov eax,[ebp+0x10] ; explicitId, the third argument
008828b6 test eax,eax
008828b8 jne 0x8828c5 ; non-zero -> use the id the command carried
008828ba lea ecx,[edi+0x80] ; else: this sim's own IDMap
008828c0 call 0x8b8b70 ; IDMap::AllocateOnLocalNode
008828c5 ... call 0x8b9350 ; IDMap::Insert(map, design+0xa0, id)
```
The client calls it with `explicitId = 0` and mints; the server calls it with the id off the wire
and mints nothing.
## 1. The arithmetic, which is the whole result
B5's formula, run backwards on RB's numbers:
| id | | counter | node |
|---:|---|---:|---:|
| **18** design | *client-allocated* | **1** | **2** |
| **34** fleet | *client-allocated* | **2** | **2** |
| 1712 | from the master counter | 107 | 0 |
| 1728 | ship in the input save | 108 | 0 |
| 1776 | | 111 | 0 |
| 272, 288 systems; 32, 496, 512 players | | 17, 18, 2, 31, 32 | 0 |
Every id that comes out of a save carries node nibble **0**. Both client-allocated ids carry node
nibble **2**, with counters **1** and **2** — the first two ids that node ever issued.
### 1.1 On twenty saves, four games, three distinct client nodes
Nibble histogram over `FltID`/`DesID`/`ShipID`/`SysID`/`PlayerID` in the whole corpus
(`verify/results/saves/*.sav`, read with `verify/save-reader/save_reader.py --dump`):
| save | ids | nibbles | the non-zero ones |
|---|---:|---|---|
| `turn1-state` | 98 | `{0: 98}` | — |
| `turn2-state` | 102 | `{0: 101, 2: 1}` | 18 |
| `turn3-state` | 104 | `{0: 102, 2: 2}` | 18, 34 |
| `cb-turn2to3-autosave` | 104 | `{0: 102, 2: 2}` | 18, 34 |
| `human-turn3-noderoute` | 90 | `{0: 86, 2: 4}` | 18, 34, 50, 66 |
| `human-turn8-traderoutes` | 151 | `{0: 141, 2: 10}` | 18 … 242 |
| `human-turn15-spyprogram` | 295 | `{0: 252, **1: 1**, 2: 42}` | **17**, 18 … 818 |
| `zuul-turn15-orders` | 150 | `{0: 139, 2: 10, **3: 1**}` | 18 … 162, **19** |
| `zuul-turn17-orders2` | 167 | `{0: 151, 2: 14, **3: 2**}` | 18 … 226, **19, 35** |
| `zuul-turn23-fleet23` | 222 | `{0: 198, 2: 22, 3: 2}` | … |
Three facts fall straight out, and none of them needed a hook.
1. **Nodes 1, 2 and 3 all occur.** "The client" is not singular. Each node's counters run
**1, 2, 3, …** contiguously from the start of the game — node 2's first id is `18` in *every*
save in the corpus, node 3's first is `19`, node 1's first is `17`.
2. **The partition is absolute.** ~2,600 ids across four games, zero collisions between spaces.
That is the entire purpose of the nibble and of `IDMap::Initialize`'s `numNodes <= 0x10` cap.
3. **Node = owning player's index + 1.** Every node-nibble object belongs to exactly one player:
| save | id | owner `PID` | that player's `PlyrIdx` | node |
|---|---:|---:|---:|---:|
| `turn3-state` | design 18, fleet 34 | 32 | 1 | 2 |
| `human-turn15` | design 17 | 16 | 0 | 1 |
| `human-turn15` | design 18 | 32 | 1 | 2 |
| `zuul-turn17` | designs 18, 34, … | 32 | 1 | 2 |
| `zuul-turn17` | designs 19, 35 | 496 | 2 | 3 |
## 2. The node index, read from the instruction stream
`StrategyServer::InitGameForPlayer` 0x007c8d90 builds the CreateGame message for one player's
client. Four instructions do the assignment:
```
007c8ddf mov esi,[eax+0x28] ; player->PlyrIdx
...
007c8ecb cmp esi,0xffffffff
007c8ece je 0x7c8ed3
007c8ed0 inc esi ; localNode = PlyrIdx + 1
007c8ed1 jmp 0x7c8ed5
007c8ed3 xor esi,esi ; PlyrIdx == -1 -> localNode = 0
007c8ed5 ... [ebx+0x90] - [ebx+0x8c], /20 ; numNodes = the server map's node count
007c8ef2 mov [ebp-0x134],ecx ; msg->numNodes
007c8ef8 push esi
007c8ef9 lea ecx,[ebx+0x84] ; the SERVER's IDMap
007c8eff mov [ebp-0x130],esi ; msg->localNode
007c8f05 call 0x8b8b80 ; IDMap::GetNodeCounter(server map, localNode)
007c8f1a mov [ebp-0x12c],eax ; msg->startId
```
The message is `{numNodes, localNode = PlyrIdx + 1, startId = the server's counter for that node}`.
The client receives it as **case 0** of `StrategyClient::RaiseEvent` 0x00783ee0 (jump table
0x00784200, handler 0x00783f05) and `StrategySim::OnCreateGame` 0x00776f20 calls
`IDMap::Initialize` on its own map at `+0x80` with exactly those three words.
So `PlyrIdx + 1` is not an inference from the saves any more. The saves and the disassembly agree,
which is the only reason either is worth reporting.
## 3. Seeding: what a save carries, and what it cannot
`IDMap::Initialize(numNodes, localNode, startId)` 0x008b9ad0, `ret 0xc`:
* refuses `numNodes > 0x10` — **sixteen nodes, and that is the four-bit nibble**;
* refuses `localNode` outside `[0, numNodes)`;
* resizes the 0x14-stride node vector, then **zeroes every node's counter** (0x008b9b20);
* writes `startId` into **`nodes[localNode].counter` alone** (0x008b9b38);
* stores `localNode` at `this->+0x18` (0x008b9b3d).
### 3.1 The save's `NM*` tags are now named
`StrategyServer::Read` 0x007d27a0 reads three ints and passes them straight in:
```
NMSz -> numNodes NMLc -> localNode NMnx -> startId
```
That **confirms and extends B5's labelled hypothesis** ("the counter is almost certainly NMnx").
It is NMnx, it is the counter *for node NMLc only*, and the two tags beside it are the node count
and the local node index. Corpus:
| save | NMSz | NMLc | NMnx | ModCount | Frame |
|---|---:|---:|---:|---:|---:|
| `turn1-state` | 16 | 0 | 106 | 0 | 1 |
| `turn2-state` | 16 | 0 | 109 | 12 | 2 |
| `turn3-state` | 16 | 0 | **111** | 24 | 3 |
`StrategyServer::LoadGame` 0x007dd530 sets a fresh game up with `Initialize(16, 0, 0)`, which is
where `NMSz 16` / `NMLc 0` come from and why the host's ids all carry nibble 0.
**A correction to `turn-command-replay.md` §4 row 2** (earned rule 11). It says the server "issued
1712 and 1776 the same turn", and that design 18 and fleet 34 are "objects that do not exist in the
input save". Neither is quite right on the canonical pair:
* 1712 is counter **107**, issued during turn **1→2** (NMnx 106 → 109 covers counters 107, 108, 109
= ids 1712, 1728, 1744). 1776 is counter 111, issued during turn **2→3** (109 → 111).
* **Design 18 is present in `turn2-state.sav`.** It is in the master `DesignIDs` list and in player
32's block, and it was allocated during turn 1→2. Only **fleet 34** is new on the canonical pair.
So the turn-2→3 build order names an id the input save already holds, and a faithful replay of
that pair has to mint exactly **one** id, not two. That is a materially smaller gap than the row
claims.
Everything else in the row stands, including the load-bearing part: allocate on apply and you get
`1792` where the original has `34`.
### 3.2 A falsified prediction, and the open item it leaves
Prediction ID-P3 said, in full:
> If NMLc = 0 holds, then the node-2 counter that issued 18 and 34 is NOT in the save, is NOT
> restored by Initialize (which zeroes it), and therefore **restarts at zero on every load**.
NMLc is 0, and the first two clauses are read directly out of the code: no save carries a per-node
array, `Initialize` zeroes the rest, `IDMap::Insert` 0x008b9350 **never touches a counter** (read to
the end: it logs a duplicate, refuses id 0, bounds-checks `id & 0xF`, writes the id into `obj->+4`
and inserts into `nodes[n]`'s map — no counter write), and `InitGameForPlayer` seeds a client from
`GetNodeCounter`, which therefore returns 0 for every client node in a freshly loaded game.
**The third clause is falsified by a save already in the corpus.** `cb-turn2to3-autosave.sav` is a
fresh process that *loaded* `turn2-state.sav` (which contains design 18 = node 2, counter 1) and
pressed End Turn. The fleet it created is **34 — counter 2**, not 18. If the client's counter had
started at 0 the fleet would have been 18, colliding with a design that already exists.
Two explanations survive, and this lane cannot separate them without a hook:
* **(A)** something restores the server's node-2 counter to 1 during the load, and the client is
seeded with it. I read `Initialize`, `Insert`, `AllocateID`, `FindNode`, `GetNodeCounter` and
`InitGameForPlayer` and found no such write. Absence in six functions is not absence.
* **(B)** the counter really does start at 0 and the client made **one earlier allocation that turn
which never reached the save** — a local design object, a task, an order — taking counter 1, so
the fleet got counter 2. This fits the read code exactly and requires nothing unread.
**The probe is one hook and it is cheap** (earned rule 18, and rule 20 — instrument the *entry*):
detour `IDMap::AllocateID` 0x008b8ae0 and log `(this, nodeIndex, resulting id, return address)` for
one `turn2-state → turn3-state` End Turn. (A) predicts the client's very first allocation that turn
returns 34; (B) predicts it returns 18 from a site that is not the fleet creator. **A count alone
cannot separate them** — that is exactly the shape rule 20 exists for.
Whichever it is, the consequence for a reimplementation is the same and is stated in §5.
### 3.3 The collision hazard, stated as a hypothesis
If (B) is right, then a client's counter is a pure function of **allocation order within the
process** and a save/load loses it. Load a save in which a client has already allocated *n* objects
and let it allocate again, and it re-issues `(1<<4)|k` — an id the map already holds. `IDMap::Insert`
logs `IDMap: Object already exists with id %d.` and **inserts anyway** (0x008b9367: the log is
followed by fall-through, not a return). That would be an original defect, not ours.
Labelled hypothesis. The same one-hook probe settles it, and `human-turn15-spyprogram.sav` — node 2
at counter 51 — is the workload that would make it loud.
## 4. Layout, for the record
```
IDMap (StrategySim + 0x80)
+0x08 / +0x0c / +0x10 vector<NodeEntry>, STRIDE 0x14 (allocator +0x10, rule 5)
+0x18 int localNodeIndex
NodeEntry 0x14 bytes
+0x00 .. +0x0f std::map<int, void*> id -> object, for THIS node
+0x10 int counter pre-incremented; 0 is never issued
```
`AllocateID(node)`: 0 if `node == -1` or the node does not exist; else `++nodes[node].counter`
(pre-increment, and on wrap to 0 it logs and increments again, so `INVALID_NETWORK_ID` is never
handed out); `id = (counter << 4) | (node & 0xF)`, with an overflow log if the counter no longer
round-trips.
## 5. What a faithful replay has to do
1. **A per-node counter array, not one counter.** Sixteen of them.
2. **Node 0 for the sim that owns the board; node `PlyrIdx + 1` for each player's client.**
3. **Restore only node `NMLc`'s counter from `NMnx`.** There is nothing else in the save to
restore, and inventing a per-node array in the save format would diverge from the original.
4. **Mint the id at command *emission*, not at apply**, and carry it in the command — the applier
must honour a non-zero id and mint only on zero (`StrategySim::CreateDesign`'s branch).
5. **Do not model the client counter as a function of the save.** It is a function of allocation
order in the process, and until §3.2 is settled a replay of the canonical pair should take the
client's ids from the capture rather than derive them.
For the canonical pair specifically, (4) plus "design 18 is already in the input save" means the
replay needs exactly one minted id, `34`, and it is the *second* thing node 2 allocates that turn.
## 6. What this lane did NOT do
1. **No hook, no VM run, no new instrument.** Every number is from the checked-in corpus and the
image. That is also the limit: §3.2 is open precisely because nothing ran.
2. **The second `LoadTechFile` loop and the message *send* were not read.** I read
`InitGameForPlayer` up to the point where the message is filled, and `OnCreateGame` from the
point where it is received; the transport between them is assumed, not read. It is a single
process, so nothing observable depends on it — but it is not read.
3. **`FUN_008a9030`'s counter at `+0x23c` is a different mechanism** (a Mars scene object with its
own `IDMap` at `+0x68`, `Initialize(1, 0, 0)`, and a separate per-object counter gated on
`obj->+0x20 & 0xF == 2`). It is not game state and it is not this. Recorded so the next lane
that greps for `& 0xF` does not chase it.
4. **Multiplayer is untested.** Every save in the corpus is `NMLc 0`. A real client machine would
save with `NMLc = k` and its own counter, and *the server's* copy of that counter is the thing
§3.2 is about. Nothing here has been exercised across two machines.
5. **`ShipID` allocation was not traced to its site.** Ships in the corpus are all node 0, so they
are minted server-side; I did not read which function does it.
## 7. Reproducing every number here
```sh
cd ~/sots-re
uv run python3 verify/save-reader/save_reader.py --dump verify/results/saves/turn3-state.sav \
| grep -E 'NMSz|NMLc|NMnx'
# nibble histogram: see the one-screen script in section 1.1's method note below
uv run python3 dumps/b6dis.py 0x008b9ad0 # IDMap::Initialize
uv run python3 dumps/b6dis.py 0x007c8d90 # InitGameForPlayer; the +1 is at 0x007c8ed0
```
The histogram is `re.match(r'\s*@[0-9a-f]+\s+(\S+)\s+int\??\s+(-?\d+)', line)` over the dump,
keeping the five id tags and counting `value & 15`. No tool was added for it — it is four lines and
a tool would only hide what it does.
## 8. VM140 released
The game was never launched and nothing was written. `C:\SOTS\SavedGames` verified over SSH before
release — **8 files**, and the three oracle hashes match:
| file | bytes | sha256[0:16] | |
|---|---:|---|---|
| `(Autosave).sav` | 67,219 | `978041acd168b56e` | ✅ oracle |
| `(Autosave EndTurn).sav` | 66,732 | `bb4fd9ac89f41e3b` | ✅ oracle |
| `(Autosave Backup).sav` | 67,219 | `978041acd168b56e` | ✅ oracle |
| `MyGameverify1rtD.sav` | 66,732 | `bb4fd9ac89f41e3b` | |
| `MyGameverify1rtE.sav` | 66,732 | `bb4fd9ac89f41e3b` | |
| `MyGameverify1verify1.sav` | 66,732 | `bb4fd9ac89f41e3b` | |
| `ref-turn2.sav` | 66,739 | `ab4ac2d7e2977260` | |
| `zuul-turn5.sav` | 59,131 | `48559ab5b719b332` | |
Note for the next VM lane: a filename with parentheses cannot be hashed through
`ssh … 'certutil -hashfile "…(Autosave).sav"'` — `cmd.exe` eats the parens and returns
"Check the spelling", which reads exactly like a missing file. Use
`powershell -NoProfile -c "(Get-FileHash -LiteralPath '…' -Algorithm SHA256).Hash"`. My first
attempt reported the three oracle files as absent and they were never absent.
**Please mark VM140 free.**

146
findings/subsystems/techid-name-map.md Normal file
View file

@ -0,0 +1,146 @@
# `techId` -> tech name: an offline lookup, not a capture to transcribe
Lane ID, 2026-09-08. Host work only.
Closes row 8 of `turn-command-replay.md` §4 and the open value that kept Rung C-set's "we can name
all *k*" from closing.
> **282 is `XNC_TrnsHum2`.**
And it was **already in our own notes**: lane L4 observed it live in run R2
(`ai-order-capture.md`, the two-capture table — "player 512 target: `XNC_TrnsHum2`, techId 282").
Nobody connected it, and lane CB then produced 282 as an unmapped value. Earned rule 18's corollary,
paid for again: **search the notes before the binary.**
---
## 1. The rule
> **`techId` is the 0-based index of the tech's name in the master tech list sorted by
> `_stricmp` — case-insensitively.**
293 nodes in stock `TechTree/MasterTechList.tech`, so `techId` runs 0..292. That is a complete,
offline, data-derived mapping: **no capture, no transcription, no VM.** The replayer can stop
carrying the observed name in its own column.
Every observed point, including the two the rule was not fitted to:
| techId | name | how it was observed |
|---:|---|---|
| 90 | `DRV_PlsFiss` | RB / L4 capture, player 496 |
| 144 | `IND_Waldo` | RB / L4 capture, player 32 |
| **282** | **`XNC_TrnsHum2`** | L4 `ai-order-capture.md` run R2, player 512 — **independent of the fit** |
| 288 | `XNC_TrnsMorr2` | RB / L4 capture, player 512 |
The rule was fitted on the three RB reported and then hit 282 on the nose against a live
observation taken days earlier by a different lane on a different run. It also reproduces L4's
structural claim about the family: the six-member `XNC_Trns<Species>2` set is exactly
**{282 Hum2, 284 Hvr2, 286 Lir2, 288 Morr2, 290 Trk2, 292 Zuul2}**, interleaved with the `…3`
variants — which is why L4 saw two picks "six apart".
## 2. Read from the instruction stream, not fitted
`MasterTechTree::MasterTechTree` 0x0058b870, immediately after the `LoadTechFile` loop:
```
0058b9df lea eax,[ebx+0x14] ; the PARSE-ORDER list
0058b9e3 lea ecx,[ebx+0x24]
0058b9e6 call 0x8b3210 ; vector::operator= -- +0x24 is a COPY of +0x14
0058b9eb..0058b9ff
push pred / push count=(last-first)/4 / push last / push first
call 0x583b10 ; MSVC std::sort::_Sort
0058ba19 xor eax,eax
0058ba23 mov edx,[ebx+0x24]
0058ba26 mov ecx,[edx+eax*4] ; sortedList[i]
0058ba29 mov [ecx],eax ; **def->techId = i**
0058ba31 inc eax
0058ba37 jl 0x58ba23
```
Three things this settles that the fit could not.
**It is a sorted COPY.** `+0x14` keeps parse order; `+0x24` is the sorted one and the id is its
index. That is why the parse-order pointer-table dump in `strategic-turn-internals.md`
(`turn3/techtable.txt`, which starts `IND_Waldo, IND_RefCoat, …`) does not match the ids, and why
RB correctly concluded the ids "are not indices into anything we hold" — we held the wrong list.
**The comparator is `_stricmp`, so the order is case-insensitive.** It is inlined into
`_Unguarded_partition` FUN_00581130 at 0x00581190 and 0x005811b8 as
`_stricmp(a->name, b->name) < 0`, reading a `std::string` at `TechDef+0x04` (SSO-tested at `+0x14`
against `0x10`); the import at `0x009dd328` resolves to **`MSVCR100.dll _stricmp`**.
This matters. **48 of the 293 ids differ between byte-wise ASCII order and case-insensitive order**
— the `CCC_AI*`/`CCC_Adv*` run at 21–28, `DRN_COL`/`DRN_Cmbt`, `DRV_NodFoc`/`DRV_Node`,
`DRV_ROOT`/`DRV_Rad…`, `IND_DSCon`/`IND_Decon`, `IND_ROOT`/`IND_RefCoat`, `WEP_APRtech`/`WEP_Ac…`,
`WEP_KKMsl`/`WEP_KelTrp`, `WEP_MWMsl`/`WEP_Mas…`, `WEP_Nukes`/`WEP_NukMine`, `WEP_Pls*`. **None of
the four observed points discriminates**, so this half is disassembly-only and the corpus cannot
confirm it. Flagged as such (earned rule 6): if a future capture ever reports a techId in one of
those 48 slots, it is a real test and it should be reported as one. No two names collide under
case-folding (checked over all 293), so `std::sort`'s instability cannot bite.
**Why sorted at all.** The same `+0x24` vector is the binary-search index: `FUN_0057f120`
(`std::lower_bound`) resolves every `requires` / `allows` name reference against it at 0x0058baed.
The id is the index into a lookup table that had to be sorted anyway.
## 3. The two tech id spaces, which is what made this look hard
| | space | base | size | where |
|---|---|---:|---:|---|
| **wire / command** | index into the `_stricmp`-sorted master list | 0 | 293 | `TechDef+0x00`; the research-target gate payload |
| **`TechID` enum** | index into `MasterTechTree->resolved[]` | **10000** | **196** | a `{name, id}` table in `.rdata`, `CCC_AdvSens = 10001` |
`MasterTechTree::GetTechDef` 0x0057d610 and `IsTech` 0x0057d5d0 take the **enum** id:
`if (id == 0xc5) return 0; id -= 10000; if ((unsigned)id > 0xc3) return 0; return resolved[id];`
(`0xc5` = 197 is the "no tech" sentinel.)
`TechTree::HasResearched` 0x0057d810 is the bridge and shows both in four instructions:
```
eax = enumId - 10000; eax = master->resolved[eax]; ; ENUM space -> TechDef*
eax = *eax; ; TechDef+0x00 = the WIRE techId
eax = this->+0x10[eax]; ; per-player node vector, indexed by WIRE id
return eax && eax->+0x14 == 4;
```
The enum space exists so that C++ code can name a specific tech; the wire space is a dense index
over *all* techs. `TechTree::IsResearchable` 0x0057e820 — called by the research-target gate applier
at 0x0088ff49 with the payload word straight off the command — bounds-checks against the length of
the 293-entry vector, which is what proves the payload is in the wire space.
## 4. The table, and how to regenerate it
`tools/techid_table.py` derives the whole mapping from the data file (or from the checked-in parse
oracle) and self-checks the four observed points:
```sh
uv run python3 tools/techid_table.py # prints all 293
uv run python3 tools/techid_table.py 282 288 144 90 # or just some
```
It is deliberately short: the rule is one `sorted(key=str.lower)` and a tool that obscured that
would be worse than no tool. It reads `$SOTS_GOB_DIR/TechTree/MasterTechList.tech` if that is set
(the same variable the engine's `realdata_test` uses) and otherwise the checked-in parse oracle, so
it needs no data tree. **It refuses to print a table whose four observed points do not agree** —
a short parse shifts every id after the gap, and the observed points are the only check it has.
Honest bound: the raw-`.tech` regex path is exercised here only against a synthetic three-block
fixture (both `tech {` and `tech` + newline + `{`); the numbers in this note all come through the
parse-oracle path.
## 5. What this lane did NOT do
1. **No live confirmation of its own.** The 282 point is lane L4's observation, re-read. Nothing new
ran. The strongest available further test is free: any future capture reporting a techId in the
48 sort-ambiguous slots.
2. **Only stock data.** A mod adding a tech file renumbers **every id after the insertion point**,
because the id is a sorted index over the union. That is a property of the original and a real
hazard for anyone replaying a modded game; untested here.
3. **The second `LoadTechFile` loop in the ctor (0x0058ba7e) was not read.** If it can add nodes
*after* 0x0058ba37, ids would be assigned before those nodes exist. Stock data has exactly one
tech file (`data-model.md`) and the 293-entry fit holds, so it does not bite here — but it is
unread, and it is the one way the rule could be incomplete.
4. **The 196-entry `.rdata` enum table was not dumped in full.** It is at the address behind
`MasterTechTree::ResolveTechIds` 0x00581c10; `scratchpad/techfx/techtable196.txt` has an old
dump of it (`CCC_AdvSens = 10001`, `IND_Waldo = 10002`, …). Note that
`nvo-tshn-visible-owner.md` calls `CCC_AdvSens` "tech id 10000"; the table says 10001. One of
the two is off by one and this lane did not resolve which.

92
ghidra/addresses.d/lane-id.json Normal file
View file

@ -0,0 +1,92 @@
{
"entries": [
{
"name": "IDMap_Initialize",
"addr": "0x008b9ad0",
"convention": "thiscall",
"prototype": "bool (IDMap* this, int numNodes, int localNode, int startId) // ret 0xc. THE SEEDING ENTRY POINT, and it names the save's three NM* tags. Calls FUN_008b8f80 (clear), then REFUSES numNodes > 0x10 ('IDMap: Cannot support %d nodes.') -- SIXTEEN is the hard cap, and it is the low nibble of IDMap_AllocateID's id. Refuses localNode outside [0, numNodes) ('IDMap: Node %d does not exist, valid nodes are 0-%d.'). On success: resize the 0x14-stride node vector at this->+0x08 via FUN_008b99d0, ZERO EVERY NODE'S COUNTER (the loop at 0x008b9b20 stores 0 to +eax+0x10, eax += 0x14), then write startId into nodes[localNode].counter ALONE (0x008b9b38), then this->+0x18 = localNode (0x008b9b3d). SO: exactly one node's counter is ever seeded; every other node starts at 0, and nothing else in the class ever restores one -- IDMap_Insert 0x008b9350 does NOT touch a counter. Four call sites: StrategyServer_Read +0x105 = Initialize(NMSz, NMLc, NMnx) on S+0x84 -- WHICH NAMES THE TAGS: NMSz = node count, NMLc = LOCAL NODE INDEX, NMnx = that node's counter; StrategyServer_LoadGame +0x107 = Initialize(16, 0, 0); StrategySim_OnCreateGame +0x45 = Initialize(msg->+4, msg->+8, msg->+0xc) on the CLIENT's map at +0x80; and FUN_008a9f80 +0x1ce = Initialize(1, 0, 0) on an unrelated Mars scene object's map at +0x68",
"status": "verified",
"source": "findings/subsystems/id-allocation.md"
},
{
"name": "IDMap_AllocateOnLocalNode",
"addr": "0x008b8b70",
"convention": "thiscall",
"prototype": "int (IDMap* this) // 10 B: push [ecx+0x18] (the local node index); call IDMap_AllocateID; ret. The IDMap-relative form of the same thunk IDMap_AllocateLocalID 0x0080f710 wraps from the StrategySim side (that one is `ecx += 0x80` then jmp here). Five direct call sites, all StrategySim methods reached on BOTH the server and a client: StrategySim_CreateDesign +0xe0, FUN_008713a0 +0x3df, SystemBuildQueue_AttachBuiltShip 0x0088e7f0 +0x42, TradeManager_SpawnEncounterSquadron 0x0088f070 +0x216 and +0x28f",
"status": "verified",
"source": "findings/subsystems/id-allocation.md"
},
{
"name": "IDMap_GetNodeCounter",
"addr": "0x008b8b80",
"convention": "thiscall",
"prototype": "int (IDMap* this, int nodeIndex) // 29 B, ret 4: node = IDMap_FindNode(nodeIndex); return node ? node->+0x10 : 0. A READ of a per-node counter with no side effect. Its only interesting caller is StrategyServer_InitGameForPlayer 0x007c8f05, which uses it to SEED a client: the CreateGame message carries the server's current counter for the node that client is about to own. Since a save restores only ONE node's counter (Initialize zeroes the rest), this returns 0 for every client node in a freshly loaded game",
"status": "verified",
"source": "findings/subsystems/id-allocation.md"
},
{
"name": "IDMap_FindNode",
"addr": "0x008b8a70",
"convention": "thiscall",
"prototype": "void* (IDMap* this, int nodeIndex) // 101 B, ret 4. Bounds-checks nodeIndex against (this->+0x0c - this->+0x08)/0x14 -- the 0x66666667 / sar 3 divide-by-20 that pins the NodeEntry stride at 0x14 -- logs 'IDMap: Node %d does not exist, valid nodes are 0-%d.' and returns 0 when out of range; else returns _Myfirst + nodeIndex*0x14. Shared by IDMap_AllocateID and IDMap_GetNodeCounter",
"status": "verified",
"source": "findings/subsystems/id-allocation.md"
},
{
"name": "StrategySim_OnCreateGame",
"addr": "0x00776f20",
"convention": "thiscall",
"prototype": "void (StrategySim* this, CreateGameMsg* msg) // ret 4. CASE 0 of StrategyClient::RaiseEvent 0x00783ee0's 0x2c-entry jump table at 0x00784200 (the handler body is at 0x00783f05). FIRST ACT: IDMap_Initialize(this+0x80, msg->+0x04 numNodes, msg->+0x08 localNode, msg->+0x0c startId) -- THIS IS WHERE A CLIENT'S ID SPACE IS SET UP, and the only path by which an IDMap ever gets a local node index other than 0. Then copies the rest of the message into the sim: +0x10 -> this->+0x08, +0x48 -> this->+0x10, +0x14 -> this->+0xb8, the two floats at +0x18/+0x1c -> this->+0xbc/+0xc0, the bools at +0x20/+0x21 -> this->+0xc4/+0xc5, +0x6c -> this->+0x154, +0x4c -> this->+0xf8, then the vectors from this+0x40 on",
"status": "verified",
"source": "findings/subsystems/id-allocation.md"
},
{
"name": "StrategyServer_InitGameForPlayer",
"addr": "0x007c8d90",
"convention": "thiscall",
"prototype": "void (StrategyServer* this /*the S frame*/, int playerObjectId, int arg2) // THE NODE-INDEX ASSIGNMENT, in four instructions. Resolves the player through this->+0x84 (the id-to-object lookup FUN_008b9240) and takes esi = player->PlyrIdx(+0x28). Then at 0x007c8ecb: `cmp esi,-1 / je L / inc esi / jmp / L: xor esi,esi` -- localNode = (PlyrIdx == -1) ? 0 : PlyrIdx + 1. NODE 0 IS THE SERVER'S; PLAYER k GETS NODE k+1. It then fills the CreateGame message on the stack at [ebp-0x138]: +0x00 vtable 0x00a252c0, +0x04 numNodes = (server IDMap node vector length, the /20 divide at 0x007c8ee1 over ebx+0x8c/+0x90), +0x08 localNode = that esi, +0x0c startId = IDMap_GetNodeCounter(S+0x84, localNode). Sends it, and StrategySim_OnCreateGame is what receives it. See findings/subsystems/id-allocation.md section 3 for why startId is always 0 for a client after a save load",
"status": "verified",
"source": "findings/subsystems/id-allocation.md"
},
{
"name": "StrategyServer_LoadGame",
"addr": "0x007dd530",
"convention": "thiscall",
"prototype": "void (StrategyServer* this, ...) // At +0x107 it calls IDMap_Initialize(this+0x84, 16, 0, 0) -- SIXTEEN NODES, LOCAL NODE 0, COUNTER 0. That is where the corpus's NMSz 16 / NMLc 0 come from, and it is why every id created by the host carries low nibble 0",
"status": "verified",
"source": "findings/subsystems/id-allocation.md"
},
{
"name": "StrategySim_CreateDesign",
"addr": "0x008827e0",
"convention": "thiscall",
"prototype": "void (StrategySim* this, ServerPlayer* owner, void* params, int explicitId, char, char, char, char) // ret 0x18. THE CLIENT-ALLOCATES / SERVER-HONOURS SPLIT, in one branch. operator new(0x1a8) -> ctor FUN_00874c70 -> FUN_0057c6d0(params); design->+0x130 = owner, design->+0x134 = this->+0x08. Then at 0x008828b3: `if (explicitId != 0) use it; else id = IDMap_AllocateOnLocalNode(this+0x80)`, followed by IDMap_Insert(this+0x80, design+0xa0, id). BECAUSE StrategyClient AND StrategyServer BOTH DERIVE FROM StrategySim, this is the SAME code on both sides: the client runs it with explicitId 0 and mints an id on ITS node, puts that id in the turn command, and the server runs it again with explicitId set and mints nothing. A reimplementation that allocates on apply produces every AI-created id wrong",
"status": "verified",
"source": "findings/subsystems/id-allocation.md"
},
{
"name": "MasterTechTree_SortAndNumber",
"addr": "0x0058b9df",
"convention": "label",
"status": "verified",
"prototype": "THE techId -> NAME MAP, in eight instructions inside MasterTechTree_ctor 0x0058b870, immediately after the LoadTechFile loop. 0x0058b9df: vector::operator=(this+0x24, this+0x14) -- the sorted list at +0x24 is a COPY of the parse-order list at +0x14, which is why a parse-order dump does not match the ids. 0x0058b9ff: std::sort(first=[this+0x24], last=[this+0x28], ideal=(last-first)/4, pred) = FUN_00583b10, MSVC _Sort (the _ISORT_MAX 0x20 test and the _Ideal 3/4 halving are both there); the predicate is INLINED in FUN_00581130 at 0x00581190 and 0x005811b8 as `_stricmp(a->name, b->name) < 0` on TechDef+0x04 (a std::string, SSO-tested at +0x14 against 0x10) -- MSVCR100 _stricmp, so the order is CASE-INSENSITIVE, not byte-wise. 0x0058ba19..0x0058ba37: `for (i = 0; i < n; ++i) sortedList[i]->+0x00 = i;` -- THE WIRE techId IS LITERALLY THE 0-BASED INDEX INTO THE _stricmp-SORTED MASTER LIST. The same sorted vector is then the binary-search index: FUN_0057f120 (std::lower_bound) resolves every `requires`/`allows` name against it at 0x0058baed. NOT the 10000-based TechID enum, which is a separate 196-entry .rdata table",
"source": "findings/subsystems/techid-name-map.md"
},
{
"name": "TechTree_IsResearchable",
"addr": "0x0057e820",
"convention": "thiscall",
"prototype": "bool (TechTree* this /*per-player*/, int techId) // THE WIRE-SPACE BOUNDS CHECK, and it is what proves the space. Rejects techId < 0 and techId >= (this->+0x14 - this->+0x10)/4 -- the per-player node vector, 293 entries in stock data. Then esi = masterList[techId] from this->+0x04 -> +0x24/+0x28 (the SORTED vector, MasterTechTree_SortAndNumber) and edx = this->+0x10[techId] (the player's node). Reads the name as a std::string at masterEntry+0x04 with the SSO test at +0x18 -- so TechDef is {+0x00 int techId, +0x04 std::string name, ...}. Called by the research-target gate applier at 0x0088ff49 with the techId straight off the command payload ([block-0x28])",
"status": "verified",
"source": "findings/subsystems/techid-name-map.md"
},
{
"name": "MSVC_Sort",
"addr": "0x00583b10",
"convention": "cdecl",
"prototype": "void (T** first, T** last, int ideal, Pred pred) // MSVC std::sort's _Sort: the `(last-first)/4 <= 0x20` insertion-sort cutoff at 0x00583b26, the `ideal -= ideal/2 + ideal/4` heap-sort fallback counter, and _Unguarded_partition FUN_00581130. Identified here because MasterTechTree_ctor uses it to build the tech id space; the same body will be reached from anywhere else that sorts a pointer vector",
"status": "verified",
"source": "findings/subsystems/techid-name-map.md"
}
]
}

92
tools/techid_table.py Normal file
View file

@ -0,0 +1,92 @@
#!/usr/bin/env python3
"""techId -> tech name, derived offline from the tech tree data.
The wire techId is the 0-based index of the tech's name in the master tech list sorted with
_stricmp -- case-insensitively. Read from MasterTechTree's constructor: it copies the parse-order
list to a second vector, std::sort()s that copy with an inlined `_stricmp(a->name, b->name) < 0`,
then walks it writing `def->techId = i`. See findings/subsystems/techid-name-map.md.
This is NOT the 10000-based TechID enum, which is a separate 196-entry .rdata table.
Usage:
tools/techid_table.py # all of them
tools/techid_table.py 282 288 144 90 # just these
"""
import glob
import json
import os
import re
import sys
ROOT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..")
# Four (techId, name) pairs observed live, by two lanes on different runs. 282 is the one the
# rule was NOT fitted to -- it comes from lane L4's run R2 (findings/subsystems/ai-order-capture.md)
# and is the check that this file is right rather than merely self-consistent.
OBSERVED = {90: "DRV_PlsFiss", 144: "IND_Waldo", 282: "XNC_TrnsHum2", 288: "XNC_TrnsMorr2"}
# Where the 293 names can come from, in order of preference: the raw game file under
# $SOTS_GOB_DIR (the same variable the engine's realdata_test uses), then the checked-in parse
# oracle in sots-engine's tests, which needs no data tree at all.
SOURCES = [p for p in [
os.path.join(os.environ["SOTS_GOB_DIR"], "TechTree", "MasterTechList.tech")
if os.environ.get("SOTS_GOB_DIR") else None,
os.path.join(ROOT, "..", "sots-engine", "tests", "mars_parse", "build", "oracle",
"TechTree", "MasterTechList.tech.json"),
] if p]
def names_from_tech_file(path):
"""Pull every `tech { name "X" ... }` block name out of the raw .tech file."""
text = open(path, encoding="latin-1").read()
return re.findall(r'^\s*tech\b[^\n]*\n(?:[^\n]*\n)*?\s*name\s+"([^"]+)"', text, re.M)
def names_from_oracle(path):
return [t["name"] for t in json.load(open(path))["tech"]]
def load_names():
for pat in SOURCES:
for path in sorted(glob.glob(pat)):
if os.path.exists(path):
return (names_from_oracle(path) if path.endswith(".json")
else names_from_tech_file(path)), path
for path in sorted(glob.glob(os.path.join(ROOT, "**", "MasterTechList.tech"), recursive=True)):
return names_from_tech_file(path), path
sys.exit("no MasterTechList.tech (or its parse oracle) found; see SOURCES in this file")
def table():
names, src = load_names()
if len(set(n.lower() for n in names)) != len(names):
sys.exit("case-folded duplicate tech names: the sorted order is not total, stop")
# _stricmp order. str.lower reproduces it exactly for these pure-ASCII names.
return sorted(names, key=str.lower), src
def main(argv):
ids, src = table()
bad = [(i, want, ids[i] if i < len(ids) else "<out of range>")
for i, want in OBSERVED.items() if i >= len(ids) or ids[i] != want]
if bad:
print(f"# {len(ids)} names parsed from {src}", file=sys.stderr)
for i, want, got in bad:
print(f"MISMATCH techId {i}: observed {want!r}, derived {got!r}", file=sys.stderr)
print("The observed points are the only check this file has. Do not trust the table "
"until they agree -- a short parse (a missed block) shifts EVERY id after it.",
file=sys.stderr)
return 1
print(f"# {len(ids)} techs from {os.path.relpath(src, ROOT)}; "
f"{len(OBSERVED)}/{len(OBSERVED)} observed points agree", file=sys.stderr)
wanted = [int(a) for a in argv] if argv else range(len(ids))
for i in wanted:
if 0 <= i < len(ids):
print(f"{i:4d} {ids[i]}{' <- observed' if i in OBSERVED else ''}")
else:
print(f"{i:4d} <out of range 0..{len(ids) - 1}>")
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))