diff --git a/docs/W3-watchpoints.md b/docs/W3-watchpoints.md new file mode 100644 index 0000000..509baf7 --- /dev/null +++ b/docs/W3-watchpoints.md @@ -0,0 +1,70 @@ +# W3 — a second watch mode, and what the two runs answered + +Companion to `docs/W3-predictions.md` (written and committed before the build) and to +`sots-re/findings/subsystems/nvo-tshn-visible-owner.md` + +`sots-re/findings/control-flow/w3-containers-status-handlers.md` (the full reports). + +Builds on lane W2's module without changing how it works. + +## What was added + +`watch.mode=modcount|tshn` in `src/shim/hooks/watchpoints.{h,cpp}`. **No second hook, no second +detour, no change to the VEH or to the arming mechanism** — only different arithmetic on the `S` +the existing `ApplyAllTurnCommands` detour already holds. `modcount` is W2's set and stays the +default so its run remains reproducible; `tshn` arms: + +| slot | target | +|---|---| +| 0 | the `TShn` word of an `NVO` map record on a system with `AFlags == 0` | +| 1 | that map's `_Mysize` | +| 2 | trade-route vector `_Mylast` (`*(S+4+0x154) + 0x40`) | +| 3 | spy-program vector `_Mylast` (`*(S+4+0x158) + 0x14`) | + +Two things about the `tshn` arming are worth keeping if the module is extended again: + +- **The target is chosen by predicate at arm time, not hard-coded**, and all 28 systems are logged + with name, `AFlags` and both map sizes. A hard-coded pointer that lands on the wrong object + produces the same empty hit list as "nothing writes this" (method rule 1). +- **The `ServerSystem` base is probed, not assumed.** Two published accounts disagreed by 8 about + where the `NVO` map lives, because `ServerSystem::Write` runs on the IStreamable subobject at + `+0x8`. The code tries both candidate bases across every system and logs how many validated under + each (`+0x274`: 9 systems; `+0x26c`: 0). That turned a documentation dispute into a measurement. + +Configs: `shim.cfg.w3tshn`, `shim.cfg.w3mod`, `shim.cfg.w3control`. `w3tshn` and `w3control` differ +in exactly one key (`watch=`), which is what makes the rule-19 control real. + +## Rule 19: the control was taken again, and it passed again + +One End Turn from `ref-turn2.sav` with the four `tshn` watchpoints armed reproduced the determinism +oracle byte for byte — `(Autosave EndTurn).sav` `bb4fd9ac…`, `(Autosave).sav` `978041ac…`, both +identical to the pre-run files. So the four *new* addresses are as neutral as W2's four were. That +mattered: W2 proved a data breakpoint is neutral for one address set, not for all of them, and this +lane's set includes a heap node that the game reallocates freely. + +The second run (turn-1 workload, `watch.mode=modcount`) has **no oracle of its own** and says so. + +## What the two runs answered + +1. **`NVO.TShn`'s writer, trapped live**, with its whole call chain confirmed frame by frame from + the recorded `ebp` chain: `RefreshVisibleOwnerIfKnown 0x0075bd70` → `RecordVisibleOwner + 0x0075bca0` → `SetVisibleOwner 0x0075b880` → `NVO::operator[] 0x0075a890` → the store + `0x0075b961`. The gate is `ServerSystem::IsKnownTo 0x00746390`, which is `IsVisibleTo` **or** + (sensor contact ∧ `CCC_AdvSens`) — not `AFlags` alone, which is why lane E3 could not fit it. + The resulting model predicts all 158 `NVO` records in the 11-save corpus with zero mismatches, + including the single frozen one. +2. **Four traps, not one, and the extra pair is a result**: the refresh runs twice per End Turn, in + driver phase 24 and again in combat-done phase 25. Both phase attributions confirmed for free. +3. **The trade and spy containers, read out of a live game for the first time.** Both managers + non-null, both vectors default-constructed with all three pointers zero, zero traps on either + `_Mylast` across the turn. The workload is *absent*, measured — which is the confirmation + instrument two earlier lanes needed and did not have. +4. **All ten command `ModCount` handlers are now named**, including two that only exist on a turn-1 + workload (`OnCommand_SetResearchProject`, inlined at `0x0088fe0a`, and + `OnCommand_CreateDesign 0x00882910`). + +## Reusing it + +`watch.mode` is the extension point. To watch something else, add a mode and one arming function; +everything else — the canary, the VEH, the flusher, the hit format — is unchanged. Keep the canary +self-test and keep the `ref-turn2` oracle control: together they cost about two minutes and they +are what makes the numbers evidence rather than output. diff --git a/src/shim/hooks/watchpoints.cpp b/src/shim/hooks/watchpoints.cpp index 7bd4b9d..9300c9e 100644 --- a/src/shim/hooks/watchpoints.cpp +++ b/src/shim/hooks/watchpoints.cpp @@ -19,16 +19,19 @@ bool g_enabled = false; int g_players = 2; char g_outPath[MAX_PATH] = {}; +// Which four addresses the single arming point computes. `modcount` is lane W2's set and is the +// default so its run stays reproducible byte for byte; `tshn` is lane W3's. +enum class Mode { ModCount, Tshn }; +Mode g_mode = Mode::ModCount; + // ---- what is being watched ------------------------------------------------------------------ // -// Slot 0 and 1 are the two words the naming disagreement is about; slots 2 and 3 are the first two -// players' Status. Four slots is the hardware limit and it is exactly enough. -const char* const kSlotName[4] = { - "S+0x8 (A2:ModCount / T:PhaseCounter)", - "S+0xc (A2:Frame / addresses.json:ModCount)", - "player[0]+0x164 Status", - "player[1]+0x164 Status", -}; +// Filled in by the arming function, because in `tshn` mode two of the four slots are only nameable +// once the target object has been found. Printed with the address, so a reader of the log never +// has to trust that slot N is what the brief said it would be. +char g_slotName[4][96] = {}; + +void SetSlotName(int i, const char* s) { std::snprintf(g_slotName[i], sizeof g_slotName[i], "%s", s); } // ---- hit records ----------------------------------------------------------------------------- // @@ -180,6 +183,206 @@ void LogF(const char* fmt, ...) { bool g_armed = false; +// ---- lane W3: reaching the NVO map, the trade vector and the spy vector ----------------------- +// +// All three hang off the same `S` the arming detour already has, so this adds no second hook. +// +// The offsets below were re-derived from the instruction stream of `ServerSystem::Write` +// 0x00749630 rather than taken from a table, because two published accounts disagreed by 8. The +// reason for the disagreement: `Write` is entered on the **IStreamable subobject at +// ServerSystem+0x8** (the RTTI COL offset for its vftable is +0x8), so every displacement in its +// body is 8 less than the ServerSystem-relative offset. `[edi+0x26c]`/`[edi+0x270]` at +// 0x0074a195/0x0074a07d are therefore `NVO._Myhead`/`_Mysize` at ServerSystem **+0x274/+0x278**. +// +// This code does not *rely* on that reconciliation. It probes both candidate bases on every +// system, counts how many validate under each, and logs both counts -- rule 1: an arm that landed +// on the wrong object is indistinguishable from "nothing writes this". + +constexpr std::uint32_t kSysNvoHead = 0x274; // ServerSystem-relative +constexpr std::uint32_t kSysAFlags = 0xd4; +constexpr std::uint32_t kSysName = 0xa8; // std::string, 0x1c bytes +constexpr std::uint32_t kSysNveSize = 0x288; +constexpr std::uint32_t kNodeIsNil = 0x8d; // an NVO node is ~0x90 bytes; NVE's is 0x20 +constexpr std::uint32_t kNodeValue = 0x10; // {int16 touched, int16 TShn, int32 OID, ...} + +inline bool Readable(std::uintptr_t p, std::size_t n) { + return p != 0 && !IsBadReadPtr(reinterpret_cast(p), n); +} +inline std::uint32_t U32(std::uintptr_t p) { return *reinterpret_cast(p); } +inline std::uint8_t U8(std::uintptr_t p) { return *reinterpret_cast(p); } + +struct NvoProbe { + bool ok = false; + std::uintptr_t head = 0; + std::uint32_t size = 0; + std::uintptr_t root = 0; // == _Myhead->_Parent + std::uint32_t key = 0; // the root node's key: a player INDEX (Write reads players[key]) +}; + +// Validate `sys + headOff` as an MSVC `_Tree` header {_Myhead, _Mysize}. The head node is the nil +// sentinel, so its _Isnil byte must be 1 and the root it parents must have _Isnil 0. +NvoProbe ProbeNvo(std::uintptr_t sys, std::uint32_t headOff) { + NvoProbe r; + const std::uintptr_t hp = sys + headOff; + if (!Readable(hp, 8)) return r; + r.head = U32(hp); + r.size = U32(hp + 4); + if (!Readable(r.head, 0x90)) return r; + if (U8(r.head + kNodeIsNil) != 1) return r; + if (r.size == 0 || r.size > 64) return r; + const std::uintptr_t root = U32(r.head + 4); + if (!Readable(root, 0x90)) return r; + if (U8(root + kNodeIsNil) != 0) return r; + const std::uint32_t key = U32(root + 0xc); + if (key > 63) return r; // player index, not a handle + r.root = root; + r.key = key; + r.ok = true; + return r; +} + +// MSVC std::string (0x1c): union _Bx at +0, _Mysize +0x10, _Myres +0x14; short strings live in the +// union. Copies at most `cap-1` bytes and always NUL-terminates. +void ReadStdString(std::uintptr_t s, char* out, std::size_t cap) { + out[0] = '\0'; + if (!Readable(s, 0x18)) return; + const std::uint32_t len = U32(s + 0x10); + const std::uint32_t res = U32(s + 0x14); + const std::uintptr_t p = (res < 16) ? s : static_cast(U32(s)); + if (len == 0 || len > 0x100 || !Readable(p, len)) return; + std::size_t n = len < cap - 1 ? len : cap - 1; + for (std::size_t i = 0; i < n; ++i) { + const char c = static_cast(U8(p + i)); + out[i] = (c >= 32 && static_cast(c) < 127) ? c : '?'; + } + out[n] = '\0'; +} + +// A std::vector here is {_Myfirst,_Mylast,_Myend,_Alval} = 0x10, allocator LAST (method rule 5). +// Returns the element count and reports the raw triple, because "the offset names something else" +// and "the container is empty" have to be separable from the log alone. +int VectorCount(std::uintptr_t vec, std::uint32_t* first, std::uint32_t* last, std::uint32_t* end, + std::uint32_t stride) { + *first = *last = *end = 0; + if (!Readable(vec, 12)) return -1; + *first = U32(vec); + *last = U32(vec + 4); + *end = U32(vec + 8); + if (*last < *first || (*last - *first) % stride) return -1; + return static_cast((*last - *first) / stride); +} + +// The systems vector in the S+4 frame. Lane E3 decoded tail phase 17 as `mov edx,[esi+0x44]` = +// systems.begin with esi = S, i.e. (S+4)+0x40 -- the same frame `StrategyServer_off_Players` +// (0x50) is expressed in. Recorded in ghidra/addresses.d/lane-w3.json; kept local rather than +// pulled from the generated header because a concurrent lane owns that header this session. +constexpr std::uint32_t kServerOffSystems = 0x40; + +// The two containers lane W2 §8.3 located but did not arm. Both are one add from `S`. +constexpr std::uint32_t kServerOffTradeMgr = 0x154; // -> ServerTradeManagerImpl* +constexpr std::uint32_t kTradeMgrOffVec = 0x3c; // _Myfirst; _Mylast at +0x40 +constexpr std::uint32_t kServerOffSpyMgr = 0x158; // -> ServerSpyManager* +constexpr std::uint32_t kSpyMgrOffVec = 0x10; // _Myfirst; _Mylast at +0x14 + +// Lane W3's four slots. Slot 0 is the whole point: the `TShn` word of an NVO record on a system +// whose AFlags is zero. Slot 1 is its map's _Mysize, which is what makes slot 0's silence mean +// something -- if the node were freed and reallocated, slot 0 would be watching dead memory and +// would report a confident nothing (rule 20's distinction, one level down). +void ArmTshnSlots(std::uintptr_t S) { + const std::uintptr_t sysVec = S + 4 + kServerOffSystems; + std::uint32_t f = 0, l = 0, e = 0; + const int nSys = VectorCount(sysVec, &f, &l, &e, 4); + LogF("watch: systems vector @0x%08x first=0x%08x last=0x%08x end=0x%08x count=%d " + "(expect 28 on the reference game)", + static_cast(sysVec), f, l, e, nSys); + + // Does the vector hold the ServerSystem base, or its IStreamable subobject 8 bytes in? Probe + // both and let the systems themselves decide, rather than trusting either published table. + const std::uint32_t headOff[2] = {kSysNvoHead, kSysNvoHead - 8}; + int okAt[2] = {0, 0}; + for (int a = 0; a < 2; ++a) { + for (int i = 0; i < nSys && i < 256; ++i) { + const std::uintptr_t sys = U32(f + i * 4); + if (!Readable(sys, 0x300)) continue; + if (ProbeNvo(sys, headOff[a]).ok) ++okAt[a]; + } + } + const int adj = (okAt[0] >= okAt[1]) ? 0 : 1; + const std::uint32_t d = adj ? 8 : 0; + LogF("watch: NVO header probe -- ServerSystem+0x%x validated on %d systems, +0x%x on %d; " + "using +0x%x (pointer delta %u)", + kSysNvoHead, okAt[0], kSysNvoHead - 8, okAt[1], headOff[adj], d); + + std::uintptr_t target = 0; + NvoProbe tp; + char tname[40] = {}; + for (int i = 0; i < nSys && i < 256; ++i) { + const std::uintptr_t sys = U32(f + i * 4); + if (!Readable(sys, 0x300)) continue; + const NvoProbe p = ProbeNvo(sys, headOff[adj]); + const std::uint32_t af = + Readable(sys + kSysAFlags - d, 4) ? U32(sys + kSysAFlags - d) : 0xffffffffu; + const std::uint32_t nve = + Readable(sys + kSysNveSize - d, 4) ? U32(sys + kSysNveSize - d) : 0xffffffffu; + char nm[40]; + ReadStdString(sys + kSysName - d, nm, sizeof nm); + LogF("watch: sys[%d] @0x%08x '%s' AFlags=0x%x NVO=%u NVE=%u root=0x%08x key=%u ok=%d", i, + static_cast(sys), nm, af, p.size, nve, static_cast(p.root), p.key, + p.ok ? 1 : 0); + if (!target && af == 0 && p.ok) { + target = sys; + tp = p; + std::snprintf(tname, sizeof tname, "%s", nm); + } + } + + if (target) { + g_watchAddr[0] = tp.root + kNodeValue; + g_watchAddr[1] = target + kSysNvoHead + 4 - d; + std::snprintf(g_slotName[0], sizeof g_slotName[0], + "'%s' NVO root+0x10 {touched:i16,TShn:i16} key=player %u", tname, tp.key); + std::snprintf(g_slotName[1], sizeof g_slotName[1], "'%s' NVO._Mysize (=%u at arm)", tname, + tp.size); + } else { + LogF("watch: NO system has AFlags==0 with a non-empty NVO -- slots 0/1 UNSET. That is a " + "FAILED TARGET SELECTION, not a measurement; every zero below is unmeasured."); + SetSlotName(0, "(no target found)"); + SetSlotName(1, "(no target found)"); + } + + // The workload confirmation. Printing these two counts is the cheap half of the answer: two + // lanes have failed to build a trade/spy workload, and nobody has yet printed the containers + // from a live game to say whether a workload took. + const struct { + std::uint32_t mgrOff, vecOff; + const char* what; + } cont[2] = { + {kServerOffTradeMgr, kTradeMgrOffVec, "trade routes"}, + {kServerOffSpyMgr, kSpyMgrOffVec, "spy programs"}, + }; + for (int i = 0; i < 2; ++i) { + const std::uintptr_t slot = S + 4 + cont[i].mgrOff; + const std::uintptr_t mgr = Readable(slot, 4) ? U32(slot) : 0; + std::uint32_t vf = 0, vl = 0, ve = 0; + int n = -1; + if (Readable(mgr, cont[i].vecOff + 12)) + n = VectorCount(mgr + cont[i].vecOff, &vf, &vl, &ve, 4); + LogF("watch: %s -- manager=0x%08x (S+4+0x%x) vector@mgr+0x%x first=0x%08x last=0x%08x " + "end=0x%08x count=%d", + cont[i].what, static_cast(mgr), cont[i].mgrOff, cont[i].vecOff, vf, vl, ve, + n); + if (mgr) { + g_watchAddr[2 + i] = mgr + cont[i].vecOff + 4; + std::snprintf(g_slotName[2 + i], sizeof g_slotName[2 + i], + "%s vector _Mylast (mgr+0x%x, count %d at arm)", cont[i].what, + cont[i].vecOff + 4, n); + } else { + std::snprintf(g_slotName[2 + i], sizeof g_slotName[2 + i], "%s (manager null)", + cont[i].what); + } + } +} + DWORD WINAPI FlusherThread(LPVOID) { for (;;) { Sleep(2000); @@ -203,11 +406,22 @@ extern "C" void WatchOnApplyAll(void* self) { return; } const std::uintptr_t S = reinterpret_cast(self); - g_watchAddr[0] = S + 0x8; - g_watchAddr[1] = S + 0xc; + g_watchAddr[0] = 0; + g_watchAddr[1] = 0; g_watchAddr[2] = 0; g_watchAddr[3] = 0; + if (g_mode == Mode::Tshn) { + ArmTshnSlots(S); + } else { + + g_watchAddr[0] = S + 0x8; + g_watchAddr[1] = S + 0xc; + SetSlotName(0, "S+0x8 (A2:ModCount / T:PhaseCounter)"); + SetSlotName(1, "S+0xc (A2:Frame / addresses.json:ModCount)"); + SetSlotName(2, "player[0]+0x164 Status"); + SetSlotName(3, "player[1]+0x164 Status"); + // players vector lives at S+0x54 (raw +0x50 in the S+4 frame -- see the ctor enumeration in // StrategyServer_base_delta). Read defensively: a wrong pointer here must not fault. if (g_players > 0) { @@ -228,6 +442,8 @@ extern "C" void WatchOnApplyAll(void* self) { } } + } // end Mode::ModCount + // Instrument self-test: put slot 3 on a word we own, write it, and require exactly one trap // before any game number is trusted. const std::uintptr_t saved3 = g_watchAddr[3]; @@ -255,7 +471,7 @@ extern "C" void WatchOnApplyAll(void* self) { g_armedTid = GetCurrentThreadId(); g_armed = true; for (int i = 0; i < 4; ++i) - LogF("watch: slot %d -> %s = 0x%08x%s", i, kSlotName[i], + LogF("watch: slot %d -> %s = 0x%08x%s", i, g_slotName[i], static_cast(g_watchAddr[i]), g_watchAddr[i] ? "" : " (unset)"); LogF("watch: ARMED on tid %lu dr7=0x%08x, S=%p (ApplyAllTurnCommands this)", g_armedTid, dr7, self); @@ -282,6 +498,12 @@ bool watch_apply_config(const char* key, const char* value, std::string* err) { else if (err) *err = "expected on|off"; return true; } + if (std::strcmp(key, "watch.mode") == 0) { + if (std::strcmp(value, "modcount") == 0) g_mode = Mode::ModCount; + else if (std::strcmp(value, "tshn") == 0) g_mode = Mode::Tshn; + else if (err) *err = "expected modcount|tshn"; + return true; + } if (std::strcmp(key, "watch.out") == 0) { std::snprintf(g_outPath, sizeof g_outPath, "%s", value); return true; diff --git a/src/shim/hooks/watchpoints.h b/src/shim/hooks/watchpoints.h index af1d2c0..0351392 100644 --- a/src/shim/hooks/watchpoints.h +++ b/src/shim/hooks/watchpoints.h @@ -34,7 +34,16 @@ namespace shim::hooks { // shim.cfg keys owned here. Returns true if `key` was ours (whether or not the value parsed). // watch=off|on install the ApplyAllTurnCommands arming detour (default off) // watch.out= hit log (default \shim.watch.txt) -// watch.players= how many player Status words to watch (0..2, default 2) +// watch.players= how many player Status words to watch (0..2, default 2; modcount mode) +// watch.mode=modcount|tshn which four addresses the arming point computes (default modcount) +// +// `modcount` is lane W2's set: S+0x8, S+0xc and two players' Status. +// `tshn` is lane W3's: the `TShn` word of an NVO map record on a system with AFlags == 0, that +// map's _Mysize, and the `_Mylast` of the trade-route and spy-program vectors. All four are +// reachable from the same `S` the arming detour already holds, so the mode adds no second hook -- +// only different arithmetic. The target system is chosen by predicate at arm time and its name, +// AFlags and map sizes are logged, because "the arm landed on the wrong object" and "nothing +// writes this" produce the same empty hit list (method rule 1). bool watch_apply_config(const char* key, const char* value, std::string* err); bool watch_enabled(); diff --git a/src/shim/shim.cfg.w3control b/src/shim/shim.cfg.w3control new file mode 100644 index 0000000..f4b4147 --- /dev/null +++ b/src/shim/shim.cfg.w3control @@ -0,0 +1,21 @@ +# Lane W3: the same watchpoint module, pointed at the NVO/TShn record and the trade + spy +# containers. Identical to shim.cfg.w3control except for the single key `watch=`, so the pair is a +# real rule-19 control: the armed run and the control run differ in nothing else. +hooks=trace +hook.Shim::SelfTest::Fill=off +hook.Mars::GlobalConsts::LoadFile=off +hook.Game::WeaponDictionary::Init=off +hook.Game::SectionDictionary::SectionDictionary=off +hook.Game::StrategyServer::ProcessFleetMovement=off +hook.Game::TechTree::ProcessResearch=trace +hook.Game::ServerPlayer::ComputeBudget=trace +hook.Game::ServerPlayer::OnTechResearched=trace +hook.Game::ServerSystem::ProcessTurn=trace +hook.Game::StrategyServer::MoveFleet=trace +trace.path=C:\SOTS\shim.trace.jsonl +trace.inline_max=256 +trace.flush=always +watch=off +watch.players=2 +watch.mode=tshn +watch.out=C:\SOTS\shim.watch.txt diff --git a/src/shim/shim.cfg.w3mod b/src/shim/shim.cfg.w3mod new file mode 100644 index 0000000..b3cc04f --- /dev/null +++ b/src/shim/shim.cfg.w3mod @@ -0,0 +1,21 @@ +# Lane W3 run 2 (lane AI4 probe): the SAME binary as shim.cfg.w3tshn with ONE key changed -- +# watch.mode -- so the module arms lane W2's ModCount set instead. Paired with shim.cfg.w3control +# for rule 19; byte-neutrality of this build was established on ref-turn2 with mode=tshn. +hooks=trace +hook.Shim::SelfTest::Fill=off +hook.Mars::GlobalConsts::LoadFile=off +hook.Game::WeaponDictionary::Init=off +hook.Game::SectionDictionary::SectionDictionary=off +hook.Game::StrategyServer::ProcessFleetMovement=off +hook.Game::TechTree::ProcessResearch=trace +hook.Game::ServerPlayer::ComputeBudget=trace +hook.Game::ServerPlayer::OnTechResearched=trace +hook.Game::ServerSystem::ProcessTurn=trace +hook.Game::StrategyServer::MoveFleet=trace +trace.path=C:\SOTS\shim.trace.jsonl +trace.inline_max=256 +trace.flush=always +watch=on +watch.players=2 +watch.mode=modcount +watch.out=C:\SOTS\shim.watch.txt diff --git a/src/shim/shim.cfg.w3tshn b/src/shim/shim.cfg.w3tshn new file mode 100644 index 0000000..adcd0a3 --- /dev/null +++ b/src/shim/shim.cfg.w3tshn @@ -0,0 +1,21 @@ +# Lane W3: the same watchpoint module, pointed at the NVO/TShn record and the trade + spy +# containers. Identical to shim.cfg.w3control except for the single key `watch=`, so the pair is a +# real rule-19 control: the armed run and the control run differ in nothing else. +hooks=trace +hook.Shim::SelfTest::Fill=off +hook.Mars::GlobalConsts::LoadFile=off +hook.Game::WeaponDictionary::Init=off +hook.Game::SectionDictionary::SectionDictionary=off +hook.Game::StrategyServer::ProcessFleetMovement=off +hook.Game::TechTree::ProcessResearch=trace +hook.Game::ServerPlayer::ComputeBudget=trace +hook.Game::ServerPlayer::OnTechResearched=trace +hook.Game::ServerSystem::ProcessTurn=trace +hook.Game::StrategyServer::MoveFleet=trace +trace.path=C:\SOTS\shim.trace.jsonl +trace.inline_max=256 +trace.flush=always +watch=on +watch.players=2 +watch.mode=tshn +watch.out=C:\SOTS\shim.watch.txt