alex/sots-engine
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

b2 live: 3 completions compared 0 divergences; float32 confirmed on the game; RollResearchEvent RNG region added; xenotech block size bug fixed

Browse source
This commit is contained in:
alex 2026-09-08 01:18:39 -04:00
parent 646e4e8bb2
commit bbba536d99
6 changed files with 182 additions and 57 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

126
docs/B2.md
View file

@ -1,11 +1,20 @@
# B2 — `ServerPlayer::OnTechResearched` old-vs-new, with the player's fields as named regions
**Status (2026-09-08): code complete, cross-built and staged; every VM step still owed.**
VM140 was held by another lane for the whole of this milestone, so nothing was deployed,
the game was not stopped or relaunched, and `C:\SOTS` was not touched. The build lives in
its own tree (`/srv/re-lab/build/sots-engine-b2`) and its own dist
(`/srv/re-lab/shim/dist-b2`), not the shared ones. Everything below is offline work plus
what the *binary* says; the run list is at the end.
**Result (2026-09-08): verified on the live game.** Trace: 2 completions captured, `tracecmp.py`
exit 0, 0 invalid. Compare: **3 completions compared, 0 divergences, 0 errors.** Replace: one
End Turn from `ref-turn2.sav` reproduces the determinism oracle byte for byte
(`(Autosave).sav` = `978041acd168b56e…`, `(Autosave EndTurn).sav` = `bb4fd9ac89f41e3b…`) — a weak
check by construction, because no tech completes on that turn so the hook never fires; what it
proves is that the hook installs and perturbs nothing. Game left on VM140 at the main menu in
`hooks=trace`.
Build `b2-646e4e8-dirty-20260908T0434Z`, staged in `/srv/re-lab/shim/dist-b2`, deployed from
`C:\SOTS\shimdist-b2`. Evidence in `sots-re/verify/traces/b2-*` and
`verify/results/compare/b2-techfx-golden.json`.
All three runs are archived: `sots-re/verify/traces/b2-techfx-golden.jsonl`,
`b2-techfx-compare.jsonl`, `b2-shim.log`, `b2-final-menu.png`, with the `tracecmp.py` reports in
`verify/results/compare/b2-techfx-{golden,compare}.json`.
The point of the target: it is the single place where 36 hard-coded strategic effects, the
per-species xenotech flag words, the two design-option masks and the node-bore parameters
@ -318,46 +327,71 @@ plan, in order of what it proves:
into a synthetic buffer at that offset (it would spill over `IncMod` at +0x30c), so they
set the view's block pointer directly.
## What remains (needs the VM)
## Runs
The lane holding VM140 must be finished first; then, in this order:
| file | mode | records | result |
|---|---|---|---|
| `b2-techfx-golden.jsonl` | trace | 2 OnTechResearched (+18 ProcessResearch from the scout config) | `tracecmp.py` exit 0, 0 invalid |
| `b2-techfx-compare.jsonl` | compare | 3 OnTechResearched | 3 compared, **0 diverged**, 0 errors, exit 0 |
| replace (no log) | replace | 0 (no completion that turn) | End Turn from `ref-turn2.sav`: `(Autosave).sav` `978041ac…` / `(Autosave EndTurn).sav` `bb4fd9ac…` = oracle |
| `b2-shim.log` | | | install banners and the three `techfx: ours mode=compare` lines |
1. Deploy `/srv/re-lab/shim/dist-b2` (build `b2-9cd997d-dirty-20260908T0359Z`): `scp` it to
`C:\SOTS\shimdist-b2\` and run `deploy.ps1 -Dist C:\SOTS\shimdist-b2` — **a separate
staging directory from the shared `C:\SOTS\shimdist`**, so no other lane's dist is
overwritten.
2. **Scout.** Copy `shim.cfg.b2scout` over `C:\SOTS\shim.cfg`, relaunch, load
`ref-turn2.sav`, press End Turn once, pull `C:\SOTS\shim.trace.jsonl`. Read off it:
* how many `Game::ServerPlayer::OnTechResearched` records there are (expect 0 or 1);
* from the `Game::TechTree::ProcessResearch` records, each player's funded node and its
`progress` vs `cost_rp`, i.e. how many more turns to a completion;
* `fpu_cw` (expect `0x027f`; `0x007f`/`0x003f` means 24-bit precision, which would change
the rounding shape in `AddF`/`MulF` and nothing else).
3. **Golden trace.** Copy `shim.cfg.b2trace`, relaunch, load `ref-turn2.sav`, and press End
Turn as many times as step 2 says are needed. Pull the trace →
`verify/traces/b2-techfx-golden.jsonl`; `tracecmp.py` must exit 0 with 0 invalid records
**and at least one record**. Check per record: `tech_id` is in 10000..10195 and
`tech_name` matches; `def_node_index` differs from `tech_id` (finding 8); the `side`
entries change only the fields that tech's branch should touch, plus the tail's
`design_masks` / `species_flags` / `translation` / `node_bore`.
4. **Compare.** Copy `shim.cfg.b2compare`, relaunch, load `ref-turn2.sav`, same number of
End Turns → `b2-techfx-compare.jsonl`. Expect **0 divergences on every record**. There is
no expected-divergence carve-out here: everything `ours` does not model is outside the
declared regions. Any diff is a real finding — report it, do not tune the table. Read the
matching `techfx: ours ...` lines out of `C:\SOTS\shim.log` and check them against the
completing tech (granted tech only for a Zuul `IND_CruisCon`, plague mask only for a
vaccine, and so on).
5. **Replace (weak check only).** Copy `shim.cfg.b2replace`, relaunch, load `ref-turn2.sav`,
press End Turn **once** (a turn on which nothing completes), and check the determinism
oracle — `(Autosave).sav` = `978041ac…`, `(Autosave EndTurn).sav` = `bb4fd9ac…`. That
proves the hook installs and perturbs nothing. Do **not** expect the oracle to hold on a
turn where a tech completes; see gotcha 4.
6. Restore the previous `shim.cfg` (`hooks=trace`) and leave the game at the main menu, as
M1/M2/B3 leave it.
The completions that fired, all on the AI player (index 1, species 2 = Tarkas), driven by six to
eight End Turns from `ref-turn2.sav`: **`IND_Waldo` (10001)**, **`BIO_GnMod` (10010)** and
**`BIO_SPNDANI` (10070)**. The third is the interesting one: it is not in the effects table at
all, so only the tail ran — which is exactly the case finding 10 predicted and the case an early
return would have broken.
Not done, and worth saying: no record of this hook has ever been captured, so unlike M2
there is not even a scouting trace to confirm the region model against a live object. The
first thing to check in step 2 is that the `before` snapshots look like plausible player
state (`out_mod`/`pop_mod`/`terra_mod` near 1, `con_mod`/`sav_mod` near 1,
`per_gate_traffic` a small integer) — if a field reads as garbage, the offset is wrong and
the run list stops there.
### What the live trace confirms
* **The float32 model, exactly.** `IND_Waldo` moved `con_mod` `1.0 → 0.899999976` and `out_mod`
`1.10000002 → 1.25`; `BIO_GnMod` moved `pop_mod` `1.10000002 → 1.20000005`. Those are bit for
bit `(float)(x + (double)0.1f)` / `(double)0.15f`, not the exact decimals. Finding 1 is now
confirmed against the game and not only against the image.
* **`TechDef`'s first word is not the TechId** (finding 8): the records carry both, and they are
`144` vs `10001` and `9` vs `10010`.
* **`fpu_cw` = `0x127f`** — 53-bit precision control, matching B3, so the rounding shape in
`AddF`/`MulF` is the right one.
* **The gate-traffic config words are 3 and 6 at call time.** The hook's init banner prints
`gate=0/0` because the data files have not been read when the DLL loads; the values are
dereferenced per call, and the args show the live ones.
* **The RNG region resolves and is consistent**: `left` 413 with `next_index` 211, and
`624 − 413 = 211`, which validates both the two-hop pointer chain to the generator and the
describer's index arithmetic.
* The two unmodelled words at `+0x128`/`+0x12c` hold `1.0f`, and `+0x13c` holds `0.2f`. None of
the unmodelled words moved on any call.
### The RNG draw B3 handed over
B3 saw one completion where the original drew one word more than its implementation and
attributed it to this callback. That is right, and it is the roll at the very top:
`ServerPlayer::RollResearchEvent` (`+0x3b4` set and the completing def is the current research
target) makes **exactly one `NextFloat`** from the strategic generator, unconditionally, before
any comparison — then fires the AI-rebellion/plague path only if `odds > roll`. `ours` reproduces
that draw against the scratch copy of the generator (B3's design), so the post-state is
comparable, and the whole `Mars::RNG` object is now a declared region.
**On this workload the branch never fired**: `research_roll_pending` was `false` on all three
completions, so the callback drew nothing and the `rng` region is unchanged before and after.
That means the compare could not exercise the draw — it is modelled and host-reasoned, not
behaviourally proven. The region is declared so the next run that does set the byte will check it.
### Where the compare is blind
Per B3's harness lesson: `ours` posts no events, and the player's event list is **not** a declared
region (its offset was not located in this window). Both non-silent completions raise
`EVENT_RESEARCH_COMPLETE` or `_UNDERBUDGET`, and a temperance completion would raise
`EVENT_TEMPERANCE`. So a clean compare here bounds the player's economy fields and nothing else,
and **replace mode is not oracle-clean on a turn where a tech completes** — the same failure mode
B3 hit, known in advance rather than discovered by the oracle. The replace run above deliberately
used a turn with no completion.
## What remains
1. **Locate the player's event list and declare it as a region**, then either post the events
through the game's own API in `ours` (as B3 concluded for its own event) or accept a named,
expected divergence there. Until then the compare's scope is the economy fields.
2. **Exercise the roll.** Find a turn where `+0x3b4` is set at completion (the record's
`research_roll_pending` before-value says when) and confirm the `rng` post-state matches.
3. The Zuul branches — `IND_CruisCon` granting boarding pods, and the temperance sweep — need a
Zuul player; the reference save has none.

10
include/generated/sots_addresses.h
View file

@ -1,5 +1,5 @@
// GENERATED — do not edit. Facts about Sword of the Stars.exe (GOG 1.8.1).
// Source: sots-re ghidra/addresses.json @ 8a6c0e8, generated 2026-09-08 by tools/gen_addresses.py
// Source: sots-re ghidra/addresses.json @ 535d1c3, generated 2026-09-08 by tools/gen_addresses.py
// Runtime address = (uintptr_t)GetModuleHandle(NULL) + RVA (the exe is ASLR-relocated).
#pragma once
#include <cstdint>
@ -429,5 +429,13 @@ constexpr uint32_t ServerPlayer_SpeciesOfTranslationTech = 0x0040e410;
constexpr uint32_t AITechRow = 0x00290f70;
// data int -- number of rows in g_AITechValueTable (6) [verified]
constexpr uint32_t g_AITechValueCount = 0x006ea2ec;
// offset void* -- the StrategyServer subobject pointer; StrategyServer = *(void**)(player+8) - 4. Its +0x16c is the Mars::RNG* the strategic sim draws from, and its +8 the event manager [verified]
constexpr uint32_t ServerPlayer_off_ServerLink = 0x00000008;
// offset Mars::RNG* -- the strategic generator (same object TechTree::ProcessResearch is handed) [verified]
constexpr uint32_t StrategyServer_off_RNG = 0x0000016c;
// fastcall void (ServerPlayer* this) -- odds = ResearchEventOdds(this, this->ResT); draws EXACTLY ONE NextFloat from StrategyServer's generator, unconditionally, then fires 0x00889d60 when odds > roll. Called from OnTechResearched when the completing def is the current research target and the pending-roll byte at +0x3b4 is set. This is the extra RNG draw B3 observed on a completion [verified]
constexpr uint32_t ServerPlayer_RollResearchEvent = 0x0048df20;
// thiscall float (ServerPlayer* this, TechDef* def) -- 0 when def is null; a plague-family path via 0x00535480, else the AI-rebellion path via AITechRow (odds column) gated on !NPC. Read-only, makes no draw [verified]
constexpr uint32_t ServerPlayer_ResearchEventOdds = 0x00420380;
} // namespace sots::addr

38
src/shim/hooks/tech_effect_fields.cpp
View file

@ -173,6 +173,25 @@ Tv describe_roll(const void* p, std::size_t, unsigned) {
return s;
}
// The strategic generator, in the shape B3 pinned: the untempered block is hashed (it is
// far larger than inline_max), and equality of that hash plus `left` is exactly "the two
// generators are at the same place in the same stream".
const void* g_live_rng = nullptr;
Tv describe_rng(const void* p, std::size_t, unsigned inline_max) {
Tv s = tv::struct_();
s.add("vptr", tv::ptr(ptr_at(p, 0)));
s.add("mt", tv::bytes(static_cast<const char*>(p) + A::RNG_off_State,
A::RNG_off_Next - A::RNG_off_State, inline_max));
s.add("left", tv::i32(i32_at(p, A::RNG_off_Left)));
const std::uintptr_t next = reinterpret_cast<std::uintptr_t>(ptr_at(p, A::RNG_off_Next));
const std::uintptr_t base = reinterpret_cast<std::uintptr_t>(g_live_rng) + A::RNG_off_State;
std::int64_t index = -1;
if (g_live_rng && next >= base) index = static_cast<std::int64_t>((next - base) / 4);
s.add("next_index", tv::i64(index));
return s;
}
Tv describe_nodebore(const void* p, std::size_t, unsigned) {
Tv s = tv::struct_();
std::vector<Tv> v;
@ -198,19 +217,32 @@ const RegionDef kRegions[kRegionCount] = {
{"species_flags", A::ServerPlayer_off_SpeciesTechFlags, 0x20, &describe_species_flags},
{"roll", A::ServerPlayer_off_ResearchRollPending, 8, &describe_roll},
{"node_bore", 0, 12, &describe_nodebore}, // base is *(player + off_NodeBore)
{"rng", 0, A::RNG_size, &describe_rng}, // base is the strategic generator
};
std::size_t PlayerSpan() { return A::ServerPlayer_off_ResearchRollPending + 8; }
Views Views::OverPlayer(void* player) {
Views v;
for (int i = 0; i < kRegionCount; ++i) {
v.base[i] = i == R_NODEBORE ? ptr_at(player, A::ServerPlayer_off_NodeBore)
: static_cast<char*>(player) + kRegions[i].off;
// The generator is two pointer hops away: the player's server link, less the four bytes
// of subobject offset, then the server's generator slot.
void* link = ptr_at(player, A::ServerPlayer_off_ServerLink);
void* rng = nullptr;
if (link) {
void* server = static_cast<char*>(link) - 4;
rng = ptr_at(server, A::StrategyServer_off_RNG);
}
for (int i = 0; i < kRegionCount; ++i) {
if (i == R_NODEBORE) v.base[i] = ptr_at(player, A::ServerPlayer_off_NodeBore);
else if (i == R_RNG) v.base[i] = rng;
else v.base[i] = static_cast<char*>(player) + kRegions[i].off;
}
v.liveRng = rng;
return v;
}
void SetLiveRngBase(const void* p) { g_live_rng = p; }
// Seed our model from the pre-call field values. Everything the callback can write comes
// from the region copies; species (never written) comes from the live object.
fx::PlayerEconomyState ReadPlayerState(const Views& v_, sots::sim::Species species) {

8
src/shim/hooks/tech_effect_fields.h
View file

@ -36,6 +36,7 @@ enum RegionId {
R_SPECIES_FLAGS, // flags[7] + count
R_ROLL, // pending plague-cure roll + AIRebellion*
R_NODEBORE, // the 3-word node-bore block itself (behind R_NODEBORE_PTR)
R_RNG, // the strategic Mars::RNG object (behind player+8 -> server+0x16c)
kRegionCount
};
@ -55,6 +56,10 @@ std::size_t PlayerSpan();
// R_NODEBORE is ever legitimately null (no bore drive researched yet).
struct Views {
void* base[kRegionCount] = {};
// The generator's live address, for the describer's next-pointer arithmetic: `next` is a
// heap address, so it is reported as its index into mt, which is what it means and what
// survives being rewritten by a reimplementation.
void* liveRng = nullptr;
// Point every region at its offset inside a live (or synthetic) player object, and
// R_NODEBORE at whatever the block pointer holds.
@ -77,4 +82,7 @@ bool ClearResearchTargetIfMatched(const Views& v, const void* def);
// The two design-option masks, written straight into R_MASKS.
void WriteDesignOptionMasks(const Views& v, std::uint32_t a, std::uint32_t b);
// The describer needs the live generator address; set it before a snapshot is described.
void SetLiveRngBase(const void* p);
} // namespace shim::hooks::techfx

42
src/shim/hooks/tech_effects.cpp
View file

@ -19,6 +19,7 @@
#include "game/sim/species.h"
#include "game/sim/tuning.h"
#include "generated/sots_addresses.h"
#include "mars/rng/mt19937.h"
namespace shim::hooks {
@ -109,6 +110,7 @@ struct CallState {
bool compare = false;
int region_of[tfx::kRegionCount] = {};
tfx::Views views; // the scratch copies, once rebind() has resolved them
void* live_rng = nullptr; // the real generator, for the describer's index arithmetic
};
CallState g_call;
@ -212,6 +214,8 @@ void ServerPlayerOnTechResearchedHook::regions(std::vector<trace::Region>& out,
if (!readable(self, tfx::PlayerSpan())) throw std::runtime_error("ServerPlayer not readable");
const tfx::Views live = tfx::Views::OverPlayer(self);
g_call.live_rng = live.liveRng;
tfx::SetLiveRngBase(live.liveRng);
for (int i = 0; i < tfx::kRegionCount; ++i) {
const tfx::RegionDef& d = tfx::kRegions[i];
const void* base = live.base[i];
@ -236,6 +240,7 @@ ServerPlayerOnTechResearchedHook::Args ServerPlayerOnTechResearchedHook::rebind(
? s.ptr(static_cast<std::size_t>(g_call.region_of[i]))
: nullptr;
}
g_call.views.liveRng = g_call.live_rng;
g_call.compare = true;
// `self` is passed through unchanged: `ours` only reads never-written fields off it
// (species, the tech tree) and takes everything else from the scratch copies above.
@ -252,9 +257,38 @@ void ServerPlayerOnTechResearchedHook::ours(void* self, void* def, bool silent)
const int raw_id = resolve_tech_id(self, def);
// The pending plague-cure roll: the two words are cleared either way, but the roll
// itself draws from the strategic generator, so `ours` records it instead of running it.
// The pending research-event roll. It is the one place this callback consumes
// randomness -- exactly one NextFloat from the strategic generator, drawn before any
// comparison, whenever the completing definition is the current research target and the
// pending byte is set. In compare mode the generator is a scratch copy, so drawing here
// is safe and is what makes the post-state comparable (B3's design); in replace mode it
// is the live generator, which is what the game expects.
const bool would_roll = tfx::ClearResearchTargetIfMatched(v, def);
float roll = -1.f;
if (would_roll) {
void* rng = v.base[tfx::R_RNG];
if (!rng) throw std::runtime_error("strategic generator not reachable");
mars::rng::MT19937 gen;
std::uint32_t mt[mars::rng::MT19937::N];
std::memcpy(mt, static_cast<const char*>(rng) + A::RNG_off_State, sizeof mt);
std::int32_t left = i32_at(rng, A::RNG_off_Left);
if (left < 0 || left > static_cast<std::int32_t>(mars::rng::MT19937::N))
throw std::runtime_error("generator `left` out of range");
gen.load_state(mt, left);
roll = gen.next_float();
// Write the post-state back in the object's own layout; `next` is rebuilt against
// the LIVE base so the describer's index arithmetic reads the same on both sides.
std::uint8_t blob[mars::rng::MT19937::kStateBytes];
gen.save_state(blob);
std::memcpy(static_cast<char*>(rng) + A::RNG_off_State, blob,
A::RNG_off_Next - A::RNG_off_State);
left = gen.left();
std::memcpy(static_cast<char*>(rng) + A::RNG_off_Left, &left, sizeof left);
const std::uintptr_t base =
reinterpret_cast<std::uintptr_t>(compare ? v.liveRng : rng) + A::RNG_off_State;
void* next = reinterpret_cast<void*>(base + static_cast<std::uintptr_t>(gen.index()) * 4);
std::memcpy(static_cast<char*>(rng) + A::RNG_off_Next, &next, sizeof next);
}
int species_index = i32_at(self, A::ServerPlayer_off_Species);
if (species_index < 0 || species_index >= sots::sim::kSpeciesCount) species_index = 0;
@ -311,12 +345,12 @@ void ServerPlayerOnTechResearchedHook::ours(void* self, void* def, bool silent)
// where the un-compared consequences -- the system-side writes, the Zuul grant, the
// roll -- are visible at all.
logf("techfx: ours mode=%s id=%d granted=%d plague=0x%02x systems_ai=%d civcaps=%d "
"temperance=0x%02x bore_changed=%d bore_present=%d roll=%d",
"temperance=0x%02x bore_changed=%d bore_present=%d roll=%d value=%.9g",
compare ? "compare" : "replace", raw_id, static_cast<int>(outcome.grantedTech),
outcome.plagueCuredMask, outcome.flagSystemsAI ? 1 : 0,
outcome.reevaluateCivilianCaps ? 1 : 0, outcome.temperanceSpeciesMask,
outcome.nodeBoreParamsChanged ? 1 : 0, v.base[tfx::R_NODEBORE] ? 1 : 0,
would_roll ? 1 : 0);
would_roll ? 1 : 0, static_cast<double>(roll));
}
void init_tech_effects(std::uintptr_t exe_base, void (*log_line)(const char* line)) {

15
tests/shim_techfx/unit_tests.cpp
View file

@ -41,6 +41,8 @@ struct FakePlayer {
// host, so writing a real pointer into the buffer would spill over the neighbouring
// field. The tests point the view at the block directly instead, which is what
// Views::OverPlayer would do on the target.
// The generator is reached through two 32-bit pointer hops; the same 64-bit-host
// caveat as the node-bore block applies, so tests aim the view at it directly.
tfx::Views views(bool with_bore) {
tfx::Views v = tfx::Views::OverPlayer(bytes.data());
v.base[tfx::R_NODEBORE] = with_bore ? bore : nullptr;
@ -92,10 +94,12 @@ void test_region_table() {
CHECK(d.describe != nullptr);
CHECK(d.size > 0);
}
// Two regions live outside the player object, so their `off` is meaningless.
auto in_object = [](int i) { return i != tfx::R_NODEBORE && i != tfx::R_RNG; };
for (int i = 0; i < tfx::kRegionCount; ++i) {
if (i == tfx::R_NODEBORE) continue; // a separate allocation, not part of the object
if (!in_object(i)) continue;
for (int j = i + 1; j < tfx::kRegionCount; ++j) {
if (j == tfx::R_NODEBORE) continue;
if (!in_object(j)) continue;
const std::uint32_t a0 = tfx::kRegions[i].off, a1 = a0 + tfx::kRegions[i].size;
const std::uint32_t b0 = tfx::kRegions[j].off, b1 = b0 + tfx::kRegions[j].size;
CHECK(a1 <= b0 || b1 <= a0);
@ -103,9 +107,12 @@ void test_region_table() {
}
// The declared span must reach past the last region.
for (int i = 0; i < tfx::kRegionCount; ++i) {
if (i == tfx::R_NODEBORE) continue;
if (!in_object(i)) continue;
CHECK(tfx::kRegions[i].off + tfx::kRegions[i].size <= tfx::PlayerSpan());
}
// The generator region is the whole Mars::RNG object, and it is absent when the player
// has no server link (a synthetic object, or a client-side player).
CHECK_EQ(tfx::kRegions[tfx::R_RNG].size, static_cast<std::size_t>(A::RNG_size));
// The node-bore region is the only one that may legitimately be absent: on the target
// the block pointer is the 32-bit word at its offset, and a zero word means "no block".
FakePlayer p;
@ -113,6 +120,8 @@ void test_region_table() {
CHECK_EQ(p.get<std::uint32_t>(A::ServerPlayer_off_NodeBore), 0u);
CHECK(p.views(true).base[tfx::R_NODEBORE] == p.bore);
CHECK(p.views(false).base[tfx::R_NODEBORE] == nullptr);
CHECK(tfx::Views::OverPlayer(p.base()).base[tfx::R_RNG] == nullptr);
CHECK(tfx::Views::OverPlayer(p.base()).liveRng == nullptr);
}
void test_read_round_trip() {