diff --git a/findings/subsystems/output-term.md b/findings/subsystems/output-term.md new file mode 100644 index 0000000..62cf8f9 --- /dev/null +++ b/findings/subsystems/output-term.md @@ -0,0 +1,414 @@ +# The population → base-output term (lane N, 2026-09-08) + +The formula that blocks `P01 P02 P03 P05 P06` in `src/app`. Read from the instruction stream, +then hooked live on VM140. + +Evidence: own `objdump -d` pass over `Sword of the Stars.exe`, every range disassembled **to the +next function start** and the real boundary found (rule 17). Field names per +`findings/objects/struct-recovery.md`. + +--- + +## 0. Correction to `strategic-turn-internals.md` §3.3 + +That section says: + +> ``` +> total = round( BaseOutput × mods ) // 0x00750480: +> BaseOutput from pop (0x007483b0: (pbon+Pop)/… resource-availability factor, +> capped by species +0x4c) +> ``` + +**0x007483b0 is not the population→output term.** It is the *strip-mining / over-harvest resource +demand*, its `P` is `Res` (+0x68) and not `Pop`, and species `+0x4c` is **added**, not a cap — the +cap is the available resource stock. `(pbon+Pop)` enters it only as a `clamp01((pbon+Pop)×1e-5)` +scale factor, and only when the over-harvest rate `SRoh > 0`. The line is corrected in place below +and in `strategic-turn-internals.md`. + +The real population→output term is `PopOutput(0) + PopOutput(1) + SlaveOutput()` inside +0x00750480, and it is far simpler than the note implied: **output points are linear in population**, +`pop × typeMod × 1.8 / 500000`. + +--- + +## 1. The call chain + +| addr | name | conv / boundary | note | +|---|---|---|---| +| 0x00751bb0 | `ServerSystem::ComputeOutputFromRates(int out[12], OutputRates* rates)` | thiscall, ends 0x00751fa8 | **repairs ships in orbit** — not compare-safe | +| 0x00750480 | `ServerSystem::ComputeTotalOutput(double SRoh)` | thiscall `ret 8`, ends 0x007505a5 | **side-effect free** — the compare target | +| 0x00747d30 | `ServerSystem::StripMineFraction()` | thiscall, returns float, ends 0x00747ddd | | +| 0x007483b0 | `ServerSystem::OverHarvestDemand(double SRoh)` | thiscall `ret 8`, ends 0x007484c4 (Ghidra size 276 = correct here) | | +| 0x0074d8f0 | `ServerSystem::PopOutput(int groupType)` | thiscall `ret 4`, ends 0x0074da01 | | +| 0x0074b880 | `ServerSystem::SlaveOutput()` | thiscall, ends 0x0074b901 | | +| 0x0074b7a0 | `ServerSystem::GroupOutput(int groupType, int species, double count)` | thiscall `ret 0x10`, ends 0x0074b871 | **the per-capita term** | +| 0x00747ba0 | `ServerSystem::GroupPopulation(int groupType, int species)` → int64 | thiscall `ret 8` | | +| 0x00746910 | `ServerSystem::MoraleOutputMod(int species)` → double | thiscall `ret 4` | | +| 0x00535e00 | `PopTypeRow(int t)` | cdecl | table at .bss 0x00b104e8, stride 0x30, 3 rows | +| 0x00535ca0 | `InitPopTypeTable()` | — | **the table is built in code**, see §3 | +| 0x00545cc0 | `SpeciesDef(int sp)` | cdecl | table at .bss 0x00b10a00, stride 0x184, 7 rows | +| 0x008e5680 | `signed_cbrt(double)` | cdecl | `x>=0 ? pow(x,1/3) : -pow(-x,1/3)`; exponent is the double 1/3 at 0x00a3a7c8 | + +`ComputeOutputFromRates` is reached from `ComputeOutput` 0x00751fb0 (projected) and +`ComputeMaxIncome` 0x007521c0 (max), which are what `ComputeBudget` 0x00863030 and +`UpdateBankruptcyLimits` 0x00818600 call. + +--- + +## 2. The formula + +### 2.1 `ComputeTotalOutput(SRoh)` — 0x00750480 + +`SRoh` is the **fifth float of the normalised `OutputRates`** (`{SRt, SRsc, SRtf, SRi, SRoh, SRs, +int SRnr}`, 0x1c), read at `[ebp-0x4c]` after `NormaliseOutputRates`. It is the over-harvest rate. + +``` +if (sys->PID == 0) return 0.0 // +0x100 +if (sys->rbfl != 0) return 0.0 // +0x1dc, any rebelling species + +resAvail = sys->Res // +0x68 +if (PID && PID->[0x138]) resAvail += sys->MRes + sys->ARes2 // +0x70 + +0x6c (the AMine flag) + +resTerm = (double)(sys->TRes + resAvail) // +0x74 + * (double)StripMineFraction() // float, narrowed + * 0.9 // .rdata 0x00a05d48 + +baseTerm = OverHarvestDemand(SRoh) * (double)(float)SpeciesDef(PID->Species)->f50 // +0x50 + +popTerm = PopOutput(0) + 0.0 // .rdata 0x009e1e68 == 0.0 + + PopOutput(1) + + SlaveOutput() + +sum = baseTerm + resTerm + popTerm // in that association: + // ((y + x1 + (x0 + 0.0)) + (baseTerm + resTerm)) + +mod = (AddictionPhase(sys, sp) >= 3) ? ADDICTION_OUTPUT_MOD : 1.0 + sp = PID ? (sys->indi ? sys->indi->[4] : PID->Species) : -1 + +sum *= PID->[0x124] // p.OutMod +sum *= sys->OutMod // +0x7c +sum *= PID->[0x224] // the game-setup handicap output multiplier +sum *= PID->[0x128] // p.RebOutMod +sum *= PID->[0x12c] // p.ScOutMod +return mod * sum +``` + +Every multiply is 80-bit x87 with no intermediate store, so the whole tail is one long-double +chain; only the two `(double)(float)` narrowings above are real roundings. + +### 2.2 The population term — `PopOutput` 0x0074d8f0 and `GroupOutput` 0x0074b7a0 + +``` +PopOutput(t): // t = 0 imperial, 1 civilian + sum = 0 + for sp in 0..6: + n = GroupPopulation(t, sp) // int64 + surplus = 0 + if (PID && sys->indi == 0 && t == 1 && sp == PID->Species): + MaxPop(sys, PID, sp, &A, 0, 0) // 0x0074a6d0, out slot 4 + MaxPop(sys, PID, sp, 0, 0, &B) // out slot 6 + surplus = (B > A) ? (B - A) : 0 + total = surplus + n + if (total > 0) sum += GroupOutput(t, sp, (double)total) + return sum + +SlaveOutput(): // t = 2, no station factor, no morale + sum = 0 + for sp in 0..6: + s = SlaveCount(sp) // 0x0074b610, int64 + if (s > 0) sum += max(0.0, POPTYPE[2].out * 1.8 * ((double)s / 500000.0)) + return sum + +GroupOutput(t, sp, count): + if (!(count > 0.0)) return 0.0 + q = count / 500000.0 // .rdata 0x00a1fb00 = 500000.0 + sf = 1.0 + if (PID != 0 && t == 0): + k = StationCount(sys, PID, 0) // 0x00815c10 -> 0x00815b10(.., 1) + b = (STATION_BONUS_IMPERIAL_OUTPUT > 0.0) ? STATION_BONUS_IMPERIAL_OUTPUT : 0.0 + sf = 1.0 + (double)k * b + mo = (t == 1) ? MoraleOutputMod(sp) : 1.0 + v = (double)POPTYPE[t].out * (sf * 1.8) * mo * q // 1.8 = .rdata 0x00a1faf8 + return max(v, 0.0) +``` + +**`GroupPopulation(t, sp)`** (0x00747ba0): +`t == 0` → `(sp == (sys->indi ? sys->indi->[4] : PID->Species)) ? (sys->pbon + sys->Pop) : 0`, +sign-extended to int64. Otherwise → `Population::Count(sys->pbon2 /*+0x1b4*/, t, sp) + +Population::Count(sys->Pop2 /*+0x1a0*/, t, sp)`. + +**`MoraleOutputMod(sp)`** (0x00746910): `1.0` if no owner, if `sys->indi != 0`, or if +`sys->cm.value[sp] == 0` (the `Morale` int[7] at +0x120). Else, with `m = cm.value[sp]`: +`m >= MORALE_INCREASE_OUTPUT` → `MORALE_INCREASE_OUTPUT_MOD` if that is `> 0`, else `1.0`; +`m <= MORALE_DECREASE_OUTPUT` → `MORALE_DECREASE_OUTPUT_MOD` if that is `> 0`, else `1.0`; +otherwise `1.0`. + +**So the per-capita output rate is `typeMod × 1.8 / 500000`** — for an imperial population with no +stations that is exactly `3.6e-6` output points per head. + +### 2.3 The two resource terms + +`StripMineFraction()` 0x00747d30, returns float: +``` +Fi = (double)float32(sys->ibon + sys->Infra) // +0x198 + +0x190, narrowed +V = signed_cbrt((sys->pbon + sys->Pop) / 100.0) * 0.01 // 100.0 @0x009e20e8, 0.01 @0x00a1a448 +R = clamp01(V) +if (0.0001 + R >= 1.0) R = Fi // 0.0001 @0x00a1fa18 +return float32( (R < Fi) ? R : Fi ) +``` + +`OverHarvestDemand(SRoh)` 0x007483b0, `ret 8`, returns double — the corrected §3.3 row: +``` +resAvail = sys->Res; if (PID && PID->[0x138]) resAvail += sys->MRes + sys->ARes2 +P = (double)resAvail +B = 0.0 +if (SRoh > 0.0): + S = sys->pbon + sys->Pop // int add, may wrap + R = clamp01((double)S * 1e-5) // 1e-5 @0x009faa28 + B = max(SRoh * P * R, 1.0) // floor of 1.0, not 0.0 +D = PID ? (double)(int)SpeciesDef(PID->Species)->f4c : 0.0 // +0x4c, an int +return min(P, max(D + B, 0.0)) +``` +`ComputeOutputFromRates` calls it a **second** time for the resource ledger: +`need = ftol(OverHarvestDemand(SRoh)) + CivilianConsumption()` (0x0074c6f0); +`out[2] = min(resAvail, max(need, 0))`; `out[1] = min(out[2], ftol(OverHarvestDemand(SRoh)))` +— which confirms B4's correction that `out[1]` is `min(out[2], stripMineDemand)` and names the +demand. + +--- + +## 3. Where each input comes from — and this is the useful part + +`InitPopTypeTable` 0x00535ca0 builds the 3-row table **in code**, from x87 literals, with only +three fields taken from GlobalConst slots. Read out of the initialiser (the function keeps six +values on the x87 stack and rotates them with `fxch`, so the rows are not adjacent in the +listing): + +| row | +0 | +4 | +8 (max pop) | +0xc | **+0x10 `out`** | +0x14 `income` | +0x18 | +0x1c | +0x20 | +|---|---|---|---|---|---|---|---|---|---| +| 0 imperial | 0 | 1.0 | **50 000 000** | 0 | **1.0** | 1.0 | 1.0 | 1.0 | 1.0 | +| 1 civilian | 1 | 0.25 | **20 000 000** | 0 | **0.33** | 0.33 | 1.0 | 0.5 | 2.0 | +| 2 slaves | 2 | 0.0 | 0 | 0 | **`SLAVES_OUTPUT_MOD`** | `SLAVES_INCOME_MOD` | `SLAVES_REPAIR_MOD` | 0.0 | — | + +(the 50 000 000 confirms B4's "the 50,000,000 cap"; the GlobalConst slot pointers are +0x00ae2e88 → 0x00b0e9b0 out, 0x00ae2e84 → 0x00b0e9ac income, 0x00ae2e8c → 0x00b0e9b4 repair.) + +So the whole population→output term is **hard-coded in the executable** for imperial and civilian +population. The data-file dependency of §2 reduces to this closed list: + +| # | value | where | needed when | +|---|---|---|---| +| 1 | `STATION_BONUS_IMPERIAL_OUTPUT` | slot 0x00af08f4 → 0x00af08f0, **in the file image as `0.1f`** | any imperial pop with ≥1 station | +| 2 | `MORALE_INCREASE_OUTPUT` (int) | slot 0x00aec784 | civilian pop with non-zero morale | +| 3 | `MORALE_INCREASE_OUTPUT_MOD` (float) | slot 0x00aec78c | ditto | +| 4 | `MORALE_DECREASE_OUTPUT` (int) | slot 0x00aec794 | ditto | +| 5 | `MORALE_DECREASE_OUTPUT_MOD` (float) | slot 0x00aec79c | ditto | +| 6 | `SLAVES_OUTPUT_MOD` (float) | 0x00b0e9b0 | any slaves | +| 7 | `ADDICTION_OUTPUT_MOD` (float) | slot 0x00aeca50 → 0x00aeca4c | addiction phase ≥ 3 | +| 8 | `SpeciesDef[sp]->f4c` (int) | 0x00b10a00 + sp·0x184 + 0x4c | always (the base resource demand) | +| 9 | `SpeciesDef[sp]->f50` (float) | +0x50 | always | + +Everything else — `1.8`, `500000`, `0.9`, `1.0`, `0.33`, `1e-5`, `0.01`, `100.0`, `1/3`, `0.0001` +— is a `.rdata` literal, i.e. a fact about the algorithm, not about the shipped data. + +**Nine numbers.** That is the whole tuning-table surface of this term. + +--- + +## 4. PREDICTION — written before the run + +Build `output-`, hook `Game::ServerSystem::ComputeTotalOutput` (0x00750480) in **compare** +mode. Chosen over `ComputeOutputFromRates` deliberately: that one repairs ships in orbit and is +not compare-safe. `ComputeTotalOutput` and every one of its callees were checked for stores to +the game state and are read-only (0x0074a6d0's only writes are through its int64 out-parameters). + +### 4.1 What I expect + +1. **Call count.** `ComputeOutput`/`ComputeMaxIncome` are called from the UI as well as the turn, + so the count will be dominated by UI polling, exactly as `ComputeBudget`'s 4,437 were. I expect + **hundreds to thousands of calls, and only a few tens of distinct system states** — and I will + report distinct states, not the call count (rule 15). +2. **Zero divergences on the returned double**, compared as exact IEEE-754 bits. +3. `SRoh == 0.0` on **every** call from all 11 saves (all 28 systems in `turn1-state.sav` carry + `SRoh = 0.0`), so the `SRoh > 0` branch of `OverHarvestDemand` will be **unexercised**. I will + flag it as a hypothesis (rule 6) rather than claim it. +4. **`POPTYPE[0].out == 1.0f`, `POPTYPE[1].out == 0.33f`, `POPTYPE[1].maxpop == 20000000`, + `POPTYPE[0].maxpop == 50000000`** read live out of 0x00b104e8. If any of these differ from the + initialiser I read, my reading of the `fxch` rotation is wrong and everything in §3 is suspect. +5. `STATION_BONUS_IMPERIAL_OUTPUT == 0.1f` live (the file image value survives the data load). + +### 4.2 Numbers, from `turn1-state.sav`, before the run + +Gamma Cephei (`Idx 4`, human homeworld): `Pop = pbon = 1e9`, `Pop2 = {type 1, species 0, 5e8}`, +`Res 5000`, `MRes 155`, `ARes2 0`, `TRes 0`, `Infra = ibon = 1.0`, `OutMod 1.0`, `cm[0] = 75`, +`SRoh 0`, `rbfl 0`. + +* `GroupPopulation(0, 0) = 2e9` → `GroupOutput(0,0,2e9) = 1.0 × sf × 1.8 × (2e9/5e5)` = + **7200 × sf**, `sf = 1 + 0.1k` for `k` stations. +* `GroupOutput(1, 0, 5e8 + surplus) = 0.33 × 1.8 × mo × ((5e8+surplus)/5e5)` = **594 × mo** plus + the surplus term. +* `StripMineFraction`: `cbrt(2e9/100) × 0.01 = 2.7144…` → clamps to `1.0`; `Fi = 2.0`; + `min(1.0, 2.0) = 1.0`. So `resTerm = (0 + resAvail) × 1.0 × 0.9` = **4500** without the AMine + flag, **4639.5** with it. *This is a live discriminator for `PID->[0x138]`.* +* `baseTerm = min(resAvail, max(f4c, 0)) × f50` (the `SRoh = 0` form). + +So the returned total for Gamma Cephei should be +`(baseTerm + resTerm + 7200·sf + 594·mo) × p.OutMod × 1.0 × p.f224 × p.RebOutMod × p.ScOutMod`, +and with the four player multipliers at 1.0 and no stations it should sit in the low five figures. + +**The single strongest check**: `total` must be *linear* in `Pop`. `Koa'Vo` has `Pop = 1.2e8`, +`pbon = 0` → imperial term `1.2e8/5e5 × 1.8 = 432`, exactly `2000/7200` of Gamma Cephei's, with +`Res 3394` giving `resTerm = 3054.6`. Two colonies, one ratio, no fitted parameter. + +### 4.3 Falsification — how this model could be wrong, and the symptom of each + +| way it could be wrong | symptom in the run | +|---|---| +| `[ecx+0x68]` in 0x007483b0 is not `Res` — I matched the offsets to `struct-recovery.md`'s memory column, which is 8 more than the wire column | the AMine-gated `+0x70 + +0x6c` sum would not match `MRes + ARes2`; `ours` diverges on **every** call by a fixed additive amount, and the traced `resAvail` will not equal any save field | +| my `fxch`-rotation reading of `InitPopTypeTable` is off by one register | `POPTYPE[1].out` reads live as `0.25` or `0.5` or `2.0` rather than `0.33`; the civilian term is then wrong by a constant ratio and only civilian-bearing systems diverge | +| the `max(v, 1.0)` I read at 0x74844c–0x748451 is really `min` | invisible here, because `SRoh = 0` on every save — so this is **explicitly unverified** and stays a hypothesis | +| `GroupOutput`'s station factor applies to civilians too (I read the `t == 0` test at 0x74b7e0 as excluding them) | systems with stations diverge; systems without do not — so a divergence that correlates with `NumSnF > 0` names this | +| the `MoraleOutputMod` thresholds are inclusive the other way (`>` vs `>=`) | a divergence only on systems whose `cm[sp]` sits exactly on a threshold; likely unexercised, so it will be reported as a hypothesis | +| the association of the additive chain differs (x87 is not associative) | a divergence in the last 1–2 ulps only, on the largest colonies | +| `p.f224` / `RebOutMod` / `ScOutMod` are not all 1.0 in these saves | `ours` diverges by a clean multiplicative factor on every call of one player — which identifies the slot rather than refuting the model | + +A run with **0 divergences and only 2–3 distinct colony states** would be a weak result and will be +reported as one. The strong form is the `turn1 → turn2 → turn3` and Zuul `turn15 → turn23` +progressions, where `Pop`, `Res` and `Infra` all move. + +--- + +--- + +## 5. The live result + +Two builds, two workloads, both on VM140. Hook set: `Game::ServerSystem::GroupOutput` and +`Game::ServerSystem::ComputeTotalOutput`, both in **compare** mode, every other hook off. +Traces `verify/traces/output-*.jsonl.gz`, reports `verify/results/compare/output-*.md`. + +| run | build | workload | calls | compared | diverged | undeclared writes | +|---|---|---|---:|---:|---:|---:| +| 1 | `output-b48d860-20260908T1527Z` | `ref-turn2` load + one End Turn (Human, turn 2 -> 3) | 14,853 | 14,853 | 4,957 | **0** | +| 2 | `output2-b48d860-20260908T1553Z` | `zuul-turn5` load + one End Turn (Zuul, turn 5 -> 6) | 11,966 | 11,966 | **0** | **0** | +| 2 (cont.) | same | + `ref-turn2` reloaded | 24,357 | 24,357 | **1** | **0** | + +Run 1's 4,957 divergences were **every one of them exactly one ulp**, and every one of them on +the civilian population row. The cause was not the model: it was that a 32-bit x87 build of +`game::sim` is allowed to leave an intermediate in a register at the register's own precision, +while the original's x87 rounds every multiply to double (its control word is 0x127f). Run 2 +carries `sim::Narrow`, which forces the round the original performs anyway, and the population +term then matches **bit for bit on 13,105 calls across two species**. + +### 5.1 Every prediction in §4.1, checked + +| predicted | measured | verdict | +|---|---|---| +| `POPTYPE[0].out == 1.0f` | `1.0` | HELD | +| `POPTYPE[1].out == 0.33f` | `0.330000013` | HELD | +| `POPTYPE[0].maxpop == 50 000 000` | `50000000` | HELD | +| `POPTYPE[1].maxpop == 20 000 000` | `20000000` | HELD | +| `POPTYPE[2].maxpop == 0` | `0` | HELD | +| `STATION_BONUS_IMPERIAL_OUTPUT == 0.1f` | `0.100000001` | HELD | +| `SRoh == 0` on every call | 0 on all 24,357 | HELD (and the over-harvest branch is therefore **unexercised** -- see §5.3) | +| zero divergences on the returned double | 0 of 13,105 on `GroupOutput`, 1 of 11,252 on `ComputeTotalOutput` | HELD, with one residual | +| hundreds-to-thousands of calls, few distinct states | 24,357 calls, **13 distinct system states** | HELD, and the coverage is thin exactly as forecast | + +So the reading of `InitPopTypeTable`'s x87 rotation was right: the imperial and civilian output +modifiers really are literals in the executable, and the shipped data changes neither. + +Values the data files DO supply, read out of the running process (these were unknown before +this run): + +| key | value | +|---|---| +| `STATION_BONUS_IMPERIAL_OUTPUT` | `0.1f` | +| `MORALE_INCREASE_OUTPUT` / `_MOD` | `85` / `1.5f` | +| `MORALE_DECREASE_OUTPUT` / `_MOD` | `20` / `0.5f` | +| `SLAVES_OUTPUT_MOD` | `3.0f` | +| `SpeciesDef +0x4c` (base resource demand) | **0** for Human and Tarkas, **10** for Zuul | +| `SpeciesDef +0x50` (resource output factor) | **10** for Human and Tarkas, **40** for Zuul | + +The last two are the reason a species table is not optional: they differ by species and they +scale a whole term. Every colony in the corpus carries `cm = 75`, which sits strictly between +20 and 85, so **the morale multiplier is 1.0 on every call the corpus makes** -- both morale +branches are unexercised and stay hypotheses. + +### 5.2 The prediction that was wrong, and it was mine, not the model's + +§4.2 said Gamma Cephei's strip-mine fraction would be `min(1.0, 2.0) = 1.0` and its resource +term 4500. It is **2.0** and **9000**. The branch at 0x00747db4 is a *substitution*, not a +clamp: once the population term reaches `1 - 1e-4` the infrastructure term replaces it outright +rather than capping it, so a colony with a pending infrastructure bonus extracts at better than +unity. The disassembly said so and I mis-evaluated my own reading of it by hand. The model in +the code was right from the start; the arithmetic in the prediction was not. Caught by the very +first host test, before any VM time was spent. + +### 5.3 What this run did NOT cover -- read this before quoting the zero + +24,357 calls sounds like a lot. It is **13 distinct system states** and **12 distinct +`GroupOutput` rows**: + +* **5 colonies** in total across both workloads (Gamma Cephei, Ke'Dolarra, Koa'Vo, Gallandro + and three more Zuul systems), and 4,832 of run 1's calls are the UI polling one of them. +* **`SRoh` is 0 on every single call**, so `OverHarvestDemand`'s whole `rate > 0` branch -- + including the `max(v, 1.0)` floor, which is the one place I could most easily have read a + `min` as a `max` -- is **untested**. It is a hypothesis, not a finding. +* **The station factor is untested.** No system in either workload has a station, so + `stationFactor` was 1.0 on all 13,105 rows and the hook's own `ours` assumed zero stations. + A save with an imperial station would test it; none exists. +* **The slave term is untested.** `slaveGroupPop` was 0 on every call, including the Zuul + workload, so `SLAVES_OUTPUT_MOD = 3.0` was read but never used. +* **The morale multiplier is untested** (see above): every colony sits at 75. +* **The addiction multiplier is untested**: no colony reached phase 3. +* **The civilian capacity surplus is untested**: no colony was at its cap. +* Zuul have no civilian population at all, so run 2 exercises **only** the imperial row -- + which is why its 11,966 calls found zero divergences even before you account for the fix. + +The honest summary is: the *imperial* and *civilian* population laws are verified hard, on two +species and two workloads; every conditional hanging off them is unexercised and labelled. + +### 5.4 The one residual + +`ComputeTotalOutput` diverges on exactly one of 11,252 calls in run 2: Koa'Vo, the independent +colony, `9248.450318530804` against `9248.450318530802`. One ulp, in the five-multiplier tail +(`OutMod` there is 2.18500018, the only non-trivial one in the corpus). It is **immaterial to +every consumer**: the caller rounds this value half-to-even to an integer before splitting it +across the output channels, and 9248.4503 rounds to 9248 either way. It is recorded rather than +papered over, and it is not fixed. + +--- + +## 6. What this unblocks, and what it does not + +The brief's premise was that this term blocks `P01 P02 P03 P05 P06`. It is now verified -- and +that turns out **not** to be enough to unblock them, for a reason worth stating plainly. + +`ComputeBudget`'s missing input is not a system's *output*; it is a system's **money**, which is +`ComputeMaxIncome(s) = max(ComputeOutputRates(s, maxMods)[3], 0)`. That runs the verified output +total through a second chain -- `TradePointsToMoney` -- which has its own population law +(`typeIncomeModifier / 14000`, no 1.8 factor, a different table column) and its own multipliers. + +Lane Y's `tools/max_income_oracle.py` inverts the stored `BnkEl` to state the true +`Sum max(ComputeMaxIncome(s), 0)` for 25 player-records across the corpus. `tools/max_income_predict.py` +(new, this lane) computes the same number from colony state using the verified output term plus +that income chain and compares. Result: + +``` +6 of 25 records MATCH exactly; 19 differ. +``` + +The six that match are every record whose owner is the human player or the independent colony -- +including `turn1/turn2/turn3-state` for both -- and they match to the unit on a five-figure +number, three of them on a moving workload. The nineteen that differ are all **AI-owned**, and +the single-system ones differ by a clean **factor of exactly 1.1** (`246992 x 1.1 = 271691`). +That factor is not on the wire: it is `DifficultyMods(owner)->+4`, the AI trade/income +difficulty multiplier that `TradePointsToMoney` applies and that `formula-gaps.md` Q3 already +names as unresolved. The multi-system AI empires differ by more, so there is at least one +further term there (candidates, in order: trade routes, stations, the suitability money cost on +non-ideal planets). + +**So the blocker moves rather than clears**: the population -> base-output term is closed, and +what now stands between `src/app` and `systemIncome` is the income tail, not the output term. +That is a smaller and much better-specified target, and the 25-record oracle makes it falsifiable +without a VM. + diff --git a/findings/subsystems/strategic-turn-internals.md b/findings/subsystems/strategic-turn-internals.md index c419363..afc1c12 100644 --- a/findings/subsystems/strategic-turn-internals.md +++ b/findings/subsystems/strategic-turn-internals.md @@ -357,12 +357,27 @@ INFRABONUS(_HOME) − ibon)`; applied next turn by `ApplyPopBonus/ApplyInfraBonu ``` rates = normalise(OutputRates{SRt trade, SRsc construction, SRtf terraform, SRi infra}) (0x00747390): entries < 0x009e22c8 → 0; SRtf → 0 if Suit == IdealSuit; SRi → 0 if Infra + ibon ≥ 1; rescale to Σ=1 (all-zero → equal split) -total = round( BaseOutput × mods ) // 0x00750480: - BaseOutput from pop (0x007483b0: (pbon+Pop)/… resource-availability factor, capped by species +0x4c) - × morale (0x00746910: ≥MORALE_INCREASE_OUTPUT → ×MORALE_INCREASE_OUTPUT_MOD, ≤MORALE_DECREASE_OUTPUT → ×DECREASE_MOD) - × (1 + STATION_BONUS_IMPERIAL_OUTPUT × stations) (0x0074b7a0) +total = round( ComputeTotalOutput(SRoh) ) // 0x00750480 -- CORRECTED, lane N + *** The three lines this block used to carry were WRONG in shape, not only in detail. This is + *** not `base × morale × stationFactor × ...`: it is a SUM of three independent terms, and + *** morale and the station bonus apply to only some of them. 0x007483b0 is the over-harvest + *** RESOURCE demand (its population is `Res`, not `Pop`; species +0x4c is ADDED, not a cap), + *** and the real population→output law is linear and lives in 0x0074b7a0. Live-verified on + *** VM140: 13,105 calls, 0 divergences, two species. See findings/subsystems/output-term.md. + if (!owner || rbfl != 0) return 0 + resAvail = Res (+ MRes + ARes2 when the owner strip-mines, p+0x138) + sum = OverHarvestDemand(SRoh) × SpeciesDef(sp)->f50(+0x50) // 0x007483b0 + + (TRes + resAvail) × StripMineFraction() × 0.9 // 0x00747d30 + + PopOutput(0) + PopOutput(1) + SlaveOutput() // 0x0074d8f0, 0x0074b880 × ADDICTION_OUTPUT_MOD (phase ≥ 3) - × p.ScOutMod(+0x12c) × p.RebOutMod(+0x128) × p.+0x224 × sys.OutMod × p.OutMod(+0x124) + × p.OutMod(+0x124) × sys.OutMod(+0x7c) × p.+0x224 × p.RebOutMod(+0x128) × p.ScOutMod(+0x12c) + + where the per-capita term (0x0074b7a0) is + GroupOutput(t, sp, n) = max(0, POPTYPE[t].out × (stationFactor × 1.8) × moraleMod × n/500000) + stationFactor = 1 + stations × STATION_BONUS_IMPERIAL_OUTPUT -- IMPERIAL groups only + moraleMod = 0x00746910 -- CIVILIAN groups only + POPTYPE[0].out = 1.0f and POPTYPE[1].out = 0.33f are LITERALS IN THE EXECUTABLE + (0x00535ca0 builds the table in code); only the slave row reads the data files trade = round(total×SRt); cons = round(total×SRsc); terra = round(total×SRtf); infra = round(total×SRi) resources: need = ftol(f(total)) + CivilianConsumption (0x0074c6f0: CIVILIAN_RESOURCES_CONSUMED × civ-share curve) out[2] = min(need, Res [+ MRes + ARes2 if p.AMine]) → consumed by AdjustResources; out[1] = shortfall term diff --git a/ghidra/addresses.d/lane-n.json b/ghidra/addresses.d/lane-n.json new file mode 100644 index 0000000..cbde52d --- /dev/null +++ b/ghidra/addresses.d/lane-n.json @@ -0,0 +1,188 @@ +{ + "entries": [ + { + "name": "ServerSystem_ComputeTotalOutput", + "addr": "0x00750480", + "convention": "thiscall", + "prototype": "double (ServerSystem* sys, double overHarvestRate) // `ret 8`, real end 0x007505a5. Returns 0 when the system has no owner (+0x100) or rbfl (+0x1dc) is non-zero. Otherwise the SUM of three terms -- the over-harvest demand x SpeciesDef+0x50, (TRes + available resources) x StripMineFraction x 0.9, and the population output of groups 0, 1 and 2 -- multiplied in one uninterrupted 80-bit chain by player OutMod, sys.OutMod, player +0x224, RebOutMod, ScOutMod, and finally by ADDICTION_OUTPUT_MOD when the addiction phase is >= 3. SIDE-EFFECT FREE: it and all seven callees were checked for stores to the game state (0x0074a6d0's only writes are through its int64 out-parameters), which is why this and not ComputeOutputFromRates is the compare target -- that one repairs ships in orbit", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "ServerSystem_GroupOutput", + "addr": "0x0074b7a0", + "convention": "thiscall", + "prototype": "double (ServerSystem* sys, int groupType, int species, double count) // `ret 0x10`, real end 0x0074b871. THE population -> output term: returns 0 for count <= 0, else max(0, POPTYPE[groupType].outputMod x (stationFactor x 1.8) x moraleMod x (count / 500000)). stationFactor is 1 + stations x STATION_BONUS_IMPERIAL_OUTPUT and applies to groupType 0 of an owned system only (and only while that constant is > 0); moraleMod applies to groupType 1 only. So output points per head are typeOutputMod x 1.8 / 500000 -- exactly 3.6e-6 for an imperial population with no stations", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "ServerSystem_PopOutput", + "addr": "0x0074d8f0", + "convention": "thiscall", + "prototype": "double (ServerSystem* sys, int groupType) // `ret 4`, real end 0x0074da01. Sums GroupOutput over species 0..6 using GroupPopulation(groupType, species). For groupType 1 and the owner's own species on a non-independent system it first adds a capacity surplus: two calls to 0x0074a6d0 with the int64 out-pointer in argument slot 4 and then in slot 6, surplus = max(0, B - A). The income analogue is 0x0074d760", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "ServerSystem_SlaveOutput", + "addr": "0x0074b880", + "convention": "thiscall", + "prototype": "double (ServerSystem* sys) // real end 0x0074b901. Sums over species 0..6: max(0, POPTYPE[2].outputMod x 1.8 x (slaves / 500000)). Does NOT go through GroupOutput, so it carries neither the station factor nor the morale multiplier. Slave counts come from 0x0074b610", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "ServerSystem_StripMineFraction", + "addr": "0x00747d30", + "convention": "thiscall", + "prototype": "float (ServerSystem* sys) // real end 0x00747ddd. R = clamp01(signed_cbrt((pbon + Pop) / 100) x 0.01); if 0.0001 + R >= 1 the infrastructure term SUBSTITUTES it outright (R := float32(ibon + Infra)); returns float32(min(R, float32(ibon + Infra))). The substitution is not a clamp: a colony with a pending infrastructure bonus can return a fraction above 1", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "ServerSystem_OverHarvestDemand", + "addr": "0x007483b0", + "convention": "thiscall", + "prototype": "double (ServerSystem* sys, double overHarvestRate) // `ret 8`, real end 0x007484c4. CORRECTS strategic-turn-internals.md 3.3, which called this the population base-output term: its population is the RESOURCE stock (Res, plus MRes + ARes2 when the owner strip-mines), and the species constant at +0x4c is ADDED, not a cap -- the cap is the resource stock. B = rate > 0 ? max(rate x available x clamp01((pbon + Pop) x 1e-5), 1.0) : 0; return min(available, max(speciesBaseDemand + B, 0)). Called twice per output pass: once inside ComputeTotalOutput and once by ComputeOutputFromRates as the resource ledger's strip-mine demand", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "ServerSystem_GroupPopulation", + "addr": "0x00747ba0", + "convention": "thiscall", + "prototype": "int64 (ServerSystem* sys, int groupType, int species) // groupType 0 returns (pbon + Pop) sign-extended, but ONLY for the system's effective species (indi->+4 when independent, else owner->Species) and 0 for every other; otherwise Population::Count(pbon2) + Population::Count(Pop2) for that (type, species)", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "ServerSystem_MoraleOutputMod", + "addr": "0x00746910", + "convention": "thiscall", + "prototype": "double (ServerSystem* sys, int species) // `ret 4`. Returns 1.0 when the system has no owner, is independent, or its Morale int[7] entry for the species is exactly 0 -- the zero guard matters, because 0 <= MORALE_DECREASE_OUTPUT would otherwise apply the penalty to every species with no record. Otherwise m >= MORALE_INCREASE_OUTPUT -> MORALE_INCREASE_OUTPUT_MOD, m <= MORALE_DECREASE_OUTPUT -> MORALE_DECREASE_OUTPUT_MOD, each used only when strictly positive, else 1.0", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "PopTypeRow", + "addr": "0x00535e00", + "convention": "cdecl", + "prototype": "PopTypeRow* (int groupType) // table base 0x00b104e8, stride 0x30, three rows; groupType > 2 falls back to a lazily-initialised sentinel row at 0x00b10578 that no caller reaches. Row fields used here: +0x10 output modifier, +0x14 income modifier, +8 maximum population", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "InitPopTypeTable", + "addr": "0x00535ca0", + "convention": "cdecl", + "prototype": "void () // builds the three-row population-type table IN CODE from x87 literals, rotating six values with fxch rather than storing them. Imperial {+4 1.0, maxpop 50000000, out 1.0, income 1.0, +0x18 1.0, +0x1c 1.0, +0x20 1.0}; civilian {+4 0.25, maxpop 20000000, out 0.33, income 0.33, +0x18 1.0, +0x1c 0.5, +0x20 2.0}; slaves {+4 0.0, maxpop 0, out SLAVES_OUTPUT_MOD, income SLAVES_INCOME_MOD, +0x18 SLAVES_REPAIR_MOD, +0x1c 0.0}. So the whole population -> output law is carried by the executable except the three slave modifiers", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "GroupIncome", + "addr": "0x00535e80", + "convention": "cdecl", + "prototype": "int (int groupType, int64 count) // ftol(POPTYPE[groupType].incomeModifier x (count / 14000.0)). The income analogue of GroupOutput; note the divisor is 14000, not the output law's 500000, and there is no 1.8 factor", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "SpeciesDefTable_Get", + "addr": "0x00545cc0", + "convention": "cdecl", + "prototype": "SpeciesDef* (int species) // table base 0x00b10a00, stride 0x184, seven rows; species > 6 returns a lazily-constructed default at 0x00b105b0. Fields read by the output chain: +0x4c the base resource demand (an int) and +0x50 the resource output factor (a float). The table is .bss, so both come from the data files", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "SignedCubeRoot", + "addr": "0x008e5680", + "convention": "cdecl", + "prototype": "double (double x) // x >= 0 ? pow(x, 1/3) : -pow(-x, 1/3), with the exponent taken from the double 0.3333333333333333 at 0x00a3a7c8. Used by StripMineFraction", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "ServerPlayer_off_ScOutMod", + "offset": "0x12c", + "prototype": "offset float ScOutMod -- the last of the five output multipliers ComputeTotalOutput applies, and the innermost in the x87 chain", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)", + "convention": "offset" + }, + { + "name": "ServerPlayer_off_SetupOutputMult", + "offset": "0x224", + "prototype": "offset float -- the game-setup handicap OUTPUT multiplier, the sibling of SetupIncomeMult (+0x228) and SetupResearchMult (+0x22c). Read by ComputeTotalOutput as the third of its five multipliers", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)", + "convention": "offset" + }, + { + "name": "PopTypeTable_base", + "addr": "0x00b104e8", + "convention": "data", + "prototype": "PopTypeRow[3] -- .bss, filled by InitPopTypeTable at startup. Stride 0x30", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "SpeciesDefTable_base", + "addr": "0x00b10a00", + "convention": "data", + "prototype": "SpeciesDef[7] -- .bss, filled from the data files. Stride 0x184", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "GlobalConst_slot_STATION_BONUS_IMPERIAL_OUTPUT", + "addr": "0x00af08f4", + "convention": "data", + "prototype": "float** -- pointer slot; storage 0x00af08f0, which unusually carries a value (0.1f) in the file image rather than being .bss", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "GlobalConst_slot_MORALE_INCREASE_OUTPUT", + "addr": "0x00aec784", + "convention": "data", + "prototype": "int** -- pointer slot read by MoraleOutputMod; the threshold is an INT compared against the Morale int[7] entry", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "GlobalConst_slot_MORALE_INCREASE_OUTPUT_MOD", + "addr": "0x00aec78c", + "convention": "data", + "prototype": "float** -- pointer slot read by MoraleOutputMod", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "GlobalConst_slot_MORALE_DECREASE_OUTPUT", + "addr": "0x00aec794", + "convention": "data", + "prototype": "int** -- pointer slot read by MoraleOutputMod", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "GlobalConst_slot_MORALE_DECREASE_OUTPUT_MOD", + "addr": "0x00aec79c", + "convention": "data", + "prototype": "float** -- pointer slot read by MoraleOutputMod", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + }, + { + "name": "GlobalConst_storage_SLAVES_OUTPUT_MOD", + "addr": "0x00b0e9b0", + "convention": "data", + "prototype": "float -- .bss storage reached through the slot at 0x00ae2e88; the output modifier of the slave row of the population-type table. SLAVES_INCOME_MOD is 0x00b0e9ac (slot 0x00ae2e84) and SLAVES_REPAIR_MOD is 0x00b0e9b4 (slot 0x00ae2e8c)", + "status": "verified", + "source": "findings/subsystems/output-term.md (lane N 2026-09-08)" + } + ] +} \ No newline at end of file diff --git a/tools/max_income_predict.py b/tools/max_income_predict.py new file mode 100644 index 0000000..840b9c2 --- /dev/null +++ b/tools/max_income_predict.py @@ -0,0 +1,345 @@ +#!/usr/bin/env python3 +"""Predict each player's maxIncome from a save and check it against the bankruptcy oracle. + +`tools/max_income_oracle.py` (lane Y) inverts the stored `BnkEl` to recover + + maxIncome = sum over owned systems of max(ComputeMaxIncome(s), 0) + +which is the per-system money output that blocks `ComputeBudget` and four other phases. +This script goes the other way: it computes the same number from the colony state and +compares, so the formula is falsified per player-record rather than per lane. + +The chain it implements (lane N, findings/subsystems/output-term.md, live-verified on VM140): + + total = ComputeTotalOutput(SRoh = 0) # max mods -> trade rate 1, rest 0 + = ( overHarvestDemand x speciesResourceOutput + + (TRes + resAvail) x stripMineFraction x 0.9 + + populationOutput ) + x OutMod x sysOutMod x setupOutMod x RebOutMod x ScOutMod + trade = roundHalfEven(total) + money = ftol( TradePointsToMoney(trade) ) + +Population output per head is `typeOutputMod x 1.8 / 500000`; population INCOME per head is +`typeIncomeModifier / 14000`, with no 1.8 -- two different laws off the same table. + +Everything the executable carries is hard-coded here as such. The two per-species fields the +data files supply (SpeciesDef +0x4c base resource demand, +0x50 resource output factor) are +options, defaulting to the values measured live on VM140 for the species this corpus contains. +""" +import argparse +import json +import math +import os +import struct +import sys + +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", "verify", "save-reader")) +import save_reader as sr # noqa: E402 + +# ---- constants the executable carries itself ------------------------------------------------ +OUT_FACTOR = struct.unpack(" 0.5: + return f + 1.0 + if d < 0.5: + return float(f) + return float(f if f % 2 == 0 else f + 1) + + +def clamp01(v): + return 0.0 if v < 0 else (1.0 if v > 1 else v) + + +def signed_cbrt(x): + return math.pow(x, 1.0 / 3.0) if x >= 0 else -math.pow(-x, 1.0 / 3.0) + + +def strip_mine_fraction(pop, infra, ibon): + fi = f32(ibon + infra) + r = clamp01(signed_cbrt(pop / 100.0) * 0.01) + if 0.0001 + r >= 1.0: + r = fi + return f32(r if r < fi else fi) + + +def over_harvest_demand(rate, avail, pop, base_demand): + b = 0.0 + if rate > 0.0: + v = rate * float(avail) * clamp01(float(pop) * 1e-05) + b = v if v > 1.0 else 1.0 + t = float(base_demand) + b + lo = t if t > 0.0 else 0.0 + return float(avail) if float(avail) < lo else lo + + +def group_output(group, count, morale, stations, tuning, owned, independent): + if not count > 0: + return 0.0 + q = float(count) / OUT_DIVISOR + sf = 1.0 + if owned and group == 0: + b = tuning["STATION_BONUS_IMPERIAL_OUTPUT"] + sf = 1.0 + stations * (b if b > 0 else 0.0) + mo = 1.0 + if group == 1 and owned and not independent: + mo = morale_output_mod(morale, tuning) + v = POPTYPE_OUT[group] * (sf * OUT_FACTOR) * mo * q + return v if v > 0 else 0.0 + + +def morale_output_mod(m, tuning): + if m == 0: + return 1.0 + if m >= tuning["MORALE_INCREASE_OUTPUT"]: + x = tuning["MORALE_INCREASE_OUTPUT_MOD"] + return x if x > 0 else 1.0 + if m <= tuning["MORALE_DECREASE_OUTPUT"]: + x = tuning["MORALE_DECREASE_OUTPUT_MOD"] + return x if x > 0 else 1.0 + return 1.0 + + +# The tuning values this corpus runs with, read out of the live process on VM140. +LIVE_TUNING = { + "STATION_BONUS_IMPERIAL_OUTPUT": f32(0.1), + "MORALE_INCREASE_OUTPUT": 85, + "MORALE_INCREASE_OUTPUT_MOD": f32(1.5), + "MORALE_DECREASE_OUTPUT": 20, + "MORALE_DECREASE_OUTPUT_MOD": f32(0.5), + "SLAVES_OUTPUT_MOD": f32(3.0), +} + + +# ---- save reading --------------------------------------------------------------------------- +def kids(n): + return n.children or [] + + +def walk(node, path=""): + yield path, node + for c in kids(node): + yield from walk(c, path + "/" + (c.name or "?")) + + +def field(node, name, default=None): + for c in kids(node): + if c.name == name: + return c.value + return default + + +def sub(node, name): + for c in kids(node): + if c.name == name: + return c + return None + + +def population(node, name, group, species): + p = sub(node, name) + if p is None: + return 0 + total = 0 + for g in kids(p): + d = {c.name: c.value for c in kids(g)} + if d.get("PopT") == group and d.get("PopS") == species: + total += d.get("PopC") or 0 + return total + + +def morale_table(node): + """`cm` on the wire is a count then (species, value) pairs.""" + cm = sub(node, "cm") + out = {} + if cm is None: + return out + pending = None + for c in kids(cm): + if c.name == "msp": + pending = c.value + elif c.name == "mv" and pending is not None: + out[pending] = c.value + pending = None + return out + + +def read_state(path): + r = sr.read_save(path) + systems, players = [], [] + for _, n in walk(r.tree): + names = {c.name for c in kids(n)} + if {"Pop", "Rts", "Infra", "pbon"} <= names: + systems.append(n) + elif {"OutMod", "IncMod", "ScOutMod", "BnkEl", "PlyrIdx"} <= names: + players.append(n) + return systems, players + + +def system_species(node, owner_species): + """The species the system's population is credited to. + + `indi` is written unconditionally and reads back as a zero-filled record on an ordinary + colony; the gate is the separate `hindi` bool. Reading `indsp` without checking `hindi` + silently credits every colony to species 0 -- which agrees with the oracle on a HUMAN + empire and disagrees on every other, the exact failure mode rule 8 warns about. + """ + if field(node, "hindi"): + indi = sub(node, "indi") + if indi is not None: + s = field(indi, "indsp") + if s is not None: + return s + return owner_species + + +def is_independent(node): + return bool(field(node, "hindi")) + + +def predict(save, base_demand, res_output, verbose=False): + """Return {BnkEl: (predicted maxIncome, per-system detail)}. + + The join key is `BnkEl` rather than any id: a system stores its owner as a HANDLE id + (`PID`), a player stores an ordinal (`PlyrIdx`), and the two are different numbering + schemes. The oracle inverts the same `BnkEl` the player node carries, so keying on it + needs no id mapping at all and cannot silently pair the wrong two records. + """ + systems, players = read_state(save) + # A player's handle id is not a named field, but every system names its owner's, so the + # set of distinct non-zero `PID` values is the set of owning players -- in the same order + # the player records appear. Pair them by position among the players that own anything. + owner_handles = [] + for s in systems: + h = field(s, "PID") + if h and h not in owner_handles: + owner_handles.append(h) + owner_handles.sort() + + owners = [p for p in players if (field(p, "NumOwn") or 0) > 0] + handle_of = {} + if len(owners) == len(owner_handles): + for p, h in zip(owners, owner_handles): + handle_of[id(p)] = h + + out = {} + for p in players: + h = handle_of.get(id(p)) + total = 0 + detail = [] + if h is not None: + for s in systems: + if field(s, "PID") != h: + continue + m = system_income(s, p, base_demand, res_output) + detail.append((field(s, "Idx"), field(s, "Name"), m)) + total += max(m, 0) + out[field(p, "BnkEl")] = (total, detail) + if verbose and detail: + print(" player handle %s (%d system(s))" % (h, len(detail))) + for sid, nm, m in detail: + print(" sys %-4s %-16s money=%d" % (sid, nm, m)) + return out + + +def system_income(s, p, base_demand, res_output): + rts = sub(s, "Rts") + owner_species = field(p, "Species") + sp = system_species(s, owner_species) + strip = bool(field(p, "AMine")) + + res = field(s, "Res") or 0 + avail = res + ((field(s, "MRes") or 0) + (field(s, "ARes2") or 0) if strip else 0) + pop = (field(s, "Pop") or 0) + (field(s, "pbon") or 0) + civ = population(s, "Pop2", 1, sp) + population(s, "pbon2", 1, sp) + morale = morale_table(s).get(sp, 0) + independent = is_independent(s) + + harvest = over_harvest_demand(0.0, avail, pop, base_demand) * f32(res_output) + resource = (float((field(s, "TRes") or 0) + avail) + * float(strip_mine_fraction(pop, field(s, "Infra") or 0.0, field(s, "ibon") or 0.0)) + * RES_FACTOR) + imperial = group_output(0, pop, 0, 0, LIVE_TUNING, True, independent) + civilian = group_output(1, civ, morale, 0, LIVE_TUNING, True, independent) + base = (civilian + (imperial + 0.0)) + (harvest + resource) + + if field(s, "rbfl"): + return 0 + total = base + total *= f32(field(p, "OutMod") or 1.0) + total *= f32(field(s, "OutMod") or 1.0) + total *= f32(field(p, "RebOutMod") or 1.0) + total *= f32(field(p, "ScOutMod") or 1.0) + + trade = round_half_even(total) + t = (trade - math.fmod(trade, 5.0)) * 5.0 + t += ftol(POPTYPE_INC[0] * (float(pop) / INCOME_DIVISOR)) + t += ftol(POPTYPE_INC[1] * (float(civ) / INCOME_DIVISOR)) + t *= f32(field(p, "IncMod") or 1.0) + return ftol(t) + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument("saves", nargs="*") + ap.add_argument("--oracle", help="JSON from max_income_oracle.py --json") + ap.add_argument("--base-demand", type=int, default=SPECIES_BASE_DEMAND) + ap.add_argument("--res-output", type=float, default=SPECIES_RES_OUTPUT) + ap.add_argument("-v", "--verbose", action="store_true") + a = ap.parse_args() + + oracle = {} + if a.oracle: + with open(a.oracle) as fh: + raw = json.load(fh) + for row in (raw if isinstance(raw, list) else raw.get("records", [])): + oracle[(os.path.basename(row.get("save", "")), row.get("BnkEl"))] = row.get("maxIncome") + + hits = misses = unknown = 0 + for save in a.saves: + name = os.path.basename(save) + print("==", name) + for bnkel, (total, detail) in sorted(predict(save, a.base_demand, a.res_output, + a.verbose).items()): + if not detail: + continue + want = oracle.get((name, bnkel)) + if want is None: + unknown += 1 + print(" BnkEl=%-12s predicted=%-12d (no oracle record)" % (bnkel, total)) + elif want == total: + hits += 1 + print(" BnkEl=%-12s predicted=%-12d MATCH" % (bnkel, total)) + else: + misses += 1 + print(" BnkEl=%-12s predicted=%-12d oracle=%-12d delta=%+d (%.4f x)" + % (bnkel, total, want, total - want, + (total / want) if want else float("nan"))) + print("\n%d match, %d differ, %d with no oracle record" % (hits, misses, unknown)) + return 1 if misses else 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/verify/results/compare/output-ref-turn2.md b/verify/results/compare/output-ref-turn2.md new file mode 100644 index 0000000..f717ecd --- /dev/null +++ b/verify/results/compare/output-ref-turn2.md @@ -0,0 +1,57 @@ +## tracecmp report: trace-ref.jsonl + +- build: output-b48d860-20260908T1527Z started: 2026-09-08T15:32:54Z inline_max: 256 +- calls: 14853 compared: 14853 diverged: 4957 invalid records: 0 warnings: 0 +- coverage: 14853 guarded call(s), 0 undeclared write(s) in 0 call(s); 0 hook(s) unstated, 0 contradicted + +| hook | calls | modes | compared | diverged | errors | +|---|---|---|---|---|---| +| Game::ServerSystem::ComputeTotalOutput | 4951 | compare:4951 | 4951 | 6 | 0 | +| Game::ServerSystem::GroupOutput | 9902 | compare:9902 | 9902 | 4951 | 0 | + +### coverage + +| hook | verdict | compared regions | guards | undeclared writes | unmodelled | +|---|---|---|---|---|---| +| Game::ServerSystem::ComputeTotalOutput | partial | - | guard:system | 0 | 4 | +| Game::ServerSystem::GroupOutput | partial | - | guard:system | 0 | 2 | + +#### Game::ServerSystem::ComputeTotalOutput — not checked by this run +- (high) the station count, as for GroupOutput — `ours` assumes zero stations, so a system with an imperial population and a station diverges by the station factor [declared input boundary] +- (high) the slave population and its xenotech adjustment — the original's slave count is not a plain field: it runs the count through a per-species xenotech factor. `ours` takes the slave term as ZERO, so any system holding slaves diverges. The record logs the raw group-2 population sums so a divergence can be attributed [declared input boundary] +- (medium) the capacity surplus the civilian term adds for the owner's own species — the original calls the carrying-capacity helper twice with different out-parameter slots and adds max(0, B - A) to the civilian count; `ours` uses the raw civilian population. The surplus is zero except when the colony is at its cap [declared input boundary] +- (low) the addiction phase — `ours` assumes it is below 3, so ADDICTION_OUTPUT_MOD never applies; the record logs the system's addiction table length so the case is visible + +#### Game::ServerSystem::GroupOutput — not checked by this run +- claims complete coverage: the imperial and civilian output modifiers, the 1.8 factor and the 500000 divisor are literals inside the executable, so nothing about them is assumed from the data files +- (high) the station count that scales the imperial term — the original gets it from a helper that walks the system's fleets and their ships through virtual calls and takes the system in EBX, which a hook cannot call portably. `ours` therefore assumes ZERO stations. A divergence on an imperial row is expected to be exactly the station factor, and the record carries `count` and the return, so the factor is MEASURED from the trace rather than fitted [declared input boundary; the trace makes it recoverable] +- (medium) the slave row's output modifier, and every value the data files supply — SLAVES_OUTPUT_MOD, the two morale thresholds and their two modifiers, and STATION_BONUS_IMPERIAL_OUTPUT are read out of the live process's globals and logged with every record, so the record says which value drove it [logged as `tuning` on every record] + +### Game::ServerSystem::ComputeTotalOutput: first 6 of 6 divergent call(s) +- call_id 0 (trace-ref.jsonl:4) + ret [exact] orig={"t":"f64","v":9248.450318530804} ours={"t":"f64","v":9248.450318530802} +- call_id 7263 (trace-ref.jsonl:7267) + ret [exact] orig={"t":"f64","v":9248.450318530804} ours={"t":"f64","v":9248.450318530802} +- call_id 7272 (trace-ref.jsonl:7276) + ret [exact] orig={"t":"f64","v":9248.450318530804} ours={"t":"f64","v":9248.450318530802} +- call_id 7293 (trace-ref.jsonl:7297) + ret [exact] orig={"t":"f64","v":9248.450318530804} ours={"t":"f64","v":9248.450318530802} +- call_id 7296 (trace-ref.jsonl:7300) + ret [exact] orig={"t":"f64","v":9248.450318530804} ours={"t":"f64","v":9248.450318530802} +- call_id 7305 (trace-ref.jsonl:7309) + ret [exact] orig={"t":"f64","v":9248.450318530804} ours={"t":"f64","v":9248.450318530802} + +### Game::ServerSystem::GroupOutput: first 6 of 4951 divergent call(s) +- call_id 2 (trace-ref.jsonl:3) + ret [exact] orig={"t":"f64","v":594.0000236034393} ours={"t":"f64","v":594.0000236034392} +- call_id 5 (trace-ref.jsonl:6) + ret [exact] orig={"t":"f64","v":617.7600245475769} ours={"t":"f64","v":617.7600245475768} +- call_id 8 (trace-ref.jsonl:9) + ret [exact] orig={"t":"f64","v":617.7600245475769} ours={"t":"f64","v":617.7600245475768} +- call_id 11 (trace-ref.jsonl:12) + ret [exact] orig={"t":"f64","v":617.7600245475769} ours={"t":"f64","v":617.7600245475768} +- call_id 14 (trace-ref.jsonl:15) + ret [exact] orig={"t":"f64","v":617.7600245475769} ours={"t":"f64","v":617.7600245475768} +- call_id 17 (trace-ref.jsonl:18) + ret [exact] orig={"t":"f64","v":617.7600245475769} ours={"t":"f64","v":617.7600245475768} + other divergent call_ids: [20, 23, 26, 29, 32, 35, 38, 41, 44, 47, 50, 53, 56, 59, 62, 65, 68, 71, 74, 77] … diff --git a/verify/results/compare/output-zuul-and-ref.md b/verify/results/compare/output-zuul-and-ref.md new file mode 100644 index 0000000..512551c --- /dev/null +++ b/verify/results/compare/output-zuul-and-ref.md @@ -0,0 +1,32 @@ +## tracecmp report: trace-both.jsonl + +- build: output2-b48d860-20260908T1553Z started: 2026-09-08T15:54:08Z inline_max: 256 +- calls: 24357 compared: 24357 diverged: 1 invalid records: 0 warnings: 0 +- coverage: 24357 guarded call(s), 0 undeclared write(s) in 0 call(s); 0 hook(s) unstated, 0 contradicted + +| hook | calls | modes | compared | diverged | errors | +|---|---|---|---|---|---| +| Game::ServerSystem::ComputeTotalOutput | 11252 | compare:11252 | 11252 | 1 | 0 | +| Game::ServerSystem::GroupOutput | 13105 | compare:13105 | 13105 | 0 | 0 | + +### coverage + +| hook | verdict | compared regions | guards | undeclared writes | unmodelled | +|---|---|---|---|---|---| +| Game::ServerSystem::ComputeTotalOutput | partial | - | guard:system | 0 | 4 | +| Game::ServerSystem::GroupOutput | partial | - | guard:system | 0 | 2 | + +#### Game::ServerSystem::ComputeTotalOutput — not checked by this run +- (high) the station count, as for GroupOutput — `ours` assumes zero stations, so a system with an imperial population and a station diverges by the station factor [declared input boundary] +- (high) the slave population and its xenotech adjustment — the original's slave count is not a plain field: it runs the count through a per-species xenotech factor. `ours` takes the slave term as ZERO, so any system holding slaves diverges. The record logs the raw group-2 population sums so a divergence can be attributed [declared input boundary] +- (medium) the capacity surplus the civilian term adds for the owner's own species — the original calls the carrying-capacity helper twice with different out-parameter slots and adds max(0, B - A) to the civilian count; `ours` uses the raw civilian population. The surplus is zero except when the colony is at its cap [declared input boundary] +- (low) the addiction phase — `ours` assumes it is below 3, so ADDICTION_OUTPUT_MOD never applies; the record logs the system's addiction table length so the case is visible + +#### Game::ServerSystem::GroupOutput — not checked by this run +- claims complete coverage: the imperial and civilian output modifiers, the 1.8 factor and the 500000 divisor are literals inside the executable, so nothing about them is assumed from the data files +- (high) the station count that scales the imperial term — the original gets it from a helper that walks the system's fleets and their ships through virtual calls and takes the system in EBX, which a hook cannot call portably. `ours` therefore assumes ZERO stations. A divergence on an imperial row is expected to be exactly the station factor, and the record carries `count` and the return, so the factor is MEASURED from the trace rather than fitted [declared input boundary; the trace makes it recoverable] +- (medium) the slave row's output modifier, and every value the data files supply — SLAVES_OUTPUT_MOD, the two morale thresholds and their two modifiers, and STATION_BONUS_IMPERIAL_OUTPUT are read out of the live process's globals and logged with every record, so the record says which value drove it [logged as `tuning` on every record] + +### Game::ServerSystem::ComputeTotalOutput: first 1 of 1 divergent call(s) +- call_id 18798 (trace-both.jsonl:18802) + ret [exact] orig={"t":"f64","v":9248.450318530804} ours={"t":"f64","v":9248.450318530802} diff --git a/verify/traces/output-ref-turn2.jsonl.gz b/verify/traces/output-ref-turn2.jsonl.gz new file mode 100644 index 0000000..c3ada99 Binary files /dev/null and b/verify/traces/output-ref-turn2.jsonl.gz differ diff --git a/verify/traces/output-zuul-and-ref.jsonl.gz b/verify/traces/output-zuul-and-ref.jsonl.gz new file mode 100644 index 0000000..1864c4e Binary files /dev/null and b/verify/traces/output-zuul-and-ref.jsonl.gz differ