f965c8c769
Settles the 0x18-vs-0x1c contradiction lane X raised. 0x1c is right, everywhere, and there is exactly one std::string instantiation in this binary: _Bx@0, _Mysize@0x10, _Myres@0x14, _Alval@0x18. ObservedTech+0x24 is that string's trailing empty-allocator word, not the unaccounted data field it was read as. Three complete enumerations of the element each skip it: ObservedTech::Write 0x00817cf0, the ctor 0x008562a0, and the copy ctor inlined at 0x0079a184. Generalised with a new scanner, tools/strfootprint.py, which recovers every (base, disp, tag) handed to the Mars::Stream string helpers: 65 std::string members off a non-stack base across every serializer in the exe, ZERO with a sibling member inside the 0x1c span, and 51 of the 52 measurable inter-member gaps exactly 0x1c. Corroborated by the vector<string> walk stride (add esi,0x1c @0x00699c29), PostEvent's by-value strings at [ebp+8]/[ebp+0x24] with RET 0x4c, and MoraleEvent 0x50 = name@0x34 + 0x1c. Blast radius: zero recovered struct tables were wrong. Every string-bearing layout already used 0x1c spans and 0x1c gaps -- ServerPlayer::pswd @0x2dc..0x2f7, the row flagged for re-checking, included. Only prose carried the 0x18 number: the loader-prototypes conventions line, the GlobalConst_ParseString prototype, and the ObservedTech element table. struct-recovery S0 additionally had _Mysize/_Myres transposed (size@0x14, res@0x18) while every table in the same file used the correct offsets; fixed. ObservedTech's four on-disk fields are now mapped rather than guessed, by reading the serializer as lane X suggested: +0x04 uint16 otnF, +0x06 uint16 otnL, +0x08 bool odet (ONE BYTE, WriteBool), +0x0c std::string otch (0x1c), +0x28 int owith = 0x2c exactly. That matches save_reader.py's on-disk order already. Game:: ObservedWeapon (0x00817bc0/0x00817b10) is the same element with tag owep. Oracles unaffected and re-run: save_reader 36/36 and --strict exit 0 on all three real saves; state_checksum 38 tests OK, coverage PROVED byte-for-byte on turn1 and turn3. sots-engine wip/strings 32d3e36 syncs the header and corrects two stale "unpinned" comments: clean_room_check OK, host ctest 33/33. Standing rule this produced: never size a struct member from the offsets the code touches. This build's STL puts the empty allocator LAST in both string (0x1c) and vector (0x10), and an empty allocator is never loaded or stored, so a touch-scan undercounts by exactly 4 every time. Size from an enumeration instead.
46 lines
14 KiB
Markdown
46 lines
14 KiB
Markdown
# Open questions
|
||
|
||
Each links to the finding that raised it. Promoted to backlog or closed by **re-quartermaster**.
|
||
|
||
- **Compiler archive mismatch** — imports say MSVCR100/MSVCP100 (VS2010), but Ghidra applied
|
||
`windows_vs12_32` datatypes (closest available). Confirm the exact toolchain from the PE Rich
|
||
header; decide which datatype archive best fits. (from [[01-fingerprint]])
|
||
- **Battle-load root cause** — inferred as single-threaded engine + core-count-sensitive code;
|
||
must be confirmed by a `ForceSingleCore=0` vs `=1` diff once the main loop is mapped.
|
||
(from [[00-strategy]])
|
||
- **Save-editor struct coverage** — community editors cover players/tech/systems; fleet/combat
|
||
state coverage may be partial. Treat as strong hints, verify each field against real saves.
|
||
- **Lockstep determinism** — `@SYNC ERROR DETECTED: Frame %d` indicates a deterministic lockstep sim. Big for reimplementation + old-vs-new differential (determinism makes equivalence checkable frame-by-frame). Confirm scope. (from [[strings-and-config]])
|
||
- **Mods override mechanism** — `.gob` = `gobio::ZipFileSystem` with a `NativeFileSystem` loose-file override; this is how `./Mods` wins over archived files. Verify load order. (from [[strings-and-config]])
|
||
- **Battle-load root cause** — the shipped fix is a blunt process-affinity pin (`FUN_0089ee70`) with NO CPU-topology detection, so there is no adaptive branch. The real cause is emergent multi-core contention: which thread(s)? Investigate the 3 `CreateThread` sites (`FUN_00902350`, `FUN_00736e30`, startup `FUN_008a0e50`), the streaming-sound thread, and lockstep sync waits. (from [[ghidra-recon]])
|
||
- **Save-struct contradictions (R1 vs R2)** — field widths (`Abdn`/`Dstyd`/`ltis` Int16 vs Int32; `Bats2`), R2's `OID = PID*16` owner-handle claim, species id 4 (`_NPC` vs 'AI Rebellion'). Resolve against the binary's Streamable read code. (from [[save-editor-structs]])
|
||
- **Unlabeled save blocks** — `CdPlayer` (unknown1..35), empty `SimSystemDetailSpy`, opaque ~2500 B RNG blob. Analyst targets once the Streamable readers are located. (from [[save-editor-structs]])
|
||
- **Missing HUD scripts** — exe references `GUI/Combat/CombatHUD.script`, `SensorHUD.script`, `NoHUD.script` but none ship in the gobs or loose; likely dev-only overrides via the gobio native-FS fallback. Confirm via `CombatScreen` load path. Also: `.script` files are display configs, not widget layouts (corrects round-one note). (from [[ui-screen-map]])
|
||
- **Tech `allows` default per-race %** — `tech_tree.json` edges only carry races written in the `allows` string; the default for an unlisted race (believed 100%) is engine code. Ghidra target in the tech loader. (from [[data-parsers]])
|
||
- **Engine parser leniency** — 12 shipped shipsections are syntactically broken (unclosed `{`, extra `}`) yet load; keys and identifiers are case-insensitive. Reimplementation must match this leniency. `.effect` is its own `TXT`/`BEGIN-END` format, not brace-block (corrects round one). (from [[data-parsers]])
|
||
- **Struct recovery leftovers** — `PlayerColorID` exact on-disk width (writer `FUN_008b9cb0` undecompiled); `ServerSystem+0x10` owner type; `TechTree` per-tech body; `CdPlayer` block. R2's `OID = PID*16` is an id-allocation pattern, not in this code. (from [[struct-recovery]])
|
||
- **Resolved (R1/R2 contradictions)** — `Bats2`/`rcex` are int64 (R2 wrong); `Abdn`/`Dstyd` bools, `ltis` int; `TRM`/`CstR/E/T`/`shrm`/`RefCap`/`RepCap`/PlayerView `Infra` are floats; `pswd` string; `TShn`/`ETS`/diplomacy counters int16 in memory, int32 on disk; `Nexp` carries `xid/xmin/xmax/xper`; `FtOrig` is Vector3. (from [[struct-recovery]])
|
||
- **Battle-load, narrowed** — sim/combat load run on the main thread; the only other threads are net watchdog, TIME_CRITICAL audio streaming (`g_musicCS`), and a star-map mesh builder. Hypothesis: audio-thread critical-section contention or D3D9 runtime/driver threads on many cores. Needs a dynamic profile (x32dbg / ETW) under the software-GPU stack. (from [[turn-spine]])
|
||
- **Spine leftovers** — `StrategyServer` struct partial (41 fields); static-initialiser region 0x009be000–0x009c1400 undisassembled; `Mars::Stream` vftable not located; several small ProcessTurn phase fns unnamed. (from [[turn-spine]])
|
||
- **Save framing ambiguities (settle on first real save)** — padding joint `[len][name][value][pad]` vs split; bool vs int for names with len%4==0 (no type byte); on-disk tags for Summary/CreateParameters and count/element tags inside framed arrays unknown (positional for now). A real 3-char bool tag (`NPC`,`Dep`,`hsp`) settles padding. (from [[SAVE_FORMAT]])
|
||
- **Strict-parse gaps vs real save (verifier output, round 3)** — (1) `ServerSystem`: 7/28 systems carry `VFlags` where `Name`/`vnh` were expected — a conditional layout (uncolonized/special systems?); (2) `halt[].haltv` is a 31/35-byte struct, not a bool; (3) optional `indi` (IndependenceInfo?) precedes `NVE`/`PID` in some systems; (4) `PrisonerHold.prisoners[]` entries lack `PrNSp`; (5) `/createParams/key` is int; (6) `turnstats.hist.stats.tch` is int; (7) optional trailing `zdsi`/`zdst`. Padding = **joint** (settled). 7 resyncs / 2728 raw bytes ≈ RNG blob. (from [[turn2-strict-issues]])
|
||
- **RESOLVED: strict-parse gaps** — 5/8 were reader bugs (ASCII plausibility test applied to string values; empty string == int 0), 3 real: `PrNSp` only when `PrMax>0`; `indi` unconditional in `NVO` nodes (system-level gated by `hindi`); `zdsi/zdst` pairs under `zdsc`. `Key` is an empty string; `tch` int. Summary/CreateParams tag lists confirmed. Reader patch in flight. (from [[schema-gaps-resolved]])
|
||
- **RESOLVED: save framing** — padding is joint; `"."` = NULL tag (VectorHelper count+elements, Vector3 bodies); empty strings are 4 zero bytes; `CDT` is a tagged frame + opaque `CD` frames. All three real saves parse `--strict` clean. Note: `SvSctOb` IS present in all saves (patch doc §7 was wrong). (from [[SAVE_FORMAT]])
|
||
- **(parked) Combat loader job API** — does tactical-battle loading go through the job helpers `0x8fc160`/`0x8fa5b0` with the `TryEnterCriticalSection→Sleep(1000)` completion poll (`0x0071ea60`)? If yes, that is a plausible many-core stall (lost try-lock = +1 s per poll). End Turn also has a deliberate `Sleep(computed)` ≈0.85 s at `0x00838da9`. (from [[battle-load-profile]])
|
||
- **Ship-design code rules (Ghidra, from SHIP_DESIGN_RULES.md §8)** — hidden default rider designs are built WITHOUT tech gating (where?); is section class-equality enforced or merely unoffered; confirm the turret class-accept table (`standard ← {standard,missile,grapple}`, `strafe ← {standard}`; grapple weakest); hull-class tech injection (`IND_CruisCon`/`IND_DreadCon`) + station exemption; option defaults/auto-upgrade and what `option_cost` multiplies; weapon cost per bank vs per mount; mass/speed formulas; `faiDes`/`dWep` meaning. (from [[SHIP_DESIGN_RULES]])
|
||
- **Corrections from turn internals** — `ServerPlayer+0xf9` is the AI flag (not bTurnDone); `FUN_007d7f70` is encounter detection; ProcessTurn phase 2 is trade, not diplomacy; the spine's `FUN_00840fe0` 'income' is special projects — income is `FUN_00863030` ComputeBudget. `TECHBEN_*` data blocks are inert (effects hard-coded via the 116-entry tech-name table @0x00a19718). Open: TRA/TRP producers, `FPsp2` derivation, `PERGATETRAFFIC_*` readers, exact BnkEl/BnkPr expression. (from [[strategic-turn-internals]])
|
||
- **`Mars::Application::Initialize` signature (P2-M0 finding)** — a C++ `__thiscall` detour that called the original crashed inside Initialize; the trace hook now uses a convention-agnostic asm stub. Verify the real signature (stack args? EDX use? return value?) before any `compare`/`replace` hook on it. Same caution for every `[unverified]` entry in `ghidra/addresses.json`. (from [[m0]])
|
||
- **Formula gaps surfaced by the game/sim port (feed back to Ghidra)** — which bankruptcy limit carries the 3.3 factor and when the bankruptcy start turn is stamped; the suitability→carrying-capacity hazard curve shape; the trade-points→money system-income tail; the `POPBONUS_INC` population increment; the expense-slider request term; which running total the tech-income bonus / savings aid read; node-line speed clamp at the influence radius; whether `DecayAllResearch` also hits the current target. (from [[game-sim]])
|
||
- **RESOLVED: Initialize signature** — `bool __thiscall Application::Initialize(this, AppStartup*)`, RET 4; Run = `void(this)`; OnTick = `bool(this)`. Plain thiscall wrappers are safe for Run/OnTick; Initialize needs the extra arg. (from [[loader-prototypes]])
|
||
- **PARITY BUG (fix in mars/text + Python flat_kv): duplicate config keys are FIRST-occurrence-wins in the engine** (entry erased on use; duplicates log 'multiply defined'); our readers implement last-wins. Also the brace parser DROPS a final key/value pair that lacks a trailing newline, and uses whitespace-only delimiting (braces not delimiters) with `" ' \`` quotes and no escapes — mars/parse must match these for compare mode; verify on the 12 broken sections + a no-trailing-newline fixture. (from [[loader-prototypes]])
|
||
- **Not traced end-to-end** — `Species/_NPC/weapons/*.weapon` loading and the `.effect` dictionary entry `EffectDictionary_Load` (0x008b42b0, unverified). (from [[loader-prototypes]])
|
||
- **RESOLVED: End-Turn oracle** — deterministic and cross-process: compare autosaves by sha256, no masking. Only when diffing a re-save of a *loaded post-turn* autosave: mask `Player.Status` (→0) and recompute/ignore `Summary.Checksum` (additive). (from [[determinism-oracle]])
|
||
- **SAVE_FORMAT tag corrections (fix Python reader + spec)** — real on-disk tags: `otnF` (not `ontF`) in Odes/Owep/Otch, `nextid` (not `nextId`) in NdGr2, Design = `FAIDes/DHide/DWep/DName`; `ords`/`wpts` are real tags. Python's positional R() matching hid these. RNG: float mapping `(float)(y*2^-32)`, `next_int` mask, and lazy-vs-eager twist at `left==0` still need binary confirmation. (from [[mars-stream]])
|
||
- **RESOLVED: tech `allows` default** — unlisted species = 1.0 (confirmed in `FUN_005822d0` tree-creation roll, strategic-turn-internals §2); `game/data` uses 100. Still open from game/data: what the engine's converter does with the 34 malformed tokens (`force_right o`, `crew false`, `1.0f`, `0-5`, ``90\``); repeated scalars in a block are last-wins per the sequential if/else consumers (loader-prototypes §M3) — confirm on a fixture in compare mode. (from [[game-data]])
|
||
- **RESOLVED: SAVE_FORMAT tag names** — all corrections confirmed by bytes and applied to the Python reader + spec (§10). Residual doc debt: `findings/objects/save-editor-structs.md` and `verify/design-rules/SHIP_DESIGN_RULES.md` still quote R1's `ontF/faiDes/nextId` spellings (R1-provenance; annotate rather than rewrite). (from [[SAVE_FORMAT]])
|
||
- **RESOLVED (2026-09-08, lane S): `std::string` layout and size, once and for all** — `_Bx@0, _Mysize@0x10, _Myres@0x14, _Alval@0x18`, **sizeof 0x1c**, and there is exactly **one** instantiation in this binary. `struct-recovery.md` §0 was right (its §0 prose had `size`/`res` transposed — fixed); `turn-spine.md` §1.1 was wrong (already annotated); `loader-prototypes.md` and one `addresses.json` prototype said 0x18 — fixed. Lane X's `ObservedTech` 0x18 reading was a mis-attribution: `+0x24` is the string's trailing `_Alval`, not a data field, proved by three complete enumerations of the element (`ObservedTech::Write` 0x00817cf0, ctor 0x008562a0, copy ctor 0x0079a184) and generalised over the whole exe by `tools/strfootprint.py` (65 string members, 0 collisions inside the 0x1c span, 51/52 gaps exactly 0x1c). **Zero recovered struct tables were wrong** — `pswd` included. Standing rule that follows: *never size a struct member from the offsets the code touches* — this build's STL puts the empty allocator **last** in both `string` (0x1c) and `vector` (0x10), and an empty allocator is never loaded or stored, so a touch-scan undercounts by 4 every time. See `observedtech-append.md` §9. (from [[re-windows-2000s-howto]], [[observedtech-append]])
|
||
- **RESOLVED: formula gaps (all 8)** — see [[formula-gaps]]; game/sim's low-confidence functions can now be pinned. **RESOLVED: std::string** = `_Bx@0,_Mysize@0x10,_Myres@0x14,_Alval@0x18`, sizeof 0x1c (struct-recovery §0 right; turn-spine §1.1 WRONG — annotated). **CORRECTION:** the 116-entry table @0x00a19718 is a name-membership list, not the effects table; use `g_TechIdNames` (196) + `OnTechResearched`. Open: values of the 6-entry AI-tech bonus table (0x00a17888); producer of ServerPlayer +0x224/+0x228/+0x22c beyond the setup-record copy (0x0077b620). (from [[tech-effects]])
|
||
- **RESOLVED: RNG semantics** — `Seed` is `thiscall(this, uint32)` RET 4 (MT19937 init + immediate twist); `Twist` takes `this` in ECX only; **twist is LAZY** (`if (left==0) Twist()` inside the draw), `left` lives at `+0x9c4`, confirming the save blob layout. **float mapping, needs a targeted check (not a red flag):** `NextFloat` decompiles as returning `float10`, but that is simply how x86 float returns look (value in `ST(0)`), so it is weak evidence of extended-precision *computation*; a single `y * 2^-32` multiply rounds once either way. B3 should still compare the mapping explicitly (exact constant and whether the draw is `y*2^-32`, `(y>>8)*2^-24`, or divided by 2^32-1), and the x87-vs-SSE float-parity policy from the RE how-to is still needed before the x64 standalone. `RNG_NextInt` signature still unverified. (own Ghidra pass)
|
||
- **Struct-modelling hazard (found by M2)** — an MSVC-2010 `std::vector` member is **three words**, so a naive C translation put the next pointer at `+0x18` when it is really at `+0x14`; the hook silently reported the unmodelled word (string bytes `"TION"`) as a real field. Any hand-modelled game struct must pin its size with `static_assert` and account for 3-word vectors. (from [[M2]])
|
||
- **x87 precision-control mode at runtime** — 53-bit MSVC default vs 24-bit if D3D9 grabbed the FPU; moves only the last bit (~0.094% of words). `float_from_pc24()` models it and the B3 hook logs `fpu_cw` per call, so the first B3 trace settles it. (from [[B3]])
|
||
- **Some truths are unreachable by compare** — B3's draw-divisor bug differed on 0.78% of draws yet flipped 0 decisions in 1e6, so a behavioural compare would have passed while the code was wrong. Static reading of the instruction stream remains necessary; compare validates, it does not discover. (from [[B3]])
|