sots-re/findings/subsystems/multiplayer-combat.md
alex 4ef630b266 L2: record the pgrep -f tap141i0 hazard - it kills the guest, not just tcpdump
QEMU's command line contains ifname=tap141i0, so pgrep -f on the tap name matches
the VM alongside the capture. Stopping the tcpdump that way SIGTERM'd VM141.
Nothing was lost (all artefacts were already off the guest and it restarted with
a clean volume) but it is a plug-pull on an exclusively-held guest. Use
pgrep -x tcpdump.
2026-09-08 18:31:00 -04:00

315 lines
20 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Multiplayer combat, run on the real game: two humans, one battle, lockstep
- **Type:** subsystem (live verification)
- **Status:** **verified** — a player-versus-player battle was fought inside a two-player LAN game
with no GameSpy service of any kind, and the whole run was captured outside the guest
- **Confidence:** high on everything below; every claim is a screenshot, a socket table, or a packet
- **Owner / date:** lane L2 · 2026-09-08 · guest **VM141** (`sots-re-win10-b`)
- **Predictions, committed before the run:** `sots-engine/docs/L2-predictions.md`
- **Builds on:** `multiplayer-tier0-verified.md` (lane W2) and `multiplayer-gamespy.md` (lane G2)
---
## 0. The headline
**Combat multiplayer works, and it needs no GameSpy either.** Two human players in one LAN-mode
game fought a battle at a contested colony: both got the encounter query, both chose *Fight
Manually*, both loaded the tactical combat screen, and both ran the same battle off **one shared
lockstep clock**. Over the whole session — lobby, twelve strategic turns, the encounter query and
the battle — the capture on the host side of the guest's NIC shows **zero** packets to UDP 27900,
UDP 27901, TCP 28910, TCP 28900, TCP 6667 or TCP 6500, and **zero** DNS lookups for any
`*.gamespy.com` name.
W2 closed Tier 0 with the words "the most likely place for the direct path to still be broken".
It is not broken. **The multiplayer revival is complete end to end: create, join, launch, play,
and fight.**
**And the one prediction that failed is the most interesting result.** `CombatHostPort` **3370 is
never bound — not at lobby time, not at turn time, and not during the battle.** G2 read the default
out of `Game_LoadNetworkConfig` (`0x005a0610`) correctly, and W2 correctly reported it unbound at
lobby time and inferred "it must be opened later, when a combat starts". That inference is wrong on
this arm: the combat ran to a shared clock over the **already-open strategy connection**, with the
socket table unchanged from the moment the lobby opened. See §4 — this matters for anyone writing
port-forwarding instructions, and it corrects the community advice.
---
## 1. What was run
One Windows guest (VM141), GOG 1.8.1, two instances of the shipped executable, `hooks=off` — **the
instrument was removed** (method rule 19). The guest was cloned from VM140 with lane W3's
`hooks=trace` shim config in place; that was replaced with `hooks=off` before the game was launched
(§8).
- **Instance A**, pid 2256 — host. Launched by the `SOTS` scheduled task with no arguments.
- **Instance B**, pid 6620 — joiner. `"C:\SOTS\Sword of the Stars.exe" /concurrent /join 127.0.0.1:3369`
via the `SOTSB` scheduled task. **W2's P5 reconfirmed on a second guest**: two instances, two
windows, two main loops.
Game: *Host Multi-Player* → **Custom** + **LAN**, 28 stars, **Distance 5 ly** (the minimum), 2
players, **Initial Colonies 10**, **Random Encounters 100%**, Strategic Turn Length *Unlimited*,
Combat Turn Length *240 s*. Slot 2 was switched from *Computer* to **Open** before B joined — a
clone of VM140 comes up with slot 2 set to AI, and a joiner cannot take an AI slot.
Three instruments, all outside the game process:
1. `tcpdump -i tap141i0` on **spicy**, rotating into `/root/l2cap/` — the host side of the guest's
tap device, outside the guest entirely.
2. `Get-NetUDPEndpoint` / `Get-NetTCPConnection` filtered to the two SOTS pids, polled over SSH from
a separate process (`C:\SOTS\l2\socks.ps1`).
3. `qm monitor 141 screendump` for the screen, via `tools/vmshot.py` — reads the QEMU framebuffer,
nothing inside the guest.
**Blind spot, stated up front, same as W2's:** both instances talk over 127.0.0.1, so the game's own
3369 traffic never reaches the tap. §5 turns that into a *control* rather than a hole.
## 2. Provoking a battle — what it actually took
This is worth recording because it cost most of the lane and the next lane should not re-derive it.
**Random encounters at 100% do not fire on a static empire.** Eight strategic turns were played with
Random Encounters at maximum, 20 colonies between the two players, and **no encounter at all**.
The reason is visible in the system banner: `DE 00 CR 00 DN 00`. With *Initial Colonies 10* and
*Initial Technologies 0*, **neither player starts with a single ship** — so "Manage Fleets", "Move"
and "Special" are greyed out on every system, and there is nothing for an encounter to happen to.
So a battle needs, in order:
1. **Build warships.** *Build* → Destroyer → `Armor` → *Add To Queue*. At 16 M treasury and 17 k
industrial output a batch of four destroyers completes in **2 turns**.
2. **Order a fleet somewhere hostile.** Select the fleet row in *Fleets at <system>*, press
*Move*, then click the destination **star** (not its banner, and not where the tooltip appears).
3. Because both players were Human, node travel applies: targeting a system off the node network
raises *"Alpha Fleet cannot reach <X> using nodes. Do you want to plot a sublight course?"*
Accepting gives a **sublight** course — here 28 turns to Nu Ophiuchi at Speed 4.00 / 0.2 sublight.
4. **The sublight leg is what produced the battle**, and faster than the nominal ETA: on turn 12 the
fleet drifted into **Avalon**, a colony belonging to the *other* player, and the encounter fired.
**Three UI traps, each of which cost a run here:**
- **The starmap needs real mouse motion.** `SetCursorPos` alone does not move the game's cursor —
the move-mode arrow and the destination tooltip never update. The driver must follow the warp with
an actual `mouse_event(MOUSEEVENTF_MOVE, …)` relative jiggle. W2's `ui.ps1` has no such jiggle,
which is why its recipe covers only fixed-position buttons and never the map.
- **The banner is not the star.** System banners sit up and to the left of their star with a short
leader stem, and a neighbouring banner will cover the star underneath it. Clicking the banner does
nothing; clicking where the hover tooltip is drawn does nothing. Zoom in (mouse wheel) until the
banners separate, then click the star itself.
- **A modal move-warning dialog silently stops the turn loop.** An unattended "end turn on both
instances" loop ran four full cycles against a *"plot a sublight course?"* dialog and the turn
number never moved. Any automated turn driver must screenshot and check before clicking.
`tools/vmshot.py --one 141` plus a PIL crop is the whole read-back loop; the game's own text is
legible at 3–7× upscale and that is how every reading below was taken.
## 3. The battle — P2, P5, and the lockstep result
**P5 — the encounter query reaches both players — confirmed.** Both instances raised the C1
*Encounter at Avalon* screen (`SNMDoEncounterQuery`), each showing **both** player cards: the
attacker `re(2)` with *Ships: 4 / DE: 4 / 4 DE Armor*, and the defender `re` with *No ships
available* (host's own view) / *No ships detected* (attacker's view). The action strip offered the
full `ENCACTION_*` set — *Fight Manually*, auto-resolve, fight-if, auto-resolve-peacefully, and on
the attacker's side the two surrender variants. **Both players independently chose *Fight
Manually*** and pressed *Done*.
**The combat launched.** Both instances showed the C2 loading screen — the `CL_*` versus backdrop
titled *Encounter at Avalon*, `re(2)` **VS** `re` — and then the C3 tactical HUD:
| | Instance A (pid 2256, host, defender) | Instance B (pid 6620, joiner, attacker) |
|---|---|---|
| Title | `Avalon` | `Avalon` |
| Own forces | none — planet only | `DE Armor (1)` selected, weapons panel, 3 gun groups |
| Player list | `re(2)` in the enemy colour | `re(2)` |
| Fleet Commands strip | present | present |
| Combat clock | running | running |
**P2 — combat resolves in lockstep — confirmed, and the clock is the proof.** The two instances
were sampled alternately, and the combat clock is a *single shared value*, not two independent
countdowns:
| Wall clock | Instance sampled | Combat clock |
|---|---|---|
| 18:00:39 | B (attacker) | 3:51 |
| 18:01:48 | A (defender) | 3:31 |
| 18:04:33 | B (attacker) | 2:42 |
| 18:05:59 | B (attacker) | 2:18 |
Those four readings lie on one line at **≈ 0.29 × real time** regardless of which instance is
sampled — 240 s of combat took roughly 14 minutes of wall clock on a guest running two copies of a
D3D9 game through DXVK on llvmpipe. A defender sampled at 18:01:48 sitting exactly on the line drawn
by attacker samples at 18:00:39 and 18:04:33 is what lockstep looks like from the outside: **one
simulation, two views, neither free-running.** No `@SYNC ERROR` appeared on either instance.
**The sim does not stall when a window loses focus.** Instance B advanced 69 s of combat time during
a stretch when instance A held the foreground continuously. This matters for anyone automating the
lab: the two-instance configuration is not a focus-ping-pong problem.
**The battle ended cleanly and both players came out of it together.** The 240 s combat timer ran to
`0:00` and the round expired (`CombatExpiredFunctor`); both instances raised the C4 end dialog
**"Round draw."** — the `COMBAT_*` string for an expired round. Dismissing it returned **both**
instances to the strategy screen on **Turn 13** (the encounter fired on turn 12), each with its own
empire intact: the attacker on Nunki with its fleet detached at Avalon, the defender on Regulus with
its own two-destroyer Alpha Fleet. `SNMAllCombatDone` → `SNMResumePlaying` did what
`ui-screen-map.md` §2.2 step 6 says it does.
Neither instance hung at `COMBAT_WAITINGFORPLAYER` or `COMBAT_WAITINGFORSERVER`, neither crashed,
neither showed a sync error, and the two turn counters never diverged. **P2 confirmed in full.**
## 4. P1 — `CombatHostPort` 3370 — **falsified**
This was the prediction with a number attached, and it failed cleanly.
The UDP socket table for the two SOTS pids was polled continuously — every 700 ms inside the guest
from lobby creation to turn 12, then every 2 s from outside across the whole battle (18:01:24 to
18:16, covering combat load, the full 240 s of tactical combat, the round-draw dialog and the return
to the strategy screen). **Both logs contain exactly one line each.** The socket table never changed:
```
UDP pid=2256 0.0.0.0:3369 <- host, HostPort
UDP pid=2256 0.0.0.0:50020 <- host, ephemeral
UDP pid=6620 0.0.0.0:50021 <- joiner, ephemeral
```
There is **no TCP at all** for either process at any point, confirming G2's import-table reading
(`bind`/`sendto`/`recvfrom` but no `listen`, no `accept`) on a second workload.
**3370 was never bound**: not when the lobby opened, not across twelve strategic turns, not at the
encounter query, not during combat loading, and not while the tactical simulation was running with
both players in it. W2 observed it unbound at lobby time and wrote "it must be opened later, when a
combat starts". On the direct-join arm, with the strategy host also hosting the battle, **it is not**.
The straightforward reading is that the combat lockstep is carried over the **already-established
strategy connection** — the host's 3369 socket and the two ephemeral sockets that carried
`SNMEndTurn` and `SNMDoEncounterQuery` also carry the `CNM*` combat messages. `CombatHostPort` is
presumably for the case the ini key exists to serve: a **separate** combat host, i.e. the
`SNMHostCombat` → `SNMHostCombatReply` handshake electing a *different* player (or a dedicated
server) to host the battle, which cannot happen in a two-player game where the strategy host is
already a participant.
**Consequences.**
- **Port-forwarding advice should say 3369 and only 3369.** PCGamingWiki and the community threads
G2 quotes say "LAN mode + forward port 3369", and that is exactly right; nothing here supports
adding 3370, at least for two players over a direct join.
- **Not generalised.** This is one battle, two players, one guest, with the strategy host as a
combatant. Three or more players, a battle between two *clients* with the host merely relaying, or
`sots_server.exe` as a dedicated host, could each elect a different combat host and bind 3370.
That is the obvious next experiment and it is **not run** here.
- `LanScanPortRange` still covers **3369..3370** for the LAN browse broadcast, so 3370 is not dead
in the binary — it is just not bound on this path.
## 5. P3 and P4 — GameSpy at combat time, and the control
**P3 — no GameSpy traffic at combat time — confirmed.** Across the whole run, sampled twice (once
mid-game and once with the battle in progress):
| Sample | Total packets on `tap141i0` | GameSpy ports | `*.gamespy.com` DNS |
|---|---|---|---|
| Through turn 9 | 42,381 | **0** | **0** |
| With the battle in progress | 82,470 | **0** | **0** |
| **Whole run, both capture files** | **101,835** | **0** | **0** |
Ports counted: UDP 27900, UDP 27901, TCP 28910, TCP 28900, TCP 6667, TCP 6500. This closes the
question G2 left open from the other direction: `GameSpy_NNBeginNegotiationWithSocket`
(`0x00412590`) has two call sites, the browser-mediated join and the host reporting path, and **the
combat host/join path is a third thing that reaches neither**. No `natneg1`/`natneg2` lookup ever
happened, at lobby time or at combat time.
The only names resolved in the combat window were Windows telemetry (`fd.api.iris.microsoft.com`,
`v10.events.data.microsoft.com`, `wpad.home.arpa`, …) plus exactly one
`www.kerberos-productions.com` — the MOTD fetch W2 documented, once per launch.
**P4 — combat traffic stays on loopback — confirmed, and it is the control.** Zero packets to or
from UDP 3369 or 3370 appeared on the tap, while 101,835 other packets did. The instrument is
demonstrably live and demonstrably blind to the game's own traffic, which is exactly the shape of
evidence that makes the zero in P3 mean something: **the capture cannot see SOTS's own packets, and
it still sees every other packet the guest emits — so a GameSpy packet would have been caught.**
It also settles a smaller question: **the combat host does not announce itself on the LAN.** Nothing
was broadcast to 3369 or 3370 when the battle started, so a combat session is not discoverable the
way a lobby is.
## 6. Adopting VM141 — what a ZFS clone of VM140 actually needed
Recorded because four more guests exist and this took under fifteen minutes.
The clone (`zfs clone rpool/data/vm-140-disk-0@clone-base`, fresh MAC `BC:24:11:8A:11:F0`) had **no
IPv4 address in any ARP table on spicy** and `qm agent` reported *"QEMU guest agent is not running"*
(none is installed on these guests). A 40-second `tcpdump -i tap141i0 -e "ether src <mac>"` caught
**one** packet — an IPv6 neighbour solicitation — and no DHCP at all, which looks exactly like a
clone whose inherited static IPv4 has been disabled for a duplicate-address conflict with its parent.
**It was none of those things. The guest was simply idle.** Driving the console with
`qm sendkey 141 …` into an elevated `cmd` and reading `ipconfig /all` off a `screendump` showed a
perfectly healthy **DHCP** lease, `192.168.10.143/16`, obtained at 16:36:40 — half an hour before
anyone looked. SSH with Alex's key worked on the first try.
**The lesson for the remaining guests: do not diagnose a cloned guest from ARP or from a short
packet capture.** An idle Windows box on a quiet LAN emits nothing for minutes at a time, and its
absence from the neighbour table only means nobody has talked to it. Take a `screendump` first.
Console driving via `qm sendkey` works and is worth knowing:
- `qm sendkey 141 meta_l` opens the Start menu; `meta_l-r` opens *Run* — but **key delivery lags by
tens of seconds**, so a screendump taken 2 s after a keystroke will show the *previous* state.
Send, wait, then shoot; a "nothing happened" screenshot is usually just early.
- Letters/digits map to their own names, and `shift-<key>` gives the shifted form; there is no
string primitive, so a typing helper that expands a string into one `qm sendkey` per character
(batched into a single SSH call) is the practical unit.
- The `re` account is a local administrator with UAC on: `cmd` + `ctrl-shift-ret` in *Run* raises the
consent dialog, which `alt-y` accepts.
What the clone carried over correctly and did **not** need fixing: the game at `C:\SOTS`, DXVK, the
`SOTS`/`SOTSUI`/`SOTSUI2` scheduled tasks, the OpenSSH host keys and authorized keys, and — usefully
— **the Windows Defender Firewall rule for the game**, so the "has blocked some features of this
app" prompt W2 had to click through on VM140 never appeared here.
**One hazard found the hard way, and it will bite the next lane too: `pgrep -f tap141i0` matches
QEMU itself.** Stopping the capture with `pgrep -f "tap141i0" | xargs kill` on spicy killed the
guest — QEMU's command line contains `...,ifname=tap141i0,...`, so the VM's own process matched
alongside `tcpdump`, and the journal records `kvm: terminating on signal 15`. It cost nothing here
because every artefact was already off the guest, and the guest restarted clean (`fsutil dirty query
C:` → not dirty), but it is a plug-pull on someone's exclusive guest. **Kill the capture with
`pgrep -x tcpdump`, never with `-f` on the tap name.** The same trap catches `pkill -f "tcpdump -i
tap141i0"` run over SSH, where the pattern also matches the remote shell that is running it.
What it carried over that **had** to be fixed:
- **`shim.cfg` was lane W3's `hooks=trace`.** Any lane that wants the shipped game must overwrite it;
`C:\SOTS\shim.cfg.l2backup` holds what was there.
- **The duplicate hostname.** Both guests answer to `SOTS-RE` and share a SID. It was left alone —
SSH by IP is unaffected and nothing in this lane needed NetBIOS — but two machines with one name on
one LAN is a latent trap for anything that resolves by name.
- **`C:\SOTS\launch-b.cmd` does not exist**, although `C:\SOTS\w2\sotsb.xml` invokes it and
`C:\SOTS\w2\runB.txt` proves it ran on VM140. It has to be recreated (it reads the argument line
from `C:\SOTS\w2\argsB.txt`); a copy is at `C:\SOTS\l2\` on VM141.
- **Slot 2 defaults to *Computer*.** A `/join` client cannot take an AI slot; click the small red
cross at the slot card's top-right corner to set it *Open* before launching the joiner.
## 7. Files
- Capture: `verify/results/multiplayer/l2-combat.pcap` (on spicy at `/root/l2cap/`)
- Screenshots: `verify/results/multiplayer/l2-*.png`
- Socket-table log: `verify/results/multiplayer/l2-socks.log`
- Predictions, committed before the run: `sots-engine/docs/L2-predictions.md`
- Helper left on the guest: `C:\SOTS\l2\` (`ui.ps1` with wheel/jiggle support, `socks.ps1`,
`poller.ps1`, `shim.cfg`, `l2ui.xml`); scheduled task **L2UI**; task **SOTSB** now registered.
## 8. What this run did NOT cover (rule 15)
- **One guest, loopback only.** No two-machine combat, no real network latency, no NAT, no packet
loss. `CombatLatency` defaults to 1000 ms and was never stressed.
- **One battle, two players, and the strategy host was a combatant.** The `SNMHostCombat` /
`SNMHostCombatReply` election never had a choice to make. Three-plus players, or a battle between
two clients, is where 3370 might yet appear (§4).
- **Very lopsided.** Four destroyers against an undefended colony. No reinforcement waves, no
reserves, no bombardment, no ceasefire negotiation, no surrender, no combat between fleets.
- **One combat round.** Multi-round battles, `SNMCombatTimeRemaining` extension, and the
querying/combat/resume loop over several successive turns are untested.
- **No sync log.** `SyncCheckCombat` is on by default but `SyncLogCombat` is off, and it was left
off; a desync would have shown as a symptom, not as a log.
- **`sots_server.exe` still not run.** The community's "the dedicated server does not seem to work"
bug remains unreproduced, and it is now the cheapest remaining multiplayer item — and, per §4, the
one most likely to be where `CombatHostPort` earns its name.
- **`hooks=off` throughout**, so nothing internal was observed: no `CNM*` message types, no wire
format, no sync-check payload. Everything here is UI, sockets and packets.