sots-re/findings/subsystems/multiplayer-combat.md
alex 4ef630b266 L2: record the pgrep -f tap141i0 hazard - it kills the guest, not just tcpdump
QEMU's command line contains ifname=tap141i0, so pgrep -f on the tap name matches
the VM alongside the capture. Stopping the tcpdump that way SIGTERM'd VM141.
Nothing was lost (all artefacts were already off the guest and it restarted with
a clean volume) but it is a plug-pull on an exclusively-held guest. Use
pgrep -x tcpdump.
2026-09-08 18:31:00 -04:00

20 KiB
Raw Blame History

Multiplayer combat, run on the real game: two humans, one battle, lockstep

  • Type: subsystem (live verification)
  • Status: verified — a player-versus-player battle was fought inside a two-player LAN game with no GameSpy service of any kind, and the whole run was captured outside the guest
  • Confidence: high on everything below; every claim is a screenshot, a socket table, or a packet
  • Owner / date: lane L2 · 2026-09-08 · guest VM141 (sots-re-win10-b)
  • Predictions, committed before the run: sots-engine/docs/L2-predictions.md
  • Builds on: multiplayer-tier0-verified.md (lane W2) and multiplayer-gamespy.md (lane G2)

0. The headline

Combat multiplayer works, and it needs no GameSpy either. Two human players in one LAN-mode game fought a battle at a contested colony: both got the encounter query, both chose Fight Manually, both loaded the tactical combat screen, and both ran the same battle off one shared lockstep clock. Over the whole session — lobby, twelve strategic turns, the encounter query and the battle — the capture on the host side of the guest's NIC shows zero packets to UDP 27900, UDP 27901, TCP 28910, TCP 28900, TCP 6667 or TCP 6500, and zero DNS lookups for any *.gamespy.com name.

W2 closed Tier 0 with the words "the most likely place for the direct path to still be broken". It is not broken. The multiplayer revival is complete end to end: create, join, launch, play, and fight.

And the one prediction that failed is the most interesting result. CombatHostPort 3370 is never bound — not at lobby time, not at turn time, and not during the battle. G2 read the default out of Game_LoadNetworkConfig (0x005a0610) correctly, and W2 correctly reported it unbound at lobby time and inferred "it must be opened later, when a combat starts". That inference is wrong on this arm: the combat ran to a shared clock over the already-open strategy connection, with the socket table unchanged from the moment the lobby opened. See §4 — this matters for anyone writing port-forwarding instructions, and it corrects the community advice.


1. What was run

One Windows guest (VM141), GOG 1.8.1, two instances of the shipped executable, hooks=off — the instrument was removed (method rule 19). The guest was cloned from VM140 with lane W3's hooks=trace shim config in place; that was replaced with hooks=off before the game was launched (§8).

  • Instance A, pid 2256 — host. Launched by the SOTS scheduled task with no arguments.
  • Instance B, pid 6620 — joiner. "C:\SOTS\Sword of the Stars.exe" /concurrent /join 127.0.0.1:3369 via the SOTSB scheduled task. W2's P5 reconfirmed on a second guest: two instances, two windows, two main loops.

Game: Host Multi-Player → Custom + LAN, 28 stars, Distance 5 ly (the minimum), 2 players, Initial Colonies 10, Random Encounters 100%, Strategic Turn Length Unlimited, Combat Turn Length 240 s. Slot 2 was switched from Computer to Open before B joined — a clone of VM140 comes up with slot 2 set to AI, and a joiner cannot take an AI slot.

Three instruments, all outside the game process:

  1. tcpdump -i tap141i0 on spicy, rotating into /root/l2cap/ — the host side of the guest's tap device, outside the guest entirely.
  2. Get-NetUDPEndpoint / Get-NetTCPConnection filtered to the two SOTS pids, polled over SSH from a separate process (C:\SOTS\l2\socks.ps1).
  3. qm monitor 141 screendump for the screen, via tools/vmshot.py — reads the QEMU framebuffer, nothing inside the guest.

Blind spot, stated up front, same as W2's: both instances talk over 127.0.0.1, so the game's own 3369 traffic never reaches the tap. §5 turns that into a control rather than a hole.

2. Provoking a battle — what it actually took

This is worth recording because it cost most of the lane and the next lane should not re-derive it.

Random encounters at 100% do not fire on a static empire. Eight strategic turns were played with Random Encounters at maximum, 20 colonies between the two players, and no encounter at all. The reason is visible in the system banner: DE 00 CR 00 DN 00. With Initial Colonies 10 and Initial Technologies 0, neither player starts with a single ship — so "Manage Fleets", "Move" and "Special" are greyed out on every system, and there is nothing for an encounter to happen to.

So a battle needs, in order:

  1. Build warships. Build → Destroyer → Armor → Add To Queue. At 16 M treasury and 17 k industrial output a batch of four destroyers completes in 2 turns.
  2. Order a fleet somewhere hostile. Select the fleet row in Fleets at <system>, press Move, then click the destination star (not its banner, and not where the tooltip appears).
  3. Because both players were Human, node travel applies: targeting a system off the node network raises "Alpha Fleet cannot reach <X> using nodes. Do you want to plot a sublight course?" Accepting gives a sublight course — here 28 turns to Nu Ophiuchi at Speed 4.00 / 0.2 sublight.
  4. The sublight leg is what produced the battle, and faster than the nominal ETA: on turn 12 the fleet drifted into Avalon, a colony belonging to the other player, and the encounter fired.

Three UI traps, each of which cost a run here:

  • The starmap needs real mouse motion. SetCursorPos alone does not move the game's cursor — the move-mode arrow and the destination tooltip never update. The driver must follow the warp with an actual mouse_event(MOUSEEVENTF_MOVE, …) relative jiggle. W2's ui.ps1 has no such jiggle, which is why its recipe covers only fixed-position buttons and never the map.
  • The banner is not the star. System banners sit up and to the left of their star with a short leader stem, and a neighbouring banner will cover the star underneath it. Clicking the banner does nothing; clicking where the hover tooltip is drawn does nothing. Zoom in (mouse wheel) until the banners separate, then click the star itself.
  • A modal move-warning dialog silently stops the turn loop. An unattended "end turn on both instances" loop ran four full cycles against a "plot a sublight course?" dialog and the turn number never moved. Any automated turn driver must screenshot and check before clicking.

tools/vmshot.py --one 141 plus a PIL crop is the whole read-back loop; the game's own text is legible at 3–7× upscale and that is how every reading below was taken.

3. The battle — P2, P5, and the lockstep result

P5 — the encounter query reaches both players — confirmed. Both instances raised the C1 Encounter at Avalon screen (SNMDoEncounterQuery), each showing both player cards: the attacker re(2) with Ships: 4 / DE: 4 / 4 DE Armor, and the defender re with No ships available (host's own view) / No ships detected (attacker's view). The action strip offered the full ENCACTION_* set — Fight Manually, auto-resolve, fight-if, auto-resolve-peacefully, and on the attacker's side the two surrender variants. Both players independently chose Fight Manually and pressed Done.

The combat launched. Both instances showed the C2 loading screen — the CL_* versus backdrop titled Encounter at Avalon, re(2) VS re — and then the C3 tactical HUD:

Instance A (pid 2256, host, defender) Instance B (pid 6620, joiner, attacker)
Title Avalon Avalon
Own forces none — planet only DE Armor (1) selected, weapons panel, 3 gun groups
Player list re(2) in the enemy colour re(2)
Fleet Commands strip present present
Combat clock running running

P2 — combat resolves in lockstep — confirmed, and the clock is the proof. The two instances were sampled alternately, and the combat clock is a single shared value, not two independent countdowns:

Wall clock Instance sampled Combat clock
18:00:39 B (attacker) 3:51
18:01:48 A (defender) 3:31
18:04:33 B (attacker) 2:42
18:05:59 B (attacker) 2:18

Those four readings lie on one line at ≈ 0.29 × real time regardless of which instance is sampled — 240 s of combat took roughly 14 minutes of wall clock on a guest running two copies of a D3D9 game through DXVK on llvmpipe. A defender sampled at 18:01:48 sitting exactly on the line drawn by attacker samples at 18:00:39 and 18:04:33 is what lockstep looks like from the outside: one simulation, two views, neither free-running. No @SYNC ERROR appeared on either instance.

The sim does not stall when a window loses focus. Instance B advanced 69 s of combat time during a stretch when instance A held the foreground continuously. This matters for anyone automating the lab: the two-instance configuration is not a focus-ping-pong problem.

The battle ended cleanly and both players came out of it together. The 240 s combat timer ran to 0:00 and the round expired (CombatExpiredFunctor); both instances raised the C4 end dialog "Round draw." — the COMBAT_* string for an expired round. Dismissing it returned both instances to the strategy screen on Turn 13 (the encounter fired on turn 12), each with its own empire intact: the attacker on Nunki with its fleet detached at Avalon, the defender on Regulus with its own two-destroyer Alpha Fleet. SNMAllCombatDone → SNMResumePlaying did what ui-screen-map.md §2.2 step 6 says it does.

Neither instance hung at COMBAT_WAITINGFORPLAYER or COMBAT_WAITINGFORSERVER, neither crashed, neither showed a sync error, and the two turn counters never diverged. P2 confirmed in full.

4. P1 — CombatHostPort 3370 — falsified

This was the prediction with a number attached, and it failed cleanly.

The UDP socket table for the two SOTS pids was polled continuously — every 700 ms inside the guest from lobby creation to turn 12, then every 2 s from outside across the whole battle (18:01:24 to 18:16, covering combat load, the full 240 s of tactical combat, the round-draw dialog and the return to the strategy screen). Both logs contain exactly one line each. The socket table never changed:

UDP pid=2256 0.0.0.0:3369     <- host, HostPort
UDP pid=2256 0.0.0.0:50020    <- host, ephemeral
UDP pid=6620 0.0.0.0:50021    <- joiner, ephemeral

There is no TCP at all for either process at any point, confirming G2's import-table reading (bind/sendto/recvfrom but no listen, no accept) on a second workload.

3370 was never bound: not when the lobby opened, not across twelve strategic turns, not at the encounter query, not during combat loading, and not while the tactical simulation was running with both players in it. W2 observed it unbound at lobby time and wrote "it must be opened later, when a combat starts". On the direct-join arm, with the strategy host also hosting the battle, it is not.

The straightforward reading is that the combat lockstep is carried over the already-established strategy connection — the host's 3369 socket and the two ephemeral sockets that carried SNMEndTurn and SNMDoEncounterQuery also carry the CNM* combat messages. CombatHostPort is presumably for the case the ini key exists to serve: a separate combat host, i.e. the SNMHostCombat → SNMHostCombatReply handshake electing a different player (or a dedicated server) to host the battle, which cannot happen in a two-player game where the strategy host is already a participant.

Consequences.

  • Port-forwarding advice should say 3369 and only 3369. PCGamingWiki and the community threads G2 quotes say "LAN mode + forward port 3369", and that is exactly right; nothing here supports adding 3370, at least for two players over a direct join.
  • Not generalised. This is one battle, two players, one guest, with the strategy host as a combatant. Three or more players, a battle between two clients with the host merely relaying, or sots_server.exe as a dedicated host, could each elect a different combat host and bind 3370. That is the obvious next experiment and it is not run here.
  • LanScanPortRange still covers 3369..3370 for the LAN browse broadcast, so 3370 is not dead in the binary — it is just not bound on this path.

5. P3 and P4 — GameSpy at combat time, and the control

P3 — no GameSpy traffic at combat time — confirmed. Across the whole run, sampled twice (once mid-game and once with the battle in progress):

Sample Total packets on tap141i0 GameSpy ports *.gamespy.com DNS
Through turn 9 42,381 0 0
With the battle in progress 82,470 0 0
Whole run, both capture files 101,835 0 0

Ports counted: UDP 27900, UDP 27901, TCP 28910, TCP 28900, TCP 6667, TCP 6500. This closes the question G2 left open from the other direction: GameSpy_NNBeginNegotiationWithSocket (0x00412590) has two call sites, the browser-mediated join and the host reporting path, and the combat host/join path is a third thing that reaches neither. No natneg1/natneg2 lookup ever happened, at lobby time or at combat time.

The only names resolved in the combat window were Windows telemetry (fd.api.iris.microsoft.com, v10.events.data.microsoft.com, wpad.home.arpa, …) plus exactly one www.kerberos-productions.com — the MOTD fetch W2 documented, once per launch.

P4 — combat traffic stays on loopback — confirmed, and it is the control. Zero packets to or from UDP 3369 or 3370 appeared on the tap, while 101,835 other packets did. The instrument is demonstrably live and demonstrably blind to the game's own traffic, which is exactly the shape of evidence that makes the zero in P3 mean something: the capture cannot see SOTS's own packets, and it still sees every other packet the guest emits — so a GameSpy packet would have been caught.

It also settles a smaller question: the combat host does not announce itself on the LAN. Nothing was broadcast to 3369 or 3370 when the battle started, so a combat session is not discoverable the way a lobby is.

6. Adopting VM141 — what a ZFS clone of VM140 actually needed

Recorded because four more guests exist and this took under fifteen minutes.

The clone (zfs clone rpool/data/vm-140-disk-0@clone-base, fresh MAC BC:24:11:8A:11:F0) had no IPv4 address in any ARP table on spicy and qm agent reported "QEMU guest agent is not running" (none is installed on these guests). A 40-second tcpdump -i tap141i0 -e "ether src <mac>" caught one packet — an IPv6 neighbour solicitation — and no DHCP at all, which looks exactly like a clone whose inherited static IPv4 has been disabled for a duplicate-address conflict with its parent.

It was none of those things. The guest was simply idle. Driving the console with qm sendkey 141 … into an elevated cmd and reading ipconfig /all off a screendump showed a perfectly healthy DHCP lease, 192.168.10.143/16, obtained at 16:36:40 — half an hour before anyone looked. SSH with Alex's key worked on the first try.

The lesson for the remaining guests: do not diagnose a cloned guest from ARP or from a short packet capture. An idle Windows box on a quiet LAN emits nothing for minutes at a time, and its absence from the neighbour table only means nobody has talked to it. Take a screendump first.

Console driving via qm sendkey works and is worth knowing:

  • qm sendkey 141 meta_l opens the Start menu; meta_l-r opens Run — but key delivery lags by tens of seconds, so a screendump taken 2 s after a keystroke will show the previous state. Send, wait, then shoot; a "nothing happened" screenshot is usually just early.
  • Letters/digits map to their own names, and shift-<key> gives the shifted form; there is no string primitive, so a typing helper that expands a string into one qm sendkey per character (batched into a single SSH call) is the practical unit.
  • The re account is a local administrator with UAC on: cmd + ctrl-shift-ret in Run raises the consent dialog, which alt-y accepts.

What the clone carried over correctly and did not need fixing: the game at C:\SOTS, DXVK, the SOTS/SOTSUI/SOTSUI2 scheduled tasks, the OpenSSH host keys and authorized keys, and — usefully — the Windows Defender Firewall rule for the game, so the "has blocked some features of this app" prompt W2 had to click through on VM140 never appeared here.

One hazard found the hard way, and it will bite the next lane too: pgrep -f tap141i0 matches QEMU itself. Stopping the capture with pgrep -f "tap141i0" | xargs kill on spicy killed the guest — QEMU's command line contains ...,ifname=tap141i0,..., so the VM's own process matched alongside tcpdump, and the journal records kvm: terminating on signal 15. It cost nothing here because every artefact was already off the guest, and the guest restarted clean (fsutil dirty query C: → not dirty), but it is a plug-pull on someone's exclusive guest. Kill the capture with pgrep -x tcpdump, never with -f on the tap name. The same trap catches pkill -f "tcpdump -i tap141i0" run over SSH, where the pattern also matches the remote shell that is running it.

What it carried over that had to be fixed:

  • shim.cfg was lane W3's hooks=trace. Any lane that wants the shipped game must overwrite it; C:\SOTS\shim.cfg.l2backup holds what was there.
  • The duplicate hostname. Both guests answer to SOTS-RE and share a SID. It was left alone — SSH by IP is unaffected and nothing in this lane needed NetBIOS — but two machines with one name on one LAN is a latent trap for anything that resolves by name.
  • C:\SOTS\launch-b.cmd does not exist, although C:\SOTS\w2\sotsb.xml invokes it and C:\SOTS\w2\runB.txt proves it ran on VM140. It has to be recreated (it reads the argument line from C:\SOTS\w2\argsB.txt); a copy is at C:\SOTS\l2\ on VM141.
  • Slot 2 defaults to Computer. A /join client cannot take an AI slot; click the small red cross at the slot card's top-right corner to set it Open before launching the joiner.

7. Files

  • Capture: verify/results/multiplayer/l2-combat.pcap (on spicy at /root/l2cap/)
  • Screenshots: verify/results/multiplayer/l2-*.png
  • Socket-table log: verify/results/multiplayer/l2-socks.log
  • Predictions, committed before the run: sots-engine/docs/L2-predictions.md
  • Helper left on the guest: C:\SOTS\l2\ (ui.ps1 with wheel/jiggle support, socks.ps1, poller.ps1, shim.cfg, l2ui.xml); scheduled task L2UI; task SOTSB now registered.

8. What this run did NOT cover (rule 15)

  • One guest, loopback only. No two-machine combat, no real network latency, no NAT, no packet loss. CombatLatency defaults to 1000 ms and was never stressed.
  • One battle, two players, and the strategy host was a combatant. The SNMHostCombat / SNMHostCombatReply election never had a choice to make. Three-plus players, or a battle between two clients, is where 3370 might yet appear (§4).
  • Very lopsided. Four destroyers against an undefended colony. No reinforcement waves, no reserves, no bombardment, no ceasefire negotiation, no surrender, no combat between fleets.
  • One combat round. Multi-round battles, SNMCombatTimeRemaining extension, and the querying/combat/resume loop over several successive turns are untested.
  • No sync log. SyncCheckCombat is on by default but SyncLogCombat is off, and it was left off; a desync would have shown as a symptom, not as a log.
  • sots_server.exe still not run. The community's "the dedicated server does not seem to work" bug remains unreproduced, and it is now the cheapest remaining multiplayer item — and, per §4, the one most likely to be where CombatHostPort earns its name.
  • hooks=off throughout, so nothing internal was observed: no CNM* message types, no wire format, no sync-check payload. Everything here is UI, sockets and packets.