flybrain/infra/docs/runbook.md

flybrain infra runbook

Operational procedures for the two fly demo containers (fly-pokemon the release container, fly-platformer the platformer container) plus the throwaway spike (fly-spike the dev container). Every command below runs on the host (<host-ip>) as root, unless marked "operator box". There is no sshd in any fly container — everything goes through pct exec/pct push/pct pull.

Read docs/design/infra.md first if you have not; this runbook assumes its vocabulary (flysim, flystage, flystage-web, flycast, flypush, flybridge, fly.target) without re-explaining it.

Do not touch the neighbouring production container

The neighbouring production container is another service's production workload. Nothing in infra/ targets it, references it, or shares a dataset/pool path with it beyond the same rpool SSD mirror both live on. If a disk-full or write-storm incident on a fly container ever threatens rpool, the priority order is: stop the fly container's writes first (pct stop <ctid> if needed), never touch the neighbouring production container. See docs/design/infra.md section 0 and section 8 ("SSD write endurance") for why the checkpoint cadence design specifically exists to protect this mirror.

Start / stop

Single verb, fly.target:

pct exec <ctid> -- systemctl start fly.target   # xvfb, pulse, mediamtx, flysim,
                                                  # flystage-web, flystage, flycast
pct exec <ctid> -- systemctl stop  fly.target
pct exec <ctid> -- systemctl status fly.target

flypush is deliberately not part of fly.target (see "Local-to-Twitch flip" below) — stopping/starting the target never touches it, and it is not affected by a fly.target restart. flybridge is in fly.target but as Wants=, so a missing/broken bridge never blocks target start/stop.

Expect flycast to take up to about two minutes longer than everything else on a cold start or after an xvfb restart: it is After=flystage.service and waits (up to 120 s, then starts anyway with a warning) for the page to be painted and its audio client attached, which is the capture-freeze ordering gate — see "Capture freeze" below.

To restart just one unit after a config change: pct exec <ctid> -- systemctl restart <unit>.service. 05-deploy.sh already does this for you when it pushes a changed unit file and detects a daemon-reload is needed, but a config file inside /etc/fly/ (chromium-flags, pulse.pa, mediamtx.yml) needs a manual restart of the unit that reads it.

Local-to-Twitch flip

Local test mode (PUSH_TARGET=local in the env file) leaves flypush.service disabled; everything else is identical to production. Flipping to Twitch:

# 1. Install the stream key (from the operator box, where `pass` lives):
FORCE_SECRETS=1 infra/06-secrets.sh <release-env>

# 2. Enable flypush:
pct exec <release-ctid> -- systemctl enable --now flypush.service

Flip back: pct exec <release-ctid> -- systemctl disable --now flypush.service. Neither direction restarts the sim, the page, the encoder, or the recording — that is the entire reason flycast and flypush are split into two units (docs/design/infra.md section 3).

After flipping to Twitch, also enable the 23h restart guard: pct exec <release-ctid> -- systemctl enable --now flypush-restart.timer (or re-run 07-enable.sh with PUSH_TARGET=twitch set in the env file, which does both).

Rotate the stream key

1. Generate a new key on Twitch's dashboard for the channel.
2. Update the pass entry (twitch/<channel>-key or twitch/<platformer-channel>-key) on The operator box.
3. Re-run secrets install: FORCE_SECRETS=1 infra/06-secrets.sh <release-env>. This overwrites /etc/fly/creds/twitch-key.cred with the new encrypted credential.
4. Restart flypush to pick it up: pct exec <release-ctid> -- systemctl restart flypush.service.
5. Confirm: pct exec <release-ctid> -- journalctl -u flypush -n 50 shows a clean reconnect, and infra/verify.sh <release-env>'s key-prefix check passes against the new key.

Rotate immediately if a key ever appears in a log, a commit, or chat — the design's threat model (docs/design/infra.md section 3) is explicit that the key's presence in /proc/<pid>/cmdline is an accepted, contained residual risk, not a reason to be casual about rotation.

On-screen chat: ban a phrase, or kill the panel

The right rail carries a live CHAT panel. Chat is AutoMod-passed, sanitized by flybridge, sanitized again and deny-list filtered by flysim, and carried in a 12-line ring in the feed header (docs/control-api.md, POST /chat). Chat never reaches the simulation, so nothing here can affect the fly — only what the audience sees on the broadcast.

Ban a phrase or a name, live, without dropping a frame:

# On the host. The file is operator-owned: 05-deploy.sh installed it once and never
# overwrites it. One pattern per line, matched case-insensitively anywhere in the
# name or the sanitized text; `#` comments.
pct exec <release-ctid> -- sh -c 'printf "%s\n" "some phrase" >> /srv/fly/chat-deny.txt'
pct exec <release-ctid> -- systemctl kill -s HUP flysim.service
pct exec <release-ctid> -- journalctl -u flysim -n 5   # "chat deny list reloaded patterns=N ... forced=true"

SIGHUP re-reads that file and does nothing else — no restart, no reload of anything else, no interruption. flysim also re-reads it once a minute by itself, so a forgotten signal costs at most 60 seconds.

Turn the whole panel off (the kill switch):

# Source end, no Twitch credentials needed. Survives a flysim restart because
# 05-deploy.sh writes it from the env file.
#   1. set FLY_CHAT_ENABLED=0 in the release env file
#   2. infra/05-deploy.sh <release-env>
#   3. pct exec <release-ctid> -- systemctl restart flysim.service
# In an emergency, without a deploy:
pct exec <release-ctid> -- sh -c 'sed -i s/FLY_CHAT_ENABLED=1/FLY_CHAT_ENABLED=0/ /etc/fly/fly.env'
pct exec <release-ctid> -- systemctl restart flysim.service

POST /chat then answers 403 and the feed header omits chat entirely, so the page blanks the panel rather than freezing the last seven lines. Put the env file back afterwards, or the next deploy will turn it on again. flybridge has its own flag (FEATURE_ONSCREEN_CHAT) but the flysim switch is the one to reach for: it is the source end, and it also stops anything else that might be posting to the endpoint.

Check what the filters are doing:

curl -s 127.0.0.1:9101/metrics | grep fly_chat_
# fly_chat_accepted_total, fly_chat_ring_lines, and one
# fly_chat_rejected_total{reason="..."} series per rule (url, charset, deny_list, ...).

Chat dead, bridge active (flybridge EventSub)

The symptom: nothing the bot says appears in chat, !fly and friends do nothing, the on-screen CHAT panel stops filling — and systemctl status flybridge says active (running) with NRestarts=0. Measured once for real: the release container, 2026-09-16 11:45:34 to 12:47:13 UTC, an hour of it, ended by a hand-typed restart.

What the bridge now does by itself. It exits. If the channel.chat.message EventSub subscription has not been confirmed created for EVENTSUB_GRACE_MS (default 60 s) after any socket disconnect, create failure or revocation, it logs one FATAL: line and exits 75; flybridge.service has Restart=always, RestartSec=15 and no start limit, so it comes back 15 s later with a FRESH EventSub websocket transport — which is the only thing that clears Twitch's "number of websocket transports limit exceeded". Expect roughly one attempt per 75 s for as long as the cause lasts, one FATAL: line each, and at most one notice in chat per 10 minutes. So in the normal case there is nothing to do but read the journal and find out why.

First question: is it actually dead, or did it already heal?

ssh the host pct exec <release-ctid> -- curl -s 127.0.0.1:7410/health | python3 -m json.tool
# chatSubscriptionHealthy  false => chat IS dead right now. THIS is the field to read.
# eventSubConnected        was TRUE for the whole hour on 2026-09-16 — a connected socket is
#                          not a working subscription. Never conclude anything from it alone.
# eventSubListenerCount    1 when the bot and broadcaster roles are one Twitch account (today),
#                          2 once a separate bot account exists. A 2 on a one-account channel
#                          means the shared auth provider did not get wired: two transports for
#                          one account is the shape that caused the outage.
# lastSelfHealAtIso        non-null => this process is a restart after a self-heal.

Then the journal. Three lines matter:

ssh the host pct exec <release-ctid> -- journalctl -u flybridge -n 200 --no-pager

• flybridge: FATAL: the channel.chat.message EventSub subscription has not been confirmed for 60000 ms — the self-heal fired. The same line names every loss reason it saw and, when it was Twitch's transport limit, says so. This is the line to grep for.
• flybridge: EventSub ... socket for <id> disconnected: ... — the trigger (code 1006 on 2026-09-16).
• flybridge: EventSub subscription create failure for channel.chat.message...: Encountered HTTP status code 429 ... number of websocket transports limit exceeded — Twitch refusing the re-create because stale transports from this account have not been reaped. Time fixes this, and RestartSec=15 is the wait. Do NOT shorten it, and do not add a retry loop.

ssh the host pct exec <release-ctid> -- curl -s 127.0.0.1:7410/metrics | grep flybridge_eventsub
# ..._chat_subscription_confirmed_total  climbs once per successful (re-)create
# ..._chat_subscription_lost_total       climbs on every disconnect/failure/revocation
# ..._transport_limit_total              > 0 => the 429 above
# ..._subscription_create_failures_total all subscriptions, not just chat
# ..._reconnects_total                   socket drops

When to intervene, and with what.

What the journal says	What it means	Do
Repeating FATAL: + transport_limit for more than ~10 min	Something else is holding this account's websocket transports (a second bridge, a dev box, a stale tools/mock-twitch.sh)	Find and stop the other client. GET https://api.twitch.tv/helix/eventsub/subscriptions with the broadcaster token lists them; delete the strays.
Repeating FATAL: + create failure ... missing scope	The token lost a scope (re-authorized with the wrong --role, or Twitch revoked it)	Re-run tools/authorize.mts — see "Rotate the bot token" below. The bridge will NOT recover from this on its own; restarting cannot add a scope.
Repeating FATAL: + revoked (authorization_revoked)	The channel or the user revoked the app	Re-authorize. Same as above.
One FATAL: then a quiet, chatSubscriptionHealthy: true bridge	It healed.	Nothing. Note it in the status log if it was during a stream.
active (running), chatSubscriptionHealthy: false, and NO FATAL: line after more than 2 min	The watchdog itself is not running — an old bundle	Check /opt/fly/current/bridge/index.js is from a release that has it, then systemctl restart flybridge.

The manual hammer, still correct and still fast, is pct exec <release-ctid> -- systemctl restart flybridge. It costs nothing but the startup notice (suppressed if one went out in the last 10 minutes) and does not touch flysim, the page or the broadcast — fly.target only Wants= this unit, so chat can never take the fly down.

Rotate the bot token (flybridge)

Not yet applicable until flybridge exists and is wired up (P3, docs/design/infra.md section 7). Once it is: re-run tools/authorize.mts (docs/design/stage-bridge.md section B1) on the operator box to get a fresh refresh token, which RefreshingAuthProvider's onRefresh persists to /var/lib/fly/bridge/tokens.json automatically from then on — a one-time manual step, not a recurring one. The twitch-app credential (client id/secret) only needs rotating if the app itself is compromised; follow the same 06-secrets.sh pattern as the stream key.

Restore a checkpoint locally

flysim restores automatically on startup from manifest.json -> latest -> previous -> highest archived generation -> best milestone, in that order (docs/design/flysim.md section 8). To force a specific generation by hand:

pct exec <ctid> -- systemctl stop flysim.service
pct exec <ctid> -- cat /srv/fly/state/manifest.json   # note the generation you want
# Edit manifest.json's "latest" field to point at that generation (systemd-creds/pct
# push a corrected manifest.json, or edit in place with pct exec ... -- sh -c '...').
pct exec <ctid> -- systemctl start flysim.service
pct exec <ctid> -- curl -s http://127.0.0.1:7401/status | jq .checkpoint

If every candidate checkpoint fails validation, flysim exits non-zero and does not auto-reset (docs/design/flysim.md section 8: "no automatic fresh start, ever"). That is deliberate — a silent reset would be indistinguishable from real progress on stream. A deliberate reset means moving /srv/fly/state aside by hand.

Restore from the backup host

# On the host:
DATE=20260101   # the backup date you want
CTID=<release-ctid>
scp -r <backup-user>@<backup-host>:<backup-path>/fly-pokemon/$DATE/* <host-stage>/fly-restore/
pct exec $CTID -- systemctl stop flysim.service
pct push $CTID <host-stage>/fly-restore/manifest.json /srv/fly/state/manifest.json
pct push $CTID <host-stage>/fly-restore/<generation>.checkpoint /srv/fly/state/<generation>.checkpoint
pct exec $CTID -- systemctl start flysim.service

To restore into the spike CT instead (the P1 restore drill in docs/design/infra.md section 7, drill 6: "Restore from the backup host into the dev container and boot flysim on it. Assert rank and badges match."), point the pct push calls at the dev container and provision the spike with the corresponding <dev-env> GAME set to match, then compare GET /status's milestone.rank and game.badges against the source container's before the restore.

Cutting a release

The release container fly-pokemon is the release container (ROLE=release in <release-env>) — docs/stream-mvp-plan.md, "Release container" (the operator, 2026-09-16): "the release container runs TAGGED commits only". The dev container fly-spike (ROLE=dev in <dev-env>) is where everything else happens: builds, measurements, deploy trials, the VirtualGL experiment. Never develop or measure on The release container; never deploy an untagged or dirty tree there.

1. Tag, on a clean main, with every test suite green:

   # operator box
   infra/build/tag-release.sh v0.2.0
   

   tag-release.sh refuses to create the tag — and never pushes anything — if you are not on main, the tree is dirty, the tag already exists (locally or on the remote), or any of npm test, npm run typecheck, cargo test --workspace (in services/flysim), or infra/tests/lint.sh fails.

2. Package, from that exact tagged commit (fly-build CT or the operator box):

   infra/build/build-flysim.sh
   infra/build/package-release.sh v0.2.0 ./flysim path/to/stage/dist path/to/bridge/dist /path/to/out
   

   To build the flysim binary with the CUDA LIF backend compiled in, set FLY_CARGO_FEATURES (default empty — every release through v0.1.3 was built with it unset, and an unset build is byte-for-byte the command those used):

   FLY_CARGO_FEATURES=cuda infra/build/build-flysim.sh ./flysim
   

   That is the only supported value today. It needs no CUDA toolkit on the build box (the PTX is committed and embedded), and ldd on the result lists no CUDA library — cudarc is built with dynamic-loading, so libcuda is dlopened on first use and a run without FLY_LIF_CUDA=1 never opens the driver. The same binary is therefore deployable to a GPU-less container, and build-flysim.sh fails the build if a CUDA library ever shows up in the link. Because the GPU tick is bit-exact with the CPU one, --print-compatibility is byte-identical to a CPU-only build of the same tree, so a checkpoint carries over in both directions (measured: infra/docs/cuda-on-dev.md). Turning the backend on is a separate, per-container switch — FLY_LIF_CUDA=1 in the env file, see "CPU partition (cpuset)" below.

   The tarball's MANIFEST records provenance as #-prefixed lines ahead of the sha256 checksums (GNU sha256sum -c skips comment lines, so 05-deploy.sh's post-extraction verification still passes):

   # flybrain release MANIFEST
   # version=v0.2.0
   # git_tag=v0.2.0
   # git_commit=<full sha>
   <sha256>  flysim
   ...
   

3. Deploy to the release container, from a checkout that is ITSELF at that same clean tag — this is the check that matters, not the tarball's own MANIFEST:

   # on the host, from a checkout/worktree at v0.2.0:
   infra/05-deploy.sh <release-env> /path/to/out/flybrain-v0.2.0.tar.gz
   

   <release-env> has ROLE=release, so before touching the container at all 05-deploy.sh runs git describe --exact-match --tags and git status --porcelain against the tree it is itself running from (infra/lib/common.sh's require_release_tag) and refuses the entire deploy — not just the release-artifact step — if either check fails. The refusal (verified against a throwaway git repo by infra/tests/lint.sh) reads:

   05-deploy: ROLE=release refuses to deploy: source tree at <path> is not exactly at an annotated tag (git describe --exact-match --tags failed)
   

   or, for a tagged-but-dirty tree:

   05-deploy: ROLE=release refuses to deploy: source tree at <path> is not clean (git status --porcelain is non-empty)
   

   (both printed after die()'s own [<UTC timestamp>] FATAL: prefix). A tarball whose own version does not match the tree's tag is refused too, with a message naming both versions and the package-release.sh command to rebuild it correctly. On success, the release directory is named after the tag (/opt/fly/releases/v0.2.0/), and the last line printed is an claim-log-style deploy line — copy it into $AGENT_CLAIM_LOG by hand (the same the LAN convention as "claim the container ID" in infra/README.md):

   05-deploy: 2026-0X-XX: deployed fly-pokemon (the release container) role=release tag=v0.2.0 release=v0.2.0
   

   ROLE=dev (the dev container, <dev-env>) skips all of the above — any commit deploys, and the release directory is named after a short sha + timestamp instead of a tag, the historical behaviour, unchanged.

4. Verify: infra/verify.sh <release-env>.

5. Rollback: see "Roll back a release" immediately below. Because the release container's release directories are named after tags, "the previous release" and "the previous tag" are the same /opt/fly/releases/<tag>/ directory — ln -sfn it back and restart.

Roll back a release

pct exec <ctid> -- ls /opt/fly/releases/           # find the previous version
pct exec <ctid> -- sh -c 'ln -sfn /opt/fly/releases/<previous-version> /opt/fly/current.tmp && mv -T /opt/fly/current.tmp /opt/fly/current'
pct exec <ctid> -- systemctl restart flysim.service flystage-web.service flystage.service

One ln -sfn plus a restart, per docs/design/infra.md section 2. Nothing else needs touching: flycast/flypush/mediamtx do not read anything under /opt/fly/current. On the release container, <previous-version> is a tag (e.g. v0.1.0) — "Cutting a release" above.

CPU partition (cpuset)

05-deploy.sh derives the in-guest AllowedCPUs= drop-ins for every app unit (flysim, xvfb, flystage, flystage-web, flycast, pulse, mediamtx) from two env-file values, via lib/common.sh's cpuset_partition: CPUSET (whole physical cores reserved for the container, 01-create-ct.sh's lxc.cgroup2.cpuset.cpus) and RAYON_THREADS (flysim's Rayon pool size — also written as RAYON_NUM_THREADS into /etc/fly/fly.env, from the same RAYON_THREADS_EFFECTIVE value, so the two can never drift apart).

The split is three groups, not two:

1. flysim gets the first RAYON_THREADS cpus of CPUSET.
2. flycast gets the last ENCODER_CORES cpus (env var, default 2) of what remains.
3. Everything else — xvfb/flystage/flystage-web/pulse/mediamtx — shares whatever is left over in between.

flycast is split off into its own group, separate from the page/capture group, because of a live finding on the release container (2026-09-16): Chromium's compositor starved when it shared cores with the x264 encoder — 63% of captured frames came back unchanged across two fly-watchdog passes, against 2% on the NVENC dev box, where the encoder is off-CPU entirely. That 63% is the 73/115 half of a pair taken with the wrong instrument (a second x11grab of the display rather than the encoder output) and has not been re-measured — see "Capture freeze" above and the correction note in infra/docs/p0-measurements.md. The split itself stays: it is cheap and live on both containers, but do not cite the number as evidence until someone re-runs it. Before this fix, only flysim/xvfb/flystage/flycast got the drop-in at all (a two-way split); flystage-web/pulse/mediamtx ran on the container's full, unpartitioned cpuset.

X264_PRESET (env var, written into /etc/fly/fly.env as FLY_X264_PRESET, read by bin/flycast-launch's encoder_x264()) is the matching encoder-side knob: veryfast is the wire default, but the release container (2026-09-16, release box, x264 at 6000k) measured it contending within flycast's own (then two-cpu) ENCODER_CORES group and moved to superfast for headroom.

"Extra cores" (2026-09-16, later the same day): the first fix above still ran CPUSET at eight cpus total (RAYON_THREADS=4 / ENCODER_CORES=2, the function's default), and that eight-cpu set could not hold x264 + page/browser + sim all at once — the same Chromium-compositor starvation the three-way split exists to fix, just not fully fixed by three thin groups on eight cores. The operator chose to widen CPUSET to all ten of node 1's whole physical cores rather than shrink flysim or flycast, and set ENCODER_CORES=3 explicitly (<release-env>, CORES=10 to match). The release container's live values — which cpus, how many threads, how many encoder cores — are a record of one box, not a constant, and live in the operator's infra repo (services/flybrain/runbook-host.md). cpuset_partition CPUSET RAYON_THREADS ENCODER_CORES reproduces the split from those three numbers; infra/tests/lint.sh checks it against both the current shape and the earlier, narrower one.

FLY_LIF_CUDA — the LIF tick on the GPU, and what it does to the partition

FLY_LIF_CUDA=1 in an env file makes 05-deploy.sh write FLY_LIF_CUDA=1 into /etc/fly/fly.env, which is what flysim reads at startup to attach the CUDA LIF backend. Default 0, written for every container either way so the file says out loud which backend is configured. Two separate switches, deliberately:

	switch	where	effect
build	FLY_CARGO_FEATURES=cuda	build-flysim.sh	the backend is compiled in, libcuda still only dlopened
run	FLY_LIF_CUDA=1	the env file → /etc/fly/fly.env	the backend is attached

A binary without the feature ignores FLY_LIF_CUDA=1 silently, so confirm the backend from flysim's own log, not from the env file:

pct exec <ctid> -- journalctl -u flysim -b --no-pager | grep -i 'cuda\|lif backend'

It needs GPU=1 (the /dev/nvidia* device block plus the userspace driver in the container — docs/design/gpu.md sections 1 and 2) and the card the committed PTX targets, sm_75.

It changes RAYON_THREADS. With the sweep and the propagation on the card, flysim holds real time on about one host core, so the dev container (<dev-env>) runs RAYON_THREADS=2 instead of 3 — two rather than one because the pool still shards plasticity.observe. The freed cpu goes to the page group, which is the whole reason to do this: CPUSET=0,2,4,6,8,10,12,14 with RAYON_THREADS=2 gives flysim 0,2, xvfb/flystage/flystage-web/pulse/mediamtx 4,6,8,10 and flycast 12,14 (infra/tests/lint.sh checks that shape too). Set it back to 3 if you set FLY_LIF_CUDA=0 — two cpus is below real time for the CPU kernel.

Flipping backends does not cost the checkpoint: the GPU tick is bit-exact, so the compatibility string 05-deploy.sh gates on is unchanged and a container can move between the two across restarts in either direction. Measurements, the golden-suite run on the card, the 3 h soak and the restore drill: infra/docs/cuda-on-dev.md.

Capture freeze

The symptom: the stream looks alive — audio perfect, frame= advancing at 30 fps in /run/fly/flycast.progress, 0 dup / 0 drop, page healthy, display updating — and the picture only changes about once a second. It lasts for the life of the flycast ffmpeg process. Full findings, including the ten dev-box experiments and what is not established, in infra/docs/capture-freeze.md.

Measure it on the encoder output. Never on a second x11grab of :99 — the display is fine during this failure, so a display-side grab reports PASS on the very thing you are looking for (see that doc's section 4, and the correction note in infra/docs/p0-measurements.md).

# On the host. Newest segment, last 3 s: count frames identical to the one before them.
# 80+ of ~90 is frozen; a healthy stream reads 0-6.
pct exec <ctid> -- sh -c '
  seg=$(ls -1t /srv/fly/media/rec/*.ts | head -n1); echo "$seg";
  ffmpeg -hide_banner -sseof -4 -i "$seg" -t 3 \
    -vf "tblend=all_mode=difference,blackframe=amount=99.5:threshold=8" \
    -an -f null - 2>&1 | grep -c "Parsed_blackframe.*] frame:"'

# Or read what the watchdog already measured (every 5 minutes, check 9):
pct exec <ctid> -- grep fly_capture /var/lib/node_exporter/textfile/fly_watchdog.prom
pct exec <ctid> -- journalctl -t fly-watchdog -g 'capture freeze' --since -2h

fly_capture_identical_frames is the last probe's count (-1 before the first probe) and fly_capture_freeze_restarts_total counts the restarts check 9 issued.

Restart procedure. Restarting flycast alone clears it, and nothing else needs to move:

pct exec <ctid> -- systemctl restart flycast.service
# Under 10 s of stream gap (the wait-for-stage settle is 5 s of it, and it should pass
# immediately because the page is already up); the recording rolls to a new segment;
# flysim, the page and flypush are untouched. Then confirm with the probe (0-6 identical):
pct exec <ctid> -- journalctl -u flycast -n 30

The watchdog does this itself after two consecutive bad probes, at most once per 30 minutes, so if the counter is climbing do not also restart by hand — read journalctl -t fly-watchdog -g 'capture freeze' first.

If it comes back immediately after a restart, the ordering gate did not help and the freeze has a path this repo has not seen: check journalctl -u flycast -g wait-for-stage for what wait-for-stage decided, run pct exec <ctid> -- runuser -u fly -- env XDG_RUNTIME_DIR=/run/fly PULSE_SERVER=unix:/run/fly/pulse/native /opt/fly/bin/wait-for-stage --once for the three readiness checks by hand, and record it against infra/docs/capture-freeze.md section 5 rather than restarting in a loop.

Why the ordering gate exists. flycast.service is After=flystage.service and runs ExecStartPre=/opt/fly/bin/wait-for-stage 120, which returns only once the X display answers, a Chromium kiosk window is mapped on :99, the pulse sink stream has at least one sink-input, and all three have held for 5 s. A new pulse client attaching during ffmpeg's first one to three seconds is what triggers the freeze, and Chromium's audio stream at page load is exactly such a client. After 120 s the wait warns and starts the encoder anyway (a black-but-running stream beats no stream), so a flycast that starts while the page is missing is expected behaviour, not a bug — but it is the state the freeze likes, so check the probe afterwards.

Loop suspected

The symptom: everything is healthy and nothing is happening. The sim advances, the page draws, the stream encodes, the macros all report done — and the fly is pressing the same short cycle over the same two tiles, hour after hour, with the stuck-o-meter climbing. The live case was Viridian on 2026-09-17: GO NPC, GO OUT, NEXT, GO FRONTIER every ~3 brain seconds for 2 h 17 min on rung 8. Mechanism and audit in infra/docs/macros-traps.md.

The watchdog never acts on this. Check 10 detects and reports; a human or a review agent decides. It does not restart flysim, it does not press anything, it does not touch the game — a loop is a bug in what a macro's target choice considers, and a bounce would only restore the same loop with the stream interrupted for nothing. Do not "fix" it by restarting either.

# What check 10 measured, every 5 minutes.
pct exec <ctid> -- grep -E 'fly_loop|fly_places' /var/lib/node_exporter/textfile/fly_watchdog.prom
pct exec <ctid> -- journalctl -t fly-watchdog -g 'loop ' --since -6h
# The report, written on every probe, for a human or a review agent:
pct exec <ctid> -- cat /run/fly/wd/loop.json | jq .

series	reading
fly_loop_suspected	1 = a short cycle is repeating and no new ground was covered. 0 = looked, clear
fly_loop_period	length in macro labels of the repeating block (0 = nothing repeats, -1 = no probe yet)
fly_loop_repeats	how many times that block repeats at the end of the 10-brain-minute window
fly_loop_distinct_macros	distinct macro names started in the window
fly_places_delta	growth in game.uniqueLocations since the previous probe (-1 = no previous probe)

The flag needs both halves: at most 3 distinct macro names with the block repeating 20+ times, or one macro at 95%+ of the window — and no growth in the exploration count. A repeating macro over ground that keeps growing is a walk longer than the 600-frame cap, not a trap (macros-traps.md: GO FRONTIER x19 across 93 tiles), and the watchdog is deliberately quiet about it. The thresholds are WD_LOOP_* in infra/bin/fly-watchdog; the 3-name ceiling is narrower than the 4-macro cycle that was actually live, so a cycle of four or more names reports through fly_loop_period/fly_loop_repeats without raising the flag. Read the gauges, not just the flag.

What to do with it. Reproduce it off the stream, from the box's own state, and fix the macro — never the stream:

# 1. Pull the checkpoint the loop is in, before it rolls out of the manifest.
pct exec <ctid> -- cat /srv/fly/state/manifest.json | jq .        # note "latest"
pct pull <ctid> /srv/fly/state/<generation>.checkpoint \
    <host-stage>/<ctid>-loop-$(date -u +%Y%m%d%H%M).checkpoint
pct pull <ctid> /run/fly/wd/loop.json <host-stage>/<ctid>-loop.json
pct pull <ctid> /srv/fly/state/events.jsonl <host-stage>/<ctid>-events.jsonl
# ... then copy both onto the dev box, into .local/checkpoints/ (never committed).

# 2. Run the trap hunt over it on the dev box, which flags every two-brain-minute
#    window by tiles and by repeated sequence (infra/docs/macros-traps.md, "The trap hunt").
FLY_ROM="$HOME/fly-plays-pokemon/Pokemon Red (U) [S][BF].gb" FLY_MACRO_BRAIN=data/fafb-v783 \
  FLY_TRAP_CHECKPOINT=.local/checkpoints/release-loop-<stamp>.checkpoint \
  FLY_TRAP_MINUTES=20 cargo run --release -p flysim --example trap_hunt

The report's sequence names which macros to read first, milestone.rank/label says which rung's goals they were aiming at, and map says where. Add what you find as a row in infra/docs/macros-traps.md's audit table — including "left, and why" — and fix it as a change to a macro's target choice, never to a ranking, a prior or a fallback action (docs/design/macros.md section 12).

Check 10 goes quiet on its own, with one loop cleared line, as soon as the exploration count moves again. That line is how a deployed fix is confirmed from the outside.

Disk full

The three independent brakes, in the order they trigger (docs/design/infra.md section 8):

1. ZFS quota on <bulk-pool>/subvol-<ctid>-disk-1 (600G by default) — the filesystem itself refuses new writes once hit. This is "the one that actually protects the neighbours" (the neighbouring GPU container, the metrics container, another container on the host, another guest on the host on the same pool).
2. fly-retention.timer, hourly, prunes segments > 7 days, highlights > 90 days, orphan checkpoints.
3. fly-watchdog's disk guard (bin/fly-watchdog's check_disk): >85% triggers an immediate extra retention pass; >95% sets the fly_watchdog_disk_critical alarm metric and runs retention in --aggressive mode (1-day segment window instead of 7). Known gap: at 95% the design calls for also dropping the recording leg of flycast while keeping the stream up; this infra pass does not implement a no-recording variant of the fixed flycast.service command (see that unit's own header comment). If retention alone cannot get ahead of a runaway recorder, the manual fallback is pct exec <ctid> -- systemctl stop flycast.service (this also stops the stream — accept the outage over risking the pool) followed by manual cleanup of /srv/fly/media/rec, then systemctl start flycast.service.

Check current usage: pct exec <ctid> -- df -h /srv/fly/media /srv/fly/state.

GPU driver version lockstep

docs/design/gpu.md sections 2 and 8. The single permanent operational coupling the GPU work adds: the host's NVIDIA kernel modules and each container's NVIDIA userspace must be the same version string, today 580.76.05. The host's modules are a hand-patched build against 7.0.14-11-pve; the patch repo is the operator's own kernel-patch repo (per the operator's own GPU-driver notes, the stock sources do not build against 7.x). The container userspace comes from the stock <host-stage>/NVIDIA-Linux-x86_64-580.76.05.run, installed by 02-base.sh with --no-kernel-modules --silent --no-x-check.

NVIDIA_VERSION in the env file is the declared version, and verify.sh asserts three things agree: that value, the container's nvidia-smi, and the host's nvidia-smi.

What breaks, in what order, after a host driver or kernel upgrade

1. The guest's libcuda/libnvidia-encode refuse to talk to a mismatched kernel module. nvidia-smi inside the CT prints "Failed to initialize NVML: Driver/library version mismatch" and flycast cannot open an NVENC session. The automatic fallback in bin/flycast-launch catches this: the stream continues on libx264 at about 2 cores, fly_encoder_backend{backend="x264"} goes to 1, and fly-watchdog logs encoder degraded: and raises fly_watchdog_encoder_degraded. Without that fallback flycast would crash-loop and the stream would go black.

2. A kernel update rebuilds the modules via DKMS, but the stock sources do not build against 7.x, so fresh patches may be needed. Always check nvidia-smi on the host and in every guest after a kernel update — including the neighbouring GPU container, which shares this card.

3. A driver upgrade is an ordered, three-step operation. Do the spike CT first:

   # 1. on the host: patch + install the new kernel module (nvidia-kernel-patches)
   # 2. reboot the host, so fly-nvidia-majors.service rewrites the majors for the
   #    newly loaded module (the uvm major moves on essentially every boot)
   # 3. for each CT: bump NVIDIA_VERSION in env/<name>.env, then
   infra/02-base.sh <release-env>      # installs the matching userspace
   infra/verify.sh  <release-env>      # lockstep + char devices + majors
   

   Never upgrade the host driver without a window in which the streams may be on the x264 path. Two containers at ~2 cores each on a 2015 Haswell box is survivable but not free.

Symptom-to-cause, fastest first

Symptom	Look at
nvidia-smi in the CT: "Driver/library version mismatch"	host driver moved; re-run 02-base.sh (step 3 above)
nvidia-smi in the CT: "No devices were found"	passthrough, not the driver. ls -l /dev/nvidia* in the CT — a regular file where a character device should be means the bind's source was missing at container start
/dev/nvidia-uvm present but NVENC fails	stale majors. /usr/local/sbin/fly-nvidia-majors.sh <ctid>, then restart the container
fly_encoder_backend{backend="x264"} 1 with FLY_ENCODER=nvenc	the fallback fired. nvidia-smi --query-gpu=memory.used,utilization.encoder --format=csv — the GPU workload in the neighbouring container can hold 6-7 of the 8 GB
Everything fine, flycast still crash-looping	not the GPU. journalctl -u flycast -n 50; a non-session error is deliberately not faulted over to x264

The card is shared with the neighbouring GPU container (ml, the GPU workload in the neighbouring container), which belongs to the LLM work. Compute mode stays Default; never set nvidia-smi -c EXCLUSIVE_PROCESS, which would let whichever container got there first lock the card out from under the other.

The host reboot order

Fly containers have no special reboot ordering requirement relative to each other (onboot=1, startup order=4,up=60, per 01-create-ct.sh — they start after the neighbouring production service/order-lower services, with a 60s stagger). After any the host reboot:

There is one host-side ordering requirement, added by the GPU work (docs/design/gpu.md section 1). Two units must complete before pve-guests.service, because lxc.* keys are read only at container start and the nvidia-uvm major is allocated dynamically and moves on essentially every boot:

nvidia-devnodes.service   (already on the host: materialises /dev/nvidia*)
    -> fly-nvidia-majors.service   (rewrites the majors into /etc/pve/lxc/<id>.conf, one per fly guest plus the neighbouring GPU container)
        -> pve-guests.service      (starts the guests, which read those configs)

fly-nvidia-majors.service is Before=pve-guests.service and the fly CTs' startup order=4,up=60 keeps them behind it. If that unit is missing or disabled, the containers come up with a stale uvm major and no announcement: verify.sh's GPU section is what catches it. Install per infra/host/README.md.

Do not be alarmed by the shape of /etc/pve/lxc/<id>.conf after a boot. PVE re-emits that file on every lifecycle operation with its own keys sorted alphabetically, every comment hoisted to the top of the file and every raw lxc.* key moved to the end, so the # BEGIN fly-nvidia / # END fly-nvidia pair will be sitting at the top with nothing between it while the eight lines it generated sit at the bottom (measured on PVE 9.2.10, 2026-09-16). The passthrough is fine — the lxc.* lines are what LXC reads, and they are all there. fly-nvidia-majors.sh converges lines rather than bytes and still reports unchanged in that state. What would be a fault is the same script reporting changed on two consecutive runs.

After any the host reboot:

1. Confirm the neighbouring production container (the neighbouring production service) is healthy first — it is the priority guest on this host.
2. systemctl status fly-nvidia-majors.service on the host — want active (exited), status=0/SUCCESS. This is the check the operator's own GPU-driver notes call the silent one.
3. Confirm the fly containers came back: pct list, then infra/verify.sh <release-env> and <platformer-env>. The GPU section asserts every /dev/nvidia* is a character device, that the conf's majors still match /proc/devices, and that the driver versions are in lockstep.
4. flysim restores from its checkpoint automatically; expect a brief gap while wait-for-health gates flystage's Chromium start (up to 120s per flystage.service's ExecStartPre timeout).
5. If flypush was enabled before the reboot, confirm it reconnects to Twitch within a couple of minutes; if not, check journalctl -u flypush for an RTMP-level error before assuming a key problem.

Known gaps (recorded, not smuggled in as "done")

• No alerting path. Everything degrades to a Prometheus textfile metric and a journal line at err with the fly-watchdog: prefix. There is no page/SMS/Slack path from this host (docs/design/infra.md section 0 and section 8).
• 95% disk guard does not stop flycast's recording leg, only prunes harder — see "Disk full" above.
• fly-recap's event/segment schema is inferred, not contract-specified. Its header comment explains exactly which fields are assumed; update it once services/flysim actually ships events.jsonl/segments.csv.
• flybridge.service's ExecStart path is a placeholder (services/bridge does not exist yet). It will fail to start until that package ships, which is expected under Wants= semantics and does not block fly.target.
• /dev/shm resize via a dev-shm.mount drop-in is inert — confirmed on the P0 spike (2026-09-15), no longer a maybe. An unprivileged Debian 13 LXC on PVE 9 has no dev-shm.mount unit at all (LXC mounts /dev/shm itself before systemd starts), so the drop-in does nothing. Two consequences. First, /dev/shm comes up at 94.4G (half the host's RAM), not the small tmpfs the design feared, so Chromium will not run out of space there; the residual risk is the reverse, a runaway charging 94G of tmpfs against an 8G container limit and being OOM-killed rather than getting ENOSPC. Second, it cannot be fixed from inside the guest either: the mount carries uid=100000 from the unprivileged id-map and mount -o remount,size=1G /dev/shm as root in the guest fails with tmpfs: Invalid uid '100000'. The only real fix is a host-side lxc.mount.entry in /etc/pve/lxc/<ctid>.conf — an the operator by-hand step, which nothing reachable through pct exec can do. verify.sh's size check should be read as "is it at least 1G", not "is it exactly 1G".

P0 spike checklist

Run the ten measurements in docs/design/infra.md section 4 against the dev container (<dev-env>) and fill in infra/docs/p0-measurements.md. Do not skip ahead to P1 provisioning of the real containers until that file's go/no-go line is checked GO.

infra/provision.sh <dev-env>
# ... run the measurements by hand against the dev container (P0 is manual instrumentation,
# not scripted — see docs/design/infra.md section 4 for exactly what to run) ...
# when done and no longer needed:
pct stop <dev-ctid> && pct destroy <dev-ctid>