> Design document produced 2026-09-15 by a planning agent. Binding contracts are ../feed-protocol.md and ../control-api.md; where this document differs, the contracts win. # Two 24/7 Twitch demo containers on the host: infrastructure and rollout Research only. Nothing on the host was touched, no files written. Every address, CT ID and free-space claim below is marked VERIFY where it rests on the infra repo's docs rather than on a live `pct list`. ## 0. What the operator's existing infrastructure constrains - The host = PVE 9.2.10, ``, 2x Xeon E5-2660 v3 (20C/40T, AVX2, no AVX-512), 188 GB RAM. An SSD mirror backs `local-zfs` with something over 150 GB free, and a wide raidz2 array (tens of TiB usable) is registered with pvesm for `images,rootdir`. The exact devices, sizes and free space are in the operator's infra repo; treat any figure here as indicative. - ZFS ARC is pinned at 6 GB in `/etc/modprobe.d/zfs-arc.conf`. Do not raise it. - The LAN convention is DHCP plus a router-side reservation keyed on MAC, not in-guest static. The two guests that ignore that convention (two unrelated guests) are both documented as hazards, and one of them caused a live address collision with `metrics`. Follow the convention. - The reserved range is already densely populated — around a dozen holders, plus the hosts themselves, the management interfaces and the switch. Read the reservation table in the operator's infra repo before asking for an address, and ask for one by MAC. - The other containers on the host have no direct SSH; the neighbours' own deploy scripts scp to the host and then `pct push`. Mirror that: the fly containers get no sshd, all ops go through `pct exec`. - Provisioning on these hosts has always been shell (`deploy-prod.sh`, `deploy.sh`, a hand-installed backup script, a burn-in directory, a migration directory). There is no Ansible anywhere in the operator's repos, and the only `pct create` in those docs is one mention. The closest template is a neighbouring container running a comparable self-hosted web service: Debian 13 unprivileged, 4c/6G/1G swap, 40G on `bulk-array`, `onboot=1`, `features: nesting=1,keyctl=1`. - There is no alerting path at all. `the infra repo's panels page` "Known gaps" says Alertmanager and Grafana are unbuilt and postfix on the metrics container is loopback-only. Plan for "the panel tile is the alarm", and record the gap. ## 1. Container spec Two containers, identical except for an env file. | | fly-pokemon | fly-platformer | |---|---|---| | CT ID | 150 (VERIFY `pct list`) | 151 (VERIFY) | | hostname | `fly-pokemon` | `fly-platformer` | | IP | `` (VERIFY free: the router's lease table + `arping`) | `` (VERIFY) | | Twitch | channel 1 | channel 2 | Hostname note: `fly-platformer` rather than `fly-mario`, because the ROM choice is still open between Super Mario Land and Kirby's Dream Land (`docs/streaming-plan.md` section 6) and the container name should survive that decision. `streaming-plan.md` and the proposed `pass` entries already say `fly-platformer`. ``` pct create local:vztmpl/debian-13-standard_13.6-1_amd64.tar.zst \ --hostname fly-pokemon --ostype debian --unprivileged 1 \ --cores 8 --memory 8192 --swap 2048 \ --features nesting=1 \ --rootfs local-zfs:24 \ --mp0 local-zfs:16,mp=/srv/fly/state \ --mp1 bulk-array:600,mp=/srv/fly/media \ --net0 name=eth0,bridge=vmbr0,ip=dhcp \ --onboot 1 --startup order=4,up=60 \ --description "fly demo: flysim/flystage/flybridge/flycast (flybrain/infra)" ``` Why `nesting=1`: modern Chromium has no setuid sandbox, it uses the namespace sandbox, which means the zygote calls `clone(CLONE_NEWUSER|CLONE_NEWPID|CLONE_NEWNS)`. An unprivileged LXC runs under the PVE `generated` AppArmor profile, which blocks nested user and mount namespace creation; without nesting the zygote dies with "Failed to move to new namespace: Operation not permitted" and Chromium never paints. `nesting=1` switches to the nesting AppArmor profile and permits it. One-line precheck inside the CT as the `fly` user: `unshare --user --pid true` must exit 0. Fallback if it cannot be enabled: `--no-sandbox`, acceptable in principle because the page is local content from `127.0.0.1` with no navigation surface, but it removes the renderer's defence in depth for a process that runs for weeks, so treat it as a temporary workaround and not the design. `keyctl=1` is not needed (no Docker here); add only if something complains. Cores and memory: 8/8192 as decided. Two containers take 16 of 40 threads and 16 GB of the 60 GB available, which leaves the box at well under half. Swap 2048 so Chromium's cold pages page out instead of triggering the kernel OOM killer (VERIFY the host actually has host swap; if not, drop swap to 0 and rely on the per-unit `MemoryMax` below). > **2026-09-15, VERIFIED, and the answer is no: the host's swap is 0.** Read live from > The host during the GPU probe (`docs/design/gpu.md` section 0). A container swap > allocation has no backing store on a host with no swap, so `--swap 2048` bought exactly > nothing — the paragraph above is wrong on its own terms and its parenthetical resolves > to "drop swap to 0". the env files now set `SWAP_MB=0`, and the per-unit `MemoryMax` > values plus `fly-watchdog` are the entire OOM story: a Chromium leak hits > `MemoryMax=3G` on `flystage`, gets killed there, and `flysim` (`4G`, `Nice=-5`) keeps > playing. Also affects section 8's risk list, which assumed the swap cushion existed. > The `cpuset` pinning added by the GPU work (`docs/design/gpu.md` section 1) changes the > "16 of 40 threads" arithmetic too: the two containers now take named whole physical > cores on NUMA node 0 rather than whatever PVE picked. Per-unit memory caps matter more than the container total: `MemoryMax=4G` on flysim, `3G` on flystage, `512M` on flycast. A Chromium leak then kills flystage, the watchdog restarts it, and flysim keeps playing. Without caps the kernel picks the biggest RSS, which is usually Chromium but not always. `/dev/shm` in an LXC defaults to a small tmpfs and Chromium's renderer will fall over on it. Fix it properly with a tmpfs mount unit sizing `/dev/shm` at 1 GB rather than using `--disable-dev-shm-usage`, which just moves the same traffic onto the rootfs. ### Storage sizing Rolling recording at 720p30, 3000 kbps video plus 160 kbps AAC: - 3160 kbit/s = 395 kB/s = 1.42 GB/h - 34.1 GB/day, 239 GB for 7 days, about 244 GB with MPEG-TS overhead - plus 90 days of daily recaps (5 min at ~6 Mbps, ~225 MB each) = ~20 GB - plus a checkpoint archive on the array `mp1 = bulk-array:600` mounted `/srv/fly/media` covers 7 days with 2x headroom. Set on the dataset (host side): ``` zfs set quota=600G recordsize=1M /subvol--disk-1 ``` The quota is the single most important safety measure in this plan. Guests on the bulk array include the neighbouring GPU container, the metrics container, another container on the host and another guest on the host; a runaway recorder with no quota takes the monitoring stack and another project's NFS share down with it. `recordsize=1M` suits 10-minute sequential video files. Keep `compression=lz4` (its early abort makes incompressible video nearly free) rather than turning compression off. `mp0 = local-zfs:16` mounted `/srv/fly/state` holds checkpoints, the event log, `status.json` and the segment index. It is on the SSD mirror because checkpoint commits are fsync-latency-sensitive, and it is a separate dataset so it can be snapshotted and backed up on its own and carries its own quota. **2026-09-15 note:** the resolution/bitrate decision above changed (`docs/streaming-plan.md` section 3's dated note) — the broadcast canvas is now native 1920x1080 and the Twitch target is 1080p30 at 6000 kbps CBR, not the 720p30/3000 kbps this section's sizing assumed. Recomputed at 6160 kbit/s (6000 video + 160 audio): 770 kB/s, 2.77 GB/h, about 66.5 GB/day, **about 465 GB for 7 days** — roughly 1.9x the 244 GB figure above. The 600G `mp1` quota still clears that with headroom, but not the same 2x margin the original sizing intended; **recommend the bulk-array quota become 900G** to restore comparable headroom (the 90-day recap and checkpoint-archive figures above are unaffected — recaps and checkpoints don't scale with live-stream resolution). Left as a note rather than rewriting the paragraph above, which records the original decision. ### A finding that changes the checkpoint design `docs/server-sessions.md` in the prototype commits a checkpoint every 5 s with fsync plus atomic rename. Estimate the envelope: 139,255 neurons of state (~3 MB), the plastic KC to MBON weight and eligibility arrays (~1 to 4 MB), the binjgb save state (~0.2 MB), so 5 to 15 MB per generation. At 10 MB every 5 s that is 173 GB/day of writes to the SSD pool. A 512G consumer SATA mirror is rated around 200 TBW, which is under a year of life, and that mirror also hosts another service's production workload. Do not do that. Recommended instead: 1. Hot state ring in tmpfs at `/run/fly/state` (size-capped, 64 MB), written as often as the sim likes. This is what the watchdog reads for freshness. 2. Durable checkpoint to `/srv/fly/state` every 300 s, plus one on every milestone rank-up and one on clean shutdown. 3. Result: roughly 3 GB/day of SSD writes, and a crash costs at most 5 minutes of play, which is irrelevant for a 24/7 stream. P0 must measure the real envelope size before this is finalised. If it turns out to be under 1 MB the 5 s cadence is fine and this becomes a non-issue. ## 2. Provisioning: shell, under `~/flybrain/infra/` Shell, not Ansible. Reasons: the operator has never used Ansible and the host has no control node or inventory; `pct` is not idempotent in a way Ansible's lxc modules improve on, so the useful idempotency is check-then-act either way; the whole thing is roughly 400 lines; and adding a new dependency to the one host that runs another service's production workload is a poor trade. The cost of shell is no drift detection, so the plan includes an explicit `verify.sh` that asserts desired state and exits non-zero, which is the piece people usually skip. ``` ~/flybrain/infra/ README.md provision.sh # runs 01..07 for one env file, resumable verify.sh # asserts desired state, exit 1 on drift lib/common.sh # log/die/on_pve_host/ct_exec/ct_push/converge_file/need 01-create-ct.sh # pct create + pct set, guarded on `pct config` 02-base.sh # apt, /dev/shm, journald caps, fly user, dirs, tmpfiles 03-node.sh # pinned node 22 tarball from nodejs.org + sha256 04-mediamtx.sh # pinned release tarball + sha256 + /etc/mediamtx.yml 05-deploy.sh # flysim binary, flystage bundle, flybridge, units 06-secrets.sh # pass -> systemd-creds, root 0600 07-enable.sh # enable/start in order, health gates units/*.service units/*.timer units/fly.target config/mediamtx.yml config/pulse.pa config/chromium-flags config/journald.conf bin/fly-watchdog bin/fly-recap bin/fly-backup-stage bin/fly-retention bin/flypush bin/wait-for-health bin/wait-for-x bin/wait-for-stage docs/runbook.md ``` Idempotency rules: `set -euo pipefail` everywhere; `pct config $CTID >/dev/null 2>&1 || pct create ...`; `apt-get install -y` is naturally converging; config files are shipped with `converge_file` which compares sha256, pushes only on difference, and signals whether a `daemon-reload` plus restart is needed. Running `provision.sh` twice in a row must produce no restarts the second time, and that is a test in `verify.sh`. Packages: `xvfb x11-utils x11-xserver-utils xauth chromium pulseaudio pulseaudio-utils ffmpeg fonts-dejavu-core fonts-liberation2 fonts-noto-core fonts-noto-color-emoji fontconfig prometheus-node-exporter curl ca-certificates jq rsync sysstat procps zstd`. No sshd. `apt-mark hold chromium ffmpeg` so an unattended security upgrade cannot swap the two components most likely to break the stream; upgrade them deliberately on the spike CT first. PulseAudio, not pipewire-pulse. The null-sink plus monitor pattern is exactly what PulseAudio does natively, it configures from a single `--file=` script, and it needs no dbus session or wireplumber. pipewire in a container with no logind session is the more fragile of the two, for no gain here. Node 22 as a pinned tarball from nodejs.org into `/opt/node-v22.x`, verified by sha256, symlinked into `/usr/local/bin`. Hermetic, no third-party apt repo, matches the the operator's posture of pinning artifacts. Debian 13 ships Node 20, so the distro package is not an option; NodeSource works but adds a repo whose trixie support is another thing to monitor. Rust never lands in prod. Build `flysim` on a dedicated throwaway build container, The throwaway build container `fly-build` (Debian 13, 16 cores, `onboot=0`, destroyable), not on the WSL box. Reason: glibc. Building against a newer glibc than Debian 13 trixie's and running on trixie fails at load, and the build box being the same template on the same host removes that class of problem entirely. Also, and this is the sharp edge: ``` RUSTFLAGS="-C target-cpu=haswell" # correct: E5-2660 v3 is Haswell RUSTFLAGS="-C target-cpu=native" # WRONG off-host: Zen3 emits instructions # Haswell does not have, illegal instruction at run time ``` Reject `x86_64-unknown-linux-musl` static linking: musl's allocator is materially slower under the allocation pattern of a per-millisecond neuron sweep, and real-time factor is the whole ballgame. Ship a dynamically linked glibc binary built on the matching template. Releases go to `/opt/fly/releases//` with `/opt/fly/current` as the symlink, so a rollback is one `ln -sfn` plus a restart. sha256 of every artifact recorded in `/opt/fly/releases//MANIFEST`. `fly` user: `adduser --system --group --home /var/lib/fly --shell /usr/sbin/nologin fly`. Directories via `/etc/tmpfiles.d/fly.conf`: ``` d /run/fly 0750 fly fly - d /run/fly/pulse 0750 fly fly - d /run/fly/state 0750 fly fly - d /run/fly/wd 0750 fly fly - d /var/lib/fly 0750 fly fly - d /var/lib/fly/chrome 0700 fly fly - d /srv/fly/state 0750 fly fly - d /srv/fly/media/rec 0750 fly fly - d /srv/fly/media/highlights 0750 fly fly - d /etc/fly 0755 root root - d /etc/fly/creds 0700 root root - ``` ## 3. systemd units One `fly.target` declares the whole set so the runbook has a single verb and ordering lives in one place. All units `After=network-online.target`, all app units `User=fly`. ``` fly.target xvfb.service Xvfb :99 pulse.service null sink "stream" mediamtx.service local RTMP/HLS/WebRTC ingest flysim.service Type=notify, WatchdogSec=30 -> flystage.service waits on flysim /health -> flycast.service encode once, tee to mediamtx + segments -> flypush.service copy-only remux to Twitch (disabled in local mode) flybridge.service Wants, never Requires (chat must not be able to stop the sim) ``` **flycast's ordering is load-bearing, and `After=` only (2026-09-16).** The arrow from flystage to flycast above used to be a diagram convention — flycast's `After=`/`Requires=` named `xvfb` and `pulse` but not `flystage`, so an `xvfb` restart co-started the page and the encoder. That race is the capture freeze: a new PulseAudio client attaching to the null sink during roughly the first one to three seconds of flycast's ffmpeg leaves its x11grab leg permanently starved (about one new picture a second, 29 repeats, audio perfect, 0 dup / 0 drop, `frame=` advancing at 30 fps), and Chromium's audio stream at page load is exactly such a client. Measured 3 h 50 min of frozen broadcast on the release container and reproduced six times on the dev container; `infra/docs/capture-freeze.md` has the experiments. So `flycast.service` now carries `After=xvfb.service pulse.service mediamtx.service flystage.service` plus `ExecStartPre=/opt/fly/bin/wait-for-stage 120` (X answers, a Chromium kiosk window mapped on `:99`, the `stream` sink has a sink-input, all three held for 5 s; after 120 s it warns and starts anyway) and `TimeoutStartSec=180`, since two ExecStartPre waits totalling 150 s do not fit systemd's default 90 s start timeout. `flystage.service` stays out of flycast's `Requires=` deliberately: ordering is what the fix needs, and a `Requires=` would let a dead or restarting page take the encoder, the recording and the broadcast down with it — the opposite of "the browser is display only; the sim is a service" in section 7's decisions. ### The one structural change to the plan in `piped-noodling-valley.md` Split `flycast` into two units. `flycast` encodes once and tees only to local sinks (MediaMTX and the segment recorder). `flypush` is a pure `-c copy` remux from `rtmp://127.0.0.1:1935/live/fly` to Twitch. Reasons: 1. A Twitch outage or backpressure can no longer stall the encoder or the local recording. With a single unit, a blocked RTMP write blocks the muxer loop for every tee leg (`-use_fifo 1` mitigates it, does not eliminate it). 2. The 23 h restart then bounces a copy-only process, so there is no re-encode hiccup, the local recording is continuous across the seam, and the recap never sees a gap. 3. "Flip to Twitch" becomes "enable one unit", with nothing else restarting. 4. Cost is about 0.05 core. The single-unit variant stays documented in the runbook as the simpler fallback. ### xvfb.service ``` ExecStart=/usr/bin/Xvfb :99 -screen 0 1280x720x24 -nolisten tcp -noreset -dpi 96 \ +extension RANDR -extension GLX Restart=always RestartSec=2 ``` Depth 24 so the framebuffer is plain RGB, which converts cleanly to `yuv420p`. Keep MIT-SHM (on by default): `x11grab` uses XShm for fast frame reads and losing it costs real CPU. `-extension GLX` is deliberate: decision 3 in the plan puts the brain map on a 2D canvas precisely to avoid SwiftShader WebGL, so nothing needs server-side GLX, and removing it stops Chromium probing a software GLX stack it should not use. If WebGL ever comes back, the answer is `--use-gl=angle --use-angle=swiftshader --enable-unsafe-swiftshader` plus re-enabling GLX, and a fresh CPU measurement. P0 tests both settings. ### pulse.service ``` User=fly Environment=XDG_RUNTIME_DIR=/run/fly PULSE_RUNTIME_PATH=/run/fly/pulse ExecStart=/usr/bin/pulseaudio -n --file=/etc/fly/pulse.pa --exit-idle-time=-1 \ --disallow-exit --log-target=journal Restart=always RestartSec=2 ``` No `--system` mode (upstream discourages it and it disables per-user modules) and no logind or `enable-linger` dependency. A plain system unit running as `fly` with an explicit `PULSE_RUNTIME_PATH` under `/run/fly` sidesteps the whole "container has no session" problem: the socket is at a fixed path both Chromium and ffmpeg can be pointed at. `/etc/fly/pulse.pa`: ``` load-module module-null-sink sink_name=stream rate=48000 channels=2 \ sink_properties=device.description=stream set-default-sink stream load-module module-native-protocol-unix socket=/run/fly/pulse/native ``` `-n` means no default script, so `module-udev-detect`, `module-console-kit`, `module-systemd-login` and bluetooth are never loaded, all of which fail noisily in a container. Critically, `module-suspend-on-idle` is NOT loaded: if the null sink suspends while the page is silent, ffmpeg's pulse input stalls and you get audio gaps or a hard desync at the next sound. Verify with `pactl list short sinks` showing `stream` as `RUNNING` while Chromium plays, and `IDLE` never becoming `SUSPENDED`. 48 kHz throughout, matching AAC output, so nothing resamples. ### flysim.service ``` Type=notify NotifyAccess=main WatchdogSec=30 Environment=FLY_GAME=${GAME} FLY_ROM=/srv/fly/rom/${ROM_SHA256}.gb Environment=FLY_STATE_HOT=/run/fly/state FLY_STATE=/srv/fly/state Environment=FLY_CONTROL_ADDR=127.0.0.1:7380 FLY_METRICS_ADDR=0.0.0.0:9101 Environment=RAYON_NUM_THREADS=6 Restart=always RestartSec=2 StartLimitIntervalSec=300 StartLimitBurst=5 Nice=-5 CPUWeight=400 MemoryMax=4G ``` `sd_notify` is worth the small effort: it is one `sendmsg` to `$NOTIFY_SOCKET`, no crate needed. The important detail is that `WATCHDOG=1` must be sent from inside the simulation loop, not from a helper thread, because then a stalled sweep actually trips the watchdog. `WatchdogSec=30` against a 30 Hz loop is generous enough to survive a checkpoint fsync. If the Rust side is not ready at P1, fall back to `Type=exec` and let the external watchdog cover it on checkpoint freshness; add `Type=notify` before P2. Two listeners, one process, and the separation is deliberate: the control API (`/stimulate`, `/reward`, `/pause`, `/checkpoint`, and explicitly no button endpoint) binds `127.0.0.1` only, while a read-only `/metrics` plus `/status.json` binds the CT address on 9101 so Prometheus on the metrics container can reach it. No mutating route is served on the LAN listener. `RAYON_NUM_THREADS=6` leaves headroom for the encoder; P0 finds the knee. ### flystage.service ``` After=xvfb.service pulse.service flysim.service Requires=xvfb.service pulse.service Environment=DISPLAY=:99 PULSE_SERVER=unix:/run/fly/pulse/native ExecStartPre=/opt/fly/bin/wait-for-x :99 30 ExecStartPre=/opt/fly/bin/wait-for-health http://127.0.0.1:7380/health 120 ExecStart=/usr/bin/chromium $(cat /etc/fly/chromium-flags | tr '\n' ' ') http://127.0.0.1:7380/stage Restart=always RestartSec=5 MemoryMax=3G ``` Flags, with the reason each one is there: | Flag | Why | |---|---| | `--kiosk --window-position=0,0 --window-size=1280,720` | one window filling the root, no browser chrome in frame | | `--user-data-dir=/var/lib/fly/chrome` | writable profile outside a nologin home, survives restarts | | `--no-first-run --no-default-browser-check --disable-search-engine-choice-screen` | no first-run UI on stream | | `--noerrdialogs --disable-session-crashed-bubble --disable-infobars --hide-scrollbars` | nothing modal can appear over the broadcast | | `--autoplay-policy=no-user-gesture-required` | page-played audio has no click to wait for | | `--disable-background-timer-throttling` | Chromium throttles background timers to ~1 Hz | | `--disable-backgrounding-occluded-windows` | an occluded window must not be treated as background | | `--disable-renderer-backgrounding` | keep renderer priority | | `--disable-ipc-flooding-protection` | a 30 Hz feed plus canvas draws exceeds the default 10/s/frame cap | | `--disable-gpu --disable-software-rasterizer` | 2D canvas only, Skia CPU raster, no SwiftShader cost | | `--force-device-scale-factor=1 --force-color-profile=srgb` | deterministic pixels into x264 | | `--disable-lcd-text` | subpixel antialiasing becomes colour fringing after 4:2:0 subsampling | | `--password-store=basic --use-mock-keychain` | no gnome-keyring in the container | | `--disable-features=Translate,MediaRouter,OptimizationHints,CalculateNativeWinOcclusion` | fewer background subsystems, and occlusion calculation is meaningless on Xvfb | | `--remote-debugging-port=9222` (bind 127.0.0.1) | needed for the P0 measurements and the CDP fallback watchdog | Rejected, with reasons, because the brief asked: `--headless` (see `streaming-plan.md` section 2b, screencast has no rate guarantee and burns cores on JPEG round-trips); `--disable-gpu-vsync` and `--disable-frame-rate-limit` (they uncap paint rate, which burns cores for nothing when ffmpeg owns the output clock at 30 fps, and make CPU unpredictable); `--enable-unsafe-swiftshader` (only if WebGL returns); `--mute-audio` (the whole point is page-played audio); `--disable-dev-shm-usage` (fix `/dev/shm` instead). Also set `/etc/fonts/local.conf` to grayscale antialiasing to match `--disable-lcd-text`. ### flycast.service (encode once, local sinks only) The unit as shipped also carries `After=... flystage.service`, `ExecStartPre=/opt/fly/bin/wait-for-stage 120` and `TimeoutStartSec=180` — the capture-freeze ordering gate described above. The ffmpeg command below is unchanged by that fix (beyond the 1080p/6000k and `fps=30:round=near` / `-fps_mode:v cfr` notes already recorded elsewhere): `-analyzeduration`/`-probesize` were considered and rejected, because ffmpeg's documentation of those two demuxer-probing options does not support an argument about a device input's first seconds, and the mechanism is not established (`infra/docs/capture-freeze.md` section 2). ``` ExecStartPre=/opt/fly/bin/wait-for-x :99 30 ExecStartPre=/opt/fly/bin/wait-for-stage 120 ExecStart=/usr/bin/ffmpeg -nostdin -loglevel warning -nostats \ -thread_queue_size 1024 -f x11grab -draw_mouse 0 -framerate 30 -video_size 1280x720 -i :99.0+0,0 \ -thread_queue_size 1024 -f pulse -name flycast -sample_rate 48000 -channels 2 -i stream.monitor \ -filter_complex "[0:v]format=yuv420p[v];[1:a]aresample=async=1:min_hard_comp=0.100:first_pts=0[a]" \ -map "[v]" -map "[a]" \ -c:v libx264 -preset veryfast -profile:v high -level 4.1 \ -b:v 3000k -minrate 3000k -maxrate 3000k -bufsize 6000k \ -g 60 -keyint_min 60 -sc_threshold 0 -r 30 -bf 2 -x264-params "nal-hrd=cbr:filler=1" \ -c:a aac -b:a 160k -ar 48000 -ac 2 \ -progress /run/fly/flycast.progress -stats_period 5 \ -f tee -use_fifo 1 \ -fifo_options "queue_size=120:drop_pkts_on_overflow=1:attempt_recovery=1:recovery_wait_time=1" \ "[f=flv:onfail=ignore]rtmp://127.0.0.1:1935/live/fly|[f=segment:segment_format=mpegts:segment_time=600:strftime=1:reset_timestamps=1:segment_list=/srv/fly/state/segments.csv:segment_list_type=csv:segment_list_flags=+live:segment_list_size=0]/srv/fly/media/rec/%Y%m%d-%H%M%S.ts" ExecStartPost=/bin/sh -c 'date -u +%%s > /srv/fly/state/flycast-start' Restart=always RestartSec=5 CPUWeight=100 MemoryMax=512M ``` Decisions inside that command worth stating: - No `-tune zerolatency`, despite the question in the brief. It disables lookahead and B-frames and therefore needs more bitrate for the same quality. We are not latency bound (a few seconds of glass-to-glass is fine for chat interaction), and 3000 kbps on a text-heavy 720p frame needs the efficiency. `-tune stillimage` is also rejected: it raises deblocking and smears 1 px UI edges. No tune, `-bf 2`, and `nal-hrd=cbr:filler=1` for true CBR, which Twitch prefers. - MPEG-TS segments, not MP4. A `.ts` file killed mid-write is still playable and concat-friendly; a truncated MP4 has no moov atom and is garbage. This matters because every ffmpeg restart truncates the open segment. - `-use_fifo 1` on the tee muxer is the specific fix for one slow leg blocking the others. - `segment_list` as CSV gives the recap script an exact segment-to-time index instead of parsing filenames and hoping. `flycast-start` records the wallclock epoch of stream start so offsets are unambiguous. - `aresample=async=1` is the long-run A/V drift fix. x11grab and pulse are two independent clocks; over hours the pulse leg drifts and the fix is to let the resampler stretch rather than to accumulate. ### flypush.service ``` After=mediamtx.service flycast.service LoadCredentialEncrypted=twitch-key:/etc/fly/creds/twitch-key.cred EnvironmentFile=/etc/fly/flypush.env ExecStart=/opt/fly/bin/flypush Restart=always RestartSec=15 StartLimitIntervalSec=0 ``` `/opt/fly/bin/flypush` reads `$CREDENTIALS_DIRECTORY/twitch-key`, builds `rtmps://ingest.global-contribute.live-video.net/app/$KEY`, and execs: ``` exec ffmpeg -nostdin -loglevel warning -nostats -rw_timeout 5000000 \ -i rtmp://127.0.0.1:1935/live/fly -c copy -f flv \ -progress /run/fly/flypush.progress "$URL" ``` P0 must confirm `ffmpeg -protocols | grep rtmps` on the Debian 13 build (native TLS RTMP, not librtmp). `StartLimitIntervalSec=0` because a multi-hour Twitch outage must not permanently defeat the restart logic. ### Timers | Unit | Schedule | Does | |---|---|---| | `flypush-restart.timer` | `OnUnitActiveSec=23h`, `RandomizedDelaySec=30m`, `AccuracySec=1m` | `systemctl restart flypush`, the 48 h guard | | `fly-recap.timer` | `OnCalendar=*-*-* 04:20`, `Persistent=true`, `RandomizedDelaySec=10m` | cut yesterday's highlights | | `fly-retention.timer` | `OnUnitActiveSec=1h` | prune segments > 7 d, highlights > 90 d, checkpoints per policy | | `fly-watchdog.timer` | `OnBootSec=2min`, `OnUnitActiveSec=60s`, `AccuracySec=5s` | the health loop below | | `fly-backup.timer` | on the host, not in the CT, 02:50 daily | stage and rsync to the backup host | `RandomizedDelaySec=30m` on the restart timer keeps the two channels from reconnecting at the same instant. Retention runs hourly rather than daily on purpose: a runaway recorder must not get 24 hours of rope. `fly-recap` reads `/srv/fly/state/events.jsonl` for yesterday (boundary computed in `America/New_York` because that is the audience's yesterday, while every filename and event timestamp is UTC, which is a real source of off-by-one bugs), selects milestone rank-ups, badges, first-visit areas, deaths and sugar redemptions, maps each timestamp to `(segment file, offset)` via `segments.csv` plus `flycast-start`, cuts each clip with `-ss/-t -c copy` (2 s GOP means 2 s granularity, which is fine), concatenates with `-f concat`, and re-encodes only the final few minutes. `Nice=19`, `IOSchedulingClass=idle`, `CPUWeight=20` so it can never starve the live stream. ### fly-watchdog, 60 s Checks in order, each with its own remediation, restarting only the failed unit: 1. flysim: `/health` returns 200 and hot-state mtime age < 30 s. Else restart flysim. 2. flystage: flysim's `frames_sent_total` and `feed_clients` from `/status.json`. Clients zero, or the counter flat across two passes, means the page is dead or frozen even though Chromium is alive. Restart flystage. This is the check that catches the failure mode a process-liveness check cannot see. 3. flycast: `frame=` in `/run/fly/flycast.progress` advancing. Flat for two passes, restart flycast. 4. mediamtx: `GET 127.0.0.1:9997/v3/paths/get/live/fly` shows `ready: true` and `bytesReceived` advancing. Else restart mediamtx, then flycast. 5. flypush: `frame=` in its own progress file advancing. Twitch-side liveness is checked centrally, not here, so two containers do not both poll Helix. 6. Disk guard: `/srv/fly/media` above 85% triggers an immediate prune; above 95% drops the recording leg while keeping the stream up, and raises the alarm metric. 7. Process age guard: flypush uptime above 24 h forces a restart even if the timer misfired. This is the local belt for the 48 h cap. Skipped when flypush is not enabled or not active. Restarts here count separately from the checks above, as `fly_watchdog_restarts_total{unit="flypush",reason="age"}`, so a dashboard can tell a scheduled age-guard bounce from a real progress-stall restart (check 5). Escalation: per-unit consecutive-failure counters in `/run/fly/wd/.fails` (tmpfs, so a reboot clears them). Three consecutive failures of the same unit inside 10 minutes restarts the dependent chain; five triggers `systemctl reboot` from inside the container, gated by `/var/lib/fly/wd/last-reboot` so it cannot reboot more than once an hour, with a hard stop after three reboots in six hours. Past that it stops trying and only alarms: a reboot loop on a host that also runs another service's production workload is worse than a dead demo. Notification: there is no alerting path, so (a) journal at `err` with the stable prefix `fly-watchdog:`, (b) a Prometheus textfile `/var/lib/node_exporter/textfile/fly_watchdog.prom` carrying `fly_watchdog_restarts_total{unit=...}` and `fly_watchdog_escalations_total`, which puts it on the wall panels. Building a real alert path is listed as a known gap, not smuggled into this project. ### Secrets: exact paths and modes | What | Where | Mode | |---|---|---| | source of truth | `pass twitch/-key`, `twitch/-key`, `twitch/helix-client-id`, `twitch/helix-client-secret`, `twitch/fly-pokemon-bot-token` | WSL box only | | stream key, in CT | `/etc/fly/creds/twitch-key.cred` | `root:root 0400`, dir `0700` | | bridge app creds, in CT | `/etc/fly/creds/twitch-app.cred` | `root:root 0400` | | bridge refresh tokens | `/var/lib/fly/bridge/tokens.json` | `fly:fly 0600` | | non-secret env | `/etc/fly/flypush.env`, `/etc/fly/fly.env` | `root:root 0644` | | Helix poller creds | `/etc/fly-twitch/helix.env` on the metrics container | `root:root 0600` | `06-secrets.sh` reads from `pass` on the WSL box and pipes straight into `pct exec -- systemd-creds encrypt --name=twitch-key - /etc/fly/creds/twitch-key.cred`. The secret never touches a file on the WSL box or on the host, and never appears in an argv (it arrives on stdin). Be honest about what `systemd-creds` buys in an unprivileged LXC: there is no TPM, so it falls back to `/var/lib/systemd/credential.secret`, which lives in the same container. Root in the container, or root on the host, can read it either way. The real value is that the key is not in the repo, not in the journal, not in any unit file, not in `/proc/*/environ` for any process except the one unit that declares `LoadCredentialEncrypted=`, and is scoped to that unit's lifetime. The residual exposure is `/proc//cmdline`: ffmpeg takes the RTMP URL as an argument, and there is no way around that. Containment is that the container has one non-root service user and no sshd. `verify.sh` greps the journal and the whole repo for the key's first 8 characters and fails if it finds them. The wrapper never runs `set -x`. Rotation is in the runbook and takes under a minute. Never rsync secrets off-box: `fly-backup-stage` explicitly excludes `/etc/fly/creds`, `/var/lib/fly/bridge`, and `/srv/fly/rom`. ### Backups, mirroring the neighbouring service's nightly job Same shape as `the backup script of another service on the host`: a root cron/timer on **the host** (not in the container), so the backup host credentials stay on the hypervisor and never enter a guest, and The host root's pubkey is already authorized for the backup host's backup account. 02:50 daily, staggered after the neighbouring production service's 02:30. `/fly-backup/backup.sh` per container: 1. `pct exec -- cat /srv/fly/state/manifest.json` and note `latest`/`previous`. 2. `pct pull` those generation files, then `pct pull` the manifest **last**. A concurrent commit can only add a newer generation, so the snapshot is a valid earlier point in time and the server never has to stop. (This is the ordering `docs/streaming-plan.md` section 4 argues for.) 3. `pct pull` `events.jsonl` and yesterday's highlight. 4. md5 change detection, then dated copy to `@:/fly-pokemon/`. 5. Write `backup_*` textfile metrics (last success epoch, bytes) with `host` and `role` labels, matching the existing collector convention. Do not back up the rolling recordings: 240 GB/week of material whose only durable value is the highlights. Retain 14 daily and 8 weekly checkpoint sets on the backup host; keep highlights indefinitely (they are small). Log to `/fly-backup/backup.log` and expect "unchanged, skip" lines, exactly like the the neighbouring production service job. ## 4. Local test mode the env file carries `PUSH_TARGET=local|twitch`. In `local`, `07-enable.sh` leaves `flypush` disabled and everything else runs identically, so the only difference between the test rig and production is one unit. MediaMTX config, with the security split that matters on a flat /16 that has a public NPM edge: ``` rtmpAddress: 127.0.0.1:1935 # publish and pull are loopback only hls: yes # LAN preview lives here hlsAddress: :8888 hlsVariant: mpegts # VLC-friendly; fMP4 low-latency HLS is flaky in VLC hlsAlwaysRemux: yes # HLS is ready with no reader, so probes need no fake viewer webrtc: yes webrtcAddress: :8889 webrtcLocalUDPAddress: :8189 api: yes apiAddress: 127.0.0.1:9997 metrics: yes metricsAddress: 127.0.0.1:9998 paths: live/fly: { source: publisher } ``` From the LAN: - VLC: `http://:8888/live/fly/index.m3u8` - browser: `http://:8889/live/fly` - RTMP stays loopback, so no `rtmp://` URL exists by design Explicitly: do not create an NPM proxy host for 8888 or 8889. If LAN-open HLS is not acceptable, add MediaMTX `authInternalUsers` with a read-only password. Flip to Twitch: ``` pass twitch/-key | ssh root@ \ 'pct exec -- systemd-creds encrypt --name=twitch-key - /etc/fly/creds/twitch-key.cred' pct exec -- systemctl enable --now flypush.service ``` Flip back is `systemctl disable --now flypush`. Neither direction restarts the sim, the page, the encoder or the recording. That is the whole reason for the split. ### Phase 0 spike checklist (throwaway the dev container `fly-spike`, `onboot=0`) Measure, in this order, and write every number into `infra/docs/p0-measurements.md`: 1. `unshare --user --pid true` as `fly` exits 0 (nesting works, no `--no-sandbox` needed). 2. flysim real-time factor on the host cores: 30 min runs at `RAYON_NUM_THREADS` of 1, 2, 4, 6, 8, feed off, reading `fly_sim_realtime_factor`. Record the knee. Calibration: Node reached 0.92x on a 5800X3D, so Node on Haswell projects to 0.4-0.5x. Rust with rayon at 6 threads should clear 1.0x. Below 0.5x is a design problem, not a tuning problem. 3. Chromium CPU with the 2D canvas map: `pidstat -u 5 360` filtered to chromium, 30 min. Target under 1.0 core total across all chromium processes. 4. ffmpeg CPU at veryfast 720p30, then faster, then ultrafast. Target under 1.5 core at veryfast. Judge quality at 3000 kbps by eyeballing 1 px UI borders and the 4x game panel, not by a number. 5. Total per-container load: `systemd-cgtop -1 --order=cpu` on the container slice while everything runs. Target 6 of 8 cores steady state. 6. A/V sync from a recorded file: record 60 minutes, then compare a reward tone against its on-screen flash frame by frame at minute 1 and minute 55. The drift is what matters, not the offset. Target under 100 ms per hour; the fix is `aresample=async=1`, which is already in the command. 7. Checkpoint envelope size and commit latency on `local-zfs` (this decides the cadence question in section 1). 8. 4 h unattended: RSS flat, `document.visibilityState === "visible"` throughout over CDP, rAF cadence stable, `/dev/shm` not exhausted, journal within its cap, Xvfb never restarted, pulse sink never `SUSPENDED`. 9. `ffmpeg -protocols | grep rtmps`, and `ffmpeg -f x11grab` using XShm. 10. VLC on the LAN plays the MediaMTX HLS output with production flags. Go/no-go to P1: all of 1 through 10 pass, real-time factor at or above 0.8x, total load at or below 6 cores, drift under 100 ms/h, RSS flat over 4 h. ## 5. Monitoring Existing: the metrics container `metrics` at `` runs Prometheus (`:9090`) and node_exporter (`:9100`), with custom textfile collectors (`gpu_*`, `service_up`, `guest_*`, `disk_*`, `backup_*`, `remote_pool_*`) that all carry `host` and `role` labels. `panel-bridge` (stdlib Python on `127.0.0.1:9099`) polls Prometheus once and fans label-stripped frames to the wall panels over SSE, with per-source `[age, budget]` freshness. There is no alerting path. Proposal, fitting that architecture rather than adding a parallel one: 1. `prometheus-node-exporter` in each fly CT with `--collector.textfile.directory=/var/lib/node_exporter/textfile`, listening on the CT address `:9100`. Two new scrape targets on the metrics container with `labels: {host: fly-pokemon, role: stream}`. VERIFY the exact `prometheus.yml` path and textfile directory on the metrics container; the infra repo's docs name the collectors but not the paths. 2. flysim's own `/metrics` on the read-only listener `:9101`. Series: `fly_sim_realtime_factor`, `fly_sim_steps_total`, `fly_emulator_frames_total`, `fly_frames_sent_total`, `fly_feed_clients`, `fly_ratchet_rank`, `fly_badges`, `fly_reward_total`, `fly_stuck_seconds`, `fly_checkpoint_age_seconds`, `fly_checkpoint_generation`, `fly_uptime_seconds`, `fly_sugar_redemptions_total`. Plus from the watchdog textfile: `fly_encoder_fps`, `fly_encoder_dropped_frames_total`, `fly_push_up`, `fly_media_use_ratio`, `fly_watchdog_restarts_total{unit}`. 3. A status JSON at `http://:9101/status.json`, written every 5 s, shaped for the panels rather than for Prometheus: `{demo, live, rank, rank_name, badges, uptime_s, realtime_factor, last_event, checkpoint_age_s, encoder_fps, twitch_live, viewers, ts}`. `panel-bridge` polls it once and adds a `fly` section to the SSE frame, so the tablets keep issuing zero extra requests and inherit the existing freshness machinery. Give it its own freshness budget (10 s) in the per-source table. wall-panels then gets a "fly live / rank / uptime" tile, and the honesty rules already in that project apply: if the source is stale, grey it out rather than showing a confident stale rank. 4. Twitch Helix liveness from exactly one place: `fly-twitch-liveness.timer` on **the metrics container**, every 120 s. One app access token via client credentials, one request `GET /helix/streams?user_login=&user_login=`, writing `fly_twitch_live{channel=}`, `fly_twitch_viewers{channel=}`, `fly_twitch_started_at{channel=}` to the textfile collector. The metrics container already owns "is it up" semantics and already has the collector directory. The app limit is 800 req/min, so 120 s is free. An empty `data` array means offline (`streaming-plan.md` marks that unverified; confirm by hand against a live and an offline channel in P2). 5. The two interesting derived conditions, both visible on the panel: `fly_push_up == 1 and fly_twitch_live == 0` for two consecutive polls means the local side thinks it is streaming and Twitch disagrees, which is the failure you cannot see from inside the container. And `time() - fly_twitch_started_at > 40h` means the 48 h cap is approaching despite the 23 h timer, which is the remote brace for the local process-age guard. ## 6. The second demo What differs: the ROM, the reward adapter, the decoder preset, the channel, and the env file. Nothing else. ``` # CTID= HOSTNAME=fly-platformer IP= GAME=super-mario-land # or kirbys-dream-land after the RAM-search spike ROM_SHA256= REWARD_ADAPTER=fly-sml-rstdp-v1 DECODER_PRESET=platformer # sustained holds, retuned cooldowns TWITCH_CHANNEL= PASS_KEY=twitch/-key PUSH_TARGET=local ``` One binary, not two. `flysim` compiles both adapters and selects by `FLY_GAME`, with the game id folded into the compatibility string so a Pokemon checkpoint is rejected by a platformer run and the reverse. Reasons: one artifact to build, sign and ship; one golden test suite; and the shared-image claim is then literally true rather than aspirational. The one real code difference to flag, because it is easy to under-scope: the motor decoder is calibrated at startup and tuned to Pokemon's menu-driven pacing. A platformer needs sustained button holds, so `DECODER_PRESET=platformer` is a reviewed change with its own tests, not a config tweak. `streaming-plan.md` section 7 phase 2 step 4 already says this. ROM handling: staged to `/srv/fly/rom/.gb`, mode `0400 fly:fly`, never in the repo (`.gitignore` already blocks `*.gb`), never in the backup rsync, never on screen, never linked. ## 7. Rollout, with go/no-go gates **P0, spike (the dev container, throwaway).** The ten measurements in section 4. Nothing shared, nothing live, no real checkpoint directory. Go: all ten pass at the stated thresholds. No-go paths: real-time factor under 0.5x sends us back to the sim design; Chromium over 2 cores sends the brain map to a smaller canvas; nesting failing sends us to `--no-sandbox` with that risk written down. **P1, fly-pokemon provisioned from the script, local only, 48 h.** Claim the release container in The host's `$AGENT_CLAIM_LOG` first (the operator's rule, in `the host's own agent notes`). Run `provision.sh ` from a clean checkout, then `verify.sh`. Stream to MediaMTX for 48 h with `flypush` disabled. Restore drills, all of them, with the observed recovery time recorded: 1. `kill -9` flysim. Expect: systemd restarts it, it restores from the latest checkpoint, the encoder and recording never drop, the page reconnects. Measure the visible gap. 2. `pct reboot `. Expect: every unit returns in order, HLS is back within N seconds. Record N. 3. Corrupt the latest checkpoint (`dd` one byte into it). Expect: flysim detects the bad checksum and falls back to `previous`. **If the checkpoint envelope has no checksum, that is a code gap to close before P1 completes**, because this drill is the one that protects against a silent bad-state loop. 4. Fill `/srv/fly/media` to 96%. Expect: retention prunes, the stream survives, the metric fires. 5. `pct set -net0 ...,link_down=1` for 10 minutes. Expect: flycast and the recording are untouched, flypush retries, recovery is automatic. 6. Restore from the backup host into the dev container and boot flysim on it. Assert rank and badges match. Also in P1: nightly backup wired and proven once; retention proven over a full 7-day cycle; the recap timer produces a watchable highlight three days running. Go: 48 h with no manual intervention, all six drills pass, provision script run twice with no second-run restarts, `verify.sh` green. **P2, Twitch test channel, 7 days.** Enable flypush against a throwaway test channel. Verify the 23 h restart leaves the sim and the recording running and produces one tidy VOD per day; verify whether the VOD splits and what the reconnect grace window actually is (both marked unverified in `streaming-plan.md`); verify the Helix poller flips offline when flypush stops and back when it returns; verify the discrepancy condition in section 5 item 5 by deliberately breaking it. Grep the journal and the repo for the key prefix. Go: 7 unattended days, at least 6 clean daily restarts, offline alert fires correctly in a deliberate test, no key found anywhere it should not be. **P3, public channel plus flybridge.** Rotate to the real channel's key. Enable flybridge for read-only chat first, then the rate-capped sugar path, with the cap enforced on the flysim side as the real limit and on the bridge side only for UX. Moderation policy in writing before any chat text is rendered into the frame. Go: overlay plus bridge adds under 0.3 core, cap enforcement proven by attacking it, moderation policy exists. **P4, fly-platformer.** RAM-search spike, adapter and catalog, decoder preset, compatibility string bump with a cross-rejection test, then `provision.sh `. Both containers live. Go: the host load average stays under 0.7x its thread count with both live, no farming loop reachable in a 1 h soak, and both pools trending flat on usage. **Every phase, on completion.** Append to the host's `$AGENT_CLAIM_LOG`, and update The operator's infra repo (`master`, `git pull --rebase` first, never force-push, markdown only, no `.sh` artifacts): new `the infra repo's flybrain page`; `the host's notes` guest table gains the release container/151; `main.md` static reservations table gains the two reserved addresses; `the infra repo's panels page` gains the fly tile and the new panel-bridge source. The scripts themselves live in `~/flybrain/infra/`, pushed to the operator's own git remote (API-create the repo first, push-to-create is disabled). ## 8. Risks - **Chromium in an unprivileged LXC.** Namespace sandbox needs nesting; without it the zygote aborts. Gated by the `unshare` precheck in P0. `--no-sandbox` works but is a real reduction in defence for a process running for weeks, so it is a fallback with a written justification, not a default. - **PulseAudio with no session.** No logind, no dbus session, no udev. Mitigated by `-n --file=` with only two modules loaded and a fixed socket path. The specific trap is `module-suspend-on-idle`: if it loads, the null sink suspends during silence and ffmpeg's pulse input stalls. Verify the sink never reaches `SUSPENDED` over a 4 h run. - **x11grab tearing.** Xvfb has no vblank, so a capture can read a frame mid-paint and show a horizontal seam. The content is largely static panels with a nearest-neighbour game view, so the exposure is low, and XShm capture plus Chromium's double-buffered compositor keeps it rare. Accept it, confirm by eye on a recorded file in P0, and do not chase it with `--disable-gpu-vsync`, which makes it worse and costs cores. - **Disk growth.** 240 GB per container per week if retention fails, on a pool that also carries the neighbouring GPU container, the metrics container, another container on the host and another guest on the host. Three independent brakes: the ZFS `quota=600G` (which is the one that actually protects the neighbours), the hourly retention timer, and the watchdog's 85%/95% guard. The quota is not optional. - **SSD write endurance from the checkpoint cadence.** Section 1 covers the arithmetic. A 5 s full-checkpoint cadence would put roughly 173 GB/day onto the mirror that hosts another service's production workload. Fix with the tmpfs hot ring plus a 300 s durable cadence, and confirm the envelope size in P0 before committing to numbers. - **48 h cap edge cases.** The 23 h timer resets on every crash-restart, and each restart ends the broadcast, so the cap itself is safe from that direction. The genuine edge case is the opposite: Twitch counts per broadcast, not per process, so a silent RTMP re-establish inside one ffmpeg lifetime can leave a broadcast older than the process thinks. Hence both guards: local process age over 24 h forces a restart, and remote `started_at` over 40 h alarms and forces one. - **Secrets.** The unavoidable residue is the RTMP URL in `/proc//cmdline`. Containment: no sshd, one service user, `verify.sh` greps the journal and repo for the key prefix, secrets excluded from the backup, rotation documented and under a minute. `systemd-creds` in an LXC has no TPM, so be honest in the docs about what it does and does not protect against. - **Haswell single-thread performance**, the pre-existing risk. P0 measurement 2 is the gate; there is no software fix downstream of it. - **Host time sync.** A container cannot set its own clock, so every recap boundary and event timestamp depends on the host being NTP-synced. VERIFY chrony or timesyncd on the host, and have `verify.sh` assert clock skew under 1 s. - **Unattended upgrades swapping Chromium or ffmpeg under a live stream.** `apt-mark hold` both, canary them on the spike CT, apply deliberately. - **No alerting path at all.** Everything above degrades to a panel tile and a journal line. Record it as a known gap rather than pretending the watchdog notifies anyone. ## Critical files to create - `infra/provision.sh` and `infra/lib/common.sh` - `the release env file` (plus ``, ``) - `infra/units/` (`xvfb`, `pulse`, `mediamtx`, `flysim`, `flystage`, `flycast`, `flypush`, `flybridge`, `fly.target`, five timers) - `infra/bin/fly-watchdog` and `infra/bin/fly-recap` - `infra/docs/runbook.md` (start/stop, local-to-Twitch flip, rotate stream key, rotate bot token, restore checkpoint locally and from the backup host, roll back a release via the `current` symlink, disk full, panel tile meanings, the host reboot order, and "do not touch the neighbouring production container") - `the operator's infra repo/the infra repo's flybrain page` (plus the three existing-doc edits in §7) ``` ### Critical Files for Implementation - infra/provision.sh (with `lib/common.sh`, `verify.sh`, the env file) - infra/units/flycast.service (and `flypush.service`, `flysim.service`, `flystage.service`, `fly.target`) - infra/bin/fly-watchdog (and `fly-recap`, `fly-retention`, `flypush`) - infra/docs/runbook.md - The operator's infra repo/the infra repo's flybrain page (plus `the host's notes`, `main.md`, `the infra repo's panels page` edits)