flybrain/docs/design/infra.md

Design document produced 2026-09-15 by a planning agent. Binding contracts are ../feed-protocol.md and ../control-api.md; where this document differs, the contracts win.

Two 24/7 Twitch demo containers on the host: infrastructure and rollout

Research only. Nothing on the host was touched, no files written. Every address, CT ID and free-space claim below is marked VERIFY where it rests on the infra repo's docs rather than on a live pct list.

0. What the operator's existing infrastructure constrains

• The host = PVE 9.2.10, <host-ip>, 2x Xeon E5-2660 v3 (20C/40T, AVX2, no AVX-512), 188 GB RAM. An SSD mirror backs local-zfs with something over 150 GB free, and a wide raidz2 array (tens of TiB usable) is registered with pvesm for images,rootdir. The exact devices, sizes and free space are in the operator's infra repo; treat any figure here as indicative.
• ZFS ARC is pinned at 6 GB in /etc/modprobe.d/zfs-arc.conf. Do not raise it.
• The LAN convention is DHCP plus a router-side reservation keyed on MAC, not in-guest static. The two guests that ignore that convention (two unrelated guests) are both documented as hazards, and one of them caused a live address collision with metrics. Follow the convention.
• The reserved range is already densely populated — around a dozen holders, plus the hosts themselves, the management interfaces and the switch. Read the reservation table in the operator's infra repo before asking for an address, and ask for one by MAC.
• The other containers on the host have no direct SSH; the neighbours' own deploy scripts scp to the host and then pct push. Mirror that: the fly containers get no sshd, all ops go through pct exec.
• Provisioning on these hosts has always been shell (deploy-prod.sh, deploy.sh, a hand-installed backup script, a burn-in directory, a migration directory). There is no Ansible anywhere in the operator's repos, and the only pct create in those docs is one mention. The closest template is a neighbouring container running a comparable self-hosted web service: Debian 13 unprivileged, 4c/6G/1G swap, 40G on bulk-array, onboot=1, features: nesting=1,keyctl=1.
• There is no alerting path at all. the infra repo's panels page "Known gaps" says Alertmanager and Grafana are unbuilt and postfix on the metrics container is loopback-only. Plan for "the panel tile is the alarm", and record the gap.

1. Container spec

Two containers, identical except for an env file.

	fly-pokemon	fly-platformer
CT ID	150 (VERIFY pct list)	151 (VERIFY)
hostname	fly-pokemon	fly-platformer
IP	<release-ct-ip> (VERIFY free: the router's lease table + arping)	<platformer-ct-ip> (VERIFY)
Twitch	channel 1	channel 2

Hostname note: fly-platformer rather than fly-mario, because the ROM choice is still open between Super Mario Land and Kirby's Dream Land (docs/streaming-plan.md section 6) and the container name should survive that decision. streaming-plan.md and the proposed pass entries already say fly-platformer.

pct create <release-ctid> local:vztmpl/debian-13-standard_13.6-1_amd64.tar.zst \
  --hostname fly-pokemon --ostype debian --unprivileged 1 \
  --cores 8 --memory 8192 --swap 2048 \
  --features nesting=1 \
  --rootfs local-zfs:24 \
  --mp0 local-zfs:16,mp=/srv/fly/state \
  --mp1 bulk-array:600,mp=/srv/fly/media \
  --net0 name=eth0,bridge=vmbr0,ip=dhcp \
  --onboot 1 --startup order=4,up=60 \
  --description "fly demo: flysim/flystage/flybridge/flycast (flybrain/infra)"

Why nesting=1: modern Chromium has no setuid sandbox, it uses the namespace sandbox, which means the zygote calls clone(CLONE_NEWUSER|CLONE_NEWPID|CLONE_NEWNS). An unprivileged LXC runs under the PVE generated AppArmor profile, which blocks nested user and mount namespace creation; without nesting the zygote dies with "Failed to move to new namespace: Operation not permitted" and Chromium never paints. nesting=1 switches to the nesting AppArmor profile and permits it. One-line precheck inside the CT as the fly user: unshare --user --pid true must exit 0. Fallback if it cannot be enabled: --no-sandbox, acceptable in principle because the page is local content from 127.0.0.1 with no navigation surface, but it removes the renderer's defence in depth for a process that runs for weeks, so treat it as a temporary workaround and not the design. keyctl=1 is not needed (no Docker here); add only if something complains.

Cores and memory: 8/8192 as decided. Two containers take 16 of 40 threads and 16 GB of the 60 GB available, which leaves the box at well under half. Swap 2048 so Chromium's cold pages page out instead of triggering the kernel OOM killer (VERIFY the host actually has host swap; if not, drop swap to 0 and rely on the per-unit MemoryMax below).

2026-09-15, VERIFIED, and the answer is no: the host's swap is 0. Read live from The host during the GPU probe (docs/design/gpu.md section 0). A container swap allocation has no backing store on a host with no swap, so --swap 2048 bought exactly nothing — the paragraph above is wrong on its own terms and its parenthetical resolves to "drop swap to 0". the env files now set SWAP_MB=0, and the per-unit MemoryMax values plus fly-watchdog are the entire OOM story: a Chromium leak hits MemoryMax=3G on flystage, gets killed there, and flysim (4G, Nice=-5) keeps playing. Also affects section 8's risk list, which assumed the swap cushion existed. The cpuset pinning added by the GPU work (docs/design/gpu.md section 1) changes the "16 of 40 threads" arithmetic too: the two containers now take named whole physical cores on NUMA node 0 rather than whatever PVE picked.

Per-unit memory caps matter more than the container total: MemoryMax=4G on flysim, 3G on flystage, 512M on flycast. A Chromium leak then kills flystage, the watchdog restarts it, and flysim keeps playing. Without caps the kernel picks the biggest RSS, which is usually Chromium but not always.

/dev/shm in an LXC defaults to a small tmpfs and Chromium's renderer will fall over on it. Fix it properly with a tmpfs mount unit sizing /dev/shm at 1 GB rather than using --disable-dev-shm-usage, which just moves the same traffic onto the rootfs.

Storage sizing

Rolling recording at 720p30, 3000 kbps video plus 160 kbps AAC:

• 3160 kbit/s = 395 kB/s = 1.42 GB/h
• 34.1 GB/day, 239 GB for 7 days, about 244 GB with MPEG-TS overhead
• plus 90 days of daily recaps (5 min at ~6 Mbps, ~225 MB each) = ~20 GB
• plus a checkpoint archive on the array

mp1 = bulk-array:600 mounted /srv/fly/media covers 7 days with 2x headroom. Set on the dataset (host side):

zfs set quota=600G recordsize=1M <bulk-pool>/subvol-<ctid>-disk-1

The quota is the single most important safety measure in this plan. Guests on the bulk array include the neighbouring GPU container, the metrics container, another container on the host and another guest on the host; a runaway recorder with no quota takes the monitoring stack and another project's NFS share down with it. recordsize=1M suits 10-minute sequential video files. Keep compression=lz4 (its early abort makes incompressible video nearly free) rather than turning compression off.

mp0 = local-zfs:16 mounted /srv/fly/state holds checkpoints, the event log, status.json and the segment index. It is on the SSD mirror because checkpoint commits are fsync-latency-sensitive, and it is a separate dataset so it can be snapshotted and backed up on its own and carries its own quota.

2026-09-15 note: the resolution/bitrate decision above changed (docs/streaming-plan.md section 3's dated note) — the broadcast canvas is now native 1920x1080 and the Twitch target is 1080p30 at 6000 kbps CBR, not the 720p30/3000 kbps this section's sizing assumed. Recomputed at 6160 kbit/s (6000 video + 160 audio): 770 kB/s, 2.77 GB/h, about 66.5 GB/day, about 465 GB for 7 days — roughly 1.9x the 244 GB figure above. The 600G mp1 quota still clears that with headroom, but not the same 2x margin the original sizing intended; recommend the bulk-array quota become 900G to restore comparable headroom (the 90-day recap and checkpoint-archive figures above are unaffected — recaps and checkpoints don't scale with live-stream resolution). Left as a note rather than rewriting the paragraph above, which records the original decision.

A finding that changes the checkpoint design

docs/server-sessions.md in the prototype commits a checkpoint every 5 s with fsync plus atomic rename. Estimate the envelope: 139,255 neurons of state (~3 MB), the plastic KC to MBON weight and eligibility arrays (~1 to 4 MB), the binjgb save state (~0.2 MB), so 5 to 15 MB per generation. At 10 MB every 5 s that is 173 GB/day of writes to the SSD pool. A 512G consumer SATA mirror is rated around 200 TBW, which is under a year of life, and that mirror also hosts another service's production workload.

Do not do that. Recommended instead:

1. Hot state ring in tmpfs at /run/fly/state (size-capped, 64 MB), written as often as the sim likes. This is what the watchdog reads for freshness.
2. Durable checkpoint to /srv/fly/state every 300 s, plus one on every milestone rank-up and one on clean shutdown.
3. Result: roughly 3 GB/day of SSD writes, and a crash costs at most 5 minutes of play, which is irrelevant for a 24/7 stream.

P0 must measure the real envelope size before this is finalised. If it turns out to be under 1 MB the 5 s cadence is fine and this becomes a non-issue.

2. Provisioning: shell, under ~/flybrain/infra/

Shell, not Ansible. Reasons: the operator has never used Ansible and the host has no control node or inventory; pct is not idempotent in a way Ansible's lxc modules improve on, so the useful idempotency is check-then-act either way; the whole thing is roughly 400 lines; and adding a new dependency to the one host that runs another service's production workload is a poor trade. The cost of shell is no drift detection, so the plan includes an explicit verify.sh that asserts desired state and exits non-zero, which is the piece people usually skip.

~/flybrain/infra/
  README.md
  provision.sh                 # runs 01..07 for one env file, resumable
  verify.sh                    # asserts desired state, exit 1 on drift
  lib/common.sh                # log/die/on_pve_host/ct_exec/ct_push/converge_file/need
  01-create-ct.sh              # pct create + pct set, guarded on `pct config`
  02-base.sh                   # apt, /dev/shm, journald caps, fly user, dirs, tmpfiles
  03-node.sh                   # pinned node 22 tarball from nodejs.org + sha256
  04-mediamtx.sh               # pinned release tarball + sha256 + /etc/mediamtx.yml
  05-deploy.sh                 # flysim binary, flystage bundle, flybridge, units
  06-secrets.sh                # pass -> systemd-creds, root 0600
  07-enable.sh                 # enable/start in order, health gates
  <release-env>  <platformer-env>  <dev-env>
  units/*.service  units/*.timer  units/fly.target
  config/mediamtx.yml  config/pulse.pa  config/chromium-flags  config/journald.conf
  bin/fly-watchdog  bin/fly-recap  bin/fly-backup-stage  bin/fly-retention
  bin/flypush  bin/wait-for-health  bin/wait-for-x  bin/wait-for-stage
  docs/runbook.md

Idempotency rules: set -euo pipefail everywhere; pct config $CTID >/dev/null 2>&1 || pct create ...; apt-get install -y is naturally converging; config files are shipped with converge_file which compares sha256, pushes only on difference, and signals whether a daemon-reload plus restart is needed. Running provision.sh twice in a row must produce no restarts the second time, and that is a test in verify.sh.

Packages: xvfb x11-utils x11-xserver-utils xauth chromium pulseaudio pulseaudio-utils ffmpeg fonts-dejavu-core fonts-liberation2 fonts-noto-core fonts-noto-color-emoji fontconfig prometheus-node-exporter curl ca-certificates jq rsync sysstat procps zstd. No sshd. apt-mark hold chromium ffmpeg so an unattended security upgrade cannot swap the two components most likely to break the stream; upgrade them deliberately on the spike CT first.

PulseAudio, not pipewire-pulse. The null-sink plus monitor pattern is exactly what PulseAudio does natively, it configures from a single --file= script, and it needs no dbus session or wireplumber. pipewire in a container with no logind session is the more fragile of the two, for no gain here.

Node 22 as a pinned tarball from nodejs.org into /opt/node-v22.x, verified by sha256, symlinked into /usr/local/bin. Hermetic, no third-party apt repo, matches the the operator's posture of pinning artifacts. Debian 13 ships Node 20, so the distro package is not an option; NodeSource works but adds a repo whose trixie support is another thing to monitor.

Rust never lands in prod. Build flysim on a dedicated throwaway build container, The throwaway build container fly-build (Debian 13, 16 cores, onboot=0, destroyable), not on the WSL box. Reason: glibc. Building against a newer glibc than Debian 13 trixie's and running on trixie fails at load, and the build box being the same template on the same host removes that class of problem entirely. Also, and this is the sharp edge:

RUSTFLAGS="-C target-cpu=haswell"     # correct: E5-2660 v3 is Haswell
RUSTFLAGS="-C target-cpu=native"      # WRONG off-host: Zen3 emits instructions
                                      # Haswell does not have, illegal instruction at run time

Reject x86_64-unknown-linux-musl static linking: musl's allocator is materially slower under the allocation pattern of a per-millisecond neuron sweep, and real-time factor is the whole ballgame. Ship a dynamically linked glibc binary built on the matching template.

Releases go to /opt/fly/releases/<version>/ with /opt/fly/current as the symlink, so a rollback is one ln -sfn plus a restart. sha256 of every artifact recorded in /opt/fly/releases/<version>/MANIFEST.

fly user: adduser --system --group --home /var/lib/fly --shell /usr/sbin/nologin fly. Directories via /etc/tmpfiles.d/fly.conf:

d /run/fly            0750 fly  fly  -
d /run/fly/pulse      0750 fly  fly  -
d /run/fly/state      0750 fly  fly  -
d /run/fly/wd         0750 fly  fly  -
d /var/lib/fly        0750 fly  fly  -
d /var/lib/fly/chrome 0700 fly  fly  -
d /srv/fly/state      0750 fly  fly  -
d /srv/fly/media/rec  0750 fly  fly  -
d /srv/fly/media/highlights 0750 fly fly -
d /etc/fly            0755 root root -
d /etc/fly/creds      0700 root root -

3. systemd units

One fly.target declares the whole set so the runbook has a single verb and ordering lives in one place. All units After=network-online.target, all app units User=fly.

fly.target
  xvfb.service        Xvfb :99
  pulse.service       null sink "stream"
  mediamtx.service    local RTMP/HLS/WebRTC ingest
  flysim.service      Type=notify, WatchdogSec=30
   -> flystage.service   waits on flysim /health
       -> flycast.service   encode once, tee to mediamtx + segments
           -> flypush.service  copy-only remux to Twitch (disabled in local mode)
  flybridge.service   Wants, never Requires (chat must not be able to stop the sim)

flycast's ordering is load-bearing, and After= only (2026-09-16). The arrow from flystage to flycast above used to be a diagram convention — flycast's After=/Requires= named xvfb and pulse but not flystage, so an xvfb restart co-started the page and the encoder. That race is the capture freeze: a new PulseAudio client attaching to the null sink during roughly the first one to three seconds of flycast's ffmpeg leaves its x11grab leg permanently starved (about one new picture a second, 29 repeats, audio perfect, 0 dup / 0 drop, frame= advancing at 30 fps), and Chromium's audio stream at page load is exactly such a client. Measured 3 h 50 min of frozen broadcast on the release container and reproduced six times on the dev container; infra/docs/capture-freeze.md has the experiments.

So flycast.service now carries After=xvfb.service pulse.service mediamtx.service flystage.service plus ExecStartPre=/opt/fly/bin/wait-for-stage 120 (X answers, a Chromium kiosk window mapped on :99, the stream sink has a sink-input, all three held for 5 s; after 120 s it warns and starts anyway) and TimeoutStartSec=180, since two ExecStartPre waits totalling 150 s do not fit systemd's default 90 s start timeout. flystage.service stays out of flycast's Requires= deliberately: ordering is what the fix needs, and a Requires= would let a dead or restarting page take the encoder, the recording and the broadcast down with it — the opposite of "the browser is display only; the sim is a service" in section 7's decisions.

The one structural change to the plan in piped-noodling-valley.md

Split flycast into two units. flycast encodes once and tees only to local sinks (MediaMTX and the segment recorder). flypush is a pure -c copy remux from rtmp://127.0.0.1:1935/live/fly to Twitch. Reasons:

1. A Twitch outage or backpressure can no longer stall the encoder or the local recording. With a single unit, a blocked RTMP write blocks the muxer loop for every tee leg (-use_fifo 1 mitigates it, does not eliminate it).
2. The 23 h restart then bounces a copy-only process, so there is no re-encode hiccup, the local recording is continuous across the seam, and the recap never sees a gap.
3. "Flip to Twitch" becomes "enable one unit", with nothing else restarting.
4. Cost is about 0.05 core.

The single-unit variant stays documented in the runbook as the simpler fallback.

xvfb.service

ExecStart=/usr/bin/Xvfb :99 -screen 0 1280x720x24 -nolisten tcp -noreset -dpi 96 \
          +extension RANDR -extension GLX
Restart=always  RestartSec=2

Depth 24 so the framebuffer is plain RGB, which converts cleanly to yuv420p. Keep MIT-SHM (on by default): x11grab uses XShm for fast frame reads and losing it costs real CPU. -extension GLX is deliberate: decision 3 in the plan puts the brain map on a 2D canvas precisely to avoid SwiftShader WebGL, so nothing needs server-side GLX, and removing it stops Chromium probing a software GLX stack it should not use. If WebGL ever comes back, the answer is --use-gl=angle --use-angle=swiftshader --enable-unsafe-swiftshader plus re-enabling GLX, and a fresh CPU measurement. P0 tests both settings.

pulse.service

User=fly
Environment=XDG_RUNTIME_DIR=/run/fly PULSE_RUNTIME_PATH=/run/fly/pulse
ExecStart=/usr/bin/pulseaudio -n --file=/etc/fly/pulse.pa --exit-idle-time=-1 \
          --disallow-exit --log-target=journal
Restart=always  RestartSec=2

No --system mode (upstream discourages it and it disables per-user modules) and no logind or enable-linger dependency. A plain system unit running as fly with an explicit PULSE_RUNTIME_PATH under /run/fly sidesteps the whole "container has no session" problem: the socket is at a fixed path both Chromium and ffmpeg can be pointed at. /etc/fly/pulse.pa:

load-module module-null-sink sink_name=stream rate=48000 channels=2 \
    sink_properties=device.description=stream
set-default-sink stream
load-module module-native-protocol-unix socket=/run/fly/pulse/native

-n means no default script, so module-udev-detect, module-console-kit, module-systemd-login and bluetooth are never loaded, all of which fail noisily in a container. Critically, module-suspend-on-idle is NOT loaded: if the null sink suspends while the page is silent, ffmpeg's pulse input stalls and you get audio gaps or a hard desync at the next sound. Verify with pactl list short sinks showing stream as RUNNING while Chromium plays, and IDLE never becoming SUSPENDED. 48 kHz throughout, matching AAC output, so nothing resamples.

flysim.service

Type=notify  NotifyAccess=main  WatchdogSec=30
Environment=FLY_GAME=${GAME} FLY_ROM=/srv/fly/rom/${ROM_SHA256}.gb
Environment=FLY_STATE_HOT=/run/fly/state FLY_STATE=/srv/fly/state
Environment=FLY_CONTROL_ADDR=127.0.0.1:7380 FLY_METRICS_ADDR=0.0.0.0:9101
Environment=RAYON_NUM_THREADS=6
Restart=always  RestartSec=2
StartLimitIntervalSec=300  StartLimitBurst=5
Nice=-5  CPUWeight=400  MemoryMax=4G

sd_notify is worth the small effort: it is one sendmsg to $NOTIFY_SOCKET, no crate needed. The important detail is that WATCHDOG=1 must be sent from inside the simulation loop, not from a helper thread, because then a stalled sweep actually trips the watchdog. WatchdogSec=30 against a 30 Hz loop is generous enough to survive a checkpoint fsync. If the Rust side is not ready at P1, fall back to Type=exec and let the external watchdog cover it on checkpoint freshness; add Type=notify before P2.

Two listeners, one process, and the separation is deliberate: the control API (/stimulate, /reward, /pause, /checkpoint, and explicitly no button endpoint) binds 127.0.0.1 only, while a read-only /metrics plus /status.json binds the CT address on 9101 so Prometheus on the metrics container can reach it. No mutating route is served on the LAN listener. RAYON_NUM_THREADS=6 leaves headroom for the encoder; P0 finds the knee.

flystage.service

After=xvfb.service pulse.service flysim.service
Requires=xvfb.service pulse.service
Environment=DISPLAY=:99 PULSE_SERVER=unix:/run/fly/pulse/native
ExecStartPre=/opt/fly/bin/wait-for-x :99 30
ExecStartPre=/opt/fly/bin/wait-for-health http://127.0.0.1:7380/health 120
ExecStart=/usr/bin/chromium $(cat /etc/fly/chromium-flags | tr '\n' ' ') http://127.0.0.1:7380/stage
Restart=always  RestartSec=5  MemoryMax=3G

Flags, with the reason each one is there:

Flag	Why
--kiosk --window-position=0,0 --window-size=1280,720	one window filling the root, no browser chrome in frame
--user-data-dir=/var/lib/fly/chrome	writable profile outside a nologin home, survives restarts
--no-first-run --no-default-browser-check --disable-search-engine-choice-screen	no first-run UI on stream
--noerrdialogs --disable-session-crashed-bubble --disable-infobars --hide-scrollbars	nothing modal can appear over the broadcast
--autoplay-policy=no-user-gesture-required	page-played audio has no click to wait for
--disable-background-timer-throttling	Chromium throttles background timers to ~1 Hz
--disable-backgrounding-occluded-windows	an occluded window must not be treated as background
--disable-renderer-backgrounding	keep renderer priority
--disable-ipc-flooding-protection	a 30 Hz feed plus canvas draws exceeds the default 10/s/frame cap
--disable-gpu --disable-software-rasterizer	2D canvas only, Skia CPU raster, no SwiftShader cost
--force-device-scale-factor=1 --force-color-profile=srgb	deterministic pixels into x264
--disable-lcd-text	subpixel antialiasing becomes colour fringing after 4:2:0 subsampling
--password-store=basic --use-mock-keychain	no gnome-keyring in the container
--disable-features=Translate,MediaRouter,OptimizationHints,CalculateNativeWinOcclusion	fewer background subsystems, and occlusion calculation is meaningless on Xvfb
--remote-debugging-port=9222 (bind 127.0.0.1)	needed for the P0 measurements and the CDP fallback watchdog

Rejected, with reasons, because the brief asked: --headless (see streaming-plan.md section 2b, screencast has no rate guarantee and burns cores on JPEG round-trips); --disable-gpu-vsync and --disable-frame-rate-limit (they uncap paint rate, which burns cores for nothing when ffmpeg owns the output clock at 30 fps, and make CPU unpredictable); --enable-unsafe-swiftshader (only if WebGL returns); --mute-audio (the whole point is page-played audio); --disable-dev-shm-usage (fix /dev/shm instead). Also set /etc/fonts/local.conf to grayscale antialiasing to match --disable-lcd-text.

flycast.service (encode once, local sinks only)

The unit as shipped also carries After=... flystage.service, ExecStartPre=/opt/fly/bin/wait-for-stage 120 and TimeoutStartSec=180 — the capture-freeze ordering gate described above. The ffmpeg command below is unchanged by that fix (beyond the 1080p/6000k and fps=30:round=near / -fps_mode:v cfr notes already recorded elsewhere): -analyzeduration/-probesize were considered and rejected, because ffmpeg's documentation of those two demuxer-probing options does not support an argument about a device input's first seconds, and the mechanism is not established (infra/docs/capture-freeze.md section 2).

ExecStartPre=/opt/fly/bin/wait-for-x :99 30
ExecStartPre=/opt/fly/bin/wait-for-stage 120
ExecStart=/usr/bin/ffmpeg -nostdin -loglevel warning -nostats \
  -thread_queue_size 1024 -f x11grab -draw_mouse 0 -framerate 30 -video_size 1280x720 -i :99.0+0,0 \
  -thread_queue_size 1024 -f pulse -name flycast -sample_rate 48000 -channels 2 -i stream.monitor \
  -filter_complex "[0:v]format=yuv420p[v];[1:a]aresample=async=1:min_hard_comp=0.100:first_pts=0[a]" \
  -map "[v]" -map "[a]" \
  -c:v libx264 -preset veryfast -profile:v high -level 4.1 \
  -b:v 3000k -minrate 3000k -maxrate 3000k -bufsize 6000k \
  -g 60 -keyint_min 60 -sc_threshold 0 -r 30 -bf 2 -x264-params "nal-hrd=cbr:filler=1" \
  -c:a aac -b:a 160k -ar 48000 -ac 2 \
  -progress /run/fly/flycast.progress -stats_period 5 \
  -f tee -use_fifo 1 \
  -fifo_options "queue_size=120:drop_pkts_on_overflow=1:attempt_recovery=1:recovery_wait_time=1" \
  "[f=flv:onfail=ignore]rtmp://127.0.0.1:1935/live/fly|[f=segment:segment_format=mpegts:segment_time=600:strftime=1:reset_timestamps=1:segment_list=/srv/fly/state/segments.csv:segment_list_type=csv:segment_list_flags=+live:segment_list_size=0]/srv/fly/media/rec/%Y%m%d-%H%M%S.ts"
ExecStartPost=/bin/sh -c 'date -u +%%s > /srv/fly/state/flycast-start'
Restart=always  RestartSec=5  CPUWeight=100  MemoryMax=512M

Decisions inside that command worth stating:

• No -tune zerolatency, despite the question in the brief. It disables lookahead and B-frames and therefore needs more bitrate for the same quality. We are not latency bound (a few seconds of glass-to-glass is fine for chat interaction), and 3000 kbps on a text-heavy 720p frame needs the efficiency. -tune stillimage is also rejected: it raises deblocking and smears 1 px UI edges. No tune, -bf 2, and nal-hrd=cbr:filler=1 for true CBR, which Twitch prefers.
• MPEG-TS segments, not MP4. A .ts file killed mid-write is still playable and concat-friendly; a truncated MP4 has no moov atom and is garbage. This matters because every ffmpeg restart truncates the open segment.
• -use_fifo 1 on the tee muxer is the specific fix for one slow leg blocking the others.
• segment_list as CSV gives the recap script an exact segment-to-time index instead of parsing filenames and hoping. flycast-start records the wallclock epoch of stream start so offsets are unambiguous.
• aresample=async=1 is the long-run A/V drift fix. x11grab and pulse are two independent clocks; over hours the pulse leg drifts and the fix is to let the resampler stretch rather than to accumulate.

flypush.service

After=mediamtx.service flycast.service
LoadCredentialEncrypted=twitch-key:/etc/fly/creds/twitch-key.cred
EnvironmentFile=/etc/fly/flypush.env
ExecStart=/opt/fly/bin/flypush
Restart=always  RestartSec=15
StartLimitIntervalSec=0

/opt/fly/bin/flypush reads $CREDENTIALS_DIRECTORY/twitch-key, builds rtmps://ingest.global-contribute.live-video.net/app/$KEY, and execs:

exec ffmpeg -nostdin -loglevel warning -nostats -rw_timeout 5000000 \
  -i rtmp://127.0.0.1:1935/live/fly -c copy -f flv \
  -progress /run/fly/flypush.progress "$URL"

P0 must confirm ffmpeg -protocols | grep rtmps on the Debian 13 build (native TLS RTMP, not librtmp). StartLimitIntervalSec=0 because a multi-hour Twitch outage must not permanently defeat the restart logic.

Timers

Unit	Schedule	Does
flypush-restart.timer	OnUnitActiveSec=23h, RandomizedDelaySec=30m, AccuracySec=1m	systemctl restart flypush, the 48 h guard
fly-recap.timer	OnCalendar=*-*-* 04:20, Persistent=true, RandomizedDelaySec=10m	cut yesterday's highlights
fly-retention.timer	OnUnitActiveSec=1h	prune segments > 7 d, highlights > 90 d, checkpoints per policy
fly-watchdog.timer	OnBootSec=2min, OnUnitActiveSec=60s, AccuracySec=5s	the health loop below
fly-backup.timer	on the host, not in the CT, 02:50 daily	stage and rsync to the backup host

RandomizedDelaySec=30m on the restart timer keeps the two channels from reconnecting at the same instant. Retention runs hourly rather than daily on purpose: a runaway recorder must not get 24 hours of rope.

fly-recap reads /srv/fly/state/events.jsonl for yesterday (boundary computed in America/New_York because that is the audience's yesterday, while every filename and event timestamp is UTC, which is a real source of off-by-one bugs), selects milestone rank-ups, badges, first-visit areas, deaths and sugar redemptions, maps each timestamp to (segment file, offset) via segments.csv plus flycast-start, cuts each clip with -ss/-t -c copy (2 s GOP means 2 s granularity, which is fine), concatenates with -f concat, and re-encodes only the final few minutes. Nice=19, IOSchedulingClass=idle, CPUWeight=20 so it can never starve the live stream.

fly-watchdog, 60 s

Checks in order, each with its own remediation, restarting only the failed unit:

1. flysim: /health returns 200 and hot-state mtime age < 30 s. Else restart flysim.
2. flystage: flysim's frames_sent_total and feed_clients from /status.json. Clients zero, or the counter flat across two passes, means the page is dead or frozen even though Chromium is alive. Restart flystage. This is the check that catches the failure mode a process-liveness check cannot see.
3. flycast: frame= in /run/fly/flycast.progress advancing. Flat for two passes, restart flycast.
4. mediamtx: GET 127.0.0.1:9997/v3/paths/get/live/fly shows ready: true and bytesReceived advancing. Else restart mediamtx, then flycast.
5. flypush: frame= in its own progress file advancing. Twitch-side liveness is checked centrally, not here, so two containers do not both poll Helix.
6. Disk guard: /srv/fly/media above 85% triggers an immediate prune; above 95% drops the recording leg while keeping the stream up, and raises the alarm metric.
7. Process age guard: flypush uptime above 24 h forces a restart even if the timer misfired. This is the local belt for the 48 h cap. Skipped when flypush is not enabled or not active. Restarts here count separately from the checks above, as fly_watchdog_restarts_total{unit="flypush",reason="age"}, so a dashboard can tell a scheduled age-guard bounce from a real progress-stall restart (check 5).

Escalation: per-unit consecutive-failure counters in /run/fly/wd/<unit>.fails (tmpfs, so a reboot clears them). Three consecutive failures of the same unit inside 10 minutes restarts the dependent chain; five triggers systemctl reboot from inside the container, gated by /var/lib/fly/wd/last-reboot so it cannot reboot more than once an hour, with a hard stop after three reboots in six hours. Past that it stops trying and only alarms: a reboot loop on a host that also runs another service's production workload is worse than a dead demo.

Notification: there is no alerting path, so (a) journal at err with the stable prefix fly-watchdog:, (b) a Prometheus textfile /var/lib/node_exporter/textfile/fly_watchdog.prom carrying fly_watchdog_restarts_total{unit=...} and fly_watchdog_escalations_total, which puts it on the wall panels. Building a real alert path is listed as a known gap, not smuggled into this project.

Secrets: exact paths and modes

What	Where	Mode
source of truth	pass twitch/<channel>-key, twitch/<platformer-channel>-key, twitch/helix-client-id, twitch/helix-client-secret, twitch/fly-pokemon-bot-token	WSL box only
stream key, in CT	/etc/fly/creds/twitch-key.cred	root:root 0400, dir 0700
bridge app creds, in CT	/etc/fly/creds/twitch-app.cred	root:root 0400
bridge refresh tokens	/var/lib/fly/bridge/tokens.json	fly:fly 0600
non-secret env	/etc/fly/flypush.env, /etc/fly/fly.env	root:root 0644
Helix poller creds	/etc/fly-twitch/helix.env on the metrics container	root:root 0600

06-secrets.sh reads from pass on the WSL box and pipes straight into pct exec <release-ctid> -- systemd-creds encrypt --name=twitch-key - /etc/fly/creds/twitch-key.cred. The secret never touches a file on the WSL box or on the host, and never appears in an argv (it arrives on stdin).

Be honest about what systemd-creds buys in an unprivileged LXC: there is no TPM, so it falls back to /var/lib/systemd/credential.secret, which lives in the same container. Root in the container, or root on the host, can read it either way. The real value is that the key is not in the repo, not in the journal, not in any unit file, not in /proc/*/environ for any process except the one unit that declares LoadCredentialEncrypted=, and is scoped to that unit's lifetime.

The residual exposure is /proc/<pid>/cmdline: ffmpeg takes the RTMP URL as an argument, and there is no way around that. Containment is that the container has one non-root service user and no sshd. verify.sh greps the journal and the whole repo for the key's first 8 characters and fails if it finds them. The wrapper never runs set -x. Rotation is in the runbook and takes under a minute.

Never rsync secrets off-box: fly-backup-stage explicitly excludes /etc/fly/creds, /var/lib/fly/bridge, and /srv/fly/rom.

Backups, mirroring the neighbouring service's nightly job

Same shape as the backup script of another service on the host: a root cron/timer on the host (not in the container), so the backup host credentials stay on the hypervisor and never enter a guest, and The host root's pubkey is already authorized for the backup host's backup account. 02:50 daily, staggered after the neighbouring production service's 02:30.

<host-stage>/fly-backup/backup.sh per container:

1. pct exec <release-ctid> -- cat /srv/fly/state/manifest.json and note latest/previous.
2. pct pull those generation files, then pct pull the manifest last. A concurrent commit can only add a newer generation, so the snapshot is a valid earlier point in time and the server never has to stop. (This is the ordering docs/streaming-plan.md section 4 argues for.)
3. pct pull events.jsonl and yesterday's highlight.
4. md5 change detection, then dated copy to <backup-user>@<backup-host>:<backup-path>/fly-pokemon/.
5. Write backup_* textfile metrics (last success epoch, bytes) with host and role labels, matching the existing collector convention.

Do not back up the rolling recordings: 240 GB/week of material whose only durable value is the highlights. Retain 14 daily and 8 weekly checkpoint sets on the backup host; keep highlights indefinitely (they are small). Log to <host-stage>/fly-backup/backup.log and expect "unchanged, skip" lines, exactly like the the neighbouring production service job.

4. Local test mode

the env file carries PUSH_TARGET=local|twitch. In local, 07-enable.sh leaves flypush disabled and everything else runs identically, so the only difference between the test rig and production is one unit.

MediaMTX config, with the security split that matters on a flat /16 that has a public NPM edge:

rtmpAddress: 127.0.0.1:1935     # publish and pull are loopback only
hls: yes                        # LAN preview lives here
hlsAddress: :8888
hlsVariant: mpegts              # VLC-friendly; fMP4 low-latency HLS is flaky in VLC
hlsAlwaysRemux: yes             # HLS is ready with no reader, so probes need no fake viewer
webrtc: yes
webrtcAddress: :8889
webrtcLocalUDPAddress: :8189
api: yes
apiAddress: 127.0.0.1:9997
metrics: yes
metricsAddress: 127.0.0.1:9998
paths:
  live/fly: { source: publisher }

From the LAN:

• VLC: http://<release-ct-ip>:8888/live/fly/index.m3u8
• browser: http://<release-ct-ip>:8889/live/fly
• RTMP stays loopback, so no rtmp://<release-ct-ip> URL exists by design

Explicitly: do not create an NPM proxy host for 8888 or 8889. If LAN-open HLS is not acceptable, add MediaMTX authInternalUsers with a read-only password.

Flip to Twitch:

pass twitch/<channel>-key | ssh root@<host-ip> \
  'pct exec <release-ctid> -- systemd-creds encrypt --name=twitch-key - /etc/fly/creds/twitch-key.cred'
pct exec <release-ctid> -- systemctl enable --now flypush.service

Flip back is systemctl disable --now flypush. Neither direction restarts the sim, the page, the encoder or the recording. That is the whole reason for the split.

Phase 0 spike checklist (throwaway the dev container fly-spike, onboot=0)

Measure, in this order, and write every number into infra/docs/p0-measurements.md:

1. unshare --user --pid true as fly exits 0 (nesting works, no --no-sandbox needed).
2. flysim real-time factor on the host cores: 30 min runs at RAYON_NUM_THREADS of 1, 2, 4, 6, 8, feed off, reading fly_sim_realtime_factor. Record the knee. Calibration: Node reached 0.92x on a 5800X3D, so Node on Haswell projects to 0.4-0.5x. Rust with rayon at 6 threads should clear 1.0x. Below 0.5x is a design problem, not a tuning problem.
3. Chromium CPU with the 2D canvas map: pidstat -u 5 360 filtered to chromium, 30 min. Target under 1.0 core total across all chromium processes.
4. ffmpeg CPU at veryfast 720p30, then faster, then ultrafast. Target under 1.5 core at veryfast. Judge quality at 3000 kbps by eyeballing 1 px UI borders and the 4x game panel, not by a number.
5. Total per-container load: systemd-cgtop -1 --order=cpu on the container slice while everything runs. Target 6 of 8 cores steady state.
6. A/V sync from a recorded file: record 60 minutes, then compare a reward tone against its on-screen flash frame by frame at minute 1 and minute 55. The drift is what matters, not the offset. Target under 100 ms per hour; the fix is aresample=async=1, which is already in the command.
7. Checkpoint envelope size and commit latency on local-zfs (this decides the cadence question in section 1).
8. 4 h unattended: RSS flat, document.visibilityState === "visible" throughout over CDP, rAF cadence stable, /dev/shm not exhausted, journal within its cap, Xvfb never restarted, pulse sink never SUSPENDED.
9. ffmpeg -protocols | grep rtmps, and ffmpeg -f x11grab using XShm.
10. VLC on the LAN plays the MediaMTX HLS output with production flags.

Go/no-go to P1: all of 1 through 10 pass, real-time factor at or above 0.8x, total load at or below 6 cores, drift under 100 ms/h, RSS flat over 4 h.

5. Monitoring

Existing: the metrics container metrics at <metrics-ct-ip> runs Prometheus (:9090) and node_exporter (:9100), with custom textfile collectors (gpu_*, service_up, guest_*, disk_*, backup_*, remote_pool_*) that all carry host and role labels. panel-bridge (stdlib Python on 127.0.0.1:9099) polls Prometheus once and fans label-stripped frames to the wall panels over SSE, with per-source [age, budget] freshness. There is no alerting path.

Proposal, fitting that architecture rather than adding a parallel one:

1. prometheus-node-exporter in each fly CT with --collector.textfile.directory=/var/lib/node_exporter/textfile, listening on the CT address :9100. Two new scrape targets on the metrics container with labels: {host: fly-pokemon, role: stream}. VERIFY the exact prometheus.yml path and textfile directory on the metrics container; the infra repo's docs name the collectors but not the paths.
2. flysim's own /metrics on the read-only listener :9101. Series: fly_sim_realtime_factor, fly_sim_steps_total, fly_emulator_frames_total, fly_frames_sent_total, fly_feed_clients, fly_ratchet_rank, fly_badges, fly_reward_total, fly_stuck_seconds, fly_checkpoint_age_seconds, fly_checkpoint_generation, fly_uptime_seconds, fly_sugar_redemptions_total. Plus from the watchdog textfile: fly_encoder_fps, fly_encoder_dropped_frames_total, fly_push_up, fly_media_use_ratio, fly_watchdog_restarts_total{unit}.
3. A status JSON at http://<ct-ip>:9101/status.json, written every 5 s, shaped for the panels rather than for Prometheus: {demo, live, rank, rank_name, badges, uptime_s, realtime_factor, last_event, checkpoint_age_s, encoder_fps, twitch_live, viewers, ts}. panel-bridge polls it once and adds a fly section to the SSE frame, so the tablets keep issuing zero extra requests and inherit the existing freshness machinery. Give it its own freshness budget (10 s) in the per-source table. wall-panels then gets a "fly live / rank / uptime" tile, and the honesty rules already in that project apply: if the source is stale, grey it out rather than showing a confident stale rank.
4. Twitch Helix liveness from exactly one place: fly-twitch-liveness.timer on the metrics container, every 120 s. One app access token via client credentials, one request GET /helix/streams?user_login=<ch1>&user_login=<ch2>, writing fly_twitch_live{channel=}, fly_twitch_viewers{channel=}, fly_twitch_started_at{channel=} to the textfile collector. The metrics container already owns "is it up" semantics and already has the collector directory. The app limit is 800 req/min, so 120 s is free. An empty data array means offline (streaming-plan.md marks that unverified; confirm by hand against a live and an offline channel in P2).
5. The two interesting derived conditions, both visible on the panel: fly_push_up == 1 and fly_twitch_live == 0 for two consecutive polls means the local side thinks it is streaming and Twitch disagrees, which is the failure you cannot see from inside the container. And time() - fly_twitch_started_at > 40h means the 48 h cap is approaching despite the 23 h timer, which is the remote brace for the local process-age guard.

6. The second demo

What differs: the ROM, the reward adapter, the decoder preset, the channel, and the env file. Nothing else.

# <platformer-env>
CTID=<platformer-ctid>  HOSTNAME=fly-platformer  IP=<platformer-ct-ip>
GAME=super-mario-land            # or kirbys-dream-land after the RAM-search spike
ROM_SHA256=<sha256>
REWARD_ADAPTER=fly-sml-rstdp-v1
DECODER_PRESET=platformer        # sustained holds, retuned cooldowns
TWITCH_CHANNEL=<ch2>
PASS_KEY=twitch/<platformer-channel>-key
PUSH_TARGET=local

One binary, not two. flysim compiles both adapters and selects by FLY_GAME, with the game id folded into the compatibility string so a Pokemon checkpoint is rejected by a platformer run and the reverse. Reasons: one artifact to build, sign and ship; one golden test suite; and the shared-image claim is then literally true rather than aspirational.

The one real code difference to flag, because it is easy to under-scope: the motor decoder is calibrated at startup and tuned to Pokemon's menu-driven pacing. A platformer needs sustained button holds, so DECODER_PRESET=platformer is a reviewed change with its own tests, not a config tweak. streaming-plan.md section 7 phase 2 step 4 already says this.

ROM handling: staged to /srv/fly/rom/<sha256>.gb, mode 0400 fly:fly, never in the repo (.gitignore already blocks *.gb), never in the backup rsync, never on screen, never linked.

7. Rollout, with go/no-go gates

P0, spike (the dev container, throwaway). The ten measurements in section 4. Nothing shared, nothing live, no real checkpoint directory. Go: all ten pass at the stated thresholds. No-go paths: real-time factor under 0.5x sends us back to the sim design; Chromium over 2 cores sends the brain map to a smaller canvas; nesting failing sends us to --no-sandbox with that risk written down.

P1, fly-pokemon provisioned from the script, local only, 48 h. Claim the release container in The host's $AGENT_CLAIM_LOG first (the operator's rule, in the host's own agent notes). Run provision.sh <release-env> from a clean checkout, then verify.sh. Stream to MediaMTX for 48 h with flypush disabled. Restore drills, all of them, with the observed recovery time recorded:

1. kill -9 flysim. Expect: systemd restarts it, it restores from the latest checkpoint, the encoder and recording never drop, the page reconnects. Measure the visible gap.
2. pct reboot <release-ctid>. Expect: every unit returns in order, HLS is back within N seconds. Record N.
3. Corrupt the latest checkpoint (dd one byte into it). Expect: flysim detects the bad checksum and falls back to previous. If the checkpoint envelope has no checksum, that is a code gap to close before P1 completes, because this drill is the one that protects against a silent bad-state loop.
4. Fill /srv/fly/media to 96%. Expect: retention prunes, the stream survives, the metric fires.
5. pct set <release-ctid> -net0 ...,link_down=1 for 10 minutes. Expect: flycast and the recording are untouched, flypush retries, recovery is automatic.
6. Restore from the backup host into the dev container and boot flysim on it. Assert rank and badges match.

Also in P1: nightly backup wired and proven once; retention proven over a full 7-day cycle; the recap timer produces a watchable highlight three days running. Go: 48 h with no manual intervention, all six drills pass, provision script run twice with no second-run restarts, verify.sh green.

P2, Twitch test channel, 7 days. Enable flypush against a throwaway test channel. Verify the 23 h restart leaves the sim and the recording running and produces one tidy VOD per day; verify whether the VOD splits and what the reconnect grace window actually is (both marked unverified in streaming-plan.md); verify the Helix poller flips offline when flypush stops and back when it returns; verify the discrepancy condition in section 5 item 5 by deliberately breaking it. Grep the journal and the repo for the key prefix. Go: 7 unattended days, at least 6 clean daily restarts, offline alert fires correctly in a deliberate test, no key found anywhere it should not be.

P3, public channel plus flybridge. Rotate to the real channel's key. Enable flybridge for read-only chat first, then the rate-capped sugar path, with the cap enforced on the flysim side as the real limit and on the bridge side only for UX. Moderation policy in writing before any chat text is rendered into the frame. Go: overlay plus bridge adds under 0.3 core, cap enforcement proven by attacking it, moderation policy exists.

P4, fly-platformer. RAM-search spike, adapter and catalog, decoder preset, compatibility string bump with a cross-rejection test, then provision.sh <platformer-env>. Both containers live. Go: the host load average stays under 0.7x its thread count with both live, no farming loop reachable in a 1 h soak, and both pools trending flat on usage.

Every phase, on completion. Append to the host's $AGENT_CLAIM_LOG, and update The operator's infra repo (master, git pull --rebase first, never force-push, markdown only, no .sh artifacts): new the infra repo's flybrain page; the host's notes guest table gains the release container/151; main.md static reservations table gains the two reserved addresses; the infra repo's panels page gains the fly tile and the new panel-bridge source. The scripts themselves live in ~/flybrain/infra/, pushed to the operator's own git remote (API-create the repo first, push-to-create is disabled).

8. Risks

• Chromium in an unprivileged LXC. Namespace sandbox needs nesting; without it the zygote aborts. Gated by the unshare precheck in P0. --no-sandbox works but is a real reduction in defence for a process running for weeks, so it is a fallback with a written justification, not a default.
• PulseAudio with no session. No logind, no dbus session, no udev. Mitigated by -n --file= with only two modules loaded and a fixed socket path. The specific trap is module-suspend-on-idle: if it loads, the null sink suspends during silence and ffmpeg's pulse input stalls. Verify the sink never reaches SUSPENDED over a 4 h run.
• x11grab tearing. Xvfb has no vblank, so a capture can read a frame mid-paint and show a horizontal seam. The content is largely static panels with a nearest-neighbour game view, so the exposure is low, and XShm capture plus Chromium's double-buffered compositor keeps it rare. Accept it, confirm by eye on a recorded file in P0, and do not chase it with --disable-gpu-vsync, which makes it worse and costs cores.
• Disk growth. 240 GB per container per week if retention fails, on a pool that also carries the neighbouring GPU container, the metrics container, another container on the host and another guest on the host. Three independent brakes: the ZFS quota=600G (which is the one that actually protects the neighbours), the hourly retention timer, and the watchdog's 85%/95% guard. The quota is not optional.
• SSD write endurance from the checkpoint cadence. Section 1 covers the arithmetic. A 5 s full-checkpoint cadence would put roughly 173 GB/day onto the mirror that hosts another service's production workload. Fix with the tmpfs hot ring plus a 300 s durable cadence, and confirm the envelope size in P0 before committing to numbers.
• 48 h cap edge cases. The 23 h timer resets on every crash-restart, and each restart ends the broadcast, so the cap itself is safe from that direction. The genuine edge case is the opposite: Twitch counts per broadcast, not per process, so a silent RTMP re-establish inside one ffmpeg lifetime can leave a broadcast older than the process thinks. Hence both guards: local process age over 24 h forces a restart, and remote started_at over 40 h alarms and forces one.
• Secrets. The unavoidable residue is the RTMP URL in /proc/<pid>/cmdline. Containment: no sshd, one service user, verify.sh greps the journal and repo for the key prefix, secrets excluded from the backup, rotation documented and under a minute. systemd-creds in an LXC has no TPM, so be honest in the docs about what it does and does not protect against.
• Haswell single-thread performance, the pre-existing risk. P0 measurement 2 is the gate; there is no software fix downstream of it.
• Host time sync. A container cannot set its own clock, so every recap boundary and event timestamp depends on the host being NTP-synced. VERIFY chrony or timesyncd on the host, and have verify.sh assert clock skew under 1 s.
• Unattended upgrades swapping Chromium or ffmpeg under a live stream. apt-mark hold both, canary them on the spike CT, apply deliberately.
• No alerting path at all. Everything above degrades to a panel tile and a journal line. Record it as a known gap rather than pretending the watchdog notifies anyone.

Critical files to create

• infra/provision.sh and infra/lib/common.sh
• the release env file (plus <platformer-env>, <dev-env>)
• infra/units/ (xvfb, pulse, mediamtx, flysim, flystage, flycast, flypush, flybridge, fly.target, five timers)
• infra/bin/fly-watchdog and infra/bin/fly-recap
• infra/docs/runbook.md (start/stop, local-to-Twitch flip, rotate stream key, rotate bot token, restore checkpoint locally and from the backup host, roll back a release via the current symlink, disk full, panel tile meanings, the host reboot order, and "do not touch the neighbouring production container")
• the operator's infra repo/the infra repo's flybrain page (plus the three existing-doc edits in §7)

### Critical Files for Implementation
- infra/provision.sh (with `lib/common.sh`, `verify.sh`, the env file)
- infra/units/flycast.service (and `flypush.service`, `flysim.service`, `flystage.service`, `fly.target`)
- infra/bin/fly-watchdog (and `fly-recap`, `fly-retention`, `flypush`)
- infra/docs/runbook.md
- The operator's infra repo/the infra repo's flybrain page (plus `the host's notes`, `main.md`, `the infra repo's panels page` edits)