# flybrain/infra Provisioning, systemd units, and a runbook for the two 24/7 Twitch demo LXC containers (`fly-pokemon` the release container, `fly-platformer` the platformer container) plus a throwaway measurement CT (`fly-spike` the dev container), all on **the host**. Implements `docs/design/infra.md`; where `docs/feed-protocol.md` or `docs/control-api.md` disagree with that design doc on a port number or endpoint shape, this code follows the contract docs (feed `ws://127.0.0.1:7400/feed`, control `http://127.0.0.1:7401`) — see each unit file's own header comment for exactly where. The GPU path (NVIDIA passthrough, the in-container driver, NVENC encoding, CPU pinning) implements `docs/design/gpu.md` and is switched entirely by `GPU=1` in the env file. The recommendation there is narrow on purpose: **the GPU is for the encoder, not for Chromium.** Read it before touching anything under `host/`, the `lxc.*` blocks in `01-create-ct.sh`, or `bin/flycast-launch`. Read `docs/design/infra.md` in full before changing anything here, then `infra/docs/runbook.md` for operations. ## Layout ``` provision.sh runs 01..07 for one env file, resumable verify.sh asserts desired state, exit 1 on drift lib/common.sh log/die/need/converge_file/converge_conf_block/pct wrappers host/* the two HOST-side files (fly-nvidia-majors.{sh,service}), installed by hand on the host — see host/README.md 01-create-ct.sh .. 07-enable.sh env/example.env the template; real env files live outside this repo units/*.service .timer .target config/* mediamtx.yml, pulse.pa, chromium-flags, chromium-flags.gpu, chat-deny.txt (installed once to /srv/fly, then operator-owned) (gpu: spike-only, insufficient on its own), chromium-flags.vgl (VirtualGL EGL, measured passing, dev containers only — infra/docs/virtualgl-spike.md), journald.conf, fonts-local.conf, fly-tmpfiles.conf, dev-shm-override.conf, fly-sudoers, serve.mjs + serve.sh (flystage-web's static server) bin/* fly-watchdog, fly-recap, fly-retention, fly-backup-stage, flypush, flystage-launch, flycast-launch, wait-for-x, wait-for-stage, wait-for-health build/* build-flysim.sh (runs in the fly-build CT), package-release.sh docs/runbook.md docs/*.md method + pointers; the records live in the infra repo tests/lint.sh bash -n / shellcheck + systemd-analyze verify + ExecStart* sanity ``` Every script under `infra/` takes the **path to an env file** as its first argument, is `set -euo pipefail`, and is safe to re-run (`converge_file` in `lib/common.sh` compares sha256 before pushing anything, and every `pct create`/`adduser`/`mkdir -p` call is naturally idempotent or explicitly guarded). **The real env files are not in this repo.** `infra/env/example.env` is the template; the deployable ones live in the operator's infra repo and are passed by path from outside the checkout (`infra/env/README.md`). Throughout this document ``, `` and `` stand for those paths — e.g. `/etc/fly/env/fly-pokemon.env`, or a bare name with `FLY_ENV_DIR` set. Other angle-bracket values (``, ``, ``, …) are placeholders for the same reason: this repo is public and must not name the operator's network. **Nothing in this repo runs against the host by itself.** These scripts are meant to be run by a human, on the host, deliberately, one step at a time if needed (`--from-step N`). They never run from CI, from this worktree's own test harness, or unattended. ## Quick start (once P0 has gone GO) ```sh # On the host, as root: infra/provision.sh infra/verify.sh ``` Add `--release path/to/flybrain-.tar.gz` to `provision.sh` once a release exists (`infra/build/build-flysim.sh` + `infra/build/package-release.sh`); without it, `provision.sh` still converges every unit, config file, and bin/ helper, which is the expected state until `services/flysim`, `apps/stage`, and `services/bridge` are built. ### Provisioning a release container before the first release `ROLE=release` (the release container) makes `05-deploy.sh` refuse any tree that is not exactly at a clean annotated tag — which is also true of the units-and-config-only pass above, so a brand-new release container could not be provisioned at all until a tag existed. `PRERELEASE_UNITS=1` is the one documented way through, and it is narrow: no release tarball (so nothing is installed under `/opt/fly/releases` and `/opt/fly/current` is never flipped), the variable typed by a human, and a container that has never had a release deployed to it. From the first tagged deploy onwards it refuses forever, so it can never be used to slip an untagged unit file onto a live stream. ```sh # on the host, provisioning the release container for the first time (no release exists yet): PRERELEASE_UNITS=1 infra/provision.sh ``` Do **not** run `07-enable.sh` (or `provision.sh` past step 6) on a release container with no release: `fly.target` would start `flysim` against a `/opt/fly/current` that does not exist. Run steps 1-6 and leave the target disabled until the first tagged deploy — that is what `verify.sh` now expects to find on a pre-release container (it checks that the unit *files* are converged and skips the active/enabled assertions). ## The dev/release split `ROLE` in each env file (default `dev` when unset — see `infra/lib/common.sh`'s `load_env`) picks which of two shapes a container gets (docs/stream-mvp-plan.md, "Release container", the operator 2026-09-16): - **`ROLE=dev`** — `` (the dev container `fly-spike`). Builds, measurements, deploy trials, the VirtualGL experiment happen here. GPU passthrough and NVENC stay on this box. `infra/05-deploy.sh` deploys any commit, clean or not, and names the release directory after a short sha + timestamp — the historical behaviour, unchanged. - **`ROLE=release`** — `` (the release container `fly-pokemon`), the actual stream. `GPU=0`, `FLY_ENCODER=x264` for the first release (no NVIDIA passthrough on this container at all; NVENC stays dev-only until the operator says otherwise). `05-deploy.sh` refuses to deploy unless the source tree it is itself run from is exactly at a clean annotated `vX.Y.Z` tag, names the release directory after that tag, and prints the tag in its final claim-log-style deploy line. Nothing experimental runs here; never develop or measure on this box. See "Cutting a release" below for the tag-to-deploy flow, and `infra/docs/runbook.md`'s section of the same name for the full command sequence including rollback. ## Cutting a release 1. On a clean `main`: `infra/build/tag-release.sh vX.Y.Z`. It runs `npm test`, `npm run typecheck`, `cargo test --workspace` (in `services/flysim`), and `infra/tests/lint.sh` itself, and refuses to create the tag if any of them fail, the tree is dirty, or the current branch is not `main`. It does not push the tag. 2. Build and package from that tagged commit: `infra/build/build-flysim.sh`, then `infra/build/package-release.sh vX.Y.Z `. The MANIFEST inside the tarball records `git_tag=vX.Y.Z` and `git_commit=` as `#`-prefixed lines ahead of the sha256 checksums. 3. Deploy, from a checkout that is ITSELF at that same clean tag: `infra/05-deploy.sh /path/to/flybrain-vX.Y.Z.tar.gz`. ``'s `ROLE=release` makes `05-deploy.sh` check the checkout it is running from (not the tarball) — `git describe --exact-match --tags` and `git status --porcelain` — and refuse the entire deploy if either fails. 4. `infra/verify.sh `. 5. Rollback to the previous release: `infra/docs/runbook.md`'s "Roll back a release" — one `ln -sfn` of `/opt/fly/current` to the previous tag's `/opt/fly/releases//` directory, plus a restart. Releases on this container are named after tags, so "the previous release" and "the previous tag" are the same directory. Full command sequence, including the exact refusal message and what a passing run prints, is in `infra/docs/runbook.md`'s own "Cutting a release" section. ## Host-side steps the operator runs by hand These are **not** run by any script in this repo, on purpose — either because they are one-time/rare, because they are safety-critical enough to want a human looking at the number, or because they touch host state (`$AGENT_CLAIM_LOG`, the ZFS pool) that no per-container script should be reaching outside its own container's boundary. ### 1. Claim the container ID Before running `provision.sh` for the first time against a real CT ID, append a claim to The host's agent claim log — the file `AGENT_CLAIM_LOG` names (the operator's rule; the path is in the operator's infra repo, not here): ``` 2026-0X-XX: claiming the release container (fly-pokemon) and the platformer container (fly-platformer) for flybrain/infra. ``` ### 2. `pct create` (reference — `01-create-ct.sh` runs this for you) `01-create-ct.sh` runs this exact command (guarded on `pct config` so a second run is a no-op); it is reproduced here so the manual command and the scripted one never drift apart silently. The one difference is the template argument: the script resolves the newest `debian-13-standard_*_amd64.tar.zst` actually present in `pveam list local` (override with `TEMPLATE=`) rather than hardcoding a patch level, because the patch level moves. `13.6-1` below is what the host had on 2026-09-15. ```sh pct create local:vztmpl/debian-13-standard_13.6-1_amd64.tar.zst \ --hostname fly-pokemon --ostype debian --unprivileged 1 \ --cores 8 --memory 8192 --swap 0 \ --features nesting=1 \ --rootfs local-zfs:24 \ --mp0 local-zfs:16,mp=/srv/fly/state \ --mp1 bulk-array:600,mp=/srv/fly/media \ --net0 name=eth0,bridge=vmbr0,ip=dhcp \ --onboot 1 --startup order=4,up=60 \ --description "fly demo: flysim/flystage/flybridge/flycast (flybrain/infra)" ``` `--swap 0`, not the design doc's 2048: the host's swap is **0** (read live on 2026-09-15, `docs/design/gpu.md` section 0), so a container swap allocation has no backing store and buys nothing. Per-unit `MemoryMax` is the whole OOM story. `01-create-ct.sh` additionally converges two `lxc.*` sentinel blocks into `/etc/pve/lxc/.conf` — the NVIDIA passthrough block when `GPU=1`, and the `cpuset.cpus`/`cpuset.mems` pinning whenever `CPUSET` is set at all, which includes CPU-only containers like the release box — and restarts the container if either changed. Those are raw `lxc.*` keys, which `pct set` does not accept, so they are not part of the `pct create` line above. See step 5 below and `docs/design/gpu.md` section 1. Substitute `151`/`fly-platformer` for the second container. **VERIFY** the two reserved addresses are free (the router's lease table + `arping`) before creating — see `docs/design/infra.md` section 0's reserved holder table, and request the the router's DHCP reservation by MAC after the container's first boot (the LAN convention is DHCP + a the router reservation, not in-guest static — see that same section for why those two guests are documented hazards for having done it the other way). ### 3. ZFS quota `--mp1 bulk-array:600,mp=/srv/fly/media` in the `pct create` above already sizes that subvolume, which Proxmox's ZFS storage plugin implements on the underlying dataset at allocation time — as **`refquota`**, not `quota`, and the dataset is numbered **per storage pool**, not per mountpoint index. Measured on the dev container (P0 spike run 2): the `bulk-array` volume is `/subvol--disk-0` (it is the first volume this guest has on *that* pool, even though it is `mp1`), while `local-zfs` holds `rpool/data/subvol--disk-0` (rootfs) and `-disk-1` (`mp0`, `/srv/fly/state`). `zfs get quota` on any of them returns `none`, which reads as "no limit" and is wrong. So confirm it like this — find the dataset first, ask for `refquota`, and only then decide whether anything needs setting: ```sh # which dataset is /srv/fly/media, really: pct config | grep '^mp1:' # -> bulk-array:subvol--disk-N,mp=/srv/fly/media zfs list -r -o name,refquota,quota,used,recordsize | grep subvol- # the limit PVE actually set (expect 600G here; `quota` will say none): zfs get -H -o value refquota /subvol--disk-0 # only if refquota is missing or wrong, and with the exact dataset name from above: zfs set refquota=600G recordsize=1M /subvol--disk-0 ``` `refquota` caps what the guest itself can write, which is the runaway-recorder case this guard exists for. It does **not** cover snapshots of that dataset, so if a snapshot schedule is ever pointed at these subvolumes, add a `quota` on top of the `refquota`. This is deliberately **not** run by any script — `docs/design/infra.md` section 1 calls it "the single most important safety measure in this plan" (guests on the bulk array include the neighbouring GPU container, the metrics container, another container on the host and another guest on the host; a runaway recorder with no quota takes the monitoring stack and another project's NFS share down with it), and safety-critical host-level ZFS changes get a human's eyes on the exact number, every time. `verify.sh` checks the *result* of this (via `df` from inside the container, since it never runs `zfs` on the host itself) but does not set it. ### 4. Pin the cpuset, and partition it between the units `--cores 8` does **not** give the container eight cores. PVE turns it into an automatic cpuset of eight host *threads*, and on the host (2 × E5-2660 v3, SMT on, two NUMA nodes) the set it picks is neither whole-cored nor single-socket: the dev container got `3,6,13,15,20,23,26,36` on the P0 spike's second run — six physical cores, two of which contributed both SMT siblings, split three and three across the two sockets. The first run got a different eight-thread set with seven physical cores. This is measured, not theoretical: with the automatic set and the old `RAYON_NUM_THREADS=6`, flysim burned 4.1 of 8 cores, could not hold real time (`fly_lag_seconds` at 31 s eight minutes in) and starved x11grab into ~4 duplicated and ~4 dropped frames a second. **Both halves are scripted now** (2026-09-16), and neither half depends on `GPU`: the pinning is the sim's business, not the encoder's, and the measurement that justifies it was taken on x264. `01-create-ct.sh` writes the host-side `lxc.cgroup2.cpuset.cpus`/`.mems` from `CPUSET`/`CPUMEMS` in the env file, and `05-deploy.sh` generates the in-guest `AllowedCPUs=` drop-ins from that same `CPUSET`, via `lib/common.sh`'s `cpuset_partition` — flysim gets the first `RAYON_THREADS` cpus, flycast gets the last `ENCODER_CORES` (default 2) of what is left, and xvfb/flystage/flystage-web/ pulse/mediamtx share whatever is left over in between — so the thread count and the core count cannot drift apart. flycast gets its own group, separate from the page/xvfb group, because the release container measured Chromium's compositor starving when it shared cores with the x264 encoder (63% of captured frames unchanged, against 2% on the NVENC dev box). `05-deploy.sh` refuses to write the drop-ins until the conf actually carries `CPUSET`, because an `AllowedCPUs=` outside the container's own cpuset leaves `cpuset.cpus.effective` empty and the unit unstartable. `CPUSET` is now eight WHOLE physical cores on one socket (node 0 for the release container, the GPU-local socket; node 1 for the platformer container), not the four-cores-plus-SMT-siblings set `docs/design/gpu.md` section 1 originally specified: a four-core set cannot be partitioned so that flysim owns three whole cores AND Chromium gets its 1.4, and P0 run 2 measured that sharing physical cores is exactly what breaks real time. **Temporarily, the release container is on node 1** (`CPUSET=1,3,5,7,9,11,13,15`, `CPUMEMS=1` — the override block at the bottom of ``, 2026-09-16). The dev container `fly-spike`, the dev/demo box, is running on node 0's eight whole cores; two stream containers on the same physical cores is the sharing the measurement above says breaks real time. The release container moves back to node 0 when the dev container is retired — the env file's own comment carries the procedure, and node 1 is the platformer container's documented home, so the two cannot both be pinned there. The manual procedure below is kept for the diagnosis — what PVE picks, and how to see what it really is: ```sh # 1. See what PVE picked, and what it really is: pct exec -- lscpu -e # ONLINE=yes rows are the container's cpuset for c in ; do \ printf 'cpu%s node=%s siblings=%s\n' "$c" \ "$(cat /sys/devices/system/cpu/cpu$c/topology/physical_package_id)" \ "$(cat /sys/devices/system/cpu/cpu$c/topology/thread_siblings_list)"; done # 2. Replace it with whole cores on ONE socket (host-side, /etc/pve/lxc/.conf): # eight distinct physical cores of node1, no SMT siblings, no cross-node hop. echo 'lxc.cgroup2.cpuset.cpus: 1,3,5,7,9,11,13,15' >> /etc/pve/lxc/.conf pct stop && pct start # 3. Partition it inside the guest so the sim, the browser and the encoder never # share a physical core (drop-ins, since the numbers are per host): pct exec -- mkdir -p /etc/systemd/system/flysim.service.d printf '[Service]\nAllowedCPUs=1,3,5\n' | \ pct exec -- tee /etc/systemd/system/flysim.service.d/cpuset.conf for u in xvfb flystage flycast; do \ pct exec -- mkdir -p /etc/systemd/system/$u.service.d; \ printf '[Service]\nAllowedCPUs=7,9,11,13,15\n' | \ pct exec -- tee /etc/systemd/system/$u.service.d/cpuset.conf; done pct exec -- systemctl daemon-reload pct exec -- systemctl restart flysim.service flycast.service flystage.service ``` `RAYON_NUM_THREADS` in `units/flysim.service` must match the number of physical cores flysim's own `AllowedCPUs` covers — it ships as 3 for that reason, with the measurements in its header comment. `AllowedCPUs` does work through cgroup2 delegation in an unprivileged LXC (`cpuset.cpus.effective` comes back exactly as asked); it is deliberately not committed to the unit files because the numbers are per host and per container. ### 5. Secrets `06-secrets.sh ` reads `pass twitch/-key` on the operator box and pipes it straight into `systemd-creds encrypt` inside the container — see that script's own header for the exact relay shape (direct `pct exec` if run on the host itself, `ssh` relay otherwise, mirroring `docs/design/infra.md` section 4's own flip-to-Twitch example). Nothing here needs a manual step beyond having the `pass` entries populated ahead of time. ### 5. `fly-nvidia-majors.service` (GPU, one time) Required before any `GPU=1` container is provisioned. Full instructions and the reboot ordering are in **`infra/host/README.md`**; the short form, on the host as root: ```sh install -o root -g root -m 0755 infra/host/fly-nvidia-majors.sh /usr/local/sbin/fly-nvidia-majors.sh install -o root -g root -m 0644 infra/host/fly-nvidia-majors.service /etc/systemd/system/fly-nvidia-majors.service /usr/local/sbin/fly-nvidia-majors.sh --dry-run 122 150 151 199 # prints the diff, writes nothing systemctl daemon-reload && systemctl enable --now fly-nvidia-majors.service ``` Why by hand and not in `provision.sh`: it writes `/etc/pve`, which nothing reachable through `pct exec` can do, it has to be ordered against `pve-guests.service`, and its default id list **includes the neighbouring GPU container** (`ml`, the GPU workload in the neighbouring container), whose passthrough majors are currently stale. The operator approved touching the neighbouring GPU container on 2026-09-15 — the majors service is what repairs it, and it is a production container, so that approval is the reason this is not automated. Do not extend the list to other people's containers without the same conversation. The reason it exists at all: the `nvidia-uvm` and `nvidia-caps` majors are allocated dynamically and move across boots (uvm went 509 → 511 on the LLM host on 2026-08-29), so a major written once into a container config goes stale **silently** — `docs/design/gpu.md` section 1. `verify.sh`'s GPU section asserts the unit is enabled and active, and that the conf's majors still match `/proc/devices`. Optional, not scripted, and a separate decision: `nvidia-smi -pm 1` (persistence mode). With a 24/7 client attached it changes little. **Do not** set `nvidia-smi -c EXCLUSIVE_PROCESS` — compute mode stays `Default` or the neighbouring GPU container breaks (`docs/design/gpu.md` section 5). The stock installer `/NVIDIA-Linux-x86_64-580.76.05.run` also has to be present on The host for `02-base.sh` to push into each container. It was there on 2026-09-15. Not the `-custom.run`: only its kernel modules were patched, and those stay on the host. ## The infra-repo update checklist **Do not edit `the operator's infra repo` from this repo or this worktree.** `docs/design/infra.md` section 7 ("Every phase, on completion") lists exactly what to add there, on `the operator's infra repo`'s own `master` branch (`git pull --rebase` first, never force-push, markdown only, no `.sh` artifacts — the operator's infra repo holds documentation, not scripts): - New `the infra repo's flybrain page` describing this whole setup (ports, units, timers, secrets shape, backup shape) at the level of detail `the infra repo's panels page` and the operator's other services already use. - `the host's notes`'s guest table gains rows for the release container `fly-pokemon` and the platformer container `fly-platformer`. - `main.md`'s static reservations table gains the two reserved addresses (once VERIFYed free and reserved in the router). - `the infra repo's panels page` gains the "fly live / rank / uptime" panel tile and the new `panel-bridge` source entry (`docs/design/infra.md` section 5), including the freshness budget (10s) in that doc's per-source table. Do this **after** each rollout phase completes (P0 spike results, P1 provisioning proven, P2 Twitch test channel, P3 public + bridge, P4 second demo), not once at the end — `docs/design/infra.md` section 7 is explicit that this is a per-phase step. ## Testing this repo ```sh infra/tests/lint.sh ``` Runs `shellcheck` (falls back to `bash -n` if not installed) on every script — including `host/` and the extension-less `bin/` launchers — `systemd-analyze verify` (skipped if not installed) on every unit file including `host/*.service`, and a structural check that every `ExecStart*=` directive points at a standard system path, something this repo deploys under `/opt/fly/bin` or `/opt/fly/current`, or a `/usr/local/sbin/` script that exists in `host/`. It also runs two behavioural smoke tests, both entirely local: - `bin/flycast-launch --print nvenc` / `--print x264` assemble the real ffmpeg command and are checked for the right encoder, the `fps=30:round=near` filter and `-fps_mode:v cfr` on **both** backends, and the absence of a leftover `-r 30`. - `host/fly-nvidia-majors.sh` is run against a fake `/proc/devices` and a temp conf dir: changed-then-unchanged, the live majors written, the block placed before a `[snapshot]` section, and a refusal to write when a major reads empty. This never touches the host, `pct`, any container, or `/etc/pve` — it is static analysis plus two dry runs against a temp directory.