flybrain/infra/host/README.md

infra/host — the two files that run on the host itself

Everything else under infra/ runs against a container. These two run on the host, because they write /etc/pve/lxc/<id>.conf, which no pct exec can reach, and because they have to be ordered against pve-guests.service, which is not a per-container concern.

They are installed by hand, by the operator, once. provision.sh does not install them and must not: a boot-ordered host unit that rewrites a production container's config (the neighbouring GPU container) is not something a per-demo provisioning script should be doing behind the operator's back.

File	Installs as
fly-nvidia-majors.sh	/usr/local/sbin/fly-nvidia-majors.sh, mode 0755, root:root
fly-nvidia-majors.service	/etc/systemd/system/fly-nvidia-majors.service, mode 0644

Background: docs/design/gpu.md section 1 ("The dynamic nvidia-uvm major"). /dev/nvidia-uvm's and /dev/nvidia-caps's majors are allocated dynamically and change across boots (uvm moved 509 → 511 on the LLM host on 2026-08-29), so a literal major written once into a container config goes stale silently. 195 (nvidia0 / nvidiactl / nvidia-modeset) is compile-time static and never drifts.

Install

From a checkout of this repo on the host, as root:

install -o root -g root -m 0755 infra/host/fly-nvidia-majors.sh /usr/local/sbin/fly-nvidia-majors.sh
install -o root -g root -m 0644 infra/host/fly-nvidia-majors.service /etc/systemd/system/fly-nvidia-majors.service

# Look before you leap: --dry-run prints the diff it would apply and writes nothing.
/usr/local/sbin/fly-nvidia-majors.sh --dry-run 122 150 151 199

systemctl daemon-reload
systemctl enable --now fly-nvidia-majors.service
systemctl status fly-nvidia-majors.service     # want: active (exited), Main PID ... (code=exited, status=0/SUCCESS)

Expected first-run output on a host that has never had this: 122 changed (its stale c 509:* / c 234:* lines get corrected), and missing for any fly CT that does not exist yet. A second run must print unchanged for every existing CT — if it does not, something else is rewriting those configs and that needs explaining before going further.

A pct stop/pct start will shuffle the file, and that is normal. PVE re-emits the conf on every lifecycle operation with its own keys sorted, every comment hoisted to the top and every raw lxc.* key moved to the end, so after one container restart the # BEGIN fly-nvidia / # END fly-nvidia pair sits at the top of the file with nothing between it and the lines it generated sit at the bottom. That is expected. The script converges lines rather than bytes and still reports unchanged (verified on the host, 2026-09-16, and asserted in infra/tests/lint.sh); if it ever reports changed on a second consecutive run, that is the bug to chase.

Installed on the host on 2026-09-16 (GPU run for the dev container). First run: 122 changed, 199 changed, 150/151 missing. The neighbouring GPU container's own c 509:* rwm line is left alone on purpose: 509 is mei (the Intel ME interface) in this host's /proc/devices today, not nvidia-uvm, and the script never deletes an allow line for a major that resolves to a non-NVIDIA device. It is reported on every run and wants a by-hand deletion from whoever owns the neighbouring GPU container.

the neighbouring GPU container is in the default id list on purpose, and the operator approved it (2026-09-15). It is a production container running the GPU workload in the neighbouring container on this same card, its majors are currently stale (docs/design/gpu.md section 0), and including it repairs them. It is also the reason this file exists as a hand-install step rather than a scripted one. Do not extend the list to other people's containers without the same conversation.

pct stop / pct start is needed for a config change to take effect on a running container — lxc.* keys are read only at container start. At boot that is automatic (this unit is Before=pve-guests.service). Mid-life, restart the container yourself; infra/01-create-ct.sh does that for the fly CTs when it sees the block change.

Reboot order

docs/runbook.md's "the host reboot order" section carries the operational version of this. The short form:

nvidia-devnodes.service   (materialises /dev/nvidia*)
    ->  fly-nvidia-majors.service   (rewrites the majors into the guest configs)
        ->  pve-guests.service      (starts the containers, which read those configs)

Both NVIDIA units must complete before pve-guests.service. The fly containers' startup order=4,up=60 is what keeps them behind it. After any reboot of the host, the check that matters is infra/verify.sh <release-env>, whose GPU section asserts that the majors in the conf still match /proc/devices and that every /dev/nvidia* inside the container is a character device rather than the empty regular file that bind,optional,create=file leaves behind when the source node was missing at start.

Optional host hygiene, not installed here

• nvidia-smi -pm 1 (persistence mode). With a 24/7 client attached it changes little, and it is a host-wide change, so it stays a deliberate the operator decision rather than a line in a script. docs/design/gpu.md section 5.
• Do not set nvidia-smi -c EXCLUSIVE_PROCESS. Compute mode stays Default; exclusive mode would let whichever container got there first lock the card and would break the neighbouring GPU container.