dkp/dkp-skills-and-conventions-share
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
Page: Building a Debian VM on Proxmox for agents
Pages
Bootstrapping opencode Building a Debian VM on Proxmox for agents Home Rules and mechanisms Sending into a shared git channel part 1 rules Sending into a shared git channel part 2 mechanisms
1 Building a Debian VM on Proxmox for agents
dkp edited this page 2026-08-30 18:59:49 -04:00
Table of Contents

Building a Debian 13 VM on Proxmox, for running agents

A step-by-step build guide for provisioning a full VM — not an LXC container — on a Proxmox VE node, with its disk on a redundant ZFS array, provisioned by cloud-init and reachable by SSH key. The end result is a machine intended to host coding agents.

Verified against: Proxmox VE 9.2.10 (pve-manager/9.2.10), kernel 7.0.14-11-pve on the host; Debian 13 "trixie" from the official generic cloud image, kernel 6.12.107, in the guest. Built and verified 2026-08-30.

Audience: a peer engineer and that engineer's AI agent. Every step carries an explicit verification command and the output that means "correct", because an agent cannot eyeball a GUI.

Provenance marks, used throughout:

  • [v] — verified: this exact command was run and its output was seen.
  • [c] — claim: passed through unchecked, or reasoning rather than measurement. This is this page's spelling of the wiki's [unverified] convention.

An unmarked statement is prose, not a claim about a command's behaviour.

On the numbers in this guide. Per the wiki's convention: values shown in the verification blocks are measurements from one real machine, not illustrations, and they are left concrete on purpose so an agent has something to pattern-match against. Every environment-specific value is also collected once in the Substitution Table (§2) and referenced as {{NAME}} in the command bodies — so re-deriving them for your own machine means editing that table, not hunting through the prose. Do re-derive them. Sizes, VMIDs, addresses and pool names will all differ on yours.

A note on what this page is not. It describes how one machine was actually built. It does not run, install, or ask anything of a reading agent. Several steps record a decision that went the other way on the day, and why — those are kept deliberately, because the reasoning is the transferable part.

1. Prerequisites

Before starting, you need all six of these. The guide does not create them.

# Prerequisite How to check it
1 A Proxmox VE node, version 8 or 9 §3 verifies this without credentials
2 Root shell access to that node over SSH, by key §4
3 An SSH keypair on your workstation ssh-keygen -l -f ~/.ssh/id_ed25519.pub [v]
4 A Proxmox storage pool with images content and free space §5
5 A Linux bridge on the node attached to your LAN §5
6 Outbound HTTPS from the node, to fetch the cloud image §6

Today's build used PVE 9.2.10 on a single (non-clustered) node [v]. Nothing below is 9-specific except the import-from form in §8, which needs PVE 7.2 or newer [c].

2. Substitution table — every environment-specific value, in one place

Replace {{NAME}} everywhere it appears. The "today's value" column is what the reference build used; re-derive each one for your own machine rather than copying it.

Placeholder Meaning Today's value
{{PVE_HOST}} SSH target for the Proxmox node spicy (192.168.3.201)
{{PVE_FQDN}} Node's certificate CN spicy.home.arpa
{{VMID}} Numeric guest ID, free in both qm list and pct list 133
{{VM_NAME}} Proxmox-side VM name itslocal
{{STORAGE}} Proxmox storage ID backed by the redundant array bulk-array
{{ZPOOL}} The ZFS pool {{STORAGE}} sits on bulk-storage
{{BRIDGE}} Linux bridge on the LAN vmbr0
{{CORES}} vCPU count 12
{{RAM_MB}} RAM in MiB 32768
{{DISK_SIZE}} Final disk size 500G
{{CIUSER}} cloud-init username created in the guest kala
{{PUBKEY_LOCAL}} Path on your workstation to the ONE public key to inject ~/.ssh/id_ed25519.pub
{{VM_IP_CIDR}} Static address + prefix for the VM 192.168.3.204/16
{{VM_IP}} Same address without the prefix 192.168.3.204
{{GATEWAY}} LAN default gateway 192.168.0.1
{{IMG_URL}} Cloud image URL see §6
{{IMG_PATH}} Where the image lands on the node /tmp/debian-13-generic-amd64.qcow2

The two values you must not guess: {{STORAGE}} and {{VM_IP_CIDR}}. Guessing the first can silently land a 500 GB disk on the boot disk. Guessing the second is what the /16 trap in §12.1 is about.

3. Identify the hypervisor — without any credential

Useful when you are handed a host and told "there's a hypervisor on it". Proxmox VE serves its UI and API on fixed TCP 8006, and the banner is unauthenticated.

curl -k -s -S -m 10 -i https://{{PVE_HOST}}:8006/

VERIFICATION — correct output contains all three of these [v]:

HTTP/1.1 200 OK
Server: pve-api-daemon/3.0
<title>spicy - Proxmox Virtual Environment</title>

Server: pve-api-daemon is the tell. The CSS cache-buster carries component versions [v for the string, c for the interpretation]:

<link ... href="/pve2/css/ext6-pve.css?ver=9.2.10" />      -> pve-manager 9.2.10
<link ... href="/pwt/css/ext6-pmx.css?ver=v5.2.7-..." />   -> widget-toolkit 5.2.7

The REST API, by contrast, is closed [v]:

curl -k -s -S -m 10 -i https://{{PVE_HOST}}:8006/api2/json/version

→ HTTP/1.1 401 No ticket. So there is no credential-free path to storage, capacity or the guest list. You need §4.

Do not over-read the TLS certificate. OU = PVE Cluster Node appears on every PVE node including a standalone one; it does not mean the node is clustered [c].

4. Get a shell — and read §7 before you choose the account

ssh -o BatchMode=yes -o StrictHostKeyChecking=yes -o ConnectTimeout=5 root@{{PVE_HOST}} id

VERIFICATION — correct output [v]:

uid=0(root) gid=0(root) groups=0(root)

BatchMode=yes matters: it makes the probe fail fast instead of hanging on a password prompt, which is what an unattended agent needs.

If you get Permission denied (publickey,password) — that message is identical whether your key is missing or the account does not exist. Today that ambiguity cost hours: only a non-root account had been tested, and the conclusion "we are locked out" was wrong; root authenticated fine on the first try [v]. Test root@ explicitly before concluding you lack access.

Then confirm the version from inside:

ssh -o BatchMode=yes root@{{PVE_HOST}} pveversion

VERIFICATION: a line beginning pve-manager/9. (today: 9.2.10) [v].

5. Recon — storage, capacity, free VMID, bridge, network prefix

Run all of these before creating anything. Each is read-only.

5.1 Which storage pool is the redundant array?

ssh -o BatchMode=yes root@{{PVE_HOST}} pvesm status

VERIFICATION — today's output [v]:

Name              Type     Status     Total (KiB)      Used (KiB) Available (KiB)        %
bulk-array     zfspool     active     23275080349      3542208037     19732872312   15.22%
local              dir     active       174683904        15057920       159625984    8.62%
local-zfs      zfspool     active       279044948       119418924       159626024   42.80%

bulk-array is the array (≈22 TiB total, ≈18 TiB free). local-zfs is the boot pool — that is the one you must not land on if the requirement is "storage on the array". Set {{STORAGE}} from this table, never from memory.

5.2 Is it thin or thick? — read storage.cfg, not pvesm

ssh -o BatchMode=yes root@{{PVE_HOST}} cat /etc/pve/storage.cfg

VERIFICATION — today's output [v]:

dir: local
	path /var/lib/vz
	content import,iso,backup,vztmpl

zfspool: local-zfs
	pool rpool/data
	content rootdir,images
	sparse 1

zfspool: bulk-array
	pool bulk-storage
	content images,rootdir
	mountpoint /bulk-storage

The trap: local-zfs has sparse 1; bulk-array does not. No sparse line means thick provisioning — a 500 GB disk genuinely reserves ~507 GB on the pool the moment you create it, rather than growing as it fills [v].

Plan capacity on the provisioned size, not the used size. If you want thin behaviour, that is a storage-config change on the node (out of scope here, and someone's decision, not an agent's).

Confirm the reservation after the build:

ssh -o BatchMode=yes root@{{PVE_HOST}} zfs list -o name,used,refreservation,volsize {{ZPOOL}}/vm-{{VMID}}-disk-0

VERIFICATION: refreservation is non-zero and close to volsize on a thick pool [v]. On a thin pool it reads none.

5.3 Array health

ssh -o BatchMode=yes root@{{PVE_HOST}} zpool list
ssh -o BatchMode=yes root@{{PVE_HOST}} zpool status {{ZPOOL}}

VERIFICATION: state: ONLINE, and errors: No known data errors [v]. Today: raidz1 across 3 spinning disks, plus one SATA SSD as a cache (L2ARC) vdev; 32.7T raw, 29.3T free [v]. Do not provision onto a DEGRADED pool.

5.4 Free VMID — check BOTH lists

Containers and VMs share one ID space on Proxmox. A VMID is free only if it is absent from both.

ssh -o BatchMode=yes root@{{PVE_HOST}} qm list

VERIFICATION — today [v]:

      VMID NAME                 STATUS     MEM(MB)    BOOTDISK(GB) PID
       112 poly-build-vm        stopped    32768             64.00 0
       131 blade-forge          running    16384             80.00 7392
       132 blade-vmtest         stopped    2048               8.00 0
       900 instr-template       stopped    4096              10.00 0

ssh -o BatchMode=yes root@{{PVE_HOST}} pct list

VERIFICATION — today [v]: 16 containers at IDs 100 103 105 110 120 121 122 123 124 130 142 143 202 203 206 211.

This list is a moving target. Container 121 appeared in one reading and was gone from a later one the same day, because it was live-migrated to another node [v]. Re-run pct list immediately before qm create, not from notes taken an hour earlier.

Union of the two: 100 103 105 110 112 120 121 122 123 124 130 131 132 142 143 202 203 206 211 900. 133 is free in both — hence {{VMID}}=133 [v].

qm create refuses a VMID already used by a VM, but checking pct list too is what stops you from picking an ID a container holds [c].

5.5 The bridge

ssh -o BatchMode=yes root@{{PVE_HOST}} ip -br link

VERIFICATION — today's relevant lines [v]:

vmbr0            UP             44:a8:42:1e:be:c1 <BROADCAST,MULTICAST,UP,LOWER_UP>
vmbr1            UNKNOWN        4a:3f:8d:e9:cf:c7 <BROADCAST,MULTICAST,UP,LOWER_UP>

Two bridges exist. vmbr0 is the LAN one — proven by the next check, not by its name.

5.6 The network prefix — verify it, do not assume /24

ssh -o BatchMode=yes root@{{PVE_HOST}} ip -4 addr show {{BRIDGE}}
ssh -o BatchMode=yes root@{{PVE_HOST}} ip route show default

VERIFICATION — today [v]:

6: vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...
    inet 192.168.3.201/16 scope global vmbr0

default via 192.168.0.1 dev vmbr0 proto kernel onlink

/16, not /24. The whole first phase of today's work assumed a /24 and was wrong [v]. A /16 here means hosts are scattered across 192.168.0.x, 192.168.3.x and 192.168.10.x, and the gateway (192.168.0.1) is outside the /24 you would have guessed. A VM configured 192.168.3.204/24 with gateway 192.168.0.1 would not route at all.

5.7 Capacity

ssh -o BatchMode=yes root@{{PVE_HOST}} free -h
ssh -o BatchMode=yes root@{{PVE_HOST}} lscpu

Today: 188 GiB host RAM, ZFS ARC capped at 64 GiB [v]. Budget the VM's RAM against what the running guests already hold, not against total.

6. Download the cloud image (on the node)

Debian's generic image is the direct equivalent of Ubuntu's cloudimg — a pre-built, cloud-init-enabled disk image, not an installer ISO. There is no qm install-media step and no console-driven installer.

ssh -o BatchMode=yes root@{{PVE_HOST}} wget -nv -O {{IMG_PATH}} https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2

VERIFICATION — correct output [v]:

2026-08-30 14:14:23 URL:https://<mirror>/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2 [433651712/433651712] -> "/tmp/debian-13-generic-amd64.qcow2" [1]

Byte counts equal on both sides of the slash, and the redirect lands on a Debian CDN mirror (today chuangtzu.ftp.acc.umu.se) — that is normal, not a hijack [v].

Confirm the file:

ssh -o BatchMode=yes root@{{PVE_HOST}} ls -l {{IMG_PATH}}

VERIFICATION: size ≈ 433 MB (433651712 bytes today) [v].

Ubuntu translation, for a reader who knows Ubuntu:

Ubuntu Debian 13 equivalent
noble-server-cloudimg-amd64.img debian-13-generic-amd64.qcow2
default cloud-init user ubuntu default cloud-init user debian
cloud-images.ubuntu.com cloud.debian.org/images/cloud/trixie/latest/
snap, PPAs neither exists; apt only

You override the default user with --ciuser in §8, so debian vs ubuntu only bites you if you forget to set it and then try to log in as ubuntu.

Integrity, and an honest gap: Debian publishes SHA512SUMS and a detached SHA512SUMS.sign alongside the image. Today's build did not verify the checksum [v — it was not run]. If you care, fetch SHA512SUMS from the same directory and check before importing. Stated as a gap rather than papered over.

7. ⚠ CREDENTIAL DECISIONS — read this section before running §8

Five decisions. Each was made deliberately today; each is stated with its reasoning so you can make it differently on purpose rather than by accident.

7.1 THE BIG ONE — --sshkeys /root/.ssh/authorized_keys injects EVERY key

Proxmox's qm set --sshkeys <file> takes a file of public keys and hands it to cloud-init, which writes it into the new user's ~/.ssh/authorized_keys in the guest. Every guide on the internet shows the convenient form:

# DO NOT DO THIS unless you have read the file and want every key in it
qm set {{VMID}} --sshkeys /root/.ssh/authorized_keys

The hazard, stated generically: a hypervisor's root authorized_keys is usually a multi-party file. It accumulates the key of everyone who has ever needed to administer the node — colleagues, a vendor, an old laptop, a decommissioned CI runner. Seeding a new VM from it silently grants every one of those parties login to the new machine, as your {{CIUSER}}, with whatever sudo rights cloud-init gives that user. Nobody is notified. It does not show up as a "sharing" action; it shows up as a one-line convenience flag. On the reference node that file held four keys belonging to more than one person [v], and the VM's user has passwordless sudo [v] — so the grant is root in the guest, not a shell.

The safe recipe. Steps 2 and 3 below are exactly what was run today, and they worked [v]. Step 1 is this guide's addition — it was NOT run today, so treat it as recommended rather than proven [c].

# 1. Look at the file first. Count the keys. Decide, per key, on purpose.
ssh -o BatchMode=yes root@{{PVE_HOST}} 'wc -l < /root/.ssh/authorized_keys'
ssh -o BatchMode=yes root@{{PVE_HOST}} 'cut -d" " -f3 /root/.ssh/authorized_keys'

# 2. Copy exactly the ONE key you intend to grant, to its own file.
scp -o BatchMode=yes {{PUBKEY_LOCAL}} root@{{PVE_HOST}}:/tmp/onekey.pub

# 3. Point --sshkeys at THAT file.
ssh -o BatchMode=yes root@{{PVE_HOST}} qm set {{VMID}} --sshkeys /tmp/onekey.pub

Step 1's second command prints only the comment field of each key (usually an email or user@host), not the key material — enough to identify them, without copying secrets around.

VERIFICATION that only one key landed — check inside the guest after boot:

ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} 'wc -l < ~/.ssh/authorized_keys'

Correct output: 1. If it is 4, you used the wrong file; the fix is to edit the guest's authorized_keys, and to treat the other three keyholders as having had access in the interim.

Cleanup: /tmp/onekey.pub is a public key, so leaving it is not a disclosure — but delete it anyway so the next person does not reuse a stale file believing it is current.

7.2 SSH host-key acceptance — verified TOFU, and what it costs

Two ways to accept a new host's key:

# Blind trust-on-first-use. Accepts whatever key answers, no questions asked.
ssh -o StrictHostKeyChecking=accept-new {{CIUSER}}@{{VM_IP}} id

versus

# Verified TOFU, route A -- the serial console, which needs no network at all.
# `--serial0 socket` in 8.1 exists for this. cloud-init prints an
# "SSH HOST KEY FINGERPRINTS" banner to the console on first boot.
ssh -o BatchMode=yes -t root@{{PVE_HOST}} qm terminal {{VMID}}
#   (press Enter; scroll back to the fingerprint banner; ^O to exit qm terminal)

# Verified TOFU, route B -- via the guest agent. ONLY works after Section 10,
# so it does not help on a first connection. Use A, or accept that B is
# a re-verification rather than a first one.
ssh -o BatchMode=yes root@{{PVE_HOST}} "qm guest exec {{VMID}} -- ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub"

# Then, from your workstation, over the network you do not yet trust:
ssh-keyscan -t ed25519 {{VM_IP}} 2>/dev/null | ssh-keygen -lf -
# A human compares the two SHA256 strings and accepts only if identical.

Provenance on that block: the ssh-keyscan ... | ssh-keygen -lf - form was run today, against a different host, and printed a fingerprint [v]. Routes A and B were not run and are reasoned, not measured [c] -- route B in particular has the ordering problem noted in its comment.

What accept-new actually costs you: it trusts whatever answers on that address at that moment. On a flat LAN with no DHCP reservations, an address can be held by something other than what you think — and accept-new will record that impostor's key permanently, after which the real host trips a REMOTE HOST IDENTIFICATION HAS CHANGED warning and looks like the attacker.

Recommendation: verified TOFU. It costs exactly one human step — a person comparing two fingerprint strings — and that step cannot be delegated to the agent doing the connecting, because an agent that can read the fingerprint over the untrusted channel is not independently verifying anything.

Honest record of what happened today, since this guide should not claim more discipline than was exercised [v]:

  • For the new VM, -o StrictHostKeyChecking=accept-new was used, and it succeeded: Warning: Permanently added '192.168.3.204' (ED25519) to the list of known hosts. Blind TOFU, on a host that had existed for ten minutes on a LAN under the operator's control — low risk, but not the recommended path.
  • For a different host (sloth), the same accept-new flag was denied by the workstation's permission classifier [v], and the fingerprint route (ssh-keyscan | ssh-keygen -lf -) was used instead so a human could verify out of band before accepting.

So the recommendation above is the lesson, drawn from the host where the guard rail fired — not a description of the VM step. Marked plainly because a guide that overstates its own rigour teaches the wrong habit.

7.3 Root-only on the node, versus creating a named admin user

Ruled: root-only, here. The reasoning, not just the ruling:

Proxmox has a real user/ACL system (pveum), and on a node several people administer, named accounts with scoped roles are correct — you get per-person audit trails in /var/log/pve/tasks/, and you can revoke one person without rotating everyone.

It was not done here because:

  1. A named user did not exist and creating one is a write on the node. The task was to build a VM, not to restructure the node's identity model. A provisioning job that quietly adds an admin account is doing something nobody asked for.
  2. qm needs root or near-root anyway. Creating a VM, importing a disk to a ZFS pool and resizing it require VM.Allocate, Datastore.AllocateSpace and Disk.Resize — most of PVEVMAdmin plus storage rights. A named user with those permissions is not meaningfully less privileged for this task; it is better attributed, which is a different benefit [c].
  3. The lower-privilege option was noted and is the right one for recon. A read-only API token (PVEAuditor, scoped to /) answers every question in §5 — storage, capacity, guest inventory, bridges — and can create nothing. If you are only surveying a node, ask for that, not for a shell [c].

Reconsider this if: more than one person administers the node, or you need to answer "who created this VM" later. Then create a named user with PVEVMAdmin on /vms and Datastore.AllocateSpace on the storage.

7.4 An unauthenticated inference endpoint on a flat LAN

The reference environment has a local LLM inference endpoint at http://192.168.3.221:11434 -- an Ollama server in LXC 121 (ollama) on a second Proxmox node (sloth), with two NVIDIA GPUs bind-mounted into the container [v, per this campaign's item-#9 work]. That container was migrated from the first node to the second earlier the same day, which is why an earlier reading of the first node's pct list showed it and a later one did not [v]. Reachability from the new VM was proved, not assumed: curl -s -m 10 http://192.168.3.221:11434/api/tags from inside the VM returns the full model list [v].

State this plainly, because the reader inherits it: HTTP, no TLS, no authentication, on a flat /16 with no segmentation. Anything on that network can use it — enumerate the models, send arbitrary prompts, consume the GPUs, and read whatever any client sends through it. There is no per-client identity, so there is no audit trail and nothing to revoke.

That is an accepted risk on a home LAN with known devices. It stops being acceptable the moment the network carries a guest SSID, an IoT VLAN, or a contractor's laptop. If you replicate this pattern, the honest mitigations are network-level (a separate VLAN, or an nftables rule restricting source addresses), not application-level — Ollama has no auth to turn on [c].

Related: the container's address is DHCP-assigned and will move [v]. Anything that hardcodes 192.168.3.221 breaks on the next lease change.

7.5 Never put a credential on a command line or in a URL

Anything you type as an argument is visible in at least three places you do not control: your shell history file, the process table (ps aux, readable by every user on the box for the lifetime of the process), and any log that records commands — including, on this workstation, the agent transcript itself. A credential in a URL is worse: it additionally reaches proxy logs, server access logs, and Referer headers.

So, never:

curl https://user:token@host/...          # NO
qm ... --password 'hunter2'               # NO
export API_KEY=sk-...                     # NO, if typed interactively

Instead:

  • Prefer key-based auth with no secret in the command at all. Every authenticated command in this guide is an ssh that carries a key by reference, never a password. That is why the guide has no secrets in it.
  • If a token is unavoidable, read it from a file: curl -H @/path/to/header-file or curl --netrc-file /path/to/netrc, with the file chmod 600.
  • For a Proxmox API token, put it in a file and source it in a non-interactive script, so it never enters your interactive history.
  • --sshkeys is fine — a public key is not a credential. It is the selection of which public keys, per §7.1, that matters.

8. Build the VM

Four commands. Run them in this order.

8.1 Create the shell (no disk yet)

ssh -o BatchMode=yes root@{{PVE_HOST}} qm create {{VMID}} \
  --name {{VM_NAME}} \
  --memory {{RAM_MB}} \
  --cores {{CORES}} \
  --sockets 1 \
  --cpu host \
  --net0 virtio,bridge={{BRIDGE}} \
  --scsihw virtio-scsi-single \
  --ostype l26 \
  --agent 1 \
  --serial0 socket

This is the exact command run today (with --name dev-agents; the rename to itslocal came later, §11) [v].

Each flag, since an agent should not have to guess:

Flag Why
--cpu host passes the host CPU's features through; needed for anything using modern instruction sets
--scsihw virtio-scsi-single one virtio-scsi controller per disk; the modern PVE default, enables discard/IO-thread
--ostype l26 tells PVE "Linux 2.6+"; affects device defaults
--agent 1 enables the QEMU guest agent channel hypervisor-side. The guest package is separate — see §10
--serial0 socket gives you qm terminal {{VMID}} as a console. Cloud images write their boot log to serial; without this you are blind if networking fails
no --boot yet set in 8.2, after there is a disk to boot

VERIFICATION:

ssh -o BatchMode=yes root@{{PVE_HOST}} qm config {{VMID}}

Correct output contains name: {{VM_NAME}}, cores: 12, memory: 32768, net0: virtio=...,bridge=vmbr0, agent: 1, and no scsi0: line yet [v].

8.2 Import the disk, attach cloud-init, set user, key and network — one command

ssh -o BatchMode=yes root@{{PVE_HOST}} qm set {{VMID}} \
  --scsi0 {{STORAGE}}:0,import-from={{IMG_PATH}} \
  --ide2 {{STORAGE}}:cloudinit \
  --boot order=scsi0 \
  --ciuser {{CIUSER}} \
  --sshkeys /tmp/onekey.pub \
  --ipconfig0 ip={{VM_IP_CIDR}},gw={{GATEWAY}}

This is today's command, with two deliberate differences flagged below [v].

Reading the arguments:

  • --scsi0 {{STORAGE}}:0,import-from=<path> — the :0 means "allocate a volume sized from the imported image" (the image's own 3 GB). This is the modern replacement for qm importdisk + a separate attach; it does both in one step, and lands the volume on {{STORAGE}} [v].
  • --ide2 {{STORAGE}}:cloudinit — creates the cloud-init drive: a tiny generated ISO carrying the user, key and network config into the guest on first boot. Without this, --ciuser, --sshkeys and --ipconfig0 do nothing — they are only ever delivered through this drive.
  • --boot order=scsi0 — boot the imported disk. No install media, no ide2 in the boot order: the cloud-init drive is data, not bootable.
  • --sshkeys /tmp/onekey.pub — §7.1. Not /root/.ssh/authorized_keys.
  • --ipconfig0 ip=...,gw=... — the static address. §12.1 explains why static rather than ip=dhcp.

Two differences from what literally ran today, both improvements:

  1. Today's command used --ipconfig0 ip=dhcp, DHCP failed to lease, and a second qm set --ipconfig0 ip=192.168.3.204/16,gw=192.168.0.1 plus a stop/start was needed [v]. Setting the static address here skips that.
  2. Today's --sshkeys pointed at /tmp/danyel.pub; renamed to /tmp/onekey.pub for a general reader. Same mechanism [v].

VERIFICATION — the import is not instant. Correct output ends with [v]:

transferred 3.0 GiB of 3.0 GiB (100.00%)
scsi0: successfully created disk 'bulk-array:vm-133-disk-0,size=3G'
generating cloud-init ISO

Both lines matter: successfully created disk names the storage — read it and confirm it says {{STORAGE}}, not local-zfs — and generating cloud-init ISO confirms --ide2 took effect.

Then:

ssh -o BatchMode=yes root@{{PVE_HOST}} qm config {{VMID}}

Correct output now includes [v]:

boot: order=scsi0
ciuser: kala
ide2: bulk-array:vm-133-cloudinit,media=cdrom
ipconfig0: ip=192.168.3.204/16,gw=192.168.0.1
scsi0: bulk-array:vm-133-disk-0,size=3G
sshkeys: ssh-ed25519%20AAAA...%20user%40example.com%0A

Note sshkeys is stored URL-encoded (%20 for space, %0A for newline). That is normal. It also means the key comment — often an email address — is readable in qm config output by anyone who can run it. Redact that line before pasting qm config output anywhere.

size=3G is expected at this point: it is the cloud image's own size. §8.3 fixes it.

8.3 Grow the disk

ssh -o BatchMode=yes root@{{PVE_HOST}} qm disk resize {{VMID}} scsi0 {{DISK_SIZE}}

[v — run today, twice: once to 200G, then to 500G after the size decision changed. Once is sufficient; resizing up again works.]

VERIFICATION:

ssh -o BatchMode=yes root@{{PVE_HOST}} qm config {{VMID}} | grep scsi0

Correct output: scsi0: bulk-array:vm-133-disk-0,size=500G [v].

The trap: this grows the virtual disk, not the filesystem. At this point the guest would see a 500 GB block device with a 3 GB partition on it. The filesystem reaches the new size only because the cloud image runs cloud-init's growpart + resizefs modules on boot, which grow the partition and then the filesystem to fill the device.

Consequences an agent must plan for:

  • Resize before first boot (as here) and it is done in one step.
  • Resize after boot and you must reboot the guest, or run growpart /dev/sda 1 and resize2fs /dev/sda1 inside it by hand.
  • On an image without cloud-init, growpart never runs and the extra space stays invisible forever. This is why the generic/cloudimg images are the right base and a plain installer ISO is not [c].

Verification that growpart actually ran is in §9 — df -h /, not qm config.

8.4 Start

ssh -o BatchMode=yes root@{{PVE_HOST}} qm start {{VMID}}
ssh -o BatchMode=yes root@{{PVE_HOST}} qm status {{VMID}}

VERIFICATION: status: running [v].

9. Verify the guest end-to-end

Do not accept "the start command exited 0" as proof the VM works. Four checks.

9.1 It answers on the network

ping -c 40 -W 1 {{VM_IP}}

VERIFICATION: replies begin within ~30 s of qm start [v]. A cloud image's first boot runs cloud-init before sshd is fully up, so allow a minute.

9.2 The right user exists, with the right key

ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} id

(First connection: accept the host key per §7.2.)

VERIFICATION — today's output [v]:

uid=1000(kala) gid=1000(kala) groups=1000(kala),4(adm),20(dialout),24(cdrom),25(floppy),27(sudo),29(audio),30(dip),44(video),46(plugdev)

27(sudo) is the one that matters — cloud-init put the user in sudo and wrote /etc/sudoers.d/90-cloud-init-users granting ALL=(ALL) NOPASSWD:ALL [v]. That is passwordless root in the guest for every key in authorized_keys — which is why §7.1 is the section it is.

And the key count, per §7.1:

ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} 'wc -l < ~/.ssh/authorized_keys'

VERIFICATION: 1.

9.3 growpart reached the filesystem

ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} df -h /

VERIFICATION — today's output [v]:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       492G  1.2G  471G   1% /

492G against a 500G volume is correct — the difference is GiB-vs-GB plus partition overhead. If this says 3.0G, growpart did not run: check cloud-init status in the guest and that the disk was resized before first boot.

ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} lsblk

VERIFICATION: sda1 occupies essentially all of sda [v].

9.4 The OS is what you think

ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} 'cat /etc/os-release; uname -r'

VERIFICATION — today [v]: Debian GNU/Linux 13.6 (trixie), kernel 6.12.107+deb13-amd64.

10. qemu-guest-agent on Debian — the trap that cost the most time

--agent 1 in §8.1 enabled the hypervisor side. Nothing is listening in the guest until you install the package — and on Debian, installing it is not enough.

Before installing, confirm the failure mode so you recognise it [v]:

ssh -o BatchMode=yes root@{{PVE_HOST}} qm guest cmd {{VMID}} network-get-interfaces

Output before the agent runs: QEMU guest agent is not running [v].

10.1 Install

ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} sudo -n apt-get update
ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} sudo -n DEBIAN_FRONTEND=noninteractive apt-get -y install qemu-guest-agent

(sudo -n = never prompt; it fails loudly instead of hanging an unattended agent on a password prompt. DEBIAN_FRONTEND=noninteractive stops debconf opening a dialog on a pipe.)

VERIFICATION — and here is the trap, in the installer's own words [v]:

Setting up qemu-guest-agent (1:10.0.11+ds-0+deb13u1) ...
qemu-guest-agent.service is a disabled or a static unit, not starting it.

Read that line. On Ubuntu, installing qemu-guest-agent enables and starts it, and you are done. On Debian 13 the unit is static — it has no [Install] section, so it cannot be enabled, and the package deliberately does not start it. It is meant to be device-activated: systemd starts it when /dev/virtio-ports/org.qemu.guest_agent.0 appears, via dev-virtio\x2dports-org.qemu.guest_agent.0.device. On a VM that was already running when you installed it, that device event has already passed, so nothing starts it.

10.2 Start it explicitly

ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} sudo -n systemctl start qemu-guest-agent

[v]

Do not run systemctl enable qemu-guest-agent — it will fail or warn, because a static unit has no install target. That failure is not a problem to fix; it is the design.

VERIFICATION — from the hypervisor side, which is the side that matters:

ssh -o BatchMode=yes root@{{PVE_HOST}} qm guest cmd {{VMID}} network-get-interfaces

Correct output: a JSON array of interfaces including lo and ens18/eth0 with the VM's IPv4 address [v]. The QEMU guest agent is not running message is gone.

10.3 Prove it survives a reboot — do not assume it

Because the unit is device-activated rather than enabled, "will it come back?" is a real question, not a formality. Answer it:

ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} sudo -n systemctl reboot
ping -c 45 -W 1 {{VM_IP}}
ssh -o BatchMode=yes root@{{PVE_HOST}} qm guest cmd {{VMID}} get-osinfo
ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} uptime

VERIFICATION [v]: get-osinfo returns JSON naming Debian 13 after the reboot, and uptime confirms the machine really restarted:

 18:33:29 up 1 min,  1 user,  load average: 0.08, 0.02, 0.01

This was run today and it passed — the device unit does fire on boot [v]. It is in the guide because proving it costs 90 seconds and assuming it costs a mystery later.

Ubuntu translation: on Ubuntu you would write systemctl enable --now qemu-guest-agent and never think about it again. On Debian, --now is the only half that works, and the persistence comes from a different mechanism (device activation) that you should verify once.

11. Rename (optional) — and the caveat that bit today

Proxmox-side rename:

ssh -o BatchMode=yes root@{{PVE_HOST}} qm set {{VMID}} --name {{VM_NAME}}

[v — this is how dev-agents became itslocal.]

VERIFICATION: qm config {{VMID}} | grep name → name: itslocal [v].

The caveat: qm set --name changes the hypervisor's label only. The guest's own /etc/hostname is untouched, and it was set at first boot by cloud-init from the name the VM had then. Verified today [v]:

ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} hostname

→ dev-agents, not itslocal.

So the machine answers to one name in the Proxmox UI and a different one at its own shell prompt. To change the guest's hostname too:

ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} sudo -n hostnamectl set-hostname {{VM_NAME}}

[c — not run today. Note that cloud-init may reset it on next boot unless preserve_hostname: true is set in /etc/cloud/cloud.cfg.]

Cleanest option: pick the final name before §8.1 and never rename.

12. Traps, collected

Each of these cost real time today. They are the part of this guide the official documentation will not give you.

# Trap Where
1 qemu-guest-agent on Debian is a static, device-activated unit — installing it does not start it, and enable does not apply §10
2 Debian's generic = Ubuntu's cloudimg; the default cloud-init user is debian, not ubuntu §6
3 A ZFS pool exposed to Proxmox may be thick (no sparse 1 in storage.cfg) — a 500 GB disk really reserves 500 GB §5.2
4 The guest disk is a zvol, so recordsize does not apply. volblocksize does, and it is immutable after creation §13
5 Growing the disk needs cloud-init's growpart on the next boot to reach the filesystem §8.3
6 Verify the network prefix. Today's LAN is a /16 with the gateway outside the obvious /24, and no DHCP on the segment — hence a static address §5.6, §12.1 below
7 Containers and VMs share the VMID space — a free ID must be free in qm list AND pct list §5.4
8 Permission denied (publickey,password) is identical for "missing key" and "no such account" — test root@ explicitly §4
9 --sshkeys /root/.ssh/authorized_keys injects every key on the node §7.1
10 qm set --name does not change the guest's hostname §11

12.1 No DHCP on the segment

Today's VM was created with --ipconfig0 ip=dhcp, booted, and after several minutes held an IPv6 link-local address and no IPv4 lease [v]. There is no DHCP server on that segment; the other hosts are statically addressed.

How to diagnose this rather than waiting:

ssh -o BatchMode=yes root@{{PVE_HOST}} qm guest cmd {{VMID}} network-get-interfaces

If the guest interface shows only fe80::... and no 192.168.x.x, DHCP is not answering. The fix, applied today [v]:

ssh -o BatchMode=yes root@{{PVE_HOST}} qm set {{VMID}} --ipconfig0 ip={{VM_IP_CIDR}},gw={{GATEWAY}}
ssh -o BatchMode=yes root@{{PVE_HOST}} qm stop {{VMID}}
ssh -o BatchMode=yes root@{{PVE_HOST}} qm start {{VMID}}

A stop/start is required, not qm reboot — the cloud-init drive is regenerated and re-read at machine start, and a warm reboot may not pick it up [c, though the stop/start is what was actually done and it worked [v]].

Pick the static address by checking it is unused first:

ssh -o BatchMode=yes root@{{PVE_HOST}} ip neigh
ping -c 3 -W 1 {{VM_IP}}

VERIFICATION: the address is absent from ip neigh and the ping gets no reply before you assign it.

13. Storage tuning — what you can change, and what you cannot

Worth stating because the obvious ZFS advice does not apply to a VM disk.

A Proxmox VM disk on a zfspool storage is a zvol — a block device — not a filesystem dataset. Therefore:

ssh -o BatchMode=yes root@{{PVE_HOST}} zfs get recordsize {{ZPOOL}}/vm-{{VMID}}-disk-0

VERIFICATION: returns - [v]. recordsize is a dataset property and has no meaning here. Every "tune ZFS recordsize for your workload" article is about datasets, and following it here does nothing.

The property that does apply:

ssh -o BatchMode=yes root@{{PVE_HOST}} zfs get volblocksize,compression {{ZPOOL}}/vm-{{VMID}}-disk-0

Today: volblocksize=16K, compression=lz4 (achieving 1.49x on this volume) [v].

volblocksize is fixed at creation and cannot be changed on an existing zvol [c — documented ZFS behaviour]. If you want a different value you must set the storage's blocksize in /etc/pve/storage.cfg before creating the disk, and recreate. On a 3-wide raidz1 with ashift=12, 16K is a reasonable default and there is nothing to do [c].

compression=lz4 is inherited from the pool and is the right choice for source trees — cheap on CPU, pays for itself against slow disks. Leave it.

14. What this build did NOT do

Stated so nobody assumes it.

  • No development toolchain. The guest is a bare Debian 13 generic cloud image plus qemu-guest-agent — 324 dpkg packages, of which the manually-selected set is exactly that [v]. No git, no node, no tmux, no rg. python3 3.13.5 is present, pulled in as a cloud-init dependency rather than chosen [v]. curl, vim, nano, sudo, ca-certificates are present [v].
  • No image checksum verification (§6).
  • No firewall rules. Proxmox's own firewall was not enabled for this VM, and nothing restricts the guest from reaching the hypervisor's API on 8006.
  • No backup job, no snapshot schedule. qm snapshot {{VMID}} <name> is available and cheap on ZFS; nothing was configured.
  • No swap in the guest [v]. Memory exhaustion is a cliff ending in the OOM killer, not a slowdown.
  • Two kernel limits left at their defaults that will bite a many-process workload before RAM does: fs.inotify.max_user_instances = 128 (a per-uid limit, so every process run by the same user draws from one pool of 128) and the interactive ulimit -n soft limit of 1024 [v]. Raise both if you intend to run many file-watching processes as one user.

15. The complete sequence, condensed

For an agent that has read the above and wants the list. Every line was run today except where marked.

# --- recon (read-only) ---
curl -k -s -S -m 10 -i https://{{PVE_HOST}}:8006/
ssh -o BatchMode=yes -o StrictHostKeyChecking=yes root@{{PVE_HOST}} id
ssh -o BatchMode=yes root@{{PVE_HOST}} pveversion
ssh -o BatchMode=yes root@{{PVE_HOST}} pvesm status
ssh -o BatchMode=yes root@{{PVE_HOST}} cat /etc/pve/storage.cfg
ssh -o BatchMode=yes root@{{PVE_HOST}} zpool status {{ZPOOL}}
ssh -o BatchMode=yes root@{{PVE_HOST}} qm list
ssh -o BatchMode=yes root@{{PVE_HOST}} pct list
ssh -o BatchMode=yes root@{{PVE_HOST}} ip -br link
ssh -o BatchMode=yes root@{{PVE_HOST}} ip -4 addr show {{BRIDGE}}
ssh -o BatchMode=yes root@{{PVE_HOST}} ip route show default
ssh -o BatchMode=yes root@{{PVE_HOST}} free -h

# --- key hygiene (see 7.1) ---
ssh -o BatchMode=yes root@{{PVE_HOST}} 'cut -d" " -f3 /root/.ssh/authorized_keys'
scp -o BatchMode=yes {{PUBKEY_LOCAL}} root@{{PVE_HOST}}:/tmp/onekey.pub

# --- image ---
ssh -o BatchMode=yes root@{{PVE_HOST}} wget -nv -O {{IMG_PATH}} https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2

# --- build ---
ssh -o BatchMode=yes root@{{PVE_HOST}} qm create {{VMID}} --name {{VM_NAME}} --memory {{RAM_MB}} --cores {{CORES}} --sockets 1 --cpu host --net0 virtio,bridge={{BRIDGE}} --scsihw virtio-scsi-single --ostype l26 --agent 1 --serial0 socket
ssh -o BatchMode=yes root@{{PVE_HOST}} qm set {{VMID}} --scsi0 {{STORAGE}}:0,import-from={{IMG_PATH}} --ide2 {{STORAGE}}:cloudinit --boot order=scsi0 --ciuser {{CIUSER}} --sshkeys /tmp/onekey.pub --ipconfig0 ip={{VM_IP_CIDR}},gw={{GATEWAY}}
ssh -o BatchMode=yes root@{{PVE_HOST}} qm disk resize {{VMID}} scsi0 {{DISK_SIZE}}
ssh -o BatchMode=yes root@{{PVE_HOST}} qm start {{VMID}}

# --- verify ---
ssh -o BatchMode=yes root@{{PVE_HOST}} qm status {{VMID}}
ssh -o BatchMode=yes root@{{PVE_HOST}} qm config {{VMID}}
ping -c 40 -W 1 {{VM_IP}}
# accept the host key per 7.2 (verified TOFU), then:
ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} id
ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} 'wc -l < ~/.ssh/authorized_keys'
ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} df -h /
ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} lsblk

# --- guest agent (see 10) ---
ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} sudo -n apt-get update
ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} sudo -n DEBIAN_FRONTEND=noninteractive apt-get -y install qemu-guest-agent
ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} sudo -n systemctl start qemu-guest-agent
ssh -o BatchMode=yes root@{{PVE_HOST}} qm guest cmd {{VMID}} network-get-interfaces
ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} sudo -n systemctl reboot
ping -c 45 -W 1 {{VM_IP}}
ssh -o BatchMode=yes root@{{PVE_HOST}} qm guest cmd {{VMID}} get-osinfo
ssh -o BatchMode=yes {{CIUSER}}@{{VM_IP}} uptime

16. Decisions carried from the original build, not re-argued here

  • A full VM, not an LXC container. Ruled explicitly. This guide does not offer the container path even though it is cheaper.
  • Storage on the redundant array, not the boot disk. Ruled. This is what {{STORAGE}} is for, and §5.1/§8.2's verification exists to catch a wrong substitution.
  • Debian 13. Chosen. Debian-specific behaviour is translated into Ubuntu terms wherever it differs (§6, §10).
  • The VM is named itslocal — renamed from dev-agents after the build (§11). The guest OS hostname is a SEPARATE setting that renaming the VM does not touch; it was initially missed and has since been set to itslocal as well.

17. Corrections this guide makes to the brief it was written from

Recorded because the guide should not silently paper over a wrong brief.

  1. accept-new was NOT refused for the VM. The initial brief for this guide said the session was denied StrictHostKeyChecking=accept-new and a human verified the fingerprint out of band. That happened for a different host (sloth). For this VM, accept-new was used and succeeded [v]. §7.2 states both, and presents verified TOFU as a recommendation rather than a description.
  2. --sshkeys, --ciuser, --ide2, --scsi0 and --ipconfig0 were one qm set, not five steps [v]. §8.2 keeps them together, as run.
  3. The disk was resized twice (200G then 500G) [v]. The guide gives the single-resize form and notes the history.
  4. --ipconfig0 ip=dhcp was set first and failed; the static address came after, with a stop/start [v]. §8.2 sets static up front and §12.1 keeps the diagnosis.
  5. The rename to itslocal is Proxmox-side only; the guest still calls itself dev-agents AT THE TIME THIS GUIDE WAS WRITTEN [v]. FIXED SINCE: hostnamectl set-hostname itslocal was run and /etc/hosts corrected; hostname -f now returns itslocal [v]. The underlying point stands and is the reason §11 exists -- the Proxmox VM name and the guest OS hostname are independent, and renaming one does not rename the other.
  6. qm create did not carry --scsi0, --cdrom or --boot. Earlier draft command sketches in the campaign record show an ISO-install shape that was never run; the cloud-image path replaced it [v].