Proxmox

Proxmox VE

Production infrastructure demands production discipline. Every Proxmox configuration must be secure by default, redundant where it matters, and automated where possible.

Route to Reference

• VM, LXC, and OCI container management — [${CLAUDE_SKILL_DIR}/references/vm-and-lxc.md]: VM vs LXC vs OCI

comparison, OCI container support (PVE 9.1 tech preview), Docker-on-Proxmox decision guidance, configuration options, template workflows, linked vs full clone trade-offs

• Storage backends — [${CLAUDE_SKILL_DIR}/references/storage-backends.md]: Backend capability matrix, ZFS tuning

(ARC/L2ARC/SLOG/volblocksize), Ceph configuration, LVM-Thin monitoring, storage selection decision tree

• Networking — [${CLAUDE_SKILL_DIR}/references/networking.md]: Bridge configuration, VLAN layout, bonding modes,

SDN zones (VXLAN/EVPN), MTU considerations, OVS vs Linux bridge, firewall lockout prevention

• Clustering and HA — [${CLAUDE_SKILL_DIR}/references/clustering-and-ha.md]: Corosync configuration, quorum math,

QDevice setup, split-brain prevention, fencing methods, HA groups, migration, quorum loss recovery

• API and automation — [${CLAUDE_SKILL_DIR}/references/api-automation.md]: REST API architecture, pvesh/qm/pct

reference, Terraform patterns, cloud-init customization (cicustom, network v2), hookscript lifecycle, CI/CD pipelines

• Backup strategies — [${CLAUDE_SKILL_DIR}/references/backup-strategies.md]: vzdump modes, PBS architecture,

encryption key management, garbage collection safety, verification jobs, retention policies, off-site sync patterns

Guest Management

VM vs LXC vs OCI Decision

• Default to LXC for trusted Linux workloads — near-zero overhead, high density
• Use VMs when the workload requires: non-Linux OS, full kernel isolation, PCIe passthrough, or live migration
• Use OCI containers (PVE 9.1+, tech preview) for single-purpose microservices from Docker Hub/GHCR — lightweight

deployment without a Docker VM. Not suitable for multi-container stacks or workloads requiring Docker Compose

• Use a Docker VM for full Docker/Compose/Kubernetes workflows, multi-container stacks, or advanced networking

(macvlan, overlay). Still the most flexible and compatible option for container-heavy workloads

• Use unprivileged containers (default) — they map container UID 0 to a non-privileged host UID, preventing

container escape attacks

• Only use privileged containers when unprivileged mode is incompatible (specific device access, certain NFS mounts)

VM Configuration

• Use VirtIO drivers for disk and network — mandatory for production performance
• Enable QEMU Guest Agent inside every VM for graceful shutdown, snapshot consistency, and IP address reporting
• Use OVMF (UEFI) + Q35 machine type for PCIe passthrough and Secure Boot
• Set CPU type to host for maximum performance in homogeneous clusters; use x86-64-v2-AES or the lowest common CPU

generation when live migration across different CPU generations is required

• Enable memory ballooning for dynamic RAM management
• Enable NUMA topology for VMs with many cores on multi-socket hosts

Container Configuration

• Set explicit memory limits — containers without limits can exhaust host RAM
• Enable nesting (features: nesting=1) only when required (Docker inside LXC)
• For Docker workloads in LXC: unprivileged + nesting=1 + keyctl=1 — but note this is unsupported and can break

on host updates (CVE-2025-52881 broke Docker-in-LXC setups; workaround: lxc.apparmor.profile: unconfined)

• Use bind mounts to share host directories, not NFS/CIFS mounts inside the container
• PVE 9.0 removed cgroup v1 entirely — containers requiring cgroup v1 must move to VMs

OCI Application Containers (PVE 9.1 — Tech Preview)

• Pull OCI images from Docker Hub/GHCR/Quay and run as LXC containers — no Docker engine required
• Limitations: no in-place updates, no Docker Compose, no orchestration, no shell in most containers
• Use for: single-purpose lightweight services; use a Docker VM for multi-container stacks
• See [${CLAUDE_SKILL_DIR}/references/vm-and-lxc.md] for full OCI details, Docker-on-Proxmox decision guide, and

Docker VM best practices

Templates and Cloning

• Build templates with the guest agent and cloud-init pre-installed
• Convert to template: qm template or backup as .tar.gz for containers
• Use linked clones for development/ephemeral workloads (fast, shared base disk)
• Use full clones for production (independent, no template dependency)

Storage

Backend Selection

• Local redundancy, data integrity, snapshots → ZFS
• Local snapshots/clones without ZFS overhead → LVM-Thin
• Shared storage for HA clusters (3+ nodes) → Ceph
• Simple shared storage (existing NAS/SAN) → NFS or iSCSI
• Deduplicated backups → PBS

ZFS Rules

• Use HBA in IT mode — never hardware RAID controllers with write-back cache
• ARC sizing: cap explicitly in /etc/modprobe.d/zfs.conf; 25-30% of host RAM for mixed workloads. PVE 8.1+

defaults to 10% (max 16GB) for new installs. Rule of thumb: 2GB base + 1GB per TB of storage

• Set ashift=12 for 4K sector disks (most modern drives); incorrect ashift halves IOPS due to sector misalignment
• Volblocksize (VMs on zvols): 16k for mirrors/RAID10; increase for wide RAIDZ to reduce write amplification. Can

only be set at zvol creation

• Recordsize (containers on datasets): tune per workload — 8k for Postgres, 16k for MariaDB, 1M for large sequential

files

• Enable compression=lz4 — can actually increase I/O performance by writing less data to disk
• SLOG: 8-32GB enterprise NVMe with power-loss protection; only holds ~5s of write data. Benefits sync-heavy

workloads (databases, NFS)

• L2ARC: only if ARC hit ratio is low and adding RAM is not possible; size 5-20x RAM; budget 1GB RAM per 50GB of

L2ARC for metadata

• Schedule regular scrubs (weekly or monthly)
• Plan capacity upfront — ZFS pools cannot be shrunk
• Never create swap on a ZFS zvol — causes blocking I/O during backups

Ceph Rules

• Minimum 3 nodes for proper quorum and data distribution
• Dedicated 10GbE+ network for Ceph traffic (25GbE+ recommended for NVMe OSDs), separate from Corosync — Ceph rebalance

traffic saturates links and causes Corosync instability

• Use SSD/NVMe for OSD WAL/DB
• Dedicated disks for OSDs — never share with the host OS
• PVE 9.0 defaults to Ceph Squid (v19.2)

Storage Anti-Patterns

• Running ZFS or Ceph on top of hardware RAID with write-back cache — defeats data integrity guarantees
• Overprovisioning LVM-Thin without monitoring — a full thin pool causes I/O errors for all guests on that pool
• Storing backups on the same physical disks as production data
• Letting ZFS ARC consume unbounded RAM — causes VM crashes when the hypervisor and VMs compete for memory

Networking

Core Model

Every guest connects to a Linux bridge. Use VLAN-aware bridges (single bridge with 802.1Q tagging) instead of per-VLAN bridges. The VLAN-aware checkbox must be explicitly enabled — it is off by default.

VLAN Best Practices

• Place the management interface on a dedicated VLAN — never share with guest traffic
• Configure trunk ports on physical switches for the Proxmox host — frames with VLAN IDs not allowed by the switch

are silently dropped

• Assign VLAN tags per guest in the network configuration
• Verify no double-tagging mismatch between Proxmox and switch native VLAN

Traffic Separation

• Management / Corosync — 1GbE (dedicated)
• Ceph cluster + public — 10GbE (dedicated), 25GbE recommended
• Migration — 10GbE (recommended)
• Guest — depends on workload

Critical rule: Never combine Corosync traffic with high-bandwidth Ceph or migration traffic on a single 1GbE link. Corosync is latency-sensitive — network contention causes cluster instability and false fencing.

SDN

Use SDN (VXLAN zones) for overlay networking across nodes without physical switch changes. Use EVPN for advanced multi-tenant setups with BGP routing. SDN is fully supported and installed by default since PVE 8.1.

SDN gotchas:

• VXLAN adds 50-byte header — set VNet MTU to 1450 (or 1370 with IPSEC)
• SDN changes are staged, not live — click Apply at Datacenter level
• DHCP requires a gateway configured on the subnet
• Multiple EVPN exit nodes require disabling rp_filter in sysctl
• OVS vs Linux bridge: OVS is automatically VLAN-aware and may resolve 10GbE throughput bottlenecks seen with native

Linux bridge

SDN-Firewall integration (PVE 8.3+): SDN automatically generates IPSets for VNets and IPAM-managed guests — use these in firewall rules for simplified maintenance. The nftables firewall can filter forwarded traffic at host and VNet levels (e.g., restrict SNAT or inter-zone traffic).

Fabrics (PVE 9.0+): Automated routing between cluster nodes using FRRouting with OpenFabric (IS-IS-based) or OSPF. Fabrics simplify underlay network configuration for Ceph full-mesh and EVPN/VXLAN deployments.

Clustering and High Availability

Cluster Rules

• Use a dedicated network for Corosync — latency under 5ms required
• Configure redundant Corosync links (up to 8 supported via Kronosnet) on separate physical networks
• For 2-node clusters, deploy a QDevice on a third machine for quorum — a 2-node cluster without QDevice is a

split-brain generator

• QDevice is discouraged for odd-numbered clusters — it becomes a single point of failure due to (N-1) vote allocation
• If using LACP bonds for Corosync, set bond-lacp-rate fast on both node and switch — default slow rate has 90s

failover, causing fencing after ~60s

• Avoid balance-rr, balance-xor, balance-tlb, balance-alb bond modes for Corosync — they cause asymmetric

connectivity and mass fencing

• Never join a node with existing VMs/containers to a cluster — start fresh
• Update nodes one at a time — the LRM requests a service freeze from the CRM during updates; if both are updating,

the watchdog fences the node

HA Requirements

• Shared or replicated storage accessible from all HA nodes
• Working fencing mechanism — test before relying on HA
• Minimum 3 quorum votes (3 nodes, or 2 nodes + QDevice)
• Configure HA groups with node priorities for controlled failover

Fencing

Fencing guarantees a failed node is offline before its services restart elsewhere. HA without fencing is a data corruption risk — two nodes writing to shared storage simultaneously causes irrecoverable damage.

• Verify watchdog status: ha-manager status
• Test fencing by simulating node failure before going to production
• Use hardware watchdog (iTCO_wdt via /etc/default/pve-ha-manager) when available, software watchdog (softdog) as

fallback

• Configure WATCHDOG_MODULE=iTCO_wdt in /etc/default/pve-ha-manager

Quorum Loss Recovery

If the cluster loses quorum, pmxcfs becomes read-only — no VM operations are possible. Emergency recovery:

• pvecm expected 1 forces single-node quorum — use only to restore vital guests or fix the quorum issue itself
• Never make cluster changes (add/remove nodes, storage, guests) while expected votes are overridden

Migration

• Live migration (VMs only): requires shared/replicated storage, brief pause at cutover
• vGPU live migration (PVE 8.4+): VMs using NVIDIA vGPU (mediated devices) can now be live-migrated between nodes

with compatible GPU hardware — previously required shutdown

• VMs with full PCIe passthrough devices still cannot be live-migrated — use cluster-wide resource mappings for HA
• Offline migration (VMs and containers): guest stops, data transfers, guest starts on target
• Use a dedicated high-bandwidth network for migration traffic

API and Automation

Authentication

• Use API tokens with privilege separation for all automation — never root credentials
• Store token secrets in vault or environment variables — never in code
• Use ticket authentication only for interactive or short-lived tools

CLI Tools

• pvesh — direct REST API access from CLI
• qm — VM lifecycle management
• pct — container lifecycle management
• ha-manager — HA resource management
• pvecm — cluster management
• pvesm — storage management

Terraform

• Use API tokens with privilege separation — never root credentials
• Use cloud-init templates as the base for Terraform-managed VMs — templates must have qemu-guest-agent installed or

Terraform hangs on "still creating"

• Store Terraform state remotely (GitLab HTTP backend, S3, Consul) — never local state for shared infrastructure
• Use lifecycle { ignore_changes } for fields Proxmox modifies outside Terraform (e.g., disk size after manual resize)
• CI/CD pipeline: validate -> plan (save artifact) -> apply (manual trigger)
• Never commit API secrets to git — use CI/CD variables (TF_VAR_*)

Cloud-Init

• Prepare a base VM with qemu-guest-agent and cloud-init installed
• Add a Cloud-Init drive (IDE or SCSI CD-ROM)
• Configure network, SSH keys, and user data via the Cloud-Init panel or API
• Convert to template, deploy via linked clone
• Use SSH key authentication — cloud-init password storage is less secure
• cicustom for advanced needs: reference custom YAML snippets for user, network, and meta data from a

snippets-capable storage

• Store cicustom snippets on shared storage (CephFS) in clusters for HA
• Windows templates: use Cloudbase-Init with configdrive2 format + Sysprep

Backups

Backup Rules

• Use PBS for production — deduplication, incremental backups, verification, encryption
• Use snapshot mode for VMs (crash-consistent, no downtime); zstd compression
• Follow the 3-2-1 rule: 3 copies, 2 media types, 1 off-site
• Retention: keep-daily=7,keep-weekly=4,keep-monthly=6,keep-yearly=1

PBS Security

• Restrict PVE backup user/token to create-only access (no delete) on PBS
• Separate PBS admin credentials from PVE access
• Store encryption keys separately from the backed-up system — password manager + offline backup
• Never disable gc-atime-safety-check; use dedicated remote users per sync job

Non-Negotiable

• Test restores regularly. A backup that cannot be restored is worthless.
• Monitor backup jobs. A silently failing backup is worse than no backup.
• Document the restore procedure.

PCIe Passthrough

Requirements

• CPU: VT-d (Intel) or AMD-Vi enabled in BIOS/UEFI
• IOMMU enabled in kernel: intel_iommu=on or amd_iommu=on
• VFIO modules loaded: vfio, vfio_iommu_type1, vfio_pci, vfio_virqfd
• Dedicated IOMMU group for the passthrough device

Configuration

• Use OVMF (UEFI) + Q35 machine type; SeaBIOS if GPU lacks UEFI ROM
• Blacklist host driver or bind via vfio-pci IDs in /etc/modprobe.d/
• Pass through all device functions — GPU requires video + audio; USB-C controllers must also be bound to vfio-pci
• For GPU: x-vga=1 for primary, vga: none; output via physical monitor, dummy plug, or Looking Glass
• VMs with full passthrough cannot be live-migrated — use resource mappings (/cluster/mapping/pci) for HA

GPU-Specific Issues

• NVIDIA Error 43 (Windows): set CPU type to host, add options kvm ignore_msrs=1 in /etc/modprobe.d/kvm.conf
• AMD reset bug (Vega, Polaris, some Navi): GPU fails to reset after VM shutdown, preventing reuse without host

reboot. Fix: install vendor-reset kernel module for vendor-specific reset quirks. RDNA2+ generally unaffected

• NVIDIA vGPU: officially supported since vGPU Software 18 on PVE. Requires valid NVIDIA entitlement. Ampere+ GPUs

need SR-IOV enabled first via pve-nvidia-vgpu-helper. PVE 8.4+ supports **live migration of vGP

…

Source & license

This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code.

• Author: xobotyi
• Source: xobotyi/cc-foundry
• License: MIT

Install and usage instructions live in the source repository linked above.