Proxmox VE / Ubuntu 24.04 / NVIDIA CUDA
Installing NVIDIA Driver and CUDA 12.9 on Tesla V100 in Ubuntu 24.04 VM on Proxmox VE #
This is a guide to PCIe passthrough Tesla V100 to an OVMF virtual machine on Proxmox VE and install
NVIDIA driver and CUDA Toolkit 12.9 on the Ubuntu 24.04 side.
It also includes workarounds for BAR0 is 0M @ 0x0 /This PCI I/O region assigned to your NVIDIA device is invalid
which are common in OVMF environments.
Prerequisites for This Article #
- IOMMU / VFIO settings on the Proxmox host side are already complete
- GPU passthrough to guest is functional
- Guest OS is Ubuntu 24.04 LTS
- This article assumes a compute-use VM
Verification Configuration #
Proxmox VE / KVM
OVMF (UEFI)
q35
Tesla V100 PCIe 32GB
Ubuntu 24.04 LTS
12.9
Why Use CUDA 12.9 #
Tesla V100 is a Volta-generation GPU. Starting with CUDA 13 and later, offline compilation and
library support for Volta have been dropped, so it is safer to keep the V100 development environment
fixed on CUDA 12.x series. This article uses CUDA Toolkit 12.9.
STEP 1
Configure VM on Proxmox Side for GPU Passthrough #
First, configure the virtual machine for GPU passthrough.
If OVMF / q35 / Host PCI settings are already complete, proceed to the next step.
qm stop 101
qm set 101 --bios ovmf
qm set 101 --machine q35
qm set 101 --hostpci0 65:00,pcie=1
qm start 101Note: 101 is the VMID example, 65:00 is a GPU PCI address example.
Replace with values from your actual environment.
STEP 2
Expand MMIO Aperture in OVMF Environment #
This is the biggest pitfall. In OVMF UEFI VMs, GPU BAR resources are often not sufficiently mapped,
resulting in BAR0 is 0M @ 0x0 orThis PCI I/O region assigned to your NVIDIA device is invalid.
In such cases, add X-PciMmio64Mb to
expand the MMIO aperture.
Adding this setting actually resolved the issue wherenvidia-smi failed to work properly on OVMF virtual machines in Proxmox.
qm stop 101
qm set 101 --args '-fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=262144'
qm start 101Key Point: After the change, perform a
complete VM shutdown and then start it again, not just a guest OS reboot.
STEP 3
Install Required Packages on Ubuntu 24.04 #
Before installing the NVIDIA driver and CUDA Toolkit, install kernel headers and build-related tools.
sudo apt update
sudo apt install -y linux-headers-$(uname -r) gcc g++ make wget pciutilsSTEP 4
Add NVIDIA Official Repository #
Add the NVIDIA official APT repository for Ubuntu 24.04.
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt updateSTEP 5
Install NVIDIA Driver #
Fix the driver to the 580 series before installation. V100 is still supported by the 580 series.
sudo apt install -y nvidia-driver-pinning-580
sudo apt install -y cuda-driversIf you want a compute-only configuration without GUI, you can also
switch to a minimal libnvidia-compute and nvidia-dkms-based setup
depending on your use case, but it is clearer to first verify normal operation with cuda-drivers.
STEP 6
Fix and Install CUDA Toolkit 12.9 #
To avoid tracking the latest CUDA version, use the metapackage cuda-toolkit-12-9 to
lock to the 12.9 series.
sudo apt install -y cuda-toolkit-12-9If Capacity is Tight: The full Toolkit is quite large, so you can start
with just runtime and compiler.
sudo apt install -y cuda-runtime-12-9 cuda-compiler-12-9
# Add development libraries later if needed
sudo apt install -y cuda-libraries-dev-12-9STEP 7
Set PATH and Reboot #
Set PATH and library path to make nvcc easier to use, then reboot.
echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
sudo rebootSTEP 8
Verify Operation #
After reboot, verify that the GPU is visible, the driver loads properly,
and CUDA compiler is usable.
source ~/.bashrc
uname -r
lspci | grep -i nvidia
nvidia-smi
nvidia-smi -L
nvcc --version
dmesg -T | grep -E 'NVRM|BAR0|nvidia'If installed correctly, nvidia-smi will display the Tesla V100,
and nvcc --version will display CUDA 12.9.
Troubleshooting #
BAR0 is 0M @ 0x0 Appears #
This is almost certainly an OVMF MMIO aperture shortage. First, applyqm set 101 --args '-fw_cfg name=opt/ovmf/X-PciMmio64Mb,string=262144',
perform a complete VM shutdown, and then start it again.
nvidia-smi Cannot Communicate with Driver #
The driver kernel module may not have loaded. First, review the OVMF MMIO
settings, then check the module status.
lsmod | grep -E 'nvidia|nouveau'
sudo journalctl -k -b | grep -Ei 'nvidia|nouveau|secure|module verification'Disk Space Runs Out During Toolkit Installation #
The full Toolkit is large, so installation may fail midway on a VM with limited free space.
In that case, clean the cache and switch to a minimal package configuration for safety.
df -h /
sudo apt clean
sudo apt autoremove -y
sudo apt --fix-broken installSummary #
Installing NVIDIA driver and CUDA Toolkit on a Tesla V100 in an Ubuntu 24.04 VM
on Proxmox VE is not difficult in itself. However, on OVMF virtual machines,
insufficient MMIO aperture can prevent the NVIDIA driver from starting, so
configuring X-PciMmio64Mb becomes a practical key point.
When building the same configuration going forward, it is best to configure
OVMF / q35 / PCIe and
MMIO aperture expansion together from the start.
Reference Information #
Disclaimer #
This article has been created based on the verification environment at the time of writing. We do not warrant
the accuracy, completeness, usefulness, or operation in specific environments of the content described.
The procedures and behavior described may differ depending on version differences or update status of OS, kernel, GPU, firmware, BIOS/UEFI, Proxmox VE,
NVIDIA driver, CUDA Toolkit, etc.
- The procedures in this article include GPU passthrough, kernel module installation, and startup configuration changes.
- Before implementation, prepare backup, snapshot, and maintenance plans and verify thoroughly in a test environment.
- We are not responsible for any defects, downtime, failures, data loss, or other damages caused by using the content of this article.
- Implementation in production environments is at the user’s discretion and responsibility.
- Product names, service names, and company names mentioned are trademarks or registered trademarks of their respective companies.
