When you click on links to various merchants on this site and make a purchase, this can result in this site earning a commission. Affiliate programs and affiliations include, but are not limited to, the eBay Partner Network. As an Amazon Associate I earn from qualifying purchases. #ad #promotions
This machine is KILLER! I love that it has so much flexibility also, something lacking from the bare metal approach we did the first time around. GPUs are expensive, so having them be used by all the services that can use them helps their overall value to my homelab.
You DO NOT have to have Multiple GPUs to take advantage of this, even a single GPU is best deployed in this manner if you need to multipurpose use it. You DO NOT have to have a Workstation or Server to take advantage of this. A single GPU desktop all these same steps apply, but you may run into VRAM limits and slowdowns as a result. Aim for 32GB system RAM if possible, but 16 should be okay as well. Any CPU will work.
VIDEO Accompanies Written Tutorial Below
Here are the commands and steps to get you setup and running ASAP. At the time of writing the specs and versions are as follows.
How to Install Proxmox for Ai Server with LXC Docker passthru
Steps to Passthru and Setup a Proxmox
GPU LXC DOCKER Powerhouse
Download Proxmox latest PVE version
Download RUFUS and burn your USB thumb drive
Put the thumb drive into Desktop, Workstation or Server and power it up. If you have a iKVM, congrats!
Check your monitor to enter BOOT MENU. Once there select the thumb drive you just made and install to EMPTY Drive. Your data will be overwritten so you have been warned. Installing to a ZFS Mirror is a great idea!
Once greeted with the proxmox install screen fill in the details of your location, Boot drive to install to, and Networking.
Proxmox Tips and Tricks
Update your Proxmox Server and at the end reboot
Install Nvidia Drivers for GPUs for Proxmox
You need to install the non-free, also called proprietary, NVIDIA drivers that support the Proxmox host system. As of the time of this writing that is Proxmox 12 version based off Debian 12. You may be prompted to restart at various times. Do so.
apt install pve-headers-$(uname -r)
curl -fSsL https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/3bf863cc.pub | gpg –dearmor | tee /usr/share/keyrings/nvidia-drivers.gpg > /dev/null 2>&1
apt update
apt install dirmngr ca-certificates software-properties-common apt-transport-https dkms curl -y
echo ‘deb [signed-by=/usr/share/keyrings/nvidia-drivers.gpg] https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/ /’ | tee /etc/apt/sources.list.d/nvidia-drivers.list
apt install nvidia-driver cuda nvidia-smi nvidia-settings nvtop
Nvidia Container Toolkit for Proxmox LXC
apt install nvidia-container-toolkit
nvidia-ctk runtime configure –runtime=docker
reboot
ls -al /dev/nvidia*
Copy and paste the above to a seperate notepad. You can see I have 4 GPUS numbers 0,1,2,3 which I will address in the next part.
nano /etc/pve/lxc/XXXYOUR ID HEREXXX.conf (mine is 100 for instance, 100.conf)
Add the following values added into your conf at the bottom, paying attention to the device numbers . Mine may differ from yours.
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 511:* rwm
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia1 dev/nvidia1 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia2 dev/nvidia2 none bind,optional,create=file
lxc.mount.entry: /dev/nvidia3 dev/nvidia3 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
REBOOT
Reopen shell and run nvtop cmd. Do you see your GPU/s? If so proceed.
Install Docker for Proxmox in an LXC for GPU Passthru
You can get a docker LXC template as well as many others from: https://tteck.github.io/Proxmox/
We are using the Docker LXC template provided.
bash -c “$(wget -qLO – https://github.com/tteck/Proxmox/raw/main/ct/docker.sh)”
I specified my extra space endowed 4TB NVMe’s for this to install to, and used defaults.
Once installed, you should adjust Disk space, RAM, and Processors allocated
sudo apt update -y && sudo apt upgrade -y && apt install gpg nvtop glances git
curl -fSsL https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/3bf863cc.pub | gpg –dearmor | tee /usr/share/keyrings/nvidia-drivers.gpg > /dev/null 2>&1
apt install dirmngr ca-certificates software-properties-common apt-transport-https dkms curl -y
echo ‘deb [signed-by=/usr/share/keyrings/nvidia-drivers.gpg] https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/ /’ | tee /etc/apt/sources.list.d/nvidia-drivers.list
apt update
apt install nvidia-driver cuda nvidia-smi nvidia-settings -y
apt install nvidia-container-toolkit
nvidia-ctk runtime configure –runtime=docker
nano /etc/nvidia-container-runtime/config.toml
uncomment and edit the line and change it to read as follows. Im unsure on the next commands direct security implications. Do let me know if there are any please in the videos comments.
no-cgroups = true
CTRL-X to exit, and save.
REBOOT
Run NVTOP, you should see your GPUs presented.
mkdir -p /opt/stacks /opt/dockge
cd /opt/dockge
/opt/dockge
docker compose up -d
browse to your Proxmox hosts IP address with :5001 appended for the port #
In dockge, use the docker paste conversion field. Paste this in:
docker run -d -p 3000:8080 –gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data –name open-webui –restart always ghcr.io/open-webui/open-webui:ollama
It will convert it to compose, click save, and run.
Create your openwebui login credentials (not sent out from localnet) and then download a model and test it out!