When you click on links to various merchants on this site and make a purchase, this can result in this site earning a commission. Affiliate programs and affiliations include, but are not limited to, the eBay Partner Network. As an Amazon Associate I earn from qualifying purchases. #ad #promotions
This is guide 1 in our series on setting up a local ai homelab with Proxmox 9. YOU HAVE TO HAVE FOLLOW THE GUIDES SEQUENTIALLY IF UP TO Step 2, Llama.cpp and Unsloth setup guide! We create a backup you will see me using in ALMOST EVERY future guide, you need it also.
Step 1 Ollama and OpenWEBUI: THIS GUIDE
Step 2 Llama.cpp and Unsloth setup guide
Step 3 vLLM setup guide
ARTICLE IS UPDATED
9/23/2025 – added details on rig
10/30/2025 – updated to support current CUDA version 3.13 which recent drivers support
This Local Ai Server Guide does not sacrifice on the quality of the end result, which is critical, but does reduce the number of things to be managed and removes docker completely as factor. Removing Docker does not directly impactful to the overall end product at all and makes many tasks very much easier to understand and alter. Removing Docker from the equation also moves us away from a company that while having many fully open source components, does limit and may limit in the future other parts of their ecosystem to paid services. Incus is the way, and LXC is the current path. Do note that ALL of this also works WITHOUT GPUs as well but at slow speed and smaller models. This would impact your experience with various models and sizes.
Self Hosted Ai Server Goals
Hardware Overview
I will be using the quad 3090 rig which we have covered previously as it will allow me to show you how to use multiple GPUs in the setup, but the same steps work exactly the same for a single 3060 12GB $350 ai server build. This works on ANY rig even if you only have CPU you can still run inference on smaller models like a 1B or 3B very performantly.
Quad Ai Rig – UPDATED GUIDE w/alternatives
BIOS Settings
Important to disable UEFI if you can in your motherboard as it will provide clearer resolution if there are issues and I recommend setting up CSM mode as well so you can have older gear work easily. You performance wise will not miss out on anything unless you have the latest and greatest gear or Intel Dedicated GPUs. You will gain clarity if you run into issues and make troubleshooting “missing” devices much easier. Also ensure you have enabled iommu, which is usually default on. I strongly recommend setting up 2x drives for storage in RAID1 so if one fails you do not hit downtime while you replace the failed drive nor lose data. Adding in extra storage SSD/HDD is also a good idea, but realistically getting a home NAS is the best bet and mapping that into Proxmox as a NFS or SMB/CIFS share to store models is a great idea. Do not setup RAID1 with the motherboards hardware options, we will do that inside Proxmox directly with ZFS and software RAID1.
Installing Proxmox
Visit Proxmox Website Here and grab the latest iso downloadable. At this time it is 9.0-1. Use rufus.ie or balena ethcher to burn the ISO to a USB drive if you do not have a BMC capable server or workstation. If you do have a KVM you can just mount the ISO to the virtual CD drive and install that way. Insert your USB and boot up.
Select Install Proxmox VE (Graphical) and accept at the license screen and you will get the hard drive selection screen. Click options…
Then you need to select from the dropdown zfs (RAID1) which will give you a software raid1 in case an NVMe fails. You can select other options from here if you want, but this is a very safe option. It will have in “disk setup” view the hard drives connected to your system listed. If you have multiple hard drives, you need to select only the ones you want to utilize for the installation. WARNING, ANY DATA IS OVERWRITTEN ON THE DRIVES. Next select “Advanced Options” on this same view. I highly recommend subtracting 1 GB from your partition setup so in case you overrun the disk and it locks up you can use a boot rescue usb to boot from and resize your partition and then it will reboot. Do not mess with the other options, they are set appropriately.
Set your country and timezone, create a password, set an email address (this email address will get sent notifications from your system. I recommend setting a valid email. It does not get sent to proxmox or anything) The default username is root just FYI. Now lets set your host on the network. Proxmox runs on a machine and runs as a server so you can access this from any client (browser) on your network. You can also access if from the Proxmox app (official) in the google app store for android here: https://play.google.com/store/search?q=proxmox also Proxmox has a GREAT mobile interface, check it out on a tablet! You can makeup an internal Hostname or if you have an internal hostname set it here. Note the subdomain in the front, pve, is a default. You could change this to something like prox0 (literal name) or gpu1 (functional name) or orion (fun name) or whatever, but it does need to be set. This does not have to be a real domain and if you do not have an internal domain you can use .lan (normal ppl) or even .home.arpa (if you are some sort of spec freak) as you will not be using this to access your server. You need to set a valid IP however in your network range. IMPORTANT use a IP Address that does NOT reside inside your DHCP reservation range. Often your gateway and DNS will be in that same range like 192.168.1.1 if you have a normal setup.
After that it will install and reboot.
Login to your Proxmox Host Machine
Go to a client machine and input your IP address in in the above instance that would be https://192.168.1.40:8006 and enter root for your username and whatever password you entered. Move past the nag screen, you can disable that here in a moment.
Initial Proxmox Configuration
You MUST do this before the rest of the steps below of you will not have a successful install of the drivers that will survive updates of the Kernel in Proxmox.
Click the hostname (prox0 if you are following along, on the left hand side) and then repositories. Disable the “enterprise” repos. Add a repo. Add no-subscription. Click refresh and then update. Apply the updates.
It will need a reboot after this step.
Alternatively, you can run the Proxmox Helper post installer script featured in the video: https://community-scripts.github.io/ProxmoxVE/scripts?id=post-pve-install which also can remove the nag screen.
Install OpenWebUI LXC
Visit the Community Helper scripts openwebui link to copy the link to the openwebui container. You can and should review the source scripts as well by pasting in the script uri to a browser window and reviewing.
On the HOST terminal you need to paste this in and choose advanced. This will allow you to set a root password for your LXC container and customize other options. Pay attention to your Disk size, it is easy to grow an LXC container but not easy to shrink it. You will also very likely find and like certain models and use them often, and remove others. This will free up space on your disk. 35Gb is a decent starting place recommendation, it is just what I used for illustration. If you have 12GB VRAM, I would suggest 60GB. If you have 24GB VRAM I would suggest 100GB. If you have 48GB VRAM then 200GB disk would be reasonable and if you have 96GBVRAM 400GB+ would be likely.
Set all your desired params and start the installation. It will take a decent amount of time to create the LXC container. Do not close this, it will halt and ask you if you want to install Ollama at some point. For this guide you need to select Y however you could also install a separate container with Ollama as well easily. If you do that, you need to select that new network address later in the OWUI user interface.
This script is updatable in the future as well. Just enter the container shell, and type update and it will update both OpenWEBUI and Ollama. After it is finished installing, do not yet shut down the LXC Container. We will be pushing a driver into it and installing.
Install NVIDIA Driver on Proxmox Host
IF YOU HAVE BLACKWELL 50×0 series Nvidia GPU’s Use MIT Drivers and not Proprietary version during installation. The same package is downloaded, but different option selected during installation process.
First you need to update the Proxmox Host system. Please follow all these steps in this order or you will have to rerun the initramfs install portion later after you have a new kernel installed from proxmox. I do NOT recommend installing from the debian sources as that is an older version of the nvidia driver which can and will have issues with some newer cards and can and will have lower performance.
Now reboot if for some reason you need to hit enter still on your terminal.
Next visit the nvidia unix drivers archive page here: https://www.nvidia.com/en-us/drivers/unix/ and click the Latest Production Branch Version: XXXXXX link.
This will take you to that drivers page. Hover the Download button and right click, copy link.
Return to your proxmox HOST terminal and type wget then paste in the link. Hit enter. It will download that file to your location in your proxmox HOST machine.
Now type ls and view the name of the downloaded file. It should start with NVIDIA-blah-blah.run essentially. Next grant execute permissions for yourself to install.
hit enter. You can now execute this installer.
To execute the installer type the following:
You will get a popup and be presented options. IF YOU HAVE BLACKWELL 50×0 series Nvidia GPU’s Use MIT Drivers and not Proprietary version during installation. The same package is downloaded, but different option selected during installation process. Follow the video selection and choose Nvidia Proprietary (ADA or lower cards). Do not worry about X (that is a desktop in linux land) and the 32 bit warnings. It will warn you about modprobe and headers. Click install those. If it demands a restart first, let it restart. It SHOULD.
When it is rebooted come back and rerun
Now install again and this time let it finish. Run nvtop to display the GPUs on the host terminal.
You should see all of your GPUs presented here. They will have a default wattage for the gpu, but we are going to adjust this down which will help idle and have very minimal impact on your GPU performance for inference.
Enable GPU Passthrough Reboot Persistence
Type Crontab -e and then select nano option and hit enter. At the top, type in the following. CTRL-X to save, then at reboot you will have this automatically applied and it will prevent the host GPU from failing to initialize as a side benefit.
At the top of the file we will enter the following:
It will look like this:
Now we need to install the Nvidia driver inside the LXC container.
Install NVIDIA Driver in LXC Container
First update the container OS.
Take note of the container number ID. In the host terminal still type ls -la. It will print the full name of the nvidia driver. It may look something like this: NVIDIA-Linux-x86_64-580.76.05.run
Type the following:
Enter the LXC container:
Grant exec privilege inside the container:
Install the driver with a no kernel modules flag:
This time you can just install Nvidia Proprietary with all the defaults it presents.
This should be a very straightforward installation. You can shutdown the container now.
Next we need to physically passthrough the GPUs to the container as follows.
Passthrough GPU/s to the LXC container
In the proxmox HOST terminal type:
It will print your devices. They will read something like this:
Click on your openwebui container 100 (or whatever it is called/numbered) and then select Resources, Add, Device Passthrough.
It will bring up a popup that you can use to put the /dev/nvidia0 and related entries into. That popup looks like this. You need to fill in several of these, one for each of the entries displayed in the terminal.
If you have 4 GPUs you would end up with an LXC Resources tab that looks like this when all 4 GPUs are passed through. If you only had 1 GPU, it would be just nvidia0. They are orange until they are applied, or if the container is shutdown they will just apply when started. To apply if a container is running, you need to stop and start the LXC container, not just restart. The same applies for VMs FYI as a management concept for attaching certain types of hardware.
Once it is started, enter the terminal and run nvtop again. It should display your GPUs like this inside the container terminal now.
Almost done!
Now reboot the machine to test this functions properly. The hard part is now over! It is a good time to backup the LXC container before you make changes inside it “just in case” and proxmox makes this super easy and fast! We will use the amazing snapshot functionality I demonstrated in the video for this.
Create a Backup
This is good to do for all your containers as it allows them to be restored. There is a LOT of additional things that you can do that are built into proxmox for backups, they even have a backup server. For now lets keep it simple and run a single backup. Select your container, then backup, then backup now, then possibly add some notes like {{guestname}} – initial GPU setup and click backup. It will take a bit but then you have a easy, portable, backup to restore to fast. Note this is NOT the same as snapshots. That is a whole other concept that I would not get into right away as it can get complex. If you have a NAS or connected portable usb drive, you can copy this backup off this host. A good idea to eventually tackle.
Resize your Proxmox 9 Hosted LXC container
I set this in the video to 25GB initially, and had to increase it to 40GB. LLMs are not small files when you get to the capability sets of a quad 3090 rig with 512GB ram, so be conservative in your allocations to avoid running out of space. This can be done by clicking the container, resources, root disk, volume actions, resize volume, then growing the allocated space. I outlined some suggested sizes to start at based on your amount of VRAM above. This is ADDED space. Note there is NOT an easy surefire way to shrink this! You can try traditional methods to shrink a drive, but expect failure! Do not go super large or you will regret it when you see how many additional LXC containers we are adding in here, unless you have a few, like 8 TB’s, of fast SSD or NVMe storage space that is.
This ONLY can grow your LXC disk size. Input the number of NEW EXTRA GiB space you want. In this instance, I have selected 10 GiB.
Now we see I have 60 GiB of space for the LXC container to use.
You do not need to resize your space inside the LXC container.
OpenWebUI Overview
Okay here is possibly the craziest annotation screencap you have ever seen. This will get you familiar with the interface that you land on first for OpenWEBUI which is a great, but somewhat complex interface. I am either sorry or you are welcome for this insanity.
Downloading a Local Ai LLM Model from Ollama in OpenWEBUI
There you have it! You now know how to accomplish all the most common functionality in Proxmox 9 to get an OpenWEBUI + Ollama container up with full GPU passthru and even most of the common things like snapshotting and growing space you are sure to run into and be better off if you use. If you have questions about any of this, or find anything that is not working as expected please drop a comment in the video below. I monitor and answer the most recent How-To videos, which this is.
How To Run Deepseek V3.1 in Ollama
Thanks to Unsloth you can run deepseek easily if you have 168GB of RAM/VRAM in combo at a Q1 quant. To do this just use the command below to pull the model
If you want to run a bigger quant in ollama you can follow along with the steps to merge the gguf files. You will have to have llama.cpp installed at that point however so likely better to just use that for running at that point. Follow our Llama.cpp guide to get up and running easily.
BELOW IS THE MOST RECENT VIDEO THAT COVERS PROXMOX 9 + OLLAMA + OPENWEBUI SETUP.