Local Ai Deepseek R1 671b Home Server at $500 Price

Deepseek R1 has been out for a few months now and I got the curiosity spark while tinkering with an “Ultimate HP Z440” setup I was testing based on feedback from my recent $750 24GB VRAM Ai Server video. 512GB of RAM is a solution-looking-for-problem scenario also and I thought why not! I was pretty surprised to see that not only did the single socket HP Z440 work with LRDIMMS in 4×4 but didn’t seem to have any problems aside from an annoying boot screen error that you have to hit enter to bypass. Seriously looking for a way around that issue! In the next few tables I will lay out the testing I did not only on Deepseek R1 671b Q4 but also Gemma 3, QwQ and Cogito on CPU only.

The Ultimate HP Z440 Home Server

Building a dedicated home server that can handle more then 16 pcie lanes falls well into my favorite category of systems, servers and workstations. These systems have more PCIe lanes, usually substantially more cores, substantially more RAM slots and max capacity and important for ai workloads… more than 2 memory channels. That is important to provide higher system bandwidth between the CPU and the RAM in the system. While I did test a lot, I did not test 1 DPC vs 2 DPC as with 8 dimms and 64GB sticks that is 512GB, the minimum for running Deepseek R1 671b locally. I also did not have the 3090 processing, num_gpu=0 was set, even though it was plugged in at first for the E5-2696 V4 test I was running and yes there is a whole separate video on all that out soon. This was validated with no processes in nvtop.

Local Ai Server Base Config

This server is configured for these test as follows for the E5-2650 V4 CPU:

The E5-2696 V4 I also tested but that is a much more expensive chip in the 150 range. While it is generally better overall of course for Broadwell generation CPU with 22 cores and more cache. It has the same socket bandwidth for the top-end Broadwell chips also which is important for inference workloads, at 76.8 GB/s and four RAM channels. It does have 40 pcie lanes on-chip at gen3 speed to round it out.

Local LLM Performance Benchmarks

CONFIG: ARMA TEST – CPU ONLY – E5-2650V4 – 512GB DDR4

CONFIG: ARMA TEST – CPU ONLY – E5-2696V4 – 512GB DDR4

Benchmark Analysis

The Deepseek 671b did see the Armageddon with a twist question hit better performance on extended token generation on the E5-2696 v4 and this is notable. However possibly more notable is that on a 4b model, the 2650 v4 absolutely rocks. I think even Gemma 3 at 12b and q8 is usable for many folks not needing real-time chat speeds at nearly 4 tps. QwQ is a bit of an overthinker and on a CPU, this hurts, especially as it is a 32b model and even q4 performance is low. The amount of RAM demand for CPU inference however needs to be examined in a more dedicated manner to determine what dimm per channel layout and performance you in certain configs I do not know off the bat, but will figure that out eventually. If I could double performance on Gemma 3 12b q8 with the 2696 v4 I would be very impressed. I doubt it will line up like that however but do bookmark the site and check back in a week or so.

$500 Local Ai Server Tips and Tricks

• The Conclusions section of the video has A LOT of answers to common questions, my reasoning, tips, and pitfalls talked about that are not listed in this article. You should checkout that part of the video no doubt.
• Opt for hitting 4 populated dimm slots. I suspect this will yield the highest BW. Will validate soon.
• Deepseek R1 Q4 in a Guff is about 500GB in size. Hence the 1TB SSD in the build. Do note that even with a very good SATA SSD like the Samsung Evo 870 in 1TB, which has a bigger cache and that lines highest base r/w rate at ~550MB/s, it is still really slow to load! You need to adjust any timeouts to a minimum of 15 mins. You should also be careful not to bump it out of RAM with other models at the same time.
• Deepseek R1 is a great and revolutionary reasoning model, however the nature of reasoning models is they take time to arrive at an answer. This does slow down interactions and especially in CPU based setups, a lot. I do use DS 671B on my Rome EPYC but not every day, hence it is a secondary I use for specific tasks. At the typical 3 tps I get on that system with it, it takes a long time to get me an answer. I do ask it some of the most challenging questions I have and more complex code tasks. We will see R2 likely pretty soon, and I would be surprised to see the size go down. I think it is likely goes up.
• Other CPU Models run pretty decently on this machine at lower quants or lower b sizes. The Q4 Gemma 3 4b is in what I call my happy zone even, above 15 tps. A good additional model to run and augment is the Q8 Gemma 3 12b, but it runs at just under 4 tps on the 2650v4
• I only tested the 2696 v4 on Deepseek R1 671 so far, but I will test on the other models listed here and then we can see if the difference we saw between the two chips holds up. I’ll post that back here when I get it done.
• Both CPUs idle at ~60w in the system populated as seen in the video. The 2650 v4 hits 160s max and the 2696 v4 hits 180s max.
• I suspect QPI now is highly to blame for the inference speed results I got during the first DS R1 tests on the R930’s. Those quad socket machines are NOT for the faint of heart, be warned.
• HP Also has other cheap “higher end” Z lineup towers, the Z640 and the Z840 you might consider

Full Video Review

Hilarious Armageddon with a Twist response during video

gemma3:12b-it-q8_0