Testing out some of the great new ZFS functions from the Unraid team’s latest releases has been a fun way to get deeper into the specifics of how this fun and unique operating system handles the users shares for clients. It is easy to get decent performance with just a few SSDs which does not come as a big surprise. The performance we see with spindle based hard drives however is surprisingly good with the out of the box ready configurations that Unraid brings for the GUI experience most users will utilize. Let’s take a look at several configurable settings and plugins that I used to create a great end to end experience for a 148TB Raid Z1 pool over 40Gbit smb networking.
Before we start, here are some screenshots of the speeds we got with Crystal Disk Bench which was set to NVMe mode for access to the shares. This is an easy tool to use to rapidly test tweaks you make in a consistent fashion so I would recommend you use this especially if you have 10Gbit or higher networking.
Above we have the Raid0 performance we hit on our Dell Poweredge R520 Server with 3x Intel Optane 900P NVMe drives in PCIe form factor. This was after a fair amount of tuning time and settings adjustments to get the proper setup that allowed me to not have major network congestion, which we will be diving into more here.
Next, we can see the performance of 8 Seagate 22TB drives in a Raid Z1 configuration with 1 group and 8 hard drives. Impressive! This was not what I expected to see from spindles for this Raid level, but there are clearly some automatic things happening under the hood around the ZFS pool that allow for a very decent out of the box experience with Unraid.
And of course, if you missed it, there is a video that accompanies this article here.
Settings to Enable
App Store > Tips and Tweaks plugin
- Disable Flow Control > Yes (on client machine also and switch)
- Disable NIC Offload > Yes (on client machine also)
- Ethernet NIC RX > 4096 (on client machine also)
- Ethernet NIC TX > 4096 (on client machine also)
- NIC selection (you only want your NIC you are moving high speed over here)
- Dirty BG Ratio > 41% (this you need to tune while moving typical worksize on Unraid Shares)
- Dirty Ratio > 40% (1% less then BG ratio, RAM size dependent)
- Turbo Boost Mode > YES
- Governor > PERFORMANCE
- Scheduler > On Demand
- Apply and reboot
# allow testing with buffers up to 128MB
net.core.rmem_max=134217728
net.core.wmem_max=134217728
# increase Linux autotuning TCP buffer limit to 64MB
net.ipv4.tcp_rmem=4096 87380 67108864
net.ipv4.tcp_wmem=4096 65536 67108864
# testing bbr
net.ipv4.tcp_congestion_control=bbr
# jumbo frames up
net.ipv4.tcp_mtu_probing=1
# recommended to enable ‘fair queueing’
net.core.default_qdisc=fq
SYSLINUX Conf Changes for Networking at 10Gb or 40Gbit +
# allow testing with buffers up to 128MB
net.core.rmem_max=134217728
net.core.wmem_max=134217728
# increase Linux autotuning TCP buffer limit to 64MB
net.ipv4.tcp_rmem=4096 87380 67108864
net.ipv4.tcp_wmem=4096 65536 67108864
# testing bbr
net.ipv4.tcp_congestion_control=bbr
# jumbo frames up
net.ipv4.tcp_mtu_probing=1
# enable ‘fair queueing’ if you run bbr congestion control
net.core.default_qdisc=fq
Global Share Settings
Just turn them all to YES. I tried every combo and provided I had exclusive shares enabled + disk paths and used those, I got these fast speeds. Disabling user shares and Direct IO changed nothing as I was not hitting FUSE.
Client Hardware (Windows 11 Workstation)
CLIENT (or server) WORKSTATION MOBO
AMD Threadripper Pro 5955wx https://geni.us/ThreadripperPRO_5955wx
256GB DDR4 3200 RAM https://geni.us/256GB_8x32GB_DDR4
EVGA 1600W PSU https://geni.us/1600w-PSU-EVGA
Fractal Meshify XL 2 Case https://geni.us/FractalMeshify_2XLCase
ConnectX-3 Pro 40Gbit NICs https://geni.us/connectx-3-pro
QSFP+cable https://geni.us/40gb-AOC-20M
Unraid Server Hardware (Dell R520 Alternatives)
SERVER
Dell r730 https://geni.us/Dell_Poweredge_r730
Dell r720 https://geni.us/OAJ7Fl
Dell r720xd https://geni.us/5wG9n6
RAM
DDR4 RAM https://geni.us/DDR4_ECC_8x32GB
SWITCH
Mellanox sx6036 Switch https://geni.us/Mellanox_SX6036
22TB HARD DRIVES
https://shop.digitalspaceport.com/products/22tb-seagate-exos-x22-7200rpm-sata-6gb-s-3-5-white-label-hard-drive
2 Responses