Home Networking with 10Gb, 40Gb and 100Gb Tips, Tricks and Installation

When you click on links to various merchants on this site and make a purchase, this can result in this site earning a commission. Affiliate programs and affiliations include, but are not limited to, the eBay Partner Network. As an Amazon Associate I earn from qualifying purchases. #ad #promotions

10Gb, 25Gb, 40Gb and 100Gb Home Networking

There are a number of reasons to get into really fast networking. The biggest reason is it is now cheap! Finally even 40Gb nics, switches, DAC’s, Fiber and Transceivers have come down to prices that make them very solid recommendations. 10Gb Ethernet has been cheap for a pretty long time, but 40Gb Ethernet has been much higher. 100Gb networking coming out of Datacenters in large numbers now we are also seeing quality 100GbE options arise. Upgrading your home network to 10Gb, 25Gb, 40Gb, or 100Gb can greatly improve your local network experience, but it requires some planning and varying degrees of installation. In this article, I’ll provide you with tips, tricks, and best practices I have learned along the way to my high-speed network.

Accompanying Video

Homelab Networking Needs

It is a good idea to have an idea as to why you are wanting to go faster. Let me be clear in saying I don’t think you should drill holes in walls unless you are qualified and (maybe) own the walls and have good reasons. WiFi 6 and 7 are fast, but far from reliable and consistent. Sometimes it helps to read what others think so here is my “why” moment.

Top 5 reasons for 10Gb, 25Gb, 40Gb or 100Gb Fiber Ethernet

  1. I have been into birding photography for a while and started using RAW file types to preserve editing capabilities for later. These pictures are large, but thumbnail generation can be slow. The slowness is based on LATENCY. Fiber based, especially in 25Gb+ networks, are excellent in low latency. Thumbnail generation and first view is up to 10x faster commonly for me, as I also use low latency high iops NVMe caches.
  2. I like resilient networks. This is why I use ex enterprise gear. It simply doesn’t fail easy.
  3. Remote storage is really cool. Mapped network drives to ZFS arrays, awesome. Backups and migrations over 1gb networking, not awesome.
  4. I had an opportunity to install fiber when we remodeled the house. It is very easy to do when you have walls open and no opposition to messes and getting into the attic is already happening.
  5. VM backed storage. Low latency is key for a good time. CAT is higher latency, DACs are lower latency, Fiber via Active cables or Transceivers are the lowest.

Networking Gear Recommendations

The real play is the 40Gb gear which is amazing at the prices it is at now. 100Gb is coming down a lot but the performance gain per $ is highest in 40. 10GbE is good as you don’t need to adjust anything to hit peaks around 9.6Gbit/s. It surprisingly is about the same cost as 40GbE gear, if you can get a 40GbE switch that you are happy with. It is the best choice in my opinion by a long shot.

Networking Tips and Tricks

  • Try to buy gear all coded to the switch maker you have. Mellanox switch, Mellanox cards, Mellanox cables ideally.
    • In higher performance networking gear, cables are coded to a mfg code. This can present compatibility issues.
  • Your Ethernet traffic goes through the processor on your client and server computers. This slows down the traffic.
    • RDMA can sidestep this but has a lot to configure
    • A 4Ghz CPU I often see as hitting 40Gbits. 3Ghz around 30Gbits. This may be a correlation.
  • You can wreck your network fast when you start making changes to congestion algos, tcp windows, rss, offloading and more.
    • You likely do progress to this, but have a system image backup first in case you cant remember which tweak did what that broke things. Blindly following other peoples settings is a fast way to a reinstall.
  • Plan your slot layout and check the manual and/or bios to see what each slot will negotiate at.
  • Molex is a great maker of cables, check for them.
  • Switches may have license restrictions with them. Ask the seller to verify if they can run ethernet traffic on it.
  • Don’t buy a “for parts” 100GbE switch and think you can fix it is my recommendation after going that route. It was a big pita to get it working and I spent a lot buying other parts.
  • Check switches for “ATOM” bug that can kill their CPU.
  • If you are good with testdisk, you might be able to recover a complete operating system.
    • It is a good skill to have and I do recommend it.
  • If your throughput is stuck at around 22-25Gbit/s check you are not on a PCIe 3 x 4 slot. Use lspci -vvv to query the device.
  • 56GbE is an unlock for Mellanox. You need to jump through a lot of extra hoops to get it setup but it is likely worth it.
  • Heatsinks on all transceivers that are 40GbE is a good idea. Absolute MUST on CWDM4 100GbE optics.
    • It only takes a few minutes of heavy traffic to burn up a no airflow CWDM4. They are cheap for a reason.
  • Watch bend radius on any transceiver based fiber drops for too much tension.
  • If you see network drops, which can be also checked in dmesg easily, then check each component in the line to find the cause.
    • Overheated Transceivers will often come back up after going down from getting hot.
    • Once they are damaged, putting a heatsink on them wont solve the problems.
  • Always have an extra of X gear. DAC, Fiber, Card, Transceiver
    • It is hard to have two switches in failover, but very possible. Costs will rise a lot for this.
  • All enterprise switches are unpleasant to listen to if close. Put them in a seperate room.
    • If you are brave, mods can be done to reduce noise but airflow MUST remain good in the switch.
  • Always have at least 1 DAC on hand that is know good at each level of XXGbE for troubleshooting.
  • Use closet rack from a home store and ladder hanger brackets for cheap raceway.
  • Direct airflow is an absolute must. Even on ConnectX-2 cards. Make it a part of the plan upfront.
  • Use 9k MTU in your network if your switch supports it. 1500 WILL fragment your packets down and require more round trips.
    • Use 9216 MTU on the switch and 9000 on clients if available
    • Hitting block storage? Try out 8400 on clients.
  • Do not buy long range optics. They run hotter.
  • DACs in Racks is a saying for a good reason. They are rugged and work well.
    • Fiber can also work, but being careful is very important.
  • Buy or 3d print a loop spool for your wraps.
  • You WILL have issues at times. This is normal so be capable to address them as best as possible.
    • Learning in your homelab about networking is strongly encouraged, but not in segments you need to stay up.
  • Velcro can really help you run fiber
  • Drilling into walls can also hit electrical. Please be careful.
    • Check codes if they impact you running low voltage.
    • Get an electrician in you don’t like getting in the attic and you have far spans. Serious health risks
  • Save the tips from cables.
  • Do not knot or allow to get snagged into a knot when pulling fiber. If it bends sharp 90 degrees, it is likely damaged.
  • Avoid CAT based networking as it is higher latency.
  • If running in attic, running in conduit is advisable
    • You can use Cantex plastic electrical conduit in long spaces that are hard to reach but which will hit the other side.
    • Use a cable puller in the conduit to pull an extra mule tape for easy future pulls.
  • Let gravity pull down the any fiber cables when you can until you need to apply pressure pulling.
  • Top slots in Dell (and likely other OEMs) are full x16 gen3 in many cheap desktop class cards.
    • Great to place a 100Gb nic in to allow full speed
    • GPUs often do not need x16 to perform well, especially true for inference work
    • Bifurcation should be checked in your motherboards BIOS
  • Use glow sticks to run in walls if you have no conduit.
  • Keystones plates are very good to use but can induce loss.
  • Check for TX or RX pauses and errors in your switch.
  • You can form loops with zip-ties or velcro and not pull them tight. Then you can staple those to the wall in the attic and run fiber through those.
  • Sketching out your network before using something like Draw.io is a great idea. It is faster to ideate on
    • Eventually you should make a pretty network diagram.
  • Same gear will behave largely the same when it is dealing with VLANs. That is another reason to go with all one brand as much as possible.
  • Check your switch for errors often. On high quality gear and good runs you will see 0. I see 0 on the SN2700 for instance.
  • If buying a long range active DAC really be sure you measured your entire run very well. Then go up 1 size just in case.
  • I would avoid 25GbE switches at the prices they seems to be stuck at.
    • I call this the unifi effect
  • Mikrotik is cheap and pretty decent, but top end performance can present issues on some switches.
    • Possibly didn’t have fast enough or enough nand for buffers on some switches leading to extra retry activity.
    • They don’t update very often which for an L2/L3 mode non routing switch is fine but don’t put it directly on the internet unless running routerOS
  • Netgear has given me many problems over the years. Be wary.
  • Don’t look at the lasers in optics or fiber to see if it is on. If you do make a good joke about seeing it on all the time.
JOIN AS A YOUTUBE MEMBER

Table of Contents

Hardware Review

Local Ai Setup Guide