PB Chia Farm Storage Planning Blueprint 1-2 petabytes

link to full visio diagram

I had someone ask me recently for a 2+ PB chia farm plan in a YouTube Comment. At first, I was like, uhhh… but then I decided this is actually a very interesting topic since it covered Chia Farming at larger scales yet still in a home datacenter level.

Since I just covered the topic of how to flash Dell H710, H310 and H810 HBA‘s and touched on their uses for things like disk shelves and JBOD arrays this seemed like a good time to write this. I am also in the longer-term planning phase for my Chia Farm, network redesign and energy optimization path to (my) optimal solution for the most profitable solution I can attain. Let’s dive in!

Server Racks and things you have to consider with racks

If you just jump into a server rack without a plan, headaches will be sure to follow. You really must plan out a diagram of what you need before you start. I am going to take you step by step through that process. Consider these questions first: what parts you are going to assemble, location it will live in, environmental concerns (humidity, fire, air conditioning), power conditioning and availability.

You need to know what your TARGET PLOTTED PB is going into this, or you will undersize your rack and other components.

Rack Sizes

Racks mainly are 42U/48U monsters that you can find locally. They will require you to use a truck to move them and will also need at least 2 people to do this without stress. Harder and more expensive to find locally are 24U racks or half racks. (My 24U below)

These are a bit of a premium for reasons on secondhand markets, because they are not what is used en masse in most operations centers. The good news is you can get an (Paid Link) open frame rack pretty cheap off Amazon in a 24U capacity. 2.16PB (unformatted) is the Maximum capacity for a 1-2 PB Chia Farm in a 24U format with a dedicated plotter and dedicated full node and power conditioning. That is also not the optimum price/TB either.

Power usage

Did you know you can increase capacity at a given location by using 220v A/C instead of 120V A/C? This is very important when planning a rack size. A24U / 42U / 48U rack can operate with a 220VAC, however most 42U and 48U racks will not be optimized running off 120VAC once you have them loaded with components. You need to also size the available capacity you have in your house for additional AC. Check out my guide here on Home Datacenter Power Usage for more on that.

In a nutshell now that you have read that, here is a basic formula for delivering safe power margins to an drop.

Find the total watts of all equipment then we see if that fits into that breaker’s safety margin. Let’s do our 24U @ 120vac rack as an example.

Plotter Dell R720XD w/128GB – 320w – (Paid Link) https://amzn.to/2V3dJgc

Full Node R720XD 32GB – 140w – (Paid Link) https://amzn.to/2UYgqQ5

4x DS4246 Shelves – 1280w – (Paid Link) https://ebay.us/12BUvT

2x APC UPS Units – 200w – (Paid Link) https://amzn.to/2WxS68b

TOTAL EQUIP       –  1940w

On a standard 20A circuit breaker rated for 80%, the maximum watts are 0.8*20A*120V = 1920 watts.

Ah we are so close, but DON’T FUDGE IT! This problem disappears if you use a 30-amp breaker, heavier wire, and 220VAC. You could also run 2x 20amp circuits and be actually much better as each APC would have a dedicated circuit. That is what I did since I had a bunch of extra 12 AWG Romex.

Networking

The beauty of disk shelf SAS jbods is…they do not need networking. The SAS connection is it; they behave as direct connected devices to the HBA. Nice right! You still need to factor for remote access to the UPSs at 1 run ea. and 2-4 runs to the full node and 1 run to the plotter. An 8 port GB switch can get you there. IF you run multiple plotters, you need 10GB to write each plot without hitting a bottleneck.

Location

Do you want a server rack by your bed? No. Whether you can you FIT a server rack into a spot (turns, stairs, etc.) will heavily factor into this. You MUST think about that before you start. Moving one into a basement could be really hard but a great overall area. Seldom should you move one upstairs. A garage or first floor space is a great spot. You can even build a little room to accommodate this if you like. You need to factor in the space behind the rack. Do not smush the back up to a wall.

You also will have casters and locking riser feet on most. Once you rise, level and load a 42U rack, you are not going to be moving it without a lot of work. A 24U rack is much more flexible in this manner, but loaded you are still looking at around 1200-1400lbs on a 24U. A 42U Netshelter (great racks) can in the 2200-3000lbs range. Check out this video for more about my Chia 24U rack.

A Word of Caution on Electrical Usage

Also let me point out you will be severely harmed or die if it falls on you or a loved one. Do not allow children EVER to be around a server rack. Many, many reasons for that. Putting 1200lbs on a second floor in a 2×4 foot space also you might now see is not a great idea for certain structures. You need to be able to run power to it. You need to be able to run networking to it. Placing a server rack against the side of a wall is A okay but make sure you have 2-4 ft unobstructed space behind it for air to exhaust and access. This also allows you to anchor the top to studs in the wall, for safety.

Cooling a PB Chia Farm

Running a rack of mainly compute in high temps is one thing, even some machines like Dell’s fresh air compliant configs are built for this. Running a bunch of hard drives in high temps, however, is not okay. You will greatly shorten their lifespan. You have to factor in cooling for this for another reason, humidity. This will cause all sorts of problems that can range from minor surface corrosion, to dripping water and shorts.

A dedicated AC unit sized to the intake needs of a server rack is critical in a non-air conditioned space like a garage, and sometimes in high humidity basements if plastic on walls cannot bring down levels of humidity. Check out this video for more on that.

Large disk Arrays for Chia Farming

How far does the rabbit hole go? In a home environment you will eventually be bound by your residential excess electrical capacity. With a 200amp home service, and a typical home, you might have 60-80 amps extra you can use unless you have a pool or a well. Spending more for high efficiency appliances can get you extra margin as well, but there is a limit. Let’s say 20 amps per 2 PB Chia Farm, or 10 amps per each PB Chia Farm.

Next let’s say you can squeeze 80-100 extra amps with some very creative energy conservation. This nets you someplace around 8-10PB max in a house. Yes, you can do better than this by using, say, 4 48U racks bolted together in a garage and not need to have as many RX720’s and cut out maybe 1-1.5KW, but ballpark here.

Proper Airflow

At a certain size, it makes sense to use drives that have a lot of air moving across them. What is good at doing that? Disk shelves. What is not so great at doing that? USB enclosures. Not to say you can’t or shouldn’t, but at some scale there will be an impact.

Chassis type

4U units are pretty awesome, but you know what is even better? Fitting 48 or 60 drives into 4U. This comes at a price though and with the inflated demand due to… Chia… I think those costs scale faster than 24 drive units scale. The NetApp disk shelf also has a lot of public info about use of them in jbod scenarios that makes them very well documented for the space.

Noise levels

This needs to be factored in for your own safety when you are around them for any period of time more than a few mins. You do not want to do permanent hearing damage to yourself. Also, if the wall next to a server rack is not sound deadened in just the right manner, spillover noise will be a thing. That could get annoying if someone like a neighbor or family member is on the other side of that wall (or your home theater) so ensure that is well planned in advance.

Economics of a 1 PB Chia Farm (or larger)

I feel like I have covered the basics of the setup we are approaching here well up to this point, so let’s look at the components, services, and expenses that will go into this.

Components Needed

R720XD Plotter and Full Node Costs, Wattages, Purposes at (8/29/2021) Costs

Top Plotter is running MadMax with 128GB DDR3 – (Paid Link) https://amzn.to/2V3dJgc
Top Backplane is connected to Full Node below
Flashed H710
140w full node
320w plotter
Full node running 2630 32GB ram – (Paid Link) https://amzn.to/2UYgqQ5

Full node needs an H810 external flashed or similar external
HBA – H810 (Paid Link) https://ebay.us/OxeUI5
LSI 9207-8e (Paid Link) https://ebay.us/Z3l7Bs
I think it’s pretty easy to get 1000 for both with rails, cables, ram, cpu’s, caddies (not drives)

DS4246 Costs, Wattages at (8/29/2021) Costs

DS4246 (needs caddies, Paid Link) https://ebay.us/12BUvT
24 – 3D printed caddies (cheapest way to get caddies)
DS4246 (includes caddies, much more expensive, Paid Link) https://ebay.us/6FK94h

24 Caddies needed for each (printed or stock)
Rails needed for each @ 50
Each costs 300 shipped (try for a bulk deal!)
320w ea. with drives filled @ 120v
Est total 400 each, 1200 total

24U Rack and UPS Costs, Wattages at (8/29/2021) Costs

Rack @ 250 for 24U – (Paid Link) https://amzn.to/3yrx74a
APC 2U 1500w APC Units each on Separate 20amp circuit – (Paid Link) https://amzn.to/2WxS68b
APC’s @ 500 ea.

Rack and UPS’s 1250

120 3.5″ HDD Hard Drives at (8/29/2021) Costs

120 drives capacity total (raw unformatted values, this goes down when formatted)

8TB = 960TB               21,600 @ 179ea.        22.5/TB
10TB = 1.2PB               26,400 @ 220ea.          22/TB
12TB = 1.44PB                31,200 @ 260ea.    21.67/TB
14TB = 1.68PB                34,800 @ 290ea.   20.71/TB
16TB = 1.96PB                45,600 @ 380ea.   23.26/TB
18TB = 2.16PB                54,000 @ 450ea.         25/TB

 

CURRENT PRICES

GRAND TOTAL at current Costs

Rack and UPS =          1250
4x DS4246 =                1200
RX720XD’s =              1000
Drives @ 14TB =     34,800
—————————————–
TOTAL                       38,250

Note this is assuming you think like I do about cost/TB as being a primary consideration, then you are looking at 38,500 for just the parts listed at today’s prices. This does not include all the cables, screws, nuts, wires, bolts, GBE switch, time, effort that will be needed also, but I think an extra ~500 covers the materials costs. What your time and effort are worth, only you can determine that for yourself.

Done all at once, this is about a half-day setup if you have experience. If it is your first time, it may be more like 2-5 days depending on how bad things get for you.

Services Needed

Are you an electrician or do you know how to safely run your own electric? This is not a good time to start yourself unless you have extensive experience wiring electrical, sizing electrical loads, and using the right wires, receptacles and breakers. Seriously. My stepdad is an electrician, and my mom was a house flipper when I was growing up.

I have a lot of hands-on experience running wire, installing panels, receptacles and conduit, sizing loads and wires, spending insane amounts of time in hot Texas attics, knowing what local and regional code is and is not.

In addition, I live in an unincorporated pocket of a county that does not prohibit self-installation. This is serious danger and not for the first timer, fire and death are very real and insurance companies will not be understanding. Hire an electrician. Allot 1000 for them and a day to run 2x 20amp circuits in extreme conditions like 3/12 attic and 100+ ft. Should be less for easier installs. Have you priced Romex lately? It’s getting better but is still really bad.

Other Expenses

Screws – Try to get them included! If not, allot per 24 bays * # of dish shelves – (Paid Link) – https://amzn.to/3Dpohrn

Switches – A cheaper 8 port GBE switch can get this handled in a 24u size, but a 24 port can scale to most of whatever you grow into later for marginal increase and has 2 10GB ports. Allot 140 – (Paid Link) – https://amzn.to/3yqiAFS

Network Cables – Allot 20 – (Paid Link) – https://amzn.to/3Dxi5xA

External H810 HBA – Allot 60 – (Paid Link) – https://ebay.us/OxeUI5

SAS Cables – Allot 20ea, daisy-chained, 4X 8088 to 8088 – (Paid Link) https://amzn.to/3mG126k

ELECTRIC – It is not free, so you really do have to factor in that cost, I am at 0.10 usd per kw here. Estimate this 24U rack to use 1.84 KW/hr when plotting and 1.6kw when not. I am not going to concern this article with plotting and filling your drives and those details, instead I will just point you to the YouTube channel as it has a lot of content on plotting and speed of plotting you can dig into.

ALL IN, WE ARE CALLING IT 40,000 (to be simple)

Chia Price Volatility and Impacts

So, we have our base costs here now in a ballpark fashion, let’s think about this and if it is a good or bad idea based on current and variable future economics. This is a fair-sized economic investment, and no one should view this article as me recommending you endeavor to build such a machine. In fact… don’t skip ahead but do finish reading to the end… I am using a few different calculators, Flexpool Chia Calculator  N.B. You should NOT use a calculator for making any future earnings nor price predictions. They cannot see the future.

Flex’s calculator is showing at 1.5PB of plotted size, 9.03462 XCH (2,191.80) per month. Let’s multiply out that 9 XCH/mo by 12. This would be 108 xch/yr on a pool. If you have everything dialed in and efficient and trust yourself, you could add .25 to each XCH. That would be 11.25 XCH per month and 135 per year solo; I hope you don’t have anything go wrong.

Future Price is Always Unpredictable – All Time High – (High Side)

Yes, it is unpredictable. Let’s, however, use today’s price to see what we would make in a year (this number is BS and worthless, DO NOT USE for more than a thought experiment) Okay then let’s say 243 per XCH as of 8/29/2021. Multiple this out by 108 and 135.  That is 26,244 and 32,805, respectively. Not bad. This is not factoring in time, electric, wear and tear, maintenance, your own time, but it’s still not horrible.

If we assume we total out at 1 year cash out interval (probably plus a day also), then we are looking at a loss of -13,756 with pool or -7,195 without in this scenario. Not too bad….or is it?

Future Price is Always Unpredictable – All Time Low – (Low Side)

Now let’s flip the coin and look at some less favorable scenarios. Let’s assume that we hit the price point of the All Time low, which I think was July 20 at 177 per xch. Now our 108 is 19116 and our 135 is 23895. Hum. Now let’s see what that First-year cash out looks like. Our pool loss is -20,884 and our solo loss is -16,105. Easily, we are seeing that we would need to be thinking in a more like 2-year term for our repayment on initial capital investment here and a 4-year project. So, should this be a deal breaker?

How I View Internal Rate of Return (IRR) and a PB CHIA FARM

This is not financial advice. This is just how I personally view it. Let’s view it like a business, it makes sense to me to view it exactly like that. I already own several businesses, and I have sold businesses in the past and have been an entrepreneur for a long time. Next, think about this from the tax perspective, then think about this from the economics possibilities perspective and especially from the Internal Rate of Return perspective.

Why? We should have a target IRR that we like to hit in a business. So, what exactly is IRR and how is it calculated? This is a question of finance, and something you can learn more about here, but let’s run through this calculation, shall we? This is based off of 1-year annual withdrawal, and future withdrawals at the same interval.

WHOA! 63% 4yr IRR!!! OMGFY!

Before we build a PB Chia Farm!

These numbers do not account for things like TAXES, which will be variable per person and situation, electric, time, management, maintenance of parts outside warranty, and MANY other things. Simply put, I have not taken the time yet to do a statement of cashflows for the PB Chia Farm project yet so I don’t even know yet what the real numbers will be.

I CAN SAY with a shmaybe level of confidence divide these by 2 for a fairly safe estimate of what the reality will be based off these ridiculously incomplete numbers. So that is 31.5%, over 4yrs. Now let’s divide that by 4 to get an annualized IRR. 7.875%, this is the “high side” estimate I would use to evaluate this. Sobering? Yes.

AGAIN, LET ME STATE THIS IS NOT FINANCIAL ADVICE!

On the low side, divide by 2 gets us to 10% 4yr IRR. Now divide that by 4 years. 2.5% annualized IRR. Humm. Okay so that’s not so hot on the low side, right? Now factor in variability of what we are dealing with (crypto) and yeah oh snap okay. Things can get crazy here. It is probably best to assume they will also. What does that mean? Well, it means a few things for me (NOT YOU, BECAUSE THIS IS NOT FINANCIAL ADVICE REMEMBER)

My personal takeaways from this thought experiment

1. This project has a very high dependence on costs associated with it that are tied to Hard Drives. I think things at the current price levels are hard to feel good about honestly. If these same 14TB hdds were $15/TB, that would shave 5.71 per TB per drive off the price. Times 14 then times 120 and that’s 25,200 vs the current 34,800. WOW to the tune of 9,600 usd upfront costs!
2. Having the ability to scale is nice, doing it slowly is good for me.
3. This world is WILDLY UNPREDICTABLE! You have no idea what tomorrow will hold. I don’t either. I have no idea why, but I am scared for the long-term direction of things at their current trajectory. Not just in the crypto sphere. Given that, does it make sense for me to allocate a bulk of additional funds to this right now? Not in my perspective at these hard drive prices.
4. Understanding and refining the project to work at an optimum level has been pretty challenging, and I have a high level of tech affinity and overall background use of high-end compute and networking. I have gotten a glimpse at the problems some are facing out there in the YT Comments, and it is not a smooth ride. It shouldn’t really be, this is a brand new crypto and things are always rough out of the starting line.
5. Chia official team has actually done a really good job pivoting resources, learning as they go, solving pop-up problems, and that is commendable. That is pretty different than what you would see in let’s say…ETH.
6. Chia has a long-term position in mind that is consistent and has not altered, which is something I am very wary of. Many other cryptos have a lot more “finger in the wind” approach to things, and that is a good way to have issue come up.
7. I am glad I have the capacity to crank up the plots, I am hopeful for price/tb to fall more. Glad to have the basis for the infrastructure to scale out a bit.
8. Given all that, I think the course I am on currently with my current 160TB of plots is great. Not looking to expand at the prices of HDDs right now, but closer to 15-16/tb I am very interested.

Wrapping up our assessment of a PB Chia Farm

In summary, these numbers and figures are very rough at best. I already have many of the skills I need to grow in the future, but I am looking for ways to make my internal network more robust, fault tolerant and effective. I am also looking at the best way to monitor my farm, optimize responses and minimize downtime potentials.

These things have my attention currently and have me shifted over into phase 2 of this journey. Optimization. Join along in my YouTube channel as I live the chia mining life and try to share what I learn along the way!