Here is the ROUGH process I use to get to the gross figures you see above (that is 36 plots, with room for another 2-4 in the global) on less expensive storage solutions like Dell Gen 12 servers, and older cheap dual socket xeons using affordable DDR3. This entire setup can be purchased for 1500 USD, and it’s also pretty readily available (R720’s, I am looking at you).
1) Unlearn what people have repeatedly told you, that you MUST use the most expensive NVME storage, fastest ram, highest end processors…… it’s just not an economical nor cost effective solution (maybe you get 4x in some adapter cards running, but that’s still a hard max.) Consider this a brief guide around the storage aspect of plotting. The largest amount of disinfo is encompassed in the NVME myth.
2) You need sustained high write bursts on lots of threads (threads) to the IO subsystem storage controller. The plotter uses 60% writes/40% reads. It has jitter per each plot’s threads. Yes, it reads a shitload also (it has to as it’s reading what it has written). This leads you to a different kind of need than a single card or drive can handle; que depth matters. You NEED an ability to spool a fat que of cmds to the storage controller and have them not bottleneck. If you use servers with backplanes that are SAS2 or better, you have a major leg up. You can (and must) flash that controller to get a higher peak que depth. How big of a difference are we talking? 25 vs 600 que depth once flashed to IT mode.
Guides to flash (cheap and plentiful) RAID cards you say? Here we go:
3) So now you have a flashed HBA card (even in a desktop with forward breakout cables to 8 sata ssd drives, you will beat out by a huge margin performance in plotting vs a NVME. In addition, you will have massively more size. What’s that saying? Size Matters 🤭You need 4TB at least, 8TB is much preferable. I postulate ANYONE can get to 8TB of incredibly fast storage for ~800 total cost.
4) You WANT a higher quality ssd drive with good performance outside the cache, this is critical. MX500, Samsung EVO 870, Hynix Gold are all names to look for. I personally use Samsung EVO 870 SSD’s. So, you have these 4-8 drives now connected to your HBA and recognized in ????? operating system? Let’s talk Operating Systems.
5) Get an empty USB drive. Go download the UnRAID installer. Install UnRAID on it. You get a nice 30-day trial with 30 extra days in extensions. UnRAID seriously rules for how fast you can get everything set up, monitor and control very quickly. Time is money, and UnRAID saves me a lot of time. Set as boot drive. Boot it up. Go to settings > users > root and create a password. Configure your drives. Install the Machinaris docker. Install your keys. Next, we Plan and Plot. (You HAVE NOT setup your storage areas yet, that’s step 9 below)
6) You are using HPOOL right? If you have maybe 1PB in storage then sure, solo on. If not STOP LISTENING TO OTHER PPL. Get in an hpool ASAP. You will instantly feel better seeing your daily XCH grow. INSTANTLY! It’s like 1 hour to set up max. No arguments on this point unless you have above mentioned sizes. Just go do it. PLEASE STOP LISTENING TO “OFFICIAL” and start earning some Chia now.
7) Now count the number of cores total in your machine. That is the starting point, but we will tweak that number up by a lot. Those 36 plots which I am tweaking up to 40 are running on dual 2667 v2 processors. I started at 24 and have been tweaking it up to currently 36 and plan on ending up around 42. With a 25 min stagger and ~16 hour finish you can start to see the plotting power of this rig daily.
8) Count up your GB ram, divide by 2, 3 or 4. Whichever maxes out a nice round number closest that is around your processor core count (this gets tweaked a lot later) So for instance I have 128GB here (actually 192, but I am going to remove some) divided by 4 = 32. Close enough to give growth potential and easy for me starting at 24 plots and 8192 on ram for each plot. You won’t use all this ram at once. Not even close. Not even remotely close. You will be watching the first several hours at any rate so will catch anything about to go bad. No need to have more ram than provides a 4 number.
9) We now have our number of plots (24), ram (8192), and we need to know our plot space. Remember that talk about size? Here is why it’s important. We need to have 24 active plots and the ability to grow that number without adding disks. We also need huge raw IO performance. 8 SSD’s in raid0 will absolutely saturate a SAS2 controller on a gen3 port (which can get you like 9.6*.8 theoretical bandwidth) Yes, overperformance of some LSI cards on gen 3 slots like the H710 can net you extra GB’s in a big way, and yes, it is odd. UnRAID makes setting up RAID0 pools easy and dumb proof.
So, you do not want to add storage to the “array” unless it is plot storage. You want to create 2 cache pools. 1 named plotting, 1 named plotstorage. I suggest 6 1tb drives in raid0 for plotting pool, 2 1tb drives in raid0 for plotstorage. Once you assign the drives, start the array. Then click the top named device in each pool, go to balance settings, select convert to raid0. Click apply. It happens fast. Then select balance and apply again. Do the same for the next pool. You will see 6TB and 2TB as outlined above for each.
You can get all this on 1 HBA card like a H710 flashed to IT mode with 2 forward breakout cables. Holds 8 drives without an expander like that.
10) Set your stagger to 45. This will get moved down if it’s safe. You have to let the first batch get to finished before you can adjust and know for sure. This as well as the number of plots are what you need to tweak later. RAM should remain constant per prior calcs.
11) Put the settings into the configuration for Machinaris and hit start. You must babysit (hourly checks after the first 2-3 hours) until the first plots finish out. You can adjust the plotman config and restart without killing all the plots. Nice.
Staggering and max plot counting is the key to breaking the threads/plots barrier. Start out with a few simple calculations then tweak it upward from there. You do NOT want to overdo it to start, as you will run into a log jam that is much harder to correct and doesn’t point to which variable you fucked up. So, you have to just restart with things lower again. PITA and avoidable with above formula.
So, you have the first plots in stage 4 and moved to temp2 or storage? Now look at the top and see what kind of deviation you have for your next plot and top plot. As per my image, I have 1:21 gap. I’ll subtract out the 24 mins stagger and say it’s 1 hour. I’ll next add 2 more plots to the global provided I am not maxing out my storage or ram, which is a quick check.