# AW: Thoughts on big SSD arrays?

- From: Markus Stockhausen \<hidden\>
- Date: 2015-08-03 10:52:55 
- List: linux-raid
- Page: https://ratatoskr.run/linux-raid/2015/08/6630152
- Thread: https://ratatoskr.run/linux-raid/2015/07/6630148/t.md

```text
> Von: linux-raid-owner@vger.kernel.org [linux-raid-owner@vger.kernel.org]&quot; im Auftrag von &quot;Matt Garman [matthew.garman@gmail.com]
> Gesendet: Freitag, 31. Juli 2015 17:23
> An: Mdadm
> Betreff: Thoughts on big SSD arrays?
> 
> Every few years I reprise this topic on this mailing list[1], [2].
> Basically I'm just brainstorming what is possible on the DIY front
> versus purchased solutions from a traditional "big iron" storage
> vendor.  Our particular use case is "ultra-high parallel sequential
> read throughput".  Our workload is effectively WORM: we do a small
> daily incremental write, and then the rest of the time it's constant
> re-reading of the data.  Literally 99:1 read:write
> 
> I continue to be inspired by the "Dirt Cheap Data Warehouse (DCDW)"
> [3].  SSD are getting bigger and prices are dropping rapidly (2 TB
> SSDs available now for $800).  With our WORM-like workload, I believe
> we can safely get away with consumer drives, as durability shouldn't
> be an issue.
> 
> So at this point I'm just putting out a feeler---has anyone out there
> actually built a massive SSD array, using either Linux software raid
> or hardware raid (technically off-topic for this list, though I hope
> the discussion is interesting enough to let it slide).  If so, how big
> of an array (i.e. drives/capacity)?  What was the target versus actual
> performance?  Any particularly challenging issues that came up?

Hi Matt,

maybe not 100% matching... We wanted to start a 22 HDD md raid 6 last 
year for one of our storage servers. Doing some tests we experienced 
slow random write I/Os and got discouraged. So we headed over to
hardware raid. 

To get a clearer picture we did further analysis. The bootleneck arised 
from two sources. md issued only 4K I/Os and it always did a reconstruct
write. Thus massive write amplification. Several patches in 4.1 mitigate
the situation. In between I like a lot of things in md raid 4/5/6. Especially 
the parity calucation massively benefits from high power CPUs.

Back to the reason for my explanation: The biggest "area under construction"
in md raid 4/5/6 is the internal stripe cache handling. Allocating, flushing
and freeing stripes is based on solid but not tuned algorithms (like LRU). 
IIRC it does not even have a scheduling or queueing. That could become
your bottleneck. Maybe massive CPU power easily compensates the gap. 

That said. Your read mostly setup might avoid a lot of headaches but you 
need to push it to the limits yourself. I do not see benchmarks in the net 
that closely fit your case. So I advise to enable ramdisks in the Linux 
kernel and build a raid 6 on top of /dev/ramX. A synthetic test according
to your needs should give an idea of scalability and overhead of md raid. 

Good luck.

Markus
```

## Attachments

- [InterScan\_Disclaimer.txt](https://ratatoskr.run/linux-raid/2015/08/6630152/attachment/0) (text/plain, 1694 bytes)