From: vincent Ferrer <hidden> Date: 2012-08-16 02:56:05
Hi All,
Have questions about making raid5 thread use all the cores on my
linux based storage server !
- My storage server has upto 8 cores running linux kernel 2.6.32.27.
- I created a raid5 device of 10 SSDs .
- It seems I only have single raid5 kernel thread, limiting my
WRITE throughput to single cpu core/thread.
Question : What are my options to make my raid5 thread use all the
CPU cores ?
My SSDs can do much more but single raid5 thread
from mdadm is becoming the bottleneck.
To overcome above single-thread-raid5 limitation (for now) I re-configured.
1) I partitioned all my 10 SSDs into 8 partitions:
2) I created 8 raid5 threads. Each raid5 thread having
partition from each of the 8 SSDs
3) My WRITE performance quadrupled because I have 8 RAID5 threads.
Question: Is this workaround a normal practice or may give me
maintenance problems later on.
cheers
vincy F.
chico
From: Stan Hoeppner <hidden> Date: 2012-08-16 05:58:48
On 8/15/2012 9:56 PM, vincent Ferrer wrote:
- My storage server has upto 8 cores running linux kernel 2.6.32.27.
- I created a raid5 device of 10 SSDs .
- It seems I only have single raid5 kernel thread, limiting my
WRITE throughput to single cpu core/thread.
The single write threads of md/RAID5/6/10 are being addressed by patches
in development. Read the list archives for progress/status. There were
3 posts to the list today regarding the RAID5 patch.
Question : What are my options to make my raid5 thread use all the
CPU cores ?
My SSDs can do much more but single raid5 thread
from mdadm is becoming the bottleneck.
To overcome above single-thread-raid5 limitation (for now) I re-configured.
1) I partitioned all my 10 SSDs into 8 partitions:
2) I created 8 raid5 threads. Each raid5 thread having
partition from each of the 8 SSDs
3) My WRITE performance quadrupled because I have 8 RAID5 threads.
Question: Is this workaround a normal practice or may give me
maintenance problems later on.
No it is not normal practice. I 'preach' against it regularly when I
see OPs doing it. It's quite insane. The glaring maintenance problem
is that when one SSD fails, and at least one will, you'll have 8 arrays
to rebuild vs one. This may be acceptable to you, but not to the
general population. With rust drives, and real workloads, it tends to
hammer the drive heads prodigiously, increasing latency and killing
performance, and decreasing drive life. That's not an issue with SSD,
but multiple rebuilds is. That and simply keeping track of 80 partitions.
There are a couple of sane things you can do today to address your problem:
1. Create a RAID50, a layered md/RAID0 over two 5 SSD md/RAID5 arrays.
This will double your threads and your IOPS. It won't be as fast as
your Frankenstein setup and you'll lose one SSD of capacity to
additional parity. However, it's sane, stable, doubles your
performance, and you have only one array to rebuild after an SSD
failure. Any filesystem will work well with it, including XFS if
aligned properly. It gives you an easy upgrade path-- as soon as the
threaded patches hit, a simple kernel upgrade will give your two RAID5
arrays the extra threads, so you're simply out one SSD of capacity. You
won't need to, and probably won't want to rebuild the entire thing after
the patch. With the Frankenstein setup you'll be destroying and
rebuilding arrays. And if these are consumer grade SSDs, you're much
better off having two drives worth of redundancy anyway, so a RAID50
makes good sense all around.
2. Make 5 md/RAID1 mirrors and concatenate them with md/RAID linear.
You'll get one md write thread per RAID1 device utilizing 5 cores in
parallel. The linear driver doesn't use threads, but passes offsets to
the block layer, allowing infinite core scaling. Format the linear
device with XFS and mount with inode64. XFS has been fully threaded for
15 years. Its allocation group design along with the inode64 allocator
allows near linear parallel scaling across a concatenated device[1],
assuming your workload/directory layout is designed for parallel file
throughput.
#2, with a parallel write workload, may be competitive with your
Frankenstein setup in both IOPS and throughput, even with 3 fewer RAID
threads and 4 fewer SSD "spindles". It will outrun the RAID50 setup
like it's standing still. You'll lose half your capacity to redundancy
as with RAID10, but you'll have 5 write threads for md/RAID1, one per
SSD pair. One core should be plenty to drive a single SSD mirror, with
plenty of cycles to spare for actual applications, while sparing 3 cores
for apps as well. You'll get unlimited core scaling with both md/linear
and XFS. This setup will yield the best balance of IOPS and throughput
performance for the amount of cycles burned on IO, compared to
Frankenstein and the RAID50.
[1] If you are one of the uneducated masses who believe dd gives an
accurate measure of storage performance, then ignore option #2. Such a
belief would indicate you thoroughly lack understanding of storage
workloads, and thus you will be greatly disappointed with the dd numbers
this configuration will give you.
--
Stan
From: Mikael Abrahamsson <hidden> Date: 2012-08-16 07:03:03
On Thu, 16 Aug 2012, Stan Hoeppner wrote:
performance, and decreasing drive life. That's not an issue with SSD,
but multiple rebuilds is.
I remember seeing MD waiting to rebuild when it detects components being on the same physical devices, doesn't it do that (anymore)?
--
Mikael Abrahamsson email: swmike@swm.pp.se
From: David Brown <hidden> Date: 2012-08-16 07:52:46
On 16/08/2012 07:58, Stan Hoeppner wrote:
On 8/15/2012 9:56 PM, vincent Ferrer wrote:
quoted
- My storage server has upto 8 cores running linux kernel 2.6.32.27.
- I created a raid5 device of 10 SSDs .
- It seems I only have single raid5 kernel thread, limiting my
WRITE throughput to single cpu core/thread.
The single write threads of md/RAID5/6/10 are being addressed by patches
in development. Read the list archives for progress/status. There were
3 posts to the list today regarding the RAID5 patch.
quoted
Question : What are my options to make my raid5 thread use all the
CPU cores ?
My SSDs can do much more but single raid5 thread
from mdadm is becoming the bottleneck.
To overcome above single-thread-raid5 limitation (for now) I re-configured.
1) I partitioned all my 10 SSDs into 8 partitions:
2) I created 8 raid5 threads. Each raid5 thread having
partition from each of the 8 SSDs
3) My WRITE performance quadrupled because I have 8 RAID5 threads.
Question: Is this workaround a normal practice or may give me
maintenance problems later on.
No it is not normal practice. I 'preach' against it regularly when I
see OPs doing it. It's quite insane. The glaring maintenance problem
is that when one SSD fails, and at least one will, you'll have 8 arrays
to rebuild vs one. This may be acceptable to you, but not to the
general population. With rust drives, and real workloads, it tends to
hammer the drive heads prodigiously, increasing latency and killing
performance, and decreasing drive life. That's not an issue with SSD,
but multiple rebuilds is. That and simply keeping track of 80 partitions.
The rebuilds will, I believe, be done sequentially rather than in parallel. And each rebuild will take 1/8 of the time a full array rebuild would have done. So it really should not be much more time or wear-and-tear for a rebuild of this monster setup, compared to a single raid5 array rebuild. (With hard disks, it would be worse due to head seeks - but still not as bad as you imply, if I am right about the rebuilds being done sequentially.)
However, there was a recent thread here about someone with a similar setup (on hard disks) who had a failure during such a rebuild and had lots of trouble. That makes me sceptical to this sort of multiple array setup (in addition to Stan's other points).
And of course, all Stan's other points about maintenance, updates to later kernels with multiple raid5 threads, etc., still stand.
There are a couple of sane things you can do today to address your problem:
1. Create a RAID50, a layered md/RAID0 over two 5 SSD md/RAID5 arrays.
This will double your threads and your IOPS. It won't be as fast as
your Frankenstein setup and you'll lose one SSD of capacity to
additional parity. However, it's sane, stable, doubles your
performance, and you have only one array to rebuild after an SSD
failure. Any filesystem will work well with it, including XFS if
aligned properly. It gives you an easy upgrade path-- as soon as the
threaded patches hit, a simple kernel upgrade will give your two RAID5
arrays the extra threads, so you're simply out one SSD of capacity. You
won't need to, and probably won't want to rebuild the entire thing after
the patch. With the Frankenstein setup you'll be destroying and
rebuilding arrays. And if these are consumer grade SSDs, you're much
better off having two drives worth of redundancy anyway, so a RAID50
makes good sense all around.
2. Make 5 md/RAID1 mirrors and concatenate them with md/RAID linear.
You'll get one md write thread per RAID1 device utilizing 5 cores in
parallel. The linear driver doesn't use threads, but passes offsets to
the block layer, allowing infinite core scaling. Format the linear
device with XFS and mount with inode64. XFS has been fully threaded for
15 years. Its allocation group design along with the inode64 allocator
allows near linear parallel scaling across a concatenated device[1],
assuming your workload/directory layout is designed for parallel file
throughput.
#2, with a parallel write workload, may be competitive with your
Frankenstein setup in both IOPS and throughput, even with 3 fewer RAID
threads and 4 fewer SSD "spindles". It will outrun the RAID50 setup
like it's standing still. You'll lose half your capacity to redundancy
as with RAID10, but you'll have 5 write threads for md/RAID1, one per
SSD pair. One core should be plenty to drive a single SSD mirror, with
plenty of cycles to spare for actual applications, while sparing 3 cores
for apps as well. You'll get unlimited core scaling with both md/linear
and XFS. This setup will yield the best balance of IOPS and throughput
performance for the amount of cycles burned on IO, compared to
Frankenstein and the RAID50.
For those that don't want to use XFS, or won't have balanced directories in their filesystem, or want greater throughput of larger files (rather than greater average throughput of multiple parallel accesses), you can also take your 5 raid1 mirror pairs and combine them with raid0. You should get similar scaling (the cpu does not limit raid0). For some applications (such as mail server, /home mount, etc.), the XFS over a linear concatenation is probably unbeatable. But for others (such as serving large media files), a raid0 over raid1 pairs could well be better. As always, it depends on your load - and you need to test with realistic loads or at least realistic simulations.
[1] If you are one of the uneducated masses who believe dd gives an
accurate measure of storage performance, then ignore option #2. Such a
belief would indicate you thoroughly lack understanding of storage
workloads, and thus you will be greatly disappointed with the dd numbers
this configuration will give you.
However, there was a recent thread here about someone with a similar
setup (on hard disks) who had a failure during such a rebuild and had
lots of trouble...
That was me. The rebuilds _did_ happen sequentially, not in parallel. The trouble was that by the time the first one finished, the critical-section backup for the second one was considered too old to be restored, so mdadm --assemble balked. The fix was simply to tell mdadm --assemble to go forward with the old backup; all is well now.
So from my experience, yes, multiple partitions (with magnetic disks) are definitely an administrative hassle, and can sometimes get you into actual trouble. Barring some real benefit to set against those real costs, I'd say it'd be better to not do it. [ :) ]
(BTW David, I found that your constructive commentary added much to the discussion, both here and in my original thread -- thanks.)
-- Flynn
--
America will never be destroyed from the outside. If we falter, and lose
our freedoms, it will be because we destroyed ourselves.
(Abraham Lincoln)
From: vincent Ferrer <hidden> Date: 2012-08-16 22:11:28
On Wed, Aug 15, 2012 at 10:58 PM, Stan Hoeppner [off-list ref] wrote:
On 8/15/2012 9:56 PM, vincent Ferrer wrote:
quoted
- My storage server has upto 8 cores running linux kernel 2.6.32.27.
- I created a raid5 device of 10 SSDs .
No it is not normal practice. I 'preach' against it regularly when I
see OPs doing it. It's quite insane.
There are a couple of sane things you can do today to address your problem:
Stan
Hi Stan,
Follow-up question for 2 types of setups i may have to prepare:
1) setup A has 80 SSDs. Question: Should I still create one
raid5 device or should I create 8 raid5 device each having 10 SSDs ?
My linux based storage server may be accessed by upto 10-20
physically different clients.
2) Setup B has only 12 SSDs. Question: Is it more practical to
have only one raid5 device, even though I may have 4-5 physically
different clients or create 2 raid5 devices each having 6 SSDs.
Reason I am asking because I have seen enterprise storage arrays from
EMC/IBM where new raid5 device is created on demand and (storage
firmware may spread across automatically across all the available
drives/spindles or can be intelligently selected by storage admin by
analyzing workload to avoid hot-spots)
Partitioning was only done because I am still waiting budget approval
to buy SSDs.
regards
vincy
From: Stan Hoeppner <hidden> Date: 2012-08-17 07:15:55
On 8/16/2012 2:52 AM, David Brown wrote:
On 16/08/2012 07:58, Stan Hoeppner wrote:
quoted
On 8/15/2012 9:56 PM, vincent Ferrer wrote:
quoted
- My storage server has upto 8 cores running linux kernel 2.6.32.27.
- I created a raid5 device of 10 SSDs .
- It seems I only have single raid5 kernel thread, limiting my
WRITE throughput to single cpu core/thread.
The single write threads of md/RAID5/6/10 are being addressed by patches
in development. Read the list archives for progress/status. There were
3 posts to the list today regarding the RAID5 patch.
quoted
Question : What are my options to make my raid5 thread use all the
CPU cores ?
My SSDs can do much more but single raid5 thread
from mdadm is becoming the bottleneck.
To overcome above single-thread-raid5 limitation (for now) I
re-configured.
1) I partitioned all my 10 SSDs into 8 partitions:
2) I created 8 raid5 threads. Each raid5 thread having
partition from each of the 8 SSDs
3) My WRITE performance quadrupled because I have 8 RAID5
threads.
Question: Is this workaround a normal practice or may give me
maintenance problems later on.
No it is not normal practice. I 'preach' against it regularly when I
see OPs doing it. It's quite insane. The glaring maintenance problem
is that when one SSD fails, and at least one will, you'll have 8 arrays
to rebuild vs one. This may be acceptable to you, but not to the
general population. With rust drives, and real workloads, it tends to
hammer the drive heads prodigiously, increasing latency and killing
performance, and decreasing drive life. That's not an issue with SSD,
but multiple rebuilds is. That and simply keeping track of 80
partitions.
The rebuilds will, I believe, be done sequentially rather than in
parallel. And each rebuild will take 1/8 of the time a full array
rebuild would have done. So it really should not be much more time or
wear-and-tear for a rebuild of this monster setup, compared to a single
raid5 array rebuild. (With hard disks, it would be worse due to head
seeks - but still not as bad as you imply, if I am right about the
rebuilds being done sequentially.)
However, there was a recent thread here about someone with a similar
setup (on hard disks) who had a failure during such a rebuild and had
lots of trouble. That makes me sceptical to this sort of multiple array
setup (in addition to Stan's other points).
And of course, all Stan's other points about maintenance, updates to
later kernels with multiple raid5 threads, etc., still stand.
quoted
There are a couple of sane things you can do today to address your
problem:
1. Create a RAID50, a layered md/RAID0 over two 5 SSD md/RAID5 arrays.
This will double your threads and your IOPS. It won't be as fast as
your Frankenstein setup and you'll lose one SSD of capacity to
additional parity. However, it's sane, stable, doubles your
performance, and you have only one array to rebuild after an SSD
failure. Any filesystem will work well with it, including XFS if
aligned properly. It gives you an easy upgrade path-- as soon as the
threaded patches hit, a simple kernel upgrade will give your two RAID5
arrays the extra threads, so you're simply out one SSD of capacity. You
won't need to, and probably won't want to rebuild the entire thing after
the patch. With the Frankenstein setup you'll be destroying and
rebuilding arrays. And if these are consumer grade SSDs, you're much
better off having two drives worth of redundancy anyway, so a RAID50
makes good sense all around.
2. Make 5 md/RAID1 mirrors and concatenate them with md/RAID linear.
You'll get one md write thread per RAID1 device utilizing 5 cores in
parallel. The linear driver doesn't use threads, but passes offsets to
the block layer, allowing infinite core scaling. Format the linear
device with XFS and mount with inode64. XFS has been fully threaded for
15 years. Its allocation group design along with the inode64 allocator
allows near linear parallel scaling across a concatenated device[1],
assuming your workload/directory layout is designed for parallel file
throughput.
#2, with a parallel write workload, may be competitive with your
Frankenstein setup in both IOPS and throughput, even with 3 fewer RAID
threads and 4 fewer SSD "spindles". It will outrun the RAID50 setup
like it's standing still. You'll lose half your capacity to redundancy
as with RAID10, but you'll have 5 write threads for md/RAID1, one per
SSD pair. One core should be plenty to drive a single SSD mirror, with
plenty of cycles to spare for actual applications, while sparing 3 cores
for apps as well. You'll get unlimited core scaling with both md/linear
and XFS. This setup will yield the best balance of IOPS and throughput
performance for the amount of cycles burned on IO, compared to
Frankenstein and the RAID50.
For those that don't want to use XFS, or won't have balanced directories
in their filesystem, or want greater throughput of larger files (rather
than greater average throughput of multiple parallel accesses), you can
also take your 5 raid1 mirror pairs and combine them with raid0. You
should get similar scaling (the cpu does not limit raid0). For some
applications (such as mail server, /home mount, etc.), the XFS over a
linear concatenation is probably unbeatable. But for others (such as
serving large media files), a raid0 over raid1 pairs could well be
better. As always, it depends on your load - and you need to test with
realistic loads or at least realistic simulations.
Sure, a homemade RAID10 would work as it avoids the md/RAID10 single
write thread. I intentionally avoided mentioning this option for a few
reasons:
1. Anyone needing 10 SATA SSDs obviously has a parallel workload
2. Any thread will have up to 200-500MB/s available (one SSD)
with a concat, I can't see a single thread needing 4.5GB/s of B/W
If so, md/RAID isn't capable, not on COTS hardware
3. With a parallel workload requiring this many SSDs, XFS is a must
4. With a concat, mkfs.xfs is simple, no stripe aligning, etc
~$ mkfs.xfs /dev/md0
--
Stan
From: David Brown <hidden> Date: 2012-08-17 07:29:44
On 17/08/2012 09:15, Stan Hoeppner wrote:
On 8/16/2012 2:52 AM, David Brown wrote:
quoted
For those that don't want to use XFS, or won't have balanced directories
in their filesystem, or want greater throughput of larger files (rather
than greater average throughput of multiple parallel accesses), you can
also take your 5 raid1 mirror pairs and combine them with raid0. You
should get similar scaling (the cpu does not limit raid0). For some
applications (such as mail server, /home mount, etc.), the XFS over a
linear concatenation is probably unbeatable. But for others (such as
serving large media files), a raid0 over raid1 pairs could well be
better. As always, it depends on your load - and you need to test with
realistic loads or at least realistic simulations.
Sure, a homemade RAID10 would work as it avoids the md/RAID10 single
write thread. I intentionally avoided mentioning this option for a few
reasons:
1. Anyone needing 10 SATA SSDs obviously has a parallel workload
2. Any thread will have up to 200-500MB/s available (one SSD)
with a concat, I can't see a single thread needing 4.5GB/s of B/W
If so, md/RAID isn't capable, not on COTS hardware
3. With a parallel workload requiring this many SSDs, XFS is a must
4. With a concat, mkfs.xfs is simple, no stripe aligning, etc
~$ mkfs.xfs /dev/md0
These are all good points. There is always a lot to learn from your posts.
My only concern with XFS over linear concat is that its performance depends on the spread of allocation groups across the elements of the concatenation (the raid1 pairs in this case), and that in turn depends on the directory structure. (I'm sure you'll correct me if I'm wrong in this - indeed I would /like/ to be wrong!) If you have large numbers of top-level directories and a spread of access, then this is ideal. But if you have very skewed access with most access within only one or two top-level directories, then as far as I understand XFS allocation groups, access will then be concentrated heavily on only one (or a few) of the concat elements.
raid0 of the raid1 pairs may not be the best way to spread out access (assuming XFS linear concat is not a good fit for the workload), but it might still be an improvement. Perhaps a good solution would be raid0 with a very large chunk size - that make most accesses non-striped (as you say, the user probably doesn't need striping), thus allowing more parallel accesses, while scattering the accesses evenly across all raid1 elements?
Of course, we are still waiting to hear a bit about the OP's real load.
From: David Brown <hidden> Date: 2012-08-17 07:52:40
On 17/08/2012 00:11, vincent Ferrer wrote:
On Wed, Aug 15, 2012 at 10:58 PM, Stan Hoeppner [off-list ref] wrote:
quoted
On 8/15/2012 9:56 PM, vincent Ferrer wrote:
quoted
- My storage server has upto 8 cores running linux kernel 2.6.32.27.
- I created a raid5 device of 10 SSDs .
No it is not normal practice. I 'preach' against it regularly when I
see OPs doing it. It's quite insane.
There are a couple of sane things you can do today to address your problem:
Stan
Hi Stan,
Follow-up question for 2 types of setups i may have to prepare:
1) setup A has 80 SSDs. Question: Should I still create one
raid5 device or should I create 8 raid5 device each having 10 SSDs ?
My linux based storage server may be accessed by upto 10-20
physically different clients.
I have difficultly imagining the sort of workload that would justify 80 SSDs. Certainly you have to think about far more than just the disks or the raid setup - you would be looking at massive network bandwidth, multiple servers with large PCI express buses, etc. Probably you would want dedicated SAN hardware of some sort. Otherwise you could get pretty much the same performance and capacity using 10 hard disks (and maybe a little extra ram to improve caching).
But as a general rule, you want to limit the number of disks (or partitions) you have in a single raid5 to perhaps 6 devices. With too many devices, you increase the probability that you will get a failure, and then a second failure during a rebuild. You can use raid6 for extra protection - but that also (currently) suffers from the single-thread bottleneck.
Remember also that raid5 (or raid6) requires a RMW for updates larger than a single block but smaller than a full stripe - that means it needs to read from every disk in the array before it can write. The wider the array, the bigger effect this is.
2) Setup B has only 12 SSDs. Question: Is it more practical to
have only one raid5 device, even though I may have 4-5 physically
different clients or create 2 raid5 devices each having 6 SSDs.
Again, I would put only 6 disks in a raid5.
Reason I am asking because I have seen enterprise storage arrays from
EMC/IBM where new raid5 device is created on demand and (storage
firmware may spread across automatically across all the available
drives/spindles or can be intelligently selected by storage admin by
analyzing workload to avoid hot-spots)
Partitioning was only done because I am still waiting budget approval
to buy SSDs.
regards
vincy
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Stan Hoeppner <hidden> Date: 2012-08-17 08:29:37
On 8/16/2012 5:11 PM, vincent Ferrer wrote:
On Wed, Aug 15, 2012 at 10:58 PM, Stan Hoeppner [off-list ref] wrote:
quoted
On 8/15/2012 9:56 PM, vincent Ferrer wrote:
quoted
- My storage server has upto 8 cores running linux kernel 2.6.32.27.
- I created a raid5 device of 10 SSDs .
No it is not normal practice. I 'preach' against it regularly when I
see OPs doing it. It's quite insane.
There are a couple of sane things you can do today to address your problem:
Stan
Hi Stan,
Follow-up question for 2 types of setups i may have to prepare:
1) setup A has 80 SSDs.
This is simply silly. There's no need for exaggeration here.
2) Setup B has only 12 SSDs. Question: Is it more practical to
have only one raid5 device, even though I may have 4-5 physically
different clients or create 2 raid5 devices each having 6 SSDs.
As I repeat in every such case: What is your workload? Creating a
storage specification is driven by the requirements of the workload.
--
Stan
From: Stan Hoeppner <hidden> Date: 2012-08-17 10:52:03
On 8/17/2012 2:29 AM, David Brown wrote:
On 17/08/2012 09:15, Stan Hoeppner wrote:
quoted
On 8/16/2012 2:52 AM, David Brown wrote:
quoted
For those that don't want to use XFS, or won't have balanced directories
in their filesystem, or want greater throughput of larger files (rather
than greater average throughput of multiple parallel accesses), you can
also take your 5 raid1 mirror pairs and combine them with raid0. You
should get similar scaling (the cpu does not limit raid0). For some
applications (such as mail server, /home mount, etc.), the XFS over a
linear concatenation is probably unbeatable. But for others (such as
serving large media files), a raid0 over raid1 pairs could well be
better. As always, it depends on your load - and you need to test with
realistic loads or at least realistic simulations.
Sure, a homemade RAID10 would work as it avoids the md/RAID10 single
write thread. I intentionally avoided mentioning this option for a few
reasons:
1. Anyone needing 10 SATA SSDs obviously has a parallel workload
2. Any thread will have up to 200-500MB/s available (one SSD)
with a concat, I can't see a single thread needing 4.5GB/s of B/W
If so, md/RAID isn't capable, not on COTS hardware
3. With a parallel workload requiring this many SSDs, XFS is a must
4. With a concat, mkfs.xfs is simple, no stripe aligning, etc
~$ mkfs.xfs /dev/md0
These are all good points. There is always a lot to learn from your posts.
My only concern with XFS over linear concat is that its performance
depends on the spread of allocation groups across the elements of the
concatenation (the raid1 pairs in this case), and that in turn depends
on the directory structure. (I'm sure you'll correct me if I'm wrong in
this - indeed I would /like/ to be wrong!) If you have large numbers of
top-level directories and a spread of access, then this is ideal. But
if you have very skewed access with most access within only one or two
top-level directories, then as far as I understand XFS allocation
groups, access will then be concentrated heavily on only one (or a few)
of the concat elements.
This depends on the allocator. inode32, the default allocator, does
RAID0 with files--each file being a chunk. All inodes go in AG0, all
files round robin'd across the other AGs. Great for parallel streaming
workloads on a mirror concat, obviously not metadata intensive
workloads, as metadata is on the first spindle.
The optional inode64 allocator spreads inodes and files across all AGs.
Every new dir is created in a different AG round robin, regardless of
the on disk location of the parent dir. File however always get created
in their parent dir. Much better for metadata workloads. It's just as
good with parallel streaming workloads if the user has read the XFS
Users Guide and does some manual placement.
raid0 of the raid1 pairs may not be the best way to spread out access
(assuming XFS linear concat is not a good fit for the workload), but it
might still be an improvement. Perhaps a good solution would be raid0
with a very large chunk size - that make most accesses non-striped (as
you say, the user probably doesn't need striping), thus allowing more
parallel accesses, while scattering the accesses evenly across all raid1
elements?
No matter how anyone tries to slice it, striped RAID is only optimal for
streaming writes/reads of large files. This represents less than 1% of
real world workloads. The rest are all concurrent relatively small file
workloads, and for these using an intelligent filesystem with an
allocation group design (XFS, JFS) will yield better performance.
The only real benefit of striped RAID over concat, for the majority of
workloads, is $/GB.
Of course, we are still waiting to hear a bit about the OP's real load.
It seems clear he has some hardware, real or fantasy, in need of a
workload, so I'm not holding my breath.
--
Stan
From: David Brown <hidden> Date: 2012-08-17 11:47:49
On 17/08/2012 12:52, Stan Hoeppner wrote:
On 8/17/2012 2:29 AM, David Brown wrote:
quoted
On 17/08/2012 09:15, Stan Hoeppner wrote:
quoted
On 8/16/2012 2:52 AM, David Brown wrote:
quoted
For those that don't want to use XFS, or won't have balanced directories
in their filesystem, or want greater throughput of larger files (rather
than greater average throughput of multiple parallel accesses), you can
also take your 5 raid1 mirror pairs and combine them with raid0. You
should get similar scaling (the cpu does not limit raid0). For some
applications (such as mail server, /home mount, etc.), the XFS over a
linear concatenation is probably unbeatable. But for others (such as
serving large media files), a raid0 over raid1 pairs could well be
better. As always, it depends on your load - and you need to test with
realistic loads or at least realistic simulations.
Sure, a homemade RAID10 would work as it avoids the md/RAID10 single
write thread. I intentionally avoided mentioning this option for a few
reasons:
1. Anyone needing 10 SATA SSDs obviously has a parallel workload
2. Any thread will have up to 200-500MB/s available (one SSD)
with a concat, I can't see a single thread needing 4.5GB/s of B/W
If so, md/RAID isn't capable, not on COTS hardware
3. With a parallel workload requiring this many SSDs, XFS is a must
4. With a concat, mkfs.xfs is simple, no stripe aligning, etc
~$ mkfs.xfs /dev/md0
These are all good points. There is always a lot to learn from your posts.
My only concern with XFS over linear concat is that its performance
depends on the spread of allocation groups across the elements of the
concatenation (the raid1 pairs in this case), and that in turn depends
on the directory structure. (I'm sure you'll correct me if I'm wrong in
this - indeed I would /like/ to be wrong!) If you have large numbers of
top-level directories and a spread of access, then this is ideal. But
if you have very skewed access with most access within only one or two
top-level directories, then as far as I understand XFS allocation
groups, access will then be concentrated heavily on only one (or a few)
of the concat elements.
This depends on the allocator. inode32, the default allocator, does
RAID0 with files--each file being a chunk. All inodes go in AG0, all
files round robin'd across the other AGs. Great for parallel streaming
workloads on a mirror concat, obviously not metadata intensive
workloads, as metadata is on the first spindle.
The optional inode64 allocator spreads inodes and files across all AGs.
Every new dir is created in a different AG round robin, regardless of
the on disk location of the parent dir. File however always get created
in their parent dir. Much better for metadata workloads. It's just as
good with parallel streaming workloads if the user has read the XFS
Users Guide and does some manual placement.
It sounds like I /have/ misunderstood things - at least regarding the inode64 allocator (which will surely be the best choice for a large array). I had though that while directories "/a/" and "/b/" get different allocation groups, directories "/a/1/" and "/a/2/" would go in the same AG as "/a/". What you are saying is that this is not correct - each of these four directories would go in a separate AG. File "/a/1/x" would go in the same AG as "/a/1/", of course. Assuming this is the case, XFS over linear concat sounds more appealing for a much wider set of applications than I had previously thought.
quoted
raid0 of the raid1 pairs may not be the best way to spread out access
(assuming XFS linear concat is not a good fit for the workload), but it
might still be an improvement. Perhaps a good solution would be raid0
with a very large chunk size - that make most accesses non-striped (as
you say, the user probably doesn't need striping), thus allowing more
parallel accesses, while scattering the accesses evenly across all raid1
elements?
No matter how anyone tries to slice it, striped RAID is only optimal for
streaming writes/reads of large files. This represents less than 1% of
real world workloads. The rest are all concurrent relatively small file
workloads, and for these using an intelligent filesystem with an
allocation group design (XFS, JFS) will yield better performance.
The only real benefit of striped RAID over concat, for the majority of
workloads, is $/GB.
quoted
Of course, we are still waiting to hear a bit about the OP's real load.
It seems clear he has some hardware, real or fantasy, in need of a
workload, so I'm not holding my breath.
From: Stan Hoeppner <hidden> Date: 2012-08-18 04:55:41
On 8/17/2012 6:47 AM, David Brown wrote:
On 17/08/2012 12:52, Stan Hoeppner wrote:
It sounds like I /have/ misunderstood things - at least regarding the
inode64 allocator (which will surely be the best choice for a large
array). I had though that while directories "/a/" and "/b/" get
different allocation groups, directories "/a/1/" and "/a/2/" would go in
the same AG as "/a/". What you are saying is that this is not correct -
each of these four directories would go in a separate AG. File "/a/1/x"
would go in the same AG as "/a/1/", of course. Assuming this is the
case, XFS over linear concat sounds more appealing for a much wider set
of applications than I had previously thought.
I may bear some blame for this. Long ago I thought the inode64
allocator worked as you describe. I may have spread that
misinformation. If so, my apologies to all.
Indeed the inode64 allocator creates new directories evenly across all
AGs in a round robin fashion. Thus it will work better with most
workloads on storage with some level of concatenation than with straight
striped RAID on rust. This is due to excessive seeks across all the
AGs, which line up from outer to inner tracks when striping. With SSD
seek starvation is irrelevant, so concat and striping are pretty much equal.
quoted
The only real benefit of striped RAID over concat, for the majority of
workloads, is $/GB.
To be more clear, I was obviously referring to striped parity RAID here,
not RAID10, which has the same cost as concat+mirror.
--
Stan
From: David Brown <hidden> Date: 2012-08-18 08:59:05
On 18/08/12 06:55, Stan Hoeppner wrote:
On 8/17/2012 6:47 AM, David Brown wrote:
quoted
On 17/08/2012 12:52, Stan Hoeppner wrote:
quoted
It sounds like I /have/ misunderstood things - at least regarding the
inode64 allocator (which will surely be the best choice for a large
array). I had though that while directories "/a/" and "/b/" get
different allocation groups, directories "/a/1/" and "/a/2/" would go in
the same AG as "/a/". What you are saying is that this is not correct -
each of these four directories would go in a separate AG. File "/a/1/x"
would go in the same AG as "/a/1/", of course. Assuming this is the
case, XFS over linear concat sounds more appealing for a much wider set
of applications than I had previously thought.
I may bear some blame for this. Long ago I thought the inode64
allocator worked as you describe. I may have spread that
misinformation. If so, my apologies to all.
Indeed the inode64 allocator creates new directories evenly across all
AGs in a round robin fashion.
Thanks for clearing this up. I understand much better now why you are such a fan of XFS over concat - spreading all directories around like this will give better performance for a much wider set of workloads.
Thus it will work better with most
workloads on storage with some level of concatenation than with straight
striped RAID on rust. This is due to excessive seeks across all the
AGs, which line up from outer to inner tracks when striping. With SSD
seek starvation is irrelevant, so concat and striping are pretty much equal.
quoted
quoted
The only real benefit of striped RAID over concat, for the majority of
workloads, is $/GB.
To be more clear, I was obviously referring to striped parity RAID here,
not RAID10, which has the same cost as concat+mirror.