I've tried a variety of settings... ranging from 17 to 32768.

Yes.. with stripe_cache_size set to 17, I see a C/T of rmw's.  And my TP goes in the toilet -- even with the RAM disks, I get only about 30M/s.

Bob Kierski
Senior Storage Performance Engineer
Cray Inc.
380 Jackson Street
Suite 210
St. Paul, MN 55101
Tele: 651-967-9590
Fax:  651-605-9001
Cell: 651-890-7461


-----Original Message-----
From: Phil Turmel [mailto:philip@turmel.org] 
Sent: Wednesday, December 02, 2015 9:37 AM
To: Robert Kierski; Dallas Clement
Cc: linux-raid@vger.kernel.org
Subject: Re: RAID 5,6 sequential writing seems slower in newer kernels

On 12/02/2015 10:28 AM, Robert Kierski wrote:

Thanks for the response.

Nice try... But, the reason I’m using the 3.18.4 kernel is that it has the parallelization.  I've got group_thread_cnt set to 32.  I'm watching the CPU's with mpstat, and they're pretty much idle.  I'm also watching the system traces with perf.  It claims that only 11.9% of my time is spent doing the xor.

Hmm. Ok.

I've got my CS set at 128k.  I have noticed that if I set the CS to 32k, the TP is about 2x.  I'm pretty sure the problem is that the 1M writes I'm doing are being broken into 4K pages, and then reassembled before going to disk.

I think you're right.  What is your stripe cache size?

Also, this is independent of the IO Scheduler.  I've tried all 3 and got the same results.

If your stripe cache is too small, sequential writes with large chunks can exhaust the cache before complete stripes are written, turning all of those partial stripe writes into read-modify-write cycles.