On Wed, 26 Nov 2014 13:41:39 +0530 Manish Awasthi
[off-list ref] wrote:
On 11/25/2014 08:07 AM, NeilBrown wrote:
quoted
On Mon, 24 Nov 2014 13:40:06 +0530 Manish Awasthi
[off-list ref] wrote:
quoted
Hi,
We benchmarked the md raid driver performance on 3-18-rc3 kernel and
compared the results with that of 3.6.11. The reason for this exercise
is to understand if multithreaded raid driver has any performance
benefits over 3.6.11 which is single threaded. Here are some details
about the setup
Thanks for doing this!!!! I love it when people report test results.
quoted
System: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz 4 cores (8threads),
8GB RAM.
Setup: 3 SSDs create a raid5 array
test tool: iozone (only read/re-read, write/re-write tested), blocksize:
4k-64k, filesize: 1Gig to 200Gig
Comparison was done for speed of data transfer in kBytes/sec and also
the CPU utilization as reported by iozone.
raid on 3.18.0-rc3 performed much worse than raid on 3.6.11.
Read/Write: raid on 3.18.0-rc3 operated at almost half the speed of raid
on 3.6.11
That really isn't very good.... Can you try some of the kernels in between
and see if there was a single point where performance dropped, or if there
were several steps?
Can you give me some starting point when multithread support for raid
was added. It might be a good starting point. For instance, I have a
benchmark on 3.6.11, now I'd like to go to the first kernel that has
support for multithread raid and then take it from there.
Multi-thread support appeared in 3.12.
quoted
quoted
CPU Utilization: With md raid on 3.18.0-rc3, the CPU utilization was
less than half of md raid on 3.6.11 on WRITE operations. However, for
READ operations, 3.18-0.rc3 had more CPU utilization than 3.6.11.
Can you use "perf" to determine where the extra time is going?
perf record
run test
stop perf
perf report
or something like that.
I can do this but as I mentioned below, Its better if I can get to
understand all the possible tweaks that can be done to get the optimal
results unless ofcourse you expect 3.18.0 to perform better than that of
3.6.11 even if default case without any tweaks.
I have no particular expectations. I like to see concrete measurements and
then try to interpret them.
I prefer to compare default setting (no tweaks) in the first instance.
Because that is what most people will be using.
quoted
quoted
Also, I noticed that scaling up the CPU cores of the system scales down
the raid througput with 3.18.0-rc3.
This is by writing numbers to "group_thread_cnt" ??? Can you provide a simple
table comparing thread count to throughput? Or maybe a graph. I love
graphs :-)
I did not tweak anything on the 3.18.0 kernel. I assumed all the
required support is built-in and did not bother to go into the depth of
the code as we're still in nascent stages where we are comparing the
data on specific kernel versions. Can you point me to some text that can
describe tweaks like "group_thread_cnt" etc?
Multi-threading is disabled by default, so if you haven't explicitly enabled
it, then it cannot be affecting your performance.
If you
echo 8 > /sys/block/mdXXX/md/group_thread_cnt
it will use 8 thread to perform 'xor' calculations and submit IO requests.
quoted
quoted
I do have detailed logs of the comparison but I'm not sure I should send
those on this mailing list.
A few megabytes? Yes. 100Meg? No.
Whatever data I have on comparison is attached, I have consolidated this
from log files to excel. See if this helps.
Thanks. I'll have a look at the tables and see if anything looks interesting.
Thanks,
NeilBrown
quoted
If you could put them on a website somewhere that I can browse or download
I'll try to have a look.
quoted
If my observation aligns with someone else's, then what is really the
gain with multithreaded raid.
Some testing shows real improvements. Obviously we cannot test everything
and I'm very glad to have extra testing from other people.
If we can quantify the regressions and confirm exactly when they occurred, we
can start looking for a solution.
Thanks a lot!
NeilBrown
quoted
Manish
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html