On Mon, Aug 19, 2019 at 11:37:09PM +0500, Roman Mamedov wrote:

quoted

quoted

Default Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disabled

So does it have a BBU? (Battery backup unit)

 
Yes, it does and it's working.
But because I found that with write caching enabled, it seemd to take all
the writes from the raid rebuild in a big queue, and starving I/O for others
requests that I wanted to happen "right now" (like /bin/ls actually being
loaded and running), and reading how the perc 5/i is a crap card, I turned
off its IO caching, leaving the work to linux' block buffer and the 32GB of
RAM in the server that are mostly allocated to disk IO caching.

quoted

I tried to disable the card's write cache to let linux and its 32GB of
RAM, do it better, but I didn't see a real improvement:

I'd expect that on the contrary, you should look for ways to enable it, and
force-enable even without that BBU (in case of lack of one), because it feels
like what you did is disable disks' own write buffering, and not (only) the
card's!

Yes, you may be correct on that. I can re-enable it, but it was terrible
with it on, too.

What you are observing seems to me like what "dd" does with "oflag=dsync" (and
comparable performance that it gets). Definitely feels like it's in some
"extra safe mode" and, say, every 4KB piece of data leads to full flush to
disk before accepting to write the next 4KB.

That sounds plausible indeed.

More things to try, check if it's possible to set up disks not as 1-member
RAID0, but 1-member "linear" ("JBOD"), or even 1-member RAID1, who knows maybe
some of this would work better.

Assuming I can do this without losing the entire filesystem, I can try, but
if it can't do single drive raid0, I doubt changing this to single drive
raid1 would make things much better.
Then again, once you're hitting things that aren't working as they should...

I have an H700 in the mail that should arrive tonight, I'll try swapping
that first and see what happens.

Thanks for the answer,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/