On 10/01/2022 13:38, Jaromír Cápík wrote:

Nope, I haven't read the code. I only see a low sync speed (fluctuating from 20
to 80MB/s) whilst the drives can perform much better doing sequential reading
and writing (250MB/s per drive and up to 600MB/s all 4 drives in total).
During the sync I hear a high noise caused by heads flying there and back and
that smells.

Okay, so read performance from the array is worse than you would expect 
from a single drive. And the heads should not be "flying there and back" 
- they should just be streaming data. That's actually worrying - a VERY 
plausible explanation is that your drives are on the verge of failure!!

The chosen drives have poor seeking performance and small caches and are
probably unable to reorder the operations to be more sequential. The whole
solution is 'economic' since the organisation owning the solution is poor and
cannot afford better hardware.

The drives shouldn't need to reorder the operations - a rebuild is an 
exercise in pure streaming ... unless there are so many badblocks the 
whole drive is a mess ...

That also means RAID6 is not an option. But we shouldn't search excuses what's
wrong on the chosen scenario when the code is potentially suboptimal :] We're
trying to make Linux better, right? :]

I'm searching for someone, who knows the code well and can confirm my findings
or who could point me at anything I could try in order to increase the rebuild
speed. So far I've tried changing the readahead, minimum resync speed, stripe
cache size, but it increased the resync speed by few percent only.

Actually, you might find (counter-intuitive though it sounds) REDUCING 
the max sync speed might be better ... I'd guess from what you say, 
about 60MB/s.

The other thing is, could you be confusing MB and Mb? Three 250Mb drives 
would peak at about 80MB.

I believe I would be able to write my own userspace application for rebuilding
the array offline with much higher speed ... just doing XOR of bytes at the same
offsets. That would prove the current rebuild strategy is suboptimal.

Of course it would mean a new code if it doesn't work like suggested and I know
it could be difficult and requiring a deep knowledge of the linux-raid code that
unfortunately I don't have.

What make/model are your drives? What does smartdrv say about them? And 
take a look at

https://www.ept.ca/features/everything-need-know-hard-drive-vibration/

The thing that worries me is your reference to repeated seeks. That 
should NOT be happening. Unless of course the system is in heavy use at 
the same time as the rebuild.

Cheers,
Wol