tj wrote:

Bill Davidsen wrote:

quoted

tj wrote:

quoted

Thomas Jager wrote:

quoted

Hi list.

I run a file server on MD raid-5.
If a client reads one big file and at the same time another client 
tries to write a file, the thread writing just sits in 
uninterruptible sleep until the reader has finished. Only very 
small amount of writes get trough while the reader is still working.
I'm having some trouble pinpointing the problem.
It's not consistent either sometimes it works as expected both the 
reader and writer gets some transactions. On huge reads I've seen 
the writer blocked for 30-40 minutes without any significant writes 
happening (Maybe a few megabytes, of several gigs waiting). It 
happens with NFS, SMB and FTP, and local with dd. And seems to be 
connected to raid-5. This does not happen on block devices without 
raid-5. I'm also wondering if it can have anything to do with 
loop-aes? I use loop-aes on top of the md, but then again i have 
not observed this problem on loop-devices with disk backend. I do 
know that loop-aes degrades performance but i didn't think it would 
do something like this?

I've seen this problem in 2.6.16-2.6.21

All disks in the array is connected to a controller with a SiI 3114 
chip.

I just noticed something else. A couple of slow readers where 
running on my raid-5 array. Then i started a copy from another local 
disk to the array. Then i got the extremely long wait. I noticed 
something in iostat:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          3.90    0.00   48.05   31.93    0.00   16.12

Device:            tps    kB_read/s    kB_wrtn/s    kB_read    kB_wrtn
....
sdg               0.80        25.55         0.00        128          0
sdh             154.89       632.34         0.00       3168          0
sdi               0.20        12.77         0.00         64          0
sdj               0.40        25.55         0.00        128          0
sdk               0.40        25.55         0.00        128          0
sdl               0.80        25.55         0.00        128          0
sdm               0.80        25.55         0.00        128          0
sdn               0.60        23.95         0.00        120          0
md0             199.20       796.81         0.00       3992          0

All disks are member of the same raid array (md0). One of the disks 
has a ton of transactions compared to the other disks. Read 
operations as far as i can tell. Why? May be connected with my problem? 

Two thoughts on that, if you are doing a lot of directory operations, 
it's possible that the inodes being used most are all in one chunk.

Hi thanks for the reply.

It's not directory operations AFAIK. Reading a few files (3 in this 
case) and writing one.

quoted

The other possibility is that these a journal writes and reflect 
updates to the atime. The way to see if this is in some way  related 
is to mount (remount) with noatime: "mount -o remount,noatime 
/dev/md0 /wherever" and retest. If this is journal activity you can 
do several things to reduce the problem, which I'll go into (a) if it 
seems to be the problem, and (b) if someone else doesn't point you to 
an existing document or old post on the topic. Oh, you could also try 
mounting the filesystem as etc2, assuming that it's ext3 now. I 
wouldn't run that way, but it's useful as a diagnostic tool.

I don't use ext3 i use ReiserFS. ( It seemed like a good idea at the 
time. ) It's mounted with  -o  noatime.
I've done some more testing and i seems like it might be connected to 
mount --bind. If i write to a binded mount i get the slow writes. But 
if i write directly to the real mount i don't. It might just be a 
random occurrence, as the problem always has been inconsistent. Thoughts? 

I don't beat on the bind mounts, let me do a test and get back.

-- 
bill davidsen [off-list ref]
  CTO TMR Associates, Inc
  Doing interesting things with small computers since 1979