On Fri, Dec 12, 2008 at 5:14 AM, Justin Piszcz [off-list ref] wrote:

Kernel: 2.6.27.8

I have a RAID-6 volume with 6 disks:
mdadm --create -v /dev/md0 --level=6 -n 6 -e 0.90 /dev/sd[e-j]1

While it is building...

# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid6 sdj1[5] sdi1[4] sdh1[3] sdg1[2] sdf1[1] sde1[0]
     1562834944 blocks level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU]
     [===>.................]  resync = 19.0% (74237440/390708736)
finish=346.7min speed=15208K/sec

unused devices: <none>

During this time, I am copying files to it at ~60-100MiB/s using 2 rsyncs
over NFS from 2 separate machines => this one [over the network] (on a pci-e
x1 / built-in to the motherboard).

top output:
top - 07:10:06 up 12:32, 30 users,  load average: 3.46, 4.49, 4.39
Tasks: 316 total,   2 running, 313 sleeping,   1 stopped,   0 zombie
Cpu(s): 12.1%us,  3.7%sy,  0.0%ni, 32.5%id, 49.2%wa,  1.1%hi,  1.2%si,
 0.0%st
Mem:   8100836k total,  8057480k used,    43356k free,        0k buffers
Swap: 16787884k total,      144k used, 16787740k free,  6609412k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5349 root      15  -5     0    0    0 R   10  0.0  11:59.94 md0_raid5
   1 root      20   0 10312  748  620 S    0  0.0   0:02.14 init

I see this over and over in the kernel log (maybe it will stop when the
RAID6
rebuild is completed)?  So far, my rsyncs are continuing just fine and
nothing except klogd/swapper processes have been killed.

[44893.846532] swapper: page allocation failure. order:0, mode:0x20

[..]

Raid optimizations used:

# cat oraid.sh |grep -v ^#|grep .
. /etc/profile
echo "Optimizing RAID Arrays..."
cd /sys/block
DISKS=$(/bin/ls -1d sd[e-j])
echo "Setting read-ahead to 32 MiB for /dev/md0"
blockdev --setra 65536 /dev/md0
echo "Setting stripe_cache_size to 16 MiB for /dev/md0"
echo 16384 > /sys/block/md0/md/stripe_cache_size

If you set the stripe_cache_size high enough you can strangle the
system.  Do you really need 384MiB of raid cache?