2.6.27.8: OOM killer: [swapper: page allocation failure]

Started by Justin Piszcz on Dec 12, 2008 · 4 messages from 2 authors · Last reply Dec 16, 2008

open the first message on its own page

Message details and related content
Kernel: 2.6.27.8

I have a RAID-6 volume with 6 disks:
mdadm --create -v /dev/md0 --level=6 -n 6 -e 0.90 /dev/sd[e-j]1

While it is building...

# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid6 sdj1[5] sdi1[4] sdh1[3] sdg1[2] sdf1[1] sde1[0]
      1562834944 blocks level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU]
      [===>.................]  resync = 19.0% (74237440/390708736) finish=346.7min speed=15208K/sec

unused devices: <none>

During this time, I am copying files to it at ~60-100MiB/s using 2 rsyncs over NFS from 2 separate machines => this one [over the network] (on a pci-e x1 / built-in to the motherboard).

top output:
top - 07:10:06 up 12:32, 30 users,  load average: 3.46, 4.49, 4.39
Tasks: 316 total,   2 running, 313 sleeping,   1 stopped,   0 zombie
Cpu(s): 12.1%us,  3.7%sy,  0.0%ni, 32.5%id, 49.2%wa,  1.1%hi,  1.2%si,  0.0%st
Mem:   8100836k total,  8057480k used,    43356k free,        0k buffers
Swap: 16787884k total,      144k used, 16787740k free,  6609412k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5349 root      15  -5     0    0    0 R   10  0.0  11:59.94 md0_raid5
    1 root      20   0 10312  748  620 S    0  0.0   0:02.14 init

I see this over and over in the kernel log (maybe it will stop when the RAID6
rebuild is completed)?  So far, my rsyncs are continuing just fine and nothing except klogd/swapper processes have been killed.

[44893.846532] swapper: page allocation failure. order:0, mode:0x20
[44893.846538] Pid: 0, comm: swapper Not tainted 2.6.27.8 #2
[44893.846540] [44893.846540] Call Trace:
[44893.846542]  <IRQ>  [<ffffffff80279d4d>] __alloc_pages_internal+0x3cd/0x4d0
[44893.846551]  [<ffffffff8029876b>] cache_alloc_refill+0x2fb/0x570
[44893.846553]  [<ffffffff80298abb>] __kmalloc+0xdb/0xe0
[44893.846557]  [<ffffffff8058289d>] __alloc_skb+0x6d/0x150
[44893.846559]  [<ffffffff80583407>] __netdev_alloc_skb+0x17/0x40
[44893.846563]  [<ffffffff804a248b>] e1000_alloc_rx_buffers+0x25b/0x2e0

673 more lines

Message details and related content

On Fri, 12 Dec 2008, Justin Piszcz wrote:
Kernel: 2.6.27.8

I have a RAID-6 volume with 6 disks:
mdadm --create -v /dev/md0 --level=6 -n 6 -e 0.90 /dev/sd[e-j]1

While it is building...
Removed the extra disks, put them in another host but thought it was an
interesting bug.? so sent it to the list.

Justin.
Message details and related content
On Fri, Dec 12, 2008 at 5:14 AM, Justin Piszcz [off-list ref] wrote:
Kernel: 2.6.27.8

I have a RAID-6 volume with 6 disks:
mdadm --create -v /dev/md0 --level=6 -n 6 -e 0.90 /dev/sd[e-j]1

While it is building...

# cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md0 : active raid6 sdj1[5] sdi1[4] sdh1[3] sdg1[2] sdf1[1] sde1[0]
     1562834944 blocks level 6, 64k chunk, algorithm 2 [6/6] [UUUUUU]
     [===>.................]  resync = 19.0% (74237440/390708736)
finish=346.7min speed=15208K/sec

unused devices: <none>

During this time, I am copying files to it at ~60-100MiB/s using 2 rsyncs
over NFS from 2 separate machines => this one [over the network] (on a pci-e
x1 / built-in to the motherboard).

top output:
top - 07:10:06 up 12:32, 30 users,  load average: 3.46, 4.49, 4.39
Tasks: 316 total,   2 running, 313 sleeping,   1 stopped,   0 zombie
Cpu(s): 12.1%us,  3.7%sy,  0.0%ni, 32.5%id, 49.2%wa,  1.1%hi,  1.2%si,
 0.0%st
Mem:   8100836k total,  8057480k used,    43356k free,        0k buffers
Swap: 16787884k total,      144k used, 16787740k free,  6609412k cached

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 5349 root      15  -5     0    0    0 R   10  0.0  11:59.94 md0_raid5
   1 root      20   0 10312  748  620 S    0  0.0   0:02.14 init

I see this over and over in the kernel log (maybe it will stop when the
RAID6
rebuild is completed)?  So far, my rsyncs are continuing just fine and
nothing except klogd/swapper processes have been killed.

[44893.846532] swapper: page allocation failure. order:0, mode:0x20
[..]
Raid optimizations used:

# cat oraid.sh |grep -v ^#|grep .
. /etc/profile
echo "Optimizing RAID Arrays..."
cd /sys/block
DISKS=$(/bin/ls -1d sd[e-j])
echo "Setting read-ahead to 32 MiB for /dev/md0"
blockdev --setra 65536 /dev/md0
echo "Setting stripe_cache_size to 16 MiB for /dev/md0"
echo 16384 > /sys/block/md0/md/stripe_cache_size
If you set the stripe_cache_size high enough you can strangle the
system.  Do you really need 384MiB of raid cache?
Message details and related content

On Mon, 15 Dec 2008, Dan Williams wrote:
On Fri, Dec 12, 2008 at 5:14 AM, Justin Piszcz [off-list ref] wrote:
quoted
[44893.846532] swapper: page allocation failure. order:0, mode:0x20
[..]
quoted
Raid optimizations used:

# cat oraid.sh |grep -v ^#|grep .
. /etc/profile
echo "Optimizing RAID Arrays..."
cd /sys/block
DISKS=$(/bin/ls -1d sd[e-j])
echo "Setting read-ahead to 32 MiB for /dev/md0"
blockdev --setra 65536 /dev/md0
echo "Setting stripe_cache_size to 16 MiB for /dev/md0"
echo 16384 > /sys/block/md0/md/stripe_cache_size
If you set the stripe_cache_size high enough you can strangle the
system.  Do you really need 384MiB of raid cache?
The weird thing is, with md/RAID-5, I never had a problem.

Justin.
Keyboard shortcuts
hback out one level
jnext message in thread / next series-diff position
kprevious message in thread / previous series-diff position
ldrill in
Escclose help / fold thread tree
?toggle this help