Re: OOPS in raid10.c:1448 in vanilla 2.6.13.2

On Tuesday September 20, guus@luna.nl wrote:

Hi,

On a new server I've been playing with software raid10.

...

Sep 20 19:25:09 lorentz kernel: ------------[ cut here ]------------
Sep 20 19:25:09 lorentz kernel: kernel BUG at drivers/md/raid10.c:1448!

This is fixed in 2.6.14-rc1.  I've attached the two patches that
resolves this and a related problem.

However I should point out that what you did was guaranteed to kill
the array (though mdadm-2.1 should have been able to re-assemble it
with --force):

# mdadm --create /dev/md1 --level=10 --raid-devices=4 /dev/sd[abcd]2

Now the four slots are 
   sda2 sdb2 sdc2 sdd2

sda2 and sdb2 have the same data, as do sdc2 and sdd2

Sep 20 19:08:23 lorentz kernel: raid10: Disk failure on sdb2, disabling
device.
Sep 20 19:08:23 lorentz kernel: ^IOperation continuing on 3 devices

now you have
   sda2 missing sdc2 sdd2

Then removed sdc from the server. Also took a while, but sdc2 also got
marked faulty.

now
   sda2 missing missing sdd2
all data is still safe.

Filesystem on /mnt stays usable during all this (slight hickup when a
disk is removed, but keeps going)
Then reinserted sdc. To get it resynced I did:
# mdadm /dev/md1 -r /dev/sdc2
# mdadm /dev/md1 -a /dev/sdc2
And it happily resynced and made sdc2 healthy again.

now
   sda2 sdc2 missing sdd2

note that sdc2 took the first empty slot.

Then removed sdd and sdd2 got marked faulty.

Now
   sda2 sdc2 missing missing

so you have lost half your data.

                                             Filesystem kept working.

This is a bit odd - it should have stopped working, but that depends
on how hard you pushed the filesystem.  It would have failed if you
tried to read or write in the area covered by the 2 two devices.

So at that point I was happy. At that point I just wanted to readd both
disks and get on with installing the server. So I reinserted both disks
m, waited several minutes and did:
# mdadm /dev/md1 -r /dev/sdb2
# mdadm /dev/md1 -r /dev/sdd2
# mdadm /dev/md1 -a /dev/sdb2

now you have
  sda2 sdc2 sdb2-not-sync missing

it tried to recover sdb2, but there was no-where to recover from.
This caused the BUG.  The patches cause this situation to simply
abort the recovery rather than BUG.

NeilBrown

Hi,

First of all: thanks for the speedy reply! Both patches applied cleanly
to 2.6.13.2 and work as expected. No more oops :) I hope these patches
make it into the next stable release of the 2.6.13 branch.

On Wed, 2005-09-21 at 10:30 +0200, Neil Brown wrote:

<snip>

quoted

Filesystem on /mnt stays usable during all this (slight hickup when a
disk is removed, but keeps going)
Then reinserted sdc. To get it resynced I did:
# mdadm /dev/md1 -r /dev/sdc2
# mdadm /dev/md1 -a /dev/sdc2
And it happily resynced and made sdc2 healthy again.

now
   sda2 sdc2 missing sdd2

note that sdc2 took the first empty slot.

That's where I went wrong. I thought it would take it's "old" place
again at the second empty slot.

<snip the rest>

Thanks again!

Regards,

Guus Houtzager
-- 
                              Luna.nl B.V.
                Puntegaalstraat 109 * 3024 EB Rotterdam
              T 010 7502000 * F 010 7502002 * www.luna.nl