Re: Linux RAID autodetect partitions go missing from /dev, but fdisk can see them

Hi Neil!

Neil Brown wrote:

On Wednesday December 10, torarnv@gmail.com wrote:

quoted

I have a very strange problem that I've been trying to debug for
days now. I had a RAID5 with four drives and one spare,
/dev/sd[bcde]1 + /dev/sdf1, and everything was working fine, until
one day one of the drives in the array (sdb) no longer had a
partition (sdb1). Letting the spare take over I ignored this for a
few days, but then it happened again, this time with sdc1.

quoted

I'm hoping someone on this list may have ran into this before, or
have any tips on how I can continue debugging this, because I have to
admit I'm a little lost...

Yes, it does sound rather weird.

First of all, thank you so much for helping me out with this, as I'm
still very lost :)

In addition to the things listed in the first e-mail, I've also tried
installing the latest kernel from kernel.org, but that did not solve
anything. Also, in case it's relevant, I'm running openSUSE 10.3.

Can you:

  mdadm -Esv

http://pastebin.com/d7b14d14e

For some reason it seems to think that /dev/sdc and /dev/sdb are part of
the array, while it really is /dev/sdc1 and /dev/sdb1. I'm guessing
since they are missing somehow from the device nodes in /dev mdadm
assumes the disk itself is the member?

and
  mdadm --stop /dev/md0
  strace -o /tmp/str -s 200 mdadm --assemble --scan --verbose /dev/md0

http://pastebin.com/f2c1db2e4

The original array had sd[bcde]1 + sdf1 as spare. Then sdb1 went missing
and the spare kicked in, and then sdc1 went missing, leaving me with a
degraded array.

Also the contents of /etc/mdadm.conf might help.

http://pastebin.com/f573346ef

Is there anything else I can run, cat, and/or paste that would shed
light over what's going on?

Thanks,

Thank _you_ :)

Tor Arne

quoted

raid support in. The symptoms are:

  - The kernel seems to detect the partitions (lines 396 and 407 in the
dmesg [1])

  - But once the boot process finishes and the RAID is started, there is
no longer any sdc1 or sdb1, so the RAID fails to start (lines 550-576 in
dmesg [1])

  - Running fdisk -l shows that the drives in question (sdb and sdc) do
have similar partitions as the other working drives, namely one Linux
RAID autodetect partition each (see command output [2])

  - But, the partitions are missing from /proc/partitions (see [3])

  - Manually adding device nodes using mknod works, but doing file -sL
on the device gives "writable, no read permission", even though
permissions are the same as the other sd* nodes in /dev

  - Running 'partprobe -s' successfully finds the two missing partitions
and adds device nodes, and the nodes can be 'file -sL'ed, but when
trying to assemble the array again with these new nodes in the system,
I'm told that sdc1 is not found, and after the --assemble is done, the
device nodes are once again missing (!) see [4]

  - I've tried using the 'dmraid' command to look for fakeraid
partitions or meta data on the drives, which I was told could mess up
the auto-detection of Linux software ride partitions, but could not find
any issues.


As you can tell I've exhausted all my current options, so any help on
what I could try next would be very much appreciated. I am especially
curious as to why I lose the partitions when mdadm tries to assemble the
array?

Thanks!

Tor Arne Vestbø

[1] http://pastebin.com/m15b9c275   dmesg
[2] http://pastebin.com/f50fb323a   fdisk -l
[3] http://pastebin.com/f4547c2ca   cat /proc/partitions
[4] http://pastebin.com/m4475c9ae   partprobe + mdadm --assemble
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html