Re: mdadm problems with partitioned md devices

Neil Brown wrote:

On Tuesday November 23, Carl-Daniel Hailfinger wrote:

quoted

Hi,

[please CC: me]
while trying to use md to overcome limitations of dm, I
stumbled upon the following problem: It is impossible to
have more than 4 partitioned md devices since mdadm will
create the 5th device with major:minor 255:0, which is
bogus.

You need user-space (libc in particular) that supports the new 32bit
device numbers in 2.6.  I don't know which version of glibc that would
be, or which distributions support it.  debian/unstable (which I use)
doesn't support it yet :-(

Use SUSE. They support it even in older distributions like 9.1 which
is freely downloadable by now. At least the glibc-2.3.3 shipped with
SUSE 9.1 already supports a large dev_t.

You could test your distribution with
   mknod /tmp/test b 253 1024
   ls -l /tmp/test
   rm /tmp/test

and see if the major/minor is reported as "253, 1024" or "253, 0".
If the later, you need new glibc.
If the former, your userspace is up-to-date but there must be a bug in
mdadm.
If this is the case, let me know and I will look into it.

I fear there is a bug in mdadm or in the headers it includes:

linux:~ # mknod /tmp/test b 253 1024
linux:~ # ls -l /tmp/test
brw-r--r--  1 root root 253, 1024 Nov 24 00:59 /tmp/test
linux:~ # rm /tmp/test
linux:~ # strace mknod /tmp/test b 253 1024 2>&1|grep ^mknod
mknod("/tmp/test", S_IFBLK|0666, makedev(253, 1024)) = 0
linux:~ # rm /dev/md_d*
linux:~ # strace -o out.md0 mdadm -B /dev/md_d0 -f -l linear -n 1 -ap63 /dev/hdb5
mdadm: array /dev/md_d0 built and started.
linux:~ # strace -o out.md1 mdadm -B /dev/md_d1 -f -l linear -n 1 -ap63 /dev/hdb6
mdadm: array /dev/md_d1 built and started.
linux:~ # strace -o out.md2 mdadm -B /dev/md_d2 -f -l linear -n 1 -ap63 /dev/hdb7
mdadm: array /dev/md_d2 built and started.
linux:~ # strace -o out.md3 mdadm -B /dev/md_d3 -f -l linear -n 1 -ap63 /dev/hdb8
mdadm: array /dev/md_d3 built and started.
linux:~ # strace -o out.md4 mdadm -B /dev/md_d4 -f -l linear -n 1 -ap63 /dev/hdb9
mdadm: error opening /dev/md_d4: No such device or address
linux:~ # grep mknod out.md*|grep -v _p
out.md0:mknod("/dev/md_d0", S_IFBLK|0600, makedev(254, 0)) = 0
out.md1:mknod("/dev/md_d1", S_IFBLK|0600, makedev(254, 64)) = 0
out.md2:mknod("/dev/md_d2", S_IFBLK|0600, makedev(254, 128)) = 0
out.md3:mknod("/dev/md_d3", S_IFBLK|0600, makedev(254, 192)) = 0
out.md4:mknod("/dev/md_d4", S_IFBLK|0600, makedev(255, 0)) = 0
linux:~ #

Doing a more thorough analysis, it seems the MAJOR and MINOR macros
used by mdadm via includes can only handle the small dev_t.

Oh well... I just fixed it. Basically you have to replace all
occurences of MKDEV(X) with makedev(x), MAJOR(x) with major(x) and
MINOR(x) with minor(x) and it will work just fine. I have attached
a patch but I don't want to just mindlessly do a search/replace
operation without checking for issues with tools I didn't test,
so please test if it works for you.

Proof that it works for me:
linux:~ # strace -o out.md4 mdadm -B /dev/md_d4 -f -l linear -n 1 -ap63 /dev/hdb9
mdadm: array /dev/md_d4 built and started.
linux:~ # grep ^mknod out.md*|grep -v _p
out.md0:mknod("/dev/md_d0", S_IFBLK|0600, makedev(254, 0)) = 0
out.md1:mknod("/dev/md_d1", S_IFBLK|0600, makedev(254, 64)) = 0
out.md2:mknod("/dev/md_d2", S_IFBLK|0600, makedev(254, 128)) = 0
out.md3:mknod("/dev/md_d3", S_IFBLK|0600, makedev(254, 192)) = 0
out.md4:mknod("/dev/md_d4", S_IFBLK|0600, makedev(254, 256)) = 0
linux:~ #
[after some more experimenting]
linux:~ # ls -l /dev/md_d255
brw-rw----  1 root disk 254, 16320 Nov 24 03:03 /dev/md_d255
linux:~ #

However, it seems I have run into the 256 md devices limit ;-)

But I stumbled on another bug: It is possible to specify numbers >63
for the -ap parameter. Since the kernel always assumes 64 minors
per partitioned md device, this will cause mdadm to create devices
which overlap in minor ranges. Please introduce a limit for that
parameter to avoid nasty surprises.

And a feature request: Would it be possible to allow naming like
/dev/d0
/dev/d0p1
/dev/d0p2
etc. ? Right now, an unserscore character is enforced after a digit
and this prevents me from emulating some traditional device names.

Expert question: Since the only thing I do with md is abuse it
for supporting 63 partitions on generic block devices, is there
a tool instead of mdadm which fits my purposes better?

And another bug, this time in the kernel: Partitions for md devices
don't disappear from sysfs after deletion of the underlying md
device until the deleted device is opened again. Please fix that
(if it seems to difficult, my "partitioned loop device" patch
could serve as a starting point).

Thanks for reading this far.

Regards,
Carl-Daniel
-- 
http://www.hailfinger.org/