On Sunday May 19, cwebster@ec.rr.com wrote:
Looking for help from list members in resolving software RAID problems.
Here is the help:
1/ You seem to have two copies of raid5.o which seems to be confusing
modprobe.
I don't know why, but in one instance it tried to load
/lib/raid5.o
and gets :unresolved symbol md_unregister_thread_R4ba824f9
The other time it correctly loads
/lib/modules/2.4.18-0.92sparc/kernel/drivers/md/raid5.o
I suggest removing /lib/raid5.c
2/ 2.4 raid is *not* upwards compatable with patched 2.2 raid on Sparc64
or any architecture that requires a 64bit integer to be 64bit
aligned instead of just 32bit aligned.
I'm sorry, but this was a bug in the raid patches and it is not
practical to reliably detect raid devices that were built
badly because of this bug. The bug results in the on-disc
superblock being laid out wrongly.
You will need to remake the array for 2.4.
If you remake the array correctly, you will not lose any
data.
Alternately, if you are feeling adventurous, you could download
and compile mdadm 1.0.1 from
http://www.cse.unsw.edu.au/~neilb/source/mdadm/
For each device that should be part of the array (e.g. /dev/sda1)
run
mdadm --examine /dev/sda1
This should show you some sensible information up to the Checksum,
and then bad information after that.
Then run
mdadm --examine --sparc2.2 /dev/sda1
This should show you the same sensible information, and then
sensible information after the Checksum as well.
If it does, then run
mdadm --examine --sparc2.2update /dev/sda1
and this should show the same sensible information, and then update
the superblock on the device to have this information too.
After this,
mdadm --examine --sparc2.2 /dev/sda1
should show the correct information.
If this works, you can then run the same command on the order
devices, and then assemble the array with
mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 ...
or whatever your actual device names are.
Note: I have not tested this code, but it looks OK.
If you try it and something goes wrong, then give me as much
information as you can and I will do my best to make sure you get
your data back.
Note2: --sparc2.2 and --sparc2.2update are not yet documented,
except in this mail item.
If you confirm that they work, I will add some documentation.
NeilBrown
Thank you for the excellent information and suggestions. I would like to try
using mdadm, but had some trouble compiling it. Also, please see notes on
raid5 modules below.
-----Original Message-----
From: Neil Brown [mailto:neilb@notabene]On Behalf Of Neil Brown
Sent: Monday, May 20, 2002 5:27 AM
To: cwebster@ec.rr.com
Cc: linux-raid
Subject: Re: Software RAID Stopped Working With Aurora Kernel
On Sunday May 19, cwebster@ec.rr.com wrote:
quoted
Looking for help from list members in resolving software RAID problems.
Here is the help:
1/ You seem to have two copies of raid5.o which seems to be confusing
modprobe.
I don't know why, but in one instance it tried to load
/lib/raid5.o
and gets :unresolved symbol md_unregister_thread_R4ba824f9
The other time it correctly loads
/lib/modules/2.4.18-0.92sparc/kernel/drivers/md/raid5.o
I suggest removing /lib/raid5.c
Strangely, the only time this error occurred was when I tried to boot with
"md0" devices listed in /etc/fstab. I don't know why it reported this path.
There are no modules directly under /lib. Here's where all the modules are.
I will remove all but the current kernel modules. In fact, I'll be removing
all the 2.2 kernels since I can't boot them any more anyway.
[root@winggear redhat]# find /lib -name raid5.o
/lib/modules/2.2.14-5.0/block/raid5.o
/lib/modules/2.2.19-6.2.1/block/raid5.o
/lib/modules/2.2.19-6.2.12/block/raid5.o
/lib/modules/2.2.19-6.2.16/block/raid5.o
/lib/modules/2.4.18-0.91sparc/kernel/drivers/md/raid5.o
/lib/modules/2.4.18-0.92sparc/kernel/drivers/md/raid5.o
Alternately, if you are feeling adventurous, you could download
and compile mdadm 1.0.1 from
http://www.cse.unsw.edu.au/~neilb/source/mdadm/
Using your source RPM:
[root@winggear redhat]# rpm -bc --target sparc-redhat-linux SPECS/mdadm.spec
Building target platforms: sparc-redhat-linux
Building for target sparc-redhat-linux
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.90707
+ umask 022
+ cd /usr/src/redhat/BUILD
+ cd /usr/src/redhat/BUILD
+ rm -rf mdadm-1.0.1
+ /bin/gzip -dc /usr/src/redhat/SOURCES/mdadm-1.0.1.tgz
+ tar -xf -
gzip: /usr/src/redhat/SOURCES/mdadm-1.0.1.tgz: decompression OK, trailing
garbag
e ignored
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd mdadm-1.0.1
++ /usr/bin/id -u
+ '[' 0 = 0 ']'
+ /bin/chown -Rhf root .
++ /usr/bin/id -u
+ '[' 0 = 0 ']'
+ /bin/chgrp -Rhf root .
+ /bin/chmod -Rf a+rX,g-w,o-w .
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.90717
+ umask 022
+ cd /usr/src/redhat/BUILD
+ cd mdadm-1.0.1
+ make 'CXFLAGS=-O2 -m32 -mtune=ultrasparc' SYSCONFDIR=/etc
gcc -Wall -Werror -Wstrict-prototypes -DCONFFILE=\"/etc/mdadm.conf\" -O2 -m3
2 -m
tune=ultrasparc -c -o mdadm.o mdadm.c
cc1: warnings being treated as errors
In file included from mdadm.h:57,
from mdadm.c:30:
md_p.h: In function `md_event':
md_p.h:168: warning: left shift count >= width of type
make: *** [mdadm.o] Error 1
error: Bad exit status from /var/tmp/rpm-tmp.90717 (%build)
RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.90717 (%build)
============= Compilers and Libraries ================
gcc-2.96-101
gcc-sparc32-2.96-101gcc-c++-2.96-101
gcc-g77-2.96-101
gcc3-g77-3.0.1-3
gcc-c++-sparc32-2.96-101
gcc3-3.0.1-3
libgcc-3.0.1-3
gcc3-c++-3.0.1-3
gcc-objc-2.96-101
gcc-chill-2.96-101
glib-1.2.10-5
glibc-2.2.4-19
glib-devel-1.2.10-5
compat-glibc-6.2-2.1.3.2
glibc-profile-2.2.4-19
glib10-1.0.6-10
glibc-devel-2.2.4-19
glibc-common-2.2.4-19
Neil Brown wrote:
On Sunday May 19, cwebster@ec.rr.com wrote:
quoted
Looking for help from list members in resolving software RAID problems.
Here is the help:
For each device that should be part of the array (e.g. /dev/sda1)
run
mdadm --examine /dev/sda1
This should show you some sensible information up to the Checksum,
and then bad information after that.
Yup, just like you said... sensible before, nonsense after.
Then run
mdadm --examine --sparc2.2 /dev/sda1
This should show you the same sensible information, and then
sensible information after the Checksum as well.
This looks much better, but not just right. I would expect that the list
following "Chunk Size" would be close to the /etc/raidtab that was being
used prior to the Aurora upgrade when it was last working. At any rate,
the "this" device and the last device listed are definitely not correct.
I'm not sure what "this" should be, but I assume it should list
parameters for the device being queried.
What I'm getting with "mdadm --examine --sparc2.2":
The first device listed ("this") always has a minor number of "8". Major
numbers for the "this" device on each raid drive, a through f, are
4,5,0,2,3,1, respectively. I don't know if this is random or somehow
correlates to the original raid device sequence. None of these is an
actual raid device, though.
The last device listed is always:
5 0 5 8 33 faulty sync
The remaining devices in the middle of each list, although out of
sequence in terms of /etc/raidtab, have the correct major/minor device
numbers. To be honest, though, I don't remember if the device list in
/proc/mdstat prior to the Aurora upgrade actually matched /etc/raidtab
in terms of raid device numbers either. If I remember correctly, they
seemed to be reversed but it worked so I left it alone. I do distinctly
remember that, prior to the Aurora upgrade, all raid devices were
operational and there was no reconstruction in progress. The system was
virtually quiescent.
How should the spare drive show up in these lists? /dev/sdc1 is missing
from all the lists. Could this have been the actual spare?
I've included below, the complete output from "mdadm --examine
--sparc2.2" for each raid device. First, I've listed the contents of
/etc/raidtab and a long listing of the actual raid devices. Please let
me know if this makes any sense to you.
Thanks for the help!
--Cal Webster
==============================================================================================
## Contents of /etc/raidtab
------------------------------------------------
#
# 'persistent' RAID5 setup, with one spare disk:
#
raiddev /dev/md0
raid-level 5
nr-raid-disks 5
nr-spare-disks 1
persistent-superblock 1
chunk-size 128
device /dev/sdb1
raid-disk 1
device /dev/sdc1
raid-disk 2
device /dev/sdd1
raid-disk 3
device /dev/sde1
raid-disk 4
device /dev/sda1
raid-disk 0
device /dev/sdf1
spare-disk 0
------------------------------------------------
## Raid devices:
----------------------------------------------------------------------
brw-rw---- 1 root disk 8, 1 May 5 1998 sda1
brw-rw---- 1 root disk 8, 17 May 5 1998 sdb1
brw-rw---- 1 root disk 8, 33 May 5 1998 sdc1
brw-rw---- 1 root disk 8, 49 May 5 1998 sdd1
brw-rw---- 1 root disk 8, 65 Apr 16 1999 sde1
brw-rw---- 1 root disk 8, 81 Apr 16 1999 sdf1
----------------------------------------------------------------------
----------------[ mdadm --examine --sparc2.2 /dev/sda1
]----------------
/dev/sda1:
Magic : a92b4efc
Version : 00.90.00
UUID : 494dd54f:15e7b548:90bae5c3:532ac910
Creation Time : Thu Apr 26 12:29:38 2001
Raid Level : raid5
Device Size : 17681664 (16.86 GiB 18.15 GB)
Raid Devices : 5
Total Devices : 6
Preferred Minor : 0
Update Time : Mon May 6 18:41:41 2002
State : clean, no-errors
Active Devices : 5
Working Devices : 6
Failed Devices : 0
Spare Devices : 1
Checksum : 6515abae - correct
--- adjusting superblock for 2.2/sparc compatability ---
Events : 0.171
Layout : left-asymmetric
Chunk Size : 128K
Number Major Minor RaidDevice State
this 0 4 8 1
0 0 8 17 1 active sync /dev/sdb1
1 1 8 81 2 active sync /dev/sdf1
2 2 8 49 3 active sync /dev/sdd1
3 3 8 65 4 active sync /dev/sde1
4 4 8 1 0 active sync /dev/sda1
5 0 5 8 33 faulty sync
----------------[ mdadm --examine --sparc2.2 /dev/sdb1
]----------------
/dev/sdb1:
Magic : a92b4efc
Version : 00.90.00
UUID : 494dd54f:15e7b548:90bae5c3:532ac910
Creation Time : Thu Apr 26 12:29:38 2001
Raid Level : raid5
Device Size : 17681664 (16.86 GiB 18.15 GB)
Raid Devices : 5
Total Devices : 6
Preferred Minor : 0
Update Time : Mon May 6 18:41:41 2002
State : clean, no-errors
Active Devices : 5
Working Devices : 6
Failed Devices : 0
Spare Devices : 1
Checksum : 6515abbb - correct
--- adjusting superblock for 2.2/sparc compatability ---
Events : 0.171
Layout : left-asymmetric
Chunk Size : 128K
Number Major Minor RaidDevice State
this 0 0 8 17 faulty
0 0 8 17 1 active sync /dev/sdb1
1 1 8 81 2 active sync /dev/sdf1
2 2 8 49 3 active sync /dev/sdd1
3 3 8 65 4 active sync /dev/sde1
4 4 8 1 0 active sync /dev/sda1
5 0 5 8 33 faulty sync
----------------[ mdadm --examine --sparc2.2 /dev/sdc1
]----------------
/dev/sdc1:
Magic : a92b4efc
Version : 00.90.00
UUID : 494dd54f:15e7b548:90bae5c3:532ac910
Creation Time : Thu Apr 26 12:29:38 2001
Raid Level : raid5
Device Size : 17681664 (16.86 GiB 18.15 GB)
Raid Devices : 5
Total Devices : 6
Preferred Minor : 0
Update Time : Mon May 6 18:41:41 2002
State : clean, no-errors
Active Devices : 5
Working Devices : 6
Failed Devices : 0
Spare Devices : 1
Checksum : 6515abce - correct
--- adjusting superblock for 2.2/sparc compatability ---
Events : 0.171
Layout : left-asymmetric
Chunk Size : 128K
Number Major Minor RaidDevice State
this 0 5 8 33 faulty sync
0 0 8 17 1 active sync /dev/sdb1
1 1 8 81 2 active sync /dev/sdf1
2 2 8 49 3 active sync /dev/sdd1
3 3 8 65 4 active sync /dev/sde1
4 4 8 1 0 active sync /dev/sda1
5 0 5 8 33 faulty sync
----------------[ mdadm --examine --sparc2.2 /dev/sdd1
]----------------
/dev/sdd1:
Magic : a92b4efc
Version : 00.90.00
UUID : 494dd54f:15e7b548:90bae5c3:532ac910
Creation Time : Thu Apr 26 12:29:38 2001
Raid Level : raid5
Device Size : 17681664 (16.86 GiB 18.15 GB)
Raid Devices : 5
Total Devices : 6
Preferred Minor : 0
Update Time : Mon May 6 18:41:41 2002
State : clean, no-errors
Active Devices : 5
Working Devices : 6
Failed Devices : 0
Spare Devices : 1
Checksum : 6515abdf - correct
--- adjusting superblock for 2.2/sparc compatability ---
Events : 0.171
Layout : left-asymmetric
Chunk Size : 128K
Number Major Minor RaidDevice State
this 0 2 8 49 faulty active /dev/fd0h1200
0 0 8 17 1 active sync /dev/sdb1
1 1 8 81 2 active sync /dev/sdf1
2 2 8 49 3 active sync /dev/sdd1
3 3 8 65 4 active sync /dev/sde1
4 4 8 1 0 active sync /dev/sda1
5 0 5 8 33 faulty sync
----------------[ mdadm --examine --sparc2.2 /dev/sde1
]----------------
/dev/sde1:
Magic : a92b4efc
Version : 00.90.00
UUID : 494dd54f:15e7b548:90bae5c3:532ac910
Creation Time : Thu Apr 26 12:29:38 2001
Raid Level : raid5
Device Size : 17681664 (16.86 GiB 18.15 GB)
Raid Devices : 5
Total Devices : 6
Preferred Minor : 0
Update Time : Mon May 6 18:41:41 2002
State : clean, no-errors
Active Devices : 5
Working Devices : 6
Failed Devices : 0
Spare Devices : 1
Checksum : 6515abf1 - correct
--- adjusting superblock for 2.2/sparc compatability ---
Events : 0.171
Layout : left-asymmetric
Chunk Size : 128K
Number Major Minor RaidDevice State
this 0 3 8 65 sync /dev/hda8
0 0 8 17 1 active sync /dev/sdb1
1 1 8 81 2 active sync /dev/sdf1
2 2 8 49 3 active sync /dev/sdd1
3 3 8 65 4 active sync /dev/sde1
4 4 8 1 0 active sync /dev/sda1
5 0 5 8 33 faulty sync
----------------[ mdadm --examine --sparc2.2 /dev/sdf1
]----------------
/dev/sdf1:
Magic : a92b4efc
Version : 00.90.00
UUID : 494dd54f:15e7b548:90bae5c3:532ac910
Creation Time : Thu Apr 26 12:29:38 2001
Raid Level : raid5
Device Size : 17681664 (16.86 GiB 18.15 GB)
Raid Devices : 5
Total Devices : 6
Preferred Minor : 0
Update Time : Mon May 6 18:41:41 2002
State : clean, no-errors
Active Devices : 5
Working Devices : 6
Failed Devices : 0
Spare Devices : 1
Checksum : 6515abfd - correct
--- adjusting superblock for 2.2/sparc compatability ---
Events : 0.171
Layout : left-asymmetric
Chunk Size : 128K
Number Major Minor RaidDevice State
this 0 1 8 81 active /dev/ram8
0 0 8 17 1 active sync /dev/sdb1
1 1 8 81 2 active sync /dev/sdf1
2 2 8 49 3 active sync /dev/sdd1
3 3 8 65 4 active sync /dev/sde1
4 4 8 1 0 active sync /dev/sda1
5 0 5 8 33 faulty sync