Re: recovering after a /dev/sda failure on raid1

5 messages, 4 authors, 2002-08-05 · open the first message on its own page

Re: recovering after a /dev/sda failure on raid1

From: Neil Brown <hidden>
Date: 2002-08-02 21:26:29

On Thursday August 1, vindex@apartia.org wrote:

Hi,

I have a root raid1 partition on /dev/sda1 & /dev/sdb1 (swap on
/dev/sda2 & /dev/sdb2). The server boots directly from the raid
partition. 

Now /dev/sda1 and /dev/sda2 have both failed and been removed from the
array and I am getting ready to replace the disk tonight.
Lucky you :-)
What is the best way to proceed to minimize downtime?

My concern is that if I power down and replace /dev/sda the machine
won't be able to reboot without a rescue CD (lilo.conf has root=/dev/md0
and boot=/dev/md0) or will it? 

When the bios (Dell Poweredge 1500) will try /dev/sda's mbr and fail,
will it then automatically try /dev/sdb?
With most bioses I have seen you can explicitly tell it which device
to boot from.  But cannot say for-sure about Dell Poweredge.
Or should I swap /dev/sdb on the scsi ribbon to have it take the first
place and thus become /dev/sda or will this just confuse the kernel
raid driver? (the letter on scsi drives is dependent on their place on
the ribbon cable, isn't it?)
It isn't the position on the ribbon cable.  It is the position in the
scsi device number ordering.  If  you make sure the new drive has a
larger number than the old drive, the old drive will appear as sda.
Alternatively I was thinking of booting with a rescue CD (after
replacing /dev/sda) with the "root=/dev/md0", creating my partitions,
running lilo and rebooting into production for final reconstruction.
Would that be the safest bet?
This sounds like the best bet to me.  You do have to boot twice, but
if you try the other, less well understood (by you atleast) approach,
there is an even chance you will need to reboot a couple of times
anyway.

NeilBrown

Thanks in advance for your insight, cheers,

-- 
vindex@apartia.org 
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: recovering after a /dev/sda failure on raid1

From: Danilo Godec <hidden>
Date: 2002-08-02 23:03:34

On Sat, 3 Aug 2002, Neil Brown wrote:
quoted
My concern is that if I power down and replace /dev/sda the machine
won't be able to reboot without a rescue CD (lilo.conf has root=/dev/md0
and boot=/dev/md0) or will it?
A recent enough lilo knows raid partition and will install it self on both
disks. So, in theory both disk should be able to boot you system.
quoted
When the bios (Dell Poweredge 1500) will try /dev/sda's mbr and fail,
will it then automatically try /dev/sdb?
It might not even try /dev/sdb if /dev/sda exists...
But with some recent bioses you can choose which drive you want to boot,
sou you could just choose your 2nd SCSI drive and it should work.


   D.

Re: recovering after a /dev/sda failure on raid1

From: Louis-David Mitterrand <hidden>
Date: 2002-08-05 09:26:41

On Sat, Aug 03, 2002 at 07:26:29AM +1000, Neil Brown wrote:
On Thursday August 1, vindex@apartia.org wrote:
quoted
I have a root raid1 partition on /dev/sda1 & /dev/sdb1 (swap on
/dev/sda2 & /dev/sdb2). The server boots directly from the raid
partition. 

Now /dev/sda1 and /dev/sda2 have both failed and been removed from the
array and I am getting ready to replace the disk tonight.
Lucky you :-)
It went fast fortunately ;-) And yes, I am very lucky to have raid1
notify me by email through mdadm of a disk failure.

FWIW the disk that is malfunctioning is a 3-month-old Fujitsu 15k 36G
(MAM3367MP) which is an expensive server-grade disk. The reason I
selected Fujitsu was because of reported quality problems on IBM disks
and Fujitsu's good reputation on SCSI (their IDE line is bad however).
I am looking for informed opinions on these disks and recommendations
for future purchases. What are the most reliable SCSI disks out there?

It must be: fast, affordable, reliable, (select any two ;-)
quoted
What is the best way to proceed to minimize downtime?

My concern is that if I power down and replace /dev/sda the machine
won't be able to reboot without a rescue CD (lilo.conf has root=/dev/md0
and boot=/dev/md0) or will it? 

When the bios (Dell Poweredge 1500) will try /dev/sda's mbr and fail,
will it then automatically try /dev/sdb?
With most bioses I have seen you can explicitly tell it which device
to boot from.  But cannot say for-sure about Dell Poweredge.
Yes, I found it's in the SCSI bios itself. Very configurable.

Unfortunately when I tried booting from /dev/sdb the screen filled with
010101010 instead of "lilo". And this is on debian unstable, having run
lilo on /dev/md0 just prior booting. In fact I found that having root
and boot set to /dev/md0 in lilo.conf does not allow me to boot my raid1
partition. However if I set root=/dev/sda all goes well. Any trick here?
(both disks are identical)
quoted
Or should I swap /dev/sdb on the scsi ribbon to have it take the first
place and thus become /dev/sda or will this just confuse the kernel
raid driver? (the letter on scsi drives is dependent on their place on
the ribbon cable, isn't it?)
It isn't the position on the ribbon cable.  It is the position in the
scsi device number ordering.  If  you make sure the new drive has a
larger number than the old drive, the old drive will appear as sda.
I'm really ashamed to have asked that one, memory lapse on my part.
quoted
Alternatively I was thinking of booting with a rescue CD (after
replacing /dev/sda) with the "root=/dev/md0", creating my partitions,
running lilo and rebooting into production for final reconstruction.
Would that be the safest bet?
This sounds like the best bet to me.  You do have to boot twice, but
if you try the other, less well understood (by you atleast) approach,
there is an even chance you will need to reboot a couple of times
anyway.
Having failed to boot /dev/sdb this is what I ended up doing and all
went well.

Thanks for your help and ideas, cheers,

-- 
ldm@apartia.org 

Re: recovering after a /dev/sda failure on raid1

From: Louis-David Mitterrand <hidden>
Date: 2002-08-05 09:34:40

On Sat, Aug 03, 2002 at 01:03:34AM +0200, Danilo Godec wrote:
On Sat, 3 Aug 2002, Neil Brown wrote:
quoted
quoted
My concern is that if I power down and replace /dev/sda the machine
won't be able to reboot without a rescue CD (lilo.conf has root=/dev/md0
and boot=/dev/md0) or will it?
A recent enough lilo knows raid partition and will install it self on both
disks. So, in theory both disk should be able to boot you system.
In theory, because I was unable to boot this debian sid system with
lilo-22.2 and these settings:

	boot=/dev/md0
	root=/dev/md0
	install=/boot/boot-menu.b
	delay=20
	map=/boot/map
	read-only

	image=/vmlinuz
		label=Linux

After altering the SCSI bios setting to boot from /dev/sdb the screen
filled with 01010101010.

However this is not specific to booting on degraded mode. I have the
same problem when both disks are fine. The machine will only boot after
I changed "boot=/dev/sda" and run lilo. I can then revert to
"boot=/dev/md0" and re-run lilo and from there booting will work fine.
quoted
quoted
When the bios (Dell Poweredge 1500) will try /dev/sda's mbr and fail,
will it then automatically try /dev/sdb?
It might not even try /dev/sdb if /dev/sda exists...
But with some recent bioses you can choose which drive you want to boot,
sou you could just choose your 2nd SCSI drive and it should work.
Yes, I found that out, sorry for the basic questions but I was a bit
stressed changing a disk on a production server and having it not
restart. But all went well in the end.

Thanks again to kernel-raid developers, on another server I replaced a
Mylex card with a kernel raid5 partition and the speed gain is
tremendous!

Cheers,

-- 
ldm@apartia.org 

Re: recovering after a /dev/sda failure on raid1

From: Maurice Hilarius <hidden>
Date: 2002-08-05 18:35:47

With regards to your message at 03:26 AM 8/5/02, Louis-David Mitterrand. 
Where you stated:
<<snip>>
FWIW the disk that is malfunctioning is a 3-month-old Fujitsu 15k 36G
(MAM3367MP) which is an expensive server-grade disk. The reason I
selected Fujitsu was because of reported quality problems on IBM disks
and Fujitsu's good reputation on SCSI (their IDE line is bad however).
I am looking for informed opinions on these disks and recommendations
for future purchases. What are the most reliable SCSI disks out there?

It must be: fast, affordable, reliable, (select any two ;-)
The Fujitsu disks have a track record of good reliability.
The IDE disk you mention were discontinued a year ago.
In our experiences Fujitsu disks have as low a failure rate as can be found.
Drives we have used with higher failure rates (SCSI) came from IBM and Seagate.
The best failure rate we have seen is from Hitachi.
In the case of Fujitsu, Hitachi, and Seagate the rates are very close.
Quantum/Maxtor SCSI are a bit worse.
IBM are substantially worse.
This is based on our DOA and in service SCSI disk failures over the past year.


With our best regards,

Maurice W. Hilarius       Telephone: 01-780-456-9771
Hard Data Ltd.               FAX:       01-780-456-9772
11060 - 166 Avenue        mailto:maurice@harddata.com
Edmonton, AB, Canada      http://www.harddata.com/
    T5X 1Y3

2.3TB RAID5 NAS server - dual AthlonMP CPU, Linux, $10,995 CAD / $6850 USD
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help