From: Tony Coffman <hidden> Date: 2008-08-26 15:32:54
I've have a Centos5 box running a software raid-1 set on a pair of SATA
drives.
The SATA controller or driver has a flaw.
Every 150 days or so, one of the two drives will experience errors and fail.
Subsequent tests always show the drive and cable to be ok. We bought a
couple of replacement drives before we figured that out :-(
On the last event this weekend, I went searching for a way to get the
raid back online with no host downtime. I found the technique that
deletes the drive and then brings it back online with a bus scan using
the /sys filesystem delete and rescan entities.
I didn't realize that you could also perform a rescan on a single LUN.
I'll have to use that next time.
My question - since I've done a delete/rescan bus operation, my device
name and major,minor numbers have changed.
Original
[0:0:0:0] disk ATA ST3250410AS 3.AA /dev/sdc
Current
[0:0:0:0] disk ATA ST3250410AS 3.AA /dev/sdc
If I re-add the device to the raid set using the new device name, will
it cause any problems on the next boot?
The drive appears to be fine. I can read all blocks with no errors.
Partition table looks ok, etc..
In the future if I rescan just the single LUN, I'm pretty sure I won't
run into again this but I'd like to avoid an outage on this event if
possible.
Thanks and regards,
--Tony
From: David Lethe <hidden> Date: 2008-08-27 11:11:07
-----Original Message-----
From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
owner@vger.kernel.org] On Behalf Of Tony Coffman
Sent: Tuesday, August 26, 2008 10:33 AM
To: linux-raid@vger.kernel.org
Subject: Device naming and raid1
I've have a Centos5 box running a software raid-1 set on a pair of
SATA
drives.
The SATA controller or driver has a flaw.
Every 150 days or so, one of the two drives will experience errors and
fail.
Subsequent tests always show the drive and cable to be ok. We bought
a
couple of replacement drives before we figured that out :-(
On the last event this weekend, I went searching for a way to get the
raid back online with no host downtime. I found the technique that
deletes the drive and then brings it back online with a bus scan using
the /sys filesystem delete and rescan entities.
I didn't realize that you could also perform a rescan on a single LUN.
I'll have to use that next time.
My question - since I've done a delete/rescan bus operation, my device
name and major,minor numbers have changed.
Original
[0:0:0:0] disk ATA ST3250410AS 3.AA /dev/sdc
Current
[0:0:0:0] disk ATA ST3250410AS 3.AA /dev/sdc
If I re-add the device to the raid set using the new device name, will
it cause any problems on the next boot?
The drive appears to be fine. I can read all blocks with no errors.
Partition table looks ok, etc..
In the future if I rescan just the single LUN, I'm pretty sure I won't
run into again this but I'd like to avoid an outage on this event if
possible.
Thanks and regards,
--Tony
--
To unsubscribe from this list: send the line "unsubscribe linux-raid"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.htm
Don't be too quick to say the drive(s) are good, or for that matter,
making any assumptions about what is bad or good. (Well, OK, let's
assume the monitor is good). If the drives are reporting errors and
the drives fail, why not trap the error messages and do some diagnostics
while drives are still in that failed state? Error messages tell you
what the errors are. Make yourself a bootable CDROM or USB and next
time the drives lockup and/or start spitting out errors, then capture
everything. Then boot to the external device (do NOT cycle power), and
run one of many possible diagnostics to confirm or eliminate the disks.
From: Tony Coffman <hidden> Date: 2008-08-27 13:21:14
David Lethe wrote:
quoted
-----Original Message-----
From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-
owner@vger.kernel.org] On Behalf Of Tony Coffman
Sent: Tuesday, August 26, 2008 10:33 AM
To: linux-raid@vger.kernel.org
Subject: Device naming and raid1
I've have a Centos5 box running a software raid-1 set on a pair of
SATA
quoted
drives.
The SATA controller or driver has a flaw.
Every 150 days or so, one of the two drives will experience errors and
fail.
Subsequent tests always show the drive and cable to be ok. We bought
a
quoted
couple of replacement drives before we figured that out :-(
On the last event this weekend, I went searching for a way to get the
raid back online with no host downtime. I found the technique that
deletes the drive and then brings it back online with a bus scan using
the /sys filesystem delete and rescan entities.
I didn't realize that you could also perform a rescan on a single LUN.
I'll have to use that next time.
My question - since I've done a delete/rescan bus operation, my device
name and major,minor numbers have changed.
Original
[0:0:0:0] disk ATA ST3250410AS 3.AA /dev/sdc
Current
[0:0:0:0] disk ATA ST3250410AS 3.AA /dev/sdc
If I re-add the device to the raid set using the new device name, will
it cause any problems on the next boot?
The drive appears to be fine. I can read all blocks with no errors.
Partition table looks ok, etc..
In the future if I rescan just the single LUN, I'm pretty sure I won't
run into again this but I'd like to avoid an outage on this event if
possible.
Thanks and regards,
--Tony
--
To unsubscribe from this list: send the line "unsubscribe linux-raid"
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.htm
Don't be too quick to say the drive(s) are good, or for that matter,
making any assumptions about what is bad or good. (Well, OK, let's
assume the monitor is good). If the drives are reporting errors and
the drives fail, why not trap the error messages and do some diagnostics
while drives are still in that failed state? Error messages tell you
what the errors are. Make yourself a bootable CDROM or USB and next
time the drives lockup and/or start spitting out errors, then capture
everything. Then boot to the external device (do NOT cycle power), and
run one of many possible diagnostics to confirm or eliminate the disks.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Thanks much for the reply. For the purposes of this discussion you can
assume that I've already re-established confidence in the drive, the
cable, and the controller and that the data on the drives is worthless
and I just want to get maximum uptime without causing a raid assemble
problem on the next reboot.
Any idea on my original question? If I re-add the drive using the
/dev/sdc name will I have problems on the next boot when the drive is
named /dev/sda?
Based on my experience with Linux and other software raid
implementations, I'm strongly inclined to think that the device naming
doesn't matter - the system will scan the drives at boot looking for
raid sets and re-assemble them no matter what major and minor numbers or
device names are. I'm not opposed to finding out the hard way but I'd
really like to get a definitive answer now because by the time this
system is next rebooted I'll probably have long forgotten about this.
Regards,
--Tony
Thanks much for the reply. For the purposes of this discussion you can
assume that I've already re-established confidence in the drive, the
cable, and the controller and that the data on the drives is worthless
and I just want to get maximum uptime without causing a raid assemble
problem on the next reboot.
Good.
Any idea on my original question? If I re-add the drive using the
/dev/sdc name will I have problems on the next boot when the drive is
named /dev/sda?
Since this seems to be block device it really does not matter.
Based on my experience with Linux and other software raid
implementations, I'm strongly inclined to think that the device naming
doesn't matter - the system will scan the drives at boot looking for
Kindly read some decent kernel documentation before you jump up and
say this. Kindly surf the net and read some decent article's before you
do any precious upgrades for now.
Sujit
--
--linux(2.4/2.6),bsd(4.5.x+),solaris(2.5+)
http://www.gagme.com/greg/linux/raid-lvm.php
you can try this with the spare drives you have.
basically what you have to do is to check whether the drive
now linked to another device name, is the reason for this
problem.
once it shows unplugged or failed you can, use your new
replacement drive and reboot.
Kindly read the comments to this article, Which is very
usefull.
On 8/27/08, Sujit Karataparambil [off-list ref] wrote:
quoted
Thanks much for the reply. For the purposes of this discussion you can
assume that I've already re-established confidence in the drive, the
cable, and the controller and that the data on the drives is worthless
and I just want to get maximum uptime without causing a raid assemble
problem on the next reboot.
Good.
quoted
Any idea on my original question? If I re-add the drive using the
/dev/sdc name will I have problems on the next boot when the drive is
named /dev/sda?
Since this seems to be block device it really does not matter.
quoted
Based on my experience with Linux and other software raid
implementations, I'm strongly inclined to think that the device naming
doesn't matter - the system will scan the drives at boot looking for
Kindly read some decent kernel documentation before you jump up and
say this. Kindly surf the net and read some decent article's before you
do any precious upgrades for now.
Sujit
--
--linux(2.4/2.6),bsd(4.5.x+),solaris(2.5+)
From: Tony Coffman <hidden> Date: 2008-08-27 14:48:27
Sujit Karataparambil wrote:
http://www.gagme.com/greg/linux/raid-lvm.php
you can try this with the spare drives you have.
basically what you have to do is to check whether the drive
now linked to another device name, is the reason for this
problem.
once it shows unplugged or failed you can, use your new
replacement drive and reboot.
Kindly read the comments to this article, Which is very
usefull.
On 8/27/08, Sujit Karataparambil [off-list ref] wrote:
quoted
quoted
Thanks much for the reply. For the purposes of this discussion you can
assume that I've already re-established confidence in the drive, the
cable, and the controller and that the data on the drives is worthless
and I just want to get maximum uptime without causing a raid assemble
problem on the next reboot.
Good.
quoted
Any idea on my original question? If I re-add the drive using the
/dev/sdc name will I have problems on the next boot when the drive is
named /dev/sda?
Since this seems to be block device it really does not matter.
quoted
Based on my experience with Linux and other software raid
implementations, I'm strongly inclined to think that the device naming
doesn't matter - the system will scan the drives at boot looking for
Kindly read some decent kernel documentation before you jump up and
say this. Kindly surf the net and read some decent article's before you
do any precious upgrades for now.
Sujit
--
--linux(2.4/2.6),bsd(4.5.x+),solaris(2.5+)
Sujit,
Thanks for the replies and the link. I appreciate them.
I spent several hours this week reading the kernel documentation
(md.txt), the mdadm man pages, the linux-raid wiki, and reading articles
on the net before posting to the list.
I highly recommend this wiki page from IBM for anybody who has an issue
similar to mine (temporary failure that knocks a drive offline who wants
to bring it back online without a reboot). This really helped me
understand the processes for running a highly available raid-1 set on
the linux kernel. The hardware is different but the same principles work.
http://www-941.ibm.com/collaboration/wiki/pages/viewpage.action?pageId=3625
For anybody who is interested, I grabbed a test system this morning to
simulate my situation (temporary drive failure). You can, in fact,
bring the failed drive back online with a different device name,
remirror it, and reboot with no issues on Centos5. The key, as Steve
Fairbairn pointed out, is that the mdadm.conf file is setup to use the
RAID UUID. I moved the drives into a different scan order with a the
same test and that works also. My situation is slightly complicated by
the fact that I'm booting off these same drives so I had to mirror the
MBR on the second drive but since I had already done this previously on
the wonky system this was easily achieved.
see here
http://www.dirigo.net/tuxTips/avoidingProblems/GrubMdMbr.php
I'm running a couple of more tests today to see what happens if I rescan
the device to bring it back online with the same device name - I'll post
the result in case anybody is interested. I expect that I'll have to
initiate the rebuild but don't expect any other problems.
Regards,
--Tony
From: Steve Fairbairn <hidden> Date: 2008-08-27 13:42:55
Any idea on my original question? If I re-add the drive
using the /dev/sdc name will I have problems on the next boot
when the drive is named /dev/sda?
Based on my experience with Linux and other software raid
implementations, I'm strongly inclined to think that the
device naming doesn't matter - the system will scan the
drives at boot looking for raid sets and re-assemble them no
matter what major and minor numbers or device names are. I'm
not opposed to finding out the hard way but I'd really like
to get a definitive answer now because by the time this
system is next rebooted I'll probably have long forgotten about this.
Not an expert in any sense of the word, but if the array is being
identified by its UUID and not by a specific list of devices in
mdadm.conf, then the array should assemble fine after a reboot with the
device names having changed. I recently had a similar situation where I
added a new PCI-E x1 2 port SATA II card, which promptly pushed all my
other sdx devices down two places. The raid mounted as expected and all
was/is wonderful.
Hope this helps,
Steve.