I had a somewhat strange error today.
One of my servers has a RAID1 array. Two partitions at the end of the
disk; the RAID superblocks are at the end of the partition.
After a hard reboot today, one of the disks managed to not have its
partition table scanned correctly, most probably because the disk was
hung and the ("intelligent") controller got confused about it. After the
initial scan, however, it came up correctly.
This error caused mdadm to "successfully" build a RAID1 from /dev/sda3
and /dev/sdb (instead of /dev/sdb3). Needless to say, the resulting
volume was somewhat unuseable. To say the least.
My server's mdadm.conf has a 'DEVICE=partitions' line. I suppose that
replacing these with a pattern that explicitly only matches partitions,
not disks, would make the problem go away, and that the lesson from
today's disaster recovery effort is to always explicitly list the allowed
partition names, instead of being lazy and using 'DEVICE=partitions'.
--
Specifying partition names may cause problems if the disk names
changed. Specify the UUID instead.
On Tue, Sep 15, 2009 at 10:22 PM, Matthias Urlichs [off-list ref] wrote:
I had a somewhat strange error today.
One of my servers has a RAID1 array. Two partitions at the end of the
disk; the RAID superblocks are at the end of the partition.
After a hard reboot today, one of the disks managed to not have its
partition table scanned correctly, most probably because the disk was
hung and the ("intelligent") controller got confused about it. After the
initial scan, however, it came up correctly.
This error caused mdadm to "successfully" build a RAID1 from /dev/sda3
and /dev/sdb (instead of /dev/sdb3). Needless to say, the resulting
volume was somewhat unuseable. To say the least.
My server's mdadm.conf has a 'DEVICE=partitions' line. I suppose that
replacing these with a pattern that explicitly only matches partitions,
not disks, would make the problem go away, and that the lesson from
today's disaster recovery effort is to always explicitly list the allowed
partition names, instead of being lazy and using 'DEVICE=partitions'.
--
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 2009-09-15 at 22:24 +0300, Majed B. wrote:
Specifying partition names may cause problems if the disk names
changed. Specify the UUID instead.
What I was trying to say is "use a pattern that matches all of your
partitions and none of your actual disks". That should be safe from
changing disk file names.
Besides, specifying UUIDs does not work because it runs into the exact
problem we're trying to solve in the first place. Behold the "blkid"
output from the server in question:
/dev/sdb2: UUID="1583e643-7004-a86e-4eff-81c58ea53278" TYPE="mdraid"
/dev/sdb: UUID="1583e643-7004-a86e-4eff-81c58ea53278" TYPE="mdraid"
Interestingly, for some reason it doesn't do that with /dev/sda -- most
probably because the disks in question use different head/sector values
and thus have slightly different partition layout.
Weird. I haven't seen that before on my disks/partitions.
user@Adam:~$ ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 10 2009-09-14 18:05
cf70cb2b-8e1e-44f3-b361-4a27e169eef2 -> ../../sdi1
lrwxrwxrwx 1 root root 10 2009-09-14 18:05
f54698f9-72a0-4226-9ada-34af2699a25c -> ../../sdi5
My array is offline at the moment, so I can't print the disks' info,
but as you can see, each partition has its own UUID, and if each
partition belonged to a different array then it would have a different
UUID.
Sorry for your loss :/
On Tue, Sep 15, 2009 at 10:33 PM, Matthias Urlichs [off-list ref] wrote:
On Tue, 2009-09-15 at 22:24 +0300, Majed B. wrote:
quoted
Specifying partition names may cause problems if the disk names
changed. Specify the UUID instead.
What I was trying to say is "use a pattern that matches all of your
partitions and none of your actual disks". That should be safe from
changing disk file names.
Besides, specifying UUIDs does not work because it runs into the exact
problem we're trying to solve in the first place. Behold the "blkid"
output from the server in question:
/dev/sdb2: UUID="1583e643-7004-a86e-4eff-81c58ea53278" TYPE="mdraid"
/dev/sdb: UUID="1583e643-7004-a86e-4eff-81c58ea53278" TYPE="mdraid"
Interestingly, for some reason it doesn't do that with /dev/sda -- most
probably because the disks in question use different head/sector values
and thus have slightly different partition layout.
On Sep 15, 2009, at 3:22 PM, Matthias Urlichs wrote:
I had a somewhat strange error today.
One of my servers has a RAID1 array. Two partitions at the end of the
disk; the RAID superblocks are at the end of the partition.
After a hard reboot today, one of the disks managed to not have its
partition table scanned correctly, most probably because the disk was
hung and the ("intelligent") controller got confused about it. After
the
initial scan, however, it came up correctly.
This error caused mdadm to "successfully" build a RAID1 from /dev/sda3
and /dev/sdb (instead of /dev/sdb3). Needless to say, the resulting
volume was somewhat unuseable. To say the least.
My server's mdadm.conf has a 'DEVICE=partitions' line. I suppose that
replacing these with a pattern that explicitly only matches
partitions,
not disks, would make the problem go away, and that the lesson from
today's disaster recovery effort is to always explicitly list the
allowed
partition names, instead of being lazy and using 'DEVICE=partitions'.
Wrong lesson. The correct lesson to gather from this is to prefer
version 1.1 or 1.2 superblocks wherever possible. Superblocks at the
beginning of the device disappear when there is no partition table,
superblocks at the end can be confused for superblocks belonging to
the whole device when there is no partition table.
--
Doug Ledford [off-list ref]
GPG KeyID: CFBFF194
http://people.redhat.com/dledford
InfiniBand Specific RPMS
http://people.redhat.com/dledford/Infiniband
Weird. I haven't seen that before on my disks/partitions.
user@Adam:~$ ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 10 2009-09-14 18:05
cf70cb2b-8e1e-44f3-b361-4a27e169eef2 -> ../../sdi1
lrwxrwxrwx 1 root root 10 2009-09-14 18:05
f54698f9-72a0-4226-9ada-34af2699a25c -> ../../sdi5
My array is offline at the moment, so I can't print the disks' info,
but as you can see, each partition has its own UUID, and if each
partition belonged to a different array then it would have a different
UUID.
As did Matthias' setup. However, if you fail to read the partition
table on a device, *and* the superblock on any given partition is in
the exact same location as it would be if you put a superblock on the
entire device, then mdadm can get confused and think the superblock
belongs to the entire device and not just the partition that the
superblock is at the end of. When that happens, it will think the
data starts at block 0 of the drive, instead of block 0 of the
partition and the data read will be inconsistent. That's what
happened to Matthias.
Sorry for your loss :/
On Tue, Sep 15, 2009 at 10:33 PM, Matthias Urlichs <matthias@urlichs.de
quoted
wrote:
On Tue, 2009-09-15 at 22:24 +0300, Majed B. wrote:
quoted
Specifying partition names may cause problems if the disk names
changed. Specify the UUID instead.
What I was trying to say is "use a pattern that matches all of your
partitions and none of your actual disks". That should be safe from
changing disk file names.
Besides, specifying UUIDs does not work because it runs into the
exact
problem we're trying to solve in the first place. Behold the "blkid"
output from the server in question:
/dev/sdb2: UUID="1583e643-7004-a86e-4eff-81c58ea53278" TYPE="mdraid"
/dev/sdb: UUID="1583e643-7004-a86e-4eff-81c58ea53278" TYPE="mdraid"
Interestingly, for some reason it doesn't do that with /dev/sda --
most
probably because the disks in question use different head/sector
values
and thus have slightly different partition layout.
--
Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-
raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, September 16, 2009 6:00 am, Doug Ledford wrote:
On Sep 15, 2009, at 3:22 PM, Matthias Urlichs wrote:
quoted
I had a somewhat strange error today.
One of my servers has a RAID1 array. Two partitions at the end of the
disk; the RAID superblocks are at the end of the partition.
After a hard reboot today, one of the disks managed to not have its
partition table scanned correctly, most probably because the disk was
hung and the ("intelligent") controller got confused about it. After
the
initial scan, however, it came up correctly.
This error caused mdadm to "successfully" build a RAID1 from /dev/sda3
and /dev/sdb (instead of /dev/sdb3). Needless to say, the resulting
volume was somewhat unuseable. To say the least.
My server's mdadm.conf has a 'DEVICE=partitions' line. I suppose that
replacing these with a pattern that explicitly only matches
partitions,
not disks, would make the problem go away, and that the lesson from
today's disaster recovery effort is to always explicitly list the
allowed
partition names, instead of being lazy and using 'DEVICE=partitions'.
Wrong lesson. The correct lesson to gather from this is to prefer
version 1.1 or 1.2 superblocks wherever possible. Superblocks at the
beginning of the device disappear when there is no partition table,
superblocks at the end can be confused for superblocks belonging to
the whole device when there is no partition table.
1.0 also protects from this problem. The 1.x metadata stores the
offset of the superblock from the start of the device. That will appear
to be wrong if you find a partition-superblock when reading from a
whole-device, so mdadm will reject the device as not having a valid
superblock.
NeilBrown
From: Andre Noll <hidden> Date: 2009-09-17 08:25:28
On 07:01, NeilBrown wrote:
quoted
Wrong lesson. The correct lesson to gather from this is to prefer
version 1.1 or 1.2 superblocks wherever possible. Superblocks at the
beginning of the device disappear when there is no partition table,
superblocks at the end can be confused for superblocks belonging to
the whole device when there is no partition table.
1.0 also protects from this problem. The 1.x metadata stores the
offset of the superblock from the start of the device. That will appear
to be wrong if you find a partition-superblock when reading from a
whole-device, so mdadm will reject the device as not having a valid
superblock.
BTW: Why are new arrays still created with 0.90 metadata format by
default?
Andre
--
The only person who always got his work done by Friday was Robinson Crusoe
0.9 shouldn't be the default if 1.0 or 1.x is better anyway :/
On Thu, Sep 17, 2009 at 8:09 PM, Matthias Urlichs [off-list ref] wrote:
On Thu, 2009-09-17 at 12:44 -0400, Doug Ledford wrote:
quoted
quoted
quoted
1.0 also protects from this problem.
So what's the best way to update a 0.9 superblock to 1.0 format?
mdadm --assemble --metadata=1.0 --update=summaries ... ?
I don't think you can. You just have to create a new array with the
new superblock.
Hmm. Looks like you're right.
Consider this to be a feature request, then. If 1.0 is more reasonable
than 0.9, there should be a "safe" upgrade path.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html