Hi,
I just set up a new server running stable Debian Sarge, fresh install,
mdadm 1.9.0, linux kernel 2.6.8-2
and all my (test) data has vanished into the ether.
What did I do wrong? I don't want this to happen in real life :).
I have 6x 400Gb Western Digital sata drives attached to sata controllers. One
pair of channels is on the motherboard using VIA VT8237 controller
and 2 highpoint 1520 sata cards give 4 other channels.
/dev/sda
/dev/sdb
/dev/sdc
/dev/sdd
/dev/sde
/dev/sdf
I partioned each hard drive completely as 1 partition, type fd
ie fdisk then "n p 1 enter t fd w q":
/dev/sda1
/dev/sdb1
/dev/sdc1
/dev/sdd1
/dev/sde1
/dev/sdf1
I then created 3 raid1 arrays
mdadm -Cv -n2 -l1 /dev/md0 /dev/sda1 /dev/sdb1
mdadm -Cv -n2 -l1 /dev/md1 /dev/sdc1 /dev/sdd1
mdadm -Cv -n2 -l1 /dev/md2 /dev/sde1 /dev/sdf1
I then formated
mkfs.ext3 /dev/md0
mkfs.ext3 /dev/md1
mkfs.ext3 /dev/md2
I then mounted
mount /dev/md0 /home/big0
mount /dev/md1 /home/big1
mount /dev/md2 /home/big2
I then waited until all syncing was done.
Then I copied 3 gb of data to the 3 arrays from another machine over the
network via rsync.
I did df -h and indeed the data was there. I cd into the directories and there
was data there.
Then I did
shutdown -h now
however I did not!! (?evil) umount the different devices
/dev/md0 /dev/md1 /dev/md2 before doing that.
Now I reboot into linux.
I try to assemble the arrays
mdadm -A /dev/md0 /dev/sda1 /dev/sdb1
mdadm --detail --scan gives me nothing at all.
And I get nothing.
I do fdisk -l
and no trace of any of the partitions. It is as if there are no
partitions /dev/sdX1, where X is a-f.
I do
fdisk /dev/sda
I get message:
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF
disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
The number of cylinders for this disk is set to 48641.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
p shows nothing!!
??????????????????????????????
Did I screw up everything by not unmounting the devices? What happened?
Thanks,
Mitchell Laks
From: Mike Hardy <hidden> Date: 2005-12-28 04:33:03
Mitchell Laks wrote:
Hi,
I just set up a new server running stable Debian Sarge, fresh install,
mdadm 1.9.0, linux kernel 2.6.8-2
and all my (test) data has vanished into the ether.
What did I do wrong? I don't want this to happen in real life :).
[ description of sane-looking md setup snipped ]
Did I screw up everything by not unmounting the devices? What happened?
Failure to unmount the devices should have resulted in possibly last
writes but a journalling filesystem would have kept things consistent.
Failure to nicely stop the arrays also could have possibly degraged the
arrays if they were being written to as the machine went down, but they
should be there.
In short: it *definitely* should *not* have come back with what you saw.
I have no clue why this happened, all I can say is that I use md on
around 10 business critical production machines in exactly the way you
describe, and I have not seen this (though I've seen the failure modes I
described!).
Are you positively sure that nothing else weird could have been going
on? Some layer that remaps drive names? funky hardware? write caching
capable of holding 3GB before writing out? Write caching of some sort is
actually my best guess.
If there's no extra data to explain this, I'm at a loss.
Anyone else?
-Mike
From: Ross Vandegrift <hidden> Date: 2005-12-28 04:50:21
On Tue, Dec 27, 2005 at 11:40:48PM -0500, Mitchell Laks wrote:
I then created 3 raid1 arrays
mdadm -Cv -n2 -l1 /dev/md0 /dev/sda1 /dev/sdb1
mdadm -Cv -n2 -l1 /dev/md1 /dev/sdc1 /dev/sdd1
mdadm -Cv -n2 -l1 /dev/md2 /dev/sde1 /dev/sdf1
Are you 100% sure that you didn't do:
mdadm -Cv -n2 -l1 /dev/md0 /dev/sda /dev/sdb, etc, etc?
(ie, note lack of subdevice for partition!!!)
That's the only thing I can image that would cause this:
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF
disklabel
It sounds like you created md devices out of whole disks instead of
partitions and overwrote the partition information. I don't *think*
this should be a problem, but I don't 100% know...
Try this to see if it finds your array:
mdadm --assemble /dev/md0 /dev/sda /dev/sdb
however I did not!! (?evil) umount the different devices
/dev/md0 /dev/md1 /dev/md2 before doing that.
When you shutdown cleanly, your system's scripts almost certainly did
that for you! ::-)
--
Ross Vandegrift
ross@lug.udel.edu
"The good Christian should beware of mathematicians, and all those who
make empty prophecies. The danger already exists that the mathematicians
have made a covenant with the devil to darken the spirit and to confine
man in the bonds of Hell."
--St. Augustine, De Genesi ad Litteram, Book II, xviii, 37
In short: it *definitely* should *not* have come back with what you saw.
I have no clue why this happened, all I can say is that I use md on
around 10 business critical production machines in exactly the way you
describe, and I have not seen this (though I've seen the failure modes I
described!).
I myself have been using md, mdadm for more than a year in multiple machines.
I am at a loss. The only difference is now SATA drives with SATA controllers.
Are you positively sure that nothing else weird could have been going
on? Some layer that remaps drive names? funky hardware? write caching
capable of holding 3GB before writing out? Write caching of some sort is
actually my best guess.
9.8Gb actually! same 3.2 Gb sent to each of the 3 raid 1 drive arrays to
really give them a workout, sent with rsync.
Only kicker is that this setup is Sata disks using the kernel sata drivers as
modules. I have always used Pata drives before.
I get no kernel error messages at all.
dmesg from the reboot tells me
First for the modules that come with linux: and the devices
libata version 1.02 loaded.
sata_via version 0.20
ACPI: PCI interrupt 0000:00:0f.0[B] -> GSI 20 (level, low) -> IRQ 185
sata_via(0000:00:0f.0): routed to hard irq line 10
ata1: SATA max UDMA/133 cmd 0xC000 ctl 0xB802 bmdma 0xA800 irq 185
ata2: SATA max UDMA/133 cmd 0xB400 ctl 0xB002 bmdma 0xA808 irq 185
ata1: dev 0 cfg 49:2f00 82:746b 83:7f01 84:4023 85:7469 86:3c01 87:4023
88:407f
ata1: dev 0 ATA, max UDMA/133, 781422768 sectors: lba48
ata1: dev 0 configured for UDMA/133
scsi1 : sata_via
ata2: dev 0 cfg 49:2f00 82:746b 83:7f01 84:4023 85:7469 86:3c01 87:4023
88:407f
ata2: dev 0 ATA, max UDMA/133, 781422768 sectors: lba48
ata2: dev 0 configured for UDMA/133
scsi2 : sata_via
Vendor: ATA Model: WDC WD4000YR-01P Rev: 01.0
Type: Direct-Access ANSI SCSI revision: 05
SCSI device sde: 781422768 512-byte hdwr sectors (400088 MB)
SCSI device sde: drive cache: write back
/dev/scsi/host1/bus0/target0/lun0: unknown partition table
Attached scsi disk sde at scsi1, channel 0, id 0, lun 0
Vendor: ATA Model: WDC WD4000YR-01P Rev: 01.0
Type: Direct-Access ANSI SCSI revision: 05
SCSI device sdf: 781422768 512-byte hdwr sectors (400088 MB)
SCSI device sdf: drive cache: write back
/dev/scsi/host2/bus0/target0/lun0: unknown partition table
Attached scsi disk sdf at scsi2, channel 0, id 0, lun 0
Notice the unknown partition table:
Then for the data from dmesg about the devices mounted on the highpoint
rocketraid 1520 which used the proprietary hpt37x2.ko kernel module that I
added to initrd and /lib/modules/`uname -r`/kernel/drivers/ide or whatever
hpt37x2: no version for "scsi_remove_host" found: kernel tainted.
HPT37x2 RAID Controller driver
SCSI device sda: 781422767 512-byte hdwr sectors (400088 MB)
sda: asking for cache data failed
sda: assuming drive cache: write through
/dev/scsi/host0/bus0/target0/lun0: p1
SCSI device sda: 781422767 512-byte hdwr sectors (400088 MB)
sda: asking for cache data failed
sda: assuming drive cache: write through
/dev/scsi/host0/bus0/target0/lun0: p1
SCSI device sdb: 781422767 512-byte hdwr sectors (400088 MB)
sdb: asking for cache data failed
sdb: assuming drive cache: write through
/dev/scsi/host0/bus0/target1/lun0: p1
SCSI device sdb: 781422767 512-byte hdwr sectors (400088 MB)
sdb: asking for cache data failed
sdb: assuming drive cache: write through
/dev/scsi/host0/bus0/target1/lun0: p1
SCSI device sdc: 781422767 512-byte hdwr sectors (400088 MB)
sdc: asking for cache data failed
sdc: assuming drive cache: write through
/dev/scsi/host0/bus0/target2/lun0: p1
SCSI device sdc: 781422767 512-byte hdwr sectors (400088 MB)
sdc: asking for cache data failed
sdc: assuming drive cache: write through
/dev/scsi/host0/bus0/target2/lun0: p1
SCSI device sdd: 781422767 512-byte hdwr sectors (400088 MB)
sdd: asking for cache data failed
sdd: assuming drive cache: write through
/dev/scsi/host0/bus0/target3/lun0: p1
SCSI device sdd: 781422767 512-byte hdwr sectors (400088 MB)
sdd: asking for cache data failed
sdd: assuming drive cache: write through
/dev/scsi/host0/bus0/target3/lun0: p1
If there's no extra data to explain this, I'm at a loss.
I wish I knew what else to check. I just bought 16 SATA drives for 2 servers!
Anyone else?
-Mike
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html