Re: Data Offset

Swapped out the new drive for the old. The new drive is still labelled
as /dev/sdf, hopefully.. I decided to check times before proceeding,
and to make sure the drives were in the right order. I corrected them
to go by `mdadm --examine`'s output as best I could.

Here's the output of `mdadm --examine /dev/sdf` and the result of
executing the given `./mdadm` command (with re-ordered drives), The
binary compiled from git sources crashed with a segmentation fault
while attempting to print out a failure writing the superblock. I've
tried the drives (with proper sizes) in other combinations, according
to both what you posted and what mdadm --examine says the "Device
Role" is. I haven't found a working combination; is it possible my
drives got swapped around on reboot? There's a re-run of mdadm
--examine at the end of my post.

root@leyline:~/mdadm# mdadm --examine /dev/sdf
/dev/sdf:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 9759ad94:75e30b6b:8a726b4d:177a6eda
           Name : leyline:1  (local to host leyline)
  Creation Time : Mon Sep 12 13:19:00 2011
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 3907027120 (1863.02 GiB 2000.40 GB)
     Array Size : 7813048320 (7451.10 GiB 8000.56 GB)
  Used Dev Size : 3906524160 (1862.78 GiB 2000.14 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : 2edc16c6:cf45ad32:04b026a4:956ce78b

    Update Time : Fri Jun  1 03:11:54 2012
       Checksum : b3e49e59 - correct
         Events : 2127454

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAA. ('A' == active, '.' == missing)
root@leyline:~/mdadm# ./mdadm -C /dev/md1 -e 1.2 -l 5 -n 5
--assume-clean -c 512 /dev/sdc3:2048s /dev/sdf:2048s /dev/sdb3:2048s
/dev/sdd3:2048s /dev/sde3:1024s
mdadm: /dev/sdc3 appears to be part of a raid array:
    level=raid5 devices=5 ctime=Tue Jun  5 00:10:46 2012
mdadm: /dev/sdf appears to contain an ext2fs file system
    size=242788K  mtime=Fri Oct  7 16:55:40 2011
mdadm: /dev/sdf appears to be part of a raid array:
    level=raid5 devices=5 ctime=Mon Sep 12 13:19:00 2011
mdadm: /dev/sdb3 appears to be part of a raid array:
    level=raid5 devices=5 ctime=Tue Jun  5 00:10:46 2012
mdadm: /dev/sdd3 appears to be part of a raid array:
    level=raid5 devices=5 ctime=Tue Jun  5 00:10:46 2012
Continue creating array? yes
Segmentation fault

Since I couldn't find any fault with running it again (but I am not a
smart man, or I would not be in this position), I decided to run
valgrind over it:
==3206== Memcheck, a memory error detector
==3206== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
==3206== Using Valgrind-3.6.0.SVN-Debian and LibVEX; rerun with -h for
copyright info
==3206== Command: ./mdadm -C /dev/md1 -e 1.2 -l 5 -n 5 --assume-clean
-c 512 /dev/sdc3:2048s /dev/sdf:2048s /dev/sdb3:2048s /dev/sdd3:2048s
/dev/sde3:1024s
==3206==
==3206== Warning: noted but unhandled ioctl 0x1261 with no size/direction hints
==3206==    This could cause spurious value errors to appear.
==3206==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on
writing a proper wrapper.
==3206== Warning: noted but unhandled ioctl 0x1261 with no size/direction hints
==3206==    This could cause spurious value errors to appear.
==3206==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on
writing a proper wrapper.
==3206== Warning: noted but unhandled ioctl 0x1261 with no size/direction hints
==3206==    This could cause spurious value errors to appear.
==3206==    See README_MISSING_SYSCALL_OR_IOCTL for guidance on
writing a proper wrapper.
mdadm: /dev/sdc3 appears to be part of a raid array:
    level=raid5 devices=5 ctime=Tue Jun  5 00:14:20 2012
mdadm: /dev/sdf appears to contain an ext2fs file system
    size=242788K  mtime=Fri Oct  7 16:55:40 2011
mdadm: /dev/sdf appears to be part of a raid array:
    level=raid5 devices=5 ctime=Tue Jun  5 00:14:20 2012
mdadm: /dev/sdb3 appears to be part of a raid array:
    level=raid5 devices=5 ctime=Tue Jun  5 00:14:20 2012
mdadm: /dev/sdd3 appears to be part of a raid array:
    level=raid5 devices=5 ctime=Tue Jun  5 00:14:20 2012
Continue creating array? ==3206== Invalid read of size 8
==3206==    at 0x43C9B7: write_init_super1 (super1.c:1327)
==3206==    by 0x41F1B9: Create (Create.c:951)
==3206==    by 0x407231: main (mdadm.c:1464)
==3206==  Address 0x8 is not stack'd, malloc'd or (recently) free'd
==3206==
==3206==
==3206== Process terminating with default action of signal 11 (SIGSEGV)
==3206==  Access not within mapped region at address 0x8
==3206==    at 0x43C9B7: write_init_super1 (super1.c:1327)
==3206==    by 0x41F1B9: Create (Create.c:951)
==3206==    by 0x407231: main (mdadm.c:1464)
==3206==  If you believe this happened as a result of a stack
==3206==  overflow in your program's main thread (unlikely but
==3206==  possible), you can try to increase the size of the
==3206==  main thread stack using the --main-stacksize= flag.
==3206==  The main thread stack size used in this run was 8388608.
==3206==
==3206== HEAP SUMMARY:
==3206==     in use at exit: 37,033 bytes in 350 blocks
==3206==   total heap usage: 673 allocs, 323 frees, 4,735,171 bytes allocated
==3206==
==3206== LEAK SUMMARY:
==3206==    definitely lost: 832 bytes in 8 blocks
==3206==    indirectly lost: 18,464 bytes in 4 blocks
==3206==      possibly lost: 0 bytes in 0 blocks
==3206==    still reachable: 17,737 bytes in 338 blocks
==3206==         suppressed: 0 bytes in 0 blocks
==3206== Rerun with --leak-check=full to see details of leaked memory
==3206==
==3206== For counts of detected and suppressed errors, rerun with: -v
==3206== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 4 from 4)

mdadm --examine of all my drives (again):
/dev/sdb3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 9ed0b17f:e9a7a813:a9139679:4f8f999b
           Name : leyline:1  (local to host leyline)
  Creation Time : Tue Jun  5 00:14:35 2012
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 3906525098 (1862.78 GiB 2000.14 GB)
     Array Size : 7813046272 (7451.10 GiB 8000.56 GB)
  Used Dev Size : 3906523136 (1862.78 GiB 2000.14 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : feb94069:b2afeb6e:ae6b2af2:f9e3cee4

    Update Time : Tue Jun  5 00:14:35 2012
       Checksum : 1909d79e - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAAA ('A' == active, '.' == missing)
/dev/sdc3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 9ed0b17f:e9a7a813:a9139679:4f8f999b
           Name : leyline:1  (local to host leyline)
  Creation Time : Tue Jun  5 00:14:35 2012
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 3906525098 (1862.78 GiB 2000.14 GB)
     Array Size : 7813046272 (7451.10 GiB 8000.56 GB)
  Used Dev Size : 3906523136 (1862.78 GiB 2000.14 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : ed6116ac:4f91c2dd:4ada53df:0e14fc2a

    Update Time : Tue Jun  5 00:14:35 2012
       Checksum : fe0cffd8 - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAAA ('A' == active, '.' == missing)
/dev/sdd3:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 9ed0b17f:e9a7a813:a9139679:4f8f999b
           Name : leyline:1  (local to host leyline)
  Creation Time : Tue Jun  5 00:14:35 2012
     Raid Level : raid5
   Raid Devices : 5

 Avail Dev Size : 3906523136 (1862.78 GiB 2000.14 GB)
     Array Size : 7813046272 (7451.10 GiB 8000.56 GB)
    Data Offset : 2048 sectors
   Super Offset : 8 sectors
          State : clean
    Device UUID : d839ca02:1d14cde3:65b54275:8caa0275

    Update Time : Tue Jun  5 00:14:35 2012
       Checksum : 3ac0c483 - correct
         Events : 0

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAAA ('A' == active, '.' == missing)
mdadm: No md superblock detected on /dev/sde3.



On Mon, Jun 4, 2012 at 5:57 PM, NeilBrown [off-list ref] wrote:

On Mon, 04 Jun 2012 20:26:05 +0200 Pierre Beck [off-list ref] wrote:

quoted

I'll try and clear up some confusion (I was in IRC with freeone3000).

/dev/sdf is an empty drive, a replacement for a failed drive. The Array
attempted to assemble, but failed and reported one drive as spare. This
is the moment we saved the --examines.

In expectation of a lost write due to drive write-cache, we executed
--assemble --force, which kicked another drive.

@James: remove /dev/sdf for now and replace /dev/sde3, which indeed has
a very outdated update time, with the non-present drive. Post an
--examine of that drive. It should report update time Jun 1st.

We tried to re-create the array with --assume-clean. But mdadm chose a
different data offset for the drives. A re-create with proper data
offset will be necessary.

OK, try:

  git clone -b data_offset git://neil.brown.name/mdadm
  cd mdadm
  make

  ./mdadm -C /dev/md1 -e 1.2 -l 5 -n 5 --assume-clean -c 512 \
     /dev/sdc3:2048s /dev/sdb3:2048s ???  /dev/sdd3:1024s ???

The number after ':' after a device name is a data offset.  's' means sectors.
With out 's' it means Kilobytes.
I don't know what should be at slot 2 or 4 so I put '???'.  You should fill it
in.  You should also double check the command and double check the names of
your devices.
Don't install this mdadm, and don't use it for anything other than
re-creating this array.

Good luck.

NeilBrown

quoted

Greetings,

Pierre Beck


Am 04.06.2012 05:35, schrieb NeilBrown:

quoted

On Fri, 1 Jun 2012 19:48:41 -0500 freeone3000[off-list ref]  wrote:

quoted

Sorry.

/dev/sde fell out of the array, so I replaced the physical drive with
what is now /dev/sdf. udev may have relabelled the drive - smartctl
states that the drive that is now /dev/sde works fine.
/dev/sdf is a new drive. /dev/sdf has a single, whole-disk partition
with type marked as raid. It is physically larger than the others.

/dev/sdf1 doesn't have a mdadm superblock. /dev/sdf seems to, so I
gave output of that device instead of /dev/sdf1, despite the
partition. Whole-drive RAID is fine, if it gets it working.

What I'm attempting to do is rebuild the RAID from the data from the
other four drives, and bring the RAID back up without losing any of
the data. /dev/sdb3, /dev/sdc3, /dev/sdd3, and what is now /dev/sde3
should be used to rebuild the array, with /dev/sdf as a new drive. If
I can get the array back up with all my data and all five drives in
use, I'll be very happy.

You appear to have 3 devices that are happy:
   sdc3 is device 0   data-offset 2048
   sdb3 is device 1   data-offset 2048
   sdd3 is device 3   data-offset 1024

nothing claims to be device 2 or 4.

sde3 looks like it was last in the array on 23rd May, a little over
a week before your report.  Could that have been when "sde fell out of the
array" ??
Is it possible that you replaced the wrong device?
Or is it possible the the array was degraded when sde "fell out" resulting
in data loss?

I need more precise history to understand what happened, as I cannot suggest
a fixed until I have that understanding.

When did the array fail?
How certain are you that you replaced the correct device?
Can you examine the drive that you removed and see what it says?
Are you certain that the array wasn't already degraded?

NeilBrown

-- 
James Moore
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html