On Sat, Jul 21, 2007 at 07:54:14PM +0200, Rene Herman wrote:
quoted
sfdisk -d already works most of the time. Not as a verbatim tool (I
actually semi-frequently use a "sfdisk -d /dev/hda | sfdisk" invocation
as a way to _rewrite_ the CHS fields to other values after changing
machines around on a disk) but something you'd backup on the FS level
should, in my opinion, need to be less fragile than would be possible
with just 512 bytes available.
*IF* you remember to store the sfdisk -d somewhere useful. In my "How
To Recover From Hard Drive Catastrophies" classes, I tell them to
print out a copy of "sfdisk -l /dev/hda ; sfdisk -d /dev/hda" and tape
it to the side of the computer. I also tell them do regular backups.
What to make a guess how many them actually follow this good advice?
Far fewer than I would like, I suspect...
What I'm suggesting is the equivalent of sfdisk -d, except we'd be
doing it automatically without requiring the user to take any kind of
explicit action. Is it perfect? No, although the edge conditions are
quite rare these days and generally involve users using legacy systems
and/or doing Weird Shit such that They Really Should Know To Do Their
Own Explicit Backups. But for the novice users, it should work Just
Fine.
Sounds great, but it may be advisable to hook this into the partition
modification routines instead of mkfs/fsck. Which would mean that the
partition manager could ask the kernel to instruct its fs subsystem to
update the backup partition table for each known fs-type that supports such
a feature.
Thanks!
--
Al
On Sun, Jul 22, 2007 at 07:10:31AM +0300, Al Boldi wrote:
Sounds great, but it may be advisable to hook this into the partition
modification routines instead of mkfs/fsck. Which would mean that the
partition manager could ask the kernel to instruct its fs subsystem to
update the backup partition table for each known fs-type that supports such
a feature.
Well, let's think about this a bit. What are the requirements?
1) The partition manager should be able explicitly request that a new
backup of the partition tables be stashed in each filesystem that has
room for such a backup. That way, when the user affirmatively makes a
partition table change, it can get backed up in all of the right
places automatically.
2) The fsck program should *only* stash a backup of the partition
table if there currently isn't one in the filesystem. It may be that
the partition table has been corrupted, and so merely doing an fsck
should not transfer a current copy of the partition table to the
filesystem-secpfic backup area. It could be that the partition table
was only partially recovered, and we don't want to overwrite the
previously existing backups except on an explicit request from the
system administrator.
3) The mkfs program should automatically create a backup of the
current partition table layout. That way we get a backup in the newly
created filesystem as soon as it is created.
4) The exact location of the backup may vary from filesystem to
filesystem. For ext2/3/4, bytes 512-1023 are always unused, and don't
interfere with the boot sector at bytes 0-511, so that's the obvious
location. Other filesystems may have that location in use, and some
other location might be a better place to store it. Ideally it will
be a well-known location, that isn't dependent on finding an inode
table, or some such, but that may not be possible for all filesystems.
OK, so how about this as a solution that meets the above requirements?
/sbin/partbackup <device> [<fspart>]
Will scan <device> (i.e., /dev/hda, /dev/sdb, etc.) and create
a 512 byte partition backup, using the format I've previously
described. If <fspart> is specified on the command line, it
will use the blkid library to determine the filesystem type of
<fspart>, and then attempt to execute
/dev/partbackupfs.<fstype> to write the partition backup to
<fspart>. If <fspart> is '-', then it will write the 512 byte
partition table to stdout. If <fspart> is not specified on
the command line, /sbin/partbackup will iterate over all
partitions in <device>, use the blkid library to attempt to
determine the correct filesystem type, and then execute
/sbin/partbackupfs.<fstype> if such a backup program exists.
/sbin/partbackupfs.<fstype> <fspart>
... is a filesystem specific program for filesystem type
<fstype>. It will assure that <fspart> (i.e., /dev/hda1,
/dev/sdb3) is of an appropriate filesystem type, and then read
512 bytes from stdin and write it out to <fspart> to an
appropriate place for that filesystem.
Partition managers will be encouraged to check to see if
/sbin/partbackup exists, and if so, after the partition table is
written, will check to see if /sbin/partbackup exists, and if so, to
call it with just one argument (i.e., /sbin/partbackup /dev/hdb).
They SHOULD provide an option for the user to suppress the backup from
happening, but the backup should be the default behavior.
An /etc/mkfs.<fstype> program is encouraged to run /sbin/partbackup
with two arguments (i.e., /sbin/partbackup /dev/hdb /dev/hdb3) when
creating a filesystem.
An /etc/fsck.<fstype> program is encouraged to check to see if a
partition backup exists (assuming the filesystem supports it), and if
not, call /sbin/partbackup with two arguments.
A filesystem utility package for a particular filesystem type is
encouraged to make the above changes to its mkfs and fsck programs, as
well as provide an /sbin/partbackupfs.<fstype> program.
I would do this all in userspace, though. Is there any reason to get
the kernel involved? I don't think so.
- Ted
On Sun, Jul 22, 2007 at 07:10:31AM +0300, Al Boldi wrote:
quoted
Sounds great, but it may be advisable to hook this into the partition
modification routines instead of mkfs/fsck. Which would mean that the
partition manager could ask the kernel to instruct its fs subsystem to
update the backup partition table for each known fs-type that supports
such a feature.
Well, let's think about this a bit. What are the requirements?
1) The partition manager should be able explicitly request that a new
backup of the partition tables be stashed in each filesystem that has
room for such a backup. That way, when the user affirmatively makes a
partition table change, it can get backed up in all of the right
places automatically.
2) The fsck program should *only* stash a backup of the partition
table if there currently isn't one in the filesystem. It may be that
the partition table has been corrupted, and so merely doing an fsck
should not transfer a current copy of the partition table to the
filesystem-secpfic backup area. It could be that the partition table
was only partially recovered, and we don't want to overwrite the
previously existing backups except on an explicit request from the
system administrator.
3) The mkfs program should automatically create a backup of the
current partition table layout. That way we get a backup in the newly
created filesystem as soon as it is created.
4) The exact location of the backup may vary from filesystem to
filesystem. For ext2/3/4, bytes 512-1023 are always unused, and don't
interfere with the boot sector at bytes 0-511, so that's the obvious
location. Other filesystems may have that location in use, and some
other location might be a better place to store it. Ideally it will
be a well-known location, that isn't dependent on finding an inode
table, or some such, but that may not be possible for all filesystems.
OK, so how about this as a solution that meets the above requirements?
/sbin/partbackup <device> [<fspart>]
Will scan <device> (i.e., /dev/hda, /dev/sdb, etc.) and create
a 512 byte partition backup, using the format I've previously
described. If <fspart> is specified on the command line, it
will use the blkid library to determine the filesystem type of
<fspart>, and then attempt to execute
/dev/partbackupfs.<fstype> to write the partition backup to
<fspart>. If <fspart> is '-', then it will write the 512 byte
partition table to stdout. If <fspart> is not specified on
the command line, /sbin/partbackup will iterate over all
partitions in <device>, use the blkid library to attempt to
determine the correct filesystem type, and then execute
/sbin/partbackupfs.<fstype> if such a backup program exists.
/sbin/partbackupfs.<fstype> <fspart>
... is a filesystem specific program for filesystem type
<fstype>. It will assure that <fspart> (i.e., /dev/hda1,
/dev/sdb3) is of an appropriate filesystem type, and then read
512 bytes from stdin and write it out to <fspart> to an
appropriate place for that filesystem.
Partition managers will be encouraged to check to see if
/sbin/partbackup exists, and if so, after the partition table is
written, will check to see if /sbin/partbackup exists, and if so, to
call it with just one argument (i.e., /sbin/partbackup /dev/hdb).
They SHOULD provide an option for the user to suppress the backup from
happening, but the backup should be the default behavior.
An /etc/mkfs.<fstype> program is encouraged to run /sbin/partbackup
with two arguments (i.e., /sbin/partbackup /dev/hdb /dev/hdb3) when
creating a filesystem.
An /etc/fsck.<fstype> program is encouraged to check to see if a
partition backup exists (assuming the filesystem supports it), and if
not, call /sbin/partbackup with two arguments.
A filesystem utility package for a particular filesystem type is
encouraged to make the above changes to its mkfs and fsck programs, as
well as provide an /sbin/partbackupfs.<fstype> program.
Great!
I would do this all in userspace, though. Is there any reason to get
the kernel involved? I don't think so.
Yes, doing things in userspace, when possible, is much better. But, a change
in the partition table has to be relayed to the kernel, and when that change
happens to be on a mounted disk, then the partition manager complains of not
being able to update the kernel's view. So how can this be addressed?
Thanks!
--
Al
On Sun, Jul 22, 2007 at 07:10:31AM +0300, Al Boldi wrote:
quoted
Sounds great, but it may be advisable to hook this into the partition
modification routines instead of mkfs/fsck. Which would mean that the
partition manager could ask the kernel to instruct its fs subsystem to
update the backup partition table for each known fs-type that supports such
a feature.
Well, let's think about this a bit. What are the requirements?
1) The partition manager should be able explicitly request that a new
backup of the partition tables be stashed in each filesystem that has
room for such a backup. That way, when the user affirmatively makes a
partition table change, it can get backed up in all of the right
places automatically.
2) The fsck program should *only* stash a backup of the partition
table if there currently isn't one in the filesystem. It may be that
the partition table has been corrupted, and so merely doing an fsck
should not transfer a current copy of the partition table to the
filesystem-secpfic backup area. It could be that the partition table
was only partially recovered, and we don't want to overwrite the
previously existing backups except on an explicit request from the
system administrator.
3) The mkfs program should automatically create a backup of the
current partition table layout. That way we get a backup in the newly
created filesystem as soon as it is created.
4) The exact location of the backup may vary from filesystem to
filesystem. For ext2/3/4, bytes 512-1023 are always unused, and don't
interfere with the boot sector at bytes 0-511, so that's the obvious
location. Other filesystems may have that location in use, and some
other location might be a better place to store it. Ideally it will
be a well-known location, that isn't dependent on finding an inode
table, or some such, but that may not be possible for all filesystems.
To be on the safe side, maybe also add a checksum, timestamp and
something identifying the disk the filesystem was created on.
Regards,
Indan
From: Rene Herman <hidden> Date: 2007-07-23 08:15:21
On 07/22/2007 06:28 PM, Theodore Tso wrote:
[ Al -- don't drop CCs please ]
Well, let's think about this a bit. What are the requirements?
1) The partition manager should be able explicitly request that a new
backup of the partition tables be stashed in each filesystem that has
room for such a backup. That way, when the user affirmatively makes a
partition table change, it can get backed up in all of the right
places automatically.
D-Bus? ;-)
2) The fsck program should *only* stash a backup of the partition
table if there currently isn't one in the filesystem. It may be that
the partition table has been corrupted, and so merely doing an fsck
should not transfer a current copy of the partition table to the
filesystem-secpfic backup area. It could be that the partition table
was only partially recovered, and we don't want to overwrite the
previously existing backups except on an explicit request from the
system administrator.
3) The mkfs program should automatically create a backup of the
current partition table layout. That way we get a backup in the newly
created filesystem as soon as it is created.
On an integrated system like this, do you consider it acceptable to only do
the MS-DOS partitions and not the other types that may be present _inside_
those partitions? (MINIX subpartitions, BSD slices, ...). I believe those
should really also be done, but this would require keeping more information
again.
4) The exact location of the backup may vary from filesystem to
filesystem. For ext2/3/4, bytes 512-1023 are always unused, and don't
interfere with the boot sector at bytes 0-511, so that's the obvious
location. Other filesystems may have that location in use, and some
other location might be a better place to store it. Ideally it will
be a well-known location, that isn't dependent on finding an inode
table, or some such, but that may not be possible for all filesystems.
OK, so how about this as a solution that meets the above requirements?
/sbin/partbackup <device> [<fspart>]
Will scan <device> (i.e., /dev/hda, /dev/sdb, etc.) and create
a 512 byte partition backup, using the format I've previously
described. If <fspart> is specified on the command line, it
will use the blkid library to determine the filesystem type of
<fspart>, and then attempt to execute
/dev/partbackupfs.<fstype> to write the partition backup to
<fspart>. If <fspart> is '-', then it will write the 512 byte
partition table to stdout. If <fspart> is not specified on
the command line, /sbin/partbackup will iterate over all
partitions in <device>, use the blkid library to attempt to
determine the correct filesystem type, and then execute
/sbin/partbackupfs.<fstype> if such a backup program exists.
I've cleaned up what I posted yesterday a bit and made it into the type of
standalone-by-design program you suggest here (the version from yesterday
required a -DTEST to be so). Not with the blkid bits though. Just dumps the
sector to stdout (and a textual version to stderr if compiled with -DDEBUG).
I (very) briefly looked at blkid but unless I'm mistaken blkid needs device
names? The documentation seems to be missing. When scanning the device for
the partition table, we've built a list of partitions with offsets into the
device and it would be nice if we could hand the fd and the offset off to
something directly. If the program has to construct device names itself
there's another truckload of pitfalls right there.
It wouldn't be hard to log minors as such, but you'd also need to be very
sure you'd always do this in the same order as the kernel so that what I
consider to be "/dev/sda2" is the same the kernel considers it to be. This
is again rather fragile.
It might in fact make sense to just ask the kernel for the partitions on a
device and not bother with scanning anything ourselves. Ie, just walk sysfs.
Would you agree? This siginificantly reduces the risk of things getting out
of sync, both scanning order and implementation.
The kernel doesn't currently store/export everything you'd want to store in
a backup (as far as I'm aware, that is) but that could conceivably change.
It would make things significantly less fragile.
Rene.
} __attribute__((packed));
enum {
ENTRY_FLAG_PRIMARY = 0x01,
ENTRY_FLAG_BOOTABLE = 0x80,
};
struct backup {
uint8_t signature[8];
uint16_t type;
uint8_t heads;
uint8_t sectors;
uint8_t count;
uint8_t __1[3];
struct entry table[31];
} __attribute__((packed));
#define BACKUP_SIGNATURE "PARTBAK1"
enum {
BACKUP_TYPE_MBR = 1,
};
struct backup backup = {
.signature = BACKUP_SIGNATURE,
.type = BACKUP_TYPE_MBR,
};
#define ARRAY_SIZE(arr) (sizeof arr / sizeof arr[0])
int is_extended(struct partition *partition)
{
int ret = 0;
switch (partition->sys_ind) {
case DOS_EXTENDED:
case WIN98_EXTENDED:
case LINUX_EXTENDED:
ret = 1;
}
return ret;
}
unsigned char *get_sector(int fd, uint64_t n)
{
unsigned char *sector;
if (lseek64(fd, n << 9, SEEK_SET) < 0) {
perror("lseek64");
return NULL;
}
sector = malloc(512);
if (!sector) {
fprintf(stderr, "malloc: out of memory\n");
return NULL;
}
if (read(fd, sector, 512) != 512) {
perror("read");
free(sector);
return NULL;
}
return sector;
}
void put_sector(unsigned char *sector)
{
free(sector);
}
#define TABLE_OFFSET (512 - 2 - 4 * sizeof(struct partition))
inline struct partition *table(unsigned char *sector)
{
return (struct partition *)(sector + TABLE_OFFSET);
}
int do_sector(int fd, uint32_t offset, uint32_t start)
{
unsigned char *sector;
struct partition *cur;
struct partition *ext = NULL;
int ret = 0;
sector = get_sector(fd, offset + start);
if (!sector)
return -1;
if (sector[510] != 0x55 || sector[511] != 0xaa) {
ret = -1;
goto out;
}
for (cur = table(sector); cur < table(sector) + 4; cur++) {
struct entry *entry;
if (!cur->size)
continue;
cur->end_head += 1;
if (backup.heads < cur->end_head)
backup.heads = cur->end_head;
cur->end_sector &= 0x3f;
if (backup.sectors < cur->end_sector)
backup.sectors = cur->end_sector;
if (is_extended(cur)) {
if (!offset) {
ret = do_sector(fd, cur->start, 0);
if (ret < 0)
goto out;
} else if (!ext)
ext = cur;
continue;
}
if (backup.count == ARRAY_SIZE(backup.table)) {
fprintf(stderr, "do_sector: out of space!\n");
ret = -1;
goto out;
}
entry = backup.table + backup.count++;
entry->flags = cur->boot_ind;
if (!offset)
entry->flags |= ENTRY_FLAG_PRIMARY;
entry->type = cur->sys_ind;
entry->start = cur->start + start;
entry->size = cur->size;
LE/BE issues here...
}
if (ext)
ret = do_sector(fd, offset, ext->start);
out:
put_sector(sector);
return ret;
}
void show_backup(void)
{
#ifdef DEBUG
int i;
fprintf(stderr, "signature: ");
for (i = 0; i < 8; i++)
fprintf(stderr, "%c", backup.signature[i]);
fprintf(stderr, "\n");
fprintf(stderr, " type: %d\n", backup.type);
fprintf(stderr, " heads: %d\n", backup.heads);
fprintf(stderr, " sectors: %d\n", backup.sectors);
fprintf(stderr, " count: %d\n", backup.count);
for (i = 0; i < backup.count; i++) {
fprintf(stderr, "\n");
fprintf(stderr, "%2d: flags: %02x\n", i, backup.table[i].flags);
fprintf(stderr, "%2d: type: %02x\n", i, backup.table[i].type);
fprintf(stderr, "%2d: start: %llu\n", i, backup.table[i].start);
fprintf(stderr, "%2d: size: %u\n", i, backup.table[i].size);
You'll output wrong values here, depending on your host system.
}
#endif
}
int main(int argc, char *argv[])
{
int fd;
struct stat buf;
if (argc != 2) {
printf("%s <blkdev>\n", argv[0]);
return EXIT_FAILURE;
}
fd = open(argv[1], O_RDONLY);
if (fd < 0) {
perror("open");
return EXIT_FAILURE;
}
if (fstat(fd, &buf) < 0) {
perror("stat");
return EXIT_FAILURE;
}
if (!S_ISBLK(buf.st_mode))
fprintf(stderr, "warning: not a block device\n");
if (do_sector(fd, 0, 0) < 0)
return EXIT_FAILURE;
if (close(fd) < 0) {
perror("close");
return EXIT_FAILURE;
}
if (write(STDOUT_FILENO, &backup, sizeof backup) != sizeof backup) {
perror("write");
return EXIT_FAILURE;
}
show_backup();
return EXIT_SUCCESS;
}
Looks like a useful program, but you'd definively fix the LE/BE
issues. If you do that, it'll be able to even run on BE machines, too.
MfG, JBG
--
Jan-Benedict Glaw jbglaw@lug-owl.de +49-172-7608481
Signature of: Träume nicht von Dein Leben: Lebe Deinen Traum!
the second :
This can stay for now. The partition table backup would indeed need some
defined byte-order but it might be "whatever order the filesystem it's
backed up onto uses". Since it's not directly written to any filesystem for
now, host order will do currently.
Looks like a useful program, but you'd definively fix the LE/BE issues.
If you do that, it'll be able to even run on BE machines, too.
I might disagree with the usefulness. I believe that preferably a backup
should be made with help from the kernel (if only by walking sysfs) to avoid
things getting out of sync between kernel and backup program.
(this program largely does the same as the kernel does but even now there's
already a difference in so far that I didn't bother to de-garbage the 3rd
and 4th entries in the second level extendeds).
Rene.
On Mon, Jul 23, 2007 at 10:15:21AM +0200, Rene Herman wrote:
On an integrated system like this, do you consider it acceptable to only do
the MS-DOS partitions and not the other types that may be present _inside_
those partitions? (MINIX subpartitions, BSD slices, ...). I believe those
should really also be done, but this would require keeping more information
again.
Well, I'm considering this to be a MBR backup scheme, so Minix and BSD
slices are legacy systems which are out of scope. If they are busted
in the same way as MBR in terms of not having redundant backups of
critical data, when they have a lot fewer excuses that MBR, and they
can address that issue in their own way. The number of Linux users
that also have Minix and BSD partitions are a vanishingly small number
in any case.
I (very) briefly looked at blkid but unless I'm mistaken blkid needs device
names? The documentation seems to be missing. When scanning the device for
the partition table, we've built a list of partitions with offsets into the
device and it would be nice if we could hand the fd and the offset off to
something directly. If the program has to construct device names itself
there's another truckload of pitfalls right there.
Yeah, good point, I'd have to add that support into blkid. It's been
on my todo list, but I just haven't gotten around to it yet.
It might in fact make sense to just ask the kernel for the partitions on a
device and not bother with scanning anything ourselves. Ie, just walk
sysfs. Would you agree? This siginificantly reduces the risk of things
getting out of sync, both scanning order and implementation.
My concern of sysfs is that #1, it won't work on older kernels since
you would need to add new fields to backup what we want, and #2, I'm
still fundamentally distrustful of sysfs because there isn't a bright
line between what is an exported interface that will never change, and
something which is considered an "internal implementation detail" that
can change whenever some kernel hacker feels like it. (Or when some
kernel hacker is careless...) So as far as I'm concerned sysfs is a
terrible, TERRIBLE way to export a published interface where we
promise stability to userspace.
So I'd just as soon do this in userspace; after all, the entire
partition manager (and there are multiple ones; fdisk, sfdisk, gpart,
etc.) all in userspace, and that needs to be in synch with the kernel
partition reading code anyway. So one more userspace implementation
is in my mind much cleaner than trying to push the needed
functionality into sysfs, and then hoping against hope that it doesn't
accidentally change in the future.
- Ted
This can stay for now. The partition table backup would indeed need
some defined byte-order but it might be "whatever order the filesystem
it's backed up onto uses". Since it's not directly written to any
filesystem for now, host order will do currently.
quoted
Looks like a useful program, but you'd definively fix the LE/BE issues.
If you do that, it'll be able to even run on BE machines, too.
I might disagree with the usefulness. I believe that preferably a
backup should be made with help from the kernel (if only by walking
sysfs) to avoid things getting out of sync between kernel and backup
program.
(this program largely does the same as the kernel does but even now
there's already a difference in so far that I didn't bother to
de-garbage the 3rd and 4th entries in the second level extendeds).
How can I politely say this code really needs comments?
To quote the late R. W. Benway, "If it was hard to write it should be
hard to understand."
(regarding code in FORTRAN II on punched cards, ca 1965)
Rene.
--
bill davidsen [off-list ref]
CTO TMR Associates, Inc
Doing interesting things with small computers since 1979
From: Rene Herman <hidden> Date: 2007-07-24 04:08:32
On 07/23/2007 03:58 PM, Theodore Tso wrote:
Well, I'm considering this to be a MBR backup scheme, so Minix and BSD
slices are legacy systems which are out of scope. If they are busted in
the same way as MBR in terms of not having redundant backups of critical
data, when they have a lot fewer excuses that MBR, and they can address
that issue in their own way. The number of Linux users that also have
Minix and BSD partitions are a vanishingly small number in any case.
I'd in fact expect quite a few people to have a FreeBSD partition around.
And MINIX if they are in university and in an operating systems course...
But "they should take whatever precautions they want themselves" is a valid
argument.
[ blkid ]
Yeah, good point, I'd have to add that support into blkid. It's been
on my todo list, but I just haven't gotten around to it yet.
I'll for now stop updating the partbackup thingy as posted. Given that Linux
only follows the first extended in the list of extendeds (which sort of
destroys the nice recursion anyway) it might want to be iterative instead of
recursive if the thing has a future -- not very important though.
My concern of sysfs is that #1, it won't work on older kernels since
you would need to add new fields to backup what we want,
I'd be okay with that.
and #2, I'm still fundamentally distrustful of sysfs because there isn't
a bright line between what is an exported interface that will never
change, and something which is considered an "internal implementation
detail" that can change whenever some kernel hacker feels like it. (Or
when some kernel hacker is careless...) So as far as I'm concerned sysfs
is a terrible, TERRIBLE way to export a published interface where we
promise stability to userspace.
Oh come on, that's going overboard a bit, it's not all _that_ bad! Finding
say "sda" will be possible without breaking too many times. Admittedly, the
kernel's partittion scanning order is also not likely to change as it would
certainly break userspace, but code duplication, with the possiblity of bugs
slipping in at least userspace-ways (considering the kernel the reference no
matter what it does) is a concern. Somewhat. A little.
So I'd just as soon do this in userspace; after all, the entire partition
manager (and there are multiple ones; fdisk, sfdisk, gpart, etc.) all in
userspace, and that needs to be in synch with the kernel partition
reading code anyway. So one more userspace implementation is in my mind
much cleaner than trying to push the needed functionality into sysfs, and
then hoping against hope that it doesn't accidentally change in the
future.
* rene envisions /lib/libpart.so...
Not to mention my Grand Visions of a totally new Linux native partitioning
scheme probably modelled after BSD slices (as also mentioned in a previous
message just now). Or perhaps LVM already fills that role comfortably. It's
certainly what I hear everyone talking about these days.
Rene.