Re: [PATCH] RAID-6 check standalone

From
Piergiorgio Sartor <hidden>
Date
2011-03-21 10:40:07
Reply to
NeilBrown, 2011-03-21
Hi Neil,

On Mon, Mar 21, 2011 at 02:02:44PM +1100, NeilBrown wrote:
On Mon, 21 Feb 2011 21:45:51 +0100 Piergiorgio Sartor
[off-list ref] wrote:
quoted
Hi Neil,

please find attached a patch, to mdadm-3.2 base, including
a standalone versione of the raid-6 check.

This is basically a re-working (and hopefully improvement)
of the already implemented check in "restripe.c".

I splitted the check function into "collect" and "stats",
so that the second one could be easily replaced.
The API is also simplified.

The command line option are reduced, since we only level
is raid-6, but the ":offset" option is included.

The output reports the block/stripe rotation, P/Q errors
and the possible HDD (or unknown).

BTW, the patch applies also to the already patched "restripe.c",
including the last ":offset" patch (which is not yet in git).
Sorry for not replying for a month!!  I manage to lose track of this then
couldn't find it when I went looking, then got distracted, then....
don't worry, I know how easy is to loose track of
such items. That's why I kept (slowly) insisting.
Thanks for your patience and persistence.

I have applied this patch to me git now - with some minor changes.
Thanks.

One question, I did not find the previous patch to "restripe.c",
which was adding the ":offset" capability. This was in order to
be able to cope with metadata 1.1 or 1.2 (offset to be determined).
quoted
Other item is that due to "sysfs.c" linking (see below) the
"Makefile" needed some changes, I hope this is not a problem.
I have reverted most of these changes and the linking with sysfs.c isn't
needed yet.  When it is we can sort out how best to achieve it.
quoted
Next steps (TODO list you like) would be:

1) Add the "sysfs.c" code in order to retrieve the HDDs info
from the MD device. It is already linked, together with the
whole (mdadm) universe, since it seems it cannot leave alone.
I'll need some advice or hint on how to do use it. I checked
"sysfs.c", but before I dig deep into it maybe better to
have some advice (maybe just one function call will do it).
What exactly do you want to achieve?

I suspect you want to a open the md device, then

  info = sysfs_read(fd, -1,
  GET_LEVEL|GET_LAYOUT_GET_DISKS|GET_CHUNK|GET_DEVS|GET_OFFSET);

(possibly with other flags) and then extract the info you want from the data
structure returned - but I'm only guessing at what you might want.
I think this more or less correct.

I would like to pass, to "raid6check", the md device as only
parameter, maybe with some start/stop position, and derive all
the needed information from that.

That is, it should be: raid6check /dev/mdX start stop
Or, possibly: raid6check /dev/mdX start length
Or just: raid6check /dev/mdX

This means, the information that should be derived is:

1) level, in order to confirm it is 6
2) layout
3) disk components
4) offset for each component
5) chunk size

That should be all, I guess.

Your "sysfs_read" seems to do exactly this, please confirm.
quoted
2) Add the suspend lo/hi control. Fellow John Robinson was
suggesting to look into "Grow.c", which I did, but I guess
the same story as 1) is valid: better to have some hint on
where to look before wasting time.
This would be:

  sysfs_set_num(info, NULL, "suspend_lo", offset*data_disks);
  sysfs_set_num(info, NULL, "suspend_hi", (offset+chunksize)*data_disks);

to freeze one stripe.  Then work in there.
The addresses are addresses in the array, hence the multiplication
by data_disks (which is raid_disks - 2 for RAID6).

Don't hold the array suspended for too long or something might get
upset.  And allocate any memory you need first, and call
    	mlockall(MCL_CURRENT | MCL_FUTURE);

first to be even more safe.
Thanks for the explanation. How is, then, the "unfreeze"?
Just writing (0) to both hi and lo?
quoted
3) Add a repair option (future). This should have different
levels, like "all", "disk", "stripe". That is, fix everything
(more or less like "repair"), fix only if a disk is clearly
having problems, fix each stripe which has clearly a problem
(but maybe different stripes may belong to different HDDs).

So, for the point 1) and 2) would be nice to have some more
detail on where to look what. Point 3) we will discuss later.
Agreed.
Hopefully the info I have provided is sufficient to help you progress.
I'll ask, if I need more information.
Thanks,
NeilBrown
Thank you,

bye,

pg
quoted
Thanks, please consider for inclusion,

bye,

pg
quoted

diff -urN a/Makefile b/Makefile
--- a/Makefile	2011-02-01 05:31:07.000000000 +0100
+++ b/Makefile	2011-02-20 00:03:44.929998038 +0100
@@ -98,7 +98,7 @@
 MAN5DIR = $(MANDIR)/man5
 MAN8DIR = $(MANDIR)/man8
 
-OBJS =  mdadm.o config.o policy.o mdstat.o  ReadMe.o util.o Manage.o Assemble.o Build.o \
+OBJSX = config.o policy.o mdstat.o  ReadMe.o util.o Manage.o Assemble.o Build.o \
 	Create.o Detail.o Examine.o Grow.o Monitor.o dlink.o Kill.o Query.o \
 	Incremental.o \
 	mdopen.o super0.o super1.o super-ddf.o super-intel.o bitmap.o \
@@ -106,6 +106,8 @@
 	restripe.o sysfs.o sha1.o mapfile.o crc32.o sg_io.o msg.o \
 	platform-intel.o probe_roms.o
 
+OBJS = mdadm.o $(OBJSX)
+
 SRCS =  mdadm.c config.c policy.c mdstat.c  ReadMe.c util.c Manage.c Assemble.c Build.c \
 	Create.c Detail.c Examine.c Grow.c Monitor.c dlink.c Kill.c Query.c \
 	Incremental.c \
@@ -182,6 +184,9 @@
 test_stripe : restripe.c mdadm.h
 	$(CC) $(CXFLAGS) $(LDFLAGS) -o test_stripe -DMAIN restripe.c
 
+raid6check : raid6check.o mdadm.h $(OBJSX)
+	$(CC) $(CXFLAGS) $(LDFLAGS) -o raid6check raid6check.o $(OBJSX)
+
 mdassemble : $(ASSEMBLE_SRCS) $(INCL)
 	rm -f $(OBJS)
 	$(DIET_GCC) $(ASSEMBLE_FLAGS) -o mdassemble $(ASSEMBLE_SRCS)  $(STATICSRC)
@@ -265,7 +270,7 @@
 	mdadm.Os mdadm.O2 mdmon.O2 \
 	mdassemble mdassemble.static mdassemble.auto mdassemble.uclibc \
 	mdassemble.klibc swap_super \
-	init.cpio.gz mdadm.uclibc.static test_stripe mdmon \
+	init.cpio.gz mdadm.uclibc.static test_stripe raid6check raid6check.o mdmon \
 	mdadm.8
 
 dist : clean
diff -urN a/raid6check.c b/raid6check.c
--- a/raid6check.c	1970-01-01 01:00:00.000000000 +0100
+++ b/raid6check.c	2011-02-21 21:40:27.813656214 +0100
@@ -0,0 +1,263 @@
+/*
+ * raid6check - extended consistency check for RAID-6
+ *
+ * Copyright (C) 2011 Piergiorgio Sartor
+ *
+ *
+ *    This program is free software; you can redistribute it and/or modify
+ *    it under the terms of the GNU General Public License as published by
+ *    the Free Software Foundation; either version 2 of the License, or
+ *    (at your option) any later version.
+ *
+ *    This program is distributed in the hope that it will be useful,
+ *    but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *    GNU General Public License for more details.
+ *
+ *    You should have received a copy of the GNU General Public License
+ *    along with this program; if not, write to the Free Software
+ *    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
+ *
+ *    Author: Piergiorgio Sartor
+ *    Based on "restripe.c" from "mdadm" codebase
+ */
+
+#include "mdadm.h"
+#include <stdint.h>
+
+int geo_map(int block, unsigned long long stripe, int raid_disks,
+	    int level, int layout);
+void qsyndrome(uint8_t *p, uint8_t *q, uint8_t **sources, int disks, int size);
+void make_tables(void);
+
+/* Collect per stripe consistency information */
+void raid6_collect(int chunk_size, uint8_t *p, uint8_t *q,
+		   char *chunkP, char *chunkQ, int *results)
+{
+	int i;
+	int data_id;
+	uint8_t Px, Qx;
+	extern uint8_t raid6_gflog[];
+
+	for(i = 0; i < chunk_size; i++) {
+		Px = (uint8_t)chunkP[i] ^ (uint8_t)p[i];
+		Qx = (uint8_t)chunkQ[i] ^ (uint8_t)q[i];
+
+		if((Px != 0) && (Qx == 0))
+			results[i] = -1;
+
+		if((Px == 0) && (Qx != 0))
+			results[i] = -2;
+
+		if((Px != 0) && (Qx != 0)) {
+			data_id = (raid6_gflog[Qx] - raid6_gflog[Px]);
+			if(data_id < 0) data_id += 255;
+			results[i] = data_id;
+		}
+
+		if((Px == 0) && (Qx == 0))
+			results[i] = -255;
+	}
+}
+
+/* Try to find out if a specific disk has problems */
+int raid6_stats(int *results, int raid_disks, int chunk_size)
+{
+	int i;
+	int curr_broken_disk = -255;
+	int prev_broken_disk = -255;
+	int broken_status = 0;
+	
+	for(i = 0; i < chunk_size; i++) {
+
+		if(results[i] != -255)
+			curr_broken_disk = results[i];
+
+		if(curr_broken_disk >= raid_disks)
+			broken_status = 2;
+
+		switch(broken_status) {
+		case 0:
+			if(curr_broken_disk != -255) {
+				prev_broken_disk = curr_broken_disk;
+				broken_status = 1;
+			}
+			break;
+
+		case 1:
+			if(curr_broken_disk != prev_broken_disk)
+				broken_status = 2;
+			break;
+
+		case 2:
+		default:
+			curr_broken_disk = prev_broken_disk = -65535;
+			break;
+		}
+	}
+
+	return curr_broken_disk;
+}
+
+int check_stripes(int *source, unsigned long long *offsets,
+		  int raid_disks, int chunk_size, int level, int layout,
+		  unsigned long long start, unsigned long long length, char *name[])
+{
+	/* read the data and p and q blocks, and check we got them right */
+	char *stripe_buf = malloc(raid_disks * chunk_size);
+	char **stripes = malloc(raid_disks * sizeof(char*));
+	char **blocks = malloc(raid_disks * sizeof(char*));
+	uint8_t *p = malloc(chunk_size);
+	uint8_t *q = malloc(chunk_size);
+	int *results = malloc(chunk_size * sizeof(int));
+
+	int i;
+	int diskP, diskQ;
+	int data_disks = raid_disks - 2;
+
+	extern int tables_ready;
+
+	if (!tables_ready)
+		make_tables();
+
+	for ( i = 0 ; i < raid_disks ; i++)
+		stripes[i] = stripe_buf + i * chunk_size;
+
+	while (length > 0) {
+		int disk;
+
+		for (i = 0 ; i < raid_disks ; i++) {
+			lseek64(source[i], offsets[i]+start, 0);
+			read(source[i], stripes[i], chunk_size);
+		}
+		for (i = 0 ; i < data_disks ; i++) {
+			int disk = geo_map(i, start/chunk_size, raid_disks,
+					   level, layout);
+			blocks[i] = stripes[disk];
+			printf("%d->%d\n", i, disk);
+		}
+
+		qsyndrome(p, q, (uint8_t**)blocks, data_disks, chunk_size);
+		diskP = geo_map(-1, start/chunk_size, raid_disks,
+				level, layout);
+		if (memcmp(p, stripes[diskP], chunk_size) != 0) {
+			printf("P(%d) wrong at %llu\n", diskP,
+			       start / chunk_size);
+		}
+		diskQ = geo_map(-2, start/chunk_size, raid_disks,
+				level, layout);
+		if (memcmp(q, stripes[diskQ], chunk_size) != 0) {
+			printf("Q(%d) wrong at %llu\n", diskQ,
+			       start / chunk_size);
+		}
+		raid6_collect(chunk_size, p, q,
+			      stripes[diskP], stripes[diskQ], results);
+		disk = raid6_stats(results, raid_disks, chunk_size);
+
+		if(disk >= -2) {
+			disk = geo_map(disk, start/chunk_size, raid_disks,
+				       level, layout);
+		}
+		if(disk >= 0) {
+			printf("Possible failed disk: %d --> %s\n", disk, name[disk]);
+		}
+		if(disk == -65535) {
+			printf("Failure detected, but disk unknown\n");
+		}
+
+		length -= chunk_size;
+		start += chunk_size;
+	}
+
+	free(stripe_buf);
+	free(stripes);
+	free(blocks);
+	free(p);
+	free(q);
+	free(results);
+
+	return 0;
+}
+
+unsigned long long getnum(char *str, char **err)
+{
+	char *e;
+	unsigned long long rv = strtoull(str, &e, 10);
+	if (e==str || *e) {
+		*err = str;
+		return 0;
+	}
+	return rv;
+}
+
+int main(int argc, char *argv[])
+{
+	/* raid_disks chunk_size layout start length devices...
+	 */
+	int *fds;
+	char *buf;
+	unsigned long long *offsets;
+	int raid_disks, chunk_size, layout;
+	int level = 6;
+	unsigned long long start, length;
+	int i;
+
+	char *err = NULL;
+	if (argc < 8) {
+		fprintf(stderr, "Usage: raid6check raid_disks"
+			" chunk_size layout start length devices...\n");
+		exit(1);
+	}
+
+	raid_disks = getnum(argv[1], &err);
+	chunk_size = getnum(argv[2], &err);
+	layout = getnum(argv[3], &err);
+	start = getnum(argv[4], &err);
+	length = getnum(argv[5], &err);
+	if (err) {
+		fprintf(stderr, "test_stripe: Bad number: %s\n", err);
+		exit(2);
+	}
+	if (argc != raid_disks + 6) {
+		fprintf(stderr, "test_stripe: wrong number of devices: want %d found %d\n",
+			raid_disks, argc-6);
+		exit(2);
+	}
+	fds = malloc(raid_disks * sizeof(*fds));
+	offsets = malloc(raid_disks * sizeof(*offsets));
+	memset(offsets, 0, raid_disks * sizeof(*offsets));
+
+	for (i=0; i<raid_disks; i++) {
+		char *p;
+		p = strchr(argv[6+i], ':');
+
+		if(p != NULL) {
+			*p++ = '\0';
+			offsets[i] = atoll(p) * 512;
+		}
+		fds[i] = open(argv[6+i], O_RDWR);
+		if (fds[i] < 0) {
+			perror(argv[6+i]);
+			fprintf(stderr,"test_stripe: cannot open %s.\n", argv[6+i]);
+			exit(3);
+		}
+	}
+
+	buf = malloc(raid_disks * chunk_size);
+
+	int rv = check_stripes(fds, offsets,
+			       raid_disks, chunk_size, level, layout,
+			       start, length, &argv[6]);
+	if (rv != 0) {
+		fprintf(stderr,
+			"test_stripe: test_stripes returned %d\n", rv);
+		exit(1);
+	}
+
+	free(fds);
+	free(offsets);
+	free(buf);
+
+	exit(0);
+}
+
diff -urN a/restripe.c b/restripe.c
--- a/restripe.c	2011-02-19 22:36:29.707651642 +0100
+++ b/restripe.c	2011-02-19 18:13:14.468626986 +0100
@@ -33,7 +33,7 @@
  *
  */
 
-static int geo_map(int block, unsigned long long stripe, int raid_disks,
+int geo_map(int block, unsigned long long stripe, int raid_disks,
 		   int level, int layout)
 {
 	/* On the given stripe, find which disk in the array will have
@@ -223,7 +223,7 @@
 	}
 }
 
-static void qsyndrome(uint8_t *p, uint8_t *q, uint8_t **sources, int disks, int size)
+void qsyndrome(uint8_t *p, uint8_t *q, uint8_t **sources, int disks, int size)
 {
 	int d, z;
 	uint8_t wq0, wp0, wd0, w10, w20;
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 

piergiorgio
Thread (27 messages) 27 messages, 3 authors, last reply 2011-08-09
Keyboard shortcuts
hback out one level
jnext message in thread / next series-diff position
kprevious message in thread / previous series-diff position
ldrill in
Escclose help / fold thread tree
?toggle this help