Out of sync mirrors

Hi,
I am using linux 2.6.8 for my research on raid systems.

I need to have the disks of an array out of sync for a while.
I want to be able to stop a disk request queue (blk_stop_queue) and
start it sometime later. Meanwhile its mirrors should be serving r/w
as if this mirror was up.
Whenever the stopped mirror comes back it will serve all the requests
already sent to it.

I am queueing up the requests sent to all others while my mirror queue
is stopped.

The problem is that the other mirrors stop serving as well. Even if I
don't stop the queue, only delay its request a little bit, still the
others also slow down.

Question: how do I get raid to allow mirrors to be out of sync?

Obviously, I am missing something out in the code.

Any clues?

Thanks,
Ana Paula

Ana Paula wrote:

Hi,
I am using linux 2.6.8 for my research on raid systems.

I need to have the disks of an array out of sync for a while.
I want to be able to stop a disk request queue (blk_stop_queue) and
start it sometime later. Meanwhile its mirrors should be serving r/w
as if this mirror was up.
Whenever the stopped mirror comes back it will serve all the requests
already sent to it.

I am queueing up the requests sent to all others while my mirror queue
is stopped.

The problem is that the other mirrors stop serving as well. Even if I
don't stop the queue, only delay its request a little bit, still the
others also slow down.

Question: how do I get raid to allow mirrors to be out of sync?

Obviously, I am missing something out in the code.

Any clues?
  

Take a look at "write-mostly" and see if creative use of it will assist. 
I suspect you're doing something which needs a bitmap, as well.

-- 
Bill Davidsen [off-list ref]
  "Woe unto the statesman who makes war without a reason that will still
  be valid when the war is over..." Otto von Bismark 

I got an idea today, of not letting the sync thread start while the
disks are out of sync.

I am not sure if it will suffice to fool the raid.

-AP

On Thu, Jul 24, 2008 at 6:37 PM, Bill Davidsen [off-list ref] wrote:

Ana Paula wrote:

quoted

Hi,
I am using linux 2.6.8 for my research on raid systems.

I need to have the disks of an array out of sync for a while.
I want to be able to stop a disk request queue (blk_stop_queue) and
start it sometime later. Meanwhile its mirrors should be serving r/w
as if this mirror was up.
Whenever the stopped mirror comes back it will serve all the requests
already sent to it.

I am queueing up the requests sent to all others while my mirror queue
is stopped.

The problem is that the other mirrors stop serving as well. Even if I
don't stop the queue, only delay its request a little bit, still the
others also slow down.

Question: how do I get raid to allow mirrors to be out of sync?

Obviously, I am missing something out in the code.

Any clues?

Take a look at "write-mostly" and see if creative use of it will assist. I suspect you're doing something which needs a bitmap, as well.

--
Bill Davidsen [off-list ref]
 "Woe unto the statesman who makes war without a reason that will still
 be valid when the war is over..." Otto von Bismark

Hi,
I am using linux 2.6.8 for my research on raid systems.

I need to have the disks of an array out of sync for a while.
I want to be able to stop a disk request queue (blk_stop_queue) and
start it sometime later. Meanwhile its mirrors should be serving r/w
as if this mirror was up.
Whenever the stopped mirror comes back it will serve all the requests
already sent to it.

I am queueing up the requests sent to all others while my mirror queue
is stopped.

The problem is that the other mirrors stop serving as well. Even if I
don't stop the queue, only delay its request a little bit, still the
others also slow down.

Question: how do I get raid to allow mirrors to be out of sync?

Obviously, I am missing something out in the code.

Any clues?

Thanks,
Ana Paula

Ana:

You are making this way too hard for yourself
If you want to stop the queue but let the RAID system continue
processing the queue, then just sending I/O from whatever program you
are using to generate the load.  This won't let you start up with a
partially unflushed queue, but if for whatever reason you need to
measure effects of pending I/Os and timing, then you can obtain a
reasonably good simulation by increasing the queue depth and repeating
the test.

As for A way to get RAID (I assume software RAID, a la the md driver),
to get out of sync is to "remove" a disk drive.  If you are afraid to
crack the case, then use a cheap USB enclosure for one of the disks and
then turn it off or unplug the USB connector.  You can also "remove" a
disk by sending it a command to go into a self-test, perform a low-level
format, send a mode select to change the block size, there are lots of
ways if you think outside of the box.

All of these "failure" scenarios will interact differently, just turning
a disk drive off creates a real failure, not a quasi-emulated more
graceful failure that may be easier for you to time and measure the
effects.

Good luck.  

David @ santools.com

Ana:

You are making this way too hard for yourself
If you want to stop the queue but let the RAID system continue
processing the queue, then just sending I/O from whatever program you
are using to generate the load.  This won't let you start up with a
partially unflushed queue, but if for whatever reason you need to
measure effects of pending I/Os and timing, then you can obtain a
reasonably good simulation by increasing the queue depth and repeating
the test.

I didn't quite understood what you are saying. I need to stop the
queue in order to see its growth.

As for A way to get RAID (I assume software RAID, a la the md driver),
to get out of sync is to "remove" a disk drive.  If you are afraid to
crack the case, then use a cheap USB enclosure for one of the disks and
then turn it off or unplug the USB connector.  You can also "remove" a
disk by sending it a command to go into a self-test, perform a low-level
format, send a mode select to change the block size, there are lots of
ways if you think outside of the box.

By doing this I wouldn't see its queue building up, would I?

Basically I need to simulate a raid with disks of different speeds.
Where a faster mirror would process its requests before the slower
ones.
For reads it processes more requests to keep the raid load balanced.
But for writes I can't load balance, all have to write. The problem is
that the faster is slowing down to meet the slower disk throughput.

See the queue build up is key.

Thanks,
AP

All of these "failure" scenarios will interact differently, just turning
a disk drive off creates a real failure, not a quasi-emulated more
graceful failure that may be easier for you to time and measure the
effects.

Good luck.

David @ santools.com


--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html