Re: [PATCH, RFC 2/2] dm: support REQ_FLUSH directly

Hi Christoph,

On 08/04/2010 03:51 AM +0900, Christoph Hellwig wrote:

Adapt device-mapper to the new world order where even bio based devices
get simple REQ_FLUSH requests for cache flushes, and need to submit
them downwards for implementing barriers.

<snip>

<snip>

Current request-based device-mapper's flush code depends on
the block-layer's barrier behavior which dispatches only one request
at a time when flush is needed.
In other words, current request-based device-mapper can't handle
other requests while a flush request is in progress.

I'll take a look how I can fix the request-based device-mapper to
cope with it.  I think it'll take time for carefull investigation.

Thanks,
Kiyoshi Ueda

On Wed, Aug 04, 2010 at 01:57:37PM +0900, Kiyoshi Ueda wrote:

quoted

-		if (unlikely(dm_rq_is_flush_request(rq))) {
+		if (rq->cmd_flags & REQ_FLUSH) {
 			BUG_ON(md->flush_request);
 			md->flush_request = rq;
 			blk_start_request(rq);

Current request-based device-mapper's flush code depends on
the block-layer's barrier behavior which dispatches only one request
at a time when flush is needed.
In other words, current request-based device-mapper can't handle
other requests while a flush request is in progress.

I'll take a look how I can fix the request-based device-mapper to
cope with it.  I think it'll take time for carefull investigation.

Given that request based device mapper doesn't even look at the
block numbers from what I can see just removing any special casing
for REQ_FLUSH should probably do it.

Hi Christoph,

(08/04/10 17:54), Christoph Hellwig wrote:

On Wed, Aug 04, 2010 at 01:57:37PM +0900, Kiyoshi Ueda wrote:

quoted

quoted

-		if (unlikely(dm_rq_is_flush_request(rq))) {
+		if (rq->cmd_flags & REQ_FLUSH) {
 			BUG_ON(md->flush_request);
 			md->flush_request = rq;
 			blk_start_request(rq);

Current request-based device-mapper's flush code depends on
the block-layer's barrier behavior which dispatches only one request
at a time when flush is needed.
In other words, current request-based device-mapper can't handle
other requests while a flush request is in progress.

I'll take a look how I can fix the request-based device-mapper to
cope with it.  I think it'll take time for carefull investigation.

Given that request based device mapper doesn't even look at the
block numbers from what I can see just removing any special casing
for REQ_FLUSH should probably do it.

Special casing is necessary because device-mapper may have to
send multiple copies of REQ_FLUSH request to multiple
targets, while normal request is just sent to single target.

Thanks,
-- 
Jun'ichi Nomura, NEC Corporation

On Wed, Aug 04 2010 at 10:16pm -0400,
Jun'ichi Nomura [off-list ref] wrote:

Hi Christoph,

(08/04/10 17:54), Christoph Hellwig wrote:

quoted

On Wed, Aug 04, 2010 at 01:57:37PM +0900, Kiyoshi Ueda wrote:

quoted

quoted

-		if (unlikely(dm_rq_is_flush_request(rq))) {
+		if (rq->cmd_flags & REQ_FLUSH) {
 			BUG_ON(md->flush_request);
 			md->flush_request = rq;
 			blk_start_request(rq);

Current request-based device-mapper's flush code depends on
the block-layer's barrier behavior which dispatches only one request
at a time when flush is needed.
In other words, current request-based device-mapper can't handle
other requests while a flush request is in progress.

I'll take a look how I can fix the request-based device-mapper to
cope with it.  I think it'll take time for carefull investigation.

Given that request based device mapper doesn't even look at the
block numbers from what I can see just removing any special casing
for REQ_FLUSH should probably do it.

Special casing is necessary because device-mapper may have to
send multiple copies of REQ_FLUSH request to multiple
targets, while normal request is just sent to single target.

Yes, request-based DM is meant to have all the same capabilities as
bio-based DM.  So in theory it should support multiple targets but in
practice it doesn't.  DM's multipath target is the only consumer of
request-based DM and it only ever clones a single flush request
(num_flush_requests = 1).

So why not remove all of request-based DM's barrier infrastructure and
simply rely on the revised block layer to sequence the FLUSH+WRITE
request for request-based DM?

Given that we do not have a request-based DM target that requires
cloning multiple FLUSH requests its unused code that is delaying DM
support for the new FLUSH+FUA work (NOTE: bio-based DM obviously still
needs work in this area).

Once we have a need for using request-based DM for something other than
multipath we can take a fresh look at implementing rq-based FLUSH+FUA.

Mike

p.s. I know how hard NEC worked on request-based DM's barrier support;
so I'm not suggesting this lightly.  For me it just seems like we're
carrying complexity in DM that hasn't ever been required.

On Thu, Aug 26 2010 at  6:50pm -0400,
Mike Snitzer [off-list ref] wrote:

Once we have a need for using request-based DM for something other than
multipath we can take a fresh look at implementing rq-based FLUSH+FUA.

Mike

p.s. I know how hard NEC worked on request-based DM's barrier support;
so I'm not suggesting this lightly.  For me it just seems like we're
carrying complexity in DM that hasn't ever been required.

To be clear: the piece that I was saying wasn't required is the need to
for request-based DM to clone a FLUSH to send to multiple targets
(saying as much was just a confusing distraction.. please ignore that).

Anyway, my previous email's question still stands.

Mike Snitzer wrote:

On Thu, Aug 26 2010 at  6:50pm -0400,
Mike Snitzer [off-list ref] wrote:

quoted

Once we have a need for using request-based DM for something other than
multipath we can take a fresh look at implementing rq-based FLUSH+FUA.

Mike

p.s. I know how hard NEC worked on request-based DM's barrier support;
so I'm not suggesting this lightly.  For me it just seems like we're
carrying complexity in DM that hasn't ever been required.

To be clear: the piece that I was saying wasn't required is the need to
for request-based DM to clone a FLUSH to send to multiple targets
(saying as much was just a confusing distraction.. please ignore that).

Anyway, my previous email's question still stands.

On a slightly related note: DM suggests a reason for the lower layer, or the
request queues, to implement the trivial optimisation of discarding
FLUSHes if there's been no WRITE since the previous FLUSH.

That was mentioned elsewhere in this big thread as not being worth
even the small effort - because the filesystem is able to make good
decisions anyway.

But once you have something like RAID or striping, it's quite common
for the filesystem to issue a FLUSH when only a subset of the target
devices have received WRITEs through the RAID/striping layer since
they last received a FLUSH.

-- Jamie

Hi Mike,

(08/27/10 07:50), Mike Snitzer wrote:

quoted

Special casing is necessary because device-mapper may have to
send multiple copies of REQ_FLUSH request to multiple
targets, while normal request is just sent to single target.

Yes, request-based DM is meant to have all the same capabilities as
bio-based DM.  So in theory it should support multiple targets but in
practice it doesn't.  DM's multipath target is the only consumer of
request-based DM and it only ever clones a single flush request
(num_flush_requests = 1).

This is correct. But,

So why not remove all of request-based DM's barrier infrastructure and
simply rely on the revised block layer to sequence the FLUSH+WRITE
request for request-based DM?

Given that we do not have a request-based DM target that requires
cloning multiple FLUSH requests its unused code that is delaying DM
support for the new FLUSH+FUA work (NOTE: bio-based DM obviously still
needs work in this area).

the above mentioned 'special casing' is not a hard part.
See the attached patch.

The hard part is discerning the error type for flush failure
as discussed in the other thread.
And as Kiyoshi wrote, that's an existing problem so it can
be worked on as a separate issue than the new FLUSH work.

Thanks,
-- 
Jun'ichi Nomura, NEC Corporation


Cope with new sequencing of flush requests in the block layer.

Request-based dm used to depend on the barrier sequencer in the block layer
in that, when a flush request is dispatched, there are no other requests
in-flight. So it reused md->pending counter for checking completion of
cloned flush requests.

This patch separates the pending counter for flush request
as a prepartion for the new FLUSH work, where a flush request can be
dispatched while other normal requests are in-flight.

Index: linux-2.6.36-rc2/drivers/md/dm.c
===================================================================

--- linux-2.6.36-rc2.orig/drivers/md/dm.c
+++ linux-2.6.36-rc2/drivers/md/dm.c

@@ -162,6 +162,7 @@ struct mapped_device {
 
 	/* A pointer to the currently processing pre/post flush request */
 	struct request *flush_request;
+	atomic_t flush_pending;
 
 	/*
 	 * The current mapping.

@@ -777,10 +778,16 @@ static void store_barrier_error(struct m
  * the md may be freed in dm_put() at the end of this function.
  * Or do dm_get() before calling this function and dm_put() later.
  */
-static void rq_completed(struct mapped_device *md, int rw, int run_queue)
+static void rq_completed(struct mapped_device *md, int rw, int run_queue, bool is_flush)
 {
 	atomic_dec(&md->pending[rw]);
 
+	if (is_flush) {
+		atomic_dec(&md->flush_pending);
+		if (!atomic_read(&md->flush_pending))
+			wake_up(&md->wait);
+	}
+
 	/* nudge anyone waiting on suspend queue */
 	if (!md_in_flight(md))
 		wake_up(&md->wait);

@@ -837,7 +844,7 @@ static void dm_end_request(struct reques
 	} else
 		blk_end_request_all(rq, error);
 
-	rq_completed(md, rw, run_queue);
+	rq_completed(md, rw, run_queue, is_barrier);
 }
 
 static void dm_unprep_request(struct request *rq)

@@ -880,7 +887,7 @@ void dm_requeue_unmapped_request(struct 
 	blk_requeue_request(q, rq);
 	spin_unlock_irqrestore(q->queue_lock, flags);
 
-	rq_completed(md, rw, 0);
+	rq_completed(md, rw, 0, false);
 }
 EXPORT_SYMBOL_GPL(dm_requeue_unmapped_request);
 

@@ -1993,6 +2000,7 @@ static struct mapped_device *alloc_dev(i
 
 	atomic_set(&md->pending[0], 0);
 	atomic_set(&md->pending[1], 0);
+	atomic_set(&md->flush_pending, 0);
 	init_waitqueue_head(&md->wait);
 	INIT_WORK(&md->work, dm_wq_work);
 	INIT_WORK(&md->barrier_work, dm_rq_barrier_work);

@@ -2375,7 +2383,7 @@ void dm_put(struct mapped_device *md)
 }
 EXPORT_SYMBOL_GPL(dm_put);
 
-static int dm_wait_for_completion(struct mapped_device *md, int interruptible)
+static int dm_wait_for_completion(struct mapped_device *md, int interruptible, bool for_flush)
 {
 	int r = 0;
 	DECLARE_WAITQUEUE(wait, current);

@@ -2388,6 +2396,8 @@ static int dm_wait_for_completion(struct
 		set_current_state(interruptible);
 
 		smp_mb();
+		if (for_flush && !atomic_read(&md->flush_pending))
+			break;
 		if (!md_in_flight(md))
 			break;
 

@@ -2408,14 +2418,14 @@ static int dm_wait_for_completion(struct
 
 static void dm_flush(struct mapped_device *md)
 {
-	dm_wait_for_completion(md, TASK_UNINTERRUPTIBLE);
+	dm_wait_for_completion(md, TASK_UNINTERRUPTIBLE, false);
 
 	bio_init(&md->barrier_bio);
 	md->barrier_bio.bi_bdev = md->bdev;
 	md->barrier_bio.bi_rw = WRITE_BARRIER;
 	__split_and_process_bio(md, &md->barrier_bio);
 
-	dm_wait_for_completion(md, TASK_UNINTERRUPTIBLE);
+	dm_wait_for_completion(md, TASK_UNINTERRUPTIBLE, false);
 }
 
 static void process_barrier(struct mapped_device *md, struct bio *bio)

@@ -2512,11 +2522,12 @@ static int dm_rq_barrier(struct mapped_d
 			clone = clone_rq(md->flush_request, md, GFP_NOIO);
 			dm_rq_set_target_request_nr(clone, j);
 			atomic_inc(&md->pending[rq_data_dir(clone)]);
+			atomic_inc(&md->flush_pending);
 			map_request(ti, clone, md);
 		}
 	}
 
-	dm_wait_for_completion(md, TASK_UNINTERRUPTIBLE);
+	dm_wait_for_completion(md, TASK_UNINTERRUPTIBLE, true);
 	dm_table_put(map);
 
 	return md->barrier_error;

@@ -2705,7 +2716,7 @@ int dm_suspend(struct mapped_device *md,
 	 * We call dm_wait_for_completion to wait for all existing requests
 	 * to finish.
 	 */
-	r = dm_wait_for_completion(md, TASK_INTERRUPTIBLE);
+	r = dm_wait_for_completion(md, TASK_INTERRUPTIBLE, false);
 
 	down_write(&md->io_lock);
 	if (noflush)

On Thu, Aug 26 2010 at  9:43pm -0400,
Jun'ichi Nomura [off-list ref] wrote:

Hi Mike,

(08/27/10 07:50), Mike Snitzer wrote:

quoted

quoted

Special casing is necessary because device-mapper may have to
send multiple copies of REQ_FLUSH request to multiple
targets, while normal request is just sent to single target.

Yes, request-based DM is meant to have all the same capabilities as
bio-based DM.  So in theory it should support multiple targets but in
practice it doesn't.  DM's multipath target is the only consumer of
request-based DM and it only ever clones a single flush request
(num_flush_requests = 1).

This is correct. But,

quoted

So why not remove all of request-based DM's barrier infrastructure and
simply rely on the revised block layer to sequence the FLUSH+WRITE
request for request-based DM?

Given that we do not have a request-based DM target that requires
cloning multiple FLUSH requests its unused code that is delaying DM
support for the new FLUSH+FUA work (NOTE: bio-based DM obviously still
needs work in this area).

the above mentioned 'special casing' is not a hard part.
See the attached patch.

Yes, Tejun suggested something like this in one of the threads.  Thanks
for implementing it.

But do you agree that the request-based barrier code (added in commit
d0bcb8786) could be reverted given the new FLUSH work?

We no longer need waiting now that ordering isn't a concern.  Especially
so given rq-based doesn't support multiple targets.  As you know, from
dm_table_set_type:

        /*
         * Request-based dm supports only tables that have a single target now.
         * To support multiple targets, request splitting support is needed,
         * and that needs lots of changes in the block-layer.
         * (e.g. request completion process for partial completion.)
         */

I think we need to at least benchmark the performance of dm-mpath
without any of this extra, soon to be unnecessary, code.

Maybe my concern is overblown...

The hard part is discerning the error type for flush failure
as discussed in the other thread.
And as Kiyoshi wrote, that's an existing problem so it can
be worked on as a separate issue than the new FLUSH work.

Right, Mike Christie will be refreshing his patchset that should enable
us to resolve that separate issue.

Thanks,
Mike

Hi Mike,

(08/27/10 13:08), Mike Snitzer wrote:

quoted

the above mentioned 'special casing' is not a hard part.
See the attached patch.

Yes, Tejun suggested something like this in one of the threads.  Thanks
for implementing it.

But do you agree that the request-based barrier code (added in commit
d0bcb8786) could be reverted given the new FLUSH work?

No, it's a separate thing.
If we don't need to care about the case where multiple clones
of flush request are necessary, the special casing of flush
request can be removed regardless of the new FLUSH work.

We no longer need waiting now that ordering isn't a concern.  Especially

The waiting is not for ordering, but for multiple clones.

so given rq-based doesn't support multiple targets.  As you know, from
dm_table_set_type:

        /*
         * Request-based dm supports only tables that have a single target now.
         * To support multiple targets, request splitting support is needed,
         * and that needs lots of changes in the block-layer.
         * (e.g. request completion process for partial completion.)
         */

This comment is about multiple targets.
The special code for barrier is for single target whose
num_flush_requests > 1. Different thing.

I think we need to at least benchmark the performance of dm-mpath
without any of this extra, soon to be unnecessary, code.

If there will be no need for supporting a request-based target
with num_flush_requests > 1, the special handling of flush
can be removed.

And since there is no such target in the current tree,
I don't object if you remove that part of code for good reason.

Thanks,
-- 
Jun'ichi Nomura, NEC Corporation

On Fri, Aug 27 2010 at  1:52am -0400,
Jun'ichi Nomura [off-list ref] wrote:

Hi Mike,

(08/27/10 13:08), Mike Snitzer wrote:

quoted

But do you agree that the request-based barrier code (added in commit
d0bcb8786) could be reverted given the new FLUSH work?

No, it's a separate thing.
If we don't need to care about the case where multiple clones
of flush request are necessary, the special casing of flush
request can be removed regardless of the new FLUSH work.

Ah, yes thanks for clarifying.  But we've never cared about multiple
clone of a flush so it's odd that such elaborate infrastructure was
introduced without a need.

quoted

We no longer need waiting now that ordering isn't a concern.  Especially

The waiting is not for ordering, but for multiple clones.

quoted

so given rq-based doesn't support multiple targets.  As you know, from
dm_table_set_type:

        /*
         * Request-based dm supports only tables that have a single target now.
         * To support multiple targets, request splitting support is needed,
         * and that needs lots of changes in the block-layer.
         * (e.g. request completion process for partial completion.)
         */

This comment is about multiple targets.
The special code for barrier is for single target whose
num_flush_requests > 1. Different thing.

Yes, I need to not send mail just before going to bed..
 

quoted

I think we need to at least benchmark the performance of dm-mpath
without any of this extra, soon to be unnecessary, code.

If there will be no need for supporting a request-based target
with num_flush_requests > 1, the special handling of flush
can be removed.

And since there is no such target in the current tree,
I don't object if you remove that part of code for good reason.

OK, certainly something to keep in mind.  But _really_ knowing the
multipath FLUSH+FUA performance difference (extra special-case code vs
none) requires a full FLUSH conversion of request-based DM anyway.

In general, request-based DM's barrier/flush code does carry a certain
maintenance overhead.  It is quite a bit of distracting code in the core
DM which isn't buying us anything.. so we _could_ just remove it and
never look back (until we have some specific need for num_flush_requests

1 in rq-based DM).

Mike

Hi Mike,

(08/27/10 23:13), Mike Snitzer wrote:

quoted

If there will be no need for supporting a request-based target
with num_flush_requests > 1, the special handling of flush
can be removed.

And since there is no such target in the current tree,
I don't object if you remove that part of code for good reason.

OK, certainly something to keep in mind.  But _really_ knowing the
multipath FLUSH+FUA performance difference (extra special-case code vs
none) requires a full FLUSH conversion of request-based DM anyway.

In general, request-based DM's barrier/flush code does carry a certain
maintenance overhead.  It is quite a bit of distracting code in the core
DM which isn't buying us anything.. so we _could_ just remove it and
never look back (until we have some specific need for num_flush_requests

quoted

1 in rq-based DM).

So, I'm not objecting to your idea.
Could you please create a patch to remove that?

Thanks,
-- 
Jun'ichi Nomura, NEC Corporation

On 08/30/2010 06:45 AM, Jun'ichi Nomura wrote:

Hi Mike,

(08/27/10 23:13), Mike Snitzer wrote:

quoted

quoted

If there will be no need for supporting a request-based target
with num_flush_requests > 1, the special handling of flush
can be removed.

And since there is no such target in the current tree,
I don't object if you remove that part of code for good reason.

OK, certainly something to keep in mind.  But _really_ knowing the
multipath FLUSH+FUA performance difference (extra special-case code vs
none) requires a full FLUSH conversion of request-based DM anyway.

In general, request-based DM's barrier/flush code does carry a certain
maintenance overhead.  It is quite a bit of distracting code in the core
DM which isn't buying us anything.. so we _could_ just remove it and
never look back (until we have some specific need for num_flush_requests

quoted

1 in rq-based DM).

So, I'm not objecting to your idea.
Could you please create a patch to remove that?

I did that yesterday.  Will post the patch soon.

Thanks.

-- 
tejun

On Mon, Aug 30 2010 at  4:33am -0400,
Tejun Heo [off-list ref] wrote:

On 08/30/2010 06:45 AM, Jun'ichi Nomura wrote:

quoted

Hi Mike,

(08/27/10 23:13), Mike Snitzer wrote:

quoted

quoted

If there will be no need for supporting a request-based target
with num_flush_requests > 1, the special handling of flush
can be removed.

And since there is no such target in the current tree,
I don't object if you remove that part of code for good reason.

OK, certainly something to keep in mind.  But _really_ knowing the
multipath FLUSH+FUA performance difference (extra special-case code vs
none) requires a full FLUSH conversion of request-based DM anyway.

In general, request-based DM's barrier/flush code does carry a certain
maintenance overhead.  It is quite a bit of distracting code in the core
DM which isn't buying us anything.. so we _could_ just remove it and
never look back (until we have some specific need for num_flush_requests

quoted

1 in rq-based DM).

So, I'm not objecting to your idea.
Could you please create a patch to remove that?

I did that yesterday.  Will post the patch soon.

I did it yesterday also, mine builds on your previous DM patchset...

I'll review your recent patchset, from today, to compare and will share
my findings.

I was hoping we could get the current request-based code working with
your new FLUSH+FUA work without removing support for num_flush_requests
(yet).  And then layer in the removal to give us the before and after so
we would know the overhead associated with keeping/dropping
num_flush_requests.  But like I said earlier "we _could_ just remove it
and never look back".

Thanks,
Mike

Hello,

On 08/30/2010 02:43 PM, Mike Snitzer wrote:

I did it yesterday also, mine builds on your previous DM patchset...

I'll review your recent patchset, from today, to compare and will share
my findings.

Thanks. :-)

I was hoping we could get the current request-based code working with
your new FLUSH+FUA work without removing support for num_flush_requests
(yet).  And then layer in the removal to give us the before and after so
we would know the overhead associated with keeping/dropping
num_flush_requests.  But like I said earlier "we _could_ just remove it
and never look back".

I tried but it's not very easy because the original implementation
depended on the block layer suppressing other requests while flush
sequence is in progress.  The painful part was that block layer no
longer sorts requeued flush requests in front of other front inserted
requests, so explicit queue suppressing can't be implemented simply.
Another route would be adding a separate wait/wakeup logic for flushes
(someone posted a demo patch for that which was almost there but not
fully), but it seemed like a aimless effort to build a new facility to
rip it out in the next patch.  After all, the whole thing seemed
somewhat pointless given that writes can't be routed to multiple
targets (if writes can't target multiple devices, flushes won't need
to either).

Thanks.

-- 
tejun