Re: [PATCH 1/4] raid1: Add a filed array_frozen to indicate whether raid in freeze state.
From: Alexander Lyakas <hidden>
Date: 2016-06-14 09:33:50
Hello Jianpeng Ma, Neil,
This commit seems to introduce a severe regression, leading to IO
getting stuck on the whole array.
Previously, the condition of wait_barrier() was:
wait_event_lock_irq(conf->wait_barrier,
!conf->barrier ||
(conf->nr_pending &&
current->bio_list &&
!bio_list_empty(current->bio_list)),
conf->resync_lock);
This means that if current->bio_list is not empty, we will not wait
for the barrier to drop.
But right now the condition is:
wait_event_lock_irq(conf->wait_barrier,
!conf->array_frozen &&
(!conf->barrier ||
(conf->nr_pending &&
current->bio_list &&
!bio_list_empty(current->bio_list))),
conf->resync_lock);
This means that if somebody calls freeze_array (for example,
md_do_sync), then freeze_array unconditionally sets:
conf->array_frozen = 1;
And then if somebody calls wait_barrier and its current->bio_list is
not empty, it will still wait, because array_frozen=1. But
freeze_array is waiting for the bio complete, but this bio sits in
current->bio_list and will never get submitted, because we are waiting
in wait_barrier. DEADLOCK.
For me it happens on kernel 3.18, when raid1.c::make_request() submits
a READ bio, which has raid1_end_read_request completion routine. But
this completion routine never gets called, and I also confirmed that
no IO is stuck on lower levels.
Additional notes:
# Please see my email titled "RAID1: deadlock between freeze_array and
blk plug?" in http://permalink.gmane.org/gmane.linux.raid/52693. The
problem described there also stems from the fact that now
freeze_array() sets array_frozen=1 unconditionally. And wait_barrier()
will always wait if array_frozen!=0.
# Also now freeze_array() is non-reentrant. Meaning that if two
threads call freeze_array in parallel, it will not work, because
array_frozen can only be 1 or 0. Example, when freeze_array can be
called by two threads:
- We have 3-way RAID1 with one drive missing. We want to recover this drive.
- we have an in-flight READ request
- md_do_sync starts recovery and calls mddev->pers->quiesce(mddev, 1),
which waits for the in-flight READ to complete
- meanwhile the READ bio fails
- so raid1d calls handle_read_error, which calls freeze_array
Result: we have two freeze_array calls in parallel. But we don't know
to account for them properly, because array_frozen can only be 1 or 0.
Please advise how to fix this problem. Please kindly note that the fix
should apply to kernel 3.18; which is a longterm kernel that we use.
Thanks,
Alex.
On Wed, Aug 28, 2013 at 2:40 PM, majianpeng [off-list ref] wrote:quoted hunk ↗ jump to hunk
Because the following patch will rewrite the contend between nornal IO and resync IO. So we used a parameter to indicate whether raid is in freeze array. Signed-off-by: Jianpeng Ma <redacted> --- drivers/md/raid1.c | 15 +++++++-------- drivers/md/raid1.h | 1 + 2 files changed, 8 insertions(+), 8 deletions(-)diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index d60412c..92a6d29 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c@@ -829,6 +829,7 @@ static void raise_barrier(struct r1conf *conf) /* Now wait for all pending IO to complete */ wait_event_lock_irq(conf->wait_barrier, + !conf->array_frozen && !conf->nr_pending && conf->barrier < RESYNC_DEPTH, conf->resync_lock);@@ -860,10 +861,11 @@ static void wait_barrier(struct r1conf *conf) * count down. */ wait_event_lock_irq(conf->wait_barrier, - !conf->barrier || + !conf->array_frozen && + (!conf->barrier || (conf->nr_pending && current->bio_list && - !bio_list_empty(current->bio_list)), + !bio_list_empty(current->bio_list))), conf->resync_lock); conf->nr_waiting--; }@@ -884,8 +886,7 @@ static void freeze_array(struct r1conf *conf, int extra) { /* stop syncio and normal IO and wait for everything to * go quite. - * We increment barrier and nr_waiting, and then - * wait until nr_pending match nr_queued+extra + * We wait until nr_pending match nr_queued+extra * This is called in the context of one normal IO request * that has failed. Thus any sync request that might be pending * will be blocked by nr_pending, and we need to wait for@@ -895,8 +896,7 @@ static void freeze_array(struct r1conf *conf, int extra) * we continue. */ spin_lock_irq(&conf->resync_lock); - conf->barrier++; - conf->nr_waiting++; + conf->array_frozen = 1; wait_event_lock_irq_cmd(conf->wait_barrier, conf->nr_pending == conf->nr_queued+extra, conf->resync_lock,@@ -907,8 +907,7 @@ static void unfreeze_array(struct r1conf *conf) { /* reverse the effect of the freeze */ spin_lock_irq(&conf->resync_lock); - conf->barrier--; - conf->nr_waiting--; + conf->array_frozen = 0; wake_up(&conf->wait_barrier); spin_unlock_irq(&conf->resync_lock); }diff --git a/drivers/md/raid1.h b/drivers/md/raid1.h index 0ff3715..331a98a 100644 --- a/drivers/md/raid1.h +++ b/drivers/md/raid1.h@@ -65,6 +65,7 @@ struct r1conf { int nr_waiting; int nr_queued; int barrier; + int array_frozen; /* Set to 1 if a full sync is needed, (fresh device added). * Cleared when a sync completes. --1.8.1.2