On Sun, 18 Jan 2015 21:33:50 -0500 (EST) Xiao Ni [off-list ref] wrote:

----- Original Message -----

quoted

From: "Joe Lawrence" <redacted>
To: "Xiao Ni" <redacted>
Cc: "NeilBrown" <redacted>, linux-raid@vger.kernel.org, "Bill Kuzeja" <redacted>
Sent: Friday, January 16, 2015 11:10:31 PM
Subject: Re: RAID1 removing failed disk returns EBUSY

On Fri, 16 Jan 2015 00:20:12 -0500
Xiao Ni [off-list ref] wrote:

quoted

Hi Joe

   Thanks for reminding me. I didn't do that. Now it can remove
   successfully after writing
"idle" to sync_action.

   I thought wrongly that the patch referenced in this mail is fixed for
   the problem.

So it sounds like even with 3.18 and a new mdadm, this bug still
persists?

-- Joe

--

Hi Joe

   I'm a little confused now. Does the patch 45eaf45dfa4850df16bc2e8e7903d89021137f40 from linux-stable
resolve the problem?

   My environment is:

[root@dhcp-12-133 mdadm]# mdadm --version
mdadm - v3.3.2-18-g93d3bd3 - 18th December 2014  (this is the newest upstream)
[root@dhcp-12-133 mdadm]# uname -r
3.18.2


   My steps are:

[root@dhcp-12-133 mdadm]# lsblk 
sdb                       8:16   0 931.5G  0 disk 
└─sdb1                    8:17   0     5G  0 part 
sdc                       8:32   0 186.3G  0 disk 
sdd                       8:48   0 931.5G  0 disk 
└─sdd1                    8:49   0     5G  0 part 
[root@dhcp-12-133 mdadm]# mdadm -CR /dev/md0 -l1 -n2 /dev/sdb1 /dev/sdd1 --assume-clean
mdadm: Note: this array has metadata at the start and
    may not be suitable as a boot device.  If you plan to
    store '/boot' on this device please ensure that
    your boot-loader understands md/v1.x metadata, or use
    --metadata=0.90
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.

   Then I unplug the disk.

[root@dhcp-12-133 mdadm]# lsblk 
sdc                       8:32   0 186.3G  0 disk  
sdd                       8:48   0 931.5G  0 disk  
└─sdd1                    8:49   0     5G  0 part  
  └─md0                   9:0    0     5G  0 raid1 
[root@dhcp-12-133 mdadm]# echo faulty > /sys/block/md0/md/dev-sdb1/state 
[root@dhcp-12-133 mdadm]# echo remove > /sys/block/md0/md/dev-sdb1/state 
-bash: echo: write error: Device or resource busy
[root@dhcp-12-133 mdadm]# echo idle > /sys/block/md0/md/sync_action 
[root@dhcp-12-133 mdadm]# echo remove > /sys/block/md0/md/dev-sdb1/state 

I cannot reproduce this - using linux 3.18.2.  I'd be surprised if mdadm
version affects things.

This error (Device or resoource busy) implies that rdev->raid_disk is >= 0
(tested in state_store()).

->raid_disk is set to -1 by remove_and_add_spares() providing:
  1/ it isn't Blocked (which is very unlikely)
  2/ hot_remove_disk succeeds, which it will if nr_pending is zero, and
  3/ nr_pending is zero.

So it seems most likely that either:
 1/ nr_pending is non-zero, or
 2/ remove_and_add_spares() didn't run.

nr_pending can only get set if IO is generated, and your sequence of steps
don't show any IO.  It is possible that something else (e.g. started by udev)
triggered some IO.  How long that IO can stay pending might depend on exactly
how you unplug the device.
In my tests I used
   echo 1 > /sys/block/sdXX/../../delete
which may have a different effect to what you do.

However the fact that writing 'idle' to sync_action releases the device seems
to suggest the nr_pending has dropped to zero.  So either
  - remove_and_add_spares didn't run, or
  - remove_and_add_spares ran during a small window when nr_pending was
    elevated, and then didn't run again when nr_pending was reduced to zero.

Ahh.... that rings bells....

I have the following patch in the SLES kernel which I have applied to
mainline yet (and given how old it is, that is really slack of me).

Can you apply the following and see if the symptom goes away please?

Thanks,
NeilBrown

From: Hannes Reinecke <hare@suse.de>
Date: Thu, 26 Jul 2012 11:12:18 +0200
Subject: [PATCH] md: wakeup thread upon rdev_dec_pending()

After each call to rdev_dec_pending() we should wakeup the
md thread if the device is found to be faulty.
Otherwise we'll incur heavy delays on failing devices.

Signed-off-by: Neil Brown <redacted>
Signed-off-by: Hannes Reinecke <hare@suse.de>

diff --git a/drivers/md/md.h b/drivers/md/md.h
index 03cec5bdcaae..4cc2f59b2994 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h

@@ -439,13 +439,6 @@ struct mddev {
 	void (*sync_super)(struct mddev *mddev, struct md_rdev *rdev);
 };
 
-static inline void rdev_dec_pending(struct md_rdev *rdev, struct mddev *mddev)
-{
-	int faulty = test_bit(Faulty, &rdev->flags);
-	if (atomic_dec_and_test(&rdev->nr_pending) && faulty)
-		set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
-}
-
 static inline void md_sync_acct(struct block_device *bdev, unsigned long nr_sectors)
 {
 	atomic_add(nr_sectors, &bdev->bd_contains->bd_disk->sync_io);

@@ -624,4 +617,14 @@ static inline int mddev_check_plugged(struct mddev *mddev)
 	return !!blk_check_plugged(md_unplug, mddev,
 				   sizeof(struct blk_plug_cb));
 }
+
+static inline void rdev_dec_pending(struct md_rdev *rdev, struct mddev *mddev)
+{
+	int faulty = test_bit(Faulty, &rdev->flags);
+	if (atomic_dec_and_test(&rdev->nr_pending) && faulty) {
+		set_bit(MD_RECOVERY_NEEDED, &mddev->recovery);
+		md_wakeup_thread(mddev->thread);
+	}
+}
+
 #endif /* _MD_MD_H */