From: Christoph Hellwig <hch@lst.de> Date: 2021-08-18 14:47:38
Hi Jens,
this series does some refactoring and then adds support to return errors
from add_disk (rebasing a patch from Luis). I think that alone is a huge
improvement as it leaves a disk for which add_disk failed in a defined
status, but the real improvement will be actually handling the errors in
the drivers. This series contains two trivial conversions. Luis has
a tree with conversions for all drivers in the tree, which will be fed
incrementally once this goes in. Hopefully we can convert all the
commonly used drivers in this merge window.
This series sits on top of:
"ensure each gendisk always has a request_queue reference v2"
A git tree is available here:
git://git.infradead.org/users/hch/block.git
Gitweb:
http://git.infradead.org/users/hch/block.git/shortlog/refs/heads/add-disk-error-handling
Diffstat:
block/blk-integrity.c | 12 +-
block/blk-sysfs.c | 9 --
block/blk.h | 7 -
block/disk-events.c | 7 -
block/genhd.c | 186 ++++++++++++++++++++++--------------------
drivers/block/null_blk/main.c | 3
drivers/block/virtio_blk.c | 7 +
include/linux/genhd.h | 8 -
8 files changed, 125 insertions(+), 114 deletions(-)
From: Christoph Hellwig <hch@lst.de> Date: 2021-08-18 14:49:04
Add a sanity check to del_gendisk to do nothing when the disk wasn't
successfully added. This papers over the complete lack of add_disk
error handling, which is about to get fixed gradually.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
block/genhd.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
From: Christoph Hellwig <hch@lst.de> Date: 2021-08-18 14:49:46
There is no real reason these should be separate. Also simplify the
groups assignment a bit.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
block/genhd.c | 131 +++++++++++++++++++++++---------------------------
1 file changed, 60 insertions(+), 71 deletions(-)
@@ -409,71 +409,6 @@ static void disk_scan_partitions(struct gendisk *disk)blkdev_put(bdev,FMODE_READ);}-staticvoidregister_disk(structdevice*parent,structgendisk*disk,-conststructattribute_group**groups)-{-structdevice*ddev=disk_to_dev(disk);-interr;--ddev->parent=parent;--dev_set_name(ddev,"%s",disk->disk_name);--/* delay uevents, until we scanned partition table */-dev_set_uevent_suppress(ddev,1);--if(groups){-WARN_ON(ddev->groups);-ddev->groups=groups;-}-if(device_add(ddev))-return;-if(!sysfs_deprecated){-err=sysfs_create_link(block_depr,&ddev->kobj,-kobject_name(&ddev->kobj));-if(err){-device_del(ddev);-return;-}-}--/*-*avoidprobabledeadlockcausedbyallocatingmemorywith-*GFP_KERNELinruntime_resumecallbackofitsallancestor-*devices-*/-pm_runtime_set_memalloc_noio(ddev,true);--disk->part0->bd_holder_dir=-kobject_create_and_add("holders",&ddev->kobj);-disk->slave_dir=kobject_create_and_add("slaves",&ddev->kobj);--/*-*XXX:thisisamess,can'twaitforrealerrorhandlinginadd_disk.-*Makesure->slave_dirisNULLifwefailedsomeoftheregistration-*sothatthecleanupinbd_unlink_disk_holderworksproperly.-*/-if(bd_register_pending_holders(disk)<0){-kobject_put(disk->slave_dir);-disk->slave_dir=NULL;-}--if(disk->flags&GENHD_FL_HIDDEN)-return;--disk_scan_partitions(disk);--/* announce the disk and partitions after all partitions are created */-dev_set_uevent_suppress(ddev,0);-disk_uevent(disk,KOBJ_ADD);--if(disk->bdi->dev){-err=sysfs_create_link(&ddev->kobj,&disk->bdi->dev->kobj,-"bdi");-WARN_ON(err);-}-}-/***device_add_disk-adddiskinformationtokernellist*@parent:parentdeviceforthedisk
From: Christoph Hellwig <hch@lst.de> Date: 2021-08-18 14:50:20
Once bdev_add is called userspace can open the block device. Ensure
that the struct device, which is used for refcounting of the disk
besides various other things, is fully setup at that point.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
block/genhd.c | 36 +++++++++++++++++-------------------
1 file changed, 17 insertions(+), 19 deletions(-)
From: Christoph Hellwig <hch@lst.de> Date: 2021-08-18 14:52:27
Doing all the sysfs file creation before adding the bdev and thus
allowing it to be opened will simplify the about to be added error
handling.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
block/genhd.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
From: Christoph Hellwig <hch@lst.de> Date: 2021-08-18 14:53:26
Ensure that all the sysfs bits are set up before bdev_add is called,
as that will make the upcomding error handling much easier. However
this means the call to disk_update_readahead has to be split as that
requires a bdi. Also remove various sanity checks that don't make
sense now that blk_register_queue only has a single caller.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
block/blk-sysfs.c | 9 ---------
block/genhd.c | 5 +++--
2 files changed, 3 insertions(+), 11 deletions(-)
@@ -856,15 +856,6 @@ int blk_register_queue(struct gendisk *disk)structdevice*dev=disk_to_dev(disk);structrequest_queue*q=disk->queue;-if(WARN_ON(!q))-return-ENXIO;--WARN_ONCE(blk_queue_registered(q),-"%s is registering an already registered queue\n",-kobject_name(&dev->kobj));--disk_update_readahead(disk);-ret=blk_trace_init_sysfs(dev);if(ret)returnret;
From: Christoph Hellwig <hch@lst.de> Date: 2021-08-18 14:56:07
From: Luis Chamberlain <mcgrof@kernel.org>
Properly unwind on errors in device_add_disk. This is the initial work
as drivers are not converted yet, which will follow in separate patches.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
[hch: major rebase. All bugs are probably mine]
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
block/genhd.c | 92 +++++++++++++++++++++++++++----------------
include/linux/genhd.h | 8 ++--
2 files changed, 62 insertions(+), 38 deletions(-)
From: Christoph Hellwig <hch@lst.de> Date: 2021-08-18 14:57:01
From: Luis Chamberlain <mcgrof@kernel.org>
We never checked for errors on add_disk() as this function
returned void. Now that this is fixed, use the shiny new
error handling.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
drivers/block/virtio_blk.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
From: Christoph Hellwig <hch@lst.de> Date: 2021-08-18 14:57:43
From: Luis Chamberlain <mcgrof@kernel.org>
We never checked for errors on add_disk() as this function
returned void. Now that this is fixed, use the shiny new
error handling. The actual cleanup in case of error is
already handled by the caller of null_gendisk_register().
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/block/null_blk/main.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
From: Hannes Reinecke <hare@suse.de> Date: 2021-08-19 10:41:57
On 8/18/21 4:45 PM, Christoph Hellwig wrote:
quoted hunk
Add a sanity check to del_gendisk to do nothing when the disk wasn't
successfully added. This papers over the complete lack of add_disk
error handling, which is about to get fixed gradually.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
block/genhd.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
From: Hannes Reinecke <hare@suse.de> Date: 2021-08-19 10:48:41
On 8/18/21 4:45 PM, Christoph Hellwig wrote:
There is no real reason these should be separate. Also simplify the
groups assignment a bit.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
block/genhd.c | 131 +++++++++++++++++++++++---------------------------
1 file changed, 60 insertions(+), 71 deletions(-)
Reviewed-by: Hannes Reinecke <hare@suse.de>
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
From: Hannes Reinecke <hare@suse.de> Date: 2021-08-20 06:06:47
On 8/18/21 4:45 PM, Christoph Hellwig wrote:
Once bdev_add is called userspace can open the block device. Ensure
that the struct device, which is used for refcounting of the disk
besides various other things, is fully setup at that point.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
block/genhd.c | 36 +++++++++++++++++-------------------
1 file changed, 17 insertions(+), 19 deletions(-)
Reviewed-by: Hannes Reinecke <hare@suse.de>
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
From: Hannes Reinecke <hare@suse.de> Date: 2021-08-20 06:08:59
On 8/18/21 4:45 PM, Christoph Hellwig wrote:
Doing all the sysfs file creation before adding the bdev and thus
allowing it to be opened will simplify the about to be added error
handling.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
block/genhd.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
Reviewed-by: Hannes Reinecke <hare@suse.de>
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
From: Hannes Reinecke <hare@suse.de> Date: 2021-08-20 06:09:47
On 8/18/21 4:45 PM, Christoph Hellwig wrote:
Ensure that all the sysfs bits are set up before bdev_add is called,
as that will make the upcomding error handling much easier. However
this means the call to disk_update_readahead has to be split as that
requires a bdi. Also remove various sanity checks that don't make
sense now that blk_register_queue only has a single caller.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
block/blk-sysfs.c | 9 ---------
block/genhd.c | 5 +++--
2 files changed, 3 insertions(+), 11 deletions(-)
Reviewed-by: Hannes Reinecke <hare@suse.de>
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
From: Hannes Reinecke <hare@suse.de> Date: 2021-08-20 06:12:24
On 8/18/21 4:45 PM, Christoph Hellwig wrote:
From: Luis Chamberlain <mcgrof@kernel.org>
Properly unwind on errors in device_add_disk. This is the initial work
as drivers are not converted yet, which will follow in separate patches.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
[hch: major rebase. All bugs are probably mine]
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
block/genhd.c | 92 +++++++++++++++++++++++++++----------------
include/linux/genhd.h | 8 ++--
2 files changed, 62 insertions(+), 38 deletions(-)
Reviewed-by: Hannes Reinecke <hare@suse.de>
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
From: Hannes Reinecke <hare@suse.de> Date: 2021-08-20 06:12:58
On 8/18/21 4:45 PM, Christoph Hellwig wrote:
From: Luis Chamberlain <mcgrof@kernel.org>
We never checked for errors on add_disk() as this function
returned void. Now that this is fixed, use the shiny new
error handling.
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
---
drivers/block/virtio_blk.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
Reviewed-by: Hannes Reinecke <hare@suse.de>
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
From: Hannes Reinecke <hare@suse.de> Date: 2021-08-20 06:13:25
On 8/18/21 4:45 PM, Christoph Hellwig wrote:
From: Luis Chamberlain <mcgrof@kernel.org>
We never checked for errors on add_disk() as this function
returned void. Now that this is fixed, use the shiny new
error handling. The actual cleanup in case of error is
already handled by the caller of null_gendisk_register().
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
drivers/block/null_blk/main.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
Reviewed-by: Hannes Reinecke <hare@suse.de>
Cheers,
Hannes
--
Dr. Hannes Reinecke Kernel Storage Architect
hare@suse.de +49 911 74053 688
SUSE Software Solutions GmbH, Maxfeldstr. 5, 90409 Nürnberg
HRB 36809 (AG Nürnberg), Geschäftsführer: Felix Imendörffer
Hi Jens,
this series does some refactoring and then adds support to return errors
from add_disk (rebasing a patch from Luis). I think that alone is a huge
improvement as it leaves a disk for which add_disk failed in a defined
status, but the real improvement will be actually handling the errors in
the drivers. This series contains two trivial conversions. Luis has
a tree with conversions for all drivers in the tree, which will be fed
incrementally once this goes in. Hopefully we can convert all the
commonly used drivers in this merge window.
From: Luis Chamberlain <mcgrof@kernel.org> Date: 2021-08-23 19:45:13
On Mon, Aug 23, 2021 at 12:57:41PM -0600, Jens Axboe wrote:
On 8/18/21 8:45 AM, Christoph Hellwig wrote:
quoted
Hi Jens,
this series does some refactoring and then adds support to return errors
from add_disk (rebasing a patch from Luis). I think that alone is a huge
improvement as it leaves a disk for which add_disk failed in a defined
status, but the real improvement will be actually handling the errors in
the drivers. This series contains two trivial conversions. Luis has
a tree with conversions for all drivers in the tree, which will be fed
incrementally once this goes in. Hopefully we can convert all the
commonly used drivers in this merge window.
Applied, thanks.
Do you have a branch published which has this by any chance? I checked
but can't see anything obvious.
Luis
On Mon, Aug 23, 2021 at 12:57:41PM -0600, Jens Axboe wrote:
quoted
On 8/18/21 8:45 AM, Christoph Hellwig wrote:
quoted
Hi Jens,
this series does some refactoring and then adds support to return errors
from add_disk (rebasing a patch from Luis). I think that alone is a huge
improvement as it leaves a disk for which add_disk failed in a defined
status, but the real improvement will be actually handling the errors in
the drivers. This series contains two trivial conversions. Luis has
a tree with conversions for all drivers in the tree, which will be fed
incrementally once this goes in. Hopefully we can convert all the
commonly used drivers in this merge window.
Applied, thanks.
Do you have a branch published which has this by any chance? I checked
but can't see anything obvious.
It's in the core branch, just hadn't been pushed out yet. Now it is,
find it in for-5.15/block
--
Jens Axboe
Hi Christoph,
On 2021/8/18 22:45, Christoph Hellwig wrote:
Once bdev_add is called userspace can open the block device. Ensure
that the struct device, which is used for refcounting of the disk
besides various other things, is fully setup at that point.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Sorry for bring up an old commit.
From my understanding, before this commit, bdev_add() will
be called prior to device_add() so insert_inode_hash()
in bdev_add() will be called before users can see blkdev
in the devtmpfs.
But after this change, blkdev can be seen before
insert_inode_hash(), which opens a race (although I'm not
sure if it's an expected behavior) blkdev_get_no_open() will
find nothing (e.g. mounting) even blkdev in devtmpfs is
there.
Also before commit 22ae8ce8b892 ("block: simplify bdev/disk
lookup in blkdev_get"), the corresponding bd_inode was created
by bdget() so at least it doesn't have such race window too.
One use case is that some userspace applications expect that
once a blkdev is visible in devtmpfs, it can be mounted
immediately. I'm not sure if it's a correct expectation
(or if there is some better way to know when bd_inode is
ready instead of just retring mount operations) or it's
just needed to be fixed.
Such race can be often observed if a virtio-blk device is
hotpluged into a VM and mount immediately at least on
Linux 6.6.
Thanks,
Gao Xiang
From: Christoph Hellwig <hch@lst.de> Date: 2025-10-31 09:09:29
On Fri, Oct 31, 2025 at 03:46:02PM +0800, Gao Xiang wrote:
From my understanding, before this commit, bdev_add() will
be called prior to device_add() so insert_inode_hash()
in bdev_add() will be called before users can see blkdev
in the devtmpfs.
But after this change, blkdev can be seen before
insert_inode_hash(), which opens a race (although I'm not
sure if it's an expected behavior) blkdev_get_no_open() will
find nothing (e.g. mounting) even blkdev in devtmpfs is
there.
We're not supposed to see the uevent notification before the
block device is ready. Do you see that earlier, or do you have
code busy polling for a node?
Hi Christoph,
On 2025/10/31 17:09, Christoph Hellwig wrote:
On Fri, Oct 31, 2025 at 03:46:02PM +0800, Gao Xiang wrote:
quoted
From my understanding, before this commit, bdev_add() will
be called prior to device_add() so insert_inode_hash()
in bdev_add() will be called before users can see blkdev
in the devtmpfs.
But after this change, blkdev can be seen before
insert_inode_hash(), which opens a race (although I'm not
sure if it's an expected behavior) blkdev_get_no_open() will
find nothing (e.g. mounting) even blkdev in devtmpfs is
there.
We're not supposed to see the uevent notification before the
block device is ready.
Thanks for the quick response.
Right, sorry yes, disk_uevent(KOBJ_ADD) is in the end.
Do you see that earlier, or do you have
code busy polling for a node?
Personally I think it will break many userspace programs
(although I also don't think it's a correct expectation.)
After recheck internally, the userspace program logic is:
- stat /dev/vdX;
- if exists, mount directly;
- if non-exists, listen uevent disk_add instead.
Previously, for devtmpfs blkdev files, such stat/mount
assumption is always valid.
After this change, it's only valid for udev generated blkdev
files (because they're generated by uevent). again I'm not
saying it's a good assumption, but I guess many developpers
who don't know this will write such user code, including our
internal users.
Thanks,
Gao Xiang
From: Christoph Hellwig <hch@lst.de> Date: 2025-10-31 09:45:58
On Fri, Oct 31, 2025 at 05:36:45PM +0800, Gao Xiang wrote:
Right, sorry yes, disk_uevent(KOBJ_ADD) is in the end.
quoted
Do you see that earlier, or do you have
code busy polling for a node?
Personally I think it will break many userspace programs
(although I also don't think it's a correct expectation.)
We've had this behavior for a few years, and this is the first report
I've seen.
After recheck internally, the userspace program logic is:
- stat /dev/vdX;
- if exists, mount directly;
- if non-exists, listen uevent disk_add instead.
Previously, for devtmpfs blkdev files, such stat/mount
assumption is always valid.
That assumption doesn't seem wrong. But why does the device node
get created earlier? My assumption was that it would only be
created by the KOBJ_ADD uevent. Adding the device model maintainers
as my little dig through the core drivers/base/ code doesn't find
anything to the contrary, but maybe I don't fully understand it.
On Fri, Oct 31, 2025 at 05:36:45PM +0800, Gao Xiang wrote:
quoted
Right, sorry yes, disk_uevent(KOBJ_ADD) is in the end.
quoted
Do you see that earlier, or do you have
code busy polling for a node?
Personally I think it will break many userspace programs
(although I also don't think it's a correct expectation.)
We've had this behavior for a few years, and this is the first report
I've seen.
quoted
After recheck internally, the userspace program logic is:
- stat /dev/vdX;
- if exists, mount directly;
- if non-exists, listen uevent disk_add instead.
Previously, for devtmpfs blkdev files, such stat/mount
assumption is always valid.
That assumption doesn't seem wrong.
;-) I was thought UNIX mknod doesn't imply the device is
ready or valid in any case (but dev files in devtmpfs
might be an exception but I didn't find some formal words)...
so uevent is clearly a right way, but..
But why does the device node
get created earlier? My assumption was that it would only be
created by the KOBJ_ADD uevent. Adding the device model maintainers
as my little dig through the core drivers/base/ code doesn't find
anything to the contrary, but maybe I don't fully understand it.
AFAIK, device_add() is used to trigger devtmpfs file
creation, and it can be observed if frequently
hotpluging device in the VM and mount. Currently
I don't have time slot to build an easy reproducer,
but I think it's a real issue anyway.
Thanks,
Gao Xiang
On Fri, Oct 31, 2025 at 05:54:10PM +0800, Gao Xiang wrote:
On 2025/10/31 17:45, Christoph Hellwig wrote:
quoted
On Fri, Oct 31, 2025 at 05:36:45PM +0800, Gao Xiang wrote:
quoted
Right, sorry yes, disk_uevent(KOBJ_ADD) is in the end.
quoted
Do you see that earlier, or do you have
code busy polling for a node?
Personally I think it will break many userspace programs
(although I also don't think it's a correct expectation.)
We've had this behavior for a few years, and this is the first report
I've seen.
quoted
After recheck internally, the userspace program logic is:
- stat /dev/vdX;
- if exists, mount directly;
- if non-exists, listen uevent disk_add instead.
Previously, for devtmpfs blkdev files, such stat/mount
assumption is always valid.
That assumption doesn't seem wrong.
;-) I was thought UNIX mknod doesn't imply the device is
ready or valid in any case (but dev files in devtmpfs
might be an exception but I didn't find some formal words)...
so uevent is clearly a right way, but..
Yes, anyone can do a mknod and attempt to open a device that isn't
present.
when devtmpfs creates the device node, it should be there. Unless it
gets removed, and then added back, so you could race with userspace, but
that's not normal.
quoted
But why does the device node
get created earlier? My assumption was that it would only be
created by the KOBJ_ADD uevent. Adding the device model maintainers
as my little dig through the core drivers/base/ code doesn't find
anything to the contrary, but maybe I don't fully understand it.
AFAIK, device_add() is used to trigger devtmpfs file
creation, and it can be observed if frequently
hotpluging device in the VM and mount. Currently
I don't have time slot to build an easy reproducer,
but I think it's a real issue anyway.
As I say above, that's not normal, and you have to be root to do this,
so I don't understand what you are trying to prevent happening? What is
the bug and why is it just showing up now (i.e. what changed to cause
it?)
thanks,
greg k-h
Hi Greg,
On 2025/10/31 17:58, Greg Kroah-Hartman wrote:
On Fri, Oct 31, 2025 at 05:54:10PM +0800, Gao Xiang wrote:
quoted
On 2025/10/31 17:45, Christoph Hellwig wrote:
quoted
On Fri, Oct 31, 2025 at 05:36:45PM +0800, Gao Xiang wrote:
quoted
Right, sorry yes, disk_uevent(KOBJ_ADD) is in the end.
quoted
Do you see that earlier, or do you have
code busy polling for a node?
Personally I think it will break many userspace programs
(although I also don't think it's a correct expectation.)
We've had this behavior for a few years, and this is the first report
I've seen.
quoted
After recheck internally, the userspace program logic is:
- stat /dev/vdX;
- if exists, mount directly;
- if non-exists, listen uevent disk_add instead.
Previously, for devtmpfs blkdev files, such stat/mount
assumption is always valid.
That assumption doesn't seem wrong.
;-) I was thought UNIX mknod doesn't imply the device is
ready or valid in any case (but dev files in devtmpfs
might be an exception but I didn't find some formal words)...
so uevent is clearly a right way, but..
Yes, anyone can do a mknod and attempt to open a device that isn't
present.
when devtmpfs creates the device node, it should be there. Unless it
gets removed, and then added back, so you could race with userspace, but
that's not normal.
quoted
quoted
But why does the device node
get created earlier? My assumption was that it would only be
created by the KOBJ_ADD uevent. Adding the device model maintainers
as my little dig through the core drivers/base/ code doesn't find
anything to the contrary, but maybe I don't fully understand it.
AFAIK, device_add() is used to trigger devtmpfs file
creation, and it can be observed if frequently
hotpluging device in the VM and mount. Currently
I don't have time slot to build an easy reproducer,
but I think it's a real issue anyway.
As I say above, that's not normal, and you have to be root to do this,
Just thinking out if I am a random reporter, I could
report the original symptom now because we face it,
but everyone has his own internal business or even
with limited kernel ability for example, in any
case, there is no such expectation to rush someone
into build a clean reproducer.
Nevertheless, I will take time on the reproducer, and
I think it could just add some artificial delay just
after device_add(). I could try anyway, but no rush.
so I don't understand what you are trying to prevent happening? What is
Hi Greg,
On 2025/10/31 17:58, Greg Kroah-Hartman wrote:
quoted
On Fri, Oct 31, 2025 at 05:54:10PM +0800, Gao Xiang wrote:
quoted
On 2025/10/31 17:45, Christoph Hellwig wrote:
...
quoted
quoted
quoted
But why does the device node
get created earlier? My assumption was that it would only be
created by the KOBJ_ADD uevent. Adding the device model maintainers
as my little dig through the core drivers/base/ code doesn't find
anything to the contrary, but maybe I don't fully understand it.
AFAIK, device_add() is used to trigger devtmpfs file
creation, and it can be observed if frequently
hotpluging device in the VM and mount. Currently
I don't have time slot to build an easy reproducer,
but I think it's a real issue anyway.
As I say above, that's not normal, and you have to be root to do this,
I just spent time to reproduce with dynamic loop devices and
actually it's easy if msleep() is located artificiallly,
the diff as below:
then
# losetup /dev/loop996 test.img & sleep 1; stat /dev/loop996; trace-cmd record -p function_graph mount -t ext4 /dev/loop996 mnt &> /dev/null
It shows
Hi Greg,
On 2025/10/31 17:58, Greg Kroah-Hartman wrote:
quoted
On Fri, Oct 31, 2025 at 05:54:10PM +0800, Gao Xiang wrote:
quoted
On 2025/10/31 17:45, Christoph Hellwig wrote:
...
quoted
quoted
quoted
quoted
But why does the device node
get created earlier? My assumption was that it would only be
created by the KOBJ_ADD uevent. Adding the device model maintainers
as my little dig through the core drivers/base/ code doesn't find
anything to the contrary, but maybe I don't fully understand it.
AFAIK, device_add() is used to trigger devtmpfs file
creation, and it can be observed if frequently
hotpluging device in the VM and mount. Currently
I don't have time slot to build an easy reproducer,
but I think it's a real issue anyway.
As I say above, that's not normal, and you have to be root to do this,
I just spent time to reproduce with dynamic loop devices and
actually it's easy if msleep() is located artificiallly,
the diff as below:
then
# losetup /dev/loop996 test.img & sleep 1; stat /dev/loop996; trace-cmd record -p function_graph mount -t ext4 /dev/loop996 mnt &> /dev/null
It shows
minor revision:
loop_control_ioctl
loop_add
add_disk_fwnode
__add_disk
device_add
devtmpfs_create_node // here create devtmpfs blkdev file, but racy
add_disk_final
bdev_add
insert_inode_hash // just seen by bdev_file_open_by_dev()
disk_uevent(disk, KOBJ_ADD)
I actually think it's enough to explain the root.
Thanks,
Gao Xiang
On Fri, Oct 31, 2025 at 06:12:05PM +0800, Gao Xiang wrote:
Hi Greg,
On 2025/10/31 17:58, Greg Kroah-Hartman wrote:
quoted
On Fri, Oct 31, 2025 at 05:54:10PM +0800, Gao Xiang wrote:
quoted
On 2025/10/31 17:45, Christoph Hellwig wrote:
quoted
On Fri, Oct 31, 2025 at 05:36:45PM +0800, Gao Xiang wrote:
quoted
Right, sorry yes, disk_uevent(KOBJ_ADD) is in the end.
quoted
Do you see that earlier, or do you have
code busy polling for a node?
Personally I think it will break many userspace programs
(although I also don't think it's a correct expectation.)
We've had this behavior for a few years, and this is the first report
I've seen.
quoted
After recheck internally, the userspace program logic is:
- stat /dev/vdX;
- if exists, mount directly;
- if non-exists, listen uevent disk_add instead.
Previously, for devtmpfs blkdev files, such stat/mount
assumption is always valid.
That assumption doesn't seem wrong.
;-) I was thought UNIX mknod doesn't imply the device is
ready or valid in any case (but dev files in devtmpfs
might be an exception but I didn't find some formal words)...
so uevent is clearly a right way, but..
Yes, anyone can do a mknod and attempt to open a device that isn't
present.
when devtmpfs creates the device node, it should be there. Unless it
gets removed, and then added back, so you could race with userspace, but
that's not normal.
quoted
quoted
But why does the device node
get created earlier? My assumption was that it would only be
created by the KOBJ_ADD uevent. Adding the device model maintainers
as my little dig through the core drivers/base/ code doesn't find
anything to the contrary, but maybe I don't fully understand it.
AFAIK, device_add() is used to trigger devtmpfs file
creation, and it can be observed if frequently
hotpluging device in the VM and mount. Currently
I don't have time slot to build an easy reproducer,
but I think it's a real issue anyway.
As I say above, that's not normal, and you have to be root to do this,
Just thinking out if I am a random reporter, I could
report the original symptom now because we face it,
but everyone has his own internal business or even
with limited kernel ability for example, in any
case, there is no such expectation to rush someone
into build a clean reproducer.
Nevertheless, I will take time on the reproducer, and
I think it could just add some artificial delay just
after device_add(). I could try anyway, but no rush.
quoted
so I don't understand what you are trying to prevent happening? What is
On Fri, Oct 31, 2025 at 08:23:32PM +0800, Gao Xiang wrote:
quoted hunk
On 2025/10/31 18:12, Gao Xiang wrote:
quoted
Hi Greg,
On 2025/10/31 17:58, Greg Kroah-Hartman wrote:
quoted
On Fri, Oct 31, 2025 at 05:54:10PM +0800, Gao Xiang wrote:
quoted
On 2025/10/31 17:45, Christoph Hellwig wrote:
...
quoted
quoted
quoted
quoted
But why does the device node
get created earlier? My assumption was that it would only be
created by the KOBJ_ADD uevent. Adding the device model maintainers
as my little dig through the core drivers/base/ code doesn't find
anything to the contrary, but maybe I don't fully understand it.
AFAIK, device_add() is used to trigger devtmpfs file
creation, and it can be observed if frequently
hotpluging device in the VM and mount. Currently
I don't have time slot to build an easy reproducer,
but I think it's a real issue anyway.
As I say above, that's not normal, and you have to be root to do this,
I just spent time to reproduce with dynamic loop devices and
actually it's easy if msleep() is located artificiallly,
the diff as below:
@@ -497,6 +499,9 @@ static int __add_disk(struct device *parent, struct gendisk *disk,if(ret)gotoout_free_ext_minor;+if(disk->major==LOOP_MAJOR)+msleep(2500);// delay 2.5s for all loops+
Yes, so you need to watch for the uevent to happen, THEN it is safe to
access the block device. Doing it before then isn't a good idea :)
But, if you think this is an issue, do you have a patch that passes your
testing to fix it?
thanks,
greg k-h
On Fri, Oct 31, 2025 at 06:12:05PM +0800, Gao Xiang wrote:
quoted
Hi Greg,
On 2025/10/31 17:58, Greg Kroah-Hartman wrote:
quoted
On Fri, Oct 31, 2025 at 05:54:10PM +0800, Gao Xiang wrote:
quoted
On 2025/10/31 17:45, Christoph Hellwig wrote:
quoted
On Fri, Oct 31, 2025 at 05:36:45PM +0800, Gao Xiang wrote:
quoted
Right, sorry yes, disk_uevent(KOBJ_ADD) is in the end.
quoted
Do you see that earlier, or do you have
code busy polling for a node?
Personally I think it will break many userspace programs
(although I also don't think it's a correct expectation.)
We've had this behavior for a few years, and this is the first report
I've seen.
quoted
After recheck internally, the userspace program logic is:
- stat /dev/vdX;
- if exists, mount directly;
- if non-exists, listen uevent disk_add instead.
Previously, for devtmpfs blkdev files, such stat/mount
assumption is always valid.
That assumption doesn't seem wrong.
;-) I was thought UNIX mknod doesn't imply the device is
ready or valid in any case (but dev files in devtmpfs
might be an exception but I didn't find some formal words)...
so uevent is clearly a right way, but..
Yes, anyone can do a mknod and attempt to open a device that isn't
present.
when devtmpfs creates the device node, it should be there. Unless it
gets removed, and then added back, so you could race with userspace, but
that's not normal.
quoted
quoted
But why does the device node
get created earlier? My assumption was that it would only be
created by the KOBJ_ADD uevent. Adding the device model maintainers
as my little dig through the core drivers/base/ code doesn't find
anything to the contrary, but maybe I don't fully understand it.
AFAIK, device_add() is used to trigger devtmpfs file
creation, and it can be observed if frequently
hotpluging device in the VM and mount. Currently
I don't have time slot to build an easy reproducer,
but I think it's a real issue anyway.
As I say above, that's not normal, and you have to be root to do this,
Just thinking out if I am a random reporter, I could
report the original symptom now because we face it,
but everyone has his own internal business or even
with limited kernel ability for example, in any
case, there is no such expectation to rush someone
into build a clean reproducer.
Nevertheless, I will take time on the reproducer, and
I think it could just add some artificial delay just
after device_add(). I could try anyway, but no rush.
quoted
so I don't understand what you are trying to prevent happening? What is
So you see cases where the device node is present, you try to open it,
but yet there is no real block device behind it at all?
Roughly yes, block devices have a pseudo filesystem, briefly
it registered the block device with device_add() so the
devtmpfs file is visible then but bdev_add() is not called yet
so for example, mounting like bdev_file_open_by_dev() cannot
find this and return ENXIO.
quoted
quoted
the bug and why is it just showing up now (i.e. what changed to cause
it?)
I don't know, I think just because 6.6 is a relatively
newer kernel, and most userspace logic has retry logic
to cover this up.
6.6 has been out for 2 years now, this is a long time in kernel
development cycles for things to just start showing up now.
I think for most cases devices are added during boot so
it's hard to find, but in the stress hotplug cases, it
can be observed easily honestly.
Thanks,
Gao Xiang
On Fri, Oct 31, 2025 at 08:23:32PM +0800, Gao Xiang wrote:
quoted
On 2025/10/31 18:12, Gao Xiang wrote:
quoted
Hi Greg,
On 2025/10/31 17:58, Greg Kroah-Hartman wrote:
quoted
On Fri, Oct 31, 2025 at 05:54:10PM +0800, Gao Xiang wrote:
quoted
On 2025/10/31 17:45, Christoph Hellwig wrote:
...
quoted
quoted
quoted
quoted
But why does the device node
get created earlier? My assumption was that it would only be
created by the KOBJ_ADD uevent. Adding the device model maintainers
as my little dig through the core drivers/base/ code doesn't find
anything to the contrary, but maybe I don't fully understand it.
AFAIK, device_add() is used to trigger devtmpfs file
creation, and it can be observed if frequently
hotpluging device in the VM and mount. Currently
I don't have time slot to build an easy reproducer,
but I think it's a real issue anyway.
As I say above, that's not normal, and you have to be root to do this,
I just spent time to reproduce with dynamic loop devices and
actually it's easy if msleep() is located artificiallly,
the diff as below:
@@ -497,6 +499,9 @@ static int __add_disk(struct device *parent, struct gendisk *disk,if(ret)gotoout_free_ext_minor;+if(disk->major==LOOP_MAJOR)+msleep(2500);// delay 2.5s for all loops+
Yes, so you need to watch for the uevent to happen, THEN it is safe to
access the block device. Doing it before then isn't a good idea :)
But, if you think this is an issue, do you have a patch that passes your
testing to fix it?
I just raise it up for some ideas, and this change is
buried into the code refactor and honestly I need to
look into the codebase and related patchsets first.
Currently I have dozens of other development stuffs
on hand, if it's really a regression, I do hope
Christoph or other folks who are familiar with the code
could try to address this.
Thanks,
Gao Xiang
if (ret)
goto out_free_ext_minor;
+ if (disk->major == LOOP_MAJOR)
+ msleep(2500); // delay 2.5s for all loops
+
Yes, so you need to watch for the uevent to happen, THEN it is safe to
access the block device. Doing it before then isn't a good idea :)
But, if you think this is an issue, do you have a patch that passes your
testing to fix it?
I just raise it up for some ideas, and this change is
buried into the code refactor and honestly I need to
look into the codebase and related patchsets first.
Currently I have dozens of other development stuffs
on hand, if it's really a regression, I do hope
Christoph or other folks who are familiar with the code
could try to address this.
I've provided a reproducible way:
https://lore.kernel.org/linux-block/ec8b1c76-c211-49a5-a056-6a147faddd3b@linux.alibaba.com
As the author of these gendisk/bdev enhancement commits, what's
your opinion on this?
In other words, do you think it's a regression, or just a behavior
change but not a regression? Also, a minor confirmation:
if it is a regression on your side, would you like to address it?
Due to further code changes, I proposed a temporary workaround
for our 6.6 kernels as below (I don't think it's clean but we will
do more tests), but due to limited time, currently I don't have
time to come up with a cleaner solution and track this until the
upstream fix lands.
Thanks,
Gao Xiang
block/blk.h | 1 +
block/genhd.c | 18 ++++++++++++++++--
block/partitions/core.c | 6 +++++-
3 files changed, 22 insertions(+), 3 deletions(-)
@@ -376,6 +376,9 @@ static struct block_device *add_partition(struct gendisk *disk, int partno,gotoout_put;}+bdev->bd_inode->i_state|=I_NEW;+bdev_add(bdev,devt);+/* delay uevent until 'holders' subdir is created */dev_set_uevent_suppress(pdev,1);err=device_add(pdev);
@@ -398,7 +401,7 @@ static struct block_device *add_partition(struct gendisk *disk, int partno,err=xa_insert(&disk->part_tbl,partno,bdev,GFP_KERNEL);if(err)gotoout_del;-bdev_add(bdev,devt);+unlock_new_inode(bdev->bd_inode);/* suppress uevent if the disk suppresses it */if(!dev_get_uevent_suppress(ddev))
From: Christian Brauner <brauner@kernel.org> Date: 2025-11-05 12:30:31
On Fri, Oct 31, 2025 at 10:44:53PM +0800, Gao Xiang wrote:
On 2025/10/31 22:34, Greg Kroah-Hartman wrote:
quoted
On Fri, Oct 31, 2025 at 08:23:32PM +0800, Gao Xiang wrote:
quoted
On 2025/10/31 18:12, Gao Xiang wrote:
quoted
Hi Greg,
On 2025/10/31 17:58, Greg Kroah-Hartman wrote:
quoted
On Fri, Oct 31, 2025 at 05:54:10PM +0800, Gao Xiang wrote:
quoted
On 2025/10/31 17:45, Christoph Hellwig wrote:
...
quoted
quoted
quoted
quoted
But why does the device node
get created earlier? My assumption was that it would only be
created by the KOBJ_ADD uevent. Adding the device model maintainers
as my little dig through the core drivers/base/ code doesn't find
anything to the contrary, but maybe I don't fully understand it.
AFAIK, device_add() is used to trigger devtmpfs file
creation, and it can be observed if frequently
hotpluging device in the VM and mount. Currently
I don't have time slot to build an easy reproducer,
but I think it's a real issue anyway.
As I say above, that's not normal, and you have to be root to do this,
I just spent time to reproduce with dynamic loop devices and
actually it's easy if msleep() is located artificiallly,
the diff as below:
@@ -497,6 +499,9 @@ static int __add_disk(struct device *parent, struct gendisk *disk,if(ret)gotoout_free_ext_minor;+if(disk->major==LOOP_MAJOR)+msleep(2500);// delay 2.5s for all loops+
Yes, so you need to watch for the uevent to happen, THEN it is safe to
access the block device. Doing it before then isn't a good idea :)
But, if you think this is an issue, do you have a patch that passes your
testing to fix it?
I just raise it up for some ideas, and this change is
buried into the code refactor and honestly I need to
look into the codebase and related patchsets first.
Currently I have dozens of other development stuffs
on hand, if it's really a regression, I do hope
Christoph or other folks who are familiar with the code
could try to address this.
If it's easy to do without much of a regression or performance risk then
the device node should only show up once the device is actually ready.
It's certainly best-practive to wait for the uevent though.
Hi Christian,
On 2025/11/5 20:30, Christian Brauner wrote:
On Fri, Oct 31, 2025 at 10:44:53PM +0800, Gao Xiang wrote:
quoted
On 2025/10/31 22:34, Greg Kroah-Hartman wrote:
quoted
On Fri, Oct 31, 2025 at 08:23:32PM +0800, Gao Xiang wrote:
quoted
On 2025/10/31 18:12, Gao Xiang wrote:
quoted
Hi Greg,
On 2025/10/31 17:58, Greg Kroah-Hartman wrote:
quoted
On Fri, Oct 31, 2025 at 05:54:10PM +0800, Gao Xiang wrote:
quoted
On 2025/10/31 17:45, Christoph Hellwig wrote:
...
quoted
quoted
quoted
quoted
But why does the device node
get created earlier? My assumption was that it would only be
created by the KOBJ_ADD uevent. Adding the device model maintainers
as my little dig through the core drivers/base/ code doesn't find
anything to the contrary, but maybe I don't fully understand it.
AFAIK, device_add() is used to trigger devtmpfs file
creation, and it can be observed if frequently
hotpluging device in the VM and mount. Currently
I don't have time slot to build an easy reproducer,
but I think it's a real issue anyway.
As I say above, that's not normal, and you have to be root to do this,
I just spent time to reproduce with dynamic loop devices and
actually it's easy if msleep() is located artificiallly,
the diff as below:
@@ -497,6 +499,9 @@ static int __add_disk(struct device *parent, struct gendisk *disk,if(ret)gotoout_free_ext_minor;+if(disk->major==LOOP_MAJOR)+msleep(2500);// delay 2.5s for all loops+
Yes, so you need to watch for the uevent to happen, THEN it is safe to
access the block device. Doing it before then isn't a good idea :)
But, if you think this is an issue, do you have a patch that passes your
testing to fix it?
I just raise it up for some ideas, and this change is
buried into the code refactor and honestly I need to
look into the codebase and related patchsets first.
Currently I have dozens of other development stuffs
on hand, if it's really a regression, I do hope
Christoph or other folks who are familiar with the code
could try to address this.
If it's easy to do without much of a regression or performance risk then
the device node should only show up once the device is actually ready.
Yeah, agreed.
It's certainly best-practive to wait for the uevent though.
Currently our internal applications will try to adapt uevent
detection too, but as a public cloud provider, we may still
need a fallback to avoid users' potential blame on this,
anyway.
Thanks,
Gao Xiang