The new WARN_ON_ONCE is only an additional logic check. When it
triggers it also should trigger when you undo the rest of the change.
But when it triggers something with the driver logic must be off.
(Or my understanding of the intent of the code is worse than assumed:-)
Looking into the d_ref logic I see two additional problems not addressed
by the original patch when sg_add_sfp() fails:
1) sg_open() is then also calling first scsi_device_put() and then
sg_device_destroy() via kref_put(). That's the wrong order.
2) When sg_add_sfp() fails we never call kref_get(&sdp->d_ref).
Thus we shoud not call kref_get() here at all.
Thus your warning above could be triggered by an error within
sg_add_sfp(): In that case d_ref would already be zero when the code
gets to the warning.
Can you check the debug patch below and provide output?
When I'm right the warning should be gone and you should just get the
"Modification triggered" instead. When I'm wrong we should at least see,
how many references d_ref has left.
Alexander
---
drivers/scsi/sg.c | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
From: Sachin Sant <hidden> Date: 2024-03-29 14:37:55
Can you check the debug patch below and provide output?
When I'm right the warning should be gone and you should just get the
"Modification triggered" instead. When I'm wrong we should at least see,
how many references d_ref has left.
From: Alexander Wetzel <hidden> Date: 2024-04-01 09:57:27
commit 27f58c04a8f4 ("scsi: sg: Avoid sg device teardown race")
introduced an incorrect WARN_ON_ONCE() and missed a sequence where
sg_device_destroy() after scsi_device_put() when handling errors.
sg_device_destroy() is accessing the parent scsi_device request_queue which
will already be set to NULL when the preceding call to scsi_device_put()
removed the last reference to the parent scsi_device.
Drop the incorrect WARN_ON_ONCE() - allowing more than one concurrent
access to the sg device - and make sure sg_device_destroy() is not used
after scsi_device_put() in the error handling.
Link: https://lore.kernel.org/all/5375B275-D137-4D5F-BE25-6AF8ACAE41EF@linux.ibm.com
Fixes: 27f58c04a8f4 ("scsi: sg: Avoid sg device teardown race")
Cc: stable@vger.kernel.org
Signed-off-by: Alexander Wetzel <redacted>
---
The WARN_ON_ONCE() was kind of stupid to add:
We get add reference for each sg_open(). So opening a second session and
then closing either one will trigger the warning... Nothing to warn
about here.
Alexander
---
drivers/scsi/sg.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
From: Alexander Wetzel <hidden> Date: 2024-04-01 10:03:27
commit 27f58c04a8f4 ("scsi: sg: Avoid sg device teardown race")
introduced an incorrect WARN_ON_ONCE() and missed a sequence where
sg_device_destroy() was used after scsi_device_put().
sg_device_destroy() is accessing the parent scsi_device request_queue which
will already be set to NULL when the preceding call to scsi_device_put()
removed the last reference to the parent scsi_device.
Drop the incorrect WARN_ON_ONCE() - allowing more than one concurrent
access to the sg device - and make sure sg_device_destroy() is not used
after scsi_device_put() in the error handling.
Link: https://lore.kernel.org/all/5375B275-D137-4D5F-BE25-6AF8ACAE41EF@linux.ibm.com
Fixes: 27f58c04a8f4 ("scsi: sg: Avoid sg device teardown race")
Cc: stable@vger.kernel.org
Signed-off-by: Alexander Wetzel <redacted>
---
Changes compared to V1: fixed commit message
The WARN_ON_ONCE() was kind of stupid to add:
We get add reference for each sg_open(). So opening a second session and
then closing either one will trigger the warning... Nothing to warn
about here.
Alexander
---
drivers/scsi/sg.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
From: Bart Van Assche <bvanassche@acm.org> Date: 2024-04-01 17:10:09
On 4/1/24 03:03, Alexander Wetzel wrote:
commit 27f58c04a8f4 ("scsi: sg: Avoid sg device teardown race")
introduced an incorrect WARN_ON_ONCE() and missed a sequence where
sg_device_destroy() was used after scsi_device_put().
Isn't that too negative? I think that the WARN_ON_ONCE() mentioned above
has proven to be useful: it helped to catch a bug.
sg_device_destroy() is accessing the parent scsi_device request_queue which
will already be set to NULL when the preceding call to scsi_device_put()
removed the last reference to the parent scsi_device.
Drop the incorrect WARN_ON_ONCE() - allowing more than one concurrent
access to the sg device - and make sure sg_device_destroy() is not used
after scsi_device_put() in the error handling.
Link: https://lore.kernel.org/all/5375B275-D137-4D5F-BE25-6AF8ACAE41EF@linux.ibm.com
Fixes: 27f58c04a8f4 ("scsi: sg: Avoid sg device teardown race")
The "goto sg_put" removed by this patch was introduced by commit
cc833acbee9d ("sg: O_EXCL and other lock handling"). Since the latter
commit is older than the one mentioned above, shouldn't the Fixes tag
refer to the latter commit?
From: Alexander Wetzel <hidden> Date: 2024-04-01 19:01:59
On 01.04.24 19:09, Bart Van Assche wrote:
On 4/1/24 03:03, Alexander Wetzel wrote:
quoted
commit 27f58c04a8f4 ("scsi: sg: Avoid sg device teardown race")
introduced an incorrect WARN_ON_ONCE() and missed a sequence where
sg_device_destroy() was used after scsi_device_put().
Isn't that too negative? I think that the WARN_ON_ONCE() mentioned above
has proven to be useful: it helped to catch a bug.
It helped to find the other issue. But the WARN_ON_ONCE() here is still
plain wrong. Any only explained by my lack of understanding of the code
and stupid assumptions I should have checked a bit more.
The warning always triggers when we have more than one user of the sg
device (and then free one of them).
While trying to understand the issue I tripped over the other wrong
sequence. Which is probably very seldom really executed...
That said I can of course update the wording when you have a better
suggestion. But I only have some variations of "Ups... sorry. I thought
that was a good idea. Turns out it's not"
quoted
sg_device_destroy() is accessing the parent scsi_device request_queue
which
will already be set to NULL when the preceding call to scsi_device_put()
removed the last reference to the parent scsi_device.
Drop the incorrect WARN_ON_ONCE() - allowing more than one concurrent
access to the sg device - and make sure sg_device_destroy() is not used
after scsi_device_put() in the error handling.
Link:
https://lore.kernel.org/all/5375B275-D137-4D5F-BE25-6AF8ACAE41EF@linux.ibm.com
Fixes: 27f58c04a8f4 ("scsi: sg: Avoid sg device teardown race")
The "goto sg_put" removed by this patch was introduced by commit
cc833acbee9d ("sg: O_EXCL and other lock handling"). Since the latter
commit is older than the one mentioned above, shouldn't the Fixes tag
refer to the latter commit?
The order was not wrong till commit db59133e9279 ("scsi: sg: fix
blktrace debugfs entries leakage"), the one my original patch tried to
fix. Prior to that one sg_device_destroy() was not using the scsi device
request_queue.
I guess I (or one maintainer) could add that commit here again, too...
My reasoning here is, that this patch here fixes what my first patch got
wrong.
Which is already heading into the stable trees. And I would prefer to
not have any kernel release with commit 27f58c04a8f4 ("scsi: sg: Avoid
sg device teardown race") without this fix, too.
Please add a comment above "return retval" that explains which code will
drop the sg reference.
Hm, don't get that. That kref_put() is the one dropping the reference.
The matching kref_get() is in sg_add_sfp(). Which is called a few lines
prior to the code here (line 350).
The patch is literally only swapping the order of scsi_device_put() and
kref_put().
Which *again* causes a use-after free. So I'll send out v3 immediately
and if any of the thinks discussed here require a v4 we'll do that.
Alexander
From: Alexander Wetzel <hidden> Date: 2024-04-01 19:11:01
commit 27f58c04a8f4 ("scsi: sg: Avoid sg device teardown race")
introduced an incorrect WARN_ON_ONCE() and missed a sequence where
sg_device_destroy() was used after scsi_device_put().
sg_device_destroy() is accessing the parent scsi_device request_queue which
will already be set to NULL when the preceding call to scsi_device_put()
removed the last reference to the parent scsi_device.
Drop the incorrect WARN_ON_ONCE() - allowing more than one concurrent
access to the sg device - and make sure sg_device_destroy() is not used
after scsi_device_put() in the error handling.
Link: https://lore.kernel.org/all/5375B275-D137-4D5F-BE25-6AF8ACAE41EF@linux.ibm.com
Fixes: 27f58c04a8f4 ("scsi: sg: Avoid sg device teardown race")
Cc: stable@vger.kernel.org
Signed-off-by: Alexander Wetzel <redacted>
---
Changes compared to V1: fixed commit message
Changes compared to V2: Fix use-after free
---
drivers/scsi/sg.c | 18 ++++++++++--------
1 file changed, 10 insertions(+), 8 deletions(-)
@@ -301,11 +302,12 @@ sg_open(struct inode *inode, struct file *filp)/* This driver's module count bumped by fops_get in <linux/fs.h> *//* Prevent the device driver from vanishing while we sleep */-retval=scsi_device_get(sdp->device);+device=sdp->device;+retval=scsi_device_get(device);if(retval)gotosg_put;-retval=scsi_autopm_get_device(sdp->device);+retval=scsi_autopm_get_device(device);if(retval)gotosdp_put;
@@ -313,7 +315,7 @@ sg_open(struct inode *inode, struct file *filp)*checkifO_NONBLOCK.PermitsSCSIcommandstobeissued*duringerrorrecovery.Treadcarefully.*/if(!((flags&O_NONBLOCK)||-scsi_block_when_processing_errors(sdp->device))){+scsi_block_when_processing_errors(device))){retval=-ENXIO;/* we are in error recovery for this device */gotoerror_out;
From: Sachin Sant <hidden> Date: 2024-04-02 06:02:14
On 2 Apr 2024, at 12:40 AM, Alexander Wetzel [off-list ref] wrote:
commit 27f58c04a8f4 ("scsi: sg: Avoid sg device teardown race")
introduced an incorrect WARN_ON_ONCE() and missed a sequence where
sg_device_destroy() was used after scsi_device_put().
sg_device_destroy() is accessing the parent scsi_device request_queue which
will already be set to NULL when the preceding call to scsi_device_put()
removed the last reference to the parent scsi_device.
Drop the incorrect WARN_ON_ONCE() - allowing more than one concurrent
access to the sg device - and make sure sg_device_destroy() is not used
after scsi_device_put() in the error handling.
Link: https://lore.kernel.org/all/5375B275-D137-4D5F-BE25-6AF8ACAE41EF@linux.ibm.com
Fixes: 27f58c04a8f4 ("scsi: sg: Avoid sg device teardown race")
Cc: stable@vger.kernel.org
Signed-off-by: Alexander Wetzel <redacted>
---
Thanks for the fix. I tested this patch and confirm it fixes the reported problem.
Tested-by: Sachin Sant <redacted>
— Sachin
/* This driver's module count bumped by fops_get in <linux/fs.h> */
/* Prevent the device driver from vanishing while we sleep */
- retval = scsi_device_get(sdp->device);
+ device = sdp->device;
+ retval = scsi_device_get(device);
if (retval)
goto sg_put;
Are all the sdp->device -> device changes essential? Isn't there a
preference to minimize patches that will end up in the stable trees?
Only the very last change is essential:
- scsi_device_put(sdp->device);
- goto sg_put;
+ kref_put(&sdp->d_ref, sg_device_destroy);
+ scsi_device_put(device);
+ return retval;
Not using a (required) local variable and de-referencing it again and
looks strange for anyone reading the code. While the additional lines in
the patch are trivial to review...
Alexander
From: Bart Van Assche <bvanassche@acm.org> Date: 2024-04-04 16:34:28
On 4/1/24 12:10, Alexander Wetzel wrote:
commit 27f58c04a8f4 ("scsi: sg: Avoid sg device teardown race")
introduced an incorrect WARN_ON_ONCE() and missed a sequence where
sg_device_destroy() was used after scsi_device_put().
sg_device_destroy() is accessing the parent scsi_device request_queue which
will already be set to NULL when the preceding call to scsi_device_put()
removed the last reference to the parent scsi_device.
Drop the incorrect WARN_ON_ONCE() - allowing more than one concurrent
access to the sg device - and make sure sg_device_destroy() is not used
after scsi_device_put() in the error handling.
From: Martin K. Petersen <hidden> Date: 2024-04-06 01:59:38
On Mon, 01 Apr 2024 21:10:38 +0200, Alexander Wetzel wrote:
commit 27f58c04a8f4 ("scsi: sg: Avoid sg device teardown race")
introduced an incorrect WARN_ON_ONCE() and missed a sequence where
sg_device_destroy() was used after scsi_device_put().
sg_device_destroy() is accessing the parent scsi_device request_queue which
will already be set to NULL when the preceding call to scsi_device_put()
removed the last reference to the parent scsi_device.
[...]
Applied to 6.9/scsi-fixes, thanks!
[1/1] scsi: sg: Avoid race in error handling & drop bogus warn
https://git.kernel.org/mkp/scsi/c/d4e655c49f47
--
Martin K. Petersen Oracle Linux Engineering