Our testcase(briefly described as fsstress on dm thin-provisioning which
ext4 see volume size with 100G but actual size 10G) trigger a hungtask
bug since ext4_writepages fall into a infinite loop:
static int ext4_writepages(xxx)
{
...
while (!done && mpd.first_page <= mpd.last_page) {
...
ret = mpage_prepare_extent_to_map(&mpd);
if (!ret) {
...
ret = mpage_map_and_submit_extent(handle,
&mpd,&give_up_on_write);
<----- will return -ENOSPC
...
}
...
if (ret == -ENOSPC && sbi->s_journal) {
<------ we cannot break since we will get ENOSPC forever
jbd2_journal_force_commit_nested(sbi->s_journal);
ret = 0;
continue;
}
...
}
}
Got ENOSPC with follow stack:
...
ext4_ext_map_blocks
ext4_ext_convert_to_initialized
ext4_ext_zeroout
ext4_issue_zeroout
...
submit_bio_wait <-- bio to thinpool will return ENOSPC
'df22291ff0fd ("ext4: Retry block allocation if we have free blocks
left")' add the logic to retry block allcate since we may get free block
after we commit a transaction. But the ENOSPC from thin-provisioning
will confuse ext4, and lead the upper infinite loop. Fix it by convert
the err to EIO.
Signed-off-by: yangerkun <redacted>
---
fs/ext4/inode.c | 3 +++
1 file changed, 3 insertions(+)
From: Jan Kara <jack@suse.cz> Date: 2021-08-04 13:35:35
On Wed 04-08-21 20:50:44, yangerkun wrote:
Our testcase(briefly described as fsstress on dm thin-provisioning which
ext4 see volume size with 100G but actual size 10G) trigger a hungtask
bug since ext4_writepages fall into a infinite loop:
static int ext4_writepages(xxx)
{
...
while (!done && mpd.first_page <= mpd.last_page) {
...
ret = mpage_prepare_extent_to_map(&mpd);
if (!ret) {
...
ret = mpage_map_and_submit_extent(handle,
&mpd,&give_up_on_write);
<----- will return -ENOSPC
...
}
...
if (ret == -ENOSPC && sbi->s_journal) {
<------ we cannot break since we will get ENOSPC forever
jbd2_journal_force_commit_nested(sbi->s_journal);
ret = 0;
continue;
}
...
}
}
Got ENOSPC with follow stack:
...
ext4_ext_map_blocks
ext4_ext_convert_to_initialized
ext4_ext_zeroout
ext4_issue_zeroout
...
submit_bio_wait <-- bio to thinpool will return ENOSPC
'df22291ff0fd ("ext4: Retry block allocation if we have free blocks
left")' add the logic to retry block allcate since we may get free block
after we commit a transaction. But the ENOSPC from thin-provisioning
will confuse ext4, and lead the upper infinite loop. Fix it by convert
the err to EIO.
Signed-off-by: yangerkun <redacted>
Thanks for the patch. As a quick fix for the problem this is probably fine.
But longer term we might need to implement a configurable behavior for this
because just dropping data on the floor (which is what would happen here)
need not be what sysadmin wants and blocking until space is provisioned may be
actually a preferable behavior. Anyway for now feel free to add:
Reviewed-by: Jan Kara <jack@suse.cz>
Honza
On Wed, Aug 04, 2021 at 03:35:29PM +0200, Jan Kara wrote:
On Wed 04-08-21 20:50:44, yangerkun wrote:
quoted
Our testcase(briefly described as fsstress on dm thin-provisioning which
ext4 see volume size with 100G but actual size 10G) trigger a hungtask
bug since ext4_writepages fall into a infinite loop:
Got ENOSPC with follow stack:
...
ext4_ext_map_blocks
ext4_ext_convert_to_initialized
ext4_ext_zeroout
ext4_issue_zeroout
...
submit_bio_wait <-- bio to thinpool will return ENOSPC
Thanks for the patch. As a quick fix for the problem this is probably fine.
But longer term we might need to implement a configurable behavior for this
because just dropping data on the floor (which is what would happen here)
need not be what sysadmin wants and blocking until space is provisioned may be
actually a preferable behavior. Anyway for now feel free to add:
Reviewed-by: Jan Kara <jack@suse.cz>
Hmm, I wonder if this would be a better fix. (Not yet tested, may fry
your file system, etc....) What do folks think?
- Ted
If the underlying storage device is using thin-provisioning, it's
possible for a zeroout operation to return ENOSPC.
Commit df22291ff0fd ("ext4: Retry block allocation if we have free blocks
left") added logic to retry block allocation since we might get free block
after we commit a transaction. But the ENOSPC from thin-provisioning
will confuse ext4, and lead to an infinite loop.
Since using zeroout instead of splitting the extent node is an
optimization, if it fails, we might as well fall back to splitting the
extent node.
Reported-by: yangerkun <redacted>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
---
I've run this through my battery of tests, and it doesn't cause any
regressions. Yangerkun, can you test this and see if this works for
you?
fs/ext4/extents.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
If the underlying storage device is using thin-provisioning, it's
possible for a zeroout operation to return ENOSPC.
Commit df22291ff0fd ("ext4: Retry block allocation if we have free blocks
left") added logic to retry block allocation since we might get free block
after we commit a transaction. But the ENOSPC from thin-provisioning
will confuse ext4, and lead to an infinite loop.
Since using zeroout instead of splitting the extent node is an
optimization, if it fails, we might as well fall back to splitting the
extent node.
Reported-by: yangerkun <redacted>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
---
I've run this through my battery of tests, and it doesn't cause any
regressions. Yangerkun, can you test this and see if this works for
you?
If the underlying storage device is using thin-provisioning, it's
possible for a zeroout operation to return ENOSPC.
Commit df22291ff0fd ("ext4: Retry block allocation if we have free blocks
left") added logic to retry block allocation since we might get free
block
after we commit a transaction. But the ENOSPC from thin-provisioning
will confuse ext4, and lead to an infinite loop.
Since using zeroout instead of splitting the extent node is an
optimization, if it fails, we might as well fall back to splitting the
extent node.
Reported-by: yangerkun <redacted>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
---
I've run this through my battery of tests, and it doesn't cause any
regressions. Yangerkun, can you test this and see if this works for
you?
Will do it.
Thanks for the patch, it can help us to pass the testcase. And after
some review, it's really a better fix for me.
Thanks,
Kun.
From: Jan Kara <jack@suse.cz> Date: 2021-08-16 10:05:53
On Fri 13-08-21 11:18:01, Theodore Ts'o wrote:
On Wed, Aug 04, 2021 at 03:35:29PM +0200, Jan Kara wrote:
quoted
On Wed 04-08-21 20:50:44, yangerkun wrote:
quoted
Our testcase(briefly described as fsstress on dm thin-provisioning which
ext4 see volume size with 100G but actual size 10G) trigger a hungtask
bug since ext4_writepages fall into a infinite loop:
Got ENOSPC with follow stack:
...
ext4_ext_map_blocks
ext4_ext_convert_to_initialized
ext4_ext_zeroout
ext4_issue_zeroout
...
submit_bio_wait <-- bio to thinpool will return ENOSPC
Thanks for the patch. As a quick fix for the problem this is probably fine.
But longer term we might need to implement a configurable behavior for this
because just dropping data on the floor (which is what would happen here)
need not be what sysadmin wants and blocking until space is provisioned may be
actually a preferable behavior. Anyway for now feel free to add:
Reviewed-by: Jan Kara <jack@suse.cz>
Hmm, I wonder if this would be a better fix. (Not yet tested, may fry
your file system, etc....) What do folks think?
Yes, that looks indeed better. I'd note that even splitting extent may fail
due to ENOSPC on thin-provisioned storage but the chances are *much* lower.
Honza
On Mon, Aug 16, 2021 at 12:05:45PM +0200, Jan Kara wrote:
Yes, that looks indeed better. I'd note that even splitting extent may fail
due to ENOSPC on thin-provisioned storage but the chances are *much* lower.
Indeed, any kind of metadata update (updating an inode atime, creating
a new inode, deleting a directory entry, etc.) can fail due to ENOSPC
on thin-provision storage, leading to a potentially corrupted file
system since some writes will succeed, while others won't --- and
that's not a scenario that's super well tested, nor is there much we
can do other than remounting the file system read/only or forcing a
reboot.
But at least we won't throw the kernel into an infinite loop, which is
what yangerkun was reporting...
- Ted
Hi,
Rethink about this problem. Should we consider other place which call
ext4_issue_zeroout? Maybe it can trigger the problem too(in theory, not
really happened)...
How about include follow patch which not only transfer ENOSPC to EIO.
But also stop to overwrite the error return by ext4_ext_insert_extent in
ext4_split_extent_at.
Besides, 308c57ccf431 ("ext4: if zeroout fails fall back to splitting
the extent node") can work together with this patch.
@@ -3243,7 +3245,7 @@ static int ext4_split_extent_at(handle_t *handle, ext4_ext_pblock(&orig_ex)); }- if (!err) {+ if (!ret) { /* update the extent length and mark as
@@ -427,6 +427,9 @@ int ext4_issue_zeroout(struct inode *inode,
ext4_lblk_t lblk, ext4_fsblk_t pblk,
if (ret > 0)
ret = 0;
+ if (ret == -ENOSPC)
+ ret = -EIO;
+
return ret;
}
在 2021/8/14 5:27, Theodore Ts'o 写道:
quoted hunk
If the underlying storage device is using thin-provisioning, it's
possible for a zeroout operation to return ENOSPC.
Commit df22291ff0fd ("ext4: Retry block allocation if we have free blocks
left") added logic to retry block allocation since we might get free block
after we commit a transaction. But the ENOSPC from thin-provisioning
will confuse ext4, and lead to an infinite loop.
Since using zeroout instead of splitting the extent node is an
optimization, if it fails, we might as well fall back to splitting the
extent node.
Reported-by: yangerkun <redacted>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
---
I've run this through my battery of tests, and it doesn't cause any
regressions. Yangerkun, can you test this and see if this works for
you?
fs/ext4/extents.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
From: Jan Kara <jack@suse.cz> Date: 2021-11-23 09:27:43
Hello,
On Sun 26-09-21 19:35:01, yangerkun wrote:
Rethink about this problem. Should we consider other place which call
ext4_issue_zeroout? Maybe it can trigger the problem too(in theory, not
really happened)...
How about include follow patch which not only transfer ENOSPC to EIO. But
also stop to overwrite the error return by ext4_ext_insert_extent in
ext4_split_extent_at.
Besides, 308c57ccf431 ("ext4: if zeroout fails fall back to splitting the
extent node") can work together with this patch.
I've got back to this. The ext4_ext_zeroout() calls in
ext4_split_extent_at() seem to be there as fallback when insertion of a new
extent fails due to ENOSPC / EDQUOT. If even ext4_ext_zeroout(), then I
think returning an error as the code does now is correct and we don't have
much other option. Also we are really running out of disk space so I think
returning ENOSPC is fine. What exact scenario are you afraid of?
Honza
@@ -3243,7 +3245,7 @@ static int ext4_split_extent_at(handle_t *handle, ext4_ext_pblock(&orig_ex)); }- if (!err) {+ if (!ret) { /* update the extent length and mark as initialized
@@ -427,6 +427,9 @@ int ext4_issue_zeroout(struct inode *inode, ext4_lblk_t
lblk, ext4_fsblk_t pblk,
if (ret > 0)
ret = 0;
+ if (ret == -ENOSPC)
+ ret = -EIO;
+
return ret;
}
在 2021/8/14 5:27, Theodore Ts'o 写道:
quoted
If the underlying storage device is using thin-provisioning, it's
possible for a zeroout operation to return ENOSPC.
Commit df22291ff0fd ("ext4: Retry block allocation if we have free blocks
left") added logic to retry block allocation since we might get free block
after we commit a transaction. But the ENOSPC from thin-provisioning
will confuse ext4, and lead to an infinite loop.
Since using zeroout instead of splitting the extent node is an
optimization, if it fails, we might as well fall back to splitting the
extent node.
Reported-by: yangerkun <redacted>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
---
I've run this through my battery of tests, and it doesn't cause any
regressions. Yangerkun, can you test this and see if this works for
you?
fs/ext4/extents.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
Rethink about this problem. Should we consider other place which call
ext4_issue_zeroout? Maybe it can trigger the problem too(in theory, not
really happened)...
How about include follow patch which not only transfer ENOSPC to EIO. But
also stop to overwrite the error return by ext4_ext_insert_extent in
ext4_split_extent_at.
Besides, 308c57ccf431 ("ext4: if zeroout fails fall back to splitting the
extent node") can work together with this patch.
I've got back to this. The ext4_ext_zeroout() calls in
ext4_split_extent_at() seem to be there as fallback when insertion of a new
extent fails due to ENOSPC / EDQUOT. If even ext4_ext_zeroout(), then I
think returning an error as the code does now is correct and we don't have
much other option. Also we are really running out of disk space so I think
returning ENOSPC is fine. What exact scenario are you afraid of?
I am afraid about the EDQUOT from ext4_ext_insert_extent may be
overwrite by ext4_ext_zeroout with ENOSPC. And this may lead to dead
loop since ext4_writepages will retry once get ENOSPC? Maybe I am wrong...
@@ -3243,7 +3245,7 @@ static int ext4_split_extent_at(handle_t *handle, ext4_ext_pblock(&orig_ex)); }- if (!err) {+ if (!ret) { /* update the extent length and mark as initialized
@@ -427,6 +427,9 @@ int ext4_issue_zeroout(struct inode *inode, ext4_lblk_t
lblk, ext4_fsblk_t pblk,
if (ret > 0)
ret = 0;
+ if (ret == -ENOSPC)
+ ret = -EIO;
+
return ret;
}
在 2021/8/14 5:27, Theodore Ts'o 写道:
quoted
If the underlying storage device is using thin-provisioning, it's
possible for a zeroout operation to return ENOSPC.
Commit df22291ff0fd ("ext4: Retry block allocation if we have free blocks
left") added logic to retry block allocation since we might get free block
after we commit a transaction. But the ENOSPC from thin-provisioning
will confuse ext4, and lead to an infinite loop.
Since using zeroout instead of splitting the extent node is an
optimization, if it fails, we might as well fall back to splitting the
extent node.
Reported-by: yangerkun <redacted>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
---
I've run this through my battery of tests, and it doesn't cause any
regressions. Yangerkun, can you test this and see if this works for
you?
fs/ext4/extents.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
From: Jan Kara <jack@suse.cz> Date: 2021-11-24 10:37:44
On Wed 24-11-21 17:01:12, yangerkun wrote:
On 2021/11/23 17:27, Jan Kara wrote:
quoted
Hello,
On Sun 26-09-21 19:35:01, yangerkun wrote:
quoted
Rethink about this problem. Should we consider other place which call
ext4_issue_zeroout? Maybe it can trigger the problem too(in theory, not
really happened)...
How about include follow patch which not only transfer ENOSPC to EIO. But
also stop to overwrite the error return by ext4_ext_insert_extent in
ext4_split_extent_at.
Besides, 308c57ccf431 ("ext4: if zeroout fails fall back to splitting the
extent node") can work together with this patch.
I've got back to this. The ext4_ext_zeroout() calls in
ext4_split_extent_at() seem to be there as fallback when insertion of a new
extent fails due to ENOSPC / EDQUOT. If even ext4_ext_zeroout(), then I
think returning an error as the code does now is correct and we don't have
much other option. Also we are really running out of disk space so I think
returning ENOSPC is fine. What exact scenario are you afraid of?
I am afraid about the EDQUOT from ext4_ext_insert_extent may be overwrite by
ext4_ext_zeroout with ENOSPC. And this may lead to dead loop since
ext4_writepages will retry once get ENOSPC? Maybe I am wrong...
OK, so passing back original error instead of the error from
ext4_ext_zeroout() makes sense. But I don't think doing much more is needed
- firstly, ENOSPC or EDQUOT should not happen in ext4_split_extent_at()
called from ext4_writepages() because we should have reserved enough
space for extent splits when writing data. So hitting that is already
unexpected. Committing transaction holding blocks that are expected to be
free is the most likely reason for us seeing ENOSPC and returning EIO in
that case would be bug. Secondly, returning EIO instead of ENOSPC is IMO a
bit confusing for upper layers and makes it harder to analyze where the
real problem is...
Honza
--
Jan Kara [off-list ref]
SUSE Labs, CR
Rethink about this problem. Should we consider other place which call
ext4_issue_zeroout? Maybe it can trigger the problem too(in theory, not
really happened)...
How about include follow patch which not only transfer ENOSPC to EIO. But
also stop to overwrite the error return by ext4_ext_insert_extent in
ext4_split_extent_at.
Besides, 308c57ccf431 ("ext4: if zeroout fails fall back to splitting the
extent node") can work together with this patch.
I've got back to this. The ext4_ext_zeroout() calls in
ext4_split_extent_at() seem to be there as fallback when insertion of a new
extent fails due to ENOSPC / EDQUOT. If even ext4_ext_zeroout(), then I
think returning an error as the code does now is correct and we don't have
much other option. Also we are really running out of disk space so I think
returning ENOSPC is fine. What exact scenario are you afraid of?
I am afraid about the EDQUOT from ext4_ext_insert_extent may be overwrite by
ext4_ext_zeroout with ENOSPC. And this may lead to dead loop since
ext4_writepages will retry once get ENOSPC? Maybe I am wrong...
OK, so passing back original error instead of the error from
ext4_ext_zeroout() makes sense. But I don't think doing much more is needed
- firstly, ENOSPC or EDQUOT should not happen in ext4_split_extent_at()
called from ext4_writepages() because we should have reserved enough
space for extent splits when writing data. So hitting that is already
ext4_da_write_begin
ext4_da_get_block_prep
ext4_insert_delayed_block
ext4_da_reserve_space
It seems we will only reserve space for data, no for metadata...
unexpected. Committing transaction holding blocks that are expected to be
free is the most likely reason for us seeing ENOSPC and returning EIO in
that case would be bug.
Agree. EIO from ext4_ext_zeroout that overwrite the ENOSPC from
ext4_ext_insert_extent seems buggy too. Maybe we should ignore the error
from ext4_ext_zeroout and return the error from ext4_ext_insert_extent
once ext4_ext_zeroout in ext4_split_extent_at got a error. Something
like this:
@@ -3278,7 +3279,7 @@ static int ext4_split_extent_at(handle_t *handle, ext4_ext_pblock(&orig_ex)); }- if (!err) {+ if (!err1) { /* update the extent length and mark as
From: Jan Kara <jack@suse.cz> Date: 2021-11-24 17:15:17
On Wed 24-11-21 20:11:43, yangerkun wrote:
On 2021/11/24 18:37, Jan Kara wrote:
quoted
On Wed 24-11-21 17:01:12, yangerkun wrote:
quoted
On 2021/11/23 17:27, Jan Kara wrote:
quoted
Hello,
On Sun 26-09-21 19:35:01, yangerkun wrote:
quoted
Rethink about this problem. Should we consider other place which call
ext4_issue_zeroout? Maybe it can trigger the problem too(in theory, not
really happened)...
How about include follow patch which not only transfer ENOSPC to EIO. But
also stop to overwrite the error return by ext4_ext_insert_extent in
ext4_split_extent_at.
Besides, 308c57ccf431 ("ext4: if zeroout fails fall back to splitting the
extent node") can work together with this patch.
I've got back to this. The ext4_ext_zeroout() calls in
ext4_split_extent_at() seem to be there as fallback when insertion of a new
extent fails due to ENOSPC / EDQUOT. If even ext4_ext_zeroout(), then I
think returning an error as the code does now is correct and we don't have
much other option. Also we are really running out of disk space so I think
returning ENOSPC is fine. What exact scenario are you afraid of?
I am afraid about the EDQUOT from ext4_ext_insert_extent may be overwrite by
ext4_ext_zeroout with ENOSPC. And this may lead to dead loop since
ext4_writepages will retry once get ENOSPC? Maybe I am wrong...
OK, so passing back original error instead of the error from
ext4_ext_zeroout() makes sense. But I don't think doing much more is needed
- firstly, ENOSPC or EDQUOT should not happen in ext4_split_extent_at()
called from ext4_writepages() because we should have reserved enough
space for extent splits when writing data. So hitting that is already
ext4_da_write_begin
ext4_da_get_block_prep
ext4_insert_delayed_block
ext4_da_reserve_space
It seems we will only reserve space for data, no for metadata...
quoted
unexpected. Committing transaction holding blocks that are expected to be
free is the most likely reason for us seeing ENOSPC and returning EIO in
that case would be bug.
Agree. EIO from ext4_ext_zeroout that overwrite the ENOSPC from
ext4_ext_insert_extent seems buggy too. Maybe we should ignore the error
from ext4_ext_zeroout and return the error from ext4_ext_insert_extent
once ext4_ext_zeroout in ext4_split_extent_at got a error. Something
like this:
@@ -3278,7 +3279,7 @@ static int ext4_split_extent_at(handle_t *handle, ext4_ext_pblock(&orig_ex)); }- if (!err) {+ if (!err1) { /* update the extent length and mark as initialized