From: Christoph Hellwig <hch@infradead.org> Date: 2021-08-13 07:50:12
On Fri, Aug 13, 2021 at 02:05:10PM +0800, Guoqing Jiang wrote:
From: Guoqing Jiang <redacted>
We can't split bio with more than BIO_MAX_VECS sectors, otherwise the
below call trace was triggered because we could allocate oversized
write behind bio later.
[ 8.097936] bvec_alloc+0x90/0xc0
[ 8.098934] bio_alloc_bioset+0x1b3/0x260
[ 8.099959] raid1_make_request+0x9ce/0xc50 [raid1]
Which bio_alloc_bioset is this? The one in alloc_behind_master_bio?
In which case I think you want to limit the reduction of max_sectors
to just the write behind case, and clearly document what is going on.
In general the size of a bio only depends on the number of vectors, not
the total I/O size. But alloc_behind_master_bio allocates new backing
pages using order 0 allocations, so in this exceptional case the total
size oes actually matter.
While we're at it: this huge memory allocation looks really deadlock
prone.
On Fri, Aug 13, 2021 at 02:05:10PM +0800, Guoqing Jiang wrote:
quoted
From: Guoqing Jiang <redacted>
We can't split bio with more than BIO_MAX_VECS sectors, otherwise the
below call trace was triggered because we could allocate oversized
write behind bio later.
[ 8.097936] bvec_alloc+0x90/0xc0
[ 8.098934] bio_alloc_bioset+0x1b3/0x260
[ 8.099959] raid1_make_request+0x9ce/0xc50 [raid1]
Which bio_alloc_bioset is this? The one in alloc_behind_master_bio?
Yes, it should be the one since bio_clone_fast calls bio_alloc_bioset
with 0 iovecs.
In which case I think you want to limit the reduction of max_sectors
to just the write behind case, and clearly document what is going on.
Ok, thanks.
In general the size of a bio only depends on the number of vectors, not
the total I/O size. But alloc_behind_master_bio allocates new backing
pages using order 0 allocations, so in this exceptional case the total
size oes actually matter.
While we're at it: this huge memory allocation looks really deadlock
prone.
Hmm, let me think more about it, or could you share your thought? 😉
Thanks,
Guoqing
From: Christoph Hellwig <hch@infradead.org> Date: 2021-08-14 07:56:21
On Fri, Aug 13, 2021 at 04:38:59PM +0800, Guoqing Jiang wrote:
Ok, thanks.
quoted
In general the size of a bio only depends on the number of vectors, not
the total I/O size. But alloc_behind_master_bio allocates new backing
pages using order 0 allocations, so in this exceptional case the total
size oes actually matter.
While we're at it: this huge memory allocation looks really deadlock
prone.
Hmm, let me think more about it, or could you share your thought? ????
Well, you'd need a mempool which can fit the max payload of a bio,
that is BIO_MAX_VECS pages.
FYI, this is what I'd do instead of this patch for now. We don't really
need a vetor per sector, just per page. So this limits the I/O
size a little less.
On Sat, Aug 14, 2021 at 08:55:21AM +0100, Christoph Hellwig wrote:
quoted hunk
On Fri, Aug 13, 2021 at 04:38:59PM +0800, Guoqing Jiang wrote:
quoted
Ok, thanks.
quoted
In general the size of a bio only depends on the number of vectors, not
the total I/O size. But alloc_behind_master_bio allocates new backing
pages using order 0 allocations, so in this exceptional case the total
size oes actually matter.
While we're at it: this huge memory allocation looks really deadlock
prone.
Hmm, let me think more about it, or could you share your thought? ????
Well, you'd need a mempool which can fit the max payload of a bio,
that is BIO_MAX_VECS pages.
FYI, this is what I'd do instead of this patch for now. We don't really
need a vetor per sector, just per page. So this limits the I/O
size a little less.
+
if (max_sectors < bio_sectors(bio)) {
struct bio *split = bio_split(bio, max_sectors,
GFP_NOIO, &conf->bio_split);
Here the limit is max single-page vectors, and the above way may not work,
such as:
0 ~ 254: each bvec's length is 512
255: bvec's length is 8192
the total length is just 512*255 + 8192 = 138752 bytes = 271 sectors, but it
still may need 257 bvecs, which can't be allocated via bio_alloc_bioset().
One solution is to add queue limit of max_single_page_bvec, and let
blk_queue_split() handle it.
Thanks,
Ming
Hi Ming and Christoph,
On 8/14/21 4:57 PM, Ming Lei wrote:
On Sat, Aug 14, 2021 at 08:55:21AM +0100, Christoph Hellwig wrote:
quoted
On Fri, Aug 13, 2021 at 04:38:59PM +0800, Guoqing Jiang wrote:
quoted
Ok, thanks.
quoted
In general the size of a bio only depends on the number of vectors, not
the total I/O size. But alloc_behind_master_bio allocates new backing
pages using order 0 allocations, so in this exceptional case the total
size oes actually matter.
While we're at it: this huge memory allocation looks really deadlock
prone.
Hmm, let me think more about it, or could you share your thought? ????
Well, you'd need a mempool which can fit the max payload of a bio,
that is BIO_MAX_VECS pages.
IIUC, the behind bio is allocated from bio_set (mddev->bio_set) which is
allocated in md_run by
call bioset_init, so the mempool (bvec_pool) of this bio_set is created
by biovec_init_pool which
uses global biovec slabs. Do we really need another mempool? Or, there
is no potential deadlock
for this case.
quoted
FYI, this is what I'd do instead of this patch for now. We don't really
need a vetor per sector, just per page. So this limits the I/O
size a little less.
+
if (max_sectors < bio_sectors(bio)) {
struct bio *split = bio_split(bio, max_sectors,
GFP_NOIO, &conf->bio_split);
Here the limit is max single-page vectors, and the above way may not work,
such as:ust splitted and not
0 ~ 254: each bvec's length is 512
255: bvec's length is 8192
the total length is just 512*255 + 8192 = 138752 bytes = 271 sectors, but it
still may need 257 bvecs, which can't be allocated via bio_alloc_bioset().
Thanks for deeper looking! I guess it is because how vcnt is calculated.
One solution is to add queue limit of max_single_page_bvec, and let
blk_queue_split() handle it.
The path (blk_queue_split -> blk_bio_segment_split -> bvec_split_segs)
which respects max_segments
of limit. Do you mean introduce max_single_page_bvec to limit? Then
perform similar checking as for
max_segment.
Thanks,
Guoqing
On Mon, Aug 16, 2021 at 02:27:48PM +0800, Guoqing Jiang wrote:
Hi Ming and Christoph,
On 8/14/21 4:57 PM, Ming Lei wrote:
quoted
On Sat, Aug 14, 2021 at 08:55:21AM +0100, Christoph Hellwig wrote:
quoted
On Fri, Aug 13, 2021 at 04:38:59PM +0800, Guoqing Jiang wrote:
quoted
Ok, thanks.
quoted
In general the size of a bio only depends on the number of vectors, not
the total I/O size. But alloc_behind_master_bio allocates new backing
pages using order 0 allocations, so in this exceptional case the total
size oes actually matter.
While we're at it: this huge memory allocation looks really deadlock
prone.
Hmm, let me think more about it, or could you share your thought? ????
Well, you'd need a mempool which can fit the max payload of a bio,
that is BIO_MAX_VECS pages.
IIUC, the behind bio is allocated from bio_set (mddev->bio_set) which is
allocated in md_run by
call bioset_init, so the mempool (bvec_pool) of this bio_set is created by
biovec_init_pool which
uses global biovec slabs. Do we really need another mempool? Or, there is no
potential deadlock
for this case.
quoted
quoted
FYI, this is what I'd do instead of this patch for now. We don't really
need a vetor per sector, just per page. So this limits the I/O
size a little less.
+
if (max_sectors < bio_sectors(bio)) {
struct bio *split = bio_split(bio, max_sectors,
GFP_NOIO, &conf->bio_split);
Here the limit is max single-page vectors, and the above way may not work,
such as:ust splitted and not
0 ~ 254: each bvec's length is 512
255: bvec's length is 8192
the total length is just 512*255 + 8192 = 138752 bytes = 271 sectors, but it
still may need 257 bvecs, which can't be allocated via bio_alloc_bioset().
Thanks for deeper looking! I guess it is because how vcnt is calculated.
quoted
One solution is to add queue limit of max_single_page_bvec, and let
blk_queue_split() handle it.
The path (blk_queue_split -> blk_bio_segment_split -> bvec_split_segs) which
respects max_segments
of limit. Do you mean introduce max_single_page_bvec to limit? Then perform
similar checking as for
max_segment.
Yes, then the bio is guaranteed to not reach max single-page bvec limit,
just like what __blk_queue_bounce() does.
thanks,
Ming
Yeah, max_sectors is in size units, I messed that up.
quoted
+
if (max_sectors < bio_sectors(bio)) {
struct bio *split = bio_split(bio, max_sectors,
GFP_NOIO, &conf->bio_split);
Here the limit is max single-page vectors, and the above way may not work,
such as:
0 ~ 254: each bvec's length is 512
255: bvec's length is 8192
the total length is just 512*255 + 8192 = 138752 bytes = 271 sectors, but it
still may need 257 bvecs, which can't be allocated via bio_alloc_bioset().
Yes, we still need the rounding magic that alloc_behind_master_bio uses
here.
One solution is to add queue limit of max_single_page_bvec, and let
blk_queue_split() handle it.
Yeah, max_sectors is in size units, I messed that up.
quoted
quoted
+
if (max_sectors < bio_sectors(bio)) {
struct bio *split = bio_split(bio, max_sectors,
GFP_NOIO, &conf->bio_split);
Here the limit is max single-page vectors, and the above way may not work,
such as:
0 ~ 254: each bvec's length is 512
255: bvec's length is 8192
the total length is just 512*255 + 8192 = 138752 bytes = 271 sectors, but it
still may need 257 bvecs, which can't be allocated via bio_alloc_bioset().
Yes, we still need the rounding magic that alloc_behind_master_bio uses
here.
But it is wrong to use max sectors to limit number of bvecs(segments), isn't it?
Thanks,
Ming
From: Christoph Hellwig <hch@infradead.org> Date: 2021-08-17 05:07:26
On Mon, Aug 16, 2021 at 07:40:48PM +0800, Ming Lei wrote:
quoted
quoted
0 ~ 254: each bvec's length is 512
255: bvec's length is 8192
the total length is just 512*255 + 8192 = 138752 bytes = 271 sectors, but it
still may need 257 bvecs, which can't be allocated via bio_alloc_bioset().
Yes, we still need the rounding magic that alloc_behind_master_bio uses
here.
But it is wrong to use max sectors to limit number of bvecs(segments), isn't it?
The raid1 write behind code cares about the size ofa bio it can reach by
adding order 0 pages to it. The bvecs are part of that and I think the
calculation in the patch documents that a well.
On Tue, Aug 17, 2021 at 06:06:12AM +0100, Christoph Hellwig wrote:
On Mon, Aug 16, 2021 at 07:40:48PM +0800, Ming Lei wrote:
quoted
quoted
quoted
0 ~ 254: each bvec's length is 512
255: bvec's length is 8192
the total length is just 512*255 + 8192 = 138752 bytes = 271 sectors, but it
still may need 257 bvecs, which can't be allocated via bio_alloc_bioset().
Yes, we still need the rounding magic that alloc_behind_master_bio uses
here.
But it is wrong to use max sectors to limit number of bvecs(segments), isn't it?
The raid1 write behind code cares about the size ofa bio it can reach by
adding order 0 pages to it. The bvecs are part of that and I think the
calculation in the patch documents that a well.
Thinking of further, your and Guoqing's patch are correct & enough since
bio_copy_data() just copies bytes(sectors) stream from fs bio to the
write behind bio.
Thanks,
Ming
Hi all,
I just had the occasion to test the new patch as landed in arch linux
5.14.7. Unfortunately it does not work for me. Attached you can find a
modification that works for me, though I am not really sure why
write_behind seems not to be set to true on my configuration. If there
is any more data I can provide to help you to investigate, please let me
know.
Thanks for any clues,
Jens
My configuration:
[root@vdr jens]# mdadm --detail -v /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Fri Dec 26 09:50:53 2014
Raid Level : raid1
Array Size : 1953381440 (1862.89 GiB 2000.26 GB)
Used Dev Size : 1953381440 (1862.89 GiB 2000.26 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Sep 24 17:30:51 2021
State : active
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Consistency Policy : bitmap
Name : vdr:0 (local to host vdr)
UUID : 5532ffda:ccbc790f:b50c4959:8f0fd43f
Events : 32805
Number Major Minor RaidDevice State
2 8 33 0 active sync /dev/sdc1
3 8 17 1 active sync /dev/sdb1
[root@vdr jens]# mdadm -X /dev/sdb1
Filename : /dev/sdb1
Magic : 6d746962
Version : 4
UUID : 5532ffda:ccbc790f:b50c4959:8f0fd43f
Events : 32804
Events Cleared : 32804
State : OK
Chunksize : 64 MB
Daemon : 5s flush period
Write Mode : Allow write behind, max 4096
Sync Size : 1953381440 (1862.89 GiB 2000.26 GB)
Bitmap : 29807 bits (chunks), 3 dirty (0.0%)
[root@vdr jens]# mdadm -X /dev/sdc1
Filename : /dev/sdc1
Magic : 6d746962
Version : 4
UUID : 5532ffda:ccbc790f:b50c4959:8f0fd43f
Events : 32804
Events Cleared : 32804
State : OK
Chunksize : 64 MB
Daemon : 5s flush period
Write Mode : Allow write behind, max 4096
Sync Size : 1953381440 (1862.89 GiB 2000.26 GB)
Bitmap : 29807 bits (chunks), 3 dirty (0.0%)
Am 17.08.21 um 14:32 schrieb Ming Lei:
On Tue, Aug 17, 2021 at 06:06:12AM +0100, Christoph Hellwig wrote:
quoted
On Mon, Aug 16, 2021 at 07:40:48PM +0800, Ming Lei wrote:
quoted
quoted
quoted
0 ~ 254: each bvec's length is 512
255: bvec's length is 8192
the total length is just 512*255 + 8192 = 138752 bytes = 271 sectors, but it
still may need 257 bvecs, which can't be allocated via bio_alloc_bioset().
Yes, we still need the rounding magic that alloc_behind_master_bio uses
here.
But it is wrong to use max sectors to limit number of bvecs(segments), isn't it?
The raid1 write behind code cares about the size ofa bio it can reach by
adding order 0 pages to it. The bvecs are part of that and I think the
calculation in the patch documents that a well.
Thinking of further, your and Guoqing's patch are correct & enough since
bio_copy_data() just copies bytes(sectors) stream from fs bio to the
write behind bio.
Thanks,
Ming
Hi all,
I just had the occasion to test the new patch as landed in arch linux
5.14.7. Unfortunately it does not work for me. Attached you can find a
modification that works for me, though I am not really sure why
write_behind seems not to be set to true on my configuration. If there
is any more data I can provide to help you to investigate, please let
me know.
Thanks for the report! As commented in bugzilla, this is because
write-behind
IO still happens even without write-mostly device. I will send new patch
after
you confirm it works.
1. https://bugzilla.kernel.org/show_bug.cgi?id=213181
Thanks,
Guoqing