Bug in vm_map_ram - it does an unconditional GFP_KERNEL allocation
here, and we are in a GFP_NOFS context. We can't pass a gfp_mask to
vm_map_ram(), so until vm_map_ram() grows that we can't fix it...
This trivial patch should fix it.
The only behavior change is the XFS part:
@@ -406,7 +406,7 @@ _xfs_buf_map_pages( do { bp->b_addr = vm_map_ram(bp->b_pages, bp->b_page_count,- -1, PAGE_KERNEL);+ -1, GFP_NOFS, PAGE_KERNEL); if (bp->b_addr) break; vm_unmap_aliases();
Does that look fine to you?
Thanks,
Fengguang
---
From 7301975d3211da2ce07723c294cf3260229fe84b Mon Sep 17 00:00:00 2001
@@ -209,6 +209,7 @@ static void *vb2_dma_sg_vaddr(void *buf_priv)buf->vaddr=vm_map_ram(buf->pages,buf->sg_desc.num_pages,-1,+GFP_KERNEL,PAGE_KERNEL);/* add offset in case userptr is not page-aligned */
Bug in vm_map_ram - it does an unconditional GFP_KERNEL allocation
here, and we are in a GFP_NOFS context. We can't pass a gfp_mask to
vm_map_ram(), so until vm_map_ram() grows that we can't fix it...
This trivial patch should fix it.
The only behavior change is the XFS part:
@@ -406,7 +406,7 @@ _xfs_buf_map_pages( do { bp->b_addr = vm_map_ram(bp->b_pages, bp->b_page_count,- -1, PAGE_KERNEL);+ -1, GFP_NOFS, PAGE_KERNEL); if (bp->b_addr) break; vm_unmap_aliases();
@@ -209,6 +209,7 @@ static void *vb2_dma_sg_vaddr(void *buf_priv)buf->vaddr=vm_map_ram(buf->pages,buf->sg_desc.num_pages,-1,+GFP_KERNEL,PAGE_KERNEL);/* add offset in case userptr is not page-aligned */
It shouldn't work because vmap_page_range still can allocate GFP_KERNEL by pud_alloc in vmap_pud_range.
For it, I tried [1] but other mm guys want to add WARNING [2] so let's avoiding gfp context passing.
[1] https://lkml.org/lkml/2012/4/23/77
[2] https://lkml.org/lkml/2012/5/2/340
Thanks.
--
Kind regards,
Minchan Kim
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
From: Dave Chinner <david@fromorbit.com> Date: 2012-06-14 02:15:47
On Thu, Jun 14, 2012 at 09:49:02AM +0800, Fengguang Wu wrote:
quoted hunk
On Thu, Jun 14, 2012 at 11:20:26AM +1000, Dave Chinner wrote:
quoted
Bug in vm_map_ram - it does an unconditional GFP_KERNEL allocation
here, and we are in a GFP_NOFS context. We can't pass a gfp_mask to
vm_map_ram(), so until vm_map_ram() grows that we can't fix it...
This trivial patch should fix it.
The only behavior change is the XFS part:
This function isn't always called in GFP_NOFS context - readahead
uses different memory allocation semantics (no retry, no warn), so
there are flags that tell it what to do. i.e.
- -1, PAGE_KERNEL);
+ -1, xb_to_gfp(flags), PAGE_KERNEL);
Otherwise looks fine to me...
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
Hello, guys.
On Thu, Jun 14, 2012 at 11:07:53AM +0900, Minchan Kim wrote:
It shouldn't work because vmap_page_range still can allocate
GFP_KERNEL by pud_alloc in vmap_pud_range. For it, I tried [1] but
other mm guys want to add WARNING [2] so let's avoiding gfp context
passing.
[1] https://lkml.org/lkml/2012/4/23/77
[2] https://lkml.org/lkml/2012/5/2/340
Yeah, vmalloc area doesn't support !GFP_KERNEL allocations and as
Minchan said, changing this would require updating page table
allocation functions on all archs. This is the same reason why percpu
allocator doesn't support !GFP_KERNEL allocations which in turn made
blk-throttle implement its own private percpu pool.
If xfs can't live without GFP_NOFS vmalloc allocations, either it has
to implement its own pool or maybe it's time to implement !GFP_KERNEL
allocs for vmalloc area. I don't know.
Thanks.
--
tejun
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
From: Minchan Kim <minchan@kernel.org> Date: 2012-06-14 02:39:36
On 06/14/2012 11:21 AM, Tejun Heo wrote:
Hello, guys.
On Thu, Jun 14, 2012 at 11:07:53AM +0900, Minchan Kim wrote:
quoted
It shouldn't work because vmap_page_range still can allocate
GFP_KERNEL by pud_alloc in vmap_pud_range. For it, I tried [1] but
other mm guys want to add WARNING [2] so let's avoiding gfp context
passing.
[1] https://lkml.org/lkml/2012/4/23/77
[2] https://lkml.org/lkml/2012/5/2/340
Yeah, vmalloc area doesn't support !GFP_KERNEL allocations and as
Minchan said, changing this would require updating page table
allocation functions on all archs. This is the same reason why percpu
allocator doesn't support !GFP_KERNEL allocations which in turn made
blk-throttle implement its own private percpu pool.
If xfs can't live without GFP_NOFS vmalloc allocations, either it has
to implement its own pool or maybe it's time to implement !GFP_KERNEL
allocs for vmalloc area. I don't know.
From: Dave Chinner <david@fromorbit.com> Date: 2012-06-14 03:34:29
On Thu, Jun 14, 2012 at 11:07:53AM +0900, Minchan Kim wrote:
Hi Fengguang,
On 06/14/2012 10:49 AM, Fengguang Wu wrote:
quoted
On Thu, Jun 14, 2012 at 11:20:26AM +1000, Dave Chinner wrote:
quoted
On Wed, Jun 13, 2012 at 08:39:32PM +0800, Fengguang Wu wrote:
quoted
Hi Christoph, Dave,
I got this lockdep warning on XFS when running the xfs tests:
[rant warning]
quoted
quoted
Bug in vm_map_ram - it does an unconditional GFP_KERNEL allocation
here, and we are in a GFP_NOFS context. We can't pass a gfp_mask to
vm_map_ram(), so until vm_map_ram() grows that we can't fix it...
This trivial patch should fix it.
.....
It shouldn't work because vmap_page_range still can allocate GFP_KERNEL by pud_alloc in vmap_pud_range.
For it, I tried [1] but other mm guys want to add WARNING [2] so let's avoiding gfp context passing.
Oh, wonderful, you're pulling the "it's not a MM issue, don't use
vmalloc" card.
https://lkml.org/lkml/2012/4/24/29
"vmalloc was never supposed to use gfp flags for allocation
"context" restriction. I.e., it was always supposed to have
blocking, fs, and io capable allocation context."
vmalloc always was a badly neglected, ugly step-sister of kmalloc
that was kept in the basement and only brought out when the tax
collector called. But that inner ugliness doesn't change the fact
that beatiful things have been built around it. XFS has used
vm_map_ram() and it's predecessor since it was first ported to linux
some 13 or 14 years ago, so the above claim is way out of date. i.e.
vmalloc has been used in GFP_NOFS context since before that flag
even existed....
http://lkml.org/lkml/2012/4/24/67
"I would say add a bit of warnings and documentation, and see what
can be done about callers."
Wonderful. Well, there's about 2 years of work queued up for me
before I even get to the do the open heart surgery that would allow
XFS to handle memory allocation failures at this level without
causing the filesystem to shut down.
Andrew Morton's response:
https://lkml.org/lkml/2012/4/24/413
"There are gruesome problems in block/blk-throttle.c (thread
"mempool, percpu, blkcg: fix percpu stat allocation and remove
stats_lock"). It wants to do an alloc_percpu()->vmalloc() from the
IO submission path, under GFP_NOIO.
Changing vmalloc() to take a gfp_t does make lots of sense, although
I worry a bit about making vmalloc() easier to use!"
OK, so according to Andrew there is no technical reason why it can't
be done, it's just handwaving about "vmalloc is bad"....
https://lkml.org/lkml/2012/5/2/452
"> Where are these offending callsites?
dm:
...
ubi:
....
ext4:
....
ntfs:
....
ubifs:
....
mm:
....
ceph:
...."
So, we've got a bunch of filesystems that require vmalloc under
GFP_NOFS conditions. Perhaps there's a reason for needing to be able
to do this in filesystem code? Like, perhaps, avoiding memory
reclaim deadlocks?
https://lkml.org/lkml/2012/5/3/27
"Note that in writeback paths, a "good citizen" filesystem should
not require any allocations, or at least it should be able to
tolerate allocation failures. So fixing that would be a good idea
anyway."
Oh, please. I have been hearing this for years, and are we any
closer to it? No, we are further away from ever being able to
acheive this than ever. Face it, filesystems require memory
allocation to write dirty data to disk, and the amount is almost
impossible to define. Hence mempools can't be used because we can't
give any guarantees of forward progress. And for vmalloc?
Filesystems widely use vmalloc/vm_map_ram because kmalloc fails on
large contiguous allocations. This renders kmalloc unfit for
purpose, so we have to fall back to single page allocation and
vm_map_ram or vmalloc so that the filesystem can function properly.
And to avoid deadlocks, all memory allocation must be able to
specify GFP_NOFS to prevent the MM subsystem from recursing into the
filesystem. Therefore, vmalloc needs to support GFP_NOFS.
I don't care how you make it happen, just fix it. Trying to place
the blame on the filesystem folk for using vmalloc in GFP_NOFS
contexts is a total and utter cop-out, because mm folk of all people
should know that non-zero order kmalloc is not a reliable
alternative....
[end rant]
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
From: David Rientjes <rientjes@google.com> Date: 2012-06-14 03:53:42
On Thu, 14 Jun 2012, Dave Chinner wrote:
Oh, please. I have been hearing this for years, and are we any
closer to it? No, we are further away from ever being able to
acheive this than ever. Face it, filesystems require memory
allocation to write dirty data to disk, and the amount is almost
impossible to define. Hence mempools can't be used because we can't
give any guarantees of forward progress. And for vmalloc?
Filesystems widely use vmalloc/vm_map_ram because kmalloc fails on
large contiguous allocations. This renders kmalloc unfit for
purpose, so we have to fall back to single page allocation and
vm_map_ram or vmalloc so that the filesystem can function properly.
And to avoid deadlocks, all memory allocation must be able to
specify GFP_NOFS to prevent the MM subsystem from recursing into the
filesystem. Therefore, vmalloc needs to support GFP_NOFS.
I don't care how you make it happen, just fix it. Trying to place
the blame on the filesystem folk for using vmalloc in GFP_NOFS
contexts is a total and utter cop-out, because mm folk of all people
should know that non-zero order kmalloc is not a reliable
alternative....
I'd actually like to see a demonstrated problem (i.e. not theoretical)
where vmalloc() stalls indefinitely because its passed GFP_NOFS. I've
never seen one reported.
This is because the per-arch pte allocators have hardwired GFP_KERNEL
flags, but then again they also have __GFP_REPEAT which would cause them
to loop infinitely in the page allocator if a page was not reclaimed,
which has little success without __GFP_FS. But nobody has ever reported a
livelock that was triaged back to passing !__GFP_FS to vmalloc().
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
From: Minchan Kim <minchan@kernel.org> Date: 2012-06-14 05:51:03
On 06/14/2012 12:34 PM, Dave Chinner wrote:
On Thu, Jun 14, 2012 at 11:07:53AM +0900, Minchan Kim wrote:
quoted
Hi Fengguang,
On 06/14/2012 10:49 AM, Fengguang Wu wrote:
quoted
On Thu, Jun 14, 2012 at 11:20:26AM +1000, Dave Chinner wrote:
quoted
On Wed, Jun 13, 2012 at 08:39:32PM +0800, Fengguang Wu wrote:
quoted
Hi Christoph, Dave,
I got this lockdep warning on XFS when running the xfs tests:
[rant warning]
quoted
quoted
quoted
Bug in vm_map_ram - it does an unconditional GFP_KERNEL allocation
here, and we are in a GFP_NOFS context. We can't pass a gfp_mask to
vm_map_ram(), so until vm_map_ram() grows that we can't fix it...
This trivial patch should fix it.
.....
quoted
It shouldn't work because vmap_page_range still can allocate GFP_KERNEL by pud_alloc in vmap_pud_range.
For it, I tried [1] but other mm guys want to add WARNING [2] so let's avoiding gfp context passing.
Oh, wonderful, you're pulling the "it's not a MM issue, don't use
vmalloc" card.
https://lkml.org/lkml/2012/4/24/29
"vmalloc was never supposed to use gfp flags for allocation
"context" restriction. I.e., it was always supposed to have
blocking, fs, and io capable allocation context."
vmalloc always was a badly neglected, ugly step-sister of kmalloc
that was kept in the basement and only brought out when the tax
collector called. But that inner ugliness doesn't change the fact
that beatiful things have been built around it. XFS has used
vm_map_ram() and it's predecessor since it was first ported to linux
some 13 or 14 years ago, so the above claim is way out of date. i.e.
vmalloc has been used in GFP_NOFS context since before that flag
even existed....
http://lkml.org/lkml/2012/4/24/67
"I would say add a bit of warnings and documentation, and see what
can be done about callers."
Wonderful. Well, there's about 2 years of work queued up for me
before I even get to the do the open heart surgery that would allow
XFS to handle memory allocation failures at this level without
causing the filesystem to shut down.
Andrew Morton's response:
https://lkml.org/lkml/2012/4/24/413
"There are gruesome problems in block/blk-throttle.c (thread
"mempool, percpu, blkcg: fix percpu stat allocation and remove
stats_lock"). It wants to do an alloc_percpu()->vmalloc() from the
IO submission path, under GFP_NOIO.
Changing vmalloc() to take a gfp_t does make lots of sense, although
I worry a bit about making vmalloc() easier to use!"
OK, so according to Andrew there is no technical reason why it can't
be done, it's just handwaving about "vmalloc is bad"....
https://lkml.org/lkml/2012/5/2/452
"> Where are these offending callsites?
dm:
...
ubi:
....
ext4:
....
ntfs:
....
ubifs:
....
mm:
....
ceph:
...."
So, we've got a bunch of filesystems that require vmalloc under
GFP_NOFS conditions. Perhaps there's a reason for needing to be able
to do this in filesystem code? Like, perhaps, avoiding memory
reclaim deadlocks?
https://lkml.org/lkml/2012/5/3/27
"Note that in writeback paths, a "good citizen" filesystem should
not require any allocations, or at least it should be able to
tolerate allocation failures. So fixing that would be a good idea
anyway."
Oh, please. I have been hearing this for years, and are we any
closer to it? No, we are further away from ever being able to
acheive this than ever. Face it, filesystems require memory
allocation to write dirty data to disk, and the amount is almost
impossible to define. Hence mempools can't be used because we can't
give any guarantees of forward progress. And for vmalloc?
Filesystems widely use vmalloc/vm_map_ram because kmalloc fails on
large contiguous allocations. This renders kmalloc unfit for
purpose, so we have to fall back to single page allocation and
vm_map_ram or vmalloc so that the filesystem can function properly.
And to avoid deadlocks, all memory allocation must be able to
specify GFP_NOFS to prevent the MM subsystem from recursing into the
filesystem. Therefore, vmalloc needs to support GFP_NOFS.
I don't care how you make it happen, just fix it. Trying to place
the blame on the filesystem folk for using vmalloc in GFP_NOFS
contexts is a total and utter cop-out, because mm folk of all people
should know that non-zero order kmalloc is not a reliable
alternative....
[end rant]
Cheers,
Dave.
Again, hot potato.
I understand your claim and biased on you.
Day by day, guys are adding their memory pool for avoiding this problem.
IMHO, It's not good. If we can support context gfp passing in vmalloc, we could save many code.
It means less error-prone as well as less code size.
Ccing Nick. I hope we reach a agreement about this problem in this chance.
--
Kind regards,
Minchan Kim
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
From: Andreas Dilger <hidden> Date: 2012-06-14 07:52:18
On 2012-06-13, at 9:34 PM, Dave Chinner wrote:
On Thu, Jun 14, 2012 at 11:07:53AM +0900, Minchan Kim wrote:
quoted
Hi Fengguang,
On 06/14/2012 10:49 AM, Fengguang Wu wrote:
quoted
On Thu, Jun 14, 2012 at 11:20:26AM +1000, Dave Chinner wrote:
quoted
On Wed, Jun 13, 2012 at 08:39:32PM +0800, Fengguang Wu wrote:
quoted
Hi Christoph, Dave,
I got this lockdep warning on XFS when running the xfs tests:
[rant warning]
I heartily agree with Dave on this. There are plenty of places
that would be simplified by making vmalloc() work better. Instead,
there are workarounds (e.g. allocating 2-level trees of pointers
to smaller kmalloc'd elements, instead of a single large array)
spread around that add to code complexity and memory inefficiency.
I don't think there is a real danger of people starting to abuse
vmalloc() for everything, but does it really make sense to keep
some component of the kernel at sub-par functionality/efficiency
just to discourage its use?
Cheers, Andreas
quoted
quoted
quoted
Bug in vm_map_ram - it does an unconditional GFP_KERNEL allocation
here, and we are in a GFP_NOFS context. We can't pass a gfp_mask to
vm_map_ram(), so until vm_map_ram() grows that we can't fix it...
This trivial patch should fix it.
.....
quoted
It shouldn't work because vmap_page_range still can allocate GFP_KERNEL by pud_alloc in vmap_pud_range.
For it, I tried [1] but other mm guys want to add WARNING [2] so let's avoiding gfp context passing.
Oh, wonderful, you're pulling the "it's not a MM issue, don't use
vmalloc" card.
https://lkml.org/lkml/2012/4/24/29
"vmalloc was never supposed to use gfp flags for allocation
"context" restriction. I.e., it was always supposed to have
blocking, fs, and io capable allocation context."
vmalloc always was a badly neglected, ugly step-sister of kmalloc
that was kept in the basement and only brought out when the tax
collector called. But that inner ugliness doesn't change the fact
that beatiful things have been built around it. XFS has used
vm_map_ram() and it's predecessor since it was first ported to linux
some 13 or 14 years ago, so the above claim is way out of date. i.e.
vmalloc has been used in GFP_NOFS context since before that flag
even existed....
http://lkml.org/lkml/2012/4/24/67
"I would say add a bit of warnings and documentation, and see what
can be done about callers."
Wonderful. Well, there's about 2 years of work queued up for me
before I even get to the do the open heart surgery that would allow
XFS to handle memory allocation failures at this level without
causing the filesystem to shut down.
Andrew Morton's response:
https://lkml.org/lkml/2012/4/24/413
"There are gruesome problems in block/blk-throttle.c (thread
"mempool, percpu, blkcg: fix percpu stat allocation and remove
stats_lock"). It wants to do an alloc_percpu()->vmalloc() from the
IO submission path, under GFP_NOIO.
Changing vmalloc() to take a gfp_t does make lots of sense, although
I worry a bit about making vmalloc() easier to use!"
OK, so according to Andrew there is no technical reason why it can't
be done, it's just handwaving about "vmalloc is bad"....
https://lkml.org/lkml/2012/5/2/452
"> Where are these offending callsites?
dm:
...
ubi:
....
ext4:
....
ntfs:
....
ubifs:
....
mm:
....
ceph:
...."
So, we've got a bunch of filesystems that require vmalloc under
GFP_NOFS conditions. Perhaps there's a reason for needing to be able
to do this in filesystem code? Like, perhaps, avoiding memory
reclaim deadlocks?
https://lkml.org/lkml/2012/5/3/27
"Note that in writeback paths, a "good citizen" filesystem should
not require any allocations, or at least it should be able to
tolerate allocation failures. So fixing that would be a good idea
anyway."
Oh, please. I have been hearing this for years, and are we any
closer to it? No, we are further away from ever being able to
acheive this than ever. Face it, filesystems require memory
allocation to write dirty data to disk, and the amount is almost
impossible to define. Hence mempools can't be used because we can't
give any guarantees of forward progress. And for vmalloc?
Filesystems widely use vmalloc/vm_map_ram because kmalloc fails on
large contiguous allocations. This renders kmalloc unfit for
purpose, so we have to fall back to single page allocation and
vm_map_ram or vmalloc so that the filesystem can function properly.
And to avoid deadlocks, all memory allocation must be able to
specify GFP_NOFS to prevent the MM subsystem from recursing into the
filesystem. Therefore, vmalloc needs to support GFP_NOFS.
I don't care how you make it happen, just fix it. Trying to place
the blame on the filesystem folk for using vmalloc in GFP_NOFS
contexts is a total and utter cop-out, because mm folk of all people
should know that non-zero order kmalloc is not a reliable
alternative....
[end rant]
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html