From: David Rientjes <rientjes@google.com> Date: 2012-09-09 22:41:02
On Sun, 9 Sep 2012, Pavel Machek wrote:
On 3.6.0-rc2+, I tried to turn on the wireless, but got
root@amd:~# ifconfig wlan0 10.0.0.6 up
SIOCSIFFLAGS: Cannot allocate memory
SIOCSIFFLAGS: Cannot allocate memory
root@amd:~#
It looks like it uses "a bit too big" allocations to allocate
firmware...? Order five allocation....
Hmm... then I did "echo 3 > /proc/sys/vm/drop_caches" and now the
network works. Is it VM problem that it failed to allocate memory when
it was freeable?
From: Pavel Machek <hidden> Date: 2012-09-10 11:11:21
On Sun 2012-09-09 15:40:55, David Rientjes wrote:
On Sun, 9 Sep 2012, Pavel Machek wrote:
quoted
On 3.6.0-rc2+, I tried to turn on the wireless, but got
root@amd:~# ifconfig wlan0 10.0.0.6 up
SIOCSIFFLAGS: Cannot allocate memory
SIOCSIFFLAGS: Cannot allocate memory
root@amd:~#
It looks like it uses "a bit too big" allocations to allocate
firmware...? Order five allocation....
Hmm... then I did "echo 3 > /proc/sys/vm/drop_caches" and now the
network works. Is it VM problem that it failed to allocate memory when
it was freeable?
From: Andrew Morton <akpm@linux-foundation.org> Date: 2012-09-11 23:25:41
On Mon, 10 Sep 2012 13:11:13 +0200
Pavel Machek [off-list ref] wrote:
On Sun 2012-09-09 15:40:55, David Rientjes wrote:
quoted
On Sun, 9 Sep 2012, Pavel Machek wrote:
quoted
On 3.6.0-rc2+, I tried to turn on the wireless, but got
root@amd:~# ifconfig wlan0 10.0.0.6 up
SIOCSIFFLAGS: Cannot allocate memory
SIOCSIFFLAGS: Cannot allocate memory
root@amd:~#
It looks like it uses "a bit too big" allocations to allocate
firmware...? Order five allocation....
Hmm... then I did "echo 3 > /proc/sys/vm/drop_caches" and now the
network works. Is it VM problem that it failed to allocate memory when
it was freeable?
Asking for a 256k allocation is pretty crazy - this is an operating
system kernel, not a userspace application.
I'm wondering if this is due to a recent change, but I'm having trouble
working out where the allocation call site is.
From: Eric Dumazet <hidden> Date: 2012-09-12 05:16:37
On Tue, 2012-09-11 at 16:25 -0700, Andrew Morton wrote:
Asking for a 256k allocation is pretty crazy - this is an operating
system kernel, not a userspace application.
I'm wondering if this is due to a recent change, but I'm having trouble
working out where the allocation call site is.
--
From: Marc MERLIN <hidden> Date: 2012-09-12 05:57:28
On Wed, Sep 12, 2012 at 07:16:28AM +0200, Eric Dumazet wrote:
On Tue, 2012-09-11 at 16:25 -0700, Andrew Morton wrote:
quoted
Asking for a 256k allocation is pretty crazy - this is an operating
system kernel, not a userspace application.
I'm wondering if this is due to a recent change, but I'm having trouble
working out where the allocation call site is.
--
(Adding Marc Merlin to CC, since he reported same problem)
Thats the firmware loading in iwlwifi driver. Not sure if it can use SG.
drivers/net/wireless/iwlwifi/iwl-drv.c
iwl_alloc_ucode() -> iwl_alloc_fw_desc() -> dma_alloc_coherent()
It seems some sections of /lib/firmware/iwlwifi*.ucode files are above
128 Kbytes, so dma_alloc_coherent() try order-5 allocations
Thanks for looping me in, yes, this looks very familiar to me :)
In the other thread, Johannes Berg gave me this patch which is supposed to
help: http://p.sipsolutions.net/11ea33b376a5bac5.txt
Unfortunately due to very long work days, I haven't had the time to try it
out yet, but I will soon.
Would that help in this case too?
And to answer David Rientjes, I also have compaction on:
gandalfthegreat:~# zgrep CONFIG_COMPACTION /proc/config.gz
CONFIG_COMPACTION=y
Full config:
http://marc.merlins.org/tmp/config-3.5.2-amd64-preempt-noide-20120731
If that helps for comparison, my thread is here:
http://www.spinics.net/lists/linux-wireless/msg96438.html
Thanks,
Marc
--
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
.... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/
On Tue, Sep 11, 2012 at 04:25:36PM -0700, Andrew Morton wrote:
On Mon, 10 Sep 2012 13:11:13 +0200
Pavel Machek [off-list ref] wrote:
quoted
On Sun 2012-09-09 15:40:55, David Rientjes wrote:
quoted
On Sun, 9 Sep 2012, Pavel Machek wrote:
quoted
On 3.6.0-rc2+, I tried to turn on the wireless, but got
root@amd:~# ifconfig wlan0 10.0.0.6 up
SIOCSIFFLAGS: Cannot allocate memory
SIOCSIFFLAGS: Cannot allocate memory
root@amd:~#
It looks like it uses "a bit too big" allocations to allocate
firmware...? Order five allocation....
Hmm... then I did "echo 3 > /proc/sys/vm/drop_caches" and now the
network works. Is it VM problem that it failed to allocate memory when
it was freeable?
Asking for a 256k allocation is pretty crazy - this is an operating
system kernel, not a userspace application.
I'm wondering if this is due to a recent change, but I'm having trouble
working out where the allocation call site is.
iwlwifi/iwlegacy do such kind of allocation for ages, since iwlwifi driver
inclusion in 2.6.24 (however firmware was smaller then).
I can fix that in iwlegacy similar as Johannes did it in iwlwifi, but this
actually seems to be allocator regression. We use GFP_KERNEL allocation,
kernel can wait for free memory and/or swap out pages, I do not understand
why this fail.
Stanislaw
On Tue, Sep 11, 2012 at 04:25:36PM -0700, Andrew Morton wrote:
On Mon, 10 Sep 2012 13:11:13 +0200
Pavel Machek [off-list ref] wrote:
quoted
On Sun 2012-09-09 15:40:55, David Rientjes wrote:
quoted
On Sun, 9 Sep 2012, Pavel Machek wrote:
quoted
On 3.6.0-rc2+, I tried to turn on the wireless, but got
root@amd:~# ifconfig wlan0 10.0.0.6 up
SIOCSIFFLAGS: Cannot allocate memory
SIOCSIFFLAGS: Cannot allocate memory
root@amd:~#
It looks like it uses "a bit too big" allocations to allocate
firmware...? Order five allocation....
Hmm... then I did "echo 3 > /proc/sys/vm/drop_caches" and now the
network works. Is it VM problem that it failed to allocate memory when
it was freeable?
Asking for a 256k allocation is pretty crazy - this is an operating
system kernel, not a userspace application.
I'm wondering if this is due to a recent change, but I'm having trouble
working out where the allocation call site is.
It may be indirectly due to a recent change and this was somewhat
deliberate. Order-5 is larger than PAGE_ALLOC_COSTLY_ORDER and I doubt
__GFP_REPEAT was set so it is treated as something that can fail in
preference to aggressively reclaiming pages to satisfy the allocation. In
older kernels with lumpy reclaim and an aggressive kswapd it would have
probably succeeded but now it errs on the side of failing early instead
assuming that the caller can recover. Drivers that depend on order-5
allocations to succeed for correct operation are somewhat frowned upon.
--
Mel Gorman
SUSE Labs
On Wed, Sep 12, 2012 at 11:18:26AM +0100, Mel Gorman wrote:
On Tue, Sep 11, 2012 at 04:25:36PM -0700, Andrew Morton wrote:
quoted
On Mon, 10 Sep 2012 13:11:13 +0200
Pavel Machek [off-list ref] wrote:
quoted
On Sun 2012-09-09 15:40:55, David Rientjes wrote:
quoted
On Sun, 9 Sep 2012, Pavel Machek wrote:
quoted
On 3.6.0-rc2+, I tried to turn on the wireless, but got
root@amd:~# ifconfig wlan0 10.0.0.6 up
SIOCSIFFLAGS: Cannot allocate memory
SIOCSIFFLAGS: Cannot allocate memory
root@amd:~#
It looks like it uses "a bit too big" allocations to allocate
firmware...? Order five allocation....
Hmm... then I did "echo 3 > /proc/sys/vm/drop_caches" and now the
network works. Is it VM problem that it failed to allocate memory when
it was freeable?
Asking for a 256k allocation is pretty crazy - this is an operating
system kernel, not a userspace application.
I'm wondering if this is due to a recent change, but I'm having trouble
working out where the allocation call site is.
It may be indirectly due to a recent change and this was somewhat
deliberate. Order-5 is larger than PAGE_ALLOC_COSTLY_ORDER and I doubt
__GFP_REPEAT was set so it is treated as something that can fail in
preference to aggressively reclaiming pages to satisfy the allocation. In
older kernels with lumpy reclaim and an aggressive kswapd it would have
probably succeeded but now it errs on the side of failing early instead
assuming that the caller can recover. Drivers that depend on order-5
allocations to succeed for correct operation are somewhat frowned upon.
So, can this problem be solved like on below patch, or I should rather
split firmware loading into chunks similar like was already iwlwifi did?
I think this will certainly make memory compaction more aggressive by
avoiding the logic to defer calling compaction in the page allocator, but
because we lack lumpy reclaim this still has a higher probability of
failing than it had in the past because it will fail if 128KB of memory is
reclaimed that may not happen to be contiguous for an order-5 allocation
to succeed.
On Wed, Oct 03, 2012 at 11:07:13AM -0700, David Rientjes wrote:
On Wed, 3 Oct 2012, Stanislaw Gruszka wrote:
quoted
So, can this problem be solved like on below patch, or I should rather
split firmware loading into chunks similar like was already iwlwifi did?
Hmm, I looked at iwl3945 code and looks loading firmware in chunks is
nothing that can be easily done. 3945 bootstrap code expect that runtime
ucode will be placed in physically continue memory, and there are no
separate instructions for copy and for execute, just one to perform both
those actions. Maybe loading firmware in chunks can be done using
undocumented features of the device, but I'm eager to do this.
I think this will certainly make memory compaction more aggressive by
avoiding the logic to defer calling compaction in the page allocator, but
because we lack lumpy reclaim this still has a higher probability of
failing than it had in the past because it will fail if 128KB of memory is
reclaimed that may not happen to be contiguous for an order-5 allocation
to succeed.
I understand that complex systems like virtual memory manager require
various design compromises. In this case decision was to make memory
allocator to perform faster in cost of possible allocation failures.
I'm not quite sure if that was good decision, but I think VM developers
know best what is good for VM.
However, maybe this allocation issue initially reported here, was
caused by some bug, which is possibly now fixed i.e. memory leak on
some driver or subsystem.
Pavel, do you still can reproduce this problem on released 3.6 ?
Thanks
Stanislaw
From: Pavel Machek <hidden> Date: 2012-10-06 12:08:54
Hi!
On Fri 2012-10-05 10:37:00, Stanislaw Gruszka wrote:
On Wed, Oct 03, 2012 at 11:07:13AM -0700, David Rientjes wrote:
quoted
On Wed, 3 Oct 2012, Stanislaw Gruszka wrote:
quoted
So, can this problem be solved like on below patch, or I should rather
split firmware loading into chunks similar like was already iwlwifi did?
Hmm, I looked at iwl3945 code and looks loading firmware in chunks is
nothing that can be easily done. 3945 bootstrap code expect that runtime
ucode will be placed in physically continue memory, and there are no
separate instructions for copy and for execute, just one to perform both
those actions. Maybe loading firmware in chunks can be done using
undocumented features of the device, but I'm eager to do this.
Just allocate memory during boot?
Pavel, do you still can reproduce this problem on released 3.6 ?
On Sat, Oct 06, 2012 at 02:08:50PM +0200, Pavel Machek wrote:
On Fri 2012-10-05 10:37:00, Stanislaw Gruszka wrote:
quoted
On Wed, Oct 03, 2012 at 11:07:13AM -0700, David Rientjes wrote:
quoted
On Wed, 3 Oct 2012, Stanislaw Gruszka wrote:
quoted
So, can this problem be solved like on below patch, or I should rather
split firmware loading into chunks similar like was already iwlwifi did?
Hmm, I looked at iwl3945 code and looks loading firmware in chunks is
nothing that can be easily done. 3945 bootstrap code expect that runtime
ucode will be placed in physically continue memory, and there are no
separate instructions for copy and for execute, just one to perform both
those actions. Maybe loading firmware in chunks can be done using
undocumented features of the device, but I'm eager to do this.
Just allocate memory during boot?
On driver I can reserve memory during module load, but also this isn't
something I prefer to do.
quoted
Pavel, do you still can reproduce this problem on released 3.6 ?
It happened again yesterday on 3.6.0-rc6+. I don't think mm changed
between -rc6 and final...
Could you check __GFP_REPEAT oneline patch posted previously ?
And if that fail again, provide full dmesg (on your previous messages
there is vmap() failure, which I do not understand, where it come
from) ?
Thanks
Stanislaw