-----Original Message-----
From: Nagal, Amit UTC CCS 
Sent: Monday, June 3, 2019 11:08 AM
To: 'Matthew Wilcox' <willy@infradead.org>
Cc: linux-kernel@vger.kernel.org; linux-mm@kvack.org; CHAWLA, RITU UTC CCS <redacted>; netdev@vger.kernel.org; Netter, Christian M UTC CCS <redacted>
Subject: RE: [External] Re: linux kernel page allocation failure and tuning of page cache



-----Original Message-----
From: Matthew Wilcox [mailto:willy@infradead.org]
Sent: Saturday, June 1, 2019 1:01 AM
To: Nagal, Amit UTC CCS <redacted>
Cc: linux-kernel@vger.kernel.org; linux-mm@kvack.org; CHAWLA, RITU UTC CCS <redacted>; netdev@vger.kernel.org
Subject: [External] Re: linux kernel page allocation failure and tuning of page cache

1) the platform is low memory platform having memory 64MB.

2)  we are doing around 45MB TCP data transfer from PC to target using netcat utility .On Target , a process receives data over socket and writes the data to flash disk .

I think your network is faster than your disk ...

Ok . I need to check it . But how does this affect page reclaim procedure .

5) sometimes , we observed kernel memory getting exhausted as page allocation failure happens in kernel  with the backtrace is printed below :
# [  775.947949] nc.traditional: page allocation failure: order:0,
mode:0x2080020(GFP_ATOMIC)

We're in the soft interrupt handler at this point, so we have very few options for freeing memory; we can't wait for I/O to complete, for example.

That said, this is a TCP connection.  We could drop the packet silently without such a noisy warning.  Perhaps just collect statistics on how many packets we dropped due to a low memory situation.

                                           total       used       free     shared    buffers     cached
             Mem:                     57         56               1          0              0                  20
-/+ buffers/cache:         35         22
Swap:                                    0          0                 0
eth0      Link encap:Ethernet  HWaddr D6:8C:FD:9D:35:AC
          inet addr:169.254.1.1  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::d48c:fdff:fe9d:35ac/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:8466 errors:1 dropped:0 overruns:3 frame:5
          TX packets:1085 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:12230677 (11.6 MiB)  TX bytes:107425 (104.9 KiB)
          Interrupt:77 Base address:0x3000

Not too many packet drops seem to happen .  

[  775.956362] CPU: 0 PID: 1288 Comm: nc.traditional Tainted: G           O    4.9.123-pic6-g31a13de-dirty #19
[  775.966085] Hardware name: Generic R7S72100 (Flattened Device Tree) 
[  775.972501] [<c0109829>] (unwind_backtrace) from [<c010796f>]
(show_stack+0xb/0xc) [  775.980118] [<c010796f>] (show_stack) from 
[<c0151de3>] (warn_alloc+0x89/0xba) [  775.987361] [<c0151de3>]
(warn_alloc) from [<c0152043>] (__alloc_pages_nodemask+0x1eb/0x634)
[  775.995790] [<c0152043>] (__alloc_pages_nodemask) from [<c0152523>]
(__alloc_page_frag+0x39/0xde) [  776.004685] [<c0152523>]
(__alloc_page_frag) from [<c03190f1>] (__netdev_alloc_skb+0x51/0xb0) [ 
776.013217] [<c03190f1>] (__netdev_alloc_skb) from [<c02c1b6f>]
(sh_eth_poll+0xbf/0x3c0) [  776.021342] [<c02c1b6f>] (sh_eth_poll) 
from [<c031fd8f>] (net_rx_action+0x77/0x170) [  776.029051] 
[<c031fd8f>] (net_rx_action) from [<c011238f>]
(__do_softirq+0x107/0x160) [  776.036896] [<c011238f>] (__do_softirq) 
from [<c0112589>] (irq_exit+0x5d/0x80) [  776.044165] [<c0112589>]
(irq_exit) from [<c012f4db>] (__handle_domain_irq+0x57/0x8c) [  776.052007] [<c012f4db>] (__handle_domain_irq) from [<c01012e1>] (gic_handle_irq+0x31/0x48) [  776.060362] [<c01012e1>] (gic_handle_irq) from [<c0108025>] (__irq_svc+0x65/0xac) [  776.067835] Exception stack(0xc1cafd70 to 0xc1cafdb8)
[  776.072876] fd60:                                     0002751c c1dec6a0 0000000c 521c3be5
[  776.081042] fd80: 56feb08e f64823a6 ffb35f7b feab513d f9cb0643 
0000056c c1caff10 ffffe000 [  776.089204] fda0: b1f49160 c1cafdc4
c180c677 c0234ace 200e0033 ffffffff [  776.095816] [<c0108025>]
(__irq_svc) from [<c0234ace>] (__copy_to_user_std+0x7e/0x430) [ 
776.103796] [<c0234ace>] (__copy_to_user_std) from [<c0241715>]
(copy_page_to_iter+0x105/0x250) [  776.112503] [<c0241715>]
(copy_page_to_iter) from [<c0319aeb>]
(skb_copy_datagram_iter+0xa3/0x108)
[  776.121469] [<c0319aeb>] (skb_copy_datagram_iter) from [<c03443a7>]
(tcp_recvmsg+0x3ab/0x5f4) [  776.130045] [<c03443a7>] (tcp_recvmsg) 
from [<c035e249>] (inet_recvmsg+0x21/0x2c) [  776.137576] [<c035e249>]
(inet_recvmsg) from [<c031009f>] (sock_read_iter+0x51/0x6e) [ 
776.145384] [<c031009f>] (sock_read_iter) from [<c017795d>]
(__vfs_read+0x97/0xb0) [  776.152967] [<c017795d>] (__vfs_read) from 
[<c01781d9>] (vfs_read+0x51/0xb0) [  776.159983] [<c01781d9>]
(vfs_read) from [<c0178aab>] (SyS_read+0x27/0x52) [  776.166837] [<c0178aab>] (SyS_read) from [<c0105261>] (ret_fast_syscall+0x1/0x54) [  776.174308] Mem-Info:
[  776.176650] active_anon:2037 inactive_anon:23 isolated_anon:0 [ 
776.176650]  active_file:2636 inactive_file:7391 isolated_file:32 [ 
776.176650]  unevictable:0 dirty:1366 writeback:1281 unstable:0

Almost all the dirty pages are under writeback at this point.

[  776.176650]  slab_reclaimable:719 slab_unreclaimable:724 [ 
776.176650]  mapped:1990 shmem:26 pagetables:159 bounce:0 [ 
776.176650]  free:373 free_pcp:6 free_cma:0

We have 373 free pages, but refused to allocate one of them to GFP_ATOMIC?
I don't understand why that failed.  We also didn't try to steal an inactive_file or inactive_anon page, which seems like an obvious thing we might want to do.

Yes that's where I am concerned . we do not have swap device so I am assuming perhaps inactive_anon pages are not stolen , but inactive_file pages could have been used . 

[  776.209062] Node 0 active_anon:8148kB inactive_anon:92kB 
active_file:10544kB inactive_file:29564kB unevictable:0kB 
isolated(anon):0kB isolated(file):128kB mapped:7960kB dirty:5464kB 
writeback:5124kB shmem:104kB writeback_tmp:0kB unstable:0kB
pages_scanned:0 all_unreclaimable? no [  776.233602] Normal 
free:1492kB min:964kB low:1204kB high:1444kB active_anon:8148kB 
inactive_anon:92kB active_file:10544kB inactive_file:29564kB 
unevictable:0kB writepending:10588kB present:65536kB managed:59304kB 
mlocked:0kB slab_reclaimable:2876kB slab_unreclaimable:2896kB 
kernel_stack:1152kB pagetables:636kB bounce:0kB free_pcp:24kB 
local_pcp:24kB free_cma:0kB [  776.265406] lowmem_reserve[]: 0 0 [ 
776.268761] Normal: 7*4kB (H) 5*8kB (H) 7*16kB (H) 5*32kB (H) 6*64kB
(H) 2*128kB (H) 2*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 
1492kB
10071 total pagecache pages
[  776.284124] 0 pages in swap cache
[  776.287446] Swap cache stats: add 0, delete 0, find 0/0 [ 
776.292645] Free swap  = 0kB [  776.295532] Total swap = 0kB [ 
776.298421] 16384 pages RAM [  776.301224] 0 pages HighMem/MovableOnly 
[  776.305052] 1558 pages reserved