Dear Linux folks,


With Linux 4.9.32 (and older versions), the following messages are 
reported under high network load.

[34615.695622] swapper/0: page allocation failure: order:0, mode:0x2080020(GFP_ATOMIC)
[34615.695631] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.9.32.mx64.159 #1
[34615.695633] Hardware name: Dell Inc. PowerEdge R910/0KYD3D, BIOS 2.10.0 08/29/2013
[34615.695634]  ffff883f7f803b98 ffffffff813e2ac6 ffffffff81f33ce0 0000000000000000
[34615.695636]  ffff883f7f803c18 ffffffff8113fdac 0208002000000011 ffffffff81f33ce0
[34615.695638]  ffff883f7f803bc0 ffff884000000010 ffff883f7f803c28 ffff883f7f803bd8
[34615.695640] Call Trace:
[34615.695642]  <IRQ>
[34615.695651]  [<ffffffff813e2ac6>] dump_stack+0x4d/0x67
[34615.695655]  [<ffffffff8113fdac>] warn_alloc+0x11c/0x140
[34615.695657]  [<ffffffff811402ab>] __alloc_pages_nodemask+0x45b/0xe70
[34615.695661]  [<ffffffff81183cec>] alloc_pages_current+0xac/0x160
[34615.695667]  [<ffffffff816edfc0>] bnx2x_alloc_rx_sge+0x170/0x220
[34615.695669]  [<ffffffff816f0c8b>] bnx2x_poll+0x5bb/0x19d0
[34615.695670]  [<ffffffff816f083f>] ? bnx2x_poll+0x16f/0x19d0
[34615.695676]  [<ffffffff819eb089>] net_rx_action+0x1f9/0x340
[34615.695680]  [<ffffffff81a9c228>] __do_softirq+0x88/0x29b
[34615.695684]  [<ffffffff81064206>] irq_exit+0x76/0x80
[34615.695686]  [<ffffffff81a9bfd3>] do_IRQ+0x63/0xf0
[34615.695689]  [<ffffffff81a9a4bf>] common_interrupt+0x7f/0x7f
[34615.695689]  <EOI>
[34615.695696]  [<ffffffff81903574>] ? cpuidle_enter_state+0x144/0x2e0
[34615.695697]  [<ffffffff81903569>] ? cpuidle_enter_state+0x139/0x2e0
[34615.695699]  [<ffffffff81903747>] cpuidle_enter+0x17/0x20
[34615.695703]  [<ffffffff8109e103>] call_cpuidle+0x23/0x40
[34615.695704]  [<ffffffff8109e324>] cpu_startup_entry+0x104/0x1e0
[34615.695706]  [<ffffffff81a93ef7>] rest_init+0x77/0x80
[34615.695708]  [<ffffffff8235fec3>] start_kernel+0x3b3/0x3c0
[34615.695710]  [<ffffffff8235f28f>] x86_64_start_reservations+0x2a/0x2c
[34615.695711]  [<ffffffff8235f3f9>] x86_64_start_kernel+0x168/0x176
[34615.695712] Mem-Info:
[34615.695747] active_anon:275374 inactive_anon:304826 isolated_anon:0
[34615.695747]  active_file:252290929 inactive_file:8578985 isolated_file:128
[34615.695747]  unevictable:0 dirty:4122065 writeback:166205 unstable:0
[34615.695747]  slab_reclaimable:1643416 slab_unreclaimable:57412
[34615.695747]  mapped:3551 shmem:8 pagetables:2275 bounce:0
[34615.695747]  free:265929 free_pcp:42297 free_cma:0
[34615.695757] Node 0 active_anon:276944kB inactive_anon:277380kB active_file:249203496kB inactive_file:9551688kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:1276kB dirty:5052260kB writeback:111676kB shmem:8kB writeback_tmp:0kB unstable:0kB pages_scanned:128 all_unreclaimable? no
[34615.695769] Node 1 active_anon:144016kB inactive_anon:144900kB active_file:253005028kB inactive_file:7623120kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:2560kB dirty:3717196kB writeback:93784kB shmem:12kB writeback_tmp:0kB unstable:0kB pages_scanned:34 all_unreclaimable? no
[34615.695782] Node 2 active_anon:164000kB inactive_anon:164004kB active_file:254373252kB inactive_file:7768156kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:3164kB dirty:3754752kB writeback:157076kB shmem:4kB writeback_tmp:0kB unstable:0kB pages_scanned:32 all_unreclaimable? no
[34615.695795] Node 3 active_anon:516536kB inactive_anon:633020kB active_file:252581940kB inactive_file:9372976kB unevictable:0kB isolated(anon):0kB isolated(file):128kB mapped:7204kB dirty:3964052kB writeback:302284kB shmem:8kB writeback_tmp:0kB unstable:0kB pages_scanned:29 all_unreclaimable? no
[34615.695797] Node 0 DMA free:15904kB min:0kB low:12kB high:24kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15904kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[34615.695800] lowmem_reserve[]: 0 1672 257669 257669
[34615.695803] Node 0 DMA32 free:1023928kB min:104kB low:1816kB high:3528kB active_anon:12kB inactive_anon:156kB active_file:639616kB inactive_file:37904kB unevictable:0kB writepending:9860kB present:2067680kB managed:1722564kB mlocked:0kB slab_reclaimable:4572kB slab_unreclaimable:140kB kernel_stack:416kB pagetables:0kB bounce:0kB free_pcp:13752kB local_pcp:188kB free_cma:0kB
[34615.695807] lowmem_reserve[]: 0 0 255997 255997
[34615.695810] Node 0 Normal free:6036kB min:16256kB low:278396kB high:540536kB active_anon:276932kB inactive_anon:277224kB active_file:248563880kB inactive_file:9513784kB unevictable:0kB writepending:5154076kB present:266338304kB managed:262141028kB mlocked:0kB slab_reclaimable:2858988kB slab_unreclaimable:63008kB kernel_stack:15936kB pagetables:2592kB bounce:0kB free_pcp:38680kB local_pcp:132kB free_cma:0kB
[34615.695813] lowmem_reserve[]: 0 0 0 0
[34615.695816] Node 1 Normal free:5840kB min:16388kB low:280624kB high:544860kB active_anon:144016kB inactive_anon:144900kB active_file:253005028kB inactive_file:7623120kB unevictable:0kB writepending:3810980kB present:268435456kB managed:264238448kB mlocked:0kB slab_reclaimable:2012588kB slab_unreclaimable:60932kB kernel_stack:16712kB pagetables:1736kB bounce:0kB free_pcp:40480kB local_pcp:768kB free_cma:0kB
[34615.695819] lowmem_reserve[]: 0 0 0 0
[34615.695821] Node 2 Normal free:6020kB min:16388kB low:280624kB high:544860kB active_anon:164000kB inactive_anon:164004kB active_file:254373252kB inactive_file:7768156kB unevictable:0kB writepending:3911828kB present:268435456kB managed:264238448kB mlocked:0kB slab_reclaimable:1196052kB slab_unreclaimable:43876kB kernel_stack:21056kB pagetables:1716kB bounce:0kB free_pcp:39772kB local_pcp:740kB free_cma:0kB
[34615.695825] lowmem_reserve[]: 0 0 0 0
[34615.695827] Node 3 Normal free:5988kB min:16388kB low:280624kB high:544860kB active_anon:516536kB inactive_anon:633020kB active_file:252581940kB inactive_file:9372976kB unevictable:0kB writepending:4266336kB present:268435456kB managed:264237500kB mlocked:0kB slab_reclaimable:501464kB slab_unreclaimable:61692kB kernel_stack:37208kB pagetables:3056kB bounce:0kB free_pcp:36504kB local_pcp:728kB free_cma:0kB
[34615.695830] lowmem_reserve[]: 0 0 0 0
[34615.695832] Node 0 DMA: 0*4kB 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15904kB
[34615.695839] Node 0 DMA32: 174*4kB (UME) 564*8kB (UME) 504*16kB (UME) 247*32kB (UME) 102*64kB (UME) 23*128kB (UE) 12*256kB (UME) 24*512kB (UME) 15*1024kB (UME) 2*2048kB (U) 234*4096kB (M) = 1023928kB
[34615.695848] Node 0 Normal: 241*4kB (UME) 38*8kB (UME) 24*16kB (UME) 52*32kB (UM) 11*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 1*2048kB (H) 0*4096kB = 6068kB
[34615.695855] Node 1 Normal: 484*4kB (MEH) 36*8kB (UMEH) 38*16kB (UMEH) 64*32kB (UMH) 7*64kB (H) 4*128kB (H) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5840kB
[34615.695862] Node 2 Normal: 305*4kB (UMEH) 423*8kB (MEH) 12*16kB (MH) 8*32kB (H) 5*64kB (H) 3*128kB (H) 2*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6268kB
[34615.695870] Node 3 Normal: 105*4kB (UME) 758*8kB (M) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 6484kB
[34615.695876] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[34615.695877] Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[34615.695878] Node 2 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[34615.695878] Node 3 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[34615.695879] 260870776 total pagecache pages
[34615.695880] 405 pages in swap cache
[34615.695881] Swap cache stats: add 2291, delete 1886, find 8/9
[34615.695881] Free swap  = 993530660kB
[34615.695882] Total swap = 993539792kB
[34615.695882] 268432085 pages RAM
[34615.695883] 0 pages HighMem/MovableOnly
[34615.695883] 4283612 pages reserved

In Linux kernel bug #12135 [1], Andrew comments [2]:

yes, it's just a diagnostic warning that the network is allocating
memory faster than the VM system is reclaiming it.  Everything should
recover and run to completion.

You should be able to make the warning go away by increasing /proc/sys/vm/min_free_kbytes.

On that system with 1 TB of memory, the value is 64 MB.

$ more /proc/sys/vm/min_free_kbytes
65536

Do you have suggestions on how to tune the system to get rid of the 
messages? Could the Linux kernel be improved?


Kind regards,

Paul


[1] https://bugzilla.kernel.org/show_bug.cgi?id=12135
[2] https://bugzilla.kernel.org/show_bug.cgi?id=12135#c2