Re: [mmotm-2016-07-18-16-40] page allocation failure: order:2, mode:0x2000000(GFP_NOWAIT)
From: Kirill A. Shutemov <hidden>
Date: 2016-07-20 15:19:10
Subsystem:
library code, the rest · Maintainers:
Andrew Morton, Linus Torvalds
On Wed, Jul 20, 2016 at 03:39:24PM +0200, Vlastimil Babka wrote:
On 07/20/2016 01:53 PM, Michal Hocko wrote:quoted
On Wed 20-07-16 14:44:17, Kirill A. Shutemov wrote:quoted
Hello, Looks like current mmotm is broken. See trace below.Why do you think it is broken? This is order-2 NOWAIT allocation. So we are relying on atomic highorder reserve and kcompactd to make sufficient progress. It is hard to find out more without the full log including the meminfo.Also it seems to come from kasan allocating stackdepot space to record who freed a slab object, or something.quoted
quoted
It's easy to reproduce in my setup: virtual machine with some amount of swap space and try allocate about the size of RAM in userspace (I used usemem[1] for that).Have you tried to bisect it?
Bisected to a590d2628f08 ("mm, kasan: switch SLUB to stackdepot, enable
memory quarantine for SLUB").
I guess it's candidate for __GFP_WARN. Not sure if there's a better
solution.
This helps:
diff --git a/lib/stackdepot.c b/lib/stackdepot.c
index 53ad6c0831ae..60f77f1d470a 100644
--- a/lib/stackdepot.c
+++ b/lib/stackdepot.c@@ -242,6 +242,7 @@ depot_stack_handle_t depot_save_stack(struct stack_trace *trace, */ alloc_flags &= ~GFP_ZONEMASK; alloc_flags &= (GFP_ATOMIC | GFP_KERNEL); + alloc_flags |= __GFP_NOWARN; page = alloc_pages(alloc_flags, STACK_ALLOC_ORDER); if (page) prealloc = page_address(page);
quoted
Some of the recent compaction changes might have made a difference.AFAIK recent compaction changes are not in mmotm yet. The node-based lru reclaim might have shifted some balances perhaps.quoted
quoted
Any clues? [1] http://www.spinics.net/lists/linux-mm/attachments/gtarazbJaHPaAT.gtar [ 39.413099] kswapd2: page allocation failure: order:2, mode:0x2000000(GFP_NOWAIT) [ 39.414122] CPU: 2 PID: 64 Comm: kswapd2 Not tainted 4.7.0-rc7-mm1-00428-gc3e13e4dab1b-dirty #2878 [ 39.416018] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.9.1-0-gb3ef39f-prebuilt.qemu-project.org 04/01/2014 [ 39.416018] 0000000000000002 ffff88002f807690 ffffffff81c8fb0d 1ffff10005f00ed6 [ 39.416018] 0000000000000000 0000000000000002 ffff88002f807750 ffff88002f8077a8 [ 39.416018] ffffffff813e728b ffff88002f8077a8 0200000000000000 0000000041b58ab3 [ 39.416018] Call Trace: [ 39.416018] <IRQ> [<ffffffff81c8fb0d>] dump_stack+0x95/0xe8 [ 39.416018] [<ffffffff813e728b>] warn_alloc_failed+0x1cb/0x250 [ 39.416018] [<ffffffff813e70c0>] ? zone_watermark_ok_safe+0x250/0x250 [ 39.416018] [<ffffffff81153788>] ? __kernel_text_address+0x78/0xa0 [ 39.416018] [<ffffffff813e7f4c>] __alloc_pages_nodemask+0x92c/0x1fe0 [ 39.416018] [<ffffffff8119047c>] ? sched_clock_cpu+0x12c/0x1e0 [ 39.416018] [<ffffffff81d24a17>] ? depot_save_stack+0x1b7/0x5b0 [ 39.416018] [<ffffffff813d5ac2>] ? mempool_free_slab+0x22/0x30 [ 39.416018] [<ffffffff81313595>] ? is_ftrace_trampoline+0xe5/0x120 [ 39.416018] [<ffffffff813e7620>] ? __free_pages+0x90/0x90 [ 39.416018] [<ffffffff811f9870>] ? debug_show_all_locks+0x290/0x290 [ 39.416018] [<ffffffff81190023>] ? sched_clock_local+0x43/0x120 [ 39.416018] [<ffffffff81cef815>] ? check_preemption_disabled+0x35/0x190 [ 39.416018] [<ffffffff81cef815>] ? check_preemption_disabled+0x35/0x190 [ 39.416018] [<ffffffff81cef98c>] ? debug_smp_processor_id+0x1c/0x20 [ 39.416018] [<ffffffff811ee78d>] ? get_lock_stats+0x1d/0x90 [ 39.416018] [<ffffffff81245617>] ? debug_lockdep_rcu_enabled+0x77/0x90 [ 39.416018] [<ffffffff81153788>] ? __kernel_text_address+0x78/0xa0 [ 39.416018] [<ffffffff8105d05b>] ? print_context_stack+0x7b/0x100 [ 39.416018] [<ffffffff814d42dc>] alloc_pages_current+0xbc/0x1f0 [ 39.416018] [<ffffffff81d24d5f>] depot_save_stack+0x4ff/0x5b0 [ 39.416018] [<ffffffff813d5ac2>] ? mempool_free_slab+0x22/0x30 [ 39.416018] [<ffffffff814eb4c7>] kasan_slab_free+0x157/0x180 [ 39.416018] [<ffffffff8107c58b>] ? save_stack_trace+0x2b/0x50 [ 39.416018] [<ffffffff814eb453>] ? kasan_slab_free+0xe3/0x180 [ 39.416018] [<ffffffff814e73e5>] ? kmem_cache_free+0x95/0x300 [ 39.416018] [<ffffffff813d5ac2>] ? mempool_free_slab+0x22/0x30 [ 39.416018] [<ffffffff813d59a9>] ? mempool_free+0xd9/0x1d0 [ 39.416018] [<ffffffff81be96e5>] ? bio_free+0x145/0x220 [ 39.416018] [<ffffffff81bea8bf>] ? bio_put+0x8f/0xb0 [ 39.416018] [<ffffffff814a7bfe>] ? end_swap_bio_write+0x22e/0x310 [ 39.416018] [<ffffffff81bf1687>] ? bio_endio+0x187/0x1f0 [ 39.416018] [<ffffffff81c0e89b>] ? blk_update_request+0x1bb/0xc30 [ 39.416018] [<ffffffff81c3238c>] ? blk_mq_end_request+0x4c/0x130 [ 39.416018] [<ffffffff8208a330>] ? virtblk_request_done+0xb0/0x2a0 [ 39.416018] [<ffffffff81c2d17d>] ? __blk_mq_complete_request_remote+0x5d/0x70 [ 39.416018] [<ffffffff8129fe3c>] ? flush_smp_call_function_queue+0xdc/0x3a0 [ 39.416018] [<ffffffff812a0548>] ? generic_smp_call_function_single_interrupt+0x18/0x20 [ 39.416018] [<ffffffff8109c654>] ? smp_call_function_single_interrupt+0x64/0x90 [ 39.416018] [<ffffffff829584a9>] ? call_function_single_interrupt+0x89/0x90 [ 39.416018] [<ffffffff81c350f6>] ? blk_mq_map_request+0xe6/0xc00 [ 39.416018] [<ffffffff81c36f6f>] ? blk_sq_make_request+0x9af/0xca0 [ 39.416018] [<ffffffff81c0b05e>] ? generic_make_request+0x30e/0x660 [ 39.416018] [<ffffffff81c0b540>] ? submit_bio+0x190/0x470 [ 39.416018] [<ffffffff814a8fd8>] ? __swap_writepage+0x6e8/0x940 [ 39.416018] [<ffffffff814a926a>] ? swap_writepage+0x3a/0x70 [ 39.416018] [<ffffffff8141376b>] ? shrink_page_list+0x1bdb/0x2f00 [ 39.416018] [<ffffffff81416038>] ? shrink_inactive_list+0x538/0xc70 [ 39.416018] [<ffffffff81417a1b>] ? shrink_node_memcg+0xa1b/0x1160 [ 39.416018] [<ffffffff81418436>] ? shrink_node+0x2d6/0xc60 [ 39.416018] [<ffffffff8141bf1e>] ? kswapd+0x82e/0x1460 [ 39.416018] [<ffffffff81156d4a>] ? kthread+0x24a/0x2e0 [ 39.416018] [<ffffffff8295773f>] ? ret_from_fork+0x1f/0x40 [ 39.416018] [<ffffffff81190023>] ? sched_clock_local+0x43/0x120 [ 39.416018] [<ffffffff81cef815>] ? check_preemption_disabled+0x35/0x190 [ 39.416018] [<ffffffff81cef815>] ? check_preemption_disabled+0x35/0x190 [ 39.416018] [<ffffffff81cef815>] ? check_preemption_disabled+0x35/0x190 [ 39.416018] [<ffffffff81353e65>] ? time_hardirqs_off+0x45/0x2f0 [ 39.416018] [<ffffffff814e73c5>] ? kmem_cache_free+0x75/0x300 [ 39.416018] [<ffffffff81353e65>] ? time_hardirqs_off+0x45/0x2f0 [ 39.416018] [<ffffffff814e747f>] ? kmem_cache_free+0x12f/0x300 [ 39.416018] [<ffffffff813d5ac2>] ? mempool_free_slab+0x22/0x30 [ 39.416018] [<ffffffff814e73e5>] kmem_cache_free+0x95/0x300 [ 39.416018] [<ffffffff813d5aa0>] ? mempool_free+0x1d0/0x1d0 [ 39.416018] [<ffffffff813d5ac2>] mempool_free_slab+0x22/0x30 [ 39.416018] [<ffffffff813d59a9>] mempool_free+0xd9/0x1d0 [ 39.416018] [<ffffffff81be96e5>] bio_free+0x145/0x220 [ 39.416018] [<ffffffff814a79d0>] ? SyS_madvise+0x13c0/0x13c0 [ 39.416018] [<ffffffff81bea8bf>] bio_put+0x8f/0xb0 [ 39.416018] [<ffffffff814a7bfe>] end_swap_bio_write+0x22e/0x310 [ 39.416018] [<ffffffff814a79d0>] ? SyS_madvise+0x13c0/0x13c0 [ 39.416018] [<ffffffff81bf1687>] bio_endio+0x187/0x1f0 [ 39.416018] [<ffffffff81c0e89b>] blk_update_request+0x1bb/0xc30 [ 39.416018] [<ffffffff81c2d120>] ? blkdev_issue_zeroout+0x3f0/0x3f0 [ 39.416018] [<ffffffff81c3238c>] blk_mq_end_request+0x4c/0x130 [ 39.416018] [<ffffffff8208a330>] virtblk_request_done+0xb0/0x2a0 [ 39.416018] [<ffffffff81c2d17d>] __blk_mq_complete_request_remote+0x5d/0x70 [ 39.416018] [<ffffffff8129fe3c>] flush_smp_call_function_queue+0xdc/0x3a0 [ 39.416018] [<ffffffff812a0548>] generic_smp_call_function_single_interrupt+0x18/0x20 [ 39.416018] [<ffffffff8109c654>] smp_call_function_single_interrupt+0x64/0x90 [ 39.416018] [<ffffffff829584a9>] call_function_single_interrupt+0x89/0x90 [ 39.416018] <EOI> [<ffffffff8120138b>] ? lock_acquire+0x15b/0x340 [ 39.416018] [<ffffffff81c350a9>] ? blk_mq_map_request+0x99/0xc00 [ 39.416018] [<ffffffff81c350f6>] blk_mq_map_request+0xe6/0xc00 [ 39.416018] [<ffffffff81c350a9>] ? blk_mq_map_request+0x99/0xc00 [ 39.416018] [<ffffffff81c8d434>] ? blk_integrity_merge_bio+0xb4/0x3b0 [ 39.416018] [<ffffffff81c35010>] ? blk_mq_alloc_request+0x490/0x490 [ 39.416018] [<ffffffff81c0dd66>] ? blk_attempt_plug_merge+0x226/0x2c0 [ 39.416018] [<ffffffff81c36f6f>] blk_sq_make_request+0x9af/0xca0 [ 39.416018] [<ffffffff81c365c0>] ? blk_mq_insert_requests+0x940/0x940 [ 39.416018] [<ffffffff81c07d20>] ? blk_exit_rl+0x60/0x60 [ 39.416018] [<ffffffff81c027b0>] ? handle_bad_sector+0x1e0/0x1e0 [ 39.416018] [<ffffffff81190023>] ? sched_clock_local+0x43/0x120 [ 39.416018] [<ffffffff81cef815>] ? check_preemption_disabled+0x35/0x190 [ 39.416018] [<ffffffff81cef815>] ? check_preemption_disabled+0x35/0x190 [ 39.416018] [<ffffffff81cef815>] ? check_preemption_disabled+0x35/0x190 [ 39.416018] [<ffffffff81c0b05e>] generic_make_request+0x30e/0x660 [ 39.416018] [<ffffffff81c0ad50>] ? blk_plug_queued_count+0x160/0x160 [ 39.416018] [<ffffffff81cef98c>] ? debug_smp_processor_id+0x1c/0x20 [ 39.416018] [<ffffffff81353ba2>] ? time_hardirqs_on+0xb2/0x330 [ 39.416018] [<ffffffff8152890b>] ? unlock_page_memcg+0x7b/0x130 [ 39.416018] [<ffffffff81c0b540>] submit_bio+0x190/0x470 [ 39.416018] [<ffffffff811dff60>] ? woken_wake_function+0x60/0x60 [ 39.416018] [<ffffffff81c0b3b0>] ? generic_make_request+0x660/0x660 [ 39.416018] [<ffffffff813f769d>] ? __test_set_page_writeback+0x36d/0x8c0 [ 39.416018] [<ffffffff814a8fd8>] __swap_writepage+0x6e8/0x940 [ 39.416018] [<ffffffff814a79d0>] ? SyS_madvise+0x13c0/0x13c0 [ 39.416018] [<ffffffff814a88f0>] ? generic_swapfile_activate+0x490/0x490 [ 39.416018] [<ffffffff814abd45>] ? swap_info_get+0x165/0x240 [ 39.416018] [<ffffffff814affda>] ? page_swapcount+0xba/0xf0 [ 39.416018] [<ffffffff82956ba1>] ? _raw_spin_unlock+0x31/0x50 [ 39.416018] [<ffffffff814affdf>] ? page_swapcount+0xbf/0xf0 [ 39.416018] [<ffffffff814a926a>] swap_writepage+0x3a/0x70 [ 39.416018] [<ffffffff8141376b>] shrink_page_list+0x1bdb/0x2f00 [ 39.416018] [<ffffffff81411b90>] ? putback_lru_page+0x3b0/0x3b0 [ 39.416018] [<ffffffff81cef9ac>] ? __this_cpu_preempt_check+0x1c/0x20 [ 39.416018] [<ffffffff81438ed4>] ? __mod_node_page_state+0x94/0xe0 [ 39.416018] [<ffffffff81190023>] ? sched_clock_local+0x43/0x120 [ 39.416018] [<ffffffff81cef815>] ? check_preemption_disabled+0x35/0x190 [ 39.416018] [<ffffffff81cef815>] ? check_preemption_disabled+0x35/0x190 [ 39.416018] [<ffffffff811ee78d>] ? get_lock_stats+0x1d/0x90 [ 39.416018] [<ffffffff8140fcb0>] ? __isolate_lru_page+0x3b0/0x3b0 [ 39.416018] [<ffffffff81cef98c>] ? debug_smp_processor_id+0x1c/0x20 [ 39.416018] [<ffffffff81353ba2>] ? time_hardirqs_on+0xb2/0x330 [ 39.416018] [<ffffffff811f6535>] ? trace_hardirqs_on_caller+0x405/0x590 [ 39.416018] [<ffffffff81416038>] shrink_inactive_list+0x538/0xc70 [ 39.416018] [<ffffffff81415b00>] ? putback_inactive_pages+0xaa0/0xaa0 [ 39.416018] [<ffffffff81416770>] ? shrink_inactive_list+0xc70/0xc70 [ 39.416018] [<ffffffff81190023>] ? sched_clock_local+0x43/0x120 [ 39.416018] [<ffffffff81cef815>] ? check_preemption_disabled+0x35/0x190 [ 39.416018] [<ffffffff81cef815>] ? check_preemption_disabled+0x35/0x190 [ 39.416018] [<ffffffff81cd5780>] ? _find_next_bit.part.0+0xe0/0x120 [ 39.416018] [<ffffffff8140e654>] ? pgdat_reclaimable_pages+0x764/0x9d0 [ 39.416018] [<ffffffff8140f2ec>] ? pgdat_reclaimable+0x13c/0x1d0 [ 39.416018] [<ffffffff8140f3cc>] ? lruvec_lru_size+0x4c/0xa0 [ 39.416018] [<ffffffff81417a1b>] shrink_node_memcg+0xa1b/0x1160 [ 39.416018] [<ffffffff81417000>] ? shrink_active_list+0x890/0x890 [ 39.416018] [<ffffffff81cef815>] ? check_preemption_disabled+0x35/0x190 [ 39.416018] [<ffffffff81cef98c>] ? debug_smp_processor_id+0x1c/0x20 [ 39.416018] [<ffffffff8151f208>] ? mem_cgroup_iter+0x1b8/0xd10 [ 39.416018] [<ffffffff82956c2c>] ? _raw_spin_unlock_irq+0x2c/0x50 [ 39.416018] [<ffffffff81418436>] shrink_node+0x2d6/0xc60 [ 39.416018] [<ffffffff81418160>] ? shrink_node_memcg+0x1160/0x1160 [ 39.416018] [<ffffffff8140f3cc>] ? lruvec_lru_size+0x4c/0xa0 [ 39.416018] [<ffffffff8141bf1e>] kswapd+0x82e/0x1460 [ 39.416018] [<ffffffff8141b6f0>] ? mem_cgroup_shrink_node+0x600/0x600 [ 39.416018] [<ffffffff81171cc8>] ? finish_task_switch+0x178/0x5b0 [ 39.416018] [<ffffffff82956c2c>] ? _raw_spin_unlock_irq+0x2c/0x50 [ 39.416018] [<ffffffff811f6535>] ? trace_hardirqs_on_caller+0x405/0x590 [ 39.416018] [<ffffffff82956c37>] ? _raw_spin_unlock_irq+0x37/0x50 [ 39.416018] [<ffffffff81171cc8>] ? finish_task_switch+0x178/0x5b0 [ 39.416018] [<ffffffff811de8f0>] ? __wake_up_common+0x150/0x150 [ 39.416018] [<ffffffff82948c3e>] ? __schedule+0x55e/0x1b60 [ 39.416018] [<ffffffff81156a32>] ? __kthread_parkme+0x172/0x240 [ 39.416018] [<ffffffff81156d4a>] kthread+0x24a/0x2e0 [ 39.416018] [<ffffffff8141b6f0>] ? mem_cgroup_shrink_node+0x600/0x600 [ 39.416018] [<ffffffff81156b00>] ? __kthread_parkme+0x240/0x240 [ 39.416018] [<ffffffff81171c9c>] ? finish_task_switch+0x14c/0x5b0 [ 39.416018] [<ffffffff8295773f>] ret_from_fork+0x1f/0x40 [ 39.416018] [<ffffffff81156b00>] ? __kthread_parkme+0x240/0x240 -- Kirill A. Shutemov
-- Kirill A. Shutemov -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>