Re: [PATCH 00/31] numa/core patches
From: Hugh Dickins <hughd@google.com>
Date: 2012-10-31 07:26:40
Also in:
lkml
On Tue, 30 Oct 2012, Johannes Weiner wrote:
On Tue, Oct 30, 2012 at 02:29:25PM +0800, Zhouping Liu wrote:quoted
On 10/29/2012 01:56 AM, Johannes Weiner wrote:quoted
On Fri, Oct 26, 2012 at 11:08:00AM +0200, Peter Zijlstra wrote:quoted
On Fri, 2012-10-26 at 17:07 +0800, Zhouping Liu wrote:quoted
[ 180.918591] RIP: 0010:[<ffffffff8118c39a>] [<ffffffff8118c39a>] mem_cgroup_prepare_migration+0xba/0xd0 [ 182.681450] [<ffffffff81183b60>] do_huge_pmd_numa_page+0x180/0x500 [ 182.775090] [<ffffffff811585c9>] handle_mm_fault+0x1e9/0x360 [ 182.863038] [<ffffffff81632b62>] __do_page_fault+0x172/0x4e0 [ 182.950574] [<ffffffff8101c283>] ? __switch_to_xtra+0x163/0x1a0 [ 183.041512] [<ffffffff8101281e>] ? __switch_to+0x3ce/0x4a0 [ 183.126832] [<ffffffff8162d686>] ? __schedule+0x3c6/0x7a0 [ 183.211216] [<ffffffff81632ede>] do_page_fault+0xe/0x10 [ 183.293705] [<ffffffff8162f518>] page_fault+0x28/0x30Johannes, this looks like the thp migration memcg hookery gone bad, could you have a look at this?Oops. Here is an incremental fix, feel free to fold it into #31.Hello Johannes, maybe I don't think the below patch completely fix this issue, as I found a new error(maybe similar with this): [88099.923724] ------------[ cut here ]------------ [88099.924036] kernel BUG at mm/memcontrol.c:1134! [88099.924036] invalid opcode: 0000 [#1] SMP [88099.924036] Modules linked in: lockd sunrpc kvm_amd kvm amd64_edac_mod edac_core ses enclosure serio_raw bnx2 pcspkr shpchp joydev i2c_piix4 edac_mce_amd k8temp dcdbas ata_generic pata_acpi megaraid_sas pata_serverworks usb_storage radeon i2c_algo_bit drm_kms_helper ttm drm i2c_core [88099.924036] CPU 7 [88099.924036] Pid: 3441, comm: stress Not tainted 3.7.0-rc2Jons+ #3 Dell Inc. PowerEdge 6950/0WN213 [88099.924036] RIP: 0010:[<ffffffff81188e97>] [<ffffffff81188e97>] mem_cgroup_update_lru_size+0x27/0x30Thanks a lot for your testing efforts, I really appreciate it. I'm looking into it, but I don't expect power to get back for several days where I live, so it's hard to reproduce it locally. But that looks like an LRU accounting imbalance that I wasn't able to tie to this patch yet. Do you see weird numbers for the lru counters in /proc/vmstat even without this memory cgroup patch? Ccing Hugh as well.
Sorry, I didn't get very far with it tonight. Almost certain to be a page which was added to lru while it looked like a 4k page, but taken off lru as a 2M page: we are taking a 2M page off lru here, it's likely to be the page in question, but not necessarily. There's quite a few put_page()s in do_huge_pmd_numa_page(), and it would help if we could focus on the one which is giving the trouble, but I don't know which that is. Zhouping, if you can, please would you do an "objdump -ld vmlinux >bigfile" of your kernel, then extract from bigfile just the lines from "<do_huge_pmd_numa_page>:" to whatever is the next function, and post or mail privately just that disassembly. That should be good to identify which of the put_page()s is involved. do_huge_pmd_numa_page() does look a bit worrying, but I've not pinned the misaccounting seen to the aspects which have worried me so far. Where is a check for page_mapcount(page) being 1? And surely it's unsafe to to be migrating the page when it was found !PageLRU? It's quite likely to be sitting in a pagevec or on a local list somewhere, about to be added to lru at any moment. Hugh -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>