From: Abdul Haleem <hidden> Date: 2017-09-20 07:40:48
On Tue, 2017-09-12 at 12:11 +0530, abdul wrote:
Hi,
Memory hot-unplug on PowerVM LPAR running next-20170911 results in
Faulting instruction address: 0xc0000000002b56c4
which maps to the below code path:
0xc0000000002b56c4 is in __rmqueue (./include/linux/list.h:104).
99 * This is only for internal list manipulation where we know
100 * the prev/next entries already!
101 */
102 static inline void __list_del(struct list_head * prev, struct
list_head * next)
103 {
104 next->prev = prev;
105 WRITE_ONCE(prev->next, next);
106 }
107
108 /**
I see another kernel Oops when running transparent hugepages
de-fragmentation test.
And the faulty instruction address again pointing to same code line
0xc00000000026f9f4 is in compaction_alloc (./include/linux/list.h:104)
steps to recreate:
-----------------
1. Enable transparent hugepages ("always")
2. Turn off the defrag $ echo 0 > khugepaged/defrag
3. Write random to memory path
4. Set huge pages numbers
5. Turn on defrag $ echo 1 > khugepaged/defrag
new trace:
----------
Unable to handle kernel paging request for data at address
0x5deadbeef0000108
Faulting instruction address: 0xc00000000026f9f4
Oops: Kernel access of bad area, sig: 11 [#1]
LE SMP NR_CPUS=2048 NUMA PowerNV
Dumping ftrace buffer:
(ftrace buffer empty)
Modules linked in: bridge iptable_mangle ipt_MASQUERADE
nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4
nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4
xt_tcpudp tun stp llc kvm_hv kvm iptable_filter vmx_crypto
powernv_op_panel powernv_rng leds_powernv rng_core ipmi_powernv
led_class ipmi_devintf ipmi_msghandler binfmt_misc nfsd ip_tables
x_tables autofs4 [last unloaded: bridge]
CPU: 52 PID: 803 Comm: kcompactd1 Not tainted
4.13.0-next-20170915-autotest #1
task: c0000007f2380000 task.stack: c0000007f2400000
NIP: c00000000026f9f4 LR: c0000000002d1328 CTR: c00000000026f980
REGS: c0000007f24037d0 TRAP: 0380 Not tainted
(4.13.0-next-20170915-autotest)
MSR: 9000000002009033 <SF,HV,VEC,EE,ME,IR,DR,RI,LE> CR: 22822088 XER:
00000000
CFAR: c0000000002d1324 SOFTE: 1
GPR00: c0000000002d1328 c0000007f2403a50 c0000000010bd500
f000000003dcd100
GPR04: c0000007f2403c90 c0000007f2403af0 f0000000021628a0
5deadbeef0000100
GPR08: 5deadbeef0000200 5deadbeef0000200 5deadbeef0000100
0000000000000060
GPR12: c00000000026f980 c00000000fd51e00 f000000002163700
0000000020000000
GPR16: 0000000000000000 0000000080000000 0000000000000000
c00000000026c3d0
GPR20: 0000000000000003 0000000000000001 c0000007f2403ca0
c0000007f2403c90
GPR24: c00000000026f980 0000000000000000 f0000000021636c0
f000000003dcd100
GPR28: 5deadbeef0000100 5deadbeef0000200 0000000000000001
c0000007f2403c90
NIP [c00000000026f9f4] compaction_alloc+0x74/0x350
LR [c0000000002d1328] migrate_pages+0x268/0x10c0
Call Trace:
[c0000007f2403a50] [c000000000239584] free_hot_cold_page+0x2b4/0x310
(unreliable)
[c0000007f2403ad0] [c0000000002d1328] migrate_pages+0x268/0x10c0
[c0000007f2403bc0] [c000000000270814] compact_zone+0x294/0xb30
[c0000007f2403c70] [c0000000002714c8] kcompactd_do_work+0x168/0x300
[c0000007f2403d40] [c000000000271718] kcompactd+0xb8/0x250
[c0000007f2403dc0] [c0000000001102f0] kthread+0x160/0x1a0
[c0000007f2403e30] [c00000000000bc60] ret_from_kernel_thread+0x5c/0x7c
Instruction dump:
419e008c 3d405dea e87f0000 614adbee 794a07c6 654af000 e9030008 e8e30000
3863ffe0 7d495378 614a0100 61290200 <f9070008> f8e80000 f9430020
f9230028
---[ end trace 27b8c4e55ceebc7d ]---
On Wed, Sep 20, 2017 at 12:40 AM, Abdul Haleem
[off-list ref] wrote:
On Tue, 2017-09-12 at 12:11 +0530, abdul wrote:
quoted
Hi,
Memory hot-unplug on PowerVM LPAR running next-20170911 results in
Faulting instruction address: 0xc0000000002b56c4
which maps to the below code path:
0xc0000000002b56c4 is in __rmqueue (./include/linux/list.h:104).
99 * This is only for internal list manipulation where we know
100 * the prev/next entries already!
101 */
102 static inline void __list_del(struct list_head * prev, struct
list_head * next)
103 {
104 next->prev = prev;
105 WRITE_ONCE(prev->next, next);
106 }
107
108 /**
I see another kernel Oops when running transparent hugepages
de-fragmentation test.
And the faulty instruction address again pointing to same code line
0xc00000000026f9f4 is in compaction_alloc (./include/linux/list.h:104)
steps to recreate:
-----------------
1. Enable transparent hugepages ("always")
2. Turn off the defrag $ echo 0 > khugepaged/defrag
3. Write random to memory path
4. Set huge pages numbers
5. Turn on defrag $ echo 1 > khugepaged/defrag
new trace:
----------
Unable to handle kernel paging request for data at address
0x5deadbeef0000108
This looks like use-after-list-removal, that value appears to be LIST_POISON1.
Try enabling CONFIG_DEBUG_LIST to see if you get better details?
-Kees
--
Kees Cook
Pixel Security
From: Abdul Haleem <hidden> Date: 2017-09-29 10:50:56
On Wed, 2017-09-20 at 12:54 -0700, Kees Cook wrote:
On Wed, Sep 20, 2017 at 12:40 AM, Abdul Haleem
[off-list ref] wrote:
quoted
On Tue, 2017-09-12 at 12:11 +0530, abdul wrote:
quoted
Hi,
Memory hot-unplug on PowerVM LPAR running next-20170911 results in
Faulting instruction address: 0xc0000000002b56c4
which maps to the below code path:
0xc0000000002b56c4 is in __rmqueue (./include/linux/list.h:104).
99 * This is only for internal list manipulation where we know
100 * the prev/next entries already!
101 */
102 static inline void __list_del(struct list_head * prev, struct
list_head * next)
103 {
104 next->prev = prev;
105 WRITE_ONCE(prev->next, next);
106 }
107
108 /**
I see another kernel Oops when running transparent hugepages
de-fragmentation test.
And the faulty instruction address again pointing to same code line
0xc00000000026f9f4 is in compaction_alloc (./include/linux/list.h:104)
steps to recreate:
-----------------
1. Enable transparent hugepages ("always")
2. Turn off the defrag $ echo 0 > khugepaged/defrag
3. Write random to memory path
4. Set huge pages numbers
5. Turn on defrag $ echo 1 > khugepaged/defrag
new trace:
----------
Unable to handle kernel paging request for data at address
0x5deadbeef0000108
This looks like use-after-list-removal, that value appears to be LIST_POISON1.
Try enabling CONFIG_DEBUG_LIST to see if you get better details?
With above config enabled I see below messages and also call traces. But
no kernel Oops.
BUG: Bad page state in process drmgr pfn:770c7
page:f000000001dc31c0 count:0 mapcount:0 mapping:f000000001dc31c8
index:0x1
flags: 0x33ffff800000000()
raw: 033ffff800000000 f000000001dc31c8 0000000000000001 00000000ffffffff
raw: 5deadbeef0000100 5deadbeef0000200 0000000000000000 0000000000000000
page dumped because: non-NULL mapping
--
Regard's
Abdul Haleem
IBM Linux Technology Centre
From: Abdul Haleem <hidden> Date: 2017-10-03 10:31:18
On Wed, 2017-09-20 at 12:54 -0700, Kees Cook wrote:
On Wed, Sep 20, 2017 at 12:40 AM, Abdul Haleem
[off-list ref] wrote:
quoted
On Tue, 2017-09-12 at 12:11 +0530, abdul wrote:
quoted
Hi,
Memory hot-unplug on PowerVM LPAR running next-20170911 results in
Faulting instruction address: 0xc0000000002b56c4
which maps to the below code path:
0xc0000000002b56c4 is in __rmqueue (./include/linux/list.h:104).
99 * This is only for internal list manipulation where we know
100 * the prev/next entries already!
101 */
102 static inline void __list_del(struct list_head * prev, struct
list_head * next)
103 {
104 next->prev = prev;
105 WRITE_ONCE(prev->next, next);
106 }
107
108 /**
I see another kernel Oops when running transparent hugepages
de-fragmentation test.
And the faulty instruction address again pointing to same code line
0xc00000000026f9f4 is in compaction_alloc (./include/linux/list.h:104)
steps to recreate:
-----------------
1. Enable transparent hugepages ("always")
2. Turn off the defrag $ echo 0 > khugepaged/defrag
3. Write random to memory path
4. Set huge pages numbers
5. Turn on defrag $ echo 1 > khugepaged/defrag
new trace:
----------
Unable to handle kernel paging request for data at address
0x5deadbeef0000108
This looks like use-after-list-removal, that value appears to be LIST_POISON1.
Try enabling CONFIG_DEBUG_LIST to see if you get better details?