From: Sachin Sant <hidden> Date: 2009-11-04 12:38:43
Sachin Sant wrote:
Today's next tree failed to boot on a POWER 6 box with :
------------[ cut here ]------------
kernel BUG at mm/mmap.c:2135!
Oops: Exception in kernel mode, sig: 5 [#2]
SMP NR_CPUS=1024 NUMA pSeries
Problem exists with today's next as well.
Likely cause for this problem seems to the following commit.
If i revert this patch the machine boots fine.
commit a0668cdc154e54bf0c85182e0535eea237d53146
powerpc/mm: Cleanup management of kmem_caches for pagetables
Thanks
-Sachin
--
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
From: David Gibson <hidden> Date: 2009-11-05 00:16:49
On Wed, Nov 04, 2009 at 06:08:44PM +0530, Sachin Sant wrote:
Sachin Sant wrote:
quoted
Today's next tree failed to boot on a POWER 6 box with :
------------[ cut here ]------------
kernel BUG at mm/mmap.c:2135!
Oops: Exception in kernel mode, sig: 5 [#2]
SMP NR_CPUS=1024 NUMA pSeries
Problem exists with today's next as well.
Likely cause for this problem seems to the following commit.
If i revert this patch the machine boots fine.
commit a0668cdc154e54bf0c85182e0535eea237d53146
powerpc/mm: Cleanup management of kmem_caches for pagetables
Ugh. Ok, it's not at all obvious how my patch could cause this bug.
Can you send your .config?
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
From: Sachin Sant <hidden> Date: 2009-11-05 04:42:50
David Gibson wrote:
On Wed, Nov 04, 2009 at 06:08:44PM +0530, Sachin Sant wrote:
quoted
Sachin Sant wrote:
quoted
Today's next tree failed to boot on a POWER 6 box with :
------------[ cut here ]------------
kernel BUG at mm/mmap.c:2135!
Oops: Exception in kernel mode, sig: 5 [#2]
SMP NR_CPUS=1024 NUMA pSeries
Problem exists with today's next as well.
Likely cause for this problem seems to the following commit.
If i revert this patch the machine boots fine.
commit a0668cdc154e54bf0c85182e0535eea237d53146
powerpc/mm: Cleanup management of kmem_caches for pagetables
Ugh. Ok, it's not at all obvious how my patch could cause this bug.
Can you send your .config?
Attached here is the config.
Thanks
-Sachin
--
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
From: Sachin Sant <hidden> Date: 2009-11-12 11:16:43
David Gibson wrote:
On Wed, Nov 04, 2009 at 06:08:44PM +0530, Sachin Sant wrote:
quoted
Sachin Sant wrote:
quoted
Today's next tree failed to boot on a POWER 6 box with :
------------[ cut here ]------------
kernel BUG at mm/mmap.c:2135!
Oops: Exception in kernel mode, sig: 5 [#2]
SMP NR_CPUS=1024 NUMA pSeries
Problem exists with today's next as well.
Likely cause for this problem seems to the following commit.
If i revert this patch the machine boots fine.
commit a0668cdc154e54bf0c85182e0535eea237d53146
powerpc/mm: Cleanup management of kmem_caches for pagetables
Ugh. Ok, it's not at all obvious how my patch could cause this bug.
Can you send your .config?
--
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
From: David Gibson <hidden> Date: 2009-11-13 01:37:29
On Thu, Nov 12, 2009 at 04:46:40PM +0530, Sachin Sant wrote:
David Gibson wrote:
quoted
On Wed, Nov 04, 2009 at 06:08:44PM +0530, Sachin Sant wrote:
quoted
Sachin Sant wrote:
quoted
Today's next tree failed to boot on a POWER 6 box with :
------------[ cut here ]------------
kernel BUG at mm/mmap.c:2135!
Oops: Exception in kernel mode, sig: 5 [#2]
SMP NR_CPUS=1024 NUMA pSeries
Problem exists with today's next as well.
Likely cause for this problem seems to the following commit.
If i revert this patch the machine boots fine.
commit a0668cdc154e54bf0c85182e0535eea237d53146
powerpc/mm: Cleanup management of kmem_caches for pagetables
Ugh. Ok, it's not at all obvious how my patch could cause this bug.
Can you send your .config?
Still present in today's next.
Sorry, I've been sidetracked by other issues and have only managed to
look into this today. My initial attempts to reproduce the bug with
your config on both POWER6 and POWER5+ have failed though.
Is it possible to get the complete boot log from this system - not
just the [cut here] section around the BUG()? This should help to
determine exactly when during boot the bug is being triggered.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
From: David Gibson <hidden> Date: 2009-11-13 02:10:50
On Fri, Nov 13, 2009 at 12:37:29PM +1100, David Gibson wrote:
On Thu, Nov 12, 2009 at 04:46:40PM +0530, Sachin Sant wrote:
quoted
David Gibson wrote:
quoted
On Wed, Nov 04, 2009 at 06:08:44PM +0530, Sachin Sant wrote:
quoted
Sachin Sant wrote:
quoted
Today's next tree failed to boot on a POWER 6 box with :
------------[ cut here ]------------
kernel BUG at mm/mmap.c:2135!
Oops: Exception in kernel mode, sig: 5 [#2]
SMP NR_CPUS=1024 NUMA pSeries
Problem exists with today's next as well.
Likely cause for this problem seems to the following commit.
If i revert this patch the machine boots fine.
commit a0668cdc154e54bf0c85182e0535eea237d53146
powerpc/mm: Cleanup management of kmem_caches for pagetables
Ugh. Ok, it's not at all obvious how my patch could cause this bug.
Can you send your .config?
Still present in today's next.
Sorry, I've been sidetracked by other issues and have only managed to
look into this today. My initial attempts to reproduce the bug with
your config on both POWER6 and POWER5+ have failed though.
Is it possible to get the complete boot log from this system - not
just the [cut here] section around the BUG()? This should help to
determine exactly when during boot the bug is being triggered.
Also, could you try booting the kernel with the patch below, which
should give a bit more information about the problem.
Index: working-2.6/mm/mmap.c
===================================================================
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
From: Sachin Sant <hidden> Date: 2009-11-13 09:35:34
David Gibson wrote:
quoted hunk
so, could you try booting the kernel with the patch below, which
should give a bit more information about the problem.
Index: working-2.6/mm/mmap.c
===================================================================
Here is the information collected with today's next.
(2.6.32-rc7-20091113)
------------[ cut here ]------------
kernel BUG at mm/mmap.c:2139!
cpu 0x3: Vector: 700 (Program Check) at [c0000000fae1b7e0]
pc: c000000000150e88: .exit_mmap+0x1ac/0x1d4
lr: c000000000150e78: .exit_mmap+0x19c/0x1d4
sp: c0000000fae1ba60
msr: 8000000000029032
current = 0xc0000000fada8be0
paca = 0xc000000000bb2c00
pid = 84, comm = cat
kernel BUG at mm/mmap.c:2139!
enter ? for help
[c0000000fae1bb10] c000000000093d24 .mmput+0x54/0x164
[c0000000fae1bba0] c000000000098f30 .exit_mm+0x17c/0x1a0
[c0000000fae1bc50] c00000000009b310 .do_exit+0x248/0x784
[c0000000fae1bd30] c00000000009b900 .do_group_exit+0xb4/0xe8
[c0000000fae1bdc0] c00000000009b948 .SyS_exit_group+0x14/0x28
[c0000000fae1be30] c0000000000085b4 syscall_exit+0x0/0x40
--- Exception: c01 (System Call) at 00000fff89a8ff40
SP (fffdf8a2460) is in userspace
Have attached the complete boot log.
At the time of crash values of mm and mm->nr_ptes were
<7>exit_mmap(): mm c0000000fa9f9580 nr_ptes 1
Thanks
-Sachin
--
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
From: David Gibson <hidden> Date: 2009-11-17 05:25:42
On Fri, Nov 13, 2009 at 03:05:33PM +0530, Sachin Sant wrote:
quoted hunk
David Gibson wrote:
quoted
so, could you try booting the kernel with the patch below, which
should give a bit more information about the problem.
Index: working-2.6/mm/mmap.c
===================================================================
while (vma)
vma = remove_vma(vma);
+ if (nr_ptes != 0)
+ printk("exit_mmap(): mm %p nr_ptes %d\n", mm, mm->nr_ptes);
BUG_ON(mm->nr_ptes > (FIRST_USER_ADDRESS+PMD_SIZE-1)>>PMD_SHIFT);
}
Here is the information collected with today's next.
(2.6.32-rc7-20091113)
------------[ cut here ]------------
kernel BUG at mm/mmap.c:2139!
cpu 0x3: Vector: 700 (Program Check) at [c0000000fae1b7e0]
pc: c000000000150e88: .exit_mmap+0x1ac/0x1d4
lr: c000000000150e78: .exit_mmap+0x19c/0x1d4
sp: c0000000fae1ba60
msr: 8000000000029032
current = 0xc0000000fada8be0
paca = 0xc000000000bb2c00
pid = 84, comm = cat
kernel BUG at mm/mmap.c:2139!
enter ? for help
[c0000000fae1bb10] c000000000093d24 .mmput+0x54/0x164
[c0000000fae1bba0] c000000000098f30 .exit_mm+0x17c/0x1a0
[c0000000fae1bc50] c00000000009b310 .do_exit+0x248/0x784
[c0000000fae1bd30] c00000000009b900 .do_group_exit+0xb4/0xe8
[c0000000fae1bdc0] c00000000009b948 .SyS_exit_group+0x14/0x28
[c0000000fae1be30] c0000000000085b4 syscall_exit+0x0/0x40
--- Exception: c01 (System Call) at 00000fff89a8ff40
SP (fffdf8a2460) is in userspace
Have attached the complete boot log.
At the time of crash values of mm and mm->nr_ptes were
<7>exit_mmap(): mm c0000000fa9f9580 nr_ptes 1
Hrm. Ok. I am truly baffled. Well, below is a revised debug patch
which I hope will shed some sort of light on things. I do also notice
from your full log that it looks like the bug is happening shortly
after we start userspace. So it may be differences in my userspace
set up that meant I haven't been able to reproduce it. I'll have
another look at that when I get a chance.
Index: working-2.6/mm/mmap.c
===================================================================
@@ -348,6 +350,8 @@ int __pte_alloc(struct mm_struct *mm, pmspin_lock(&mm->page_table_lock);if(!pmd_present(*pmd)){/* Has another populated it ? */mm->nr_ptes++;+printk("__pte_alloc() -> mm %p addr 0x%lx nr_ptes %d\n",mm,+address,mm->nr_ptes);pmd_populate(mm,pmd,new);new=NULL;}
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
From: Sachin Sant <hidden> Date: 2009-11-17 07:37:06
David Gibson wrote:
Hrm. Ok. I am truly baffled. Well, below is a revised debug patch
which I hope will shed some sort of light on things. I do also notice
Thanks for the debug patch. I have attached the collected information.
from your full log that it looks like the bug is happening shortly
after we start userspace. So it may be differences in my userspace
set up that meant I haven't been able to reproduce it. I'll have
another look at that when I get a chance.
Let me know if you need access to the system on which i can recreate the
bug. I can make that system available for you to debug this issue.
Thanks
-Sachin
--
---------------------------------
Sachin Sant
IBM Linux Technology Center
India Systems and Technology Labs
Bangalore, India
---------------------------------
From: David Gibson <hidden> Date: 2009-11-18 03:32:28
On Tue, Nov 17, 2009 at 01:07:03PM +0530, Sachin Sant wrote:
David Gibson wrote:
quoted
Hrm. Ok. I am truly baffled. Well, below is a revised debug patch
which I hope will shed some sort of light on things. I do also notice
Thanks for the debug patch. I have attached the collected information.
quoted
from your full log that it looks like the bug is happening shortly
after we start userspace. So it may be differences in my userspace
set up that meant I haven't been able to reproduce it. I'll have
another look at that when I get a chance.
Let me know if you need access to the system on which i can recreate the
bug. I can make that system available for you to debug this issue.
That's probably a good idea. I'm still pretty baffled by this, so it
will probably take several more rounds of debug patches to start
getting a handle on it.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson