From: Olaf Hering <hidden> Date: 2006-03-05 14:09:35
On Sun, Feb 26, Linus Torvalds wrote:
Have I missed anything? Holler. And please keep reminding about any
regressions since 2.6.15.
I see random memory corruption on an early G3 ibook.
Testcase is an openSuSE 10.1 installation. 2.6.15 works ok modulo 2 bugs
to get it booted at all, and the usual udev breakage.
plain 2.6.16-rc5-git7 locks up after a few packages, no ping.
Our SuSE kernel does not lockup, but ext2 shows access beyond end of
device after > 200 packages, or the rpmdb gets corrupt, or both. With reiserfs
it gets past 100 packages, then reiserfs complains about fs corruption.
plain -rc2 shows the same reiserfs corruption.
plain -rc1 dies after a few packages, it jumps to 0x0 in softirq.
I'm trying to compile the git snapshots now, which is a real challenge..
From: Olaf Hering <hidden> Date: 2006-03-05 18:59:25
On Sun, Mar 05, Olaf Hering wrote:
On Sun, Feb 26, Linus Torvalds wrote:
quoted
Have I missed anything? Holler. And please keep reminding about any
regressions since 2.6.15.
I see random memory corruption on an early G3 ibook.
Testcase is an openSuSE 10.1 installation. 2.6.15 works ok modulo 2 bugs
to get it booted at all, and the usual udev breakage.
plain 2.6.16-rc5-git7 locks up after a few packages, no ping.
Our SuSE kernel does not lockup, but ext2 shows access beyond end of
device after > 200 packages, or the rpmdb gets corrupt, or both. With reiserfs
it gets past 100 packages, then reiserfs complains about fs corruption.
plain -rc2 shows the same reiserfs corruption.
plain -rc1 dies after a few packages, it jumps to 0x0 in softirq.
-git5 works, -git7 showed reiserfs corruption. -git6 died, jumped from
__do_softirq to 0x0, will try once again.
git5->6 has the mutex changes, but also lots of powerpc changes. Lets
see if I can narrow it down further.
The ibook has 160mb, installation is done via modular nfs
(ro,v3,rsize=32768,wsize=32768,hard,nolock,proto=tcp,addr=1.1.1.3)
I havent seen this on a B&W G3 with 256mb, nor on other ppc32 systems.
plain 2.6.16-rc5-git7 locks up after a few packages, no ping.
Our SuSE kernel does not lockup, but ext2 shows access beyond end of
device after > 200 packages, or the rpmdb gets corrupt, or both. With reiserfs
it gets past 100 packages, then reiserfs complains about fs corruption.
plain -rc2 shows the same reiserfs corruption.
plain -rc1 dies after a few packages, it jumps to 0x0 in softirq.
-git5 works, -git7 showed reiserfs corruption. -git6 died, jumped from
__do_softirq to 0x0, will try once again.
Since there are several git users in the ppc camp, one thing that always
helps is that when you test a -git snapshot, you also say what the "git
ID" was.
I'm assuming that when you say "-git5 works", you mean 2.6.15-git5.
In this case:
2.6.15-git5: 5367f2d67c7d0bf1faae90e6e7b4e2ac3c9b5e0f
2.6.15-git6: 977127174a7dff52d17faeeb4c4949a54221881f
2.6.15-git7: 05f6ece6f37f987e9de643f6f76e8fb5d5b9e014
git5->6 has the mutex changes, but also lots of powerpc changes. Lets
see if I can narrow it down further.
If you can try out git, the best way to proceed is
git bisect start
git bisect good 5367f2d67c7d0bf1faae90e6e7b4e2ac3c9b5e0f
git bisect bad 977127174a7dff52d17faeeb4c4949a54221881f
which should help narrow it down pretty efficiently (I'm marking -git6 as
bad, on the logic that the bug being chased is "corruption _or_ jumping to
address 0". It's much harder if you want to chase down just one bug, when
the other bug might stand in your way).
And yes, that range contains not just powerpc updates, but also PCI layer,
mutex changes, crypto and V4L/DVB. Doing just three or four bisection
trials would help narrow it down a lot (now it's 448 commits - doing three
bisctions should narrow it down into less than 60 commits and likely which
subsystem, while doing another bisection or two would get us into a few
tens of commits).
"git bisect" really is very powerful and easy to use.
Linus
From: Olaf Hering <hidden> Date: 2006-03-05 20:42:35
On Sun, Mar 05, Linus Torvalds wrote:
"git bisect" really is very powerful and easy to use.
Indeed. The one "between" gi5 and git6
(93b47684f60cf25e8cefe19a21d94aa0257fdf36) is fails also. There are no
mutex changes left, so I suspect some ppc bug.
With this changeset, my first attempt ran into this deadlock, the second
attempt lead to the reiserfs corruption.
See attached 93b47684f60cf25e8cefe19a21d94aa0257fdf36.log
I'm now at 03929c76f3e5af919fb762e9882a9c286d361e7d, which fails as
well. dmesg shows this:
Adding 295332k swap on /dev/hda10. Priority:-1 extents:1 across:295332k
ReiserFS: warning: is_leaf: wrong item type for item *3.5*[-1 -1 0xffffffff DIRECT], item_len 65535, item_location 65535, free_space(entry_count) 65535
ReiserFS: hda11: warning: vs-5150: search_by_key: invalid format found in block 0. Fsck?
ReiserFS: hda11: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2 4652 0x0 SD]
ReiserFS: warning: is_leaf: wrong item type for item *3.5*[-1 -1 0xffffffff DIRECT], item_len 65535, item_location 65535, free_space(entry_count) 65535
ReiserFS: hda11: warning: vs-5150: search_by_key: invalid format found in block 0. Fsck?
ReiserFS: hda11: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2 4652 0x0 SD]
ReiserFS: warning: is_leaf: wrong item type for item *3.5*[-1 -1 0xffffffff DIRECT], item_len 65535, item_location 65535, free_space(entry_count) 65535
ReiserFS: hda11: warning: vs-5150: search_by_key: invalid format found in block 0. Fsck?
ReiserFS: hda11: warning: vs-13070: reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2 4652 0x0 SD]
There are only powerpc related changes left.
Its a bit time comsuming until I get to the point of where it fails,
lets see how far I get this evening.
From: Paul Mackerras <hidden> Date: 2006-03-05 21:50:53
Olaf Hering writes:
I'm now at 03929c76f3e5af919fb762e9882a9c286d361e7d, which fails as
well. dmesg shows this:
The range from git5 to there includes David Woodhouse's syscall
entry/exit revamp (401d1f029bebb7153ca704997772113dc36d9527) and the
follow-ons which fix it for 32-bit:
9687c587596b54a77f08620595f5686ea35eed97
623703f620453c798b6fa3eb79ad8ea27bfd302a
There are also commits from Ben H that change the way we parse
addresses from the OF device tree. If you can bisect a bit further
that would be good, although you may strike problems between the 401d
and 6237 commits I mentioned above.
It would be interesting to take 401d and then apply 9687 and 6237
directly on top of it and try that, and if it fails, then try
1cd8e506209223ed10da805d99be55e268f4023c (the parent of 401d).
Paul.
From: Olaf Hering <hidden> Date: 2006-03-05 22:22:06
On Mon, Mar 06, Paul Mackeras wrote:
Olaf Hering writes:
quoted
I'm now at 03929c76f3e5af919fb762e9882a9c286d361e7d, which fails as
well. dmesg shows this:
The range from git5 to there includes David Woodhouse's syscall
entry/exit revamp (401d1f029bebb7153ca704997772113dc36d9527) and the
follow-ons which fix it for 32-bit:
9687c587596b54a77f08620595f5686ea35eed97
623703f620453c798b6fa3eb79ad8ea27bfd302a
There are also commits from Ben H that change the way we parse
addresses from the OF device tree. If you can bisect a bit further
that would be good, although you may strike problems between the 401d
and 6237 commits I mentioned above.
I will check this tomorrow.
quick update:
d4e4b3520c4df46cf1d15a56379a6fa57e267b7d, locks up, tried two times
404849bbd2bfd62e05b36f4753f6e1af6050a824 + 3 buildfixes:
31df1678d7732b94178a6e457ed6666e4431212f
8dacaedf04467e32c50148751a96150e73323cdc
d2dd482bc17c3bc240045f80a7c4b4d5cea5e29c
This one has the syscall changes, but not the two fixes you mentioned.
It gets far, but at the point where it locks up with the d4eb, it
crashes in run_timer_softirq, branched to 0x1f4. Maybe its the result of
the missing fixes. Will continue tomorrow.
From: Olaf Hering <hidden> Date: 2006-03-05 22:44:40
On Sun, Mar 05, Olaf Hering wrote:
404849bbd2bfd62e05b36f4753f6e1af6050a824 + 3 buildfixes:
31df1678d7732b94178a6e457ed6666e4431212f
8dacaedf04467e32c50148751a96150e73323cdc
d2dd482bc17c3bc240045f80a7c4b4d5cea5e29c
This one has the syscall changes, but not the two fixes you mentioned.
It gets far, but at the point where it locks up with the d4eb, it
crashes in run_timer_softirq, branched to 0x1f4. Maybe its the result of
the missing fixes. Will continue tomorrow.
From: Olaf Hering <hidden> Date: 2006-03-06 07:48:15
On Sun, Mar 05, Olaf Hering wrote:
On Sun, Mar 05, Olaf Hering wrote:
quoted
404849bbd2bfd62e05b36f4753f6e1af6050a824 + 3 buildfixes:
31df1678d7732b94178a6e457ed6666e4431212f
8dacaedf04467e32c50148751a96150e73323cdc
d2dd482bc17c3bc240045f80a7c4b4d5cea5e29c
This one has the syscall changes, but not the two fixes you mentioned.
It gets far, but at the point where it locks up with the d4eb, it
crashes in run_timer_softirq, branched to 0x1f4. Maybe its the result of
the missing fixes. Will continue tomorrow.
Another try with that version, now I see the corruption before the
package where it locked up before (glibc-locale, rather large).
Will backout the syscall change and try again with 404849bbd2bfd62e05b36f4753f6e1af6050a824.
Its not the syscall change at least. Looking further.
From: Olaf Hering <hidden> Date: 2006-03-06 16:48:21
On Mon, Mar 06, Paul Mackeras wrote:
There are also commits from Ben H that change the way we parse
addresses from the OF device tree. If you can bisect a bit further
that would be good, although you may strike problems between the 401d
and 6237 commits I mentioned above.
What I have right now is this, which got me in a non-compiling state.
I will pick the udbg stuff and apply the relevant changes to -git5.
==> .git/HEAD <==
463ce0e103f419f51b1769111e73fe8bb305d0ec
==> .git/refs/bisect/bad <==
51d3082fe6e55aecfa17113dbe98077c749f724c
==> .git/refs/bisect/good-5367f2d67c7d0bf1faae90e6e7b4e2ac3c9b5e0f <==
5367f2d67c7d0bf1faae90e6e7b4e2ac3c9b5e0f
==> .git/refs/bisect/good-d1405b869850982f05c7ec0d3f137ca27588192f <==
d1405b869850982f05c7ec0d3f137ca27588192f
==> .git/BISECT_LOG <==
git-bisect start
# good: [5367f2d67c7d0bf1faae90e6e7b4e2ac3c9b5e0f] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband
git-bisect good 5367f2d67c7d0bf1faae90e6e7b4e2ac3c9b5e0f
# bad: [977127174a7dff52d17faeeb4c4949a54221881f] Merge master.kernel.org:/pub/scm/linux/kernel/git/gregkh/pci-2.6
git-bisect bad 977127174a7dff52d17faeeb4c4949a54221881f
# bad: [93b47684f60cf25e8cefe19a21d94aa0257fdf36] drivers/*rest*: Replace pci_module_init() with pci_register_driver()
git-bisect bad 93b47684f60cf25e8cefe19a21d94aa0257fdf36
# bad: [03929c76f3e5af919fb762e9882a9c286d361e7d] ppc32: cpm_uart: fix xchar sending
git-bisect bad 03929c76f3e5af919fb762e9882a9c286d361e7d
# bad: [d4e4b3520c4df46cf1d15a56379a6fa57e267b7d] powerpc: fix for "Update OF address parsers"
git-bisect bad d4e4b3520c4df46cf1d15a56379a6fa57e267b7d
# bad: [404849bbd2bfd62e05b36f4753f6e1af6050a824] powerpc: Remove some unneeded fields from the paca
git-bisect bad 404849bbd2bfd62e05b36f4753f6e1af6050a824
# good: [d1405b869850982f05c7ec0d3f137ca27588192f] powerpc: Add OF address parsing code (#2)
git-bisect good d1405b869850982f05c7ec0d3f137ca27588192f
# bad: [e199500c6280aadf98c185db99fd24ab61ebe0c7] powerpc: partly merge iseries do_IRQ
git-bisect bad e199500c6280aadf98c185db99fd24ab61ebe0c7
# bad: [2c5bd01f8f5d7c655d9d1aa60b696d980947e3be] powerpc: convert macio_asic to use prom_parse
git-bisect bad 2c5bd01f8f5d7c655d9d1aa60b696d980947e3be
# bad: [51d3082fe6e55aecfa17113dbe98077c749f724c] powerpc: Unify udbg (#2)
git-bisect bad 51d3082fe6e55aecfa17113dbe98077c749f724c
From: Olaf Hering <hidden> Date: 2006-03-06 22:20:09
On Mon, Mar 06, Olaf Hering wrote:
On Mon, Mar 06, Paul Mackeras wrote:
quoted
There are also commits from Ben H that change the way we parse
addresses from the OF device tree. If you can bisect a bit further
that would be good, although you may strike problems between the 401d
and 6237 commits I mentioned above.
What I have right now is this, which got me in a non-compiling state.
I will pick the udbg stuff and apply the relevant changes to -git5.
Here is the working vs. non-working thing.
So its the udbg patch. My used .config is attached, use yes '' | make oldconfig
to get it in shape.
From: Olaf Hering <hidden> Date: 2006-03-06 23:02:37
On Mon, Mar 06, Olaf Hering wrote:
On Mon, Mar 06, Olaf Hering wrote:
quoted
On Mon, Mar 06, Paul Mackeras wrote:
quoted
There are also commits from Ben H that change the way we parse
addresses from the OF device tree. If you can bisect a bit further
that would be good, although you may strike problems between the 401d
and 6237 commits I mentioned above.
What I have right now is this, which got me in a non-compiling state.
I will pick the udbg stuff and apply the relevant changes to -git5.
I tried with CONFIG_BOOTX_TEXT disabled. same result. This is the list
of patches I used on top of 2.6.15:
patches.kernel.org/patch-2.6.15-git5
patches.suse/get_cramfs_inode-revert.patch
patches.suse/suse-ppc-legacy-io.patch
patches.arch/0022-powerpc-incorrect-rmo_top-handling-in-prom_init.txt
patches.suse/9100b205fdc70b300894954ebebbf2709c5ed525.patch
patches.suse/3d1229d6ae92ed1994f4411b8493327ef8f4b76f.patch
patches.suse/d1405b869850982f05c7ec0d3f137ca27588192f.patch
patches.suse/463ce0e103f419f51b1769111e73fe8bb305d0ec.patch
patches.suse/51d3082fe6e55aecfa17113dbe98077c749f724c.patch
patches.suse/31df1678d7732b94178a6e457ed6666e4431212f.patch
patches.suse/8dacaedf04467e32c50148751a96150e73323cdc.patch
patches.suse/52020d2bda9fe447bb50674a2e39e4064b6a10b5.patch
From: Olaf Hering <hidden> Date: 2006-03-11 21:59:42
On Tue, Mar 07, Olaf Hering wrote:
On Mon, Mar 06, Olaf Hering wrote:
quoted
On Mon, Mar 06, Olaf Hering wrote:
quoted
On Mon, Mar 06, Paul Mackeras wrote:
quoted
There are also commits from Ben H that change the way we parse
addresses from the OF device tree. If you can bisect a bit further
that would be good, although you may strike problems between the 401d
and 6237 commits I mentioned above.
What I have right now is this, which got me in a non-compiling state.
I will pick the udbg stuff and apply the relevant changes to -git5.
I tried with CONFIG_BOOTX_TEXT disabled. same result. This is the list
of patches I used on top of 2.6.15:
patches.kernel.org/patch-2.6.15-git5
patches.suse/get_cramfs_inode-revert.patch
patches.suse/suse-ppc-legacy-io.patch
patches.arch/0022-powerpc-incorrect-rmo_top-handling-in-prom_init.txt
patches.suse/9100b205fdc70b300894954ebebbf2709c5ed525.patch
patches.suse/3d1229d6ae92ed1994f4411b8493327ef8f4b76f.patch
patches.suse/d1405b869850982f05c7ec0d3f137ca27588192f.patch
patches.suse/463ce0e103f419f51b1769111e73fe8bb305d0ec.patch
patches.suse/51d3082fe6e55aecfa17113dbe98077c749f724c.patch
patches.suse/31df1678d7732b94178a6e457ed6666e4431212f.patch
patches.suse/8dacaedf04467e32c50148751a96150e73323cdc.patch
patches.suse/52020d2bda9fe447bb50674a2e39e4064b6a10b5.patch
51d3082fe6e55aecfa17113dbe98077c749f724c enabled powersave_nap
unconditionally. But early G3 cpus can not handle it.
I sent a patch in another thread.