Again crash. Need more posts of panic or this message have full info
that needed to fix bug?
Hi,
Please, try to not create new threads each time: reply to the previous
one if you have something new. And this one doesn't seem to show more.
You have written earlier it's '1-5 times on week', so you should have
got used to it a little, so no need to panic...
You would better try to write if there was some previous kernel
version, which worked better for you?
It seems, there could be some locking problem and your script could
mess htb queue from the second cpu (or is interrupted). Probably you
could have something more in logs about this, and maybe even this
script could be helpful (you should mask secret things only). Or maybe
you could try to add some echos to this script to figure out the
part which is the most suspected. Of course .config and dmesg (zipped)
could be helpful too.
If it's possible you can try it shortly without e.g. netconsole or
even without CONFIG_SMP.
Regards,
Jarek P.
On Wed, Aug 29, 2007 at 01:34:47PM +0200, Jarek Poplawski wrote:
quoted
On 29-08-2007 11:34, Badalian Vyacheslav wrote:
quoted
Again crash. Need more posts of panic or this message have full info
that needed to fix bug?
...
quoted
If it's possible you can try it shortly without e.g. netconsole or
even without CONFIG_SMP.
...or maybe even dare to try something current like 2.6.23-rc4?
Jarek P.
we have this kernel panic (then delete HTB) at all 2.6.18-x versions.
on older kernel (2.6.x) we have another panic (then delete tc filter)...
summary we have TC panics 1 year ago ;) Sysctl option "reboot on panic"
save us. Now we up 2 backup computers and may try any patches to fix
this problem.
Also on 2.6.22 have strange dead. Black screen, no response to keyboard,
no info in netconsole, HardDisk led is stable red. "Black Dead"
On Wed, Aug 29, 2007 at 04:53:52PM +0400, Badalian Vyacheslav wrote:
...
we have this kernel panic (then delete HTB) at all 2.6.18-x versions.
on older kernel (2.6.x) we have another panic (then delete tc filter)...
summary we have TC panics 1 year ago ;) Sysctl option "reboot on panic"
I'm not sure: do you mean it was less often? Did you try to report it
here? (Delete HTB: qdisc or classes?)
save us. Now we up 2 backup computers and may try any patches to fix
this problem.
Also on 2.6.22 have strange dead. Black screen, no response to keyboard,
no info in netconsole, HardDisk led is stable red. "Black Dead"
Yes, with all black it could be harder... Maybe 'set -x' at the
beginning (after #!/bin/sh line) of a script could manage to save
something before reboot or send with netconsole (but there could be
a lot of this with a large script...). Netconsole could be troublesome
too. One HTB deadlock problem during similar deleting was fixed in
2.6.23-rc (HTB timer problem) but the log was different. Anyway,
we probably need some more information (and trying).
Jarek P.
On Wed, Aug 29, 2007 at 04:53:52PM +0400, Badalian Vyacheslav wrote:
...
quoted
we have this kernel panic (then delete HTB) at all 2.6.18-x versions.
on older kernel (2.6.x) we have another panic (then delete tc filter)...
summary we have TC panics 1 year ago ;) Sysctl option "reboot on panic"
I'm not sure: do you mean it was less often? Did you try to report it
here? (Delete HTB: qdisc or classes?)
i was can't catch bug. now i have configured netconsole to catch panics.
for every clinet run command like:
### command to recreate HTB
tc filter del dev eth1 protocol ip parent 1:0 prio 5 handle 4:9:a1 u32
tc filter del dev eth0 protocol ip parent 1:0 prio 5 handle 4:9:a1 u32
tc class del dev eth1 parent 1:6 classid 1:1c
tc class del dev eth0 parent 1:6 classid 1:1c
tc class del dev eth1 parent 1:8 classid 1:1c
tc class del dev eth0 parent 1:8 classid 1:1c
tc class add dev eth1 parent 1:8 classid 1:1c htb rate 1kbit ceil
5000kbit burst 1b cburst 625b quantum 1500
tc qdisc add dev eth1 parent 1:1c handle 28 sfq perturb 10
tc class add dev eth0 parent 1:8 classid 1:1c htb rate 1kbit ceil
5000kbit burst 1b cburst 625b quantum 1500
tc qdisc add dev eth0 parent 1:1c handle 28 sfq perturb 10
tc filter add dev eth1 protocol ip parent 1:0 prio 5 handle 8:73:6 u32
ht 8:73: match ip dst 87.255.6.115 flowid 1:1c
tc filter add dev eth0 protocol ip parent 1:0 prio 5 handle 8:73:6 u32
ht 8:73: match ip src 87.255.6.115 flowid 1:1c
tc filter add dev eth1 protocol ip parent 1:0 prio 5 handle 4:9:a1 u32
ht 4:9: match ip dst 172.16.161.9 flowid 1:1c
tc filter add dev eth0 protocol ip parent 1:0 prio 5 handle 4:9:a1 u32
ht 4:9: match ip src 172.16.161.9 flowid 1:1c
###
i try delete class "parent 1:6" and "parent 1:8" because i not know
what parent was. (limited speed class 1:6 or unlimited speed class 1:8)
if i delete class - qdisc delete automatic.
if computer not have traffic - all normal (test system work 3 week).
if we have lot of traffic system have 1-5 kernel panics at week.
quoted
save us. Now we up 2 backup computers and may try any patches to fix
this problem.
Also on 2.6.22 have strange dead. Black screen, no response to keyboard,
no info in netconsole, HardDisk led is stable red. "Black Dead"
Yes, with all black it could be harder... Maybe 'set -x' at the
beginning (after #!/bin/sh line) of a script could manage to save
something before reboot or send with netconsole (but there could be
a lot of this with a large script...). Netconsole could be troublesome
too. One HTB deadlock problem during similar deleting was fixed in
2.6.23-rc (HTB timer problem) but the log was different. Anyway,
we probably need some more information (and trying).
In my desktop system i have "Black dead" (2.6.22-r5) All freeze (on
monitor KDE desctop. mouse, keyboard, network and other not work. HDD
led is on. No panics.)
Say that info you need. I will try get it.
PS. And also have we have strange bug in another computer (2.6.22-r5).
Have computer XEON_CPUx2 (4 CPU)
after boot have CPU0 and CPU3 SI = ~50%
after some time CPU0 SI = 0% and ksoftirqd/2 process have 100% cpu usage!
nat-new ~ # cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 403 0 0 0 IO-APIC-edge timer
1: 448 0 0 0 IO-APIC-edge i8042
6: 3 0 0 0 IO-APIC-edge floppy
8: 3 0 0 0 IO-APIC-edge rtc
9: 18 0 0 0 IO-APIC-fasteoi acpi
12: 4 0 0 0 IO-APIC-edge i8042
16: 100838998 0 656832858 0 IO-APIC-fasteoi eth0
17: 756133415 0 124233955 1 IO-APIC-fasteoi eth1
18: 0 0 0 0 IO-APIC-fasteoi
uhci_hcd:usb1
19: 27167 0 0 0 IO-APIC-fasteoi gdth
20: 0 0 0 0 IO-APIC-fasteoi
uhci_hcd:usb2
NMI: 0 0 0 0
LOC: 89312505 89314019 89310139 89313972
ERR: 0
MIS: 0
changes only LOC interrupts!
Maybe its info intresting for you. =)
Best regals.
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
On Thu, Aug 30, 2007 at 12:16:32AM +0400, slavon@bigtelecom.ru wrote:
Quoting Jarek Poplawski [off-list ref]:
quoted
On Wed, Aug 29, 2007 at 04:53:52PM +0400, Badalian Vyacheslav wrote:
...
quoted
we have this kernel panic (then delete HTB) at all 2.6.18-x versions.
on older kernel (2.6.x) we have another panic (then delete tc filter)...
summary we have TC panics 1 year ago ;) Sysctl option "reboot on panic"
I'm not sure: do you mean it was less often? Did you try to report it
here? (Delete HTB: qdisc or classes?)
i was can't catch bug. now i have configured netconsole to catch panics.
for every clinet run command like:
If some error repeats you should report it even without logs. Sometimes
people here could help to catch this, but at least they know something
is wrong around and look at the code more carefully.
### command to recreate HTB
tc filter del dev eth1 protocol ip parent 1:0 prio 5 handle 4:9:a1 u32
...
I need more time to think about it.
In my desktop system i have "Black dead" (2.6.22-r5) All freeze (on
monitor KDE desctop. mouse, keyboard, network and other not work. HDD
led is on. No panics.)
Say that info you need. I will try get it.
I still think, at least .config and dmesg could be interesting.
PS. And also have we have strange bug in another computer (2.6.22-r5).
Have computer XEON_CPUx2 (4 CPU)
after boot have CPU0 and CPU3 SI = ~50%
after some time CPU0 SI = 0% and ksoftirqd/2 process have 100% cpu usage!
nat-new ~ # cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 403 0 0 0 IO-APIC-edge timer
...
LOC: 89312505 89314019 89310139 89313972
ERR: 0
MIS: 0
changes only LOC interrupts!
Maybe its info intresting for you. =)
Yes. It seems something loops or breaks with disabled interrupts. If
it's possible on this box try this 2.6.23-rc4 (and as minimum devices
and as maximum debug options in config as possible). Without anything
in logs or from the screen it could be hard, so maybe you need to
experiment with different configs and kernel versions.
Thanks,
Jarek P.
PS: if it's possible you can try this patch maybe with some fake load
plus these tc scripts (for testing only, linux 2.6.22.5).
---
diff -Nurp linux-2.6.22.5-/net/sched/sch_htb.c linux-2.6.22.5/net/sched/sch_htb.c
@@ -688,7 +696,11 @@ static void htb_rate_timer(unsigned long/* lock queue so that we can muck with it */-spin_lock_bh(&sch->dev->queue_lock);+if(!spin_trylock_bh(&sch->dev->queue_lock)){+q->rttim.expires=jiffies+1;+add_timer(&q->rttim);+return;+}q->rttim.expires=jiffies+HZ;add_timer(&q->rttim);
@@ -1306,7 +1318,8 @@ static void htb_destroy(struct Qdisc *scqdisc_watchdog_cancel(&q->watchdog);#ifdef HTB_RATECM-del_timer_sync(&q->rttim);+if(!del_timer_sync(&q->rttim))+del_timer(&q->rttim);#endif/* This line used to be after htb_destroy_class call belowandsurprisinglyitworkedin2.4.Butitmustprecedeit
On Thu, Aug 30, 2007 at 08:31:10AM +0200, Jarek Poplawski wrote:
On Thu, Aug 30, 2007 at 12:16:32AM +0400, slavon@bigtelecom.ru wrote:
...
quoted
PS. And also have we have strange bug in another computer (2.6.22-r5).
Have computer XEON_CPUx2 (4 CPU)
after boot have CPU0 and CPU3 SI = ~50%
after some time CPU0 SI = 0% and ksoftirqd/2 process have 100% cpu usage!
nat-new ~ # cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 403 0 0 0 IO-APIC-edge timer
...
quoted
LOC: 89312505 89314019 89310139 89313972
ERR: 0
MIS: 0
changes only LOC interrupts!
Maybe its info intresting for you. =)
Yes. It seems something loops or breaks with disabled interrupts. If
On the other hand disabling local interrupts shouldn't be enough here,
so it's really strange... Did you get this remotely? Are you sure LOC
only? (Anyway this 2.6.23-rc4 should be interesting.)
Jarek P.
On Thu, Aug 30, 2007 at 08:31:10AM +0200, Jarek Poplawski wrote:
quoted
On Thu, Aug 30, 2007 at 12:16:32AM +0400, slavon@bigtelecom.ru wrote:
...
quoted
quoted
PS. And also have we have strange bug in another computer (2.6.22-r5).
Have computer XEON_CPUx2 (4 CPU)
after boot have CPU0 and CPU3 SI = ~50%
after some time CPU0 SI = 0% and ksoftirqd/2 process have 100% cpu usage!
nat-new ~ # cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 403 0 0 0 IO-APIC-edge timer
...
quoted
LOC: 89312505 89314019 89310139 89313972
ERR: 0
MIS: 0
changes only LOC interrupts!
Maybe its info intresting for you. =)
Yes. It seems something loops or breaks with disabled interrupts. If
On the other hand disabling local interrupts shouldn't be enough here,
so it's really strange... Did you get this remotely? Are you sure LOC
only? (Anyway this 2.6.23-rc4 should be interesting.)
Jarek P.
Only LOC changes... icmp answer = 50-70ms... after 1-2 hours traffic
level is down and SI on CPU0 and CPU2 change to above 50%. ksoftirqd
free CPU usage. I have this bug 3-4 times in week. If you need info what
i can see only in bug still processing - i may try get this info for you.
maybe help:
1U server INTEL, mb se7501w2
nat-new ~ # lspci
00:00.0 Host bridge: Intel Corporation E7501 Memory Controller Hub (rev 01)
00:00.1 Class ff00: Intel Corporation E7500/E7501 Host RASUM Controller
(rev 01)
00:03.0 PCI bridge: Intel Corporation E7500/E7501 Hub Interface C
PCI-to-PCI Bridge (rev 01)
00:03.1 Class ff00: Intel Corporation E7500/E7501 Hub Interface C RASUM
Controller (rev 01)
00:1d.0 USB Controller: Intel Corporation 82801CA/CAM USB Controller #1
(rev 02)
00:1d.1 USB Controller: Intel Corporation 82801CA/CAM USB Controller #2
(rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 42)
00:1f.0 ISA bridge: Intel Corporation 82801CA LPC Interface Controller
(rev 02)
00:1f.1 IDE interface: Intel Corporation 82801CA Ultra ATA Storage
Controller (rev 02)
00:1f.3 SMBus: Intel Corporation 82801CA/CAM SMBus Controller (rev 02)
01:0c.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
02:1c.0 PIC: Intel Corporation 82870P2 P64H2 I/OxAPIC (rev 04)
02:1d.0 PCI bridge: Intel Corporation 82870P2 P64H2 Hub PCI Bridge (rev 04)
02:1e.0 PIC: Intel Corporation 82870P2 P64H2 I/OxAPIC (rev 04)
02:1f.0 PCI bridge: Intel Corporation 82870P2 P64H2 Hub PCI Bridge (rev 04)
03:07.0 Ethernet controller: Intel Corporation 82546EB Gigabit Ethernet
Controller (Copper) (rev 01)
03:07.1 Ethernet controller: Intel Corporation 82546EB Gigabit Ethernet
Controller (Copper) (rev 01)
04:08.0 RAID bus controller: Intel Corporation RAID Controller
On Thu, Aug 30, 2007 at 01:09:11PM +0400, Badalian Vyacheslav wrote:
Jarek Poplawski ??????????:
...
quoted
On the other hand disabling local interrupts shouldn't be enough here,
so it's really strange... Did you get this remotely? Are you sure LOC
only? (Anyway this 2.6.23-rc4 should be interesting.)
...
Only LOC changes... icmp answer = 50-70ms... after 1-2 hours traffic
level is down and SI on CPU0 and CPU2 change to above 50%. ksoftirqd
free CPU usage. I have this bug 3-4 times in week. If you need info what
i can see only in bug still processing - i may try get this info for you.
Any additional info could be helpful. I'm not sure if all these
computers do similar htb processing, or it's another problem?
As I've written before htb before 2.6.23-rc1 has a problem with
timer lockup during qdisc_destroy, so softirqs would be hit.
If it's htb's fault 2.6.23-rc4 or my testing patch should help.
I try to find in htb code another weak points. BTW, if during
such lockups any processes are killed 'by hand' etc., without
restarting the whole system, please let us know.
On Thu, Aug 30, 2007 at 01:09:11PM +0400, Badalian Vyacheslav wrote:
quoted
Jarek Poplawski ??????????:
...
quoted
quoted
On the other hand disabling local interrupts shouldn't be enough here,
so it's really strange... Did you get this remotely? Are you sure LOC
only? (Anyway this 2.6.23-rc4 should be interesting.)
...
quoted
Only LOC changes... icmp answer = 50-70ms... after 1-2 hours traffic
level is down and SI on CPU0 and CPU2 change to above 50%. ksoftirqd
free CPU usage. I have this bug 3-4 times in week. If you need info what
i can see only in bug still processing - i may try get this info for you.
Any additional info could be helpful. I'm not sure if all these
computers do similar htb processing, or it's another problem?
As I've written before htb before 2.6.23-rc1 has a problem with
timer lockup during qdisc_destroy, so softirqs would be hit.
If it's htb's fault 2.6.23-rc4 or my testing patch should help.
I try to find in htb code another weak points. BTW, if during
such lockups any processes are killed 'by hand' etc., without
restarting the whole system, please let us know.
I will try patch ;)
"CPU Si" is another bug at another computer. - NAT
NAT: simple have iptables rules (NAT) and ipcad to generate netflow.
FW: have iptables rules to close FORWARD access and TC rules for shape.
Scheme of network:
CORE <-> (FW 2.6.22 and FW-BackupLink) <-> NAT <-> OUT_ROUTER =
500mbs/500mbs traffic
FW have kernel panics if i try Delete HTB class (i will try patch)
NAT sometime have "si usage bug" if traffic more what 250mbs.
txt.txt have dmesg and lspci -vv info for NAT
Sorry for my English spell
I try you patch. Also i try add more debug options to kernel. I catch
(BUG: spinlock lockup on CPU#3, tc/6403, f742e200)
All info in file. Ready for next patch ;)
quoted
Jarek Poplawski ??????????:
...
quoted
quoted
On the other hand disabling local interrupts shouldn't be enough here,
so it's really strange... Did you get this remotely? Are you sure LOC
only? (Anyway this 2.6.23-rc4 should be interesting.)
...
quoted
Only LOC changes... icmp answer = 50-70ms... after 1-2 hours traffic
level is down and SI on CPU0 and CPU2 change to above 50%. ksoftirqd
free CPU usage. I have this bug 3-4 times in week. If you need info what
i can see only in bug still processing - i may try get this info for you.
Any additional info could be helpful. I'm not sure if all these
computers do similar htb processing, or it's another problem?
As I've written before htb before 2.6.23-rc1 has a problem with
timer lockup during qdisc_destroy, so softirqs would be hit.
If it's htb's fault 2.6.23-rc4 or my testing patch should help.
I try to find in htb code another weak points. BTW, if during
such lockups any processes are killed 'by hand' etc., without
restarting the whole system, please let us know.
On Fri, Aug 31, 2007 at 11:04:21AM +0400, Badalian Vyacheslav wrote:
I try you patch. Also i try add more debug options to kernel. I catch
(BUG: spinlock lockup on CPU#3, tc/6403, f742e200)
All info in file. Ready for next patch ;)
I've to look at this, but actually my patch wrongly added this one
warning. I hope you did this on a testing machine...
Anyway there could be something new but I need some time. If possible,
2.6.23-rc4 should be the best thing for testing yet.
Thanks,
Jarek P.
i not have testing mashine.
we have 2 mashine and dynamic routing. if 1 mashine down - all traffic
go to second mashine.
I can test is on this mashines but i need that testing mashine will
reboot on kernel panic (sysctl message). No freezes =)
Ok. i try 2.6.23-rc4.
On Fri, Aug 31, 2007 at 11:04:21AM +0400, Badalian Vyacheslav wrote:
quoted
I try you patch. Also i try add more debug options to kernel. I catch
(BUG: spinlock lockup on CPU#3, tc/6403, f742e200)
All info in file. Ready for next patch ;)
I've to look at this, but actually my patch wrongly added this one
warning. I hope you did this on a testing machine...
Anyway there could be something new but I need some time. If possible,
2.6.23-rc4 should be the best thing for testing yet.
Thanks,
Jarek P.
On Fri, Aug 31, 2007 at 12:25:22PM +0400, Badalian Vyacheslav wrote:
i not have testing mashine.
we have 2 mashine and dynamic routing. if 1 mashine down - all traffic
go to second mashine.
I can test is on this mashines but i need that testing mashine will
reboot on kernel panic (sysctl message). No freezes =)
Ok. i try 2.6.23-rc4.
quoted
On Fri, Aug 31, 2007 at 11:04:21AM +0400, Badalian Vyacheslav wrote:
quoted
I try you patch. Also i try add more debug options to kernel. I catch
(BUG: spinlock lockup on CPU#3, tc/6403, f742e200)
BTW, I think after this BUG could be something more about other CPUs.
Could you check this?
Jarek P.
PS: If it's possible try to not cut too much: if there is something
confidential mask this with some XXX or mark the cut with... There
could matter what else is run at the same time (including other
driver's warnings). If you think it's too much for a list you can
send it to me only.
On Fri, Aug 31, 2007 at 12:25:22PM +0400, Badalian Vyacheslav wrote:
i not have testing mashine.
we have 2 mashine and dynamic routing. if 1 mashine down - all traffic
go to second mashine.
I can test is on this mashines but i need that testing mashine will
reboot on kernel panic (sysctl message). No freezes =)
Ok. i try 2.6.23-rc4.
...but without testing machine it can be too much risk! New versions
of kernel can break your applications (sometimes they should be at
least rebuilded).
So, maybe you would better try this, 'less testing', version of my patch:
Jarek P.
---
diff -Nurp linux-2.6.22.5-/net/sched/sch_htb.c linux-2.6.22.5/net/sched/sch_htb.c
@@ -688,7 +688,11 @@ static void htb_rate_timer(unsigned long/* lock queue so that we can muck with it */-spin_lock_bh(&sch->dev->queue_lock);+if(!spin_trylock_bh(&sch->dev->queue_lock)){+q->rttim.expires=jiffies+1;+add_timer(&q->rttim);+return;+}q->rttim.expires=jiffies+HZ;add_timer(&q->rttim);
@@ -1306,7 +1310,8 @@ static void htb_destroy(struct Qdisc *scqdisc_watchdog_cancel(&q->watchdog);#ifdef HTB_RATECM-del_timer_sync(&q->rttim);+if(!del_timer_sync(&q->rttim))+del_timer(&q->rttim);#endif/* This line used to be after htb_destroy_class call belowandsurprisinglyitworkedin2.4.Butitmustprecedeit
i not have testing mashine.
we have 2 mashine and dynamic routing. if 1 mashine down - all traffic
go to second mashine.
I can test is on this mashines but i need that testing mashine will
reboot on kernel panic (sysctl message). No freezes =)
Ok. i try 2.6.23-rc4.
...but without testing machine it can be too much risk! New versions
of kernel can break your applications (sometimes they should be at
least rebuilded).
So, maybe you would better try this, 'less testing', version of my patch:
I risk only if RC kernel broke hardware. Mashines use only iptables and
TC. All Scrips backuped.
But if i need to reboot PC by hand (if it freeze) i need drive to
servers-room =)
> BTW, I think after this BUG could be something more about other CPUs.
> Could you check this?
I send to you all info that catch NETCONSOLE. i don't cut any info...
Now 1 mashine work at *"2.6.23-rc4-git2
<http://kernel.org/pub/linux/kernel/v2.6/snapshots/patch-2.6.23-rc4-git2.bz2>"
Backup mashine use 2.6.18*
On Fri, Aug 31, 2007 at 01:33:04PM +0400, Badalian Vyacheslav wrote:
quoted
quoted
i not have testing mashine.
we have 2 mashine and dynamic routing. if 1 mashine down - all traffic
go to second mashine.
I can test is on this mashines but i need that testing mashine will
reboot on kernel panic (sysctl message). No freezes =)
Ok. i try 2.6.23-rc4.
...but without testing machine it can be too much risk! New versions
of kernel can break your applications (sometimes they should be at
least rebuilded).
So, maybe you would better try this, 'less testing', version of my patch:
I risk only if RC kernel broke hardware. Mashines use only iptables and
TC. All Scrips backuped.
But sometime a new kernel can break binary compatibility with previous
one (e.g. after data structures change) and e.g. iptables or iproute
tools stop working or work in an unpredictable way. There were a few
such changes before 2.6.20 - I don't track current changes too much.
And I'm sure your system uses much more than iptables or TC, even
without your knowledge.
But if i need to reboot PC by hand (if it freeze) i need drive to
servers-room =)
Of course, if previous kernel always boots by default, or even there
is a possibility to use for testing different partition with a copy
of main system this risk should be much smaller.
quoted
BTW, I think after this BUG could be something more about other CPUs.
Could you check this?
I send to you all info that catch NETCONSOLE. i don't cut any info...
So, it seems something was broken. But, I meant, there can be sometimes
interesting things a few lines before or after the infos too.
But sometime a new kernel can break binary compatibility with previous
one (e.g. after data structures change) and e.g. iptables or iproute
tools stop working or work in an unpredictable way. There were a few
such changes before 2.6.20 - I don't track current changes too much.
And I'm sure your system uses much more than iptables or TC, even
without your knowledge.
I can rollback =) Backup system get all functions. I can risk for one
system of two for get and fix all bugs.
Dynamic routing work fine and automatic switch between systems = 1-4
seconds.
quoted
I send to you all info that catch NETCONSOLE. i don't cut any info...
So, it seems something was broken. But, I meant, there can be sometimes
interesting things a few lines before or after the infos too.
I can only see that say netconsole. If i look to monitor i look last
lines. last line is "====...". Scrolling not work
netconsole run as module and start after system do full load. Then
netconsole is up - i run generator of tc scripts.
BTW, -git versions are usually more risky than -rc. And, maybe, let
this 2.6.18 better stay away from this testing...
I look changes between 2.6.23-rc4 and 2.6.23-rc4-git2 and think that
paches look good and no do any critical things =)
Now i do many script runs to simulate bug. if i get it on 2.6.23-rc4 - i
post it here. 2.6.23-rc4 not have htb_timer function.
Badalian Vyacheslav
On Fri, Aug 31, 2007 at 02:59:55PM +0400, Badalian Vyacheslav wrote:
May be this bug eq "[PATCH] [NET_SCHED] sch_prio.c: remove duplicate
call of tc_classify()"?
quoted
I get kernel panic on 2.6.23-rc4-git2
This is netconsole log!
...
So, it looks like you have found a really new (unknown) HTB bug,
congratulations! We've only to find where is it hidden now...
I don't think tc_classify could do such a harm to HTB, and can't
see similar double calling as in sch_prio.
Jarek P.
On Fri, Aug 31, 2007 at 02:59:55PM +0400, Badalian Vyacheslav wrote:
quoted
May be this bug eq "[PATCH] [NET_SCHED] sch_prio.c: remove duplicate
call of tc_classify()"?
quoted
I get kernel panic on 2.6.23-rc4-git2
This is netconsole log!
...
So, it looks like you have found a really new (unknown) HTB bug,
congratulations! We've only to find where is it hidden now...
I don't think tc_classify could do such a harm to HTB, and can't
see similar double calling as in sch_prio.
Jarek P.
Great! +)
I ready for patching and testing anything patches to find and fix problem =)
Also i remember that bug up only if i delete HTB class and mashine have
lot of traffic.
I get statistic for last panic:
Script try to delete 25278 classes, Creating 8360 classes. Adding 6732
filters
Panic after 27 restarts of script.
On Fri, Aug 31, 2007 at 02:48:31PM +0400, Badalian Vyacheslav wrote:
...
I can only see that say netconsole. If i look to monitor i look last
lines. last line is "====...". Scrolling not work
netconsole run as module and start after system do full load. Then
netconsole is up - i run generator of tc scripts.
It would be interesting to know if this bug did ever happen without
netconsole, or, if it wasn't tested this way, if it's possible to
do such a test (not necessarily today)?
quoted
BTW, -git versions are usually more risky than -rc. And, maybe, let
this 2.6.18 better stay away from this testing...
I look changes between 2.6.23-rc4 and 2.6.23-rc4-git2 and think that
paches look good and no do any critical things =)
Now i do many script runs to simulate bug. if i get it on 2.6.23-rc4 - i
post it here. 2.6.23-rc4 not have htb_timer function.
I'll not be able to assist you until monday (but I'll try to look
into the code and maybe to prepare some new patch - but it needs
a lot of checking to not add too much of this locking as well).
I think you can stay with a kernel whichever you like - I'm not sure
any config changes or even more debugging can change much, but maybe
I'm wrong. It looks to me like some locking is missing or interrupted.
If you are working weekends and find something new, don't wait: maybe
somebody else here could be interested too.
Cheers,
Jarek P.
Ok =) I hope in next week you found bug place and fix it!
PS. if you ask where i can read "kernel panic dump logic" literature and
try find bugline in code.
I read dump and see that bug in function "rb_insert_color" + some shift
(in asm?) that called from htb_dequeue? But in htb_dequeue not have
calling rb_insert_color =( Or some nodes in trace was skipped?
Its for change up my education ;)
I'll not be able to assist you until monday (but I'll try to look
into the code and maybe to prepare some new patch - but it needs
a lot of checking to not add too much of this locking as well).
I think you can stay with a kernel whichever you like - I'm not sure
any config changes or even more debugging can change much, but maybe
I'm wrong. It looks to me like some locking is missing or interrupted.
If you are working weekends and find something new, don't wait: maybe
somebody else here could be interested too.
Cheers,
Jarek P.
I found that bug in this place
(gdb) l *0xc01c8973
0xc01c8973 is in rb_insert_color (lib/rbtree.c:80).
75
76 while ((parent = rb_parent(node)) && rb_is_red(parent))
77 {
78 gparent = rb_parent(parent);
79
80 if (parent == gparent->rb_left)
81 {
82 {
83 register struct rb_node *uncle =
gparent->rb_right;
84 if (uncle && rb_is_red(uncle))
if i not wrong understand message "unable to handle kernel NULL pointer
dereference at virtual address 00000008" its was known that "gparent ==
Null"?
Or i hope or i try find a mare's-nest?
Ok =) I hope in next week you found bug place and fix it!
PS. if you ask where i can read "kernel panic dump logic" literature
and try find bugline in code.
I read dump and see that bug in function "rb_insert_color" + some
shift (in asm?) that called from htb_dequeue? But in htb_dequeue not
have calling rb_insert_color =( Or some nodes in trace was skipped?
Its for change up my education ;)
quoted
I'll not be able to assist you until monday (but I'll try to look
into the code and maybe to prepare some new patch - but it needs
a lot of checking to not add too much of this locking as well).
I think you can stay with a kernel whichever you like - I'm not sure
any config changes or even more debugging can change much, but maybe
I'm wrong. It looks to me like some locking is missing or interrupted.
If you are working weekends and find something new, don't wait: maybe
somebody else here could be interested too.
Cheers,
Jarek P.
-
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Aug 31, 2007 at 06:51:24PM +0400, Badalian Vyacheslav wrote:
I found that bug in this place
(gdb) l *0xc01c8973
0xc01c8973 is in rb_insert_color (lib/rbtree.c:80).
...
if i not wrong understand message "unable to handle kernel NULL pointer
dereference at virtual address 00000008" its was known that "gparent ==
Null"?
Or i hope or i try find a mare's-nest?
Your errors trigger in rbtree, which does indexing for HTB, but since
it's something quite rare I think there is a very small probability
that it's caused by HTB class/level handling (but it's possible, too),
but more probable (to me) these indexes are corrupted by something e.g.
like accessing them without proper locking.
Below I attach a patch for testing: it adds some lock debugging (plus
one place: htb_put is locked). There is mainly checking of locks
needed for writing to rbtree, but it doesn't check all readings yet,
so there will be still something to check if this patch doesn't help
to find anything.
It should be applied to 2.6.23-rc4, but if you prefer 2.6.22.5 version
let me know (BTW, I hope you let us know if you have to apply any
other patches/changes to these kernels...).
quoted
Ok =) I hope in next week you found bug place and fix it!
PS. if you ask where i can read "kernel panic dump logic" literature
and try find bugline in code.
I read dump and see that bug in function "rb_insert_color" + some
shift (in asm?) that called from htb_dequeue? But in htb_dequeue not
have calling rb_insert_color =( Or some nodes in trace was skipped?
Its for change up my education ;)
@@ -52,6 +52,7 @@onelessthantheirparent.*/+#define DEBUG_HTB#define HTB_HSIZE 16 /* classid hash size */#define HTB_HYSTERESIS 1 /* whether to use mode hysteresis for speedup */#define HTB_VER 0x30011 /* major must be matched with number suplied by TC as version */
@@ -127,6 +128,9 @@ struct htb_class {intprio;/* For parent to leaf return possible here */intquantum;/* we do backup. Finally full replacement *//* of un.leaf originals should be done. */+#ifdef DEBUG_HTB+structQdisc*sch;+#endif};staticinlinelongL2T(structhtb_class*cl,structqdisc_rate_table*rate,
@@ -175,6 +179,23 @@ struct htb_sched {longdirect_pkts;};+#ifdef DEBUG_HTB+staticinlineinthtb_queue_locked(structhtb_class*cl)+{+if(cl->sch){+if(!spin_is_locked(&cl->sch->dev->queue_lock)||+!in_softirq()){+cl->sch=NULL;+return0;+}+}+return1;+}+#define DEBUG_QUEUE_LOCKED(cl) WARN_ON(!htb_queue_locked(cl))+#else+#define DEBUG_QUEUE_LOCKED(dev) do { } while (0)+#endif+/* compute hash of size HTB_HSIZE for given handle */staticinlineinthtb_hash(u32h){
@@ -1367,6 +1399,9 @@ static int htb_change_class(struct Qdiscfor(prio=0;prio<TC_HTB_NUMPRIO;prio++)RB_CLEAR_NODE(&cl->node[prio]);+#ifdef DEBUG_HTB+cl->sch=sch;+#endif/* create leaf qdisc early because it uses kmalloc(GFP_KERNEL)sothatcan'tbeusedinsideofsch_tree_lock--thankstoKarlisPeisenieks*/
Your errors trigger in rbtree, which does indexing for HTB, but since
it's something quite rare I think there is a very small probability
that it's caused by HTB class/level handling (but it's possible, too),
but more probable (to me) these indexes are corrupted by something e.g.
like accessing them without proper locking.
Below I attach a patch for testing: it adds some lock debugging (plus
one place: htb_put is locked). There is mainly checking of locks
needed for writing to rbtree, but it doesn't check all readings yet,
so there will be still something to check if this patch doesn't help
to find anything.
It should be applied to 2.6.23-rc4, but if you prefer 2.6.22.5 version
let me know (BTW, I hope you let us know if you have to apply any
other patches/changes to these kernels...).
Ok... i was apply patch and see that its say... thanks...
Maybe you see bug 2 (wrong level calculation) and 3 (class not leaf but
have qdisc) at
http://bugzilla.kernel.org/show_bug.cgi?id=8971
On Mon, Sep 03, 2007 at 12:31:39PM +0400, Badalian Vyacheslav wrote:
May you also see that i need change to fix this:
qdisc handle can >= 10 000
i have more then 10 000 qdiscs =(
As far as I know qdisc handle is hex, so you can have e.g.:
handle 999a (or a999 too). But, does it mean your kernel has
any changes around this? If so, please, describe them all (or
send your patch).
I've read your bugzilla log with: "2. HTB levels wrong calculate!",
but can't check this now: could you add there what are these levels
you can see after: tc -d class show dev eth0 ?
Jarek P.