[2.6.0-test9] QoS HTB crash...

9 messages, 3 authors, 2003-12-07 · open the first message on its own page

[2.6.0-test9] QoS HTB crash...

From: Daniel Blueman <hidden>
Date: 2003-10-30 16:10:41

With the LARTC 'wondershaper' HTB script [1] for good latency over ADSL, I
get an oops [3] when sending traffic via ppp0 (and when bringing the interface
down).

Kernel is 2.6.0-test9 and 'debug 3333333' appended to the 'tc' command, to
show HTB debug information [2] (not shown in [1]).

Please CC me when replying, and I can provide further details, debugging,
testing etc...this problem has been around for a while it seems.
--- [1] (relevant lines from http://lartc.org/wondershaper/)
tc qdisc add dev ppp0 root handle 1: htb default 20
tc class add dev ppp0 parent 1: classid 1:1 htb rate 210kbit burst 6k
tc class add dev ppp0 parent 1:1 classid 1:10 htb rate 210kbit burst 6k prio
1
tc class add dev ppp0 parent 1:1 classid 1:20 htb rate 189kbit burst 6k prio
2
tc class add dev ppp0 parent 1:1 classid 1:30 htb rate 168kbit burst 6k prio
2
--- [2] (HTB debug messages)
HTB init, kernel part version 3.13
htb_init sch=dc28e7f8 handle=10000 r2q=10
htb_dump sch=dc28e7f8, handle=10000
htb*g j=179519 lj=0
htb*r7 m=0
htb*r6 m=0
htb*r5 m=0
htb*r4 m=0
htb*r3 m=0
htb*r2 m=0
htb*r1 m=0
htb*r0 m=0
htb_get clid=10010 q=dc28e86c cl=00000000 ref=0
htb_get clid=10010 q=dc28e86c cl=00000000 ref=0
htb_get clid=10020 q=dc28e86c cl=00000000 ref=0
htb_get clid=10020 q=dc28e86c cl=00000000 ref=0
htb_get clid=10030 q=dc28e86c cl=00000000 ref=0
htb_get clid=10030 q=dc28e86c cl=00000000 ref=0
htb_tcf q=dc28e86c clid=0 fref=0 fl=00000000
htb_bind q=dc28e86c clid=10010 cl=00000000 fref=0
htb_tcf q=dc28e86c clid=0 fref=1 fl=df4ecd7c
htb_bind q=dc28e86c clid=10010 cl=00000000 fref=1
htb_tcf q=dc28e86c clid=0 fref=2 fl=df4ecd7c
htb_bind q=dc28e86c clid=10010 cl=00000000 fref=2
htb_tcf q=dc28e86c clid=0 fref=3 fl=df4ecd7c
htb_bind q=dc28e86c clid=10020 cl=00000000 fref=3
htb_reset sch=dc28e7f8, handle=10000
--- [3] (ksymoops-processed oops report)
Oops: 0000 [#1]
CPU:    0
EIP:    0060:[<c02cbbad>]    Not tainted
quoted
EIP; c02cbbad <htb_enqueue+ab/126>   <=====
 
quoted
ebx; ffffffff <__kernel_rt_sigreturn+1bbf/????>
ecx; dc28e86c <_end+1be20f60/3fb906f4>
esi; d9cf3f50 <_end+19886644/3fb906f4>
edi; dc28e7f8 <_end+1be20eec/3fb906f4>
ebp; dbef5ab8 <_end+1ba881ac/3fb906f4>
esp; dbef5a9c <_end+1ba88190/3fb906f4>
 
Trace; c02b3570 <dev_queue_xmit+593/73a>
Trace; c02e9c98 <ip_finish_output2+0/1c4>
Trace; c02e9d97 <ip_finish_output2+ff/1c4>
Trace; c02c0259 <nf_hook_slow+ef/13a>
Trace; c02e9c98 <ip_finish_output2+0/1c4>
Trace; c02e76d7 <ip_finish_output+21d/222>
Trace; c02e9c98 <ip_finish_output2+0/1c4>
Trace; c02bff12 <nf_iterate+6b/9f>
Trace; c02e9c80 <dst_output+15/2d>
Trace; c02c0259 <nf_hook_slow+ef/13a>
Trace; c02e9c6b <dst_output+0/2d>
Trace; c02e9620 <ip_push_pending_frames+3ac/408>
Trace; c02e9c6b <dst_output+0/2d>
Trace; c030da69 <udp_push_pending_frames+134/23c>
Trace; c030e1b4 <udp_sendmsg+60d/f3d>
Trace; c031900a <inet_sendmsg+4b/56>
Trace; c02aa4cc <sock_sendmsg+92/af>
Trace; c02aa582 <sock_recvmsg+99/b4>
Trace; c02a9f68 <move_addr_to_kernel+7e/a7>
Trace; c02b01a0 <verify_iovec+80/fa>
Trace; c02ac13d <sys_sendmsg+256/2f4>
Trace; c0141e28 <filemap_nopage+21c/2f9>
Trace; c0155610 <do_no_page+29b/631>
Trace; c0155c54 <handle_mm_fault+132/30c>
Trace; c018becd <inode_update_time+8e/b9>
Trace; c0119419 <do_page_fault+34b/56a>
Trace; c02ac67e <sys_socketcall+261/29a>
Trace; c0164327 <sys_write+5b/5d>
Trace; c010a3eb <syscall_call+7/b>
 
Code;  c02cbbad <htb_enqueue+ab/126>
00000000 <_EIP>:
Code;  c02cbbad <htb_enqueue+ab/126>   <=====
   0:   8b 43 04                  mov    0x4(%ebx),%eax   <=====
Code;  c02cbbb0 <htb_enqueue+ae/126>
   3:   89 44 24 04               mov    %eax,0x4(%esp,1)
Code;  c02cbbb4 <htb_enqueue+b2/126>
   7:   c7 04 24 2b 30 38 c0      movl   $0xc038302b,(%esp,1)
Code;  c02cbbbb <htb_enqueue+b9/126>
   e:   e8 e3 6e e5 ff            call   ffe56ef6 <_EIP+0xffe56ef6>
Code;  c02cbbc0 <htb_enqueue+be/126>
  13:   31 00                     xor    %eax,(%eax)
 
 <0>Kernel panic: Fatal exception in interrupt

-- 
Daniel J Blueman

NEU FÜR ALLE - GMX MediaCenter - für Fotos, Musik, Dateien...
Fotoalbum, File Sharing, MMS, Multimedia-Gruß, GMX FotoService

Jetzt kostenlos anmelden unter http://www.gmx.net
+++ GMX - die erste Adresse für Mail, Message, More! +++

Re: [2.6.0-test9] QoS HTB crash...

From: devik <hidden>
Date: 2003-10-30 20:08:27

Hi,

thanks for the report. I know that there is an issue regarding
HTB in 2.6.x. Please send me net/sched/sch_htb.o,
net/sched/sch_htb.c (just to be sure) and be sure that you
build the kernel with debugging symbols (see debugging section
of menuconfig/xconfig).

thanks,
-------------------------------
    Martin Devera aka devik
Linux kernel QoS/HTB maintainer
  http://luxik.cdi.cz/~devik/

On Thu, 30 Oct 2003, Daniel Blueman wrote:
quoted hunk
With the LARTC 'wondershaper' HTB script [1] for good latency over ADSL, I
get an oops [3] when sending traffic via ppp0 (and when bringing the interface
down).

Kernel is 2.6.0-test9 and 'debug 3333333' appended to the 'tc' command, to
show HTB debug information [2] (not shown in [1]).

Please CC me when replying, and I can provide further details, debugging,
testing etc...this problem has been around for a while it seems.
--- [1] (relevant lines from http://lartc.org/wondershaper/)
tc qdisc add dev ppp0 root handle 1: htb default 20
tc class add dev ppp0 parent 1: classid 1:1 htb rate 210kbit burst 6k
tc class add dev ppp0 parent 1:1 classid 1:10 htb rate 210kbit burst 6k prio
1
tc class add dev ppp0 parent 1:1 classid 1:20 htb rate 189kbit burst 6k prio
2
tc class add dev ppp0 parent 1:1 classid 1:30 htb rate 168kbit burst 6k prio
2
--- [2] (HTB debug messages)
HTB init, kernel part version 3.13
htb_init sch=dc28e7f8 handle=10000 r2q=10
htb_dump sch=dc28e7f8, handle=10000
htb*g j=179519 lj=0
htb*r7 m=0
htb*r6 m=0
htb*r5 m=0
htb*r4 m=0
htb*r3 m=0
htb*r2 m=0
htb*r1 m=0
htb*r0 m=0
htb_get clid=10010 q=dc28e86c cl=00000000 ref=0
htb_get clid=10010 q=dc28e86c cl=00000000 ref=0
htb_get clid=10020 q=dc28e86c cl=00000000 ref=0
htb_get clid=10020 q=dc28e86c cl=00000000 ref=0
htb_get clid=10030 q=dc28e86c cl=00000000 ref=0
htb_get clid=10030 q=dc28e86c cl=00000000 ref=0
htb_tcf q=dc28e86c clid=0 fref=0 fl=00000000
htb_bind q=dc28e86c clid=10010 cl=00000000 fref=0
htb_tcf q=dc28e86c clid=0 fref=1 fl=df4ecd7c
htb_bind q=dc28e86c clid=10010 cl=00000000 fref=1
htb_tcf q=dc28e86c clid=0 fref=2 fl=df4ecd7c
htb_bind q=dc28e86c clid=10010 cl=00000000 fref=2
htb_tcf q=dc28e86c clid=0 fref=3 fl=df4ecd7c
htb_bind q=dc28e86c clid=10020 cl=00000000 fref=3
htb_reset sch=dc28e7f8, handle=10000
--- [3] (ksymoops-processed oops report)
Oops: 0000 [#1]
CPU:    0
EIP:    0060:[<c02cbbad>]    Not tainted
quoted
quoted
EIP; c02cbbad <htb_enqueue+ab/126>   <=====
quoted
quoted
ebx; ffffffff <__kernel_rt_sigreturn+1bbf/????>
ecx; dc28e86c <_end+1be20f60/3fb906f4>
esi; d9cf3f50 <_end+19886644/3fb906f4>
edi; dc28e7f8 <_end+1be20eec/3fb906f4>
ebp; dbef5ab8 <_end+1ba881ac/3fb906f4>
esp; dbef5a9c <_end+1ba88190/3fb906f4>
Trace; c02b3570 <dev_queue_xmit+593/73a>
Trace; c02e9c98 <ip_finish_output2+0/1c4>
Trace; c02e9d97 <ip_finish_output2+ff/1c4>
Trace; c02c0259 <nf_hook_slow+ef/13a>
Trace; c02e9c98 <ip_finish_output2+0/1c4>
Trace; c02e76d7 <ip_finish_output+21d/222>
Trace; c02e9c98 <ip_finish_output2+0/1c4>
Trace; c02bff12 <nf_iterate+6b/9f>
Trace; c02e9c80 <dst_output+15/2d>
Trace; c02c0259 <nf_hook_slow+ef/13a>
Trace; c02e9c6b <dst_output+0/2d>
Trace; c02e9620 <ip_push_pending_frames+3ac/408>
Trace; c02e9c6b <dst_output+0/2d>
Trace; c030da69 <udp_push_pending_frames+134/23c>
Trace; c030e1b4 <udp_sendmsg+60d/f3d>
Trace; c031900a <inet_sendmsg+4b/56>
Trace; c02aa4cc <sock_sendmsg+92/af>
Trace; c02aa582 <sock_recvmsg+99/b4>
Trace; c02a9f68 <move_addr_to_kernel+7e/a7>
Trace; c02b01a0 <verify_iovec+80/fa>
Trace; c02ac13d <sys_sendmsg+256/2f4>
Trace; c0141e28 <filemap_nopage+21c/2f9>
Trace; c0155610 <do_no_page+29b/631>
Trace; c0155c54 <handle_mm_fault+132/30c>
Trace; c018becd <inode_update_time+8e/b9>
Trace; c0119419 <do_page_fault+34b/56a>
Trace; c02ac67e <sys_socketcall+261/29a>
Trace; c0164327 <sys_write+5b/5d>
Trace; c010a3eb <syscall_call+7/b>

Code;  c02cbbad <htb_enqueue+ab/126>
00000000 <_EIP>:
Code;  c02cbbad <htb_enqueue+ab/126>   <=====
   0:   8b 43 04                  mov    0x4(%ebx),%eax   <=====
Code;  c02cbbb0 <htb_enqueue+ae/126>
   3:   89 44 24 04               mov    %eax,0x4(%esp,1)
Code;  c02cbbb4 <htb_enqueue+b2/126>
   7:   c7 04 24 2b 30 38 c0      movl   $0xc038302b,(%esp,1)
Code;  c02cbbbb <htb_enqueue+b9/126>
   e:   e8 e3 6e e5 ff            call   ffe56ef6 <_EIP+0xffe56ef6>
Code;  c02cbbc0 <htb_enqueue+be/126>
  13:   31 00                     xor    %eax,(%eax)

 <0>Kernel panic: Fatal exception in interrupt

--
Daniel J Blueman

NEU FÜR ALLE - GMX MediaCenter - für Fotos, Musik, Dateien...
Fotoalbum, File Sharing, MMS, Multimedia-Gruß, GMX FotoService

Jetzt kostenlos anmelden unter http://www.gmx.net
+++ GMX - die erste Adresse für Mail, Message, More! +++

Re: [2.6.0-test9] QoS HTB crash...

From: David S. Miller <hidden>
Date: 2003-10-30 21:16:15

On Thu, 30 Oct 2003 20:50:16 +0100 (CET)
devik [off-list ref] wrote:
thanks for the report. I know that there is an issue regarding
HTB in 2.6.x. Please send me net/sched/sch_htb.o,
net/sched/sch_htb.c (just to be sure) and be sure that you
build the kernel with debugging symbols (see debugging section
of menuconfig/xconfig).
I think the problem is the changes that were made
in 2.5.x to htb_next_rb_node().  It used to be:

static void htb_next_rb_node(rb_node_t **n)
{
        rb_node_t *p;
        if ((*n)->rb_right) {
                /* child at right. use it or its leftmost ancestor */
                *n = (*n)->rb_right;
                while ((*n)->rb_left)
                        *n = (*n)->rb_left;
                return;
        }
        while ((p = (*n)->rb_parent) != NULL) {
                /* if we've arrived from left child then we have next node */
                if (p->rb_left == *n) break;
                *n = p;
        }
        *n = p;
}

But it was changed into:

static void htb_next_rb_node(struct rb_node **n)
{
        *n = rb_next(*n);
}

This is wrong, the new code has much different side effects
than the original code.

This looks like the problem, devik what do you think?

Re: [2.6.0-test9] QoS HTB crash...

From: devik <hidden>
Date: 2003-10-31 07:46:40

Hmm - I have to look at 2.6's definition of rb_next. It
might be the case ! I'll check it.

Thanks, devik

On Thu, 30 Oct 2003, David S. Miller wrote:
On Thu, 30 Oct 2003 20:50:16 +0100 (CET)
devik [off-list ref] wrote:
quoted
thanks for the report. I know that there is an issue regarding
HTB in 2.6.x. Please send me net/sched/sch_htb.o,
net/sched/sch_htb.c (just to be sure) and be sure that you
build the kernel with debugging symbols (see debugging section
of menuconfig/xconfig).
I think the problem is the changes that were made
in 2.5.x to htb_next_rb_node().  It used to be:

static void htb_next_rb_node(rb_node_t **n)
{
        rb_node_t *p;
        if ((*n)->rb_right) {
                /* child at right. use it or its leftmost ancestor */
                *n = (*n)->rb_right;
                while ((*n)->rb_left)
                        *n = (*n)->rb_left;
                return;
        }
        while ((p = (*n)->rb_parent) != NULL) {
                /* if we've arrived from left child then we have next node */
                if (p->rb_left == *n) break;
                *n = p;
        }
        *n = p;
}

But it was changed into:

static void htb_next_rb_node(struct rb_node **n)
{
        *n = rb_next(*n);
}

This is wrong, the new code has much different side effects
than the original code.

This looks like the problem, devik what do you think?

Re: [2.6.0-test9] QoS HTB crash...

From: Daniel Blueman <hidden>
Date: 2003-11-01 18:13:35

Having applied this patch, I still get this issue when I kill pppd:

Oops: 0002 [#1]
CPU:    0
quoted
EIP; c02ced99 <htb_unbind_filter+49/6b>   <=====
quoted
ebx; d9c0586c <_end+19797f60/3fb906f4>
ecx; dc95adf8 <_end+1c4ed4ec/3fb906f4>
edx; d9c057f8 <_end+19797eec/3fb906f4>
esi; dc95adf8 <_end+1c4ed4ec/3fb906f4>
edi; df4eceac <_end+1f07f5a0/3fb906f4>
ebp; d9c4bdfc <_end+197de4f0/3fb906f4>
esp; d9c4bde4 <_end+197de4d8/3fb906f4>
Trace; c011a647 <kernel_map_pages+28/5d>
Trace; c02d3876 <u32_destroy_key+6a/6c>
Trace; c02d3b1f <u32_clear_hnode+2e/48>
Trace; c02d3b5e <u32_destroy_hnode+25/a9>
Trace; c011a647 <kernel_map_pages+28/5d>
Trace; c02d3cc4 <u32_destroy+e2/110>
Trace; c02ce073 <htb_destroy_class+18b/1c3>
Trace; c02cdedc <htb_destroy_filters+1d/29>
Trace; c02ce112 <htb_destroy+67/d2>
Trace; c02c1ea9 <qdisc_destroy+81/92>
Trace; c02c260a <dev_shutdown+aa/268>
Trace; c01398e8 <wakeme_after_rcu+0/c>
Trace; c02b59e2 <unregister_netdevice+db/352>
Trace; c02bd6ce <rtnl_lock+22/37>
Trace; c0235f8c <unregister_netdev+19/25>
Trace; c023c1ce <ppp_shutdown_interface+21e/3ba>
Trace; c0183836 <dput+23/723>
Trace; c016527c <__fput+7c/cd>
Trace; c02363a5 <ppp_release+5f/61>
Trace; c01652bb <__fput+bb/cd>
Trace; c0163353 <filp_close+57/81>
Trace; c0163481 <sys_close+104/228>
Trace; c010a3eb <syscall_call+7/b>

Code;  c02ced99 <htb_unbind_filter+49/6b>
00000000 <_EIP>:
Code;  c02ced99 <htb_unbind_filter+49/6b>   <=====
   0:   83 ae 58 01 00 00 01      subl   $0x1,0x158(%esi)   <=====
Code;  c02ceda0 <htb_unbind_filter+50/6b>
   7:   8b 5d f8                  mov    0xfffffff8(%ebp),%ebx
Code;  c02ceda3 <htb_unbind_filter+53/6b>
   a:   8b 75 fc                  mov    0xfffffffc(%ebp),%esi
Code;  c02ceda6 <htb_unbind_filter+56/6b>
   d:   89 ec                     mov    %ebp,%esp
Code;  c02ceda8 <htb_unbind_filter+58/6b>
   f:   5d                        pop    %ebp
Code;  c02ceda9 <htb_unbind_filter+59/6b>
  10:   c3                        ret
Code;  c02cedaa <htb_unbind_filter+5a/6b>
  11:   83 ab 3c 00 00 00 00      subl   $0x0,0x3c(%ebx)

 <0>Kernel panic: Fatal exception in interrupt

---
On Thu, 30 Oct 2003 20:50:16 +0100 (CET)
devik [off-list ref] wrote:
quoted
thanks for the report. I know that there is an issue regarding
HTB in 2.6.x. Please send me net/sched/sch_htb.o,
net/sched/sch_htb.c (just to be sure) and be sure that you
build the kernel with debugging symbols (see debugging section
of menuconfig/xconfig).
I think the problem is the changes that were made
in 2.5.x to htb_next_rb_node().  It used to be:

static void htb_next_rb_node(rb_node_t **n)
{
        rb_node_t *p;
        if ((*n)->rb_right) {
                /* child at right. use it or its leftmost ancestor */
                *n = (*n)->rb_right;
                while ((*n)->rb_left)
                        *n = (*n)->rb_left;
                return;
        }
        while ((p = (*n)->rb_parent) != NULL) {
                /* if we've arrived from left child then we have next node
*/
                if (p->rb_left == *n) break;
                *n = p;
        }
        *n = p;
}

But it was changed into:

static void htb_next_rb_node(struct rb_node **n)
{
        *n = rb_next(*n);
}

This is wrong, the new code has much different side effects
than the original code.

This looks like the problem, devik what do you think?
-- 
Daniel J Blueman

NEU FÜR ALLE - GMX MediaCenter - für Fotos, Musik, Dateien...
Fotoalbum, File Sharing, MMS, Multimedia-Gruß, GMX FotoService

Jetzt kostenlos anmelden unter http://www.gmx.net
+++ GMX - die erste Adresse für Mail, Message, More! +++

HTB 2.6.0-test10 oops related patch

From: devik <hidden>
Date: 2003-12-06 14:18:37

Hello Dave,

I discovered reason of the first Daniel's OOPS. It is change
which was not forward-ported to 2.6.
It is attached. However other Daniel's oops (pppd detach
related) is of unknown reason yet. I tried to duplicate all
conditions but pppoe (have no pppoe device here) and I was able
to kill pppd with htb under full load with no problem.
I'll look into it again today.

-------------------------------
    Martin Devera aka devik
Linux kernel QoS/HTB maintainer
  http://luxik.cdi.cz/~devik/

Re: HTB 2.6.0-test10 oops related patch

From: David S. Miller <hidden>
Date: 2003-12-07 07:10:47

On Sat, 6 Dec 2003 15:18:37 +0100 (CET)
devik [off-list ref] wrote:
I discovered reason of the first Daniel's OOPS. It is change
which was not forward-ported to 2.6.
It is attached.
Thanks a lot for tracking this problem down Devik.

Patch applied, thanks again.

Re: HTB 2.6.0-test10 oops related patch

From: devik <hidden>
Date: 2003-12-07 11:30:20

Hello Dave,

I just spotted the second bug too. (For those interested it is described
in the patch).
Please revert the small patch I sent you yesterday before applying the
new one. It contains yesterday's one too (it is because it is for 2.4 too).
Please apply it against both 2.4.23 and 2.6.0-test10. It should apply
cleanly in 2.4.23 case and with some offsets to 2.6.0-test10.

Patch comments:
- Fixed oops in debugging of enqueue/requeue.
- Fixed oops in htb_destroy.

Note that I tested 2.6 only because I'm still downloading 2.4.22 patch
on my slow line. I just wanted to sent the patch ASAP. I'll inform you
about result later but I expect no problems.

-------------------------------
    Martin Devera aka devik
Linux kernel QoS/HTB maintainer
  http://luxik.cdi.cz/~devik/

On Sat, 6 Dec 2003, David S. Miller wrote:
On Sat, 6 Dec 2003 15:18:37 +0100 (CET)
devik [off-list ref] wrote:
quoted
I discovered reason of the first Daniel's OOPS. It is change
which was not forward-ported to 2.6.
It is attached.
Thanks a lot for tracking this problem down Devik.

Patch applied, thanks again.

Re: HTB 2.6.0-test10 oops related patch

From: David S. Miller <hidden>
Date: 2003-12-07 23:29:31

On Sun, 7 Dec 2003 12:30:20 +0100 (CET)
devik [off-list ref] wrote:
I just spotted the second bug too. (For those interested it is described
in the patch).
Please revert the small patch I sent you yesterday before applying the
new one. It contains yesterday's one too (it is because it is for 2.4 too).
Please apply it against both 2.4.23 and 2.6.0-test10. It should apply
cleanly in 2.4.23 case and with some offsets to 2.6.0-test10.
Great work, applied.

Thanks.
Keyboard shortcuts
hback out one level
jnext message in thread
kprevious message in thread
ldrill in
Escclose help / fold thread tree
?toggle this help