From: Andrew Morton <akpm@linux-foundation.org> Date: 2007-06-25 05:24:49
On Sun, 24 Jun 2007 21:57:19 -0700 (PDT) bugme-daemon@bugzilla.kernel.org wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=8668
Summary: HTB Deadlock
Product: Networking
Version: 2.5
KernelVersion: 2.6.19.7
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: normal
Priority: P1
Component: Other
AssignedTo: acme@ghostprotocols.net
ReportedBy: ranko@spidernet.net
Most recent kernel where this bug did not occur:
Distribution:
Hardware Environment:
Software Environment:
Problem Description:
Greetings,
I've been experiencing problems with HTB where the whole machine locks
up. This usually happens when the whole qdisc is being removed and
occasionally when a leaf is being removed.
Common is that it always happens when some sort of removal is in
progress.
Console output I have captured is at the end of this message. The same
behavior exists from vanilla 2.6.19.7 and above. It is possible that the
problem also exist in the earlier versions however I did not go further
back.
I also believe I have found where the actual problem is:
qdisc_destroy() function is always called with dev->queue_lock locked.
htb_destroy() function up the stack is using del_timer_sync() call to
deactivate HTB qdisc timers.
yep, I would agree with that analysis. del_timer_sync() under a lock is
quite dangerous in this regard.
If the (misspelled) comment over htb_destroy() is true, current mainline
appears still to have this bug.
quoted
From the comments in the source where del_timer_sync() is defined:
---copy/paste---
/**
* del_timer_sync - deactivate a timer and wait for the handler to finish.
* @timer: the timer to be deactivated
*
* This function only differs from del_timer() on SMP: besides deactivating
* the timer it also makes sure the handler has finished executing on other
* CPUs.
*
* Synchronization rules: Callers must prevent restarting of the timer,
* otherwise this function is meaningless. It must not be called from
* interrupt contexts. The caller must not hold locks which would prevent
* completion of the timer's handler. The timer's handler must not call
* add_timer_on(). Upon exit the timer is not queued and the handler is
* not running on any CPU.
*
* The function returns whether it has deactivated a pending timer or not.
*/
---copy/paste---
Now, htb_rate_timer() does exactly what appears to be the source of the
problem - it tries obtain dev->queue_lock - and given the right moment
(timer fired handler while qdisc_destroy was holding the lock) - system
locks up - del_timer_sync is waiting for handler to finish while the
handler is waiting for the dev->queue_lock.
Of course I could also be completely wrong here and missing something
not so obvious.
I could also attempt to fix this but I haven't dealt with this code in
the past so I was hoping someone with better insight might just have an
elegant solution up his sleeve.
Best regards,
Ranko
PS: If this is not the right place for this report - please let me
know.
-----------CONSOLE (2.6.19.7)-----------
BUG: soft lockup detected on CPU#3!
[<c013c890>] softlockup_tick+0x93/0xc2
[<c0127585>] update_process_times+0x26/0x5c
[<c0111cd5>] smp_apic_timer_interrupt+0x97/0xb2
[<c0104373>] apic_timer_interrupt+0x1f/0x24
[<c02e007b>] klist_next+0x4/0x8a
[<c02e2570>] _spin_unlock_irqrestore+0xa/0xc
[<c012729b>] try_to_del_timer_sync+0x47/0x4f
[<c01272b1>] del_timer_sync+0xe/0x14
[<f8b8a85b>] htb_destroy+0x20/0x7b [sch_htb]
[<c028f196>] qdisc_destroy+0x44/0x8d
[<f8b89645>] htb_destroy_class+0xd0/0x12d [sch_htb]
[<f8b895c7>] htb_destroy_class+0x52/0x12d [sch_htb]
[<f8b8a87a>] htb_destroy+0x3f/0x7b [sch_htb]
[<c028f196>] qdisc_destroy+0x44/0x8d
[<f8b89645>] htb_destroy_class+0xd0/0x12d [sch_htb]
[<f8b895c7>] htb_destroy_class+0x52/0x12d [sch_htb]
[<f8b8a87a>] htb_destroy+0x3f/0x7b [sch_htb]
[<c028f196>] qdisc_destroy+0x44/0x8d
[<c0290ba9>] tc_get_qdisc+0x1a3/0x1ef
[<c0290a06>] tc_get_qdisc+0x0/0x1ef
[<c028a366>] rtnetlink_rcv_msg+0x158/0x215
[<c028a20e>] rtnetlink_rcv_msg+0x0/0x215
[<c0294598>] netlink_run_queue+0x88/0x11d
[<c028a1c0>] rtnetlink_rcv+0x26/0x42
[<c0294b0c>] netlink_data_ready+0x12/0x54
[<c0293843>] netlink_sendskb+0x1c/0x33
[<c0294a11>] netlink_sendmsg+0x1ee/0x2d7
[<c0278ff7>] sock_sendmsg+0xe5/0x100
[<c01306b9>] autoremove_wake_function+0x0/0x37
[<c01306b9>] autoremove_wake_function+0x0/0x37
[<c0278ff7>] sock_sendmsg+0xe5/0x100
[<c01cd8be>] copy_from_user+0x33/0x69
[<c027913f>] sys_sendmsg+0x12d/0x243
[<c02e2564>] _read_unlock_irq+0x5/0x7
[<c013fb2b>] find_get_page+0x37/0x42
[<c01423dd>] filemap_nopage+0x30c/0x3a3
[<c014bb99>] __handle_mm_fault+0x21c/0x943
[<c02e24c5>] _spin_unlock_bh+0x5/0xd
[<c027b475>] sock_setsockopt+0x63/0x59d
[<c0151801>] anon_vma_prepare+0x1b/0xcb
[<c027a2ea>] sys_socketcall+0x24f/0x271
[<c02e3ad0>] do_page_fault+0x0/0x600
[<c01038f1>] sysenter_past_esp+0x56/0x79
=======================
BUG: soft lockup detected on CPU#1!
[<c013c890>] softlockup_tick+0x93/0xc2
[<c0127585>] update_process_times+0x26/0x5c
[<c0111cd5>] smp_apic_timer_interrupt+0x97/0xb2
[<c0104373>] apic_timer_interrupt+0x1f/0x24
[<c01c007b>] blk_do_ordered+0x70/0x27e
[<c01ce788>] _raw_spin_lock+0xaa/0x13e
[<f8b8b422>] htb_rate_timer+0x18/0xc4 [sch_htb]
[<c0127539>] run_timer_softirq+0x163/0x189
[<f8b8b40a>] htb_rate_timer+0x0/0xc4 [sch_htb]
[<c0123315>] __do_softirq+0x70/0xdb
[<c01233bb>] do_softirq+0x3b/0x42
[<c0111cda>] smp_apic_timer_interrupt+0x9c/0xb2
[<c0104373>] apic_timer_interrupt+0x1f/0x24
[<c0101cc3>] mwait_idle_with_hints+0x3b/0x3f
[<c0101cd3>] mwait_idle+0xc/0x1b
[<c010271c>] cpu_idle+0x63/0x79
=======================
BUG: soft lockup detected on CPU#2!
[<c013c890>] softlockup_tick+0x93/0xc2
[<c0127585>] update_process_times+0x26/0x5c
[<c0111cd5>] smp_apic_timer_interrupt+0x97/0xb2
[<c0104373>] apic_timer_interrupt+0x1f/0x24
[<c01c007b>] blk_do_ordered+0x70/0x27e
[<c01ce788>] _raw_spin_lock+0xaa/0x13e
[<c02846df>] dev_queue_xmit+0x53/0x2e4
[<c0286e20>] neigh_connected_output+0x80/0xa0
[<c02a213a>] ip_output+0x1b5/0x24b
[<c02a0b56>] ip_finish_output+0x0/0x192
[<c029dfef>] ip_forward+0x1c8/0x2b9
[<c029ddf0>] ip_forward_finish+0x0/0x37
[<c029c962>] ip_rcv+0x2a5/0x538
[<c029c100>] ip_rcv_finish+0x0/0x2aa
[<c027f3bc>] __netdev_alloc_skb+0x12/0x2a
[<c029c6bd>] ip_rcv+0x0/0x538
[<c0282a1e>] netif_receive_skb+0x218/0x318
[<c0270008>] bitmap_get_counter+0x41/0x1e6
[<f8a6146d>] e1000_clean_rx_irq+0x12c/0x4ef [e1000]
[<f8a61341>] e1000_clean_rx_irq+0x0/0x4ef [e1000]
[<f8a60612>] e1000_clean+0xe5/0x130 [e1000]
[<c0284573>] net_rx_action+0xbc/0x1d5
[<c0123315>] __do_softirq+0x70/0xdb
[<c01233bb>] do_softirq+0x3b/0x42
[<c01058c2>] do_IRQ+0x6c/0xda
[<c01042e2>] common_interrupt+0x1a/0x20
[<c0101cc3>] mwait_idle_with_hints+0x3b/0x3f
[<c0101cd3>] mwait_idle+0xc/0x1b
[<c010271c>] cpu_idle+0x63/0x79
=======================
BUG: soft lockup detected on CPU#0!
[<c013c890>] softlockup_tick+0x93/0xc2
[<c0127585>] update_process_times+0x26/0x5c
[<c0111cd5>] smp_apic_timer_interrupt+0x97/0xb2
[<c0104373>] apic_timer_interrupt+0x1f/0x24
[<c01cd2eb>] delay_tsc+0x7/0x13
[<c01cd323>] __delay+0x6/0x7
[<c01ce796>] _raw_spin_lock+0xb8/0x13e
[<c02846df>] dev_queue_xmit+0x53/0x2e4
[<c0286e20>] neigh_connected_output+0x80/0xa0
[<c02a213a>] ip_output+0x1b5/0x24b
[<c02a0b56>] ip_finish_output+0x0/0x192
[<c029dfef>] ip_forward+0x1c8/0x2b9
[<c029ddf0>] ip_forward_finish+0x0/0x37
[<c029c962>] ip_rcv+0x2a5/0x538
[<c029c100>] ip_rcv_finish+0x0/0x2aa
[<c027e774>] __alloc_skb+0x47/0xf3
[<c029c6bd>] ip_rcv+0x0/0x538
[<c0282a1e>] netif_receive_skb+0x218/0x318
[<c0270008>] bitmap_get_counter+0x41/0x1e6
[<f88fac1d>] tg3_poll+0x6d3/0x906 [tg3]
[<c0284573>] net_rx_action+0xbc/0x1d5
[<c0123315>] __do_softirq+0x70/0xdb
[<c01233bb>] do_softirq+0x3b/0x42
[<c01058c2>] do_IRQ+0x6c/0xda
[<c01042e2>] common_interrupt+0x1a/0x20
[<c0101cc3>] mwait_idle_with_hints+0x3b/0x3f
[<c0101cd3>] mwait_idle+0xc/0x1b
[<c010271c>] cpu_idle+0x63/0x79
[<c03a9780>] start_kernel+0x353/0x423
[<c03a91cd>] unknown_bootoption+0x0/0x260
=======================
-----------CONSOLE-----------
Steps to reproduce:
--
Configure bugmail: http://bugzilla.kernel.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.
From: Patrick McHardy <hidden> Date: 2007-06-25 09:29:01
Andrew Morton wrote:
On Sun, 24 Jun 2007 21:57:19 -0700 (PDT) bugme-daemon@bugzilla.kernel.org wrote:
quoted
I've been experiencing problems with HTB where the whole machine locks
up. This usually happens when the whole qdisc is being removed and
occasionally when a leaf is being removed.
It shouldn't happen when leaves are removed, you might be running
into some endless dequeue loops however that got fixed in 2.6.20.
quoted
Common is that it always happens when some sort of removal is in
progress.
Console output I have captured is at the end of this message. The same
behavior exists from vanilla 2.6.19.7 and above. It is possible that the
problem also exist in the earlier versions however I did not go further
back.
I also believe I have found where the actual problem is:
qdisc_destroy() function is always called with dev->queue_lock locked.
htb_destroy() function up the stack is using del_timer_sync() call to
deactivate HTB qdisc timers.
yep, I would agree with that analysis. del_timer_sync() under a lock is
quite dangerous in this regard.
If the (misspelled) comment over htb_destroy() is true, current mainline
appears still to have this bug.
It is. This patch I had originally planned for 2.6.23 switches HTB
to the generic estimator, which shouldn't suffer from this.
Ranko, can you try if it fixes your timer problem?
From: Patrick McHardy <hidden> Date: 2007-06-25 09:30:33
Patrick McHardy wrote:
It is. This patch I had originally planned for 2.6.23 switches HTB
to the generic estimator, which shouldn't suffer from this.
Ranko, can you try if it fixes your timer problem?
On Mon, 2007-06-25 at 11:30 +0200, Patrick McHardy wrote:
Patrick McHardy wrote:
quoted
It is. This patch I had originally planned for 2.6.23 switches HTB
to the generic estimator, which shouldn't suffer from this.
Ranko, can you try if it fixes your timer problem?
It is. This patch I had originally planned for 2.6.23 switches HTB
to the generic estimator, which shouldn't suffer from this.
BTW, maybe I look at this too short, but is this del_timer()
in gen_kill_estimator() enough? I cannot see nothing against
a timer just running and doing mod_timer() again...
Regards,
Jarek P.
From: Patrick McHardy <hidden> Date: 2007-06-27 11:45:21
Jarek Poplawski wrote:
On 25-06-2007 11:28, Patrick McHardy wrote:
...
quoted
It is. This patch I had originally planned for 2.6.23 switches HTB
to the generic estimator, which shouldn't suffer from this.
BTW, maybe I look at this too short, but is this del_timer()
in gen_kill_estimator() enough? I cannot see nothing against
a timer just running and doing mod_timer() again...
Yes, but nothing bad can happen, the timer will find an empty
list and do nothing. It would make more sense to check for
an empty list before restarting the timer though.
Could you send a patch for that?
On Wed, Jun 27, 2007 at 01:44:08PM +0200, Patrick McHardy wrote:
Jarek Poplawski wrote:
quoted
On 25-06-2007 11:28, Patrick McHardy wrote:
...
quoted
It is. This patch I had originally planned for 2.6.23 switches HTB
to the generic estimator, which shouldn't suffer from this.
BTW, maybe I look at this too short, but is this del_timer()
in gen_kill_estimator() enough? I cannot see nothing against
a timer just running and doing mod_timer() again...
Yes, but nothing bad can happen, the timer will find an empty
list and do nothing. It would make more sense to check for
an empty list before restarting the timer though.
Could you send a patch for that?
Probably I could, but it's your idea!
I look at this just now, and maybe it's enough for asking,
but definitely not enough for patch. I'll try to check this
more in the evening, so I could send something tomorrow.
So if it's not only about kindness, feel free to do it
sooner and I've no doubts - better.
Thanks for so instant reply!
Jarek P.
On Wed, Jun 27, 2007 at 02:10:13PM +0200, Jarek Poplawski wrote:
...
- > So if it's not only about kindness, feel free to do it
+ > So if it's only about kindness, feel free to do it
Sorry!
Jarek P.
From: Patrick McHardy <hidden> Date: 2007-06-27 14:55:03
Jarek Poplawski wrote:
On Wed, Jun 27, 2007 at 01:44:08PM +0200, Patrick McHardy wrote:
quoted
quoted
BTW, maybe I look at this too short, but is this del_timer()
in gen_kill_estimator() enough? I cannot see nothing against
a timer just running and doing mod_timer() again...
Yes, but nothing bad can happen, the timer will find an empty
list and do nothing. It would make more sense to check for
an empty list before restarting the timer though.
Could you send a patch for that?
Probably I could, but it's your idea!
I look at this just now, and maybe it's enough for asking,
but definitely not enough for patch. I'll try to check this
more in the evening, so I could send something tomorrow.
So if it's not only about kindness, feel free to do it
sooner and I've no doubts - better.
From: Patrick McHardy <hidden> Date: 2007-06-27 15:10:52
Patrick McHardy wrote:
Jarek Poplawski wrote:
quoted
I look at this just now, and maybe it's enough for asking,
but definitely not enough for patch. I'll try to check this
more in the evening, so I could send something tomorrow.
So if it's not only about kindness, feel free to do it
sooner and I've no doubts - better.
I can take care of it, no problem.
OK, this patch should fix the Jarek noticed (and a few more).
It does not fix the original HTB problem though.
From: Patrick McHardy <hidden> Date: 2007-06-27 15:26:16
Patrick McHardy wrote:
[NET]: gen_estimator: fix locking and timer related bugs
That one still left a race, we could be reinitalizing the timer
while it is still running. This patch additionally makes sure
each timer is only initialized once.
On Wed, Jun 27, 2007 at 05:25:45PM +0200, Patrick McHardy wrote:
Patrick McHardy wrote:
quoted
[NET]: gen_estimator: fix locking and timer related bugs
That one still left a race, we could be reinitalizing the timer
while it is still running. This patch additionally makes sure
each timer is only initialized once.
[NET]: gen_estimator: fix locking and timer related bugs
As noticed by Jarek Poplawski [off-list ref], the timer removal in
gen_kill_estimator races with the timer function rearming the timer.
Additionally there are a few more related problems that seem to be
relicts from the timer when the estimator was qdisc specific and
- relicts from the timer when the estimator was qdisc specific and
+ relicts from the time when the estimator was qdisc specific and
could rely on the rtnl or dev->qdisc_lock:
I've lost some time thinking about this rtnl and checking where
these gen_ functions are used, and how much foolish could be
asking about this here, so, it seems there should be some policy
about commenting required locking in networking - I mean after
reading e.g. sch_generic.c you could wrongly think no comments
means: no locking required. (And probably it would be better/
easier for "the more experienced" to do some supplements, if you
know what I mean...)
quoted hunk
- the check whether the list is empty and a timer needs to be started
when adding a new estimator doesn't take the lock, so it races
against concurrent additions, which can result in the timer beeing
added twice or getting reinitialized after being added.
- the new estimator's next pointer is also set without holding the
lock, again racing against concurrent additions with possible
list corruption as a result.
- the timer deletion when killing an estimator is also not under
the lock and races against timer arming when adding a new estimator.
Fix by holding the lock around the entire list addition and initial
timer arming. Removal is not done explicitly anymore, instead the
timer function only rearms the timer when there are still estimators
present.
Signed-off-by: Patrick McHardy <redacted>
---
commit b6a0c468c258d96c6f132fc71ca74225235bc223
tree 6f61004cf4810a4826aa5c7477e4d455ae3a5698
parent 48d8d7ee5dd17c64833e0343ab4ae8ef01cc2648
author Patrick McHardy [off-list ref] Wed, 27 Jun 2007 17:06:02 +0200
committer Patrick McHardy [off-list ref] Wed, 27 Jun 2007 17:24:13 +0200
net/core/gen_estimator.c | 27 +++++++++++----------------
1 files changed, 11 insertions(+), 16 deletions(-)
On Wed, Jun 27, 2007 at 04:53:48PM +0200, Patrick McHardy wrote:
Jarek Poplawski wrote:
quoted
On Wed, Jun 27, 2007 at 01:44:08PM +0200, Patrick McHardy wrote:
quoted
quoted
BTW, maybe I look at this too short, but is this del_timer()
in gen_kill_estimator() enough? I cannot see nothing against
a timer just running and doing mod_timer() again...
Yes, but nothing bad can happen, the timer will find an empty
list and do nothing. It would make more sense to check for
an empty list before restarting the timer though.
Maybe this time I've looked at this too much, but actually
I think some "badness" is possible: if I'm not missing something
such timer would rearm constantly, so, there could be some
collision when gen_new_estimator() tries to add new timer.
quoted
quoted
Could you send a patch for that?
Probably I could, but it's your idea!
I look at this just now, and maybe it's enough for asking,
but definitely not enough for patch. I'll try to check this
more in the evening, so I could send something tomorrow.
So if it's not only about kindness, feel free to do it
sooner and I've no doubts - better.
I can take care of it, no problem.
Thanks. I've some technical limitations, so such small changes
take much time if there is a need to get the most up-to-date
kernel and do at least "compile testing". I'm not much about
credits, so I'd prefer to do such things only if it's not very
urgent and could save some signifying amount of your time too.
BTW #2, I hope it's about some new policy, but I cannot see
any #ifdef CONFIG_NET_ESTIMATOR in this sch_htb patch.
Jarek P.
On Wed, Jun 27, 2007 at 05:25:45PM +0200, Patrick McHardy wrote:
...
Additionally there are a few more related problems that seem to be
relicts from the timer when the estimator was qdisc specific and
could rely on the rtnl or dev->qdisc_lock:
- the check whether the list is empty and a timer needs to be started
when adding a new estimator doesn't take the lock, so it races
against concurrent additions, which can result in the timer beeing
added twice or getting reinitialized after being added.
- the new estimator's next pointer is also set without holding the
lock, again racing against concurrent additions with possible
list corruption as a result.
- the timer deletion when killing an estimator is also not under
the lock and races against timer arming when adding a new estimator.
Fix by holding the lock around the entire list addition and initial
timer arming. Removal is not done explicitly anymore, instead the
timer function only rearms the timer when there are still estimators
present.
...
quoted hunk
@@ -202,7 +201,6 @@ void gen_kill_estimator(struct gnet_stats_basic *bstats, struct gen_estimator *est, **pest; for (idx=0; idx <= EST_MAX_INTERVAL; idx++) {- int killed = 0; pest = &elist[idx].list; while ((est=*pest) != NULL) {
So, maybe this list walking here needs some locking too?
Jarek P.
From: Patrick McHardy <hidden> Date: 2007-06-28 12:25:06
Jarek Poplawski wrote:
quoted
@@ -202,7 +201,6 @@ void gen_kill_estimator(struct gnet_stats_basic *bstats, struct gen_estimator *est, **pest; for (idx=0; idx <= EST_MAX_INTERVAL; idx++) {- int killed = 0; pest = &elist[idx].list; while ((est=*pest) != NULL) {
So, maybe this list walking here needs some locking too?
It depends on whether estimators should be able to rely on
the rtnl in the future or be completely responsible for their
own locking. My patch yesterday was made under the assumption
that they shouldn't rely on external locking, which seemed to
be the right thing for a "generic" implementation. OTOH its
still specific to networking, so relying on the rtnl doesn't
sound too unreasonable too. I'm beginning to thing I made
the wrong choice with my patch.
I'm busy right now, would you mind looking into a patch that
only deals with the timer races, but still relies on the
rtnl?
*bstats,
struct gen_estimator *est, **pest;
for (idx=0; idx <= EST_MAX_INTERVAL; idx++) {
- int killed = 0;
pest = &elist[idx].list;
while ((est=*pest) != NULL) {
So, maybe this list walking here needs some locking too?
It depends on whether estimators should be able to rely on
the rtnl in the future or be completely responsible for their
own locking. My patch yesterday was made under the assumption
that they shouldn't rely on external locking, which seemed to
be the right thing for a "generic" implementation. OTOH its
still specific to networking, so relying on the rtnl doesn't
sound too unreasonable too. I'm beginning to thing I made
the wrong choice with my patch.
I'm busy right now, would you mind looking into a patch that
only deals with the timer races, but still relies on the
rtnl?
In that case this patch looks OK & enough.
My earlier proposals are only of cosmetical value.
Jarek P.
*bstats,
struct gen_estimator *est, **pest;
for (idx=0; idx <= EST_MAX_INTERVAL; idx++) {
- int killed = 0;
pest = &elist[idx].list;
while ((est=*pest) != NULL) {
So, maybe this list walking here needs some locking too?
It depends on whether estimators should be able to rely on
the rtnl in the future or be completely responsible for their
own locking. My patch yesterday was made under the assumption
that they shouldn't rely on external locking, which seemed to
be the right thing for a "generic" implementation. OTOH its
still specific to networking, so relying on the rtnl doesn't
sound too unreasonable too. I'm beginning to thing I made
the wrong choice with my patch.
I'm busy right now, would you mind looking into a patch that
only deals with the timer races, but still relies on the
rtnl?
In that case this patch looks OK & enough.
Its overkill in that case. The concurrent additions and removals
can't happen.
*bstats,
struct gen_estimator *est, **pest;
for (idx=0; idx <= EST_MAX_INTERVAL; idx++) {
- int killed = 0;
pest = &elist[idx].list;
while ((est=*pest) != NULL) {
...
Its overkill in that case. The concurrent additions and removals
can't happen.
BTW, if we talk about overkills: is there any reason to do these
for & while until the end? I can't see why anybody should add the
same *bstats & *rate_est more than once (or max twice if we let
to add, change & remove them independently). With a large number
of classes this could matter.
Regards,
Jarek P.
On Fri, Jun 29, 2007 at 09:02:41AM +0200, Jarek Poplawski wrote:
...
same *bstats & *rate_est more than once (or max twice if we let
to add, change & remove them independently).
...but this doesn't look sensible at all!
So, maybe, if we would need something counted with two intervals...
But, nobody seems to use such possibility, anyway.
Jarek P.