This seems to be more complicated that it appears. It looks as though
Linux has re-used the LSB of the old TOS field for some "link local"
flag which is used by routing.
It's not immediately obvious whether pfifo_fast is using this new
interpretation though. If it isn't, the fix should be to remove the
RTO_ONLINK bit from the mask it's using on the tos field. The other
half of the mask correctly excludes the ECN bits from the field.
CC netdev, where linux network dev can take a look.
I would say that this is a wrong analysis :
1) ECN uses two low order bits of TOS byte
2) pfifo_fast uses skb->priority
skb->priority = rt_tos2priority(iph->tos);
#define IPTOS_TOS_MASK 0x1E
#define IPTOS_TOS(tos) ((tos)&IPTOS_TOS_MASK)
static inline char rt_tos2priority(u8 tos)
{
return ip_tos2prio[IPTOS_TOS(tos)>>1];
}
No interference between two mechanisms, unless sysadmin messed up things
(skb_edit)
This seems to be more complicated that it appears. It looks as though
Linux has re-used the LSB of the old TOS field for some "link local"
flag which is used by routing.
It's not immediately obvious whether pfifo_fast is using this new
interpretation though. If it isn't, the fix should be to remove the
RTO_ONLINK bit from the mask it's using on the tos field. The other
half of the mask correctly excludes the ECN bits from the field.
CC netdev, where linux network dev can take a look.
I would say that this is a wrong analysis :
1) ECN uses two low order bits of TOS byte
2) pfifo_fast uses skb->priority
skb->priority = rt_tos2priority(iph->tos);
#define IPTOS_TOS_MASK 0x1E
#define IPTOS_TOS(tos) ((tos)&IPTOS_TOS_MASK)
static inline char rt_tos2priority(u8 tos)
{
return ip_tos2prio[IPTOS_TOS(tos)>>1];
}
No interference between two mechanisms, unless sysadmin messed up things
(skb_edit)
David, it seems ip_tos2prio is wrong on its 2nd entry :
#define TC_PRIO_BESTEFFORT 0
#define TC_PRIO_FILLER 1
#define TC_PRIO_BULK 2
#define TC_PRIO_INTERACTIVE_BULK 4
#define TC_PRIO_INTERACTIVE 6
#define TC_PRIO_CONTROL 7
#define TC_PRIO_MAX 15
net/ipv4/route.c:170:#define ECN_OR_COST(class) TC_PRIO_##class
const __u8 ip_tos2prio[16] = {
TC_PRIO_BESTEFFORT, /* 0 : for flow without ECN */
ECN_OR_COST(FILLER), /* 1 : flow with ECN */
...
};
This means ECN enabled flows got TC_PRIO_FILLER (what the hell is
that ?)
pfifo_fast has :
static const u8 prio2band[TC_PRIO_MAX+1] =
{ 1, 2, 2, 2, 1, 2, 0, 0 , 1, 1, 1, 1, 1, 1, 1, 1 };
So a non ECN enabled flow goes to band 1, while an ECN enabled one is in
band 2 (!). Thus, ECN enabled flows have a chance being droped more
often than non ECN flows. Thats not fair...
What do you think ?
Thanks
CC netdev, where linux network dev can take a look.
I would say that this is a wrong analysis :
1) ECN uses two low order bits of TOS byte
2) pfifo_fast uses skb->priority
skb->priority = rt_tos2priority(iph->tos);
#define IPTOS_TOS_MASK 0x1E
#define IPTOS_TOS(tos) ((tos)&IPTOS_TOS_MASK)
static inline char rt_tos2priority(u8 tos)
{
return ip_tos2prio[IPTOS_TOS(tos)>>1];
}
No interference between two mechanisms, unless sysadmin messed up things
(skb_edit)
David, it seems ip_tos2prio is wrong on its 2nd entry :
#define TC_PRIO_BESTEFFORT 0
#define TC_PRIO_FILLER 1
#define TC_PRIO_BULK 2
#define TC_PRIO_INTERACTIVE_BULK 4
#define TC_PRIO_INTERACTIVE 6
#define TC_PRIO_CONTROL 7
#define TC_PRIO_MAX 15
net/ipv4/route.c:170:#define ECN_OR_COST(class) TC_PRIO_##class
const __u8 ip_tos2prio[16] = {
TC_PRIO_BESTEFFORT, /* 0 : for flow without ECN */
ECN_OR_COST(FILLER), /* 1 : flow with ECN */
...
};
This means ECN enabled flows got TC_PRIO_FILLER (what the hell is
that ?)
pfifo_fast has :
static const u8 prio2band[TC_PRIO_MAX+1] =
{ 1, 2, 2, 2, 1, 2, 0, 0 , 1, 1, 1, 1, 1, 1, 1, 1 };
So a non ECN enabled flow goes to band 1, while an ECN enabled one is in
band 2 (!). Thus, ECN enabled flows have a chance being droped more
often than non ECN flows. Thats not fair...
What do you think ?
Well, that makes 3 of us that think it's wrong. Can we get more?
(I'll run through the math again in the morning)
It's most often not actually "enablement" but "assertion", when for
example an ECN bit is put on an ACK packet (by an application, or qdisc)
, it drops that ACK packet into the 2 queue - leaving all the other
non-ECN asserted packets in that flow to flow out ahead of it.
Or so dan siemon & I & now you, think. It's late and I really want to recheck
the math and the shifts in the morning. However, if true... this would
explain much ECN related weirdness precisely where it has been hard to
measure, on heavily loaded systems.
I think this is a good short term fix, but it will mildly upset people
that actually still use minimum cost and don't use ECN. That said,
RFC1349 has been obsolete for a decade now, and ECN enabled servers are
at 12% penetration according to MIT.
Still, long term, doing a sch_pfifo_dscp that would be fully compliant
with the relevant modern RFCs and eventually making that the standard
would be good.
--
Dave Taht
http://nex-6.taht.net
From: Eric Dumazet <hidden> Date: 2011-03-15 06:15:53
Le lundi 14 mars 2011 à 23:27 -0600, Dave Täht a écrit :
Well, that makes 3 of us that think it's wrong. Can we get more?
(I'll run through the math again in the morning)
It's most often not actually "enablement" but "assertion", when for
example an ECN bit is put on an ACK packet (by an application, or qdisc)
, it drops that ACK packet into the 2 queue - leaving all the other
non-ECN asserted packets in that flow to flow out ahead of it.
There are two ECN bits, not one.
The low order bit is not taken into account by skb->priority mapping.
The high order bit cannot be changed during flow lifetime.
(So : no OOO (Out Of Order) problems on say TCP flows)
Or so dan siemon & I & now you, think. It's late and I really want to recheck
the math and the shifts in the morning. However, if true... this would
explain much ECN related weirdness precisely where it has been hard to
measure, on heavily loaded systems.
I think this is a good short term fix, but it will mildly upset people
that actually still use minimum cost and don't use ECN. That said,
RFC1349 has been obsolete for a decade now, and ECN enabled servers are
at 12% penetration according to MIT.
If minimum cost was asked by people, their packets had chance being
dropped. Why should they be upset ?
ECN should be favored anyway in 2011, now everybody is ready.
Still, long term, doing a sch_pfifo_dscp that would be fully compliant
with the relevant modern RFCs and eventually making that the standard
would be good.
sch_pfifo_fast is not the place we perform the TOS -> priority mapping.
Its done in another layer.
That is of litle effect, given TOS values are meaningfull only inside a
domain. Nobody can force everyone to use same semantics on the Internet,
even with a standard RFC. I doubt people using linux machines at home
really need DSCP at all.
What we could do instead is to favor a bit ECN enabled connections,
using 4 bands instead of 3 for pfifo_fast (linux default qdisc, probably
the most used qdisc)
band 0 : high priority packets (like now)
band 1 : (old band 1, ECN capable flows)
band 2 : (old band 1, no ECN flows)
band 3 : low priority packets (old band 2)
Note : pfifo_fast is mostly used on end hosts, not on routers (where
admins setup non default qdiscs), and typical end hosts never experiment
packet drops on their qdiscs, because they are now plugged to Gigabit
LANS, and device queuelength is so big.
From: Jonathan Morton <hidden> Date: 2011-03-15 17:09:11
On 15 Mar, 2011, at 8:15 am, Eric Dumazet wrote:
band 0 : high priority packets (like now)
band 1 : (old band 1, ECN capable flows)
band 2 : (old band 1, no ECN flows)
band 3 : low priority packets (old band 2)
This seems good to me. It would provide a concrete (if minor) enticement to turn ECN on.
- Jonathan
From: Eric Dumazet <hidden> Date: 2011-03-15 18:28:42
Le mardi 15 mars 2011 à 19:09 +0200, Jonathan Morton a écrit :
On 15 Mar, 2011, at 8:15 am, Eric Dumazet wrote:
quoted
band 0 : high priority packets (like now)
band 1 : (old band 1, ECN capable flows)
band 2 : (old band 1, no ECN flows)
band 3 : low priority packets (old band 2)
This seems good to me. It would provide a concrete (if minor) enticement to turn ECN on.
Here is a patch to implement that, on top of net-next-2.6 git tree
qdisc pfifo_fast 0: dev eth1 root refcnt 2 bands 4 priomap 2 1 3 3 2 3 0 0 2 2 2 2 2 2 2 2
Sent 168 bytes 2 pkt (dropped 0, overlimits 0 requeues 0)
backlog 0b 0p requeues 0
@@ -412,19 +412,39 @@ static struct Qdisc noqueue_qdisc = {};-staticconstu8prio2band[TC_PRIO_MAX+1]={-1,2,2,2,1,2,0,0,1,1,1,1,1,1,1,1+/* 4-band FIFO queue: old style, but should be a bit faster than+genericprio+fifocombination.+*/++enum{+BAND_HIGH_PRIO,+BAND_MEDIUM_ECN_PRIO,+BAND_MEDIUM_PRIO,+BAND_LOW_PRIO,++PFIFO_FAST_BANDS};-/* 3-band FIFO queue: old style, but should be a bit faster than-genericprio+fifocombination.+/*+*WegivealitleincenttoECNflows*/+staticconstu8prio2band[TC_PRIO_MAX+1]={+[TC_PRIO_BESTEFFORT]=BAND_MEDIUM_PRIO,+[TC_PRIO_FILLER]=BAND_MEDIUM_ECN_PRIO,+[TC_PRIO_BULK]=BAND_LOW_PRIO,+[TC_PRIO_BULK+1]=BAND_LOW_PRIO,+[TC_PRIO_INTERACTIVE_BULK]=BAND_MEDIUM_PRIO,+[TC_PRIO_INTERACTIVE_BULK+1]=BAND_LOW_PRIO,+[TC_PRIO_INTERACTIVE]=BAND_HIGH_PRIO,+[TC_PRIO_CONTROL]=BAND_HIGH_PRIO,++[TC_PRIO_CONTROL+1...TC_PRIO_MAX]=BAND_MEDIUM_PRIO,+};-#define PFIFO_FAST_BANDS 3/**Privatedataforapfifo_fastschedulercontaining:-*-queuesforthethreeband+*-queuesforthefourbands*-bitmapindicatingwhichofthebandscontainskbs*/structpfifo_fast_priv{
From: Jonathan Morton <hidden> Date: 2011-03-15 18:37:07
On 15 Mar, 2011, at 8:28 pm, Eric Dumazet wrote:
quoted
quoted
band 0 : high priority packets (like now)
band 1 : (old band 1, ECN capable flows)
band 2 : (old band 1, no ECN flows)
band 3 : low priority packets (old band 2)
This seems good to me. It would provide a concrete (if minor) enticement to turn ECN on.
Here is a patch to implement that, on top of net-next-2.6 git tree
Does this take both ECN bits into account? The ECT(0), ECT(1) and ECE codepoints all need to be recognised equally.
- Jonathan
From: Eric Dumazet <hidden> Date: 2011-03-15 19:57:04
Le mardi 15 mars 2011 à 20:37 +0200, Jonathan Morton a écrit :
On 15 Mar, 2011, at 8:28 pm, Eric Dumazet wrote:
quoted
quoted
quoted
band 0 : high priority packets (like now)
band 1 : (old band 1, ECN capable flows)
band 2 : (old band 1, no ECN flows)
band 3 : low priority packets (old band 2)
This seems good to me. It would provide a concrete (if minor) enticement to turn ECN on.
Here is a patch to implement that, on top of net-next-2.6 git tree
Does this take both ECN bits into account? The ECT(0), ECT(1) and ECE codepoints all need to be recognised equally.
This is done in a different layer, as already explained.
Current linux code ignores low order bit when doing TOS -> skb->priority
mapping.
From: David Miller <davem@davemloft.net> Date: 2011-03-15 22:51:14
From: Eric Dumazet <redacted>
Date: Tue, 15 Mar 2011 05:42:46 +0100
Le lundi 14 mars 2011 à 21:24 +0100, Eric Dumazet a écrit :
David, it seems ip_tos2prio is wrong on its 2nd entry :
Indeed, and in context, this is simply a thinko which has survived since
ECN support first got added to the tree, here's the relevant hunk:
--------------------
From: Eric Dumazet <hidden> Date: 2011-03-15 23:57:38
From: Dan Siemon <redacted>
ECN support incorrectly maps ECN BESTEFFORT packets to TC_PRIO_FILLER
(1) instead of TC_PRIO_BESTEFFORT (0)
This means ECN enabled flows are placed in pfifo_fast/prio low priority
band, giving ECN enabled flows [ECT(0) and CE codepoints] higher drop
probabilities.
This is rather unfortunate, given we would like ECN being more widely
used.
Ref : http://www.coverfire.com/archives/2011/03/13/pfifo_fast-and-ecn/
Signed-off-by: Dan Siemon <redacted>
Signed-off-by: Eric Dumazet <redacted>
Cc: Dave Täht <redacted>
Cc: Jonathan Morton <redacted>
---
Note: I left TC_PRIO_FILLER definition, this can be removed later.
net/ipv4/route.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
From: David Miller <davem@davemloft.net> Date: 2011-03-16 01:53:46
From: Eric Dumazet <redacted>
Date: Wed, 16 Mar 2011 00:56:07 +0100
From: Dan Siemon <redacted>
ECN support incorrectly maps ECN BESTEFFORT packets to TC_PRIO_FILLER
(1) instead of TC_PRIO_BESTEFFORT (0)
This means ECN enabled flows are placed in pfifo_fast/prio low priority
band, giving ECN enabled flows [ECT(0) and CE codepoints] higher drop
probabilities.
This is rather unfortunate, given we would like ECN being more widely
used.
Ref : http://www.coverfire.com/archives/2011/03/13/pfifo_fast-and-ecn/
Signed-off-by: Dan Siemon <redacted>
Signed-off-by: Eric Dumazet <redacted>
Cc: Dave Täht <redacted>
Cc: Jonathan Morton <redacted>