From: Jamal Hadi Salim <hidden> Date: 2005-01-30 22:12:04
This is in relation to providing functionality that IMQ was intending
to using the dummy device and tc actions. Ive copied as many people as i
could dig who i know may have interest in this.
Please forward this to any other list which may have interest
in the subject. It still needs some cleaning up; however, i dont wanna
sit on it for another year - and now that mirred is out there, this is a
good time.
Advantage over current IMQ; cleaner in particular in in SMP;
with a _lot_ less code.
Old Dummy device functionality is preserved while new one only
kicks in if you use actions. Didnt have to write a new device and finaly
made a real dumb device to be a little smarter ;->
IMQ USES
--------
As far as i know the reasons listed below is why people use IMQ.
It would be nice to know of anything else that i missed because this
is the requirements list i used.
1) qdiscs/policies that are per device as opposed to system wide.
IMQ allows for sharing across multiple devices.
2) Allows for queueing incoming traffic for shaping instead of
dropping. I am not aware of any study that shows policing is
worse than shaping in achieving the end goal of rate control.
I would be interested if anyone is experimenting. Nevertheless,
this is still an alternative as opposed to making a system wide
ingress change.
3) Very interesting use: if you are serving p2p you may wanna give
preference to your own localy originated traffic (when responses come
back) vs someone using your system to do bittorent. So QoSing based on
state comes in as the solution. What people did to achive this was stick
the IMQ somewhere prelocal hook.
I think this is a pretty neat feature to have in Linux in general.
(i.e not just for IMQ).
But i wont go back to putting netfilter hooks in the device to satisfy
this. I also dont think its worth it hacking dummy some more to be
aware of say L3 info and play ip rule tricks to achieve this.
--> Instead the plan is to have a contrack related action. This action
will selectively either query/create contrack state on incoming packets.
Packets could then be redirected to dummy based on what happens -> eg
on incoming packets; if we find they are of known state we could send to
a different queue than one which didnt have existing state. This
all however is dependent on whatever rules the admin enters.
What you can do with dummy currently with actions
--------------------------------------------------
Lets say you are policing packets from alias 192.168.200.200/32
you dont want those to exceed 100kbps going out.
tc filter add dev eth0 parent 1: protocol ip prio 10 u32 \
match ip src 192.168.200.200/32 flowid 1:2 \
action police rate 100kbit burst 90k drop
If you run tcpdump on eth0 you will see all packets going out
with src 192.168.200.200/32 dropped or not
Extend the rule a little to see only the ones that made it out:
tc filter add dev eth0 parent 1: protocol ip prio 10 u32 \
match ip src 192.168.200.200/32 flowid 1:2 \
action police rate 10kbit burst 90k drop \
action mirred egress mirror dev dummy0
Now fire tcpdump on dummy0 to see only those packets ..
tcpdump -n -i dummy0 -x -e -t
Essentially a good debugging/logging interface.
If you replace mirror with redirect, those packets will be
blackholed and will never make it out. This redirect behavior
changes with new patch (but not the mirror).
What you can do with dummy and attached patch
----------------------------------------------
Essentially provide functionality that most people use IMQ;
sample below:
--------
export TC="/sbin/tc"
$TC qdisc add dev dummy0 root handle 1: prio
$TC qdisc add dev dummy0 parent 1:1 handle 10: sfq
$TC qdisc add dev dummy0 parent 1:2 handle 20: tbf rate 20kbit buffer
1600 limit 3000
$TC qdisc add dev dummy0 parent 1:3 handle 30:
sfq
$TC filter add dev dummy0 protocol ip pref 1 parent 1: handle 1 fw
classid 1:1
$TC filter add dev dummy0 protocol ip pref 2 parent 1: handle 2 fw
classid 1:2
ifconfig dummy0 up
$TC qdisc add dev eth0 ingress
# redirect all IP packets arriving in eth0 to dummy0
# use mark 1 --> puts them onto class 1:1
$TC filter add dev eth0 parent ffff: protocol ip prio 10 u32 \
match u32 0 0 flowid 1:1 \
action ipt -j MARK --set-mark 1 \
action mirred egress redirect dev dummy0
--------
Run A Little test:
from another machine ping so that you have packets going into the box:
-----
[root@jzny action-tests]# ping 10.22
PING 10.22 (10.0.0.22): 56 data bytes
64 bytes from 10.0.0.22: icmp_seq=0 ttl=64 time=2.8 ms
64 bytes from 10.0.0.22: icmp_seq=1 ttl=64 time=0.6 ms
64 bytes from 10.0.0.22: icmp_seq=2 ttl=64 time=0.6 ms
--- 10.22 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.6/1.3/2.8 ms
[root@jzny action-tests]#
-----
Now look at some stats:
---
[root@jmandrake]:~# $TC -s filter show parent ffff: dev eth0
filter protocol ip pref 10 u32
filter protocol ip pref 10 u32 fh 800: ht divisor 1
filter protocol ip pref 10 u32 fh 800::800 order 2048 key ht 800 bkt 0
flowid 1:1
match 00000000/00000000 at 0
action order 1: tablename: mangle hook: NF_IP_PRE_ROUTING
target MARK set 0x1
index 1 ref 1 bind 1 installed 4195sec used 27sec
Sent 252 bytes 3 pkts (dropped 0, overlimits 0)
action order 2: mirred (Egress Redirect to device dummy0) stolen
index 1 ref 1 bind 1 installed 165 sec used 27 sec
Sent 252 bytes 3 pkts (dropped 0, overlimits 0)
[root@jmandrake]:~# $TC -s qdisc
qdisc sfq 30: dev dummy0 limit 128p quantum 1514b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
qdisc tbf 20: dev dummy0 rate 20Kbit burst 1575b lat 2147.5s
Sent 210 bytes 3 pkts (dropped 0, overlimits 0)
qdisc sfq 10: dev dummy0 limit 128p quantum 1514b
Sent 294 bytes 3 pkts (dropped 0, overlimits 0)
qdisc prio 1: dev dummy0 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1
1
Sent 504 bytes 6 pkts (dropped 0, overlimits 0)
qdisc ingress ffff: dev eth0 ----------------
Sent 308 bytes 5 pkts (dropped 0, overlimits 0)
[root@jmandrake]:~# ifconfig dummy0
dummy0 Link encap:Ethernet HWaddr 00:00:00:00:00:00
inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
RX packets:6 errors:0 dropped:3 overruns:0 frame:0
TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:32
RX bytes:504 (504.0 b) TX bytes:252 (252.0 b)
-----
Dummy continues to behave like it always did.
You send it any packet not originating from the actions it will drop
them.
[In this case the three dropped packets were ipv6 ndisc].
My goal here is to start a discussion to see if people agree this is
a good replacement for IMQ or whether to go another path.
Clearly i would prefer to have this change in, but I am not religious
and would listen to reason about how it should be done as long as no
uneccessary clutter happens.
Patch attached.
cheers,
jamal
2) Allows for queueing incoming traffic for shaping instead of
dropping. I am not aware of any study that shows policing is
worse than shaping in achieving the end goal of rate control.
I would be interested if anyone is experimenting. Nevertheless,
this is still an alternative as opposed to making a system wide
ingress change.
Policing didn't work with IPv6 last time I checked.
--
Hasso Tepper
Elion Enterprises Ltd.
WAN administrator
2) Allows for queueing incoming traffic for shaping instead of
dropping. I am not aware of any study that shows policing is
worse than shaping in achieving the end goal of rate control.
I would be interested if anyone is experimenting. Nevertheless,
this is still an alternative as opposed to making a system wide
ingress change.
Policing didn't work with IPv6 last time I checked.
Really? I take it this is using the u32 classifier?
What filter did you use?
cheers,
jamal
This?
tc filter add dev eth1.101 parent ffff: protocol all prio 50 handle \
0x101 fw police rate 1024kbit burst 60k drop flowid :101
What are you trying to do? Are you also trying to rate limit ARPs etc
in one shot?
Does this even get hit at all? tc -s would show you stats. I suspect
for one it is not being hit.
Maybe you are trying to use iptables marks that happen
a long time after the ingress has seen the packets (which would
explain why it is not being hit)? This would be true kernels > 2.6.8
but not before ..
In other words, it may be a config issue.
If you tell me what it is you are trying to do i could try and set it
up when i come back from work today.
cheers,
jamal
This?
tc filter add dev eth1.101 parent ffff: protocol all prio 50 handle \
0x101 fw police rate 1024kbit burst 60k drop flowid :101
What are you trying to do? Are you also trying to rate limit ARPs etc
in one shot?
All traffic coming from eth1.101 interface.
Does this even get hit at all? tc -s would show you stats. I suspect
for one it is not being hit.
As far as I remember situation was exactly as I described. This worked for
IPv4 traffic, but not for IPv6 traffic.
Maybe you are trying to use iptables marks that happen
a long time after the ingress has seen the packets (which would
explain why it is not being hit)? This would be true kernels > 2.6.8
but not before ..
This test was done with 2.6.6.
In other words, it may be a config issue.
Would be nice ;).
If you tell me what it is you are trying to do i could try and set it
up when i come back from work today.
I'd like to limit _all_ traffic coming in from one particular interface to
the one common limit. No matter what traffic it is - IPv4 or IPv6. Sum of
traffic should be the one I specify.
--
Hasso Tepper
Elion Enterprises Ltd.
WAN administrator
It depends on whether you have CONFIG_NET_CLS_ACT enabled or not.
If so, the ingress qdisc is hit before PREROUTING and thus can't
see the mark for a good reason. Simply removing the dependcy on
the mark resolves the issue for you.
If you don't have CONFIG_NET_CLS_ACT enabled you would see the
mark if the ingress qdisc would register on the IPv6 PREROUTING
hook but apparently it doesn't.
The patch below should fix it, it is completely untested though.
Yeah, that would fix it. Note however, that i am trying to highly
discourage use of iptables and i would rather let people who use
iptables to suffer;-> (sounds rude i know). At some point i plan to
remove the dependency on iptables altogether. So i am not sure whether i
should encourage pushing of this patch or not ;->
All this hooking in 100 hooks is one of the reasons i disliked IMQ as
well
cheers,
jamal
PS:- also note in 2.6.6 tc action was not yet in, so
On Mon, 2005-01-31 at 08:28, Thomas Graf wrote:
It depends on whether you have CONFIG_NET_CLS_ACT enabled or not.
If so, the ingress qdisc is hit before PREROUTING and thus can't
see the mark for a good reason. Simply removing the dependcy on
the mark resolves the issue for you.
If you don't have CONFIG_NET_CLS_ACT enabled you would see the
mark if the ingress qdisc would register on the IPv6 PREROUTING
hook but apparently it doesn't.
The patch below should fix it, it is completely untested though.
From: Thomas Graf <tgraf@suug.ch> Date: 2005-01-31 14:06:30
Yeah, that would fix it. Note however, that i am trying to highly
discourage use of iptables and i would rather let people who use
iptables to suffer;-> (sounds rude i know). At some point i plan to
remove the dependency on iptables altogether.
Heh, I think it isn't rude, giving people a little clap to join
the "good side" isn't that bad ;->
So i am not sure whether i should encourage pushing of this patch or not ;->
I don't care that much, the patch is there, everyone can patch and
distributions can pick it up. I agree that we should remove the
dependency on iptables but I'd also like to see the dependency on the
action bits to go away at the same time.
Yeah, that would fix it. Note however, that i am trying to highly
discourage use of iptables and i would rather let people who use
iptables to suffer;-> (sounds rude i know). At some point i plan to
remove the dependency on iptables altogether.
Heh, I think it isn't rude, giving people a little clap to join
the "good side" isn't that bad ;->
Unfortunately killing it totaly would break all sorts of scripts.
Wish we could do this though ;->
Go ahead and push the patch to Dave even - I am just gonna look the
other way;->
quoted
So i am not sure whether i should encourage pushing of this patch or not ;->
I don't care that much, the patch is there, everyone can patch and
distributions can pick it up. I agree that we should remove the
dependency on iptables but I'd also like to see the dependency on the
action bits to go away at the same time.
Agreed.
The main reason not to use the iptables bits is performance; also its
just the wrong spot in the stack.
cheers,
jamal
What are you trying to do? Are you also trying to rate limit ARPs etc
in one shot?
All traffic coming from eth1.101 interface.
eth1.101 is an alias? You may have issues there. Maybe not if the
attach to that interface worked.
quoted
tc filter add dev eth1.101 parent ffff: protocol all prio 50 handle \
0x101 fw police rate 1024kbit burst 60k drop flowid :101
Does this even get hit at all? tc -s would show you stats. I suspect
for one it is not being hit.
As far as I remember situation was exactly as I described. This worked for
IPv4 traffic, but not for IPv6 traffic.
Off hand i cant see why .. Unless the ipv6 packets didnt get marked
properly but the v4 ones did?
quoted
Maybe you are trying to use iptables marks that happen
a long time after the ingress has seen the packets (which would
explain why it is not being hit)? This would be true kernels > 2.6.8
but not before ..
This test was done with 2.6.6.
Ok, in that case iptables prerouting would have come before ingress; so
as long as you marked the packets with iptables it should work fine.
quoted
In other words, it may be a config issue.
Would be nice ;).
I am still thinking it is. What are your iptables v6 markers?
quoted
If you tell me what it is you are trying to do i could try and set it
up when i come back from work today.
I'd like to limit _all_ traffic coming in from one particular interface to
the one common limit. No matter what traffic it is - IPv4 or IPv6. Sum of
traffic should be the one I specify.
Theres other ways to do it[1] but if theres a bug in this it needs
fixing.
cheers,
jamal
[1]
Example, you could do this:
tc filter add dev eth1 parent ffff: protocol ip prio 1 \
u32 match u32 0 0 flowid 1:15 \
action police index 1 rate 1024kbit burst 60k drop index 1
Note the use of "index 1" to select a policer.
Then repeat replacing ip with ip6; make sure that "index 1" for policer
stays. You could do this to share also across devices.
Example, on egress of eth0 also use the same 1Mbps
tc filter add dev eth0 parent 1:0 protocol ip prio 6 u32 \
match ip src 10.0.0.21/32 flowid 1:16 \
action police index 1 rate 1024kbit burst 60k drop index 1
Now with new action stuff you could instead just have said:
tc actions add \
action police index 1 rate 1024kbit burst 60k drop index 1
And then later just referenced it without having to repeat the rate
like so:
filter add dev eth0 parent ffff: protocol ip prio 6 u32 match ip src \
10.0.0.21/32 flowid 1:16 \
action police index 1
Again, this does not excuse a bug if it exists ...
eth1.101 is an alias? You may have issues there. Maybe not if the
attach to that interface worked.
No, vlan.
Theres other ways to do it[1] but if theres a bug in this it needs
fixing.
cheers,
jamal
[1]
Example, you could do this:
tc filter add dev eth1 parent ffff: protocol ip prio 1 \
u32 match u32 0 0 flowid 1:15 \
action police index 1 rate 1024kbit burst 60k drop index 1
Note the use of "index 1" to select a policer.
Then repeat replacing ip with ip6; make sure that "index 1" for policer
stays. You could do this to share also across devices.
Example, on egress of eth0 also use the same 1Mbps
tc filter add dev eth0 parent 1:0 protocol ip prio 6 u32 \
match ip src 10.0.0.21/32 flowid 1:16 \
action police index 1 rate 1024kbit burst 60k drop index 1
Now with new action stuff you could instead just have said:
tc actions add \
action police index 1 rate 1024kbit burst 60k drop index 1
And then later just referenced it without having to repeat the rate
like so:
filter add dev eth0 parent ffff: protocol ip prio 6 u32 match ip src \
10.0.0.21/32 flowid 1:16 \
action police index 1
Hmmm ... I even didn't know about index. Yes, something like that would do
as well probably. I'll do some tests later today with this. Actions don't
help me though as I'm using 2.4 kernel for production.
--
Hasso Tepper
Elion Enterprises Ltd.
WAN administrator
eth1.101 is an alias? You may have issues there. Maybe not if the
attach to that interface worked.
No, vlan.
That should be fine then
quoted
Theres other ways to do it[1] but if theres a bug in this it needs
fixing.
[..]
quoted
And then later just referenced it without having to repeat the rate
like so:
filter add dev eth0 parent ffff: protocol ip prio 6 u32 match ip src \
10.0.0.21/32 flowid 1:16 \
action police index 1
Hmmm ... I even didn't know about index. Yes, something like that would do
as well probably. I'll do some tests later today with this. Actions don't
help me though as I'm using 2.4 kernel for production.
Theres an extra "index 1" in all those examples. remove the first one (I
am sure you will find out when experimenting what the correct syntax
is).
Unfortunately this "index" thing continues to be a big secret although i
have pointed it a few times. Bart should probably add it to his HOWTO.
All actions also have indices and are therefore shareable.
cheers,
jamal
Unfortunately this "index" thing continues to be a big secret although i
have pointed it a few times. Bart should probably add it to his HOWTO.
All this stuff deserves better documentation ;). Call me oldfashioned, but
IMHO would be good start to have all keywords at least somewhat documented
in iproute2 man pages. I know, writing documentation is boring task etc,
but at least you should mention all features in man pages. This gives to
someone at least chance to kick you "hey, what's this?" ;).
This is somewhat related to killing the chance to use iptables as well ...
Iptables has better documentation and people use it just because of that.
Just checked ... "tc action help" in newest iproute2 returns nothing as
well :(.
--
Hasso Tepper
Elion Enterprises Ltd.
WAN administrator
Unfortunately this "index" thing continues to be a big secret although i
have pointed it a few times. Bart should probably add it to his HOWTO.
All this stuff deserves better documentation ;). Call me oldfashioned, but
IMHO would be good start to have all keywords at least somewhat documented
in iproute2 man pages. I know, writing documentation is boring task etc,
but at least you should mention all features in man pages. This gives to
someone at least chance to kick you "hey, what's this?" ;).
But 50 emails later noone documented anything .. And every summer
someone tries to write a qdisc to achive this ;->
This is somewhat related to killing the chance to use iptables as well ...
Iptables has better documentation and people use it just because of that.
Just checked ... "tc action help" in newest iproute2 returns nothing as
well :(.
Should work. Are you sure you got the latest one?
Also make sure you compile tc action in the scheduler part of kernel
config.
cheers,
jamal
On Mon, Jan 31, 2005 at 04:46:14PM +0200, Hasso Tepper wrote:
This is somewhat related to killing the chance to use iptables as well ...
Iptables has better documentation and people use it just because of that.
I'm afraid I have to agree on this one. The idea behind iptables is
easy to grasp, whereas tc isn't totally obvious, and all tc 'tutorials'
out there just give you a long list of commands to type in but don't
really explain you what goes on under the hood.
And you can't just expect everyone to "Go look at the source."
--L
On Mon, 2005-01-31 at 13:00, Lennert Buytenhek wrote:
On Mon, Jan 31, 2005 at 04:46:14PM +0200, Hasso Tepper wrote:
quoted
This is somewhat related to killing the chance to use iptables as well ...
Iptables has better documentation and people use it just because of that.
I'm afraid I have to agree on this one.
Well, if you look at the 2 requirements behind IMQ, has nothing todo
with iptables i.e does not at all require presence of iptables.
So motivation is to meet those requirements not kill iptables.
The idea behind iptables is
easy to grasp, whereas tc isn't totally obvious, and all tc 'tutorials'
out there just give you a long list of commands to type in but don't
really explain you what goes on under the hood.
And you can't just expect everyone to "Go look at the source."
Agreed, tc is less usable and has a lot less people puking code at it.
The usability part has to be fixed. And i think you will see that with
ematch and eaction code showing up. Credit goes to Bart and co and their
website for putting a lot of docs together. Usability certainly needs to
improve!
cheers,
jamal
From: Thomas Graf <tgraf@suug.ch> Date: 2005-01-31 13:58:10
2) Allows for queueing incoming traffic for shaping instead of
dropping. I am not aware of any study that shows policing is
worse than shaping in achieving the end goal of rate control.
I would be interested if anyone is experimenting. Nevertheless,
this is still an alternative as opposed to making a system wide
ingress change.
Agreed, the problem should be solved on egress by delaying ACKs
so the other side's congestion control slows down. I still don't
have a solution which works for all ip stacks and ended up tuning
parameters based on TTL numbers guessing the operating system.
For me, the purpose of ingress policing is to apply some policy for
control datagrams and other unwanted traffic. One example would be
dropping echo requests comming from nmap which reduces egress
bandwidth consumption by 13% my border routers.
tc filter add dev $DEV parent ffff: protocol ip prio 10 \
u32 match u32 0x10000 0xff0000 at 8 \
match u32 0x1c 0xffff at 0 \
match u32 0x8000000 0xf000000 at 20 \
police mtu 1 drop flowid :1
I should convert this to actions at some point ;->
--> Instead the plan is to have a contrack related action. This action
will selectively either query/create contrack state on incoming packets.
Packets could then be redirected to dummy based on what happens -> eg
on incoming packets; if we find they are of known state we could send to
a different queue than one which didnt have existing state. This
all however is dependent on whatever rules the admin enters.
We could also do it in the meta ematch but this relies on the packet
already having passed the conntrack code. How do you plan to do this
in ingress?
tc filter add dev eth0 parent 1: protocol ip prio 10 u32 \
match ip src 192.168.200.200/32 flowid 1:2 \
action police rate 10kbit burst 90k drop \
action mirred egress mirror dev dummy0
This is extremely useful. I'm not sure but I think you also had plans
to allow mirroring to userspace?
My goal here is to start a discussion to see if people agree this is
a good replacement for IMQ or whether to go another path.
Sounds good to me. No complains from my side. I'll have a closer look
at the patch later on.
2) Allows for queueing incoming traffic for shaping instead of
dropping. I am not aware of any study that shows policing is
worse than shaping in achieving the end goal of rate control.
I would be interested if anyone is experimenting. Nevertheless,
this is still an alternative as opposed to making a system wide
ingress change.
Agreed, the problem should be solved on egress by delaying ACKs
so the other side's congestion control slows down.
Or dropping packets. TCP will adjust itself either way; at least
thats true according to this formula [rfc3448] (originally derived from
Reno, but people are finding it works fine with all other variants of
TCP CC):
-----
The throughput equation is:
s
X = ----------------------------------------------------------
R*sqrt(2*b*p/3) + (t_RTO * (3*sqrt(3*b*p/8) * p * (1+32*p^2)))
Where:
X is the transmit rate in bytes/second.
s is the packet size in bytes.
R is the round trip time in seconds.
p is the loss event rate, between 0 and 1.0, of the number of loss
events as a fraction of the number of packets transmitted.
t_RTO is the TCP retransmission timeout value in seconds.
b is the number of packets acknowledged by a single TCP
acknowledgement.
----
dropping mucks with "p" and delaying ACKs (shaping) mucks with "R".
Plug into that formula either one and you see they affect the
result for X the same way.
I am really hoping that someone will do experimental analysis - cant
believe no hungry students these days out there.
I still don't
have a solution which works for all ip stacks and ended up tuning
parameters based on TTL numbers guessing the operating system.
For me, the purpose of ingress policing is to apply some policy for
control datagrams and other unwanted traffic. One example would be
dropping echo requests comming from nmap which reduces egress
bandwidth consumption by 13% my border routers.
tc filter add dev $DEV parent ffff: protocol ip prio 10 \
u32 match u32 0x10000 0xff0000 at 8 \
match u32 0x1c 0xffff at 0 \
match u32 0x8000000 0xf000000 at 20 \
police mtu 1 drop flowid :1
I should convert this to actions at some point ;->
You should ;->
And now you can actually _really_ drop, above will let some packets
through. More interestingly is you can now randomly drop or
determistically (drop every 10th packet)
quoted
--> Instead the plan is to have a contrack related action. This action
will selectively either query/create contrack state on incoming packets.
Packets could then be redirected to dummy based on what happens -> eg
on incoming packets; if we find they are of known state we could send to
a different queue than one which didnt have existing state. This
all however is dependent on whatever rules the admin enters.
We could also do it in the meta ematch but this relies on the packet
already having passed the conntrack code. How do you plan to do this
in ingress?
Something along the lines of what OBSD firewall does but selectively (If
i understood those OBSD fanatics at SUCON;-> correctly)..they track
at ingress before ip stack. The difference is we can allow selective
tracking; something along the lines of:
tc filter add dev $DEV parent ffff: protocol ip prio 10 \
u32 match u32 0x10000 0xff0000 at 8 \
action track \
action metamark here depending on whether we found contrack etc
I have the layout scribbeled on paper somewhere .. I will look it up
and provide more details
Track should just use iptables contracking code instead of reinventing
it.
quoted
tc filter add dev eth0 parent 1: protocol ip prio 10 u32 \
match ip src 192.168.200.200/32 flowid 1:2 \
action police rate 10kbit burst 90k drop \
action mirred egress mirror dev dummy0
This is extremely useful. I'm not sure but I think you also had plans
to allow mirroring to userspace?
Yes via mmaped packet sockets. The other way (induced by laziness, so i
dont have to write a single line of code) is to
have redirection to ring device that was posted a while back by someone
since it provides a bridge between mmaped packet socket like interface
and kernel.
quoted
My goal here is to start a discussion to see if people agree this is
a good replacement for IMQ or whether to go another path.
Sounds good to me. No complains from my side. I'll have a closer look
at the patch later on.
From: Thomas Graf <tgraf@suug.ch> Date: 2005-01-31 15:15:32
Or dropping packets. TCP will adjust itself either way; at least
thats true according to this formula [rfc3448] (originally derived from
Reno, but people are finding it works fine with all other variants of
TCP CC):
-----
The throughput equation is:
s
X = ----------------------------------------------------------
R*sqrt(2*b*p/3) + (t_RTO * (3*sqrt(3*b*p/8) * p * (1+32*p^2)))
Where:
X is the transmit rate in bytes/second.
s is the packet size in bytes.
R is the round trip time in seconds.
p is the loss event rate, between 0 and 1.0, of the number of loss
events as a fraction of the number of packets transmitted.
t_RTO is the TCP retransmission timeout value in seconds.
b is the number of packets acknowledged by a single TCP
acknowledgement.
----
Agreed, this was my first attempt and my current code is still based on
this. I'm trying to avoid a retransmit battle, therefore I try to
delay packets if possible with the hope that it's either just a peak
or the slow down is fast enough. I use a simplified RED and
tcp_xmit_retransmit_queue() input to avoid flick flack effects which
works pretty well for bulky streams. A burst buffer takes care
of interactive traffic with peaks but this doesn't work perfectly fine
yet. Overall, my attempt works pretty well if the other side uses
reno/bic and quite well for westwood and vegas. The problem is not that
it doesn't work at all but achieving a certain _stable_ rate is very
difficult, the delta of the requested and real rate is up to 25% depending
on the constancy of the rtt and wether they follow one of the proposed
tcp cc algorithms. The cc guessing code helps a bit but isn't very
accurate.
Something along the lines of what OBSD firewall does but selectively (If
i understood those OBSD fanatics at SUCON;-> correctly)..they track
at ingress before ip stack. The difference is we can allow selective
tracking; something along the lines of:
This means we'd have to do the most important sanity cehcks ourselves
like checksum and ip header consistencity. Which basically means a
duplication of ip_rcv() and ipv6_rcv().
tc filter add dev $DEV parent ffff: protocol ip prio 10 \
u32 match u32 0x10000 0xff0000 at 8 \
action track \
action metamark here depending on whether we found contrack etc
I have the layout scribbeled on paper somewhere .. I will look it up
and provide more details
Track should just use iptables contracking code instead of reinventing
it.
This is exactly my thinking as well but I'd do it as ematch. Given
we pass the netfilter conntrack code we'd then have access to the
meta data of it such as direction, state and other attributes.
tc filter add dev $DEV parent ffff: protocol ip prio 10 \
u32 match u32 0x10000 0xff0000 at 8 \
and conntrack \
and meta nf_state eq ESTABLISHED \
and meta nf_status eq SEEN_REPLY \
action metamark here depending on whether we found contrack etc
Agreed, this was my first attempt and my current code is still based on
this. I'm trying to avoid a retransmit battle, therefore I try to
delay packets if possible with the hope that it's either just a peak
or the slow down is fast enough. I use a simplified RED and
tcp_xmit_retransmit_queue() input to avoid flick flack effects which
works pretty well for bulky streams. A burst buffer takes care
of interactive traffic with peaks but this doesn't work perfectly fine
yet. Overall, my attempt works pretty well if the other side uses
reno/bic and quite well for westwood and vegas. The problem is not that
it doesn't work at all but achieving a certain _stable_ rate is very
difficult, the delta of the requested and real rate is up to 25% depending
on the constancy of the rtt and wether they follow one of the proposed
tcp cc algorithms. The cc guessing code helps a bit but isn't very
accurate.
My experience is that you end up dropping no more than a packet in a
burst with policing before TCP adjusts. Also depending on the gap
between bursts, that may be the only packet you drop altogether.
In long flows such as file transfers, avergae of one packet ever gets
dropped.
quoted
Something along the lines of what OBSD firewall does but selectively (If
i understood those OBSD fanatics at SUCON;-> correctly)..they track
at ingress before ip stack. The difference is we can allow selective
tracking; something along the lines of:
This means we'd have to do the most important sanity cehcks ourselves
like checksum and ip header consistencity. Which basically means a
duplication of ip_rcv() and ipv6_rcv().
checksum and other validity of ip header will have to be written as an
action if needed. Infact csum is on my list of mini actions. I could
decide to change something on egress of outgoing ip packet in pedit
and would therefore require to recompute csum.
quoted
tc filter add dev $DEV parent ffff: protocol ip prio 10 \
u32 match u32 0x10000 0xff0000 at 8 \
action track \
action metamark here depending on whether we found contrack etc
I have the layout scribbeled on paper somewhere .. I will look it up
and provide more details
Track should just use iptables contracking code instead of reinventing
it.
This is exactly my thinking as well but I'd do it as ematch. Given
we pass the netfilter conntrack code we'd then have access to the
meta data of it such as direction, state and other attributes.
tc filter add dev $DEV parent ffff: protocol ip prio 10 \
u32 match u32 0x10000 0xff0000 at 8 \
and conntrack \
and meta nf_state eq ESTABLISHED \
and meta nf_status eq SEEN_REPLY \
action metamark here depending on whether we found contrack etc
Ok, I think both approaches are correct. ematch does the check/get
essentially; and action will create the set/tracking if needed.
For the example i gave, you are absolutely correct, ematch is
sufficient.
cheers,
jamal
From: Thomas Graf <tgraf@suug.ch> Date: 2005-01-31 15:59:29
My experience is that you end up dropping no more than a packet in a
burst with policing before TCP adjusts. Also depending on the gap
between bursts, that may be the only packet you drop altogether.
In long flows such as file transfers, avergae of one packet ever gets
dropped.
I mostly agree but not completely. It's definitely true that most of
the problems I'm fighting today are causes by the attempt to be too
perfect in calculating. Going a step backwards solves most of the
problems and probably works just fine for most cases. One of the main
problem I'm facing here are big file transfers on low latency links with
modified ip stacks to allow for a "faster" slow start (those are the
reason why I'm trying to do this). An attempt to drop only a few
packets results in a stronger incremenal growth. I'm not quite sure
why that happens yet but a more aggresive policing stategy helped a
lot. I agree that if we plan to put something like this into mainline
those problem domains should be separated to not overcomlicate the
whole thing.
checksum and other validity of ip header will have to be written as an
action if needed. Infact csum is on my list of mini actions. I could
decide to change something on egress of outgoing ip packet in pedit
and would therefore require to recompute csum.
Sounds good. We'll need to address this anyway, the classifiers rely
on the ip header being valid which is no longer assured.
Ok, I think both approaches are correct. ematch does the check/get
essentially; and action will create the set/tracking if needed.
For the example i gave, you are absolutely correct, ematch is
sufficient.
Right, so we can do something like the meta ematch/action split. What
attributes to you intend to be modifieable? A neat thing would be
to overwrite the state and thus assign a packet to another connection
which could be used to reimplement fast nat together with pedit.
My experience is that you end up dropping no more than a packet in a
burst with policing before TCP adjusts. Also depending on the gap
between bursts, that may be the only packet you drop altogether.
In long flows such as file transfers, avergae of one packet ever gets
dropped.
I mostly agree but not completely. It's definitely true that most of
the problems I'm fighting today are causes by the attempt to be too
perfect in calculating. Going a step backwards solves most of the
problems and probably works just fine for most cases. One of the main
problem I'm facing here are big file transfers on low latency links with
modified ip stacks to allow for a "faster" slow start (those are the
reason why I'm trying to do this). An attempt to drop only a few
packets results in a stronger incremenal growth. I'm not quite sure
why that happens yet but a more aggresive policing stategy helped a
lot. I agree that if we plan to put something like this into mainline
those problem domains should be separated to not overcomlicate the
whole thing.
Would be interesting to combine policing and random dropping to see
what happens.
I think this is something you should be able to find a student to abuse
so they can write a paper ;-> Probably Linux may not even be the right
place to do it to start with - rather simulations until you get it right
then code it into Linux.
Sounds good. We'll need to address this anyway, the classifiers rely
on the ip header being valid which is no longer assured.
true - i was thinking of restoring stateless NAT at this level as well.
So csum would be needed. The csum could be programmed to either
validate only or recompute; those are the only two arguements to it that
i could think of. I suppose first thing is to put out the eaction patch
then add this action. I will try to sneak in some time this week and
write the eaction.
quoted
Ok, I think both approaches are correct. ematch does the check/get
essentially; and action will create the set/tracking if needed.
For the example i gave, you are absolutely correct, ematch is
sufficient.
Right, so we can do something like the meta ematch/action split. What
attributes to you intend to be modifieable?
Essentially on ingress create state; i have to find my notes to give you
precise answer. But one of the parameters was to select the level of
state tracking (such as "track IP only" - not sure how doable that is
with contrack)
A neat thing would be
to overwrite the state and thus assign a packet to another connection
which could be used to reimplement fast nat together with pedit.
Stateless NAT doesnt really need contracking. pedit (taught to speak
english) + eaction csum should do it.
cheers,
jamal
From: Thomas Graf <tgraf@suug.ch> Date: 2005-01-31 18:15:53
Would be interesting to combine policing and random dropping to see
what happens.
Indeed.
Probably Linux may not even be the right place to do it to start with
- rather simulations until you get it right then code it into Linux.
I haven't left ns sim so far, my calculations are based on some of the
linux specific cc modifications though, i.e. I modified ns sim a bit
to provide the same information. I'm thinking about moving to umlsim,
it should provide a better real world simulation.
true - i was thinking of restoring stateless NAT at this level as well.
So csum would be needed. The csum could be programmed to either
validate only or recompute; those are the only two arguements to it that
i could think of. I suppose first thing is to put out the eaction patch
then add this action. I will try to sneak in some time this week and
write the eaction.
Sounds good, we could put up a ematch csum for validation and a eaction
for recomputation. I'll wait for your code to show up.
quoted
Right, so we can do something like the meta ematch/action split. What
attributes to you intend to be modifieable?
Essentially on ingress create state; i have to find my notes to give you
precise answer. But one of the parameters was to select the level of
state tracking (such as "track IP only" - not sure how doable that is
with contrack)
So you want to have a generic conntrack action capable of dynamically
taking whatever information into account that the user requests? This
remembers me of the esfq effort which could benefit from this, it
extends sfq to take the definition for a flow as a parameter. We could
share some code here.
Stateless NAT doesnt really need contracking. pedit (taught to speak
english) + eaction csum should do it.
Right, given we don't need any reverse translation. Still it would be
neat to set the conntrack attributes so one could use iptables later
on, I'm not sure how doable this is though.
Something different...
This sounds all very good but I think we're still sucessfully ignoring
one of the most important points, usability. Most modifications over
the last few months have complicated things, introduced different behaviour
depending on compile time options and userspace tools which are either
outdated or having features being completely undocumented. Some of the
recent additions don't even show up in the usage text of iproute2. So
I think we should at least part time focus a little more on the big
picture and make things consitent and more useable. At least 50% of the
functionaility currently in mainline is completely unused because nobody
knows about it. I'm in no way against any of the recent additions but
maybe we can also put some more effort into usability.
I haven't left ns sim so far, my calculations are based on some of the
linux specific cc modifications though, i.e. I modified ns sim a bit
to provide the same information. I'm thinking about moving to umlsim,
it should provide a better real world simulation.
The problem is if theres any bugs in the way algos are implemented in
Linux you are influenced by that truth. Starting with ns and then
validating on Linux is a great way to do it.
Sounds good, we could put up a ematch csum for validation and a eaction
for recomputation. I'll wait for your code to show up.
Cross your fingers; worst case by weekend i should get something out.
quoted
Essentially on ingress create state; i have to find my notes to give you
precise answer. But one of the parameters was to select the level of
state tracking (such as "track IP only" - not sure how doable that is
with contrack)
So you want to have a generic conntrack action capable of dynamically
taking whatever information into account that the user requests? This
remembers me of the esfq effort which could benefit from this, it
extends sfq to take the definition for a flow as a parameter. We could
share some code here.
I dont think contrack was designed for this kind of effort. If we totaly
fail to do it using contrack then we could go a different path.
sfq already stores some rough view of the state; not sure if it can
benefit from this.
quoted
Stateless NAT doesnt really need contracking. pedit (taught to speak
english) + eaction csum should do it.
Right, given we don't need any reverse translation. Still it would be
neat to set the conntrack attributes so one could use iptables later
on, I'm not sure how doable this is though.
If you are NATing (stateless) you should enter rules for both
directions; Maybe we could write a wrapper where user only enters
outgoing rule and that automatically generates the incoming rule as
well.
Something different...
This sounds all very good but I think we're still sucessfully ignoring
one of the most important points, usability.
Absolutely.
Most modifications over
the last few months have complicated things, introduced different behaviour
depending on compile time options and userspace tools which are either
outdated or having features being completely undocumented. Some of the
recent additions don't even show up in the usage text of iproute2. So
I think we should at least part time focus a little more on the big
picture and make things consitent and more useable. At least 50% of the
functionaility currently in mainline is completely unused because nobody
knows about it. I'm in no way against any of the recent additions but
maybe we can also put some more effort into usability.
I think the eactions etc are adding a lot of value towards usability.
Hasso Tepper was ealrier complaining about this same issue.
As an example, I think u32 and ematches would improve a great deal now
and be more understandable. True, work/time still needs to be invested.
cheers,
jamal
From: Thomas Graf <tgraf@suug.ch> Date: 2005-01-31 22:53:28
* jamal [off-list ref] 2005-01-31 15:18
On Mon, 2005-01-31 at 13:15, Thomas Graf wrote:
quoted
I haven't left ns sim so far, my calculations are based on some of the
linux specific cc modifications though, i.e. I modified ns sim a bit
to provide the same information. I'm thinking about moving to umlsim,
it should provide a better real world simulation.
The problem is if theres any bugs in the way algos are implemented in
Linux you are influenced by that truth. Starting with ns and then
validating on Linux is a great way to do it.
Absolutely, that's why I first went to ns sim but a nice theory is
worth nothing if it doesn't work in the real world.
I dont think contrack was designed for this kind of effort. If we totaly
fail to do it using contrack then we could go a different path.
sfq already stores some rough view of the state; not sure if it can
benefit from this.
I was thinking of the parameters to define what a flow consists of.
Extended SFQ basically allows you to define the hash function. I think
I misunderstood you before and you don't want allow adjustable
states on only a subset of the attributes, e.g. only L3 data.
If you are NATing (stateless) you should enter rules for both
directions; Maybe we could write a wrapper where user only enters
outgoing rule and that automatically generates the incoming rule as
well.
Agreed iff we don't enforce it.
I think the eactions etc are adding a lot of value towards usability.
Hasso Tepper was ealrier complaining about this same issue.
As an example, I think u32 and ematches would improve a great deal now
and be more understandable. True, work/time still needs to be invested.
I'd guess that the basic classifier will make the race because the
documentation will be smaller due to the lack of parameters. ;->
But yes I agree, I think we're making small step forwards and hopefully
the network config shell/tool/whatever will ease the steps to configure
things. My primary goal is to allow using it without looking up
parameters all the time, given one is aware of the common terms and basic
concepts. I'll have some more time next week and will try to implement the
traffic control bits or at least some of them. The wind forecast is pretty
good for the next days so I won't have too much time. ;->
I was thinking of the parameters to define what a flow consists of.
Extended SFQ basically allows you to define the hash function. I think
I misunderstood you before and you don't want allow adjustable
states on only a subset of the attributes, e.g. only L3 data.
Why bother putting extra classifier functionality into a qdisc?
you should be able to rip off the classifier from sfq so you dont depend
on it; you can then select one of n queues (eaction meta set class 1:X
based on result of sfq classifier - or you can have it set the classids
based on resulting hash index)
quoted
I think the eactions etc are adding a lot of value towards usability.
Hasso Tepper was ealrier complaining about this same issue.
As an example, I think u32 and ematches would improve a great deal now
and be more understandable. True, work/time still needs to be invested.
I'd guess that the basic classifier will make the race because the
documentation will be smaller due to the lack of parameters. ;->
Well, even if it is just being able to describe in english the u32
parameters and displaying them in english (by using a ID stored)
its already huge progress.
But yes I agree, I think we're making small step forwards and hopefully
the network config shell/tool/whatever will ease the steps to configure
things. My primary goal is to allow using it without looking up
parameters all the time, given one is aware of the common terms and basic
concepts.
Online easy to use help is always valuable.
I'll have some more time next week and will try to implement the
traffic control bits or at least some of them. The wind forecast is pretty
good for the next days so I won't have too much time. ;->
Weather is also predicted to be good here for the week; we are planning to
get out of our igloos and go tobagoning;->
cheers,
jamal
From: Thomas Graf <tgraf@suug.ch> Date: 2005-02-01 12:51:47
Why bother putting extra classifier functionality into a qdisc?
you should be able to rip off the classifier from sfq so you dont depend
on it; you can then select one of n queues (eaction meta set class 1:X
based on result of sfq classifier - or you can have it set the classids
based on resulting hash index)
Excellent idea, this would allow for various hash functions to be used
in a single sfq. We can use skb->tc_index for it so we can easly combine
it with a underlying dsmark. The hardest part is to find a intuitive
form to define the hash, it should be possible to for example define
a hash based on daddr + hproto only completely ignoring saddr. The
perutrbation must be made optional, sometimes the hash will not produce
any unwanted collisions (hash based on dscp for example) so modifying it
wouldn't make sense. We can fork sfq and make a gsfq which takes the
hash from tc_index and disabled perturbation if it is set to 0.
Thoughts?
Well, even if it is just being able to describe in english the u32
parameters and displaying them in english (by using a ID stored)
its already huge progress.
True. I've some notes on paper describing a match definition db which
basically defines a u32 like match and assigns a name and id to it, it
is stored in a external database file so everyone can define their
own pre defined matches without recompiling.
I've put together some code printing a tc tree as a whole and added
it to netconfig. It's just a start and still contains redundant
information which can be removed but I think it's already a step
forward because it all gets down to one command. Currently it's only
possible to filter on the device but I'll extend this later so one
can extract a part of the tree. One pretty ugly thing is that cbq
creates a qdisc and class with the same handle which gets quite
confusing if one wants list the filters attached to a certain handle
because they will show up for both, the qdisc and the root class.
Full output for a default ethernet device:
lsx# tc tree full where device eth0
eth0 ether 00:02:44:63:ed:53 mtu 1500 <BROADCAST,MULTICAST,UP>
txqlen 1000 weight 64 qdisc pfifo_fast irq 19 index 4 brd ff:ff:ff:ff:ff:ff
pfifo_fast qdisc dev eth0 handle none parent none bands 3
refcnt 1 priomap [1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1]
besteffort => 1 0x8 => 1
filler => 2 0x9 => 1
bulk => 2 0xa => 1
0x3 => 2 0xb => 1
interactive_bulk => 1 0xc => 1
0x5 => 2 0xd => 1
interactive => 0 0xe => 1
control => 0 0xf => 1
Brief output for some cbq classes:
lsx# tc tree where device eth0
eth0 ether 00:02:44:63:ed:53 mtu 1500 <BROADCAST,MULTICAST,UP>
cbq qdisc dev eth0 handle 10: parent none rate 11.92MiB/s (95Mbit) prio 8
cbq class dev eth0 handle 10: parent root rate 11.92MiB/s (95Mbit) prio 8
u32 cls dev eth0 handle none parent 10: prio 10 protocol ip
u32 cls dev eth0 handle 8000: parent 10: prio 10 protocol ip divisor 1
u32 cls dev eth0 handle 8000:800 parent 10: prio 10 protocol ip target 10:12
cbq class dev eth0 handle 10:12 parent 10: rate 11.92MiB/s (95Mbit) prio 3
sfq qdisc dev eth0 handle 8003: parent 10:12 quantum 1514 perturb 0us
Full output with stats for some cbq classes:
lsx# tc tree stats where device eth0
eth0 ether 00:02:44:63:ed:53 mtu 1500 <BROADCAST,MULTICAST,UP>
txqlen 1000 weight 64 qdisc cbq irq 19 index 4 brd ff:ff:ff:ff:ff:ff
Stats: bytes packets errors dropped fifo-err compressed
RX 46.65 MiB 42211 0 0 0 0
TX 1.91 MiB 16234 0 0 0 0
Errors: length over crc frame missed multicast
RX 0 0 0 0 0 0
Errors: aborted carrier heartbeat window collision
TX 0 0 0 0 0
cbq qdisc dev eth0 handle 10: parent none rate 11.92MiB/s (95Mbit) prio 8
refcnt 1 avgpkt 1400 mpu 64 cell 16 allot 1514 weight 95Mbit
minidle 65535999us maxidle 2us offtime 0us level 1 ewma_log 5
penalty 0us strategy classic split none defmap 0x00000000 police ok
Stats: bytes packets drops overlimits qlen backlog
44.02 KiB 380 0 0 0 0
0.00 B/s 0 pps
borrows overact avgidle undertime
0 0 114 0
cbq class dev eth0 handle 10: parent root rate 11.92MiB/s (95Mbit) prio 8
avgpkt 1400 mpu 64 cell 16 allot 1514 weight 95Mbit
minidle 65535999us maxidle 2us offtime 0us level 1 ewma_log 5
penalty 0us strategy classic split none defmap 0x00000000 police ok
Stats: bytes packets drops overlimits qlen backlog
44.02 KiB 380 0 0 0 0
0.00 B/s 0 pps
borrows overact avgidle undertime
0 0 114 0
u32 cls dev eth0 handle none parent 10: prio 10 protocol ip
Stats: bytes packets drops overlimits qlen backlog
0.00 B 0 0 0 0 0
0.00 B/s 0 pps
u32 cls dev eth0 handle 8000: parent 10: prio 10 protocol ip divisor 1
Stats: bytes packets drops overlimits qlen backlog
0.00 B 0 0 0 0 0
0.00 B/s 0 pps
u32 cls dev eth0 handle 8000:800 parent 10: prio 10 protocol ip target 10:12
nkeys 1 ht key 0x800 hash 0x0 <TERMINAL>
match u32 at 8 & 0x00ff0000 == 0x00010000 successful 34
Stats: bytes packets drops overlimits qlen backlog
0.00 B 0 0 0 0 0
0.00 B/s 0 pps
successful hits
34 379
cbq class dev eth0 handle 10:12 parent 10: rate 11.92MiB/s (95Mbit) prio 3
child-qdisc 8003: avgpkt 500 mpu 0 cell 8 allot 1514 weight 95Mbit
minidle 65535999us maxidle 0us offtime 0us level 0 ewma_log 5
penalty 0us strategy classic split none defmap 0x00000000 police ok
Stats: bytes packets drops overlimits qlen backlog
3.25 KiB 34 0 0 0 0
0.00 B/s 0 pps
borrows overact avgidle undertime
0 0 40 0
sfq qdisc dev eth0 handle 8003: parent 10:12 quantum 1514 perturb 0us
refcnt 1 limit 128 divisor 1024 flows 128
Stats: bytes packets drops overlimits qlen backlog
3.25 KiB 34 0 0 0 0
0.00 B/s 0 pps
Why bother putting extra classifier functionality into a qdisc?
you should be able to rip off the classifier from sfq so you dont depend
on it; you can then select one of n queues (eaction meta set class 1:X
based on result of sfq classifier - or you can have it set the classids
based on resulting hash index)
Excellent idea, this would allow for various hash functions to be used
in a single sfq. We can use skb->tc_index for it so we can easly combine
Let the meta action do that. Just set the skb->tc_classid in my opinion;
recall we can change classid now ..
it with a underlying dsmark. The hardest part is to find a intuitive
form to define the hash, it should be possible to for example define
a hash based on daddr + hproto only completely ignoring saddr. The
perutrbation must be made optional, sometimes the hash will not produce
any unwanted collisions (hash based on dscp for example) so modifying it
wouldn't make sense. We can fork sfq and make a gsfq which takes the
hash from tc_index and disabled perturbation if it is set to 0.
Thoughts?
You can let the user define that via tc but have a default;
eg:
tc dev eth0 add sfq ematch
tc dev eth0 set sfq pertub xxx
match u32 ...
ematch sfq
ematch meta classid 1:2
eaction meta set tcindex 101
eaction meta set fwmark ..
etc
I have to run, havent looked at the rest of your email - will later.
cheers,
jamal
From: Thomas Graf <tgraf@suug.ch> Date: 2005-02-01 22:44:36
Let the meta action do that. Just set the skb->tc_classid in my opinion;
recall we can change classid now ..
True, I don't really care but it's already quite confusing. The priority
of the packet is described in viarous field depeding on which qdisc/cls
being used, we have skb->priority with sch_prio, tc_index for dsmark
and cls_tcindex and now tc_classid directly. Some even use u32 to
match on DSCP and set a nfmark. I can already feel the perfect confusion
once we open up access for rt_classid, realm and other routing fields.
I'm always aiming for easy to understand solutions, this doesn't mean
it to be simple. Why is nfmark so heavly used? Because it's damn simple
to undertand, you can set and read it and that's it. The only thing one
has to care about is to make sure that is actually gets set before it being
read and to make sure all userspace apps read it in the same base ;->
(This is basically one of the issue in usability, the lack of clearliness
in what base number are read the displayed. Often they get displayed in
hex without a 0x prefix but are read with strtol(...,0) resulting in
a decimal reading if no prefix is specified)
Long rant short statement, I'm pleading for a generic way to transfer
such things between a classifier and a qdisc because it's simply
easier to explain and use.
... eaction meta set tc_index ip_saddr_proto_hash
... qdisc sfq tcindex-hash
where ip_saddr_proto_hash is not a variable but rather a special meta
value calulated in the kernel.
You can let the user define that via tc but have a default;
eg:
tc dev eth0 add sfq ematch
tc dev eth0 set sfq pertub xxx
match u32 ...
ematch sfq
ematch meta classid 1:2
eaction meta set tcindex 101
eaction meta set fwmark ..
I think we're on the same road or at least going into the same direction.
However I'm not sure whether it's a good to have ematches return
some values besides true/false. I'd rather like to see an eaction store
it in the skb and the sfq catching it up again. Of course we can have the
userspace part be configured within the sfq.
Let the meta action do that. Just set the skb->tc_classid in my opinion;
recall we can change classid now ..
True, I don't really care but it's already quite confusing. The priority
of the packet is described in viarous field depeding on which qdisc/cls
being used, we have skb->priority with sch_prio, tc_index for dsmark
and cls_tcindex and now tc_classid directly. Some even use u32 to
match on DSCP and set a nfmark. I can already feel the perfect confusion
once we open up access for rt_classid, realm and other routing fields.
I'm always aiming for easy to understand solutions, this doesn't mean
it to be simple. Why is nfmark so heavly used? Because it's damn simple
to undertand, you can set and read it and that's it. The only thing one
has to care about is to make sure that is actually gets set before it being
read and to make sure all userspace apps read it in the same base ;->
(This is basically one of the issue in usability, the lack of clearliness
in what base number are read the displayed. Often they get displayed in
hex without a 0x prefix but are read with strtol(...,0) resulting in
a decimal reading if no prefix is specified)
So let me put it this way:
You cant avoid passing around metadata between the different blocks.
Whether the metadata is set by the admin or by some other block along
the packet path is way of life.
All of the metadata defined and attached around skbs so far has a
standalone semantical meaning whic unfortunately cant just be hidden
from the user. Its the unfortunate consequence of giving someone a
weapon )they may shoot their toe off).
As an example:
People have been setting flowid/classid for years via the classifiers
to stamp session a flow belongs to. All we are doing with
skb->tc_classid is giving more power to them. i.e before you get to the
queue given certain computation/state you may decide to belong to a
different session.
sfq as a matter of setting the hash is computing what flow you belong to
and thats why i suggested tc_classid (in this case not set by the admin,
rather by a smart stateful classifier).
Long rant short statement, I'm pleading for a generic way to transfer
such things between a classifier and a qdisc because it's simply
easier to explain and use.
... eaction meta set tc_index ip_saddr_proto_hash
... qdisc sfq tcindex-hash
where ip_saddr_proto_hash is not a variable but rather a special meta
value calulated in the kernel.
Let me see if i understood correctly: Instead of giving static values
(such as 0x10) you want to assign a variable(ip_saddr_proto_hash) which
is computed at runtime to tcindex?
Thats a parallel issue though but indeed useful .
quoted
You can let the user define that via tc but have a default;
eg:
tc dev eth0 add sfq ematch
tc dev eth0 set sfq pertub xxx
match u32 ...
ematch sfq
ematch meta classid 1:2
eaction meta set tcindex 101
eaction meta set fwmark ..
I think we're on the same road or at least going into the same direction.
However I'm not sure whether it's a good to have ematches return
some values besides true/false. I'd rather like to see an eaction store
it in the skb and the sfq catching it up again. Of course we can have the
userspace part be configured within the sfq.
A classifier is allowed to select/set the class/flow/sessionID.
The sfq hash result should at least set/map to the minor value of the
classid
cheers,
jamal
From: Thomas Graf <tgraf@suug.ch> Date: 2005-02-02 15:40:41
First of all, sorry for the massive amount of typos in my last post.
I could barely see anything due to the sun shining onto my display.
You cant avoid passing around metadata between the different blocks.
Whether the metadata is set by the admin or by some other block along
the packet path is way of life.
Agreed.
All of the metadata defined and attached around skbs so far has a
standalone semantical meaning whic unfortunately cant just be hidden
from the user. Its the unfortunate consequence of giving someone a
weapon )they may shoot their toe off).
Agreed, I'm just trying avoid further confusion. I think my country
has one of highest if not the higest density of fully automatic
assault weapons (because everyone liable to miltary service needs
to have one at home), everyone owning one is forced to practice once
a year and shooting is a common sport. OTOH, we have one of the lowest
crime rates. Why's that? Because almost everyone is well educated
in terms of weapon saftey, so I think this should be our way as well.
So yes, we can definitely add more complexity but we should try to make
it easy to understand and use.
sfq as a matter of setting the hash is computing what flow you belong to
and thats why i suggested tc_classid (in this case not set by the admin,
rather by a smart stateful classifier).
If the value for tc_classid is directly set by the user then I agree.
What I want to avoid is having hidden uses of parameters which can also
be modified by the user. It results in a backward compatibility hell
later on because we can't just add another use for it without possibily
breaking scripts.
Let me see if i understood correctly: Instead of giving static values
(such as 0x10) you want to assign a variable(ip_saddr_proto_hash) which
is computed at runtime to tcindex?
Thats a parallel issue though but indeed useful .
OK, so basically we weren't talking of exactly the same thing. In a
user setting only context your argumentation makes sense. Let me follow
on this thought a little further, what I basically want is a generic
way to influence various qdiscs, be it a hashing index for sfq, a
priority value for priority band based qdiscs, etc.
tc_classid isn't a bad choice but it gets complex once we want a
classful qdiscs to be able to use this input parameter.
Summarizing what we currently have:
tc_index: May contain a dscp value if dsmark is told to fetch the dscp
field, the minor part of priority if dsmark is told to map
priority values via handle values, or the minor part of the
classid in a classifier result via ingress classification or
a classifier attached to a dsmark. cls_tcindex, gred, and
meta ematch use it as input value.
nf_mark: cls_fw map to classids, meta ematch may read it, meta
eaction may set it.
tc_classid: Actions may set it to overwrite the result of a classifer,
meta ematch may read it and I guess meta eaction may
write to it.
tc_verd: Set early in net stack, used to track location and tc
relevant flags.
tclassid: Set withing routing db, may be read via meta ematch.
At the moment all of them can be described properly and it should be
easy to understand if the relations are outlined properly.
Assuming we allow setting tc_classid to overwrite the sfq internal
hash we introduce a not so obvious double use because tc_classid is
assumed to at least partly point to a class. We can redefine tc_classid
as being a generic flow/session descriptor but then it should be moved
out of being used to overwrite the classid within actions. Assuming I
have a classifier which normally classifies into a child class but
sometimes I want the traffic to go into a leaf sfq qdisc by using the
action to overwrite the result. It will then be impossible to overwrite
the sfq hash because I would no longer be able to overwrite the
classifier result. It's probably possible to find some working solution
by having the minor part being the sfq input or vice versa but it
gets really nasty. Therefore I think we should make a difference between
the current use of tc_classid to overwrite the classifier result and
giving qdiscs some kind of input not directly related to their handle.
Thoughts?
From: Thomas Graf <tgraf@suug.ch> Date: 2005-02-02 15:55:43
tc_index: May contain a dscp value if dsmark is told to fetch the dscp
field, the minor part of priority if dsmark is told to map
priority values via handle values, or the minor part of the
classid in a classifier result via ingress classification or
a classifier attached to a dsmark. cls_tcindex, gred, and
meta ematch use it as input value.
Assuming we use tc_index to provide the hash...
- we don't need to worry about any definitions. tc_index already stands for
some kind of index grouping together various packets.
- we can directly use sfq to do fair queueing on dscp values and skb
priority including specialized map with a underlying dsmark or cls_tcindex.
From: "David S. Miller" <davem@davemloft.net> Date: 2005-01-31 20:28:04
On 31 Jan 2005 10:40:44 -0500
jamal [off-list ref] wrote:
My experience is that you end up dropping no more than a packet in a
burst with policing before TCP adjusts. Also depending on the gap
between bursts, that may be the only packet you drop altogether.
In long flows such as file transfers, avergae of one packet ever gets
dropped.
Keep in mind that this does not help people with connection
heavy access patterns. If you have a lot of people doing
small transactions, ACK pacing as well as data traffic
dropping is necessary.
The heart of TCP pacing is ACK rates. All of it's data
sending is clocked via ACK arrival.
Therefore the best scheme seems to be ACK pacing along
with data dropping. The ACK pacing is the "nice" policing
where as the data dropping is the big hammer. Ideally, the
ACK pacing will produce the desired data rate and thus the
data dropping will not be necessary.
ACK pacing is more desirable also because of schemes such
as VEGAS congestion control which wish to test the limits
of a link without any data drops. It's basic idea is that
"if my delay increases, yet my throughput does not, I am
doing nothing more than eating router queue space and
therefore have gone beyond the limits of this path, back off"
I know there are problems with VEGAS, but it is a good example
to use in showing that the way to tame TCP's data sending rate
is by controlling the ACKs not by dropping the data, as a first
order method of policing.
From: Andy Furniss <hidden> Date: 2005-02-01 01:02:12
Thomas Graf wrote:
quoted
Or dropping packets. TCP will adjust itself either way; at least
thats true according to this formula [rfc3448] (originally derived from
Reno, but people are finding it works fine with all other variants of
TCP CC):
-----
The throughput equation is:
s
X = ----------------------------------------------------------
R*sqrt(2*b*p/3) + (t_RTO * (3*sqrt(3*b*p/8) * p * (1+32*p^2)))
Where:
X is the transmit rate in bytes/second.
s is the packet size in bytes.
R is the round trip time in seconds.
p is the loss event rate, between 0 and 1.0, of the number of loss
events as a fraction of the number of packets transmitted.
t_RTO is the TCP retransmission timeout value in seconds.
b is the number of packets acknowledged by a single TCP
acknowledgement.
WRT policers I never figured out where you would put the effects of
playing with the burst size parameter and it's effects with few/many
connections and any burstiness caused into an equasion like that.
quoted
----
Agreed, this was my first attempt and my current code is still based on
this. I'm trying to avoid a retransmit battle, therefore I try to
delay packets if possible with the hope that it's either just a peak
or the slow down is fast enough. I use a simplified RED and
tcp_xmit_retransmit_queue() input to avoid flick flack effects which
works pretty well for bulky streams. A burst buffer takes care
of interactive traffic with peaks but this doesn't work perfectly fine
yet. Overall, my attempt works pretty well if the other side uses
reno/bic and quite well for westwood and vegas. The problem is not that
it doesn't work at all but achieving a certain _stable_ rate is very
difficult, the delta of the requested and real rate is up to 25% depending
on the constancy of the rtt and wether they follow one of the proposed
tcp cc algorithms. The cc guessing code helps a bit but isn't very
accurate.
This sounds cool. For me in someways I think it could be nicer (in the
case of shaping from the wrong end of a slow link) to delay the real
packets - that way the tcps of the clients get to see the smoothed
version of the traffic and you can delay udp aswell.
How intelligent and how much, if any, per connection state do you/could
you keep? I think being able to set a class that behaves as full before
it is, removing the s from sfq, de piggybacking acks and singling out
and handling slowstart connections specially could really help the world
of shaping from the wrong end of slow links.
There's always playing with rwin, but maybe that's abit OTT :-)
Andy.
From: Thomas Graf <tgraf@suug.ch> Date: 2005-02-01 13:31:38
quoted
quoted
X = ----------------------------------------------------------
R*sqrt(2*b*p/3) + (t_RTO * (3*sqrt(3*b*p/8) * p * (1+32*p^2)))
Where:
X is the transmit rate in bytes/second.
s is the packet size in bytes.
R is the round trip time in seconds.
p is the loss event rate, between 0 and 1.0, of the number of loss
events as a fraction of the number of packets transmitted.
t_RTO is the TCP retransmission timeout value in seconds.
b is the number of packets acknowledged by a single TCP
acknowledgement.
WRT policers I never figured out where you would put the effects of
playing with the burst size parameter and it's effects with few/many
connections and any burstiness caused into an equasion like that.
A burst buffer has impact on R on later packets, it can "smooth" R
and X and thus results in more stable rates. Depending on the actual
burst, it can avoid retransmits which stabilizes the rate as well.
This sounds cool. For me in someways I think it could be nicer (in the
case of shaping from the wrong end of a slow link) to delay the real
packets - that way the tcps of the clients get to see the smoothed
version of the traffic and you can delay udp aswell.
It's impossible to never drop anything, for udp we can either drop
it or use ECN and hope the other ip stack takes care of it or the
application implements its own cc algorithm. Basically you can already
do that with (G)RED. Most UDP users which provide a continous stream
such as video streams, implement some kind of key datagram which contains
the number of datagrams received since the last key datagram and the
application throttles down based on that so dropping is often the only
way to achieve a general working solution. Delaying UDP packets and
then drop them if the buffer is full is very dangerous, often the
protocols based on UDP rely on the assumption that datagrams get lost
randomly and not succcessive. We can think about precicse policing
for UDP again once the current poor application level cc algorithms
have failed and the industry accepted ECN as the right thing. For
now most of them still suffer from the NIH syndrom in this area.
How intelligent and how much, if any, per connection state do you/could
you keep?
I keep a rate estimator for every flow on ingress in a hash table and
lookup it up on egress with the flow parameters reversed. It gets
pretty expensive on huge amounts of connection usually one doesn't
want to do per connection policing on such boxes. ;->
From: Andy Furniss <hidden> Date: 2005-02-01 15:03:49
Thomas Graf wrote:
quoted
quoted
quoted
X = ----------------------------------------------------------
R*sqrt(2*b*p/3) + (t_RTO * (3*sqrt(3*b*p/8) * p * (1+32*p^2)))
Where:
X is the transmit rate in bytes/second.
s is the packet size in bytes.
R is the round trip time in seconds.
p is the loss event rate, between 0 and 1.0, of the number of loss
events as a fraction of the number of packets transmitted.
t_RTO is the TCP retransmission timeout value in seconds.
b is the number of packets acknowledged by a single TCP
acknowledgement.
WRT policers I never figured out where you would put the effects of
playing with the burst size parameter and it's effects with few/many
connections and any burstiness caused into an equasion like that.
A burst buffer has impact on R on later packets, it can "smooth" R
and X and thus results in more stable rates. Depending on the actual
burst, it can avoid retransmits which stabilizes the rate as well.
But it's not a real rate limiting buffer in the policer case is it?
quoted
This sounds cool. For me in someways I think it could be nicer (in the
case of shaping from the wrong end of a slow link) to delay the real
packets - that way the tcps of the clients get to see the smoothed
version of the traffic and you can delay udp aswell.
It's impossible to never drop anything, for udp we can either drop
it or use ECN and hope the other ip stack takes care of it or the
application implements its own cc algorithm. Basically you can already
do that with (G)RED. Most UDP users which provide a continous stream
such as video streams, implement some kind of key datagram which contains
the number of datagrams received since the last key datagram and the
application throttles down based on that so dropping is often the only
way to achieve a general working solution. Delaying UDP packets and
then drop them if the buffer is full is very dangerous, often the
protocols based on UDP rely on the assumption that datagrams get lost
randomly and not succcessive. We can think about precicse policing
for UDP again once the current poor application level cc algorithms
have failed and the industry accepted ECN as the right thing. For
now most of them still suffer from the NIH syndrom in this area.
Interesting stuff. I was thinking of game udp where just dropping would
simulate what the user should have done anyway, but costing you
bandwidth. If alot of gamers share a slow link then if you lag them out
they know it's time to turn the rate down.
quoted
How intelligent and how much, if any, per connection state do you/could
you keep?
I keep a rate estimator for every flow on ingress in a hash table and
lookup it up on egress with the flow parameters reversed. It gets
pretty expensive on huge amounts of connection usually one doesn't
want to do per connection policing on such boxes. ;->
Nice - are you planning to add anything to tweak things for the wrong
end of the bottleneck problems?
Andy.
From: Thomas Graf <tgraf@suug.ch> Date: 2005-02-02 13:28:16
quoted
quoted
WRT policers I never figured out where you would put the effects of
playing with the burst size parameter and it's effects with few/many
connections and any burstiness caused into an equasion like that.
A burst buffer has impact on R on later packets, it can "smooth" R
and X and thus results in more stable rates. Depending on the actual
burst, it can avoid retransmits which stabilizes the rate as well.
But it's not a real rate limiting buffer in the policer case is it?
Abstractly speaking, burst specifies the maximum amount of time allowed
for a single packet to sit in the burst buffer. Although the burst is
configured as the size of the buffer it is transformed into a time
delta before providing it to the kernel. Because the policer doesn't
enqueue things the packet simply gets dropped if it would exceed that
time. It's not _exactly_ like this but it gives you an idea what
happens, net/sched/police.c isn't that big so one coffee should do it.
Nice - are you planning to add anything to tweak things for the wrong
end of the bottleneck problems?
I hope so, once I figured out an acceptable compromise between a good
result and simplicity. Currently it would be way to expensive and hard
to use.
From: Andre Correa <hidden> Date: 2005-01-31 16:27:03
Hi all,
it turned an year since we (me and some cool folks) got the original IMQ
from "death". During this year we updated kernel and iptables patches
for every available version, created some new features (like hooking
after and before NAT, multiple IMQ devices, solved modules problems,
etc), and helped lots of users in our mailling list. The wish list grew,
we created a site/FAQ/WiKi. We are still missing "dumb device"
functionality. Our site is www.linuximq.net
Complicated or not, clean or not, its being working in some interresting
scenarios with lots of load on it. I feel fine for being able to help
the community somehow with it. Found no time yet to check Jamal's new
patches but we would use dummy as the base for "real device"
functionality development.
At least its nice to find we are discussing how to do it, not anymore if
IMQ functionality is needed, cause it really is.
Going one way or another we should not let users alone again with nobody
taking care of this like it happened before. I plan keeping IMQ updated
with new kernel versions as usual.
Jamal, when you say "to replace" you mean it may get into vanila kernel?
Do you plan keeping it updated from now on?
Either way, can we call this new thing something else, because actual
users may not want to migrate, so both should work together. A user
should be able to patch a kernel with both.
We (at linuximq.net) would be more then happy to help with it.
Andre
Jamal Hadi Salim wrote:
quoted hunk
This is in relation to providing functionality that IMQ was intending
to using the dummy device and tc actions. Ive copied as many people as i
could dig who i know may have interest in this.
Please forward this to any other list which may have interest
in the subject. It still needs some cleaning up; however, i dont wanna
sit on it for another year - and now that mirred is out there, this is a
good time.
Advantage over current IMQ; cleaner in particular in in SMP;
with a _lot_ less code.
Old Dummy device functionality is preserved while new one only
kicks in if you use actions. Didnt have to write a new device and finaly
made a real dumb device to be a little smarter ;->
IMQ USES
--------
As far as i know the reasons listed below is why people use IMQ.
It would be nice to know of anything else that i missed because this
is the requirements list i used.
1) qdiscs/policies that are per device as opposed to system wide.
IMQ allows for sharing across multiple devices.
2) Allows for queueing incoming traffic for shaping instead of
dropping. I am not aware of any study that shows policing is
worse than shaping in achieving the end goal of rate control.
I would be interested if anyone is experimenting. Nevertheless,
this is still an alternative as opposed to making a system wide
ingress change.
3) Very interesting use: if you are serving p2p you may wanna give
preference to your own localy originated traffic (when responses come
back) vs someone using your system to do bittorent. So QoSing based on
state comes in as the solution. What people did to achive this was stick
the IMQ somewhere prelocal hook.
I think this is a pretty neat feature to have in Linux in general.
(i.e not just for IMQ).
But i wont go back to putting netfilter hooks in the device to satisfy
this. I also dont think its worth it hacking dummy some more to be
aware of say L3 info and play ip rule tricks to achieve this.
--> Instead the plan is to have a contrack related action. This action
will selectively either query/create contrack state on incoming packets.
Packets could then be redirected to dummy based on what happens -> eg
on incoming packets; if we find they are of known state we could send to
a different queue than one which didnt have existing state. This
all however is dependent on whatever rules the admin enters.
What you can do with dummy currently with actions
--------------------------------------------------
Lets say you are policing packets from alias 192.168.200.200/32
you dont want those to exceed 100kbps going out.
tc filter add dev eth0 parent 1: protocol ip prio 10 u32 \
match ip src 192.168.200.200/32 flowid 1:2 \
action police rate 100kbit burst 90k drop
If you run tcpdump on eth0 you will see all packets going out
with src 192.168.200.200/32 dropped or not
Extend the rule a little to see only the ones that made it out:
tc filter add dev eth0 parent 1: protocol ip prio 10 u32 \
match ip src 192.168.200.200/32 flowid 1:2 \
action police rate 10kbit burst 90k drop \
action mirred egress mirror dev dummy0
Now fire tcpdump on dummy0 to see only those packets ..
tcpdump -n -i dummy0 -x -e -t
Essentially a good debugging/logging interface.
If you replace mirror with redirect, those packets will be
blackholed and will never make it out. This redirect behavior
changes with new patch (but not the mirror).
What you can do with dummy and attached patch
----------------------------------------------
Essentially provide functionality that most people use IMQ;
sample below:
--------
export TC="/sbin/tc"
$TC qdisc add dev dummy0 root handle 1: prio
$TC qdisc add dev dummy0 parent 1:1 handle 10: sfq
$TC qdisc add dev dummy0 parent 1:2 handle 20: tbf rate 20kbit buffer
1600 limit 3000
$TC qdisc add dev dummy0 parent 1:3 handle 30:
sfq
$TC filter add dev dummy0 protocol ip pref 1 parent 1: handle 1 fw
classid 1:1
$TC filter add dev dummy0 protocol ip pref 2 parent 1: handle 2 fw
classid 1:2
ifconfig dummy0 up
$TC qdisc add dev eth0 ingress
# redirect all IP packets arriving in eth0 to dummy0
# use mark 1 --> puts them onto class 1:1
$TC filter add dev eth0 parent ffff: protocol ip prio 10 u32 \
match u32 0 0 flowid 1:1 \
action ipt -j MARK --set-mark 1 \
action mirred egress redirect dev dummy0
--------
Run A Little test:
from another machine ping so that you have packets going into the box:
-----
[root@jzny action-tests]# ping 10.22
PING 10.22 (10.0.0.22): 56 data bytes
64 bytes from 10.0.0.22: icmp_seq=0 ttl=64 time=2.8 ms
64 bytes from 10.0.0.22: icmp_seq=1 ttl=64 time=0.6 ms
64 bytes from 10.0.0.22: icmp_seq=2 ttl=64 time=0.6 ms
--- 10.22 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.6/1.3/2.8 ms
[root@jzny action-tests]#
-----
Now look at some stats:
---
[root@jmandrake]:~# $TC -s filter show parent ffff: dev eth0
filter protocol ip pref 10 u32
filter protocol ip pref 10 u32 fh 800: ht divisor 1
filter protocol ip pref 10 u32 fh 800::800 order 2048 key ht 800 bkt 0
flowid 1:1
match 00000000/00000000 at 0
action order 1: tablename: mangle hook: NF_IP_PRE_ROUTING
target MARK set 0x1
index 1 ref 1 bind 1 installed 4195sec used 27sec
Sent 252 bytes 3 pkts (dropped 0, overlimits 0)
action order 2: mirred (Egress Redirect to device dummy0) stolen
index 1 ref 1 bind 1 installed 165 sec used 27 sec
Sent 252 bytes 3 pkts (dropped 0, overlimits 0)
[root@jmandrake]:~# $TC -s qdisc
qdisc sfq 30: dev dummy0 limit 128p quantum 1514b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
qdisc tbf 20: dev dummy0 rate 20Kbit burst 1575b lat 2147.5s
Sent 210 bytes 3 pkts (dropped 0, overlimits 0)
qdisc sfq 10: dev dummy0 limit 128p quantum 1514b
Sent 294 bytes 3 pkts (dropped 0, overlimits 0)
qdisc prio 1: dev dummy0 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1
1
Sent 504 bytes 6 pkts (dropped 0, overlimits 0)
qdisc ingress ffff: dev eth0 ----------------
Sent 308 bytes 5 pkts (dropped 0, overlimits 0)
[root@jmandrake]:~# ifconfig dummy0
dummy0 Link encap:Ethernet HWaddr 00:00:00:00:00:00
inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING NOARP MTU:1500 Metric:1
RX packets:6 errors:0 dropped:3 overruns:0 frame:0
TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:32
RX bytes:504 (504.0 b) TX bytes:252 (252.0 b)
-----
Dummy continues to behave like it always did.
You send it any packet not originating from the actions it will drop
them.
[In this case the three dropped packets were ipv6 ndisc].
My goal here is to start a discussion to see if people agree this is
a good replacement for IMQ or whether to go another path.
Clearly i would prefer to have this change in, but I am not religious
and would listen to reason about how it should be done as long as no
uneccessary clutter happens.
Patch attached.
cheers,
jamal
------------------------------------------------------------------------
@@ -35,11 +42,128 @@#include<linux/etherdevice.h>#include<linux/init.h>#include<linux/moduleparam.h>+#ifdef CONFIG_NET_CLS_ACT+#include<net/pkt_sched.h>+#endif++#define TX_TIMEOUT (2*HZ)++#define TX_Q_LIMIT 32+structdummy_private{+structnet_device_statsstats;+#ifdef CONFIG_NET_CLS_ACT+structtasklet_structdummy_tasklet;+inttasklet_pending;+/* mostly debug stats leave in for now */+unsignedlongstat_r1;+unsignedlongstat_r2;+unsignedlongstat_r3;+unsignedlongstat_r4;+unsignedlongstat_r5;+unsignedlongstat_r6;+unsignedlongstat_r7;+unsignedlongstat_r8;+structsk_buff_headrq;+structsk_buff_headtq;+#endif+};++#ifdef CONFIG_NET_CLS_ACT+staticvoidri_tasklet(unsignedlongdev);+#endif+staticintnumdummies=1;staticintdummy_xmit(structsk_buff*skb,structnet_device*dev);staticstructnet_device_stats*dummy_get_stats(structnet_device*dev);+staticvoiddummy_timeout(structnet_device*dev);+staticintdummy_open(structnet_device*dev);+staticintdummy_close(structnet_device*dev);++staticvoiddummy_timeout(structnet_device*dev){++intcpu=smp_processor_id();++dev->trans_start=jiffies;+printk("%s: BUG tx timeout on CPU %d\n",dev->name,cpu);+if(spin_is_locked((&dev->xmit_lock)))+printk("xmit lock grabbed already\n");+if(spin_is_locked((&dev->queue_lock)))+printk("queue lock grabbed already\n");+}++#ifdef CONFIG_NET_CLS_ACT+staticvoidri_tasklet(unsignedlongdev){++structnet_device*dv=(structnet_device*)dev;+structdummy_private*dp=((structnet_device*)dev)->priv;+structnet_device_stats*stats=&dp->stats;+structsk_buff*skb=NULL;++dp->stat_r4+=1;+if(NULL==(skb=skb_peek(&dp->tq))){+dp->stat_r5+=1;+if(spin_trylock(&dv->xmit_lock)){+dp->stat_r8+=1;+while(NULL!=(skb=skb_dequeue(&dp->rq))){+skb_queue_tail(&dp->tq,skb);+}+spin_unlock(&dv->xmit_lock);+}else{+/* reschedule */+dp->stat_r1+=1;+gotoresched;+}+}++while(NULL!=(skb=skb_dequeue(&dp->tq))){+__u32from=G_TC_FROM(skb->tc_verd);++skb->tc_verd=0;+skb->tc_verd=SET_TC_NCLS(skb->tc_verd);+stats->tx_packets++;+stats->tx_bytes+=skb->len;+if(from&AT_EGRESS){+dp->stat_r6+=1;+dev_queue_xmit(skb);+}elseif(from&AT_INGRESS){++dp->stat_r7+=1;+netif_rx(skb);+}else{+/* if netfilt is compiled in and packet is+tagged,wecouldreinjectthepacketback+thiswouldmakeitdoremaining10%+ofwhatcurrentIMQdoes+ifsomeonereallyreallyinsiststhen+thisisthespot..jhs*/+dev_kfree_skb(skb);+stats->tx_dropped++;+}+}++if(spin_trylock(&dv->xmit_lock)){+dp->stat_r3+=1;+if(NULL==(skb=skb_peek(&dp->rq))){+dp->tasklet_pending=0;+if(netif_queue_stopped(dv))+//netif_start_queue(dv);+netif_wake_queue(dv);+}else{+dp->stat_r2+=1;+spin_unlock(&dv->xmit_lock);+gotoresched;+}+spin_unlock(&dv->xmit_lock);+}else{+resched:+dp->tasklet_pending=1;+tasklet_schedule(&dp->dummy_tasklet);+}++}+#endifstaticintdummy_set_address(structnet_device*dev,void*p){
@@ -62,12 +186,17 @@/* Initialize the device structure. */dev->get_stats=dummy_get_stats;dev->hard_start_xmit=dummy_xmit;+dev->tx_timeout=&dummy_timeout;+dev->watchdog_timeo=TX_TIMEOUT;+dev->open=&dummy_open;+dev->stop=&dummy_close;+dev->set_multicast_list=set_multicast_list;dev->set_mac_address=dummy_set_address;/* Fill in device structure with ethernet-generic values. */ether_setup(dev);-dev->tx_queue_len=0;+dev->tx_queue_len=TX_Q_LIMIT;dev->change_mtu=NULL;dev->flags|=IFF_NOARP;dev->flags&=~IFF_MULTICAST;
@@ -97,12 +272,41 @@module_param(numdummies,int,0);MODULE_PARM_DESC(numdummies,"Number of dummy pseudo devices");+staticintdummy_close(structnet_device*dev)+{++#ifdef CONFIG_NET_CLS_ACT+structdummy_private*dp=((structnet_device*)dev)->priv;++tasklet_kill(&dp->dummy_tasklet);+skb_queue_purge(&dp->rq);+skb_queue_purge(&dp->tq);+#endif+netif_stop_queue(dev);+return0;+}++staticintdummy_open(structnet_device*dev)+{++#ifdef CONFIG_NET_CLS_ACT+structdummy_private*dp=((structnet_device*)dev)->priv;++tasklet_init(&dp->dummy_tasklet,ri_tasklet,(unsignedlong)dev);+skb_queue_head_init(&dp->rq);+skb_queue_head_init(&dp->tq);+#endif+netif_start_queue(dev);+return0;+}++staticint__initdummy_init_one(intindex){structnet_device*dev_dummy;interr;-dev_dummy=alloc_netdev(sizeof(structnet_device_stats),+dev_dummy=alloc_netdev(sizeof(structdummy_private),"dummy%d",dummy_setup);if(!dev_dummy)
-----------------------------------------------------------------------
Confidentiality Notice: This e-mail communication and any attachments
may contain confidential and privileged information for the use of the
designated recipients named above. If you are not the intended
recipient, you are hereby notified that you have received this
communication in error and that any review, disclosure, dissemination,
distribution or copying of it or its contents is prohibited. If you have
received this communication in error, please notify me immediately by
replying to this message and deleting it from your computer.
Thank you.
From: Jamal Hadi Salim <hidden> Date: 2005-01-31 16:51:44
On Mon, 2005-01-31 at 11:27, Andre Correa wrote:
Hi all,
it turned an year since we (me and some cool folks) got the original IMQ
from "death". During this year we updated kernel and iptables patches
for every available version, created some new features (like hooking
after and before NAT, multiple IMQ devices, solved modules problems,
etc), and helped lots of users in our mailling list. The wish list grew,
we created a site/FAQ/WiKi. We are still missing "dumb device"
functionality. Our site is www.linuximq.net
nice. Since you are deeply involved i think you can help put closure to
this.
Complicated or not, clean or not, its being working in some interresting
scenarios with lots of load on it. I feel fine for being able to help
the community somehow with it. Found no time yet to check Jamal's new
patches but we would use dummy as the base for "real device"
functionality development.
At least its nice to find we are discussing how to do it, not anymore if
IMQ functionality is needed, cause it really is.
The people have spoken i suppose is the right way to describe it.
If a lot of people use it, then its existence is justified. The problem
has been misrepresenation of why its needed.
Going one way or another we should not let users alone again with nobody
taking care of this like it happened before. I plan keeping IMQ updated
with new kernel versions as usual.
Jamal, when you say "to replace" you mean it may get into vanila kernel?
Do you plan keeping it updated from now on?
The plan is to make that small update to the kernel to achieve the
funtionality that IMQ provides. It doesnt have to be me who does the
updating thereafter; you can own this for example. The goal is to meet
those requirements with little noise in the kernel. If we have things in
the kernel, then there should be no need to maintain separate patches.
Either way, can we call this new thing something else, because actual
users may not want to migrate, so both should work together. A user
should be able to patch a kernel with both.
I dont have an issue with renaming but i dont see any overwhelming
reason to do it on a new device when dummy seems to be sufficient.
Take a look at that patch and see what functionality is missing.
Forget about the iptable hooks. See the thread of discussion
to see how the plan to meet those requirements looks like - see if
something is missing. Please read the text i posted - it is verbose but
would give a good explanation.
We (at linuximq.net) would be more then happy to help with it.
Like i said you guys can own this - just wanna reduce cruft in the
kernel.
cheers,
jamal
From: Andy Furniss <hidden> Date: 2005-01-31 22:39:42
Jamal Hadi Salim wrote:
This is in relation to providing functionality that IMQ was intending
to using the dummy device and tc actions. Ive copied as many people as i
could dig who i know may have interest in this.
Please forward this to any other list which may have interest
in the subject. It still needs some cleaning up; however, i dont wanna
sit on it for another year - and now that mirred is out there, this is a
good time.
Advantage over current IMQ; cleaner in particular in in SMP;
with a _lot_ less code.
Old Dummy device functionality is preserved while new one only
kicks in if you use actions. Didnt have to write a new device and finaly
made a real dumb device to be a little smarter ;->
IMQ USES
--------
As far as i know the reasons listed below is why people use IMQ.
It would be nice to know of anything else that i missed because this
is the requirements list i used.
1) qdiscs/policies that are per device as opposed to system wide.
IMQ allows for sharing across multiple devices.
2) Allows for queueing incoming traffic for shaping instead of
dropping. I am not aware of any study that shows policing is
worse than shaping in achieving the end goal of rate control.
I would say the end goal is shaping not just rate control. Shaping
meaning different things to different people and ingress shaping being
different from egress.
For me it's from the wrong end of a relativly narrow (512kbit)
bottleneck link that has a 600ms fifo at the other end. My aim to
sacrifice as little bandwidth as possible while not adding latency
bursts for gaming and per user bandwidth allocation (with sharing of
unused) and sfq within that for bulk tcp traffic.
If I was shaping LAN traffic, then policers/drops would be OK for me -
but for a slow link I think queueing and dropping are better/give more
control eg. I get to use sfq which should not drop the one packet a 56k
user has managed to send me in the face of lots of incoming from low
latency high bandwidth servers.
Even if it's possible I bet few can easily get policers to setup the
complex sharing/prioritisations that you can with HTB or HFSC.
I would be interested if anyone is experimenting. Nevertheless,
this is still an alternative as opposed to making a system wide
ingress change.
3) Very interesting use: if you are serving p2p you may wanna give
preference to your own localy originated traffic (when responses come
back) vs someone using your system to do bittorent. So QoSing based on
state comes in as the solution. What people did to achive this was stick
the IMQ somewhere prelocal hook.
I think this is a pretty neat feature to have in Linux in general.
(i.e not just for IMQ).
I think flexibility is always good - tunnels, ipsec etc. may need it - I
don't know from personal use, though.
But i wont go back to putting netfilter hooks in the device to satisfy
this. I also dont think its worth it hacking dummy some more to be
aware of say L3 info and play ip rule tricks to achieve this.
--> Instead the plan is to have a contrack related action. This action
will selectively either query/create contrack state on incoming packets.
I don't understand exactly what you mean here - for my setup to work I
need to see denatted addresses and mark (connbytes - it helps me be
extra nasty to multiple simoultaneous connections in slowstart and
prioritise browsing over bulk) in prerouting mangle. Of course if I can
use netfilter to classify and save into contrack then I could do
evrything in netfilter and then use something like connmark to save it
per connection.
Packets could then be redirected to dummy based on what happens -> eg
on incoming packets; if we find they are of known state we could send to
a different queue than one which didnt have existing state. This
all however is dependent on whatever rules the admin enters.
How does the admin enter the rules - netfilter or other?
Andy.
2) Allows for queueing incoming traffic for shaping instead of
dropping. I am not aware of any study that shows policing is
worse than shaping in achieving the end goal of rate control.
I would say the end goal is shaping not just rate control. Shaping
meaning different things to different people and ingress shaping being
different from egress.
I know for a while the Bart howto was mislabeling the meaning of
policing - not sure about shaping.
Shaping has a precise definition that involves a queue and a
non-working-conserving scheduler in order to rate control. This doesnt
matter where it happens (egress or ingress). Policing on the other hand
is work conserving etc.
For me it's from the wrong end of a relativly narrow (512kbit)
bottleneck link that has a 600ms fifo at the other end. My aim to
sacrifice as little bandwidth as possible while not adding latency
bursts for gaming and per user bandwidth allocation (with sharing of
unused) and sfq within that for bulk tcp traffic.
If I was shaping LAN traffic, then policers/drops would be OK for me -
but for a slow link I think queueing and dropping are better/give more
control eg. I get to use sfq which should not drop the one packet a 56k
user has managed to send me in the face of lots of incoming from low
latency high bandwidth servers.
Even if it's possible I bet few can easily get policers to setup the
complex sharing/prioritisations that you can with HTB or HFSC.
sfq has a built in classifier that can efficiently separate those
flows so you can achieve semi-fairness; so its not the shaping perse
that helps, rather you ended up using a clever scheme that can isolate
flows and benefited from shaping as a result; the hashing function
should prove weak when a lot of flows start showing up.
You could write some interesting classifier (as an example steal the one
that sfq has) and achieve the same end results with policing. This would
be easier now with ematches ..
quoted
But i wont go back to putting netfilter hooks in the device to satisfy
this. I also dont think its worth it hacking dummy some more to be
aware of say L3 info and play ip rule tricks to achieve this.
--> Instead the plan is to have a contrack related action. This action
will selectively either query/create contrack state on incoming packets.
I don't understand exactly what you mean here - for my setup to work I
need to see denatted addresses and mark (connbytes - it helps me be
extra nasty to multiple simoultaneous connections in slowstart and
prioritise browsing over bulk) in prerouting mangle. Of course if I can
use netfilter to classify and save into contrack then I could do
evrything in netfilter and then use something like connmark to save it
per connection.
You may be refering to requirement #3 then?
In other words what you are doing is best served by knowing the state?
Are pre/post routing sufficient as netfilter hooks for your case?
quoted
Packets could then be redirected to dummy based on what happens -> eg
on incoming packets; if we find they are of known state we could send to
a different queue than one which didnt have existing state. This
all however is dependent on whatever rules the admin enters.
How does the admin enter the rules - netfilter or other?
Just like i showed in that post (It was long - so dont wanna cutnpaste
here).
cheers,
jamal
From: Andy Furniss <hidden> Date: 2005-02-01 14:53:29
jamal wrote:
On Mon, 2005-01-31 at 17:39, Andy Furniss wrote:
quoted
Jamal Hadi Salim wrote:
quoted
quoted
2) Allows for queueing incoming traffic for shaping instead of
dropping. I am not aware of any study that shows policing is
worse than shaping in achieving the end goal of rate control.
I would say the end goal is shaping not just rate control. Shaping
meaning different things to different people and ingress shaping being
different from egress.
I know for a while the Bart howto was mislabeling the meaning of
policing - not sure about shaping.
Shaping has a precise definition that involves a queue and a
non-working-conserving scheduler in order to rate control. This doesnt
matter where it happens (egress or ingress). Policing on the other hand
is work conserving etc.
Ok, but shaping to LARTC posters means being able to classify traffic
and set up sharing/priorotising of classes. This is the reason most need
to be able to queue - they want to use htb/hfsc for complicated setups
and until now were not aware that it was even possible to replicate this
in policers and if it becomes feasable it will probably appear daunting
to do compared with HTB and all the existing docs/scripts.
For me, I think queuing and dropping is better than just dropping, you
can affect tcp by delaying eg. 1 ack per packet rather than delayed acks
and clocking out the packets helps smooth burstiness, which hurts
latency if you are doing traffic control from the wrong end of the
bottleneck.
quoted
For me it's from the wrong end of a relativly narrow (512kbit)
bottleneck link that has a 600ms fifo at the other end. My aim to
sacrifice as little bandwidth as possible while not adding latency
bursts for gaming and per user bandwidth allocation (with sharing of
unused) and sfq within that for bulk tcp traffic.
If I was shaping LAN traffic, then policers/drops would be OK for me -
but for a slow link I think queueing and dropping are better/give more
control eg. I get to use sfq which should not drop the one packet a 56k
user has managed to send me in the face of lots of incoming from low
latency high bandwidth servers.
Even if it's possible I bet few can easily get policers to setup the
complex sharing/prioritisations that you can with HTB or HFSC.
sfq has a built in classifier that can efficiently separate those
flows so you can achieve semi-fairness; so its not the shaping perse
that helps, rather you ended up using a clever scheme that can isolate
flows and benefited from shaping as a result; the hashing function
should prove weak when a lot of flows start showing up.
You could write some interesting classifier (as an example steal the one
that sfq has) and achieve the same end results with policing. This would
be easier now with ematches ..
The idea of loosing the s from sfq and doing multilevel hash/mapping is
attractive - of course I would want to queue each flow and have the
queue be variable length for each flow depending on occupancy of other
flows. I suppose a per flow intelligent dropping scheme would be even
better. It would be nice to be able to set/control queuelength for link
bandwidth, nothing classful in linux tc does this.
quoted
quoted
But i wont go back to putting netfilter hooks in the device to satisfy
this. I also dont think its worth it hacking dummy some more to be
aware of say L3 info and play ip rule tricks to achieve this.
--> Instead the plan is to have a contrack related action. This action
will selectively either query/create contrack state on incoming packets.
I don't understand exactly what you mean here - for my setup to work I
need to see denatted addresses and mark (connbytes - it helps me be
extra nasty to multiple simoultaneous connections in slowstart and
prioritise browsing over bulk) in prerouting mangle. Of course if I can
use netfilter to classify and save into contrack then I could do
evrything in netfilter and then use something like connmark to save it
per connection.
You may be refering to requirement #3 then?
In other words what you are doing is best served by knowing the state?
As long as I could use netfilter to mark/classify connections then I
think I can sort my setup, don't know about others.
Are pre/post routing sufficient as netfilter hooks for your case?
Yes but depends on where in pre/postrouting. For me after/before nat, as
I say above though I could workaround if I classify connections with
netfilter. For others as long as they can filter on a mark/classify set
in forward, then I think it will be OK for them.
quoted
quoted
Packets could then be redirected to dummy based on what happens -> eg
on incoming packets; if we find they are of known state we could send to
a different queue than one which didnt have existing state. This
all however is dependent on whatever rules the admin enters.
How does the admin enter the rules - netfilter or other?
Just like i showed in that post (It was long - so dont wanna cutnpaste
here).
I am not sure what exactly can can't be done in your example:
># redirect all IP packets arriving in eth0 to dummy0
># use mark 1 --> puts them onto class 1:1
>$TC filter add dev eth0 parent ffff: protocol ip prio 10 u32 \
>match u32 0 0 flowid 1:1 \
What I can do here depends where it hooks packets.
>action ipt -j MARK --set-mark 1 \
I don't know what table I am using - may be OK as long as I can test for
a mark I made earlier in the egress dummy case or test connmark/state I
set for that connection in the ingress case.
>action mirred egress redirect dev dummy0
Andy.
I know for a while the Bart howto was mislabeling the meaning of
policing - not sure about shaping.
Shaping has a precise definition that involves a queue and a
non-working-conserving scheduler in order to rate control. This doesnt
matter where it happens (egress or ingress). Policing on the other hand
is work conserving etc.
Ok, but shaping to LARTC posters means being able to classify traffic
and set up sharing/priorotising of classes.
This is the reason most need
to be able to queue - they want to use htb/hfsc for complicated setups
Close - but you are missing the rate control requirement.
You can do the above with prio qdisc for example but that does not
equate to shaping. Understood about Lartc users definitions. However,
note that they are influenced by what people tell them or what people
write in docs. Help in making sure the redefinition doesnt propagate.
Theres a very precise meaning to shaping and it is _exactly_ the way i
described it above. Clue people who ask questions.
and until now were not aware that it was even possible to replicate this
in policers
I am sure i have written about 50 emails on this topic over the last 5
years;-> look at the archives. I even joked about it here:
http://www.cyberus.ca/~hadi/patches/action/README over 2 years ago.
look at the text reading "it must be the summer heat again; weve had
someone doing that every year around summer"
Unfortunately people who are writting docs havent picked it up for
whatever reasons. I am hoping we finaly get this documented somewhere.
Can you help fix this?
and if it becomes feasable it will probably appear daunting
to do compared with HTB and all the existing docs/scripts.
From a usability perspective i agree with you.
Its still doable is all i can say ;-> (but you are correct in that it
may not be for the weak hearts)
As a reminder of some of the big discussions on shared and hierachical
policing - look at the many many discussions I had with devik on this
topic a few years back.
It resulted in the birth of HTB (which is essentially a hierachy of the
same token bucket meters used in policers; hierachical shared policers
are doable - look at iproute2/examples/diffserv). HTB otoh has proven
useful due to simplicty - so it stands on its own merit now.
I think there may be peculiar occasions where you may need to have
queues to shape traffic to a local app - but thats peculiar.
For me, I think queuing and dropping is better than just dropping, you
can affect tcp by delaying eg. 1 ack per packet rather than delayed acks
and clocking out the packets helps smooth burstiness,
True - but i question the usefulness of localy terminating TCP packets
being shaped. For packets being forwarded, the shaping happens on
egress.
which hurts
latency if you are doing traffic control from the wrong end of the
bottleneck.
Not sure i followed the latency connection.
As long as I could use netfilter to mark/classify connections then I
think I can sort my setup, don't know about others.
Great. yes, you can.
quoted
Are pre/post routing sufficient as netfilter hooks for your case?
Yes but depends on where in pre/postrouting. For me after/before nat, as
I say above though I could workaround if I classify connections with
netfilter. For others as long as they can filter on a mark/classify set
in forward, then I think it will be OK for them.
You can mark in netfilter or even in tc and use those marks in both
places.
I am not sure what exactly can can't be done in your example:
># redirect all IP packets arriving in eth0 to dummy0
># use mark 1 --> puts them onto class 1:1
>$TC filter add dev eth0 parent ffff: protocol ip prio 10 u32 \
>match u32 0 0 flowid 1:1 \
What I can do here depends where it hooks packets.
>action ipt -j MARK --set-mark 1 \
I don't know what table I am using - may be OK as long as I can test for
a mark I made earlier in the egress dummy case or test connmark/state I
set for that connection in the ingress case.
That would be doable. Thanks for taking the time Andy.
cheers,
jamal
From: Andy Furniss <hidden> Date: 2005-02-04 00:33:54
jamal wrote:
On Tue, 2005-02-01 at 09:53, Andy Furniss wrote:
quoted
jamal wrote:
quoted
I know for a while the Bart howto was mislabeling the meaning of
policing - not sure about shaping.
Shaping has a precise definition that involves a queue and a
non-working-conserving scheduler in order to rate control. This doesnt
matter where it happens (egress or ingress). Policing on the other hand
is work conserving etc.
Ok, but shaping to LARTC posters means being able to classify traffic
and set up sharing/priorotising of classes.
This is the reason most need
to be able to queue - they want to use htb/hfsc for complicated setups
Close - but you are missing the rate control requirement.
You can do the above with prio qdisc for example but that does not
equate to shaping. Understood about Lartc users definitions. However,
note that they are influenced by what people tell them or what people
write in docs. Help in making sure the redefinition doesnt propagate.
Theres a very precise meaning to shaping and it is _exactly_ the way i
described it above. Clue people who ask questions.
I see your point.
quoted
and until now were not aware that it was even possible to replicate this
in policers
I am sure i have written about 50 emails on this topic over the last 5
years;-> look at the archives. I even joked about it here:
http://www.cyberus.ca/~hadi/patches/action/README over 2 years ago.
look at the text reading "it must be the summer heat again; weve had
someone doing that every year around summer"
Unfortunately people who are writting docs havent picked it up for
whatever reasons. I am hoping we finaly get this documented somewhere.
Can you help fix this?
I could write up some what I did type stuff. Once I work out what to do
and how to do it :-)
quoted
and if it becomes feasable it will probably appear daunting
to do compared with HTB and all the existing docs/scripts.
quoted
From a usability perspective i agree with you.
Its still doable is all i can say ;-> (but you are correct in that it
may not be for the weak hearts)
As a reminder of some of the big discussions on shared and hierachical
policing - look at the many many discussions I had with devik on this
topic a few years back.
It resulted in the birth of HTB (which is essentially a hierachy of the
same token bucket meters used in policers; hierachical shared policers
are doable - look at iproute2/examples/diffserv). HTB otoh has proven
useful due to simplicty - so it stands on its own merit now.
I think there may be peculiar occasions where you may need to have
queues to shape traffic to a local app - but thats peculiar.
I'll have to read up abit.
quoted
For me, I think queuing and dropping is better than just dropping, you
can affect tcp by delaying eg. 1 ack per packet rather than delayed acks
and clocking out the packets helps smooth burstiness,
True - but i question the usefulness of localy terminating TCP packets
being shaped. For packets being forwarded, the shaping happens on
egress.
I know it's a bit odd, but then if I had just one PC I would want to
rather than police for reasons below.
quoted
which hurts
latency if you are doing traffic control from the wrong end of the
bottleneck.
Not sure i followed the latency connection.
I am shaping a relativly slow link from the wrong end. My objective is
to avoid the 600ms buffer at ISP/Teleco getting filled as it adds
latency for my interactive traffic. If I have a dozen bulk tcp
connections running then policing encourages each to send data in bursts
at link speed, because delayed acks will pair packets and say a group of
four passes without dropping it causes another group of four from that
connection at link speed. Add to that the different or variable rtts of
the 12 connections it means that there will be times when large bunches
of big packets arrive together and delay the interactive traffic.
If I shape and dequeue each connection round robin and the aggeragate
rate is below link speed then the aggregate flow is smoothed better. If
the rates are low enough I will delay longer than delayed ack timers and
get one packet per ack aswell. It's still not perfect of course.
quoted
As long as I could use netfilter to mark/classify connections then I
think I can sort my setup, don't know about others.
Great. yes, you can.
quoted
quoted
Are pre/post routing sufficient as netfilter hooks for your case?
Yes but depends on where in pre/postrouting. For me after/before nat, as
I say above though I could workaround if I classify connections with
netfilter. For others as long as they can filter on a mark/classify set
in forward, then I think it will be OK for them.
You can mark in netfilter or even in tc and use those marks in both
places.
Great.
quoted
I am not sure what exactly can can't be done in your example:
quoted
# redirect all IP packets arriving in eth0 to dummy0
# use mark 1 --> puts them onto class 1:1
$TC filter add dev eth0 parent ffff: protocol ip prio 10 u32 \
match u32 0 0 flowid 1:1 \
What I can do here depends where it hooks packets.
quoted
action ipt -j MARK --set-mark 1 \
I don't know what table I am using - may be OK as long as I can test for
a mark I made earlier in the egress dummy case or test connmark/state I
set for that connection in the ingress case.
That would be doable. Thanks for taking the time Andy.