Hi
I am using kernel mode pppoe, and while using mirred on ppp interfaces (to
shape users upstream) noticed very strange behaviour (and IMHO wrong)
$2 - ppp interface
tc filter add dev $2 parent ffff: protocol ip prio 10 u32 \
match u32 0 0 flowid 1:1 \
action mirred egress redirect dev ifb0"
If i do tcpdump on ppp interface i can see normal ip traffic
PPPoE_146 ~ # tcpdump -ni ppp6 -c 100
tcpdump: WARNING:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ppp6, link-type LINUX_SLL (Linux cooked), capture size 96 bytes
11:36:16.541742 IP 172.16.131.221.2182 > 64.4.50.51.443: Flags [F.], seq
2636760792, ack 2812561777, win 64963, length 0
If i do tcpdump on ifb0 i will see PPPoE incapsulated traffic!
PPPoE_146 ~ # tcpdump -ni ifb0 -c 100
tcpdump: WARNING: ifb0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ifb0, link-type EN10MB (Ethernet), capture size 96 bytes
11:36:58.949727 PPPoE [ses 0x15] IP 172.16.98.22.2081 > 94.75.218.20.80:
Flags [S], seq 3205622298, win 65535, options [mss 1440,nop,nop,sackOK],
length 0
11:36:58.531473 PPPoE [ses 0xd] IP 172.16.98.14.16526 > 128.16.130.164.445:
Flags [S], seq 98566173, win 65535, options [mss 1440,nop,nop,sackOK], length
0
Is it expected that redirecting ppp interface, that supposed to be clean IP
traffic is becoming eth encapsulated traffic?
Probably some bad interaction with kernel mode pppoe?
On Mon, 2009-10-12 at 11:43 +0300, Denys Fedoryschenko wrote:
Is it expected that redirecting ppp interface, that supposed to be clean IP
traffic is becoming eth encapsulated traffic?
No. Imagine if there were other types of packets non-ip for example,
what do you do then?
this feature is as close as you can get when you do switch level
mirroring or redirection. If you want to edit header before redirect
etc, use pedit (refer to recent discussion with someone who wanted to
replicate packets for redundant routing purposes);
cheers,
jamal
How it can be ethernet packets (with PPPoE headers) if i am redirecting from
ppp interface, not ethernet interface?
PPPoE_146 ~ # tcpdump -ni ppp0 -e -vvv -s 1500 -c 4
tcpdump: WARNING:
tcpdump: listening on ppp0, link-type LINUX_SLL (Linux cooked), capture size
1500 bytes
17:03:17.015598 Out ethertype IPv4 (0x0800), length 68: (tos 0x0, ttl 111, id
45623, offset 0, flags [DF], proto TCP (6), length 52)
68.231.189.241.24800 > 172.16.131.199.1060: Flags [.], cksum 0xd208
(correct), seq 783840165, ack 1980178761, win 32748, options [nop,nop,sack 1
{1441:8741}], length 0
PPPoE_146 ~ # tcpdump -ni ifb0 -e -vvv -s 1500 -c 4
tcpdump: WARNING:
tcpdump: WARNING: ifb0: no IPv4 address assigned
tcpdump: listening on ifb0, link-type EN10MB (Ethernet), capture size 1500
bytes
17:04:01.625547 00:0a:cd:17:5e:08 > 00:0a:cd:14:6b:67, ethertype PPPoE S
(0x8864), length 70: PPPoE [ses 0xb] IP (0x0021), length 50: (tos 0x0, ttl
128, id 51706, offset 0, flags [DF], proto TCP (6), length 48)
Btw i test, if i use userspace pppoe (synchronous) i will get different output
in tcpdump
PPPoE_146 ~ # tcpdump -ni ifb0 -vvvv
tcpdump: WARNING:
tcpdump: WARNING: ifb0: no IPv4 address assigned
tcpdump: listening on ifb0, link-type EN10MB (Ethernet), capture size 96 bytes
17:15:41.512553 00:28:00:d6:40:00 > ff:03:00:21:45:00, ethertype Unknown
(0x8006), length 44:
0x0000: df49 ac10 8320 cf2e 1c51 0416 0050 ce7b .I.......Q...P.{
0x0010: e177 1caf 491d 5010 ffff 7afe 0000 .w..I.P...z...
17:15:41.520802 00:34:00:d7:40:00 > ff:03:00:21:45:00, ethertype Unknown
(0x8006), length 112:
0x0000: df3c ac10 8320 cf2e 1c51 0416 0050 ce7b .<.......Q...P.{
0x0010: e177 1caf 491d 8010 ffff 686e 0000 0101 .w..I.....hn....
0x0020: 050a 1caf 4ebd 1caf 545d ff03 0021 4500 ....N...T]...!E.
0x0030: 0034 00d8 4000 8006 df3b ac10 8320 cf2e .4..@....;......
0x0040: 1c51 0416 0050 ce7b e177 1caf 491d 8010 .Q...P.{.w..I...
0x0050: ffff ..
17:15:41.528825 00:34:00:d9:40:00 > ff:03:00:21:45:00, ethertype Unknown
(0x8006), length 56:
0x0000: df3a ac10 8320 cf2e 1c51 0416 0050 ce7b .:.......Q...P.{
0x0010: e177 1caf 491d 8010 ffff 5d2e 0000 0101 .w..I.....].....
0x0020: 050a 1caf 4ebd 1caf 5f9d ....N..._.
With PPTP it is as expected, IP packets (like in ppp interface tcpdump).
On Monday 12 October 2009 16:10:47 jamal wrote:
On Mon, 2009-10-12 at 11:43 +0300, Denys Fedoryschenko wrote:
quoted
Is it expected that redirecting ppp interface, that supposed to be clean
IP traffic is becoming eth encapsulated traffic?
No. Imagine if there were other types of packets non-ip for example,
what do you do then?
this feature is as close as you can get when you do switch level
mirroring or redirection. If you want to edit header before redirect
etc, use pedit (refer to recent discussion with someone who wanted to
replicate packets for redundant routing purposes);
cheers,
jamal
On Mon, 2009-10-12 at 17:15 +0300, Denys Fedoryschenko wrote:
How it can be ethernet packets (with PPPoE headers) if i am redirecting from
ppp interface, not ethernet interface?
You are running this on ingress. If you run on egress you will see
proper header.
_Most people_ want to see the whole original header for qos accounting
purposes etc. PPPOE claims to be ethernet, PPP does not. Hence
difference in observation.
We could add an override feature to say "i only want
see what was passed not wire packet".
If this is important to you let me know and i will try to make time to
add it (with hopefully not breaking old scripts); if you feel brave as
well i could guide you on how to add it.
cheers,
jamal
I don't have problem with existing behaviour, since i am using other way of
shaping, for my case using pktedit to assign priority to SKB and shaping by
it.
But generally problem is was told by one of russian developers who is working
on firmware for few models of broadband routers, he asked to help on ISP
forum, and if possible to explain this to someone who can give advice, and
maybe tell that probably there is a bug.
It is explained here (but it is russian language, i dont think worth spending
time and translating) http://forum.nag.ru/forum/index.php?showtopic=51760
His case a bit complicated, since that ISP who provide pppoe use some kind of
ugly, oversized packets, and as he mention they can be fragmented. I didn't
understand exactly how it can be possible and how it can break u32 filter.
Here is one more guy who asked to help about that
http://mailman.ds9a.nl/pipermail/lartc/2007q3/021561.html
His issue that he wants to shape by ip, not by interface, and while it is
ethernet traffic in ifb, it is not trivial. I told him just to use offset in
u32,but he said there is case with possible packet fragmentation (not sure it
can happen) and shaper won't match.
I am just asking to take a look, if there is any bug.
If it is not, then just simple question, it will work reliably if i just use
u32 filter with offset on ifb?
On Monday 12 October 2009 23:57:44 jamal wrote:
On Mon, 2009-10-12 at 17:15 +0300, Denys Fedoryschenko wrote:
quoted
How it can be ethernet packets (with PPPoE headers) if i am redirecting
from ppp interface, not ethernet interface?
You are running this on ingress. If you run on egress you will see
proper header.
_Most people_ want to see the whole original header for qos accounting
purposes etc. PPPOE claims to be ethernet, PPP does not. Hence
difference in observation.
We could add an override feature to say "i only want
see what was passed not wire packet".
If this is important to you let me know and i will try to make time to
add it (with hopefully not breaking old scripts); if you feel brave as
well i could guide you on how to add it.
On Tue, 2009-10-13 at 00:54 +0300, Denys Fedoryschenko wrote:
I don't have problem with existing behaviour, since i am using other way of
shaping, for my case using pktedit to assign priority to SKB and shaping by
it.
I am dissapointed Denys, you dont like ipt?;->
But generally problem is was told by one of russian developers who is working
on firmware for few models of broadband routers, he asked to help on ISP
forum, and if possible to explain this to someone who can give advice, and
maybe tell that probably there is a bug.
[..]
If it is not, then just simple question, it will work reliably if i just use
u32 filter with offset on ifb?
Yes, of course you can if you add offset sizeof pppoe header.
But:
It looks like there is a genuine need for this feature.
The challenge is: I am trying to be generic across devices of many
different types (ethernet, atm, virtual etc) at many entry points,
ingress, egress local, forwarding etc.
This feature that this person would need will only work if you _know_
what you are doing; i.e in this case, I can very easily turn it off with
a simple command - but the user must know that they do this on ingress
side. I can cook a very quick patch for kernel and user space - you
think this user can test it?
cheers,
jamal
On Tue, 2009-10-13 at 00:54 +0300, Denys Fedoryschenko wrote:
quoted
I don't have problem with existing behaviour, since i am using other way
of shaping, for my case using pktedit to assign priority to SKB and
shaping by it.
I am dissapointed Denys, you dont like ipt?;->
It kills me :-) Each new version it doesn't work and i notice, i'm almost one
who use it :-) Probably i should wait till netfilter API and iptables
conversion will stabilize somehow.
Plus skbedit in some cases will be faster, if i eliminate iptables, unloading
modules even, basic filtering can be done by iproute2 too, i won't have
netfilter locks that make things slow on SMP (at least what i heard here and
what oprofile shows, that MARK was small CPU hog to compare with skbedit).
I am happily running 2k pppoe users on Quad Core CPU/on supercheap r8169
(better nic not available here) with skbedit and flow classifier. It can do
more even, i think.
After switching to skbedit things improve a lot (before 1k users was near max)
quoted
But generally problem is was told by one of russian developers who is
working on firmware for few models of broadband routers, he asked to help
on ISP forum, and if possible to explain this to someone who can give
advice, and maybe tell that probably there is a bug.
[..]
quoted
If it is not, then just simple question, it will work reliably if i just
use u32 filter with offset on ifb?
Yes, of course you can if you add offset sizeof pppoe header.
But:
It looks like there is a genuine need for this feature.
The challenge is: I am trying to be generic across devices of many
different types (ethernet, atm, virtual etc) at many entry points,
ingress, egress local, forwarding etc.
This feature that this person would need will only work if you _know_
what you are doing; i.e in this case, I can very easily turn it off with
a simple command - but the user must know that they do this on ingress
side. I can cook a very quick patch for kernel and user space - you
think this user can test it?
I can test even, even if he won't.
As i understand, for pppoe case, he can just skip offset for ethernet and
pppoe header, and he can filter by ip, or not?
Current way is maybe better, cause someone who want to count everything with
ethernet and pppoe headers - can, and who want without - also can (by setting
offset , just a bit more difficult.
Like
/sbin/tc filter add dev eth1 protocol 0x8864 parent 2:0 prio 1 u32 \
match u32 0x$IPREMOTE_HEX 0xffffffff at 24 flowid 2:$ID
(found in LARTC)
On Tue, 2009-10-13 at 01:44 +0300, Denys Fedoryschenko wrote:
It kills me :-) Each new version it doesn't work and i notice, i'm almost one
who use it :-) Probably i should wait till netfilter API and iptables
conversion will stabilize somehow.
I am a little frustrated - but yeah, waiting may help. I tend to do a
lot of private support to fix integration with iptables problems, so you
are not the only user ;->. I wish one distro gets it right.
The most hopeful seems to be debian.
Plus skbedit in some cases will be faster, if i eliminate iptables, unloading
modules even, basic filtering can be done by iproute2 too, i won't have
netfilter locks that make things slow on SMP (at least what i heard here and
what oprofile shows, that MARK was small CPU hog to compare with skbedit).
Makes sense.
I am happily running 2k pppoe users on Quad Core CPU/on supercheap r8169
(better nic not available here) with skbedit and flow classifier. It can do
more even, i think.
I bet pppd in user space is probably your biggest problem in terms of
performance.
After switching to skbedit things improve a lot (before 1k users was near max)
Not using netfilter will improve your numbers. So can skbedit do fwmark
as well?
I can test even, even if he won't.
As i understand, for pppoe case, he can just skip offset for ethernet and
pppoe header, and he can filter by ip, or not?
Current way is maybe better, cause someone who want to count everything with
ethernet and pppoe headers - can, and who want without - also can (by setting
offset , just a bit more difficult.
Like
/sbin/tc filter add dev eth1 protocol 0x8864 parent 2:0 prio 1 u32 \
match u32 0x$IPREMOTE_HEX 0xffffffff at 24 flowid 2:$ID
(found in LARTC)
yes, something like that.
It may be easier to tcpdump -x on both pppoe and ifb and see how the
packets look like at what offset. If that doesnt work well, I will work
on a patch...
cheers,
jamal
I bet pppd in user space is probably your biggest problem in terms of
performance.
No. :-)
First problem, on restart and massive users login i am getting some weird
locking problem with modprobe,probably related to busybox/mdev.
When interface come up, maybe it is problem of mdev, it is trying to do
modprobe with ip address as attribute. Probably they must not modprobe on
virtual interface. It is not easy to track who is reason of such call.
As result:
[ 174.564503] request_module: runaway loop modprobe 172.16.3.1
[ 174.564702] request_module: runaway loop modprobe 172.16.3.1
[ 174.801355] request_module: runaway loop modprobe 172.16.106.1
[ 174.801487] request_module: runaway loop modprobe 172.16.106.1
[ 175.011415] request_module: runaway loop modprobe 172.16.106.1
load average (even it doesn't mean anything in terms of CPU load) jumps to
72-80.
Another bottleneck is u32 (i can optimize but) and some strange locks
appearing at top of perf, maybe same as logon case.
And yes, pppd also appearing, but seems just registering new sysctls (for new
interface?) takes a lot of resources.
I can post perf -a -f g if you are interested for "logging in" case and
regular operation.
quoted
After switching to skbedit things improve a lot (before 1k users was near
max)
Not using netfilter will improve your numbers. So can skbedit do fwmark
as well?
I dont need it, i am using skb->priority as a key for flow classifier.
It looks weird, a lot of obsolete code there, but it is very nice, i dont need
to touch almost anything on ifb, and i can predefine even on startup required
amount of classes and just change rate when required.
Here is key part of shaper code:
$TC filter add dev ifb0 protocol ip pref 32 parent 1: handle 1 flow map key
priority baseclass 1:64
Then i create classes for id's i need (at my case id related to ppp interface
number).
and when ppp interface come up also:
(lowid is related to ppp interface or user id)
filter add dev $2 parent ffff: protocol ip prio 10 u32 \
match u32 0 0 flowid 1:1 \
action skbedit priority 0x${lowid} pipe \
action mirred egress redirect dev ifb0
yes, something like that.
It may be easier to tcpdump -x on both pppoe and ifb and see how the
packets look like at what offset. If that doesnt work well, I will work
on a patch...
Well another way is just to use as you suggest - egress on output
interface(s).
On Tue, 2009-10-13 at 15:45 +0300, Denys Fedoryschenko wrote:
First problem, on restart and massive users login i am getting some weird
locking problem with modprobe,probably related to busybox/mdev.
When interface come up, maybe it is problem of mdev, it is trying to do
modprobe with ip address as attribute. Probably they must not modprobe on
virtual interface. It is not easy to track who is reason of such call.
As result:
[ 174.564503] request_module: runaway loop modprobe 172.16.3.1
[ 174.564702] request_module: runaway loop modprobe 172.16.3.1
[ 174.801355] request_module: runaway loop modprobe 172.16.106.1
[ 174.801487] request_module: runaway loop modprobe 172.16.106.1
[ 175.011415] request_module: runaway loop modprobe 172.16.106.1
load average (even it doesn't mean anything in terms of CPU load) jumps to
72-80.
Strange. But would probably make sense to modprobe the virtual interface
then assign ip address to it.
Another bottleneck is u32 (i can optimize but) and some strange locks
appearing at top of perf, maybe same as logon case.
like you say u32 can be optimized
And yes, pppd also appearing, but seems just registering new sysctls (for new
interface?) takes a lot of resources.
back in the days (when i was involved in pppoe) pppd was a big problem
because it was not rentrant, so to solve the problem you exec a new
process for each user..
Ben Lahaise did a lot of work in this area and i am pretty sure decided
not to use pppd.
I can post perf -a -f g if you are interested for "logging in" case and
regular operation.
Sure - but i may not be much use to you..
I have CCed Ben.
Here is key part of shaper code:
Ok - makes sense. I know some people using mark for ingress pseudo RPF.
And given instability of netfilter interface, I may just end up patching
skbedit.
Well another way is just to use as you suggest - egress on output
interface(s).
Sure; If it becomes an issue let me know.
cheers,
jamal
From: Benjamin LaHaise <hidden> Date: 2009-10-13 15:10:21
Hi folks,
On Tue, Oct 13, 2009 at 09:02:38AM -0400, jamal wrote:
On Tue, 2009-10-13 at 15:45 +0300, Denys Fedoryschenko wrote:
quoted
Another bottleneck is u32 (i can optimize but) and some strange locks
appearing at top of perf, maybe same as logon case.
like you say u32 can be optimized
quoted
And yes, pppd also appearing, but seems just registering new sysctls (for new
interface?) takes a lot of resources.
back in the days (when i was involved in pppoe) pppd was a big problem
because it was not rentrant, so to solve the problem you exec a new
process for each user..
Ben Lahaise did a lot of work in this area and i am pretty sure decided
not to use pppd.
Yes, I've been maintaining the ppp implementation (Babylon) initially
written by SpellCaster back in '98. My main focus of late has been
dealing with PPPoE and L2TP, and specifically with terminating a lot of
L2TP LNS traffic. Babylon itself scales fairly well to large numbers of
interfaces, as does the network stack these days. Currently the bottleneck
in interface creation is in the sysctl code.
The other big problem with large numbers of interfaces is interface
deletion. The synchronize_rcu() call in the interface teardown ends up
making a single daemon with thousands of interfaces take a long time
to clean up after itself.
-ben
Ok - makes sense. I know some people using mark for ingress pseudo RPF.
And given instability of netfilter interface, I may just end up patching
skbedit.
Ok, I just whipped a quick patch for skbedit to set the mark - and it
seems to work well after some basic testing. I will post.
cheers,
jamal
Denys Fedoryschenko wrote, On 10/13/2009 12:44 AM:
...
As i understand, for pppoe case, he can just skip offset for ethernet and
pppoe header, and he can filter by ip, or not?
Current way is maybe better, cause someone who want to count everything with
ethernet and pppoe headers - can, and who want without - also can (by setting
offset , just a bit more difficult.
Like
/sbin/tc filter add dev eth1 protocol 0x8864 parent 2:0 prio 1 u32 \
match u32 0x$IPREMOTE_HEX 0xffffffff at 24 flowid 2:$ID
(found in LARTC)
Maybe I miss something, but generally (for IP, TCP etc. matches) it
should work "as usual". I think you and those other users you quoted
were mislead by that tcpdump on ifb. Probably in some configs you
might needed this 'protocol 0x8864' or 'protocol all'. You should
see it on ppp's tcpdump then, like yours:
On Wed, Oct 14, 2009 at 09:11:55PM +0200, Jarek Poplawski wrote:
Denys Fedoryschenko wrote, On 10/13/2009 12:44 AM:
...
quoted
As i understand, for pppoe case, he can just skip offset for ethernet and
pppoe header, and he can filter by ip, or not?
Current way is maybe better, cause someone who want to count everything with
ethernet and pppoe headers - can, and who want without - also can (by setting
offset , just a bit more difficult.
Like
/sbin/tc filter add dev eth1 protocol 0x8864 parent 2:0 prio 1 u32 \
match u32 0x$IPREMOTE_HEX 0xffffffff at 24 flowid 2:$ID
(found in LARTC)
Maybe I miss something, but generally (for IP, TCP etc. matches) it
should work "as usual". I think you and those other users you quoted
were mislead by that tcpdump on ifb. Probably in some configs you
might needed this 'protocol 0x8864' or 'protocol all'. You should
see it on ppp's tcpdump then, like yours:
Hmm... Of course, like yours, where we can't see it ;-)
so 'protocol ip' is enough.