kernel mode pppoe ppp if + ifb + mirred redirect, ethernet packets in ifb?!

Hi

I am using kernel mode pppoe, and while using mirred on ppp interfaces (to 
shape users upstream) noticed very strange behaviour (and IMHO wrong)

$2 - ppp interface
tc filter add dev $2 parent ffff: protocol ip prio 10 u32 \
match u32 0 0 flowid 1:1 \
action mirred egress redirect dev ifb0"

If i do tcpdump on ppp interface i can see normal ip traffic

PPPoE_146 ~ # tcpdump -ni ppp6 -c 100
tcpdump: WARNING:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ppp6, link-type LINUX_SLL (Linux cooked), capture size 96 bytes
11:36:16.541742 IP 172.16.131.221.2182 > 64.4.50.51.443: Flags [F.], seq 
2636760792, ack 2812561777, win 64963, length 0

If i do tcpdump on ifb0 i will see PPPoE incapsulated traffic! 
PPPoE_146 ~ # tcpdump -ni ifb0 -c 100
tcpdump: WARNING: ifb0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ifb0, link-type EN10MB (Ethernet), capture size 96 bytes
11:36:58.949727 PPPoE  [ses 0x15] IP 172.16.98.22.2081 > 94.75.218.20.80: 
Flags [S], seq 3205622298, win 65535, options [mss 1440,nop,nop,sackOK], 
length 0
11:36:58.531473 PPPoE  [ses 0xd] IP 172.16.98.14.16526 > 128.16.130.164.445: 
Flags [S], seq 98566173, win 65535, options [mss 1440,nop,nop,sackOK], length 
0

Is it expected that redirecting ppp interface, that supposed to be clean IP 
traffic is becoming eth encapsulated traffic?
Probably some bad interaction with kernel mode pppoe?

On Mon, 2009-10-12 at 11:43 +0300, Denys Fedoryschenko wrote:

Is it expected that redirecting ppp interface, that supposed to be clean IP 
traffic is becoming eth encapsulated traffic?

No. Imagine if there were other types of packets non-ip for example,
what do you do then? 
this feature is as close as you can get when you do switch level
mirroring or redirection. If you want to edit header before redirect
etc, use pedit (refer to recent discussion with someone who wanted to
replicate packets for redundant routing purposes); 

cheers,
jamal

How it can be ethernet packets (with PPPoE headers) if i am redirecting from 
ppp interface, not ethernet interface?

PPPoE_146 ~ # tcpdump -ni ppp0 -e -vvv -s 1500 -c 4
tcpdump: WARNING:
tcpdump: listening on ppp0, link-type LINUX_SLL (Linux cooked), capture size 
1500 bytes
17:03:17.015598 Out ethertype IPv4 (0x0800), length 68: (tos 0x0, ttl 111, id 
45623, offset 0, flags [DF], proto TCP (6), length 52)
    68.231.189.241.24800 > 172.16.131.199.1060: Flags [.], cksum 0xd208 
(correct), seq 783840165, ack 1980178761, win 32748, options [nop,nop,sack 1 
{1441:8741}], length 0

PPPoE_146 ~ # tcpdump -ni ifb0 -e -vvv -s 1500 -c 4
tcpdump: WARNING:
tcpdump: WARNING: ifb0: no IPv4 address assigned
tcpdump: listening on ifb0, link-type EN10MB (Ethernet), capture size 1500 
bytes
17:04:01.625547 00:0a:cd:17:5e:08 > 00:0a:cd:14:6b:67, ethertype PPPoE S 
(0x8864), length 70: PPPoE  [ses 0xb] IP (0x0021), length 50: (tos 0x0, ttl 
128, id 51706, offset 0, flags [DF], proto TCP (6), length 48)

Btw i test, if i use userspace pppoe (synchronous) i will get different output 
in tcpdump
PPPoE_146 ~ # tcpdump -ni ifb0 -vvvv
tcpdump: WARNING:
tcpdump: WARNING: ifb0: no IPv4 address assigned
tcpdump: listening on ifb0, link-type EN10MB (Ethernet), capture size 96 bytes
17:15:41.512553 00:28:00:d6:40:00 > ff:03:00:21:45:00, ethertype Unknown 
(0x8006), length 44:
        0x0000:  df49 ac10 8320 cf2e 1c51 0416 0050 ce7b  .I.......Q...P.{
        0x0010:  e177 1caf 491d 5010 ffff 7afe 0000       .w..I.P...z...
17:15:41.520802 00:34:00:d7:40:00 > ff:03:00:21:45:00, ethertype Unknown 
(0x8006), length 112:
        0x0000:  df3c ac10 8320 cf2e 1c51 0416 0050 ce7b  .<.......Q...P.{
        0x0010:  e177 1caf 491d 8010 ffff 686e 0000 0101  .w..I.....hn....
        0x0020:  050a 1caf 4ebd 1caf 545d ff03 0021 4500  ....N...T]...!E.
        0x0030:  0034 00d8 4000 8006 df3b ac10 8320 cf2e  .4..@....;......
        0x0040:  1c51 0416 0050 ce7b e177 1caf 491d 8010  .Q...P.{.w..I...
        0x0050:  ffff                                     ..

9 more lines

On Mon, 2009-10-12 at 11:43 +0300, Denys Fedoryschenko wrote:

quoted

Is it expected that redirecting ppp interface, that supposed to be clean
IP traffic is becoming eth encapsulated traffic?

No. Imagine if there were other types of packets non-ip for example,
what do you do then?
this feature is as close as you can get when you do switch level
mirroring or redirection. If you want to edit header before redirect
etc, use pedit (refer to recent discussion with someone who wanted to
replicate packets for redundant routing purposes);

cheers,
jamal

On Mon, 2009-10-12 at 17:15 +0300, Denys Fedoryschenko wrote:

How it can be ethernet packets (with PPPoE headers) if i am redirecting from 
ppp interface, not ethernet interface?

You are running this on ingress. If you run on egress you will see
proper header.
_Most people_ want to see the whole original header for qos accounting
purposes etc. PPPOE claims to be ethernet, PPP does not. Hence
difference in observation. 

We could add an override feature to say "i only want 
see what was passed not wire packet". 
If this is important to you let me know and i will try to make time to
add it (with hopefully not breaking old scripts); if you feel brave as
well i could guide you on how to add it.

cheers,
jamal

I don't have problem with existing behaviour, since i am using other way of 
shaping, for my case using pktedit to assign priority to SKB and shaping by 
it.

But generally problem is was told by one of russian developers who is working 
on firmware for few models of broadband routers, he asked to help on ISP 
forum, and if possible to explain this to someone who can give advice, and 
maybe tell that probably there is a bug.

It is explained here (but it is russian language, i dont think worth spending 
time and translating) http://forum.nag.ru/forum/index.php?showtopic=51760
His case a bit complicated, since that ISP who provide pppoe use some kind of 
ugly, oversized packets, and as he mention they can be fragmented. I didn't 
understand exactly how it can be possible and how it can break u32 filter.

Here is one more guy who asked to help about that
http://mailman.ds9a.nl/pipermail/lartc/2007q3/021561.html

His issue that he wants to shape by ip, not by interface, and while it is 
ethernet traffic in ifb, it is not trivial. I told him just to use offset in 
u32,but he said there is case with possible packet fragmentation (not sure it 
can happen) and shaper won't match.
I am just asking to take a look, if there is any bug.
If it is not, then just simple question, it will work reliably if i just use 
u32 filter with offset on ifb?



On Monday 12 October 2009 23:57:44 jamal wrote:

On Mon, 2009-10-12 at 17:15 +0300, Denys Fedoryschenko wrote:

quoted

How it can be ethernet packets (with PPPoE headers) if i am redirecting
from ppp interface, not ethernet interface?

You are running this on ingress. If you run on egress you will see
proper header.
_Most people_ want to see the whole original header for qos accounting
purposes etc. PPPOE claims to be ethernet, PPP does not. Hence
difference in observation.

We could add an override feature to say "i only want
see what was passed not wire packet".
If this is important to you let me know and i will try to make time to
add it (with hopefully not breaking old scripts); if you feel brave as
well i could guide you on how to add it.

On Tue, 2009-10-13 at 00:54 +0300, Denys Fedoryschenko wrote:

I don't have problem with existing behaviour, since i am using other way of 
shaping, for my case using pktedit to assign priority to SKB and shaping by 
it.

I am dissapointed Denys, you dont like ipt?;->

But generally problem is was told by one of russian developers who is working 
on firmware for few models of broadband routers, he asked to help on ISP 
forum, and if possible to explain this to someone who can give advice, and 
maybe tell that probably there is a bug.

[..]

If it is not, then just simple question, it will work reliably if i just use 
u32 filter with offset on ifb?

Yes, of course you can if you add offset sizeof pppoe header.
But:
It looks like there is a genuine need for this feature.

The challenge is: I am trying to be generic across devices of many
different types (ethernet, atm, virtual etc) at many entry points,
ingress, egress local, forwarding etc. 
This feature that this person would need will only work if you _know_
what you are doing; i.e in this case, I can very easily turn it off with
a simple command - but the user must know that they do this on ingress
side. I can cook a very quick patch for kernel and user space - you
think this user can test it?

cheers,
jamal

On Tuesday 13 October 2009 01:07:30 jamal wrote:

On Tue, 2009-10-13 at 00:54 +0300, Denys Fedoryschenko wrote:

quoted

I don't have problem with existing behaviour, since i am using other way
of shaping, for my case using pktedit to assign priority to SKB and
shaping by it.

I am dissapointed Denys, you dont like ipt?;->

It kills me :-) Each new version it doesn't work and i notice, i'm almost one  
who use it :-) Probably i should wait till netfilter API and iptables 
conversion will stabilize somehow.

Plus skbedit in some cases will be faster, if i eliminate iptables, unloading 
modules even, basic filtering can be done by iproute2 too, i won't have 
netfilter locks that make things slow on SMP (at least what i heard here and 
what oprofile shows, that MARK was small CPU hog to compare with skbedit).

I am happily running 2k pppoe users on Quad Core CPU/on supercheap r8169
(better nic not available here) with skbedit and flow classifier. It can do 
more even, i think.
After switching to skbedit things improve a lot (before 1k users was near max)

quoted

But generally problem is was told by one of russian developers who is
working on firmware for few models of broadband routers, he asked to help
on ISP forum, and if possible to explain this to someone who can give
advice, and maybe tell that probably there is a bug.

[..]

quoted

If it is not, then just simple question, it will work reliably if i just
use u32 filter with offset on ifb?

Yes, of course you can if you add offset sizeof pppoe header.
But:
It looks like there is a genuine need for this feature.

The challenge is: I am trying to be generic across devices of many
different types (ethernet, atm, virtual etc) at many entry points,
ingress, egress local, forwarding etc.
This feature that this person would need will only work if you _know_
what you are doing; i.e in this case, I can very easily turn it off with
a simple command - but the user must know that they do this on ingress
side. I can cook a very quick patch for kernel and user space - you
think this user can test it?

I can test even, even if he won't.

As i understand, for pppoe case, he can just skip offset for ethernet and 
pppoe header, and he can filter by ip, or not?
Current way is maybe better, cause someone who want to count everything with 
ethernet and pppoe headers - can, and who want without - also can (by setting 
offset , just a bit more difficult.

Like 
/sbin/tc filter add dev eth1 protocol 0x8864  parent 2:0 prio 1 u32 \
match u32 0x$IPREMOTE_HEX 0xffffffff at 24 flowid 2:$ID
(found in LARTC)

cheers,
jamal

On Tue, 2009-10-13 at 01:44 +0300, Denys Fedoryschenko wrote:

It kills me :-) Each new version it doesn't work and i notice, i'm almost one  
who use it :-) Probably i should wait till netfilter API and iptables 
conversion will stabilize somehow.

I am a little frustrated - but yeah, waiting may help. I tend to do a
lot of private support to fix integration with iptables problems, so you
are not the only user ;->. I wish one distro gets it right.
The most hopeful seems to be debian.

Plus skbedit in some cases will be faster, if i eliminate iptables, unloading 
modules even, basic filtering can be done by iproute2 too, i won't have 
netfilter locks that make things slow on SMP (at least what i heard here and 
what oprofile shows, that MARK was small CPU hog to compare with skbedit).

Makes sense.

I am happily running 2k pppoe users on Quad Core CPU/on supercheap r8169
(better nic not available here) with skbedit and flow classifier. It can do 
more even, i think.

I bet pppd in user space is probably your biggest problem in terms of
performance.

After switching to skbedit things improve a lot (before 1k users was near max)

Not using netfilter will improve your numbers. So can skbedit do fwmark
as well?

I can test even, even if he won't.
As i understand, for pppoe case, he can just skip offset for ethernet and 
pppoe header, and he can filter by ip, or not?
Current way is maybe better, cause someone who want to count everything with 
ethernet and pppoe headers - can, and who want without - also can (by setting 
offset , just a bit more difficult.

Like 
/sbin/tc filter add dev eth1 protocol 0x8864  parent 2:0 prio 1 u32 \
match u32 0x$IPREMOTE_HEX 0xffffffff at 24 flowid 2:$ID
(found in LARTC)

yes, something like that. 
It may be easier to tcpdump -x on both pppoe and ifb and see how the
packets look like at what offset. If that doesnt work well, I will work
on a patch...

cheers,
jamal

I bet pppd in user space is probably your biggest problem in terms of
performance.

No. :-)
First problem, on restart and massive users login i am getting some weird 
locking problem with modprobe,probably related to busybox/mdev.
When interface come up, maybe it is problem of mdev, it is trying to do 
modprobe with ip address as attribute. Probably they must not modprobe on 
virtual interface. It is not easy to track who is reason of such call.

As result:
[  174.564503] request_module: runaway loop modprobe 172.16.3.1
[  174.564702] request_module: runaway loop modprobe 172.16.3.1
[  174.801355] request_module: runaway loop modprobe 172.16.106.1
[  174.801487] request_module: runaway loop modprobe 172.16.106.1
[  175.011415] request_module: runaway loop modprobe 172.16.106.1
load average (even it doesn't mean anything in terms of CPU load) jumps to 
72-80.

Another bottleneck is u32 (i can optimize but) and some strange locks 
appearing at top of perf, maybe same as logon case.
And yes, pppd also appearing, but seems just registering new sysctls (for new 
interface?) takes a lot of resources.

I can post perf -a -f g if you are interested for "logging in" case and 
regular operation.

quoted

After switching to skbedit things improve a lot (before 1k users was near
max)

Not using netfilter will improve your numbers. So can skbedit do fwmark
as well?

I dont need it, i am using skb->priority as a key for flow classifier.
It looks weird, a lot of obsolete code there, but it is very nice, i dont need 
to touch almost anything on ifb, and i can predefine even on startup required 
amount of classes and just change rate when required.

Here is key part of shaper code:

$TC filter add dev ifb0 protocol ip pref 32 parent 1: handle 1 flow map key 
priority baseclass 1:64

Then i create classes for id's i need (at my case id related to ppp interface 
number).

and when ppp interface come up also:
(lowid is related to ppp interface or user id)

filter add dev $2 parent ffff: protocol ip prio 10 u32 \
match u32 0 0 flowid 1:1 \
action skbedit priority 0x${lowid} pipe \
action mirred egress redirect dev ifb0

yes, something like that.
It may be easier to tcpdump -x on both pppoe and ifb and see how the
packets look like at what offset. If that doesnt work well, I will work
on a patch...

Well another way is just to use  as you suggest - egress on output 
interface(s).

cheers,
jamal

On Tue, 2009-10-13 at 15:45 +0300, Denys Fedoryschenko wrote:

First problem, on restart and massive users login i am getting some weird 
locking problem with modprobe,probably related to busybox/mdev.
When interface come up, maybe it is problem of mdev, it is trying to do 
modprobe with ip address as attribute. Probably they must not modprobe on 
virtual interface. It is not easy to track who is reason of such call.

As result:
[  174.564503] request_module: runaway loop modprobe 172.16.3.1
[  174.564702] request_module: runaway loop modprobe 172.16.3.1
[  174.801355] request_module: runaway loop modprobe 172.16.106.1
[  174.801487] request_module: runaway loop modprobe 172.16.106.1
[  175.011415] request_module: runaway loop modprobe 172.16.106.1
load average (even it doesn't mean anything in terms of CPU load) jumps to 
72-80.

Strange. But would probably make sense to modprobe the virtual interface
then assign ip address to it.

Another bottleneck is u32 (i can optimize but) and some strange locks 
appearing at top of perf, maybe same as logon case.

like you say u32 can be optimized

And yes, pppd also appearing, but seems just registering new sysctls (for new 
interface?) takes a lot of resources.

back in the days (when i was involved in pppoe) pppd was a big problem
because it was not rentrant, so to solve the problem you exec a new
process for each user..
Ben Lahaise did a lot of work in this area and i am pretty sure decided
not to use pppd.

I can post perf -a -f g if you are interested for "logging in" case and 
regular operation.

Sure - but i may not be much use to you..
I have CCed Ben.

Here is key part of shaper code:

Ok - makes sense. I know some people using mark for ingress pseudo RPF.
And given instability of netfilter interface, I may just end up patching
skbedit.

Well another way is just to use  as you suggest - egress on output 
interface(s).

Sure; If it becomes an issue let me know.

cheers,
jamal

Hi folks,

On Tue, Oct 13, 2009 at 09:02:38AM -0400, jamal wrote:

On Tue, 2009-10-13 at 15:45 +0300, Denys Fedoryschenko wrote:

quoted

Another bottleneck is u32 (i can optimize but) and some strange locks 
appearing at top of perf, maybe same as logon case.

like you say u32 can be optimized

quoted

And yes, pppd also appearing, but seems just registering new sysctls (for new 
interface?) takes a lot of resources.

back in the days (when i was involved in pppoe) pppd was a big problem
because it was not rentrant, so to solve the problem you exec a new
process for each user..
Ben Lahaise did a lot of work in this area and i am pretty sure decided
not to use pppd.

Yes, I've been maintaining the ppp implementation (Babylon) initially 
written by SpellCaster back in '98.  My main focus of late has been 
dealing with PPPoE and L2TP, and specifically with terminating a lot of 
L2TP LNS traffic.  Babylon itself scales fairly well to large numbers of 
interfaces, as does the network stack these days.  Currently the bottleneck 
in interface creation is in the sysctl code.

The other big problem with large numbers of interfaces is interface 
deletion.  The synchronize_rcu() call in the interface teardown ends up 
making a single daemon with thousands of interfaces take a long time 
to clean up after itself.

		-ben

On Tue, 2009-10-13 at 09:18 -0400, jamal wrote:

Ok - makes sense. I know some people using mark for ingress pseudo RPF.
And given instability of netfilter interface, I may just end up patching
skbedit.

Ok, I just whipped a quick patch for skbedit to set the mark - and it
seems to work well after some basic testing. I will post.

cheers,
jamal

Denys Fedoryschenko wrote, On 10/13/2009 12:44 AM:
...

As i understand, for pppoe case, he can just skip offset for ethernet and 
pppoe header, and he can filter by ip, or not?
Current way is maybe better, cause someone who want to count everything with 
ethernet and pppoe headers - can, and who want without - also can (by setting 
offset , just a bit more difficult.

Like 
/sbin/tc filter add dev eth1 protocol 0x8864  parent 2:0 prio 1 u32 \
match u32 0x$IPREMOTE_HEX 0xffffffff at 24 flowid 2:$ID
(found in LARTC)

Maybe I miss something, but generally (for IP, TCP etc. matches) it
should work "as usual". I think you and those other users you quoted
were mislead by that tcpdump on ifb. Probably in some configs you
might needed this 'protocol 0x8864' or 'protocol all'. You should
see it on ppp's tcpdump then, like yours:

PPPoE_146 ~ # tcpdump -ni ppp0 -e -vvv -s 1500 -c 4
tcpdump: WARNING:
tcpdump: listening on ppp0, link-type LINUX_SLL (Linux cooked), capture size 
1500 bytes
17:03:17.015598 Out ethertype IPv4 (0x0800), length 68: (tos 0x0, ttl 111, id 

Cheers,
Jarek P.

On Wed, Oct 14, 2009 at 09:11:55PM +0200, Jarek Poplawski wrote:

Denys Fedoryschenko wrote, On 10/13/2009 12:44 AM:
...

quoted

As i understand, for pppoe case, he can just skip offset for ethernet and 
pppoe header, and he can filter by ip, or not?
Current way is maybe better, cause someone who want to count everything with 
ethernet and pppoe headers - can, and who want without - also can (by setting 
offset , just a bit more difficult.

Like 
/sbin/tc filter add dev eth1 protocol 0x8864  parent 2:0 prio 1 u32 \
match u32 0x$IPREMOTE_HEX 0xffffffff at 24 flowid 2:$ID
(found in LARTC)

Maybe I miss something, but generally (for IP, TCP etc. matches) it
should work "as usual". I think you and those other users you quoted
were mislead by that tcpdump on ifb. Probably in some configs you
might needed this 'protocol 0x8864' or 'protocol all'. You should
see it on ppp's tcpdump then, like yours:

Hmm... Of course, like yours, where we can't see it ;-)
so 'protocol ip' is enough.

quoted

PPPoE_146 ~ # tcpdump -ni ppp0 -e -vvv -s 1500 -c 4
tcpdump: WARNING:
tcpdump: listening on ppp0, link-type LINUX_SLL (Linux cooked), capture size 
1500 bytes
17:03:17.015598 Out ethertype IPv4 (0x0800), length 68: (tos 0x0, ttl 111, id 

BTW, let's note this 'protocol 0x8864' was used with dev eth1, so it's
a different case.

Jarek P.