Re: kernel mode pppoe ppp if + ifb + mirred redirect, ethernet packets in ifb?!

I bet pppd in user space is probably your biggest problem in terms of
performance.

No. :-)
First problem, on restart and massive users login i am getting some weird 
locking problem with modprobe,probably related to busybox/mdev.
When interface come up, maybe it is problem of mdev, it is trying to do 
modprobe with ip address as attribute. Probably they must not modprobe on 
virtual interface. It is not easy to track who is reason of such call.

As result:
[  174.564503] request_module: runaway loop modprobe 172.16.3.1
[  174.564702] request_module: runaway loop modprobe 172.16.3.1
[  174.801355] request_module: runaway loop modprobe 172.16.106.1
[  174.801487] request_module: runaway loop modprobe 172.16.106.1
[  175.011415] request_module: runaway loop modprobe 172.16.106.1
load average (even it doesn't mean anything in terms of CPU load) jumps to 
72-80.

Another bottleneck is u32 (i can optimize but) and some strange locks 
appearing at top of perf, maybe same as logon case.
And yes, pppd also appearing, but seems just registering new sysctls (for new 
interface?) takes a lot of resources.

I can post perf -a -f g if you are interested for "logging in" case and 
regular operation.

quoted

After switching to skbedit things improve a lot (before 1k users was near
max)

Not using netfilter will improve your numbers. So can skbedit do fwmark
as well?

I dont need it, i am using skb->priority as a key for flow classifier.
It looks weird, a lot of obsolete code there, but it is very nice, i dont need 
to touch almost anything on ifb, and i can predefine even on startup required 
amount of classes and just change rate when required.

Here is key part of shaper code:

$TC filter add dev ifb0 protocol ip pref 32 parent 1: handle 1 flow map key 
priority baseclass 1:64

Then i create classes for id's i need (at my case id related to ppp interface 
number).

and when ppp interface come up also:
(lowid is related to ppp interface or user id)

filter add dev $2 parent ffff: protocol ip prio 10 u32 \
match u32 0 0 flowid 1:1 \
action skbedit priority 0x${lowid} pipe \
action mirred egress redirect dev ifb0

yes, something like that.
It may be easier to tcpdump -x on both pppoe and ifb and see how the
packets look like at what offset. If that doesnt work well, I will work
on a patch...

Well another way is just to use  as you suggest - egress on output 
interface(s).

cheers,
jamal