Re: Intro into qdisc writing?

On 10/08/2021 10.34, Eric Dumazet wrote:

On 8/10/21 5:17 AM, Thorsten Glaser wrote:

quoted

Hi,

I hope this is the right place to ask this kind of questions,
and not just send patches ☺

I’m currently working with a… network simulator of sorts, which
has so far mostly used htb, netem, dualpi2 and fq_codel to do the
various tricks needed for whatever they do, but now I have rather
specific change requests (one of which I already implemented).

The next things on my list basically involve delaying all traffic
or a subset of traffic for a certain amount of time (in the one‑ to
two-digit millisecond ballpark, so rather long, in CPU time). I’ve
seen the netem source use qdisc_watchdog_schedule_ns for this, but,
unlike the functions I used in my earlier module changes, I cannot
find any documentation for this.

Similarily, is there an intro of sorts for qdisc writing, the things
to know, concepts, locking, whatever is needed?

My background is multi-decade low-level programmer, but so far only
userland, libc variants and bootloaders, not kernel, and what bit of
kernel I touched so far was in BSD land so any pointers welcome.

If it helps: while this is for a customer project, so far everything
coming out of it is published under OSS licences; mostly at
https://github.com/tarent/sch_jens/tree/master/sch_jens as regards
the kernel module (and ../jens/ for the relayfs client example) but
https://github.com/tarent/ECN-Bits has a related userspace project.

Thanks in advance,
//mirabilos

Instead of writing a new qdisc, you could simply use FQ packet scheduler,
and a eBPF program adjusting skb->tstamp depending on your needs.

https://legacy.netdevconf.info/0x14/session.html?talk-replacing-HTB-with-EDT-and-BPF

Good link and reference.

If you want to see some code doing this via BPF see: 
https://github.com/xdp-project/bpf-examples/blob/master/traffic-pacing-edt/

I've unfortunately not had time to document the 'traffic-pacing-edt' 
code, as this was done under time pressure, for solving a production 
problem at an ISP. They needed packet pacing or transmission smoothing 
due to switches with too small buffers to handle bursts, but as close to 
1Gbit/s as possible as they sold 1G to their customers.

The comments in the code and scripts should hopefully be enough for you 
to understand the concept. Eric's slides describe the overall concept 
and background.


The main code you want to look at is in 'edt_pacer_vlan.c' [1], but 
notice that is assumes it have lockless access to the datastructure. 
This assumption is only true because the XDP-prog in 'xdp_cpumap_qinq.c' 
[2] moves packets associated with the datastructure to the right CPU 
(and invokes/starts the normal networks stack on that CPU).


[1] 
https://github.com/xdp-project/bpf-examples/blob/master/traffic-pacing-edt/edt_pacer_vlan.c

[2] 
https://github.com/xdp-project/bpf-examples/blob/master/traffic-pacing-edt/xdp_cpumap_qinq.c


--Jesper