From: Paweł Staszewski <hidden> Date: 2017-12-11 21:48:04
W dniu 2017-12-11 o 22:23, Paweł Staszewski pisze:
Hi
I just upgraded some testing host to 4.15.0-rc2+ kernel
And after some time of traffic processing - when traffic on all ports reach about 3Mpps - memleak started.
Graph attached from memory usage: https://ibb.co/idK4zb
HW config:
Intel E5
8x Intel 82599 (used ixgbe driver from kernel)
Interfaces with vlans attached
All 8 ethernet ports are in one LAG group configured by team.
With current settings
(this host is acting as a router - and bgpd process is eating same amount of memory from the beginning about 5.2GB)
cat /proc/meminfo
MemTotal: 32770588 kB
MemFree: 11342492 kB
MemAvailable: 10982752 kB
Buffers: 84704 kB
Cached: 83180 kB
SwapCached: 0 kB
Active: 5105320 kB
Inactive: 46252 kB
Active(anon): 4985448 kB
Inactive(anon): 1096 kB
Active(file): 119872 kB
Inactive(file): 45156 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 4005280 kB
SwapFree: 4005280 kB
Dirty: 236 kB
Writeback: 0 kB
AnonPages: 4983752 kB
Mapped: 13556 kB
Shmem: 2852 kB
Slab: 1013124 kB
SReclaimable: 45876 kB
SUnreclaim: 967248 kB
KernelStack: 7152 kB
PageTables: 12164 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 20390572 kB
Committed_AS: 396568 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 0 kB
VmallocChunk: 0 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
CmaTotal: 0 kB
CmaFree: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 1407572 kB
DirectMap2M: 20504576 kB
DirectMap1G: 13631488 kB
ps aux --sort -rss
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 6758 1.8 14.9 5044996 4886964 ? Sl 01:22 23:21 /usr/local/sbin/bgpd -d -u root -g root -I --ignore_warnings
root 6752 0.0 0.1 86272 61920 ? Ss 01:22 0:16 /usr/local/sbin/zebra -d -u root -g root -I --ignore_warnings
root 6766 12.6 0.0 51592 29196 ? S 01:22 157:48 /usr/sbin/snmpd -p /var/run/snmpd.pid -Ln
root 7494 0.0 0.0 708976 5896 ? Ssl 01:22 0:09 /opt/collectd/sbin/collectd
root 15531 0.0 0.0 67864 5056 ? Ss 21:57 0:00 sshd: paol [priv]
root 4915 0.0 0.0 271912 4904 ? Ss 01:21 0:25 /usr/sbin/syslog-ng --persist-file /var/lib/syslog-ng/syslog-ng.persist --cfgfile /etc/syslog-ng/syslog-ng.conf --pidfile /run/syslog-ng.pid
root 4278 0.0 0.0 37220 4164 ? Ss 01:21 0:00 /lib/systemd/systemd-udevd --daemon
root 5147 0.0 0.0 32072 3232 ? Ss 01:21 0:00 /usr/sbin/sshd
root 5203 0.0 0.0 28876 2436 ? S 01:21 0:00 teamd -d -f /etc/teamd.conf
root 17372 0.0 0.0 17924 2388 pts/2 R+ 22:13 0:00 ps aux --sort -rss
root 4789 0.0 0.0 5032 2176 ? Ss 01:21 0:00 mdadm --monitor --scan --daemonise --pid-file /var/run/mdadm.pid --syslog
root 7511 0.0 0.0 12676 1920 tty4 Ss+ 01:22 0:00 /sbin/agetty 38400 tty4 linux
root 7510 0.0 0.0 12676 1896 tty3 Ss+ 01:22 0:00 /sbin/agetty 38400 tty3 linux
root 7512 0.0 0.0 12676 1860 tty5 Ss+ 01:22 0:00 /sbin/agetty 38400 tty5 linux
root 7513 0.0 0.0 12676 1836 tty6 Ss+ 01:22 0:00 /sbin/agetty 38400 tty6 linux
root 7509 0.0 0.0 12676 1832 tty2 Ss+ 01:22 0:00 /sbin/agetty 38400 tty2 linux
And latest kernel that everything was working is: 4.14.3
Some observations - when i disable tso on all cards there is more memleak.
When traffic starts to drop - there is less and less memleak
below link to memory usage graph:
https://ibb.co/hU97kG
And there is rising slab_unrecl - Amount of unreclaimable memory used for slab kernel allocations
Forgot to add that im using hfsc and qdiscs like pfifo on classes.
From: John Fastabend <john.fastabend@gmail.com> Date: 2017-12-11 22:15:59
On 12/11/2017 01:48 PM, Paweł Staszewski wrote:
W dniu 2017-12-11 o 22:23, Paweł Staszewski pisze:
quoted
Hi
I just upgraded some testing host to 4.15.0-rc2+ kernel
And after some time of traffic processing - when traffic on all ports
reach about 3Mpps - memleak started.
[...]
quoted
Some observations - when i disable tso on all cards there is more
memleak.
When traffic starts to drop - there is less and less memleak
below link to memory usage graph:
https://ibb.co/hU97kG
And there is rising slab_unrecl - Amount of unreclaimable memory used
for slab kernel allocations
Forgot to add that im using hfsc and qdiscs like pfifo on classes.
Maybe some error case I missed in the qdisc patches I'm looking into
it.
Thanks,
John
From: Paweł Staszewski <hidden> Date: 2017-12-11 22:27:50
W dniu 2017-12-11 o 23:15, John Fastabend pisze:
On 12/11/2017 01:48 PM, Paweł Staszewski wrote:
quoted
W dniu 2017-12-11 o 22:23, Paweł Staszewski pisze:
quoted
Hi
I just upgraded some testing host to 4.15.0-rc2+ kernel
And after some time of traffic processing - when traffic on all ports
reach about 3Mpps - memleak started.
[...]
quoted
quoted
Some observations - when i disable tso on all cards there is more
memleak.
When traffic starts to drop - there is less and less memleak
below link to memory usage graph:
https://ibb.co/hU97kG
And there is rising slab_unrecl - Amount of unreclaimable memory used
for slab kernel allocations
Forgot to add that im using hfsc and qdiscs like pfifo on classes.
Maybe some error case I missed in the qdisc patches I'm looking into
it.
Thanks,
John
This is how it looks like when corelated on graph - traffic vs mem
https://ibb.co/njpkqG
Typical hfsc class + qdisc:
### Client interface vlan1616
tc qdisc del dev vlan1616 root
tc qdisc add dev vlan1616 handle 1: root hfsc default 100
tc class add dev vlan1616 parent 1: classid 1:100 hfsc ls m2 200Mbit ul m2 200Mbit
tc qdisc add dev vlan1616 parent 1:100 handle 100: pfifo limit 128
### End TM for client interface
tc qdisc del dev vlan1616 ingress
tc qdisc add dev vlan1616 handle ffff: ingress
tc filter add dev vlan1616 parent ffff: protocol ip prio 50 u32 match ip src 0.0.0.0/0 police rate 200Mbit burst 200M mtu 32k drop flowid 1:1
And this is same for about 450 vlan interfaces
Good thing is that compared to 4.14.3 i have about 5% less cpu load on 4.15.0-rc2+
When hfsc will be lockless or tbf - then it will be really huge difference in cpu load on x86 when using traffic shaping - so really good job John.
From: Paweł Staszewski <hidden> Date: 2017-12-12 17:57:35
W dniu 2017-12-11 o 23:27, Paweł Staszewski pisze:
W dniu 2017-12-11 o 23:15, John Fastabend pisze:
quoted
On 12/11/2017 01:48 PM, Paweł Staszewski wrote:
quoted
W dniu 2017-12-11 o 22:23, Paweł Staszewski pisze:
quoted
Hi
I just upgraded some testing host to 4.15.0-rc2+ kernel
And after some time of traffic processing - when traffic on all ports
reach about 3Mpps - memleak started.
[...]
quoted
quoted
Some observations - when i disable tso on all cards there is more
memleak.
When traffic starts to drop - there is less and less memleak
below link to memory usage graph:
https://ibb.co/hU97kG
And there is rising slab_unrecl - Amount of unreclaimable memory used
for slab kernel allocations
Forgot to add that im using hfsc and qdiscs like pfifo on classes.
Maybe some error case I missed in the qdisc patches I'm looking into
it.
Thanks,
John
This is how it looks like when corelated on graph - traffic vs mem
https://ibb.co/njpkqG
Typical hfsc class + qdisc:
### Client interface vlan1616
tc qdisc del dev vlan1616 root
tc qdisc add dev vlan1616 handle 1: root hfsc default 100
tc class add dev vlan1616 parent 1: classid 1:100 hfsc ls m2 200Mbit ul m2 200Mbit
tc qdisc add dev vlan1616 parent 1:100 handle 100: pfifo limit 128
### End TM for client interface
tc qdisc del dev vlan1616 ingress
tc qdisc add dev vlan1616 handle ffff: ingress
tc filter add dev vlan1616 parent ffff: protocol ip prio 50 u32 match ip src 0.0.0.0/0 police rate 200Mbit burst 200M mtu 32k drop flowid 1:1
And this is same for about 450 vlan interfaces
Good thing is that compared to 4.14.3 i have about 5% less cpu load on 4.15.0-rc2+
When hfsc will be lockless or tbf - then it will be really huge difference in cpu load on x86 when using traffic shaping - so really good job John.
From: John Fastabend <john.fastabend@gmail.com> Date: 2017-12-13 21:05:59
On 12/12/2017 09:57 AM, Paweł Staszewski wrote:
W dniu 2017-12-11 o 23:27, Paweł Staszewski pisze:
quoted
W dniu 2017-12-11 o 23:15, John Fastabend pisze:
quoted
On 12/11/2017 01:48 PM, Paweł Staszewski wrote:
quoted
W dniu 2017-12-11 o 22:23, Paweł Staszewski pisze:
quoted
Hi
I just upgraded some testing host to 4.15.0-rc2+ kernel
And after some time of traffic processing - when traffic on all ports
reach about 3Mpps - memleak started.
[...]
quoted
quoted
Some observations - when i disable tso on all cards there is more
memleak.
When traffic starts to drop - there is less and less memleak
below link to memory usage graph:
https://ibb.co/hU97kG
And there is rising slab_unrecl - Amount of unreclaimable memory used
for slab kernel allocations
Forgot to add that im using hfsc and qdiscs like pfifo on classes.
Maybe some error case I missed in the qdisc patches I'm looking into
it.
Thanks,
John
This is how it looks like when corelated on graph - traffic vs mem
https://ibb.co/njpkqG
Typical hfsc class + qdisc:
### Client interface vlan1616
tc qdisc del dev vlan1616 root
tc qdisc add dev vlan1616 handle 1: root hfsc default 100
tc class add dev vlan1616 parent 1: classid 1:100 hfsc ls m2 200Mbit ul m2 200Mbit
tc qdisc add dev vlan1616 parent 1:100 handle 100: pfifo limit 128
### End TM for client interface
tc qdisc del dev vlan1616 ingress
tc qdisc add dev vlan1616 handle ffff: ingress
tc filter add dev vlan1616 parent ffff: protocol ip prio 50 u32 match ip src 0.0.0.0/0 police rate 200Mbit burst 200M mtu 32k drop flowid 1:1
And this is same for about 450 vlan interfaces
Good thing is that compared to 4.14.3 i have about 5% less cpu load on 4.15.0-rc2+
When hfsc will be lockless or tbf - then it will be really huge difference in cpu load on x86 when using traffic shaping - so really good job John.
It seems I was able to produce a similar memleak with qdisc patches
reverted and running TCP traffic overnight. I guess we can do a bisect
and track it down. Will try to get a "good" run tonight.
Thanks,
John