Re: Regression: net/ipv6/mld running system out of memory (not a leak)
From: Rafał Miłecki <zajec5@gmail.com>
Date: 2020-03-03 06:16:50
On 12.02.2020 08:37, Rafał Miłecki wrote:
I run Linux based OpenWrt distribution on home wireless devices (ARM
routers and access points with brcmfmac wireless driver). I noticed
that using wireless monitor mode interface results in my devices (128
MiB RAM) running out of memory in about 2 days. This is NOT a memory
leak as putting wireless down brings back all the memory.
Interestingly this memory drain requires at least one of:
net.ipv6.conf.default.forwarding=1
net.ipv6.conf.all.forwarding=1
to be set. OpenWrt happens to use both by default.
This regression was introduced by the commit 1666d49e1d41 ("mld: do
not remove mld souce list info when set link down") - first appeared
in 4.10 and then backported. This bug exists in 4.9.14 and 4.14.169.
Reverting that commit from 4.9.14 and 4.14.169 /fixes/ the problem.I have some interesting debugging results to share. I decided to log all kfree() operations in mcast.c. Logging mld_clear_delrec() provided some promising info. With kernel 4.14.169 (affected by regression) bringing monitor ifaces down doesn't result in freeing any memory in mld_clear_delrec(): down [ 73.487846] [ipv6_mc_down] idev->dev->name:mon-phy0 [ 73.917823] [ipv6_mc_down] idev->dev->name:mon-phy1 down [ 76.767781] [ipv6_mc_down] idev->dev->name:mon-phy0 [ 77.157790] [ipv6_mc_down] idev->dev->name:mon-phy1 down [ 79.658260] [ipv6_mc_down] idev->dev->name:mon-phy0 [ 80.047805] [ipv6_mc_down] idev->dev->name:mon-phy1 down [ 83.067773] [ipv6_mc_down] idev->dev->name:mon-phy0 [ 83.447778] [ipv6_mc_down] idev->dev->name:mon-phy1 down [ 86.555700] [ipv6_mc_down] idev->dev->name:mon-phy0 [ 86.985706] [ipv6_mc_down] idev->dev->name:mon-phy1 However removing interfaces reveals there were duplicated entries in idev->mc_tomb lists: remove [ 89.694038] [ipv6_mc_down] idev->dev->name:wlan1 [ 89.717953] [ipv6_mc_destroy_dev] idev->dev->name:mon-phy1 [ 89.723433] [ipv6_mc_down] idev->dev->name:mon-phy1 [ 89.728375] [ipv6_mc_destroy_dev -> mld_clear_delrec] kfree(pmc:c6e2c600) ff02::2 [ 89.736013] [ipv6_mc_destroy_dev -> mld_clear_delrec] kfree(pmc:c620b100) ff02::2 [ 89.743643] [ipv6_mc_destroy_dev -> mld_clear_delrec] kfree(pmc:c620b900) ff02::2 [ 89.751275] [ipv6_mc_destroy_dev -> mld_clear_delrec] kfree(pmc:c620b680) ff02::2 [ 89.758901] [ipv6_mc_destroy_dev -> mld_clear_delrec] kfree(pmc:c620b580) ff02::2 [ 89.868542] [ipv6_mc_down] idev->dev->name:wlan0 [ 89.878203] [ipv6_mc_destroy_dev] idev->dev->name:wlan1 [ 89.883479] [ipv6_mc_down] idev->dev->name:wlan1 [ 89.888208] [ipv6_mc_destroy_dev -> mld_clear_delrec] kfree(pmc:c6e2ca80) ff02::2 [ 89.895901] [ipv6_mc_destroy_dev -> mld_clear_delrec] kfree(pmc:c6e2ca00) ff02::1:ff10:e018 [ 89.904412] [ipv6_mc_destroy_dev -> mld_clear_delrec] kfree(pmc:c6e2c980) ff02::1:ff00:0 [ 89.937564] [ipv6_mc_destroy_dev] idev->dev->name:mon-phy0 [ 89.943062] [ipv6_mc_down] idev->dev->name:mon-phy0 [ 89.948070] [ipv6_mc_destroy_dev -> mld_clear_delrec] kfree(pmc:c6e2c300) ff02::2 [ 89.955728] [ipv6_mc_destroy_dev -> mld_clear_delrec] kfree(pmc:c620b780) ff02::2 [ 89.963392] [ipv6_mc_destroy_dev -> mld_clear_delrec] kfree(pmc:c620b980) ff02::2 [ 89.971035] [ipv6_mc_destroy_dev -> mld_clear_delrec] kfree(pmc:c620b700) ff02::2 [ 89.978688] [ipv6_mc_destroy_dev -> mld_clear_delrec] kfree(pmc:c620b080) ff02::2 [ 90.068433] [ipv6_mc_destroy_dev] idev->dev->name:wlan0 [ 90.073672] [ipv6_mc_down] idev->dev->name:wlan0 [ 90.078357] [ipv6_mc_destroy_dev -> mld_clear_delrec] kfree(pmc:c620bf00) ff02::2 [ 90.085992] [ipv6_mc_destroy_dev -> mld_clear_delrec] kfree(pmc:c620be80) ff02::1:ff0e:5018 [ 90.094530] [ipv6_mc_destroy_dev -> mld_clear_delrec] kfree(pmc:c620bd80) ff02::1:ff00:0 It appears that every interface up & down sequence results in adding a new ff02::2 entry to the idev->mc_tomb. Doing that over and over will obviously result in running out of memory at some point. That list isn't cleared until removing an interface. I searched and found ff02::2 to be described as "all routers" address. Any idea why is it being added over and over? Just to make my debugging complete I tried the same logging with the kernel 4.4.194 (before regression). It clears idev->mc_tomb list on every interface down so that list never grows so big. down [ 119.241112] [ipv6_mc_down] idev->dev->name:mon-phy0 [ 119.246025] [ipv6_mc_down -> mld_clear_delrec] kfree(pmc:c6c80b40) ff02::2 [ 119.649975] [ipv6_mc_down] idev->dev->name:mon-phy1 [ 119.654873] [ipv6_mc_down -> mld_clear_delrec] kfree(pmc:c6c80300) ff02::2 down [ 125.220060] [ipv6_mc_down] idev->dev->name:mon-phy0 [ 125.224969] [ipv6_mc_down -> mld_clear_delrec] kfree(pmc:c6c80540) ff02::2 [ 125.580000] [ipv6_mc_down] idev->dev->name:mon-phy1 [ 125.584900] [ipv6_mc_down -> mld_clear_delrec] kfree(pmc:c6c80180) ff02::2 down [ 128.520013] [ipv6_mc_down] idev->dev->name:mon-phy0 [ 128.524921] [ipv6_mc_down -> mld_clear_delrec] kfree(pmc:c6c800c0) ff02::2 [ 128.879994] [ipv6_mc_down] idev->dev->name:mon-phy1 [ 128.884899] [ipv6_mc_down -> mld_clear_delrec] kfree(pmc:c6c80e40) ff02::2 down [ 131.820028] [ipv6_mc_down] idev->dev->name:mon-phy0 [ 131.824934] [ipv6_mc_down -> mld_clear_delrec] kfree(pmc:c6ca53c0) ff02::2 [ 132.179992] [ipv6_mc_down] idev->dev->name:mon-phy1 [ 132.184894] [ipv6_mc_down -> mld_clear_delrec] kfree(pmc:c6ca5240) ff02::2 down [ 134.759991] [ipv6_mc_down] idev->dev->name:mon-phy0 [ 134.764901] [ipv6_mc_down -> mld_clear_delrec] kfree(pmc:c6ca5540) ff02::2 [ 135.119995] [ipv6_mc_down] idev->dev->name:mon-phy1 [ 135.124897] [ipv6_mc_down -> mld_clear_delrec] kfree(pmc:c6ca5a80) ff02::2 remove [ 140.746744] [ipv6_mc_down] idev->dev->name:wlan1 [ 140.751432] [ipv6_mc_down -> mld_clear_delrec] kfree(pmc:c6ca5480) ff02::2 [ 140.758457] [ipv6_mc_down -> mld_clear_delrec] kfree(pmc:c6c800c0) ff02::1:ff10:e018 [ 140.766367] [ipv6_mc_down -> mld_clear_delrec] kfree(pmc:c6c80e40) ff02::1:ff00:0 [ 140.791016] [ipv6_mc_down] idev->dev->name:wlan0 [ 140.795686] [ipv6_mc_down -> mld_clear_delrec] kfree(pmc:c6ca5c00) ff02::2 [ 140.802814] [ipv6_mc_down -> mld_clear_delrec] kfree(pmc:c6ca59c0) ff02::1:ff0e:5018 [ 140.810818] [ipv6_mc_down -> mld_clear_delrec] kfree(pmc:c6ca50c0) ff02::1:ff00:0 [ 140.820225] [ipv6_mc_destroy_dev] idev->dev->name:mon-phy1 [ 140.825849] [ipv6_mc_down] idev->dev->name:mon-phy1 [ 140.830785] [ipv6_mc_destroy_dev] igmp6_group_dropped(c6c80840) [ 140.902121] [ipv6_mc_destroy_dev] idev->dev->name:mon-phy0 [ 140.907622] [ipv6_mc_down] idev->dev->name:mon-phy0 [ 140.912622] [ipv6_mc_destroy_dev] igmp6_group_dropped(c6cc4a80) [ 140.983259] [ipv6_mc_destroy_dev] idev->dev->name:wlan1 [ 140.988472] [ipv6_mc_down] idev->dev->name:wlan1 [ 140.993151] [ipv6_mc_destroy_dev] igmp6_group_dropped(c72f6780) [ 141.013696] [ipv6_mc_destroy_dev] idev->dev->name:wlan0 [ 141.018933] [ipv6_mc_down] idev->dev->name:wlan0 [ 141.023722] [ipv6_mc_destroy_dev] igmp6_group_dropped(c5c4da80)