Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

23 messages, 5 authors, 2021-09-16 · open the first message on its own page

Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Martin Zaharinov <hidden>
Date: 2021-08-05 20:54:00

Hi Net dev team


Please check this error :
Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html

But not find any solution.

Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.

Server is work fine users is down/up 500+ users .
But in one moment server make spike and affect other vlans in same server .
And in accel I see many row with this error.

Is there options to find and fix this bug.

With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.


[2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected


Martin

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Greg KH <gregkh@linuxfoundation.org>
Date: 2021-08-06 04:40:09

On Thu, Aug 05, 2021 at 11:53:50PM +0300, Martin Zaharinov wrote:

Hi Net dev team


Please check this error :
Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html

But not find any solution.

Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.

Server is work fine users is down/up 500+ users .
But in one moment server make spike and affect other vlans in same server .
And in accel I see many row with this error.

Is there options to find and fix this bug.

With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.


[2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected

These are userspace error messages, not kernel messages.

What kernel version are you using?

thanks,

greg k-h

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Martin Zaharinov <hidden>
Date: 2021-08-06 05:41:01

Hi Greg

Latest kernel 5.13.8.

I try old version from 5.10 to 5.13 and its same error.

Martin

On 6 Aug 2021, at 7:40, Greg KH [off-list ref] wrote:

On Thu, Aug 05, 2021 at 11:53:50PM +0300, Martin Zaharinov wrote:

quoted

Hi Net dev team


Please check this error :
Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html

But not find any solution.

Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.

Server is work fine users is down/up 500+ users .
But in one moment server make spike and affect other vlans in same server .
And in accel I see many row with this error.

Is there options to find and fix this bug.

With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.


[2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected

These are userspace error messages, not kernel messages.

What kernel version are you using?

thanks,

greg k-h

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Martin Zaharinov <hidden>
Date: 2021-08-08 15:14:15

Add Pali Rohár,

If have any idea .

Martin

On 6 Aug 2021, at 7:40, Greg KH [off-list ref] wrote:

On Thu, Aug 05, 2021 at 11:53:50PM +0300, Martin Zaharinov wrote:

quoted

Hi Net dev team


Please check this error :
Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html

But not find any solution.

Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.

Server is work fine users is down/up 500+ users .
But in one moment server make spike and affect other vlans in same server .
And in accel I see many row with this error.

Is there options to find and fix this bug.

With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.


[2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected

These are userspace error messages, not kernel messages.

What kernel version are you using?

thanks,

greg k-h

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Pali Rohár <pali@kernel.org>
Date: 2021-08-08 15:23:23

Hello!

On Sunday 08 August 2021 18:14:09 Martin Zaharinov wrote:

Add Pali Rohár,

If have any idea .

Martin

quoted

On 6 Aug 2021, at 7:40, Greg KH [off-list ref] wrote:

On Thu, Aug 05, 2021 at 11:53:50PM +0300, Martin Zaharinov wrote:

quoted

Hi Net dev team


Please check this error :
Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html

But not find any solution.

Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.

Server is work fine users is down/up 500+ users .
But in one moment server make spike and affect other vlans in same server .

When this error started to happen? After kernel upgrade? After pppd
upgrade? Or after system upgrade? Or when more users started to
connecting?

quoted

And in accel I see many row with this error.

Is there options to find and fix this bug.

With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.


[2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected

These are userspace error messages, not kernel messages.

What kernel version are you using?

Yes, we need to know, what kernel version are you using.

quoted

thanks,

greg k-h

And also another question, what version of pppd daemon are you using?

Also, are you able to dump state of ppp channels and ppp units? It is
needed to know to which tty device, file descriptor (or socket
extension) is (or should be) particular ppp channel bounded.

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Martin Zaharinov <hidden>
Date: 2021-08-08 15:29:36

Hi Pali

Kernel 5.13.8


The problem is from kernel 5.8 > I try all major update 5.9, 5.10, 5.11 ,5.12

I use accel-pppd daemon (not pppd) .

And yes after users started to connecting .

When system boot and connect first time all user connect without any problem .
In time of work user disconnect and connect (power cut , fiber cut or other problem in network) , but in time of spike (may be make lock or other problem ) disconnect ~ 400-500 users  and affect other users. Process go to load over 100% and In statistic I see many finishing connection and many start connection. 
And in this time in log get many lines with   ioctl(PPPIOCCONNECT): Transport endpoint is not connected. After finish (unlock or other) stop to see this error and system is back to normal. And connect all disconnected users.

Martin

On 8 Aug 2021, at 18:23, Pali Rohár [off-list ref] wrote:

Hello!

On Sunday 08 August 2021 18:14:09 Martin Zaharinov wrote:

quoted

Add Pali Rohár,

If have any idea .

Martin

quoted

On 6 Aug 2021, at 7:40, Greg KH [off-list ref] wrote:

On Thu, Aug 05, 2021 at 11:53:50PM +0300, Martin Zaharinov wrote:

quoted

Hi Net dev team


Please check this error :
Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html

But not find any solution.

Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.

Server is work fine users is down/up 500+ users .
But in one moment server make spike and affect other vlans in same server .

When this error started to happen? After kernel upgrade? After pppd
upgrade? Or after system upgrade? Or when more users started to
connecting?

quoted

And in accel I see many row with this error.

Is there options to find and fix this bug.

With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.


[2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected

These are userspace error messages, not kernel messages.

What kernel version are you using?

Yes, we need to know, what kernel version are you using.

quoted

thanks,

greg k-h

And also another question, what version of pppd daemon are you using?

Also, are you able to dump state of ppp channels and ppp units? It is
needed to know to which tty device, file descriptor (or socket
extension) is (or should be) particular ppp channel bounded.

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Pali Rohár <pali@kernel.org>
Date: 2021-08-09 15:15:36

On Sunday 08 August 2021 18:29:30 Martin Zaharinov wrote:

Hi Pali

Kernel 5.13.8


The problem is from kernel 5.8 > I try all major update 5.9, 5.10, 5.11 ,5.12

I use accel-pppd daemon (not pppd) .

I'm not using accel-pppd, so cannot help here.

I would suggest to try "git bisect" kernel version which started to be
problematic for accel-pppd.

Providing state of ppp channels and ppp units could help to debug this
issue, but I'm not sure if accel-pppd has this debug feature. IIRC only
process which has ppp file descriptors can retrieve and dump this
information.

And yes after users started to connecting .

When system boot and connect first time all user connect without any problem .
In time of work user disconnect and connect (power cut , fiber cut or other problem in network) , but in time of spike (may be make lock or other problem ) disconnect ~ 400-500 users  and affect other users. Process go to load over 100% and In statistic I see many finishing connection and many start connection. 
And in this time in log get many lines with   ioctl(PPPIOCCONNECT): Transport endpoint is not connected. After finish (unlock or other) stop to see this error and system is back to normal. And connect all disconnected users.

Martin

quoted

On 8 Aug 2021, at 18:23, Pali Rohár [off-list ref] wrote:

Hello!

On Sunday 08 August 2021 18:14:09 Martin Zaharinov wrote:

quoted

Add Pali Rohár,

If have any idea .

Martin

quoted

On 6 Aug 2021, at 7:40, Greg KH [off-list ref] wrote:

On Thu, Aug 05, 2021 at 11:53:50PM +0300, Martin Zaharinov wrote:

quoted

Hi Net dev team


Please check this error :
Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html

But not find any solution.

Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.

Server is work fine users is down/up 500+ users .
But in one moment server make spike and affect other vlans in same server .

When this error started to happen? After kernel upgrade? After pppd
upgrade? Or after system upgrade? Or when more users started to
connecting?

quoted

And in accel I see many row with this error.

Is there options to find and fix this bug.

With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.


[2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected

These are userspace error messages, not kernel messages.

What kernel version are you using?

Yes, we need to know, what kernel version are you using.

quoted

thanks,

greg k-h

And also another question, what version of pppd daemon are you using?

Also, are you able to dump state of ppp channels and ppp units? It is
needed to know to which tty device, file descriptor (or socket
extension) is (or should be) particular ppp channel bounded.

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Martin Zaharinov <hidden>
Date: 2021-08-10 18:27:20

Add Guillaume Nault

On 9 Aug 2021, at 18:15, Pali Rohár [off-list ref] wrote:

On Sunday 08 August 2021 18:29:30 Martin Zaharinov wrote:

quoted

Hi Pali

Kernel 5.13.8


The problem is from kernel 5.8 > I try all major update 5.9, 5.10, 5.11 ,5.12

I use accel-pppd daemon (not pppd) .

I'm not using accel-pppd, so cannot help here.

I would suggest to try "git bisect" kernel version which started to be
problematic for accel-pppd.

Providing state of ppp channels and ppp units could help to debug this
issue, but I'm not sure if accel-pppd has this debug feature. IIRC only
process which has ppp file descriptors can retrieve and dump this
information.

quoted

And yes after users started to connecting .

When system boot and connect first time all user connect without any problem .
In time of work user disconnect and connect (power cut , fiber cut or other problem in network) , but in time of spike (may be make lock or other problem ) disconnect ~ 400-500 users  and affect other users. Process go to load over 100% and In statistic I see many finishing connection and many start connection. 
And in this time in log get many lines with   ioctl(PPPIOCCONNECT): Transport endpoint is not connected. After finish (unlock or other) stop to see this error and system is back to normal. And connect all disconnected users.

Martin

quoted

On 8 Aug 2021, at 18:23, Pali Rohár [off-list ref] wrote:

Hello!

On Sunday 08 August 2021 18:14:09 Martin Zaharinov wrote:

quoted

Add Pali Rohár,

If have any idea .

Martin

quoted

On 6 Aug 2021, at 7:40, Greg KH [off-list ref] wrote:

On Thu, Aug 05, 2021 at 11:53:50PM +0300, Martin Zaharinov wrote:

quoted

Hi Net dev team


Please check this error :
Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html

But not find any solution.

Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.

Server is work fine users is down/up 500+ users .
But in one moment server make spike and affect other vlans in same server .

When this error started to happen? After kernel upgrade? After pppd
upgrade? Or after system upgrade? Or when more users started to
connecting?

quoted

And in accel I see many row with this error.

Is there options to find and fix this bug.

With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.


[2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected

These are userspace error messages, not kernel messages.

What kernel version are you using?

Yes, we need to know, what kernel version are you using.

quoted

thanks,

greg k-h

And also another question, what version of pppd daemon are you using?

Also, are you able to dump state of ppp channels and ppp units? It is
needed to know to which tty device, file descriptor (or socket
extension) is (or should be) particular ppp channel bounded.

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Martin Zaharinov <hidden>
Date: 2021-08-11 11:10:41

And one more that see.

Problem is come when accel start finishing sessions,
Now in server have 2k users and restart on one of vlans 3 Olt with 400 users and affect other vlans ,
And problem is start when start destroying dead sessions from vlan with 3 Olt and this affect all other vlans.
May be kernel destroy old session slow and entrained other users by locking other sessions.
is there a way to speed up the closing of stopped/dead sessions.

Martin

On 9 Aug 2021, at 18:15, Pali Rohár [off-list ref] wrote:

On Sunday 08 August 2021 18:29:30 Martin Zaharinov wrote:

quoted

Hi Pali

Kernel 5.13.8


The problem is from kernel 5.8 > I try all major update 5.9, 5.10, 5.11 ,5.12

I use accel-pppd daemon (not pppd) .

I'm not using accel-pppd, so cannot help here.

I would suggest to try "git bisect" kernel version which started to be
problematic for accel-pppd.

Providing state of ppp channels and ppp units could help to debug this
issue, but I'm not sure if accel-pppd has this debug feature. IIRC only
process which has ppp file descriptors can retrieve and dump this
information.

quoted

And yes after users started to connecting .

When system boot and connect first time all user connect without any problem .
In time of work user disconnect and connect (power cut , fiber cut or other problem in network) , but in time of spike (may be make lock or other problem ) disconnect ~ 400-500 users  and affect other users. Process go to load over 100% and In statistic I see many finishing connection and many start connection. 
And in this time in log get many lines with   ioctl(PPPIOCCONNECT): Transport endpoint is not connected. After finish (unlock or other) stop to see this error and system is back to normal. And connect all disconnected users.

Martin

quoted

On 8 Aug 2021, at 18:23, Pali Rohár [off-list ref] wrote:

Hello!

On Sunday 08 August 2021 18:14:09 Martin Zaharinov wrote:

quoted

Add Pali Rohár,

If have any idea .

Martin

quoted

On 6 Aug 2021, at 7:40, Greg KH [off-list ref] wrote:

On Thu, Aug 05, 2021 at 11:53:50PM +0300, Martin Zaharinov wrote:

quoted

Hi Net dev team


Please check this error :
Last time I write for this problem : https://www.spinics.net/lists/netdev/msg707513.html

But not find any solution.

Config of server is : Bonding port channel (LACP)  > Accel PPP server > Huawei switch.

Server is work fine users is down/up 500+ users .
But in one moment server make spike and affect other vlans in same server .

When this error started to happen? After kernel upgrade? After pppd
upgrade? Or after system upgrade? Or when more users started to
connecting?

quoted

And in accel I see many row with this error.

Is there options to find and fix this bug.

With accel team I discus this problem  and they claim it is kernel bug and need to find solution with Kernel dev team.


[2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected

These are userspace error messages, not kernel messages.

What kernel version are you using?

Yes, we need to know, what kernel version are you using.

quoted

thanks,

greg k-h

And also another question, what version of pppd daemon are you using?

Also, are you able to dump state of ppp channels and ppp units? It is
needed to know to which tty device, file descriptor (or socket
extension) is (or should be) particular ppp channel bounded.

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Guillaume Nault <hidden>
Date: 2021-08-11 16:40:09

On Tue, Aug 10, 2021 at 09:27:14PM +0300, Martin Zaharinov wrote:

Add Guillaume Nault

quoted

On 9 Aug 2021, at 18:15, Pali Rohár [off-list ref] wrote:

On Sunday 08 August 2021 18:29:30 Martin Zaharinov wrote:

quoted

[2021-08-05 13:52:05.294] vlan912: 24b205903d09718e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.298] vlan912: 24b205903d097162: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:05.626] vlan641: 24b205903d09711b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:11.000] vlan912: 24b205903d097105: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:17.852] vlan912: 24b205903d0971ae: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:21.113] vlan641: 24b205903d09715b: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:27.963] vlan912: 24b205903d09718d: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.249] vlan496: 24b205903d097184: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:30.992] vlan420: 24b205903d09718a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:33.937] vlan640: 24b205903d0971cd: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.032] vlan912: 24b205903d097182: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:40.420] vlan912: 24b205903d0971d5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan912: 24b205903d09713a: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:42.799] vlan614: 24b205903d0971e5: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.102] vlan912: 24b205903d097190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097153: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.850] vlan479: 24b205903d097141: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.852] vlan912: 24b205903d097198: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:43.977] vlan637: 24b205903d097148: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-08-05 13:52:44.528] vlan637: 24b205903d0971c3: ioctl(PPPIOCCONNECT): Transport endpoint is not connected

The PPPIOCCONNECT ioctl returns -ENOTCONN if the ppp channel has been
unregistered.

From a user space point of view, this means that accel-ppp establishes
PPPoE sessions, starts negociating PPP connection parameters on top of
them (LCP and authentication) and finally the PPPoE sessions get
disconnected before accel-ppp connects them to ppp units (units are
roughly the "pppX" network devices).

Unregistration of PPPoE channels can happen for the following reasons:

  * Changing some parameters of the network interface used by the
    PPPoE connection: MAC address, MTU, bringing the device down.

  * Reception of a PADT (PPPoE disconnection message sent from the peer).

  * Closing the PPPoE socket.

  * Re-connecting a PPPoE socket with a different session ID (this
    unregisters the previous channel and creates a new one, so that
    shouldn't be the problem you're facing here).

Given that this seems to affect all PPPoE connections, I guess
something happened to the underlying network interface (1st bullet
point).

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Guillaume Nault <hidden>
Date: 2021-08-11 16:48:42

On Wed, Aug 11, 2021 at 02:10:32PM +0300, Martin Zaharinov wrote:

And one more that see.

Problem is come when accel start finishing sessions,
Now in server have 2k users and restart on one of vlans 3 Olt with 400 users and affect other vlans ,
And problem is start when start destroying dead sessions from vlan with 3 Olt and this affect all other vlans.
May be kernel destroy old session slow and entrained other users by locking other sessions.
is there a way to speed up the closing of stopped/dead sessions.

What are the CPU stats when that happen? Is it users space or kernel
space that keeps it busy?

One easy way to check is to run "mpstat 1" for a few seconds when the
problem occurs.

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Martin Zaharinov <hidden>
Date: 2021-09-07 06:16:24

Hi 
Sorry for delay but not easy to catch moment .


See this is mpstatl 1 :

Linux 5.14.1 (demobng) 	09/07/21 	_x86_64_	(12 CPU)

11:12:16     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
11:12:17     all    0.17    0.00    6.66    0.00    0.00    4.13    0.00    0.00    0.00   89.05
11:12:18     all    0.25    0.00    8.36    0.00    0.00    4.88    0.00    0.00    0.00   86.51
11:12:19     all    0.26    0.00    9.62    0.00    0.00    3.91    0.00    0.00    0.00   86.21
11:12:20     all    0.85    0.00    6.00    0.00    0.00    4.31    0.00    0.00    0.00   88.84
11:12:21     all    0.08    0.00    4.45    0.00    0.00    4.79    0.00    0.00    0.00   90.67
11:12:22     all    0.17    0.00    9.50    0.00    0.00    4.58    0.00    0.00    0.00   85.75
11:12:23     all    0.00    0.00    6.92    0.00    0.00    2.48    0.00    0.00    0.00   90.61
11:12:24     all    0.17    0.00    5.45    0.00    0.00    4.27    0.00    0.00    0.00   90.11
11:12:25     all    0.25    0.00    5.38    0.00    0.00    4.79    0.00    0.00    0.00   89.58
11:12:26     all    0.60    0.00    1.45    0.00    0.00    2.65    0.00    0.00    0.00   95.30
11:12:27     all    0.42    0.00    6.91    0.00    0.00    4.47    0.00    0.00    0.00   88.20
11:12:28     all    0.00    0.00    6.75    0.00    0.00    4.18    0.00    0.00    0.00   89.07
11:12:29     all    0.17    0.00    3.52    0.00    0.00    5.11    0.00    0.00    0.00   91.20
11:12:30     all    1.45    0.00   10.14    0.00    0.00    3.49    0.00    0.00    0.00   84.92
11:12:31     all    0.09    0.00    5.11    0.00    0.00    4.77    0.00    0.00    0.00   90.03
11:12:32     all    0.25    0.00    3.11    0.00    0.00    4.46    0.00    0.00    0.00   92.17
Average:     all    0.32    0.00    6.21    0.00    0.00    4.21    0.00    0.00    0.00   89.26


I attache and one screenshot from perf top (Screenshot is send on preview mail)

And I see in lsmod 

pppoe                  20480  8198
pppox                  16384  1 pppoe
ppp_generic            45056  16364 pppox,pppoe
slhc                   16384  1 ppp_generic

To slow remove pppoe session .

And from log : 

[2021-09-07 11:01:11.129] vlan3020: ebdd1c5d8b5900f6: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:01:53.621] vlan643: ebdd1c5d8b59014e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:00.359] vlan1616: ebdd1c5d8b590195: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:05.859] vlan3020: ebdd1c5d8b5900d8: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:08.258] vlan3005: ebdd1c5d8b590190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:13.820] vlan643: ebdd1c5d8b590152: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:15.839] vlan727: ebdd1c5d8b590144: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:20.139] vlan1693: ebdd1c5d8b59019f: ioctl(PPPIOCCONNECT): Transport endpoint is not connected

On 11 Aug 2021, at 19:48, Guillaume Nault [off-list ref] wrote:

On Wed, Aug 11, 2021 at 02:10:32PM +0300, Martin Zaharinov wrote:

quoted

And one more that see.

Problem is come when accel start finishing sessions,
Now in server have 2k users and restart on one of vlans 3 Olt with 400 users and affect other vlans ,
And problem is start when start destroying dead sessions from vlan with 3 Olt and this affect all other vlans.
May be kernel destroy old session slow and entrained other users by locking other sessions.
is there a way to speed up the closing of stopped/dead sessions.

What are the CPU stats when that happen? Is it users space or kernel
space that keeps it busy?

One easy way to check is to run "mpstat 1" for a few seconds when the
problem occurs.

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Martin Zaharinov <hidden>
Date: 2021-09-07 06:42:20

Perf top from text


PerfTop:   28391 irqs/sec  kernel:98.0%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    17.01%  [nf_conntrack]           [k] nf_ct_iterate_cleanup
     9.73%  [kernel]                 [k] mutex_spin_on_owner
     9.07%  [pppoe]                  [k] pppoe_rcv
     2.77%  [nf_nat]                 [k] device_cmp
     1.66%  [kernel]                 [k] osq_lock
     1.65%  [kernel]                 [k] _raw_spin_lock
     1.61%  [kernel]                 [k] __local_bh_enable_ip
     1.35%  [nf_nat]                 [k] inet_cmp
     1.30%  [kernel]                 [k] __netif_receive_skb_core.constprop.0
     1.16%  [kernel]                 [k] menu_select
     0.99%  [kernel]                 [k] cpuidle_enter_state
     0.96%  [ixgbe]                  [k] ixgbe_clean_rx_irq
     0.86%  [kernel]                 [k] __dev_queue_xmit
     0.70%  [kernel]                 [k] __cond_resched
     0.69%  [sch_cake]               [k] cake_dequeue
     0.67%  [nf_tables]              [k] nft_do_chain
     0.63%  [kernel]                 [k] rcu_all_qs
     0.61%  [kernel]                 [k] fib_table_lookup
     0.57%  [kernel]                 [k] __schedule
     0.57%  [kernel]                 [k] skb_release_data
     0.54%  [kernel]                 [k] sched_clock
     0.54%  [kernel]                 [k] __copy_skb_header
     0.53%  [kernel]                 [k] dev_queue_xmit_nit
     0.53%  [kernel]                 [k] _raw_spin_lock_irqsave
     0.50%  [kernel]                 [k] kmem_cache_free
     0.48%  libfrr.so.0.0.0          [.] 0x00000000000ce970
     0.47%  [ixgbe]                  [k] ixgbe_clean_tx_irq
     0.45%  [kernel]                 [k] timerqueue_add
     0.45%  [kernel]                 [k] lapic_next_deadline
     0.45%  [kernel]                 [k] csum_partial_copy_generic
     0.44%  [nf_flow_table]          [k] nf_flow_offload_ip_hook
     0.44%  [kernel]                 [k] kmem_cache_alloc
     0.44%  [nf_conntrack]           [k] nf_conntrack_lock

On 7 Sep 2021, at 9:16, Martin Zaharinov [off-list ref] wrote:

Hi 
Sorry for delay but not easy to catch moment .


See this is mpstatl 1 :

Linux 5.14.1 (demobng) 	09/07/21 	_x86_64_	(12 CPU)

11:12:16     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
11:12:17     all    0.17    0.00    6.66    0.00    0.00    4.13    0.00    0.00    0.00   89.05
11:12:18     all    0.25    0.00    8.36    0.00    0.00    4.88    0.00    0.00    0.00   86.51
11:12:19     all    0.26    0.00    9.62    0.00    0.00    3.91    0.00    0.00    0.00   86.21
11:12:20     all    0.85    0.00    6.00    0.00    0.00    4.31    0.00    0.00    0.00   88.84
11:12:21     all    0.08    0.00    4.45    0.00    0.00    4.79    0.00    0.00    0.00   90.67
11:12:22     all    0.17    0.00    9.50    0.00    0.00    4.58    0.00    0.00    0.00   85.75
11:12:23     all    0.00    0.00    6.92    0.00    0.00    2.48    0.00    0.00    0.00   90.61
11:12:24     all    0.17    0.00    5.45    0.00    0.00    4.27    0.00    0.00    0.00   90.11
11:12:25     all    0.25    0.00    5.38    0.00    0.00    4.79    0.00    0.00    0.00   89.58
11:12:26     all    0.60    0.00    1.45    0.00    0.00    2.65    0.00    0.00    0.00   95.30
11:12:27     all    0.42    0.00    6.91    0.00    0.00    4.47    0.00    0.00    0.00   88.20
11:12:28     all    0.00    0.00    6.75    0.00    0.00    4.18    0.00    0.00    0.00   89.07
11:12:29     all    0.17    0.00    3.52    0.00    0.00    5.11    0.00    0.00    0.00   91.20
11:12:30     all    1.45    0.00   10.14    0.00    0.00    3.49    0.00    0.00    0.00   84.92
11:12:31     all    0.09    0.00    5.11    0.00    0.00    4.77    0.00    0.00    0.00   90.03
11:12:32     all    0.25    0.00    3.11    0.00    0.00    4.46    0.00    0.00    0.00   92.17
Average:     all    0.32    0.00    6.21    0.00    0.00    4.21    0.00    0.00    0.00   89.26


I attache and one screenshot from perf top (Screenshot is send on preview mail)

And I see in lsmod 

pppoe                  20480  8198
pppox                  16384  1 pppoe
ppp_generic            45056  16364 pppox,pppoe
slhc                   16384  1 ppp_generic

To slow remove pppoe session .

And from log : 

[2021-09-07 11:01:11.129] vlan3020: ebdd1c5d8b5900f6: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:01:53.621] vlan643: ebdd1c5d8b59014e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:00.359] vlan1616: ebdd1c5d8b590195: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:05.859] vlan3020: ebdd1c5d8b5900d8: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:08.258] vlan3005: ebdd1c5d8b590190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:13.820] vlan643: ebdd1c5d8b590152: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:15.839] vlan727: ebdd1c5d8b590144: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:20.139] vlan1693: ebdd1c5d8b59019f: ioctl(PPPIOCCONNECT): Transport endpoint is not connected

quoted

On 11 Aug 2021, at 19:48, Guillaume Nault [off-list ref] wrote:

On Wed, Aug 11, 2021 at 02:10:32PM +0300, Martin Zaharinov wrote:

quoted

And one more that see.

Problem is come when accel start finishing sessions,
Now in server have 2k users and restart on one of vlans 3 Olt with 400 users and affect other vlans ,
And problem is start when start destroying dead sessions from vlan with 3 Olt and this affect all other vlans.
May be kernel destroy old session slow and entrained other users by locking other sessions.
is there a way to speed up the closing of stopped/dead sessions.

What are the CPU stats when that happen? Is it users space or kernel
space that keeps it busy?

One easy way to check is to run "mpstat 1" for a few seconds when the
problem occurs.

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Martin Zaharinov <hidden>
Date: 2021-09-11 06:26:52

Hi Guillaume

Main problem is overload of service because have many finishing ppp (customer) last two day down from 40-50 to 100-200 users and make problem when is happen if try to type : ip a wait 10-20 sec to start list interface .
But how to find where is a problem any locking or other.
And is there options to make fast remove ppp interface from kernel to reduce this load.


Martin

On 7 Sep 2021, at 9:42, Martin Zaharinov [off-list ref] wrote:

Perf top from text


PerfTop:   28391 irqs/sec  kernel:98.0%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

   17.01%  [nf_conntrack]           [k] nf_ct_iterate_cleanup
    9.73%  [kernel]                 [k] mutex_spin_on_owner
    9.07%  [pppoe]                  [k] pppoe_rcv
    2.77%  [nf_nat]                 [k] device_cmp
    1.66%  [kernel]                 [k] osq_lock
    1.65%  [kernel]                 [k] _raw_spin_lock
    1.61%  [kernel]                 [k] __local_bh_enable_ip
    1.35%  [nf_nat]                 [k] inet_cmp
    1.30%  [kernel]                 [k] __netif_receive_skb_core.constprop.0
    1.16%  [kernel]                 [k] menu_select
    0.99%  [kernel]                 [k] cpuidle_enter_state
    0.96%  [ixgbe]                  [k] ixgbe_clean_rx_irq
    0.86%  [kernel]                 [k] __dev_queue_xmit
    0.70%  [kernel]                 [k] __cond_resched
    0.69%  [sch_cake]               [k] cake_dequeue
    0.67%  [nf_tables]              [k] nft_do_chain
    0.63%  [kernel]                 [k] rcu_all_qs
    0.61%  [kernel]                 [k] fib_table_lookup
    0.57%  [kernel]                 [k] __schedule
    0.57%  [kernel]                 [k] skb_release_data
    0.54%  [kernel]                 [k] sched_clock
    0.54%  [kernel]                 [k] __copy_skb_header
    0.53%  [kernel]                 [k] dev_queue_xmit_nit
    0.53%  [kernel]                 [k] _raw_spin_lock_irqsave
    0.50%  [kernel]                 [k] kmem_cache_free
    0.48%  libfrr.so.0.0.0          [.] 0x00000000000ce970
    0.47%  [ixgbe]                  [k] ixgbe_clean_tx_irq
    0.45%  [kernel]                 [k] timerqueue_add
    0.45%  [kernel]                 [k] lapic_next_deadline
    0.45%  [kernel]                 [k] csum_partial_copy_generic
    0.44%  [nf_flow_table]          [k] nf_flow_offload_ip_hook
    0.44%  [kernel]                 [k] kmem_cache_alloc
    0.44%  [nf_conntrack]           [k] nf_conntrack_lock

quoted

On 7 Sep 2021, at 9:16, Martin Zaharinov [off-list ref] wrote:

Hi 
Sorry for delay but not easy to catch moment .


See this is mpstatl 1 :

Linux 5.14.1 (demobng) 	09/07/21 	_x86_64_	(12 CPU)

11:12:16     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
11:12:17     all    0.17    0.00    6.66    0.00    0.00    4.13    0.00    0.00    0.00   89.05
11:12:18     all    0.25    0.00    8.36    0.00    0.00    4.88    0.00    0.00    0.00   86.51
11:12:19     all    0.26    0.00    9.62    0.00    0.00    3.91    0.00    0.00    0.00   86.21
11:12:20     all    0.85    0.00    6.00    0.00    0.00    4.31    0.00    0.00    0.00   88.84
11:12:21     all    0.08    0.00    4.45    0.00    0.00    4.79    0.00    0.00    0.00   90.67
11:12:22     all    0.17    0.00    9.50    0.00    0.00    4.58    0.00    0.00    0.00   85.75
11:12:23     all    0.00    0.00    6.92    0.00    0.00    2.48    0.00    0.00    0.00   90.61
11:12:24     all    0.17    0.00    5.45    0.00    0.00    4.27    0.00    0.00    0.00   90.11
11:12:25     all    0.25    0.00    5.38    0.00    0.00    4.79    0.00    0.00    0.00   89.58
11:12:26     all    0.60    0.00    1.45    0.00    0.00    2.65    0.00    0.00    0.00   95.30
11:12:27     all    0.42    0.00    6.91    0.00    0.00    4.47    0.00    0.00    0.00   88.20
11:12:28     all    0.00    0.00    6.75    0.00    0.00    4.18    0.00    0.00    0.00   89.07
11:12:29     all    0.17    0.00    3.52    0.00    0.00    5.11    0.00    0.00    0.00   91.20
11:12:30     all    1.45    0.00   10.14    0.00    0.00    3.49    0.00    0.00    0.00   84.92
11:12:31     all    0.09    0.00    5.11    0.00    0.00    4.77    0.00    0.00    0.00   90.03
11:12:32     all    0.25    0.00    3.11    0.00    0.00    4.46    0.00    0.00    0.00   92.17
Average:     all    0.32    0.00    6.21    0.00    0.00    4.21    0.00    0.00    0.00   89.26


I attache and one screenshot from perf top (Screenshot is send on preview mail)

And I see in lsmod 

pppoe                  20480  8198
pppox                  16384  1 pppoe
ppp_generic            45056  16364 pppox,pppoe
slhc                   16384  1 ppp_generic

To slow remove pppoe session .

And from log : 

[2021-09-07 11:01:11.129] vlan3020: ebdd1c5d8b5900f6: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:01:53.621] vlan643: ebdd1c5d8b59014e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:00.359] vlan1616: ebdd1c5d8b590195: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:05.859] vlan3020: ebdd1c5d8b5900d8: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:08.258] vlan3005: ebdd1c5d8b590190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:13.820] vlan643: ebdd1c5d8b590152: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:15.839] vlan727: ebdd1c5d8b590144: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:20.139] vlan1693: ebdd1c5d8b59019f: ioctl(PPPIOCCONNECT): Transport endpoint is not connected

quoted

On 11 Aug 2021, at 19:48, Guillaume Nault [off-list ref] wrote:

On Wed, Aug 11, 2021 at 02:10:32PM +0300, Martin Zaharinov wrote:

quoted

And one more that see.

Problem is come when accel start finishing sessions,
Now in server have 2k users and restart on one of vlans 3 Olt with 400 users and affect other vlans ,
And problem is start when start destroying dead sessions from vlan with 3 Olt and this affect all other vlans.
May be kernel destroy old session slow and entrained other users by locking other sessions.
is there a way to speed up the closing of stopped/dead sessions.

What are the CPU stats when that happen? Is it users space or kernel
space that keeps it busy?

One easy way to check is to run "mpstat 1" for a few seconds when the
problem occurs.

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Martin Zaharinov <hidden>
Date: 2021-09-14 06:17:01

Hi Nault

See this stats :

Linux 5.14.2 (testb)   09/14/21        _x86_64_        (12 CPU)

11:33:44     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
11:33:45     all    1.75    0.00   18.85    0.00    0.00    5.00    0.00    0.00    0.00   74.40
11:33:46     all    1.74    0.00   17.88    0.00    0.00    4.72    0.00    0.00    0.00   75.66
11:33:47     all    2.23    0.00   17.62    0.00    0.00    5.05    0.00    0.00    0.00   75.10
11:33:48     all    1.82    0.00   13.64    0.00    0.00    5.70    0.00    0.00    0.00   78.84
11:33:49     all    1.50    0.00   13.46    0.00    0.00    5.15    0.00    0.00    0.00   79.90
11:33:50     all    3.06    0.00   13.96    0.00    0.00    4.79    0.00    0.00    0.00   78.20
11:33:51     all    1.40    0.00   16.53    0.00    0.00    5.21    0.00    0.00    0.00   76.86
11:33:52     all    4.43    0.00   19.44    0.00    0.00    6.56    0.00    0.00    0.00   69.57
11:33:53     all    1.51    0.00   16.40    0.00    0.00    4.77    0.00    0.00    0.00   77.32
11:33:54     all    1.51    0.00   16.55    0.00    0.00    4.71    0.00    0.00    0.00   77.23
11:33:55     all    1.00    0.00   13.21    0.00    0.00    5.90    0.00    0.00    0.00   79.90
Average:     all    2.00    0.00   16.14    0.00    0.00    5.23    0.00    0.00    0.00   76.63


  PerfTop:   28046 irqs/sec  kernel:96.3%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    23.37%  [nf_conntrack]           [k] nf_ct_iterate_cleanup
    17.76%  [kernel]                 [k] mutex_spin_on_owner
     9.47%  [pppoe]                  [k] pppoe_rcv
     7.71%  [kernel]                 [k] osq_lock
     2.77%  [nf_nat]                 [k] inet_cmp
     2.59%  [nf_nat]                 [k] device_cmp
     2.55%  [kernel]                 [k] __local_bh_enable_ip
     2.04%  [kernel]                 [k] _raw_spin_lock
     1.23%  [kernel]                 [k] __cond_resched
     1.16%  [kernel]                 [k] rcu_all_qs
     1.13%  libfrr.so.0.0.0          [.] 0x00000000000ce970
     0.79%  [nf_conntrack]           [k] nf_conntrack_lock
     0.75%  libfrr.so.0.0.0          [.] 0x00000000000ce94e
     0.53%  [kernel]                 [k] __netif_receive_skb_core.constprop.0
     0.46%  [kernel]                 [k] fib_table_lookup
     0.46%  [ip_tables]              [k] ipt_do_table
     0.45%  [ixgbe]                  [k] ixgbe_clean_rx_irq
     0.37%  [kernel]                 [k] __dev_queue_xmit
     0.34%  [nf_conntrack]           [k] __nf_conntrack_find_get.isra.0
     0.33%  [ixgbe]                  [k] ixgbe_clean_tx_irq
     0.30%  [kernel]                 [k] menu_select
     0.25%  [kernel]                 [k] vlan_do_receive
     0.21%  [kernel]                 [k] ip_finish_output2
     0.21%  [ixgbe]                  [k] ixgbe_poll
     0.20%  [kernel]                 [k] _raw_spin_lock_irqsave
     0.19%  [kernel]                 [k] get_rps_cpu
     0.19%  libc.so.6                [.] 0x0000000000186afa
     0.19%  [kernel]                 [k] queued_read_lock_slowpath
     0.19%  [kernel]                 [k] do_poll.constprop.0
     0.19%  [kernel]                 [k] cpuidle_enter_state
     0.18%  [kernel]                 [k] dev_hard_start_xmit
     0.18%  [kernel]                 [k] ___slab_alloc.constprop.0
     0.17%  zebra                    [.] 0x00000000000b9271
     0.16%  [kernel]                 [k] csum_partial_copy_generic
     0.16%  zebra                    [.] 0x00000000000b91f1
     0.16%  [kernel]                 [k] page_frag_free
     0.16%  [kernel]                 [k] kmem_cache_alloc
     0.15%  [kernel]                 [k] __skb_flow_dissect
     0.15%  [kernel]                 [k] sched_clock
     0.15%  libc.so.6                [.] 0x00000000000965a2
     0.15%  [kernel]                 [k] kmem_cache_free_bulk.part.0
     0.15%  [pppoe]                  [k] pppoe_flush_dev
     0.15%  [ixgbe]                  [k] ixgbe_tx_map
     0.14%  [kernel]                 [k] _raw_spin_lock_bh
     0.14%  [kernel]                 [k] fib_table_flush
     0.14%  [kernel]                 [k] native_irq_return_iret
     0.14%  [kernel]                 [k] __dev_xmit_skb
     0.13%  [kernel]                 [k] nf_hook_slow
     0.13%  [kernel]                 [k] fib_lookup_good_nhc
     0.12%  [kernel]                 [k] __fget_files
     0.12%  [kernel]                 [k] process_backlog
     0.12%  [xt_dtvqos]              [k] 0x00000000000008d1
     0.12%  [kernel]                 [k] __list_del_entry_valid
     0.12%  [kernel]                 [k] skb_release_data
     0.12%  [kernel]                 [k] ip_route_input_slow
     0.11%  [kernel]                 [k] netif_skb_features
     0.11%  [kernel]                 [k] sock_poll
     0.11%  [kernel]                 [k] __schedule
     0.11%  [kernel]                 [k] __softirqentry_text_start


And on time of problem when try to write : ip a 
to list interface wait 15-20 sec i finaly have options to simulate but users is angry when down internet.

In case need to know why system is overloaded when deconfig ppp interface.


Best regards,
Martin

On 11 Sep 2021, at 9:26, Martin Zaharinov [off-list ref] wrote:

Hi Guillaume

Main problem is overload of service because have many finishing ppp (customer) last two day down from 40-50 to 100-200 users and make problem when is happen if try to type : ip a wait 10-20 sec to start list interface .
But how to find where is a problem any locking or other.
And is there options to make fast remove ppp interface from kernel to reduce this load.


Martin

quoted

On 7 Sep 2021, at 9:42, Martin Zaharinov [off-list ref] wrote:

Perf top from text


PerfTop:   28391 irqs/sec  kernel:98.0%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

  17.01%  [nf_conntrack]           [k] nf_ct_iterate_cleanup
   9.73%  [kernel]                 [k] mutex_spin_on_owner
   9.07%  [pppoe]                  [k] pppoe_rcv
   2.77%  [nf_nat]                 [k] device_cmp
   1.66%  [kernel]                 [k] osq_lock
   1.65%  [kernel]                 [k] _raw_spin_lock
   1.61%  [kernel]                 [k] __local_bh_enable_ip
   1.35%  [nf_nat]                 [k] inet_cmp
   1.30%  [kernel]                 [k] __netif_receive_skb_core.constprop.0
   1.16%  [kernel]                 [k] menu_select
   0.99%  [kernel]                 [k] cpuidle_enter_state
   0.96%  [ixgbe]                  [k] ixgbe_clean_rx_irq
   0.86%  [kernel]                 [k] __dev_queue_xmit
   0.70%  [kernel]                 [k] __cond_resched
   0.69%  [sch_cake]               [k] cake_dequeue
   0.67%  [nf_tables]              [k] nft_do_chain
   0.63%  [kernel]                 [k] rcu_all_qs
   0.61%  [kernel]                 [k] fib_table_lookup
   0.57%  [kernel]                 [k] __schedule
   0.57%  [kernel]                 [k] skb_release_data
   0.54%  [kernel]                 [k] sched_clock
   0.54%  [kernel]                 [k] __copy_skb_header
   0.53%  [kernel]                 [k] dev_queue_xmit_nit
   0.53%  [kernel]                 [k] _raw_spin_lock_irqsave
   0.50%  [kernel]                 [k] kmem_cache_free
   0.48%  libfrr.so.0.0.0          [.] 0x00000000000ce970
   0.47%  [ixgbe]                  [k] ixgbe_clean_tx_irq
   0.45%  [kernel]                 [k] timerqueue_add
   0.45%  [kernel]                 [k] lapic_next_deadline
   0.45%  [kernel]                 [k] csum_partial_copy_generic
   0.44%  [nf_flow_table]          [k] nf_flow_offload_ip_hook
   0.44%  [kernel]                 [k] kmem_cache_alloc
   0.44%  [nf_conntrack]           [k] nf_conntrack_lock

quoted

On 7 Sep 2021, at 9:16, Martin Zaharinov [off-list ref] wrote:

Hi 
Sorry for delay but not easy to catch moment .


See this is mpstatl 1 :

Linux 5.14.1 (demobng) 	09/07/21 	_x86_64_	(12 CPU)

11:12:16     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
11:12:17     all    0.17    0.00    6.66    0.00    0.00    4.13    0.00    0.00    0.00   89.05
11:12:18     all    0.25    0.00    8.36    0.00    0.00    4.88    0.00    0.00    0.00   86.51
11:12:19     all    0.26    0.00    9.62    0.00    0.00    3.91    0.00    0.00    0.00   86.21
11:12:20     all    0.85    0.00    6.00    0.00    0.00    4.31    0.00    0.00    0.00   88.84
11:12:21     all    0.08    0.00    4.45    0.00    0.00    4.79    0.00    0.00    0.00   90.67
11:12:22     all    0.17    0.00    9.50    0.00    0.00    4.58    0.00    0.00    0.00   85.75
11:12:23     all    0.00    0.00    6.92    0.00    0.00    2.48    0.00    0.00    0.00   90.61
11:12:24     all    0.17    0.00    5.45    0.00    0.00    4.27    0.00    0.00    0.00   90.11
11:12:25     all    0.25    0.00    5.38    0.00    0.00    4.79    0.00    0.00    0.00   89.58
11:12:26     all    0.60    0.00    1.45    0.00    0.00    2.65    0.00    0.00    0.00   95.30
11:12:27     all    0.42    0.00    6.91    0.00    0.00    4.47    0.00    0.00    0.00   88.20
11:12:28     all    0.00    0.00    6.75    0.00    0.00    4.18    0.00    0.00    0.00   89.07
11:12:29     all    0.17    0.00    3.52    0.00    0.00    5.11    0.00    0.00    0.00   91.20
11:12:30     all    1.45    0.00   10.14    0.00    0.00    3.49    0.00    0.00    0.00   84.92
11:12:31     all    0.09    0.00    5.11    0.00    0.00    4.77    0.00    0.00    0.00   90.03
11:12:32     all    0.25    0.00    3.11    0.00    0.00    4.46    0.00    0.00    0.00   92.17
Average:     all    0.32    0.00    6.21    0.00    0.00    4.21    0.00    0.00    0.00   89.26


I attache and one screenshot from perf top (Screenshot is send on preview mail)

And I see in lsmod 

pppoe                  20480  8198
pppox                  16384  1 pppoe
ppp_generic            45056  16364 pppox,pppoe
slhc                   16384  1 ppp_generic

To slow remove pppoe session .

And from log : 

[2021-09-07 11:01:11.129] vlan3020: ebdd1c5d8b5900f6: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:01:53.621] vlan643: ebdd1c5d8b59014e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:00.359] vlan1616: ebdd1c5d8b590195: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:05.859] vlan3020: ebdd1c5d8b5900d8: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:08.258] vlan3005: ebdd1c5d8b590190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:13.820] vlan643: ebdd1c5d8b590152: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:15.839] vlan727: ebdd1c5d8b590144: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:20.139] vlan1693: ebdd1c5d8b59019f: ioctl(PPPIOCCONNECT): Transport endpoint is not connected

quoted

On 11 Aug 2021, at 19:48, Guillaume Nault [off-list ref] wrote:

On Wed, Aug 11, 2021 at 02:10:32PM +0300, Martin Zaharinov wrote:

quoted

And one more that see.

Problem is come when accel start finishing sessions,
Now in server have 2k users and restart on one of vlans 3 Olt with 400 users and affect other vlans ,
And problem is start when start destroying dead sessions from vlan with 3 Olt and this affect all other vlans.
May be kernel destroy old session slow and entrained other users by locking other sessions.
is there a way to speed up the closing of stopped/dead sessions.

What are the CPU stats when that happen? Is it users space or kernel
space that keeps it busy?

One easy way to check is to run "mpstat 1" for a few seconds when the
problem occurs.

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Guillaume Nault <hidden>
Date: 2021-09-14 08:07:48

On Tue, Sep 14, 2021 at 09:16:55AM +0300, Martin Zaharinov wrote:

Hi Nault

See this stats :

Linux 5.14.2 (testb)   09/14/21        _x86_64_        (12 CPU)

11:33:44     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
11:33:45     all    1.75    0.00   18.85    0.00    0.00    5.00    0.00    0.00    0.00   74.40
11:33:46     all    1.74    0.00   17.88    0.00    0.00    4.72    0.00    0.00    0.00   75.66
11:33:47     all    2.23    0.00   17.62    0.00    0.00    5.05    0.00    0.00    0.00   75.10
11:33:48     all    1.82    0.00   13.64    0.00    0.00    5.70    0.00    0.00    0.00   78.84
11:33:49     all    1.50    0.00   13.46    0.00    0.00    5.15    0.00    0.00    0.00   79.90
11:33:50     all    3.06    0.00   13.96    0.00    0.00    4.79    0.00    0.00    0.00   78.20
11:33:51     all    1.40    0.00   16.53    0.00    0.00    5.21    0.00    0.00    0.00   76.86
11:33:52     all    4.43    0.00   19.44    0.00    0.00    6.56    0.00    0.00    0.00   69.57
11:33:53     all    1.51    0.00   16.40    0.00    0.00    4.77    0.00    0.00    0.00   77.32
11:33:54     all    1.51    0.00   16.55    0.00    0.00    4.71    0.00    0.00    0.00   77.23
11:33:55     all    1.00    0.00   13.21    0.00    0.00    5.90    0.00    0.00    0.00   79.90
Average:     all    2.00    0.00   16.14    0.00    0.00    5.23    0.00    0.00    0.00   76.63


  PerfTop:   28046 irqs/sec  kernel:96.3%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    23.37%  [nf_conntrack]           [k] nf_ct_iterate_cleanup
    17.76%  [kernel]                 [k] mutex_spin_on_owner
     9.47%  [pppoe]                  [k] pppoe_rcv
     7.71%  [kernel]                 [k] osq_lock
     2.77%  [nf_nat]                 [k] inet_cmp
     2.59%  [nf_nat]                 [k] device_cmp
     2.55%  [kernel]                 [k] __local_bh_enable_ip
     2.04%  [kernel]                 [k] _raw_spin_lock
     1.23%  [kernel]                 [k] __cond_resched
     1.16%  [kernel]                 [k] rcu_all_qs
     1.13%  libfrr.so.0.0.0          [.] 0x00000000000ce970
     0.79%  [nf_conntrack]           [k] nf_conntrack_lock
     0.75%  libfrr.so.0.0.0          [.] 0x00000000000ce94e
     0.53%  [kernel]                 [k] __netif_receive_skb_core.constprop.0
     0.46%  [kernel]                 [k] fib_table_lookup
     0.46%  [ip_tables]              [k] ipt_do_table
     0.45%  [ixgbe]                  [k] ixgbe_clean_rx_irq
     0.37%  [kernel]                 [k] __dev_queue_xmit
     0.34%  [nf_conntrack]           [k] __nf_conntrack_find_get.isra.0
     0.33%  [ixgbe]                  [k] ixgbe_clean_tx_irq
     0.30%  [kernel]                 [k] menu_select
     0.25%  [kernel]                 [k] vlan_do_receive
     0.21%  [kernel]                 [k] ip_finish_output2
     0.21%  [ixgbe]                  [k] ixgbe_poll
     0.20%  [kernel]                 [k] _raw_spin_lock_irqsave
     0.19%  [kernel]                 [k] get_rps_cpu
     0.19%  libc.so.6                [.] 0x0000000000186afa
     0.19%  [kernel]                 [k] queued_read_lock_slowpath
     0.19%  [kernel]                 [k] do_poll.constprop.0
     0.19%  [kernel]                 [k] cpuidle_enter_state
     0.18%  [kernel]                 [k] dev_hard_start_xmit
     0.18%  [kernel]                 [k] ___slab_alloc.constprop.0
     0.17%  zebra                    [.] 0x00000000000b9271
     0.16%  [kernel]                 [k] csum_partial_copy_generic
     0.16%  zebra                    [.] 0x00000000000b91f1
     0.16%  [kernel]                 [k] page_frag_free
     0.16%  [kernel]                 [k] kmem_cache_alloc
     0.15%  [kernel]                 [k] __skb_flow_dissect
     0.15%  [kernel]                 [k] sched_clock
     0.15%  libc.so.6                [.] 0x00000000000965a2
     0.15%  [kernel]                 [k] kmem_cache_free_bulk.part.0
     0.15%  [pppoe]                  [k] pppoe_flush_dev
     0.15%  [ixgbe]                  [k] ixgbe_tx_map
     0.14%  [kernel]                 [k] _raw_spin_lock_bh
     0.14%  [kernel]                 [k] fib_table_flush
     0.14%  [kernel]                 [k] native_irq_return_iret
     0.14%  [kernel]                 [k] __dev_xmit_skb
     0.13%  [kernel]                 [k] nf_hook_slow
     0.13%  [kernel]                 [k] fib_lookup_good_nhc
     0.12%  [kernel]                 [k] __fget_files
     0.12%  [kernel]                 [k] process_backlog
     0.12%  [xt_dtvqos]              [k] 0x00000000000008d1
     0.12%  [kernel]                 [k] __list_del_entry_valid
     0.12%  [kernel]                 [k] skb_release_data
     0.12%  [kernel]                 [k] ip_route_input_slow
     0.11%  [kernel]                 [k] netif_skb_features
     0.11%  [kernel]                 [k] sock_poll
     0.11%  [kernel]                 [k] __schedule
     0.11%  [kernel]                 [k] __softirqentry_text_start


And on time of problem when try to write : ip a 
to list interface wait 15-20 sec i finaly have options to simulate but users is angry when down internet.

Probably some contention on the rtnl lock.

In case need to know why system is overloaded when deconfig ppp interface.

Does it help if you disable conntrack?

Best regards,
Martin

quoted

On 11 Sep 2021, at 9:26, Martin Zaharinov [off-list ref] wrote:

Hi Guillaume

Main problem is overload of service because have many finishing ppp (customer) last two day down from 40-50 to 100-200 users and make problem when is happen if try to type : ip a wait 10-20 sec to start list interface .
But how to find where is a problem any locking or other.
And is there options to make fast remove ppp interface from kernel to reduce this load.


Martin

quoted

On 7 Sep 2021, at 9:42, Martin Zaharinov [off-list ref] wrote:

Perf top from text


PerfTop:   28391 irqs/sec  kernel:98.0%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

  17.01%  [nf_conntrack]           [k] nf_ct_iterate_cleanup
   9.73%  [kernel]                 [k] mutex_spin_on_owner
   9.07%  [pppoe]                  [k] pppoe_rcv
   2.77%  [nf_nat]                 [k] device_cmp
   1.66%  [kernel]                 [k] osq_lock
   1.65%  [kernel]                 [k] _raw_spin_lock
   1.61%  [kernel]                 [k] __local_bh_enable_ip
   1.35%  [nf_nat]                 [k] inet_cmp
   1.30%  [kernel]                 [k] __netif_receive_skb_core.constprop.0
   1.16%  [kernel]                 [k] menu_select
   0.99%  [kernel]                 [k] cpuidle_enter_state
   0.96%  [ixgbe]                  [k] ixgbe_clean_rx_irq
   0.86%  [kernel]                 [k] __dev_queue_xmit
   0.70%  [kernel]                 [k] __cond_resched
   0.69%  [sch_cake]               [k] cake_dequeue
   0.67%  [nf_tables]              [k] nft_do_chain
   0.63%  [kernel]                 [k] rcu_all_qs
   0.61%  [kernel]                 [k] fib_table_lookup
   0.57%  [kernel]                 [k] __schedule
   0.57%  [kernel]                 [k] skb_release_data
   0.54%  [kernel]                 [k] sched_clock
   0.54%  [kernel]                 [k] __copy_skb_header
   0.53%  [kernel]                 [k] dev_queue_xmit_nit
   0.53%  [kernel]                 [k] _raw_spin_lock_irqsave
   0.50%  [kernel]                 [k] kmem_cache_free
   0.48%  libfrr.so.0.0.0          [.] 0x00000000000ce970
   0.47%  [ixgbe]                  [k] ixgbe_clean_tx_irq
   0.45%  [kernel]                 [k] timerqueue_add
   0.45%  [kernel]                 [k] lapic_next_deadline
   0.45%  [kernel]                 [k] csum_partial_copy_generic
   0.44%  [nf_flow_table]          [k] nf_flow_offload_ip_hook
   0.44%  [kernel]                 [k] kmem_cache_alloc
   0.44%  [nf_conntrack]           [k] nf_conntrack_lock

quoted

On 7 Sep 2021, at 9:16, Martin Zaharinov [off-list ref] wrote:

Hi 
Sorry for delay but not easy to catch moment .


See this is mpstatl 1 :

Linux 5.14.1 (demobng) 	09/07/21 	_x86_64_	(12 CPU)

11:12:16     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
11:12:17     all    0.17    0.00    6.66    0.00    0.00    4.13    0.00    0.00    0.00   89.05
11:12:18     all    0.25    0.00    8.36    0.00    0.00    4.88    0.00    0.00    0.00   86.51
11:12:19     all    0.26    0.00    9.62    0.00    0.00    3.91    0.00    0.00    0.00   86.21
11:12:20     all    0.85    0.00    6.00    0.00    0.00    4.31    0.00    0.00    0.00   88.84
11:12:21     all    0.08    0.00    4.45    0.00    0.00    4.79    0.00    0.00    0.00   90.67
11:12:22     all    0.17    0.00    9.50    0.00    0.00    4.58    0.00    0.00    0.00   85.75
11:12:23     all    0.00    0.00    6.92    0.00    0.00    2.48    0.00    0.00    0.00   90.61
11:12:24     all    0.17    0.00    5.45    0.00    0.00    4.27    0.00    0.00    0.00   90.11
11:12:25     all    0.25    0.00    5.38    0.00    0.00    4.79    0.00    0.00    0.00   89.58
11:12:26     all    0.60    0.00    1.45    0.00    0.00    2.65    0.00    0.00    0.00   95.30
11:12:27     all    0.42    0.00    6.91    0.00    0.00    4.47    0.00    0.00    0.00   88.20
11:12:28     all    0.00    0.00    6.75    0.00    0.00    4.18    0.00    0.00    0.00   89.07
11:12:29     all    0.17    0.00    3.52    0.00    0.00    5.11    0.00    0.00    0.00   91.20
11:12:30     all    1.45    0.00   10.14    0.00    0.00    3.49    0.00    0.00    0.00   84.92
11:12:31     all    0.09    0.00    5.11    0.00    0.00    4.77    0.00    0.00    0.00   90.03
11:12:32     all    0.25    0.00    3.11    0.00    0.00    4.46    0.00    0.00    0.00   92.17
Average:     all    0.32    0.00    6.21    0.00    0.00    4.21    0.00    0.00    0.00   89.26


I attache and one screenshot from perf top (Screenshot is send on preview mail)

And I see in lsmod 

pppoe                  20480  8198
pppox                  16384  1 pppoe
ppp_generic            45056  16364 pppox,pppoe
slhc                   16384  1 ppp_generic

To slow remove pppoe session .

And from log : 

[2021-09-07 11:01:11.129] vlan3020: ebdd1c5d8b5900f6: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:01:53.621] vlan643: ebdd1c5d8b59014e: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:00.359] vlan1616: ebdd1c5d8b590195: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:05.859] vlan3020: ebdd1c5d8b5900d8: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:08.258] vlan3005: ebdd1c5d8b590190: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:13.820] vlan643: ebdd1c5d8b590152: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:15.839] vlan727: ebdd1c5d8b590144: ioctl(PPPIOCCONNECT): Transport endpoint is not connected
[2021-09-07 11:02:20.139] vlan1693: ebdd1c5d8b59019f: ioctl(PPPIOCCONNECT): Transport endpoint is not connected

quoted

On 11 Aug 2021, at 19:48, Guillaume Nault [off-list ref] wrote:

On Wed, Aug 11, 2021 at 02:10:32PM +0300, Martin Zaharinov wrote:

quoted

And one more that see.

Problem is come when accel start finishing sessions,
Now in server have 2k users and restart on one of vlans 3 Olt with 400 users and affect other vlans ,
And problem is start when start destroying dead sessions from vlan with 3 Olt and this affect all other vlans.
May be kernel destroy old session slow and entrained other users by locking other sessions.
is there a way to speed up the closing of stopped/dead sessions.

What are the CPU stats when that happen? Is it users space or kernel
space that keeps it busy?

One easy way to check is to run "mpstat 1" for a few seconds when the
problem occurs.

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Florian Westphal <fw@strlen.de>
Date: 2021-09-14 09:50:25

Guillaume Nault [off-list ref] wrote:

quoted

And on time of problem when try to write : ip a 
to list interface wait 15-20 sec i finaly have options to simulate but users is angry when down internet.

Probably some contention on the rtnl lock.

Yes, I'll create a patch.

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Martin Zaharinov <hidden>
Date: 2021-09-14 10:01:38

Hi Nault and Florian

Nault : 

No not test need conntrack to log user traffic.

Florian: 

If you make patch send to test please.


Martin

On 14 Sep 2021, at 12:50, Florian Westphal [off-list ref] wrote:

Guillaume Nault [off-list ref] wrote:

quoted

And on time of problem when try to write : ip a 
to list interface wait 15-20 sec i finaly have options to simulate but users is angry when down internet.

Probably some contention on the rtnl lock.

Yes, I'll create a patch.

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Martin Zaharinov <hidden>
Date: 2021-09-14 10:53:26

Florian Hi 
One more 

please see i try to remove nf_nat and xt_MASQUERADE 

and on time of problem need 50-80 sec to remove and overload system .

see perf from this moment: 




 PerfTop:    1738 irqs/sec  kernel:85.0%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    40.63%  [nf_conntrack]    [k] nf_ct_iterate_cleanup
    21.23%  [kernel]          [k] __local_bh_enable_ip
    10.93%  [kernel]          [k] __cond_resched
     9.20%  [kernel]          [k] _raw_spin_lock
     8.91%  [kernel]          [k] rcu_all_qs
     5.83%  [nf_conntrack]    [k] nf_conntrack_lock
     0.10%  [kernel]          [k] mutex_spin_on_owner
     0.08%  telegraf          [.] 0x0000000000021bf0
     0.06%  [kernel]          [k] osq_lock
     0.06%  [kernel]          [k] kallsyms_expand_symbol.constprop.0
     0.05%  [kernel]          [k] format_decode
     0.04%  [kernel]          [k] rtnl_fill_ifinfo.constprop.0.isra.0
     0.04%  perf              [.] 0x00000000000bc7b3
     0.04%  [kernel]          [k] memcpy_erms
     0.03%  [kernel]          [k] string
     0.03%  [kernel]          [k] menu_select
     0.03%  [kernel]          [k] nla_put
     0.03%  [kernel]          [k] vsnprintf



Martin

On 14 Sep 2021, at 12:50, Florian Westphal [off-list ref] wrote:

Guillaume Nault [off-list ref] wrote:

quoted

And on time of problem when try to write : ip a 
to list interface wait 15-20 sec i finaly have options to simulate but users is angry when down internet.

Probably some contention on the rtnl lock.

Yes, I'll create a patch.

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Florian Westphal <fw@strlen.de>
Date: 2021-09-14 11:00:44

Martin Zaharinov [off-list ref] wrote:

[ Trimming CC list ]

Florian: 

If you make patch send to test please.

Attached.  No idea if it helps, but 'ip' should stay responsive
even when masquerade processes netdevice events.

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Martin Zaharinov <hidden>
Date: 2021-09-15 14:25:25

Hey Florian

make test in lab and look much better that before.

see this perf 

 PerfTop:    6551 irqs/sec  kernel:77.8%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

    15.70%  [ixgbe]           [k] ixgbe_read_reg
    13.33%  [kernel]          [k] mutex_spin_on_owner
     7.65%  [kernel]          [k] osq_lock
     2.85%  libfrr.so.0.0.0   [.] 0x00000000000ce970
     1.94%  libfrr.so.0.0.0   [.] 0x00000000000ce94e
     1.19%  libc.so.6         [.] 0x0000000000186afa
     1.15%  [kernel]          [k] do_poll.constprop.0
     0.99%  [kernel]          [k] inet_dump_ifaddr
     0.94%  libteam.so.5.6.1  [.] 0x0000000000006470
     0.79%  libc.so.6         [.] 0x0000000000186e57
     0.71%  [ixgbe]           [k] ixgbe_update_mc_addr_list_generic
     0.65%  [kernel]          [k] __fget_files
     0.61%  [kernel]          [k] sock_poll
     0.57%  libteam.so.5.6.1  [.] 0x0000000000009e7d
     0.51%  perf              [.] 0x00000000000bc7b3
     0.51%  libteam.so.5.6.1  [.] 0x0000000000006501
     0.48%  [kernel]          [k] next_uptodate_page
     0.46%  [kernel]          [k] _raw_read_lock_bh
     0.43%  libc.so.6         [.] 0x0000000000186eac
     0.42%  bgpd              [.] 0x0000000000070a46
     0.41%  [pppoe]           [k] pppoe_flush_dev
     0.39%  [kernel]          [k] zap_pte_range


This happened when remove and add new interface on time of drop and reconnect users.


now : ip a command work fine !


Martin

On 14 Sep 2021, at 14:00, Florian Westphal [off-list ref] wrote:

Martin Zaharinov [off-list ref] wrote:

[ Trimming CC list ]

quoted

Florian: 

If you make patch send to test please.

Attached.  No idea if it helps, but 'ip' should stay responsive
even when masquerade processes netdevice events.
<defer_masq_work.diff>

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Martin Zaharinov <hidden>
Date: 2021-09-15 14:37:39

and this : 

  PerfTop:   26378 irqs/sec  kernel:61.4%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

     5.65%  libfrr.so.0.0.0   [.] 0x00000000000ce970
     5.56%  [kernel]          [k] osq_lock
     5.22%  [kernel]          [k] mutex_spin_on_owner
     3.66%  [pppoe]           [k] pppoe_flush_dev
     3.01%  libfrr.so.0.0.0   [.] 0x00000000000ce94e
     1.98%  libc.so.6         [.] 0x00000000000965a2
     1.84%  libc.so.6         [.] 0x0000000000186afa
     1.55%  libc.so.6         [.] 0x0000000000186e57
     1.54%  zebra             [.] 0x00000000000b9271
     1.46%  zebra             [.] 0x00000000000b91f1
     1.46%  libteam.so.5.6.1  [.] 0x0000000000006470
     1.44%  libc.so.6         [.] 0x00000000000965a0
     1.30%  libteam.so.5.6.1  [.] 0x0000000000009e7d
     1.08%  [kernel]          [k] fib_table_flush
     1.02%  libc.so.6         [.] 0x0000000000186eac
     0.93%  [kernel]          [k] do_poll.constprop.0
     0.85%  libc.so.6         [.] 0x0000000000186afe
     0.80%  dtvbras           [.] 0x0000000000014be8
     0.78%  [kernel]          [k] queued_read_lock_slowpath
     0.72%  [kernel]          [k] next_uptodate_page
     0.64%  [kernel]          [k] zap_pte_range
     0.64%  bgpd              [.] 0x0000000000070a46
     0.61%  [kernel]          [k] fib_table_insert

On 15 Sep 2021, at 17:25, Martin Zaharinov [off-list ref] wrote:

Hey Florian

make test in lab and look much better that before.

see this perf 

PerfTop:    6551 irqs/sec  kernel:77.8%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

   15.70%  [ixgbe]           [k] ixgbe_read_reg
   13.33%  [kernel]          [k] mutex_spin_on_owner
    7.65%  [kernel]          [k] osq_lock
    2.85%  libfrr.so.0.0.0   [.] 0x00000000000ce970
    1.94%  libfrr.so.0.0.0   [.] 0x00000000000ce94e
    1.19%  libc.so.6         [.] 0x0000000000186afa
    1.15%  [kernel]          [k] do_poll.constprop.0
    0.99%  [kernel]          [k] inet_dump_ifaddr
    0.94%  libteam.so.5.6.1  [.] 0x0000000000006470
    0.79%  libc.so.6         [.] 0x0000000000186e57
    0.71%  [ixgbe]           [k] ixgbe_update_mc_addr_list_generic
    0.65%  [kernel]          [k] __fget_files
    0.61%  [kernel]          [k] sock_poll
    0.57%  libteam.so.5.6.1  [.] 0x0000000000009e7d
    0.51%  perf              [.] 0x00000000000bc7b3
    0.51%  libteam.so.5.6.1  [.] 0x0000000000006501
    0.48%  [kernel]          [k] next_uptodate_page
    0.46%  [kernel]          [k] _raw_read_lock_bh
    0.43%  libc.so.6         [.] 0x0000000000186eac
    0.42%  bgpd              [.] 0x0000000000070a46
    0.41%  [pppoe]           [k] pppoe_flush_dev
    0.39%  [kernel]          [k] zap_pte_range


This happened when remove and add new interface on time of drop and reconnect users.


now : ip a command work fine !


Martin

quoted

On 14 Sep 2021, at 14:00, Florian Westphal [off-list ref] wrote:

Martin Zaharinov [off-list ref] wrote:

[ Trimming CC list ]

quoted

Florian: 

If you make patch send to test please.

Attached.  No idea if it helps, but 'ip' should stay responsive
even when masquerade processes netdevice events.
<defer_masq_work.diff>

Re: Urgent Bug report: PPPoE ioctl(PPPIOCCONNECT): Transport endpoint is not connected

From: Martin Zaharinov <hidden>
Date: 2021-09-16 20:00:41

Small Updates

After switch from frr to bird bgp reduce load from frr
but still when have disconnect 5k+ users have slow pppoe_flush_dev




   PerfTop:   15606 irqs/sec  kernel:77.7%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (all, 12 CPUs)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

     8.24%  [kernel]              [k] osq_lock
     7.55%  [kernel]              [k] mutex_spin_on_owner
     7.04%  [pppoe]               [k] pppoe_flush_dev
     2.77%  libteam.so.5.6.1      [.] 0x0000000000009e7d
     2.67%  libteam.so.5.6.1      [.] 0x0000000000006470
     1.90%  [kernel]              [k] fib_table_flush
     1.73%  [kernel]              [k] queued_read_lock_slowpath
     1.68%  [kernel]              [k] next_uptodate_page
     1.36%  ip                    [.] 0x0000000000011b74
     1.23%  ip                    [.] 0x00000000000121b0
     1.09%  [kernel]              [k] zap_pte_range
     0.99%  libteam.so.5.6.1      [.] 0x0000000000006501
     0.88%  dtvbras               [.] 0x0000000000014be8
     0.87%  [kernel]              [k] inet_dump_ifaddr
     0.74%  [kernel]              [k] filemap_map_pages
     0.72%  [kernel]              [k] neigh_flush_dev.isra.0
     0.66%  [kernel]              [k] snmp_get_cpu_field
     0.65%  [kernel]              [k] fib_table_insert
     0.63%  [kernel]              [k] native_irq_return_iret
     0.63%  libteam.so.5.6.1      [.] 0x0000000000005c78
     0.60%  [kernel]              [k] copy_page
     0.52%  libteam.so.5.6.1      [.] 0x000000000000647f
     0.50%  [kernel]              [k] _raw_spin_lock
     0.48%  libc.so.6             [.] 0x00000000000965a2
     0.45%  [kernel]              [k] _raw_read_lock_bh
     0.44%  [kernel]              [k] release_pages
     0.42%  [kernel]              [k] clear_page_erms
     0.42%  [kernel]              [k] page_remove_rmap
     0.41%  [kernel]              [k] queued_spin_lock_slowpath
     0.38%  [kernel]              [k] kmem_cache_alloc
     0.36%  [kernel]              [k] vma_interval_tree_insert
     0.36%  libteam.so.5.6.1      [.] 0x0000000000009e6f
     0.36%  [kernel]              [k] do_set_pte


sessions:
  starting: 296
  active: 3868
  finishing: 6748

On 14 Sep 2021, at 14:00, Florian Westphal [off-list ref] wrote:

Martin Zaharinov [off-list ref] wrote:

[ Trimming CC list ]

quoted

Florian: 

If you make patch send to test please.

Attached.  No idea if it helps, but 'ip' should stay responsive
even when masquerade processes netdevice events.
<defer_masq_work.diff>

`h`	back out one level
`j`	next message in thread
`k`	previous message in thread
`l`	drill in
`Esc`	close help / fold thread tree
`?`	toggle this help