Hi,
I am working on a self-written routing agent
(https://github.com/OLSR/OONF) and am stuck on a problem with netlink
that I cannot explain with an userspace error.
I am using a netlink socket for setting routes
(RTM_NEWROUTE/RTM_DELROUTE), querying the kernel for the current routes
in the database (via a RTM_GETROUTE dump) and for getting multicast
messages for ongoing routing changes.
After a few netlink messages I get to the point where the kernel just
does not responst to a RTM_NEWROUTE. No error, no answer, despite the
NLM_F_ACK flag set)... but sometime when (during shutdown of the routing
agent) the program sends another route command (most times a
RTM_DELROUTE) I get a single netlink packet with a "successful" response
for both the "missing" RTM_NEWROUTE and one for the new RTM DELROUTE
sequence number.
I am testing two routing agents, each of them in a systemd-nspawn based
container connected over a bridge on the host system on a current Debian
Testing (kernel 4.18.0-1-amd64).
I am directly using the netlink sockets, without any other userspace
library in between.
I have checked the hexdumps of a couple of netlink messages (including
the ones just before the bug happens) by hand and they seem to be okay.
When I tried to add a "netlink listener" socket for futher debugging (ip
link add nlmon0 type nlmon) the problem vanished until I removed the
listener socket again.
Any ideas how to debug this problem? Unfortunately I have no short
example program to trigger the bug... I have rarely seen the problem for
years (once every couple of months), but until a few days ago I never
managed to reproduce it.
Henning Rogge
--
Diplom-Informatiker Henning Rogge , Fraunhofer-Institut für
Kommunikation, Informationsverarbeitung und Ergonomie FKIE
Kommunikationssysteme (KOM)
Zanderstrasse 5, 53177 Bonn, Germany
Telefon +49 228 50212-469
mailto:henning.rogge@fkie.fraunhofer.de http://www.fkie.fraunhofer.de
From: Stephen Hemminger <stephen@networkplumber.org> Date: 2018-10-13 02:25:53
On Fri, 12 Oct 2018 09:30:40 +0200
Henning Rogge [off-list ref] wrote:
Hi,
I am working on a self-written routing agent
(https://github.com/OLSR/OONF) and am stuck on a problem with netlink
that I cannot explain with an userspace error.
I am using a netlink socket for setting routes
(RTM_NEWROUTE/RTM_DELROUTE), querying the kernel for the current routes
in the database (via a RTM_GETROUTE dump) and for getting multicast
messages for ongoing routing changes.
After a few netlink messages I get to the point where the kernel just
does not responst to a RTM_NEWROUTE. No error, no answer, despite the
NLM_F_ACK flag set)... but sometime when (during shutdown of the routing
agent) the program sends another route command (most times a
RTM_DELROUTE) I get a single netlink packet with a "successful" response
for both the "missing" RTM_NEWROUTE and one for the new RTM DELROUTE
sequence number.
I am testing two routing agents, each of them in a systemd-nspawn based
container connected over a bridge on the host system on a current Debian
Testing (kernel 4.18.0-1-amd64).
I am directly using the netlink sockets, without any other userspace
library in between.
I have checked the hexdumps of a couple of netlink messages (including
the ones just before the bug happens) by hand and they seem to be okay.
When I tried to add a "netlink listener" socket for futher debugging (ip
link add nlmon0 type nlmon) the problem vanished until I removed the
listener socket again.
Any ideas how to debug this problem? Unfortunately I have no short
example program to trigger the bug... I have rarely seen the problem for
years (once every couple of months), but until a few days ago I never
managed to reproduce it.
Henning Rogge
Are you reading the responses to your requests? If you don't read
the response, the socket will get flow blocked.
On Fri, 12 Oct 2018 09:30:40 +0200
Henning Rogge [off-list ref] wrote:
quoted
Hi,
I am working on a self-written routing agent
(https://github.com/OLSR/OONF) and am stuck on a problem with netlink
that I cannot explain with an userspace error.
I am using a netlink socket for setting routes
(RTM_NEWROUTE/RTM_DELROUTE), querying the kernel for the current routes
in the database (via a RTM_GETROUTE dump) and for getting multicast
messages for ongoing routing changes.
After a few netlink messages I get to the point where the kernel just
does not responst to a RTM_NEWROUTE. No error, no answer, despite the
NLM_F_ACK flag set)... but sometime when (during shutdown of the routing
agent) the program sends another route command (most times a
RTM_DELROUTE) I get a single netlink packet with a "successful" response
for both the "missing" RTM_NEWROUTE and one for the new RTM DELROUTE
sequence number.
I am testing two routing agents, each of them in a systemd-nspawn based
container connected over a bridge on the host system on a current Debian
Testing (kernel 4.18.0-1-amd64).
I am directly using the netlink sockets, without any other userspace
library in between.
I have checked the hexdumps of a couple of netlink messages (including
the ones just before the bug happens) by hand and they seem to be okay.
When I tried to add a "netlink listener" socket for futher debugging (ip
link add nlmon0 type nlmon) the problem vanished until I removed the
listener socket again.
Any ideas how to debug this problem? Unfortunately I have no short
example program to trigger the bug... I have rarely seen the problem for
years (once every couple of months), but until a few days ago I never
managed to reproduce it.
Henning Rogge
Are you reading the responses to your requests? If you don't read
the response, the socket will get flow blocked.
Yes, I do...
all netlink sockets the program uses are constantly watched for traffic
coming from the kernel (with an epoll()-based event loop, no edge-trigger).
I even have a rate limitation towards the kernel, only sending a
"pagesize" full of netlink data towards the kernel, then waiting for the
reply before sending more (I had the blocking problem a few years ago
when experimenting with LOTS of routes).
Henning Rogge
--
Diplom-Informatiker Henning Rogge , Fraunhofer-Institut für
Kommunikation, Informationsverarbeitung und Ergonomie FKIE
Kommunikationssysteme (KOM)
Zanderstrasse 5, 53177 Bonn, Germany
Telefon +49 228 50212-469
mailto:henning.rogge@fkie.fraunhofer.de http://www.fkie.fraunhofer.de
Does anyone else have an idea how to debug this problem?
Henning Rogge
Am 15.10.2018 um 07:25 schrieb Henning Rogge:
Am 12.10.2018 um 20:51 schrieb Stephen Hemminger:
quoted
On Fri, 12 Oct 2018 09:30:40 +0200
Henning Rogge [off-list ref] wrote:
quoted
Hi,
I am working on a self-written routing agent
(https://github.com/OLSR/OONF) and am stuck on a problem with netlink
that I cannot explain with an userspace error.
I am using a netlink socket for setting routes
(RTM_NEWROUTE/RTM_DELROUTE), querying the kernel for the current routes
in the database (via a RTM_GETROUTE dump) and for getting multicast
messages for ongoing routing changes.
After a few netlink messages I get to the point where the kernel just
does not responst to a RTM_NEWROUTE. No error, no answer, despite the
NLM_F_ACK flag set)... but sometime when (during shutdown of the routing
agent) the program sends another route command (most times a
RTM_DELROUTE) I get a single netlink packet with a "successful" response
for both the "missing" RTM_NEWROUTE and one for the new RTM DELROUTE
sequence number.
I am testing two routing agents, each of them in a systemd-nspawn based
container connected over a bridge on the host system on a current Debian
Testing (kernel 4.18.0-1-amd64).
I am directly using the netlink sockets, without any other userspace
library in between.
I have checked the hexdumps of a couple of netlink messages (including
the ones just before the bug happens) by hand and they seem to be okay.
When I tried to add a "netlink listener" socket for futher debugging (ip
link add nlmon0 type nlmon) the problem vanished until I removed the
listener socket again.
Any ideas how to debug this problem? Unfortunately I have no short
example program to trigger the bug... I have rarely seen the problem for
years (once every couple of months), but until a few days ago I never
managed to reproduce it.
Henning Rogge
Are you reading the responses to your requests? If you don't read
the response, the socket will get flow blocked.
Yes, I do...
all netlink sockets the program uses are constantly watched for traffic
coming from the kernel (with an epoll()-based event loop, no edge-trigger).
I even have a rate limitation towards the kernel, only sending a
"pagesize" full of netlink data towards the kernel, then waiting for the
reply before sending more (I had the blocking problem a few years ago
when experimenting with LOTS of routes).
Henning Rogge
Henning Rogge
--
Diplom-Informatiker Henning Rogge , Fraunhofer-Institut für
Kommunikation, Informationsverarbeitung und Ergonomie FKIE
Kommunikationssysteme (KOM)
Zanderstrasse 5, 53177 Bonn, Germany
Telefon +49 228 50212-469
mailto:henning.rogge@fkie.fraunhofer.de http://www.fkie.fraunhofer.de