ss command not showing raw sockets? (regression)

Hi there,

I've got following installation:
* ping 32 bit version
* Linux 4.4.0 x86_64 (yes, somewhat ancient)
* iproute2  4.9.0 or 4.20.0 or 5.10.0

With one ping command active, there are two raw sockets on my system: one for IPv4 and one for IPv6 (just one of those is used).

My problem is that

ss -awp

shows 
* two raw sockets (4.9.0)
* any raw socket = bug (4.20.0)
* any raw socket = bug (5.10.0)


So is this a bug or is this wont-fix (then, if it is related to kernel version, package maintainers may be interested)?


Best regards
 Jonas Bechtel

On Sun, 15 Aug 2021 23:17:38 +0200 Jonas Bechtel wrote:

I've got following installation:
* ping 32 bit version
* Linux 4.4.0 x86_64 (yes, somewhat ancient)
* iproute2  4.9.0 or 4.20.0 or 5.10.0

With one ping command active, there are two raw sockets on my system:
one for IPv4 and one for IPv6 (just one of those is used).

My problem is that

ss -awp

shows 
* two raw sockets (4.9.0)
* any raw socket = bug (4.20.0)
* any raw socket = bug (5.10.0)

Could you clarify how the bug manifests itself? Does ss crash?

So is this a bug or is this wont-fix (then, if it is related to
kernel version, package maintainers may be interested)?

On 8/16/21 4:08 PM, Jakub Kicinski wrote:

On Sun, 15 Aug 2021 23:17:38 +0200 Jonas Bechtel wrote:

quoted

I've got following installation:
* ping 32 bit version
* Linux 4.4.0 x86_64 (yes, somewhat ancient)
* iproute2  4.9.0 or 4.20.0 or 5.10.0

With one ping command active, there are two raw sockets on my system:
one for IPv4 and one for IPv6 (just one of those is used).

My problem is that

ss -awp

shows 
* two raw sockets (4.9.0)
* any raw socket = bug (4.20.0)
* any raw socket = bug (5.10.0)

Could you clarify how the bug manifests itself? Does ss crash?

I take it kernel version is constant and iproute2 version changes,
correct? Can you download the source and do a git bisect?

quoted

So is this a bug or is this wont-fix (then, if it is related to
kernel version, package maintainers may be interested)?

On Mon, 16 Aug 2021 15:08:00 -0700 Jakub Kicinski wrote:

On Sun, 15 Aug 2021 23:17:38 +0200 Jonas Bechtel wrote:

quoted

I've got following installation:
* ping 32 bit version
* Linux 4.4.0 x86_64 (yes, somewhat ancient)
* iproute2  4.9.0 or 4.20.0 or 5.10.0

With one ping command active, there are two raw sockets on my system:
one for IPv4 and one for IPv6 (just one of those is used).

My problem is that

ss -awp

shows 
* two raw sockets (4.9.0)
* any raw socket = bug (4.20.0)
* any raw socket = bug (5.10.0)  

Could you clarify how the bug manifests itself? Does ss crash?

quoted

So is this a bug or is this wont-fix (then, if it is related to
kernel version, package maintainers may be interested)?  

I had a look, I don't see anything out of the ordinary. I checked with
v4.6, I don't have a 4.4 box handy. It seems ss got support for dumping
over netlink in the 4.9. On a 4.4 kernel it should fall back to using
procfs tho, raw_show() calls inet_show_netlink() which should fails and
therefore the code should fall through to the old procfs stuff.

No idea why that doesn't happen for you. Is this vanilla 4.4 or does it
have backports? Is there a /sys/module/raw_diag/ directory on your
system after you run those commands?

Does setting PROC_NET_RAW make the newer iproute version work for you?

$ PROC_NET_RAW=/proc/net/raw ss -awp

On Tue, 17 Aug 2021 08:04:51 -0700
Jakub Kicinski [off-list ref] wrote with subject
"Re: ss command not showing raw sockets? (regression)":

On Mon, 16 Aug 2021 15:08:00 -0700 Jakub Kicinski wrote:

quoted

On Sun, 15 Aug 2021 23:17:38 +0200 Jonas Bechtel wrote:

quoted

I've got following installation:
* ping 32 bit version
* Linux 4.4.0 x86_64 (yes, somewhat ancient)
* iproute2  4.9.0 or 4.20.0 or 5.10.0

With one ping command active, there are two raw sockets on my
system: one for IPv4 and one for IPv6 (just one of those is used).

My problem is that

ss -awp

shows 
* two raw sockets (4.9.0)
* any raw socket = bug (4.20.0)
* any raw socket = bug (5.10.0)  

Could you clarify how the bug manifests itself? Does ss crash?

quoted

So is this a bug or is this wont-fix (then, if it is related to
kernel version, package maintainers may be interested)?  

I had a look, I don't see anything out of the ordinary. I checked with
v4.6, I don't have a 4.4 box handy. It seems ss got support for
dumping over netlink in the 4.9. On a 4.4 kernel it should fall back
to using procfs tho, raw_show() calls inet_show_netlink() which
should fails and therefore the code should fall through to the old
procfs stuff.

No idea why that doesn't happen for you. Is this vanilla 4.4 or does
it have backports? Is there a /sys/module/raw_diag/ directory on your
system after you run those commands?

It's was Knoppix distributed package. I don't know about the exact contents, there's also no hint in package description. I just know that it works without initrd, as it directly mounts the root disk.

No, there's /sys/module but no /sys/module/raw_diag/ neither before nor after running those commands.

Does setting PROC_NET_RAW make the newer iproute version work for you?

$ PROC_NET_RAW=/proc/net/raw ss -awp

Yes, this did the trick. (And again I was thinking programs were doing something "magical", but in the end it's just a file they access)


Furthermore I checked with Linux 4.19.0 amd64 RT (Debian package; from package description: "This kernel includes the PREEMPT_RT realtime patch set."). With this kernel there was no need for PROC_NET_RAW. All iproute versions worked out of the box and showed even command name, pid and fd number (that's why ss traverses all /proc/[pids]/fd/ directories?).


See attached log file, with kernel versions and iproute2 versions printed.


@kuba With PROC_NET_RAW I consider the problem is found, isn't it? So I will not download/bisect<->build or otherwise investigate the problem until one of you explicitely asks me to do so.

I have now redirected invocation of command with set PROC_NET_RAW on my system, and may (try to) update to Linux 4.19.

Thank you!


Best Regards
 jbechtel

On Tue, 17 Aug 2021 20:21:35 +0200 Jonas Bechtel wrote:

On Tue, 17 Aug 2021 08:04:51 -0700
Jakub Kicinski [off-list ref] wrote with subject

quoted

I had a look, I don't see anything out of the ordinary. I checked with
v4.6, I don't have a 4.4 box handy. It seems ss got support for
dumping over netlink in the 4.9. On a 4.4 kernel it should fall back
to using procfs tho, raw_show() calls inet_show_netlink() which
should fails and therefore the code should fall through to the old
procfs stuff.

No idea why that doesn't happen for you. Is this vanilla 4.4 or does
it have backports? Is there a /sys/module/raw_diag/ directory on your
system after you run those commands?  

It's was Knoppix distributed package. I don't know about the exact
contents, there's also no hint in package description. I just know
that it works without initrd, as it directly mounts the root disk.

No, there's /sys/module but no /sys/module/raw_diag/ neither before
nor after running those commands.

Hm. Could you share the config for that kernel? I don't know Knoppix
but there should be a /boot/config-* or /proc/config.gz, hopefully.

quoted

Does setting PROC_NET_RAW make the newer iproute version work for
you?

$ PROC_NET_RAW=/proc/net/raw ss -awp  

Yes, this did the trick. (And again I was thinking programs were
doing something "magical", but in the end it's just a file they
access)


Furthermore I checked with Linux 4.19.0 amd64 RT (Debian package;
from package description: "This kernel includes the PREEMPT_RT
realtime patch set."). With this kernel there was no need for
PROC_NET_RAW. All iproute versions worked out of the box and showed
even command name, pid and fd number (that's why ss traverses all
/proc/[pids]/fd/ directories?).


See attached log file, with kernel versions and iproute2 versions
printed.


@kuba With PROC_NET_RAW I consider the problem is found, isn't it? So
I will not download/bisect<->build or otherwise investigate the
problem until one of you explicitely asks me to do so.

I have now redirected invocation of command with set PROC_NET_RAW on
my system, and may (try to) update to Linux 4.19.

I suspect the bisection would end up at the commit which added 
the netlink dump support, so you can hold off for now, yes.

My best guess right now is that Knoppix has a cut-down kernel 
config and we don't handle that case correctly.

On 8/17/21 12:44 PM, Jakub Kicinski wrote:

quoted

@kuba With PROC_NET_RAW I consider the problem is found, isn't it? So
I will not download/bisect<->build or otherwise investigate the
problem until one of you explicitely asks me to do so.

I have now redirected invocation of command with set PROC_NET_RAW on
my system, and may (try to) update to Linux 4.19.

I suspect the bisection would end up at the commit which added 
the netlink dump support, so you can hold off for now, yes.

agreed.

My best guess right now is that Knoppix has a cut-down kernel 
config and we don't handle that case correctly.

CONFIG_INET_RAW_DIAG (or INET_DIAG) is probably disabled. surprised the
netlink dump does not return an error and it falls back to the proc file:

        if (!getenv("PROC_NET_RAW") && !getenv("PROC_ROOT") &&
            inet_show_netlink(f, NULL, IPPROTO_RAW) == 0)
                return 0;

can you strace it?

On Tue, 17 Aug 2021 13:54:53 -0600 David Ahern wrote:

On 8/17/21 12:44 PM, Jakub Kicinski wrote:

quoted

quoted

@kuba With PROC_NET_RAW I consider the problem is found, isn't it? So
I will not download/bisect<->build or otherwise investigate the
problem until one of you explicitely asks me to do so.

I have now redirected invocation of command with set PROC_NET_RAW on
my system, and may (try to) update to Linux 4.19.  

I suspect the bisection would end up at the commit which added 
the netlink dump support, so you can hold off for now, yes.  

agreed.

quoted

My best guess right now is that Knoppix has a cut-down kernel 
config and we don't handle that case correctly.
  

CONFIG_INET_RAW_DIAG (or INET_DIAG) is probably disabled. surprised the
netlink dump does not return an error and it falls back to the proc file:

        if (!getenv("PROC_NET_RAW") && !getenv("PROC_ROOT") &&
            inet_show_netlink(f, NULL, IPPROTO_RAW) == 0)
                return 0;

can you strace it?

Ah, good point, strace will show it. 

/me goes off to look at the strace Jonas sent off list.

Well this is unexpected:

sendmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=76, type=DCCPDIAG_GETSOCK, ...

--->8----------------

From: Jakub Kicinski <kuba@kernel.org>
Subject: ss: fix fallback to procfs for raw and sctp sockets

sockdiag_send() diverts to tcpdiag_send() to try the older
netlink interface. tcpdiag_send() works for TCP and DCCP
but not other protocols. Instead of rejecting unsupported
protocols (and missing RAW and SCTP) match on supported ones.

Fixes: 41fe6c34de50 ("ss: Add inet raw sockets information gathering via netlink diag interface")
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
---
 misc/ss.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/misc/ss.c b/misc/ss.c
index 894ad40574f1..b39f63fe3b17 100644
--- a/misc/ss.c
+++ b/misc/ss.c

@@ -3404,13 +3404,13 @@ static int tcpdiag_send(int fd, int protocol, struct filter *f)
 	struct iovec iov[3];
 	int iovlen = 1;
 
-	if (protocol == IPPROTO_UDP || protocol == IPPROTO_MPTCP)
-		return -1;
-
 	if (protocol == IPPROTO_TCP)
 		req.nlh.nlmsg_type = TCPDIAG_GETSOCK;
-	else
+	else if (protocol == IPPROTO_DCCP)
 		req.nlh.nlmsg_type = DCCPDIAG_GETSOCK;
+	else
+		return -1;
+
 	if (show_mem) {
 		req.r.idiag_ext |= (1<<(INET_DIAG_MEMINFO-1));
 		req.r.idiag_ext |= (1<<(INET_DIAG_SKMEMINFO-1));

-- 
2.31.1

On 8/17/21 3:37 PM, Jakub Kicinski wrote:

That looks correct to me.

Jonas: can you build iproute2 and test?

On Tue, 17 Aug 2021 18:47:06 -0600
David Ahern [off-list ref] wrote with subject
"Re: ss command not showing raw sockets? (regression)":

On 8/17/21 3:37 PM, Jakub Kicinski wrote:

quoted

Ah, good point, strace will show it. 

/me goes off to look at the strace Jonas sent off list.

Well this is unexpected:

sendmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=76,
type=DCCPDIAG_GETSOCK, ... --->8----------------  

From: Jakub Kicinski <kuba@kernel.org>
Subject: ss: fix fallback to procfs for raw and sctp sockets

sockdiag_send() diverts to tcpdiag_send() to try the older
netlink interface. tcpdiag_send() works for TCP and DCCP
but not other protocols. Instead of rejecting unsupported
protocols (and missing RAW and SCTP) match on supported ones.

Fixes: 41fe6c34de50 ("ss: Add inet raw sockets information
gathering via netlink diag interface") Signed-off-by: Jakub
Kicinski [off-list ref] ---
 misc/ss.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/misc/ss.c b/misc/ss.c
index 894ad40574f1..b39f63fe3b17 100644
--- a/misc/ss.c
+++ b/misc/ss.c

@@ -3404,13 +3404,13 @@ static int tcpdiag_send(int fd, int

protocol, struct filter *f) struct iovec iov[3];
 	int iovlen = 1;
 
-	if (protocol == IPPROTO_UDP || protocol == IPPROTO_MPTCP)
-		return -1;
-
 	if (protocol == IPPROTO_TCP)
 		req.nlh.nlmsg_type = TCPDIAG_GETSOCK;
-	else
+	else if (protocol == IPPROTO_DCCP)
 		req.nlh.nlmsg_type = DCCPDIAG_GETSOCK;
+	else
+		return -1;
+
 	if (show_mem) {
 		req.r.idiag_ext |= (1<<(INET_DIAG_MEMINFO-1));
 		req.r.idiag_ext |= (1<<(INET_DIAG_SKMEMINFO-1));
  

That looks correct to me.

Jonas: can you build iproute2 and test?

I've cloned branch main as instructed in https://wiki.linuxfoundation.org/networking/iproute2. Most recent commit is 9b7ea92b9e3f. After building, no socket was listed in table.

Then I [manually] applied the patch and rebuilt. The patched version works well, I do see the two sockets right now.

Command was in both cases misc/ss -awp

On 8/18/21 1:57 PM, Jonas Bechtel wrote:

On Tue, 17 Aug 2021 18:47:06 -0600
David Ahern [off-list ref] wrote with subject
"Re: ss command not showing raw sockets? (regression)":

quoted

On 8/17/21 3:37 PM, Jakub Kicinski wrote:

quoted

Ah, good point, strace will show it. 

/me goes off to look at the strace Jonas sent off list.

Well this is unexpected:

sendmsg(3, {msg_name={sa_family=AF_NETLINK, nl_pid=0,
nl_groups=00000000}, msg_namelen=12, msg_iov=[{iov_base={{len=76,
type=DCCPDIAG_GETSOCK, ... --->8----------------  

From: Jakub Kicinski <kuba@kernel.org>
Subject: ss: fix fallback to procfs for raw and sctp sockets

sockdiag_send() diverts to tcpdiag_send() to try the older
netlink interface. tcpdiag_send() works for TCP and DCCP
but not other protocols. Instead of rejecting unsupported
protocols (and missing RAW and SCTP) match on supported ones.

Fixes: 41fe6c34de50 ("ss: Add inet raw sockets information
gathering via netlink diag interface") Signed-off-by: Jakub
Kicinski [off-list ref] ---
 misc/ss.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/misc/ss.c b/misc/ss.c
index 894ad40574f1..b39f63fe3b17 100644
--- a/misc/ss.c
+++ b/misc/ss.c

@@ -3404,13 +3404,13 @@ static int tcpdiag_send(int fd, int

protocol, struct filter *f) struct iovec iov[3];
 	int iovlen = 1;
 
-	if (protocol == IPPROTO_UDP || protocol == IPPROTO_MPTCP)
-		return -1;
-
 	if (protocol == IPPROTO_TCP)
 		req.nlh.nlmsg_type = TCPDIAG_GETSOCK;
-	else
+	else if (protocol == IPPROTO_DCCP)
 		req.nlh.nlmsg_type = DCCPDIAG_GETSOCK;
+	else
+		return -1;
+
 	if (show_mem) {
 		req.r.idiag_ext |= (1<<(INET_DIAG_MEMINFO-1));
 		req.r.idiag_ext |= (1<<(INET_DIAG_SKMEMINFO-1));
  

That looks correct to me.

Jonas: can you build iproute2 and test?

I've cloned branch main as instructed in https://wiki.linuxfoundation.org/networking/iproute2. Most recent commit is 9b7ea92b9e3f. After building, no socket was listed in table.

Then I [manually] applied the patch and rebuilt. The patched version works well, I do see the two sockets right now.

Command was in both cases misc/ss -awp

Awesome, thanks for doing that. Jakub: want to send a formal patch with
his Tested-by?