[bug] networking broke, ssh: connect to port 22: Protocol error

randconfig qa on x86.git ran into the following new networking related 
problem on latest -git: with the attached .config the testbox comes up 
but cannot establish any TCP connections due to -ENOPROTO in 
sys_connect().

The error comes from this condition in inet_stream_connect():

        /* Connection was closed by RST, timeout, ICMP error
         * or another process disconnected us.
         */
        if (sk->sk_state == TCP_CLOSE)
                goto sock_error;

ICMP pings do work to the machine. Netfilter is on in the .config, maybe 
some new option prevents TCP connections from being established?

CONFIG_SECURITY_NETWORK and CONFIG_SECURITY_SMACK is enabled as well. 
(but that shouldnt throw a no-protocol error)

	Ingo

From: Ingo Molnar <redacted>
Date: Wed, 6 Feb 2008 12:38:29 +0100

randconfig qa on x86.git ran into the following new networking related 
problem on latest -git: with the attached .config the testbox comes up 
but cannot establish any TCP connections due to -ENOPROTO in 
sys_connect().

Make sure you have the following fix in your tree.

It might be the cause.

commit 5d8c0aa9433b09387d9021358baef7939f9b32c4
Author: Pavel Emelyanov [off-list ref]
Date:   Tue Feb 5 03:14:44 2008 -0800

    [INET]: Fix accidentally broken inet(6)_hash_connect's port offset calculations.
    
    The port offset calculations depend on the protocol family, but, as
    Adrian noticed, I broke this logic with the commit
    
    	5ee31fc1ecdcbc234c8c56dcacef87c8e09909d8
    	[INET]: Consolidate inet(6)_hash_connect.
    
    Return this logic back, by passing the port offset directly into the
    consolidated function.
    
    Signed-off-by: Pavel Emelyanov [off-list ref]
    Noticed-by: Adrian Bunk [off-list ref]
    Signed-off-by: David S. Miller [off-list ref]

diff --git a/include/net/inet_hashtables.h b/include/net/inet_hashtables.h
index 48ac620..97dc35a 100644
--- a/include/net/inet_hashtables.h
+++ b/include/net/inet_hashtables.h

@@ -389,7 +389,7 @@ static inline struct sock *inet_lookup(struct net *net,
 }
 
 extern int __inet_hash_connect(struct inet_timewait_death_row *death_row,
-		struct sock *sk,
+		struct sock *sk, u32 port_offset,
 		int (*check_established)(struct inet_timewait_death_row *,
 			struct sock *, __u16, struct inet_timewait_sock **),
 			       void (*hash)(struct sock *sk));

diff --git a/net/ipv4/inet_hashtables.c b/net/ipv4/inet_hashtables.c
index 90f422c..9cac6c0 100644
--- a/net/ipv4/inet_hashtables.c
+++ b/net/ipv4/inet_hashtables.c

@@ -398,7 +398,7 @@ out:
 EXPORT_SYMBOL_GPL(inet_unhash);
 
 int __inet_hash_connect(struct inet_timewait_death_row *death_row,
-		struct sock *sk,
+		struct sock *sk, u32 port_offset,
 		int (*check_established)(struct inet_timewait_death_row *,
 			struct sock *, __u16, struct inet_timewait_sock **),
 		void (*hash)(struct sock *sk))

@@ -413,7 +413,7 @@ int __inet_hash_connect(struct inet_timewait_death_row *death_row,
 	if (!snum) {
 		int i, remaining, low, high, port;
 		static u32 hint;
-		u32 offset = hint + inet_sk_port_offset(sk);
+		u32 offset = hint + port_offset;
 		struct hlist_node *node;
 		struct inet_timewait_sock *tw = NULL;
 

@@ -502,7 +502,7 @@ EXPORT_SYMBOL_GPL(__inet_hash_connect);
 int inet_hash_connect(struct inet_timewait_death_row *death_row,
 		      struct sock *sk)
 {
-	return __inet_hash_connect(death_row, sk,
+	return __inet_hash_connect(death_row, sk, inet_sk_port_offset(sk),
 			__inet_check_established, __inet_hash_nolisten);
 }
 

diff --git a/net/ipv6/inet6_hashtables.c b/net/ipv6/inet6_hashtables.c
index 43f3993..99fd25f 100644
--- a/net/ipv6/inet6_hashtables.c
+++ b/net/ipv6/inet6_hashtables.c

@@ -236,7 +236,7 @@ static inline u32 inet6_sk_port_offset(const struct sock *sk)
 int inet6_hash_connect(struct inet_timewait_death_row *death_row,
 		       struct sock *sk)
 {
-	return __inet_hash_connect(death_row, sk,
+	return __inet_hash_connect(death_row, sk, inet6_sk_port_offset(sk),
 			__inet6_check_established, __inet6_hash);
 }
 

* David Miller [off-list ref] wrote:

From: Ingo Molnar <redacted>
Date: Wed, 6 Feb 2008 12:38:29 +0100

quoted

randconfig qa on x86.git ran into the following new networking related 
problem on latest -git: with the attached .config the testbox comes up 
but cannot establish any TCP connections due to -ENOPROTO in 
sys_connect().

Make sure you have the following fix in your tree.

It might be the cause.

commit 5d8c0aa9433b09387d9021358baef7939f9b32c4
Author: Pavel Emelyanov [off-list ref]
Date:   Tue Feb 5 03:14:44 2008 -0800

this is already upstream. As i mentioned above i tested latest -git. 
(HEAD 551e4fb2465b8)

So no, it does not fix the problem. The config i sent is a rather 
generic one, it should boot on most whitebox PCs. TCP connections will 
fail immediately, all the time.

(I reverted 5d8c0aa943 as well, that didnt solve the problem either.)

	Ingo

From: Ingo Molnar <redacted>
Date: Wed, 6 Feb 2008 13:22:48 +0100

So no, it does not fix the problem. The config i sent is a rather 
generic one, it should boot on most whitebox PCs. TCP connections will 
fail immediately, all the time.

I suspect this got added recently with how often and how thoroughly
you test things :-)

If you can only give us the last GIT head that worked on that machine
it might help us narrow things down a lot.

If you have time for a bisect, even better but not absolutely
required.

* David Miller [off-list ref] wrote:

From: Ingo Molnar <redacted>
Date: Wed, 6 Feb 2008 13:22:48 +0100

quoted

So no, it does not fix the problem. The config i sent is a rather 
generic one, it should boot on most whitebox PCs. TCP connections 
will fail immediately, all the time.

I suspect this got added recently with how often and how thoroughly 
you test things :-)

yeah, although various other upstream breakages prevented real long 
randconfig series in the past 2-3 days. I'd say it's either in this pull 
from your tree:

  Author: Linus Torvalds [off-list ref]
  Date:   Tue Feb 5 10:09:07 2008 -0800

    Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6
    * git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6: (21 commits)

or perhaps in this one:

  Author: Linus Torvalds [off-list ref]
  Date:   Mon Feb 4 07:43:36 2008 -0800

    Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6

    * git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6: (77 commits)

i'll figure it out, it's totally reproducible so it should be easy to 
bisect. Just wanted to know whether you had anything queued up already 
for something like this.

	Ingo

* Ingo Molnar [off-list ref] wrote:

yeah, although various other upstream breakages prevented real long 
randconfig series in the past 2-3 days. I'd say it's either in this 
pull from your tree:

ok, i have bisected it down but the result made no sense, so i 
double-checked it and noticed that the .config mutated during the test.

the diff below is the diff between the 'good' and 'bad' .config, with 
this notable detail:

 @@ -2336,7 +2350,7 @@ CONFIG_SECURITY_NETWORK=y
  CONFIG_SECURITY_CAPABILITIES=y
  # CONFIG_SECURITY_FILE_CAPABILITIES is not set
  # CONFIG_SECURITY_ROOTPLUG is not set
 -# CONFIG_SECURITY_SMACK is not set
 +CONFIG_SECURITY_SMACK=y
  CONFIG_XOR_BLOCKS=m
  CONFIG_ASYNC_CORE=m
  CONFIG_ASYNC_MEMCPY=m

so i disabled CONFIG_SECURITY_SMACK, and viola, just 2 hours of hard 
work later networking works on my testbox again :-/

And we have this 1 day old commit:

  commit e114e473771c848c3cfec05f0123e70f1cdbdc99
  Author: Casey Schaufler [off-list ref]
  Date:   Mon Feb 4 22:29:50 2008 -0800

      Smack: Simplified Mandatory Access Control Kernel

that adds SMACK.

So unlike some other security modules like SELINUX, enabling SMACK 
breaks un-aware userspace and breaks TCP networking?

I dont think that's expected behavior - and i'd definitely like to 
enable SMACK in automated tests to check for regressions, etc.

	Ingo

--- .config.good	2008-02-06 14:13:35.000000000 +0100
+++ .config.bad	2008-02-06 14:17:28.000000000 +0100

@@ -1,7 +1,7 @@
 #
 # Automatically generated make config: don't edit
 # Linux kernel version: 2.6.24
-# Wed Feb  6 14:11:27 2008
+# Wed Feb  6 14:15:22 2008
 #
 # CONFIG_64BIT is not set
 CONFIG_X86_32=y

@@ -94,15 +94,16 @@ CONFIG_FUTEX=y
 CONFIG_ANON_INODES=y
 # CONFIG_EPOLL is not set
 CONFIG_SIGNALFD=y
-CONFIG_TIMERFD=y
+# CONFIG_TIMERFD is not set
 CONFIG_EVENTFD=y
 # CONFIG_SHMEM is not set
 # CONFIG_VM_EVENT_COUNTERS is not set
 # CONFIG_SLAB is not set
 # CONFIG_SLUB is not set
 CONFIG_SLOB=y
-# CONFIG_PROFILING is not set
+CONFIG_PROFILING=y
 # CONFIG_MARKERS is not set
+CONFIG_OPROFILE=y
 CONFIG_HAVE_OPROFILE=y
 # CONFIG_KPROBES is not set
 CONFIG_HAVE_KPROBES=y

@@ -691,7 +692,7 @@ CONFIG_MAC80211_RC_DEFAULT=""
 # CONFIG_MAC80211_RC_PID is not set
 # CONFIG_MAC80211_RC_SIMPLE is not set
 # CONFIG_MAC80211_DEBUGFS is not set
-# CONFIG_MAC80211_DEBUG_PACKET_ALIGNMENT is not set
+CONFIG_MAC80211_DEBUG_PACKET_ALIGNMENT=y
 # CONFIG_MAC80211_DEBUG is not set
 CONFIG_IEEE80211=m
 # CONFIG_IEEE80211_DEBUG is not set

@@ -744,6 +745,7 @@ CONFIG_CDROM_PKTCDVD=y
 CONFIG_CDROM_PKTCDVD_BUFFERS=8
 CONFIG_CDROM_PKTCDVD_WCACHE=y
 CONFIG_ATA_OVER_ETH=y
+CONFIG_VIRTIO_BLK=y
 # CONFIG_MISC_DEVICES is not set
 # CONFIG_IDE is not set
 

@@ -796,9 +798,19 @@ CONFIG_BLK_DEV_3W_XXXX_RAID=y
 CONFIG_SCSI_3W_9XXX=m
 CONFIG_SCSI_ACARD=y
 # CONFIG_SCSI_AACRAID is not set
-# CONFIG_SCSI_AIC7XXX is not set
-# CONFIG_SCSI_AIC7XXX_OLD is not set
-# CONFIG_SCSI_AIC79XX is not set
+CONFIG_SCSI_AIC7XXX=y
+CONFIG_AIC7XXX_CMDS_PER_DEVICE=32
+CONFIG_AIC7XXX_RESET_DELAY_MS=5000
+# CONFIG_AIC7XXX_DEBUG_ENABLE is not set
+CONFIG_AIC7XXX_DEBUG_MASK=0
+# CONFIG_AIC7XXX_REG_PRETTY_PRINT is not set
+CONFIG_SCSI_AIC7XXX_OLD=m
+CONFIG_SCSI_AIC79XX=y
+CONFIG_AIC79XX_CMDS_PER_DEVICE=32
+CONFIG_AIC79XX_RESET_DELAY_MS=5000
+CONFIG_AIC79XX_DEBUG_ENABLE=y
+CONFIG_AIC79XX_DEBUG_MASK=0
+# CONFIG_AIC79XX_REG_PRETTY_PRINT is not set
 CONFIG_SCSI_AIC94XX=m
 # CONFIG_AIC94XX_DEBUG is not set
 CONFIG_SCSI_DPT_I2O=m

@@ -1181,6 +1193,7 @@ CONFIG_NETCONSOLE=y
 CONFIG_NETPOLL=y
 # CONFIG_NETPOLL_TRAP is not set
 CONFIG_NET_POLL_CONTROLLER=y
+# CONFIG_VIRTIO_NET is not set
 CONFIG_ISDN=y
 # CONFIG_ISDN_I4L is not set
 # CONFIG_ISDN_CAPI is not set

@@ -2043,7 +2056,8 @@ CONFIG_INFINIBAND_AMSO1100=m
 CONFIG_INFINIBAND_AMSO1100_DEBUG=y
 # CONFIG_INFINIBAND_CXGB3 is not set
 CONFIG_MLX4_INFINIBAND=m
-# CONFIG_INFINIBAND_NES is not set
+CONFIG_INFINIBAND_NES=m
+CONFIG_INFINIBAND_NES_DEBUG=y
 CONFIG_INFINIBAND_IPOIB=m
 CONFIG_INFINIBAND_IPOIB_CM=y
 CONFIG_INFINIBAND_IPOIB_DEBUG=y

@@ -2336,7 +2350,7 @@ CONFIG_SECURITY_NETWORK=y
 CONFIG_SECURITY_CAPABILITIES=y
 # CONFIG_SECURITY_FILE_CAPABILITIES is not set
 # CONFIG_SECURITY_ROOTPLUG is not set
-# CONFIG_SECURITY_SMACK is not set
+CONFIG_SECURITY_SMACK=y
 CONFIG_XOR_BLOCKS=m
 CONFIG_ASYNC_CORE=m
 CONFIG_ASYNC_MEMCPY=m

@@ -2396,7 +2410,9 @@ CONFIG_CRYPTO_AUTHENC=y
 # CONFIG_CRYPTO_HW is not set
 CONFIG_VIRTUALIZATION=y
 # CONFIG_LGUEST is not set
-# CONFIG_VIRTIO_PCI is not set
+CONFIG_VIRTIO=y
+CONFIG_VIRTIO_RING=y
+CONFIG_VIRTIO_PCI=y
 # CONFIG_VIRTIO_BALLOON is not set
 
 #

On Wed, 2008-02-06 at 14:35 +0100, Ingo Molnar wrote:

* Ingo Molnar [off-list ref] wrote:

quoted

yeah, although various other upstream breakages prevented real long 
randconfig series in the past 2-3 days. I'd say it's either in this 
pull from your tree:

ok, i have bisected it down but the result made no sense, so i 
double-checked it and noticed that the .config mutated during the test.

the diff below is the diff between the 'good' and 'bad' .config, with 
this notable detail:

 @@ -2336,7 +2350,7 @@ CONFIG_SECURITY_NETWORK=y
  CONFIG_SECURITY_CAPABILITIES=y
  # CONFIG_SECURITY_FILE_CAPABILITIES is not set
  # CONFIG_SECURITY_ROOTPLUG is not set
 -# CONFIG_SECURITY_SMACK is not set
 +CONFIG_SECURITY_SMACK=y
  CONFIG_XOR_BLOCKS=m
  CONFIG_ASYNC_CORE=m
  CONFIG_ASYNC_MEMCPY=m

so i disabled CONFIG_SECURITY_SMACK, and viola, just 2 hours of hard 
work later networking works on my testbox again :-/

And we have this 1 day old commit:

  commit e114e473771c848c3cfec05f0123e70f1cdbdc99
  Author: Casey Schaufler [off-list ref]
  Date:   Mon Feb 4 22:29:50 2008 -0800

      Smack: Simplified Mandatory Access Control Kernel

that adds SMACK.

So unlike some other security modules like SELINUX, enabling SMACK 
breaks un-aware userspace and breaks TCP networking?

I dont think that's expected behavior - and i'd definitely like to 
enable SMACK in automated tests to check for regressions, etc.

It is expected behavior for Smack due to default use of CIPSO for packet
labeling, see:
http://lkml.org/lkml/2007/10/14/210

-- 
Stephen Smalley
National Security Agency

--- Ingo Molnar <mingo@elte.hu> wrote:

* Ingo Molnar [off-list ref] wrote:

quoted

yeah, although various other upstream breakages prevented real long 
randconfig series in the past 2-3 days. I'd say it's either in this 
pull from your tree:

ok, i have bisected it down but the result made no sense, so i 
double-checked it and noticed that the .config mutated during the test.

the diff below is the diff between the 'good' and 'bad' .config, with 
this notable detail:

 @@ -2336,7 +2350,7 @@ CONFIG_SECURITY_NETWORK=y
  CONFIG_SECURITY_CAPABILITIES=y
  # CONFIG_SECURITY_FILE_CAPABILITIES is not set
  # CONFIG_SECURITY_ROOTPLUG is not set
 -# CONFIG_SECURITY_SMACK is not set
 +CONFIG_SECURITY_SMACK=y
  CONFIG_XOR_BLOCKS=m
  CONFIG_ASYNC_CORE=m
  CONFIG_ASYNC_MEMCPY=m

so i disabled CONFIG_SECURITY_SMACK, and viola, just 2 hours of hard 
work later networking works on my testbox again :-/

And we have this 1 day old commit:

  commit e114e473771c848c3cfec05f0123e70f1cdbdc99
  Author: Casey Schaufler [off-list ref]
  Date:   Mon Feb 4 22:29:50 2008 -0800

      Smack: Simplified Mandatory Access Control Kernel

that adds SMACK.

So unlike some other security modules like SELINUX, enabling SMACK 
breaks un-aware userspace and breaks TCP networking?

I dont think that's expected behavior - and i'd definitely like to 
enable SMACK in automated tests to check for regressions, etc.

As Stephen mentions later, Smack uses CIPSO. sshd does not like
any IP options because of traceroute, and must be built with that
check disabled with the current Smack version. I have been looking
at using unlabeled packets for the "ambient" label, it appears that
doing so would make life simpler. I will get right on it.

Application behavior in the presence of IP options isn't
always what I think it ought to be.


Casey Schaufler
casey@schaufler-ca.com

* Casey Schaufler [off-list ref] wrote:

quoted

So unlike some other security modules like SELINUX, enabling SMACK 
breaks un-aware userspace and breaks TCP networking?

I dont think that's expected behavior - and i'd definitely like to 
enable SMACK in automated tests to check for regressions, etc.

As Stephen mentions later, Smack uses CIPSO. sshd does not like any IP 
options because of traceroute, and must be built with that check 
disabled with the current Smack version. I have been looking at using 
unlabeled packets for the "ambient" label, it appears that doing so 
would make life simpler. I will get right on it.

ok - feel free to send me any patches to test.

	Ingo