From: Andrew Morton <akpm@linux-foundation.org> Date: 2008-08-13 05:12:39
(switched to email. Please respond via emailed reply-to-all, not via the
bugzilla web interface).
On Tue, 12 Aug 2008 22:04:41 -0700 (PDT) bugme-daemon@bugzilla.kernel.org wrote:
http://bugzilla.kernel.org/show_bug.cgi?id=11316
Summary: severe performance regression for iptables nat routing
Product: Networking
Version: 2.5
KernelVersion: 2.6.27-rc3
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: Netfilter/Iptables
AssignedTo: networking_netfilter-iptables@kernel-bugs.osdl.org
ReportedBy: alex.williamson@hp.com
Latest working kernel version: 2.6.26.2
Earliest failing kernel version: 2.6.27-rc2 (maybe earlier)
Distribution: Ubuntu
Hardware Environment: x86_64
Software Environment: 32bit userspace/64bit kernel
Problem Description: When using iptables to intercept addr:port and reroute
through an ssh tunnel, I see a huge performance hit on the 2.6.27-rc series
relative to 2.6.26 (34KB/s vs 1+MB/s).
Steps to reproduce:
Setup and ssh tunnel to one of the kernel.org servers using a system on your
local network:
ssh -L 8888:204.152.191.37:80 <local system>
Leave the ssh session running. In a new terminal (on your local system),
verify performance of direct access versus the tunnel:
wget -O /dev/null
http://204.152.191.37/pub/linux/kernel/v2.6/linux-2.6.26.2.tar.bz2
wget -O /dev/null
http://127.0.0.1:8888/pub/linux/kernel/v2.6/linux-2.6.26.2.tar.bz2
These should be roughly the same. Now setup iptables so that when you try to
access 204.152.191.37:80 you'll automatically be redirected to the ssh tunnel:
sudo iptables -t nat -N bug
sudo iptables -t nat -I OUTPUT 1 -j bug
sudo iptables -t nat -A bug -d 204.152.191.37 -p tcp --dport 80 -j DNAT
--to-destination 127.0.0.1:8888
Repeat the performance test:
wget -O /dev/null
http://204.152.191.37/pub/linux/kernel/v2.6/linux-2.6.26.2.tar.bz2
wget -O /dev/null
http://127.0.0.1:8888/pub/linux/kernel/v2.6/linux-2.6.26.2.tar.bz2
On 2.6.27-rc2+ My rate quickly drops down to ~34KB/s using the iptables nat'd
wget (204.152.191.37) while the ssh tunnel still runs 1+MB/s. On 2.6.26 I get
similar performance for both paths.
From: Alex Williamson <hidden> Date: 2008-08-14 02:14:21
git bisect traced the problem back to this changeset:
commit e5a4a72d4f88f4389e9340d383ca67031d1b8536
Author: Lennert Buytenhek [off-list ref]
Date: Sun Aug 3 01:23:10 2008 -0700
net: use software GSO for SG+CSUM capable netdevices
I've verified that I can toggle the slowness by reverting this patch on
top of 8d0968ab (current head). The problem is readily reproducible
using Ubuntu Hardy in a KVM VM with upstream, defconfig kernel.
On Tue, 2008-08-12 at 22:12 -0700, Andrew Morton wrote:
(switched to email. Please respond via emailed reply-to-all, not via the
bugzilla web interface).
On Tue, 12 Aug 2008 22:04:41 -0700 (PDT) bugme-daemon@bugzilla.kernel.org wrote:
quoted
http://bugzilla.kernel.org/show_bug.cgi?id=11316
Summary: severe performance regression for iptables nat routing
Product: Networking
Version: 2.5
KernelVersion: 2.6.27-rc3
Platform: All
OS/Version: Linux
Tree: Mainline
Status: NEW
Severity: high
Priority: P1
Component: Netfilter/Iptables
AssignedTo: networking_netfilter-iptables@kernel-bugs.osdl.org
ReportedBy: alex.williamson@hp.com
Latest working kernel version: 2.6.26.2
Earliest failing kernel version: 2.6.27-rc2 (maybe earlier)
Distribution: Ubuntu
Hardware Environment: x86_64
Software Environment: 32bit userspace/64bit kernel
Problem Description: When using iptables to intercept addr:port and reroute
through an ssh tunnel, I see a huge performance hit on the 2.6.27-rc series
relative to 2.6.26 (34KB/s vs 1+MB/s).
Steps to reproduce:
Setup and ssh tunnel to one of the kernel.org servers using a system on your
local network:
ssh -L 8888:204.152.191.37:80 <local system>
Leave the ssh session running. In a new terminal (on your local system),
verify performance of direct access versus the tunnel:
wget -O /dev/null
http://204.152.191.37/pub/linux/kernel/v2.6/linux-2.6.26.2.tar.bz2
wget -O /dev/null
http://127.0.0.1:8888/pub/linux/kernel/v2.6/linux-2.6.26.2.tar.bz2
These should be roughly the same. Now setup iptables so that when you try to
access 204.152.191.37:80 you'll automatically be redirected to the ssh tunnel:
sudo iptables -t nat -N bug
sudo iptables -t nat -I OUTPUT 1 -j bug
sudo iptables -t nat -A bug -d 204.152.191.37 -p tcp --dport 80 -j DNAT
--to-destination 127.0.0.1:8888
Repeat the performance test:
wget -O /dev/null
http://204.152.191.37/pub/linux/kernel/v2.6/linux-2.6.26.2.tar.bz2
wget -O /dev/null
http://127.0.0.1:8888/pub/linux/kernel/v2.6/linux-2.6.26.2.tar.bz2
On 2.6.27-rc2+ My rate quickly drops down to ~34KB/s using the iptables nat'd
wget (204.152.191.37) while the ssh tunnel still runs 1+MB/s. On 2.6.26 I get
similar performance for both paths.
From: David Miller <davem@davemloft.net> Date: 2008-08-14 02:20:59
From: Alex Williamson <redacted>
Date: Wed, 13 Aug 2008 20:08:20 -0600
git bisect traced the problem back to this changeset:
commit e5a4a72d4f88f4389e9340d383ca67031d1b8536
Author: Lennert Buytenhek [off-list ref]
Date: Sun Aug 3 01:23:10 2008 -0700
net: use software GSO for SG+CSUM capable netdevices
I've verified that I can toggle the slowness by reverting this patch on
top of 8d0968ab (current head). The problem is readily reproducible
using Ubuntu Hardy in a KVM VM with upstream, defconfig kernel.
Patrick I wonder if there a case where iptables NAT will COW the packet
when it really doesn't need to.
It seems, if anything, using GSO should make things go a little bit
faster not slower... Hmmm...
Anyways, if we can't figure this one out soon we can easily revert.
From: Patrick McHardy <hidden> Date: 2008-08-14 11:04:28
David Miller wrote:
From: Alex Williamson <redacted>
Date: Wed, 13 Aug 2008 20:08:20 -0600
quoted
git bisect traced the problem back to this changeset:
commit e5a4a72d4f88f4389e9340d383ca67031d1b8536
Author: Lennert Buytenhek [off-list ref]
Date: Sun Aug 3 01:23:10 2008 -0700
net: use software GSO for SG+CSUM capable netdevices
I've verified that I can toggle the slowness by reverting this patch on
top of 8d0968ab (current head). The problem is readily reproducible
using Ubuntu Hardy in a KVM VM with upstream, defconfig kernel.
Patrick I wonder if there a case where iptables NAT will COW the packet
when it really doesn't need to.
I don't think so, its using skb_make_writable everywhere, which checks
for skb_clone_writable, which should usually avoid COWing local TCP
packets. It would also be unlikely to have that much of a performance
impact (1MB/s -> 34kb/s).
It seems, if anything, using GSO should make things go a little bit
faster not slower... Hmmm...
Alex, could you post a tcpdump from both loopback and the outgoing
device from the machine you're doing NAT on?
From: Alex Williamson <hidden> Date: 2008-08-14 15:08:12
On Thu, 2008-08-14 at 13:04 +0200, Patrick McHardy wrote:
I don't think so, its using skb_make_writable everywhere, which checks
for skb_clone_writable, which should usually avoid COWing local TCP
packets. It would also be unlikely to have that much of a performance
impact (1MB/s -> 34kb/s).
quoted
It seems, if anything, using GSO should make things go a little bit
faster not slower... Hmmm...
Alex, could you post a tcpdump from both loopback and the outgoing
device from the machine you're doing NAT on?
Attached, let me know if you want more options, this is just -vv -n.
The NAT'ing system is at 10.0.2.15 and the ssh tunnel target is
192.168.1.60. Thanks,
Alex
--
Alex Williamson HP Open Source & Linux Org.
From: David Miller <davem@davemloft.net> Date: 2008-08-14 22:00:54
From: Patrick McHardy <redacted>
Date: Thu, 14 Aug 2008 13:04:25 +0200
David Miller wrote:
quoted
Patrick I wonder if there a case where iptables NAT will COW the packet
when it really doesn't need to.
I don't think so, its using skb_make_writable everywhere, which checks
for skb_clone_writable, which should usually avoid COWing local TCP
packets. It would also be unlikely to have that much of a performance
impact (1MB/s -> 34kb/s).
I think he is NAT'ing locally generated traffic, look at the bugzilla
entry.
He has two cases of the same wget transfer, one is direct and another
uses a 127.0.0.1:XXXX URL that does the transfer over an SSH tunnel.
Normally they go roughly at the same rate.
Then he adds iptables NAT entries that redirect the first transfer
case over the SSH tunnel addr/port. And it is this case that degrades
in performance with the GSO changeset.
So it is locally generated TCP traffic, NAT'd to another port and IP
address (specifically, redirected to 127.0.0.1:8888).
Perhaps the problem has something to do with the fact that as far as
TCP is concerned, the destination device can do SG and CSUM and thus
GSO. But then iptables NATs this traffic to loopback. I think that
is what leads to some kind of slowpath.
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2008-08-15 04:44:38
Alex Williamson [off-list ref] wrote:
Attached, let me know if you want more options, this is just -vv -n.
The NAT'ing system is at 10.0.2.15 and the ssh tunnel target is
192.168.1.60. Thanks,
Right, the underlying TCP connection is going well, but the NATed
connection is getting checksum errors. Please send us the raw
packet dump on lo (tcpdump -s 1600 -w file) so we can see what's
wrong.
Actually, I think know what's going on but a raw packet dump should
confirm whether we're getting a partial checksum.
Thanks,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2008-08-15 05:36:01
On Fri, Aug 15, 2008 at 02:44:26PM +1000, Herbert Xu wrote:
Actually, I think know what's going on but a raw packet dump should
confirm whether we're getting a partial checksum.
Nevermind, I think I've found the problem.
loopback: Drop obsolete ip_summed setting
Now that the network stack can handle inbound packets with partial
checksums, we should no longer clobber the ip_summed field in the
loopback driver. This is because CHECKSUM_UNNECESSARY implies that
the checksum field is actually valid which is not true for loopback
packets since it's only partial (and thus complemented).
This allows packets from lo to then be SNATed to an external source
while still preserving the checksum's validity.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
--
From: Alex Williamson <hidden> Date: 2008-08-15 05:50:15
On Fri, 2008-08-15 at 15:35 +1000, Herbert Xu wrote:
On Fri, Aug 15, 2008 at 02:44:26PM +1000, Herbert Xu wrote:
quoted
Actually, I think know what's going on but a raw packet dump should
confirm whether we're getting a partial checksum.
Nevermind, I think I've found the problem.
loopback: Drop obsolete ip_summed setting
Now that the network stack can handle inbound packets with partial
checksums, we should no longer clobber the ip_summed field in the
loopback driver. This is because CHECKSUM_UNNECESSARY implies that
the checksum field is actually valid which is not true for loopback
packets since it's only partial (and thus complemented).
This allows packets from lo to then be SNATed to an external source
while still preserving the checksum's validity.
Nope, that doesn't fix it. NAT'd throughput remains about the same.
Thanks,
Alex
--
Alex Williamson HP Open Source & Linux Org.
From: David Miller <davem@davemloft.net> Date: 2008-08-15 21:56:56
From: Herbert Xu <herbert@gondor.apana.org.au>
Date: Fri, 15 Aug 2008 15:35:48 +1000
loopback: Drop obsolete ip_summed setting
Now that the network stack can handle inbound packets with partial
checksums, we should no longer clobber the ip_summed field in the
loopback driver. This is because CHECKSUM_UNNECESSARY implies that
the checksum field is actually valid which is not true for loopback
packets since it's only partial (and thus complemented).
This allows packets from lo to then be SNATed to an external source
while still preserving the checksum's validity.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
I've applied this one too, let me know if I should not have :)