From: Chris Clayton <hidden> Date: 2012-09-17 15:44:18
Hi,
I'm having a problem with networking. I'm running Windows XP as a KVM
guest on a laptop running kernel 3.6.0-rc6. The identical configuration
works fine with kernels 3.5.4 and 3.4.11 (and has done so, largely
unchanged, since since KVM was introduced in 2.6.<whatever>.)
The configuration is:
XP guest: 192.168.200.1 (gateway 192.168.200.254)
tap0: 192.168.200.254
host: 192.168.0.40 (gateway 192.168.0.1)
router: 192.168.0.1
The script that starts up the firewall includes the following commands:
# Load the connection-sharing for qemu/kvm guests
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
...
# allow traffic to and from the qemu/kvm virtual networks
NETS="200 201"
for net in $NETS; do
iptables -A INPUT -s 192.168.$net.0/24 -j ACCEPT
iptables -A OUTPUT -d 192.168.$net.0/24 -j ACCEPT
done
...
The network-related modules that are loaded are:
$ lsmod
Module Size Used by
tun 12412 0
xt_state 891 1
iptable_filter 852 1
ipt_MASQUERADE 1222 1
iptable_nat 3087 1
nf_nat 10901 2 ipt_MASQUERADE,iptable_nat
nf_conntrack_ipv4 4942 4 nf_nat,iptable_nat
nf_defrag_ipv4 815 1 nf_conntrack_ipv4
nf_conntrack 37644 5
ipt_MASQUERADE,nf_nat,xt_state,iptable_nat,nf_conntrack_ipv4
...
r8169 47159 0
From the host I can successfully ping the guest, tap0 and the router as
you would expect, but from the guest, although I can ping the host and
tap0, I cannot ping the router. In practice, this means I have no
internet access from the guest. As I say, this configuration works
perfectly under 3.5.x and 3.4.x kernels.
I'll do a coarse-grained "bisect" of Linus' 3.6 release candidates and
report back, but does anyone have any prime-suspect patches that may be
at the cause of this problem?
Let me know if there are any other diagnostics I can provide. Also, as
I'm not subscribed to netdev, please cc me to any reply.
Thanks,
Chris
From: Chris Clayton <hidden> Date: 2012-09-18 14:21:29
On 09/17/12 16:44, Chris Clayton wrote:
Hi,
I'm having a problem with networking. I'm running Windows XP as a KVM
guest on a laptop running kernel 3.6.0-rc6. The identical configuration
works fine with kernels 3.5.4 and 3.4.11 (and has done so, largely
unchanged, since since KVM was introduced in 2.6.<whatever>.)
The configuration is:
XP guest: 192.168.200.1 (gateway 192.168.200.254)
tap0: 192.168.200.254
host: 192.168.0.40 (gateway 192.168.0.1)
router: 192.168.0.1
The script that starts up the firewall includes the following commands:
# Load the connection-sharing for qemu/kvm guests
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
...
# allow traffic to and from the qemu/kvm virtual networks
NETS="200 201"
for net in $NETS; do
iptables -A INPUT -s 192.168.$net.0/24 -j ACCEPT
iptables -A OUTPUT -d 192.168.$net.0/24 -j ACCEPT
done
...
The network-related modules that are loaded are:
$ lsmod
Module Size Used by
tun 12412 0
xt_state 891 1
iptable_filter 852 1
ipt_MASQUERADE 1222 1
iptable_nat 3087 1
nf_nat 10901 2 ipt_MASQUERADE,iptable_nat
nf_conntrack_ipv4 4942 4 nf_nat,iptable_nat
nf_defrag_ipv4 815 1 nf_conntrack_ipv4
nf_conntrack 37644 5
ipt_MASQUERADE,nf_nat,xt_state,iptable_nat,nf_conntrack_ipv4
...
r8169 47159 0
From the host I can successfully ping the guest, tap0 and the router as
you would expect, but from the guest, although I can ping the host and
tap0, I cannot ping the router. In practice, this means I have no
internet access from the guest. As I say, this configuration works
perfectly under 3.5.x and 3.4.x kernels.
I'll do a coarse-grained "bisect" of Linus' 3.6 release candidates and
report back, but does anyone have any prime-suspect patches that may be
at the cause of this problem?
-rc1 turned out to have the problem so I've bisected between 3.5 and
3.6-rc1. I arrived at:
$ git bisect bad
d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5 is the first bad commit
commit d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5
Author: David S. Miller [off-list ref]
Date: Tue Jul 17 12:58:50 2012 -0700
ipv4: Cache input routes in fib_info nexthops.
Caching input routes is slightly simpler than output routes, since we
don't need to be concerned with nexthop exceptions. (locally
destined, and routed packets, never trigger PMTU events or redirects
that will be processed by us).
However, we have to elide caching for the DIRECTSRC and non-zero itag
cases.
Signed-off-by: David S. Miller [off-list ref]
:040000 040000 6bbc75c1cbe62bf84ea412d3b98adf2b614779cd
3ad7256b4a71e63ca4530977c0550121ea803d35 M include
:040000 040000 18c2a950a53c4eec9bfa12185d1e382dfed74af8
a2ab6157d6cd54930da395758c6ded3a225d1f04 M net
The bisect log:
git bisect start
# bad: [0d7614f09c1ebdbaa1599a5aba7593f147bf96ee] Linux 3.6-rc1
git bisect bad 0d7614f09c1ebdbaa1599a5aba7593f147bf96ee
# good: [28a33cbc24e4256c143dce96c7d93bf423229f92] Linux 3.5
git bisect good 28a33cbc24e4256c143dce96c7d93bf423229f92
# bad: [614a6d4341b3760ca98a1c2c09141b71db5d1e90] Merge branch 'for-3.6'
of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
git bisect bad 614a6d4341b3760ca98a1c2c09141b71db5d1e90
# bad: [320f5ea0cedc08ef65d67e056bcb9d181386ef2c] genetlink: define
lockdep_genl_is_held() when CONFIG_LOCKDEP
git bisect bad 320f5ea0cedc08ef65d67e056bcb9d181386ef2c
# good: [0cd06647b7c24f6633e32a505930a9aa70138c22] Merge branch 'master'
of git://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth-next
git bisect good 0cd06647b7c24f6633e32a505930a9aa70138c22
# good: [dbfa600148a25903976910863c75dae185f8d187] cxgb3: set maximal
number of default RSS queues
git bisect good dbfa600148a25903976910863c75dae185f8d187
# good: [efdfad3205403e1d1c5c0bdcbdb647ddd89bfaa3] bnx2: Try to recover
from PCI block reset
git bisect good efdfad3205403e1d1c5c0bdcbdb647ddd89bfaa3
# good: [1bf91cdc1bba94ea062a9147d924815c13f029f2] ixgbe: Drop
references to deprecated pci_ DMA api and instead use dma_ API
git bisect good 1bf91cdc1bba94ea062a9147d924815c13f029f2
# good: [b6dfd939fdc249fcf8cd7b8006f76239b33eb581] ixgbe: add support
for new 82599 device
git bisect good b6dfd939fdc249fcf8cd7b8006f76239b33eb581
# good: [3ba97381343b271296487bf073eb670d5465a8b8] net: ethernet:
davinci_emac: add pm_runtime support
git bisect good 3ba97381343b271296487bf073eb670d5465a8b8
# bad: [5e9965c15ba88319500284e590733f4a4629a288] Merge branch
'kill_rtcache'
git bisect bad 5e9965c15ba88319500284e590733f4a4629a288
# good: [f5b0a8743601a4477419171f5046bd07d1c080a0] net: Document
dst->obsolete better.
git bisect good f5b0a8743601a4477419171f5046bd07d1c080a0
# bad: [ba3f7f04ef2b19aace38f855aedd17fe43035d50] ipv4: Kill
FLOWI_FLAG_RT_NOCACHE and associated code.
git bisect bad ba3f7f04ef2b19aace38f855aedd17fe43035d50
# good: [f2bb4bedf35d5167a073dcdddf16543f351ef3ae] ipv4: Cache output
routes in fib_info nexthops.
git bisect good f2bb4bedf35d5167a073dcdddf16543f351ef3ae
# bad: [d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5] ipv4: Cache input
routes in fib_info nexthops.
git bisect bad d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5
Checking out the parent commit
(f2bb4bedf35d5167a073dcdddf16543f351ef3ae) and building and installing
the kernel gives a working configuration, so I'm pretty confident in the
outcome of the bisect. Reversing the patch gives errors, so I've not
tested master with the patch reversed.
Let me know if I can help in any way to identify a fix.
Chris
Let me know if there are any other diagnostics I can provide. Also, as
I'm not subscribed to netdev, please cc me to any reply.
Thanks,
Chris
From: Chris Clayton <hidden> Date: 2012-09-18 14:31:48
quoted
...
r8169 47159 0
From the host I can successfully ping the guest, tap0 and the router as
you would expect, but from the guest, although I can ping the host and
tap0, I cannot ping the router. In practice, this means I have no
internet access from the guest. As I say, this configuration works
perfectly under 3.5.x and 3.4.x kernels.
I'll do a coarse-grained "bisect" of Linus' 3.6 release candidates and
report back, but does anyone have any prime-suspect patches that may be
at the cause of this problem?
-rc1 turned out to have the problem so I've bisected between 3.5 and
3.6-rc1. I arrived at:
$ git bisect bad
d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5 is the first bad commit
commit d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5
Author: David S. Miller [off-list ref]
Date: Tue Jul 17 12:58:50 2012 -0700
ipv4: Cache input routes in fib_info nexthops.
Caching input routes is slightly simpler than output routes, since we
don't need to be concerned with nexthop exceptions. (locally
destined, and routed packets, never trigger PMTU events or redirects
that will be processed by us).
However, we have to elide caching for the DIRECTSRC and non-zero itag
cases.
Signed-off-by: David S. Miller [off-list ref]
:040000 040000 6bbc75c1cbe62bf84ea412d3b98adf2b614779cd
3ad7256b4a71e63ca4530977c0550121ea803d35 M include
:040000 040000 18c2a950a53c4eec9bfa12185d1e382dfed74af8
a2ab6157d6cd54930da395758c6ded3a225d1f04 M net
The bisect log:
git bisect start
# bad: [0d7614f09c1ebdbaa1599a5aba7593f147bf96ee] Linux 3.6-rc1
git bisect bad 0d7614f09c1ebdbaa1599a5aba7593f147bf96ee
# good: [28a33cbc24e4256c143dce96c7d93bf423229f92] Linux 3.5
git bisect good 28a33cbc24e4256c143dce96c7d93bf423229f92
# bad: [614a6d4341b3760ca98a1c2c09141b71db5d1e90] Merge branch 'for-3.6'
of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
git bisect bad 614a6d4341b3760ca98a1c2c09141b71db5d1e90
# bad: [320f5ea0cedc08ef65d67e056bcb9d181386ef2c] genetlink: define
lockdep_genl_is_held() when CONFIG_LOCKDEP
git bisect bad 320f5ea0cedc08ef65d67e056bcb9d181386ef2c
# good: [0cd06647b7c24f6633e32a505930a9aa70138c22] Merge branch 'master'
of git://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth-next
git bisect good 0cd06647b7c24f6633e32a505930a9aa70138c22
# good: [dbfa600148a25903976910863c75dae185f8d187] cxgb3: set maximal
number of default RSS queues
git bisect good dbfa600148a25903976910863c75dae185f8d187
# good: [efdfad3205403e1d1c5c0bdcbdb647ddd89bfaa3] bnx2: Try to recover
from PCI block reset
git bisect good efdfad3205403e1d1c5c0bdcbdb647ddd89bfaa3
# good: [1bf91cdc1bba94ea062a9147d924815c13f029f2] ixgbe: Drop
references to deprecated pci_ DMA api and instead use dma_ API
git bisect good 1bf91cdc1bba94ea062a9147d924815c13f029f2
# good: [b6dfd939fdc249fcf8cd7b8006f76239b33eb581] ixgbe: add support
for new 82599 device
git bisect good b6dfd939fdc249fcf8cd7b8006f76239b33eb581
# good: [3ba97381343b271296487bf073eb670d5465a8b8] net: ethernet:
davinci_emac: add pm_runtime support
git bisect good 3ba97381343b271296487bf073eb670d5465a8b8
# bad: [5e9965c15ba88319500284e590733f4a4629a288] Merge branch
'kill_rtcache'
git bisect bad 5e9965c15ba88319500284e590733f4a4629a288
# good: [f5b0a8743601a4477419171f5046bd07d1c080a0] net: Document
dst->obsolete better.
git bisect good f5b0a8743601a4477419171f5046bd07d1c080a0
# bad: [ba3f7f04ef2b19aace38f855aedd17fe43035d50] ipv4: Kill
FLOWI_FLAG_RT_NOCACHE and associated code.
git bisect bad ba3f7f04ef2b19aace38f855aedd17fe43035d50
# good: [f2bb4bedf35d5167a073dcdddf16543f351ef3ae] ipv4: Cache output
routes in fib_info nexthops.
git bisect good f2bb4bedf35d5167a073dcdddf16543f351ef3ae
# bad: [d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5] ipv4: Cache input
routes in fib_info nexthops.
git bisect bad d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5
Checking out the parent commit
(f2bb4bedf35d5167a073dcdddf16543f351ef3ae) and building and installing
the kernel gives a working configuration, so I'm pretty confident in the
outcome of the bisect. Reversing the patch gives errors, so I've not
tested master with the patch reversed.
Let me know if I can help in any way to identify a fix.
Sorry, I forgot to say that I also have tried running TinyCore Linux as
a KVM guest on a 3.6.0-rc6 kernel, and I can ping the router fine, so
the problem seems to be something specifically related to ruuning
Windows XP as the guest. I don't have any other guests installed so
that's as much as I can say, although I could maybe install a Win7 guest
tomorrow if that would help.
Chris
quoted
Let me know if there are any other diagnostics I can provide. Also, as
I'm not subscribed to netdev, please cc me to any reply.
Thanks,
Chris
From: Eric Dumazet <hidden> Date: 2012-09-18 14:40:47
On Tue, 2012-09-18 at 15:31 +0100, Chris Clayton wrote:
quoted
quoted
...
r8169 47159 0
From the host I can successfully ping the guest, tap0 and the router as
you would expect, but from the guest, although I can ping the host and
tap0, I cannot ping the router. In practice, this means I have no
internet access from the guest. As I say, this configuration works
perfectly under 3.5.x and 3.4.x kernels.
I'll do a coarse-grained "bisect" of Linus' 3.6 release candidates and
report back, but does anyone have any prime-suspect patches that may be
at the cause of this problem?
-rc1 turned out to have the problem so I've bisected between 3.5 and
3.6-rc1. I arrived at:
$ git bisect bad
d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5 is the first bad commit
commit d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5
Author: David S. Miller [off-list ref]
Date: Tue Jul 17 12:58:50 2012 -0700
ipv4: Cache input routes in fib_info nexthops.
Caching input routes is slightly simpler than output routes, since we
don't need to be concerned with nexthop exceptions. (locally
destined, and routed packets, never trigger PMTU events or redirects
that will be processed by us).
However, we have to elide caching for the DIRECTSRC and non-zero itag
cases.
Signed-off-by: David S. Miller [off-list ref]
:040000 040000 6bbc75c1cbe62bf84ea412d3b98adf2b614779cd
3ad7256b4a71e63ca4530977c0550121ea803d35 M include
:040000 040000 18c2a950a53c4eec9bfa12185d1e382dfed74af8
a2ab6157d6cd54930da395758c6ded3a225d1f04 M net
The bisect log:
git bisect start
# bad: [0d7614f09c1ebdbaa1599a5aba7593f147bf96ee] Linux 3.6-rc1
git bisect bad 0d7614f09c1ebdbaa1599a5aba7593f147bf96ee
# good: [28a33cbc24e4256c143dce96c7d93bf423229f92] Linux 3.5
git bisect good 28a33cbc24e4256c143dce96c7d93bf423229f92
# bad: [614a6d4341b3760ca98a1c2c09141b71db5d1e90] Merge branch 'for-3.6'
of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
git bisect bad 614a6d4341b3760ca98a1c2c09141b71db5d1e90
# bad: [320f5ea0cedc08ef65d67e056bcb9d181386ef2c] genetlink: define
lockdep_genl_is_held() when CONFIG_LOCKDEP
git bisect bad 320f5ea0cedc08ef65d67e056bcb9d181386ef2c
# good: [0cd06647b7c24f6633e32a505930a9aa70138c22] Merge branch 'master'
of git://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth-next
git bisect good 0cd06647b7c24f6633e32a505930a9aa70138c22
# good: [dbfa600148a25903976910863c75dae185f8d187] cxgb3: set maximal
number of default RSS queues
git bisect good dbfa600148a25903976910863c75dae185f8d187
# good: [efdfad3205403e1d1c5c0bdcbdb647ddd89bfaa3] bnx2: Try to recover
from PCI block reset
git bisect good efdfad3205403e1d1c5c0bdcbdb647ddd89bfaa3
# good: [1bf91cdc1bba94ea062a9147d924815c13f029f2] ixgbe: Drop
references to deprecated pci_ DMA api and instead use dma_ API
git bisect good 1bf91cdc1bba94ea062a9147d924815c13f029f2
# good: [b6dfd939fdc249fcf8cd7b8006f76239b33eb581] ixgbe: add support
for new 82599 device
git bisect good b6dfd939fdc249fcf8cd7b8006f76239b33eb581
# good: [3ba97381343b271296487bf073eb670d5465a8b8] net: ethernet:
davinci_emac: add pm_runtime support
git bisect good 3ba97381343b271296487bf073eb670d5465a8b8
# bad: [5e9965c15ba88319500284e590733f4a4629a288] Merge branch
'kill_rtcache'
git bisect bad 5e9965c15ba88319500284e590733f4a4629a288
# good: [f5b0a8743601a4477419171f5046bd07d1c080a0] net: Document
dst->obsolete better.
git bisect good f5b0a8743601a4477419171f5046bd07d1c080a0
# bad: [ba3f7f04ef2b19aace38f855aedd17fe43035d50] ipv4: Kill
FLOWI_FLAG_RT_NOCACHE and associated code.
git bisect bad ba3f7f04ef2b19aace38f855aedd17fe43035d50
# good: [f2bb4bedf35d5167a073dcdddf16543f351ef3ae] ipv4: Cache output
routes in fib_info nexthops.
git bisect good f2bb4bedf35d5167a073dcdddf16543f351ef3ae
# bad: [d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5] ipv4: Cache input
routes in fib_info nexthops.
git bisect bad d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5
Checking out the parent commit
(f2bb4bedf35d5167a073dcdddf16543f351ef3ae) and building and installing
the kernel gives a working configuration, so I'm pretty confident in the
outcome of the bisect. Reversing the patch gives errors, so I've not
tested master with the patch reversed.
Let me know if I can help in any way to identify a fix.
Sorry, I forgot to say that I also have tried running TinyCore Linux as
a KVM guest on a 3.6.0-rc6 kernel, and I can ping the router fine, so
the problem seems to be something specifically related to ruuning
Windows XP as the guest. I don't have any other guests installed so
that's as much as I can say, although I could maybe install a Win7 guest
tomorrow if that would help.
It would help to have some traffic sample, maybe.
Especially if the problem is not easily reproductible for us.
(I dont have Windows XP nor Win7)
Also the bisect might point to a commit with an already fixed bug :
commit 4331debc51ee1ce319f4a389484e0e8e05de2aca
Author: Eric Dumazet [off-list ref]
Date: Wed Jul 25 05:11:23 2012 +0000
ipv4: rt_cache_valid must check expired routes
commit d2d68ba9fe8 (ipv4: Cache input routes in fib_info nexthops.)
introduced rt_cache_valid() helper. It unfortunately doesn't check if
route is expired before caching it.
I noticed sk_setup_caps() was constantly called on a tcp workload.
Signed-off-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
From: Chris Clayton <hidden> Date: 2012-09-18 14:44:01
quoted
Sorry, I forgot to say that I also have tried running TinyCore Linux as
a KVM guest on a 3.6.0-rc6 kernel, and I can ping the router fine, so
the problem seems to be something specifically related to ruuning
Windows XP as the guest. I don't have any other guests installed so
that's as much as I can say, although I could maybe install a Win7 guest
tomorrow if that would help.
Sorry again, but ignore the message above, please. Wrong kernel used in
test. In fact, I get the same failure to ping the router running on a
6.6.0-rc6 kernel.
Apologies for the noise.
Chris
From: Chris Clayton <hidden> Date: 2012-09-18 15:51:15
Thanks for the reply, Eric.
quoted
quoted
-rc1 turned out to have the problem so I've bisected between 3.5 and
3.6-rc1. I arrived at:
$ git bisect bad
d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5 is the first bad commit
commit d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5
Author: David S. Miller [off-list ref]
Date: Tue Jul 17 12:58:50 2012 -0700
ipv4: Cache input routes in fib_info nexthops.
Caching input routes is slightly simpler than output routes, since we
don't need to be concerned with nexthop exceptions. (locally
destined, and routed packets, never trigger PMTU events or redirects
that will be processed by us).
However, we have to elide caching for the DIRECTSRC and non-zero itag
cases.
Signed-off-by: David S. Miller [off-list ref]
:040000 040000 6bbc75c1cbe62bf84ea412d3b98adf2b614779cd
3ad7256b4a71e63ca4530977c0550121ea803d35 M include
:040000 040000 18c2a950a53c4eec9bfa12185d1e382dfed74af8
a2ab6157d6cd54930da395758c6ded3a225d1f04 M net
The bisect log:
git bisect start
# bad: [0d7614f09c1ebdbaa1599a5aba7593f147bf96ee] Linux 3.6-rc1
git bisect bad 0d7614f09c1ebdbaa1599a5aba7593f147bf96ee
# good: [28a33cbc24e4256c143dce96c7d93bf423229f92] Linux 3.5
git bisect good 28a33cbc24e4256c143dce96c7d93bf423229f92
# bad: [614a6d4341b3760ca98a1c2c09141b71db5d1e90] Merge branch 'for-3.6'
of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
git bisect bad 614a6d4341b3760ca98a1c2c09141b71db5d1e90
# bad: [320f5ea0cedc08ef65d67e056bcb9d181386ef2c] genetlink: define
lockdep_genl_is_held() when CONFIG_LOCKDEP
git bisect bad 320f5ea0cedc08ef65d67e056bcb9d181386ef2c
# good: [0cd06647b7c24f6633e32a505930a9aa70138c22] Merge branch 'master'
of git://git.kernel.org/pub/scm/linux/kernel/git/bluetooth/bluetooth-next
git bisect good 0cd06647b7c24f6633e32a505930a9aa70138c22
# good: [dbfa600148a25903976910863c75dae185f8d187] cxgb3: set maximal
number of default RSS queues
git bisect good dbfa600148a25903976910863c75dae185f8d187
# good: [efdfad3205403e1d1c5c0bdcbdb647ddd89bfaa3] bnx2: Try to recover
from PCI block reset
git bisect good efdfad3205403e1d1c5c0bdcbdb647ddd89bfaa3
# good: [1bf91cdc1bba94ea062a9147d924815c13f029f2] ixgbe: Drop
references to deprecated pci_ DMA api and instead use dma_ API
git bisect good 1bf91cdc1bba94ea062a9147d924815c13f029f2
# good: [b6dfd939fdc249fcf8cd7b8006f76239b33eb581] ixgbe: add support
for new 82599 device
git bisect good b6dfd939fdc249fcf8cd7b8006f76239b33eb581
# good: [3ba97381343b271296487bf073eb670d5465a8b8] net: ethernet:
davinci_emac: add pm_runtime support
git bisect good 3ba97381343b271296487bf073eb670d5465a8b8
# bad: [5e9965c15ba88319500284e590733f4a4629a288] Merge branch
'kill_rtcache'
git bisect bad 5e9965c15ba88319500284e590733f4a4629a288
# good: [f5b0a8743601a4477419171f5046bd07d1c080a0] net: Document
dst->obsolete better.
git bisect good f5b0a8743601a4477419171f5046bd07d1c080a0
# bad: [ba3f7f04ef2b19aace38f855aedd17fe43035d50] ipv4: Kill
FLOWI_FLAG_RT_NOCACHE and associated code.
git bisect bad ba3f7f04ef2b19aace38f855aedd17fe43035d50
# good: [f2bb4bedf35d5167a073dcdddf16543f351ef3ae] ipv4: Cache output
routes in fib_info nexthops.
git bisect good f2bb4bedf35d5167a073dcdddf16543f351ef3ae
# bad: [d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5] ipv4: Cache input
routes in fib_info nexthops.
git bisect bad d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5
Checking out the parent commit
(f2bb4bedf35d5167a073dcdddf16543f351ef3ae) and building and installing
the kernel gives a working configuration, so I'm pretty confident in the
outcome of the bisect. Reversing the patch gives errors, so I've not
tested master with the patch reversed.
Let me know if I can help in any way to identify a fix.
Sorry, I forgot to say that I also have tried running TinyCore Linux as
a KVM guest on a 3.6.0-rc6 kernel, and I can ping the router fine, so
the problem seems to be something specifically related to ruuning
Windows XP as the guest. I don't have any other guests installed so
that's as much as I can say, although I could maybe install a Win7 guest
tomorrow if that would help.
I hope you've seen my later email in which I reported my error in my
testing that led me to believe that all was OK with a linux client. In
fact, The router is inaccessible from both the Windows XP and the Linux
clients.
It would help to have some traffic sample, maybe.
I'll need help here. How would I go about collecting that traffic. I
have wireshark installed, but haven't used it for years. Would a trace
from that be helpful? It might take me a while to figure out how to
capture it?
Especially if the problem is not easily reproductible for us.
(I dont have Windows XP nor Win7)
Also the bisect might point to a commit with an already fixed bug :
This fix is already in 3.6.0-rc6. BTW, I've pulled the latest changes
from kernel.org this afternoon, but that hasn't helped.
commit 4331debc51ee1ce319f4a389484e0e8e05de2aca
Author: Eric Dumazet [off-list ref]
Date: Wed Jul 25 05:11:23 2012 +0000
ipv4: rt_cache_valid must check expired routes
commit d2d68ba9fe8 (ipv4: Cache input routes in fib_info nexthops.)
introduced rt_cache_valid() helper. It unfortunately doesn't check if
route is expired before caching it.
I noticed sk_setup_caps() was constantly called on a tcp workload.
Signed-off-by: Eric Dumazet [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
From: Chris Clayton <hidden> Date: 2012-09-22 06:26:36
I guess you network developer folks are either very busy or this
regression is proving a bit troublesome to identify, so I've opened a
bugzilla report to keep track of it. The report number is 47761.
Chris
On 09/19/12 16:26, Chris Clayton wrote:
quoted
It would help to have some traffic sample, maybe.
Especially if the problem is not easily reproductible for us.
From: Chris Clayton <hidden> Date: 2012-09-27 11:50:59
Just for information - I've pulled Linus' tree this morning and the
problem is still present. Also, Gunther Piaz has reported, via the
bugzilla entry, that he too has hit this regression.
On 09/22/12 07:26, Chris Clayton wrote:
I guess you network developer folks are either very busy or this
regression is proving a bit troublesome to identify, so I've opened a
bugzilla report to keep track of it. The report number is 47761.
Chris
On 09/19/12 16:26, Chris Clayton wrote:
quoted
quoted
It would help to have some traffic sample, maybe.
Especially if the problem is not easily reproductible for us.
From: Eric Dumazet <hidden> Date: 2012-09-27 12:14:07
On Thu, 2012-09-27 at 12:50 +0100, Chris Clayton wrote:
Just for information - I've pulled Linus' tree this morning and the
problem is still present. Also, Gunther Piaz has reported, via the
bugzilla entry, that he too has hit this regression.
I tried to reproduce the bug, and my kvm guests have no problem.
I guess you need to precisely describe how you setup your network, so
that I can reproduce the problem and eventually fix it.
Thanks
From: Chris Clayton <hidden> Date: 2012-09-27 18:05:28
On 09/27/12 13:14, Eric Dumazet wrote:
On Thu, 2012-09-27 at 12:50 +0100, Chris Clayton wrote:
quoted
Just for information - I've pulled Linus' tree this morning and the
problem is still present. Also, Gunther Piaz has reported, via the
bugzilla entry, that he too has hit this regression.
I tried to reproduce the bug, and my kvm guests have no problem.
I guess you need to precisely describe how you setup your network, so
that I can reproduce the problem and eventually fix it.
You've seen the bits from my firewall setup script that relate to this
issue. I start the WinXP client with another script:
#!/bin/sh
if [ -e $HOME/kvm/var/run/kvm-winxp.pid ]; then
echo "winxp is already running ..." > /dev/stderr
exit 1
fi
# make sure the kvm modules are loaded
if test -z "$(grep '\<kvm\>' /proc/misc)"; then
sudo modprobe kvm-intel
while test -z "$(grep '\<kvm\>' /proc/misc)"; do
true
done
fi
# make sure tun module is loaded
if test ! -e /dev/net/tun; then
sudo modprobe tun
fi
# figure out the cpu to use
QVER=$(qemu-kvm --version | cut -d' ' -f 4 | sed 's/,/./')
# assumes major version is 1
MINORVER=$(echo $QVER | cut -d'.' -f 2)
if [ $MINORVER -ge 1 ]; then
CPU="host"
else
CPU="qemu64"
fi
# set up the network interface
TAPDEV=$(sudo tunctl -b -u $(whoami))
sudo ifconfig $TAPDEV 192.168.200.254 netmask 255.255.255.0 broadcast
192.168.200.255
# start Windows XP
qemu-kvm -drive file=$HOME/kvm/winxp.qcow2,index=0,cache=none,if=virtio
-cpu $CPU -smp cores=1,threads=2 -soundhw es1370 \
-m 768 -net nic,model=virtio,macaddr=$(getmacaddr) -net
tap,ifname=$TAPDEV -startdate $(date +%Y-%m-%dT%H:%M:%S) \
-name kxplaptop -pidfile $HOME/kvm/var/run/kvm-winxp.pid $*
# stop the network interface
sudo ifconfig $TAPDEV down
sudo tunctl -d $TAPDEV &>/dev/null
# tidy up
rm -f $HOME/kvm/var/run/kvm-winxp.pid
The call to getmacaddr just returns the next in a sequence of mac
addresses. qemu-kvm is a symlink to /usr/bin/qemu-system-i386. I first
found the problem whilst running qemu-kvm version 1.1.1 although I've
since updated to 1.2.0.
By the way, I doubt it will make a difference, but, although my laptop
has a 64bit CPU, I am running a 32 bit kernel and, obviously, user space.
Let me know if you need anything else.
Thanks
From: Eric Dumazet <hidden> Date: 2012-09-27 21:03:51
On Thu, 2012-09-27 at 19:05 +0100, Chris Clayton wrote:
On 09/27/12 13:14, Eric Dumazet wrote:
quoted
On Thu, 2012-09-27 at 12:50 +0100, Chris Clayton wrote:
quoted
Just for information - I've pulled Linus' tree this morning and the
problem is still present. Also, Gunther Piaz has reported, via the
bugzilla entry, that he too has hit this regression.
I tried to reproduce the bug, and my kvm guests have no problem.
I guess you need to precisely describe how you setup your network, so
that I can reproduce the problem and eventually fix it.
You've seen the bits from my firewall setup script that relate to this
issue. I start the WinXP client with another script:
#!/bin/sh
if [ -e $HOME/kvm/var/run/kvm-winxp.pid ]; then
echo "winxp is already running ..." > /dev/stderr
exit 1
fi
# make sure the kvm modules are loaded
if test -z "$(grep '\<kvm\>' /proc/misc)"; then
sudo modprobe kvm-intel
while test -z "$(grep '\<kvm\>' /proc/misc)"; do
true
done
fi
# make sure tun module is loaded
if test ! -e /dev/net/tun; then
sudo modprobe tun
fi
# figure out the cpu to use
QVER=$(qemu-kvm --version | cut -d' ' -f 4 | sed 's/,/./')
# assumes major version is 1
MINORVER=$(echo $QVER | cut -d'.' -f 2)
if [ $MINORVER -ge 1 ]; then
CPU="host"
else
CPU="qemu64"
fi
# set up the network interface
TAPDEV=$(sudo tunctl -b -u $(whoami))
sudo ifconfig $TAPDEV 192.168.200.254 netmask 255.255.255.0 broadcast
192.168.200.255
# start Windows XP
qemu-kvm -drive file=$HOME/kvm/winxp.qcow2,index=0,cache=none,if=virtio
-cpu $CPU -smp cores=1,threads=2 -soundhw es1370 \
-m 768 -net nic,model=virtio,macaddr=$(getmacaddr) -net
tap,ifname=$TAPDEV -startdate $(date +%Y-%m-%dT%H:%M:%S) \
-name kxplaptop -pidfile $HOME/kvm/var/run/kvm-winxp.pid $*
# stop the network interface
sudo ifconfig $TAPDEV down
sudo tunctl -d $TAPDEV &>/dev/null
# tidy up
rm -f $HOME/kvm/var/run/kvm-winxp.pid
The call to getmacaddr just returns the next in a sequence of mac
addresses. qemu-kvm is a symlink to /usr/bin/qemu-system-i386. I first
found the problem whilst running qemu-kvm version 1.1.1 although I've
since updated to 1.2.0.
By the way, I doubt it will make a difference, but, although my laptop
has a 64bit CPU, I am running a 32 bit kernel and, obviously, user space.
Let me know if you need anything else.
It works for me.
Hmm, maybe your guest is using DHCP and DHCP fails ?
Could you check ?
From: Eric Dumazet <hidden> Date: 2012-09-27 21:17:11
On Thu, 2012-09-27 at 23:03 +0200, Eric Dumazet wrote:
On Thu, 2012-09-27 at 19:05 +0100, Chris Clayton wrote:
quoted
On 09/27/12 13:14, Eric Dumazet wrote:
quoted
On Thu, 2012-09-27 at 12:50 +0100, Chris Clayton wrote:
quoted
Just for information - I've pulled Linus' tree this morning and the
problem is still present. Also, Gunther Piaz has reported, via the
bugzilla entry, that he too has hit this regression.
I tried to reproduce the bug, and my kvm guests have no problem.
I guess you need to precisely describe how you setup your network, so
that I can reproduce the problem and eventually fix it.
You've seen the bits from my firewall setup script that relate to this
issue. I start the WinXP client with another script:
#!/bin/sh
if [ -e $HOME/kvm/var/run/kvm-winxp.pid ]; then
echo "winxp is already running ..." > /dev/stderr
exit 1
fi
# make sure the kvm modules are loaded
if test -z "$(grep '\<kvm\>' /proc/misc)"; then
sudo modprobe kvm-intel
while test -z "$(grep '\<kvm\>' /proc/misc)"; do
true
done
fi
# make sure tun module is loaded
if test ! -e /dev/net/tun; then
sudo modprobe tun
fi
# figure out the cpu to use
QVER=$(qemu-kvm --version | cut -d' ' -f 4 | sed 's/,/./')
# assumes major version is 1
MINORVER=$(echo $QVER | cut -d'.' -f 2)
if [ $MINORVER -ge 1 ]; then
CPU="host"
else
CPU="qemu64"
fi
# set up the network interface
TAPDEV=$(sudo tunctl -b -u $(whoami))
sudo ifconfig $TAPDEV 192.168.200.254 netmask 255.255.255.0 broadcast
192.168.200.255
# start Windows XP
qemu-kvm -drive file=$HOME/kvm/winxp.qcow2,index=0,cache=none,if=virtio
-cpu $CPU -smp cores=1,threads=2 -soundhw es1370 \
-m 768 -net nic,model=virtio,macaddr=$(getmacaddr) -net
tap,ifname=$TAPDEV -startdate $(date +%Y-%m-%dT%H:%M:%S) \
-name kxplaptop -pidfile $HOME/kvm/var/run/kvm-winxp.pid $*
# stop the network interface
sudo ifconfig $TAPDEV down
sudo tunctl -d $TAPDEV &>/dev/null
# tidy up
rm -f $HOME/kvm/var/run/kvm-winxp.pid
The call to getmacaddr just returns the next in a sequence of mac
addresses. qemu-kvm is a symlink to /usr/bin/qemu-system-i386. I first
found the problem whilst running qemu-kvm version 1.1.1 although I've
since updated to 1.2.0.
By the way, I doubt it will make a difference, but, although my laptop
has a 64bit CPU, I am running a 32 bit kernel and, obviously, user space.
Let me know if you need anything else.
It works for me.
Hmm, maybe your guest is using DHCP and DHCP fails ?
Yes it seems the problem. On the host I tried :
# ip ro get 8.8.8.8 from 192.168.200.1 iif tap1
8.8.8.8 from 192.168.200.1 via 172.30.42.1 dev eth0
cache iif *
So if the guest tries to send a frame to 8.8.8.8 we are going to forward
the packet to eth0
But if the guest tries to send to 255.255.255.255, we try to deliver the
packet to the host itself, instead of broadcasting to eth0
# ip ro get 255.255.255.255 from 192.168.200.1 iif tap1
broadcast 255.255.255.255 from 192.168.200.1 dev lo
cache <local,brd> iif *
David, maybe you'll have an idea ?
Thanks
Yes it seems the problem. On the host I tried :
# ip ro get 8.8.8.8 from 192.168.200.1 iif tap1
8.8.8.8 from 192.168.200.1 via 172.30.42.1 dev eth0
cache iif *
So if the guest tries to send a frame to 8.8.8.8 we are going to forward
the packet to eth0
But if the guest tries to send to 255.255.255.255, we try to deliver the
packet to the host itself, instead of broadcasting to eth0
# ip ro get 255.255.255.255 from 192.168.200.1 iif tap1
broadcast 255.255.255.255 from 192.168.200.1 dev lo
cache <local,brd> iif *
David, maybe you'll have an idea ?
Perhaps this was introduced by:
commit 7bd86cc282a458b66c41e3f6676de6656c99b8db
Author: Yan, Zheng [off-list ref]
Date: Sun Aug 12 20:09:59 2012 +0000
ipv4: Cache local output routes
Commit caacf05e5ad1abf causes big drop of UDP loop back performance.
The cause of the regression is that we do not cache the local output
routes. Each time we send a datagram from unconnected UDP socket,
the kernel allocates a dst_entry and adds it to the rt_uncached_list.
It creates lock contention on the rt_uncached_lock.
Reported-by: Alex Shi [off-list ref]
Signed-off-by: Yan, Zheng [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
Yes it seems the problem. On the host I tried :
# ip ro get 8.8.8.8 from 192.168.200.1 iif tap1
8.8.8.8 from 192.168.200.1 via 172.30.42.1 dev eth0
cache iif *
So if the guest tries to send a frame to 8.8.8.8 we are going to forward
the packet to eth0
But if the guest tries to send to 255.255.255.255, we try to deliver the
packet to the host itself, instead of broadcasting to eth0
# ip ro get 255.255.255.255 from 192.168.200.1 iif tap1
broadcast 255.255.255.255 from 192.168.200.1 dev lo
cache <local,brd> iif *
David, maybe you'll have an idea ?
Perhaps this was introduced by:
Thanks, David.
Unfortunately, reversing that patch does not fix the problem. The pings
from the KVM client to the router still time out.
I have bisected this (see
http://marc.info/?l=linux-netdev&m=134797809611847&w=2) and that rendered:
$ git bisect bad
d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5 is the first bad commit
commit d2d68ba9fe8b38eb03124b3176a013bb8aa2b5e5
Author: David S. Miller [off-list ref]
Date: Tue Jul 17 12:58:50 2012 -0700
ipv4: Cache input routes in fib_info nexthops.
Caching input routes is slightly simpler than output routes, since we
don't need to be concerned with nexthop exceptions. (locally
destined, and routed packets, never trigger PMTU events or redirects
that will be processed by us).
However, we have to elide caching for the DIRECTSRC and non-zero itag
cases.
Signed-off-by: David S. Miller [off-list ref]
:040000 040000 6bbc75c1cbe62bf84ea412d3b98adf2b614779cd
3ad7256b4a71e63ca4530977c0550121ea803d35 M include
:040000 040000 18c2a950a53c4eec9bfa12185d1e382dfed74af8
a2ab6157d6cd54930da395758c6ded3a225d1f04 M net
Unfortunately, the related patches don't reverse cleanly, but a kernel
built from a git checkout of the parent commit (
f2bb4bedf35d5167a073dcdddf16543f351ef3ae) works fine.
quoted hunk
commit 7bd86cc282a458b66c41e3f6676de6656c99b8db
Author: Yan, Zheng [off-list ref]
Date: Sun Aug 12 20:09:59 2012 +0000
ipv4: Cache local output routes
Commit caacf05e5ad1abf causes big drop of UDP loop back performance.
The cause of the regression is that we do not cache the local output
routes. Each time we send a datagram from unconnected UDP socket,
the kernel allocates a dst_entry and adds it to the rt_uncached_list.
It creates lock contention on the rt_uncached_lock.
Reported-by: Alex Shi [off-list ref]
Signed-off-by: Yan, Zheng [off-list ref]
Signed-off-by: David S. Miller [off-list ref]
From: Chris Clayton <hidden> Date: 2012-09-28 09:22:19
On 09/27/12 22:17, Eric Dumazet wrote:
On Thu, 2012-09-27 at 23:03 +0200, Eric Dumazet wrote:
quoted
On Thu, 2012-09-27 at 19:05 +0100, Chris Clayton wrote:
quoted
On 09/27/12 13:14, Eric Dumazet wrote:
quoted
On Thu, 2012-09-27 at 12:50 +0100, Chris Clayton wrote:
quoted
Just for information - I've pulled Linus' tree this morning and the
problem is still present. Also, Gunther Piaz has reported, via the
bugzilla entry, that he too has hit this regression.
I tried to reproduce the bug, and my kvm guests have no problem.
I guess you need to precisely describe how you setup your network, so
that I can reproduce the problem and eventually fix it.
You've seen the bits from my firewall setup script that relate to this
issue. I start the WinXP client with another script:
#!/bin/sh
if [ -e $HOME/kvm/var/run/kvm-winxp.pid ]; then
echo "winxp is already running ..." > /dev/stderr
exit 1
fi
# make sure the kvm modules are loaded
if test -z "$(grep '\<kvm\>' /proc/misc)"; then
sudo modprobe kvm-intel
while test -z "$(grep '\<kvm\>' /proc/misc)"; do
true
done
fi
# make sure tun module is loaded
if test ! -e /dev/net/tun; then
sudo modprobe tun
fi
# figure out the cpu to use
QVER=$(qemu-kvm --version | cut -d' ' -f 4 | sed 's/,/./')
# assumes major version is 1
MINORVER=$(echo $QVER | cut -d'.' -f 2)
if [ $MINORVER -ge 1 ]; then
CPU="host"
else
CPU="qemu64"
fi
# set up the network interface
TAPDEV=$(sudo tunctl -b -u $(whoami))
sudo ifconfig $TAPDEV 192.168.200.254 netmask 255.255.255.0 broadcast
192.168.200.255
# start Windows XP
qemu-kvm -drive file=$HOME/kvm/winxp.qcow2,index=0,cache=none,if=virtio
-cpu $CPU -smp cores=1,threads=2 -soundhw es1370 \
-m 768 -net nic,model=virtio,macaddr=$(getmacaddr) -net
tap,ifname=$TAPDEV -startdate $(date +%Y-%m-%dT%H:%M:%S) \
-name kxplaptop -pidfile $HOME/kvm/var/run/kvm-winxp.pid $*
# stop the network interface
sudo ifconfig $TAPDEV down
sudo tunctl -d $TAPDEV &>/dev/null
# tidy up
rm -f $HOME/kvm/var/run/kvm-winxp.pid
The call to getmacaddr just returns the next in a sequence of mac
addresses. qemu-kvm is a symlink to /usr/bin/qemu-system-i386. I first
found the problem whilst running qemu-kvm version 1.1.1 although I've
since updated to 1.2.0.
By the way, I doubt it will make a difference, but, although my laptop
has a 64bit CPU, I am running a 32 bit kernel and, obviously, user space.
Let me know if you need anything else.
It works for me.
Hmm, maybe your guest is using DHCP and DHCP fails ?
No, the WinXP guest is configured with a fixed IP address
(192.168.200.1). Subnet mask is 255.255.255.0, and default gateway is
192.168.200.254. DNS is 192.168.0.1.
Yes it seems the problem. On the host I tried :
# ip ro get 8.8.8.8 from 192.168.200.1 iif tap1
8.8.8.8 from 192.168.200.1 via 172.30.42.1 dev eth0
cache iif *
So if the guest tries to send a frame to 8.8.8.8 we are going to forward
the packet to eth0
But if the guest tries to send to 255.255.255.255, we try to deliver the
packet to the host itself, instead of broadcasting to eth0
# ip ro get 255.255.255.255 from 192.168.200.1 iif tap1
broadcast 255.255.255.255 from 192.168.200.1 dev lo
cache <local,brd> iif *
David, maybe you'll have an idea ?
Thanks
From: Eric Dumazet <hidden> Date: 2012-09-28 11:26:37
On Fri, 2012-09-28 at 10:22 +0100, Chris Clayton wrote:
No, the WinXP guest is configured with a fixed IP address
(192.168.200.1). Subnet mask is 255.255.255.0, and default gateway is
192.168.200.254. DNS is 192.168.0.1.
I have no problem with such a setup, with a linux guest.
Could you send again a tcpdump, but including link-level header ?
(option -e)
Ideally, you could send two traces, one taken on tap0, and another taken
on eth0.
From: Chris Clayton <hidden> Date: 2012-09-28 14:28:26
On 09/28/12 12:26, Eric Dumazet wrote:
On Fri, 2012-09-28 at 10:22 +0100, Chris Clayton wrote:
quoted
No, the WinXP guest is configured with a fixed IP address
(192.168.200.1). Subnet mask is 255.255.255.0, and default gateway is
192.168.200.254. DNS is 192.168.0.1.
I have no problem with such a setup, with a linux guest.
Could you send again a tcpdump, but including link-level header ?
(option -e)
Ideally, you could send two traces, one taken on tap0, and another taken
on eth0.
From: Chris Clayton <hidden> Date: 2012-09-30 15:26:05
Hi Eric,
On 09/28/12 12:26, Eric Dumazet wrote:
On Fri, 2012-09-28 at 10:22 +0100, Chris Clayton wrote:
quoted
No, the WinXP guest is configured with a fixed IP address
(192.168.200.1). Subnet mask is 255.255.255.0, and default gateway is
192.168.200.254. DNS is 192.168.0.1.
I have no problem with such a setup, with a linux guest.
Could you send again a tcpdump, but including link-level header ?
(option -e)
Ideally, you could send two traces, one taken on tap0, and another taken
on eth0.
From: Eric Dumazet <hidden> Date: 2012-09-30 19:45:35
On Sun, 2012-09-30 at 16:26 +0100, Chris Clayton wrote:
Hi Eric,
On 09/28/12 12:26, Eric Dumazet wrote:
quoted
On Fri, 2012-09-28 at 10:22 +0100, Chris Clayton wrote:
quoted
No, the WinXP guest is configured with a fixed IP address
(192.168.200.1). Subnet mask is 255.255.255.0, and default gateway is
192.168.200.254. DNS is 192.168.0.1.
I have no problem with such a setup, with a linux guest.
Could you send again a tcpdump, but including link-level header ?
(option -e)
Ideally, you could send two traces, one taken on tap0, and another taken
on eth0.
Below are two more traces that I think may well be more useful than
those I sent on Friday. They are taken with tcpdump directly (after some
reading up on that application) rather than tcpdump translations of pcap
files captured with netsniff-ng. Also, they are taken concurrently, so
they show the traffic on tap0 and eth0 at the time of an unsuccessful
attempt to ping the router from the WinXP KVM client. The command was:
sudo tcpdump -nev -i eth0 -Z chris >eth0.trace
Could you send "netstat -s" before/after your tests ?
From: Chris Clayton <hidden> Date: 2012-10-01 08:36:03
On 09/30/12 20:45, Eric Dumazet wrote:
On Sun, 2012-09-30 at 16:26 +0100, Chris Clayton wrote:
quoted
Hi Eric,
On 09/28/12 12:26, Eric Dumazet wrote:
quoted
On Fri, 2012-09-28 at 10:22 +0100, Chris Clayton wrote:
quoted
No, the WinXP guest is configured with a fixed IP address
(192.168.200.1). Subnet mask is 255.255.255.0, and default gateway is
192.168.200.254. DNS is 192.168.0.1.
I have no problem with such a setup, with a linux guest.
Could you send again a tcpdump, but including link-level header ?
(option -e)
Ideally, you could send two traces, one taken on tap0, and another taken
on eth0.
Below are two more traces that I think may well be more useful than
those I sent on Friday. They are taken with tcpdump directly (after some
reading up on that application) rather than tcpdump translations of pcap
files captured with netsniff-ng. Also, they are taken concurrently, so
they show the traffic on tap0 and eth0 at the time of an unsuccessful
attempt to ping the router from the WinXP KVM client. The command was:
sudo tcpdump -nev -i eth0 -Z chris >eth0.trace
Could you send "netstat -s" before/after your tests ?
Before:
$ netstat -s
Ip:
485 total packets received
10 forwarded
0 incoming packets discarded
473 incoming packets delivered
383 requests sent out
Icmp:
0 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
0 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
Tcp:
12 active connections openings
0 passive connection openings
6 failed connection attempts
0 connection resets received
5 connections established
374 segments received
306 segments send out
0 segments retransmited
0 bad segments received.
6 resets sent
Udp:
164 packets received
0 packets to unknown port received.
0 packet receive errors
67 packets sent
RcvbufErrors: 0
SndbufErrors: 0
UdpLite:
InDatagrams: 0
NoPorts: 0
InErrors: 0
OutDatagrams: 0
RcvbufErrors: 0
SndbufErrors: 0
error parsing /proc/net/snmp: Success
After:
$ netstat -s
Ip:
519 total packets received
21 forwarded
0 incoming packets discarded
496 incoming packets delivered
406 requests sent out
Icmp:
4 ICMP messages received
4 input ICMP message failed.
ICMP input histogram:
echo replies: 4
0 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
IcmpMsg:
InType0: 4
Tcp:
13 active connections openings
0 passive connection openings
6 failed connection attempts
0 connection resets received
5 connections established
381 segments received
316 segments send out
0 segments retransmited
0 bad segments received.
6 resets sent
Udp:
173 packets received
0 packets to unknown port received.
0 packet receive errors
69 packets sent
RcvbufErrors: 0
SndbufErrors: 0
UdpLite:
InDatagrams: 0
NoPorts: 0
InErrors: 0
OutDatagrams: 0
RcvbufErrors: 0
SndbufErrors: 0
error parsing /proc/net/snmp: Success
So icmp replies come back and are delivered to host instead of being
forwarded.
I wonder if MASQUERADE broke...
Could you send
iptables -t -nat -nvL
conntrack -L # while ping is running from guest
So icmp replies come back and are delivered to host instead of being
forwarded.
I wonder if MASQUERADE broke...
Could you send
iptables -t -nat -nvL
$ iptables -t -nat -nvL
iptables v1.4.15: can't initialize iptables table `-nat': Table does not
exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
conntrack -L # while ping is running from guest
$ conntrack -L
conntrack v1.2.2 (conntrack-tools): Operation failed: invalid parameters
Forgive me for asking, but why is the problem not down to the change
that I identified by bisecting? The title of the patch is "ipv4: Cache
local output routes" and, although I'm a million miles from being an
expert here, to me it does make it look a good candidate.
http://marc.info/?l=linux-netdev&m=134797809611847&w=2
So icmp replies come back and are delivered to host instead of being
forwarded.
I wonder if MASQUERADE broke...
Could you send
iptables -t -nat -nvL
$ iptables -t -nat -nvL
iptables v1.4.15: can't initialize iptables table `-nat': Table does not
exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
Thats not expected, you described you used MASQUERADE target, so
"iptables -t nat -nvL" should display something.
Forgive me for asking, but why is the problem not down to the change
that I identified by bisecting? The title of the patch is "ipv4: Cache
local output routes" and, although I'm a million miles from being an
expert here, to me it does make it look a good candidate.
http://marc.info/?l=linux-netdev&m=134797809611847&w=2
Because I cant reproduce your problem at all, using your setup.
So icmp replies come back and are delivered to host instead of being
forwarded.
I wonder if MASQUERADE broke...
Could you send
iptables -t -nat -nvL
$ iptables -t -nat -nvL
iptables v1.4.15: can't initialize iptables table `-nat': Table does not
exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
Thats not expected, you described you used MASQUERADE target, so
"iptables -t nat -nvL" should display something.
To check this I've booted a 3.5.4 kernel. I get the same response to the
two commands. I also double checked that, with a 3.5.4 kernel, pinging
the router and browsing the internet from the client work and they do.
Except for the packets and bytes columns, the command iptables -nvL
gives the following output under both 3.5.4 and 3.6.0 kernels:
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
3757 3240K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
state RELATED,ESTABLISHED
14 840 ACCEPT all -- * * 127.0.0.1 127.0.0.1
41 4362 ACCEPT all -- * * 192.168.0.0/24 0.0.0.0/0
90 12780 ACCEPT all -- * * 192.168.200.0/24 0.0.0.0/0
0 0 ACCEPT all -- * * 192.168.201.0/24 0.0.0.0/0
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT 4470 packets, 3065K bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 3243 packets, 349K bytes)
pkts bytes target prot opt in out source destination
64 8344 ACCEPT all -- * * 0.0.0.0/0 192.168.200.0/24
0 0 ACCEPT all -- * * 0.0.0.0/0 192.168.201.0/24
quoted
Forgive me for asking, but why is the problem not down to the change
that I identified by bisecting? The title of the patch is "ipv4: Cache
local output routes" and, although I'm a million miles from being an
expert here, to me it does make it look a good candidate.
http://marc.info/?l=linux-netdev&m=134797809611847&w=2
Because I cant reproduce your problem at all, using your setup.
So icmp replies come back and are delivered to host instead of being
forwarded.
I wonder if MASQUERADE broke...
Could you send
iptables -t -nat -nvL
$ iptables -t -nat -nvL
iptables v1.4.15: can't initialize iptables table `-nat': Table does not
exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
Thats not expected, you described you used MASQUERADE target, so
"iptables -t nat -nvL" should display something.
To check this I've booted a 3.5.4 kernel. I get the same response to the
two commands. I also double checked that, with a 3.5.4 kernel, pinging
the router and browsing the internet from the client work and they do.
Except for the packets and bytes columns, the command iptables -nvL
gives the following output under both 3.5.4 and 3.6.0 kernels:
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
3757 3240K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
state RELATED,ESTABLISHED
14 840 ACCEPT all -- * * 127.0.0.1 127.0.0.1
41 4362 ACCEPT all -- * * 192.168.0.0/24 0.0.0.0/0
90 12780 ACCEPT all -- * * 192.168.200.0/24 0.0.0.0/0
0 0 ACCEPT all -- * * 192.168.201.0/24 0.0.0.0/0
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT 4470 packets, 3065K bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 3243 packets, 349K bytes)
pkts bytes target prot opt in out source destination
64 8344 ACCEPT all -- * * 0.0.0.0/0 192.168.200.0/24
0 0 ACCEPT all -- * * 0.0.0.0/0 192.168.201.0/24
I am lost, since n your first mail you said :
-----------------------------------------------------------------------------
# Load the connection-sharing for qemu/kvm guests
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
...
# allow traffic to and from the qemu/kvm virtual networks
NETS="200 201"
for net in $NETS; do
iptables -A INPUT -s 192.168.$net.0/24 -j ACCEPT
iptables -A OUTPUT -d 192.168.$net.0/24 -j ACCEPT
done
...
The network-related modules that are loaded are:
$ lsmod
Module Size Used by
tun 12412 0
xt_state 891 1
iptable_filter 852 1
ipt_MASQUERADE 1222 1
iptable_nat 3087 1
nf_nat 10901 2 ipt_MASQUERADE,iptable_nat
nf_conntrack_ipv4 4942 4 nf_nat,iptable_nat
nf_defrag_ipv4 815 1 nf_conntrack_ipv4
nf_conntrack 37644 5
ipt_MASQUERADE,nf_nat,xt_state,iptable_nat,nf_conntrack_ipv4
...
r8169 47159 0
-----------------------------------------------
Now you say you dont have nat ?
Something is wrong.
So icmp replies come back and are delivered to host instead of being
forwarded.
I wonder if MASQUERADE broke...
Could you send
iptables -t -nat -nvL
$ iptables -t -nat -nvL
iptables v1.4.15: can't initialize iptables table `-nat': Table does not
exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
Thats not expected, you described you used MASQUERADE target, so
"iptables -t nat -nvL" should display something.
To check this I've booted a 3.5.4 kernel. I get the same response to the
two commands. I also double checked that, with a 3.5.4 kernel, pinging
the router and browsing the internet from the client work and they do.
Except for the packets and bytes columns, the command iptables -nvL
gives the following output under both 3.5.4 and 3.6.0 kernels:
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
3757 3240K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0
state RELATED,ESTABLISHED
14 840 ACCEPT all -- * * 127.0.0.1 127.0.0.1
41 4362 ACCEPT all -- * * 192.168.0.0/24 0.0.0.0/0
90 12780 ACCEPT all -- * * 192.168.200.0/24 0.0.0.0/0
0 0 ACCEPT all -- * * 192.168.201.0/24 0.0.0.0/0
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy ACCEPT 4470 packets, 3065K bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 3243 packets, 349K bytes)
pkts bytes target prot opt in out source destination
64 8344 ACCEPT all -- * * 0.0.0.0/0 192.168.200.0/24
0 0 ACCEPT all -- * * 0.0.0.0/0 192.168.201.0/24
I am lost, since n your first mail you said :
-----------------------------------------------------------------------------
# Load the connection-sharing for qemu/kvm guests
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
...
# allow traffic to and from the qemu/kvm virtual networks
NETS="200 201"
for net in $NETS; do
iptables -A INPUT -s 192.168.$net.0/24 -j ACCEPT
iptables -A OUTPUT -d 192.168.$net.0/24 -j ACCEPT
done
...
The network-related modules that are loaded are:
$ lsmod
Module Size Used by
tun 12412 0
xt_state 891 1
iptable_filter 852 1
ipt_MASQUERADE 1222 1
iptable_nat 3087 1
nf_nat 10901 2 ipt_MASQUERADE,iptable_nat
nf_conntrack_ipv4 4942 4 nf_nat,iptable_nat
nf_defrag_ipv4 815 1 nf_conntrack_ipv4
nf_conntrack 37644 5
ipt_MASQUERADE,nf_nat,xt_state,iptable_nat,nf_conntrack_ipv4
...
r8169 47159 0
-----------------------------------------------
Now you say you dont have nat ?
Something is wrong.
Here's the complete script that starts up my firewall. I can't recall
having changed this at all for two or three years, other than when a
replacement router changed the network from 192.168.1.x or I add (or
remove) other networks to (from) the $NETS list for other KVM clients
$ cat /etc/rc.d/rc.firewall
#! /bin/sh
case "$1" in
stop)
echo 0 > /proc/sys/net/ipv4/ip_forward
# clear out the current settings
iptables -F
iptables -X
iptables -Z
;;
start)
# Load the connection-sharing for qemu/kvm guests
echo 1 > /proc/sys/net/ipv4/ip_forward
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Allow anything internal to this machine (i.e. localhost)
# is this really necessary?
iptables -A INPUT -s 127.0.0.1 -d 127.0.0.1 -j ACCEPT
# Allow any traffic from nodes on home network
iptables -A INPUT -s 192.168.0.0/24 -j ACCEPT
# and traffic to and from the qemu/kvm virtual networks
NETS="200 201"
for net in $NETS; do
iptables -A INPUT -s 192.168.$net.0/24 -j ACCEPT
iptables -A OUTPUT -d 192.168.$net.0/24 -j ACCEPT
done
# drop everything else
# iptables -A INPUT -j LOG --log-level 4 --log-prefix "FIREWALL: "
iptables -A INPUT -j DROP
;;
restart|reload)
$0 stop
$0 start
;;
status)
iptables -L
;;
*)
echo "Usage: $0 {start|stop|restart|reload|status}"
exit 1
;;
esac
eth0 is set up by calling /sbin/ifup from udev on the add event for eth0
(wlan0 is disabled on the laptop, so that won't be getting in the way).
Here's the script (the SSID is not really XXXXX:
$ cat /sbin/ifup
#!/bin/sh
PATH="/usr/bin:/usr/sbin:/sbin:/bin"
export PATH
SSID=XXXXX
#logger "$0 called with arguments $@"
if [ "$1" = "wlan0" ]; then
# Bring the interface up before the iwconfig stuff below
# assign ip address later else association with AP fails when using WPA
ifconfig wlan0 up
# Configure the wireless adapter
iw wlan0 connect $SSID
# start wpa_supplicant
if [ -z `pgrep wpa_supplicant` ]; then
wpa_supplicant -c/etc/wpa_supplicant/wpa_supplicant.conf
-iwlan0 -Dwext -B -f/var/log/wpa_supplicant.log
fi
# wait until associated with the AP - can take a while with WPA
secs=0
until iw wlan0 link | grep -q "SSID: $SSID"; do
let secs++
if [ $secs -ge 20 ]; then
logger -p user.err -t IFUP "Failed to associate with AP
within 20 seconds"
exit -1
fi
sleep 1
done
# set the regulatory domain (kernel >= 2.6.28)
iw reg set GB
ifconfig wlan0 192.168.0.140 netmask 255.255.255.0 up
route add default gw 192.168.0.1 netmask 0.0.0.0 metric 1
exit 0
fi
if [ "$1" = "eth0" ] ; then
# load the module if necessary
if ! grep -q eth0 /proc/net/dev; then
modprobe r8169
fi
# wait up to 5 seconds for eth0 to appear
secs=0
until grep -q eth0 /proc/net/dev; do
let secs++
if [ $secs -ge 5 ]; then
logger -p user.err -t IFUP "eth0 failed to appear within 5
seconds"
exit -1
fi
sleep 1
done
ifconfig eth0 192.168.0.40 netmask 255.255.255.0 up
route add default gw 192.168.0.1 netmask 0.0.0.0 metric 1
exit 0
fi
When the KVM client is running the routing on the host is:
$ route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
default router.local.la 0.0.0.0 UG 1 0 0 eth0
Unix * 255.0.0.0 U 0 0 0 lo
local.lan * 255.255.255.0 U 0 0 0 eth0
192.168.200.0 * 255.255.255.0 U 0 0 0 tap0
Like I say, the set up has been like this for ages and has worked. It's
only since I started using 3.6 kernels that I've had a problem. I don't
recall anything from the nat table ever having been listed by iptables -L.
From: Dave Jones <hidden> Date: 2012-10-01 19:34:41
On Mon, Oct 01, 2012 at 11:15:50AM +0200, Eric Dumazet wrote:
> >
> > $ netstat -s
> > Icmp:
> > 4 ICMP messages received
> > 4 input ICMP message failed.
> > ICMP input histogram:
> > echo replies: 4
>
> So icmp replies come back and are delivered to host instead of being
> forwarded.
>
> I wonder if MASQUERADE broke...
I hit something that sounds just like this a few months back..
http://lists.openwall.net/netdev/2012/07/25/53
It "went away" a few builds later, but I've seen it happen
again from time to time.
Dave
From: David Miller <davem@davemloft.net> Date: 2012-10-01 20:01:22
From: Dave Jones <redacted>
Date: Mon, 1 Oct 2012 15:34:34 -0400
On Mon, Oct 01, 2012 at 11:15:50AM +0200, Eric Dumazet wrote:
> >
> > $ netstat -s
> > Icmp:
> > 4 ICMP messages received
> > 4 input ICMP message failed.
> > ICMP input histogram:
> > echo replies: 4
>
> So icmp replies come back and are delivered to host instead of being
> forwarded.
>
> I wonder if MASQUERADE broke...
I hit something that sounds just like this a few months back..
http://lists.openwall.net/netdev/2012/07/25/53
It "went away" a few builds later, but I've seen it happen
again from time to time.
Yep I remembe that report.
If you can find a way to more reliably trigger the case, that would
help us immensely.
From: Eric Dumazet <hidden> Date: 2012-10-01 20:04:49
On Mon, 2012-10-01 at 16:01 -0400, David Miller wrote:
From: Dave Jones <redacted>
Date: Mon, 1 Oct 2012 15:34:34 -0400
quoted
On Mon, Oct 01, 2012 at 11:15:50AM +0200, Eric Dumazet wrote:
> >
> > $ netstat -s
> > Icmp:
> > 4 ICMP messages received
> > 4 input ICMP message failed.
> > ICMP input histogram:
> > echo replies: 4
>
> So icmp replies come back and are delivered to host instead of being
> forwarded.
>
> I wonder if MASQUERADE broke...
I hit something that sounds just like this a few months back..
http://lists.openwall.net/netdev/2012/07/25/53
It "went away" a few builds later, but I've seen it happen
again from time to time.
Yep I remembe that report.
If you can find a way to more reliably trigger the case, that would
help us immensely.
I am building a KMEMCHECK kernel, as a last try before my night ;)
From: Eric Dumazet <hidden> Date: 2012-10-02 15:35:39
On Mon, 2012-10-01 at 22:04 +0200, Eric Dumazet wrote:
On Mon, 2012-10-01 at 16:01 -0400, David Miller wrote:
quoted
If you can find a way to more reliably trigger the case, that would
help us immensely.
I am building a KMEMCHECK kernel, as a last try before my night ;)
This was a total disaster. KMEMCHECK dies horribly on my machine
David, shouldnt we use a nh_rth_forward instead of a nh_rth_input in
__mkroute_input() ?
(And change rt_cache_route() as well ?)
I am testing a patch right now.
From: Eric Dumazet <hidden> Date: 2012-10-02 15:48:46
From: Eric Dumazet <edumazet@google.com>
On Tue, 2012-10-02 at 17:35 +0200, Eric Dumazet wrote:
On Mon, 2012-10-01 at 22:04 +0200, Eric Dumazet wrote:
quoted
On Mon, 2012-10-01 at 16:01 -0400, David Miller wrote:
quoted
quoted
If you can find a way to more reliably trigger the case, that would
help us immensely.
I am building a KMEMCHECK kernel, as a last try before my night ;)
This was a total disaster. KMEMCHECK dies horribly on my machine
David, shouldnt we use a nh_rth_forward instead of a nh_rth_input in
__mkroute_input() ?
(And change rt_cache_route() as well ?)
I am testing a patch right now.
Yeah, this patch seems to fix the bug for me.
[PATCH] ipv4: properly cache forward routes
commit d2d68ba9fe8 (ipv4: Cache input routes in fib_info nexthops.)
introduced a regression for forwarding.
This was hard to reproduce but the symptom was that packets were
delivered to local host instead of being forwarded.
Add a separate cache (nh_rth_forward) to solve the problem.
Many thanks to Chris Clayton for his patience and help.
Reported-by: Chris Clayton <redacted>
Bisected-by: Chris Clayton [off-list ref]
Reported-by: Dave Jones <redacted>
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
include/net/ip_fib.h | 1 +
net/ipv4/fib_semantics.c | 1 +
net/ipv4/route.c | 16 ++++++++--------
3 files changed, 10 insertions(+), 8 deletions(-)
From: Dave Jones <hidden> Date: 2012-10-02 15:57:45
On Tue, Oct 02, 2012 at 05:48:39PM +0200, Eric Dumazet wrote:
> From: Eric Dumazet [off-list ref]
>
> On Tue, 2012-10-02 at 17:35 +0200, Eric Dumazet wrote:
> > On Mon, 2012-10-01 at 22:04 +0200, Eric Dumazet wrote:
> > > On Mon, 2012-10-01 at 16:01 -0400, David Miller wrote:
> >
> > > > If you can find a way to more reliably trigger the case, that would
> > > > help us immensely.
> > >
> > > I am building a KMEMCHECK kernel, as a last try before my night ;)
> >
> > This was a total disaster. KMEMCHECK dies horribly on my machine
> >
> > David, shouldnt we use a nh_rth_forward instead of a nh_rth_input in
> > __mkroute_input() ?
> >
> > (And change rt_cache_route() as well ?)
> >
> > I am testing a patch right now.
>
> Yeah, this patch seems to fix the bug for me.
Good work! Any idea why it didn't happen on every build for me ?
From your description, this should have failed every time ?
From: Eric Dumazet <hidden> Date: 2012-10-02 16:06:37
On Tue, 2012-10-02 at 11:57 -0400, Dave Jones wrote:
Good work! Any idea why it didn't happen on every build for me ?
From your description, this should have failed every time ?
Well, it seems that as long as you had forwarded packets and a route not
yet cached in nh_rth_input, we were using a brand new route (and correct
one)
But as soon as a locally generated traffic did cache a route in
nh_rth_input, forwarded packets immediately were using this cache and
were delivered (and dropped) to local host.
Maybe my patch is not the good fix, but at least its a step in
understanding the problem.
Thanks
On Mon, 2012-10-01 at 16:01 -0400, David Miller wrote:
quoted
From: Dave Jones <davej <at> redhat.com>
Date: Mon, 1 Oct 2012 15:34:34 -0400
quoted
On Mon, Oct 01, 2012 at 11:15:50AM +0200, Eric Dumazet wrote:
> >
> > $ netstat -s
> > Icmp:
> > 4 ICMP messages received
> > 4 input ICMP message failed.
> > ICMP input histogram:
> > echo replies: 4
>
> So icmp replies come back and are delivered to host instead of being
> forwarded.
>
> I wonder if MASQUERADE broke...
I hit something that sounds just like this a few months back..
http://lists.openwall.net/netdev/2012/07/25/53
It "went away" a few builds later, but I've seen it happen
again from time to time.
Yep I remembe that report.
If you can find a way to more reliably trigger the case, that would
help us immensely.
I am building a KMEMCHECK kernel, as a last try before my night ;)
i,
I'm facing this kind of problem, too, but it is a little different; from the
kvm guest I can ping the local host and any host outside my local (physical)
network, but cannot ping other hosts in the local (physical) net. This happens
whith guests in a virtual switch (vde) or in any bridged tun/tap. I switched
back to 3.5.4, for now.
Thanks
Edivaldo de Araújo Pereira
From: David Miller <davem@davemloft.net> Date: 2012-10-02 18:25:08
From: Eric Dumazet <redacted>
Date: Tue, 02 Oct 2012 17:48:39 +0200
[PATCH] ipv4: properly cache forward routes
commit d2d68ba9fe8 (ipv4: Cache input routes in fib_info nexthops.)
introduced a regression for forwarding.
This was hard to reproduce but the symptom was that packets were
delivered to local host instead of being forwarded.
Add a separate cache (nh_rth_forward) to solve the problem.
Many thanks to Chris Clayton for his patience and help.
Reported-by: Chris Clayton <redacted>
Bisected-by: Chris Clayton [off-list ref]
Reported-by: Dave Jones <redacted>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Thanks for figuring this out, I'll think about this more
deeply.
From: Alexander Duyck <hidden> Date: 2012-10-02 21:13:53
On 10/02/2012 11:25 AM, David Miller wrote:
From: Eric Dumazet <redacted>
Date: Tue, 02 Oct 2012 17:48:39 +0200
quoted
[PATCH] ipv4: properly cache forward routes
commit d2d68ba9fe8 (ipv4: Cache input routes in fib_info nexthops.)
introduced a regression for forwarding.
This was hard to reproduce but the symptom was that packets were
delivered to local host instead of being forwarded.
Add a separate cache (nh_rth_forward) to solve the problem.
Many thanks to Chris Clayton for his patience and help.
Reported-by: Chris Clayton <redacted>
Bisected-by: Chris Clayton [off-list ref]
Reported-by: Dave Jones <redacted>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Thanks for figuring this out, I'll think about this more
deeply.
I think something may have been missed in this patch.
With it applied to net-next I am unable to remove the ixgbe driver after
running a routing traffic test. The specific message I am getting is:
unregister_netdevice: waiting for eth2 to become free. Usage count = -7
Thanks,
Alex
From: Eric Dumazet <hidden> Date: 2012-10-02 21:35:36
On Tue, 2012-10-02 at 14:14 -0700, Alexander Duyck wrote:
I think something may have been missed in this patch.
With it applied to net-next I am unable to remove the ixgbe driver after
running a routing traffic test. The specific message I am getting is:
unregister_netdevice: waiting for eth2 to become free. Usage count = -7
Yes, I realized later that rt_set_nexthop(), called from
__mkroute_input() was responsible to do the caching...
So another version is needed, I'll do that tomorrow unless David can fix
the problem while I sleep a bit ;)
David, shouldnt we use a nh_rth_forward instead of a nh_rth_input in
__mkroute_input() ?
(And change rt_cache_route() as well ?)
I am testing a patch right now.
Yeah, this patch seems to fix the bug for me.
[PATCH] ipv4: properly cache forward routes
commit d2d68ba9fe8 (ipv4: Cache input routes in fib_info nexthops.)
introduced a regression for forwarding.
This was hard to reproduce but the symptom was that packets were
delivered to local host instead of being forwarded.
Add a separate cache (nh_rth_forward) to solve the problem.
Can it be a problem related to fib_info reuse
from different routes. For example, when local IP address
is created for subnet we have:
broadcast 192.168.0.255 dev DEV proto kernel scope link src 192.168.0.1
192.168.0.0/24 dev DEV proto kernel scope link src 192.168.0.1
local 192.168.0.1 dev DEV proto kernel scope host src 192.168.0.1
The "dev DEV proto kernel scope link src 192.168.0.1" is
a reused fib_info structure where we put cached routes.
The result can be same fib_info for 192.168.0.255 and
192.168.0.0/24. RTN_BROADCAST is cached only for input
routes. Incoming broadcast to 192.168.0.255 can be cached
and can cause problems for traffic forwarded to 192.168.0.0/24.
So, this patch should solve the problem because it
separates the broadcast from unicast traffic.
And the ip_route_input_slow caching will work for
local and broadcast input routes (above routes 1 and 3) just
because they differ in scope and use different fib_info.
Another possible failure is for output routes:
multicast 224.0.0.0/4 fib_info
with unicast
192.168.0.0/24 fib_info
The multicast sets RTCF_MULTICAST | RTCF_LOCAL
and can cause problems for generated unicast traffic on
fib_info reuse. Depends on the scope, for multicast it is
usually scope global, so may be it is difficult to happen
in practice.
__mkroute_output works for local/unicast routes
because they differ in scope.
Many thanks to Chris Clayton for his patience and help.
Reported-by: Chris Clayton <redacted>
Bisected-by: Chris Clayton [off-list ref]
Reported-by: Dave Jones <redacted>
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
include/net/ip_fib.h | 1 +
net/ipv4/fib_semantics.c | 1 +
net/ipv4/route.c | 16 ++++++++--------
3 files changed, 10 insertions(+), 8 deletions(-)
From: David Miller <davem@davemloft.net> Date: 2012-10-03 02:55:37
From: Eric Dumazet <redacted>
Date: Tue, 02 Oct 2012 17:48:39 +0200
[PATCH] ipv4: properly cache forward routes
commit d2d68ba9fe8 (ipv4: Cache input routes in fib_info nexthops.)
introduced a regression for forwarding.
This was hard to reproduce but the symptom was that packets were
delivered to local host instead of being forwarded.
Add a separate cache (nh_rth_forward) to solve the problem.
Many thanks to Chris Clayton for his patience and help.
Reported-by: Chris Clayton <redacted>
Bisected-by: Chris Clayton [off-list ref]
Reported-by: Dave Jones <redacted>
Signed-off-by: Eric Dumazet <edumazet@google.com>
I'm still having trouble understanding how this can happen,
which is probably why I introduced this bug in the first
place :-)
Only INPUT routes created by ip_route_input_slow() cache using
nh_rth_input.
Routes for locally destinations vs. forwarded destinations will
resolve to different fib_info objects.
If at some point a new route is added which turns a local destination
into one for which we forward, normal invalidation of cached routes
ought to fix it.
There's some sequence of events I don't understand that causes the
corrupt route cache, can you show it to me?
Thanks.
Can it be a problem related to fib_info reuse
from different routes. For example, when local IP address
is created for subnet we have:
broadcast 192.168.0.255 dev DEV proto kernel scope link src 192.168.0.1
192.168.0.0/24 dev DEV proto kernel scope link src 192.168.0.1
local 192.168.0.1 dev DEV proto kernel scope host src 192.168.0.1
The "dev DEV proto kernel scope link src 192.168.0.1" is
a reused fib_info structure where we put cached routes.
The result can be same fib_info for 192.168.0.255 and
192.168.0.0/24. RTN_BROADCAST is cached only for input
routes. Incoming broadcast to 192.168.0.255 can be cached
and can cause problems for traffic forwarded to 192.168.0.0/24.
So, this patch should solve the problem because it
separates the broadcast from unicast traffic.
Now I understand the problem.
I think the way to fix this is to add cfg->fc_type as another
thing that fib_info objects are key'd by.
I think it also would fix your obscure output multicast case too.
From: Eric Dumazet <hidden> Date: 2012-10-03 07:28:53
On Wed, 2012-10-03 at 02:24 +0300, Julian Anastasov wrote:
Hello,
On Tue, 2 Oct 2012, Eric Dumazet wrote:
quoted
quoted
David, shouldnt we use a nh_rth_forward instead of a nh_rth_input in
__mkroute_input() ?
(And change rt_cache_route() as well ?)
I am testing a patch right now.
Yeah, this patch seems to fix the bug for me.
[PATCH] ipv4: properly cache forward routes
commit d2d68ba9fe8 (ipv4: Cache input routes in fib_info nexthops.)
introduced a regression for forwarding.
This was hard to reproduce but the symptom was that packets were
delivered to local host instead of being forwarded.
Add a separate cache (nh_rth_forward) to solve the problem.
Can it be a problem related to fib_info reuse
from different routes. For example, when local IP address
is created for subnet we have:
broadcast 192.168.0.255 dev DEV proto kernel scope link src 192.168.0.1
192.168.0.0/24 dev DEV proto kernel scope link src 192.168.0.1
local 192.168.0.1 dev DEV proto kernel scope host src 192.168.0.1
The "dev DEV proto kernel scope link src 192.168.0.1" is
a reused fib_info structure where we put cached routes.
The result can be same fib_info for 192.168.0.255 and
192.168.0.0/24. RTN_BROADCAST is cached only for input
routes. Incoming broadcast to 192.168.0.255 can be cached
and can cause problems for traffic forwarded to 192.168.0.0/24.
So, this patch should solve the problem because it
separates the broadcast from unicast traffic.
And the ip_route_input_slow caching will work for
local and broadcast input routes (above routes 1 and 3) just
because they differ in scope and use different fib_info.
Another possible failure is for output routes:
multicast 224.0.0.0/4 fib_info
with unicast
192.168.0.0/24 fib_info
The multicast sets RTCF_MULTICAST | RTCF_LOCAL
and can cause problems for generated unicast traffic on
fib_info reuse. Depends on the scope, for multicast it is
usually scope global, so may be it is difficult to happen
in practice.
__mkroute_output works for local/unicast routes
because they differ in scope.
Thanks Julian for these informations.
BTW, it seems we dont properly increase UDP MIB counters when a
multicast message is not delivered to at least one socket.
Lets fix this to ease future bug hunting.
I hate when "netstat -s" is useless and we have to use dropwatch to
figure out where we drop a frame.
[PATCH] udp: increment UDP_MIB_NOPORTS in multicast receive
We should increment UDP_MIB_NOPORTS in the case we found
no socket to deliver a copy of one incoming UDP message.
(RFC 4113 udpNoPorts)
Signed-off-by: Eric Dumazet <edumazet@google.com>
---
net/ipv4/udp.c | 1 +
net/ipv6/udp.c | 1 +
2 files changed, 2 insertions(+)
From: David Stevens <hidden> Date: 2012-10-03 12:46:03
netdev-owner@vger.kernel.org wrote on 10/03/2012 03:28:48 AM:
BTW, it seems we dont properly increase UDP MIB counters when a
multicast message is not delivered to at least one socket.
If an interface is in promiscuous mode or there are false
positives in a multicast address filter, wouldn't this count as
"drops" packets that were never intended for this machine?
I think an otherwise valid multicast or broadcast packet that doesn't
have a local receiver is not an error and shouldn't be counted.
+-DLS
From: Eric Dumazet <hidden> Date: 2012-10-03 13:15:57
On Wed, 2012-10-03 at 08:45 -0400, David Stevens wrote:
netdev-owner@vger.kernel.org wrote on 10/03/2012 03:28:48 AM:
quoted
BTW, it seems we dont properly increase UDP MIB counters when a
multicast message is not delivered to at least one socket.
If an interface is in promiscuous mode or there are false
positives in a multicast address filter, wouldn't this count as
"drops" packets that were never intended for this machine?
Yes, probably. So we drop them and its expected.
I think an otherwise valid multicast or broadcast packet that doesn't
have a local receiver is not an error and shouldn't be counted.
Hmmm
This counter is not an "error counter", just a "counter".
RFC definitions are exactly :
udpNoPorts OBJECT-TYPE
SYNTAX Counter32
MAX-ACCESS read-only
STATUS current
DESCRIPTION
"The total number of received UDP datagrams for which
there was no application at the destination port.
udpInErrors OBJECT-TYPE
SYNTAX Counter32
MAX-ACCESS read-only
STATUS current
DESCRIPTION
"The number of received UDP datagrams that could not be
delivered for reasons other than the lack of an
application at the destination port.
So when a host receives an UDP datagram but there was no application
at the destination port we should increment udpNoPorts, and its not
an error but just a fact.
Now _if_ some reader interprets udpNoPorts increases as an indication
of errors, this reader is wrong.
From: David Stevens <hidden> Date: 2012-10-03 14:25:37
Eric Dumazet [off-list ref] wrote on 10/03/2012 09:15:51 AM:
So when a host receives an UDP datagram but there was no application
at the destination port we should increment udpNoPorts, and its not
an error but just a fact.
Of course. I think our difference is on the definition of
"receives".
I don't think a packet delivered locally due to promiscuous mode,
broadcast
or an imperfect multicast address filter match is a host UDP datagram
receive.
These packets really shouldn't be delivered to UDP at all; they are not
addressed to this host (at least the non-broadcast, no-membership ones).
A unicast UDP packet that doesn't match a local IP address does
not
increment this counter. A promiscuous mode multicast delivery is no
different,
except that the destination alone doesn't tell us if it is for us.
I think counting these will primarily lead to administrators
seeing
non-zero drops and wasting their time trying to track them down.
+-DLS
Can it be a problem related to fib_info reuse
from different routes. For example, when local IP address
is created for subnet we have:
broadcast 192.168.0.255 dev DEV proto kernel scope link src 192.168.0.1
192.168.0.0/24 dev DEV proto kernel scope link src 192.168.0.1
local 192.168.0.1 dev DEV proto kernel scope host src 192.168.0.1
The "dev DEV proto kernel scope link src 192.168.0.1" is
a reused fib_info structure where we put cached routes.
The result can be same fib_info for 192.168.0.255 and
192.168.0.0/24. RTN_BROADCAST is cached only for input
routes. Incoming broadcast to 192.168.0.255 can be cached
and can cause problems for traffic forwarded to 192.168.0.0/24.
So, this patch should solve the problem because it
separates the broadcast from unicast traffic.
Now I understand the problem.
I think the way to fix this is to add cfg->fc_type as another
thing that fib_info objects are key'd by.
I think it also would fix your obscure output multicast case too.
I've seen the discussion about whether Eric's patch is OK or not, but
thought I'd give it a spin anyway. It applies to 3.6.0 with some fuzz,
but I can confirm that with the patch applied I can now ping my router
and browse the internet from a KVM client, so the Eric's diagnosis
matches the problem I reported.
However, after closing the client, I got an oops. I've taken a
photograph of the screen and uploaded it to
http://i714.photobucket.com/albums/ww149/chris2553/IMAG0059.jpg. As it's
not the final patch, this may be a red herring, but I thought I'd better
give a heads up anyway.
Chris
From: Eric Dumazet <hidden> Date: 2012-10-03 15:29:19
On Wed, 2012-10-03 at 10:09 -0400, David Stevens wrote:
Eric Dumazet [off-list ref] wrote on 10/03/2012 09:15:51 AM:
quoted
So when a host receives an UDP datagram but there was no application
at the destination port we should increment udpNoPorts, and its not
an error but just a fact.
Of course. I think our difference is on the definition of
"receives".
A receive is a packet delivered to this host.
Interface being promiscuous or not doesnt really matter.
I don't think a packet delivered locally due to promiscuous mode,
broadcast
or an imperfect multicast address filter match is a host UDP datagram
receive.
These packets really shouldn't be delivered to UDP at all; they are not
addressed to this host (at least the non-broadcast, no-membership ones).
Thats the bug we currently are tracking. If some error is happening and
packet is delivered instead of being forwarded or dropped, we need a
counter being incremented to catch the bug.
A unicast UDP packet that doesn't match a local IP address does
not
increment this counter.
It _does_ increment this counter right now, not sure what you mean.
We currently correctly increment udpNoPorts if we receive an unicast UDP
packet that doesnt find a matching socket (because socket(s) are bound
to specific addresses instead of ANY_ADDR)
This is an extension of the "there was no application at the destination
port" to "there was no application at the destination port and
destination address"
A promiscuous mode multicast delivery is no
different,
except that the destination alone doesn't tell us if it is for us.
I think counting these will primarily lead to administrators
seeing
non-zero drops and wasting their time trying to track them down.
Well, as I said, seeing increments of this counter is perfectly fine and
matches RFC. It permits better diagnostics. Hiding bugs is not very
helpful.
Most of the time I am trying to track a bug in linux network stack, the
very first thing I ask to reporters is to post "netstat -s" before/after
their tests exactly because I want to see _some_ counters be incremented
and catch obvious problems.
And alas, many drops in our stack are not correctly reported because we
forgot to increment a counter at the right place.
I am fine adding a new SNMP McastDrops counter if you feel its better.
# grep Udp: /proc/net/snmp
Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors McastDrops
Udp: 11449164 15473 514616 290821178 0 184352 134
"netstat -s -u" would display :
Udp:
11449164 packets received
15473 packets to unknown port received.
514616 packet receive errors
290821178 packets sent
SndbufErrors: 184352
McastDrops: 134
Non official patch since net-next is not open :
include/linux/snmp.h | 1 +
net/ipv4/proc.c | 1 +
net/ipv4/udp.c | 2 ++
net/ipv6/proc.c | 2 ++
net/ipv6/udp.c | 2 ++
5 files changed, 8 insertions(+)
From: David Stevens <hidden> Date: 2012-10-03 17:31:40
Eric Dumazet [off-list ref] wrote on 10/03/2012 11:29:13 AM:
quoted
Of course. I think our difference is on the definition of
"receives".
A receive is a packet delivered to this host.
Interface being promiscuous or not doesnt really matter.
A receive is a packet *addressed* to this host. My point was
that running tcpdump/wireshark to look at other hosts' traffic
shouldn't affect any UDP MIB (these are ordinarily filtered by IP),
but I forgot that we are checking in software, as well as the HW multicast
address filter, for multicast group membership. So promiscuous mode and
imperfect NIC MAF hashes shouldn't actually result in local delivery
and that problem isn't there at all.
I do think, still, that it is common to have broadcasts and
multicasts (for joined groups, even) with traffic completely uninteresting
to this host and that having a drop counter going up for those will
appear to be losses and errors when they are completely harmless and
irrelevant.
But since it can't be incremented for items that are not actually
addressed to the local host, as I originally thought, I don't object
anymore. Sorry for the sidetrack -- I should've verified that originally.
+-DLS
From: Rick Jones <hidden> Date: 2012-10-03 17:39:57
On 10/03/2012 07:09 AM, David Stevens wrote:
Of course. I think our difference is on the definition of
"receives". I don't think a packet delivered locally due to
promiscuous mode, broadcast or an imperfect multicast address filter
match is a host UDP datagram receive. These packets really shouldn't
be delivered to UDP at all; they are not addressed to this host (at
least the non-broadcast, no-membership ones). A unicast UDP packet
that doesn't match a local IP address does not increment this
counter. A promiscuous mode multicast delivery is no different,
except that the destination alone doesn't tell us if it is for us.
I think counting these will primarily lead to administrators seeing
non-zero drops and wasting their time trying to track them down.
I would tend to agree with David on this one. Or they might cease trying
to track them down because they've gotten so many "false positives."
Isn't "meant for me" vs "not meant for me" at the heard of "drops"
versus "discards?"
Once the packet is in the host, is it tagged in some way with "this was
received as promiscuous/whatnot?"
rick
From: David Miller <davem@davemloft.net> Date: 2012-10-03 19:30:31
From: David Stevens <redacted>
Date: Wed, 3 Oct 2012 13:31:30 -0400
Eric Dumazet [off-list ref] wrote on 10/03/2012 11:29:13 AM:
quoted
quoted
Of course. I think our difference is on the definition of
"receives".
A receive is a packet delivered to this host.
Interface being promiscuous or not doesnt really matter.
A receive is a packet *addressed* to this host.
Although I'm largely ambivalent, this one sentence tipped me over
towards David's side on this issue.
But this is easy to resolve Eric, just simply make a new custom
counter that counts these new cases you care about and document it
properly.
Thanks.
Can it be a problem related to fib_info reuse
from different routes. For example, when local IP address
is created for subnet we have:
broadcast 192.168.0.255 dev DEV proto kernel scope link src 192.168.0.1
192.168.0.0/24 dev DEV proto kernel scope link src 192.168.0.1
local 192.168.0.1 dev DEV proto kernel scope host src 192.168.0.1
The "dev DEV proto kernel scope link src 192.168.0.1" is
a reused fib_info structure where we put cached routes.
The result can be same fib_info for 192.168.0.255 and
192.168.0.0/24. RTN_BROADCAST is cached only for input
routes. Incoming broadcast to 192.168.0.255 can be cached
and can cause problems for traffic forwarded to 192.168.0.0/24.
So, this patch should solve the problem because it
separates the broadcast from unicast traffic.
Now I understand the problem.
I think the way to fix this is to add cfg->fc_type as another
thing that fib_info objects are key'd by.
I think it also would fix your obscure output multicast case too.
Agreed. I don't see problem with this idea.
It will avoid confusions with rt_type.
Regards
--
Julian Anastasov [off-list ref]
From: Eric Dumazet <hidden> Date: 2012-10-04 11:25:32
On Tue, 2012-10-02 at 17:48 +0200, Eric Dumazet wrote:
From: Eric Dumazet <edumazet@google.com>
On Tue, 2012-10-02 at 17:35 +0200, Eric Dumazet wrote:
quoted
On Mon, 2012-10-01 at 22:04 +0200, Eric Dumazet wrote:
quoted
On Mon, 2012-10-01 at 16:01 -0400, David Miller wrote:
quoted
quoted
If you can find a way to more reliably trigger the case, that would
help us immensely.
I am building a KMEMCHECK kernel, as a last try before my night ;)
This was a total disaster. KMEMCHECK dies horribly on my machine
David, shouldnt we use a nh_rth_forward instead of a nh_rth_input in
__mkroute_input() ?
(And change rt_cache_route() as well ?)
I am testing a patch right now.
OK so I implemented David idea and it seems to work.
Testers are needed, thanks ! ;)
[PATCH] ipv4: add a fib_type to fib_info
commit d2d68ba9fe8 (ipv4: Cache input routes in fib_info nexthops.)
introduced a regression for forwarding.
This was hard to reproduce but the symptom was that packets were
delivered to local host instead of being forwarded.
David suggested to add fib_type to fib_info so that we dont
inadvertently share same fib_info for different purposes.
With help from Julian Anastasov who provided very helpful
hints, reproduced here :
<quote>
Can it be a problem related to fib_info reuse
from different routes. For example, when local IP address
is created for subnet we have:
broadcast 192.168.0.255 dev DEV proto kernel scope link src
192.168.0.1
192.168.0.0/24 dev DEV proto kernel scope link src 192.168.0.1
local 192.168.0.1 dev DEV proto kernel scope host src 192.168.0.1
The "dev DEV proto kernel scope link src 192.168.0.1" is
a reused fib_info structure where we put cached routes.
The result can be same fib_info for 192.168.0.255 and
192.168.0.0/24. RTN_BROADCAST is cached only for input
routes. Incoming broadcast to 192.168.0.255 can be cached
and can cause problems for traffic forwarded to 192.168.0.0/24.
So, this patch should solve the problem because it
separates the broadcast from unicast traffic.
And the ip_route_input_slow caching will work for
local and broadcast input routes (above routes 1 and 3) just
because they differ in scope and use different fib_info.
</quote>
Many thanks to Chris Clayton for his patience and help.
Reported-by: Chris Clayton <redacted>
Bisected-by: Chris Clayton [off-list ref]
Reported-by: Dave Jones <redacted>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Julian Anastasov <ja@ssi.bg>
---
include/net/ip_fib.h | 1 +
net/ipv4/fib_semantics.c | 2 ++
2 files changed, 3 insertions(+)
From: Chris Clayton <hidden> Date: 2012-10-04 13:08:30
On 10/04/12 12:25, Eric Dumazet wrote:
On Tue, 2012-10-02 at 17:48 +0200, Eric Dumazet wrote:
quoted
From: Eric Dumazet <edumazet@google.com>
On Tue, 2012-10-02 at 17:35 +0200, Eric Dumazet wrote:
quoted
On Mon, 2012-10-01 at 22:04 +0200, Eric Dumazet wrote:
quoted
On Mon, 2012-10-01 at 16:01 -0400, David Miller wrote:
quoted
quoted
If you can find a way to more reliably trigger the case, that would
help us immensely.
I am building a KMEMCHECK kernel, as a last try before my night ;)
This was a total disaster. KMEMCHECK dies horribly on my machine
David, shouldnt we use a nh_rth_forward instead of a nh_rth_input in
__mkroute_input() ?
(And change rt_cache_route() as well ?)
I am testing a patch right now.
OK so I implemented David idea and it seems to work.
Testers are needed, thanks ! ;)
I've tested 3.6.0 with this patch applied and networking in a WinXP KVM
client is now working fine. The patch applies cleanly to 3.6.0, so I
assume the patch will be forwarded to stable in due course.
Tested-by: Chris Clayton <redacted>
quoted hunk
[PATCH] ipv4: add a fib_type to fib_info
commit d2d68ba9fe8 (ipv4: Cache input routes in fib_info nexthops.)
introduced a regression for forwarding.
This was hard to reproduce but the symptom was that packets were
delivered to local host instead of being forwarded.
David suggested to add fib_type to fib_info so that we dont
inadvertently share same fib_info for different purposes.
With help from Julian Anastasov who provided very helpful
hints, reproduced here :
<quote>
Can it be a problem related to fib_info reuse
from different routes. For example, when local IP address
is created for subnet we have:
broadcast 192.168.0.255 dev DEV proto kernel scope link src
192.168.0.1
192.168.0.0/24 dev DEV proto kernel scope link src 192.168.0.1
local 192.168.0.1 dev DEV proto kernel scope host src 192.168.0.1
The "dev DEV proto kernel scope link src 192.168.0.1" is
a reused fib_info structure where we put cached routes.
The result can be same fib_info for 192.168.0.255 and
192.168.0.0/24. RTN_BROADCAST is cached only for input
routes. Incoming broadcast to 192.168.0.255 can be cached
and can cause problems for traffic forwarded to 192.168.0.0/24.
So, this patch should solve the problem because it
separates the broadcast from unicast traffic.
And the ip_route_input_slow caching will work for
local and broadcast input routes (above routes 1 and 3) just
because they differ in scope and use different fib_info.
</quote>
Many thanks to Chris Clayton for his patience and help.
Reported-by: Chris Clayton <redacted>
Bisected-by: Chris Clayton [off-list ref]
Reported-by: Dave Jones <redacted>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Julian Anastasov <ja@ssi.bg>
---
include/net/ip_fib.h | 1 +
net/ipv4/fib_semantics.c | 2 ++
2 files changed, 3 insertions(+)
From: Eric Dumazet <hidden> Date: 2012-10-04 13:32:13
On Thu, 2012-10-04 at 14:08 +0100, Chris Clayton wrote:
I've tested 3.6.0 with this patch applied and networking in a WinXP KVM
client is now working fine. The patch applies cleanly to 3.6.0, so I
assume the patch will be forwarded to stable in due course.
Tested-by: Chris Clayton <redacted>
From: David Miller <davem@davemloft.net> Date: 2012-10-04 18:14:16
From: Eric Dumazet <redacted>
Date: Thu, 04 Oct 2012 15:32:08 +0200
On Thu, 2012-10-04 at 14:08 +0100, Chris Clayton wrote:
quoted
I've tested 3.6.0 with this patch applied and networking in a WinXP KVM
client is now working fine. The patch applies cleanly to 3.6.0, so I
assume the patch will be forwarded to stable in due course.
Tested-by: Chris Clayton <redacted>
Thanks for testing.
Applied and queued up for -stable, thanks everyone.
Note that this change means we can completely remove the type fields
from fib_alias and fib_result when net-next opens up, as the value can
be fetched from the fib_info directly now.