On Tue, Feb 12, 2013 at 08:06:04AM +0100, Peter Lieven wrote:

On 23.01.2013 11:03, Michael S. Tsirkin wrote:

quoted

On Tue, Jan 22, 2013 at 10:04:07AM +0100, Peter Lieven wrote:

quoted

On 23.11.2012 12:01, Michael S. Tsirkin wrote:

quoted

On Fri, Nov 23, 2012 at 10:41:21AM +0100, Peter Lieven wrote:

quoted

Am 23.11.2012 um 08:02 schrieb Stefan Hajnoczi:

quoted

On Thu, Nov 22, 2012 at 03:29:52PM +0100, Peter Lieven wrote:

quoted

is anyone aware of a problem with the linux network bridge that in very rare circumstances stops
a bridge from sending pakets to a tap device?

My problem occurs in conjunction with vanilla qemu-kvm-1.2.0 and Ubuntu Kernel 3.2.0-34.53
which is based on Linux 3.2.33.

I was not yet able to reproduce the issue, it happens in really rare cases. The symptom is that
the tap does not have any TX packets. RX is working fine. I see the packets coming in at
the physical interface on the host, but they are not forwarded to the tap interface.
The bridge itself has learnt the mac address of the vServer that is connected to the tap interface.
It does not help to toggle the bridge link status,  the tap interface status or the interface in the vServer.
It seems that problem occurs if a tap interface that has previously been used, but set to nonpersistent
is set persistent again and then is by chance assigned to the same vServer (=same mac address on same
bridge) again. Unfortunately it seems not to be reproducible.

Not sure but this patch from Michael Tsirkin may help - it solves an
issue with persistent tap devices:

http://patchwork.ozlabs.org/patch/198598/

Hi Stefan,

thanks for the pointer. I have seen this patch, but I have neglected it because it was dealing
with persistent taps. But maybe the taps in the kernel are not deleted directly.
Can you remember what the syptomps of the above issue have been? Sorry for
being vague, but I currently have no clue whats going on.

Can someone who has more internal knowledge of the bridging/tap code say if qemu can
be responsible at all if the tap device is not receiving packets from the bridge.

If I have the following config. Lets say packets coming in via physical interface eth1.123,
and a bridge called br123.I further have a virtual machine with tap0. Both eth1.123
and tap0 are member of br123.

If the issue occurs the vServer has no network connectivity inbound. If I sent a ping
from the vServer I see it on tap0 and leaving on eth1.123. I see further the arp reply coming
in via eth1.123, but the reply can't be seen on tap0.

Peter

If guest is not consuming packets, a TX queue in tap device
will with time overrun (there's space for 1000 packets there).
This is code from tun:

        if (skb_queue_len(&tfile->socket.sk->sk_receive_queue)
                          >= dev->tx_queue_len / tun->numqueues){
                if (!(tun->flags & TUN_ONE_QUEUE)) {
                        /* Normal queueing mode. */
                        /* Packet scheduler handles dropping of further
 * packets. */
                        netif_stop_subqueue(dev, txq);

                        /* We won't see all dropped packets
 * individually, so overrun
                         * error is more appropriate. */
                        dev->stats.tx_fifo_errors++;


So you can detect that this triggered by looking at fifo errors counter in device.

Once this happens TX queue is stopped, then you hit this path:

                        if (!netif_xmit_stopped(txq)) {
                                __this_cpu_inc(xmit_recursion);
                                rc = dev_hard_start_xmit(skb, dev, txq);
                                __this_cpu_dec(xmit_recursion);
                                if (dev_xmit_complete(rc)) {
                                        HARD_TX_UNLOCK(dev, txq);
                                        goto out;
                                }
                        }

so packets are not passed to device anymore.
It will stay this way until guest consumes some packets and
queue is restarted.

After some time I again have a vServer in this state. It seems not like there
are no TX errors.

# ifconfig tap10
tap10     Link encap:Ethernet  HWaddr 7a:59:20:6f:e7:e5
          inet6 addr: fe80::7859:20ff:fe6f:e7e5/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:197431 errors:0 dropped:0 overruns:0 frame:0
          TX packets:264309 errors:0 dropped:0 overruns:2 carrier:0
          collisions:0 txqueuelen:500
          RX bytes:13842063 (13.8 MB)  TX bytes:35092821 (35.0 MB)

It seems like the bridge is not forwarding any packets to the tap device anymore altough it has learnt
the MAC-Adresses and there are also broadcast packets coming in.

Any more ideas where I could debug?

Peter

quoted

quoted

quoted

Stefan

Hmm. So there are two overrun errors that triggered, so
it's possible after the second one the queue got stuck in an xoff state.
You'd have to use something like systemtap or kdb to poke at the
queue state to see whether xoff flag is set and/or look
at the receive queue length.

For future, we can try to set TUN_ONE_QUEUE flag on the interface,
or try applying this patch
5d097109257c03a71845729f8db6b5770c4bbedc
in kernel see if this helps.

If have set this option for 2 weeks now and not seen this problem again.
How does this flag work with the recently added tap multiqueue support?

Peter

This will be the only option in 3.8.

-- 
MST