As subject says, 802.3ad bonding is not working with virtio network model.
The only errors I see is:
No 802.3ad response from the link partner for any adapters in the bond.
Dumping the network traffic shows that no LACP packets are sent from the
host running with virtio driver, changing to for example e1000 solves
this problem
with no configuration changes.
Is this a known problem?
--
Bj(/)rnar
Mon, Jan 25, 2016 at 05:24:48PM CET, bjornar.ness@gmail.com wrote:
As subject says, 802.3ad bonding is not working with virtio network model.
The only errors I see is:
No 802.3ad response from the link partner for any adapters in the bond.
Dumping the network traffic shows that no LACP packets are sent from the
host running with virtio driver, changing to for example e1000 solves
this problem
with no configuration changes.
Is this a known problem?
I believe the problem is virtio_net for obvious reasons does not report
speed and duplex. Bonding 3ad mode makes that unconfortable :)
Use team ;)
From: Nikolay Aleksandrov <hidden> Date: 2016-01-28 13:53:43
On 01/28/2016 02:33 PM, Jiri Pirko wrote:
Mon, Jan 25, 2016 at 05:24:48PM CET, bjornar.ness@gmail.com wrote:
quoted
As subject says, 802.3ad bonding is not working with virtio network model.
The only errors I see is:
No 802.3ad response from the link partner for any adapters in the bond.
Dumping the network traffic shows that no LACP packets are sent from the
host running with virtio driver, changing to for example e1000 solves
this problem
with no configuration changes.
Is this a known problem?
Can you show your bond's /proc/net/bonding/bond<X> ? And also in order to
better see what's going on I'd suggest enabling the pr_debug() calls in the 3ad
code:
echo 'file bond_3ad.c +p' > /sys/kernel/debug/dynamic_debug/control
(assuming you have debugfs mounted at /sys/kernel/debug)
Then you can follow the logs to see what's going on.
I can clearly see LACP packets sent over virtio net devices:
14:53:05.323490 52:54:00:51:25:3c > 01:80:c2:00:00:02, ethertype Slow Protocols (0x8809), length 124: LACPv1, length 110
I believe the problem is virtio_net for obvious reasons does not report
speed and duplex. Bonding 3ad mode makes that unconfortable :)
root@dev:~# ethtool -i eth1
driver: virtio_net
root@dev:~# ethtool eth1
Settings for eth1:
Supported ports: [ ]
Supported link modes: Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
*Speed: 10Mb/s*
*Duplex: Full*
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
MDI-X: Unknown
Link detected: yes
The bonding catches that correctly,
[54569.138572] bond0: Adding slave eth1
[54569.139686] bond0: Port 1 Received status full duplex update from adapter
[54569.139690] bond0: Port 1 Received link speed 2 update from adapter
The debug messages are from enabled 3ad mode pr_debugs().
Added 2 virtio_net adapters and they successfully went in a single LAG
when it was enabled on the other side.
root@dev:~# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535
System MAC address: 52:54:00:51:25:3c
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 5
Partner Key: 0
Partner Mac Address: 52:54:00:2f:30:f7
Slave Interface: eth1
MII Status: up
Speed: 10 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 52:54:00:51:25:3c
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 52:54:00:51:25:3c
port key: 5
port priority: 255
port number: 1
port state: 61
details partner lacp pdu:
system priority: 65535
system mac address: 52:54:00:2f:30:f7
oper key: 0
port priority: 255
port number: 1
port state: 73
Slave Interface: eth2
MII Status: up
Speed: 10 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 52:54:00:bf:57:16
Slave queue ID: 0
Aggregator ID: 1
Actor Churn State: monitoring
Partner Churn State: monitoring
Actor Churned Count: 0
Partner Churned Count: 0
details actor lacp pdu:
system priority: 65535
system mac address: 52:54:00:51:25:3c
port key: 5
port priority: 255
port number: 2
port state: 61
details partner lacp pdu:
system priority: 65535
system mac address: 52:54:00:2f:30:f7
oper key: 0
port priority: 255
port number: 1
port state: 73
From: Nikolay Aleksandrov <hidden> Date: 2016-01-28 14:10:53
On 01/28/2016 02:53 PM, Nikolay Aleksandrov wrote:
On 01/28/2016 02:33 PM, Jiri Pirko wrote:
quoted
Mon, Jan 25, 2016 at 05:24:48PM CET, bjornar.ness@gmail.com wrote:
quoted
As subject says, 802.3ad bonding is not working with virtio network model.
The only errors I see is:
No 802.3ad response from the link partner for any adapters in the bond.
Dumping the network traffic shows that no LACP packets are sent from the
host running with virtio driver, changing to for example e1000 solves
this problem
with no configuration changes.
Is this a known problem?
Can you show your bond's /proc/net/bonding/bond<X> ? And also in order to
better see what's going on I'd suggest enabling the pr_debug() calls in the 3ad
code:
echo 'file bond_3ad.c +p' > /sys/kernel/debug/dynamic_debug/control
(assuming you have debugfs mounted at /sys/kernel/debug)
Then you can follow the logs to see what's going on.
I can clearly see LACP packets sent over virtio net devices:
14:53:05.323490 52:54:00:51:25:3c > 01:80:c2:00:00:02, ethertype Slow Protocols (0x8809), length 124: LACPv1, length 110
Nevermind my message, Jiri was right. My setup wasn't correct.
I have a virtio_net get_settings hack that enables me to test 802.3ad over virtio_net
locally.
Sorry about the noise.
quoted
I believe the problem is virtio_net for obvious reasons does not report
speed and duplex. Bonding 3ad mode makes that unconfortable :)
[snip]
Right, of course. My setup has some patches that add get_settings to virtio_net
to make it play nice for testing 803.2ad.
From: Nikolay Aleksandrov <hidden> Date: 2016-01-29 21:31:43
On 01/25/2016 05:24 PM, Bjørnar Ness wrote:
As subject says, 802.3ad bonding is not working with virtio network model.
The only errors I see is:
No 802.3ad response from the link partner for any adapters in the bond.
Dumping the network traffic shows that no LACP packets are sent from the
host running with virtio driver, changing to for example e1000 solves
this problem
with no configuration changes.
Is this a known problem?
[Including bonding maintainers for comments]
Hi,
Here's a workaround patch for virtio_net devices that "cheats" the
duplex test (which is the actual problem). I've tested this locally
and it works for me.
I'd let the others comment on the implementation, there're other signs
that can be used to distinguish a virtio_net device so I'm open to suggestions.
Also feedback if this is at all acceptable would be appreciated.
Could you give it a try ?
@@ -334,6 +334,7 @@ static u16 __get_link_speed(struct port *port)staticu8__get_duplex(structport*port){structslave*slave=port->slave;+conststructnet_device*slave_dev=slave->dev;u8retval=0x0;/* handling a special case: when the configuration starts with
From: Jay Vosburgh <hidden> Date: 2016-01-29 21:45:05
Nikolay Aleksandrov [off-list ref] wrote:
On 01/25/2016 05:24 PM, Bjørnar Ness wrote:
quoted
As subject says, 802.3ad bonding is not working with virtio network model.
The only errors I see is:
No 802.3ad response from the link partner for any adapters in the bond.
Dumping the network traffic shows that no LACP packets are sent from the
host running with virtio driver, changing to for example e1000 solves
this problem
with no configuration changes.
Is this a known problem?
[Including bonding maintainers for comments]
Hi,
Here's a workaround patch for virtio_net devices that "cheats" the
duplex test (which is the actual problem). I've tested this locally
and it works for me.
I'd let the others comment on the implementation, there're other signs
that can be used to distinguish a virtio_net device so I'm open to suggestions.
Also feedback if this is at all acceptable would be appreciated.
Should virtio instead provide an arbitrary speed and full duplex
to ethtool, as veth does?
Creating a magic whitelist of devices deep inside the 802.3ad
implementation seems less desirable.
-J
---
-Jay Vosburgh, jay.vosburgh@canonical.com
From: Nikolay Aleksandrov <hidden> Date: 2016-01-29 21:48:29
On 01/29/2016 10:45 PM, Jay Vosburgh wrote:
Nikolay Aleksandrov [off-list ref] wrote:
quoted
On 01/25/2016 05:24 PM, Bjørnar Ness wrote:
quoted
As subject says, 802.3ad bonding is not working with virtio network model.
The only errors I see is:
No 802.3ad response from the link partner for any adapters in the bond.
Dumping the network traffic shows that no LACP packets are sent from the
host running with virtio driver, changing to for example e1000 solves
this problem
with no configuration changes.
Is this a known problem?
[Including bonding maintainers for comments]
Hi,
Here's a workaround patch for virtio_net devices that "cheats" the
duplex test (which is the actual problem). I've tested this locally
and it works for me.
I'd let the others comment on the implementation, there're other signs
that can be used to distinguish a virtio_net device so I'm open to suggestions.
Also feedback if this is at all acceptable would be appreciated.
Should virtio instead provide an arbitrary speed and full duplex
to ethtool, as veth does?
Creating a magic whitelist of devices deep inside the 802.3ad
implementation seems less desirable.
TBH, I absolutely agree. In fact here's what we've been doing:
add set_settings which allows the user to set any speed/duplex
and get_settings of course to retrieve that. This is also useful
for testing other stuff that requires speed and duplex, not only
for the bonding case.
I'll add the virtio_net maintainers to the discussion, see if it's
okay with everyone and I'll move to send patches once net-next opens up.
Thanks!
From: David Miller <davem@davemloft.net> Date: 2016-01-30 06:59:24
From: Nikolay Aleksandrov <redacted>
Date: Fri, 29 Jan 2016 22:48:26 +0100
On 01/29/2016 10:45 PM, Jay Vosburgh wrote:
quoted
Nikolay Aleksandrov [off-list ref] wrote:
quoted
On 01/25/2016 05:24 PM, Bjørnar Ness wrote:
quoted
As subject says, 802.3ad bonding is not working with virtio network model.
The only errors I see is:
No 802.3ad response from the link partner for any adapters in the bond.
Dumping the network traffic shows that no LACP packets are sent from the
host running with virtio driver, changing to for example e1000 solves
this problem
with no configuration changes.
Is this a known problem?
[Including bonding maintainers for comments]
Hi,
Here's a workaround patch for virtio_net devices that "cheats" the
duplex test (which is the actual problem). I've tested this locally
and it works for me.
I'd let the others comment on the implementation, there're other signs
that can be used to distinguish a virtio_net device so I'm open to suggestions.
Also feedback if this is at all acceptable would be appreciated.
Should virtio instead provide an arbitrary speed and full duplex
to ethtool, as veth does?
Creating a magic whitelist of devices deep inside the 802.3ad
implementation seems less desirable.
TBH, I absolutely agree. In fact here's what we've been doing:
add set_settings which allows the user to set any speed/duplex
and get_settings of course to retrieve that. This is also useful
for testing other stuff that requires speed and duplex, not only
for the bonding case.
I also agree. Having a whitelist is just rediculous.
There should be a default speed/duplex setting for such devices as well.
We can pick one that will be use universally for these kinds of devices.
Sat, Jan 30, 2016 at 07:59:24AM CET, davem@davemloft.net wrote:
From: Nikolay Aleksandrov <redacted>
Date: Fri, 29 Jan 2016 22:48:26 +0100
quoted
On 01/29/2016 10:45 PM, Jay Vosburgh wrote:
quoted
Nikolay Aleksandrov [off-list ref] wrote:
quoted
On 01/25/2016 05:24 PM, Bjørnar Ness wrote:
quoted
As subject says, 802.3ad bonding is not working with virtio network model.
The only errors I see is:
No 802.3ad response from the link partner for any adapters in the bond.
Dumping the network traffic shows that no LACP packets are sent from the
host running with virtio driver, changing to for example e1000 solves
this problem
with no configuration changes.
Is this a known problem?
[Including bonding maintainers for comments]
Hi,
Here's a workaround patch for virtio_net devices that "cheats" the
duplex test (which is the actual problem). I've tested this locally
and it works for me.
I'd let the others comment on the implementation, there're other signs
that can be used to distinguish a virtio_net device so I'm open to suggestions.
Also feedback if this is at all acceptable would be appreciated.
Should virtio instead provide an arbitrary speed and full duplex
to ethtool, as veth does?
Creating a magic whitelist of devices deep inside the 802.3ad
implementation seems less desirable.
TBH, I absolutely agree. In fact here's what we've been doing:
add set_settings which allows the user to set any speed/duplex
and get_settings of course to retrieve that. This is also useful
for testing other stuff that requires speed and duplex, not only
for the bonding case.
I also agree. Having a whitelist is just rediculous.
There should be a default speed/duplex setting for such devices as well.
We can pick one that will be use universally for these kinds of devices.
Exposing made up speed for veth and virtio_net sounds odd to me. User
see 10000Mb/s but it makes no sense at all. It is just confusing.
I believe this is bonding bug and should be fixed in there. Team works
fine with virtio_net device and lacp runner.
From: Nikolay Aleksandrov <hidden> Date: 2016-01-30 11:41:21
On 01/30/2016 07:59 AM, David Miller wrote:
From: Nikolay Aleksandrov <redacted>
Date: Fri, 29 Jan 2016 22:48:26 +0100
quoted
On 01/29/2016 10:45 PM, Jay Vosburgh wrote:
quoted
Nikolay Aleksandrov [off-list ref] wrote:
quoted
On 01/25/2016 05:24 PM, Bjørnar Ness wrote:
quoted
As subject says, 802.3ad bonding is not working with virtio network model.
The only errors I see is:
No 802.3ad response from the link partner for any adapters in the bond.
Dumping the network traffic shows that no LACP packets are sent from the
host running with virtio driver, changing to for example e1000 solves
this problem
with no configuration changes.
Is this a known problem?
[Including bonding maintainers for comments]
Hi,
Here's a workaround patch for virtio_net devices that "cheats" the
duplex test (which is the actual problem). I've tested this locally
and it works for me.
I'd let the others comment on the implementation, there're other signs
that can be used to distinguish a virtio_net device so I'm open to suggestions.
Also feedback if this is at all acceptable would be appreciated.
Should virtio instead provide an arbitrary speed and full duplex
to ethtool, as veth does?
Creating a magic whitelist of devices deep inside the 802.3ad
implementation seems less desirable.
TBH, I absolutely agree. In fact here's what we've been doing:
add set_settings which allows the user to set any speed/duplex
and get_settings of course to retrieve that. This is also useful
for testing other stuff that requires speed and duplex, not only
for the bonding case.
I also agree. Having a whitelist is just rediculous.
There should be a default speed/duplex setting for such devices as well.
We can pick one that will be use universally for these kinds of devices.
Yes, that's the other thing - the default setting, from a brief grepping
I see that veth uses 10Gbps, tun uses 10Mbps and batman-adv uses 10Mbps.
If we add a default get_settings that can be used by virtual devices
in ethtool that returns 10Gbps with the settings set like veth does
sounds good to me. What do you think ?
In fact they all set the same settings (apart from speed) so we can
consolidate them in a single default setting init function and for the
ones using different speed do something like:
__ethtool_init_settings(); <- sets everything like veth would with 10Mbps speed
__ethtool_cmd_speed_set(cmd, SPEED_10000);
And we're done.
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2016-01-31 14:35:39
On Fri, Jan 29, 2016 at 01:45:00PM -0800, Jay Vosburgh wrote:
Nikolay Aleksandrov [off-list ref] wrote:
quoted
On 01/25/2016 05:24 PM, Bjørnar Ness wrote:
quoted
As subject says, 802.3ad bonding is not working with virtio network model.
The only errors I see is:
No 802.3ad response from the link partner for any adapters in the bond.
Dumping the network traffic shows that no LACP packets are sent from the
host running with virtio driver, changing to for example e1000 solves
this problem
with no configuration changes.
Is this a known problem?
[Including bonding maintainers for comments]
Hi,
Here's a workaround patch for virtio_net devices that "cheats" the
duplex test (which is the actual problem). I've tested this locally
and it works for me.
I'd let the others comment on the implementation, there're other signs
that can be used to distinguish a virtio_net device so I'm open to suggestions.
Also feedback if this is at all acceptable would be appreciated.
Should virtio instead provide an arbitrary speed and full duplex
to ethtool, as veth does?
Creating a magic whitelist of devices deep inside the 802.3ad
implementation seems less desirable.
-J
Absolutely but why not tream DUPLEX_UNKNOWN as DUPLEX_FULL
and allow LACP?
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2016-01-31 14:50:38
On Sat, Jan 30, 2016 at 12:41:16PM +0100, Nikolay Aleksandrov wrote:
On 01/30/2016 07:59 AM, David Miller wrote:
quoted
From: Nikolay Aleksandrov <redacted>
Date: Fri, 29 Jan 2016 22:48:26 +0100
quoted
On 01/29/2016 10:45 PM, Jay Vosburgh wrote:
quoted
Nikolay Aleksandrov [off-list ref] wrote:
quoted
On 01/25/2016 05:24 PM, Bjørnar Ness wrote:
quoted
As subject says, 802.3ad bonding is not working with virtio network model.
The only errors I see is:
No 802.3ad response from the link partner for any adapters in the bond.
Dumping the network traffic shows that no LACP packets are sent from the
host running with virtio driver, changing to for example e1000 solves
this problem
with no configuration changes.
Is this a known problem?
[Including bonding maintainers for comments]
Hi,
Here's a workaround patch for virtio_net devices that "cheats" the
duplex test (which is the actual problem). I've tested this locally
and it works for me.
I'd let the others comment on the implementation, there're other signs
that can be used to distinguish a virtio_net device so I'm open to suggestions.
Also feedback if this is at all acceptable would be appreciated.
Should virtio instead provide an arbitrary speed and full duplex
to ethtool, as veth does?
Creating a magic whitelist of devices deep inside the 802.3ad
implementation seems less desirable.
TBH, I absolutely agree. In fact here's what we've been doing:
add set_settings which allows the user to set any speed/duplex
and get_settings of course to retrieve that. This is also useful
for testing other stuff that requires speed and duplex, not only
for the bonding case.
I also agree. Having a whitelist is just rediculous.
There should be a default speed/duplex setting for such devices as well.
We can pick one that will be use universally for these kinds of devices.
Yes, that's the other thing - the default setting, from a brief grepping
I see that veth uses 10Gbps, tun uses 10Mbps and batman-adv uses 10Mbps.
If we add a default get_settings that can be used by virtual devices
in ethtool that returns 10Gbps with the settings set like veth does
sounds good to me. What do you think ?
In fact they all set the same settings (apart from speed) so we can
consolidate them in a single default setting init function and for the
ones using different speed do something like:
__ethtool_init_settings(); <- sets everything like veth would with 10Mbps speed
__ethtool_cmd_speed_set(cmd, SPEED_10000);
And we're done.
I think leaving speed alone would be preferable if possible.
Faking speed might cause strange effects e.g. bonding prefering
virtio over a real device unexpectedly.
Also, wouldn't it break same link speed requirements?
Same applies to duplex really.
We could easily extend virtio with ability to pass
speed/duplex info from host, but that's not always
known to host and there are existing hypervisors,
so we are still left with the question of what to do
when this info is not available.
Is virtio somehow special that we want LACP for it
if duplex state and speed are unknown, but
don't want it for other devices?
If no, let's allow it for all such devices?
-->
bond_3ad: allow LACP when duplex is unknown
Devices would sometimes set speed and duplex as unknown.
It seem reasonable to allow LACP in this case.
Signed-off-by: Michael S. Tsirkin <mst@redhat.com>
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2016-01-31 14:58:42
On Fri, Jan 29, 2016 at 10:48:26PM +0100, Nikolay Aleksandrov wrote:
On 01/29/2016 10:45 PM, Jay Vosburgh wrote:
quoted
Nikolay Aleksandrov [off-list ref] wrote:
quoted
On 01/25/2016 05:24 PM, Bjørnar Ness wrote:
quoted
As subject says, 802.3ad bonding is not working with virtio network model.
The only errors I see is:
No 802.3ad response from the link partner for any adapters in the bond.
Dumping the network traffic shows that no LACP packets are sent from the
host running with virtio driver, changing to for example e1000 solves
this problem
with no configuration changes.
Is this a known problem?
[Including bonding maintainers for comments]
Hi,
Here's a workaround patch for virtio_net devices that "cheats" the
duplex test (which is the actual problem). I've tested this locally
and it works for me.
I'd let the others comment on the implementation, there're other signs
that can be used to distinguish a virtio_net device so I'm open to suggestions.
Also feedback if this is at all acceptable would be appreciated.
Should virtio instead provide an arbitrary speed and full duplex
to ethtool, as veth does?
Creating a magic whitelist of devices deep inside the 802.3ad
implementation seems less desirable.
TBH, I absolutely agree. In fact here's what we've been doing:
add set_settings which allows the user to set any speed/duplex
and get_settings of course to retrieve that. This is also useful
for testing other stuff that requires speed and duplex, not only
for the bonding case.
This looks like a very reasonable thing to do:
user might have knowledge of the actual speed through
some side-channel.
We might also propagate it to hypervisor in the future.
And this sound useful even if bonding is changed
to allow DUPLEX_UNKNOWN.
So please post this patch.
I'll add the virtio_net maintainers to the discussion, see if it's
okay with everyone and I'll move to send patches once net-next opens up.
Thanks!
From: Rick Jones <hidden> Date: 2016-02-01 18:49:54
On 01/29/2016 10:59 PM, David Miller wrote:
There should be a default speed/duplex setting for such devices as well.
We can pick one that will be use universally for these kinds of devices.
There is at least one monitoring tool - collectl - which gets a trifle
upset when the actual speed through an interface is significantly
greater than the reported link speed. I have to wonder how unique it is
in that regard.
Doesn't mean there can't be a default, but does suggest it should be
rather high.
rick jones