vhost [query] : support for multiple ports and non VMDQ devices in vhost switch

Hi,

I am working on an NXP platform where we intend to use user space vhost switch (examples/vhost) as backend for VIRTIO devices. But there are two limitations in current vhost-switch (examples/vhost)that are restricting my use case:

1. The vhost-switch application is tightly integrated with Intel VMDQ. Since my device doesn't have VMDQ i can not use this application directly.

2. The vhost-switch application supports only one external or physical port (non virtio devices), but my requirement is to have multiple physical ports and multiple virtio devices.

In summary my requirement is to do whatever vhost-switch is doing, in addition to that add support for following:

1. support devices that don't have VMDQ.
2. Support multiple physical ports.

I need suggestions on the approach i should take: whether to add support of above mentioned in existing vhost-switch (examples/vhost) or write another application (based on librte_vhost only) to support my requirements.

I'll work on it after the suggestion i get from the list, and send the RFC patch.

Thanks,
Pankaj

On Tue, Aug 09, 2016 at 04:42:33PM +0530, Pankaj Chauhan wrote:

Hi,

I am working on an NXP platform where we intend to use user space vhost
switch (examples/vhost) as backend for VIRTIO devices. But there are two
limitations in current vhost-switch (examples/vhost)that are restricting my
use case:

1. The vhost-switch application is tightly integrated with Intel VMDQ. Since
my device doesn't have VMDQ i can not use this application directly.

Sorry being late (I was on biz trip last week).

Yes, a vhost example should not do that. We have an internal TODO to
remove it. Actually, to make it optional, and Jianfeng was working on
that. Well, seems that you two have already had some discussions.

2. The vhost-switch application supports only one external or physical port
(non virtio devices), but my requirement is to have multiple physical ports
and multiple virtio devices.

What are you going to achieve? BTW, have you tried testpmd (with vhost-pmd)?
I'm with impression that it might be a better option to you.

	--yliu

In summary my requirement is to do whatever vhost-switch is doing, in
addition to that add support for following:

1. support devices that don't have VMDQ.
2. Support multiple physical ports.

I need suggestions on the approach i should take: whether to add support of
above mentioned in existing vhost-switch (examples/vhost) or write another
application (based on librte_vhost only) to support my requirements.

I'll work on it after the suggestion i get from the list, and send the RFC
patch.

Thanks,
Pankaj

Hi,

Please review below proposal of Pankaj and myself after an offline discussion. (Pankaj, please correct me if I'm going somewhere wrong).

a. Remove HW dependent option, --strip-vlan, because different kinds of NICs behave differently. It's a bug fix.
b. Abstract switching logic into a framework, so that we can develop different kinds of switching logics. In this phase, we will have two switching logics: (1) a simple software-based mac learning switching; (2) VMDQ based switching. Any other advanced switching logics can be proposed based on this framework.
c. Merge tep_termination example vxlan as a switching logic of the framework.

To be decided:
d. Support multiple physical ports.
e. Keep the current way to use vhost lib directly or use vhost pmd instead.

On 8/16/2016 10:56 AM, Yuanhan Liu wrote:

On Tue, Aug 09, 2016 at 04:42:33PM +0530, Pankaj Chauhan wrote:

quoted

Hi,

I am working on an NXP platform where we intend to use user space vhost
switch (examples/vhost) as backend for VIRTIO devices. But there are two
limitations in current vhost-switch (examples/vhost)that are restricting my
use case:

1. The vhost-switch application is tightly integrated with Intel VMDQ. Since
my device doesn't have VMDQ i can not use this application directly.

Sorry being late (I was on biz trip last week).

Yes, a vhost example should not do that. We have an internal TODO to
remove it. Actually, to make it optional, and Jianfeng was working on
that. Well, seems that you two have already had some discussions.

quoted

2. The vhost-switch application supports only one external or physical port
(non virtio devices), but my requirement is to have multiple physical ports
and multiple virtio devices.

What are you going to achieve? BTW, have you tried testpmd (with vhost-pmd)?

Example testpmd will not allow those complex switching logics.

Thanks,
Jianfeng

I'm with impression that it might be a better option to you.

	--yliu

quoted

In summary my requirement is to do whatever vhost-switch is doing, in
addition to that add support for following:

1. support devices that don't have VMDQ.
2. Support multiple physical ports.

I need suggestions on the approach i should take: whether to add support of
above mentioned in existing vhost-switch (examples/vhost) or write another
application (based on librte_vhost only) to support my requirements.

I'll work on it after the suggestion i get from the list, and send the RFC
patch.

Thanks,
Pankaj

On 8/16/2016 8:26 AM, Yuanhan Liu wrote:

On Tue, Aug 09, 2016 at 04:42:33PM +0530, Pankaj Chauhan wrote:

quoted

Hi,

I am working on an NXP platform where we intend to use user space vhost
switch (examples/vhost) as backend for VIRTIO devices. But there are two
limitations in current vhost-switch (examples/vhost)that are restricting my
use case:

1. The vhost-switch application is tightly integrated with Intel VMDQ. Since
my device doesn't have VMDQ i can not use this application directly.

Sorry being late (I was on biz trip last week).

Yes, a vhost example should not do that. We have an internal TODO to
remove it. Actually, to make it optional, and Jianfeng was working on
that. Well, seems that you two have already had some discussions.

Yes I had a discussion with Jianfeng and he has sent the work items on the list.

quoted

2. The vhost-switch application supports only one external or physical port
(non virtio devices), but my requirement is to have multiple physical ports
and multiple virtio devices.

What are you going to achieve? BTW, have you tried testpmd (with vhost-pmd)?
I'm with impression that it might be a better option to you.

	--yliu

My use case is that my machine/board which is not sitting as end node of network but somewhere in between like an router. So the traffic looks something like this:

Physical port 1 -> Enter VM(s) through virtio -> exit from physical port 2

For above use case i need a vhost-back-end which supports multiple physical ports.

Thanks for the suggestion of vhost-pmd ( i was not aware of that), i'll explore possibility of using it for my use case of multiple physical ports.

thanks,
Pankaj

Hi Jianfeng,

On 08/17/2016 04:33 AM, Tan, Jianfeng wrote:

Hi,

Please review below proposal of Pankaj and myself after an offline
discussion. (Pankaj, please correct me if I'm going somewhere wrong).

a. Remove HW dependent option, --strip-vlan, because different kinds of
NICs behave differently. It's a bug fix.
b. Abstract switching logic into a framework, so that we can develop
different kinds of switching logics. In this phase, we will have two
switching logics: (1) a simple software-based mac learning switching;
(2) VMDQ based switching. Any other advanced switching logics can be
proposed based on this framework.
c. Merge tep_termination example vxlan as a switching logic of the
framework.

I was also thinking of making physical port optional and add MAC learning,
so this is all good for me.

Let me know if I can help in implementation, I'll be happy to
contribute.

To be decided:
d. Support multiple physical ports.
e. Keep the current way to use vhost lib directly or use vhost pmd instead.

Do you see advantages of using vhost lib directly vs. pmd?
Wouldn't using vhost pmd make achieving zero-copy harder?
(I'm not sure, I didn't investigate the topic much for now).

Also, if we use pmd directly, then it would no more be a vhost switch
only, as it could potentially be used with physical NICs also.

Any thoughts?

Thanks,
Maxime

On 8/16/2016 10:56 AM, Yuanhan Liu wrote:

quoted

On Tue, Aug 09, 2016 at 04:42:33PM +0530, Pankaj Chauhan wrote:

quoted

Hi,

I am working on an NXP platform where we intend to use user space vhost
switch (examples/vhost) as backend for VIRTIO devices. But there are two
limitations in current vhost-switch (examples/vhost)that are
restricting my
use case:

1. The vhost-switch application is tightly integrated with Intel
VMDQ. Since
my device doesn't have VMDQ i can not use this application directly.

Sorry being late (I was on biz trip last week).

Yes, a vhost example should not do that. We have an internal TODO to
remove it. Actually, to make it optional, and Jianfeng was working on
that. Well, seems that you two have already had some discussions.

quoted

2. The vhost-switch application supports only one external or
physical port
(non virtio devices), but my requirement is to have multiple physical
ports
and multiple virtio devices.

What are you going to achieve? BTW, have you tried testpmd (with
vhost-pmd)?

Example testpmd will not allow those complex switching logics.

Thanks,
Jianfeng

quoted

I'm with impression that it might be a better option to you.

    --yliu

quoted

In summary my requirement is to do whatever vhost-switch is doing, in
addition to that add support for following:

1. support devices that don't have VMDQ.
2. Support multiple physical ports.

I need suggestions on the approach i should take: whether to add
support of
above mentioned in existing vhost-switch (examples/vhost) or write
another
application (based on librte_vhost only) to support my requirements.

I'll work on it after the suggestion i get from the list, and send
the RFC
patch.

Thanks,
Pankaj

Hi Maxime,

On 8/17/2016 7:18 PM, Maxime Coquelin wrote:

Hi Jianfeng,

On 08/17/2016 04:33 AM, Tan, Jianfeng wrote:

quoted

Hi,

Please review below proposal of Pankaj and myself after an offline
discussion. (Pankaj, please correct me if I'm going somewhere wrong).

a. Remove HW dependent option, --strip-vlan, because different kinds of
NICs behave differently. It's a bug fix.
b. Abstract switching logic into a framework, so that we can develop
different kinds of switching logics. In this phase, we will have two
switching logics: (1) a simple software-based mac learning switching;
(2) VMDQ based switching. Any other advanced switching logics can be
proposed based on this framework.
c. Merge tep_termination example vxlan as a switching logic of the
framework.

I was also thinking of making physical port optional and add MAC learning,
so this is all good for me.

To make it clear, we are not proposing to eliminate physical port, instead, we just eliminate the binding of VMDQ and virtio ports, superseding it with a MAC learning switching.

Let me know if I can help in implementation, I'll be happy to
contribute.

Thank you for participating. Currently, I'm working on item a (will be a quick and simple fix). Pankaj is working on item b (which would be a huge change). Item c is depending on item b. So let's wait RFC patch from Pankaj and see what we can help.

quoted

To be decided:
d. Support multiple physical ports.
e. Keep the current way to use vhost lib directly or use vhost pmd instead.

Do you see advantages of using vhost lib directly vs. pmd?
Wouldn't using vhost pmd make achieving zero-copy harder?
(I'm not sure, I didn't investigate the topic much for now).

Yes, by using vhost lib, we can add back the removed feature zero-copy. But my understanding is zero-copy (nic-to-vm or vm-to-nic) or delayed copy (vm-to-vm) would be great and common features, which should be integrated into vhost lib and enabled in vhost pmd, so that all applications can benefit from it. And in fact, Yuanhan is working on the delayed copy now. An exception is rx-side-zero-copy, I don't know if it's common enough to be integrated in a vhost lib, because it'll require hardware queue binding.

Besides, vhost pmd would be easier to use than vhost lib (personal opinion). Secondly, vhost pmd would be more clear in logic, 1:1:1 mapping among vhost port, unix socket path, and virtio port. Thirdly, by using vhost pmd, we can treat vhost ports the way of physical ports, otherwise, we use different API to receive/transmit packets.

Also, if we use pmd directly, then it would no more be a vhost switch
only, as it could potentially be used with physical NICs also.

You mean we are building a switch instead of vhost switch? Yes, a switch can switch packets between virtio-virtio and virtio-physical nic.

Thanks,
Jianfeng

Any thoughts?

Thanks,
Maxime

Hi,

On 08/18/2016 04:35 AM, Tan, Jianfeng wrote:

Hi Maxime,

On 8/17/2016 7:18 PM, Maxime Coquelin wrote:

quoted

Hi Jianfeng,

On 08/17/2016 04:33 AM, Tan, Jianfeng wrote:

quoted

Hi,

Please review below proposal of Pankaj and myself after an offline
discussion. (Pankaj, please correct me if I'm going somewhere wrong).

a. Remove HW dependent option, --strip-vlan, because different kinds of
NICs behave differently. It's a bug fix.
b. Abstract switching logic into a framework, so that we can develop
different kinds of switching logics. In this phase, we will have two
switching logics: (1) a simple software-based mac learning switching;
(2) VMDQ based switching. Any other advanced switching logics can be
proposed based on this framework.
c. Merge tep_termination example vxlan as a switching logic of the
framework.

I was also thinking of making physical port optional and add MAC
learning,
so this is all good for me.

To make it clear, we are not proposing to eliminate physical port,
instead, we just eliminate the binding of VMDQ and virtio ports,
superseding it with a MAC learning switching.

So you confirm we could have setup with only VMs, and no physical
NIC? That's what I meant when saying "making physical port optional".

quoted

Let me know if I can help in implementation, I'll be happy to
contribute.

Thank you for participating. Currently, I'm working on item a (will be a
quick and simple fix). Pankaj is working on item b (which would be a
huge change). Item c is depending on item b. So let's wait RFC patch
from Pankaj and see what we can help.

Good, let's wait for Pankaj's RFC.

quoted

quoted

To be decided:
d. Support multiple physical ports.
e. Keep the current way to use vhost lib directly or use vhost pmd
instead.

Do you see advantages of using vhost lib directly vs. pmd?
Wouldn't using vhost pmd make achieving zero-copy harder?
(I'm not sure, I didn't investigate the topic much for now).

Yes, by using vhost lib, we can add back the removed feature zero-copy.
But my understanding is zero-copy (nic-to-vm or vm-to-nic) or delayed
copy (vm-to-vm) would be great and common features, which should be
integrated into vhost lib and enabled in vhost pmd, so that all
applications can benefit from it. And in fact, Yuanhan is working on the
delayed copy now. An exception is rx-side-zero-copy, I don't know if
it's common enough to be integrated in a vhost lib, because it'll
require hardware queue binding.

Ok, I'm interrested in knowing how vm-to-vm delayed copy will be
implemented.

Besides, vhost pmd would be easier to use than vhost lib (personal
opinion). Secondly, vhost pmd would be more clear in logic, 1:1:1
mapping among vhost port, unix socket path, and virtio port. Thirdly, by
using vhost pmd, we can treat vhost ports the way of physical ports,
otherwise, we use different API to receive/transmit packets.

I'm 100% aligned with you on this, the vhost pmd makes things more
standard, so more flexible.

quoted

Also, if we use pmd directly, then it would no more be a vhost switch
only, as it could potentially be used with physical NICs also.

You mean we are building a switch instead of vhost switch? Yes, a switch
can switch packets between virtio-virtio and virtio-physical nic.

And physical-physical also, as we will be standard API with the
vhost-pmd, nothing will prevent using it with only physical switches,
no?

Thanks,
Maxime

Thanks,
Jianfeng

quoted

Any thoughts?

Thanks,
Maxime

On Wed, Aug 17, 2016 at 03:54:21PM +0530, Pankaj Chauhan wrote:

My use case is that my machine/board which is not sitting as end node of
network but somewhere in between like an router. So the traffic looks
something like this:

Physical port 1 -> Enter VM(s) through virtio -> exit from physical port 2

I'm not quite sure testpmd has this kind of support or not, routing the
data from a specific port to another specific port. Zhihong might have
the answer.

	--yliu

For above use case i need a vhost-back-end which supports multiple physical
ports.

Thanks for the suggestion of vhost-pmd ( i was not aware of that), i'll
explore possibility of using it for my use case of multiple physical ports.

Hi Maxime,

-----Original Message-----
From: Maxime Coquelin [mailto:maxime.coquelin@redhat.com]
Sent: Thursday, August 18, 2016 3:43 PM
To: Tan, Jianfeng; Yuanhan Liu; Pankaj Chauhan
Cc: dev@dpdk.org; hemant.agrawal@nxp.com; shreyansh.jain@nxp.com
Subject: Re: [dpdk-dev] vhost [query] : support for multiple ports and non
VMDQ devices in vhost switch

Hi,

On 08/18/2016 04:35 AM, Tan, Jianfeng wrote:

quoted

Hi Maxime,

On 8/17/2016 7:18 PM, Maxime Coquelin wrote:

quoted

Hi Jianfeng,

On 08/17/2016 04:33 AM, Tan, Jianfeng wrote:

quoted

Hi,

Please review below proposal of Pankaj and myself after an offline
discussion. (Pankaj, please correct me if I'm going somewhere wrong).

a. Remove HW dependent option, --strip-vlan, because different kinds

of

quoted

quoted

quoted

NICs behave differently. It's a bug fix.
b. Abstract switching logic into a framework, so that we can develop
different kinds of switching logics. In this phase, we will have two
switching logics: (1) a simple software-based mac learning switching;
(2) VMDQ based switching. Any other advanced switching logics can be
proposed based on this framework.
c. Merge tep_termination example vxlan as a switching logic of the
framework.

I was also thinking of making physical port optional and add MAC
learning,
so this is all good for me.

To make it clear, we are not proposing to eliminate physical port,
instead, we just eliminate the binding of VMDQ and virtio ports,
superseding it with a MAC learning switching.

So you confirm we could have setup with only VMs, and no physical
NIC? That's what I meant when saying "making physical port optional".

Yes, this case would be supported too.

quoted

quoted

Let me know if I can help in implementation, I'll be happy to
contribute.

Thank you for participating. Currently, I'm working on item a (will be a
quick and simple fix). Pankaj is working on item b (which would be a
huge change). Item c is depending on item b. So let's wait RFC patch
from Pankaj and see what we can help.

Good, let's wait for Pankaj's RFC.

quoted

quoted

quoted

To be decided:
d. Support multiple physical ports.
e. Keep the current way to use vhost lib directly or use vhost pmd
instead.

Do you see advantages of using vhost lib directly vs. pmd?
Wouldn't using vhost pmd make achieving zero-copy harder?
(I'm not sure, I didn't investigate the topic much for now).

Yes, by using vhost lib, we can add back the removed feature zero-copy.
But my understanding is zero-copy (nic-to-vm or vm-to-nic) or delayed
copy (vm-to-vm) would be great and common features, which should be
integrated into vhost lib and enabled in vhost pmd, so that all
applications can benefit from it. And in fact, Yuanhan is working on the
delayed copy now. An exception is rx-side-zero-copy, I don't know if
it's common enough to be integrated in a vhost lib, because it'll
require hardware queue binding.

Ok, I'm interrested in knowing how vm-to-vm delayed copy will be
implemented.

quoted

Besides, vhost pmd would be easier to use than vhost lib (personal
opinion). Secondly, vhost pmd would be more clear in logic, 1:1:1
mapping among vhost port, unix socket path, and virtio port. Thirdly, by
using vhost pmd, we can treat vhost ports the way of physical ports,
otherwise, we use different API to receive/transmit packets.

I'm 100% aligned with you on this, the vhost pmd makes things more
standard, so more flexible.

quoted

quoted

Also, if we use pmd directly, then it would no more be a vhost switch
only, as it could potentially be used with physical NICs also.

You mean we are building a switch instead of vhost switch? Yes, a switch
can switch packets between virtio-virtio and virtio-physical nic.

And physical-physical also, as we will be standard API with the
vhost-pmd, nothing will prevent using it with only physical switches,
no?

Oh yes, I agree.

Thanks,
Jianfeng

Thanks,
Maxime

quoted

Thanks,
Jianfeng

quoted

Any thoughts?

Thanks,
Maxime

2016-08-18 10:36, Tan, Jianfeng:

From: Maxime Coquelin [mailto:maxime.coquelin@redhat.com]

quoted

quoted

On 8/17/2016 7:18 PM, Maxime Coquelin wrote:

quoted

On 08/17/2016 04:33 AM, Tan, Jianfeng wrote:

quoted

quoted

quoted

quoted

b. Abstract switching logic into a framework, so that we can develop
different kinds of switching logics. In this phase, we will have two
switching logics: (1) a simple software-based mac learning switching;
(2) VMDQ based switching. Any other advanced switching logics can be
proposed based on this framework.
c. Merge tep_termination example vxlan as a switching logic of the
framework.

[...]

quoted

quoted

quoted

quoted

To be decided:
d. Support multiple physical ports.

[...]

quoted

quoted

quoted

Also, if we use pmd directly, then it would no more be a vhost switch
only, as it could potentially be used with physical NICs also.

You mean we are building a switch instead of vhost switch? Yes, a switch
can switch packets between virtio-virtio and virtio-physical nic.

And physical-physical also, as we will be standard API with the
vhost-pmd, nothing will prevent using it with only physical switches,
no?

Oh yes, I agree.

It looks to be a nice project.
I think you should start a new git repository in DPDK apps:
	http://dpdk.org/browse/
And we could remove tep_termination and vhost examples.

On 08/18/2016 04:35 AM, Tan, Jianfeng wrote:

Hi Maxime,

On 8/17/2016 7:18 PM, Maxime Coquelin wrote:

quoted

Hi Jianfeng,

On 08/17/2016 04:33 AM, Tan, Jianfeng wrote:

quoted

Hi,

Please review below proposal of Pankaj and myself after an offline
discussion. (Pankaj, please correct me if I'm going somewhere wrong).

a. Remove HW dependent option, --strip-vlan, because different kinds of
NICs behave differently. It's a bug fix.
b. Abstract switching logic into a framework, so that we can develop
different kinds of switching logics. In this phase, we will have two
switching logics: (1) a simple software-based mac learning switching;
(2) VMDQ based switching. Any other advanced switching logics can be
proposed based on this framework.
c. Merge tep_termination example vxlan as a switching logic of the
framework.

I was also thinking of making physical port optional and add MAC
learning,
so this is all good for me.

To make it clear, we are not proposing to eliminate physical port,
instead, we just eliminate the binding of VMDQ and virtio ports,
superseding it with a MAC learning switching.

quoted

Let me know if I can help in implementation, I'll be happy to
contribute.

Thank you for participating. Currently, I'm working on item a (will be a
quick and simple fix). Pankaj is working on item b (which would be a
huge change). Item c is depending on item b. So let's wait RFC patch
from Pankaj and see what we can help.

Pankaj, so I organize myself , do you have an idea of when the RFC
patch will be available?

Thanks,
Maxime

On 8/24/2016 12:58 PM, Maxime Coquelin wrote:

On 08/18/2016 04:35 AM, Tan, Jianfeng wrote:

quoted

Hi Maxime,

On 8/17/2016 7:18 PM, Maxime Coquelin wrote:

quoted

Hi Jianfeng,

On 08/17/2016 04:33 AM, Tan, Jianfeng wrote:

quoted

Hi,

Please review below proposal of Pankaj and myself after an offline
discussion. (Pankaj, please correct me if I'm going somewhere wrong).

a. Remove HW dependent option, --strip-vlan, because different kinds of
NICs behave differently. It's a bug fix.
b. Abstract switching logic into a framework, so that we can develop
different kinds of switching logics. In this phase, we will have two
switching logics: (1) a simple software-based mac learning switching;
(2) VMDQ based switching. Any other advanced switching logics can be
proposed based on this framework.
c. Merge tep_termination example vxlan as a switching logic of the
framework.

I was also thinking of making physical port optional and add MAC
learning,
so this is all good for me.

To make it clear, we are not proposing to eliminate physical port,
instead, we just eliminate the binding of VMDQ and virtio ports,
superseding it with a MAC learning switching.

quoted

Let me know if I can help in implementation, I'll be happy to
contribute.

Thank you for participating. Currently, I'm working on item a (will be a
quick and simple fix). Pankaj is working on item b (which would be a
huge change). Item c is depending on item b. So let's wait RFC patch
from Pankaj and see what we can help.

Pankaj, so I organize myself , do you have an idea of when the RFC
patch will be available?

I am almost finishing the first version of RFC patch set, hoping to send it today itself.

Thanks,
Pankaj

Thanks,
Maxime