[Question]About KVM network zero-copy feature!

Hi,All

I searched from git-log, and found that until now we have vhost TX zero-copy experiment feature, how
about RX zero-copy?

For XEN, net-back also only has TX zero-copy, Is there any reason that RX zero-copy still not implemented?

Thanks!

On Fri, 10 Aug 2012 11:34:32 +0800
"Peter Huang(Peng)" [off-list ref] wrote:

Hi,All

I searched from git-log, and found that until now we have vhost TX zero-copy experiment feature, how
about RX zero-copy?

For XEN, net-back also only has TX zero-copy, Is there any reason that RX zero-copy still not implemented?

There is no guarantee that packet will ever be read by receiver. This means zero-copy could
create memory back pressure stalls.

On Fri, 10 Aug 2012 11:34:32 +0800
"Peter Huang(Peng)" [off-list ref] wrote:

quoted

I searched from git-log, and found that until now we have vhost TX zero-copy experiment feature, how
about RX zero-copy?

On 08/11/2012 04:55 PM, Stephen Hemminger wrote:

There is no guarantee that packet will ever be read by receiver. This means zero-copy could
create memory back pressure stalls.

It would be handy if this could be an optional feature, perhaps not 
enabled by default due to the problem with stalls you mentioned. I would 
love to see RX zero-copy implemented natively in KVM, as it might 
alleviate the need for custom solutions like vPF_RING:

http://www.ntop.org/products/pf_ring/vpf_ring/

Every time a packet is copied, especially from kernel space to user 
space, there is an opportunity for it to be dropped on its way to the 
receiving application - which is unacceptable when monitoring high-speed 
networks for security or bandwidth accounting purposes.

I am attempting to find a highly-efficient way to deploy virtualized 
network monitoring sensors (Snort, for example). Ideally I want to 
exploit symmetric hardware-based RSS and SR-IOV functionality for 
load-balancing and packet distribution completely in ASIC. I've found 
other existing work in this area (also using custom drivers) indicating 
significant performance gains in the non-virtualized case:

http://www.ndsl.kaist.edu/~shinae/papers/TR-symRSS.pdf

Is there any interest in exploring native RX zero-copy within the 
mainline KVM networking code?

Thanks,
Robert Vineyard

(adding Xin Xiaohui to the conversation for comment)

According to the NetworkingTodo page on the KVM wiki, zero-copy RX for 
macvtap is in fact on the roadmap, assigned to Xin:

http://www.linux-kvm.org/page/NetworkingTodo

The Release Notes for RHEL 6.2 (originally published on 12/06/2011) also 
specifically mention macvtap/vhost zero-copy capabilities as being 
included as a Technology Preview:

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/6.2_Release_Notes/virtualization.html

Since I've been unable to find much information on the current status of 
this development, can anyone confirm if this functionality is still in 
the works? If so, is there any planned ETA?

Thanks,
Robert Vineyard


On 08/11/2012 05:54 PM, Robert Vineyard wrote:

quoted

On Fri, 10 Aug 2012 11:34:32 +0800
"Peter Huang(Peng)" [off-list ref] wrote:

quoted

I searched from git-log, and found that until now we have vhost TX
zero-copy experiment feature, how
about RX zero-copy?

On 08/11/2012 04:55 PM, Stephen Hemminger wrote:

quoted

There is no guarantee that packet will ever be read by receiver. This
means zero-copy could
create memory back pressure stalls.

It would be handy if this could be an optional feature, perhaps not
enabled by default due to the problem with stalls you mentioned. I would
love to see RX zero-copy implemented natively in KVM, as it might
alleviate the need for custom solutions like vPF_RING:

http://www.ntop.org/products/pf_ring/vpf_ring/

Every time a packet is copied, especially from kernel space to user
space, there is an opportunity for it to be dropped on its way to the
receiving application - which is unacceptable when monitoring high-speed
networks for security or bandwidth accounting purposes.

I am attempting to find a highly-efficient way to deploy virtualized
network monitoring sensors (Snort, for example). Ideally I want to
exploit symmetric hardware-based RSS and SR-IOV functionality for
load-balancing and packet distribution completely in ASIC. I've found
other existing work in this area (also using custom drivers) indicating
significant performance gains in the non-virtualized case:

http://www.ndsl.kaist.edu/~shinae/papers/TR-symRSS.pdf

Is there any interest in exploring native RX zero-copy within the
mainline KVM networking code?

Thanks,
Robert Vineyard
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

On Sat, Aug 11, 2012 at 08:42:44PM -0400, Robert Vineyard wrote:

(adding Xin Xiaohui to the conversation for comment)

According to the NetworkingTodo page on the KVM wiki, zero-copy RX
for macvtap is in fact on the roadmap, assigned to Xin:

http://www.linux-kvm.org/page/NetworkingTodo

AFAIK Xin left Intel and is not working on it.
Contributions are welcome.

The Release Notes for RHEL 6.2 (originally published on 12/06/2011)
also specifically mention macvtap/vhost zero-copy capabilities as
being included as a Technology Preview:

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/6.2_Release_Notes/virtualization.html

Since I've been unable to find much information on the current
status of this development, can anyone confirm if this functionality
is still in the works? If so, is there any planned ETA?

Thanks,
Robert Vineyard

I think this means TX.

-- 
MST

Hi, Michael

IIt will be usefull if we can implement rx zero-copy, could you give
 me some help if you know some technical details or some
references on the internet?

I am wondering may be I can take a deep look on it first then decide
if I can take it over or not.

Thanks a lot.

On 2012/8/12 17:37, Michael S. Tsirkin wrote:

On Sat, Aug 11, 2012 at 08:42:44PM -0400, Robert Vineyard wrote:

quoted

(adding Xin Xiaohui to the conversation for comment)

According to the NetworkingTodo page on the KVM wiki, zero-copy RX
for macvtap is in fact on the roadmap, assigned to Xin:

http://www.linux-kvm.org/page/NetworkingTodo

AFAIK Xin left Intel and is not working on it.
Contributions are welcome.

On 08/12/2012 05:37 AM, Michael S. Tsirkin wrote:

AFAIK Xin left Intel and is not working on it.
Contributions are welcome.

That's too bad... do you know of anyone else (at Intel or otherwise) who 
might be familiar enough with the existing codebase to get me started?

 From the code I've looked at, it appears that among other things 
they're using the splice(2) / vmsplice(2) system calls to effect 
"copying" without actually copying data. If I understand the semantics 
correctly, these calls are basically shuffling pointers around to avoid 
unnecessary memcpy(3) / mmap(2) calls. I've even seen a "zero-copy" 
version of sendfile(2) that essentially wraps it around a call to splice(2).

I may be able to hack something together based on the current zero-copy 
TX implementation, but as I'm still wrapping my head around several of 
the concepts I just described, it may be awhile before I can produce 
anything useful. I have quite a bit of experience developing for Linux 
in C, but this would be my first attempt at writing kernel/device-driver 
code.

quoted

The Release Notes for RHEL 6.2 (originally published on 12/06/2011)
also specifically mention macvtap/vhost zero-copy capabilities as
being included as a Technology Preview:

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/6.2_Release_Notes/virtualization.html

I think this means TX.

It does. I meant to clarify that point in my original email... yes, only 
TX zero-copy is currently implemented, and it is still marked as 
"experimental". Outside of the custom solutions like PF_RING that I 
mentioned, I don't know that I've seen zero-copy for RX.

-- Robert Vineyard

On 08/12/2012 11:45 PM, Robert Vineyard wrote:

On 08/12/2012 05:37 AM, Michael S. Tsirkin wrote:

quoted

Contributions are welcome.

That's too bad... do you know of anyone else (at Intel or otherwise) who
might be familiar enough with the existing codebase to get me started?

I suppose I should have done my homework... *ahem* time for me to finish 
RTFM on git...

I'll come back after digging through more kernel code :-)

-- Robert

Last patchset version was this:
[PATCH v16 00/17] Provide a zero-copy method on KVM virtio-net

IIRC the main issue was the need to integrate the patchset with
macvtap (as opposed to using a separate device).

On Mon, Aug 13, 2012 at 09:24:54AM +0800, Peter Huang(Peng) wrote:

Hi, Michael

IIt will be usefull if we can implement rx zero-copy, could you give
 me some help if you know some technical details or some
references on the internet?

I am wondering may be I can take a deep look on it first then decide
if I can take it over or not.

Thanks a lot.

On 2012/8/12 17:37, Michael S. Tsirkin wrote:

quoted

On Sat, Aug 11, 2012 at 08:42:44PM -0400, Robert Vineyard wrote:

quoted

(adding Xin Xiaohui to the conversation for comment)

According to the NetworkingTodo page on the KVM wiki, zero-copy RX
for macvtap is in fact on the roadmap, assigned to Xin:

http://www.linux-kvm.org/page/NetworkingTodo

AFAIK Xin left Intel and is not working on it.
Contributions are welcome.