Gregory Haskins [off-list ref] writes:
What might be useful is if you could expand a bit more on what the high level
use cases for this.
Questions that come to mind and that would be good to answer:
This seems to be aimed at having multiple VMs talk
to each other, but not talk to the rest of the world, correct?
Is that a common use case?
Wouldn't they typically have a default route anyways and be able to talk to each
other this way?
And why can't any such isolation be done with standard firewalling? (it's known that
current iptables has some scalability issues, but there's work going on right
now to fix that).
What would be the use cases for non networking devices?
How would the interfaces to the user look like?
-Andi
--
ak@linux.intel.com -- Speaking for myself only.
Gregory Haskins [off-list ref] writes:
What might be useful is if you could expand a bit more on what the high level
use cases for this.
Questions that come to mind and that would be good to answer:
This seems to be aimed at having multiple VMs talk
to each other, but not talk to the rest of the world, correct?
Is that a common use case?
Actually we didn't design specifically for either type of environment.
I think it would, in fact, be well suited to either type of
communication model, even concurrently (e.g. an intra-vm ipc channel
resource could live right on the same bus as a virtio-net and a
virtio-disk resource)
Wouldn't they typically have a default route anyways and be able to talk to each
other this way?
And why can't any such isolation be done with standard firewalling? (it's known that
current iptables has some scalability issues, but there's work going on right
now to fix that).
vbus itself, and even some of the higher level constructs we apply on
top of it (like venet) are at a different scope than I think what you
are getting at above. Yes, I suppose you could create a private network
using the existing virtio-net + iptables. But you could also do the
same using virtio-net and a private bridge devices as well. That is not
what we are trying to address.
What we *are* trying to address is making an easy way to declare virtual
resources directly in the kernel so that they can be accessed more
efficiently. Contrast that to the way its done today, where the models
live in, say, qemu userspace.
So instead of having
guest->host->qemu::virtio-net->tap->[iptables|bridge], you simply have
guest->host->[iptables|bridge]. How you make your private network (if
that is what you want to do) is orthogonal...its the path to get there
that we changed.
What would be the use cases for non networking devices?
How would the interfaces to the user look like?
I am not sure if you are asking about the guests perspective or the
host-administators perspective.
First now lets look at the low-level device interface from the guests
perspective. We can cover the admin perspective in a separate doc, if
need be.
Each device in vbus supports two basic verbs: CALL, and SHM
int (*call)(struct vbus_device_proxy *dev, u32 func,
void *data, size_t len, int flags);
int (*shm)(struct vbus_device_proxy *dev, int id, int prio,
void *ptr, size_t len,
struct shm_signal_desc *sigdesc, struct shm_signal **signal,
int flags);
CALL provides a synchronous method for invoking some verb on the device
(defined by "func") with some arbitrary data. The namespace for "func"
is part of the ABI for the device in question. It is analogous to an
ioctl, with the primary difference being that its remotable (it invokes
from the guest driver across to the host device).
SHM provides a way to register shared-memory with the device which can
be used for asynchronous communication. The memory is always owned by
the "north" (the guest), while the "south" (the host) simply maps it
into its address space. You can optionally establish a shm_signal
object on this memory for signaling in either direction, and I
anticipate most shm regions will use this feature. Each shm region has
an "id" namespace, which like the "func" namespace from the CALL method
is completely owned by the device ABI. For example, we have might have
id's of "RX-RING" and "TX-RING", etc.
From there, we can (hopefully) build an arbitrary type of IO service to
map on top. So for instance, for venet-tap, we have CALL verbs for
things like MACQUERY, and LINKUP, and we have SHM ids for RX-QUEUE and
TX-QUEUE. We can write a driver that speaks this ABI on the bottom
edge, and presents a normal netif interface on the top edge. So the
actual consumption of these resources can look just like another other
resource of a similar type.
-Greg
On Wed, Apr 01, 2009 at 08:03:49AM -0400, Gregory Haskins wrote:
Andi Kleen wrote:
quoted
Gregory Haskins [off-list ref] writes:
What might be useful is if you could expand a bit more on what the high level
use cases for this.
Questions that come to mind and that would be good to answer:
This seems to be aimed at having multiple VMs talk
to each other, but not talk to the rest of the world, correct?
Is that a common use case?
Actually we didn't design specifically for either type of environment.
But surely you must have some specific use case in mind? Something
that it does better than the various methods that are available
today. Or rather there must be some problem you're trying
to solve. I'm just not sure what that problem exactly is.
What we *are* trying to address is making an easy way to declare virtual
resources directly in the kernel so that they can be accessed more
efficiently. Contrast that to the way its done today, where the models
live in, say, qemu userspace.
So instead of having
guest->host->qemu::virtio-net->tap->[iptables|bridge], you simply have
guest->host->[iptables|bridge]. How you make your private network (if
So is the goal more performance or simplicity or what?
quoted
What would be the use cases for non networking devices?
How would the interfaces to the user look like?
I am not sure if you are asking about the guests perspective or the
host-administators perspective.
I was wondering about the host-administrators perspective.
-Andi
--
ak@linux.intel.com -- Speaking for myself only.
On Wed, Apr 01, 2009 at 08:03:49AM -0400, Gregory Haskins wrote:
quoted
Andi Kleen wrote:
quoted
Gregory Haskins [off-list ref] writes:
What might be useful is if you could expand a bit more on what the high level
use cases for this.
Questions that come to mind and that would be good to answer:
This seems to be aimed at having multiple VMs talk
to each other, but not talk to the rest of the world, correct?
Is that a common use case?
Actually we didn't design specifically for either type of environment.
But surely you must have some specific use case in mind? Something
that it does better than the various methods that are available
today. Or rather there must be some problem you're trying
to solve. I'm just not sure what that problem exactly is.
Performance. We are trying to create a high performance IO infrastructure.
Ideally we would like to see things like virtual-machines have
bare-metal performance (or as close as possible) using just pure
software on commodity hardware. The data I provided shows that
something like KVM with virtio-net does a good job on throughput even on
10GE, but the latency is several orders of magnitude slower than
bare-metal. We are addressing this issue and others like it that are a
result of the current design of out-of-kernel emulation.
quoted
What we *are* trying to address is making an easy way to declare virtual
resources directly in the kernel so that they can be accessed more
efficiently. Contrast that to the way its done today, where the models
live in, say, qemu userspace.
So instead of having
guest->host->qemu::virtio-net->tap->[iptables|bridge], you simply have
guest->host->[iptables|bridge]. How you make your private network (if
So is the goal more performance or simplicity or what?
(Answered above)
quoted
quoted
What would be the use cases for non networking devices?
How would the interfaces to the user look like?
I am not sure if you are asking about the guests perspective or the
host-administators perspective.
I was wondering about the host-administrators perspective.
Ah, ok. Sorry about that. It was probably good to document that other
thing anyway, so no harm.
So about the host-administrator interface. The whole thing is driven by
configfs, and the basics are already covered in the documentation in
patch 2, so I wont repeat it here. Here is a reference to the file for
everyone's convenience:
http://git.kernel.org/?p=linux/kernel/git/ghaskins/vbus/linux-2.6.git;a=blob;f=Documentation/vbus.txt;h=e8a05dafaca2899d37bd4314fb0c7529c167ee0f;hb=f43949f7c340bf667e68af6e6a29552e62f59033
So a sufficiently privileged user can instantiate a new bus (e.g.
container) and devices on that bus via configfs operations. The types
of devices available to instantiate are dictated by whatever vbus-device
modules you have loaded into your particular kernel. The loaded modules
available are enumerated under /sys/vbus/deviceclass.
Now presumably the administrator knows what a particular module is and
how to configure it before instantiating it. Once they instantiate it,
it will present an interface in sysfs with a set of attributes. For
example, an instantiated venet-tap looks like this:
ghaskins@test:~> tree /sys/vbus/devices
/sys/vbus/devices
`-- foo
|-- class -> ../../deviceclass/venet-tap
|-- client_mac
|-- enabled
|-- host_mac
|-- ifname
`-- interfaces
`-- 0 -> ../../../instances/bar/devices/0
Some of these attributes, like "class" and "interfaces" are default
attributes that are filled in by the infrastructure. Other attributes,
like "client_mac" and "enabled" are properties defined by the venet-tap
module itself. So the administrator can then set these attributes as
desired to manipulate the configuration of the instance of the device,
on a per device basis.
So now imagine we have some kind of disk-io vbus device that is designed
to act kind of like a file-loopback device. It might define an
attribute allowing you to specify the path to the file/block-dev that
you want it to export.
(Warning: completely fictitious "tree" output to follow ;)
ghaskins@test:~> tree /sys/vbus/devices
/sys/vbus/devices
`-- foo
|-- class -> ../../deviceclass/vdisk
|-- src_path
`-- interfaces
`-- 0 -> ../../../instances/bar/devices/0
So the admin would instantiate this "vdisk" device and do:
'echo /path/to/my/exported/disk.dat > /sys/vbus/devices/foo/src_path'
To point the device to the file on the host that it wants to present as
a vdisk. Any guest that has access to the particular bus that contains
this device would then see it as a standard "vdisk" ABI device (as if
there where such a thing, yet) and could talk to it using a vdisk
specific driver.
A property of a vbus is that it is inherited by children. Today, I do
not have direct support in qemu for creating/configuring vbus devices.
Instead what I do is I set up the vbus and devices from bash, and then
launch qemu-kvm so it inherits the bus. Someday (soon, unless you guys
start telling me this whole idea is rubbish ;) I will add support so you
could do things like "-net nic,model=venet" and that would trigger qemu
to go out and create the container/device on its own. TBD.
I hope this helps to clarify!
-Greg
On Wed, Apr 01, 2009 at 08:03:49AM -0400, Gregory Haskins wrote:
quoted
Andi Kleen wrote:
quoted
Gregory Haskins [off-list ref] writes:
What might be useful is if you could expand a bit more on what the high level
use cases for this.
Questions that come to mind and that would be good to answer:
This seems to be aimed at having multiple VMs talk
to each other, but not talk to the rest of the world, correct?
Is that a common use case?
Actually we didn't design specifically for either type of environment.
But surely you must have some specific use case in mind? Something
that it does better than the various methods that are available
today. Or rather there must be some problem you're trying
to solve. I'm just not sure what that problem exactly is.
Performance. We are trying to create a high performance IO infrastructure.
Actually, I should also state that I am interested in enabling some new
kinds of features based on having in-kernel devices like this. For
instance (and this is still very theoretical and half-baked), I would
like to try to support RT guests.
[adding linux-rt-users]
I think one of the things that we need in order to do that is being able
to convey vcpu priority state information to the host in an efficient
way. I was thinking that a shared-page per vcpu could have something
like "current" and "theshold" priorties. The guest modifies "current"
while the host modifies "threshold". The guest would be allowed to
increase its "current" priority without a hypercall (after all, if its
already running presumably it is already of sufficient priority that the
scheduler). But if the guest wants to drop below "threshold", it needs
to hypercall the host to give it an opportunity to schedule() a new task
(vcpu or not).
The host, on the other hand, could apply a mapping so that the guests
priority of RT1-RT99 might map to RT20-RT30 on the host, or something
like that. We would have to take other considerations as well, such as
implicit boosting on IRQ injection (e.g. the guest could be in HLT/IDLE
when an interrupt is injected...but by virtue of injecting that
interrupt we may need to boost it to (guest-relative) RT50).
Like I said, this is all half-baked right now. My primary focus is
improving performance, but I did try to lay the groundwork for taking
things in new directions too..rt being an example.
Hope that helps!
-Greg
On Wed, Apr 01, 2009 at 10:19:49AM -0400, Gregory Haskins wrote:
quoted
quoted
But surely you must have some specific use case in mind? Something
that it does better than the various methods that are available
today. Or rather there must be some problem you're trying
to solve. I'm just not sure what that problem exactly is.
Performance. We are trying to create a high performance IO infrastructure.
Ok. So the goal is to bypass user space qemu completely for better
performance. Can you please put this into the initial patch
description?
So the administrator can then set these attributes as
desired to manipulate the configuration of the instance of the device,
on a per device basis.
How would the guest learn of any changes in there?
I think the interesting part would be how e.g. a vnet device
would be connected to the outside interfaces.
So the admin would instantiate this "vdisk" device and do:
'echo /path/to/my/exported/disk.dat > /sys/vbus/devices/foo/src_path'
So it would act like a loop device? Would you reuse the loop device
or write something new?
How about VFS mount name spaces?
-Andi
--
ak@linux.intel.com -- Speaking for myself only.
From: Anthony Liguori <hidden> Date: 2009-04-01 18:46:20
Andi Kleen wrote:
On Wed, Apr 01, 2009 at 10:19:49AM -0400, Gregory Haskins wrote:
quoted
quoted
quoted
But surely you must have some specific use case in mind? Something
that it does better than the various methods that are available
today. Or rather there must be some problem you're trying
to solve. I'm just not sure what that problem exactly is.
Performance. We are trying to create a high performance IO infrastructure.
Ok. So the goal is to bypass user space qemu completely for better
performance. Can you please put this into the initial patch
description?
FWIW, there's nothing that prevents in-kernel back ends with virtio so
vbus certainly isn't required for in-kernel backends.
That said, I don't think we're bound today by the fact that we're in
userspace. Rather we're bound by the interfaces we have between the
host kernel and userspace to generate IO. I'd rather fix those
interfaces than put more stuff in the kernel.
Regards,
Anthony Liguori
On Wed, Apr 01, 2009 at 10:19:49AM -0400, Gregory Haskins wrote:
quoted
quoted
quoted
But surely you must have some specific use case in mind? Something
that it does better than the various methods that are available
today. Or rather there must be some problem you're trying
to solve. I'm just not sure what that problem exactly is.
Performance. We are trying to create a high performance IO infrastructure.
Ok. So the goal is to bypass user space qemu completely for better
performance. Can you please put this into the initial patch
description?
Yes, good point. I will be sure to be more explicit in the next rev.
quoted
So the administrator can then set these attributes as
desired to manipulate the configuration of the instance of the device,
on a per device basis.
How would the guest learn of any changes in there?
The only events explicitly supported by the infrastructure of this
nature would be device-add and device-remove. So when an admin adds or
removes a device to a bus, the guest would see driver::probe() and
driver::remove() callbacks, respectively. All other events are left (by
design) to be handled by the device ABI itself, presumably over the
provided shm infrastructure.
So for instance, I have on my todo list to add a third shm-ring for
events in the venet ABI. One of the event-types I would like to
support is LINK_UP and LINK_DOWN. These events would be coupled to the
administrative manipulation of the "enabled" attribute in sysfs. Other
event-types could be added as needed/appropriate.
I decided to do it this way because I felt it didn't make sense for me
to expose the attributes directly, since they are often back-end
specific anyway. Therefore I leave it to the device-specific ABI which
has all the necessary tools for async events built in.
I think the interesting part would be how e.g. a vnet device
would be connected to the outside interfaces.
Ah, good question. This ties into the statement I made earlier about
how presumably the administrative agent would know what a module is and
how it works. As part of this, they would also handle any kind of
additional work, such as wiring the backend up. Here is a script that I
use for testing that demonstrates this:
------------------
#!/bin/bash
set -e
modprobe venet-tap
mount -t configfs configfs /config
bridge=vbus-br0
brctl addbr $bridge
brctl setfd $bridge 0
ifconfig $bridge up
createtap()
{
mkdir /config/vbus/devices/$1-dev
echo venet-tap > /config/vbus/devices/$1-dev/type
mkdir /config/vbus/instances/$1-bus
ln -s /config/vbus/devices/$1-dev /config/vbus/instances/$1-bus
echo 1 > /sys/vbus/devices/$1-dev/enabled
ifname=$(cat /sys/vbus/devices/$1-dev/ifname)
ifconfig $ifname up
brctl addif $bridge $ifname
}
createtap client
createtap server
--------------------
This script creates two buses ("client-bus" and "server-bus"),
instantiates a single venet-tap on each of them, and then "wires" them
together with a private bridge instance called "vbus-br0". To complete
the picture here, you would want to launch two kvms, one of each of the
client-bus/server-bus instances. You can do this via /proc/$pid/vbus. E.g.
# (echo client-bus > /proc/self/vbus; qemu-kvm -hda client.img....)
# (echo server-bus > /proc/self/vbus; qemu-kvm -hda server.img....)
(And as noted, someday qemu will be able to do all the setup that the
script did, natively. It would wire whatever tap it created to an
existing bridge with qemu-ifup, just like we do for tun-taps today)
One of the key details is where I do "ifname=$(cat
/sys/vbus/devices/$1-dev/ifname)". The "ifname" attribute of the
venet-tap is a read-only attribute that reports back the netif interface
name that was returned when the device did a register_netdev() (e.g.
"eth3"). This register_netdev() operation occurs as a result of echoing
the "1" into the "enabled" attribute. Deferring the registration until
the admin explicitly does an "enable" gives the admin a chance to change
the MAC address of the virtual-adapter before it is registered (note:
the current code doesnt support rw on the mac attributes yet..i need a
parser first).
quoted
So the admin would instantiate this "vdisk" device and do:
'echo /path/to/my/exported/disk.dat > /sys/vbus/devices/foo/src_path'
So it would act like a loop device? Would you reuse the loop device
or write something new?
Well, keeping in mind that I haven't even looked at writing a block
device for this infrastructure yet....my blanket statement would be
"lets reuse as much as possible" ;) If the existing loop infrastructure
would work here, great!
How about VFS mount name spaces?
Yeah, ultimately I would love to be able to support a fairly wide range
of the normal userspace/kernel ABI through this mechanism. In fact, one
of my original design goals was to somehow expose the syscall ABI
directly via some kind of syscall proxy device on the bus. I have since
backed away from that idea once I started thinking about things some
more and realized that a significant number of system calls are really
inappropriate for a guest type environment due to their ability to
block. We really dont want a vcpu to block.....however, the AIO type
system calls on the other hand, have much more promise. ;) TBD.
For right now I am focused more on the explicit virtual-device type
transport (disk, net, etc). But in theory we should be able to express
a fairly broad range of services in terms of the call()/shm() interfaces.
-Greg
From: Chris Wright <hidden> Date: 2009-04-01 20:41:20
* Anthony Liguori (anthony@codemonkey.ws) wrote:
Andi Kleen wrote:
quoted
On Wed, Apr 01, 2009 at 10:19:49AM -0400, Gregory Haskins wrote:
quoted
Performance. We are trying to create a high performance IO infrastructure.
Ok. So the goal is to bypass user space qemu completely for better
performance. Can you please put this into the initial patch
description?
FWIW, there's nothing that prevents in-kernel back ends with virtio so
vbus certainly isn't required for in-kernel backends.
Indeed.
That said, I don't think we're bound today by the fact that we're in
userspace. Rather we're bound by the interfaces we have between the
host kernel and userspace to generate IO. I'd rather fix those
interfaces than put more stuff in the kernel.
And more stuff in the kernel can come at the potential cost of weakening
protection/isolation.
On Wed, Apr 01, 2009 at 10:19:49AM -0400, Gregory Haskins wrote:
quoted
quoted
quoted
But surely you must have some specific use case in mind? Something
that it does better than the various methods that are available
today. Or rather there must be some problem you're trying
to solve. I'm just not sure what that problem exactly is.
Performance. We are trying to create a high performance IO
infrastructure.
Ok. So the goal is to bypass user space qemu completely for better
performance. Can you please put this into the initial patch
description?
FWIW, there's nothing that prevents in-kernel back ends with virtio so
vbus certainly isn't required for in-kernel backends.
I think there is a slight disconnect here. This is *exactly* what I am
trying to do. You can of course do this many ways, and I am not denying
it could be done a different way than the path I have chosen. One
extreme would be to just slam a virtio-net specific chunk of code
directly into kvm on the host. Another extreme would be to build a
generic framework into Linux for declaring arbitrary IO types,
integrating it with kvm (as well as other environments such as lguest,
userspace, etc), and building a virtio-net model on top of that.
So in case it is not obvious at this point, I have gone with the latter
approach. I wanted to make sure it wasn't kvm specific or something
like pci specific so it had the broadest applicability to a range of
environments. So that is why the design is the way it is. I understand
that this approach is technically "harder/more-complex" than the "slam
virtio-net into kvm" approach, but I've already done that work. All we
need to do now is agree on the details ;)
That said, I don't think we're bound today by the fact that we're in
userspace.
You will *always* be bound by the fact that you are in userspace. Its
purely a question of "how much" and "does anyone care". Right now,
the anwer is "a lot (roughly 45x slower)" and "at least Greg's customers
do". I have no doubt that this can and will change/improve in the
future. But it will always be true that no matter how much userspace
improves, the kernel based solution will always be faster. Its simple
physics. I'm cutting out the middleman to ultimately reach the same
destination as the userspace path, so userspace can never be equal.
I agree that the "does anyone care" part of the equation will approach
zero as the latency difference shrinks across some threshold (probably
the single microsecond range), but I will believe that is even possible
when I see it ;)
Regards,
-Greg
And more stuff in the kernel can come at the potential cost of weakening
protection/isolation.
Note that the design of vbus should prevent any weakening...though if
you see a hole, please point it out.
(On that front, note that I still have some hardening to do, such as not
calling BUG_ON() in venet-tap if the ring is in a funk, etc)
Regards,
-Greg
Note that the design of vbus should prevent any weakening
Could you elaborate?
Absolutely.
So you said that something in the kernel could weaken the
protection/isolation. And I fully agree that whatever we do here has to
be done carefully...more carefully than a userspace derived counterpart,
naturally.
So to address this, I put in various mechanisms to (hopefully? :) ensure
we can still maintain proper isolation, as well as protect the host,
other guests, and other applications from corruption. Here are some of
the highlights:
*) As I mentioned, a "vbus" is a form of a kernel-resource-container.
It is designed so that the view of a vbus is a unique namespace of
device-ids. Each bus has its own individual namespace that consist
solely of the devices that have been placed on that bus. The only way
to create a bus, and/or create a device on a bus, is via the
administrative interface on the host.
*) A task can only associate with, at most, one vbus at a time. This
means that a task can only see the device-id namespace of the devices on
its associated bus and thats it. This is enforced by the host kernel by
placing a reference to the associated vbus on the task-struct itself.
Again, the only way to modify this association is via a host based
administrative operation. Note that multiple tasks can associate to the
same vbus, which would commonly be used by all threads in an app, or all
vcpus in a guest, etc.
*) the asynchronous nature of the shm/ring interfaces implies we have
the potential for asynchronous faults. E.g. "crap" in the ring might
not be discovered at the EIP of the guest vcpu when it actually inserts
the crap, but rather later when the host side tries to update the ring.
A naive implementation would have the host do a BUG_ON() when it
discovers the discrepancy (note that I still have a few of these to fix
in the venet-tap code). Instead, what should happen is that we utilize
an asynchronous fault mechanism that allows the guest to always be the
one punished (via something like a machine-check for guests, or SIGABRT
for userspace, etc)
*) "south-to-north path signaling robustness". Because vbus supports a
variety of different environments, I call guest/userspace "north', and
the host/kernel "south". When the north wants to communicate with the
kernel, its perfectly ok to stall the north indefinitely if the south is
not ready. However, it is not really ok to stall the south when
communicating with the north because this is an attack vector. E.g. a
malicous/broken guest could just stop servicing its ring to cause
threads in the host to jam up. This is bad. :) So what we do is we
design all south-to-north signaling paths to be robust against
stalling. What they do instead is manage backpressure a little bit more
intelligently than simply blocking like they might in the guest. For
instance, in venet-tap, a "transmit" from netif that has to be injected
in the south-to-north ring when it is full will result in a
netif_stop_queue(). etc.
I cant think of more examples right now, but I will update this list
if/when I come up with more. I hope that satisfactorily answered your
question, though!
Regards,
-Greg
On Wed, Apr 01, 2009 at 04:29:57PM -0400, Gregory Haskins wrote:
quoted
description?
Yes, good point. I will be sure to be more explicit in the next rev.
quoted
quoted
So the administrator can then set these attributes as
desired to manipulate the configuration of the instance of the device,
on a per device basis.
How would the guest learn of any changes in there?
The only events explicitly supported by the infrastructure of this
nature would be device-add and device-remove. So when an admin adds or
removes a device to a bus, the guest would see driver::probe() and
driver::remove() callbacks, respectively. All other events are left (by
design) to be handled by the device ABI itself, presumably over the
provided shm infrastructure.
Ok so you rely on a transaction model where everything is set up
before it is somehow comitted to the guest? I hope that is made
explicit in the interface somehow.
This script creates two buses ("client-bus" and "server-bus"),
instantiates a single venet-tap on each of them, and then "wires" them
together with a private bridge instance called "vbus-br0". To complete
the picture here, you would want to launch two kvms, one of each of the
client-bus/server-bus instances. You can do this via /proc/$pid/vbus. E.g.
# (echo client-bus > /proc/self/vbus; qemu-kvm -hda client.img....)
# (echo server-bus > /proc/self/vbus; qemu-kvm -hda server.img....)
(And as noted, someday qemu will be able to do all the setup that the
script did, natively. It would wire whatever tap it created to an
existing bridge with qemu-ifup, just like we do for tun-taps today)
The usual problem with that is permissions. Just making qemu-ifup suid
it not very nice. It would be good if any new design addressed this.
the current code doesnt support rw on the mac attributes yet..i need a
parser first).
parser in kernel space always sounds scary to me.
Yeah, ultimately I would love to be able to support a fairly wide range
of the normal userspace/kernel ABI through this mechanism. In fact, one
of my original design goals was to somehow expose the syscall ABI
directly via some kind of syscall proxy device on the bus. I have since
That sounds really scary for security.
backed away from that idea once I started thinking about things some
more and realized that a significant number of system calls are really
inappropriate for a guest type environment due to their ability to
block. We really dont want a vcpu to block.....however, the AIO type
Not only because of blocking, but also because of security issues.
After all one of the usual reasons to run a guest is security isolation.
In general the more powerful the guest API the more risky it is, so some
self moderation is probably a good thing.
-Andi
--
ak@linux.intel.com -- Speaking for myself only.
On Wed, Apr 01, 2009 at 04:29:57PM -0400, Gregory Haskins wrote:
quoted
quoted
description?
Yes, good point. I will be sure to be more explicit in the next rev.
quoted
quoted
So the administrator can then set these attributes as
desired to manipulate the configuration of the instance of the device,
on a per device basis.
How would the guest learn of any changes in there?
The only events explicitly supported by the infrastructure of this
nature would be device-add and device-remove. So when an admin adds or
removes a device to a bus, the guest would see driver::probe() and
driver::remove() callbacks, respectively. All other events are left (by
design) to be handled by the device ABI itself, presumably over the
provided shm infrastructure.
Ok so you rely on a transaction model where everything is set up
before it is somehow comitted to the guest? I hope that is made
explicit in the interface somehow.
Well, its not an explicit transaction model, but I guess you could think
of it that way.
Generally you set the device up before you launch the guest. By the
time the guest loads and tries to scan the bus for the initial
discovery, all the devices would be ready to go.
This does bring up the question of hotswap. Today we fully support
hotswap in and out, but leaving this "enabled" transaction to the
individual device means that the device-id would be visible in the bus
namespace before the device may want to actually communicate. Hmmm
Perhaps I need to build this in as a more explicit "enabled"
feature...and the guest will not see the driver::probe() until this happens.
quoted
This script creates two buses ("client-bus" and "server-bus"),
instantiates a single venet-tap on each of them, and then "wires" them
together with a private bridge instance called "vbus-br0". To complete
the picture here, you would want to launch two kvms, one of each of the
client-bus/server-bus instances. You can do this via /proc/$pid/vbus. E.g.
# (echo client-bus > /proc/self/vbus; qemu-kvm -hda client.img....)
# (echo server-bus > /proc/self/vbus; qemu-kvm -hda server.img....)
(And as noted, someday qemu will be able to do all the setup that the
script did, natively. It would wire whatever tap it created to an
existing bridge with qemu-ifup, just like we do for tun-taps today)
The usual problem with that is permissions. Just making qemu-ifup suid
it not very nice. It would be good if any new design addressed this.
Well, its kind of out of my control. venet-tap ultimately creates a
simple netif interface which we must do something with. Once its
created, "wiring" it up to something like a linux-bridge is no different
than something like a tun-tap, so the qemu-ifup requirement doesn't change.
The one thing I can think of is it would be possible to build a
"venet-switch" module, and this could be done without using brctl or
qemu-ifup...but then I would lose all the benefits of re-using that
infrastructure. I do not recommend we actually do this, but it would
technically be a way to address your concern.
quoted
the current code doesnt support rw on the mac attributes yet..i need a
parser first).
parser in kernel space always sounds scary to me.
Heh..why do you think I keep procrastinating ;)
quoted
Yeah, ultimately I would love to be able to support a fairly wide range
of the normal userspace/kernel ABI through this mechanism. In fact, one
of my original design goals was to somehow expose the syscall ABI
directly via some kind of syscall proxy device on the bus. I have since
That sounds really scary for security.
quoted
backed away from that idea once I started thinking about things some
more and realized that a significant number of system calls are really
inappropriate for a guest type environment due to their ability to
block. We really dont want a vcpu to block.....however, the AIO type
Not only because of blocking, but also because of security issues.
After all one of the usual reasons to run a guest is security isolation.
Oh yeah, totally agreed. Not that I am advocating this, because I have
abandoned the idea. But back when I was thinking of this, I would have
addressed the security with the vbus and syscall-proxy-device objects
themselves. E.g. if you dont instantiate a syscall-proxy-device on the
bus, the guest wouldnt have access to syscalls at all. And you could
put filters into the module to limit what syscalls were allowed, which
UID to make the guest appear as, etc.
In general the more powerful the guest API the more risky it is, so some
self moderation is probably a good thing.
From: Anthony Liguori <hidden> Date: 2009-04-02 00:30:14
Gregory Haskins wrote:
Anthony Liguori wrote:
I think there is a slight disconnect here. This is *exactly* what I am
trying to do.
If it were exactly what you were trying to do, you would have posted a
virtio-net in-kernel backend implementation instead of a whole new
paravirtual IO framework ;-)
quoted
That said, I don't think we're bound today by the fact that we're in
userspace.
You will *always* be bound by the fact that you are in userspace.
Again, let's talk numbers. A heavy-weight exit is 1us slower than a
light weight exit. Ideally, you're taking < 1 exit per packet because
you're batching notifications. If you're ping latency on bare metal
compared to vbus is 39us to 65us, then all other things being equally,
the cost imposed by doing what your doing in userspace would make the
latency be 66us taking your latency from 166% of native to 169% of
native. That's not a huge difference and I'm sure you'll agree there
are a lot of opportunities to improve that even further.
And you didn't mention whether your latency tests are based on ping or
something more sophisticated as ping will be a pathological case that
doesn't allow any notification batching.
I agree that the "does anyone care" part of the equation will approach
zero as the latency difference shrinks across some threshold (probably
the single microsecond range), but I will believe that is even possible
when I see it ;)
Note the other hat we have to where is not just virtualization developer
but Linux developer. If there are bad userspace interfaces for IO that
impose artificial restrictions, then we need to identify those and fix them.
Regards,
Anthony Liguori
Anthony Liguori wrote:
I think there is a slight disconnect here. This is *exactly* what
I am
trying to do.
If it were exactly what you were trying to do, you would have posted a
virtio-net in-kernel backend implementation instead of a whole new
paravirtual IO framework ;-)
semantics, semantics ;)
but ok, fair enough.
quoted
quoted
That said, I don't think we're bound today by the fact that we're in
userspace.
You will *always* be bound by the fact that you are in userspace.
Again, let's talk numbers. A heavy-weight exit is 1us slower than a
light weight exit. Ideally, you're taking < 1 exit per packet because
you're batching notifications. If you're ping latency on bare metal
compared to vbus is 39us to 65us, then all other things being equally,
the cost imposed by doing what your doing in userspace would make the
latency be 66us taking your latency from 166% of native to 169% of
native. That's not a huge difference and I'm sure you'll agree there
are a lot of opportunities to improve that even further.
Ok, so lets see it happen. Consider the gauntlet thrown :) Your
challenge, should you chose to accept it, is to take todays 4000us and
hit a 65us latency target while maintaining 10GE line-rate (at least
1500 mtu line-rate).
I personally don't want to even stop at 65. I want to hit that 36us!
In case you think that is crazy, my first prototype of venet was hitting
about 140us, and I shaved 10us here, 10us there, eventually getting down
to the 65us we have today. The low hanging fruit is all but harvested
at this point, but I am not done searching for additional sources of
latency. I just needed to take a breather to get the code out there for
review. :)
And you didn't mention whether your latency tests are based on ping or
something more sophisticated
Well, the numbers posted were actually from netperf -t UDP_RR. This
generates a pps from a continuous (but non-bursted) RTT measurement. So
I invert the pps result of this test to get the average rtt time. I
have also confirmed that ping jives with these results (e.g. virtio-net
results were about 4ms, and venet were about 0.065ms as reported by ping).
as ping will be a pathological case
Ah, but this is not really pathological IMO. There are plenty of
workloads that exhibit request-reply patterns (e.g. RPC), and this is a
direct measurement of the systems ability to support these
efficiently. And even unidirectional flows can be hampered by poor
latency (think PTP clock sync, etc).
Massive throughput with poor latency is like Andrew Tanenbaum's
station-wagon full of backup tapes ;) I think I have proven we can
actually get both with a little creative use of resources.
that doesn't allow any notification batching.
Well, if we can take anything away from all this: I think I have
demonstrated that you don't need notification batching to get good
throughput. And batching on the head-end of the queue adds directly to
your latency overhead, so I don't think its a good technique in general
(though I realize that not everyone cares about latency, per se, so
maybe most are satisfied with the status-quo).
quoted
I agree that the "does anyone care" part of the equation will approach
zero as the latency difference shrinks across some threshold (probably
the single microsecond range), but I will believe that is even possible
when I see it ;)
Note the other hat we have to where is not just virtualization
developer but Linux developer. If there are bad userspace interfaces
for IO that impose artificial restrictions, then we need to identify
those and fix them.
Fair enough, and I would love to take that on but alas my
development/debug bandwidth is rather finite these days ;)
-Greg
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2009-04-02 03:10:22
Anthony Liguori [off-list ref] wrote:
That said, I don't think we're bound today by the fact that we're in
userspace. Rather we're bound by the interfaces we have between the
host kernel and userspace to generate IO. I'd rather fix those
interfaces than put more stuff in the kernel.
I'm sorry but I totally disagree with that. By having our IO
infrastructure in user-space we've basically given up the main
advantage of kvm, which is that the physical drivers operate in
the same environment as the hypervisor.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2009-04-02 03:12:25
Chris Wright [off-list ref] wrote:
quoted
That said, I don't think we're bound today by the fact that we're in
userspace. Rather we're bound by the interfaces we have between the
host kernel and userspace to generate IO. I'd rather fix those
interfaces than put more stuff in the kernel.
And more stuff in the kernel can come at the potential cost of weakening
protection/isolation.
I cant think of more examples right now, but I will update this list
if/when I come up with more. I hope that satisfactorily answered your
question, though!
Yes, that helps, thanks.
There's still the simple issue of guest/host interface widening w/ kernel
resident backend where a plain ol' bug (good that you thought about the
isolation) can take out more than single guest. Always the balance... ;-)
From: Avi Kivity <hidden> Date: 2009-04-02 06:47:23
Herbert Xu wrote:
Anthony Liguori [off-list ref] wrote:
quoted
That said, I don't think we're bound today by the fact that we're in
userspace. Rather we're bound by the interfaces we have between the
host kernel and userspace to generate IO. I'd rather fix those
interfaces than put more stuff in the kernel.
I'm sorry but I totally disagree with that. By having our IO
infrastructure in user-space we've basically given up the main
advantage of kvm, which is that the physical drivers operate in
the same environment as the hypervisor.
I don't understand this. If we had good interfaces, all that userspace
would do is translate guest physical addresses to host physical
addresses, and translate the guest->host protocol to host API calls. I
don't see anything there that benefits from being in the kernel.
Can you elaborate?
--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
From: Avi Kivity <hidden> Date: 2009-04-02 06:51:47
Gregory Haskins wrote:
I think there is a slight disconnect here. This is *exactly* what I am
trying to do. You can of course do this many ways, and I am not denying
it could be done a different way than the path I have chosen. One
extreme would be to just slam a virtio-net specific chunk of code
directly into kvm on the host. Another extreme would be to build a
generic framework into Linux for declaring arbitrary IO types,
integrating it with kvm (as well as other environments such as lguest,
userspace, etc), and building a virtio-net model on top of that.
So in case it is not obvious at this point, I have gone with the latter
approach. I wanted to make sure it wasn't kvm specific or something
like pci specific so it had the broadest applicability to a range of
environments. So that is why the design is the way it is. I understand
that this approach is technically "harder/more-complex" than the "slam
virtio-net into kvm" approach, but I've already done that work. All we
need to do now is agree on the details ;)
virtio is already non-kvm-specific (lguest uses it) and non-pci-specific
(s390 uses it).
quoted
That said, I don't think we're bound today by the fact that we're in
userspace.
You will *always* be bound by the fact that you are in userspace. Its
purely a question of "how much" and "does anyone care". Right now,
the anwer is "a lot (roughly 45x slower)" and "at least Greg's customers
do". I have no doubt that this can and will change/improve in the
future. But it will always be true that no matter how much userspace
improves, the kernel based solution will always be faster. Its simple
physics. I'm cutting out the middleman to ultimately reach the same
destination as the userspace path, so userspace can never be equal.
If you have a good exit mitigation scheme you can cut exits by a factor
of 100; so the userspace exit costs are cut by the same factor. If you
have good copyless networking APIs you can cut the cost of copies to
zero (well, to the cost of get_user_pages_fast(), but a kernel solution
needs that too).
--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2009-04-02 08:53:29
Avi Kivity [off-list ref] wrote:
virtio is already non-kvm-specific (lguest uses it) and non-pci-specific
(s390 uses it).
I think Greg's work shows that putting the backend in the kernel
can dramatically reduce the cost of a single guest->host transaction.
I'm sure the same thing would work for virtio too.
If you have a good exit mitigation scheme you can cut exits by a factor
of 100; so the userspace exit costs are cut by the same factor. If you
have good copyless networking APIs you can cut the cost of copies to
zero (well, to the cost of get_user_pages_fast(), but a kernel solution
needs that too).
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2009-04-02 08:54:42
On Thu, Apr 02, 2009 at 09:46:49AM +0300, Avi Kivity wrote:
I don't understand this. If we had good interfaces, all that userspace
would do is translate guest physical addresses to host physical
addresses, and translate the guest->host protocol to host API calls. I
don't see anything there that benefits from being in the kernel.
Can you elaborate?
From: Avi Kivity <hidden> Date: 2009-04-02 09:03:21
Herbert Xu wrote:
Avi Kivity [off-list ref] wrote:
quoted
virtio is already non-kvm-specific (lguest uses it) and non-pci-specific
(s390 uses it).
I think Greg's work shows that putting the backend in the kernel
can dramatically reduce the cost of a single guest->host transaction.
I'm sure the same thing would work for virtio too.
Virtio suffers because we've had no notification of when a packet is
actually submitted. With the notification, the only difference should
be in the cost of a kernel->user switch, which is nowhere nearly as
dramatic.
quoted
If you have a good exit mitigation scheme you can cut exits by a factor
of 100; so the userspace exit costs are cut by the same factor. If you
have good copyless networking APIs you can cut the cost of copies to
zero (well, to the cost of get_user_pages_fast(), but a kernel solution
needs that too).
Given the choice of having to mitigate or not having the problem
in the first place, guess what I would prefer :)
There is no choice. Exiting from the guest to the kernel to userspace
is prohibitively expensive, you can't do that on every packet.
--
error compiling committee.c: too many arguments to function
From: Avi Kivity <hidden> Date: 2009-04-02 09:04:23
Herbert Xu wrote:
On Thu, Apr 02, 2009 at 09:46:49AM +0300, Avi Kivity wrote:
quoted
I don't understand this. If we had good interfaces, all that userspace
would do is translate guest physical addresses to host physical
addresses, and translate the guest->host protocol to host API calls. I
don't see anything there that benefits from being in the kernel.
Can you elaborate?
I think Greg has expressed it clearly enough.
At the end of the day, the numbers speak for themselves. So if
and when there's a user-space version that achieves the same or
better results, then I will change my mind :)
Like Anthony said, the problem is with the kernel->user interfaces. We
won't have a good user space virtio implementation until that is fixed.
--
error compiling committee.c: too many arguments to function
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2009-04-02 09:05:51
On Thu, Apr 02, 2009 at 12:03:32PM +0300, Avi Kivity wrote:
Like Anthony said, the problem is with the kernel->user interfaces. We
won't have a good user space virtio implementation until that is fixed.
If it's just the interface that's bad, then it should be possible
to do a proof-of-concept patch to show that this is the case.
Even if we have to redesign the interface, at least you can then
say that you guys were right all along :)
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2009-04-02 09:17:49
On Thu, Apr 02, 2009 at 12:02:09PM +0300, Avi Kivity wrote:
There is no choice. Exiting from the guest to the kernel to userspace
is prohibitively expensive, you can't do that on every packet.
I was referring to the bit between the kernel and userspace.
In any case, I just looked at the virtio mitigation code again
and I am completely baffled at why we need it. Look at Greg's
code or the netback/netfront notification, why do we need this
completely artificial mitigation when the ring itself provides
a natural way of stemming the flow?
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
From: Avi Kivity <hidden> Date: 2009-04-02 09:27:49
Herbert Xu wrote:
On Thu, Apr 02, 2009 at 12:02:09PM +0300, Avi Kivity wrote:
quoted
There is no choice. Exiting from the guest to the kernel to userspace
is prohibitively expensive, you can't do that on every packet.
I was referring to the bit between the kernel and userspace.
In any case, I just looked at the virtio mitigation code again
and I am completely baffled at why we need it. Look at Greg's
code or the netback/netfront notification, why do we need this
completely artificial mitigation when the ring itself provides
a natural way of stemming the flow?
If the vcpu thread does the transmit, then it will always complete
sending immediately:
guest: push packet, notify qemu
qemu: disable notification
qemu: pop packet
qemu: copy to tap
qemu: ??
At this point, qemu must enable notification again, since we have no
notification from tap that the transmit completed. The only alternative
is the timer.
If we do the transmit through an extra thread, then scheduling latency
buys us some time:
guest: push packet, notify qemu
qemu: disable notification
qemu: schedule iothread
iothread: pop packet
iothread: copy to tap
iothread: check for more packets
iothread: enable notification
If tap told us when the packets were actually transmitted, life would be
wonderful:
guest: push packet, notify qemu
qemu: disable notification
qemu: pop packet
qemu: queue on tap
qemu: return to guest
hardware: churn churn churn
tap: packet is out
iothread: check for more packets
iothread: enable notification
--
error compiling committee.c: too many arguments to function
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2009-04-02 09:33:56
On Thu, Apr 02, 2009 at 05:29:36PM +0800, Herbert Xu wrote:
On Thu, Apr 02, 2009 at 12:27:17PM +0300, Avi Kivity wrote:
quoted
If tap told us when the packets were actually transmitted, life would be
wonderful:
And why do we need this? Because we are in user space!
I'll continue to wait for your patch and numbers :)
And in case you're working on that patch, this might interest
you. Check out the netdev thread titled "TX time stamping".
Now that we assign the tap skb with its own sk, these two scenarios
are pretty much identical.
I also noitced despite davem's threats to revert the patch, it
has now made Linus's tree :)
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
From: Avi Kivity <hidden> Date: 2009-04-02 09:44:27
Herbert Xu wrote:
On Thu, Apr 02, 2009 at 12:38:46PM +0300, Avi Kivity wrote:
quoted
Why does a kernel solution not need to know when a packet is transmitted?
Because you can install your own destructor?
So we're back to "the problem is with the kernel->user interface, not
userspace being cursed into slowness".
--
error compiling committee.c: too many arguments to function
I think there is a slight disconnect here. This is *exactly* what I am
trying to do. You can of course do this many ways, and I am not denying
it could be done a different way than the path I have chosen. One
extreme would be to just slam a virtio-net specific chunk of code
directly into kvm on the host. Another extreme would be to build a
generic framework into Linux for declaring arbitrary IO types,
integrating it with kvm (as well as other environments such as lguest,
userspace, etc), and building a virtio-net model on top of that.
So in case it is not obvious at this point, I have gone with the latter
approach. I wanted to make sure it wasn't kvm specific or something
like pci specific so it had the broadest applicability to a range of
environments. So that is why the design is the way it is. I understand
that this approach is technically "harder/more-complex" than the "slam
virtio-net into kvm" approach, but I've already done that work. All we
need to do now is agree on the details ;)
virtio is already non-kvm-specific (lguest uses it) and
non-pci-specific (s390 uses it).
Ok, then to be more specific, I need it to be more generic than it
already is. For instance, I need it to be able to integrate with
shm_signals. If we can do that without breaking the existing ABI, that
would be great! Last I looked, it was somewhat entwined here so I didnt
try...but I admit that I didnt try that hard since I already had the IOQ
library ready to go.
quoted
quoted
That said, I don't think we're bound today by the fact that we're in
userspace.
You will *always* be bound by the fact that you are in userspace. Its
purely a question of "how much" and "does anyone care". Right now,
the anwer is "a lot (roughly 45x slower)" and "at least Greg's customers
do". I have no doubt that this can and will change/improve in the
future. But it will always be true that no matter how much userspace
improves, the kernel based solution will always be faster. Its simple
physics. I'm cutting out the middleman to ultimately reach the same
destination as the userspace path, so userspace can never be equal.
If you have a good exit mitigation scheme you can cut exits by a
factor of 100; so the userspace exit costs are cut by the same
factor. If you have good copyless networking APIs you can cut the
cost of copies to zero (well, to the cost of get_user_pages_fast(),
but a kernel solution needs that too).
"exit mitigation' schemes are for bandwidth, not latency. For latency
it all comes down to how fast you can signal in both directions. If
someone is going to do a stand-alone request-reply, its generally always
going to be at least one hypercall and one rx-interrupt. So your speed
will be governed by your signal path, not your buffer bandwidth.
What Ive done is shown that you can use techniques other than buffering
the head of the queue to do exit mitigation for bandwidth, while still
maintaining a very short signaling path for latency. And I also argue
that the latter will always be optimal in the kernel, though I know by
which degree is still TBD. Anthony thinks he can make the difference
negligible, and I would love to see it but am skeptical.
-Greg
virtio is already non-kvm-specific (lguest uses it) and
non-pci-specific (s390 uses it).
I think Greg's work shows that putting the backend in the kernel
can dramatically reduce the cost of a single guest->host transaction.
I'm sure the same thing would work for virtio too.
Virtio suffers because we've had no notification of when a packet is
actually submitted. With the notification, the only difference should
be in the cost of a kernel->user switch, which is nowhere nearly as
dramatic.
quoted
quoted
If you have a good exit mitigation scheme you can cut exits by a
factor of 100; so the userspace exit costs are cut by the same
factor. If you have good copyless networking APIs you can cut the
cost of copies to zero (well, to the cost of get_user_pages_fast(),
but a kernel solution needs that too).
Given the choice of having to mitigate or not having the problem
in the first place, guess what I would prefer :)
There is no choice. Exiting from the guest to the kernel to userspace
is prohibitively expensive, you can't do that on every packet.
Now you are making my point ;) This is part of the cost of your
signaling path, and it directly adds to your latency time. You can't
buffer packets here if the guest is only going to send one and wait for
a response and expect that to perform well. And this is precisely what
drove me to look at avoiding going back to userspace in the first place.
-Greg
On Thu, Apr 02, 2009 at 12:27:17PM +0300, Avi Kivity wrote:
quoted
If tap told us when the packets were actually transmitted, life
would be wonderful:
And why do we need this? Because we are in user space!
Why does a kernel solution not need to know when a packet is transmitted?
You do not need to know when the packet is copied (which I currently
do). You only need it for zero-copy (of which I would like to support,
but as I understand it there are problems with the reliability of proper
callback (i.e. skb->destructor).
Its "fire and forget" :)
-Greg
From: Avi Kivity <hidden> Date: 2009-04-02 11:44:25
Gregory Haskins wrote:
quoted
virtio is already non-kvm-specific (lguest uses it) and
non-pci-specific (s390 uses it).
Ok, then to be more specific, I need it to be more generic than it
already is. For instance, I need it to be able to integrate with
shm_signals.
Why?
quoted
If you have a good exit mitigation scheme you can cut exits by a
factor of 100; so the userspace exit costs are cut by the same
factor. If you have good copyless networking APIs you can cut the
cost of copies to zero (well, to the cost of get_user_pages_fast(),
but a kernel solution needs that too).
"exit mitigation' schemes are for bandwidth, not latency. For latency
it all comes down to how fast you can signal in both directions. If
someone is going to do a stand-alone request-reply, its generally always
going to be at least one hypercall and one rx-interrupt. So your speed
will be governed by your signal path, not your buffer bandwidth.
The userspace path is longer by 2 microseconds (for two additional
heavyweight exits) and a few syscalls. I don't think that's worthy of
putting all the code in the kernel.
--
error compiling committee.c: too many arguments to function
From: Avi Kivity <hidden> Date: 2009-04-02 11:48:45
Gregory Haskins wrote:
quoted
There is no choice. Exiting from the guest to the kernel to userspace
is prohibitively expensive, you can't do that on every packet.
Now you are making my point ;) This is part of the cost of your
signaling path, and it directly adds to your latency time.
It adds a microsecond. The kvm overhead of putting things in userspace
is low enough, I don't know why people keep mentioning it. The problem
is the kernel/user networking interfaces.
You can't
buffer packets here if the guest is only going to send one and wait for
a response and expect that to perform well. And this is precisely what
drove me to look at avoiding going back to userspace in the first place.
We're not buffering any packets. What we lack is a way to tell the
guest that we're done processing all packets in the ring (IOW, re-enable
notifications).
--
error compiling committee.c: too many arguments to function
From: Avi Kivity <hidden> Date: 2009-04-02 11:59:56
Gregory Haskins wrote:
quoted
Why does a kernel solution not need to know when a packet is transmitted?
You do not need to know when the packet is copied (which I currently
do). You only need it for zero-copy (of which I would like to support,
but as I understand it there are problems with the reliability of proper
callback (i.e. skb->destructor).
Its "fire and forget" :)
It's more of a "schedule and forget" which I think brings you the win.
The host disables notifications and schedules the actual tx work (rx
from the host's perspective). So now the guest and host continue
producing and consuming packets in parallel. So long as the guest is
faster (due to the host being throttled?), notifications continue to be
disabled.
If you changed your rx_isr() to process the packets immediately instead
of scheduling, I think throughput would drop dramatically.
Mark had a similar change for virtio. Mark?
--
error compiling committee.c: too many arguments to function
From: Rusty Russell <hidden> Date: 2009-04-02 12:13:43
On Thursday 02 April 2009 21:36:07 Gregory Haskins wrote:
You do not need to know when the packet is copied (which I currently
do). You only need it for zero-copy (of which I would like to support,
but as I understand it there are problems with the reliability of proper
callback (i.e. skb->destructor).
But if you have a UP guest, there will *never* be another packet in the queue
at this point, since it wasn't running.
As Avi said, you can do the processing in another thread and go back to the
guest; lguest pre-virtio did a hacky "weak" wakeup to ensure the guest ran
again before the thread did for exactly this kind of reason.
While Avi's point about a "powerful enough userspace API" is probably valid,
I don't think it's going to happen. It's almost certainly less code to put a
virtio_net server in the kernel, than it is to create such a powerful
interface (see vringfd & tap). And that interface would have one user in
practice.
So, let's roll out a kernel virtio_net server. Anyone?
Rusty.
virtio is already non-kvm-specific (lguest uses it) and
non-pci-specific (s390 uses it).
Ok, then to be more specific, I need it to be more generic than it
already is. For instance, I need it to be able to integrate with
shm_signals.
Why?
Well, shm_signals is what I designed to be the event mechanism for vbus
devices. One of the design criteria of shm_signal is that it should
support a variety of environments, such as kvm, but also something like
userspace apps. So I cannot make assumptions about things like "pci
interrupts", etc.
So if I want to use it in vbus, virtio-ring has to be able to use them,
as opposed to what it does today. Part of this would be a natural fit
for the "kick()" callback in virtio, but there are other problems. For
one, virtio-ring (IIUC) does its own event-masking directly in the
virtio metadata. However, really I want the higher layer ring-overlay
to do its masking in terms of the lower-layered shm_signal in order to
work the way I envision this stuff. If you look at the IOQ
implementation, this is exactly what it does.
To be clear, and Ive stated this in the past: venet is just an example
of this generic, in-kernel concept. We plan on doing much much more
with all this. One of the things we are working on is have userspace
clients be able to access this too, with an ultimately goal of
supporting things like having guest-userspace doing bypass, rdma, etc.
We are not there yet, though...only the kvm-host to guest kernel is
currently functional and is thus the working example.
I totally "get" the attraction to doing things in userspace. Its
contained, naturally isolated, easily supports migration, etc. Its also
a penalty. Bare-metal userspace apps have a direct path to the kernel
IO. I want to give guest the same advantage. Some people will care
more about things like migration than performance, and that is fine.
But others will certainly care more about performance, and that is what
we are trying to address.
quoted
quoted
If you have a good exit mitigation scheme you can cut exits by a
factor of 100; so the userspace exit costs are cut by the same
factor. If you have good copyless networking APIs you can cut the
cost of copies to zero (well, to the cost of get_user_pages_fast(),
but a kernel solution needs that too).
"exit mitigation' schemes are for bandwidth, not latency. For latency
it all comes down to how fast you can signal in both directions. If
someone is going to do a stand-alone request-reply, its generally always
going to be at least one hypercall and one rx-interrupt. So your speed
will be governed by your signal path, not your buffer bandwidth.
The userspace path is longer by 2 microseconds (for two additional
heavyweight exits) and a few syscalls. I don't think that's worthy of
putting all the code in the kernel.
By your own words, the exit to userspace is "prohibitively expensive",
so that is either true or its not. If its 2 microseconds, show me. We
need the rtt time to go from a "kick" PIO all the way to queue a packet
on the egress hardware and return. That is going to define your
latency. If you can do this such that you can do something like ICMP
ping in 65us (or anything close to a few dozen microseconds of this),
I'll shut-up about how much I think the current path sucks ;) Even so,
I still propose the concept of a frame-work for in-kernel devices for
all the other reasons I mentioned above.
-Greg
Why does a kernel solution not need to know when a packet is
transmitted?
You do not need to know when the packet is copied (which I currently
do). You only need it for zero-copy (of which I would like to support,
but as I understand it there are problems with the reliability of proper
callback (i.e. skb->destructor).
Its "fire and forget" :)
It's more of a "schedule and forget" which I think brings you the
win. The host disables notifications and schedules the actual tx work
(rx from the host's perspective). So now the guest and host continue
producing and consuming packets in parallel. So long as the guest is
faster (due to the host being throttled?), notifications continue to
be disabled.
Yep, when the "producer::consumer" ratio is > 1, we mitigate signaling.
When its < 1, we signal roughly once per packet.
If you changed your rx_isr() to process the packets immediately
instead of scheduling, I think throughput would drop dramatically.
Right, that is the point. :) This is that "soft asic" thing I was
talking about yesterday.
-Greg
From: Avi Kivity <hidden> Date: 2009-04-02 12:43:17
Gregory Haskins wrote:
Avi Kivity wrote:
quoted
Gregory Haskins wrote:
quoted
quoted
virtio is already non-kvm-specific (lguest uses it) and
non-pci-specific (s390 uses it).
Ok, then to be more specific, I need it to be more generic than it
already is. For instance, I need it to be able to integrate with
shm_signals.
Why?
Well, shm_signals is what I designed to be the event mechanism for vbus
devices. One of the design criteria of shm_signal is that it should
support a variety of environments, such as kvm, but also something like
userspace apps. So I cannot make assumptions about things like "pci
interrupts", etc.
virtio doesn't make these assumptions either. The only difference I see
is that you separate notification from the ring structure.
By your own words, the exit to userspace is "prohibitively expensive",
so that is either true or its not. If its 2 microseconds, show me.
In user/test/x86/vmexit.c, change 'cpuid' to 'out %al, $0'; drop the
printf() in kvmctl.c's test_outb().
I get something closer to 4 microseconds, but that's on a two year old
machine; It will be around two on Nehalems.
My 'prohibitively expensive' is true only if you exit every packet.
--
error compiling committee.c: too many arguments to function
From: Avi Kivity <hidden> Date: 2009-04-02 12:44:25
Gregory Haskins wrote:
quoted
It's more of a "schedule and forget" which I think brings you the
win. The host disables notifications and schedules the actual tx work
(rx from the host's perspective). So now the guest and host continue
producing and consuming packets in parallel. So long as the guest is
faster (due to the host being throttled?), notifications continue to
be disabled.
Yep, when the "producer::consumer" ratio is > 1, we mitigate signaling.
When its < 1, we signal roughly once per packet.
quoted
If you changed your rx_isr() to process the packets immediately
instead of scheduling, I think throughput would drop dramatically.
Right, that is the point. :) This is that "soft asic" thing I was
talking about yesterday.
But all that has nothing to do with where the code lives, in the kernel
or userspace.
--
error compiling committee.c: too many arguments to function
On Thursday 02 April 2009 21:36:07 Gregory Haskins wrote:
quoted
You do not need to know when the packet is copied (which I currently
do). You only need it for zero-copy (of which I would like to support,
but as I understand it there are problems with the reliability of proper
callback (i.e. skb->destructor).
But if you have a UP guest,
I assume you mean UP host ;)
there will *never* be another packet in the queue
at this point, since it wasn't running.
Yep, and I'll be the first to admit that my design only looks forward.
Its for high speed links and multi-core cpus, etc. If you have a
uniprocessor host, the throughput would likely start to suffer with my
current strategy. You could probably reclaim some of that throughput
(but trading latency) by doing as you are suggesting with the deferred
initial signalling. However, it is still a tradeoff to account for the
lower-end rig. I could certainly put a heuristic/timer on the
guest->host to mitigate this as well, but this is not my target use case
anyway so I am not sure it is worth it.
As Avi said, you can do the processing in another thread and go back to the
guest; lguest pre-virtio did a hacky "weak" wakeup to ensure the guest ran
again before the thread did for exactly this kind of reason.
While Avi's point about a "powerful enough userspace API" is probably valid,
I don't think it's going to happen. It's almost certainly less code to put a
virtio_net server in the kernel, than it is to create such a powerful
interface (see vringfd & tap). And that interface would have one user in
practice.
So, let's roll out a kernel virtio_net server. Anyone?
Hmm..well I was hoping to be able to work with you guys to make my
proposal fit this role. If there is no interest in that, I hope that my
infrastructure itself may still be considered for merging (in *some*
tree, not -kvm per se) as I would prefer to not maintain it out of tree
if it can be avoided. I think people will find that the new logic
touches very few existing kernel lines at all, and can be completely
disabled with config options so it should be relatively inconsequential
to those that do not care.
-Greg
It's more of a "schedule and forget" which I think brings you the
win. The host disables notifications and schedules the actual tx work
(rx from the host's perspective). So now the guest and host continue
producing and consuming packets in parallel. So long as the guest is
faster (due to the host being throttled?), notifications continue to
be disabled.
Yep, when the "producer::consumer" ratio is > 1, we mitigate
signaling. When its < 1, we signal roughly once per packet.
quoted
If you changed your rx_isr() to process the packets immediately
instead of scheduling, I think throughput would drop dramatically.
Right, that is the point. :) This is that "soft asic" thing I was
talking about yesterday.
But all that has nothing to do with where the code lives, in the
kernel or userspace.
Agreed, but note Ive already stated that some of my boost is likely from
in-kernel, while others are unrelated design elements such as the
"soft-asic" approach (you guys dont read my 10 page emails, do you? ;).
I don't deny that some of my ideas could be used in userspace as well
(Credit if used would be appreciated :).
-Greg
From: Avi Kivity <hidden> Date: 2009-04-02 13:07:36
Gregory Haskins wrote:
Rusty Russell wrote:
quoted
On Thursday 02 April 2009 21:36:07 Gregory Haskins wrote:
quoted
You do not need to know when the packet is copied (which I currently
do). You only need it for zero-copy (of which I would like to support,
but as I understand it there are problems with the reliability of proper
callback (i.e. skb->destructor).
But if you have a UP guest,
I assume you mean UP host ;)
I think Rusty did mean a UP guest, and without schedule-and-forget.
Hmm..well I was hoping to be able to work with you guys to make my
proposal fit this role. If there is no interest in that, I hope that my
infrastructure itself may still be considered for merging (in *some*
tree, not -kvm per se) as I would prefer to not maintain it out of tree
if it can be avoided.
The problem is that we already have virtio guest drivers going several
kernel versions back, as well as Windows drivers. We can't keep
changing the infrastructure under people's feet.
--
error compiling committee.c: too many arguments to function
On Thursday 02 April 2009 21:36:07 Gregory Haskins wrote:
quoted
You do not need to know when the packet is copied (which I currently
do). You only need it for zero-copy (of which I would like to
support,
but as I understand it there are problems with the reliability of
proper
callback (i.e. skb->destructor).
But if you have a UP guest,
I assume you mean UP host ;)
I think Rusty did mean a UP guest, and without schedule-and-forget.
That doesnt make sense to me, tho. All the testing I did was a UP
guest, actually. Why would I be constrained to run without the
scheduling unless the host was also UP?
quoted
Hmm..well I was hoping to be able to work with you guys to make my
proposal fit this role. If there is no interest in that, I hope that my
infrastructure itself may still be considered for merging (in *some*
tree, not -kvm per se) as I would prefer to not maintain it out of tree
if it can be avoided.
The problem is that we already have virtio guest drivers going several
kernel versions back, as well as Windows drivers. We can't keep
changing the infrastructure under people's feet.
Well, IIUC the virtio code itself declares the ABI as unstable, so there
technically *is* an out if we really wanted one. But I certainly
understand the desire to not change this ABI if at all possible, and
thus the resistance here.
However, theres still the possibility we can make this work in an ABI
friendly way with cap-bits, or other such features. For instance, the
virtio-net driver could register both with pci and vbus-proxy and
instantiate a device with a slightly different ops structure for each or
something. Alternatively we could write a host-side shim to expose vbus
devices as pci devices or something like that.
-Greg
From: Avi Kivity <hidden> Date: 2009-04-02 13:28:18
Gregory Haskins wrote:
Avi Kivity wrote:
quoted
Gregory Haskins wrote:
quoted
Rusty Russell wrote:
quoted
On Thursday 02 April 2009 21:36:07 Gregory Haskins wrote:
quoted
You do not need to know when the packet is copied (which I currently
do). You only need it for zero-copy (of which I would like to
support,
but as I understand it there are problems with the reliability of
proper
callback (i.e. skb->destructor).
But if you have a UP guest,
I assume you mean UP host ;)
I think Rusty did mean a UP guest, and without schedule-and-forget.
That doesnt make sense to me, tho. All the testing I did was a UP
guest, actually. Why would I be constrained to run without the
scheduling unless the host was also UP?
You aren't constrained. And your numbers show it works.
quoted
The problem is that we already have virtio guest drivers going several
kernel versions back, as well as Windows drivers. We can't keep
changing the infrastructure under people's feet.
Well, IIUC the virtio code itself declares the ABI as unstable, so there
technically *is* an out if we really wanted one. But I certainly
understand the desire to not change this ABI if at all possible, and
thus the resistance here.
virtio is a stable ABI.
However, theres still the possibility we can make this work in an ABI
friendly way with cap-bits, or other such features. For instance, the
virtio-net driver could register both with pci and vbus-proxy and
instantiate a device with a slightly different ops structure for each or
something. Alternatively we could write a host-side shim to expose vbus
devices as pci devices or something like that.
Sounds complicated...
--
error compiling committee.c: too many arguments to function
My 'prohibitively expensive' is true only if you exit every packet.
Understood, but yet you need to do this if you want something like iSCSI
READ transactions to have as low-latency as possible.
Dunno, two microseconds is too much? The wire imposes much more.
No, but thats not what we are talking about. You said signaling on
every packet is prohibitively expensive. I am saying signaling on every
packet is required for decent latency. So is it prohibitively expensive
or not?
I think most would agree that adding 2us is not bad, but so far that is
an unproven theory that the IO path in question only adds 2us. And we
are not just looking at the rate at which we can enter and exit the
guest...we need the whole path...from the PIO kick to the dev_xmit() on
the egress hardware, to the ingress and rx-injection. This includes any
and all penalties associated with the path, even if they are imposed by
something like the design of tun-tap.
Right now its way way way worse than 2us. In fact, at my last reading
this was more like 3060us (3125-65). So shorten that 3125 to 67 (while
maintaining line-rate) and I will be impressed. Heck, shorten it to
80us and I will be impressed.
-Greg
From: Avi Kivity <hidden> Date: 2009-04-02 13:46:22
Gregory Haskins wrote:
Avi Kivity wrote:
quoted
Gregory Haskins wrote:
quoted
Avi Kivity wrote:
quoted
My 'prohibitively expensive' is true only if you exit every packet.
Understood, but yet you need to do this if you want something like iSCSI
READ transactions to have as low-latency as possible.
Dunno, two microseconds is too much? The wire imposes much more.
No, but thats not what we are talking about. You said signaling on
every packet is prohibitively expensive. I am saying signaling on every
packet is required for decent latency. So is it prohibitively expensive
or not?
We're heading dangerously into the word-game area. Let's not do that.
If you have a high throughput workload with many packets per seconds
then an exit per packet (whether to userspace or to the kernel) is
expensive. So you do exit mitigation. Latency is not important since
the packets are going to sit in the output queue anyway.
If you have a request-response workload with the wire idle and latency
critical, then there's no problem having an exit per packet because (a)
there aren't that many packets and (b) the guest isn't doing any
batching, so guest overhead will swamp the hypervisor overhead.
If you have a low latency request-response workload mixed with a high
throughput workload, then you aren't going to get low latency since your
low latency packets will sit on the queue behind the high throughput
packets. You can fix that with multiqueue and then you're back to one
of the scenarios above.
I think most would agree that adding 2us is not bad, but so far that is
an unproven theory that the IO path in question only adds 2us. And we
are not just looking at the rate at which we can enter and exit the
guest...we need the whole path...from the PIO kick to the dev_xmit() on
the egress hardware, to the ingress and rx-injection. This includes any
and all penalties associated with the path, even if they are imposed by
something like the design of tun-tap.
Correct, we need to look at the whole path. That's why the wishing well
is clogged with my 'give me a better userspace interface' emails.
Right now its way way way worse than 2us. In fact, at my last reading
this was more like 3060us (3125-65). So shorten that 3125 to 67 (while
maintaining line-rate) and I will be impressed. Heck, shorten it to
80us and I will be impressed.
The 3060us thing is a timer, not cpu time. We aren't starting a JVM for
each packet. We could remove it given a notification API, or
duplicating the sched-and-forget thing, like Rusty did with lguest or
Mark with qemu.
--
error compiling committee.c: too many arguments to function
On Thursday 02 April 2009 21:36:07 Gregory Haskins wrote:
quoted
You do not need to know when the packet is copied (which I currently
do). You only need it for zero-copy (of which I would like to
support,
but as I understand it there are problems with the reliability of
proper
callback (i.e. skb->destructor).
But if you have a UP guest,
I assume you mean UP host ;)
I think Rusty did mean a UP guest, and without schedule-and-forget.
That doesnt make sense to me, tho. All the testing I did was a UP
guest, actually. Why would I be constrained to run without the
scheduling unless the host was also UP?
You aren't constrained. And your numbers show it works.
quoted
quoted
The problem is that we already have virtio guest drivers going several
kernel versions back, as well as Windows drivers. We can't keep
changing the infrastructure under people's feet.
Well, IIUC the virtio code itself declares the ABI as unstable, so there
technically *is* an out if we really wanted one. But I certainly
understand the desire to not change this ABI if at all possible, and
thus the resistance here.
virtio is a stable ABI.
Dang! Scratch that.
quoted
However, theres still the possibility we can make this work in an ABI
friendly way with cap-bits, or other such features. For instance, the
virtio-net driver could register both with pci and vbus-proxy and
instantiate a device with a slightly different ops structure for each or
something. Alternatively we could write a host-side shim to expose vbus
devices as pci devices or something like that.
Sounds complicated...
Well, the first solution would be relatively trivial...at least on the
guest side. All the other infrastructure is done and included in the
series I sent out. The changes to the virtio-net driver on the guest
itself would be minimal. The bigger effort would be converting
venet-tap to use virtio-ring instead of IOQ. But this would arguably be
less work than starting a virtio-net backend module from scratch because
you would have to not only code up the entire virtio-net backend, but
also all the pci emulation and irq routing stuff that is required (and
is already done by the vbus infrastructure). Here all the major pieces
are in place, just the xmit and rx routines need to be converted to
virtio-isms.
For the second option, I agree. Its probably too nasty and it would be
better if there was just either a virtio-net to kvm-host hack, or a more
pci oriented version of a vbus-like framework.
That said, there is certainly nothing wrong with having an alternate
option. There is plenty of precedent for having different drivers for
different subsystems, etc, even if there is overlap. Heck, even KVM has
realtek, e1000, and virtio-net, etc. Would our kvm community be willing
to work with me to get these patches merged? I am perfectly willing to
maintain them. That said, the general infrastructure should probably
not live in -kvm (perhaps -tip, -mm, or -next, etc is more
appropriate). So a good plan might be to shoot for the core going into
a more general upstream tree. When/if that happens, then the kvm
community could consider the kvm specific parts, etc. I realize this is
all pending review acceptance by everyone involved...
-Greg
My 'prohibitively expensive' is true only if you exit every packet.
Understood, but yet you need to do this if you want something like
iSCSI
READ transactions to have as low-latency as possible.
Dunno, two microseconds is too much? The wire imposes much more.
No, but thats not what we are talking about. You said signaling on
every packet is prohibitively expensive. I am saying signaling on every
packet is required for decent latency. So is it prohibitively expensive
or not?
We're heading dangerously into the word-game area. Let's not do that.
If you have a high throughput workload with many packets per seconds
then an exit per packet (whether to userspace or to the kernel) is
expensive. So you do exit mitigation. Latency is not important since
the packets are going to sit in the output queue anyway.
Agreed. virtio-net currently does this with batching. I do with the
bidir napi thing (which effectively crosses the producer::consumer > 1
threshold to mitigate the signal path).
If you have a request-response workload with the wire idle and latency
critical, then there's no problem having an exit per packet because
(a) there aren't that many packets and (b) the guest isn't doing any
batching, so guest overhead will swamp the hypervisor overhead.
Right, so the trick is to use an algorithm that adapts here. Batching
solves the first case, but not the second. The bidir napi thing solves
both, but it does assume you have ample host processing power to run the
algorithm concurrently. This may or may not be suitable to all
applications, I admit.
If you have a low latency request-response workload mixed with a high
throughput workload, then you aren't going to get low latency since
your low latency packets will sit on the queue behind the high
throughput packets. You can fix that with multiqueue and then you're
back to one of the scenarios above.
Agreed, and thats ok. Now we are getting more into 802.1p type MQ
issues anyway, if the application cared about it that much.
quoted
I think most would agree that adding 2us is not bad, but so far that is
an unproven theory that the IO path in question only adds 2us. And we
are not just looking at the rate at which we can enter and exit the
guest...we need the whole path...from the PIO kick to the dev_xmit() on
the egress hardware, to the ingress and rx-injection. This includes any
and all penalties associated with the path, even if they are imposed by
something like the design of tun-tap.
Correct, we need to look at the whole path. That's why the wishing
well is clogged with my 'give me a better userspace interface' emails.
quoted
Right now its way way way worse than 2us. In fact, at my last reading
this was more like 3060us (3125-65). So shorten that 3125 to 67 (while
maintaining line-rate) and I will be impressed. Heck, shorten it to
80us and I will be impressed.
The 3060us thing is a timer, not cpu time.
Agreed, but its still "state of the art" from an observer perspective.
The reason "why", though easily explainable, is inconsequential to most
people. FWIW, I have seen virtio-net do a much more respectable 350us
on an older version, so I know there is plenty of room for improvement.
We aren't starting a JVM for each packet.
Heh...it kind of feels like that right now, so hopefully some
improvement will at least be on the one thing that comes out of all this.
-Greg
From: Avi Kivity <hidden> Date: 2009-04-02 14:32:34
Gregory Haskins wrote:
quoted
If you have a request-response workload with the wire idle and latency
critical, then there's no problem having an exit per packet because
(a) there aren't that many packets and (b) the guest isn't doing any
batching, so guest overhead will swamp the hypervisor overhead.
Right, so the trick is to use an algorithm that adapts here. Batching
solves the first case, but not the second. The bidir napi thing solves
both, but it does assume you have ample host processing power to run the
algorithm concurrently. This may or may not be suitable to all
applications, I admit.
The alternative is to get a notification from the stack that the packet
is done processing. Either an skb destructor in the kernel, or my new
API that everyone is not rushing out to implement.
quoted
quoted
Right now its way way way worse than 2us. In fact, at my last reading
this was more like 3060us (3125-65). So shorten that 3125 to 67 (while
maintaining line-rate) and I will be impressed. Heck, shorten it to
80us and I will be impressed.
The 3060us thing is a timer, not cpu time.
Agreed, but its still "state of the art" from an observer perspective.
The reason "why", though easily explainable, is inconsequential to most
people. FWIW, I have seen virtio-net do a much more respectable 350us
on an older version, so I know there is plenty of room for improvement.
All I want is the notification, and the timer is headed into the nearest
landfill.
--
error compiling committee.c: too many arguments to function
From: Avi Kivity <hidden> Date: 2009-04-02 14:42:42
Avi Kivity wrote:
The alternative is to get a notification from the stack that the
packet is done processing. Either an skb destructor in the kernel, or
my new API that everyone is not rushing out to implement.
btw, my new api is
io_submit(..., nr, ...): submit nr packets
io_getevents(): complete nr packets
--
error compiling committee.c: too many arguments to function
From: Anthony Liguori <hidden> Date: 2009-04-02 14:49:52
Avi Kivity wrote:
Avi Kivity wrote:
quoted
The alternative is to get a notification from the stack that the
packet is done processing. Either an skb destructor in the kernel,
or my new API that everyone is not rushing out to implement.
btw, my new api is
io_submit(..., nr, ...): submit nr packets
io_getevents(): complete nr packets
I don't think we even need that to end this debate. I'm convinced we
have a bug somewhere. Even disabling TX mitigation, I see a ping
latency of around 300ns whereas it's only 50ns on the host. This defies
logic so I'm now looking to isolate why that is.
Regards,
Anthony Liguori
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2009-04-02 15:01:20
On Thu, Apr 02, 2009 at 04:07:09PM +0300, Avi Kivity wrote:
I think Rusty did mean a UP guest, and without schedule-and-forget.
Going off on a tangent here, I don't really think it should matter
whether we're UP or SMP. The ideal state is where we have the
same number of (virtual) TX queues as there are cores in the guest.
On the host side we need the backend to run at least on a core
that shares cache with the corresponding guest queue/core. If
that happens to be the same core as the guest core then it should
work as well.
IOW we should optimise it as if the host were UP.
The problem is that we already have virtio guest drivers going several
kernel versions back, as well as Windows drivers. We can't keep
changing the infrastructure under people's feet.
From: Avi Kivity <hidden> Date: 2009-04-02 15:06:17
Herbert Xu wrote:
On Thu, Apr 02, 2009 at 04:07:09PM +0300, Avi Kivity wrote:
quoted
I think Rusty did mean a UP guest, and without schedule-and-forget.
Going off on a tangent here, I don't really think it should matter
whether we're UP or SMP. The ideal state is where we have the
same number of (virtual) TX queues as there are cores in the guest.
On the host side we need the backend to run at least on a core
that shares cache with the corresponding guest queue/core. If
that happens to be the same core as the guest core then it should
work as well.
IOW we should optimise it as if the host were UP.
Good point - if we rely on having excess cores in the host, large guest
scalability will drop.
--
error compiling committee.c: too many arguments to function
From: Michael S. Tsirkin <hidden> Date: 2009-04-02 15:11:48
On Thu, Apr 02, 2009 at 10:43:19PM +1030, Rusty Russell wrote:
On Thursday 02 April 2009 21:36:07 Gregory Haskins wrote:
quoted
You do not need to know when the packet is copied (which I currently
do). You only need it for zero-copy (of which I would like to support,
but as I understand it there are problems with the reliability of proper
callback (i.e. skb->destructor).
But if you have a UP guest, there will *never* be another packet in the queue
at this point, since it wasn't running.
As Avi said, you can do the processing in another thread and go back to the
guest; lguest pre-virtio did a hacky "weak" wakeup to ensure the guest ran
again before the thread did for exactly this kind of reason.
While Avi's point about a "powerful enough userspace API" is probably valid,
I don't think it's going to happen. It's almost certainly less code to put a
virtio_net server in the kernel, than it is to create such a powerful
interface (see vringfd & tap). And that interface would have one user in
practice.
So, let's roll out a kernel virtio_net server. Anyone?
Rusty.
BTW, whatever approach is chosen, to enable zero-copy transmits, it seems that
we still must add tracking of when the skb has actually been transmitted, right?
Rusty, I think this is what you did in your patch from 2008 to add destructor
for skb data ( http://kerneltrap.org/mailarchive/linux-netdev/2008/4/18/1464944 ):
and it seems that it would make zero-copy possible - or was there some problem with
that approach? Do you happen to remember?
--
MST
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2009-04-02 15:41:22
On Thu, Apr 02, 2009 at 06:00:17PM +0300, Avi Kivity wrote:
Good point - if we rely on having excess cores in the host, large guest
scalability will drop.
Going back to TX mitigation, I wonder if we could avoid it altogether
by having a "wakeup" mechanism that does not involve a vmexit. We
have two cases:
1) UP, or rather guest runs on the same core/hyperthread as the
backend. This is the easy one, the guest simply sets a marker
in shared memory and keeps going until its time is up. Then the
backend takes over, and uses a marker for notification too.
The markers need to be interpreted by the scheduler so that it
knows the guest/backend is runnable, respectively.
2) The guest and backend runs on two cores/hyperthreads. We'll
assume that they share caches as otherwise mitigation is the last
thing to worry about. We use the same marker mechanism as above.
The only caveat is that if one core/hyperthread is idle, its
idle thread needs to monitor the marker (this would be a separate
per-core marker) to wake up the scheduler.
CCing Ingo so that he can flame me if I'm totally off the mark.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
From: Avi Kivity <hidden> Date: 2009-04-02 15:59:12
Herbert Xu wrote:
On Thu, Apr 02, 2009 at 06:00:17PM +0300, Avi Kivity wrote:
quoted
Good point - if we rely on having excess cores in the host, large guest
scalability will drop.
Going back to TX mitigation, I wonder if we could avoid it altogether
by having a "wakeup" mechanism that does not involve a vmexit. We
have two cases:
1) UP, or rather guest runs on the same core/hyperthread as the
backend. This is the easy one, the guest simply sets a marker
in shared memory and keeps going until its time is up. Then the
backend takes over, and uses a marker for notification too.
The markers need to be interpreted by the scheduler so that it
knows the guest/backend is runnable, respectively.
Let's look at this first.
What if the guest sends N packets, then does some expensive computation
(say the guest scheduler switches from the benchmark process to
evolution). So now we have the marker set at packet N, but the host
will not see it until the guest timeslice is up?
I think I totally misunderstood you. Can you repeat in smaller words?
--
error compiling committee.c: too many arguments to function
From: Anthony Liguori <hidden> Date: 2009-04-02 16:10:44
Anthony Liguori wrote:
Avi Kivity wrote:
quoted
Avi Kivity wrote:
quoted
The alternative is to get a notification from the stack that the
packet is done processing. Either an skb destructor in the kernel,
or my new API that everyone is not rushing out to implement.
btw, my new api is
io_submit(..., nr, ...): submit nr packets
io_getevents(): complete nr packets
I don't think we even need that to end this debate. I'm convinced we
have a bug somewhere. Even disabling TX mitigation, I see a ping
latency of around 300ns whereas it's only 50ns on the host. This
defies logic so I'm now looking to isolate why that is.
I'm down to 90us. Obviously, s/ns/us/g above. The exec.c changes were
the big winner... I hate qemu sometimes.
I'm pretty confident I can get at least to Greg's numbers with some
poking. I think I understand why he's doing better after reading his
patches carefully but I also don't think it'll scale with many guests
well... stay tuned.
But most importantly, we are darn near where vbus is with this patch wrt
added packet latency and this is totally from userspace with no host
kernel changes.
So no, userspace is not the issue.
Regards,
Anthony Liguori
From: Avi Kivity <hidden> Date: 2009-04-02 16:19:39
Anthony Liguori wrote:
quoted
I don't think we even need that to end this debate. I'm convinced we
have a bug somewhere. Even disabling TX mitigation, I see a ping
latency of around 300ns whereas it's only 50ns on the host. This
defies logic so I'm now looking to isolate why that is.
I'm down to 90us. Obviously, s/ns/us/g above. The exec.c changes
were the big winner... I hate qemu sometimes.
The way I read it, it will run only run slowly once per page, then
settle to a cache miss per page.
Regardless, it makes a memslot model even more attractive.
--
error compiling committee.c: too many arguments to function
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2009-04-02 16:28:41
On Thu, Apr 02, 2009 at 06:57:38PM +0300, Avi Kivity wrote:
What if the guest sends N packets, then does some expensive computation
(say the guest scheduler switches from the benchmark process to
evolution). So now we have the marker set at packet N, but the host
will not see it until the guest timeslice is up?
Well that's fine. The guest will use up the remainder of its
timeslice. After all we only have one core/hyperthread here so
this is no different than if the packets were held up higher up
in the guest kernel and the guest decided to do some computation.
Once its timeslice completes the backend can start plugging away
at the backlog.
Of course it would be better to put the backend on another core
that shares the cache or a hyperthread on the same core.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
From: Avi Kivity <hidden> Date: 2009-04-02 16:55:30
Herbert Xu wrote:
On Thu, Apr 02, 2009 at 06:57:38PM +0300, Avi Kivity wrote:
quoted
What if the guest sends N packets, then does some expensive computation
(say the guest scheduler switches from the benchmark process to
evolution). So now we have the marker set at packet N, but the host
will not see it until the guest timeslice is up?
Well that's fine. The guest will use up the remainder of its
timeslice. After all we only have one core/hyperthread here so
this is no different than if the packets were held up higher up
in the guest kernel and the guest decided to do some computation.
3ms latency for ping?
(ping will always be scheduled immediately when the reply arrives if I
understand cfs, so guest load won't delay it)
--
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2009-04-02 17:06:45
On Thu, Apr 02, 2009 at 07:54:21PM +0300, Avi Kivity wrote:
3ms latency for ping?
(ping will always be scheduled immediately when the reply arrives if I
understand cfs, so guest load won't delay it)
That only happens if the guest immediately does some CPU-intensive
computation 3ms and assuming its timeslice lasts that long.
In any case, the same thing will happen right now if the host or
some other guest on the same CPU hogs the CPU for 3ms.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2009-04-02 17:18:17
On Fri, Apr 03, 2009 at 01:06:10AM +0800, Herbert Xu wrote:
That only happens if the guest immediately does some CPU-intensive
computation 3ms and assuming its timeslice lasts that long.
In any case, the same thing will happen right now if the host or
some other guest on the same CPU hogs the CPU for 3ms.
Even better, look at the packet's TOS. If it's marked for low-
latency then vmexit immediately. Otherwise continue.
In the backend you'd just set the marker in shared memory.
Of course invert this for the host => guest direction.
Cheers,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
From: Anthony Liguori <hidden> Date: 2009-04-02 18:19:40
Avi Kivity wrote:
Anthony Liguori wrote:
quoted
quoted
I don't think we even need that to end this debate. I'm convinced
we have a bug somewhere. Even disabling TX mitigation, I see a ping
latency of around 300ns whereas it's only 50ns on the host. This
defies logic so I'm now looking to isolate why that is.
I'm down to 90us. Obviously, s/ns/us/g above. The exec.c changes
were the big winner... I hate qemu sometimes.
What, this:
UDP_RR test was limited by CPU consumption. QEMU was pegging a CPU with
only about 4000 packets per second whereas the host could do 14000. An
oprofile run showed that phys_page_find/cpu_physical_memory_rw where at
the top by a wide margin which makes little sense since virtio is zero
copy in kvm-userspace today.
That leaves the ring queue accessors that used ld[wlq]_phys and friends
that happen to make use of the above. That led me to try this terrible
hack below and low and beyond, we immediately jumped to 10000 pps. This
only works because almost nothing uses ld[wlq]_phys in practice except
for virtio so breaking it for the non-RAM case didn't matter.
We didn't encounter this before because when I changed this behavior, I
tested streaming and ping. Both remained the same. You can only expose
this issue if you first disable tx mitigation.
Anyway, if we're able to send this many packets, I suspect we'll be able
to also handle much higher throughputs without TX mitigation so that's
what I'm going to look at now.
Regards,
Anthony Liguori
From: Herbert Xu <herbert@gondor.apana.org.au> Date: 2009-04-03 01:12:01
Anthony Liguori [off-list ref] wrote:
Anyway, if we're able to send this many packets, I suspect we'll be able
to also handle much higher throughputs without TX mitigation so that's
what I'm going to look at now.
Awesome! I'm prepared to eat my words :)
On the subject of TX mitigation, can we please set a standard
on how we measure it? For instance, do we bind the the backend
qemu to the same CPU as the guest, or do we bind it to a different
CPU that shares cache? They're two completely different scenarios
and I think we should be explicit about which one we're measuring.
Thanks,
--
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} [off-list ref]
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
From: Jeremy Fitzhardinge <hidden> Date: 2009-04-03 04:43:55
Michael S. Tsirkin wrote:
Rusty, I think this is what you did in your patch from 2008 to add destructor
for skb data ( http://kerneltrap.org/mailarchive/linux-netdev/2008/4/18/1464944 ):
and it seems that it would make zero-copy possible - or was there some problem with
that approach? Do you happen to remember?
I'm planning on resurrecting it to replace the page destructor used by
Xen netback.
J