From: Rik van Riel <hidden> Date: 2012-06-26 20:41:23
On 06/26/2012 04:32 PM, Frank Swiderski wrote:
This implementation of a virtio balloon driver uses the page cache to
"store" pages that have been released to the host. The communication
(outside of target counts) is one way--the guest notifies the host when
it adds a page to the page cache, allowing the host to madvise(2) with
MADV_DONTNEED. Reclaim in the guest is therefore automatic and implicit
(via the regular page reclaim). This means that inflating the balloon
is similar to the existing balloon mechanism, but the deflate is
different--it re-uses existing Linux kernel functionality to
automatically reclaim.
Signed-off-by: Frank Swiderski<redacted>
It is a great idea, but how can this memory balancing
possibly work if someone uses memory cgroups inside a
guest?
Having said that, we currently do not have proper
memory reclaim balancing between cgroups at all, so
requiring that of this balloon driver would be
unreasonable.
The code looks good to me, my only worry is the
code duplication. We now have 5 balloon drivers,
for 4 hypervisors, all implementing everything
from scratch...
From: Frank Swiderski <hidden> Date: 2012-06-26 21:31:30
On Tue, Jun 26, 2012 at 1:40 PM, Rik van Riel [off-list ref] wrote:
On 06/26/2012 04:32 PM, Frank Swiderski wrote:
quoted
This implementation of a virtio balloon driver uses the page cache to
"store" pages that have been released to the host. The communication
(outside of target counts) is one way--the guest notifies the host when
it adds a page to the page cache, allowing the host to madvise(2) with
MADV_DONTNEED. Reclaim in the guest is therefore automatic and implicit
(via the regular page reclaim). This means that inflating the balloon
is similar to the existing balloon mechanism, but the deflate is
different--it re-uses existing Linux kernel functionality to
automatically reclaim.
Signed-off-by: Frank Swiderski<redacted>
It is a great idea, but how can this memory balancing
possibly work if someone uses memory cgroups inside a
guest?
Thanks and good point--this isn't something that I considered in the
implementation.
Having said that, we currently do not have proper
memory reclaim balancing between cgroups at all, so
requiring that of this balloon driver would be
unreasonable.
The code looks good to me, my only worry is the
code duplication. We now have 5 balloon drivers,
for 4 hypervisors, all implementing everything
from scratch...
Do you have any recommendations on this? I could (I think reasonably
so) modify the existing virtio_balloon.c and have it change behavior
based on a feature bit or other configuration. I'm not sure that
really addresses the root of what you're pointing out--it's still
adding a different implementation, but doing so as an extension of an
existing one.
fes
From: Rik van Riel <hidden> Date: 2012-06-26 21:46:47
On 06/26/2012 05:31 PM, Frank Swiderski wrote:
On Tue, Jun 26, 2012 at 1:40 PM, Rik van Riel[off-list ref] wrote:
quoted
The code looks good to me, my only worry is the
code duplication. We now have 5 balloon drivers,
for 4 hypervisors, all implementing everything
from scratch...
Do you have any recommendations on this? I could (I think reasonably
so) modify the existing virtio_balloon.c and have it change behavior
based on a feature bit or other configuration. I'm not sure that
really addresses the root of what you're pointing out--it's still
adding a different implementation, but doing so as an extension of an
existing one.
Ideally, I believe we would have two balloon
top parts in a guest (one classical balloon,
one on the LRU), and four bottom parts (kvm,
xen, vmware & s390).
That way the virt specific bits of a balloon
driver would be essentially a ->balloon_page
and ->release_page callback for pages, as well
as methods to communicate with the host.
All the management of pages, including stuff
like putting them on the LRU, or isolating
them for migration, would be done with the
same common code, regardless of what virt
software we are running on.
Of course, that is a substantial amount of
work and I feel it would be unreasonable to
block anyone's code on that kind of thing
(especially considering that your code is good),
but I do believe the explosion of balloon
code is a little worrying.
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2012-06-26 21:47:11
On Tue, Jun 26, 2012 at 02:31:26PM -0700, Frank Swiderski wrote:
On Tue, Jun 26, 2012 at 1:40 PM, Rik van Riel [off-list ref] wrote:
quoted
On 06/26/2012 04:32 PM, Frank Swiderski wrote:
quoted
This implementation of a virtio balloon driver uses the page cache to
"store" pages that have been released to the host. The communication
(outside of target counts) is one way--the guest notifies the host when
it adds a page to the page cache, allowing the host to madvise(2) with
MADV_DONTNEED. Reclaim in the guest is therefore automatic and implicit
(via the regular page reclaim). This means that inflating the balloon
is similar to the existing balloon mechanism, but the deflate is
different--it re-uses existing Linux kernel functionality to
automatically reclaim.
Signed-off-by: Frank Swiderski<redacted>
It is a great idea, but how can this memory balancing
possibly work if someone uses memory cgroups inside a
guest?
Thanks and good point--this isn't something that I considered in the
implementation.
quoted
Having said that, we currently do not have proper
memory reclaim balancing between cgroups at all, so
requiring that of this balloon driver would be
unreasonable.
The code looks good to me, my only worry is the
code duplication. We now have 5 balloon drivers,
for 4 hypervisors, all implementing everything
from scratch...
Do you have any recommendations on this? I could (I think reasonably
so) modify the existing virtio_balloon.c and have it change behavior
based on a feature bit or other configuration. I'm not sure that
really addresses the root of what you're pointing out--it's still
adding a different implementation, but doing so as an extension of an
existing one.
fes
Let's assume it's a feature bit: how would you
formulate what the feature does *from host point of view*?
--
MST
From: Frank Swiderski <hidden> Date: 2012-06-26 23:22:03
On Tue, Jun 26, 2012 at 2:47 PM, Michael S. Tsirkin [off-list ref] wrote:
On Tue, Jun 26, 2012 at 02:31:26PM -0700, Frank Swiderski wrote:
quoted
On Tue, Jun 26, 2012 at 1:40 PM, Rik van Riel [off-list ref] wrote:
quoted
On 06/26/2012 04:32 PM, Frank Swiderski wrote:
quoted
This implementation of a virtio balloon driver uses the page cache to
"store" pages that have been released to the host. The communication
(outside of target counts) is one way--the guest notifies the host when
it adds a page to the page cache, allowing the host to madvise(2) with
MADV_DONTNEED. Reclaim in the guest is therefore automatic and implicit
(via the regular page reclaim). This means that inflating the balloon
is similar to the existing balloon mechanism, but the deflate is
different--it re-uses existing Linux kernel functionality to
automatically reclaim.
Signed-off-by: Frank Swiderski<redacted>
It is a great idea, but how can this memory balancing
possibly work if someone uses memory cgroups inside a
guest?
Thanks and good point--this isn't something that I considered in the
implementation.
quoted
Having said that, we currently do not have proper
memory reclaim balancing between cgroups at all, so
requiring that of this balloon driver would be
unreasonable.
The code looks good to me, my only worry is the
code duplication. We now have 5 balloon drivers,
for 4 hypervisors, all implementing everything
from scratch...
Do you have any recommendations on this? I could (I think reasonably
so) modify the existing virtio_balloon.c and have it change behavior
based on a feature bit or other configuration. I'm not sure that
really addresses the root of what you're pointing out--it's still
adding a different implementation, but doing so as an extension of an
existing one.
fes
Let's assume it's a feature bit: how would you
formulate what the feature does *from host point of view*?
--
MST
In this implementation, the host doesn't keep track of pages in the
balloon, as there is no explicit deflate path. The host device for
this implementation should merely, for example, MADV_DONTNEED on the
pages sent in an inflate. Thus, the inflate becomes a notification
that the guest doesn't need those pages mapped in, but that they
should be available if the guest touches them. In that sense, it's
not a rigid shrink of guest memory. I'm not sure what I'd call the
feature bit though.
Was that the question you were asking, or did I misread?
fes
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2012-06-27 09:02:11
On Tue, Jun 26, 2012 at 04:21:58PM -0700, Frank Swiderski wrote:
On Tue, Jun 26, 2012 at 2:47 PM, Michael S. Tsirkin [off-list ref] wrote:
quoted
On Tue, Jun 26, 2012 at 02:31:26PM -0700, Frank Swiderski wrote:
quoted
On Tue, Jun 26, 2012 at 1:40 PM, Rik van Riel [off-list ref] wrote:
quoted
On 06/26/2012 04:32 PM, Frank Swiderski wrote:
quoted
This implementation of a virtio balloon driver uses the page cache to
"store" pages that have been released to the host. The communication
(outside of target counts) is one way--the guest notifies the host when
it adds a page to the page cache, allowing the host to madvise(2) with
MADV_DONTNEED. Reclaim in the guest is therefore automatic and implicit
(via the regular page reclaim). This means that inflating the balloon
is similar to the existing balloon mechanism, but the deflate is
different--it re-uses existing Linux kernel functionality to
automatically reclaim.
Signed-off-by: Frank Swiderski<redacted>
It is a great idea, but how can this memory balancing
possibly work if someone uses memory cgroups inside a
guest?
Thanks and good point--this isn't something that I considered in the
implementation.
quoted
Having said that, we currently do not have proper
memory reclaim balancing between cgroups at all, so
requiring that of this balloon driver would be
unreasonable.
The code looks good to me, my only worry is the
code duplication. We now have 5 balloon drivers,
for 4 hypervisors, all implementing everything
from scratch...
Do you have any recommendations on this? I could (I think reasonably
so) modify the existing virtio_balloon.c and have it change behavior
based on a feature bit or other configuration. I'm not sure that
really addresses the root of what you're pointing out--it's still
adding a different implementation, but doing so as an extension of an
existing one.
fes
Let's assume it's a feature bit: how would you
formulate what the feature does *from host point of view*?
--
MST
In this implementation, the host doesn't keep track of pages in the
balloon, as there is no explicit deflate path. The host device for
this implementation should merely, for example, MADV_DONTNEED on the
pages sent in an inflate. Thus, the inflate becomes a notification
that the guest doesn't need those pages mapped in, but that they
should be available if the guest touches them.
So guest access removes the page from the balloon,
since it cancels MADV_DONTNEED, right?
Okay. But what is the meaning of num_pages then?
For example, let's assume I set num_pages to 1,
then guest gives me a page and later accesses this
page. Is guest also required to give me another
page now? Later I send a config interrupt without
changing num_pages. Is guest required to give me another
page now?
In that sense, it's
not a rigid shrink of guest memory. I'm not sure what I'd call the
feature bit though.
Was that the question you were asking, or did I misread?
fes
Yes. It would be a good idea for you to try and write a spec IMO.
Send a patch to virtio.lyx
--
MST
From: Rusty Russell <hidden> Date: 2012-07-02 01:52:55
On Tue, 26 Jun 2012 16:21:58 -0700, Frank Swiderski [off-list ref] wrote:
On Tue, Jun 26, 2012 at 2:47 PM, Michael S. Tsirkin [off-list ref] wrote:
quoted
Let's assume it's a feature bit: how would you
formulate what the feature does *from host point of view*?
In this implementation, the host doesn't keep track of pages in the
balloon, as there is no explicit deflate path. The host device for
this implementation should merely, for example, MADV_DONTNEED on the
pages sent in an inflate. Thus, the inflate becomes a notification
that the guest doesn't need those pages mapped in, but that they
should be available if the guest touches them. In that sense, it's
not a rigid shrink of guest memory. I'm not sure what I'd call the
feature bit though.
Was that the question you were asking, or did I misread?
Hmm, the spec is unfortunately vague: !VIRTIO_BALLOON_F_MUST_TELL_HOST
implies you should tell the host (eventually). I don't know if any
implementations actually care though.
We could add a VIRTIO_BALLOON_F_NEVER_TELL_DEFLATE which would mean the
deflate vq need not be used at all.
Is it altogether impossible to know when a page is reused in your
implementation? If we could do that, we could replace our balloon with
this one.
(My deep ignorance of vm issues is hurting us here, sorry.)
Cheers,
Rusty.
From: Paolo Bonzini <pbonzini@redhat.com> Date: 2012-09-03 06:35:32
Il 02/07/2012 02:29, Rusty Russell ha scritto:
VIRTIO_BALLOON_F_MUST_TELL_HOST
implies you should tell the host (eventually). I don't know if any
implementations actually care though.
This is indeed broken, because it is a "negative" feature: it tells you
that "implicit deflate" is _not_ supported.
Right now, QEMU refuses migration if the target does not support all the
features that were negotiated. But then:
- a migration from non-MUST_TELL_HOST to MUST_TELL_HOST will succeed,
which is wrong;
- a migration from MUST_TELL_HOST to non-MUST_TELL_HOST will fail, which
is useless.
We could add a VIRTIO_BALLOON_F_NEVER_TELL_DEFLATE which would mean the
deflate vq need not be used at all.
That would work. At the same time we could deprecate MUST_TELL_HOST.
Certainly the guest implementations don't care, or we would have
experienced problems such as the one above. The QEMU implementation
also does not care but, for example, a Xen implementation would care.
Paolo
From: Rusty Russell <hidden> Date: 2012-09-06 03:03:19
Paolo Bonzini [off-list ref] writes:
Il 02/07/2012 02:29, Rusty Russell ha scritto:
quoted
VIRTIO_BALLOON_F_MUST_TELL_HOST
implies you should tell the host (eventually). I don't know if any
implementations actually care though.
This is indeed broken, because it is a "negative" feature: it tells you
that "implicit deflate" is _not_ supported.
Right now, QEMU refuses migration if the target does not support all the
features that were negotiated. But then:
- a migration from non-MUST_TELL_HOST to MUST_TELL_HOST will succeed,
which is wrong;
- a migration from MUST_TELL_HOST to non-MUST_TELL_HOST will fail, which
is useless.
quoted
We could add a VIRTIO_BALLOON_F_NEVER_TELL_DEFLATE which would mean the
deflate vq need not be used at all.
That would work. At the same time we could deprecate MUST_TELL_HOST.
Certainly the guest implementations don't care, or we would have
experienced problems such as the one above. The QEMU implementation
also does not care but, for example, a Xen implementation would care.
OK; I'm not sure we need to deprecate MUST_TELL_HOST, though since it's
never actually been used there's a good argument.
VIRTIO_BALLOON_F_SILENT_DEFLATE (or whatever it's called) would
obviously mean you couldn't ack VIRTIO_BALLOON_F_MUST_TELL_HOST.
Patches welcome!
Cheers,
Rusty.