On Sat, Oct 17, 2020 at 6:34 AM Colm MacCarthaigh [off-list ref] wrote:
On 16 Oct 2020, at 21:02, Jann Horn wrote:
quoted
On Sat, Oct 17, 2020 at 5:36 AM Willy Tarreau [off-list ref] wrote:
But in userspace, we just need a simple counter. There's no need for
us to worry about anything else, like timestamps or whatever. If we
repeatedly fork a paused VM, the forked VMs will see the same counter
value, but that's totally fine, because the only thing that matters to
userspace is that the counter changes when the VM is forked.
For user-space, even a single bit would do. We added MADVISE_WIPEONFORK
so that userspace libraries can detect fork()/clone() robustly, for the
same reasons. It just wipes a page as the indicator, which is
effectively a single-bit signal, and it works well. On the user-space
side of this, I’m keen to find a solution like that that we can use
fairly easily inside of portable libraries and applications. The “have
I forked” checks do end up in hot paths, so it’s nice if they can be
CPU cache friendly. Comparing a whole 128-bit value wouldn’t be my
favorite.
I'm pretty sure a single bit is not enough if you want to have a
single page, shared across the entire system, that stores the VM
forking state; you need a counter for that.
quoted
And actually, since the value is a cryptographically random 128-bit
value, I think that we should definitely use it to help reseed the
kernel's RNG, and keep it secret from userspace. That way, even if the
VM image is public, we can ensure that going forward, the kernel RNG
will return securely random data.
If the image is public, you need some extra new raw entropy from
somewhere. The gen-id could be mixed in, that can’t do any harm as
long as rigorous cryptographic mixing with the prior state is used, but
if that’s all you do then the final state is still deterministic and
non-secret.
Microsoft's documentation
(http://go.microsoft.com/fwlink/?LinkId=260709) says that the VM
Generation ID that we get after a fork "is a 128-bit,
cryptographically random integer value". If multiple people use the
same image, it guarantees that each use of the image gets its own,
fresh ID: The table in section "How to implement virtual machine
generation ID support in a virtualization platform" says that (among
other things) "Virtual machine is imported, copied, or cloned"
generates a new generation ID.
So the RNG state after mixing in the new VM Generation ID would
contain 128 bits of secret entropy not known to anyone else, including
people with access to the VM image.
Now, 128 bits of cryptographically random data aren't _optimal_; I
think something on the order of 256 bits would be nicer from a
theoretical standpoint. But in practice I think we'll be good with the
128 bits we're getting (since the number of users who fork a VM image
is probably not going to be so large that worst-case collision
probabilities matter).
The kernel would need to use the change as a trigger to
measure some entropy (e.g. interrupts and RDRAND, or whatever). Our just
define the machine contract as “this has to be unique random data and
if it’s not unique, or if it’s pubic, you’re toast”.
As far as I can tell from Microsoft's spec, that is a guarantee we're
already getting.
On Sat, Oct 17, 2020 at 6:34 AM Colm MacCarthaigh
[off-list ref] wrote:
quoted
For user-space, even a single bit would do. We added
MADVISE_WIPEONFORK
so that userspace libraries can detect fork()/clone() robustly, for
the
same reasons. It just wipes a page as the indicator, which is
effectively a single-bit signal, and it works well. On the user-space
side of this, I’m keen to find a solution like that that we can use
fairly easily inside of portable libraries and applications. The
“have
I forked” checks do end up in hot paths, so it’s nice if they can
be
CPU cache friendly. Comparing a whole 128-bit value wouldn’t be my
favorite.
I'm pretty sure a single bit is not enough if you want to have a
single page, shared across the entire system, that stores the VM
forking state; you need a counter for that.
You’re right. WIPEONFORK is more like a single-bit per use. If it’s
something system wide then a counter is better.
So the RNG state after mixing in the new VM Generation ID would
contain 128 bits of secret entropy not known to anyone else, including
people with access to the VM image.
Now, 128 bits of cryptographically random data aren't _optimal_; I
think something on the order of 256 bits would be nicer from a
theoretical standpoint. But in practice I think we'll be good with the
128 bits we're getting (since the number of users who fork a VM image
is probably not going to be so large that worst-case collision
probabilities matter).
This reminds me on key/IV usage limits for AES encryption, where the
same birthday bounds apply, and even though 256-bits would be better, we
routinely make 128-bit birthday bounds work for massively scalable
systems.
quoted
The kernel would need to use the change as a trigger to
measure some entropy (e.g. interrupts and RDRAND, or whatever). Our
just
define the machine contract as “this has to be unique random data
and
if it’s not unique, or if it’s pubic, you’re toast”.
As far as I can tell from Microsoft's spec, that is a guarantee we're
already getting.
On Sat, Oct 17, 2020 at 07:01:31AM +0200, Jann Horn wrote:
Microsoft's documentation
(http://go.microsoft.com/fwlink/?LinkId=260709) says that the VM
Generation ID that we get after a fork "is a 128-bit,
cryptographically random integer value". If multiple people use the
same image, it guarantees that each use of the image gets its own,
fresh ID:
No. It cannot be more unique than the source that feeds that cryptographic
transformation. All it guarantees is that the entropy source is protected
from being guessed based on the output. Applying cryptography on a simple
counter provides apparently random numbers that will be unique for a long
period for the same source, but as soon as you duplicate that code between
users and they start from the same counter they'll get the same IDs.
This is why I think that using a counter is better if you really need something
unique. Randoms only reduce predictability which helps avoiding collisions.
And I'm saying this as someone who had on his external gateway the same SSH
host key as 89 other hosts on the net, each of them using randoms to provide
a universally unique one...
Willy
On Sat, Oct 17, 2020 at 7:37 AM Willy Tarreau [off-list ref] wrote:
On Sat, Oct 17, 2020 at 07:01:31AM +0200, Jann Horn wrote:
quoted
Microsoft's documentation
(http://go.microsoft.com/fwlink/?LinkId=260709) says that the VM
Generation ID that we get after a fork "is a 128-bit,
cryptographically random integer value". If multiple people use the
same image, it guarantees that each use of the image gets its own,
fresh ID:
No. It cannot be more unique than the source that feeds that cryptographic
transformation. All it guarantees is that the entropy source is protected
from being guessed based on the output. Applying cryptography on a simple
counter provides apparently random numbers that will be unique for a long
period for the same source, but as soon as you duplicate that code between
users and they start from the same counter they'll get the same IDs.
This is why I think that using a counter is better if you really need something
unique. Randoms only reduce predictability which helps avoiding collisions.
Microsoft's spec tells us that they're giving us cryptographically
random numbers. Where they're getting those from is not our problem.
(And if even the hypervisor is not able to collect enough entropy to
securely generate random numbers, worrying about RNG reseeding in the
guest would be kinda pointless, we'd be fairly screwed anyway.)
Also note that we don't actually need to *always* reinitialize RNG
state on forks for functional correctness; it is fine if that fails
with a probability of 2^-128, because functionally everything will be
fine, and an attacker who is that lucky could also just guess an AES
key (which has the same probability of being successful). (And also
2^-128 is such a tiny number that it doesn't matter anyway.)
And I'm saying this as someone who had on his external gateway the same SSH
host key as 89 other hosts on the net, each of them using randoms to provide
a universally unique one...
If your SSH host key was shared with 89 other hosts, it evidently
wasn't generated from cryptographically random numbers. :P Either
because the key generator was not properly hooked up to the system's
entropy pool (if you're talking about the Debian fiasco), or because
the system simply did not have enough entropy available. (Or because
the key generator is broken, but I don't think that ever happened with
OpenSSH?)
On Sat, Oct 17, 2020 at 07:52:48AM +0200, Jann Horn wrote:
On Sat, Oct 17, 2020 at 7:37 AM Willy Tarreau [off-list ref] wrote:
quoted
On Sat, Oct 17, 2020 at 07:01:31AM +0200, Jann Horn wrote:
quoted
Microsoft's documentation
(http://go.microsoft.com/fwlink/?LinkId=260709) says that the VM
Generation ID that we get after a fork "is a 128-bit,
cryptographically random integer value". If multiple people use the
same image, it guarantees that each use of the image gets its own,
fresh ID:
No. It cannot be more unique than the source that feeds that cryptographic
transformation. All it guarantees is that the entropy source is protected
from being guessed based on the output. Applying cryptography on a simple
counter provides apparently random numbers that will be unique for a long
period for the same source, but as soon as you duplicate that code between
users and they start from the same counter they'll get the same IDs.
This is why I think that using a counter is better if you really need something
unique. Randoms only reduce predictability which helps avoiding collisions.
Microsoft's spec tells us that they're giving us cryptographically
random numbers. Where they're getting those from is not our problem.
(And if even the hypervisor is not able to collect enough entropy to
securely generate random numbers, worrying about RNG reseeding in the
guest would be kinda pointless, we'd be fairly screwed anyway.)
Sorry if I sound annoying, but it's a matter of terminology and needs.
Cryptograhically random means safe for use with cryptography in that it
is unguessable enough so that you can use it for encryption keys that
nobody will be able to guess. It in no ways guarantees uniqueness, just
like you don't really care if the symmetric crypto key of you VPN has
already been used once somewhere else as long as there's no way to know.
However with the good enough distribution that a CSPRNG provides,
collisions within a *same* generator are bound to a very low, predictable
rate which is by generally considered as acceptable for all use cases.
Something random (cryptographically or not) *cannot* be unique by
definition, otherwise it's not random anymore, since each draw has an
influence on the remaining list of possible draws, which is contrary to
randomness. And conversely something unique cannot be completely random
because if you know it's unique, you can already rule out all other known
values from the candidates, thus it's more predictable than random.
With this in mind, picking randoms from a same RNG is often highly
sufficient to consider they're highly likely unique within a long
period. But it's not a guarantee. And it's even less one between two
RNGs (e.g. if uniqueness is required between multiple hypervisors in
case VMs are migrated or centrally managed, which I don't know).
If what is sought here is a strong guarantee of uniqueness, using a
counter as you first suggested is better. If what is sought is pure
randomness (in the sense that it's unpredictable, which I don't think
is needed here), then randoms are better. If both are required, just
concatenate a counter and a random. And if you need them to be spatially
unique, just include a node identifier.
Now the initial needs in the forwarded message are not entirely clear
to me but I wanted to rule out the apparent mismatch between the expressed
needs for uniqueness and the proposed solutions solely based on randomness.
Cheers,
Willy
On Sat, Oct 17, 2020 at 8:44 AM Willy Tarreau [off-list ref] wrote:
On Sat, Oct 17, 2020 at 07:52:48AM +0200, Jann Horn wrote:
quoted
On Sat, Oct 17, 2020 at 7:37 AM Willy Tarreau [off-list ref] wrote:
quoted
On Sat, Oct 17, 2020 at 07:01:31AM +0200, Jann Horn wrote:
quoted
Microsoft's documentation
(http://go.microsoft.com/fwlink/?LinkId=260709) says that the VM
Generation ID that we get after a fork "is a 128-bit,
cryptographically random integer value". If multiple people use the
same image, it guarantees that each use of the image gets its own,
fresh ID:
No. It cannot be more unique than the source that feeds that cryptographic
transformation. All it guarantees is that the entropy source is protected
from being guessed based on the output. Applying cryptography on a simple
counter provides apparently random numbers that will be unique for a long
period for the same source, but as soon as you duplicate that code between
users and they start from the same counter they'll get the same IDs.
This is why I think that using a counter is better if you really need something
unique. Randoms only reduce predictability which helps avoiding collisions.
Microsoft's spec tells us that they're giving us cryptographically
random numbers. Where they're getting those from is not our problem.
(And if even the hypervisor is not able to collect enough entropy to
securely generate random numbers, worrying about RNG reseeding in the
guest would be kinda pointless, we'd be fairly screwed anyway.)
Sorry if I sound annoying, but it's a matter of terminology and needs.
Cryptograhically random means safe for use with cryptography in that it
is unguessable enough so that you can use it for encryption keys that
nobody will be able to guess. It in no ways guarantees uniqueness, just
like you don't really care if the symmetric crypto key of you VPN has
already been used once somewhere else as long as there's no way to know.
However with the good enough distribution that a CSPRNG provides,
collisions within a *same* generator are bound to a very low, predictable
rate which is by generally considered as acceptable for all use cases.
Yes.
Something random (cryptographically or not) *cannot* be unique by
definition, otherwise it's not random anymore, since each draw has an
influence on the remaining list of possible draws, which is contrary to
randomness. And conversely something unique cannot be completely random
because if you know it's unique, you can already rule out all other known
values from the candidates, thus it's more predictable than random.
Yes.
With this in mind, picking randoms from a same RNG is often highly
sufficient to consider they're highly likely unique within a long
period. But it's not a guarantee. And it's even less one between two
RNGs (e.g. if uniqueness is required between multiple hypervisors in
case VMs are migrated or centrally managed, which I don't know).
If what is sought here is a strong guarantee of uniqueness, using a
counter as you first suggested is better.
My suggestion is to use a counter *in the UAPI*, not in the hypervisor
protocol. (And as long as that counter can only miss increments in a
cryptographically negligible fraction of cases, everything's fine.)
If what is sought is pure
randomness (in the sense that it's unpredictable, which I don't think
is needed here), then randoms are better.
And this is what *the hypervisor protocol* gives us (which could be
very useful for reseeding the kernel RNG).
If both are required, just
concatenate a counter and a random. And if you need them to be spatially
unique, just include a node identifier.
Now the initial needs in the forwarded message are not entirely clear
to me but I wanted to rule out the apparent mismatch between the expressed
needs for uniqueness and the proposed solutions solely based on randomness.
Sure, from a theoretical standpoint, it would be a little bit nicer if
the hypervisor protocol included a generation number along with the
128-bit random value. But AFAIU it doesn't, so if we want this to just
work under Microsoft's existing hypervisor, we'll have to make do with
checking whether the random value changed. :P
On Sat, Oct 17, 2020 at 08:55:34AM +0200, Jann Horn wrote:
My suggestion is to use a counter *in the UAPI*, not in the hypervisor
protocol. (And as long as that counter can only miss increments in a
cryptographically negligible fraction of cases, everything's fine.)
OK I got it now and I agree.
quoted
If what is sought is pure
randomness (in the sense that it's unpredictable, which I don't think
is needed here), then randoms are better.
And this is what *the hypervisor protocol* gives us (which could be
very useful for reseeding the kernel RNG).
As an external source, yes very likely, as long as it's not trivially
observable by everyone under the same hypervisor :-)
quoted
Now the initial needs in the forwarded message are not entirely clear
to me but I wanted to rule out the apparent mismatch between the expressed
needs for uniqueness and the proposed solutions solely based on randomness.
Sure, from a theoretical standpoint, it would be a little bit nicer if
the hypervisor protocol included a generation number along with the
128-bit random value. But AFAIU it doesn't, so if we want this to just
work under Microsoft's existing hypervisor, we'll have to make do with
checking whether the random value changed. :P
From: "Jason A. Donenfeld" <Jason@zx2c4.com> Date: 2020-10-17 13:24:29
After discussing this offline with Jann a bit, I have a few general
comments on the design of this.
First, the UUID communicated by the hypervisor should be consumed by
the kernel -- added as another input to the rng -- and then userspace
should be notified that it should reseed any userspace RNGs that it
may have, without actually communicating that UUID to userspace. IOW,
I agree with Jann there. Then, it's the functioning of this
notification mechanism to userspace that is interesting to me.
There are a few design goals of notifying userspace: it should be
fast, because people who are using userspace RNGs are usually doing so
in the first place to completely avoid syscall overhead for whatever
high performance application they have - e.g. I recall conversations
with Colm about his TLS implementation needing to make random IVs
_really_ fast. It should also happen as early as possible, with no
race or as minimal as possible race window, so that userspace doesn't
begin using old randomness and then switch over after the damage is
already done.
I'm also not wedded to using Microsoft's proprietary hypervisor design
for this. If we come up with a better interface, I don't think it's
asking too much to implement that and reasonably expect for Microsoft
to catch up. Maybe someone here will find that controversial, but
whatever -- discussing ideal designs does not seem out of place or
inappropriate for how we usually approach things in the kernel, and a
closed source hypervisor coming along shouldn't disrupt that.
So, anyway, here are a few options with some pros and cons for the
kernel notifying userspace that its RNG should reseed.
1. SIGRND - a new signal. Lol.
2. Userspace opens a file descriptor that it can epoll on. Pros are
that many notification mechanisms already use this. Cons is that this
requires syscall and might be more racy than we want. Another con is
that this a new thing for userspace programs to do.
3. We stick an atomic counter in the vDSO, Jann's suggestion. Pros are
that this is extremely fast, and also simple to use and implement.
There are enough sequence points in typical crypto programs that
checking to see whether this counter has changed before doing whatever
operation seems easy enough. Cons are that typically we've been
conservative about adding things to the vDSO, and this is also a new
thing for userspace programs to do.
4. We already have a mechanism for this kind of thing, because the
same issue comes up when fork()ing. The solution was MADV_WIPEONFORK,
where userspace marks a page to be zeroed when forking, for the
purposes of the RNG being notified when its world gets split in two.
This is basically the same thing as we're discussing here with guest
snapshots, except it's on the system level rather than the process
level, and a system has many processes. But the problem space is still
almost the same, and we could simply reuse that same mechanism. There
are a few implementation strategies for that:
4a. We mess with the PTEs of all processes' pages that are
MADV_WIPEONFORK, like fork does now, when the hypervisor notifies us
to do so. Then we wind up reusing the already existing logic for
userspace RNGs. Cons might be that this usually requires semaphores,
and we're in irq context, so we'd have to hoist to a workqueue, which
means either more wake up latency, or a larger race window.
4b. We just memzero all processes' pages that are MADV_WIPEONFORK,
when the hypervisor notifies us to do so. Then we wind up reusing the
already existing logic for userspace RNGs.
4c. The guest kernel maintains an array of physical addresses that are
MADV_WIPEONFORK. The hypervisor knows about this array and its
location through whatever protocol, and before resuming a
moved/snapshotted/duplicated VM, it takes the responsibility for
memzeroing this memory. The huge pro here would be that this
eliminates all races, and reduces complexity quite a bit, because the
hypervisor can perfectly synchronize its bringup (and SMP bringup)
with this, and it can even optimize things like on-disk memory
snapshots to simply not write out those pages to disk.
A 4c-like approach seems like it'd be a lot of bang for the buck -- we
reuse the existing mechanism (MADV_WIPEONFORK), so there's no new
userspace API to deal with, and it'd be race free, and eliminate a lot
of kernel complexity.
But 4b and 3 don't seem too bad either.
Any thoughts on 4c? Is that utterly insane, or does that actually get
us somewhere close to what we want?
Jason
From: Catangiu, Adrian Costin <hidden> Date: 2020-10-17 18:06:39
After discussing this offline with Jann a bit, I have a few general
comments on the design of this.
First, the UUID communicated by the hypervisor should be consumed by
the kernel -- added as another input to the rng -- and then userspace
should be notified that it should reseed any userspace RNGs that it
may have, without actually communicating that UUID to userspace. IOW,
I agree with Jann there. Then, it's the functioning of this
notification mechanism to userspace that is interesting to me.
Agreed! The UUID/vmgenid is the glue to the hypervisor to be able to find
out about VM snapshots/forks. The really interesting (and important) topic
here is finding the right notification mechanism to userspace.
...In retrospect, I should have posted this as RFC instead of PATCH.
So, anyway, here are a few options with some pros and cons for the
kernel notifying userspace that its RNG should reseed.
1. SIGRND - a new signal. Lol.
2. Userspace opens a file descriptor that it can epoll on. Pros are
that many notification mechanisms already use this. Cons is that this
requires syscall and might be more racy than we want. Another con is
that this a new thing for userspace programs to do.
3. We stick an atomic counter in the vDSO, Jann's suggestion. Pros are
that this is extremely fast, and also simple to use and implement.
There are enough sequence points in typical crypto programs that
checking to see whether this counter has changed before doing whatever
operation seems easy enough. Cons are that typically we've been
conservative about adding things to the vDSO, and this is also a new
thing for userspace programs to do.
For each 1, 2, and 3 options, userspace programs _have to do smth new_
anyway, so I wouldn't weigh that as a con.
An atomic counter in the vDSO looks like the most bang for the buck to me.
I'm really curious to hear more opinions on why we shouldn't do it.
4. We already have a mechanism for this kind of thing, because the
same issue comes up when fork()ing. The solution was MADV_WIPEONFORK,
where userspace marks a page to be zeroed when forking, for the
purposes of the RNG being notified when its world gets split in two.
This is basically the same thing as we're discussing here with guest
snapshots, except it's on the system level rather than the process
level, and a system has many processes. But the problem space is still
almost the same, and we could simply reuse that same mechanism. There
are a few implementation strategies for that:
I don't think we can piggy back on MADV_WIPEONFORK. That madvise flag
has a clear contract of only wiping _on fork_. Overloading it with wiping
on VM-fork - while process doesn't fork - might break some of its users.
4a. We mess with the PTEs of all processes' pages that are
MADV_WIPEONFORK, like fork does now, when the hypervisor notifies us
to do so. Then we wind up reusing the already existing logic for
userspace RNGs. Cons might be that this usually requires semaphores,
and we're in irq context, so we'd have to hoist to a workqueue, which
means either more wake up latency, or a larger race window.
4b. We just memzero all processes' pages that are MADV_WIPEONFORK,
when the hypervisor notifies us to do so. Then we wind up reusing the
already existing logic for userspace RNGs.
4c. The guest kernel maintains an array of physical addresses that are
MADV_WIPEONFORK. The hypervisor knows about this array and its
location through whatever protocol, and before resuming a
moved/snapshotted/duplicated VM, it takes the responsibility for
memzeroing this memory. The huge pro here would be that this
eliminates all races, and reduces complexity quite a bit, because the
hypervisor can perfectly synchronize its bringup (and SMP bringup)
with this, and it can even optimize things like on-disk memory
snapshots to simply not write out those pages to disk.
I've previously proposed a path similar (in concept at least) with a combination
of 4 a,b and c - https://lwn.net/ml/linux-mm/B7793B7A-3660-4769-9B9A-FFCF250728BB@amazon.com/
without reusing MADV_WIPEONFORK, but by adding a dedicated
MADV_WIPEONSUSPEND.
That proposal was clunky however with many people raising concerns around
how the interface is too subtle and hard to work with.
A vmgenid driver offering a clean FS interface seemed cleaner, although, like
some of you observed, it still allows a window of time between actual VM fork
and userspace handling of the event.
One other direction that I would like to explore and I feel it’s similar to your 4c
proposal is to do smth like:
"mm: extend memfd with ability to create 'secret' memory"
https://patchwork.kernel.org/project/linux-mm/patch/20200130162340.GA14232@rapoport-lnx/
Maybe we can combine ideas from the two patches in smth like: instead of libs
using anon mem with MADV_WIPEONSUSPEND, they can use a secret_mem_fd
with a (secretmem specific) WIPEONSUSPEND flag; then instead of crawling
PTEs in the core PM code looking for MADV_WIPEONSUSPEND mappings,
we can register this secretmem device to wipe its own secrets during a pm callback.
From a crypto safety pov, the ideal solution would be one where wiping happens
before or during VM forks, not after.
Having said that, I think a vDSO (updated by the guest kernel _after_ VM fork) still
closes that gap enough to be practically safe.
Thanks,
Adrian.
Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
From: Alexander Graf <hidden> Date: 2020-10-17 18:09:32
Hi Jason,
On 17.10.20 15:24, Jason A. Donenfeld wrote:
After discussing this offline with Jann a bit, I have a few general
comments on the design of this.
First, the UUID communicated by the hypervisor should be consumed by
the kernel -- added as another input to the rng -- and then userspace
We definitely want a kernel internal notifier as well, yes :).
should be notified that it should reseed any userspace RNGs that it
may have, without actually communicating that UUID to userspace. IOW,
I also tend to agree that it makes sense to disconnect the actual UUID
we receive from the notification to user space. This would allow us to
create a generic mechanism for VM save/restore cycles across different
hypervisors. Let me add PPC and s390x people to the CC list to see
whether they have anything remotely similar to the VmGenID mechanism.
For x86 and aarch64, the ACPI and memory based VmGenID implemented here
is the most obvious option to implement IMHO. It's also already
implemented in all major hypervisors.
I agree with Jann there. Then, it's the functioning of this
notification mechanism to userspace that is interesting to me.
There are a few design goals of notifying userspace: it should be
fast, because people who are using userspace RNGs are usually doing so
in the first place to completely avoid syscall overhead for whatever
high performance application they have - e.g. I recall conversations
with Colm about his TLS implementation needing to make random IVs
_really_ fast. It should also happen as early as possible, with no
race or as minimal as possible race window, so that userspace doesn't
begin using old randomness and then switch over after the damage is
already done.
There are multiple facets and different types of consumers here. For a
user space RNG, I agree that fast and as race free as possible is key.
That's what the mmap interface is there for.
There are applications way beyond that though. What do you do with
applications that already consumed randomness? For example a cached pool
of SSL keys. Or a higher level language primitive that consumes
randomness and caches its seed somewhere in an internal data structure.
Or even worse: your system's host ssh key.
For those types of events, an mmap (or vDSO) interface does not work. We
need to actively allow user space applications to readjust to the new
environment - either internally (the language primitive case) or through
a system event, maybe even as systemd trigger (the ssh host key case).
To give everyone enough time before we consider a system as "updated to
the new environment", we have the callback logic with the "Orchestrator"
that can check whether all listeners to system wide updates confirms
they adjusted themselves.
That's what the rest of the logic is there for: A read+poll interface
and all of the orchestration logic. It's not for the user space RNG
case, it's for all of its downstream users.
I'm also not wedded to using Microsoft's proprietary hypervisor design
for this. If we come up with a better interface, I don't think it's
asking too much to implement that and reasonably expect for Microsoft
to catch up. Maybe someone here will find that controversial, but
whatever -- discussing ideal designs does not seem out of place or
inappropriate for how we usually approach things in the kernel, and a
closed source hypervisor coming along shouldn't disrupt that.
The main bonus point on this interface is that Hyper-V, VMware and QEMU
implement it already. It would be a very natural for into the ecosystem.
I agree though that we shouldn't have our user space interface
necessarily dictated by it: Other hypervisors may implement different
ways such as a simple edge IRQ that gets triggered whenever the VM gets
resumed.
So, anyway, here are a few options with some pros and cons for the
kernel notifying userspace that its RNG should reseed.
I can only stress again that we should not be laser focused on the RNG
case. In a lot of cases, data has already been generated by the RNG
before the snapshot and needs to be reinitialized after the snapshot. In
other cases such as system UUIDs, it's completely orthogonal to the RNG.
1. SIGRND - a new signal. Lol.
Doable, but a lot of plumbing in user space. It's also not necessarily a
good for for event notification in most user space applications.
2. Userspace opens a file descriptor that it can epoll on. Pros are
that many notification mechanisms already use this. Cons is that this
requires syscall and might be more racy than we want. Another con is
that this a new thing for userspace programs to do.
That's part of what this patch does, right? This patch implements
read+poll as well as mmap() for high speed reads.
3. We stick an atomic counter in the vDSO, Jann's suggestion. Pros are
that this is extremely fast, and also simple to use and implement.
There are enough sequence points in typical crypto programs that
checking to see whether this counter has changed before doing whatever
operation seems easy enough. Cons are that typically we've been
conservative about adding things to the vDSO, and this is also a new
thing for userspace programs to do.
The big con is that its use is going to be super limited to applications
that can be adapted to check their "vm generation" through a vDSO call /
read every time they consume data that may potentially need to be
regenerated.
This probably works for the pure RNG case. It falls apart for more
sophisticated things such as "redo my ssh host keys and restart the
service" or "regenerate my samba machine uuid".
4. We already have a mechanism for this kind of thing, because the
same issue comes up when fork()ing. The solution was MADV_WIPEONFORK,
where userspace marks a page to be zeroed when forking, for the
purposes of the RNG being notified when its world gets split in two.
This is basically the same thing as we're discussing here with guest
snapshots, except it's on the system level rather than the process
level, and a system has many processes. But the problem space is still
almost the same, and we could simply reuse that same mechanism. There
are a few implementation strategies for that:
Yup, that's where we started from :). And then we ran into resistance by
the mm people (on CC here). And then we looked at the problem more in
depth and checked what it would take to for example implement this for
user space RNGs in Java. It's ... more complicated than one may think at
first.
4a. We mess with the PTEs of all processes' pages that are
MADV_WIPEONFORK, like fork does now, when the hypervisor notifies us
to do so. Then we wind up reusing the already existing logic for
userspace RNGs. Cons might be that this usually requires semaphores,
and we're in irq context, so we'd have to hoist to a workqueue, which
means either more wake up latency, or a larger race window.
4b. We just memzero all processes' pages that are MADV_WIPEONFORK,
when the hypervisor notifies us to do so. Then we wind up reusing the
already existing logic for userspace RNGs.
4c. The guest kernel maintains an array of physical addresses that are
MADV_WIPEONFORK. The hypervisor knows about this array and its
location through whatever protocol, and before resuming a
moved/snapshotted/duplicated VM, it takes the responsibility for
memzeroing this memory. The huge pro here would be that this
eliminates all races, and reduces complexity quite a bit, because the
hypervisor can perfectly synchronize its bringup (and SMP bringup)
with this, and it can even optimize things like on-disk memory
snapshots to simply not write out those pages to disk.
A 4c-like approach seems like it'd be a lot of bang for the buck -- we
reuse the existing mechanism (MADV_WIPEONFORK), so there's no new
userspace API to deal with, and it'd be race free, and eliminate a lot
of kernel complexity.
But 4b and 3 don't seem too bad either.
Any thoughts on 4c? Is that utterly insane, or does that actually get
us somewhere close to what we want?
All of the options for "4" are possible and have an RFC out. Please
check out the discussion linked above :).
The problem with anything that relies on close loop reads (options 3+4)
is not going to work well with the more sophisticated use case of
derived data.
IMHO it will boil down to "both". We will need a high-speed interface
that with close-to-0 overhead tells you either the generation ID or
clears pages (options 3+4) as well as something that is bigger for
applications that can either intrinsically (sshd) or by system design
(Java) not adopt the mechanisms above easily.
That said, we need to start somewhere. I don't mind which angle we start
from. But this is a real world problem and one that will only become
more prevalent over time as VMs are used for more than only your
traditional enterprise hardware consolidation.
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
On Sat, Oct 17, 2020 at 8:09 PM Alexander Graf [off-list ref] wrote:
There are applications way beyond that though. What do you do with
applications that already consumed randomness? For example a cached pool
of SSL keys. Or a higher level language primitive that consumes
randomness and caches its seed somewhere in an internal data structure.
For deterministic protection, those would also have to poll some
memory location that tells them whether the VmGenID changed:
1. between reading entropy from their RNG pool and using it
2. between collecting data from external sources (user input, clock,
...) and encrypting it
and synchronously shoot down the connection if a change happened. If
e.g. an application inside the VM has an AES-GCM-encrypted TLS
connection and, directly after the VM is restored, triggers an
application-level timeout that sends some fixed message across the
connection, then the TLS library must guarantee that either the VM was
already committed to sending exactly that message before the VM was
forked or the message will be blocked. If we don't do that, an
attacker who captures both a single packet from the forked VM and
traffic from the old VM can decrypt the next message from the old VM
after the fork (because AES-GCM is like AES-CTR plus an authenticator,
and CTR means that when keystream reuse occurs and one of the
plaintexts is known, the attacker can simply recover the other
plaintext using XOR).
(Or maybe, in disaster failover environments, TLS 1.3 servers could
get away with rekeying the connection instead of shooting it down? Ask
your resident friendly cryptographer whether that would be secure, I
am not one.)
I don't think a mechanism based around asynchronously telling the
application and waiting for it to confirm the rotation at a later
point is going to cut it; we should have some hard semantics on when
an application needs to poll this value.
Or even worse: your system's host ssh key.
Mmmh... I think I normally would not want a VM to reset its host ssh
key after merely restoring a snapshot though? And more importantly,
Microsoft's docs say that they also change the VmGenID on disaster
failover. I think you very much wouldn't want your server to lose its
host key every time disaster failover happens. On the other hand,
after importing a public VM image, it might be a good idea.
I guess you could push that responsibility on the user, by adding an
option to the sshd_config that tells OpenSSH whether the host key
should be rotated on an ID change or not... but that still would not
be particularly pretty.
Ideally we would have the host tell us what type of events happened to
the VM, or something like that... or maybe even get the host VM
management software to ask the user whether they're importing a public
image... I really feel like with Microsoft's current protocol, we
don't get enough information to figure out what we should do about
private long-term authentication keys.
On 17 Oct 2020, at 6:24, Jason A. Donenfeld wrote:
There are a few design goals of notifying userspace: it should be
fast, because people who are using userspace RNGs are usually doing so
in the first place to completely avoid syscall overhead for whatever
high performance application they have - e.g. I recall conversations
with Colm about his TLS implementation needing to make random IVs
_really_ fast.
That’s our old friend TLS1.1 in CBC mode, which needs a random
explicit IV for every record sent. Speed is still a reason at the
margins in cases like that, but getrandom() is really fast. A stickier
problem is that getrandom() is not certified for use with every
compliance standard, and those often dictate precise use of some NIST
DRBG or NRBG construction. That keeps people proliferating user-space
RNGs even when speed isn’t as important.
It should also happen as early as possible, with no
race or as minimal as possible race window, so that userspace doesn't
begin using old randomness and then switch over after the damage is
already done.
+1 to this, and I’d add that anyone making VM snapshots that they plan
to restore from multiple times really needs to think this through top to
bottom. The system would likely need to be put in to some kind of
quiescent state when the snapshot is taken.
So, anyway, here are a few options with some pros and cons for the
kernel notifying userspace that its RNG should reseed.
1. SIGRND - a new signal. Lol.
2. Userspace opens a file descriptor that it can epoll on. Pros are
that many notification mechanisms already use this. Cons is that this
requires syscall and might be more racy than we want. Another con is
that this a new thing for userspace programs to do.
A library like OpenSSL or BoringSSL also has to account for running
inside a chroot, which also makes this hard.
Any thoughts on 4c? Is that utterly insane, or does that actually get
us somewhere close to what we want?
I still like 4c, and as a user-space crypto-person, and a VM person,
they have a lot of appeal. Alex and Adrian’s replies get into some of
the sufficiency challenge. But for user-space libraries like the *SSLs,
the JVMs, and other runtimes where RNGs show up, it could plug in easily
enough.
-
Colm
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2020-10-18 15:52:20
On Sat, Oct 17, 2020 at 03:24:08PM +0200, Jason A. Donenfeld wrote:
4c. The guest kernel maintains an array of physical addresses that are
MADV_WIPEONFORK. The hypervisor knows about this array and its
location through whatever protocol, and before resuming a
moved/snapshotted/duplicated VM, it takes the responsibility for
memzeroing this memory. The huge pro here would be that this
eliminates all races, and reduces complexity quite a bit, because the
hypervisor can perfectly synchronize its bringup (and SMP bringup)
with this, and it can even optimize things like on-disk memory
snapshots to simply not write out those pages to disk.
A 4c-like approach seems like it'd be a lot of bang for the buck -- we
reuse the existing mechanism (MADV_WIPEONFORK), so there's no new
userspace API to deal with, and it'd be race free, and eliminate a lot
of kernel complexity.
Clearly this has a chance to break applications, right?
If there's an app that uses this as a non-system-calls way
to find out whether there was a fork, it will break
when wipe triggers without a fork ...
For example, imagine:
MADV_WIPEONFORK
copy secret data to MADV_DONTFORK
fork
used to work, with this change it gets 0s instead of the secret data.
I am also not sure it's wise to expose each guest process
to the hypervisor like this. E.g. each process needs a
guest physical address of its own then. This is a finite resource.
The mmap interface proposed here is somewhat baroque, but it is
certainly simple to implement ...
--
MST
From: Andy Lutomirski <luto@kernel.org> Date: 2020-10-18 15:54:53
On Sun, Oct 18, 2020 at 8:52 AM Michael S. Tsirkin [off-list ref] wrote:
On Sat, Oct 17, 2020 at 03:24:08PM +0200, Jason A. Donenfeld wrote:
quoted
4c. The guest kernel maintains an array of physical addresses that are
MADV_WIPEONFORK. The hypervisor knows about this array and its
location through whatever protocol, and before resuming a
moved/snapshotted/duplicated VM, it takes the responsibility for
memzeroing this memory. The huge pro here would be that this
eliminates all races, and reduces complexity quite a bit, because the
hypervisor can perfectly synchronize its bringup (and SMP bringup)
with this, and it can even optimize things like on-disk memory
snapshots to simply not write out those pages to disk.
A 4c-like approach seems like it'd be a lot of bang for the buck -- we
reuse the existing mechanism (MADV_WIPEONFORK), so there's no new
userspace API to deal with, and it'd be race free, and eliminate a lot
of kernel complexity.
Clearly this has a chance to break applications, right?
If there's an app that uses this as a non-system-calls way
to find out whether there was a fork, it will break
when wipe triggers without a fork ...
For example, imagine:
MADV_WIPEONFORK
copy secret data to MADV_DONTFORK
fork
used to work, with this change it gets 0s instead of the secret data.
I am also not sure it's wise to expose each guest process
to the hypervisor like this. E.g. each process needs a
guest physical address of its own then. This is a finite resource.
The mmap interface proposed here is somewhat baroque, but it is
certainly simple to implement ...
Wipe of fork/vmgenid/whatever could end up being much more problematic
than it naively appears -- it could be wiped in the middle of a read.
Either the API needs to handle this cleanly, or we need something more
aggressive like signal-on-fork.
--Andy
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2020-10-18 15:59:52
On Sun, Oct 18, 2020 at 08:54:36AM -0700, Andy Lutomirski wrote:
On Sun, Oct 18, 2020 at 8:52 AM Michael S. Tsirkin [off-list ref] wrote:
quoted
On Sat, Oct 17, 2020 at 03:24:08PM +0200, Jason A. Donenfeld wrote:
quoted
4c. The guest kernel maintains an array of physical addresses that are
MADV_WIPEONFORK. The hypervisor knows about this array and its
location through whatever protocol, and before resuming a
moved/snapshotted/duplicated VM, it takes the responsibility for
memzeroing this memory. The huge pro here would be that this
eliminates all races, and reduces complexity quite a bit, because the
hypervisor can perfectly synchronize its bringup (and SMP bringup)
with this, and it can even optimize things like on-disk memory
snapshots to simply not write out those pages to disk.
A 4c-like approach seems like it'd be a lot of bang for the buck -- we
reuse the existing mechanism (MADV_WIPEONFORK), so there's no new
userspace API to deal with, and it'd be race free, and eliminate a lot
of kernel complexity.
Clearly this has a chance to break applications, right?
If there's an app that uses this as a non-system-calls way
to find out whether there was a fork, it will break
when wipe triggers without a fork ...
For example, imagine:
MADV_WIPEONFORK
copy secret data to MADV_DONTFORK
fork
used to work, with this change it gets 0s instead of the secret data.
I am also not sure it's wise to expose each guest process
to the hypervisor like this. E.g. each process needs a
guest physical address of its own then. This is a finite resource.
The mmap interface proposed here is somewhat baroque, but it is
certainly simple to implement ...
Wipe of fork/vmgenid/whatever could end up being much more problematic
than it naively appears -- it could be wiped in the middle of a read.
Either the API needs to handle this cleanly, or we need something more
aggressive like signal-on-fork.
--Andy
Right, it's not on fork, it's actually when process is snapshotted.
If we assume it's CRIU we care about, then I
wonder what's wrong with something like
MADV_CHANGEONPTRACE_SEIZE
and basically say it's X bytes which change the value...
--
MST
From: Andy Lutomirski <luto@kernel.org> Date: 2020-10-18 16:14:16
On Sun, Oct 18, 2020 at 8:59 AM Michael S. Tsirkin [off-list ref] wrote:
On Sun, Oct 18, 2020 at 08:54:36AM -0700, Andy Lutomirski wrote:
quoted
On Sun, Oct 18, 2020 at 8:52 AM Michael S. Tsirkin [off-list ref] wrote:
quoted
On Sat, Oct 17, 2020 at 03:24:08PM +0200, Jason A. Donenfeld wrote:
quoted
4c. The guest kernel maintains an array of physical addresses that are
MADV_WIPEONFORK. The hypervisor knows about this array and its
location through whatever protocol, and before resuming a
moved/snapshotted/duplicated VM, it takes the responsibility for
memzeroing this memory. The huge pro here would be that this
eliminates all races, and reduces complexity quite a bit, because the
hypervisor can perfectly synchronize its bringup (and SMP bringup)
with this, and it can even optimize things like on-disk memory
snapshots to simply not write out those pages to disk.
A 4c-like approach seems like it'd be a lot of bang for the buck -- we
reuse the existing mechanism (MADV_WIPEONFORK), so there's no new
userspace API to deal with, and it'd be race free, and eliminate a lot
of kernel complexity.
Clearly this has a chance to break applications, right?
If there's an app that uses this as a non-system-calls way
to find out whether there was a fork, it will break
when wipe triggers without a fork ...
For example, imagine:
MADV_WIPEONFORK
copy secret data to MADV_DONTFORK
fork
used to work, with this change it gets 0s instead of the secret data.
I am also not sure it's wise to expose each guest process
to the hypervisor like this. E.g. each process needs a
guest physical address of its own then. This is a finite resource.
The mmap interface proposed here is somewhat baroque, but it is
certainly simple to implement ...
Wipe of fork/vmgenid/whatever could end up being much more problematic
than it naively appears -- it could be wiped in the middle of a read.
Either the API needs to handle this cleanly, or we need something more
aggressive like signal-on-fork.
--Andy
Right, it's not on fork, it's actually when process is snapshotted.
If we assume it's CRIU we care about, then I
wonder what's wrong with something like
MADV_CHANGEONPTRACE_SEIZE
and basically say it's X bytes which change the value...
I feel like we may be approaching this from the wrong end. Rather
than saying "what data structure can the kernel expose that might
plausibly be useful", how about we try identifying some specific
userspace needs and see what a good solution could look like. I can
identify two major cryptographic use cases:
1. A userspace RNG. The API exposed by the userspace end is a
function that generates random numbers. The userspace code in turn
wants to know some things from the kernel: it wants some
best-quality-available random seed data from the kernel (and possibly
an indication of how good it is) as well as an indication of whether
the userspace memory may have been cloned or rolled back, or, failing
that, an indication of whether a reseed is needed. Userspace could
implement a wide variety of algorithms on top depending on its goals
and compliance requirements, but the end goal is for the userspace
part to be very, very fast.
2. A userspace crypto stack that wants to avoid shooting itself in the
foot due to inadvertently doing the same thing twice. For example, an
AES-GCM stack does not want to reuse an IV, *expecially* if there is
even the slightest chance that it might reuse the IV for different
data. This use case doesn't necessarily involve random numbers, but,
if anything, it needs to be even faster than #1.
The threats here are not really the same. For #1, a userspace RNG
should be able to recover from a scenario in which an adversary clones
the entire process *and gets to own the clone*. For example, in
Android, an adversary can often gain complete control of a fork of the
zygote -- this shouldn't adversely affect the security properties of
other forks. Similarly, a server farm could operate by having one
booted server that is cloned to create more workers. Those clones
could be provisioned with secrets and permissions post-clone, and at
attacker gaining control of a fresh clone could be considered
acceptable. For #2, in contrast, if an adversary gains control of a
clone of an AES-GCM session, they learn the key outright -- the
relevant attack scenario is that the adversary gets to interact with
two clones without compromising either clone per se.
It's worth noting that, in both cases, there could possibly be more
than one instance of an RNG or an AES-GCM session in the same process.
This means that using signals is awkward but not necessarily
impossibly. (This is an area in which Linux, and POSIX in general, is
much weaker than Windows.)
From: "Michael S. Tsirkin" <mst@redhat.com> Date: 2020-10-19 15:00:59
On Sun, Oct 18, 2020 at 09:14:00AM -0700, Andy Lutomirski wrote:
On Sun, Oct 18, 2020 at 8:59 AM Michael S. Tsirkin [off-list ref] wrote:
quoted
On Sun, Oct 18, 2020 at 08:54:36AM -0700, Andy Lutomirski wrote:
quoted
On Sun, Oct 18, 2020 at 8:52 AM Michael S. Tsirkin [off-list ref] wrote:
quoted
On Sat, Oct 17, 2020 at 03:24:08PM +0200, Jason A. Donenfeld wrote:
quoted
4c. The guest kernel maintains an array of physical addresses that are
MADV_WIPEONFORK. The hypervisor knows about this array and its
location through whatever protocol, and before resuming a
moved/snapshotted/duplicated VM, it takes the responsibility for
memzeroing this memory. The huge pro here would be that this
eliminates all races, and reduces complexity quite a bit, because the
hypervisor can perfectly synchronize its bringup (and SMP bringup)
with this, and it can even optimize things like on-disk memory
snapshots to simply not write out those pages to disk.
A 4c-like approach seems like it'd be a lot of bang for the buck -- we
reuse the existing mechanism (MADV_WIPEONFORK), so there's no new
userspace API to deal with, and it'd be race free, and eliminate a lot
of kernel complexity.
Clearly this has a chance to break applications, right?
If there's an app that uses this as a non-system-calls way
to find out whether there was a fork, it will break
when wipe triggers without a fork ...
For example, imagine:
MADV_WIPEONFORK
copy secret data to MADV_DONTFORK
fork
used to work, with this change it gets 0s instead of the secret data.
I am also not sure it's wise to expose each guest process
to the hypervisor like this. E.g. each process needs a
guest physical address of its own then. This is a finite resource.
The mmap interface proposed here is somewhat baroque, but it is
certainly simple to implement ...
Wipe of fork/vmgenid/whatever could end up being much more problematic
than it naively appears -- it could be wiped in the middle of a read.
Either the API needs to handle this cleanly, or we need something more
aggressive like signal-on-fork.
--Andy
Right, it's not on fork, it's actually when process is snapshotted.
If we assume it's CRIU we care about, then I
wonder what's wrong with something like
MADV_CHANGEONPTRACE_SEIZE
and basically say it's X bytes which change the value...
I feel like we may be approaching this from the wrong end. Rather
than saying "what data structure can the kernel expose that might
plausibly be useful", how about we try identifying some specific
userspace needs and see what a good solution could look like. I can
identify two major cryptographic use cases:
From: Christian Borntraeger <hidden> Date: 2020-10-20 09:36:42
On 17.10.20 20:09, Alexander Graf wrote:
Hi Jason,
On 17.10.20 15:24, Jason A. Donenfeld wrote:
quoted
After discussing this offline with Jann a bit, I have a few general
comments on the design of this.
First, the UUID communicated by the hypervisor should be consumed by
the kernel -- added as another input to the rng -- and then userspace
We definitely want a kernel internal notifier as well, yes :).
quoted
should be notified that it should reseed any userspace RNGs that it
may have, without actually communicating that UUID to userspace. IOW,
I also tend to agree that it makes sense to disconnect the actual UUID we receive from the notification to user space. This would allow us to create a generic mechanism for VM save/restore cycles across different hypervisors. Let me add PPC and s390x people to the CC list to see whether they have anything remotely similar to the VmGenID mechanism. For x86 and aarch64, the ACPI and memory based VmGenID implemented here is the most obvious option to implement IMHO. It's also already implemented in all major hypervisors.
Hmm, what we do have configurations (e.g. stfle bits) and we do have a notification mechanism via sclp that notifies guests when things change.
As of today neither KVM nor Linux implement the sclp change notification mechanism, but I do see value in such a thing.
quoted
I agree with Jann there. Then, it's the functioning of this
notification mechanism to userspace that is interesting to me.
Yes. Passing a notification to userspace is essential. Where I do not see a solution yet is the race between notification and
already running with the old knowledge.
quoted
There are a few design goals of notifying userspace: it should be
fast, because people who are using userspace RNGs are usually doing so
in the first place to completely avoid syscall overhead for whatever
high performance application they have - e.g. I recall conversations
with Colm about his TLS implementation needing to make random IVs
_really_ fast. It should also happen as early as possible, with no
race or as minimal as possible race window, so that userspace doesn't
begin using old randomness and then switch over after the damage is
already done.
There are multiple facets and different types of consumers here. For a user space RNG, I agree that fast and as race free as possible is key. That's what the mmap interface is there for.
There are applications way beyond that though. What do you do with applications that already consumed randomness? For example a cached pool of SSL keys. Or a higher level language primitive that consumes randomness and caches its seed somewhere in an internal data structure. Or even worse: your system's host ssh key.
For those types of events, an mmap (or vDSO) interface does not work. We need to actively allow user space applications to readjust to the new environment - either internally (the language primitive case) or through a system event, maybe even as systemd trigger (the ssh host key case).
To give everyone enough time before we consider a system as "updated to the new environment", we have the callback logic with the "Orchestrator" that can check whether all listeners to system wide updates confirms they adjusted themselves.
That's what the rest of the logic is there for: A read+poll interface and all of the orchestration logic. It's not for the user space RNG case, it's for all of its downstream users.
quoted
I'm also not wedded to using Microsoft's proprietary hypervisor design
for this. If we come up with a better interface, I don't think it's
asking too much to implement that and reasonably expect for Microsoft
to catch up. Maybe someone here will find that controversial, but
whatever -- discussing ideal designs does not seem out of place or
inappropriate for how we usually approach things in the kernel, and a
closed source hypervisor coming along shouldn't disrupt that.
The main bonus point on this interface is that Hyper-V, VMware and QEMU implement it already. It would be a very natural for into the ecosystem. I agree though that we shouldn't have our user space interface necessarily dictated by it: Other hypervisors may implement different ways such as a simple edge IRQ that gets triggered whenever the VM gets resumed.
quoted
So, anyway, here are a few options with some pros and cons for the
kernel notifying userspace that its RNG should reseed.
I can only stress again that we should not be laser focused on the RNG case. In a lot of cases, data has already been generated by the RNG before the snapshot and needs to be reinitialized after the snapshot. In other cases such as system UUIDs, it's completely orthogonal to the RNG.
quoted
1. SIGRND - a new signal. Lol.
Doable, but a lot of plumbing in user space. It's also not necessarily a good for for event notification in most user space applications.
quoted
2. Userspace opens a file descriptor that it can epoll on. Pros are
that many notification mechanisms already use this. Cons is that this
requires syscall and might be more racy than we want. Another con is
that this a new thing for userspace programs to do.
That's part of what this patch does, right? This patch implements read+poll as well as mmap() for high speed reads.
quoted
3. We stick an atomic counter in the vDSO, Jann's suggestion. Pros are
that this is extremely fast, and also simple to use and implement.
There are enough sequence points in typical crypto programs that
checking to see whether this counter has changed before doing whatever
operation seems easy enough. Cons are that typically we've been
conservative about adding things to the vDSO, and this is also a new
thing for userspace programs to do.
The big con is that its use is going to be super limited to applications that can be adapted to check their "vm generation" through a vDSO call / read every time they consume data that may potentially need to be regenerated.
This probably works for the pure RNG case. It falls apart for more sophisticated things such as "redo my ssh host keys and restart the service" or "regenerate my samba machine uuid".
quoted
4. We already have a mechanism for this kind of thing, because the
same issue comes up when fork()ing. The solution was MADV_WIPEONFORK,
where userspace marks a page to be zeroed when forking, for the
purposes of the RNG being notified when its world gets split in two.
This is basically the same thing as we're discussing here with guest
snapshots, except it's on the system level rather than the process
level, and a system has many processes. But the problem space is still
almost the same, and we could simply reuse that same mechanism. There
are a few implementation strategies for that:
Yup, that's where we started from :). And then we ran into resistance by the mm people (on CC here). And then we looked at the problem more in depth and checked what it would take to for example implement this for user space RNGs in Java. It's ... more complicated than one may think at first.
quoted
4a. We mess with the PTEs of all processes' pages that are
MADV_WIPEONFORK, like fork does now, when the hypervisor notifies us
to do so. Then we wind up reusing the already existing logic for
userspace RNGs. Cons might be that this usually requires semaphores,
and we're in irq context, so we'd have to hoist to a workqueue, which
means either more wake up latency, or a larger race window.
4b. We just memzero all processes' pages that are MADV_WIPEONFORK,
when the hypervisor notifies us to do so. Then we wind up reusing the
already existing logic for userspace RNGs.
4c. The guest kernel maintains an array of physical addresses that are
MADV_WIPEONFORK. The hypervisor knows about this array and its
location through whatever protocol, and before resuming a
moved/snapshotted/duplicated VM, it takes the responsibility for
memzeroing this memory. The huge pro here would be that this
eliminates all races, and reduces complexity quite a bit, because the
hypervisor can perfectly synchronize its bringup (and SMP bringup)
with this, and it can even optimize things like on-disk memory
snapshots to simply not write out those pages to disk.
A 4c-like approach seems like it'd be a lot of bang for the buck -- we
reuse the existing mechanism (MADV_WIPEONFORK), so there's no new
userspace API to deal with, and it'd be race free, and eliminate a lot
of kernel complexity.
But 4b and 3 don't seem too bad either.
Any thoughts on 4c? Is that utterly insane, or does that actually get
us somewhere close to what we want?
All of the options for "4" are possible and have an RFC out. Please check out the discussion linked above :).
The problem with anything that relies on close loop reads (options 3+4) is not going to work well with the more sophisticated use case of derived data.
IMHO it will boil down to "both". We will need a high-speed interface that with close-to-0 overhead tells you either the generation ID or clears pages (options 3+4) as well as something that is bigger for applications that can either intrinsically (sshd) or by system design (Java) not adopt the mechanisms above easily.
That said, we need to start somewhere. I don't mind which angle we start from. But this is a real world problem and one that will only become more prevalent over time as VMs are used for more than only your traditional enterprise hardware consolidation.
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
From: Alexander Graf <hidden> Date: 2020-10-20 09:55:21
On 20.10.20 11:35, Christian Borntraeger wrote:
On 17.10.20 20:09, Alexander Graf wrote:
quoted
Hi Jason,
On 17.10.20 15:24, Jason A. Donenfeld wrote:
quoted
After discussing this offline with Jann a bit, I have a few general
comments on the design of this.
First, the UUID communicated by the hypervisor should be consumed by
the kernel -- added as another input to the rng -- and then userspace
We definitely want a kernel internal notifier as well, yes :).
quoted
should be notified that it should reseed any userspace RNGs that it
may have, without actually communicating that UUID to userspace. IOW,
I also tend to agree that it makes sense to disconnect the actual UUID we receive from the notification to user space. This would allow us to create a generic mechanism for VM save/restore cycles across different hypervisors. Let me add PPC and s390x people to the CC list to see whether they have anything remotely similar to the VmGenID mechanism. For x86 and aarch64, the ACPI and memory based VmGenID implemented here is the most obvious option to implement IMHO. It's also already implemented in all major hypervisors.
Hmm, what we do have configurations (e.g. stfle bits) and we do have a notification mechanism via sclp that notifies guests when things change.
As of today neither KVM nor Linux implement the sclp change notification mechanism, but I do see value in such a thing.
stfle only stores architected CPU capabilities, no? The UUID is about
uniquely identifying clones of the same base image, so they can
reestablish their uniqueness.
That said, your interest means that there may be a mechanism on s390 one
day, so we should think about making it more generic.
quoted
quoted
I agree with Jann there. Then, it's the functioning of this
notification mechanism to userspace that is interesting to me.
Yes. Passing a notification to userspace is essential. Where I do not see a solution yet is the race between notification and
already running with the old knowledge.
With a post-mortem interface, we will always have a tiny race window.
I'm not really convinced that this is a serious problem yet though.
In order to do extract anything off a virtual machine that was cloned,
you need to communicate with it. If you for example stop the network
link until all of this device's users have indicated they are finished
adjusting, the race should be small enough for any practical purposes I
can think of.
Alex
Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
From: Catangiu, Adrian Costin <hidden> Date: 2020-10-20 16:54:42
Hi all,
On 17/10/2020, 21:09, "Graf (AWS), Alexander" [off-list ref] wrote:
On 17.10.20 15:24, Jason A. Donenfeld wrote:
>
> After discussing this offline with Jann a bit, I have a few general
> comments on the design of this.
>
> First, the UUID communicated by the hypervisor should be consumed by
> the kernel -- added as another input to the rng -- and then userspace
We definitely want a kernel internal notifier as well, yes :).
What would be a generic event trigger mechanism we could use from a kernel
module/driver for triggering rng reseed (possibly adding the uuid to the mix
as well)?
Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.