Re: [CRIU] Introspecting userns relationships to other namespaces?

quoted

Andrew Vagin [off-list ref] writes:

On Wed, Jul 06, 2016 at 10:46:33AM -0500, Eric W. Biederman wrote:
"Serge E. Hallyn" [off-list ref] writes:

On Wed, Jul 06, 2016 at 10:41:48AM +0200, Michael Kerrisk (man-pages) wrote:
[Rats! Doing now what I should have down to start with. Looping some
lists and CRIU and other possibly relevant people into this
conversation]

Hi Eric,

On 5 July 2016 at 23:47, Eric W. Biederman [off-list ref] wrote:
"Michael Kerrisk (man-pages)" [off-list ref] writes:

Hi Eric,

I have a question. Is there any way currently to discover which
user namespace a particular nonuser namespace is governed by?
Maybe I am missing something, but there does not seem to be a
way to do this. Also, can one discover which userns is the
parent of a given userns? Again, I can't see a way to do this.

The point here is introspecting so that a process might determine
what its capabilities are when operating on some resource governed
by a (nonuser) namespace.
To the best of my knowledge that there is not an interface to get that
information.  It would be good to have such an interface for no other
reason than the CRIU folks are going to need it at some point.  I am a
bit surprised they have not complained yet.
I don't think they need it.  They do in fact have what they need.  Assume
you have tasks T1, T2, T1_1 and T2_1;  T1 and T2 are in init_user_ns;  T1
spawned T1_1 in a new userns;  T2 spawned T2_1 which setns()d to T1_1's ns.
There's some {handwave} uid mapping, does not matter.

At restart, it doesn't matter which task originally created the new userns.
criu knows T1_1 and T2_1 are in the same userns;  it creates the userns, sets
up the mapping, and T1_1 and T2_1 setns() to it.
Given that the simple cases are so easy it probably doesn't matter in
that sense.

However we now have the case where user namespaces own pid namespaces,
and uts namespaces, and network namespaces, and ipc namespaces, and
filesystems.  Throw in some mount propagation and use of setns and
things could get confusing.   It is something that will need to be
figured out if CRIU is going to properly checkpoint containers
containing containers containing containers containing containers.
It isn't a joke:). We have a few requests to support CR of containers with
Docker containers inside. And we are going to start this task in a near
future, so we would like to have interface to get dependencies between
namespaces too.

BTW: CRIU already supports nested mount namespaces, because systemd
creates them for services.
The tricky part about this and what messes up James proposed plan is
that the interface needs to be something that returns a namespace file
descriptor.  So we can't print something out in a simple text file.
Well I suppose we could print an device number and inode number pair.
But then someone would still have to scour processes looking for a user
namespace so that is likely less than ideal.

Starting with 4.8 we are also going to need to be able to retrieve the
user namespace owner of filesystems.  That will be an interesting mix.

Eric
On Fri, 2016-07-08 at 02:44 -0500, Eric W. Biederman wrote:
Andrew Vagin [off-list ref] writes:

On Wed, Jul 06, 2016 at 10:46:33AM -0500, Eric W. Biederman wrote:
"Serge E. Hallyn" [off-list ref] writes:

On Wed, Jul 06, 2016 at 10:41:48AM +0200, Michael Kerrisk (man
-pages) wrote:
[Rats! Doing now what I should have down to start with.
Looping some
lists and CRIU and other possibly relevant people into this
conversation]

Hi Eric,

On 5 July 2016 at 23:47, Eric W. Biederman <
ebiederm-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org> wrote:
"Michael Kerrisk (man-pages)" [off-list ref]
writes:

Hi Eric,

I have a question. Is there any way currently to discover 
which user namespace a particular nonuser namespace is 
governed by? Maybe I am missing something, but there does 
not seem to be a way to do this. Also, can one discover 
which userns is the parent of a given userns? Again, I 
can't see a way to do this.

The point here is introspecting so that a process might 
determine what its capabilities are when operating on 
some resource governed by a (nonuser) namespace.
To the best of my knowledge that there is not an interface 
to get that information.  It would be good to have such an 
interface for no other reason than the CRIU folks are going 
to need it at some point.  I am a bit surprised they have
not complained yet.
I don't think they need it.  They do in fact have what they 
need.  Assume you have tasks T1, T2, T1_1 and T2_1;  T1 and T2 
are in init_user_ns;  T1 spawned T1_1 in a new userns;  T2 
spawned T2_1 which setns()d to T1_1's ns. There's some
{handwave} uid mapping, does not matter.

At restart, it doesn't matter which task originally created the 
new userns. criu knows T1_1 and T2_1 are in the same userns; 
 it creates the userns, sets up the mapping, and T1_1 and T2_1
setns() to it.
Given that the simple cases are so easy it probably doesn't 
matter in that sense.

However we now have the case where user namespaces own pid 
namespaces, and uts namespaces, and network namespaces, and ipc 
namespaces, and filesystems.  Throw in some mount propagation and 
use of setns and things could get confusing.   It is something 
that will need to be figured out if CRIU is going to properly 
checkpoint containers containing containers containing containers 
containing containers.
It isn't a joke:). We have a few requests to support CR of 
containers with Docker containers inside. And we are going to start 
this task in a near future, so we would like to have interface to 
get dependencies between namespaces too.

BTW: CRIU already supports nested mount namespaces, because systemd
creates them for services.
The tricky part about this and what messes up James proposed plan is
that the interface needs to be something that returns a namespace 
file descriptor.  So we can't print something out in a simple text
file.
I actually described two problems: the first was how we get the
information in the first place.  Currently the owning or parent user_ns
is tucked inside an opaque structure.  I think we need to move that to
ns_common where it would be the owning userns for all non-user
namespaces and the parent for the userns.

Once we actually have the information, we can also add a set of proc
links, say either

/proc/<pid>/ns/X-userns

Which might be a bit messy since it doubles the number of files, or
perhaps in a simple directory.

Well I suppose we could print an device number and inode number pair.
But then someone would still have to scour processes looking for a 
user namespace so that is likely less than ideal.
There's no reason any of the proposed methods so far have to be
exclusive: nsfs.c has a lot of flexibility.

Starting with 4.8 we are also going to need to be able to retrieve 
the user namespace owner of filesystems.  That will be an interesting
mix.
This is per mount point, isn't it? so it can't be in /proc/fs/ and it
would have to be per local mount tree.  Yes, that is a bit nasty. 
 Sounds like we might need to unfold mount or mountinfo into something
that has one directory per entry?

James

Eric

_______________________________________________
Containers mailing list
Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linuxfoundation.org/mailman/listinfo/containers
On Fri, Jul 08, 2016 at 01:38:19PM -0700, Andrew Vagin wrote:
What do you think about the idea to mount nsfs and be able to look up
any alive namespace by inum:

  $ tree .
  .
  ├── mnt{inum}
  │   └── user -> ../user{inum}
  ├── pid{inum}
  │   ├── pid{inum}
  │   │   └── user -> ../../user{inum}/user{inum}
  │   └── user -> ../user{inum}
  └── user{inum}
      └── user{inum}

https://lkml.org/lkml/2016/7/8/59

I think it solves all requirements which were mentioned in this thread.
It may need an additional entry per directory for the bit you setns.
Maybe ‘handle’?

  $ tree .
  .
  ├── mnt{inum}
  │   ├── handle -> mnt:[{inum}]
  │   └── user -> ../user{inum}
  …

but that's not a major revision.

On Fri, Jul 08, 2016 at 07:35:33AM -0700, James Bottomley wrote:
On Fri, 2016-07-08 at 02:44 -0500, Eric W. Biederman wrote:
Starting with 4.8 we are also going to need to be able to
retrieve the user namespace owner of filesystems.  That will be
an interesting mix.
This is per mount point, isn't it? so it can't be in /proc/fs/ and
it would have to be per local mount tree.  Yes, that is a bit
nasty.  Sounds like we might need to unfold mount or mountinfo
into something that has one directory per entry?
If we will be able to look up namespaces in nsfs by inum, we can
print an userns inum in mountinfo.
With the tree view you can find a namespace by inum (if it's one of
your descendants), but it's not going to be particularly efficient
(you'll have to walk the tree).  Folks that need to do that quickly
can index the tree (which would be fairly straightforward if the nsfs
mount supports inotify), but it would be nice to have a more elegant
solution for this use-case.

Cheers,
Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
On July 8, 2016 1:38:19 PM PDT, Andrew Vagin [off-list ref] wrote:
On Fri, Jul 08, 2016 at 07:35:33AM -0700, James Bottomley wrote:
On Fri, 2016-07-08 at 02:44 -0500, Eric W. Biederman wrote:
Andrew Vagin [off-list ref] writes:

On Wed, Jul 06, 2016 at 10:46:33AM -0500, Eric W. Biederman
wrote:
"Serge E. Hallyn" [off-list ref] writes:

On Wed, Jul 06, 2016 at 10:41:48AM +0200, Michael Kerrisk
(man
-pages) wrote:
[Rats! Doing now what I should have down to start with.
Looping some
lists and CRIU and other possibly relevant people into this
conversation]

Hi Eric,

On 5 July 2016 at 23:47, Eric W. Biederman <
ebiederm-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org> wrote:
"Michael Kerrisk (man-pages)" [off-list ref]
writes:

Hi Eric,

I have a question. Is there any way currently to
discover 
which user namespace a particular nonuser namespace is 
governed by? Maybe I am missing something, but there
does 
not seem to be a way to do this. Also, can one discover

which userns is the parent of a given userns? Again, I 
can't see a way to do this.

The point here is introspecting so that a process might

determine what its capabilities are when operating on 
some resource governed by a (nonuser) namespace.
To the best of my knowledge that there is not an
interface 
to get that information.  It would be good to have such
an 
interface for no other reason than the CRIU folks are
going 
to need it at some point.  I am a bit surprised they have
not complained yet.
I don't think they need it.  They do in fact have what they 
need.  Assume you have tasks T1, T2, T1_1 and T2_1;  T1 and
T2 
are in init_user_ns;  T1 spawned T1_1 in a new userns;  T2 
spawned T2_1 which setns()d to T1_1's ns. There's some
{handwave} uid mapping, does not matter.

At restart, it doesn't matter which task originally created
the 
new userns. criu knows T1_1 and T2_1 are in the same userns; 
 it creates the userns, sets up the mapping, and T1_1 and
T2_1
setns() to it.
Given that the simple cases are so easy it probably doesn't 
matter in that sense.

However we now have the case where user namespaces own pid 
namespaces, and uts namespaces, and network namespaces, and ipc

namespaces, and filesystems.  Throw in some mount propagation
and 
use of setns and things could get confusing.   It is something 
that will need to be figured out if CRIU is going to properly 
checkpoint containers containing containers containing
containers 
containing containers.
It isn't a joke:). We have a few requests to support CR of 
containers with Docker containers inside. And we are going to
start 
this task in a near future, so we would like to have interface to

get dependencies between namespaces too.

BTW: CRIU already supports nested mount namespaces, because
systemd
creates them for services.
The tricky part about this and what messes up James proposed plan
is
that the interface needs to be something that returns a namespace 
file descriptor.  So we can't print something out in a simple text
file.
I actually described two problems: the first was how we get the
information in the first place.  Currently the owning or parent
user_ns
is tucked inside an opaque structure.  I think we need to move that
to
ns_common where it would be the owning userns for all non-user
namespaces and the parent for the userns.
I'm agree with this.

Once we actually have the information, we can also add a set of proc
links, say either

/proc/<pid>/ns/X-userns

Which might be a bit messy since it doubles the number of files, or
perhaps in a simple directory.
In this case we will need to enter into each namespace to build a full
chain of dependencies.

It's tricky, because if we enter into a child userns, we can't to enter
into a parent userns from the same process, so to get the next branch,
we will need to create a new process.

			    process A
				|
init_user_ns->child_user_ns_1->child_userns_2

fork() -> B
 B: setns(/proc/A/ns/userns-parent)
readlink(/proc/B/ns/userns)

fork() -> C
 C: setns(/proc/B/ns/userns-parent)
readlink(/proc/C/ns/userns)

Well I suppose we could print an device number and inode number
pair.
But then someone would still have to scour processes looking for a 
user namespace so that is likely less than ideal.
There's no reason any of the proposed methods so far have to be
exclusive: nsfs.c has a lot of flexibility.

What do you think about the idea to mount nsfs and be able to look up
any alive namespace by inum:
I think I like it.  It will give us a way to enter any extant namespace.  It will work for Eric's fs namespaces as well.  Perhaps a /process/ns/<inum>
 Directory?

James

 $ tree .
 .
 ├── mnt{inum}
 │   └── user -> ../user{inum}
 ├── pid{inum}
 │   ├── pid{inum}
 │   │   └── user -> ../../user{inum}/user{inum}
 │   └── user -> ../user{inum}
 └── user{inum}
     └── user{inum}

https://lkml.org/lkml/2016/7/8/59

I think it solves all requirements which were mentioned in this thread.

Starting with 4.8 we are also going to need to be able to retrieve 
the user namespace owner of filesystems.  That will be an
interesting
mix.
This is per mount point, isn't it? so it can't be in /proc/fs/ and it
would have to be per local mount tree.  Yes, that is a bit nasty. 
 Sounds like we might need to unfold mount or mountinfo into
something
that has one directory per entry?
If we will be able to look up namespaces in nsfs by inum, we can print
an userns inum in mountinfo.

James

Eric

_______________________________________________
Containers mailing list
Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linuxfoundation.org/mailman/listinfo/containers
_______________________________________________
Containers mailing list
Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linuxfoundation.org/mailman/listinfo/containers

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
On July 8, 2016 1:38:19 PM PDT, Andrew Vagin [off-list ref] wrote:
On Fri, Jul 08, 2016 at 07:35:33AM -0700, James Bottomley wrote:
On Fri, 2016-07-08 at 02:44 -0500, Eric W. Biederman wrote:
Andrew Vagin [off-list ref] writes:

On Wed, Jul 06, 2016 at 10:46:33AM -0500, Eric W. Biederman
wrote:
"Serge E. Hallyn" [off-list ref] writes:

On Wed, Jul 06, 2016 at 10:41:48AM +0200, Michael Kerrisk
(man
-pages) wrote:
[Rats! Doing now what I should have down to start with.
Looping some
lists and CRIU and other possibly relevant people into this
conversation]

Hi Eric,

On 5 July 2016 at 23:47, Eric W. Biederman <
ebiederm-aS9lmoZGLiVWk0Htik3J/w@public.gmane.org> wrote:
"Michael Kerrisk (man-pages)" [off-list ref]
writes:

Hi Eric,

I have a question. Is there any way currently to
discover 
which user namespace a particular nonuser namespace is 
governed by? Maybe I am missing something, but there
does 
not seem to be a way to do this. Also, can one discover

which userns is the parent of a given userns? Again, I 
can't see a way to do this.

The point here is introspecting so that a process might

determine what its capabilities are when operating on 
some resource governed by a (nonuser) namespace.
To the best of my knowledge that there is not an
interface 
to get that information.  It would be good to have such
an 
interface for no other reason than the CRIU folks are
going 
to need it at some point.  I am a bit surprised they have
not complained yet.
I don't think they need it.  They do in fact have what they 
need.  Assume you have tasks T1, T2, T1_1 and T2_1;  T1 and
T2 
are in init_user_ns;  T1 spawned T1_1 in a new userns;  T2 
spawned T2_1 which setns()d to T1_1's ns. There's some
{handwave} uid mapping, does not matter.

At restart, it doesn't matter which task originally created
the 
new userns. criu knows T1_1 and T2_1 are in the same userns; 
 it creates the userns, sets up the mapping, and T1_1 and
T2_1
setns() to it.
Given that the simple cases are so easy it probably doesn't 
matter in that sense.

However we now have the case where user namespaces own pid 
namespaces, and uts namespaces, and network namespaces, and ipc

namespaces, and filesystems.  Throw in some mount propagation
and 
use of setns and things could get confusing.   It is something 
that will need to be figured out if CRIU is going to properly 
checkpoint containers containing containers containing
containers 
containing containers.
It isn't a joke:). We have a few requests to support CR of 
containers with Docker containers inside. And we are going to
start 
this task in a near future, so we would like to have interface to

get dependencies between namespaces too.

BTW: CRIU already supports nested mount namespaces, because
systemd
creates them for services.
The tricky part about this and what messes up James proposed plan
is
that the interface needs to be something that returns a namespace 
file descriptor.  So we can't print something out in a simple text
file.
I actually described two problems: the first was how we get the
information in the first place.  Currently the owning or parent
user_ns
is tucked inside an opaque structure.  I think we need to move that
to
ns_common where it would be the owning userns for all non-user
namespaces and the parent for the userns.
I'm agree with this.

Once we actually have the information, we can also add a set of proc
links, say either

/proc/<pid>/ns/X-userns

Which might be a bit messy since it doubles the number of files, or
perhaps in a simple directory.
In this case we will need to enter into each namespace to build a full
chain of dependencies.

It's tricky, because if we enter into a child userns, we can't to enter
into a parent userns from the same process, so to get the next branch,
we will need to create a new process.

			    process A
				|
init_user_ns->child_user_ns_1->child_userns_2

fork() -> B
 B: setns(/proc/A/ns/userns-parent)
readlink(/proc/B/ns/userns)

fork() -> C
 C: setns(/proc/B/ns/userns-parent)
readlink(/proc/C/ns/userns)

Well I suppose we could print an device number and inode number
pair.
But then someone would still have to scour processes looking for a 
user namespace so that is likely less than ideal.
There's no reason any of the proposed methods so far have to be
exclusive: nsfs.c has a lot of flexibility.

What do you think about the idea to mount nsfs and be able to look up
any alive namespace by inum:
I think I like it.  It will give us a way to enter any extant namespace.  It will work for Eric's fs namespaces as well.  Perhaps a /process/ns/<inum>
 Directory?

James

 $ tree .
 .
 ├── mnt{inum}
 │   └── user -> ../user{inum}
 ├── pid{inum}
 │   ├── pid{inum}
 │   │   └── user -> ../../user{inum}/user{inum}
 │   └── user -> ../user{inum}
 └── user{inum}
     └── user{inum}

https://lkml.org/lkml/2016/7/8/59

I think it solves all requirements which were mentioned in this thread.

Starting with 4.8 we are also going to need to be able to retrieve 
the user namespace owner of filesystems.  That will be an
interesting
mix.
This is per mount point, isn't it? so it can't be in /proc/fs/ and it
would have to be per local mount tree.  Yes, that is a bit nasty. 
 Sounds like we might need to unfold mount or mountinfo into
something
that has one directory per entry?
If we will be able to look up namespaces in nsfs by inum, we can print
an userns inum in mountinfo.

James

Eric

_______________________________________________
Containers mailing list
Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linuxfoundation.org/mailman/listinfo/containers
_______________________________________________
Containers mailing list
Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
https://lists.linuxfoundation.org/mailman/listinfo/containers

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
On Fri, Jul 08, 2016 at 07:35:33AM -0700, James Bottomley wrote:
On Fri, 2016-07-08 at 02:44 -0500, Eric W. Biederman wrote:
Andrew Vagin [off-list ref] writes:

On Wed, Jul 06, 2016 at 10:46:33AM -0500, Eric W. Biederman wrote:
"Serge E. Hallyn" [off-list ref] writes:

On Wed, Jul 06, 2016 at 10:41:48AM +0200, Michael Kerrisk (man
-pages) wrote:
[Rats! Doing now what I should have down to start with.
Looping some
lists and CRIU and other possibly relevant people into this
conversation]

Hi Eric,

On 5 July 2016 at 23:47, Eric W. Biederman <
ebiederm@xmission.com> wrote:
"Michael Kerrisk (man-pages)" [off-list ref]
writes:

Hi Eric,

I have a question. Is there any way currently to discover 
which user namespace a particular nonuser namespace is 
governed by? Maybe I am missing something, but there does 
not seem to be a way to do this. Also, can one discover 
which userns is the parent of a given userns? Again, I 
can't see a way to do this.

The point here is introspecting so that a process might 
determine what its capabilities are when operating on 
some resource governed by a (nonuser) namespace.
To the best of my knowledge that there is not an interface 
to get that information.  It would be good to have such an 
interface for no other reason than the CRIU folks are going 
to need it at some point.  I am a bit surprised they have
not complained yet.
I don't think they need it.  They do in fact have what they 
need.  Assume you have tasks T1, T2, T1_1 and T2_1;  T1 and T2 
are in init_user_ns;  T1 spawned T1_1 in a new userns;  T2 
spawned T2_1 which setns()d to T1_1's ns. There's some
{handwave} uid mapping, does not matter.

At restart, it doesn't matter which task originally created the 
new userns. criu knows T1_1 and T2_1 are in the same userns; 
 it creates the userns, sets up the mapping, and T1_1 and T2_1
setns() to it.
Given that the simple cases are so easy it probably doesn't 
matter in that sense.

However we now have the case where user namespaces own pid 
namespaces, and uts namespaces, and network namespaces, and ipc 
namespaces, and filesystems.  Throw in some mount propagation and 
use of setns and things could get confusing.   It is something 
that will need to be figured out if CRIU is going to properly 
checkpoint containers containing containers containing containers 
containing containers.
It isn't a joke:). We have a few requests to support CR of 
containers with Docker containers inside. And we are going to start 
this task in a near future, so we would like to have interface to 
get dependencies between namespaces too.

BTW: CRIU already supports nested mount namespaces, because systemd
creates them for services.
The tricky part about this and what messes up James proposed plan is
that the interface needs to be something that returns a namespace 
file descriptor.  So we can't print something out in a simple text
file.
I actually described two problems: the first was how we get the
information in the first place.  Currently the owning or parent user_ns
is tucked inside an opaque structure.  I think we need to move that to
ns_common where it would be the owning userns for all non-user
namespaces and the parent for the userns.
I'm agree with this.

Once we actually have the information, we can also add a set of proc
links, say either

/proc/<pid>/ns/X-userns

Which might be a bit messy since it doubles the number of files, or
perhaps in a simple directory.
In this case we will need to enter into each namespace to build a full
chain of dependencies.

It's tricky, because if we enter into a child userns, we can't to enter
into a parent userns from the same process, so to get the next branch,
we will need to create a new process.

				    process A
					|
init_user_ns->child_user_ns_1->child_userns_2

fork() -> B
  B: setns(/proc/A/ns/userns-parent)
readlink(/proc/B/ns/userns)

fork() -> C
  C: setns(/proc/B/ns/userns-parent)
readlink(/proc/C/ns/userns)

Well I suppose we could print an device number and inode number pair.
But then someone would still have to scour processes looking for a 
user namespace so that is likely less than ideal.
There's no reason any of the proposed methods so far have to be
exclusive: nsfs.c has a lot of flexibility.

What do you think about the idea to mount nsfs and be able to look up
any alive namespace by inum:

  $ tree .
  .
  ├── mnt{inum}
  │   └── user -> ../user{inum}
  ├── pid{inum}
  │   ├── pid{inum}
  │   │   └── user -> ../../user{inum}/user{inum}
  │   └── user -> ../user{inum}
  └── user{inum}
      └── user{inum}

https://lkml.org/lkml/2016/7/8/59

I think it solves all requirements which were mentioned in this thread.

Starting with 4.8 we are also going to need to be able to retrieve 
the user namespace owner of filesystems.  That will be an interesting
mix.
This is per mount point, isn't it? so it can't be in /proc/fs/ and it
would have to be per local mount tree.  Yes, that is a bit nasty. 
 Sounds like we might need to unfold mount or mountinfo into something
that has one directory per entry?
If we will be able to look up namespaces in nsfs by inum, we can print
an userns inum in mountinfo.

James

Eric

_______________________________________________
Containers mailing list
Containers@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/containers
_______________________________________________
Containers mailing list
Containers@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/containers

`h`	back out one level
`j`	next message in thread
`k`	previous message in thread
`l`	drill in
`Esc`	close help / fold thread tree
`?`	toggle this help