Re: Overlayfs @ Containers and checkpoint/restart micro-conference at LPC2018

On Tue, Sep 11, 2018 at 08:13:40AM -0700, James Bottomley wrote:

On Tue, 2018-09-11 at 09:52 -0400, Vivek Goyal wrote:

quoted

On Sun, Sep 09, 2018 at 11:18:54AM +0200, Christian Brauner wrote:
[..]

quoted

My team hast just started to be more involved with shifts
development a few months back. Overlayfs is definitely an
inspiration and we even once thought about making shifts an
extension of overlayfs. Seth Forshee on my team is currently
actively working on shifts and getting a POC ready.
When he has a POC based on James' patchset there will be an RFC
that will go to fsdevel and all parties of interest.
There will also be an update on shifts development during the
microconf. So even more reason for developers from overlayfs to
stop by.

So we need both shiftfs and overlayfs in container deployments,
right?

Well, no; only docker style containers need some form of overlay graph
driver, but even there it doesn't have to be the overlayfs one.  When I
build unprivileged containers, I never use overlays so for me having to
use it will be problematic as it would be even in docker for the non-
overlayfs graph drivers.

Hi James,

Ok. For us, overlayfs is now default for docker containers as it was
much faster as compared to devicemapper and vfs (due to page cache
sharing). So please keep in mind overlayfs graph driver use case 
as well while designing a solution.

For non docker containers, I am assuming all the image is in one directory
so no union is required. Also these probably are read-only containers
or this image directory is not shared with other containers for it to
work.

Perhaps we should consider this when we look at the use cases.

quoted

shiftfs to make sure each container can run in its own user namespace
and uid/gid mappings can be setup on the fly and overlayfs to provide
union of multiple layers and copy on write filesystem. I am assuming
that shiftfs is working on top of overlayfs here?

Doing shifting at VFS level using mount API was another idea
discussed at last plumbers. I saw David Howells was pushing all the
new mount API patches. Not sure if he ever got time to pursue
shifting at VFS level.

I wasn't party to the conversation, but when I discussed it with Ted
(who wants something similar for a feature changing bind mount) we need
the entire VFS api to be struct path based instead of dentry/inode
based.  That's the way it's going, but we'd need to get to the end
point so we have a struct vfsmnt available for every VFS call.

Ok, thanks. So mappings will be per mount and available in vfsmnt and
hence pass around path so that one can get to vfsmnt (instead of
dentry/inode). Makes sense.

quoted

BTW, now we have metadata only copy up patches in overlayfs as
well(4.19-rc). That speeds up chown operation with overlayfs,
needed for changing ownership of files in images for making sure
they work fine with user namespaces. In my simple testing in a VM,
a fedora image was taking around 30 seconds to chown. With metadata
only copy up that time drops to around 2-3 seconds. So till shiftfs
or shiting at VFS level gets merged, it can be used as a stop gap
solution.

Most of the snapshot based filesystem (btrfs, xfs) do this without any
need for overlayfs.

Right. But they don't share page cache yet (same with devicemapper). So
till we get page cache sharing in these file systems, overlayfs still
has the advantage of being able to launch many more containers using
same image with smaller memory requirements (and its faster too as image
does not have to be read from disk).

Thanks
Vivek