I'm re-posting this with a couple of fixes: one fixes the FPU
save/restore and another fixes argument to kunmap_atomic().
There is now a git tree at:
git://gorgona.ncl.cs.columbia.edu/pub/git/linux-cr.git
And another tree to track development and older versions:
git://gorgona.ncl.cs.columbia.edu/pub/git/linux-cr-dev.git
I could not have told it better than Dave Hansen:
--
I'd like to see these merged into -mm and on the way to mainline. The
entire freakin' world is cc'd. So sue me. :)
Why do we want it? It allows containers to be moved between physical
machines' kernels in the same way that VMWare can move VMs between
physical machines' hypervisors. There are currently at least two
out-of-tree implementations of this in the commercial world (IBM's
Metacluster and Parallels' OpenVZ/Virtuozzo) and several in the academic
world like Zap.
Why do we need it in mainline now? Because we already have plenty of
out-of-tree ones, and want to know what an in-tree one will be like. :)
What *I* want right now is the extra review and scrutiny that comes with
a mainline submission to make sure we're not going in a direction
contrary to the community.
This only supports pretty simple apps. But, I trust Ingo when he says:
quoted
Generally, if something works for simple apps already (in a robust,
compatible and supportable way) and users find it "very cool", then
support for more complex apps is not far in the future. but if you
want to support more complex apps straight away, it takes forever and
gets ugly.
We're *certainly* going to be changing the ABI (which is the format of
the checkpoint). I'd like to follow the model that we used for
ext4-dev, which is to make it very clear that this is a development-only
feature for now. Perhaps we do that by making the interface only
available through debugfs or something similar for now. Or, reserving
the syscall numbers but require some runtime switch to be thrown before
they can be used. I'm open to suggestions here.
These patches are Oren Laadan's baby. Virtually all this code is his,
but he's a bit busy at the moment finishing up his PhD.
There's a plethora of old history and some userspace tools below if you
want some more detail, but please ignore them and look at the kernel
code. :)
--
These patches implement basic checkpoint-restart [CR]. This version
(v7) supports basic tasks with simple private memory, and open files
(regular files and directories only). See original announcements below.
Oren.
--
Todo:
- Add support for x86-64 and improve ABI
- Refine or change syscall interface
- Extend to handle (multiple) tasks in a container
- Handle multiple namespaces in a container (e.g. save the filesystem
namespaces state with the file descriptors)
- Security (without CAPS_SYS_ADMIN files restore may fail)
Changelog:
[2008-Oct-17] v7:
- Fix save/restore state of FPU
- Fix argument given to kunmap_atomic() in memory dump/restore
[2008-Oct-07] v6:
- Balance all calls to cr_hbuf_get() with matching cr_hbuf_put()
(even though it's not really needed)
- Add assumptions and what's-missing to documentation
- Misc fixes and cleanups
[2008-Sep-11] v5:
- Config is now 'def_bool n' by default
- Improve memory dump/restore code (following Dave Hansen's comments)
- Change dump format (and code) to allow chunks of <vaddrs, pages>
instead of one long list of each
- Fix use of follow_page() to avoid faulting in non-present pages
- Memory restore now maps user pages explicitly to copy data into them,
instead of reading directly to user space; got rid of mprotect_fixup()
- Remove preempt_disable() when restoring debug registers
- Rename headers files s/ckpt/checkpoint/
- Fix misc bugs in files dump/restore
- Fixes and cleanups on some error paths
- Fix misc coding style
[2008-Sep-09] v4:
- Various fixes and clean-ups
- Fix calculation of hash table size
- Fix header structure alignment
- Use stand list_... for cr_pgarr
[2008-Aug-29] v3:
- Various fixes and clean-ups
- Use standard hlist_... for hash table
- Better use of standard kmalloc/kfree
[2008-Aug-20] v2:
- Added Dump and restore of open files (regular and directories)
- Added basic handling of shared objects, and improve handling of
'parent tag' concept
- Added documentation
- Improved ABI, 64bit padding for image data
- Improved locking when saving/restoring memory
- Added UTS information to header (release, version, machine)
- Cleanup extraction of filename from a file pointer
- Refactor to allow easier reviewing
- Remove requirement for CAPS_SYS_ADMIN until we come up with a
security policy (this means that file restore may fail)
- Other cleanup and response to comments for v1
[2008-Jul-29] v1:
- Initial version: support a single task with address space of only
private anonymous or file-mapped VMAs; syscalls ignore pid/crid
argument and act on current process.
--
(Dave Hansen's announcement)
At the containers mini-conference before OLS, the consensus among
all the stakeholders was that doing checkpoint/restart in the kernel
as much as possible was the best approach. With this approach, the
kernel will export a relatively opaque 'blob' of data to userspace
which can then be handed to the new kernel at restore time.
This is different than what had been proposed before, which was
that a userspace application would be responsible for collecting
all of this data. We were also planning on adding lots of new,
little kernel interfaces for all of the things that needed
checkpointing. This unites those into a single, grand interface.
The 'blob' will contain copies of select portions of kernel
structures such as vmas and mm_structs. It will also contain
copies of the actual memory that the process uses. Any changes
in this blob's format between kernel revisions can be handled by
an in-userspace conversion program.
This is a similar approach to virtually all of the commercial
checkpoint/restart products out there, as well as the research
project Zap.
These patches basically serialize internel kernel state and write
it out to a file descriptor. The checkpoint and restore are done
with two new system calls: sys_checkpoint and sys_restart.
In this incarnation, they can only work checkpoint and restore a
single task. The task's address space may consist of only private,
simple vma's - anonymous or file-mapped. The open files may consist
of only simple files and directories.
--
(Original announcement)
In the recent mini-summit at OLS 2008 and the following days it was
agreed to tackle the checkpoint/restart (CR) by beginning with a very
simple case: save and restore a single task, with simple memory
layout, disregarding other task state such as files, signals etc.
Following these discussions I coded a prototype that can do exactly
that, as a starter. This code adds two system calls - sys_checkpoint
and sys_restart - that a task can call to save and restore its state
respectively. It also demonstrates how the checkpoint image file can
be formatted, as well as show its nested nature (e.g. cr_write_mm()
-> cr_write_vma() nesting).
The state that is saved/restored is the following:
* some of the task_struct
* some of the thread_struct and thread_info
* the cpu state (including FPU)
* the memory address space
In the current code, sys_checkpoint will checkpoint the current task,
although the logic exists to checkpoint other tasks (not in the
checkpointee's execution context). A simple loop will extend this to
handle multiple processes. sys_restart restarts the current tasks, and
with multiple tasks each task will call the syscall independently.
(Actually, to checkpoint outside the context of a task, it is also
necessary to also handle restart-block logic when saving/restoring the
thread data).
It takes longer to describe what isn't implemented or supported by
this prototype ... basically everything that isn't as simple as the
above.
As for containers - since we still don't have a representation for a
container, this patch has no notion of a container. The tests for
consistent namespaces (and isolation) are also omitted.
@@ -0,0 +1,198 @@+/*+*Checkpoint/restart-architecturespecificsupportforx86+*+*Copyright(C)2008OrenLaadan+*+*ThisfileissubjecttothetermsandconditionsoftheGNUGeneralPublic+*License.SeethefileCOPYINGinthemaindirectoryoftheLinux+*distributionformoredetails.+*/++#include<asm/desc.h>+#include<asm/i387.h>++#include<linux/checkpoint.h>+#include<linux/checkpoint_hdr.h>++/* dump the thread_struct of a given task */+intcr_write_thread(structcr_ctx*ctx,structtask_struct*t)+{+structcr_hdrh;+structcr_hdr_thread*hh=cr_hbuf_get(ctx,sizeof(*hh));+structthread_struct*thread;+structdesc_struct*desc;+intntls=0;+intn,ret;++h.type=CR_HDR_THREAD;+h.len=sizeof(*hh);+h.parent=task_pid_vnr(t);++thread=&t->thread;++/* calculate no. of TLS entries that follow */+desc=thread->tls_array;+for(n=GDT_ENTRY_TLS_ENTRIES;n>0;n--,desc++){+if(desc->a||desc->b)+ntls++;+}++hh->gdt_entry_tls_entries=GDT_ENTRY_TLS_ENTRIES;+hh->sizeof_tls_array=sizeof(thread->tls_array);+hh->ntls=ntls;++ret=cr_write_obj(ctx,&h,hh);+cr_hbuf_put(ctx,sizeof(*hh));+if(ret<0)+returnret;++/* for simplicity dump the entire array, cherry-pick upon restart */+ret=cr_kwrite(ctx,thread->tls_array,sizeof(thread->tls_array));++cr_debug("ntls %d\n",ntls);++/* IGNORE RESTART BLOCKS FOR NOW ... */++returnret;+}++#ifdef CONFIG_X86_64++#error "CONFIG_X86_64 unsupported yet."++#else /* !CONFIG_X86_64 */++voidcr_write_cpu_regs(structcr_hdr_cpu*hh,structtask_struct*t)+{+structthread_struct*thread=&t->thread;+structpt_regs*regs=task_pt_regs(t);++hh->bp=regs->bp;+hh->bx=regs->bx;+hh->ax=regs->ax;+hh->cx=regs->cx;+hh->dx=regs->dx;+hh->si=regs->si;+hh->di=regs->di;+hh->orig_ax=regs->orig_ax;+hh->ip=regs->ip;+hh->cs=regs->cs;+hh->flags=regs->flags;+hh->sp=regs->sp;+hh->ss=regs->ss;++hh->ds=regs->ds;+hh->es=regs->es;++/*+*forcheckpointinprocesscontext(fromwithinacontainer)+*theGSandFSregistersshouldbesavedfromthehardware;+*otherwisetheyarealreadysabedonthethreadstructure+*/+if(t==current){+savesegment(gs,hh->gs);+savesegment(fs,hh->fs);+}else{+hh->gs=thread->gs;+hh->fs=thread->fs;+}++/*+*forcheckpointinprocesscontext(fromwithinacontainer),+*theactualsyscallistakingplaceatthisverymoment;so+*we(optimistically)subtitutethefuturereturnvalue(0)of+*thissyscallintotheorig_eax,sothatuponrestartitwill+*succeed(oritwillendlesslyretrycheckpoint...)+*/+if(t==current){+BUG_ON(hh->orig_ax<0);+hh->ax=0;+}+}++voidcr_write_cpu_debug(structcr_hdr_cpu*hh,structtask_struct*t)+{+structthread_struct*thread=&t->thread;++/* debug regs */++preempt_disable();++/*+*forcheckpointinprocesscontext(fromwithinacontainer),+*gettheactualregisters;otherwisegetthesavedvalues.+*/++if(t==current){+get_debugreg(hh->debugreg0,0);+get_debugreg(hh->debugreg1,1);+get_debugreg(hh->debugreg2,2);+get_debugreg(hh->debugreg3,3);+get_debugreg(hh->debugreg6,6);+get_debugreg(hh->debugreg7,7);+}else{+hh->debugreg0=thread->debugreg0;+hh->debugreg1=thread->debugreg1;+hh->debugreg2=thread->debugreg2;+hh->debugreg3=thread->debugreg3;+hh->debugreg6=thread->debugreg6;+hh->debugreg7=thread->debugreg7;+}++hh->debugreg4=0;+hh->debugreg5=0;++hh->uses_debug=!!(task_thread_info(t)->flags&TIF_DEBUG);++preempt_enable();+}++voidcr_write_cpu_fpu(structcr_hdr_cpu*hh,structtask_struct*t)+{+structthread_struct*thread=&t->thread;+structthread_info*thread_info=task_thread_info(t);++/* i387 + MMU + SSE logic */++preempt_disable();++hh->used_math=tsk_used_math(t)?1:0;+if(hh->used_math){+/*+*normally,noneedtounlazy_fpu(),sinceTS_USEDFPUflag+*havebeenclearedwhentaskwasconexted-switchedout...+*exceptifweareinprocesscontext,inwhichcasewedo+*/+if(thread_info->status&TS_USEDFPU)+unlazy_fpu(current);++hh->has_fxsr=cpu_has_fxsr;+memcpy(&hh->xstate,thread->xstate,sizeof(*thread->xstate));+}++preempt_enable();+}++#endif /* CONFIG_X86_64 */++/* dump the cpu state and registers of a given task */+intcr_write_cpu(structcr_ctx*ctx,structtask_struct*t)+{+structcr_hdrh;+structcr_hdr_cpu*hh=cr_hbuf_get(ctx,sizeof(*hh));+intret;++h.type=CR_HDR_CPU;+h.len=sizeof(*hh);+h.parent=task_pid_vnr(t);++cr_write_cpu_regs(hh,t);+cr_write_cpu_debug(hh,t);+cr_write_cpu_fpu(hh,t);++cr_debug("math %d debug %d\n",hh->used_math,hh->uses_debug);++ret=cr_write_obj(ctx,&h,hh);+cr_hbuf_put(ctx,sizeof(*hh));+returnret;+}
@@ -0,0 +1,194 @@+/*+*Checkpoint/restart-architecturespecificsupportforx86+*+*Copyright(C)2008OrenLaadan+*+*ThisfileissubjecttothetermsandconditionsoftheGNUGeneralPublic+*License.SeethefileCOPYINGinthemaindirectoryoftheLinux+*distributionformoredetails.+*/++#include<asm/desc.h>+#include<asm/i387.h>++#include<linux/checkpoint.h>+#include<linux/checkpoint_hdr.h>++/* read the thread_struct into the current task */+intcr_read_thread(structcr_ctx*ctx)+{+structcr_hdr_thread*hh=cr_hbuf_get(ctx,sizeof(*hh));+structtask_struct*t=current;+structthread_struct*thread=&t->thread;+intparent,ret;++parent=cr_read_obj_type(ctx,hh,sizeof(*hh),CR_HDR_THREAD);+if(parent<0){+ret=parent;+gotoout;+}++ret=-EINVAL;++#if 0 /* activate when containers are used */+if(parent!=task_pid_vnr(t))+gotoout;+#endif+cr_debug("ntls %d\n",hh->ntls);++if(hh->gdt_entry_tls_entries!=GDT_ENTRY_TLS_ENTRIES||+hh->sizeof_tls_array!=sizeof(thread->tls_array)||+hh->ntls<0||hh->ntls>GDT_ENTRY_TLS_ENTRIES)+gotoout;++if(hh->ntls>0){+structdesc_struct*desc;+intsize,cpu;++/*+*restoreTLSbyhand:whyconverttostructuser_descif+*sys_set_thread_entry()willconvertitback?+*/++size=sizeof(*desc)*GDT_ENTRY_TLS_ENTRIES;+desc=kmalloc(size,GFP_KERNEL);+if(!desc)+return-ENOMEM;++ret=cr_kread(ctx,desc,size);+if(ret>=0){+/*+*FIX:addsanitychecks(eg.thatvaluesmakes+*sense,thatwedon'toverwriteoldvalues,etc+*/+cpu=get_cpu();+memcpy(thread->tls_array,desc,size);+load_TLS(thread,cpu);+put_cpu();+}+kfree(desc);+}++ret=0;+out:+cr_hbuf_put(ctx,sizeof(*hh));+returnret;+}++#ifdef CONFIG_X86_64++#error "CONFIG_X86_64 unsupported yet."++#else /* !CONFIG_X86_64 */++intcr_read_cpu_regs(structcr_hdr_cpu*hh,structtask_struct*t)+{+structthread_struct*thread=&t->thread;+structpt_regs*regs=task_pt_regs(t);++regs->bx=hh->bx;+regs->cx=hh->cx;+regs->dx=hh->dx;+regs->si=hh->si;+regs->di=hh->di;+regs->bp=hh->bp;+regs->ax=hh->ax;+regs->ds=hh->ds;+regs->es=hh->es;+regs->orig_ax=hh->orig_ax;+regs->ip=hh->ip;+regs->cs=hh->cs;+regs->flags=hh->flags;+regs->sp=hh->sp;+regs->ss=hh->ss;++thread->gs=hh->gs;+thread->fs=hh->fs;+loadsegment(gs,hh->gs);+loadsegment(fs,hh->fs);++return0;+}++intcr_read_cpu_debug(structcr_hdr_cpu*hh,structtask_struct*t)+{+/* debug regs */++if(hh->uses_debug){+set_debugreg(hh->debugreg0,0);+set_debugreg(hh->debugreg1,1);+/* ignore 4, 5 */+set_debugreg(hh->debugreg2,2);+set_debugreg(hh->debugreg3,3);+set_debugreg(hh->debugreg6,6);+set_debugreg(hh->debugreg7,7);+}++return0;+}++intcr_read_cpu_fpu(structcr_hdr_cpu*hh,structtask_struct*t)+{+structthread_struct*thread=&t->thread;+intret;++/* i387 + MMU + SSE */++preempt_disable();++__clear_fpu(t);/* in case we used FPU in user mode */++if(!hh->used_math)+clear_used_math();+else{+if(hh->has_fxsr!=cpu_has_fxsr){+force_sig(SIGFPE,t);+return-EINVAL;+}+/* init_fpu() also calls set_used_math() */+ret=init_fpu(current);+if(ret<0)+returnret;+memcpy(thread->xstate,&hh->xstate,sizeof(*thread->xstate));+}++preempt_enable();+return0;+}++#endif /* CONFIG_X86_64 */++/* read the cpu state and registers for the current task */+intcr_read_cpu(structcr_ctx*ctx)+{+structcr_hdr_cpu*hh=cr_hbuf_get(ctx,sizeof(*hh));+structtask_struct*t=current;+intparent,ret;++parent=cr_read_obj_type(ctx,hh,sizeof(*hh),CR_HDR_CPU);+if(parent<0){+ret=parent;+gotoout;+}++ret=-EINVAL;++#if 0 /* activate when containers are used */+if(parent!=task_pid_vnr(t))+gotoout;+#endif+/* FIX: sanity check for sensitive registers (eg. eflags) */++ret=cr_read_cpu_regs(hh,t);+if(ret<0)+gotoout;+ret=cr_read_cpu_debug(hh,t);+if(ret<0)+gotoout;+ret=cr_read_cpu_fpu(hh,t);++cr_debug("math %d debug %d\n",hh->used_math,hh->uses_debug);+out:+cr_hbuf_put(ctx,sizeof(*hh));+returnret;+}
@@ -145,8 +147,17 @@ static int cr_write_task(struct cr_ctx *ctx, struct task_struct *t)}ret=cr_write_task_struct(ctx,t);-cr_debug("ret %d\n",ret);+cr_debug("task_struct: ret %d\n",ret);+if(ret<0)+gotoout;+ret=cr_write_thread(ctx,t);+cr_debug("thread: ret %d\n",ret);+if(ret<0)+gotoout;+ret=cr_write_cpu(ctx,t);+cr_debug("cpu: ret %d\n",ret);+out:returnret;}
@@ -172,8 +174,17 @@ static int cr_read_task(struct cr_ctx *ctx)intret;ret=cr_read_task_struct(ctx);-cr_debug("ret %d\n",ret);+cr_debug("task_struct: ret %d\n",ret);+if(ret<0)+gotoout;+ret=cr_read_thread(ctx);+cr_debug("thread: ret %d\n",ret);+if(ret<0)+gotoout;+ret=cr_read_cpu(ctx);+cr_debug("cpu: ret %d\n",ret);+out:returnret;}
Add those interfaces, as well as helpers needed to easily manage the
file format. The code is roughly broken out as follows:
checkpoint/sys.c - user/kernel data transfer, as well as setup of the
checkpoint/restart context (a per-checkpoint data structure for
housekeeping)
checkpoint/checkpoint.c - output wrappers and basic checkpoint handling
checkpoint/restart.c - input wrappers and basic restart handling
Patches to add the per-architecture support as well as the actual
work to do the memory checkpoint follow in subsequent patches.
Signed-off-by: Oren Laadan <redacted>
Acked-by: Serge Hallyn <redacted>
Signed-off-by: Dave Hansen <redacted>
---
Makefile | 2 +-
checkpoint/Makefile | 2 +-
checkpoint/checkpoint.c | 174 +++++++++++++++++++++++++++++++
checkpoint/restart.c | 197 ++++++++++++++++++++++++++++++++++++
checkpoint/sys.c | 219 +++++++++++++++++++++++++++++++++++++++-
fs/read_write.c | 4 +-
include/linux/checkpoint.h | 60 +++++++++++
include/linux/checkpoint_hdr.h | 75 ++++++++++++++
include/linux/magic.h | 3 +
9 files changed, 728 insertions(+), 8 deletions(-)
create mode 100644 checkpoint/checkpoint.c
create mode 100644 checkpoint/restart.c
create mode 100644 include/linux/checkpoint.h
create mode 100644 include/linux/checkpoint_hdr.h
@@ -2,4 +2,4 @@# Makefile for linux checkpoint/restart.#-obj-$(CONFIG_CHECKPOINT_RESTART)+=sys.o+obj-$(CONFIG_CHECKPOINT_RESTART)+=sys.ocheckpoint.orestart.o
@@ -0,0 +1,174 @@+/*+*Checkpointlogicandhelpers+*+*Copyright(C)2008OrenLaadan+*+*ThisfileissubjecttothetermsandconditionsoftheGNUGeneralPublic+*License.SeethefileCOPYINGinthemaindirectoryoftheLinux+*distributionformoredetails.+*/++#include<linux/version.h>+#include<linux/sched.h>+#include<linux/time.h>+#include<linux/fs.h>+#include<linux/file.h>+#include<linux/dcache.h>+#include<linux/mount.h>+#include<linux/utsname.h>+#include<linux/magic.h>+#include<linux/checkpoint.h>+#include<linux/checkpoint_hdr.h>++/**+*cr_write_obj-writearecorddescribedbyacr_hdr+*@ctx:checkpointcontext+*@h:recorddescriptor+*@buf:recordbuffer+*/+intcr_write_obj(structcr_ctx*ctx,structcr_hdr*h,void*buf)+{+intret;++ret=cr_kwrite(ctx,h,sizeof(*h));+if(ret<0)+returnret;+returncr_kwrite(ctx,buf,h->len);+}++/**+*cr_write_string-writeastring+*@ctx:checkpointcontext+*@str:stringpointer+*@len:stringlength+*/+intcr_write_string(structcr_ctx*ctx,char*str,intlen)+{+structcr_hdrh;++h.type=CR_HDR_STRING;+h.len=len;+h.parent=0;++returncr_write_obj(ctx,&h,str);+}++/* write the checkpoint header */+staticintcr_write_head(structcr_ctx*ctx)+{+structcr_hdrh;+structcr_hdr_head*hh=cr_hbuf_get(ctx,sizeof(*hh));+structnew_utsname*uts;+structtimevalktv;+intret;++h.type=CR_HDR_HEAD;+h.len=sizeof(*hh);+h.parent=0;++do_gettimeofday(&ktv);++hh->magic=CHECKPOINT_MAGIC_HEAD;+hh->major=(LINUX_VERSION_CODE>>16)&0xff;+hh->minor=(LINUX_VERSION_CODE>>8)&0xff;+hh->patch=(LINUX_VERSION_CODE)&0xff;++hh->rev=CR_VERSION;++hh->flags=ctx->flags;+hh->time=ktv.tv_sec;++uts=utsname();+memcpy(hh->release,uts->release,__NEW_UTS_LEN);+memcpy(hh->version,uts->version,__NEW_UTS_LEN);+memcpy(hh->machine,uts->machine,__NEW_UTS_LEN);++ret=cr_write_obj(ctx,&h,hh);+cr_hbuf_put(ctx,sizeof(*hh));+returnret;+}++/* write the checkpoint trailer */+staticintcr_write_tail(structcr_ctx*ctx)+{+structcr_hdrh;+structcr_hdr_tail*hh=cr_hbuf_get(ctx,sizeof(*hh));+intret;++h.type=CR_HDR_TAIL;+h.len=sizeof(*hh);+h.parent=0;++hh->magic=CHECKPOINT_MAGIC_TAIL;++ret=cr_write_obj(ctx,&h,hh);+cr_hbuf_put(ctx,sizeof(*hh));+returnret;+}++/* dump the task_struct of a given task */+staticintcr_write_task_struct(structcr_ctx*ctx,structtask_struct*t)+{+structcr_hdrh;+structcr_hdr_task*hh=cr_hbuf_get(ctx,sizeof(*hh));+intret;++h.type=CR_HDR_TASK;+h.len=sizeof(*hh);+h.parent=0;++hh->state=t->state;+hh->exit_state=t->exit_state;+hh->exit_code=t->exit_code;+hh->exit_signal=t->exit_signal;++hh->task_comm_len=TASK_COMM_LEN;++/* FIXME: save remaining relevant task_struct fields */++ret=cr_write_obj(ctx,&h,hh);+cr_hbuf_put(ctx,sizeof(*hh));+if(ret<0)+returnret;++returncr_write_string(ctx,t->comm,TASK_COMM_LEN);+}++/* dump the entire state of a given task */+staticintcr_write_task(structcr_ctx*ctx,structtask_struct*t)+{+intret;++if(t->state==TASK_DEAD){+pr_warning("CR: task may not be in state TASK_DEAD\n");+return-EAGAIN;+}++ret=cr_write_task_struct(ctx,t);+cr_debug("ret %d\n",ret);++returnret;+}++intdo_checkpoint(structcr_ctx*ctx)+{+intret;++/* FIX: need to test whether container is checkpointable */++ret=cr_write_head(ctx);+if(ret<0)+gotoout;+ret=cr_write_task(ctx,current);+if(ret<0)+gotoout;+ret=cr_write_tail(ctx);+if(ret<0)+gotoout;++/* on success, return (unique) checkpoint identifier */+ret=ctx->crid;++out:+returnret;+}
@@ -0,0 +1,197 @@+/*+*Restartlogicandhelpers+*+*Copyright(C)2008OrenLaadan+*+*ThisfileissubjecttothetermsandconditionsoftheGNUGeneralPublic+*License.SeethefileCOPYINGinthemaindirectoryoftheLinux+*distributionformoredetails.+*/++#include<linux/version.h>+#include<linux/sched.h>+#include<linux/file.h>+#include<linux/magic.h>+#include<linux/checkpoint.h>+#include<linux/checkpoint_hdr.h>++/**+*cr_read_obj-readawholerecord(cr_hdrfollowedbypayload)+*@ctx:checkpointcontext+*@h:recorddescriptor+*@buf:recordbuffer+*@n:availablebuffersize+*+*Returnssizeofpayload+*/+intcr_read_obj(structcr_ctx*ctx,structcr_hdr*h,void*buf,intn)+{+intret;++ret=cr_kread(ctx,h,sizeof(*h));+if(ret<0)+returnret;++cr_debug("type %d len %d parent %d\n",h->type,h->len,h->parent);++if(h->len<0||h->len>n)+return-EINVAL;++returncr_kread(ctx,buf,h->len);+}++/**+*cr_read_obj_type-readawholerecordofexpectedtype+*@ctx:checkpointcontext+*@buf:recordbuffer+*@n:availablebuffersize+*@type:expectedrecordtype+*+*Returnsobjectreferenceoftheparentobject+*/+intcr_read_obj_type(structcr_ctx*ctx,void*buf,intn,inttype)+{+structcr_hdrh;+intret;++ret=cr_read_obj(ctx,&h,buf,n);+if(ret<0)+returnret;++ret=-EINVAL;+if(h.type==type)+ret=h.parent;++returnret;+}++/**+*cr_read_string-readastring+*@ctx:checkpointcontext+*@str:stringbuffer+*@len:bufferbufferlength+*/+intcr_read_string(structcr_ctx*ctx,void*str,intlen)+{+returncr_read_obj_type(ctx,str,len,CR_HDR_STRING);+}++/* read the checkpoint header */+staticintcr_read_head(structcr_ctx*ctx)+{+structcr_hdr_head*hh=cr_hbuf_get(ctx,sizeof(*hh));+intparent,ret=-EINVAL;++parent=cr_read_obj_type(ctx,hh,sizeof(*hh),CR_HDR_HEAD);+if(parent<0){+ret=parent;+gotoout;+}elseif(parent!=0)+gotoout;++if(hh->magic!=CHECKPOINT_MAGIC_HEAD||hh->rev!=CR_VERSION||+hh->major!=((LINUX_VERSION_CODE>>16)&0xff)||+hh->minor!=((LINUX_VERSION_CODE>>8)&0xff)||+hh->patch!=((LINUX_VERSION_CODE)&0xff))+gotoout;++if(hh->flags&~CR_CTX_CKPT)+gotoout;++ctx->oflags=hh->flags;++/* FIX: verify compatibility of release, version and machine */++ret=0;+out:+cr_hbuf_put(ctx,sizeof(*hh));+returnret;+}++/* read the checkpoint trailer */+staticintcr_read_tail(structcr_ctx*ctx)+{+structcr_hdr_tail*hh=cr_hbuf_get(ctx,sizeof(*hh));+intparent,ret=-EINVAL;++parent=cr_read_obj_type(ctx,hh,sizeof(*hh),CR_HDR_TAIL);+if(parent<0){+ret=parent;+gotoout;+}elseif(parent!=0)+gotoout;++if(hh->magic!=CHECKPOINT_MAGIC_TAIL)+gotoout;++ret=0;+out:+cr_hbuf_put(ctx,sizeof(*hh));+returnret;+}++/* read the task_struct into the current task */+staticintcr_read_task_struct(structcr_ctx*ctx)+{+structcr_hdr_task*hh=cr_hbuf_get(ctx,sizeof(*hh));+structtask_struct*t=current;+char*buf;+intparent,ret=-EINVAL;++parent=cr_read_obj_type(ctx,hh,sizeof(*hh),CR_HDR_TASK);+if(parent<0){+ret=parent;+gotoout;+}elseif(parent!=0)+gotoout;++/* upper limit for task_comm_len to prevent DoS */+if(hh->task_comm_len<0||hh->task_comm_len>PAGE_SIZE)+gotoout;++buf=kmalloc(hh->task_comm_len,GFP_KERNEL);+if(!buf)+gotoout;+ret=cr_read_string(ctx,buf,hh->task_comm_len);+if(!ret){+/* if t->comm is too long, silently truncate */+memset(t->comm,0,TASK_COMM_LEN);+memcpy(t->comm,buf,min(hh->task_comm_len,TASK_COMM_LEN));+}+kfree(buf);++/* FIXME: restore remaining relevant task_struct fields */+out:+cr_hbuf_put(ctx,sizeof(*hh));+returnret;+}++/* read the entire state of the current task */+staticintcr_read_task(structcr_ctx*ctx)+{+intret;++ret=cr_read_task_struct(ctx);+cr_debug("ret %d\n",ret);++returnret;+}++intdo_restart(structcr_ctx*ctx)+{+intret;++ret=cr_read_head(ctx);+if(ret<0)+gotoout;+ret=cr_read_task(ctx);+if(ret<0)+gotoout;+ret=cr_read_tail(ctx);+if(ret<0)+gotoout;++/* on success, adjust the return value if needed [TODO] */+out:+returnret;+}
@@ -10,6 +10,187 @@#include<linux/sched.h>#include<linux/kernel.h>+#include<linux/fs.h>+#include<linux/file.h>+#include<linux/uaccess.h>+#include<linux/capability.h>+#include<linux/checkpoint.h>++/*+*helperstowrite/readto/fromtheimagefiledescriptor+*+*cr_uwrite()-writeauser-spacebuffertothecheckpointimage+*cr_kwrite()-writeakernel-spacebuffertothecheckpointimage+*cr_uread()-readfromthecheckpointimagetoauser-spacebuffer+*cr_kread()-readfromthecheckpointimagetoakernel-spacebuffer+*/++intcr_uwrite(structcr_ctx*ctx,void*buf,intcount)+{+structfile*file=ctx->file;+ssize_tnwrite;+intnleft;++for(nleft=count;nleft;nleft-=nwrite){+loff_tpos=file_pos_read(file);+nwrite=vfs_write(file,(char__user*)buf,nleft,&pos);+file_pos_write(file,pos);+if(nwrite<=0){+if(nwrite==-EAGAIN)+nwrite=0;+else+returnnwrite;+}+buf+=nwrite;+}++ctx->total+=count;+return0;+}++intcr_kwrite(structcr_ctx*ctx,void*buf,intcount)+{+mm_segment_toldfs;+intret;++oldfs=get_fs();+set_fs(KERNEL_DS);+ret=cr_uwrite(ctx,buf,count);+set_fs(oldfs);++returnret;+}++intcr_uread(structcr_ctx*ctx,void*buf,intcount)+{+structfile*file=ctx->file;+ssize_tnread;+intnleft;++for(nleft=count;nleft;nleft-=nread){+loff_tpos=file_pos_read(file);+nread=vfs_read(file,(char__user*)buf,nleft,&pos);+file_pos_write(file,pos);+if(nread<=0){+if(nread==-EAGAIN)+nread=0;+else+returnnread;+}+buf+=nread;+}++ctx->total+=count;+return0;+}++intcr_kread(structcr_ctx*ctx,void*buf,intcount)+{+mm_segment_toldfs;+intret;++oldfs=get_fs();+set_fs(KERNEL_DS);+ret=cr_uread(ctx,buf,count);+set_fs(oldfs);++returnret;+}++/*+*Duringcheckpointandrestartthecodewritesouts/readsindata+*to/fromthechekcpointimagefrom/toatemporarybuffer(ctx->hbuf).+*Becauseoperationscanbenested,oneshouldcallcr_hbuf_get()to+*reservespaceinthebuffer,andthencr_hbuf_put()whennolonger+*needsthatspace.+*/++/*+*ctx->hbufisusedtoholdheadersanddataofknown(orbound),+*staticsizes.Insomecases,multipleheadersmaybeallocatedin+*anestedmanner.Thesizeshouldaccommodateallheaders,nested+*ornot,onallarchs.+*/+#define CR_HBUF_TOTAL (8 * 4096)++/**+*cr_hbuf_get-reservespaceonthehbuf+*@ctx:checkpointcontext+*@n:numberofbytestoreserve+*+*Returnspointertoreservedspace+*/+void*cr_hbuf_get(structcr_ctx*ctx,intn)+{+void*ptr;++/*+*Sincerequestsdependonlogicandstaticheadersizes(noton+*userdata),spaceshouldalwayssuffice,unlesssomeoneeither+*madeastructurebiggerorcallpathdeeperthanexpected.+*/+BUG_ON(ctx->hpos+n>CR_HBUF_TOTAL);+ptr=ctx->hbuf+ctx->hpos;+ctx->hpos+=n;+returnptr;+}++/**+*cr_hbuf_put-unreservespaceonthehbuf+*@ctx:checkpointcontext+*@n:numberofbytestoreserve+*/+voidcr_hbuf_put(structcr_ctx*ctx,intn)+{+BUG_ON(ctx->hpos<n);+ctx->hpos-=n;+}++/*+*helperstomanageCRcontexts:allocatedforeachcheckpointand/or+*restartoperation,andpersistsuntiltheoperationiscompleted.+*/++/* unique checkpoint identifier (FIXME: should be per-container) */+staticatomic_tcr_ctx_count;++voidcr_ctx_free(structcr_ctx*ctx)+{+if(ctx->file)+fput(ctx->file);++kfree(ctx->hbuf);++kfree(ctx);+}++structcr_ctx*cr_ctx_alloc(pid_tpid,intfd,unsignedlongflags)+{+structcr_ctx*ctx;++ctx=kzalloc(sizeof(*ctx),GFP_KERNEL);+if(!ctx)+returnERR_PTR(-ENOMEM);++ctx->file=fget(fd);+if(!ctx->file){+cr_ctx_free(ctx);+returnERR_PTR(-EBADF);+}++ctx->hbuf=kmalloc(CR_HBUF_TOTAL,GFP_KERNEL);+if(!ctx->hbuf){+cr_ctx_free(ctx);+returnERR_PTR(-ENOMEM);+}++ctx->pid=pid;+ctx->flags=flags;++ctx->crid=atomic_inc_return(&cr_ctx_count);++returnctx;+}/***sys_checkpoint-checkpointacontainer
@@ -22,9 +203,26 @@*/asmlinkagelongsys_checkpoint(pid_tpid,intfd,unsignedlongflags){-pr_debug("sys_checkpoint not implemented yet\n");-return-ENOSYS;+structcr_ctx*ctx;+intret;++/* no flags for now */+if(flags)+return-EINVAL;++ctx=cr_ctx_alloc(pid,fd,flags|CR_CTX_CKPT);+if(IS_ERR(ctx))+returnPTR_ERR(ctx);++ret=do_checkpoint(ctx);++if(!ret)+ret=ctx->crid;++cr_ctx_free(ctx);+returnret;}+/***sys_restart-restartacontainer*@crid:checkpointimageidentifier
@@ -36,6 +234,19 @@ asmlinkage long sys_checkpoint(pid_t pid, int fd, unsigned long flags)*/asmlinkagelongsys_restart(intcrid,intfd,unsignedlongflags){-pr_debug("sys_restart not implemented yet\n");-return-ENOSYS;+structcr_ctx*ctx;+intret;++/* no flags for now */+if(flags)+return-EINVAL;++ctx=cr_ctx_alloc(crid,fd,flags|CR_CTX_RSTR);+if(IS_ERR(ctx))+returnPTR_ERR(ctx);++ret=do_restart(ctx);++cr_ctx_free(ctx);+returnret;}
@@ -0,0 +1,60 @@+#ifndef _CHECKPOINT_CKPT_H_+#define _CHECKPOINT_CKPT_H_+/*+*Genericcontainercheckpoint-restart+*+*Copyright(C)2008OrenLaadan+*+*ThisfileissubjecttothetermsandconditionsoftheGNUGeneralPublic+*License.SeethefileCOPYINGinthemaindirectoryoftheLinux+*distributionformoredetails.+*/++#define CR_VERSION 1++structcr_ctx{+pid_tpid;/* container identifier */+intcrid;/* unique checkpoint id */++unsignedlongflags;+unsignedlongoflags;/* restart: old flags */++structfile*file;+inttotal;/* total read/written */++void*hbuf;/* temporary buffer for headers */+inthpos;/* position in headers buffer */+};++/* cr_ctx: flags */+#define CR_CTX_CKPT 0x1+#define CR_CTX_RSTR 0x2++externintcr_uwrite(structcr_ctx*ctx,void*buf,intcount);+externintcr_kwrite(structcr_ctx*ctx,void*buf,intcount);+externintcr_uread(structcr_ctx*ctx,void*buf,intcount);+externintcr_kread(structcr_ctx*ctx,void*buf,intcount);++externvoid*cr_hbuf_get(structcr_ctx*ctx,intn);+externvoidcr_hbuf_put(structcr_ctx*ctx,intn);++structcr_hdr;++externintcr_write_obj(structcr_ctx*ctx,structcr_hdr*h,void*buf);+externintcr_write_string(structcr_ctx*ctx,char*str,intlen);++externintcr_read_obj(structcr_ctx*ctx,structcr_hdr*h,void*buf,intn);+externintcr_read_obj_type(structcr_ctx*ctx,void*buf,intn,inttype);+externintcr_read_string(structcr_ctx*ctx,void*str,intlen);++externintdo_checkpoint(structcr_ctx*ctx);+externintdo_restart(structcr_ctx*ctx);++/* there are from fs/read_write.c, not exported otherwise in a header */+externloff_tfile_pos_read(structfile*file);+externvoidfile_pos_write(structfile*file,loff_tpos);++#define cr_debug(fmt, args...) \+pr_debug("[CR:%s] "fmt,__func__,##args)++#endif /* _CHECKPOINT_CKPT_H_ */
Create trivial sys_checkpoint and sys_restore system calls. They will
enable to checkpoint and restart an entire container, to and from a
checkpoint image file descriptor.
The syscalls take a file descriptor (for the image file) and flags as
arguments. For sys_checkpoint the first argument identifies the target
container; for sys_restart it will identify the checkpoint image.
Signed-off-by: Oren Laadan <redacted>
Acked-by: Serge Hallyn <redacted>
Signed-off-by: Dave Hansen <redacted>
---
arch/x86/kernel/syscall_table_32.S | 2 +
checkpoint/Kconfig | 11 +++++++++
checkpoint/Makefile | 5 ++++
checkpoint/sys.c | 41 ++++++++++++++++++++++++++++++++++++
include/asm-x86/unistd_32.h | 2 +
include/linux/syscalls.h | 2 +
init/Kconfig | 2 +
kernel/sys_ni.c | 4 +++
8 files changed, 69 insertions(+), 0 deletions(-)
create mode 100644 checkpoint/Kconfig
create mode 100644 checkpoint/Makefile
create mode 100644 checkpoint/sys.c
@@ -0,0 +1,41 @@+/*+*Genericcontainercheckpoint-restart+*+*Copyright(C)2008OrenLaadan+*+*ThisfileissubjecttothetermsandconditionsoftheGNUGeneralPublic+*License.SeethefileCOPYINGinthemaindirectoryoftheLinux+*distributionformoredetails.+*/++#include<linux/sched.h>+#include<linux/kernel.h>++/**+*sys_checkpoint-checkpointacontainer+*@pid:pidofthecontainerinit(1)process+*@fd:filetowhichdumpthecheckpointimage+*@flags:checkpointoperationflags+*+*Returnspositiveidentifieronsuccess,0whenreturningfromrestart+*ornegativevalueonerror+*/+asmlinkagelongsys_checkpoint(pid_tpid,intfd,unsignedlongflags)+{+pr_debug("sys_checkpoint not implemented yet\n");+return-ENOSYS;+}+/**+*sys_restart-restartacontainer+*@crid:checkpointimageidentifier+*@fd:filefromwhichreadthecheckpointimage+*@flags:restartoperationflags+*+*Returnsnegativevalueonerror,orotherwisereturnsintherealm+*oftheoriginalcheckpoint+*/+asmlinkagelongsys_restart(intcrid,intfd,unsignedlongflags)+{+pr_debug("sys_restart not implemented yet\n");+return-ENOSYS;+}
@@ -0,0 +1,374 @@++ === Checkpoint-Restart support in the Linux kernel ===++Copyright (C) 2008 Oren Laadan++Author: Oren Laadan <orenl@cs.columbia.edu>++License: The GNU Free Documentation License, Version 1.2+ (dual licensed under the GPL v2)+Reviewers:++Application checkpoint/restart [CR] is the ability to save the state+of a running application so that it can later resume its execution+from the time at which it was checkpointed. An application can be+migrated by checkpointing it on one machine and restarting it on+another. CR can provide many potential benefits:++* Failure recovery: by rolling back an to a previous checkpoint++* Improved response time: by restarting applications from checkpoints+ instead of from scratch.++* Improved system utilization: by suspending long running CPU+ intensive jobs and resuming them when load decreases.++* Fault resilience: by migrating applications off of faulty hosts.++* Dynamic load balancing: by migrating applications to less loaded+ hosts.++* Improved service availability and administration: by migrating+ applications before host maintenance so that they continue to run+ with minimal downtime++* Time-travel: by taking periodic checkpoints and restarting from+ any previous checkpoint.+++=== Overall design++Checkpoint and restart is done in the kernel as much as possible. The+kernel exports a relative opaque 'blob' of data to userspace which can+then be handed to the new kernel at restore time. The 'blob' contains+data and state of select portions of kernel structures such as VMAs+and mm_structs, as well as copies of the actual memory that the tasks+use. Any changes in this blob's format between kernel revisions can be+handled by an in-userspace conversion program. The approach is similar+to virtually all of the commercial CR products out there, as well as+the research project Zap.++Two new system calls are introduced to provide CR: sys_checkpoint and+sys_restart. The checkpoint code basically serializes internal kernel+state and writes it out to a file descriptor, and the resulting image+is stream-able. More specifically, it consists of 5 steps:+ 1. Pre-dump+ 2. Freeze the container+ 3. Dump+ 4. Thaw (or kill) the container+ 5. Post-dump+Steps 1 and 5 are an optimization to reduce application downtime:+"pre-dump" works before freezing the container, e.g. the pre-copy for+live migration, and "post-dump" works after the container resumes+execution, e.g. write-back the data to secondary storage.++The restart code basically reads the saved kernel state and from a+file descriptor, and re-creates the tasks and the resources they need+to resume execution. The restart code is executed by each task that+is restored in a new container to reconstruct its own state.+++=== Interfaces++int sys_checkpoint(pid_t pid, int fd, unsigned long flag);+ Checkpoint a container whose init task is identified by pid, to the+ file designated by fd. Flags will have future meaning (should be 0+ for now).+ Returns: a positive integer that identifies the checkpoint image+ (for future reference in case it is kept in memory) upon success,+ 0 if it returns from a restart, and -1 if an error occurs.++int sys_restart(int crid, int fd, unsigned long flags);+ Restart a container from a checkpoint image identified by crid, or+ from the blob stored in the file designated by fd. Flags will have+ future meaning (should be 0 for now).+ Returns: 0 on success and -1 if an error occurs.++Thus, if checkpoint is initiated by a process in the container, one+can use logic similar to fork():+ ...+ crid = checkpoint(...);+ switch (crid) {+ case -1:+ perror("checkpoint failed");+ break;+ default:+ fprintf(stderr, "checkpoint succeeded, CRID=%d\n", ret);+ /* proceed with execution after checkpoint */+ ...+ break;+ case 0:+ fprintf(stderr, "returned after restart\n");+ /* proceed with action required following a restart */+ ...+ break;+ }+ ...+And to initiate a restart, the process in an empty container can use+logic similar to execve():+ ...+ if (restart(crid, ...) < 0)+ perror("restart failed");+ /* only get here if restart failed */+ ...++See below a complete example in C.+++=== Order of state dump++The order of operations, both save and restore, is as following:++* Header section: header, container information, etc.+* Global section: [TBD] global resources such as IPC, UTS, etc.+* Process forest: [TBD] tasks and their relationships+* Per task data (for each task):+ -> task state: elements of task_struct+ -> thread state: elements of thread_struct and thread_info+ -> CPU state: registers etc, including FPU+ -> memory state: memory address space layout and contents+ -> filesystem state: [TBD] filesystem namespace state, chroot, cwd, etc+ -> files state: open file descriptors and their state+ -> signals state: [TBD] pending signals and signal handling state+ -> credentials state: [TBD] user and group state, statistics+++=== Checkpoint image format++The checkpoint image format is composed of records consistings of a+pre-header that identifies its contents, followed by a payload. (The+idea here is to enable parallel checkpointing in the future in which+multiple threads interleave data from multiple processes into a single+stream).++The pre-header is defined by "struct cr_hdr" as follows:++struct cr_hdr {+ __s16 type;+ __s16 len;+ __u32 parent;+};++Here, 'type' field identifies the type of the payload, 'len' tells its+length in bytes. The 'parent' identifies the owner object instance. The+meaning of the 'parent field varies depending on the type. For example,+for type CR_HDR_MM, the 'parent identifies the task to which this MM+belongs. The payload also varies depending on the type, for instance,+the data describing a task_struct is given by a 'struct cr_hdr_task'+(type CR_HDR_TASK) and so on.++The format of the memory dump is as follows: for each VMA, there is a+'struct cr_vma'; if the VMA is file-mapped, it is followed by the file+name. Following comes the actual contents, in one or more chunk: each+chunk begins with a header that specifies how many pages it holds,+then a the virtual addresses of all the dumped pages in that chunk,+followed by the actual contents of all the dumped pages. A header with+zero number of pages marks the end of the contents for a particular+VMA. Then comes the next VMA and so on.++To illustrate this, consider a single simple task with two VMAs: one+is file mapped with two dumped pages, and the other is anonymous with+three dumped pages. The checkpoint image will look like this:++cr_hdr + cr_hdr_head+cr_hdr + cr_hdr_task+ cr_hdr + cr_hdr_mm+ cr_hdr + cr_hdr_vma + cr_hdr + string+ cr_hdr_pgarr (nr_pages = 2)+ addr1, addr2+ page1, page2+ cr_hdr_pgarr (nr_pages = 0)+ cr_hdr + cr_hdr_vma+ cr_hdr_pgarr (nr_pages = 3)+ addr3, addr4, addr5+ page3, page4, page5+ cr_hdr_pgarr (nr_pages = 0)+ cr_hdr + cr_mm_context+ cr_hdr + cr_hdr_thread+ cr_hdr + cr_hdr_cpu+cr_hdr + cr_hdr_tail+++=== Current Implementation++[2008-Oct-07]+There are several assumptions in the current implementation; they will+be gradually relaxed in future versions. The main ones are:+* A task can only checkpoint itself (missing "restart-block" logic).+* Namespaces are not saved or restored; They will be treated as a type+ of shared object.+* In particular, it is assumed that the task's file system namespace+ is the "root" for the entire container.+* It is assumed that the same file system view is available for the+ restart task(s). Otherwise, a file system snapshot is required.+++=== Sample code++Two example programs: one uses checkpoint (called ckpt) to checkpoint+itself, and another uses restart (called rstr) to restart from that+checkpoint. Note the use of "dup2" to create a copy of an open file+and show how shared objects are treated. Execute like this:++orenl:~/test$ ./ckpt > out.1+ <-- ctrl-c+orenl:~/test$ cat /tmp/cr-rest.out+hello, world!+world, hello!+(ret = 1)++orenl:~/test$ ./ckpt > out.1+ <-- ctrl-c+orenl:~/test$ cat /tmp/cr-rest.out+hello, world!+world, hello!+(ret = 2)++ <-- now change the contents of the file+orenl:~/test$ sed -i 's/world, hello!/xxxx/' /tmp/cr-rest.out+orenl:~/test$ cat /tmp/cr-rest.out+hello, world!+xxxx+(ret = 2)++ <-- and do the restart+orenl:~/test$ ./rstr < out.1+ <-- ctrl-c+orenl:~/test$ cat /tmp/cr-rest.out+hello, world!+world, hello!+(ret = 0)++(if you check the output of ps, you'll see that "rstr" changed its+name to "ckpt", as expected).++============================== ckpt.c ================================++#define _GNU_SOURCE /* or _BSD_SOURCE or _SVID_SOURCE */++#include <stdio.h>+#include <stdlib.h>+#include <string.h>+#include <errno.h>+#include <fcntl.h>+#include <unistd.h>+#include <asm/unistd.h>+#include <sys/syscall.h>++#define OUTFILE "/tmp/cr-test.out"++int main(int argc, char *argv[])+{+ pid_t pid = getpid();+ FILE *file;+ int ret;++ close(0);+ close(2);++ unlink(OUTFILE);+ file = fopen(OUTFILE, "w+");+ if (!file) {+ perror("open");+ exit(1);+ }++ if (dup2(0,2) < 0) {+ perror("dups");+ exit(1);+ }++ fprintf(file, "hello, world!\n");+ fflush(file);++ ret = syscall(__NR_checkpoint, pid, STDOUT_FILENO, 0);+ if (ret < 0) {+ perror("checkpoint");+ exit(2);+ }++ fprintf(file, "world, hello!\n");+ fprintf(file, "(ret = %d)\n", ret);+ fflush(file);++ while (1)+ ;++ return 0;+}+======================================================================++============================== rstr.c ================================++#define _GNU_SOURCE /* or _BSD_SOURCE or _SVID_SOURCE */++#include <stdio.h>+#include <stdlib.h>+#include <errno.h>+#include <fcntl.h>+#include <unistd.h>+#include <asm/unistd.h>+#include <sys/syscall.h>++int main(int argc, char *argv[])+{+ pid_t pid = getpid();+ int ret;++ ret = syscall(__NR_restart, pid, STDIN_FILENO, 0);+ if (ret < 0)+ perror("restart");++ printf("should not reach here !\n");++ return 0;+}+======================================================================+++=== Changelog++[2008-Oct-17] v7:+ - Fix save/restore state of FPU+ - Fix argument given to kunmap_atomic() in memory dump/restore++[2008-Oct-07] v6:+ - Balance all calls to cr_hbuf_get() with matching cr_hbuf_put()+ (even though it's not really needed)+ - Add 'current implementation' to docs to describe assumptions+ - Misc fixes and cleanups++[2008-Sep-11] v5:+ - Config is 'def_bool n' by default+ - Improve memory dump/restore code (following Dave Hansen's comments)+ - Change dump format (and code) to allow chunks of <vaddrs, pages>+ instead of one long list of each+ - Fix use of follow_page() to avoid faulting in non-present pages+ - Memory restore now maps user pages explicitly to copy data into them,+ instead of reading directly to user space; got rid of mprotect_fixup()+ - Remove preempt_disable() when restoring debug registers+ - Rename headers files s/ckpt/checkpoint/+ - Fix misc bugs in files dump/restore+ - Fix cleanup on some error paths+ - Fix misc coding style++[2008-Sep-04] v4:+ - Fix calculation of hash table size+ - Fix header structure alignment+ - Use stand list_... for cr_pgarr++[2008-Aug-20] v3:+ - Various fixes and clean-ups+ - Use standard hlist_... for hash table+ - Better use of standard kmalloc/kfree++[2008-Aug-09] v2:+ - Added utsname->{release,version,machine} to checkpoint header+ - Pad header structures to 64 bits to ensure compatibility+ - Address comments from LKML and linux-containers mailing list++[2008-Jul-29] v1:+In this incarnation, CR only works on single task. The address space+may consist of only private, simple VMAs - anonymous or file-mapped.+Both checkpoint and restart will ignore the first argument (pid/crid)+and instead act on themselves.
Infrastructure to handle objects that may be shared and referenced by
multiple tasks or other objects, e..g open files, memory address space
etc.
The state of shared objects is saved once. On the first encounter, the
state is dumped and the object is assigned a unique identifier (objref)
and also stored in a hash table (indexed by its physical kenrel address).
From then on the object will be found in the hash and only its identifier
is saved.
On restart the identifier is looked up in the hash table; if not found
then the state is read, the object is created, and added to the hash
table (this time indexed by its identifier). Otherwise, the object in
the hash table is used.
Signed-off-by: Oren Laadan <redacted>
Acked-by: Serge Hallyn <serue-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org>
Signed-off-by: Dave Hansen <redacted>
---
Documentation/checkpoint.txt | 46 +++++++
checkpoint/Makefile | 2 +-
checkpoint/objhash.c | 268 ++++++++++++++++++++++++++++++++++++++++++
checkpoint/sys.c | 6 +
include/linux/checkpoint.h | 20 +++
5 files changed, 341 insertions(+), 1 deletions(-)
create mode 100644 checkpoint/objhash.c
@@ -189,6 +189,52 @@ cr_hdr + cr_hdr_task cr_hdr + cr_hdr_tail+=== Shared resources (objects)++Many resources used by tasks may be shared by more than one task (e.g.+file descriptors, memory address space, etc), or even have multiple+references from other resources (e.g. a single inode that represents+two ends of a pipe).++Clearly, the state of shared objects need only be saved once, even if+they occur multiple times. We use a hash table (ctx->objhash) to keep+track of shared objects and whether they were already saved. Shared+objects are stored in a hash table as they appear, indexed by their+kernel address. (The hash table itself is not saved as part of the+checkpoint image: it is constructed dynamically during both checkpoint+and restart, and discarded at the end of the operation).++Each shared object that is found is first looked up in the hash table.+On the first encounter, the object will not be found, so its state is+dumped, and the object is assigned a unique identifier and also stored+in the hash table. Subsequent lookups of that object in the hash table+will yield that entry, and then only the unique identifier is saved,+as opposed the entire state of the object.++During restart, shared objects are seen by their unique identifiers as+assigned during the checkpoint. Each shared object that it read in is+first looked up in the hash table. On the first encounter it will not+be found, meaning that the object needs to be created and its state+read in and restored. Then the object is added to the hash table, this+time indexed by its unique identifier. Subsequent lookups of the same+unique identifier in the hash table will yield that entry, and then+the existing object instance is reused instead of creating another one.++The interface for the hash table is the following:++cr_obj_get_by_ptr() - find the unique object reference (objref)+ of the object that is pointer to by ptr [checkpoint]++cr_obj_add_ptr() - add the object pointed to by ptr to the hash table+ if not already there, and fill its unique object reference (objref)++cr_obj_get_by_ref() - return the pointer to the object whose unique+ object reference is equal to objref [restart]++cr_obj_add_ref() - add the object with given unique object reference+ (objref), pointed to by ptr to the hash table. [restart]++ === Current Implementation [2008-Oct-07]
@@ -2,5 +2,5 @@# Makefile for linux checkpoint/restart.#-obj-$(CONFIG_CHECKPOINT_RESTART)+=sys.ocheckpoint.orestart.o\+obj-$(CONFIG_CHECKPOINT_RESTART)+=sys.ocheckpoint.orestart.oobjhash.o\ckpt_mem.orstr_mem.o
Restoring the memory address space begins with nuking the existing one
of the current process, and then reading the VMA state and contents.
Call do_mmap_pgoffset() for each VMA and then read in the data.
Signed-off-by: Oren Laadan <redacted>
Acked-by: Serge Hallyn <serue-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org>
Signed-off-by: Dave Hansen <redacted>
---
arch/x86/mm/restart.c | 64 ++++++-
checkpoint/Makefile | 2 +-
checkpoint/checkpoint_arch.h | 2 +
checkpoint/checkpoint_mem.h | 5 +
checkpoint/restart.c | 42 ++++
checkpoint/rstr_mem.c | 384 ++++++++++++++++++++++++++++++++++++++
include/asm-x86/checkpoint_hdr.h | 4 +
include/linux/checkpoint.h | 3 +
8 files changed, 503 insertions(+), 3 deletions(-)
create mode 100644 checkpoint/rstr_mem.c
@@ -78,6 +78,44 @@ int cr_read_string(struct cr_ctx *ctx, void *str, int len)returncr_read_obj_type(ctx,str,len,CR_HDR_STRING);}+/**+*cr_read_fname-readafilename+*@ctx:checkpointcontext+*@fname:buffer+*@n:bufferlength+*/+intcr_read_fname(structcr_ctx*ctx,void*fname,intflen)+{+returncr_read_obj_type(ctx,fname,flen,CR_HDR_FNAME);+}++/**+*cr_read_open_fname-readafilenameandopenafile+*@ctx:checkpointcontext+*@flags:fileflags+*@mode:filemode+*/+structfile*cr_read_open_fname(structcr_ctx*ctx,intflags,intmode)+{+structfile*file;+char*fname;+intret;++fname=kmalloc(PATH_MAX,GFP_KERNEL);+if(!fname)+returnERR_PTR(-ENOMEM);++ret=cr_read_fname(ctx,fname,PATH_MAX);+cr_debug("fname '%s' flags %#x mode %#x\n",fname,flags,mode);+if(ret>=0)+file=filp_open(fname,flags,mode);+else+file=ERR_PTR(ret);++kfree(fname);+returnfile;+}+/* read the checkpoint header */staticintcr_read_head(structcr_ctx*ctx){
@@ -177,6 +215,10 @@ static int cr_read_task(struct cr_ctx *ctx)cr_debug("task_struct: ret %d\n",ret);if(ret<0)gotoout;+ret=cr_read_mm(ctx);+cr_debug("memory: ret %d\n",ret);+if(ret<0)+gotoout;ret=cr_read_thread(ctx);cr_debug("thread: ret %d\n",ret);if(ret<0)
@@ -0,0 +1,384 @@+/*+*Restartmemorycontents+*+*Copyright(C)2008OrenLaadan+*+*ThisfileissubjecttothetermsandconditionsoftheGNUGeneralPublic+*License.SeethefileCOPYINGinthemaindirectoryoftheLinux+*distributionformoredetails.+*/++#include<linux/kernel.h>+#include<linux/sched.h>+#include<linux/fcntl.h>+#include<linux/file.h>+#include<linux/fs.h>+#include<linux/pagemap.h>+#include<linux/mm_types.h>+#include<linux/mman.h>+#include<linux/mm.h>+#include<linux/err.h>+#include<linux/checkpoint.h>+#include<linux/checkpoint_hdr.h>++#include"checkpoint_arch.h"+#include"checkpoint_mem.h"++/*+*Unlikecheckpoint,restartisexecutedinthecontextofeachrestarting+*process:vmaregionsarerestoredviaacalltommap(),andthedatais+*readintotheaddressspaceofthecurrentprocess.+*/+++/**+*cr_read_pages_vaddrs-readaddressesofpagestopage-arraychain+*@ctx-restartcontext+*@nr_pages-numberofaddresstoread+*/+staticintcr_read_pages_vaddrs(structcr_ctx*ctx,unsignedlongnr_pages)+{+structcr_pgarr*pgarr;+unsignedlong*vaddrp;+intnr,ret;++while(nr_pages){+pgarr=cr_pgarr_current(ctx);+if(!pgarr)+return-ENOMEM;+nr=cr_pgarr_nr_free(pgarr);+if(nr>nr_pages)+nr=nr_pages;+vaddrp=&pgarr->vaddrs[pgarr->nr_used];+ret=cr_kread(ctx,vaddrp,nr*sizeof(unsignedlong));+if(ret<0)+returnret;+pgarr->nr_used+=nr;+nr_pages-=nr;+}+return0;+}++staticintcr_page_read(structcr_ctx*ctx,structpage*page,char*buf)+{+void*ptr;+intret;++ret=cr_kread(ctx,buf,PAGE_SIZE);+if(ret<0)+returnret;++ptr=kmap_atomic(page,KM_USER1);+memcpy(ptr,buf,PAGE_SIZE);+kunmap_atomic(ptr,KM_USER1);++return0;+}++/**+*cr_read_pages_contents-readindataofpagesinpage-arraychain+*@ctx-restartcontext+*/+staticintcr_read_pages_contents(structcr_ctx*ctx)+{+structmm_struct*mm=current->mm;+structcr_pgarr*pgarr;+unsignedlong*vaddrs;+char*buf;+inti,ret=0;++buf=kmalloc(PAGE_SIZE,GFP_KERNEL);+if(!buf)+return-ENOMEM;++down_read(&mm->mmap_sem);+list_for_each_entry_reverse(pgarr,&ctx->pgarr_list,list){+vaddrs=pgarr->vaddrs;+for(i=0;i<pgarr->nr_used;i++){+structpage*page;++ret=get_user_pages(current,mm,vaddrs[i],+1,1,1,&page,NULL);+if(ret<0)+gotoout;++ret=cr_page_read(ctx,page,buf);+page_cache_release(page);++if(ret<0)+gotoout;+}+}++out:+up_read(&mm->mmap_sem);+kfree(buf);+return0;+}++/**+*cr_read_private_vma_contents-restorecontentsofaVMAwithprivatememory+*@ctx-restartcontext+*+*Readsaheaderthatspecifieshowmanypageswillfollow,thenreads+*alistofvirtualaddressesintoctx->pgarr_listpage-arraychain,+*followedbytheactualcontentsofthecorrespondingpages.Iterates+*thesestepsuntilreachingaheaderspecifying"0"pages,whichmarks+*theendofthecontents.+*/+staticintcr_read_private_vma_contents(structcr_ctx*ctx)+{+structcr_hdr_pgarr*hh;+unsignedlongnr_pages;+intparent,ret=0;++while(1){+hh=cr_hbuf_get(ctx,sizeof(*hh));+parent=cr_read_obj_type(ctx,hh,sizeof(*hh),CR_HDR_PGARR);+if(parent!=0){+if(parent<0)+ret=parent;+else+ret=-EINVAL;+cr_hbuf_put(ctx,sizeof(*hh));+break;+}++cr_debug("nr_pages %ld\n",(unsignedlong)hh->nr_pages);++nr_pages=hh->nr_pages;+cr_hbuf_put(ctx,sizeof(*hh));++if(!nr_pages)+break;++ret=cr_read_pages_vaddrs(ctx,nr_pages);+if(ret<0)+break;+ret=cr_read_pages_contents(ctx);+if(ret<0)+break;+cr_pgarr_reset_all(ctx);+}++returnret;+}++/**+*cr_calc_map_prot_bits-convertvm_flagstommapprotection+*orig_vm_flags:sourcevm_flags+*/+staticunsignedlongcr_calc_map_prot_bits(unsignedlongorig_vm_flags)+{+unsignedlongvm_prot=0;++if(orig_vm_flags&VM_READ)+vm_prot|=PROT_READ;+if(orig_vm_flags&VM_WRITE)+vm_prot|=PROT_WRITE;+if(orig_vm_flags&VM_EXEC)+vm_prot|=PROT_EXEC;+if(orig_vm_flags&PROT_SEM)/* only (?) with IPC-SHM */+vm_prot|=PROT_SEM;++returnvm_prot;+}++/**+*cr_calc_map_flags_bits-convertvm_flagstommapflags+*orig_vm_flags:sourcevm_flags+*/+staticunsignedlongcr_calc_map_flags_bits(unsignedlongorig_vm_flags)+{+unsignedlongvm_flags=0;++vm_flags=MAP_FIXED;+if(orig_vm_flags&VM_GROWSDOWN)+vm_flags|=MAP_GROWSDOWN;+if(orig_vm_flags&VM_DENYWRITE)+vm_flags|=MAP_DENYWRITE;+if(orig_vm_flags&VM_EXECUTABLE)+vm_flags|=MAP_EXECUTABLE;+if(orig_vm_flags&VM_MAYSHARE)+vm_flags|=MAP_SHARED;+else+vm_flags|=MAP_PRIVATE;++returnvm_flags;+}++staticintcr_read_vma(structcr_ctx*ctx,structmm_struct*mm)+{+structcr_hdr_vma*hh=cr_hbuf_get(ctx,sizeof(*hh));+unsignedlongvm_size,vm_start,vm_flags,vm_prot,vm_pgoff;+unsignedlongaddr;+structfile*file=NULL;+intparent,ret=-EINVAL;++parent=cr_read_obj_type(ctx,hh,sizeof(*hh),CR_HDR_VMA);+if(parent<0){+ret=parent;+gotoerr;+}elseif(parent!=0)+gotoerr;++cr_debug("vma %#lx-%#lx type %d\n",(unsignedlong)hh->vm_start,+(unsignedlong)hh->vm_end,(int)hh->vma_type);++if(hh->vm_end<hh->vm_start)+gotoerr;++vm_start=hh->vm_start;+vm_pgoff=hh->vm_pgoff;+vm_size=hh->vm_end-hh->vm_start;+vm_prot=cr_calc_map_prot_bits(hh->vm_flags);+vm_flags=cr_calc_map_flags_bits(hh->vm_flags);++switch(hh->vma_type){++caseCR_VMA_ANON:/* anonymous private mapping */+if(vm_flags&VM_SHARED)+gotoerr;+/*+*vm_pgoffforanonymousmappingisthe"global"page+*offset(namelyfromaddr0x0),soweforceazero+*/+vm_pgoff=0;+break;++caseCR_VMA_FILE:/* private mapping from a file */+if(vm_flags&VM_SHARED)+gotoerr;+/*+*forprivatemappingusing'read-only'issufficient+*/+file=cr_read_open_fname(ctx,O_RDONLY,0);+if(IS_ERR(file)){+ret=PTR_ERR(file);+gotoerr;+}+break;++default:+gotoerr;++}++cr_hbuf_put(ctx,sizeof(*hh));++down_write(&mm->mmap_sem);+addr=do_mmap_pgoff(file,vm_start,vm_size,+vm_prot,vm_flags,vm_pgoff);+up_write(&mm->mmap_sem);+cr_debug("size %#lx prot %#lx flag %#lx pgoff %#lx => %#lx\n",+vm_size,vm_prot,vm_flags,vm_pgoff,addr);++/* the file (if opened) is now referenced by the vma */+if(file)+filp_close(file,NULL);++if(IS_ERR((void*)addr))+returnPTR_ERR((void*)addr);++/*+*CR_VMA_ANON:readinmemoryasis+*CR_VMA_FILE:readinmemoryasis+*(moretofollow...)+*/++switch(hh->vma_type){+caseCR_VMA_ANON:+caseCR_VMA_FILE:+/* standard case: read the data into the memory */+ret=cr_read_private_vma_contents(ctx);+break;+}++if(ret<0)+returnret;++cr_debug("vma retval %d\n",ret);+return0;++err:+cr_hbuf_put(ctx,sizeof(*hh));+returnret;+}++staticintcr_destroy_mm(structmm_struct*mm)+{+structvm_area_struct*vmnext=mm->mmap;+structvm_area_struct*vma;+intret;++while(vmnext){+vma=vmnext;+vmnext=vmnext->vm_next;+ret=do_munmap(mm,vma->vm_start,vma->vm_end-vma->vm_start);+if(ret<0){+pr_debug("CR: restart failed do_munmap (%d)\n",ret);+returnret;+}+}+return0;+}++intcr_read_mm(structcr_ctx*ctx)+{+structcr_hdr_mm*hh=cr_hbuf_get(ctx,sizeof(*hh));+structmm_struct*mm;+intnr,parent,ret;++parent=cr_read_obj_type(ctx,hh,sizeof(*hh),CR_HDR_MM);+if(parent<0){+ret=parent;+gotoout;+}++ret=-EINVAL;+#if 0 /* activate when containers are used */+if(parent!=task_pid_vnr(current))+gotoout;+#endif+cr_debug("map_count %d\n",hh->map_count);++/* XXX need more sanity checks */+if(hh->start_code>hh->end_code||+hh->start_data>hh->end_data||hh->map_count<0)+gotoout;++mm=current->mm;++/* point of no return -- destruct current mm */+down_write(&mm->mmap_sem);+ret=cr_destroy_mm(mm);+if(ret<0){+up_write(&mm->mmap_sem);+gotoout;+}+mm->start_code=hh->start_code;+mm->end_code=hh->end_code;+mm->start_data=hh->start_data;+mm->end_data=hh->end_data;+mm->start_brk=hh->start_brk;+mm->brk=hh->brk;+mm->start_stack=hh->start_stack;+mm->arg_start=hh->arg_start;+mm->arg_end=hh->arg_end;+mm->env_start=hh->env_start;+mm->env_end=hh->env_end;+up_write(&mm->mmap_sem);++/* FIX: need also mm->flags */++for(nr=hh->map_count;nr;nr--){+ret=cr_read_vma(ctx,mm);+if(ret<0)+gotoout;+}++ret=cr_read_mm_context(ctx,mm,hh->objref);+out:+cr_hbuf_put(ctx,sizeof(*hh));+returnret;+}
Dump the files_struct of a task with 'struct cr_hdr_files', followed by
all open file descriptors. Since FDs can be shared, they are assigned an
objref and registered in the object hash.
For each open FD there is a 'struct cr_hdr_fd_ent' with the FD, its objref
and its close-on-exec property. If the FD is to be saved (first time)
then this is followed by a 'struct cr_hdr_fd_data' with the FD state.
Then will come the next FD and so on.
This patch only handles basic FDs - regular files, directories and also
symbolic links.
Signed-off-by: Oren Laadan <redacted>
Acked-by: Serge Hallyn <redacted>
Signed-off-by: Dave Hansen <redacted>
---
checkpoint/Makefile | 2 +-
checkpoint/checkpoint.c | 4 +
checkpoint/checkpoint_file.h | 17 +++
checkpoint/ckpt_file.c | 231 ++++++++++++++++++++++++++++++++++++++++
include/linux/checkpoint.h | 7 +-
include/linux/checkpoint_hdr.h | 32 ++++++-
6 files changed, 288 insertions(+), 5 deletions(-)
create mode 100644 checkpoint/checkpoint_file.h
create mode 100644 checkpoint/ckpt_file.c
@@ -203,6 +203,10 @@ static int cr_write_task(struct cr_ctx *ctx, struct task_struct *t)cr_debug("memory: ret %d\n",ret);if(ret<0)gotoout;+ret=cr_write_files(ctx,t);+cr_debug("files: ret %d\n",ret);+if(ret<0)+gotoout;ret=cr_write_thread(ctx,t);cr_debug("thread: ret %d\n",ret);if(ret<0)
@@ -0,0 +1,231 @@+/*+*Checkpointfiledescriptors+*+*Copyright(C)2008OrenLaadan+*+*ThisfileissubjecttothetermsandconditionsoftheGNUGeneralPublic+*License.SeethefileCOPYINGinthemaindirectoryoftheLinux+*distributionformoredetails.+*/++#include<linux/kernel.h>+#include<linux/sched.h>+#include<linux/file.h>+#include<linux/fdtable.h>+#include<linux/checkpoint.h>+#include<linux/checkpoint_hdr.h>++#include"checkpoint_file.h"++#define CR_DEFAULT_FDTABLE 256 /* an initial guess */++/**+*cr_scan_fds-scanfiletableandconstructarrayofopenfds+*@files:files_structpointer+*@fdtable:(output)arrayofopenfds+*+*Returnsthenumberofopenfdsfound,andalsothefiletable+*arrayvia*fdtable.Thecallershouldfreethearray.+*+*Thecallermustvalidatethefiledescriptorscollectedinthe+*arraybeforeusingthem,e.g.byusingfcheck_files(),incase+*thetask'sfdtablechangesinthemeantime.+*/+intcr_scan_fds(structfiles_struct*files,int**fdtable)+{+structfdtable*fdt;+int*fds;+inti,n=0;+inttot=CR_DEFAULT_FDTABLE;++fds=kmalloc(tot*sizeof(*fds),GFP_KERNEL);+if(!fds)+return-ENOMEM;++/*+*Weassumethatthetargettaskisfrozen(orthatwecheckpoint+*ourselves),sowecansafelyproceedafterkrealloc()fromwhere+*weleftoff;intheworstcasesrestartwillfail.+*/++spin_lock(&files->file_lock);+rcu_read_lock();+fdt=files_fdtable(files);+for(i=0;i<fdt->max_fds;i++){+if(!fcheck_files(files,i))+continue;+if(n==tot){+/*+*fcheck_files()issafewithdrop/re-acquire+*ofthelock,becauseittests:fd<max_fds+*/+spin_unlock(&files->file_lock);+rcu_read_unlock();+tot*=2;/* won't overflow: kmalloc will fail */+fds=krealloc(fds,tot*sizeof(*fds),GFP_KERNEL);+if(!fds){+kfree(fds);+return-ENOMEM;+}+rcu_read_lock();+spin_lock(&files->file_lock);+}+fds[n++]=i;+}+rcu_read_unlock();+spin_unlock(&files->file_lock);++*fdtable=fds;+returnn;+}++/* cr_write_fd_data - dump the state of a given file pointer */+staticintcr_write_fd_data(structcr_ctx*ctx,structfile*file,intparent)+{+structcr_hdrh;+structcr_hdr_fd_data*hh=cr_hbuf_get(ctx,sizeof(*hh));+structdentry*dent=file->f_dentry;+structinode*inode=dent->d_inode;+enumfd_typefd_type;+intret;++h.type=CR_HDR_FD_DATA;+h.len=sizeof(*hh);+h.parent=parent;++hh->f_flags=file->f_flags;+hh->f_mode=file->f_mode;+hh->f_pos=file->f_pos;+hh->f_version=file->f_version;+/* FIX: need also file->uid, file->gid, file->f_owner, etc */++switch(inode->i_mode&S_IFMT){+caseS_IFREG:+fd_type=CR_FD_FILE;+break;+caseS_IFDIR:+fd_type=CR_FD_DIR;+break;+caseS_IFLNK:+fd_type=CR_FD_LINK;+break;+default:+cr_hbuf_put(ctx,sizeof(*hh));+return-EBADF;+}++/* FIX: check if the file/dir/link is unlinked */+hh->fd_type=fd_type;++ret=cr_write_obj(ctx,&h,hh);+cr_hbuf_put(ctx,sizeof(*hh));+if(ret<0)+returnret;++returncr_write_fname(ctx,&file->f_path,ctx->vfsroot);+}++/**+*cr_write_fd_ent-dumpthestateofagivenfiledescriptor+*@ctx:checkpointcontext+*@files:files_structpointer+*@fd:filedescriptor+*+*Savesthestateofthefiledescriptor;looksuptheactualfile+*pointerinthehashtable,andiffoundsavesthematchingobjref,+*otherwisecallscr_write_fd_datatodumpthefilepointertoo.+*/+staticint+cr_write_fd_ent(structcr_ctx*ctx,structfiles_struct*files,intfd)+{+structcr_hdrh;+structcr_hdr_fd_ent*hh=cr_hbuf_get(ctx,sizeof(*hh));+structfile*file=NULL;+structfdtable*fdt;+intcoe,objref,new,ret;++rcu_read_lock();+fdt=files_fdtable(files);+file=fcheck_files(files,fd);+if(file){+coe=FD_ISSET(fd,fdt->close_on_exec);+get_file(file);+}+rcu_read_unlock();++/* sanity check (although this shouldn't happen) */+if(!file){+ret=-EBADF;+gotoout;+}++new=cr_obj_add_ptr(ctx,file,&objref,CR_OBJ_FILE,0);+cr_debug("fd %d objref %d file %p c-o-e %d)\n",fd,objref,file,coe);++if(new<0){+ret=new;+gotoout;+}++h.type=CR_HDR_FD_ENT;+h.len=sizeof(*hh);+h.parent=0;++hh->objref=objref;+hh->fd=fd;+hh->close_on_exec=coe;++ret=cr_write_obj(ctx,&h,hh);+if(ret<0)+gotoout;++/* new==1 if-and-only-if file was newly added to hash */+if(new)+ret=cr_write_fd_data(ctx,file,objref);++out:+cr_hbuf_put(ctx,sizeof(*hh));+fput(file);+returnret;+}++intcr_write_files(structcr_ctx*ctx,structtask_struct*t)+{+structcr_hdrh;+structcr_hdr_files*hh=cr_hbuf_get(ctx,sizeof(*hh));+structfiles_struct*files;+int*fdtable;+intnfds,n,ret;++h.type=CR_HDR_FILES;+h.len=sizeof(*hh);+h.parent=task_pid_vnr(t);++files=get_files_struct(t);++nfds=cr_scan_fds(files,&fdtable);+if(nfds<0){+put_files_struct(files);+returnnfds;+}++hh->objref=0;/* will be meaningful with multiple processes */+hh->nfds=nfds;++ret=cr_write_obj(ctx,&h,hh);+cr_hbuf_put(ctx,sizeof(*hh));+if(ret<0)+gotoclean;++cr_debug("nfds %d\n",nfds);+for(n=0;n<nfds;n++){+ret=cr_write_fd_ent(ctx,files,fdtable[n]);+if(ret<0)+break;+}++clean:+kfree(fdtable);+put_files_struct(files);+returnret;+}
@@ -79,11 +79,12 @@ extern int cr_read_fname(struct cr_ctx *ctx, void *fname, int n);externstructfile*cr_read_open_fname(structcr_ctx*ctx,intflags,intmode);+externintdo_checkpoint(structcr_ctx*ctx);externintcr_write_mm(structcr_ctx*ctx,structtask_struct*t);-externintcr_read_mm(structcr_ctx*ctx);+externintcr_write_files(structcr_ctx*ctx,structtask_struct*t);-externintdo_checkpoint(structcr_ctx*ctx);externintdo_restart(structcr_ctx*ctx);+externintcr_read_mm(structcr_ctx*ctx);/* there are from fs/read_write.c, not exported otherwise in a header */externloff_tfile_pos_read(structfile*file);
@@ -105,4 +109,30 @@ struct cr_hdr_pgarr {__u64nr_pages;/* number of pages to saved */}__attribute__((aligned(8)));+structcr_hdr_files{+__u32objref;/* identifier for shared objects */+__u32nfds;+}__attribute__((aligned(8)));++structcr_hdr_fd_ent{+__u32objref;/* identifier for shared objects */+__s32fd;+__u32close_on_exec;+}__attribute__((aligned(8)));++/* fd types */+enumfd_type{+CR_FD_FILE=1,+CR_FD_DIR,+CR_FD_LINK+};++structcr_hdr_fd_data{+__u16fd_type;+__u16f_mode;+__u32f_flags;+__u64f_pos;+__u64f_version;+}__attribute__((aligned(8)));+#endif /* _CHECKPOINT_CKPT_HDR_H_ */
--
1.5.4.3
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
For each VMA, there is a 'struct cr_vma'; if the VMA is file-mapped,
it will be followed by the file name. Then comes the actual contents,
in one or more chunk: each chunk begins with a header that specifies
how many pages it holds, then the virtual addresses of all the dumped
pages in that chunk, followed by the actual contents of all dumped
pages. A header with zero number of pages marks the end of the contents.
Then comes the next VMA and so on.
Signed-off-by: Oren Laadan <redacted>
Acked-by: Serge Hallyn <redacted>
Signed-off-by: Dave Hansen <redacted>
---
arch/x86/mm/checkpoint.c | 31 +++
arch/x86/mm/restart.c | 1 +
checkpoint/Makefile | 3 +-
checkpoint/checkpoint.c | 53 ++++
checkpoint/checkpoint_arch.h | 2 +
checkpoint/checkpoint_mem.h | 41 +++
checkpoint/ckpt_mem.c | 500 ++++++++++++++++++++++++++++++++++++++
checkpoint/sys.c | 16 ++
include/asm-x86/checkpoint_hdr.h | 5 +
include/linux/checkpoint.h | 12 +
include/linux/checkpoint_hdr.h | 32 +++
11 files changed, 695 insertions(+), 1 deletions(-)
create mode 100644 checkpoint/checkpoint_mem.h
create mode 100644 checkpoint/ckpt_mem.c
@@ -2,4 +2,5 @@# Makefile for linux checkpoint/restart.#-obj-$(CONFIG_CHECKPOINT_RESTART)+=sys.ocheckpoint.orestart.o+obj-$(CONFIG_CHECKPOINT_RESTART)+=sys.ocheckpoint.orestart.o\+ckpt_mem.o
@@ -55,6 +55,55 @@ int cr_write_string(struct cr_ctx *ctx, char *str, int len)returncr_write_obj(ctx,&h,str);}+/**+*cr_fill_fname-returnpathnameofagivenfile+*@path:pathname+*@root:relativeroot+*@buf:bufferforpathname+*@n:bufferlength(in)andpathnamelength(out)+*/+staticchar*+cr_fill_fname(structpath*path,structpath*root,char*buf,int*n)+{+char*fname;++BUG_ON(!buf);+fname=__d_path(path,root,buf,*n);+if(!IS_ERR(fname))+*n=(buf+(*n)-fname);+returnfname;+}++/**+*cr_write_fname-writeafilename+*@ctx:checkpointcontext+*@path:pathname+*@root:relativeroot+*/+intcr_write_fname(structcr_ctx*ctx,structpath*path,structpath*root)+{+structcr_hdrh;+char*buf,*fname;+intret,flen;++flen=PATH_MAX;+buf=kmalloc(flen,GFP_KERNEL);+if(!buf)+return-ENOMEM;++fname=cr_fill_fname(path,root,buf,&flen);+if(!IS_ERR(fname)){+h.type=CR_HDR_FNAME;+h.len=flen;+h.parent=0;+ret=cr_write_obj(ctx,&h,fname);+}else+ret=PTR_ERR(fname);++kfree(buf);+returnret;+}+/* write the checkpoint header */staticintcr_write_head(structcr_ctx*ctx){
@@ -150,6 +199,10 @@ static int cr_write_task(struct cr_ctx *ctx, struct task_struct *t)cr_debug("task_struct: ret %d\n",ret);if(ret<0)gotoout;+ret=cr_write_mm(ctx,t);+cr_debug("memory: ret %d\n",ret);+if(ret<0)+gotoout;ret=cr_write_thread(ctx,t);cr_debug("thread: ret %d\n",ret);if(ret<0)
@@ -0,0 +1,500 @@+/*+*Checkpointmemorycontents+*+*Copyright(C)2008OrenLaadan+*+*ThisfileissubjecttothetermsandconditionsoftheGNUGeneralPublic+*License.SeethefileCOPYINGinthemaindirectoryoftheLinux+*distributionformoredetails.+*/++#include<linux/kernel.h>+#include<linux/sched.h>+#include<linux/slab.h>+#include<linux/file.h>+#include<linux/pagemap.h>+#include<linux/mm_types.h>+#include<linux/checkpoint.h>+#include<linux/checkpoint_hdr.h>++#include"checkpoint_arch.h"+#include"checkpoint_mem.h"++/*+*utilitiestoalloc,free,andhandle'structcr_pgarr'(page-arrays)+*(commontockpt_mem.candrstr_mem.c).+*+*Thecheckpointcontextstructurehastwomembersforpage-arrays:+*ctx->pgarr_list:listheadofthepage-arraychain+*+*Duringcheckpoint(andrestart)thechaintracksthedirtypages(page+*pointerandvirtualaddress)ofeachMM.ForaparticularMM,theseare+*alwaysaddedtotheheadofthepage-arraychain(ctx->pgarr_list).+*This"current"page-arrayadvancesasnecessary,andnewpage-array+*descriptorsareallocatedon-demand.Beforethenextchunkofpages,+*thechainisresetbutnotfreed(thatis,dereferencepagepointers).+*/++/* return first page-array in the chain */+staticinlinestructcr_pgarr*cr_pgarr_first(structcr_ctx*ctx)+{+if(list_empty(&ctx->pgarr_list))+returnNULL;+returnlist_first_entry(&ctx->pgarr_list,structcr_pgarr,list);+}++/* release pages referenced by a page-array */+staticvoidcr_pgarr_release_pages(structcr_pgarr*pgarr)+{+inti;++cr_debug("nr_used %d\n",pgarr->nr_used);+/*+*althoughbothcheckpointandrestartuse'nr_used',weonly+*collectpagesduringcheckpoint;inrestartwesimplyreturn+*/+if(!pgarr->pages)+return;+for(i=pgarr->nr_used;i--;/**/)+page_cache_release(pgarr->pages[i]);+}++/* free a single page-array object */+staticvoidcr_pgarr_free_one(structcr_pgarr*pgarr)+{+cr_pgarr_release_pages(pgarr);+kfree(pgarr->pages);+kfree(pgarr->vaddrs);+kfree(pgarr);+}++/* free a chain of page-arrays */+voidcr_pgarr_free(structcr_ctx*ctx)+{+structcr_pgarr*pgarr,*tmp;++list_for_each_entry_safe(pgarr,tmp,&ctx->pgarr_list,list){+list_del(&pgarr->list);+cr_pgarr_free_one(pgarr);+}+}++/* allocate a single page-array object */+staticstructcr_pgarr*cr_pgarr_alloc_one(unsignedlongflags)+{+structcr_pgarr*pgarr;++pgarr=kzalloc(sizeof(*pgarr),GFP_KERNEL);+if(!pgarr)+returnNULL;++pgarr->vaddrs=kmalloc(CR_PGARR_TOTAL*sizeof(unsignedlong),+GFP_KERNEL);+if(!pgarr->vaddrs)+gotonomem;++/* pgarr->pages is needed only for checkpoint */+if(flags&CR_CTX_CKPT){+pgarr->pages=kmalloc(CR_PGARR_TOTAL*sizeof(structpage*),+GFP_KERNEL);+if(!pgarr->pages)+gotonomem;+}++returnpgarr;++nomem:+cr_pgarr_free_one(pgarr);+returnNULL;+}++/* cr_pgarr_current - return the next available page-array in the chain+*@ctx:checkpointcontext+*+*Returnsthefirstpage-arrayinthelistthathasspace.Extendsthe+*listifnonehasspace.+*/+structcr_pgarr*cr_pgarr_current(structcr_ctx*ctx)+{+structcr_pgarr*pgarr;++pgarr=cr_pgarr_first(ctx);+if(pgarr&&!cr_pgarr_is_full(pgarr))+gotoout;+pgarr=cr_pgarr_alloc_one(ctx->flags);+if(!pgarr)+gotoout;+list_add(&pgarr->list,&ctx->pgarr_list);+out:+returnpgarr;+}++/* reset the page-array chain (dropping page references if necessary) */+voidcr_pgarr_reset_all(structcr_ctx*ctx)+{+structcr_pgarr*pgarr;++list_for_each_entry(pgarr,&ctx->pgarr_list,list){+cr_pgarr_release_pages(pgarr);+pgarr->nr_used=0;+}+}++/*+*Checkpointisoutsidethecontextofthecheckpointee,soonecannot+*simplyreadpagesfromuser-space.Instead,wescantheaddressspace+*ofthetargettocherry-pickpagesofinterest.Selectedpagesare+*enlistedinapage-arraychain(attachedtothecheckpointcontext).+*Tosavetheircontents,eachpageismappedtokernelmemoryandthen+*dumpedtothefiledescriptor.+*/+++/**+*cr_private_follow_page-returnpagepointerfordirtypages+*@vma-targetvma+*@addr-pageaddress+*+*Looksupthepagethatcorrespondtotheaddressinthevma,and+*returnsthepageifitwasmodified(andgrabsareferencetoit),+*orotherwisereturnsNULL(orerror).+*+*Thisfunctionshould_only_calledforprivatevma's.+*/+staticstructpage*+cr_private_follow_page(structvm_area_struct*vma,unsignedlongaddr)+{+structpage*page;++BUG_ON(vma->vm_flags&(VM_SHARED|VM_MAYSHARE));++/*+*simplifiedversionofget_user_pages():alreadyhavevma,+*onlyneedFOLL_ANON,and(fornow)ignorefaultstats.+*+*follow_page()willreturnNULLifthepageisnotpresent+*(swapped),ZERO_PAGE(0)iftheptewasn'tallocated,and+*theactualpagepointerotherwise.+*+*FIXME:consolidatewithget_user_pages()+*/++cond_resched();+while(!(page=follow_page(vma,addr,FOLL_ANON|FOLL_GET))){+intret;++/* the page is swapped out - bring it in (optimize ?) */+ret=handle_mm_fault(vma->vm_mm,vma,addr,0);+if(ret&VM_FAULT_ERROR){+if(ret&VM_FAULT_OOM)+returnERR_PTR(-ENOMEM);+elseif(ret&VM_FAULT_SIGBUS)+returnERR_PTR(-EFAULT);+else+BUG();+break;+}+cond_resched();+}++if(IS_ERR(page))+returnpage;++/*+*Weonlycareaboutdirtypages:eithernon-zeropage,or+*file-backed(copy-on-write)thatweretouched.Forthelatter,+*thepage_mapping()willbeunsetbecauseitwillnolongerbe+*mappedtotheoriginalfileafterhavingbeenmodified.+*/+if(page==ZERO_PAGE(0)){+/* this is the zero page: ignore */+page_cache_release(page);+page=NULL;+}elseif(vma->vm_file&&(page_mapping(page)!=NULL)){+/* file backed clean cow: ignore */+page_cache_release(page);+page=NULL;+}++returnpage;+}++/**+*cr_private_vma_fill_pgarr-fillapage-arraywithaddr/pagetuples+*@ctx-checkpointcontext+*@pgarr-page-arraytofill+*@vma-vmatoscan+*@start-startaddress(updated)+*+*Returnsthenumberofpagescollected+*/+staticint+cr_private_vma_fill_pgarr(structcr_ctx*ctx,structcr_pgarr*pgarr,+structvm_area_struct*vma,unsignedlong*start)+{+unsignedlongend=vma->vm_end;+unsignedlongaddr=*start;+intorig_used=pgarr->nr_used;++/* this function is only for private memory (anon or file-mapped) */+BUG_ON(vma->vm_flags&(VM_SHARED|VM_MAYSHARE));++while(addr<end){+structpage*page;++page=cr_private_follow_page(vma,addr);+if(IS_ERR(page))+returnPTR_ERR(page);++if(page){+pgarr->pages[pgarr->nr_used]=page;+pgarr->vaddrs[pgarr->nr_used]=addr;+pgarr->nr_used++;+}++addr+=PAGE_SIZE;++if(cr_pgarr_is_full(pgarr))+break;+}++*start=addr;+returnpgarr->nr_used-orig_used;+}++/* dump contents of a pages: use kmap_atomic() to avoid TLB flush */+staticintcr_page_write(structcr_ctx*ctx,structpage*page,char*buf)+{+void*ptr;++ptr=kmap_atomic(page,KM_USER1);+memcpy(buf,ptr,PAGE_SIZE);+kunmap_atomic(ptr,KM_USER1);++returncr_kwrite(ctx,buf,PAGE_SIZE);+}++/**+*cr_vma_dump_pages-dumppageslistedinthectxpage-arraychain+*@ctx-checkpointcontext+*@total-totalnumberofpages+*+*Firstdumpallvirtualaddresses,followedbythecontentsofallpages+*/+staticintcr_vma_dump_pages(structcr_ctx*ctx,inttotal)+{+structcr_pgarr*pgarr;+char*buf;+inti,ret=0;++if(!total)+return0;++list_for_each_entry_reverse(pgarr,&ctx->pgarr_list,list){+ret=cr_kwrite(ctx,pgarr->vaddrs,+pgarr->nr_used*sizeof(*pgarr->vaddrs));+if(ret<0)+returnret;+}++buf=kmalloc(PAGE_SIZE,GFP_KERNEL);+if(!buf)+return-ENOMEM;++list_for_each_entry_reverse(pgarr,&ctx->pgarr_list,list){+for(i=0;i<pgarr->nr_used;i++){+ret=cr_page_write(ctx,pgarr->pages[i],buf);+if(ret<0)+gotoout;+}+}++out:+kfree(buf);+returnret;+}++/**+*cr_write_private_vma_contents-dumpcontentsofaVMAwithprivatememory+*@ctx-checkpointcontext+*@vma-vmatoscan+*+*Collectlistsofpagesthatneedstobedumped,andcorresponding+*virtualaddressesintoctx->pgarr_listpage-arraychain.Thendump+*theaddresses,followedbythepagecontents.+*/+staticint+cr_write_private_vma_contents(structcr_ctx*ctx,structvm_area_struct*vma)+{+structcr_hdrh;+structcr_hdr_pgarr*hh;+unsignedlongaddr=vma->vm_start;+structcr_pgarr*pgarr;+unsignedlongcnt=0;+intret;++/*+*Workiteratively,collectinganddumpingatmostCR_PGARR_CHUNK+*ineachround.Eachiterationsisdividedintotwosteps:+*+*(1)scan:scanthroughthePTEsofthevmatocollectthepages+*todump(laterwe'llalsomakethemCOW),whilekeepingalist+*ofpagesandtheircorrespondingaddressesonctx->pgarr_list.+*+*(2)dump:writeoutaheaderspecifyinghowmanypages,followed+*bytheaddressesofallpagesinctx->pgarr_list,followedby+*theactualcontentsofallpages.(Then,releasethereferences+*tothepagesandresetthepage-arraychain).+*+*(Thissplitmakesthelogicsimplerbyfirstcountingthepages+*thatneedsaving.Moreimportantly,itallowsforafuture+*optimizationthatwillreduceapplicationdowntimebydeferring+*theactualwrite-outofthedatatoaftertheapplicationis+*allowedtoresumeexecution).+*+*Afterdumptingtheentirecontents,concludewithaheaderthat+*specifies0pagestomarktheendofthecontents.+*/++h.type=CR_HDR_PGARR;+h.len=sizeof(*hh);+h.parent=0;++while(addr<vma->vm_end){+pgarr=cr_pgarr_current(ctx);+if(!pgarr)+return-ENOMEM;+ret=cr_private_vma_fill_pgarr(ctx,pgarr,vma,&addr);+if(ret<0)+returnret;+cnt+=ret;++/* did we complete a chunk, or is this the last chunk ? */+if(cnt>=CR_PGARR_CHUNK||(cnt&&addr==vma->vm_end)){+hh=cr_hbuf_get(ctx,sizeof(*hh));+hh->nr_pages=cnt;+ret=cr_write_obj(ctx,&h,hh);+cr_hbuf_put(ctx,sizeof(*hh));+if(ret<0)+returnret;++ret=cr_vma_dump_pages(ctx,cnt);+if(ret<0)+returnret;++cr_pgarr_reset_all(ctx);+}+}++/* mark end of contents with header saying "0" pages */+hh=cr_hbuf_get(ctx,sizeof(*hh));+hh->nr_pages=0;+ret=cr_write_obj(ctx,&h,hh);+cr_hbuf_put(ctx,sizeof(*hh));++returnret;+}++staticintcr_write_vma(structcr_ctx*ctx,structvm_area_struct*vma)+{+structcr_hdrh;+structcr_hdr_vma*hh=cr_hbuf_get(ctx,sizeof(*hh));+intvma_type,ret;++h.type=CR_HDR_VMA;+h.len=sizeof(*hh);+h.parent=0;++hh->vm_start=vma->vm_start;+hh->vm_end=vma->vm_end;+hh->vm_page_prot=vma->vm_page_prot.pgprot;+hh->vm_flags=vma->vm_flags;+hh->vm_pgoff=vma->vm_pgoff;++if(vma->vm_flags&(VM_SHARED|VM_IO|VM_HUGETLB|VM_NONLINEAR)){+pr_warning("CR: unsupported VMA %#lx\n",vma->vm_flags);+cr_hbuf_put(ctx,sizeof(*hh));+return-ENOSYS;+}++/* by default assume anon memory */+vma_type=CR_VMA_ANON;++/*+*ifthereisabackingfile,assumeprivate-mapped+*(FIXME:checkifthefileisunlinked)+*/+if(vma->vm_file)+vma_type=CR_VMA_FILE;++hh->vma_type=vma_type;++ret=cr_write_obj(ctx,&h,hh);+cr_hbuf_put(ctx,sizeof(*hh));+if(ret<0)+returnret;++/* save the file name, if relevant */+if(vma->vm_file){+ret=cr_write_fname(ctx,&vma->vm_file->f_path,ctx->vfsroot);+if(ret<0)+returnret;+}++returncr_write_private_vma_contents(ctx,vma);+}++intcr_write_mm(structcr_ctx*ctx,structtask_struct*t)+{+structcr_hdrh;+structcr_hdr_mm*hh=cr_hbuf_get(ctx,sizeof(*hh));+structmm_struct*mm;+structvm_area_struct*vma;+intobjref,ret;++h.type=CR_HDR_MM;+h.len=sizeof(*hh);+h.parent=task_pid_vnr(t);++mm=get_task_mm(t);++objref=0;/* will be meaningful with multiple processes */+hh->objref=objref;++down_read(&mm->mmap_sem);++hh->start_code=mm->start_code;+hh->end_code=mm->end_code;+hh->start_data=mm->start_data;+hh->end_data=mm->end_data;+hh->start_brk=mm->start_brk;+hh->brk=mm->brk;+hh->start_stack=mm->start_stack;+hh->arg_start=mm->arg_start;+hh->arg_end=mm->arg_end;+hh->env_start=mm->env_start;+hh->env_end=mm->env_end;++hh->map_count=mm->map_count;++/* FIX: need also mm->flags */++ret=cr_write_obj(ctx,&h,hh);+cr_hbuf_put(ctx,sizeof(*hh));+if(ret<0)+gotoout;++/* write the vma's */+for(vma=mm->mmap;vma;vma=vma->vm_next){+ret=cr_write_vma(ctx,vma);+if(ret<0)+gotoout;+}++ret=cr_write_mm_context(ctx,mm,objref);++out:+up_read(&mm->mmap_sem);+mmput(mm);+returnret;+}
@@ -73,4 +75,34 @@ struct cr_hdr_task {__s32task_comm_len;}__attribute__((aligned(8)));+structcr_hdr_mm{+__u32objref;/* identifier for shared objects */+__u32map_count;++__u64start_code,end_code,start_data,end_data;+__u64start_brk,brk,start_stack;+__u64arg_start,arg_end,env_start,env_end;+}__attribute__((aligned(8)));++/* vma subtypes */+enumvm_type{+CR_VMA_ANON=1,+CR_VMA_FILE+};++structcr_hdr_vma{+__u32vma_type;+__u32_padding;++__u64vm_start;+__u64vm_end;+__u64vm_page_prot;+__u64vm_flags;+__u64vm_pgoff;+}__attribute__((aligned(8)));++structcr_hdr_pgarr{+__u64nr_pages;/* number of pages to saved */+}__attribute__((aligned(8)));+#endif /* _CHECKPOINT_CKPT_HDR_H_ */
--
1.5.4.3
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
Restore open file descriptors: for each FD read 'struct cr_hdr_fd_ent'
and lookup objref in the hash table; if not found (first occurence), read
in 'struct cr_hdr_fd_data', create a new FD and register in the hash.
Otherwise attach the file pointer from the hash as an FD.
This patch only handles basic FDs - regular files, directories and also
symbolic links.
Signed-off-by: Oren Laadan <redacted>
Acked-by: Serge Hallyn <redacted>
Signed-off-by: Dave Hansen <redacted>
---
checkpoint/Makefile | 2 +-
checkpoint/restart.c | 4 +
checkpoint/rstr_file.c | 246 ++++++++++++++++++++++++++++++++++++++++++++
include/linux/checkpoint.h | 1 +
4 files changed, 252 insertions(+), 1 deletions(-)
create mode 100644 checkpoint/rstr_file.c
@@ -219,6 +219,10 @@ static int cr_read_task(struct cr_ctx *ctx)cr_debug("memory: ret %d\n",ret);if(ret<0)gotoout;+ret=cr_read_files(ctx);+cr_debug("files: ret %d\n",ret);+if(ret<0)+gotoout;ret=cr_read_thread(ctx);cr_debug("thread: ret %d\n",ret);if(ret<0)
@@ -0,0 +1,246 @@+/*+*Checkpointfiledescriptors+*+*Copyright(C)2008OrenLaadan+*+*ThisfileissubjecttothetermsandconditionsoftheGNUGeneralPublic+*License.SeethefileCOPYINGinthemaindirectoryoftheLinux+*distributionformoredetails.+*/++#include<linux/kernel.h>+#include<linux/sched.h>+#include<linux/fs.h>+#include<linux/file.h>+#include<linux/fdtable.h>+#include<linux/fsnotify.h>+#include<linux/syscalls.h>+#include<linux/checkpoint.h>+#include<linux/checkpoint_hdr.h>++#include"checkpoint_file.h"++staticintcr_close_all_fds(structfiles_struct*files)+{+int*fdtable;+intnfds;++nfds=cr_scan_fds(files,&fdtable);+if(nfds<0)+returnnfds;+while(nfds--)+sys_close(fdtable[nfds]);+kfree(fdtable);+return0;+}++/**+*cr_attach_file-attachalonelyfileptrtoafiledescriptor+*@file:lonelyfilepointer+*/+staticintcr_attach_file(structfile*file)+{+intfd=get_unused_fd_flags(0);++if(fd>=0){+fsnotify_open(file->f_path.dentry);+fd_install(fd,file);+}+returnfd;+}++/**+*cr_attach_get_file-attach(andget)lonelyfileptrtoafiledescriptor+*@file:lonelyfilepointer+*/+staticintcr_attach_get_file(structfile*file)+{+intfd=get_unused_fd_flags(0);++if(fd>=0){+fsnotify_open(file->f_path.dentry);+fd_install(fd,file);+get_file(file);+}+returnfd;+}++#define CR_SETFL_MASK (O_APPEND|O_NONBLOCK|O_NDELAY|FASYNC|O_DIRECT|O_NOATIME)++/* cr_read_fd_data - restore the state of a given file pointer */+staticint+cr_read_fd_data(structcr_ctx*ctx,structfiles_struct*files,intparent)+{+structcr_hdr_fd_data*hh=cr_hbuf_get(ctx,sizeof(*hh));+structfile*file;+intrparent,ret;+intfd=0;/* pacify gcc warning */++rparent=cr_read_obj_type(ctx,hh,sizeof(*hh),CR_HDR_FD_DATA);+cr_debug("rparent %d parent %d flags %#x mode %#x how %d\n",+rparent,parent,hh->f_flags,hh->f_mode,hh->fd_type);+if(rparent<0){+ret=parent;+gotoout;+}++ret=-EINVAL;++if(rparent!=parent)+gotoout;++/* FIX: more sanity checks on f_flags, f_mode etc */++switch(hh->fd_type){+caseCR_FD_FILE:+caseCR_FD_DIR:+caseCR_FD_LINK:+file=cr_read_open_fname(ctx,hh->f_flags,hh->f_mode);+break;+default:+gotoout;+}++if(IS_ERR(file)){+ret=PTR_ERR(file);+gotoout;+}++/* FIX: need to restore uid, gid, owner etc */++fd=cr_attach_file(file);/* no need to cleanup 'file' below */+if(fd<0){+filp_close(file,NULL);+ret=fd;+gotoout;+}++/* register new <objref, file> tuple in hash table */+ret=cr_obj_add_ref(ctx,(void*)file,parent,CR_OBJ_FILE,0);+if(ret<0)+gotoout;+ret=sys_fcntl(fd,F_SETFL,hh->f_flags&CR_SETFL_MASK);+if(ret<0)+gotoout;+ret=vfs_llseek(file,hh->f_pos,SEEK_SET);+if(ret==-ESPIPE)/* ignore error on non-seekable files */+ret=0;++ret=0;+out:+cr_hbuf_put(ctx,sizeof(*hh));+returnret<0?ret:fd;+}++/**+*cr_read_fd_ent-restorethestateofagivenfiledescriptor+*@ctx:checkpointcontext+*@files:files_structpointer+*@parent:parentobjref+*+*Restoresthestateofafiledescriptor;looksuptheobjref(inthe+*header)inthehashtable,andiffoundpicksthematchingfileand+*useit;otherwisecallscr_read_fd_datatorestorethefiletoo.+*/+staticint+cr_read_fd_ent(structcr_ctx*ctx,structfiles_struct*files,intparent)+{+structcr_hdr_fd_ent*hh=cr_hbuf_get(ctx,sizeof(*hh));+structfile*file;+intnewfd,rparent,ret;++rparent=cr_read_obj_type(ctx,hh,sizeof(*hh),CR_HDR_FD_ENT);+cr_debug("rparent %d parent %d ref %d fd %d c.o.e %d\n",+rparent,parent,hh->objref,hh->fd,hh->close_on_exec);+if(rparent<0){+ret=rparent;+gotoout;+}++ret=-EINVAL;++if(rparent!=parent)+gotoout;+if(hh->objref<=0)+gotoout;++file=cr_obj_get_by_ref(ctx,hh->objref,CR_OBJ_FILE);+if(IS_ERR(file)){+ret=PTR_ERR(file);+gotoout;+}++if(file){+/* reuse file descriptor found in the hash table */+newfd=cr_attach_get_file(file);+}else{+/* create new file pointer (and register in hash table) */+newfd=cr_read_fd_data(ctx,files,hh->objref);+}++if(newfd<0){+ret=newfd;+gotoout;+}++cr_debug("newfd got %d wanted %d\n",newfd,hh->fd);++/* if newfd isn't desired fd then reposition it */+if(newfd!=hh->fd){+ret=sys_dup2(newfd,hh->fd);+if(ret<0)+gotoout;+sys_close(newfd);+}++if(hh->close_on_exec)+set_close_on_exec(hh->fd,1);++ret=0;+out:+cr_hbuf_put(ctx,sizeof(*hh));+returnret;+}++intcr_read_files(structcr_ctx*ctx)+{+structcr_hdr_files*hh=cr_hbuf_get(ctx,sizeof(*hh));+structfiles_struct*files=current->files;+inti,parent,ret;++parent=cr_read_obj_type(ctx,hh,sizeof(*hh),CR_HDR_FILES);+if(parent<0){+ret=parent;+gotoout;+}++ret=-EINVAL;+#if 0 /* activate when containers are used */+if(parent!=task_pid_vnr(current))+gotoout;+#endif+cr_debug("objref %d nfds %d\n",hh->objref,hh->nfds);+if(hh->objref<0||hh->nfds<0)+gotoout;++if(hh->nfds>sysctl_nr_open){+ret=-EMFILE;+gotoout;+}++/* point of no return -- close all file descriptors */+ret=cr_close_all_fds(files);+if(ret<0)+gotoout;++for(i=0;i<hh->nfds;i++){+ret=cr_read_fd_ent(ctx,files,hh->objref);+if(ret<0)+break;+}++ret=0;+out:+cr_hbuf_put(ctx,sizeof(*hh));+returnret;+}
@@ -85,6 +85,7 @@ extern int cr_write_files(struct cr_ctx *ctx, struct task_struct *t);externintdo_restart(structcr_ctx*ctx);externintcr_read_mm(structcr_ctx*ctx);+externintcr_read_files(structcr_ctx*ctx);/* there are from fs/read_write.c, not exported otherwise in a header */externloff_tfile_pos_read(structfile*file);
--
1.5.4.3
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
From: Andrew Morton <akpm@linux-foundation.org> Date: 2008-10-21 19:22:54
On Mon, 20 Oct 2008 01:40:28 -0400
Oren Laadan [off-list ref] wrote:
These patches implement basic checkpoint-restart [CR]. This version
(v7) supports basic tasks with simple private memory, and open files
(regular files and directories only).
This is a problem. I wouldn't want to be in a position where we merge
this code in mainline, but it's just a not-very-useful toy. Then, as
we turn it into a useful non-toy it all turns into an utter mess.
IOW, merging this code as-is will commit us to merging more code which
hasn't even been written yet. It might even commit us to solving
thus-far-unknown problems which we don't know how to solve!
It's a big blank cheque.
So.
- how useful is this code as it stands in real-world usage?
- what additional work needs to be done to it? (important!)
- how far are we down the design and implementation path with that new
work? Are we yet at least in a position where we can say "yes, this
feature can be completed and no, it won't be a horrid mess"?
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Andrew Morton <akpm@linux-foundation.org> Date: 2008-10-21 19:42:29
On Mon, 20 Oct 2008 01:40:30 -0400
Oren Laadan [off-list ref] wrote:
Add those interfaces, as well as helpers needed to easily manage the
file format. The code is roughly broken out as follows:
checkpoint/sys.c - user/kernel data transfer, as well as setup of the
checkpoint/restart context (a per-checkpoint data structure for
housekeeping)
checkpoint/checkpoint.c - output wrappers and basic checkpoint handling
checkpoint/restart.c - input wrappers and basic restart handling
Patches to add the per-architecture support as well as the actual
work to do the memory checkpoint follow in subsequent patches.
...
+int cr_kwrite(struct cr_ctx *ctx, void *buf, int count)
+{
+ mm_segment_t oldfs;
+ int ret;
+
+ oldfs = get_fs();
+ set_fs(KERNEL_DS);
+ ret = cr_uwrite(ctx, buf, count);
+ set_fs(oldfs);
+
+ return ret;
+}
The decision to write files direct from within the kernel is a bit
unusual and needs discussion and justification in the changelog,
please.
Other schemes would be to make the data available to userspace via a
pseudo-fs file, netlink, a pipe, blah, blah.
...
+/*
+ * During checkpoint and restart the code writes outs/reads in data
+ * to/from the chekcpoint image from/to a temporary buffer (ctx->hbuf).
Yuo cnat tpye.
+ * Because operations can be nested, one should call cr_hbuf_get() to
+ * reserve space in the buffer, and then cr_hbuf_put() when no longer
+ * needs that space.
Mangled grammar.
+ */
+
+/*
+ * ctx->hbuf is used to hold headers and data of known (or bound),
+ * static sizes. In some cases, multiple headers may be allocated in
+ * a nested manner. The size should accommodate all headers, nested
+ * or not, on all archs.
+ */
+#define CR_HBUF_TOTAL (8 * 4096)
+
...
+/*
+ * helpers to manage CR contexts: allocated for each checkpoint and/or
+ * restart operation, and persists until the operation is completed.
+ */
+
+/* unique checkpoint identifier (FIXME: should be per-container) */
+static atomic_t cr_ctx_count;
This never gets initialised. Use ATOMIC_INIT() here. (It doesn't
matter, but one day it might!)
...
asmlinkage long sys_checkpoint(pid_t pid, int fd, unsigned long flags)
{
- pr_debug("sys_checkpoint not implemented yet\n");
- return -ENOSYS;
+ struct cr_ctx *ctx;
+ int ret;
+
+ /* no flags for now */
+ if (flags)
+ return -EINVAL;
+
+ ctx = cr_ctx_alloc(pid, fd, flags | CR_CTX_CKPT);
+ if (IS_ERR(ctx))
+ return PTR_ERR(ctx);
+
+ ret = do_checkpoint(ctx);
+
+ if (!ret)
+ ret = ctx->crid;
+
+ cr_ctx_free(ctx);
+ return ret;
}
Is it appropriate that this be an unprivileged operation?
What happens if I pass it a pid which isn't system-wide unique?
What happens if I pass it a pid of a process which I don't own? This
is super security-sensitive and we need to go over the permission
checking with a toothcomb. It needs to be exhaustively described in
the changelog. It might have security/selinux implications - I don't
know, I didn't look, but lights are flashing and bells are ringing over
here.
What happens if I pass it a pid of a process which I _do_ own, but it
does not refer to a container's init process?
If `pid' must refer to a container's init process, isn't it always
equal to 1??
@@ -36,6 +234,19 @@ asmlinkage long sys_checkpoint(pid_t pid, int fd, unsigned long flags) */ asmlinkage long sys_restart(int crid, int fd, unsigned long flags) {- pr_debug("sys_restart not implemented yet\n");- return -ENOSYS;+ struct cr_ctx *ctx;+ int ret;++ /* no flags for now */+ if (flags)+ return -EINVAL;++ ctx = cr_ctx_alloc(crid, fd, flags | CR_CTX_RSTR);+ if (IS_ERR(ctx))+ return PTR_ERR(ctx);++ ret = do_restart(ctx);++ cr_ctx_free(ctx);+ return ret; }
Again, this is scary stuff. We're allowing unprivileged userspace to
feed random numbers into kernel data structures.
I'd like to see the security guys take a real close look at all of
this, and for them to do that effectively they should be provided with
a full description of the security design of this feature.
Might as well move these to a header and inline them everywhere.
That'd be a separate leadin patch.
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Serge E. Hallyn <hidden> Date: 2008-10-21 20:24:29
Quoting Andrew Morton (akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org):
On Mon, 20 Oct 2008 01:40:30 -0400
Oren Laadan [off-list ref] wrote:
quoted
asmlinkage long sys_checkpoint(pid_t pid, int fd, unsigned long flags)
{
- pr_debug("sys_checkpoint not implemented yet\n");
- return -ENOSYS;
+ struct cr_ctx *ctx;
+ int ret;
+
+ /* no flags for now */
+ if (flags)
+ return -EINVAL;
+
+ ctx = cr_ctx_alloc(pid, fd, flags | CR_CTX_CKPT);
+ if (IS_ERR(ctx))
+ return PTR_ERR(ctx);
+
+ ret = do_checkpoint(ctx);
+
+ if (!ret)
+ ret = ctx->crid;
+
+ cr_ctx_free(ctx);
+ return ret;
}
Is it appropriate that this be an unprivileged operation?
Early versions checked capable(CAP_SYS_ADMIN), and we reasoned that we
would later attempt to remove the need for privilege so that all users
could safely use it.
Arnd Bergmann called us on that nonsense, pointing out that it'd make
more sense to let unprivileged users use them now, so that we'll be
more careful about the security as patches roll in.
So, Oren's patchset right now only checkpoints current, despite pid
being part of the API. So the task can access its own data. When
the patch supports checkpointing another task (which Oren says he's
doing right now), then our intent is to check for ptrace access to
the target task. (Right, Oren?)
What happens if I pass it a pid which isn't system-wide unique?
pid must be checked in the caller's pid namespace. So if I've create a
container which I want to checkpoint, pid 1 in that pidns will be, say,
3497 in my pid_ns, and so 3497 is the pid I must use. If I try to pass
1, I'll try to checkpoint my own container. And, if I'm not privileged
and init is owned by root, the ptrace() check I mentioned above will
return -EPERM.
What happens if I pass it a pid of a process which I don't own? This
is super security-sensitive and we need to go over the permission
checking with a toothcomb. It needs to be exhaustively described in
the changelog. It might have security/selinux implications - I don't
know, I didn't look, but lights are flashing and bells are ringing over
here.
What happens if I pass it a pid of a process which I _do_ own, but it
does not refer to a container's init process?
I would assume that do_checkpoint() would return -EINVAL, but it's a
great question: Oren, did you have another plan?
If `pid' must refer to a container's init process, isn't it always
equal to 1??
@@ -36,6 +234,19 @@ asmlinkage long sys_checkpoint(pid_t pid, int fd, unsigned long flags) */ asmlinkage long sys_restart(int crid, int fd, unsigned long flags) {- pr_debug("sys_restart not implemented yet\n");- return -ENOSYS;+ struct cr_ctx *ctx;+ int ret;++ /* no flags for now */+ if (flags)+ return -EINVAL;++ ctx = cr_ctx_alloc(crid, fd, flags | CR_CTX_RSTR);+ if (IS_ERR(ctx))+ return PTR_ERR(ctx);++ ret = do_restart(ctx);++ cr_ctx_free(ctx);+ return ret; }
Again, this is scary stuff. We're allowing unprivileged userspace to
feed random numbers into kernel data structures.
Yes, all of the file opens and mmaps must not skip the usual security
checks. The task credentials are currently unsupported, meaning that
euid, etc, come from the caller, not the checkpoint image. When the
restoration of credentials becomes supported, then definately the
caller (of sys_restore())'s ability to setresuid/setresgid to those
values must be checked.
So that's why we don't want CAP_SYS_ADMIN required up-front. That way
we will be forced to more carefully review each of those features.
I'd like to see the security guys take a real close look at all of
this, and for them to do that effectively they should be provided with
a full description of the security design of this feature.
Right, some of the above should be spelled out somewhere. Should it be
in the patch description, in the Documentation/checkpoint.txt file,
or someplace else? Oren, do you want to filter the above information
into the right place, or do you want me to do it and send you a patch?
Might as well move these to a header and inline them everywhere.
That'd be a separate leadin patch.
thanks,
-serge
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Dave Hansen <hidden> Date: 2008-10-21 20:41:23
On Tue, 2008-10-21 at 12:21 -0700, Andrew Morton wrote:
On Mon, 20 Oct 2008 01:40:28 -0400
Oren Laadan [off-list ref] wrote:
quoted
These patches implement basic checkpoint-restart [CR]. This version
(v7) supports basic tasks with simple private memory, and open files
(regular files and directories only).
- how useful is this code as it stands in real-world usage?
Right now, an application must be specifically written to use these mew
system calls. It must be a single process and not share any resources
with other processes. The only file descriptors that may be open are
simple files and may not include sockets or pipes.
What this means in practice is that it is useful for a simple app doing
computational work.
- what additional work needs to be done to it? (important!)
- how far are we down the design and implementation path with that new
work?
We know this design can work. We have two commercial products and a
horde of academic projects doing it today using this basic design.
We're early in this particular implementation because we're trying to
release early and often.
I think we're at the point where we need a yes or no from the rest of
the community on it. Reading the patches, I'd hope a reviewer can get
an idea how this will extend to other subsystems. Do you think the
current patches aren't enough from which to extrapolate how this will be
extended?
Are we yet at least in a position where we can say "yes, this
feature can be completed and no, it won't be a horrid mess"?
It will be complete a few months after the rest of the kernel is
complete. :)
From these patches, I think you can see that this will largely be
something that can live off in its own corner of the tree. We will, of
course, need to do plenty of refactoring of existing code (like the pid
namespaces for instance) to make some of it more accessible from the
outside. We're also going to look for every opportunity to share code
with other users like the freezer.
-- Dave
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Andrew Morton <akpm@linux-foundation.org> Date: 2008-10-21 20:42:21
On Tue, 21 Oct 2008 15:24:10 -0500
"Serge E. Hallyn" [off-list ref] wrote:
quoted
I'd like to see the security guys take a real close look at all of
this, and for them to do that effectively they should be provided with
a full description of the security design of this feature.
Right, some of the above should be spelled out somewhere. Should it be
in the patch description, in the Documentation/checkpoint.txt file,
or someplace else?
Dupliction is usually bad. Documentation/checkpoint.txt would be good
(although these things tend to go out of date fast).
If you go that way, please ensure that the documentation patch is early
in the series and that the changelog says "look in here before whining,
dummy".
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoting Andrew Morton (akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org):
quoted
On Mon, 20 Oct 2008 01:40:30 -0400
Oren Laadan [off-list ref] wrote:
quoted
asmlinkage long sys_checkpoint(pid_t pid, int fd, unsigned long flags)
{
- pr_debug("sys_checkpoint not implemented yet\n");
- return -ENOSYS;
+ struct cr_ctx *ctx;
+ int ret;
+
+ /* no flags for now */
+ if (flags)
+ return -EINVAL;
+
+ ctx = cr_ctx_alloc(pid, fd, flags | CR_CTX_CKPT);
+ if (IS_ERR(ctx))
+ return PTR_ERR(ctx);
+
+ ret = do_checkpoint(ctx);
+
+ if (!ret)
+ ret = ctx->crid;
+
+ cr_ctx_free(ctx);
+ return ret;
}
Is it appropriate that this be an unprivileged operation?
Early versions checked capable(CAP_SYS_ADMIN), and we reasoned that we
would later attempt to remove the need for privilege so that all users
could safely use it.
Arnd Bergmann called us on that nonsense, pointing out that it'd make
more sense to let unprivileged users use them now, so that we'll be
more careful about the security as patches roll in.
So, Oren's patchset right now only checkpoints current, despite pid
being part of the API. So the task can access its own data. When
the patch supports checkpointing another task (which Oren says he's
doing right now), then our intent is to check for ptrace access to
the target task. (Right, Oren?)
What happens if I pass it a pid which isn't system-wide unique?
pid must be checked in the caller's pid namespace. So if I've create a
container which I want to checkpoint, pid 1 in that pidns will be, say,
3497 in my pid_ns, and so 3497 is the pid I must use. If I try to pass
1, I'll try to checkpoint my own container. And, if I'm not privileged
and init is owned by root, the ptrace() check I mentioned above will
return -EPERM.
Yup.
quoted
What happens if I pass it a pid of a process which I don't own? This
is super security-sensitive and we need to go over the permission
checking with a toothcomb. It needs to be exhaustively described in
the changelog. It might have security/selinux implications - I don't
know, I didn't look, but lights are flashing and bells are ringing over
here.
This should be covered by ptrace_may_access() test.
In the longer run, I suppose SElinux people would want a security hook
there to approve or disapprove the operation.
quoted
What happens if I pass it a pid of a process which I _do_ own, but it
does not refer to a container's init process?
I would assume that do_checkpoint() would return -EINVAL, but it's a
great question: Oren, did you have another plan?
Since we intentional provide minimal functionality to keep the patchset
simple and allow easy review - we only checkpoint one task; it doesn't
really matter because we don't deal with the entire container.
With the ability to checkpoint multiple process we will have to ensure
that we checkpoint an entire container. I planned to return -EINVAL if
the target task isn't a container init(1). Another option, if people
prefer, is to use any task in a container to "represent" the entire
container.
quoted
If `pid' must refer to a container's init process, isn't it always
equal to 1??
@@ -36,6 +234,19 @@ asmlinkage long sys_checkpoint(pid_t pid, int fd, unsigned long flags) */ asmlinkage long sys_restart(int crid, int fd, unsigned long flags) {- pr_debug("sys_restart not implemented yet\n");- return -ENOSYS;+ struct cr_ctx *ctx;+ int ret;++ /* no flags for now */+ if (flags)+ return -EINVAL;++ ctx = cr_ctx_alloc(crid, fd, flags | CR_CTX_RSTR);+ if (IS_ERR(ctx))+ return PTR_ERR(ctx);++ ret = do_restart(ctx);++ cr_ctx_free(ctx);+ return ret; }
Again, this is scary stuff. We're allowing unprivileged userspace to
feed random numbers into kernel data structures.
Yes, all of the file opens and mmaps must not skip the usual security
checks. The task credentials are currently unsupported, meaning that
euid, etc, come from the caller, not the checkpoint image. When the
Actually, the fact that task credentials are not restored makes it
more secure, because the user can't do anything beyond her current
capabilities.
For the same reason, however, unless we agree on a secure way to
elevate credentials, there are various things that we cannot restore,
even though it may be something we would want to permit.
restoration of credentials becomes supported, then definately the
caller (of sys_restore())'s ability to setresuid/setresgid to those
values must be checked.
So that's why we don't want CAP_SYS_ADMIN required up-front. That way
we will be forced to more carefully review each of those features.
quoted
I'd like to see the security guys take a real close look at all of
this, and for them to do that effectively they should be provided with
a full description of the security design of this feature.
Right, some of the above should be spelled out somewhere. Should it be
in the patch description, in the Documentation/checkpoint.txt file,
or someplace else? Oren, do you want to filter the above information
into the right place, or do you want me to do it and send you a patch?
I'll add something to the Documentation/checkpoint.txt.
Thanks,
Oren.
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Dave Hansen <hidden> Date: 2008-10-22 03:02:57
On Tue, 2008-10-21 at 22:55 -0400, Daniel Jacobowitz wrote:
I haven't been following - but why this whole container restriction?
Checkpoint/restart of individual processes is very useful too.
There are issues with e.g. IPC, but I'm not convinced they're
substantially different than the issues already present for a
container.
Containers provide isolation. Once you have isolation, you have a
discrete set of resources which you can checkpoint/restart.
Let's say you have a process you want to checkpoint. If it uses a
completely discrete IPC namespace, you *know* that nothing else depends
on those IPC ids. We don't even have to worry about who might have been
using them and when.
Also think about pids. Without containers, how can you guarantee a
restarted process that it can regain the same pid?
-- Dave
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Daniel Jacobowitz <hidden> Date: 2008-10-22 03:16:19
On Tue, Oct 21, 2008 at 09:33:19PM -0400, Oren Laadan wrote:
quoted
quoted
What happens if I pass it a pid of a process which I _do_ own, but it
does not refer to a container's init process?
I would assume that do_checkpoint() would return -EINVAL, but it's a
great question: Oren, did you have another plan?
Since we intentional provide minimal functionality to keep the patchset
simple and allow easy review - we only checkpoint one task; it doesn't
really matter because we don't deal with the entire container.
With the ability to checkpoint multiple process we will have to ensure
that we checkpoint an entire container. I planned to return -EINVAL if
the target task isn't a container init(1). Another option, if people
prefer, is to use any task in a container to "represent" the entire
container.
I haven't been following - but why this whole container restriction?
Checkpoint/restart of individual processes is very useful too.
There are issues with e.g. IPC, but I'm not convinced they're
substantially different than the issues already present for a
container.
--
Daniel Jacobowitz
CodeSourcery
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
On Tue, 2008-10-21 at 12:21 -0700, Andrew Morton wrote:
quoted
On Mon, 20 Oct 2008 01:40:28 -0400
Oren Laadan [off-list ref] wrote:
quoted
These patches implement basic checkpoint-restart [CR]. This version
(v7) supports basic tasks with simple private memory, and open files
(regular files and directories only).
- how useful is this code as it stands in real-world usage?
Right now, an application must be specifically written to use these
mew system calls. It must be a single process and not share any
resources with other processes. The only file descriptors that may be
open are simple files and may not include sockets or pipes.
What this means in practice is that it is useful for a simple app
doing computational work.
say a chemistry application doing calculations. Or a raytracer with a
large job. Both can take many hours (days!) even on very fast machine
and the restrictions on rebootability can hurt in such cases.
You should reach a minimal level of initial practical utility: say some
helper tool that allows testers to checkpoint and restore a real PovRay
session - without any modification to a stock distro PovRay.
Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Daniel Lezcano <hidden> Date: 2008-10-22 11:51:46
Ingo Molnar wrote:
* Dave Hansen [off-list ref] wrote:
quoted
On Tue, 2008-10-21 at 12:21 -0700, Andrew Morton wrote:
quoted
On Mon, 20 Oct 2008 01:40:28 -0400
Oren Laadan [off-list ref] wrote:
quoted
These patches implement basic checkpoint-restart [CR]. This version
(v7) supports basic tasks with simple private memory, and open files
(regular files and directories only).
- how useful is this code as it stands in real-world usage?
Right now, an application must be specifically written to use these
mew system calls. It must be a single process and not share any
resources with other processes. The only file descriptors that may be
open are simple files and may not include sockets or pipes.
What this means in practice is that it is useful for a simple app
doing computational work.
say a chemistry application doing calculations. Or a raytracer with a
large job. Both can take many hours (days!) even on very fast machine
and the restrictions on rebootability can hurt in such cases.
You should reach a minimal level of initial practical utility: say some
helper tool that allows testers to checkpoint and restore a real PovRay
session - without any modification to a stock distro PovRay.
There are the liblxc userspace tools doing that.
http://sourceforge.net/projects/lxc/
There are the lxc-checkpoint and lxc-restart commands to test the Oren's
patches with the external checkpoint Cedric did. These commands are
experimental and under development so a hack may be necessary for
checkpoint/restart.
I didn't tried with Oren's external checkpoint yet, but I think the
commands should work. Actually these commands relies on the freezer, so
the checkpoint command does freeze, checkpoint, unfreeze. (and kill if
specified).
lxc-create -n foo
lxc-start -n foo mypovray
lxc-checkpoint -s -n foo > myckptfile
lxc-restart -n foo < myckptfile
Thanks
-- Daniel
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Cedric Le Goater <hidden> Date: 2008-10-22 11:55:46
Ingo Molnar wrote:
* Dave Hansen [off-list ref] wrote:
quoted
On Tue, 2008-10-21 at 12:21 -0700, Andrew Morton wrote:
quoted
On Mon, 20 Oct 2008 01:40:28 -0400
Oren Laadan [off-list ref] wrote:
quoted
These patches implement basic checkpoint-restart [CR]. This version
(v7) supports basic tasks with simple private memory, and open files
(regular files and directories only).
- how useful is this code as it stands in real-world usage?
Right now, an application must be specifically written to use these
mew system calls. It must be a single process and not share any
resources with other processes. The only file descriptors that may be
open are simple files and may not include sockets or pipes.
What this means in practice is that it is useful for a simple app
doing computational work.
say a chemistry application doing calculations. Or a raytracer with a
large job. Both can take many hours (days!) even on very fast machine
even weeks in the EDA and Petroleum geophysics.
and the restrictions on rebootability can hurt in such cases.
yes, indeed.
These industries also like to be able to schedule high priority jobs
needing the full power of their clusters: checkpoint running jobs,
schedule a high priority one, restart the previous.
You should reach a minimal level of initial practical utility: say some
helper tool that allows testers to checkpoint and restore a real PovRay
session - without any modification to a stock distro PovRay.
Supporting Povray is a good target. many HPC applications have the same
resource scope.
C.
From: Daniel Jacobowitz <hidden> Date: 2008-10-22 14:29:21
On Tue, Oct 21, 2008 at 08:02:43PM -0700, Dave Hansen wrote:
Let's say you have a process you want to checkpoint. If it uses a
completely discrete IPC namespace, you *know* that nothing else depends
on those IPC ids. We don't even have to worry about who might have been
using them and when.
Also think about pids. Without containers, how can you guarantee a
restarted process that it can regain the same pid?
OK, that makes sense. In a lot of simple cases you can get by without
regaining the same pid; there's an implementation of checkpointing in
GDB that works by injecting fork calls into the child, and it is
useful for a reasonable selection of single-threaded programs.
--
Daniel Jacobowitz
CodeSourcery
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoting Andrew Morton (akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org):
quoted
On Mon, 20 Oct 2008 01:40:30 -0400
Oren Laadan [off-list ref] wrote:
quoted
asmlinkage long sys_checkpoint(pid_t pid, int fd, unsigned long flags)
{
- pr_debug("sys_checkpoint not implemented yet\n");
- return -ENOSYS;
+ struct cr_ctx *ctx;
+ int ret;
+
+ /* no flags for now */
+ if (flags)
+ return -EINVAL;
+
+ ctx = cr_ctx_alloc(pid, fd, flags | CR_CTX_CKPT);
+ if (IS_ERR(ctx))
+ return PTR_ERR(ctx);
+
+ ret = do_checkpoint(ctx);
+
+ if (!ret)
+ ret = ctx->crid;
+
+ cr_ctx_free(ctx);
+ return ret;
}
Is it appropriate that this be an unprivileged operation?
Early versions checked capable(CAP_SYS_ADMIN), and we reasoned that we
would later attempt to remove the need for privilege so that all users
could safely use it.
Arnd Bergmann called us on that nonsense, pointing out that it'd make
more sense to let unprivileged users use them now, so that we'll be
more careful about the security as patches roll in.
So, Oren's patchset right now only checkpoints current, despite pid
being part of the API. So the task can access its own data. When
the patch supports checkpointing another task (which Oren says he's
doing right now), then our intent is to check for ptrace access to
the target task. (Right, Oren?)
Correct. That's already in the additional patch in the git tree - first
I locate the task and if found, I check ptrace_may_access() (read mode).
Just thinking aloud...
Is read mode appropriate? The user can edit the statefile and restart
it. Admittedly the restart code should then do all the appropriate
checks for recreating resources, but I'm having a hard time thinking
through this straight.
Let's say hallyn is running passwd. ruid=500,euid=0. He quickly
checkpoints. Then he restarts. Will restart say "ok, the /bin/passwd
binary is setuid 0 so let hallyn take euid=0 for this?" I guess not.
But are there other resources for which this is harder to get right?
...
This should be covered by ptrace_may_access() test.
In the longer run, I suppose SElinux people would want a security hook
there to approve or disapprove the operation.
I think we'll find the ptrace() checks to be so like what we're doing
that no new check will be needed. But we should definately ask them.
Now may be too early to ask, though. The answer will be clearer once
more resources are supported.
quoted
quoted
What happens if I pass it a pid of a process which I _do_ own, but it
does not refer to a container's init process?
I would assume that do_checkpoint() would return -EINVAL, but it's a
great question: Oren, did you have another plan?
Since we intentional provide minimal functionality to keep the patchset
simple and allow easy review - we only checkpoint one task; it doesn't
really matter because we don't deal with the entire container.
With the ability to checkpoint multiple process we will have to ensure
that we checkpoint an entire container. I planned to return -EINVAL if
the target task isn't a container init(1). Another option, if people
prefer, is to use any task in a container to "represent" the entire
container.
Except we support nested containers, so unless we only support
checkpoint of the deepest container, that doesn't work.
...
quoted
quoted
Again, this is scary stuff. We're allowing unprivileged userspace to
feed random numbers into kernel data structures.
Yes, all of the file opens and mmaps must not skip the usual security
checks. The task credentials are currently unsupported, meaning that
euid, etc, come from the caller, not the checkpoint image. When the
Actually, the fact that task credentials are not restored makes it
more secure, because the user can't do anything beyond her current
capabilities.
Hmm, so do you think we just always use the caller's credentials?
If we were to use some sort of tpm-signing of statefiles, then
hallyn restarting a checkpointed /bin/passwd may become doable.
For the same reason, however, unless we agree on a secure way to
elevate credentials, there are various things that we cannot restore,
even though it may be something we would want to permit.
quoted
restoration of credentials becomes supported, then definately the
caller (of sys_restore())'s ability to setresuid/setresgid to those
values must be checked.
So that's why we don't want CAP_SYS_ADMIN required up-front. That way
we will be forced to more carefully review each of those features.
quoted
I'd like to see the security guys take a real close look at all of
this, and for them to do that effectively they should be provided with
a full description of the security design of this feature.
Right, some of the above should be spelled out somewhere. Should it be
in the patch description, in the Documentation/checkpoint.txt file,
or someplace else? Oren, do you want to filter the above information
into the right place, or do you want me to do it and send you a patch?
I'll add something to the Documentation/checkpoint.txt.
Cool, thanks Oren.
-serge
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoting Andrew Morton (akpm-de/tnXTf+JLsfHDXvbKv3WD2FQJk+8+b@public.gmane.org):
quoted
On Mon, 20 Oct 2008 01:40:30 -0400
Oren Laadan [off-list ref] wrote:
quoted
asmlinkage long sys_checkpoint(pid_t pid, int fd, unsigned long flags)
{
- pr_debug("sys_checkpoint not implemented yet\n");
- return -ENOSYS;
+ struct cr_ctx *ctx;
+ int ret;
+
+ /* no flags for now */
+ if (flags)
+ return -EINVAL;
+
+ ctx = cr_ctx_alloc(pid, fd, flags | CR_CTX_CKPT);
+ if (IS_ERR(ctx))
+ return PTR_ERR(ctx);
+
+ ret = do_checkpoint(ctx);
+
+ if (!ret)
+ ret = ctx->crid;
+
+ cr_ctx_free(ctx);
+ return ret;
}
Is it appropriate that this be an unprivileged operation?
Early versions checked capable(CAP_SYS_ADMIN), and we reasoned that we
would later attempt to remove the need for privilege so that all users
could safely use it.
Arnd Bergmann called us on that nonsense, pointing out that it'd make
more sense to let unprivileged users use them now, so that we'll be
more careful about the security as patches roll in.
So, Oren's patchset right now only checkpoints current, despite pid
being part of the API. So the task can access its own data. When
the patch supports checkpointing another task (which Oren says he's
doing right now), then our intent is to check for ptrace access to
the target task. (Right, Oren?)
Correct. That's already in the additional patch in the git tree - first
I locate the task and if found, I check ptrace_may_access() (read mode).
Just thinking aloud...
Is read mode appropriate? The user can edit the statefile and restart
it. Admittedly the restart code should then do all the appropriate
checks for recreating resources, but I'm having a hard time thinking
through this straight.
Let's say hallyn is running passwd. ruid=500,euid=0. He quickly
checkpoints. Then he restarts. Will restart say "ok, the /bin/passwd
binary is setuid 0 so let hallyn take euid=0 for this?" I guess not.
But are there other resources for which this is harder to get right?
I'd say that checkpoint and restart are separate.
In checkpoint, you read the state and save it somewhere; you don't
modify anything in the target task (container). This equivalent to
ptrace read-mode. If you could do ptrace, you could save all that
state. In fact, you could save it in a format that is suitable for
a future restart ... (or just forge one !)
In restart, we either don't trust the user and keep everything to
be done with her credentials, of we trust the root user and allow
all operations (like loading a kernel module).
We can actually have both modes of operations. How to decide that
we trust the user is a separate question: one option is to have
both checkpoint and restart executables setuid - checkpoint will
sign (in user space) the output image, and restart (in user space)
will validate the signature, before passing it to the kenrel. Surely
there are other ways...
...
quoted
This should be covered by ptrace_may_access() test.
In the longer run, I suppose SElinux people would want a security hook
there to approve or disapprove the operation.
I think we'll find the ptrace() checks to be so like what we're doing
that no new check will be needed. But we should definately ask them.
Now may be too early to ask, though. The answer will be clearer once
more resources are supported.
quoted
quoted
quoted
What happens if I pass it a pid of a process which I _do_ own, but it
does not refer to a container's init process?
I would assume that do_checkpoint() would return -EINVAL, but it's a
great question: Oren, did you have another plan?
Since we intentional provide minimal functionality to keep the patchset
simple and allow easy review - we only checkpoint one task; it doesn't
really matter because we don't deal with the entire container.
With the ability to checkpoint multiple process we will have to ensure
that we checkpoint an entire container. I planned to return -EINVAL if
the target task isn't a container init(1). Another option, if people
prefer, is to use any task in a container to "represent" the entire
container.
Except we support nested containers, so unless we only support
checkpoint of the deepest container, that doesn't work.
...
yeah.. I just though about it this mornig ;)
quoted
quoted
quoted
Again, this is scary stuff. We're allowing unprivileged userspace to
feed random numbers into kernel data structures.
Yes, all of the file opens and mmaps must not skip the usual security
checks. The task credentials are currently unsupported, meaning that
euid, etc, come from the caller, not the checkpoint image. When the
Actually, the fact that task credentials are not restored makes it
more secure, because the user can't do anything beyond her current
capabilities.
Hmm, so do you think we just always use the caller's credentials?
Nope, since we will fail to restart in many cases. We will need a way
to move from caller's credentials to saved credentials, and even from
caller's credentials to privileged credentials (e.g. to reopen a file
that was created by a setuid program prior to dropping privileges).
To do that, we will need to agree on a way to escalate/change the
credentials. This however belongs to user-space (and then the binaries
for checkpoint/restart will be setuid themselves).
There will also be the issue of mapping credentials: a user A may have
one UID/GID on once system and another UID/GID on another system, and
we may want to do the conversion. This, too, can be done in user space
prior to restart by using an appropriate filter through the checkpoint
stream.
Oren.
If we were to use some sort of tpm-signing of statefiles, then
hallyn restarting a checkpointed /bin/passwd may become doable.
quoted
For the same reason, however, unless we agree on a secure way to
elevate credentials, there are various things that we cannot restore,
even though it may be something we would want to permit.
quoted
restoration of credentials becomes supported, then definately the
caller (of sys_restore())'s ability to setresuid/setresgid to those
values must be checked.
So that's why we don't want CAP_SYS_ADMIN required up-front. That way
we will be forced to more carefully review each of those features.
quoted
I'd like to see the security guys take a real close look at all of
this, and for them to do that effectively they should be provided with
a full description of the security design of this feature.
Right, some of the above should be spelled out somewhere. Should it be
in the patch description, in the Documentation/checkpoint.txt file,
or someplace else? Oren, do you want to filter the above information
into the right place, or do you want me to do it and send you a patch?
I'll add something to the Documentation/checkpoint.txt.
Cool, thanks Oren.
-serge
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Serge E. Hallyn <hidden> Date: 2008-10-22 17:04:23
Quoting Oren Laadan (orenl@cs.columbia.edu):
Serge E. Hallyn wrote:
quoted
Quoting Oren Laadan (orenl@cs.columbia.edu):
Just thinking aloud...
Is read mode appropriate? The user can edit the statefile and restart
it. Admittedly the restart code should then do all the appropriate
checks for recreating resources, but I'm having a hard time thinking
through this straight.
Let's say hallyn is running passwd. ruid=500,euid=0. He quickly
checkpoints. Then he restarts. Will restart say "ok, the /bin/passwd
binary is setuid 0 so let hallyn take euid=0 for this?" I guess not.
But are there other resources for which this is harder to get right?
I'd say that checkpoint and restart are separate.
In checkpoint, you read the state and save it somewhere; you don't
modify anything in the target task (container). This equivalent to
ptrace read-mode. If you could do ptrace, you could save all that
state. In fact, you could save it in a format that is suitable for
a future restart ... (or just forge one !)
Yeah, that's convincing.
In restart, we either don't trust the user and keep everything to
be done with her credentials, of we trust the root user and allow
all operations (like loading a kernel module).
We can actually have both modes of operations. How to decide that
we trust the user is a separate question: one option is to have
both checkpoint and restart executables setuid - checkpoint will
sign (in user space) the output image, and restart (in user space)
will validate the signature, before passing it to the kenrel. Surely
there are other ways...
Makes sense.
...
quoted
Hmm, so do you think we just always use the caller's credentials?
Nope, since we will fail to restart in many cases. We will need a way
to move from caller's credentials to saved credentials, and even from
caller's credentials to privileged credentials (e.g. to reopen a file
that was created by a setuid program prior to dropping privileges).
Can we agree to worry about that much much later? :) Would you agree
that for the majority of use-cases, restarting with caller's credentials
will work? Or am I wrong about that?
To do that, we will need to agree on a way to escalate/change the
credentials. This however belongs to user-space (and then the binaries
for checkpoint/restart will be setuid themselves).
Ok those are less scary, and I have no problem with those.
There will also be the issue of mapping credentials: a user A may have
one UID/GID on once system and another UID/GID on another system, and
we may want to do the conversion. This, too, can be done in user space
prior to restart by using an appropriate filter through the checkpoint
stream.
User namespaces may help here too. So user A can create a new user
namespace and restart as user B in that namespace. But right now that
sounds like overkill.
-serge
Quoting Oren Laadan (orenl-eQaUEPhvms7ENvBUuze7eA@public.gmane.org):
Just thinking aloud...
Is read mode appropriate? The user can edit the statefile and restart
it. Admittedly the restart code should then do all the appropriate
checks for recreating resources, but I'm having a hard time thinking
through this straight.
Let's say hallyn is running passwd. ruid=500,euid=0. He quickly
checkpoints. Then he restarts. Will restart say "ok, the /bin/passwd
binary is setuid 0 so let hallyn take euid=0 for this?" I guess not.
But are there other resources for which this is harder to get right?
I'd say that checkpoint and restart are separate.
In checkpoint, you read the state and save it somewhere; you don't
modify anything in the target task (container). This equivalent to
ptrace read-mode. If you could do ptrace, you could save all that
state. In fact, you could save it in a format that is suitable for
a future restart ... (or just forge one !)
Yeah, that's convincing.
quoted
In restart, we either don't trust the user and keep everything to
be done with her credentials, of we trust the root user and allow
all operations (like loading a kernel module).
We can actually have both modes of operations. How to decide that
we trust the user is a separate question: one option is to have
both checkpoint and restart executables setuid - checkpoint will
sign (in user space) the output image, and restart (in user space)
will validate the signature, before passing it to the kenrel. Surely
there are other ways...
Makes sense.
...
quoted
quoted
Hmm, so do you think we just always use the caller's credentials?
Nope, since we will fail to restart in many cases. We will need a way
to move from caller's credentials to saved credentials, and even from
caller's credentials to privileged credentials (e.g. to reopen a file
that was created by a setuid program prior to dropping privileges).
Can we agree to worry about that much much later? :) Would you agree
Definitely. Even more so - I believe that's a user-space issue :)
that for the majority of use-cases, restarting with caller's credentials
will work? Or am I wrong about that?
That depends on your target audience. For HPC you're probably right.
For server applications this may not be the case (e.g. apache needs
a privileged port, and then it drops privileges).
I agree that we may safely (...) defer this discussion until the
implementation gets much beefier.
quoted
To do that, we will need to agree on a way to escalate/change the
credentials. This however belongs to user-space (and then the binaries
for checkpoint/restart will be setuid themselves).
Ok those are less scary, and I have no problem with those.
quoted
There will also be the issue of mapping credentials: a user A may have
one UID/GID on once system and another UID/GID on another system, and
we may want to do the conversion. This, too, can be done in user space
prior to restart by using an appropriate filter through the checkpoint
stream.
User namespaces may help here too. So user A can create a new user
namespace and restart as user B in that namespace. But right now that
sounds like overkill.
Indeed, virtualization is probably the solution. Here, too, I think
it's safe to defer the discussion.
Oren.
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Peter Chubb <hidden> Date: 2008-10-27 08:28:49
quoted
quoted
quoted
quoted
"Oren" == Oren Laadan [off-list ref] writes:
Oren> Nope, since we will fail to restart in many cases. We will need
Oren> a way to move from caller's credentials to saved credentials,
Oren> and even from caller's credentials to privileged credentials
Oren> (e.g. to reopen a file that was created by a setuid program
Oren> prior to dropping privileges).
You can't necessarily tell the difference between this and revocation
of privilege. For most security models, it must be possible to change
the permissions on the file, and then the restart should fail.
In our implementation, we simply refused to checkpoint setid programs.
--
Dr Peter Chubb http://www.gelato.unsw.edu.au peterc AT gelato.unsw.edu.au
http://www.ertos.nicta.com.au ERTOS within National ICT Australia
Oren> Nope, since we will fail to restart in many cases. We will need
Oren> a way to move from caller's credentials to saved credentials,
Oren> and even from caller's credentials to privileged credentials
Oren> (e.g. to reopen a file that was created by a setuid program
Oren> prior to dropping privileges).
You can't necessarily tell the difference between this and revocation
of privilege. For most security models, it must be possible to change
the permissions on the file, and then the restart should fail.
In our implementation, we simply refused to checkpoint setid programs.
True. And this works very well for HPC applications.
However, it doesn't work so well for server applications, for instance.
Also, you could use file system snapshotting to ensure that the file
system view does not change, and still face the same issue.
So I'm perfectly ok with deferring this discussion to a later time :)
Oren.
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Dave Hansen <hidden> Date: 2008-10-27 16:43:18
On Mon, 2008-10-27 at 07:03 -0400, Oren Laadan wrote:
quoted
In our implementation, we simply refused to checkpoint setid
programs.
True. And this works very well for HPC applications.
However, it doesn't work so well for server applications, for
instance.
Also, you could use file system snapshotting to ensure that the file
system view does not change, and still face the same issue.
So I'm perfectly ok with deferring this discussion to a later time :)
Oren, is this a good place to stick a process_deny_checkpoint()? Both
so we refuse to checkpoint, and document this as something that has to
be addressed later?
-- Dave
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, 2008-10-27 at 07:03 -0400, Oren Laadan wrote:
quoted
quoted
In our implementation, we simply refused to checkpoint setid
programs.
True. And this works very well for HPC applications.
However, it doesn't work so well for server applications, for
instance.
Also, you could use file system snapshotting to ensure that the file
system view does not change, and still face the same issue.
So I'm perfectly ok with deferring this discussion to a later time :)
Oren, is this a good place to stick a process_deny_checkpoint()? Both
so we refuse to checkpoint, and document this as something that has to
be addressed later?
why refuse to checkpoint ?
if I'm root, and I want to checkpoint, and later restart, my sshd server
(assuming we support listening sockets) - then why not ?
we can just let it be, and have the restart fail (if it isn't root that
does the restart); perhaps add something like warn_checkpoint() (similar
to deny, but only warns) ?
Oren.
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Matt Helsley <hidden> Date: 2008-10-27 20:52:08
On Mon, 2008-10-27 at 13:11 -0400, Oren Laadan wrote:
Dave Hansen wrote:
quoted
On Mon, 2008-10-27 at 07:03 -0400, Oren Laadan wrote:
quoted
quoted
In our implementation, we simply refused to checkpoint setid
programs.
True. And this works very well for HPC applications.
However, it doesn't work so well for server applications, for
instance.
Also, you could use file system snapshotting to ensure that the file
system view does not change, and still face the same issue.
So I'm perfectly ok with deferring this discussion to a later time :)
Oren, is this a good place to stick a process_deny_checkpoint()? Both
so we refuse to checkpoint, and document this as something that has to
be addressed later?
why refuse to checkpoint ?
If most setuid programs hold privileged resources for extended periods
of time after dropping privileges then it seems like a good idea to
refuse to checkpoint. Restart of those programs would be quite
unreliable unless/until we find a nice solution.
if I'm root, and I want to checkpoint, and later restart, my sshd server
(assuming we support listening sockets) - then why not ?
we can just let it be, and have the restart fail (if it isn't root that
does the restart); perhaps add something like warn_checkpoint() (similar
to deny, but only warns) ?
How will folks not specializing in checkpoint/restart know when to use
this as opposed to deny?
Instead, how about a flag to sys_checkpoint() -- DO_RISKY_CHECKPOINT --
which checkpoints despite !may_checkpoint?
Cheers,
-Matt Helsley
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
From: Serge E. Hallyn <hidden> Date: 2008-10-27 21:20:49
Quoting Matt Helsley (matthltc-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org):
On Mon, 2008-10-27 at 13:11 -0400, Oren Laadan wrote:
quoted
Dave Hansen wrote:
quoted
On Mon, 2008-10-27 at 07:03 -0400, Oren Laadan wrote:
quoted
quoted
In our implementation, we simply refused to checkpoint setid
programs.
True. And this works very well for HPC applications.
However, it doesn't work so well for server applications, for
instance.
Also, you could use file system snapshotting to ensure that the file
system view does not change, and still face the same issue.
So I'm perfectly ok with deferring this discussion to a later time :)
Oren, is this a good place to stick a process_deny_checkpoint()? Both
so we refuse to checkpoint, and document this as something that has to
be addressed later?
why refuse to checkpoint ?
If most setuid programs hold privileged resources for extended periods
of time after dropping privileges then it seems like a good idea to
refuse to checkpoint. Restart of those programs would be quite
unreliable unless/until we find a nice solution.
I agree with Dave and Matt. Let's assume that we have a setuid root
program which creates some resources then drops to username kooky. If
you now checkpoint and restart that program, then a stupid restart will
either
1. be done as user kooky and not be able to recreate the
resources, fail.
2. be done as user root and not drop uid back to kooky, unsafe.
For the earliest prototypes of c/r, I think saying that setuid an the
life of a container makes checkpoint impossible is the right thing to
do.
-serge
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, 2008-10-27 at 13:11 -0400, Oren Laadan wrote:
quoted
Dave Hansen wrote:
quoted
On Mon, 2008-10-27 at 07:03 -0400, Oren Laadan wrote:
quoted
quoted
In our implementation, we simply refused to checkpoint setid
programs.
True. And this works very well for HPC applications.
However, it doesn't work so well for server applications, for
instance.
Also, you could use file system snapshotting to ensure that the file
system view does not change, and still face the same issue.
So I'm perfectly ok with deferring this discussion to a later time :)
Oren, is this a good place to stick a process_deny_checkpoint()? Both
so we refuse to checkpoint, and document this as something that has to
be addressed later?
why refuse to checkpoint ?
If most setuid programs hold privileged resources for extended periods
of time after dropping privileges then it seems like a good idea to
refuse to checkpoint. Restart of those programs would be quite
unreliable unless/until we find a nice solution.
quoted
if I'm root, and I want to checkpoint, and later restart, my sshd server
(assuming we support listening sockets) - then why not ?
we can just let it be, and have the restart fail (if it isn't root that
does the restart); perhaps add something like warn_checkpoint() (similar
to deny, but only warns) ?
How will folks not specializing in checkpoint/restart know when to use
this as opposed to deny?
Instead, how about a flag to sys_checkpoint() -- DO_RISKY_CHECKPOINT --
which checkpoints despite !may_checkpoint?
I also agree with Matt - so we have a quorum :)
so just to clarify: sys_checkpoint() is to fail (with what error ?) if the
deny-checkpoint test fails.
however, if the user is risky, she can specify CR_CHECKPOINT_RISKY to force
an attempt to checkpoint as is.
does this sound right ?
Oren.
From: Dave Hansen <hidden> Date: 2008-10-27 22:10:08
On Mon, 2008-10-27 at 17:51 -0400, Oren Laadan wrote:
quoted
Instead, how about a flag to sys_checkpoint() -- DO_RISKY_CHECKPOINT --
which checkpoints despite !may_checkpoint?
I also agree with Matt - so we have a quorum :)
so just to clarify: sys_checkpoint() is to fail (with what error ?) if the
deny-checkpoint test fails.
however, if the user is risky, she can specify CR_CHECKPOINT_RISKY to force
an attempt to checkpoint as is.
This sounds like an awful lot of policy to determine *inside* the
kernel. Everybody is going to have a different definition of risky, so
this scheme will work for approximately 5 minutes until it gets
patched. :)
Is it possible to enhance our interface such that users might have some
kind of choice on these matters?
-- Dave
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
@@ -0,0 +1,374 @@++ === Checkpoint-Restart support in the Linux kernel ===++Copyright (C) 2008 Oren Laadan++Author: Oren Laadan <orenl-eQaUEPhvms7ENvBUuze7eA@public.gmane.org>++License: The GNU Free Documentation License, Version 1.2+ (dual licensed under the GPL v2)+Reviewers:++Application checkpoint/restart [CR] is the ability to save the state+of a running application so that it can later resume its execution+from the time at which it was checkpointed. An application can be+migrated by checkpointing it on one machine and restarting it on+another. CR can provide many potential benefits:++* Failure recovery: by rolling back an to a previous checkpoint
Extraneous word "an"?
+
+* Improved response time: by restarting applications from checkpoints
+ instead of from scratch.
+
+* Improved system utilization: by suspending long running CPU
+ intensive jobs and resuming them when load decreases.
+
+* Fault resilience: by migrating applications off of faulty hosts.
s/off of/off/
+
+* Dynamic load balancing: by migrating applications to less loaded
+ hosts.
+
+* Improved service availability and administration: by migrating
+ applications before host maintenance so that they continue to run
+ with minimal downtime
+
+* Time-travel: by taking periodic checkpoints and restarting from
+ any previous checkpoint.
+
+
+=== Overall design
+
+Checkpoint and restart is done in the kernel as much as possible. The
+kernel exports a relative opaque 'blob' of data to userspace which can
s/relative/relatively/
+then be handed to the new kernel at restore time. The 'blob' contains
+data and state of select portions of kernel structures such as VMAs
+and mm_structs, as well as copies of the actual memory that the tasks
+use. Any changes in this blob's format between kernel revisions can be
+handled by an in-userspace conversion program. The approach is similar
+to virtually all of the commercial CR products out there, as well as
+the research project Zap.
+
+Two new system calls are introduced to provide CR: sys_checkpoint and
+sys_restart. The checkpoint code basically serializes internal kernel
+state and writes it out to a file descriptor, and the resulting image
+is stream-able. More specifically, it consists of 5 steps:
+ 1. Pre-dump
+ 2. Freeze the container
+ 3. Dump
+ 4. Thaw (or kill) the container
+ 5. Post-dump
+Steps 1 and 5 are an optimization to reduce application downtime:
+"pre-dump" works before freezing the container, e.g. the pre-copy for
+live migration, and "post-dump" works after the container resumes
+execution, e.g. write-back the data to secondary storage.
+
+The restart code basically reads the saved kernel state and from a
Extraneous word "and"
+file descriptor, and re-creates the tasks and the resources they need
+to resume execution. The restart code is executed by each task that
+is restored in a new container to reconstruct its own state.
+
+
+=== Interfaces
+
+int sys_checkpoint(pid_t pid, int fd, unsigned long flag);
+ Checkpoint a container whose init task is identified by pid, to the
I seem to recall Andrew M. mentioning something about this. Could you
add a bit of text here to explain why "pid" is not always "1".
+ file designated by fd. Flags will have future meaning (should be 0
+ for now).
Should be 0, or must be 0? IMO, the text should be the latter. And
the code should check that -- does it?
+ Returns: a positive integer that identifies the checkpoint image
Can you add some more text here to describe the what this "positive
integer" is. E..g., how is it generated, and what does it refer to
(an address, a file descriptor, something else).
Looking further down, it seems that this is the "crid", but you could
make that clearer already here.
+ (for future reference in case it is kept in memory) upon success,
+ 0 if it returns from a restart, and -1 if an error occurs.
+
+int sys_restart(int crid, int fd, unsigned long flags);
+ Restart a container from a checkpoint image identified by crid, or
See above -- that "crid" is the thing returned by checkpoint(), right?
Make that clearer here.
+ from the blob stored in the file designated by fd. Flags will have
+ future meaning (should be 0 for now).
Again... Should be 0, or must be 0? IMO, the text should be the
latter. And the code should check that -- does it?
+ Returns: 0 on success and -1 if an error occurs.
+
+Thus, if checkpoint is initiated by a process in the container, one
+can use logic similar to fork():
+ ...
+ crid = checkpoint(...);
+ switch (crid) {
+ case -1:
+ perror("checkpoint failed");
+ break;
+ default:
+ fprintf(stderr, "checkpoint succeeded, CRID=%d\n", ret);
+ /* proceed with execution after checkpoint */
+ ...
+ break;
+ case 0:
+ fprintf(stderr, "returned after restart\n");
+ /* proceed with action required following a restart */
+ ...
+ break;
+ }
+ ...
+And to initiate a restart, the process in an empty container can use
+logic similar to execve():
+ ...
+ if (restart(crid, ...) < 0)
+ perror("restart failed");
+ /* only get here if restart failed */
+ ...
+
+See below a complete example in C.
+
+
+=== Order of state dump
+
+The order of operations, both save and restore, is as following:
s/is as following/is as follows/
+
+* Header section: header, container information, etc.
+* Global section: [TBD] global resources such as IPC, UTS, etc.
+* Process forest: [TBD] tasks and their relationships
+* Per task data (for each task):
+ -> task state: elements of task_struct
+ -> thread state: elements of thread_struct and thread_info
+ -> CPU state: registers etc, including FPU
+ -> memory state: memory address space layout and contents
+ -> filesystem state: [TBD] filesystem namespace state, chroot, cwd, etc
+ -> files state: open file descriptors and their state
+ -> signals state: [TBD] pending signals and signal handling state
+ -> credentials state: [TBD] user and group state, statistics
+
+
+=== Checkpoint image format
+
+The checkpoint image format is composed of records consistings of a
consisting
+pre-header that identifies its contents, followed by a payload. (The
+idea here is to enable parallel checkpointing in the future in which
+multiple threads interleave data from multiple processes into a single
+stream).
+
+The pre-header is defined by "struct cr_hdr" as follows:
+
+struct cr_hdr {
+ __s16 type;
+ __s16 len;
+ __u32 parent;
+};
+
+Here, 'type' field identifies the type of the payload, 'len' tells its
s/'type' field/'type'/
or
s/'type' field/the 'type' field/
+length in bytes. The 'parent' identifies the owner object instance. The
add "field" after 'parent'
+meaning of the 'parent field varies depending on the type. For example,
+for type CR_HDR_MM, the 'parent identifies the task to which this MM
Missing ' (single quote)
+belongs. The payload also varies depending on the type, for instance,
+the data describing a task_struct is given by a 'struct cr_hdr_task'
+(type CR_HDR_TASK) and so on.
+
+The format of the memory dump is as follows: for each VMA, there is a
+'struct cr_vma'; if the VMA is file-mapped, it is followed by the file
+name. Following comes the actual contents, in one or more chunk: each
s/Following comes/Following that are/
s/chunk/chunks/
+chunk begins with a header that specifies how many pages it holds,
+then a the virtual addresses of all the dumped pages in that chunk,
s/a the/the/
+followed by the actual contents of all the dumped pages. A header with
+zero number of pages marks the end of the contents for a particular
+VMA. Then comes the next VMA and so on.
+
+To illustrate this, consider a single simple task with two VMAs: one
+is file mapped with two dumped pages, and the other is anonymous with
+three dumped pages. The checkpoint image will look like this:
+
+cr_hdr + cr_hdr_head
+cr_hdr + cr_hdr_task
+ cr_hdr + cr_hdr_mm
+ cr_hdr + cr_hdr_vma + cr_hdr + string
+ cr_hdr_pgarr (nr_pages = 2)
+ addr1, addr2
+ page1, page2
+ cr_hdr_pgarr (nr_pages = 0)
+ cr_hdr + cr_hdr_vma
+ cr_hdr_pgarr (nr_pages = 3)
+ addr3, addr4, addr5
+ page3, page4, page5
+ cr_hdr_pgarr (nr_pages = 0)
+ cr_hdr + cr_mm_context
+ cr_hdr + cr_hdr_thread
+ cr_hdr + cr_hdr_cpu
+cr_hdr + cr_hdr_tail
+
+
+=== Current Implementation
+
+[2008-Oct-07]
+There are several assumptions in the current implementation; they will
+be gradually relaxed in future versions. The main ones are:
+* A task can only checkpoint itself (missing "restart-block" logic).
+* Namespaces are not saved or restored; They will be treated as a type
+ of shared object.
+* In particular, it is assumed that the task's file system namespace
+ is the "root" for the entire container.
+* It is assumed that the same file system view is available for the
+ restart task(s). Otherwise, a file system snapshot is required.
+
+
+=== Sample code
+
+Two example programs: one uses checkpoint (called ckpt) to checkpoint
+itself, and another uses restart (called rstr) to restart from that
+checkpoint. Note the use of "dup2" to create a copy of an open file
+and show how shared objects are treated. Execute like this:
+
+orenl:~/test$ ./ckpt > out.1
+ <-- ctrl-c
From: Serge E. Hallyn <hidden> Date: 2008-10-28 18:43:44
Quoting Dave Hansen (dave-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org):
On Mon, 2008-10-27 at 17:51 -0400, Oren Laadan wrote:
quoted
quoted
Instead, how about a flag to sys_checkpoint() -- DO_RISKY_CHECKPOINT --
which checkpoints despite !may_checkpoint?
I also agree with Matt - so we have a quorum :)
so just to clarify: sys_checkpoint() is to fail (with what error ?) if the
deny-checkpoint test fails.
however, if the user is risky, she can specify CR_CHECKPOINT_RISKY to force
an attempt to checkpoint as is.
This sounds like an awful lot of policy to determine *inside* the
kernel. Everybody is going to have a different definition of risky, so
this scheme will work for approximately 5 minutes until it gets
patched. :)
Is it possible to enhance our interface such that users might have some
kind of choice on these matters?
Well we could always just add a field to /proc/self/status, and let
userspace check that field (after freezing the task) for the
presence of CR_CHECKPOINT_RISKY and make up its own mind.
Though my preference is for simplicity - just refuse the checkpoint.
That way people might screan loudly enough for us to support the
features they want. If we let them just bypass and hope for the
best that starts to dilute some of the intended effect of all this.
-serge
--
To unsubscribe from this list: send the line "unsubscribe linux-api" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at http://vger.kernel.org/majordomo-info.html