From: Eric W. Biederman <hidden> Date: 2018-09-26 17:36:56
Andrey Vagin [off-list ref] writes:
On Tue, Sep 25, 2018 at 12:02:32AM +0200, Eric W. Biederman wrote:
quoted
Andrey Vagin [off-list ref] writes:
quoted
On Fri, Sep 21, 2018 at 02:27:29PM +0200, Eric W. Biederman wrote:
quoted
Dmitry Safonov [off-list ref] writes:
quoted
Discussions around time virtualization are there for a long time.
The first attempt to implement time namespace was in 2006 by Jeff Dike.
From that time, the topic appears on and off in various discussions.
There are two main use cases for time namespaces:
1. change date and time inside a container;
2. adjust clocks for a container restored from a checkpoint.
“It seems like this might be one of the last major obstacles keeping
migration from being used in production systems, given that not all
containers and connections can be migrated as long as a time dependency
is capable of messing it up.” (by github.com/dav-ell)
The kernel provides access to several clocks: CLOCK_REALTIME,
CLOCK_MONOTONIC, CLOCK_BOOTTIME. Last two clocks are monotonous, but the
start points for them are not defined and are different for each running
system. When a container is migrated from one node to another, all
clocks have to be restored into consistent states; in other words, they
have to continue running from the same points where they have been
dumped.
The main idea behind this patch set is adding per-namespace offsets for
system clocks. When a process in a non-root time namespace requests
time of a clock, a namespace offset is added to the current value of
this clock on a host and the sum is returned.
All offsets are placed on a separate page, this allows up to map it as
part of vvar into user processes and use offsets from vdso calls.
Now offsets are implemented for CLOCK_MONOTONIC and CLOCK_BOOTTIME
clocks.
Questions to discuss:
* Clone flags exhaustion. Currently there is only one unused clone flag
bit left, and it may be worth to use it to extend arguments of the clone
system call.
* Realtime clock implementation details:
Is having a simple offset enough?
What to do when date and time is changed on the host?
Is there a need to adjust vfs modification and creation times?
Implementation for adjtime() syscall.
Overall I support this effort. In my quick skim this code looked good.
Hi Eric,
Thank you for the feedback.
quoted
My feeling is that we need to be able to support running ntpd and
support one namespace doing googles smoothing of leap seconds while
another namespace takes the leap second.
What I was imagining when I was last thinking about this was one
instance of struct timekeeper aka tk_core per time namespace. That
structure already keeps offsets for all of the various clocks from
the kerne internal time sources. What would be needed would be to
pass in an appropriate time namespace pointer.
I could be completely wrong as I have not take the time to completely
trace through the code. Have you looked at pushing the time namespace
down as far as tk_core?
What I think would be the big advantage (besides ntp working) is that
the bulk of the code could be reused. Allowing testing of the kernel's
time code by setting up a new time namespace. So a person in production
could setup a time namespace with the time set ahead a little bit and
be able to verify that the kernel handles the upcoming leap second
properly.
It is an interesting idea, but I have a few questions:
1. Does it mean that timekeeping_update() will be called for each
namespace? This functions is called periodically, it updates times on the
timekeeper structure, updates vsyscall_gtod_data, etc. What will be an
overhead of this?
I don't know if periodically is a proper characterization. There may be
a code path that does that. But from what I can see timekeeping_update
is the guts of settimeofday (and a few related functions).
So it appears to make sense for timekeeping_update to be per namespace.
Hmm. Looking at what is updated in the vsyscall_gtod_data it does
look like you would have to periodically update things, but I don't know
big that period would be. As long as the period is reasonably large,
or the time namespaces were sufficiently deschronized it should not
be a problem. But that is the class of problem that could make
my ideal impractical if there is measuarable overhead.
Where were you seeing timekeeping_update being called periodically?
Interesting.
Reading the code the calling sequence there is:
tick_sched_do_timer
tick_do_update_jiffies64
update_wall_time
timekeeping_advance
timekeepging_update
If I read that properly under the right nohz circumstances that update
can be delayed indefinitely.
So I think we could prototype a time namespace that was per
timekeeping_update and just had update_wall_time iterate through
all of the time namespaces.
I don't think the naive version would scale to very many time
namespaces.
At the same time using the techniques from the nohz work and a little
smarts I expect we could get the code to scale.
I think this direction is definitely worth exploring. My experience
with namespaces is that if we don't get the advanced features working
there is little to no interest from the core developers of the code,
and the namespaces don't solve additional problems. Which makes the
namespace a hard sell. Especially when it does not solve problems the
developers of the subsystem have.
The advantage of timekeeping_update per time namespace is that it allows
different lengths of seconds per time namespace. Which allows testing
ntp and the kernel in interesting ways while still having a working
production configuration on the same system.
Eric
2018-09-26 18:36 GMT+01:00 Eric W. Biederman [off-list ref]:
The advantage of timekeeping_update per time namespace is that it allows
different lengths of seconds per time namespace. Which allows testing
ntp and the kernel in interesting ways while still having a working
production configuration on the same system.
From: Thomas Gleixner <hidden> Date: 2018-09-27 21:30:47
On Wed, 26 Sep 2018, Eric W. Biederman wrote:
Reading the code the calling sequence there is:
tick_sched_do_timer
tick_do_update_jiffies64
update_wall_time
timekeeping_advance
timekeepging_update
If I read that properly under the right nohz circumstances that update
can be delayed indefinitely.
So I think we could prototype a time namespace that was per
timekeeping_update and just had update_wall_time iterate through
all of the time namespaces.
Please don't go there. timekeeping_update() is already heavy and walking
through a gazillion of namespaces will just make it horrible,
I don't think the naive version would scale to very many time
namespaces.
:)
At the same time using the techniques from the nohz work and a little
smarts I expect we could get the code to scale.
You'd need to invoke the update when the namespace is switched in and
hasn't been updated since the last tick happened. That might be doable, but
you also need to take the wraparound constraints of the underlying
clocksources into account, which again can cause walking all name spaces
when they are all idle long enough.
From there it becomes hairy, because it's not only timekeeping,
i.e. reading time, this is also affecting all timers which are armed from a
namespace.
That gets really ugly because when you do settimeofday() or adjtimex() for
a particular namespace, then you have to search for all armed timers of
that namespace and adjust them.
The original posix timer code had the same issue because it mapped the
clock realtime timers to the timer wheel so any setting of the clock caused
a full walk of all armed timers, disarming, adjusting and requeing
them. That's horrible not only performance wise, it's also a locking
nightmare of all sorts.
Add time skew via NTP/PTP into the picture and you might have to adjust
timers as well, because you need to guarantee that they are not expiring
early.
I haven't looked through Dimitry's patches yet, but I don't see how this
can work at all without introducing subtle issues all over the place.
Thanks,
tglx
From: Thomas Gleixner <hidden> Date: 2018-09-27 21:42:21
On Thu, 27 Sep 2018, Thomas Gleixner wrote:
Add time skew via NTP/PTP into the picture and you might have to adjust
timers as well, because you need to guarantee that they are not expiring
early.
I haven't looked through Dimitry's patches yet, but I don't see how this
can work at all without introducing subtle issues all over the place.
And just a quick scan tells me that this is broken. Timers will expire
early or late. The latter is acceptible to some extent, but larger delays
might come with surprise. Expiring early is an absolute nono.
Thanks,
tglx
On Thu, Sep 27, 2018 at 11:41:49PM +0200, Thomas Gleixner wrote:
On Thu, 27 Sep 2018, Thomas Gleixner wrote:
quoted
Add time skew via NTP/PTP into the picture and you might have to adjust
timers as well, because you need to guarantee that they are not expiring
early.
I haven't looked through Dimitry's patches yet, but I don't see how this
can work at all without introducing subtle issues all over the place.
And just a quick scan tells me that this is broken. Timers will expire
early or late. The latter is acceptible to some extent, but larger delays
might come with surprise. Expiring early is an absolute nono.
Do you mean that we have to adjust all timers after changing offset for
CLOCK_MONOTONIC or CLOCK_BOOTTIME? Our idea is that offsets for
monotonic and boot times will be set immediately after creating a time
namespace before using any timers.
It is interesting to think what a use-case for changing these offsets
after creating timers. It may be useful for testing needs. A user sets a
timer in an hour and then change a clock offset forward and check that a
test application handles the timer properly.
From: Thomas Gleixner <hidden> Date: 2018-10-02 06:15:51
On Mon, 1 Oct 2018, Andrey Vagin wrote:
On Thu, Sep 27, 2018 at 11:41:49PM +0200, Thomas Gleixner wrote:
quoted
On Thu, 27 Sep 2018, Thomas Gleixner wrote:
quoted
Add time skew via NTP/PTP into the picture and you might have to adjust
timers as well, because you need to guarantee that they are not expiring
early.
I haven't looked through Dimitry's patches yet, but I don't see how this
can work at all without introducing subtle issues all over the place.
And just a quick scan tells me that this is broken. Timers will expire
early or late. The latter is acceptible to some extent, but larger delays
might come with surprise. Expiring early is an absolute nono.
Do you mean that we have to adjust all timers after changing offset for
CLOCK_MONOTONIC or CLOCK_BOOTTIME? Our idea is that offsets for
monotonic and boot times will be set immediately after creating a time
namespace before using any timers.
I explained that in detail in this thread, but it's not about the initial
setting of clock mono/boot before any timers have been armed.
It's about setting the offset or clock realtime (via settimeofday) when
timers are already armed. Also having a entirely different time domain,
e.g. separate NTP adjustments, makes that necessary.
Thanks,
tglx
Hi Thomas, Andrei, Eric,
On Tue, 2 Oct 2018 at 07:15, Thomas Gleixner [off-list ref] wrote:
On Mon, 1 Oct 2018, Andrey Vagin wrote:
quoted
On Thu, Sep 27, 2018 at 11:41:49PM +0200, Thomas Gleixner wrote:
quoted
On Thu, 27 Sep 2018, Thomas Gleixner wrote:
quoted
Add time skew via NTP/PTP into the picture and you might have to adjust
timers as well, because you need to guarantee that they are not expiring
early.
I haven't looked through Dimitry's patches yet, but I don't see how this
can work at all without introducing subtle issues all over the place.
And just a quick scan tells me that this is broken. Timers will expire
early or late. The latter is acceptible to some extent, but larger delays
might come with surprise. Expiring early is an absolute nono.
Do you mean that we have to adjust all timers after changing offset for
CLOCK_MONOTONIC or CLOCK_BOOTTIME? Our idea is that offsets for
monotonic and boot times will be set immediately after creating a time
namespace before using any timers.
I explained that in detail in this thread, but it's not about the initial
setting of clock mono/boot before any timers have been armed.
It's about setting the offset or clock realtime (via settimeofday) when
timers are already armed. Also having a entirely different time domain,
e.g. separate NTP adjustments, makes that necessary.
It looks like, there is a bit of misunderstanding each other:
Andrei was talking about the current RFC version, where we haven't
introduced offsets for clock realtime. While Thomas IIUC, is looking
how-to expand time namespace over realtime.
As CLOCK_REALTIME virtualization raises so many complex questions
like a different length of the second or list of realtime timers in ns we
haven't added any realization for it.
It seems like an initial introduction for timens can be expanded after to cover
realtime clocks too. While it may seem incomplete, it solves issues for
restoring/migration of real-world applications like nodejs, Oracle DB server
which fails after being restored if there is a leap in monotonic time.
While solving the mentioned issues, it doesn't bring overhead.
(well, Andy noted that cmp for zero-offsets on vdso can be optimized too,
which will be done in v1).
Thomas, thanks much for your input - now we know that we'll need to
introduce list for timers in namespace when we'll add realtime clocks.
Do you believe that CLOCK_MONOTONIC_SYNC would be an easier
concept than offsets per-namespace?
Thanks,
Dmitry
From: Thomas Gleixner <hidden> Date: 2018-10-02 21:27:07
Dmitry,
On Tue, 2 Oct 2018, Dmitry Safonov wrote:
On Tue, 2 Oct 2018 at 07:15, Thomas Gleixner [off-list ref] wrote:
quoted
I explained that in detail in this thread, but it's not about the initial
setting of clock mono/boot before any timers have been armed.
It's about setting the offset or clock realtime (via settimeofday) when
timers are already armed. Also having a entirely different time domain,
e.g. separate NTP adjustments, makes that necessary.
It looks like, there is a bit of misunderstanding each other:
Andrei was talking about the current RFC version, where we haven't
introduced offsets for clock realtime. While Thomas IIUC, is looking
how-to expand time namespace over realtime.
As CLOCK_REALTIME virtualization raises so many complex questions
like a different length of the second or list of realtime timers in ns we
haven't added any realization for it.
It seems like an initial introduction for timens can be expanded after to cover
realtime clocks too. While it may seem incomplete, it solves issues for
restoring/migration of real-world applications like nodejs, Oracle DB server
which fails after being restored if there is a leap in monotonic time.
Well, yes. But you really have to think about the full picture. Just adding
part of the overall solution right now, just because it can be glued into
the code easily, is not the best approach IMO as it might result in
substantial rework of the whole thing sooner than later. I really don't
want to end up with something which is not extensible and has to be
supported forever.
Just for the record, the current approach with name space offsets for
monotonic is also prone to malfunction vs. timers, unless you can prevent
changing the offset _after_ the namespace has been set up and timers have
been armed. I admit, that I did not look close enough to verify that.
While solving the mentioned issues, it doesn't bring overhead.
(well, Andy noted that cmp for zero-offsets on vdso can be optimized too,
which will be done in v1).
Thomas, thanks much for your input - now we know that we'll need to
introduce list for timers in namespace when we'll add realtime clocks.
Do you believe that CLOCK_MONOTONIC_SYNC would be an easier
concept than offsets per-namespace?
Haven't thought it through. This was just an idea in reaction to Eric's
question whether setting clock monotonic might be feasible. But yes, it
might be worth to think about it.
I think you should really define the long term requirements for time
namespaces and perhaps set some limitations in functionality upfront.
Thanks,
tglx