Marcelo,

On Fri, Oct 01 2021 at 09:05, Marcelo Tosatti wrote:

On Fri, Oct 01, 2021 at 01:02:23AM +0200, Thomas Gleixner wrote:

quoted

But even if that would be the case, then what prevents the stale time
stamps to be visible? Nothing:

T0:    t = now();
         -> pause
         -> resume
         -> magic "fixup"
T1:    dostuff(t);

Yes.

BTW, you could have a userspace notification (then applications 
could handle this if desired).

Well, we have that via timerfd with TFD_TIMER_CANCEL_ON_SET for
CLOCK_REALTIME. That's what applications which are sensitive to clock
REALTIME jumps use today.

quoted

  Now the proposed change is creating exactly the same problem:

  >> > +	if (data.flags & KVM_CLOCK_REALTIME) {
  >> > +		u64 now_real_ns = ktime_get_real_ns();
  >> > +
  >> > +		/*
  >> > +		 * Avoid stepping the kvmclock backwards.
  >> > +		 */
  >> > +		if (now_real_ns > data.realtime)
  >> > +			data.clock += now_real_ns - data.realtime;
  >> > +	}

  IOW, it takes the time between pause and resume into account and
  forwards the underlying base clock which makes CLOCK_MONOTONIC
  jump forward by exactly that amount of time.

Well, it is assuming that the

 T0:    t = now();
 T1:    pause vm()
 T2:	finish vm migration()
 T3:    dostuff(t);

Interval between T1 and T2 is small (and that the guest
clocks are synchronized up to a given boundary).

Yes, I understand that, but it's an assumption and there is no boundary
for the time jump in the proposed patches, which rings my alarm bells :)

But i suppose adding a limit to the forward clock advance 
(in the migration case) is useful:

	1) If migration (well actually, only the final steps
	   to finish migration, the time between when guest is paused
	   on source and is resumed on destination) takes too long,
	   then too bad: fix it to be shorter if you want the clocks
	   to have close to zero change to realtime on migration.

	2) Avoid the other bugs in case of large forward advance.

Maybe having it configurable, with a say, 1 minute maximum by default
is a good choice?

Don't know what 1 minute does in terms of applications etc. You have to
do some experiments on that.

An alternative would be to advance only the guests REALTIME clock, from 
data about how long steps T1-T2 took.

Yes, that's what would happen in the cooperative S2IDLE or S3 case when
the guest resumes.

quoted

So now virt came along and created a hard to solve circular dependency
problem:

   - If CLOCK_MONOTONIC stops for too long then NTP/PTP gets out of
     sync, but everything else is happy.
     
   - If CLOCK_MONOTONIC jumps too far forward, then all hell breaks
     lose, but NTP/PTP is happy.

One must handle the

 T0:    t = now();
          -> pause
          -> resume
          -> magic "fixup"
 T1:    dostuff(t);

fact if one is going to use savevm/restorevm anyway, so...
(it is kind of unfixable, unless you modify your application
to accept notifications to redo any computation based on t, isnt it?).

Well yes, but what applications can deal with is CLOCK_REALTIME jumping
because that's a property of it. Not so much the CLOCK_MONOTONIC part.

quoted

If you decide that correctness is overrated, then please document it
clearly instead of trying to pretend being correct.

Based on the above, advancing only CLOCK_REALTIME (and not CLOCK_MONOTONIC)
would be correct, right? And its probably not very hard to do.

Time _is_ hard to get right. 

So you might experiment with something like this as a stop gap:

  Provide the guest something like this:

          u64		   migration_seq;
          u64      	   realtime_delta_ns;

  in the shared clock page

  Do not forward jump clock MONOTONIC.

  On resume kick an IPI where the guest handler does:

         if (clock_data->migration_seq == migration_seq)
         	return;

         migration_seq = clock_data->migration_seq;

         ts64 = { 0, 0 };
         timespec64_add_ns(&ts64, clock_data->realtime_delta_ns);
         timekeeping_inject_sleeptime64(&ts64);

  Make sure that the IPI completes before you migrate the guest another
  time or implement it slightly smarter, but you get the idea :)

That's what we use for suspend time injection, but it should just work
without frozen tasks as well. It will forward clock REALTIME by the
amount of time spent during migration. It'll also modify the BOOTTIME
offset by the same amount, but that's not a tragedy.

The function will also reset NTP state so the NTP/PTP daemon knows that
there was a kernel initiated time jump and it can work out easily what
to do like it does on resume from an actual suspend. It will also
invoke clock_was_set() which makes all the other time related updates
trigger and wakeup tasks which have a timerfd with
TFD_TIMER_CANCEL_ON_SET armed.

This will obviously not work when the guest is in S2IDLE or S3, but for
initial experimentation you can ignore that and just avoid to do that in
the guest. :)

That still is worse than a cooperative S2IDLE/S3, but it's way more
sensible than the other two evils you have today.

Thanks very much for the detailed information! Its a good basis
for the document you ask.

I volunteer to review that documentation once it materializes :)

Thanks,

        tglx

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel