From: Jakub Drnec <hidden> Date: 2019-02-26 20:36:29
Hi all,
I think I observed a potential problem, is this the correct place to report it? (CC me, not on list)
[1.] One line summary: monotonic clock can be made to decrease on ppc64
[2.] Full description:
Setting the realtime clock can sometimes make the monotonic clock go back by over a hundred years.
Decreasing the realtime clock across the y2k38 threshold is one reliable way to reproduce.
Allegedly this can also happen just by running ntpd, I have not managed to reproduce that other
than booting with rtc at >2038 and then running ntp.
When this happens, anything with timers (e.g. openjdk) breaks rather badly.
[3.] Keywords: gettimeofday, ppc64, vdso
[4.] Kernel information
[4.1.] Kernel version: any (tested on 4.19)
[4.2.] Kernel .config file: any
[5.] Most recent kernel version which did not have the bug: not a regression
[6.] Output of Oops..: not applicable
[7.] Example program which triggers the problem
--- testcase.c
#include <stdio.h>
#include <time.h>
#include <stdlib.h>
#include <unistd.h>
long get_time() {
struct timespec tp;
if (clock_gettime(CLOCK_MONOTONIC, &tp) != 0) {
perror("clock_gettime failed");
exit(1);
}
long result = tp.tv_sec + tp.tv_nsec / 1000000000;
return result;
}
int main() {
printf("monitoring monotonic clock...\n");
long last = get_time();
while(1) {
long now = get_time();
if (now < last) {
printf("clock went backwards by %ld seconds!\n",
last - now);
}
last = now;
sleep(1);
}
return 0;
}
---
when running
# date -s 2040-1-1
# date -s 2037-1-1
program outputs: clock went backwards by 4294967295 seconds!
[8.] Environment: any ppc64, currently reproducing on qemu-system-ppc64le running debian unstable
[X.] Other notes, patches, fixes, workarounds:
The problem seems to be in vDSO code in arch/powerpc/kernel/vdso64/gettimeofday.S.
(possibly because some values used in the calculation are only 32 bit?)
Slightly silly workaround:
nuke the "cmpwi cr1,r3,CLOCK_MONOTONIC" in __kernel_clock_gettime
Now it always goes through the syscall fallback which does not have the same problem.
Regards,
Jakub Drnec
From: Daniel Axtens <hidden> Date: 2019-02-28 06:50:09
Hi,
I think I observed a potential problem, is this the correct place to report it? (CC me, not on list)
Yes
[1.] One line summary: monotonic clock can be made to decrease on ppc64
[2.] Full description:
Setting the realtime clock can sometimes make the monotonic clock go back by over a hundred years.
Decreasing the realtime clock across the y2k38 threshold is one reliable way to reproduce.
Allegedly this can also happen just by running ntpd, I have not managed to reproduce that other
than booting with rtc at >2038 and then running ntp.
When this happens, anything with timers (e.g. openjdk) breaks rather badly.
I can confirm this on a physical 64bit book3e system and powerpc/next
with the provided reproducer.
[X.] Other notes, patches, fixes, workarounds:
The problem seems to be in vDSO code in arch/powerpc/kernel/vdso64/gettimeofday.S.
(possibly because some values used in the calculation are only 32 bit?)
Slightly silly workaround:
nuke the "cmpwi cr1,r3,CLOCK_MONOTONIC" in __kernel_clock_gettime
Now it always goes through the syscall fallback which does not have the same problem.
I'll look at the asm but I'm no expert here so we might need to defer to
someone with real expertise. Falling back to the syscall sadly ruins
the performance gain of having a vDSO.
Regards,
Daniel
On Tue, Feb 26, 2019 at 9:36 PM Jakub Drnec [off-list ref] wrote:
Hi all,
I think I observed a potential problem, is this the correct place to report it? (CC me, not on list)
[1.] One line summary: monotonic clock can be made to decrease on ppc64
[2.] Full description:
Setting the realtime clock can sometimes make the monotonic clock go back by over a hundred years.
Decreasing the realtime clock across the y2k38 threshold is one reliable way to reproduce.
Allegedly this can also happen just by running ntpd, I have not managed to reproduce that other
than booting with rtc at >2038 and then running ntp.
Isn't it the expected behavior. Here is what I see for powermac:
$ git show 22db552b50fa
...
This changes the logic to cast to an unsigned 32-bit number first for
the Macintosh time and then convert that to the Unix time, which then
gives us a time in the documented 1904..2040 year range. I decided not
to use the longer 1970..2106 range that other drivers use, for
consistency with the literal interpretation of the register, but that
could be easily changed if we decide we want to support any Mac after
2040.
...
quoted hunk
When this happens, anything with timers (e.g. openjdk) breaks rather badly.
[3.] Keywords: gettimeofday, ppc64, vdso
[4.] Kernel information
[4.1.] Kernel version: any (tested on 4.19)
[4.2.] Kernel .config file: any
[5.] Most recent kernel version which did not have the bug: not a regression
[6.] Output of Oops..: not applicable
[7.] Example program which triggers the problem
--- testcase.c
#include <stdio.h>
#include <time.h>
#include <stdlib.h>
#include <unistd.h>
long get_time() {
struct timespec tp;
if (clock_gettime(CLOCK_MONOTONIC, &tp) != 0) {
perror("clock_gettime failed");
exit(1);
}
long result = tp.tv_sec + tp.tv_nsec / 1000000000;
return result;
}
int main() {
printf("monitoring monotonic clock...\n");
long last = get_time();
while(1) {
long now = get_time();
if (now < last) {
printf("clock went backwards by %ld seconds!\n",
last - now);
}
last = now;
sleep(1);
}
return 0;
}
---
when running
# date -s 2040-1-1
# date -s 2037-1-1
program outputs: clock went backwards by 4294967295 seconds!
[8.] Environment: any ppc64, currently reproducing on qemu-system-ppc64le running debian unstable
[X.] Other notes, patches, fixes, workarounds:
The problem seems to be in vDSO code in arch/powerpc/kernel/vdso64/gettimeofday.S.
(possibly because some values used in the calculation are only 32 bit?)
Slightly silly workaround:
nuke the "cmpwi cr1,r3,CLOCK_MONOTONIC" in __kernel_clock_gettime
Now it always goes through the syscall fallback which does not have the same problem.
Regards,
Jakub Drnec
From: Daniel Axtens <hidden> Date: 2019-02-28 11:19:58
Mathieu Malaterre [off-list ref] writes:
On Tue, Feb 26, 2019 at 9:36 PM Jakub Drnec [off-list ref] wrote:
quoted
Hi all,
I think I observed a potential problem, is this the correct place to report it? (CC me, not on list)
[1.] One line summary: monotonic clock can be made to decrease on ppc64
[2.] Full description:
Setting the realtime clock can sometimes make the monotonic clock go back by over a hundred years.
Decreasing the realtime clock across the y2k38 threshold is one reliable way to reproduce.
Allegedly this can also happen just by running ntpd, I have not managed to reproduce that other
than booting with rtc at >2038 and then running ntp.
Isn't it the expected behavior. Here is what I see for powermac:
$ git show 22db552b50fa
...
This changes the logic to cast to an unsigned 32-bit number first for
the Macintosh time and then convert that to the Unix time, which then
gives us a time in the documented 1904..2040 year range. I decided not
to use the longer 1970..2106 range that other drivers use, for
consistency with the literal interpretation of the register, but that
could be easily changed if we decide we want to support any Mac after
2040.
...
My interpretation of that commit is that it relates to the kernel
reading the hardware RTC on a powermac, but this issue relates to
userspace fetching the time from the vDSO. I'm also not running on a
powermac, so my hardware should be able to deal with times > 2040...
Regards,
Daniel
quoted
When this happens, anything with timers (e.g. openjdk) breaks rather badly.
[3.] Keywords: gettimeofday, ppc64, vdso
[4.] Kernel information
[4.1.] Kernel version: any (tested on 4.19)
[4.2.] Kernel .config file: any
[5.] Most recent kernel version which did not have the bug: not a regression
[6.] Output of Oops..: not applicable
[7.] Example program which triggers the problem
--- testcase.c
#include <stdio.h>
#include <time.h>
#include <stdlib.h>
#include <unistd.h>
long get_time() {
struct timespec tp;
if (clock_gettime(CLOCK_MONOTONIC, &tp) != 0) {
perror("clock_gettime failed");
exit(1);
}
long result = tp.tv_sec + tp.tv_nsec / 1000000000;
return result;
}
int main() {
printf("monitoring monotonic clock...\n");
long last = get_time();
while(1) {
long now = get_time();
if (now < last) {
printf("clock went backwards by %ld seconds!\n",
last - now);
}
last = now;
sleep(1);
}
return 0;
}
---
when running
# date -s 2040-1-1
# date -s 2037-1-1
program outputs: clock went backwards by 4294967295 seconds!
[8.] Environment: any ppc64, currently reproducing on qemu-system-ppc64le running debian unstable
[X.] Other notes, patches, fixes, workarounds:
The problem seems to be in vDSO code in arch/powerpc/kernel/vdso64/gettimeofday.S.
(possibly because some values used in the calculation are only 32 bit?)
Slightly silly workaround:
nuke the "cmpwi cr1,r3,CLOCK_MONOTONIC" in __kernel_clock_gettime
Now it always goes through the syscall fallback which does not have the same problem.
Regards,
Jakub Drnec
From: Jakub Drnec <hidden> Date: 2019-03-01 13:35:32
Hi,
quoted
Decreasing the realtime clock across the y2k38 threshold is one reliable way to reproduce.
Allegedly this can also happen just by running ntpd, I have not managed to reproduce that other
than booting with rtc at >2038 and then running ntp.
Isn't it the expected behavior. Here is what I see for powermac:
... gives us a time in the documented 1904..2040 year range. I decided not
This (or some related issue involving monotonic time going wonky) surfaced on a system running 4.5 kernel with sane clock values running ntpd. It may not be limited to setting the clock beyond 2038, that is just a convenient way to reproduce something visibly wrong.
If the "ntpd-related monotonic" issue is separate from what I reported, I did not find anything in the commit history that looks likely to have fixed that, so I think that is still a thing.
Another way of looking at the issue is that when you set the clock beyond 2038 (or the mystery ntpd thing happens), vdso and syscall versions of clock_gettime start reporting wildly different values for CLOCK_MONOTONIC. If someone manages to somehow interlace calls to those two assuming they are consistent, crazy bugs will ensue...
Regards,
Jakub
Hi all,
I think I observed a potential problem, is this the correct place to report it? (CC me, not on list)
[1.] One line summary: monotonic clock can be made to decrease on ppc64
[2.] Full description:
Setting the realtime clock can sometimes make the monotonic clock go back by over a hundred years.
Decreasing the realtime clock across the y2k38 threshold is one reliable way to reproduce.
Allegedly this can also happen just by running ntpd, I have not managed to reproduce that other
than booting with rtc at >2038 and then running ntp.
When this happens, anything with timers (e.g. openjdk) breaks rather badly.
Thanks for the report.
The problem seems to be in vDSO code in arch/powerpc/kernel/vdso64/gettimeofday.S.
You're right, the wall-to-monotonic offset (wtom_clock_sec) is a signed
32-bit value, so that seems like it's going to have problems.
If I do `date -s 2037-1-1` I see:
[ 26.024061] update_vsyscall: tk->wall_to_monotonic.tv_sec -2114341175
[ 26.042633] update_vsyscall: vdso_data->wtom_clock_sec -2114341175
Which looks sane.
But then 2040-1-1 shows:
[ 32.617020] update_vsyscall: tk->wall_to_monotonic.tv_sec -2208949168
[ 32.632642] update_vsyscall: vdso_data->wtom_clock_sec 2086018128
ie. the larger negative offset has overflowed and become positive.
But then when we go back to 2037 we get a negative offset again and
monotonic time appears to go backward and things are unhappy.
I don't know this code well, but the patch below *appears* to work. I'll
have a closer look on Monday.
cheers
[3.] Keywords: gettimeofday, ppc64, vdso
[4.] Kernel information
[4.1.] Kernel version: any (tested on 4.19)
[4.2.] Kernel .config file: any
[5.] Most recent kernel version which did not have the bug: not a regression
[6.] Output of Oops..: not applicable
[7.] Example program which triggers the problem
--- testcase.c
#include <stdio.h>
#include <time.h>
#include <stdlib.h>
#include <unistd.h>
long get_time() {
struct timespec tp;
if (clock_gettime(CLOCK_MONOTONIC, &tp) != 0) {
perror("clock_gettime failed");
exit(1);
}
long result = tp.tv_sec + tp.tv_nsec / 1000000000;
return result;
}
int main() {
printf("monitoring monotonic clock...\n");
long last = get_time();
while(1) {
long now = get_time();
if (now < last) {
printf("clock went backwards by %ld seconds!\n",
last - now);
}
last = now;
sleep(1);
}
return 0;
}
---
when running
# date -s 2040-1-1
# date -s 2037-1-1
program outputs: clock went backwards by 4294967295 seconds!
[8.] Environment: any ppc64, currently reproducing on qemu-system-ppc64le running debian unstable
[X.] Other notes, patches, fixes, workarounds:
The problem seems to be in vDSO code in arch/powerpc/kernel/vdso64/gettimeofday.S.
(possibly because some values used in the calculation are only 32 bit?)
Slightly silly workaround:
nuke the "cmpwi cr1,r3,CLOCK_MONOTONIC" in __kernel_clock_gettime
Now it always goes through the syscall fallback which does not have the same problem.
Regards,
Jakub Drnec
From: Thomas Gleixner <hidden> Date: 2019-03-02 00:41:17
On Sat, 2 Mar 2019, Michael Ellerman wrote:
quoted
The problem seems to be in vDSO code in arch/powerpc/kernel/vdso64/gettimeofday.S.
You're right, the wall-to-monotonic offset (wtom_clock_sec) is a signed
32-bit value, so that seems like it's going to have problems.
If I do `date -s 2037-1-1` I see:
[ 26.024061] update_vsyscall: tk->wall_to_monotonic.tv_sec -2114341175
[ 26.042633] update_vsyscall: vdso_data->wtom_clock_sec -2114341175
Which looks sane.
But then 2040-1-1 shows:
[ 32.617020] update_vsyscall: tk->wall_to_monotonic.tv_sec -2208949168
[ 32.632642] update_vsyscall: vdso_data->wtom_clock_sec 2086018128
ie. the larger negative offset has overflowed and become positive.
But then when we go back to 2037 we get a negative offset again and
monotonic time appears to go backward and things are unhappy.
I don't know this code well, but the patch below *appears* to work. I'll
have a closer look on Monday.
Looks about right. The core hands in the 64bit offset already.
Thanks,
tglx