Re: clock_gettime64 vdso bug on 32-bit arm, rpi-4

On Wed, May 20, 2020 at 04:41:29PM +0100, Szabolcs Nagy wrote:

The 05/19/2020 22:31, Arnd Bergmann wrote:

quoted

On Tue, May 19, 2020 at 10:24 PM Adhemerval Zanella
[off-list ref] wrote:

quoted

On 19/05/2020 16:54, Arnd Bergmann wrote:

quoted

Jack Schmidt reported a bug for the arm32 clock_gettimeofday64 vdso call last
month: https://github.com/richfelker/musl-cross-make/issues/96 and
https://github.com/raspberrypi/linux/issues/3579

As Will Deacon pointed out, this was never reported on the mailing list,
so I'll try to summarize what we know, so this can hopefully be resolved soon.

- This happened reproducibly on Linux-5.6 on a 32-bit Raspberry Pi patched
   kernel running on a 64-bit Raspberry Pi 4b (bcm2711) when calling
   clock_gettime64(CLOCK_REALTIME)

Does it happen with other clocks as well?

Unclear.

quoted

quoted

- The kernel tree is at https://github.com/raspberrypi/linux/, but I could
  see no relevant changes compared to a mainline kernel.

Is this bug reproducible with mainline kernel or mainline kernel can't be
booted on bcm2711?

Mainline linux-5.6 should boot on that machine but might not have
all the other features, so I think users tend to use the raspberry pi
kernel sources for now.

quoted

quoted

- From the report, I see that the returned time value is larger than the
  expected time, by 3.4 to 14.5 million seconds in four samples, my
  guess is that a random number gets added in at some point.

What kind code are you using to reproduce it? It is threaded or issue
clock_gettime from signal handlers?

The reproducer is very simple without threads or signals,
see the start of https://github.com/richfelker/musl-cross-make/issues/96

It does rely on calling into the musl wrapper, not the direct vdso
call.

quoted

quoted

- From other sources, I found that the Raspberry Pi clocksource runs
  at 54 MHz, with a mask value of 0xffffffffffffff. From these numbers
  I would expect that reading a completely random hardware register
  value would result in an offset up to 1.33 billion seconds, which is
  around factor 100 more than the error we see, though similar.

- The test case calls the musl clock_gettime() function, which falls back to
  the clock_gettime64() syscall on kernels prior to 5.5, or to the 32-bit
  clock_gettime() prior to Linux-5.1. As reported in the bug, Linux-4.19 does
  not show the bug.

- The behavior was not reproduced on the same user space in qemu,
  though I cannot tell whether the exact same kernel binary was used.

- glibc-2.31 calls the same clock_gettime64() vdso function on arm to
  implement clock_gettime(), but earlier versions did not. I have not
  seen any reports of this bug, which could be explained by users
  generally being on older versions.

- As far as I can tell, there are no reports of this bug from other users,
  and so far nobody could reproduce it.

note: i could not reproduce it in qemu-system with these configs:

qemu-system-aarch64 + arm64 kernel + compat vdso
qemu-system-aarch64 + kvm accel (on cortex-a72) + 32bit arm kernel
qemu-system-arm + cpu max + 32bit arm kernel

so i think it's something specific to that user's setup
(maybe rpi hw bug or gcc miscompiled the vdso or something
with that particular linux, i built my own linux 5.6 because
i did not know the exact kernel version where the bug was seen)

i don't have access to rpi (or other cortex-a53 where i
can install my own kernel) so this is as far as i got.

If we have a binary of the kernel that's known to be failing on the
hardware, it would be useful to dump its vdso and examine the
disassembly to see if it was miscompiled.

Rich

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel