Re: ❌ FAIL: Test report for kernel 5.3.13-3b5f971.cki (stable-queue)

----- Original Message -----

Hi Jan,

Jan Stancek [off-list ref] writes:

quoted

----- Original Message -----

quoted

Hello,

We ran automated tests on a recent commit from this kernel tree:

       Kernel repo:
       git://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git
            Commit: 3b5f97139acc - KVM: PPC: Book3S HV: Flush link stack
            on
            guest exit to host kernel

I can't find this commit, I assume it's roughly the same as:

  https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/commit/?h=linux-5.3.y&id=0815f75f90178bc7e1933cf0d0c818b5f3f5a20c

Hi,

yes, that looks like same one:
  https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/commit/?h=3b5f97139acc

Looking at CKI reports for past 2 weeks, there were 3 (unexplained) SIGBUS related failures:

5.3.13-3b5f971.cki@upstream-stable
LTP genpower Bus error

5.4.0-rc8-4b17a56.cki@upstream-stable
LTP genatan Bus error

5.3.11-200.fc30
xfstests
+/var/lib/xfstests/tests/generic/248: line 38: 161943 Bus error               (core dumped) $TEST_PROG $TESTFILE

All 3 are from ppc64le, all power9 systems.

quoted

quoted

The results of these automated tests are provided below.

    Overall result: FAILED (see details below)
             Merge: OK
           Compile: OK
             Tests: FAILED

All kernel binaries, config files, and logs are available for download
here:

  https://artifacts.cki-project.org/pipelines/314344

One or more kernel tests failed:

    ppc64le:
     ❌ LTP

I suspect kernel bug.

Looks that way, but I can't reproduce it on a machine here.

I have the same CPU revision and am booting the exact kernel binary &
modules linked above.

I can semi-reliably reproduce it with:
(where LTP is installed to /mnt/testarea/ltp)

while [ True ]; do
        echo 3 > /proc/sys/vm/drop_caches
        rm -f /mnt/testarea/ltp/results/RUNTEST.log /mnt/testarea/ltp/output/RUNTEST.run.log
        ./runltp -p -d results -l RUNTEST.log -o RUNTEST.run.log -f math
        grep FAIL /mnt/testarea/ltp/results/RUNTEST.log && exit 1
done

and some stress activity in other terminal (e.g. kernel build).
Sometimes in minutes, sometimes in hours. I did try couple
older kernels and could reproduce it with v4.19 and v5.0 as well.

v4.18 ran OK for 2 hours, assuming that one is good, it could be
related to xfs switching to iomap in 4.19-rc1.

Tracing so far led me to filemap_fault(), where it reached this -EIO,
before returning SIGBUS.

page_not_uptodate:
        /*
         * Umm, take care of errors if the page isn't up-to-date.
         * Try to re-read it _once_. We do this synchronously,
         * because there really aren't any performance issues here
         * and we need to check for errors.
         */
        ClearPageError(page);
        fpin = maybe_unlock_mmap_for_io(vmf, fpin);
        error = mapping->a_ops->readpage(file, page);
        if (!error) {
                wait_on_page_locked(page);
                if (!PageUptodate(page))
                        error = -EIO;
        }

...
        return VM_FAULT_SIGBUS;

quoted

There were couple of 'math' runtest related failures in recent couple days.
In all cases, some data file used by test was missing. Presumably because
binary that generates it crashed.

I managed to reproduce one failure with this CKI build, which I believe
is the same problem.

We crash early during load, before any LTP code runs:

(gdb) r
Starting program: /mnt/testarea/ltp/testcases/bin/genasin

What is this /mnt/testarea? Looks like it's setup by some of the beaker
scripts or something?

Correct, it's where beaker script installs LTP. It's not a real mount,
just a directory on /. In my case it's xfs. It should match default
Fedora-31 Server ppc64le installation.

I'm running LTP out of /home, which is ext4 directly on disk.

I tried getting the tests-beaker stuff working on my machine, but I
couldn't find all the libraries and so on it requires.

quoted

Program received signal SIGBUS, Bus error.
dl_main (phdr=0x10000040, phnum=<optimized out>, user_entry=0x7fffffffe760,
auxv=<optimized out>) at rtld.c:1362
1362        switch (ph->p_type)
(gdb) bt
#0  dl_main (phdr=0x10000040, phnum=<optimized out>,
user_entry=0x7fffffffe760, auxv=<optimized out>) at rtld.c:1362
#1  0x00007ffff7fcf3c8 in _dl_sysdep_start (start_argptr=<optimized out>,
dl_main=0x7ffff7fb37b0 <dl_main>) at ../elf/dl-sysdep.c:253
#2  0x00007ffff7fb1d1c in _dl_start_final (arg=arg@entry=0x7fffffffee20,
info=info@entry=0x7fffffffe870) at rtld.c:445
#3  0x00007ffff7fb2f5c in _dl_start (arg=0x7fffffffee20) at rtld.c:537
#4  0x00007ffff7fb14d8 in _start () from /lib64/ld64.so.2
(gdb) f 0
#0  dl_main (phdr=0x10000040, phnum=<optimized out>,
user_entry=0x7fffffffe760, auxv=<optimized out>) at rtld.c:1362
1362        switch (ph->p_type)
(gdb) l
1357      /* And it was opened directly.  */
1358      ++main_map->l_direct_opencount;
1359
1360      /* Scan the program header table for the dynamic section.  */
1361      for (ph = phdr; ph < &phdr[phnum]; ++ph)
1362        switch (ph->p_type)
1363          {
1364          case PT_PHDR:
1365            /* Find out the load address.  */
1366            main_map->l_addr = (ElfW(Addr)) phdr - ph->p_vaddr;

(gdb) p ph
$1 = (const Elf64_Phdr *) 0x10000040

(gdb) p *ph
Cannot access memory at address 0x10000040

(gdb) info proc map
process 1110670
Mapped address spaces:

          Start Addr           End Addr       Size     Offset objfile
          0x10000000         0x10010000    0x10000        0x0
          /mnt/testarea/ltp/testcases/bin/genasin
          0x10010000         0x10030000    0x20000        0x0
          /mnt/testarea/ltp/testcases/bin/genasin
      0x7ffff7f90000     0x7ffff7fb0000    0x20000        0x0 [vdso]
      0x7ffff7fb0000     0x7ffff7fe0000    0x30000        0x0
      /usr/lib64/ld-2.30.so
      0x7ffff7fe0000     0x7ffff8000000    0x20000    0x20000
      /usr/lib64/ld-2.30.so
      0x7ffffffd0000     0x800000000000    0x30000        0x0 [stack]

(gdb) x/1x 0x10000040
0x10000040:     Cannot access memory at address 0x10000040

Yeah that's weird.

quoted

# /mnt/testarea/ltp/testcases/bin/genasin
Bus error (core dumped)

However, as soon as I copy that binary somewhere else, it works fine:

# cp /mnt/testarea/ltp/testcases/bin/genasin /tmp
# /tmp/genasin
# echo $?
0

Is /tmp a real disk or tmpfs?

tmpfs

Filesystem                           Type      1K-blocks     Used  Available Use% Mounted on
devtmpfs                             devtmpfs  254530176        0  254530176   0% /dev
tmpfs                                tmpfs     267992768        0  267992768   0% /dev/shm
tmpfs                                tmpfs     267992768     9152  267983616   1% /run
/dev/mapper/fedora_ibm--p9b--03-root xfs        15718400 13029284    2689116  83% /
tmpfs                                tmpfs     267992768        0  267992768   0% /tmp
/dev/sda1                            xfs         1038336   944588      93748  91% /boot
tmpfs                                tmpfs      53598528        0   53598528   0% /run/user/0