Re: ❌ FAIL: Test report for?kernel?5.11.0-rc7 (arm-next)

----- Original Message -----

From: "Will Deacon" <will@kernel.org>
To: "Veronika Kabatova" <redacted>
Cc: "catalin marinas" <catalin.marinas@arm.com>, "CKI Project" <redacted>,
linux-arm-kernel@lists.infradead.org
Sent: Wednesday, February 10, 2021 6:31:34 PM
Subject: Re: ❌ FAIL: Test report	for?kernel?5.11.0-rc7 (arm-next)

On Wed, Feb 10, 2021 at 12:07:23PM -0500, Veronika Kabatova wrote:

quoted

quoted

On Wed, Feb 10, 2021 at 10:24:31AM -0500, Veronika Kabatova wrote:

quoted

Hi, I have a few results back:

- resubmitted the same kernel: gets stuck in the same spot
- tried the new version pushed today: gets stuck in the same spot

That's odd, as I just received a pass report for that branch!

https://lore.kernel.org/r/cki.598435E2D5.M3C5MKJ1NV@redhat.com (local)

Is it just flakey, perhaps? Obviously, that's not great either, but it
will
make bisection more challenging.

We have a large number of machines (both physical and virtual) and it's
impossible to run all tests on all of them, so they are randomly picked as
long as they fit the distro and test requirements. The distribution for
ARM tree is 1 physical and 1 "any" machine (which usually ends up being
virtual). The jobs from the report you linked ran on different machines
and didn't pick the one that failed to boot previously, so I manually
forced my testing to pick that machine to eliminate some variables.

Ah thanks, I hadn't twigged that it was a different set of hosts each time.
Makes sense.

quoted

The machine in question can on course be somewhat flaky (hard to eliminate
that possibility completely), but I checked our historical data and it
didn't fail to boot a single time other than with these two new kernels.

So the first thing we should probably try is whether vanilla -rc7 fails on
the machine causing us problems. If it does, then the arm64 queue for 5.12
is out of the equation, if not then we can try a targetted bisection.

Would you be able to try v5.11-rc7 please?

Can do.

Someone snatched up the machine in the meanwhile and appears to have it
reserved till next Monday :/ I sincerely hope that they'll release it sooner
and have queued up the test job with high priority. Will let you know right
as I get the results.

quoted

quoted

quoted

- tried the version from last week: boots ok

There is an extra message from the run that managed to boot, which is
not
present with any of the runs that failed:

EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus
value

But this message is not present with the stable run that I mentioned
previously.

Interesting. Are those messages in the logs anywhere? It would be handy
to
include them, if possible.

The messages are from before the kernel boot banner which is the marker
we use for log inclusion (to reduce console log spam from distro
installation which uses a different kernel and thus makes debugging less
straightforward). The same EFI messages are present before the kernel
banner in the new report you linked, and with the passing job from the
previous runs as well:

EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1]
[    0.000000] Linux version 5.11.0-rc7
(cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc
(GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version
2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021
[    0.000000] efi: EFI v2.70 by EDK II
[    0.000000] efi: SMBIOS 3.0=0x1bf760000 MEMATTR=0x1be656018 ACPI
2.0=0x1bc030000 RNG=0x1bf86cf98 MEMRESERVE=0x1bc3d3e18
[    0.000000] efi: seeding entropy pool
....

and

EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1]
[    0.000000] Linux version 5.11.0-rc7
(cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc
(GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version
2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021
[    0.000000] efi: EFI v2.70 by American Megatrends
....

The failing machine/kernel combos get stuck right after that last EFI
line before the kernel messages come in.

Sorry, just to be clear here: do we always fail when we have the "KASLR
will be disabled" message, or do some machines pass with that?

No. The message is sometimes present, sometimes not (likely machine
dependent), and doesn't seem to influence the boot results. Both of the
log excerpts above are from jobs that booted.


Veronika

Thanks again,

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel