Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git
Commit: 4176e42aa565 - Merge branch 'for-next/cpufeature' into for-kernelci
The results of these automated tests are provided below.
Overall result: FAILED (see details below)
Merge: OK
Compile: OK
Tests: FAILED
All kernel binaries, config files, and logs are available for download here:
https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix=datawarehouse-public/2021/02/09/623431
One or more kernel tests failed:
aarch64:
❌ Boot test
We hope that these logs can help you find the problem quickly. For the full
detail on our testing procedures, please scroll to the bottom of this message.
Please reply to this email if you have any questions about the tests that we
ran or if you have any suggestions on how to make future tests more effective.
,-. ,-.
( C ) ( K ) Continuous
`-',-.`-' Kernel
( I ) Integration
`-'
______________________________________________________________________________
Compile testing
---------------
We compiled the kernel for 1 architecture:
aarch64:
make options: make -j30 INSTALL_MOD_STRIP=1 targz-pkg
Hardware testing
----------------
We booted each kernel and ran the following tests:
aarch64:
Host 1:
✅ Boot test
✅ ACPI table test
✅ ACPI enabled test
✅ LTP
✅ Loopdev Sanity
✅ Memory: fork_mem
✅ Memory function: memfd_create
✅ AMTU (Abstract Machine Test Utility)
✅ Networking bridge: sanity
✅ Networking socket: fuzz
✅ Networking: igmp conformance test
✅ Networking route: pmtu
✅ Networking route_func - local
✅ Networking route_func - forward
✅ Networking TCP: keepalive test
✅ Networking UDP: socket
✅ Networking tunnel: geneve basic test
✅ Networking tunnel: gre basic
✅ L2TP basic test
✅ Networking tunnel: vxlan basic
✅ Networking ipsec: basic netns - transport
✅ Networking ipsec: basic netns - tunnel
✅ Libkcapi AF_ALG test
✅ pciutils: update pci ids test
✅ ALSA PCM loopback test
✅ ALSA Control (mixer) Userspace Element test
✅ storage: SCSI VPD
🚧 ✅ CIFS Connectathon
🚧 ✅ POSIX pjd-fstest suites
🚧 ✅ Firmware test suite
🚧 ✅ jvm - jcstress tests
🚧 ✅ Memory function: kaslr
🚧 ✅ Ethernet drivers sanity
🚧 ✅ Networking firewall: basic netfilter test
🚧 ✅ audit: audit testsuite test
🚧 ✅ trace: ftrace/tracer
Host 2:
❌ Boot test
⚡⚡⚡ selinux-policy: serge-testsuite
⚡⚡⚡ storage: software RAID testing
🚧 ⚡⚡⚡ xfstests - ext4
🚧 ⚡⚡⚡ xfstests - xfs
🚧 ⚡⚡⚡ xfstests - btrfs
🚧 ⚡⚡⚡ IPMI driver test
🚧 ⚡⚡⚡ IPMItool loop stress test
🚧 ⚡⚡⚡ Storage blktests
🚧 ⚡⚡⚡ Storage block - filesystem fio test
🚧 ⚡⚡⚡ Storage block - queue scheduler test
🚧 ⚡⚡⚡ Storage nvme - tcp
🚧 ⚡⚡⚡ Storage: swraid mdadm raid_module test
🚧 ⚡⚡⚡ stress: stress-ng
Test sources: https://gitlab.com/cki-project/kernel-tests
💚 Pull requests are welcome for new tests or improvements to existing tests!
Aborted tests
-------------
Tests that didn't complete running successfully are marked with ⚡⚡⚡.
If this was caused by an infrastructure issue, we try to mark that
explicitly in the report.
Waived tests
------------
If the test run included waived tests, they are marked with 🚧. Such tests are
executed but their results are not taken into account. Tests are waived when
their results are not reliable enough, e.g. when they're just introduced or are
being fixed.
Testing timeout
---------------
We aim to provide a report within reasonable timeframe. Tests that haven't
finished running yet are marked with ⏱.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Which system (e.g. soc) is host 2 and are there are known infra issues at
the moment? I did push some changes which affect the early boot path, so we
may well be running into a kernel bug, but I'd just like to make sure before
we dive in trying to debug that, especially as we haven't seen failures on
other systems (and host 1 seems ok).
In the meantime, I'll push a new branch without those early changes to see
if that does any better.
Thanks,
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: "Will Deacon" <will@kernel.org>
To: "CKI Project" <redacted>
Cc: "catalin marinas" <catalin.marinas@arm.com>, linux-arm-kernel@lists.infradead.org
Sent: Wednesday, February 10, 2021 10:29:46 AM
Subject: Re: ❌ FAIL: Test report for kernel 5.11.0-rc7 (arm-next)
Hi CKI folks,
Please can you help me with the error below?
On Tue, Feb 09, 2021 at 09:07:50PM -0000, CKI Project wrote:
Which system (e.g. soc) is host 2 and are there are known infra issues at
the moment? I did push some changes which affect the early boot path, so we
may well be running into a kernel bug, but I'd just like to make sure before
we dive in trying to debug that, especially as we haven't seen failures on
other systems (and host 1 seems ok).
Hi, the machine in question is a Cavium ThunderX2 Sabre. It booted a stable
kernel just a few days back okay. The last messages I can see in the raw
console log from this run are:
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
and then it times out after hour and half. I'm not aware of any ongoing
issues, however sometimes the link between the lab controller and the
machines can sometimes go wrong after reboot and lead to a similarly
looking problem.
I'll resubmit the test job on that same machine to check if that was
the case and let you know right after it boots.
Veronika
In the meantime, I'll push a new branch without those early changes to see
if that does any better.
Thanks,
Will
From: "Veronika Kabatova" <redacted>
To: "Will Deacon" <will@kernel.org>
Cc: "catalin marinas" <catalin.marinas@arm.com>, "CKI Project" <redacted>,
linux-arm-kernel@lists.infradead.org
Sent: Wednesday, February 10, 2021 1:01:36 PM
Subject: Re: ❌ FAIL: Test report for kernel 5.11.0-rc7 (arm-next)
----- Original Message -----
quoted
From: "Will Deacon" <will@kernel.org>
To: "CKI Project" <redacted>
Cc: "catalin marinas" <catalin.marinas@arm.com>,
linux-arm-kernel@lists.infradead.org
Sent: Wednesday, February 10, 2021 10:29:46 AM
Subject: Re: ❌ FAIL: Test report for kernel 5.11.0-rc7 (arm-next)
Hi CKI folks,
Please can you help me with the error below?
On Tue, Feb 09, 2021 at 09:07:50PM -0000, CKI Project wrote:
Which system (e.g. soc) is host 2 and are there are known infra issues at
the moment? I did push some changes which affect the early boot path, so we
may well be running into a kernel bug, but I'd just like to make sure
before
we dive in trying to debug that, especially as we haven't seen failures on
other systems (and host 1 seems ok).
Hi, the machine in question is a Cavium ThunderX2 Sabre. It booted a stable
kernel just a few days back okay. The last messages I can see in the raw
console log from this run are:
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
and then it times out after hour and half. I'm not aware of any ongoing
issues, however sometimes the link between the lab controller and the
machines can sometimes go wrong after reboot and lead to a similarly
looking problem.
I'll resubmit the test job on that same machine to check if that was
the case and let you know right after it boots.
Hi, I have a few results back:
- resubmitted the same kernel: gets stuck in the same spot
- tried the new version pushed today: gets stuck in the same spot
- tried the version from last week: boots ok
There is an extra message from the run that managed to boot, which is not
present with any of the runs that failed:
EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value
But this message is not present with the stable run that I mentioned
previously.
We didn't change the builder container for a while now and since I tested
things on the same machine one run after other, there should also be no
distro/package changes involved and the only difference should be the
kernel.
Veronika
Veronika
quoted
In the meantime, I'll push a new branch without those early changes to see
if that does any better.
Thanks,
Will
Which system (e.g. soc) is host 2 and are there are known infra issues at
the moment? I did push some changes which affect the early boot path, so we
may well be running into a kernel bug, but I'd just like to make sure
before
we dive in trying to debug that, especially as we haven't seen failures on
other systems (and host 1 seems ok).
Hi, the machine in question is a Cavium ThunderX2 Sabre. It booted a stable
kernel just a few days back okay. The last messages I can see in the raw
console log from this run are:
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
and then it times out after hour and half. I'm not aware of any ongoing
issues, however sometimes the link between the lab controller and the
machines can sometimes go wrong after reboot and lead to a similarly
looking problem.
I'll resubmit the test job on that same machine to check if that was
the case and let you know right after it boots.
Hi, I have a few results back:
- resubmitted the same kernel: gets stuck in the same spot
- tried the new version pushed today: gets stuck in the same spot
- tried the version from last week: boots ok
There is an extra message from the run that managed to boot, which is not
present with any of the runs that failed:
EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus value
But this message is not present with the stable run that I mentioned
previously.
Interesting. Are those messages in the logs anywhere? It would be handy to
include them, if possible.
Cheers,
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: "Will Deacon" <will@kernel.org>
To: "Veronika Kabatova" <redacted>
Cc: "catalin marinas" <catalin.marinas@arm.com>, "CKI Project" <redacted>,
linux-arm-kernel@lists.infradead.org
Sent: Wednesday, February 10, 2021 5:09:37 PM
Subject: Re: ❌ FAIL: Test report for kernel?5.11.0-rc7 (arm-next)
Hi Veronika,
Thanks for the help with this.
On Wed, Feb 10, 2021 at 10:24:31AM -0500, Veronika Kabatova wrote:
quoted
quoted
quoted
On Tue, Feb 09, 2021 at 09:07:50PM -0000, CKI Project wrote:
Which system (e.g. soc) is host 2 and are there are known infra issues
at
the moment? I did push some changes which affect the early boot path,
so we
may well be running into a kernel bug, but I'd just like to make sure
before
we dive in trying to debug that, especially as we haven't seen failures
on
other systems (and host 1 seems ok).
Hi, the machine in question is a Cavium ThunderX2 Sabre. It booted a
stable
kernel just a few days back okay. The last messages I can see in the raw
console log from this run are:
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
and then it times out after hour and half. I'm not aware of any ongoing
issues, however sometimes the link between the lab controller and the
machines can sometimes go wrong after reboot and lead to a similarly
looking problem.
I'll resubmit the test job on that same machine to check if that was
the case and let you know right after it boots.
Hi, I have a few results back:
- resubmitted the same kernel: gets stuck in the same spot
- tried the new version pushed today: gets stuck in the same spot
We have a large number of machines (both physical and virtual) and it's
impossible to run all tests on all of them, so they are randomly picked as
long as they fit the distro and test requirements. The distribution for
ARM tree is 1 physical and 1 "any" machine (which usually ends up being
virtual). The jobs from the report you linked ran on different machines
and didn't pick the one that failed to boot previously, so I manually
forced my testing to pick that machine to eliminate some variables.
The machine in question can on course be somewhat flaky (hard to eliminate
that possibility completely), but I checked our historical data and it
didn't fail to boot a single time other than with these two new kernels.
quoted
- tried the version from last week: boots ok
There is an extra message from the run that managed to boot, which is not
present with any of the runs that failed:
EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus
value
But this message is not present with the stable run that I mentioned
previously.
Interesting. Are those messages in the logs anywhere? It would be handy to
include them, if possible.
The messages are from before the kernel boot banner which is the marker
we use for log inclusion (to reduce console log spam from distro
installation which uses a different kernel and thus makes debugging less
straightforward). The same EFI messages are present before the kernel
banner in the new report you linked, and with the passing job from the
previous runs as well:
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1]
[ 0.000000] Linux version 5.11.0-rc7 (cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc (GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version 2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021
[ 0.000000] efi: EFI v2.70 by EDK II
[ 0.000000] efi: SMBIOS 3.0=0x1bf760000 MEMATTR=0x1be656018 ACPI 2.0=0x1bc030000 RNG=0x1bf86cf98 MEMRESERVE=0x1bc3d3e18
[ 0.000000] efi: seeding entropy pool
....
and
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1]
[ 0.000000] Linux version 5.11.0-rc7 (cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc (GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version 2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021
[ 0.000000] efi: EFI v2.70 by American Megatrends
....
The failing machine/kernel combos get stuck right after that last EFI
line before the kernel messages come in.
Let me know if I should test some other versions or if you need some
other information!
Veronika
From: Will Deacon <will@kernel.org> Date: 2021-02-10 17:32:52
On Wed, Feb 10, 2021 at 12:07:23PM -0500, Veronika Kabatova wrote:
quoted
On Wed, Feb 10, 2021 at 10:24:31AM -0500, Veronika Kabatova wrote:
quoted
Hi, I have a few results back:
- resubmitted the same kernel: gets stuck in the same spot
- tried the new version pushed today: gets stuck in the same spot
We have a large number of machines (both physical and virtual) and it's
impossible to run all tests on all of them, so they are randomly picked as
long as they fit the distro and test requirements. The distribution for
ARM tree is 1 physical and 1 "any" machine (which usually ends up being
virtual). The jobs from the report you linked ran on different machines
and didn't pick the one that failed to boot previously, so I manually
forced my testing to pick that machine to eliminate some variables.
Ah thanks, I hadn't twigged that it was a different set of hosts each time.
Makes sense.
The machine in question can on course be somewhat flaky (hard to eliminate
that possibility completely), but I checked our historical data and it
didn't fail to boot a single time other than with these two new kernels.
So the first thing we should probably try is whether vanilla -rc7 fails on
the machine causing us problems. If it does, then the arm64 queue for 5.12
is out of the equation, if not then we can try a targetted bisection.
Would you be able to try v5.11-rc7 please?
quoted
quoted
- tried the version from last week: boots ok
There is an extra message from the run that managed to boot, which is not
present with any of the runs that failed:
EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus
value
But this message is not present with the stable run that I mentioned
previously.
Interesting. Are those messages in the logs anywhere? It would be handy to
include them, if possible.
The messages are from before the kernel boot banner which is the marker
we use for log inclusion (to reduce console log spam from distro
installation which uses a different kernel and thus makes debugging less
straightforward). The same EFI messages are present before the kernel
banner in the new report you linked, and with the passing job from the
previous runs as well:
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1]
[ 0.000000] Linux version 5.11.0-rc7 (cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc (GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version 2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021
[ 0.000000] efi: EFI v2.70 by EDK II
[ 0.000000] efi: SMBIOS 3.0=0x1bf760000 MEMATTR=0x1be656018 ACPI 2.0=0x1bc030000 RNG=0x1bf86cf98 MEMRESERVE=0x1bc3d3e18
[ 0.000000] efi: seeding entropy pool
....
and
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1]
[ 0.000000] Linux version 5.11.0-rc7 (cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc (GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version 2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021
[ 0.000000] efi: EFI v2.70 by American Megatrends
....
The failing machine/kernel combos get stuck right after that last EFI
line before the kernel messages come in.
Sorry, just to be clear here: do we always fail when we have the "KASLR
will be disabled" message, or do some machines pass with that?
Thanks again,
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: "Will Deacon" <will@kernel.org>
To: "Veronika Kabatova" <redacted>
Cc: "catalin marinas" <catalin.marinas@arm.com>, "CKI Project" <redacted>,
linux-arm-kernel@lists.infradead.org
Sent: Wednesday, February 10, 2021 6:31:34 PM
Subject: Re: ❌ FAIL: Test report for?kernel?5.11.0-rc7 (arm-next)
On Wed, Feb 10, 2021 at 12:07:23PM -0500, Veronika Kabatova wrote:
quoted
quoted
On Wed, Feb 10, 2021 at 10:24:31AM -0500, Veronika Kabatova wrote:
quoted
Hi, I have a few results back:
- resubmitted the same kernel: gets stuck in the same spot
- tried the new version pushed today: gets stuck in the same spot
We have a large number of machines (both physical and virtual) and it's
impossible to run all tests on all of them, so they are randomly picked as
long as they fit the distro and test requirements. The distribution for
ARM tree is 1 physical and 1 "any" machine (which usually ends up being
virtual). The jobs from the report you linked ran on different machines
and didn't pick the one that failed to boot previously, so I manually
forced my testing to pick that machine to eliminate some variables.
Ah thanks, I hadn't twigged that it was a different set of hosts each time.
Makes sense.
quoted
The machine in question can on course be somewhat flaky (hard to eliminate
that possibility completely), but I checked our historical data and it
didn't fail to boot a single time other than with these two new kernels.
So the first thing we should probably try is whether vanilla -rc7 fails on
the machine causing us problems. If it does, then the arm64 queue for 5.12
is out of the equation, if not then we can try a targetted bisection.
Would you be able to try v5.11-rc7 please?
Can do.
Someone snatched up the machine in the meanwhile and appears to have it
reserved till next Monday :/ I sincerely hope that they'll release it sooner
and have queued up the test job with high priority. Will let you know right
as I get the results.
quoted
quoted
quoted
- tried the version from last week: boots ok
There is an extra message from the run that managed to boot, which is
not
present with any of the runs that failed:
EFI stub: ERROR: FIRMWARE BUG: efi_loaded_image_t::image_base has bogus
value
But this message is not present with the stable run that I mentioned
previously.
Interesting. Are those messages in the logs anywhere? It would be handy
to
include them, if possible.
The messages are from before the kernel boot banner which is the marker
we use for log inclusion (to reduce console log spam from distro
installation which uses a different kernel and thus makes debugging less
straightforward). The same EFI messages are present before the kernel
banner in the new report you linked, and with the passing job from the
previous runs as well:
EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1]
[ 0.000000] Linux version 5.11.0-rc7
(cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc
(GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version
2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021
[ 0.000000] efi: EFI v2.70 by EDK II
[ 0.000000] efi: SMBIOS 3.0=0x1bf760000 MEMATTR=0x1be656018 ACPI
2.0=0x1bc030000 RNG=0x1bf86cf98 MEMRESERVE=0x1bc3d3e18
[ 0.000000] efi: seeding entropy pool
....
and
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable, KASLR will be disabled
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x431f0af1]
[ 0.000000] Linux version 5.11.0-rc7
(cki@runner-3uc3rmvr-project-2-concurrent-2lpn99) (aarch64-linux-gnu-gcc
(GCC) 10.2.1 20200826 (Red Hat Cross 10.2.1-3), GNU ld version
2.35.1-1.fc33) #1 SMP Wed Feb 10 09:47:23 UTC 2021
[ 0.000000] efi: EFI v2.70 by American Megatrends
....
The failing machine/kernel combos get stuck right after that last EFI
line before the kernel messages come in.
Sorry, just to be clear here: do we always fail when we have the "KASLR
will be disabled" message, or do some machines pass with that?
No. The message is sometimes present, sometimes not (likely machine
dependent), and doesn't seem to influence the boot results. Both of the
log excerpts above are from jobs that booted.
Veronika
From: Will Deacon <will@kernel.org> Date: 2021-02-10 18:57:17
On Wed, Feb 10, 2021 at 01:06:08PM -0500, Veronika Kabatova wrote:
----- Original Message -----
quoted
From: "Will Deacon" <will@kernel.org>
To: "Veronika Kabatova" <redacted>
Cc: "catalin marinas" <catalin.marinas@arm.com>, "CKI Project" <redacted>,
linux-arm-kernel@lists.infradead.org
Sent: Wednesday, February 10, 2021 6:31:34 PM
Subject: Re: ❌ FAIL: Test report for?kernel?5.11.0-rc7 (arm-next)
On Wed, Feb 10, 2021 at 12:07:23PM -0500, Veronika Kabatova wrote:
quoted
quoted
On Wed, Feb 10, 2021 at 10:24:31AM -0500, Veronika Kabatova wrote:
quoted
Hi, I have a few results back:
- resubmitted the same kernel: gets stuck in the same spot
- tried the new version pushed today: gets stuck in the same spot
We have a large number of machines (both physical and virtual) and it's
impossible to run all tests on all of them, so they are randomly picked as
long as they fit the distro and test requirements. The distribution for
ARM tree is 1 physical and 1 "any" machine (which usually ends up being
virtual). The jobs from the report you linked ran on different machines
and didn't pick the one that failed to boot previously, so I manually
forced my testing to pick that machine to eliminate some variables.
Ah thanks, I hadn't twigged that it was a different set of hosts each time.
Makes sense.
quoted
The machine in question can on course be somewhat flaky (hard to eliminate
that possibility completely), but I checked our historical data and it
didn't fail to boot a single time other than with these two new kernels.
So the first thing we should probably try is whether vanilla -rc7 fails on
the machine causing us problems. If it does, then the arm64 queue for 5.12
is out of the equation, if not then we can try a targetted bisection.
Would you be able to try v5.11-rc7 please?
Can do.
Brill, thanks.
Someone snatched up the machine in the meanwhile and appears to have it
reserved till next Monday :/ I sincerely hope that they'll release it sooner
and have queued up the test job with high priority. Will let you know right
as I get the results.
Just tell them the machine is broken and they really don't want to use it
for anything important ;)
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: "Will Deacon" <will@kernel.org>
To: "Veronika Kabatova" <redacted>
Cc: "catalin marinas" <catalin.marinas@arm.com>, "CKI Project" <redacted>,
linux-arm-kernel@lists.infradead.org
Sent: Wednesday, February 10, 2021 7:56:03 PM
Subject: Re: ❌ FAIL: Test report?for?kernel?5.11.0-rc7 (arm-next)
On Wed, Feb 10, 2021 at 01:06:08PM -0500, Veronika Kabatova wrote:
quoted
----- Original Message -----
quoted
From: "Will Deacon" <will@kernel.org>
To: "Veronika Kabatova" <redacted>
Cc: "catalin marinas" <catalin.marinas@arm.com>, "CKI Project"
[off-list ref],
linux-arm-kernel@lists.infradead.org
Sent: Wednesday, February 10, 2021 6:31:34 PM
Subject: Re: ❌ FAIL: Test report for?kernel?5.11.0-rc7 (arm-next)
On Wed, Feb 10, 2021 at 12:07:23PM -0500, Veronika Kabatova wrote:
quoted
quoted
On Wed, Feb 10, 2021 at 10:24:31AM -0500, Veronika Kabatova wrote:
quoted
Hi, I have a few results back:
- resubmitted the same kernel: gets stuck in the same spot
- tried the new version pushed today: gets stuck in the same spot
We have a large number of machines (both physical and virtual) and it's
impossible to run all tests on all of them, so they are randomly picked
as
long as they fit the distro and test requirements. The distribution for
ARM tree is 1 physical and 1 "any" machine (which usually ends up being
virtual). The jobs from the report you linked ran on different machines
and didn't pick the one that failed to boot previously, so I manually
forced my testing to pick that machine to eliminate some variables.
Ah thanks, I hadn't twigged that it was a different set of hosts each
time.
Makes sense.
quoted
The machine in question can on course be somewhat flaky (hard to
eliminate
that possibility completely), but I checked our historical data and it
didn't fail to boot a single time other than with these two new
kernels.
So the first thing we should probably try is whether vanilla -rc7 fails
on
the machine causing us problems. If it does, then the arm64 queue for
5.12
is out of the equation, if not then we can try a targetted bisection.
Would you be able to try v5.11-rc7 please?
Can do.
Brill, thanks.
quoted
Someone snatched up the machine in the meanwhile and appears to have it
reserved till next Monday :/ I sincerely hope that they'll release it
sooner
and have queued up the test job with high priority. Will let you know right
as I get the results.
Just tell them the machine is broken and they really don't want to use it
for anything important ;)
Our wishes were granted by the lab fairy and the machine was returned
rather quickly :)
The 5.11-rc7 kernel boots.
Veronika
From: Will Deacon <will@kernel.org> Date: 2021-02-10 20:19:30
On Wed, Feb 10, 2021 at 02:31:45PM -0500, Veronika Kabatova wrote:
quoted
quoted
quoted
quoted
The machine in question can on course be somewhat flaky (hard to
eliminate
that possibility completely), but I checked our historical data and it
didn't fail to boot a single time other than with these two new
kernels.
So the first thing we should probably try is whether vanilla -rc7 fails
on
the machine causing us problems. If it does, then the arm64 queue for
5.12
is out of the equation, if not then we can try a targetted bisection.
Would you be able to try v5.11-rc7 please?
Can do.
Brill, thanks.
quoted
Someone snatched up the machine in the meanwhile and appears to have it
reserved till next Monday :/ I sincerely hope that they'll release it
sooner
and have queued up the test job with high priority. Will let you know right
as I get the results.
Just tell them the machine is broken and they really don't want to use it
for anything important ;)
Our wishes were granted by the lab fairy and the machine was returned
rather quickly :)
The 5.11-rc7 kernel boots.
Fantastic! Then it's something in the arm64 for-next/core tree that was
added since then. The diff isn't huge and one change stands out for me, so
let me try reverting that and I'll update the branch...
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: Will Deacon <will@kernel.org> Date: 2021-02-10 20:33:21
On Wed, Feb 10, 2021 at 08:17:53PM +0000, Will Deacon wrote:
On Wed, Feb 10, 2021 at 02:31:45PM -0500, Veronika Kabatova wrote:
quoted
quoted
quoted
quoted
quoted
The machine in question can on course be somewhat flaky (hard to
eliminate
that possibility completely), but I checked our historical data and it
didn't fail to boot a single time other than with these two new
kernels.
So the first thing we should probably try is whether vanilla -rc7 fails
on
the machine causing us problems. If it does, then the arm64 queue for
5.12
is out of the equation, if not then we can try a targetted bisection.
Would you be able to try v5.11-rc7 please?
Can do.
Brill, thanks.
quoted
Someone snatched up the machine in the meanwhile and appears to have it
reserved till next Monday :/ I sincerely hope that they'll release it
sooner
and have queued up the test job with high priority. Will let you know right
as I get the results.
Just tell them the machine is broken and they really don't want to use it
for anything important ;)
Our wishes were granted by the lab fairy and the machine was returned
rather quickly :)
The 5.11-rc7 kernel boots.
Fantastic! Then it's something in the arm64 for-next/core tree that was
added since then. The diff isn't huge and one change stands out for me, so
let me try reverting that and I'll update the branch...
Ok, I updated for-kernelci so that it contains the arm64 for-next/core
branch merged into -rc7, but with a couple of patches reverted on top.
HEAD is e56137cc7606 ("Revert "arm64/mm: Fix pfn_valid() for ZONE_DEVICE
based memory""). Please can you try that?
Cheers,
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
From: "Will Deacon" <will@kernel.org>
To: "Veronika Kabatova" <redacted>
Cc: "catalin marinas" <catalin.marinas@arm.com>, linux-arm-kernel@lists.infradead.org, "CKI Project"
[off-list ref]
Sent: Wednesday, February 10, 2021 9:32:06 PM
Subject: Re: ❌ FAIL: Test?report?for?kernel?5.11.0-rc7 (arm-next)
On Wed, Feb 10, 2021 at 08:17:53PM +0000, Will Deacon wrote:
quoted
On Wed, Feb 10, 2021 at 02:31:45PM -0500, Veronika Kabatova wrote:
quoted
quoted
quoted
quoted
quoted
The machine in question can on course be somewhat flaky (hard to
eliminate
that possibility completely), but I checked our historical data
and it
didn't fail to boot a single time other than with these two new
kernels.
So the first thing we should probably try is whether vanilla -rc7
fails
on
the machine causing us problems. If it does, then the arm64 queue
for
5.12
is out of the equation, if not then we can try a targetted
bisection.
Would you be able to try v5.11-rc7 please?
Can do.
Brill, thanks.
quoted
Someone snatched up the machine in the meanwhile and appears to have
it
reserved till next Monday :/ I sincerely hope that they'll release it
sooner
and have queued up the test job with high priority. Will let you know
right
as I get the results.
Just tell them the machine is broken and they really don't want to use
it
for anything important ;)
Our wishes were granted by the lab fairy and the machine was returned
rather quickly :)
The 5.11-rc7 kernel boots.
Fantastic! Then it's something in the arm64 for-next/core tree that was
added since then. The diff isn't huge and one change stands out for me, so
let me try reverting that and I'll update the branch...
Ok, I updated for-kernelci so that it contains the arm64 for-next/core
branch merged into -rc7, but with a couple of patches reverted on top.
HEAD is e56137cc7606 ("Revert "arm64/mm: Fix pfn_valid() for ZONE_DEVICE
based memory""). Please can you try that?
Hi,
I didn't actually have to, the autopick picked the machine for the testing
and it happily booted :) \o/
Apparently IT cut off our email sending access because we're sending too
many emails so that's why you don't have the report yet. When we work
around it I'll make sure to retrigger the sending so you have it.
Veronika
From: Will Deacon <will@kernel.org> Date: 2021-02-11 11:52:47
On Thu, Feb 11, 2021 at 05:46:02AM -0500, Veronika Kabatova wrote:
quoted
On Wed, Feb 10, 2021 at 08:17:53PM +0000, Will Deacon wrote:
quoted
On Wed, Feb 10, 2021 at 02:31:45PM -0500, Veronika Kabatova wrote:
quoted
quoted
quoted
quoted
quoted
The machine in question can on course be somewhat flaky (hard to
eliminate
that possibility completely), but I checked our historical data
and it
didn't fail to boot a single time other than with these two new
kernels.
So the first thing we should probably try is whether vanilla -rc7
fails
on
the machine causing us problems. If it does, then the arm64 queue
for
5.12
is out of the equation, if not then we can try a targetted
bisection.
Would you be able to try v5.11-rc7 please?
Can do.
Brill, thanks.
quoted
Someone snatched up the machine in the meanwhile and appears to have
it
reserved till next Monday :/ I sincerely hope that they'll release it
sooner
and have queued up the test job with high priority. Will let you know
right
as I get the results.
Just tell them the machine is broken and they really don't want to use
it
for anything important ;)
Our wishes were granted by the lab fairy and the machine was returned
rather quickly :)
The 5.11-rc7 kernel boots.
Fantastic! Then it's something in the arm64 for-next/core tree that was
added since then. The diff isn't huge and one change stands out for me, so
let me try reverting that and I'll update the branch...
Ok, I updated for-kernelci so that it contains the arm64 for-next/core
branch merged into -rc7, but with a couple of patches reverted on top.
HEAD is e56137cc7606 ("Revert "arm64/mm: Fix pfn_valid() for ZONE_DEVICE
based memory""). Please can you try that?
I didn't actually have to, the autopick picked the machine for the testing
and it happily booted :) \o/
Phew, so I'll drop those from linux-next as well. Do you know if "earlycon"
works on the problematic machine? If possible, it would be helpful to try
booting the bad kernel with earlycon on the cmdline to see if it manages to
say anything in its dying breath.
Apparently IT cut off our email sending access because we're sending too
many emails so that's why you don't have the report yet. When we work
around it I'll make sure to retrigger the sending so you have it.
From: "Will Deacon" <will@kernel.org>
To: "Veronika Kabatova" <redacted>
Cc: "catalin marinas" <catalin.marinas@arm.com>, linux-arm-kernel@lists.infradead.org, "CKI Project"
[off-list ref]
Sent: Thursday, February 11, 2021 12:50:50 PM
Subject: Re: ❌ FAIL:?Test?report?for?kernel?5.11.0-rc7 (arm-next)
On Thu, Feb 11, 2021 at 05:46:02AM -0500, Veronika Kabatova wrote:
quoted
quoted
On Wed, Feb 10, 2021 at 08:17:53PM +0000, Will Deacon wrote:
quoted
On Wed, Feb 10, 2021 at 02:31:45PM -0500, Veronika Kabatova wrote:
quoted
quoted
quoted
quoted
quoted
The machine in question can on course be somewhat flaky (hard
to
eliminate
that possibility completely), but I checked our historical
data
and it
didn't fail to boot a single time other than with these two
new
kernels.
So the first thing we should probably try is whether vanilla
-rc7
fails
on
the machine causing us problems. If it does, then the arm64
queue
for
5.12
is out of the equation, if not then we can try a targetted
bisection.
Would you be able to try v5.11-rc7 please?
Can do.
Brill, thanks.
quoted
Someone snatched up the machine in the meanwhile and appears to
have
it
reserved till next Monday :/ I sincerely hope that they'll
release it
sooner
and have queued up the test job with high priority. Will let you
know
right
as I get the results.
Just tell them the machine is broken and they really don't want to
use
it
for anything important ;)
Our wishes were granted by the lab fairy and the machine was returned
rather quickly :)
The 5.11-rc7 kernel boots.
Fantastic! Then it's something in the arm64 for-next/core tree that was
added since then. The diff isn't huge and one change stands out for me,
so
let me try reverting that and I'll update the branch...
Ok, I updated for-kernelci so that it contains the arm64 for-next/core
branch merged into -rc7, but with a couple of patches reverted on top.
HEAD is e56137cc7606 ("Revert "arm64/mm: Fix pfn_valid() for ZONE_DEVICE
based memory""). Please can you try that?
I didn't actually have to, the autopick picked the machine for the testing
and it happily booted :) \o/
Phew, so I'll drop those from linux-next as well. Do you know if "earlycon"
works on the problematic machine? If possible, it would be helpful to try
booting the bad kernel with earlycon on the cmdline to see if it manages to
say anything in its dying breath.
No idea, we don't have the option enabled and I can't find any information
about it. I submitted a new job with the option and let's see whether it
works or not. The machine is reserved again so there may be some delays.
quoted
Apparently IT cut off our email sending access because we're sending too
many emails so that's why you don't have the report yet. When we work
around it I'll make sure to retrigger the sending so you have it.
That's nice of them...
Update for the drama loving folks: IT is actually innocent in this.
Someone deployed a misconfigured application in the same cluster as our
reporting system is in, and this application is sending out emails
every 3 seconds. We're collateral damage. Working on resolution, but
right now it seems that the situation is under control.
Veronika
From: "Will Deacon" <will@kernel.org>
To: "Veronika Kabatova" <redacted>
Cc: "catalin marinas" <catalin.marinas@arm.com>,
linux-arm-kernel@lists.infradead.org, "CKI Project"
[off-list ref]
Sent: Thursday, February 11, 2021 12:50:50 PM
Subject: Re: ❌ FAIL:?Test?report?for?kernel?5.11.0-rc7 (arm-next)
On Thu, Feb 11, 2021 at 05:46:02AM -0500, Veronika Kabatova wrote:
quoted
quoted
On Wed, Feb 10, 2021 at 08:17:53PM +0000, Will Deacon wrote:
quoted
On Wed, Feb 10, 2021 at 02:31:45PM -0500, Veronika Kabatova wrote:
quoted
quoted
quoted
quoted
quoted
The machine in question can on course be somewhat flaky
(hard
to
eliminate
that possibility completely), but I checked our historical
data
and it
didn't fail to boot a single time other than with these two
new
kernels.
So the first thing we should probably try is whether vanilla
-rc7
fails
on
the machine causing us problems. If it does, then the arm64
queue
for
5.12
is out of the equation, if not then we can try a targetted
bisection.
Would you be able to try v5.11-rc7 please?
Can do.
Brill, thanks.
quoted
Someone snatched up the machine in the meanwhile and appears to
have
it
reserved till next Monday :/ I sincerely hope that they'll
release it
sooner
and have queued up the test job with high priority. Will let
you
know
right
as I get the results.
Just tell them the machine is broken and they really don't want
to
use
it
for anything important ;)
Our wishes were granted by the lab fairy and the machine was
returned
rather quickly :)
The 5.11-rc7 kernel boots.
Fantastic! Then it's something in the arm64 for-next/core tree that
was
added since then. The diff isn't huge and one change stands out for
me,
so
let me try reverting that and I'll update the branch...
Ok, I updated for-kernelci so that it contains the arm64 for-next/core
branch merged into -rc7, but with a couple of patches reverted on top.
HEAD is e56137cc7606 ("Revert "arm64/mm: Fix pfn_valid() for
ZONE_DEVICE
based memory""). Please can you try that?
I didn't actually have to, the autopick picked the machine for the
testing
and it happily booted :) \o/
Phew, so I'll drop those from linux-next as well. Do you know if "earlycon"
works on the problematic machine? If possible, it would be helpful to try
booting the bad kernel with earlycon on the cmdline to see if it manages to
say anything in its dying breath.
No idea, we don't have the option enabled and I can't find any information
about it. I submitted a new job with the option and let's see whether it
works or not. The machine is reserved again so there may be some delays.
I finally got access to the machine and don't have any good news. The hang
is from before early printk takes effect. The output is identical to what
I sent before, with nothing resembling kernel lines in there.
Veronika
quoted
quoted
Apparently IT cut off our email sending access because we're sending too
many emails so that's why you don't have the report yet. When we work
around it I'll make sure to retrigger the sending so you have it.
That's nice of them...
Update for the drama loving folks: IT is actually innocent in this.
Someone deployed a misconfigured application in the same cluster as our
reporting system is in, and this application is sending out emails
every 3 seconds. We're collateral damage. Working on resolution, but
right now it seems that the situation is under control.
Veronika
From: Will Deacon <will@kernel.org> Date: 2021-02-15 18:25:16
On Mon, Feb 15, 2021 at 08:13:18AM -0500, Veronika Kabatova wrote:
quoted
quoted
On Thu, Feb 11, 2021 at 05:46:02AM -0500, Veronika Kabatova wrote:
quoted
quoted
On Wed, Feb 10, 2021 at 08:17:53PM +0000, Will Deacon wrote:
quoted
On Wed, Feb 10, 2021 at 02:31:45PM -0500, Veronika Kabatova wrote:
Ok, I updated for-kernelci so that it contains the arm64 for-next/core
branch merged into -rc7, but with a couple of patches reverted on top.
HEAD is e56137cc7606 ("Revert "arm64/mm: Fix pfn_valid() for
ZONE_DEVICE
based memory""). Please can you try that?
I didn't actually have to, the autopick picked the machine for the
testing
and it happily booted :) \o/
Phew, so I'll drop those from linux-next as well. Do you know if "earlycon"
works on the problematic machine? If possible, it would be helpful to try
booting the bad kernel with earlycon on the cmdline to see if it manages to
say anything in its dying breath.
No idea, we don't have the option enabled and I can't find any information
about it. I submitted a new job with the option and let's see whether it
works or not. The machine is reserved again so there may be some delays.
I finally got access to the machine and don't have any good news. The hang
is from before early printk takes effect. The output is identical to what
I sent before, with nothing resembling kernel lines in there.
Oh well, thanks for giving it a shot. I've already dropped the for-next/mm
branch from my pull request for 5.12, but please keep a note of the
problematic machine if possible so that we can try further debugging in
future.
Cheers,
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel