From: Abdul Haleem <hidden> Date: 2017-05-25 08:29:17
Hi,
Test : boot
Machine : Power 8 PowerVM LPAR
Kernel version : 4.12.0-rc2-next-20170524
Toolchain : gcc version 5.2.1
Config : attached
next-20170517 was last good version where we did not see the failure.
from next-20170519 to today's next-20170524, we see the problem
everyday.
Kernels fails to boot on PowerVM LPAR with these errors;
Freeing unused kernel memory: 4416K
This architecture does not have kernel memory protection.
Loading, please wait...
mount: mounting udev on /dev failed: Device or resource busy
W: devtmpfs not available, falling back to tmpfs for /dev
starting version 222
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin:
Running /scripts/local-top ... done.
chvt: can't open console
Gave up waiting for root device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Check root= (did the system wait for the right device?)
- Missing modules (cat /proc/modules; ls /dev)
chvt: can't open console
ALERT! /dev/disk/by-uuid/3aee8de3-5b8f-4a89-9e89-108efe213983 does not
exist. Dropping to a shell!
hid: module verification failed: signature and/or required key missing -
tainting kernel
hidraw: raw HID events driver (C) Jiri Kosina
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
Couldn't get a file descriptor referring to the console
(initramfs)
detail logs are attached.
--
Regard's
Abdul Haleem
IBM Linux Technology Centre
Hi,
Test : boot
Machine : Power 8 PowerVM LPAR
Kernel version : 4.12.0-rc2-next-20170524
Toolchain : gcc version 5.2.1
Config : attached
next-20170517 was last good version where we did not see the failure.
from next-20170519 to today's next-20170524, we see the problem
everyday.
Kernels fails to boot on PowerVM LPAR with these errors;
Freeing unused kernel memory: 4416K
This architecture does not have kernel memory protection.
Loading, please wait...
mount: mounting udev on /dev failed: Device or resource busy
W: devtmpfs not available, falling back to tmpfs for /dev
starting version 222
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin:
Running /scripts/local-top ... done.
chvt: can't open console
Gave up waiting for root device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Check root= (did the system wait for the right device?)
- Missing modules (cat /proc/modules; ls /dev)
chvt: can't open console
ALERT! /dev/disk/by-uuid/3aee8de3-5b8f-4a89-9e89-108efe213983 does not
exist. Dropping to a shell!
Seems like the by-uuid node for your harddrive containing rootfs didn't
show up.
The EBUSY from mount(/dev) seems like the biggest suspect.
These:
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2017-05-25 13:02:15
Abdul Haleem [off-list ref] writes:
Hi,
Test : boot
Machine : Power 8 PowerVM LPAR
Kernel version : 4.12.0-rc2-next-20170524
Toolchain : gcc version 5.2.1
Config : attached
next-20170517 was last good version where we did not see the failure.
from next-20170519 to today's next-20170524, we see the problem
everyday.
Kernels fails to boot on PowerVM LPAR with these errors;
Freeing unused kernel memory: 4416K
This architecture does not have kernel memory protection.
Loading, please wait...
mount: mounting udev on /dev failed: Device or resource busy
W: devtmpfs not available, falling back to tmpfs for /dev
It'll be:
ee35011fd032 ("initramfs: make initramfs honor CONFIG_DEVTMPFS_MOUNT")
cheers
From: Rob Landley <hidden> Date: 2017-05-26 06:24:56
On 05/25/2017 04:24 PM, Stephen Rothwell wrote:
Hi Michael,
On Thu, 25 May 2017 23:02:06 +1000 Michael Ellerman [off-list ref] wrote:
quoted
It'll be:
ee35011fd032 ("initramfs: make initramfs honor CONFIG_DEVTMPFS_MOUNT")
And Andrew has asked me to drop that patch from linux-next which will
happen today.
What approach do the kernel developers suggest I take here?
I would have thought letting it soak in linux-next for a release so
people could fix userspace bugs would be the next step, but this sounds
like that's not an option?
Is the behavior the patch implements wrong?
Rob
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2017-05-29 05:07:04
Rob Landley [off-list ref] writes:
On 05/25/2017 04:24 PM, Stephen Rothwell wrote:
quoted
Hi Michael,
On Thu, 25 May 2017 23:02:06 +1000 Michael Ellerman [off-list ref] wrote:
quoted
It'll be:
ee35011fd032 ("initramfs: make initramfs honor CONFIG_DEVTMPFS_MOUNT")
And Andrew has asked me to drop that patch from linux-next which will
happen today.
What approach do the kernel developers suggest I take here?
Well I'm just *a* kernel developer, but rule #1 is don't break userspace.
I would have thought letting it soak in linux-next for a release so
people could fix userspace bugs would be the next step, but this sounds
like that's not an option?
You say they're userspace bugs, userspace will say it's a bug that the
kernel has changed its behaviour.
Is the behavior the patch implements wrong?
Yes, because it breaks existing setups for no particularly good reason.
If CONFIG_DEVTMPFS_MOUNT had always meant devtmpfs was mounted in the
initramfs then that would have been fine.
But because it didn't, there are now systems out there that depend on
the existing behaviour, and changing it is therefore wrong IMHO.
As I said in another mail you can avoid breaking existing setups by
adding a new config option to control mounting devtmpfs in the
initramfs. It's a pity to need yet another config option, but such is
life.
cheers
@@ -48,16 +48,10 @@ config DEVTMPFS_MOUNTbool"Automount devtmpfs at /dev, after the kernel mounted the rootfs"depends onDEVTMPFShelp-Thiswillinstructthekerneltoautomaticallymountthe-devtmpfsfilesystemat/dev,directlyafterthekernelhas-mountedtherootfilesystem.Thebehaviorcanbeoverridden-withthecommandlineparameter:devtmpfs.mount=0|1.-Thisoptiondoesnotaffectinitramfsbasedbooting,here-thedevtmpfsfilesystemalwaysneedstobemountedmanually-aftertherootfsismounted.-Withthisoptionenabled,itallowstobringupasystemin-rescuemodewithinit=/bin/sh,evenwhenthe/devdirectory-ontherootfsiscompletelyempty.+Automaticallymountdevtmpfsat/devontherootfilesystem,which+letsthesystemtocomeupinrescuemodewith[rd]init=/bin/sh.+Overridewithdevtmpfs.mount=0onthecommandline.Initramfscan+createa/devdirasneeded,otherrootfsneedsthemountpoint.configSTANDALONEbool"Select only drivers that don't need compile-time external firmware"
@@ -1065,12 +1065,6 @@ static noinline void __init kernel_init_freeable(void)do_basic_setup();-/* Open the /dev/console on the rootfs, this should never fail */-if(sys_open((constchar__user*)"/dev/console",O_RDWR,0)<0)-pr_err("Warning: unable to open an initial console.\n");--(void)sys_dup(0);-(void)sys_dup(0);/**checkifthereisanearlyuserspaceinit.Ifyes,letitdoall*thework
@@ -1082,8 +1076,17 @@ static noinline void __init kernel_init_freeable(void)if(sys_access((constchar__user*)ramdisk_execute_command,0)!=0){ramdisk_execute_command=NULL;prepare_namespace();+}elseif(IS_ENABLED(CONFIG_DEVTMPFS_MOUNT)){+sys_mkdir("/dev",0755);+devtmpfs_mount("/dev");}+/* Open the /dev/console on the rootfs, this should never fail */+if(sys_open((constchar__user*)"/dev/console",O_RDWR,0)<0)+pr_err("Warning: unable to open an initial console.\n");+(void)sys_dup(0);+(void)sys_dup(0);+/**Ok,wehavecompletedtheinitialbootup,and*we'reessentiallyupandrunning.Getridofthe
From: Rob Landley <redacted>
Make initramfs honor CONFIG_DEVTMPFS_MOUNT, and move
/dev/console open after devtmpfs mount.
Add workaround for Debian bug that was copied by Ubuntu.
Is that a bug only for Debian ? Why ?
Why should a Debian bug be fixed by a workaround in the mainline kernel ?
@@ -48,16 +48,10 @@ config DEVTMPFS_MOUNTbool"Automount devtmpfs at /dev, after the kernel mounted the rootfs"depends onDEVTMPFShelp-Thiswillinstructthekerneltoautomaticallymountthe-devtmpfsfilesystemat/dev,directlyafterthekernelhas-mountedtherootfilesystem.Thebehaviorcanbeoverridden-withthecommandlineparameter:devtmpfs.mount=0|1.-Thisoptiondoesnotaffectinitramfsbasedbooting,here-thedevtmpfsfilesystemalwaysneedstobemountedmanually-aftertherootfsismounted.-Withthisoptionenabled,itallowstobringupasystemin-rescuemodewithinit=/bin/sh,evenwhenthe/devdirectory-ontherootfsiscompletelyempty.+Automaticallymountdevtmpfsat/devontherootfilesystem,which+letsthesystemtocomeupinrescuemodewith[rd]init=/bin/sh.+Overridewithdevtmpfs.mount=0onthecommandline.Initramfscan+createa/devdirasneeded,otherrootfsneedsthemountpoint.
Why modifying the initial text ?
Why talking about rescue mode only, whereas this feature mainly concerns
the standard mode.
quoted hunk
config STANDALONE
bool "Select only drivers that don't need compile-time external firmware"
@@ -1065,12 +1065,6 @@ static noinline void __init kernel_init_freeable(void)do_basic_setup();-/* Open the /dev/console on the rootfs, this should never fail */-if(sys_open((constchar__user*)"/dev/console",O_RDWR,0)<0)-pr_err("Warning: unable to open an initial console.\n");--(void)sys_dup(0);-(void)sys_dup(0);/**checkifthereisanearlyuserspaceinit.Ifyes,letitdoall*thework
Why not, but couldn't we also expect the initramfs to already contains
that mountpoint ?
+ devtmpfs_mount("/dev");
}
+ /* Open the /dev/console on the rootfs, this should never fail */
+ if (sys_open((const char __user *) "/dev/console", O_RDWR, 0) < 0)
+ pr_err("Warning: unable to open an initial console.\n");
+ (void) sys_dup(0);
+ (void) sys_dup(0);
+
/*
* Ok, we have completed the initial bootup, and
* we're essentially up and running. Get rid of the
On Wed, Sep 13, 2017 at 06:51:25PM -0500, Rob Landley wrote:
From: Rob Landley <redacted>
Make initramfs honor CONFIG_DEVTMPFS_MOUNT, and move
/dev/console open after devtmpfs mount.
Add workaround for Debian bug that was copied by Ubuntu.
Signed-off-by: Rob Landley <redacted>
---
v2 discussion: http://lkml.iu.edu/hypermail/linux/kernel/1705.2/05611.html
drivers/base/Kconfig | 14 ++++----------
fs/namespace.c | 14 ++++++++++++++
init/main.c | 15 +++++++++------
3 files changed, 27 insertions(+), 16 deletions(-)
Always run scripts/checkpatch.pl so you don't get grumpy emails from
reviewers telling you to run scripts/checkpatch.pl... telling you to run
scripts/checkpatch.pl... telling you to run scripts/checkpatch.pl...
telling you to run scripts/checkpatch.pl...
From: Rob Landley <hidden> Date: 2017-09-17 04:03:23
On 09/14/2017 04:17 AM, Christophe LEROY wrote:
Le 14/09/2017 à 01:51, Rob Landley a écrit :
quoted
From: Rob Landley <redacted>
Make initramfs honor CONFIG_DEVTMPFS_MOUNT, and move
/dev/console open after devtmpfs mount.
Add workaround for Debian bug that was copied by Ubuntu.
That's some discussion of version 2 of this patch, which was merged for
a while last dev cycle, then backed out again because it triggered the
same bug in a number of system init scripts:
http://lkml.iu.edu/hypermail/linux/kernel/1705.2/07072.htmlhttp://lkml.iu.edu/hypermail/linux/kernel/1705.3/01182.htmlhttp://lkml.iu.edu/hypermail/linux/kernel/1705.3/01505.htmlhttp://lkml.iu.edu/hypermail/linux/kernel/1705.3/01320.html
All of whom copied the broken error "recovery" path from debian. If they
checked whether it was already mounted, or didn't _blank_ the /dev
directory in response to mounting the exact same filesystem over itself
giving -EBUSY, the system would work fine. Heck, if you built a kernel
with a static /dev in initramfs and no devtmpfs configured in, the
script would break things exactly the same way. The breakage is that
script takes a hammer to a perfectly functional /dev directory and then
continues the boot with an empty /dev. That's bonkers.
Why should a Debian bug be fixed by a workaround in the mainline kernel ?
That was my argument last time, and the answer was "Breaking userspace
is bad, mmmkay." Even when userspace is doing something REALLY OBVIOUSLY
STUPID and it is _clearly_ their fault, as long as they got there first
they've established the status quo and it doesn't matter how silly it is.
This was explicitly stated to me here:
http://lkml.iu.edu/hypermail/linux/kernel/1705.3/03292.html
I.E. don't argue with me, argue with him. :)
So, I added a workaround with a printk in hopes of embarassing them into
someday fixing it.
Rob
From: Henrique de Moraes Holschuh <hmh@hmh.eng.br> Date: 2017-09-17 13:51:23
On Sat, 16 Sep 2017, Rob Landley wrote:
So, I added a workaround with a printk in hopes of embarassing them into
someday fixing it.
Oh, it will be fixed in Debian alright. I am just waiting the issue to
settle a bit to file the bug reports, or maybe even send in the Debian
patches myself (note that I am not responsible for the code in question,
so I am not wearing a brown paperbag at this time). Even if I didn't do
it, there are several other Debian Developers reading LKML that could do
it (provided they noticed this specific thread and are aware of the
situation) :p
I can even push for the fixes to be accepted into the stable and
oldstable branches of Debian, but that can take anything from a few
weeks to several months, due to the way our stable releases work. But
it would eventually happen.
Whether such fixes will ever make it to LTS branches, especially
Ubuntu's, *that* I don't know.
--
Henrique Holschuh
From: Rob Landley <hidden> Date: 2017-09-20 03:30:21
On 09/17/2017 08:51 AM, Henrique de Moraes Holschuh wrote:
On Sat, 16 Sep 2017, Rob Landley wrote:
quoted
So, I added a workaround with a printk in hopes of embarassing them into
someday fixing it.
Oh, it will be fixed in Debian alright.
Cool!
But part of the problem is people upgrade the kernel on existing
deployed root filesystems, some of which are a fork off of a fork off of
debian, so we won't exhaust the broken userspace for probably a couple
years.
I'd put it in feature-removal-schedule.txt but Linus zapped that, so...
I am just waiting the issue to
settle a bit to file the bug reports, or maybe even send in the Debian
patches myself (note that I am not responsible for the code in question,
so I am not wearing a brown paperbag at this time). Even if I didn't do
it, there are several other Debian Developers reading LKML that could do
it (provided they noticed this specific thread and are aware of the
situation) :p
There was a previous thread last merge window they didn't notice. I was
hoping the warning would be obvious enough. :)
I can even push for the fixes to be accepted into the stable and
oldstable branches of Debian, but that can take anything from a few
weeks to several months, due to the way our stable releases work. But
it would eventually happen.
Whether such fixes will ever make it to LTS branches, especially
Ubuntu's, *that* I don't know.
I have no idea what that powerpc system was, the guy didn't say...
Rob
From: Michael Ellerman <mpe@ellerman.id.au> Date: 2017-09-21 10:13:58
Rob Landley [off-list ref] writes:
On 09/14/2017 04:17 AM, Christophe LEROY wrote:
quoted
Le 14/09/2017 =C3=A0 01:51, Rob Landley a =C3=A9crit :
quoted
From: Rob Landley <redacted>
Make initramfs honor CONFIG_DEVTMPFS_MOUNT, and move
/dev/console open after devtmpfs mount.
Add workaround for Debian bug that was copied by Ubuntu.
That's some discussion of version 2 of this patch, which was merged for
a while last dev cycle, then backed out again because it triggered the
same bug in a number of system init scripts:
http://lkml.iu.edu/hypermail/linux/kernel/1705.2/07072.htmlhttp://lkml.iu.edu/hypermail/linux/kernel/1705.3/01182.htmlhttp://lkml.iu.edu/hypermail/linux/kernel/1705.3/01505.htmlhttp://lkml.iu.edu/hypermail/linux/kernel/1705.3/01320.html
All of whom copied the broken error "recovery" path from debian. If they
checked whether it was already mounted, or didn't _blank_ the /dev
directory in response to mounting the exact same filesystem over itself
giving -EBUSY, the system would work fine. Heck, if you built a kernel
with a static /dev in initramfs and no devtmpfs configured in, the
script would break things exactly the same way. The breakage is that
script takes a hammer to a perfectly functional /dev directory and then
continues the boot with an empty /dev. That's bonkers.
quoted
Why should a Debian bug be fixed by a workaround in the mainline kernel ?
That was my argument last time, and the answer was "Breaking userspace
is bad, mmmkay." Even when userspace is doing something REALLY OBVIOUSLY
STUPID and it is _clearly_ their fault, as long as they got there first
they've established the status quo and it doesn't matter how silly it is.
This was explicitly stated to me here:
http://lkml.iu.edu/hypermail/linux/kernel/1705.3/03292.html
I.E. don't argue with me, argue with him. :)
I'm still here. And I'm still right :)
No one wants their system to stop booting because of this obscure
functionality.
Just put it behind a new config option which defaults off. No
workarounds required, no broken systems, no long email threads required.
cheers