Re: [PATCH 5/7] printk: Make %pS and friends print module build ID

Quoting Petr Mladek (2021-03-03 02:25:58)

On Mon 2021-03-01 09:47:47, Stephen Boyd wrote:

quoted

The %pS printk format (among some others) is used to print kernel
addresses symbolically. When the kernel prints an address inside of a
module, the kernel prints the addresses' symbol name along with the
module's name that contains the address. Let's make kernel stacktraces
easier to identify on KALLSYMS builds by including the build ID of a
module when we print the address.

This is especially helpful for crash debugging with pstore or crashdump
kernels. If we have the build ID for the module in the stacktrace we can
request the debug symbols for the module from a remote debuginfod server

I have read the thread so for. IMHO, all mentioned variants complicate
the logs a lot. Either the backtrace lines are hard to parse.
Or the OOps/panic blocks gets too long when the ID is mentioned
for every loaded module. IMHO, neither proposed solution
is acceptable to be always used.

The modules line is already pretty hard to read once it gets beyond 8 or
10 modules. Probably it should be broken up into multiple lines just so
it can be read more easily. I find myself having to hunt in there
already even without the build ID encoded there. I've also seen folks
cut out that line in commit text and when posting to the mailing list
because it's just long and noisy already.

First, I think that only I some developers would actually use
this information. Many of them either know what module was
used or they do not have an easy way to get the debugging
information by the ID. So, it should be configurable
at minimum.

It should be configurable because it isn't used or is hard to use?
Wouldn't a config variable limit the uptake of this information and
further reinforce the fact that nobody will use it? If we always exposed
it then maybe we would get new users. I imagine that we could have a
robot search crash reports and find the "crashiest" places in the kernel
all the way down to the line level and poke kernel developers to look
and see why it crashes so much there. If the information is behind a
config then the benefit of that analysis will be limited.

Second, I am not sure that it should be part of each OOps/panic blob.
One solution would be to print the ID by the module loader when
the module gets loaded. It would be mentioned earlier in the log
then.

Please no. The kernel log could wrap around before an oops/panic happens
and then the ID would be lost.

Or we could make it available, for example, via /proc. It is a kind
of information that might be gathered later on a rebooted system.
SUSE has "supportconfig" command that allows to gather a lot
of similar information about the system. We use it when
analyzing crashdumps and kernel bugs in general.

Please no. The build ID is already available via
/sys/module/<modulename>/sections/<sectionname> and /sys/kernel/notes
(for vmlinux) but that won't help for panics that reboot. If a panic
happens and a new kernel is booted then post processing on the modules
and vmlinux could all be incorrect. Furthermore, the modules will have
to be found and parsed by some userspace crash processing tool after the
reboot.

I'd really prefer to rely on the standard build ID vs. a per-distro
bespoke solution to finding the information about the binaries the
kernel was running. It's just simpler to not need to find out this sort
of information about the system when the kernel knows what binary is
running already. This is the same reason coredump_filter exists to let
coredumping code figure out the build ID of the process that crashes vs.
connecting that to some system information daemon.
 

Anyway, I consider this a very detailed information that is not
suitable for everyone. It should be availabe on request, like
for example, backtraces from all CPUs, list of tasks, memory info.

I suppose I can make a config option for this module printing stuff in
the "Modules linked in:" line. Then we can remove it if most distros
decide to enable it? I'm slightly disappointed that it can't just be
printed all the time for every stacktrace but if there isn't opposition
if it's all behind a config option then I will be happy to get 99% of
the code upstream.