On Tue, Jan 25, 2011 at 03:09:48PM +0100, Ingo Molnar wrote:

* Ahmed S. Darwish [off-list ref] wrote:

quoted

Hi,

I've faced some very early panics in latest kernel. Being a run of the mill
x86 laptop, the machine is void of debugging aids like serial ports or
network boot.

As a possible solution, below patches prototypes the idea of persistently
storing the kernel log ring to a hard disk partition using the enhanced BIOS
0x13 services.

[...]

quoted

Diffstat:

 arch/x86/kernel/saveoops-rmode.S |  483 ++++++++++++++++++++++++++++++++++++++
 arch/x86/include/asm/saveoops.h  |   15 ++
 arch/x86/kernel/saveoops.c       |  219 +++++++++++++++++
 arch/x86/kernel/setup.c          |    9 +
 arch/x86/kernel/Makefile         |    3 +
 lib/Kconfig.debug                |   15 ++
 6 files changed, 744 insertions(+), 0 deletions(-)

Ok, i have to admit that while i'm a rabid BIOS-hater i find this debug feature very 
very interesting, for the plain reason that if it's implemented in a robust and 
clever way then this has a chance to improve debuggability of pretty much any Linux 
laptop quite enormously!

While we generally thoroughly hate BIOSes from beginning to end, one thing can be 
said, a BIOS bootstraps very early during bootup, and it's relatively simple to 
trigger as well.

Yes, the BIOS subset used is the same one used by SYSLINUX, grub, etc to load
the kernel. If the kernel is on, these functions are hopefully quite stable.

The complete __roadblock__ I'm currently facing though is restoring the disk
controllers to the state originally setup by the BIOS Power-on self-test (POST).
I hope such re-initialization is even technically feasible.

Without such re-initialization, we'll just be risking the BIOS code exploding.
That was the case in the 5-minute hang described in the cover sheet (PATCH #0).

Also, since latest kernels do not stomp on BIOS data structures anymore (low RAM), 
there's some good chance it's still functional at the point of crash - be that an 
early crash or a later crash.

Yes, I've noticed that thankfully the kernel reserves the BIOS EBDA area. I'm
not sure though if it reserves the real-mode vector table area 0x0 -> 0x400?

I think the biggest areas of practical concern would be:

 - Can this mechanism ever, under any circumstance corrupt any real data, destroy 
   the MBR or do other nasties. Can you think of any additional fail-safe measures 
   where you could _further robustify the BIOS calls_ to make sure it can never go 
   to the wrong sector(s)? I really do not want to think of trusting a BIOS to 
   _write to my disk_.

Admittedly, I was quite worried while testing this on disk. As a possible
safety trigger, I'm thinking of coding a small script under "tools/" that will
write a cookie in the desired area. If the real-mode code did not find such
cookie in the expected place, it abandons the write operation.

 - Is there some hidden disk area somewhere on PCs, or somewhere on a real partition
   on typical Linux distributions, which we could use without having to reinstall
   the box? This would increase utility and availability enormously. I'm thinking of 
   partition _ends_ which sometimes get rounded in an awkward way and which are 
   potentially skipped by most Linux filesystems. Even a very small, 512 bytes of 
   area would be extremely useful for debugging weird suspend hangs ...

I did not have to re-partition the box here. A kindof a hacky solution was
disabling the swap partition and using it for storing the log. That would make
the feature available without re-installing the box, at the cost of temporarily
disabling swap.

Another testing method was booting the kernel from a USB stick and writing the
log there. If there are other free places in the boot disk, it can be used. But
I'm afraid it will make saveoops much more dangerous than what it already is
(by virtue of being much closer to file-system structures and such).

 - Could we automate the recovery of the dump, and just put it into the regular 
   kernel log on the next (successful) bootup (on a feature-enabled kernel)? That 
   would make the log of the 'previous crash' very conveniently available in dmesg 
   and the syslog. Tools like kerneloops could make use of it immediately.

If I can make the code reliably work, it can be very very useful in a huge
number of situations such as these and more. The catch is the "If" part :)

All in one, a very intriguing idea IMO, and the hardest bits (lowlevel x86 
transition) is all implemented already.

There is a place under arch/x86 that already does the lmode->rmode transition?

That was (so far) the hardest part; I quite searched the tree but did not find
any, so I rolled my own lmode->rmode switch code in the file "saveoops-rmode.S"

	Ingo

Thanks,

-- 
Darwish
http://darwish.07.googlepages.com