Hello all,
one of my drivers is causing a kernel panic and I _think_ it happens in the 1st call to the interrupt routine.
What kind of information can I extract from the following ?
Is it like a core dump that I can load with the executable in the debugger to know exactly what happened (I doubt it) ?
Oops: Exception in kernel mode, sig: 4 [#1]
Xilinx Virtex
last sysfs file:
Modules linked in: xad
NIP: c0002328 LR: c0011de8 CTR: c001d77c
REGS: c778de20 TRAP: 0700 Not tainted (2.6.34)
MSR: 00021030 <ME,CE,IR,DR> CR: 24000044 XER: 00000000
TASK = c6ce80a0[241] 'SoftNoy' THREAD: c778c000
GPR00: 00000000 c778ded0 c6ce80a0 00000026 c6dbe000 00000000 e146dcab 00000000
GPR08: 02134be0 00000000 000020e7 00000001 000020e6 100265d8 00000000 1007c600
GPR16: 100acd0c 100822e4 1009024d bfa39a48 c7452080 c05bf0e8 c05bf02c c0207d6c
GPR24: c778c03c 00000004 c6cc7040 c05c1b88 00000001 00000004 c6cc73c0 00000026
NIP [c0002328] set_context+0x0/0x10
LR [c0011de8] switch_mmu_context+0x194/0x1b8
Call Trace:
[c778ded0] [c001a810] pick_next_task_fair+0xec/0x130 (unreliable)
[c778def0] [c0203514] schedule+0x300/0x394
[c778df40] [c000f63c] recheck+0x0/0x24
Instruction dump:
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 <00000000> 00000000 00000000 00000000
Kernel stack overflow in process c6ce80a0, r1=c778c070
NIP: c000d270 LR: c000f3c8 CTR: c0017fd0
REGS: c778bfc0 TRAP: 0501 Tainted: G D (2.6.34)
MSR: 00029030 <EE,ME,CE,IR,DR> CR: 24000048 XER: 00000000
TASK = c6ce80a0[241] 'SoftNoy' THREAD: c778c000
GPR00: 00029030 c778c070 c6ce80a0 c778c090 08000000 ffff32d8 00000001 00000001
GPR08: ffff32da 00000000 00021032 c000d110 06ce82a8
NIP [c000d270] program_check_exception+0x160/0x228
LR [c000f3c8] ret_from_except_full+0x0/0x4c
Call Trace:
Instruction dump:
38090004 901f0080 480000d8 3ca00003 7fe4fb78 80df0080 60a50001 38600005
480000a8 7c0000a6 60008000 7c000124 <77c00c04> 41a20068 4bffef89 2f83fff2
Kernel panic - not syncing: kernel stack overflow
Call Trace:
Rebooting in 180 seconds..
My driver is xad.ko, though /dev/xps-acqui-data. The user program is SoftNoy.
The code for the ISR (note that this code works fine on the same driver for a slightly different piece of custom
hardware):
static irqreturn_t XadIsr(int irq, void *dev_id) {
Xad.control_reg->fin_in = 0;
Xad.interrupt_reg->ISR = 1;
Xad.interrupt_IPIF_reg->ISR = 4;
Xad.control_reg->flux_address[0] = BUFFER_PHY_BASE + BUF_SZ*(++Xad.Icnt % BUF_NB);
Xad.control_reg->flux_address[1] = Xad.control_reg->flux_address[0] + BUF_SZ/2;
if (Xad.Icnt<Xad.Rcnt+BUF_NB)
Xad.control_reg->flux_start=255; // Arm the next interrupt
else {
// There aren't any buffers available for the next read. We'll do the start in the read routine
Xad.Suspended=1;
Xad.OverflowsSinceLastRead++;
Xad.Overflow++;
DBG_ADD_CHAR('*');
if (Verbose) printk(KERN_WARNING SD "%dth buffer overflow: %d-%d=%d>=%d\n" FL,
Xad.Overflow, Xad.Icnt, Xad.Rcnt, Xad.Icnt-Xad.Rcnt, BUF_NB);
}
wake_up_interruptible(&Xad.wait);
return IRQ_HANDLED;
}
--
Guillaume Dargaud
http://www.gdargaud.net/
Hello all,
one of my drivers is causing a kernel panic and I _think_ it happens in the
1st call to the interrupt routine. What kind of information can I extract
from the following ?
Is it like a core dump that I can load with the executable in the debugger
to know exactly what happened (I doubt it) ?
Oops: Exception in kernel mode, sig: 4 [#1]
Xilinx Virtex
last sysfs file:
Modules linked in: xad
NIP: c0002328 LR: c0011de8 CTR: c001d77c
REGS: c778de20 TRAP: 0700 Not tainted (2.6.34)
MSR: 00021030 <ME,CE,IR,DR> CR: 24000044 XER: 00000000
TASK = c6ce80a0[241] 'SoftNoy' THREAD: c778c000
GPR00: 00000000 c778ded0 c6ce80a0 00000026 c6dbe000 00000000 e146dcab
00000000 GPR08: 02134be0 00000000 000020e7 00000001 000020e6 100265d8
00000000 1007c600 GPR16: 100acd0c 100822e4 1009024d bfa39a48 c7452080
c05bf0e8 c05bf02c c0207d6c GPR24: c778c03c 00000004 c6cc7040 c05c1b88
00000001 00000004 c6cc73c0 00000026 NIP [c0002328] set_context+0x0/0x10
LR [c0011de8] switch_mmu_context+0x194/0x1b8
Call Trace:
[c778ded0] [c001a810] pick_next_task_fair+0xec/0x130 (unreliable)
[c778def0] [c0203514] schedule+0x300/0x394
[c778df40] [c000f63c] recheck+0x0/0x24
Instruction dump:
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 <00000000> 00000000 00000000 00000000
Kernel stack overflow in process c6ce80a0, r1=c778c070
NIP: c000d270 LR: c000f3c8 CTR: c0017fd0
REGS: c778bfc0 TRAP: 0501 Tainted: G D (2.6.34)
MSR: 00029030 <EE,ME,CE,IR,DR> CR: 24000048 XER: 00000000
TASK = c6ce80a0[241] 'SoftNoy' THREAD: c778c000
GPR00: 00029030 c778c070 c6ce80a0 c778c090 08000000 ffff32d8 00000001
00000001 GPR08: ffff32da 00000000 00021032 c000d110 06ce82a8
NIP [c000d270] program_check_exception+0x160/0x228
LR [c000f3c8] ret_from_except_full+0x0/0x4c
Call Trace:
Instruction dump:
38090004 901f0080 480000d8 3ca00003 7fe4fb78 80df0080 60a50001 38600005
480000a8 7c0000a6 60008000 7c000124 <77c00c04> 41a20068 4bffef89 2f83fff2
Kernel panic - not syncing: kernel stack overflow
Call Trace:
Rebooting in 180 seconds..
My driver is xad.ko, though /dev/xps-acqui-data. The user program is
SoftNoy. The code for the ISR (note that this code works fine on the same
driver for a slightly different piece of custom hardware):
static irqreturn_t XadIsr(int irq, void *dev_id) {
Xad.control_reg->fin_in = 0;
Xad.interrupt_reg->ISR = 1;
Xad.interrupt_IPIF_reg->ISR = 4;
Xad.control_reg->flux_address[0] = BUFFER_PHY_BASE + BUF_SZ*(++Xad.Icnt %
BUF_NB); Xad.control_reg->flux_address[1] =
Xad.control_reg->flux_address[0] + BUF_SZ/2;
if (Xad.Icnt<Xad.Rcnt+BUF_NB)
Xad.control_reg->flux_start=255; // Arm the next interrupt
else {
// There aren't any buffers available for the next read. We'll do the
start in the read routine Xad.Suspended=1;
Xad.OverflowsSinceLastRead++;
Xad.Overflow++;
DBG_ADD_CHAR('*');
if (Verbose) printk(KERN_WARNING SD "%dth buffer overflow:
%d-%d=%d>=%d\n" FL, Xad.Overflow, Xad.Icnt, Xad.Rcnt, Xad.Icnt-Xad.Rcnt,
BUF_NB);
}
wake_up_interruptible(&Xad.wait);
return IRQ_HANDLED;
}
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2011-07-07 22:58:26
On Tue, 2011-07-05 at 16:19 +0200, Guillaume Dargaud wrote:
Hello all,
one of my drivers is causing a kernel panic and I _think_ it happens in the 1st call to the interrupt routine.
What kind of information can I extract from the following ?
Is it like a core dump that I can load with the executable in the debugger to know exactly what happened (I doubt it) ?
Kernel stack overflow in process c6ce80a0, r1=c778c070
That's bad...
NIP: c000d270 LR: c000f3c8 CTR: c0017fd0
REGS: c778bfc0 TRAP: 0501 Tainted: G D (2.6.34)
MSR: 00029030 <EE,ME,CE,IR,DR> CR: 24000048 XER: 00000000
TASK = c6ce80a0[241] 'SoftNoy' THREAD: c778c000
GPR00: 00029030 c778c070 c6ce80a0 c778c090 08000000 ffff32d8 00000001 00000001
GPR08: ffff32da 00000000 00021032 c000d110 06ce82a8
NIP [c000d270] program_check_exception+0x160/0x228
LR [c000f3c8] ret_from_except_full+0x0/0x4c
Call Trace:
Instruction dump:
38090004 901f0080 480000d8 3ca00003 7fe4fb78 80df0080 60a50001 38600005
480000a8 7c0000a6 60008000 7c000124 <77c00c04> 41a20068 4bffef89 2f83fff2
Kernel panic - not syncing: kernel stack overflow
Call Trace:
Rebooting in 180 seconds..
My driver is xad.ko, though /dev/xps-acqui-data. The user program is SoftNoy.
The code for the ISR (note that this code works fine on the same driver for a slightly different piece of custom
hardware):
static irqreturn_t XadIsr(int irq, void *dev_id) {
Xad.control_reg->fin_in = 0;
Xad.interrupt_reg->ISR = 1;
Xad.interrupt_IPIF_reg->ISR = 4;
Xad.control_reg->flux_address[0] = BUFFER_PHY_BASE + BUF_SZ*(++Xad.Icnt % BUF_NB);
Xad.control_reg->flux_address[1] = Xad.control_reg->flux_address[0] + BUF_SZ/2;
if (Xad.Icnt<Xad.Rcnt+BUF_NB)
Xad.control_reg->flux_start=255; // Arm the next interrupt
else {
// There aren't any buffers available for the next read. We'll do the start in the read routine
Xad.Suspended=1;
Xad.OverflowsSinceLastRead++;
Xad.Overflow++;
DBG_ADD_CHAR('*');
if (Verbose) printk(KERN_WARNING SD "%dth buffer overflow: %d-%d=%d>=%d\n" FL,
Xad.Overflow, Xad.Icnt, Xad.Rcnt, Xad.Icnt-Xad.Rcnt, BUF_NB);
}
wake_up_interruptible(&Xad.wait);
return IRQ_HANDLED;
}
What is "Xad." ? (btw, coding style FAIL !)
Are you trying to write to HW registers using a structure like that
without using the appropriate MMIO register accessors ?
In that case, your accesses may happen our of order since you don't have
memory barriers (among other potential problems).
The crash looks like you aren't properly clearing the interrupt
condition on the HW, it remains asserted, tho it shouldn't overflow like
that, something seems wrong with your PIC.
What HW is this ? What PIC ? It looks like the interrupt source isn't
masked on the PIC itself while it's being handled or something...
Cheers,
Ben.
That's the struct I use to access the control registers of the hardware.
About the coding style, don't worry it's never going to make it into mainstream as there's only one piece of that
hardware ever built ! (which is also why I didn't respect things like allowing multiple devices, please don't nail me to
the cross for that). And it's only my 2nd real Linux driver...
Are you trying to write to HW registers using a structure like that
without using the appropriate MMIO register accessors ?
In that case, your accesses may happen our of order since you don't have
memory barriers (among other potential problems).
Yes. I discovered the out() functions afterwards. But I insert asm(eieio) to avoid 'out of order' problems.
The crash looks like you aren't properly clearing the interrupt
condition on the HW, it remains asserted, tho it shouldn't overflow like
that, something seems wrong with your PIC.
Is there some constraints I should tell the electronics guys ? Should the interrupt be raised for less than some max
duration ? It's on a raising signal, so I don't see why that should be an issue.
What HW is this ? What PIC ? It looks like the interrupt source isn't
masked on the PIC itself while it's being handled or something...
The hardware is a heavily modified Xilinx ML405 derivative.
The PIC is a XPS_INTC (in VHDL)
--
Guillaume Dargaud
http://www.gdargaud.net/Antarctica/
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2011-07-09 23:16:44
On Fri, 2011-07-08 at 09:26 +0200, Guillaume Dargaud wrote:
quoted
What is "Xad." ? (btw, coding style FAIL !)
That's the struct I use to access the control registers of the hardware.
About the coding style, don't worry it's never going to make it into mainstream as there's only one piece of that
hardware ever built ! (which is also why I didn't respect things like allowing multiple devices, please don't nail me to
the cross for that). And it's only my 2nd real Linux driver...
quoted
Are you trying to write to HW registers using a structure like that
without using the appropriate MMIO register accessors ?
In that case, your accesses may happen our of order since you don't have
memory barriers (among other potential problems).
Yes. I discovered the out() functions afterwards. But I insert asm(eieio) to avoid 'out of order' problems.
Yeah well, you may have the compiler playing tricks too. Use
{read,write}{b,w,l} instead, or the _be variants to avoid byteswap.
quoted
The crash looks like you aren't properly clearing the interrupt
condition on the HW, it remains asserted, tho it shouldn't overflow like
that, something seems wrong with your PIC.
Is there some constraints I should tell the electronics guys ? Should the interrupt be raised for less than some max
duration ? It's on a raising signal, so I don't see why that should be an issue.
What do you mean by "raising signal" ? It's meant to be positive edge
sensitive ? Maybe that's your problem, ie, maybe you haven't configued
the interrupt controller for edge trigger but for level trigger
instead ?
quoted
What HW is this ? What PIC ? It looks like the interrupt source isn't
masked on the PIC itself while it's being handled or something...
The hardware is a heavily modified Xilinx ML405 derivative.
The PIC is a XPS_INTC (in VHDL)
Ok, I'm not familiar with that PIC. You need to check what's going on
between the PIC, your interrupt source and the kernel.
Normally, if it's an edge interrupt, it's a single event that gets
latched by the PIC. The kernel will then call ack() on that PIC driver
(irq_chip) which should clear that latch -before- getting into your
device driver for processing.
Also, the interrupt shall either be masked while processing or if it
re-enters, the PIC code shall try to mask it (lazy masking) until the
original handler completes at which point it gets unmasked. That shall
be handled by the standard flow handlers, so it really depends on how
you hookup your PIC in SW.
It looks like one of these things isn't happening, but it's hard to tell
without seeing more of the code & vhdl
Cheers,
Ben.
Ok, I'm not familiar with that PIC. You need to check what's going on
between the PIC, your interrupt source and the kernel.
Normally, if it's an edge interrupt, it's a single event that gets
latched by the PIC. The kernel will then call ack() on that PIC driver
(irq_chip) which should clear that latch -before- getting into your
device driver for processing.
Also, the interrupt shall either be masked while processing or if it
re-enters, the PIC code shall try to mask it (lazy masking) until the
original handler completes at which point it gets unmasked. That shall
be handled by the standard flow handlers, so it really depends on how
you hookup your PIC in SW.
This should be all this:
static int xad_driver_probe(struct of_device* dev, const struct of_device_id *match) {
struct device_node *dn = dev->node;
Xad.irq = irq_of_parse_and_map(dn, 0);
rc=request_irq(Xad.irq, XadIsr, IRQF_TRIGGER_RISING | IRQF_DISABLED | IRQF_SHARED /*| IRQF_SAMPLE_RANDOM*/,
"XadIsr", &Xad);
IIRC IRQF_DISABLED is obsolete (I've tried without).
What mystifies me is that:
- my same code on slightly different hardware works perfectly (the differences are not relevant to the driver but to the
user application).
- a simplified standalone code works (so, non-linux).
--
Guillaume Dargaud
http://www.gdargaud.net/