From: Joe Korty <hidden> Date: 2007-07-26 18:17:22
Create /proc/all-interrupts for some architectures.
Create a version of /proc/interrupts that displays _every_
IRQ vector, not just those that someone thought might be
interesting, and add an entry in the commentary column
for those vectors which lacked such a comment.
Rationale: /proc/interrupts is not truly useful unless it
displays every IRQ vector, not just those somebody thought
would be interesting. For example, since /proc/interrupts
does not display the rescheduling interrupt, the occurance
of rescheduling interrupt floods ends up affecting
latencies, yet without an entry in /proc/interrupts, it
is difficult to discern why latencies are being affected.
Rather than modify /proc/interrupts, this patch creates
a new version of /proc/interrupts, on the off-chance
that adding new lines to /proc/interrupts, and appending
new fields to the end of old lines, might break some
longstanding script. However, these kinds of changes
traditionally do not affect scripts, so it might be
reasonable to fold /proc/all-interrupts back into
/proc/interrupts.
Signed-off-by: Joe Korty <redacted>
Index: 2.6.22.1-rt8/arch/i386/kernel/apic.c
===================================================================
@@ -1297,7 +1299,7 @@apic_write(APIC_ESR,0);v1=apic_read(APIC_ESR);ack_APIC_irq();-atomic_inc(&irq_err_count);+atomic_inc(&__get_cpu_var(irq_err_counts));/* Here is what the APIC error bits mean:0:SendCSerror
@@ -51,7 +51,6 @@#include"io_ports.h"int(*ioapic_renumber_irq)(intioapic,intirq);-atomic_tirq_mis_count;/* Where if anywhere is the i8259 connect in external int mode */staticstruct{intpin,apic;}ioapic_i8259={-1,-1};
@@ -1127,7 +1130,7 @@apic_write(APIC_ESR,0);v1=apic_read(APIC_ESR);ack_APIC_irq();-atomic_inc(&irq_err_count);+atomic_inc(&__get_cpu_var(irq_err_counts));/* Here is what the APIC error bits mean:0:SendCSerror
@@ -295,7 +295,15 @@ the IO-APIC automatically retry the transmission, so it should not be a big problem, but you should read the SMP-FAQ.-In this context it could be interesting to note the new irq directory in 2.4.+In 2.6.* a new file, /proc/all-interrupts appears on some architectures.+This adds to the /proc/interrupts output all the system interrupt vectors+that /proc/interrupts missed, and it fills in the commentary field for each+system vector that lacked it. This added functionality was made a seperate+file, rather than merged into /proc/interrupts, as /proc/interrupts has+not changed in a long time, thus raising the specter that _any_ change+might break ancient, longstanding scripts and programs.++An interesting addition to 2.4 is the new irq directory. It could be used to set IRQ to CPU affinity, this means that you can "hook" an IRQ to only one CPU, or to exclude a CPU of handling IRQs. The contents of the irq subdir is one subdir for each IRQ, and one file; prof_cpu_mask
From: "H. Peter Anvin" <hpa@zytor.com> Date: 2007-07-26 18:58:34
Joe Korty wrote:
Create /proc/all-interrupts for some architectures.
Create a version of /proc/interrupts that displays _every_
IRQ vector, not just those that someone thought might be
interesting, and add an entry in the commentary column
for those vectors which lacked such a comment.
Rationale: /proc/interrupts is not truly useful unless it
displays every IRQ vector, not just those somebody thought
would be interesting. For example, since /proc/interrupts
does not display the rescheduling interrupt, the occurance
of rescheduling interrupt floods ends up affecting
latencies, yet without an entry in /proc/interrupts, it
is difficult to discern why latencies are being affected.
Rather than modify /proc/interrupts, this patch creates
a new version of /proc/interrupts, on the off-chance
that adding new lines to /proc/interrupts, and appending
new fields to the end of old lines, might break some
longstanding script. However, these kinds of changes
traditionally do not affect scripts, so it might be
reasonable to fold /proc/all-interrupts back into
/proc/interrupts.
I think that would be the right thing to do. We have added things to
/proc/interrupts in the past.
-hpa
From: Sven-Thorsten Dietrich <hidden> Date: 2007-07-30 17:33:41
On Thu, 2007-07-26 at 11:56 -0700, H. Peter Anvin wrote:
Joe Korty wrote:
quoted
Create /proc/all-interrupts for some architectures.
Create a version of /proc/interrupts that displays _every_
IRQ vector, not just those that someone thought might be
interesting, and add an entry in the commentary column
for those vectors which lacked such a comment.
Rationale: /proc/interrupts is not truly useful unless it
displays every IRQ vector, not just those somebody thought
would be interesting. For example, since /proc/interrupts
does not display the rescheduling interrupt, the occurance
of rescheduling interrupt floods ends up affecting
latencies, yet without an entry in /proc/interrupts, it
is difficult to discern why latencies are being affected.
Rather than modify /proc/interrupts, this patch creates
a new version of /proc/interrupts, on the off-chance
that adding new lines to /proc/interrupts, and appending
new fields to the end of old lines, might break some
longstanding script. However, these kinds of changes
traditionally do not affect scripts, so it might be
reasonable to fold /proc/all-interrupts back into
/proc/interrupts.
I think that would be the right thing to do. We have added things to
/proc/interrupts in the past.
Hi Andrew,
Would it make sense to drop this patch into -mm for feedback?
Thanks,
Sven
From: Andrew Morton <akpm@linux-foundation.org> Date: 2007-07-30 19:33:53
On Mon, 30 Jul 2007 10:33:17 -0700
Sven-Thorsten Dietrich [off-list ref] wrote:
On Thu, 2007-07-26 at 11:56 -0700, H. Peter Anvin wrote:
quoted
Joe Korty wrote:
quoted
Create /proc/all-interrupts for some architectures.
Create a version of /proc/interrupts that displays _every_
IRQ vector, not just those that someone thought might be
interesting, and add an entry in the commentary column
for those vectors which lacked such a comment.
Rationale: /proc/interrupts is not truly useful unless it
displays every IRQ vector, not just those somebody thought
would be interesting. For example, since /proc/interrupts
does not display the rescheduling interrupt, the occurance
of rescheduling interrupt floods ends up affecting
latencies, yet without an entry in /proc/interrupts, it
is difficult to discern why latencies are being affected.
Rather than modify /proc/interrupts, this patch creates
a new version of /proc/interrupts, on the off-chance
that adding new lines to /proc/interrupts, and appending
new fields to the end of old lines, might break some
longstanding script. However, these kinds of changes
traditionally do not affect scripts, so it might be
reasonable to fold /proc/all-interrupts back into
/proc/interrupts.
I think that would be the right thing to do. We have added things to
/proc/interrupts in the past.
Hi Andrew,
Would it make sense to drop this patch into -mm for feedback?
It's a lot of code for something which might be useful to someone sometime.
It's a bit of a crappy changelog too. I'd at least like to see a list of
all the new fields.
It should be OK to add new lines to /proc/interrupts? That file varies a
lot between machines adn between architectures - as long as the new lines
have similar layout it is unlikely that anything will break.
+ atomic_inc(&__get_cpu_var(irq_thermal_counts));
The patch does atomic ops on cpu-local variables. This isn't needed, and
is expensive.
If the field is only ever modified from hard interrupt context then you can
make the field unsigned long and use plain old `foo++'.
If the field is modified from both hard-IRQ and from non-IRQ then use a
local_t and local_inc.
Or even, given that this is just a statistic and grrat precision is not
needed, use unsigned long and f++ even if that _is_ racy. Because the
consequences of a race will just be a single lost count, which we dont'
care about enough to add the additional overhead of an atomic op.
But I'm a bit dubious about the whole thing, really. As I said, it's a lot
of code. More justfication is needed for its addition, I'd suggest.
From: Joe Korty <hidden> Date: 2007-07-30 19:43:43
On Mon, Jul 30, 2007 at 12:32:06PM -0700, Andrew Morton wrote:
On Mon, 30 Jul 2007 10:33:17 -0700
Sven-Thorsten Dietrich [off-list ref] wrote:
quoted
On Thu, 2007-07-26 at 11:56 -0700, H. Peter Anvin wrote:
quoted
Joe Korty wrote:
quoted
Create /proc/all-interrupts for some architectures.
quoted
Would it make sense to drop this patch into -mm for feedback?
It's a lot of code for something which might be useful to someone sometime.
It's a bit of a crappy changelog too. I'd at least like to see a list of
all the new fields.
It should be OK to add new lines to /proc/interrupts? That file varies a
lot between machines adn between architectures - as long as the new lines
have similar layout it is unlikely that anything will break.
+ atomic_inc(&__get_cpu_var(irq_thermal_counts));
The patch does atomic ops on cpu-local variables. This isn't needed, and
is expensive.
If the field is only ever modified from hard interrupt context then you can
make the field unsigned long and use plain old `foo++'.
If the field is modified from both hard-IRQ and from non-IRQ then use a
local_t and local_inc.
Or even, given that this is just a statistic and grrat precision is not
needed, use unsigned long and f++ even if that _is_ racy. Because the
consequences of a race will just be a single lost count, which we dont'
care about enough to add the additional overhead of an atomic op.
Hi Andrew,
Thanks for the comments. I'll, at least, make the changes you suggested.
(the /proc/interrupts version has the benefit of being smaller too).
Joe
From: Joe Korty <hidden> Date: 2007-07-31 00:18:28
Add missing IRQs and IRQ descriptions to /proc/interrupts.
/proc/interrupts is most useful when it displays every
IRQ vector in use by the system, not just those somebody
thought would be interesting.
This patch inserts the following vector displays to the
i386 and x86_64 platforms, as appropriate:
rescheduling interrupts
TLB flush interrupts
function call interrupts
thermal event interrupts
threshold interrupts
spurious interrupts
A threshold interrupt occurs when ECC memory correction
is occuring at too high a frequency. Thresholds are used
by the ECC hardware as occasional ECC failures are part
of normal operation, but long sequences of ECC failures
usually indicate a memory chip that is about to fail.
Thermal event interrupts occur when a temperature threshold
has been exceeded for some CPU chip. IIRC, a thermal
interrupt is also generated when the temperature drops
back to a normal level.
A spurious interrupt is an interrupt that was raised then
lowered by the device before it could be fully processed
by the APIC. Hence the apic sees the interrupt but does
not know what device it came from. For this case the APIC
hardware will assume a vector of 0xff.
Rescheduling, call, and TLB flush interrupts are sent from
one CPU to another per the needs of the OS. Typically,
their statistics would be used to discover if an
interrupt flood of the given type has been occuring.
Signed-off-by: Joe Korty <redacted>
Index: 2.6.23-rc1-git7/arch/i386/kernel/apic.c
===================================================================
From: Joe Korty <hidden> Date: 2007-07-31 14:21:07
[ changes from v2:
added documentation
merged some #ifdef CONFIG_SMP's
]
Add missing IRQs and IRQ descriptions to /proc/interrupts.
/proc/interrupts is most useful when it displays every
IRQ vector in use by the system, not just those somebody
thought would be interesting.
This patch inserts the following vector displays to the
i386 and x86_64 platforms, as appropriate:
rescheduling interrupts
TLB flush interrupts
function call interrupts
thermal event interrupts
threshold interrupts
spurious interrupts
A threshold interrupt occurs when ECC memory correction
is occuring at too high a frequency. Thresholds are used
by the ECC hardware as occasional ECC failures are part
of normal operation, but long sequences of ECC failures
usually indicate a memory chip that is about to fail.
Thermal event interrupts occur when a temperature threshold
has been exceeded for some CPU chip. IIRC, thermal interrupts
can also be generated when the temperature drops back to
the normal range.
A spurious interrupt is an interrupt that was raised then
lowered by the device before it could be fully processed
by the APIC. Hence the apic sees the interrupt but does
not know what device it came from. For this case the APIC
hardware will assume a vector of 0xff.
Rescheduling, call, and TLB flush interrupts are sent from
one CPU to another per the needs of the OS. Typically,
their statistics would be used to discover interrupt
flooding.
Signed-off-by: Joe Korty <redacted>
Index: 2.6.23-rc1-git7/arch/i386/kernel/apic.c
===================================================================
@@ -347,7 +347,37 @@ the IO-APIC automatically retry the transmission, so it should not be a big problem, but you should read the SMP-FAQ.-In this context it could be interesting to note the new irq directory in 2.4.+In 2.6.2* /proc/interrupts was expanded again. This time the goal was for+/proc/interrupts to display every IRQ vector in use by the system, not+just those considered 'most important'. The new vectors are:++ THR -- a threshold interrupt occurs when ECC memory correction is+ occuring at too high a frequency. Threshold interrupt machinery is+ often put into the ECC logic, as occasional ECC memory failures are+ part of normal memory operation, but sequences of ECC failures over+ some short interval usually indicate a memory chip that is about to fail.++ TRM -- a thermal event interrupt occurs when a temperature threshold+ has been exceeded for some CPU chip. This interrupt may also be generated+ when the temperature drops back to normal.++ SPU -- a spurious interrupt is some interrupt that was raised then lowered+ by some IO device before it could be fully processed by the APIC. Hence+ the APIC sees the interrupt but does not know what device it came from.+ For this case the APIC will generate the interrupt with a IRQ vector+ of 0xff.++ RES, CAL, TLB -- rescheduling, call and tlb flush interrupts are+ sent from one CPU to another per the needs of the OS. Typically,+ their statistics are used by kernel developers and interested users to+ determine the occurance of interrupt floods of the given type.++The above IRQ vectors are displayed only when relevent. For example,+the threshold vector does not exist on x86_64 platforms. Others are+suppressed when the system is a uniprocessor. As of this writing, only+i386 and x86_64 platforms support the new IRQ vector displays.++Of some interest is the introduction of the /proc/irq directory to 2.4. It could be used to set IRQ to CPU affinity, this means that you can "hook" an IRQ to only one CPU, or to exclude a CPU of handling IRQs. The contents of the irq subdir is one subdir for each IRQ, and one file; prof_cpu_mask
I think it would be better to use 5-6 char identifiers even
when it whacks the columns a bit; otherwise
nobody will know what it means. e.g. SCHED here.
Also there you should update proc(5) and send a patch
to the manpage maintainer.
From: Joe Korty <hidden> Date: 2007-07-31 20:34:00
On Tue, Jul 31, 2007 at 07:02:01PM +0200, Andi Kleen wrote:
Hi Andi,
Thanks for the review. I implemented many of your suggestions and for
the rest, here mention why not, in case you want to respond further.
Regards,
Joe
Joe Korty [off-list ref] writes:
quoted
A threshold interrupt occurs when ECC memory correction
is occuring at too high a frequency.
It's configurable and the default is off. Also
it's only on AMD hardware.
v4 now has a comment to the Documentation section noting
this.
quoted
Thresholds are used
by the ECC hardware as occasional ECC failures are part
of normal operation,
Occasional ECC _corrections_ are normal (due to stray alpha particles)
but ECC _failures_ are not. Document corrected.
Wouldn't it be safer on preemptible kernels to have that inside
the irq_exit?
Although irq_exit() releases the preemption block, it doesn't seem to
release the APIC interrupt block, at least for i386. And as an interrupt
block also blocks preemption and process migration, it seems that it would
be safe to do the increments after the irq_exit(). But I've moved them
all inside in v4, just in case I am wrong, or this changes in the future
(eg, PREEMPT_RT).
quoted
+ seq_printf(p, "RES: ");
I think it would be better to use 5-6 char identifiers
even when it whacks the columns a bit; otherwise nobody
will know what it means. e.g. SCHED here.
v3 addresses this. The normally empty 'description' column at the end of
each line now holds a description of each vector. The three-character
line-prefix names are there only to make the new lines match the syntax
and format of the other lines in /proc/interrupts.
Also there you should update proc(5) and send a patch
to the manpage maintainer.
Will do.
Thanks,
Joe
PS: also fixed up the whitespace.
From: Joe Korty <hidden> Date: 2007-07-31 21:08:00
[ v3-v4 changelog:
s/irq_spur_counts/irq_spurious_counts/g (Andrew Morton)
tweaked documentation (Andi Kleen)
moved increments before irq_exit as appropriate (Andi Kleen)
whitespace cleanup (Andi Kleen)
]
Add missing IRQs and IRQ descriptions to /proc/interrupts,
version 4.
/proc/interrupts is most useful when it displays every
IRQ vector in use by the system, not just those somebody
thought would be interesting.
This patch inserts the following vector displays to the
i386 and x86_64 platforms.
rescheduling interrupts
TLB flush interrupts
function call interrupts
thermal event interrupts
threshold interrupts
spurious interrupts
A threshold interrupt occurs when ECC memory correction
is occuring at too high a frequency. Thresholds are used
by the ECC hardware as occasional ECC corrections are part
of normal operation (alpha particles), but long sequences
of ECC corrections usually indicate a memory chip that
is about to fail. Note that not every system has ECC
threshold logic, and those that do, can require it to
be specifically enabled.
Thermal event interrupts occur when a temperature threshold
has been exceeded for some CPU chip. I am not sure,
but I think a thermal interrupt is also generated when
the temperature drops back to a normal level.
A spurious interrupt is an interrupt that was raised then
lowered by the device before it could be fully processed
by the APIC. Hence the apic sees the interrupt but does
not know what device it came from. For this case the APIC
hardware will assume a vector of 0xff.
Rescheduling, call, and TLB flush interrupts are sent from
one CPU to another per the needs of the OS. Typically,
their statistics would be used to discover interrupt
flooding.
Signed-off-by: Joe Korty <redacted>
Index: 2.6.23-rc1-git7/arch/i386/kernel/apic.c
===================================================================
@@ -347,7 +347,40 @@ the IO-APIC automatically retry the transmission, so it should not be a big problem, but you should read the SMP-FAQ.-In this context it could be interesting to note the new irq directory in 2.4.+In 2.6.2* /proc/interrupts was expanded again. This time the goal was for+/proc/interrupts to display every IRQ vector in use by the system, not+just those considered 'most important'. The new vectors are:++ THR -- a threshold interrupt occurs when ECC memory correction is occuring+ at too high a frequency. Threshold interrupt machinery is often put+ into the ECC logic, as occasional ECC memory corrections are part of+ normal operation (due to random alpha particles), but sequences of+ ECC corrections or outright failures over some short interval usually+ indicate a memory chip that is about to fail. Note that not every+ platform has ECC threshold logic, and those that do generally require+ it to be explicitly turned on.++ TRM -- a thermal event interrupt occurs when a temperature threshold+ has been exceeded for some CPU chip. This interrupt may also be generated+ when the temperature drops back to normal.++ SPU -- a spurious interrupt is some interrupt that was raised then lowered+ by some IO device before it could be fully processed by the APIC. Hence+ the APIC sees the interrupt but does not know what device it came from.+ For this case the APIC will generate the interrupt with a IRQ vector+ of 0xff.++ RES, CAL, TLB -- rescheduling, call and tlb flush interrupts are+ sent from one CPU to another per the needs of the OS. Typically,+ their statistics are used by kernel developers and interested users to+ determine the occurance of interrupt floods of the given type.++The above IRQ vectors are displayed only when relevent. For example,+the threshold vector does not exist on x86_64 platforms. Others are+suppressed when the system is a uniprocessor. As of this writing, only+i386 and x86_64 platforms support the new IRQ vector displays.++Of some interest is the introduction of the /proc/irq directory to 2.4. It could be used to set IRQ to CPU affinity, this means that you can "hook" an IRQ to only one CPU, or to exclude a CPU of handling IRQs. The contents of the irq subdir is one subdir for each IRQ, and one file; prof_cpu_mask
From: Lennart Sorensen <hidden> Date: 2007-08-01 13:29:25
On Mon, Jul 30, 2007 at 08:17:18PM -0400, Joe Korty wrote:
Add missing IRQs and IRQ descriptions to /proc/interrupts.
/proc/interrupts is most useful when it displays every
IRQ vector in use by the system, not just those somebody
thought would be interesting.
This patch inserts the following vector displays to the
i386 and x86_64 platforms, as appropriate:
rescheduling interrupts
TLB flush interrupts
function call interrupts
thermal event interrupts
threshold interrupts
spurious interrupts
A threshold interrupt occurs when ECC memory correction
is occuring at too high a frequency. Thresholds are used
by the ECC hardware as occasional ECC failures are part
of normal operation, but long sequences of ECC failures
usually indicate a memory chip that is about to fail.
Thermal event interrupts occur when a temperature threshold
has been exceeded for some CPU chip. IIRC, a thermal
interrupt is also generated when the temperature drops
back to a normal level.
A spurious interrupt is an interrupt that was raised then
lowered by the device before it could be fully processed
by the APIC. Hence the apic sees the interrupt but does
not know what device it came from. For this case the APIC
hardware will assume a vector of 0xff.
Rescheduling, call, and TLB flush interrupts are sent from
one CPU to another per the needs of the OS. Typically,
their statistics would be used to discover if an
interrupt flood of the given type has been occuring.
Since you are playing with /proc/interrupts, is there any chance you
could make it make sense too?
Right now I have the annoying issue that /proc/interrupts only shows
interrupts that it sees a handler for, and ignores the rest. For some
reason irq 3 never shows up in /proc/interrupts on my system, even
though I have serial ports on both IRQ 3 and 4. Only irq 4 shows
'serial' as the handler, and is the only one shown. irq 3 is getting
lots of interrupts, and the serial driver is handling them, it just
doesn't show up in /proc/interrupts. procinfo does show irq 3 although
with no handler listed. I think any interrupt with a handler AND any
interrupt with a non zero count ought to be shown. I only happened to
try procinfo by chance and discover that while I thought IRQ 3 wasn't
working, the only issue was that /proc/interrupts simply wasn't showing
all the relevant information.
Of course I would also love to know why irq 3 isn't showing serial as
the handler, but I have no idea why that is. This is all using Debian's
2.6.18 kernel.
--
Len Sorensen