From: Russell Currey <hidden> Date: 2017-03-03 04:47:31
eeh_handle_special_event() is called when an EEH event is detected but
can't be narrowed down to a specific PE. This function looks through
every PE to find one in an erroneous state, then calls the regular event
handler eeh_handle_normal_event() once it knows which PE has an error.
However, if eeh_handle_normal_event() found that the PE cannot possibly
be recovered, it will remove the PE and associated devices. This leads
to a use after free in eeh_handle_special_event() as it attempts to clear
the "recovering" state on the PE after eeh_handle_normal_event() returns.
Thus, make sure the PE is valid when attempting to clear state in
eeh_handle_special_event().
Cc: <redacted> #3.10+
Reported-by: Alexey Kardashevskiy <redacted>
Signed-off-by: Russell Currey <redacted>
---
arch/powerpc/kernel/eeh_driver.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
@@ -983,6 +983,19 @@ static void eeh_handle_special_event(void)if(rc==EEH_NEXT_ERR_FROZEN_PE||rc==EEH_NEXT_ERR_FENCED_PHB){eeh_handle_normal_event(pe);++/*+*eeh_handle_normal_event()canfreethePEifit+*determinesthatthePEcannotpossiblyberecovered.+*MakesurethePEstillexistsbeforechangingits+*state.+*/+if(!pe||(pe->type&EEH_PE_INVALID)+||(pe->state&EEH_PE_REMOVED)){+pr_warn("EEH: not clearing state on bad PE\n");+continue;+}+eeh_pe_state_clear(pe,EEH_PE_RECOVERING);}else{pci_lock_rescan_remove();
On Fri, Mar 03, 2017 at 03:47:18PM +1100, Russell Currey wrote:
eeh_handle_special_event() is called when an EEH event is detected but
can't be narrowed down to a specific PE. This function looks through
every PE to find one in an erroneous state, then calls the regular event
handler eeh_handle_normal_event() once it knows which PE has an error.
However, if eeh_handle_normal_event() found that the PE cannot possibly
be recovered, it will remove the PE and associated devices. This leads
to a use after free in eeh_handle_special_event() as it attempts to clear
the "recovering" state on the PE after eeh_handle_normal_event() returns.
Thus, make sure the PE is valid when attempting to clear state in
eeh_handle_special_event().
From the changelog, I don't see how the PE is free'd. Could you explain
if (rc == EEH_NEXT_ERR_FROZEN_PE ||
rc == EEH_NEXT_ERR_FENCED_PHB) {
eeh_handle_normal_event(pe);
+
+ /*
+ * eeh_handle_normal_event() can free the PE if it
+ * determines that the PE cannot possibly be recovered.
+ * Make sure the PE still exists before changing its
+ * state.
+ */
+ if (!pe || (pe->type & EEH_PE_INVALID)
+ || (pe->state & EEH_PE_REMOVED)) {
+ pr_warn("EEH: not clearing state on bad PE\n");
+ continue;
+ }
+
It seems not correct. @pe has set to the valid PE in advance, the !pe is
always false? If the PE has been free'd, how can we access @pe->type here
and how can we make sure PE_INVALID and PE_REMOVED flag wasn't overwritten
by somebody else?
eeh_handle_special_event() is called when an EEH event is detected but
can't be narrowed down to a specific PE. This function looks through
every PE to find one in an erroneous state, then calls the regular event
handler eeh_handle_normal_event() once it knows which PE has an error.
However, if eeh_handle_normal_event() found that the PE cannot possibly
be recovered, it will remove the PE and associated devices. This leads
to a use after free in eeh_handle_special_event() as it attempts to clear
the "recovering" state on the PE after eeh_handle_normal_event() returns.
Thus, make sure the PE is valid when attempting to clear state in
eeh_handle_special_event().
Cc: <redacted> #3.10+
Reported-by: Alexey Kardashevskiy <redacted>
Signed-off-by: Russell Currey <redacted>
---
arch/powerpc/kernel/eeh_driver.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
On Fri, Mar 03, 2017 at 03:47:18PM +1100, Russell Currey wrote:
quoted
eeh_handle_special_event() is called when an EEH event is detected but
can't be narrowed down to a specific PE. This function looks through
every PE to find one in an erroneous state, then calls the regular event
handler eeh_handle_normal_event() once it knows which PE has an error.
However, if eeh_handle_normal_event() found that the PE cannot possibly
be recovered, it will remove the PE and associated devices. This leads
to a use after free in eeh_handle_special_event() as it attempts to clear
the "recovering" state on the PE after eeh_handle_normal_event() returns.
Thus, make sure the PE is valid when attempting to clear state in
eeh_handle_special_event().
From the changelog, I don't see how the PE is free'd. Could you explain
a bit about it?
This is a backtrace when kfree(pe) is done:
dump_stack+0xb0/0xf0 (unreliable)
eeh_rmv_from_parent_pe+0x2f8/0x330
eeh_remove_device+0x128/0x170
pcibios_release_device+0x2c/0x70
pci_release_dev+0x5c/0xb0
device_release+0x58/0xf0
kobject_put+0x144/0x2e0
put_device+0x24/0x40
pci_remove_bus_device+0x14c/0x190
pci_hp_remove_devices+0xac/0x170
eeh_handle_normal_event+0x120/0x560
eeh_handle_special_event+0x328/0x3b0
eeh_handle_event+0x74/0xa0
eeh_event_handler+0x260/0x280
kthread+0x14c/0x190
ret_from_kernel_thread+0x5c/0x74
if (rc == EEH_NEXT_ERR_FROZEN_PE ||
rc == EEH_NEXT_ERR_FENCED_PHB) {
eeh_handle_normal_event(pe);
+
+ /*
+ * eeh_handle_normal_event() can free the PE if it
+ * determines that the PE cannot possibly be recovered.
+ * Make sure the PE still exists before changing its
+ * state.
+ */
+ if (!pe || (pe->type & EEH_PE_INVALID)
+ || (pe->state & EEH_PE_REMOVED)) {
+ pr_warn("EEH: not clearing state on bad PE\n");
+ continue;
+ }
+
It seems not correct. @pe has set to the valid PE in advance, the !pe is
always false? If the PE has been free'd, how can we access @pe->type here
and how can we make sure PE_INVALID and PE_REMOVED flag wasn't overwritten
by somebody else?
On Fri, Mar 03, 2017 at 04:59:11PM +1100, Alexey Kardashevskiy wrote:
On 03/03/17 15:47, Russell Currey wrote:
quoted
eeh_handle_special_event() is called when an EEH event is detected but
can't be narrowed down to a specific PE. This function looks through
every PE to find one in an erroneous state, then calls the regular event
handler eeh_handle_normal_event() once it knows which PE has an error.
However, if eeh_handle_normal_event() found that the PE cannot possibly
be recovered, it will remove the PE and associated devices. This leads
to a use after free in eeh_handle_special_event() as it attempts to clear
the "recovering" state on the PE after eeh_handle_normal_event() returns.
Thus, make sure the PE is valid when attempting to clear state in
eeh_handle_special_event().
Cc: <redacted> #3.10+
Reported-by: Alexey Kardashevskiy <redacted>
Signed-off-by: Russell Currey <redacted>
---
arch/powerpc/kernel/eeh_driver.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
The bug is that pe becomes stale after eeh_handle_normal_event() returned
and dereferencing it afterwards is broken.
Correct, it won't cause a kernel crash as @pe is deferencing linear mapped
area whose address is always valid. I think the proper fix would be to use
eeh_handle_normal_event() to indicate the @pe has been released and don't
access it any more.
quoted
+ pr_warn("EEH: not clearing state on bad PE\n");
The message like this isn't meaningful, no need to have it. The messages that
have prefix "EEH:" is informative messages. We definitely needn't this here.
However, the message might be not needed in next revision.
On Fri, Mar 03, 2017 at 04:59:11PM +1100, Alexey Kardashevskiy wrote:
quoted
On 03/03/17 15:47, Russell Currey wrote:
quoted
eeh_handle_special_event() is called when an EEH event is detected but
can't be narrowed down to a specific PE. This function looks through
every PE to find one in an erroneous state, then calls the regular event
handler eeh_handle_normal_event() once it knows which PE has an error.
However, if eeh_handle_normal_event() found that the PE cannot possibly
be recovered, it will remove the PE and associated devices. This leads
to a use after free in eeh_handle_special_event() as it attempts to clear
the "recovering" state on the PE after eeh_handle_normal_event() returns.
Thus, make sure the PE is valid when attempting to clear state in
eeh_handle_special_event().
Cc: <redacted> #3.10+
Reported-by: Alexey Kardashevskiy <redacted>
Signed-off-by: Russell Currey <redacted>
---
arch/powerpc/kernel/eeh_driver.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
The bug is that pe becomes stale after eeh_handle_normal_event() returned
and dereferencing it afterwards is broken.
Correct, it won't cause a kernel crash as @pe is deferencing linear mapped
area whose address is always valid.
Dereferencing pe would not crash but dereferencing any pointer from the
pnv_ioda_pe struct would (as it would random stuff or a poison).
I think the proper fix would be to use
eeh_handle_normal_event() to indicate the @pe has been released and don't
access it any more.
Correct. The problem is that the callstack from my other reply is a bit too
long to make an trivial patch :)
quoted
quoted
+ pr_warn("EEH: not clearing state on bad PE\n");
The message like this isn't meaningful, no need to have it. The messages that
have prefix "EEH:" is informative messages. We definitely needn't this here.
However, the message might be not needed in next revision.
On Fri, Mar 03, 2017 at 04:59:11PM +1100, Alexey Kardashevskiy wrote:
quoted
On 03/03/17 15:47, Russell Currey wrote:
quoted
eeh_handle_special_event() is called when an EEH event is detected but
can't be narrowed down to a specific PE. This function looks through
every PE to find one in an erroneous state, then calls the regular event
handler eeh_handle_normal_event() once it knows which PE has an error.
However, if eeh_handle_normal_event() found that the PE cannot possibly
be recovered, it will remove the PE and associated devices. This leads
to a use after free in eeh_handle_special_event() as it attempts to clear
the "recovering" state on the PE after eeh_handle_normal_event() returns.
Thus, make sure the PE is valid when attempting to clear state in
eeh_handle_special_event().
Cc: <redacted> #3.10+
Reported-by: Alexey Kardashevskiy <redacted>
Signed-off-by: Russell Currey <redacted>
---
arch/powerpc/kernel/eeh_driver.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
The bug is that pe becomes stale after eeh_handle_normal_event() returned
and dereferencing it afterwards is broken.
Correct, it won't cause a kernel crash as @pe is deferencing linear mapped
area whose address is always valid.
Dereferencing pe would not crash but dereferencing any pointer from the
pnv_ioda_pe struct would (as it would random stuff or a poison).
quoted
I think the proper fix would be to use
eeh_handle_normal_event() to indicate the @pe has been released and don't
access it any more.
Correct. The problem is that the callstack from my other reply is a bit too
long to make an trivial patch :)
Any update on this?
quoted
quoted
quoted
+ pr_warn("EEH: not clearing state on bad PE\n");
The message like this isn't meaningful, no need to have it. The messages that
have prefix "EEH:" is informative messages. We definitely needn't this here.
However, the message might be not needed in next revision.