From: Adam Zilkie <hidden> Date: 2009-09-02 21:37:28
Hi Chris,
I am having a problem similar to what you described in this discussion.
We are using the ppc arch with 2.6.24 with CONFIG_SEQUOIA with compiles
arch/ppc/kernel/head_44x.c (quite different
from /arch/powerpc/kernel/head_32.S). I would like to apply your
backporting patch to this architecture. Any help would be appreciated.
Regards,
Adam
--
Adam Zilkie
Software Designer,
International Datacasting Corp.
This message and the documents attached hereto are intended only for the addressee and may contain privileged or confidential information. Any unauthorized disclosure is strictly prohibited. If you have received this message in error, please notify us immediately so that we may correct our internal records. Please then delete the original message. Thank you.
From: Chris Pringle <hidden> Date: 2009-09-03 08:15:15
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following section:
#ifdef CONFIG_44x
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_GUARDED)
#else
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif
Try adding _PAGE_COHERENT to the appropriate line above and see if that
fixes your issue - this causes the 'M' bit to be set on the page which
sure enforce cache coherency. If it doesn't, you'll need to check the
'M' bit isn't being masked out in head_44x.S (it was originally masked
out on arch/powerpc, but was fixed in later kernels when the cache
coherency issues with non-SMP systems were resolved).
The patch I had fixed two problems on 2.6.26 for 'powerpc':
1) It stopped the 'M' bit being masked out (head_32.S)
2) It set the cache coherency ('M' bit) flag on each page table entry
(pgtable-ppc32.h)
Hope this helps!
Cheers,
Chris
Adam Zilkie wrote:
Hi Chris,
I am having a problem similar to what you described in this discussion.
We are using the ppc arch with 2.6.24 with CONFIG_SEQUOIA with compiles
arch/ppc/kernel/head_44x.c (quite different
from /arch/powerpc/kernel/head_32.S). I would like to apply your
backporting patch to this architecture. Any help would be appreciated.
Regards,
Adam
--
______________________________
Chris Pringle
Software Design Engineer
Miranda Technologies Ltd.
Hithercroft Road
Wallingford
Oxfordshire OX10 9DG
UK
Tel. +44 1491 820206
Fax. +44 1491 820001
www.miranda.com
____________________________
Miranda Technologies Limited
Registered in England and Wales CN 02017053
Registered Office: James House, Mere Park, Dedmere Road, Marlow, Bucks, SL7 1FJ
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-09-03 09:57:37
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following section:
#ifdef CONFIG_44x
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_GUARDED)
#else
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif
Try adding _PAGE_COHERENT to the appropriate line above and see if that
fixes your issue - this causes the 'M' bit to be set on the page which
sure enforce cache coherency. If it doesn't, you'll need to check the
'M' bit isn't being masked out in head_44x.S (it was originally masked
out on arch/powerpc, but was fixed in later kernels when the cache
coherency issues with non-SMP systems were resolved).
I have some doubts about the usefulness of doing that for 4xx. AFAIK,
the 440 core just ignores M.
The problem lies probably elsewhere. Maybe the L2 cache coherency isn't
enabled or not working ?
The L1 cache on 440 is simply not coherent, so drivers have to make sure
they use the appropriate DMA APIs which will do cache flushing when
needed.
Adam, what driver is causing you that sort of problems ?
Cheers,
Ben.
From: Adam Zilkie <hidden> Date: 2009-09-03 16:17:55
Ben,
Thanks for your info.
Are you sure there is L2 cache on the 440?
I am seeing this problem with our custom IDE driver which is based on
pretty old code. Our driver uses pci_alloc_consistent() to allocate the
physical DMA memory and alloc_pages() to allocate a virtual page. It
then uses pci_map_sg() to map to a scatter/gather buffer. Perhaps I
should convert these to the DMA API calls as you suggest.
Regards,
Adam
On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt wrote:
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following section:
#ifdef CONFIG_44x
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_GUARDED)
#else
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif
Try adding _PAGE_COHERENT to the appropriate line above and see if that
fixes your issue - this causes the 'M' bit to be set on the page which
sure enforce cache coherency. If it doesn't, you'll need to check the
'M' bit isn't being masked out in head_44x.S (it was originally masked
out on arch/powerpc, but was fixed in later kernels when the cache
coherency issues with non-SMP systems were resolved).
I have some doubts about the usefulness of doing that for 4xx. AFAIK,
the 440 core just ignores M.
The problem lies probably elsewhere. Maybe the L2 cache coherency isn't
enabled or not working ?
The L1 cache on 440 is simply not coherent, so drivers have to make sure
they use the appropriate DMA APIs which will do cache flushing when
needed.
Adam, what driver is causing you that sort of problems ?
Cheers,
Ben.
--
Adam Zilkie
Software Designer,
International Datacasting Corp.
This message and the documents attached hereto are intended only for the addressee and may contain privileged or confidential information. Any unauthorized disclosure is strictly prohibited. If you have received this message in error, please notify us immediately so that we may correct our internal records. Please then delete the original message. Thank you.
On Thu, Sep 03, 2009 at 12:04:50PM -0400, Adam Zilkie wrote:
Ben,
Thanks for your info.
Are you sure there is L2 cache on the 440?
It depends on which 440 SoC you have. It also depends on that being
configured in the kernel even if it does exist.
I am seeing this problem with our custom IDE driver which is based on
pretty old code. Our driver uses pci_alloc_consistent() to allocate the
physical DMA memory and alloc_pages() to allocate a virtual page. It
then uses pci_map_sg() to map to a scatter/gather buffer. Perhaps I
should convert these to the DMA API calls as you suggest.
I would suggest updating the code. I have no idea if that is the problem,
but it should probably be done anyway.
josh
It depends on the SoC you are using. SoC like 460EX (Canyonlands board)
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC. 440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
I am seeing this problem with our custom IDE driver which is based on=20
pretty old code. Our driver uses pci_alloc_consistent() to allocate the
physical DMA memory and alloc_pages() to allocate a virtual page. It=20
then uses pci_map_sg() to map to a scatter/gather buffer. Perhaps I=20
should convert these to the DMA API calls as you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent() should
work too
Thanks
Prodyut=09
On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt wrote:
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
=20
If you have a look in include/asm-ppc/pgtable.h for the following
#else
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif
=20
Try adding _PAGE_COHERENT to the appropriate line above and see if
that=20
quoted
fixes your issue - this causes the 'M' bit to be set on the page
which=20
quoted
sure enforce cache coherency. If it doesn't, you'll need to check
the=20
quoted
'M' bit isn't being masked out in head_44x.S (it was originally
masked=20
quoted
out on arch/powerpc, but was fixed in later kernels when the cache=20
coherency issues with non-SMP systems were resolved).
=20
I have some doubts about the usefulness of doing that for 4xx. AFAIK,
the 440 core just ignores M.
=20
The problem lies probably elsewhere. Maybe the L2 cache coherency
isn't
enabled or not working ?
=20
The L1 cache on 440 is simply not coherent, so drivers have to make
sure
they use the appropriate DMA APIs which will do cache flushing when
needed.
=20
Adam, what driver is causing you that sort of problems ?
=20
Cheers,
Ben.
=20
=20
--=20
Adam Zilkie
Software Designer,
International Datacasting Corp.
This message and the documents attached hereto are intended only for the
addressee and may contain privileged or confidential information. Any
unauthorized disclosure is strictly prohibited. If you have received
this message in error, please notify us immediately so that we may
correct our internal records. Please then delete the original message.
Thank you.
_______________________________________________
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
--------------------------------------------------------
CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, =
is for the sole use of the intended recipient(s) and contains =
information that is confidential and proprietary to AppliedMicro =
Corporation or its subsidiaries. It is to be used solely for the purpose =
of furthering the parties' business relationship. All unauthorized =
review, use, disclosure or distribution is prohibited. If you are not =
the intended recipient, please contact the sender by reply e-mail and =
destroy all copies of the original message.
From: Adam Zilkie <hidden> Date: 2009-09-08 18:11:56
Hi Prodyut,
Yes, I am using the 440EPx (same as the sequoia board).
Our ideDriver is DMA'ing blocks of 192-byte data over the PCI bus (using
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing)
end up being partially corrupted when we try to parse the data in the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA memory
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of 16byte
cache lines) is overwritten with old data from the last time the buffer
was used.
Regards,
Adam
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands board)
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC. 440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is based on
pretty old code. Our driver uses pci_alloc_consistent() to allocate the
quoted
physical DMA memory and alloc_pages() to allocate a virtual page. It
then uses pci_map_sg() to map to a scatter/gather buffer. Perhaps I
should convert these to the DMA API calls as you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent() should
work too
Thanks
Prodyut
On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
#else
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif
Try adding _PAGE_COHERENT to the appropriate line above and see if
that
quoted
quoted
fixes your issue - this causes the 'M' bit to be set on the page
which
quoted
quoted
sure enforce cache coherency. If it doesn't, you'll need to check
the
quoted
quoted
'M' bit isn't being masked out in head_44x.S (it was originally
masked
quoted
quoted
out on arch/powerpc, but was fixed in later kernels when the cache
coherency issues with non-SMP systems were resolved).
I have some doubts about the usefulness of doing that for 4xx. AFAIK,
the 440 core just ignores M.
The problem lies probably elsewhere. Maybe the L2 cache coherency
isn't
quoted
enabled or not working ?
The L1 cache on 440 is simply not coherent, so drivers have to make
sure
quoted
they use the appropriate DMA APIs which will do cache flushing when
needed.
Adam, what driver is causing you that sort of problems ?
Cheers,
Ben.
--
Adam Zilkie
Software Designer,
International Datacasting Corp.
This message and the documents attached hereto are intended only for the addressee and may contain privileged or confidential information. Any unauthorized disclosure is strictly prohibited. If you have received this message in error, please notify us immediately so that we may correct our internal records. Please then delete the original message. Thank you.
Yes, I am using the 440EPx (same as the sequoia board).=20
Our ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing)
end up being partially corrupted when we try to parse the data in the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of 16byte
cache lines) is overwritten with old data from the last time the
buffer
was used.=20
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA region has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
Hi Adam,
=20
quoted
Are you sure there is L2 cache on the 440?
=20
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC. 440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
=20
quoted
I am seeing this problem with our custom IDE driver which is based on
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
=20
quoted
physical DMA memory and alloc_pages() to allocate a virtual page. It=20
then uses pci_map_sg() to map to a scatter/gather buffer. Perhaps I=20
should convert these to the DMA API calls as you suggest.
=20
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent() should
work too
=20
Thanks
Prodyut=09
=20
On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
=20
If you have a look in include/asm-ppc/pgtable.h for the following
#else
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif
=20
Try adding _PAGE_COHERENT to the appropriate line above and see if
that=20
quoted
quoted
fixes your issue - this causes the 'M' bit to be set on the page
which=20
quoted
quoted
sure enforce cache coherency. If it doesn't, you'll need to check
the=20
quoted
quoted
'M' bit isn't being masked out in head_44x.S (it was originally
masked=20
quoted
quoted
out on arch/powerpc, but was fixed in later kernels when the cache
quoted
quoted
coherency issues with non-SMP systems were resolved).
=20
I have some doubts about the usefulness of doing that for 4xx.
AFAIK,
quoted
the 440 core just ignores M.
=20
The problem lies probably elsewhere. Maybe the L2 cache coherency
isn't
quoted
enabled or not working ?
=20
The L1 cache on 440 is simply not coherent, so drivers have to make
sure
quoted
they use the appropriate DMA APIs which will do cache flushing when
needed.
=20
Adam, what driver is causing you that sort of problems ?
=20
Cheers,
Ben.
=20
=20
--=20
Adam Zilkie
Software Designer,
International Datacasting Corp.
This message and the documents attached hereto are intended only for the
addressee and may contain privileged or confidential information. Any
unauthorized disclosure is strictly prohibited. If you have received
this message in error, please notify us immediately so that we may
correct our internal records. Please then delete the original message.
Thank you.
--------------------------------------------------------
CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, =
is for the sole use of the intended recipient(s) and contains =
information that is confidential and proprietary to AppliedMicro =
Corporation or its subsidiaries. It is to be used solely for the purpose =
of furthering the parties' business relationship. All unauthorized =
review, use, disclosure or distribution is prohibited. If you are not =
the intended recipient, please contact the sender by reply e-mail and =
destroy all copies of the original message.
From: Adam Zilkie <hidden> Date: 2009-09-08 19:41:54
All,
We have found that using flush_dcache_range() after each DMA solves the
problem. Ideally, we'd like to be able to allocate the virtual page in
cache inhibited memory to avoid the performance loss from all the flush
calls. To do this, we'd have to change our TLB sizes and reserve a TLB
in memory as cache inhibited (using the 'I' bit). Will update if this
works as well. Thanks for your help in this.
Regards,
Adam
On Tue, 2009-09-08 at 11:59 -0700, Prodyut Hazarika wrote:
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board).
Our ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing)
end up being partially corrupted when we try to parse the data in the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of 16byte
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA region has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC. 440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is based on
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual page. It
then uses pci_map_sg() to map to a scatter/gather buffer. Perhaps I
should convert these to the DMA API calls as you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent() should
work too
Thanks
Prodyut
On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
#else
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif
Try adding _PAGE_COHERENT to the appropriate line above and see if
that
quoted
quoted
fixes your issue - this causes the 'M' bit to be set on the page
which
quoted
quoted
sure enforce cache coherency. If it doesn't, you'll need to check
the
quoted
quoted
'M' bit isn't being masked out in head_44x.S (it was originally
masked
quoted
quoted
out on arch/powerpc, but was fixed in later kernels when the cache
quoted
quoted
quoted
coherency issues with non-SMP systems were resolved).
I have some doubts about the usefulness of doing that for 4xx.
AFAIK,
quoted
quoted
the 440 core just ignores M.
The problem lies probably elsewhere. Maybe the L2 cache coherency
isn't
quoted
enabled or not working ?
The L1 cache on 440 is simply not coherent, so drivers have to make
sure
quoted
they use the appropriate DMA APIs which will do cache flushing when
needed.
Adam, what driver is causing you that sort of problems ?
Cheers,
Ben.
--
Adam Zilkie
Software Designer,
International Datacasting Corp.
This message and the documents attached hereto are intended only for the
addressee and may contain privileged or confidential information. Any
unauthorized disclosure is strictly prohibited. If you have received
this message in error, please notify us immediately so that we may
correct our internal records. Please then delete the original message.
Thank you.
--------------------------------------------------------
CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and contains information that is confidential and proprietary to AppliedMicro Corporation or its subsidiaries. It is to be used solely for the purpose of furthering the parties' business relationship. All unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
--
Adam Zilkie
Software Designer,
International Datacasting Corp.
This message and the documents attached hereto are intended only for the addressee and may contain privileged or confidential information. Any unauthorized disclosure is strictly prohibited. If you have received this message in error, please notify us immediately so that we may correct our internal records. Please then delete the original message. Thank you.
We have found that using flush_dcache_range() after each DMA solves
the
problem. Ideally, we'd like to be able to allocate the virtual page in
cache inhibited memory to avoid the performance loss from all the
flush
calls. To do this, we'd have to change our TLB sizes and reserve a TLB
in memory as cache inhibited (using the 'I' bit). Will update if this
works as well. Thanks for your help in this.
Aren't you using dma_alloc_coherent to get buffers that are shared
between CPU and external devices?
Thanks
Prodyut
On Tue, 2009-09-08 at 11:59 -0700, Prodyut Hazarika wrote:
Hi Adam,
=20
quoted
Yes, I am using the 440EPx (same as the sequoia board).=20
Our ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on
timing)
quoted
end up being partially corrupted when we try to parse the data in
the
quoted
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of
16byte
quoted
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.=20
=20
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA region
has
the CacheInhibit bit set.
You will need a BDI connected to your system.
=20
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for example on
how to do it.
=20
Thanks
Prodyut
=20
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
=20
quoted
Are you sure there is L2 cache on the 440?
=20
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC.
440EPx
quoted
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is
L2C.
quoted
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
=20
quoted
I am seeing this problem with our custom IDE driver which is based
on
=20
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
=20
quoted
physical DMA memory and alloc_pages() to allocate a virtual page.
It=20
quoted
quoted
then uses pci_map_sg() to map to a scatter/gather buffer. Perhaps I
quoted
quoted
should convert these to the DMA API calls as you suggest.
=20
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent()
should
quoted
work too
=20
Thanks
Prodyut=09
=20
On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
=20
If you have a look in include/asm-ppc/pgtable.h for the
#else
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif
=20
Try adding _PAGE_COHERENT to the appropriate line above and see
if
quoted
that=20
quoted
quoted
fixes your issue - this causes the 'M' bit to be set on the page
which=20
quoted
quoted
sure enforce cache coherency. If it doesn't, you'll need to
check
quoted
the=20
quoted
quoted
'M' bit isn't being masked out in head_44x.S (it was originally
masked=20
quoted
quoted
out on arch/powerpc, but was fixed in later kernels when the
cache
=20
quoted
quoted
quoted
coherency issues with non-SMP systems were resolved).
=20
I have some doubts about the usefulness of doing that for 4xx.
AFAIK,
quoted
quoted
the 440 core just ignores M.
=20
The problem lies probably elsewhere. Maybe the L2 cache coherency
isn't
quoted
enabled or not working ?
=20
The L1 cache on 440 is simply not coherent, so drivers have to
make
quoted
sure
quoted
they use the appropriate DMA APIs which will do cache flushing
when
quoted
quoted
needed.
=20
Adam, what driver is causing you that sort of problems ?
=20
Cheers,
Ben.
=20
=20
--=20
Adam Zilkie
Software Designer,
International Datacasting Corp.
=20
This message and the documents attached hereto are intended only for
the
addressee and may contain privileged or confidential information. Any
unauthorized disclosure is strictly prohibited. If you have received
this message in error, please notify us immediately so that we may
correct our internal records. Please then delete the original message.
Thank you.
--------------------------------------------------------
=20
CONFIDENTIALITY NOTICE: This e-mail message, including any
attachments, is for the sole use of the intended recipient(s) and
contains information that is confidential and proprietary to
AppliedMicro Corporation or its subsidiaries. It is to be used solely
for the purpose of furthering the parties' business relationship. All
unauthorized review, use, disclosure or distribution is prohibited. If
you are not the intended recipient, please contact the sender by reply
e-mail and destroy all copies of the original message.
=20
--=20
Adam Zilkie
Software Designer,
International Datacasting Corp.
This message and the documents attached hereto are intended only for the
addressee and may contain privileged or confidential information. Any
unauthorized disclosure is strictly prohibited. If you have received
this message in error, please notify us immediately so that we may
correct our internal records. Please then delete the original message.
Thank you.
From: Adam Zilkie <hidden> Date: 2009-09-08 20:11:54
We are using pci_alloc_consistent()
Adam
On Tue, 2009-09-08 at 12:56 -0700, Prodyut Hazarika wrote:
quoted
We have found that using flush_dcache_range() after each DMA solves
the
quoted
problem. Ideally, we'd like to be able to allocate the virtual page in
cache inhibited memory to avoid the performance loss from all the
flush
quoted
calls. To do this, we'd have to change our TLB sizes and reserve a TLB
in memory as cache inhibited (using the 'I' bit). Will update if this
works as well. Thanks for your help in this.
Aren't you using dma_alloc_coherent to get buffers that are shared
between CPU and external devices?
Thanks
Prodyut
On Tue, 2009-09-08 at 11:59 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board).
Our ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on
timing)
quoted
quoted
end up being partially corrupted when we try to parse the data in
the
quoted
quoted
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of
16byte
quoted
quoted
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA region
has
quoted
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC.
440EPx
quoted
quoted
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is
L2C.
quoted
quoted
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is based
on
quoted
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual page.
It
quoted
quoted
quoted
then uses pci_map_sg() to map to a scatter/gather buffer. Perhaps I
quoted
quoted
quoted
should convert these to the DMA API calls as you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent()
should
quoted
quoted
work too
Thanks
Prodyut
On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the
#else
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif
Try adding _PAGE_COHERENT to the appropriate line above and see
if
quoted
quoted
that
quoted
quoted
fixes your issue - this causes the 'M' bit to be set on the page
which
quoted
quoted
sure enforce cache coherency. If it doesn't, you'll need to
check
quoted
quoted
the
quoted
quoted
'M' bit isn't being masked out in head_44x.S (it was originally
masked
quoted
quoted
out on arch/powerpc, but was fixed in later kernels when the
cache
quoted
quoted
quoted
quoted
coherency issues with non-SMP systems were resolved).
I have some doubts about the usefulness of doing that for 4xx.
AFAIK,
quoted
quoted
the 440 core just ignores M.
The problem lies probably elsewhere. Maybe the L2 cache coherency
isn't
quoted
enabled or not working ?
The L1 cache on 440 is simply not coherent, so drivers have to
make
quoted
quoted
sure
quoted
they use the appropriate DMA APIs which will do cache flushing
when
quoted
quoted
quoted
needed.
Adam, what driver is causing you that sort of problems ?
Cheers,
Ben.
--
Adam Zilkie
Software Designer,
International Datacasting Corp.
This message and the documents attached hereto are intended only for
the
quoted
addressee and may contain privileged or confidential information. Any
unauthorized disclosure is strictly prohibited. If you have received
this message in error, please notify us immediately so that we may
correct our internal records. Please then delete the original message.
Thank you.
--------------------------------------------------------
CONFIDENTIALITY NOTICE: This e-mail message, including any
attachments, is for the sole use of the intended recipient(s) and
contains information that is confidential and proprietary to
AppliedMicro Corporation or its subsidiaries. It is to be used solely
for the purpose of furthering the parties' business relationship. All
unauthorized review, use, disclosure or distribution is prohibited. If
you are not the intended recipient, please contact the sender by reply
e-mail and destroy all copies of the original message.
quoted
--
Adam Zilkie
Software Designer,
International Datacasting Corp.
This message and the documents attached hereto are intended only for the addressee and may contain privileged or confidential information. Any unauthorized disclosure is strictly prohibited. If you have received this message in error, please notify us immediately so that we may correct our internal records. Please then delete the original message. Thank you.
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-09-09 01:34:26
On Tue, 2009-09-08 at 16:00 -0400, Adam Zilkie wrote:
We are using pci_alloc_consistent()
Then your flush should have no effect since pci_alloc_consistent will
return I=1 mapped memory, unless you don't have
CONFIG_NOT_COHERENT_CACHE for some reason.
Cheers,
Ben.
Adam
On Tue, 2009-09-08 at 12:56 -0700, Prodyut Hazarika wrote:
quoted
quoted
We have found that using flush_dcache_range() after each DMA solves
the
quoted
problem. Ideally, we'd like to be able to allocate the virtual page in
cache inhibited memory to avoid the performance loss from all the
flush
quoted
calls. To do this, we'd have to change our TLB sizes and reserve a TLB
in memory as cache inhibited (using the 'I' bit). Will update if this
works as well. Thanks for your help in this.
Aren't you using dma_alloc_coherent to get buffers that are shared
between CPU and external devices?
Thanks
Prodyut
On Tue, 2009-09-08 at 11:59 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board).
Our ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on
timing)
quoted
quoted
end up being partially corrupted when we try to parse the data in
the
quoted
quoted
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of
16byte
quoted
quoted
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA region
has
quoted
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC.
440EPx
quoted
quoted
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is
L2C.
quoted
quoted
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is based
on
quoted
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual page.
It
quoted
quoted
quoted
then uses pci_map_sg() to map to a scatter/gather buffer. Perhaps I
quoted
quoted
quoted
should convert these to the DMA API calls as you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent()
should
quoted
quoted
work too
Thanks
Prodyut
On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the
#else
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif
Try adding _PAGE_COHERENT to the appropriate line above and see
if
quoted
quoted
that
quoted
quoted
fixes your issue - this causes the 'M' bit to be set on the page
which
quoted
quoted
sure enforce cache coherency. If it doesn't, you'll need to
check
quoted
quoted
the
quoted
quoted
'M' bit isn't being masked out in head_44x.S (it was originally
masked
quoted
quoted
out on arch/powerpc, but was fixed in later kernels when the
cache
quoted
quoted
quoted
quoted
coherency issues with non-SMP systems were resolved).
I have some doubts about the usefulness of doing that for 4xx.
AFAIK,
quoted
quoted
the 440 core just ignores M.
The problem lies probably elsewhere. Maybe the L2 cache coherency
isn't
quoted
enabled or not working ?
The L1 cache on 440 is simply not coherent, so drivers have to
make
quoted
quoted
sure
quoted
they use the appropriate DMA APIs which will do cache flushing
when
quoted
quoted
quoted
needed.
Adam, what driver is causing you that sort of problems ?
Cheers,
Ben.
--
Adam Zilkie
Software Designer,
International Datacasting Corp.
This message and the documents attached hereto are intended only for
the
quoted
addressee and may contain privileged or confidential information. Any
unauthorized disclosure is strictly prohibited. If you have received
this message in error, please notify us immediately so that we may
correct our internal records. Please then delete the original message.
Thank you.
--------------------------------------------------------
CONFIDENTIALITY NOTICE: This e-mail message, including any
attachments, is for the sole use of the intended recipient(s) and
contains information that is confidential and proprietary to
AppliedMicro Corporation or its subsidiaries. It is to be used solely
for the purpose of furthering the parties' business relationship. All
unauthorized review, use, disclosure or distribution is prohibited. If
you are not the intended recipient, please contact the sender by reply
e-mail and destroy all copies of the original message.
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-09-08 21:35:06
On Tue, 2009-09-08 at 15:30 -0400, Adam Zilkie wrote:
All,
We have found that using flush_dcache_range() after each DMA solves the
problem. Ideally, we'd like to be able to allocate the virtual page in
cache inhibited memory to avoid the performance loss from all the flush
calls. To do this, we'd have to change our TLB sizes and reserve a TLB
in memory as cache inhibited (using the 'I' bit). Will update if this
works as well. Thanks for your help in this.
I think the problem is that you are manipulating the TLB directly, which
you shouldn't have to do. You also shouldn't have to use
flush_dcache_range() yourself neither.
It should all be handled by the DMA and PCI DMA APIs, you are just not
using those correctly.
You have two choice. You can either allocate memory permanently mapped
with I=1, in which case, use pci_alloc_consistent() (or
dma_alloc_coherent(), same thing).
Or you can use "normal" memory and ensure you flush/invalidate the cache
at the right time, which you can do with something like
pci_map_sg/pci_unmap_sg (or dma_* variants) or the dma_sync_* functions.
It's all pretty standard mechanisms in Linux, other platforms also have
non-coherent DMA (such as some ARMs) and those functions are generic.
Cheers,
Ben.
Regards,
Adam
On Tue, 2009-09-08 at 11:59 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board).
Our ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing)
end up being partially corrupted when we try to parse the data in the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of 16byte
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA region has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC. 440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is based on
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual page. It
then uses pci_map_sg() to map to a scatter/gather buffer. Perhaps I
should convert these to the DMA API calls as you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent() should
work too
Thanks
Prodyut
On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
#else
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif
Try adding _PAGE_COHERENT to the appropriate line above and see if
that
quoted
quoted
fixes your issue - this causes the 'M' bit to be set on the page
which
quoted
quoted
sure enforce cache coherency. If it doesn't, you'll need to check
the
quoted
quoted
'M' bit isn't being masked out in head_44x.S (it was originally
masked
quoted
quoted
out on arch/powerpc, but was fixed in later kernels when the cache
quoted
quoted
quoted
coherency issues with non-SMP systems were resolved).
I have some doubts about the usefulness of doing that for 4xx.
AFAIK,
quoted
quoted
the 440 core just ignores M.
The problem lies probably elsewhere. Maybe the L2 cache coherency
isn't
quoted
enabled or not working ?
The L1 cache on 440 is simply not coherent, so drivers have to make
sure
quoted
they use the appropriate DMA APIs which will do cache flushing when
needed.
Adam, what driver is causing you that sort of problems ?
Cheers,
Ben.
--
Adam Zilkie
Software Designer,
International Datacasting Corp.
This message and the documents attached hereto are intended only for the
addressee and may contain privileged or confidential information. Any
unauthorized disclosure is strictly prohibited. If you have received
this message in error, please notify us immediately so that we may
correct our internal records. Please then delete the original message.
Thank you.
--------------------------------------------------------
CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is for the sole use of the intended recipient(s) and contains information that is confidential and proprietary to AppliedMicro Corporation or its subsidiaries. It is to be used solely for the purpose of furthering the parties' business relationship. All unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message.
Hi,
Why manage cache lines manually, if appropriate code is a part of
__dma_sync / dma_sync_single_for_device of DMA API ? (implies
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)
Prodyut Hazarika wrote:
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board).
Our ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing)
end up being partially corrupted when we try to parse the data in the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of 16byte
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA region has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC. 440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is based on
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual page. It
then uses pci_map_sg() to map to a scatter/gather buffer. Perhaps I
should convert these to the DMA API calls as you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent() should
work too
Thanks
Prodyut
On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
From: Tom Burns <hidden> Date: 2009-09-09 13:56:00
Hi,
With the default config for the Sequoia board on 2.6.24, calling
pci_dma_sync_sg_for_cpu() results in executing
invalidate_dcache_range() in arch/ppc/kernel/misc.S from __dma_sync().
This OOPses on PPC440 since it tries to call directly the assembly
instruction dcbi, which can only be executed in supervisor mode. We
tried that before resorting to manual cache line management with
usermode-safe assembly calls.
Regards,
Tom Burns
International Datacasting Corporation
Mikhail Zolotaryov wrote:
Hi,
Why manage cache lines manually, if appropriate code is a part of
__dma_sync / dma_sync_single_for_device of DMA API ? (implies
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)
Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board). Our
ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing)
end up being partially corrupted when we try to parse the data in the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of 16byte
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA region has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC. 440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is based on
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual page.
It then uses pci_map_sg() to map to a scatter/gather buffer.
Perhaps I should convert these to the DMA API calls as you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent() should
work too
Thanks
Prodyut
On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
Hi Tom,
possible solution could be to use tasklet to perform DMA-related job (as
in most cases DMA transfer is interrupt driven - makes sense).
Tom Burns wrote:
Hi,
With the default config for the Sequoia board on 2.6.24, calling
pci_dma_sync_sg_for_cpu() results in executing
invalidate_dcache_range() in arch/ppc/kernel/misc.S from
__dma_sync(). This OOPses on PPC440 since it tries to call directly
the assembly instruction dcbi, which can only be executed in
supervisor mode. We tried that before resorting to manual cache line
management with usermode-safe assembly calls.
Regards,
Tom Burns
International Datacasting Corporation
Mikhail Zolotaryov wrote:
quoted
Hi,
Why manage cache lines manually, if appropriate code is a part of
__dma_sync / dma_sync_single_for_device of DMA API ? (implies
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)
Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board). Our
ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing)
end up being partially corrupted when we try to parse the data in the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of 16byte
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA region has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC. 440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is based on
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual page.
It then uses pci_map_sg() to map to a scatter/gather buffer.
Perhaps I should convert these to the DMA API calls as you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent() should
work too
Thanks
Prodyut
On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
From: Tom Burns <hidden> Date: 2009-09-09 14:21:01
Hi Mikhail,
Sorry, this DMA code is in a tasklet. Are you suggesting the processor
is in supervisor mode at that time? Calling pci_dma_sync_sg_for_cpu()
from the tasklet context is what generates the OOPS. The entire oops is
as follows, if it's relevant:
Oops: kernel access of bad area, sig: 11 [#1]
NIP: c0003ab0 LR: c0010c30 CTR: 02400001
REGS: df117bd0 TRAP: 0300 Tainted: P (2.6.24.2)
MSR: 00029000 <EE,ME> CR: 44224042 XER: 20000000
DEAR: 3fd39000, ESR: 00800000
TASK = de5db7d0[157] 'cat' THREAD: df116000
GPR00: e11e5854 df117c80 de5db7d0 3fd39000 02400001 0000001f 00000002
0079a169
GPR08: 00000001 c0310000 00000000 c0010c84 24224042 101c0dac c0310000
10177000
GPR16: deb14200 df116000 e12062d0 e11f6104 de0f16c0 e11f0000 c0310000
e11f59cc
GPR24: e11f62d0 e11f0000 e11f0000 00000000 00000002 defee014 3fd39008
87d39009
NIP [c0003ab0] invalidate_dcache_range+0x1c/0x30
LR [c0010c30] __dma_sync+0x58/0xac
Call Trace:
[df117c80] [0000000a] 0xa (unreliable)
[df117c90] [e11e5854] DoTasklet+0x67c/0xc90 [ideDriverDuo_cyph]
[df117ce0] [c001ee24] tasklet_action+0x60/0xcc
[df117cf0] [c001ef04] __do_softirq+0x74/0xe0
[df117d10] [c00067a8] do_softirq+0x54/0x58
[df117d20] [c001edb4] irq_exit+0x48/0x58
[df117d30] [c00069d0] do_IRQ+0x6c/0xc0
[df117d40] [c00020e0] ret_from_except+0x0/0x18
[df117e00] [c00501e0] unmap_vmas+0x2c4/0x560
[df117e90] [c0053ebc] exit_mmap+0x64/0xec
[df117ec0] [c00171ac] mmput+0x50/0xd4
[df117ed0] [c001aef8] exit_mm+0x80/0xe0
[df117ef0] [c001c818] do_exit+0x134/0x6f8
[df117f30] [c001ce14] do_group_exit+0x38/0x74
[df117f40] [c0001a80] ret_from_syscall+0x0/0x3c
Instruction dump:
7c0018ac 38630020 4200fff8 7c0004ac 4e800020 38a0001f 7c632878 7c832050
7c842a14 5484d97f 4d820020 7c8903a6 <7c001bac> 38630020 4200fff8
7c0004ac
Kernel panic - not syncing: Aiee, killing interrupt handler!
Rebooting in 180 seconds..
Cheers,
Tom
Mikhail Zolotaryov wrote:
Hi Tom,
possible solution could be to use tasklet to perform DMA-related job
(as in most cases DMA transfer is interrupt driven - makes sense).
Tom Burns wrote:
quoted
Hi,
With the default config for the Sequoia board on 2.6.24, calling
pci_dma_sync_sg_for_cpu() results in executing
invalidate_dcache_range() in arch/ppc/kernel/misc.S from
__dma_sync(). This OOPses on PPC440 since it tries to call directly
the assembly instruction dcbi, which can only be executed in
supervisor mode. We tried that before resorting to manual cache line
management with usermode-safe assembly calls.
Regards,
Tom Burns
International Datacasting Corporation
Mikhail Zolotaryov wrote:
quoted
Hi,
Why manage cache lines manually, if appropriate code is a part of
__dma_sync / dma_sync_single_for_device of DMA API ? (implies
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)
Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board). Our
ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing)
end up being partially corrupted when we try to parse the data in the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of 16byte
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA region
has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC.
440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is
L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is
based on
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual page.
It then uses pci_map_sg() to map to a scatter/gather buffer.
Perhaps I should convert these to the DMA API calls as you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent() should
work too
Thanks
Prodyut On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt
wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
Hi Tom,
In my case __dma_sync() calls flush_dcache_range() (it's due to
alignment) from a tasklet - no OOPS. It uses dcbf instruction instead of
dcbi - that's the difference as dcbf is not privileged.
Tom Burns wrote:
Hi Mikhail,
Sorry, this DMA code is in a tasklet. Are you suggesting the
processor is in supervisor mode at that time? Calling
pci_dma_sync_sg_for_cpu() from the tasklet context is what generates
the OOPS. The entire oops is as follows, if it's relevant:
Oops: kernel access of bad area, sig: 11 [#1]
NIP: c0003ab0 LR: c0010c30 CTR: 02400001
REGS: df117bd0 TRAP: 0300 Tainted: P (2.6.24.2)
MSR: 00029000 <EE,ME> CR: 44224042 XER: 20000000
DEAR: 3fd39000, ESR: 00800000
TASK = de5db7d0[157] 'cat' THREAD: df116000
GPR00: e11e5854 df117c80 de5db7d0 3fd39000 02400001 0000001f 00000002
0079a169
GPR08: 00000001 c0310000 00000000 c0010c84 24224042 101c0dac c0310000
10177000
GPR16: deb14200 df116000 e12062d0 e11f6104 de0f16c0 e11f0000 c0310000
e11f59cc
GPR24: e11f62d0 e11f0000 e11f0000 00000000 00000002 defee014 3fd39008
87d39009
NIP [c0003ab0] invalidate_dcache_range+0x1c/0x30
LR [c0010c30] __dma_sync+0x58/0xac
Call Trace:
[df117c80] [0000000a] 0xa (unreliable)
[df117c90] [e11e5854] DoTasklet+0x67c/0xc90 [ideDriverDuo_cyph]
[df117ce0] [c001ee24] tasklet_action+0x60/0xcc
[df117cf0] [c001ef04] __do_softirq+0x74/0xe0
[df117d10] [c00067a8] do_softirq+0x54/0x58
[df117d20] [c001edb4] irq_exit+0x48/0x58
[df117d30] [c00069d0] do_IRQ+0x6c/0xc0
[df117d40] [c00020e0] ret_from_except+0x0/0x18
[df117e00] [c00501e0] unmap_vmas+0x2c4/0x560
[df117e90] [c0053ebc] exit_mmap+0x64/0xec
[df117ec0] [c00171ac] mmput+0x50/0xd4
[df117ed0] [c001aef8] exit_mm+0x80/0xe0
[df117ef0] [c001c818] do_exit+0x134/0x6f8
[df117f30] [c001ce14] do_group_exit+0x38/0x74
[df117f40] [c0001a80] ret_from_syscall+0x0/0x3c
Instruction dump:
7c0018ac 38630020 4200fff8 7c0004ac 4e800020 38a0001f 7c632878 7c832050
7c842a14 5484d97f 4d820020 7c8903a6 <7c001bac> 38630020 4200fff8
7c0004ac
Kernel panic - not syncing: Aiee, killing interrupt handler!
Rebooting in 180 seconds..
Cheers,
Tom
Mikhail Zolotaryov wrote:
quoted
Hi Tom,
possible solution could be to use tasklet to perform DMA-related job
(as in most cases DMA transfer is interrupt driven - makes sense).
Tom Burns wrote:
quoted
Hi,
With the default config for the Sequoia board on 2.6.24, calling
pci_dma_sync_sg_for_cpu() results in executing
invalidate_dcache_range() in arch/ppc/kernel/misc.S from
__dma_sync(). This OOPses on PPC440 since it tries to call directly
the assembly instruction dcbi, which can only be executed in
supervisor mode. We tried that before resorting to manual cache
line management with usermode-safe assembly calls.
Regards,
Tom Burns
International Datacasting Corporation
Mikhail Zolotaryov wrote:
quoted
Hi,
Why manage cache lines manually, if appropriate code is a part of
__dma_sync / dma_sync_single_for_device of DMA API ? (implies
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)
Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board). Our
ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on
timing)
end up being partially corrupted when we try to parse the data in
the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of
16byte
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA
region has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for
example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC.
440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is
L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is
based on
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual
page. It then uses pci_map_sg() to map to a scatter/gather
buffer. Perhaps I should convert these to the DMA API calls as
you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent()
should
work too
Thanks
Prodyut On Thu, 2009-09-03 at 19:57 +1000, Benjamin
Herrenschmidt wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-09-11 01:58:05
On Wed, 2009-09-09 at 17:40 +0300, Mikhail Zolotaryov wrote:
Hi Tom,
In my case __dma_sync() calls flush_dcache_range() (it's due to
alignment) from a tasklet - no OOPS. It uses dcbf instruction instead of
dcbi - that's the difference as dcbf is not privileged.
What it calls depends on the direction of the transfer. The tasklet runs
in priviledged mode, dcbi should work just fine... if passed a correct
address :-)
Cheers,
Ben.
Tom Burns wrote:
quoted
Hi Mikhail,
Sorry, this DMA code is in a tasklet. Are you suggesting the
processor is in supervisor mode at that time? Calling
pci_dma_sync_sg_for_cpu() from the tasklet context is what generates
the OOPS. The entire oops is as follows, if it's relevant:
Oops: kernel access of bad area, sig: 11 [#1]
NIP: c0003ab0 LR: c0010c30 CTR: 02400001
REGS: df117bd0 TRAP: 0300 Tainted: P (2.6.24.2)
MSR: 00029000 <EE,ME> CR: 44224042 XER: 20000000
DEAR: 3fd39000, ESR: 00800000
TASK = de5db7d0[157] 'cat' THREAD: df116000
GPR00: e11e5854 df117c80 de5db7d0 3fd39000 02400001 0000001f 00000002
0079a169
GPR08: 00000001 c0310000 00000000 c0010c84 24224042 101c0dac c0310000
10177000
GPR16: deb14200 df116000 e12062d0 e11f6104 de0f16c0 e11f0000 c0310000
e11f59cc
GPR24: e11f62d0 e11f0000 e11f0000 00000000 00000002 defee014 3fd39008
87d39009
NIP [c0003ab0] invalidate_dcache_range+0x1c/0x30
LR [c0010c30] __dma_sync+0x58/0xac
Call Trace:
[df117c80] [0000000a] 0xa (unreliable)
[df117c90] [e11e5854] DoTasklet+0x67c/0xc90 [ideDriverDuo_cyph]
[df117ce0] [c001ee24] tasklet_action+0x60/0xcc
[df117cf0] [c001ef04] __do_softirq+0x74/0xe0
[df117d10] [c00067a8] do_softirq+0x54/0x58
[df117d20] [c001edb4] irq_exit+0x48/0x58
[df117d30] [c00069d0] do_IRQ+0x6c/0xc0
[df117d40] [c00020e0] ret_from_except+0x0/0x18
[df117e00] [c00501e0] unmap_vmas+0x2c4/0x560
[df117e90] [c0053ebc] exit_mmap+0x64/0xec
[df117ec0] [c00171ac] mmput+0x50/0xd4
[df117ed0] [c001aef8] exit_mm+0x80/0xe0
[df117ef0] [c001c818] do_exit+0x134/0x6f8
[df117f30] [c001ce14] do_group_exit+0x38/0x74
[df117f40] [c0001a80] ret_from_syscall+0x0/0x3c
Instruction dump:
7c0018ac 38630020 4200fff8 7c0004ac 4e800020 38a0001f 7c632878 7c832050
7c842a14 5484d97f 4d820020 7c8903a6 <7c001bac> 38630020 4200fff8
7c0004ac
Kernel panic - not syncing: Aiee, killing interrupt handler!
Rebooting in 180 seconds..
Cheers,
Tom
Mikhail Zolotaryov wrote:
quoted
Hi Tom,
possible solution could be to use tasklet to perform DMA-related job
(as in most cases DMA transfer is interrupt driven - makes sense).
Tom Burns wrote:
quoted
Hi,
With the default config for the Sequoia board on 2.6.24, calling
pci_dma_sync_sg_for_cpu() results in executing
invalidate_dcache_range() in arch/ppc/kernel/misc.S from
__dma_sync(). This OOPses on PPC440 since it tries to call directly
the assembly instruction dcbi, which can only be executed in
supervisor mode. We tried that before resorting to manual cache
line management with usermode-safe assembly calls.
Regards,
Tom Burns
International Datacasting Corporation
Mikhail Zolotaryov wrote:
quoted
Hi,
Why manage cache lines manually, if appropriate code is a part of
__dma_sync / dma_sync_single_for_device of DMA API ? (implies
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)
Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board). Our
ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on
timing)
end up being partially corrupted when we try to parse the data in
the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of
16byte
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA
region has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for
example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC.
440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is
L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is
based on
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual
page. It then uses pci_map_sg() to map to a scatter/gather
buffer. Perhaps I should convert these to the DMA API calls as
you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent()
should
work too
Thanks
Prodyut On Thu, 2009-09-03 at 19:57 +1000, Benjamin
Herrenschmidt wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
On Wed, 2009-09-09 at 17:40 +0300, Mikhail Zolotaryov wrote:
quoted
Hi Tom,
In my case __dma_sync() calls flush_dcache_range() (it's due to
alignment) from a tasklet - no OOPS. It uses dcbf instruction instead of
dcbi - that's the difference as dcbf is not privileged.
What it calls depends on the direction of the transfer.
Would not agree with you in this point as __dma_sync() code is:
case DMA_FROM_DEVICE:
/*
* invalidate only when cache-line aligned otherwise
there is
* the potential for discarding uncommitted data from
the cache
*/
if ((start & (L1_CACHE_BYTES - 1)) || (size &
(L1_CACHE_BYTES - 1)))
flush_dcache_range(start, end);
else
invalidate_dcache_range(start, end);
break;
So, actual instruction used depends on address/size alignment.
The tasklet runs
in priviledged mode, dcbi should work just fine... if passed a correct
address :-)
Cheers,
Ben.
quoted
Tom Burns wrote:
quoted
Hi Mikhail,
Sorry, this DMA code is in a tasklet. Are you suggesting the
processor is in supervisor mode at that time? Calling
pci_dma_sync_sg_for_cpu() from the tasklet context is what generates
the OOPS. The entire oops is as follows, if it's relevant:
Oops: kernel access of bad area, sig: 11 [#1]
NIP: c0003ab0 LR: c0010c30 CTR: 02400001
REGS: df117bd0 TRAP: 0300 Tainted: P (2.6.24.2)
MSR: 00029000 <EE,ME> CR: 44224042 XER: 20000000
DEAR: 3fd39000, ESR: 00800000
TASK = de5db7d0[157] 'cat' THREAD: df116000
GPR00: e11e5854 df117c80 de5db7d0 3fd39000 02400001 0000001f 00000002
0079a169
GPR08: 00000001 c0310000 00000000 c0010c84 24224042 101c0dac c0310000
10177000
GPR16: deb14200 df116000 e12062d0 e11f6104 de0f16c0 e11f0000 c0310000
e11f59cc
GPR24: e11f62d0 e11f0000 e11f0000 00000000 00000002 defee014 3fd39008
87d39009
NIP [c0003ab0] invalidate_dcache_range+0x1c/0x30
LR [c0010c30] __dma_sync+0x58/0xac
Call Trace:
[df117c80] [0000000a] 0xa (unreliable)
[df117c90] [e11e5854] DoTasklet+0x67c/0xc90 [ideDriverDuo_cyph]
[df117ce0] [c001ee24] tasklet_action+0x60/0xcc
[df117cf0] [c001ef04] __do_softirq+0x74/0xe0
[df117d10] [c00067a8] do_softirq+0x54/0x58
[df117d20] [c001edb4] irq_exit+0x48/0x58
[df117d30] [c00069d0] do_IRQ+0x6c/0xc0
[df117d40] [c00020e0] ret_from_except+0x0/0x18
[df117e00] [c00501e0] unmap_vmas+0x2c4/0x560
[df117e90] [c0053ebc] exit_mmap+0x64/0xec
[df117ec0] [c00171ac] mmput+0x50/0xd4
[df117ed0] [c001aef8] exit_mm+0x80/0xe0
[df117ef0] [c001c818] do_exit+0x134/0x6f8
[df117f30] [c001ce14] do_group_exit+0x38/0x74
[df117f40] [c0001a80] ret_from_syscall+0x0/0x3c
Instruction dump:
7c0018ac 38630020 4200fff8 7c0004ac 4e800020 38a0001f 7c632878 7c832050
7c842a14 5484d97f 4d820020 7c8903a6 <7c001bac> 38630020 4200fff8
7c0004ac
Kernel panic - not syncing: Aiee, killing interrupt handler!
Rebooting in 180 seconds..
Cheers,
Tom
Mikhail Zolotaryov wrote:
quoted
Hi Tom,
possible solution could be to use tasklet to perform DMA-related job
(as in most cases DMA transfer is interrupt driven - makes sense).
Tom Burns wrote:
quoted
Hi,
With the default config for the Sequoia board on 2.6.24, calling
pci_dma_sync_sg_for_cpu() results in executing
invalidate_dcache_range() in arch/ppc/kernel/misc.S from
__dma_sync(). This OOPses on PPC440 since it tries to call directly
the assembly instruction dcbi, which can only be executed in
supervisor mode. We tried that before resorting to manual cache
line management with usermode-safe assembly calls.
Regards,
Tom Burns
International Datacasting Corporation
Mikhail Zolotaryov wrote:
quoted
Hi,
Why manage cache lines manually, if appropriate code is a part of
__dma_sync / dma_sync_single_for_device of DMA API ? (implies
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)
Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board). Our
ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on
timing)
end up being partially corrupted when we try to parse the data in
the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of
16byte
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA
region has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for
example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC.
440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is
L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is
based on
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual
page. It then uses pci_map_sg() to map to a scatter/gather
buffer. Perhaps I should convert these to the DMA API calls as
you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent()
should
work too
Thanks
Prodyut On Thu, 2009-09-03 at 19:57 +1000, Benjamin
Herrenschmidt wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-09-11 07:32:14
On Fri, 2009-09-11 at 10:17 +0300, Mikhail Zolotaryov wrote:
Benjamin Herrenschmidt wrote:
quoted
On Wed, 2009-09-09 at 17:40 +0300, Mikhail Zolotaryov wrote:
quoted
Hi Tom,
In my case __dma_sync() calls flush_dcache_range() (it's due to
alignment) from a tasklet - no OOPS. It uses dcbf instruction instead of
dcbi - that's the difference as dcbf is not privileged.
What it calls depends on the direction of the transfer.
Would not agree with you in this point as __dma_sync() code is:
Well, it -does- depend on the direction of the transfer... and -also- on
the size & alignement :-)
Anyway, that is probably not the problem. From the log I've seen, it
just looks like a page fault due to a bad virtual address passed there.
Cheers,
Ben.
case DMA_FROM_DEVICE:
/*
* invalidate only when cache-line aligned otherwise
there is
* the potential for discarding uncommitted data from
the cache
*/
if ((start & (L1_CACHE_BYTES - 1)) || (size &
(L1_CACHE_BYTES - 1)))
flush_dcache_range(start, end);
else
invalidate_dcache_range(start, end);
break;
So, actual instruction used depends on address/size alignment.
quoted
The tasklet runs
in priviledged mode, dcbi should work just fine... if passed a correct
address :-)
Cheers,
Ben.
quoted
Tom Burns wrote:
quoted
Hi Mikhail,
Sorry, this DMA code is in a tasklet. Are you suggesting the
processor is in supervisor mode at that time? Calling
pci_dma_sync_sg_for_cpu() from the tasklet context is what generates
the OOPS. The entire oops is as follows, if it's relevant:
Oops: kernel access of bad area, sig: 11 [#1]
NIP: c0003ab0 LR: c0010c30 CTR: 02400001
REGS: df117bd0 TRAP: 0300 Tainted: P (2.6.24.2)
MSR: 00029000 <EE,ME> CR: 44224042 XER: 20000000
DEAR: 3fd39000, ESR: 00800000
TASK = de5db7d0[157] 'cat' THREAD: df116000
GPR00: e11e5854 df117c80 de5db7d0 3fd39000 02400001 0000001f 00000002
0079a169
GPR08: 00000001 c0310000 00000000 c0010c84 24224042 101c0dac c0310000
10177000
GPR16: deb14200 df116000 e12062d0 e11f6104 de0f16c0 e11f0000 c0310000
e11f59cc
GPR24: e11f62d0 e11f0000 e11f0000 00000000 00000002 defee014 3fd39008
87d39009
NIP [c0003ab0] invalidate_dcache_range+0x1c/0x30
LR [c0010c30] __dma_sync+0x58/0xac
Call Trace:
[df117c80] [0000000a] 0xa (unreliable)
[df117c90] [e11e5854] DoTasklet+0x67c/0xc90 [ideDriverDuo_cyph]
[df117ce0] [c001ee24] tasklet_action+0x60/0xcc
[df117cf0] [c001ef04] __do_softirq+0x74/0xe0
[df117d10] [c00067a8] do_softirq+0x54/0x58
[df117d20] [c001edb4] irq_exit+0x48/0x58
[df117d30] [c00069d0] do_IRQ+0x6c/0xc0
[df117d40] [c00020e0] ret_from_except+0x0/0x18
[df117e00] [c00501e0] unmap_vmas+0x2c4/0x560
[df117e90] [c0053ebc] exit_mmap+0x64/0xec
[df117ec0] [c00171ac] mmput+0x50/0xd4
[df117ed0] [c001aef8] exit_mm+0x80/0xe0
[df117ef0] [c001c818] do_exit+0x134/0x6f8
[df117f30] [c001ce14] do_group_exit+0x38/0x74
[df117f40] [c0001a80] ret_from_syscall+0x0/0x3c
Instruction dump:
7c0018ac 38630020 4200fff8 7c0004ac 4e800020 38a0001f 7c632878 7c832050
7c842a14 5484d97f 4d820020 7c8903a6 <7c001bac> 38630020 4200fff8
7c0004ac
Kernel panic - not syncing: Aiee, killing interrupt handler!
Rebooting in 180 seconds..
Cheers,
Tom
Mikhail Zolotaryov wrote:
quoted
Hi Tom,
possible solution could be to use tasklet to perform DMA-related job
(as in most cases DMA transfer is interrupt driven - makes sense).
Tom Burns wrote:
quoted
Hi,
With the default config for the Sequoia board on 2.6.24, calling
pci_dma_sync_sg_for_cpu() results in executing
invalidate_dcache_range() in arch/ppc/kernel/misc.S from
__dma_sync(). This OOPses on PPC440 since it tries to call directly
the assembly instruction dcbi, which can only be executed in
supervisor mode. We tried that before resorting to manual cache
line management with usermode-safe assembly calls.
Regards,
Tom Burns
International Datacasting Corporation
Mikhail Zolotaryov wrote:
quoted
Hi,
Why manage cache lines manually, if appropriate code is a part of
__dma_sync / dma_sync_single_for_device of DMA API ? (implies
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)
Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board). Our
ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on
timing)
end up being partially corrupted when we try to parse the data in
the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of
16byte
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA
region has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for
example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC.
440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is
L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is
based on
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual
page. It then uses pci_map_sg() to map to a scatter/gather
buffer. Perhaps I should convert these to the DMA API calls as
you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent()
should
work too
Thanks
Prodyut On Thu, 2009-09-03 at 19:57 +1000, Benjamin
Herrenschmidt wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-09-11 01:57:21
On Wed, 2009-09-09 at 10:10 -0400, Tom Burns wrote:
Hi Mikhail,
Sorry, this DMA code is in a tasklet. Are you suggesting the processor
is in supervisor mode at that time? Calling pci_dma_sync_sg_for_cpu()
from the tasklet context is what generates the OOPS. The entire oops is
as follows, if it's relevant:
Yes. A tasklet runs in supervisor mode.
Oops: kernel access of bad area, sig: 11 [#1]
That seems to indicate that your passed an incorrect address to dcbi
(ie, an address that isn't currently mapped, 0x3fd39000 according to
your log), so that looks like a driver bug to me (unless something is
busted in 2.6.24 version of __dma_sync on 44x...)
Cheers,
Ben.
Hi Tom,
possible solution could be to use tasklet to perform DMA-related job
(as in most cases DMA transfer is interrupt driven - makes sense).
Tom Burns wrote:
quoted
Hi,
With the default config for the Sequoia board on 2.6.24, calling
pci_dma_sync_sg_for_cpu() results in executing
invalidate_dcache_range() in arch/ppc/kernel/misc.S from
__dma_sync(). This OOPses on PPC440 since it tries to call directly
the assembly instruction dcbi, which can only be executed in
supervisor mode. We tried that before resorting to manual cache line
management with usermode-safe assembly calls.
Regards,
Tom Burns
International Datacasting Corporation
Mikhail Zolotaryov wrote:
quoted
Hi,
Why manage cache lines manually, if appropriate code is a part of
__dma_sync / dma_sync_single_for_device of DMA API ? (implies
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)
Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board). Our
ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing)
end up being partially corrupted when we try to parse the data in the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of 16byte
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA region
has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC.
440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is
L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is
based on
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual page.
It then uses pci_map_sg() to map to a scatter/gather buffer.
Perhaps I should convert these to the DMA API calls as you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent() should
work too
Thanks
Prodyut On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt
wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
From: Tom Burns <hidden> Date: 2009-09-10 20:01:28
Hi,
Thank you everyone for your help.
I've been looking into the other dma/pci API calls (dma_alloc_coherent,
pci_alloc_consistent). I don't see how either of these return memory
mapped to a TLB with the I bit set to 1 in kernel 2.6.24. In our kernel
code, the only use of the PPC44x_TLB_I define is in head_44x.S in
_start. We have CONFIG_NON_COHERENT_CACHE enabled.
We changed our code to use dma_alloc_coherent, removed our manual
cacheline flushing, and saw the corrupted data return. To me this means
dma_alloc_coherent cannot be setting the I=1 bit in the TLB entry.
I tried, using our JTAG debugger (BDI3000), to pause operation after
calling dma_alloc_coherent to examine the TLB entry for the memory
returned by the call (which was just past
CONFIG_CONSISTENT_START=0xff100000). The TLB list loaded at the time
that I paused operation did not show a mapping for this area. I guess
the kernel swaps TLB entries on the fly so it isn't limited to only 64
entries? I will try to sleep in the same context as the
dma_alloc_coherent call to try to catch the TLB entry while loaded to
see if it has the I bit set.
If that fails, any ideas?
Thanks,
Tom Burns
International Datacasting Corporation
Mikhail Zolotaryov wrote:
Hi Tom,
possible solution could be to use tasklet to perform DMA-related job
(as in most cases DMA transfer is interrupt driven - makes sense).
Tom Burns wrote:
quoted
Hi,
With the default config for the Sequoia board on 2.6.24, calling
pci_dma_sync_sg_for_cpu() results in executing
invalidate_dcache_range() in arch/ppc/kernel/misc.S from
__dma_sync(). This OOPses on PPC440 since it tries to call directly
the assembly instruction dcbi, which can only be executed in
supervisor mode. We tried that before resorting to manual cache line
management with usermode-safe assembly calls.
Regards,
Tom Burns
International Datacasting Corporation
Mikhail Zolotaryov wrote:
quoted
Hi,
Why manage cache lines manually, if appropriate code is a part of
__dma_sync / dma_sync_single_for_device of DMA API ? (implies
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)
Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board). Our
ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing)
end up being partially corrupted when we try to parse the data in the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of 16byte
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA region
has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC.
440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is
L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is
based on
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual page.
It then uses pci_map_sg() to map to a scatter/gather buffer.
Perhaps I should convert these to the DMA API calls as you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent() should
work too
Thanks
Prodyut On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt
wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
Tom Burns wrote=20
Hi,
=20
Thank you everyone for your help.
=20
I've been looking into the other dma/pci API calls
(dma_alloc_coherent,
pci_alloc_consistent). I don't see how either of these return memory
mapped to a TLB with the I bit set to 1 in kernel 2.6.24. In our
kernel
code, the only use of the PPC44x_TLB_I define is in head_44x.S in
_start. We have CONFIG_NON_COHERENT_CACHE enabled.
=20
We changed our code to use dma_alloc_coherent, removed our manual
cacheline flushing, and saw the corrupted data return. To me this
means
dma_alloc_coherent cannot be setting the I=3D1 bit in the TLB entry.
=20
I tried, using our JTAG debugger (BDI3000), to pause operation after
calling dma_alloc_coherent to examine the TLB entry for the memory
returned by the call (which was just past
CONFIG_CONSISTENT_START=3D0xff100000). The TLB list loaded at the =
time
that I paused operation did not show a mapping for this area. I guess
the kernel swaps TLB entries on the fly so it isn't limited to only 64
entries? I will try to sleep in the same context as the
dma_alloc_coherent call to try to catch the TLB entry while loaded to
see if it has the I bit set.
=20
If that fails, any ideas?
=20
Thanks,
Tom Burns
International Datacasting Corporation
=20
There is also a patch that was submitted for 440EPX a couple of years
back. The 440EPX SOC causes hangs with Memory Read Multiple (MRM)
commands. Whether MRM is used or not depends on the value of
PCI_CACHE_LINE_SIZE register. I see that the changes are no longer
present in linux 2.6.30+ kernels. Although the patch certainly resolved
the hang issue with Silicon Image 680 PATA card as the 680 driver
attempts to use MRM commands - I don't know if it would resolve the data
corruption issue. It is certainly worth trying in my opinion. Below is a
link to the patch submission:
http://git.denx.de/?p=3Dlinux-2.6-denx.git;a=3Dcommit;h=3Dcffefde924123e6=
85327
48dd58fcb780eab5e219
Mikhail Zolotaryov wrote:
quoted
Hi Tom,
possible solution could be to use tasklet to perform DMA-related job
(as in most cases DMA transfer is interrupt driven - makes sense).
Tom Burns wrote:
quoted
Hi,
With the default config for the Sequoia board on 2.6.24, calling
pci_dma_sync_sg_for_cpu() results in executing
invalidate_dcache_range() in arch/ppc/kernel/misc.S from
__dma_sync(). This OOPses on PPC440 since it tries to call
directly
quoted
quoted
the assembly instruction dcbi, which can only be executed in
supervisor mode. We tried that before resorting to manual cache
line
quoted
quoted
management with usermode-safe assembly calls.
Regards,
Tom Burns
International Datacasting Corporation
Mikhail Zolotaryov wrote:
quoted
Hi,
Why manage cache lines manually, if appropriate code is a part of
__dma_sync / dma_sync_single_for_device of DMA API ? (implies
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)
Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board). Our
ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on
timing)
quoted
quoted
quoted
quoted
quoted
end up being partially corrupted when we try to parse the data
in
the
quoted
quoted
quoted
quoted
quoted
virtual page. We have confirmed the data is good before the PCI-
IDE
quoted
quoted
quoted
quoted
quoted
bridge. We are creating two 8K pages and map them to physical
DMA
quoted
quoted
quoted
quoted
quoted
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of
16byte
quoted
quoted
quoted
quoted
quoted
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA
region
quoted
quoted
quoted
quoted
has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines
appropriately,
quoted
quoted
quoted
quoted
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for
example on
quoted
quoted
quoted
quoted
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC.
440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there
is
quoted
quoted
quoted
quoted
quoted
L2C.
For example, in canyonlands.dts (460EX based board), we have the
L2C
quoted
quoted
quoted
quoted
quoted
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is
based on
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to
allocate
quoted
quoted
quoted
quoted
quoted
quoted
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual
page.
quoted
quoted
quoted
quoted
quoted
quoted
It then uses pci_map_sg() to map to a scatter/gather buffer.
Perhaps I should convert these to the DMA API calls as you
suggest.
quoted
quoted
quoted
quoted
quoted
quoted
Could you give more details on the consistency problem? It is a
good
quoted
quoted
quoted
quoted
quoted
idea to change to the new DMA APIs, but pci_alloc_consistent()
should
quoted
quoted
quoted
quoted
quoted
work too
Thanks
Prodyut On Thu, 2009-09-03 at 19:57 +1000, Benjamin
Herrenschmidt
quoted
quoted
quoted
quoted
quoted
wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-09-11 02:44:47
On Thu, 2009-09-10 at 13:30 -0700, Pravin Bathija wrote:
There is also a patch that was submitted for 440EPX a couple of years
back. The 440EPX SOC causes hangs with Memory Read Multiple (MRM)
commands. Whether MRM is used or not depends on the value of
PCI_CACHE_LINE_SIZE register. I see that the changes are no longer
present in linux 2.6.30+ kernels. Although the patch certainly resolved
the hang issue with Silicon Image 680 PATA card as the 680 driver
attempts to use MRM commands - I don't know if it would resolve the data
corruption issue. It is certainly worth trying in my opinion. Below is a
link to the patch submission:
http://git.denx.de/?p=linux-2.6-denx.git;a=commit;h=cffefde924123e685327
48dd58fcb780eab5e219
The changes in the above repository is a quick hack that can't be merged
as-is (and afaik hasn't been submitted).
If indeed we need to clamp the PCI cache line size on those critters,
then we need something in ppc4xx_pci.c to detect the need, set
pci_cache_line_size to 0 and eventually fixup the existing values in
devices in case u-boot don't have them right.
Care to send a patch ? :-)
Cheers,
Ben
From: Stefan Roese <sr@denx.de> Date: 2009-09-11 05:12:33
On Friday 11 September 2009 04:44:34 Benjamin Herrenschmidt wrote:
On Thu, 2009-09-10 at 13:30 -0700, Pravin Bathija wrote:
quoted
There is also a patch that was submitted for 440EPX a couple of years
back. The 440EPX SOC causes hangs with Memory Read Multiple (MRM)
commands. Whether MRM is used or not depends on the value of
PCI_CACHE_LINE_SIZE register. I see that the changes are no longer
present in linux 2.6.30+ kernels. Although the patch certainly resolved
the hang issue with Silicon Image 680 PATA card as the 680 driver
attempts to use MRM commands - I don't know if it would resolve the data
corruption issue. It is certainly worth trying in my opinion. Below is a
link to the patch submission:
http://git.denx.de/?p=linux-2.6-denx.git;a=commit;h=cffefde924123e685327
48dd58fcb780eab5e219
The changes in the above repository is a quick hack that can't be merged
as-is (and afaik hasn't been submitted).
If indeed we need to clamp the PCI cache line size on those critters,
then we need something in ppc4xx_pci.c to detect the need, set
pci_cache_line_size to 0 and eventually fixup the existing values in
devices in case u-boot don't have them right.
Care to send a patch ? :-)
It's already there. See commit:
5ce4b59653b2c2053cd9a011918ac1e4747f24cc
powerpc/4xx: Workaround for PPC440EPx/GRx PCI_28 Errata
Cheers,
Stefan
Stefan Roese wrote:
Correct. IIRC, some PATA driver as Pravin already mentioned.
Cheers,
Stefan
Thanks Stefan. The whole intention of the patch/hack (or whatever one might call it :) ) was to avoid rogue drivers from setting pci_cache_line_size to non-zero value even though the underlying hardware doesn't support MRM calls. Nonetheless this approach works only if the drivers use the kernel API for PCI config space access provided by the powerpc platform driver.
Regards,
Pravin
On Friday 11 September 2009 07:17:50 Benjamin Herrenschmidt wrote:
On Fri, 2009-09-11 at 07:12 +0200, Stefan Roese wrote:
quoted
It's already there. See commit:
5ce4b59653b2c2053cd9a011918ac1e4747f24cc
powerpc/4xx: Workaround for PPC440EPx/GRx PCI_28 Errata
Ok, that's another way to do it. Will catch nasty drivers who
try to write directly rather than clear pci_cache_line_size I suppose...
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-09-11 05:40:37
On Thu, 2009-09-10 at 22:35 -0700, Pravin Bathija wrote:
Thanks Stefan. The whole intention of the patch/hack (or whatever one
might call it :) ) was to avoid rogue drivers from setting
pci_cache_line_size to non-zero value even though the underlying
hardware doesn't support MRM calls. Nonetheless this approach works
only if the drivers use the kernel API for PCI config space access
provided by the powerpc platform driver.
Do you know many drivers that do config space accesses without using
the config space accessors ?
Such drivers should be banned to oblivion.
Cheers,
Ben.
Do you know many drivers that do config space accesses without using
the config space accessors ?
Such drivers should be banned to oblivion.
Cheers,
Ben.
I'm not aware of such drivers in the 2.6.30+ kernel.
Thanks,
Pravin
On Thu, 2009-09-10 at 22:35 -0700, Pravin Bathija wrote:
Thanks Stefan. The whole intention of the patch/hack (or whatever one
might call it :) ) was to avoid rogue drivers from setting
pci_cache_line_size to non-zero value even though the underlying
hardware doesn't support MRM calls. Nonetheless this approach works
only if the drivers use the kernel API for PCI config space access
provided by the powerpc platform driver.
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-09-11 02:00:15
On Thu, 2009-09-10 at 15:53 -0400, Tom Burns wrote:
Hi,
Thank you everyone for your help.
I've been looking into the other dma/pci API calls (dma_alloc_coherent,
pci_alloc_consistent). I don't see how either of these return memory
mapped to a TLB with the I bit set to 1 in kernel 2.6.24. In our kernel
code, the only use of the PPC44x_TLB_I define is in head_44x.S in
_start. We have CONFIG_NON_COHERENT_CACHE enabled.
It uses _PAGE_NO_CACHE which translates into the I bit in the TLB from
the TLB miss handler.
We changed our code to use dma_alloc_coherent, removed our manual
cacheline flushing, and saw the corrupted data return. To me this means
dma_alloc_coherent cannot be setting the I=1 bit in the TLB entry.
It must and it is. However, what are you allocating with
dma_alloc_coherent ? The actual data buffers must -also- be properly
cache managed, probably via dma_sync().
It would help tremendously if you sent us the source of the driver :-)
I tried, using our JTAG debugger (BDI3000), to pause operation after
calling dma_alloc_coherent to examine the TLB entry for the memory
returned by the call (which was just past
CONFIG_CONSISTENT_START=0xff100000). The TLB list loaded at the time
that I paused operation did not show a mapping for this area. I guess
the kernel swaps TLB entries on the fly so it isn't limited to only 64
entries? I will try to sleep in the same context as the
dma_alloc_coherent call to try to catch the TLB entry while loaded to
see if it has the I bit set.
Ben.
If that fails, any ideas?
Thanks,
Tom Burns
International Datacasting Corporation
Mikhail Zolotaryov wrote:
quoted
Hi Tom,
possible solution could be to use tasklet to perform DMA-related job
(as in most cases DMA transfer is interrupt driven - makes sense).
Tom Burns wrote:
quoted
Hi,
With the default config for the Sequoia board on 2.6.24, calling
pci_dma_sync_sg_for_cpu() results in executing
invalidate_dcache_range() in arch/ppc/kernel/misc.S from
__dma_sync(). This OOPses on PPC440 since it tries to call directly
the assembly instruction dcbi, which can only be executed in
supervisor mode. We tried that before resorting to manual cache line
management with usermode-safe assembly calls.
Regards,
Tom Burns
International Datacasting Corporation
Mikhail Zolotaryov wrote:
quoted
Hi,
Why manage cache lines manually, if appropriate code is a part of
__dma_sync / dma_sync_single_for_device of DMA API ? (implies
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)
Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board). Our
ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing)
end up being partially corrupted when we try to parse the data in the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of 16byte
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA region
has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC.
440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is
L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is
based on
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual page.
It then uses pci_map_sg() to map to a scatter/gather buffer.
Perhaps I should convert these to the DMA API calls as you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent() should
work too
Thanks
Prodyut On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt
wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
I tried, using our JTAG debugger (BDI3000), to pause operation after=20
calling dma_alloc_coherent to examine the TLB entry for the memory=20
returned by the call (which was just past=20
CONFIG_CONSISTENT_START=3D0xff100000). The TLB list loaded at the =
time=20
that I paused operation did not show a mapping for this area. I guess
the kernel swaps TLB entries on the fly so it isn't limited to only 64
entries? I will try to sleep in the same context as the=20
dma_alloc_coherent call to try to catch the TLB entry while loaded to=20
see if it has the I bit set.
If that fails, any ideas?
Sleeping won't cause the entry to appear at the TLB.
After the dma_alloc call, try to deference the pointer returned.
As a result of the dereference, a DataTLB Miss will happen which will
result in
the appropriate entry put in TLB.
Then do a JTAG break right after the dereference, and you should be able
to see the TLB entry.
Thanks
Prodyut
Mikhail Zolotaryov wrote:
Hi Tom,
possible solution could be to use tasklet to perform DMA-related job=20
(as in most cases DMA transfer is interrupt driven - makes sense).
Tom Burns wrote:
quoted
Hi,
With the default config for the Sequoia board on 2.6.24, calling=20
pci_dma_sync_sg_for_cpu() results in executing
invalidate_dcache_range() in arch/ppc/kernel/misc.S from=20
__dma_sync(). This OOPses on PPC440 since it tries to call directly=20
the assembly instruction dcbi, which can only be executed in=20
supervisor mode. We tried that before resorting to manual cache line
quoted
management with usermode-safe assembly calls.
Regards,
Tom Burns
International Datacasting Corporation
Mikhail Zolotaryov wrote:
quoted
Hi,
Why manage cache lines manually, if appropriate code is a part of=20
__dma_sync / dma_sync_single_for_device of DMA API ? (implies=20
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)
Prodyut Hazarika wrote:
quoted
Hi Adam,
=20
quoted
Yes, I am using the 440EPx (same as the sequoia board). Our=20
ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
=20
(using
=20
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on
timing)
quoted
quoted
quoted
quoted
end up being partially corrupted when we try to parse the data in
the
quoted
quoted
quoted
quoted
virtual page. We have confirmed the data is good before the
PCI-IDE
quoted
quoted
quoted
quoted
bridge. We are creating two 8K pages and map them to physical DMA
=20
memory
=20
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of
16byte
quoted
quoted
quoted
quoted
cache lines) is overwritten with old data from the last time the
=20
buffer
=20
quoted
was used. =20
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA region
quoted
quoted
quoted
has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines
appropriately,
quoted
quoted
quoted
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for example
on
quoted
quoted
quoted
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
=20
quoted
Hi Adam,
=20
quoted
Are you sure there is L2 cache on the 440?
=20
It depends on the SoC you are using. SoC like 460EX (Canyonlands
=20
board)
=20
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC.=20
440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is=20
L2C.
For example, in canyonlands.dts (460EX based board), we have the
L2C
quoted
quoted
quoted
quoted
entry.
L2C0: l2c {
...
}
=20
quoted
I am seeing this problem with our custom IDE driver which is=20
based on
=20
=20
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to
allocate
quoted
quoted
quoted
quoted
quoted
=20
the
=20
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual page.
quoted
quoted
quoted
quoted
quoted
It then uses pci_map_sg() to map to a scatter/gather buffer.=20
Perhaps I should convert these to the DMA API calls as you
suggest.
quoted
quoted
quoted
quoted
quoted
=20
Could you give more details on the consistency problem? It is a
good
quoted
quoted
quoted
quoted
idea to change to the new DMA APIs, but pci_alloc_consistent()
should
quoted
quoted
quoted
quoted
work too
Thanks
Prodyut On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt
quoted
quoted
quoted
quoted
wrote:
=20
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
=20
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the
#else
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif
Try adding _PAGE_COHERENT to the appropriate line above and see
if
quoted
quoted
quoted
quoted
quoted
quoted
=20
that =20
quoted
quoted
fixes your issue - this causes the 'M' bit to be set on the page
=20
which =20
quoted
quoted
sure enforce cache coherency. If it doesn't, you'll need to
check
quoted
quoted
quoted
quoted
quoted
quoted
=20
the =20
quoted
quoted
'M' bit isn't being masked out in head_44x.S (it was originally
=20
masked =20
quoted
quoted
out on arch/powerpc, but was fixed in later kernels when the
cache
quoted
quoted
quoted
quoted
quoted
quoted
=20
=20
quoted
quoted
quoted
coherency issues with non-SMP systems were resolved).
=20
I have some doubts about the usefulness of doing that for 4xx.
=20
AFAIK,
=20
quoted
quoted
the 440 core just ignores M.
The problem lies probably elsewhere. Maybe the L2 cache coherency
=20
isn't
=20
quoted
enabled or not working ?
The L1 cache on 440 is simply not coherent, so drivers have to
make
quoted
quoted
quoted
quoted
quoted
=20
sure
=20
quoted
they use the appropriate DMA APIs which will do cache flushing
when
quoted
quoted
quoted
quoted
quoted
needed.
Adam, what driver is causing you that sort of problems ?
Cheers,
Ben.
=20
--------------------------------------------------------
CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, =
is for the sole use of the intended recipient(s) and contains =
information that is confidential and proprietary to AppliedMicro =
Corporation or its subsidiaries. It is to be used solely for the purpose =
of furthering the parties' business relationship. All unauthorized =
review, use, disclosure or distribution is prohibited. If you are not =
the intended recipient, please contact the sender by reply e-mail and =
destroy all copies of the original message.
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-09-11 01:56:08
On Wed, 2009-09-09 at 09:43 -0400, Tom Burns wrote:
Hi,
With the default config for the Sequoia board on 2.6.24, calling
pci_dma_sync_sg_for_cpu() results in executing
invalidate_dcache_range() in arch/ppc/kernel/misc.S from __dma_sync().
This OOPses on PPC440 since it tries to call directly the assembly
instruction dcbi, which can only be executed in supervisor mode. We
tried that before resorting to manual cache line management with
usermode-safe assembly calls.
Wait a minute.... usermode ? You are doing all of that from userspace ?
I don't understand the story here. You can't call all those kernel APIs
form userspace in the first place, indeed... But then an IDE driver has
nothing to do in userspace neither.
Cheers,
Ben.
Regards,
Tom Burns
International Datacasting Corporation
Mikhail Zolotaryov wrote:
quoted
Hi,
Why manage cache lines manually, if appropriate code is a part of
__dma_sync / dma_sync_single_for_device of DMA API ? (implies
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)
Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board). Our
ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing)
end up being partially corrupted when we try to parse the data in the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of 16byte
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA region has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC. 440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is based on
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual page.
It then uses pci_map_sg() to map to a scatter/gather buffer.
Perhaps I should convert these to the DMA API calls as you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent() should
work too
Thanks
Prodyut
On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
From: Tom Burns <hidden> Date: 2009-09-11 14:01:36
Hi Ben,
Benjamin Herrenschmidt wrote:
On Wed, 2009-09-09 at 09:43 -0400, Tom Burns wrote:
quoted
Hi,
With the default config for the Sequoia board on 2.6.24, calling
pci_dma_sync_sg_for_cpu() results in executing
invalidate_dcache_range() in arch/ppc/kernel/misc.S from __dma_sync().
This OOPses on PPC440 since it tries to call directly the assembly
instruction dcbi, which can only be executed in supervisor mode. We
tried that before resorting to manual cache line management with
usermode-safe assembly calls.
Wait a minute.... usermode ? You are doing all of that from userspace ?
I don't understand the story here. You can't call all those kernel APIs
form userspace in the first place, indeed... But then an IDE driver has
nothing to do in userspace neither.
Sorry, I was referring to whether or not the CPU is in supervisor mode.
Our code is all in the kernel, not userspace :). I can see now from the
value of MSR at the time of the OOPS that the processor was in
supervisor mode, I missed that earlier. Looks like you're right about
the bad address, I will investigate.
Thanks,
Tom
Cheers,
Ben.
quoted
Regards,
Tom Burns
International Datacasting Corporation
Mikhail Zolotaryov wrote:
quoted
Hi,
Why manage cache lines manually, if appropriate code is a part of
__dma_sync / dma_sync_single_for_device of DMA API ? (implies
CONFIG_NOT_COHERENT_CACHE enabled, as default for Sequoia Board)
Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Yes, I am using the 440EPx (same as the sequoia board). Our
ideDriver is DMA'ing blocks of 192-byte data over the PCI bus
(using
quoted
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing)
end up being partially corrupted when we try to parse the data in the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA
memory
quoted
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of 16byte
cache lines) is overwritten with old data from the last time the
buffer
quoted
was used.
This looks like a cache coherency problem.
Can you ensure that the TLB entries corresponding to the DMA region has
the CacheInhibit bit set.
You will need a BDI connected to your system.
Also, you will need to invalidate and flush the lines appropriately,
since in 440 cores,
L1Cache coherency is managed entirely by software.
Please look at drivers/net/ibm_newemac/mal.c and core.c for example on
how to do it.
Thanks
Prodyut
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands
board)
quoted
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC. 440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is based on
quoted
quoted
pretty old code. Our driver uses pci_alloc_consistent() to allocate
the
quoted
quoted
physical DMA memory and alloc_pages() to allocate a virtual page.
It then uses pci_map_sg() to map to a scatter/gather buffer.
Perhaps I should convert these to the DMA API calls as you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent() should
work too
Thanks
Prodyut
On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-09-08 21:29:52
On Tue, 2009-09-08 at 14:01 -0400, Adam Zilkie wrote:
Hi Prodyut,
Yes, I am using the 440EPx (same as the sequoia board).
Our ideDriver is DMA'ing blocks of 192-byte data over the PCI bus (using
the Sil0680A PCI-IDE bridge). Most of the DMA's (depending on timing)
end up being partially corrupted when we try to parse the data in the
virtual page. We have confirmed the data is good before the PCI-IDE
bridge. We are creating two 8K pages and map them to physical DMA memory
using single-entry scatter/gather structs. When a DMA block is
corrupted, we see a random portion of it (always a multiple of 16byte
cache lines) is overwritten with old data from the last time the buffer
was used.
Smells to me like you aren't properly using the dma or pci dma APIs to
flush/invalidate the cache around your transfers.
Ben.
Regards,
Adam
On Thu, 2009-09-03 at 13:27 -0700, Prodyut Hazarika wrote:
quoted
Hi Adam,
quoted
Are you sure there is L2 cache on the 440?
It depends on the SoC you are using. SoC like 460EX (Canyonlands board)
have L2Cache.
It seems you are using a Sequoia board, which has a 440EPx SoC. 440EPx
has a 440 cpu core, but no L2Cache.
Could you please tell me which SoC you are using?
You can also refer to the appropriate dts file to see if there is L2C.
For example, in canyonlands.dts (460EX based board), we have the L2C
entry.
L2C0: l2c {
...
}
quoted
I am seeing this problem with our custom IDE driver which is based on
pretty old code. Our driver uses pci_alloc_consistent() to allocate the
quoted
physical DMA memory and alloc_pages() to allocate a virtual page. It
then uses pci_map_sg() to map to a scatter/gather buffer. Perhaps I
should convert these to the DMA API calls as you suggest.
Could you give more details on the consistency problem? It is a good
idea to change to the new DMA APIs, but pci_alloc_consistent() should
work too
Thanks
Prodyut
On Thu, 2009-09-03 at 19:57 +1000, Benjamin Herrenschmidt wrote:
quoted
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
quoted
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
#else
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif
Try adding _PAGE_COHERENT to the appropriate line above and see if
that
quoted
quoted
fixes your issue - this causes the 'M' bit to be set on the page
which
quoted
quoted
sure enforce cache coherency. If it doesn't, you'll need to check
the
quoted
quoted
'M' bit isn't being masked out in head_44x.S (it was originally
masked
quoted
quoted
out on arch/powerpc, but was fixed in later kernels when the cache
coherency issues with non-SMP systems were resolved).
I have some doubts about the usefulness of doing that for 4xx. AFAIK,
the 440 core just ignores M.
The problem lies probably elsewhere. Maybe the L2 cache coherency
isn't
quoted
enabled or not working ?
The L1 cache on 440 is simply not coherent, so drivers have to make
sure
quoted
they use the appropriate DMA APIs which will do cache flushing when
needed.
Adam, what driver is causing you that sort of problems ?
Cheers,
Ben.
Hi,
This doesn't seem right. If we are talking about a single CPU core chip,
i.e., just one data cache, then setting M is typically a) useless and
could even b) cause a performance penalty depending on a chip's
implementation.
The M bit is required if *other* cores with caches need to see changes
for coherency of their caches. You wouldn't set it for one core only
because your own core knows about its own cache.
The possible performance penalty could happen because you need some way
to tell the others that they better intercept a transaction. And that
could, depending on the chip, by a clock extra or so per transaction.
Now, in theory, a DMA engine could have caches, read from cache content
first, and could snoop the bus on global transactions like another core,
but I have never heard of such a beast.=20
Hope this helps,
Heinz
-----Original Message-----
From: linuxppc-dev-bounces+heinz.wrobel=3Dfreescale.com@lists.ozlabs.org
[mailto:linuxppc-dev-bounces+heinz.wrobel=3Dfreescale.com@lists.ozlabs.or=
g
] On Behalf Of Chris Pringle
Sent: Donnerstag, 3. September 2009 10:05
To: azilkie@datacast.com
Cc: Tom Burns; Andrea Zypchen; linuxppc-dev@lists.ozlabs.org
Subject: Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
section:
#ifdef CONFIG_44x
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_GUARDED)
#else
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif
Try adding _PAGE_COHERENT to the appropriate line above and see if that
fixes your issue - this causes the 'M' bit to be set on the page which
sure enforce cache coherency. If it doesn't, you'll need to check the
'M' bit isn't being masked out in head_44x.S (it was originally masked
out on arch/powerpc, but was fixed in later kernels when the cache
coherency issues with non-SMP systems were resolved).
The patch I had fixed two problems on 2.6.26 for 'powerpc':
1) It stopped the 'M' bit being masked out (head_32.S)
2) It set the cache coherency ('M' bit) flag on each page table entry
(pgtable-ppc32.h)
Hope this helps!
Cheers,
Chris
Adam Zilkie wrote:
Hi Chris,
I am having a problem similar to what you described in this
discussion.
We are using the ppc arch with 2.6.24 with CONFIG_SEQUOIA with=20
compiles arch/ppc/kernel/head_44x.c (quite different from=20
/arch/powerpc/kernel/head_32.S). I would like to apply your=20
backporting patch to this architecture. Any help would be appreciated.
Regards,
Adam
=20
--=20
______________________________
Chris Pringle
Software Design Engineer
Miranda Technologies Ltd.
Hithercroft Road
Wallingford
Oxfordshire OX10 9DG
UK
Tel. +44 1491 820206
Fax. +44 1491 820001
www.miranda.com
____________________________
Miranda Technologies Limited
Registered in England and Wales CN 02017053 Registered Office: James
House, Mere Park, Dedmere Road, Marlow, Bucks, SL7 1FJ
_______________________________________________
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
From: Chris Pringle <hidden> Date: 2009-09-03 12:45:57
In our case, we were suffering coherency issues on an 8260 when using
DMA with PCI. Setting the 'M' bit cured all of our DMA coherency issues.
There is a comment in "pgtable-ppc32.h" on 2.6.29.6 that says:
"We always set _PAGE_COHERENT when SMP is enabled *or* the processor
might need it for DMA coherency". Freescale had also suggested setting
the 'M' bit when we submitted a support request.
I've no idea how this bit affects other PowerPC chips. Looking briefly
through some of the header files, it looks as if the 'M' bit should not
be set for 44x, so the issue is probably not the same as the one I had.
Cheers,
Chris
Wrobel Heinz-R39252 wrote:
Hi,
This doesn't seem right. If we are talking about a single CPU core chip,
i.e., just one data cache, then setting M is typically a) useless and
could even b) cause a performance penalty depending on a chip's
implementation.
The M bit is required if *other* cores with caches need to see changes
for coherency of their caches. You wouldn't set it for one core only
because your own core knows about its own cache.
The possible performance penalty could happen because you need some way
to tell the others that they better intercept a transaction. And that
could, depending on the chip, by a clock extra or so per transaction.
Now, in theory, a DMA engine could have caches, read from cache content
first, and could snoop the bus on global transactions like another core,
but I have never heard of such a beast.
Hope this helps,
Heinz
-----Original Message-----
From: linuxppc-dev-bounces+heinz.wrobel=freescale.com@lists.ozlabs.org
[mailto:linuxppc-dev-bounces+heinz.wrobel=freescale.com@lists.ozlabs.org
] On Behalf Of Chris Pringle
Sent: Donnerstag, 3. September 2009 10:05
To: azilkie@datacast.com
Cc: Tom Burns; Andrea Zypchen; linuxppc-dev@lists.ozlabs.org
Subject: Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
section:
#ifdef CONFIG_44x
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_GUARDED)
#else
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif
Try adding _PAGE_COHERENT to the appropriate line above and see if that
fixes your issue - this causes the 'M' bit to be set on the page which
sure enforce cache coherency. If it doesn't, you'll need to check the
'M' bit isn't being masked out in head_44x.S (it was originally masked
out on arch/powerpc, but was fixed in later kernels when the cache
coherency issues with non-SMP systems were resolved).
The patch I had fixed two problems on 2.6.26 for 'powerpc':
1) It stopped the 'M' bit being masked out (head_32.S)
2) It set the cache coherency ('M' bit) flag on each page table entry
(pgtable-ppc32.h)
Hope this helps!
Cheers,
Chris
Adam Zilkie wrote:
quoted
Hi Chris,
I am having a problem similar to what you described in this
discussion.
quoted
We are using the ppc arch with 2.6.24 with CONFIG_SEQUOIA with
compiles arch/ppc/kernel/head_44x.c (quite different from
/arch/powerpc/kernel/head_32.S). I would like to apply your
backporting patch to this architecture. Any help would be appreciated.
Regards,
Adam
____________________________
Miranda Technologies Limited
Registered in England and Wales CN 02017053
Registered Office: James House, Mere Park, Dedmere Road, Marlow, Bucks, SL7 1FJ
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2009-09-06 21:32:42
On Thu, 2009-09-03 at 13:20 +0100, Wrobel Heinz-R39252 wrote:
Hi,
This doesn't seem right. If we are talking about a single CPU core chip,
i.e., just one data cache, then setting M is typically a) useless and
could even b) cause a performance penalty depending on a chip's
implementation.
The M bit is required if *other* cores with caches need to see changes
for coherency of their caches. You wouldn't set it for one core only
because your own core knows about its own cache.
The possible performance penalty could happen because you need some way
to tell the others that they better intercept a transaction. And that
could, depending on the chip, by a clock extra or so per transaction.
Now, in theory, a DMA engine could have caches, read from cache content
first, and could snoop the bus on global transactions like another core,
but I have never heard of such a beast.
Actually there are some freescale part, afaik, that require M for proper
cache coherency :-) I don't have names off the top of my mind, I think
it has to be with PCI inbound buffers.
In this case, however, it's 440 on which I believe M is simply ignored.
Cheers,
Ben.
Hope this helps,
Heinz
-----Original Message-----
From: linuxppc-dev-bounces+heinz.wrobel=freescale.com@lists.ozlabs.org
[mailto:linuxppc-dev-bounces+heinz.wrobel=freescale.com@lists.ozlabs.org
] On Behalf Of Chris Pringle
Sent: Donnerstag, 3. September 2009 10:05
To: azilkie@datacast.com
Cc: Tom Burns; Andrea Zypchen; linuxppc-dev@lists.ozlabs.org
Subject: Re: AW: PowerPC PCI DMA issues (prefetch/coherency?)
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following
section:
#ifdef CONFIG_44x
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_GUARDED)
#else
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif
Try adding _PAGE_COHERENT to the appropriate line above and see if that
fixes your issue - this causes the 'M' bit to be set on the page which
sure enforce cache coherency. If it doesn't, you'll need to check the
'M' bit isn't being masked out in head_44x.S (it was originally masked
out on arch/powerpc, but was fixed in later kernels when the cache
coherency issues with non-SMP systems were resolved).
The patch I had fixed two problems on 2.6.26 for 'powerpc':
1) It stopped the 'M' bit being masked out (head_32.S)
2) It set the cache coherency ('M' bit) flag on each page table entry
(pgtable-ppc32.h)
Hope this helps!
Cheers,
Chris
Adam Zilkie wrote:
quoted
Hi Chris,
I am having a problem similar to what you described in this
discussion.
quoted
We are using the ppc arch with 2.6.24 with CONFIG_SEQUOIA with
compiles arch/ppc/kernel/head_44x.c (quite different from
/arch/powerpc/kernel/head_32.S). I would like to apply your
backporting patch to this architecture. Any help would be appreciated.
Regards,
Adam
From: Adam Zilkie <hidden> Date: 2009-09-03 16:07:55
Chris,
I noticed the following comment in pgtable.h:
* - CACHE COHERENT bit (M) has no effect on PPC440 core, because it
* doesn't support SMP. So we can use this as software bit, like
* DIRTY.
And _PAGE_COHERENT is not defined for the 44x (giving a compile error
when I add it the _PAGE_BASE line as you suggested). This would confirm
that the M bit is meaningless for the PPC440
Regards,
Adam
On Thu, 2009-09-03 at 09:05 +0100, Chris Pringle wrote:
Hi Adam,
If you have a look in include/asm-ppc/pgtable.h for the following section:
#ifdef CONFIG_44x
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED | _PAGE_GUARDED)
#else
#define _PAGE_BASE (_PAGE_PRESENT | _PAGE_ACCESSED)
#endif
Try adding _PAGE_COHERENT to the appropriate line above and see if that
fixes your issue - this causes the 'M' bit to be set on the page which
sure enforce cache coherency. If it doesn't, you'll need to check the
'M' bit isn't being masked out in head_44x.S (it was originally masked
out on arch/powerpc, but was fixed in later kernels when the cache
coherency issues with non-SMP systems were resolved).
The patch I had fixed two problems on 2.6.26 for 'powerpc':
1) It stopped the 'M' bit being masked out (head_32.S)
2) It set the cache coherency ('M' bit) flag on each page table entry
(pgtable-ppc32.h)
Hope this helps!
Cheers,
Chris
Adam Zilkie wrote:
quoted
Hi Chris,
I am having a problem similar to what you described in this discussion.
We are using the ppc arch with 2.6.24 with CONFIG_SEQUOIA with compiles
arch/ppc/kernel/head_44x.c (quite different
from /arch/powerpc/kernel/head_32.S). I would like to apply your
backporting patch to this architecture. Any help would be appreciated.
Regards,
Adam
--
Adam Zilkie
Software Designer,
International Datacasting Corp.
This message and the documents attached hereto are intended only for the addressee and may contain privileged or confidential information. Any unauthorized disclosure is strictly prohibited. If you have received this message in error, please notify us immediately so that we may correct our internal records. Please then delete the original message. Thank you.