DMA consistent allocation is limited to 2MB on 2.6.11 for ppc440?

Hi,

The page table 'consistent_pte' covers the uncached DMA consistent
allocation space. Its size is only one page, each page has 512 PTEs.
That means only 2MB memory are available for DMA. For some
applications this is not enough. So how to eliminate this limitation?

I noticed that there are two configuration macros related to this
issue: CONFIG_CONSISTENT_START and CONFIG_CONSISTENT_SIZE. The current
value for CONFIG_CONSISTENT_SIZE is 2MB, which is exactly able to be
covered by one page. I guess that's why consistent_pte is initialized
to just one page. So simply changing CONFIG_CONSISTENT_SIZE cannot
remove the 2MB limitation. dma_alloc_init() has to be modified to
initialize consisten_pte to some pages which are enough to cover
CONFIG_CONSISTENT_SIZE memory space. Right?

Does increasing CONSISTENT_SIZE have any side effects?

Thanks,
-Shawn.

On Wed, May 18, 2005 at 05:54:39PM -0700, Shawn Jin wrote:

The page table 'consistent_pte' covers the uncached DMA consistent
allocation space. Its size is only one page, each page has 512 PTEs.
That means only 2MB memory are available for DMA. For some
applications this is not enough. So how to eliminate this limitation?

No, _all_ physical memory is available for DMA. "consistent" pool is 
used for small non-cached allocations, e.g. buffer descriptors, etc. 
Don't use it for actual data buffers.

-- 
Eugene

No, _all_ physical memory is available for DMA. "consistent" pool is
used for small non-cached allocations, e.g. buffer descriptors, etc.
Don't use it for actual data buffers.

We used to pci_alloc_consistent() to allocate DMA buffers on 2.4.
There was no problem to allocate more than 2MB buffer. The
implementation of pci_alloc_consistent() on 2.6 differs from that on
2.4, which calls dma_alloc_coherent(). Following this, I traced down
to 'consistent_pte', which is the root cause of 2MB limitation.

Then what's your recommendation to allocate a big chunk of memory for
pci device? From this perspective, 2.6 is not compatible with 2.4,
which I think, is pretty bad.

Thanks,
-Shawn.

On Wed, May 18, 2005 at 06:52:52PM -0700, Shawn Jin wrote:

Then what's your recommendation to allocate a big chunk of memory for
pci device? 

My recommendation - don't do this. Why do you need to allocate this 
big chunk of consistent memory in the first place? You can do DMA 
_without_ allocating "consistent" memory. In fact, this is how 
virtually all devices work in Linux. For more info about DMA API - 
look at Documentation/DMA-API.txt.

Technically, you can make consistent pool bigger, if you really insist 
on using this approach.

-- 
Eugene

My recommendation - don't do this. Why do you need to allocate this
big chunk of consistent memory in the first place? You can do DMA
_without_ allocating "consistent" memory. In fact, this is how
virtually all devices work in Linux. For more info about DMA API -
look at Documentation/DMA-API.txt.

The driver stack we've been developing (already 4.0) uses consistent
memory a lot for DMA. The stack is for many kinds of high performance
storage IO e.g. iSCSI, FC. It works fine on 2.4.x because there is no
such 2MB consistent pool limitation.

Technically, you can make consistent pool bigger, if you really insist
on using this approach.

I want to understand the motivation and the rationale of choosing
'consistent_pte' on 2.6.x, what impact there would be to increase the
consistent pool, and so on. Some pointers to articles, posts are more
helpful.

Thanks a lot,
-Shawn.