From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2011-03-24 22:17:10
Hi Takashi !
While working on endian-fixing xHCI with Matt (CC), we discovered the
source of our problems with usb-audio on a board we were working on.
c32d977b8157bf67cdf47729ce7dd054a26eb534
"ALSA: pcm - Call pgprot_noncached() for vmalloc'ed buffers"
I'm afraid that this is totally bogus :-)
I don't know on what arch it is safe to have the same memory be mapped
cachable in the kernel (and accessed via this cached mapping) and
non-cachable in userspace, but I can confidently say that wherever it
works it does so by accident.
In the case of usb-audio, what we observed is that the user application
was writing samples using an uncached mapping, so directly to memory,
which does -not- invalidate conflicting cache lines on the way, an the
kernel would then memcpy those data to the USB buffers using a cached
mapping (vmalloc) and essentially get stale stuff from the cache instead
of the real samples.
Worse, on some processors, it's actually -illegal- to create (and even
more to -access-) a conflicting mapping of a page of memory, ie, have it
mapped cached somewhere and uncached somewhere else. It will lockup some
processors and afaik, some x86 as well.
In fact, cache coherent architectures often don't support mapping memory
uncached -at-all- so something like snd_pcm_lib_mmap_noncached()
shouldn't exist, or at least be under arch control. There's no case
where it's "always safe". There will almost always be a cache alias in
the linear mapping unless special arch specific sauce has been applied.
Now, there's another problem on top of that, which is that
snd_pcm_default_mmap() will not work properly the "other way around" on
powerpc, where the mapping -needs- to be uncached bcs you are running on
a non cache coherent embedded CPU and trying to mmap DMA memory, but
that's something that needs fixing inside powerpc by properly defining
dma_mmap_coherent() & ARCH_HAS_DMA_MMAP_COHERENT (I thought we had added
it a while back but it's not upstream, patch must have got lost). We
must also make sure we don't go down that path for vmalloc memory
though.
Cheers,
Ben.
At Fri, 25 Mar 2011 09:16:48 +1100,
Benjamin Herrenschmidt wrote:
Hi Takashi !
While working on endian-fixing xHCI with Matt (CC), we discovered the
source of our problems with usb-audio on a board we were working on.
c32d977b8157bf67cdf47729ce7dd054a26eb534
"ALSA: pcm - Call pgprot_noncached() for vmalloc'ed buffers"
I'm afraid that this is totally bogus :-)
I don't know on what arch it is safe to have the same memory be mapped
cachable in the kernel (and accessed via this cached mapping) and
non-cachable in userspace, but I can confidently say that wherever it
works it does so by accident.
In the case of usb-audio, what we observed is that the user application
was writing samples using an uncached mapping, so directly to memory,
which does -not- invalidate conflicting cache lines on the way, an the
kernel would then memcpy those data to the USB buffers using a cached
mapping (vmalloc) and essentially get stale stuff from the cache instead
of the real samples.
Worse, on some processors, it's actually -illegal- to create (and even
more to -access-) a conflicting mapping of a page of memory, ie, have it
mapped cached somewhere and uncached somewhere else. It will lockup some
processors and afaik, some x86 as well.
In fact, cache coherent architectures often don't support mapping memory
uncached -at-all- so something like snd_pcm_lib_mmap_noncached()
shouldn't exist, or at least be under arch control. There's no case
where it's "always safe". There will almost always be a cache alias in
the linear mapping unless special arch specific sauce has been applied.
I see. I'll take your removal patch.
It should be applied to stable kernel, too, right?
Now, there's another problem on top of that, which is that
snd_pcm_default_mmap() will not work properly the "other way around" on
powerpc, where the mapping -needs- to be uncached bcs you are running on
a non cache coherent embedded CPU and trying to mmap DMA memory, but
that's something that needs fixing inside powerpc by properly defining
dma_mmap_coherent() & ARCH_HAS_DMA_MMAP_COHERENT (I thought we had added
it a while back but it's not upstream, patch must have got lost).
Mea culpa. I moved to a different team and have had little time for
the upstream development since then...
We
must also make sure we don't go down that path for vmalloc memory
though.
Yes.
Your patch looks good. Thanks for taking care of this!
Takashi
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2011-03-25 09:15:54
On Fri, 2011-03-25 at 09:01 +0100, Takashi Iwai wrote:
quoted
In fact, cache coherent architectures often don't support mapping memory
uncached -at-all- so something like snd_pcm_lib_mmap_noncached()
shouldn't exist, or at least be under arch control. There's no case
where it's "always safe". There will almost always be a cache alias in
the linear mapping unless special arch specific sauce has been applied.
I see. I'll take your removal patch.
It should be applied to stable kernel, too, right?
I suppose so :-)
quoted
Now, there's another problem on top of that, which is that
snd_pcm_default_mmap() will not work properly the "other way around" on
powerpc, where the mapping -needs- to be uncached bcs you are running on
a non cache coherent embedded CPU and trying to mmap DMA memory, but
that's something that needs fixing inside powerpc by properly defining
dma_mmap_coherent() & ARCH_HAS_DMA_MMAP_COHERENT (I thought we had added
it a while back but it's not upstream, patch must have got lost).
Mea culpa. I moved to a different team and have had little time for
the upstream development since then...
That's ok, I did a new one, which you have noticed already anyways :-)
quoted
We
must also make sure we don't go down that path for vmalloc memory
though.
Yes.
I haven't actually checked, but I assume that the test
substream->dma_buffer.dev.type == SNDRV_DMA_TYPE_DEV
In snd_pcm_default_mmap() takes care of that, please correct me if
I'm wrong in which case we'll need something else there.
Your patch looks good. Thanks for taking care of this!
Are you taking care of sending it upstream ?
Cheers,
Ben.
At Fri, 25 Mar 2011 20:15:33 +1100,
Benjamin Herrenschmidt wrote:
quoted
quoted
We
must also make sure we don't go down that path for vmalloc memory
though.
Yes.
I haven't actually checked, but I assume that the test
substream->dma_buffer.dev.type == SNDRV_DMA_TYPE_DEV
In snd_pcm_default_mmap() takes care of that, please correct me if
I'm wrong in which case we'll need something else there.
Well, in the case of usb-audio, it's not handled via
dma_mmap_coherent(), as the page isn't allocated via
dma_alloc_coherent() but vmalloc().
The bad commit was to overcome some problems on SH platform, IIRC,
when it's used with dmix -- i.e. concurrent accesses on the mmapped
buffer from multiple processes. But, this looks obviously like a
wrong approach.
Actually, the buffer allocated there in usb-audio is an intermediate
buffer, that isn't directly transferred to hardware. We may need a
bit more consideration what is the best way to solve that issue (if
it's still really present).
quoted
Your patch looks good. Thanks for taking care of this!
Are you taking care of sending it upstream ?
I'll apply the patch to remove vmalloc pgprot thingy surely to sound
tree and include in the next pull request.
Others should be sent through arch maintainers (PPC and ARM), right?
thanks,
Takashi
From: Benjamin Herrenschmidt <benh@kernel.crashing.org> Date: 2011-03-25 10:23:47
On Fri, 2011-03-25 at 11:12 +0100, Takashi Iwai wrote:
At Fri, 25 Mar 2011 20:15:33 +1100,
Benjamin Herrenschmidt wrote:
quoted
quoted
quoted
We
must also make sure we don't go down that path for vmalloc memory
though.
Yes.
I haven't actually checked, but I assume that the test
substream->dma_buffer.dev.type == SNDRV_DMA_TYPE_DEV
In snd_pcm_default_mmap() takes care of that, please correct me if
I'm wrong in which case we'll need something else there.
Well, in the case of usb-audio, it's not handled via
dma_mmap_coherent(), as the page isn't allocated via
dma_alloc_coherent() but vmalloc().
Right, I just wanted to make sure I was right to assume that a page
allocated by vmalloc() was going to fail the above test in
snd_pcm_default_mmap() and thus -not- get into dma_mmap_coherent()...
just double checking as I'm not totally familiar with the intricacies of
the pcm code :-)
The bad commit was to overcome some problems on SH platform, IIRC,
when it's used with dmix -- i.e. concurrent accesses on the mmapped
buffer from multiple processes. But, this looks obviously like a
wrong approach.
Is this a vivt architecture ? Maybe enforcing some restrictions on the
virtual addresses so they hit the same cache congruence classes ?
Actually, the buffer allocated there in usb-audio is an intermediate
buffer, that isn't directly transferred to hardware. We may need a
bit more consideration what is the best way to solve that issue (if
it's still really present).
Right. I wouldn't expect vmalloc stuff to hit HW in most cases anyways,
though I do wonder why you don't pass the buffer directly to the HCD and
avoid that intermediate step but that's a completely different
question :-)
quoted
quoted
Your patch looks good. Thanks for taking care of this!
Are you taking care of sending it upstream ?
I'll apply the patch to remove vmalloc pgprot thingy surely to sound
tree and include in the next pull request.
Others should be sent through arch maintainers (PPC and ARM), right?
Well, I am ppc maintainer so that's sorted :-) I've CCed Russell for the
other, it's up to him, I have no specific dependency there, it's just an
easy cleanup I stumbled upon.
Cheers,
Ben.
thanks,
Takashi
--
To unsubscribe from this list: send the line "unsubscribe linux-arch" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
At Fri, 25 Mar 2011 21:23:26 +1100,
Benjamin Herrenschmidt wrote:
On Fri, 2011-03-25 at 11:12 +0100, Takashi Iwai wrote:
quoted
At Fri, 25 Mar 2011 20:15:33 +1100,
Benjamin Herrenschmidt wrote:
quoted
quoted
quoted
We
must also make sure we don't go down that path for vmalloc memory
though.
Yes.
I haven't actually checked, but I assume that the test
substream->dma_buffer.dev.type == SNDRV_DMA_TYPE_DEV
In snd_pcm_default_mmap() takes care of that, please correct me if
I'm wrong in which case we'll need something else there.
Well, in the case of usb-audio, it's not handled via
dma_mmap_coherent(), as the page isn't allocated via
dma_alloc_coherent() but vmalloc().
Right, I just wanted to make sure I was right to assume that a page
allocated by vmalloc() was going to fail the above test in
snd_pcm_default_mmap() and thus -not- get into dma_mmap_coherent()...
just double checking as I'm not totally familiar with the intricacies of
the pcm code :-)
quoted
The bad commit was to overcome some problems on SH platform, IIRC,
when it's used with dmix -- i.e. concurrent accesses on the mmapped
buffer from multiple processes. But, this looks obviously like a
wrong approach.
Is this a vivt architecture ? Maybe enforcing some restrictions on the
virtual addresses so they hit the same cache congruence classes ?
Possibly, yes. But we need to re-test the problem first with the
recent kernel.
quoted
Actually, the buffer allocated there in usb-audio is an intermediate
buffer, that isn't directly transferred to hardware. We may need a
bit more consideration what is the best way to solve that issue (if
it's still really present).
Right. I wouldn't expect vmalloc stuff to hit HW in most cases anyways,
though I do wonder why you don't pass the buffer directly to the HCD and
avoid that intermediate step but that's a completely different
question :-)
The direct buffer was used partly in the older versions. But there
was a problem in the accuracy of the buffer playback position, so this
was switched to full double-buffering.
thanks,
Takashi