On Tue, Dec 04, 2007 at 01:14:43PM +1100, Benjamin Herrenschmidt wrote:

 .../... (snip scary bunch of errata)

quoted

- "FEr PCI-#4" (Detailed by Application Note AN-84):

[This isn't strictly a coherency issue but having coherency enabled
exacerbates the problem.]  Basically, the bridge can let the cpu read
a pci device's registers before all of the data the PCI devices has
written has actually made it to memory.  This and the fact that the
device's write data may be stuck in the PCI Slave Write Buffer
(which isn't checked for coherency), the cpu can get stale data.

There are no plans to fix that erratum.

So if I understand correctly, there's no plan to fix a major PCI spec
violation which prevent any kind of reliable implementation whatsoever ?

That's just for that particular part (e.g., 64360).  Newer parts like
the 64460 have it fixed.

quoted

So, the answer depends on what part & what rev of the part you have
(e.g., the pegasos doesn't use the MPSC and apparently has the other
issues worked around so it can turn on coherency but the prpmc2800
doesn't so it needs coherency off).

BTW, I haven't forgotten the inherent bug you described when coherency
is off (/me too lazy to find link to the email) but AFAIK I've never run
into it.  However, if I turn on coherency and stress the PCI bus, it
hangs (I can't even look at memory thru a bdi).

Well, as it is today, the "classic" MMU code cannot deal with !coherent.
The entire linear mapping is always mapped cacheable with BATs, so stuff
may be brought into the cache at any time, potentially polluting DMA
data.

Dealing with that would be hard. It might be possible by using G on the
entire linear mapping like we do on 4xx (yuck), and/or by not using
D-BATs (the kernel will blow up in various areas without I-BATs).

Hrm, I didn't realize it was in such bad shape.  I'll have to take a
closer look.

Mark