Re: [PATCH 0/3] x86_64, sfc: 128-bit memory-mapped I/O

On Tue, 2012-08-21 at 18:59 -0700, H. Peter Anvin wrote:

On 08/21/2012 06:43 PM, Ben Hutchings wrote:

quoted

On Tue, 2012-08-21 at 18:38 -0700, H. Peter Anvin wrote:

quoted

On 08/21/2012 06:17 PM, Ben Hutchings wrote:

quoted

Current Solarflare network controllers have 128-bit memory-mapped
registers which are normally accessed through a series of I/O
operations.  However, it is also possible to access them with a single
MOVAPS instruction on x86_64, and this is measurably faster as it
requires only one PCIe transaction.

Also, have you considered doing this with write combining instead?

We tried it, and it goes horribly wrong.  On some systems, the writes
are not combined, but they are reordered in a way the hardware doesn't
support.  See the comment at the top of drivers/net/ethernet/sfc/io.h.

Yes, you have to make sure you properly enforce the necessary ordering
requirements manually (I think you can do that with sfence).

We did put an sfence after the writes to each register.  But some
systems only want to combine writes that cover an entire cache line, and
the writes covering a 128-bit register get broken back up into multiple
writes at the PCIe level.  And on some systems these are sent in
decreasing address order, which breaks the rules for writing to
TX_DESC_UPD.

To avoid this we'd have to put an sfence in between the writes to a
register, leaving us back where we started.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.