Re: RDMA power failure write atomicity

There are several factors that make this problem hard. For many modern
servers, DMA data is written to last level cache via DDIO, i.e., it
will not go to the NVDIMM unless the remote CPU flushes the cache /
cache lines. On servers where data is written to DRAM (or to an NVDIMM
attached to memory bus), the data can (probably) still be buffered by
the CPU's memory controller.

I am not sure how much control RDMA NICs have over these factors.
AFAIK, there is no PCIe command to flush either cache lines or memory
controller buffers, so flushing to DRAM this is beyond what RDMA NICs
can currently accomplish.

--Anuj (rdma_guy)

On Fri, Mar 11, 2016 at 7:26 PM, Vladislav Bolkhovitin [off-list ref] wrote:

I'm aware of this proposal. Unfortunately, it is quite orthogonal to my question,
because it is about how to ensure persistence of RDMA writes. Atomicity it is
mentioning as well as general RDMA atomicity is atomicity with regard of parallel
commands acting on the same locations. However, I'm asking about power failure
atomicity, which is something different.

For instance, you are doing RDMA WRITE of 10 bytes of data. If a power failure happen
while this operation is in progress, what data will end up on the target location? All
10 bytes new? All 10 bytes old? Or mix of 5 bytes new and five bytes old? Power failure
atomicity I mean is guarantee that the data either old, or new, never mix of old and
new data.

Thanks,
Vlad

Asgeir Eiriksson wrote on 03/10/2016 05:33 PM:

quoted

Vladislav,

This is an area of active R&D

You might be interested in the following (at ietf.org):

Title           : RDMA Durable Write Commit
        Authors         : Tom Talpey
                               Jim Pinkerton
                          <>
      Filename      : draft-talpey-rdma-commit-00.txt
      Pages          : 24
      Date            : 2016-02-19

Regards,

‘Asgeir

quoted

On Mar 10, 2016, at 3:45 PM, Vladislav Bolkhovitin [off-list ref] wrote:

Hello,

I'm currently considering to use NVDIMM behind RDMA and wonder what is RDMA power
failure write atomicity? I mean, what is minimal size and alignment guaranteed to be
written atomically in face of power failure (or some other similar failure), i.e.
either written in full, or not written at all?

For memory writes on Intel it is 8 bytes with 8 bytes alignment. Is there anything like
this for RDMA? Or different vendors/implementation have so different expectations and
promises, so you can not assume anything >1 byte?

I can't find such info anywhere.

Thanks,
Vlad

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html