On 18/02/2021 18:23, Jason Gunthorpe wrote:

On Thu, Feb 18, 2021 at 05:52:16PM +0200, Gal Pressman wrote:

quoted

On 18/02/2021 14:53, Jason Gunthorpe wrote:

quoted

On Thu, Feb 18, 2021 at 11:13:43AM +0200, Gal Pressman wrote:

quoted

I'm a bit confused about the meaning of the ibv_req_notify_cq() verb:
"Upon the addition of a new CQ entry (CQE) to cq, a completion event will be
added to the completion channel associated with the CQ."

What is considered a new CQE in this case?
The next CQE from the user's perspective, i.e. any new CQE that wasn't consumed
by the user's poll cq?
Or any new CQE from the device's perspective?

new CQE from the device perspective.

quoted

For example, if at the time of ibv_req_notify_cq() call the CQ has received 100
completions, but the user hasn't polled his CQ yet, when should he be notified?
On the 101 completion or immediately (since there are completions waiting on the
CQ)?

101 completion

It is only meaningful to call it when the CQ is empty.

Thanks, so there's an inherent race between the user's CQ poll and the next arm?

I think the specs or man pages talk about this, the application has to
observe empty, do arm, then poll again then sleep on the cq if empty.

quoted

Do you know what's the purpose of the consumer index in the arm doorbell that's
implemented by many providers?

The consumer index is needed by HW to prevent CQ overflow, presumably
the drivers push to reduce the cases where the HW has to read it from
PCI

Thanks, that makes sense.

I found the following sentence in CX PRM:
"If new CQEs are posted to the CQ after the reporting of a completion event and
these CQEs are not yet consumed, then an event will be generated immediately
after the request for notification is executed."

Doesn't that contradict the expected behavior?