Re: Question on mlx5 PMD txq memory registration

Thursday, July 20, 2017 7:22 PM, Sagi Grimberg:

quoted

quoted

As I said, there are primitives which are designed to handle frequent
reads and rare mutations.

Even with such primitives, rarely lock is more than never lock.

You do realize that the cache mutation involves ibv_dereg_mr() right?
Any locking scheme for mutation is negligible compared to that, and
rcu_read_lock is practically free.

The fix for the issue you presented includes not to dereg MR on datapath. MR will be deregistered only on device close.
MRs replace on cache can still happen, but it won't involve any de-registration. 

quoted

quoted

quoted

It is one possibility discussed also with Mellanox guys, the point
is this breaks the security point of view which is also an important stuff.

What security aspect? The entire dpdk model builds on top of physical
addresses awareness running under root permissions.
I'm not saying
exposing it to the application nor granting remote permissions to the
physical space.

mlx5_pmd is a network driver, and as a driver, it should allowed to
use the device dma lkey as it sees fit. I honestly think its pretty
much mandatory considering the work-arounds mlx5_pmd tries to do
(which we agreed are broken).

True, There are many PMDs which can work only with physical memory.
However Mellanox NICs have the option to work with virtual one thus

provide more security.

I don't understand the security argument. Its completely private to the
driver. anything under librte is equivalent to an OS wrt networking, so I fail to
see what is the security feature your talking about.

You are correct that as a root you are able to do whatever you want on the server.
The security I refer to is to protect against badly written code.

The fact the PMD only registers the mempools, and use the device engine to translate the VA, provide some protection.
For example, one DPDK process will not be able to access the memory of other DPDK process *by mistake*.

I am not saying using the reserved lkey is not a good suggestion, and we plan to test its value.
All I am saying is there are maybe other option to provide the same performance with the extra protection mentioned above.
One of them can be to use indirect keys. One indirect key to represent 64b memory area, and other regular keys for the hugepages.
The rest of the memory area can be filled with keys pointing to /dev/null. 

quoted

The fact running under root doesn't mean you have privileges to access

every physical page on the server (even if you try very hard to be aware).

But dpdk core mbufs structs are built this way.

quoted

The issue here, AFAIU, is performance.
We are now looking into ways to provide the same performance as if it was

only a single lkey, while preserving the security feature.

Hmm, What exactly do you have in mind?

I'm hoping that you are not referring to ODP. If you are, I think that latency
unpredictability would be a huge non-starter, page-faults are way too
expensive for dpdk users.

No ODP :). 
As all relevant DPDK memory is on top of hugepages, there is no reason to avoid registration and pinning  in advance. 

Again, personally, I don't think that using virtual memory registration are very
useful for a network driver, It's a driver, not an application.

quoted

quoted

quoted

If this is added in the future it will certainly be as an option,
this way both will be possible, the application could then choose
about security vs performance.

Why should the application even be aware of that? Does any other
driver expose the user how it maps pkt mbufs to the NIC? Just like
the MR handling, its 100% internal to mlx5 and no reason why the user
should ever be exposed to any of these details.

Other option is the reserved lkey as you suggested, but it will lose the

security guarantees.

OK, obviously I'm missing something. Can you articulate what do you mean
by "security guarantees"?

quoted

Like every performance optimization it should be the application decision.

I tend to disagree with you on this. The application must never be aware of
the nitty details of the underlying driver implementation. In fact, propagating
this information to the application sounds like a layering violation.

quoted

In fact, there are some discussions on the ML of exposing the option
to use va instead of pa. [1]

My understanding is that the correct proposal was to hide this knowledge
from the user. Also the use-case is not security (which I'm not even sure
what it means yet).

quoted

quoted

quoted

This is also a in progress in PMD part, it should be part of the
next DPDK release.

That is *very* good to hear! Can you guys share a branch? I'm willing
to take it for testing.

The branch is still pre-mature. It may be good enough for external testing in

about two weeks.

quoted

Contact me directly and I will provide it to you.

Awesome!

Thanks.