On Thu, Jun 28, 2018 at 9:54 AM, Burakov, Anatoly <anatoly.burakov@intel.com

wrote:

On 27-Jun-18 5:52 PM, Alejandro Lucero wrote:

quoted

On Wed, Jun 27, 2018 at 2:24 PM, Burakov, Anatoly <
anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>> wrote:

    On 27-Jun-18 11:13 AM, Alejandro Lucero wrote:



        On Wed, Jun 27, 2018 at 9:17 AM, Burakov, Anatoly
        <anatoly.burakov@intel.com <mailto:anatoly.burakov@intel.com>
        <mailto:anatoly.burakov@intel.com

        <mailto:anatoly.burakov@intel.com>>> wrote:

             On 26-Jun-18 6:37 PM, Alejandro Lucero wrote:

                 This RFC tries to handle devices with addressing
        limitations.
                 NFP devices
                 4000/6000 can just handle addresses with 40 bits implying
                 problems for handling
                 physical address when machines have more than 1TB of
        memory. But
                 because how
                 iovas are configured, which can be equivalent to physical
                 addresses or based on
                 virtual addresses, this can be a more likely problem.

                 I tried to solve this some time ago:

        https://www.mail-archive.com/dev@dpdk.org/msg45214.html
        <https://www.mail-archive.com/dev@dpdk.org/msg45214.html>
                        <https://www.mail-archive.com/
dev@dpdk.org/msg45214.html
        <https://www.mail-archive.com/dev@dpdk.org/msg45214.html>>

                 It was delayed because there was some changes in
        progress with
                 EAL device
                 handling, and, being honest, I completely forgot about
this
                 until now, when
                 I have had to work on supporting NFP devices with DPDK
and
                 non-root users.

                 I was working on a patch for being applied on main DPDK
        branch
                 upstream, but
                 because changes to memory initialization during the
        last months,
                 this can not
                 be backported to stable versions, at least the part
        where the
                 hugepages iovas
                 are checked.

                 I realize stable versions only allow bug fixing, and this
                 patchset could
                 arguably not be considered as so. But without this, it
        could be,
                 although
                 unlikely, a DPDK used in a machine with more than 1TB,
        and then
                 NFP using
                 the wrong DMA host addresses.

                 Although virtual addresses used as iovas are more
        dangerous, for
                 DPDK versions
                 before 18.05 this is not worse than with physical
        addresses,
                 because iovas,
                 when physical addresses are not available, are based on a
                 starting address set
                 to 0x0.


             You might want to look at the following patch:

        http://patches.dpdk.org/patch/37149/
        <http://patches.dpdk.org/patch/37149/>
             <http://patches.dpdk.org/patch/37149/
        <http://patches.dpdk.org/patch/37149/>>

             Since this patch, IOVA as VA mode uses VA addresses, and
        that has
             been backported to earlier releases. I don't think there's
        any case
             where we used zero-based addresses any more.


        But memsegs get the iova based on hugepages physaddr, and for VA
        mode that is based on 0x0 as starting point.

        And as far as I know, memsegs iovas are what end up being used
        for IOMMU mappings and what devices will use.


    For when physaddrs are available, IOVA as PA mode assigns IOVA
    addresses to PA, while IOVA as VA mode assigns IOVA addresses to VA
    (both 18.05+ and pre-18.05 as per above patch, which was applied to
    pre-18.05 stable releases).

    When physaddrs aren't available, IOVA as VA mode assigns IOVA
    addresses to VA, both 18.05+ and pre-18.05, as per above patch.


This is right.

    If physaddrs aren't available and IOVA as PA mode is used, then i as
    far as i can remember, even though technically memsegs get their
    addresses set to 0x0 onwards, the actual addresses we get in
    memzones etc. are RTE_BAD_IOVA.


This is not right. Not sure if this was the intention, but if PA mode and
physaddrs not available, this code inside vfio_type1_dma_map:

if(rte_eal_iova_mode() == RTE_IOVA_VA)

dma_map.iova = dma_map.vaddr;

else

dma_map.iova = ms[i].iova;


does the IOMMU mapping using the iovas and not the vaddr, with the iovas
starting at 0x0.

Yep, you're right, apologies. I confused this with no-huge option.

So, what do you think about the patchset? Could it be this applied to
stable versions?

I'll send a patch for current 18.05 code which will have the dma mask and
the hugepage check, along with changes for doing the mmaps below the dma
mask limit.

--
Thanks,
Anatoly