Re: [PATCH v8 10/15] octeontx2-af: Reconfig MSIX base with IOVA

On Tue, Oct 9, 2018 at 1:27 PM Arnd Bergmann [off-list ref] wrote:

On Tue, Oct 9, 2018 at 9:03 AM Sunil Kovvuri [off-list ref] wrote:

quoted

On Mon, Oct 8, 2018 at 5:38 PM Arnd Bergmann [off-list ref] wrote:

quoted

On Sun, Oct 7, 2018 at 5:01 PM [off-list ref] wrote:

quoted

+       /* HW interprets RVU_AF_MSIXTR_BASE address as an IOVA, hence
+        * create a IOMMU mapping for the physcial address configured by
+        * firmware and reconfig RVU_AF_MSIXTR_BASE with IOVA.
+        */
+       cfg = rvu_read64(rvu, BLKADDR_RVUM, RVU_PRIV_CONST);
+       max_msix = cfg & 0xFFFFF;
+       phy_addr = rvu_read64(rvu, BLKADDR_RVUM, RVU_AF_MSIXTR_BASE);
+       iova = dma_map_single(rvu->dev, (void *)phy_addr,
+                             max_msix * PCI_MSIX_ENTRY_SIZE,
+                             DMA_BIDIRECTIONAL);
+       if (dma_mapping_error(rvu->dev, iova))
+               return -ENOMEM;
+
+       rvu_write64(rvu, BLKADDR_RVUM, RVU_AF_MSIXTR_BASE, (u64)iova);
+       rvu->msix_base_iova = iova;
+

I'm a bit puzzled by how this works. Does this rely on a specific iommu
driver implementation? Normally a physical  address makes no sense to the
implementation backing dma_map_single() that tries to convert a
linear kernel virtual address into a physical address.

        Arnd

I understand what you are pointing at, but we did test this.
IOMMU on this silicon is standard ARM64 SMMUv3.

I didn't doubt that you get the right results, I just couldn't see how ;-)

quoted

All dma_map_single does is virt_to_page and iommu_dma_map_page
converts it back i.e page_to_phys.
So the IOMMU driver gets the same physical address passed above and
creates a iova translation mapping.

For reference below is the captured debug info for the same
=====
[   19.435968] rvu_setup_msix_resources: phy_addr 0x3200000 iova 0xfff80000
[   19.436967] rvu_setup_msix_resources: virt_to_page(phy_addr)
0xffff7fe00000c800 page_to_phys(page) 0x3200000
offset_in_page(phy_addr) 0x00
=====

I think if you enable CONFIG_DEBUG_VIRTUAL, the virt_to_page()
above should trigger a warning in

phys_addr_t __virt_to_phys(unsigned long x)
{
        WARN(!__is_lm_address(x),
             "virt_to_phys used for non-linear address: %pK (%pS)\n",
              (void *)x,
              (void *)x);

        return __virt_to_phys_nodebug(x);
}

Can you verify that?

        Arnd

No, it isn't, as for 64bit systems CONFIG_SPARSEMEM_VMEMMAP is enabled.
But is there any alternative to what is being done ?

Thanks,
Sunil.