RE: [PATCH V2 5/6] net: netvsc: Add Isolation VM support for netvsc driver

-----Original Message-----
From: Michael Kelley (LINUX) <redacted>
Sent: Wednesday, November 24, 2021 12:03 PM
To: Tianyu Lan <redacted>; tglx@linutronix.de; mingo@redhat.com; bp@alien8.de;
dave.hansen@linux.intel.com; x86@kernel.org; hpa@zytor.com; luto@kernel.org;
peterz@infradead.org; jgross@suse.com; sstabellini@kernel.org; boris.ostrovsky@oracle.com;
KY Srinivasan [off-list ref]; Haiyang Zhang [off-list ref]; Stephen
Hemminger [off-list ref]; wei.liu@kernel.org; Dexuan Cui [off-list ref];
joro@8bytes.org; will@kernel.org; davem@davemloft.net; kuba@kernel.org; jejb@linux.ibm.com;
martin.petersen@oracle.com; hch@lst.de; m.szyprowski@samsung.com; robin.murphy@arm.com;
Tianyu Lan [off-list ref]; thomas.lendacky@amd.com; xen-
devel@lists.xenproject.org
Cc: iommu@lists.linux-foundation.org; linux-hyperv@vger.kernel.org; linux-
kernel@vger.kernel.org; linux-scsi@vger.kernel.org; netdev@vger.kernel.org; vkuznets
[off-list ref]; brijesh.singh@amd.com; konrad.wilk@oracle.com;
parri.andrea@gmail.com; dave.hansen@intel.com
Subject: RE: [PATCH V2 5/6] net: netvsc: Add Isolation VM support for netvsc driver

From: Tianyu Lan <redacted> Sent: Tuesday, November 23, 2021 6:31 AM

quoted

In Isolation VM, all shared memory with host needs to mark visible to
host via hvcall. vmbus_establish_gpadl() has already done it for
netvsc rx/tx ring buffer. The page buffer used by vmbus_sendpacket_
pagebuffer() stills need to be handled. Use DMA API to map/umap these
memory during sending/receiving packet and Hyper-V swiotlb bounce
buffer dma address will be returned. The swiotlb bounce buffer has
been masked to be visible to host during boot up.

Allocate rx/tx ring buffer via dma_alloc_noncontiguous() in Isolation
VM. After calling vmbus_establish_gpadl() which marks these pages
visible to host, map these pages unencrypted addes space via dma_vmap_noncontiguous().

The big unresolved topic is how best to do the allocation and mapping of the big netvsc
send and receive buffers.  Let me summarize and make a recommendation.

Background
==========
1.  Each Hyper-V synthetic network device requires a large pre-allocated receive
     buffer (defaults to 16 Mbytes) and a similar send buffer (defaults to 1 Mbyte).
2.  The buffers are allocated in guest memory and shared with the Hyper-V host.
     As such, in the Hyper-V SNP environment, the memory must be unencrypted
     and accessed in the Hyper-V guest with shared_gpa_boundary (i.e., VTOM)
     added to the physical memory address.
3.  The buffers need *not* be contiguous in guest physical memory, but must be
     contiguously mapped in guest kernel virtual space.
4.  Network devices may come and go during the life of the VM, so allocation of
     these buffers and their mappings may be done after Linux has been running for
     a long time.
5.  Performance of the allocation and mapping process is not an issue since it is
     done only on synthetic network device add/remove.
6.  So the primary goals are an appropriate logical abstraction, code that is
     simple and straightforward, and efficient memory usage.

Approaches
==========
During the development of these patches, four approaches have been
implemented:

1.  Two virtual mappings:  One from vmalloc() to allocate the guest memory, and
     the second from vmap_pfns() after adding the shared_gpa_boundary.   This is
     implemented in Hyper-V or netvsc specific code, with no use of DMA APIs.
     No separate list of physical pages is maintained, so for creating the second
     mapping, the PFN list is assembled temporarily by doing virt-to-phys()
     page-by-page on the vmalloc mapping, and then discarded because it is no
     longer needed.  [v4 of the original patch series.]

2.  Two virtual mappings as in (1) above, but implemented via new DMA calls
     dma_map_decrypted() and dma_unmap_encrypted().  [v3 of the original
     patch series.]

3.  Two virtual mappings as in (1) above, but implemented via DMA noncontiguous
      allocation and mapping calls, as enhanced to allow for custom map/unmap
      implementations.  A list of physical pages is maintained in the dma_sgt_handle
      as expected by the DMA noncontiguous API.  [New split-off patch series v1 & v2]

4.   Single virtual mapping from vmap_pfns().  The netvsc driver allocates physical
      memory via alloc_pages() with as much contiguity as possible, and maintains a
      list of physical pages and ranges.   Single virtual map is setup with vmap_pfns()
      after adding shared_gpa_boundary.  [v5 of the original patch series.]

Both implementations using DMA APIs use very little of the existing DMA machinery.  Both
require extensions to the DMA APIs, and custom ops functions.
While in some sense the netvsc send and receive buffers involve DMA, they do not require
any DMA actions on a per-I/O basis.  It seems better to me to not try to fit these two
buffers into the DMA model as a one-off.  Let's just use Hyper-V specific code to allocate
and map them, as is done with the Hyper-V VMbus channel ring buffers.

That leaves approaches (1) and (4) above.  Between those two, (1) is simpler even though
there are two virtual mappings.  Using alloc_pages() as in (4) is messy and there's no
real benefit to using higher order allocations.
(4) also requires maintaining a separate list of PFNs and ranges, which offsets some of
the benefits to having only one virtual mapping active at any point in time.

I don't think there's a clear "right" answer, so it's a judgment call.  We've explored
what other approaches would look like, and I'd say let's go with
(1) as the simpler approach.  Thoughts?

I agree with the following goal:
"So the primary goals are an appropriate logical abstraction, code that is
     simple and straightforward, and efficient memory usage."

And the Approach #1 looks better to me as well.

Thanks,
- Haiyang