Re: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing library

-----Original Message-----
From: Elena Agostini <redacted>
Sent: Tuesday, September 7, 2021 00:11
To: Jerin Jacob <redacted>
Cc: Wang, Haiyue <redacted>; NBU-Contact-Thomas Monjalon <redacted>; Jerin
Jacob [off-list ref]; dpdk-dev [off-list ref]; Stephen Hemminger [off-list ref];
David Marchand [off-list ref]; Andrew Rybchenko [off-list ref]; Honnappa
Nagarahalli [off-list ref]; Yigit, Ferruh [off-list ref]; techboard@dpdk.org
Subject: RE: [dpdk-dev] [RFC PATCH v2 0/7] heterogeneous computing library

quoted

quoted

I'd like to introduce (with a dedicated option) the memory API in
testpmd to provide an example of how to TX/RX packets using device

memory.

Not sure without embedding sideband communication mechanism how it
can notify to GPU and back to CPU. If you could share the example API
sequence that helps to us understand the level of coupling with testpmd.

There is no need of communication mechanism here.
Assuming there is not workload to process network packets (to not complicate
things), the steps are:
1) Create a DPDK mempool with device external memory using the hcdev (or gpudev) library
2) Use that mempool to tx/rx/fwd packets

As an example, you look at my l2fwd-nv application here: https://github.com/NVIDIA/l2fwd-nv

To enhance the 'rte_extmem_register' / 'rte_pktmbuf_pool_create_extbuf' ?

	if (l2fwd_mem_type == MEM_HOST_PINNED) {
		ext_mem.buf_ptr = rte_malloc("extmem", ext_mem.buf_len, 0);
		CUDA_CHECK(cudaHostRegister(ext_mem.buf_ptr, ext_mem.buf_len, cudaHostRegisterMapped));
		void *pDevice;
		CUDA_CHECK(cudaHostGetDevicePointer(&pDevice, ext_mem.buf_ptr, 0));
		if (pDevice != ext_mem.buf_ptr)
			rte_exit(EXIT_FAILURE, "GPU pointer does not match CPU pointer\n");
	} else {
		ext_mem.buf_iova = RTE_BAD_IOVA;
		CUDA_CHECK(cudaMalloc(&ext_mem.buf_ptr, ext_mem.buf_len));
		if (ext_mem.buf_ptr == NULL)
			rte_exit(EXIT_FAILURE, "Could not allocate GPU memory\n");

		unsigned int flag = 1;
		CUresult status = cuPointerSetAttribute(&flag, CU_POINTER_ATTRIBUTE_SYNC_MEMOPS, (CUdeviceptr)ext_mem.buf_ptr);
		if (CUDA_SUCCESS != status) {
			rte_exit(EXIT_FAILURE, "Could not set SYNC MEMOP attribute for GPU memory at %llx\n", (CUdeviceptr)ext_mem.buf_ptr);
		}
		ret = rte_extmem_register(ext_mem.buf_ptr, ext_mem.buf_len, NULL, ext_mem.buf_iova, GPU_PAGE_SIZE);
		if (ret)
			rte_exit(EXIT_FAILURE, "Could not register GPU memory\n");
	}
	ret = rte_dev_dma_map(rte_eth_devices[l2fwd_port_id].device, ext_mem.buf_ptr, ext_mem.buf_iova, ext_mem.buf_len);
	if (ret)
		rte_exit(EXIT_FAILURE, "Could not DMA map EXT memory\n");
	mpool_payload = rte_pktmbuf_pool_create_extbuf("payload_mpool", l2fwd_nb_mbufs,
											0, 0, ext_mem.elt_size, 
											rte_socket_id(), &ext_mem, 1);
	if (mpool_payload == NULL)
		rte_exit(EXIT_FAILURE, "Could not create EXT memory mempool\n");